AdvDPO: Adversarial Negative-Augmented Preference Optimization for Text-to-Image Diffusion Models

Abstract

Diffusion Direct Preference Optimization (DPO) provides a reward-free route for aligning text-to-image diffusion models with human preferences, but its supervision is commonly restricted to fixed winner–loser pairs. When dispreferred examples cover only a limited subset of failure modes, optimization can over-emphasize local dataset preferences and provide insufficient corrective contrasts. We present AdvDPO, an adversarial candidate-negative preference optimization framework for diffusion models. AdvDPO couples the policy with an auxiliary adversarial diffusion network that produces a local perturbation direction from the preferred diffusion state. The resulting candidate augments the original preference pair while preserving the prompt, timestep, and noise target. The policy learns from both real and synthesized preference pairs, whereas the adversarial network is alternately trained with reversed preference supervision and an attack objective that keeps synthesized candidates challenging for the current policy. On three test sets and two diffusion backbones, AdvDPO achieves the strongest automatic scores among the compared methods and the largest preference share in the anonymous user study.

1. Introduction

Text-to-image diffusion models have become a leading paradigm for high-fidelity and prompt-conditioned image generation [1,2]. Nevertheless, a pretrained denoising objective and textual conditioning do not fully encode subjective human judgments of visual quality, semantic faithfulness, and perceptual plausibility. Therefore, post-training alignment is required in order to adapt a pretrained generator to the preferences expressed by its users.

Direct Preference Optimization (DPO) is attractive for this purpose because it learns directly from pairwise comparisons without online reinforcement learning or explicit reward maximization [3]. Diffusion DPO transfers this principle to denoising models by comparing the reference-relative denoising errors of preferred and dispreferred images [4]. However, realistic preference datasets provide only a small number of labeled comparisons for each prompt; consequently, the loser side may expose only a narrow subset of possible failure modes, and training on fixed pairs can leave the policy without contrasts against other plausible but undesirable generations.

We address this limitation with AdvDPO, which augments diffusion DPO with adversarially synthesized candidate negatives. AdvDPO couples a diffusion policy with an auxiliary adversarial U-Net that predicts a local perturbation direction from the preferred noisy latent. The synthesized state retains the prompt, timestep, and noise target of the preferred state, which keeps the additional comparison controlled and locally coupled. The policy is optimized with both the original winner–loser pair and the synthesized winner–candidate pair. In alternating steps, the adversary is optimized with label-flipped supervision and an attack term that adapts the synthesized candidate to the current policy. The auxiliary branch is discarded after training, and does not add inference-time cost.

The main contributions of this work are as follows:

• We extend diffusion DPO with adaptive candidate negatives that supplement fixed winner–loser supervision without additional human annotations.
• We introduce a local adversarial perturbation strategy and an alternating objective that preserve diffusion-state comparability while adapting the candidate to the evolving policy.
• Experiments on SD1.5 and SDXL together with component ablations and an anonymous user study show that AdvDPO outperforms the compared baselines on the four preference metrics and receives the highest weighted preference share in the user study.

The remainder of this article is organized as follows: Section 2 reviews text-to-image diffusion and preference alignment methods; Section 3 introduces diffusion training and diffusion DPO; Section 4 presents AdvDPO; Section 5 reports the experiments; and Section 6 provides conclusions and discusses the paper’s limitations.

2. Related Work

2.1. Text-to-Image Diffusion Models

Modern text-to-image diffusion systems build on denoising diffusion probabilistic models [5] and classifier-free guidance [6]. Contrastive Language–Image Pre-training (CLIP) learns shared image–text representations [7]. Large-scale systems such as Imagen [8] have established strong text-conditioned synthesis, while latent diffusion models reduce computational cost by operating in a learned latent space [1]. SDXL subsequently strengthened the architecture and conditioning design for high-resolution generation [2]. Related advances in flow matching [9] and latent adversarial diffusion distillation [10] have further broadened the design space for efficient and high-quality generation.

A recent survey in Electronics identified controllability, semantic grounding, and human-centered quality assessment as central challenges for modern image generation systems [11]. Application-oriented work in the same journal has also shown that LoRA can specialize a pretrained text-to-image diffusion model under limited data and computational resources [12]. In addition, multiscale temporal modeling supports task-specific visual representation learning [13]. Despite these advances in fidelity, prompt understanding, and adaptation efficiency, the underlying generative objective does not clearly distinguish outputs preferred by users from outputs that are merely plausible. This gap motivates a preference-oriented post-training approach.

2.2. Preference Alignment for Diffusion Models

Early text-to-image alignment research developed both evaluation signals and optimization procedures. ImageReward [14] and HPSv2 [15] learn human judgments, while TIFA evaluates text–image faithfulness through question answering [16] and Pick-a-Pic provides large-scale pairwise preference data for direct supervision [17]. RichHF further collects fine-grained human feedback beyond a single scalar judgment [18]. Reward-based optimization includes reinforcement learning fine-tuning in DPOK [19] and direct fine-tuning on differentiable rewards [20], including reward backpropagation in AlignProp [21]. Other methods align semantic concepts or use image understanding feedback, as exemplified by CoMat [22] and DreamSync [23]. While these approaches demonstrate the value of feedback-driven post-training, reward optimization also adds modeling and optimization complexity, and can be sensitive to errors in the feedback signal.

Diffusion DPO avoids an explicit reward model by expressing the pairwise objective through reference-relative denoising errors [4]. Recent work in Electronics has proposed xDPO, which attenuates Kullback–Leibler (KL) regularization and introduces a higher-order divergence term to balance preference improvement against preservation of the pretrained distribution [24]. Subsequent methods improve the objective or the construction of diffusion comparisons. KTO adapts human utility optimization to diffusion alignment [25]; DSPO directly optimizes diffusion scores [26]; InPO [27] and Inversion-DPO [28] use inversion trajectories to improve preference comparisons; SmPO smooths the objective through ReNoise inversion [29]; and SPO targets aesthetic improvement through stepwise preference optimization [30]. Other variants explore noise-aware step level rewards [31], reference-free margin objectives [32], and negative-preference learning with conditional or self-generated supervision [33]. A broader survey discusses these approaches within the general design space of diffusion model alignment [34].

Diffusion-NPO uses a negative-preference model for sampling guidance, while InPO constructs comparisons through inversion and MaPO optimizes a reference-free preference margin. AdvDPO instead couples local candidate construction with online adversarial policy adaptation. Most pairwise approaches still rely on fixed dispreferred examples in which each fixed loser supplies only one observed contrast for a prompt, and may not represent other plausible failure modes; its usefulness can also diminish as the policy improves. AdvDPO supplements these fixed pairs by synthesizing an additional policy-aware candidate locally around the preferred diffusion state. This candidate augments the DPO comparison set during training, and the auxiliary branch is removed before inference.

3. Preliminaries

3.1. Diffusion Models

Let $x_0$ be a clean latent and let t denote a diffusion timestep. The forward process adds Gaussian noise according to

$$x_t=\sqrt{{\overline{\alpha }}_t}{x}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,$$

(1)

where $ϵ\sim \mathcal{N}(0,I)$ and ${\overline{\alpha }}_t$ is determined by the noise schedule. A conditional denoising network ${ϵ}_{\theta }(x_t,t,c)$ predicts the injected noise under text condition c. Its per-sample training error is

$${\mathsf{\ell }}_{\theta }(x_t,t,c)={∥ϵ-{ϵ}_{\theta }(x_t,t,c)∥}_2^2.$$

(2)

The latent-space formulation makes this training procedure practical for high-resolution text-to-image generation [1].

3.2. Direct Preference Optimization for Diffusion Models

Let $(x_0^w,x_0^l,c)$ denote a preference triplet with prompt c, preferred image $x_0^w$, and dispreferred image $x_0^l$. For a sampled timestep and noise, the two images are noised under the same diffusion condition using Equation (1). Following diffusion DPO [4], their preference loss is computed relative to a frozen reference denoiser ${ϵ}_{\mathrm{ref}}$:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{DPO}}(x^a,x^b;\theta )=-\log \sigma \left\{-{\displaystyle \frac{\beta }{2}}\left[({\mathsf{\ell }}_{\theta }^a-{\mathsf{\ell }}_{\mathrm{ref}}^a)-({\mathsf{\ell }}_{\theta }^b-{\mathsf{\ell }}_{\mathrm{ref}}^b)\right]\right\}\end{array}$$

(3)

where $x^a$ is preferred over $x^b$, ${\mathsf{\ell }}_{\theta }^u$ abbreviates the denoising error for sample u, and $\beta$ controls preference strength. The reference-relative terms encourage a larger denoising improvement on the preferred image than on the dispreferred image. AdvDPO retains this reward-free objective and augments its comparison set with adaptive candidate negatives.

4. Method

AdvDPO keeps the standard winner–loser comparison, synthesizes an additional candidate negative from the preferred state, and alternates policy and adversary updates. As illustrated in Figure 1, the synthesized negative is designed so as to remain comparable to the preferred image while exposing a weakness of the current policy.

Figure 1. Overview of AdvDPO. The adversarial U-Net produces a local perturbation from a preferred diffusion state, while the policy learns from both real and synthesized preference pairs. The displayed candidate is a DDIM-decoded visualization; construction and loss computation operate in latent space. Forward arrows show computations, return arrows indicate updates, and flames mark the network updated in each phase.

4.1. Local Adversarial Perturbation

Instead of independently sampling another negative, AdvDPO constructs a local candidate from the preferred sample. Given $x_t^w$, the conditional adversarial network ${\mathcal{A}}_{\varphi }$ predicts a noise-space direction $d_{\varphi }$ and forms

$$\begin{array}{cc}\hfill d_{\varphi }& ={\mathcal{A}}_{\varphi }(x_t^w,t,c),\hfill \\ \hfill x_t^s& =x_t^w-\gamma \sqrt{1-{\overline{\alpha }}_t}{d}_{\varphi },\hfill \end{array}$$

(4)

where $\gamma$ controls the perturbation magnitude. Equation (4) can be obtained by replacing the winner noise with $ϵ+\gamma d_{\varphi }$, reconstructing an $x_0$ estimate, and re-noising it with the original $ϵ$. The shared noise cancels across the reconstruction and re-noising operations, yielding the local update above. The synthesized state $x_t^s$ is evaluated with the same prompt c, timestep t, and target noise $ϵ$ as $x_t^w$, meaning that the construction is a local adversarial perturbation in diffusion-state space. The perturbation strength sampling rule used in the experiments is specified in the training details of Section 5.1.

In the upper panel of Figure 1, frozen VAE encoding and forward noising produce $x_t^w$; the adversary predicts $d_{\varphi }$ and constructs $x_t^s$ through Equation (4). The predictions ${\widehat{ϵ}}_w$ and ${\widehat{ϵ}}_s$ are evaluated under the shared diffusion condition. The lower panel reverses the original preference labels for the adversarial branch, with its errors and reversed-preference loss defined in Equations (5) and (6); the dashed connectors link the synthesized-pair loss to both objectives in Equation (7).

4.2. Adversarial Preference Optimization

The policy ${ϵ}_{\theta }$ and adversary ${\mathcal{A}}_{\varphi }$ are conditional diffusion U-Nets with different roles and trainable parameters. The policy and its frozen reference ${ϵ}_{\mathrm{ref}}$ are initialized from the same generation backbone. The adversary is initialized from a pretrained negative-preference U-Net. SD1.5 fully fine-tunes this U-Net with a separate frozen copy as ${\mathcal{A}}_{{\varphi }_0}$; SDXL adapts it with trainable low-rank adapters, and disabling them gives the frozen adversarial reference. This construction supplies a fixed reference for each branch while allowing the policy and adversary to be optimized independently. The complete parameterization and freezing configuration is reported in the training details of Section 5.1.

For $u\in \{w,l\}$, define

$$\begin{array}{cc}\hfill {\mathsf{\ell }}_{\varphi }^u& ={∥ϵ-{\mathcal{A}}_{\varphi }(x_t^u,t,c)∥}_2^2,\hfill \\ \hfill {\mathsf{\ell }}_{{\varphi }_0}^u& ={∥ϵ-{\mathcal{A}}_{{\varphi }_0}(x_t^u,t,c)∥}_2^2.\hfill \end{array}$$

(5)

The label-flipped term treats $x^l$ as preferred over $x^w$ for the adversarial branch:

$${\mathcal{L}}_{\mathrm{rev}}\left(\varphi \right)=-\log \sigma \left\{-{\displaystyle \frac{{\beta }_{\mathrm{adv}}}{2}}\left[({\mathsf{\ell }}_{\varphi }^l-{\mathsf{\ell }}_{{\varphi }_0}^l)-({\mathsf{\ell }}_{\varphi }^w-{\mathsf{\ell }}_{{\varphi }_0}^w)\right]\right\}.$$

(6)

The factor $1/2$ in Equations (3) and (6) uses the denoising error scaling convention of diffusion DPO. AdvDPO alternates between

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{policy}}& ={\mathcal{L}}_{\mathrm{DPO}}(x^w,x^l;\theta )+{\mathcal{L}}_{\mathrm{DPO}}(x^w,x^s\left(\varphi \right);\theta ),\hfill \\ \hfill {\mathcal{L}}_{\mathrm{adv}}& ={\mathcal{L}}_{\mathrm{rev}}\left(\varphi \right)-{\lambda }_{\mathrm{att}}{\mathcal{L}}_{\mathrm{DPO}}(x^w,x^s\left(\varphi \right);\mathrm{sg}\left(\theta \right)).\hfill \end{array}$$

(7)

On even optimization steps, $\varphi$ is fixed andthe candidate is detached, with the policy updated using the real-pair and synthesized-pair losses. On odd steps, the policy weights are held fixed while gradients are propagated through its prediction on $x^s\left(\varphi \right)$ to update the trainable adversarial parameters. Minimizing the negative attack term maximizes the current policy’s DPO loss on the synthesized pair. The numerical loss weights and optimization settings are consolidated in the training details of Section 5.1.

Training algorithm.

Step 1:

Initialize the policy and its reference from the same backbone. Pretrain the adversary on reversed winner–loser labels and fix the branch references; select full adversarial parameters for SD1.5 or adversarial LoRA parameters for SDXL.

Step 2:

Sample a preference batch, shared t and $ϵ$ within each pair, and one batch-wise $\gamma$; form the noisy winner and loser states and construct $x_t^s$ with Equation (4).

Step 3:

On an even update, detach $x_t^s$, hold $\varphi$ fixed, and minimize ${\mathcal{L}}_{\mathrm{policy}}$ over $\theta$.

Step 4:

On an odd update, fix policy and reference weights while retaining the candidate-dependent computation graph, and minimize ${\mathcal{L}}_{\mathrm{adv}}$ over $\varphi$.

Step 5:

Repeat Steps 2–4 for the configured schedule and retain only the policy for inference.

5. Experiments

We evaluate AdvDPO on SD1.5 and SDXL using three preference-oriented test sets. The experiments include automatic metrics, qualitative comparisons, an anonymous user study, component ablations, and sensitivity analyses of the attack loss weight and perturbation strength.

5.1. Experimental Setup

5.1.1. Base Models and Baselines

We instantiate AdvDPO with SD1.5 [1] and SDXL [2], which differ substantially in model capacity and native image resolution. For SD1.5, we compare against the pretrained backbone, diffusion DPO [4], KTO [25], InPO [27], and Diffusion-NPO [33]. For SDXL, the comparison includes the pretrained backbone, diffusion DPO, MaPO [32], InPO, and Diffusion-NPO. These baselines cover direct pairwise optimization, utility-based alignment, inversion-assisted preference learning, reference-free margin optimization, and negative-preference guidance. Their selection emphasizes representative mechanisms relevant to candidate augmentation rather than an exhaustive comparison with every related alignment method.

5.1.2. Datasets and Evaluation Metrics

Training uses the strict winner–loser pairs from Pick-a-Pic v2 [17]. From 959,040 released training comparisons, we exclude 107,747 ties ($\mathtt{label}\_\mathtt{0}=0.5$) and retain 851,293 strict pairs ($\mathtt{label}\_\mathtt{0}\in \{0,1\}$), containing 928,068 distinct image identifiers across both sides. Original preference labels and the released train/test partitioning are retained. Generation evaluation uses 500 unique Pick-a-Pic test prompts, 3200 HPD prompts, and all 1632 PartiPrompts. We evaluate on Pick-a-Pic [17], PartiPrompts [35], and HPD [15]. Pick-a-Pic represents the pairwise human preference domain used for training, HPD provides a separate preference-oriented benchmark, and PartiPrompts emphasizes broad and compositionally varied text prompts. For every method, one image is generated for each prompt under the same sampling protocol inthe main preference score comparisons. All methods use 50 sampling steps; SD1.5 uses $512\times 512$ images and guidance scale 7.5, while SDXL uses $1024\times 1024$ images and guidance scale 5.0. Training data preparation for AdvDPO follows DiffusionDPO and InPO, although their training budgets and hardware differ. PickScore (PS) [17] predicts pairwise choices learned from Pick-a-Pic, Human Preference Score (HPS) [15] estimates general human preference, Multi-dimensional Preference Score (MPS) [36] aggregates multiple preference dimensions, and ImageReward (IR) [14] provides a learned reward for text–image alignment and visual quality. We report both the mean and median of each metric; the mean summarizes aggregate performance, whereas the median reflects a typical sample and is less sensitive to extreme scores or a skewed score distribution.

5.1.3. Training Details

AdvDPO is trained on eight NVIDIA A100 GPUs (NVIDIA, Santa Clara, CA, USA). The main software environment comprised Python 3.12.0 and PyTorch 2.5.0 (CUDA 12.1). We use AdamW for SD1.5 and Adafactor for SDXL with a learning rate of $1\times {10}^{-8}$, an effective batch size of 1024, and mixed-precision training. The policy U-Net is optimized in full forboth backbones. SD1.5 also fully fine-tunes the adversarial U-Net, while SDXL adapts the adversary with rank-64 LoRA modules using a scaling factor of 32; the reference networks, VAE, and text encoders remain frozen. The policy and adversary are updated alternately with $\beta ={\beta }_{\mathrm{adv}}=2000$ and ${\lambda }_{\mathrm{att}}=0.1$, while $\gamma$ is sampled once per batch from $\mathcal{U}(0.1,1)$. No separate validation set was used for hyperparameter selection. Negative-preference pretraining uses 400 updates with reversed preference labels, followed by 800 alternating steps comprising 400 updates for each branch. The auxiliary adversarial network and any LoRA parameters are removed after training; therefore, inference uses only the optimized policy, and has the same architecture and sampling cost as its backbone.

Computational cost. Table 1 reports resources for the respective training schedules. AdvDPO times include negative-preference pretraining and both alternating branches; GPU hours are time multiplied by GPU count. The policy/adversary trainable parameter counts are 859.52/859.52 million for SD1.5 and 2567.46/92.90 million for SDXL. The smaller overall training budget contributes to AdvDPO’s shorter time than the reported DiffusionDPO schedule, while its additional model, optimizer states, and candidate-dependent backpropagation increase memory use. InPO additionally uses iterative DDIM inversion. Hardware and schedules differ, so these totals do not establish equal per-update cost. In a separately retrained SD1.5 comparison at the same batch size and closely matched compute (55.06 versus approximately 55.20 A100 GPU-hours), AdvDPO improved PS/HPS/MPS/IR over DPO by 0.40/0.45/0.42/0.3004 on PartiPrompts. The complete budget comparison is provided in the Supplementary Materials (SA).

Table 1. Training resources for the reported schedules. Time is estimated; memory is peak allocated memory per GPU.

Backbone	Method	Batch Size (Pairs)	GPUs	Time (h)	GPU- Hours	Memory (GiB)
SD1.5	DPO	2048	16 A100	13.90	222.40	22.03
	InPO	1024	8 A800	7.50	60.00	22.03
	AdvDPO	1024	8 A100	6.90	55.20	63.93
SDXL	DPO	2048	16 A100	65.80	1052.80	36.90
	InPO	1024	8 A100	31.86	254.88	36.90
	AdvDPO	1024	8 A100	49.59	396.72	74.86

5.2. Results and Analysis

5.2.1. Quantitative Results

Table 2 reports the SD1.5 comparison. AdvDPO ranks first in every average and median column. On Pick-a-Pic, its margins over the strongest competing averages are $0.24$ for PS, $0.29$ for HPS, $0.39$ for MPS, and $0.1954$ for IR. The ordering is unchanged on the other test sets: the average PS and MPS margins on HPD are $0.42$ and $0.27$, and the average HPS and IR margins on PartiPrompts are $0.22$ and $0.1219$.

Table 2. SD1.5 comparison: avg./med. scores are reported; higher is better.

Dataset	Method	PS		HPS		MPS		IR
		Avg.	Med.	Avg.	Med.	Avg.	Med.	Avg.	Med.
Pick-a-Pic	SD1.5 [1]	20.67	20.67	26.47	26.51	9.35	9.62	−0.1459	−0.1146
	DPO [4]	21.05	21.02	26.91	27.10	9.82	10.04	0.0727	0.1615
	KTO [25]	21.20	21.18	27.58	27.64	10.03	10.12	0.5339	0.6475
	InPO [27]	21.49	21.51	27.76	27.85	10.29	10.35	0.6212	0.7860
	NPO [33]	21.11	21.06	26.95	27.02	9.97	10.32	−0.0376	0.0396
	AdvDPO	21.73	21.78	28.05	28.12	10.68	10.91	0.8166	1.0625
HPD	SD1.5 [1]	21.05	21.03	27.31	27.31	11.61	11.78	0.1524	0.2213
	DPO [4]	21.48	21.47	27.74	27.76	12.22	12.50	0.3610	0.4330
	KTO [25]	21.68	21.65	28.52	28.57	12.85	13.02	0.6195	0.6998
	InPO [27]	22.05	22.05	28.70	28.75	13.09	13.20	0.8459	1.0005
	NPO [33]	21.40	21.36	27.42	27.41	12.20	12.36	0.0122	0.0924
	AdvDPO	22.47	22.22	28.86	28.86	13.36	13.51	0.9312	1.0983
PartiPrompts	SD1.5 [1]	21.51	21.54	27.48	27.58	9.44	9.59	0.2159	0.3510
	DPO [4]	21.76	21.78	27.80	27.86	9.85	10.05	0.3827	0.5626
	KTO [25]	21.84	21.82	28.31	28.37	10.02	10.27	0.5502	0.6652
	InPO [27]	22.03	22.04	28.46	28.48	10.27	10.47	0.7691	1.0204
	NPO [33]	21.73	21.74	27.61	27.67	9.88	10.10	0.1357	0.3009
	AdvDPO	22.20	22.22	28.68	28.73	10.56	10.72	0.8910	1.1046

Bold values indicate the best result for each metric within each dataset, including ties.

AdvDPO consistently achieves the highest average and median scores across all three test sets. These results demonstrate improvements in both aggregate performance and typical sample quality.

Table 3 gives the results for the larger backbone, with AdvDPO leading every average and median column. Its average score margins on Pick-a-Pic are $0.29$ for PS, $0.39$ for HPS, $0.42$ for MPS, and $0.2437$ for IR. On HPD, the HPS and MPS margins are $0.27$ and $0.35$. The differences are smaller on PartiPrompts, including $0.16$ for PS and $0.0849$ for IR, but all four average metrics remain highest for AdvDPO.

Table 3. SDXL comparison: avg./med. scores are reported; higher is better.

Dataset	Method	PS		HPS		MPS		IR
		Avg.	Med.	Avg.	Med.	Avg.	Med.	Avg.	Med.
Pick-a-Pic	SDXL [2]	22.17	22.24	27.97	27.98	11.54	11.84	0.5679	0.7563
	DPO [4]	22.64	22.62	28.59	28.53	11.99	12.17	0.8034	1.0297
	MaPO [32]	22.32	22.34	28.54	28.63	11.66	11.82	0.6998	0.8463
	InPO [27]	22.58	22.56	28.52	28.64	11.92	12.20	0.8926	1.0770
	NPO [33]	22.22	22.25	28.15	28.18	11.51	11.88	0.5464	0.6061
	AdvDPO	22.93	23.00	28.98	28.95	12.41	12.53	1.1363	1.3944
HPD	SDXL [2]	22.78	22.77	28.63	28.62	14.27	14.43	0.7801	0.8828
	DPO [4]	23.13	23.15	29.17	29.17	14.64	14.79	1.0249	1.1881
	MaPO [32]	22.85	22.82	28.99	29.01	14.36	14.45	0.8749	0.9947
	InPO [27]	23.28	23.23	29.24	29.26	14.66	14.82	1.0466	1.1947
	NPO [33]	22.91	22.86	28.91	28.89	14.33	14.48	0.7795	0.8857
	AdvDPO	23.42	23.38	29.51	29.50	15.01	15.15	1.1412	1.3031
PartiPrompts	SDXL [2]	22.63	22.65	28.42	28.47	11.25	11.67	0.6956	0.9216
	DPO [4]	22.93	22.93	28.91	28.89	11.81	12.11	1.0330	1.2906
	MaPO [32]	22.59	22.54	28.56	28.46	11.28	11.63	0.7888	1.2698
	InPO [27]	23.00	22.99	28.84	28.86	11.65	11.95	1.0248	1.2699
	NPO [33]	22.64	22.57	28.50	28.50	11.18	11.46	0.6571	0.8563
	AdvDPO	23.16	23.16	29.15	29.15	11.90	12.12	1.1179	1.3882

Bold values indicate the best result for each metric within each dataset, including ties.

SD1.5 and SDXL give the same overall ranking despite their different capacities and native resolutions. The agreement among PS, HPS, MPS, and IR also makes the comparison less dependent on the behavior of any single evaluator. Across five independent SD1.5 training runs on Pick-a-Pic and PartiPrompts, AdvDPO has higher mean scores than DPO on all four metrics. A separate repeated-generation analysis uses five matched generation seeds per prompt and 10,000 paired prompt–text–cluster bootstrap resamples; all 95% intervals for AdvDPO-minus-DPO and AdvDPO-minus-InPO mean differences are above zero on both datasets. Full training seed and repeated-generation statistics are provided in the Supplementary Materials (SB).

5.2.2. Qualitative Results

Figure 2 presents prompts with progressively richer constraints, ranging from object depiction and motion to spatial composition and text rendering. For the first two prompts, AdvDPO produces a well-formed single balloon and a coherent pickup truck with clear motion cues. In the third prompt, which requires an Imari vase below a wooden chair, AdvDPO preserves both the object identity and the specified spatial relation. In the text rendering case, it generates a single legible “START”. DPO and InPO also produce coherent balloon depictions, and DPO renders readable “START” text. The four examples in Figure 2 were manually selected to illustrate object depiction, motion, spatial relations, and text rendering; they serve as qualitative illustrations, and are not intended as a statistically representative sample of generated outputs. All methods use seed 0, Euler sampling with 50 steps, guidance scale 5.0, and $1024\times 1024$ resolution. Across these examples, the observed counting, spatial, and textual accuracy qualitatively complements the quantitative results in Table 3.

Figure 2. Qualitative comparison of SDXL-based methods. Columns (left to right): SDXL, DPO, MaPO, InPO, and AdvDPO; rows progress from object depiction to motion, spatial relations, and text rendering. Illustrative examples use identical generation settings and seed 0 across methods.

5.2.3. Compositional Evaluation

We evaluate SD1.5, SFT, DPO, InPO, and AdvDPO on all 553 GenEval prompts [37], with four images per prompt and method (2212 images per method). All use $512\times 512$ resolution, 50 PNDM steps, and guidance scale 9. For each prompt, a generator initialized with seed 42 produced four noise samples matched across methods. The overall score is the unweighted mean of six task accuracy results. Table 4 reports the results.

Table 4. GenEval results under matched generation settings; higher is better.

Method	Single Object	Two Objects	Counting	Colors	Position	Color Binding	Overall
SD1.5	0.9844	0.3308	0.3969	0.7606	0.0250	0.0450	0.4238
SFT	0.9906	0.4242	0.4562	0.8351	0.0375	0.0525	0.4660
DPO	0.9781	0.3864	0.4250	0.8191	0.0375	0.0500	0.4494
InPO	0.9812	0.4747	0.4781	0.8191	0.0700	0.0825	0.4843
AdvDPO	0.9781	0.4899	0.5094	0.8271	0.0900	0.1050	0.4999

Bold values indicate the best result in each metric column, including ties.

AdvDPO achieves the highest overall score (0.4999, versus 0.4843 for InPO and 0.4494 for DPO) and leads in two-object composition, counting, position, and color binding. SFT is strongest on single-object generation and colors. Absolute position and color binding scores remain low even for AdvDPO (0.0900 and 0.1050); individual failures include incorrect object counts, omitted objects, and incorrect color assignment, which are sometimes handled better by DPO or InPO.

5.2.4. Synthesized Candidate Analysis

To examine the candidate-negative assumption, we analyze 200 preferred training images with distinct captions at $t\in \{100,300,500,700\}$ and $\gamma \in \{0.1,0.4,0.7,1.0\}$, yielding 3200 comparisons. Both states undergo the same DDIM reverse sampling and VAE decoding; the matched decoded winner is the baseline, since decoding at larger t can itself alter the source image.

Figure 3 quantifies relative latent displacement, CLIP image cosine similarity, and automatic preference ordering. From $(t,\gamma )=(100,0.1)$ to $(700,1.0)$, mean relative displacement increases from 0.0288 to 0.9256 and image similarity decreases from 0.9959 to 0.5487. Direct prompt alignment also changes, with the mean CLIP text–image cosine difference from the matched winner being $+0.0002$ and $-0.1097$ at these endpoints. These results indicate that stronger perturbations can reduce semantic consistency despite shared diffusion conditioning. Matched winners receive a higher PickScore in 85.4% of comparisons (2733/3200), ranging from 59.0% to 97.0% at the endpoints. The automatic preference rankings and matched visual comparisons support the use of synthesized samples as candidate negatives, while human preference annotations are associated with the original training pairs.

Figure 3. SD1.5 winner–candidate analysis over 200 preferred images at each timestep–strength setting: (a) mean relative latent displacement $\parallel x_t^s-x_t^w{\parallel }_2/{\parallel x_t^w\parallel }_2$, (b) mean CLIP image cosine similarity between candidate and matched decoded winner, and (c) percentage receiving higher PickScore for the matched winner; panel (c) reports automatic predictions.

Figure 4 shows that small perturbations remain close to this baseline, while larger strengths and timesteps reduce detail, sharpness, and structural fidelity. Additional matched visualizations are provided in the Supplementary Materials (SF).

Figure 4. SD1.5 synthesized candidate visualizations: Pikachu with a small black dog (top) and an outdoor portrait (bottom). Rows show $t=100,300,500,700$; columns show the source winner, matched decoded winner, and candidates with $\gamma =0.1,0.4,0.7,1.0$. The matched winner and candidates use the same DDIM reverse sampling and VAE decoding.

5.2.5. User Study

We additionally conducted an anonymous user study over a pool of 128 image groups generated by SDXL, DPO, MaPO, InPO, and AdvDPO. Twenty-two participants were each randomly assigned up to 50 group views, yielding 603 completed group-level evaluations; completion counts varied by participant. Method identities were hidden as Images A–E, and display positions were randomized for each participant–group combination. For each displayed group, participants were instructed to select one best image, or two as jointly best when they considered them equally good and difficult to distinguish. Each completed evaluation contributed a single point, divided equally among all selected methods; consequently, each method receives 0.5 points in the case of an observed two-way tie. The weighted share is the method’s accumulated score divided by the sum of weighted scores over all methods. For each AdvDPO-versus-baseline comparison, W denotes selection of AdvDPO without the baseline, L selection of the baseline without AdvDPO, T denotes selection of both as jointly best, and U denotes selection of neither. Another method may be co-selected in W or L; only T denotes a tie, while U leaves the relative preference between the two unselected methods unobserved. The decisive win rate is $W/(W+L)$ and excludes both T and U. Table 5 summarizes the user study results.

Table 5. Anonymous user study results: W/L/T/U is reported from the AdvDPO perspective, and the decisive win rate excludes both T and U.

Method	Sole Best	Tied Best	Selected	Weighted Score	Share	AdvDPO W/L/T/U	Decisive Win Rate
AdvDPO	165	61	226	195.5	32.42%	–	–
InPO	110	58	168	139.0	23.05%	202/144/24/233	58.38%
DPO	110	56	166	138.0	22.89%	200/140/26/237	58.82%
MaPO	51	33	84	67.5	11.19%	218/76/8/301	74.15%
SDXL	48	30	78	63.0	10.45%	223/75/3/302	74.83%

Bold values indicate the highest counts, weighted score, and preference share.

AdvDPO receives the largest weighted share at 32.42%, compared with 23.05% for InPO and 22.89% for DPO. After excluding T and U, AdvDPO wins 58.38% of decisive comparisons against InPO and 58.82% against DPO; the corresponding rates against MaPO and the SDXL base model are 74.15% and 74.83%. Across the four pairwise comparisons, T and U together account for 42.62–51.24% of responses, whereas 119 of the 603 evaluations (19.73%) identify two joint-best outputs. The decisive win rate is conditional on selecting one compared method without the other, and should be interpreted together with the weighted share. The weighted share uses all sole and tied selections, while the decisive win rate uses only responses that distinguish AdvDPO from a given baseline. AdvDPO leads under both calculations, matching its ranking under the automatic metrics in Table 2 and Table 3. Additional agreement, confidence interval, and forced choice analyses are provided in the Supplementary Materials (SC).

5.3. Ablation Study

5.3.1. Component Analysis

Table 6 reports the SDXL ablation on PartiPrompts. The DPO row keeps only ${\mathcal{L}}_{\mathrm{DPO}}(x^w,x^l)$ in ${\mathcal{L}}_{\mathrm{policy}}$ and uses the same evaluation as Table 3, yielding identical mean scores. The fixed-${\mathcal{A}}_{\varphi }$ variant uses synthesized pairs but freezes an adversarial network pretrained with ${\mathcal{L}}_{\mathrm{rev}}$, while the w/o ${\mathcal{L}}_{\mathrm{rev}}$ variant removes label-flipped supervision and the w/o negative attack term variant removes policy-adaptive adversarial training. Relative to DPO, the fixed adversary improves PS, HPS, MPS, and IR by $0.16$, $0.10$, $0.06$, and $0.0774$, respectively; thus, adding controlled synthesized comparisons is beneficial even before the adversary is adapted to the current policy.

Table 6. SDXL ablation on PartiPrompts. Mean scores are reported.

Variant	PS	HPS	MPS	IR
SDXL Base	22.63	28.42	11.25	0.6956
${\mathcal{L}}_{\mathrm{DPO}}(x^w,x^l)$	22.93	28.91	11.81	1.0330
DPO + random candidate	22.63	28.42	11.26	0.6984
${\mathcal{L}}_{\mathrm{policy}}$ w/fixed ${\mathcal{A}}_{\varphi }$	23.09	29.01	11.87	1.1104
w/o ${\mathcal{L}}_{\mathrm{rev}}$	23.17	29.13	11.81	1.1043
w/o attack term	22.85	28.73	11.85	1.0384
AdvDPO	23.16	29.15	11.90	1.1179

Bold values indicate the best result in each metric column, including ties.

Removing ${\mathcal{L}}_{\mathrm{rev}}$ changes PS from 23.16 to 23.17 while reducing HPS from 29.15 to 29.13, MPS from 11.90 to 11.81, and IR from 1.1179 to 1.1043. Therefore, the complete objective achieves the best HPS, MPS, and IR along with the second-highest PS, with a difference of 0.01. We hypothesize that label-flipped supervision helps ${\mathcal{A}}_{\varphi }$ to retain loser-oriented denoising behavior instead of adapting exclusively to current policy weaknesses.

Removing the attack term reduces PS and HPS to 22.85 and 28.73, below the DPO values of 22.93 and 28.91; its MPS and IR gains over DPO are only $0.04$ and $0.0054$. Thus, after adversarial updating begins, ${\mathcal{L}}_{\mathrm{rev}}$ alone does not preserve the gains obtained by the complete objective. The two terms serve different roles: label-flipped supervision maintains loser-oriented behavior, while the attack term directs the candidate toward errors of the current policy; the aggregate result favors retaining both. The random-candidate control replaces $d_{\varphi }$ with $z\sim \mathcal{N}(0,I)$ in Equation (4) while retaining the same batch-wise $\gamma$, diffusion condition, and one additional pair; it uses 400 policy updates. Its scores are below policy-only DPO on all four metrics, showing that extra random negatives alone do not reproduce AdvDPO’s gains. The w/o ${\mathcal{L}}_{\mathrm{rev}}$ variant uses attack-only adaptation, while the w/o attack variant retains reversed-preference synthesis.

5.3.2. Sensitivity to Attack Loss Weight

After fixing the reported configuration, we conducted a post hoc sensitivity analysis of the attack loss weight ${\lambda }_{\mathrm{att}}$ on PartiPrompts with the SDXL backbone while keeping all other settings fixed. As shown in Table 7, increasing ${\lambda }_{\mathrm{att}}$ from 0 to $0.1$ improves PS from $22.85$ to $23.16$, HPS from $28.73$ to $29.15$, MPS from $11.85$ to $11.90$, and IR from $1.0384$ to $1.1179$. The result at ${\lambda }_{\mathrm{att}}=0$ coincides with the w/o attack term variant in Table 6, while ${\lambda }_{\mathrm{att}}=0.05$ already recovers part of the performance gained by the complete objective.

Table 7. SDXL attack loss weight ablation on PartiPrompts. Mean scores are reported; higher is better.

${\mathit{\lambda }}_{\mathbf{att}}$	PS	HPS	MPS	IR
0	22.85	28.73	11.85	1.0384
$0.05$	22.94	29.04	11.88	1.0877
$0.1$	23.16	29.15	11.90	1.1179
$0.5$	23.14	29.12	11.90	1.1036
1	23.04	29.08	11.89	1.0883

Bold values indicate the best result in each metric column, including ties.

The prespecified setting ${\lambda }_{\mathrm{att}}=0.1$ achieves the highest PS, HPS, and IR and ties for the highest MPS. At $0.5$, PS and HPS are lower by $0.02$ and $0.03$, MPS remains tied, and IR decreases to $1.1036$. All four metrics decline further at ${\lambda }_{\mathrm{att}}=1$.

5.3.3. Sensitivity to Perturbation Strength

After the reported configuration had been fixed, we similarly conducted a post hoc sensitivity analysis of the perturbation coefficient $\gamma$ on PartiPrompts with the SDXL backbone while keeping the data, optimization schedule, and all other hyperparameters fixed. As shown in Table 8, increasing the fixed coefficient from $0.1$ to $0.4$ improves PS from $22.90$ to $23.12$, HPS from $28.93$ to $29.10$, MPS from $11.82$ to $11.85$, and IR from $1.0388$ to $1.1054$. At $\gamma =0.7$, PS and HPS are slightly lower than at $0.4$, whereas MPS and IR are slightly higher. At $\gamma =1$, all four metrics decrease relative to the intermediate settings, showing a non-monotonic response to the fixed perturbation strength. The prespecified batch-wise sampling strategy $\gamma \sim \mathcal{U}(0.1,1)$ achieves the best value on every metric, with $23.16$ PS, $29.15$ HPS, $11.90$ MPS, and $1.1179$ IR. Thus, within the evaluated settings, sampling a range of perturbation magnitudes provides stronger aggregate performance than using a single fixed scale.

Table 8. SDXL perturbation strength ablation on PartiPrompts. Mean scores are reported; higher is better.

$\mathit{\gamma }$ Setting	PS	HPS	MPS	IR
$0.1$	22.90	28.93	11.82	1.0388
$0.4$	23.12	29.10	11.85	1.1054
$0.7$	23.10	29.05	11.86	1.1072
1	23.00	29.01	11.81	1.0531
$\mathcal{U}(0.1,1)$ (default)	23.16	29.15	11.90	1.1179

Bold values indicate the best result in each metric column, including ties.

6. Conclusions and Limitations

6.1. Conclusions

This paper has presented AdvDPO, a preference optimization framework that augments diffusion DPO with locally-perturbed adversarially synthesized candidate negatives. The perturbation preserves the prompt, timestep, and noise target while supplying adaptive comparisons beyond fixed winner–loser pairs. Alternating optimization updates the policy with both real and synthesized preference pairs while training an adversarial diffusion U-Net to generate informative directions. Across the evaluated datasets and the SD1.5 and SDXL backbones, AdvDPO outperforms the evaluated baselines on the four preference evaluation metrics and obtains the highest weighted preference share in the anonymous user study.

6.2. Limitations

AdvDPO trains both the policy and adversarial networks on offline preference data, so unsafe or biased patterns in those data can be inherited during alignment. Alongside the preference alignment evaluation, we report an exploratory SafetyChecker analysis in Supplementary Materials (SJ). Its findings motivate incorporating explicit safety-aware preference objectives and broader safety evaluation protocols.

AdvDPO requires an additional adversarial U-Net and optimizer states during training. Backpropagation through candidate-dependent policy predictions increases intermediate activations, leading to higher peak memory and limiting feasible batch sizes. Negative-preference pretraining and alternating optimization add training stages and model evaluations; only inference avoids the auxiliary branch.

Finally, the synthesized negative samples were evaluated using automatic preference predictions from PickScore rather than direct human preference judgments.