Diffusion Preference Alignment via Attenuated Kullback–Leibler Regularization

Abstract

Direct preference optimization (DPO) has been successfully applied to align large language models (LLMs) with human preferences. In recent years, DPO has also been used to improve the generation quality of text-to-image diffusion models. However, existing techniques often rely on a single type of reward model. They are also prone to overfitting to inaccurate reward signals. As a result, model quality cannot be continuously improved. To address these limitations, we propose xDPO. This method introduces a novel regularization approach that implicitly defines reward functions for both preferred and non-preferred samples. This design greatly enhances the flexibility of reward modeling. The experimental results show that, after fine-tuning Stable Diffusion v1.5, xDPO achieves significant improvements in human preference evaluations compared to previous DPO methods. It also improves training efficiency by approximately 1.5 times. Meanwhile, xDPO maintains image–text alignment performance that is comparable to the original model.

1. Introduction

In recent years, diffusion models have emerged as a powerful generative framework capable of efficiently sampling from and learning complex data distributions [1,2,3]. However, effectively aligning the outputs of such models with human aesthetics and preferences—particularly in the text-to-image (T2I) domain—remains a significant challenge. The prevailing approach is to first pretrain on large-scale unlabeled datasets to acquire general generative capabilities, followed by fine-tuning on specific downstream tasks or directly leveraging human feedback. This fine-tuning aims to make the model outputs better conform to human preferences while preserving the performance of the pretrained model [4]. Fine-tuning is typically formulated as a distribution optimization problem, where a regularization term is added to encourage the optimized distribution to remain close to the pretrained distribution. Representative methods include reinforcement learning from human feedback (RLHF) [5] and direct preference optimization (DPO) [6,7].

However, empirical studies have shown that DPO suffers from over-optimization issues [8], leading to a degraded quality of generated samples. To address this problem, we propose a novel joint regularization technique that transforms the implicit reward function from a fixed form into a more flexible and designable one, and apply this method to diffusion models.

Our main contributions include:

• We introduce a higher-order divergence regularization method, incorporating a tunable coefficient $\eta$ and higher-order divergence terms into the conventional KL regularization. This enables a more refined implicit reward function design, surpassing previous approaches that rely solely on simple KL penalties.
• For diffusion models, we redefine the data likelihood and construct the xDPO framework, from which we derive a concise and efficient loss function to achieve direct preference optimization.
• Through comprehensive experiments, xDPO demonstrates approximately 1.6 times the training efficiency of Diffusion-DPO significantly improves the quality of generated images (see Figure 1), and receives consistent approval from both human evaluators and preference evaluation models.

2. Related Work

2.1. Text-to-Image Generative Models

Diffusion models convert Gaussian noise into images via an iterative denoising process [2,11,12]. Text-to-image (T2I) generative models are designed to produce high-quality, high-fidelity images conditioned on input text prompts. Moreover, these models have driven advances in related areas such as image editing [13], video generation [14], and 3D modeling [15]. However, state-of-the-art diffusion models are typically trained on large-scale, noisy datasets and often exhibit systematic biases relative to human preferences, indicating that there remains substantial room for improvement.

2.2. Diffusion Models Alignment

The alignment of text-to-image (T2I) diffusion models with human preferences has garnered widespread attention. In this domain, supervised fine-tuning (SFT) remains the most commonly used approach for preference alignment. Inspired by the success of Reinforcement Learning from Human Feedback (RLHF) in large language models [4,16,17], researchers have begun developing image-preference reward models. Representative examples include an aesthetic predictor leveraging learned visual embeddings [18], the ImageReward framework [19], the PickScore metric for aesthetic comparison [20], and the HPSv2 human preference scorer [9].

Building upon these human-preference reward models, [21] proposed Diffusion Policy Optimization for Knowledge (DPOK), while [22] introduced Denoising Diffusion Policy Optimization (DDPO). Both methods formalize the denoising (or sampling) process of diffusion models as a Markov decision process and employ policy gradient techniques for fine-tuning. Some approaches improve preference alignment by adjusting the training data distribution to favor samples with strong visual appeal and highly consistent textual descriptions [23,24,25]. Additionally, regenerating captions for pre-collected web images has been used to enhance the accuracy and richness of textual annotations [26,27], further advancing preference alignment in text-to-image diffusion models.

Additionally, Diffusion-DPO [7] and D3PO [28] leverage the duality between reward learning and policy optimization inherent in direct preference optimization (DPO), fine-tuning solely on offline human preference pairs at each denoising step; DenseReward [29] further assigns higher weights to early denoising steps within this framework. Diffusion-KTO [30] replaces DPO with Kahneman–Tversky Optimization (KTO) [31], relying exclusively on binary feedback per image for fine-tuning. However, DPO depends on KL regularization, resulting in overly simplistic implicit reward functions. DPO techniques often suffer from the learned policy $p_{\theta }$ gradually diverging from the reference policy $p_{\mathrm{ref}}$ during fine-tuning, resulting in degraded model performance. To mitigate this over-optimization issue, the $\chi$PO framework [32] introduces a ${\chi }^2$-divergence term alongside the KL divergence to constrain the distance between the reference distribution $p_{\mathrm{ref}}$ and the target distribution $p_{\theta }$. Inspired by this, we incorporate a tunable coefficient $\eta$ into the original KL regularization and integrate a higher-order divergence regularizer to more precisely control the discrepancy between $p_{\theta }$ and $p_{\mathrm{ref}}$. Experimental results demonstrate that our method achieves significant performance improvements compared to Diffusion-DPO.

3. Preliminaries

3.1. Diffusion Models

This process comprises a forward (diffusion) phase and a reverse (denoising) phase [1,2,3,33]. In Denoising Diffusion Probabilistic Models (DDPMs) [2], image generation is cast as a Markov chain. In the forward pass, Gaussian noise $ϵ\sim \mathcal{N}(0,I)$ is gradually added to a clean sample $x_0\sim p_{\mathrm{data}}\left(x_0\right)$ over T timesteps under a noise schedule governed by ${\alpha }_t$ and ${\sigma }_t$. Concretely, for each $t\in [0,T]$, one has $x_t={\alpha }_t{x}_0+{\sigma }_tϵ$. In the reverse (denoising) phase, generation proceeds by starting from pure Gaussian noise and iteratively removing noise via a learned network ${ϵ}_{\theta }$, ultimately recovering a high-quality image. During training, the model parameters $\theta$ are optimized by minimizing the evidence lower bound (ELBO). A common reparameterization of this objective is

$$\begin{array}{c}\hfill L_{\mathrm{DDPM}}={\mathbb{E}}_{x_0,t,ϵ}\left[\lambda \left(t\right){∥ϵ-{ϵ}_{\theta }(x_t,c,t)∥}^2\right],\end{array}$$

(1)

where $t\sim \mathcal{U}(0,T)$,$x_t\sim q(x_t\mid x_0)=\mathcal{N}\left(\sqrt{{\overline{\alpha }}_t}{x}_0,(1-{\overline{\alpha }}_t)I\right)$, $\lambda \left(t\right)$ is a weighting function dependent on the timestep, and c denotes any conditioning information.

3.2. Human Preference Optimization

3.2.1. RLHF

RLHF employs maximum likelihood estimation to fit the Bradley–Terry (BT) model on the dataset $\mathcal{D}$, thereby obtaining an explicit parameterized reward model r. The BT loss is defined as

$$\begin{array}{c}\hfill L_{BT}\left(\varphi \right):=-{\mathbb{E}}_{(x_0^w,x_0^l,c)\sim \mathcal{D}}\left[log\sigma \left(r_{\varphi }(x_0^w,c)-r_{\varphi }(x_0^l,c)\right)\right].\end{array}$$

(2)

Reinforcement Learning from Human Feedback (RLHF) proceeds by first learning a reward model $r(x,c)$, then optimizing the conditional generation distribution $p_{\theta }(x_{0:T}\mid c)$ to maximize expected reward while regularizing its divergence from a pretrained reference distribution $p_{\mathrm{ref}}(x_0\mid c)$. Concretely, the training objective can be written as

$$\begin{array}{c}\hfill L_{\mathrm{RLHF}}={\mathbb{E}}_{c\sim D}{\mathbb{E}}_{x_0\sim p_{\theta }(x_0\mid c)}\left[r(x_0,c)\right]-\beta D_{\mathrm{KL}}\left[p_{\theta }(x_0\mid c)\parallel p_{\mathrm{ref}}(x_0\mid c)\right].\end{array}$$

(3)

where $D_{\mathrm{KL}}[p_{\theta }\parallel p_{\mathrm{ref}}]$ measures the KL divergence between the learned policy $p_{\theta }(y\mid x)$ and the reference policy $p_{\mathrm{ref}}(y\mid x)$, preventing $p_{\theta }$ from collapsing and helping to maintain output diversity. The hyperparameter $\beta >0$ balances the expected reward against the KL penalty.

3.2.2. Direct Preference Optimization

Direct preference optimization (DPO) is a fine-tuning paradigm that leverages the correspondence between the optimal policy $p^{\ast }(y\mid x)$ and the reward model $r(x,y)$ via $r(x,c)=\beta log{\displaystyle \frac{p_{\theta }(x\mid c)}{p_{\mathrm{ref}}(x\mid c)}}+\beta logZ\left(c\right),$ and then optimizes the model parameters using human-preference triplets $(c,x_0^{+},x_0^{-})$. The DPO training objective can be written as

$$\begin{array}{c}\hfill L_{\mathrm{DPO}}\left(\theta \right)=-{\mathbb{E}}_{c,x_0^{+},x_0^{-}}\left[log\sigma \left(\beta log\frac{p_{\theta }(x_0^{+}\mid c)}{p_{\mathrm{ref}}(x_0^{+}\mid c)}-\beta log\frac{p_{\theta }(x_0^{-}\mid c)}{p_{\mathrm{ref}}(x_0^{-}\mid c)}\right)\right]\end{array}$$

(4)

where $\beta$ controls the extent to which $p_{\theta }(x\mid c)$ may deviate from the reference distribution $p_{\mathrm{ref}}(x\mid c)$.

Diffusion-DPO adapts the DPO algorithm from Equation (4), which is used to optimize $p_{\theta }$, for application to diffusion models by casting the generation process as a Markov decision process (MDP). The corresponding objective is defined as follows:

$$\begin{array}{c}\hfill L_{\mathrm{Diffusion}-\mathrm{DPO}}=-\mathbb{E}\left[log\sigma \left(\beta log{\displaystyle \frac{p_{\theta }(x_t^w\mid x_{t+1}^w,c)}{p_{\mathrm{ref}}(x_t^w\mid x_{t+1}^w,c)}}-\beta log{\displaystyle \frac{p_{\theta }(x_t^l\mid x_{t+1}^l,c)}{p_{\mathrm{ref}}(x_t^l\mid x_{t+1}^l,c)}}\right)\right]\end{array}$$

(5)

where $x_0^w,x_0^l,c\sim \mathcal{D},t\sim \mathcal{U}(0,T),x_{t,t+1}^w\sim p\left(x_{t,t+1}^w\mid x_0^w\right),x_{t-1,t}^l\sim p\left(x_{t-1,t}^l\mid x_0^l\right).$ The DPO-based approach can directly optimize the target distribution $p_{\theta }$ without relying on reinforcement learning algorithms.

4. Method

Consider a dataset annotated with human preferences, denoted as ${\mathcal{D}}_{\mathrm{pref}}=(c,x_0^w,x_0^l)$, where each entry comprises a text prompt c and two associated images: $x_0^w$ (the “preferred” sample) and $x_0^l$ (the “non-preferred” sample). We wish to train a generation policy ${\pi }_{\theta }$ that, when sampling $x_0\sim p_{\theta }(·\mid c)$, preferentially outputs higher-quality, human-favored images. To improve the optimization of the generation policy ${\pi }_{\theta }$, we implicitly design the reward function. In particular, xDPO introduces a coefficient $\eta$ to adjust the weight of the KL regularizer and further incorporates a higher-order divergence term defined by a generator function: $g\left(z\right)=\eta z+\frac{1}{n+1}\gamma z^{n+1}$, thereby imposing a stronger penalty on deviations from the reference policy $p_{\mathrm{ref}}$. Accordingly, the resulting reinforcement learning objective becomes

$$\begin{array}{cc}\hfill \underset{p_{\theta }}{max}{\mathbb{E}}_{c\sim {\mathcal{D}}_c}& {\mathbb{E}}_{x_0\sim p_{\theta }(x_0\mid c)}\left[r(x_0,c)\right]\hfill \\ & -\beta {\mathbb{E}}_{c\sim {\mathcal{D}}_c}\left[\eta D_{\mathrm{KL}}\left(p_{\theta }(·\mid c)\parallel {\pi }_{\mathrm{ref}}(·\mid c)\right)+D_g\left(p_{\theta }(·\mid c)\parallel p_{\mathrm{ref}}(·\mid c)\right)\right]\hfill \end{array}$$

(6)

where the hyperparameter $\beta >0$ controls the overall strength of the combined regularization. Treating the diffusion process as a multi-step Markov decision process (MDP) and following [7,21,28,30], we optimize a reward model at each time step to obtain the following objective:

$$\begin{array}{cc}\hfill L_{\mathrm{RLHF}}=& {\mathbb{E}}_{c\sim D}{\mathbb{E}}_{x_{0:T}\sim p_{\theta }(·\mid c)}\left[\sum _{t=0}^{T-1}r(x_t,c)\right]\hfill \\ \hfill & -\beta \left(\eta D_{\mathrm{KL}}\left[p_{\theta }(x_{0:T}\mid c)\parallel p_{\mathrm{ref}}(x_{0:T}\mid c)\right]+D_g\left[p_{\theta }(x_{0:T}\mid c)\parallel p_{\mathrm{ref}}(x_{0:T}\mid c)\right]\right),\hfill \end{array}$$

(7)

where $p_{\mathrm{ref}}(x_{0:T}\mid c)$ is the reference distribution given by the pretrained diffusion model, and $\beta >0$ weights the combined regularization terms.

Here, we define the composite divergence $D_f\left(\pi \parallel {\pi }_{\mathrm{ref}}\right)={\mathbb{E}}_{c\sim D}\left[\eta D_{\mathrm{kl}}\left[\pi (·\mid c)\parallel {\pi }_{\mathrm{ref}}(·\mid c)\right]+D_{\mathrm{g}}\left[\pi (·\mid c)\parallel {\pi }_{\mathrm{ref}}(·\mid c)\right]\right]$ with generator $f\left(z\right):=\eta \left(zlogz-z\right)+{\displaystyle \frac{1}{n+1}}\gamma z^{n+1}$, and use $\beta$ to control the overall strength of this regularization. We present the divergences and corresponding derivatives of xDPO and Diffusion-DPO in

Table 1. The divergences and corresponding derivatives of xDPO and Diffusion-DPO.

	$\mathit{f}\mathbf{\left(}\mathit{z}\mathbf{\right)}$	${\mathit{f}}^{\mathbf{\prime }}\mathbf{\left(}\mathit{z}\mathbf{\right)}$
Diffusion-DPO (Reverse KL)	$zlogz$	$logz$
xDPO	$\eta \left(zlogz-z\right)+{\displaystyle \frac{1}{n+1}}\gamma z^{n+1}$	$\eta logz+\gamma z^n$

. The derivative of the combined KL + $g\left(z\right)$ regularizer is $f_{\chi }^{\prime }\left(z\right)=\eta logz+\gamma z^n$, which satisfies $0\notin \mathrm{dom}\left(f_{\chi }^{\prime }\left(z\right)\right)$. Consequently, in the text-to-image diffusion setting (see Figure 2), one can reparameterize the objective as in Equation (7) of [34].

$$\begin{array}{c}\hfill \sum _{t=0}^{T-1}r(x_t,c)=\beta \left(\eta ln{\displaystyle \frac{p_{\theta }(x_{0:T}\mid c)}{p_{\mathrm{ref}}(x_{0:T}\mid c)}}+\gamma {\left({\displaystyle \frac{p_{\theta }(x_{0:T}\mid c)}{p_{\mathrm{ref}}(x_{0:T}\mid c)}}\right)}^n\right).\end{array}$$

(8)

Accordingly, we define the per-step reward as

$$\begin{array}{c}\hfill r(x_t,c)=\beta \left(\eta ln{\displaystyle \frac{p_{\theta }(x_t\mid x_{t+1},c)}{p_{\mathrm{ref}}(x_t\mid x_{t+1},c)}}+\gamma {\left({\displaystyle \frac{p_{\theta }(x_t\mid x_{t+1},c)}{p_{\mathrm{ref}}(x_t\mid x_{t+1},c)}}\right)}^n\right).\end{array}$$

(9)

Invoking the reverse process of the T2I diffusion model $p_{\theta }(x_t\mid x_{t+1},c)$ (see Equation (1), this reward can be written in closed form:

$$\begin{array}{cc}\hfill r(x_t,c)& =-\beta \omega (\hfill \\ & \eta {∥{ϵ}_{\theta }(x_{t+1},t+1)+{ϵ}_{t+1}∥}_2^2+\gamma exp\left(n\parallel {ϵ}_{\theta }(x_{t+1},t+1)-{ϵ}_{t+1}{\parallel }_2^2\right)\hfill \\ & -\eta {∥{ϵ}_{\mathrm{ref}}(x_{t+1},t+1)-{ϵ}_{t+1}∥}_2^2-\gamma exp\left(n\parallel {ϵ}_{\mathrm{ref}}(x_{t+1},t+1)-{ϵ}_{t+1}{\parallel }_2^2\right)).\hfill \end{array}$$

(10)

where $\omega ={\displaystyle \frac{{\beta }_t{\alpha }_{t-1}}{2(1-{\overline{\alpha }}_{t-1}){\alpha }_t}}$ is typically set as a constant in practice [2,35], ${ϵ}_{t+t}\sim \mathcal{N}(0,I)$ represents the noises added in the forward (diffusion) process for “preferred” (w) and “less preferred” (l) samples, and ${ϵ}_{\mathrm{ref}}$ denotes the denoising network of the reference model. This loss optimizes the difference in mean-squared error between the learned denoiser ${ϵ}_{\theta }$ and the reference denoiser ${ϵ}_{\mathrm{ref}}$ on preferred versus non-preferred noisy images.

We generalize the optimization of $p_{\theta }$ in the Bradley–Terry preference model (Equation (2)) to the diffusion setting. By substituting Equation (10) into Equation (2) and rearranging terms, we obtain the following overall training objective:

$$\begin{array}{cc}\hfill L_{\mathrm{xDPO}}& ={\mathbb{E}}_{\begin{array}{c}(x_0^w,x_0^l,c)\sim \mathcal{D},t\sim \mathcal{U}(0,T),\\ x_t^w\sim p(x_t\mid x_0^w,c),x_t^l\sim p(x_t\mid x_0^l,c)\end{array}}\left[log\sigma \left(r(x_t^w,c)-r(x_t^l,c)\right)\right],\hfill \end{array}$$

(11)

where $\sigma (·)$ is the sigmoid function, c the text prompt, and $(x_0^w,x_0^l,c)\sim \mathcal{D}$. Here, $t\sim \mathcal{U}(0,T)$, and $x_t^w$, $x_t^l$ are the intermediate noisy images at step t for the “winning” and “losing” samples, respectively, drawn via $(x_t^w,x_{t+1}^w)\sim p(x_{t+1}\mid x_t)$ and $(x_t^l,x_{t+1}^l)\sim p(x_{t+1}\mid x_t).$

5. Experiments

5.1. Model and Dataset

In this section, we demonstrate the effectiveness of xDPO for personalized preference alignment through a series of experiments. Following the Diffusion-DPO [7] approach, we fine-tune the open-source Stable Diffusion 1.5 (SD1.5) base model [12] on the human-preference image pairs in the Pick-a-Picv2 dataset [20], using the xDPO objective function Equation (11). The Pick-a-Picv2 dataset comprises 0.8 million preference triplets of the form (preferred image, non-preferred image, input prompt).

5.2. Hyperparameters

For all SD1.5 experiments, we employ the AdamW optimizer [36]. Training is performed on two NVIDIA RTX 4090 GPUs (NVIDIA, Santa Clara, CA, USA), with a per-GPU micro-batch size of one preference pair and gradient accumulation over 128 steps. The learning rate is set to $1\times {10}^{-8}$. Empirically, we observed stable and strong performance for $\beta$ values in the range [1000, 2000], and therefore report results using the configuration $\beta =1000$.

5.3. Evaluation

We evaluated the performance of the xDPO model in aligning with human preferences using both automated metrics and a user study. In the automated image quality evaluation, we employed four metrics: the first three were PickScore [20], HPSV2 [9], and ImageReward [19], each trained on human-preference datasets to predict which image a user is more likely to prefer under the same prompt. These three metrics not only assess preference prediction but also incorporate aesthetic judgment to some extent. In addition, we used CLIP [37] to measure prompt responsiveness by computing the image–text similarity score.

We generated images with xDPO and four baseline models—pretrained SD1.5, Diffusion-DPO [7], SePPO [38], and Diffusion-KTO [30]—on two benchmarks: PartiPrompt [10] (1632 prompts) and HPSv2 [9] (3200 prompts). We then applied the aforementioned automated metrics to all outputs and reported both average scores and win-rate statistics. For a fair comparison, Diffusion-DPO  [7], Diffusion-KTO [30], and SePPO [38] were evaluated using their officially released checkpoints, and all models (including xDPO) employed the default hyperparameters from [7]: a guidance scale of 7.5 and 50 denoising steps. For the user study, we presented human evaluators with paired images—one from xDPO and one from Diffusion-DPO—generated under the same prompt, and asked two questions: 1. General Preference (Q1): Given the same prompt, which image do you personally prefer? 2. Prompt Alignment (Q2): Which image better matches the textual description? Each comparison was rated by five annotators, and the majority vote (≥3 out of 5) determined the group decision. We aggregated their choices into win-rate percentages over two sets of prompts: 100 randomly sampled from PartiPrompts [10] and 200 randomly sampled from the HPSv2 benchmark [9].

5.4. Quantitative Results

Table 2. The results of the model-feedback evaluation on the HPSv2 and Parti-Prompt datasets are presented below, with the highest scores highlighted in bold.

Dataset	Method	HPSV2 ↑	PickScore ↑	Image Reward ↑	CLIP ↑	Aesthetic ↑
HPSV2	SD v1-5  [12]	26.97	20.690	0.125	0.349	5.46
	Diffusion-DPO  [7]	27.28	21.124	0.315	0.354	5.56
	Diffusion-KTO [30]	$\mathbf{27.99}$	21.332	0.696	0.352	5.69
	SePPO [38]	27.88	21.496	0.616	0.354	$\mathbf{5.76}$
	xDPO (ours)	27.98	$\mathbf{21.694}$	$\mathbf{0.737}$	$\mathbf{0.354}$	5.66
PartiPrompts	SD v1-5  [12]	26.956	21.240	0.244	0.336	5.260
	Diffusion-DPO  [7]	27.193	21.489	0.396	0.341	5.339
	Diffusion-KTO  [30]	$\mathbf{27.740}$	21.544	0.633	0.339	5.473
	SePPO  [38]	27.611	21.667	0.572	0.338	$\mathbf{5.504}$
	xDPO (ours)	$27.677$	$\mathbf{21.823}$	$\mathbf{0.725}$	$\mathbf{0.344}$	5.439

reports the absolute reward scores of each reward model across all datasets. Compared to SD1.5 and Diffusion-DPO, xDPO achieves substantial improvements on every reward metric. The bottom of Figure 3 presents a side-by-side comparison of xDPO, Diffusion-DPO, Diffusion-KTO, and the original SD1.5 on the HPSv2 dataset. On HPSv2, xDPO achieves the highest PickScore, ImageReward, and CLIP scores. It is also worth noting that xDPO delivers a significant improvement in training efficiency, training 1.6 times faster than Diffusion-DPO (as shown in Figure 4), while generating higher-quality images. Furthermore, in human evaluations (see the top of Figure 3), annotators demonstrated a clear preference for our xDPO model, which received higher support rates than both baseline models, DPO-SD1.5 and Diffusion-DPO. Specifically, for the “Overall Preference” (Q1) task on the HPSv2 dataset, xDPO outperformed DPO-SD1.5 with a 76.4% win rate and surpassed Diffusion-DPO with a 73.4% win rate. These results clearly demonstrate that our approach provides a more efficient solution for fine-tuning diffusion models based on human preferences.

5.5. Qualitative Results

Figure 1 illustrates images generated by xDPO, which demonstrate outstanding appeal, vivid color palettes, dramatic lighting, strong compositional balance, and lifelike human and animal anatomical structure. Figure 5 presents a direct visual comparison of xDPO with Diffusion-DPO and other baseline methods. In the first row, although all models faithfully reproduce the “pixel art” aesthetic, Diffusion-DPO fails to capture the critical “on the moon” detail; by contrast, xDPO accurately renders this element, achieving superior fine-grained fidelity and overall image quality. In the second row, our xDPO aligned model not only depicts the vibrant hues and mist-enshrouded rock formations with remarkable precision but also outperforms Diffusion-DPO in rendering the glass sphere—faithfully reproducing perspective distortion, specular highlights, reflections, and the subtle stretching and compression at the periphery. In the bottom row, when conditioned on prompts containing detailed descriptions of hand anatomy, xDPO generates anatomically more accurate hand structures than those produced by the Diffusion-DPO method.

5.6. Ablations

As the alignment process progresses, the model increases the conditional probability of the preferred sample, $p_{\theta }(x_t^w\mid x_{t+1}^w,c)$, while decreasing that of the non-preferred sample, $p_{\theta }(x_t^l\mid x_{t+1}^l,c)$. Consequently, the ratio for the preferred sample $z_w={\displaystyle \frac{p_{\theta }(x_t^w\mid x_{t+1}^w,c)}{p_{\mathrm{ref}}(x_t^w\mid x_{t+1}^w,c)}}$ tends to become greater than 1, whereas the ratio for the non-preferred sample $z_l={\displaystyle \frac{p_{\theta }(x_t^l\mid x_{t+1}^l,c)}{p_{\mathrm{ref}}(x_t^l\mid x_{t+1}^l,c)}}$ tends to fall below 1. Based on this observation, we introduce the $x^n$ divergence penalty to further amplify the function values in the region where $z>1$, thereby strengthening the gradient signals associated with preferred samples during training. As illustrated in Figure 2 (right), this design enhances the model’s responsiveness to human preferences. In our preliminary exploration, we directly introduced the $x^n$ divergence penalty and found that it significantly constrained the model’s flexibility in adjustment, slowed the convergence of the training loss, and had a limited effect on improving the reward score. To mitigate this issue, we introduced a coefficient $\gamma$ and applied $1/n$ weighting scheme to reduce the impact of the $x^n$ term. The experimental results confirm the effectiveness of this strategy. To investigate the impact of different values of n on model performance, we conducted comparative experiments with $n=1,3,6$. Rows 3 to 5 of

Table 3. Ablation and comparison results on SD1.5 using the HPDv2 test set, with the highest scores highlighted in bold.

	Median (HPDv2)
	HPSV2	PickScore	Image Reward	CLIP
Base-SD1.5	26.97	20.690	0.125	0.349
DPO-SD1.5	27.28	21.124	0.315	0.354
$\eta =1$, $n=1$, $\gamma =1/1$	$27.83$	$21.53$	$0.643$	$0.357$
$\eta =1$, $n=3$, $\gamma =1/3$	$27.87$	21.62	0.644	0.355
$\eta =1$, $n=6$, $\gamma =1/6$	27.88	21.57	0.653	0.355
$\eta =0.5$, $n=6$, $\gamma =1/6$	27.98	21.70	0.702	0.355
$\eta =0.1$, $n=6$, $\gamma =1/6$	27.97	$\mathbf{21.709}$	0.731	$\mathbf{0.357}$
$\eta =0.05$, $n=6$, $\gamma =1/6$	27.98	21.694	$\mathbf{0.737}$	$0.354$

show the performance of xDPO under different n values. The results indicate that the model performs best when $n=6$. In addition, we explored balancing the optimization strength between positive and negative samples by reducing the value of $\eta$. Rows 6 to 8 of Table 3 present the results of experiments where only $\eta$ was adjusted. The findings show that setting $\eta$ to $0.05$ or $0.1$ enables the model to achieve a more balanced generation quality across various reward functions.

6. Conclusions and Limitations

6.1. Conclusions

This paper proposes xDPO, an efficient diffusion model alignment method based on human preferences. xDPO directly fine-tunes diffusion models by implicitly designing reward functions. Experiments on fine-tuning T2I diffusion models with the Pick-a-Pic v2 dataset demonstrate that xDPO significantly improves training efficiency compared to Diffusion-DPO. Moreover, xDPO achieves robust alignment across various reward models, fully showcasing its potential to enhance the performance of text-to-image diffusion models.

6.2. Limitations

Due to its reliance on offline fine-tuning, the performance of xDPO is highly sensitive to the quality of the training dataset. If the dataset contains harmful, violent, or pornographic content, the model may generate images with inappropriate elements. Therefore, we believe that adopting an online training paradigm holds greater promise and stability for future developments.