Utility-Based Preference Training for Effective Synthetic Text Classification

Abstract

High-quality synthetic text can mitigate annotation scarcity in text classification. However, standard preference optimization often produces samples that are fluent but weakly label-specific. We present Utility-weighted Direct Preference Optimization (U-DPO), a preference-optimization framework for class-conditional synthetic data generation. In U-DPO, a task-specific classifier provides a margin-based external score for each candidate generation, which is combined with an embedding-based internal similarity score to form an overall utility. These utilities are used (i) to mine preference pairs from multiple candidates per class and (ii) to weigh each DPO update by the utility gap between preferred and dispreferred samples. This design encourages the generator to concentrate on learning informative, label-discriminative preference comparisons rather than treating all pairs equally. Across two multiclass scientific-abstract benchmarks (arXiv and WOS-11967), U-DPO consistently improves downstream SciBERT classification accuracy compared with both vanilla synthetic generation and standard DPO fine-tuning, with gains up to 0.88 percentage points on arXiv and 0.83 percentage points on WOS-11967 depending on the generator. An additional GPT-4.5-based evaluation also indicates a higher mean quality score for U-DPO samples with reduced variance.

1. Introduction

Text classification is a core task in natural language processing (NLP), underpinning applications from sentiment analysis to topic categorization. Recent Large Language Models (LLMs) have made it increasingly feasible to generate labeled synthetic text to supplement limited human annotations [1,2]. When effective, synthetic augmentation can reduce labeling costs and expand data coverage, which is particularly valuable in low-resource settings [3,4].

A central practical challenge; however, is label specificity [5]. Prompt-based generations can be fluent and coherent yet only weakly discriminative for the intended class. Such label-ambiguous samples may provide a limited learning signal and can even degrade downstream performance due to noise and distribution mismatch [6,7,8].

To improve the usefulness of synthetic data, prior work has explored stronger prompting strategies and filtering mechanisms. In parallel, preference optimization has emerged as a powerful paradigm for aligning LLM outputs, including Reinforcement Learning from Human Feedback (RLHF) [9] and Direct Preference Optimization (DPO) [10]. DPO is particularly attractive because it offers a stable, reinforcement-learning-free objective and has been extended to better handle preference strength and robustness [11,12,13].

Despite these advances, an important gap remains for classification-oriented synthetic generation. Standard preference optimization typically treats all labeled preference pairs (preferred vs. dispreferred) as similarly informative. In class-conditional generation; however, preference pairs can differ substantially in downstream utility, which comprises both label discriminativeness and intrinsic text quality. A “dispreferred” sample is not necessarily useless; it may still contain strong class-discriminative cues and exhibit high linguistic fluency. In such cases, blindly forcing the model away from a valid dispreferred sample can be counterproductive. This suggests that preference learning for synthetic data should not apply uniform updates. Instead, it should account for the size of the utility gap (reflecting differences in semantic quality and classification confidence) to avoid penalizing high-quality candidates.

To address this gap, we propose Utility-weighted Direct Preference Optimization (U-DPO). U-DPO leverages utility in two complementary ways. First, it performs utility-guided pair construction to prioritize informative preference pairs. Second, it applies utility-gap weighting to reduce updates on ambiguous pairs, especially when the dispreferred candidate may still be beneficial for classification.

We focus on scientific abstract classification because it is a realistic setting where labeled data are costly and domain-specific terminology increases label ambiguity. Moreover, classes often exhibit distinct writing conventions (e.g., characteristic terminology and structure), so class-conditional generation must capture both topical content and label-specific cues. This makes scientific abstracts a strong stress test for utility-based preference training, where label-discriminativeness is critical. While our experiments focus on scientific corpora, the proposed utility-weighted preference mechanism is model-agnostic and can extend to other classification domains.

Accordingly, we evaluate U-DPO on two multiclass scientific-abstract classification benchmarks (arXiv and WOS-11967), comparing against prompt-based synthetic generation and standard DPO tuning. Across multiple open-source text generators, U-DPO consistently improves downstream SciBERT classification accuracy and produces synthetic samples with stronger label consistency. This demonstrates that incorporating task utility into preference optimization yields more effective synthetic training data.

Our main contributions are as follows:

• Task-utility preference construction: We introduce a utility-based mechanism to mine class-conditional preference pairs from multiple candidate generations using classifier-margin confidence and embedding-based semantic quality signals.
• Utility-gap–weighted preference optimization: We propose U-DPO, which scales DPO updates by the utility gap between preferred and dispreferred samples to focus learning on more informative comparisons and avoid counterproductive updates.
• Empirical validation on scientific multiclass benchmarks: We demonstrate consistent downstream improvements on arXiv and WOS-11967 across multiple generators, supported by analyses showing enhanced label consistency of generated samples.

2. Related Work

2.1. LLMs for Text Classification

Recent studies have explored the capabilities of LLMs in text classification through zero-shot and few-shot prompting, as well as fine-tuning [14,15]. These works show that LLMs can often perform surprisingly well without task-specific training data, but their effectiveness varies by task and setting [16]. LLMs have demonstrated competitive performance in specialized tasks such as scientific edit intent classification [17]. However, large-scale evaluations find that zero-shot prompting is most effective on simpler tasks like sentiment analysis. Fine-tuned models, even smaller ones, often remain stronger on more complex classification problems [18]. A recent multilingual study found that smaller fine-tuned transformers can even surpass few-shot LLMs in accuracy across most categories, suggesting that in-context learning alone is often insufficient for optimal performance [19].

2.2. Prompt-Based Synthetic Data Generation

Prompt-based synthetic data generation has emerged as a promising strategy for training text classifiers [3]. Instead of manually collecting or annotating data, researchers prompt LLMs to produce labeled examples, which can be used to augment or even replace human-labeled training sets [5]. Such approaches have shown growing effectiveness for domain adaptation and general-purpose classification [20]. For instance, recent studies show that LLMs can generate domain-general sentiment datasets [21], fully synthetic training corpora without human labels [22], and even code-mixed data for multilingual sentiment classification [23]. Despite these successes, important limitations have also been observed. In one health-related classification task, augmenting an unbalanced dataset with GPT-generated samples did not produce performance improvements [6]. This finding suggests that the utility of synthetic data depends on the target task. It also indicates that generation strategies must be carefully tailored to avoid issues such as bias inheritance or model collapse [24,25]. These findings suggest that synthetic data can be fluent yet not classifier-useful. This motivates utility-weighted preference learning that prioritizes label-discriminative samples and down-weights ambiguous generations.

2.3. Preference Optimization for Text Generation

Aligning LLMs with human values is critical for their safe and effective deployment. The standard approach, Reinforcement Learning from Human Feedback (RLHF) [9], optimizes a model to produce outputs preferred by humans and has been widely used to train aligned language models [26,27]. However, RLHF can be complex and unstable, especially in classification settings where defining reliable reward functions is challenging [28]. To address these issues, Direct Preference Optimization (DPO) [10] was introduced as a more stable, RL-free alternative that learns directly from preference data. Building on this idea, subsequent work has adapted DPO to more complex alignment tasks. For instance, variants like IPO address overfitting [11], while KTO learns from binary good/bad labels instead of pairs [12]. Other approaches introduce adaptive reward margins or offsets to better reflect preference strength during optimization, such as AlphaDPO [13] and ODPO [29]. These advances show that preference optimization can be adapted to diverse task requirements through objective-level modifications, providing a practical and scalable framework for alignment. Our work complements this line by making the training emphasis task-specific for synthetic data generation. Unlike approaches that modify the optimization objective with static margins, we propose a dynamic reweighting mechanism based on a composite sample utility. Concretely, we define a task-specific utility by combining two factors, label discriminativeness and intrinsic text quality. We then use the utility gap between the preferred and dispreferred samples to scale the gradient of each update. This strategy amplifies learning from high-contrast pairs that significantly improve both class separability and text well-formedness, while effectively dampening the signal from ambiguous near-ties. By prioritizing informative comparisons over noisy ones, U-DPO tailors preference optimization specifically for downstream classification performance.

3. Theoretical Analysis

We briefly review DPO and its theoretical basis before introducing our utility-based extension.

3.1. RLHF Objective and Optimal Policy

The standard RLHF objective seeks a policy ${\pi }_{\theta }$ that maximizes the expected reward from a learned reward model $r_{\varphi }(x,y)$ while remaining close to a reference policy ${\pi }_{\mathrm{ref}}$, typically a supervised fine-tuned model. This is formulated as:

$$\underset{\theta }{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.25em} {\mathbb{E}}_{x\sim \mathcal{D},\hspace{0.17em}y\sim {\pi }_{\theta }(y\mid x)}[r_{\varphi }(x,y)]\hspace{0.25em} -\hspace{0.25em} \beta \hspace{0.17em}{D}_{KL}({\pi }_{\theta }(y\mid x)\hspace{0.17em}\parallel \hspace{0.17em}{\pi }_{\mathrm{ref}}(y\mid x)),$$

(1)

where $\beta >0$ controls the strength of the KL penalty. Under mild assumptions, the optimal policy ${\pi }^{*}$ has the closed form

$${\pi }^{*}(y\mid x)={\displaystyle \frac{1}{Z(x)}}\hspace{0.17em}{\pi }_{\mathrm{ref}}(y\mid x)\hspace{0.17em}\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{1}{\beta }}{r}^{*}(x,y)\right),$$

(2)

where $Z(x)$ is a normalizing partition function. This relation connects the optimal policy ${\pi }^{*}$ to an implicit reward function $r^{*}$.

3.2. The Bradley-Terry Model and the DPO Objective

DPO leverages this mapping by modeling human preferences via the Bradley–Terry model [30]. Given a pair of candidate responses $\left(y_w,y_l\right)$ for the same input $x$, the probability that $y_w$ is preferred over $y_l$ is

$$p^{*}\left(y_w\succ y_l∣x\right)=\sigma \left(r^{*}\left(x,y_w\right)-r^{*}\left(x,y_l\right)\right),$$

(3)

where $\sigma (\cdot )$ is the logistic sigmoid. Substituting the policy-based representation of $r^{*}$ from (2), ref. [10] express this probability in terms of policies:

$$p^{*}(y_w\succ y_l\mid x)=\sigma (\beta \mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{{\pi }^{*}(y_w\mid x)}{{\pi }_{\mathrm{ref}}(y_w\mid x)}}-\beta \mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{{\pi }^{*}(y_l\mid x)}{{\pi }_{\mathrm{ref}}(y_l\mid x)}}).$$

(4)

Using an implicit reward ${\widehat{r}}_{\theta }(x,y)=\beta \mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\mathrm{ref}}(y\mid x)}}$, the DPO loss is defined as the negative log-likelihood over a dataset of preference triplets ${\mathcal{D}}_{\mathrm{pref}}=\{(x,y_w,y_l)\}$:

$${\mathcal{L}}_{\mathrm{DPO}}({\pi }_{\theta };{\pi }_{\mathrm{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim {\mathcal{D}}_{\mathrm{pref}}}\left[\mathrm{l}\mathrm{o}\mathrm{g} \sigma ({\widehat{r}}_{\theta }(x,y_w)-{\widehat{r}}_{\theta }(x,y_l))\right].$$

(5)

This objective directly optimizes ${\pi }_{\theta }$ to respect observed preferences, without training an explicit reward model or performing RL.

3.3. Utility-Weighted DPO

In our setting, preference pairs are induced by a proxy utility signal, and their reliability can vary substantially. Pairs with small utility gaps behave like near-ties and are more sensitive to noise or miscalibration in the utility scorer. We therefore model U-DPO as a weighted variant of DPO that assigns each pair a nonnegative weight w based on its utility gap. Here, $w$ represents the confidence in the induced preference. Larger utility gaps indicate more reliable preferences, whereas near ties are likely to produce unstable preference labels and weak learning signals.

Concretely, we optimize a weighted Bradley–Terry negative log-likelihood:

$${\mathcal{L}}_{\mathrm{U}\text{-}\mathrm{DPO}}({\pi }_{\theta };{\pi }_{\mathrm{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim {\mathcal{D}}_{\mathrm{pref}}}\left[w(x,y_w,y_l) \ast \mathrm{l}\mathrm{o}\mathrm{g} \sigma ({\widehat{r}}_{\theta }(x,y_w)-{\widehat{r}}_{\theta }(x,y_l))\right].$$

(6)

When $w(x,y_w,y_l)=1$ for all pairs, this reduces to standard DPO. Intuitively, weighting emphasizes more informative comparisons while down-weighting ambiguous pairs, which is desirable when preferences are automatically constructed from task utility.

4. Utility-Weighted DPO for Class-Conditional Synthetic Data

Standard DPO is agnostic to the downstream task: it aligns with whatever preferences are provided (e.g., helpful vs. unhelpful responses). For synthetic data generation in classification, we seek preferences that reflect how useful a sample is for training a classifier. Moreover, preference pairs are not equally informative in this setting; thus, applying uniform updates across pairs can be suboptimal. Unlike prior methods, we leverage a task-specific utility to adaptively adjust the update strength for each sample. This approach emphasizes comparisons that improve class separability while down-weighting ambiguous pairs. Crucially, the utility is composite: it dynamically reflects the relative difference between candidate samples while also accounting for intrinsic text quality, enabling U-DPO to favor both label-aligned and well-formed generations. An overview of the entire framework is illustrated in Figure 1.

4.1. Problem Setup

Let $\mathcal{C}=\{c_1,\dots ,c_K\}$ denote the set of class labels. Our goal is to train a conditional language model ${\pi }_{\theta }(x\mid c)$ that generates synthetic texts $x$ such that, for each label $c$, the resulting dataset is effective for training a downstream classifier.

We adopt a preference-learning view. For each label $c$, we consider pairs of candidate generations $\left(x^{+},x^{-}\right)$ intended to reflect relative quality with respect to the class semantics, where $x^{+}$ is the preferred (higher-utility) sample and $x^{-}$ the less preferred one.

4.2. Utility-Based Preference Signal

Standard DPO typically relies on preference pairs, often human-annotated, that reflect generic notions of quality.

In contrast, our Utility-weighted DPO (U-DPO) redefines the preference signal in terms of a task-specific utility function $r_{\mathrm{util}}$ designed to approximate the value of a sample for classification. Instead of directly modeling a latent reward $r^{*}$, we construct preference pairs automatically based on utility differences.

We view this as a form of reward shaping within the DPO framework: $r_{\mathrm{util}}$ is a proxy for the ideal reward, namely, downstream classification performance. The utility function combines external label confidence and internal semantic quality, described below.

However, employing a classifier-defined utility warrants careful consideration regarding feedback loops. Using such a utility may introduce bias propagation if the auxiliary classifier is biased or miscalibrated. This could potentially amplify spurious cues in the generated samples.

A related risk is circular evaluation, which occurs when the utility scorer and downstream evaluator are identical or closely aligned. This overlap can cause the model to overfit to the evaluator’s peculiarities, thereby exaggerating the apparent gains. To mitigate direct circularity, we distinguish the models used in our pipeline. Specifically, we use RoBERTa for utility scoring while reserving SciBERT solely for downstream evaluation.

4.3. Data Construction and Utility Function

Since collecting human-labeled preference pairs is expensive, we construct them automatically using an auxiliary classifier $f_{\varphi }$ trained on available real data.

For each class $y\in \mathcal{C}$,

• Sample $n=8$ candidate texts ${\tilde{x}}_1,\dots ,{\tilde{x}}_n$ from ${\pi }_{\theta }(\cdot \mid y)$. Candidate texts are generated with a fixed 2-shot academic-abstract prompt to ensure a consistent scientific style across classes. The exact prompt template is provided in Appendix A.
• Compute a margin score for each candidate using $f_{\varphi }$:

  $$m(\tilde{x})=f_{\varphi }(\tilde{x}{)}_y-\underset{j\ne y}{\mathrm{m}\mathrm{a}\mathrm{x}} f_{\varphi }(\tilde{x}{)}_j,$$

  (7)

  where $f_{\varphi }(\tilde{x}{)}_j$ is the predicted probability for class $j$. Candidates with higher $m(\tilde{x})$ are more confidently classified as a label $y$.
• Select high-margin candidates as preferred $x^{+}$ and medium-margin candidates as less preferred $x^{-}$, forming automatic preference pairs. To mitigate bias, the utility classifier is trained only on the real training split and is kept fixed. It is never trained on synthetic samples, and it does not access the downstream test set.
• Margin scores alone; however, they may be noisy or biased and do not capture aspects such as fluency or coherence. We therefore introduce an internal utility term $r_{int}(x)$ that measures the well-formedness of a candidate text independent of label confidence. Concretely, we implement $r_{int}(x)$ as a MiniLM-based text-quality scorer: a pretrained MiniLM (microsoft/MiniLM-L12-H384-uncased) encoder ${\rm E}(·)$ is kept frozen, and a lightweight linear head g(·) is trained to predict a continuous quality score in [0, 1]. Given a candidate text x, we obtain its representation $h(x)={\rm E}(x)$ (final-layer [CLS] embedding) and compute

  $$r_{int}(x)=\sigma (g(h(x))),$$

  (8)

  where σ(⋅) denotes the sigmoid function. To train the quality head, we use the Ultrafeedback dataset, which provides supervision signals aimed at assessing intrinsic text quality independent of the downstream classification task. In other words, the head is optimized to score how good the text is as text, rather than how confidently it supports a particular label.

We define the external utility $r_{\mathrm{ext}}(\tilde{x},y)$ using the auxiliary classifier margin:

$$r_{ext}(\tilde{x},y)=m(\tilde{x}),$$

(9)

Since the raw margin scale can vary across datasets and classifier calibrations, we apply min-max normalization over the candidate pool used for pair construction:

$$\widehat{r_{\mathrm{ext}}}\left(\tilde{x},y\right)=\widehat{m}\left(\tilde{x}\right)= {\displaystyle \frac{m\left(\tilde{x}\right)- m_{min}}{m_{max }-m_{min }}}$$

(10)

Finally, we combine the internal text-quality score and the external label-confidence score into a single utility used for preference learning:

$$r_{\mathrm{util}}(\tilde{x},y)=\lambda \hspace{0.17em}{r}_{\mathrm{int}}(\tilde{x})+(1-\lambda )\hspace{0.17em}\widehat{r_{\mathrm{ext}}}(\tilde{x},y),$$

(11)

where 0 ≤ λ ≤ 1 balances semantic quality and label confidence. To focus optimization on informative pairs, we weight each pair by a modulation factor depending on the absolute utility gap. Additional details and minimal ablation studies on $\lambda$ and the contribution of $r_{\mathrm{int}}$ and $r_{\mathrm{ext}}$ are deferred to Appendix A.

$$w(x^{+},x^{-})={ r}_{\mathrm{util}}(x^{+},y)-r_{\mathrm{util}}(x^{-},y).$$

(12)

The above equation down-weights training signals for pairs whose utility values are similar, as such pairs are likely to provide ambiguous supervision.

4.4. U-DPO Objective

Given preference pairs $\left(x,x^{+},x^{-}\right)$ constructed as above, we define the U-DPO loss by re-weighting the standard DPO loss with the modulation factor:

$${\mathcal{L}}_{\mathrm{U}\text{-}\mathrm{DPO}}(\theta )={\mathbb{E}}_{\left(x,x^{+},x^{-}\right)}[w(x^{+},x^{-})\hspace{0.25em} {\mathcal{l}}_{\mathrm{DPO}}(x,x^{+},x^{-};\theta )],$$

(13)

where ${\mathcal{l}}_{\mathrm{DPO}}$ denotes the per-triplet contribution to the DPO loss in (5). In practice, we implement U-DPO by multiplying the standard DPO loss for each pair by $w(x^{+},x^{-})$, so that pairs with stronger utility differences exert stronger gradients.

Intuitively, U-DPO encourages the generator ${\pi }_{\theta }$ to produce outputs that are simultaneously (i) semantically plausible and (ii) highly discriminative under the classifier, thereby optimizing directly for the downstream classification task.

For clarity,

Table 1. Summary of utility components in U-DPO.

Term	Definition	Range	Purpose
${\mathrm{r}}_{\mathrm{i}\mathrm{n}\mathrm{t}}$	$\mathsf{\sigma }(\mathrm{g}(\mathrm{h}(\mathrm{x})))$	[0, 1]	Intrinsic text quality
$r_{ext}$	$m(\tilde{x})=f_{\varphi }(\tilde{x}{)}_y-\underset{j\ne y}{\mathrm{m}\mathrm{a}\mathrm{x}} f_{\varphi }(\tilde{x}{)}_j$	$\mathbb{R}$	Label confidence from auxiliary classifier
$\widehat{r_{\mathrm{ext}}}$	$\widehat{m}\left(\tilde{x}\right)={\displaystyle \frac{m\left(\tilde{x}\right)-m_{min}}{m_{max }-m_{min}}}$	[0, 1]	Min–max normalized margin to match scales with ${\mathrm{r}}_{\mathrm{i}\mathrm{n}\mathrm{t}}$
$r_{\mathrm{util}}$	$\lambda \hspace{0.17em}{r}_{\mathrm{int}}(\tilde{x})+(1-\lambda )\hspace{0.17em}\widehat{r_{\mathrm{ext}}}(\tilde{x},y)$	[0, 1]	Overall utility combining quality + label evidence
$\lambda$	Mixing weight in $r_{util}$	[0, 1]	Trade-off: λ↑ quality emphasis, λ↓ label confidence emphasis
$w$	${ r}_{\mathrm{util}}(x^{+},y)-r_{\mathrm{util}}(x^{-},y)$	[0, 1]	Pair weight: down-weights ambiguous pairs and emphasizes informative utility gaps.

Trade-off: λ↑ indicates quality emphasis, whereas λ↓ indicates label-confidence emphasis.

summarizes the key utility components used in U-DPO, along with the definition of the pair weight $w$.

5. Computational Complexity

We analyze the computational overhead of U-DPO relative to standard DPO and simple supervised fine-tuning.

5.1. Complexity of DPO Fine-Tuning

In standard DPO, each training step processes a batch of preference triplets $\left(\mathit{x},{\mathit{y}}_{\mathit{w}},{\mathit{y}}_{\mathit{l}}\right)$. The main cost arises from computing log-probabilities under both the policy model ${\mathit{\pi }}_{\mathit{\theta }}$ and the frozen reference model ${\mathit{\pi }}_{\mathrm{ref}}$, requiring two forward passes per candidate. For a Transformer-based model with sequence length $\mathit{N}$ and hidden dimension $\mathit{d}$, a single forward pass has time complexity $\mathcal{O}({\mathit{N}}^2\mathit{d})$ [31]. For batch size $\mathit{B}$, the per-step complexity is roughly:

$$\mathcal{O}(\mathit{B}\cdot 2\cdot ({\mathit{T}}_{{\mathit{\pi }}_{\mathit{\theta }}}+{\mathit{T}}_{{\mathit{\pi }}_{\mathrm{ref}}}))\approx \mathcal{O}(\mathit{B}\hspace{0.17em}{\mathit{N}}^2\mathit{d}),$$

where ${\mathit{T}}_{{\mathit{\pi }}_{\mathit{\theta }}}$ and ${\mathit{T}}_{{\mathit{\pi }}_{\mathrm{ref}}}$ denote single forward-pass costs, which are typically comparable as the architectures are matched. Memory usage is also substantial, since activations for both models must be stored [32].

5.2. Additional Complexity of U-DPO

U-DPO introduces an offline data-construction phase to build utility-filtered preference pairs. For each class $\mathit{c}\in \mathcal{C}$, this phase comprises:

• Candidate Generation. For each prompt, we generate $\mathit{n}$ candidate sequences from the generator ${\mathit{\pi }}_{\mathit{\theta }}$, incurring complexity $\mathcal{O}(\mathit{n}\hspace{0.17em}{\mathit{N}}_{\mathrm{gen}}^2\mathit{d})$, where ${\mathit{N}}_{\mathrm{gen}}$ is the length of the generated sequences.
• Margin Scoring. Each candidate is fed to the classifier ${\mathit{f}}_{\mathit{\varphi }}$ (e.g., SciBERT) of dimension ${\mathit{d}}_{\mathrm{cls}}$, with complexity $\mathcal{O}(\mathit{n}\hspace{0.17em}{\mathit{N}}_{\mathrm{cls}}^2{\mathit{d}}_{\mathrm{cls}})$.
• Internal Utility Scoring. Each candidate is encoded by an embedding model (e.g., MiniLM) to compute ${\mathit{r}}_{\mathrm{int}}$, with complexity $\mathcal{O}(\mathit{n}\hspace{0.17em}{\mathit{N}}_{\mathrm{emb}}^2{\mathit{d}}_{\mathrm{emb}})$.

Thus, the total per-prompt cost is dominated by candidate generation and the two utility-scoring passes. Since preference-pair construction is performed once before U-DPO fine-tuning, this overhead is front-loaded and can be amortized across training runs. Moreover, in our implementation, we pre-compute offline to avoid keeping multiple scoring models on the GPU during fine-tuning; as a result, U-DPO training incurs only a negligible additional cost from simple scalar weighting.

Table 2. Runtime breakdown and additional overhead introduced by U-DPO. Units are minutes for training phases and milliseconds for inference and scoring steps.

Component	Time	Notes
External Scorer training	22 min	One-time
Internal Scorer training	25 min	One-time
Candidate generation (8 samples)	16,640 ms	Baseline
External Scoring	206 ms	Overhead
Internal Scoring	50 ms	Overhead
Total Overhead	257 ms	$\approx$1.54%

reports the resulting wall-clock overhead measured in our setup.

6. Experiments

6.1. Experimental Setup

We evaluate our approach on two multiclass scientific classification datasets, arXiv [33], with 11 classes, and WOS-11967 [34], with 33 classes. Both datasets consist of scholarly abstracts.

The overall experimental configuration is summarized in

Table 3. Summary of experimental configuration.

Dataset	arXiv	WOS-11967
Category	11	33
Train set	28,388	9473
Test set	2500	2394
Component	Details
Classification Model	SciBERT
Utility Function Model	MiniLM
	RoBERTa
LLMs for Generation	LLaMA 3.2 1B,
	LLaMa 3.2 3B,
	Phi-4-mini 3.8B
DPO Settings	β = 1.0
	learning rate = 2 × 10−5
	batch size = 2
	epochs = 3

. We use SciBERT (uncased) [35] as the classifier backbone and MiniLM-L12-H384-uncased [36] as the embedding model for computing semantic utility scores. Prior studies have shown that SciBERT consistently outperforms BERT [37] and RoBERTa [38] on scientific NLP benchmarks, highlighting its domain relevance.

Synthetic training data is generated using three open-source LLMs—LLaMA 3.2 1B, 3B [39], and Phi-4-mini [40]—with class-conditional prompts. DPO training is performed using HuggingFace TRL with custom modifications to incorporate our utility-aware pair selection. All experiments are conducted on a single NVIDIA A6000 GPU. Synthetic data are generated using a 2-shot prompting strategy, where two randomly selected examples with the same label are used as input to the LLM. For each prompt, we generate (n = 5) samples. For evaluation, we report accuracy.

6.2. Training-Time Preference Consistency

To evaluate how well the generator aligns with the preference signal during training, we compute the DPO reward accuracy. This metric is defined as the fraction of preference pairs in which the model assigns a higher log-probability to the preferred sample ${\mathit{x}}^{+}$. Figure 2 plots reward accuracy over training steps for standard DPO and U-DPO. U-DPO consistently achieves higher reward accuracy, indicating that utility-filtered pairs provide a cleaner, more informative training signal. In contrast, standard DPO exhibits more fluctuations, likely due to noise from unfiltered pair selection.

6.3. Margin-Based Quality Assessment

To verify whether U-DPO enhances the class consistency of generated text, we evaluate synthetic samples using the margin score defined in Section 4.3. We analyze this score using samples generated by the Phi-4-mini model on the ArXiv dataset. As shown in Figure 3, both standard DPO and U-DPO result in higher average and median margin scores compared to generation without preference optimization. Notably, U-DPO yields further gains, suggesting that incorporating utility signals strengthens the model’s ability to generate label-consistent outputs.

6.4. Classification Performance with Synthetic Data

To directly assess the classification performance, we train models exclusively on generated samples from the arXiv dataset. We compare three regimes: (1) generation from the base LLM, (2) generation using standard DPO, and (3) generation via our U-DPO framework. As shown in Figure 4, classifiers trained using U-DPO samples consistently outperform those trained on data from the base LLM and standard DPO. This demonstrates that utility-based training leads to significant improvements in downstream task performance.

6.5. Evaluating Synthetic–Real Data Augmentation

To evaluate practical utility, we measure performance when combining synthetic samples with a fixed subset of real data. We vary the number of generated samples per class and found that 50 samples per class achieved the highest performance on average.

As shown in

Table 4. Classification accuracy of the fine-tuned SciBERT baseline, SciBERT prompting, and GPT-4o prompting on the arXiv and WOS-11967 datasets. Results of open-source LLMs—LLaMA 3.2 1B (denoted LLaMA-1B), LLaMA 3.2 3B (LLaMA-3B), and Phi-4-mini (Phi-4)—are also included for comparison. All values are means over 20 runs. p-values (two-sided paired t-tests over 20 matched runs) are reported below the table.

SciBERT Baseline Dataset		Accuracy		Dataset		Accuracy
arXiv		0.8828		WOS-11967		0.9005
Dataset	Method	LLaMA-1B	LLaMA-3B	Phi-4	Zero Shot	k = 2	k = 3	k = 5
arXiv	Prompt-base	-	-	-	0.78	0.85	0.86	0.88
	Base Synthetic	0.8864	0.8868	0.8824	-	-	-	-
	DPO Synthetic	0.8872	0.8876	0.8904	-	-	-	-
	U-DPO	0.8884	0.8896	0.8912	-	-	-	-
WOS	Prompt-based	-	-	-	0.64	0.78	0.82	0.85
	Base Synthetic	0.9076	0.8885	0.8824	-	-	-	-
	DPO Synthetic	0.9118	0.9112	0.8904	-	-	-	-
	U-DPO	0.9143	0.9156	0.9143	-	-	-	-

, augmenting real data with synthetic samples leads to consistent accuracy improvements across both datasets. U-DPO yields the most substantial gains, indicating its effectiveness in hybrid settings. We also compare against GPT-4o prompting baselines, and classifiers trained on U-DPO synthetic data outperform both zero-shot and few-shot setups.

Furthermore, paired t-tests on 20 independent runs indicate that U-DPO consistently and significantly outperforms both the baseline (p < 0.001) and standard DPO (p < 0.05). As shown in Figure 5, U-DPO also produces more stable and higher-accuracy distributions between trials, strengthening the case for its robustness.

6.6. LLM-Based Evaluation with GPT-4.5

To assess the quality of the generated synthetic samples, we employ GPT-4.5 as an automated evaluator. Each sample is rated on a 0–5 scale based on its relevance, fluency, and class alignment. A total of 132 synthetic samples were evaluated.

Table 5. GPT4.5-based Evaluation of Synthetic Samples from the Standard DPO and U-DPO.

Statistic	Standard DPO	U-DPO
Mean	4.05	4.14
Median	4.50	4.50
Std. Dev.	1.09	0.89

reports the statistics of scores assigned to generations from Standard DPO and U-DPO. U-DPO achieves a higher average score (4.14 vs. 4.05) and a lower standard deviation (0.89 vs. 1.09), suggesting it produces more consistent and higher-quality outputs.

To complement classifier-based metrics, we assess the intrinsic quality of generated samples using GPT-4.5 as an automatic evaluator. We sample 132 synthetic texts and ask GPT-4.5 to rate each on a 0–5 scale with respect to relevance, fluency, and alignment with the intended class. As in Table 3, U-DPO samples achieve a higher mean score (4.14 vs. 4.05) and lower standard deviation (0.89 vs. 1.09) than standard DPO, indicating more consistently high-quality outputs. Medians are comparable (4.5), suggesting that U-DPO primarily reduces variance and tail failures.

6.7. Discussion of Experimental Findings

Overall, the experiments show that U-DPO yields consistent improvements. The gains are observed across multiple generators and datasets. This suggests that utility-weighted preference training is robust to the choice of generator. Importantly, these trends are stable over 20 independent runs with different random seeds, indicating that the improvements are not driven by a favorable single run but reflect a reproducible effect.

Gains are observed in:

• Margin-based label consistency—measured by the auxiliary classifier’s target-vs-competitor margin, indicating stronger class-discriminative cues in generated samples (Section 6.3);
• Downstream classification accuracy—evaluated under both the synthetic-only training regime and the hybrid real–synthetic augmentation setting (Section 6.4 and Section 6.5);
• LLM-based quality assessments—ratings of fluency/relevance and class alignment that corroborate the quantitative results (Section 6.6).

We also note that smaller models, such as LLaMA 3.2 1B, generally benefit less than larger models, suggesting that model capacity influences the effectiveness of utility-based preference optimization. Nonetheless, even the smaller models show consistent trends in the same direction. Interestingly, the downstream gains in the full-data regime are modest for LLaMA 3.2 1B. However, U-DPO yields substantial improvements at the generation stage. It produces more label-discriminative and higher-utility samples, leading to clearer gains in the synthetic augmentation setting where synthetic data quality matters most.

7. Limitations

Despite its benefits, U-DPO has several limitations:

First, U-DPO depends on the quality of the auxiliary classifier $f_{\varphi }$ and the resulting utility signal. Although we reduce direct circularity by using RoBERTa for utility scoring and SciBERT for downstream evaluation, the utility remains classifier-defined and may still encode bias or miscalibration. If $f_{\varphi }$ is biased or poorly calibrated, U-DPO may over-optimize this proxy signal rather than true generalization performance [41].

Second, our empirical evaluation is restricted to scientific article classification on two benchmark datasets. It remains unclear how well U-DPO transfers to other types of tasks, such as short-form social media texts, multi-label classification, or domains with fuzzier label boundaries and higher stylistic variance.

Third, reliance on synthetic data—even when quality-controlled—raises concerns about model collapse and bias amplification [24,25]. Iteratively training on synthetic data generated by models that were themselves trained on synthetic or biased corpora can reduce diversity and distort the learned distribution. While our hybrid experiments, which mix real and synthetic samples, show promising results, the long-term impact of heavy synthetic data usage warrants further study.

Finally, U-DPO introduces computational overhead due to the offline generation and scoring of candidate samples. Although we restrict experiments to relatively lightweight models (up to 4B parameters), scaling U-DPO to larger models or to label spaces with many classes and fine-grained distinctions could be costly.

8. Conclusions

We presented Utility DPO (U-DPO), a utility-based preference optimization framework for class-conditional synthetic text generation in multiclass classification. U-DPO redefines the preference signal using a task-specific utility that combines classifier margin (label discriminativeness) with semantic quality, enabling LLMs to generate synthetic data that is more label-consistent and more useful for downstream training than baseline generation or standard DPO tuning.

Across arXiv and WOS-11967, classifiers trained on U-DPO-generated data consistently outperform those trained on base or DPO synthetic data in both fully synthetic and hybrid real–synthetic settings. Complementary analyses—including margin-based label consistency and LLM-based quality assessment—support the same conclusion: incorporating task utility into preference optimization steers generation toward classification-relevant signal, not only surface fluency. Overall, our results highlight task-specific preference optimization as a practical route to more data-efficient model development with reduced reliance on large-scale manual annotation.

9. Future Work

Richer utility signals. Future work could extend ${\gamma }_{util}$ beyond a single classifier’s margin by incorporating ensemble agreement, uncertainty estimates, or more nuanced heuristics to better approximate true downstream utility and reduce noise [42].

Broader tasks and domains. Applying U-DPO to short-form text, multi-label or hierarchical classification, code classification, and domain-specific settings (e.g., medical or legal text) would further test generality and robustness [7]. Cross-modal extensions, such as generating text conditioned on images, are also promising [43].

Efficiency improvements. Since utility scoring is a major source of overhead, techniques such as ranking distillation, selective pair mining, caching, or joint training of the generator and auxiliary classifier could reduce computation while maintaining or improving generation quality.

Scalability across resource regimes. While this work demonstrates the efficacy of U-DPO in data-constrained settings, extending the analysis to moderate-to-high resource regimes remains an important direction to characterize the scaling behavior and potential saturation points of the proposed method.