Abstract
While reinforcement learning is widely used to align large language models (LLMs), its application to open-ended settings remains limited by reliance on preference datasets, making it difficult to scale to non-reasoning tasks such as safety. Applying direct preference optimization (DPO) to LLM safety alignment requires preferred and dispreferred responses for harmful instructions, making scalable preference construction a key bottleneck. We therefore propose GapDPO, a setwise preference optimization method that enables DPO-based safety alignment without reward models, verifiers, or preference annotation. GapDPO constructs preference sets by generating safe responses with our proposed refusal-style prompts and pairing them with affirmative targets from the dataset. GapDPO then maximizes the mean reward gap between the two sets while penalizing within-set variance. On AdvBench and HarmBench, GapDPO reduces the average attack success rate under PAIR, GCG, and AmpleGCG from 21.3% for refusal-style fine-tuning and 15.0% for single-pair DPO to 10.6%, while increasing strict set separation on training behaviors from 51% to as high as 63%. The improvement is mainly observed under GCG attacks and is accompanied by more refusals of benign requests. The averaged pairwise DPO on the same preference sets performs comparably, indicating that these sets account for most of the gain over single-pair DPO.