DPO-ESRGAN: Perceptually Enhanced Super-Resolution Using Direct Preference Optimization

Abstract

Super-resolution (SR) is a long-standing task in the field of computer vision that aims to improve the quality and resolution of an image. ESRGAN is a representative generative adversarial network specialized to produce perceptually convincing SR images. However, it often fails to recover local details and still produces blurry or unnatural visual artifacts, resulting in producing SR images that people do not prefer. To address this problem, we propose to adopt Direct Preference Optimization (DPO), which was originally devised to fine-tune large language models based on human preferences. To this end, we develop a method for applying DPO to ESRGAN, and add a DPO loss for training the ESRGAN generator. Through ×4 SR experiments utilizing benchmark datasets, it is demonstrated that the proposed method can produce SR images with a significantly higher perceptual quality and higher human preference than ESRGAN and other ESRGAN variants that have modified the loss or network structure of ESRGAN. Specifically, when compared to ESRGAN, the proposed method achieved, on average, 0.32 lower PieAPP values, 0.79 lower NIQE values, and 0.05 higher PSNR values on the BSD100 dataset, as well as 0.32 lower PieAPP values, 0.32 lower NIQE values, and 0.17 higher PSNR values on the Set14 dataset.

1. Introduction

Super-resolution (SR) aims to improve the quality and resolution of an image and is used as pre-process in various vision tasks such as remote sensing, surveillance, microscopy, and medical imaging [1]. Recent approaches are based on deep learning and are being implemented using various network architectures such as convolutional neural networks (CNNs), generative adversarial networks (GANs), Transformers, and diffusion models. Compared to the others, GAN-based approaches have shown more promising results in many benchmark datasets [1,2], drawing more attention in the computer vision and graphics society.

ESRGAN [3], which is the most well-known GAN-based approach for SR, is capable of generating SR images with significantly improved perceptual quality compared to existing SR approaches. However, while the adversarial and perceptual losses are useful for understanding local or global contexts, they hinder the recovery of pixel-level details. This causes unnatural visual artifacts, resulting in SR images that people do not prefer. To reduce these undesirable artifacts, ESRGAN performed network interpolation as follows: ${\pi }_{\theta }=(1-\alpha ){\pi }_{\theta }^{PSNR}+\alpha {\pi }_{\theta }^{GAN}$, where ${\pi }_{\theta }^{PSNR}$ is a pre-trained model obtained by minimizing L1 loss and ${\pi }_{\theta }^{GAN}$ is a GAN model obtained by fine-tuning ${\pi }_{\theta }^{PSNR}$ using the adversarial and perceptual losses. However, this approach is not an effective solution and causes the SR images to blur. Various attempts have been made to improve the performance of ESRGAN by modifying loss functions or network architectures [4,5,6]. They performed better on some perceptual metrics than ESRGAN, but did not directly consider human preferences in the training process. Therefore, they still suffer from the problem of generating low-preference SR images. In this study, our goal is to develop an effective method for training SR models to generate images that people do prefer (thus, with significantly improved perceptual quality). Inspired by Direct Preference Optimization (DPO), a learning method for fine-tuning language models based on human preferences [7], we propose a DPO-based loss function to reflect human preferences in the training process of SR models. The problem is that SR models including ESRGAN have different frameworks from language models, which means that applying DPO to SR models is not straightforward. To this end, we review the DPO formula in detail and describe how to modify it to apply it to SR models.

Our study is the first attempt to directly apply DPO to SR models. We adopt ESRGAN as the baseline model, but our method can be applied to other SR models with different network architectures without modification.

2. Related Works

2.1. Image Super-Resolution

SR is the process of enhancing a low-resolution (LR) image by increasing its resolution to reveal the underlying high-resolution (HR) image. The nature of SR is inherently ill-posed because the same LR image can be generated from different HR images. However, deep learning has easily resolved this problem and revolutionized the SR process. Early deep learning-based SR methods have used L1 or L2 loss, achieving higher PSNR—a common image quantitative quality assessment metric. However, PSNR is known to be poorly correlated with human visual perception [8], and PSNR-oriented SR produces blurry results because it does not consider the creation of high-frequency details.

Focusing more on the perceptual quality of generated images, GANs, consisting of a generator network and a discriminator network that are trained in an adversarial manner, have proven to be particularly effective for SR. SRGAN [9] introduced GAN into SR for the first time, generating more visually convincing SR images at the expense of PSNR. ESRGAN [3], an improvement over SRGAN, further increased the perceptual quality of SR images by modifying the SRGAN generator, discriminator, and loss function. Afterwards, ESRGAN has evolved in two different ways. First, more advanced network architectures and more sophisticated loss functions have been introduced [4,5,6,10,11]. Second, to address the problem of poor performance in real-world images, more accurate and realistic image degradation models have been explored [12,13]. In both ways, various attempts are still being reported.

Since SwinIR [14], the first attempt to introduce Transformer to SR, was reported, Transformer has also been applied to SR. Due to its effectiveness in learning the global context of images, Transformer helped generate SR images with more plausible and natural textures. However, due to its high computational complexity, Transformer is mainly used in combination with CNNs, showing much better performance [15].

Recently, diffusion models, which are generative models devised for image generation, have shown a high potential in SR by generating SR images that are even more visually pleasing and realistic [16]. However, diffusion models face challenges such as a high computational complexity and color shift, which require continuous follow-up studies.

In this way, SR has been implemented using various network architectures. However, this study focuses on GAN-based approaches due to their maturity and computational efficiency, as well as focusing on how to further improve their ability to generate perceptually enhanced SR images. In particular, we adopt ESRGAN as the baseline model because it is the most representative and widely used of the GAN-based SR approaches.

2.2. Direct Preference Optimization

DPO is a method that helps large, unsupervised language models better match human preferences using a simple classification approach [7]. DPO was originally proposed for language model alignment and has spread quickly as an effective fine-tuning method in various natural language processing tasks such as instruction tuning, summarization, and dialog generation. Rafailov et al. [7] showed that DPO can achieve an excellent performance in summarization tasks, and various extended studies have been reported since then. To name a few, a method of calibrating the preference optimization loss to prevent over-confidence [17], a multi-objective optimization framework that addresses the diversity of human preferences [18], a method that uses self-retrospection to enhance preference optimization [19], a method of performing fine-grained preference optimization at the token level rather than the full responses [20], and a method of adding a regularization term to the preference optimization loss to disentangle response quality from length, controlling the length bias in preference [21], have been proposed.

Recently, DPO has also been expanding into the field of image processing and computer vision, such as image generation and text-to-image alignment. Wallace et al. [22] applied DPO to a diffusion-based text-to-image generation model and made it better aligned with human preferences. Lee et al. [23] proposed a modified DPO known as the direct consistency optimization, which controls the deviation between the fine-tuned and reference models, allowing the personalized text-to-image diffusion model to generate images consistent to both subject and style. Their method is not based on human preferences, but shows that the alignment ability of DPO is also effective for training personalized image generation models. Croitoru et al. [24] proposed Curriculum DPO, which is a method that combines curriculum learning and DPO to gradually learn from a series of easy to difficult examples by dividing text–image pairs based on difficulty. Through this, both stability and performance were improved in the alignment of the diffusion model. These studies show that DPO can be useful in the field of computer vision, but most of them are still limited to text-to-image diffusion models. They do not deviate from the original DPO framework, indicating that they cannot be applied to other models or network architectures. Most recently, as the only study to apply DPO to SR, direct semantic preference optimization (DSPO) [25], which is a method for suppressing visual artifacts by introducing semantic-level preference alignment in real-world SR, has been proposed. However, DSPO is also based on diffusion models; thus, it is not applicable to other network architectures and requires instance-level preference datasets.

Unlike the existing vision-domain DPO-based methods, our method provides a breakthrough to extend DPO to models or network architectures other than diffusion models, without the need for human-labeled preference datasets.

3. Proposed Method

3.1. Preliminaries

DPO is at the heart of the proposed method, so this section briefly reviews it.

Reinforcement learning methods such as reinforcement learning from human feedback (RLHF) [26], which fine-tune generative models to generate the output that people prefer, have recently emerged. They have played an important role in improving the performance of large-scale models (LMs) such as language models and diffusion models. RLHF trains a reward model ($r_{\varphi }$) using the human preference dataset (D) by minimizing the negative log-likelihood loss, $-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \left(r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)\right)\right]$, where $\sigma$ is the sigmoid function, while $y_w$ and $y_l$ represent the preferred and non-preferred outputs for the input x. Then, RLHF optimizes LMs (${\pi }_{\theta }$) to produce outputs with high rewards while constraining them not to stray too far from the pre-trained LMs’ outputs, as follows:

$$\mathrm{RLHF}:\underset{{\pi }_{\theta }}{max}{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }\left(y\right|x)}\left[r_{\varphi }(x,y)-\beta {\mathbb{D}}_{KL}\left({\pi }_{\theta }\left(y\right|x),{\pi }_{ref}\left(y\right|x)\right)\right].$$

(1)

Here, $\beta$ is a parameter controlling the deviation from the pre-trained reference model (${\pi }_{ref}$), while ${\mathbb{D}}_{KL}$ represents the Kullback–Leibler (KL) divergence. Since Equation (1) is not differentiated, optimization is performed through proximal policy optimization [27], which is one of the reinforcement learning methods. That is, with Equation (1) as the final reward for reinforcement learning, the policy model is trained so that the reward increases.

DPO sticks to the KL-constrained reward maximization, which is the strategy of RLHF for preference learning. However, DPO eliminated the process of training a reward model in advance, as well as the process of reinforcement learning for policy optimization, by reparameterizing the reward model with some algebra and applying the reparameterization to the Bradley–Terry preference model [28], enabling us to solve the RLHF problem using a simple differentiable loss, as follows:

$${\mathcal{L}}_{DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \left(\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_w\right|x)}{{\pi }_{ref}\left(y_w\right|x)}}-\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_l\right|x)}{{\pi }_{ref}\left(y_l\right|x)}}\right)\right].$$

(2)

Here, $\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y\right|x)}{{\pi }_{ref}\left(y\right|x)}}$ is an implicit reward model defined by the log probability of outputs. Therefore, the reward, i.e., the relative log probability of preferred to non-preferred outputs, increases during the training of the policy model (${\pi }_{\theta }$). The common parameter of the two reward terms, $\beta$, plays the same role as in Equation (1). That is, a large $\beta$ increases the reward, resulting in a reduced DPO loss and the suppressed learning of ${\pi }_{\theta }$. This constrains ${\pi }_{\theta }$ from deviating from the reference model. The detailed process for deriving Equation (2) from Equation (1) can be found in [7]. The fine-tuned language model using the DPO loss (Equation (2)) showed a better performance in summary and single-turn problems compared to the RLHF fine-tuned model [7].

We adopt ESRGAN as the baseline model, so we briefly review it here. ESRGAN is a GAN-based SR model [3]. It consists of a generator and a discriminator, which are trained in an adversarial manner. In other words, the generator is trained to generate more natural SR images, while the discriminator is trained to better distinguish between the generated SR images and the real HR images. This allows the generator to generate high-perceptual-quality SR images. The generator is trained with the following combination loss:

$${\mathcal{L}}_{ESRGAN}={\alpha }_1{\mathcal{L}}_{adv}+{\alpha }_2{\mathcal{L}}_{percep}+{\alpha }_3{\mathcal{L}}_1.$$

(3)

Here, ${\mathcal{L}}_{adv}$, ${\mathcal{L}}_{percep}$, and ${\mathcal{L}}_1$ are adversarial, perceptual, and L1 losses, respectively. The adversarial loss is computed from the discriminator output, indicating the probability that the generated image is discriminated as unrealistic. The perceptual loss represents the difference between the VGG [29] features of the SR and HR images. In summary, to train the generator, ESRGAN uses losses that reflect the perceptual quality and authenticity of images, but that are not directly related to human preferences.

3.2. Introducing DPO into the SR Process

We propose an SR-DPO loss in the training process to improve ESRGAN’s performance via preference learning. Here, we describe how to derive the SR-DPO loss.

In order to calculate the DPO loss in Equation (2), the preferred output ($y_w$) and the non-preferred output ($y_l$) for the input x are required. However, unlike language models or diffusion models that produce probabilistically different outputs from the inputs, SR models always produce the same outputs from the inputs. Therefore, the DPO loss of Equation (2) cannot be directly applied to SR models such as ESRGAN. To resolve this inherent problem, we obtain preferred and non-preferred outputs from the outputs of $N_B$ images included in the input batch (denoted as $x_B$). That is, the DPO loss is modified to be calculated using the input $x_w$, which generates the output $y_w$ with the highest preference among the input images, and the input $x_l$, which generates the output $y_l$ with the lowest preference. The preference calculation is performed using the pre-trained PieAPP model [30], and the output with the lowest PieAPP value is determined as the preferred output, while the output with the highest PieAPP value is determined as the non-preferred output. The PieAPP model learned human preferences for images, i.e., a deep CNN model was trained to predict the probability that humans will prefer an image over the other using large-scale human-labeled preference datasets. It outputs image quality values correlated with real human preference responses; therefore, it fits our purpose well. Since we use the pre-trained and frozen PieAPP model to calculate human preferences, we can omit the explicit or implicit reward model training process required by DPO or RLHF. Furthermore, we no longer need to prepare the human preference datasets. Based on this, Equation (2) is modified as follows:

$${\mathcal{L}}_{SR\_DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{E}}_{x_B\sim \mathcal{I},(y_w,y_l)\sim {\pi }_{\theta }\left(y_B\right|x_B)}\left[\log \sigma \left(\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_w\right|x_B)}{{\pi }_{ref}\left(y_w\right|x_B)}}-\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_l\right|x_B)}{{\pi }_{ref}\left(y_l\right|x_B)}}\right)\right].$$

(4)

Here, $\mathcal{I}$ is an image dataset that does not include human preferences and $\pi$ represents a model in which the SR model and the frozen PieAPP model are connected. The output of $\pi$ is a PieAPP value, not a probability, unlike the original DPO. However, this does not matter because the reward value is relatively computed from the ratio between preferred and non-preferred $\pi$ outputs. However, considering that the smaller the PieAPP value is, the more preferred the image is, as opposed to the reward value of DPO, we need to change the order of two reward terms as follows:

$${\mathcal{L}}_{SR\_DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{E}}_{x_B\sim \mathcal{I},(y_w,y_l)\sim {\pi }_{\theta }\left(y_B\right|x_B)}\left[\log \sigma \left(\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_l\right|x_B)}{{\pi }_{ref}\left(y_l\right|x_B)}}-\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_w\right|x_B)}{{\pi }_{ref}\left(y_w\right|x_B)}}\right)\right].$$

(5)

This loss allows SR models to be trained to make the non-preferred results less preferred and the preferred results more preferred. However, considering that the PieAPP value is always above 0, it is relatively easier for SR models to be trained in the direction of making the PieAPP values larger than in the direction of making the PieAPP values smaller; therefore, it is highly likely that only the training that makes SR images that are not preferred more non-preferred will proceed. Above all, the performance of SR models can be degraded if SR images that are not preferred are made to be more non-preferred. For this reason, we can prevent backpropagation through the first reward term (which is simply implemented by using the PyTorch detach function), enabling only the training that makes the SR images that are preferred more preferred, as follows:

$${\mathcal{L}}_{SR\_DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{E}}_{x_B\sim \mathcal{I},(y_w,y_l)\sim {\pi }_{\theta }\left(y_B\right|x_B)}\left[\log \sigma \left(\beta \log {\displaystyle \frac{{\pi }_{{\theta }_{no\_grad}}\left(y_l\right|x_B)}{{\pi }_{ref}\left(y_l\right|x_B)}}-\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_w\right|x_B)}{{\pi }_{ref}\left(y_w\right|x_B)}}\right)\right].$$

(6)

However, this method is not very helpful in improving the performance of SR models. This will be shown in the experimental results later.

Unlike language models, SR models can perform better by training in a way that increases the preference of relatively non-preferred SR images. Therefore, we return to Equation (4). However, we prevent backpropagation through the first reward term, thus disabling the training that makes the preferred SR images less preferred. As a result, our final SR-DPO loss is defined as follows:

$${\mathcal{L}}_{SR\_DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{E}}_{x_B\sim \mathcal{I},(y_w,y_l)\sim {\pi }_{\theta }\left(y_B\right|x_B)}\left[\log \sigma \left(\beta \log {\displaystyle \frac{{\pi }_{{\theta }_{no\_grad}}\left(y_w\right|x_B)}{{\pi }_{ref}\left(y_w\right|x_B)}}-\beta \log {\displaystyle \frac{{\pi }_{\theta }\left(y_l\right|x_B)}{{\pi }_{ref}\left(y_l\right|x_B)}}\right)\right].$$

(7)

This loss serves to increase the preference for low-preference SR images while maintaining the preference of high-preference SR images by comparing the preferences of SR images in the batch.

3.3. Determining the Reference Model

DPO is an approach to fine-tune a given reference model through preference learning. Therefore, we need to determine the reference model for our study. At first, we considered using a fully trained ESRGAN generator with the L1, perceptual, and adversarial losses as a reference model. However, this forces ESRGAN to be tweaked, relying too heavily on PieAPP values, resulting in a problem whereby PieAPP values decrease compared to the reference model; however, the performance in other evaluation metrics is rather poor. This will be shown in the experimental results later.

As a result, we use an ESRGAN generator trained only with L1 loss as a reference model. However, in the subsequent fine-tuning process, the SR-DPO loss and the perceptual and adversarial losses used in ESRGAN are used together. This allows us to further improve the perceptual quality of SR images through preference learning without degrading the original performance of ESRGAN.

Additionally, we also consider not using the reference model. That is, unlike DPO, where the relative reward to the reference model is maximized, we attempt to maximize the absolute reward of the policy model. This is the same as not using the KL divergence constraint in Equation (1). The SR-DPO loss without the reference model is defined as follows:

$${\mathcal{L}}_{SR\_DPO\_NoRef}\left({\pi }_{\theta }\right)=-{\mathbb{E}}_{x_B\sim \mathcal{I},(y_w,y_l)\sim {\pi }_{\theta }\left(y_B\right|x_B)}\left[\log \sigma \left(\beta \log {\pi }_{{\theta }_{no\_grad}}\left(y_w\right|x_B)-\beta \log {\pi }_{\theta }\left(y_l\right|x_B)\right)\right].$$

(8)

3.4. Training ESRGAN Using the SR-DPO Loss

The proposed method can be summarized as fine-tuning the ESRGAN generator using an additional loss that is related to user preferences. Figure 1 shows the process flow of the proposed method. The SR-DPO loss is shared among the images in the input batch because it is computed once per batch, while the other losses are computed independently for each image. The generator is fine-tuned to make $y_l^{\left(i\right)}$ more preferred, while being constrained to make it similar to $y_l^{\left(0\right)}$. That is, regardless of whether the reference model is used or not, the ESRGAN generator is pre-trained first with only the L1 loss and fine-tuned with the discriminator using a combination loss, as follows:

$${\mathcal{L}}_{+DPO}={\mathcal{L}}_{ESRGAN}+{\alpha }_4{\mathcal{L}}_{SR\_DPO}.$$

(9)

Here, ${\mathcal{L}}_{ESRGAN}$ represents the total loss of the original ESRGAN in Equation (3). In ${\mathcal{L}}_{SR\_DPO}$, $\beta$ is set to 0.1, because too small a value cannot preserve the performance of the reference model, while a value greater than 0.2 degrades the performance of preference learning [7].

4. Experimental Results and Discussion

4.1. Setup

For the experiment, we trained ESRGAN and its variants using the DIV2K [31] training dataset. To increase the amount of training data, data augmentation was performed through random horizontal or vertical flips. To test the trained models, we used the DIV2K validation dataset, Urban100 [32], BSD100 [33], Set5 [34], and Set14 [35].

For training, HR images were randomly cropped to $128\times 128$ and downsampled using bicubic interpolation with a scaling factor of 4 to obtain LR images. All the training parameters of the original ESRGAN method were maintained, except that the mini-batch size ($N_B$) was set to 4. The number of epochs (N) was 500. Only the generator was trained with the L1 loss during the first 200 epochs, and the generator and discriminator were adversarially trained using all the losses in Equations (3) and (9), where ${\alpha }_1=5\times {10}^{-3}$, ${\alpha }_2=1.0$, ${\alpha }_3=1\times {10}^{-2}$, and ${\alpha }_4=1.0$, during the remaining epochs. The learning rate was set to $1\times {10}^{-4}$ and was halved at $[0.125,0.250,0.500,0.750]\times N$ epochs. The model was optimized using Adam with ${\beta }_1=0.9$ and ${\beta }_2=0.999$. The implementation was carried out using PyTorch 2.1.0 and training on a PC with Intel i7 2.1 GHz CPU and NVIDIA RTX 3090 GPU.

To evaluate the quantitative and qualitative quality of SR images, we computed the values of PSNR, SSIM, LPIPS, PieAPP, and NIQE. NIQE is a no-reference image quality metric, and the others are full-reference ones. PSNR has been most commonly used to measure the quality of reconstructed images, comparing the maximum value of pixels ($Peak$) with the mean-squared error ($MSE$) between the SR and HR images [36,37,38]. This is defined as follows: $PSNR=20{log}_{10}Peak-10{log}_{10}MSE$. SSIM estimates the similarity between SR and HR images and is defined as follows: $SSIM=l(y,\overline{y})·c(y,\overline{y})·s(y,\overline{y})$. Here, y and $\overline{y}$ represent SR and HR images, respectively. l, c, and s represent the luminance, contrast, and structural similarities between two images, respectively, and can be computed from the average, standard deviation, and covariance of the pixel values [36,39,40]. Therefore, higher PSNR and SSIM values indicate better image quality. LPIPS extracts features from images using a CNN model pre-trained on an image classification task and computes the L2 distance between the features, as follows: $LPIPS={\sum }_l{\displaystyle \frac{1}{H_l{W}_l}}{\sum }_{h,w}{\left[w_l(f_{h,w}^l-{\overline{f}}_{h,w}^l)\right]}^2$ [41]. Here, $f^l$ and ${\overline{f}}^l$ are the SR and HR feature maps obtained from the l-th layer of the pre-trained CNN model. $w_l$, $H_l$, and $W_l$ are the scale factor, the height, and the width of the feature maps for layer l, respectively. PieAPP trains a CNN model using a pairwise learning framework to predict the probability that humans will prefer one image over the other on human-labeled preference datasets, and the trained model is used to measure the perceptual difference between images [30]. The PieAPP value is more correlated with human preferences. NIQE measures deviations from statistical regularities observed in natural images, without reference images [42]. It extracts statistical features based on a natural scene statistic (NSS) model from images and computes a multivariate Gaussian (MVG) fit of the features. The quality of an image is expressed as the distance between its MVG fit and the MVG fit of the NSS features extracted from natural images, as follows: $NIQE=\sqrt{{({\nu }_1-{\nu }_2)}^T{\left({\displaystyle \frac{{\Sigma }_1+{\Sigma }_2}{2}}\right)}^{-1}({\nu }_1-{\nu }_2)}$. Here, $\nu$ and $\Sigma$ are the mean vectors and covariance matrices of the MVG fits. Therefore, lower PieAPP, LPIPS, and NIQE values indicate better image quality.

4.2. Effectiveness of Using the SR-DPO Loss

To show the effectiveness of using the SR-DPO loss, we compared ESRGANs trained with different losses—${\mathcal{L}}_{ESRGAN}$, ${\mathcal{L}}_{+PieAPP}$, and ${\mathcal{L}}_{+DPO}$. ${\mathcal{L}}_{+PieAPP}$ is the addition of the PieAPP values of generated SR images to ${\mathcal{L}}_{ESRGAN}$, which is proposed in [5] and has proven to outperform other loss modifications.

Table 1. Comparison of ESRGANs trained with different losses. ${\mathcal{L}}_{+PieAPP}$ and ${\mathcal{L}}_{+DPO}$ represent the addition of the PieAPP loss and the proposed SR-DPO loss to ${\mathcal{L}}_{ESRGAN}$, respectively.

		PSNR ↑	SSIM ↑	LPIPS ↓	PieAPP ↓	NIQE ↓
DIV2K_valid	${\mathcal{L}}_{ESRGAN}$ [3]	25.6795	0.7278	0.1237	0.5507	4.6551
	${\mathcal{L}}_{+PieAPP}$ [5]	25.4889	0.7275	0.1414	0.2749	4.3927
	${\mathcal{L}}_{+DPO}$	25.9753	07409	0.1244	0.2626	4.2588
Urban100	${\mathcal{L}}_{ESRGAN}$ [3]	22.0044	0.6752	0.1587	0.8355	4.5552
	${\mathcal{L}}_{+PieAPP}$ [5]	21.6073	0.6635	0.1831	0.5079	4.4453
	${\mathcal{L}}_{+DPO}$	22.1408	0.6860	0.1598	0.5072	4.4582
BSD100	${\mathcal{L}}_{ESRGAN}$ [3]	23.4732	0.6037	0.1653	0.6956	5.4894
	${\mathcal{L}}_{+PieAPP}$ [5]	22.8331	0.5874	0.1901	0.4011	5.0825
	${\mathcal{L}}_{+DPO}$	23.5257	0.5998	0.1696	0.3798	4.6984
Set5	${\mathcal{L}}_{ESRGAN}$ [3]	26.6157	0.8027	0.0755	0.4486	5.4038
	${\mathcal{L}}_{+PieAPP}$ [5]	26.0691	0.7881	0.0917	0.3511	4.9664
	${\mathcal{L}}_{+DPO}$	26.8175	0.7996	0.0748	0.3564	5.3249
Set14	${\mathcal{L}}_{ESRGAN}$ [3]	23.8251	0.7659	0.1446	0.9212	4.9711
	${\mathcal{L}}_{+PieAPP}$ [5]	22.8668	0.7521	0.1626	0.5072	4.5064
	${\mathcal{L}}_{+DPO}$	23.9951	0.7653	0.1432	0.5985	4.6497

shows the results of the quality metrics. Compared to ${\mathcal{L}}_{ESRGAN}$, ${\mathcal{L}}_{+DPO}$ contributed to a significant reduction in PieAPP values indicating user preferences, and it can be seen that its effectiveness is higher than ${\mathcal{L}}_{+PieAPP}$. In addition, it can be seen that ${\mathcal{L}}_{+PieAPP}$ has the problem of increasing the LPIPS values, but ${\mathcal{L}}_{+DPO}$ has similar LPIPS values to ${\mathcal{L}}_{ESRGAN}$. Quantitative quality metrics such as PSNR or SSIM are also degraded using ${\mathcal{L}}_{+PieAPP}$, but are maintained or improved using ${\mathcal{L}}_{+DPO}$. Both ${\mathcal{L}}_{+PieAPP}$ and ${\mathcal{L}}_{+DPO}$ contributed to reducing the NIQE values, and the performance difference between the two was not significant. Consequently, it can be said that ${\mathcal{L}}_{+DPO}$ effectively improves the quantitative and perceptual image quality of SR images without compromising the performance of ESRGAN, and ${\mathcal{L}}_{+DPO}$ outperforms ${\mathcal{L}}_{+PieAPP}$.

Figure 2 shows some SR images generated using ${\mathcal{L}}_{ESRGAN}$, ${\mathcal{L}}_{+PieAPP}$, and ${\mathcal{L}}_{+DPO}$, respectively. ${\mathcal{L}}_{ESRGAN}$ suffered from recovering local details and caused unnatural visual artifacts (e.g., line artifacts in the crosswalk or severely distorted lines in the wood fence). ${\mathcal{L}}_{+PieAPP}$ reduced the visual artifacts in some images, but tended to lose structural contexts more than ${\mathcal{L}}_{ESRGAN}$, which may be the reason why its SSIM and LPIPS values were worse than ${\mathcal{L}}_{ESRGAN}$. As expected, ${\mathcal{L}}_{+DPO}$ recovered local details much better (e.g., the boundaries of the white dots on the butterfly and the stripes on the tiger are clear, and their shapes are more similar to the ground-truth), while minimizing unnatural visual artifacts (the line artifacts at the crosswalk are not strong).

In this study, we do not replace or eliminate the losses originally used for ESRGAN, as the goal is to fine-tune ESRGAN by adding the SR-DPO loss to the existing losses. In other words, the SR-DPO loss cannot replace the original losses. Referring to [5], it is predicable that any replacement or elimination will significantly degrade the performance of ESRGAN because the original losses play an important role in achieving a high perceptual quality of the SR images.

4.3. Performance Comparison Based on How to Design the SR-DPO Loss

Section 3.2 described how we designed the SR-DPO loss. In this section, we show how much the performance degrades when the SR-DPO loss is designed differently. In

Table 2. Performance changes based on how the SR-DPO loss is designed. Equations (4)–(7) represent the DPO losses we designed to introduce DPO to the SR process in Section 3.2, and Equation (7) is our final SR-DPO loss.

		PSNR ↑	SSIM ↑	LPIPS ↓	PieAPP ↓	NIQE ↓
DIV2K_valid	Equation (4)	24.0414	0.7183	0.1616	1.0361	4.3884
	Equation (5)	20.7606	0.7073	0.2426	2.0506	5.5587
	Equation (6)	20.3597	0.6494	0.2698	2.0456	4.9085
	Equation (7)	25.9753	0.7409	0.1244	0.2626	4.2588
Urban100	Equation (4)	20.8525	0.6496	0.1966	1.1522	4.2652
	Equation (5)	18.4527	0.6154	0.2761	1.8359	5.1611
	Equation (6)	18.3813	0.5535	0.3031	1.8517	4.7215
	Equation (7)	22.1408	0.6860	0.1598	0.5072	4.4582
BSD100	Equation (4)	22.7156	0.5885	0.2064	1.1245	5.4479
	Equation (5)	20.6213	0.5903	0.2952	1.7535	6.9459
	Equation (6)	20.1085	0.5366	0.3418	1.8276	6.0400
	Equation (7)	23.5257	0.5998	0.1696	0.3798	4.6984
Set5	Equation (4)	24.2179	0.7456	0.1198	0.5111	4.9955
	Equation (5)	21.4562	0.7253	0.1712	0.9298	6.8279
	Equation (6)	19.7723	0.6271	0.2044	1.9173	6.5020
	Equation (7)	26.8175	0.7996	0.0748	0.3564	5.3249
Set14	Equation (4)	22.8015	0.7515	0.1814	1.3275	5.0441
	Equation (5)	20.2181	0.7316	0.2638	2.6234	6.0784
	Equation (6)	18.8591	0.6548	0.3062	1.5138	5.6434
	Equation (7)	23.9951	0.7653	0.1432	0.5985	4.6497

, we compared the results of using Equations (4), (5), (6), and (7), respectively. As aforementioned, it was not good for SR to make the non-preferred results more non-preferred and the preferred results more preferred (Equation (5)). The image quality became severely worse on all metrics. Rather, making preferred SR images less preferred and non-preferred images more preferred resulted in better results (Equation (4)). It seems that the performance improvement by making non-preferred images preferred is relatively large. However, making preferred SR images less preferred has hindered performance improvement, as expected. Performing only training that makes the preferred SR images more preferred also did not result in performance improvement (Equation (6)). Only the proposed scheme that makes non-preferred images preferred has resulted in high performance improvements (Equation (7)).

4.4. Influence of the Reference Model

As mentioned in Section 3.3, different reference models were considered. First, a fully trained ESRGAN with ${\mathcal{L}}_{ESRGAN}$ was used as a reference model and was fine-tuned with ${\mathcal{L}}_{SR\_DPO}$. Second, an ESRGAN trained with ${\mathcal{L}}_1$ was used as a reference model and was fine-tuned with ${\mathcal{L}}_{+DPO}$. Third, no reference model was used and the L1-trained ESRGAN was fine-tuned with ${\mathcal{L}}_{+DPO\_NoRef}$, which replaced ${\mathcal{L}}_{SR\_DPO}$ of ${\mathcal{L}}_{+DPO}$ with ${\mathcal{L}}_{SR\_DPO\_NoRef}$ in Equation (8).

Table 3. Performance changes with reference model in DPO-based SR.

	Reference Model	PSNR ↑	SSIM ↑	LPIPS ↓	PieAPP ↓
DIV2K_valid	ESRGAN trained with ${\mathcal{L}}_{ESRGAN}$	15.2341	0.5422	0.3866	0.4054
	ESRGAN trained with ${\mathcal{L}}_1$	25.9753	0.7409	0.1244	0.2626
	No reference model	25.8533	0.7452	0.1259	0.2547
Urban100	ESRGAN trained with ${\mathcal{L}}_{ESRGAN}$	13.9608	0.4315	0.4033	0.6819
	ESRGAN trained with ${\mathcal{L}}_1$	22.1408	0.6860	0.1598	0.5072
	No reference model	22.1086	0.6193	0.1624	0.4574
Set14	ESRGAN trained with ${\mathcal{L}}_{ESRGAN}$	13.3721	0.5037	0.4509	0.5808
	ESRGAN trained with ${\mathcal{L}}_1$	23.9951	0.7653	0.1432	0.5985
	No reference model	23.8403	0.7581	0.1498	0.5224

shows the results. In the first case, the PieAPP value decreased compared to the reference ESRGAN, but the performance was significantly reduced in other evaluation metrics by focusing too much on lowering the PieAPP value. In fact, our DPO method has to maintain the original performance due to the KL constraint. However, a small $\beta$ could not guarantee performance maintenance. This also implies that the DPO loss is not in redundancy with the perceptual and adversarial losses of ESRGAN. When the L1-trained ESRGAN was used as a reference model, preference learning was performed effectively while maintaining the performance of the reference model, and PieAPP and LPIPS values were greatly improved. When no reference model was used, PieAPP values could have been much lower by not caring about maintaining the performance of the reference model, but the performance was worse than that when using the L1-trained reference model in other evaluation metrics.

4.5. Performance Comparison of the PieAPP and LPIPS Models for Preference Calculation

As explained in Section 3.2, our method uses the pre-trained PieAPP model for preference calculation. To show its suitability, we attempted to use the pre-trained LPIPS model [41] for the calculation and analyzed its results. Like the PieAPP value, the smaller the LPIPS value, the more preferred the image; therefore, our formula for DPO-based SR can be used as is. The results are shown in

Table 4. Performance when using the LPIPS model for preference calculation.

	PSNR ↑	SSIM ↑	LPIPS ↓	PieAPP ↓	NIQE ↓
DIV2K_valid	25.9707	0.7446	0.1186	0.5191	4.0304
Urban100	22.1383	0.6866	0.1566	0.7609	4.0996
BSD100	23.9345	0.6150	0.1613	0.6565	4.6466
Set5	26.8093	0.8044	0.0718	0.5903	4.9209
Set14	24.2466	0.7757	0.1337	0.7321	4.4916

. Even with the LPIPS model, the perceptual quality of the SR images could be improved while the performance of the reference model (vanilla ESRGAN) could be maintained. However, compared to the results of using the PieAPP model for preference calculation (the results of ${\mathcal{L}}_{+DPO}$ in Table 1), the PieAPP values are significantly higher, although the LPIPS and NIQE values were slightly lower. Low NIQE values did not always imply perceptually better SR results. As shown in Figure 3, the LPIPS model tended to produce a rich but unnatural texture. The LPIPS model also caused a color shift in the SR images. In addition, in the remaining metrics (PSNR and SSIM), the performance of using the LPIPS model was worse. The performance difference between the datasets was not discernible. As a result, we confirm that the PieAPP model is more suitable for preference calculation. This was predictable because the PieAPP model was trained to infer the difference between human preferences, while the LPIPS model was trained to infer the difference between perceptual features, as explained in Section 4.1.

4.6. Influence of the Batch Size for Preference Calculation

As explained in Section 3.2, our DPO method determines preferred and non-preferred outputs within a batch, indicating that the performance may be dependent on the batch size ($N_B$). Therefore, we wanted to analyze the effect of batch size on performance. However, due to the lack of GPU memory, the batch size could not be set higher than 8. We simply compared the results of increasing the batch size to 8 with the results presented previously (obtained with a batch size of 4).

Table 5. Performance when increasing the batch size to 8.

	PSNR ↑	SSIM ↑	LPIPS ↓	PieAPP ↓	NIQE ↓
DIV2K_valid	25.7276	0.7134	0.1282	0.2828	4.1035
Urban100	21.9077	0.6654	0.1601	0.5648	4.0682
BSD100	23.5713	0.6057	0.1716	0.3612	4.9198
Set5	26.6223	0.8011	0.0784	0.3221	4.7340
Set14	23.3602	0.7508	0.1448	0.5263	4.6600

shows the results when the batch size is 8. Compared to the results when the batch size is 4 (the results of ${\mathcal{L}}_{+DPO}$ in Table 1), although there are some differences depending on the dataset, the difference in performance was not discernible according to the batch size. If anything, the PSNR and LPIPS values were slightly better with the smaller batch size. Therefore, smaller batch sizes are preferred.

4.7. Performance Comparison with Other ESRGAN Improvement Models

Our DPO method modifies only the loss function without modifying the network architecture of ESRGAN. In Table 1, it was shown that our method outperforms other methods that have attempted to modify the loss function. Here, our DPO method is compared with recent methods that have attempted to modify the network architecture of the ESRGAN generator or discriminator—Real-ESRGAN [13], StarSRGAN [43], ESRGAN-DP [10], A-ESRGAN [11], MSA-ESRGAN [6], and SeD [44]. The results are given in

Table 6. Comparison with other ESRGAN improvement models.

		PSNR ↑	SSIM ↑	LPIPS ↓	PieAPP ↓	NIQE ↓
DIV2K_valid	Bicubic	26.6942	0.7663	0.3407	0.5804	7.2218
	Our DPO method	25.9753	0.7409	0.1244	0.2626	4.2588
	Real-ESRGAN [13]	21.8301	0.6209	0.2758	1.6711	3.5661
	StarSRGAN [43]	24.5426	0.7147	0.1464	0.6914	3.0836
	ESRGAN-DP [10]	26.5069	0.7555	0.0912	0.3996	2.8800
	A-ESRGAN [11]	22.7443	0.6550	0.2358	1.3846	3.1801
	MSA-ESRGAN [6]	24.8405	0.7204	0.1799	1.1802	3.5811
	SeD [44]	27.7939	0.7934	0.0751	0.3469	3.1302
Urban100	Bicubic	21.6991	0.6517	0.4205	1.1409	7.1941
	Our DPO method	22.1408	0.6860	0.1598	0.5072	4.4582
	Real-ESRGAN [13]	18.1046	0.5409	0.2666	2.2056	4.2697
	StarSRGAN [43]	20.3558	0.6483	0.1712	1.1067	3.5930
	ESRGAN-DP [10]	22.7345	0.7149	0.1086	0.6753	3.6833
	A-ESRGAN [11]	18.7962	0.5721	0.2451	1.4945	3.5516
	MSA-ESRGAN [6]	21.0561	0.6575	0.1804	1.6531	4.0658
	SeD [44]	24.3847	0.7714	0.0887	0.6458	3.9645
BSD100	Bicubic	24.6507	0.6415	0.4561	0.7388	7.5764
	Our DPO method	23.5257	0.5998	0.1696	0.3798	4.6984
	Real-ESRGAN [13]	21.1057	0.5145	0.3272	2.0294	3.9643
	StarSRGAN [43]	22.7458	0.6093	0.1769	0.9638	3.8182
	ESRGAN-DP [10]	23.9337	0.6397	0.1328	0.6465	3.3965
	A-ESRGAN [11]	21.3103	0.5405	0.2711	1.5606	3.6621
	MSA-ESRGAN [6]	23.5678	0.6121	0.2372	1.2789	4.0619
	SeD [44]	25.0264	0.6638	0.1224	0.5664	3.5617
Set5	Bicubic	26.6902	0.7899	0.3004	0.9857	8.2823
	Our DPO method	26.8175	0.7996	0.0748	0.3564	5.3249
	Real-ESRGAN [13]	21.6163	0.6269	0.2277	1.5581	5.7241
	StarSRGAN [43]	24.9597	0.7691	0.1107	0.7751	4.3798
	ESRGAN-DP [10]	28.3406	0.8287	0.0598	0.3684	4.1061
	A-ESRGAN [11]	21.9001	0.6427	0.1735	0.6849	4.8551
	MSA-ESRGAN [6]	24.3240	0.7403	0.1438	1.4387	6.5172
	SeD [44]	29.3011	0.8511	0.0521	0.3867	5.3606
Set14	Bicubic	24.2384	0.7737	0.3862	0.7347	7.7363
	Our DPO method	23.9951	0.7653	0.1432	0.5985	4.6497
	Real-ESRGAN [13]	20.3295	0.5336	0.2977	2.7718	4.2195
	StarSRGAN [43]	23.3203	0.6298	0.1732	1.0428	3.8000
	ESRGAN-DP [10]	24.4899	0.6859	0.1158	0.7899	3.3932
	A-ESRGAN [11]	20.7806	0.6487	0.2367	1.7234	3.6678
	MSA-ESRGAN [6]	23.3530	0.7386	0.1979	1.3044	4.5716
	SeD [44]	25.4956	0.8026	0.0969	0.6217	3.8461

.

In the PieAPP values, the performance of our DPO method was the best. However, because our DPO method aims to improve the perceptual quality or human preference of SR images while maintaining the performance of ESRGAN, the degree of improvement in other performance indicators is not so high, resulting in a lower performance than SeD (However, the results of SeD were obtained using the weight file provided by the authors) [44]. The weight file was obtained using a different training dataset from ours and a larger mini-batch size. This may be why SeD performs better than our DPO method). This shows that even if we try to improve the performance of the generator by improving the loss function in the GAN structure, the degree of performance improvement may not be high unless the performance of the discriminator is improved together. This also implies that the performance of ESRGAN can be improved more significantly by improving the performance of the discriminator. ESRGAN’s discriminator was not helpful for recovering local details at the pixel level because it classifies images at the image level. This is why most ESRGAN improvements focused on improving the discriminator. It seems that SeD has effectively solved the ESRGAN discriminator problem. However, when incorporated into the image-wise discriminator, even SeD experienced significant performance degradation [44]. For this reason, as shown in Figure 4, our DPO method was not able to recover local fine details as clearly as SeD (e.g., the dense stripes on the roof are broken or distorted). Nevertheless, our DPO method performed better than the other methods. ESRGAN-DP achieved values close to those of SeD in all metrics, thanks to the use of ResNet features that complement VGG features in calculating ${\mathcal{L}}_{percep}$. However, ESRGAN-DP still had a limitation in recovering fine details, and often produced annoying visual artifacts or structural distortions that degraded the perceptual quality of SR images. The overall perceptual quality was worse than that of our DPO method. Real-ESRGAN, StarSRGAN, A-ESRGAN, and MSA-ESRGAN have also improved the performance of the ESRGAN discriminator, i.e., allowing for the classification of images at the pixel level by leveraging the U-Net discriminator [45]. However, they improved only the NIQE value and underperformed in other performance metrics. This seems to be due to recovering the rich but unnatural local texture (which can be seen in Figure 4). A-ESRGAN tended to smooth out the fine details and thicken the lines, and Real-ESRGAN and MSA-ESRGAN also reported this tendency. A-ESRGAN and Real-ESRGAN also distorted local structures more significantly. StarSRGAN integrated the network architectures and losses used in Real-ESRGAN, A-ESRGAN, and ESRGAN-DP, thus achieving better performance than Real-ESRGAN, A-ESRGAN, and MSA-ESRGAN. However, StarSRGAN still produced SR images with structural distortion and unnatural textures and performed worse than our DPO method, which simply introduced the DPO loss to the vanilla ESRGAN. In all performance metrics except NIQE, Real-ESRGAN, StarSRGAN, A-ESRGAN, and MSA-ESRGAN performed worse than our DPO method.

5. Conclusions and Future Works

This study proposed DPO-ESRGAN, a method that applied DPO to ESRGAN. The DPO formula was modified to fit the SR process. DPO-ESRGAN used the pre-trained PieAPP model for preference calculation and could be trained without human preference datasets. Through ×4 SR experiments utilizing benchmark datasets, it was demonstrated that DPO-ESRGAN can generate SR images with a significantly higher perceptual quality and human preference than ESRGAN and other ESRGAN variants. DPO-ESRGAN worked well for small batch sizes. Although we were unable to perform a full analysis, smaller batch sizes yielded slightly better results. This study aims to generate SR images with high perceptual quality and human preference. Therefore, our method is more useful in applications such as multimedia industry and commercial image/video enhancement than applications that require high fidelity, such as forensics, medical diagnosis, and biometrics.

In this study, we adopted ESRGAN as the baseline model, but our DPO method can be applied to other SR models with different network architectures without modification. However, to verify this applicability, experimental validation is required. Related experiments remain a part of our future study.

As discussed in Section 4.7, our DPO method did not achieve as impressive a performance improvement as expected, due to the tendency to maintain the performance of the reference model, as well as the lack of discriminator performance. For further improvements, we believe that the performance of the discriminator must be improved, which remains to be explored in a future study.

Unlike what was mentioned in Section 3.4, $\beta$ may operate differently in models other than language models. Therefore, an experimental analysis of the sensitivity of our DPO method to $\beta$ would be an interesting future study.

PieAPP values directly encode human preferences. Therefore, we concluded that our DPO method contributes to improving the human preference of SR images based on the analysis of their PieAPP values. However, a direct human evaluation of the SR images would better support the conclusion. We plan to conduct human evaluation experiments in the near future. This is the highest-priority future work in order that we can fully validate our claims.