Increasing the Robustness of Image Quality Assessment Models Through Adversarial Training

Abstract

The adversarial robustness of image quality assessment (IQA) models to adversarial attacks is emerging as a critical issue. Adversarial training has been widely used to improve the robustness of neural networks to adversarial attacks, but little in-depth research has examined adversarial training as a way to improve IQA model robustness. This study introduces an enhanced adversarial training approach tailored to IQA models; it adjusts the perceptual quality scores of adversarial images during training to enhance the correlation between an IQA model’s quality and the subjective quality scores. We also propose a new method for comparing IQA model robustness by measuring the Integral Robustness Score; this method evaluates the IQA model resistance to a set of adversarial perturbations with different magnitudes. We used our adversarial training approach to increase the robustness of five IQA models. Additionally, we tested the robustness of adversarially trained IQA models to 16 adversarial attacks and conducted an empirical probabilistic estimation of this feature.

1. Introduction

Image quality assessment (IQA) is essential to developing image- and video-processing algorithms. These algorithms assess the perceptual quality of processed images and reduce reliance on costly subjective tests. IQA models fall into two main categories, depending on the availability of the original image: full reference (FR), which involves estimating the difference between the original and processed images; reduced reference (RR), which involves calculating the difference between features extracted from the original and processed images; and no reference (NR), which only involves analyzing the distorted image using scene statistics gathered from a large dataset. FR-IQA methods are commonly used to evaluate the quality of video compression [1]. RR-IQA methods can be useful in scenarios where a reference image is only partially available, or when the original and processed images are not perfectly aligned [2]. NR-IQA techniques are used to assess the quality of generated images [3].

In recent years, IQA models have shifted toward deep learning, surpassing traditional approaches in correlation with subjective quality. However, this shift has also introduced new vulnerabilities, as all trainable methods, including IQA models, are susceptible to adversarial attacks. These attacks can occur in many real-life image-quality-measurement scenarios [4,5], such as cheating on benchmarks, manipulating web-search results, increasing the energy consumption and latency of streaming video, and storing unclear video-surveillance footage. Researchers often use IQA models to estimate the perceptual quality of image- and video-enhancement algorithms, e.g., underwater image enhancement [6,7,8,9,10], low-light image enhancement [11], and matting of videos [12]. When IQA models are used as the optimization objective to improve the visual quality, unintended attacks may occur, leading to visible image artifacts. Current non-robust learning-based IQA models exhibit low resistance to adversarial attacks, and their use as optimization objectives may lead to visible artifacts.

Recent papers have proposed adversarial attack methods for IQA models [13,14,15,16,17,18], but only a few of them offer defense methods. Adversarial training is essential to improving neural network resistance to these attacks. Surprisingly, however, little research has considered adversarial training for IQA models. Only in one paper suggesting a new adversarial attack did the authors try adversarial training for three models [13]. They concluded that adversarial training is a promising defense for deep IQA models and said further research is necessary to investigate more sophisticated techniques.

The main drawback of applying vanilla adversarial training to increase IQA models’ robustness is a falloff in their correlation with subjective scores, as a previous study has shown [13]. This effect is because these models are more sensitive to image noise than computer-vision models such as object classification and detection. When attacking the IQA model, the adversarial noise affects the subjective quality labels. Even an unsuccessful attack that leaves the IQA score unchanged will likely alter an adversarial image’s perceptual quality. Unlike this case, an unsuccessful attack on an image classifier leaves the adversarial image’s target class unchanged. Also, training an IQA model from scratch only on the attacked data generated at each iteration, as in the standard adversarial training approach, may decrease the correlation.

This paper studies adversarial training’s efficiency in improving IQA model robustness to adversarial attacks. We propose a new strategy to adjust perceptual-quality scores for adversarial images during training. Our main contributions are the following:

• A new adversarial-training approach with a perceptual-quality-labels penalty. It outperforms standard adversarial training by correlating with subjective quality.
• A new method for comparing the robustness of trained IQA models, as well as a new robustness-evaluation metric.
• Comprehensive experiments that tested our method’s efficiency on four IQA models: LinearityIQA, KonCept512, HyperIQA, and WSP-IQA.

This paper is organized as follows. Section 2 discusses related work: the task and trends in image quality assessment, the existing adversarial attacks on IQA models, adversarial training as a defense against adversarial attacks, and the existing IQA models designed to be robust to adversarial attacks. In Section 3, we describe the proposed method: first, we formulate the problem, and then we describe the main ideas of our adjustments to the vanilla adversarial training procedure. Section 4 describes our experiments’ methodology, including a list of datasets and IQA models, evaluation metrics, and main details of experiment implementation. In Section 5, we show the results of the experiments. Section 6 discusses the main findings from the results, and Section 7 summarizes the conclusions of this work. The code is available at https://github.com/wianluna/metrics_at (accessed on 31 October 2024).

2. Related Work

2.1. Image Quality Assessment

IQA methods aim to make quantitative evaluations that closely approximate human perceptual judgments. NR-IQA models function without an original image or any information about that image; they may serve a broader range of applications than other models, such as controlling video-streaming quality and assessing the quality of text-to-image generation. Initially, the IQA model’s design depended on the task and how its developers understood visual-quality perception by the human eye. Early approaches seldom involved machine learning [19]; examples include error visibility (e.g., PSNR), structural similarity (e.g., SSIM [20]), and information theory (e.g., VIF [21]).

Because subjective quality image and video datasets have proliferated, new IQA methods employ deep learning to model empirical statistics of natural images in the NR case or feature learning in the FR case. The latest comparisons reveal that deep-learning-based IQA outperforms traditional alternatives [22,23]. Image-quality measurement is widening thanks to the appearance of new narrow task-specific models. For example, recognition-aware IQA evaluates image quality as a computer vision’s ability to detect or recognize objects, as well as image generation or image super-resolution models. Many IQA models that handle perceptual quality employ CNN-based architectures [24,25,26,27,28], but many also employ ViT [29,30].

2.2. Adversarial Attacks on IQA Models

Adversarial examples are specially crafted inputs that add small imperceptible perturbations to images such that the perturbed input can, with high confidence, mislead an IQA model to yield a false prediction [31]. Depending on the attacker’s knowledge of the victim model, adversarial attacks are either white-box or black-box. In the former case, an attacker has some knowledge about the model, such as its architecture, weights, loss function, or training data. These attacks often use gradients and generate high-confidence adversarial examples that are invisible to the human eye. In the black-box case, an attacker has no direct access to the target model and sometimes only evaluates it on the basis of chosen examples without gradient feedback. Therefore, adversarial examples in this situation are weaker than in the white-box situation. White-box attacks normally have few applications. In the case of IQA models, however, they are practical because IQA model architectures are often available, as with public benchmarks. White-box attacks can obtain gradients to optimize the objective function by taking a single small step (one-step attacks) or multiple small steps (iterative attacks) along the gradient direction in image space.

Several adversarial attacks have targeted IQA models. UAP [15], for example, creates a universal adversarial noise that increases the IQA model score. The generated perturbation is independent of the input image. FACPA [16] is an improved version of this attack; it produces adversarial noise for each input image, increasing success and perceptual quality. The main challenge in attacking IQA models is hiding the perturbations. Korhonen et al. [13] suggested a Sobel filter to hide adversarial distortions in textured regions. An invisible one-iteration (IOI) attack [17] also uses high-frequency filtering to mask perturbations. A subjective comparison proves this attack is invisible, even when applied to videos frame by frame. Zhang et al. [14] used Chebyshev distance, SSIM, LPIPS, DISTS, and other FR metrics to manage visual distortions.

2.3. Adversarial Training

Adversarial training is a popular way to improve the resistance of deep learning models to adversarial attacks. Each training step requires generating a perturbation, thus increasing the computational time. Researchers have used an early version of adversarial training, the Fast Gradient Sign Method (FGSM) [32], to generate perturbed examples. But adversarial training based on noniterative attacks such as FGSM only yields resistance to weak attacks; the model remains vulnerable to stronger iterative attacks. A way to overcome this limitation is adversarial training through multistep Projected Gradient Descent (PGD) [33], which achieves resistance to iterative attacks but can be computationally expensive. Wong et al. designed fast adversarial training [34], which combines FGSM adversarial training and the random perturbation initialization of PGD attacks to accelerate training. This method is almost as effective as PGD-based training but has a much lower computational cost.

In their recent work, Singh et al. [35] generated adversarial examples using two-step Auto PGD (APGD-2) [36]. This adaptive attack realizes a higher loss than PGD in the same iteration budget, reducing both the number of iterations for adversarial training and the computational cost. For NR-IQA models, Korhonen et al. [13] applied adversarial training to make KonCept512, PHIQNet, and TRIQ more robust. These models exhibited a lower correlation with subjective quality than the original ones did.

2.4. Robust IQA Models

Few papers thus far have focused on making particular IQA models more robust. Netflix introduced VMAF NEG (No Enhancement Gain) [37], which is more resistant to contrast enhancement than the original VMAF. VMAF utilizes support-vector regression based on fundamental low- and mid-level frame attributes. The NEG version entails regression over clipped values of base features. In particular, the authors introduced VIF and DLM enhancement-gain limits. These parameters allow limits on the maximum VIF and DLM scores, which are features of the VMAF support-vector regressor. Although this modification increases VMAF’s resistance to attacks [38], limiting base features in a fixed range only protects the model from extreme attacks; for example, increasing scores by 100–200%. An attacker can still increase the VMAF NEG score in its clipped range, however.

Kettunen et al. [39] proposed E-LPIPS, a modification of LPIPS [40]. It applies a series of random transformations to the image before passing the altered image through the neural network. It then calculates the average output over all inputs. This ensemble approach can make manipulating the results more difficult, but it also requires many model runs to generate a single prediction, limiting its real-world applicability. R-LPIPS [41] is another LPIPS variation. The authors left the network architecture unchanged and used adversarial training, during which time they applied adversarial perturbations to just one image in a triplet. They employed cross-entropy loss to generate the adversarial example.

Currently, there are very few studies focusing on improving the stability of NR-IQA models. Recently, Liu et al. [42] proposed the Norm regularization Training (NT) method using $l_1$-regularization of the gradient norm to enhance the robustness of NR-IQA models against adversarial attacks.

3. Proposed Method

3.1. Problem Setting

Adversarial attacks on IQA models. Our investigation assumes an attacker is trying to increase the IQA model scores by adding small perturbations to images. Recent papers have examined these attacks, which may occur more frequently in the real world than other attacks. Attempts to decrease an image’s estimated quality are all the same up to the attack direction sign. Therefore, this approach avoids violating our study’s generality.

For a given NR-IQA model $f_{\theta }:X\to \mathbb{R}$ parameterized by the vector of weights $\theta$, where $X\in {[0,255]}^{3\times H\times W}$ is a distribution of images, we can mathematically describe an adversarial attack on an input image x with a perturbation $\delta$ bounded by $\epsilon$ with a norm of p as follows:

$$\underset{{\parallel \delta \parallel }_p\le \epsilon }{\max }{f}_{\theta }(x+\delta ).$$

(1)

An attack on FR-IQA models can be reformulated as an attack on NR-IQA by manipulating just one of the reference and distorted images. For a given FR-IQA model $f_{\theta }$ parameterized by $\theta$ which takes a reference image $\widehat{x}\subseteq {[0,255]}^{3\times H\times W}$ and distorted image $x\subseteq {[0,255]}^{3\times H\times W}$ as an input, an adversarial attack with a perturbation $\delta$ bounded by $\epsilon$ of norm p is formulated as follows:

$$\underset{{\parallel \delta \parallel }_p\le \epsilon }{\max }{f}_{\theta }(\widehat{x},x+\delta ).$$

(2)

Adversarial training for IQA models. Given a NR-IQA or FR-IQA model $f_{\theta }$ parameterized by $\theta$; a distribution of the training data D, where each training sample contains an image $x\subseteq {[0,255]}^{3\times H\times W}$ and associated quality label $y\in \mathbb{R}$; and a loss function $\mathcal{L}$ of the model $f_{\theta }$, vanilla adversarial training is a min-max problem where the inner maximization problem aims to discover strong adversarial examples $x+\delta$ bounded by the allowable perturbation magnitude $\epsilon$ of a norm p. The outer minimization problem fine-tunes the model parameters as follows:

$$\underset{\theta }{\min }{\mathbb{E}}_{(x,y)\sim D}\left[\underset{{\parallel \delta \parallel }_p\le \epsilon }{\max }\mathcal{L}(f_{\theta }(x+\delta ),y)\right]$$

(3)

As we mentioned in our related work discussion (Section 2), the perceptual quality of perturbed images may change, leading to different quality labels. We therefore add another term to the objective function, as follows:

$$\underset{\theta }{\min }{\mathbb{E}}_{(x,y)\sim D}\left[\underset{{\parallel \delta \parallel }_p\le \epsilon }{\max }\mathcal{L}(f_{\theta }\left(x\right),y)+\mathcal{L}(f_{\theta }(x+\delta ),y^{\prime })\right],$$

(4)

where y is the original subjective quality score and $y^{\prime }$ is the adjusted subjective quality score.

3.2. Method

We propose the following improvements to the vanilla adversarial training procedure to adapt it for IQA models.

Pre-training and fine-tuning. Many IQA models extract deep features from convolutional neural networks pre-trained on the ImageNet dataset to perform image classification [25,26,27,28,40]. Such transfer learning enables the receipt of more stable features than training on smaller datasets labeled for IQA. Because adversarial training is a nontrivial optimization problem, we propose initializing it from pre-trained standard IQA model weights and then fine-tuning them on clean and perturbed images. Doing so allows us to make the model resistant to adversarial perturbations while keeping a high correlation with subjective quality—a feature we demonstrated in our experiments.

Subjective scores correction. Obtaining a subjective quality score for perturbed images during each training epoch would be extremely expensive. Therefore, we propose three strategies to correct the subjective quality scores of clean images after adding adversarial perturbations.

Minimal MOS. The simplest approach to adjusting scores for adversarial images is to set the minimum value for the entire dataset D as follows:

$$y^{\prime }=\underset{y\sim D}{\min }y$$

(5)

Percentage-based penalty. Another strategy involves penalizing the original subjective evaluation with a constant. During training, adversarial images receive an assigned target value as follows:

$$y^{\prime }=y-p\times \mathrm{diam}\left(D\right),$$

(6)

where p is 5/100 or 10/100 in our experiments, and $\mathrm{diam}\left(D\right)=\underset{y,z\in D}{\sup }\left\{\right|y-z\left|\right\}$. This approach’s complexity lies in the emergence of a new hyperparameter, which may be inconvenient in practice.

FR-based penalty. We extended our approach by penalizing subjective quality labels with FR metrics such as SSIM [20], MS-SSIM [43], and PSNR. Also, we incorporated penalties based on the trainable metric LPIPS [40], which emphasizes perceptual similarity. The formula for the target-value assignment then becomes the following:

$$y^{\prime }=y-\mathit{norm}(M(x,x+\delta ),M)\ast y,$$

(7)

where $M\in$ {SSIM, MS-SSIM, PSNR, LPIPS}. Here, norm denotes a metric-dependent mapping of the original values that transforms the range of M to the closed interval $[0,1]$:

$$\mathit{norm}(x,M)=\left\{\begin{array}{c}1-x,\mathrm{if}\mathrm{M}\mathrm{is}\mathrm{SSIM}\mathrm{or}\mathrm{MS}-\mathrm{SSIM},\hfill \\ {\displaystyle \frac{75-x}{45}},\mathrm{if}\mathrm{M}\mathrm{is}\mathrm{PSNR},\hfill \\ x,\mathrm{if}\mathrm{M}\mathrm{is}\mathrm{LPIPS}.\hfill \end{array}\right.$$

(8)

Attack methods and magnitudes. The choice of attack for generating adversarial examples during training is crucial to adversarial learning’s effectiveness. Previous research shows that models trained using PGD are resistant to weaker attacks. However, choosing a stronger attack for training may cause the model’s performance to decrease more. Therefore, we applied three attacks previously used for adversarial training of image classifiers [32,34,35]: a one-step FGSM attack [32]; a one-step PGD (PDG-1) attack, which is an FGSM with an initial random perturbation [34]; and an adaptive Auto PGD (APGD-2) attack [36], for which we used two iterations. For FR-IQA models, we propose attacking both distorted images simultaneously for greater strength compared with only one image, as [41] suggests. Additionally, we examined the impact on model robustness of the adversarial-perturbation magnitude during training. Previous works on adversarial training commonly consider a single value for the perturbation magnitude $\epsilon$. Our study explores different $\epsilon$ values from the set {2, 4, 8, 10}/255.

The complete proposed training procedure is summarized in Algorithm 1 and Figure 1. Our ablation study (Section 5.1) shows the impact of each component of our adversarial training.

Algorithm 1 Proposed Adversarial Training for T Epochs, Given Attack Magnitude $\epsilon$, Step Size $\alpha$, And Dataset D for Pre-Trained NR-IQA Model $f_{\theta }$, Loss Function $\mathcal{L}$, and FR Metric M

for $t=1\dots T$ do  for $(x_i,y_i)\in D$ do   // PGD-1 adversarial attack:   $\delta =Uniform(-\epsilon ,\epsilon )$   $\delta =\delta +\alpha ·sign\left({\nabla }_{\delta }\mathcal{L}(f_{\theta }(x_i+\delta ),y_i)\right)$   $\delta =\max \left(\min \right(\delta ,\epsilon ),-\epsilon )$   // Subjective score correction:   $y_i^{\prime }=y_i-\mathit{norm}(M(x_i,x_i+\delta ),M)\ast y_i$ // norm is calculated according to Equation (7)   $x_i=concatenate(x_i,x_i^{\prime })$   $y_i=concatenate(y_i,y_i^{\prime })$   $\theta =\theta -\beta ·{\nabla }_{\theta }l(f_{\theta }(x_i+\delta ),y_i)$// Update model with some optimizer and step size β  end for end for

Figure 1. Procedure for proposed IQA models adversarial training.

4. Materials and Methods

4.1. Experimental Setup

NR-IQA datasets and models. We used KonIQ-10k [25], a popular IQA dataset that commonly serves for training and evaluating IQA models, to perform adversarial training of the NR-IQA model. This dataset comprises 10,073 images paired with subjective quality values based on Mean Opinion Scores (MOSs). To add an extra evaluation layer, we also used the NIPS2017 [44] dataset, which serves in attack development and testing and contains 1000 images without MOSs. The decision to incorporate the NIPS2017 dataset in our experiments was to validate model robustness across varied datasets, extending beyond the scope of their original training domain.

We evaluated our method using four NR-IQA models originally developed using KonIQ-10k and showed a high correlation with subjective quality in public benchmarks. KonCept512 [25] was the first IQA model trained on the KonIQ-10k dataset. WSP-IQA [27] employs weighted spatial pooling (WSP) instead of global average pooling (GAP) to aggregate spatial information from different-size weight maps. HyperIQA [26] uses a hyper-convolutional network to predict the weights of fully connected layers. LinearityIQA [28] introduces the norm-in-norm loss function, which enables faster convergence than MSE or MAE when serving with the ResNet architecture.

FR-IQA datasets and models. For full-reference assessment, we enhanced LPIPS [40]. Among learning-based FR-IQA models that have proven vulnerable to adversarial attacks, it is the most widely used option with an open training dataset [39,45]. Training of the original model version used the BAPPS [40] dataset, so our approach involves modifying the training process using this dataset. BAPPS consists of 36,344 image triplets: one image is the reference and the other two are distorted. Each triplet has two alternative forced-choice (2AFC) scores, indicating which of the two distorted images is more similar to the reference image. Additionally, we used the KADID-10k dataset [46], which contains 10,125 distorted images derived from 81 reference images. Each distorted image is associated with a subjective quality score based on MOS.

4.2. Evaluation Metrics

Performance evaluation. We used the Spearman rank order correlation coefficient (SROCC) to measure the correlation of NR-IQA model scores with subjective ratings. For FR-IQA models, we measured two-alternative forced choice (2AFC), which Zhang et al. [40] used.

Robustness evaluation. We applied FGSM and PGD attacks with 10 iterations to test adversarially trained models. Our evaluation of IQA model robustness employed two metrics.

Robustness score (R) [14] is the average ratio of the maximum allowable quality-prediction change to the actual change over all adversarial images in a logarithmic scale:

$$R={\displaystyle \frac{1}{N}}\sum _{i=1}^N\log ({\displaystyle \frac{\max \{{\beta }_1-f\left(x_i\right),f\left(x_i\right)-{\beta }_2\}}{|f\left(x_i\right)-f\left(x_i^{\prime }\right)|}}),$$

(9)

where f is an IQA model, $x_i$ and $x_i^{\prime }$ are, respectively, the clean and perturbed versions of $i^{th}$ image from the dataset; ${\beta }_1$ is the maximum MOS value; and ${\beta }_2$ is the minimum MOS value.

Integral Robustness Score (IR-score). We introduce a new method to measure the resistance of adversarially trained models to perturbations of various magnitudes. Unlike R, our integral robustness score (IR-score) accounts for attack success versus perturbation-budget dependency, as Carlini et al. recommended [47].

We generated adversarial examples with different perturbation sizes $\epsilon$ from {2, 4, 8, 10}/255 for each image in the test set and each IQA model. We then obtained the IQA model scores for images before and after the attack and performed min-max normalization. Our next step was to calculate the minimum and maximum scores for each IQA model using the training dataset. Afterward, we transformed the IQA model scores into the unified domain using neural optimal transport [48]. This step is necessary because the scores of different IQA models are nonuniformly distributed. We therefore trained a small neural network with four linear layers and one-dimensional input, as the authors in [4] suggested. Our experiments used LinearityIQA as a unified domain. Next was computing the absolute gain: the score difference for each image pair before and after the attack. We performed the abovementioned procedure for the original and adversarially trained models. The final step was to calculate the IR-score as the difference between two integrals (for the original and adversarially trained models) of absolute gain-$\epsilon$ curves averaged over all test-dataset images. Formally, for a given IQA model f parameterized by $\theta$ and adversarially trained model $f_{AT}$ parameterized by ${\theta }_{AT}$, the IR-score for a set of images ${\left\{x_i\right\}}_{i=1}^N$ is the following:

$$\begin{array}{c}\hfill \mathrm{IR}-\mathrm{score}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\int }_{\epsilon }\left(\widehat{f}\left(x_i\right)-\widehat{f}(x_i+\delta (\theta ,\epsilon ))-{\widehat{f}}_{AT}\left(x_i\right)+{\widehat{f}}_{AT}(x_i+\delta ({\theta }_{AT},\epsilon ))\right)d\epsilon ,\end{array}$$

(10)

where $\delta (\theta ,\epsilon )$ is the attack perturbation with a magnitude of $\epsilon$ applied to the model parameterized by $\theta$, and $\widehat{f}\left(x_i\right)$ is a metric score mapped by neural optimal transport (NOT) [48], as follows:

$$\begin{array}{c}\hfill \widehat{f}\left(x_i\right)\leftarrow \mathrm{NOT}\left(\mathrm{diam}\left(f_{\theta }\left(x_i\right)\right)\right).\end{array}$$

(11)

4.3. Implementation Details

The pretraining of each IQA model employed the settings described in the source articles. Subsequently, we conducted 30 fine-tuning epochs for NR-IQA models, applying our proposed method with the hyperparameters listed in Table A2. LPIPS involved five epochs using the Adam optimizer with a learning rate of 10⁻⁴ as well as five epochs with a linearly decreasing learning rate. To train their LPIPS model, the authors of [49] chose the features of various classification models trained on ImageNet. Following the R-LPIPS researchers [41], we used the AlexNet features.

Selecting the best model first required analysis of the SROCC for WSP-IQA, HyperIQA, and LinearityIQA and of the Pearson linear correlation coefficient (PLCC) for KonCept512, in accordance with the original paper’s recommendation [25]. We measured correlation coefficients on the validation subset after each training epoch and saved the best-performing model. During adversarial retraining, we saved the best model after three epochs, as we expected to obtain a more robust model exhibiting a natural decrease in correlation with subjective scores over more epochs. All our experiments with NR-IQA models were performed using eight Nvidia Tesla A100 80-Gb GPUs, while experiments with FR-IQA models used four NVIDIA RTX A6000 GPUs.

5. Results

5.1. Ablation Study

Role of pre-training and training data in adversarial robustness. Table 1 shows pre-training’s impact on the performance and robustness of the adversarially trained LinearityIQA as measured by the SROCC and IR-score. Similarly, Table 2 shows the performance and robustness of the adversarially trained FR-IQA model LPIPS. AT means vanilla adversarial training with an FGSM attack, +clean means concatenated batches of clean and adversarial data, +pretr. means pre-training. For training, we set the allowable perturbation magnitude $\epsilon$ to 4/255.

Table 1. Influence of pre-training and fine-tuning on NR-IQA models. Bold denotes the best value and underlined denotes the second best.

Training		IR-Score ↑
Strategy	SROCC	FGSM		PGD-10
		KonIQ-10K	NIPS2017	KonIQ-10K	NIPS2017
AT	0.784	1.001	1.011	0.424	0.323
+ pretr.	0.717	1.554	1.685	0.565	0.603
+ clean	0.924	2.092	2.218	0.516	0.527
+ clean, pretr.	0.925	1.984	2.248	0.454	0.451

Table 2. Influence of pre-training and fine-tuning on FR-IQA model LPIPS for KADID-10k dataset. Bold denotes the best value, and underlined denotes the second best.

Training Strategy	SROCC	IR-Score ↑
		FGSM	PGD-10
AT	0.668	0.665	0.597
+ pretr.	0.817	0.801	0.732
+ clean	0.701	0.696	0.621
+ clean, pretr.	0.832	0.872	0.765

Although vanilla adversarial training for classification typically involves training the model solely on perturbed data, our experiments showed that this approach is ineffective for IQA models. Adversarial training of these models without pre-training causes the model’s correlation with subjective quality to decline sharply.

Subjective score correction. Table 3 shows that assigning minimal MOS labels and PSNR-based score correction produces a greater decrease in correlation with subjective quality on clean images, a phenomenon we want to avoid. The best correlations are for fixed penalties of 5% and 10%, followed by FR-based penalties on LPIPS and SSIM. A fixed rate, however, will require manual selection of a penalty percentage, which may depend on image content, its corruption level, and other properties. In total, LPIPS-based penalty shows better IR-score than fixed rate, keeping high correlation with original quality labels.

Table 3. SROCC ↑ and IR-score of LinearityIQA model on KonIQ-10k dataset for different MOS-penalty strategies and attack types in our adversarial training. Bold denotes the best value, and underlined denotes the second best. SROCC of the original LinearityIQA is 0.931.

		SROCC (Train $\mathit{\epsilon }=2/255$)			IR-Score ↑ (Trained with PGD-1)
Trained with		FGSM	PGD-1	APGD-2	$\mathbf{\epsilon }=\mathbf{2}/\mathbf{255}$	$\mathbf{\epsilon }=\mathbf{4}/\mathbf{255}$	$\mathbf{\epsilon }=\mathbf{8}/\mathbf{255}$
Label strategy	–	0.920	0.917	0.858	0.848	1.231	1.013
	Min	0.908	0.911	0.870	6.252	5.730	4.253
	−5%	0.920	0.922	0.882	1.331	1.301	0.773
	−10%	0.922	0.923	0.901	1.819	1.637	1.264
	PSNR	0.906	0.907	0.922	5.938	5.872	4.504
	SSIM	0.917	0.922	0.886	1.350	1.905	2.458
	MS-SSIM	0.917	0.918	0.878	1.002	1.384	1.165
	LPIPS	0.920	0.921	0.906	1.857	2.608	2.679
	Avg	0.916	0.918	0.888

Impact of attack type during training. Table 3 shows the correlation between model predictions and subjective-quality scores for the original NR-IQA model LinearityIQA and its adversarially trained versions. The results showed that NR-IQA models trained with a PGD-1 attack correlated better with subjective quality. PGD-1 is a stronger FGSM version requiring one iteration of forward and backward passes. Interestingly, the use of a stronger attack during image-classifier training decreases the model’s accuracy on clean images [33,34]. Table 4 presents the results for different versions of the FR-IQA model, LPIPS. More sophisticated attacks increased robustness and decreased SROCC.

Table 4. SROCC and IR-score of LPIPS on KADID-10k for different attack types in our adversarial training. Bold denotes the best value, and underlined denotes the second best. SROCC of the original LPIPS is 0.893.

FR-IQA Model	Train $\mathit{\epsilon }$	SROCC	IR-Score ↑
			FGSM	PGD-10
R-LPIPS [41]		0.858	0.791	0.777
AT-LPIPS (FGSM)	2/255	0.855	0.804	0.782
	4/255	0.843	0.811	0.791
	8/255	0.845	0.807	0.786
	10/255	0.832	0.792	0.775
AT-LPIPS (PGD-1)	2/255	0.848	0.817	0.801
	4/255	0.837	0.821	0.808
	8/255	0.856	0.814	0.799
	10/255	0.849	0.802	0.783
AT-LPIPS (APGD-2)	2/255	0.841	0.802	0.788
	4/255	0.834	0.805	0.791
	8/255	0.830	0.799	0.783
	10/255	0.835	0.788	0.775

Impact of perturbation magnitude during training. Figure 2 illustrates the impact of the allowable perturbation values on the NR-IQA model’s correlation and robustness. The upper row of plots illustrates the resistance of each model to a simple FGSM attack. The lower row illustrates the same models’ resistance to a stronger PGD attack with 10 iterations. Models trained using a lower $\epsilon$ value correlate less well with MOS values, but during evaluation, they become more resistant to a stronger attack. LPIPS exhibits a similar effect, as Table 4: smaller magnitudes are used while adversarial training yields better resistance to PGD.

Figure 2. Impact of subjective score correction, perturbation magnitude, and attack type during training for the LinearityIQA NR-IQA model and KonIQ-10k dataset.

5.2. Overall Results

Our final adversarial training method employed the best strategies according to our experiment results noted in the ablation study (Section 5.1): we fine-tuned pre-trained IQA models on clean and attacked images using the PGD-1 attack with $\epsilon =2/255$; for NR-IQA models, we computed a penalty for subjective scores using the SSIM metric. Table 5 compares the original NR-IQA models with the robust versions from vanilla adversarial training, Norm regularization Training, and our proposed method. Our adversarially trained models demonstrate better resistance than the base and NT versions. Moreover, the SROCC decrease in models trained with our process is no more than 1.8%—a substantial improvement over vanilla adversarial training.

Table 5. SROCC↑ on clean data, $R\uparrow$ and IR-score ↑ of different NR-IQA models trained using different techniques. The SROCC value is for KonIQ-10k without attacks. The training time is calculated using an Nvidia Tesla A100 80-Gb GPU. Bold denotes the best value for each model, and underlined denotes the second best.

		SROCC	$\mathit{R}\uparrow$ [14] ($\mathit{\epsilon }=2/255$)				IR-Score ↑				Train.
NR-IQA Model		↑	FGSM		PGD-10		FGSM		PGD-10		Time
			KonIQ	NIPS	KonIQ	NIPS	KonIQ	NIPS	KonIQ	NIPS	(min)
LinearityIQA [28]	base	0.931	0.699	0.780	0.182	0.266	-	-	-	-	73
	AT	0.824	1.507	1.628	1.485	1.573	0.162	0.069	0.901	0.918	237
	NT [42]	0.930	0.797	0.844	0.304	0.382	0.029	−0.356	0.239	0.227	79
	AT (ours)	0.922	1.555	1.567	0.731	0.921	1.020	1.366	0.657	0.726	275
KonCept512 [25]	base	0.925	0.706	0.620	0.353	0.033	-	-	-	-	93
	AT	0.868	1.246	1.383	0.873	1.135	0.547	0.850	0.657	0.957	222
	NT [42]	0.822	1.459	1.246	1.096	1.008	0.687	0.784	0.774	0.946	101
	AT (ours)	0.913	1.265	1.416	0.632	0.726	1.044	0.900	0.538	0.831	255
HyperIQA [26]	base	0.894	0.531	0.505	−0.168	−0.090	-	-	-	-	133
	AT	0.778	1.624	1.779	1.376	1.634	0.547	0.682	0.945	0.961	141
	NT [42]	0.846	1.093	1.013	0.742	0.733	0.243	0.043	0.826	0.775	219
	AT (ours)	0.891	0.848	1.625	0.622	1.010	1.385	1.413	0.762	0.911	163
WSP-IQA [27]	base	0.916	0.618	0.711	0.124	0.285	-	-	-	-	68
	AT	0.882	0.948	1.140	0.405	0.609	0.534	0.662	0.459	0.506	76
	NT [42]	0.915	0.644	0.742	0.109	0.280	0.279	0.243	−0.094	−0.109	92
	AT (ours)	0.899	1.135	1.224	0.319	0.461	1.352	1.428	0.355	0.317	117

Table 6 shows the adversarial robustness of the FR-IQA LPIPS model trained with different approaches. Our approach to attacking two images simultaneously demonstrates improved robustness while insignificantly reducing the SROCC compared to vanilla adversarial training used in R-LPIPS.

Table 6. SROCC on clean data, 2AFC ↑, $R\uparrow$ and IR-score↑ of different LPIPS versions. The training time is calculated using an NVIDIA RTX A6000 GPU. Bold denotes the best value, and underlined denotes the second best.

LPIPS Version	SROCC	2AFC ↑			$\mathit{R}\uparrow$		IR-Score ↑		Train. Time
		No Attack	FGSM	PGD-10	FGSM	PGD-10	FGSM	PGD-10	(min)
base LPIPS	0.893	0.742	0.260	0.102	0.525	0.311	-	-	46
R-LPIPS [41]	0.858	0.741	0.487	0.306	0.701	0.658	0.791	0.777	663
AT-LPIPS (ours)	0.852	0.742	0.495	0.440	0.753	0.737	0.817	0.801	101

For each final model, we provide information on training time. Although our method for NR-IQA requires more GPU hours than previous methods, this does not affect inference time.

6. Discussion

Applicability of proposed method to other adversarial attacks. We tested the robustness of the adversarially trained NR-IQA Linearity model to 16 other attacks. Appendix A provides the details about attacks and their hyperparameters. Figure 3 shows the R values for each pair of attack and version of IQA model. The proposed approach improves NR-IQA model resistance to almost every attack, except AdvCF [50]. This is because the training process used the noise attack, and models were therefore intended to be resistant to perturbations rather than color distortion, which is how AdvCF works. Also, the SSIM-based label penalty chosen in the ablation study as the best strategy showed the best results in this experiment since it provided the best robustness for 13 of 16 unforeseen attacks.

Figure 3. Adversarial robustness $R\uparrow$ for LinearityIQA. Attack methods appear on the x-axis and LinearityIQA versions on the y-axis. Columns and rows are sorted by mean value.

Performance on clean images. Since vanilla adversarial training decreases model performance on clean data, we tested the performance of our adversarial training method when images did not contain adversarial noise. This property is more pronounced for IQA models because we added adversarial noise to images during training. The proposed approach helped reduce this effect of NR-IQA models: it decreased the correlation by less than 1.8% for four NR-IQA models. For LPIPS, our approach yielded 4.6% lower SROCC compared with the original LPIPS and less than 1% compared with R-LPIPS.

Empirical probabilistic estimation. When comparing the robustness of adversarially trained models, we launched each attack once and varied only the perturbation norm. We conducted an empirical study involving 100 attack launches for each parameter set to show that our adversarially trained models provide statistically significant improvement. Then, we used the Wilcoxon signed-rank test with the alternative hypothesis that the attack gains for the original and adversarially trained models differ. The test yielded a p-value of 0, confirming that adversarial training enhances model resistance to adversarial attacks. Details of the experiment appear in the Appendix B.

7. Conclusions

This paper presents a novel adversarial training method for IQA models. Unlike the vanilla approaches created for computer-vision models, our method is for IQA specifically, a feature that allowed us to create an IQA model that is robust to adversarial attacks without significantly reducing correlation with human image-quality perception. We conducted a comprehensive ablation study to determine the best adversarial training components for IQA. We also introduced a new evaluation strategy for our method and compared the robustness of IQA models. Our results showed that the best approach is to fine-tune IQA models on clean and adversarial images with a low perturbation level ($\epsilon =2/255$ in our experiments). In the case of NR-IQA models, the adjusted perceptual-quality scores for perturbed images generated during adversarial training maintain a high correlation with subjective quality. For FR-IQA LPIPS model, our approach of adding perturbations to two images during training improves robustness relative to other methods.

Future work. This study primarily focused on applying attacks during training with a restriction on the $l_{\infty }$ norm. An interesting variation would be to explore perceptually constrained attacks for quality metrics.