DualNetIQ: Texture-Insensitive Image Quality Assessment with Dual Multi-Scale Feature Maps

Abstract

The precise assessment of image quality that matches human perception is still a major challenge in the field of digital imaging. Digital images play a crucial role in many technological and media applications. The existing deep convolutional neural network (CNN)-based image quality assessment (IQA) methods have advanced considerably, but there remains a critical need to improve the performance of existing methods while maintaining explicit tolerance to visual texture resampling and texture similarity. This paper introduces DualNetIQ, a novel full-reference IQA method that leverages the strengths of deep learning architectures to exhibit robustness against resampling effects on visual textures. DualNetIQ includes two main stages: feature extraction from the reference and distorted images, and similarity measurement based on combining global texture and structure similarity metrics. In particular, DualNetIQ takes features from input images using a group of hybrid pre-trained multi-scale feature maps carefully chosen from VGG19 and SqueezeNet pre-trained CNN models to find differences in texture and structure between the reference image and the distorted image. The Grey Wolf Optimizer (GWO) calculates the weighted combination of global texture and structure similarity metrics to assess the similarity between reference and distorted images. The unique advantage of the proposed method is that it does not require training or fine-tuning the CNN deep learning model. Comprehensive experiments and comparisons on five databases, including various distortion types, demonstrate the superiority of the proposed method over state-of-the-art models, particularly in image quality prediction and texture similarity tasks.

1. Introduction

Visual perception plays a vital role in our daily lives, and over the past century, there has been a significant technological shift in the field of imaging. Advances in mobile multimedia technology now allow people to access numerous high-resolution images on their smartphones or other devices at any time. However, various processes in an image system such as capture, reproduction, compression, storage, transmission, and restoration can introduce noise, distortions, or artifacts. As a result, monitoring and assessing image quality is essential. The demand for accurate image quality assessment (IQA) has grown with the increasing reliance on digital imaging in fields like communications, surveillance, medical imaging, and visual information. Thus, precise IQA methods are fundamental for many image-based applications. The objective of IQA is to develop algorithms that automatically assess digital image quality in a manner consistent with human perception [1,2].

Notably, existing IQA methods are generally classified into three categories: full-reference (FR), reduced-reference (RR), and no-reference (NR) methods. The FR-IQA techniques, such as mean square error (MSE) and the structural similarity index (SSIM) [3], evaluate image quality by directly comparing the distorted image with a pristine reference image. However, these methods often fail to align with human visual perception, especially when comparing images with similar textures that may differ in the exact arrangement of features but appear nearly identical to the human eye (see Figure 1). This discrepancy arises from their dependence on pixel-wise accuracy, making them overly sensitive to point-by-point differences between images that share the same overall texture.

In response to the limitations of FR-IQA methods, and with the rise of deep learning across various fields, deep learning has significantly impacted IQA [7,8,9]. Deep learning-based models have shifted the paradigm by leveraging large-scale datasets to automatically learn complex distortion patterns, eliminating the need for manually crafted features that traditional methods rely on. Moreover, the use of convolutional neural networks (CNNs) for IQA tasks has been further enhanced by transfer learning, where pre-trained networks initially trained on large-scale image recognition tasks exhibit strong capabilities in capturing intricate patterns and textures, closely resembling the processing of the human visual system [9,10,11,12].

Despite the advancements in the realm of IQA, there remains a need for further improvements in IQA models, particularly regarding texture similarity and sensitivity to texture resampling effects. The key challenge lies in developing an IQA model that can accurately assess image quality while withstanding the impact of texture resampling commonly encountered during image processing tasks. While models such as deep image structure and texture similarity (DISTS) [11] and deep distance correlation (DeepDC) [12] have explored the concept of IQA models that are less influenced by texture resampling, they should also strive for alignment with human visual perception. It is important to note that both DISTS and DeepDC utilize the VGG pre-trained model [13] for feature extraction. However, relying on a single deep feature extractor like VGG may not fully capture the complete range of image appearance, structure, and texture, especially under various distortion types. While DISTS and DeepDC have achieved satisfactory IQA results, there remains a need for further improvement in the performance of IQA methods, particularly in traditional image quality assessment.

This paper introduces a novel full-reference IQA method, termed DualNetIQ, designed to be resilient to resampling effects on visual textures by leveraging the capabilities of deep learning architectures. The proposed DualNetIQ method has two steps: first, it pulls out features from both the original image and the distorted image; second, it measures similarity by combining global texture and structure similarity metrics. In particular, this method takes features from input images by using a mix of pre-trained feature maps carefully chosen from the VGG19 and SqueezeNet CNN models to find differences in texture and structure. Each deep CNN network uses hybrid feature maps in the proposed IQA method to pull out specific patterns from images. This makes sure that a full description of the textures and structures of both the original image and the distorted image under different types of distortion is given. The proposed scheme then evaluates the similarity between the reference and distorted images by calculating a weighted combination of global texture and structure similarity metrics, which we then optimize using the Grey Wolf Optimizer (GWO). This research highlights the following key contributions:

• Proposing a novel IQA method called DualNetIQ for texture-insensitive full reference IQA. Unlike existing IQA methods that employ a single deep feature extractor, the proposed method utilizes hybrid multi-scale feature maps selected from robust pre-trained CNNs models (VGG19 and SqueezeNet) to assess the quality of textured images effectively and withstand the effects of texture resampling under different distortion types. DualNetIQ makes a complete description of the textures and structures of the reference and the distorted images under different types of distortion. It addresses the limitations of similar existing models, particularly in handling various types of distortions.
• Proposing an efficient method for assessing the similarity between the reference and distorted images by optimally combining structure and texture similarity metrics utilizing the GWO algorithm, allowing the model to align closely with human perceptual judgement.
• Providing an extensive ablation study and comparative analysis against existing methods on five IQA datasets (LIVE [14], CSIQ [15], TID2013 [16], KADID-10k [17], and PIPAL [18]), one texture similarity dataset, SynTEX [4], and one perceptual similarity dataset, BAPPS [10]. These datasets encompass a wide range of distortions, including traditional synthetic artifacts and those created by modern image processing algorithms. The employment of such datasets provides a rigorous benchmark that confirms the adaptability and robustness of the DualNetIQ method across various evaluation scenarios.

The remainder of this study includes four sections: Section 2 presents a brief review of existing research on IQA methods. Section 3 introduces the proposed DualNetIQ model in detail. Section 4 provides experimental comparisons and discussion to evaluate the model’s performance. Lastly, Section 5 concludes the study.

2. Related Work

The field of IQA has seen significant advancements in recent years. It began with traditional IQA methods, which were primarily based on modeling the human visual system. In the last decade, IQA techniques have evolved to include deep learning approaches, whether through training neural networks from scratch or leveraging pre-trained models. Additionally, hybrid models that combine knowledge-driven and data-driven methods have emerged. Below, we provide a brief overview of the most prominent traditional approaches and deep learning-based IQA methods.

2.1. Traditional Full-Reference IQA Methods

For over five decades, knowledge-driven techniques have been the primary approach in FR-IQA models. Early FR-IQA methods, like peak signal-to-noise ratio (PSNR) and mean squared error (MSE), compared things in a way that was certain by computing differences between pixels. However, these methods demonstrated poor correlation with the human visual system (HVS) [19]. Wang et al. introduced the SSIM metric [3], recognizing that earlier models failed to capture perceptual image quality effectively. This marked a paradigm shift, incorporating structural information, contrast, and luminance into the evaluation process. Despite these advances, SSIM and its variants, such as multi-scale SSIM (MS-SSIM) [20], information weighting SSIM (IW-SSIM) [21], and complex wavelet SSIM (CW-SSIM) [22], faced challenges in handling complex distortions and accurately representing textures.

Additional techniques, such as visual saliency similarity (VSI) [23], Visual Information Fidelity (VIF) [24], and feature similarity (FSIM) [25], have been proposed to improve performance. While these models have shown significant improvements, they still face challenges in accurately handling visual textures and certain distortion types.

2.2. Deep Learning-Based IQA Methods

Deep learning-based models have significantly outperformed traditional metrics in capturing high-level perceptual cues. In recent years, numerous deep learning-based IQA methods have been proposed. For example, Liang et al. [26] introduced a dual-path CNN network that processes both distorted and reference image patches to predict corresponding quality scores. Kim and Lee [7] developed an IQA method called DeepQA in which a CNN learns the behavior of the HVS from the underlying data distribution of IQA datasets.

The successful application of pre-trained networks like AlexNet [27], SqueezeNet [28], and VGG [13] across various vision tasks has inspired the use of deep features in FR-IQA, significantly enhancing the accuracy and reliability of these estimations [8]. Gao et al. [9] introduced the DeepSim IQA model, which utilizes a deep CNN pre-trained on the ImageNet database [29]. In their approach, pairs of distorted and reference images are fed into the CNN, and each output layer serves as a feature map. Global image quality scores are then computed by pooling local similarities between the feature maps from the distorted and reference images. The authors of LPIPS [10] used pre-trained CNN models to extract feature stacks from various network layers and normalize them in the channel dimension to compute the distance between reference and distorted image patches.

The development of transformers has greatly energized IQA and ushered in a new era in artificial intelligence. Junyong You and Korhonen [30] employed transformers to effectively manage images of diverse resolutions and improve their focus on global features. Building on this, Ke et al. [31] presented a novel technique that combines scale embedding and hash-based 2D spatial embedding to make it easier to handle images with different sizes and aspect ratios. Many vision transformer-based methods have since emerged [32,33,34,35] that have shown excellent performance in image quality assessment.

While all these models have achieved acceptable results, they did not address the issue of sensitivity to texture resampling.

Recent studies have aimed to address the issue of texture resampling sensitivity. One of the first deep learning-based models designed to explicitly handle this problem is the DISTS model [11]. DISTS employs a pre-trained VGG model to extract features from both reference and corrupt images. It then constructs a similar measure by combining texture and structural terms across all feature maps. The authors of [36] proposed an improved version of DISTS called A-DISTS by introducing adaptive strategies to calculate the structure and texture similarities. They employed a dispersion index to adaptively localize the texture regions at different scales. DeepDC [12] proposed the distance correction to measure the similarity between the reference and corrupted images in the deep feature domain.

Although the methods mentioned above can address texture resampling and texture similarity, they still have limitations, such as sensitivity to specific distortion types, largely due to their reliance on a single pre-trained CNN feature extractor. According to the no-free-lunch theorem, using one pre-trained model (i.e., a pre-trained VGG19 network) for feature extraction may work well for certain distortions but not for all. Specifically, DISTS and DeepDC perform a little poorly on the large-scale IQA dataset TID2013, which involves 3000 images with 24 types of distortion. These methods yield limited results when dealing with high-frequency noise, impulse noise, local block-wise distortion, color saturation, and changes in contrast. In contrast, this paper proposes the DualNetIQ model, which utilizes carefully selected multi-scale feature maps from multiple powerful pre-trained models, enabling it to handle various distortion types. DualNetIQ offers improved computational efficiency and reliability in image quality assessment and texture similarity tasks.

3. Proposed Method

3.1. DualNetIQ Model

Figure 2 presents an overview of the architecture of the proposed DualNetIQ model. In this process, there are two main steps: using multi-scale hybrid feature maps to extract features and using the proposed optimal combined similarity metric to measure similarity in the deep feature domain. Given the clean reference image (I) and the damaged image (G), DualNetIQ uses robust pre-trained models to choose the best multiscale feature maps to represent the input images. Notably, effectively capturing the perceptual quality of images requires a robust image representation approach. Since perceptual distances are non-uniform in pixel space [37,38], the goal of an image representation function is to transform the pixel representation into a space that is more perceptually uniform. In the proposed DualNetIQ model, we carefully select hybrid multi-scale feature maps from powerful pre-trained CNN models to extract robust representations of pristine reference and distorted images across various distortion types. Through an exhaustive ablation study [39] on different CNN models, including VGG19 [13], ResNet [40], AlexNet [27], and SqueezeNet [28], we found that VGG19 and SqueezeNet performed the best for feature representation. We also expanded the comparative study to include a stage-by-stage analysis to identify the most effective stages for our task. Additionally, this study involved some transformer-based models for further evaluation.

As shown in Figure 2, DualNetIQ includes two identical branches: the top branch extracts representation for the reference image while the bottom branch extracts representation for the corrupted image. Each branch consists of eight layers: the first layer represents the input images, the second and third layers are feature maps rendered from the VGG19 pre-trained model, and the last five layers are feature maps rendered from the SqueezeNet pre-trained model. The feature vector extracted by each feature map is passed to the optimal combined similarity metric (OCSM) to compute the similarity score.

The feature representation of the input reference image, $f\left(I\right),$ and the representation of the distorted image $f\left(G\right)$ are formed by concatenating the features resulting from the feature maps of the top and bottom branches:

$$f\left(I\right)=\{{\tilde{I}}_j^{\left(i\right)};i=0,\dots m;j=1,\dots ,n_i\}$$

(1)

$$f\left(G\right)=\{{\tilde{G}}_j^{\left(i\right)};i=0,\dots m;j=1,\dots ,n_i\}$$

(2)

Here, m represents the number of feature maps in each branch, $f$ stands for the feature representation, $n_i$ denotes the dimensions of features extracted from the i-th feature map, and ${\tilde{I}}^{\left(0\right)}$ and ${\tilde{G}}^{\left(0\right)}$ are the input reference and corrupted images.

Table 1. Description of the stages of the proposed model.

Stage (i)	Description	Output Feature Dimension (ni)
1	Input image	3 × 256 × 256
2	VGG19 stage1: conv1_8	256 × 64 × 64
3	VGG19 stage2: conv2_4	512 × 32 × 32
4	SqueezeNet stage1: conv1_1	64 × 127 × 127
5	SqueezeNet stage2: conv2_6 (fire1 + fire2)	128 × 64 × 64
6	SqueezeNet stage3: conv3_6 (fire3 + fire4)	256 × 32 × 32
7	SqueezeNet stage4: conv4_6 (fire5 + fire6)	384 × 16 × 16
8	SqueezeNet stage5: conv5_6 (fire7 + fire8)	512 × 16 × 16

provides a description of the stages of the proposed model as well as the output feature dimension for each stage. The first stage corresponds to the input image. The second and third stages correspond to the convolutional responses from conv1_8 and conv2_4 layers of the VGG19 network. The fourth stage captures the convolutional response from the conv1_1 layer of SqueezeNet. The remaining four stages correspond to the responses from fire modules 2, 4, 6, and 8, respectively. As described in [28], a fire module consists of a squeeze convolution layer with 1 × 1 filters, followed by an expanding layer that combines both $1\times 1$ and $3\times 3$ convolution filters.

3.2. Optimal Combined Similarity Metric

At each stage, DualNetIQ computes global texture and structure similar measurements between the corresponding feature maps of reference and distorted images. The overall image quality score, $D(I,G)$, is determined through a weighted summation of the similarity scores across all stages. The formula is given by:

$$D(I,G)=1-\sum _{i=0}^m\sum _{j=1}^{ni}\left({\kappa }_{ij}l\left({\check{I}}_j^{\left(i\right)},{\check{G}}_j^{\left(i\right)}\right)+{\xi }_{ij}s\left({\check{I}}_j^{\left(i\right)},{\check{G}}_j^{\left(i\right)}\right)\right)$$

(3)

where {${\kappa }_{ij}$, ${\xi }_{ij}$} are positive learned values that are constrained such that

$$\sum _{i=0}^n\sum _{j=1}^{ni}{(\mathsf{\kappa }}_{ij}+{\xi }_{ij})=1$$

${\check{I}}_j^{\left(i\right)}$ and ${\check{G}}_j^{\left(i\right)}$ are defined in Equations (1) and (2), respectively. $l\left({\check{I}}_j^{\left(i\right)},{\check{G}}_j^{\left(i\right)}\right)$ and $s\left({\check{I}}_j^{\left(i\right)},{\check{G}}_j^{\left(i\right)}\right)$ represent texture similarity and structure similarity, respectively, and are equal:

$$l\left({\check{I}}_j^{\left(i\right)},{\check{G}}_j^{\left(i\right)}\right)={\displaystyle \frac{2{\mu }_{\tilde{I}j}^{\left(i\right)}{\mu }_{\tilde{G}j}^{\left(i\right)}+c_1}{{\left({\mu }_{\tilde{I}j}^{\left(i\right)}\right)}^2+{\left({\mu }_{Gj}^{\left(i\right)}\right)}^2+c_1}}$$

(4)

$$s\left({\check{I}}_j^{\left(i\right)},{\check{G}}_j^{\left(i\right)}\right)={\displaystyle \frac{2{\sigma }_{\tilde{I}j}^{\left(i\right)}{\sigma }_{\tilde{G}j}^{\left(i\right)}+c_2}{{\left({\sigma }_{\tilde{I}j}^{\left(i\right)}\right)}^2+{\left({\sigma }_{\tilde{G}j}^{\left(i\right)}\right)}^2+c_2}}$$

(5)

where ${\mu }_I$ and ${\sigma }_I$ are the mean and standard deviation of the image x, respectively, and ${\sigma }_{IG}$ is the covariance of the I and G images.

The parameters in Equation (3), known as $\mathrm{ĸ}$ and $\xi$, determine the weight of texture and structure similarity, so they are important in our model performance improvement. It should be noted that we explored many optimization methods to select the most appropriate tuning method for our model, including Random Search [41], Blend Search [42], Ax Search, Nevergrad Search [43], Bayesian Opt/HyperBand (BOHB) Search [44], and Bayesian Optimization (BO) Search within Raytune framework [45]. Moreover, we investigated some metaheuristic algorithms such as Particle Swarm Optimization [46], Grey Wolf Optimization (GWO) [47], and Tree-structured Parzen Estimator [48]. Ultimately, GWO yielded the best results, so we adopted this algorithm for the tuning of ĸ and $\xi$.

Grey wolves are at the top of the food chain. They exhibit a complex hunting behavior where they encircle their prey. This behavior is mathematically modeled in GWO with the following equations:

$$\stackrel{⃑}{D}=|\stackrel{⃑}{C}.{\stackrel{⃑}{X}}_p\left(t\right)-\stackrel{⃑}{X}(t\left)\right|$$

(6)

$$\stackrel{⃑}{X}\left(t+1\right)={\stackrel{⃑}{X}}_p\left(t\right)-\stackrel{⃑}{A}.\stackrel{⃑}{D}$$

(7)

where ${\stackrel{⃑}{X}}_p\left(t\right)$ is the position of the prey (best solution obtained so far), $\stackrel{⃑}{X}(t)$ is the position vector of a grey wolf, and $\stackrel{⃑}{A}$ and $\stackrel{⃑}{C}$ are coefficient vectors. These vectors are calculated as:

$$\stackrel{⃑}{A}=2\stackrel{⃑}{a}.{\stackrel{⃑}{r}}_1-\stackrel{⃑}{a}$$

(8)

$$\stackrel{⃑}{C}=2.{\stackrel{⃑}{r}}_2$$

(9)

where $\stackrel{⃑}{a}$ decreases linearly from 2 to 0 over the course of iterations, and ${\stackrel{⃑}{r}}_1$ and ${\stackrel{⃑}{r}}_2$ are random vectors in [0, 1].

Grey wolves have a hierarchy made up of four kinds of wolves: $\alpha , \beta , \delta ,$ and $\omega$. This hierarchy is very important in the coordination of group hunting and survival systems; GWO takes advantage of this when it comes to problem-solving techniques in complex optimization. The social hierarchy in the GWO algorithm is modeled by considering the best solution as alpha ($\alpha$), the second-best solution as beta ($\beta$), the third-best solution as delta ($\delta$), and other remaining solutions are omegas ($\omega$). This hierarchy is very important for guiding the optimization process where $\alpha$, $\beta ,$ and $\delta$ wolves guide $\omega$ wolves towards promising areas of search space while maintaining a balance between exploration and exploitation. The wolves update their positions in terms of the best three solutions so far obtained $(\alpha , \beta , \delta )$ to show how they hunt by tracking, chasing, and attacking. The next equations present how to update positions:

$${\stackrel{⃑}{X}}_1={\stackrel{⃑}{X}}_{\alpha }-{\stackrel{⃑}{A}}_1.|{\stackrel{⃑}{C}}_1.{\stackrel{⃑}{X}}_{\alpha }-\stackrel{⃑}{X}|$$

(10)

$${\stackrel{⃑}{X}}_2={\stackrel{⃑}{X}}_{\beta }-{\stackrel{⃑}{A}}_2.|{\stackrel{⃑}{C}}_2.{\stackrel{⃑}{X}}_{\beta }-\stackrel{⃑}{X}|$$

(11)

$${\stackrel{⃑}{X}}_3={\stackrel{⃑}{X}}_{\delta }-{\stackrel{⃑}{A}}_3.|{\stackrel{⃑}{C}}_3.{\stackrel{⃑}{X}}_{\delta }-\stackrel{⃑}{X}|$$

(12)

$$\stackrel{⃑}{X}\left(t+1\right)=({\stackrel{⃑}{X}}_1+{\stackrel{⃑}{X}}_2+{\stackrel{⃑}{X}}_3)/3$$

(13)

Figure 3 shows the steps of using GWO to fine-tune our model parameters. The fitness function for this method is

$$fitness=|1-\rho (gt\_scores,d\_scores\left)\right|$$

(14)

where $\rho (gt\_scores,d\_scores)$ is the Spearman’s correlation coefficient (SRCC) between ground truth scores and the scores computed by our model using the instance $\kappa$ and $\xi$ values suggested by the GWO algorithm. Equation (14) is the equation that GWO seeks to minimize in order to obtain the optimal values of $\kappa$ and $\xi$ that result in the best alignment between the human subjective scores and the quality expectations resulting from our model. Application of GWO is best suited because this algorithm’s performance in exploring complex and high-dimensional search spaces helps us refine parameters influencing the model’s outputs.

We conduct this parameter learning on a rich dataset containing various types of image distortions. This ensures that the tuned parameters will enable DualNetIQ to perform exceptionally well across various types of distortions, thereby enhancing its usability and dependability in practical applications. The values of κ and ξ that yield the smallest value of 1-SRCC are the final optimal values. This will confirm not only that the quality predictions made by the model are accurate but that they are also in tune with the subtle perceptual preferences of human observers.

3.3. Implementation of DualNetIQ

The implementation of DualNetIQ, as outlined in Algorithm 1, begins with loading the tuned parameters, $\kappa$ and $\xi$, and then the pre-trained models are loaded, VGG19 and SqueezeNet, partially freezing their parameters to maintain the features that have been learned while lowering computational complexity. Code is available at https://github.com/Hossam-Mady/DualNetIQ.

Algorithm 1: Pseudocode of DualNetIQ
1 2 3 4 5	Load tuned κ and ξ from files Load pretrained VGG19 and SqueezeNet models Define stages as sequential models for different layers of VGG19 and SqueezeNet Freeze model parameters Set mean and standard deviation for input normalization Define channel sizes list ([3,256,512,64,128,256,384,512])
6 7 8 9 10 11 12 13 14 15	def forward_once(x):                      Normalize x using mean and std                      # Pass x through VGG19 stages                      Pass x through VGG19_stage1 -> h1                      Pass h1 through VGG19_stage2 -> h2                     # Pass x through SqueezeNet stages                      Pass x through SqueezeNet_stage1 -> h3                      Pass h3 through SqueezeNet_stage2 -> h4                      Pass h4 through SqueezeNet_stage3 -> h5                      Pass h5 through SqueezeNet_stage4 -> h6                      Pass h6 through SqueezeNet_stage5 -> h7                      Return [x, h1, h2, h3, h4, h5, h6, h7]
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31	def forward(x, y, require_grad = False, batch_average = False):                      Normalize kappa and xi                      term1 = 0                      term2 = 0                      for i in range(len(channel_sizes)):                                           x_mean, y_mean = Compute mean of features from x and y                                           S1 = Compute similarity using x_mean, y_mean                                           term1 += kappa[i] * S1                                           x_var, y_var = Compute variance of features from x and y                                            S2 = Compute similarity using x_var, y_var                                            term2 += xi[i] * S2                       score = 1 − (term1 + term2).squeeze()                       if batch_average:                                           return score.mean()                       else:                                           return score

The VGG19 and SqueezeNet stages are then defined, with each stage representing specific layers within each architecture. Additionally, we specify input normalization, which provides the mean and standard deviation to properly scale the input images for feeding into the network. The forward_once function passes an input image through the defined stages of VGG19 and SqueezeNet. First, it normalizes the image and then feeds it sequentially through the VGG19 stages, extracting features h1 and h2, and through the SqueezeNet stages, extracting features h3 to h7. The algorithm returns a list of features from both models, which subsequently serve as the basis for computing similarity. The core of the algorithm is the forward function, which computes the similarity between the features extracted from two images, $x$ and $y$. It normalizes first the $\kappa$ and $\xi$ parameters, and it initializes two terms, term1 and term2, which will accumulate the score. For each stage, it calculates the meaning and variance of the features from both images. Then, it computes two similarities, $S1$ (texture similarity) and $S2$ (structure similarity), weighted respectively by kappa[i] and xi[i]. The combination of these two weighted similarities produces an image quality score, normalized to fall within a specific range.

3.4. Evaluation Metrics

The correlation between the objective IQA quality score and mean opinion scores (MOSs) is a commonly used measure to evaluate the objective performance of IQA. Three metrics are commonly used to measure this correlation: SRCC, KRCC, and PLCC.

While PLCC gauges the predicted quality score’s linearity, SRCC and KRCC assess the predicted quality score’s monotonicity. We use SRCC and KRCC, as PLCC requires the produced scores to exhibit linearity in relation to subjective ratings; higher SRCC and KRCC values indicate higher-caliber prediction outcomes.

$$SRCC=1-{\displaystyle \frac{6{\sum }_{i=1}^n{d_i}^2}{n(n^2-1)}}$$

(15)

$$KRCC={\displaystyle \frac{2(C-D)}{n(n-1)}}$$

(16)

where $d_i$ is the difference between the $i^{th}$ image’s ranks in the objective and the subjective ratings, $C$ is the number of concordant pairs, $D$ is the number of discordant pairs, and n is the number of observations.

4. Results and Discussion

In this section, we discuss the comprehensive tests conducted to validate the effectiveness of our suggested model. These consisted of standard image quality prediction evaluation, perceptual similarity measurement, and texture similarity assessment. Finally, we conducted ablation studies to demonstrate the impact of each component on the model.

4.1. Parameter Setting

As previously mentioned, we optimized the weights using the GWO method. It was configured with a population size equal to four and a maximum number of iterations of 50 iterations. The dimensionality of the search space was set to 16, reflecting the number of parameters (the total of the kappa and xi values over all stages). Parameter values were restricted within the range [0, 1]. The error function was defined as the absolute value of (1-SRCC), and the goal was to minimize this value.

We performed all computations on an NVIDIA GeForce GTX 1650 GPU using PyTorch version 2.2.0, with all computations offloaded to the CUDA-enabled GPU to enhance performance and reduce runtime. Our system was running Windows 10 64-bit with CUDA version 12.1 for GPU acceleration. This experiment was conducted on a machine with an Intel(R) core (TM) i7-9700K processor (3.60 GHz) and 32 GB of RAM.

Algorithm 1 shows the steps taken on the reference image and the distorted image to calculate the final assessment of the quality of the distorted image.

4.2. Performance in Quality Prediction and Texture Similarity

In this section, we compare DualNetIQ with 11 FR-IQA models on five standard IQA datasets (LIVE [14], CSIQ [15], TID2013 [16], KADID-10k [17], and PIPAL [18]), as well as on a specialized texture similarity dataset, SynTEX [4], which contains 105 synthesized texture images created using five different methods of texture synthesis based on 21 high-quality textures. The 11 FR-IQA methods include six knowledge-driven models (PSNR, SSIM [3], MS-SSIM [20], FSIM [25], VIF [24], and NLPD [49]) and five data-driven CNN-based models (PieAPP [50], LPIPS [10], DISTS [11], A-DISTS [36], and DeepDC [12]). For consistent evaluation, all models were evaluated using their original implementations, and we selected the default implementation of LPIPS, which is LPIPS-VGG-lin.

Table 2. Performance comparison of the DualNetIQ model against multiple state-of-the-art FR-IQA models on four standard IQA datasets, as well as one texture similarity dataset.

Method	Quality Prediction										Texture Similarity
	LIVE [14]		CSIQ [15]		TID2013 [16]		KADID-10k [17]		PIPAL [18]		SynTEX [4]
	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC
PSNR	0.873	0.680	0.809	0.599	0.687	0.496	0.676	0.488	0.407	0.233	0.320	0.211
SSIM [3]	0.931	0.763	0.872	0.678	0.720	0.527	0.724	0.537	0.498	0.276	0.620	0.446
MS-SSIM [20]	0.951	0.805	0.906	0.730	0.786	0.605	0.826	0.635	0.552	0.291	0.632	0.454
FSIMc [25]	0.965	0.837	0.931	0.768	0.851	0.666	0.854	0.665	0.569	0.304	0.081	0.077
VIF [24]	0.964	0.828	0.911	0.743	0.677	0.518	0.679	0.507	0.443	0.261	0.606	0.492
NLPD [49]	0.937	0.778	0.932	0.769	0.800	0.625	0.812	0.623	0.469	0.255	0.606	0.464
PieAPP [50]	0.919	0.750	0.892	0.715	0.876	0.683	0.836	0.647	0.700	0.492	0.715	0.532
LPIPS [10]	0.932	0.765	0.876	0.689	0.670	0.497	0.843	0.653	0.573	0.323	0.663	0.478
DISTS [11]	0.954	0.811	0.939	0.780	0.830	0.639	0.887	0.709	0.624	0.433	0.923	0.759
A-DISTS [36]	0.955	0.812	0.942	0.796	0.836	0.642	0.890	0.715	0.622	0.431	0.760	-
DeepDC [12]	0.940	0.781	0.937	0.774	0.844	0.651	0.905	0.733	0.684	0.467	0.896	0.727
DualNetIQ (ours)	0.955	0.815	0.930	0.764	0.865	0.678	0.897	0.719	0.635	0.452	0.938	0.792

presents the comparative results that provide a number of important insights. Every existing IQA dataset shows that the proposed DualNetIQ is strong, outperforming older methods like PSNR, SSIM, and MS-SSIM as well as newer CNN-based methods like PieAPP and LPIPS. It is intriguing that DualNetIQ does better than DISTS, A-DISTS, and DeepDC in most of the IQA datasets. This demonstrates its ability to conduct more comprehensive structural and textural analysis, a crucial step in accurately evaluating image quality. DualNetIQ achieves the second-highest SRCC and KRCC values on the LIVE, TID2013, and KADID-10K datasets. Notably, the difference between its performance and the top-performing model on KADID-10K (DeepDC) is only 0.008. It is clear that DualNetIQ enhances performance on TID2013, achieving strong results compared to other methods, especially those that are insensitive to texture resampling. On the CSIQ dataset, DualNetIQ delivers comparable results (SRCC = 0.930 and KRCC = 0.764). Furthermore, when evaluating individual distortions on CSIQ, it outperforms other models. When it comes to the pipal dataset, which has a lot of GAN-generated images that are hard to work with because they have synthesized textures that are not in the reference images, DualNetIQ also does well.

With $SRCC=0.938$ and $KRCC=0.792$, DualNetIQ is much better than any knowledge-based or advanced data-driven model at figuring out how similar two textures are across the SynTEX dataset. DualNetIQ’s superior feature extraction capabilities, which effectively extract key attributes related to visual texture, account for its performance edge. This aspect presents a challenge for traditional models, as they rely on precise pixel correspondence. Also, it is worth noting that DualNetIQ outperforms modern methods known for their ability and superiority in texture similarity assessment, such as DISTS and DeepDC.

4.3. Performance on Perceptual Similarity Measurment

In addition to the traditional image quality evaluation, DualNetIQ performance was also evaluated for the perceptual similarity measurement on the BAPPS dataset. BAPPS contains a diverse collection of image patches that includes distortions produced by real-world algorithms such as super resolution, video deblurring, and frame interpolation, as well as synthetic ones, whether traditional or CNN-based. The evaluation of perceptual similarities in BAPPS follows a Two-Alternative Forced Choice (2AFC) experimental setup. This score quantifies the consistency between the model and human preference; the higher the score, the more consistent the model is, providing a strong indication of its perceptual alignment with human visual perception.

The results shown in

Table 3. Performance comparison of various IQA models on the BAPPS [10] dataset. The table displays the 2AFC scores for each model, which quantify the agreement with human judgments on image quality. Scores range from [0, 1], with higher values indicating better performance and closer alignment with human perceptual evaluations.

Method	Synthetic Distortions			Distortions by Real-World Algorithms					All
	Traditional	CNN-Based	All	Super Resolution	Video Deblurring	Colorization	Frame Interpolation	All
Human	0.808	0.844	0.826	0.734	0.671	0.688	0.686	0.695	0.739
PSNR	0.573	0.801	0.687	0.642	0.590	0.624	0.543	0.614	0.633
SSIM [3]	0.605	0.806	0.705	0.647	0.589	0.624	0.573	0.617	0.640
MS-SSIM [20]	0.585	0.768	0.676	0.638	0.589	0.524	0.572	0.596	0.617
FSIMc [25]	0.627	0.794	0.710	0.660	0.590	0.573	0.581	0.615	0.640
VSI [23]	0.630	0.818	0.724	0.668	0.592	0.597	0.568	0.622	0.648
VIF [24]	0.556	0.744	0.650	0.651	0.594	0.515	0.597	0.603	0.615
NLPD [49]	0.550	0.764	0.657	0.655	0.584	0.528	0.552	0.600	0.615
GMSD [51]	0.609	0.772	0.690	0.677	0.594	0.517	0.575	0.613	0.633
DeepIQA [1]	0.703	0.794	0.748	0.660	0.582	0.585	0.598	0.615	0.650
PieAPP [50]	0.727	0.770	0.746	0.684	0.585	0.594	0.598	0.627	0.659
LPIPS [10]	0.714	0.814	0.764	0.705	0.605	0.625	0.630	0.641	0.692
DISTS [11]	0.749	0.824	0.786	0.705	0.600	0.629	0.625	0.649	0.685
DeepDC [12]	0.757	0.825	0.796	0.712	0.608	0.631	0.626	0.655	0.692
DualNetIQ (ours)	0.739	0.832	0.797	0.719	0.606	0.648	0.631	0.661	0.693

highlight the excellent performance of DualNetIQ both for synthetic and real-world distortions. The proposed model outperformed all methods in managing distortions generated by real-world algorithms by achieving the highest 2AFC score of 0.661 and it also surpassed all methods in managing synthetic distortions by achieving the highest 2AFC score of 0.797 and thus the highest 2AFC score in total, which was 0.693.

4.4. Performance Comparison Across Diversity of Distortion Types

Table 4. Performance of DualNetIQ against texture-resampling-insensitive methods over a subset of distortion types in the TID2013 dataset.

Noise Type	DISTS [11]		A-DISTS [36]		DeepDC [12]		DualNetIQ
	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC
Quantization Noise	0.831	0.644	0.799	0.62	0.872	0.682	0.862	0.672
Gaussian Blur	0.938	0.776	0.938	0.774	0.954	0.808	0.941	0.78
Contrast Change	0.488	0.335	0.433	0.289	0.472	0.297	0.472	0.303
JPEG Compression	0.904	0.702	0.907	0.709	0.917	0.725	0.921	0.736
JPEG 2000 Compression	0.944	0.795	0.945	0.797	0.949	0.801	0.946	0.798
Additive Gaussian Noise	0.877	0.677	0.858	0.651	0.868	0.667	0.883	0.688
High-Frequency Noise	0.878	0.659	0.872	0.651	0.879	0.663	0.895	0.677
Impulse Noise	0.713	0.506	0.708	0.503	0.693	0.48	0.8	0.591
Image Denoising	0.905	0.736	0.892	0.714	0.912	0.755	0.923	0.762
Mean Shift	0.801	0.594	0.801	0.598	0.737	0.538	0.804	0.604
Change of Color Saturation	0.814	0.612	0.839	0.644	0.717	0.518	0.836	0.642
Multiplicative Gaussian Noise	0.83	0.616	0.795	0.578	0.823	0.611	0.826	0.612

shows how well DualNetIQ performed compared to DISTS, A-DISTS, and DeepDC for a certain group of noise types in the TID2013 dataset, using SRCC and KRCC. The results show that our proposed model achieved strong generalization across a wide range of distortion types. The proposed model consistently achieved the best or at least the second-best result for each type of distortion, demonstrating its remarkable versatility and robustness.

Figure 4 shows a boxplot of SRCC values for IQA metrics for different types of distortion in the TID2013, CSIQ, and LIVE datasets. It is easy to see that our suggested model, DualNetIQ, had the highest median and upper whisker. DeepDC demonstrated a slightly lower yet comparable median; however, it presented outliers that suggest limitations associated with certain distortion types. Conversely, A-DISTS recorded the lowest median, and despite the absence of visible outliers, the behavior of its lower whisker was comparable to or worse than that of other metrics’ outliers. DISTS had a median slightly lower than both DualNetIQ and DeepDC, albeit still comparable, and it also exhibited outliers that highlighted limitations for specific distortion types.

Furthermore, Figure 5 provides a similar analysis for KRCC values. DualNetIQ also had the highest median, as well as the highest upper and lower whiskers. DeepDC exhibited a slightly lower median and the smallest lower whisker, suggesting that it performs poorly under certain distortion classes. A-DISTS had the lowest median, with its lower whisker slightly higher than DeepDC’s. Furthermore, DISTS exhibited a marginally lower median than both DualNetIQ and DeepDC, all while maintaining a lower whisker.

In summary, it is evident that DualNetIQ performs better than the other models, resolving issues and improving generalizability across different kinds of distortion.

4.5. Ablation Study

We conducted a series of ablation experiments to delve into the influences of various CNNs acting as feature extractors in the DualNetIQ framework. Specifically, the aim was to study how different CNN selections affect the performance of the model.

First, we carefully tuned κ and ξ parameters in each configuration using the GWO algorithm to ensure the validity and reliability of our findings. This study was essential because it provided a means to identify settings that would result in optimum performance for each CNN network and ensure fair comparison.

Second, we tested the performance of each CNN variant (VGG19, SqueezeNet, AlexNet, and ResNet50) on a wide range of five datasets, including four benchmark IQA datasets and one texture similarity dataset.

Table 5. Ablation study results: comparative performance of DualNetIQ using different CNN architectures as standalone feature extractors.

CNN Model	Quality Prediction								Texture Similarity
	LIVE [14]		CSIQ [15]		TID2013 [16]		KADID-10k [17]		SynTEX [4]
	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC	SRCC	KRCC
VGG19	0.954	0.813	0.941	0.786	0.833	0.648	0.896	0.718	0.928	0.773
SqueezeNet	0.953	0.812	0.896	0.714	0.858	0.668	0.881	0.695	0.930	0.778
AlexNet	0.942	0.797	0.900	0.716	0.833	0.640	0.860	0.666	0.892	0.713
ResNet50	0.862	0.681	0.816	0.633	0.774	0.578	0.801	0.612	0.631	0.490
VGG19 + SqueezeNet (DualNetIQ)	0.955	0.815	0.930	0.764	0.865	0.678	0.897	0.719	0.938	0.792

shows how each architecture performed in terms of evaluating texture similarity and image quality, offering a detailed comparison of the performance metrics for each CNN. It also presents the results of combining the VGG19 and SqueezeNet networks (i.e., the proposed DualNetIQ model).

As one can see in Table 5, the VGG19 and SqueezeNet networks achieved SRCC values higher than 0.9 with the LIVE and SynTEX datasets. However, AlexNet obtained good results with LIVE, CSIQ, TID2013, and KADID-10k, and it achieved limited results with the SynTEX dataset (KRCC values < 0.72). In turn, ResNet50 obtained limited results with the SynTEX dataset (SRCC values < 0.7 and KRCC values < 0.5). According to the results, VGG19 demonstrated commendable performance, approaching that of DualNetIQ in certain cases, and even outperforming in the case of CSIQ. However, it is important to note that DualNetIQ outperformed VGG19 across all datasets if we consider the different distortion types separately in CSIQ. In addition, DualNetIQ reduced the computational load of the model by selecting and utilizing only selective layers from VGG19, particularly by replacing some of the computation-intensive layers from VGG19 with more efficient counterparts from SqueezeNet. The quality prediction became much better, especially on the TID2013 dataset, where it achieved SRCC = 0.865 and KRCC = 0.678, compared to SRCC = 0.833 and KRCC = 0.648 when VGG19 was used alone. Additionally, on the SynTEX dataset, adding efficient SqueezeNet layers increased SRCC to 0.938 and KRCC to 0.792, which was better than the SRCC of 0.928 and KRCC of 0.733 that was achieved with VGG19 alone.

4.6. Complexity Comparison

We compared our proposed method, DualNetIQ, with a set of methods used to evaluate image quality in computational complexity, a perspective that is crucial for real-time applications. The methods used in comparison were FPS and FLOPS. As shown in

Table 6. FPS and FLOPS results for five IQA models including DualNetIQ.

Model	FPS	FLOPS (Billion)
LPIPS [10]	9.68	401
PieAPP [50]	0.36	827.3
DISTS [11]	32.9	40.125
DeepDC [12]	27.77	56.69
DualNetIQ (ours)	29.8	46.92

, DualNetIQ achieved a competitive FPS value of 29.8 and FLOPS of 46.92 billion, which was the second-best result after DISTS, which achieved a slightly better result. However, DualNetIQ outperformed DISTS in terms of image quality assessment and texture similarity, particularly when dealing with certain types of distortions.

It should be noted that one limitation of the model is that it has not been extensively tested on images with unusual colors such as night shots and underwater. Also, we did not present results for real systems (e.g., in surveillance cameras or vision systems) in this study. Additionally, the performance of the proposed model with other transformer architectures [52] could be studied. All these points will be addressed in future studies.

5. Conclusions and Future Work

In this paper, we proposed DualNetIQ, an FR-IQA model with explicit tolerance to texture resampling. Our model utilizes two powerful CNNs, VGG19 and SqueezeNet, to extract features from both reference and distorted images. The eight feature maps are then combined using the optimal combined similarity metric, with its weighting parameters fine-tuned using GWO. Our model architecture enables it to address the limitations of traditional metrics as well as deep learning-based metrics that rely on a single CNN for feature extraction. DualNetIQ was tested on IQA, texture similarity, perceptual similarity, and different types of distortion. The results show that our model works better and more efficiently than other existing models.

Future work may involve incorporating additional metrics to calculate the similarity at each stage of the DualNetIQ model, with the aim of enhancing its performance. Additionally, different methods for parameter tuning may be investigated. The effectiveness of various architectures, such as EfficientNet or Swin Transformer, with the proposed model will be also assessed. Moreover, we will conduct real-world deployment tests and evaluate the model’s performance in handling images with unusual colors.