Saliency-Guided Local Full-Reference Image Quality Assessment

Abstract

Research and development of image quality assessment (IQA) algorithms have been in the focus of the computer vision and image processing community for decades. The intent of IQA methods is to estimate the perceptual quality of digital images correlating as high as possible with human judgements. Full-reference image quality assessment algorithms, which have full access to the distortion-free images, usually contain two phases: local image quality estimation and pooling. Previous works have utilized visual saliency in the final pooling stage. In addition to this, visual saliency was utilized as weights in the weighted averaging of local image quality scores, emphasizing image regions that are salient to human observers. In contrast to this common practice, visual saliency is applied in the computation of local image quality in this study, based on the observation that local image quality is determined both by local image degradation and visual saliency simultaneously. Experimental results on KADID-10k, TID2013, TID2008, and CSIQ have shown that the proposed method was able to improve the state-of-the-art’s performance at low computational costs.

1. Introduction

Image quality assessment (IQA) is a hot research topic in the image processing community, since it can be applied in a wide range of practical and important applications, such as the optimization of computer vision system parameters [1] and monitoring the quality of image displays [2], it is also helpful in the process of benchmarking image and video compression and denoising algorithms [3,4]. In practice, the final perceivers and users of digital images are human beings. As a consequence, the most reliable way of evaluating the perceptual quality of digital images is subjective user studies involving a group of vision experts and human observers either in a laboratory environment [5] or a crowdsourcing experiment [6]. Further, these subjective evaluation methods result in publicly available benchmark IQA databases, which can be used in objective IQA. Namely, objective IQA tries to construct mathematical algorithms that estimate the perceptual quality of digital images as consistently as possible with human judgement. To help the development of IQA methods, publicly available benchmark databases, such as TID2013 [7] and CSIQ [8], contain digital images with their mean opinion score (MOS) values, which are the averages of individual human quality ratings. Traditionally, IQA is divided into three classes [9]: full-reference (FR), no-reference (NR), and reduced-reference (RR). The main difference is the degree of access to the reference, i.e., the distortion-free images. Namely, FR-IQA algorithms have full access to the reference image, while NR-IQA ones have no access to the them, and RR-IQA methods have limited access to the reference images.

Human vision research has increased the understanding of the human visual system. During observation of an image, people tend to focus on the visually significant part of a scene. As a consequence, the content of the image is not treated equally. This is why considering visual saliency in FR-IQA has gained a lot of attention in the literature [10,11,12]. In the literature, the traditional and well-known PSNR [13] was first improved using visual saliency computation by dividing the input images into different regions and assigning weights to them according to their estimated saliency. This work was followed by a line of papers utilizing visual saliency from different features. Specifically, Zhang et al. [14] utilized the phase congruency feature [15] as a visual saliency measure to create weights for defined feature similarity metrics. Wang and Li [16] compiled local distortion maps from the reference and distorted images. To quantify visual quality, these local distortion maps were pooled together by using saliency as a weighting function. The common point of algorithms using visual saliency is that visual saliency is used as a weighting function in the comparison of local image quality maps.

1.1. Motivation and Contributions

Over the evolution of the human race, humans have evolved to pay different amounts of attention to different regions of a visual scene as it is viewed. This mechanism obviously affects human perceptual quality judgements. Previously published FR-IQA methods [14,16,17,18] utilize visual saliency in the final pooling stage of the algorithm. To be more specific, visual saliency is used as weight in the weighted averaging of local image quality scores, emphasizing image regions that are more salient to the human visual system. The main contribution of this paper was the following. As opposed to other previously published algorithms, visual saliency was utilized by this work in the computation of local image quality and not in the pooling stage, which is usual in the literature. In fact, the human visual system perceives image degradation with higher probability in visually significant regions and tends to neglect degradation in visually less significant regions. In other words, local quality degradation is simultaneously influenced by both local image degradation and visual saliency. The effect of visual saliency on the perception of local image quality is demonstrated below. If humans observe Figure 1a, they to tend to focus their attention on the child’s face rather than on the vegetation in the background. To be specific, we added the same amount of noise to two different visually significant areas of the image. In Figure 1a, it was added to the region containing the the vegetation in the left corner, while it was added to the face in Figure 1b. Comparing these two figures, the human visual system was less likely to perceive the degradation if it was not in a visually salient area.

Based on the above-mentioned observations, a visual-saliency-guided FR-IQA framework was introduced where the visual saliency adaptively adjusted local image quality. Within this framework, ESSIM (edge strength similarity-based image quality metric) [19] method was further developed. Furthermore, it was demonstrated that the proposed method was able to improve state-of-the-art performance at low computational costs on four large and widely accepted IQA benchmark databases, i.e., KADID-10k [20], TID2013 [7], TID2008 [21], and CSIQ [8].

1.2. Organization of the Paper

After this introduction, related and previous papers are reviewed in Section 2. Next, our proposed method is described in Section 3. Databases, evaluation metrics, and implementation details are given in Section 4. Numerical, experimental results, and a comparison to the state-of-the-art is presented in Section 5. At the end, a conclusion is drawn in Section 6.

2. Related Work

As already mentioned, FR-IQA algorithms predict the perceptual quality of distorted images with full access to the reference images. The traditional and probably the simplest methods for FR-IQA are mean squared error (MSE) and peak signal-to-noise ratio (PSNR), which rely on the measurement of the distortion energy of the images. However, their performance lags behind other more sophisticated metrics, since their results are not consistent with human quality perception [22]. Hence, many FR-IQA methods have tried to build on the properties of the human visual system (HVS) to compile effective algorithms. For example, the structural similarity index (SSIM) [23] utilizes the observation that HVS is sensitive to contrast and structural distortions. More specifically, SSIM [23] applies local sliding windows on the reference and distorted images and measures luminance, contrast, and structure similarity using predefined functions containing average, variance, and covariance computations. Finally, the perceptual quality of the distorted image is obtained by taking the arithmetic mean of the similarity values of the local sliding windows. Brunet et al. [24] investigated the mathematical properties of SSIM and pointed out that SSIM satisfies the identity of indiscernibles and exhibits symmetry properties, excluding the triangle inequality. SSIM has become quickly popular in the signal processing community and gained a significant amount of attention for further research [25]. For example, Wang et al. [26] extended SSIM into MS-SSIM by conducting multi-scale processing. Further, Sampat et al. [27] utilized complex wavelet domains to define a new structural similarity index. In contrast, Chen et al. [28] modified SSIM to compare edge information between the reference and the distorted images. In addition to structural degradation, image gradient is also a popular feature in FR-IQA. Specifically, Liu et al. [29] defined an FR-IQA metric using gradient similarity between the reference and distorted images. Similarly, Zhu and Wang [30] utilized gradient similarity, but scale information was also incorporated into their metric.

Motivated by the success of deep learning in many image processing tasks [31,32,33], deep learning has also appeared in FR-IQA. For example, significant amount of proposed works compare deep activations of convolutional neural networks (CNNs) for a reference–distorted image pair to establish an FR quality metric. For example, Amirshahi et al. [34] extracted feature maps using an AlexNet [35] CNN. Subsequently, the extracted feature maps were compared using histogram intersection kernels [36] and a similarity value was produced for each feature map. To obtain perceptual quality, the arithmetic mean of these similarity values was taken. Later, this approach was developed further [37] by replacing the histogram intersection kernels by traditional image similarity metrics. Another line of work focused on training end-to-end deep architectures on large IQA benchmark databases [38,39,40]. Chubarau et al. [39] investigated vision transformers and self-attention for FR-IQA. Namely, a context-aware sampling procedure was introduced to extract patches from the reference and the distorted images. Next, the patches were encoded by a vision transformer.

Another class of FR-IQA methods uses existing and available FR-IQA metrics to construct a new metric with improved performance. For instance, Okarma [41] studied the characteristics of three different metrics. In addition, the authors introduced a combined metric by taking the exponentiated product of the examined three metrics. In contrast, other proposals utilized optimization techniques to find optimal weights for a linear combination of already existing FR-IQA techniques [42,43,44]. Instead of optimization, Lukin et al. [45] trained a neural network from scratch using traditional FR-IQA metrics as image features to obtain improved estimation performance.

3. Proposed Method

In this section, our proposed method is introduced and described. Specifically, we first gave an overview about the ESSIM [19] method to better understand our contribution to the saliency-guided determination of local image quality. In the following, the ESSIM [19] algorithm is first briefly described, and then the saliency-guided SG-ESSIM is described in detail.

3.1. ESSIM Method

To quantify quality degradation of a distorted image $\left(g\right)$ with respect to a reference image $\left(f\right)$, the ESSIM is defined as

$$ESSIM(f,g)=\frac{1}{N}\sum _{i=1}^N\frac{2E(f,i)E(g,i)+C}{{\left(E(f,i)\right)}^2+{\left(E(g,i)\right)}^2+C},$$

(1)

where C is a small constant to avoid the division by zero. To be more specific, C has two roles in Equation (1): as already mentioned, it is necessary to avoid having a denominator equal to zero; secondly C is also a scaling factor. If $C⟶\infty$, the result of similarity measure is 1. Further, $E(f,i)$ is used to characterize the edge strength around pixel i in the reference image f, and is determined as

$$E(f,i)=max(E_i^{1,3}\left(f\right),E_i^{2,4}\left(f\right)),$$

(2)

where $E_i^{2,4}\left(f\right)$ is the edge strength in the diagonal directions and is determined as

$$E_i^{2,4}\left(f\right)={|\partial f_i^2-\partial f_i^4|}^p.$$

(3)

Similarly, $E_i^{1,3}\left(f\right)$ stands for the edge strength but in the vertical and horizontal directions and is computed as

$$E_i^{1,3}\left(f\right)={|\partial f_i^1-\partial f_i^3|}^p.$$

(4)

In Equations (3) and (4), $\partial f_i^j$ stands for the directional derivatives at pixel i in reference image f. The applied directions in the ESSIM are the following: $0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }$. Further, the fractional derivatives in the ESSIM are implemented with the help of $5\times 5$ Scharr operators [46]. Moreover, p is used to adapt the edge strength in the algorithm. The edge strength of distorted image g around pixel i is defined as

$$E(g,i)=\left\{\begin{array}{c}\hfill E_i^{1,3}\left(g\right),\mathrm{if}E(f,i)=E_i^{1,3}\left(f\right)\hfill \\ \hfill E_i^{2,4}\left(g\right),\mathrm{if}E(f,i)=E_i^{2,4}\left(f\right)\hfill \end{array}\right.$$

(5)

in order to guarantee that the edge strengths of f and g are compared in the same direction. Otherwise, $E_i^{1,3}\left(g\right)$ and $E_i^{2,4}\left(g\right)$ are determined similarly to Equations (3) and (4).

3.2. SG-ESSIM Method

As one can observe from the derivation of the ESSIM [19], the impact of visual saliency was not incorporated into these metrics. In this section, we describe how we enhanced the ESSIM by using visual saliency in the measurement of local image quality. Hence, it was highlighted that local image quality degradation was jointly characterized by the objective degradation of edge strength and significance.

As proposed by the authors of the ESSIM [19], our algorithm first determined the edge strength maps of the reference and distorted images as recommended in the ESSIM [19]. Subsequently, it calculated the saliency-guided local image quality map. Finally, it pooled the local image quality map to obtain the overall perceptual quality of the distorted image.

Using Equations (2) and (5), which describe $E(f,i)$ and $E(g,i)$ in the ESSIM, our visual saliency-guided local similarity map is defined as

$$SM(E(f,i),E(g,i))=\frac{2·E(f,i)·E(g,i)+H\left(V\right(i\left)\right)}{{\left(E(f,i)\right)}^2+{\left(E(g,i)\right)}^2+H\left(V\left(i\right)\right)},$$

(6)

where $V\left(i\right)$ stands for the visual saliency measure at pixel location at pixel location i. Moreover, $H(·)$ is a decreasing function defined as

$$H\left(V\left(i\right)\right)=K·e^{-\frac{V\left(i\right)}{h}},$$

(7)

where e is Euler’s number, K is a scaling factor, and h is an attenuation factor (Equation (7) is a decreasing function because the similarity metric defined by Equation (9) also needs to be regulated). To be more specific, $V\left(i\right)$ takes the edge strength maps of the reference and distorted images and returns their pixel-wise maximum:

$$V\left(i\right)=max\left(E\right(f,i),E(g,i\left)\right).$$

(8)

Equation (8) implies that regions with strong edges are more salient to the human visual system than those with weak edges since edge information conveys essential information about the visual scene to humans [47]. A larger value of $V\left(i\right)=max\left(E\right(f,i),E(g,i\left)\right)$ indicated a higher visual saliency at pixel location i. Moreover, the computation of $V\left(i\right)$ did not result in a significant increase in the computational costs since $E(f,i)$ and $E(g,i)$ were determined anyway in the ESSIM [19]. Substituting Equations (7) and (8) into Equation (9), we obtain the following equation for visual saliency guided local similarity map:

$$SM(E(f,i),E(g,i))=\frac{2·E(f,i)·E(g,i)+K·e^{-\frac{max\left(E\right(f,i),E(g,i\left)\right)}{h}}}{{\left(E(f,i)\right)}^2+{\left(E(g,i)\right)}^2+K·e^{-\frac{max\left(E\right(f,i),E(g,i\left)\right)}{h}}}.$$

(9)

Finally, the overall predicted perceptual quality score of the image was estimated by averaging the visual saliency-guided (SG) local similarity map

$$SG-ESSIM=\frac{1}{N}\sum _{i=1}^{N}SM(E(f,i),E(g,i)),$$

(10)

where N is the total number of pixels.

The hyperparameters of the SG-ESSIM, which have to be set, were K and h. These parameters were determined on a subset of the TID2008 [21] database, which contained 5 random reference images and 340 corresponding distorted images. Specifically, those hyperparameters were chosen where the highest Spearman’s rank order correlation coefficient between the ground truth and the predicted values was obtained. As a result, we chose K = 51,000 and $h=0.5$ in our MATLAB implementation. Since the SG-ESSIM was not a machine-learning-based approach, this random subset was also used in the evaluation process.

4. Materials

In this section, the description of the applied benchmark IQA databases are given first. Second, we give the applied evaluation metrics and protocol. Finally, the implementation details of the proposed method are given.

4.1. Databases

In this study, four large publicly available IQA benchmark databases (KADID-10k [20], TID2013 [7], TID2008 [21], and CSIQ [8]) were used to evaluate and compare the state-of-the-art to the proposed method. The most important information about the applied databases is summarized in

Table 1. Summary of benchmark databases used in this study.

Database	Ref. Images	Dist. Images	Dist. Types	Dist. Levels
KADID-10k [20]	81	10,125	25	5
TID2013 [7]	25	3000	24	5
TID2008 [21]	25	1700	17	4
CSIQ [8]	30	866	6	4–5

. Namely, these databases contained a small set of distortion-free reference images from which distorted images were derived using different distortion types at different distortion levels. Moreover, the distorted images were annotated with subjective quality ratings. Therefore, they were suitable and accepted for the evaluation and ranking of FR-IQA metrics in the literature [48]. Figure 2 depicts the empirical MOS distributions in the applied databases.

4.2. Evaluation Metrics and Protocol

Ranking of FR-IQA methods relies on measuring the correlation between predicted and ground truth quality scores of benchmark IQA databases, such as KADID-10k [20]. To be more specific, the performance of an FR-IQA metric was evaluated using three different criteria in this study: Pearson’s linear correlation coefficient (PLCC), Spearman’s rank order correlation coefficient (SROCC), and Kendall’s rank order correlation coefficient (KROCC). In addition, a nonlinear logistic regression was applied before the calculation of the PLCC as recommended in [49], because nonlinear relationship exists between the ground truth and predicted scores:

$$Q={\beta }_1\left(\frac{1}{2}-\frac{1}{1+e^{{\beta }_2(Q_p-{\beta }_3)}}\right)+{\beta }_4{Q}_p+{\beta }_5,$$

(11)

where Q and $Q_p$ are the fitted and predicted scores, respectively. In addition, the regression described by Equation (11) is determined by the ${\beta }_i$$(i\in \{1,2,\cdots ,5\left\}\right)$ parameters.

4.3. Implementation Details

MATLAB R2021a was used to implement the proposed FR-IQA metric using the functions of the Image Processing Toolbox. The computer configuration applied in our experiments is outlined in

Table 2. Computer configuration applied in our experiments.

Computer model	STRIX Z270H Gaming
Operating system	Windows 10
Memory	15 GB
CPU	Intel(R) Core(TM) i7-7700K CPU 4.20 GHz (8 cores)
GPU	Nvidia GeForce GTX 1080

.

5. Experimental Results and Analysis

In this section, our experimental results and analysis are presented using four common benchmark IQA databases: KADID-10k [20], TID2013 [7], TID2008 [21], and CSIQ [8]. More specifically, the proposed SG-ESSIM method was compared to thirteen other state-of-the-art FR-IQA metrics (2stepQA [50], CSV [51], DISTS [52], ESSIM [19], GSM [29], IW-SSIM [16], MAD [8], MS-SSIM [26], PSNR, ReSIFT [53], RVSIM [54], SSIM [23], SUMMER [55]), whose original source codes were made publicly available by their authors, on the above-mentioned databases in terms of correlations with ground truth quality ratings. Furthermore, their hyperparameter values (if any) were applied in the examined state-of-the-art methods, which were recommended as default values by the original authors in their MATLAB implementations. In addition to this, a detailed comparison of TID2013 [7] and TID2008 [21] with respect to different noise types and noise levels is also presented.

The comparison, in terms of the PLCC, SROCC, and KROCC, in KADID-10k [20], TID2013 [7], TID2008 [21], and CSIQ [8] is summarized in

Table 3. Comparison of the SG-ESSIM to several other state-of-the-art algorithms on KADID-10k [20] and TID2013 [7]. The highest values are typed in bold, while the second highest ones are underlined.

	KADID-10k [20]			TID2013 [7]
FR-IQA Metric	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
2stepQA [50]	0.768	0.771	0.571	0.736	0.733	0.550
CSV [51]	0.671	0.669	0.531	0.852	0.848	0.657
DISTS [52]	0.809	0.814	0.626	0.759	0.711	0.524
ESSIM [19]	0.644	0.823	0.634	0.740	0.797	0.627
GSM [29]	0.780	0.780	0.588	0.789	0.787	0.593
IW-SSIM [16]	0.781	0.756	0.524	0.832	0.778	0.598
MAD [8]	0.716	0.724	0.535	0.827	0.778	0.600
MS-SSIM [26]	0.819	0.821	0.630	0.794	0.785	0.604
PSNR	0.479	0.676	0.488	0.616	0.646	0.467
ReSIFT [53]	0.648	0.628	0.468	0.630	0.623	0.471
RVSIM [54]	0.728	0.719	0.540	0.763	0.683	0.520
SSIM [23]	0.670	0.671	0.489	0.618	0.616	0.437
SUMMER [55]	0.719	0.723	0.540	0.623	0.622	0.472
SG-ESSIM	0.739	0.838	0.650	0.878	0.805	0.636

and

Table 4. Comparison of SG-ESSIM to several other state-of-the-art algorithms on TID2008 [21] and CSIQ [8]. The highest values are typed in bold, while the second highest ones are underlined.

	TID2008 [21]			CSIQ [8]
FR-IQA Metric	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
2stepQA [50]	0.757	0.769	0.574	0.841	0.849	0.655
CSV [51]	0.852	0.851	0.659	0.933	0.933	0.766
DISTS [52]	0.705	0.668	0.488	0.930	0.930	0.764
ESSIM [19]	0.658	0.876	0.696	0.814	0.933	0.768
GSM [29]	0.782	0.781	0.578	0.906	0.910	0.729
IW-SSIM [16]	0.842	0.856	0.664	0.804	0.921	0.753
MAD [8]	0.831	0.829	0.639	0.950	0.947	0.796
MS-SSIM [26]	0.838	0.846	0.648	0.913	0.917	0.743
PSNR	0.447	0.489	0.346	0.853	0.809	0.599
ReSIFT [53]	0.627	0.632	0.484	0.884	0.868	0.695
RVSIM [54]	0.789	0.743	0.566	0.923	0.903	0.728
SSIM [23]	0.669	0.675	0.485	0.812	0.812	0.606
SUMMER [55]	0.817	0.823	0.623	0.826	0.830	0.658
SG-ESSIM	0.853	0.888	0.708	0.836	0.944	0.786

. From these experimental, numerical results, it could be seen that the proposed SG-ESSIM was able to provide the best results in terms of the SROCC and KROCC on KADID-10k [20] and the PLCC on TID2013 [7], respectively. Moreover, it gave the best results for all correlation values in TID2008 [21]. On CSIQ [8], the second best results in terms of the SROCC and KROCC were achieved by the proposed method. To illustrate the overall performance of the examined methods on these databases, direct and weighted average (using the numbers of images as weights) performance values are summarized in

Table 5. Comparison of SG-ESSIM to several other state-of-the-art algorithms. Direct and weighted average PLCC, SROCC, and KROCC values are presented. Measured on KADID-10k [20], TID2013 [7], TID2008 [21], and CSIQ [8]. The highest values are typed in bold, while the second highest ones are underlined.

	Direct Average			Weighted Average
FR-IQA Metric	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
2stepQA [50]	0.776	0.781	0.587	0.765	0.768	0.572
CSV [51]	0.827	0.825	0.653	0.740	0.738	0.582
DISTS [52]	0.801	0.781	0.601	0.795	0.785	0.599
ESSIM [19]	0.714	0.857	0.681	0.673	0.830	0.647
GSM [29]	0.814	0.815	0.622	0.789	0.789	0.596
IW-SSIM [16]	0.815	0.828	0.635	0.800	0.780	0.570
MAD [8]	0.831	0.820	0.643	0.763	0.758	0.573
MS-SSIM [26]	0.841	0.842	0.656	0.821	0.822	0.633
PSNR	0.599	0.655	0.475	0.520	0.660	0.470
ReSIFT [53]	0.697	0.688	0.530	0.655	0.641	0.483
RVSIM [54]	0.801	0.762	0.589	0.752	0.725	0.549
SSIM [23]	0.692	0.694	0.504	0.668	0.669	0.485
SUMMER [55]	0.746	0.750	0.573	0.720	0.720	0.540
SG-ESSIM	0.827	0.869	0.695	0.783	0.843	0.661

. It could be seen that the proposed method was able to deliver the second best results in terms of the SROCC and KROCC, both in direct and weighted averages. In

Table 6. Comparison of TID2013’s [7] distortion types. SROCC values are given. The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [50]	CSV [51]	DISTS [52]	ESSIM [19]	GSM [29]	MAD [8]	MS-SSIM [26]	ReSIFT [53]	RVSIM [54]	SSIM [23]	SG-ESSIM
AGN	0.817	0.938	0.845	0.911	0.899	0.912	0.624	0.831	0.886	0.848	0.936
ANC	0.590	0.862	0.786	0.806	0.823	0.800	0.387	0.749	0.836	0.779	0.855
SCN	0.860	0.939	0.859	0.938	0.927	0.929	0.683	0.839	0.868	0.851	0.935
MN	0.395	0.748	0.814	0.711	0.704	0.658	0.372	0.702	0.734	0.775	0.715
HFN	0.828	0.927	0.868	0.890	0.884	0.902	0.704	0.869	0.895	0.889	0.920
IN	0.715	0.848	0.674	0.825	0.813	0.743	0.766	0.824	0.865	0.810	0.833
QN	0.886	0.892	0.810	0.904	0.911	0.895	0.720	0.745	0.869	0.817	0.911
GB	0.853	0.933	0.926	0.970	0.954	0.915	0.762	0.937	0.970	0.910	0.969
DEN	0.900	0.952	0.899	0.956	0.955	0.922	0.819	0.907	0.926	0.876	0.963
JPEG	0.867	0.944	0.897	0.923	0.933	0.924	0.784	0.905	0.930	0.893	0.950
JP2K	0.891	0.966	0.931	0.946	0.934	0.929	0.790	0.928	0.946	0.806	0.949
JGTE	0.806	0.800	0.906	0.826	0.866	0.768	0.582	0.712	0.831	0.701	0.823
J2TE	0.854	0.887	0.865	0.902	0.893	0.854	0.742	0.835	0.882	0.813	0.899
NEPN	0.775	0.811	0.833	0.799	0.804	0.803	0.792	0.693	0.771	0.634	0.801
BLOCK	0.044	0.183	0.302	0.649	0.588	−0.322	0.382	0.440	0.545	0.564	0.623
MS	0.660	0.654	0.752	0.712	0.728	0.708	0.732	0.418	0.559	0.738	0.706
CC	0.430	0.227	0.464	0.453	0.466	0.420	0.027	−0.055	0.132	0.355	0.452
CCS	−0.258	0.809	0.789	−0.297	0.676	−0.059	−0.055	−0.209	0.366	0.742	0.010
MGN	0.747	0.884	0.790	0.853	0.831	0.888	0.653	0.765	0.853	0.804	0.900
CN	0.858	0.924	0.907	0.910	0.902	0.904	0.596	0.882	0.914	0.797	0.916
LCNI	0.902	0.965	0.932	0.957	0.945	0.950	0.713	0.897	0.933	0.877	0.952
ICQD	0.808	0.919	0.832	0.904	0.901	0.867	0.739	0.770	0.871	0.820	0.928
CA	0.702	0.845	0.879	0.839	0.835	0.760	0.568	0.838	0.871	0.740	0.835
SSR	0.926	0.976	0.944	0.965	0.961	0.949	0.801	0.944	0.956	0.822	0.964
All	0.733	0.848	0.711	0.797	0.787	0.778	0.785	0.623	0.683	0.616	0.805

and

Table 7. Comparison of TID2008’s [21] distortion types. SROCC values are given. The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [50]	CSV [51]	DISTS [52]	ESSIM [19]	GSM [29]	MAD [8]	MS-SSIM [26]	ReSIFT [53]	RVSIM [54]	SSIM [23]	SG-ESSIM
AGN	0.766	0.922	0.812	0.875	0.855	0.872	0.610	0.771	0.840	0.805	0.913
ANC	0.627	0.893	0.811	0.792	0.821	0.803	0.354	0.762	0.829	0.780	0.900
SCN	0.814	0.932	0.838	0.909	0.904	0.901	0.727	0.810	0.837	0.800	0.920
MN	0.450	0.781	0.830	0.744	0.736	0.673	0.304	0.728	0.760	0.797	0.825
HFN	0.818	0.936	0.870	0.899	0.889	0.894	0.749	0.881	0.886	0.871	0.921
IN	0.659	0.819	0.626	0.777	0.764	0.650	0.767	0.777	0.836	0.776	0.786
QN	0.850	0.894	0.770	0.884	0.903	0.851	0.708	0.730	0.836	0.784	0.873
GB	0.877	0.923	0.909	0.966	0.948	0.896	0.759	0.904	0.963	0.866	0.964
DEN	0.919	0.970	0.931	0.974	0.971	0.928	0.786	0.923	0.939	0.873	0.963
JPEG	0.895	0.948	0.894	0.938	0.937	0.931	0.774	0.914	0.926	0.880	0.959
JP2K	0.910	0.984	0.953	0.966	0.949	0.941	0.837	0.935	0.970	0.745	0.972
JGTE	0.851	0.790	0.907	0.859	0.871	0.781	0.606	0.735	0.860	0.666	0.855
J2TE	0.845	0.852	0.833	0.875	0.880	0.802	0.742	0.778	0.854	0.769	0.863
NEPN	0.803	0.752	0.882	0.742	0.784	0.801	0.749	0.761	0.732	0.588	0.729
Block	0.441	0.770	0.618	0.876	0.843	−0.362	0.765	0.743	0.782	0.804	0.905
MS	0.655	0.594	0.681	0.611	0.638	0.563	0.711	0.322	0.525	0.629	0.683
CC	0.597	0.330	0.649	0.624	0.634	0.548	0.042	−0.018	0.194	0.502	0.642
All	0.769	0.851	0.668	0.876	0.781	0.829	0.846	0.632	0.743	0.675	0.888

, detailed results could be seen on TID2013 [7] and TID2008 [21] with respect to the different distortion types. Similarly,

Table 8. Comparison of SROCC of each FR-IQA metrics on TID2013’s [7] distortion levels (Level 1 represents the lowest level of degradation, while Level 5 represents the highest one). The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [50]	CSV [51]	DISTS [52]	ESSIM [19]	GSM [29]	MAD [8]	MS-SSIM [26]	ReSIFT [53]	RVSIM [54]	SSIM [23]	SG-ESSIM
Level 1	0.246	0.424	0.235	0.388	0.372	0.388	0.166	0.181	0.248	0.204	0.448
Level 2	0.394	0.626	0.440	0.547	0.512	0.368	0.049	0.401	0.430	0.276	0.569
Level 3	0.539	0.635	0.367	0.638	0.523	0.442	0.240	0.415	0.416	0.084	0.660
Level 4	0.571	0.749	0.606	0.766	0.669	0.284	0.172	0.699	0.702	0.208	0.787
Level 5	0.663	0.787	0.664	0.875	0.745	0.308	0.397	0.788	0.803	0.202	0.861
All	0.733	0.848	0.711	0.797	0.787	0.778	0.785	0.623	0.683	0.616	0.805

and

Table 9. Comparison of SROCC for each FR-IQA metric on TID2008’s [21] distortion levels (Level 1 represents the lowest level of degradation, while Level 5 represents the highest one). The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [50]	CSV [51]	DISTS [52]	ESSIM [19]	GSM [29]	MAD [8]	MS-SSIM [26]	ReSIFT [53]	RVSIM [54]	SSIM [23]	SG-ESSIM
Level 1	0.470	0.638	0.566	0.655	0.639	0.432	0.067	0.457	0.634	0.368	0.691
Level 2	0.619	0.683	0.381	0.773	0.636	0.520	0.221	0.437	0.513	0.105	0.807
Level 3	0.573	0.774	0.581	0.826	0.677	0.239	0.059	0.707	0.761	0.190	0.849
Level 4	0.610	0.829	0.628	0.905	0.718	0.232	0.275	0.788	0.825	0.241	0.891
All	0.769	0.851	0.668	0.876	0.781	0.829	0.846	0.632	0.743	0.675	0.888

illustrate the detailed results with respect to the noise levels. TID2013 [7] contained images with 24 different distortion types, such as AGN (additive Gaussian noise), ANC (additive noise in color components), SCN (spatially correlated noise), MN (masked noise), HFN (high frequency noise), IN (impulse noise), QN (quantization noise), GB (Gaussian blur), DEN (image denoising), JPEG (JPEG compression), JP2K (JPEG2000 compression), JGTE (JPEG transmission errors), J2TE (JPEG2000 transmission errors), NEPN (noneccentricity pattern noise), BLOCK (local block-wise distortions of different intensity), MS (mean shift), CC (contrast change), CCS (change in color saturation), MGN (multiplicative Gaussian noise), CN (comfort noise), LCNI (lossy compression of noisy images), ICQD (image color quantization with dither), CA (chromatic aberrations), and SSR (sparse sampling and reconstruction). TID2008 [21] consisted of the first 17 distortion types from TID2013 [7]. From the presented results, it could be observed that SG-ESSIM was able to produce the best SROCC values on five distortion types of TID2013 [7] and on four distortion types of TID2008 [21]. On the other hand, it provided the second best results on nine distortion types of TID2013 [7] and on seven distortion types of TID2008 [21]. Moreover, the SG-ESSIM was able to produce the best SROCC values on three distortion levels of TID2013 [7], while it gave the second best results on the remaining two distortion levels. In TID2008 [21], it provided the best results on three distortion levels and the second best result on the remaining distortion level.

In Figure 3, the execution time (logarithmic scale) versus SROCC scatter plot measured on KADID-10k [20] is depicted. From this figure, it could be seen that the proposed SG-ESSIM method was the fourth fastest algorithm out of the examined thirteen ones. Considering execution time and estimation performance together, the SG-ESSIM provided a competitive result against the state-of-the-art. Similarly, Figure 4 depicts the execution versus the SROCC on the CSIQ [8] database. Here, we could also observe that considering execution time and performance together, the proposed method was able to achieve a competitive result.

6. Conclusions

The goal of FR-IQA is to predict digital images’ perceptual quality with full access to the distortion-free, reference images. FR-IQA methods usually contain two stages: local image quality estimation and pooling. In addition, visual saliency is utilized in the pooling stage as weights for the weighted averaging of local image quality. Unlike this traditional approach, we applied visual saliency in the computation of local image quality, motivated by the fact that local image quality is determined both by local image degradation and visual saliency simultaneously. Experimental results on four publicly available benchmark IQA databases showed that the proposed algorithm was able to outperform the state-of-the-art and was characterized by low computational costs. Future work could involve the incorporation of local and global saliency into a novel FR-IQA metric. The source code of the proposed method is available at: https://github.com/Skythianos/SG-ESSIM (accessed on 12 June 2022).