New Combined Metric for Full-Reference Image Quality Assessment

Abstract

In recent years, many new metrics highly correlated with the Mean Opinion Score (MOS) have been proposed for assessing image quality through Full-Reference Image Quality Assessment (FR-IQA) methods, such as MDSI, HPSI, and GMSD. Eight of these selected metrics, which compare reference and distorted images in a symmetrical manner, are briefly described in this article, and their performance is evaluated using correlation criteria (PLCC, SROCC, and KROCC), as well as RMSE. The aim of this paper is to develop a new, efficient quality index based on a combination of several high-performance metrics already utilized in the field of Image Quality Assessment (IQA). The study was conducted on four benchmark image databases (TID2008, TID2013, KADID-10k, and PIPAL) and identified the three best-performing metrics for each database. The paper introduces a New Combined Metric (NCM), which is a weighted sum of three component metrics, and demonstrates its superiority over each of its component metrics across all the examined databases. An optimization method for determining the weights of the NCM is also presented. Additionally, an alternative version of the combined metric, based on the fastest metrics and employing symmetric calculations for pairs of compared images, is discussed. This version also demonstrates strong performance.

1. Introduction

Every day, a huge number of digital cameras generate a vast stream of images. Due to the multitude of applications of imaging devices in areas such as vision-based quality control of components in manufacturing processes, security monitoring, object detection systems in automotive applications, and analysis of diagnostic images in medicine, there has been a strong increase in the demand for Image Quality Assessment (IQA) methods.

Image quality can be assessed either subjectively or objectively. Subjective methods rely on the perceptual evaluation of image quality by human observers, which means that conducting these assessments incurs significant financial costs and requires a large number of participants. In contrast, objective methods utilize mathematical models to determine the values of various metrics related to image quality. Among Image Quality Assessment (IQA) methods, the most advanced are those that perform a symmetrical comparison between distorted images and their originals, referred to as Full-Reference IQA (FR-IQA) techniques. The scores generated by each IQA metric can be evaluated against subjective assessments, such as the Mean Opinion Score (MOS) or the Difference Mean Opinion Score (DMOS), derived from human viewers. For FR-IQA methods, the correlation coefficients obtained from comparisons with MOS indicate the effectiveness of the metric: the higher the coefficient, the more closely the metric aligns with human perception. For many years, efforts have been made in the field of FR-IQA to improve and refine existing quality metrics. Significant importance is attached to attempts to combine one metric with other quality measures to increase the correlation of the resulting quality index with MOS. Meanwhile, the rapid development of machine-learning and deep-learning techniques provides new methods for image quality assessment. The FR-IQA problem can be understood as a challenge in developing mathematical models that can perceptually assess the image quality in alignment with human judgment.

Over the years, numerous metrics for FR-IQA have been proposed that take various aspects of the Human Visual System (HVS) into account. Recently, attempts have been made to enhance the effectiveness of FR-IQA by combining existing metrics to create a “super” index. Theoretical foundations for such metric fusion can be found in Liu’s work [1], where it is applied to very old and classical metrics such as PSNR, VSNR, SSIM, and VIF. In the paper by Okarma [2], the properties of three FR-IQA metrics (MS-SSIM, VIF, and R-SVD) were analyzed, and a combined quality metric based on their product was proposed. It is named the Combined Quality Metric (CQM), and its three correlation coefficients increased in relation to the correlation coefficients with MOS of individual product multipliers.

Later, this concept was further developed using optimization or regression techniques to determine the optimal weights or exponents in the products of existing FR-IQA indices [3,4]. The Combined Image Similarity Index (CISI) proposed by Okarma in [3] employs metrics similar to those used in the CQM. However, instead of the R-SVD metric, it utilizes the FSIMc metric. The CISI index demonstrates a higher correlation with Mean Opinion Score (MOS) compared to the CQM index. In the 2013 article by Okarma [4], a new EHIS metric is introduced, which is based on the product of four multipliers: two familiar from CQM (MS-SSIM and VIF) and two new ones (WFSIMc and RFSIM). This approach improves the correlation with MOS over the combined CISI metric.

Another metrics fusion strategy was proposed by the author of [5], who presented several versions of a linear combination (weighted sum) based on metrics selected from a dozen FR-IQA metrics. He referred to these new combined metrics as Linearly Combined Similarity Measures (LCSIM). The use of linear combination metrics requires determining the weighting factors for the FR-IQA metrics, which is achieved by solving the RMSE error minimization task using a genetic algorithm.

Another line of work in creating methods based on the fusion of metrics utilizes machine-learning techniques. One example can be seen in [6], where the results of six traditional FR-IQA metrics (FSIMc, PSNR-HMA, PSNR-HVS, SFF, SR-SIM, and VIF) were used as a feature vector for training and testing a four-layer neural network. The output results produced by the neural network demonstrate a significant improvement over those achieved by the input metrics. Currently, deep neural networks, particularly CNNs, can learn the best combinations of metrics to optimize image quality assessment [7,8].

Combining metrics leads to increased computational complexity, which can be an issue in the context of real-time applications. Nevertheless, the fusion of quality metrics for FR-IQA is a promising research direction that could significantly improve the correlation of objective quality assessments with MOS.

In this paper, we consider a new combined metric for FR-IQA, based on metrics that are highly correlated with MOS and well-known from the literature. The structure of this paper is as follows: following the introduction, Section 2 provides an overview of eight relatively new and promising FR-IQA metrics. Section 3 presents their linear combination, along with the experimental results. Finally, Section 4 concludes the paper.

2. Overview of Highly Correlated FR-IQA Metrics

2.1. Feature SIMilarity Index FSIMc (Color Version)

In [9], the Feature SIMilarity Index (FSIM) was introduced as a quality assessment metric for grayscale images, along with its color version, FSIMc. The local quality of the assessed image, $f_2$, when symmetrically compared to the reference image, $f_1$, is represented by two low-level similarity maps derived from phase congruences ($P{C}_1$, $P{C}_2$) [10]:

$$S_{PC}(x,y)={\displaystyle \frac{2P{C}_1(x,y)·P{C}_2(x,y)+T_1}{P{C}_1^2(x,y)+P{C}_2^2(x,y)+T_1}},$$

(1)

and gradient magnitudes ($G_1$, $G_2$):

$$S_G(x,y)={\displaystyle \frac{2G_1(x,y)·G_2(x,y)+T_2}{G_1^2(x,y)+G_2^2(x,y)+T_2}},$$

(2)

where $G_1$ and $G_2$ are Scharr gradient operators and $T_1$ and $T_2$ are positive constants introduced to enhance the stability of the formulas. Phase Congruence (PC) quantifies the presence and intensity of local features, including edges, corners, and textures. It is derived from the analysis of phase information in the frequency domain of the image (specifically, the phase of its Fourier transform). As an additional weighting factor for the similarity maps, the following $P{C}_m$ is used:

$$P{C}_m(x,y)=max(P{C}_1(x,y),P{C}_2(x,y)).$$

(3)

The final expression for the proposed image quality index is given by:

$$FSIM={\displaystyle \frac{{\sum }_{(x,y)\in \Omega }{S}_{PC}(x,y)·S_G(x,y)·P{C}_m(x,y)}{{\sum }_{(x,y)\in \Omega }P{C}_m(x,y)}}.$$

(4)

FSIMc, the color extension of FSIM, incorporates the chrominance components I and Q. The calculation of the index values starts by decomposing the compared images, $f_1$ and $f_2$, into their YIQ color components, where Y represents luminance and I and Q represent chrominance. Similar to the earlier defined similarity maps, additional similarity maps for the I and Q components were introduced:

$$S_I(x,y)={\displaystyle \frac{2I_1(x,y)·I_2(x,y)+T_3}{I_1^2(x,y)+I_2^2(x,y)+T_3}},$$

(5)

$$S_Q(x,y)={\displaystyle \frac{2Q_1(x,y)·Q_2(x,y)+T_4}{Q_1^2(x,y)+Q_2^2(x,y)+T_4}},$$

(6)

where $T_3$ and $T_4$ are positive constants. The overall chrominance similarity map is given by:

$$S_C(x,y)=S_I(x,y)·S_Q(x,y).$$

(7)

The inclusion of chromatic components in the FSIM index results in the following version of the formula for color images:

$$FSIMc={\displaystyle \frac{{\sum }_{(x,y)\in \Omega }{S}_{PC}(x,y)·S_G(x,y)·{\left[S_C(x,y)\right]}^{\lambda }·P{C}_m(x,y)}{{\sum }_{(x,y)\in \Omega }P{C}_m(x,y)}},$$

(8)

where the positive value of the $\lambda$ exponent highlights the significance of chrominance in the color image quality assessment process. For subsequent studies utilizing FSIMc, the following parameter values were employed, as specified in [9]: $T_1=0.85$, $T_2=160$, $T_3=T_4=200$, and $\lambda =0.03$.

2.2. Mean Deviation Similarity Index MDSI

Many IQA metrics work as follows: they determine local distortions in the images, build similarity maps, and implement a pooling strategy based on the mean, weighted mean, standard deviation, etc. An example of this approach to IQA index modeling is the Mean Deviation Similarity Index (MDSI), described in [11]. The calculation of MDSI starts with converting the RGB color space components of the input images to a luminance component:

$$L=0.2989R+0.5870G+0.1140B$$

(9)

and two chromaticity components:

$$\begin{array}{c}\hfill \left[\begin{array}{c}H\\ M\end{array}\right]=\left[\begin{array}{ccc}0.30& 0.04& -0.35\\ 0.34& -0.6& 0.17\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right].\end{array}$$

(10)

This index is derived from the calculation of gradient similarity ($GS$) for structural distortions and chromaticity similarity ($CS$) for color distortions. The local structural similarity map is typically computed using gradient values. Traditionally, structural similarity maps are obtained by calculating the gradient values separately for the original and distorted images. The traditional approach for the MDSI metric has been improved by integrating the gradient value map, which combines the luminance channel values from both images:

$$f=0.5(L_r+L_d),$$

(11)

where f represents the fused luminance image, r is the reference image, and d refers to the distorted image. The formulas for the proposed structural similarity are given below:

$$\begin{array}{c}\hfill G{S}_{rf}(x)={\displaystyle \frac{2G_r(x)G_f(x)+C_2}{G_r^2(x)+G_f^2(x)+C_2}},\end{array}$$

(12)

$$\begin{array}{c}\hfill G{S}_{df}(x)={\displaystyle \frac{2G_d(x)G_f(x)+C_2}{G_d^2(x)+G_f^2(x)+C_2}},\end{array}$$

(13)

$$\widehat{GS}(x)=GS(x)+[G{S}_{df}(x)-G{S}_{rf}(x)].$$

(14)

The gradient magnitude is calculated using the simple Prewitt operator. Additionally, the authors of the MDSI index have adjusted the method for evaluating local chromaticity similarity. In contrast to the previously discussed IQA metrics, such as FSIM or VSI, which assess chromaticity separately for the two chrominance components, this approach combines them in a different way. In the case of MDSI, it was suggested to calculate the color similarity for both chrominance components simultaneously, using the following formula:

$$\widehat{CS}(x)={\displaystyle \frac{2(H_r(x)H_d(x)+M_r(x)M_d(x))+C_3}{H_r^2(x)+H_d^2(x)+M_r{(x)}^2+M_d{(x)}^2+C_3}},$$

(15)

where $C_3$ is a constant introduced for numerical stability. The joint color similarity map, $\widehat{CS}(x)$, is then combined with the $\widehat{GS}(x)$ map using a weighted mean:

$$\widehat{GCS}(x)=\alpha \widehat{GS}(x)+(1-\alpha )\widehat{CS}(x),$$

(16)

where $\alpha$ determines the relative importance of the similarity maps $\widehat{GS}(x)$ and $\widehat{CS}(x)$. The final step involves converting the resulting $\widehat{GCS}$ map into an MDSI score through a pooling strategy based on a specific deviation method:

$$MDSI={\left[{\displaystyle \frac{1}{N}}\sum _{i=1}^N|{\widehat{GCS}}_i^{1/4}-({\displaystyle \frac{1}{N}}\sum _{i=1}^N|{\widehat{GCS}}_i^{1/4})|\right]}^{1/4}.$$

(17)

The original article [11] provides suggestions for selecting various parameters that influence the performance of the MDSI index.

2.3. Haar Wavelet Perceptual Similarity Index HPSI

The Haar Wavelet-based Perceptual Similarity Index (HPSI) [12] is a relatively novel and computationally efficient similarity metric for FR-IQA. HPSI uses coefficients obtained from Haar wavelet decomposition for assessing local similarities between two images. The one-dimensional Haar filters are given by:

$$h_1^{1D}={\displaystyle \frac{1}{\sqrt{2}}}·[1,1],$$

(18)

$$g_1^{1D}={\displaystyle \frac{1}{\sqrt{2}}}·[-1,1],$$

(19)

where $h_1^{1D}$ represents the low-pass scaling filter and $g_1^{1D}$ refers to the corresponding high-pass wavelet filter. For any scale, $j\in \mathbb{N}$, two-dimensional Haar filters can be constructed as follows:

$$g_j^{(1)}=g_1^{1D}\otimes h_1^{1D},$$

(20)

$$g_j^{(2)}=h_1^{1D}\otimes g_1^{1D},$$

(21)

where the symbol ⊗ denotes the outer product, and the one-dimensional filters $h_j^{1D}$ and $g_j^{1D}$ for $j>1$ are defined as:

$$g_j^{1D}=h_1^{1D}\ast {(g_{j-1}^{1D})}_{\uparrow 2},$$

(22)

$$h_j^{1D}=h_1^{1D}\ast {(h_{j-1}^{1D})}_{\uparrow 2},$$

(23)

where $\uparrow 2$ is the dyadic upsampling operator and ∗ denotes the one-dimensional convolution operator. To effectively predict the perceptual similarity perceived by human viewers, it may be beneficial to apply an additional nonlinear mapping to the local similarities derived from the high-frequency Haar wavelet filter responses. This nonlinearity is represented by a logistic function, defined with a parameter $\alpha >0$, as follows:

$$l_{\alpha }(x)={\displaystyle \frac{1}{1+e^{-\alpha x}}}.$$

(24)

For two grayscale images, $f_1$ and $f_2$, the local similarity measure employed to calculate the HPSI is derived from the first two steps of the two-dimensional discrete Haar wavelet transform, as expressed by the following formula:

$$H{S}_{f_1,f_2}^{(k)}\left[x\right]=l_{\alpha }({\displaystyle \frac{1}{2}}\sum _{j=1}^{2}S(|(g_j^{(k)}\ast f_1)\left[x\right]|,|(g_j^{(k)}\ast f_2)\left[x\right]|,C)),$$

(25)

where $C>0$, $k\in \left\{1,2\right\}$ selects either horizontal or vertical Haar wavelet filters, S denotes the similarity measure, and ∗ is the two-dimensional convolution operator. Similar to FSIMc, HPSI also applies a specific weighting map, which is derived here from the response of a single low-frequency Haar wavelet filter:

$$W_f^{(k)}\left[x\right]=|(g_3^{(k)}\ast f)\left[x\right]|,$$

(26)

where $k\in \left\{1,2\right\}$ again differentiates between horizontal and vertical filters. The final expression for the HPSI for grayscale images $f_1$ and $f_2$ is provided as a weighted average of the local similarity map, $H{S}_{f_1,f_2}^{(k)}$:

$$HPS{I}_{f_1,f_2}=l_{\alpha }^{-1}{({\displaystyle \frac{{\displaystyle \sum _x}{\displaystyle \sum _{k=1}^2}H{S}_{f_1,f_2}^{(k)}\left[x\right]·W_{f_1,f_2}^{(k)}\left[x\right]}{{\displaystyle \sum _x}{\displaystyle \sum _{k=1}^2}{W}_{f_1,f_2}^{(k)}\left[x\right]}})}^2,$$

(27)

where:

$$W_{f_1,f_2}^{(k)}\left[x\right]=max(W_{f_1}^{(k)}\left[x\right],W_{f_2}^{(k)}\left[x\right]).$$

(28)

HPSI can be extended for color images in the YIQ color space using a third local similarity map based on the chrominance components I and Q. This map, $H{S}_{f_1,f_2}^{(3)}$, is defined as:

$$H{S}_{f_1,f_2}^{(3)}\left[x\right]=l_{\alpha }({\displaystyle \frac{1}{2}}(S(|(m\ast f_1^I)\left[x\right]|,|(m\ast f_2^I)\left[x\right]|,C)+S(|(m\ast f_1^Q)\left[x\right]|,|(m\ast f_2^Q)\left[x\right]|,C))),$$

(29)

where m is a $2\times 2$ mean filter and then:

$$W_{f_1^Y,f_2^Y}^{(3)}\left[x\right]={\displaystyle \frac{1}{2}}(W_{f_1^Y,f_2^Y}^{(1)}\left[x\right]+W_{f_1^Y,f_2^Y}^{(2)}\left[x\right]).$$

(30)

The final form of the HPSI for color images is defined as follows:

$$HPSI{c}_{f_1,f_2}=l_{\alpha }^{-1}{({\displaystyle \frac{{\displaystyle \sum _x}{\displaystyle \sum _{k=1}^3}H{S}_{f_1,f_2}^{(k)}\left[x\right]·W_{f_1^Y,f_2^Y}^{(k)}\left[x\right]}{{\displaystyle \sum _x}{\displaystyle \sum _{k=1}^3}{W}_{f_1^Y,f_2^Y}^{(k)}\left[x\right]}})}^2.$$

(31)

The fast computation time of the HPSI may explain its high usefulness in various tasks.

2.4. Visual Saliency with Color Appearance and Gradient Similarity Index VCGS

The VCGS index [13] uses color space CIELAB and combines three feature similarity maps: visual salience with color appearance similarity map, $S_{VC}$, gradient similarity map, $S_G$, and chrominance similarity map, $S_C$. The first of these maps is calculated using a formula based on visual saliency with color appearance (VC) for both images:

$$S_{VC}={\displaystyle \frac{2V{C}_1·V{C}_2+K_{VC}}{V{C}_1^2+V{C}_2^2+K_{VC}}},$$

(32)

where $K_{VC}$ is a small constant that controls the numerical stability of the formula. The gradient similarity map using the Scharr operator applied to the L component is calculated according to the formula:

$$S_G={\displaystyle \frac{2G_1·G_2+K_G}{G_1^2+G_2^2+K_G}},$$

(33)

where $K_G$ is a small constant that controls the numerical stability of the formula. The third map measures the similarity of the $a^{*}$ and $b^{*}$ chrominance components in the CIELAB color space and is given by the formula:

$$S_C={\displaystyle \frac{2a_1·a_2+K_C}{a_1^2+a_2^2+K_C}}·{\displaystyle \frac{2b_1·b_2+K_C}{b_1^2+b_2^2+K_C}},$$

(34)

where $K_C$ is a small constant that controls the numerical stability of the formula. The final form of the VCGS metric is given by the following formula:

$$\mathrm{VCGS}={\displaystyle \frac{{\displaystyle \sum _{\Omega }}{S}_{VC}·{\left(S_G\right)}^{\alpha }·{\left(S_C\right)}^{\lambda }·V{C}_m}{{\displaystyle \sum _{\Omega }}V{C}_m}},$$

(35)

where $\Omega$ is the spatial domain, $V{C}_m=max(V{C}_1,V{C}_2)$ is used to weight the relevance of two maps in overall similarity, and $\alpha$ and $\lambda$ represent the relative importance of the similarity maps depending on where they occur.

2.5. SuperPixel SIMilarity Index SPSIM

Superpixel-based SIMilarity (SPSIM) [14] utilizes superpixel segmentation for feature extraction. Superpixels are clusters of neighboring pixels that share similar characteristics, such as color, intensity, or structure. This pixel grouping results in a mosaic consisting of a significantly smaller number of superpixels, which facilitates faster subsequent processing. A key advantage of using superpixel-based segmentation over other oversegmentation algorithms is the ability to predefine the number of generated superpixels. Additionally, superpixel segmentation improves the distinction of perceptually significant regions in the image. Among the various superpixel generation algorithms, we can distinguish between graph-based, gradient-based, clustering-based, and watershed-based methods, among others [15]. The shape and size of superpixels depend on the applied algorithm, with each pixel belonging to exactly one superpixel. These algorithms control the number and properties of the superpixels, such as compactness and minimum size. One of the most popular and efficient algorithms for superpixel segmentation is the k-means-based Simple Linear Iterative Clustering (SLIC) algorithm [16]. This algorithm is notable for producing superpixels with a consistent shape and size. A key benefit of SLIC is that segmentation only requires specifying the desired number of superpixels in the output image. Consequently, the SLIC algorithm is used in the SPSIM quality index discussed in this paper. For each superpixel, the algorithm calculates the mean CIELAB color values and the Local Binary Pattern (LBP) features. Superpixels are initially generated on the reference image and then applied to both the reference and distorted images.

The SPSIM index calculation algorithm relies on pixel gradient similarity and luminance-chrominance superpixel similarity. The YUV color space, rather than RGB, is utilized for SPSIM computation, where Y represents luminance and U and V denote chrominance components. If $s_i$ is used to represent a superpixel containing pixel i, the following formulas can be written for luminance $L_i$ and luminance similarity $M_L(i)$:

$$L_i={\displaystyle \frac{1}{\left|s_i\right|}}\sum _{j\in s_i}Y(j),M_L\left(i\right)={\displaystyle \frac{2L_r\left(i\right)L_d\left(i\right)+T_1}{L_r^2\left(i\right)+L_d^2(i)+T_1}},$$

(36)

where $Y(j)$ represents the luminance of pixel j and $L_r(i)$ and $L_d(i)$ denote the average luminance values for superpixel $s_i$ in the reference and distorted images, respectively. $T_1$ is a positive constant introduced to prevent instability in the equation. Similar expressions can be formulated for both the U and V chrominance components:

$$U_i={\displaystyle \frac{1}{\left|s_i\right|}}\sum _{j\in s_i}U\left(j\right),M_U\left(i\right)={\displaystyle \frac{2U_r\left(i\right)U_d\left(i\right)+T_1}{U_r^2\left(i\right)+U_d^2\left(i\right)+T_1}},$$

(37)

$$V_i={\displaystyle \frac{1}{\left|s_i\right|}}\sum _{j\in s_i}V\left(j\right),M_U\left(i\right)={\displaystyle \frac{2V_r\left(i\right)V_d\left(i\right)+T_1}{V_r^2\left(i\right)+V_d^2\left(i\right)+T_1}}.$$

(38)

The chrominance similarity, $M_C$, can then be calculated as shown below:

$$M_C(i)=M_U(i)M_V(i).$$

(39)

The gradient similarity, $M_G$, is described by the following formula:

$$M_G\left(i\right)={\displaystyle \frac{2G_r\left(i\right)G_d\left(i\right)+T_2}{G_r^2\left(i\right)+G_d^2\left(i\right)+T_2}},$$

(40)

where the gradient magnitude, G, is composed of two components calculated using a simple Prewitt operator, and $T_1$ and $T_2$ are constants selected by the authors of the algorithm to account for contrast-related errors. Further information on the determination of $T_1$ and $T_2$ can be found in [14]. The formula for calculating the similarity of superpixel i in both images is as follows:

$$M\left(i\right)=M_G\left(i\right){\left[M_L\left(i\right)\right]}^{\alpha }{e}^{\beta (M_C(i)-1)},$$

(41)

where the parameters $\alpha$ and $\beta$ represent the weights for the luminance and chrominance components, respectively. Finally, the SPSIM index is calculated as a weighted sum of $M(i)$ and the corresponding weights, which are determined based on the texture complexity ($TC$), described by the standard deviation ($std$) and kurtosis ($Kurt$) of the superpixels:

$$T{C}_r(i)={\displaystyle \frac{std(S_r(i))}{Kurt\left[S_r(i)\right]+3}}, T{C}_d(i)={\displaystyle \frac{std(S_d(i))}{Kurt\left[S_d(i)\right]+3}},$$

(42)

$$w(i)=exp(0.05·abs(T{C}_d(i)-T{C}_r(i))),$$

(43)

$$SPSIM={\displaystyle \frac{{\sum }_{i=1}^{N}M(i)w(i)}{{\sum }_{i=1}^{N}w(i)}},$$

(44)

where $S_r(i)$ and $S_d(i)$ are, respectively, the superpixels in the reference and distorted images that contain the i-th pixel.

2.6. Local Global Variation Index LGV and Saliency Weighted Local Global Variation Index SWLGV

Varga [17] introduced new quality indices that utilize both gradients in the image and Grünwald–Letnikov fractional derivatives. While gradients capture local variations within the image, fractional derivatives describe global variations, represented by the flowing global similarity map:

$$S_G(x,y)={\displaystyle \frac{2·{}^{GL}D^{\alpha }R(x,y)·{}^{GL}D^{\alpha }D(x,y)+c_1}{{\left({}^{GL}D^{\alpha }R(x,y)\right)}^2+{\left({}^{GL}D^{\alpha }D(x,y)\right)}^2+c_1}},$$

(45)

where $R(x,y)$ is the reference image, $D(x,y)$ is the distorted image, ${}^{GL}D^{\alpha }$ is the $\alpha$-order Grünwald–Letnikov fractional derivative, and $c_1$ is a constant number that provides numerical stability. The order of the fractional derivative was set to $\alpha =0.6$. The $3\times 3$ Scharr operator was used to compute the gradients for the local gradient map, $S_L(x,y)$:

$$S_L(x,y)={\displaystyle \frac{2·G_R(x,y)·G_D(x,y)+c_2}{G_R^2(x,y)+G_D^2(x,y)+c_2}},$$

(46)

where $c_2$ is a constant that ensures numerical stability. The similarity map between the two compared images was calculated using the two previously defined gradient maps and the exponential coefficient $\lambda =0.7$:

$$S(x,y)={(S_G(x,y))}^{\lambda }·{(S_L(x,y))}^{1-\lambda }.$$

(47)

Finally, the similarity map obtained is fused with the saliency map. This index is referred to as the Local Global Variation (LGV):

$$LGV={\displaystyle \frac{1}{M·N}}\sum _{x=1}^M\sum _{y=1}^{N}S(x,y),$$

(48)

where $M·N$ is the resolution of images.

The SWLGV index, in contrast to LGV, also incorporates the mechanism of visual saliency. It emphasizes the differences between the reference and distorted images in the most distinctive regions. By labeling the maps of the distinguishing regions as $S{M}_R(x,y)$ for the reference image and $S{M}_D(x,y)$ for the distorted image, we can create a formula for the image pair:

$$SM(x,y)=max(S{M}_R(x,y),S{M}_D(x,y)),$$

(49)

where $S{M}_R(x,y)$ and $S{M}_D(x,y)$ are visual saliency maps built as proposed in [18]. The SWLGV index is defined as the weighted average of $S(x,y)$ and $SM(x,y)$, where $SM(x,y)$ represents the weights:

$$SWLGV={\displaystyle \frac{{\displaystyle \sum _{i=1}^M}{\displaystyle \sum _{j=1}^N}SM(x,y)·S(x,y)}{{\displaystyle \sum _{i=1}^M}{\displaystyle \sum _{j=1}^N}SM(x,y)}}.$$

(50)

2.7. Gradient Magnitude Similarity Deviation Index GMSD

GMSD [19] is a relatively simple metric, which is based on a gradient similarity map and uses a $3\times 3$ Prewitt filter. The magnitudes of the gradients of images r and d at position i, denoted by $m_r(i)$ and $m_d(i)$, are calculated as follows:

$$m_r(i)=\sqrt{{(r\otimes h_x)}^2(i)+{(r\otimes h_y)}^2(i)},$$

(51)

$$m_d(i)=\sqrt{{(d\otimes h_x)}^2(i)+{(d\otimes h_y)}^2(i)},$$

(52)

where ⊗ denotes a convolution operation. The magnitude gradient similarity map, $GMS(i)$, is then calculated as follows:

$$GMS(i)={\displaystyle \frac{2m_r(i)m_d(i)+c}{m_r^2(i)+m_d^2(i)+c}},$$

(53)

where c is a constant number that provides numerical stability. The formulas above demonstrate a symmetrical approach to both referenced and distorted images. The average gradient value from the $GMS(i)$ map was then determined as:

$$GMSM={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}GMS(i),$$

(54)

where N is a number of pixels in image. Finally, the GMSD index is defined by the formula:

$$GMSD=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(GMS(i)-GMSM)}^2}.$$

(55)

2.8. Evaluation Criteria for IQA

Individual IQA metrics are commonly compared with the subjective ratings of specific images. To assess the linearity, monotonicity, and accuracy of these predictions, four criteria are used: the Pearson Linear Correlation Coefficient (PLCC), the Spearman Rank Order Correlation Coefficient (SROCC), the Kendall Rank Order Correlation Coefficient (KROCC), and the Root Mean Squared Error (RMSE). The formulas for these comparisons are provided below:

$$PLCC={\displaystyle \frac{{\sum }_{i=1}^N(p_i-\overline{p})(s_i-\overline{s})}{\sqrt{{\sum }_{i=1}^N{(p_i-\overline{p})}^2{(s_i-\overline{s})}^2}}},$$

(56)

where $p_i$ and $s_i$ represent the raw values of the subjective and objective measures, respectively, and $\overline{p}$ and $\overline{s}$ are the mean values of the subjective and objective measures.

Spearman Rank Order Correlation Coefficient is given by the formula:

$$SROCC=1-{\displaystyle \frac{6{\sum }_{i=1}^N{d}_i^2}{N(N^2-1)}},$$

(57)

where $d_i$ represents the difference between the ranks of both measures for the i-th observation and N is the total number of observations.

Kendall Rank Order Correlation Coefficient (KROCC) is provided by the formula:

$$KROCC={\displaystyle \frac{N_c-N_d}{0.5(N-1)N}},$$

(58)

where $N_c$ and $N_d$ denote the counts of concordant and discordant pairs.

Root Mean Squared Error (RMSE) is given by the following equation:

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(p_i-s_i)}^2},$$

(59)

where $p_i$ and $s_i$ are defined as above.

The above correlation coefficients are useful tools for objectively assessing the agreement between IQA computational models and subjective MOS assessments. However, they capture specific aspects of this relationship, such as linearity in the case of PLCC or monotonicity in the cases of SROCC and KROCC. SROCC and KROCC are suitable for scenarios where the relationship between variables is nonlinear, with KROCC offering high robustness to small changes in the data. RMSE, on the other hand, is a measure of error primarily used to evaluate the accuracy of a model’s predictions. Unlike PLCC, SROCC, and KROCC, it is not a measure of correlation, and its role is fundamentally different. RMSE quantifies the average distance between predicted and actual values. It is sensitive to the magnitude of errors because the differences are squared before averaging, meaning that larger deviations have a disproportionate impact on the final value. High RMSE values indicate poor agreement between predicted and actual values, but RMSE does not provide information about the type of relationship (e.g., whether it is monotonic, linear, or otherwise).

As recommended in [20], a nonlinear mapping was applied to calculate PLCC and RMSE. This process involves the use of a fitting function, usually a logistic function with five beta parameters, ${\beta }_1, {\beta }_2, {\beta }_3, {\beta }_4, {\beta }_5$, to better represent the relationship between predicted performance, x, and MOS.

$$p(x,\beta )={\beta }_1({\displaystyle \frac{1}{2}}-{\displaystyle \frac{1}{1+exp({\beta }_2(x-{\beta }_3))}})+{\beta }_{4}x+{\beta }_5.$$

(60)

3. The New Combined Metric (NCM) and Its Experimental Research

The New Combined Metric (NCM):

$$NCM=\alpha ·M_1+\beta ·M_2+\gamma ·M_3,$$

(61)

where $M_1$, $M_2$, and $M_3$ are the selected FR-IQA metrics for given dataset and $\alpha$, $\beta$, and $\gamma$ are the optimized weights.

3.1. Selected IQA Databases

Four benchmark databases, TID2008 [21], TID2013 [22], KADID-10k [23], and PIPAL [24], were chosen for the research. These databases are distinguished by a large set of reference images, diverse distortion types, and varying levels of their presence in the images. For each image in the databases, Mean Opinion Scores (MOSs) are experimentally gathered by collecting assessments from multiple human observers.

The TID2008 image database consists of 1700 distorted images, generated using 17 different distortion types, each applied at four levels to 25 reference images (Figure 1). MOS was provided based on the work of 838 human observers and 256,428 comparisons. All images have a resolution of 512 × 384 pixels.

Figure 1. Reference images of the TID2008 and TID2013 databases [21].

The TID2013 image database is an updated and expanded version of TID2008. It retains the same set of reference images (Figure 1), but the number of distortion types has been increased to 24, and the distortion levels have been raised to five. The database includes 3000 distorted digital images. Additionally, the size of the research group from which the average subjective ratings were derived has been enlarged. MOS ratings were collected from 524,340 comparisons made by 971 observers. The image resolution remains unchanged.

Online crowdsourcing for image assessment has enabled the creation of larger databases. One such large database, KADID-10k (Konstanz Artificially Distorted Image Quality Database) [23], contains 10,125 digital images with subjective quality scores (MOSs). It was developed and published by 2209 crowd workers. This database includes a limited selection of reference images (81) (Figure 2), a restricted number of artificial distortion types (25), and five levels for each distortion type. Recently, KADID-10k has become widely used for deep-learning models for image quality assessment [25]. The artificial distortions present in the KADID-10k database include spatial distortions, noise, blurs, and more. The image resolution remains unchanged.

Figure 2. Reference images of the KADID-10k database [23].

PIPAL is a large IQA dataset, first introduced in 2020 by [24], that increased the number of reference images to 250. In fact, these are 288 × 288 fragments from images in the DIV2K and Flickr2K high-resolution image collections (Figure 3), with distortion types increased to 40 and distorted images increased to 29,000, and it contains 1,130,000 human ratings. In this image database, the Elo rating system was used to assign the Mean Opinion Scores (MOSs). Currently, the PIPAL dataset is used in many challenges as a benchmark for IQA algorithms.

Figure 3. Examples of reference images from the PIPAL database [24].

The key information regarding the selected IQA benchmark databases is presented in Table 1.

Table 1. Comparison of the selected IQA databases.

		No. of	No. of		No. of Dist.
Database	Year	Ref.	Dist.	Environment	Images
TID2008	2008	25	17	lab	1700
TID2013	2013	25	24	lab	3000
KADID-10k	2019	81	25	crowdsourcing	10,125
PIPAL	2020	250	40	crowdsourcing	29,000

3.2. Experimental Tests

The experimental study began with the determination of the PLCC, SROCC, and KROCC correlation coefficients, and RMSE values for eight selected highly correlated metrics. The results of these tests for the four study datasets are included in Table 2. The three highest correlation coefficients and the three lowest RMSE values are shown in different colors (the best results in red, the second results in green, and the third results in blue).

Table 2. Values of correlation coefficients and RMSE for FR-IQA metrics.

Database	Metric	FSIMc [9]	MDSI [11]	HPSI [12]	VCGS [13]	SPSIM [14]	LGV [17]	SWLGV [17]	GMSD [19]
TID2008	PLCC	0.876	0.916	0.907	0.878	0.893	0.865	0.874	0.879
	SROCC	0.884	0.921	0.910	0.897	0.910	0.881	0.889	0.891
	KROCC	0.699	0.751	0.737	0.717	0.730	0.696	0.711	0.709
	RMSE	0.647	0.538	0.566	0.643	0.605	0.674	0.652	0.640
TID2013	PLCC	0.877	0.909	0.893	0.900	0.909	0.778	0.797	0.855
	SROCC	0.851	0.890	0.873	0.893	0.904	0.807	0.807	0.804
	KROCC	0.667	0.712	0.692	0.717	0.725	0.638	0.641	0.634
	RMSE	0.596	0.518	0.557	0.541	0.517	0.779	0.749	0.642
KADID-10k	PLCC	0.851	0.864	0.885	0.868	0.874	0.815	0.835	0.805
	SROCC	0.854	0.885	0.885	0.871	0.874	0.820	0.840	0.847
	KROCC	0.665	0.702	0.699	0.683	0.687	0.630	0.655	0.664
	RMSE	0.568	0.544	0.505	0.538	0.525	0.627	0.595	0.643
PIPAL	PLCC	0.615	0.598	0.641	0.554	0.578	0.529	0.543	0.629
	SROCC	0.589	0.585	0.589	0.534	0.562	0.519	0.536	0.583
	KROCC	0.416	0.408	0.417	0.370	0.391	0.359	0.372	0.414
	RMSE	0.104	0.106	0.101	0.110	0.108	0.112	0.111	0.103

The three correlation coefficients and the RMSE value were aggregated into one score. A point scale with values from 1 to 8 was adopted, where the highest points were awarded to the highest correlation coefficients and the lowest RMSE values. This ranking is shown in Table 3, where the point values for each dataset are also summarized in the columns. The number of points determined the three component metrics for each dataset. The three highest scores are highlighted in bold.

Table 3. Ranking of FR-IQA metrics.

Database	Metric	FSIMc [9]	MDSI [11]	HPSI [12]	VCGS [13]	SPSIM [14]	LGV [17]	SWLGV [17]	GMSD [19]
TID2008	PLCC	3	8	7	4	6	1	2	5
	SROCC	2	8	6	5	7	1	3	4
	KROCC	2	8	7	5	6	1	4	3
	RMSE	3	8	7	4	6	1	2	5
	TOTAL	10	32	27	18	25	4	11	17
TID2013	PLCC	4	7	5	6	8	1	2	3
	SROCC	4	6	5	7	8	3	2	1
	KROCC	4	6	5	7	8	2	3	1
	RMSE	4	7	5	6	8	1	2	3
	TOTAL	16	26	20	26	32	7	9	8
KADID-10k	PLCC	4	5	8	6	7	2	3	1
	SROCC	4	8	7	5	6	1	2	3
	KROCC	4	8	7	5	6	1	2	3
	RMSE	4	5	8	6	7	2	3	1
	TOTAL	16	26	30	22	26	6	10	8
PIPAL	PLCC	6	5	8	3	4	1	2	7
	SROCC	8	6	7	2	4	1	3	5
	KROCC	7	5	8	2	4	1	3	6
	RMSE	6	5	8	3	4	1	2	7
	TOTAL	27	21	31	10	16	4	10	25

The three metrics selected from the table served as the components $M_1$, $M_2$, and $M_3$ for the linear combination that determines the New Combined Metric, as defined in Formula (61).

Determining the NCM value requires calculating the $\alpha$, $\beta$, and $\gamma$ weights present in this formula. These weights are optimized in the Matlab environment using the $fmincon$ function. In the optimization task, the PLCC linear correlation coefficient is maximized. The obtained values of the weights for each dataset are given in Table 4. Based on these weights and component metrics, the values of the combined NCM metric were determined. The results are shown in Table 5. The results for the three component metrics are shown in red, while green is used for the best score achieved by the combined NCM metric in each case.

Table 4. Optimized weight values for the NCM metric.

Criterion:		Three Best		Three Fast
Database	Weight	Value	Metrics	Value	Metrics
TID2008	$\alpha$	0.578	MDSI [11]	0.618	MDSI [11]
	$\beta$	0.285	HPSI [12]	0.372	HPSI [12]
	$\gamma$	0.136	SPSIM [14]	0.010	GMSD [19]
TID2013	$\alpha$	0.459	MDSI [11]	0.680	MDSI [11]
	$\beta$	0.086	VCGS [13]	0.310	HPSI [12]
	$\gamma$	0.455	SPSIM [14]	0.010	GMSD [19]
KADID-10k	$\alpha$	0.386	MDSI [11]	0.342	MDSI [11]
	$\beta$	0.404	HPSI [12]	0.486	HPSI [12]
	$\gamma$	0.210	SPSIM [14]	0.172	GMSD [19]
PIPAL	$\alpha$	0.320	FSIMc [9]	0.317	MDSI [11]
	$\beta$	0.507	HPSI [12]	0.584	HPSI [12]
	$\gamma$	0.174	GMSD [19]	0.099	GMSD [19]

Table 5. Results of NCM metric for the three best component metrics.

Db.	Met.	FSIMc [9]	MDSI [11]	HPSI [12]	VCGS [13]	SPSIM [14]	LGV [17]	SWLGV [17]	GMSD [19]	NCM	${\mathit{M}}_1$, ${\mathit{M}}_2$, ${\mathit{M}}_3$
TID2008	PLCC	0.876	0.916	0.907	0.878	0.893	0.865	0.874	0.879	0.924	SPSIM [14]
	SROCC	0.884	0.921	0.910	0.897	0.910	0.881	0.889	0.891	0.924	MDSI [11]
	KROCC	0.699	0.751	0.737	0.717	0.730	0.696	0.711	0.709	0.759	HPSI [12]
	RMSE	0.647	0.538	0.566	0.643	0.605	0.674	0.652	0.640	0.514
TID2013	PLCC	0.877	0.909	0.893	0.900	0.909	0.778	0.797	0.855	0.922	SPSIM [14]
	SROCC	0.851	0.890	0.873	0.893	0.904	0.807	0.807	0.804	0.906	MDSI [11]
	KROCC	0.667	0.712	0.692	0.717	0.725	0.638	0.641	0.634	0.732	VCGS [13]
	RMSE	0.596	0.518	0.557	0.541	0.517	0.779	0.749	0.642	0.481
KADID-10k	PLCC	0.851	0.864	0.885	0.868	0.874	0.815	0.835	0.805	0.896	SPSIM [14]
	SROCC	0.854	0.885	0.885	0.871	0.874	0.820	0.840	0.847	0.897	MDSI [11]
	KROCC	0.665	0.702	0.699	0.683	0.687	0.630	0.655	0.664	0.717	HPSI [12]
	RMSE	0.568	0.544	0.505	0.538	0.525	0.627	0.595	0.643	0.480
PIPAL	PLCC	0.615	0.598	0.641	0.554	0.577	0.529	0.543	0.629	0.653	GMSD [19]
	SROCC	0.589	0.585	0.589	0.534	0.562	0.519	0.536	0.583	0.608	FSIMc [9]
	KROCC	0.416	0.408	0.417	0.370	0.391	0.359	0.372	0.414	0.432	HPSI [12]
	RMSE	0.104	0.106	0.101	0.110	0.108	0.112	0.111	0.103	0.100

In order to visualize the good quality of the proposed NCM index, scatter plots of the proposed eight metrics and the NCM for the tested bases are shown in Figure 4, Figure 5, Figure 6 and Figure 7. The scatter plots and their fitted curves show that the proposed combined NCM metric closely matches the MOS estimates for each of the databases.

Figure 4. Scatter plots of subjective MOS against IQA metrics obtained from the TID2008 database.

Figure 5. Scatter plots of subjective MOS against IQA metrics obtained from the TID2013 database.

Figure 6. Scatter plots of subjective MOS against IQA metrics obtained from the KADID-10k database.

Figure 7. Scatter plots of subjective MOS against IQA metrics obtained from the PIPAL database.

A study of computation times for the considered FR-IQA metrics was also conducted. The average computation times for each of the databases are shown in Table 6. The three fastest metrics are highlighted in bold. The NCM computation time, which is not included in Table 6, is approximately equal to the sum of the computation times of its component metrics.

Table 6. Computation times (s) for IQA metrics.

Database	FSIMc [9]	MDSI [11]	HPSI [12]	VCGS [13]	SPSIM [14]	LGV [17]	SWLGV [17]	GMSD [19]
TID2008	0.089	0.014	0.028	0.212	0.128	0.435	5.986	0.019
TID2013	0.104	0.016	0.037	0.214	0.113	0.339	6.130	0.019
KADID-10k	0.136	0.026	0.046	0.299	0.143	0.451	5.890	0.028
PIPAL	0.172	0.011	0.012	0.091	0.176	0.619	2.628	0.009

For the three fastest FR-IQA metrics highlighted in red (see Table 7), a linear combination was formed by redetermining the optimal $\alpha$, $\beta$, and $\gamma$ weights (see Table 5) from the perspective of PLCC maximization. The resulting NCM metric using the three fastest metrics achieved the best performance, as marked in green in Table 7.

Table 7. Results of the NCM metric for the three fastest component metrics.

Db.	Met.	FSIMc [9]	MDSI [11]	HPSI [12]	VCGS [13]	SPSIM [14]	LGV [17]	SWLGV [17]	GMSD [19]	NCM	${\mathit{M}}_1$, ${\mathit{M}}_2$, ${\mathit{M}}_3$
TID2008	PLCC	0.876	0.916	0.907	0.878	0.893	0.865	0.874	0.879	0.922	GMSD
	SROCC	0.884	0.921	0.910	0.897	0.910	0.881	0.889	0.891	0.923	MDSI
	KROCC	0.699	0.751	0.737	0.717	0.730	0.696	0.711	0.709	0.757	HPSI
	RMSE	0.647	0.538	0.566	0.643	0.605	0.674	0.652	0.640	0.519
TID2013	PLCC	0.877	0.909	0.893	0.900	0.909	0.778	0.797	0.855	0.912	GMSD
	SROCC	0.851	0.890	0.873	0.893	0.904	0.807	0.807	0.804	0.892	MDSI
	KROCC	0.667	0.712	0.692	0.717	0.725	0.638	0.641	0.634	0.716	HPSI
	RMSE	0.596	0.518	0.557	0.541	0.517	0.779	0.749	0.642	0.508
KADID-10k	PLCC	0.851	0.864	0.885	0.868	0.874	0.815	0.835	0.805	0.897	GMSD
	SROCC	0.854	0.885	0.885	0.871	0.874	0.820	0.840	0.847	0.897	MDSI
	KROCC	0.665	0.702	0.699	0.683	0.687	0.630	0.655	0.664	0.719	HPSI
	RMSE	0.568	0.544	0.505	0.538	0.525	0.627	0.595	0.643	0.479
PIPAL	PLCC	0.615	0.598	0.641	0.554	0.577	0.529	0.543	0.629	0.655	GMSD
	SROCC	0.589	0.585	0.589	0.534	0.562	0.519	0.536	0.583	0.608	MDSI
	KROCC	0.416	0.408	0.417	0.370	0.391	0.359	0.372	0.414	0.432	HPSI
	RMSE	0.104	0.106	0.101	0.110	0.108	0.112	0.111	0.103	0.100

The study was conducted in the MATLAB R2024a programming environment on a computer with the specifications provided in Table 8.

Table 8. Parameters of desktop computer used for experiments.

Processor	Intel(R) Core(TM) i5-7400 CPU @ 3.00 GHz (4 cores)
RAM	32 GB
OS	Windows 10
Env.	Matlab 2024a

The best results obtained using the NCM were additionally compared with those of other combined metrics presented in the literature [26,27]. The comparison is highlighted in bold in Table 9. The authors of [26] used combined metrics (MFMOGP3, MFMOGP4) based on the additive combination of component metrics, ranging from 8 to 10 metrics. In [27], combined metrics (OFIQA) based on the product form, involving between 4 and 17 factor-metrics, were proposed. For Table 9, we selected the best results from both of the above-mentioned works. For the TID2008 database, the results are comparable, while for the TID2013 database the proposed NCM index achieves the highest correlation coefficients. We conducted the comparison on the TID2008 and TID2013 databases, as both older works on combined metrics did not consider newer databases (KADID-10k, PIPAL).

Table 9. Comparison of the proposed NCM metric with other combined metrics.

Database	Metric	MFMOGP3 [26]	MFMOGP4 [26]	OFIQA [27]	NCM
TID2008	PLCC	0.925	0.902	0.910	0.922
	SROCC	0.923	0.911	0.915	0.923
	KROCC	0.757	0.727	0.738	0.757
	RMSE	0.511	0.580	0.557	0.519
TID2013	PLCC	0.883	0.914	0.906	0.922
	SROCC	0.868	0.902	0.890	0.923
	KROCC	0.688	0.725	0.713	0.757
	RMSE	0.581	0.503	0.526	0.519

4. Conclusions

From the existing literature on FR-IQA metrics, it is evident that there is no single metric that significantly outperforms the others. Therefore, the idea of creating a linear or nonlinear combination of several top metrics has emerged. The proposed approach opted for an additive combination of the three metrics with the highest correlation coefficients and the lowest RMSE. The resulting combined NCM metric was based on component metrics that depended on the selected database. NCM achieved the best results among all tested metrics across all tested databases. In addition, a case was examined where the three fastest metrics, i.e., MDSI, HPSI, and GMSD, were selected as components. The combined metric obtained in this case also achieved the best results compared to all the tested metrics. Potential extensions of the proposed approach include replacing the linear combination of metrics to their nonlinear combination, exploring alternative methods for optimizing weight selection, and more.