Enhancing Underwater Image Quality Assessment with Influential Perceptual Features

Abstract

In the multifaceted field of oceanic engineering, the quality of underwater images is paramount for a range of applications, from marine biology to robotic exploration. This paper presents a novel approach in underwater image quality assessment (UIQA) that addresses the current limitations by effectively combining low-level image properties with high-level semantic features. Traditional UIQA methods predominantly focus on either low-level attributes such as brightness and contrast or high-level semantic content, but rarely both, which leads to a gap in achieving a comprehensive assessment of image quality. Our proposed methodology bridges this gap by integrating these two critical aspects of underwater imaging. We employ the least-angle regression technique for balanced feature selection, particularly in high-level semantics, to ensure that the extensive feature dimensions of high-level content do not overshadow the fundamental low-level properties. The experimental results of our method demonstrate a remarkable improvement over existing UIQA techniques, establishing a new benchmark in both accuracy and reliability for underwater image assessment.

1. Introduction

The burgeoning field of oceanic engineering heavily relies on the quality of underwater images for a multitude of applications ranging from marine biology research to underwater robotics. Underwater image quality assessment (UIQA) plays a pivotal role in ensuring the reliability and efficacy of these applications [1,2]. High-quality underwater images are essential for accurate data analysis and interpretation, making UIQA an indispensable aspect of oceanic exploration and monitoring.

Image quality assessment (IQA) fundamentally revolves around three principal categories: full-reference (FR) [3,4], reduced-reference (RR) [5,6], and no-reference (NR) [7,8]. FR IQA methods compare a degraded image with a high-quality reference image, while RR IQA methods require only partial information from the reference. However, in most underwater imaging scenarios, distortion-free reference images are not available, rendering FR and RR methods impractical. Consequently, NR IQA, which does not require any reference image, becomes particularly relevant for UIQA. This paradigm shift towards NR IQA is driven by its adaptability in dynamic, unpredictable underwater environments where obtaining reference images is often unfeasible.

Over the last decade, various IQA methods have been proposed and achieved impressive performance. For instance, Ma et al. [9] proposed the ASCAM-Former that introduces the channel-wised self-attention into IQA. The spatial and channel dependencies among features are characterized, rendering to a comprehensive quality evaluation. Nizami et al. [10] proposed an NR-IQA method that combines visual salience and a convolution neural network (CNN), where the salient region of the image is identified first and then sent to the CNN for quality evaluation. Rehman et al. [11] designed a network applicable for both natural scene images and screen content. The network is based on the patch salience, which is determined by the texture and edge properties of the image. Sendjasni et al. [12] proposed an NR-IQA model for 360 degree images using a perceptually weighted multichannel CNN. Zhou et al. [13] further dealt with the 360 degree images through deep local and global spatiotemporal feature aggregation. HOU et al. [14] established an underwater IQA database of 60 degraded underwater images, which involves many unique distortions of underwater images, including blue-green scenes, greenish scenes, hazy scenes, and low-light scenes. Nizami et al. [15] proposed to utilize the discrete cosine transform (DCT) features for quality evaluation. Bouris et al. [16] developed a CNN-based IQA model for the quality evaluation of optic disc photographs, wherein a large-scale dataset is dedicatedly established. Nizami et al. [17] proposed an NR-IQA method that can extract quality-aware features in an elegant and efficient manner based on the Harris affine detector and scale invariant feature transformation. Zhou et al. [18] investigated the NR-IQA strategy for evaluating the perceptual quality of omnidirectional images (360 degree images), where an effective metric is proposed based on the low-level statistics and high-level semantics of omnidirectional images. Jiang et al. [19] proposed a full-reference solution for omnidirectional images by characterizing the degradation in a hierarchical manner.

Existing UIQA methods [20,21,22,23] predominantly focus on assessing low-level image attributes such as brightness, colorfulness, contrast, and sharpness. While these factors are crucial in determining the perceptual quality of underwater images, such methods often fall short by not considering the high-level semantic content that these images convey. The semantic aspect, encompassing the contextual and compositional elements of the image, is vital for a comprehensive understanding and utilization of the underwater visual data [24,25]. This oversight limits the scope and effectiveness of current UIQA techniques.

Moreover, while the advent of deep-learning techniques in IQA has significantly advanced the field, there remains a tendency in these methods to overlook the unique characteristics of underwater imagery, such as the specific distortions and color shifts caused by water medium. Deep-learning approaches often focus extensively on high-level feature extraction, potentially neglecting the intrinsic low-level properties specific to underwater images. This imbalance can lead to suboptimal performance when these models are applied to UIQA tasks [26], necessitating a more integrated approach that considers both low- and high-level image characteristics.

Our work aims to bridge this gap by developing an innovative approach that amalgamates both the low-level perceptual properties and abstract semantic features of underwater images, herein namely underwater image quality metric with low-level properties and selected high-level semantics (UIQM-LSH). Recognizing the disparity in feature dimensions between these two types of features, we employ the least-angle regression method (LARS) [27,28] for effective feature selection on the high-level semantic attributes. This approach ensures that the high-level features do not overshadow the crucial low-level properties, thus achieving a balanced and comprehensive feature representation for UIQA [29,30,31]. Our proposed UIQM-LSH stands out by adeptly integrating the diverse aspects of underwater images, providing a more nuanced and effective assessment of image quality. Through extensive experiments, UIQM-LSH demonstrates superior performance against existing UIQA or IQA approaches.

In summary, this work makes several new contributions:

• We have developed an innovative methodology, namely UIQM-LSH, which integrates both low-level image properties and high-level semantic features in underwater images. By considering both the intrinsic visual elements such as brightness, colorfulness, and sharpness, along with the contextual and compositional high-level semantic information, our method provides a nuanced understanding of underwater image quality, addressing a significant gap in existing UIQA methods.
• We implement the least-angle regression method for efficient feature selection in high-level semantics. This technique effectively balances the feature representation, preventing high-level features from dominating the assessment. By fine-tuning the feature selection process, our approach ensures that both low-level and high-level features contribute appropriately to the overall image quality assessment, leading to a more accurate and reliable evaluation.

2. Proposed Method

In this section, we elaborate on the proposed UIQA metric, namely Underwater Image Quality Metric with Low-level properties and Selected High-level semantics (UIQM-LSH), which is capable of characterizing the perceptual properties of underwater images reliably and comprehensively. The proposed UIQM-LSH consists of three branches, i.e., (1) a low-level feature extraction branch for characterizing the low-level properties of underwater images, (2) a high-level semantic branch for extracting the image content-related features, and (3) a feature balancing branch to prevent the extracted features from being dominated by high-level features. The processed features and the corresponding MOS values are sent to the support vector machine (SVM) to derive the final quality score. Figure 1 illustrates the overall architecture of the UIQM-LSH. We will cover each of these modules in detail in the following sections.

2.1. Low-Level Underwater Properties Characterization

2.1.1. Color Cast

Underwater image quality is often impaired by color cast. The underwater images usually appear bluish or greenish. Color cast occurs due to unequal absorption of various colors of light underwater, specifically, red light suffers greater loss due to the absorption by the water. It is indicated in [32] that in the CIELAB color space, the chromaticity distribution on the ab chrominance coordinate plane of underwater images with color cast is relatively concentrated and has a single peak with a small degree of dispersion, and the mean of chromaticity is large. Conversely, images that do not exhibit color cast or have a less severe degree of color cast have a chromaticity distribution on the ab chrominance coordinate plane that exhibits multiple peaks with a large degree of dispersion and a relatively small chromaticity mean. Based on the above results, the degree of color cast of underwater images can be characterized by calculating the chromaticity mean and variance in the ab chrominance space. The chromaticity means in a and b chrominance channel can be calculated as follows:

$$\overline{c_a}=\frac{{\sum }_{i=1}^M{c}_{ai}}{M}$$

$$\overline{c_b}=\frac{{\sum }_{i=1}^M{c}_{bi}}{M}$$

where $\overline{c_a}$ and $\overline{c_b}$, respectively, represent the average chromaticity values of all pixel points in chrominance channels a and b. M is the amount of all pixels of the underwater image. The overall chromaticity mean can be denoted as $\overline{c}$:

$$\begin{array}{c}\hfill \overline{c}=\sqrt{{\overline{c_a}}^2+{\overline{c_b}}^2}\end{array}$$

(1)

Then, we calculate the variances of the chromaticity distribution in chrominance channels a and b

$${\mathit{\sigma }}_a^2=\frac{{\sum }_{i=1}^M{\left(c_{ai}-\overline{c_a}\right)}^2}{M}$$

$${\mathit{\sigma }}_b^2=\frac{{\sum }_{i=1}^M{\left(c_{bi}-\overline{c_b}\right)}^2}{M}$$

where ${\mathit{\sigma }}_a^2$ and ${\mathit{\sigma }}_b^2$ represent the variance of the chromaticity distribution of the underwater image in a and b chrominance channels, respectively. Then, ${\mathit{\sigma }}_a^2$ and ${\mathit{\sigma }}_b^2$ can be used to characterize the average chromaticity variance of the image ${\mathit{\sigma }}^2$:

$$\begin{array}{c}\hfill {\mathit{\sigma }}^2=\sqrt{{\mathit{\sigma }}_a^2+{\mathit{\sigma }}_a^2}\end{array}$$

(2)

Finally, we evaluate the color cast degree using the ratio of chromaticity mean to variance, denoted by d:

$$\begin{array}{c}\hfill d=\frac{\overline{c}}{{\mathit{\sigma }}^2+\mathit{\tau }}\end{array}$$

(3)

where $\mathit{\tau }$ is a small constant introduced to reduce instability, typically set to 0.0001 based on empirical values. The larger the ratio index d is, the greater the degree of color cast in the image is. Figure 2 shows some underwater images with color cast. Intuitively, as the degree of color cast in the underwater images increases, the value of the ratio index d correspondingly increases.

2.1.2. Brightness

Due to the absorption of light by the water, underwater images often suffer from a lack of brightness. The overall brightness and the uniformity of brightness distribution are two key factors in evaluating the quality of underwater images. When evaluating the overall brightness, a method to calculate the gray-scale mean value of pixel points can be adopted. First, the underwater image obtained is converted into a gray-scale image G, and the gray-scale values of each pixel point in the image are obtained and the brightness mean value is calculated, which is recorded as $\overline{g}$.

$$\begin{array}{c}\hfill \overline{g}=\frac{{\sum }_{i=1}^M{g}_i}{M}\end{array}$$

(4)

where m represents the number of pixel points in the image and $g_i$ represents the gray-scale values of pixel points in gray-scale image G. To evaluate the uniformity of brightness distribution in underwater images, it is necessary to statistic the brightness variations in different parts of the image. Here, the image is divided equally into n blocks geometrically, and the average brightness of each pixel point in each image block is calculated to obtain a set of brightness means containing n image blocks. Let the maximum element be $g_{max}$, which corresponds to the mean brightness of the brightest part of the image block. The minimum element is denoted by $g_{min}$. The brightness variations in underwater images can be calculated as:

$$\begin{array}{c}\hfill h=\frac{g_{max}-g_{min}}{g_{max}+\mathit{\eta }}\end{array}$$

(5)

where h represents the overall brightness variation degree of the image. $\mathit{\eta }$ is a small constant introduced to reduce instability, typically set to 0.0001 based on empirical values. The smaller the value h is, the more uniform the brightness distribution of the underwater image is.

2.1.3. Contrast

Contrast is an important parameter for evaluating image quality. In this study, we evaluated the impact of contrast on underwater image quality from two aspects: local contrast and global contrast. When describing local contrast, we referred to the concept of contrast energy (CE) and calculated the contrast energy of the image in three chrominance channels $gray,yb,rg$, which can be linearly represented by the $RGB$ chrominance channels [33]. Specifically, the calculation methods are as follows: $gray=0.299R+0.587G+0.114B$, $yb=0.5(R+G)-B$, and $rg=R-G$. The calculation method for CE is as follows:

$$\begin{array}{c}\hfill C{E}_k=\frac{\mathit{\beta }·Q\left(k\right)}{Q\left(k\right)+\mathit{\beta }·G}-e_k\end{array}$$

(6)

$$\begin{array}{c}\hfill Q\left(k\right)=\sqrt{{\left(I_k\otimes d_h\right)}^2+{\left(I_k\otimes d_v\right)}^2}\end{array}$$

(7)

where k represents one of the color channels {$gray$, $yb$, or $rg$}. $d_h$ and $d_v$ are the horizontal and vertical second derivatives of the Gaussian function. ⊗ represents convolution computation, and $\mathit{\beta }$ is the maximum value of $Q\left(k\right)$. G represents contrast gain. $n_k$ represents the noise threshold for the color channel k.

Evaluating the global contrast of underwater images can be achieved by analyzing the grayscale histogram of the image. Generally, images with high contrast have relatively flat grayscale histogram distribution with a wide range of grayscale values, while images with low contrast have more steeply shaped grayscale histograms with a smaller range of grayscale values. Based on the distribution characteristics of the grayscale histogram, the global contrast of the image can be reflected by comparing the grayscale histogram of the image with the uniform distribution. Here, we calculate the Kullback–Leibler divergence (KL divergence) between the grayscale histogram distribution of the image and the uniform distribution to represent the flatness of the grayscale histogram. The smaller the KL divergence is, the closer the grayscale histogram is to a uniform distribution, and the higher the global contrast is. Let P represent the grayscale histogram distribution of the underwater image and Q represent the uniform distribution. The KL divergence between P and Q is calculated as follows,

$$\begin{array}{c}\hfill D_{K\_L}\left(\mathbf{P}||\mathbf{Q}\right)=-\int P(x)logQ(x)dx+\int P(x)logP(x)dx\end{array}$$

(8)

Due to the asymmetry of the KL divergence (i.e., $D_{JS}\left(\mathbf{P}||\mathbf{Q}\right)=D_{KL}(\mathbf{P}||\mathbf{Q})$), we adopt the symmetric JS divergence to characterize the flatness of the image grayscale histogram. The JS divergence can be derived from the KL divergence as follows,

$$\begin{array}{c}\hfill D_{J\_S}\left(\mathbf{P}||\mathbf{Q}\right)=\frac{1}{2}\left[D_{K\_L}\left(\mathbf{P}||\mathbf{N}\right)+D_{K\_L}\left(\mathbf{Q}||\mathbf{N}\right)\right]\end{array}$$

(9)

where $N=\frac{P+Q}{2}$. Finally, we use the J-S divergence to characterize the global contrast of the underwater images. In Figure 3, we plotted the results of enhancing the contrast of high-contrast and low-contrast images through histogram equalization. The results of the histogram distribution show that after histogram equalization, both the high-contrast original image and the low-contrast original image exhibit a more even histogram distribution, resulting in improved contrast. Additionally, we calculated the $D_{JS}$ values for both original images. It is shown that the low-contrast image has a higher JS divergence compared to the high-contrast image.

2.2. Vision Transformer

Image quality assessment is the perception of the content of an image by the human visual system (HVS), including a combination of global high-level quality perception and local low-level features [25,34]. In the previous section, we concentrated on evaluating the local features of underwater images. This section presents a global evaluation of image semantic quality based on the spatial domain. Specifically, we scan and sense the image quality in different spatial domains and obtain the global image quality after comparing and fusing the sensing results. In [35], we learnt that the quality relationship between different spatial domains of an image is significant in assessing the quality of the global image.

In recent years, due to the enhanced global perception capabilities of the transformer, it has found extensive applications in both the field of natural language processing and computer vision [36,37,38]. Therefore, we propose utilizing transformer modules to extract the quality relationships between adjacent spatial regions and obtain the global quality of an image.

When provided with an input image I, we first break down the image into N patches, each with a resolution of p ∗ p and $N=\frac{H_{\mathrm{I}}{W}_{\mathrm{I}}}{p^2}$. N is the total number of patches, and each image patch is represented as a vector containing that specific patch’s pixel information. After flattening each patch, they are passed through a linear projection layer to map each patch into a D-dimension embedding. Next, we embed a learnable embedding called the Image-Quality-Collection cls token ${\mathit{C}}_{\mathrm{IQC}}\in {\mathbb{R}}^{1\times D}$ to concatenate with the perceived underwater image information. Finally, we incorporate positional encoding into these $N+1$ embeddings to capture the spatial relationships within the underwater image.

Self-Attention Module

After obtaining the embedding vector $\mathit{E}=\{{\mathit{C}}_{\mathrm{IQC}},{\mathit{E}}_1,\dots {\mathit{E}}_N\}\in {\mathbb{R}}^{(N+1)\times D}$, we use the multi-head self-attention (MHSA) block to obtain the self-attention scores. The MHSA block includes h heads, which means attending to different semantic information and feature representations while obtaining a dimension of $d=\frac{D}{h}$. Following the normalization layer, the self-attention layer generates projections for $query\left(Q\right)$, $key\left(K\right)$, and $value\left(V\right)$ for ${\mathit{Q}}_{\mathsf{\ell }},{\mathit{K}}_{\mathsf{\ell }},{\mathit{V}}_{\mathsf{\ell }}\in {\mathbb{R}}^{(N+1)\times d}$. For each head, the computation process of the self-attention layer is defined as follows:

$$\begin{array}{c}\hfill A({\mathit{Q}}_{\mathsf{\ell }},{\mathit{K}}_{\mathsf{\ell }},{\mathit{V}}_{\mathsf{\ell }})=softmax\left(\frac{{\mathit{Q}}_{\mathsf{\ell }}{{\mathit{K}}_{\mathsf{\ell }}}^T}{\sqrt{d_k}}\right){\mathit{V}}_{\mathsf{\ell }},\hspace{1em}\mathsf{\ell }=1\dots h.\end{array}$$

(10)

where $d_k$ is the dimension of K. We compute the self-attention scores for each head separately and concatenate them to obtain the final self-attention map. The following is the calculation process for the transformer encoder.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill MHSA\left(\mathit{Q},\mathit{K},\mathit{V}\right)=& Concat(A({\mathit{Q}}_1,{\mathit{K}}_1,{\mathit{V}}_1),\dots ,\hfill \\ & A({\mathit{Q}}_h,{\mathit{K}}_h,{\mathit{V}}_h))\mathit{W}\hfill \\ \hfill {\mathit{O}}_M=& MHSA\left(\mathit{Q},\mathit{K},\mathit{V}\right)+\mathit{E}\hfill \\ \hfill {\mathit{O}}_{Encoder}=& MLP\left(LN\left({\mathit{O}}_M\right)\right)+{\mathit{O}}_M,\hfill \end{array}\end{array}$$

where W refers to the weights of the linear projection layer and $\mathrm{LM}(·)$ stands for the layer normalization.

2.3. Least Angle Regression for Adaptive Feature Selection

The excessive number of high-level features we learn by cls token in the transformer will affect the role of low level in quality assessment. We here use the least angle regression (LARS) method to select those influential high-level features and thus reduce the dimension of the high-level features [39].

By constructing a first-order penalty function, the LARS method determines that the coefficients of some variables are zero to delete some invalid variables and realize the role of dimensionality reduction [40]. In the LARS model, we assume normalized data $\overline{\mathit{X}}\in {\mathbb{R}}^{N\times M}$ and the regression coefficient vector $\mathit{\theta }\in {\mathbb{R}}^M$ satisfing the conditions of

$$\begin{array}{c}\hfill y=\overline{X}\mathit{\theta }\end{array}$$

(11)

$$\begin{array}{c}\hfill \mathrm{subject}\mathrm{to}\sum _{j=1}^m\left|{\mathit{\theta }}_j\right|⩽t,\end{array}$$

(12)

where N represents the number of samples and M represents the number of features, making y $\in {\mathbb{R}}^n$, and t is a positive constraint value. The linear regression model to be solved is as follows:

$$\begin{array}{c}\hfill \min S(\mathit{\theta })={(y-\overline{X}\mathit{\theta })}^T(y-\overline{X}\mathit{\theta })+\mathit{\lambda }{\parallel \mathit{\theta }\parallel }_1.\end{array}$$

(13)

The idea of the LARS model to solve Equation (13) is to initialize all variable values of $\overline{\mathit{X}}$ to zero first and make predictions about y by finding the independent variable $x_i\in \overline{\mathit{X}}$ with the highest correlation with y [41,42]. The LARS model is then solved by finding the independent variable xi with the highest correlation with the current residuals. Then, find an independent variable $x_j\in \overline{\mathit{X}}$ belonging along the direction of this independent variable with the highest correlation with the current residuals. Then, proceed along the angular bisector of $x_i$ with $x_j$ until a third predictor variable is found. Therefore, the directions of the three predictor variables are equiangular, and so on, iterating until all the required variables are obtained. In the process of travelling, the correlation between the residuals and the selected variables will gradually decrease so that the irrelevant variables can be eliminated, thus achieving dimensionality reduction. For this purpose, let $c={\left\{c_1,c_2,\cdots ,c_M\right\}}^T$ be the vector of the correlation calculated as:

$$\begin{array}{c}\hfill c={\overline{X}}^T(y-\overline{X}\mathit{\theta }),\end{array}$$

(14)

where the difference between y and $\overline{X}\mathit{\theta }$ is named the residual vector. Each element in c embodies a correlation between the corresponding column in $\overline{X}$ and its residual vector.

Now, we assume that k iterations have been performed and that the resulting regression vector is $u_k$. The set of k variables is selected to form a set $X_A$ (A⊂$\overline{X}$) and use more relevant variables in $\overline{X}$ to populate set $X_A$ with the number of iterations. Each column in set $X_A$ consists of the column corresponding to the position of the maximum value of c indexed in the $\overline{X}$ matrix, i.e., $\left|c_{max}\right|={max}_i\left|c_i\right|$. At the same time, we define the following relationships:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill G_A& =X_A^T{X}_A\hfill \\ \hfill A_A& ={\left(l_A^T{G}_A^{-1}{l}_A\right)}^{-1/2},\hfill \end{array}\end{array}$$

(15)

where $l_A$ is the vector of ones whose length is the number of elements in set A. Additionally, we define:

$$\begin{array}{c}\hfill u_A=X_A{A}_A{G}_A^{-1}{l}_A,\end{array}$$

(16)

where $X_A={\left(\cdots {,}_{s_j}{x}_j,\cdots \right)}_{j\in A}$, $s_j=sign\left\{c_j\right\}$.

This method calculates $a={\overline{X}}^T{u}_A$ to determine the variables needed during the iteration. From there, the regression vector is updated:

$$\begin{array}{c}\hfill {\mathit{\mu }}_k={\mathit{\mu }}_{k-1}+{\mathit{\gamma }}_k{\mathit{u}}_A,\end{array}$$

(17)

where ${\mathit{\gamma }}_k$ can be calculated as

$$\begin{array}{c}\hfill {\mathit{\gamma }}_k=\underset{j\in A^c}{min}\left(\frac{C_{max}-c_j}{A_A-a_j},\frac{C_{max}+c_j}{A_A+a_j}\right).\end{array}$$

(18)

By choosing a smaller value of ${\mathit{\gamma }}_k$, we update subset $X_A$ while calculating $C_{max}^{\prime }=C_{max}-{\mathit{\gamma }}_k{A}_A$. The LARS model iterates until the values of c become negative or the desired number of variables is selected, thereby achieving dimensionality reduction.

2.4. Qualty Score Assessment

After obtaining the feature maps with LARS dimensionality reduction, we use SVR to map the feature maps to the final underwater image quality scores [8]. We want to learn a mapping that predicts the underwater quality score from the feature map x, defined as:

$$\begin{array}{c}\hfill f\left(\mathbf{x}\right)=\langle \mathit{\alpha },\mathbf{x}\rangle +b,\end{array}$$

(19)

where $\mathit{\alpha }$ and b indicate the model weights and bias, and $\langle ·,·\rangle$ is the inner product of the parameters. In the SVR model, we introduce two slack variables, ${\mathit{\xi }}_i$ and ${\mathit{\xi }}_i^{*}$, to introduce soft intervals to construct the following optimization problem:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill min\frac{1}{2}{\parallel \mathit{\alpha }\parallel }^2+\mathit{\lambda }\left(\sum _{i=1}^n{\mathit{\xi }}_i+\sum _{i=1}^n{\mathit{\xi }}_i^{*}\right)\\ \hfill \mathrm{s}.\mathrm{t}.〈\mathit{\alpha },{\mathbf{x}}_i〉+b-y_i\le \mathit{\epsilon }+{\mathit{\xi }}_i\\ \hfill y_i-〈\mathit{\alpha },{\mathbf{x}}_i〉-b\le \mathit{\epsilon }+{\mathit{\xi }}_i^{*}\\ \hfill {\mathit{\xi }}_i,{\mathit{\xi }}_i^{*}\ge 0,i=1,2,\dots ,n\end{array}\end{array}$$

(20)

where $\mathit{\lambda }$ is a constant used to balance the accuracy of the model. In this model, our kernel function chosen is $K\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=exp\left(-\mathit{\gamma }{∥{\mathbf{x}}_i-{\mathbf{x}}_j∥}^2\right)$ to solve the above equation and finally obtain the predicted score.

3. Experiments Results and Discussion

3.1. Dataset and Evaluation Protocols

In order to verify the effectiveness of our proposed UIQM-LSH, we conducted extensive validation experiments on the authentic underwater dataset, namely the underwater image quality assessment database (UWIQA). UWIQA contains 890 underwater images of different qualities [23] and is the largest public UIQA dataset at present. The labels, namely MOSs, are obtained by single-stimulus methodology, where each image is scored by a number of observers and the average value of these scores is considered the MOS. In Figure 4, we give example images of the UWIQA database. As can be seen, these images contain rich ocean features and a wide range of distortion types, thus ensuring a reliable evaluation of the proposed method.

The IQA performance metrics commonly used for image quality evaluation are proposed by the Video Quality Experts Group (VQEG) [43], which recommends the use of the Spearman rank-order correlation coefficient (SRCC) and Kendall’s rank-order correlation coefficient (KRCC) for the monotonicity of the prediction, Pearsons linear correlation coefficient (PLCC) for the accuracy of the prediction, and root mean square error (RMSE) for the consistency. These four metrics are used to assess the performance of the IQA method. Among them, SRCC and KRCC measure the degree to which the model’s predicted scores vary with the mean opinion score (MOS). They are not affected by the magnitude of the change. SRCC and KRCC can be calculated as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill SRCC=& 1-\frac{6{\sum }_{n=1}^N{d}_n^2}{N(N^2-1)}\hfill \\ \hfill KRCC=& \frac{N_c-N_d}{0.5\times (N-1)\times N}.\hfill \end{array}\end{array}$$

(21)

Here, N denotes the total number of images in the dataset and $d_n$ indicates the ranking difference between the subjective score (MOS) and the objective score for each pair. $N_c$ and $N_d$ denote the number of consistent and inconsistent images in the dataset. At the same time, PLCC and RMSE are used to measure the accuracy and consistency of the predicted scores by the model. PLCC measures the degree of linear correlation between the predicted and reference quality scores. In contrast, RMSE measures the average size of the difference between the predicted and reference quality scores. Both PLCC and RMSE are used to evaluate the performance of image quality assessment models. The PLCC and RMSE can be expressed as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill PLCC=& \frac{{\sum }_{n=1}^N(P_n-\widehat{P})(M_n-\widehat{M})}{\sqrt{{\sum }_{n=1}^N{(P_n-\widehat{P})}^2{(M_n-\widehat{M})}^2}}\hfill \\ \hfill RMSE=& \sqrt{\frac{1}{N}\sum {\left(P_n-M_n\right)}^2},\hfill \end{array}\end{array}$$

(22)

where $P_n$ and $M_n$ denote the predicted objective score of the n-th picture and the subjective score of the MOS. $\widehat{P}$ and $\widehat{M}$ represent the mean values of the predicted objective score and the subjective MOS score, respectively. In the above metrics, the closer the values of SRCC, KRCC, and PLCC are to 1, and the closer the value of RMSE is to 0, the higher the performance of the model algorithm is indicated.

Before calculating these metrics, we need to map the objective prediction algorithm to the subjective MOS values using a nonlinear fitting function, which is formulated as:

$$\begin{array}{c}\hfill O\left(P\right)={\mathit{\beta }}_1\left(\frac{1}{2}-\frac{1}{1+\exp \left({\mathit{\beta }}_2\left(P-{\mathit{\beta }}_3\right)\right)}\right)+{\mathit{\beta }}_{4}P+{\mathit{\beta }}_5,\end{array}$$

(23)

where P and $O\left(P\right)$ indicate the predicted objective and mapped scores, ${\mathit{\beta }}_i\mathrm{for}i=1\sim 5$ are the model parameters obtained in the fitting. The logistic function in Equation (23) is utilized to model the non-linearity inherent in the human subjective evaluation. It transforms the continuous predicted scores into values that better align with the categorical MOSs, thereby rendering to more reliable model evaluation.

3.2. Implementation Details

We implemented the proposed UIQM-LSH using Python and the Pytorch framework. In the transformer, each input image was randomly cropped into 10 image patches, each of which had the size of 224 × 224. In addition to random cropping, other data augmentation strategies, including flipping and rotation, were utilized. Each image patch was transformed into a number of patches with the size of 16. The number of transformer layers was set to 12, each of which contained 6 heads. We followed the standard train-test strategy, with 80% training data and testing 20% data. The training and testing data are not overlapped. The learning rate was set to ${10}^{-4}$ with a decay factor of 10. The ADAM optimizer was utilized for training with a batch size of 16. The experiment was repeated 10 times, each with a different data splitting, in order to mitigate the performance bias. The averages of SRCC, KRCC, PLCC, and RMSE of the underwater image were reported.

3.3. Prediction Performance Evaluation

The proposed method UIQM-LSH is compared with 13 mainstream UIQA or IQA methods, including BRISQUE [44], NFERM [45], DBCNN [46], HyperIQA [35], TReS [47], PaQ-2-PiQ [48], NIQE [49], NPQI [50], SNP-NIQE [20], UCIQE [21], UIQM [22], CCF [51], and FDUM [23]. Note that the first nine IQA methods are for natural image processing and the last four IQA methods and the proposed UIQM-LSH are for underwater application scenarios.

As shown in

Table 1. Overall performance evaluation comparison on UWIQA database, where the best performances are highlighted in bold.

IQA Metrics	Type	UWIQA
		SRCC	KRCC	PLCC	RMSE
DBCNN [46]	IQA (supervised)	0.5392	0.4065	0.5179	0.1241
HyperIQA [35]	IQA (supervised)	0.6501	0.5040	0.6799	0.1114
TReS [47]	IQA (supervised)	0.5038	0.3817	0.4942	0.1302
GraphIQA [52]	IQA (supervised)	0.7708	0.6609	0.7842	0.0801
PaQ-2-PiQ [48]	IQA (unsupervised)	0.6341	0.4955	0.6203	0.1221
BRISQUE [53]	IQA (unsupervised)	0.3456	0.2562	0.3669	0.1415
NFERM [45]	IQA (unsupervised)	0.3486	0.2595	0.3925	0.1398
NIQE [49]	IQA (unsupervised)	0.4347	0.3243	0.4687	0.1343
NPQI [50]	IQA (unsupervised)	0.6078	0.4667	0.6361	0.1173
SNP-NIQE [20]	IQA (unsupervised)	0.5516	0.4199	0.5897	0.1228
UCIQE [21]	UIQA (unsupervised)	0.6271	0.4863	0.6261	0.1185
UIQM [22]	UIQA (unsupervised)	0.5960	0.4563	0.5928	0.1225
CCF [51]	UIQA (unsupervised)	0.4456	0.3344	0.4634	0.1348
FDUM [23]	UIQA (unsupervised)	0.6780	0.5289	0.6462	0.1160
UIQM-LSH (Pro.)	IQA (supervised)	0.7580	0.6528	0.7890	0.0790

, our proposed UIQM-LSH achieves the best prediction results in SRCC, KRCC, PLCC, and RMSE indicators of all IQA methods, thereby strongly reflecting the advantages of the proposed method. In addition, for the IQA methods, the deep-learning (DL)-based IQA methods, such as DBCNN, HyperIQA, and TReS, perform much better than the hand-crafted feature-based IQA methods. This observation is not surprising since DL methods generally possess a strong feature learning capability. However, such a merit is often with the sacrifice of computational efficiency. By further analyzing Table 1, we can find out that the hand-crafted feature-based methods, such as UCIQE and FDUM, achieve better performance than some DL-based methods while their feature dimensions are much smaller. This indicates the crucial role of the low-level perceptual features in the UIQA task. Such an observation is also evident by our proposed UIQM-LSH. The architecture of UIQM-LSH has a similar network backbone to TReS, i.e., the ViT architecture. However, the UIQM-LSH surpasses TReS significantly. This result can be attributed to the dedicated consideration of the unique properties of underwater images, such as colorfulness and brightness. In other words, relying on the DL network solely without considering the task-oriented factors can hardly achieve an optimal result.

We then perform the statistical significance test on the subjective MOS values versus the mass fraction obtained from the mapping. The statistical differentiation between our proposed method and other IQA methods is assessed through t-tests. The values of “1”, “0”, and “−1” indicate that the proposed method is superior, equal, and inferior to the compared algorithms, respectively. In

Table 2. Statistical significance test results on the UWIQA database. The data of “1”, “0”, and “−1” indicate that the proposed method is superior, equal, and inferior to the compared algorithms, respectively, (95% confidence).

t-Test	BRISQUE	NFERM	NIQE	SNP-NIQE	DBCNN	HyperIQA	NPQI	TReS	PaQ-2-PiQ	UCIQE	UIQM	CCF	FDUM
UWIQA	1	1	1	1	1	1	1	1	1	1	1	1	1

, we see that all values are “1”, indicating that our proposed method is statistically superior to all other methods.

3.4. Cross-Dataset Validation

Having testified the IQA methods on each database, we further evaluate their prediction performance through the cross-database validation methodology. The cross-database validation tests the generalization ability of IQA metrics, which is essential to ensure the reliability and effectiveness of the models. In implementation, we initially train an IQA model for each method on one database and then test the trained models on the other database without any fine-tuning or parameter adaptation. Here, we utilize another database, namely UID2021 [14], which contains 960 underwater images in total. The experimental results in terms of SRCC, KRCC, PLCC, and RMSE on both databases are presented in

Table 3. Cross-database validation, where the best performances are highlighted in bold.

IQA Metrics	UWIQA (Trained on UID2021)				UID2021 (Trained on UWIQA)
	SRCC	KRCC	PLCC	RMSE	SRCC	KRCC	PLCC	RMSE
DBCNN [46]	0.4851	0.3492	4904	0.1208	0.6089	0.4706	0.6237	1.4198
HyperIQA [35]	0.5516	0.4287	0.5625	0.1209	0.6703	0.5263	0.6679	1.4132
TReS [47]	0.3459	0.2346	0.3673	0.1341	0.5224	0.4097	0.5321	1.5339
UIQM-LSH (Pro.)	0.6307	0.5777	0.6340	0.1309	0.7189	0.6645	0.7197	1.3933

.

3.5. Ablation Study

The proposed UIQM-LSH leverages both low-level and high-level perceptual properties to characterize the quality of underwater images. It is therefore necessary to examine their individual contributions to the model performance. To this end, we first conduct the ablation analysis to examine the effectiveness of the designed three modules, i.e., low-level feature extraction, high-level feature extraction, and the LARS module. Experiments were conducted on the UWIQA database, and the experimental settings are the same as those described in Section 3.2. The experimental results in terms of the average of SRCC, KRCC, PLCC, and RMSE are reported in

Table 4. Performance evaluation of different types of features on UWIQA database. The high-level features are processed without or with the LARS method. The best performances are highlighted in bold.

Feature Types	LARS	UWIQA
		SRCC	KRCC	PLCC	RMSE
Low-level perceptual features	✗	0.6748	0.5272	0.6383	0.1066
High-level perceptual features		0.6944	0.5916	0.6964	0.0928
Combination		0.7183	0.6237	0.7060	0.0915
Low-level perceptual features	✓	0.6748	0.5272	0.6383	0.1066
High-level perceptual features		0.7246	0.6299	0.7673	0.0801
Combination		0.7580	0.6528	0.7890	0.0790

.

In Table 4, we can obtain several meaningful observations. Firstly, either low-level features or high-level features can achieve a moderate performance, and their combinations deliver the best prediction. Hence, it is evident that both types of features are effective and contribute to the overall model performance. Secondly, we can find out that the high-level features achieve better prediction accuracy than the low-level features. This is reasonable since the number of dimensions of high-level features is far larger than the low-level features, i.e., 768 high-level features vs. 8 low-level features. Such a large dimension can capture a wide variety of aspects of an image, such as the details about the objects presence, the scene context, and the interactions between elements. As a result, it can offer a richness and complexity of information that aligns closely with the multifaceted nature of the underwater image quality. However, the large number of high-level features may also result in the model being biased. As can be seen, the improvement of introducing the low-level features into original high-level features is much less than that into dimension reduced of high-level features. Thirdly, the LARS method can effectively promote the model performance further. It optimizes the high-level features, presenting a more balancing and effective feature fusion. The LARS method reduces redundancy and noise by selecting the most relevant high-level features and enhances the model’s generalization capabilities. More importantly, it leads to a more interpretable and computationally efficient model.

The experiments in Table 4 were conducted from the module level. However, by introducing the low-level features and the LARS method, the proposed UIQM-LSH is more interpretable than the common black-box DL methods. We further analyze UIQM-LSH from the feature level. In particular, we ablate each low-level feature, i.e., brightness, colorfulness, and contrast, to examine their individual contribution. The experimental results are reported in

Table 5. Performance evaluation of low-level perceptual features on UWIQA database. The best performances are highlighted in bold.

Feature Types	UWIQA
	SRCC	KRCC	PLCC	RMSE
Colorfulness	0.4192	0.3201	0.4396	0.1244
Brightness	0.5028	0.3777	0.5200	0.1266
Contrast	0.6325	0.5552	0.6172	0.1165
Colorfulness + Contrast	0.6418	0.5035	0.6196	0.1090
Colorfulness + Brightness	0.5859	0.4773	0.5936	0.1108
Contrast + Brightness	0.6578	0.5414	0.6198	0.1157
Contrast + Brightness + Colorfulness	0.6748	0.5272	0.6383	0.1066

. As can be observed, each type of feature is effective in characterizing the quality of underwater images. The combination of any two out of the three feature types presents a better performance than the single feature type. This finding indicates that the different types of features complement each other. By in-depth analysis of Table 5, the image contrast contributes most to the model performance. One of the main reasons is that the difference in either image brightness or colorfulness can impact the image contrast as well. Higher contrast enhances the visibility of objects and details, making images more visually appealing and informative, especially in situations where light is limited and scattered in water. On the other hand, the colorfulness feature may be less effective than contrast and brightness in estimating the quality of underwater images due to the limited color diversity in underwater environments. Though the color cast may impact the human subjective evaluation on the underwater image, a relatively higher quality score may be still assigned when the color-casted image is visually clear.

4. Conclusions

This paper deals with the challenging issues in the field of underwater quality assessment and proposes a novel method, namely underwater image quality metric with low-level properties and selected high-level semantics (UIQM-LSH). This approach bridges the gap in current UIQA methodologies by integrating both low-level image attributes and high-level semantic content. By implementing the least-angle regression method for feature selection, our model ensures a balanced representation of these diverse features, thus enhancing the accuracy and reliability of image quality assessments. The superior performance of UIQM-LSH over existing UIQA methods, as demonstrated through extensive testing, highlights its potential as a more effective tool for various applications in oceanic engineering, marine biology, and underwater robotics. This work exhibits promising results to significantly enhance the understanding and exploration of underwater environments.