Delving into Underwater Image Utility: Benchmark Dataset and Prediction Model

Abstract

High-quality underwater images are essential for both human visual perception and machine analysis in marine vision applications. Although significant progress has been achieved in Underwater Image Quality Assessment (UIQA), almost all existing UIQA methods focus on the visual perception-oriented image quality issue and cannot be used to gauge the utility of underwater images for the use in machine vision applications. To address this issue, in this work, we focus on the problem of automatic underwater image utility assessment (UIUA). On the one hand, we first construct a large-scale Object Detection-oriented Underwater Image Utility Assessment (OD-UIUA) dataset, which includes 1200 raw underwater images, corresponding to 12,000 enhanced results by 10 representative underwater image enhancement (UIE) algorithms and 13,200 underwater image utility scores (UIUSs) for all raw and enhanced underwater images in the dataset. On the other hand, based on this newly constructed OD-UIUA dataset, we train a deep UIUA network (DeepUIUA) that can automatically and accurately predict UIUS. To the best of our knowledge, this is the first benchmark dataset for UIUA and also the first model focusing on the specific UIUA problem. We comprehensively compare the performance of our proposed DeepUIUA model with that of 14 state-of-the-art no-reference image quality assessment (NR-IQA) methods by using the OD-UIUA dataset as the benchmark. Extensive experiments showcase that our proposed DeepUIUA model has superior performance compared with the existing NR-IQA methods in assessing UIUS. The OD-UIUA dataset and the source code of our DeepUIUA model will be released.

1. Introduction

High-quality underwater images are essential for marine scientific research, resource development, environmental monitoring, and other applications [1,2,3,4]. However, due to light absorption and scattering effects in underwater environments, raw underwater images usually suffer from various degradation issues such as color cast, blur, low light, and reduced contrast, which not only affects image quality, but also poses great challenges for underwater machine vision tasks [5,6,7,8,9,10]. In order to improve the quality of underwater images, researchers have developed a variety of underwater image enhancement (UIE) algorithms [11,12,13], which have achieved impressive results in restoring image colors and improving contrast and details. However, the effectiveness of these UIE methods in improving underwater machine analysis performance remains unclear. In addition, almost all existing Underwater Image Quality Assessment (UIQA) methods [14,15,16,17,18] focus on measuring human visual perception-oriented image quality, which cannot be directly used to measure the utility of underwater images in machine vision tasks, i.e., underwater image utility assessment (UIUA). Thus, how to fairly gauge the utility of underwater images in machine vision applications remains a challenging and unsolved problem [19,20].

Similar to UIQA, which focuses on measuring human visual perception-oriented image quality, UIUA aims to assess the utility of underwater images in specific machine vision-related tasks [21]. A comprehensive underwater image dataset with reliable utility score annotations plays an indispensable role for establishing reliable UIUA metrics, just as a well-constructed dataset for UIQA [22]. Such a dataset should include relevant data that can help in accurately assessing the utility of underwater images in complex underwater scenarios, which parallels the role of benchmark datasets in the development of UIQA.

Existing UIQA datasets [23,24] contain raw underwater images, enhanced counterparts, and human visual perception-oriented quality scores. These datasets have facilitated significant advances in the development of UIQA metrics, allowing more accurate evaluation of visual quality and also driving significant progress in the design of more advanced UIE algorithms [25]. However, these datasets do not provide annotations tailored to assessing the utility of underwater images in machine vision applications. The lack of underwater image utility score annotations restricts researchers from measuring the practical value of underwater images for real-world machine vision tasks. Clearly, a large-scale benchmark dataset is essential for performing a fair and comprehensive performance evaluation of UIUA metrics [26]. This kind of dataset should include raw underwater images with diverse underwater scenes and typical degradation types, corresponding enhanced results generated by representative UIE algorithms, and the corresponding utility scores. Although collecting raw underwater images and producing representative enhanced results is relatively straightforward, ensuring the reliability of utility scores continues to be a crucial challenge.

In addition to the absence of a benchmark dataset for UIUA, a dedicated objective UIUA metric is also lacking at the current stage. Although significant progress has been achieved in UIQA during the past decade, the focus of UIQA is different from that of UIUA. For the classical UIQA task, the most critical issue is to extract visual quality-aware features that can well reflect subjective perception of humans in terms of different image attributes. Meanwhile, for the UIUA task, the goal is to extract task-oriented features from the input images. Although one can resort to the end-to-end deep learning technique to directly train a UIUA network, the feature extraction module should still be carefully designed so as to increase the interpretability of the network architecture. This gap between perception-oriented and task-oriented features makes existing UIQA metrics unable to achieve the desired performance on the UIUA task.

In order to address the above issues, in this work, we first construct a large-scale Object Detection-oriented UIUA dataset (OD-UIUA) that contains diverse underwater images and the corresponding utility scores, and then propose an effective deep UIUA network (DeepUIUA) that can predict the utility of underwater images well. In summary, the main contributions of this paper are as follows:

• To the best of our knowledge, we are the first to construct a large-scale Object Detection-oriented Underwater Image Utility Assessment (OD-UIUA) dataset for the task of UIUA. The OD-UIUA dataset comprises 1200 raw underwater images, corresponding 12,000 enhanced images generated by 10 representative UIE algorithms, and 13,200 utility scores for all images in the dataset. The construction of such a dataset not only provides a fair and reliable benchmark platform for the UIUA task but also inspires further innovations of machine vision-oriented UIE algorithms.
• Unlike vanilla UIQA methods that mainly focus on visual perception-oriented image quality evaluation, we train a deep UIUA network (DeepUIUA) specifically tailored to assess the utility of underwater images for machine vision tasks (in particular, the object detection task). DeepUIUA is capable of accurately capturing object region-oriented features to facilitate accurately predicting the utility score of underwater images. Extensive experiments demonstrate that our proposed DeepUIUA model achieves superior predictive accuracy compared with the existing state-of-the-art no-reference image quality assessment (NR-IQA) methods in assessing the utility of underwater images.

The remaining parts of this paper are organized as follows: Section 2 reviews materials and methods. Section 3 presents the results. Section 4 introduces the discussion. Finally, Section 5 draws the conclusion of this paper.

2. Materials and Methods

2.1. Related Works

2.1.1. Underwater Image Dataset

The collection and analysis of underwater image data are of great importance to marine scientific research. High-quality underwater image datasets not only provide researchers with accurate and reliable data, but also expand the use of underwater image technology in various application scenarios [27]. For this reason, many researchers have proposed various underwater image datasets in recent years. For example, Liu et al. [28] built a multi-view underwater imaging system, collected more than 4000 real underwater images, and divided them into three subsets: the Underwater Image Quality Set (UIQS), the Underwater Color Cast Set (UCCS), and the Underwater Higher-Level Task-driven Set (UHTS), which were investigated to address three challenging enhancement requirements. Li et al. [29] also compiled a dataset by collecting 45 underwater images from existing datasets and real underwater scenes for testing purposes. However, a common issue faced by these datasets is the lack of corresponding reference images. To solve this problem, Li et al. [30] constructed an underwater image enhancement dataset containing 950 images, each enhanced by 12 different algorithms, and finally, volunteers selected the best enhancement result as the reference image, generating 890 raw underwater images with reference and 60 challenging images. To advance perceptual image enhancement research, Islam et al. [31] gathered over 20,000 underwater images using 7 different camera devices, comprising more than 12,000 paired images and 8000 unpaired images. More recently, Jiang et al. [23] built a quality evaluation dataset of UIE containing subjective annotations, which contained 100 raw underwater images and their 1000 enhanced images processed by 10 UIE algorithms, and each enhanced image carries a subjective ranking score, which provides the basic truth for underwater image enhancement quality evaluation. Furthermore, Liu et al. [32] created a large-scale UIQA database by collecting 5369 underwater images from existing underwater computer vision studies, covering a broad range of underwater scenes and assigning subjective quality scores to each image.

Although these datasets contribute significantly to UIQA, there remains a notable gap: lacking large-scale datasets with utility scores. This limitation hinders the broader application of underwater images in machine vision tasks, highlighting the urgent need for further research to fill this gap.

2.1.2. Underwater Image Enhancement

Underwater image enhancement significantly improves visual information quality and plays a crucial role in various underwater research fields [33,34]. Existing underwater image enhancement algorithms can be broadly divided into two categories: traditional methods based on handcrafted feature extraction and deep learning-based algorithms. For instance, Fu et al. [35] proposed a two-branch deep learning network designed to compensate global color cast and local contrast degradation and further enhance image quality through compressed histogram equalization. Chen et al. [36] used a convolutional neural network to estimate backward scattering and direct transmission and by generating enhanced underwater images via a reconstruction module. Ankita et al. [37] developed a lightweight deep neural network that maintains the image enhancement effect while reducing the model computational requirements. Zhang et al. [38] improved image quality by minimizing color loss and applying local adaptive contrast enhancement, combining color shift and maximum decay mapping for local color correction, followed by contrast enhancement using integral and squared integral mapping. Xiao et al. [39] proposed a lightweight network guided by statistical methods, using a bi-statistical white balance module for color distortion correction and adaptive contrast enhancement through a multi-color space stretching module. Guo et al. [40] investigated the impact of ranked UIQA methods, developing a loss function to guide image enhancement effect. Li et al. [41] designed a quality discriminative network containing a Siamese multilayered inference structure, improving the generation effect by comparing the qualities of multiple enhancement candidates through a comparative learning framework. Jiang et al. [42] introduced a two-stage lightweight UIE network. The first stage decomposes the complex underwater degradation into smaller subproblems, and the second stage boosts detail perception using multi-branch color enhancement and pixel attention modules. Zhang et al. [43] performed color correction through decay mapping, combined the maximum information entropy and fast integrated optimization strategies to enhance the global and local contrast of the image, and finally combined the high- and low-frequency components of different scale images through a weighted wavelet fusion strategy to generate high-quality underwater images. Rao et al. [44] proposed a two-stage enhancement approach that performs color compensation through a probabilistic color compensation network, and then performed image enhancement. This network improves color performance and enhances model robustness by estimating color probability distributions through multi-scale texture and color feature fusion.

While these methods perform well in enhancing visual visibility, they primarily focus on human visual perception, leaving challenges in optimizing the utility of underwater images in machine vision applications. However, some researchers are now beginning to study the relationship between underwater image enhancement and object detection performance. Wang et al. [45] conducted experimental research on the impact of underwater image enhancement algorithms on object detectors, and the results showed that underwater image enhancement can suppress object detection performance, especially when detecting difficult samples. Ashraf et al. [3] conducted a comparative analysis of the detection performance before and after enhancing a single image and found that most enhanced images performed equally or better than the raw image when evaluated separately, but excessive enhancement would reduce detection performance.

2.1.3. Underwater Image Quality Assessment

In recent years, UIQA has gradually become a hot research topic [46]. Yang et al. [47] introduced the first metric for assessing underwater color image quality UCIQE, which combines chromaticity, saturation, and contrast to quantify color cast, blur, and low-contrast issues in underwater images. Panetta et al. [48] proposed UIQM, which is inspired by the human visual system (HVS), to evaluate underwater image quality by measuring color, sharpness, and contrast. Wang et al. [49] proposed CCF, a metric based on the principle of underwater imaging, combining the color index, the contrast index, and the fog density index. Yang et al. [50] improved the existing manual extraction of colorimetry, contrast, and sharpness method and proposed a new quality evaluation index, FDUM, for underwater images. Jiang et al. [23] found that unsatisfactory color bias and luminance would lead to the degradation of visual quality, and thus extracted quality-aware features in the chromaticity and luminance components, and proposed NUIQ through SVM model training. To evaluate underwater image enhancement quality, Guo et al. [51] combined the underwater image imaging model and visual attributes to construct UWEQM, a benchmark for underwater image enhancement quality evaluation. Zheng et al. [24] proposed UIF, combining naturalness, clarity, and structural metrics, and used a saliency pooling strategy to improve the effectiveness of underwater image enhancement quality evaluation. However, previous IQA methods mainly focus on intrinsic image distortions, overlooking artificial distortions introduced by enhancement algorithms. To address this, Wang et al. [52] proposed GLCQE to measure the quality of the enhanced image by constructing the optimal enhancement direction and based on the luminance and chromaticity modules. Liu et al. [53] proposed UIQI, a metric that comprehensively analyzes luminance, chromaticity, sharpness, contrast, fog density, noise, color cast, and color depth. In addition, Liu et al. [32] used a deep learning approach to evaluate underwater image quality from both global and local perspectives, introducing a benchmark dataset and combining attention mechanisms with a visual transformer to better characterize image quality.

However, most of these UIQA methods are designed based on the characteristics of the HVS and focus on the human visual perception-oriented image quality issue [54]. Although these methods fulfill the needs of human vision, they have significant shortcomings in assessing the utility of underwater images in machine vision applications. Therefore, how to develop UIQA methods that can accurately assess the utility of underwater images has become a critical issue that needs to be addressed.

2.2. Construction of OD-UIUA Dataset

By systematically reviewing previous works, we found that the field of UIUA still lacks large-scale datasets with utility scores, and the UIQA methods cannot be used to gauge the utility of underwater images in machine vision applications. In this section, we provide a detailed description of the OD-UIUA dataset, covering raw underwater image collection, enhanced image generation, and utility score generation.

Overall, our constructed OD-UIUA dataset contains 13,200 underwater images and the corresponding underwater image utility scores (UIUSs). The 13,200 underwater images consist of 1200 raw underwater images and 12,000 enhanced images. Section 2.2.1 describes the collection process for the 1200 raw underwater images. Section 2.2.2 details the generation process of the 12,000 enhanced images. Section 2.2.3 describes the acquisition of UIUSs. To provide a comprehensive understanding of the dataset, Section 2.2.4 presents the statistical characteristics and utility analysis of the OD-UIUA dataset.

2.2.1. Raw Underwater Image Collection

To reflect real-world underwater application scenarios for machine vision tasks, we follow the key principle that the collected raw underwater images should cover a diversity of biological categories and the typical underwater degradation types. Specifically, to cover a diversity of biological categories, we select the images containing typical marine organisms such as different types of echini, starfish, holothurians, and scallops. To cover typical underwater degradation types, we select images containing different degrees of color cast, low light, reduced contrast, etc. Based on this principle, we select 1200 raw underwater images from the existing underwater object detection dataset DUO [55]. Sample images in Figure 1 illustrate practical underwater scenarios with biodiversity and typical degradation types.

2.2.2. Enhanced Image Generation

To enable the evaluation of underwater enhanced results with different qualities, we generate $1200\times 10$ = 12,000 enhanced results via ten representative UIE methods in last 5 years. The ten UIE methods contains eight deep learning-based methods and two traditional methods. The eight deep learning-based methods include GLCHE [35], DLIFM [36], Shallow-UWNet [37], USLN [39], NU₂Net [40], CLUIE-Net [41], FiveAPlus-Net [42], and P2CNet [44]. The other two traditional methods include MLLE [38] and WWPF [43]. As shown in Figure 2, the five representative raw underwater images in column (a) and their corresponding ten enhanced results shown in column (b) to column (k) present underwater images with different qualities.

In order to construct the UIUA dataset, it is necessary to obtain the utility scores of all images. In the following section, we describe the utility testing for obtaining the utility scores of these images.

2.2.3. Utility Score Generation

To obtain the UIUSs of 13,200 images and ensure that the utility scores are fair and reliable, the UIUS of each image is averaged by five mean Average Precision (mAP) values generated by five classical object detectors including three single-stage detectors (RetinaNet [56], FCOS [57], and TOOD [58]) and two two-stage detectors (Faster R-CNN [59] and Cascade R-CNN [60]). The calculation of the UIUS can be expressed by the following equation:

$$UIUS={\displaystyle \frac{{\sum }_{i=1}^{5}mA{P}_i}{5}}$$

(1)

where $UIUS$ represents the UIUS of a single image in an OD-UIUA dataset, and $mA{P}_i$ represents the mAP value generated by the i-th object detector.

To obtain reasonable mAP values for underwater images, we construct a new underwater image dataset and retrain all these five detectors. Figure 3 shows the process flowchart for constructing datasets and generating UIUS. This new dataset is divided into 11 subsets: 1 subset for the raw underwater images and other 10 subsets for the enhanced underwater images processed by 10 representative UIE algorithms described in Section 2.1.2. The raw subset contains 7717 images selected from the DUO dataset in which the duplicate images and the images without objects are eliminated. Among these 7717 images, 1200 raw underwater images selected in Section 2.2.1 are regarded as the testing set, and the remaining 6517 images are regarded as the training set. The other 10 enhanced subsets contain 7717 × 10 = 77,170 images generated by the 10 UIE algorithms from the 7717 raw images. In addition, each of the 5 detectors are trained on each training subset individually; i.e., every detector will be trained on the above-mentioned 11 subsets to obtain 11 pre-trained models, finally generating a total of 5 × 11 = 55 pre-trained models. These models are used to assess the performance of object detection (generating mAP values) across different testing subsets, providing a comprehensive understanding of the impact of detection accuracy from UIE. Finally, the UIUS of each image is calculated by averaging the mAP values obtained from the detections made by the 55 pre-trained models on the corresponding subsets.

The OD-UIUA dataset has been constructed after finishing the above utility annotation. In the next section, we analyze the OD-UIUA dataset in detail and explore its statistical properties and utility assessment.

2.2.4. Dataset Analysis

To deeply understand the OD-UIUA dataset, we conduct a comprehensive analysis focusing on the proportions of different object categories, the number of objects, category diversity, UIUS distribution, and the impact of UIE algorithms on UIUS. The detailed results of these analysis are presented below:

First, in order to show the basic information of the OD-UIUA dataset, we analyze the biological composition of the OD-UIUA dataset in terms of the proportions of different object categories. Figure 4 shows the proportions of different object categories in the OD-UIUA dataset. Specifically, the proportions of echini, starfish, holothurians, scallops are $68.1$%, $18.9$%, $10.6$%, and $2.4$%, respectively.

Second, to reflect the complexity of the scenes in the OD-UIUA dataset, we analyze the proportions between the number of objects and categories per image. Figure 5a shows the proportion of the number of objects per image. Specifically, the proportions of 1–2, 3–5, 6–10, and ≥11 are $12.7$%, $27.1$%, $29.1$%, and $31.1$%, respectively. Figure 5b shows the proportion of the number of categories per image. Specifically, the proportions of 1, 2, and ≥3 are $28.7$%, $44.6$%, and $26.7$% respectively. Images containing different categories of organisms not only improve the learning ability of relationships between categories but also promote the generalization ability of UIUA methods.

Subsequently, we further analyze the distribution of UIUS in the OD-UIUA dataset. As illustrated in Figure 6, the UIUSs cover the entire horizontal axis, and this distribution not only intuitively reflects the diversity of the OD-UIUA dataset but also indicates that the UIUA methods trained on the OD-UIUA dataset can effectively evaluate samples with different utility levels.

In addition, analyzing UIUSs has potential value, such as the impact of different UIE algorithms on the performance of image object detection. In the next part, we will analyze around this point.

We present the 11 mean UIUSs of all images in each subset (constructed in Section 2.2.3) in Figure 7, which shows that all the 10 mean UIUSs of 1 enhanced subset are lower than the mean UIUS of a raw subset. This suggests that existing UIE algorithms produce the negative effect on object detection, while the visual quality of the enhanced images is improved by these UIE algorithms. Does this mean that each enhanced image will degrade the performance of object detection?

To answer the above question, we present the UIUSs of 100 consecutive raw underwater images and their corresponding enhanced images, as shown in Figure 8. The 100 consecutive raw underwater images are selected from the OD-UIUA dataset according to their UIUSs from low to high. The results indicate that the 10 UIE algorithms can either increase or decrease UIUS, which reveals the uncertainty of the enhancement effect on object detection performance. This uncertainty, combined with the conclusion from Figure 7, shows that although the UIE algorithms produce the negative effect on the overall performance of object detection, it is still possible for the UIE algorithm to produce the positive effect on the object detection performance of a single image. Meanwhile, these observations imply that no single UIE algorithm can improve the object detection performance across all types of underwater images.

The above analysis presents a new challenge to UIQA methods. Almost all existing UIQA methods focus on the visual perception-oriented image quality issue and cannot be used to gauge the utility of underwater images in machine vision applications. Therefore, it is urgent to propose a UIUA method that can objectively gauge the utility of underwater images. In the next section, we will introduce our proposed UIUA method.

2.3. Proposed DeepUIUA

With the OD-UIUA dataset, we proposed a UIUA method to assess the object detection performance of underwater images, called DeepUIUA. The proposal of DeepUIUA not only aims to assess the utility of underwater images, but also calls for further research on UIUA.

Existing UIQA methods usually focus on the degree of degradation or the type of distortion of the entire image. However, they are not the main factors in improving the UIUA. In contrast, we consider that the UIUA method should focus on the quality of the object region so that the utility score can accurately reflects the utility of the underwater image. To achieve this, DeepUIUA employs a two-stage design that enables the UIUA model to focus on the quality of object regions. The architecture of the proposed DeepUIUA is shown in Figure 9.

2.3.1. Stage1: Object Region-Oriented Feature Capturing Network

In the first stage, a backbone network optimized for focusing on object regions to capture features is obtained via following the architecture of the YOLOX [61] including a backbone network, a neck layer, and a detection head. The following part will introduce the architecture.

Backbone: To enhance the feature extraction capability of the backbone network, we replace the backbone network of YOLOX with ResNet50 [62] in this paper due to the following two reasons: First, ResNet50, a classical convolutional neural network, is widely used in visual tasks by virtue of its excellent feature extraction ability, and choosing it as the backbone network can ensure the reliability of the model. Second, ResNet50 provides abundant pre-trained weights, which can effectively improve the initial performance of the model.

Neck: In order to cope with the common problems of uneven illumination and color shift in underwater environments and improve the ability of the backbone network to focus on underwater objects with different scales, we utilize a Feature Pyramid Network (FPN) and a Path Aggregation Network (PAN) at the neck layer to extract and fuse multi-scale features. The FPN enhances the recognition performance of blurred underwater objects by integrating high-level semantic information through a top–down structure, while PAN increases the expression ability of different scale features through a bottom–up path aggregation, allowing the detection network to be more stable when dealing with underwater objects of different sizes and shapes in the face of a complex underwater scene. As shown in Figure 9, we extract features $F_1$, $F_2$, and $F_3$ from layers 4, 3, and 2 of the backbone network and obtain the first set of multi-scale features $F_1^{\prime }$, $F_2^{\prime }$, and $F_3^{\prime }$ through the FPN. Then the first set of multi-scale features is transmitted to PAN to produce the second set of multi-scale features $F_1^{″}$, $F_2^{″}$, and $F_3^{″}$, which are subsequently input to the detection head for detection. The process of multi-scale feature extraction and fusion can be expressed with the following equation:

$$F_1^{\prime },F_2^{\prime },F_3^{\prime }=FPN(F_1,F_2,F_3)$$

(2)

$$F_1^{″},F_2^{″},F_3^{″}=PAN(F_1^{\prime },F_2^{\prime },F_3^{\prime })$$

(3)

Head: The detection head uses the multi-scale features from the neck layer to perform object classification, bounding box regression, and confidence estimation. Due to the multi-scale characteristic of the features, the detection head can identify both large and small objects, thereby improving the accuracy of detecting objects of different sizes in an underwater environment.

2.3.2. Stage2: Multi-Scale Quality Assessment Network

In the second stage, the predicted UIUSs of underwater images are generated by incorporating the pre-trained backbone network from the first stage and the quality prediction module. The following part will introduce the backbone and the quality prediction module.

Backbone: To ensure that quality scores reflect the underwater images’ object detection performance more accurately, we use the ResNet50 model trained in the first stage as the backbone network in this stage so that image features are focused on underwater object areas. The input image I is processed through the backbone network ${\mathcal{F}}_{pt-backbone}$ to extract features $F_a$, $F_b$, and $F_c$ from layers 2, 3, and 4. The feature extraction can be expressed with the following equation:

$$F_a,F_b,F_c={\mathcal{F}}_{pt-backbone}\left(I\right)$$

(4)

Quality Prediction Module: The quality prediction module utilizes the multi-scale features of underwater images for quality prediction. Shallow features have rich detail information and higher resolution, which can accurately reflect the quality of small objects, while deep features have a larger receptive field and richer semantic information, which can accurately reflect the quality of large objects. The lightweight and concise detection head in the object detection model has the ability to detect different-sized objects through feature maps of different scales, which is also important for UIUA. Therefore, the design of our DeepUIUA draws on this idea to accurately assess the underwater objects utility of different sizes through feature maps of different scales. Specifically, we extract multi-scale features $F_a$, $F_b$, and $F_c$ from layers 2, 3, and 4 of the backbone network and input them to the quality prediction module for prediction. These three multi-scale features are passed through average pooling layers ${\mathcal{F}}_{Avgpool}$ cascaded by three different scales’ fully connected layers ${\mathcal{F}}_{FC}$ to produce three output values. These values are concatenated to form a fused feature vector $\widehat{F}$. Then, $\widehat{F}$ is input into a fully connected layer $FC$, and finally, the utility score q of the image is obtained, represented by the following formula:

$$\widehat{F}=Concat\left({\mathcal{F}}_{FC}\left({\mathcal{F}}_{Avgpool}(F_a,F_b,F_c)\right)\right)$$

(5)

$$q=FC\left(\widehat{F}\right)$$

(6)

Total Loss: In order to optimize the performance of the model, we minimize the $L_1$ loss function between the predicted UIUSs and the baseline UIUSs. The $L_1$ loss function is defined as follows:

$$L_1=\sum _{i=1}^N|q_i-s_i|$$

(7)

where N is the total number of training images, and $q_i$ and $s_i$ denote the predicted UIUSs and baseline UIUS of the i-th training image, respectively.

3. Results

In this section, we first detail the experimental protocol, and then compare the performance of DeepUIUA with that of the SOTA NR-IQA methods.

3.1. Experimental Protocol

3.1.1. Utility Score Generation Details

As stated, the UIUS of each image is generated by averaging the mean Average Precision (mAP) values generated by applying the five object detectors RetinaNet [56], FCOS [57], TOOD [58], Faster R-CNN [59], and Cascade R-CNN [60] on each image. Specifically, these object detectors are implemented on the MMDetection [63] platform. We use the same SGD optimizer with the same parameters for all detectors during training. All momentum values are set to $0.9$, and weight decay values are set to $1\times {10}^{-4}$.

3.1.2. Training Details of DeepUIUA

In the first stage, to train the object detection network, we utilize images from the raw subset as mentioned in Section 2.2.3 and their corresponding object detection labels from the DUO dataset. In addition, to ensure that the network sufficiently utilizes the image detail, the original resolutions of the raw subset images are maintained during this stage. In the second stage, to train the quality prediction network, we utilize images from the OD-UIUA dataset and their corresponding UIUS. Meanwhile, we uniformly adjust the input images to the resolution of 224 × 224. The proposed DeepUIUA is implemented in the PyTorch framework. The Adam optimizer with an initial learning rate of $1\times {10}^{-4}$ and a training batch size of 32 is used for training the proposed DeepUIUA on a server with NVIDIA GTX3090.

3.1.3. Evaluation Metrics

To evaluate the prediction performance of DeepUIUA, we use four widely adopted statistical metrics: Spearman Rank Correlation Coefficient (SRCC), Kendall Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE). Higher values of SRCC, KRCC, and PLCC and lower values of RMSE indicate the better predictive performance of the method.

3.2. Performance Comparison with the State-of-the-Art Methods

We compare DeepUIUA with 14 SOTA deep learning-based NR-IQA methods, including CNNIQA [64], IQA-CNN+ [65], WaDIQaM-NR [66], DBCNN [67], HyperIQA [68], MetaIQA [69], TRes [70], DACNN [71], VCRNet [72], DEIQT [73], StairIQA [74], TTA-IQA [75], TOPIQ [76], and ATUIQP [32]. The first 13 methods focus on natural image visual quality assessment, and the last method, ATUIQP, is specifically for underwater image visual quality assessment. For the evaluation experiments, the OD-UIUA dataset is divided into 6 groups. Each group contains 2200 images: 200 raw underwater images and 2000 corresponding enhanced images. The performance of model is evaluated via cross-validation. In each round of experiments, one group is used as the testing set, and the remaining five groups are used as the training set. We conducted training and testing on each group in turn, and the final evaluation metric is the average results across the 6 experiments.

Table 1. Performance comparison of 14 state-of-the-art methods and the proposed DeepUIUA on the OD-UIUA dataset.

Comparison Method	PLCC	SRCC	KRCC	RMSE
CNN-IQA [64]	0.4625	0.4450	0.3052	0.1536
IQA-CNN++ [65]	0.4529	0.4435	0.3037	0.1548
WaDIQaM-NR [66]	0.4112	0.3729	0.2537	0.1632
DBCNN [67]	0.4788	0.4647	0.3210	0.1569
HyperIQA [68]	0.5301	0.5036	0.3517	0.1470
MetaIQA [69]	0.5070	0.4907	0.3423	0.1538
TRes [70]	0.3444	0.3410	0.2314	0.1895
DACNN [71]	0.4413	0.4117	0.2817	0.1559
VCRNet [72]	0.4580	0.4279	0.2947	0.1557
DEIQT [73]	0.5287	0.4912	0.3416	0.1549
StairIQA [74]	0.5307	0.4971	0.3486	0.1474
TTA-IQA [75]	0.3617	0.3270	0.2220	0.1803
TOPIQ [76]	0.5179	0.4891	0.3407	0.1528
ATUIQP [32]	0.2512	0.2214	0.1476	0.1737
Ours	0.6004	0.5595	0.3963	0.1386

presents the comparison results between DeepUIUA and these 14 SOTA NR-IQA methods.

Three key conclusions can be drawn from the result in Table 1. First, the performance of NR-IQA methods utilizing pre-trained initialization remarkably surpasses that of the NR-IQA methods without pre-trained initialization. Specifically, The NR-IQA methods, such as StairIQA, HyperIQA, and DEIQT, integrating pre-trained initialization, perform better than the NR-IQA methods, such as CNN-IQA, WaDIQaM-NR, and ATUIQP, without pre-trained initialization across all metrics. This is because pre-trained initialization provides the network with prior knowledge, which enables the network to better understand and adapt to the features of underwater images. Second, ATUIQP, which is specifically designed for UIQA, performs worst among all compared methods. Finally, our proposed DeepUIUA achieves the best performance compared with existing SOTA NR-IQA methods. The combination of a pre-training technique and multi-scale features designed for underwater machine vision tasks enables DeepUIUA to accurately assess the utility of an underwater image.

As shown in Figure 10, to intuitively visualize the consistency between UIUS and DeepUIUA prediction scores, we selected six underwater images representing different levels of utility from the OD-UIUA dataset. With the increase in UIUS, the localization accuracy and image confidence are obviously improved. For example, comparing image (a) with image (f), image (a), with UIUS $0.341$, results in poor object localization accuracy and a low confidence level of 50–65%, as marked by the boxes. In contrast, image (f), with UIUS $0.807$, shows accurate object localization with a high confidence level of above 80%. In addition, the quality scores predicted by our proposed DeepUIUA are highly consistent with UIUS, further demonstrating the effectiveness of DeepUIUA in UIUA.

To evaluate the computational complexity of each NR-IQA model, we conduct a systematic comparative analysis of parameter counts, FLOPs, and running times of existing NR-IQA methods to assess their computational efficiency. To ensure fairness in the comparison, all NR-IQA methods are tested on a 224 × 224 image. Additionally, the running times of these methods are implemented on an NVIDIA RTX 4090 GPU. The detailed results are shown in

Table 2. Parameters, FLOPs, and running times of different NR-IQA methods.

Comparison Method	Params (MB)	FLOPs (G)	Running Times (s)
CNN-IQA [64]	0.35	0.73	0.011
IQA-CNN++ [65]	0.05	0.08	0.007
WaDIQaM-NR [66]	3.28	4.98	0.038
DBCNN [67]	16.50	15.31	0.116
HyperIQA [68]	4.34	27.38	0.015
MetaIQA [69]	1.83	13.24	0.034
TRes [70]	8.39	34.46	0.053
DACNN [71]	0.33	2.90	0.143
VCRNet [72]	10.27	11.41	0.043
DEIQT [73]	4.26	22.77	0.019
StairIQA [74]	5.11	30.49	0.015
TTA-IQA [75]	8.30	34.40	0.043
ATUIQP [32]	4.87	28.10	0.019
Ours	25.61	4.13	0.106

. From the table, we can see that although IQA-CNN++ demonstrates the best performance in the comparison of computational complexities, it performs poorly in terms of the quality assessment metrics. However, although our proposed method does not have an advantage in computational complexity, it achieves state-of-the-art performance in terms of quality assessment metrics. In summary, our proposed method strikes the best balance between performance and computational complexity.

4. Discussion

4.1. Ablation Study

4.1.1. Ablation of the Proposed Components

To validate the effectiveness of the components in DeepUIUA, we conduct a series of ablation experiments. The experimental results are summarized in

Table 3. Composition and performance comparison of ablation models.

	BL	PT	MSQP	PLCC	SRCC	KRCC	RMSE
AM1	✓			0.5202	0.4951	0.3459	0.1488
AM2	✓	✓		0.5663	0.5263	0.3701	0.1443
AM3	✓		✓	0.5395	0.5110	0.3570	0.1481
AM4	✓	✓	✓	0.6004	0.5595	0.3963	0.1386

. Specifically, “BL” (Baseline) refers to the general pre-trained ResNet50 network without the training on underwater object detection datasets; this is the setting of the baseline, which provides a baseline comparison for subsequent additions of components. “PT” refers to the Pre-Trained backbone network trained by the first stage of DeepUIUA, and “MSQP” refers to the Multi-Scale Quality Prediction module in the second stage of DeepUIUA. Our proposed network utilizes these two components to improve the underwater image utility assessment, and therefore requires a comparison in an ablation study to validate the effectiveness of these two components. Four ablation models, i.e., AM1∼AM4, respectively represent with BL only, with BL+PT, with BL+MSQP, and with BL+PT+MSQP, i.e., the whole network of stage 2.

From the table, we can see that adding any components to the baseline can effectively improve the performance of the model. Specifically, by comparing the ablation model AM1 with AM2 and AM3, we observe that both PT and MSQP improve the performance of the model, with PT showing a more remarkable impact. For example, compared with the model AM1, the PLCC value of the models AM2 and AM3 increased by $0.0461$ and $0.0193$, respectively. Comparing the ablation model AM4 with AM2 and AM3, the combination of PT and MSQP is able to further improve the overall performance of the model. For example, compared with the models AM2 and AM3, the PLCC value of the model AM4 increased by $0.0341$ and $0.0609$, respectively. To sum up, both PT and MSQP, which combine the pre-training technique and multi-scale features, play a crucial role in improving the performance of the UIUA model.

4.1.2. Visual Explanation

In order to deeply understand the merits of simultaneously using PT and MSQP from another view, we present the visual activations on six representative samples from the OD-UIUA dataset, as shown in Figure 11. These visual activations are the Grad-CAM maps of the features outputted by the four ablation models. The six representative samples in column (a) include two raw underwater images in the first row and forth row and their corresponding four enhanced images in the remaining rows. The Grad-CAM activation maps of these six samples in AM1∼AM4 are shown in column (b)∼(e). Specifically, comparing the images in the first 3 rows of column (b) and column (c) and the images in the last 3 rows of column (b) and column (d), the activation in echinus regions of single using PT and single using MSQP is better than that of single using BL. Meanwhile, comparing all the images of column (b)∼(d) with that of column (e), the activation on echinus regions of simultaneously using PT and MSQP is better than that of others, which indicates that simultaneously using PT and MSQP effectively guides the network to focus on the underwater object and provide these critical features for prediction. Thus, our proposed DeepUIUA simultaneously consisting PT and MSQP is highly suitable to analyze the utility of underwater images.

4.2. Limitations

Based on the previous experimental verification, the effectiveness of the DeepUIUA method has been fully demonstrated. However, in practical applications, there is still space for optimization of this method in particular situations. In the three typical images shown in Figure 12, there is a notable disparity between the UIUS and the predicted score, which may be due to the impact of various degradation and distortion factors in the underwater environment, resulting in biased prediction results. In addition, due to the complexity and diversity of underwater environments, the current OD-UIUA dataset has limitations in scale. In future research, we intend to expand our exploration of different underwater environments and biological species by increasing the size of the dataset and further optimizing the assessment algorithm. At the same time, based on the evaluation results of algorithms, we will implement targeted improvements to underwater image enhancement and object detection techniques, thereby promoting the overall progress of underwater image processing technology.

5. Conclusions

In this work, we explore the UIUA from two aspects. First, we constructed the large-scale OD-UIUA dataset, which contains 1200 raw underwater images, corresponding to 12,000 enhanced results by 10 representative UIE algorithms, and 13,200 utility scores for all images. The proposed OD-UIUA dataset not only provides a fair and reliable benchmark for the UIUA task but also promotes further development of machine vision-oriented UIE algorithms. Second, we propose a two-stage UIUA model, DeepUIUA, which accurately predicts utility scores by fully employing the merits of a pre-training technique and multi-scale features. Extensive performance comparisons with 14 SOTA NR-IQA methods on the OD-UIUA dataset demonstrate the superiority of the proposed DeepUIUA, exceeding suboptimal methods by more than 10% in multiple evaluation metrics, which is expected to play a significant role in future machine vision applications.