Unpaired Underwater Image Enhancement Based on CycleGAN

Abstract

Underwater image enhancement recovers degraded underwater images to produce corresponding clear images. Image enhancement methods based on deep learning usually use paired data to train the model, while such paired data, e.g., the degraded images and the corresponding clear images, are difficult to capture simultaneously in the underwater environment. In addition, how to retain the detailed information well in the enhanced image is another critical problem. To solve such issues, we propose a novel unpaired underwater image enhancement method via a cycle generative adversarial network (UW-CycleGAN) to recover the degraded underwater images. Our proposed UW-CycleGAN model includes three main modules: (1) A content loss regularizer is adopted into the generator in CycleGAN, which constrains the detailed information existing in one degraded image to remain in the corresponding generated clear image; (2) A blur-promoting adversarial loss regularizer is introduced into the discriminator to reduce the blur and noise in the generated clear images; (3) We add the DenseNet block to the generator to retain more information of each feature map in the training stage. Finally, experimental results on two unpaired underwater image datasets produced satisfactory performance compared to the state-of-the-art image enhancement methods, which proves the effectiveness of the proposed model.

1. Introduction

With the rapid development of marine resources, underwater robots are necessary to replace humans to work in the complex underwater environment. The underwater robot mainly relies on its visual ability to achieve several tasks, such as object recognition, localization, 3D reconstruction, and route guidance. Due to the light absorption and scattering properties in water, underwater images usually contain color distortion and low contrast. Therefore, how to enhance underwater images becomes an urgent problem for practical underwater applications [1].

Over recent decades, underwater image enhancement has attracted an increasing amount of attention. Wang et al. [2] divided underwater image enhancement methods into three main categories: spatial-domain methods, transform-domain methods, and the popular deep learning-based methods.

The spatial-domain methods usually tune the grayscale range of one image to enhance its contrast and reduce the color distortion [3]. Traditional methods include gray world [4], white balance [4], automatic white balance [5], histogram equalization, adaptive histogram equalization [6], contrast limited adaptive histogram equalization [7], and its variations. Although these methods have had success in enhancing degraded images, they still have significant limitations for severely degraded underwater images, which introduce red artifacts and noise.

The transform-domain methods transfer an underwater image to the frequency domain, then, they enhance the image contrast by amplifying the high-frequency information and suppressing the low-frequency information. Classic transform-domain methods include a low-pass filter [8], high-pass filter [9], homomorphic filter [10], and wavelet transform [11,12,13]. Although these methods decrease the noise and enhance the contrast of an underwater image, the performance of color correction is bad.

The above two categories of methods just enhance each underwater image independently, without learning procedures. Deep learning-based methods can exploit an end-to-end automatic training mechanism to enhance underwater images, which learns the intrinsic underwater features from a set of underwater images. In [14], they replaced the handcrafted features with the nonparametric deep features for the image representation. Other researchers [15,16] introduced the convolutional neural network (CNN) into underwater image enhancement applications. A residual CNN was further proposed in [17]. Furthermore, [18] provided a deep pixel-to-pixel network by designing an encoding–decoding framework. In [19], they utilized domain adversarial learning to enhance underwater images. In [20,21], they improved the quality of visual underwater scenes using Generative Adversarial Networks (GAN), and then [22] proposed a fusion adversarial network. Finally, Hu et al. [23] introduced the natural image quality evaluation to a supervised generative adversarial network.

These methods improved the visual effect and quality of underwater images, but they require a large amount of paired data, i.e., each degraded image has a corresponding clear image. Paired data is difficult to obtain in an underwater environment, which also causes difficulty for underwater image enhancement. Therefore, researchers usually use synthetic data to construct paired data. Figure 1 shows some samples of unpaired underwater images.

To solve the problem of deep learning-based underwater image enhancement methods requiring paired data, we propose a novel underwater cycle generative adversarial network (UW-CycleGAN) for image enhancement, which just needs one set of unpaired underwater degraded images and clear images to train the proposed model. A brief illustration of UW-CycleGAN is shown in Figure 2.

The main contributions of this paper are briefly summarized as follows:

• We introduce a content loss regularizer into the generator in CycleGAN, which keeps more detailed information in the corresponding generated clear image. This strategy is different from CartoonGAN [24];
• We add a blur-promoting adversarial loss regularizer into the discriminator in CycleGAN, which reduces the effects of blur and noise and enhances the image clarity;
• We exploit the improved DenseNet Block in the generator to strengthen the forward transfer of feature maps, so that every feature map can be utilized;
• We test our proposed UW-CycleGAN on different types of underwater images and obtain a satisfactory performance.

We develop an end-to-end underwater image enhancement system. The structure of this paper is organized as follows: The necessary knowledge about underwater image enhancement is reviewed in Section 2. An improved underwater CycleGAN model for unpaired data, so-called UW-CycleGAN, is proposed in Section 3. The experimental results on two underwater image datasets are illustrated in Section 4. Finally, we conclude this paper in Section 5.

2. Underwater Image Enhancement

As we mentioned above, capturing paired data in the underwater environment is difficult. To study the intrinsic relationship between the degraded image and the corresponding clear image, some researchers designed a simplified physical model according to the refraction, scattering, and attenuation properties of light [25],

$$\begin{array}{c}\hfill I_{\lambda }\left(x\right)=J_{\lambda }\left(x\right)·t_{\lambda }\left(x\right)+(1-t_{\lambda }\left(x\right))·B_{\lambda },\lambda \in \{r,g,b\}\end{array}$$

(1)

where $I_{\lambda }\left(x\right)$ denotes the degraded image captured by underwater cameras, $J_{\lambda }\left(x\right)$ means the corresponding restored clear image, $t_{\lambda }\left(x\right)$ is the medium transmission map, $B_{\lambda }$ represents the well-proportioned background light, and $\lambda$ gives the light wavelength.

In order to enhance $J_{\lambda }\left(x\right)$, the key problem of the traditional physical models is to estimate $t_{\lambda }\left(x\right)$ and $B_{\lambda }$, since only image $I_{\lambda }\left(x\right)$ is known. Although this physical model does not need paired data, some assumptions and prior knowledge are required to evaluate $t_{\lambda }\left(x\right)$ and $B_{\lambda }$, which severely limits the practical applications.

In recent years, many researchers have applied CNN to process underwater images and achieved good results in underwater image enhancement applications. However, CNN based methods need paired data to train their network models, and researchers have to use synthetic data instead. Fortunately, CycleGAN can utilize unpaired data for the conversion of image style, which offers a new direction for underwater image enhancement.

3. Underwater CycleGAN (UW-CycleGAN)

Deep learning-based image enhancement methods usually need paired underwater images to train network models. To solve this problem, we propose a CycleGAN-based underwater image enhancement method (UW-CycleGAN), which can utilize unpaired data to train its model.

Suppose we have the unpaired degraded image set X and clear image set Y. One complete procedure of UW-CycleGAN is shown in Figure 2:

(1)

The mapping function G generates the clear image $G\left(x\right)$ from $x\in X$.

(2)

Another mapping function F reconstructs the degraded image x by $G\left(x\right)\to F\left(G\right(x\left)\right)$.

(3)

Discriminator $D_Y$ judges whether the generated image $G\left(x\right)$ and clear image y derive from the same distribution.

In addition, $y\to F\left(y\right)\to G\left(F\right(y\left)\right)$ and $D_X$ are the similar inverse processes, which ensures the model invertibility. We display some samples of x, $G\left(x\right)$, and $F\left(G\right(x\left)\right)$ in Figure 3, respectively.

3.1. Loss Function

Zhu et al. [26] proposed a CycleGAN framework to achieve unpaired image-to-image translation, which consisted of adversarial loss and cycle consistency loss.

The adversarial loss restricts the generated image $G\left(x\right)$ and $F\left(y\right)$ to derive from the same distribution with the clear image y and degraded image x, respectively:

$$\begin{array}{cc}\hfill L_{adv}(G,D_Y)& =E_{y\sim P_{data}\left(y\right)}\left[{\left(D_Y\left(y\right)\right)}^2\right]\hfill \\ \hfill & +E_{x\sim P_{data}\left(x\right)}\left[{(1-D_Y\left(G\left(x\right)\right))}^2\right]\hfill \\ \hfill L_{adv}(F,D_X)& =E_{x\sim P_{data}\left(x\right)}\left[{\left(D_X\left(x\right)\right)}^2\right]\hfill \\ \hfill & +E_{y\sim P_{data}\left(y\right)}\left[{(1-D_X\left(F\left(y\right)\right))}^2\right]\hfill \end{array}$$

(2)

where $P_{data}\left(x\right)$ and $P_{data}\left(y\right)$ represent the distributions of underwater degraded images and clear images, respectively.

The cycle consistency loss ensures the reconstructed images are similar to the input images,

$$\begin{array}{cc}\hfill L_{cyc}(G,F)& =E_{x\sim P_{data}\left(x\right)}[\Vert F\left(G\left(x\right)\right)-{x\Vert }_1]\hfill \\ \hfill & +E_{y\sim P_{data}\left(y\right)}[\Vert G\left(F\left(y\right)\right)-{y\Vert }_1].\hfill \end{array}$$

(3)

where ${\parallel ·\parallel }_1$ means the ${\ell }_1$-norm.

3.1.1. Content Loss

The cycle consistency loss (3) only minimizes the difference between the input image x (or y) and its reconstructed image $F\left(G\right(x\left)\right)$ (or $G\left(F\right(y\left)\right)$), which ignores whether the generated image $G\left(x\right)$ (or $F\left(y\right)$) is visually similar to x (or y). Therefore, we add the content loss regularizer measured by ${\ell }_1$ norm,

$$\begin{array}{cc}\hfill L_{con}(G,F)& =E_{x\sim P_{data}\left(x\right)}[\Vert G\left(x\right)-{x\Vert }_1]\hfill \\ \hfill & +E_{y\sim P_{data}\left(y\right)}[\Vert F\left(y\right)-{y\Vert }_1]\hfill \end{array}$$

(4)

However, the above function makes the generated image $G\left(x\right)$ (or $F\left(y\right)$) too similar to the input image x (or y) due to the element-wise subtraction, retaining almost all the information of the input image x (or y). We want to keep the detailed information of the input image x (or y) unchanged, meanwhile, calibrating the image color to enhance the visual quality of the generated image $G\left(x\right)$ (or $F\left(y\right)$).

In order to achieve this purpose, a VGG19 pretraining network is used to extract the $conv4\_4$ layer feature maps of the input and generated images. We also employ ${\ell }_1$-norm to measure the content loss, since ${\ell }_1$-norm is more robust to noise and outliers, which can recover the underwater image details well. Thus, the new content loss regularizer is rewritten as,

$$\begin{array}{cc}\hfill L_{con}(G,F)& =E_{x\sim P_{data}\left(x\right)}[\Vert VGG\left(G\left(x\right)\right)-{VGG\left(x\right)\Vert }_1]\hfill \\ \hfill & +E_{y\sim P_{data}\left(y\right)}[\Vert VGG\left(F\left(y\right)\right)-{VGG\left(y\right)\Vert }_1]\hfill \end{array}$$

(5)

where $VGG(·)$ denotes the VGG19 feature map in this paper.

3.1.2. Blur-Promoting Adversarial Loss

Although the content of the image generated by G is consistent with its corresponding input image, a large amount of noise and blur are also generated at the same time, which effects the visual performance. We need to make the discriminator robust to blur. Therefore, a blur dataset Z is constructed by adding Gaussian blur to the clear image dataset Y. The discriminator $D_Y$ should judge $z\in Z$ as the fake image and y as the real image, so that the images generated by generator G can be clearer. With this idea, the adversarial loss can be rewritten as follows:

$$\begin{array}{cc}\hfill L_{badv}(G,D_Y)& =E_{y\sim P_{data}\left(y\right)}\left[{\left(D_Y\left(y\right)\right)}^2\right]\hfill \\ \hfill & +E_{z\sim P_{data}\left(z\right)}\left[{(1-D_Y\left(z\right))}^2\right]\hfill \\ \hfill & +E_{x\sim P_{data}\left(x\right)}\left[{(1-D_Y\left(G\left(x\right)\right))}^2\right],\hfill \end{array}$$

(6)

where $P_{data}\left(z\right)$ represents the distribution of underwater clear images with Gaussian blur.

3.1.3. Full Loss Funtion

Finally, we construct the full loss function of UW-CycleGAN as follows,

$$\begin{array}{cc}\hfill \mathcal{L}(G,F,D_X,D_Y)& =L_{badv}(G,D_Y)+L_{adv}(F,D_X)\hfill \\ \hfill & +L_{cyc}(G,F)+L_{con}(G,F),\hfill \end{array}$$

(7)

where the generator G, F and the discriminator $D_X$, $D_Y$ can be updated by

$$\begin{array}{cc}\hfill G^{*},F^{*}& =arg\underset{G,F}{min}\underset{D_x,D_y}{max}\mathcal{L}(G,F,D_X,D_Y).\hfill \end{array}$$

(8)

It should be noted, we equally treat each loss regularizer and never set several hyperparameters to tune experimental results.

3.2. Network Architectures

As illustrated in Figure 2, our UW-CycleGAN network architecture consists of two main generators and two discriminators.

Generators G and F have the same network structure with different parameters, which directly adopts the encoder–decoder structure in this paper. Firstly, one flat convolution stage with convolution kernel size of $7\times 7$ and step length of 1 and two down-convolution stages with convolution kernel size of $3\times 3$ and step length of 2 are exploited to spatially compress and encode the input images. Then, three DenseNet blocks are used to transfer the feature maps and preserve their high-level features. The detailed structure of the DenseNet block is shown in Figure 4b [27]. In the T layer, with 128 convolution kernels size of $1\times 1$ and step length of 1, we reduce the size of feature maps from $64\times 64\times 256$ to $64\times 64\times 128$. In the L1 layer, L1 includes 64 convolution kernels size of $1\times 1$ and step length 1 and 16 convolution kernels size of $3\times 3$ and step length 1, so the size of the output feature maps is $64\times 64\times 16$. Then, we concatenate the output of the T layer and the output of the L1 layer to obtain a feature map size of $64\times 64\times 144$ as the inputs for the L2 layer. Similarly, the outputs of the L1 and L2 layers are concatenated as the input for the L3 layer. After several similar operations, we obtain the output size of $64\times 64\times 256$ in the L8 layer. Finally, the generated clear images are reconstructed by two up-convolutions, which contain one convolution kernel size of $3\times 3$ and step length $1/2$ and one final convolution kernel size of $7\times 7$ and step length 1.

Discriminators $D_X$ and $D_Y$ also contain the same network structure with different parameters. In the discriminator network, a Markov discriminator, which comprises five full convolution layers outputs a “0–1” indicator matrix size of $70\times 70$ and then calculates the mean value of all elements in the matrix as the real/fake output at last.

4. Experiment and Evaluation

In this section, UW-CycleGAN is tested on two real-world unpaired underwater image datasets and is compared with several classic image enhancement methods to evaluate the superiority of UW-CycleGAN. Finally, the ablation experiments verify the importance of each component in our UW-CycleGAN model.

4.1. Datasets and Metrics

URPC2019 (http://www.cnurpc.org/a/js/2019/0805/125.html accessed on 17 December 2021) contains over 4000 underwater images [28], and we took a subset in this paper. We chose 670 underwater images as the training set, in which 335 degraded images belonged to training set X, and the remaining 335 clear images belonged to training set Y. There was no paired relationship between training sets X and Y. The Gaussian blur set E was formed by performing a Gaussian blur operation on the previous training set Y. The testing set consisted of 70 degraded images. We set the color image size of both the training set and the test set as $256\times 256\times 3$.

EUVP (http://irvlab.cs.umn.edu/resources/euvp-dataset accessed on 17 December 2021) contains over 6446 underwater human images. We choose 405 degraded images as the training set X and 405 clear images as the training set Y. The testing set consisted of 200 degraded images. The color image size was also set as $256\times 256\times 3$.

To fairly assess these image enhancement methods from different aspects, we selected three standard metrics, which were average gradient (AG) [29], information entropy (IE) [30], and underwater image quality measure (UIQM) [31]. Lower values of IE reflect better performance, while AG and UIQM are the opposite. The entire network was coded in the pytorch framework and implemented on a workstation with 8 Nvidia Tesla P100 GPUs.

4.2. Experimental Assessment

If only the clear image set Y was used to train the discriminator, the generated image usually had an obvious blur. To solve this problem, we exploited the Gaussian blur set Z to train the discriminator, and the generator could output clear images well.

We compared the proposed model with three traditional underwater image enhancement methods:

• De-scattering Underwater image (Deunderwater) [32]
• Diving into Haze-Lines: Color Restoration of Underwater Images (HL) [33]
• Unsupervised Color Correction Method (UCM) [34]

and three deep learning-based methods:

• Fast Underwater Image Enhancement for Improved Visual Perception (FUnIE-GAN-UP) [35]
• Generative Adversarial Networks for Photo Cartoonization (CartoonGAN) [24]
• Unpaired Image-to-Image Translation using Cycle Consistent Adversarial Networks (CycleGAN) [26].

Figure 5 displays the enhancement results of five underwater scene images on the URPC dataset. Deunderwater recovered the image color to a certain extent, but the contrast and details of the generated image were not good. Although HL restored the image detailed information well, the generated image had the problems of poor contrast and color distortion. UCM and FUnIE-GAN-UP recovered the image color and detail well, while the contrast was relatively bad. CartoonGAN and CycleGAN improved the image color and contrast excellently, but blur existed in the image detail. Our UW-CycleGAN obtained good performance in image contrast, color, and detail.

Figure 6 shows the vision enhancement results of underwater human images on the EUVP dataset. Obviously, Deunderwater, HL and UCM had color distortion problems. FUnIE-GAN-UP, CatroonGAN, and CycleGAN performed reasonably well, and their enhanced images were comparable to our UW-CycleGAN, but UW-CycleGAN was still the best in terms of the image detail and clarity.

Table 1. Quality evaluation of all methods on the URPC dataset.

	AG ↑	IE ↓	UIQM ↑
Deunderwater	7.5047	7.8178	5.1460
HL	7.3021	7.4033	4.0719
UCM	4.9102	7.3955	3.8221
FUnIE-GAN-UP	5.9444	7.3819	4.2130
CartoonGAN	4.9079	7.2567	4.4997
CycleGAN	6.4737	7.2785	4.8380
UW-CycleGAN	7.6345	7.1824	5.1689

and

Table 2. Quality evaluation of all methods on the EUVP dataset.

	AG ↑	IE ↓	UIQM ↑
Deunderwater	2.4945	7.7830	1.5500
HL	2.0565	7.4271	1.5769
UCM	2.5489	7.2451	2.0124
FUnIE-GAN-UP	2.6014	7.3463	0.9782
CartoonGAN	2.7224	6.7422	2.0883
CycleGAN	2.9969	6.7452	2.6075
UW-CycleGAN	3.1370	6.4827	2.7497

estimate the underwater image enhancement performance of all methods under three standard metrics. Although the value performance of traditional methods (Deunderwater, HL, and UCM) in both tables are not bad, their visualization performance in Figure 5 and Figure 6 are obviously unsatisfactory. Deep learning-based methods work relatively steadily in both value and visulization performance. Our UW-CycleGAN obtained the best experimental results measured by all objective evaluation metrics.

4.3. Ablation Experiments

We designed a set of ablation experiments to further analyze the importance of each module in our UW-CycleGAN method, and the experimental results are shown in Figure 7 and

Table 3. Ablation Experiments: Quality evaluation of all methods on URPC dataset.

	AG ↑	IE ↓	UIQM ↑
w/o $L_{Content}$	7.3567	7.3727	5.1268
w/o $L_{Blur}$	7.5490	7.2750	5.1530
$G_{ResNet}$	6.9984	7.2864	5.0271
UW-CycleGAN	7.6345	7.1824	5.1689

. We introduce each ablation experiment as follows:

(i) w/o $L_{Content}$: We removed the content loss from UW-CycleGAN, which led to serious detail loss and image blur in the generated images.

(ii) w/o $L_{Blur}$: Without the blur-adversarial loss, the generated images remain intact but had slight blurring.

(iii) $G_{ResNet}$: DenseNet block in UW-CycleGAN was replaced by ResNet-block. Although the subjective difference between $G_{ResNet}$ and our UW-CycleGAN is not obvious in Figure 7, the objective evaluation results in Table 3 verify the advantages of DenseNet-block.

In the above, the effect of each component in our UW-CycleGAN was verified.

5. Conclusions

Underwater vehicle vision has important research value in underwater applications. We proposed an end-to-end underwater image enhancement method for unpaired data (UW-CycleGAN). To be specific, we firstly added a content loss regularizer to the generator in traditional CycleGAN through a VGG19 pretraining network. Then, a blur-promoting adversarial loss regularizer was adopted in the discriminator. Finally, we replaced the commonly used ResNet-block in CycleGAN with the DenseNet block in the coding layer. Compared with several image enhancement methods, our proposed methods restored the underwater degraded images with blue-green background and blur into clear images effectively. We also performed ablation experiments to verify the importance of each module in UW-CycleGAN.