Hybrid Regularized Variational Minimization Method to Promote Visual Perception for Intelligent Surface Vehicles Under Hazy Weather Condition

Abstract

Intelligent surface vehicles, including unmanned surface vehicles (USVs) and autonomous surface vehicles (ASVs), have gained significant attention from both academic and industrial communities. However, shipboard maritime images captured under hazy weather conditions inevitably suffer from a blurred, distorted appearance. Low-quality maritime images can lead to negative effects on high-level computer vision tasks, such as object detection, recognition and tracking, etc. To avoid the negative influence of low-quality maritime images, it is necessary to develop a visual perception enhancement method for intelligent surface vehicles. To generate satisfactory haze-free maritime images, we propose development of a novel transmission map estimation and refinement framework. In this work, the coarse transmission map is obtained by the weighted fusion of transmission maps generated by dark channel prior (DCP)- and luminance-based estimation methods. To refine the transmission map, we take the segmented smooth feature of the transmission map into account. A joint variational framework with total generalized variation (TGV) and relative total variation (RTV) regularizers is accordingly proposed. The joint variational framework is effectively solved by an alternating-direction numerical algorithm, which decomposes the original nonconvex nonsmooth optimization problem into several subproblems. Each subproblem could be efficiently and easily handled using the existing optimization algorithm. Finally, comprehensive experiments are conducted on synthetic and realistic maritime images. The imaging results have illustrated that our method can outperform or achieve comparable results with other competing dehazing methods. The promoted visual perception is beneficial to improve navigation safety for intelligent surface vehicles under hazy weather conditions.

1. Introduction

Intelligent surface vehicles, e.g., unmanned surface vehicles (USVs) and autonomous surface vehicles (ASVs), etc., have received scholarly attention for promoting the construction and development of maritime intelligent transportation systems. Many onboard sensors, e.g., visible-light/thermal-infrared cameras, automatic identification systems (AIS), radar, etc., have been widely exploited to improve the capacity of situational awareness for surface vehicles under complex navigation environments. Owing to the higher performance–price ratio, visible-light cameras attract more attention and play an important role in promoting safe navigation. However, visible-light maritime images easily suffer from a blurred, distorted appearance under hazy weather conditions. The visually degraded images will lead to negative influences on subsequent tasks for surface vehicles, such as object detection, recognition, tracking, visual navigation, etc. [1,2]. Therefore, it is necessary to effectively restore high-quality images from the haze-degraded images.

Currently, significant progress has been made in image dehazing techniques, which can be broadly categorized into three types: contrast enhancement methods, physics-based imaging methods, and deep learning-based methods. Although these approaches can effectively improve image quality in general scenarios, they still exhibit certain limitations. Contrast enhancement methods tend to lose valuable information; physics-based methods have shortcomings in estimating the transmission map and modeling image degradation under complex maritime environments; deep learning-based methods, while achieving excellent performance, rely heavily on high-quality training data and still show estimation errors in the transmission map for sky and water regions.

Therefore, there is an urgent need for an image dehazing method that can adapt to complex maritime scenarios, reduce dependence on high-quality training data, and maintain accurate transmission estimation in sky and water regions, thereby enhancing the visual perception and navigational safety of surface vessels under hazy conditions. This study proposes a corresponding solution to address these issues and validates through experiments its effectiveness in improving image quality and enhancing maritime object detection performance.

1.1. Related Works

1.1.1. Contrast Enhancement Methods

Images captured in haze environments have low contrast. Therefore, many methods have been proposed to eliminate haze through contrast stretching. Early attempts, e.g., histogram equalization (HE) and its extended versions [3,4] transformed the histogram into a uniform distribution for enhancing the contrast of hazy images. Zhou et al. [5] proposed a single image dehazing method based on Retinex theory [6], which decomposed one image into reflection and illumination for enhancing separately. Ancuti et al. [7] introduced a multi-scale fusion-based strategy to improve the visibility of hazy images, designing weight maps to effectively fuse image information. However, since the inherent degradation characteristic of hazy images is ignored, valuable information is easily lost in contrast enhancement methods. Therefore, it is challenging to eliminate the degrading effect of hazy images by these methods.

1.1.2. Physical-Based Methods

Through the analysis of haze generation, numerous physical-based methods have been proposed. Specifically, these methods regard the image dehazing task as an inverse problem, where the parameters of image degradation models are estimated from hazy images. For instance, Fattal [8] proposed a transmission estimation method based on surface shading to restore clear images. Berman et al. [9] found that a haze-free image can be well approximated by a few hundred distinct colors to form clusters in RGB space, i.e., non-local prior. In particular, each color cluster is represented by a haze line. A method for restoring a haze-free version by identifying those haze-lines was thus constructed. Based on the fact that pixels of small patches in natural images typically exhibit a 1D distribution in RGB color space, Fattal introduced color-lines prior [10]. In addition, Fattal proposed an image dehazing method base on Markov random field, which is dedicated to estimating transmission maps on noisy and scattered scenes. He et al. [11] proposed the dark channel prior (DCP) according to statistics of natural haze-free images. Based on DCP, He proposed a novel method to estimate transmission and restore the haze-free image. However, DCP-based methods [11,12] easily fail in sky and water regions due to the deviation of transmission estimation. To avoid this problem, Zhu et al. [13] proposed a fusion of luminance and dark channel prior (F-LDCP) strategy to estimate the transmission of sky and water regions accurately. According to our research, physical-based methods can excavate the potential details in the degraded image as much as possible. However, limitations of this kind of method still exist in maritime scenes due to the complexity and ill-posedness of the dehazing problem.

1.1.3. Deep Learning-Based Methods

With the development of convolutional neural networks (CNNs), various deep learning-based methods have been proposed and have made significant advances in dehazing issues. According to our research, deep learning-based image dehazing methods can be divided into parameter estimation and end-to-end methods. DehazeNet [14] was a typical dehazing network for parameter estimation, which obtains dehazed versions by estimating transmission. Subsequently, Ren et al. [15] proposed a multi-scale convolutional neural network (MSCNN) for transmission estimation. However, it is difficult to collect transmission and atmospheric light of images. To solve the problem caused by the dataset, an unsupervised dehazing strategy [16] was designed. Instead of using the training method utilising paired datasets, this method generates transmission by minimizing the energy function of the dark channel prior. Chen et al. [17] defined a new feature called patch map and proposed the patch-map-based hybrid learning DehazeNet (PMHLD). More recently, a deep network-enabled three-stage dehazing network was developed to enhance the imaging quality for visual Internet of Things (IoT)-based intelligent transportation systems [18]. A parameter-adaptive fusion method, related to the new color channel and preserved brightness, was proposed to perform single image dehazing in intelligent transportation systems [19]. In addition, an unsupervised dehazing method, taking into consideration the patch-line and fuzzy clustering-line priors, was presented to accurately estimate the atmospheric scattering parameters and haze-free images [18]. The improved image quality helps enhance ship detection [20] accuracy under hazy weather conditions and contributes to traffic safety. Although the deep learning methods have demonstrated superior performance for image dehazing tasks, strong dependence on high-quality training images is still a problem that cannot be ignored in maritime applications.

1.2. Motivation and Contributions

Although image dehazing has made remarkable achievements, few studies have been proposed for intelligent surface vehicles and maritime scenes. Undoubtedly, maritime images contain more homogeneous regions than plain images, e.g., sky and water. However, many methods fail to suppress the haze in these regions. For instance, dark channel prior (DCP) [11] easily cause a block effect in backgrounds [21]. It is thus necessary to develop more advanced dehazing methods to improve the imaging quality under hazy weather conditions.

Motivated by the great performance of regularized methods in maritime image enhancement tasks [22,23], we proposed a joint variational regularized framework for refining the transmission map. In this paper, maritime image dehazing is performed in two steps, as illustrated in Figure 1. For solving the worse performance of DCP in sky regions, the transmission maps estimated by different methods are integrated as the coarse transmission map. Subsequently, according to the segmented smooth feature of the transmission map, TGV (Total Generalized Variation) [24] and RTV (Relative Total Variation) [25] regularizers are adopted to this joint variational framework to constrain the transmission map, as detailed in Algorithm 2. TGV is a regularization method that denoises images while preserving edges by balancing first- and higher-order gradients, whereas RTV distinguishes structures from textures based on the relative variation of local gradients, allowing textures to be smoothed while retaining important structures. It has the capacity to restore hazy images and improve detection results.

Algorithm 1 Hybrid Regularized Variational Dehazing

1: Input: Hazy image $\mathbf{I}$, Optimal parameters ${\lambda }_{1\sim 5}$, ${\beta }_{1,2}$. 2: Initialize: $i=0$. 3: Estimate $\mathbf{A},t^d$ according to DCP theory [11], Equation (6). 4: Estimate $t^l$ according to Luminance-based model Equation (7). 5: Fusion $\overline{t}$ by $(t^d,t^l)$ according to Equation (9). 6: Build refined model Equation (15), decompose into sub-problems. 7: Define $X_0,Y_0,{\overline{J}}_0,t_0$. 8: While a stopping criterion is not satisfied do 9:      Update $X_{i+1}$ form Equation (19) for fixed $t_i$, 10:    Update $Y_{i+1}$ using primal-dual algorithm for fixed ${\overline{J}}_i$, 11:    Update ${\overline{J}}_{i+1}$ form Equation (22) for fixed $t_i$ and $Y_{i+1}$, 12:    Update $t_{i+1}$ form Algorithm 2 for fixed $X_{i+1},J_{i+1},t_i,$ and $\overline{t}$, 13:    $i\leftarrow i+1$, 14: Restore $\mathbf{J}$ according to Equation (3). 15: Output: Haze-free image $\mathbf{J}$.

In conclusion, the main contributions of this work are summarized as follows

• Hybrid Variational Method. A hybrid variational model, which combines with TGV and RTV regularizers, is proposed to refine the transmission map. The hybrid regularizer effectively solves the problems of over-smoothness and artifact interference.
• Numerical Optimization Algorithm. The original transmission-refined model is a nonconvex nonsmooth optimization problem, which is decomposed into simpler subproblems based on the alternating direction method and easily handled by existing numerical methods.
• Competitive Dehazing Performance. Experiments conducted on synthetic and realistic maritime images proved that the proposed approach could robustly and effectively restore the visibility of hazy images in maritime scenes.

The remainder of this paper is organized into several sections. Section 2 analyzes the image degradation process. Section 3 briefly introduces the fusion process of coarse transmission maps. In Section 4, we propose a hybrid regularized variational dehazing model and the corresponding numerical optimization algorithm. The experimental results on both synthetic and realistic maritime images are presented in Section 5. Finally, we summarize our contributions to this paper in Section 6.

2. Problem Formulation

According to the atmospheric scattering theory firstly introduced by McCartney [26], images captured by imaging sensors are mainly composed of two components, i.e., (1) incident light reflected from the object, which is attenuated by the influence of the scattering medium, e.g., particles, water vapor, etc., and (2) atmospheric light reflected from the scattering medium. The hazy image can thus be formulated as follows

$$\mathbf{I}\left(\mathrm{x}\right)=\mathbf{J}\left(\mathrm{x}\right)t\left(\mathrm{x}\right)+\mathbf{A}(1-t(\mathrm{x}\left)\right),$$

(1)

where $\mathrm{x}$ is the pixel coordinate, $\mathbf{I}$ is the observed hazy image, $\mathbf{J}$ is the haze-free image needed to be reconstructed, $\mathbf{A}$ is the global atmospheric light, and t is the transmission map related to the depth map, i.e.,

$$t\left(\mathrm{x}\right)=e^{-\beta d\left(\mathrm{x}\right)},$$

(2)

where $\beta$ is the scattering coefficient related to wavelength, and d is the distance between the target and imaging sensor.

The purpose of image dehazing is to restore the latent haze-free image $\mathbf{J}$ from the observed hazy image $\mathbf{I}$. According to Equation (1), the restored image $\mathbf{J}$ can be estimated as follows

$$\mathbf{J}\left(\mathrm{x}\right)={\displaystyle \frac{\mathbf{I}\left(\mathrm{x}\right)-\mathbf{A}}{\max (t\left(\mathrm{x}\right),t_{ϵ})}}+\mathbf{A}.$$

(3)

with $t_{ϵ}>0$ being a small constant to avoid instability.

Image dehazing is a typical ill-posed inverse problem since both transmission map t and atmospheric light $\mathbf{A}$ are commonly unknown. The dehazing performance strongly depends upon the accurate estimation of the transmission map. To achieve high-quality dehazing, it is thus important to accurately estimate the transmission map using satisfactory physical priors related to latent haze-free images. In this work, we will propose a coarse-to-fine strategy to accurately estimate the transmission map, assisting in yielding high-quality dehazing results.

3. Estimation of Coarse Transmission Map

This section will introduce both DCP and luminance priors to estimate the coarse transmission map.

3.1. DCP-Based Transmission Map

Through the statistical analysis of outdoor haze-off images, He et al. [11] concluded that some pixels with extremely low intensities exist in at least one color channel for most of the local image areas except large sky regions. This observation forms a famous prior (i.e., DCP), which is described as follows

$$J^{\mathrm{dark}}\left(\mathrm{x}\right)=\underset{\mathrm{y}\in \mathrm{\Omega }\left(\mathrm{x}\right)}{\min }\left(\underset{c\in \{r,g,b\}}{\min }{\mathbf{J}}_c\left(\mathrm{y}\right)\right)\to 0,$$

(4)

where $J^{\mathrm{dark}}$ denotes the dark channel, $\mathrm{\Omega }\left(\mathrm{x}\right)$ is a local patch centered at $\mathrm{x}$. The concept of DCP is essentially beneficial for estimating the transmission map t in Equation (3). In particular, it is necessary to first calculate the atmospheric light $\mathbf{A}$. Among the pixels, related to the top $0.1\%$ of brightest pixels in the dark channel, the one with the maximum intensity in $\mathbf{I}$ could be selected as the global atmospheric light $\mathbf{A}$. After the estimation of $\mathbf{A}$, we can calculate the dark channel of $\mathbf{I}$ and $\mathbf{J}$ in Equation (1), and then put the minimum operators on both sides, i.e.,

$$\underset{\mathrm{y}\in \mathrm{\Omega }\left(\mathrm{x}\right)}{\min }\left(\underset{c}{\min }{\displaystyle \frac{{\mathbf{I}}_c\left(\mathrm{y}\right)}{{\mathbf{A}}_c}}\right)=t\left(\mathrm{x}\right)\underset{\mathrm{y}\in \mathrm{\Omega }\left(\mathrm{x}\right)}{\min }\left(\underset{c}{\min }{\displaystyle \frac{{\mathbf{J}}_c\left(\mathrm{y}\right)}{{\mathbf{A}}_c}}\right)+1-t\left(\mathrm{x}\right).$$

(5)

Based on the assumption of DCP in Equation (4), the DCP-based transmission map $t^{\mathrm{d}}$ can be estimated from Equation (5), i.e.,

$$t^{\mathrm{d}}\left(\mathrm{x}\right)\approx 1-\mu \underset{\mathrm{y}\in \mathrm{\Omega }\left(\mathrm{x}\right)}{\min }\left(\underset{c}{\min }{\displaystyle \frac{{\mathbf{I}}_c\left(\mathrm{y}\right)}{{\mathbf{A}}_c}}\right),$$

(6)

where $\mu$ is a constant parameter and empirically set to $0.95$. It could guarantee natural-looking results in maritime practice.

For most hazy images without large sky regions, DCP could achieve satisfactory dehazing in terms of both quantitative and qualitative evaluations. The maritime images, captured for vehicle surveillance, often contain large sky and water regions. Within these regions, the assumption of DCP is inevitably invalid because the intensity value of the dark channel is much greater than zero. According to Equations (5) and (6), it is easy to derive an underestimation of the dark channel, leading to the inaccurate $t^{\mathrm{d}}$, obviously smaller than the actual version. As illustrated in Figure 2, the DCP-based transmission map $t^{\mathrm{d}}$ is almost equal to zero in the sky and water regions. It can make the dehazing suffer from serious overly dark appearances and halo artifacts. It is thus necessary to accurately estimate the transmission map in large sky and water regions, where DCP commonly fails [27].

3.2. Luminance-Based Transmission Map

Inspired by previous studies [13,27], the luminance-based model can be used to estimate the transmission map in the sky and water regions where DCP fails. The luminance-based transmission map model is essentially based on the luminance channel of a CIE color space. The corresponding transmission map is estimated as follows

$$t^l\left(\mathrm{x}\right)=e^{-\beta \widehat{L}\left(\mathrm{x}\right)},$$

(7)

where $\beta$ is the scattering coefficient, $\widehat{L}$ is the revised value of luminance channel. The scattering coefficient is affected by many factors such as light wavelength, aerosol type, polarization state, and weather conditions. For the sake of convenience, only the wavelengths that play a more important role in light scattering are considered in our method. In this work, the scattering coefficient $\beta$ of the red, green, and blue channels in RGB images are set to 0.3324, 0.3433, and 0.3502, respectively. Note that the transmission map correlates with the depth map. To estimate the transmission more naturally, it is necessary to obtain the depth map as accurately as possible. Therefore, the luminance is stretched as follows

$$\widehat{L}\left(\mathrm{x}\right)={\displaystyle \frac{\tau }{L^{*}}}L\left(\mathrm{x}\right),$$

(8)

where L is the luminance of the input image, $L^{*}$ is the $95\%$ percentile value of the luminance channel, $\tau$ is a parameter that describes the accurate depth range, which controls the estimation of the transmission map by DCP, and luminance assumptions have a similar distribution.

3.3. Weighted Transmission Map

The coarse transmission map $\overline{t}\left(\mathrm{x}\right)$ is obtained by fusing transmission maps (i.e., $t^d$ and $t^l$) with the weight $\omega$ at every pixel location $\mathrm{x}$, i.e.,

$$\overline{\mathbf{t}}\left(\mathrm{x}\right)=\omega \left(\mathrm{x}\right){\mathbf{t}}^{\mathrm{d}}\left(\mathrm{x}\right)+(1-\omega \left(\mathrm{x}\right)){\mathbf{t}}^{\mathrm{l}}\left(\mathrm{x}\right),$$

(9)

where weight $\omega \left(\mathrm{x}\right)\in [0,1]$. It is able to take full advantages of $t^d$ and $t^l$. In particular, DCP only performs well in transmission map estimation in non-sky and non-water regions, whereas the luminance-based transmission map seems more natural-looking in these regions. The DCP model usually shows satisfactory performance, except for large sky and water regions. The luminance-based model is capable of solving this problem. Therefore, when the pixel $\mathrm{x}$ belongs to sky regions, $\omega \left(\mathrm{x}\right)$ should be close to 0 and $\overline{t}\left(\mathrm{x}\right)\to t^l\left(\mathrm{x}\right)$; conversely, $\omega \left(\mathrm{x}\right)$ close to 1 and $\overline{t}\left(\mathrm{x}\right)\to t^d\left(\mathrm{x}\right)$ when the pixel $\mathrm{x}$ belongs to foreground regions. The mathematical model is expressed as follows

$$\begin{array}{c}\hfill \omega \left(\mathrm{x}\right)={\displaystyle \frac{1}{1+e^{-{\theta }_1{t}^d\left(\mathrm{x}\right)+{\theta }_2}}},\end{array}$$

(10)

where ${\theta }_1$, ${\theta }_2$ are parameters that control the weight distribution. The details definition of ${\theta }_1$, ${\theta }_2$ are given by

$${\theta }_1={\displaystyle \frac{20}{\max (t^d\left(\mathrm{x}\right))-\min (t^d\left(\mathrm{x}\right))}},$$

(11)

and

$${\theta }_2=10+{\theta }_1\min (t^d\left(\mathrm{x}\right)).$$

(12)

The illustration of the weighted transmission map is displayed in Figure 3. It can be found that our weighted transmission map seems more reasonable due to the $t^d$ and $t^l$. However, it contains some undesirable blocky effects because the transmission map is not always uniform in a local patch. We will propose a hybrid variational model to refine the coarse transmission map to improve the visual image quality.

4. Estimation of Refined Transmission Map

In this section, a hybrid variational model and its effective numerical method are proposed to further refine the coarse transmission map $\overline{\mathbf{t}}$ in Equation (9).

4.1. Hybrid Variational Model

To make transmission map refinement easier, we propose to exploit a more compact imaging model instead of Equation (1). In particular, we first introduce $\overline{\mathbf{I}}=\mathbf{A}-\mathbf{I}$ and $\overline{\mathbf{J}}=\mathbf{A}-\mathbf{J}$, and then rewrite Equation (1) as follows

$$\overline{\mathbf{I}}\left(\mathrm{x}\right)=\overline{\mathbf{J}}\left(\mathrm{x}\right)\mathbf{t}\left(\mathrm{x}\right).$$

(13)

To guarantee a stable solution, the initial value of ${\overline{\mathbf{J}}}^0$ for our numerical method is pre-selected as follows

$${\overline{\mathbf{J}}}^0\left(\mathrm{x}\right)={\displaystyle \frac{\mathbf{A}-\mathbf{I}\left(\mathrm{x}\right)}{\max \left\{\mathbf{t}\left(\mathrm{x}\right),{\mathbf{t}}_{ϵ}\right\}}},$$

(14)

where ${\mathbf{t}}_{ϵ}$ is a small constant, avoiding the numerical instability. To maintain the geometrical structures in restored images, it is necessary to accurately refine the transmission map. Therefore, the proper constraints on transmission map refinement should be considered. For the sake of simplicity, we first define $I={\mathbf{I}}_c,\overline{I}={\overline{\mathbf{I}}}_c$, $\overline{J}={\overline{\mathbf{J}}}_c$, and $\overline{t}={\overline{\mathbf{t}}}_c$ for $c\in \{r,g,b\}$. The hybrid variational model for estimation of the refined transmission map t is then given by

$$\begin{array}{c}\hfill \underset{\overline{J},t}{\min }\left\{{\displaystyle \frac{{\lambda }_1}{2}}{∥\overline{I}-\overline{J}t∥}_2^2+{\displaystyle \frac{{\lambda }_2}{2}}{∥t-\overline{t}∥}_2^2+{\lambda }_3{∥\nabla t-\nabla I∥}_1+{\lambda }_4\mathrm{TGV}\left(\overline{J}\right)+{\lambda }_5\mathrm{RTV}\left(t\right)\right\},\end{array}$$

(15)

where ${\lambda }_{1\le i\le 5}$ is a positive regularization parameter.

It is well known that TGV tends to encourage piece-wise smooth solutions, beneficial for yielding a more natural-looking restoration during image dehazing. In this work, we directly introduce the second-order TGV to regularize the latent sharp image $\overline{J}$ in Equation (15), i.e.,

$$\mathrm{TGV}\left(\overline{J}\right)=\underset{L}{\min }\left\{{\alpha }_1{∥\nabla \overline{J}-L∥}_1+{\alpha }_2{∥\mathcal{E}\left(L\right)∥}_1\right\},$$

with ${\alpha }_1$ and ${\alpha }_2$ being positive parameters, and $\mathcal{E}$ denoting the symmetrized derivative operator. It is able to suppress the unwanted blocking artifacts inevitably yielded by traditional TV and its extensions. In theory, the transmission map should contain a piece-wise constant appearance within objects and obvious edges at depth discontinuities. To model these properties, the relative total variation (RTV) is exploited to enhance estimation of the transmission map, i.e.,

$$\mathrm{RTV}\left(t\right)={\displaystyle \frac{{\mathcal{D}}_{x}t}{{\mathcal{L}}_{x}t+ϵ}}+{\displaystyle \frac{{\mathcal{D}}_{y}t}{{\mathcal{L}}_{y}t+ϵ}},$$

(16)

where ${\mathcal{D}}_{x}t$ and ${\mathcal{D}}_{y}t$, respectively, denote the window total variations along the horizontal and vertical directions, ${\mathcal{L}}_{x}t$ and ${\mathcal{L}}_{y}t$ are the windowed inherent variations. More details on TGV and RTV regularizers can be found in Refs. [24,25].

The first three components contribute to a unified data-fidelity term in Equation (15). In particular, the first squared L2-norm term complies with the imaging theorem in Equation (13), which is able to reduce the error of transmission map estimation. The second term measures the squared L2-norm distance between t and $\overline{t}$, which can enable the robustness of estimation. The third L1-norm term, adopted to preserve the edges of the transmission map, is capable of maintaining the significant structures essentially related to the original hazy images.

4.2. Numerical Optimization Algorithm

The non-smooth terms make the hybrid variational model (15) more difficult to optimize. To generate stable and accurate solutions, we propose to exploit the alternating direction method (ADM) to effectively handle the corresponding optimization problem [28,29]. In particular, we first introduce two intermediate variables $X=\nabla t-\nabla I$ and $Y=\overline{J}$, and then transform Equation (15) into the constrained version as follows

$$\begin{array}{cc}\hfill & \underset{X,Y,\overline{J},t}{\min }\left\{{\displaystyle \frac{{\lambda }_1}{2}}{∥\overline{I}-\overline{J}t∥}_2^2+{\displaystyle \frac{{\lambda }_2}{2}}{∥t-\overline{t}∥}_2^2+{\lambda }_3{∥X∥}_1+{\lambda }_4\mathrm{TGV}\left(Y\right)+{\lambda }_5\mathrm{RTV}\left(t\right)\right\},\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.X=\nabla t-\nabla I,Y=\overline{J},\hfill \end{array}$$

(17)

where ${\beta }_1$ and ${\beta }_2$ denote the penalty parameters to control the weights of penalty terms.

To guarantee the solution stability, the ADM is adopted to decompose Equation (17) into several simpler subproblems related to X, Y, $\overline{J}$, and t. These subproblems could be easily solved through existing numerical algorithms.

4.2.1. X-Subproblem

Given the fixed value of t, the X subproblem is essentially a least-squares problem with L1-norm regularization, which is given as follows

$$X=\underset{X}{\min }\left\{{\lambda }_3{∥\nabla t-\nabla I∥}_1+{\displaystyle \frac{{\beta }_1}{2}}{∥X-(\nabla t-\nabla I)∥}_2^2\right\},$$

(18)

which can be effectively handled by the following shrinkage operator [30]

$$\begin{array}{cc}\hfill X& =\mathrm{shrinkage}(\nabla t-\nabla I,{\lambda }_3/{\beta }_1)\hfill \\ \hfill & =\max \left(\right|\nabla t-\nabla I|-{\lambda }_3/{\beta }_1,0)\circ \mathrm{sign}(\nabla t-\nabla I),\hfill \end{array}$$

(19)

where ∘ denotes the point-wise product operator, $\mathrm{sign}(·)$ denotes the signum function.

4.2.2. Y Subproblem

Given the fixed value of $\overline{J}$, the Y-subproblem is essentially a TGV-based denoising model [24], i.e.,

$$Y=\underset{Y}{\min }\left\{{\lambda }_4\mathrm{TGV}\left(Y\right)+{\displaystyle \frac{{\beta }_2}{2}}{∥Y-\overline{J}∥}_2^2\right\},$$

(20)

which could be easily solved using the primal-dual algorithm of Chambolle–Pock [31,32]. For more details on this popular algorithm, please refer to [31,32] and the references therein.

4.2.3. $\overline{J}$ Subproblem

Given the fixed values of t and Y, the $\overline{J}$ subproblem is essentially a problem of calculating extremum, i.e.,

$$\overline{J}=\underset{\overline{J}}{\min }\left\{{\displaystyle \frac{{\lambda }_1}{2}}{∥\overline{I}-\overline{J}t∥}_2^2+{\displaystyle \frac{{\beta }_2}{2}}{∥Y-\overline{J}∥}_2^2\right\},$$

(21)

whose solution can be easily calculated by the following formula

$$\overline{J}={\displaystyle \frac{{\lambda }_{1}I/t+{\beta }_{2}Y}{{\lambda }_1+{\beta }_2}}.$$

(22)

4.2.4. T Subproblem

Given the fixed values of X and J, we can define the functions of $f\left(t\right)$ and $g\left(t\right)$ as follows

$$f\left(t\right)={\displaystyle \frac{{\lambda }_1}{2}}{∥\overline{I}-\overline{J}t∥}_2^2+{\displaystyle \frac{{\lambda }_2}{2}}{∥t-\overline{t}∥}_2^2+{\displaystyle \frac{{\beta }_1}{2}}{∥X-(\nabla t-\nabla I)∥}_2^2,$$

(23)

and

$$g\left(t\right)={\lambda }_5\left({\displaystyle \frac{{\mathcal{D}}_{x}t}{{\mathcal{L}}_{x}t+ϵ}}+{\displaystyle \frac{{\mathcal{D}}_{y}t}{{\mathcal{L}}_{y}t+ϵ}}\right).$$

(24)

The t subproblem in Equation (17) is equivalent to solving the following convex optimization model

$$\underset{t}{\min }\left\{F\left(t\right)=f\left(t\right)+g\left(t\right)\right\},$$

(25)

where $f\left(t\right)$ is a continuously differentiable function satisfying the Lipschitz condition, $g\left(t\right)$ is a normalized function constraining the parameter t. Motivated by [30], we can directly exploit the fast iterative shrinkage-thresholding algorithm (FISTA) to solve the above optimization problem. Therefore, we first convert the optimization problem (25) into a quadratic approximation formula. For any $L>0$, $F\left(t\right)$ can approximate a given point y as follows

$$Q_L(t,y)=f\left(y\right)+\langle t-y,\nabla f\left(y\right)\rangle +{\displaystyle \frac{L}{2}}{∥t-y∥}_2^2+g\left(t\right).$$

(26)

At the ($k+1$)-th iteration of FISTA, the minimization of $Q_L(t_k,y)$ is represented via the following proximal operator

$$\begin{array}{cc}\hfill p_L\left(y\right)& =\underset{t}{\min }\left\{Q_L(t_k,y)\right\}\hfill \\ \hfill & =\underset{t}{\min }\left\{g\left(t\right)+{\displaystyle \frac{L}{2}}{∥t-(t_k-{\displaystyle \frac{1}{L}}\nabla f\left(t_k\right))∥}_2^2\right\},\hfill \end{array}$$

(27)

where $g(·)$ represents the RTV regularizer. According to the definition in Equation (23), $\nabla f\left(t_k\right)$ is given by

$$\begin{array}{cc}\hfill \mathrm{\Theta }=& \nabla f\left(t_k\right)\hfill \\ \hfill =& \left\{{\lambda }_1(t_k-{\displaystyle \frac{\overline{I}}{\overline{J}}})+{\lambda }_2(t_k-\overline{t})+{\beta }_1{\nabla }^T\left(\nabla t_k-(X+\nabla I)\right)\right\}.\hfill \end{array}$$

(28)

Intuitively, Equation (27) is essentially an RTV-based smoothing model, which can be handled by the generalized relative total variation (GRTV) [33]. In summary, the whole optimization procedure of the t subproblem is summarized in Algorithm 2.

Algorithm 2 FISTA for t subproblem

1: Input: $\overline{I}$, $\overline{J}$, $\tilde{t}$, X, I, t, ${\lambda }_1$, ${\lambda }_2$, ${\beta }_1$. 2: Initialize: $t_0=t,{\vartheta }_0=1,{\widehat{t}}_0=\tilde{t}$. 3: While a stopping criterion is not satisfied do 4:    Calculate $\mathrm{\Theta }=\nabla f\left(t_k\right)$ according to Equation (28), 5:    Update ${\widehat{t}}_k=p_L\left(t_k\right)=\mathbf{GRTVsmooth}(t_k-{\displaystyle \frac{1}{L}}\mathrm{\Theta })$, 6:    Update ${\vartheta }_{k+1}={\displaystyle \frac{1+\sqrt{1+4{\vartheta }_k^2}}{2}}$, 7:    Update $t_{k+1}={\widehat{t}}_k+{\displaystyle \frac{{\vartheta }_k-1}{{\vartheta }_{k+1}}}({\widehat{t}}_k-{\widehat{t}}_{k-1})$, 8:    $k=k+1$, 9: $t_{\mathrm{result}}=t_k$. 10: Output: Solution of t subproblem: $t_{\mathrm{result}}$.

4.3. Latent Sharp Image Restoration

The proposed numerical method is effective to handle the nonconvex nonsmooth optimization problem Equation (15). After optimization, there are still some weak artifacts, guided filtering [34] can be used to enhance the edge structure information before the optimization process. The guided filtering filters the coarse transmission map through a guided image $\mathbf{I}$, making the output transmission similar to the coarse transmission map in intensity, but the edges are enhanced to be similar to the guided image $\mathbf{I}$. Once the final refined transmission map t is obtained, it is easy to yield the haze-free image $\mathbf{J}$ according to Equation (1). The whole optimization procedure is summarized in Algorithm 1. In addition, the procedure of transmission map refinement is illustrated in Figure 4.

5. Experimental Results and Analysis

5.1. Experimental Data

Experiments will be conducted on both synthetic and real-world images to verify the superior dehazing performance of our method. The synthetic images are generated based on a physical model. Specifically, we manually selected 100 high-quality maritime images from the public OverwaterHaze dataset [35] as ground truths. Examples of the synthetic data are shown in Figure 5, where the degraded images were generated using Equation (1) with $t\in \{0.6,0.4,0.2\}$ and $\mathbf{A}=0.8$. The real-world images are collected from actual hazy scenes and publicly available online datasets.

5.2. Experimental Settings

To verify the effectiveness and practicability of dehazing methods, numerous experiments are implemented on both synthetic and realistic hazy images. Our method will be compared with several state-of-the-art dehazing methods, including the dark channel prior (DCP) [11], non-local prior (Non-Local) [9], fusion of luminance and dark channel prior (F-LDCP) [13], gradient residual minimization (GRM) [36], all-in-one dehazing network (AOD-Net) [37], gated context aggregation network (GCANet) [38], and multi-scale convolutional neural network (MSCNN) [15]. In particular, the first four methods belong to the traditional dehazing methods. In contrast, the others are deep learning-driven dehazing methods. Analogous to previous variational dehazing methods, our method is also dependent on the optimal selections of (regularization) parameters. In our experiments, the tunable parameters were empirically optimized on the validation set and set as follows: ${\lambda }_1=15\times {10}^{-1}$, ${\lambda }_2=5\times {10}^{-1}$, ${\lambda }_3=3\times {10}^2$, ${\lambda }_4=1$, ${\lambda }_5=3\times {10}^{-2}$, ${\beta }_1=1\times {10}^{-1}$, ${\beta }_2=5$, and $\tau =3.4$. Extensive experiments have demonstrated the effectiveness and robustness of these manually selected parameters under different hazy conditions. To achieve fair experimental comparisons, other dehazing methods will be implemented with the optimal parameters provided by the authors. Our experiments were conducted on a machine equipped with an Intel(R) Core(TM) i5-10600KF CPU @ 4.10GHz and an 11GB NVIDIA GeForce RTX 2080 TI, running Python 3.8 and PyTorch 1.6.0.

5.3. Dehazing Experiments on Synthetic Hazy Images

Comprehensive dehazing experiments on synthetic images will be conducted in this subsection. For the sake of better visual comparisons, several representative dehazing results are visually illustrated in Figure 6. It is obvious that DCP, Non-Local, and GCANet fail to generate the unnatural-looking restored images, due to the significant color distortion. Some unpleasant blocking artifacts are also produced in homogeneous regions (e.g., sky or water areas). Compared with these methods, F-LDCP could restore the hazy images with higher visual quality. However, the occurrence of color abnormality in homogeneous regions could create unnatural appearances. It is intractable to totally remove the haze effects through the AOD-Net, GRM, and MSCNN. The remaining haze has obviously degraded the image details and color appearance, leading to unsatisfactory and unstable visual quality. By comparison, our method can not only eliminate the effects of unwanted artifacts and color distortion, but also produce the most natural-looking restored images under different imaging conditions.

To quantitatively evaluate the imaging performance, PSNR [39], SSIM [40], FSIM [41], and ${\mathrm{FSIM}}_c$ [41] are simultaneously exploited to compare our method with other dehazing methods. In theory, the higher scores essentially indicate the higher-quality dehazing results. The quantitative results are summarized in detail in

Table 1. PSNR, SSIM, FSIM, and ${\mathrm{FSIM}}_C$ comparisons (mean ± std) of all methods based on synthesized hazy maritime images. The best results are highlighted in bold.

Methods	PSNR	SSIM	FSIM	${\mathrm{FSIM}}_{\mathit{C}}$
DCP [11]	14.70 ± 2.09	0.807 ± 0.044	0.913 ± 0.028	0.909 ± 0.029
Non-local [9]	18.53 ± 2.72	0.862 ± 0.051	0.926 ± 0.034	0.922 ± 0.034
F-LDCP [13]	20.91 ± 2.12	0.916 ± 0.034	0.956 ± 0.013	0.951 ± 0.015
GRM [36]	16.88 ± 1.42	0.761 ± 0.078	0.885 ± 0.033	0.882 ± 0.033
AOD-Net [37]	18.15 ± 1.01	0.870 ± 0.034	0.884 ± 0.017	0.882 ± 0.017
GCANet [38]	19.07 ± 4.37	0.874 ± 0.069	0.938 ± 0.037	0.930 ± 0.041
MSCNN [15]	18.71 ± 2.14	0.874 ± 0.044	0.939 ± 0.014	0.937 ± 0.014
Ours	21.92 ± 2.35	0.933 ± 0.019	0.971 ± 0.012	0.967 ± 0.013

. It can be found that our method generates the most optimal evaluation scores in all metrics. These superior imaging results mainly benefit from the robust transmission estimation and hybrid variational model.

5.4. Dehazing Experiments on Realistic Hazy Images

We further carry out several dehazing experiments on realistic hazy maritime images in this subsection. The visual comparisons of real-world dehazing methods are illustrated in Figure 7, Figure 8 and Figure 9. Unlike image dehazing in traditional urban transportation, the more general imaging scenarios with heterogeneous haze make maritime image dehazing more difficult in practice. The dehazing results illustrate that the competing methods fail to robustly reconstruct the hazy images. In particular, due to the improper prior assumptions, e.g., DCP and Non-Local, in sky and water areas, both DCP and Non-Local generate dehazed images with poor contrast or color distortion. In addition, some unwanted artifacts easily appear around the ships, resulting in image quality degradation. L-DCP is able to effectively remove the hazy effects. However, the restored results remain susceptible to the color shift interference, especially in the sky and water regions of relatively homogeneous colors. The negative effects, e.g., unwanted artifacts and color distortion, could also be observed in restored images generated by GCANet. By contrast, AOD-Net, GRM, and MSCNN are capable of eliminating the impact of haze and preserving the main image structures. However, the resulting restored images sometimes suffer from obvious degradation in intensity contrast. Compared with these state-of-the-art methods, our method has the capacity to effectively remove the hazing effects, eliminating the imaging contrast and preserving the color appearance. The visual image quality can be obviously improved under different scenarios. This is mainly due to the fact that our variational model performs well in accurately and robustly estimating the transmission map, which directly determines the reconstruction of hazy images.

5.5. Influences of Dehazing on Ship Detection

Object detection, recognition, and tracking provide reliable foundations for vehicle surveillance in video-empowered maritime transportation systems. However, the surveillance performance is strongly dependent on high-quality maritime images. There is thus a potential to employ our haze visibility-enhancement method to promote maritime vehicle surveillance under adverse hazy conditions. To investigate the influences of dehazing on ship detection, it is necessary to implement detection experiments on original hazy images and their restored versions. In particular, an efficient one-stage detector, YOLOv4 [42], which can achieve a good balance between detection accuracy and efficiency, is selected to detect maritime vehicles. The YOLOv4-based detection results are visually shown in Figure 10. In these hazy images, the visual information is inevitably obscured due to the hazing effects. Note that the training datasets often do not contain the degraded images collected under hazy conditions. The detection accuracy and robustness will be easily degraded accordingly. After the implementation of our method, the visual image quality will be significantly improved, resulting in high-quality ship detection results. More ship detection experiments under different imaging scenarios could be visually found in Figure 11, Figure 12 and Figure 13. It can be found that our method is capable of restoring the degraded images, leading to improved detection performance. Therefore, our method can be considered as a useful component for promoting surveillance of maritime vehicles in real-world scenarios during hazy weather conditions.

6. Conclusions

Visibility enhancement plays an essential role in guaranteeing the navigational safety of intelligent surface vehicles under hazy conditions. However, it is commonly intractable to accurately estimate the transmission map due to the enormous sky and water regions. To overcome this limitation, we first generated the coarse transmission map through weighing fuse transmission maps estimated by different methods. To further refine the coarse transmission map, a variational framework with hybrid regularizers was proposed. The proposed method could preserve important image structures and detailed information in various hazy conditions. Both synthetic and realistic experiments have illustrated that our method could perform superior quantitative and qualitative evaluations.

It is worth noting that our method is essentially a physically guided variational dehazing approach. The proposed method is relatively sensitive to certain parameters, such as the weights of the hybrid regularizers, which require careful tuning. These factors should be carefully considered when applying the method to real-time scenarios or large-scale datasets. Inspired by the successful combination of physical modeling and deep learning [43,44], future work could incorporate the proposed hybrid regularizers as constraints into the loss functions of deep learning-based dehazing models. This integration may reduce the model’s sensitivity to parameters and potentially further enhance imaging performance.