Density Estimation of Fog in Image Based on Dark Channel Prior

Abstract

This paper proposes a method and an original index for the estimation of fog density using images or videos. The proposed method had the advantages of convenient operation and low costs for applications in automatic driving and environmental monitoring. The index was constructed based on a dark channel map and the pseudo-edge details of the foggy image. The effectiveness of the fog density index was demonstrated and validated through experiments on the two existing open datasets. The experimental results showed that the presented index could correctly estimate the fog density of images: (1) the estimated fog density value was consistent with the corresponding label in the Color Hazy Image Database (CHIC) in terms of rank order; (2) the estimated fog density level was consistent with the corresponding label in the Cityscapes database and the accuracy reached as high as 0.9812; (3) the proposed index could be used to evaluate the performance of a video defogging algorithm in terms of residual fog.

1. Introduction

Fog or haze in images and videos leads to low visibility and thus, causes major problems in transportation and computer vision applications. Under poor visibility conditions, road traffic accidents often increase greatly. In order to ensure people’s safety whilst traveling, it is necessary for the relevant authorities to make the decision of whether or not to close highways, cancel flights, etc. However, a reliable decision requires an accurate estimation of fog density and images and videos that are taken during reduced visibility conditions are often characterized by blurred details, faded color, lower contrast and overall poor visibility and thus, degrade the performance of outdoor vision systems, such as object recognition, segmentation and remote sensing. In order to accurately detect and process extracted features for computer vision applications, image defogging is an imperative preprocessing step. However, the fog-relevant features closely correlate with the perception of fog density within the image [1], so it is necessary to estimate the fog density in order to select the most appropriate defogging methods. As a result, the estimation of fog density is critical and has a wide range of applications. The definition of fog and the division standard of fog density levels are somewhat different within the fields of physics and meteorology [2]. In terms of measuring methods, fog levels can be estimated using subjective human judgment or measured using instruments, such as a smoke density tester or an optical fog sensor [3]. For most people, foggy weather is often simply divided into three levels, according to visibility: shallow fog, moderate fog and dense fog [4]. According to Koschmieder’s law, horizontal visibility can be estimated simply using the atmospheric extinction coefficient, which is highly relative to the daytime fog density [5,6]. The visibility is then measurable using instruments such as a “visiometer”. Therefore, in meteorology, atmospheric horizontal visibility is usually used to judge fog density levels [5].

Fog density estimation is not just a subjective visual evaluation with the naked eye, but is rather an objective estimation that is based on facts and measurements. In recent years, image-based fog density estimation methods have received an increasing amount of attention [1]. A representative method is the Fog Aware Density Evaluator (FADE) algorithm [7], which is based on the model of NSS (Natural Scene Statistics) and fog-aware statistical features. A pixel-based fog density estimation algorithm was proposed in [8], which is based on two priors and is constructed in the HSV space. Mori et al. [9] proposed a fog density estimation method that uses in-vehicle camera images and millimeter wave (mmW) radar data [9]. This method estimates fog density by evaluating the visibility of and distance to the vehicle in front. In addition, Jiang et al. [1] proposed a surrogate model that uses optical depth to estimate fog density. The authors of [10] found three fog-relevant statistical features by observing the RGB values of foggy images and developed a fog density estimator (SFDE) that uses a linear combination of those three features.

Machine learning-based methods for the evaluation of atmospheric horizontal visibility and the estimation of fog density are hot topics at present. Based on the convolution neural network of the Alex model, a visibility detection method that combines monitoring camera equipment and a deep learning algorithm was proposed in [11]. To reduce the number of unnecessary nonlinear components in the network and maintain the performance as the number of convolution layers gradually increases, a new and improved DiracNet convolution neural network was proposed and a haze visibility detection method was constructed in [12]. In addition, a method in which visibility is estimated using a continuous surveillance video by taking the mean square error into account and constructing a convolutional neural network (MSEBCNN) was developed in [13].

The estimation of the amount of fog in images allows us to construct more effective image defogging algorithms [1,10]. For example, by repairing the medium transmission of bright regions using FADE [7], the color distortion that is often produced by many image dehazing algorithms can be improved effectively [14]. Recently, Yeh’s fog density estimation method [8] has often been employed to improve the estimation of atmospheric light and transmission maps. Aiming to find a solution to the defect of many traditional single image dehazing algorithms that fail in sky areas, the authors in [15] used the haze density weight function to reduce the halo effect in sky areas. In addition, the estimation of the amount of fog in scene images allows us to establish more appropriate foggy image datasets. Recently, several foggy image databases have been built, which contain reference fog-free images and the related foggy images that are covered with different levels of fog, such as the Color Hazy Image Database (CHIC) [16], which consists of real images that are characterized into nine different levels of fog density, and the Real world Benchmark Dataset (BeDDE) [4], in which each image features a fog level label. These databases were helpful for us to investigate the quality of our haze model according to fog density and the efficiency of the dehazing algorithm.

Compared to the instrument measurement methods, these methods are straightforward [7] and have the advantages of low costs and high efficiency. However, only a few image processing algorithms have been proposed for fog density estimation and research on this issue is far from extensive. Since our defogging method, which was based on Dark Channel Prior (DCP) [17], was drawn from statistical analysis and had clear meteorological meanings, we assumed that the inverse process of image dehazing using DCP could reveal the levels of fog density. Based on this assumption and the pseudo-edge details of the image, an alternative method for quickly detecting fog density using images is proposed in this paper.

2. Method

With a foggy image, we first estimated fog density using a Dark Channel Prior (DCP) [17] map and the pseudo-edge details of the image and then calculated the fog density level. The process for obtaining the fog density level contained four steps: (1) input images with fog density level labels; (2) calculate dark channel map and pseudo-edges of the input images; (3) estimate the parameters in the model by training a benchmark dataset; (4) apply the constructed model to the estimation of fog density values and levels in images or videos. Note that when there was no labeled dataset, we either used the default values for the parameters or a transfer study.

2.1. Density Index

Our physics-based image defogging method using single input image relied on prior assumptions to estimate the unknown physical parameters of the model. HE et al. [17] proposed the Dark Channel Prior (DCP) algorithm while investigating a large number of fog-free images. DCP suggests that the minimum intensity of a non-sky local patch in an outdoor fog-free image is close to zero. For convenience, we present He’s Dark Channel Prior (DCP) alorithm here as a proposition:

Proposition 1

([17]). When image I is fog-free, the minimum values of the three color channels ($r,g$ and b) in the image are close to 0, i.e.,

$$I^{dark}(i,j)=\underset{c\in r,g,b}{min}(\underset{y\in \Omega (i,j)}{min}{I}^c(y))\approx 0,$$

(1)

In Formula (1), $(i,j)$ is the pixel position at the ith row and jth column of the image, $\Omega (i,j)$ is a neighborhood of $(i,j)$ and $I^{dark}(i,j)$ the dark channel of pixel $I(i,j)$. Image $I^{dark}$ was named as the Dark Channel Map (DCM).

The converse negative of Proposition 1 can be expressed as:

Proposition 2.

When $I^{dark}(i,j)=\underset{c\in r,g,b}{min}(\underset{y\in \Omega (i,j)}{min}{I}^c(y))>>0$, image I is covered in fog.

This assumption may fail when the image contains large sky or white regions, which causes the DCP algorithm to become invalid [17].

In order to evaluate the validity of Proposition 2, five images with different levels of fog density from [16] (first row) and their dark channel maps (second row) were studied and are presented in Figure 1. From this evaluation, we found that the brighter the dark channel image, the thicker the fog, i.e., the gray value tended toward 1 (0 is dark and 1 is white).

Usually, foggy images are blurrier and have low contrast, which means that edge information is also closely related to the fog density. This observation has been validated using a large set of images. As shown in the third row of Figure 1, the thicker the fog, the more pseudo-edge details. Due to the effects of the fog, the edges are pseudo-edges.

Our approach was based on the dark channel maps and pseudo-edge details of the images and the fog density index $F_{den}$ of the image was first defined as:

$$F_{den}=\frac{1}{1+\alpha e^{-\beta (\sqrt{F_{nave}{E}_r}-{\mathcal{M}}_{FE})}}.$$

(2)

This is a logistic function, which is also called the sigmoid function, and its domain is the real number set and range of the open interval (0, 1). The use of Equation (2) meant that the relative fog density $F_{den}$ was defined based on the average value of the product of dark channels that were related $F_{nave}$ and the proportion of the edge information. In this equation, $\alpha$ and $\beta$ are the scaling parameters and ${\mathcal{M}}_{FE}$ is the translation parameter that was used in the standardization process. The three parameters were positive numbers and could be obtained by training the standard database.

Moreover, in Formula (2), $E_r$, the pseudo-edge details of the image are the ratios of edge pixel points to image points and the edge image was obtained using the Canny operator [18]. The average brightness of the dark channel maps, except for the sky and white regions, in the images was represented by $F_{ave}$:

$$F_{ave}=\frac{1}{n_0}\sum _{i=1}^m\sum _{j=1}^n(I^{dark}(i,j)\le {\delta }_w),$$

(3)

where m and n are the number of rows and columns in the image, $n_0$ is the number of non-sky pixels and ${\delta }_w$ is the segmentation parameter of the pixels in the sky and white image blocks, generally ${\delta }_w=0.95$. $F_{ave}$ was standardized as:

$$F_{nave}=\left\{\begin{array}{cc}0,\hfill & F_{ave}\le {\theta }_L\hfill \\ \frac{F_{ave}-{\theta }_L}{{\theta }_U-{\theta }_L},\hfill & {\theta }_L<F_{ave}\le {\theta }_U\hfill \\ 1,\hfill & F_{ave}>{\theta }_U\hfill \end{array}\right.$$

(4)

where $F_{nave}$ is the standardized $F_{ave}$ and ${\theta }_U$ and ${\theta }_L$ are two thresholds with experimental values of ${\theta }_U=0.88$ and ${\theta }_L=0.18$, which were introduced to distinguish the sky and white regions, the foggy regions and the fog-free areas.

2.2. Fog Density Levels

According to application requirement, the fog density level $F_{Lev}$ of an image could be classified into four levels: fog-free, shallow fog, moderate fog and dense fog, which were represented by 0, 1, 2 and 3, respectively. That is, $F_{Lev}$ was a piecewise constant function:

$$F_{Lev}=\left\{\begin{array}{cc}0,\hfill & I\mathrm{is}\mathrm{fog}-\mathrm{free};\hfill \\ 1,\hfill & I\mathrm{is}\mathrm{shallow}\mathrm{fog};\hfill \\ 2,\hfill & I\mathrm{is}\mathrm{moderate}\mathrm{fog};\hfill \\ 3,\hfill & I\mathrm{is}\mathrm{heavy}\mathrm{fog}.\hfill \end{array}\right.$$

(5)

Different from the manual methods [4], this paper estimated the density level $F_{Lev}$ of an image automatically, according to the fog density index $F_{den}$:

$$F_{Lev}=\left\{\begin{array}{cc}0,\hfill & F_{den}\le {\delta }_0;\hfill \\ 1,\hfill & {\delta }_0<F_{den}\le {\delta }_1;\hfill \\ 2,\hfill & {\delta }_1<F_{den}\le {\delta }_2;\hfill \\ 3,\hfill & {\delta }_2<F_{den}.\hfill \end{array}\right.$$

(6)

The parameters ${\delta }_0,{\delta }_1$ and ${\delta }_2$ in Formula (6) were estimated using an analysis of a set of foggy images with fog level labels from several accessible databases, for example, the Cityscapes Dataset [19,20].

Our method for the estimation of parameters was based on the normal distribution hypothesis of the density value $Y_i$ of each category obeying a normal distribution $N({\mu }_i,{\sigma }_i^2)$, $i=0,1,2,3$. In general, ${\mu }_0\le {\mu }_1\le {\mu }_2\le {\mu }_3$. The probability density functions were denoted as ${\varphi }_i(y)$. Our algorithm consisted of two main steps.

The first step was to estimate the values of the parameters ${\mu }_i,{\sigma }_i^2,i=0,1,2,3$ using the ith class of images ($F_{Lev}=i$ in Formula (5) and $F_{den}$ in Formula (2)).

The second step was to calculate the parameters ${\delta }_0,{\delta }_1$ and ${\delta }_2$ by finding the intersection points between the adjacent probability density functions ${\varphi }_i(y)$. These intersection points were successively marked as ${\delta }_0,{\delta }_1$ and ${\delta }_2$, from small to big, as shown in Figure 2.

One of the advantages of the proposed method was that the probability of correct classification could be estimated theoretically. Here, we take the case of fog-free as an example:

$$P(F_{Lev}=0|I\mathrm{is}\mathrm{fog}-\mathrm{free})={\int }_{-\infty }^{{\delta }_0}{\varphi }_0(y)dy,$$

(7)

Similarly, the probability of misclassification could be estimated as:

$$P(F_{Lev}=1|I\mathrm{is}\mathrm{fog}-\mathrm{free})={\int }_{{\delta }_0}^{{\delta }_1}{\varphi }_0(y)dy,$$

(8)

Theoretically, the probabilities of the correct classification or misclassification of all cases could also be calculated using similar formulae. The results formed a matrix, which was named as a confusion matrix (described in the next section).

By substituting the estimated values of ${\delta }_0,{\delta }_1$ and ${\delta }_2$ into Formula (6), the fog density level ($F_{Lev}$) of an image I could be determined according to its fog density value $F_{den}$.

3. Results

In order to evaluate the proposed method, the experimental results are presented in this section. The experiments were conducted mainly for the estimation of fog density values and fog density levels in images, as well as changes of fog density in videos.

3.1. Fog Density Estimation

To verify the effectiveness of the fog density index $F_{den}$, the $F_{den}$ of images from the Color Hazy Image Database (CHIC) [16] was calculated and compared to the results from SFDE [10], FADE [7] and the ground truth. Figure 3 shows 10 images from Scene1 in CHIC and these images are all labeled with a density from 1 (heavy foggy) to 10 (clear). Note that the CHIC database is characterized by the presence of the ground truth reference images.

Figure 4 shows the fog density values that were evaluated using SFDE [10], FADE [7] and our method. By comparing these three curves, we found the following characteristics:

(1)

The three curves were all monotonically decreasing, which was highly consistent with the changes in the real fog density values of the images in the dataset (ground truth information);

(2)

The fog density values that were estimated using our method were limited to the interval 0 to 1, in which 0 meant clear and 1 represented heavy fog, while FADE did not have a limited interval and the density values that were calculated using SFDE tended to be greater than 0.5. Obviously, our fog density values had a more intuitive meaning;

(3)

Our curve was in sharp decline from the fourth image to the fifth image, which illustrated that the fog density in the first four images was significantly different from that in the last six images. This correlated with the real situation.

3.2. Fog Density Levels

The other experimental dataset that we employed was “lindau”, which is a subset of the Cityscapes dataset [19,20]. The “lindau” dataset consists of 177 images, of which 59 images are covered with shallow fog, 59 are covered with moderate fog and 59 are covered with heavy fog. Figure 5 presents 12 images from four different scenes in the dataset. By applying Formula (2) to 36 of the images, which consisted of these 12 images covered with three different density levels of fog, the fog density values $F_{den}$ were calculated and are displayed in Figure 6. Figure 5 and Figure 6 show that the fog density in most of these images was very low. According to general standards, all of these images would be classified into the shallow fog category, both visually and numerically. However, this dataset divided the values into three different categories (shallow fog, moderate fog and heavy fog) based on a refined standard, which meant that in some application scenarios, it was necessary to distinguish the fog density levels within a small range of variation. Our method could realize this fine classification through parameter learning when the labeled training dataset was feasible.

In order to identify the density level $F_{Lev}$ according to the practice application, it was necessary to first calculate the value of parameter ${\mathcal{M}}_{FE}$ in Formula (2) and ${\delta }_0,{\delta }_1$ in Formula (6) using the training data. Since only three density levels (shallow, moderate and heavy) were considered, we did not need to estimate the parameter ${\delta }_2$.

We chose the first 36 images (12 images with three fog densities) as the training data. Six of them are shown in the first row of Figure 5. After training, the parameters ${\mathcal{M}}_{FE}$, ${\delta }_0$ and ${\delta }_1$ were evaluated as ${\mathcal{M}}_{FE}=0.155$, ${\delta }_0=0.235$ and ${\delta }_1=0.5509$. The normal distribution fitting curves are shown in Figure 7. The confusion matrix of the training results is listed in the three left-hand columns of

Table 1. The accuracy of fog density level estimation in the training stage and the test (marked with “*”) stage.

Level	${\mathit{F}}_{\mathbf{Lev}}=0$	${\mathit{F}}_{\mathbf{Lev}}=1$	${\mathit{F}}_{\mathbf{Lev}}=2$	${\mathit{F}}_{\mathbf{Lev}}^{*}=0$	${\mathit{F}}_{\mathbf{Lev}}^{*}=1$	${\mathit{F}}_{\mathbf{Lev}}^{*}=2$
Shallow	0.9929	0.0071	0.0000	0.9149	0.0426	0.0426
Moderate	0.0149	0.9689	0.0162	0.4894	0.2553	0.2553
Heavy	0.0000	0.0181	0.9819	0.1489	0.1489	0.7021

. The accuracy of the fog density level estimation was 0.9812.

The last 141 images (47 images with three fog density levels) in the “lindau” database were chosen as our test data. The confusion matrix of the test dataset is listed in the three right-hand columns of Table 1. The accuracy of the fog density level estimation was 0.6241.

3.3. Fog Density Levels in Videos

The fog density level of the scenes that were evaluated using a single image could predict changes in fog density during a video. We downloaded a video of foggy mountains (thanks to the open permission of the uploader) from the Bilibili website (https://www.bilibili.com/video/BV1Rb411y7qR?p=9 accessed on 17 May 2021). The first 12 s of the video, which is composed of 317 frames, was used as the experimental data. The fog density values $F_{den}$ of the frames were drawn as a discrete curve (for convenience, we called it the $F_{den}$ curve), which is shown in Figure 8, where the symbol “#” denotes the frame sequence number.

To evaluate the image defogging algorithms, various objective evaluation indexes have been adopted, such as the classical indexes “SSIM” and “PSNR” [21] and the recently proposed indexes “visible edge ratio” and “average gradient ratio” [22]. However, to evaluate the performance of video defogging algorithms, additional indexes are needed to evaluate the temporal and spatial coherence of the defogged results. Since $F_{den}$ could be used not only to measure the degree of image pollution and the image defogging effect but also changes in the fog density in a video, it was suitable for the evaluation of the video’s temporal and spatial coherence.

According to Equation (2), the fog density $F_{den}$ of each frame of the defogged video formed a sequence. We found that the changes in the sequence were closely related to the smoothness of the video, especially the defogged video. In general, the flatter the $F_{den}$ curve, the smoother the motion in the video. Conversely, violent fluctuations in the $F_{den}$ curve meant that the flicker of the dehazed video increased. To validate our observations, experiments were conducted on the foggy video “Cross” [23], which consists of 240 frames. Four existing video dehazing algorithms were evaluating using the $F_{den}$ curves. These four algorithms were: dehazing based on a neural network (DNet) [24], advanced histogram equalization (HistEq) [23], jointly estimating scene depth (JoDep) [25] and spatial–temporal MRF (STMRF) [26]. The experimental results are shown in Figure 9 and Figure 10. Figure 9 exhibits the defogged images of one frame that were produced by the four algorithms.

The $F_{den}$ curves of the original foggy video and the defogged videos that were produced by the four algorithms are displayed in Figure 10. It can be seen that the original video was taken under heavy fog weather conditions, the result of the DNet algorithm retained the most fog, the curve of HistEq algorithm shows the most and biggest fluctuations (which meant that there were the most flickers in that video) and the curves of the STMRF and JoDep algorithms were relatively smooth and almost coincided with the straight line Fden = 0 (which meant that the defogged videos had less flickers and the least amount of residual fog).

4. Conclusions

The aim of our method was to estimate the relative fog density by simply using an improved logistic function, which was related to the dark channel maps and pseudo-edge details of the images. The fog density values, which are directly proportional to the visibility, were standardized into the interval 0 to 1, which was conducive for the comparison of different images. For the practical application, we constructed an index of fog density levels using the estimated relative fog density values. The parameters that were involved in the level classification were estimated using a trained set of foggy images with fog level labels from several accessible databases. Due to this, the number of levels could be increased or decreased flexibly, according to the application requirements.

From the experimental results using open image datasets, several conclusions could be drawn:

(1)

In the experiment using the Color Hazy Image Database (CHIC), our index was consistent with the labeled fog density values in terms of rank order;

(2)

In the experiment using the Cityscapes database, our index was consistent with the labeled fog density values. The accuracy reached as high as 0.9812.

It should be noted that our method had its limitations. It did not uncover the quantitative relationship between visibility and fog density, which will become our research topic in the future. One approach that is also worth exploring is as follows: take photos and calculate the fog density levels using an image-based fog density estimation method (such as the proposed method) while measuring fog density via various meteorological sensors [2,27,28], then establish the quantitative relationship between the measured and estimated fog density values and improve the accuracy using multi-scene model correction. There is no doubt that the discovery of this relationship will promote the application of our method in image and video processing.