Single Remote Sensing Image Dehazing Using Robust Light-Dark Prior

: Haze, generated by ﬂoaters (semitransparent clouds, fog, snow, etc.) in the atmosphere, can signiﬁcantly degrade the utilization of remote sensing images (RSIs). However, the existing techniques for single image dehazing rarely consider that the haze is superimposed by ﬂoaters and shadow, and they often aggravate the degree of the haze shadow and dark region. In this paper, a single RSI dehazing method based on robust light-dark prior (RLDP) is proposed, which utilizes the proposed hybrid model and is robust to outlier pixels. In the proposed RLDP method, the haze is ﬁrst removed by a robust dark channel prior (RDCP). Then, the shadow is removed with a robust light channel prior (RLCP). Further, a cube root mean enhancement (CRME)-based stable state search criterion is proposed for solving the difﬁcult problem of patch size setting. The experiment results on benchmark and Landsat 8 RSIs demonstrate that the RLDP method could effectively remove haze.


Introduction
Haze affects the visibility of an optical remote sensing image (RSI), thereby, increasing the difficulty of many geographic monitoring procedures such as resource survey, environmental monitoring, and disaster analysis.As an example, semitransparent clouds are polymers formed by a mixture of small water droplets or small ice crystals that reflect and diffuse light in the air, thereby reducing the visibility of ground objects [1].Haze is caused when the light reaching the optical camera is attenuated, while shadow is conversely caused when the light reaching ground is attenuated.Therefore, it is necessary to remove both haze and shadow from RSIs in advance to improve image clarity and provide assurance for subsequent tasks.
Optical RSI dehazing is considerably different from traditional image dehazing, in that there is a large distance between the optical sensor and the ground objects, irregular haze intensity, and absence of sky regions [2].Additionally, optical RSIs take ground objects from a top-down perspective, and the presence of haze will also cause shadow.To solve the problems existing in prior-based methods, we propose an RLDP-based single RSI dehazing method which can simultaneously remove haze and shadow.At the same time, the patch size parameter can be adaptively searched according to the image contrast.The experiment results on benchmark RSIs and Landsat 8 RSIs demonstrate that RLDP can simultaneously remove haze and shadow.
The main contributions are as follows: • For the regularity of satellite imaging, a semitransparent cloud was taken as an example, and a shadowing model and corresponding hybrid model were proposed.Then, a two-stage dehazing algorithm was proposed based on the haze hybrid model; • An RLDP-based single RSI dehazing method which removes haze based on the robust dark channel prior (RDCP) and removes shadow with the robust light channel prior (RLCP) was proposed.A statistical-based 3σ criterion is also used to improve the robustness of RLDP; • In order to solve the patch size setting problem, a CRME-based appropriate patch size search criterion was proposed.This method can adaptively adjust the patch size according to the contrast measure of a single image.

State of the Art
Researchers have developed various methods of computer vision to achieve image dehazing.Among such methods, prior-based and data-driven methods are the most popular [3,4].
The prior-based approaches are based on the atmosphere scattering model [5], which can be expressed as follows: where I(x) is the observed hazy image, J(x) is the haze-free image to be recovered, x is the position of any pixel, A is the global atmospheric light, and t(x) = e −βd(x) is the transmission map, with β being the atmospheric scattering parameter and d(x) being the scene depth.The prior-based methods estimate the ill-posed parameters t(x) and A by prior assumptions.They then use Equation ( 1) to reconstruct the haze-free image J(x) to achieve image dehazing.He et al. [6] pioneered an efficient dark channel prior to approximate model parameters.Salazar et al. [7] presented a fast restoration method based on the prior proposed by He and morphological reconstruction.In addition, some other techniques learn the distributions of different parts scattering models [8,9].Liu et al. [5] added a boundary constraint to Equation (1).They then applied a non-local total variation regularization to solve the color distortion when the patch size is small.Nishino et al. [8] constructed the statistical image model using a factorial Markov random field.They then estimated the parameters using a joint estimation algorithm.Berman et al. [9] estimated initial transmission using the deviation of haze-lines in the hazy images, which is based on a global image color space distribution prior assumption.Bui et al. [10] estimated transmission by a novel color ellipsoid prior to statistically fitting haze color pixel clusters.Berman et al. [11] proposed a robust optimization scheme based on the previous haze-line model, which can achieve faster and better dehazing effects.Wang et al. [12] proposed a fast algorithm to solve the linear assumption model based on dark channel.Zhu et al. [13] offered a pixel-wise fusion weight map to guide multi-exposure image fusion, which improves the performance and robustness of dehazing.Han et al. [14] proposed a mutiscale guided filtering method.Their method could dehaze well by fusing dehazing layer and detail enhancement layer at two scales.Li et al. [15] used bright and dark channel priors for dehazing the segmented sky and non-sky regions separately and proposed a weighting method to fuse the parameters.Such methods are simple and easy to implement, but still suffer with the problems of easy distortion of image color, aggravation of shadow degree, and difficulty in parameter setting.Data-driven methods are based on large datasets and try to learn the mapping of hazy-clear image pairs [16][17][18][19].Li et al. [20] pioneered a direct image dehazing network, which is designed using a re-formulated Equation (1) to learn the model parameters jointly.Li et al. [21] presented a conditional GAN including an autoencoder structure for better dehazing.Qu et al. [22] gave an enhanced Pix2pix dehazing network using a GAN to reconstruct a clear image and an enhancer to improve the reconstructing effect.Li et al. [23] presented a multi-stage hybrid dehazing network using the autoencoder and RNN structure.Chen et al.
[24] defined a pixel-wise patch map to enhance DCP and presented a bi-attentive GAN to learn the patch map adaptively.Zhao et al. [25] proposed a novel dehazing framework that fuses DCP and GAN.Their framework can be trained on unpaired datasets to improve the restoring visibility and result's realness.Compared to the methods based on paired synthetic datasets, Chen et al. [26,27] proposed two unpaired dehazing methods with PDR-GAN and MO-GAN, which also achieved good results.Song et al. [28] introduced the Swin Transformer and designed an improved multiple variants of DehazeFormer for dehazing, which earned a superior effect.Such methods are capable of removing opaque clouds (completely occluding ground objects) in RSIs, but have the following problems: (1) increasingly complex neural networks are used to improve the dehazing effect without considering the resource occupation.(2) in satellite remote sensing systems, it is infeasible to obtain multiple real-world training image pairs of the same location at once.Due to the forementioned issues, the focus of the paper study is on the prior-based single image dehazing method.
Researchers have proposed various methods to address the problem of optical RSI dehazing in recent years.Sun et al. [29] presented a cloud removal network, which first used the RNN to segment cloud regions.They then used the autoencoder to reconstruct cloud-free RSI.Li et al. [30] presented a first-coarse-then-fine network, which first extracts coarse features using autoencoder, and then, extracts detailed features using a subnet.Chen et al. [31] presented a hybrid high-resolution learning framework to achieve single RSI dehazing with fine spatial details.Ding et al. [32] proposed an extended image formation model and divided RSI dehazing into scene depth fusion and adaptive color constancy.Wen et al. [33] presented a Wasserstein GAN-based thin cloud removal network, which effectively reduces both exposure and dark region.Chen et al. [27] gave a memory-oriented GAN for recording haze distribution patterns of single RSI dehazing in an unpaired manner.Bie et al. [34] proposed an RSI dehazing network including feature extraction based on a global attention mechanism and autoencoder with a Gaussian process.Zhou et al. [35] presented a distortion coding and composite loss-based GAN, which can effectively remove thick clouds, as well as thin clouds.

Robust Light-Dark Prior
The proposed shadowing model and hybrid model of optical RSI are explored in detail with the semitransparent cloud as an example.Then, the RLDP and corresponding parameter estimation method is presented.

Haze Hybrid Model
The process of original atmosphere scattering model [36] is shown in Figure 1a.As shown in Equation ( 1), the actual light that reaches the satellite camera consists of two components: direct attenuation and airlight.Direct attenuation is the light reflected from ground object that passes through the cloud and is attenuated by the cloud [37], while airlight is the light reflected from the cloud particles to the satellite camera.The shadow of the semitransparent cloud deepened the color of ground objects captured by the satellite camera, as shown in Figure 1b.Such findings could be attributed to the actual light received by the satellite camera being the indirect attenuation reflected after being attenuated by the cloud.The main differences between Figure 1a,b are: first, the cloud is suspended between the ground object and the camera in the atmosphere scattering model, while the cloud is suspended between the airlight and object in the shadowing model.Second, the cloud particles reflect airlight into the camera in the atmosphere scattering model, while the cloud particles form shadow on the ground object in the shadowing model.Thus, the proposed shadowing model can be expressed as follows: where J(x) represents the observed haze-free but shadowing image, S(x) denotes the shadow-free image to be recovered, and s(x) = e −βd(x+γ) is the shadow transmission map, with β, d(x + γ), and γ being the atmospheric scattering parameter, the cloud transparency and bias, respectively.There are two differences between Equation ( 16) and Equation ( 1): first, the cloud particles do not reflect light into the camera, and, therefore, do not superimpose airlight; second, the bias γ = h ċot θ is introduced, indicating that the transparency of cloud at x is bias from the position of cloud, being mainly related to the height of cloud and the angle of airlight.
A more complex haze hybrid model, as shown in Figure 1c, is the final target.In the model, the cloud is suspended between the airlight and the ground object.It is also between the ground object and the satellite camera.The light emanates from the source, attenuates for the first time when passing through the cloud, and then attenuates for the second time when reflected by the object passing through the cloud.The light entering the camera consists of the indirect-direct attenuated light and the airlight.Indirect attenuated light represents the light that passes through the clouds twice.As such, the proposed hybrid model can be expressed as follows: In Equation (3), t(x), A, and s(x) are all unknown; so, restoring the original image S(x) from Equation (3) seems more difficult.However, the restoration of the original image can be decomposed into two steps: haze removal and shadow removal.In the haze removal stage, any existing dehazing method can be used to estimate the dehazing parameters, such as DCP [6] and CEP [10].In the shadow removal stage, a shadow transmission s(x) estimating method based on the assumption of RLCP is presented.

Robust Light-Dark Prior
The initial prior-based method can be described as selecting a prior vector close to 0 or very low from the pixels in local patches.Recent methods enhance this prior, such as dark channel prior [6], color-lines [38], haze-lines [11], and color ellipsoid prior [10].Symmetrically, the initial light channel prior (LCP) can be calculated as follows: where S c,j denotes a color channel c of image S at a pixel j and Ω i is a local patch centered at i.If we directly solve for the global maximum, it is vulnerable to outlier pixels; we equivalently convert Equation (4) as follows: As the vectors from shadowing image signals are more dense and normally distributed, it is assumed that the pixel values of each channel in Ω(x) obey Gaussian distribution i.e., where mean: Based on the 3σ criterion of general Gaussian distribution [39], the probability of data within [µ − 3σ, µ + 3σ] exceeds 99%.Therefore, to exclude the interference of outlier pixel values in the local patches and improve the robustness of the algorithm, we set the maximum value of the patch to µ + 3σ, i.e., max From Equations ( 5) and ( 7), the robust light channel prior (RLCP) is obtained as follows: Similarly, we improve the dark channel prior using the 3σ criterion, and the proposed robust dark channel prior (RDCP) is formed as follows: Our final robust light-dark prior (RLDP) is based on RLCP and RDCP while satisfying both observations θ RLCP i /s i → min max c∈{r,g,b} S c and θ RDCP i → 0. The former is based on the observation that at least one color channel has some pixels whose intensity is very high and close to the minimum of channel maximums in most RGB patches.

Haze Removal Based on RLDP
Robust atmospheric light estimation: Atmospheric light is usually estimated by the brightest pixels (most haze-opaque pixels), but is easily confused by bright ground objects (especially glaciers, snow, etc.) in RSIs.To address this issue, we use the 3σ criterion when estimating A and use the top 0.1% pixel max of the robust dark channel.The robust atmospheric light estimation method is as follows: Robust transmission map estimation: RDCP satisfies the observation θ RDCP i → 0 and Equations ( 1) and ( 9).Thus, Referring to [6], the robust transmission map can be estimated as follows: where ω(0 < ω ≤ 1) is a constant parameter used to bring the estimation of t closer to the RLCP observation.The value of ω can be determined by the specific application and can be empirically set to 0.95 in this paper.
Robust shadow transmission map estimation: RLCP satisfies the observation θ RLCP i /s i → min max c∈{r,g,b} S c and Equations ( 8) and ( 16); thus, max c∈{r,g,b} (µ c,i + 3σ c,i ) In Figure 2, a pseudo-color composite image of Landsat 8 is shown on the right side.The red box marks a glacier patch (special ground object), and its statistical distribution is depicted in the lower left corner.The green box marks an additional patch, and its statistical distribution is depicted in the upper left corner.The g, b channel values of the glacier patch are too high and do not meet the RLDP assumption.Therefore, to eliminate the confusion of glaciers, we take the minimum of the maximum of three channels (minmaxv).It can also be seen from the statistics that there are some outliers far from most of the pixels in the two patches.We can eliminate the interference of these outliers through the 3σ criterion when calculating the RLDP.Therefore, the shadow transmission map can be estimated as follows: where s(0 ≤ s < minmaxv) is a constant used to bring the estimation of s closer to the RLDP observation.Parameter s can usually be determined by the specific application.
Otherwise, it can be determined by the component of the shadow vector, which is the robust average dark channel of the brightest area: The fast RLDP implementation of the pseudocode is shown in Algorithm 1.This algorithm first assesses the global atmospheric light A by Equations ( 10) and (15), and then, estimates the t i and s i of each pixel i by Equations ( 12) and (14).Finally, the image dehazing is completed through two stages: recovering the haze-free image J and restoring shadow-free image I. Calculate s i by (Equation ( 14)). 8: Restore the haze-free image: Restore the shadow-free image: end for 12: end for

CRME-Based Appropriate Patch Size Search Criterion
As with other priori-based methods, the proposed RLDP requires a pre-set parameter of patch size Ω .As shown in Figure 3, a hazy RSI is shown on the left, and two sets of RLDP dehazing results with different patch sizes (using six patch size values at intervals between 1 and 120) are shown on the right.The upper set (green box) is the dehazing result of the less hazy area, while the lower set (red box) is the dehazing result of the dense hazy area.When setting the patch size to be too small, the image contrast is too high and the color is distorted after dehazing.Instead, when setting the patch size to be too large, the details become blurred and the dehazing effect becomes poor.Moreover, areas in one RSI with different hazy degrees are affected to varying degrees according to the patch size setting.Therefore, determining how to set a global patch size or a local adaptive patch size is an urgent problem.To solve the patch size setting problem, most of the priori-based methods are set a fixed value based on historical experience to avoid a patch size that is too high or too low.In recent years, several adaptive approaches have been proposed.Chen et al. [40] designed a CNN-based patch size selection model, which can adaptively and automatically select the patch size corresponding to each pixel.Chen et al.
[24] defined a pixel-wise patch map and presented a bi-attentive GAN for adaptively learning it by minimizing the error function for each pixel.By analyzing the test results on 50 hazy images, Hong et al. [41] set the patch size to the value when the contrast metric tended to be maximum.
In order to find an appropriate patch size, there is a trade-off in contrast.Inspired by [41], a high Color/Cube RME (CRME), which is an index that measures the relative difference of the patch center and all the neighbors in the patch, indicates a high image contrast.Unlike [41], a stable CRME formula was proposed to search for an appropriate patch size in the shortest possible time.Appropriate patch size (sps) is defined as the search step (where patch size is monotonically increased from 20 to 120) when CRME reaches a stable state for the first time.
where θ is a tiny constant (fixed to 0.001 in this paper) used to measure the stable state of CRME.Generally, the larger θ value, the easier it is to reach a stable state, corresponding to a smaller patch size, and vice versa, corresponding to a larger patch size.

Experimental Setup
Quantitative experiments were conducted on an open-source: Remote sensing Image Cloud rEmoving dataset 1 (RICE1) [42], which included 108 alpine images, 118 flat images, 110 sandy images, 54 mountainous images, and 110 sea images.Overall, RICE1 contained 500 semitransparent cloud and clear RSI pairs collected from Google Earth, all of which were 512 × 512 in size.Subsequently, several 652-band pseudo-color composite images of Landsat 8 images were selected.The study area is a plateau high-altitude area located at row 138, column 37, containing large glaciers.All experiments were performed on an Intel(R) Core (TM) i7-11700 @ 2.50 GHz and 32.0 GB RAM hardware environment with Python 3.9.
In order to quantitatively verify the dehazing performance of RLDP on RSI pairs, three image quality evaluation indicators, peak signal to noise ratio (PSNR), structural similarity (SSIM), and Commission Internationale de l'Eclairage difference evaluation 2000 (CIEDE2000) were used to evaluate the image quality.PSNR measures the quality of image dehazing, with higher values indicating lower distortion and better results.SSIM measures the similarity between the dehazed image and clear image in terms of brightness, contrast and structure.The closer the value is to 1, the more similar and the better results.Compared with the above two indicators, CIEDE2000 is more in line with subjective visual perception, and the lower the value, the smaller the color difference.

Quantitative Evaluations
As shown in Table 1, the average SSIMs on flat, sandy, mountainous, and overall images of the proposed RLDP exceeded other comparison methods, demonstrating that RLDP could perform better dehazing in terms of brightness, contrast, and structure.The average PSNRs on flat, sandy, and overall images of the proposed RLDP exceeded other comparison methods, emanating that there is less distortion in recovered images using RLDP.The average CIEDE2000 of the proposed method wins on alpine, flat, sandy, sea, and overall, indicating that RLDP restores images with higher color quality.RLDP is less effective on alpine and sea, due to the color distribution of the images being narrow.This does not meet the observation of the robust prior, resulting in low t and brighter restored images.Such issues can be remedied by combining a σ threshold with other methods.The DCP and CEEF-TMM could remove semitransparent clouds well, but the recovered images were dark overall.The CL could better restore the original color, but the dehazing effect on some images is poor, particularly for the second images.The CEP had the optimal dehazing effect, but made the image significantly distorted, with the bright areas exposed and the dark areas darkened.The images recovered by the HL had better visibility, but serious exposure could be seen.The UNTV is mainly used for underwater image dehazing, and the color distortion was serious during the present RSI image dehazing.The ACT worked better on the fourth image, 'Sea', but there were slight exposure and color deviation problems in other images.The proposed RLDP method can inherit a better dehazing effect of DCP and can solve the problem that the DCP method makes images darker.In addition, RLDP can make the details clearer in cloud shadow and dark regions of the images.Despite such advantages, the inhomogeneous fog removal effect still needs to be improved due to the bias in the parameter estimation, especially in the atmospheric light estimation (as shown in the fifth and eighth images).The aforementioned experiments show that RLDP can solve the problem of dark region aggravation of existing dehazing methods and can reduce the shadow caused by semitransparent cloud.RLDP is able to increase the visibility of the details in cloud shadow and dark region, and can therefore achieve a better image dehazing quality.

Conclusions
In this paper, an RLDP-based method for single RSI dehazing was proposed.The proposed hybrid model is utilized to estimate parameters, the 3σ criterion is used to improve robustness, and the appropriate patch size search criterion is adopted to improve adaptability.RLDP assumes that the robust dark channel prior or the robust light channel prior is satisfied in most RGB patches.The results of qualitative and quantitative experiments demonstrate the better performance of RLDP.The prior-based approaches become ineffective in the presence of opaque clouds, and thus, further efforts will be made in future research to apply deep learning methods and study complex scenes with the coexistence of semitransparent clouds, opaque clouds, and shadow.

Figure 1 .
Figure 1.The physical model of three different atmosphere scattering models.(a) Original atmosphere scattering model.(b) Shadowing model.(c) Haze hybrid model.

Figure 2 .
Figure 2. Statistical color constructions of ground and glacier patches.

Figure 3 .
Figure 3.A hazy RSI and its 2 sets of RLDP dehazing resluts with different patch size (using 6 patch size values at intervals between 1 and 120) in different areas.The upper set is the dehazing result of the less hazy area and the lower set is the dehazing result of the dense hazy area.

Figures 4 and 5
Figures4 and 5show the dehazing effect on five benchmark RSIs and six Landsat 8 images (to show the details more clearly, the images were cropped to the size of 512 × 512).The DCP and CEEF-TMM could remove semitransparent clouds well, but the recovered images were dark overall.The CL could better restore the original color, but the dehazing effect on some images is poor, particularly for the second images.The CEP had the optimal dehazing effect, but made the image significantly distorted, with the bright areas exposed and the dark areas darkened.The images recovered by the HL had better visibility, but serious exposure could be seen.The UNTV is mainly used for underwater image dehazing, and the color distortion was serious during the present RSI image dehazing.The ACT worked better on the fourth image, 'Sea', but there were slight exposure and color deviation problems in other images.The proposed RLDP method can inherit a better dehazing effect of DCP and can solve the problem that the DCP method makes images darker.In addition, RLDP can make the details clearer in cloud shadow and dark regions of the images.Despite such advantages, the inhomogeneous fog removal effect still needs to be improved due to the bias in the parameter estimation, especially in the atmospheric light estimation (as shown in the fifth and eighth images).