A Shearlets-Based Method for Rain Removal from Single Images

Featured Application: The Shearletes-based rain removal method proposed in this paper can be applied to the destriping of remote sensing images and removing other directional noises. Abstract: This work focuses on the problem of rain removal from a single image. The directional multilevel system, Shearlets, is used to describe the intrinsic directional and structure sparse priors of rain streaks and the background layer. In this paper, a Shearlets-based convex rain removal model is proposed, which involves three sparse regularizers: including the sparse regularizer of rain streaks and two sparse regularizers of the Shearlets transform of background layer in the rain drops’ direction and the Shearlets transform of rain streaks in the perpendicular direction. The split Bregman algorithm is utilized to solve the proposed convex optimization model, which ensures the global optimal solution. Comparison tests with three state-of-the-art methods are implemented on synthetic and real rainy images, which suggests that the proposed method is efﬁcient both in rain removal and details preservation of the background layer.


Introduction
Rain removal from a single image is an important issue in processing outdoor vision problems. In fact, the outdoor images are often degraded by the rain streak and other bad weather conditions, and these bad weather conditions can lead to the change of local or global intensities and color contrast in real images, thus causing unclear visible scenes. Such degradation severely affects the performance of algorithms in computer vision systems. Therefore, the removal of rain streaks is essential.
In recent years, the rain removal problem has caused more and more attention. There are many approaches for rain removal using different modelings. They can be classified into two kinds, including the video-based methods and the single image-based methods. In fact, there are many literatures for the video-based methods [1][2][3][4][5][6][7][8][9][10], the main idea is taking advantage of the rich detail of multiple relative images and the similarity of the image sequences to detect rain streaks and recover the background layer. For instance, Garg and Nayar [1] proposed a correlation model to capture the dynamics of rain in video, then removed rain streaks by a physics-based motion blur model. That is, once rain streaks are detected, the corresponding pixel value of rain is obtained by averaging the rain-free temporal neighbors. In [4], Garg and Nayar found that the video rain visibility relies heavily on the exposure time and the depth of field, based on this fact, they introduced a self-adaption parameter model to remove rain streaks efficiently. In addition, Zhang in [2] gives a new rain removal model, which is based on two priors, one is rain streaks that are almost the same in RGB channels and the other is rain streaks that do not appear everywhere in video. Recently, Jiang et al. proposed a novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors in [10], by fully considering the discriminatively intrinsic characteristics of rain streaks and clean videos, which needs neither rain detection nor time-consuming dictionary learning stage. This method is based on the following priors, including that rain streaks are sparse and smooth in the rain-drop direction and the clean video is smooth along the rain-perpendicular direction, which are both global and local correlation in time direction.
For the single images-based methods, it is more difficult than video-based method due to the limited single image information, thus there are fewer works for single image rain removal. The most successful methods are given in literatures [11][12][13][14][15][16][17][18][19][20][21]. For instance, in [11], based on the assumption that a rainy image can be represented by the rain layer and the background layer, the author proposed a dictionary learning method-based nonlinear screen blend model for rain removal from single images.
The key for this method is that by sparse coding of two layers using the sparse dictionary, by learning sparse dictionary, the accurate layer decompositions are obtained. Moreover, in [14], Kang et al. decomposed the rainy image into low frequency and high frequency parts, and applied an MCA-based dictionary learning model to split the rain streaks layer in the high frequency domain. Following this idea, the author in [16] introduced the structure information into consideration, but the estimated background layer tends to be blurry. Then Li et al. gives another additional model in [13], which is a linear super-imposition of the desired background layer and the rain streak layer, which is an energy minimization model, and the Gaussian mixture model is utilized to learn the corresponding patch-based priors for two layers. The advantage of this model is that it can detect different orientations and scales of the rain streak, which leads to state-of-the-art rain removal results. In fact, these methods described above do remove the rain streaks from rainy images, but the recovery of background seems to be over-smoothed, that is, some details of the background layer are lost. The main reason for details missing in general methods is that the directional property of two layers is out of consideration, which is critical in edges detection. Based on this fact, the author in [21] proposed a directional global sparse model for single image rain removal, in this work, undirectional total removal (UTV) [22] is introduced to describe the basic directional property of single rainy images. The proposed optimization model is based on two directional sparse priors of rain streaks and background layer respectively and a generally sparse prior on rain streaks, that is, rain streaks are sparse in the vertical direction, while the rain-free image (background layer) is sparse in the horizontal direction; furthermore, the rain streak is approximately considered as sparse when the rain is not heavy. The directional characters are captured by rotating rainy images appropriately, which ensures that rain streaks are mainly concentrated in the vertical direction.
In addition, the deep learning-based method gives another way for single image and video rain removal [23][24][25][26], which is totally different from the traditional image processing. The key for this approach is the training network. By learning from lots of rainy images, the learning model can split rain streaks from rainy images after thousands of training images, which is time-consuming, but once the training model gets good generalization, it can deal with lots of rainy images simultaneously, with an accuracy of at least 80 percent. With the development of deep learning, the accuracy for splitting rain streaks can be improved further, and this approach will be widely used for other image processing and computer vision tasks.
In this work, we consider removing rain streaks from single rainy images. This is different from the method in [21], which removes directional rain streaks by rotating the rainy image to the direction that rain streaks are approximately distributed in the vertical direction, then UTV is utilized to capture the 'vertical' characteristics of rain streaks. This is a two-step method when the rain drops are in a non-vertical direction. In fact, directional multilevel transform, e.g., Shearlets transform, can be used to substitute the two-step transform above due to its multi-direction property, also the multilevel system is suitable for detecting rain streaks as the shape of rain streaks is not necessarily rectangular. In addition, we found that the Shearlets decomposition coefficients of rainy images in specific scale and directional frequency bands are sparse. Some statistical analysis (see Figure 1) suggests that the Shearlets coefficients of rainy images in the rain drops' direction has sparse structure, also the Shearlets coefficients of rain-free (background) layer in the rain streaks direction is sparse. Moreover, another sparse prior is the rain streaks itself, which can be approximately considered as sparse when the rain is not heavy (partly sparse when the rain is heavy). Combining these three sparse priors above, we propose a convex optimization model and the split Bregman algorithm [27] is utilized to solve the proposed model. Numerical experiments on synthetic and real rainy images demonstrate that our Shearlets-based method outperforms the recently widely used rain removal methods in [11,13,21]. The outline of this paper is given as follows: Section 2 gives a brief introduction of the motivation. Section 3 reviews the Shearltes transform. In Section 4, a Shearlets-based rain streaks removal model, and the corresponding algorithm, are proposed. Section 5 compares our model with three state-of-art methods, some discussions of rain removal results are given in Section 6. In the end, the conclusion is given in Section 7.

Motivations of the Proposed Method
In general, the rain streak removal model can be represented as follows: where o ∈ R M×N is the rainy image, r ∈ R M×N represents rain streaks layer, and b ∈ R M×N is the background layer (rain-free layer).
Motivated by the work in [14], which decomposed rainy images into high-and low-frequency, then the high-frequency parts can be represented by the addition of rain streaks layer and rain-free layer by performing dictionary learning and sparse coding. This method does remove rain streaks from the rainy image, but the rain-free layer estimated tends to be blurry, which implies that the high-frequency decomposition leads to the loss of details in high frequency not only from rain streaks, but also from the details of the background layer. This makes us consider splitting rain streaks in specific frequency bands, so that more details can be preserved. Therefore, the directional multilevel transform, such as Shearlets transform, is a good choice as it can be used to separate the frequency bands of rain streaks only.
In fact, the Shearlets transform is a powerful tool in detecting directional singularities in frequency band [28][29][30]. Different from UTV, which is used to enforce the sparse prior of the gradient along vertical and horizontal directions, the Shearlets can be utilized to describe the variation along different directions in various scales (see Figure 2). Since the rain streaks can be viewed as a particular directional singularity in different scales, which makes it possible to remove from the rainy image without additional details missing. However, loss of the details from the rain-free layer in corresponding frequency bands cannot be avoided. In order to minimize this kind of details loss, the multi-direction Shearlets is used to preserve most original image details. Compared with the recent state-of-the-art directional UTV rotation-based rain removal model in [21], the proposed model has the following advantages: - The directional multilevel transform, Shearlets transform, is utilized to describe the sparse structure of rain streaks and the background layer. Different from the rotation UTV method, the Shearlets-based method can obtain the gradient variation in different directions and different scales efficiently, thus the recovery keeps more directional singularities details. -For different directions of rain streaks, the Shearlets transform can capture the rain streak layer details due to its multi-direction property. Moreover, the fast algorithm for solving the proposed model can be obtained as the discrete Shearlets transform and inverse are available. - The split Bregman algorithm is utilized to solve the proposed convex optimization model, which guarantees that the solver is global optimal. The computation of algorithm includes three soft-thresholding processes and two Shearlets transforms, and the total computing complexity is where N is the total number of pixels.

Shearlets
Shearlets system is first given in [31] by Kutyniok et al, which not only inherits the multiscale structure of wavelets, but also possesses the directional windows in the frequency domain by introducing the shear operator, thus making it possible for the Shearlets transform to capture the directional singularities in images and high dimensional signals. This section gives a retrospection of the basic definition and construction of Shearlets, for more details refer to [28,32].
For the basic wavelet function φ ∈ L 2 (R 2 ), it is supposed to satisfy whereφ represents the Fourier transform of φ, and it has the following separated form: with φ 1 ∈ L 2 (R), φ 2 ∈ L 2 (R 2 ). Then the Shearlets multilevel system S is generated by where T t is the translation operator, T t φ = φ(· − t). D s is the scaling operator and S s is the shear operator, their definitions are given as where j ∈ Z is the scaling parameter, which decides the decomposition layer. s ∈ R is the shearing parameter, which decides the multi-direction property. t is the translation parameter, which decides the number of sub-bands in each scale. For the signal f ∈ L 2 (R 2 ), its continuous Shearlets transform where S maps f into the Shearlets coefficients S(φ) f (j, s, t) with different scale j, shearing s, and translation t. Then the discrete Shearlets system can be given as where The multilevel system Ψ(ψ; c) corresponds to the low frequency area in the frequency domain, while the system Φ(φ; c) andΦ(φ; c) corresponds to the high frequency region in vertical and horizontal direction respectively. The sub-band of S(ψ, φ; c) in the frequency domain is shown in Figure 3.

The Proposed Optimization Model
This subsection describes the proposed convex model with three sparse prior regularizers and some non-negative constraints as follows: 1.
The sparse constraint of the Shearets coefficients of the background layer in the rain drops' direction. In fact, the Shearlets decomposition coefficients of the background layer in high frequency can be approximately considered (in the rain drops' direction) as sparse (see Figure 1) due to the intensity distribution for multilevel coefficients. In order to describe this sparse regularizer, the 1 norm of multilevel coefficients in different scales, along the direction of the rain-free layer is used, which also reflects the discontinuity of rain streaks, specifically where S r b is the Shearlets transform of the rain-free layer in the rain drops' direction.

2.
The sparse constraint of the Shearlets coefficients of the rain streaks across the rain drops' direction.
In the real scenes of rainfall, the shape of rain streaks is a stretched ellipse in a specific direction reflected in pixels, therefore the directional multilevel transform is more sensitive than UTV in detecting the directional singularities in images. From Figure 2, it shows that the Shearlets coefficient in scale 2 across the rain drops' direction of the rain streak has sparse structure. Similarly, Reg 2 (r) = S b r 1 .
where S b r is the Shearlets transform of the rain streak across the rain drops' direction.

3.
The sparse constraint of rain streaks (see Figure 1). In general, the rain streak is sparse when the rain is not heavy, therefore its sparsity can be described by 0 norm, which represents the number of nonzero elements. Here, the 1 is utilized to replace the 0 norm due to its non-convexity, thus we have the following sparse regularizer of rain streaks 4. Some non-negative constraints of r and b. For the rain streaks removal problem, the pixel of the rain streaks layer r and the background layer b are non-negative, therefore the following constraints hold By analyzing the property of multi-direction Shearlets transform of rain streaks and background layers, the corresponding sparse regularizers are obtained. With constraints of r and b, the optimization model for solving single image rain removal problem can be given as follows: where γ i , i = 1, 2, 3 are positive regularization parameters. The convexity of (13) guarantees its solver to be global optimal.

Solving the Proposed Model
To solve the proposed model, the split Bregman algorithm is utilized [27], and the split step is implemented by introducing new variables: as the 1 -norm is not differentiable, thus (13) has the following equivalent form for non-negative constrains of r and b, they can be implemented by a projection operator. The corresponding Bregman iterations can be given as Then the v i -sub-problem i = 1, 2, 3 can be given as they can be solved accurately by soft-thresholding strategy as follows for r = 1, ..., N, N represents the total number of pixels, where Sh(x, y) = sign(x) max(|x| − y, 0); with The r-sub-problem can be given by minimizing the following equation since the 2 norm is differentiable, we have the following closed-form solver for r-sub-problem where I is the identity matrix. From [28], the Shearlets transform is constructed to be a Parseval frame, which implies that S * r S r = I, then the r-sub-problem above can be solved by Fast Fourier Transform (FFT, F ) efficiently.
where F −1 represents the Fast Inverse Fourier Transform. In the end, the constrains for r ≥ 0 and b ≥ 0 can be projected by the following formula: Having explained the main idea of our method, the final resulting algorithm is displayed in Algorithm 1.

Numerical Experiments
In this section, some comparison tests are performed to validate the effectiveness of the proposed method. We compare the proposed method with three recent state-of-art rain removal methods, including a dictionary learning-based algorithm in [11] (15ICCV), a minimization model with the learned rain layer prior method in [13] (16CVPR), and a directional global sparse model in [21] (18UTV). All tests are implemented in a laptop, CPU: Inter Core i5, memory: 16 GB, OS: Windows 10, testing software: Matlab R2016a.
Since humans are more sensitive to changes in luminance, all the RGB testing images are converted to YUV space. The proposed method is utilized to remove rain streaks only in the luminance channel (three compared methods are also implemented on the luminance channel). To be more persuasive, we use two kinds of data for experiments, the synthetic data, and real data. For the synthetic data, the relative error (RelErr), peak signal-noise ration (PSNR), and the structural similarity (SSIM) [33] are used to estimate the performance of different methods.

•
The relative error (RelErr) is defined as where u and u r are the original signal and the recovered signal, and · F represents the Frobenius norm.

•
The peak signal-noise ration (PSNR) is defined as where u max is the maximum value of u, and where M, N are the size of the signal u.

•
The structural similarity (SSIM) is defined as where µ u and µ u r represent the average of u and u r , σ u , and σ u r represent the standard deviation of u and u r , σ uu r is the covariance of u, and u r , c 1 , c 2 > 0.

Comparison Tests of the Synthetic Data
The testing results and discussions for synthetic data are given in this subsection. First, the synthetic rainy image is generated as follows: (1) adding the salt and pepper noise with a random density den (which decides the density of rain streaks) to a zero matrix; (2) convoluting (1) with a motion kernel (including the directional parameter θ and motion parameter len); (3) adding (2) to the rain-free images to obtain the rainy image [14]. The parameters of synthetic rainy image for testing are given in Table 1. Then these RGB rainy images are converted to YUV channels, and only the rain streaks in the Y channel are removed using the 15ICCV method, the 16CVPR method, the 18UTV method, and the proposed method. In this test, we choose 5 images as the testing images, which can be downloaded from the dataset "UCID" (http://homepages.lboro.ac.uk/cgs/datasets/ucid/ucid.html) and (https://pixabay.com/). The testing image are given in Figure 4. The visual results for 4 methods are displayed in Figure 5. The first column displays the de-rain results by the 15ICCV method, the second column shows the de-rain results by the 16CVPR method, the third column gives the de-rain results by the 18UTV method, and the last column shows the de-rain results by the proposed method.  From Figure 5 and the corresponding zooming in images in Figure 6, we find that the method 15ICCV fails to remove the rain streak completely, which leads to the lower SSIM value, and while methods 16CVPR and 18UTV perform better than 15ICCV, they cannot outperform our method. Comparing with three methods, the proposed method removes rain streaks and preserves the details of the background layer more efficiently. Then we have the conclusion as follows: • the method 15ICCV cannot remove rain streaks completely, especially with heavy rain; • the method 16CVPR does remove rain streaks completely, but the resulted background seems to be over-smooth; • the method 18UTV can remove rain streaks completely, but rain streaks are detected by TV, which leads to a non-smooth background, especially in the heavy rain case; • the proposed method using multilevel system performs better both in rain streaks removal and details preservation in heavy and light rain cases.
The quantitative results for different methods with different rain streaks' direction are presented in Table 2, including PSNR, SSIM, and RelErr values for both the background layer and the rain streak layer in heavy and light rain cases.
In addition, we randomly choose 20 images from UCID dataset to further test the rain removal performance in different directional rainy images. The quantitative results of four different methods in terms of RelErr, PSNR, and SSIM value are shown in Figure 7.
15ICCV 16CVPR 18UTV O Figure 6. The corresponding local enlarged view of four methods for synthetic images.

Results and Discussion for the Real Data
This subsection tests the performance of different methods under real rainy images. In Figure 8, the real rainy images and their local enlarged view are displayed. The testing images can be downloaded from the website.

Rainy image
Rainy image The visual results recovered by four methods are displayed in Figure 9, the first column shows de-rain results by the 15ICCV method, the second column displays the de-rain results by the 16CVPR method, the third column gives the de-rain results by the 18UTV method, and the last row shows the de-rain results by the proposed method.

15ICCV
16CVPR 18UTV Ours Figure 9. The visual results of four methods for real rainy images.
From Figure 9 and the corresponding local enlarged view in Figure 10, we can get similar results as the synthetic data. The method 15ICCV still has rain streaks remaining in the background layer, methods 16CVPR and 18UTV do significantly remove the rain streaks, but fail to detect directional singularities, while the proposed method performs well both in rain removal and detail preservation of the rain-free layer. Two test results suggest that the proposed method can remove the directional rain streaks efficiently. Figure 10. The corresponding local enlarged view of four methods for real rainy images.

Computation Complexity
The computation complexity of our algorithm includes two Shearlets transforms computation and four sub-problems with closed-form solutions. For two Shearlets transforms, its computation complexity is about O(N log N) + O(N). The computation complexity of four sub-problems is O(N), thus the total complexity for Algorithm 1 is O(N log N) + O(N), where N represents the total number of pixels. Table 2 shows the computation time comparison of four methods. For 'Pic1' with size [559 × 314 × 3], the computing time of the proposed method is 2.39 s, while the 15ICCV and 16CVPR methods take 72 s and 946 s respectively, and the 18UTV method only takes 0.87 s (without direction rotation). Our explanation is that the computation of Shearlets transform needs more computing time than UTV. Thus, we should strike a balance between the algorithm efficiency and computation complexity.

Description for Parameters
The proposed method includes two kinds of parameters, the model parameter and the algorithm parameter. For the model parameter, γ 1 , γ 2 , γ 3 are used to constrain the sparsity of Shearlets decomposition coefficients in high frequency, thus the recovery is sensitive to their changes. But different testing images with directional rain streaks are not sensitive to the parameters γ i , α i , i = 1, 2, 3. In the testing, we set γ 1 For the algorithm parameter tol, M iter , they are chosen to be tol = 1e − 3 and M iter = 250. (for some images, parameters are chosen different to obtain the best result). The default setting for the 15ICCV, 16CVPR, and 18UTV methods parameters can be found in [21]. We give the average quantitative performance of 20 images of UCID in Table 3.

Simple Discussion of Regularization Terms
In order to understand the effect of three sparse terms, some tests are implemented to reveal their contribution for rain removal. We choose real rainy image "gentlemen" as the testing image (see Figure 11a). The visual results after discarding sparse terms Reg 1 , Reg 2 , Reg 3 are listed in Figure 11c-e. From the recovery, we find that the regularizers Reg 1 , Reg 2 have a remarkable effect on the rain removal, since Reg 1 , Reg 2 describe the sparsity of the Shearlets transform of rainy images in the rain drops' direction and the perpendicular direction respectively.

Convergency of the Proposed Algorithm
In Figure 12, we display the convergency curve of the proposed algorithm for the testing image 'gentleman'. It is easy to find that the RelErr value decreases as the iteration number increases, while the PSNR value and SSIM value increase with the iteration number increasing, which suggests that the proposed algorithm is stable.

Conclusions
This work proposed an efficient convex rain removal model for single images, which is based on the sparse prior of rainy streaks and the background layer. The split Bregman algorithm is utilized to solve our model, which ensures the global optimal. We test the synthetic and real data to demonstrate that the proposed method performs better both in rain removal and details preservation of background layer than the comparable three methods.
Author Contributions: G.S. and C.C. designed the convex optimization model for rain removal of single images and performed the experiments, G.S. and J.L. analyzed the results. G.S. wrote the paper. All authors have read and approved the final manuscript.