A Shearlets-Based Method for Rain Removal from Single Images

Featured Application

The Shearletes-based rain removal method proposed in this paper can be applied to the destriping of remote sensing images and removing other directional noises.

Abstract

This work focuses on the problem of rain removal from a single image. The directional multilevel system, Shearlets, is used to describe the intrinsic directional and structure sparse priors of rain streaks and the background layer. In this paper, a Shearlets-based convex rain removal model is proposed, which involves three sparse regularizers: including the sparse regularizer of rain streaks and two sparse regularizers of the Shearlets transform of background layer in the rain drops’ direction and the Shearlets transform of rain streaks in the perpendicular direction. The split Bregman algorithm is utilized to solve the proposed convex optimization model, which ensures the global optimal solution. Comparison tests with three state-of-the-art methods are implemented on synthetic and real rainy images, which suggests that the proposed method is efficient both in rain removal and details preservation of the background layer.

1. Introduction

Rain removal from a single image is an important issue in processing outdoor vision problems. In fact, the outdoor images are often degraded by the rain streak and other bad weather conditions, and these bad weather conditions can lead to the change of local or global intensities and color contrast in real images, thus causing unclear visible scenes. Such degradation severely affects the performance of algorithms in computer vision systems. Therefore, the removal of rain streaks is essential.

In recent years, the rain removal problem has caused more and more attention. There are many approaches for rain removal using different modelings. They can be classified into two kinds, including the video-based methods and the single image-based methods. In fact, there are many literatures for the video-based methods [1,2,3,4,5,6,7,8,9,10], the main idea is taking advantage of the rich detail of multiple relative images and the similarity of the image sequences to detect rain streaks and recover the background layer. For instance, Garg and Nayar [1] proposed a correlation model to capture the dynamics of rain in video, then removed rain streaks by a physics-based motion blur model. That is, once rain streaks are detected, the corresponding pixel value of rain is obtained by averaging the rain-free temporal neighbors. In [4], Garg and Nayar found that the video rain visibility relies heavily on the exposure time and the depth of field, based on this fact, they introduced a self-adaption parameter model to remove rain streaks efficiently. In addition, Zhang in [2] gives a new rain removal model, which is based on two priors, one is rain streaks that are almost the same in RGB channels and the other is rain streaks that do not appear everywhere in video. Recently, Jiang et al. proposed a novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors in [10], by fully considering the discriminatively intrinsic characteristics of rain streaks and clean videos, which needs neither rain detection nor time-consuming dictionary learning stage. This method is based on the following priors, including that rain streaks are sparse and smooth in the rain-drop direction and the clean video is smooth along the rain-perpendicular direction, which are both global and local correlation in time direction.

For the single images-based methods, it is more difficult than video-based method due to the limited single image information, thus there are fewer works for single image rain removal. The most successful methods are given in literatures [11,12,13,14,15,16,17,18,19,20,21]. For instance, in [11], based on the assumption that a rainy image can be represented by the rain layer and the background layer, the author proposed a dictionary learning method-based nonlinear screen blend model for rain removal from single images. The key for this method is that by sparse coding of two layers using the sparse dictionary, by learning sparse dictionary, the accurate layer decompositions are obtained. Moreover, in [14], Kang et al. decomposed the rainy image into low frequency and high frequency parts, and applied an MCA-based dictionary learning model to split the rain streaks layer in the high frequency domain. Following this idea, the author in [16] introduced the structure information into consideration, but the estimated background layer tends to be blurry. Then Li et al. gives another additional model in [13], which is a linear super-imposition of the desired background layer and the rain streak layer, which is an energy minimization model, and the Gaussian mixture model is utilized to learn the corresponding patch-based priors for two layers. The advantage of this model is that it can detect different orientations and scales of the rain streak, which leads to state-of-the-art rain removal results. In fact, these methods described above do remove the rain streaks from rainy images, but the recovery of background seems to be over-smoothed, that is, some details of the background layer are lost. The main reason for details missing in general methods is that the directional property of two layers is out of consideration, which is critical in edges detection. Based on this fact, the author in [21] proposed a directional global sparse model for single image rain removal, in this work, undirectional total removal (UTV) [22] is introduced to describe the basic directional property of single rainy images. The proposed optimization model is based on two directional sparse priors of rain streaks and background layer respectively and a generally sparse prior on rain streaks, that is, rain streaks are sparse in the vertical direction, while the rain-free image (background layer) is sparse in the horizontal direction; furthermore, the rain streak is approximately considered as sparse when the rain is not heavy. The directional characters are captured by rotating rainy images appropriately, which ensures that rain streaks are mainly concentrated in the vertical direction.

In addition, the deep learning-based method gives another way for single image and video rain removal [23,24,25,26], which is totally different from the traditional image processing. The key for this approach is the training network. By learning from lots of rainy images, the learning model can split rain streaks from rainy images after thousands of training images, which is time-consuming, but once the training model gets good generalization, it can deal with lots of rainy images simultaneously, with an accuracy of at least 80 percent. With the development of deep learning, the accuracy for splitting rain streaks can be improved further, and this approach will be widely used for other image processing and computer vision tasks.

In this work, we consider removing rain streaks from single rainy images. This is different from the method in [21], which removes directional rain streaks by rotating the rainy image to the direction that rain streaks are approximately distributed in the vertical direction, then UTV is utilized to capture the ‘vertical’ characteristics of rain streaks. This is a two-step method when the rain drops are in a non-vertical direction. In fact, directional multilevel transform, e.g., Shearlets transform, can be used to substitute the two-step transform above due to its multi-direction property, also the multilevel system is suitable for detecting rain streaks as the shape of rain streaks is not necessarily rectangular. In addition, we found that the Shearlets decomposition coefficients of rainy images in specific scale and directional frequency bands are sparse. Some statistical analysis (see Figure 1) suggests that the Shearlets coefficients of rainy images in the rain drops’ direction has sparse structure, also the Shearlets coefficients of rain-free (background) layer in the rain streaks direction is sparse. Moreover, another sparse prior is the rain streaks itself, which can be approximately considered as sparse when the rain is not heavy (partly sparse when the rain is heavy). Combining these three sparse priors above, we propose a convex optimization model and the split Bregman algorithm [27] is utilized to solve the proposed model. Numerical experiments on synthetic and real rainy images demonstrate that our Shearlets-based method outperforms the recently widely used rain removal methods in [11,13,21].

The outline of this paper is given as follows: Section 2 gives a brief introduction of the motivation. Section 3 reviews the Shearltes transform. In Section 4, a Shearlets-based rain streaks removal model, and the corresponding algorithm, are proposed. Section 5 compares our model with three state-of-art methods, some discussions of rain removal results are given in Section 6. In the end, the conclusion is given in Section 7.

2. Motivations of the Proposed Method

In general, the rain streak removal model can be represented as follows:

$$o=r+b;$$

(1)

where $o\in R^{M\times N}$ is the rainy image, $r\in R^{M\times N}$ represents rain streaks layer, and $b\in R^{M\times N}$ is the background layer (rain-free layer).

Motivated by the work in [14], which decomposed rainy images into high- and low-frequency, then the high-frequency parts can be represented by the addition of rain streaks layer and rain-free layer by performing dictionary learning and sparse coding. This method does remove rain streaks from the rainy image, but the rain-free layer estimated tends to be blurry, which implies that the high-frequency decomposition leads to the loss of details in high frequency not only from rain streaks, but also from the details of the background layer. This makes us consider splitting rain streaks in specific frequency bands, so that more details can be preserved. Therefore, the directional multilevel transform, such as Shearlets transform, is a good choice as it can be used to separate the frequency bands of rain streaks only.

In fact, the Shearlets transform is a powerful tool in detecting directional singularities in frequency band [28,29,30]. Different from UTV, which is used to enforce the sparse prior of the gradient along vertical and horizontal directions, the Shearlets can be utilized to describe the variation along different directions in various scales (see Figure 2). Since the rain streaks can be viewed as a particular directional singularity in different scales, which makes it possible to remove from the rainy image without additional details missing. However, loss of the details from the rain-free layer in corresponding frequency bands cannot be avoided. In order to minimize this kind of details loss, the multi-direction Shearlets is used to preserve most original image details.

Compared with the recent state-of-the-art directional UTV rotation-based rain removal model in [21], the proposed model has the following advantages:

-

The directional multilevel transform, Shearlets transform, is utilized to describe the sparse structure of rain streaks and the background layer. Different from the rotation UTV method, the Shearlets-based method can obtain the gradient variation in different directions and different scales efficiently, thus the recovery keeps more directional singularities details.

-

For different directions of rain streaks, the Shearlets transform can capture the rain streak layer details due to its multi-direction property. Moreover, the fast algorithm for solving the proposed model can be obtained as the discrete Shearlets transform and inverse are available.

-

The split Bregman algorithm is utilized to solve the proposed convex optimization model, which guarantees that the solver is global optimal. The computation of algorithm includes three soft-thresholding processes and two Shearlets transforms, and the total computing complexity is $O(NlogN)+O\left(N\right)$, where N is the total number of pixels.

3. Shearlets

Shearlets system is first given in [31] by Kutyniok et al, which not only inherits the multiscale structure of wavelets, but also possesses the directional windows in the frequency domain by introducing the shear operator, thus making it possible for the Shearlets transform to capture the directional singularities in images and high dimensional signals. This section gives a retrospection of the basic definition and construction of Shearlets, for more details refer to [28,32].

For the basic wavelet function $\varphi \in L^2\left(R^2\right)$, it is supposed to satisfy

$$C_{\varphi }={\int }_{R^2}\frac{|\widehat{\varphi }{\left(\omega \right)|}^2}{\left|\omega \right|}d\omega <\infty ,$$

(2)

where $\widehat{\varphi }$ represents the Fourier transform of $\varphi$, and it has the following separated form:

$$\widehat{\varphi }({\omega }_1,{\omega }_2)={\widehat{\varphi }}_1\left({\omega }_1\right){\widehat{\varphi }}_2\left(\frac{{\omega }_2}{{\omega }_1}\right),$$

(3)

with ${\varphi }_1\in L^2\left(R\right),{\varphi }_2\in L^2\left(R^2\right)$. Then the Shearlets multilevel system S is generated by

$$S\left(\varphi \right)=\{{\varphi }_{j,s,t}=T_t{D}_2^j{S}_s\varphi :j\in Z,s\in R,t\in R^2\},$$

(4)

where $T_t$ is the translation operator, $T_t\varphi =\varphi (·-t)$. $D_s$ is the scaling operator and $S_s$ is the shear operator, their definitions are given as

$$D_2^j=\left(\begin{array}{cc}{2}^j& 0\\ 0& 2^{j/2}\end{array}\right),S_s=\left(\begin{array}{cc}1& s\\ 0& 1\end{array}\right).$$

(5)

where $j\in Z$ is the scaling parameter, which decides the decomposition layer. $s\in R$ is the shearing parameter, which decides the multi-direction property. t is the translation parameter, which decides the number of sub-bands in each scale. For the signal $f\in L^2\left(R^2\right)$, its continuous Shearlets transform

$$f\to S\left(\varphi \right)f(j,s,t)=<f,{\varphi }_{j,s,t}>,j\in Z,s\in R,t\in R^2,$$

(6)

where S maps f into the Shearlets coefficients $S\left(\varphi \right)f(j,s,t)$ with different scale j, shearing s, and translation t. Then the discrete Shearlets system can be given as

Definition 1

([28]). Let $\psi ,\varphi \in L^2\left(R^2\right)$, $c=(c_1,c_2)\in R^2$, the definition of the discrete Shearlets system $S(\psi ,\varphi ;c)$ is

$$S(\psi ,\varphi ;c)=\mathsf{\Psi }(\psi ;c)\cup \mathsf{\Phi }(\varphi ;c)\cup \tilde{\mathsf{\Phi }}(\tilde{\varphi };c),$$

(7)

where

$$\begin{array}{cc}\hfill \mathsf{\Psi }(\psi ;c)& =\{{\psi }_{j,m}=T_t{D}_2^j\psi =2^{3/4}\psi (D_2^j·-cm):m\in Z^2\},\hfill \\ \hfill \mathsf{\Phi }(\varphi ;c)& =\{{\varphi }_{j,k,m}=2^{3/4}\varphi (S_k{D}_2^j·-cm):j\ge 0,|k|\le 2^{j/2},m\in Z^2\},\hfill \\ \hfill \tilde{\mathsf{\Phi }}(\tilde{\varphi };c)& =\{{\tilde{\varphi }}_{j,k,m}=2^{3/4}\tilde{\varphi }(S_k{D}_2^j·-cm):j\ge 0,|k|\le 2^{j/2},m\in Z^2\}.\hfill \end{array}$$

(8)

with $\widehat{\psi }={\widehat{\varphi }}_1\left({\omega }_1\right){\widehat{\varphi }}_2\left({\omega }_2\right)$, and $\tilde{\varphi }({\xi }_1,{\xi }_2)=\varphi ({\xi }_2,{\xi }_1)$.

The multilevel system $\mathsf{\Psi }(\psi ;c)$ corresponds to the low frequency area in the frequency domain, while the system $\mathsf{\Phi }(\varphi ;c)$ and $\tilde{\mathsf{\Phi }}(\tilde{\varphi };c)$ corresponds to the high frequency region in vertical and horizontal direction respectively. The sub-band of $S(\psi ,\varphi ;c)$ in the frequency domain is shown in Figure 3.

4. The Proposed Method

4.1. The Proposed Optimization Model

This subsection describes the proposed convex model with three sparse prior regularizers and some non-negative constraints as follows:

• The sparse constraint of the Shearets coefficients of the background layer in the rain drops’ direction. In fact, the Shearlets decomposition coefficients of the background layer in high frequency can be approximately considered (in the rain drops’ direction) as sparse (see Figure 1) due to the intensity distribution for multilevel coefficients. In order to describe this sparse regularizer, the ${\ell }_1$ norm of multilevel coefficients in different scales, along the direction of the rain-free layer is used, which also reflects the discontinuity of rain streaks, specifically

  $$Re{g}_1\left(b\right)=\parallel S_r{b\parallel }_1={\parallel S_r(o-r)\parallel }_1;$$

  (9)

  where $S_{r}b$ is the Shearlets transform of the rain-free layer in the rain drops’ direction.
• The sparse constraint of the Shearlets coefficients of the rain streaks across the rain drops’ direction. In the real scenes of rainfall, the shape of rain streaks is a stretched ellipse in a specific direction reflected in pixels, therefore the directional multilevel transform is more sensitive than UTV in detecting the directional singularities in images. From Figure 2, it shows that the Shearlets coefficient in scale 2 across the rain drops’ direction of the rain streak has sparse structure. Similarly,

  $$Re{g}_2\left(r\right)={\parallel S_{b}r\parallel }_1.$$

  (10)

  where $S_{b}r$ is the Shearlets transform of the rain streak across the rain drops’ direction.
• The sparse constraint of rain streaks (see Figure 1). In general, the rain streak is sparse when the rain is not heavy, therefore its sparsity can be described by ${\ell }_0$ norm, which represents the number of nonzero elements. Here, the ${\ell }_1$ is utilized to replace the ${\ell }_0$ norm due to its non-convexity, thus we have the following sparse regularizer of rain streaks

  $$Re{g}_3\left(r\right)={\parallel r\parallel }_1.$$

  (11)
• Some non-negative constraints of r and b. For the rain streaks removal problem, the pixel of the rain streaks layer r and the background layer b are non-negative, therefore the following constraints hold

  $$Co{n}_1(r,b):r\ge 0;b\ge 0.$$

  (12)

By analyzing the property of multi-direction Shearlets transform of rain streaks and background layers, the corresponding sparse regularizers are obtained. With constraints of r and b, the optimization model for solving single image rain removal problem can be given as follows:

$$\underset{s}{min}{\gamma }_1\parallel S_r{(o-r)\parallel }_1+{\gamma }_2\parallel S_b{r\parallel }_1+{\gamma }_3{\parallel r\parallel }_1,s.t.r\ge 0;b\ge 0;$$

(13)

where ${\gamma }_i,i=1,2,3$ are positive regularization parameters. The convexity of (13) guarantees its solver to be global optimal.

4.2. Solving the Proposed Model

To solve the proposed model, the split Bregman algorithm is utilized [27], and the split step is implemented by introducing new variables:

$$\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)=\left(\begin{array}{c}{S}_r(o-r)\\ S_{b}r\\ r\end{array}\right)$$

(14)

as the ${\ell }_1$-norm is not differentiable, thus (13) has the following equivalent form

$$\begin{array}{c}\hfill min{\gamma }_1\parallel v_1{\parallel }_1+{\gamma }_2\parallel v_2{\parallel }_1+{\gamma }_3{\parallel v_3\parallel }_1\\ \hfill s.t.v_1=S_r(o-r),v_2=S_{b}r,v_3=r;r\ge 0;b\ge 0;\end{array}$$

(15)

for non-negative constrains of r and b, they can be implemented by a projection operator. The corresponding Bregman iterations can be given as

$$\left\{\begin{array}{cc}\hfill (v_1^{k+1},v_2^{k+1},v_3^{k+1},r^{k+1})& =arg\underset{v_1,v_2,v_3,r}{min}{\gamma }_1\parallel v_1{\parallel }_1+{\gamma }_2\parallel v_2{\parallel }_1+{\gamma }_3\parallel v_3{\parallel }_1+\frac{{\alpha }_1}{2}\parallel S_r(o-r)\hfill \\ \hfill & -v_1-b_{v_1}^k{\parallel }_2^2+\frac{{\alpha }_2}{2}\parallel S_{b}r-v_2-b_{v_2}^k{\parallel }_2^2+\frac{{\alpha }_3}{2}{\parallel r-v_3-b_{v_3}^k\parallel }_2^2;\hfill \\ \hfill b_{v_1}^{k+1}& =b_{v_1}^k+S_r(o-r)-v_1;\hfill \\ \hfill b_{v_2}^{k+1}& =b_{v_2}^k+S_{b}r-v_2;\hfill \\ \hfill b_{v_3}^{k+1}& =b_{v_3}^k+r-v_3;\hfill \end{array}\right.$$

(16)

Then the $v_i$-sub-problem $i=1,2,3$ can be given as

$$\left\{\begin{array}{cc}\hfill v_1^{k+1}& =arg\underset{v_1}{min}{\gamma }_1\parallel v_1{\parallel }_1+\frac{{\alpha }_1}{2}{\parallel S_r(o-r)-v_1-b_{v_1}^k\parallel }_2^2;\hfill \\ \hfill v_2^{k+1}& =arg\underset{v_2}{min}{\gamma }_2\parallel v_2{\parallel }_1+\frac{{\alpha }_2}{2}{\parallel S_{b}r-v_2-b_{v_2}^k\parallel }_2^2;\hfill \\ \hfill v_3^{k+1}& =arg\underset{v_3}{min}{\gamma }_3\parallel v_3{\parallel }_1+\frac{{\alpha }_3}{2}{\parallel r-v_3-b_{v_3}^k\parallel }_2^2;\hfill \end{array}\right.$$

(17)

they can be solved accurately by soft-thresholding strategy as follows

$$\left\{\begin{array}{cc}\hfill v_1^{k+1}& =\mathrm{Sh}\left((S_r(o-r^k)+b_{v_1}^k)\left(r\right),\frac{{\gamma }_1}{{\alpha }_1}\right);\hfill \\ \hfill v_2^{k+1}& =\mathrm{Sh}\left((S_b{r}^k+b_{v_2}^k)\left(r\right),\frac{{\gamma }_2}{{\alpha }_2}\right);\hfill \\ \hfill v_3^{k+1}& =\mathrm{Sh}\left((r^k+b_{v_3}^k)\left(r\right),\frac{{\gamma }_3}{{\alpha }_3}\right);\hfill \end{array}\right.$$

(18)

for $r=1,\dots ,N$, N represents the total number of pixels, where

$$\mathrm{Sh}(x,y)=\mathrm{sign}\left(x\right)max\left(\right|x|-y,0);$$

(19)

with

$$\mathrm{sign}\left(x\right)=\left\{\begin{array}{cc}1,\hfill & x>0;\hfill \\ 0,\hfill & x=0;\hfill \\ -1,\hfill & x<0.\hfill \end{array}\right.$$

(20)

The r-sub-problem can be given by minimizing the following equation

$$r^{k+1}=arg\underset{r}{min}\frac{{\alpha }_1}{2}\parallel S_r(o-r)-v_1^k-b_{v_1}^k{\parallel }_2^2+\frac{{\alpha }_2}{2}\parallel S_{b}r-v_2^k-b_{v_2}^k{\parallel }_2^2+\frac{{\alpha }_3}{2}{\parallel r-v_3^k-b_{v_3}^k\parallel }_2^2,$$

(21)

since the ${\ell }_2$ norm is differentiable, we have the following closed-form solver for r-sub-problem

$$\begin{array}{cc}\hfill & ({\alpha }_1{S}_r^{*}{S}_r+{\alpha }_2{S}_b^{*}{S}_b+{\alpha }_{3}I)r={\alpha }_2{S}_b^{*}(v_2^k+b_{v_2}^k)+{\alpha }_3(v_3^k+b_{v_3}^k)-{\alpha }_1{S}_r^{*}(v_1^k-b_{v_1}^k-S_{r}o),\hfill \end{array}$$

(22)

where I is the identity matrix. From [28], the Shearlets transform is constructed to be a Parseval frame, which implies that $S_r^{*}{S}_r=I$, then the r-sub-problem above can be solved by Fast Fourier Transform (FFT, $\mathcal{F}$) efficiently.

$$r^{k+1}={\mathcal{F}}^{-1}\left(\frac{{\alpha }_1\mathcal{F}\left(S_r^{*}{S}_r\right)+{\alpha }_2\mathcal{F}\left(S_b^{*}{S}_b\right)+{\alpha }_3\mathcal{F}\left(I\right)}{{\alpha }_2\mathcal{F}\left(S_b^{*}(v_2^k+b_{v_2}^k)\right)+{\alpha }_3\mathcal{F}(v_3^k+b_{v_3}^k)-{\alpha }_1\mathcal{F}\left(S_r^{*}(v_1^k-b_{v_1}^k-S_{r}o)\right)}\right),$$

(23)

where ${\mathcal{F}}^{-1}$ represents the Fast Inverse Fourier Transform.

In the end, the constrains for $r\ge 0$ and $b\ge 0$ can be projected by the following formula:

$$\begin{array}{cc}\hfill r^{k+1}& =min(o,max(r^{k+1},0));\hfill \\ \hfill b^{k+1}& =min(o,max(o-r^{k+1},0));\hfill \end{array}$$

(24)

Having explained the main idea of our method, the final resulting algorithm is displayed in Algorithm 1.

Algorithm 1: The split Bregman algorithm for proposed model (13)

5. Numerical Experiments

In this section, some comparison tests are performed to validate the effectiveness of the proposed method. We compare the proposed method with three recent state-of-art rain removal methods, including a dictionary learning-based algorithm in [11] (15ICCV), a minimization model with the learned rain layer prior method in [13] (16CVPR), and a directional global sparse model in [21] (18UTV). All tests are implemented in a laptop, CPU: Inter Core i5, memory: 16 GB, OS: Windows 10, testing software: Matlab R2016a.

Since humans are more sensitive to changes in luminance, all the RGB testing images are converted to YUV space. The proposed method is utilized to remove rain streaks only in the luminance channel (three compared methods are also implemented on the luminance channel). To be more persuasive, we use two kinds of data for experiments, the synthetic data, and real data. For the synthetic data, the relative error (RelErr), peak signal–noise ration (PSNR), and the structural similarity (SSIM) [33] are used to estimate the performance of different methods.

• The relative error (RelErr) is defined as

  $$\mathrm{RelErr}=\frac{\parallel u-u^r{\parallel }_F^2}{{\parallel u\parallel }_F^2},$$

  (25)

  where u and $u^r$ are the original signal and the recovered signal, and ${\parallel ·\parallel }_F$ represents the Frobenius norm.
• The peak signal–noise ration (PSNR) is defined as

  $$\mathrm{PSNR}=10log\frac{MN{u}_{max}^2}{\mathrm{MSE}},$$

  (26)

  where $u_{max}$ is the maximum value of u, and

  $$\mathrm{MSE}=\frac{1}{MN}\sum _{i=1}^M\sum _{j=1}^N{\parallel u(i,j)-u^r(i,j)\parallel }_F^2,$$

  (27)

  where M, N are the size of the signal u.
• The structural similarity (SSIM) is defined as

  $$\mathrm{SSIM}=\frac{(2{\mu }_u{\mu }_{u^r}+c_1)(2{\sigma }_{u{u}^r}+c_2)}{({\mu }_u^2+{\mu }_{u^r}^2+c_1)({\sigma }_u^2+{\sigma }_{u^r}^2+c_2)},$$

  (28)

  where ${\mu }_u$ and ${\mu }_{u^r}$ represent the average of u and $u^r$, ${\sigma }_u$, and ${\sigma }_{u^r}$ represent the standard deviation of u and $u^r$, ${\sigma }_{u{u}^r}$ is the covariance of u, and $u^r$, $c_1,c_2>0$.

5.1. Comparison Tests of the Synthetic Data

The testing results and discussions for synthetic data are given in this subsection. First, the synthetic rainy image is generated as follows: (1) adding the salt and pepper noise with a random density $den$ (which decides the density of rain streaks) to a zero matrix; (2) convoluting (1) with a motion kernel (including the directional parameter $\theta$ and motion parameter $len$); (3) adding (2) to the rain-free images to obtain the rainy image [14]. The parameters of synthetic rainy image for testing are given in

Table 1. The parameters list.

Parameters	$\mathit{den}$	$\mathit{len}$	$\mathit{\theta }°$
1	[0.02, 0.04, 0.06, 0.08]	10	30
2	0.04	[10, 20, 30, 40]	15
3	0.04	10	[−15, −5, 0, 5, 15]

. Then these RGB rainy images are converted to YUV channels, and only the rain streaks in the Y channel are removed using the 15ICCV method, the 16CVPR method, the 18UTV method, and the proposed method. In this test, we choose 5 images as the testing images, which can be downloaded from the dataset “UCID” (http://homepages.lboro.ac.uk/cgs/datasets/ucid/ucid.html) and (https://pixabay.com/).

The testing image are given in Figure 4. The visual results for 4 methods are displayed in Figure 5. The first column displays the de-rain results by the 15ICCV method, the second column shows the de-rain results by the 16CVPR method, the third column gives the de-rain results by the 18UTV method, and the last column shows the de-rain results by the proposed method.

From Figure 5 and the corresponding zooming in images in Figure 6, we find that the method 15ICCV fails to remove the rain streak completely, which leads to the lower SSIM value, and while methods 16CVPR and 18UTV perform better than 15ICCV, they cannot outperform our method. Comparing with three methods, the proposed method removes rain streaks and preserves the details of the background layer more efficiently. Then we have the conclusion as follows:

• the method 15ICCV cannot remove rain streaks completely, especially with heavy rain;
• the method 16CVPR does remove rain streaks completely, but the resulted background seems to be over-smooth;
• the method 18UTV can remove rain streaks completely, but rain streaks are detected by TV, which leads to a non-smooth background, especially in the heavy rain case;
• the proposed method using multilevel system performs better both in rain streaks removal and details preservation in heavy and light rain cases.

The quantitative results for different methods with different rain streaks’ direction are presented in

Table 2. Results comparison on synthetic data.

	Rainy Type	Heavy							Light
		Background			Rainy Streaks			Times(s)	Background			Rainy Streaks			Times(s)
Image	Methods	PSNR	SSIM	RelErr	PSNR	SSIM	RelErr		PSNR	SSIM	RelErr	PSNR	SSIM	RelErr
Pic1	15ICCV	28.79	0.8673	11.832	27.51	0.4831	15.642	71.54	29.84	0.9002	10.273	27.16	0.495	11.416	78.32
	16CVPR	30.49	0.8892	7.682	30.67	0.6417	7.394	948.27	30.76	0.9123	10.791	30.52	0.533	10.376	941.96
	18UTV	30.83	0.9027	7.591	30.91	0.6793	7.427	0.87	32.87	0.9345	5.964	32.79	0.6497	5.873	0.68
	Ours	31.62	0.9115	8.244	30.76	0.6954	7.082	2.39	33.28	0.9452	5.417	33.19	0.6719	5.314	2.26
Pic2	15ICCV	27.12	0.8117	12.027	27.01	0.4193	12.314	73.94	28.96	0.8936	10.217	28.74	0.4019	10.127	82.37
	16CVPR	28.54	0.8629	9.264	28.31	0.5763	10.028	926.85	29.94	0.9187	9.026	29.78	0.4693	10.201	931.57
	18UTV	30.17	0.9056	8.3146	30.09	0.6493	8.217	0.94	32.69	0.9221	7.424	31.79	0.6019	7.619	0.79
	Ours	31.68	0.9195	8.027	30.59	0.6952	8.335	4.08	32.94	0.9207	7.506	32.84	0.6394	6.701	3.15
Pic3	15ICCV	27.64	0.8327	10.837	27.28	0.4237	10.316	98.69	30.96	0.9017	7.462	29.85	0.5173	7.438	96.54
	16CVPR	28.94	0.8725	8.497	29.03	0.4887	9.287	837.29	32.15	0.9102	7.4179	32.36	0.5782	7.419	864.61
	18UTV	30.42	0.9013	7.694	30.25	0.5473	7.415	1.29	32.83	0.9274	7.018	32.59	0.5995	6.131	1.25
	Ours	31.52	0.9113	8.917	30.83	0.591	7.4953	4.76	33.65	0.9431	6.294	33.46	0.6554	5.831	4.93
Pic4	15ICCV	26.84	0.7916	13.94	26.95	0.4017	14.92	80.64	29.07	0.8114	9.148	28.96	0.4771	10.167	82.17
	16CVPR	28.41	0.8792	11.64	28.44	0.4657	10.13	1147.09	32.18	0.9104	8.365	32.06	0.4993	9.725	1129.68
	18UTV	32.17	0.9208	8.61	32.07	0.6129	8.486	1.17	32.36	0.9502	7.628	31.59	0.6107	8.847	1.89
	Ours	31.88	0.9159	8.05	32.16	0.6283	7.971	3.87	32.87	0.9221	7.391	32.72	0.6194	7.872	3.84
Pic5	15ICCV	29.76	0.8339	10.017	30.01	0.4996	9.172	44.93	32.69	0.9017	7.286	32.54	0.5162	7.836	31.26
	16CVPR	31.94	0.8992	8.274	31.79	0.5194	7.139	428.16	33.94	0.9317	6.264	33.72	0.5997	6.938	441.27
	18UTV	33.26	0.9331	6.192	33.86	0.5917	6.985	0.61	34.92	0.9427	5.947	34.94	0.627	5.846	0.57
	Ours	31.86	0.9141	7.214	30.94	0.5711	8.718	2.95	32.09	0.9197	7.172	30.52	0.6844	9.873	3.17

, including PSNR, SSIM, and RelErr values for both the background layer and the rain streak layer in heavy and light rain cases.

In addition, we randomly choose 20 images from UCID dataset to further test the rain removal performance in different directional rainy images. The quantitative results of four different methods in terms of RelErr, PSNR, and SSIM value are shown in Figure 7.

5.2. Results and Discussion for the Real Data

This subsection tests the performance of different methods under real rainy images. In Figure 8, the real rainy images and their local enlarged view are displayed. The testing images can be downloaded from the website.

The visual results recovered by four methods are displayed in Figure 9, the first column shows de-rain results by the 15ICCV method, the second column displays the de-rain results by the 16CVPR method, the third column gives the de-rain results by the 18UTV method, and the last row shows the de-rain results by the proposed method.

From Figure 9 and the corresponding local enlarged view in Figure 10, we can get similar results as the synthetic data. The method 15ICCV still has rain streaks remaining in the background layer, methods 16CVPR and 18UTV do significantly remove the rain streaks, but fail to detect directional singularities, while the proposed method performs well both in rain removal and detail preservation of the rain-free layer. Two test results suggest that the proposed method can remove the directional rain streaks efficiently.

6. Some Discussion for the Proposed Method

6.1. Computation Complexity

The computation complexity of our algorithm includes two Shearlets transforms computation and four sub-problems with closed-form solutions. For two Shearlets transforms, its computation complexity is about $O(NlogN)+O\left(N\right)$. The computation complexity of four sub-problems is $O\left(N\right)$, thus the total complexity for Algorithm 1 is $O(NlogN)+O\left(N\right)$, where N represents the total number of pixels. Table 2 shows the computation time comparison of four methods. For ’Pic1’ with size $[559\times 314\times 3]$, the computing time of the proposed method is 2.39 s, while the 15ICCV and 16CVPR methods take 72 s and 946 s respectively, and the 18UTV method only takes 0.87 s (without direction rotation). Our explanation is that the computation of Shearlets transform needs more computing time than UTV. Thus, we should strike a balance between the algorithm efficiency and computation complexity.

6.2. Description for Parameters

The proposed method includes two kinds of parameters, the model parameter and the algorithm parameter. For the model parameter, ${\gamma }_1,{\gamma }_2,{\gamma }_3$ are used to constrain the sparsity of Shearlets decomposition coefficients in high frequency, thus the recovery is sensitive to their changes. But different testing images with directional rain streaks are not sensitive to the parameters ${\gamma }_i,{\alpha }_i,i=1,2,3$. In the testing, we set ${\gamma }_1={\gamma }_2=1e-3$, ${\gamma }_3=1e-2$, ${\alpha }_i=8$.

For the algorithm parameter $tol,M_{iter}$, they are chosen to be $tol=1e-3$ and $M_{iter}=250$. (for some images, parameters are chosen different to obtain the best result). The default setting for the 15ICCV, 16CVPR, and 18UTV methods parameters can be found in [21]. We give the average quantitative performance of 20 images of UCID in

Table 3. Average quantitative performance of 20 images from UCID with different types of rain streaks.

Rain Type	Methods	Background			Rainy Streaks			Time(s)
		PSNR	SSIM	RMSE	PSNR	SSIM	RMSE
$den=0.04$	15ICCV	28.74 ± 2.49	0.8293 ± 0.0513	11.57 6 ± 2.895	26.19 ± 3.42	0.4702 ± 0.0527	13.042 ± 3.521	144.27
$len=40$	16CVPR	29.64 ± 2.01	0.8846 ± 0.0312	9.142 ± 1.259	27.41 ± 1.39	0.5327 ± 0.0932	9.741 ± 1.692	1152.64
$theta=0$	18UTV	30.74 ± 0.63	0.8811 ± 0.0737	9.747 ± 0.994	27.94 ± 0.53	0.6017 ± 0.0574	9.226 ± 1.726	1.59
	Ours	31.02 ± 1.722	0.8842 ± 0.0397	8.117 ± 0.692	28.52 ± 0.63	0.6064 ± 0.0827	8.017 ± 0.923	3.94
$den=0.08$	15ICCV	26.39 ± 1.64	0.7923 ± 0.6249	12.971 ± 2.063	27.73 ± 1.36	0.4713 ± 0.0539	12.973 ± 2.918	119.57
$len=40$	16CVPR	28.94 ± 1.92	0.8709 ± 0.0319	10.559 ± 1.947	28.59 ± 2.74	0.5904 ± 0.1752	9.172 ± 2.072	1397.46
$theta=-15$	18UTV	29.59 ± 2.17	0.8906 ± 0.0216	9.311 ± 1.509	28.74 ± 1.915	0.6607 ± 0.0793	10.703 ± 1.448	1.97
	Ours	30.16 ± 1.43	0.9042 ± 0.0161	9.942 ± 1.772	28.94 ± 2.73	0.692 ± 0.1154	9.954 ± 2.142	4.57
$den=0.04$	15ICCV	28.74 ± 1.89	0.8841 ± 0.0719	11.409 ± 1.173	28.97 ± 1.83	0.5516 ± 0.0967	12.712 ± 1.837	86.53
$len=40$	16CVPR	30.55 ± 1.93	0.892 ± 0.0517	10.929 ± 1.846	30.59 ± 1.73	0.6012 ± 0.0953	10.397 ± 2.066	1493.27
$theta=15$	18UTV	31.27 ± 1.53	0.9004 ± 0.0439	7.492 ± 0.793	31.37 ± 1.74	0.6112 ± 0.0571	10.973 ± 0.722	1.38
	Ours	31.42 ± 1.71	0.9087 ± 0.0164	7.928 ± 0.271	32.17 ± 0.54	0.6292 ± 0.0803	9.364 ± 0.574	4.61

.

6.3. Simple Discussion of Regularization Terms

In order to understand the effect of three sparse terms, some tests are implemented to reveal their contribution for rain removal. We choose real rainy image “gentlemen” as the testing image (see Figure 11a). The visual results after discarding sparse terms $Re{g}_1,Re{g}_2,Re{g}_3$ are listed in Figure 11c–e. From the recovery, we find that the regularizers $Re{g}_1,Re{g}_2$ have a remarkable effect on the rain removal, since $Re{g}_1,Re{g}_2$ describe the sparsity of the Shearlets transform of rainy images in the rain drops’ direction and the perpendicular direction respectively.

6.4. Convergency of the Proposed Algorithm

In Figure 12, we display the convergency curve of the proposed algorithm for the testing image ‘gentleman’. It is easy to find that the RelErr value decreases as the iteration number increases, while the PSNR value and SSIM value increase with the iteration number increasing, which suggests that the proposed algorithm is stable.

7. Conclusions

This work proposed an efficient convex rain removal model for single images, which is based on the sparse prior of rainy streaks and the background layer. The split Bregman algorithm is utilized to solve our model, which ensures the global optimal. We test the synthetic and real data to demonstrate that the proposed method performs better both in rain removal and details preservation of background layer than the comparable three methods.