A Hybrid Content-Aware Network for Single Image Deraining

Abstract

Rain streaks degrade the quality of optical images and seriously affect the effectiveness of subsequent vision-based algorithms. Although the applications of a convolutional neural network (CNN) and self-attention mechanism (SA) in single image deraining have shown great success, there are still unresolved issues regarding the deraining performance and the large computational load. The work in this paper fully coordinates and utilizes the advantages between CNN and SA and proposes a hybrid content-aware deraining network (CAD) to reduce complexity and generate high-quality results. Specifically, we construct the CADBlock, including the content-aware convolution and attention mixer module (CAMM) and the multi-scale double-gated feed-forward module (MDFM). In CAMM, the attention mechanism is used for intricate windows to generate abundant features and simple convolution is used for plain windows to reduce computational costs. In MDFM, multi-scale spatial features are double-gated fused to preserve local detail features and enhance image restoration capabilities. Furthermore, a four-token contextual attention module (FTCA) is introduced to explore the content information among neighbor keys to improve the representation ability. Both qualitative and quantitative validations on synthetic and real-world rain images demonstrate that the proposed CAD can achieve a competitive deraining performance.

1. Introduction

Due to the influence of inclement weather, images or videos often suffer from unfavorable visibility or degraded quality of the captured content. Images or videos captured under complicated rain weather environments usually affect the subsequent vision tasks (such as target detection [1,2], segmentation [3], and so on), reducing the image value significantly. Therefore, it is important preliminary work obtains clear images by removing rain streaks. The temporal information contained in videos can help to better identify and remove streaks; however, single-image deraining suffers from greater challenges and difficulties due to the lack of temporal information. Thus, it is of great interest to propose effective approaches to recover high-quality rain-free content from single images.

Currently, most studies consider that the rain image can be represented with the following equation:

$$\mathrm{O}= \mathrm{B} + \mathrm{R},$$

(1)

where O denotes the rain image, B denotes the clean background, and R denotes the rain streak layer. Image deraining aims to recover the clean background B from rain image O.

The rain removal methods can be divided into two categories: model-driven methods and data-driven methods. The model-drive methods are mainly based on dictionary learning [4], low-rank representation [5], Gaussian mixture [6], and other models. These models can only handle the rain streaks with specific physical characteristics and typically require many iterative optimization steps to determine the optimal solution, which limits their practical applications. With the rapid development of computer performance and deep learning frameworks, data-driven methods have gradually become the mainstream of rain removal [7]. Thanks to the powerful representation ability of convolutional neural networks, deep learning-based deraining methods have achieved remarkable performance. For example, Fu et al. [8] completed the rain streak removal task by introducing the end-to-end residual CNN. H. Z. et al. [9] proposed a multi-scale density-aware deraining network that automatically determines the rain-density information. Y. Z. et al. [10] separated rain streaks from the background image in a coarse-to-fine fashion by exploiting a residual multiscale pyramid architecture. Y. C. et al. [11] presented a direction-aware network endowing the discriminative representation ability of different rain streaks from the line pattern image edges for rain streak modeling. Yao et al. [12] use a filter to decompose the image into high and low frequency parts and then use CNN to remove the high-frequency part to achieve rain removal results. However, the intrinsic characteristics of convolutional operation, i.e., local receptive fields and independence of input content, hinder the model’s capacity to eliminate long-range rain degradation perturbation. Subsequently, self-attention mechanisms are proposed as a substitute to resolve the above-mentioned shortcomings.

A transformer is a typical self-attention model, which can process features over long distances due to its global computational properties and alleviates the limitations of convolution operations precisely in the field of computer vision. However, self-attention also has non-negligible disadvantages, such as a quadratic relationship between computation and image size, which limits its application environment seriously. Therefore, many efforts have been made to reduce the computational burdens [13,14].

As is known to all, the rain image is composed of rain streak layers and a clean background. To the best of our knowledge, different contents vary in complexity in dealing with deraining. For example, a flat area (e.g., sky, land) is naturally easier to process than textures, so different regions in the background own different complexities. Inspired by this, we infer that rain images can be divided into two categories: the harder regions and the simpler regions. The smaller modules are used to process the simpler regions and the bigger modules are used to treat the more complex image regions in this paper to achieve a balance between deraining performance and model complexity.

Above all, according to the drawbacks and complementarity of CNN and self-attention and the opinion that different regions apply to different networks, a hybrid content-aware deraining network (CAD) is proposed to deal with the single image deraining issue. The core of CAD is a series of CADBlocks, which include two components: the content-aware convolution and attention mixer module (CAMM) and the multi-scale double-gated feed-forward module (MDFM). In practice, CAMM is the organic combination of convolution and attention, which applies the attention mechanism for intricate windows and simple convolution for plain windows to generate less computational costs. After CAMM, MDFM blends multi-scale spatial features in a double-gated manner to ensure the network preserves local detail features and enhances image restoration capabilities.

Furthermore, since the rain streak layer and the background have a high interlacement and the rain streaks seriously destroy the content, a rain streak detection module (RSDM), which utilizes a four-token contextual attention module (FTCA) and a 3 × 3 convolution, is developed to process the global information to extract the rain streaks. The attention map generated by RSDM is fed into the subsequent feature map to encourage the network to pay more attention to the rain areas. Specifically, the supervisory information of RSDM used in training is generated by comparing the rain image with the clean background.

The main contributions made in this paper can be summarized as follows:

(1)

A hybrid content-aware deraining (CAD) network is proposed to generate high-quality deraining results, which incorporates a content-aware convolution and attention mixer module (CAMM) to apply simple convolution for plain areas and attention mechanism for complex areas and a deraining multi-scale double-gated feed-forward module (MDFM) to preserve local detail features and enhance image restoration capabilities.

(2)

An attention model named four-token contextual attention (FTCA) is designed to explore the rich context information among neighbor keys. The proposed FTCA applies four tokens, rather than the usual three tokens in transformers, to process the input features and exploit the introduced token to alleviate computational costs and enhance the expressiveness by aggregating global information.

(3)

Qualitative and quantitative evaluations both on synthetic and real-world datasets have verified that the proposed method not only exhibits excellent superiority and effectiveness in image deraining compared to other methods, but also maintains higher operational efficiency.

The rest of the paper is organized as follows: Section 2 introduces the related works on single image deraining, accelerating and lightweight framework. Section 3 elaborates on the proposed method. Section 4 presents the experiments and results of the synthetic and real-world datasets. Finally, the conclusion and discussion are given in Section 5.

2. Related Works

2.1. Single Image Deraining

Removing rain streaks from a single image has been a highly serious problem. Most traditional methods tend to explore handcrafted prior knowledge of images by determining the rain-dominant regions [15] or decomposing the rain images into high-frequency and low-frequency layers [16] to remove rain streaks. CNN-based methods [17,18] achieve better results by exploiting rain streaks direction [19] and density [9]. Subsequently, strategies such as residual connection [10], multi-scale features [20], and progressive fusion [21] are introduced to enhance the ability to restore clear images. Following this, the application of transformers [22,23,24] has made some headway in image deraining tasks and performed better than the CNN-based models. Recently, methods such as improving convolutional feed-forward network (FFN) [22,25] or adding CNN elements have been applied to enhance transformer-based approaches. However, these models struggle to strike a balance between efficiency and effectiveness. In contrast, a hybrid content-aware network proposed in this work can achieve a balance of deraining performance and computational efficiency.

2.2. Accelerating and Lightweight Framework

Accelerating framework. The complexity of deep learning models continuously increases to achieve more satisfying visuals in the low-level vision tasks, resulting in the practical application of these models being harder than before. Recently, content-aware routing has been proposed to deal with cropped patches by using different models that own different complexities to accelerate the models’ calculation rates in visual tasks. Kong et al. [26] divided the input into three categories, i.e., complex, medium, simple, and different categories are processed by corresponding networks.

Lightweight framework. The computational complexity of CNN [27] has also increased with the gradual expansion of model size and numerous strategies have been applied to reduce the complexity for a lightweight inference. For example, an efficient information fusion structure [28,29] is proposed to reduce both parameters and calculations and a simplified information distillation procedure [30,31] is introduced to obtain real-time inference on mobile devices. Moreover, varied powerful token mixers are also introduced for being lightweight. For example, both window-based SA [32] and large kernel convolution [33] achieved SOTA performance. Inspired by these categories, the content-aware routing and token mixer are integrated to design the proposed CAD, which applies complex operators for informative areas but simple operators for plain areas.

3. Method

3.1. Backbone Pipeline

Given a rain image $O\in {ℝ}^{H\times W\times 3}$, the shallow feature $X_S\in {ℝ}^{H\times W\times C}$ is first extracted by a 3 × 3 convolution, while the attention map $X_{att}\in {ℝ}^{H\times W\times 1}$ is simultaneously computed through RSDM. Subsequently, the feature $X_S$ is passed through a feature refinement block (FRB) and a series of CAD groups (CADGs) for deep feature extraction. Each CAD group consists of a stack of varying numbers of CAD blocks (CADBs). In CAD block, the CAMM is employed for local window modeling, only involving convolution and attention mechanisms. Then, the MDFM is utilized to integrate different ranges and scale information to further bolster the features for local details. Moreover, the attention map $X_{att}$ is fed to multiply the output of CAD block to enhance the rain regions. Skip connections [34] are also employed to fuse the shallow feature $X_S$ and the deep feature $X_D$. Finally, a 3 × 3 convolution and pixel-shuffle operations [35] are used to process the fused features to generate the finally reconstructed deraining image $Y\in {ℝ}^{H\times W\times 3}$. The overall workflow of the content-aware deraining (CAD) framework is illustrated in Figure 1.

3.2. Four-Token Contextual Attention Module

Conventional self-attention calculates each query-key pair, which not only results in excessive computational costs but also ignores the rich context information among neighbor keys. A new attention model named four-token contextual attention (FTCA) is proposed to alleviate the above problems.

Different from the original transformer, an additional token $A$, which can be seen as a key or query at different places, is introduced in FTCA to solve the first problem. As illustrated in Figure 2, token A is first utilized to perform calculations $M_1$ between $K$ and A as query and simultaneously perform calculations $M_2$ between $Q$ and A as key. Subsequently, the attention matrix $M\in {ℝ}^{HW\times nn},$ instead of the regular attention map of size ${ℝ}^{HW\times HW},$ is achieved by connecting the $M_1$ and $M_2$ by one 3 × 3 convolution. Finally, the feature map $Y$ is obtained by calculating the attention matrix $M$ and the pooling value $V$. For the second problem, the $K$ is calculated by employing group convolution to connect all neighbor keys within the grid, and then a skip connection is used to connect $K$ and the feature map $Y$. Given the input $X\in {ℝ}^{HW\times C}$, the proposed FTCA can be written as:

$$\begin{gathered} Q={\mathrm{Conv}}_{1\times 1}\left(X\right)\in {ℝ}^{HW\times C},K={\mathrm{Conv}}_{\mathrm{k}\times \mathrm{k}}\left(X\right)\in {ℝ}^{HW\times C}, \\ V=\mathrm{Pool}\left({\mathrm{Conv}}_{1\times 1}\left(X\right)\right)\in {ℝ}^{nn\times C},A=\mathrm{Pool}\left({\mathrm{Conv}}_{1\times 1}\left(X\right)\right)\in {ℝ}^{nn\times C}, \\ {M}_1=\mathrm{Softmax}\left( A{K}^T\right)\in {ℝ}^{nn\times HW},M_2=\mathrm{Softmax}\left(Q{A}^T\right)\in {ℝ}^{HW\times nn}, \\ M={\mathrm{Conv}}_{3\times 3}\left(\mathrm{cat}\left(\left[M_1,M_2^T\right]\right)\right)\in {ℝ}^{HW\times nn}, \\ Y={\left(M{V}^T\right)}^T+K \in {ℝ}^{HW\times C}, \end{gathered}$$

(2)

where $n$ is the pooling size and ${\mathrm{Conv}}_{\mathrm{k}\times \mathrm{k}}$ represents k×k group convolution. It should be noted that the rain streak detection module (RSDM) applies FTCA to explore the rich context information among neighbor keys, followed by a 3 × 3 convolution operation to obtain a rain streak attention map, promoting the subsequent modules to focus on the rain areas consciously.

3.3. Feature Refinement Block

The shallow feature $X_S\in {ℝ}^{H\times W\times C}$ is extracted by a 3 × 3 convolution; however, the simple convolution operation may not be able to extract information effectively from rain images and result in the loss of crucial details. Therefore, a feature refinement block (FRB), which can extract features in both spatial and channel dimensions simultaneously, is introduced to better learn the structural and textural details representation of rain images. As illustrated in Figure 3, the input $X_S$ is first processed by a 3 × 3 convolution and an average pooling to achieve channel and spatial processing, respectively. Then, the channel output $X_1$ and the spatial output $X_2$ are both multiplied by the input $X_S$ and added together to obtain the fused feature output $X_3$. Finally, skip connection is employed to connect the processed output $X_3$ and the input $X_S$ to reserve abundant crucial details. Given the input $X_S$, the detailed calculation process of FRB can be presented as:

$$\begin{gathered} X_1=\sigma \left(\mathrm{Conv}\left(\mathrm{ф}\left(\mathrm{Conv}\left(\mathrm{GAP}\left(X_S\right)\right)\right)\right)\right), \\ {X}_2=\sigma \left(\mathrm{Conv}\left(\mathrm{ф}\left(\mathrm{Conv}\left(\mathrm{Conv}\left(X_S\right)\right)\right)\right)\right), \\ {X}_3=X_S\odot X_1+X_S\odot X_2, \\ {X}_{FRB}=\mathrm{ф}\left(\mathrm{Conv}\left(X_3\right)\right)+\mathrm{Conv}\left(X_S\right), \end{gathered}$$

(3)

where $X_{FRB}$ refers to the output feature and $\mathrm{GAP}\left(·\right)$ denotes the global average pooling operation; $\odot$ is element-wise multiplication;$\sigma \left(·\right)$ and $\mathrm{ф}\left(·\right)$ denote the Sigmoid and ReLU activation functions, respectively. The mathematical notation nomenclature is listed in

Table A1. This is a mathematical notation nomenclature table.

Math Notations	Meaning
$\odot$	Element-wise multiplication
$\sigma \left(·\right)$	Sigmoid activation function
$\mathrm{ф}\left(·\right)$	ReLU activation function
$\varnothing \left(·\right)$	GELU non-linearity function

of Appendix A.

3.4. Content-Aware Deraining Block

Similar to the paradigm of the transformer block [14], the CAD block integrates the content-aware routing to token mixers, which use the attention mechanism for intricate areas and simple convolution for plain areas, followed by a multi-scale double-gated feed-forward module (MDFM) to enhance the features for local details. Specifically, the CAD block employs content-aware convolution and attention mixer module (CAMM) to achieve local window modeling, the calculation cost is greatly reduced compared to the conventional self-attention mechanism. In addition, MDFM is introduced for local feature processing, where the gating mechanism and multi-scale view are combined to enhance the local processing capabilities. The process of CAD block is depicted in Figure 4. Given the feature $X_i^{j-1}\in {ℝ}^{H\times W\times C}$ as the input of the j-th CAD block, the procedures can be expressed as:

$$\begin{gathered} X_i^{j-{\displaystyle \frac{1}{2}}}=X_i^{j-1}+\mathrm{BN}\left(CAMM\left(X_i^{j-1}\right)\right), \\ {X}_i^j=X_i^{j-{\displaystyle \frac{1}{2}}}+\mathrm{BN}\left(MDFM\left(X_i^{j-{\displaystyle \frac{1}{2}}}\right)\right), \end{gathered}$$

(4)

where $\mathrm{BN}\left(·\right)$ denotes BatchNorm layer;$X_i^{j-{\displaystyle \frac{1}{2}}}$ and $X_i^j$ denotes the output of CAMM and MDFM, respectively. In addition, $i$ denotes the i-th CAD group in CAD blocks.

Content-aware convolution and attention mixer module (CAMM). As depicted in Figure 5, the proposed CAMM exploits a predictor [36] to generate a mixer mask, an offset map, and an attention map. The offset map is generated by convolution operation, which can reduce the parameters greatly compared to the fully connection operation in [36]. In addition, different from the attention map in [36], which only contains spatial information, the attention map in the proposed CAMM integrates spatial and channel information through split operation. In practice, we evenly divide the input into two groups based on dimensions, one group performing spatial operations and the other group performing channel operations, and then multiply the outputs of the two groups to obtain a simple attention map. Specifically, the sorting mask is arranged in descending order, the first K indices are set to represent areas with complex information, while the other indices are set to represent plain areas. For the plain areas, the attention map is used to implement the simple attention for lightweight operation. For the informative complex areas, the offset map is applied to modulate the key in FTCA to endow the module with the dynamic property, promoting the deformable window of FTCA to adaptively include more useful information to enhance the representational capacity. Finally, the plain areas and the informative complex areas are integrated by a 1 × 1 convolution and an FTCA with a fixed window. Given the input $X_i^{j-1}$, the CAMM is formulated as:

$$\begin{gathered} mask,offset,A_{map}=Predictor\left(X_i^{j-1}\right), \\ {X}_{plain }, X_{complex }={\mathrm{X}}_{\mathrm{i}}^{\mathrm{j}-1}\odot mask, \\ {X}_{plain}^{\prime }=X_{plain }\odot A_{map}, \\ {X}_{complex}^{\prime }=DFTCA\left(X_{complex},X_{complex}+offset\right), \\ {X}_i^{j-{\displaystyle \frac{1}{2}}}=\mathrm{Conv}\left(X_{plain}^{\prime }+X_{complex}^{\prime }\right), \\ {X}_i^{j-{\displaystyle \frac{1}{2}}}=\mathrm{F}FTCA\left(X_i^{j-{\displaystyle \frac{1}{2}}},X_i^{j-{\displaystyle \frac{1}{2}}}\right), \end{gathered}$$

(5)

where the first input of FTCA denotes the input of Q, V, A, and the second input denotes the input of K;$\odot$ is element-wise multiplication. Here, DFTCA denotes the FTCA with the deformable window and FFTCA denotes the FTCA with the fixed window.

Multi-scale double-gated feed-forward module (MDFM). The regular feed-forward network (FFN) can extract and fuse feature information from different locations by operating on each pixel location separately and identically. One 1 × 1 convolution is used to expand the input channel dimension and another 1 × 1 convolution is used to reduce the channel to the original input dimension. Previous studies typically leverage the standard FFN to bolster local context [25]. To focus on enriching features with contextual information, S. et al. added a gating mechanism and depth-wise convolutions into the standard FFN named gated-dconv feed-forward network (GDFN) [22]. Based on GDFN, Sun et al. [25] conceived a dual-scale gated feed-forward module, which uses a 5 × 5 convolution and a 3 × 3 convolution to extract the multi-scale information. Motivated by these, we propose a multi-scale double-gated feed-forward module (MDFM), which integrates different range and scale information into the transmission process by using two gating mechanisms and three convolutions that own different kernel sizes. The process of MDFM is depicted in Figure 6. Given the input $X_{in}\in {ℝ}^{H\times W\times C}$, the proposed MDFM is formulated as:

$$\begin{gathered} X_1,X_2,X_3=\mathrm{Split}\left({\mathrm{Conv}}_{1\times 1}\left(X_{in}\right)\right), \\ X_1,X_2,X_3={\mathrm{Conv}}_{3\times 3}\left(X_1\right),{\mathrm{Conv}}_{5\times 5}\left(X_2\right),{\mathrm{Conv}}_{7\times 7}\left(X_3\right), \\ {X}_{out}={\mathrm{Conv}}_{1\times 1}\left(\varnothing \left(\varnothing \left(X_1\right)\odot X_2\right)\odot X_3\right), \end{gathered}$$

(6)

where the $\mathrm{Split}\left(·\right)$ indicates the channel equalization operation, ${\mathrm{Conv}}_{\mathrm{N}\times \mathrm{N}}$ represents N×N depth-wise convolution; $\odot$ is element-wise multiplication; and $\varnothing \left(·\right)$ denotes the GELU non-linearity. Overall, the MDFM can process spatially neighboring pixel information by the depth-wise convolutions and acquire fine details complementary by the gating mechanism, which allows each level to focus on the other levels.

3.5. Loss Function

To better enhance the structure details, a hybrid loss function is applied to optimize the proposed CAD.

The MSE loss function is adopted to reduce the pixel-level difference between the output image of the proposed network and the corresponding background image, expressed as follows:

$$L_{mse}=E_x{\left|\left|C\left(x\right)-y\right|\right|}^2,$$

(7)

where x is the input image, y denotes the ground truth, and $C\left(·\right)$ is the result generated by the proposed CAD.

The SSIM loss function is adopted to assess the structure-level similarity difference between the two types of images. It can be expressed as:

$$L_{ssim}=1-\mathrm{SSIM}\left(C\left(x\right),y\right),$$

(8)

where $SSIM\left(·\right)$ denotes the structural similarity between two images. When two images are more similar, the value of the function $SSIM\left(·\right)$ becomes closer to 0. On the contrary, the closer its value is to 1.

The attention loss function is utilized to enhance the focus on the important rain streak areas, which can be formulated as:

$$L_{att}=E_x{\left|\left|RSDM\left(x\right)-M\left(x\right)\right|\right|}^2,$$

(9)

where $RSDM\left(·\right)$ is the attention map predicted by the RSDM and the $M\left(·\right)$ denotes a binary map of the rain streaks.

The perceptual loss function is used to minimize the feature distance between the input image and the target image to achieve better image quality that is more in line with the human eye’s perception. The function can be indicated as:

$$L_{per}=L_{mse}\left(V_i\left(C\left(x\right)\right),V_i\left(y\right)\right),$$

(10)

where $V_i\left(·\right)$ represents the feature map calculated by the i-th layer in the pretrained VGG-16 network.

The Laplace edge loss function is adopted to capture the texture information related to high-frequency structure to improve the images’ detail representation and it can be expressed as:

$$L_{edge}=L_{mse}\left(K\left(C\left(x\right)\right),K\left(y\right)\right),$$

(11)

where $K\left(·\right)$ denotes the Laplace operator convolution operation.

Above all, the overall loss function can be formulated as:

$$L_{loss}={\lambda }_1{L}_{mse}+{\lambda }_2{L}_{ssim}+{\lambda }_3{L}_{att}+{\lambda }_4{L}_{per}+{\lambda }_5{L}_{edge},$$

(12)

where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, ${\lambda }_4$, and ${\lambda }_5$ are used to balance different loss terms.

4. Experiments and Discussions

4.1. Datasets and Implementation Details

Datasets. Experiments on two frequently used synthetic rain datasets are conducted to effectively assess the proposed deraining method. Rain100L [37] and Rain100H [37] both contain 1800 pairs of images for training and 100 pairs of images for testing. The rain streaks in these two groups images are different in the aspects of density and direction. In addition, the real-world datasets provided by Zhang et al. [38] and Yang et al. [37] are also used to verify the efficiency of the proposed method in practical application.

Metrics. PSNR [39] and SSIM [40] are applied to evaluate the deraining results processed by the proposed method and other compared methods.

$$\mathrm{PSNR}=10·{\log }_{10}\left({\displaystyle \frac{MA{X}^2}{MSE}}\right),$$

(13)

$$\mathrm{SSIM}\left(x,y\right)={\displaystyle \frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}},$$

(14)

where $x$ and $y$ denote the predicted rain removal image and the corresponding ground truth, respectively; MSE denotes mean squared error and MAX denotes the maximum possible value of image pixel values; $\mu$ represents the average; $\sigma$ represents the variances; and $c_1$ and $c_2$ are different constants.

Implementation details. The number of CAD groups is set to 4, the number of CAD blocks that each CAD group owns to [4,4,6,6], and the initial channel number to 60. The window size of the attention model FTCA is set to 16. Furthermore, the Adam optimizer with the initial learning rate of 1 × 10⁻³ and decayed by multiplying 0.2 at every 30 epochs until the 150th epoch. The entire network of CAD is implemented in a PyTorch framework of 1.12.1 with an Intel i9 CPU and a 4090 GPU. The proposed CAD is compared with two prior-based methods (GMM [3] and DSC [6]), six CNN-based methods (PReNet [41], VRGNet [42], MPRNet [43], SAPNet [44], HINetp [45], and DRAN [11]), two transformer-based methods (IDT [12] and DCT [46]), and one hybrid-based method (ELF [47]).

4.2. Comparison with State-of-the-Art

Synthetic datasets.

Table 1. Comparison results of PSNR and SSIM on the Rain100L and Rain100H datasets.

Datasets		Rain100L	Rain100H	Overhead
Metrics		PSNR/SSIM	PSNR/SSIM	Params [M]	FloPs [G]	Time/s
Prior-based methods	GMM [4]	26.945/0.844	16.317/0.431	-	-	-
	DSC [6]	27.271/0.837	15.579/0.396	-	-	-
CNN-based methods	PReNet [41]	32.420/0.950	26.770/0.858	0.169	16.576	0.163
	VRGNet [42]	36.662/0.973	30.077/0.887	0.169	16.576	0.120
	MPRNet [43]	36.687/0.967	30.427/0.891	3.637	137.163	0.207
	SAPNet [44]	32.291/0.951	28.046/0.867	0.182	165.938	0.198
	HINet [45]	37.591/0.971	30.649/0.894	88.671	42.629	0.698
	DRAN [11]	37.830/0.980	30.650/0.900	-	-	-
Transformer-based methods	IDT [22]	37.010/0.970	29.950/0.898	16.390	58.440	0.164
	DCT [46]	38.190/0.970	30.740/0.890	-	-	-
Hybrid-based methods	ELF [47]	36.672/0.968	30.480/0.896	1.532	66.390	0.125
	CAD (our)	39.301/0.984	30.807/0.915	1.196	32.956	0.109

shows the quantitative comparison in the datasets of Rain100L and Rain100H. Due to the fact that the codes of DRAN [11] and DCT [46] are not open source, their Params and FloPs cannot be calculated, and we directly report the experimental results in their papers. As can be seen, the proposed CAD achieves the highest PSNR and SSIM, as well as the shortest inference time. For Rain100L, the proposed CAD surpasses the second-best method DCT by 1.111 dB in PSRN. For Rain100H, the proposed CAD surpasses the CNN-based method DRAN by 0.157 dB in PSRN. In contrast to the DCT, which is based on a transformer, although the proposed CAD has a small advantage in PSNR of only 0.067 dB, the SSIM is significantly higher than DCT in Rain100H. Compared with the hybrid method ELF, which simply combines CNN and a transformer to process the entire image, the proposed CAD utilizes a predictor formed by CNN operation to generate offset maps and attention maps; the former are used to provide deformable windows for the transformer, and the latter are applied to endow the convolution with dynamic characteristics for simple regions. These improvements in CAD make the deraining results in Rain100H superior by 0.399 dB in PSNR than ELF. In addition, the proposed model achieves obvious advantages in terms of Params and FloPs among the methods involving attention mechanisms. Limited by the structure of self-attention, the proposed CAD has higher Params and FloPs compared with some CNN-based methods, such as PReNet and VRGNet, but the PSNR has increased by 4.037 dB and 2.761 dB, respectively, and the rain removal effect has been greatly improved.

Furthermore, Figure 7 presents the visual comparison results and the corresponding SSIM and PSNR on two synthetic images. As can be seen, GMM and DSC fail to produce satisfactory rain removal images and the results contain many residual rain streaks. SAPNet and VRGNet remove most rain streaks but introduce unpleasant blurry artifacts. For example, there is a clear blur at the intersection of two mountains in the first image, and the sail ropes of the sailboat are ghosted and blurred in the second image. MPRNet and HINet cause excessive texture removal and detail loss. For instance, in the first image, the mountain is excessively smooth, while in the second image, the sail ropes of the sailboat have disappeared. In contrast, the proposed CAD can preserve more image detail and achieve better visual results.

Real-world datasets. Experiments on real-world rain datasets are further conducted to demonstrate the generalization ability of the proposed method. However, the deraining performance on these images can only be evaluated visually as the corresponding rain-free images are difficult to obtain. The deraining results of all the involved methods are shown in Figure 8. One can see that the proposed CAD can remove the rain streaks clearly and produce more credible content, whereas other competitive methods fail to eliminate the apparent rain streaks. Specifically, the background images generated by GMM are very blurry in both images. DSC cannot eliminate the rain streaks in the first image and the tree trunk in the second image exhibits significant artifacts. The backgrounds in the first image of SAPNet and VRGNet are unclear, and the results of MPRNet and HINet own heavy shadows. The results of SAPNet, VRGNet, MPRNet, and HINet in the second image are similar, i.e., there are some rain residual streaks and artificial marks on the tree trunk. The satisfactory visual deraining result of the proposed CAD proves that it has good generalization ability.

In addition, a survey on user satisfaction is conducted to further demonstrate the deraining effect of CAD. We selected eight images from the real-world deraining images and arranged them randomly. Then, 100 participants were asked to rank the images from 1 to 10 based on the quality of deraining, with 10 representing the best effect. The average scores of the seven methods are shown in

Table 2. Average scores of user satisfaction.

Methods	GMM	DSC	SAPNet	VRGNet	MPRNet	HINet	CAD
Scores	4.11	3.52	6.27	7.14	8.17	8.56	9.25

, it can be seen that the proposed CAD obtained the highest scores, confirming that the deraining results of CAD are more in line with human visual perception.

4.3. Ablation Study

In this section, ablation studies are conducted to validate the main components of the proposed method, as well as the effects of different window sizes and pooling sizes in FTCA. To this end, five variants of CAD are trained on Rain100H with the strategy described in Section 4.1. In addition, three different window sizes and four pool sizes in FTCA are explored.

Effectiveness of different blocks. Based on the proposed CAD, studies that remove different modules separately as the variable are conducted to validate the effectiveness of different blocks. Specifically, the multi-scale double-gated feed-forward module (MDFM), the feature refinement block (FRB), the four-token contextual attention (FTCA) in the rain streak detection module (RSDM), the DFTCA with deformable windows, the FFTCA with fixed windows, and the content-aware convolution and attention mixer module (CAMM) are thoroughly studied. It should be noted that a gated-dconv feed-forward network (GDFN) is applied to replace the MDFM and a conventional self-attention (SA) is used to substitute FTCA. As shown in

Table 3. Ablation study on different blocks on Rain100H.

Model	MDFM	FTCA	DFTCA	FFTCA	FRB	CNN	Attention	PSNR	SSIM
model (a)	×	√	√	√	√	×	×	30.203	0.907
model (b)	√	×	√	√	√	×	×	30.615	0.912
model (c)	√	√	×	√	√	×	×	30.703	0.913
model (d)	√	√	√	×	√	×	×	30.697	0.913
model (e)	√	√	√	√	×	×	×	30.562	0.910
model (f)	√	√	×	√	√	√	×	30.361	0.907
model (g)	√	√	×	√	√	×	√	30.062	0.906
CAD (ours)	√	√	√	√	√	×	×	30.807	0.915

, comparing the proposed CAD with the model (a), MDFM achieves 0.604 dB performance gain since three scales and two gated mechanisms in MDFM can help the model handle spatially local structures for effective improvement. From model (b), model (c), and model (d), the FTCA at different positions obtains 0.192 dB, 0.104 dB, and 0.11 dB in PSNR, which shows the superiority of FTCA. In contrast with model (e), the final model shows that the FRB provides a favorable gain of 0.245 dB since the FRB can extract more detailed rain streaks. The models (f) and (g) utilize convolution and self-attention to replace CAMM. It can be observed that model (f) results in a decrease of 0.446 dB in PSNR and model (g) shows a decrease of 0.745 dB in PSNR, demonstrating the effectiveness of CAMM in rain removal.

Effectiveness of window size in FTCA. In the CAD block, the attention mechanism based on a window is used to process the complex areas to reduce the computational costs.

Table 4. Ablation study on window size on Rain100H.

Window Size	Params [M]	FLOPs [G]	PSNR	SSIM
8 × 8	1.124	32.950	30.542	0.908
16 × 16	1.196	32.956	30.807	0.915
32 × 32	1.771	33.007	30.681	0.911

illustrates the influence of different window sizes on the deraining results. From Table 4, one can observe that the model with a window size 8 × 8 reduces only 0.006 G FLOPs but reduces the PSNR 0.265 dB more than the model with a window size 16 × 16. The model with a window size 16 × 16 not only advances 0.126 dB but also reduces 0.051 G FLOPs than the model with a window size 32 × 32. Therefore, we use a window size of 16 × 16 to achieve a balance between performance and computations.

Effectiveness of pool size in FTCA. The pool operation used to generate the token A and the value V in FTCA plays an important role in reducing parameters. Therefore, a reasonable pool size $n$ is very important to balance performance and complexity.

Table 5. Ablation study on pool size on Rain100H.

Pool Size	Params [M]	FloPs [G]	PSNR	SSIM
5	1.195	32.878	30.623	0.912
6	1.196	32.956	30.807	0.915
7	1.198	33.056	30.696	0.912
8	1.201	33.215	30.702	0.912

shows the results in the situations of different pool sizes. Specifically, the model with pool size 5 reduces 0.078 G FloPs more than pool size 6 while reducing 0.184 dB in PSNR. When the pool size is 6, we obtain better results than the size of 7, with a reduction of 0.1 G FloPs and an improvement of 0.111 dB in PSNR. The pool size 8 results in an increase of 0.259 G FloPS and a reduction of 0.105 dB in PSNR compared with pool size 6. Large pooling sizes can lead to excessively high FloPs, while small pooling sizes can affect the model’s performance. As a result, the pooling size of 6 supports the proposed CAD.

5. Conclusions

In this essay, an effective hybrid network based on the content of images has been proposed for image deraining tasks. By analyzing the drawbacks of CNN and self-attention and exploiting the opinion that different regions apply different networks, a content-aware convolution and attention mixer module (CAMM) is introduced, which uses an attention mechanism for intricate windows and a simple convolution for plain windows. To enhance image restoration capabilities, the multi-scale double-gated feed-forward module (MDFM) is proposed to guide the network for preserving local detail features better. Furthermore, a four-token contextual attention module (FTCA) is introduced to explore the rich context information among neighbor keys to enhance the representation ability. Extensive experiments on both synthetic and real-world datasets have demonstrated the superiority and effectiveness of the proposed CAD framework.

Furthermore, benefiting from its lightweight design and high efficiency, the proposed CAD can be applied to smartphone cameras, autonomous driving, drone detection, and other scenarios to achieve real-time rain removal and improve visual effects. In addition to rain, harsh weather conditions such as snow and fog can also greatly affect people’s visual perception. Therefore, we will conduct experiments in a future study to verify the effectiveness of the proposed CAD in snow and fog removal and make improvements to enhance the practical value of CAD in real-world applications.