Rain Rendering and Construction of Rain Vehicle Color-24 Dataset

Abstract

The fine identification of vehicle color can assist in criminal investigation or intelligent traffic management law enforcement. Since almost all vehicle-color datasets that are used to train models are collected in good weather, the existing vehicle-color recognition algorithms typically show poor performance for outdoor visual tasks. In this paper we construct a new $RainVehicleColor$-24 dataset by rain-image rendering using $PS$ technology and a $SyRaGAN$ algorithm based on the $VehicleColor$-24 dataset. The dataset contains a total of 40,300 rain images with 125 different rain patterns, which can be used to train deep neural networks for specific vehicle-color recognition tasks. Experiments show that the vehicle-color recognition algorithms trained on the new dataset $RainVehicleColor$-24 improve accuracy to around $72\%$ and $90\%$ on rainy and sunny days, respectively. The code is available at humingdi2005@github.com.

1. Introduction

With the development of computer vision technology and hardware, vision algorithms based on deep learning have achieved unprecedented performances, and are increasingly applied to practical scenarios. For example, color recognition is applied in vehicle tracking [1,2,3,4,5,6,7,8,9,10,11]. In order to train vehicle-color recognition algorithms, scholars have constructed multiple vehicle-color datasets. For example, the vehicle-color dataset by Chen et al. [1] included eight color categories, and each image contained one vehicle. The dataset collected by Jeong et al. [3] contains seven colors, and Tilakaratna et al. [12] expanded their dataset to 13 categories. Hu et al. [4] constructed a new benchmark vehicle-color dataset, $VehicleColor$-24, with 24 colors, and proposed a novel vision color recognition ($VCR$) method based on a Smooth Modulation Neural Network with Multi-Scale Feature Fusion ($SMNN$-$MSFF$). The $VehicleColor$-24 includes 10,091 images with a total of 31,232 vehicles, and each image contains up to nine vehicles. These datasets and algorithms have been very conducive to vehicle recognition tasks. However, the images of the above datasets are mostly collected in good weather. On the other hand, criminal investigation or intelligent traffic management law enforcement often encounter bad conditions, especially rainy weather [5].

The raindrops are likely to turn into rain streaks due to their high density and fast speed. Rain streaks will typically produce reflection or refraction, and often blur and deform the images captured by cameras, which poses challenges for subsequent visual tasks. Studying low- and high-level tasks has become a hot research direction [5]. Many scholars have paid attention to the joint processing of low- and high-level tasks. The generalization in object detection is improved by embedding domain adaptation, image restoration, style transfer, or other modules into object detection methods or few-shot learning mechanisms [6,7,8,9,13,14,15,16]. However, these works all require additional modules, which will undoubtedly increase the burden of outdoor equipment. To solve this problem, a natural solution is to construct image datasets with rich diversity and in various environments for specific tasks, which can be used to train models facing subsequent high-level tasks without adding modules, while still improving generalization. However, it is very expensive to collect such datasets in practice, so this paper constructs the $RainVehicleColor$-24 dataset by rain-image-rendering technology toward this end. It aims to address the specific vehicle-color recognition task.

There exists much literature on the construction of rain image datasets. For example, Garg and Nayar (2006) used a particle simulator to synthesize rain patterns, and then superimposed the rain patterns with clean backgrounds to synthesize the rain image [17,18]. Hu et al. [19] and Tremblay et al. [20] rendered rain images based on the complex fusion of background and rain layers. Data-driven synthetic rain images based on a generative adversarial network ($GAN$) have recently received increasing attention [20,21,22]. Wang et al. [23] constructed a large-scale real-rain image dataset, i.e., $SPA$-Data (spatial attentive data). However, rain vehicle-color image datasets are rare.

Inspired by the above works, this paper constructs a $RainVehicleColor$-24 dataset by rendering rain images using photoshop ($PS$) technology and the $SyRaGAN$ algorithm [24]. The $RainVehicleColor$-24 dataset has a total of 40,300 rain images with 125 kinds of rain-streak patterns, which is beneficial in improving the generalization of the deep neural network models for the fine identification of vehicle colors. Using the benchmark datasets, $Rain100L$ and $RainVehicleColor$-24, we trained the current state-of-the-art (SOTA) algorithm $PReNet$ network [25] and the lightly weighted $LDVS$ deraining network [26] to obtain the models $PReNet1$, $LDVS1$, $PReNet2$ and $LDVS2$. After testing on both synthetic and real data, these above models showed obvious advantages for the deraining task. Vehicle-color recognition methods trained on the new datasets showed improved performance for vehicle-color classifying in both sunny and rainy conditions.

The main contributions of this paper are as follows:

(1)

This paper constructs the $RainVehicleColor$-24 dataset by rain-image-rendering technology, in order to address the specific task of the vehicle-color fine recognition. Both model-based and data-driven-based rendering are used: the former synthesizes 300 images by $PS$ to form one subset in which clean background images are from the $VehicleColor$-24; the latter, i.e., the $SyRaGAN$ network, synthesizes 8000 vehicle images to form another subset in which clean background images are also from the $VehicleColor$-24;

(2)

This dataset helps to increase the performance of vehicle-color recognition methods on rainy days since the $RainVehicleColor$-24 dataset consists paired vehicle-color rain images with various rain patterns;

(3)

We improve the performances of existing algorithms for vehicle-color identification in rainy conditions. Vehicle-color identification plays a key role in intelligent traffic management and criminal investigation. However, existing algorithms are typically trained on the datasets collected in good weather conditions, which suffer from poor performance in poor weather conditions, such as rainy weather. In this paper, we show that our newly constructed dataset is critically beneficial to the performances of existing algorithms for vehicle-color identification in rainy conditions.

The rest of this paper is structured as follows: Section 2 reviews the related work; Section 3 introduces construction of the $RainVehicleColor$-24; Section 4 compares complex experimental results in detail; Section 5 concludes the paper.

2. Related Work

2.1. Photoshop ($PS$) Technology

At present, $PS$ is the main synthetic rain image technology. Garg and Nayar (2006) synthesized various types of rain patterns, and then directly added them to the corresponding clean background images to obtain paired rain images. There has been some work on the simple stacking of background and rain layers. For example, Li et al. [17] proposed a paired rain-image test set ($Rain12$) composed of one type of rain pattern and 12 background images to synthesize 12 rain images. Yang et al. [18] constructed a dataset ($Rain100H$) containing 1900 rain/clean image pairs, with 1800 image pairs for training and 100 image pairs for testing. These datasets are often used as comparative datasets; however, the types of rain streaks are relatively simple. There has also been research on the complex fusion of background and rain layers. For example, Li et al. [27] used $PS$ to add noise to form rain patterns of different intensities and directions, and then synthesized the rain images based on the screen blend model ($SBM$), producing the dataset $DDC$-$Data$. Wang et al. [28] constructed a dataset $QSMD$-$Data$ using synthetic rain images based on a screen-shrouded model ($SBM$).

The process of $PS$ technology is as follows (as shown in Figure 1):

Figure 1. Synthesis process of rain image using photoshop software [5].

(1)

First, the rain-streak patterns under the conditions of two light sources with different angles of illumination and different camera directions are constructed.

(2)

Then, the rain images are synthesized according to the raindrop-modeling equation:

$${\omega }_n={[n(n-1)(n+2)\sigma /\left(\rho r_0^3\right)]}^{1/2},$$

(1)

$$r(t,\theta ,\varphi )=r_0[1+A_{2,0}sin\left({\omega }_{2}t\right)P_{2,0}\left(\theta \right)+A_{3,1}sin\left({\omega }_{3}t\right)cos\left(\varphi \right)P_{3,1}\left(\theta \right)],$$

(2)

where r is the surface tension, $\rho$ is the density of water, $\theta$ is the angle, $\varphi$ is the azimuth, $r_0$ is the size of the raindrop, $A_{2,0}$ and $A_{3,1}$ are the amplitudes, and ${\omega }_n$ is the frequency, $P_{n,m}\left(\theta \right)$ is the Legendre function that describes the dependence of the shape on the angle $\theta$ for the mode (n,m). The parameters are usually set by empirical knowledge ($Garg$ and $Nayar$, 2006 [29]), and the rain pattern is synthesized by formulas $\left(1\right)$ and $\left(2\right)$.

(3)

Finally, the synthesized rain pattern is directly added to the clean background image to get the rain image.

2.2. Data-Driven Rain-Image-Rendering Technology

There are also semi-automatic methods to collect rain images. For example, Qian et al. [30] collected a paired raindrop image dataset. Jin et al. [31] constructed the rain image dataset, $RaidaR$, under a wide range of circumstances, using cameras on the roof. However, due to the high cost of data collection, in practice, real paired rain images are not sufficient to train models; therefore, most methods are trained with the help of rendering data. Wei et al. [21] synthesized a rain image dataset by an unsupervised learning mechanism with a constrained $CycleGAN$ network, resulting in a dataset $Rain200L$ with more types of rain-streak patterns and more realistic visual effects. Wang et al. [22] used a Bayesian model to construct a rain-image generation network to generate more than 120 rain-streak patterns. Further exploration of more reasonable and accurate rain-image synthesis models and the generation of more realistically rendered rain images are also important research directions. In addition, the construction of a specific vehicle rain-image dataset for specific vehicle target detection will be very much meaningful and badly needed.

2.3. Single-Image Rain-Removal Algorithm

Algorithms for single-image rain removal are mainly divided into two categories: traditional model-driven method and data-driven deep neural network [5]. Model-driven algorithms rely on the statistical analysis of rain streaks and background scenes, and use priors on rain streaks and background layers to build a rain-removing model to iteratively give explicit solutions. Chen et al. [32] constructed a generalized low-rank model, and they distinguished the low-rank rain pattern as a separate layer from the background layer. Lou et al. [33] proposed a highly discriminative sparse coding method to separate the rain pattern from the background layer.

On the other hand, data-driven deep neural network algorithm for rain removal was first proposed by Fu et al. [34]. The algorithm reconstructs a clean background layer after removing the rain pattern from the high-frequency layer. Li et al. [27] divided the rain-removal network into two stages: decomposition and combination. They then used the residuals of the synthetic and original rain images to train the network to improve the rain-removal performance. Ren et al. [25] proposed a simple and effective rain-removal network, i.e., the Rain-Removal Network Baseline Progressive Recurrent Network ($PReNet$). Since then, a variety of algorithms have been proposed, e.g., lightweight pyramid network ($LPNet$) algorithm [35], local binary pattern conditional generative adversarial network ($LBP$-$CGAN$) [36], lightweight single-image deraining algorithm incorporating visual saliency ($LDVS$) [26]

Since $PReNet$ is one of the simple and effective single-image rain-removal algorithms, and $LDVS$ is one of the lightweight algorithm, they will be trained on the $Rain100L$ and $RainVehicleColor$-24 benchmark rain datasets. Moreover, we combine the subsequent target recognition algorithms to examine whether the dataset $RainVehicleColor$-24 can improve the performance of the algorithm’s modeling of the low- and high-level joint task and the low-level rain-removal task. The details are given in Section 4.2 and Section 4.3.

2.4. Vehicle-Color Recognition Algorithms

Due to its practical significance, vehicle-color recognition has attracted much attention in computer vision. The literature mainly falls into two categories: manual feature-based methods and emerging data-driven deep learning methods [1,2,3,4,5,6,7,8,9,10,11]. Among others, Hu et al. [5] proposed a novel $VCR$ method based on a Smooth Modulation Neural Network with Multi-Scale Feature Fusion ($SMNN$-$MSFF$), which is then trained and evaluated on the dataset “$VehicleColor$-24” with 24 vehicle-color classes. $VehicleColor$-24 consists of $10,091$ vehicle images from a 100 h urban road surveillance video.

In this paper, we perform rain-image rendering using the $VehicleColor$-24 dataset to construct a specific task dataset for vehicle-color recognition. The objective is to improve the performance of the low- and high-level joint tasks in vehicle-color fine recognition or improve the generalization of vehicle-color fine recognition in bad weather.

3. Construction of $\mathit{Rain} \mathit{Vehicle} \mathit{Color}$-24

3.1. $VehicleColor$-24

Firstly, 8000 vehicle images are selected from the existing $VehicleColor$-24 as a clean background image with resolution 1747 × 982. The dataset consists of 10,091 vehicle images captured from urban road surveillance videos, with a total of 31,232 vehicles and 24 colors. The authors preprocessed the dataset, including lighting adjustment, dehazing, etc. Samples from $VehicleColor$-24 are shown in Figure 2.

Figure 2. Samples from $VehicleColor$-24.

3.2. Rendering by $PS$

We randomly selected 300 images from $VehicleColor$-24, and used $PS$ software to generate rain patterns of different directions, sizes, and thicknesses by adjusting parameters such as motion blur and color level. We then superimposed 120 kinds of rain streaks with 300 clean background images from the $VehicleColor$-24 dataset to construct a subset of $RainVehicleColor$-24. As a result, we have more types of rain patterns to ensure the diversity of rain images in the new dataset. The partial rain image samples of the rain image subset are shown in Figure 3.

Figure 3. Examples of subset by $PS$. The three images are rendered by $PS$ from simple, moderately complex, and complex scenes, respectively. We set the parameters’ noise, angle, distance, and Gaussian blur as $(40\%,-20,50,0.5),(106\%,-87,48,0.3),(146\%,53,33,0.5)$, respectively, for rendering the three rain images.

Figure 3 shows sample instances of this subset, which includes three kinds of images. Every image is formed by a different rain-streak pattern imposed onto a clean vehicle image from a different scene. Three scenes are classified into simple scenes (a single vehicle in an image taken under a clear sky), moderately complex scenes (many vehicles in an image taken under a clear sky), and complex scenes (many vehicles in an image taken under gray skies), respectively.

3.3. Rendering by the $SyRaGAN$ Algorithm

In order to further enrich the types of rain streaks, this paper uses the $SyRaGAN$ algorithm [24] to render rain patterns on the $VehicleColor$-24 dataset to construct another subset. $SyRaGAN$ is inspired by the mapping network used in the latest $I2I$ (image-to-Image) translation method, which maps the random noise space to the rain-pattern representation space to generate diverse rain patterns. $SyRaGAN$ consists of the feature map network M, encoder network E, two generators G, and two discriminators D. The network input consists of the clean background image $x_c$ and the rain image $x_r$ to produce the synthesized rain images $x_{sr1}$ and $x_{sr2}$ as output, while $S_z$ and $S_r$ are the extracted rain patterns from the network (see Figure 4).

Figure 4. SyRaGAN network structure overview [24]. M is the feature extracting module, E is the encoder, and D are the discriminators. The inputs are clean background image $x_c$ and rain image $x_r$, and the outputs are the synthesizing rain images $x_{sr1},\dots ,x_{sr2}$. Further, $S_z,S_r$ are the extracted rain patterns from the network.

The rain-image rendering process is as follows. First, the clean background image and the rain image are respectively input into the $SyRaGAN$ network, and the rain noise is extracted by the mapping network M to produce various rain styles. Second, rain streak is added to the clean background image to generate the rain image. Finally, the generated images are discriminated using the discriminator D. By this way, the network is optimized to generate rain images with various styles. For each image, five kinds of rain images with different directions, sizes, and thicknesses can be correspondingly generated.

To construct a dataset with diversity as wide as possible, we sampled one kind of clean background image with a single vehicle in a image named as “simple scene”, and rendered it into five rain images with five rain-streak patterns (Figure 5). Second, we sampled one kind of clean background sample with many vehicles in a image named as “medium scene”, and rendered it into five different rain images with five rain-streak patterns (Figure 6). Third, we sampled one kind of many vehicles under gray skies, which we named as “complex scene”, and rendered it into five different rain images with five rain-streak patterns (Figure 7). In processes such as those from the clean vehicle dataset $VehicleColor$-24, 8000 image samples are fed into $SyRaGAN$ to obtain 40,000 vehicle rain-image samples. Combined with the rain data subset generated by $PS$ technology, 40,300 rain images are finally obtained, with a resolution of $512\times 384$.

Figure 5. Rendered rain images with single vehicle. (a) Original clean image; (b–f) rendered rain images with 5 kinds of different rain-streak patterns.

Figure 6. Rendered rain images with many vehicles. (a) Original clean image; (b–f) rendered rain images with 5 kinds of different rain-streak patterns.

Figure 7. Rendered rain vehicle images under gray skies (the first image is original clean background image, and the rest are rendered rain images with five different types).

To summarize, the previous dataset $VehicleColor$-24 was labeled according to 24 standard vehicle colors, but lacked corresponding rain images. This paper leverages $SyRaGAN$ and $PS$ technologies to constitute $RainVehicleColor$-24. Some samples are illustrated in Figure 8.

Figure 8. Illustrations of some samples from $RainVehicleColor$-24.

4. Experimental Results

In this paper we used the metrics $PSNR$ and $SSIM$ to evaluate the quality of the recovered images. The formulas [37] are

$$MSE=\frac{1}{H\times W}\sum _{\mathrm{i}=1}^H\sum _{\mathrm{j}=1}^W{\left(X\right(i,j)-Y(i,j\left)\right)}^2,$$

(3)

$$PSNR=10{log}_{10}\frac{{(2^n-1)}^2}{MSE},$$

(4)

$$\begin{array}{c}SSIM(X,Y)=\left(\frac{2u_X{u}_Y+C_1}{{u_X}^2+{u_Y}^2+C_1}\right)\ast \left(\frac{2{\sigma }_X{\sigma }_Y+C_2}{{{\sigma }_X}^2+{{\sigma }_Y}^2+C_2}\right)\ast \left(\frac{{\sigma }_{XY}+C_3}{{\sigma }_X{\sigma }_Y+C_3}\right).\hfill \end{array}$$

(5)

4.1. Experimental Setup

In this paper, we first used $Rain100L$ and $RainVehicleColor$-24 as benchmark datasets to train the two deraining networks $PReNet$ [25] and $LDVS$ [26], to obtain the deraining network models $PReNet1$, $LDVS1$, $PReNet2$, and $LDVS2$. We then tested them on the synthetic and real rain images. In addition, we used $Faster$-$RCNN$ [38] to detect objects for vehicle-color classification after deraining by $PReNet1$, $LDVS1$, $PReNet2$, and $LDVS2$. The experimental results showed that deraining performance is improved after being trained by the $RainVehicleColor$-24, and the performance of $Faster$-$RCNN$ in subsequent vehicle-color fine recognition processes is also improved. All mean average precisions ($mAP$s) of vehicle-color classification are improved when corresponding specific vehicle recognition deep neural networks are trained on $RainVehicleColor$-24.

4.2. $PReNet$ Model Trained on $Rain100L$ and $RainVehicleColor$-24

4.2.1. $PReNet$ Network

Ren et al. [25] proposed a better and simpler baseline deraining network with six recurrent modules. Specifically, by repeatedly unfolding a shallow $ResNet$, progressive $ResNet\left(PRN\right)$ was proposed to take advantage of recursive computation. In this paper, we introduce a recurrent-layer $LSTM$ module to extract the dependencies of deep features across stages, forming the final framework, which we refer to as progressive recurrent network ($PReNet$). As for loss functions, single-$MSE$ or negative-$SSIM$ losses are sufficient for training $PRN$ and $PReNet$. The illustration of $PReNet$ is shown in Figure 9. As shown in this paper, the $PReNet$ is one of representative SOTA methods due to its simplicity, efficiency and effectiveness.

Figure 9. Illustration of $PReNet$ [25]. Input O is the rain image, output $BT$ is the clean background image of the T stage output.

4.2.2. Comparison of Synthetic Rain Images

We used the training subset of the Rain 100L and $RainVehicleColor$-24 datasets to train the $PReNet$ network, and then we obtained the deraining network models $PReNet1$ and $PReNet2$. Finally, we tested the rain-removal performance of the two models on the $Rain100L$ test set. The results are given in Figure 10 and Table 1.

Figure 10. Test results of $PReNet1$ and $PReNet2$ on two synthetic rain images from the $Rain100L$. $PReNet1$ and $PReNet2$ represent the different models that are trained on $Rain100L$ and $RainVehicleColor$-24 training subsets, respectively.

Table 1. Comparing $PSNR$, $SSIM$ of the $PReNet1$ and the $PReNet2$ on $Rain100L$, $RainVehicleColor$-24, respectively.

Metrics	Models	PReNet1		PReNet2
Datasets
Rain100L		PSNR	SSIM	PSNR	SSIM
		32.67	0.965	32.44	0.945
Rain Vehicle Color-24		PSNR	SSIM	PSNR	SSIM
		31.62	0.955	33.51	0.973

We tested the images sampled from the synthetic $Rain100L$ test subset with $PReNet1$ trained on $Rain100L$ and $PReNet2$ trained on the $RainVehicleColor$-24. The test results are shown in Figure 10. As can be seen from Figure 10, the $PReNet1$ model has a better visual effect than the $PReNet2$ model with further supporting results given in Table 1. The $PReNet1$ outperforms $PReNet2$ on the $PSNR$ and the $SSIM$ have been increased by margins $0.23$ and $0.02$, respectively. The reason why $PReNet1$ performs better is because the test and training images used by $PReNet1$ are identically distributed, with the two coming from the same dataset $Rain100L$. On the other hand, the rain-removal effect of $PReNet2$ trained with the $RainVehiclecolor$-24 is slightly worse than the former, since the test images derained by $PReNet2$ are not identically distributed with the training images. Most of the rain streaks have been removed and the background image can be restored. This means that when there is no domain gap, the test effect is better; furthermore, the latter faces the gap between the test and training data and still maintains the performance effect. In other words, the experiments show the better quality of $RainVehicleColor$-24.

Figure 11 shows the visual effects of the test images from $RainVehiclecolor$-24 and the testing models $PReNet1$ and $PReNet2$. Figure 11 shows that they are competitive enough with $PReNet1$, since the distributions of the training and test sets are the same, and $RainVehicleColor$-24 has more varying rain patterns. The results are given in Table 1.

Figure 11. Test results on synthetic rain images from $RainVehicleColor$-24. $PReNet1$ and $PReNet2$ represent the different models, which are trained on the training subsets of $Rain100L$ and $RainVehicleColor$-24, respectively.

4.2.3. Comparison of Real Rain Images

To compare the generalization of the rain-removal methods, the $PReNet$ models, which are trained on $RainVehicleColor$-24 and $Rain100L$, are tested on the real-rain images from the real-world rain image dataset. Figure 12 and Figure 13 show the two rain-removal models do not work; however, $PReNet2$ is still better than $PReNet1$.

Figure 12. Test result on real-rain image from $Real$$Data$. $PReNet1$ and $PReNet2$ represent the different models trained on the $Rain100L$ and the $RainVehicleColor$-24 training subsets, respectively.

Figure 13. Test results on two real-rain images containing vehicles from $RIS$. $PReNet1$ and $PReNet2$ represent the different models trained on $Rain100L$ and $RainVehicleColor$-24, respectively.

4.2.4. Comparison of Recognition Effects of Low- and High-Level Joint Tasks

In this section, we divided $VehicleColor-24$ into training, verification, and test sets at a ratio of 8:1:1. We fixed the confidence threshold at 0.5. We calculated the results using the relevant confidence code under Python.

To investigate the detection results of the vehicle-color fine recognization of $FasterRCNN$, $PReNet1$+$FasterRCNN$, and $PReNet2+FasterRCNN$ [38], we first used $FasterRCNN$ for target detection without deraining preprocessing; then, we used $FasterRCNN$ for target detection after deraining by $PReNet1$ or $PReNet2$. Figure 14 shows that $FasterRCNN$ recognizes the vehicle color on rain images with lower confidence; meanwhile, $FasterRCNN$ recognizes the vehicle color on derained rain images with higher confidence. Furthermore, $PReNet2+FRCNN$ performs better than $PReNet1+FRCNN$, since the former is trained on $RainVehicleColor$-24 with more various rain streaks and a much bigger data size.

Figure 14. Objective detection test results of $FasterRCNN,PReNet1/2+FasterRCNN$ on two synthetic rain images with vehicles from the $RainVehicleColor$-24 dataset. Each subtitle is corresponding object detection method and the corresponding confidence value in the parentheses.

From Figure 14, $FasterRCNN$, $PReNet1+FasterRCNN$, and $PReNet2+FasterRCNN$ achieve around $70\%,70\%$, and $90\%$ accuracy, respectively, for white-color-vehicle target detection. Therefore, the $RainVehicleColor$-24 dataset we proposed in this paper provides a better guarantee for visual tasks, such as vehicle-color target detection.

4.3. $LDVS$ Model Tested on $Rain100L$ and $RainVehicleColor$-24

4.3.1. $LDVS$ Network

The network framework of $LDVS$ [26] is shown in Figure 15, which is mainly composed of dilated convolution and lightweight attention modules. In the main network, there is an encoder with five feature extraction modules and a convolution operation and a decoder, where each feature extraction module concatenates a dilated convolution with a lightweight attention module $CBAM$. The rain image O is inputted into the network to extract the feature maps, and the output is the rain pattern R and the clean background image B. The clean image is equal to the input rain image O minus the feature map R. The loss function L is defined as:

$$\mathrm{Ł}=SSIM(\widehat{B},B)+\alpha \left|\right|\widehat{B}-B{\left|\right|}^2.$$

(6)

Figure 15. Illustration of $LDVS$ [26].

4.3.2. Comparison on Synthetic Rain Image

Without loss of generality, we trained the $LDVS1$ and $LDVS2$ models obtained through the $LDVS$ method on $Rain100L$ and $RainVehicleColor$-24, respectively. Figure 16 and Figure 17 show the tested synthetic rain images and the comparison of rain-removal results between the $LDVS1$ and $LDVS2$ models.

Figure 16. Test results of $LDVS1$ on synthetic rain images from $Rain100L$.

Figure 17. Test results of $LDVS2$ on rain image from $RainVehicleColor$-24 test subset. The first column is the original rainy vehicle image, the second is the clean background image (GT), and the third is the one after $LDVS2$. The image in the first row is in a simple scene, the one in the second row is in a medium complexity scene, and the one in the last row is in a complex scene.

Figure 17 shows that $LDVS$ performs better in rain-removal when the test images are taken from the $RainVehicleColor$-24 dataset. There is almost no obvious residual rain streaks on images c or f, and the background is relatively clear. Table 2 gives the testing results performed on $Rain100L$, showing that the performance of $LDVS2$ is similar to $LDVS1$. When testing on the $RainVehicleColor$-24 dataset, the performance of $LDVS2$ is better than $LDVS1$.

Table 2. Comparing $PSNR$ and $SSIM$ of test effect on $RainVehicleColor$-24 from the $LDVS1$ and the $LDVS2$.

Metrics	Models	LDVS1		LDVS2
Datasets
Rain100L		PSNR	SSIM	PSNR	SSIM
		33.56	0.959	33.12	0.960
Rain Vehicle Color-24		PSNR	SSIM	PSNR	SSIM
		31.23	0.951	34.34	0.960

4.3.3. Comparison of Real Rain Images

To further test the generalization of $LDVS$, the $LDVS1$ model pre-trained on the Rain 100L dataset and the $LDVS2$ model pre-trained on the $RainVehicleColor-24$ dataset were used to test rain removal on real-data images, and the results are shown in Figure 18. The qualitative results show that the LDVS2 trained on $RainVehicleColor-24$ generalization performs better than the $LDVS1$ trained on $Rain100L$.

Figure 18. Test results of $LDVS$ on real images from $Real$$Data$. The first column is the original real rainy image, the second is after $LDVS1$, and the third is after $LDVS2$.

Figure 18 shows that $LDVS1$ or $LDVS2$ works well for the rain removal effect of the synthetic rain images. On the other hand, when working in the real world, the effect is less ideal, and there are a large number of rain streak residues.

4.4. Object Detection Models Trained by $VehicleColor$-24 and $RainVehicleColor$-24

Four models, $SSD1$, $FasterRCNN1$, $SSD2$, and $FasterRCNN2$ are trained on $Vehicle$$Color$-24 and $RainVehicleColor$-24, and all models are then tested on the $VehicleColor$-24 and $RainVehicleColor$-24 test sets. The test results are shown in Figure 19. $SSD1$ and $FasterRCNN1$ hardly recognize the color of vehicles on rainy days, while $SSD2$ and $FasterRCNN2$ can recognize any vehicle color in the rain images. For example, in Figure 19e, white vehicle detection achieves $94\%$ accuracy after $SSD2$, and in Figure 19f, white vehicle detection achieves $100\%$ accuracy after $FasterRCNN2$. Testing on the $VehicleColor$-24 test subset, the results of the models are almost identical.

Figure 19. Object detection test results on rain images from $RainVehicleColor$-24. (a,d) Rain image; (b,c) object detection test result with $SSD1$ and $FasterRCNN1$, which are trained on $VehicleColor-24$; (e,f) object detection test result with $SSD2$ and $FasterRCNN2$, which are trained on the $RainVehicleColor-24$.

Table 3 shows the average accuracy of each category of the object detection algorithms when they test on the $RainVehicleColor-24$ dataset and are trained on $VehicleColor-24$. Table 4 shows the average accuracy of each category of object detection algorithms when they test on the $RainVehicleColor-24$ test set and are trained on $RainVehicleColor-24$. It is noted that almost all color is classified more accurately by the model trained on the $RainVehicleColor-24$.

Table 3. Comparison of the average accuracy of each category of object detection algorithms on $VehicleColor-24$ and $RainVehicleColor-24$ test sets. All object detection algorithms are trained on $VehicleColor-24$.

Category	$\mathrm{SMNN}-\mathrm{MSFF}1$		$\mathrm{Faster}R\mathrm{CNN}1$		$\mathrm{SSD}1$
	VC-24	RVC-24	VC-24	RVC-24	VC-24	RVC-24
White	0.98	0.64	0.84	0.80	0.96	0.74
Black	0.97	0.52	0.82	0.31	0.95	0.38
Orange	0.98	0.85	0.81	0.71	0.96	0.81
Silver gray	0.96	0.30	0.77	0.44	0.91	0.86
Grass green	0.98	0.82	0.70	0.61	0.96	0.96
Dark gray	0.94	0.30	0.66	0.17	0.84	0.29
Dark red	0.98	0.63	0.78	0.24	0.93	0.44
Gray	0.89	0.06	0.18	0.13	0.54	0.13
Red	0.96	0.65	0.60	0.20	0.88	0.41
Cyan	0.97	0.82	0.75	0.33	0.92	0.46
Champagne	0.97	0.17	0.63	0.29	0.81	0.25
Dark blue	0.96	0.39	0.66	0.12	0.86	0.36
Blue	0.97	0.59	0.73	0.10	0.87	0.69
Dark brown	0.97	0.09	0.45	0.02	0.71	0.11
Brown	0.88	0.36	0.30	0.13	0.58	0.27
Yellow	0.97	0.66	0.51	0.13	0.79	0.18
Lemon yellow	0.99	0.88	0.87	0.84	0.93	0.70
Dark orange	0.96	0.67	0.65	0.18	0.78	0.13
Dark green	0.94	0.28	0.38	0.08	0.58	0.00
Red orange	0.99	0.33	0.24	0.00	0.61	0.00
Earthy yellow	0.97	0.50	0.62	0.50	0.74	0.10
Green	0.93	0.13	0.61	0.33	0.74	0.00
Pink	0.94	0.66	0.50	0.33	0.71	0.17
Purple	0.80	0.00	0.00	0.00	0.19	0.00
mAP	94.96%	47.22%	58.59%	29.19%	78.13%	30.23%

Table 4. Comparison of the average accuracy of each category of object detection algorithms on the $VehicleColor-24$ and $RainVehicleColor-24$ test sets. All object detection algorithms are trained on $RainVehicleColor-24$.

Category	$\mathrm{SMNN}-\mathrm{MSFF}2$		$\mathrm{Faster}\mathit{R}\mathrm{CNN}2$		$\mathrm{SSD}2$
	VC-24	RVC-24	VC-24	RVC-24	VC-24	RVC-24
White	0.62	0.60	0.94	0.96	0.94	0.95
Black	0.61	0.69	0.82	0.69	0.92	0.93
Orange	0.69	0.77	0.92	0.90	0.95	0.95
Silver gray	0.48	0.31	0.44	0.81	0.85	0.86
Grass green	0.60	0.82	0.88	0.93	0.94	0.96
Dark gray	0.47	0.43	0.57	0.69	0.71	0.67
Dark red	0.36	0.48	0.73	0.79	0.83	0.88
Gray	0.18	0.31	0.12	0.41	0.35	0.31
Red	0.37	0.44	0.62	0.56	0.79	0.76
Cyan	0.42	0.62	0.71	0.82	0.87	0.87
Champagne	0.33	0.28	0.46	0.74	0.65	0.73
Dark blue	0.60	0.52	0.66	0.79	0.78	0.75
Blue	0.29	0.56	0.44	0.69	0.87	0.69
Dark brown	0.38	0.35	0.45	0.18	0.60	0.47
Brown	0.47	0.35	0.34	0.10	0.33	0.34
Yellow	0.51	0.35	0.94	0.83	0.72	0.92
Lemon yellow	0.32	0.57	0.95	0.99	1.00	0.75
Dark orange	0.41	0.32	0.52	0.28	0.11	0.47
Dark green	0.62	0.59	0.10	0.18	0.36	0.34
Red orange	0.52	0.38	0.66	0.07	0.29	0.52
Earthy yellow	1.00	0.68	0.23	0.45	0.78	0.28
Green	0.59	0.18	0.47	0.85	0.55	0.97
Pink	0.03	0.84	0.02	0.54	1.00	0.52
Purple	0.99	0.22	0.07	0.03	0.00	0.06
mAP	49.14%	48.58%	55.13%	60.65%	70.84%	66.33%

5. Conclusions

In this paper, the $RainVehicleColor-24$ dataset is constructed by rendering rain images based on $PS$ technology and the $SyRaGAN$ algorithm. The dataset has a total of 40,300 rain images, including 125 rain patterns. The aim of constructing $RainVehicleColor-24$ is to train data-driven deep learning neural networks for specific vehicle color recognition tasks. Specifically, $RainVehicleColor-24$ consists of two subsets: one is 300 rain images rendered by Photoshop from the $RainVehicleColor-24$ database, and the other is $40,000$ rain vehicle images rendered by the $SyRaGAN$ network from another 8000 vehicle images in $VehicleColor-24$. Extensive experiments show that, when $PReNet$ and $LDVS$ are trained on the new dataset, $RainVehicleColor-24$, both deraining task and subsequent target recognition algorithms after deraining are improved. More specifically, when the model is designed for the task of fine recognition of vehicle color, the corresponding recognition accuracy is improved for good or rainy weather conditions, after the model is fine-tuned on $RainVehicleColor-24$,

For future work, we will study low- and high-level joint tasks, based on the above work. We will focus on vehicle object detection and recognition in various adverse conditions such as bad weather. We will also consider fusing fuzzy sets, rough sets, overlap functions (see [39,40,41,42]) to expand the method of this paper. These studies will be critically beneficial in fields such as criminal investigation or traffic management law enforcement.