Using Vehicle Synthesis Generative Adversarial Networks to Improve Vehicle Detection in Remote Sensing Images

Abstract

Vehicle detection based on very high-resolution (VHR) remote sensing images is beneficial in many fields such as military surveillance, traffic control, and social/economic studies. However, intricate details about the vehicle and the surrounding background provided by VHR images require sophisticated analysis based on massive data samples, though the number of reliable labeled training data is limited. In practice, data augmentation is often leveraged to solve this conflict. The traditional data augmentation strategy uses a combination of rotation, scaling, and flipping transformations, etc., and has limited capabilities in capturing the essence of feature distribution and proving data diversity. In this study, we propose a learning method named Vehicle Synthesis Generative Adversarial Networks (VS-GANs) to generate annotated vehicles from remote sensing images. The proposed framework has one generator and two discriminators, which try to synthesize realistic vehicles and learn the background context simultaneously. The method can quickly generate high-quality annotated vehicle data samples and greatly helps in the training of vehicle detectors. Experimental results show that the proposed framework can synthesize vehicles and their background images with variations and different levels of details. Compared with traditional data augmentation methods, the proposed method significantly improves the generalization capability of vehicle detectors. Finally, the contribution of VS-GANs to vehicle detection in VHR remote sensing images was proved in experiments conducted on UCAS-AOD and NWPU VHR-10 datasets using up-to-date target detection frameworks.

1. Introduction

Fast and robust vehicle detection in remote sensing images has potential applications in traffic surveillance, emergency management, and economic analysis. Moreover, the location and density information of vehicles is an essential data source for building intelligent transportation systems. However, accurate and robust vehicle detection from remote sensing images has been a challenging task for many years.

Traditional vehicle detection methods rely on handcrafted features extracted from sliding windows with different scales. For instance, Shao et al. (2012) [1] extracted Harr features and local binary patterns from images and classified vehicles with a support vector machine. Kluckner et al. (2007) [2] and Tluckner et al. (2013) [3] applied the histogram of oriented gradients and integral channel features to train the AdaBoost classifier. However, these methods are highly reliant on manually designed features and they cannot adequately cope with large variations in targets and backgrounds. Recently, the convolutional neural network (CNN) has been applied to aerial image object detection, and it achieved promising results. For instance, Faster R-CNN (Region-Convolutional Neural Networks) [4,5] and You Only Look Twice (YOLT) [6] have been applied to various benchmarks.

Many approaches have been proposed for vehicle detection [7,8,9,10], yet the robustness problem remains unsolved. Based on CNN models, recent works [11,12,13] have achieved good detection performances on several benchmarks. Hu et al. (2019) [14] proposed a scale-insensitive convolutional neural network that could detect vehicles with a large variance of scales. Gao et al. [15] presented an end-to-end vehicle detection model that unifies and combines the deep and shallow feature maps, followed with deformable convolution and region of interest (RoI) pooling. It achieved good performance on vehicle target detection in dense areas.

Built on a massive amount of training datasets, these models can achieve significant performance improvement over previous baselines. However, the performance of CNN-based vehicle detectors is heavily dependent on the quality and quantity of annotations of the training data. The imaging configuration of remotes sensing data is variant (incidence angle, distance, sensor type, etc.), which leads to a significant difference between targets in remote sensing images and those from other images. Meanwhile, labeling ground truth bounding boxes for vehicle locations in aerial images requires considerable human efforts. Therefore, it is extremely crucial to design approaches that only rely on limited labeled images.

To alleviate the demand of massive sample annotation in the deep learning era, weakly supervised methods were proposed. Li et al. proposed a weakly supervised deep learning method that only used scene-level tags, which took advantage of both the information from separate scene categories and mutual cues between scene pairs [16]. Zhao et al. proposed a multiscale image block-level fully convolutional neural network that only requires class labels without bounding boxes [17]. However, these methods still require a large number of image samples.

Augmenting the labeled dataset is a more direct and practical approach. There are some common data augmentation methods that can be used to extend datasets over time, such as flip, rotation, color jittering, random crop, etc. Montserrat et al. [18] applied some regular data augmentation methods where linear and nonlinear transforms were done on the training data to create “new” training images for object detection. Oliveira et al. [19] generated smooth face image variations to improve age estimation based on the detection of fiducial points on the face. However, all of the above methods are restricted to very straightforward transformations or are based heavily on simple image transformations and have a limited performance on improving the robustness of the model due to the restricted diversity of samples.

Based on Generative Adversarial Nets (GANs) [20] proposed by Goodfellow et al., labeled data samples could be generated with larger diversity while maintaining the core characteristics of the original data samples [21]. The overall flowchart of data augmentation with GANs is shown in Figure 1. In a GAN, the generator and classifier networks are jointly learned based on the observed training dataset, and the observed and synthesized data samples are combined to train the detector network.

In this study, a data augmentation framework called Vehicle Synthesis Generative Adversarial Networks (VS-GANs) is proposed, which utilizes GANs to generate labeled data samples for the vehicle detection task (Figure 2). The highlights of this study include the following:

—

Our proposed model is capable of generating clear and photorealistic vehicle images in remote sensing images and can fit the background well in real images.

—

The data generated by our model can be combined with real datasets to train CNN-based detectors. This data augmentation step can improve both detection performance and robustness compared with the baseline method.

—

The proposed method is convenient to apply within the training process of the CNN-based detector.

2. Vehicle Generation Model

2.1. Generative Adversarial Networks

GANs have been widely used to improve the training stability and the quality of data generation [22,23,24,25]. It has also served in many other applications, such as image translation [26,27,28,29,30], super resolution [31], and image inpainting [32,33,34]. Liu et al. [35] proposed a DP-GAN (Domain Priori GAN) for obtaining super-resolution vehicle license plate images, which achieved pixel-level detail recovery.

Some researchers have also applied GANs for object detection. For example, Inoue et al. [36] proposed an object detection method that used labeled datasets (such as PASCAL VOC) to transfer training unlabeled datasets (such as cartoon datasets) based on cycle-GAN. Li et al. [37] proposed a Perceptual Generative Adversarial Network (Perceptual GAN) model that improved small object detection through narrowing the representation difference between small objects and large ones. Zhu et al. [26] added cycle consistency loss to the original GAN, enabling the model to conduct translations without paired training examples and task-specific designed functions. To synthesize the vehicles in the noise boxes, we adopted the training method of cycle-GAN and proposed a relatively simple architecture with one generator and two discriminators.

Compared with image inpainting works [32,33,34], which aim to fill the randomly removed monochromatic patches in the original image, our framework fills the bounding box area with noise rather than monochromatic blocks to generate patches with diverse shapes and colors. The work by Liu et al. [35] exploited a similar GAN with two discriminators for image inpainting to learn more context information of the surrounding pixels. In contrast, we passed the vehicle patch cropped from the generated output into the discriminator and used it in the model to generate vehicles with different appearances.

2.2. Vehicle Synthesis-GAN

The vehicle generation model in this paper is inspired by the structure of the original GANs. In the training stage, as shown in Figure 2, we cover the vehicle in the ground truth image with random noise and then send the noise-added image into the generator. There are two adversarial losses between the generator and both discriminators: $D_v$ for discrimination vehicles and $D_b$ for background learning.

2.2.1. The Structure of the VS-GANs Model

(a) Structure of the generator

The purpose of the generator is to learn a function mapping relation F: x→y, where x is the input noise image and y is the ground truth image. As shown in Figure 3, the encoder–decoder structure in deep learning is adopted in the experiment. In order to alleviate the gradient disappearance caused by the design of a too deep model and increase the semantic information of deep features, the basic residual block structure [38] is added to the generator structure. When the noise image x is fed into the generator, a series of convolution layers is used to reduce the size and increase the depth of the output feature map. This process captures the deep features of the image. These features are extracted through a set of residual modules to extract more abundant semantic information. Then the final feature map is extracted through the decoder composed of deconvolution structures.

(b) Structure of the background discriminator

Discriminator $D_b$ is used to learn background context information. The task for our model is not only to generate a vehicle sample but also to integrate the generated vehicle sample into the background image smoothly. Therefore, the model is required to learn the background information of the image, such as the surrounding environment and lighting conditions. In this work, we use $D_b$ to distinguish real background pairs from generated background pairs. The positive pairs are the noise image x and the ground truth image y, and the generated pairs are the concatenation of the noise image x and the generated image. As shown in Figure 4, the structure of $D_b$ follows the design of the DCGAN(deep convolutional GAN) [25], but the following adjustments are made: (1) the first convolution structure is adjusted so that it can accept six input channels, (2) $D_b$ has a similar structure with PatchGAN [28], in which $D_b$ is used to distinguish the true and false parts of each area of N × N pixels (in this experiment, N is 19), and (3) in order to adapt to the design pattern of the PatchGAN and to generate better quality images, the least-squares loss function is selected as the loss function of the discriminator $D_b$.

(c) Structure of the vehicle discriminator

The discriminator $D_v$ is used to distinguish the positive vehicles from the negative vehicles. The positive vehicles are real vehicles in the ground truth image, and the negative vehicles are fake vehicles generated by the generator at the location of the noise box. The discriminator $D_v$ can drive the generator to learn the mapping relationship between noise z and real vehicles. The structure of the discriminator $D_v$ is shown in Figure 5. The main structure uses a six-layer convolutional neural network followed by a LeakyReLU layer.

2.2.2. Loss Function of VS-GANs

As shown in Figure 2, our model consists of two adversarial learning procedures, $\mathrm{G}\leftrightarrow D_b$ and $\mathrm{G}\leftrightarrow D_v$. The adversarial learning procedure between G and $D_b$ can be formulated as follows:

$$ℒ \left(\mathrm{G},D_b\right)=E_{y~p_{gt.image\left(y\right)}}[{\left(D_b\left(y\right)-1\right)}^2]+E_{x,z~p_{noise.image\left(x,z\right)}}\left[{\left(D_b\left(G\left(x,z\right)\right)\right)}^2\right]$$

(1)

where x is the image with the noise box and y is the ground truth image. We replace the original GAN loss by the least-squares loss that was used in LSGAN(Least squares GAN) [22].

In order to generate a realistic vehicle using G in the noise box z of input image x, another adversarial training process is carried out between G and $D_v$:

$$\begin{array}{c}ℒ \left(\mathrm{G},D_v\right)=-E_{y_v~p_{vehicle\left(y_v\right)}}\left[D_v\left(y_v\right)\right]+E_{z~p_{noise\left(z\right)}}\left[D_v\left(G\left(z\right)\right)\right]\\ +\lambda E_{z~p_{noise\left(z\right)}}[{({\parallel {\nabla }_z{D}_v\left(z\right)\parallel }_2-1)}^2]\end{array}$$

(2)

where z is the noise box in image x. And $y_v$ is the cropped vehicle from ground truth image y, and the gradient penalty(GP) strategy used in WGAN(Wasserstein GAN) [39] is taken to balance the training procedure.

In this paper, we use ${\ell }_1$ loss to balance the difference between the synthesized image and the ground truth image:

$${ℒ}_{{\ell }_1}\left(G\right)=E_{x,z~p_{noise.image}\left(x,z\right),y~p_{gt.image\left(y\right)}}\left[{\parallel y-G\left(x,z\right)\parallel }_1\right].$$

(3)

The final loss function is calculated by adding the previously defined losses. The $\lambda$ is a hyper-parameter to control ${\ell }_1$ loss:

$$ℒ \left(\mathrm{G},D_b,D_v\right)=ℒ \left(\mathrm{G},D_b\right)+ℒ \left(\mathrm{G},D_v\right)+\lambda {ℒ}_{{\ell }_1}\left(G\right).$$

(4)

3. Experimental Results

The experiments mainly include two parts: In the first, the generated vehicle samples with VS-GANs were compared with those derived from other data augmentation methods. In the second, vehicle detection was conducted in order to test the contribution of the proposed method.

UCAS-AOD data [40] were used to train the VS-GANs, and the quality of the synthesized images was evaluated. Vehicles generated with VS-GANs were compared with those generated by traditional data augmentation methods and other GANs. To analyze the influence of the data augmentation, we used both the real and synthesized data to train the YOLOv3 [41] and RetinaNet [42] detectors and evaluated the performance. Vehicles generated at both “proper” and “random” locations were considered to examine the effect of vehicle location on the training of the detector. The experiments were based on the PyTorch platform and accelerated with Tesla M40 Graphics Processing Units(GPUs).

3.1. Datasets

UCAS-AOD [40] is a dataset for object detection in remote sensing images and contains 510 images with 7114 vehicle samples. All images were collected from Google Earth and manually labeled. Compared to the other benchmark DOTA [43], USAC-AOD has higher resolution and a relatively similar scale and ratio, which is more suitable to train GANs.

3.2. Qualitative Analysis of the Generated Vehicle Samples

In Figure 6, we compare VS-GANs with three other basic data augmentation methods frequently used in the training procedure of CNN-based detectors. It intuitively shows the difference between VS-GANs and other data augmentation methods. The original images are in the first column, and the second column shows the noise boxes added to the location where the vehicle is about to be generated. The third column to fifth columns show the results derived by some basic data augmentation methods. We can see that the appearance of the vehicles is not changed and that they only have a linear transformation. The last column shows that the VS-GANs-based data augmentation resulted in a significant change in appearance. Moreover, the new vehicles generated by VS-GANs in the images share the characteristics of the original vehicles.

We generated noise within the bounding boxes in images from the UCAS-AOD and NWPU VHR-10 datasets. There were some labeled vehicles that are too small or partially blocked by other objects. Therefore, only those vehicles with a width between 20 and 70 pixels and a length–width ratio between 1.3 and 2.2 were selected. Then, we cropped the 300 × 300 patches around the chosen vehicles from the original images. We obtained 3409 images and randomly selected 10% of them as the testing dataset, and used the others as the training data for VS-GANs.

We added the pre-trained generator to the CNN-based training procedure to validate the effectiveness of VS-GANs. Next, 300 × 300 patches around the vehicles were cropped from the original images with a certain probability, and noise boxes were filled to cover the real vehicles in those patches. Our well-trained generator could synthesize vehicles within those noise boxes.

Several commonly used GANs were compared to our proposed method. In our experiment, we found that using the least-squares GAN [22] for $D_b$ is useful for learning the background context. VS-GANs achieved the best picture quality when applying the least-squares loss for the adversarial learning $\mathrm{G}\leftrightarrow D_b$ and keeping the GAN loss stable for $\mathrm{G}\leftrightarrow D_v$. The reasons why the two discriminators $D_b$ and $D_v$ led to different optimal GAN losses in our network were as follows. (1) For $D_v$, since we applied the PatchGAN trick and the gradient penalty strategy used by WGAN-GP [39], the least-squares GAN with least-squares loss obtained a larger error than the regular GAN loss, which made the generator more sensitive to every pixel as compared to that in the regular GAN. Thus, the generator was forced to learn too much detailed information about vehicles instead of capturing their more general characteristics. (2) The discriminator $D_b$ can benefit from the least-squares loss when learning the background context information. We expect the generator to smoothly learn the background information from the vehicles’ surrounding pixels.

The generated and real vehicles are shown in Figure 7. We can see that VS-GANs can generate vehicles with a clear shape and a more photorealistic appearance compared with some existing GAN methods. It should be noted that to achieve such a result we only used 8850 vehicles in 3409 images. Generating samples only at the position of existing vehicles may limit the variety of samples; therefore, we also generated vehicles at random positions. As shown in Figure 8, some of the generated vehicles are at unrealistic positions.

3.3. Vehicle Detection Experiments

Data augmentation based on VS-GANs was applied to train the vehicle detector with the framework of YOLOv3 and RetinaNet on datasets UCAS-AOD and NWPU VHR-10. In the first experiment, we randomly put noise boxes at the location of the vehicle to generate diverse vehicles on the training dataset, as shown in Figure 2. The experimental datasets were cropped from original UCAS-AOD data and NWPU WHR-10 data to a size of 416 × 416 pixels. In total, there were 3083 training images and 326 testing images. First, YOLOv3 and RetinaNet were used to train the detector based on the training dataset. Then, the data samples generated by VS-GANs were applied. All the derived detectors were tested on 326 testing images and the average precision (AP) was calculated when all the models converged. As shown in

Table 1. Experimental results of data augmentation based on VS-GANs.

Source of Vehicle Samples	YOLOv3	RetinaNet
UCAS + NWPU (8850 vehicles)	92.91%	84.28%
+VS-GANs augmentation	96.12%	90.78%

, although YOLOv3 and RetinaNet achieved a relatively good performance (92.91% and 84.28%, respectively), combining this data augmentation with the vehicle detector significantly improved the vehicle detector performances (to 96.12% and 90.78%, respectively).

Examples of synthetic vehicles generated by the existing GANs and the proposed VS-GANs are shown in Figure 9. It can be observed that vehicles generated by VS-GANs can be smoothly and realistically integrated into the background of the original image. To prove how the synthetic images can help boost the performance of vehicle detection between the existing GANs and VS-GANs, we used the vehicle samples to train the YOLOv3 detector. The accuracy was derived based on 5-fold cross-validation. As shown in

Table 2. Performance of vehicle detector based on the existing GANs and VS-GANs.

Source of Vehicle Samples	DCGAN	LSGAN	WGAN-GP	VS-GANs
UCAS + NWPU (8850 vehicles)	92.91% (0.66) *
+125 images (1000 synthesized vehicles)	93.84% (0.46)	93.65% (0.65)	94.16% (0.34)	95.40% (0.33)
+250 images (2000 synthesized vehicles)	94.17% (0.41)	94.77% (0.38)	95.44% (0.18)	96.12% (0.21)

* The mean value of 5-fold cross-validation is listed, followed by standard deviation in the brackets.

, adding more generated data samples achieved higher detection accuracy, and the proposed VS-GANs were shown to outperform other vehicle-generating algorithms. The standard deviation (listed in the bracket) of 5-fold cross-validation is relatively small. Hence, the analysis result is comparable and can be referenced.

The locations of noise boxes were chosen to be where vehicles were already present in the first experiment. In the second experiment, we also selected some images without vehicles from the UCAS-AUO dataset and randomly put noise boxes into these images (as shown in Figure 6). It can be observed that if the noise boxes are located at the wrong locations, the generator may fail to generate proper vehicle samples due to an unrealistic background, as shown in Figure 9.

To give more insight on the effect of the number of samples, synthesized vehicles were gradually added into training samples of VS-GANs. As shown in Figure 10, the baseline APs for YOLOv3 and RetinaNet were 92.91% and 84.28%, respectively (highlighted in the green box). With adding the screened vehicle samples from 100 to 2000, the performance gradually increased to 96.12% and 86.96% for YOLOv3 and RetinaNet, respectively. However, when adding random vehicle samples, the performance of YOLOv3 and RetinaNet dropped gradually to 89.31% and 81.02%. Moreover, by adding 2000 augmented data samples, the growth tendency of the AP was not yet stopped, which suggests that the performance can further increase. Hence, a higher upper-bound of the proposed method could be expected.

It was clearly observed that adding samples at proper positions can improve the vehicle detection results, while adding samples at improper positions has an adverse effect on the vehicle detection performance. This is because there may be some water or forest areas in the images, which could lead to unsuccessful vehicle generation, as shown in Figure 11. Unsuccessful samples from the generator have large differences compared with real vehicles, which is not conducive for vehicle detector training.

In the experiments of this paper, most of the trained, detected, and generated vehicles were small vehicles such as cars, SUVs, and minivans due to their abundance in data samples. Given sufficient samples, the proposed algorithm could also work on buses, trucks, and other types of vehicles. Figure 12 shows different types of vehicles detected by the proposed method.

4. Conclusions

This study proposed a learning method called VS-GANs, which can quickly generate high-quality annotated vehicles from remote sensing data with limited annotated vehicle samples. The synthesized vehicles can benefit the training of CNN-based vehicle detectors. The VS-GANs model can also help boost the performance of vehicle detection if the generated vehicles are placed at the right locations of images without vehicles, which demonstrates its ability to transfer knowledge. The proposed model could be useful in other related tasks in which only limited datasets are available. The key findings of this study can be summarized as follows:

(1) The vehicle samples generated by VS-GANs hold characteristics similar to those of real vehicle samples while providing diverse details.

(2) Adding synthesized samples in the training process significantly improves the performance of the vehicle detector.

(3) The location of the generated vehicle sample has to be appropriately selected to avoid introducing an unrealistic background.

The method proposed in this study has great potential to improve the performance of vehicle detectors. The generated data samples have been proved to significantly enlarge the variety of data samples and improve the performance of target detection algorithms. The discriminator trained by VS-GANs can be applied to distinguish vehicles in backgrounds. Moreover, the technique proposed in this study could be used in other target detection/identification tasks in which only limited datasets are available.