An Aircraft Object Detection Algorithm Based on Small Samples in Optical Remote Sensing Image

Abstract

In recent years, remote sensing technology has developed rapidly, and the ground resolution of spaceborne optical remote sensing images has reached the sub-meter range, providing a new technical means for aircraft object detection. Research on aircraft object detection based on optical remote sensing images is of great significance for military object detection and recognition. However, spaceborne optical remote sensing images are difficult to obtain and costly. Therefore, this paper proposes the aircraft detection algorithm, itcan detect aircraft objects with small samples. Firstly, this paper establishes an aircraft object dataset containing weak and small aircraft objects. Secondly, the detection algorithm has been proposed to detect weak and small aircraft objects. Thirdly, the aircraft detection algorithm has been proposed to detect multiple aircraft objects of varying sizes. There are 13,324 aircraft in the test set. According to the method proposed in this paper, the f1 score can achieve 90.44%. Therefore, the aircraft objects can be detected simply and efficiently by using the method proposed. It can effectively detect aircraft objects and improve early warning capabilities.

1. Introduction

At present, computer vision (CV) and natural language processing (NLP) and speech recognition are listed as three hotspot directions of artificial intelligence (AI). Computer vision can be used in many fields, such as detection of asteroids [1], meteors [2] and atmospheric events [3], self-driving [4], biomedical image detection [5], automated defect inspection [6], satellite image analysis [7], criminal analysis [8]. One of the classic applications is aircraft object detection. Object detection algorithm is used to identify which real-world objects [9] are and where the objects are in the image. For example, Figure 1 is an image in the Pascal VOC dataset [10], object detection algorithm needs to identify the bus and person in the image and locate them. Object detection algorithms can be used in a great number of fields, such as face detection, and shape detection [9]. Therefore, object detection has received great attention in the field of remote sensing, and it also faces many challenges [11].

Traditional object detection has gone through several important phases. Viola and Jones use Harr-like features [12] to train strong classifiers in AdaBoost [13,14], and then implement face detection by using sliding window search strategy. Dalal and Triggs propose the local Histogram of Oriented Gradient (HOG) as the image feature, and support vector machine (SVM) as the classifier to detect pedestrians [15]. Felzenszwalb et al. [16,17] propose a multiscale Defomable Parts Model (DPM) method [18] based on the idea that most objects can be regarded as rigid bodies. Malisiewicz et al. adopt the integration idea in machine learning, and train an SVM classifier for each positive sample by using HOG features [19]. Ren et al. design the histogram of sparse code (HSC) based on sparse representation theory [20].

With the swift improvement of computer hardware performance, convolutional neural networks (CNN) [21] have once again attracted the attention of researchers. In the Large Scale Visual Recognition Challenge (ILSVRC) 2012 [22], Alex and Hinton used the AlexNet [23] to achieve a huge leap in detection performance on the ImageNet dataset [24]. Therefore, CNN has received extensive attention in the field of computer vision.

Since the advent of AlexNet, a large number of CNN-based object detection algorithms have appeared. In 2014, Ross Girshick et al. applied CNN to object detection and proposed R-CNN algorithm [25]. Firstly, a selective search algorithm was used to generate a series of candidate regions. Then CNN was used to extract these features in these regions. Finally, the SVM classifier was trained for classification.

In 2015, Kaiming He et al. proposed SPP-net [26], which solved the problem of input image size limitation by using spatial pyramid pooling, and mapped the candidate regions in the original image directly to the feature map. This makes SPP-net faster than R-CNN by dozens of times.

In 2015, Girshick proposed Fast R-CNN [27], which uses two parallel different fully connected layers to perform classification and positioning tasks separately. Ren et al. proposed the Faster R-CNN [28] algorithm, which uses neural networks to extract candidate regions, i.e., RPN networks.

Since the extraction of the candidate region is time-consuming and complicated, some researchers directly find the position and category information of the object of interest on the input image. The most representative algorithms are YOLO [29] and Single Shot Multi-Box Detector (SSD) [30].

In 2018, Yin Zhou proposed a generic 3D detection network, i.e., VoxelNet [31]. It can perform feature extraction and bounding box prediction into a single stage, end-to-end trainable deep network.

Although deep learning has achieved great success in object detection, the detection performance will be significantly reduced in the case of insufficient datasets. Generally, spaceborne optical remote sensing images are difficult to obtain and costly. Therefore, this paper studies the aircraft object detection algorithm with small samples.

The circle frequency filter algorithm is small in calculation, good in saliency objects detection. However, the performance of circle frequency filter greatly reduces when detecting small objects and multiple objects. Therefore, this article proposed the fusion algorithm to improve the circle frequency filter detection accuracy with small samples. The experimental results show that the proposed method can correctly detect 12,215 aircraft on test set, accounting for 91.68% of the total number of aircraft in the test set. From the data analysis, it can be seen that in the case of small samples, the proposed method has better detection performance than Faster R-CNN, and the detection speed is faster.

2. Theoretical Analysis and Methods

Generally, the grayscale, shape, size and shadow of aircraft objects in remote sensing images are used to distinguish aircraft from other interfering objects. Usually aircraft objects are brighter than the background, and most aircraft have similar shapes that they have four parts of the aircraft nose, left wing, tail and right wing. As shown in Figure 2a, the portion where the aircraft fuselage and the wingspan intersect may form a rectangle whose length and width are the widths of the fuselage and the wing, respectively. If the center of gravity of the rectangle is the center of the circle, draw a circle is with a diameter larger than the width of the fuselage and smaller than the length of the fuselage and wingspan, as shown in Figure 2b. The pixel points on the circumference are extracted clockwise, and the gray value of the pixels on the circumference will show the characteristic of “bright-dark-bright-dark-bright-dark-bright-dark” [32]. As shown in Figure 2c, a bar graph is made by taking 601 pixels on the circumference as the horizontal axis and the gray value at each pixel as the vertical axis. However, when the center of the circle is not at the center of the aircraft, it does not have such a characteristic. Therefore, based on this feature, aircraft and non-aircraft can be distinguished.

Let f_k (k = 0, 1, 2, …, N − 1) denote the gray value on the circumference centered on (i, j) and the radius is R. Make a discrete Fourier transform on this one-dimension array and obtain

$$F={\displaystyle \sum _{k=0}^{N-1}{f}_k{e}^{-j\frac{2\pi }{N}kn}(k=0,1,2,\dots ,N-1)},$$

(1)

Square the above equation to get the amplitude value.

$$F^2={\left({\displaystyle \sum _{k=0}^{N-1}{f}_k\cos \frac{2\pi }{N}kn}\right)}^2+{\left({\displaystyle \sum _{k=0}^{N-1}{f}_k\sin \frac{2\pi }{N}kn}\right)}^2,$$

(2)

where n is a constant used to control the number of periods of the sine and cosine functions for the discrete Fourier transform.

The gray value curve on the circumference has 4 peaks and 4 valleys, which is similar to the sine and cosine curves of 4 cycles. So, if four cycles of sine and cosine functions are selected in Equations (1) and (2) (that is, n = 4), the amplitude value of the circumferential frequency filter [32] in the center of the aircraft is large.

Since the non-aircraft center point usually does not have the special properties, the value of the circle frequency filter [32] is small. The amplitude value is the square of the value obtained by the Fourier transform, therefore, the amplitude values on the center of the aircraft and the non-aircraft center will vary greatly. Therefore, it is easy to distinguish between aircraft centers and non-aircraft centers.

2.1. Weak and Small Object Detection Algorithm

Our dataset has 419 images, with a size of 305 × 296. It contains a total of 2175 aircraft. In order to better verify the effect of the proposed method, the aircraft dataset of RSOD-Dataset [33] and UCAS_AOD [34] are added to our dataset. RSOD-Dataset included 4993 aircraft in 446 images, with a size of 1044×915. UCAS_AOD dataset includes 7482 aircraft in 1000 images. Images in RSOD-Dataset and UCAS_AOD dataset both have three channels, therefore, they need to be converted to grayscale. Among the dataset, 2841 aircraft are weak and small objects. They are either smaller than 10×10, or the grayscale value of them is almost equal to that of the ground background. The aircraft in each image is manually marked in the form of (x, y, r), where (x, y) is the center of the circle with the characteristics of four peaks and four valleys, and r is the radius. The dataset is divided into a training set and a test set, of which the training set accounts for about 10% and the test set 90%. Training set contains 1326 aircraft. In order to compare the test results with Faster R CNN, our dataset is annotated according to the label form of the PASCAL VOC2007 [10] dataset.

The circle frequency filter detection algorithm is used to detect the saliency object, and the detection accuracy is high. The amplitude value of each point on the circumference is calculated and normalized to between 0 and 255. According to multiple experiments, the detection effect is better when the threshold is set to 80. Therefore, the gray value greater than 80 is set to 255, and the other gray values are set to 0. As shown in Figure 3b, circles are drawn on the original image centering on pixel points with a gray value of 255 in the filter result image, and the circumference is just at the center of the airplane.

As shown in Figure 4a,b there are a total of six aircraft in the picture. Only two aircraft can be correctly detected. As shown in Figure 4c,d the circle frequency filter cannot detect aircraft. Therefore, the circle frequency filter is not suitable for weak and small aircraft object detection. Based on this problem, the circle frequency filter detection algorithm is improved to specifically detect weak and small aircraft objects.

The circle frequency filter detection algorithm has the following two reasons for the inaccurate detection of weak aircraft objects. Firstly, the difference between the gray value of the aircraft object and the background is too small, resulting in a mixture of the background and the object, which is difficult to distinguish. Secondly, the edge of the aircraft is blurred, and the gray value on the circumference does not show the characteristics of four peaks and four valleys. So, it can;t determine whether it is an aircraft object.

Usually, most of the grayscale values on the aircraft object are not much different, and the background grayscale values are not much different too. We can use this feature to solve the above two problems, that is, to square each point on the image matrix. It is assumed that the grayscale value of a point x on the aircraft target is 9, and the grayscale value of the background point y adjacent to x is 4, and the difference in grayscale value is 5. Only the circle frequency filter detection algorithm cannot distinguish between the object and the background, so that such a weak object cannot be detected. Therefore, we can square the two grayscale values, and the grayscale values of the x and y points are 81 and 16, respectively. Then the gray values sampled on the circumference normalize to make the difference between the data more obvious. Based on the characteristics of the aircraft object, the data can be determined whether it is an aircraft object.

As shown in Figure 5, weak and small aircraft can be detected using the proposed algorithm. In Figure 5b, the circumferential radius of the detected aircraft is only 2 pixels, and the minimum difference between objects and background in Figure 5d is only 1 pixel. In Figure 5a,b, some of the detection circumferences are not located at the center of the aircraft. Therefore, mean shift clustering is used to cluster the detection points to the center of the aircraft.

2.2. Fusion Algorithm

As shown in Figure 5, the use of weak and small object detection algorithms to detect aircraft objects will mark multiple bounding boxes near the center of the aircraft. Since the bounding boxes are concentrated near the center of the aircraft, only one bounding box can be left on each aircraft by using the mean shift cluster.

2.2.1. Clustering Algorithm

In order to discover interesting groups, cluster analysis is applied to a variety of scientific areas [35]. In this paper, mean shift cluster algorithm is used to cluster points which are the result of weak and small object detection algorithm. Mean Shift, a method of nonparametric probability density estimation, was originally proposed by Fukunaga et al. [36]. Yizong Cheng [37] defined a family of kernel functions, which made the contribution of drift to the mean drift vector different with the difference of sample and drift point; secondly, he set a weight coefficient, which made the importance of different sample points different.

Let X be the n-dimensional Euclidean space Rⁿ. Then, donate the ith component of $x\in X$ by $x_i$, and the norm of $x\in X$ is a non-negative number $\parallel x\parallel$. A function K: X→R is said to be a kernel if there exist a profile k: [0, ∞]→R

$$K\left(x\right)=k\left({\Vert x\Vert }^2\right),$$

(3)

where k is nonnegative, nonincreasing and piecewise continuous.

Let K be a kernel and ω: S→(0, ∞) a weight function. Then, with the kernel K at $x\in X$, the sample mean is:

$$m(x)=\frac{{\displaystyle \sum _{s\in S}K(s-x)\omega (s)s}}{{\displaystyle \sum _{s\in S}K(s-x)\omega (s)}},$$

(4)

Let T be a finite set (cluster centers). The evolution of T in the form of iterations T←m(T) is called a mean shift algorithm.

Let X be the d-dimensional Euclidean space, R^d. Donate the ith component of $x\in X$ by $x_i$, and the mean vector at $x\in X$ is:

$$M_h\left(x\right)=\frac{1}{k}{\displaystyle \sum _{x_i\in S_h}\left(x_i-x\right)},$$

(5)

and S_h refers to a spherical filed with a radius with h.

$$S_h\left(x\right)=\left\{y|\left(y-x\right){\left(y-x\right)}^T\le h^2\right\},$$

(6)

With the kernel function and sample weight, the mean vector at $x\in X$ is

$$M_h(x)=\frac{{\displaystyle \sum _{x_i\in S_h}K\left(\frac{x_i-x}{h}\right)\omega (x_i)(x_i-x)}}{{\displaystyle \sum _{x_i\in S_h}K\left(\frac{x_i-x}{h}\right)\omega (x_i)}},$$

(7)

where $\omega \left(x_i\right)$ is the weight of the sample $x_i$, and ${\displaystyle \sum _{x_i\in X}\omega \left(x_i\right)=1}$.

In the process of the proposed algorithm, firstly, calculate the mean vector M_h (x) between a random point A in the sample and its surrounding radius with h. Then, the next drift direction of this point is calculated, that is A = M_h (x) + A. At the new center A, the previous steps repeat to iterate out the new center. When the point no longer moves, it forms a cluster with the surrounding points, and all the points encountered during this iteration should be grouped into this cluster. Calculate the distance between this cluster and the other cluster, and merge into the same cluster if the distance is less than the threshold, and form a new cluster if it’s not satisfied. Finally, repeat the previous steps until all points in the sample are accessed.

According to our experience, h take to be twice the radius of the circumference. After clustering the filtered results on Figure 5, the resulting graph is shown in Figure 6. Compared with other algorithms, Mean Shift can automatically discover cluster structure without specifying the number of clusters.

2.2.2. False Alarms Elimination

There are not only airplanes in the original images, but also many buildings around the airport. Detecting aircraft only using circle frequency filter can produce many false alarms. Therefore, support vector machine (SVM) is used to eliminate false alarms. Firstly, the dataset is divided into a training set and a test set. The training set is used to fit the SVM model, and the test set is used to finally evaluate the performance and classification ability of the model. The training set includes 66,300 negative samples and 13,260 positive samples. Then, the SVM model is trained on the training set. Finally, the classification ability of the model was evaluated on the testing set.

3. Results

In many images, aircraft vary in size. Therefore, we use the circle frequency filter first to detect the large and saliency aircraft, then deduct the part of these aircraft on the original image, and finally, the small aircraft. When the weak and small aircraft is included in images, the improved circle frequency filter algorithm is used for detection. The trained SVM model is used to classify the test results, and then the targets classified as non-aircraft are removed. The fusion algorithm flow is shown in

Table 1. The fusion algorithm flow.

The Proposed Algorithm
Input: Test dataset images
Execute: (1) According to the statistical radius, the circumference of the corresponding radius is taken for the pixels on each image. (2) Calculate the magnitude of the gray value on the circumference. (3) According to the amplitude value and the special characteristics of the aircraft to determine whether it is an aircraft. (4) According to the weak and small object detection algorithm, process data and calculate the magnitude to determine whether it is an aircraft. (5) Remove some false alarms by clustering algorithm. (6) Use the trained SVM to eliminate a false alarm.
Output: Images contain the location of the aircraft

.

In order to compare the detection results of the proposed method and Faster R CNN [28], we train Faster R CNN on our training set and test it. A pre-trained backbone network VGG16 model on the Imagenet [38] dataset is adopted. In order to ensure that there is no over-fitting, the maximum number of iterations is 2000. Part of the image detection map is shown in Figure 7.

There are 13,324 aircraft in the test set, which is recorded as P. A total of 18,812 samples were detected using the proposed algorithm. Using the SVM model to eliminate false alarms, 13,687 objects were obtained. Among them, 12,215 samples can be correctly detected, denoted as TP (True Positive). 1109 samples were classified as negative sample. A sample of 1472 non-aircraft objects was incorrectly judged by the improved method and recorded as FP (False Positive). A total of 1109 aircraft targets were not be detected, and they were denoted as FN (False Negative). Therefore, TN (True Negative) is 0.

Faster R CNN detects a total of 13,687 objects. Among them, 11,335 samples can be correctly detected. 4361 non-aircraft objects was incorrectly judged by Faster R CNN and recorded as FP. A total of 1989 aircraft objects were not be detected, and they were denoted as FN. TN is 0.

According to the actual test results of the dataset, precision, accuracy, recall, missing alarm and false alarm are shown in

Table 2. Some evaluation indicators.

Evaluation Indicator	The Proposed Algorithm	Faster R CNN
Precision	89.25%	72.22%
Accuracy	82.56%	64.09%
Recall	91.68%	85.07%
Missing Alarm	8.32%	14.93%
False Alarm	10.75%	27.78%
F-measure	90.44%	78.12%

. 12,215 aircraft can be correctly detected by using the proposed algorithm. It can be seen from both the visualization results and some evaluation indicators that the proposed method is better than Faster R CNN on our dataset. In the case of small samples, the detection accuracy of the object detection method based on a convolutional neural network is greatly reduced. On the test set, Faster R CNN takes about 6.383 s to detect an image, while the proposed algorithm only takes 2.951 s. Therefore, the proposed algorithm can be adopted to efficiently detect aircraft objects.

4. Conclusions

There are 13,324 aircraft in the test set, and 12,215 aircraft can be detected by using the proposed method. The recall is 91.68%, the precision is 89.25%, and the accuracy is 82.56%. Experiments show that the proposed method is simple and effective for detecting aircraft objects. Especially for small samples learning, the proposed algorithm can accurately detect aircraft objects, which greatly saves costs for the research. The research on aircraft object detection has significant value for improving early warning capability and counterattack capability. However, about 9% aircraft can’t be detected by using our algorithm, mainly because the center of the circle on the aircraft don’t have the characteristics of four peaks and four valleys. To address this problem, in the next research, a new algorithm will be proposed to detect the remaining aircraft.