YOLO-RTUAV: Towards Real-Time Vehicle Detection through Aerial Images with Low-Cost Edge Devices

Abstract

Object detection in aerial images has been an active research area thanks to the vast availability of unmanned aerial vehicles (UAVs). Along with the increase of computational power, deep learning algorithms are commonly used for object detection tasks. However, aerial images have large variations, and the object sizes are usually small, rendering lower detection accuracy. Besides, real-time inferencing on low-cost edge devices remains an open-ended question. In this work, we explored the usage of state-of-the-art deep learning object detection on low-cost edge hardware. We propose YOLO-RTUAV, an improved version of YOLOv4-Tiny, as the solution. We benchmarked our proposed models with various state-of-the-art models on the VAID and COWC datasets. Our proposed model can achieve higher mean average precision (mAP) and frames per second (FPS) than other state-of-the-art tiny YOLO models, especially on a low-cost edge device such as the Jetson Nano 2 GB. It was observed that the Jetson Nano 2 GB can achieve up to 12.8 FPS with a model size of only 5.5 MB.

1. Introduction

Unmanned aerial vehicles (UAVs), commonly known as drones, are widely available commercially at a low cost. UAVs can provide aerial views of almost everywhere without any elaborate planning and geographical constraints. This has led to the usage of UAVs in various fields, including search and rescue missions [1,2], agriculture [3], vehicle tracking [4,5,6], and environment monitoring [7,8]. In particular, UAVs have seen their applications in intelligent transportation systems (ITSs). As the transportation system is becoming more complex, vehicle detection from aerial images are increasingly important. This helps in traffic flow management, identifying vehicles, and parking lot management, to name a few. Vehicle detection is the first step in many traffic surveillance tasks. Therefore, it is viewed as the future trend in transportation and vehicle-related applications.

Generally, vehicle detection in aerial images can be classified into two categories, the traditional machine learning methods and the deep learning approaches. Traditional machine learning methods usually collect hand-crafted features such as edges and corners to classify the object in an image. Other traditional methods include the histogram of oriented gradients, frame difference, and optical flow. These techniques have competitive inference speed due to their comparatively simple computation, but usually have low accuracy as they are trained on selected features. These techniques do not perform well if the image is not seen before training.

Recently, deep learning methods have achieved a significant breakthrough in various fields in computer vision. In terms of object detection, many detection algorithms have shown great performance in image detection tasks, such as region-based convolutional neural networks (R-CNNs) [9,10,11], you only look once (YOLO) [12,13,14,15], and the single-shot detector (SSD) [16,17]. However, those algorithms are usually trained and tested on large-scale natural images, such as MSCOCO [18] and PASCAL VOC [19], which are different compared to aerial images. Aerial images tend to have a high variation due to the altitude and smaller object sizes. Varied flying altitudes cause captured images to have different sizes of objects with different resolutions, therefore creating many different visual appearances. Ultimately, several classes could share the same appearance if images are captured at higher altitudes. Besides, the lighting condition changes as the image is captured at different times, causing a big variation in the captured images. Furthermore, aerial images tend to have many objects in a single image. Vehicles may be partially occluded by trees and other constructions. Thus, using the pretrained state-of-the-art models would yield lower accuracy.

Many methods and algorithms are proposed to solve the issue of detecting small objects in aerial images [20,21,22]. However, there is less focus on developing a real-time detector on low-cost edge hardware, such as the Jetson Nano 2 GB. The tradeoff between detection accuracy and inference time has not been well addressed. Encouraged by the previously stated problems, in this work, we propose an object detection method based on YOLO, focusing on small object detection that could achieve near-real-time detection on low-cost hardware, coined YOLO-RTUAV. To achieve that, we modify the existing YOLOv4-Tiny, a one-stage object detector with relatively high accuracy and speed in detecting small objects in aerial images. We used two public datasets, namely the Vehicle Aerial Imaging from Drone (VAID) [23] and Cars Overhead With Context (COWC) datasets [24], to validate our proposed model. We performed extensive experiments, to achieve a fair comparison with the state-of-the-art models.

The rest of this paper is organized as follows. Section 2 provides a brief overview of previous works. Section 3 revisits the object detection algorithms and explains our proposed model. Section 4 describes the experimental settings used for this work, including the dataset and model parameters used. The results and discussion are given in Section 5, and Section 6 concludes this paper and provides suggestions for further works.

2. Related Works

Various techniques have been proposed to detect vehicles in aerial images. The main challenges of this field are summarized as below.

• The vehicle in the image is small in size. For example, a $5k\times 3k$ px image may contain multiple vehicles of size less than $50\times 50$ px;
• A large number of objects in a single view. A single image could contain up to hundreds of vehicles in a parking lot;
• High variation. Images taken with different altitudes will present objects of the same class with different features. For example, when the altitude is high, a vehicle will be viewed as a single rectangular object and is difficult to differentiate;
• Highly affected by the illumination condition and occlusions. Reflection due to sunlight and background noise such as trees and buildings would render the object unobserved.

Before the popularity of the deep learning approach, many traditional machine learning methods were proposed. These techniques heavily rely on hand-crafted feature extraction for image classification. This usually requires two steps to detect and classify an object. Firstly, a feature extractor is used to extract features that are important to differentiate from one another. Generally, shape, color, texture, and corners are some of the common collected features used to differentiate between classes. More sophisticated techniques that look into the image gradient, such as the histogram of oriented gradients (HOG) and scale-invariant feature transform (SIFT), are used to collect low-level features. Then, the collected features are combined with classifiers such as support vector machine (SVM), AdaBoost, bag-of-words (BoW), and random forest (RF) to detect and recognize different classes of objects [25,26,27,28,29].

Even though deep learning methods have been proven to perform better in object detection tasks, several recent studies have used traditional machine learning approaches. They are designed to use in a specific task because of the low complexity and computation. Among them, Xu et al. [30] proposed an enhanced Viola–Jones detector for vehicle identification from aerial imagery. Chen et al. [31] extracted texture, color, and high-order context features through a multi-order descriptor. Then, superpixel segmentation and patch orientation were used to detect the vehicle in high-resolution images. Cao et al. [32] proposed an affine function transformation-based object-matching framework for vehicle detection. Similar to the previous work, superpixel segmentation was adopted to generate nonredundant patches, then detection and localization were performed with a threshold matching cost. Liu et al. [4] developed a fast oriented region search algorithm to obtain the orientation and size of an object. A modified vector of locally aggregated descriptors (VLAD) was applied to differentiate between object and background from the proposals generated. Cao et al. [33] proposed a weakly supervised, multi-instance learning algorithm to learn the weak labels without explicitly labeling every object in an image. SVM was then trained to classify from the density map derived from the positive regions.

Many advancements have been made in object detection algorithms, especially on small object detection, since aerial images usually contain many small objects. Deep-learning-based object detection techniques that use convolutional neural networks (CNNs) have been regarded as the best object detectors. Generally, the current object detection techniques can be divided into two categories, namely one-stage detectors and two-stage detectors. Two-stage detectors are usually accurate, but lack speed, while one-stage detectors are fast with relatively high accuracy. We refer the readers to two very recent surveys on small object detectors based on deep learning [34] and techniques for vehicle detection from UAV images [35]. In [34], the authors highlighted some well-performing models to detect generic small objects, including improved Faster R-CNN [36,37,38], the semantic context-aware network (SCAN) [39], SSD [40,41], RefineDet [42], etc. The authors in [35] discussed three main pillars to improve the models through optimizing accuracy, achieving optimizing objectives, and reducing computational overhead.

The key ideas of improving detection on small objects in aerial images include fusion and leveraging size information. Several proposed techniques fused the features of the shallow layers with deep layers [43,44] to differentiate between objects and background noise. Besides, some works generated vehicle regions from multiple feature maps of scales and hierarchies to locate small target objects [45,46]. Skip connections were used in [43] to reduce the loss of information in deep layers, while passthrough layers were used in [47] to combine features from various resolutions. More commonly, the deep layers are stripped off to detect small objects, which helps in increasing the number of feature points per object [45,48,49,50]. Some works suggested removing large anchor boxes [51,52] or removing small blobs to avoid false positives [53].

Undoubtedly, one-stage detectors are the best in terms of accuracy and speed tradeoff. Among them, the famous detectors are the YOLO family [12,13,14,15]. Many works further reduced the size of the YOLO models, allowing real-time detection [54,55,56,57]. There are several compressed models based on SSD [58] and MobileNet [59]. Mandal et al. proposed AVDNet [60] with a smaller size compared to RetinaNet and Faster R-CNN. We would like to further improve the YOLO models for real-time detection on low-cost embedded hardware in this work. Similar works have produced great performance, but are still far from real time [58,61].

3. Methodology

3.1. Object Detection Algorithm

Object detection is one of the important tasks in image processing. Prior to 2012, object detection was usually performed with classical machine learning approaches. As discussed in Section 2, the object detection algorithms based on deep learning are classified into two large branches: one-stage detectors and two-stage detectors. The architecture of both object detection algorithms is shown in Figure 1.

Figure 1. One-stage and two-stage object detection algorithm.

We selected the best-performing one-stage and two-stage detectors to benchmark against our proposed model in this work. We selected Faster R-CNN [11] for two-stage detector, while YOLOv2, YOLOv2-Tiny, YOLOv3, YOLOv3-Tiny, YOLOv4, and YOLOv4-Tiny [13,14,15] for the one-stage detector.

3.1.1. Two-Stage Detector: Faster R-CNN

R-CNN [9] uses region proposals for object detection in an image. The region proposal is a rough guess of the object location. A fixed number of regions are extracted through a selective search. For every region proposal, a feature vector is extracted. Similar regions are merged together through the greedy algorithm, and candidate regions are obtained. Fast R-CNN [10] improves the approach by using a shared convolutional feature map generated directly from the input image to form a region of interest (RoI). Faster R-CNN [11] introduces a region proposal network (RPN) to predict the object bounding boxes and objectness score with little effect on the computational time. The architecture is shown in Figure 2.

Figure 2. Faster R-CNN architecture.

Conceptually, Faster R-CNN comprises three components, namely the feature network, RPN, and detection network. The feature network is usually a state-of-the-art pretrained image classification model, such as VGG or ResNet, with stripped classification layers. It is used to collect features from the image. In this work, we used a feature pyramid network (FPN) with ResNet50. FPN combines low-resolution and semantically powerful features with high-resolution, but semantically weak features via a top-down pathway and lateral connections. The output of the FPN is then used for the RPN and ROI pooling. The RPN is a simple network with three convolutional layers. One common convolutional layer is fed into two layers for classification and bounding box regression. The RPN will then generate several bounding box proposals that have a high probability of containing objects. The detection network or the ROI head then considers the input from the FPN and RPN to generate the final class and bounding boxes.

3.1.2. One-Stage Detector: You Only Look Once

YOLO [12] predicts and classifies bounding boxes of objects in a single pass. Compared to Faster R-CNN, there is no region proposal phase. YOLO first splits an image into $S\times S$ nonoverlapping grids. YOLO predicts the probability of a present object, the coordinates of the predicted box, and the object’s class for each cell in the grids. The network predicts B bounding boxes in each cell and the confidence scores of these boxes. Each bounding box carries five parameters, $(x,y,w,h,sc)$, where $x,y$ are the center of the predicted box, $w,h$ is the width and height of the box, and $sc$ is the confidence score. Then, the network calculates the probabilities of the classes for each cell. The output of YOLO is $(S,S,B\times 5+C)$, where C is the number of classes. The framework is shown in Figure 3. The first version of YOLO, coined YOLOv1, reportedly achieves a faster inference time, but lower accuracy compared to a single-shot detector (SSD) [16].

Figure 3. YOLO model detection framework.

YOLO9000, or more commonly known as YOLOv2 [13], was proposed to improve the accuracy and detection speed. YOLOv2 uses convolutional layers without fully connected layers (Darknet-19) and introduces anchor boxes. Anchor boxes are predefined boxes of certain shapes to capture objects of different scales and aspects. The class probabilities are calculated at every anchor box instead of a cell, as in YOLOv1. YOLOv2 uses batch normalization (BN) and a high-resolution classifier, further boosting the accuracy of the network.

YOLOv3 [14] uses three detection levels rather than the one in YOLOv1 and YOLOv2. YOLOv3 predicts three box anchors for each cell instead of the five in YOLOv2. At the same time, the detection is performed on three levels of the searching grid ($S\times S$, $2S\times 2S$, $4S\times 4S$) instead of one ($S\times S$), inspired by the feature pyramid network [62]. YOLOv3 introduces a deeper backbone network (Darknet-53) for extracting feature maps. Thus, the prediction is slower compared to YOLOv2 since more layers are introduced.

YOLOv4 [15] was introduced two years after YOLOv3. Many technical improvements were made in YOLOv4, while maintaining its computational efficiency. The improvements are grouped into bag of freebies (BoF) and bag of specials (BoS). In BoF, all improvements introduced did not affect the inference time. The authors implemented CutMix and mosaic data augmentation, DropBlock regularization, class label smoothing, complete IoU (CIoU) loss, cross mini-batch normalization (CmBN), self adversarial training (SAT), the cosine annealing scheduler, optimal hyperparameters through genetic algorithms, and multiple anchors for a single ground truth. On the other hand, BoS represents improvements that will affect the inference time slightly, but have a significant increase in accuracy. They include the Mish activation function, cross-stage partial (CSP) connections, the multi-input weighted residual connection (MiWRC), spatial pyramid pooling (SPP), the spatial attention module (SAM), the path aggregation network (PAN), and distance IoU loss (DIoU) in nonmaximum suppression (NMS). We refer the reader to the original article for more information.

This work considered six different YOLO models, which were YOLOv2, YOLOv2-Tiny, YOLOv3, YOLOv3-Tiny, YOLOv4, and YOLOv4-Tiny.

3.2. Proposed Model: YOLO-RTUAV

YOLOv4-Tiny was developed to simplify the YOLOv4 network, allowing the network to run on low-end hardware. The model only contains a two-layer output network. Therefore, YOLOv4-Tiny has ten-times fewer parameters than YOLOv4. However, it is not able to detect a multiscale object and missing detection with overlapped objects. As for aerial images, objects are usually small in size, yet occluded by surrounding items such as trees and buildings. Therefore, an improved YOLOv4-Tiny in terms of detection accuracy, inference speed, and model size is required.

To solve the problems raised above, an improved YOLOv4-Tiny is proposed, coined YOLO-RTUAV. Our proposed model is shown in Figure 4. The proposed YOLO-RTUAV network should achieve three objectives:

Figure 4. The YOLO-RTUAV model architecture. n is calculated such that $(\mathrm{number} \mathrm{of} \mathrm{classes}+5)\times 3$.

• Detect small objects in aerial images;
• Efficient, yet accurate for real-time detection in low-end hardware;
• Small in size and reduced in parameters.

The main improvements of our proposed model are summarized below:

• Changing the output network to a larger size, allowing smaller objects to be detected;
• Usage of Leaky ReLU over the Mish activation function [63] in the convolutional layers to reduce the inference time. The backbone remains the same as the original YOLOv4-Tiny, which is CSPDarknet-19;
• Usage of several $1\times 1$ convolutional layers to reduce the complexity of the models;
• DIoU-NMS is used to reduce suppression error and lower the occurrence of missed detection;
• Complete IoU loss is used to accelerate the training of the model and improve the detection accuracy;
• Mosaic data augmentation is used to reduce overfitting and allows the model to identify smaller-scale objects better.

Firstly, the detection layer size is changed to detect smaller objects. The detection layer remains two layers, with the output size of $13\times 13\times n$ replaced with $52\times 52\times n$. YOLOv4-Tiny can detect objects in a wide range of natural image objects, but cannot detect small objects in aerial images. Therefore, the highly subsampled layers are not necessary, whereby we removed the coarse detection level layer of size $13\times 13\times 36$ and replaced it with a finer detection level layer of size $52\times 52\times 36$. Given an input image of size $416\times 416\times 3$, the $26\times 26\times n$ detection layer is suitable for medium-scale object detection, and the $52\times 52\times n$ detection layer is suitable for small-scale object detection. Therefore, the proposed YOLO-RTUAV can realize smaller object detection, which is critical for aerial images.

The backbone of our proposed model is CSPDarknet-19. In contrast to YOLOv4’s CSPDarknet-53, CSPDarknet-19 has gradually reduced the number of parameters while not hurting the accuracy significantly. The cross-stage partial (CSP) module divides the feature map into two parts and combines the two parts by the cross-stage residual edge, as illustrated in Figure 4. The CSP module allows gradient flow to propagate in two different paths, increasing the correlation difference of gradient information. Therefore, the network can learn significantly better than residual blocks. To further reduce the inference time, the Mish activation layer is not used in our proposed model. Instead, Leaky ReLU is used since the calculation was not complex, requiring less computational time.

Since the second YOLO output layer is calculated from the second and third CSP layer, which has a mismatch in channel size and output shape, naively, one would use convolutional layers to match the channel size. Therefore, the agreed channel size is then 512. This, however, requires large computation when the last convolutional layer is dealing with a large channel size. To solve this issue, we first used two $1\times 1$ convolutional layers before concatenation, agreeing on the channel size to be 256. Thus, the computation needed is smaller, with no effect on detection accuracy. A further explanation is available in Appendix A.

The most commonly used nonmaximum suppression (NMS) technique is Greedy NMS, a greedy iterative procedure. Although Greedy NMS is relatively fast, it comes with several pitfalls. It suppresses everything within the neighborhood with lower confidence, keeps only the detection with the highest confidence, and returns all the bounding boxes that are not suppressed. Therefore, Greedy NMS is not optimal in this work since the objects in the dataset are usually small and overlapped objects tend to be filtered out. To address the issue, DIoU-NMS [64] was used. DIoU-NMS considers both the overlapping areas and the distance between the center points between the two borders. Thus, it can effectively optimize the process of suppressing redundant bounding boxes and reducing missed detections. The definition of DIoU-NMS is shown in Equation (1).

$$S_i=\left\{\begin{array}{cc}{S}_i,\hfill & IoU-{\mathcal{R}}_{DIoU}\left(\mathcal{M},B_i\right)<\epsilon \hfill \\ 0,\hfill & IoU-{\mathcal{R}}_{DIoU}\left(\mathcal{M},B_i\right)\ge \epsilon \hfill \end{array}\right.$$

(1)

where ${\mathcal{R}}_{DIoU}$ denotes the distance of the center points of the two boxes, $S_i$ is the classification score, and $\epsilon$ is the NMS threshold. Equation (1) compares the distance of the prediction box $\mathcal{M}$ of the highest score with the i-th box, $B_i$. If the differences between $IoU$ and ${\mathcal{R}}_{DIoU}$ exceeds $\epsilon$, the score $S_i$ remains, otherwise it is filtered out.

The complete intersection over union (CIoU) loss function is deployed for bounding box regression. CIoU has proven to improve accuracy and accelerate the training process. Usually, the objective loss function for YOLO models is the sum of bounding box loss, confidence loss, and classification loss. The bounding box regression loss function is applied to calculate the difference between the predicted and ground truth bounding boxes. The common metric to calculate the bounding box regression is the intersection over union (IoU). The IoU is used to measure the overlapping ratio between the detected bounding box, $B_p$, and the ground truth bounding box, $B_{gt}$, as shown in Equation (2).

$$IoU=\frac{area(B_p\cap B_{gt})}{area(B_p\cup B_{gt})}$$

(2)

The IoU loss is then calculated as shown in Equation (3).

$${\mathcal{L}}_{IoU}=-lnIoU=-ln\frac{area(B_p\cap B_{gt})}{area(B_p\cup B_{gt})}$$

(3)

IoU loss suffers from a lower decrease rate during training. When there is no intersection between the ground truth and predicted box, the IoU is zero. Therefore, IoU loss does not exist. This renders the inability of the loss function to reflect the distance between the ground truth and the predicted box. Therefore, CIoU loss is used to overcome the stated issue. The CIoU uses the distance between the predicted and ground truth bounding box center point and compares the aspect ratio between the ground truth and predicted bounding box. The formula is shown in Equation (4). Figure 5 illustrates the calculation of the CIoU.

$${\mathcal{L}}_{reg}={\mathcal{L}}_{CIoU}=1-IoU+\frac{{\rho }^2(B,B_{gt})}{c^2}+\alpha v$$

(4)

where $\rho (·)$ is the Euclidean distance, c is the diagonal length of the smallest enclosing box covering the two boxes, and $\alpha$ is a positive tradeoff parameter, denoted as $\alpha =\frac{v}{(1-IoU)+v}$. v measures the consistency of aspect ratio, $v=\frac{4}{{\pi }^2}{\left(arctan\frac{w_{gt}}{h_{gt}}-arctan\frac{w}{h}\right)}^2$, where w and h represent the width and height of the box, respectively.

Figure 5. CIoU calculation conceptualized.

The classification function, ${\mathcal{L}}_{cls}$, only penalizes if the object is present in the grid cell. It also penalizes the bounding box coordinate if that box is responsible for the ground box (highest IOU). It is calculated as shown in Equation (5),

$$\begin{array}{cc}\hfill {\mathcal{L}}_{cls}& =-\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{obj}\left[{\widehat{C}}_{i}log\left(C_i\right)+(1-{\widehat{C}}_i)log(1-C_i)\right]\hfill \end{array}$$

(5)

The confidence loss, ${\mathcal{L}}_{conf}$, is calculate such that,

$$\begin{array}{cc}\hfill {\mathcal{L}}_{conf}& =-{\lambda }_{noobj}\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{noobj}\left[{\widehat{C}}_{i}log\left(C_i\right)+(1-{\widehat{C}}_i)log(1-C_i)\right]\hfill \\ \hfill & -\sum _{i=0}^{S^2}{I}_{ij}^{obj}\sum _{c\in classes}\left[{\widehat{p}}_i\left(c\right)log\left(p_i\left(c\right)\right)+(1-{\widehat{p}}_i\left(c\right))log(1-p_i\left(c\right))\right]\hfill \end{array}$$

(6)

The total loss function in our proposed model is the same as YOLOv4. The loss function comprises three parts: classification loss (${\mathcal{L}}_{cls}$), regression loss (${\mathcal{L}}_{reg}$), and confidence loss (${\mathcal{L}}_{conf}$). The loss is therefore calculated as shown in Equation (7).

$$\begin{array}{cc}\hfill \mathcal{L}& ={\mathcal{L}}_{reg}+{\mathcal{L}}_{cls}+{\mathcal{L}}_{conf}\hfill \\ \hfill & =1-IoU+\frac{{\rho }^2(B,B_{gt})}{c^2}+\alpha v-\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{obj}\left[{\widehat{C}}_{i}log\left(C_i\right)+(1-{\widehat{C}}_i)log(1-C_i)\right]\hfill \\ \hfill & -{\lambda }_{noobj}\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{noobj}\left[{\widehat{C}}_{i}log\left(C_i\right)+(1-{\widehat{C}}_i)log(1-C_i)\right]\hfill \\ \hfill & -\sum _{i=0}^{S^2}{I}_{ij}^{obj}\sum _{c\in classes}\left[{\widehat{p}}_i\left(c\right)log\left(p_i\left(c\right)\right)+(1-{\widehat{p}}_i\left(c\right))log(1-p_i\left(c\right))\right]\hfill \end{array}$$

(7)

where $S^2$ is the grids generating B candidate boxes. Each candidate box will obtain the corresponding bounding boxes through the network and form $S\times S\times B$ bounding boxes. If no objects are detected in the box ($noobj$), only the confidence loss of the box is calculated. Cross-entropy error is used to calculate the confidence loss and is divided into $obj$ (object) and $noobj$ (no object). The weight coefficient, $\lambda$, is used to reduce the weight of the $noobj$ loss function. As for classification loss, cross-entropy error is used. The j-th anchor box of the i-th grid is responsible for the given ground truth, then the bounding box generated by this anchor box will calculate the classification loss function.

Furthermore, we utilized mosaic data augmentation while training the model. This combines four training images into one with certain ratios. This technique allows the model to identify objects of a smaller scale. It also encourages the model to localize different types of images in different portions of the frame. A sample of images that have undergone mosaic data augmentation is shown in Figure 6.

Figure 6. Example of mosaic data augmentation images from the VAID dataset [23].

We compare our proposed models with the original YOLO models in Table 1. It is observed that all the greatness of YOLOv4-Tiny remains while the number of layers is reduced. Since some of the layers are removed, the model size is smaller as well.

Table 1. Characteristics of various YOLO models.

Parameters	Models
	YOLOv2	YOLOv2-Tiny	YOLOv3	YOLOv3-Tiny	YOLOv4	YOLOv4-Tiny	Ours
Number of layers	31	15	106	23	161	37	35
Anchors	5	5	9	6	9	6	6
Multilevel prediction	-	-	3	2	3	2	2
Backbone	Darknet-19	9 Conv Layers	Darknet-53	Darknet-19	CSPDarknet-53	CSPDarknet-19	CSPDarknet-19
FPN	✗	✗	✓	✓	✓	✓	✓
SPP	✗	✗	✗	✗	✓	✓	✓
PAN	✗	✗	✗	✗	✓	✓	✓
Augmentation
Saturation	✓	✓	✓	✓	✓	✓	✓
Exposure	✓	✓	✓	✓	✓	✓	✓
Hue	✓	✓	✓	✓	✓	✓	✓
Mosaic	✗	✗	✗	✗	✓	✓	✓
Model Size	194 MB	43 MB	236 MB	34 MB	245 MB	23 MB	5.5 MB

4. Experiment Settings

4.1. Dataset Description

This paper utilized two datasets, namely the Vehicle Aerial Imaging from Drone (VAID) dataset [23] and the Cars Overhead With Context (COWC) dataset [24].

4.1.1. VAID Dataset

This dataset contains 5985 images with varied illumination conditions and viewing angles, collected at different places in Taiwan. It contains seven classes, and the ratio of the training, validation, and testing datasets was set as 70:20:10. The number of objects in each class is shown in Table 2. The images have a resolution of $1137\times 640$ px in JPG format. Samples of the images are shown in Figure 7.

Table 2. Training, validation, and testing split for the VAID dataset.

Class	Sedan	Minivan	Truck	Pickup Truck	Bus	Cement Truck	Trailer
Training	28,613	313	2192	2118	413	128	542
Validation	7930	129	684	610	111	48	158
Testing	3787	59	311	283	56	15	104

Figure 7. Sample images from the VAID dataset with annotated classes.

4.1.2. COWC Dataset

This dataset comprises several high-resolution RGB and grayscale images captured at different locations (Columbus of size $4965\times 3313$, Potsdam of size $2220\times 2220$, Selwyn of size $\mathrm{18,075}\times \mathrm{18,400}$). We used the updated dataset, named COWC-M, where the authors further divided the dataset into four vehicle classes instead of only one main class. The authors patched the images in the dataset with the size of $512\times 512$. For example, a single image of size $2220\times 2220$ was divided into 49 patches. The images were then further divided into training, validation, and testing with the ratio of 80:10:10. The number of objects in each set is shown in Table 3. The patches of a single image were split into the training, validation, and testing set randomly. Samples of the image patches are shown in Figure 8.

Table 3. Training, validation, and testing split for the COWC dataset.

Class	Sedan	Pickup	Other	Unknown
Training	6240	486	4012	905
Validation	780	68	617	153
Testing	653	53	426	96

Figure 8. Illustration of obtaining image patches of size $512\times 512$ from a large image in the COWC dataset.

Table 4 summarizes the datasets used in our experiments.

Table 4. Datasets used in this study.

Parameters	Dataset
	VAID	COWC
Number of images	5985	1829
in training dataset	4189	1467
in validation dataset	1197	183
in testing dataset	599	179
Number of classes	7	4
Number of objects	48,606	14,489
in training dataset	34,319	11,643
in validation dataset	9670	1618
in testing dataset	4615	1228
Spatial resolution	12.5 cm	15 cm
Vehicle size (pixels)	$20\times 40$	$24\times 48$

4.2. Experimental Setup

The framework used to train Faster R-CNN was Detectron 2 [65], with the configuration file named faster_rcnn_R_50_FPN_3x.yaml. The model was trained with 15,000 epochs, with 1000 linear warmup iterations. The training rate was reduced at 12,000 and 13,500 epochs. The initial training rate was set to 0.001. The learning rate is shown in Figure 9. The batch size was set to 4. We kept the default values for the momentum (0.9) and weight decay (0.0001).

Figure 9. The scheduled learning rate used to train the Faster R-CNN models.

As for YOLO and our proposed model, the framework used to train the models was Darknet. The model was trained with 15,000 epochs, with 1000 exponential warmup iterations according to $l{r}_i=lr\times {\left(\frac{iter}{1000}\right)}^4$. The training rate was reduced at 12,000 and 13,500 epochs. The initial training rate was set to 0.001, 0.0013, and 0.00261 for different YOLO models and is shown in Figure 10. We kept the default values for the momentum (0.949) and weight decay (0.0005) for YOLOv4, while all other models were set to a momentum of 0.9 and a weight decay of 0.0005.

Figure 10. The scheduled learning rate used to train the YOLO models.

The anchor boxes used in all the models are shown in Table 5. Our proposed model’s anchor box sizes were the same as YOLOv4-Tiny, since our initial search for optimum anchor box sizes had almost a similar set as YOLOv4-Tiny. Note that choosing a large number of prior boxes will produce greater overlap between anchor boxes and bounding boxes. However, as we increase the number of anchor boxes, the number of convolution filters in prediction filters increase linearly, which will result in a large network size and increased training time. For each models, three different input sizes ($416\times 416$, $512\times 512$, and $608\times 608$) were considered and trained.

Table 5. Anchor box sizes for different models.

Models	Anchor Box Sizes
Faster R-CNN	Size: (32,32), (64,64), (128,128), (256,256), (512,512)
	Aspect Ratio: 0.5, 1.0, 2.0
YOLOv2	(0.57273,0.677385), (1.87446,2.06253), (3.33843,5.47434), (7.88282,3.52778), (9.77052,9.16828) ${}^{†}$
YOLOv2-Tiny	(0.57273,0.677385), (1.87446,2.06253), (3.33843,5.47434), (7.88282,3.52778), (9.77052,9.16828) ${}^{†}$
YOLOv3	(10,13), (16,30), (33,23), (30,61), (62,45), (59,119), (116,90), (156,198), (373,326)
YOLOv3-Tiny	(10,14), (23,27), (37,58), (81,82), (135,169), (344,319)
YOLOv4	(12,16), (19,36), (40,28), (36,75), (76,55), (72,146), (142,110), (192,243), (459,401)
YOLOv4-Tiny	(10,14), (23,27), (37,58), (81,82), (135,169), (344,319)
Ours	(10,14), (23,27), (37,58), (81,82), (135,169), (344,319)

${}^{†}$ YOLOv2 downsamples the image with a factor of 32. For example, a $416\times 416$ image is downsampled into multiple $13\times 13$ grids, and each grid is now $32\times 32$. The actual anchor box size is now $(0.57273\times 32,0.677385\times 32),\dots$

All of the models were trained with the Nvidia Tesla T4 or Nvidia Quadro P5000 GPUs. The trained models were then supplied to two powerful hardware, Nvidia Tesla T4 and Nvidia Quadro P5000, and one edge hardware, Nvidia Jetson Nano 2 GB, to calculate the inference time.

4.3. Evaluation Criteria and Metrics

The following criteria were used to evaluate and compare the various models:

• The IoU measuring the overlap between the predicted and the ground truth bounding boxes, as discussed in Equation (2);
• The main evaluation metric used in evaluating object detection models is mAP. mAP provides a general overview of the performance of the trained model. A higher mAP represents better detection performance. Equation (8) shows how mAP is calculated;

  $$mAP=\frac{1}{C}\sum _{i=1}^{C}A{P}_i$$

  (8)

  where $A{P}_i$ is the average precision for the i-th class and C is the total number of classes. $AP$ corresponds to the area under the PR curve and is calculated as shown in Equation (9):

  $$AP=\sum _{i=1}^{n-1}(r_{i+1}-r_i)p_{interp}\left(r_{i+1}\right)$$

  (9)

  where $r_1,r_2,...,r_n$ and $p_{interp}$ is the recall ratio and precision ratio when the n-th threshold is set. Usually, $A{P}_{50}$ and $A{P}_{75}$ represents the AP (single class) calculated with the IoU set as 0.5 and 0.75, respectively. $A{P}_{50}^{95}$ is the average value of AP with the IoU ranging from 0.5 to 0.95 with a step of 0.05. $mA{P}_{50}$, $mA{P}_{75}$, and $mA{P}_{50}^{95}$, on the other hand, represent the average APs of all classes at different settings of the IoU;
• Besides mean average precision (mAP), precision, recall, and the F1-score are also common criteria for model evaluation. The computations are as follows:

  $$\mathrm{Precision}=\frac{TP}{TP+FP}$$

  (10)

  $$\mathrm{Recall}=\frac{TP}{TP+FN}$$

  (11)

  $$\mathrm{F}1-\mathrm{score}=\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

  (12)

  where $TP$, $FP$, and $FN$ represent true positive, false positive, and false negative, respectively;
• The precision–recall curve (PR curve) plots the precision against the recall rate. As the confidence index is set to a smaller value, the recall increases, signaling more objects to be detected, while the precision decreases since false positive objects increase. The PR curve is useful to visualize the performance of a trained model. A perfect score would be at $(1,1)$, where both precision and recall is at 1. Therefore, if a curve bows towards $(1,1)$, the model performs very well;
• Number of frames per second (FPS) and inference time (in milliseconds) to measure the processing speed.

5. Results and Discussion

The detection performance for each dataset (VAID and COWC) is evaluated and discussed in this section. We trained the datasets with three different input sizes ($416\times 416$, $512\times 512$, and $608\times 608$) for all detectors. The results of each detector are tabulated, and the precision–recall curve is plotted. Besides, we provide the inference time collected on different hardware. Then, we provide a general discussion of the performances of both datasets.

5.1. VAID Dataset

The detection results of the different models are shown in Table 6. We performed experiments on three different input sizes ($416\times 416$, $512\times 512$, and $608\times 608$), and the individual classes performance was recorded. We remind that the vehicle size of the objects in the images is usually $20\times 40$ px, as shown in Table 4.

Table 6. Detection result on the VAID dataset. The individual class results are their respective $A{P}_{50}$. Numbers in red, blue, and green represent the best results for the input sizes of $416\times 416$, $512\times 512$, and $608\times 608$, respectively.

Model	Input Size	$\mathit{m}\mathit{A}{\mathit{P}}_{50}$	Precision	Recall	F1 Score	Sedan	Minivan	Truck	Pickup Truck	Bus	Cement Truck	Trailer
Faster R-CNN	$416\times 416$	0.8711	0.7859	0.8068	0.7962	0.5390	0.6668	0.4668	0.3838	0.7134	0.5178	0.5746
Faster R-CNN	$512\times 512$	0.9020	0.8283	0.7813	0.8041	0.5891	0.7268	0.5209	0.4482	0.7441	0.5966	0.6199
Faster R-CNN	$608\times 608$	0.9219	0.8578	0.7657	0.8091	0.6203	0.7289	0.5690	0.5154	0.7693	0.6629	0.6823
YOLOv2	$416\times 416$	0.8017	0.7920	0.6971	0.7415	0.6874	0.9215	0.6733	0.6693	0.9462	0.8513	0.8630
YOLOv2	$512\times 512$	0.8764	0.8396	0.8022	0.8205	0.7951	0.9619	0.8118	0.7942	0.9601	0.9144	0.8970
YOLOv2	$608\times 608$	0.9139	0.8643	0.8626	0.8634	0.8592	0.9805	0.8723	0.8658	0.9721	0.9127	0.9350
YOLOv2-Tiny	$416\times 416$	0.6147	0.4887	0.5686	0.5256	0.4373	0.7750	0.3992	0.4169	0.8643	0.6975	0.7128
YOLOv2-Tiny	$512\times 512$	0.7269	0.5896	0.6926	0.6369	0.5927	0.8629	0.6110	0.5394	0.9465	0.6980	0.8375
YOLOv2-Tiny	$608\times 608$	0.8187	0.6600	0.7822	0.7159	0.7154	0.9664	0.6810	0.6823	0.9669	0.8588	0.8601
YOLOv3	$416\times 416$	0.9675	0.9374	0.9789	0.9577	0.9753	0.9819	0.9734	0.9634	0.9825	0.9446	0.9512
YOLOv3	$512\times 512$	0.9650	0.9434	0.9828	0.9627	0.9857	0.9871	0.9727	0.9670	0.9737	0.9271	0.9419
YOLOv3	$608\times 608$	0.9697	0.9495	0.9849	0.9669	0.9861	0.9849	0.9757	0.9697	0.9808	0.9370	0.9539
YOLOv3-Tiny	$416\times 416$	0.8491	0.8192	0.8566	0.8375	0.8415	0.9752	0.8359	0.7634	0.9631	0.7038	0.8610
YOLOv3-Tiny	$512\times 512$	0.8702	0.8579	0.8867	0.8721	0.9025	0.9663	0.8815	0.8028	0.9688	0.7174	0.8521
YOLOv3-Tiny	$608\times 608$	0.9486	0.8975	0.9436	0.9200	0.9471	0.9891	0.9413	0.9230	0.9708	0.9165	0.9524
YOLOv4	$416\times 416$	0.9702	0.9400	0.9843	0.9617	0.9854	0.9954	0.9760	0.9712	0.9762	0.9355	0.9519
YOLOv4	$512\times 512$	0.9708	0.9418	0.9886	0.9646	0.9865	0.9871	0.9730	0.9796	0.9837	0.9283	0.9571
YOLOv4	$608\times 608$	0.9743	0.9438	0.9883	0.9655	0.9860	0.9851	0.9785	0.9793	0.9911	0.9450	0.9555
YOLOv4-Tiny	$416\times 416$	0.8715	0.9673	0.5651	0.7134	0.7047	0.9823	0.8233	0.7411	0.9821	0.9271	0.9403
YOLOv4-Tiny	$512\times 512$	0.9370	0.9643	0.8655	0.9122	0.9041	0.9858	0.9281	0.9120	0.9801	0.9038	0.9452
YOLOv4-Tiny	$608\times 608$	0.9605	0.9588	0.9415	0.9501	0.9551	0.9843	0.9687	0.9507	0.9848	0.9264	0.9533
Ours	$416\times 416$	0.8715	0.9646	0.5686	0.7154	0.7160	0.9858	0.8332	0.7564	0.9761	0.9101	0.9230
Ours	$512\times 512$	0.9398	0.9658	0.8714	0.9162	0.9233	0.9781	0.9385	0.9060	0.9862	0.9148	0.9314
Ours	$608\times 608$	0.9605	0.9602	0.9433	0.9517	0.9642	0.9833	0.9668	0.9488	0.9898	0.9272	0.9414

From Table 6, it is observed that a larger input size would increase the detection accuracy. YOLOv2 and YOLOv2-Tiny provided the lowest detection accuracy among all. For the input size of $416\times 416$, the best result was given by YOLOv4 ($mA{P}_{50}=0.9702$), followed by YOLOv3 ($mA{P}_{50}=0.9675$) and our proposed model ($mA{P}_{50}=0.8715$). For the input size of $512\times 512$, the best result was given by YOLOv4 ($mA{P}_{50}=0.9708$), followed by YOLOv3 ($mA{P}_{50}=0.9650$) and our proposed model ($mA{P}_{50}=0.9398$). For the input size of $608\times 608$, the best result was given by YOLOv4 ($mA{P}_{50}=0.9743$), followed by YOLOv3 ($mA{P}_{50}=0.9697$) and our proposed model ($mA{P}_{50}=0.9605$). When comparing with the tiny versions of the YOLO models (YOLOv4-Tiny, YOLOv3-Tiny, and YOLOv2-Tiny), we observed that our model led in terms of detection accuracy. Our proposed model achieved similar or better results than YOLOv4-Tiny in all three input sizes. Our model always achieved higher precision than the other models (except for the input size of $416\times 416$), revealing that it can reduce the number of false positives. Regarding detection accuracy on individual classes, we found that almost all classes had significant improvement over YOLOv4-Tiny, with an increase in accuracy within 1–4%. This was due to the added finer YOLO detection head, where it can effectively detect smaller objects. Interestingly, we observed that our model performed better than YOLOv4 for the “bus” object with the input size of $512\times 512$.

Figure 11 shows the precision–recall curve for all models of various input sizes. It was confirmed that our model outperformed almost all models, except YOLOv4 and YOLOv3. Our model provided a good behavior of the precision–recall balance, as illustrated in Figure 11.

Figure 11. Precision–recall curve for all models trained on the VAID dataset.

The detection results of the various detectors are shown in Figure 12. We took two sample frames from the test set and the inference on all detectors. We selected the frame that had the most classes of objects. There are two small “sedans” on the top left corner and top right corner in the first row of the figure. Besides, there is an occluded “pickup truck” at the bottom right corner. These three objects are the key to differentiate the effectiveness of different detectors. We observed that Faster R-CNN, YOLOv2-Tiny, and YOLOv4 successfully located all the objects. However, Faster R-CNN and YOLOv2-Tiny misclassified some objects, while YOLOv4 classified all the objects correctly with a high level of confidence. Other detectors failed to locate one or two of the small objects, as described before. As for our proposed model, it failed to identify the “sedan” on both the top left corner and top right corner. A similar detection was observed in YOLOv4-Tiny, except that our model produced a detection with a higher level of confidence. Compared to Faster R-CNN, our model did not have multiple bounding boxes on a single object, signaling that the NMS was working as intended. For the image in the second row, there is much background noise, where trees and parks are the main noise that could confuse the models. It was observed that Faster R-CNN was confused with “cement truck” and “truck”. Compared to the first image, Faster R-CNN produced less classification error, and no stacking of the bounding box was observed. As for our model, it did not classify the minivan, while other objects were detected and classified correctly.

Figure 12. Detection results of various detectors (input size of $512\times 512$) on the VAID dataset.

Table 7 shows an in-depth exploration of our models in terms of the computational efficiency.

Table 7. Comparison in terms of the model size, computational requirements, and inference time on different hardware for input size of $512\times 512$ for the VAID dataset. The number in bold denotes the best result.

Model	Model Size (MB)	BFLOPs	Number of Parameters	${\mathit{mAP}}_{50}$	Inference Time (ms)			Frame Per Second (FPS)
					Tesla T4	Quadro P5000	Jetson Nano	Tesla T4	Quadro P5000	Jetson Nano
Faster R-CNN	330	134.40	41.80 M	0.9020	40.08	56.23	779.98	25.0	17.8	1.3
YOLOv2	202	44.45	50.61 M	0.8764	14.84	10.85	412.70	67.4	92.2	2.4
YOLOv2-Tiny	44.2	8.10	11.05 M	0.7269	5.22	3.64	199.04	191.6	274.7	4.1
YOLOv3	246	98.99	61.61 M	0.9650	28.85	23.28	735.55	34.7	42.96	1.4
YOLOv3-Tiny	34.8	8.27	8.69 M	0.8702	5.23	3.65	199.20	191.2	274.0	5.0
YOLOv4	256	90.29	64.04 M	0.9708	29.08	26.48	861.87	34.4	37.8	1.2
YOLOv4-Tiny	23.6	10.30	5.89 M	0.9370	5.83	3.27	80.56	71.5	305.8	12.4
Ours	5.8	7.77	1.44 M	0.9398	5.67	4.67	80.98	176.4	214.1	12.3

As observed in Table 7, even though our proposed model took a little bit longer than YOLOv2-Tiny, YOLOv3-Tiny, and YOLOv4-Tiny on the Tesla T4 and Quadro P5000 and the model size and number of parameters were smaller, mAP was higher than them. In terms of edge hardware inferencing, our model could achieve around an 81 ms inference time on the Jetson Nano, thanks to its relatively small floating-point operations (FLOPs).

The mAP versus inference time plot is shown in Figure 13. It is observed that our model used almost the same inference time, but achieved slightly higher detection accuracy than YOLOv4-Tiny.

Figure 13. Performance of various detectors on the Tesla T4, Quadro P5000, and Jetson Nano 2 GB. The graph is plotted with the inference time collected from both input sizes of $416\times 416$ and $512\times 512$ for all detectors. Note that YOLOv4 with the input size of $608\times 608$ failed to run on the Nvidia Jetson Nano 2 GB.

5.2. COWC Dataset

The second dataset used in our study was the COWC dataset. Table 8 provides the detection results on various models trained on two different input sizes, namely $416\times 416$ and $512\times 512$.

Table 8. Detection results on the COWC dataset. Numbers in blue and red represent the best result for input sizes of $512\times 512$ and $416\times 416$ respectively.

Model	Input Size	$\mathit{m}\mathit{A}{\mathit{P}}_{50}$	Precision	Recall	F1 Score	Sedan	Pickup	Other	Unknown
Faster R-CNN	$416\times 416$	0.8417	0.8339	0.6259	0.7150	0.7284	0.7008	0.6984	0.4942
Faster R-CNN	$512\times 512$	0.8921	0.8902	0.6393	0.7442	0.7534	0.7689	0.7326	0.5868
YOLOv2	$416\times 416$	0.6180	0.7202	0.6751	0.6969	0.7157	0.6975	0.6783	0.3804
YOLOv2	$512\times 512$	0.6966	0.7644	0.7687	0.7665	0.8244	0.6462	0.7606	0.5551
YOLOv2-Tiny	$416\times 416$	0.3963	0.4338	0.5684	0.4921	0.4971	0.4242	0.4372	0.2266
YOLOv2-Tiny	$512\times 512$	0.5222	0.4979	0.6775	0.5740	0.6881	0.5854	0.5437	0.2717
YOLOv3	$416\times 416$	0.9383	0.8953	0.9536	0.9235	0.9606	0.9347	0.9674	0.8905
YOLOv3	$512\times 512$	0.9375	0.9116	0.9568	0.9337	0.9602	0.9093	0.9678	0.9127
YOLOv3-Tiny	$416\times 416$	0.8197	0.7883	0.8795	0.8314	0.9111	0.7869	0.8758	0.7050
YOLOv3-Tiny	$512\times 512$	0.8609	0.8006	0.9055	0.8498	0.9104	0.8699	0.9177	0.7456
YOLOv4	$416\times 416$	0.9529	0.8757	0.9756	0.9230	0.9727	0.9208	0.9880	0.9299
YOLOv4	$512\times 512$	0.9597	0.8832	0.9731	0.9260	0.9737	0.9585	0.9778	0.9286
YOLOv4-Tiny	$416\times 416$	0.9068	0.8748	0.9389	0.9057	0.9566	0.8430	0.9622	0.8654
YOLOv4-Tiny	$512\times 512$	0.9347	0.8787	0.9552	0.9153	0.9626	0.9153	0.9659	0.8951
Ours	$416\times 416$	0.9299	0.8772	0.9422	0.9085	0.9589	0.9179	0.9670	0.8759
Ours	$512\times 512$	0.9353	0.8853	0.9552	0.9189	0.9800	0.9155	0.9655	0.8804

From Table 8, it is observed that YOLOv4 topped the $mA{P}_{50}$ for both input sizes of $416\times 416$ and $512\times 512$. YOLOv3 came in second, and our proposed model came in third in terms of $mA{P}_{50}$. In terms of precision, YOLOv3 achieved the highest precision among all, followed by our model and YOLOv4. This is different from the VAID dataset, where our proposed model topped in terms of precision. This was mainly due to the dataset used, where COWC seems to have objects with a larger size ($24\times 48$) than VAID ($20\times 40$). Recall that we replaced the YOLO output with a larger size to accommodate the finer detection of smaller sizes; therefore, our model was not able to capture the slightly larger objects well. When comparing the tiny version of YOLO, our model topped in terms of almost all metrics. Our model achieved the highest $mA{P}_{50}$, precision, and F1 score. Faster R-CNN provided lower detection accuracy since it was not designed to detect small objects. Even though the anchor size was adapted according to the training set, they were not built to detect small objects and could not yield good detection results. As for individual class $mA{P}_{50}$, we observed that our model outperformed the others for the “sedan” class for the input size of $512\times 512$, thus revealing that our model could detect smaller objects in the image since the only class in the dataset with a consistently small object size was “sedan”. Note that the size of the “sedan” in this dataset was smaller than the other classes. As for other objects, YOLOv4 yielded the highest detection accuracy. When comparing with YOLOv4-Tiny, our model outperformed it in almost all classes. In short, YOLOv4 strikes for higher detection accuracy, YOLOv3 for precision detection, and our proposed model for the balance between detection accuracy and inference time.

We provide the precision–recall curve for all the models in Figure 14. We can once again confirm that our model outperformed almost all models, except the huge YOLOv4 and YOLOv3.

Figure 14. Precision-recall curve for all models trained on the COWC dataset.

The detection results of the various detector are illustrated in Figure 15. We took three sample frames from the test set and inference on all the trained detectors of input size $512\times 512$. As shown in the first row, the image is partially occluded with four objects that are “hidden” in the shadow and contains multiple objects in the upper right corner. Our model only detected three out of four occluded objects, while YOLOv4-Tiny successfully detected all four occluded objects. As for the upper right corner section, it was observed that Faster R-CNN generated many bounding boxes, but did not classify most of the boxes correctly. The NMS in our model successfully suppressed many unwanted bounding boxes and was able to classify most of the objects correctly. Compared to YOLOv4 and YOLOv4-Tiny, our model could identify the classes more accurately. In the second and third rows, almost all models could detect and classify the object correctly. It was observed that Faster R-CNN, YOLOv3, YOLOv4, and our model correctly classified most of the objects with high confidence.

Figure 15. Detection results of various detectors (input size of $512\times 512$) on the COWC dataset.

Our model was designed to strike a balance between detection accuracy and inference speed. In Table 9, we report the comparison between our proposed model against all other models of input size $512\times 512$ on the Tesla T4, Quadro P5000, and Jetson Nano in terms of the model size, billion floating-point operations (BFLOPs), number of parameters, inference time, and FPS. Our proposed model consumed the lowest disk space, with a model size of only 5.5 MB, only 23% of YOLOv4-Tiny. Our model required the lowest computational resource, coming in with only 7.73 BFLOPs and 1.43 M parameters. The detection accuracy was ranked second after the huge YOLOv4. In terms of inference time, YOLOv2-Tiny and YOLOv3-Tiny took a shorter time to perform inference on the Tesla T4 and Quadro P5000. However, the critical study of this work was to run inference on low-cost embedded hardware. We used the Jetson Nano 2 GB as our low-cost embedded hardware, whereby we observed that our model took the lowest time to compute, which took only 78.33 ms, equivalent to around 13 FPS. With 13 FPS, near-real-time results were obtained. We include the detection accuracy vs. inference time graph for all three GPUs in Figure 16.

Table 9. Comparison in terms of the model size, computational requirements, and inference time on different hardware for the input size of $512\times 512$ for the COWC dataset. The number in bold represents the best result.

Model	Model Size (MB)	BFLOPs	Number of Parameters	${\mathit{mAP}}_{50}$	Inference Time (ms)			Frame Per Second (FPS)
					Tesla T4	Quadro P5000	Jetson Nano	Tesla T4	Quadro P5000	Jetson Nano
Faster R-CNN	330	134.40	41.80 M	0.8921	42.12	45.77	1123	23.7	21.8	0.9
YOLOv2	202	44.45	50.59 M	0.6966	12.98	12.78	440.21	77.0	78.2	2.3
YOLOv2-Tiny	44.2	8.10	11.04 M	0.5222	4.98	5.61	197.54	200.8	178.3	5.1
YOLOv3	256	98.96	61.59 M	0.9375	16.60	24.63	802.03	60.2	40.6	1.2
YOLOv3-Tiny	34.7	8.26	8.68 M	0.8609	5.06	5.66	199.90	197.6	176.7	5.0
YOLOv4	244	90.26	64.02 M	0.9597	20.24	26.71	846.78	49.4	37.4	1.2
YOLOv4-Tiny	23.5	10.29	5.89 M	0.9347	5.67	14.67	86.21	176.4	68.2	11.6
Ours	5.5	7.73	1.43M	0.9353	5.33	13.89	78.33	187.6	72.0	12.8

Figure 16. Performance of various detectors on the Tesla T4, Quadro P5000, and Jetson Nano 2 GB. The graph is plotted with the inference time collected from both input sizes of $416\times 416$ and $512\times 512$ for all detectors.

From both of the experiments, we can conclude that our model struck a balance between detection accuracy and inference speed. Our model outperformed YOLOv4-Tiny on both of the datasets, despite requiring only 5.5 MB of storage to store the model. This is particularly useful when deploying on edge devices, where storage and processing power are usually the constraint. Apart from that, with the reduction of the FLOPs, we saw a minimal affect on the detection accuracy. An in-depth ablation study and our design of the proposed model is available in Appendix A.

We observed that for both datasets suffering from class imbalance (“minivan” and “cement truck” in the VAID dataset and “pickup” in COWC), YOLOv4, YOLOv4-Tiny, and our proposed model could still achieve competitive $mA{P}_{50}$, while other models did not perform any better. Even though we tried to include augmentation to increase the number of objects for severely under-represented classes, their results stayed almost the same, since augmentation increased the objects in the classes; however, there was still a lack of representative features of the targeted classes. A possible way to solve this issue is through a resampling process, oversampling infrequent classes from the minority classes to match the quantity of the majority classes. However, we did not explore this as YOLOv4, YOLOv4-Tiny, and our proposed model did not suffer from lower $mA{P}_{50}$ on minority classes’ prediction.

6. Conclusions

We proposed an improved version of a one-stage detector, coined YOLO-RTUAV, to detect small objects in aerial images in this work. Our model was built based on YOLOv4-Tiny, specifically aimed at near-real-time inference of small objects on edge devices. Essentially, our model is lightweight with only 5.5 MB, and experiments on the Jetson Nano 2 GB showed the ability to achieve up to 13 FPS for an input size of $512\times 512$. Experiments conducted on two datasets, namely VAID and COWC, illustrated that our proposed model outperformed YOLOv4-Tiny in terms of inference time and accuracy. Our model did not perform better than the more complicated YOLOv4. However, we believe that our model struck a balance between accuracy and inference time.

While YOLO-RTUAV provided promising results for the datasets used, some issues still exist and call for further research. Firstly, YOLO-RTUAV only focuses on small objects in aerial images. The performance on medium and large objects remains unknown since the datasets used mainly contain small objects. YOLO-RTUAV is not suitable for objects with a wide range of sizes. This bottleneck is solvable with the help of more YOLO layers with various output sizes, but it comes with the cost of more computational power requirements. Secondly, our experiment on the COWC dataset showed that our model can deal with RGB and grayscale images, and objects occluded by shadows remained detectable. However, such an assumption is limited to the COWC dataset, and more experiments are required to validate our claim. Therefore, datasets with different occlusions and background noise in the training and test datasets should be considered in future work. Thirdly, the reduction of the parameters using a $1\times 1$ convolutional layer showed a slight decrease in the detection accuracy. It would be interesting to explore other techniques, which reduce the model complexity, but retain the same detection accuracy. Finally, YOLO models are highly dependent on anchor boxes to produce predictions. This is a known problem whereby, to achieve optimum detection accuracy, a clustering analysis is required to determine a set of optimum anchors before model training. In addition, the usage of anchors increases the complexity of the detection heads. We suggest exploring the possibility of anchor-free prediction, such as YOLOX [66].