Joint Optimization Loss Function for Tiny Object Detection in Remote Sensing Images

Abstract

Tiny object detection remains a formidable challenge in the field of computer vision. There are many factors that influence tiny object detection performance. In this paper, we focus primarily on the following two aspects. First, due to diminutive size and inappropriate label assignment strategy, tiny objects yield significantly fewer positive samples than larger objects, resulting in weakened supervisory signals during backpropagation and model training. Second, most existing detectors directly combine the classification loss and bounding box regression loss during training. Some improvement methods focus exclusively on either classification or localization, leading to potential discrepancies in which predictions exhibit precise localization but incorrect classifications or accurate classifications with imprecise localization. To address these issues, we propose a novel Joint Optimization Loss (JOL) that dynamically assigns optimal weights to each training sample, enabling joint optimization of both the classification and regression losses. Notably, JOL integrates seamlessly with most mainstream detectors and loss functions without requiring alterations to network architectures. Extensive experiments conducted on five benchmark datasets demonstrate the superior performance of our approach, achieving AP improvements of 1.7 and 1.5 points on the AI-TOD and SODA-D datasets, respectively, compared to the state-of-the-art method.

1. Introduction

In recent years, Artificial/intelligence (AI) applications have progressively permeated various facets of daily life, including classification of agricultural crops [1], text detection [2], semantic segmentation [3], and urban mapping [4]. In the field of remote sensing, related applications include water body detection [5,6], river islets categorization [7], and more. Today, object detection has become one of the most widely used technologies in such applications. The advent of neural networks and deep learning has significantly enhanced detection accuracy and processing speed compared to traditional methods. In particular, the development of residual network has enabled the effective training of deeper and more complex neural networks, yielding substantial accuracy improvements. Recently proposed detectors such as Co-DETR [8], InternImage [9], MoCaE [10], and EVA [11] have demonstrated state-of-the-art performance on the MS COCO 2017 dataset [12].

Despite the substantial advancements in general object detection, detecting tiny objects remains a challenging task in computer vision. Comprehensive reviews on tiny object detection, such as those presented in [13,14], highlight three primary challenges: information loss, noise interference, and insufficient training samples. Consequently, directly applying general object detectors to tiny object detection tasks often results in a marked decline in accuracy.

In recent years, increasing research attention has been directed towards enhancing tiny object detection performance. KLDet [15] proposed modeling each bounding box as a 2D Gaussian distribution, employing the Kullback–Leibler divergence (KLD) in place of the traditional IoU metric for label assignment. Similarly, GaussianAssignment [16] uses the 2D Gaussian distribution to allocate higher-quality samples to tiny objects, while HANet [17] introduced hierarchical activation to reduce information confusion. ORFENet [18] and ESG_TODNet [19] have further advanced tiny object detection by adding feature enhancement modules.

Our analysis reveals two primary factors that hinder the improvement of tiny object detection performance. First, as is well-known, the core process of model training is backpropagation, which directly influences parameter updates. This process fundamentally relies on the model’s loss function. Figure 1a illustrates the relationship between each object’s absolute size and the proportion of loss value that it contributes to the total loss. In essence, objects with a higher loss proportion receive stronger supervisory signals. Notably, there is a substantial disparity between the proportions assigned to tiny objects versus larger ones, which is a key factor contributing to the lower detection accuracy of tiny objects compared to larger counterparts. This also underscores why label assignment strategies such as DotD [20], NWD [21], and RFLA [22] can improve tiny object detection accuracy by allocating more positive samples to these objects.

Second, most existing detectors treat the classification loss and bounding box regression loss as separate entities, which can lead to an imbalance between classification and localization tasks. Figure 1b–d illustrates the relationship between the classification score and bounding box regression score for each sample, showing that certain samples achieve high classification scores yet suffer from low-quality regression predictions, while others display accurate localization but low classification scores. This imbalance adversely affects model training and detection performance. Although methods such as YOLOv8 [25] and TOOD [26] take both the classification and regression scores into account, they primarily focus on label assignment and may overlook challenging prediction samples.

In this paper, we introduce a novel Joint Optimization Loss (JOL) designed to effectively address the aforementioned challenges in tiny object detection. First, in order to alleviate the problem of tiny objects having a much smaller proportion of the loss function during model training compared to larger objects, our JOL calculates classification and bounding box regression scores, which indicate the prediction quality of each sample. It then dynamically assigns weights to training samples based on these scores, with less accurately predicted samples receiving higher weights in order to draw the model’s attention to these challenging cases. Additionally, to alleviate the phenomenon of imbalance performance between classification and bounding box regression tasks in tiny object detection, we use the product of the classification score and bounding box regression score to evaluate the performance of each sample, which can simultaneously optimize both classification and bounding box regression tasks. Notably, our proposed method integrates seamlessly with most existing detectors and loss functions, requiring no modifications to network architectures. The main contributions of this paper are summarized as follows:

• We propose a novel Joint Optimization Loss (JOL) for tiny object detection which dynamically assigns appropriate weights to each sample in order to address the imbalance between classification and bounding box regression.
• The proposed method can be flexibly integrated with most existing detectors and loss functions without modifying the architecture of the model.
• We conducted extensive experiments on four tiny object detection datasets and one general object detection dataset to demonstrate the effectiveness of the proposed method.

The remainder of this study is organized as follows: Section 2 presents an overview of general object detection and tiny object detection algorithms along with a review of relevant work on classification and bounding box regression loss functions used in object detection tasks; Section 3 provides a comprehensive description of our proposed Joint Optimization Loss, including a visual comparison between our method and the standard loss function; Section 4 offers a detailed explanation of the datasets and implementation details used in the experiments followed by an in-depth analysis of the experimental results; Section 5 concludes the paper with a brief summary.

2. Related Work

2.1. Object Detection

Based on the absolute size of the objects, object detection tasks can be broadly categorized into general object detection and tiny object detection.

General Object Detection: Mainstream general object detectors are typically classified into two-stage and one-stage detectors based on their architecture. Two-stage detectors first generate a series of proposal regions using region proposal algorithms or networks, then classify and regress these regions to produce the final detection results. Representative two-stage detectors include R-CNN [27], Fast R-CNN [28], Faster R-CNN [24], and Mask R-CNN [29]. To achieve faster detection speeds, one-stage detectors have been developed that bypass the proposal region step and instead predict results directly for the entire image. One-stage detectors can be further classified as either anchor-based (such as RetinaNet [30], SSD [31], and YOLOv3 [32]) or anchor-free (such as FCOS [33], YOLOX [34], and CenterNet [35]) depending on whether they use anchors for bounding box regression.

Tiny Object Detection: In recent years, considerable research has been dedicated to enhancing the detection performance of small and tiny objects, particularly in high-resolution images captured from satellites and Unmanned Aerial Vehicles (UAVs), which often contain numerous densely distributed tiny objects. A common approach involves cropping the original high-resolution image into smaller patches, enlarging these patches, then detecting objects within each patch individually. Studies such as ClusDet [36], DMNet [37], CRENet [38], and CDMNet [39] have followed this method, achieving substantial improvements. Another effective approach to improve detection accuracy involves designing optimized label assignment strategies. For example, DotD [20] introduced the dot distance to replace the IoU metric for evaluating bounding box matching. NWD [21] and RFLA [22] model each bounding box as a 2D Gaussian distribution, then respectively employ the normalized Gaussian Wasserstein distance and Kullback–Leibler divergence to calculate match scores. Additionally, SimD [40] provides a novel evaluation metric that comprehensively considers both location and shape similarity. Other studies have explored effective approaches such as context modeling [41,42], attention mechanisms [43,44,45], feature fusion [46,47], and super-resolution [48,49]. Foundation models have also made significant progress, particularly in the field of remote sensing object detection. DynamicVis [50] uses a dynamic regional perception architecture and novel pretraining paradigm, achieving impressive performance on multiple remote sensing image tasks. In addition, recent reviews on tiny object detection in infrared and hyperspectral images [51,52] can also provide some inspiration thanks to their full utilization of contextual information and data characteristics.

2.2. Loss Functions for Object Detection

The loss function is a critical component in training object detectors, as it directly influences their backpropagation performance. Object detection generally involves two primary tasks: determining the locations of objects within an image, and classifying each object into its respective category.

Classification Loss: The classification loss is extensively used in tasks such as object classification, object detection, and semantic segmentation. The binary cross-entropy loss is inspired by information entropy, and is commonly applied in binary classification problems. For multiclass classification problems, using the cross-entropy loss with a softmax function allows model predictions to approximate the true category while maintaining distance from incorrect categories. Considering the issue of class imbalance, which is particularly prevalent in one-stage object detectors, Lin et al. [30] proposed the focal loss, which downweights the influence of easily classified samples and emphasizes the weights of more challenging samples. Subsequent methods inspired by focal loss include GFL [53] and VarifocalNet [54], among others.

Bounding Box Regression Loss: Accurately determining the locations of target objects is a critical aspect of object detection that is directly related to the precision of detectors. Existing bounding box regression loss functions generally fall into two categories: distance-based loss functions such as the L1, L2, and smooth L1 loss [28], and IoU-based loss functions such as the IoU [55], GIoU [56], DIoU [57], CIoU [57], and EIoU [58] loss. The L1 and L2 losses respectively compute the L1 and L2 distances between the predicted and target offsets as the bounding box regression loss value. The smooth L1 loss [28] combines their strengths by maintaining a moderate gradient when the difference is small while also preventing gradient explosion when the difference is large. Based on the recognition that bounding box coordinates are interdependent, a series of IoU-based loss functions have also been proposed. The IoU loss [55] employs the negative natural logarithm of the IoU between the predicted and target bounding boxes as the loss value. The GIoU loss [56] introduces an additional term to consider the ratio of the difference set and the minimum enclosing rectangle based on the IoU. The DIoU loss [57] addresses the limitations of constant IoU and GIoU values when a bounding box is fully enclosed within another by considering both the overlap and center point distance. The CIoU loss [57] further incorporates an aspect ratio based on the DIoU, while the EIoU loss [58] addresses aspect ratio issues by integrating the width and height of the predicted and target bounding boxes.

3. Method

3.1. Joint Optimization Loss (JOL)

The core operation when using deep learning to train a model consists of gradient updating during backpropagation, which requires a loss function. Therefore, the model will be more inclined to accurately predict objects that have a larger proportion of the loss values in the total loss function. This is one of the main reasons for the detection accuracy of tiny objects being much lower than larger ones. From this perspective, researchers have proposed label assignment strategies that are more suitable for tiny object detection by increasing the number of positive samples obtained by tiny objects and improving their proportion in the model loss function, with the goal of realizing improved detection performance.

In this paper, we adopt a more fundamental approach by directly controlling the weights of each training sample. The pipeline of our proposed Joint Optimization Loss (JOL) is illustrated in Figure 2. Prior to introducing our method, we first present a simple review of the base loss function used for object detection. In most object detectors, the total loss function consists of two parts: bounding box regression loss and classification loss. We define ${\mathcal{L}}_{base}$ as the total base loss function, as shown in Equation (1):

$${\mathcal{L}}_{base}={\sum }_{i=1}^N\left[{\mathbb{1}}_i^{pos}\left({\mathcal{L}}_i^{loc}+{\mathcal{L}}_i^{cls}\right)+{\mathbb{1}}_i^{neg}{\mathcal{L}}_i^{cls}\right]$$

(1)

where N denotes the number of samples in the batch and where ${\mathcal{L}}_i^{loc}$ and ${\mathcal{L}}_i^{cls}$ represent the base losses for bounding box regression and classification of the i-th sample, respectively. For example, in Faster R-CNN [24], the base loss functions for bounding box regression and classification are the L1 loss and cross-entropy loss, respectively. In addition, ${\mathbb{1}}_i^{pos}$ and ${\mathbb{1}}_i^{neg}$ are gating identifiers that distinguish between positive and negative samples. If the i-th sample is positive, then the loss function includes both classification loss and bounding box regression loss, meaning that ${\mathbb{1}}_i^{pos}$ and ${\mathbb{1}}_i^{neg}$ are equal to 1 and 0, respectively. If the i-th sample is negative, however, the loss function only includes the classification loss, meaning that ${\mathbb{1}}_i^{pos}$ and ${\mathbb{1}}_i^{neg}$ are equal to 0 and 1, respectively.

Our analysis and observations have identified two primary weaknesses in the existing base loss function. First, as defined in Equation (1), the current total loss function assigns equal weights to both positive and negative samples, simply combining the bounding box regression loss and classification loss to form the overall loss. This approach implicitly assumes that each sample holds equal importance for the model, which may not accurately reflect their differing contributions. Second, while several studies have proposed improved loss function calculations such as the focal loss [30] and IoU loss [55], these methods typically focus on either classification or bounding box regression in isolation. We contend that this approach may be suboptimal, as the significance of each sample should be dynamically adjusted based on its contribution to the model’s overall performance. Moreover, jointly considering both classification and bounding box regression when calculating the weights may yield better results, as accurate predictions require both precise localization and correct classification.

Based on our observations and analysis, we propose the novel Joint Optimization Loss (JOL) for tiny object detection. We define the JOL as ${\mathcal{L}}_{JOL}$, shown in Equation (2):

$${\mathcal{L}}_{JOL}={\sum }_{i=1}^N\left[{\mathbb{1}}_i^{pos}{w}_i^{+}\left({\mathcal{L}}_i^{loc}+{\mathcal{L}}_i^{cls}\right)+{\mathbb{1}}_i^{neg}{w}^{-}{\mathcal{L}}_i^{cls}\right]$$

(2)

where $w_i^{+}$ and $w^{-}$ denote the positive and negative weights for the i-th sample, shown in Equation (3) and Equation (4), respectively; the definitions of the other parameters and symbols are as the same as in Equation (1):

$$w_i^{+}=\mathit{\lambda }-S_i^{loc}\times S_i^{cls},$$

(3)

$$w^{-}={\displaystyle \frac{{\sum }_{i=1}^N{\mathbb{1}}_i^{pos}{w}_i^{+}}{{\sum }_{i=1}^N{\mathbb{1}}_i^{pos}}},$$

(4)

where $S_i^{loc}$ and $S_i^{cls}$ respectively represent the bounding box regression score and classification score for the i-th sample in a certain batch. They can be calculated by any metric that can evaluate the matching degree between the prediction and target, such as the IoU for bounding box regression. The $\mathit{\lambda }$ parameter controls the scale of the weights; as $S_i^{loc}$ and $S_i^{cls}$ usually range from zero to one, $\mathit{\lambda }$ should be at least equal to one. Based on our observations, the model’s accuracy tends to stabilize when $\mathit{\lambda }$ exceeds two; detailed experimental verification can be found in the second part of the ablation experiments described in Section 4.4. Therefore, we set $\mathit{\lambda }$ to two for the remainder of this study. To balance the weights of positive and negative samples, we use the mean weight of all positive samples in the batch as the negative weight.

3.2. Visualization Analysis

To provide a more intuitive understanding of our method, we perform a visual analysis comparing the base loss function with our proposed Joint Optimization Loss. For demonstration purposes, we assume that the base loss functions for classification and bounding box regression are the cross-entropy loss and IoU loss [55], respectively. Additionally, we focus solely on the loss function for positive samples. Thus, the base loss function for a given sample is shown in Equation (5):

$${\mathcal{L}}_{base}^{\ast }=-log\left(p\right)+1-iou$$

(5)

where p denotes the predicted classification score and $iou$ represents the Intersection over Union (IoU) between the predicted bounding box and the ground-truth bounding box.

In our proposed Joint Optimization Loss (JOL), we use p and $iou$ to represent the classification score $S^{cls}$ and bounding box regression score $S^{loc}$, respectively. Accordingly, the formula for JOL is provided in Equation (6):

$${\mathcal{L}}_{JOL}^{\ast }=\left(2-p\times iou\right)\times \left(-log\left(p\right)+1-iou\right).$$

(6)

To visually demonstrate the similarity and difference between Equations (5) and (6), Figure 3 shows a 3D graph with p and $iou$ as independent variables. It is obvious that the loss values of ${\mathcal{L}}_{base}^{\ast }$ and ${\mathcal{L}}_{JOL}^{\ast }$ are similar when the sample is well-classified and regressed; on the other hand, the value of ${\mathcal{L}}_{JOL}^{\ast }$ is much larger than ${\mathcal{L}}_{base}^{\ast }$ for the inaccurate prediction sample. This indicates that our proposed Joint Optimization Loss (JOL) can ensure that the model pays more attention to objects for which prediction is more difficult. In addition, it can be seen that only a sample with high scores in both classification and bounding box regression will obtain low loss values and be regarded as a well-predicted sample.

4. Experiments

4.1. Datasets and Evaluation Metric

To comprehensively validate the effectiveness of our proposed method, we conducted experiments on three widely used tiny object detection datasets for remote sensing: VisDrone2019 [23], AI-TOD [59], and AI-TODv2 [60]. In addition, we validated our method on a recent tiny object detection benchmark dataset (SODA-D [13]) and a general object detection dataset (MS COCO 2017 [12]). The fundamental details of the datasets employed in the experiments are provided in

Table 1. Overview of datasets used in the experiments.

Dataset	Categories	Number of Images	Number of Instances	Image Size (Pixels)	Instance Absolute Size (Pixels)	Instance Relative Size (Pixels)
VisDrone2019	10	10,209	343,205	1000	35.8	0.036
AI-TOD	8	28,036	700,621	800	12.8	0.016
AI-TODv2	8	28,036	752,745	800	12.7	0.016
SODA-D	9	24,828	278,433	2901	20.3	0.007
MS COCO 2017	80	118,000	860,000	800	150	0.188

.

VisDrone2019 Dataset: The VisDrone2019 dataset [23] is a widely used benchmark for tiny object detection. It comprises 10,209 static images and 261,908 video frames extracted from 288 video clips captured by Unmanned Aerial Vehicles (UAVs) under varying scales, angles, and lighting.

AI-TOD and AI-TODv2 Datasets: The AI-TOD dataset [59] is a large-scale benchmark for tiny object detection using aerial images from remote sensing satellites. It includes 700,621 annotated objects across eight categories sourced from five public datasets, with an average object size of only 12.8 pixels. The updated AI-TODv2 dataset [60] retains the same number of images but contains additional object instances with smaller sizes and more precise annotations.

SODA-D Dataset: The SODA-D dataset [13] is a recently published benchmark for tiny object detection containing 278,433 high-quality instances collected from the MVD dataset [61] and various online sources. This dataset focuses on detecting tiny objects in diverse scenes, including city roads, highways, and rural areas.

MS COCO 2017 Dataset: The MS COCO 2017 dataset [12] is a widely used dataset for general object detection comprising over 118,000 images and 860,000 object instances across 80 categories. It serves as the official benchmark dataset for the MS COCO Detection Challenge.

Evaluation Metric: Following previous research, we chose the most commonly used COCO metric [12] to evaluate the detection performance of our method. This metric contains two basic values, namely, Precision (P) and Recall (R), respectively defined in Equations (7) and (8):

$$Precision={\displaystyle \frac{TP}{TP+FP}},$$

(7)

$$Recall={\displaystyle \frac{TP}{TP+FN}},$$

(8)

where $TP$ is an abbreviation for True Positive, representing the number of positive output samples with correct predictions; $FP$ is an abbreviation for False Positive, representing the number of positive output samples with incorrect predictions; and $FN$ is an abbreviation for False Negative, representing the number of negative output samples with incorrect predictions. Based on Equations (7) and (8), we can obtain definitions for the Average Precision ($AP$) and Mean Average Precision ($mAP$), as shown in Equations (9) and (10):

$$AP={\int }_0^{1}P\left(R\right)\mathrm{d}R,$$

(9)

$$mAP={\displaystyle \frac{1}{N_c}}\sum _{i=1}^{N_c}A{P}_i,$$

(10)

where $AP$ represents the area under the Precision–Recall (P-R) curve, which ranges from zero to one. The $mAP$ provides the average value of the $AP$ for each category, with $N_c$ denoting the number of categories in the whole dataset.

For each detector, we report ${\mathrm{mAP}}_{0.5:0.95}$, ${\mathrm{mAP}}_{0.5}$, ${\mathrm{mAP}}_{0.75}$, ${\mathrm{AP}}_{vt}$, ${\mathrm{AP}}_t$, ${\mathrm{AP}}_s$, ${\mathrm{AP}}_m$, and ${\mathrm{AP}}_l$. Among these, ${\mathrm{mAP}}_{0.5:0.95}$ indicates that the mean average precision under the IoU ranges from 0.5 to 0.95 at intervals of 0.05, ${\mathrm{mAP}}_{0.5}$ and ${\mathrm{mAP}}_{0.75}$ respectively indicate that the mean average precision under the IoU equals 0.5 or 0.75, and ${\mathrm{AP}}_{vt}$, ${\mathrm{AP}}_t$, ${\mathrm{AP}}_s$, ${\mathrm{AP}}_m$, and ${\mathrm{AP}}_l$ represent the average precision values for objects in the very tiny, tiny, small, medium, and large size categories, respectively.

4.2. Implementation Details

We implemented all of our experiments using the common MMDetection framework [62] and PyTorch [63]. For all models, we utilized ResNet-50 [64] pretrained on ImageNet [65] as the backbone. For training, we used Stochastic Gradient Descent (SGD) as the optimizer with momentum of 0.9 and weight decay of 0.0001. The batch size and number of workers were set to 2 and 4, respectively. The initial learning rate was set to 0.005 for Faster R-CNN, Cascade R-CNN, and DetectoRS and to 0.001 for RetinaNet and FCOS.

For the VisDrone2019, SODA-D, and MS COCO 2017 datasets, in line with [13], the number of training epochs was set to 12, with the learning rate decaying by a factor of 0.1 at epochs 8 and 11. For the AI-TOD and AI-TODv2 datasets, we followed the same configuration as in [15,40] to facilitate comparison with state-of-the-art methods. We trained the models for 24 epochs, applying a learning rate decay of 0.1 at epochs 16 and 22. The RPN proposal counts for Faster R-CNN, Cascade R-CNN, and DetectoRS were all set to 3000 for both the training and testing phases. All other configurations, including the data preprocessor and anchor generator, were kept consistent with the default settings in MMDetection [62]. We conducted all of the following experiments on a computer with an Intel Xeon Gold 6326 CPU and NVIDIA RTX A6000 GPU. The versions of MMDetection, PyTorch, Python, and CUDA used are equal to 3.1.0, 2.0.1, 3.9.19, and 12.2, respectively.

4.3. Results

We conducted a series of experiments on four tiny object detection datasets (VisDrone2019, AI-TOD, AI-TODv2, and SODA-D) and a general object detection dataset (MS COCO 2017). All results reported in the tables are sourced from the corresponding papers; thus, some methods may only have results on certain datasets.

Results on VisDrone2019 Dataset: VisDrone2019 dataset is a challenging benchmark with significant scale variation, containing both tiny and general-size objects. We compared the detection performance of several commonly used object detectors before and after incorporating our proposed Joint Optimization Loss (JOL), as shown in

Table 2. Main results on VisDrone2019. All models were trained on the train set and tested on the val set. The superscript displays accuracy gains against the baseline methods.

Method	${\mathrm{mAP}}_{0.5:0.95}$	${\mathrm{mAP}}_{0.5}$	${\mathrm{AP}}_{\mathit{vt}}$	${\mathrm{AP}}_{\mathit{t}}$
baseline:
RetinaNet [30]	7.9	15.9	0.1	0.7
FCOS [33]	14.1	25.5	0.1	2.1
Faster R-CNN [24]	22.3	38.0	0.1	6.2
Cascade R-CNN [66]	22.5	38.5	0.5	6.8
DetectoRS [67]	25.7	41.7	0.5	7.6
DetectoRS w/ SimD [40]	28.7	50.3	7.6	17.3
ours:
RetinaNet w/ JOL	${\mathrm{14.2}}^{+6.3}$	26.6	0.8	3.4
FCOS w/ JOL	${\mathrm{19.9}}^{+5.8}$	37.9	5.9	12.0
Faster R-CNN w/ JOL	${\mathrm{26.9}}^{+4.6}$	49.7	8.1	16.6
Cascade R-CNN w/ JOL	${\mathrm{27.8}}^{+5.3}$	49.0	8.5	16.8
DetectoRS w/ JOL	${\mathrm{29.3}}^{+3.6}$	51.6	8.4	17.9
DetectoRS w/ SimD + JOL	${\mathrm{29.8}}^{+1.1}$	52.5	8.6	18.8

. The first six rows display the baseline results, while the last five rows illustrate the results achieved with our method. For the one-stage RetinaNet and FCOS detectors, our method results in ${\mathrm{mAP}}_{0.5:0.95}$ improvements of 6.3 and 5.8 points, respectively. For the two-stage Faster R-CNN, Cascade R-CNN, and DetectoRS detectors, our method yields ${\mathrm{mAP}}_{0.5:0.95}$ improvements of 4.6, 5.3, and 3.6 points, respectively. Compared with the best-performing label assignment strategy, SimD, our method results in a 0.6-point improvement in ${\mathrm{mAP}}_{0.5:0.95}$, as shown in the sixth row of Table 2. The performance improvements are particularly notable for tiny and very tiny objects. For example, Faster R-CNN achieves detection accuracy of only 0.1 (${\mathrm{AP}}_{vt}$) and 6.2 (${\mathrm{AP}}_t$) points after applying JOL, representing increases of 8.1 and 16.6, respectively, underscoring the effectiveness of our method for detecting tiny objects. Typical visual comparisons of detection performance on the VisDrone2019 dataset are shown in Figure 4, where the improvements brought about by our method are obvious.

Results on AI-TOD Dataset: We conducted a comprehensive comparison of the detection performance of our method against various mainstream object detectors on the AI-TOD dataset, as shown in

Table 3. Main results on the AI-TOD dataset. All models were trained on the trainval set and tested on the test set. The superscript displays the accuracy gains against the baseline methods.

Method	Publication	Backbone	${\mathrm{mAP}}_{0.5:0.95}$	${\mathrm{mAP}}_{0.5}$	${\mathrm{mAP}}_{0.75}$	${\mathrm{AP}}_{\mathit{vt}}$	${\mathrm{AP}}_{\mathit{t}}$	${\mathrm{AP}}_{\mathit{s}}$	${\mathrm{AP}}_{\mathit{m}}$
two-stage anchor-based:
Faster R-CNN [24]	TPAMI 2017	ResNet-50	11.1	26.3	7.6	0.0	7.2	23.3	33.6
Cascade RPN [68]	NeurIPS 2019	ResNet-50	13.3	33.5	7.8	3.9	12.9	18.1	26.3
Cascade R-CNN [66]	CVPR 2018	ResNet-50	13.8	30.8	10.5	0.0	10.6	25.5	36.6
DetectoRS [67]	CVPR 2021	ResNet-50	14.8	32.8	1.4	0.0	10.8	18.3	38.0
one-stage anchor-based:
SSD-512 [31]	ECCV 2016	ResNet-50	7.0	21.7	2.8	1.0	4.7	11.5	13.5
RetinaNet [30]	ICCV 2017	ResNet-50	8.7	22.3	4.8	2.4	8.9	12.2	16.0
ATSS [69]	CVPR 2020	ResNet-50	12.8	30.6	8.5	1.9	11.6	19.5	29.2
anchor-free:
OTA [70]	CVPR 2021	ResNet-50	10.4	24.3	7.2	2.5	11.9	15.7	20.9
FCOS [33]	ICCV 2019	ResNet-50	12.6	30.4	8.1	2.3	12.2	17.2	25.0
M-CenterNet [59]	ICPR 2021	DLA-34	14.5	40.7	6.4	6.1	15.0	19.4	20.4
state-of-the-art method:
DetectoRS w/ RFLA [22]	ECCV 2022	ResNet-50	24.8	55.2	18.5	9.3	24.8	30.3	38.2
HANet [17]	TCSVT 2024	ResNet-50	22.1	53.7	14.4	10.9	22.2	27.3	36.8
TTFNet w/ GA [16]	TGRS 2024	MiT-B1	24.2	53.9	19.4	10.4	25.2	24.5	34.1
KLDet [15]	TGRS 2024	ResNet-50	25.2	-	-	-	-	-	-
DetectoRS w/ SimD [40]	IROS 2024	ResNet-50	26.6	55.9	21.2	13.4	27.5	30.9	37.8
ours:
RetinaNet w/ JOL	-	ResNet-50	${\mathrm{15.1}}^{+6.4}$	36.9	9.7	5.2	14.3	20.3	29.7
FCOS w/ JOL	-	ResNet-50	${\mathrm{17.8}}^{+5.2}$	41.1	13.6	8.4	18.2	21.3	29.1
Faster R-CNN w/ JOL	-	ResNet-50	${\mathrm{24.7}}^{+13.6}$	56.7	18.1	13.4	25.1	26.9	36.5
Cascade R-CNN w/ JOL	-	ResNet-50	${\mathrm{25.5}}^{+11.7}$	56.2	20.2	13.3	26.3	27.7	37.8
DetectoRS w/ JOL	-	ResNet-50	${\mathrm{27.2}}^{+12.4}$	57.4	22.1	13.9	27.8	30.7	40.1
DetectoRS w/ SimD + JOL	-	ResNet-50	${\mathrm{28.3}}^{+1.7}$	58.2	23.9	13.6	28.7	32.5	39.3

. The first four rows present the performance of the two-stage anchor-based detectors, rows 5–7 display the results from the one-stage anchor-based detectors, and rows 8–10 report the results for the anchor-free detectors. Notably, we also include state-of-the-art detection results from recently published research in rows 11–15. The final six rows illustrate the detection performance of our method combined with baseline and state-of-the-art detectors. Compared with base detectors, including RetinaNet, FCOS, Faster R-CNN, Cascade R-CNN, and DetectoRS, our method achieves ${\mathrm{mAP}}_{0.5:0.95}$ improvements of 6.4, 5.2, 13.6, 11.7, and 12.4 points, respectively, demonstrating its adaptability across various detectors. Additionally, when combined with an existing state-of-the-art method on the AI-TOD dataset, our method yields a further improvement pf 1.7 points in terms of ${\mathrm{mAP}}_{0.5:0.95}$. Visual comparisons of the detection results and Precision–Recall (P-R) curves before and after applying our method are depicted in Figure 5 and Figure 6, respectively, highlighting the significant enhancement in performance.

Results on Other Related Datasets: The detection performance on the AI-TODv2 dataset is presented in

Table 4. Main results on the AI-TODv2 dataset. All of the settings were as same as for the AI-TOD dataset. The superscript displays the accuracy gains against the baseline methods.

Method	Publication	Backbone	${\mathrm{mAP}}_{0.5:0.95}$	${\mathrm{mAP}}_{0.5}$	${\mathrm{mAP}}_{0.75}$	${\mathrm{AP}}_{\mathit{vt}}$	${\mathrm{AP}}_{\mathit{t}}$	${\mathrm{AP}}_{\mathit{s}}$	${\mathrm{AP}}_{\mathit{m}}$
two-stage anchor-based:
Faster R-CNN [24]	TPAMI 2017	ResNet-50	12.8	29.9	9.4	0.0	9.2	24.6	37.0
Cascade R-CNN [66]	NeurIPS 2019	ResNet-50	15.1	34.2	11.2	0.1	11.5	26.7	38.5
DetectoRS [67]	CVPR 2021	ResNet-50	16.1	35.5	12.5	0.1	12.6	28.3	40.0
one-stage anchor-based:
YOLOv3 [32]	ArXiv 2018	DarkNet	4.1	14.6	0.9	1.1	4.8	7.7	8.0
RetinaNet [30]	ICCV 2017	ResNet-50	8.9	24.2	4.6	2.7	8.4	13.1	20.4
SSD-512 [31]	ECCV 2016	VGG-16	10.7	32.5	4.0	2.0	8.7	16.8	28.0
anchor-free:
FCOS [33]	ICCV 2019	ResNet-50	12.0	30.2	7.3	2.2	11.1	16.6	26.9
Grid R-CNN [71]	CVPR 2019	ResNet-50	14.3	31.1	11.0	0.1	11.0	25.7	36.7
state-of-the-art method:
ESG_TODNet [19]	GRSL 2024	ResNet-50	24.6	55.1	18.1	9.5	24.0	29.4	35.6
ORFENet [18]	TGRS 2024	ResNet-50	24.8	55.4	18.2	9.7	24.4	28.7	35.1
DetectoRS w/ SimD [40]	IROS 2024	ResNet-50	26.5	57.7	20.5	11.6	26.7	30.9	38.4
ours:
RetinaNet w/ JOL	-	ResNet-50	${\mathrm{16.2}}^{+7.3}$	40.0	10.2	5.6	14.3	21.8	33.3
FCOS w/ JOL	-	ResNet-50	${\mathrm{18.6}}^{+6.6}$	42.3	13.2	8.2	17.5	22.5	31.4
Faster R-CNN w/ JOL	-	ResNet-50	${\mathrm{25.0}}^{+12.2}$	58.4	17.6	13.6	25.1	27.8	38.9
Cascade R-CNN w/ JOL	-	ResNet-50	${\mathrm{25.4}}^{+10.3}$	56.3	19.2	11.9	25.4	29.1	39.4
DetectoRS w/ JOL	-	ResNet-50	${\mathrm{26.8}}^{+10.7}$	58.0	21.2	13.2	26.9	30.6	40.5
DetectoRS w/ SimD + JOL	-	ResNet-50	${\mathrm{28.0}}^{+1.5}$	59.6	21.8	13.9	27.8	31.7	40.8

, where we also report results from several widely used detectors of the two-stage anchor-based, one-stage anchor-based, and anchor-free varieties. Compared with the baseline RetinaNet, FCOS, Faster R-CNN, Cascade R-CNN, and DetectoRS detectors, our method achieves ${\mathrm{mAP}}_{0.5:0.95}$ improvements of 7.3, 6.6, 12.2, 10.3, and 10.7 points, respectively. To further demonstrate the effectiveness of our method, we combined it with a state-of-the-art label assignment strategy for tiny object detection, as shown in the eleventh and final rows of Table 4, resulting in an increase in ${\mathrm{mAP}}_{0.5:0.95}$ from 26.5 to 28.0 points.

Table 5. Main results on the SODA-D dataset. All models were trained on the train set and tested on the test set. The superscript displays the accuracy gains against the baseline methods.

Method	${\mathrm{mAP}}_{0.5:0.95}$	${\mathrm{mAP}}_{0.5}$	${\mathrm{mAP}}_{0.75}$
baseline:
Faster R-CNN [24]	28.9	59.7	24.2
Cascade R-CNN [66]	31.2	59.9	27.8
RetinaNet [30]	28.2	57.6	23.7
CornerNet [72]	24.6	49.5	21.7
FCOS [33]	23.9	49.5	19.9
RepPoints [73]	28.0	55.6	24.7
YOLOX [34]	26.7	53.4	23.0
DetectoRS w/ SimD [40]	32.8	59.4	31.3
ours:
RetinaNet w/ JOL	${\mathrm{28.3}}^{+0.1}$	53.8	25.7
FCOS w/ JOL	${\mathrm{27.3}}^{+3.4}$	53.3	24.5
Faster R-CNN w/ JOL	${\mathrm{31.3}}^{+2.4}$	60.0	28.4
Cascade R-CNN w/ JOL	${\mathrm{32.8}}^{+1.6}$	59.8	31.2
DetectoRS w/ JOL	${\mathrm{33.4}}^{+0.6}$	60.1	31.5
DetectoRS w/ SimD + JOL	${\mathrm{34.3}}^{+1.5}$	60.4	31.9

shows detection results on the latest SODA-D tiny object detection benchmark. The first eight rows display results from existing high-performance detectors, while the last six rows present results obtained using our method. The improvements achieved by our method are substantial, especially for DetectoRS, where our method outperforms the best alternative by 1.5 ${\mathrm{mAP}}_{0.5:0.95}$ points. From Figure 7, it is evident that our method successfully detects tiny objects across various categories, demonstrating its strong ability on tiny object detection tasks. To assess the adaptability of our method, we conducted an additional comparison on the MS COCO 2017 dataset, with the results shown in

Table 6. Comparison on the MS COCO 2017 dataset. All models were based on Faster R-CNN with ResNet-50 backbone, trained on the trainval35k set, and tested on the minival set.

Method	${\mathrm{mAP}}_{0.5:0.95}$	${\mathrm{mAP}}_{0.5}$	${\mathrm{AP}}_{\mathit{s}}$	${\mathrm{AP}}_{\mathit{m}}$	${\mathrm{AP}}_{\mathit{l}}$
Faster R-CNN (baseline)	36.5	56.8	21.0	39.8	46.8
Faster R-CNN w/ JOL (ours)	37.6	58.1	21.5	41.2	48.6

. Our JOL achieves an ${\mathrm{mAP}}_{0.5:0.95}$ improvement of 1.1 points, indicating that it is also effective for general object detection tasks.

4.4. Ablation Study

Effectiveness of Regression and Classification Scores. Our proposed Joint Optimization Loss (JOL) incorporates both bounding box regression scores and classification scores to calculate the weight of each training sample. To further validate the effectiveness of our approach, we conducted an ablation study comparing performance when using only the classification score or only the bounding box regression score, with the results shown in

Table 7. Ablation study evaluating the effectiveness of regression and classification scores. All of the results are based on the AI-TOD dataset using Faster R-CNN as the detector.

Method	${\mathit{S}}^{\mathit{loc}}$	${\mathit{S}}^{\mathit{cls}}$	${\mathbf{mAP}}_{0.5:0.95}$	${\mathbf{mAP}}_{0.5}$	${\mathbf{mAP}}_{0.75}$
Faster R-CNN			11.1	26.3	7.6
Faster R-CNN w/ $S^{loc}$	✓		23.4	52.4	17.9
Faster R-CNN w/ $S^{cls}$		✓	23.2	51.9	17.6
Faster R-CNN w/ JOL	✓	✓	24.7	56.7	18.1

. The first row displays the baseline results on AI-TOD dataset with Faster R-CNN as the detector. The second and third rows respectively show the results when using only the bounding box regression score ($S^{loc}$) and only the classification score ($S^{cls}$) to calculate each sample’s weight. The fourth row presents the results with the full implementation of our method. From the results in Table 7, it can be observed that there are respective ${\mathrm{mAP}}_{0.5:0.95}$ improvements of 12.3 and 12.1 points when considering only the bounding box regression score and only the classification score. By incorporating both types of score simultaneously, we achieve an ${\mathrm{mAP}}_{0.5:0.95}$ improvement of 13.6 points over the baseline, highlighting the effectiveness of considering both scores. This result underscores the interdependence of bounding box regression and classification, which are inseparable subtasks in object detection.

Performance with Different Values of $\mathit{\lambda }$. The $\mathit{\lambda }$ parameter is used to control the scale of the weights. To determine the optimal value, we conducted an ablation study to evaluate the performance of models with different values of $\mathit{\lambda }$. As shown in

Table 8. Ablation study evaluating the performance of different $\mathit{\lambda }$ values. All results are based on the AI-TOD dataset and use Faster R-CNN as the detector.

$\mathit{\lambda }$	1	2	3	4	5
${\mathbf{mAP}}_{0.5:0.95}$	21.7	24.7	24.7	24.6	24.7

, Faster R-CNN was used as the base detector, with all models trained on the AI-TOD trainval set and tested on the AI-TOD test set. From the results presented in Table 8, it can be observed that the model’s accuracy tends to stabilize when $\mathit{\lambda }$ exceeds two. Therefore, we set $\mathit{\lambda }$ equal to two.

Analysis of Inference and Training Cost. In order to comprehensively evaluate our method, we also calculated the time and space cost of our model during inference and training, then compared the cost with that of baseline methods, including the GFLOPs and parameters of the models during inference and the speed and model size during training. As shown in

Table 9. Comparison of time and space cost during inference and training between our proposed JOL and baseline models. All models were based on the Faster R-CNN detector using AI-TOD trainval set for training and AI-TOD test set for testing, with the same configuration as reported in the experimental settings.

Method	${\mathbf{mAP}}_{0.5:0.95}$	Inference		Training
		GFLOPs	Parameters (MB)	Speed (s/batch)	Model Size (MB)
baseline:
RetinaNet	8.7	129.8	36.5	0.12	292.0
FCOS	12.6	123.3	32.1	0.12	257.5
Faster R-CNN	11.1	134.5	41.4	0.11	331.7
ours:
RetinaNet w/ JOL	${\mathrm{15.1}}^{+6.4}$	129.8	36.5	0.13	292.0
FCOS w/ JOL	${\mathrm{17.8}}^{+5.2}$	123.3	32.1	0.13	257.5
Faster R-CNN w/ JOL	${\mathrm{24.7}}^{+13.6}$	134.5	41.4	0.12	331.7

, we used the RetinaNet and FOCS one-stage detectors as well as the Faster R-CNN two-stage detector as the base detectors. The models were trained on the AI-TOD trainval set and tested on the AI-TOD test set. The GFLOPs during inference were calculated for an input image size equal to 800 × 800. The conditions for calculating the speed of the training process were the same as the experimental settings mentioned above, using an NVIDIA RTX A6000 GPU for training and a batch size equal to 2.

From the comparison between the baseline methods and our proposed method shown in Table 9, it is obvious that the GFLOPs, parameters during inference, and model size of our proposed JOL all remain consistent with those of the baseline methods, with only a slight decrease in the training speed of the model. Considering that our method can significantly improve the detection accuracy, our proposed JOL shows good overall performance on tiny object detection tasks.

5. Conclusions

In this paper, we have proposed the Joint Optimization Loss (JOL) for tiny object detection in remote sensing images, by which we aim to tackle the two primary challenges in the field of tiny object detection. In summary, two main factors contribute to the effectiveness and robustness of our method.

First, the proposed JOL dynamically assigns appropriate weights to each sample based on prediction performance, assigning higher weights to more challenging samples during training. This approach increases the proportion of supervisory signals for difficult-to-predict objects. Because tiny objects are generally more challenging to detect than larger ones, our method shows particularly notable accuracy improvements for tiny and very tiny objects on tiny object detection datasets.

Second, the proposed JOL comprehensively incorporates both classification and bounding box regression scores when assigning weights, enabling joint optimization of these two interdependent subtasks in object detection. This joint optimization helps to mitigate imbalance issues between classification and localization tasks. Furthermore, the JOL can be easily used with various datasets and detectors, making it highly adaptable and robust. Extensive experiments conducted across a series of widely used public datasets demonstrate the outstanding performance and robustness of our method, especially for detecting very tiny objects.

Although our method achieves significant improvements on tiny object detection datasets, its improvement on general object detection datasets is relatively limited. In the feature, we will focus on refining the proposed loss function to enhance the performance of tiny object detection while maintaining robust adaptability across varying object scales.