GM-DETR: Infrared Detection of Small UAV Swarm Targets Based on Detection Transformer

Highlights

What are the main findings?

• A novel DETR-based framework (GM-DETR) is proposed for the infrared detection of small UAV swarms, effectively addressing challenges like weak features, dense distribution, and motion blur.
• The GM-DETR is mainly improved in three aspects: feature fusion, query selection, and geometric position embedding, which enhances the detection effect on small targets.
• Our model also incorporates a long-term memory mechanism, boosting its robustness against scale variations in dynamic scenarios.

What are the implications of the main findings?

• Extensive experiments on a self-constructed dataset (USD) and the publicly available Drone Vehicle dataset show the state-of-the-art performance of the GM-DETR, which demonstrates the effectiveness and robustness of our model.
• The superiority of the GM-DETR in infrared small target detection proves the effectiveness of the query selection, multi-scale feature fusion, and memory mechanism methods, providing new ideas for the DETR in small target detection tasks.

Abstract

Infrared object detection is an important prerequisite for small unmanned aerial vehicle (UAV) swarm countermeasures. Owing to the limited imaging area and texture features of small UAV targets, accurate infrared detection of UAV swarm targets is challenging. In this paper, the GM-DETR is proposed for the detection of densely distributed small UAV swarm targets in infrared scenarios. Specifically, high-level and low-level features are fused by the Fine-Grained Context-Aware Fusion module, which augments texture features in the fused feature map. Furthermore, a Supervised Sampling and Sparsification module is proposed as an explicit guiding mechanism, which assists the GM-DETR to focus on high-quality queries according to the confidence value. The Geometric Relation Encoder is introduced to encode geometric relation among queries, which makes up for the information loss caused by query serialization. In the second stage of the GM-DETR, a long-term memory mechanism is introduced to make UAV detection more stable and distinguishable in motion blur scenes. In the decoder, the self-attention mechanism is improved by introducing memory blocks as additional decoding information, which enhances the robustness of the GM-DETR. In addition, we constructed a small UAV swarm dataset, UAV Swarm Dataset (USD), which comprises 7000 infrared images of low-altitude UAV swarms, as another contribution. The experimental results on the USD show that the GM-DETR outperforms other state-of-the-arts detectors and obtains the best scores (90.6 on ${\mathrm{AP}}_{75}$ and 63.8 on ${\mathrm{AP}}_{\mathit{S}}$), which demonstrates the effectiveness of the GM-DETR in detecting small UAV targets. The good performance of the GM-DETR on the Drone Vehicle dataset also demonstrates the superiority of the proposed modules in detecting small targets.

1. Introduction

In the past few years, unmanned aerial vehicle (UAV) swarms have witnessed significant proliferation across both military and civilian domains [1,2,3], offering the potential to liberate people from repetitive work and reducing the probability of accidents. However, with the frequent occurrence of illegal modifications and flight violations, the challenge of countering malicious low-altitude UAV swarms grows increasingly urgent. Accurate detection of UAV swarms constitutes a critical first step in implementing anti-UAV countermeasures and has thus emerged as a prominent research focus. Among the traditional detection methods, the imaging effect of visible image detection is significantly influenced by external factors such as inclement weather and light intensity, which often lead to unreliable performance. Therefore, it is challenging to achieve continuous and reliable monitoring under some conditions, including stormy weather and long-distance scenarios. Radar-based detection, which relies on echoes reflected by the target, suffers from low-altitude and close-range blind zones, making it susceptible to losing low-altitude targets within UAV swarms. Coupled with the limited reflective area of small UAVs, this method is inadequate for precise and efficient continuous surveillance. Conversely, infrared target detection is not subject to limitations imposed by light intensity, limited reflective area, or inclement weather, which makes continuous surveillance at extended range of low-altitude targets possible. As a result, infrared target detection presents a more suitable alternative for anti-UAV applications.

According to the standard COCO (Common Objects in Context) definition [4], small objects are typically defined as detection targets with a resolution of less than 32 × 32 pixels. In existing datasets [5,6,7], the size of UAV targets generally falls below this established threshold (32 × 32 pixels). Furthermore, in infrared imagery, the pixel area of a typical UAV target is often less than 100 pixels; in extreme cases, it may be as low as approximately 10 pixels. This characteristic categorizes UAV target detection as a typical small target detection problem, rendering most conventional object detection methods unsuitable for UAV detection tasks. The challenges arise primarily because as the depth of the detection network increases and feature processing techniques become more complex and layered, the inherent features of small UAV targets tend to become sparser. Additionally, during continuous downsampling and processing stages, these features are more likely to be obscured by background. Consequently, this leads to a reduction in detection effectiveness. In this case, Zeng et al. [8] proposed a drone detection network based on multi-scale feature fusion, which improves detection performance from the feature extraction and fusion. In Ref. [9], a UAV detector integrated with SSD and YOLOv3 was proposed. The detector achieves good detection performance on RGB images even in motion blur conditions, but the detection speed is decreased. To achieve practical applications, Daniel et al. [10] implemented the YOLOv3 algorithm on the Jetson TX2 device, which is developed by NVIDIA, a US-based technology corporation. However, they did not make improvements to or innovate with the algorithm, and the performance in complex scenarios still needs to be enhanced. In Ref. [11], a learnable NQSA network was proposed which integrates the traditional local contrast extraction method for grayscale features into the semantic dimension. Although the model achieves high detection accuracy by leveraging spatial-channel interactions through priori models, its performance degrades significantly under fluctuating environmental conditions in operational scenarios. In addition to the above methods, there are some other methods [12,13,14,15] that achieve good performance in remote sensing detection tasks.

Although the above methods achieved good results in UAV detection, these methods are mostly focused on scattered UAV targets that do not have swarm attributes. For the infrared target detection of small UAV swarms, there are four significant challenges as follows: (1) Target feature perception challenge. Because of the compact size and extended imaging distances, UAVs usually manifest as small targets in aerial images. The consequent motion-induced edge blurring effect during high-speed flight operations further compromises detection capabilities by distorting edge gradient information and degrading critical texture features. This dual challenge of limited target resolution and dynamic image degradation significantly impedes reliable feature perception in UAV detection scenarios. (2) Similar target classification challenge. Due to the lack of target texture and color features in infrared images, detectors have poor performance in classification tasks, especially when facing small UAV targets. (3) Target feature degradation challenge. The low signal-to-noise ratio makes it challenging to distinguish UAV targets and background. In feature processing, target features are prone to degradation, which may lead to false alarms. (4) Dense distribution challenge. Due to the limitation of communication distance, small UAVs in a swarm are usually distributed in a dense manner. When a swarm conducts formation transformation, mutual occlusion often occurs among UAVs, which increases the complexity of detection. Consequently, it is a significant challenge to maintain high accuracy in the infrared target detection of small UAV swarms.

In this paper, we propose the GM-DETR (Geometric relation embedding and Memory DETR) to tackle these challenges and improve the detection performance mainly from the feature representation, the query sparsification, and the attention mechanism. To enhance the texture and semantic feature representation of infrared images, we propose a Fine-Grained Context-Aware Fusion (FCAF) module, which reduces the loss of semantic features. By analyzing the target characteristics of small UAVs in infrared detection, we propose a supervised sampling method to take the center and boundary points into account, which improves the detection and localization accuracy. Finally, the detection accuracy for mutually occluded and motion blur targets is improved by explicitly encoding the geometric relations among queries and establishing the long-term memory mechanism. The main contributions of our work are summarized as follows:

1.

We propose a Fine-grained Context-Aware Fusion module that enhances small object detection by integrating both semantic and texture features while enlarging the receptive field.

2.

We present a Supervised Sampling and Sparsification method, which consists of a Center Adaptive Supervision (CAS) module and a Boundary–Center Balance Sampling (BBS) module. CAS can adaptively learn the central area of the learning object and provide supervision for subsequent query selection. By balancing the queries of the central and the boundary query area, BBS provides more localization information for the decoding stage and improves the detection accuracy.

3.

To avoid leak detection, we propose a Geometric Relation Encoder to improve the detection performance for mutually obstructed UAVs by encoding the spatial information among queries.

4.

In order to improve the robustness of the GM-DETR, we propose a Memory Updater model and a Memory-Augmented Decoder to establish a long-term memory mechanism in the DETR, which improves the detection ability of scale change and motion blur scenes.

The rest of this paper is organized as follows. Section 2 briefly reviews the development and challenges of infrared target detection. Section 3 gives the overall architecture of the GM-DETR and introduces proposed modules in detail. In Section 4, the extensive experiments are detailed, and it is shown that the GM-DETR has excellent performance in detecting UAV swarms and other small targets. In Section 5, the conclusions of this paper are summarized.

2. Related Work

2.1. Infrared Target Detection

Infrared target detection methods can be broadly categorized into model-driven and deep learning-based approaches. The efficacy of model-driven methods stems from the distinct thermodynamic characteristics that differentiate targets from backgrounds, as discussed in [16,17,18,19,20,21,22,23,24,25]. Specifically, biological organisms or artificial objects typically exhibit abrupt variations in temperature due to their material properties and internal energy. In contrast, background elements such as slopes, skies, and oceans generally display minimal temperature fluctuations characterized by continuous and gradual spatial changes. This disparity allows for a certain degree of feasibility when estimating potential targets within an image based on prior assumptions. Li et al. [16] proposed a saliency detection framework based on spectrum scale-space analysis, which can detect targets by highlighting salient regions and inhibiting repeated regions. In Refs. [17,18], a detection method based on frequency domain transformation was proposed which transforms spatial domain information into the frequency domain one and detects targets by filtering low-frequency signals. Tom et al. [19] propose a detection filter based on morphological background estimation, which exhibits spatial high-pass characteristics emphasizing target-like peaks in the data and suppressing all other clutters. In Ref. [20], a seven-layer deep convolutional neural network is designed to automatically extract small target features and suppress clutters in an end-to-end manner. The method has more powerful target detection performance and improves the signal clutter ratio gain. Cao et al. [21] proposed a derivative dissimilarity measure for the infrared detection of small targets, which suppresses complex background clutters and locate weak and dim targets precisely with a high detection ratio. However, it is important to note that the thermal imaging characteristics of both the target and its surrounding environment are influenced by geographical location, time of day, and seasonal variations. These factors can significantly undermine the effectiveness of model-driven approaches in infrared target detection. This is why the modules or hyperparameters of their methods are required to be adjusted according to different scenarios and targets.

Compared to the above methods, deep learning-based methods have better accuracy, robustness, and generalization capability in visible-light detection, and the texture features and weight parameters are obtained through extensive training. Deep learning-based methods can be broadly divided into one-stage and two-stage detection methods. One-stage detection methods perform the regression and classification tasks in a single forward propagation, which can improve the real-time detection capability, as discussed in [26,27,28]. In comparison with one-stage methods, two-stage detection methods usually have higher detection accuracy under the same conditions due to the fine-grained process of generating candidate regions, as revealed in [29,30,31,32]. Specifically, the first stage is to generate candidate regions from an image. The second stage is to extract features of candidate regions, which are used to perform classification and regression. When the two types of methods are applied to the infrared target detection, the detection performance may tend to decrease due to the lack of texture features and low resolution. Therefore, some modifications were made to improve the detection performance. Ju et al. [33] proposed a filtering module to obtain the confidence map. This method aims to predict the category and position of small infrared targets. In Ref. [34], a single shot multi-box detector was proposed to improve the detection performance on small infrared targets.

2.2. Detection Transformer and Its Variants

With the in-depth study of Transformer methods with attention mechanisms, Transformer-based methods have emerged from the deep learning-based methods. Carion et al. [35] first proposed the Detection Transformer (DETR), which transforms the object detection task into the set prediction task of pixel sequences. The DETR realizes the end-to-end training by eliminating handcrafted components such as anchor boxes and non-maximum suppression. Many methods based on the Transformer have been referred to as the DETR family. These approaches have consistently set new records for the COCO challenge, as discussed in [36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51]. In Ref. [36], it was proved that competitive performance can be achieved by the Efficient DETR, which has only three encoder layers and one decoder layer. Zhu et al. [37] proposed the deformable attention mechanism, which transforms the computational complexity into a linear correlation with the image size and improves the detection accuracy. Meng et al. [38] proposed the conditional cross-attention mechanism to narrow down the spatial range for localizing the distinct regions, which relaxes the dependence on the content embeddings. In Ref. [39], the anchor box was introduced into DETR, which improves the query interpretability and speeds up the training convergence. There are also many other modifications, such as architecture design [40,41,50,51], query sparsification [42,43,44], and training techniques [45,46,47,48,49].

3. Proposed Method

3.1. Overall Architecture

As shown in Figure 1, the GM-DETR, which adopts the two-stage pipeline, consists of five parts, that is, a backbone, a Fine-Grained Context-Aware Fusion module, an encoder, a Query Refinement module and a decoder. We utilize ResNet50 as the backbone network to extract multi-scale features ${\left\{f_m\right\}}_{m=1}^4$, whose structure is presented in [52]. The fused feature map $f_{\mathrm{out}}\in {\mathbb{R}}^{C\times H\times W}$ has the same scale as $f_1$ and is fed into the encoder, where C, H and W represent the channel, height, and width of the feature map. The Supervised Sampling and Sparsification method is proposed to make the GM-DETR focus on high-quality queries with supervised confidence values and reduce computational costs. In the Geometric Relation Encoder, the loss of the spatial information is mitigated by the Geometric Relation Embedding (GRE) module and relation-embedded self-attention. In the Query Refinement and the Decoder, the Memory Updater module and Memory-Augmented Self-Attention are proposed to improve the adaptive detection ability of UAVs.

3.2. Fine-Grained Context-Aware Fusion

In the multi-level features, high-level features are imperative for classification tasks due to their rich semantic information. Concurrently, low-level features, characterized by their high spatial resolution, can retain fine local textures and geometric details, and this process plays a key role in target localization. However, in the task of small target detection, the downsampling operation results in a progressive decline in the resolution of the feature map. Consequently, the representation of small targets in the high-level feature map is susceptible to feature degradation or even loss. At this point, its semantic advantages are transformed into misleading problems due to the absence of target features. In this case, we propose a Fine-Grained Context-Aware Fusion (FCAF) module, which utilizes cascaded dilated convolution to enhance the visual receptive field, capture the detailed target features and context information. Its architecture is shown in Figure 2. For the adjacent high-level feature $f_h$ and low-level feature $f_l$, $f_h$ is expanded to the same scale as the $f_l$ by the bilinear interpolation. After concatenating the $f_h$ and $f_l$ in the channel dimension, 1 × 1 convolution is used to reduce the computational costs, which can be represented as follows:

$$f_i^{\left(1\right)}=\mathrm{Conv}\left(\mathrm{Concat}\left(f_l,\mathrm{Interpolate}\left(f_h\right)\right)\right),$$

(1)

where $\mathrm{Conv}(·)$, $\mathrm{Concat}(·)$, $\mathrm{Interpolate}(·)$, and $f_i^{\left(1\right)}$ represent the convolution, concatenation, bilinear interpolate operation, and the input of first fusion block, respectively.

In the fusion block, we leverage convolution layers with different dilated rates to expand the receptive field, which can avoid target lost. Subsequently, Squeeze-and-Excitation is employed to adjust the channel weights and enhance the representation of small target features, which can be formulated by

$$\begin{array}{cc}\hfill f_h^{\left(1\right)}& ={\mathrm{DConv}}_3\left({\mathrm{DConv}}_2\left({\mathrm{DConv}}_1\left(f_i^{\left(1\right)}\right)\right)\right)+{\mathrm{Conv}}_{3\times 3}\left(f_i^{\left(1\right)}\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill f_o^{\left(1\right)}& =\mathrm{SE}\left(\mathrm{ReLU}\left(f_h^{\left(1\right)}\right)\right),\hfill \end{array}$$

(3)

where ${\mathrm{DConv}}_i$ represents the $3\times 3$ convolution with a dilated rate i, $f_o^{\left(1\right)}$ denotes the output of the first fusion block, and SE is the Squeeze-and-Excitation block.

To ensure the richness and robustness of feature expression, a parallel and complementary pooling strategy is designed. Combined with the Fine-Grained Context-Aware Fusion module, a channel-space dual attention mechanism is formed to achieve fine-grained feature enhancement and dynamic weight allocation, which can be formulated by

$$\begin{array}{cc}\hfill f_{\max }& =(\mathrm{MLP}\left(\mathrm{MaxP}\left(f_o^{\left(N\right)}+f_i^{\left(1\right)}\right)\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill f_{\mathrm{aver}}& =(\mathrm{MLP}\left(\mathrm{AverP}\left(f_o^{\left(N\right)}+f_i^{\left(1\right)}\right)\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill f_{\mathrm{out}}& =\sigma \left(f_{\max }+f_{\mathrm{aver}}\right),\hfill \end{array}$$

(6)

where MaxP and AverP represent the Max Pooling and Average Pooling, respectively; $f_o^{\left(N\right)}$ denotes the output of the last fusion block; $\sigma$ is the sigmoid function; $f_{\max }$ and $f_{\mathrm{aver}}$ represent the output of the Max Pooling branch and Average Pooling branch, respectively; $f_{\mathrm{out}}$ represents the final output; and N denotes the number of the fusion block.

Through the FCAF module, high-level features rich in semantic information can be progressively integrated into low-level features. By employing cascaded dilated convolution and a residual connection path, the local receptive field is expanded, thereby enhancing context information and the contrast between the target features and background. Furthermore, it also effectively mitigates the issue of feature degradation that arises from continuous downsampling, which is further explained in the ablation study.

3.3. Supervised Sampling and Sparsification

Query filtering aims to select queries containing more target information by setting certain quantization rules, achieving comparable performance while reducing the computational load of the model, as revealed in [42,44]. In order to better differentiate the queries in the foreground, Euclidean distance is utilized in the Salience DETR to replace discrete labels 0, 1. While it provides empirical guidance for query selection by concentrating queries in the target centroid, it introduces two critical limitations. First, boundary queries and context information are ignored, resulting in inaccurate localization. Second, it induces directional inconsistencies in prediction that contradict the intrinsic characteristics of infrared UAV targets, which are generally characterized by central symmetry with no significant directional variation, as shown in Figure 3a,b. In order to solve these problems, we propose a Supervised Sampling and Sparsification (SS) method to sparsify input queries and improve the quality of selected queries. This method consists of a Center Adaptive Supervision module and a Boundary–Center Balance Sampling module.

Center Adaptive Supervision. Any object has distinctive features that serve as unique identifiers, which are not necessarily concentrated in the central area. For instance, in the Drone Vehicle dataset, the distinguishing characteristics of different vehicles are not located in the body (center) but rather in the front or rear of the vehicle. Therefore, we utilize a three-layer Multilayer Perceptron (MLP) to predict the center point of a target in order to achieve category adaptive supervision.

In order to improve the problem of direction differences, we use the Chebyshev distance to calculate the confidence ${\theta }_{i,j}$ of a foreground query, which can be formulated by

$${\theta }_{i,j}=\left\{\begin{array}{cc}d\left(p,S_{\mathrm{Bbox}}\right)\hfill & p\in S_{\mathrm{Bbox}},\hfill \\ 0\hfill & p\notin S_{\mathrm{Bbox}},\hfill \end{array}\right.$$

(7)

$$d(p,S_{\mathrm{Bbox}})=1-max\left({\displaystyle \frac{\Delta x}{w}},{\displaystyle \frac{\Delta y}{h}}\right),$$

(8)

where p represents the center point generalized by MLP in a ground truth box $S_{\mathrm{Bbox}}$, and w and h denote the width and height of a ground truth box, respectively. $\Delta x$ and $\Delta y$ denote the distance from queries to the center point in x and y direction, respectively.

In the Center Adaptive Supervision module, the detection accuracy is improved by restricting sampled queries around the center region of UAV targets, where the effect of boundary points on detection was ignored. In this case, we propose the Boundary–Center Balance Sampling (BBS) module to add the proportion of boundary points, which could provide localization information for the decoding stage.

Boundary–Center Balance Sampling. For a ground truth box with a height h and a width w, due to the refinement of foreground points, the confidence level decreases from the central region of to the boundary, which results in the final sampled queries being clustered in the central region and losing the localization and context information of boundary points. In this case, we propose a boundary–center balance sampling approach. During the sampling process, BBS determines the target visual focus based on the MLP prediction results and divides queries into two categories: center and boundary. It adaptively selects a certain number of queries based on the proportion of the two types of queries. The sampling process is shown in Figure 3c. The queries selected by the BBS module are filtered by 0.1 per encoder layer, which makes the GM-DETR further focus on high-quality queries and reduces computational costs.

Through iterative training, the CAS module facilitates the GM-DETR in identifying focal centers for diverse targets within complex scenarios (e.g., feature degradation and motion blur), which mitigates directional inconsistencies in prediction. Concurrently, the BBS module achieves a balanced selection of boundary queries, which collectively enhance localization accuracy. This approach effectively mitigates the inherent difficulties in feature perception of weak small targets in infrared imagery.

3.4. Geometric Relation Encoder

The Geometric Relation Encoder consists of the Geometric Relation Embedding module and six encoder layers. In the GRE module, the two-dimensional geometric relation between queries is constructed by encoding the explicit position relation at the pixel level, and the prior position matrix is obtained as the prior position knowledge. In the encoder layer, relation-embedded self-attention is proposed to fuse prior position information into encoded image features, which helps improve the detection accuracy for dense distributed UAVs.

Geometric Relation Embedding. DETR class methods flatten a two-dimensional feature map into a one-dimensional sequence of queries, which destroys the spatial relation among queries and results in the loss of scale information. In a dense distributed small UAV swarm, UAVs are often obscured from one another. If a feature map is flattened directly, the characteristics of UAVs become intertwined; obscured UAVs may be mistakenly identified as a single, unified target. In this case, we propose the GRE module to encode the geometric relation among queries, which are fed to the decoder as supplementary spatial information.

Let $x_{\max }$ and $x_{\min }$ denote the maximum value and the minimum value of the x-coordinate in a ground truth box, respectively. Let $y_{\max }$ and $y_{\min }$ denote the maximum value and the minimum value of the y-coordinate in a ground truth box, respectively. Let the pair $\left(x_c,y_c\right)$ denote the coordinates of the center query. Let w and h denote the width and the height of a feature map, respectively. Then, one can obtain that

$$\begin{array}{cc}\hfill \left(x_c,y_c\right)& =\left({\displaystyle \frac{x_{max}+x_{min}}{2}},{\displaystyle \frac{y_{max}+y_{min}}{2}}\right),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill h& =1+y_{max}-y_{min},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill w& =1+x_{max}-x_{min}.\hfill \end{array}$$

(11)

Thus, we compute the geometric relation for queries in the feature map, which can be represented by

$$r_{ij}=\left(\begin{array}{c}log\left({\displaystyle \frac{\left|x_i-x_j\right|}{w}}\right)\hfill \\ log\left({\displaystyle \frac{\left|y_i-y_j\right|}{h}}\right)\hfill \\ log\left({\displaystyle \frac{\left|x_i-x_j\right|}{\left|x_i-x_c\right|+\epsilon }}\right)\hfill \\ log\left({\displaystyle \frac{\left|y_i-y_j\right|}{\left|y_i-y_c\right|+\epsilon }}\right)\hfill \end{array}\right),$$

(12)

where $r_{ij}\in {\mathbb{R}}^4$ represents the relative position between query i and query j, and $r_{ij}=0$, when $i=j$.

The first two items in (12) calibrate the relative position of query i and query j. The last two items embed the scale information of a UAV target. Specifically, the denominator acts as a scale factor and calibrates the distance from query i to the center query. When this distance is 0, the last two items are symmetric. In addition to the numerical stability consideration, $\epsilon$ in the denominator is used to ensure that the last two terms are zero only if $i=j$. Then, $r_{ij}$ is mapped to a high-dimensional embedding vector through a fully connected layer, which can be represented as follows:

$$R=\mathrm{FC}\left(r\right),$$

(13)

where $r\in {\mathbb{R}}^{N\times N\times 4}\left(r\left(i,j\right)=r_{ij}\right)$ is a matrix with $N=h\times w$, which includes information about the relative position among queries; FC represents a fully connected layer with a activation function; and $R\in {\mathbb{R}}^{N\times N\times d_{\mathrm{model}}}$ is the geometric relation matrix.

Relation-Embedded Self-Attention. In order to better integrate geometric relation information and reduce the impact on computational complexity, we improve the deformable attention mechanism. Given the feature map $f_{\mathrm{out}}$ and a geometric relation matrix R, the relation-embedded self-attention calculation process is shown in Figure 4 and can be represented as follows:

$$\mathrm{ReAttention}\left(q_i,p_i,f_{out},R\right)=\sum _{h=1}^H{W}_h\left[\sum _{k=1}^K\left(A_{hqk}+A_{hqk}^{Re}\right)/2·W_h^{\prime }{f}_{out}\left(p_i+△p_{hqk}\right)\right],$$

(14)

$$A_{hqk}=\mathrm{Softmax}\left({\mathrm{FC}}_1\left(q_i\right)\right),$$

(15)

$$A_{hqk}^{Re}=\mathrm{Softmax}\left({\mathrm{FC}}_2\left(R\right)\right),$$

(16)

where $q_i$ denotes selected queries; $p_i$ are reference points; h indexes attention heads; k indexes reference points; $A_{hqk}$ and $A_{hqk}^{Re}$ represent the global attention weight and the geometric relation attention weight of the k-th sampling point in the h-th attention head, respectively; $W_h$ is a learnable matrix; $W_h^{\prime }$ is an identity matrix; $△p_{hqk}\in {\mathbb{R}}^2$ denotes the sampling offsets, which are real numbers with unconstrained range; ${\sum }_{k=1}^K{A}_{hqk}$ and ${\sum }_{k=1}^K{A}_{hqk}^{Re}$ are both 1, and their sum divided by 2 is normalized, so that the total attention weight remains 1; and ${\mathrm{FC}}_1$ and ${\mathrm{FC}}_2$ both represent the fully connected layers, whose parameters are independent.

3.5. Memory Update

Memory Updater. By default, the training set is presented in a disordered manner, with no inherent connection between any two consecutive images. Even objects belonging to the same class may exhibit variations in scale and shape. Most detectors do not possess long-term memory capabilities during learning like humans do; thus, the target detection relies solely on model parameters without leveraging any prior knowledge. Nevertheless, we contend that it is feasible to establish a learning mechanism for detecting similar objects, which can enhance the detection accuracy. In this case, we propose the Memory Updater module, which is designed to facilitate long-term memory retention for detection. Specifically, memory blocks $M\in {\mathbb{R}}^{N_c\times 256}$ are learnable high-dimensional vectors, where the $N_c$ denotes the number of memory blocks. They are created using the Embedding function from the torch library during model initialization. The update process can be represented as follows:

$$\begin{array}{cc}\hfill {\widehat{M}}_i& =\mathrm{MultiHeadAttention}\left(M_i,U,U\right),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill M_i& =\mathrm{MLP}\left(\mathrm{Concat}(M_i,{\widehat{M}}_i)\right),\hfill \end{array}$$

(18)

where the $M_i$ denotes the memory block of class i, which is selected to update according to the classification probability of the encoder, and U and ${\widehat{M}}_i$ represent the output of the encoder and the new memory block of the current target, respectively.

Then, we concatenate $M_i$ and the ${\widehat{M}}_i$ using a two-layer MLP to fuse memory blocks. It should be pointed out that memory blocks are only updated during the training stage.

3.6. Memory-Augmented Decoder

Memory-Augmented Self-Attention. In a decoder of the DETR, object queries are learnable vectors that assist the model in understanding the position and feature representation of targets within an image. Self-attention is employed to facilitate attention through pairwise similarity. However, in certain scenarios, such as those involving scale variations or motion blur, the size and appearance of a target may significantly change, leading to a reduction in detection accuracy. In this case, we propose Memory-Augmented Self-Attention (MASA) to address these challenges. In particular, the updated memory block is passed to the decoder as supplementary decoding information, which encompasses prior knowledge and assists the GM-DETR to better detect objects with scale variations or limited texture features. Memory-Augmented Self-Attention can be represented as follows:

$$\begin{array}{cc}\hfill K& =\mathrm{Concat}\left(W_{k}Q,W_m·\mathrm{FC}\left(M_i\right)\right),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill V& =\mathrm{Concat}\left(W_{v}Q,W_m·\mathrm{FC}\left(M_i\right)\right),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill MASA& =\mathrm{MultiHeadAttention}(W_{q}Q,K,V),\hfill \end{array}$$

(21)

where Q represents the query object, and FC represents the fully connected layer, which is used to compress the dimension of $M_i$ to 64. $W_q$, $W_k$, $W_v$, and $W_m$ are learnable matrices.

3.7. Training Details

The GM-DETR is trained with a multi-task loss function, which is defined as follows:

$$\begin{array}{cc}\hfill \mathcal{L}& ={\lambda }_m{\mathcal{L}}_m+{\lambda }_f{\mathcal{L}}_f+{\lambda }_{enc}{\mathcal{L}}_{enc}+{\lambda }_{dn}{\mathcal{L}}_{dn},\hfill \end{array}$$

(22)

where ${\mathcal{L}}_m$ is the loss for pair-wise matching based on the Hungarian algorithm; ${\mathcal{L}}_f$ denotes the Supervised Sampling and Sparsification loss function; ${\mathcal{L}}_{enc}$ is the loss for auxiliary optimization through the output of the last encoder layer; ${\mathcal{L}}_{dn}$ is the loss for denoising models; and ${\lambda }_m$, ${\lambda }_f$, ${\lambda }_{enc}$, and ${\lambda }_{dn}$ are scaling parameters.

Loss items are specifically formulated by

$$\begin{array}{cc}\hfill {\mathcal{L}}_m& =-{\mathbb{1}}_{\left\{c_i\ne \oslash \right\}}{\widehat{p}}_{\delta \left(i\right)}\left(c_i\right)+{\mathbb{1}}_{\left\{c_i\ne \oslash \right\}}{\mathcal{L}}_{\mathrm{box}}\left(b_i,{\widehat{b}}_{\delta \left(i\right)}\right),\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathcal{L}}_f& =-{\alpha }_f{\left(1-p_f\right)}^{\gamma }log\left(p_f\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill p_f& =1+2\widehat{\theta }\theta -\left(\widehat{\theta }+\theta \right),\hfill \end{array}$$

(25)

where $\mathbb{1}$ denotes the indicator function, $\delta \left(i\right)$ is the prediction index, ${\widehat{p}}_{\delta \left(i\right)}\left(c_i\right)$ denotes the prediction probability that $\delta \left(i\right)$ is the class label $c_i$, ${\mathcal{L}}_{\mathrm{box}}$ denotes the loss between the prediction box ${\widehat{b}}_{\delta }\left(i\right)$ and the ground truth box $b_i$, ${\alpha }_f=0.25$ and $\gamma =2$ are focal parameters, $p_f$ is the foreground probability, and $\theta$ represents the confidence value, which is proposed in (7).

4. Experiments and Discussion

4.1. Experimental Setup

Datasets. We conducted extensive experiments on the self-constructed UAV Swarm Dataset (USD) and the publicly available Drone Vehicle dataset. The USD contains 7000 infrared images of low-altitude UAV swarms with a resolution of 720 × 540, capturing low-altitude UAV swarms against urban backgrounds. The data was acquired using mid-wave infrared cameras mounted on a ground-based platform, simulating real-world drone swarm operations. Data collection covered varied conditions, including time of day (day and night) and weather (clear and cloudy). To ensure diverse behavioral representation, the UAV swarms were programmed to execute multiple formations—such as linear, circular, and random patterns—along with maneuvers, including hovering, cruising, and crossing. The original video clips were converted into image sequences at 25 frames per second, with frames sampled at three-frame intervals for annotation in COCO format. The resulting infrared images encompass a wide range of scenarios, such as swarm cruising, cross-flights, and dynamic formation changes. Each image contains an average of 20–30 UAV targets. The images in the USD can suitably reflect the three challenges associated with the infrared UAV swarm detection mentioned in the first section; an example image and related statistical analysis are shown in Figure 5a and

Table 1. Statistics of USD.

Dataset Split	# Ann	# Class	Image	Resolution	Obj/Img
Train	101,310	1	4938	720 × 540	20.52
Test	24,480	1	1145	720 × 540	21.38
Val	19,956	1	985	720 × 540	20.26

. The allocation of target areas and the statistical analysis of targets categorized by size are illustrated in Figure 5b,c, respectively. The distribution of UAV targets in the USD are shown in Figure 5d. The heat map indicates the number of UAV sightings in the same area, and each blue point indicates an UAV target.

In order to make the comparison more robust, we also compared the GM-DETR with other detectors on Drone Vehicle, which is a large-scale, drone-based, RGB–infrared vehicle detection dataset. It includes 28,439 vision–infrared image pairs. Its detection challenges lie not only in accurately locating small-scale targets from an aerial perspective but also in overcoming the fine-grained feature deficiency caused by shooting height, complex background, and cross-modal differences, so as to accurately distinguish the types of vehicles. Here, we only used infrared images to evaluate the existing methods.

Evaluation Metrics. This study used the standard evaluation protocol, Average Precision (AP) and Average Recall (AR) metrics, as revealed in [4]. These metrics reflect the area under the precision–recall curve, and the formulation can be represented by

$$\begin{array}{cc}\hfill \mathrm{AP}& ={\int }_0^{1}pre\left(r\right)\mathrm{dr},\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill pre& ={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}},\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill r& ={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}},\hfill \end{array}$$

(28)

where the $pre$ denotes the detection accuracy, r represents the recall, TP is true positives, FP denotes false positives, and FN represents false negatives.

The criterion for determining a true positive is that the C-IoU of the prediction bounding box and the ground truth should be greater than the threshold of 0.5. The average AP for each class is given as mAP, while the average accuracy mAP is obtained from thresholds in the range 0.5:0.95. In addition, the computational and real-time performance are evaluated based on the number of giga-floating-point operations per second (GFLOPs) and frames processed per second (FPS).

As shown in Figure 5b,c, all the target areas in the USD are smaller than the definition of small targets in coco (32 × 32 pixels), which leads to the evaluation metrics, such as ${\mathrm{AP}}_{\mathit{M}}$ and ${\mathrm{AP}}_{\mathit{L}}$, being −1 in the final evaluation. Such results provide no meaningful information for the task of infrared detection of small UAVs and allow for no meaningful comparison among different models. In this case, we adopted 81 and 256 pixels as thresholds to redefine the pixel area ranges for large, medium, and small objects, mapping the original pixel area ranges from (96 × 96, +∞), (32 × 32, 96 × 96], and (0, 32 × 32] to (16 × 16, +∞), (9 × 9, 16 × 16], and (0, 9 × 9], respectively. This classification aligns with the definition of small infrared targets established by the Society of Photo-Optical Instrumentation Engineers [53,54].

Implementation details. In terms of the hyperparameter settings and training details, we set $H=8$ and $K=4$ for deformable attention. The number of fusion block was set to 6. As for object queries, DETR class methods conducted query selection layer by layer on multi-scale features, and the number of object queries was set to 300 to avoid the deviation of detection performance caused by inconsistent object queries. The effect of object queries on the detection performance of the GM-DETR is discussed in detail in the ablation study. The memory slots are designed to update and store prior knowledge for each object category. Therefore, $N_c$ was set to the total number of object categories plus one to include the background class. For instance, in the Drone Vehicle dataset, $N_c$ was set to 6, while for the USD with a single foreground category, $N_c$ took the value 2. We trained the GM-DETR using the AdamW optimizer [55] with a weight decay of $1\times {10}^{-4}$ on NVIDIA RTX 4090 GPUs (24 GB), which is developed by NVIDIA, a US-based technology corporation. For the initial learning rate, the backbone part was set to $1\times {10}^{-5}$, and other parts were set to $1\times {10}^{-4}$. The learning rate was decreased by 0.1 at later stages. The batch of per GPU was set to 4 and 10 for the USD and Drone Vehicle, respectively. The training epoch was set to 120 and 12 for the USD and Drone Vehicle, respectively. All methods adjusted the input size according to the image to ensure the validity of the experimental results.

4.2. Comparison with State-of-the-Art Methods

To better validate the effectiveness of the GM-DETR, we compared the performance of the GM-DETR model with models that have achieved the cutting-edge performance in recent years under the same experimental settings, namely, Faster R-CNN [29], RetinaNet [56], YOLOv5 [57], YOLOv6 [58], YOLOv7 [26], YOLOv8 [27], YOLOv10 [28], Deformable DETR [37], Conditional DETR [38], Anchor DETR [59], DAB-DETR [39], Dn-DETR [45], Dino [40], Dino-eva-01 [60], Focus-DETR [32], Co-DETR [49], and Salience DETR [44]. The evaluation results on the USD and Drone Vehicle are shown in

Table 2. Quantitative comparison on USD. The best and second-best results are marked in red and blue, respectively. ${\mathrm{AP}}_{\mathit{L}}$, ${\mathrm{AP}}_{\mathit{M}}$, and ${\mathrm{AP}}_{\mathit{S}}$ represent the average accuracy of large, medium, and small infrared targets, respectively, which are defined as (256, +∞), (81, 256], and (0, 81] pixels.

Methods	Query	AP	${\mathrm{AP}}_{50}$	${\mathrm{AP}}_{75}$	${\mathrm{AP}}_{\mathit{S}}$	${\mathrm{AP}}_{\mathit{M}}$	${\mathrm{AP}}_{\mathit{L}}$	AR	${\mathrm{AR}}_{\mathit{S}}$	${\mathrm{AR}}_{\mathit{M}}$	${\mathrm{AR}}_{\mathit{L}}$	FPS
Faster R-CNN	-	62.9	85.6	79.7	54.1	65.5	76.1	70.4	66.1	75.3	79.2	14.2
RetinaNet	-	70.1	93.6	84.1	58.9	69.6	80.3	73.7	66.5	79.3	80.5	7.6
YOLOv5	-	70.3	91.1	83.2	60.5	71.8	80.9	74.2	66.3	77.7	83.6	38.6
YOLOv7	-	63.7	86.3	80.1	56.9	67.4	75.7	68.1	64.5	74.2	78.8	53.4
YOLOv8	-	71.1	93.7	87.6	61.4	72.7	81.3	74.1	68.1	79.4	83.2	57.7
YOLOv10	-	65.7	90.6	85.5	58.3	71.8	81.4	68.6	66.5	67.1	82.7	46.1
DAB-DETR	300	69.8	94.7	85.8	58.7	75.4	80.9	71.9	67.9	78.6	83.0	8.6
Dn-DETR	300	70.9	96.1	86.4	57.6	76.1	78.4	70.3	66.7	80.4	84.8	8.9
Focus-DETR	300	70.6	94.4	88.7	58.4	76.7	81.6	73.8	68.5	79.8	85.1	9.0
Co-DETR	300	72.8	98.6	89.3	62.1	74.3	83.5	77.4	71.2	81.5	88.2	10.7
Dino-eva-01	300	72.1	96.3	89.0	62.9	77.1	83.3	76.2	71.0	80.7	87.1	4.2
Salience DETR	300	71.3	95.3	87.9	62.7	74.6	83.0	76.5	70.8	80.2	87.9	9.1
GM-DETR	300	73.7	98.4	90.6	63.8	75.1	83.8	77.9	71.3	80.0	87.5	11.3

and

Table 3. Quantitative comparison on Drone Vehicle. The best and second-best results are marked in red and blue, respectively.

Methods	Epochs	AP	${\mathrm{AP}}_{50}$	${\mathrm{AP}}_{75}$	AR	Car	Truck	Bus	Van	Freight Car	Params	GFLOPs
YOLOv5x	12	34.7	55.2	35.9	54.1	55.1	25.1	51.7	20.1	21.4	86.2 M	204.7
YOLOv6x	12	29.8	52.7	32.4	52.4	47.2	19.8	50.4	15.7	16.1	172.9 M	184.3
YOLOv8x	12	34.2	56.3	36.1	55.6	54.8	24.3	51.1	19.8	20.9	68.1 M	257.6
RetinaNet	12	32.4	54.9	34.6	46.0	53.9	22.8	48.4	17.6	19.2	38.4 M	281.8
DAB-DETR	12	26.1	54.8	22.9	47.2	40.1	16.2	45.0	13.2	16.0	43.6 M	35.1
DAB-deform-DETR	12	38.1	60.4	42.9	60.8	56.3	28.2	55.5	25.6	25.0	47.3 M	87.3
Deformable DETR	12	34.7	58.0	36.9	59.7	54.4	25.6	52.4	19.8	21.2	40.0M	64.2
Dn-DETR	12	29.6	54.4	30.2	49.1	46.4	20.2	50.1	15.5	16.0	43.7 M	35.1
Dino	12	35.3	57.7	38.2	58.4	56.3	25.5	53.4	21.1	20.2	47.5 M	97.7
Anchor DETR	12	10.9	25.7	7.2	26.7	27.1	2.6	20.8	1.5	2.6	37.0 M	40.4
Conditional DETR	12	30.6	56.9	30.5	54.6	47.9	20.2	48.3	17.9	18.8	43.4 M	34.2
Focus-DETR	12	35.8	59.1	38.6	60.6	56.1	25.3	54.7	22.7	19.9	49.2 M	64.9
Salience DETR	12	33.9	53.4	37.7	56.7	54.9	23.0	52.2	20.1	19.0	56.3 M	201.4
GM-DETR	12	38.6	58.8	41.6	61.5	59.4	27.8	56.3	24.1	25.5	53.8 M	208.5

. The GM-DETR achieves good performance on both datasets.

Comparison on USD. As shown in Table 2, except for ${\mathrm{AP}}_{\mathit{M}}$, ${\mathrm{AR}}_{\mathit{M}}$, and ${\mathrm{AP}}_{\mathit{L}}$, the GM-DETR outperforms reference methods on all metrics. It is worth noting that the GM-DETR is the only method to score above 90 on ${\mathrm{AP}}_{75}$, and the GM-DETR surpasses the second-best method, the Focus-DETR, with a large margin of 1.9% on ${\mathrm{AP}}_{75}$. In addition, we focus more on the performance in detecting small targets, i.e., ${\mathrm{AP}}_{\mathit{S}}$, as it reflects the response speed and detection distance of a model. It is clear that the GM-DETR improves by 1.1% on ${\mathrm{AP}}_{\mathit{S}}$ compared to the second-best method, the Salience DETR, which demonstrates the effectiveness of the GM-DETR in the infrared detection of small UAV targets. All methods were trained for 120 epochs. All employ ResNet50 as the backbone, except for Dino-eva-01, which uses EVA [60] as backbone.

Comparison on Drone Vehicle. Table 3 shows the comparison results on Drone Vehicle. Drone Vehicle is also a highly challenging detection dataset because of the similar appearance of air-ground targets. Detectors are required to capture sufficient appearance feature and context information to distinguish different type vehicles. The GM-DETR achieves the best performance in terms of the AP and AR metrics. To evaluate the proposed methods against the baseline Salience DETR, the ablation study is conducted using AP, ${\mathrm{AP}}_{75}$, ${\mathrm{AP}}_S$, and FPS metrics by removing specific components and testing on the USD.

As shown in

Table 4. Ablation results on USD. FCAF, SS, GRE, and Memory stand for the Feature Fusion, Supervised Sampling and Sparsification, Geometric Relation Encoder, Memory Updater, and Memory-Augmented Self-Attention, respectively. CAS and BBS are the sub-modules of Supervised sampling and Sparsification.

FCAF	SS		GRE	Memory	AP	${\mathrm{AP}}_{75}$	${\mathrm{AP}}_{\mathit{S}}$	FPS
	CAS	BBS
✗	✗	✗	✗	✗	71.3	87.9	62.7	9.1
✓	✗	✗	✗	✗	71.9	89.2	62.8	12.8
✓	✓	✗	✗	✗	71.9	89.3	62.8	12.8
✓	✓	✓	✗	✗	72.4	89.7	63.1	12.2
✓	✗	✓	✗	✗	72.1	89.5	63.0	12.2
✓	✗	✗	✓	✗	71.6	89.4	62.9	12.3
✓	✓	✓	✓	✗	73.1	90.1	63.5	11.6
✓	✓	✓	✗	✓	72.8	90.3	63.3	11.8
✓	✓	✓	✓	✓	73.7	90.6	63.8	11.3

, the ablation study demonstrates that the FCAF, SS, and GRE modules contribute significantly to the detection performance. In contrast, the Memory Updater and Memory-Augmented Decoder increases the ${\mathrm{AP}}_{75}$ but declines in other metrics. All in all, the GM-DETR achieves the best results (73.7 on AP and 90.6 on ${\mathrm{AP}}_{75}$), confirming the effectiveness of each proposed component. The convergence graphs of the ablation study are shown in Figure 6.

4.3. Ablation Study

Effect of Fine-Grained Context-Aware Fusion. To validate the fusion effect of the Fine-Grained Context-Aware Fusion module, we performed a controlled experiment with the Salience DETR and the GM-DETR, as shown in

Table 5. Quantitative comparison of Fine-Grained Context-Aware Fusion module.

Methods	AP	${\mathrm{AP}}_{50}$	${\mathrm{AP}}_{75}$	${\mathrm{AP}}_{\mathit{S}}$	${\mathrm{AP}}_{\mathit{M}}$	${\mathrm{AP}}_{\mathit{L}}$
Salience DETR	71.3	95.3	87.9	62.7	76.1	83.0
Salience DETR + FCAF	71.9	95.1	89.2	63.2	75.8	83.3
GM-DETR + FPN	72.8	97.7	89.2	63.1	75.4	83.1
GM-DETR	73.7	98.4	90.6	63.8	75.1	83.8

. Compared to the GM-DETR, the GM-DETR with FPN declines on all metrics, except ${\mathrm{AP}}_{\mathit{M}}$. In another control group, the Salience DETR with the Fine-Grained Context-Aware Fusion module gains +0.6 improvement on AP and +0.1 improvement on ${\mathrm{AP}}_{\mathit{S}}$, which shows that the Fine-Grained Context-Aware Fusion module can improve the fusion effect of multi-scale features and the detection performance. In Figure 7, we show the convergence process of detection accuracy and loss terms during training.

The loss curve, which denotes the total training loss, exhibits a consistent decline from approximately 15 to 8 over the course of training, demonstrating stable optimization and effective learning. The loss box, representing the regression loss for bounding box localization, decreases significantly from 0.032 to around 0.012, indicating the model’s enhanced capability in accurately locating small infrared UAV targets. Similarly, the loss class curve, which corresponds to classification error, declines from 0.46 to 0.34, reflecting improved discrimination between target and background under challenging infrared conditions. The synchronized convergence of decreasing losses and increasing mAP attests to the robustness and stability of the training process and validates effectiveness of the GM-DETR for the infrared detection of small UAVs.

Ablation study of Memory Updater. In order to ascertain the enhancement of the Memory Updater module and different compression values of $M_i$ on the detection effect, we conducted experiments on the Drone Vehicle dataset. The results are displayed in

Table 6. Ablation results of Memory Updater and compressed value $M_i$.

Memory Value	AP	Car	Truck	Bus	Van	Freight Car
No memory	37.2	57.6	26.5	55.2	23.1	23.8
32	38.3	59.1	27.5	56.1	23.9	25.0
64	38.6	59.4	27.8	56.3	24.1	25.5
128	38.6	59.2	27.5	56.4	24.4	25.7
64 (Mix)	37.9	58.4	27.0	55.9	23.8	24.6

. It is evident that increasing the compression value results in an enhancement of detection accuracy, albeit to a certain extent. Specifically, expanding the prior experience space is demonstrated to enhance the detection precision for different vehicles such as vans and freight cars, which are hard to recognize. No memory group proves the effectiveness of the proposed Memory Updater module. The 64 (Mix) group represents the Memory Updater as set a single memory block and compresses it into 64 in MASA, which is iteratively updated for each target and image. The result demonstrates the the effectiveness of the proposed method in setting the memory update mechanism for each class.

Collectively, the results presented in Table 6 indicate that for infrared images characterized by the absence of discriminative texture and color features, the Memory Updater module enhances the performance of the GM-DETR on the classification task for similar targets.

Ablation study of object queries. The query sparsification of DETR class methods aims to reduce training costs by decreasing the number of queries while maintaining comparable detection performance. To investigate the impact of varying the number of object queries on detection performance, we conducted extensive experiments on Drone Vehicle, and the results are presented in

Table 7. Ablation results on object queries of GM-DETR on Drone Vehicle.

Object Query	AP	Car	Truck	Bus	Van	Freight Car
100	38.0	58.2	27.3	55.6	23.8	25.2
200	38.4	58.9	27.7	56.1	24.1	25.3
300	38.6	59.4	27.8	56.3	24.1	25.5
600	38.8	60.1	27.8	56.5	24.2	25.5

. The research results show that within a certain range, the detection performance of the GM-DETR demonstrates a positive correlation with the number of queries. However, as the number of queries continues to grow, the decoding effect will decline after the introduction of too many low-quality queries. Considering both training costs and the validity of comparisons, we standardized the number of queries to 300 across all our experiments.

Ablation study of the boundary points ratio. To evaluate the influence of the boundary points ratio, a hyperparameter, on the model performance, we designed a detailed ablation experiment. This parameter plays a critical role in the model’s capacity to perceive and leverage edge features of faint and small targets. We performed extensive comparative experiments on the USD across different values of this ratio, with results illustrated in Figure 8. As shown in the figure, detection accuracy begins to decrease gradually once the number of boundary points surpasses that of center points. Furthermore, as the proportion of boundary points continues to rise, the rate of performance degradation becomes increasingly pronounced.

4.4. Visualization

Visualization results for USD. We selected six representative images from the test set to visualize the detection performance covering formation changes, dense distribution, mutual occlusion, multi-cluster flights, motion blur, and different formation shape scenes. In Figure 9, we visualize the attention results of the GM-DETR and find that it can effectively locate the foreground area and distinguish targets from background. Although the GM-DETR is also affected by watermarks and the smoke, the GM-DETR is able to eliminate their interference in the query refinement and decoding stages from the final detection results, which are shown in Figure 10. In Figure 11, we visualize the confidence values of the queries selected by the BBS module. Since only queries with high confidence were selected, there are red dots but no blue dots in the subgraphs. The visualization results demonstrate that the Boundary–Center Balance Sampling module enhances the relevance of the selected query to the target by guiding the selection of queries, which makes the GM-DETR focus on the boundary and center regions.

In Figure 10, YOLOv8 suffers from leak detection and poor accuracy. Faster R-CNN suffers from severe leak detection and misdetection. The Focus-DETR suffers from misdetection in the first image and suffers from leak detection in the second and last images. The Salience DETR is the best among the comparison methods and only suffers from leak detection in the motion blur image. In contrast, the GM-DETR performs well in all cases, especially in the mutual occlusion, dense distribution, and motion blur images.

Visualization results for Drone Vehicle. Vehicle targets in the Drone Vehicle dataset have a relatively large target scale and rich diversity of appearance features, which makes it easier to understand the differences in how different detection models respond to features. Based on the above characteristics, we selected several images to conduct a visual analysis of the baseline and GM-DETR, and corresponding attention heat maps were generated, as shown in Figure 12. Comparative analysis revealed that the GM-DETR model exhibits more focused attention distribution for distinguishable parts, such as the front or rear of the vehicle, depending on the vehicle type. This phenomenon demonstrates the effectiveness of our CAS module and BBS module. In contrast, the baseline model exhibits a more focused attention response pattern relative to the target center area.

The differences in the visual perception mechanism between the GM-DETR and the baseline model were effectively verified by visual comparison analysis. As shown in Figure 13, the confidence values of the baseline model show a significant central aggregation phenomenon. Its attention area is limited to the central region of the annotation box, and there are obvious shortcomings in its ability to capture the boundary and global contextual information of a target. In contrast, the GM-DETR leverages its CAS and BBS modules to dynamically shift focus across target types while balancing attention between central target regions and contextual boundaries. The visualization results of different methods on Drone Vehicle can be found in Figure 14.

5. Conclusions

In this paper, we propose a Transformer-based detector, the GM-DETR, to detect low-altitude small UAV targets. Specifically, we propose a Fine-Grained Context-Aware Fusion module to enhance semantic and texture features of infrared targets. To improve the quality of selected queries, we propose a Supervised Sampling and Sparsification method, which consists of a Center Adaptive Supervision module based on Chebyshev distance and a Boundary–Center Balance Sampling module. By conducting geometric relation embedding, the GM-DETR effectively tackles the missed detection of mutually obstructed small UAV targets. In the second-stage, we introduce a long-term memory mechanism to improve the robustness of the GM-DETR, which improves the detection accuracy of the multi-category Drone Vehicle dataset. In the decoder, we improve the self-attention mechanism in decoder layers by incorporating memory blocks as additional decoding information. Finally, we demonstrate that the GM-DETR achieves state-of-the-art performance by comparing it with competitive methods on the USD and Drone Vehicle. The contributions of different modules are further analyzed through the ablation study. We believe that our work provides a good case for the application of the DETR in the infrared target detection of small UAV swarms.

Owing to constraints in data collection time and airspace authorization, the current USD exhibits limitations in weather and geographical diversity. Furthermore, the use of identically appearing quadcopters during data acquisition also restricted variation in target appearance. As a result, models trained on this dataset may generalize poorly to UAV swarm detection under different environmental conditions. To address these issues, we plan to enhance model generalization by collecting data across more diverse geographic and weather conditions and by leveraging data augmentation and generative algorithms to synthesize UAV imagery under varied scenarios. In the future, our research will focus on multi-modal UAV swarm detection, with an emphasis on investigating cross-modal feature alignment and interaction fusion issues. We will also expand the categories of UAV targets and incorporate RGB-T image data into the USD to support multi-modal detection research.