An Unsupervised Moving Object Detection Network for UAV Videos

Abstract

UAV moving object detection focuses on identifying moving objects in images captured by UAVs, with broad applications in regional surveillance and event reconnaissance. Compared to general moving object detection scenarios, UAV videos exhibit unique characteristics, including foreground sparsity and varying target scales. The direct application of conventional background modeling or motion segmentation methods from general settings may yield suboptimal performance in UAV contexts. This paper introduces an unsupervised UAV moving object detection network. Domain-specific knowledge, including spatiotemporal consistency and foreground sparsity, is integrated into the loss function to mitigate false positives caused by motion parallax and platform movement. Multi-scale features are fully utilized to address the variability in target sizes. Furthermore, we have collected a UAV moving object detection dataset from various typical scenarios, providing a benchmark for this task. Extensive experiments conducted on both our dataset and existing benchmarks demonstrate the superiority of the proposed algorithm.

1. Introduction

UAV moving object detection leverages image processing techniques to identify moving objects within image sequences captured by UAVs. Here, a moving object is defined as any object that moves significantly differently from the background [1]. Moving object detection has broad application prospects in areas such as target detection, aerial surveillance, scene understanding, and event recognition [2].

Under moving observation platforms, the background of the scene continuously shifts with the movement of the platform, and the motion of the target may become confused with the motion of the background, presenting a significant challenge for moving object detection.

According to the Gestalt principle of common fate [3], elements moving in the same pattern tend to be perceived as a group. In moving object detection, the general approach is to distinguish the motion of the foreground (moving objects) from that of the background based on their motion patterns. A key challenge lies in selecting the appropriate model to represent these motion patterns and in distinguishing between the motion models of the foreground and background. Existing techniques commonly utilize constraints such as low-rank properties, subspaces, and Gaussian mixture models to differentiate foreground and background motion. The main principles employed in these typical algorithms include (1) the difference in motion patterns between the foreground and background, and (2) the temporal consistency of foreground motion.

Compared to conventional moving object detection scenarios (such as those with handheld cameras), UAV scenes present unique characteristics, including a variable field of view and relatively small target pixel sizes. These features introduce several distinct challenges for UAV moving object detection. (1) In UAV scenarios, the foreground is sparse (moving objects occupy only a small portion of the scene). (2) In UAV scenarios, the size of moving objects changes more frequently. Directly applying conventional methods from general settings may suffer from many false alarms. Fully leveraging this unique prior knowledge may lead to better results in UAV moving object detection.

However, current datasets for moving object detection in UAV scenarios remain scarce, and related research is limited. Most existing techniques rely on video object segmentation datasets such as DAVIS [4] or tracking datasets like Vivid [5], which primarily focus on observation platforms closer to the ground. These datasets typically feature a single, large target positioned at the center of the field of view. The lack of publicly accessible datasets specifically tailored for UAV-based moving object detection limits the development and evaluation of algorithms optimized for this domain. Furthermore, the integration of domain-specific prior knowledge remains largely unexplored in current detection technologies.

This paper presents a dataset named UAV moving object detection (UAVMD) specifically designed for the UAV moving object detection task. We then propose an unsupervised method based on optical flow for the task. The method leverages domain-specific prior knowledge, including foreground sparsity and the spatiotemporal consistency of foreground motion in UAV video, to guide a multi-scale segmentation network in detecting true moving objects. The algorithm operates without the need for complex annotations during training and demonstrates superior performance on UAV moving object detection, exhibiting strong generalization capabilities on unseen data.

The main contributions of this paper can be summarized as follows:

• We propose an unsupervised multi-scale UAV moving object detection network based on optical flow, which fully leverages prior knowledge in UAV scenarios. This architecture effectively separates the foreground from background by exploiting motion pattern disparities, enabling accurate detection of prominent moving objects in UAV videos.
• We incorporate prior knowledge of foreground sparsity and spatial consistency in UAV ground observation scenes using the Ising model into the loss function of the unsupervised network. This crucial step significantly mitigates false alarms arising from motion parallax and platform motion.
• We collected high-definition UAV-to-ground observation data using DJI drones, comprising 31,536 image frames at 1920 × 1080 resolution. We annotated 37 typical sequences, containing 3469 frames (spaced every 10 frames), with polygonal bounding boxes, providing a benchmark for the UAV moving object detection task.
• Our proposed method demonstrates superior performance on established benchmarks for general moving object detection and achieves outstanding results on the introduced UAV moving object detection dataset. Furthermore, it exhibits exceptional generalization capabilities across diverse scenarios.

2. Related Work

2.1. Moving Object Detection Methods Under Dynamic Observation Platforms

Typical moving object detection methods for dynamic observation platforms can be broadly classified into four categories based on different motion representation models, as shown in Figure 1: background modeling, feature point tracking, supervised deep learning, and optical flow-based methods.

Figure 1. Frameworks of typical moving object detection methods under dynamic observation platforms.

A.

Background modeling methods

Background modeling methods create background models based on characteristics such as probability or low-rank properties in sequential images, then detect moving objects that cannot fit in the model. Several prominent approaches exist within this framework: (1) Probability distribution-based methods. Stauffer et al. [6] statistically analyze the grayscale value distribution of pixels over time, using a Gaussian Mixture Model (GMM) to fit the grayscale value variations. Foreground pixels, which have a larger variance and cannot fit into the background distribution, are detected. Zivkovic [7] further utilizes an iterative equation to continuously update the GMM parameters, selecting an appropriate number of Gaussian distributions for each pixel. Lee [8] calculated an adaptive learning rate for each Gaussian model, improving the convergence efficiency of the GMM. (2) Principal Component Analysis (PCA) -based methods. The DECOLOR method [1] leverages the low-rank property of the background and uses global optimization to identify sparse foregrounds that do not fit the low-rank background matrix. To address issues such as low computational efficiency and high false alarm rates, researchers [9,10] propose improvements through subspace tracking and kinematic regularization techniques. (3) Visual background extraction methods [11]. These methods select a certain number of pixels near each site to model the background. The Euclidean distance between the pixel and the background sample values is used as a metric to classify foreground and background.

While these background modeling methods perform well under stationary camera conditions, they face challenges in dynamic camera scenarios. Motion compensation is required to eliminate the camera’s own movement. However, techniques such as homography transformations often fail to accurately compensate for complex motion, particularly in UAV views, limiting their effectiveness in these highly dynamic scenes.

B.

Feature point tracking methods

Feature point tracking methods first select feature points from image sequences and track their long-term motion trajectories for clustering, enabling motion segmentation of sparse feature points. Then, techniques such as Markov Random Fields are used for dense segmentation of all pixels [12,13]. The efficacy of these methods is intrinsically linked to the quality of the selected feature points. Insufficient selection can lead to missed detections, while excessive selection significantly increases computational demands, making it challenging to strike a balance between accuracy and real-time performance.

C.

Supervised deep learning methods

Moving object detection based on supervised deep learning leverages the powerful feature extraction capabilities of neural networks, treating the task as a classical object detection problem [14,15,16]. These networks are trained on meticulously labeled datasets of moving objects. However, analogous to appearance-based detection algorithms, this approach may not fully harness motion-specific cues. The accuracy of detection is highly susceptible to the resemblance between the test data and the training data distribution. Consequently, ensuring the network’s ability to generalize effectively across diverse and unseen scenes presents a significant challenge.

D.

Optical flow-based methods

Optical flow-based motion segmentation methods offer a compelling approach to identifying moving objects by leveraging the inherent motion representation embedded within video sequences. These methods initially compute optical flow fields, which quantify the motion of individual pixels across frames. Subsequently, moving objects are detected by analyzing the characteristics and patterns within these optical flow fields. Since optical flow estimation operates at the pixel level, these optical flow-based moving object detection methods can effectively exploit pixel-wise motion information, often demonstrating strong generalization capabilities. With improvements in optical flow estimation datasets and advancements in network architectures [17,18], inter-frame optical flow can now be estimated accurately in a short time. Optical flow-based motion detection has gained significant attention and holds great potential for further development.

2.2. Development of Optical Flow-Based Moving Object Detection Methods

Optical flow-based moving object detection methods can primarily be divided into two categories: sparse optical flow clustering methods and motion segmentation methods. Sparse optical flow clustering methods track selected feature points in the image using optical flow and classify them based on the distribution of feature point trajectories. In 2009, Sheikh [12] differentiated moving objects and backgrounds by utilizing 3D subspaces and the projection errors of feature point trajectories. In 2012, Kim [19] used the K-means algorithm to classify sparse optical flow points. Bugeau [20] developed a method based on feature point tracking and established a Markov random field model for dense segmentation. Sheikh [12] leveraged kernel density estimation to construct foreground and background distribution models, subsequently employing graph cuts on a MRF to achieve dense segmentation. Brox [21] introduced the Potts model, framing segmentation as a total variation problem and incorporating temporal constraints to ensure consistency across frames. Sparse optical flow clustering methods also face the challenge of initial feature point selection.

With the application of deep learning, the inference time for dense optical flow estimation has been significantly reduced, and more motion segmentation methods based on dense optical flow have been developed. Huang [22] utilized the RANSAC algorithm to estimate the homography transformation, thereby reconstructing the optical flow field. Moving objects were then differentiated by comparing this reconstructed flow field with the input optical flow. Sugimura [23] employed the Canny edge detector to identify moving object contours, selected seed points for the moving target and background inside and outside the contours, and then used the oneCut algorithm to segment moving objects in the optical flow field. In 2020, Zhang [24] proposed a method to detect moving objects that are hard to distinguish in dense optical flow by reconstructing the optical flow direction using the Poisson equation. Temporal consistency was then used to eliminate false positives. Zhang [25] then extended the robust seed point selection approach by using a random walk algorithm.

Thanks to its speed and strong feature representation capabilities, deep learning has increasingly been applied to motion segmentation tasks in recent years. For instance, ref. [26] employed convolutional neural networks (CNNs) to detect independently moving objects. This network was trained directly on annotated labels, taking optical flow maps as input. Ref. [27] introduced a novel approach utilizing slot attention to segment moving objects by analyzing the discrepancies between reconstructed and input optical flow fields. Ref. [28] explored an unsupervised learning paradigm, initializing their network with pre-trained semantic features and subsequently training it to discern motion patterns. MATNet [29] simultaneously utilizes motion and appearance features, employing an attention framework to achieve better segmentation results. FSNet [30] adopts a parallel-cross architecture, using cross-attention modules to continuously exchange useful clues between appearance and motion features. AMC-Net [31] assigns weights to appearance and motion features based on the evaluated importance. GWM [32] proposed approximating optical flow using a set of parametric models, each representing a distinct motion component. In 2022, Etienne Meunier [33] suggested that the input optical flow can be represented by a set of affine or polynomial motion models. This framework led to the development of a CNN-based segmentation network, which referred to the classic EM algorithm for unsupervised division of the input optical flow into distinct sections according to these motion models. In 2023, UST [34] argued that temporal consistency is also a crucial clue for optical flow segmentation. They designed a 3D network for motion segmentation, where the loss function combines a flow reconstruction term based on spatiotemporal motion models and a regularization term that enforces foreground mask temporal consistency. TMO [35] highlighted the issue of poor motion information quality in optical flow maps for certain scenes and proposed selectively utilizing motion clues. The framework aims to reduce reliance on motion features when the quality of the motion information is compromised.

Optical flow-based motion segmentation methods have shown significant potential for moving object detection, but we have noticed that few methods specifically incorporate domain-specific prior knowledge from the UAV scenarios.

2.3. Development of Moving Object Detection in UAV Scenarios

Murari [2] utilized a multi-level feature pyramid supervised network to detect moving objects in UAV videos. K. Berker [36] selected 14 sequences from the VIVID and UAV123 [37] tracking datasets, annotated the moving object with rectangular bounding boxes, and developed a detection algorithm based on homography compensation and differentiation for low-altitude UAV scenarios. Hazal [38] explores the use of sparse flow to enhance the real-time performance of moving object detection in low-altitude UAVs, with experiments also conducted on the VIVID and UAV123 tracking datasets. B. Kalantar [39] used trajectories of matched regional adjacency graphs for moving object detection. M. Siam [40] considered utilizing homography transformation to warp the input dense optical flow to mitigate the impact of motion parallax. J. Li [41] estimated the global background motion model using sparse optical flow, then compensated for it and performed background subtraction to detect UAVs in flight. Kimura [42] applied epipolar constraints and flow-vector bound constraints to differentiate moving objects from the background. Note that these epipolar constraints require camera intrinsic parameters, and accurate background subtraction demands prior knowledge of the target’s approximate speed. In real-world scenarios, obtaining such valuable prior knowledge can be difficult.

3. Approach

We consider using a spatiotemporal polynomial motion model [34] to classify the motion in the scene into two layers: background and true moving objects (foreground). Foreground sparsity and spatiotemporal consistency are incorporated into the loss function to constrain a 3D multi-scale U-Net network to learn and segment the true moving objects, helping to mitigate false positives caused by motion parallax and background motion.

3.1. Framework

Figure 2 illustrates the framework of the proposed unsupervised UAV moving object detection algorithm. We first use a dense optical flow estimation network to compute the dense optical flow ${\left\{f_{t+i}\right\}}_{i=0}^N$ between frames in the image sequence ${\left\{I_{t+i}\right\}}_{i=0}^{N+1}$, which is then fed into the segmentation network as input. In this paper, we use three consecutive optical flows, i.e., $N=2$. The classical 3D U-Net [43] architecture is employed as the segmentation network, with a 2.5D Atrous Spatial Pyramid Pooling (ASPP) module added at the bottleneck to enhance the network’s ability to capture multi-scale features. The backbone is flexible and can be substituted with any other suitable segmentation network. The network partitions the input optical flow (representing the overall motion in the scene) into $L=2$ layers, specifically the foreground layer ($l=1$) and the background layer ($l=2$), and outputs the probability ${\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N$ that each site $\left(x,y\right)$ belongs to layer $l$ (foreground or background). The pseudocode is provided in Appendix A.

Figure 2. Flowchart of the proposed method.

During the training step, after the network outputs the layer segmentation results ${\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N$, we aim to enforce consistency in motion patterns within each layer. To achieve this, we employ a motion model to represent the motion characteristics of each layer. This model aids in reconstructing the optical flow field, allowing for a comparison with the original input optical flow field and thus evaluating the accuracy of the layer segmentation. Following [34], we adopt a spatiotemporal parametric motion model ${\overline{f}}_t^{\left(x,y\right)}\left(\theta ,\lambda \right)$, as shown in Equation (1), to represent the motion of each layer.

$${\overline{f}}_t^{\left(x,y\right)}\left(\theta ,\lambda \right)=\left(\begin{array}{l}{\theta }_0+{\lambda }_{0}t+\left({\theta }_1+{\lambda }_{1}t\right)x+\left({\theta }_2+{\lambda }_{2}t\right)y,\\ {\theta }_3+{\lambda }_{3}t+\left({\theta }_4+{\lambda }_{4}t\right)x+\left({\theta }_5+{\lambda }_{5}t\right)y\end{array}\right),$$

(1)

where ${\overline{f}}_t^{\left(x,y\right)}\left(\theta ,\lambda \right)$ represents the pixel motion of the $\left(x,y\right)$ pixel location at time t, with the motion parameters ${\left\{{\theta }_{k,},{\lambda }_k\right\}}_{k=0}^5$.

This model consists of 12 parameters, with ${\left\{{\theta }_k\right\}}_{k=0}^5$ used to constrain spatial motion variations, and ${\left\{{\lambda }_k\right\}}_{k=0}^5$ to constrain temporal motion changes. We employ the L-BFGS algorithm [34,44] to compute the specific motion model parameters ${\left\{{\theta }_{k,}^l,{\lambda }_k^l\right\}}_{k=0}^5$ for both the foreground and background layers based on the network’s layer segmentation outputs. With these motion model parameters, we can reconstruct the optical flow field.

Next, by utilizing the difference between the reconstructed optical flow field and the input optical flow, along with the foreground sparsity and spatiotemporal consistency loss terms, we apply constraints to the network. This encourages the network to learn to differentiate between the foreground and background by categorizing the overall motion based on the differences in motion patterns, thereby enabling the detection of moving objects from the moving background. A detailed explanation of these loss functions will be provided in Section 3.2.

During the inference step, the layer probabilities ${\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N$ output by the network represent the desired detection results, and no further motion model estimation phase is required.

3.2. Loss Function

The network’s loss function consists of four components: a flow reconstruction term $L_r$, a spatial consistency term $L_{sc}$, a temporal consistency term $L_{tc}$, and a foreground sparsity term $L_{fs}$. The total loss $L$ is the sum of these four terms:

$$L=L_r+\alpha L_{sc}+\beta L_{tc}+\chi L_{fs},$$

(2)

where $\alpha$, $\beta$, and $\chi$ are the hyperparameters to balance the loss components.

3.2.1. Flow Reconstruction Loss

The flow reconstruction term is computed based on the L1 norm between the reconstructed flow field, obtained from the parameterized motion model, and the input optical flow. This term quantifies how well the motion model can represent the scene’s motion. According to reference [34], the flow reconstruction loss is given by:

$$L_r={\displaystyle \frac{1}{N+1}}{\displaystyle \sum _{i=0}^N\frac{{\displaystyle \sum _{\left(x,y\right)\in I}{\displaystyle \sum _{l=1}^L{p}_{t+i}^{\left(x,y\right),l}{‖f_{t+i}^{\left(x,y\right)}-{\overline{f}}_{t+i}^{\left(x,,,y\right)}\left({\theta }^l,{\lambda }^l\right)‖}_1}}}{{\displaystyle \sum _{\left(x,y\right)\in I}{‖f_{t+i}^{\left(x,,,y\right)}‖}_1}}},$$

(3)

where $I$ is the image grid, $p_{t+i}^{\left(x,y\right),l}$ denotes the probability of the $\left(x,y\right)$ site at time $t+i$ belonging to the layer $l$, which is the output of the motion segmentation network, $f_{t+i}^{\left(x,y\right)}$ is the input flow, and ${\overline{f}}_{t+i}^{\left(x,y\right)}\left({\theta }^l,{\lambda }^l\right)$ denotes the reconstructed flow derived from the motion model with the estimated parameter $\left\{{\theta }^l,{\lambda }^l\right\}$.

3.2.2. Spatial Consistency Loss

In the scenario of drone-based ground observation, the reconnaissance area in video mode is relatively small, and the target size in the image typically exceeds several tens of pixels. According to the Gestalt principle of common fate, elements of the same entity tend to exhibit similar motion patterns. The optical flow of each pixel within the same entity is spatially closely related, typically exhibiting similar motion amplitudes and directions. In the temporal dimension, with a frame rate of 24 fps, the temporal interval between frames is less than 0.05 s. The moving object’s motion is generally smooth and continuous over such a short period, without abrupt discontinuities. In the drone scenario, the moving targets can be simply modeled as rigid bodies, with no internal contraction or expansion. As shown in Figure 3, the motion of the moving target forms a cluster of prominent sites with spatiotemporal consistency in the optical flow field. That is, the mask of a moving object should exhibit strong spatiotemporal consistency.

Figure 3. An example of UAV sequence data. The first row displays the RGB images, with moving targets highlighted by red bounding boxes, while the second row presents the corresponding visualized optical flow maps.

Additionally, the smoothness constraint of optical flow [45] implies that the optical flow between pixels in an image experiences abrupt changes only at object boundaries, which occupy a small portion of the image. Overall, the optical flow field should remain smooth. Likewise, the predicted foreground mask of the moving target should demonstrate spatial smoothness, preserving spatial consistency across neighboring pixels. Inspired by the Ising model [1,46], we introduce the following spatial consistency loss $L_{sc}$, which may help eliminate point noise and false positives.

$$L_{sc}={\displaystyle \frac{1}{\left(N+1\right)\left|I\right|}}{\displaystyle \sum _{i=0}^N{\displaystyle \sum _{\left(x,y\right)\in I}\left(\left|p_{t+i}^{\left(x+1,y\right),l=1}-p_{t+i}^{\left(x,y\right),l=1}\right|+\left|p_{t+i}^{\left(x,y+1\right),l=1}-p_{t+i}^{\left(x,y\right),l=1}\right|\right)}}$$

(4)

3.2.3. Temporal Consistency Loss

As analyzed in Section 3.2.2, the motion of real moving objects is smooth, without abrupt changes in short time intervals. This implies that the segmentation results produced by the network should likewise display temporal consistency. Inspired by the work in [34], we introduce the following temporal consistency loss function:

$$L_{tc}={\displaystyle \frac{1}{2L\left|I\right|}}{\displaystyle \sum _{\left(x,y\right)\in I}{\displaystyle \sum _{l=1}^L\left(\left|p_{t+1}^{\left(x,y\right),l}-p_t^{\left(x,y\right),l}\right|+\left|p_{t+2}^{\left(x,y\right),l}-p_{t+1}^{\left(x,y\right),l}\right|\right)}}.$$

(5)

3.2.4. Foreground Sparse Loss

In the UAV-based moving object detection scenario, target sizes are typically small, often measuring around 20 × 40 pixels or smaller. This represents a mere 1/2000th of a standard 1920 × 1080 image, resulting in a highly sparse foreground comprising only these minute targets. To effectively exploit this valuable prior knowledge, we introduce a foreground sparsity loss function. This constraint guides the network to identify genuine moving targets only when their motion demonstrably diverges from the background. By enforcing this separation of foreground and background motion models, we aim to mitigate the confounding effects of motion parallax and the UAV’s own movement. The foreground sparsity loss is defined as follows:

$$L_{fs}={\displaystyle \frac{1}{\left(N+1\right)\left|I\right|}}{\displaystyle \sum _{i=0}^N{\displaystyle \sum _{\left(x,y\right)\in I}\left|p_{t+i}^{\left(x,y\right),l=1}\right|}}.$$

(6)

4. Experiments

4.1. Datasets

We utilized the DJI M3T and M3E drones to collect data for typical UAV moving objection detection scenarios. Captured scenes encompassed a variety of environments, including urban blocks, farmlands, lakesides, and country roads. The drones flew at an altitude of approximately 400 m, using focal lengths of 1X, 2X, 4X, and 7X to capture ground-based moving targets. The camera’s pitch angle included typical reconnaissance perspectives, such as 90°, 60°, and 30° downward views. Furthermore, drones operated in diverse modes, including forward reconnaissance, hovering, camera rotation, and focus tracking, to effectively capture the targets from various angles. The video image sequences were recorded at 30 fps, with a resolution of 1920 × 1080. The UAVMD dataset comprises 70 sequences totaling 31,536 images. For evaluation purposes, we annotated 37 representative short sequences, providing annotations every ten frames with polygonal bounding boxes. Figure 4 shows examples from the dataset, with annotated moving targets highlighted in red masks.

Figure 4. Samples of our UAVMD dataset.

The UAVMD dataset encompasses a diverse range of moving object types across multiple scales, encompassing motorcycles, tricycles, passenger cars, trucks, and more. During annotation, these moving targets were labeled as foreground objects without differentiation by type. The dataset incorporates targets with varying speeds and also accounts for incomplete moving targets, such as those affected by occlusion or entering/exiting the scene. And it is publicly accessible at https://github.com/fanxuxiang/UAVMD-sets-and-related-codes (accessed on 30 December 2024).

4.2. Implementation Details

For training, we utilized 23 sequences encompassing 24,849 images from the UAVMD dataset, with sequence lengths varying from hundreds to thousands of frames. An additional 10 sequences, containing 3218 images, were reserved for validation. The input optical flow was estimated using an improved RAFT (publicly accessible at https://github.com/fanxuxiang/LKCMA (accessed on 10 February 2025)). Note that the training and validation sets are completely unannotated. Annotations are reserved exclusively for testing purposes. The balance hyperparameters for various losses were set to $\alpha =0.05$, $\beta =1$, and $\chi =0.05$. Data augmentation was implemented following the strategy in [34], which randomly added global motion to enhance the diversity of the dataset. The network was trained on two RTX 4090 GPUs. During training, the flows were resized to 540 × 960. The Adam optimizer [47] was used, with a learning rate set to ${10}^{-4}$. InstanceNorm [33] was employed to enhance stability.

4.3. Evaluation Metrics

We use both pixel-level and object-level metrics for quantitative evaluation of the UAV moving object detection task.

The pixel-level evaluation metric, Jaccard index (JI), is defined as follows:

$$JI={\displaystyle \frac{pred\cap gt}{pred\cup gt}},$$

(7)

where $pred$ is the predicted foreground pixel-level mask output by the network, and $gt$ is the annotated foreground mask.

The evaluation metrics at the object level are based on $\mathrm{Precision}$ (Pr), $\mathrm{Recall}$ (Re), and comprehensive metric F1, as shown in Equation (8). The output mask is divided into different objects based on connectivity. When the Intersection over Union (IoU) between the detected object and the ground truth annotation exceeds a threshold $\sigma$, it is considered a correct detection. In this paper, $\sigma$ is set to 0.5.

$$\begin{array}{l}\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}\\ \mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}\\ \mathrm{F}1={\displaystyle \frac{2\mathrm{Precision} · \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}},\end{array}$$

(8)

where TP, FP, and FN represent the numbers of true positives, false positives, and false negatives, respectively.

4.4. Quantitative Results

To evaluate the effectiveness of the methods in UAV moving object detection, quantitative evaluations were conducted on the UAVDT test dataset. Table 1 presents a comparison between the proposed method and several classical methods, including the background modeling method DECOLOR [1], the video object segmentation method TMO [35], the unsupervised motion segmentation method EM [33], and UST [34]. The unsupervised methods were trained using the same methodology on the UAVDT data. Since only the test set is annotated, for supervised methods that require labeled data during training, we used the pre-trained weights provided in the original papers for comparison. The results demonstrate that the proposed method outperformed all other methods in terms of pixel-level JI, object-level precision, and F1 scores. Compared to the recently developed motion segmentation method UST, our method achieved a 35% improvement in JI, a 14% increase in precision, and a 9% improvement in the overall F1 score. Due to the stricter constraints imposed on foreground sparsity and spatial consistency during implementation, the recall rate of our algorithm did not show a significant improvement.

Table 1. Quantitative evaluation results on the UAVMD dataset. Due to memory and inference time limitations, the input image sequences were downsampled by a factor of 2 for DECOLOR method. The other methods were tested at the original 1080 × 1920 resolution.

Method	JI (%)	Pr (%)	Re (%)	F1 (%)
TMO [35]	21.01	21.16	19.87	20.49
EM [33]	20.83	35.02	59.40	44.06
DECOLOR [1]	45.85	44.36	68.58	53.88
UST [34]	39.07	53.94	61.33	57.40
Ours	53.01	61.85	63.44	62.64

The best evaluation results in the column are bolded.

Table 2 presents the test results on the BMS [48] dataset, a general moving object detection benchmark. Compared to our UAVDT dataset, the BMS dataset features larger target sizes, with most targets located near the center of the field of view. Since the annotations in the BMS dataset are not at the object level, evaluation was performed using pixel-level F1 scores. Note that the supervised method TMO’s training data align closely with the BMS dataset, and the BMS method itself underwent fine-tuned parameter selection specifically for the BMS dataset. Furthermore, our proposed method was not trained or validated on the BMS dataset. Notwithstanding these differences, our method achieved F1 scores comparable to those of the other methods, demonstrating strong cross-dataset generalization capabilities.

Table 2. Quantitative evaluation results on the BMS dataset.

Data	TMO [35]	BMS [48]	MD [49]	HOMO [50]	EM [33]	UST [34]	Ours
Car4	0.92	0.87	0.67	0.78	0.91	0.81	0.87
Car5	0.88	0.84	0.55	0.80	0.90	0.82	0.88
Car6	0.93	0.91	0.44	0.86	0.92	0.92	0.92
Car7	0.90	0.91	0.75	0.89	0.89	0.83	0.88
Seq. 1	0.53	0.83	0.58	0.66	0.70	0.73	0.71
Seq. 2	0.78	0.85	0.74	0.69	0.77	0.75	0.75
Seq. 5	0.67	0.63	0.40	0.46	0.30	0.49	0.60
Seq. 6	0.77	0.60	0.28	0.49	0.75	0.60	0.68
Seq. 7	0.93	0.90	0.19	0.85	0.93	0.92	0.93
Average	0.82	0.82	0.51	0.72	0.80	0.77	0.81

4.5. Qualitative Results

Figure 5 presents a comparison of the proposed method with several classical methods on examples from the UAVMD test set. As shown in Example No. 1 and No. 2, due to motion parallax and platform movements, the building edges and protruding poles are difficult to distinguish from genuine moving targets, leading to numerous false positives in methods such as EM, DECOLOR, and UST. In Example No. 3, homogeneous areas, such as road markings, also pose significant challenges for detecting real moving objects. The supervised VOS method TMO suffers from many missed detections when the distribution of the test data differs from its training data, resulting in inconsistent performance. By leveraging prior knowledge, including foreground sparsity and spatial consistency, the proposed method effectively mitigates these false positives. Additional qualitative results of the proposed method are provided in Appendix A.

Figure 5. Qualitative comparison results of different methods. The method names are listed below the images, and the detection results are displayed on the original image in the form of weighted red masks. Key regions are emphasized using red bounding boxes.

4.6. Ablation Results

Table 3 presents the ablation results of the main components of the proposed method on our UAVMD dataset. The baseline [34] uses a classic 3D U-Net architecture, incorporating flow reconstruction loss and temporal consistency loss. The term ‘+sb’ denotes the addition of foreground sparse loss, ‘+sp’ represents the addition of spatial consistency loss, and ‘Ours’ refers to the final version, which includes the foreground sparse loss, spatial consistency loss, and the ASPP component. As shown, the foreground sparsity constraint yields a 28% improvement in the JI and a 9% improvement in precision compared to the baseline, significantly reducing false positives caused by motion parallax and the platform’s own motion. The addition of the spatial consistency constraint and the multi-scale ASPP module further improves the JI, Pr, and F1 scores in the drone-based scenarios.

Table 3. Ablations of the designs.

Method	JI (%)	Pr (%)	Re (%)	F1 (%)
Baseline	39.07	53.94	61.33	57.40
Base + sb	50.09	58.92	62.46	60.64
Base + sb + sp	51.70	61.41	61.10	61.26
Ours	53.01	61.85	63.44	62.64

The best result in each metric is highlighted in bold.

4.7. Cross-Dataset Results

Figure 6 presents the test results of the proposed method across diverse scenarios and object sizes, noting that these data were completely unseen during training. The vehicles in the examples that were occluded were also successfully detected. The results demonstrate the strong cross-dataset generalization capability of our algorithm.

Figure 6. Illustration results on unseen sequences. Examples 1 and 2 are collected by ourselves, while No. 3 is sourced from the VIVID dataset [5], Example 4 from the DAVIS sequence [4], Examples 5–7 from the BMS dataset [48], and Example 8 from the CDNet 2014 dataset [51]. The detected moving objects are highlighted with red masks, with the target number and optical flow magnitude displayed as text. Arrows indicate the direction of the optical flow associated with the targets.

5. Conclusions

In this paper, we present a benchmark for UAV moving object detection. We introduce prior knowledge in UAV scenarios, including foreground sparsity and spatial consistency constraints, into the loss function and design a multi-scale unsupervised UAV moving object detection network based on optical flow. The network learns to detect moving objects by exploiting the differences in motion patterns under these constraints, significantly mitigating false positives caused by motion parallax and platform movement. Experimental results demonstrate the algorithm’s effectiveness and its cross-dataset generalization ability.

Appendix A.1. Pseudocode of the Proposed Method

Algorithm 1 presents the pseudocode of the proposed method. The image sequence ${\left\{I_{t+i}\right\}}_{i=0}^{N+1}$ is first processed by a pre-trained optical flow estimation network ${\varphi }_f$ to generate dense inter-frame optical flow ${\left\{f_{t+i}\right\}}_{i=0}^N$. During training, the computed optical flow is fed into a segmentation network ${\varphi }_s$ to estimate the probability of each pixel belonging to the foreground or background layer ${\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N$. The L-BFGS algorithm then optimizes motion model parameters ${\left\{{\theta }_{k,},{\lambda }_k\right\}}_{k=0}^5$ based on the segmentation results. These parameters are used to reconstruct the optical flow field, enabling computation of the optical flow reconstruction loss $L_r$. Additionally, spatial consistency loss $L_{sc}$, temporal consistency loss $L_{tc}$ and foreground sparse loss $L_{fs}$ are derived from the network’s segmentation output. The total loss $L$ is minimized via backpropagation using the Adam optimizer to update the segmentation network parameters. For inference, optical flows are directly processed by the trained segmentation network to produce detection results ${\left\{m_{t+i}\right\}}_{i=0}^N$.

Algorithm A1 Proposed Moving Object Detection method
1:	procedure FlowPrepare$({\left\{I_{t+i}\right\}}_{i=0}^{N+1},{\varphi }_f$)
2:	${\left\{f_{t+i}\right\}}_{i=0}^N={\varphi }_f\left({\left\{I_{t+i}\right\}}_{i=0}^{N+1}\right)$
3:	end procedure
4:	procedure TrainingStep$({\left\{f_{t+i}\right\}}_{i=0}^N,{\varphi }_s,\alpha ,\beta ,\chi$)
5:	${\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N=\mathrm{Softmax}\left({\varphi }_s\left({\left\{f_{t+i}\right\}}_{i=0}^N\right)\right)$
6:	${\left\{{\theta }_k,{\lambda }_k\right\}}_{k=0}^5=ComputeParasLBGFS\left({\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N,{\left\{f_{t+i}\right\}}_{i=0}^N\right)$
7:	$\mathrm{Compute}{L}_r$$\mathrm{with} {\left\{{\theta }_k,{\lambda }_k\right\}}_{k=0}^5$$,{\left\{f_{t+i}\right\}}_{i=0}^N$$\mathrm{and}{\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N$ (Equation 3)
8:	$\mathrm{Compute}{L}_{sc}$$,L_{tc}$$\mathrm{and}{L}_{fs}$$\mathrm{with} {\left\{p_{t+i}^{\left(x,y\right),l}\right\}}_{i=0}^N$  (Equations 4, 5 and 6)
9:	$L=L_r+\alpha L_{sc}+\beta L_{tc}+\chi L_{fs}$
10:	$update\left({\varphi }_s\right)$
11:	end procedure
12:	procedure InferenceStep$({\left\{f_{t+i}\right\}}_{i=0}^N,{\varphi }_s$)
13:	${\left\{m_{t+i}\right\}}_{i=0}^N=\mathrm{Argmax}\left({\varphi }_s\left({\left\{f_{t+i}\right\}}_{i=0}^N\right)\right)$
14:	end procedure

Appendix A.2. Example of Occlusion Scenarios

Figure A1 presents the test results of the proposed method in various occlusion scenarios. These include cases where moving objects are entering or exiting the scene, as well as instances of occlusion by trees or shadows. Since the method relies minimally on semantic cues, it effectively detects incomplete moving objects in these examples.

Figure A1. Test results in scenes with partial occlusion. Examples from the UAVMD test set.

Appendix A.3. Example of Occlusion Scenarios

Figure A2 illustrates the continuous detection results in video sequences using the proposed method. By incorporating spatiotemporal consistency constraints, the method effectively leverages temporal cues, enabling continuous detection of moving targets across time in the example sequences. The effective detection results suggest the potential of the method for downstream applications such as target localization, velocity estimation, and tracking.

Figure A2. Illustration of the continuous detection in time. Examples are drawn from the UAVMD test set.