NightTrack: Joint Night-Time Image Enhancement and Object Tracking for UAVs

Highlights

What are the main findings?

• We propose NightTrack, a unified framework that jointly optimizes low-light enhancement and object tracking, outperforming state-of-the-art methods in night-time UAV scenarios.
• By introducing Pyramid Attention Modules (PAMs) and jointly estimating illumination and noise, the framework significantly enhances the discriminability of features in low-light conditions.

What is the implication of the main finding?

• The unified paradigm of integrating enhancement and tracking offers a more effective solution for tasks with competing objectives than traditional two-stage pipelines.
• The proposed framework substantially improves the robustness and precision of night-time UAV tracking, presenting a novel perspective to enable practical applications in challenging low-light environments.

Abstract

UAV-based visual object tracking has recently become a prominent research focus in computer vision. However, most existing trackers are primarily benchmarked under well-illuminated conditions, largely overlooking the challenges that may arise in night-time scenarios. Although attempts exist to restore image brightness via low-light image enhancement before feeding frames to a tracker, such two-stage pipelines often struggle to strike an effective balance between the competing objectives of enhancement and tracking. To address this limitation, this work proposes NightTrack, a unified framework that optimizes both low-light image enhancement and UAV object tracking. While boosting image visibility, NightTrack not only explicitly preserves but also reinforces the discriminative features required for robust tracking. To improve the discriminability of low-light representations, Pyramid Attention Modules (PAMs) are introduced to enhance multi-scale contextual cues. Moreover, by jointly estimating illumination and noise curves, NightTrack mitigates the potential adverse effects of low-light environments, leading to significant gains in precision and robustness. Experimental results on multiple night-time tracking benchmarks demonstrate that NightTrack outperforms state-of-the-art methods in night-time scenes, exhibiting strong promises for further development.

1. Introduction

Unmanned aerial vehicles (UAVs) have served as a crucial application platform for visual object tracking. Benefiting from their small size, high maneuverability, and ease of operation, UAVs are particularly well-suited for tracking missions and hold significant application value in areas such as intelligent surveillance [1] and autonomous driving [2]. Driven by the availability and applicability of large-scale datasets [3,4,5], supervised learning tasks utilizing manually annotated labels have made many breakthroughs in visual object tracking research [6,7,8,9,10,11]. Performance requirements for trackers are becoming increasingly stringent with the evolution of modern technology. A key driver is that a wealth of high-value data and critical operations are exclusive to nocturnal settings, rendering trackers that rely solely on daytime data inadequate for high-stakes application scenarios. Consequently, modern applications demand that trackers not only exhibit high precision, robustness, and real-time capability in standard night-time scenarios but also maintain sustained and reliable performance under extreme conditions, such as extremely low illumination. This demand is particularly pronounced in several key domains, notably wildlife ecology and autonomous systems. In wildlife ecology, for instance, many species are nocturnal and their key behaviors—such as hunting and migration—are hidden from daylight observation. Therefore, accurate night-time tracking is crucial for obtaining valid ecological data and implementing effective conservation strategies. In the domain of autonomous systems, the operational envelope is expanding to encompass 24/7 all-weather operations. For instance, UAVs are deployed for nocturnal inspection of critical infrastructure and search-and-rescue missions, while autonomous vehicles must navigate safely on roads with poor or non-existent lighting. In both cases, system safety and mission success are fundamentally dependent on the ability to accurately perceive and interpret the environment under adverse visual conditions.

To date, the visual tracking domain has largely evaluated under favorable lighting, where discriminative features are readily extracted by simple networks. In sharp contrast, real-world night-time images suffer from severe quality degradations, including low contrast, insufficient illumination, and color distortion, which result in a significant loss of detail. Such degradation severely impairs tracker performance and challenges the practical deployment of UAVs [12]. A conventional solution is an “enhance-then-track” pipeline, which first restores image quality using low-light enhancement methods [13,14,15] before executing tracking. Despite its effectiveness, this approach has a fundamental limitation: enhancement and tracking are separate, uncorrelated modules. This means that improvements in visual quality do not translate to tracking-specific discriminability, hindering their collaborative optimization.

To address this important problem, this paper propose a novel method that integrates low-light image enhancement and single object tracking within an end-to-end framework, called NightTrack, to achieve stable tracking in challenging low-light scenarios. Specifically, NightTrack embeds an enhancement algorithm directly in the tracking pipeline, which is optimized to boost tracking performance as well as visual aesthetics conjunctively. Pyramid Attention Modules (PAMs) are designed to combat image quality degradation. Each PAM aggregates multi-scale channel context to help capture and represent local-global dependencies simultaneously, improving the discriminative power of crucial features. Additionally, a Retinex theory-based non-linear projection is employed to estimate and map the illumination and noise curves to a set of normal-light images, effectively addressing the problems caused by degradation factors such as noise in low-light images.

As illustrated in Figure 1, the proposed NightTrack outperforms state-of-the-art (SOTA) tracking baselines. It forms a novel end-to-end framework that organically combines low-light image enhancement with single object tracking, aiming to boost tracking performance in challenging night-time environments. The proposed tracker is a generic model capable of following any object instance designated by an initial bounding box, independent of predefined object categories. This design ensures high flexibility and applicability across a wide range of scenarios.

Figure 1. Performance comparison on the UAVDark135 [16] benchmark. The results show that NightTrack is able to complete night-time UAV tracking with satisfactory performance.

The main contributions of this work are summarized as follows:

(1)

A novel end-to-end framework that jointly optimizes low-light image enhancement and object tracking, which significantly improves tracking performance at night.

(2)

Pyramid Attention Modules (PAMs) that each aggregate multi-scale contextual information, thereby substantially enhancing the underlying network’s representation capability and facilitating better detail restoration in night scenes.

(3)

An integration of Retinex theory-based illumination and noise curves within the enhancement stage, coupled with a no-reference loss function that enables unsupervised learning of the feature transformation without paired data, counteracting the degradation caused by low-light conditions.

(4)

A systematic experimental evaluation of the proposed joint optimization method, demonstrating its SOTA performance on multiple night-time tracking benchmarks, outperforming baseline tracking networks.

2. Related Work

2.1. Low-Light Image Enhancement Methods

Conventional low-light image enhancement approaches can be classified into two primary paradigms. The first paradigm, as described in [17,18], seeks to achieve illumination enhancement by expanding an image’s dynamic range, which is accomplished through the manipulation of its histogram distribution at both global and local scales. The second is founded upon the Retinex theory [19]. Its fundamental principle involves decomposing an image into a reflection component and an illumination component, based on the assumption that the former is invariant under varying illumination conditions. Consequently, the enhancement task is reformulated as an illumination estimation problem, a strategy having been adopted in numerous studies [20,21,22]. However, these conventional approaches are beset by significant limitations. Their intricate optimization processes and protracted computational times preclude their integration into real-time tracking systems deployed on UAVs. Furthermore, a common shortcoming of Retinex-based models is the frequent omission of noise considerations, an oversight that results in the preservation or amplification of noise within the enhanced output, subsequently impairing the performance of downstream applications.

In recent years, deep learning-based methods have demonstrated remarkable success in numerous low-level vision tasks. An early example is LLNet [23] that applies deep learning to low-light image enhancement, establishing a deep autoencoder-based method that simultaneously performs contrast enhancement and denoising. This approach exhibits superior robustness and stability compared to traditional methods. Also, Retinex-Net [24] integrates the Retinex model with deep learning, introducing an end-to-end trainable network for both decomposition and enhancement. Specifically, it employs a decomposition network to separate the illumination and reflection components and subsequently, utilizes an enhancement network to adjust the illumination map. KinD [25] offers a design with three sub-networks for layer decomposition, reflectance restoration, and illumination adjustment, respectively. However, its reliance on three separate deep neural networks results in high computational cost and poor real-time performance. In contrast to these image reconstruction-based methods, Zero-DCE [26] presents a lightweight model for low-light enhancement, by outputting high-order curves to perform pixel-wise adjustments on the input’s dynamic range. As such, it effectively formulates the enhancement task as a curve estimation problem, supported with experimental assessments. The image-to-curve mapping requires only a lightweight network, enabling fast inference and making it suitable for integration into UAV tracking algorithms.

2.2. Image Enhancement for Downstream Vision Tasks

Whilst powerful when applicable, deep convolutional neural networks are highly sensitive to image quality degradation factors such as color distortion, blur, and noise. Consequently, the performance of downstream tasks often degrades when processing real-world video images, which are not always of high quality. A simple solution might be to reinforce the performance of downstream vision tasks, such as object detection [27], image segmentation [28], and object tracking [13,14,15], by applying low-light image enhancement. Darklighter [13] constructs a lightweight curve estimation network to alleviate the adverse effects of inadequate lighting and noise on UAV tracking in night-time scenes. SCT [14] applies the Transformer [29] and curve projection model to train in a task-inspired manner, achieving stable night tracking. PANet [15] considers both human visual perception and specific requirements of UAV tracking, highlighting the targets’ latent feature. The basic idea of these three methods is to use a plug-and-play enhancer as a preprocessing step for UAV tracking. However, this generally meets limited success because traditional enhancement methods primarily focus on improving the visual perceptual quality of images, rather than optimizing features for a specific task.

Recently, researchers have considered integrating enhancement algorithms with downstream tasks, adopting end-to-end frameworks to learn effective feature representations from the synergistic tasks, thereby strengthening the overall performance. For instance, Guo et al. has proposed an end-to-end network for object detection in night-time scenes [30]. By jointly training a dynamic image enhancement network with a detector based on Faster R-CNN [31], the resultant system significantly improves the detection performance under low-light conditions. Also, DANNet [32] employs adversarial learning to train a sub-network for image brightness restoration, mitigating the impact of low-light environments on the segmentation network. Similarly, ADTrack, a dual regression algorithm [12], has been introduced on the basis of correlation filtering, which simultaneously trains a context filter and a target-focused filter, greatly enhancing the robustness of UAV tracking at night.

3. Methodology

3.1. Overall Framework

As previously discussed, low-light image enhancement can be used to restore image quality and serve as a plug-and-play module within a tracker, effectively improving tracking performance. However, such a decoupled paradigm essentially treats enhancement and tracking as two separate tasks that do not collaborate effectively. The tracker is constrained by the enhancer because the enhanced features are not necessarily optimal for the object tracking. To overcome this limitation, an end-to-end joint training framework, NightTrack, is proposed for concurrent image enhancement and object tracking. It aims to enhance the robustness of low-light UAV object tracking by improving image features, rather than pursuing perceptual quality.

The overall framework of the proposed network, as illustrated in Figure 2, primarily consists of two stages: a low-light image enhancement stage and a tracking stage. For a given low-light image, NightTrack first employs an image (patch-wise) enhancement algorithm to iteratively output m clear images under normal illumination. Subsequently, the features of each enhanced image are fed into the tracker’s backbone network, enabling the system to adaptively select the most effective iteration step. Through end-to-end joint optimization of the enhancer and tracker, the system simultaneously learns illumination-robust representations and discriminative correlation filters, from night-time UAV videos.

Figure 2. Proposed end-to-end framework. The left side shows the enhancement stage, and the right side shows the tracking stage.

To suppress any quality degradation in low-light images caused by noise (and possibly, other adverse factors), a novel design of Pyramid Attention Modules (PAMs) are embedded within the enhancer. Furthermore, a non-linear projection mechanism is constructed based on the Retinex theory [19]. This mechanism estimates the illumination and noise curves to map the image to a normal exposure, allowing for the simultaneous optimization of image enhancement and denoising. A set of non-reference loss functions is also introduced during the training phase. This is necessary considering the scarcity of night-time UAV tracking datasets, which are insufficient to support the full training of an end-to-end network, while recognizing that manual annotation for such datasets is time-consuming, labor-intensive, and costly. On one hand, the involvement of these losses helps drive the enhancer with the objectives of brightness preservation and structural fidelity. On the other hand, through a joint optimization strategy, they facilitate the synergistic update of the enhancer and tracker. This approach strengthens the visual quality of low-light images while maximally preserving semantic information, ensuring that the tracker’s backbone can extract discriminative features and maintain robust localization accuracy in night-time scenes.

In implementation, the proposed method primarily consists of two sub-networks: an enhancement sub-network and a tracking sub-network. In the enhancement stage, a simple yet effective architecture is employed to restore image brightness, progressively reconstructing the low-light image in an iterative manner. The restored features from each iteration are fed into the tracker’s backbone, where weights are adaptively computed based on their contribution to the subsequent tracking task. This quantifies the importance of the m-th iteration, enabling the model to select the most effective features. To guide the network to focus on the target region, each PAM mines multi-scale spatial context along the channel dimension. This enhances multi-scale representations while effectively suppressing noise and redundancies that are irrelevant for tracking.

In the tracking stage, the proposed framework utilizes HipTrack [6] as the baseline, employing the enhanced images for tracking. This joint design permits gradient backpropagation from the tracking loss to the enhancer, facilitating end-to-end training. Such a synergy not only generates higher-quality images but also markedly improves tracking precision and robustness in low-light conditions (as empirically shown later through comprehensive experimental investigations).

3.2. Enhancement Stage

Figure 3 illustrates the structure of the enhancement sub-network. It utilizes a U-Net backbone [33] and incorporates Pyramid Attention Modules (PAMs), to endow the model with stronger global context awareness. Specifically, for a given low-light image $\mathbf{I}\in {\mathbb{R}}^{H\times W\times 3}$, its hierarchical feature representations are extracted sequentially through K encoders. The constitution of each encoder includes an encoding block, a downsampling layer, and a PAM. Except for the first encoder, which outputs a default number (32) of channels, each subsequent encoder doubles the number of channels while halving the spatial resolution, i.e., $h={\displaystyle \frac{H}{2^K}}$, $w={\displaystyle \frac{W}{2^K}}$.

Figure 3. Main structure of the enhancement sub-network. By estimating the illumination curve and noise curve and then performing nonlinear projection, input images are enhanced to meet the requirements of feature extraction for UAV tracking.

To realize global context aggregation, the downsampled feature maps are fed into each PAM. This module is designed to learn rich cross-scale representations over multiple spatial scales, thereby effectively mitigating the propagation of redundant features generated by the enhancement stage. During the decoding phase, the spatial resolution is progressively restored through upsampling. A Tanh activation function [34] is applied after the final two convolutional layers to produce the estimated illumination curve $\mathbf{L}\in {\mathbb{R}}^{H\times W\times 3}$ and the noise curve $\mathbf{N}\in {\mathbb{R}}^{H\times W\times 3}$, respectively. Finally, based on the Retinex theory [19], the resulting low-light image is mapped into a set of enhanced images ${\mathbf{Y}}_m\in {\mathbb{R}}^{H\times W\times 3}$. Adaptive weights are then computed, based on these enhanced images to achieve an optimal match between the enhancement results and the tracking objective, improving night-time tracking performance. The technical details implementing this enhancement stage are presented below.

3.2.1. Pyramid Attention Module

UAV images typically possess high spatial resolution, whereas existing low-light image enhancement methodologies are primarily designed for smaller images. Consequently, the direct application of such enhancement networks to high-resolution UAV images presents two principal limitations. Firstly, the network’s receptive field is constrained, impeding the effective modeling of contextual information between different regions. Secondly, directly scaling the image may lead to the degradation or loss of salient details. To mitigate these issues, the present work, inspired by [35], introduces channel attention-based pyramid pooling modules to improve the system’s capacity for global information modeling, strengthening its ability to capture inter-channel dependencies.

As illustrated in Figure 4, each PAM is composed of four parallel branches, which employ average pooling operations with kernel sizes of $1\times 1$, $2\times 2$, $3\times 3$, and $6\times 6$, respectively. This design allows for the extraction of hierarchical representations at multiple scales. These multi-scale features not only encompass rich contextual information but also preserve fine-grained details within the image. The feature maps from each scale are upsampled to the original spatial resolution via bilinear interpolation and then, are concatenated along the channel dimension to obtain the global context feature ${\mathbf{f}}_g\in {\mathbb{R}}^{h\times w\times 3}$. Algorithm 1 summarizes the main process of the proposed module.

Figure 4. Structure of Pyramid Attention Module (PAM): A pyramid parsing module is applied first to capture feature representations at different scales, with resulting features concatenated after upsampling to obtain composite feature map containing multiscale contextual information; attention weight is then separately extracted for different scale feature maps, with weights recalibrated by Softmax [36] and multiplied by corresponding feature maps. Refined feature maps are returned with multi-scale channel information.

Inspired by the work presented in [37], a multi-scale channel attention mechanism is proposed to effectively extract channel attention weights and enhance the utilization of contextual information. The mechanism first uniformly partition the global feature ${\mathbf{f}}_g$ along the channel dimension into S sub-features, denoted as ${\mathbf{f}}_i\in {\mathbb{R}}^{h\times w\times {\displaystyle \frac{C}{S}}},i=0,1,\cdots ,S-1$. Then, a global average pooling is performed to generate channel estimation:

$${\mathbf{z}}_i={\displaystyle \frac{1}{h\times w}}\sum _{i=1}^h\sum _{j=1}^w{\mathbf{f}}_i(x,y),i=0,1,\cdots ,S-1$$

(1)

Algorithm 1 Pyramid Attention Module (PAM) algorithm

Input: Feature map $\mathbf{f}$ Parameter: Set the pooling sizes: $pool\_size=\left[1,2,3,6\right]$ Define $original\_size$ is the size of $\mathbf{f}$, S is the number of sub-features Output: The feature map of the K-th encoder ${\mathbf{F}}^K$  1: for i in $pool\_size$ do  2:    ${\mathbf{f}}_{a\_i}\leftarrow \mathrm{AdaptiveAvgPool}2\mathrm{d}\left(\mathbf{f},i\right)$.  3:    ${\mathbf{f}}_{u\_i}\leftarrow \mathrm{Upsampling}\left({\mathbf{f}}_{a\_i},original\_size\right)$.  4: end for  5: ${\mathbf{f}}_g\leftarrow \mathrm{concat}\left({\mathbf{f}}_{u\_1},{\mathbf{f}}_{u\_2},{\mathbf{f}}_{u\_3},{\mathbf{f}}_{u\_6}\right)$.  6: ${\mathbf{f}}_0,{\mathbf{f}}_1,\cdots ,{\mathbf{f}}_{S-1}\leftarrow \mathrm{split}\left({\mathbf{f}}_g,S\right)$.  7: for ${\mathbf{f}}_i$ in $\left[{\mathbf{f}}_0,{\mathbf{f}}_1,\cdots ,{\mathbf{f}}_{S-1}\right]$ do  8:    Compute ${\mathbf{z}}_i$, ${\mathbf{w}}_i$, ${\mathbf{att}}_i$ by Equation (1), Equation (2), Equation (3), respectively.  9:    ${\mathbf{Out}}_i={\mathbf{f}}_i\odot {\mathbf{att}}_i$. 10: end for 11: ${\mathbf{F}}^K=Cat\left([{\mathbf{Out}}_0,{\mathbf{Out}}_1,\cdots ,{\mathbf{Out}}_{S-1}]\right)$. 12: return ${\mathbf{F}}^K$

Subsequently, a non-linear mapping is constructed through two fully connected (FC) layers to represent the inter-channel dependencies. The attention weights are computed as follows:

$${\mathbf{w}}_i=\sigma \left(F{C}_1\left(\delta \left(F{C}_0\left({\mathbf{z}}_i\right)\right)\right)\right),i=0,1,\cdots S-1$$

(2)

where $F{C}_0\in {\mathbb{R}}^{C\times {\displaystyle \frac{C}{r}}}$ and $F{C}_1\in {\mathbb{R}}^{{\displaystyle \frac{C}{r}}\times C}$ represent the fully connected (FC) layers, $\delta$ and $\sigma$ denote the ReLU and Sigmoid activation functions, respectively. The rationale for such a design is based on the observation that combining linear information between channels is generally of benefits in modeling: facilitating information interaction across high-dimensional and low-dimensional spaces. To achieve the optimization of attention weights among different sub-features, the proposed method employs the Softmax function to normalize and recalibrate the initial attention weights:

$${\mathbf{att}}_i=\mathrm{Softmax}\left({\mathbf{w}}_i\right)={\displaystyle \frac{exp\left({\mathbf{w}}_i\right)}{{\displaystyle \sum _{i=0}^{S-1}}exp\left({\mathbf{w}}_i\right)}}$$

(3)

where ${\mathbf{w}}_i$ is the attention weight from ${\mathbf{f}}_i$, and the Softmax function is used to compute the recalibrated weight ${\mathbf{att}}_i$. This operation not only supports information interaction among the multi-scale channel attentions but also enhances the correlation between local and global features. Subsequently, the calibrated attention weights ${\mathbf{att}}_i$ are multiplied with their corresponding sub-features ${\mathbf{f}}_i$ to obtain the weighted features:

$${\mathbf{Out}}_i={\mathbf{f}}_i\odot {\mathbf{att}}_i,i=0,1,\cdots ,S-1$$

(4)

where ⊙ denotes the element-wise multiplication. In summary, the proposed novel PAMs can effectively fuse multi-scale channel attention information, improving the enhancer’s utilization of channel features from different regions and enhancing the discriminative power of key features. This makes the overall system more stable and robust in complex scenarios.

3.2.2. Curve Projection Based on Retinex Theory

In the Retinex model [19], an input image $\mathbf{I}\in {\mathbb{R}}^{H\times W\times 3}$ can be regarded as the pixel-wise product of an illumination component $\mathbf{L}\in {\mathbb{R}}^{H\times W\times 3}$ and a reflection component $\mathbf{R}\in {\mathbb{R}}^{H\times W\times 3}$:

$$\mathbf{I}=\mathbf{L}\odot \mathbf{R}$$

(5)

where $\mathbf{L}$ represents the illumination intensity of the scene, $\mathbf{R}$ is determined by the intrinsic properties of the objects themselves and is independent of the illumination, and the symbol ⊙ denotes element-wise multiplication.

Since the reflection component is not affected by illumination, it is typically considered as the enhanced result. However, in low-light imaging, the original frame is often corrupted by significant noise. Direct application of the decomposition will lead to concurrent amplification of noise within the resultant reflection. To overcome this problem, the work of [22] has introduced a noise term $\mathbf{N}\in {\mathbb{R}}^{H\times W\times 3}$ into the classical Retinex model:

$$\mathbf{I}=\mathbf{L}\odot \mathbf{R}+\mathbf{N}$$

(6)

Once the illumination component $\mathbf{L}$ and the noise component $\mathbf{N}$ are estimated, the reflection component $\mathbf{R}$ can be decomposed from the input image $\mathbf{I}$, such that

$$\mathbf{R}=\left(\mathbf{I}-\mathbf{N}\right)\oslash \mathbf{L}$$

(7)

where ⊘ denotes pixel-by-pixel division. To avoid the case where the divisor is 0, let $\mathbf{E}=\mathbf{1}\oslash \mathbf{L}$. Thus, the above formula is reformulated as follows:

$$\mathbf{R}=\left(\mathbf{I}-\mathbf{N}\right)\odot \mathbf{E}$$

(8)

As with Zero-DCE [26], low-light image enhancement can be formulated as a pixel-wise high-order curve mapping problem. Given a low-light image $\mathbf{I}$, it adjusts the dynamic range of the input image pixel-wise across M iterations, by predicting a set of nonlinear mapping coefficients:

$${\mathbf{Y}}_m\left(x\right)={\mathbf{Y}}_{m-1}\left(x\right)+{\mathbf{A}}_m\left(x\right){\mathbf{Y}}_{m-1}\left(x\right)\left(\mathbf{1}-{\mathbf{Y}}_{\mathbf{m}-\mathbf{1}}\left(\mathbf{x}\right)\right)$$

(9)

where ${\mathbf{Y}}_m\left(x\right)$ represents the brightness response after the m-th iteration, and ${\mathbf{A}}_m\left(x\right)$ is a parameter matrix with the same dimensionality as the input in which each pixel independently reflects the mapping intensity learned.

The curve projection model obtained so far does not explicitly account for noise. This may result in the amplification of image noise during the dynamic range stretching, severely reducing the robustness of the UAV tracker. To suppress noise concurrently during low-light enhancement, the proposed framework introduces a noise component into the curve mapping model. By estimating the noise distribution of the low-light image, a denoising pre-processing step is applied to the iterative output:

$${\widehat{\mathbf{Y}}}_{m-1}\left(x\right)={\mathbf{Y}}_{m-1}\left(x\right)-\mathbf{N}\left(x\right)$$

(10)

$${\mathbf{Y}}_m\left(x\right)={\widehat{\mathbf{Y}}}_{m-1}\left(x\right)+\mathbf{E}\left(x\right){\widehat{\mathbf{Y}}}_{m-1}\left(x\right)\left(\mathbf{1}-{\widehat{\mathbf{Y}}}_{m-1}\left(x\right)\right)$$

(11)

where ${\mathbf{Y}}_m\left(x\right)$ is the iterative result of the m-th illumination estimation. Empirically, the number of iterations can be set to $\mathrm{M}=8$, that is, $m\in \left[1,8\right]$.

To consider worst case scenarios, the loss of the enhancement stage is herein defined as the summation of losses of from all iterative outputs:

$${\mathcal{L}}^{Enhance}=\gamma ·\sum _{m=1}^M{\mathcal{L}}_m^{Enhance}$$

(12)

where the factor $\gamma$ is used to balance the enhancement loss against the subsequent tracking loss within the overall objective function (see Appendix A.1 for details). Under the guidance of this loss, the enhancement sub-network can adaptively estimate the optimal illumination and noise curves. This mechanism effectively prevents the amplification of noise in dark areas and helping ensure the stability of UAV tracking.

3.2.3. Loss Functions

Center exposure intensity loss. For UAV tracking, the low-light enhancer is typically used to restore the template patch and search patch. Since the tracked target is always located in the center of the template patch and generally appears in the center of the search patch, the proposed method places greater emphasis on the enhancement effect on the central area of an image. To ensure that the image brightness is controlled within a reasonable range, a spatial weight map $\mathbf{W}$ is utilized to draw the enhancer’s attention to the central area. Specifically, in implementation, the enhanced image is first divided into T non-overlapping patches each with a size of $16\times 16$ and the average intensity value of each patch is approximated as the well-illumination value $\overline{s}$ (which is empirically set to 0.6). Let $\mathbf{D}$ denotes a matrix composed of the average intensity values of the patches arranged in corresponding positions. Then, the center-focused lightness loss is constructed as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_m^{cen}& ={\displaystyle \frac{1}{T}}{∥\mathbf{W}\left(\mathbf{D}-\overline{s}·\mathbf{1}\right)∥}_2^2,\hfill \\ \hfill \mathbf{W}& =ln\left(e+\sqrt{i^2+j^2}\right)\hfill \end{array}$$

(13)

where i and j indicate the spatial position of the corresponding patches, and e is Euler constant.

Illumination estimation loss. To maintain the monotonic relations between adjacent pixels, this work introduces illumination regularization loss to constrain the smoothness of the fitted curve parameter mapping $\mathbf{E}$:

$${\mathcal{L}}_m^{ill}={∥\nabla \mathbf{E}∥}_2^2$$

(14)

where ∇ denotes the first-order differential operator.

Color balance loss. For the sake of correcting any potential color deviation in enhanced images, color intensity loss is added to establish a connection between the three channels, limiting their intensities to a relatively consistent level. The color intensity loss is calculated as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_m^{col}={\displaystyle \frac{1}{W\times H}}{∥{\mathbf{Y}}^p-{\mathbf{Y}}^q∥}_2^2,s.t.,\left(p,q\right)\in \left\{\left(r,g\right),\left(r,b\right),\left(g,b\right)\right\}\end{array}$$

(15)

where ${\mathbf{Y}}^t$ is the value of the t-th channel of $\mathbf{Y}$.

Noise estimation loss. The image sequence obtained in inadequate lighting situations often suffers from severe quality degradation, such as color distortion. Effective feature extraction is therefore vital in performing high-level tasks like object tracking, since a large amount of noise may badly mislead the tracker. For this reason, a noise estimation loss is employed to suppress the noise component $\mathbf{N}$:

$${\mathcal{L}}_m^{noi}={∥\mathbf{N}∥}_2^2$$

(16)

Enhanced Iteration Loss. To summarize, the enhanced loss for each iteration can be expressed as:

$${\mathcal{L}}_m^{enhance}={\lambda }_{cen}{\mathcal{L}}_m^{cen}+{\lambda }_{ill}{\mathcal{L}}_m^{ill}+{\lambda }_{col}{\mathcal{L}}_m^{col}+{\lambda }_{noi}{\mathcal{L}}_m^{noi}$$

(17)

where ${\lambda }_{cen}$, ${\lambda }_{ill}$, ${\lambda }_{col}$, and ${\lambda }_{noi}$ are coefficients set to adjust the contributions of each loss function (see Appendix A.2 for details).

3.3. Tracking Stage

In the proposed NightTrack, the tracker is built upon HipTrack [6], while the enhancer is embedded into the template and search branches, forming an end-to-end joint optimization framework. Specifically, the enhancer is strategically employed as a preprocessing module, embedded before the backbone networks of both the template and search branches. This design ensures that the backbone networks consistently receive a high-quality input. For a given raw template and search patch, the enhancer first generates M sets of iteratively enhanced images. These M sets are then sequentially fed into the tracker’s backbone to facilitate the extraction of multiple feature sets. During the training phase, backpropagation is performed regarding a weighted loss, where the weighting scheme is directly computed from the enhancement loss of each iteration, as follows:

$${\omega }_m=\left(1-{\displaystyle \frac{{\mathcal{L}}_m^{Enhance}}{{\displaystyle \sum _{k=1}^M}{\mathcal{L}}_k^{Enhance}}}\right)·{\displaystyle \frac{M}{M-1}}$$

(18)

with ${\mathcal{L}}_m^{Enhance}$ representing the loss associated with the t-th iteration. A larger weight is thus assigned to an enhanced image, indicating its higher utility in enhancing the tracker’s performance.

The tracking loss adopts the standard loss function used in the original HipTrack [6], which is calculated as follows:

$${\mathcal{L}}^{Track}={\lambda }_{giou}{\mathcal{L}}_{giou}+{\lambda }_{l1}{\mathcal{L}}_{l1}+{\lambda }_{loc}{\mathcal{L}}_{location}$$

(19)

where ${\lambda }_{giou}$, ${\lambda }_{l1}$, and ${\lambda }_{loc}$ serve as pre-defined hyperparameters that weight the contribution of each individual loss component.

Concerning the joint training of the enhancement and tracking sub-networks, the enhancement loss ${\mathcal{L}}^{Enhance}$ is incorporated into the tracking loss function. Such a design allows the enhancer to learn sample-specific illumination transformations throughout the training process, consequently leading to the generation of more discriminative feature representations, elevating the tracking robustness. By means of this joint optimization, a synergistic interplay is established between the enhancement and tracking tasks. The total loss of the joint optimization is calculated as follows:

$${\mathcal{L}}_{total}={\mathcal{L}}^{Enhance}+{\displaystyle \frac{1}{M+1}}·\sum _{m=0}^M{\omega }_m{\mathcal{L}}^{Track}$$

(20)

Note that, ${\omega }_0$ is the weight corresponding to the original RGB image and is invariably equal to 1. Through the joint optimization, the enhancement network can learn a sample specific transformation for improving tracking performance.

Specifically, the enhancement network adaptively modulates image brightness and contrast in accordance with the requirements of the tracker, while the tracker extracts more stable and highly discriminative target representations from the enhanced features. Thus, a significant improvement in tracking performance can be achieved within complex environments, such as those characterized by low illumination. As an example, a comparative illustration of the tracking confidence maps, generated under “separate training” and “joint optimization” paradigms, is presented in Figure 5. This indicates that the joint optimization paradigm facilitates a more pronounced salience and distinctiveness of the target. In essence, such findings validate that low-light image enhancement can furnish the UAV tracker with more discriminative features, enabling the maintenance of satisfactory performance under nocturnal conditions.

Figure 5. Qualitative comparison of confidence maps under two training strategies. ‘Separate training’ indicates that the enhancer and the tracker are trained independently, whereas ‘Joint optimization’ refers to the proposed method. The colormap transitions from blue, indicating low confidence, to red, indicating high confidence. It is evident that the joint optimization strategy yields target features with higher saliency and better distinguishability. For easing visualization, the tracked object is delineated by a yellow bounding box in the first column, and the sequence name, sourced from the public UAVDark135 benchmark [16], is displayed in the top-left corner of the original frames.

4. Experimental Evaluation

A systematic evaluation of the NightTrack is herein presented. This work benchmarks the performance of various trackers on three real-world night-time datasets: UAVDark135 [16], DarkTrack2021 [14], and NAT2021 [38]. The rich diversity of these datasets, encompassing various object categories, makes them an ideal testbed for evaluating the generalization capabilities of a generic tracker. The results help validate the superiority of the proposed NightTrack for night-time UAV object tracking.

4.1. Experimental Setup

All experimental investigations carried out are conducted on a server equipped with an NVIDIA GTX 4090 GPU, and the code is implemented based on the PyTorch 2.1.2 framework. The base tracker is trained on four popular datasets in the literature: GOT-10k [4], LaSOT [5], TrackingNet [3], and COCO [39]. The model is trained for 100 epochs using the AdamW optimizer [40] with a batch size of 32. Each epoch involves 60,000 sample pairs. The template and search region sizes are set to $192\times 192$ and $384\times 384$, respectively. Notably, following the training setup as of DCPT [41], images labeled with “night” are selected from the BDD100K [42] and SHIFT [43] datasets, as well as ExDark [44] to serve as night-time data for training the jointly optimized framework.

4.2. Evaluation Metrics

The evaluation follows the One-Pass Evaluation (OPE) [45] protocol, as commonly adopted in the relevant literature, with a focus on three principal metrics: Success Rate, Precision, and Normalized Precision. The Success Rate is based on the IoU between the predicted bounding boxes and its groundtruth. It quantifies the percentage of frames where the IoU exceeds a given threshold, visualized as a Success Plot (SP). The overall tracking performance is measured by the Area Under the Curve (AUC) of the Success Plot, where a higher AUC indicates superior performance. Precision is measured by the Center Location Error (CLE), defined as the Euclidean distance between the centers of the predicted box and that of groundtruth. The Precision Plot (PP) illustrates the trackers’ robustness at different error thresholds, where the representative precision score is typically deemed to be the percentage of frames with a CLE below 20 pixels. Furthermore, the Normalized Precision metric introduces a scale-invariant assessment by normalizing the precision with respect to the groundtruth. The AUC of the Normalized Precision Plot is calculated over a range of $[0,0.5$, yielding a holistic performance measure that is invariant to object scale.

4.3. Overall Performance

A comparative analysis is conducted between the proposed NightTrack and a suite of SOTA trackers on a range of challenging night-time datasets. The competing methods run are: ProContEXT [7], HIPTrack [6], DropTrack [8], OSTrack [46], DCPT [41], DiMP50_SCT [14], SeqTrack [47], DiMP50 [48], UDAT-BAN [38], UDAT-CAR [38], and AVTrack [49]. To ensure a fair comparison, all models, including the proposed method and all baselines, were evaluated under identical conditions.

UAVDark135: UAVDark135 [16] is generally regarded as the first tracking benchmark dedicated to night-time UAV scenarios, comprising 135 videos captured by standard UAVs at night. Figure 6a illustrates the success rate, precision, and normalized precision plots on UAVDark135. The results indicate that NightTrack achieves the highest success rate (0.632) and precision (0.781) on this dataset, significantly outperforming existing mainstream trackers. Furthermore, Figure 7 provides a visual comparison of the representative frames, demonstrating the proposed method’s capacity for stable and robust tracking even challenging darkness.

Figure 6. Performances of NightTrack and SOTA trackers compared on different night-time tracking benchmarks: (a) Results on UAVDark135 [16]. (b) Results on DarkTrack2021 [14]. (c) Results on NAT2021-test [38]. (d) Results on NAT2021-L-test [38].

Figure 7. Qualitative comparison on the public benchmark UAVDark135. Tracking sequences from top to bottom are: bike6, girl6_1, and person13. The results of different trackers have been shown with different colors.

DarkTrack2021: DarkTrack2021 [14] presents an even more formidable challenge for night-time tracking. Its 110 video sequences are predominantly characterized by complex illumination scenarios, including under-exposure and over-exposure, and depict urban environments captured from more elevated and distant perspectives. The quantitative curves in Figure 6b demonstrate that NightTrack performs exceptionally well against SOTA trackers. It achieves a normalized precision of 0.745 and a precision of 0.767, surpassing the previous best-performing tracker (namely, HIPTrack [6]) by over 2.1% on both metrics, with the success rate also raised by 1.4%.

NAT2021-test and NAT2021-L-test: NAT2021 [38] provides 1400 unlabeled night-time tracking videos (and hence, typically serving as benchmarks for unsupervised training). The test set concerned comprises 180 sequences encompassing 12 challenging attributes, such as full occlusion and extremely low illumination, thereby substantially augmenting the difficulty of UAV tracking. Nevertheless, NightTrack exhibits the top performance again. As shown in Figure 6c, it achieves a success rate of 0.584, a normalized precision of 0.713, and a precision of 0.768 on this dataset. Furthermore, to assess the robustness for long-term tracking, NAT2021-L-test (23 sequences with an average length >1400 frames) is also employed. As depicted in Figure 6d, NightTrack ranks first in both success rate and normalized precision, while trailing in precision by a mere 0.6%.

4.4. Illumination-Oriented Evaluation

A primary challenge in night-time object tracking is the model’s robustness to drastic and unpredictable illumination changes, such as sudden flashes from vehicle headlights or transitions between brightly lit areas and deep shadows. To specifically evaluate the model’s capability in handling these scenarios, a dedicated analysis was conducted using the illumination variation attribute annotations provided in the UAVdark135 [16] and NAT2021-test [38] benchmarks. These annotations allow for the isolation and testing of performance exclusively on video sequences with significant lighting fluctuations.

The quantitative results on the UAVdark135 and NAT2021-test datasets are presented in Figure 8a and Figure 8b, respectively. As depicted, the proposed NightTrack consistently demonstrates superior performance across both benchmarks. On the UAVdark135 dataset, NightTrack achieves a precision of 0.781 and a success rate of 0.632, representing a significant improvement of 2.8% in precision and 1.7% in success rate over the baseline HipTrack. Similarly, on the NAT2021-test dataset, NightTrack also leads the performance chart, outperforming HipTrack by 2.7% in precision and 1.8% in success rate and surpassing other strong competitors like ProContEXT [7] by a considerable margin on both metrics.

Figure 8. Success Rate, Normalized Precision, and Precision plots on the Illumination Variation Attribute from the (a) UAVDark135 [16] and (b) NAT2021-test [38] benchmarks.

This enhanced robustness can be directly attributed to the end-to-end joint optimization framework. Unlike traditional two-stage approaches where an image enhancer is applied independently, the enhancer in this work is trained in tandem with the tracker. This co-training mechanism forces the enhancer to learn an illumination-invariant feature space that is specifically optimized for the tracking task. Consequently, when confronted with sudden changes in lighting, the model can adaptively normalize the input, preserving critical discriminative features for the tracker while suppressing distracting artifacts or noise. This ensures a more stable and reliable feature representation, leading to markedly improved tracking performance under challenging illumination conditions.

4.5. Ablation Study

A series of ablation studies are conducted to ascertain the contribution of individual components within the NightTrack framework. This study evaluates the performance of NightTrack, utilizing HipTrack [6] as the baseline, All variants are trained under identical settings, and the tracking performance on the UAVDark135 [16], DarkTrack2021 [14], and NAT2021 [38] datasets is documented in Table 1. For clarity, ‘PAM’ denotes Pyramid Attention Module; ‘Noise’ represents estimated noise curve; ‘Enhancer’ refers to a version with a separately trained enhancer; and ‘Base’ is the original tracker.

Table 1. Performance comparison of different methods on night-time tracking benchmarks.

Method	UAVDark135		DarkTrack2021		NAT2021-test		NAT2021-L-test
	Success (↑)	Precision (↑)	Success (↑)	Precision (↑)	Success (↑)	Precision (↑)	Success (↑)	Precision (↑)
Base	61.5	75.3	60.4	74.6	56.9	74.5	53.4	67.1
Base + PAM	62.3 (0.8↑)	76.7 (1.4↑)	61.3 (0.9↑)	75.9 (1.3↑)	57.8 (0.9↑)	76.0 (1.5↑)	55.3 (1.9↑)	69.2 (2.8↑)
Base + Noise	62.1 (0.6↑)	76.2 (0.9↑)	61.2 (0.8↑)	75.4 (0.8↑)	57.5 (0.6↑)	75.6 (1.1↑)	54.3 (0.9↑)	68.7 (1.6↑)
Base + Enhancer	62.3 (0.8↑)	75.9 (0.6↑)	60.3 (0.1↓)	74.9 (0.3↑)	57.4 (0.5↑)	75.5 (1.0↑)	54.7 (1.3↑)	69.0 (1.9↑)
Base + PAM + Noise	63.2 (1.7↑)	78.1 (2.8↑)	61.8 (1.4↑)	76.7 (2.1↑)	58.4 (1.5↑)	76.8 (2.3↑)	56.9 (3.5↑)	71.7 (4.6↑)

↑ indicates improvement over Base method. Conversely, ↓ indicates a performance degradation with respect to the baseline method.

Effectiveness of PAM: The efficacy of including PAMs within the system is shown in the second row of Table 1. The integration of PAMs yields consistent improvements in both success rate and precision across all evaluated datasets. This performance uplift can be attributed to the pyramid pooling architecture within each PAM, which is adept at aggregating multi-scale global contextual information.

Effectiveness of Noise Estimation: The contribution of the noise estimation component can be seen from the third row of Table 1, with a clear indication that the incorporation of noise estimation promotes night-time tracking performance. In essence, this component is of paramount importance for enhancing UAV tracking performance, as it counteracts the noise amplification that invariably accompanies the brightness enhancement process.

Effectiveness of the Joint Training Strategy: To corroborate the effectiveness of the joint optimization framework, a further experiment is conducted, wherein the enhancer and tracker are trained in isolation. The results show that while the independently trained components (row 4) may generally lead to marginal improvements, their success rate on the DarkTrack2021 dataset [14] is even lower than what is achievable by the baseline. This indicates that independent optimization can fail due to task mismatch. In contrast, the jointly optimized framework (row 5) results in consistent and superior performance, serving as strong confirmation of the advantages conferred by the end-to-end joint training paradigm.

5. Discussion

In the experimental section, comparative experiments were conducted against SOTA trackers on three widely used night-time UAV benchmarks. The results demonstrate that the proposed NightTrack achieves the best performance on the dark tracking. This success is attributed to two primary factors. Firstly, the joint optimization of the enhancer and tracker enables information complementarity between the two tasks, collectively enhancing the model’s generalization capability in night-time scenarios. Secondly, the Pyramid Attention Module and the curve projection model with illumination-noise components further accentuate the discriminative features of the target. Notably, on the NAT2021-L-test dataset, NightTrack achieved improvements of 3.5% and 4.6% over the baseline, respectively. This serves as strong evidence for the effectiveness of the proposed joint optimization framework.

The significance of these results is underscored by the unique and irreplaceable role of night-time vision in numerous practical applications where data collection during daylight is either impossible or insufficient. For example, many animal species are nocturnal; their natural behaviors and social interactions can only be observed under the cover of darkness. In law enforcement and security, illicit activities, including poaching and smuggling, often occur at night to avoid detection. Furthermore, infrastructure inspection of power lines or pipelines is frequently conducted at night to minimize disruption and capture thermal signatures of defects invisible during the day. These scenarios highlight that night-time data is not merely a variant of daytime data but a unique and critical source of information, making the development of advanced trackers like NightTrack essential.

Against this backdrop of critical need, the practical significance of NightTrack’s performance improvements becomes even more apparent. In many high-stakes applications, even a marginal increase in tracking precision can yield substantial consequences. In wildlife ecology, for example, the ability to maintain continuous, accurate observation of elusive nocturnal species is paramount. A minor improvement in tracking precision can be the difference between collecting robust, actionable data for conservation policy and gathering fragmented, inconclusive observations. This directly translates to more effective habitat management and a greater chance of survival for endangered species. Similarly, for autonomous systems, the demand for 24/7 all-weather operation is relentless. Consider UAVs deployed for nocturnal search-and-rescue missions, where the system’s ability to accurately perceive and track objects is fundamental to preventing catastrophic failures and ensuring public safety. Therefore, the performance gains of NightTrack are not merely incremental metrics but represent a tangible step towards more reliable and robust vision systems for critical missions.

Given the scarcity of low-light UAV tracking benchmarks and the high cost of annotation, future work will focus on exploring unsupervised/self-supervised paradigms. The objective is to learn illumination-robust features for objects directly from unannotated raw night-time videos to migrate the dependency on labeled data.

6. Conclusions

This paper has presented an end-to-end network framework, called NightTrack, which is engineered for the joint optimization of low-light image enhancement and visual object tracking. The architecture of NightTrack constitutes an organic integration of an enhancement sub-network and a tracking sub-network. The former is designed to adaptively accentuate salient features within an image, while concurrently mitigating the adverse effects of illumination fluctuations and noise on the downstream task. This is realized through the incorporation of the novel Pyramid Attention Modules and a curve projection model that disentangles illumination and noise components. The latter sub-network, in turn, extracts features from the enhanced images and is co-trained with the former via a joint loss function that combines both tracking and enhancement objectives. Owing to these innovations, the proposed method is experimentally shown to offer superior performance to that attainable by the state-of-the-art mechanisms, in dealing with real-world night-time UAV tracking scenarios.

Despite its success, the current research has several limitations. Firstly, the joint optimization framework introduces additional computational complexity compared to standalone trackers, which may pose challenges for deployment on resource-constrained UAV platforms. Secondly, while effective on common benchmarks, the model’s robustness against extreme and atypical night-time conditions, such as heavy fog, dense smoke, or rapid and drastic lighting changes, requires further investigation. Lastly, the model’s performance is still dependent on supervised learning with annotated data. Its capability to learn under unsupervised or weakly supervised warrants further investigation.

This study underscore the potent role of synergistically integrating upstream and downstream tasks. By co-optimizing these traditionally disparate stages within a unified framework, this work demonstrate a paradigm that effectively overcomes the inherent performance bottlenecks of standalone models, particularly under adverse environmental conditions such as low illumination. For scenarios with compromised visual quality, including low-light, rain, and haze, this integrated workflow yields substantial performance gains. Consequently, the joint optimization strategy presents a promising direction for solving tasks that are severely impacted by environmental degradation, such as low-light, rain, or haze. Furthermore, this framework shows significant potential for generalization to other vision tasks. For example, by replacing the tracking loss with a detection-specific loss, the enhancer could be co-optimized to boost object detection performance, representing a key avenue for future work.

Future research will prioritize several key directions to improve the performance of night-time object tracking. A primary focus is the development of lightweight and efficient architectures, leveraging techniques such as neural architecture search (NAS) and model distillation to facilitate real-time deployment on embedded systems. Another promising avenue is exploring multi-modal fusion, such as combining RGB with thermal or infrared data, to significantly enhance tracking robustness in complete darkness or adverse weather. Finally, advancing the unsupervised and self-supervised learning paradigms, as mentioned in the discussion, is crucial for breaking the data annotation barrier. Future studies could improve the model by designing more sophisticated pretext tasks that are inherently tailored to the characteristics of night-time scenes, thereby learning more generalized and illumination-invariant features. By pursuing these avenues, the next generation of night-time tracking models can achieve even greater levels of performance, efficiency, and adaptability.

Appendix A.1. Primary Balancing Coefficient (γ)

The coefficient $\gamma$ is used to balance the overall influence of the enhancement module against the primary tracking task, with the objective of scaling ${\mathcal{L}}^{Enhance}$ to a magnitude comparable to ${\mathcal{L}}^{Track}$. Empirical analysis revealed that the model’s performance is robust when $\gamma$ is in the range of 0.05 to 0.2. Values outside this range caused instability: a large $\gamma$ (e.g., 1.0) led to gradient explosion, while a small value prevented the enhancer from converging. Consequently, $\gamma$ was set to 0.1 for all experiments.

Appendix A.2. Internal Balancing Coefficients (λ_cen, λ_ill, λ_col, λ_noi)

These four coefficients balance the relative importance of the enhancement sub-losses: center brightness (${\mathcal{L}}_m^{cen}$), illumination (${\mathcal{L}}_m^{ill}$), color (${\mathcal{L}}_m^{col}$) and noise (${\mathcal{L}}_m^{noi}$). To ensure the enhancer addresses all aspects of degradation equally without neglecting any single component, experiments were conducted to tune these weights. The results indicated that assigning them equal importance yielded the best and most stable results. Therefore, all internal weights were set to ${\lambda }_{cen}={\lambda }_{ill}={\lambda }_{col}={\lambda }_{noi}=1.0$.