HPFNet: Hierarchical Perception Fusion Network for Infrared Small Target Detection

Highlights

What are the main findings?

• A hierarchical perception fusion framework is proposed to improve infrared small target detection under complex background interference.
• Joint global–local perception enhancement, sparse cross-level fusion, and efficient decoding effectively preserve weak target cues while suppressing background clutter.

What are the implications of the main findings?

• The proposed advanced and well-coordinated modules jointly contribute to robust infrared small target detection.
• The proposed method provides an efficient and practical solution for detecting extremely small and weak targets in cluttered infrared scenes.

Abstract

Infrared small target detection (IRSTD) is a fundamental task in remote sensing-based surveillance and early warning systems. However, extremely small target size, low signal-to-noise ratio, and complex background clutter make reliable detection highly challenging. To address these issues, we propose a Hierarchical Perception Fusion Network (HPFNet) for IRSTD. Specifically, the Patch-Wise Context Feature Extraction module (PCFE) jointly integrates the Patch Nonlocal Block, convolutional blocks and attention mechanism to enable global–local feature extraction and enhancement, thereby strengthening weak target representations. In addition, the Multi-Level Sparse Cross-Fusion module (MSCF) explicitly performs cross-level feature interaction between encoder and decoder representations, enabling effective fusion of low-level spatial details and high-level semantic cues. A dual Top-K sparsification mechanism is adopted to filters’ irrelevant background features, enabling the attention mechanism to focus more on the target region and thereby bolstering the discriminative power of feature representation. Finally, the Efficient Upsampling Module (EUM) combines upsampling with multi-branch dilated convolutions to enhance feature reconstruction and improve localization accuracy. Extensive experiments on publicly available benchmark datasets demonstrate that HPFNet consistently outperforms existing state-of-the-art IRSTD methods.

1. Introduction

Infrared small target detection (IRSTD) is a pivotal technology in modern remote sensing systems, playing a crucial role in early warning, maritime surveillance, precision guidance, and border security [1,2]. Unlike visible-light imagery, infrared (IR) images are generated based on thermal radiation differences, enabling robust perception under all-weather and night-time conditions.

Despite these advantages, IRSTD remains a highly challenging task due to the intrinsic properties of both targets and environments.

• Tiny target scale: Targets typically occupy only a few pixels and exhibit extremely limited structural or textural cues, making it difficult to differentiate them from visually similar false alarms using only local target information [3].
• Low signal-to-clutter ratio (SCR): Target responses are often submerged by severe thermal noise and complex background clutter, which makes reliable target detection particularly difficult [4].

Early IRSTD methods were predominantly model-driven and relied on handcrafted priors, including local contrast measures [5,6] and low-rank sparse decomposition (LRSD) [7,8]. Although computationally efficient, these approaches are highly sensitive to hyperparameter tuning and typically assume background continuity or sparsity, which limits their generalization ability in dynamic and complex scenes.

In recent years, deep learning has significantly advanced IRSTD research. Convolutional neural networks (CNNs), especially encoder–decoder architectures such as U-Net [9], have become the dominant paradigm due to their strong representation learning capability. Representative works, including ACM [10], DNANet [11], and ISNet [12], improve detection performance by introducing attention mechanisms and multi-scale feature fusion. More recently, Transformer-based [13,14] and State Space Model (Mamba)-based architectures [15,16] have been explored to capture long-range dependencies and global contextual information, alleviating the limited receptive field of traditional CNNs.

Nevertheless, despite continuous progress, two critical issues remain unresolved in existing deep learning-based IRSTD methods, restricting further performance improvement.

Ineffective global–local perception balance: Transformer and Mamba-based models excel at modeling global context but often struggle to preserve fine-grained local details due to patch-wise tokenization or sequential modeling mechanisms [15]. In contrast, CNN-based architectures are prone to losing weak target cues during repeated downsampling operations. Consequently, there is an urgent need for a lightweight yet effective framework that can simultaneously enhance local structural details and capture global contextual dependencies without incurring excessive computational overhead. To tackle this challenge, we introduce the Patch-Wise Context Feature Extraction (PCFE) module, which jointly enhances fine-grained local structures and patch-level global contextual perception.

Redundant and noisy cross-scale feature fusion: Most existing IRSTD networks, including strong baselines such as UIU-Net [17] and DNANet [11], adopt dense skip connections or dense attention mechanisms for cross-scale feature fusion. However, such indiscriminate fusion inevitably propagates background clutter and thermal noise from shallow to deep layers. Recent studies on sparse attention [18] indicate that dense interactions are suboptimal for IRSTD, as informative target regions are extremely sparse. Redundant feature fusion dilutes target responses and increases false alarms. To alleviate this issue, we designed the Multi-Level Sparse Cross-Fusion (MSCF) module with a dual Top-K sparsification strategy to suppress redundant background propagation during cross-scale fusion.

To address the above challenges, we propose a Hierarchical Perception Fusion Network (HPFNet) for infrared small target detection. We introduce a Patch-Wise Context Feature Extraction Module (PCFE) that enhances weak target representations through dual-scale patch-wise non-local modeling and attention refinement. Unlike conventional token-level global attention or full-resolution non-local blocks, PCFE better preserves fine-grained local cues while capturing contextual dependencies. To overcome dense and noisy skip fusion, we design a Multi-Level Sparse Cross-Fusion Module (MSCF), which explicitly models encoder–decoder cross-level correlations and incorporates a dual Top-K sparsification strategy to suppress redundant background propagation. Different from existing sparse attention methods that typically sparsify within a single feature level, MSCF performs selective interaction among multi-depth representations for clutter-suppressed fusion. Finally, we introduce an Efficient Upsampling Module (EUM) with lightweight multi-branch depthwise dilated convolutions to improve feature reconstruction and boundary recovery during decoding. In contrast to standard upsampling blocks relying on naive interpolation or single-rate dilation, EUM enables efficient receptive-field expansion with minimal computational overhead.

The main contributions of this work are summarized as follows:

• We propose HPFNet, a novel hierarchical perception fusion framework for infrared small target detection, which jointly leverages global–local perception enhancement, sparse cross-level fusion, and efficient decoding to improve robustness under complex backgrounds.
• The PCFE enhances weak target representations by integrating convolutional local structure preservation, multi-scale patch-wise global context modeling via PNB, and attention-based refinement, leading to more discriminative feature extraction for small targets.
• The MSCF enables selective and sparsified cross-level feature fusion between encoder and decoder features through sparse cross-attention, reducing the influence of background clutter and enhancing target–background discrimination.
• The EUM performs efficient upsampling and feature reconstruction using multi-branch depthwise dilated convolutions, improving boundary recovery and localization accuracy during decoding.

2. Related Work

2.1. Infrared Small Target Detection

In the field of infrared small target detection, methods diverge into traditional IRSTD methods and emerging deep learning methods [19]. The traditional IRSTD methods including Top-Hat [20], Max-Median [21], WSLCM [22], TLLCM [23], IPI [24], NRAM [25], RIPT [26] and PSTNN [27] based on hand-crafted features are tailored to specific scenarios, limiting their generalization to challenging situations. While effective in specific scenarios, traditional methods are susceptible to interference from clutter and noise. In complex real-world scenarios, modeling objects is significantly affected by model hyperparameters, resulting in poor generalization performance.

With the availability of large-scale public benchmarks [28], deep learning has become the mainstream solution for infrared small target detection. Recent methods mainly improve detection performance from three perspectives: (i) multi-scale representation learning, (ii) attention-guided feature enhancement, and (iii) robust fusion under complex backgrounds. For instance, multi-level fusion networks explicitly aggregate hierarchical features to prevent tiny target cues from vanishing in deep layers [29,30]. Attention-based models further highlight target-relevant responses while suppressing background clutter, such as boundary-aware attention and channel and spatial attention refinement [31,32]. In addition, Transformer-based designs and correlation modeling have been explored to capture long-range dependencies and pixel-wise interactions beyond local receptive fields, leading to improved localization accuracy under cluttered scenes [33,34].

Notably, our work differs from prior attention and fusion approaches by emphasizing hierarchical perception enhancement and sparse cross-level interaction. PCFE strengthens weak targets via patch-wise contextual modeling, MSCF selectively fuses multi-level cues through dual Top-K sparse cross-attention, and EUM improves decoder recovery with multi-scale upsampling. Together, they enable robust detection under complex cluttered backgrounds. In addition, recent studies have explored target interpretation and multidomain characterization in polarimetric SAR, such as the general polarimetric correlation pattern for joint-domain scattering mechanism visualization [35] and multidomain feature characterization methods for robust target analysis [36,37]. They provide complementary insights into target representation and characterization under complex sensing environments.

2.2. U-Net Variants

The U-Net architecture, with its symmetric encoder–decoder structure and efficient skip connections, has become a cornerstone of modern image segmentation [9]. Its powerful performance has led to its widespread application in various segmentation tasks and inspired numerous variants aimed at improving feature propagation and representation capabilities. Enhancements to U-Net have primarily unfolded along two key dimensions: Backbone Design Enhancements and Skip Connection Enhancements. R2U-Net replaced standard convolutional blocks with recurrent residual units, significantly improving performance without increasing model parameters [38]. UNet++ introduced dense connections and nested skip paths to narrow the semantic gap between encoder and decoder features, thereby achieving more efficient multi-scale aggregation [39].

The U-Net architecture has also been widely used in infrared small target detection, achieving excellent results. MST-UNet replaces max pooling with Haar wavelet-based downsampling and introducing multi-scale residual units and multi-branch attention mechanisms in the encoder [40]. ID-UNet proposes a densely connected U-Net architecture, enhancing information flow between shallow and deep layers [41]. MSHNet adds a multi-scale detector head to the standard U-Net and introduces a scale and location-sensitive loss function to highlight targets of different sizes and spatial locations [42]. HintU further cascades a cue generation network with a U-Net detector to solve the problems of local features that are often lost [43].

However, these methods easily transmit background clutter along with target cues, leading to the dilution of weak targets and causing false detections. Our proposed method allows the current feature to cross-fuse the high-level feature and low-level features. In the fusion process, MSCF not only explores the multiscale messages in scenes but also mitigates the negative impact of irrelevant information by employing sparse operations.

2.3. Attention Mechanisms and Sparse Attention

Attention mechanisms significantly improve the performance of computer vision models by enabling adaptive feature reweighting and selective emphasis on salient regions [44,45]. Representative works have demonstrated the effectiveness of attention mechanisms in enhancing weak target cues. MPANet [46] introduces a multi-patch attention mechanism to refine multi-scale features and improve the visibility of dim targets. Similarly, ABCNet [47] adopts a convolution–linear fusion Transformer supported by an attention-driven bilinear correlation module, thereby enabling robust target enhancement. Attention mechanisms have also been applied in hybrid fusion strategies. For example, HAAF-Net [48] adopts a dense–sparse dual-branch architecture and combines dynamic attention modulation to adaptively enhance target-specific responses. Recently, DENet [49] proposes a dual-path global–local attention mechanism, which can simultaneously capture fine-grained edge information and global contextual cues.

Recent studies have shown that sparse attention mechanisms are more suitable for infrared small target detection. Since the information-rich target regions in infrared images are extremely sparse, dense attention or self-attention mechanisms often propagate redundant background responses together with useful target details. In contrast, sparse attention strategies, such as Top-k selection, can effectively filter out irrelevant interactions and improve robustness against background interference [50,51]. These observations highlight three important insights: (1) selective enhancement is crucial for isolating truly target-relevant activations; (2) sparsity-aware attention or fusion mechanisms help alleviate noise propagation caused by dense feature interactions.

Inspired by the above studies, our proposed HPFNet incorporates attention-driven yet computationally efficient design principles. It constructs query–key–value interactions between decoder features and multi-level encoder features through cross-level correlation modeling. After the correlation matrix is computed, MSCF designs a dual-threshold Top-K sparsification strategy and adaptively fuses features through learnable weights, thereby explicitly suppressing redundant background-related correlations.

3. Methods

3.1. Overall Network Architecture

The overall architecture of the proposed HPFNet is illustrated in Figure 1. HPFNet follows a U-Net-like encoder–decoder backbone and is strengthened by three key components: PCFE, MSCF, and EUM.

Figure 1. Overall architecture of the proposed HPFNet. HPFNet incorporates three specially designed modules: PCFE, MSCF and EUM.

In both the encoder and decoder stages, PCFE is employed to build more discriminative representations through global–local feature extraction and enhancement. Specifically, PCFE integrates convolutional blocks for local structure preservation with Patch Nonlocal Blocks (PNB) for patch-wise global context modeling, and further introduces channel–spatial attention refinement to emphasize informative responses under cluttered backgrounds.

To perform effective feature interaction between encoder and decoder representations, MSCF replaces conventional dense skip connections. Instead of directly concatenating multi-level features, MSCF explicitly constructs cross-level correlations and conducts multi-level cross-attention interaction among shallow, intermediate, and deep representations. A dual Top-K sparsification mechanism is further adopted to filter irrelevant background features, enabling the attention mechanism to focus more on the target region and thereby bolstering the discriminative power of feature representation.

In the decoder, EUM performs resolution recovery and feature reconstruction. It combines upsampling with multiple parallel depthwise dilated convolution branches of different dilation rates, which enlarges the effective receptive field with low overhead and improves boundary recovery and localization accuracy during decoding.

3.2. Patch-Wise Context Feature Extraction Module

In deep encoder–decoder architectures, repeated downsampling progressively removes fine-grained spatial cues, which is detrimental to preserving weak and small target structures in the decoder stage. Moreover, under cluttered backgrounds, local textures of targets and background regions can be highly similar, easily leading to false alarms when relying only on local receptive fields. Motivated by these observations, we propose a Patch-Wise Context Feature Extraction (PCFE) module that jointly enhances local structural details and captures long-range dependencies in a computationally efficient patch-wise manner.

As illustrated in Figure 2, PCFE adopts a multi-branch design composed of a local convolutional backbone and two patch-wise context modeling branches. Given an input feature map $F\in {\mathbb{R}}^{C\times H\times W}$, we first apply a convolutional block composed of a $1\times 1$ convolution to adjust the channel dimension and obtain $F^{\prime }\in {\mathbb{R}}^{C^{\prime }\times H\times W}$. Then, $F^{\prime }$ is separately fed into a local feature extraction branch and a global contextual feature extraction branch to produce $F_{\mathrm{local}}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$ and $F_{\mathrm{global}}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$, respectively. These two outputs are fused by element-wise addition and then refined by an attention module to obtain $\tilde{F}\in {\mathbb{R}}^{C\times H\times W}$.

Figure 2. Architecture of PCFE. The module consists of a convolutional backbone for local structure preservation and dual Patch Nonlocal Blocks (PNB) for multi-scale patch-wise context modeling, followed by attention-based refinement.

The local feature extraction branch consists of three cascaded convolutions with kernel size $3\times 3$. This design yields three different outputs: $F_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}1}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$, $F_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}2}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$, and $F_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}3}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$. By summing these three outputs, we obtain the final local feature representation $F_{\mathrm{local}}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$. This aggregation integrates hierarchical local cues from shallow-to-deep convolution stages.

From a global perspective, due to the distribution of Gaussian noise in the background, local features of the background and the target may exhibit high similarity, which can easily lead to a large number of false alarms. Therefore, global contextual association information can help the network better determine target locations and suppress background clutter with similar local patterns. To this end, the global contextual feature extraction branch adopts a Patch Nonlocal Block (PNB) to model global contextual features by constructing dependencies among pixels. Since the computational complexity of self-attention is quadratic, we apply a patch-wise operation to reduce the computational burden. First, we divide $F^{\prime }$ into $p\times p$ patches, where each patch has a spatial size of $H/p\times W/p$. Then, a mean-pooling operation is applied to extract the representative feature of each patch, producing a patch-level feature map of size $p\times p$, where each element corresponds to representative feature. A Nonlocal block is then employed to model the dependency among these patches, thereby capturing global contextual information. After that, bilinear interpolation is used to upsample the patch-level feature map back to the original spatial resolution $H\times W$. Finally, a Sigmoid function is applied to generate a spatial attention weight, which is multiplied with $F^{\prime }$ to obtain the global contextual feature $F_{\mathrm{global}}$, as shown in Equation (1):

$$F_{\mathrm{global}}=\sigma \left(\mathrm{BI}\left(NL\left(MP\left(F^{\prime }\right)\right)\right)\right)\odot F^{\prime }$$

(1)

where $MP(·)$ denotes the mean-pooling operation that aggregates features into patch-level representations, $NL(·)$ denotes the nonlocal operation for modeling global contextual dependencies among patches, $\mathrm{BI}(·)$ represents bilinear interpolation used to upsample the patch-level feature map back to the original spatial resolution, and $\sigma (·)$ denotes the Sigmoid function.

Since targets may vary significantly in scale, using only a single patch size may fail to adequately attend to targets at certain scales, which prevents comprehensive extraction of global features. Therefore, to obtain multi-scale global contextual information, we divide $F^{\prime }$ into $2\times 2$ and $4\times 4$ patches, respectively, and apply the above operations to obtain two global feature maps: $F_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}1}^{p=2}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$ and $F_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}2}^{p=4}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$. These feature maps are concatenated along the channel dimension and fused by a $1\times 1$ convolutional block to obtain the final global feature $F_{\mathrm{global}}$, as shown in Equation (2):

$$F_{\mathrm{global}}=f_{1\times 1}^{\mathrm{CBR}}\left(\left[F_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}1}^{p=2},F_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}2}^{p=4}\right]\right)$$

(2)

where $[·]$ denotes channel-wise concatenation, and $f_{1\times 1}^{\mathrm{CBR}}(·)$ denotes a convolutional block consisting of a $1\times 1$ convolution, BN, and ReLU activation.

Finally, the local and global features are fused and refined by channel–spatial attention, followed by a $1\times 1$ projection to produce the final PCFE output:

$$F_{\mathrm{out}}=f_{1\times 1}^{\mathrm{CBR}}\left(\mathcal{A}\left(F_{\mathrm{local}}+F_{\mathrm{global}}\right)\right)$$

(3)

where $\mathcal{A}(·)$ denotes the attention refinement consisting of a sequence of efficient channel attention [52] and spatial attention [53] components.

Note that the dual-scale PNB design (P = 2 and P = 4) is introduced to handle extremely small targets. The smaller patch emphasizes local target cues, whereas the larger patch provides contextual suppression of clutter, and the subsequent attention module adaptively balances their contributions.

3.3. Multi-Level Sparse Cross-Fusion Module

To efficiently fuse shallow and deep features while reducing background clutter interference, we propose a Multi-Level Sparse Cross-Fusion (MSCF) module. MSCF performs cross-attention-based feature interaction and selective sparse fusion between shallow and deep representations. Specifically, MSCF constructs a cross-level correlation map between the decoder feature and encoder features, and further applies a dual Top-K sparsification mechanism to retain only the most target-relevant correlations. The overall structure of MSCF is illustrated in Figure 3.

Figure 3. Architecture of MSCF. MSCF constructs sparse cross-level correlations between decoder and encoder features and performs selective fusion via dual Top-K sparsification.

Given three feature maps from different levels, including a shallow feature $F_l\in {\mathbb{R}}^{2C\times 2H\times 2W}$, a current feature $F_m\in {\mathbb{R}}^{C\times H\times W}$, and a deep feature $F_h\in {\mathbb{R}}^{{\displaystyle \frac{C}{2}}\times {\displaystyle \frac{H}{2}}\times {\displaystyle \frac{W}{2}}}$, we first align their spatial resolutions to match that of $F_m$. Specifically, $F_l$ is downsampled by a $3\times 3$ convolutional block with stride 2 to $F_l^{\prime }$ to reduce its spatial size while adjusting its channel dimension, and $F_h$ is upsampled via bilinear interpolation followed by a $1\times 1$ convolutional block to $F_h^{\prime }$ to restore its resolution and unify the channel dimension. After alignment, we concatenate $F_l^{\prime }$ and $F_h^{\prime }$ along the channel dimension to obtain a cross-level representation $F_{\mathrm{\Sigma }}\in {\mathbb{R}}^{2C\times H\times W}$. This cascaded feature $F_{\mathrm{\Sigma }}$ is then projected to generate the query embedding for cross-level correlation modeling, while the current feature $F_m$ is used to generate the key and value embeddings.

To construct cross-level correlations, we project $F_{\mathrm{\Sigma }}$ into the query embedding space, while the current feature $F_m$ is projected into key and value embeddings:

$$\begin{array}{cc}\hfill Q& =f_{1\times 1}\left(F_{\mathrm{\Sigma }}\right)\hfill \\ \hfill K,V& =f_{1\times 1}\left(F_m\right)\hfill \end{array}$$

(4)

where $f_{1\times 1}(·)$ denotes a $1\times 1$ convolution operation. The dense correlation map is computed as:

$$M={\displaystyle \frac{Q{K}^T}{\sqrt{c}}}$$

(5)

where c is the channel dimension of Q.

The dense correlation map inevitably contains a large number of background clutter interferences. Compared with the dense map M, the sparsified map $M^{\prime }$ sheds light on more target-relevant elements while neglecting irrelevant ones. In this way, the neglected entries are not considered in the subsequent aggregation, whereas the weights of the retained relevant correlations can be enlarged after softmax normalization.

To suppress irrelevant responses, MSCF applies row-wise Top-K sparsification: for each query position, only the Top-K largest correlation values are retained, and we replace the irrelevant elements’ values by $-\infty$ rather than removing them. This way, these values are transformed into 0 through the softmax operation. Moreover, to capture diverse feature information and balance fine-grained localization with broader contextual aggregation, we perform two separate Top-K operations with different K values (denoted as $k_1$ and $k_2$) on M. Subsequently, two learnable parameters are introduced to automatically adjust the contribution ratio of these two sparsification levels, enabling the model to enhance its perceptual capability during the fusion stage:

$$M^{\prime }=\alpha ·\mathrm{Softmax}\left({\mathrm{TopK}}_1\left(M\right)\right)+\beta ·\mathrm{Softmax}\left({\mathrm{TopK}}_2\left(M\right)\right),$$

(6)

where ${\mathrm{TopK}}_1(·)$ and ${\mathrm{TopK}}_2(·)$ denote row-wise sparsification with $k_1$ and $k_2$, respectively, and $\alpha$ and $\beta$ are learnable fusion weights. This dual-sparsity strategy preserves both highly discriminative target correlations and moderately informative contextual interactions.

The dual Top-K sparsification strategy is adopted to suppress redundant background responses. Following prior sparse attention designs, the Top-K parameters are defined as sparsification ratios rather than fixed integers. Specifically, we search $k_1\in \{1/2,1/4,1/8\}$ and $k_2\in \{1/3,1/6,1/9\}$ on the training set. The best performance is achieved with $k_1=1/2$ and $k_2=1/6$, which are fixed for all experiments in this paper.

The refined sparse correlation map $M^{\prime }$ is applied to the value feature V to obtain the cross-level aggregated representation. It is then combined with the identity shortcut via residual addition, followed by a CBR block for further feature refinement:

$$F_{\mathrm{fusion}}=f^{\mathrm{CBR}}\left(M^{\prime }V+F_m\right),$$

(7)

where $f^{\mathrm{CBR}}(·)$ denotes a convolutional block consisting of Conv, BN, and ReLU. This residual refinement preserves the original feature information while enhancing informative responses through selective sparse interaction, thereby improving target–background discrimination.

3.4. Efficient Upsampling Module

To address the limitations of conventional upsampling operators and the difficulty of recovering high-resolution feature maps in the decoder stage, we design an Efficient Upsampling Module that integrates a hybrid upsampling strategy with a multi-scale context-aware enhancement mechanism. Specifically, the input feature is first enlarged via bilinear interpolation, and then refined by multiple parallel dilated convolution blocks with different dilation rates. The proposed EUM improves upsampling flexibility and feature preservation, thereby reducing information loss and enhancing structural recovery. The overall structure of EUM is illustrated in Figure 4.

Figure 4. Architecture of the proposed Efficient Upsampling Module (EUM).

Given an input feature map $F\in {\mathbb{R}}^{C\times H\times W}$, EUM first applies bilinear interpolation (BI) to enlarge the spatial resolution, yielding $F^{\prime }\in {\mathbb{R}}^{C\times 2H\times 2W}$. This parameter-free operation provides a stable and artifact-free upsampling baseline while maintaining the original feature distribution.

To compensate for the limited contextual perception of pure interpolation, $F^{\prime }$ is further processed by three parallel depthwise $3\times 3$ convolution branches with dilation rates $d\in \{1,3,5\}$. We adopt these moderate dilation rates following common practice in multi-scale decoding modules, which provides an effective balance between receptive-field expansion and precise localization for tiny targets. These branches capture complementary information at different effective receptive fields: the branch with $d=1$ focuses on local detail refinement and boundary continuity, whereas larger dilation rates ($d=3$ and $d=5$) incorporate broader contextual cues to enhance semantic consistency and suppress background interference. We do not employ excessively large dilation rates (e.g., $d=7$), since overly sparse sampling may introduce gridding artifacts and irrelevant long-range background clutter, which is unfavorable for infrared small target localization. Each depthwise convolution is followed by Batch Normalization and ReLU activation to improve feature normalization and non-linearity.

The outputs of the three branches are concatenated along the channel dimension and passed through a $1\times 1$ convolution block with BN and ReLU for channel fusion and adaptive re-weighting of multi-scale features. Finally, a residual connection adds the fused feature back to the upsampled shortcut to stabilize optimization and preserve spatial layout:

$$F_{\mathrm{out}}=F^{\prime }+f_{1\times 1}^{\mathrm{CBR}}\left(\left[f_{d=1}\left(F^{\prime }\right),f_{d=3}\left(F^{\prime }\right),f_{d=5}\left(F^{\prime }\right)\right]\right),$$

(8)

where $[·]$ denotes channel-wise concatenation, and $f_d(·)$ represents a depthwise $3\times 3$ dilated convolution followed by Batch Normalization and ReLU.

4. Experiments

This section presents the experimental setup and evaluation of the proposed HPFNet. We describe the datasets, experimental settings, evaluation metrics, quantitative analysis, qualitative analysis and ablation studies.

4.1. Datasets

We evaluate the proposed HPFNet on three widely used infrared small target datasets.

NUAA-SIRST [10]. This dataset contains 427 real infrared images with diverse backgrounds, including sky, sea, and cloud scenes. The target size is extremely small (often only a few pixels), making NUAA-SIRST a challenging and representative benchmark for single-frame IRSTD.

IRSTD-1K [12]. This dataset consists of 1000 infrared images covering complex scenarios such as urban areas, sea surfaces, forests, mountains, and sky scenes. It contains various target categories, including UAVs, aircraft, ships, vehicles, and animals, and is widely used to evaluate robustness and generalization of IRSTD methods.

NUDT-SIRST [11]. This dataset includes 1327 synthetic infrared images generated from real scenes. Although the data are synthesized, the dataset exhibits complex background distributions and diverse target shapes, and has become a standard benchmark in many recent IRSTD works.

We adopt the method used by [11] to partition the training and test sets of NUAA-SIRST and NUDT-SIRST, and [12] for splitting the IRSTD-1K.

4.2. Experimental Settings

All experiments are implemented using PyTorch (v2.1.0) and conducted on an NVIDIA RTX 4090 GPU with 24 GB memory and an Intel Core i5-13500KF CPU. We employed Soft-IoU Loss [54] as the loss function and AdamW [55] as the optimizer. We trained three separate models on four different datasets. Due to significant variations in dataset distribution and resolution, we customized the hyperparameters for each dataset as shown in Table 1. To enhance generalization ability, we apply random horizontal and vertical flipping and random rotations during training. Multi-scale deep supervision strategy is applied to stabilize optimization and accelerate convergence.

Table 1. Training settings on different datasets.

Dataset	Epochs	Lr	Batch	Resolution
NUAA-SIRST [10]	1500	0.0001	4	$512\times 512$
IRSTD-1K [12]	500	0.0001	4	$512\times 512$
NUDT-SIRST [11]	1500	0.0001	16	$256\times 256$

4.3. Evaluation Metrics

We compare the proposed HPFNet with the state-of-the-art (SOTA) methods using several standard metrics.

IoU measures the overlap between predicted target regions and ground-truth annotations:

$${\mathrm{IoU}}_i={\displaystyle \frac{T{P}_i}{T{P}_i+F{P}_i+F{N}_i}}$$

(9)

where $TP$, $FP$, and $FN$ denote true positives, false positives, and false negatives of the i-th image.

Given the extreme class imbalance in infrared small target detection, nIoU provides a dataset-level normalized measure:

$$\mathrm{nIoU}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathrm{IoU}}_i$$

(10)

where N is the total number of images.

Fa evaluates the model’s ability to suppress background clutter by measuring the proportion of background pixels mistakenly predicted as targets:

$$\mathrm{Fa}={\displaystyle \frac{FP}{FP+TP}}\times 100\%$$

(11)

Pd represents the ratio of correctly detected target pixels among all actual target pixels. The standard definition used in SIRST literature is

$$\mathrm{Pd}={\displaystyle \frac{TP}{TP+FN}}\times 100\%$$

(12)

Next to fixed-threshold metrics, ROC curves are also employed for comprehensive evaluation. They illustrate the variation in Pd with respect to Fa when the decision threshold changes.

4.4. Quantitative Analysis

We compare the proposed HPFNet with five representative traditional IRSTD algorithms, including Top-Hat [20], Max-Median [21], TLLCM [23], IPI [24], and PSTNN [27], as well as seven state-of-the-art deep learning-based methods, namely ACM [10], AGPCNet [56], DNANet [11], ISNet [12], UIU-Net [17], RDIAN [57], and MSHNet [42]. Quantitative results on three public benchmarks (NUAA-SIRST, IRSTD-1K, and NUDT-SIRST) are summarized in Table 2.

Table 2. Comparison with state-of-the-art infrared small target detection methods on NUAA-SIRST, IRSTD-1K, and NUDT-SIRST. The best results in each column are shown in bold.

Method	NUAA-SIRST				IRSTD-1K				NUDT-SIRST
	IoU	nIoU	Pd	Fa	IoU	nIoU	Pd	Fa	IoU	nIoU	Pd	Fa
Top-Hat [20]	7.14	18.27	79.84	40.12	10.06	7.44	75.11	24.32	11.52	9.41	80.01	31.63
Max-Median [21]	4.17	12.31	69.20	55.33	6.99	3.05	65.21	59.73	8.48	4.26	67.49	64.10
TLLCM [23]	18.66	27.71	82.24	16.95	10.26	18.24	68.55	24.48	11.08	23.82	77.97	67.26
IPI [24]	25.67	33.57	85.55	11.47	27.92	30.12	81.37	16.18	28.63	38.18	74.49	41.23
PSTNN [27]	30.30	33.67	72.80	48.99	24.57	17.93	71.99	35.26	14.85	23.57	66.13	44.17
ACM [10]	72.88	72.17	96.22	15.14	63.39	60.81	91.25	15.96	68.48	69.26	96.26	10.27
AGPCNet [56]	77.13	75.19	98.15	24.47	68.81	66.18	94.26	15.85	88.71	87.48	97.57	7.54
DNANet [11]	77.55	76.90	99.08	4.48	68.87	67.53	94.95	13.38	91.67	90.09	97.53	3.35
ISNet [12]	80.02	78.12	99.18	4.92	68.77	64.84	95.56	15.39	89.81	88.92	98.12	4.21
UIU-Net [17]	79.08	77.39	100.00	4.04	69.13	67.49	94.27	13.47	90.77	90.17	98.09	3.39
RDIAN [57]	72.85	73.96	99.08	28.48	64.37	64.90	92.26	18.20	81.06	81.72	97.36	14.09
MSHNet [42]	78.75	76.82	98.17	8.65	68.50	67.15	94.27	14.75	88.49	88.70	98.30	3.46
Ours	80.78	78.47	100.00	1.29	71.54	68.19	95.13	9.88	93.48	92.36	99.13	2.20

In general, a clear and consistent performance gap is observed between traditional model-driven algorithms and modern deep learning-based methods. Traditional approaches largely depend on handcrafted priors (e.g., morphological contrast or low-rank decomposition), which restrict their capability to handle complex structured clutter, dynamic backgrounds, and extremely low signal-to-clutter ratios. Consequently, they often yield poor overlap accuracy (low IoU) and suffer from severe false alarms. In contrast, deep models benefit from data-driven representation learning and hierarchical semantic modeling, enabling more robust discrimination between weak targets and background interference across challenging infrared scenes.

Compared with advanced neural network baselines such as DNANet, MSHNet, ISNet, and UIU-Net, HPFNet consistently achieves superior performance in terms of IoU, nIoU, and detection probability ($P_d$), while maintaining the lowest false-alarm rate ($F_a$) on NUAA-SIRST and NUDT-SIRST. Specifically, HPFNet reaches an IoU of 80.78% with only 1.29 false alarms on NUAA-SIRST, and obtains the best IoU of 93.48% on NUDT-SIRST with a very low $F_a$ of 2.20. These results demonstrate that HPFNet is more effective at suppressing background clutter while preserving faint target responses, leading to more accurate segmentation and improved target–background separation. The consistent gains across multiple datasets further validate the stability and generalization ability of the proposed framework. Table 3 gives the average metrics of the different algorithms on the three datasets, and we can observe that HPFNet has acceptable parameters at the highest performance and outperforms the powerful UIU-Net.

Table 3. Comprehensive evaluation metrics with competitive algorithms.

Model	Params (M)	Flops (G)	IoU	nIoU	Pd	Fa
DNA-Net [11]	4.697	14.26	79.36	78.17	97.19	7.07
UIU-Net [17]	50.54	54.42	79.66	78.35	97.46	6.97
HPFNet	18.19	35.24	81.93	79.67	98.09	4.46

To further evaluate the detection performance under varying decision thresholds, we plot ROC curves on the NUAA-SIRST dataset in Figure 5. The ROC analysis corroborates the quantitative comparisons in Table 2, showing that deep learning methods generally outperform traditional techniques, while HPFNet achieves the most favorable trade-off between detection probability and false alarms across all thresholds.

Figure 5. ROC curves of different methods on the NUAA-SIRST dataset.

4.5. Qualitative Analysis

The qualitative comparisons in Figure 6 and Figure 7 provide intuitive evidence of the detection performance of different methods on representative infrared images. Traditional model-driven approaches tend to generate a large number of false alarms (blue boxes) and missed detections (yellow boxes), particularly in cluttered backgrounds. This behavior reflects their limited robustness in complex infrared scenes, where handcrafted priors struggle to distinguish weak targets from structurally similar background responses. In contrast, deep learning-based methods substantially reduce false detections by learning hierarchical feature representations that enhance target–background separability.

Figure 6. Visual results of different IRSTD methods. The red, blue, and yellow boxes denote correct detections, false alarms, and missed detections, respectively.

Figure 7. 3D visualization comparison of detection results via different methods. The red, yellow, and blue boxes denote correct, missed, and false detections.

Among the deep learning baselines, the proposed HPFNet consistently yields the most reliable detection results across diverse scenarios. Although ISNet and UIU-Net demonstrate competitive performance in relatively simple scenes, they still suffer from occasional false alarms and missed detections under challenging conditions, as shown in Figure 6 and Figure 7. These failures mainly stem from insufficient suppression of background clutter and limited ability to exploit long-range contextual dependencies, which are essential for distinguishing tiny targets from high-intensity distractors.

In comparison, HPFNet maintains high detection accuracy and strong robustness across various complex scenes. This advantage is attributed to its hierarchical perception fusion design, which integrates patch-wise contextual modeling, selective cross-level feature fusion and efficient upsampling design. Such a design enables HPFNet to better adapt to variations in target appearance and background complexity, while suppressing clutter-induced activations more effectively. A representative example is given in the fourth image of Figure 6, where most competing methods mistakenly detect a bright structure in the lower-left region as a target. This error is primarily caused by their heavy reliance on local contrast cues without sufficient contextual modeling.

4.6. Ablation Studies

(1) Ablation Study on Module-wise Performance Gain: In this study, we initiate our investigation from the fundamental U-Net architecture, which serves as the baseline of our framework. Based on this baseline, we progressively incorporate the three proposed components, PCFE, MSCF and EUM. This incremental design enables a systematic evaluation of the individual contributions and collective effectiveness of each module in improving infrared small target detection performance.

The experimental results reported in Table 4 provide several important observations. (1) Each proposed module consistently improves detection performance, verifying the motivation of our design. (2) Compared with the baseline U-Net (strategy (a)), adding PCFE (strategy (b)) improves IoU by 3.60% and 3.53% on NUAA-SIRST and IRSTD-1K, respectively. This proves the feature extraction and enhancement capabilities of PCFE. (3) The comparison between strategy (a) and strategy (c) demonstrates the effectiveness of MSCF, which performs sparse cross-level fusion to suppress background clutter and enhance target–background discrimination. (4) Furthermore, the introduction of EUM improves feature recovery during decoding and strengthens the localization of small targets. (5) Finally, the full model (strategy (g)) achieves the best overall performance across all datasets, confirming the strong complementarity among PCFE, MSCF, and EUM.

Table 4. Ablation study on the effects of PCFE, MSCF, and EUM on three public SIRST datasets.

Strategy	Module			NUAA-SIRST		IRSTD-1K		NUDT-SIRST
	PCFE	MSCF	EUM	IoU	nIoU	IoU	nIoU	IoU	nIoU
(a)	✕	✕	✕	75.67	73.18	66.39	63.95	88.47	87.15
(b)	✓	✕	✕	79.27	77.56	69.92	67.12	92.03	90.87
(c)	✕	✓	✕	78.83	76.84	68.56	66.25	91.77	90.03
(d)	✕	✕	✓	77.94	75.62	67.48	64.88	90.12	88.96
(e)	✓	✓	✕	80.31	78.14	70.97	68.02	92.98	91.86
(f)	✓	✕	✓	80.11	77.96	70.83	67.89	92.96	91.74
(g)	✓	✓	✓	80.78	78.47	71.54	68.19	93.48	92.36

To further illustrate the trade-off between performance gain and computational cost, we report the model complexity in terms of parameters and FLOPs in Table 5.

Table 5. Complexity analysis of different module configurations.

Strategy	Module			Params	FLOPs
	PCFE	MSCF	EUM	(M)	(G)
(a)	✕	✕	✕	4.23	0.78
(b)	✓	✕	✕	13.58	22.54
(c)	✕	✓	✕	12.40	20.26
(d)	✕	✕	✓	5.78	1.53
(e)	✓	✓	✓	18.19	35.24

(2) Ablation on PCFE Design: We conducted an ablation study on the PCFE by selectively removing its key components, including the Patch Nonlocal Block (PNB) and the attention module. As shown in Table 6, the complete PCFE consistently achieves the best performance across all three datasets, demonstrating the effectiveness of jointly leveraging patch-wise contextual modeling and attention-based refinement.

Table 6. Ablation study on different designs of the PCFE.

Strategy	NUAA-SIRST		IRSTD-1K		NUDT-SIRST
	IoU	nIoU	IoU	nIoU	IoU	nIoU
w/o PNB + Attention	79.56	77.62	70.12	67.30	92.32	91.30
w/o PNB	79.81	77.97	70.54	67.71	92.82	91.80
w/o Attention	80.21	78.15	71.14	68.01	93.13	92.10
PCFE	80.78	78.47	71.54	68.19	93.48	92.36

Removing any component leads to a clear performance degradation, indicating that both PNB and attention contribute positively to feature enhancement. In particular, the performance drop caused by removing PNB is consistently larger than that caused by removing the attention module, suggesting that patch-wise contextual modeling plays a dominant role in enhancing weak target saliency and suppressing background clutter through global dependency aggregation. Meanwhile, the attention refinement module provides a complementary benefit by further strengthening channel–spatial discrimination, which helps improve target–background contrast, especially in complex scenes.

(3) Ablation on MSCF Sparsification: This experiment investigates how different sparsification strategies affect the quality of cross-level feature fusion and the final detection performance. The results are shown in Table 7. Without sparsification, MSCF aggregates all correlation responses indiscriminately, which introduces numerous background clutter interference and dilutes target-specific cues, leading to inferior performance.

Table 7. Ablation study on the Top-k sparsification strategy in the MSCF.

Setting	IoU	nIoU	Pd	Fa
w/o Top-k	79.67	77.59	99.21	3.83
only Top-$k_1$	80.35	78.12	99.49	1.96
only Top-$k_2$	80.45	78.16	99.58	1.74
Top-$k_1$ + Top-$k_2$	80.78	78.47	100.00	1.29

Introducing a single Top-k operation consistently improves performance. By filtering out low-response correlations, the fusion becomes more selective and focuses on highly informative target-related interactions, thus enhancing target saliency and suppressing background noise.

The best performance is achieved when MSCF jointly integrates both sparsification levels via learnable fusion weights. This dual-Top-k strategy leverages the complementarity between a stricter sparsification (better noise suppression) and a looser sparsification (better context preservation), yielding the highest IoU/nIoU and achieving a perfect Pd of 100.00 with the lowest Fa of 1.29. These results suggest that adaptive sparsification effectively balances target enhancement and background suppression during cross-level fusion.

5. Conclusions

In conclusion, to alleviate spatial detail degradation and background interference in infrared small target detection, we propose HPFNet, a Hierarchical Perception Fusion Network. Specifically, PCFE jointly exploits convolution-based local structure preservation and patch-wise contextual modeling, and further integrates channel–spatial attention to enhance feature discrimination from both local and global perspectives. MSCF explicitly performs cross-level interaction among shallow, intermediate, and deep features, and incorporates a dual Top-K sparsification mechanism to filter irrelevant background features, enabling the attention mechanism to focus more on the target region and thereby bolstering the discriminative power of feature representation. EUM combines upsampling with multiple parallel dilated convolution branches, which enhances feature reconstruction and improves spatial localization accuracy. Extensive experiments conducted on multiple benchmark datasets demonstrate the effectiveness and superiority of the proposed HPFNet.