FM-Net: Frequency-Aware Masked-Attention Network for Infrared Small Target Detection

Abstract

Infrared small target detection (IRSTD) aims to locate and separate targets from complex backgrounds. The challenges in IRSTD primarily come from extremely sparse target features and strong background clutter interference. However, existing methods typically perform discrimination directly on the features extracted by deep networks, neglecting the distinct characteristics of weak and small targets in the frequency domain, thereby limiting the improvement of detection capability. In this paper, we propose a frequency-aware masked-attention network (FM-Net) that leverages multi-scale frequency clues to assist in representing global context and suppressing noise interference. Specifically, we design the wavelet residual block (WRB) to extract multi-scale spatial and frequency features, which introduces a wavelet pyramid as the intermediate layer of the residual block. Then, to perceive global information on the long-range skip connections, a frequency-modulation masked-attention module (FMM) is used to interact with multi-layer features from the encoder. FMM contains two crucial elements: (a) a mask attention (MA) mechanism for injecting broad contextual feature efficiently to promote full-level semantic correlation and focus on salient regions, and (b) a channel-wise frequency modulation module (CFM) for enhancing the most informative frequency components and suppressing useless ones. Extensive experiments on three benchmark datasets (e.g., SIRST, NUDT-SIRST, IRSTD-1k) demonstrate that FM-Net achieves superior detection performance.

1. Introduction

Infrared small target detection (IRSTD) is an advanced technology in civil and military applications, such as early warning system [1], anti-drone [2], traffic monitoring [3], remote sensing [4] and so on. Since the detection platform is far away from targets and the imaging mechanism of infrared sensors is special, there are some difficulties that need to be addressed due to the following target characteristics.

• Sparse intrinsic features. Infrared targets typically occupy an extremely small number of pixels and thus lack shape and texture features.
• Low contrast. Infrared imaging relies on radiation energy of scenes, resulting in lacking color information. The boundary between targets and background is highly indistinct, so the saliency is unobvious.
• Dense clutter in complex scenes. There is heavy clutter and noise in the scenes with similar energy and structure as targets, which makes it difficult for high-precision separation.

Traditional IRSTD methods are model-driven and include three categories: filter-based methods [5,6], local-contrast-based methods [7,8] and low-rank-based methods [9,10,11]. Traditional detection methods rely heavily on prior knowledge and fine-tuned hyper-parameter. They lack flexibility and generalization when processing complex real scenes with changeable targets. In recent years, deep learning-based methods have been widely studied in IRSTD, which has promoted the rapid development of detection technology [12,13]. The convolutional neural network is the mainstream method and plays an important role [14,15,16]. Leveraging the advantage of feature fusion strategies and nested structure, Li et al. [17] proposed the dense nested attention network (DNANet) to extract and maintain target features in deep layers. Further, UIU-Net [18] built a powerful backbone through embedding a tiny UNet into a larger UNet, which implemented multi-level and multi-scale representation learning. Despite achieving promising performance, CNN-based methods have an attention bias in processing infrared images. In order to encode global and local information, Wu et al. [19] designed a multi-level transformer–CNN hybrid structure to fully exploit the long-range dependency of tiny targets. Then, Yuan et al. [20] deeply analyzed features and found that false alarms were usually caused by inaccurate estimation of backgrounds in deep layers. So, they designed a spatial-channel cross-transformer block (SCTB) to predict the context difference between targets and backgrounds, which could achieve highly robust detection [21]. However, the methods mentioned above only extract and enhance features from the spatial domain, with additional consideration given to the channel dimension. In fact, there are many valuable clues in the frequency domain that can be utilized to improve detection performance [22,23,24,25,26]. As shown in Figure 1, edges and boundaries are distinct in high-frequency components after applying WT to an infrared image. At the same time, the low-frequency component still retains lots of spatial details of targets and background. However, the gaps between global and local features, as well as spatial and frequency features, remain inadequately addressed. It is necessary to handle the above issues for better detection performance.

In order to exploit multi-scale frequency information to solve challenges in IRSTD, we propose frequency-aware masked-attention network (FM-Net) to strengthen the complementary advantages of spatial and frequency features. Specifically, we re-design residual block by embedding wavelet transform layer in the middle, which is called wavelet residual block (WRB). WRB combines multi-layer feature extraction and attention augmentation to effectively enhance target response. WRB also has varying receptive fields to deal with changeable targets. Subsequently, we propose the frequency-modulation masked-attention module (FMM) to aggregate full scale spatial and frequency feature. The masked-attention mechanism selectively highlights salient region and introduces long-range contextual perception. The following channel-wise frequency modulation module decouples features into distinguished low-frequency and high-frequency components, which are enhanced by modulated attention. The main contributions of this paper are outlined as follows:

• This paper proposes FM-Net, which effectively integrates frequency information to promote infrared small target detection. Wavelet residual blocks (WRBs) in encoders extract multi-level frequency details in feature maps, which have larger receptive fields to perceive local contexts.
• Instead of skip connection, a frequency-modulation masked-attention module (FMM) is proposed to facilitate semantic fusion across full-scale feature and learn long-range context association of infrared images.
• Experimental results on three benchmark datasets (e.g., SIRST, IRSTD-1k, NUDT-SIRST) demonstrate that our FM-Net achieves superior detection performance.

The architecture of this paper is as follows. Section 2 mainly introduces the related works on IRSTD, frequency learning and mask attention. Section 3 analyzes our FM-Net in detail. Experimental results in sing-frame datasets are showed in Section 4. In Section 5, we discuss our work and feature studies. Lastly, Section 6 presents our conclusion.

2. Related Works

2.1. Infrared Small Target Detection

IRSTD plays a significant role in infrared search and tracking system. At the beginning of related studies, scholars have proposed many traditional methods. Top-Hat [5] and Max-Median [6] are typical filter-based methods, which adopt a specific filter to suppress background noise interference. Local-contrast-based methods, such as WSLCM [7], TLLCM [8], obtain visual saliency regions and segment pixels by threshold. Then, low-rank-based methods [9,10,11] model target detection as a mathematical optimization problem for recovering low-rank and sparse tensors. However, traditional methods depend heavily on expert knowledge and manual parameter tuning. This impairs the ability to deal with complex scenes.

Recently, deep learning-based methods are rapidly evolving in IRSTD, which can learn target representation from large amounts of data. CNNs are popular detection methods and U-Net [27] is a widely adopted framework. On this basis, many useful techniques in computer vision have been applied to design excellent detection network [14,15,16,17,18], including dense connection, nested structure, specific attention and so on. Next, considering the significance of global context, researchers add a transformer block to the CNN backbone or replace skip connections, which well suppresses background clutters [19,20,21]. Lately, some new paradigms of neural networks have also been verified on IRSTD. For example, Zhang et al. [28] redesigned the general SAM to learn better feature representation of infrared targets, addressing the low-SNR issue. From a novel perspective of the pixel movement mechanism, the computational fluid dynamic network (CFD-Net) [29] facilitates robust feature learning and improves detection performance.

Despite existing detection methods having showcased promising performance, these methods primarily extract spatial features and enhance information in the channel dimension. Direct and simple processes from the spatial domain is insufficient and easily causes false alarms. To further introduce more useful information, researchers add frequency-domain processing. Zhu et al. [22] extracted local features in image patches and adopted multi-head attention to improve low-frequency and high-frequency information. He et al. [23] firstly presented the EE module to enrich features, then frequency information was retained through downsampling and fusion with wavelet transformer. Huang at al. [24] utilized two learnable masks to extract target-specific spectra and removed background clutter by maximizing the difference in frequency characteristics. Next, to suppress high-frequency disturbances, FreqODEs [25] proposed a set of ODE convolutional modules, which contained coefficient learning to lower the risk of gradient vanishing. FACFPNet [26] used multi-head self-attention to capture multi-scale frequency features and enhance global context representation.

As we considered, existing methods still struggle to perceive multi-scale context and exploit multiple-frequency feature. Inspired by frequency learning [30,31,32,33], effective information from frequent domains can help network improve performance in vision tasks. Therefore, we explore how frequency features can effectively empower IRSTD.

2.2. Mask Attention

Mask attention is firstly proposed in Mask2Former [34], which can address universal image segmentation through a single model. Mask attention is a variant of multi-head self-attention, which is a key component that helps Mask2Former converge faster and extract localized features. Then, researchers adopted it to many tasks and its cross-domain applicability has been validated. Cheng et al. [35] integrated mask attention module in U-Net, achieving robust low-resolution image segmentation. The advantage of mask attention is that it dynamically adjusts attention maps, increasing focus on target regions. Furthermore, the attention facilitates awareness of global context while effectively fusing multi-scale features. However, the lone spatial attention mechanism leads to the loss of feature details that is crucial for IRSTD. Therefore, we design a frequency-modulation masked-attention module (FMM) by combining attention with frequency augmentation, refining fine-grained information. A detailed introduction is given in Section 3.3.

2.3. Frequency Learning on Vision Tasks

Frequency-domain analyses have been a significant tool for signal processing [36]. The technology has recently been applied in deep learning and demonstrates potential in natural image processing [30,31,32,33]. It is generally believed that detailed structures and boundaries mainly lie in high-frequency components and high-level semantic information is abundant in low-frequency components [31,37]. For feature fusion, FreqFusion [32] applied different frequency filters to increase feature consistency and make object boundary clearer. This provides a robust feature fusion scheme for dense image prediction. For feature extraction, FDConv [30] improved the frequency adaptability and diversity of parallel weights, enabling better feature capture. WTConv [33] was a new method that used wavelet transform to effectively expand the receptive field of CNN. Meanwhile, the convolutions were performed on different frequency subbands to obtain a better low-frequency response. FADC [38] adjusted dilation based on local frequency components, balancing effective bandwidth and receptive field. On the other hand, frequency learning empowers multiple tasks. For image restoration, FSNet [31] dynamically decoupled features into different subbands and selected the most useful one to recover. Then, multi-scale pooling operations performed calibration to obtain a large receptive field. For pansharpening, WFANet [39] utilized the wavelet pyramid for progressive fusion, where frequency attention triplets were designed to enhance precision and reduce information confusion. For object detection, SFS-Conv [40] employed a shunt–perceive–select frequency convolution to enhance distinctiveness. In summary, frequency learning can bring gains to solving problems in vision tasks.

In this work, we insert a wavelet pyramid into the residual block, extracting multi-scale spatial and frequency features. Furthermore, channel and spatial attention modules are further joined to capture detailed structures. Subsequently, to integrate multi-scale features across levels, we design the cascaded masked-attention and frequency-modulation module, where masked attention extracts global spatial context and the frequency-modulation module adaptively selects and enhances informational frequency features.

3. Methodology

In this section, we introduce our FM-Net in detail.

3.1. Overall Architecture

Figure 2 shows the specific architecture of FM-Net. First, the input is a single frame infrared small target image. Then, it is sequentially processed by the encoder composed of a wavelet residual block (Section 3.2) to extract deep features. Encoded features are progressively fused and recover resolution through decoder and frequency-modulation masked-attention module (Section 3.3). Last, the pixel-level prediction is generated.

FM-Net employs the ResNet-18 [41] with our proposed WRB as the backbone, which obtains deep features ${\mathbf{F}}_{\mathbf{i}}\in {\mathbb{R}}^{C_i\times {\displaystyle \frac{H}{2^{i-1}}}\times {\displaystyle \frac{W}{2^{i-1}}}}, (i = 1, 2, 3, 4)$. $C_i$ is the channel dimension of features at each layer, where $C_1 = 16, C_2 = 32, C_3 = 64, C_4 = 128$. We separately explain the fifth layer in the encoder, which has 128 channels with a resolution of 32 × 32. Subsequently, encoded features ${\mathbf{F}}_{\mathbf{i}}$ are processed into patch embedding ${\mathbf{E}}_{\mathbf{i}}\in {\mathbb{R}}^{C_i\times {\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}}$ through convolutions with kernel size and stride size of $16, 8, 4, 2$. The embedding ${\mathbf{E}}_{\mathbf{i}}$ across layers are inputted into FMM for full-level features perceiving and blending. The outputs ${\mathbf{O}}_{\mathbf{i}}\in {\mathbb{R}}^{C_i\times {\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}}$ of FMM keep the same size of ${\mathbf{E}}_{\mathbf{i}}$. Skip connections are used to add ${\mathbf{E}}_{\mathbf{i}}$ and ${\mathbf{O}}_{\mathbf{i}}$ to maintain multi-granularity semantics. Then, ${\mathbf{O}}_{\mathbf{i}}$ at each layer are restructured (Rt) and recovered to the size of original ${\mathbf{F}}_{\mathbf{i}}$, using bilinear upsample. Furthermore, convolution blocks are used to obtain decoded features ${\mathbf{D}}_{\mathbf{i}}\in {\mathbb{R}}^{C_i\times {\displaystyle \frac{H}{2^{i-1}}}\times {\displaystyle \frac{W}{2^{i-1}}}}, (i = 1, 2, 3, 4)$. Skip connections are employed to merge the features between encoder and decoder. The above process can be mathematically expressed in the following formula.

$${\mathbf{D}}_{\mathbf{i}}={\mathbf{F}}_{\mathbf{i}}+{\mathbf{Rt}}_{\mathbf{i}}\left(\mathrm{FMM}({\mathbf{E}}_{\mathbf{1}},{\mathbf{E}}_{\mathbf{2}},{\mathbf{E}}_{\mathbf{3}},{\mathbf{E}}_{\mathbf{4}})\right),i=1,2,3,4.$$

(1)

Finally, following the paradigm of the decoder in UNet, we utilize channel cross-attention (CCA) [42] to assist in integrating the high-level and low-level features. The high-level feature ${\mathbf{L}}_{\mathbf{i}+\mathbf{1}},(i=1,2,3,4)$ is first upsampled to recover the resolution. CCA is used to fuse the features from upsampling and skipping. Then, the attention-enhanced and upsampled features are concatenated together and decoded by two CBL blocks. The progress can be expressed mathematically as

$${\mathbf{L}}_{\mathbf{i}}=\mathrm{CBL}\left([up\left({\mathbf{L}}_{\mathbf{i}+\mathbf{1}}\right),\mathrm{CCA}(up\left({\mathbf{L}}_{\mathbf{i}+\mathbf{1}}\right),{\mathbf{D}}_{\mathbf{i}})]\right),i=1,2,3,4,$$

(2)

where ${\mathbf{L}}_{\mathbf{i}}$ indicates the decoded feature of $i_{th}$ layer. $up(·)$ indicates the upsample operation. $[·,·]$ indicates the concatenate operation. The CBL block consists of convolution, batch normalization and ReLU.

To help the network converge efficiently and adapt to multi-granularity predictions, we utilize a multi-level deep supervision strategy to optimize the training of FM-Net. Specifically, each output of decoder ${\mathbf{L}}_{\mathbf{i}}$ is processed by point-wise convolution (PW) and sigmoid activation to generate a predicted mask ${\mathbf{M}}_{\mathbf{i}}$. The process can be mathematically expressed as

$${\mathbf{M}}_{\mathbf{i}}=\mathrm{Sigmoid}\left(\mathrm{PW}\left({\mathbf{L}}_{\mathbf{i}}\right)\right),i=1,2,3,4,5.$$

(3)

Then, multi-scale masks are upsampled to the original image resolution. Furthermore, these masks are concatenated together and fused to acquire ${\mathbf{M}}_{\mathbf{cat}}$.

$${\mathbf{M}}_{\mathbf{cat}}=\mathrm{Sigmoid}\left(\mathrm{PW}\left([up\left({\mathbf{M}}_{\mathbf{1}}\right),up\left({\mathbf{M}}_{\mathbf{2}}\right),up\left({\mathbf{M}}_{\mathbf{3}}\right),up\left({\mathbf{M}}_{\mathbf{4}}\right),up\left({\mathbf{M}}_{\mathbf{5}}\right)]\right)\right).$$

(4)

Finally, we calculate the total loss between ${\mathbf{M}}_{\mathbf{i}}$, ${\mathbf{M}}_{\mathbf{cat}}$ and ground truth (GT). As below, we use the DiceTopK loss (DT) [43] as our optimization loss:

$$l_1={\mathcal{L}}_{DT}({\mathbf{M}}_{\mathbf{1}},\mathrm{GT}),$$

(5)

$$l_i={\mathcal{L}}_{DT}(up\left({\mathbf{M}}_{\mathbf{i}}\right),\mathrm{GT}),i=2,3,4,5,$$

(6)

$$l_{cat}={\mathcal{L}}_{DT}(up\left({\mathbf{M}}_{\mathbf{cat}}\right),\mathrm{GT}),$$

(7)

$$l_{total}=l_1+l_2+l_3+l_4+l_5+l_{cat},$$

(8)

where ${\mathcal{L}}_{DT}$ indicates the DiceTopK loss, which combines Dice loss [44] and Top-K loss [45]. Dice loss can describe regional overlap and is sensitive to target contour. Top-K loss is a variant of cross-entropy (CE) loss. It retains the top k% of the worst predicted pixel to calculate loss values, which aims to focus on hard samples. In other words, the ranking criterion is the values of CE between predicted pixels and groundtruth pixels. The composite loss, DiceTopK, combines the advantages of two functions, obtaining shape and location sensitivity. The definition is as follows.

$${\mathcal{L}}_{DT}={\mathcal{L}}_{Dice}+{\mathcal{L}}_{TopK},$$

(9)

$${\mathcal{L}}_{Dice}=1-{\displaystyle \frac{2{\sum }_{c=1}^C{\sum }_{i=1}^V{p}_i^c{g}_i^c}{{\sum }_{c=1}^C{\sum }_{i=1}^V{p}_i^c+{\sum }_{c=1}^C{\sum }_{i=1}^V{g}_i^c}},$$

(10)

$${\mathcal{L}}_{Top\_k}=-{\displaystyle \frac{1}{V}}\sum _{c=1}^C\sum _{i\in \mathbf{K}}^V{g}_i^c\log \left(p_i^c\right),$$

(11)

where $p_i$ and $g_i$, respectively, denote the values of pixel i in mask, GT. V is the number of pixels in image. C is the number of classes, and $C=2$ since IRSTD is a binary classification problem. $\mathbf{K}$ is the set of $k\%$ worst predicted pixels. In this paper, $k=10$.

3.2. Wavelet Residual Block

To distinguish targets from complex background, the network needs to make full use of comprehensive information hidden in semantic features. As analyzed in Section 2, we introduce the frequency-domain process to obtain feature information from multiple bands of spectrum. It is generally believed that CNN can employ many convolution operations of multiple sizes and branches to perceive different frequency information, which leads to a lot of computation cost. In contrast, wavelet transform dose not introduce additional parameters. Our wavelet layer can efficiently extract multi-scale features. Figure 3 shows our designed wavelet residual block (WRB), which employs simple operations to make the spatial–frequency feature extraction more direct and complete. Overall, a wavelet layer is inserted into a residual block. The wavelet layer initially employs WT on feature maps and obtains low-frequency and high-frequency components. Multiple frequency components are concatenated together and enhanced by the convolution and spatial attention module (CBAM) [17]. This promotes an increased fine-grained frequency information and a reduced spatial resolution for the lower frequencies. Subsequently, inverse wavelet transform (IWT) is used to construct the output completely.

Further, we utilize WT multiple times. According to Figure 1 and Figure 3, it is found that LH, HL and HH components have more high-frequency feature details, which can be utilized to enhance target response. More processing may harm these valuable components, while the low-frequency component (LL) still has similar features as the original image; it is considered that deep features can be extracted further to perceive multi-scale frequency information. Therefore, the low-frequency component $X_j^{LL}$ is again employed in WT to obtain different frequency components. Repeating the above steps, the same operations can be performed to extract deeper features. After extracting multiple WTs, IWT is adopted to fuse multi-level feature and recover size. The enhanced $Y_{j+1}$ in the deeper level are summed with low-frequency $Z_j^{LL}$ in the lower level. Furthermore, a new convolution and CBAM are employed to extract and enhance spatial–frequency details. The process is given by

$$X_j^{LL},X_j^H=\mathrm{WT}\left(X_{j-1}^{LL}\right),$$

(12)

$$Z_j^{LL},Z_j^H=\mathrm{CBAM}\left(Conv\left([X_j^{LL},X_j^H]\right)\right),$$

(13)

$$Y_j=\mathrm{IWT}\left([Z_j^{LL}+Y_{j+1},Z_j^H]\right),$$

(14)

where $X_0^{LL}$ represents the input features of wavelet layer. $X_j^H$ represents all high-frequency maps of $j_{th}$ level. $Z_j^{*}$ denotes the intermediate processing result in the $j_{th}$ level. $Y_j$ is the aggregated output in the $j_{th}$ level. For the deepest level, there is no $Y_{j+1}$. In this paper, $j=2$. We adopt Haar wavelet transform. $Conv(·)$ represents convolution operation.

3.3. Frequency-Modulation Masked-Attention Module

Successful architecture such as SCTB [20] has considered the interaction of spatial and channel information in constructing context. The method heavily focuses on the spatial domain but neglects the crucial effect of the frequency domain in context modeling. To solve it, we develop a frequency-modulation masked-attention module (FMM) as a spatial–frequency modulator to integrate full-level encoded features. As shown in Figure 4, given the $i_{th}$ layer embedding feature ${\mathbf{E}}_{\mathbf{i}}\in {\mathbb{R}}^{C_i\times {\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}}$, the procedure of FMM can be organized as

$${\mathbf{O}}_{\mathbf{i}}=\mathcal{F}\left(\mathrm{CFM}\left(\mathrm{MA}\left([{\mathbf{E}}_{\mathbf{1}},{\mathbf{E}}_{\mathbf{2}},{\mathbf{E}}_{\mathbf{3}},{\mathbf{E}}_{\mathbf{4}}]\right)\right)\right),$$

(15)

where $\mathcal{F}(·)$ indicates the point-wise convolution used to fuse the modulated features and adjust the number of channels.

3.3.1. Mask Attention Mechanism

Mask attention is improved from multi-head self-attention, which introduces a learnable mask to adjust the attention matrix. It is considered that local features are enough to update query features [34]. Furthermore, adding an adaptive mask promotes the network to focus more on interesting regions. Assuming the input feature ${\mathbf{E}}_{\mathbf{t}}\in {\mathbb{R}}^{C_t\times h\times w}$, it is firstly reshaped to ${\mathbf{E}}_{\mathbf{t}}^{\prime }\in {\mathbb{R}}^{hw\times C_t}$, where $C_t={\sum }_{i=1}^4{C}_i$, $h={\displaystyle \frac{H}{P}}$, $w={\displaystyle \frac{W}{P}}$. Then, the features are transformed linearly to obtain query Q, key K, value V. The attention matrix with an additional mask $M\in {\mathbb{R}}^{hw\times hw}$ is generated by

$$MA=Rs(Norm(Softmax({\displaystyle \frac{Q{K}^T}{\sqrt{C_t}}}+M)V+{\mathbf{E}}_{\mathbf{t}}^{\prime })),$$

(16)

where $Q={\mathbf{E}}_{\mathbf{t}}^{\prime }{W}_q$, $K={\mathbf{E}}_{\mathbf{t}}^{\prime }{W}_k$, $V={\mathbf{E}}_{\mathbf{t}}^{\prime }{W}_v$. $W_{*}$ is the corresponding weight. $Rs(·)$ indicates the reshape operation. $Norm(·)$ represents the layer normalization. $Softmax(·)$ is the activation function. M is the learnable mask, which is computed dynamically. It helps to reduce the contribution of weakly informative regions and focus on relevant space.

3.3.2. Channel-Wise Frequency Modulation Module

MA efficiently extracts the long-range association in the spatial domain, and the clues in the frequency domain are not utilized adequately. To distinguish targets and background from the fine point, we design a channel-wise frequency modulation module (CFM) to widen the semantic gap between target and clutter. In general, input features are split into two parts in channel dimension. Then, two cascaded decoupling–modulation modules extract frequency details. The decoupler dynamically filters and selects separate frequency components. The modulator adopts the effective strategy, including disassemble, enhance and fuse, to process features. It enriches informative frequency components. Specifically, given the input X, the decoupler generates the low-pass and high-pass filters, formulated as

$$f_l=Softmax\left(BN\left(Conv\left(GAP\left(X\right)\right)\right)\right),$$

(17)

$$f_h=I-f_l,$$

(18)

where $BN(·)$ is the batch normalization. $GAP(·)$ represents the global average pooling. High-pass filters $f_h$ are obtained through subtracting the $f_l$ from the identity kernel I. Then, low-frequency and high-frequency components are acquired by

$$X_l=\sum f_{l}X,$$

(19)

$$X_h=\sum f_{h}X.$$

(20)

After decoupling, modulation uses channel-wise attention to enhance the informative components, formulated as

$$w_h,w_l=sp\left(Softmax\left([fc\left(GAP(X_l+X_h)\right),fc\left(GAP(X_l+X_h)\right)]\right)\right),$$

(21)

$$x_{global},x_{local}=sp\left(Conv(X_l{w}_l+X_h{w}_h)\right),$$

(22)

where $sp(·)$ represents the channel-wise split operation. In Formula (21), we divide the feature maps into two parts from the middle, where $w_h$ is the first half of a small channel index and $w_l$ is the second half of a large channel index. In Formula (22), we apply the decoupler and modulator modules twice with varying kernel sizes and concatenate the output features. Then, split operation divides the output features into two parts, where $x_{global}$ is the first half of a small index and $x_{local}$ is the second half of a large index.

Next, the low-frequency and high-frequency features are enhanced and aggregated from local and global perspectives. $GAP$ is firstly used to obtain low-frequency part, and the features subtract the low-frequency part to obtain the corresponding high-frequency part. To enhance two frequency subbands, the learnable scaled weights are introduced to emphasize contributions. Addition is used to fuse two frequency parts. The global and local components are concatenated together and fused by convolution. CFM decomposes, improves and reaggregates full-level features to gain multi-frequency information, which highlights target semantics.

Finally, convolutions are used to fuse full-level splicing features and adjust the channel dimension in each layer, generating the output ${\mathbf{O}}_{\mathbf{i}}$ of FMM.

4. Experiments

In this section, we introduce evaluation metrics, experiment setting, comparison with the state-of-art-art (SOTA) methods and ablation study.

4.1. Evaluation Metrics

In this paper, we compare FM-Net with several state-of-the-art (SOTA) methods on common metrics.

(1) Intersection over Union. $\mathrm{IoU}$ is a pixel-level metric, which reflects the profile description ability.

$$\mathrm{IoU}={\displaystyle \frac{A_I}{A_{\mathrm{U}}}},$$

(23)

where $A_I$ and $A_{\mathrm{U}}$, respectively, indicate the intersection and union area between prediction and ground truth.

(2) Probability of Detection. $P_d$ is a target-level metric, which reflects the localization accuracy.

$$P_d={\displaystyle \frac{N_c}{N_t}},$$

(24)

where $N_{correct}$ represents the number of correctly predicted targets. $N_t$ represents the number of total targets. When the centroid deviation between prediction results and ground truths is less than a predefined threshold, we believe those targets are correctly detected. The threshold is 3 in this paper.

(3) False Alarm Rate. $F_a$ is a target-level metric, which reflects the wrong detection of false objects.

$$F_a={\displaystyle \frac{P_f}{P_t}},$$

(25)

where $P_f$ represents the number of falsely predicted pixels. $P_t$ represents the number of total pixels. When the centroid deviation is larger than the threshold, the prediction is considered as false pixels.

(4) Receiver Operation Characteristics. The ROC curve describes the general trend of $P_d$ as $F_a$ varies. Furthermore, it indicates the total effect under a sliding threshold.

(5) Parameters. #Params reflects the number of learnable parameters in model, which is related to the size of convolution kernel. Given 2D convolution $k\in {\mathbb{R}}^{C_{in}\times K_h\times K_w}$, the input feature is F

$$\#\mathrm{Params}=C_{in}\times C_{out}\times K_h\times K_w,$$

(26)

where $C_{in}$ and $C_{out}$ represent the number of input and output channels. $K_h$ and $K_w$ represent the kernel height and width of 2D convolution.

4.2. Experiment Setting

We used three benchmark datasets, including SIRST [14], NUDT-SIRST [17] and IRSTD-1k [16], which, respectively, contain 427, 1327 and 1001 images. We followed the standard method in [17] to split the train and test sets for three datasets. During training, input images are firstly normalized and randomly flipped for data augmentation. Furthermore, all input images have a resolution of $512\times 512$ by filling or cropping. Note that a resize operation easily causes position shift and deformation for small targets, which should be avoided. The model is trained using the AdaGrad optimizer. We set the learning rate, batch size and epoch number as 0.05, 8 and 1000, respectively. Networks are implemented by PyTorch 2.3.0 on a single Nvidia GeForce RTX 3090 GPU. Traditional detection methods are implemented and tested in MATLAB R2023b on an Intel Core i9-13900HX CPU.

To evaluate the performance of FM-Net, we compare our FM-Net to some SOTA IRSTD methods. Traditional methods include filter-based Top-Hat [5] and Max-Median [6], local-contrast-based WSLCM [7] and TLLCM [8], low-rank-based IPI [9] and MSLSTIPT [4]. Deep learning methods include ACM [14], DNANet [17], UIUNet [18], MTU-Net [19], SCTransNet [20], WTAPNet [23]. In order to ensure a fair comparison, we collect and implement the official codes of the above methods. At the same time, we keep the consistent training settings and hyper-parameters of their original papers. Our code will be available at https://github.com/yongxianLiu/ (accessed on 25 May 2025).

4.3. Quantitative Results

Quantitative results are shown in

Table 1. Result comparison of detection performance and model size with the SOTA methods on three datasets, where $\mathrm{IoU}(\times {10}^{-2})$, $P_d(\times {10}^{-2})$, $F_a(\times {10}^{-6})$, #Params$\left(M\right)$, FLOPs$\left(G\right)$. The best results are in red and the second best results are in blue. Spatial and Freq refer to spatial-based and frequency-based methods, respectively.

Method			IRSTD-1k [16]			SIRST [14]			NUDT-SIRST [17]			#Params↓	FLOPs↓
			IoU↑	Pd↑	Fa↓	IoU↑	Pd↑	Fa↓	IoU↑	Pd↑	Fa↓
Traditional		Top-Hat [5]	10.06	75.11	1432	7.14	79.84	1012	20.72	78.41	166.7	-	-
		Max-Median [6]	7.00	65.21	59.73	4.17	69.20	55.33	4.197	58.41	36.89	-	-
		WSLCM [7]	3.452	72.44	6619	1.16	77.95	5446	2.283	56.82	1309	-	-
		TLLCM [8]	3.311	77.39	6738	1.03	79.09	5899	2.176	62.01	1608	-	-
		IPI [9]	27.92	81.37	16.18	25.67	85.55	11.47	17.76	74.49	41.23	-	-
		MSLSTIPT [4]	11.43	79.03	1524	10.30	82.13	1131	8.342	47.40	888.1	-	-
Deep Learning	Spatial	ACM [14]	60.33	93.94	69.31	68.34	93.92	30.12	64.86	96.72	28.59	0.398	1.33
		DNANet [17]	65.74	90.24	13.17	73.60	93.54	38.49	94.19	99.26	2.436	4.697	56.11
		UIUNet [18]	65.69	91.58	14.18	76.21	92.40	10.70	93.48	98.31	7.79	50.54	186.28
		MTU-Net [19]	66.11	93.27	36.80	74.78	93.54	22.36	74.85	93.97	46.95	8.221	24.777
		SCTransNet [20]	68.03	93.27	10.74	76.32	96.96	14.41	94.10	98.73	7.101	11.20	40.43
	Freq	WTAPNet [23]	67.21	92.75	27.82	76.26	94.29	24.83	87.71	99.07	9.537	14.83	103.24
		FM-Net (Ours)	68.13	95.62	8.085	76.75	96.96	15.710	94.76	99.15	1.838	3.606	14.171

. It is clear that deep learning-based methods are significantly superior to traditional ones in terms of detection accuracy and contour prediction. Those traditional methods are model-driven and heavily rely on tuning hyper-parameters, resulting in poor robustness to complex scenes. Meanwhile, our proposed FM-Net achieves excellent performance on three benchmark datasets. In particular, on the IRSTD-1k dataset, FM-Net achieves the best results on all metrics. On the SIRST dataset, our method achieves superior results in $\mathrm{IoU}$ and $P_d$, with overall performance close to that of SCTransNet. Although FM-Net does not perform as well as UIUNet on $F_a$, our method achieves balanced and outstanding performance at an acceptable computational cost. On the NUDT-SIRST dataset, FM-Net achieves 99.15% in terms of $P_d$, which is lower than the $P_d$ of DNANet, but with an increase in $\mathrm{IoU}$ and a significant decrease in terms of $F_a$. In summary, FM-Net effectively extracts the spatial–frequency information to detect infrared small targets in a dense clutter. It employs frequency decoupling and modulation to amplify the difference between targets and background in the frequency domain, greatly reducing the occurrence of false alarms. At the same time, FM-Net maintains target feature in deep layers and refines semantic details through enhancing multi-frequency components. Therefore, FM-Net can not only locate targets accurately but also finely contour them.

As shown in Figure 5, we plot the ROC curves of deep learning-based methods on three datasets. Our FM-Net achieves the optimal win–win situation with regard to $P_d$ and $F_a$. It is evident that FM-Net can deal with different complex scenes and changeable targets. On the IRSTD-1k dataset, FM-Net achieves the highest $P_d$ under low $F_a$, and the curve is at the top. On the NUDT-SIRST dataset, FM-Net obtains a curve similar to that of DNANet; both demonstrate a distinct advantage over other methods.

4.4. Qualitative Results

Qualitative results of six detection methods are shown in Figure 6 and Figure 7. We can observe that targets are in a wide variety of sizes and they are similar to the background with complex clutter. Our FM-Net can segment targets with precise localization and depict the dedicate contour. In Figure 6(1), FM-Net and UIUNet can match target shape, but other methods just segment partial targets or the expanded area. Moreover, for background highlights with similar characteristics to targets, FM-Net can accurately eliminate false alarms and locate targets drowned in dense clutter. However, other methods result in poor detection performance, as shown in Figure 6(2–6). In Figure 7, we present the detection results of an additional six scenes. The brightness of the background area is usually similar to or even exceeds that of targets, making separation exceptionally difficult. In visualization comparison, FM-Net exhibits robust performance to different complex scenes, which suppresses background interference and detects targets precisely. This is due to the fact that FM-Net effectively utilizes frequency information to model global and local features, extracting and maintaining detailed target semantics. Furthermore, the processing in the frequency domain helps to distinguish differential semantic features between targets and background.

4.5. Ablation Study

In this section, we modify FM-Net and evaluate the performance of several variants to demonstrate the practical benefits of our designed modules and tricks.

4.5.1. Network Framework

To obtain the best performance and corresponding network, we change network framework, including channel numbers and feature size. As shown in

Table 2. Results achieved by FM-Net’s variants with changed channel numbers on IRSTD-1k dataset, where C indicates all channel numbers of features at each level. Bold indicates the optimal values.

Model Channel	Metrics			#Params↓
	IoU↑	Pd↑	Fa↓
C = 8, 16, 32, 64, 64	66.50	88.22	9.015	0.921
C = 16, 32, 64, 128, 128	68.13	95.62	8.085	3.606
C = 32, 64, 128, 256, 256	67.65	94.28	30.745	14.209

, we change the channel number of the encoder in FM-Net. In comparison, FM-Net with C = 16, 32, 64, 128, 128 achieves the highest performance metrics. An insufficient number of channels makes it difficult to acquire lots of divisive features. However, redundant channels bring interference information and model complexity. According to Formula (26), the change of channel numbers dramatically affects model parameters. Although the model with fewer channel has fewer parameters, the detection performance is poor. All things considered, C = 16, 32, 64, 128, 128 is the most appropriate configuration.

During extraction in FM-Net, encoded features are processed into patch embeddings. To explore the effect of patch size on detection performance, we change the size of embedding and display experimental results in

Table 3. Results achieved by FM-Net’s variants with different patch size on IRSTD-1k dataset. P indicates the uniform scale of embeddings. Bold indicates the optimal values.

Model Patch Size	Metrics			#Params↓
	IoU↑	Pd↑	Fa↓
P = 16 × 16	67.96	89.23	30.593	4.392
P = 32 × 32	68.13	95.62	8.085	3.606
P = 64 × 64	67.44	92.93	13.873	3.409

. FM-Net with 32 × 32 patch obtains the best performance and has a moderate number of parameters. A larger patch will introduce lots of computational complexity, making training slow. A small patch loses feature information and semantic details, leading to performance degradation.

Then, we evaluate the gain brought to FM-Net by the multi-level deep supervision strategy. In

Table 4. Results achieved by FM-Net with different train strategy on IRSTD-1k dataset. Bold indicates the optimal values.

Train Strategy	Metrics
	IoU↑	Pd↑	Fa↓
FM-Net w/o DS	67.32	92.93	12.545
FM-Net	68.13	95.62	8.085

, FM-Net w/o DS represents network trained without deep supervision. The deep supervision strategy can help FM-Net achieve performance improvements across the board. Focusing on different levels of prediction results can facilitate network training. This is because the strategy can mitigate gradient disappearance and promote hierarchical feature learning.

4.5.2. Impact of WRB

Wavelet residual block (WRB) in the encoder employs wavelet transform to acquire multi-scale and multi-level spatial–frequency information. The block helps enhance target responses through adaptively fusing multi-frequency information, which also serves to suppress interference. To verify the effectiveness of WRB, we compare the detection performance of FM-Net with several variants of WRB.

• FM-Net w/o WL. We remove the wavelet layer and only use the residual block to extract features in the encoder.
• FM-Net w/o CBAM. We remove all CBAMs in WRB and only use convolution operation to process frequency maps.
• FM-Net w j = 1. We change the deepest level of wavelet layer, only performing feature extraction and enhancement once.
• FM-Net w j = 3. We change the deepest level of wavelet layer and the wavelet pyramid has three levels.
• FM-Net w db2. We change the type of wavelet transform, using Daubechies wavelet of order 2.
• FM-Net w coif1. We change the type of wavelet transform, using Coiflets wavelet of order 1.

Table 5. Results achieved by main variants of WRB on IRSTD-1k dataset. Bold indicates the optimal values.

Model	Metrics			#Params↓
	IoU↑	Pd↑	Fa↓
FM-Net w/o WL	67.42	86.87	25.754	3.239
FM-Net w/o CBAM	67.99	92.93	46.346	3.300
FM-Net w j = 1	68.03	91.58	11.482	3.427
FM-Net w j = 3	67.58	91.58	14.670	3.785
FM-Net w db2	67.71	93.94	20.876	3.606
FM-Net w coif1	66.71	91.25	9.394	3.606
FM-Net	68.13	95.62	8.085	3.606

shows the results achieved by FM-Net with different variants of WRB. If a wave layer is removed, the detection performance suffers a significant reduction, especially in terms of $F_a$. As shown in Figure 8, WRB helps FM-Net to obtain more specific and distinct semantic features. Removing WRB significantly reduces the network’s ability to localize targets and suppress background clutters. That is also verified in Figure 3: when feature maps pass through WRB, target semantics are clearer and some background interferences are eliminated. Then, CBAM can enhance semantic information, greatly bringing an increase in terms of IoU, $P_d$ and $F_a$. Next, $j=2$ is the optimum level of WRB. Larger levels cannot obtain more details and lead to more computation. As for types of wavelet transform, FM-Net adopts Haar wavelet and achieve the highest performance metrics.

4.5.3. Impact of FMM

The frequency-modulation masked-attention module (FMM) plays an equivalent role of skip connections, which perceives long-range semantic association and enhances target response through global attention and multi-frequency enhancement. FMM can effectively bridge the semantic gap between encoder and decoder features. To investigate the benefits of FMM, we compare FM-Net with several variants which adopt other connection methods. Experimental results are in

Table 6. Results achieved by main variants of FMM on IRSTD-1k dataset. Bold indicates the optimal values.

Model	Metrics			#Params↓
	IoU↑	Pd↑	Fa↓
FM-Net w/o FMM	67.85	92.93	25.45	2.231
FM-Net w cat	67.60	94.61	33.934	2.288
FM-Net w/o MA	66.95	93.60	17.062	3.432
FM-Net w MA2	67.64	91.25	34.978	3.780
FM-Net w MA3	67.01	91.25	46.687	3.954
FM-Net w/o CFM	67.87	94.28	9.375	2.462
FM-Net	68.13	95.62	8.085	3.606

.

• FM-Net w/o FMM. Removing FMM, we use the plain skip connection to pass encoded features to decoder.
• FM-Net w cat. We remove the FMM in this variant and still utilize convolutions to aggregate the encoded features, which are concatenated together
• FM-Net w/o MA. In this variant, we remove the MA module in FMM to investigate the benefit of MA.
• FM-Net w MA2. We change the number of MA in FMM. This variant has two cascaded MA modules to extract feature.
• FM-Net w MA3. This variant has three cascaded MA modules in FMM.
• FM-Net w/o CFM. We remove the CFM in FMM in this variant to investigate the benefit of CFM.

FM-Net w/o FMM suffers decreases of 0.28%, 2.69% and an increase of 17.365 $\times {10}^{-6}$ in terms of IoU, $P_d$ and $F_a$. This is because FMM models global correlation and enhances target response by modulating multi-frequency components. Visualization maps in Figure 9 demonstrate the effectiveness of FMM. In the L1 maps of FM-Net w/o FMM, there is a highly responsive region which does not contain targets. This easily causes false alarm. Moreover, comparing the corresponding feature maps, such as L2 and L4, we can also find that FM-Net predicts the global context better, which can facilitate the separation of targets and background. In L3, the target region of FM-Net has higher response and is more focused. It demonstrates that FMM contributes to refining semantic information in the deep layer.

The detection performance of FM-Net w cat is poor. Plain aggregation using convolution cannot bridge the semantic gap as FMM does. Comparing the results of different numbers of MA, a single attention head can obtain the optimal results.

FM-Net w/o MA suffers decreases of 1.18%, 2.02% and an increase of 8.977 $\times {10}^{-6}$ in terms of IoU, $P_d$ and $F_a$. This is because MA can model global spatial context and focus on targets’ region. FM-Net w/o CFM suffers decreases of 0.26%, 1.34% and an increase of 1.29 $\times {10}^{-6}$ in terms of IoU, $P_d$ and $F_a$. This is because CFM decouples multiple frequency components of features and exploits adaptive channel attention to enhance the informative ones. The subsection proves that FMM is valid for IRSTD.

5. Discussion

Nowadays, there is a growing number of IRSTD methods based on deep learning, yet most of them only rely on spatial features to detect targets. Studies have shown that this is not enough. In this paper, we consider that it is important to model global context and utilize frequency information for IRSTD. The study introduces the frequency-aware masked-attention network, which contains a novel extraction module containing wavelet transform and a novel aggregation module utilizing frequency modulation. The network structures are detailed and extensive comparison experiments are conducted. Furthermore, the ablation study demonstrates the effectiveness of designed modules. Feature research can further explore some processing in the frequency domain to improve detection performance. For example, some traditional frequency-domain filtering techniques in signal processing can be devised into learnable modules. On the other hand, it is also a research direction that utilizing frequency processing fully exploits the prior characteristic information of infrared targets. Current datasets have a small capacity and are not well organized. There are actually some significant differences in target types, target numbers, imaging scenes, image resolutions and motion states across different tasks. This places high demands on the robustness of detection models. To improve model adaptability to diverse data, future studies include constructing large-scale specialized data clusters and exploring common feature extraction of infrared small target images. With unified feature extraction and specialized decoding, this approach will drive algorithms to solve practical downstream tasks.

6. Conclusions

This paper proposes a frequency-aware masked-attention network (FM-Net) for IRSTD. Our FM-Net distinguishes semantic differences between targets and background well, utilizing the gain of frequency information. We design a wavelet residual block in backbone, which extracts detailed semantic features supplemented with multi-scale spatial–frequency information through forming a multi-level wavelet pyramid inside. Meanwhile, we introduce a frequency-modulation masked-attention module as the long-range connection between encoder and decoder. The module adopts spatial masked-attention to model global background and channel-wise frequency modulation to selectively enhance informative components while suppressing interference ones. On the whole, FM-Net incorporates local and global frequency information into spatial features, making it do well in getting rid of clutters and reducing false alarms. Comprehensive experiments on three benchmark datasets demonstrate the superiority of our FM-Net. However, there are some aspects of FM-Net that need to be optimized, including computational efficiency and physical interpretation. In the future, we will further explore the basic principles of frequency-domain features to reliably improve detection ability. Furthermore, we want to achieve a wavelet transform with learnable parameters, which can adaptively extract semantic details.