TSFANet: Trans-Mamba Hybrid Network with Semantic Feature Alignment for Remote Sensing Salient Object Detection

Abstract

Recent advances in deep learning have witnessed the wide application of convolutional neural networks (CNNs), Transformer models, and Mamba models in optical remote sensing image (ORSI) analysis, particularly for salient object detection (SOD) tasks in disaster warning, urban planning, and military surveillance. Although existing methods improve detection accuracy by optimizing feature extraction and attention mechanisms, they still face limitations when dealing with the inherent challenges of ORSI. These challenges mainly manifest as complex backgrounds, extreme scale variations, and topological irregularities, which severely affect detection performance. However, the deeper underlying issue lies in how to effectively align and integrate local detail features with global semantic information. To tackle these issues, we propose the Trans-Mamba Hybrid Network with Semantic Feature Alignment (TSFANet), a novel architecture that exploits intrinsic correlations between semantic information and detail features. Our network comprises three key components: (1) a Trans-Mamba Semantic-Detail Dual-Stream Collaborative Module (TSDSM) that combines CNNs-Transformer and CNNs-Mamba in a hybrid dual-branch encoder to capture both global context and multi-scale local features; (2) an Adaptive Semantic Correlation Refinement Module (ASCRM) that leverages semantic-detail feature correlations for guided feature optimization; and 3) a Semantic-Guided Adjacent Feature Fusion Module (SGAFF) that aligns and refines multi-scale semantic features. Extensive experiments on three public RSI-SOD datasets demonstrate that our method consistently outperforms 30 state-of-the-art approaches, effectively accomplishing the task of salient object detection in remote sensing imagery.

1. Introduction

Remote sensing image salient object detection (RSI-SOD) aims to identify and extract visually prominent regions and objects from complex remote sensing scenes [1,2,3,4]. By analyzing image features such as texture, edges, and color patterns, SOD effectively suppresses background interference while highlighting objects of interest. With the advancement of Transformer architectures [5], RSI-SOD has demonstrated remarkable success in various applications, including terrain change detection [6], disaster early warning [7], and military object surveillance [8]. Different from natural scene images (NSI) captured by conventional cameras, RSIs are acquired by aerial platforms and satellites from high altitudes, providing broader coverage and higher resolution. However, this unique imaging perspective introduces several critical challenges:

(1) Complex Backgrounds: As illustrated in Figure 1 (first row), conventional NSI-SOD models struggle with RSI-SOD detection, often failing to differentiate between background and foreground elements, leading to numerous false positives; (2) Extreme Scale Variations: The second and third rows of Figure 1 demonstrate how the NSI-SOD model BASNet [9], while effective for large-scale salient region object extraction, significantly underperforms in detecting small-scale salient objects; (3) Irregular Topological Structures: The fourth and fifth rows of Figure 1 reveal that existing models can only roughly extract salient regions from objects with complex topologies and intricate edge details, resulting in significant loss of fine-grained features.

Recent approaches to address these challenges can be categorized into CNNs-based, Transformer-based, Mamba-based, and hybrid architectures. The CNNs-based R3Net [10] (Figure 1e) focuses on local feature learning but struggles with global context integration. The Transformer-based GeleNet [11] (Figure 1f) excels in capturing contextual relationships but lacks precision in edge detection. The Mamba-based VMamba [12] (Figure 1c) shows improved detail extraction through sequential modeling but lacks comprehensive scene understanding. While the hybrid CNNs-Transformer model HFANet [13] (Figure 1g) combines local and global feature processing, it still suffers from false detections. The feature visualization analysis in Figure 2 reveals that current RSI-SOD models, while showing improved object region focus compared to NSI-SOD approaches, still struggle with background suppression, inconsistent multi-scale target localization, and boundary preservation. For example, the suppression of interfering backgrounds such as the runway target region in the lower right corner of the first row and the red building above the second row is still insufficient, the positioning of the large-scale building region and the small-scale car region in the third row is incorrect, and the extraction of the boundary integrity of the road region in the fourth row is insufficient.

To address these limitations, we propose the Trans-Mamba Hybrid Network with Semantic Feature Alignment (TSFANet) to align and integrate local detail features with global semantic representations. By combining the strengths of Transformer-based global modeling and Mamba-based local detail extraction, our TSFANet is able to capture richer and more robust feature representations, leading to more accurate and reliable saliency prediction. This hybrid design is motivated by the complementary characteristics of both architectures, where global semantics help distinguish salient objects from cluttered backgrounds and local details refine object boundaries and small-scale structures. The primary contributions of this work are summarized as follows:

• To achieve comprehensive feature extraction, we designed the Trans-Mamba Semantic-Detail Dual-Stream Collaborative Module (TSDSM), a novel dual-stream architecture that synergistically combines CNNs-Transformer and CNNs-Mamba branches. This hybrid structure effectively leverages Transformer’s global modeling capabilities and Mamba’s local processing advantages, enabling more accurate salient detection in complex remote sensing scenes.
• For effective alignment of local details and global semantic features, we constructed the Adaptive Semantic Correlation Refinement Module (ASCRM). This module models the correlation between semantic and local features, utilizing matrix reshaping and SoftMax activation to accurately capture the spatial information of significant regions, thereby enhancing the precision of optical remote sensing salient object detection.
• To better integrate semantic features with local details, we designed the Semantic-Guided Adjacent Feature Fusion Module (SGAFF). This module extracts the overall semantic framework using a global attention mechanism and fuses it with local features to enhance the semantic information. Through the global semantic fluid information, SGAFF effectively filters out background noise, highlights target details, and improves object detection accuracy.
• Extensive experiments on three public RSI-SOD datasets demonstrate that our method consistently outperforms 26 state-of-the-art approaches. Detailed ablation studies verify that our proposed modules effectively address the challenges of complex backgrounds, scale variations, and irregular topologies in RSI-SOD tasks.

The article is structured as follows. Section 2 provides an overview of the related works. Section 3 introduces the proposed TSFANet framework and its associated components. The experimental results and discussion analysis are presented in Section 4. Lastly, the conclusion is drawn in Section 5.

2. Related Works

2.1. State Space Models

Mamba has emerged as the most advanced state space model (SSM) variant, achieving Transformer-comparable modeling capabilities through its hardware-aware design while maintaining superior efficiency for long sequence processing. Recent developments in Mamba-based architectures can be categorized into several key research directions: Architectural Innovations: Early works focused on enhancing Mamba’s basic architecture. Matten [14] pioneered bidirectional processing with spatiotemporal attention, while MambaVision [15] introduced a restructured architecture that effectively integrates with Transformer mechanisms. nnMamba [16] advanced the field by incorporating Channel-Spatial Siamese learning with CNN integration, significantly improving long-range modeling capabilities. DualMamba [17] proposed a hybrid approach combining cross-attention modules for global modeling with residual learning for local feature extraction. Vision-Specific Adaptations: Several works have focused on adapting Mamba for vision tasks. VMamba [12] introduced the SS2D module for 2D selective scanning, enhancing contextual information collection. Weak-Mamba-UNet [18] developed a comprehensive framework combining CNN, Transformer, and Mamba-based encoders for multi-level feature processing. LocalMamba [19] proposed a dynamic, layer-wise scanning strategy for localized feature capture while maintaining global context. VIM [20] enhanced visual representation through position-aware bidirectional modeling. Task-Specific Applications: Recent research has expanded Mamba’s applications across various domains. Graph-Mamba [21] enhanced graph network modeling through input-dependent node selection. VMRNN [22] and SpikeMba [23] focused on temporal dynamics and neural information processing. CU-Mamba [24] and VMambaMorph [25] addressed medical imaging challenges through specialized architectures. MedMamba [26] developed multi-modal medical image analysis capabilities, while Pan-Mamba [27] and Vmambair [28] focused on cross-modal interaction and image restoration, respectively. Despite these advances in Mamba’s applications across various computer vision tasks, its potential in remote sensing image analysis remains largely unexplored, particularly in the context of salient object detection. Most existing Mamba-based methods for remote sensing images focus solely on semantic segmentation tasks, leaving a significant gap in RSI-SOD applications.

2.2. SOD in Natural Scene Images

Traditional Natural Scene Image Salient Object Detection (NSI-SOD) primarily relied on classical machine learning algorithms, leveraging handcrafted features such as regularization techniques [4], color variation [29], and background priors [30]. While these methods achieved initial progress, they were limited by their dependence on manually designed features and inability to extract deep semantic information. The advent of deep learning brought revolutionary advances to saliency detection. Itti et al. [1] pioneered computational visual attention with a center-surround disparity mechanism, efficiently integrating multi-scale features for salient region identification. Subsequently, CNNs-based approaches demonstrated remarkable improvements. R3Net [10] enhanced detection through recurrent networks and residual learning, alternating between low- and high-level features. AFNet [31] improved boundary accuracy by combining boundary perception with attention mechanisms, while MDF [32] strengthened feature representation through multi-level feature cascading. Amulet [33] introduced effective feature fusion strategies, and MSIN [34] enhanced feature extraction through multi-scale interaction. Edge information processing saw significant diversification. C2SNet [35] utilized boundary information as auxiliary signals, while BASNet [9] improved performance in challenging scenarios through boundary-aware mechanisms. EGNet [36] and ITSD [37] enhanced boundary prediction through edge guidance and two-stream encoding. TRACER [38] optimized feature selection through extreme attention, and PiCANet [39] employed contextual attention for dynamic feature fusion. Recent advances include PFANet’s [40] pyramid feature extraction, BBRFNet’s [41] multi-scale receptive fields, and VST’s [5] Transformer-based long-range dependency modeling. Despite these advances in NSI-SOD, optical remote sensing image saliency detection faces unique challenges due to imaging limitations, including resolution variations, illumination changes, and atmospheric interference. Direct application of existing models to remote sensing images remains problematic, highlighting the need for specialized approaches that effectively integrate multi-scale global and local features while addressing domain-specific challenges.

2.3. SOD in Optical RSIs

RSI-SOD addresses more complex remote sensing scenes compared to NSI-SOD, leading to significant methodological advances organized in four main development stages: (1) The foundational stage focused on establishing datasets and basic frameworks. Zhao et al. [42] introduced a sparsity-guided SOD method with initial RSI datasets. Li et al. [43] developed the Optical Remote Sensing for Salient Object Detection Dataset (ORSSD) and LVNet, combining nested networks with pyramid structures. Zhang et al. [44] extended this with the higher-resolution Extended Optical Remote Sensing for Salient Object Detection Dataset (EORSSD) and proposed DAFNet featuring dense attention mechanisms. Tu et al. [45] contributed the challenging Optical Remote Sensing Image Dataset with 4199 Samples (ORSI-4199) and MJRBM model. (2) CNNs-based methods then emerged to address RSI-specific challenges. EMFINet [46] combined edge awareness with multi-scale feature pyramids, while MCCNet [47] tackled complex backgrounds through multi-content completion. SARNet [48] introduced coarse-to-fine detection with semantic guidance, and ACCoNet [49] leveraged neighborhood context coordination. ICON [50] focused on boundary preservation and feature diversity enhancement. (3) Recent advances have centered on Transformer-based and hybrid architectures. GeleNet [11], the pioneering Transformer-based method, introduced direction-aware spatial attention. Subsequent works like RAGRNet [51], BSCGNet [52], and ESGNet [53] enhanced feature interaction through various graph-based and attention mechanisms. Ma et al. [54] proposed an end-to-end framework that integrates superpixel generation and region merging for remote sensing image segmentation. ASTTNet [55] and IDELNet [56] further refined Transformer-based feature extraction. (4) The latest hybrid approaches have shown promising results. HFANet [13] combined CNN and Transformer encoders for multi-scale modeling. ASNet [57] and HFCNet [58] focused on balancing global-local feature integration. PROFILE [59] and WeightNet [60] introduced sophisticated feature enhancement and noise suppression mechanisms. Ma et al. [61] introduced a deep superpixel-wise segmentation approach for remote sensing image, which leverages task-specific superpixel sampling and soft graph convolution to enhance both accuracy and computational time. FPS-U²Net [62] focuses on gradually extracting salient information from the image by aggregating multi-scale and multi-level features extracted from different stage encoders. Despite these advances, achieving optimal accuracy in complex scenarios remains challenging, particularly for regions with intricate backgrounds, scale variations, and irregular topologies. Our proposed TSFANet addresses these limitations by introducing a novel dual-encoder architecture combining CNNs-Mamba for local features and CNNs-Transformer for global representation, unified by correlation-guided multi-scale feature fusion.

3. Methodology

This section describes the proposed TSFANet. Firstly, we introduce the overall structure. Then, the details of TSDSM, ASCRM, and SGAFF are described respectively. Finally, the loss function for network training is introduced.

3.1. Overall Structure

The overall architecture of TSFANet is illustrated in Figure 3. Built upon the foundational structure of HFANet [13], TSFANet enhances the classic encoder-decoder framework to improve feature extraction and integration. The model employs a dual encoder architecture comprising CNNs-Transformer and CNNs-Mamba components. Specifically, the CNNs-Transformer module captures complex global correlations across spatial positions, while the CNNs-Mamba module integrates these global correlations with local feature details, thereby enhancing the effectiveness of feature extraction.

At each encoding stage, the ASCRM module leverages global features to guide attention shifts within the local feature maps, optimizing them to emphasize detailed characteristics of salient target regions. Additionally, to further enrich semantic content and support robust feature interaction and fusion, the SGAFF module is integrated. This module provides semantic cues through deep-layer features and detailed structural information through shallow-layer features, facilitating accurate small-target recognition and effective background suppression. Algorithm 1 details the implementation process of the method. For an input image $\mathbf{I}\in {\mathbb{R}}^{3\times 448\times 448}$, the TSDSM encoding process operates as follows: the CNNs-Transformer branch extracts global feature maps ${\mathbf{G}}_1\in {\mathbb{R}}^{64\times 224\times 224}$, ${\mathbf{G}}_2\in {\mathbb{R}}^{128\times 112\times 112}$, ${\mathbf{G}}_3\in {\mathbb{R}}^{256\times 56\times 56}$, ${\mathbf{G}}_4\in {\mathbb{R}}^{320\times 28\times 28}$, and ${\mathbf{G}}_5\in {\mathbb{R}}^{512\times 14\times 14}$. Concurrently, the CNNs-Mamba branch captures local features ${\mathbf{L}}_1\in {\mathbb{R}}^{64\times 224\times 224}$, ${\mathbf{L}}_2\in {\mathbb{R}}^{128\times 112\times 112}$, ${\mathbf{L}}_3\in {\mathbb{R}}^{256\times 56\times 56}$, ${\mathbf{L}}_4\in {\mathbb{R}}^{320\times 28\times 28}$, and ${\mathbf{L}}_5\in {\mathbb{R}}^{512\times 14\times 14}$. For each global feature ${\mathit{G}}_i$ and local feature ${\mathit{L}}_i$ ($i\in 1,2,3,4,5$) at different stages, the ASCRM module is utilized to obtain fused features ${\mathit{F}}_i$. These fused features are then processed through the GF-ASPP and S6 modules, resulting in selectively enhanced detailed features ${\mathbf{D}}_5\in {\mathbb{R}}^{1024\times 7\times 7}$. Using the SGAFF module, the fused features ${\mathit{F}}_i$ ($i\in \left\{1,2,3,4,5\right\}$) and deep features ${\mathit{D}}_i$ ($i\in \left\{1,2,3,4,5\right\}$) are combined to produce deep fused features ${\mathit{D}}_{i-1}$ ($i\in \left\{1,2,3,4,5\right\}$) for each stage. Finally, the deep fused features ${\mathit{D}}_{i-1}$ are fed into convolutional module to generate the outputs ${\mathit{O}}_i$ ($i\in 1,2,3,4,5$).

Algorithm 1: Training Framework for TSFANet

3.2. Trans-Mamba Semantic-Detail Dual-Stream Collaborative

To capture the detailed features of target regions in remote sensing images (RSI), existing methods commonly integrate CNNs and Transformers to construct encoders for feature extraction. Typically, CNNs are employed for pixel-level local feature information modeling, followed by Transformers for pixel-block-level global feature information modeling. However, this encoder architecture fails to effectively leverage both local and global features simultaneously. Inspired by Ref. [17], we propose the Trans-Mamba Semantic-Detail Dual-Stream Collaborative Module (TSDSM), which integrates CNNs-Transformer and CNNs-Mamba to form a dual-path encoder. The CNNs-Transformer branch is for global feature modeling, capturing global relationships and long-range dependencies within the image while guiding the CNNs-Mamba branch in local feature modeling. Although the two branches operate independently in feature extraction, their collaboration is achieved through the Adaptive Semantic Correlation Refinement Module (ASCRM), which is applied after each encoding stage (as illustrated in Figure 3). The ASCRM module explicitly models the intrinsic correlations between the global semantic features from the CNNs-Transformer branch and the local detail features from the CNNs-Mamba branch by learning a cross-attention weight matrix. This mechanism enables information exchange and mutual guidance between the branches, allowing global semantic cues to refine local features and vice versa. The resulting fused features integrate both holistic context and local structure, forming a unified representation for subsequent decoding and prediction.

3.2.1. CNNs-Transformer

As illustrated in Figure 3, the Res encoder is utilized to extract low-level features of RSI targets, while the Transformer encoder acquires global semantic information. The Res encoder is defined as follows

$$\mathcal{R}\left(x\right)={\mathrm{Conv}}_{1\times 1}\left({\mathrm{Conv}}_{3\times 3}\left({\mathrm{Conv}}_{1\times 1}\left(x\right)\right)\right)\oplus x$$

(1)

where ${\mathrm{Conv}}_{1\times 1}$ denotes a convolution operation with a kernel size of $1\times 1$, and ⊕ represents element-wise addition. The Transformer captures global attention by constructing query (Q), key (K), and value (V) matrices from input feature image patches. However, due to its computational complexity increasing with image resolution, the Transformer is employed only in the 3rd and 4th stages. As shown in Figure 3, the Transformer encoder partitions the input image into patches through patch embedding. After normalization, it computes the Q, K, and V matrices for these patches. The multi-head attention is then applied to assign attention weights to each image patch. Finally, the feed-forward layer and convolutional layers are integrated to further extract and aggregate features, producing a sequence of token features. The computations processes are defined as follows:

$$T\left(x\right)=\mathrm{FC}\left(\mathrm{Conv}\mathrm{DW}\left(\mathrm{FC}\left(\mathrm{LN}\left(G_{\mathrm{MH}}\right)\right)\right)\right)\oplus G_{\mathrm{MH}}$$

(2)

$$G_{\mathrm{MH}}=A_{\mathrm{MH}}\left(\mathrm{LN}\left(\mathrm{PE}\right(x\left)\right)\right)\oplus x$$

(3)

where FC denotes a fully connected feed-forward layer, $\mathrm{Conv}\mathrm{DW}$ represents a depth-wise convolution with a kernel size of $3\times 3$, LN represents layer normalization, PE indicates patch embedding operations, $G_{\mathrm{MH}}$ represents the multi-head attention feature map, and $A_{\mathrm{MH}}$ is the multi-head attention operation. The multi-head attention is calculated as:

$$A_{\mathrm{MH}}(Q,K,V)=\mathrm{SoftMax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}\right)V$$

(4)

where $\mathrm{SoftMax}\left(·\right)$ is the activation function, ⊤ denotes matrix transposition, and d is the dimensionality of input features divided by the number of multi-head attention.

3.2.2. CNNs-Mamba

As depicted in Figure 3, the CNNs-Mamba branch employs ResBlock to capture low-level features such as edges, textures, and corners of the target, thereby reducing the complexity burden on the Mamba coding module. The Mamba encoder is responsible for synthesizing the local and global information captured by the model in deeper layers. Global information guides the construction of long-range dependencies between different local features, refining the local features in key regions. As shown in Figure 3, the Mamba encoder utilizes the State Space Module (SSM) in conjunction with the Omnidirectional Sensing Module (ODPM) to perform selective scanning in various directions, achieving linear complexity and a globally effective receptive field while extracting local details by integrating global semantic information. For the input features $F\in {\mathbb{R}}^{H\times W\times C}$, after layer normalization and dimensionality reduction via a $1\times 1$ convolution, spatial features of each input channel are extracted through deep convolution. Subsequently, the ODPM selectively scans in different directions to extract distinct detail features, which are adjusted using the normalized features. To prevent gradient loss during model operations, the features are added back to the input features after dimensionality restoration via $1\times 1$ convolution, supplementing the foreground information overlooked in local feature extraction.

$$\begin{array}{cc}\hfill A_i& =\mathrm{ODPM}\left(\mathrm{DConv}\left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{LN}\left(F_i\right)\right)\right)\right)\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill L_i& ={\mathrm{Conv}}_{1\times 1}({\mathrm{Conv}}_{1\times 1}\left(A_i\right)\times \mathrm{LN}\left(F_i\right)+\mathrm{LN}\left(F_i\right))\hfill \end{array}$$

(6)

In contrast to VMamba’s 2D selective scanning, which performs forward and backward scanning in horizontal and vertical directions, the proposed ODPM introduces additional diagonal and anti-diagonal direction scanning to address the anisotropy inherent in remote sensing images. It conducts global modeling in all directions and processes each scanning sequence independently through the SSM module. This approach selectively propagates key features while discarding background noise, enabling global modeling in specific directions. Finally, all sequences are aggregated and summed, capturing detailed features from different directions within the remote sensing image. For the input feature map $I\in {\mathbb{R}}^{h\times w}$, where $I[m,n]$ denotes the token located at the m-th row and n-th column of the feature map, the horizontal scanning operations used by VMamba are defined as follows

$$s_1\left[x\right]=I[xmodw]\left[\lfloor x/w\rfloor \right]$$

(7)

$$s_2\left[x\right]=I[(N-x-1)modw]\left[\lfloor (N-x-1)/w\rfloor \right]$$

(8)

where $N=h\times w$, x represents the position index of an element in the flattened feature map, ranging from 0 to $N-1$, and $s_1,s_2\in {\mathbb{R}}^{N\times C}$ denote the one-dimensional token sequences obtained by flattening along the horizontal scanning direction. The vertical scanning operations are defined as

$$s_3\left[x\right]=I\left[\lfloor x/h\rfloor \right][xmodh],(0\le x<N)$$

(9)

$$s_4\left[x\right]=I\left[\lfloor (N-x-1)/h\rfloor \right][(N-x-1)modh]$$

(10)

where $s_3,s_4\in {\mathbb{R}}^{N\times C}$ represent the one-dimensional token sequences obtained by flattening along the vertical scanning direction. The diagonal scanning operations introduced by ODPM are computed as follows:

$$s_5\left[x\right]=I[xmodw][xmodh],(0\le x<N)$$

(11)

$$s_6\left[x\right]=I\left[((N-x-1)modw)\right][(N-x-1)modh]$$

(12)

where $s_5,s_6\in {\mathbb{R}}^{N\times C}$ denote the one-dimensional token sequences obtained by flattening along the diagonal scanning direction. The anti-diagonal scanning operations are as

$$s_7\left[x\right]=I\left[((N-x-1)modw)\right][xmodh]$$

(13)

$$s_8\left[x\right]=I[xmodw][(N-x-1)modh]$$

(14)

where $s_7,s_8\in {\mathbb{R}}^{N\times C}$ represent the one-dimensional token sequences obtained by flattening along the anti-diagonal scanning direction. After completing the scanning in horizontal, vertical, diagonal, and anti-diagonal directions, the one-dimensional token sequences obtained from forward and backward scanning in each direction are input into the SSM (S6) block to learn attention features for significant targets in each direction. Subsequently, the corresponding inverse operations are applied to each direction

$$I_k^{\prime }\left[m\right]\left[n\right]=\left\{\begin{array}{cc}{s}_k^{\prime }[n\times w+m],\hfill & k=1,7\hfill \\ s_k^{\prime }[N-1-n\times w-m],\hfill & k=2,8\hfill \\ s_k^{\prime }[m\times h+n],\hfill & k=3,5\hfill \\ s_k^{\prime }[N-1-m\times h-n],\hfill & k=4,6\hfill \end{array}\right.$$

(15)

where $k=1,\dots ,8$ represents the eight scanning directions. The final ODPM output is obtained by summing the projections from all directions as $I^{\prime }={\sum }_{k=1}^8{I}_k^{\prime }$.

3.3. Adaptive Semantic Correlation Refinement

The CNNs-Transformer and CNNs-Mamba encoders extract feature information with dimensional discrepancies. While many existing methods address this issue through feature fusion, they often overlook the intrinsic correlations within the features extracted by different encoders. Inspired by Ref. [57], we recognize that such intrinsic correlations exist within the feature information extracted by different encoders. To refine local feature details by leveraging the correlation between semantic and detailed features, we propose the Adaptive Semantic Correlation Refinement Module (ASCRM).

As illustrated in Figure 4, for the input semantic feature $G_i\in {\mathbb{R}}^{C\times H\times W}$ and local feature $L_i\in {\mathbb{R}}^{C\times H\times W}$, matrix reshaping is first applied to obtain $G_i^{\prime }\in {\mathbb{R}}^{C\times HW}$ and $L_i^{\prime }\in {\mathbb{R}}^{C\times \left(HW\right)}$. By learning a weight matrix $W_C\in {\mathbb{R}}^{C\times C}$, the reshaped semantic features are projected into the semantic correlation matrix $A_r\in {\mathbb{R}}^{C\times H\times W}$. The matrix multiplication is then utilized to compute the correlation matrix r, representing the correlation between semantic and local features. The correlation matrix r is calculated as follows:

$$r=A_r\otimes L_i={\left(G_i^{\prime }\right)}^{\top }\otimes W_C\otimes L_i^{\prime }$$

(16)

where ⊤ denotes the transpose operation and ⊗ represents matrix multiplication. The correlation matrix r is subsequently processed using the softmax function for activation and normalization. This processing enables the weighted semantic correlation matrix to capture the spatial location information of significant regions within each feature. The corrected semantic feature $G_i^{\mathrm{corr}}\in {\mathbb{R}}^{C\times H\times W}$ and local feature $L_i^{\mathrm{corr}}\in {\mathbb{R}}^{C\times H\times W}$ are defined as follows:

$$G_i^{\mathrm{corr}}=\mathrm{Reshape}(\mathrm{Reshape}\left(G_i\right)\otimes \mathrm{SoftMax}\left(r\right))$$

(17)

$$L_i^{\mathrm{corr}}=\mathrm{Reshape}(\mathrm{Reshape}\left(L_i\right)\otimes \mathrm{SoftMax}\left(r\right))$$

(18)

where $\mathrm{Reshape}\left(·\right)$ denotes the matrix reshaping operation, and $\mathrm{SoftMax}\left(·\right)$ represents the activation function. Subsequently, location information is generated for $G_i^{\mathrm{corr}}$ and $L_i^{\mathrm{corr}}$ through a $1\times 1$ convolution, resulting in $G_i^{\mathrm{map}}\in {[0,1]}^{1\times H\times W}$ and $L_i^{\mathrm{map}}\in {[0,1]}^{1\times H\times W}$, respectively. Based on these mapped values, redundant information is filtered out to obtain $G_i^{\mathrm{gate}}$ and $L_i^{\mathrm{gate}}$, calculated as follows:

$$G_i^{\mathrm{gate}}=G_i^{\mathrm{map}}\otimes G_i^{\mathrm{corr}}=\mathrm{Sigmoid}\left(\mathrm{Conv}\left(G_i^{\mathrm{corr}}\right)\right)\otimes G_i^{\mathrm{corr}}$$

(19)

$$L_i^{\mathrm{gate}}=L_i^{\mathrm{map}}\otimes L_i^{\mathrm{corr}}=\mathrm{Sigmoid}\left(\mathrm{Conv}\left(L_i^{\mathrm{corr}}\right)\right)\otimes L_i^{\mathrm{corr}}$$

(20)

where $\mathrm{Sigmoid}\left(·\right)$ denotes the sigmoid activation function, which maps feature values to the range $[0,1]$, and Conv represents the $1\times 1$ convolution operation. Through residual connections, $G_i^{\mathrm{gate}}$ is fused with $G_i$, and $L_i^{\mathrm{gate}}$ is fused with $L_i$, resulting in ${\widehat{G}}_i^{\mathrm{corr}}$ and ${\widehat{L}}_i^{\mathrm{corr}}$:

$${\widehat{G}}_i^{\mathrm{corr}}={\mathrm{Conv}}_{3\times 3}\left(G_i^{\mathrm{gate}}\oplus G_i\right)$$

(21)

$${\widehat{L}}_i^{\mathrm{corr}}={\mathrm{Conv}}_{3\times 3}\left(L_i^{\mathrm{gate}}\oplus L_i\right)$$

(22)

where ${\mathrm{Conv}}_{3\times 3}$ denotes the $3\times 3$ depthwise separable convolution, and ⊕ represents element-wise addition. For the two effective components, ${\widehat{G}}_i^{\mathrm{corr}}$ and ${\widehat{L}}_i^{\mathrm{corr}}$, concatenation is performed followed by $3\times 3$ convolution to extract the location features of their significant regions. These features are then transformed into position information $P_i$ using the $1\times 1$ convolution and the sigmoid function:

$$P_i=\mathrm{Sigmoid}\left({\mathrm{Conv}}_{1\times 1}\left({\mathrm{Conv}}_{3\times 3}\left({\widehat{G}}_i^{\mathrm{corr}}\Theta {\widehat{L}}_i^{\mathrm{corr}}\right)\right)\right)$$

(23)

where $\Theta$ denotes channel-wise concatenation. By applying the extracted position information $P_i$ to the local feature $L_i$, the refined local feature $F_i$ is obtained as follows:

$$F_i=P_i\otimes L_i\oplus L_i$$

(24)

This direct feature modulation approach provides precise collaborative semantic guidance for refining local features, thereby laying a solid foundation for the fine-grained processing of remote sensing images.

3.4. Semantic-Guided Adjacent Feature Fusion

The remote sensing image contains rich semantic information that can assist the model in accurately locating target positions. When combined with local features, this semantic information can better identify targets of varying sizes within complex backgrounds. Although many existing methods effectively learn feature information at different scales, significant semantic differences persist across layer features. Inspired by Ref. [48], we constructed the Semantic-Guided Adjacent Feature Fusion Module (SGAFF) to align and fuse features by effectively utilizing deep semantic features and details across scales.

As illustrated in Figure 5, for the deep input feature $D_i\in {\mathbb{R}}^{C\times H\times W}$ from the encoder, it is transformed into $D_i^{\prime }\in {\mathbb{R}}^{C\times HW}$ through matrix operations for computational convenience. Subsequently, the Global Attention Mechanism (GAM) is employed to capture the overall semantic framework of the feature $E_i\in {\mathbb{R}}^{HW\times N}$. The overall semantic framework $S_i$ is calculated as follows:

$$E_i=\mathrm{SoftMax}\left(\mathrm{Transpose}\left(\mathrm{GAM}\left(D_i\right)\right)\right)$$

(25)

where $\mathrm{SoftMax}\left(·\right)$ represents the activation function, and $\mathrm{Transpose}\left(·\right)$ denotes the matrix transpose operation. Semantic features aid in understanding the overall context and meaning of the image but are susceptible to noise and complex backgrounds. To mitigate these issues, the semantic framework is fused with the local features $D_i^{\prime }$ extracted from the input features to obtain the global semantic fluid information $S_i\in {\mathbb{R}}^{C\times N}$. This provides detailed information beneficial for distinguishing similar objects in different regions. The $S_i$ is calculated as:

$$S_i={\mathrm{Conv}}_{1\times 1}\left(D_i^{\prime }\right)\oplus \left({\mathrm{Conv}}_{1\times 1}\left(D_i^{\prime }\right)\otimes E_i\right)$$

(26)

where ${\mathrm{Conv}}_{1\times 1}$ represents the $1\times 1$ convolution, ⊕ denotes element-wise addition, and ⊗ represents matrix multiplication. To facilitate the fusion of features $F_{i-1}\in {\mathbb{R}}^{(C/2)\times 2H\times 2W}$ with the shallow input features from the encoder, the deep-layer input features $D_i$ are upsampled and reshaped into a feature tensor $D_i^{″}$ with dimensions $N\times 2H\times 2W$:

$$D_i^{″}=\mathrm{Reshape}\left(\mathrm{Up}\left(D_i\right)\right)$$

(27)

where $\mathrm{Reshape}\left(·\right)$ represents the matrix reshaping operation, and $\mathrm{Up}\left(·\right)$ denotes the upsampling. The semantic information $S_i$ is projected to the corresponding positions, aggregated, and integrated into the semantic feature $A_i\in {\mathbb{R}}^{C\times kH\times kW}$:

$$A_i=\mathrm{Reshape}\left(D_i^{″}\oplus \left({\mathrm{Conv}}_{1\times 1}\left(D_i^{″}\right)\otimes S_i\right)\right)$$

(28)

where $\mathrm{Reshape}\left(·\right)$ transforms the feature matrix from $C\times k^{2}HW$ to $C\times kH\times kW$ dimensions to facilitate subsequent computation and processing; ⊗ represents matrix multiplication, and ⊕ denotes element-wise addition. Since different channel dimensions contain varying feature details of the target and include background noise interference, it is essential to highlight the detailed parts of the target. To achieve this, channel dimension features are resampled to obtain attention across different channels, thereby enhancing the shallow input features $F_{i-1}$ through channel attention. To prevent feature information loss while filtering out background noise, semantic features $A_i$ are used to guide and obtain the channel attention projection $C_i\in {\mathbb{R}}^{C\times k^{2}HW}$. Through matrix reshaping, the channel attention feature $C_i^{\prime }\in {\mathbb{R}}^{C\times kH\times kW}$ is obtained:

$$C_i^{\prime }=\mathrm{Reshape}\left(\mathrm{SoftMax}\left(F_{i-1}^{\top }\otimes A_i\right)\otimes F_{i-1}\right)$$

(29)

where $\mathrm{SoftMax}\left(·\right)$ represents the softmax activation function, and ⊗ denotes matrix multiplication. The Sigmoid function is applied to the semantic feature $A_i$ and multiplied with the channel feature $C_i^{\prime }$, leveraging the semantic features to further select the target’s feature information. Then, $A_i$ is element-wise added to the semantic feature and passed through a $1\times 1$ convolution and normalization to obtain the aggregated feature $D_{i-1}\in {\mathbb{R}}^{(C/2)\times 2H\times 2W}$:

$$D_{i-1}=\varphi \left({\mathrm{Conv}}_{1\times 1}\left(C_i^{\prime }\odot \mathrm{Sigmoid}\left(A_i\right)\right)\oplus A_i\right)$$

(30)

where ⊙ represents element-wise matrix multiplication, and $\varphi$ denotes the batch normalization and ReLU operations.

3.5. Loss Function

The proposed TSFANet extends HFANet [13] by employing the designed IG Loss and weighted Intersection over Union (wIoU) as the loss functions. As shown in Figure 3, IG Loss is calculated as follows:

$$\begin{array}{c}\hfill L_{IG}={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left[\left(p_i^E{m}_1+1\right)L_{BCE}\left(p_i^S,g_i^S\right)\right.\left.+\left((1-p_i^S·g_i^E)m_2+1\right)L_{BCE}\left(p_i^E,g_i^E\right)\right]\end{array}$$

(31)

where $p_i^S$ and $p_i^E$ are the predicted saliency map and predicted edge map, respectively; $g_i^S$ and $g_i^E$ are the ground truth saliency map and edge map; i represents the index of each pixel; n is the total number of pixels; $m_1$ and $m_2$ are balancing weight factors (both set to 1 in our implementation, following Ref. [13]) that control the contribution of the edge and region terms in the loss function, and $L_{BCE}\left(\right)$ is the Binary Cross Entropy (BCE) loss. The BCE loss is defined as follows:

$$L_i^{BCE}(x,y)=-ylog\left(x\right)-(1-y)log(1-x)$$

(32)

By introducing the wIoU loss into the IG loss, $L_i^{wIoU}$ is calculated as follows:

$$L_i^{wIoU}=1-{\displaystyle \frac{{\sum }_{j=1}^n\left(p_j^s·g_j^s\right)+1}{{\sum }_{j=1}^n\left(p_j^s+g_j^s-p_j^s·g_j^s\right)+1}}$$

(33)

where $\sum (p_j^s,g_j^s)=p_j^s+g_j^s$ and $\sum (p_j^s·g_j^s)=p_j^s·g_j^s$ represent the sum and element-wise multiplication of the predicted saliency map and ground truth values at pixel j, respectively. The total loss for each stage can be expressed as

$$L_i={\lambda }_1{L}_i^{IG}+{\lambda }_2{L}_i^{wIoU}$$

(34)

$$L_{\mathrm{total}}=L_1+{\sum }_{i=2}^5{\displaystyle \frac{1}{2^{i-2}}}{L}_i$$

(35)

For the model training process, high-quality saliency prediction maps are generated in five stages, and the model is supervised using image labels and edge labels. To better balance the five detection stages of the model, the losses of the five stages are weighted and summed as Equation (35). In addition, we find that the hyperparameters ${\lambda }_1$ and ${\lambda }_2$ are all set to 1 and work best in our experiments.

4. Experiments and Analysis

4.1. Dataset Description

To comprehensively evaluate the performance of the proposed model, we conducted extensive experimental comparisons on three publicly available Remote Sensing Image Saliency Object Detection (RSI-SOD) datasets: ORSSD, EORSSD, and ORSI-4199.

• ORSSD: The first publicly available RSI-SOD dataset, ORSSD, comprises 800 optical RSI images with corresponding annotations. Of these, 600 images are designated for training, and 200 images for testing.
• EORSSD: An extended version of ORSSD, EORSSD incorporates more challenging scenarios to better assess model robustness. It includes 1400 training samples and 600 testing samples.
• ORSI-4199: The most diverse saliency detection dataset in terms of scene complexity, ORSI-4199 contains 2000 training samples and 2199 testing samples.

Following the standard protocol, the training sets of each dataset are utilized for model training, while performance evaluations are conducted on their respective test sets.

4.2. Implementation Details and Evaluation Metrics

We developed TSFANet using the PyTorch 2.0.0 framework and Python 3.8.0. Model inference is accelerated using an NVIDIA RTX 4090 GPU. To prevent overfitting and enhance the generalization performance of the model, we employed various data augmentation techniques, including random rotations, color transformations, and noise addition. For model parameter optimization, we set the initial learning rate to $1\times {10}^{-5}$, the batch size to 16, and used the Adam optimizer to update the network parameters. A polynomial learning rate strategy is applied to automatically adjust the learning rate during training. The model is trained for 300 epochs. In training and testing phases, input images are resized to $448\times 448$ pixels to maintain consistency.

To comprehensively assess the performance of our method, we adopt five widely used evaluation metrics: S-measure ($S_{\alpha }$), F-measure ($F_{\beta }$), E-measure ($E_{\zeta }$), mean absolute error (MAE), and the precision-recall (P-R) curve. The S-measure jointly considers region-aware and object-aware similarities to reflect both global and structural correspondence between the predicted saliency map and the ground truth. The F-measure calculates the weighted harmonic mean of precision and recall, with a stronger emphasis on precision, to evaluate the overall detection quality. The E-measure combines pixel-level precision and image-level structural information for a balanced measurement of saliency map accuracy. The MAE quantifies the average pixel-wise error between the predicted and true saliency maps, providing an intuitive indicator of prediction error. Finally, the P-R curve illustrates the model’s performance under different binarization thresholds by plotting precision versus recall, offering a comprehensive view of detection capability across varying criteria.

4.3. Comparison with State-of-the-Art Methods

To comprehensively evaluate the proposed method, TSFANet is benchmarked against state-of-the-art models on three public Remote Sensing Image Saliency Object Detection (RSI-SOD) datasets. The comparison involves a Mamba-based semantic image segmentation model (SSI) VMamba [12] and Samba [63] (with one class in the experiments), eight Natural Scene Image (NSI) models (PicaNet [39], R3Net [10], RAS [64], BASNet [9], CPDNet [65], EGNet [36], PoolNet [66], U2Net [67], and VST [5]), and sixteen Remote Sensing Image (RSI) models (CSNet [68], CoorNet [69], GeleNet [11], HVPNet [70], SAMNet [71], MCCNet [47], SUCA [72], ACCoNet [49], DNTD [73], MJRBM [45], ICON [50], EMFINet [46], HFANet [13], DPORTNet [74], PA-KRN [75], ERPNet [76], DBINet [77], DCNet [78], and DSINet [79]). To ensure experimental fairness, results for the 30 comparison methods were derived from publicly available data provided by the authors, with some reproduced using publicly available source code.

4.3.1. Quantitative Comparison

The quantitative comparison results of TSFANet and the other 30 latest methods on three RSI-SOD datasets are shown in

Table 1. Quantitative comparisons of our method with 30 state-of-the-art methods on the ORSSD dataset. The Top-3 results in each column are highlighted in red, blue, and green. “-” denotes the authors do not provide the corresponding results.

Methods	Publication	Type	Backbone	ORSSD
				${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
PicaNet	2018 CVPR	C-NSI	ResNet	0.8689	0.7922	0.9005	0.0260
RAS	2018 ECCV	C-NSI	ResNet	0.8829	0.8229	0.9306	0.0169
R3Net	2018 IJCAI	C-NSI	ResNet	0.9009	0.8615	0.9238	0.0170
PoolNet	2019 CVPR	C-NSI	ResNet	0.8098	0.7051	0.8513	0.0469
BASNet	2019 CVPR	C-NSI	U-Net	0.8931	0.8231	0.9206	0.0277
CPDNet	2019 CVPR	C-NSI	ResNet	0.8829	0.8363	0.9133	0.0171
EGNet	2019 ICCV	C-NSI	VGG	0.8725	0.7603	0.8959	0.0217
U2Net	2020 PR	C-NSI	U-Net	0.8716	0.7962	0.9014	0.0222
VIT	2021 ICCV	T-NSI	Transformer	0.9174	0.8613	0.9415	0.0125
CSNet	2020 TIP	C-RSI	ResNet	0.8928	0.8437	0.9181	0.0181
HVPNet	2021 TGRS	C-RSI	ResNet	0.8585	0.7431	0.8601	0.0227
SAMNet	2021 TGRS	C-RSI	ResNet	0.8721	0.7559	0.8690	0.0221
CoorNet	2021 TGRS	C-RSI	ResNet	0.9392	0.9188	0.9746	0.0098
MCCNet	2021 TGRS	C-RSI	VGG	0.9428	0.9241	0.9758	0.0087
SUCA	2021 T-GRS	C-RSI	ResNet	0.8989	0.8398	0.9391	0.0145
ACCoNet	2022 TCYB	C-RSI	VGG	0.9424	0.9157	0.9754	0.0088
DNTD	2022 TGRS	C-RSI	ResNet	0.8696	0.8163	0.9065	0.0218
MJRBM	2022 TGRS	C-RSI	ResNet	0.9194	0.8749	0.9418	0.0163
ICON	2022 TGRS	C-RSI	ResNet	0.9251	0.8851	0.9637	0.0116
EMFINet	2022 TGRS	C-RSI	ResNet	0.9365	0.9038	0.9671	0.0109
DPORTNet	2023 TGRS	C-RSI	ResNet	0.8823	0.8327	0.9119	0.0221
PA-KRN	2023 TGRS	C-RSI	ResNet	0.9231	0.8909	0.9620	0.0139
ERPNet	2023 TGRS	C-RSI	ResNet	0.9247	0.8926	0.9566	0.0135
GeleNet	2023 TIP	T-RSI	PVT	0.9075	0.8619	0.9463	0.0133
HFANet	2022 TGRS	H-RSI	Hybrid	0.9393	0.9165	0.9711	0.0092
VMamba	2024 arXiv	-	MAmba	0.7060	0.5047	0.6684	0.0584
Samba	2024 arXiv	-	MAmba	0.8628	0.8077	0.9060	0.0327
DBINet	2024 GRSL	H-RSI	Hybrid	0.8818	0.8220	0.9297	0.0173
DCNet	2024 JSTARS	H-RSI	Hybrid	0.8821	0.8359	0.9128	0.0177
DSINet	2024 Vis Comput	H-RSI	Hybrid	0.9387	0.9007	0.9709	0.0093
TSFANet	Ours	H-RSI	Hybrid	0.9446	0.9174	0.9817	0.0077

,

Table 2. Quantitative comparisons of our method with 30 state-of-the-art methods on the EORSSD dataset. The Top 3 results in each column are highlighted in red, blue, and green. “-” denotes the authors do not provide the corresponding results.

Methods	Publication	Type	Backbone	EORSSD
				${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
PicaNet	2018 CVPR	C-NSI	ResNet	0.8797	0.7762	0.8902	0.0119
RAS	2018 ECCV	C-NSI	ResNet	0.8847	0.8052	0.9265	0.0115
R3Net	2018 IJCAI	C-NSI	ResNet	0.8974	0.8268	0.9154	0.0124
PoolNet	2019 CVPR	C-NSI	ResNet	0.8556	0.7486	0.8742	0.0169
BASNet	2019 CVPR	C-NSI	U-Net	0.9031	0.8207	0.9266	0.0114
CPDNet	2019 CVPR	C-NSI	ResNet	0.8488	0.7672	0.8794	0.0156
EGNet	2019 ICCV	C-NSI	VGG	0.8605	0.7065	0.8631	0.0109
U2Net	2020 PR	C-NSI	U-Net	0.8619	0.7457	0.8581	0.0167
VIT	2021 ICCV	T-NSI	Transformer	0.9183	0.8407	0.9366	0.0074
CSNet	2020 TIP	C-RSI	ResNet	0.8399	0.7812	0.8860	0.0142
HVPNet	2021 TGRS	C-RSI	ResNet	0.8697	0.7430	0.8552	0.0112
SAMNet	2021 TGRS	C-RSI	ResNet	0.8597	0.7286	0.8543	0.0133
CoorNet	2021 TGRS	C-RSI	ResNet	0.9298	0.8890	0.9646	0.0083
MCCNet	2021 TGRS	C-RSI	VGG	0.9323	0.8874	0.9685	0.0066
SUCA	2021 T-GRS	C-RSI	ResNet	0.8985	0.8168	0.9251	0.0097
ACCoNet	2022 TCYB	C-RSI	VGG	0.9285	0.8823	0.9655	0.0074
DNTD	2022 TGRS	C-RSI	ResNet	0.8954	0.8176	0.9196	0.0114
MJRBM	2022 TGRS	C-RSI	ResNet	0.9200	0.8504	0.9354	0.0099
ICON	2022 TGRS	C-RSI	ResNet	0.9196	0.8632	0.9619	0.0073
EMFINet	2022 TGRS	C-RSI	ResNet	0.9299	0.8751	0.9601	0.0084
DPORTNet	2023 TGRS	C-RSI	ResNet	0.8937	0.7945	0.8907	0.0152
PA-KRN	2023 TGRS	C-RSI	ResNet	0.9186	0.8621	0.9537	0.0104
ERPNet	2023 TGRS	C-RSI	ResNet	0.9201	0.8565	0.9399	0.0089
GeleNet	2023 TIP	T-RSI	PVT	0.8849	0.8138	0.9273	0.0090
HFANet	2022 TGRS	H-RSI	Hybrid	0.9390	0.8951	0.9678	0.0070
VMamba	2024 arXiv	-	MAmba	0.7423	0.5104	0.6991	0.0273
Samba	2024 arXiv	-	MAmba	0.8976	0.8397	0.9416	0.0124
DBINet	2024 GRSL	H-RSI	Hybrid	0.8830	0.8152	0.9258	0.0129
DCNet	2024 JSTARS	H-RSI	Hybrid	0.8482	0.8148	0.8753	0.0161
DSINet	2024 Vis Comput	H-RSI	Hybrid	0.9315	0.8776	0.9571	0.0076
TSFANet	Ours	H-RSI	Hybrid	0.9323	0.8874	0.9685	0.0060

and

Table 3. Quantitative comparisons of our method with 30 state-of-the-art methods on the ORSI-4199 dataset. The Top 3 results in each column are highlighted in red, blue, and green. “-” denotes the authors do not provide the corresponding results.

Methods	Publication	Type	Backbone	ORSI-4199
				${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
PicaNet	2018 CVPR	C-NSI	ResNet	0.8359	0.7711	0.8860	0.0418
RAS	2018 ECCV	C-NSI	ResNet	0.8326	0.7879	0.8962	0.0433
R3Net	2018 IJCAI	C-NSI	ResNet	0.8586	0.8220	0.9116	0.0353
PoolNet	2019 CVPR	C-NSI	ResNet	0.7926	0.7133	0.8491	0.0643
BASNet	2019 CVPR	C-NSI	U-Net	0.8369	0.7862	0.8863	0.0500
CPDNet	2019 CVPR	C-NSI	ResNet	0.8361	0.8035	0.8943	0.0403
EGNet	2019 ICCV	C-NSI	VGG	0.8464	0.7959	0.8943	0.0455
U2Net	2020 PR	C-NSI	U-Net	0.8384	0.7915	0.8970	0.0433
VIT	2021 ICCV	T-NSI	Transformer	0.8764	0.8354	0.9288	0.0307
CSNet	2020 TIP	C-RSI	ResNet	0.8215	0.7540	0.8549	0.0545
HVPNet	2021 TGRS	C-RSI	ResNet	0.8468	0.7932	0.8930	0.0438
SAMNet	2021 TGRS	C-RSI	ResNet	0.8407	0.7920	0.8908	0.0452
CoorNet	2021 TGRS	C-RSI	ResNet	0.8621	0.8421	0.9204	0.0382
MCCNet	2021 TGRS	C-RSI	VGG	0.8747	0.8541	0.9346	0.0332
SUCA	2021 T-GRS	C-RSI	ResNet	0.8795	0.8500	0.9352	0.0320
ACCoNet	2022 TCYB	C-RSI	VGG	0.8775	0.8531	0.9341	0.0330
DNTD	2022 TGRS	C-RSI	ResNet	0.8446	0.8113	0.9041	0.0441
MJRBM	2022 TGRS	C-RSI	ResNet	0.8593	0.8226	0.9102	0.0390
ICON	2022 TGRS	C-RSI	ResNet	0.8753	0.8577	0.9435	0.0299
EMFINet	2022 TGRS	C-RSI	ResNet	0.8674	0.8390	0.9254	0.0346
DPORTNet	2023 TGRS	C-RSI	ResNet	0.7475	0.7609	0.8676	0.0586
PA-KRN	2023 TGRS	C-RSI	ResNet	0.8492	0.8237	0.9164	0.0399
ERPNet	2023 TGRS	C-RSI	ResNet	0.8673	0.8286	0.9147	0.0373
GeleNet	2023 TIP	T-RSI	PVT	0.7665	0.6887	0.8216	0.0711
HFANet	2022 TGRS	H-RSI	Hybrid	0.8766	0.8534	0.9334	0.0330
VMamba	2024 arXiv	-	MAmba	0.7228	0.5853	0.7448	0.0829
Samba	2024 arXiv	-	MAmba	0.8467	0.8211	0.9132	0.0446
DBINet	2024 GRSL	H-RSI	Hybrid	0.8318	0.7870	0.8952	0.0441
DCNet	2024 JSTARS	H-RSI	Hybrid	0.8358	0.8032	0.8939	0.0410
DSINet	2024 Vis Comput	H-RSI	Hybrid	0.8644	0.8554	0.9284	0.0297
TSFANet	Ours	H-RSI	Hybrid	0.8876	0.8617	0.9453	0.0277

. Clearly, our method outperforms these advanced methods across each dataset. Specifically, for the simpler ORSSD dataset, TSFANet achieves three optimal results and one third-best result. Compared with the top-performing traditional CNN-based NSI-SOD model R3Net, the $S_{\alpha }$, $F_{\beta }$, and $E_{\xi }$ scores are increased by 4.37%, 5.59%, and 5.79%, respectively, while the MAE is reduced by 0.93%. Although it trails behind the leading RSI-SOD model MCCNet by 0.67% on the $F_{\beta }$ metric, TSFANet leads by 0.18%, 0.59%, and 0.1% on the $S_{\alpha }$, $E_{\xi }$, and MAE metrics, respectively. On the extended and more challenging EORSSD dataset, TSFANet slightly lags behind HFANet by 0.67% and 0.77% on the $S_{\alpha }$ and $F_{\beta }$ indicators, respectively, but surpasses it by 0.07% and 0.1% on the $E_{\xi }$ and MAE indicators, respectively. On the most challenging ORSI-4199 dataset, TSFANet achieves the best results with an $S_{\alpha }$ of 88.76%, $F_{\beta }$ of 86.17%, $E_{\xi }$ of 94.53%, and MAE of 2.77%. To visually illustrate the model’s overall performance, Figure 6 presents the precision-recall (P-R) curves and F-measure curves of the comparison methods across the three datasets. The comparison of the curves reveals that TSFANet’s P-R curve is closest to the coordinate (1, 1), demonstrating its ability to achieve high precision while maintaining high recall. Additionally, TSFANet achieves the largest area under the F-measure curve compared to all other methods. The computational complexity analysis of the various methods is presented in

Table 4. Computational complexity of different methods. The best results are marked in bold.

Model	Params (M) ↓	Flops (G) ↓	Model	Params (M) ↓	Flops (G) ↓
PicaNet	47.219	59.786	RAS	45.31	60.19
R3Net	71.89	70.99	PoolNet	75.67	80.83
BASNet	46.10	36.57	CPDNet	69.99	75.69
EGNet	85.25	90.91	U2Net	44.01	115.313
VIT	126.62	75.44	–	–	–
CSNet	65.58	70.12	HVPNet	100.42	120.11
SAMNet	95.66	114.96	CoorNet	88.98	104.69
MCCNet	67.65	358.77	SUCA	117.34	171.66
ACCoNet	167.01	177.21	DNTD	94.89	103.79
MJRBM	85.94	90.25	ICON	94.79	89.91
EMFINet	102.45	132.47	DPORTNet	93.73	88.68
PA-KRN	87.62	73.65	ERPNet	72.34	87.55
GeleNet	48.94	25.45	HFANet	320.31	78.46
VMamba	66.80	121.31	Samba	99.16	123.57
DBINet	57.61	47.15	DCNet	101.19	82.09
DSINet	109.89	89.69	TSFANet	334.82	115.24

. It can be observed that the introduction of multiple Transformer modules leads to a large number of parameters in the baseline model HFANet, while the proposed TSFANet improves on it by discarding some of the modules with a large number of parameters, enhancing the capture of feature details and the understanding of semantic information. Although the number of parameters is the largest, the computational complexity is at a medium level, which can guarantee high performance while providing a better price–performance ratio in terms of hardware resources and practical applications. While TSFANet consistently outperforms existing methods overall, there remain certain cases, such as images containing extremely small, low-contrast, or densely clustered salient objects, where the improvement is less significant. These cases reveal that further enhancement is needed to address the detection of subtle and highly challenging targets in complex remote sensing scenes.

4.3.2. Qualitative Comparison

Qualitative evaluations across different RSI scenarios highlight the advantages of our proposed method under complex conditions. The visualization results are presented in Figure 7. Typical scenarios for analysis and comparison include (1) Objects in complex cluttered backgrounds; (2) Small-scale salient objects; (3) Topologically irregular objects. By comparing the prediction results generated by each test image, ground truth labels, and different models, it is evident that our TSFANet more closely aligns with real-world scenarios than other methods. Specifically, the advantages of saliency detection in different scenarios are as follows:

(1) Objects in Complex Cluttered Backgrounds: As shown in Figure 7a–c, human activities introduce abundant clutter information similar to the target in RSI, making it difficult for some models to achieve high-precision detection. For example, in Figure 7a, only EMFINet, SUCA, CorrNet, ACCoNet, ERPNet, MCCNet, and DBSINet effectively filter out the interference from the terminal building, which has a similar color to the target airplane on the right. In Figure 7b, BASNet, R3Net, ENFNet, RAS, EMFINet, EGNet, GeleNet, and CPD learn some non-significant object features, leading to the recognition of the right guiding surface as part of the target. In Figure 7c, due to the excessive similarity between the salient target and background features, most models fail to completely extract the road target, with only Smamba, GeleNet, VIT, HFANet, DBNet, DCNet, and DSINet extracting the road region relatively comprehensively. In contrast, our method filters redundant information by effectively extracting and fusing local features with global semantic information, enhancing edge details after identifying the foreground position.

(2) Small-Scale Salient Objects: As shown in Figure 7d–f, the large imaging range and high spatial resolution of RSI often contain numerous small-scale targets, making it challenging for some models to accurately locate and finely extract these targets. For example, in Figure 7d,e, VMamba, R3Net, and BASNet suffer from severe localization deficiencies, resulting in the loss of six ships and four airplanes. While Samba, PA-KRN, and HFANet achieve more comprehensive salient target extraction than other methods, they still lack edge details and encounter false alarms and misclassifications, such as in the road extraction scenario shown in Figure 7f. Dual coding models DBNet, DCNet and DSINet all show good ability of small-scale target location, but due to different information integration abilities of the models, the determination of target boundary still needs to be improved. The comparison results indicate that our model provides more accurate and detailed extraction of small-scale targets, underscoring the effectiveness of integrating local and global feature information for saliency detection in such scenarios.

(3) Topologically Irregular Objects: As shown in Figure 7g–i, the diversity and complexity of natural terrain in RSI lead to objects like mountains, rivers, and landmark buildings often having irregular topology and complex edge features. In Figure 7g, due to the complexity of the mall structure designed for aesthetics, except for BASNet and DPORTNet, which do not fully capture the entire mall, other models exhibit varying degrees of loss in mall edge details. In Figure 7h, the staircases on different floors of the building lead to severe misjudgments by models such as VMamba, Samba, R3Net, RAS, PoolNet, GeleNet, VIT, SUCA, and DNTD, mistakenly identifying them as part of the main target. In Figure 7i, due to the detailed lake area and background complexity, as well as irregular topology, the detection results of each model are suboptimal. Although EGNet, GeleNet, and DSINet achieve more accurate detections, there are still some false alarms. In comparison, our model excels in addressing topologically irregular targets.

In summary, our method demonstrates excellent detection accuracy in scenarios with cluttered backgrounds, multi-scale targets, and topologically irregular objects. Despite the strong qualitative performance, we observe that TSFANet can still miss very subtle objects or confuse background clutter with salient regions under extreme conditions, such as heavy shadow, severe occlusion, or ambiguous object boundaries. To address these limitations, future research will explore more robust feature fusion strategies and adaptive attention mechanisms tailored to complex and diverse remote sensing environments.

4.4. Ablation Study

As the baseline model HFANet has achieved better results on two datasets, ORSSD and EORSSD, it provides a more adequate basis for comparison to validate the new method. However, ORSI-4199 is a more challenging dataset, especially in dealing with complex backgrounds and extreme scale variations, and Baseline’s performance on this dataset still needs to be greatly improved. Therefore, we used only the ORSI-4199 dataset in our ablation experiments to better assess the performance and improvement of the model when dealing with more difficult tasks. Based on the quantitative results in

Table 5. Ablation experiments on the ORSI-4199 dataset. TSDSM, ASCRM, and SGAFF denote Trans-Mamba Semantic-Detail Dual-Stream Collaborative Encoder Module, Adaptive Semantic Correlation Refinement Attention Module and Semantic-Guided Adjacent Feature Fusion Module, respectively. The best results are marked in bold.

No.	TSDSM	ASCRM	SGAFF	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
0				0.8766	0.8534	0.9334	0.0330
1	✓			0.8792	0.8573	0.9369	0.0313
2		✓		0.8779	0.8550	0.9354	0.0321
3			✓	0.8788	0.8567	0.9362	0.0317
4	✓	✓		0.8858	0.8603	0.9418	0.0287
5	✓		✓	0.8849	0.8596	0.9403	0.0292
6		✓	✓	0.8835	0.8590	0.9388	0.0296
7	✓	✓	✓	0.8876	0.8617	0.9453	0.0277

, the evaluation metrics progressively improve with the sequential incorporation of different modules. Additionally, Figure 8 presents visualization results for the sequential integration of these modules.

4.4.1. Effect of TSDSM

We constructed a hybrid feature encoder structure combining CNN-Transformer and CNN-Mamba architectures to analyze the effectiveness of different encoder combinations.

As shown in

Table 6. Comparative experiments of different encoder structures on ORSI-4199 datasets. The best results are marked in bold.

Encoder	Evaluation Metrics
	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
Baseline	0.8835	0.8590	0.9388	0.0296
+ CNN	0.8839	0.8593	0.9391	0.0294
+ Transformer	0.8853	0.8602	0.9417	0.0283
+ Mamba	0.8847	0.8599	0.9412	0.0287
+ CNN-Transformer	0.8872	0.8614	0.9446	0.0279
+ CNN-Mamba	0.8868	0.8609	0.9440	0.0280
+ TSDSM	0.8876	0.8617	0.9453	0.0277

, various encoder architectures and their pairwise combinations were evaluated, including TSDSM using CNN, Transformer, and Mamba. The TSDSM hybrid coding structure achieved the highest evaluation scores, with $F_{\beta }$ and $E_{\xi }$ reaching 0.8617 and 0.9453, respectively. Figure 9 visualizes the feature maps obtained by each encoder structure. The CNN coding structure alone accurately captures the salient target features but lacks global semantic information, leading to significant attention to background interference. The Transformer encoder effectively captures global information but struggles with detailed feature processing. The Mamba encoder offers better global understanding through state space modeling but is insufficient for the RSI-SOD task. Combining CNNs with Transformer or Mamba reduces background interference but either lacks edge completeness or misclassifies regions. In contrast, the TSDSM structure effectively captures both global contextual and fine-grained local features, leading to superior performance.

4.4.2. Effect of ASCRM

ASCRM is designed to guide the fusion of global semantic features from the CNN-Transformer encoder and local detailed features from the CNN-Mamba encoder by leveraging global-local feature relationships, enabling precise salient region localization and extraction. Ablation experiments were performed by replacing ASCRM in TSFANet with alternative structures numbered No. 1–No. 10 in Figure 10.

As shown in

Table 7. Comparative of different ASCRM ablation modules on ORSI-4199 datasets. The best results are marked in bold.

Encoder	Evaluation Metrics
	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
No.1	0.8849	0.8596	0.9403	0.0292
No.2	0.8849	0.8597	0.9404	0.0291
No.3	0.8850	0.8599	0.9406	0.0290
No.4	0.8851	0.8600	0.9410	0.0288
No.5	0.8855	0.8602	0.9428	0.0286
No.6	0.8864	0.8605	0.9442	0.0283
No.7	0.8866	0.8607	0.9444	0.0282
No.8	0.8873	0.8614	0.9451	0.0279
No.9	0.8871	0.8610	0.9449	0.0280
No.10	0.8876	0.8617	0.9453	0.0277

, the No.10 (ASCRM) structure achieved the highest quantitative evaluation scores. This is attributed to ASCRM’s learnable weight matrix, which effectively establishes relationships between global and local features, guiding their optimization and fusion for accurate salient region extraction. Figure 11 illustrates the feature visualization results of the ASCRM ablation experiments. Structures No.1 to No.9 exhibit various limitations, such as insufficient global understanding, low confidence in detailed features, and incomplete edge extraction. In contrast, No. 10 (ASCRM) demonstrates the best detection effect, effectively focusing on salient target regions and extracting fine-grained edge features. To further validate ASCRM’s effectiveness, it was compared with advanced attention mechanisms, including AAM [39], BAM [80], CAM [81], CBAM [82], CoT-Attention [83], ECA-Attention [84], NLAM [10], RSAM [85], and Self-Attention [86]. As shown in

Table 8. Comparative experiments of different attention modules on ORSI-4199 datasets. The best results are marked in bold.

Encoder	Evaluation Metrics
	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
Baseline	0.8766	0.8534	0.9334	0.0330
+AAM	0.8770	0.8539	0.9338	0.0327
+BAM	0.8767	0.8535	0.9335	0.0329
+CAM	0.8767	0.8536	0.9336	0.0329
+CBAM	0.8769	0.8538	0.9338	0.0328
+CoT-Attention	0.8772	0.8540	0.9341	0.0326
+ECA-Attention	0.8773	0.8542	0.9342	0.0325
+NLAM	0.8775	0.8544	0.9350	0.0324
+RSAM	0.8776	0.85475	0.9351	0.0324
+Self-Attention	0.8778	0.8548	0.9353	0.00322
+ASCRM	0.8779	0.8550	0.9354	0.0321

, ASCRM achieves higher scores across all evaluation metrics. Figure 12 provides heatmaps of various attention mechanisms, where blue indicates lower model attention and red indicates higher attention. ASCRM effectively guides the model to focus on salient regions, enabling precise extraction of fine-grained edge features for targets like vehicles, airplanes, and buildings.

4.4.3. Effect of SGAFF

To effectively integrate different levels of feature information, we constructed the Semantic-Guided Adjacent Feature Fusion (SGAFF) module to align and fuse detailed features between different scales. As shown in

Table 9. Comparative experiments of different feature fusion modules on ORSI-4199 datasets. The best results are marked in bold.

Encoder	Evaluation Metrics
	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
Baseline	0.8766	0.8534	0.9334	0.0330
+ Element-wise Summation	0.8769	0.8538	0.9339	0.0328
+ Element-wise Multiplication	0.8774	0.8542	0.9343	0.0325
+ Channel Concatenation	0.8779	0.8557	0.9359	0.0321
+ SGAFF	0.8788	0.8567	0.9362	0.0317

, SGAFF outperforms existing feature fusion methods, including element-wise summation, element-wise multiplication, feature channel concatenation, and AFAM [13], achieving the best evaluation scores. Figure 13 presents feature visualization analysis comparing SGAFF with other fusion methods. Element-wise summation introduces redundant information and biases towards common regions, while element-wise multiplication enhances commonly attended regions but excludes salient areas lacking sufficient features. Channel concatenation alleviates some defects but remains limited by feature richness and confidence. AFAM improves channel concatenation by incorporating deformable convolution and residual connections but is constrained by fused feature semantics. In contrast, SGAFF effectively learns and sorts semantic information in the deep network and supplements shallow local information, resulting in optimized feature fusion between different levels.

4.4.4. Effect of the Loss Function

To evaluate the effectiveness of the IG Loss + IoU Loss hybrid function, we compared various loss functions and their combinations, including F-measure Loss, BCE Loss, CT Loss, IG Loss, and their combinations with IoU Loss.

Table 10. Comparative experiments of different loss functions on ORSI-4199 datasets. The best results are marked in bold.

Encoder	Evaluation Metrics
	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	$\mathit{MAE}\downarrow$
F-measure Loss	0.8734	0.8566	0.8802	0.0395
CT Loss	0.8745	0.8491	0.8871	0.0389
BCE Loss	0.8748	0.8543	0.9097	0.0367
IG Loss	0.8755	0.8587	0.9212	0.0314
F-measure Loss + IOU Loss	0.8761	0.8609	0.9385	0.0292
CT Loss + IOU Loss	0.8852	0.8603	0.9418	0.0295
BCE Loss + IOU Loss	0.8868	0.8610	0.9444	0.0281
IG Loss + IOU Loss	0.8876	0.8617	0.9453	0.0277

shows that F-measure Loss performs poorly in metrics other than $F_{\beta }$, as it focuses solely on optimizing the $F_{\beta }$-measure metric. CT Loss assigns higher weights to boundary pixels using fixed or adaptive thresholds but lacks consistent mutual supervision between edge and salient region predictions, resulting in only slight performance improvements over F-measure Loss. BCE Loss, which calculates cross-entropy supervision between each pixel and the label, achieved a sub-optimal evaluation score with $S_{\alpha }$, $F_{\beta }$, $E_{\xi }$, and MAE reaching 0.8748, 0.8543, 0.8797, and 0.0361, respectively. IG Loss adaptively adjusts edge and salient region prediction losses using learnable parameters during training, improving over the second-best results by 0.07%, 0.44%, 1.15%, and 0.53%, respectively. When combined with IoU Loss, IG Loss + IoU Loss achieves the best results on the ORSI-4199 dataset, further validating the effectiveness of the hybrid loss function.

5. Conclusions

This study addresses the SOD task within the RSI domain by introducing TSFANet, which comprises three core components: TSDSM, ASCRM, and SGAFF. To enhance the model’s ability to extract salient region features, TSDSM integrates CNNs-Mamba and CNNs-Transformer dual encoders. The CNNs-Mamba encoder effectively captures local features, while the CNNs-Transformer encoder models global semantic information. ASCRM leverages the correlation consistency between local detail features and global semantic features to guide and optimize the fusion of semantic information and detailed features. Addressing significant semantic differences in cross-layer features, SGAFF is designed to align and fuse deep semantic features with multi-scale details, ensuring comprehensive feature integration. Through extensive evaluations against 26 state-of-the-art methods on multiple public RSI-SOD datasets, our method has demonstrated superior performance in various rating indicators and prediction quality metrics. Additionally, ablation experiments have validated the specific contributions of each component within TSFANet, underscoring the effectiveness of the integrated modules in improving SOD performance. The successful implementation of TSFANet highlights its potential for advancing salient object detection in complex remote sensing environments.