Brain-Inspired Synergistic Adversarial Framework for Style Transfer-Guided Semantic Segmentation in Cross-Domain Remote Sensing Imagery

Abstract

Domain shifts pose significant challenges for cross-domain semantic segmentation in high-resolution remote sensing imagery. Inspired by the cognitive mechanisms of the human brain, we propose a Brain-Inspired Style Transfer and Semantic Segmentation Collaborative Adversarial Framework (SAF), which mimics neural processes such as hierarchical perception, memory retrieval, and multimodal integration to enhance cross-domain feature alignment and segmentation performance. To achieve the joint optimization of style transfer and semantic segmentation networks, we introduce three key components: a Semantic-Aware Transformer Module (SATM) that dynamically captures and preserves essential semantic features during style transfer; a Semantic-Driven Multi-feature Memory Module (SMM) that stores and retrieves historical style and semantic information to improve domain adaptability; a Domain-Invariant Style-Semantic Center Space (DSCS) that aligns style and semantic features within a shared representation space, mitigating discrepancies between style and semantic domains. Extensive experiments across multiple tasks demonstrate that SAF effectively reduces distortions and semantic inconsistencies by achieving deep style–semantic alignment. Compared to leading approaches, SAF achieves a superior balance between style adaptation and semantic preservation, significantly improving model generalization in remote sensing applications.

1. Introduction

With the continuous advancements in Earth observation and artificial intelligence technologies, the semantic segmentation of high-resolution remote sensing imagery has been widely applied in fields such as environmental protection, urban planning, disaster prediction, and damage assessment [1,2]. Semantic segmentation aims to assign a semantic label to each pixel in an image, typically under the assumption that the training samples (source domain) and test data (target domain) follow the same data distribution [3]. However, in real-world scenarios, due to geographic complexity and diverse imaging conditions (e.g., illumination changes, seasonal variations, imaging altitudes, and sensor differences), domain shift commonly occurs [4]. This phenomenon significantly hinders model generalization in targeting domains with limited or no annotations [5], making domain adaptation a critical research focus.

To address domain shift, Unsupervised Domain Adaptation (UDA) techniques have been developed to align the feature distributions of source and target domains through adversarial training [6,7]. Although adversarial UDA methods effectively reduce distributional discrepancies, they often struggle with substantial style variations induced by environmental and sensor factors in remote sensing images. [7,8]. Particularly, adversarial training alone is insufficient to fully bridge cross-domain gaps when style and semantic structures change simultaneously [9,10,11,12].

Style transfer techniques provide an effective approach to mitigating style discrepancies between the source and target domains [13,14]. By transforming source domain images to match the target domain’s style, style transfer significantly reduces inter-domain differences, offering more consistent inputs for subsequent domain adaptation tasks [14,15]. However, existing approaches typically separate style transfer and semantic segmentation into independent processes, neglecting the interaction between style adaptation and semantic structure preservation. As illustrated in Figure 1, without semantic constraints, style transfer may disrupt the alignment of critical objects such as buildings, roads, and vegetation, leading to significant semantic distortions. Recent methods have attempted to integrate Generative Adversarial Networks (GANs) into style transfer [16,17,18], but maintaining fine-grained semantic consistency during visual style adaptation remains a major challenge.

Moreover, traditional artificial neural networks mainly rely on feedforward architectures and error backpropagation, lacking the dynamic feedback, memory retrieval, and multimodal integration capabilities observed in biological brains [19,20,21]. Recent advances in brain-inspired artificial intelligence have revealed that mechanisms such as hierarchical feature processing in the visual cortex, memory retrieval in the hippocampus, and cross-modal integration in the association cortex can significantly enhance generalization and adaptability [22,23,24].

To address the challenges of complex semantics and significant style variations in the cross-domain semantic segmentation of high-resolution remote sensing imagery, we propose a novel brain-inspired framework. The main contributions of this study are summarized as follows:

• We propose a brain-inspired synergistic adversarial framework (SAF) that integrates style transfer and semantic segmentation into a unified learning process. This framework enhances domain generalization by jointly optimizing style alignment and semantic consistency.
• We propose a Semantic-Aware Style Transfer Network (STGAN) equipped with a lightweight Semantic-Aware Transformer Module (SATM), which mimics the brain’s top–down and bottom–up information flow to guide style transformation while preserving semantic structure.
• We introduce a Semantic-Driven Multi-Feature Memory Mechanism (SMM) and a Domain-Invariant Style-Semantic Space (DSCS), which collaboratively support the storage, retrieval, and alignment of semantic and style features across domains, enabling effective style adaptation and cross-domain feature synergy.

The remainder of this article is organized as follows: Section 2 reviews related works on cross-domain semantic segmentation and style transfer. Section 3 details our proposed Brain-Inspired Style Transfer and Semantic Segmentation Framework, focusing on its core components. Section 4 discusses the experimental setup and results. Section 5 explores the interactions between the various modules within our framework. Finally, Section 6 concludes the paper, summarizing our findings and suggesting directions for future research.

2. Related Work

This section presents a concise review of relevant research, focusing on cross-domain semantic segmentation methods and recent advancements in style transfer for cross-domain tasks.

2.1. Unsupervised Domain Adaptation for Cross-Domain Semantic Segmentation

Cross-domain semantic segmentation is crucial for extending the applicability of remote sensing models to diverse environments. However, domain shifts caused by differences in imaging conditions, sensor characteristics, and geographic distributions often lead to significant performance degradation when models are directly transferred across domains [5]. In recent years, UDA has mitigated this issue by using adversarial training to align feature distributions and reduce inter-domain differences [7].

Early UDA methods primarily focused on feature-level and output-level adversarial alignment. For instance, adversarial training frameworks have been employed to match the feature distributions of the source and target domains, effectively reducing inter-domain discrepancies [7,8].

Recently, vision-language models and self-supervised learning strategies have been introduced to further improve UDA performance. Lim et al. leveraged vision-language models to relabel new classes in the target domain, addressing challenges posed by inconsistent taxonomies [25]. Subsequent research has introduced consistency-based regularization to enhance domain adaptation robustness. Zhou et al. proposed uncertainty-aware consistency regularization with dynamically weighted loss to promote reliable knowledge transfer across domains [26]. Li et al. incorporated weakly supervised constraints into the training objective to mitigate sensor and geographical domain shifts in remote sensing segmentation tasks [27]. Xu et al. proposed a self-embedding GAN framework to stabilize adversarial training and reduce the dependency on labeled data [28]. He et al. developed a cross-attention decomposition transformer equipped with gradient reversal blocks to explicitly reduce feature space domain gaps [29]. Luo proposed a teacher–student framework using spatial-aware data augmentation to mitigate pseudo-label noise and class imbalance in source-free adaptation [30]. Ren designed a mutual refinement and information distillation framework to enhance domain interaction and simplify testing with curriculum learning [31]. Wang introduced semantic augmentation across classes and domains to enhance generalization to unseen target domains [32].

While significant progress has been made, most existing UDA approaches mainly focus on aligning feature distributions at the semantic output or feature level. However, they often overlook the critical influence of style-level discrepancies, especially in very-high-resolution (VHR) remote sensing imagery, where large variations in illumination, seasonal appearance, and spectral responses exist between domains. These style inconsistencies lead to semantic distortions, even when feature-level alignment is performed, ultimately limiting segmentation accuracy and cross-domain robustness. Therefore, achieving a simultaneous alignment of both style and semantic structures remains a critical open problem, which our proposed framework aims to address.

2.2. Style Transfer for Cross-Domain Adaptation

Style transfer serves as a preprocessing step in cross-domain semantic segmentation by aligning source images with the target domain’s style. It mitigates visual style discrepancies between domains while preserving semantic content, thereby providing more consistent input data for domain adaptation.

Zhao proposed a self-training-guided domain disentanglement module to extract style features from both domains, employing a cross-domain self-training mechanism to stabilize adversarial training and improve segmentation performance [33]. Ni used integer programming to transfer target domain styles to source domain images with semantic constraints for category-level alignment [34]. Li introduced a semantic-preserving GAN by incorporating representation-invariant and semantic-preserving constraints, enabling unbiased source-to-target transformations for better domain-adaptive segmentation [35]. Psychogyios combined neural style transfer with segmentation models, applying localized style transfer with a boundary smoothing loss to optimize stylized region transitions [36]. Ye developed an attention-based multi-style transfer approach for instance segmentation, integrating convolutional block attention with conditional instance normalization [37]. Zhu addressed negative transfer caused by style differences with a shape-robust domain-adaptive segmentation algorithm, combining adjustable style transfer and feature differentiation [38]. Marco proposed a continual learning framework integrating style transfer to enhance cross-domain knowledge and mitigate forgetting in autonomous driving scenarios through a robust distillation framework [39]. To bridge the domain gap between synthetic and real-world data, Li proposed a pixel and feature contrastive alignment method, combining pixel-level style transfer with feature-level semantic alignment to improve segmentation [40]. Wang introduced an unrestricted adversarial attack method, embedding adversarial perturbations into style transfer to generate natural, highly transferable adversarial images with consistent style and semantics [41].

However, style transformation may result in semantic information loss or distortion, particularly in VHR remote-sensing data with complex semantic structures. Effectively integrating style transfer with cross-domain semantic segmentation remains a critical challenge.

3. Methodology

3.1. Overview Framework

Our SAF is illustrated in Figure 2. The Semantic Segmentation Adversarial Generative Network (SSGAN) consists of two primary components: the semantic segmentation network $G_{seg}$, which is constructed using multiple TRA-Blocks, and the semantic discriminator $D_{seg}$, implemented with lightweight convolutional layers. The Style Transfer Generative Adversarial Network (STGAN) comprises the style transfer generation network $G_{style}$, which integrates multiple ResNet blocks (RBs) with the SATM, and the style discriminator $D_{style}$, ensuring that the generated images accurately reflect the target domain style.

The SATM dynamically captures key semantic attention $AL$ during the style transfer process, while the SSGAN outputs high-order semantic feature maps $AH$. Inspired by the brain’s top–down and bottom–up information flow regulation, which facilitates prediction and error correction between lower and higher cortical regions, the SATM ensures semantic feature consistency in style transformation tasks. The SMM facilitates the conversion of short-term memory into long-term memory through an explicit memory network and leverages semantic guidance to dynamically store and retrieve historical style information. DSCS, inspired by the integrative function of the prefrontal cortex in processing heterogeneous visual features, projects style and semantic features from different domains into a shared domain-invariant feature space. By explicitly modeling the interrelationship between style and semantic features, the DSCS reinforces semantic consistency at the style level, enabling dynamic cross-domain feature alignment and collaborative optimization.

This work focuses on a labeled source domain dataset $S$ and an unlabeled target domain dataset $T$, which share the same target categories but differ in shooting conditions, locations, and styles. Images sampled from these datasets are denoted as $x_s$ and $x_t$, respectively. First, $\left\{x_{s1},x_t\right\}$ are fed into the STGAN to generate the style-transferred image $x_{st}$. This process facilitates the transfer of semantic annotations from the source domain to the target domain, ensuring that $x_{st}$ not only aligns with the style distribution of the target domain image $x_t$ but also preserves the semantic consistency of the source domain image $x_{s1}$. Subsequently, $\left\{x_{s2},x_{st},x_t\right\}$ are fed into the SSGAN. $G_{seg}$ generates semantic predictions for $\left\{x_{s2},x_{st},x_t\right\}$, resulting in predicted labels $\left\{p{l}_{s2},p{l}_{st},p{l}_t\right\}$. To ensure domain consistency, $D_{seg}$ evaluates whether these predicted labels originate from the same domain.

To better coordinate style transfer and semantic segmentation in cross-domain scenarios, our framework design draws inspiration from several cognitive processes observed in the human brain. Specifically, SAF models the bidirectional information flow and biological feedback mechanisms between lower-order and higher-order cortical regions. In biological systems, lower-order sensory areas primarily process the basic features of sensory input, while higher-order areas, such as the parietal and prefrontal cortices, generate predictions based on contextual information and provide top-down feedback to dynamically regulate sensory processing. Motivated by this mechanism, the SATM extracts key but ambiguous semantic cues at a lower level and guides the SSGAN to progressively refine the semantic structure, thereby establishing a dynamic feedback loop that maintains semantic consistency throughout the style adaptation process.

Moreover, the memory retrieval and imaginative reconstruction capabilities of the hippocampus are reflected in the design of the SMM. In the brain, imagination typically involves integrating multiple memory features to reconstruct coherent experiences. Analogously, the SMM stores and retrieves historical semantic and style-related features, using semantic guidance to selectively retrieve the most contextually relevant style features, thereby enhancing the generation of semantically aligned style representations. Furthermore, inspired by the association cortex’s role in integrating multimodal sensory information, the DSCS is designed to project style and semantic features from different domains into a unified latent space. This integration facilitates the alignment and fusion of heterogeneous features, enabling effective cross-domain feature synergy and improving both style adaptation quality and semantic segmentation performance.

3.2. Style Transfer Generative Adversarial Network with Semantically Aware Modules

The proposed style transfer network framework STGAN, with semantic perception capabilities, is illustrated in Figure 3. First, $x_{s1}$ is input into the RBs to generate the initial style-transferred image ${\overline{x}}_{st}$. To further enhance the semantic consistency of the generated images, $\left\{x_{s1},{\overline{x}}_{st},x_t\right\}$ are fed into the Semantic-Aware Transformer Module (SATM) for additional processing.

As shown in Figure 3b, SATM employs several lightweight attention mechanisms to dynamically focus on key semantic features in both the source and generated domain images, effectively guiding the style transformation process. Additionally, we integrated the Style Attention Module (SAM) into SATM, focusing on capturing fine-grained visual style features. First, local-style features are independently extracted for each channel using a $3\times 3$ convolution operation. Subsequently, pointwise convolution is applied to integrate information across different channels. Global average pooling is then used to aggregate the overall style information of the image into a unified global representation. Next, the global representation is aligned with local features through element-wise fusion, ensuring consistency between local and global style features. Finally, the style attention features and semantic features are fused to obtain the final semantic attention map $AL$.

The SATM dynamically refines ${\overline{x}}_{st}$ by extracting semantic attention features. Specifically, $\left\{x_{s1},x_{st},x_t\right\}$ are fed into the SATM to obtain the low-level semantic attention maps $\left\{A{L}_{s1},A{L}_{st},A{L}_t\right\}$. The differences among $\left\{A{L}_{s1},A{L}_{st}\right\}$ are computed to measure semantic-aware loss and ensure semantic consistency during style transfer:

$$Los{s}_{as}=-{\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{(A{L}_{s1,i}-A{L}_{st,i})}^2}$$

(1)

where $K$ represents the total number of elements in the semantic feature map. $A{L}_{s1,i}$ denotes the semantic attention value at the i-th position in $x_{s1}$, and $A{L}_{st,i}$ refers to the corresponding semantic attention value in ${\overline{x}}_{st}$.

$A{L}_{st}$ is utilized to further enhance ${\overline{x}}_{st}$, resulting in a style-transferred image ${\tilde{x}}_{st}$ with semantic consistency. This process can be expressed as follows:

$${\tilde{x}}_{st}=a_s\ast {\overline{x}}_{st}\ast (f_{conv}^{1\times 1}(f_{concat}^1({\overline{x}}_{st},A{L}_{st})))+(1-a_s)\ast {\overline{x}}_{st},a_s\in \left[0,1\right]$$

(2)

where $f_{conv}^{1\times 1}$ represents the channel mapping operation performed through $1\times 1$ convolution, and $f_{concat}^1$ denotes the concatenation operation along dimension 1.

The design of the SATM is inspired by the brain’s top–down and bottom–up information flow regulation mechanism: lower-level brain regions process basic sensory input features, while higher-level regions generate corresponding feedback signals to dynamically adjust the activity of lower-level regions. For instance, when visual input is blurry, higher-level regions can infer possible content based on prior experience and provide feedback to lower-level regions, thereby modulating the visual cortex’s response to blurry input. Specifically, $x_{st}$ is input into the encoder layer of $G_{seg}$ to obtain the high-level semantic attention map $A{H}_{st}$. The difference between the initial semantic attention map $A{L}_{s1}$ and the high-level semantic attention map $A{H}_{st}$ is used to compute a semantic consistency loss, which dynamically optimizes SATM to ensure that critical semantic information is preserved during the style transfer process:

$$Los{s}_{aLH}=-{\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K\log \frac{\exp \left(A{L}_{s1,i,\mathrm{y}}\right)}{{\displaystyle {\sum }_{c=1}^C\exp \left(A{L}_{s1,i,c}\right)}}},y=A{H}_{st,i,c}$$

(3)

In this process, $A{H}_{st}$ is treated as a pseudo-label, aiming for the predicted categories of the low-level semantic attention map to approximate $A{H}_{st}$ as closely as possible. $K$ denotes the total number of pixels in each image, and $C$ represents the total number of categories. $A{L}_{s1,i,\mathrm{y}}$ is the low-level semantic attention value at the i-th pixel in pseudo-category $y$, while $A{L}_{s1,i,c}$ represents the low-level semantic attention value at the i-th pixel in category $c$.

Finally, $\left\{x_{st},x_t\right\}$ are input into $D_{style}$ to determine the authenticity of the images. $D_{style}$ adopts a three-layer convolutional structure to generate authenticity scores for each pixel, which can be expressed as follows:

$$D_{style}(x)=f_{conv}^{1\times 1}(\mathrm{Re}\mathrm{lu}(\mathrm{Norm}(f_{conv}^{1\times 1}(\mathrm{Re}\mathrm{lu}(f_{conv}^{1\times 1}(x))))))$$

(4)

where Relu refers to the Leaky Relu activation function and Norm represents the batch normalization layer.

3.3. Semantic-Driven Multi-Feature Memory Module for Style Transfer

Style transfer methods guided by a single image may deviate from the overall style distribution due to the incompleteness of style and semantic categories, necessitating the incorporation of a memory module for dynamic guidance. The hippocampus, as a critical memory region in the brain, can integrate and reconstruct multi-feature information from various sensory channels. Inspired by this mechanism, the proposed SMM adopts a dual-channel architecture consisting of a semantic memory bank and a style memory bank. By leveraging semantic guidance to dynamically store and retrieve historical style information, the SMM effectively mitigates the incompleteness of style and semantic categories in target domain images.

Figure 4a shows the structure of the SMM. In the experiment, we observed that the shallow layers of the style generator primarily capture style-specific features rather than performing specific style transformations. Therefore, $x_t$ is fed into the shallow layers of the style generator to extract the current target domain style representation $T_{style}$, while $x_{s1}$ is input into the corresponding network layers to obtain the current source domain style representation $S_{style}$. This process can be expressed as follows:

$$T_{style},S_{style}=f_{concat}^1({\mathrm{RB}}_{1,\cdots 3}(x_t,x_{s1}))$$

(5)

where $f_{concat}^1$ denotes the concatenation operation along dimension 1, and ${\mathrm{RB}}_{1,\cdots 3}(\cdot )$ represents the output features extracted from the first to the third layers of the RBs.

Additionally, we designed both short-term and long-term memory modules. The short-term memory stores the style and semantic features of the most recent five images, while the long-term memory retains the style and semantic features of up to 20 historical images. $T_{style}$ is stored into the short-term and long-term style memory $M_{style}$, and the semantic attention map $A{L}_t$ obtained by SATM is stored into the semantic short-term and long-term memory $M_{seg}$. This process can be expressed as follows:

$$\left\{\begin{array}{c}{M}_{style}=f_{concat}^0(M_{style},f_{pool}^{32\times 32}(f_{conv}^{1\times 1}(T_{style})))\\ M_{seg}=f_{concat}^0(M_{style},f_{pool}^{32\times 32}(f_{conv}^{1\times 1}(A{L}_t)))\end{array}\right.$$

(6)

where $f_{pool}^{32\times 32}$ represents adaptive average pooling to a fixed output size of $32\times 32$.

The semantic attention weight $S{T}_{seg,a}$ is obtained by applying attention layers to $A{L}_{st}$. $A{L}_{st}$ is then employed to query the semantic memory $M_{seg}$, retrieving the semantic memory feature $S{M}_{ts}$ relevant to the current task. Subsequently, $S_{style}$ is applied to query $M_{style}$, retrieving the memory features $S{M}_{ss}$ associated with the current source domain style. This process is expressed as follows:

$$S{T}_{seg,a}=\mathrm{Sigmoid}(f_{conv}^{3\times 3}\ast \mathrm{Re}\mathrm{lu}(f_{conv}^{3\times 3}\ast \mathrm{Re}\mathrm{lu}(f_{conv}^{3\times 3}\ast A{L}_{st})))$$

(7)

$$S{M}_{ts}=f_{conv}^{1\times 1}(A{L}_{st}\ast S{T}_{seg,a}\ast M_{seg})$$

(8)

$$S{M}_{ss}=S{M}_{ts}\ast S_{style}\ast M_{style}$$

(9)

where $f_{conv}^{3\times 3}$ denotes a convolution operation with a kernel size of $3\times 3$.

The short-term and long-term style memories, denoted as $S{M}_{ss,short}$ and $S{M}_{ss,long}$, are extracted separately, and then adaptively fused to generate an integrated memory $S{M}_r$. This integrated memory is subsequently used to further optimize $x_{st}$. The process is expressed as follows:

$$S{M}_r=f_{conv}^{1\times 1}(f_{concat}^1(m_1\ast S{M}_{ss,short},(1-m_1)\ast S{M}_{ss,long})),m_1\in \left[0,1\right]$$

(10)

$$x_{st}=m_2\ast {\tilde{x}}_{st}+(1-m_2)\ast S{M}_r,m_2\in \left[0,1\right]$$

(11)

where $f_{conv}^{1\times 1}$ denotes a convolution operation with a kernel size of $1\times 1$, and $f_{concat}^1$ represents concatenation along dimension 1. $m_1$ and $m_2$ are the memory fusion coefficient. The memory loss is calculated as follows:

$$Los{s}_m=-{\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{(S{M}_{r,i}-x_{st,i})}^2}$$

(12)

where $K$ represents the total number of elements in the feature map.

3.4. Domain-Invariant Style-Semantic Center Space

We propose the domain-invariant style-semantic center space (DSCS), which explicitly incorporates the interrelationship between style and semantic features into the optimization objective by introducing a style-semantic central approximation loss. Inspired by the integrative function of the prefrontal cortex in the human brain for processing multimodal information, DSCS maps style and semantic features from different domains into a shared domain-invariant feature space. The workflow of DSCS is illustrated in Figure 4b. Specifically, we use the intermediate features extracted by $G_{style}$ and $G_{seg}$ to construct the cross-domain style feature space $C{S}_{style}$ and the cross-domain semantic feature space $C{S}_{seg}$. This process is represented as follows:

$$\left\{\begin{array}{c}C{S}_{style}=f_{conv}^{1\times 1}(f_{concat}^1(G_{style}^{6,\cdots 10}(x_{s1}),G_{style}^{6,\cdots 10}(x_t)))\\ C{S}_{seg}=f_{conv}^{1\times 1}(f_{concat}^1(G_{seg}^{6,\cdots 10}(x_{s2}),G_{seg}^{6,\cdots 10} {}^0(x_{st}),G_{seg}^{6,\cdots 10}(x_t)))\end{array}\right.$$

(13)

where $G_{style}^{6,\dots 10}(\cdot )$ and $G_{seg}^{6,\dots 10}(\cdot )$ denote the features extracted from layers 6 to 10 of $G_{style}$ and $G_{seg}$, respectively.

The DSCS dynamically refines the cross-domain style-semantic center through the style-semantic center approximation loss, defined as follows:

$$Los{s}_{cs}=-{\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{(C{S}_{style,i}-C{S}_{seg,i})}^2}$$

(14)

where $K$ is the total number of elements in the feature map.

The DSCS effectively mitigates visual style discrepancies between the source and target domains, reduces the interference of style differences on semantic feature extraction, and enables the dynamic collaborative optimization of cross-domain features. By progressively aligning the features of the source and target domains, DSCS enhances the feature extraction process of the semantic segmentation network, ensuring that the generated semantic features align with the target domain distribution at the style level. Acting as a dynamic feature fusion region, the center space adapts in real-time based on the distributional differences between the source and target domains, allowing the network to incrementally adapt to the target domain feature distribution.

3.5. Overall Objective Function

From an overarching perspective, our proposed model combines the concepts of style generation adversarial and semantic generation adversarial, and its objective function consists of four main components: the style generator loss $L_{G,style}$, the style discriminator loss $L_{D,style}$, the semantic generator loss $L_{G,seg}$, and the semantic discriminator loss $L_{D,seg}$.

(1)

Style Transfer GAN

The loss for the style generator is defined as a weighted combination of several sub-losses:

$$L_{G,style}=Los{s}_{SG}+{\lambda }_{SC}Los{s}_{SC}+{\lambda }_{as}Los{s}_{as}+{\lambda }_{aLH}Los{s}_{aLH}+{\lambda }_{m}Los{s}_m$$

(15)

where ${\lambda }_{SC},{\lambda }_{as},{\lambda }_{aLH},{\lambda }_m\in \left[0,1\right]$ represent the balance coefficients for style consistency loss, semantic perception loss, semantic alignment loss, and memory loss, respectively. In our framework, the loss balancing coefficients are set to ${\lambda }_{SC}=0.2$, ${\lambda }_{as}=0.6$, ${\lambda }_{aLH}=0.2$, and ${\lambda }_m=0.4$. The model demonstrates stable performance under moderate variations in these parameters.

During the training of the style generator, the objective is for the style discriminator is to classify $x_{st}$ as belonging to the target domain. The style consistency loss $Los{s}_{SG}$ is computed as follows:

$$Los{s}_{SG}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{(D_{style}{(x_{st})}^i-d_1^i)}^2}$$

(16)

where $D_{style}{(x_{st})}^i$ represents the score given by the style discriminator for the i-th pixel of $x_{st}$, and $d_1$ is an all-ones matrix representing the target domain label. The purpose of $Los{s}_{SG}$ is to ensure that the style features of the generated image closely align with those of the target domain.

To reduce the distance between $x_{st}$ and $x_t$ while increasing the distance between $x_{st}$ and $x_{s1}$, $\left\{x_{s1},x_{st},x_t\right\}$ are fed into $G_{style}$, and the intermediate outputs from its selected layers are concatenated. These concatenated feature maps are then passed through a 2D adaptive average pooling layer for dimensionality reduction and subsequently normalized to produce $\left\{{\overline{x}}_{s1},{\overline{x}}_{st},{\overline{x}}_t\right\}$. The detailed process is defined as follows:

$$\left\{\begin{array}{c}{\overline{x}}_{s1}=f_{Norm}^{L2}(f_{pool}^{4\times 4}(f_{concat}^1{f}_{G_{style}}^{4,8,12}(x_{s1})))\\ {\overline{x}}_{st}=f_{Norm}^{L2}(f_{pool}^{4\times 4}(f_{concat}^1{f}_{G_{style}}^{4,8,12}(x_{st})))\\ {\overline{x}}_t=f_{Norm}^{L2}(f_{pool}^{4\times 4}(f_{concat}^1{f}_{G_{style}}^{4,8,12}(x_t)))\end{array}\right.$$

(17)

where $f_{Norm}^{L2}$ represents the L2 normalization operation, ensuring the feature vectors are normalized, and $f_{pool}^{4\times 4}$ is an adaptive average pooling operation that outputs a fixed size of $4\times 4$. $f_{G_{style}}^{4,8,12}$ denotes the features extracted from the 4th, 8th, and 12th layers of $G_{style}$. The style contrastive loss is defined as follows:

$$Los{s}_{SC}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{({\overline{x}}_{s1,i}-{\overline{x}}_{st,i})}^2}-c\cdot {\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{({\overline{x}}_{t,i}-{\overline{x}}_{st,i})}^2},c\in \left[0,1\right]$$

(18)

where $c$ is the contrastive loss coefficient, used to balance the weight between positive and negative sample distances.

The style discriminator assigns scores to input images, with scores closer to 1 indicating that the image style is more similar to the target domain, and scores closer to 0 suggesting that the style is more aligned with the source domain. The loss function for the style discriminator is expressed as follows:

$$L_{D,style}={\lambda }_{d,0}{\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{(D_{style}{(x_{st})}^i-d_0^i)}^2}+{\lambda }_{d,1}{\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K{(D_{style}{(x_t)}^i-d_1^i)}^2}$$

(19)

where $D_{style}{(x_{st})}^i$ and $D_{style}{(x_t)}^i$ represent the scores assigned by the discriminator to the i-th pixel of the style-transferred image and the source domain image, respectively. $d_1$ represents a matrix filled with ones, denoting the target value for the target domain, while $d_0$ represents a matrix filled with zeros, denoting the target value for the source domain. ${\lambda }_{d,0},{\lambda }_{d,1}\in \left[0,1\right]$ are weighting factors balancing the contributions of the source and target domain loss terms.

(2)

Semantic Segmentation GAN

The input to the semantic triplet adversarial generation network consists of $\left\{x_{s2},x_{st},x_t\right\}$, with corresponding semantic labels $\left\{Labe{l}_{s2},Labe{l}_{st}\right\}$. Since $x_t$ serves as the final semantic segmentation target and lacks a label, the semantic feature generator loss is defined as a weighted combination of several partial losses:

$$L_{G,seg}={\lambda }_{cs}Los{s}_{cs}+{\lambda }_{seg}Los{s}_{seg}+{\lambda }_{adv}Los{s}_{adv}$$

(20)

where ${\lambda }_{cs},{\lambda }_{seg},{\lambda }_{adv}\in \left[0,1\right]$ are balancing coefficients for the style-semantic center approximation loss $Los{s}_{cs}$, the semantic segmentation loss $Los{s}_{seg}$, and the semantic adversarial loss $Los{s}_{adv}$, respectively. The semantic segmentation loss $Los{s}_{seg}$ is computed as follows:

$$Los{s}_{seg}=-{\displaystyle \sum _{i=1}^K{\displaystyle \sum _{n=1}^{\overline{N}}{L}_i^n}}\log p\left(n\left|x_{s2,i}\right.\right)-{\displaystyle \sum _{i=1}^K{\displaystyle \sum _{n=1}^{\overline{N}}{L}_i^n}}\log p\left(n\left|x_{st,i}\right.\right)$$

(21)

where $K$ is the total number of pixels in each image, and $\overline{N}$ is the number of categories. $p\left(n\left|x_i\right.\right)$ is the probability that the i-th pixel in $x$ is predicted to belong to class $n$, and $L_i^n\in \left\{0,1\right\}$ is the corresponding label as a binary vector.

To ensure the extracted semantic features align with the same domain and maintain domain-invariant characteristics, the semantic adversarial loss $Los{s}_{adv}$ is employed. $Los{s}_{adv}$ encourages the discriminator to classify $\left\{x_{s2},x_t\right\}$ as originating from the same domain. The calculation of $Los{s}_{adv}$ is defined as follows:

$$Los{s}_{adv}=-\log \left(D_{seg}\left(x_{s2},x_t\right)\right)$$

(22)

$D_{seg}$ represents the score assigned by the discriminator, indicating the probability that a given pair of images originates from the same domain. During the training of the semantic adversarial generation model, the style-transferred image $x_{st}$ is expected to align with the target domain, while $x_{s2}$ is considered representative of the source domain. Therefore, the discriminator is designed to classify $\left\{x_t,x_{st}\right\}$ as belonging to the same domain and $\left\{x_t,x_{s2}\right\}$ as originating from different domains. The loss function for the discriminator is defined as follows:

$$L_{D,seg}=-\log \left(D_{seg}\left(x_t,x_{st}\right)\right)-\log \left(1-D_{seg}\left(x_t,x_{s2}\right)\right)$$

(23)

4. Experiments

4.1. Datasets

The ISPRS Vaihingen and Potsdam Challenge datasets are widely recognized benchmarks for the ISPRS 2D semantic labeling challenges, focusing on very-high-resolution (VHR) remote sensing imagery. Both datasets provide pixel-wise annotations for five semantic classes: impervious surfaces, buildings, low vegetation, trees, and cars. For training, images from both datasets are cropped into 512 × 512 patches with an overlap of 200 pixels in both width and height. To augment the training set, data augmentation techniques, including horizontal flipping, vertical flipping, and resizing (scaling factors of 0.5 and 1.5), are applied.

The Vaihingen dataset consists of 32 IRRG (Infrared, Red, and Green) images, each with a resolution of 2500 × 2000 pixels. Among these, 16 images contain pixel-level annotations. We randomly selected 10 labeled images for training, while the remaining annotated images were used for testing.

The Potsdam Dataset comprises 38 images, each with a resolution of 6000 × 6000 pixels and a ground sampling distance (GSD) of 5 cm. This dataset includes two channel combinations: IRRG (Infrared, Red, and Green) and RGB (Red, Green, and Blue). Among the 38 images, 24 are annotated at the pixel level. We randomly selected 10 labeled images for training, while the remaining annotated images were used for testing. Unlike Vaihingen, Potsdam’s test patches were generated without overlap.

Vaihingen IRRG and Potsdam IRRG differ not only in scene structure but also in spectral properties, requiring models to effectively address spectral discrepancies. Furthermore, the transition from Potsdam IRRG to Potsdam RGB introduces an additional challenge, as models must adapt to domain shifts caused by spectral variations while preserving semantic consistency. The diversity in urban environments, including differences in vegetation, building layouts, and road networks, further increases the complexity of cross-domain adaptation. These characteristics make the selected datasets highly suitable for evaluating the robustness and generalization of the proposed method.

4.2. Experimental Settings

(1)

Evaluation Metrics

To evaluate the performance of semantic segmentation, we adopt several widely used metrics, including Intersection over Union (IoU), mean Intersection over Union (mIoU), F1 score (F1), and mean F1 score (mF1), following the standard calculation formulas outlined in [42]. The IoU and mIoU metrics measure the overlap between the predicted pixel set $P_m$ and the ground truth set $P_{gt}$, and are defined as follows:

$$IoU\left(P_m,P_{gt}\right)={\displaystyle \frac{\left|P_m\cap P_{gt}\right|}{\left|P_m\cup P_{gt}\right|}}$$

(24)

$$mIoU={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^{N}Io{U}_i}$$

(25)

where $N$ is the total number of semantic classes. The symbols $\cap$ and $\cup$ denote the intersection and union operations, respectively. The F1 score and mean F1 score are calculated based on precision and recall, as follows:

$$F1=2{\displaystyle \frac{Pre\times Rec}{Pre+Rec}},Pre={\displaystyle \frac{tp}{tp+fp}},Rec={\displaystyle \frac{tp}{tp+fn}}$$

(26)

$$mF1={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^{N}F{1}_i},L_{adv,F}=-\log \left(D\left(x_t,x_{s1}\right)\right)$$

(27)

where $tp$, $fp$, and $fn$ represent the number of true positives, false positives, and false negatives, respectively.

Additionally, to evaluate the quality of the style-transferred images, we adopt the Fréchet Inception Distance (FID) [43]. Assuming these features follow multivariate Gaussian distributions, let $\left(m,C\right)$ and $\left(m_w,C_w\right)$ denote the mean and covariance of the generated and real image features, respectively. The FID is then computed as follows:

$$FID((m,C),(m_w,C_w))={‖m-m_w‖}_2^2+\mathrm{Tr}(C+C_w-2{(C{C}_w)}^{1/2})$$

(28)

where $\mathrm{Tr}(\cdot )$ denotes the trace of a matrix, and ${(C{C}_w)}^{1/2}$ is the matrix square root of the product of covariances. A lower FID value indicates that the generated image distribution is closer to that of the real images, reflecting better visual fidelity and domain consistency.

(2)

Comparison Method

UDA Methods: To validate the effectiveness of our proposed SAF in cross-domain semantic segmentation, we compare it with several state-of-the-art UDA models, including TriADA [44], MCD [45], ResiDualGAN [46], CIA-UDA [34], DAFormer [47], Swin-Unet [48], HRDA [49], RS3Mamba [50], CACP [51], CSI [25], CACC [52], and DGSS [53].

The baseline method refers to a standard semantic segmentation model trained solely on labeled source domain data, without applying any domain adaptation techniques or style transfer mechanisms. During inference, the trained model is directly tested on the unlabeled target domain data. Th54is setup serves as a conventional reference point for evaluating performance degradation due to domain shift. To ensure a fair comparison, the network architecture, training configurations, and loss functions of the baseline are kept consistent with those of our proposed method, except that the style transfer module and collaborative components (i.e., SATM, SMM, and DSCS) are not included.

Style Transfer Methods: To evaluate the effectiveness of our style transfer module, we compare SAF with multiple representative style transfer models, including CycleGAN [54], CUT [55], and UNSB [56].

(3)

Implementation Details

For the style transfer model, we employ the Adam optimizer for both the generator and discriminator, with a learning rate of 0.0002 and betas (0.5, 0.999). For the semantic segmentation model, the generator was optimized using the SGD optimizer, with a momentum of 0.9, a weight decay of 0.0001, and an initial learning rate of 0.00025. The discriminator is trained with the Adam optimizer, using betas (0.9, 0.99) and an initial learning rate of 0.0001. All models are trained for 100,000 iterations on a GeForce RTX 3090 GPU, and all experiments are implemented using the PyTorch framework.

(4)

Cross-Domain Task Setup

To comprehensively evaluate the cross-domain generalization capability of the proposed framework, we design two cross-domain semantic segmentation tasks using three distinct datasets: Vaihingen IRRG, Potsdam IRRG, and Potsdam RGB.

The first task focuses on adaptation from Vaihingen IRRG to Potsdam IRRG. Although both datasets are captured in the near-infrared-red-green (IRRG) spectral range, significant domain discrepancies exist due to differences in urban morphology, building scales, vegetation density, road structures, seasonal variations, and acquisition conditions. Specifically, Vaihingen features a small-town environment characterized by tightly clustered residential buildings and narrower streets, while Potsdam represents a larger, more systematically organized urban area with broader roads and larger residential and commercial structures. These factors introduce considerable shifts in object shapes, sizes, spatial distributions, and spectral responses, despite the apparent spectral similarity.

The second task addresses adaptation from Potsdam RGB to Vaihingen IRRG, introducing an even greater level of domain shift. In addition to variations in urban structure and scene layout, this task involves a cross-spectral modality gap: Potsdam imagery is captured in the visible RGB range, while Vaihingen imagery employs an IRRG configuration incorporating near-infrared information. This substantial difference in spectral characteristics alters the visual appearance of key semantic categories, particularly vegetation, impervious surfaces, and buildings, thereby posing additional challenges for feature alignment and semantic consistency.

By covering both spatial and spectral domain shifts, these two tasks provide a rigorous and comprehensive evaluation of the proposed framework’s ability to achieve robust cross-domain generalization. They encompass variations in spatial structure, spectral modality, object morphology, seasonal imaging conditions, and data acquisition parameters, ensuring that the model is tested under diverse and realistic domain adaptation scenarios.

4.3. Cross-Domain Semantic Segmentation Task Between Vai and Potsdam Irrg

Table 1. Comparison of IoU (%), mIoU (%), F1 (%), and mF1 (%) for cross-domain semantic segmentation from Vaihingen IRRG to Potsdam IRRG. “Average Results” represent the mean values of IoU and F1 scores across all semantic categories, corresponding to mIoU and mF1, respectively.

Method	Car		Building		Tree		Low Veg		Surface		Average Results
	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	mIoU	mF1
Baseline	40.1	52.5	44.3	53.1	20.9	44.6	29.5	46.2	40.8	60.2	35.1	51.3
MCD	41.4	55.1	42.8	50.3	35.6	56.2	43.7	58.2	53.7	66.7	43.4	57.7
TriADA	59.9	69.1	71.5	81.2	48.6	66.8	57.9	72.1	68.3	81.9	61.2	74.2
ResiDualGAN	58.7	68.5	72.9	82.9	46.9	65.1	53.2	67.9	61.7	77.1	58.7	72.3
CIA-UDA	57.5	67.6	71.3	82.8	47.6	66.2	55.3	68.0	65.1	78.5	59.4	72.6
DAFormer	58.6	69.9	71.2	82.5	47.4	66.3	56.6	69.9	67.1	81.7	60.2	74.1
Swin_Unet	57.1	70.6	70.6	81.3	46.0	64.8	53.6	67.8	63.2	78.1	58.1	72.5
HRDA	55.8	64.3	72.4	83.5	49.1	67.5	52.1	66.4	64.8	78.9	58.8	72.1
RS3Mamba	59.5	69.4	72.7	81.6	48.9	67.4	54.9	68.3	66.7	80.5	60.5	73.4
CACP	58.2	68.7	71.9	82.0	47.2	65.5	55.8	68.6	65.5	79.7	59.7	72.9
CSI	56.1	66.3	70.2	81.5	46.7	65.0	54.5	67.1	64.2	78.6	58.3	71.7
CACC	57.6	68.0	71.0	82.2	47.5	66.0	57.0	70.0	68.1	80.3	60.2	73.3
DGSS	58.9	69.2	71.6	82.7	48.0	66.5	56.2	69.0	66.2	80.0	60.2	73.5
SAF (ours)	61.4	72.5	73.5	83.9	50.5	68.7	58.3	72.2	71.9	82.5	63.1	75.9

presents a detailed comparison of our proposed SAF model against advanced methods for cross-domain semantic segmentation. The results demonstrate that SAF outperforms all baseline and advanced approaches, achieving the highest mIoU of 63.1% and mF1 of 75.9%. These results highlight SAF’s superior generalization capabilities and its ability to accurately segment various semantic categories across domains.

Specifically, for small objects such as “Car”, SAF achieves an IoU of 61.4% and an F1-score of 72.5%, outperforming strong methods like RS3Mamba (59.5% IoU, 69.4% F1) and DGSS (58.9% IoU, 69.2% F1), demonstrating its robustness in capturing fine-grained features under domain shifts. For large and complex structures like “Building”, SAF attains an IoU of 73.5% and an F1-score of 83.9%, exceeding recent competitive methods such as DAFormer (71.2% IoU, 82.5% F1), HRDA (72.4% IoU, 83.5% F1), and even DGSS (71.6% IoU, 82.7% F1). In natural scene categories, SAF consistently outperforms existing methods. For “Tree” and “Low Vegetation”, SAF achieves IoUs of 50.5% and 58.3%, respectively, which are higher than those achieved by TriADA (48.6% and 57.9%) and RS3Mamba (48.9% and 54.9%), showcasing stronger adaptability to category-level variations and subtle appearance differences across domains. In the “Surface” category, SAF further achieves remarkable results, with an IoU of 71.9% and an F1-score of 82.5%, again surpassing DGSS (66.2% IoU, 80.0% F1) and CACC (68.1% IoU, 80.3% F1).

These improvements can be attributed to the synergistic contributions of the SATM, SMM, and DSCS modules. Specifically, the SATM enhances semantic consistency during style adaptation by dynamically preserving critical object-level features, the SMM retrieves semantically aligned historical features to optimize cross-domain style transfer, and the DSCS reinforces domain-invariant feature embedding through the joint alignment of style and semantic representations. This collaborative design enables the SAF to reduce domain-induced semantic distortions and appearance shifts simultaneously, thereby achieving robust cross-domain segmentation performance. Notably, the SAF improves the mIoU and mF1 by 28.0% and 24.6% over the baseline, demonstrating its substantial advantage in cross-domain segmentation.

Figure 5 presents the visualization results comparing the performance of different UDA algorithms on cross-domain semantic segmentation tasks. Compared to other methods, the SAF exhibits a superior ability to produce more accurate pixel-wise category predictions, effectively capturing semantic details and preserving class consistency across domains.

Table 2. FID scores (lower is better) for different style transfer tasks.

Method	Vaihingen IRRG →Potsdam IRRG	Potsdam IRRG →Vaihingen IRRG	Vaihingen IRRG →Potsdam RGB	Potsdam RGB →Vaihingen IRRG
CycleGAN	121.5	135.1	118.9	134.6
CUT	70.9	75.6	68.8	88.1
UNSB	79.2	86.1	75.4	76.8
SAF (ours)	62.1	69.7	62.7	61.9

provides the FID scores for different methods across various domain adaptation scenarios. Lower FID scores indicate better alignment between the source and target domains. Among the compared methods, SAF consistently achieves the lowest FID scores across all scenarios, demonstrating superior style transfer and domain adaptation performance. Specifically, the SAF outperforms CycleGAN, CUT, and UNSB by significant margins, highlighting its effectiveness in preserving content fidelity while aligning style characteristics. These improvements can be attributed to the integration of the SATM and SMM: the SATM ensures that style transformation is guided by semantically critical features rather than purely visual patterns, while the SMM provides memory-assisted retrieval of semantically consistent style features, reinforcing domain-aligned generation. By jointly optimizing semantic preservation and style consistency, the SAF achieves high-quality, domain-consistent results in cross-domain adaptation tasks.

Figure 6 presents a visual comparison of target domain images generated by different style transfer algorithms for the Vaihingen IRRG to Potsdam IRRG task. The images generated by CycleGAN exhibit noticeable overall style changes but introduce significant visual artifacts in local details, such as blurred building edges and distorted vehicle shapes in specific regions. Although UNSB mitigates some of these artifacts, it still struggles with detail blurriness, particularly at road intersections and vegetation boundaries. In contrast, CUT improved visual consistency but compromised critical textures in vegetation and buildings. Compared to these methods, the proposed SAF approach achieves a better balance between style preservation and semantic structure integrity. The style-transferred images generated by SAF effectively mitigate visual artifacts and mitigate detail blurriness, while maintaining fine-grained textures and accurate semantic information.

4.4. Cross-Domain Semantic Segmentation Task Between Potsdam Rgb and Vai

Table 3. Comparison of IoU (%), mIoU (%), F1 (%), and mF1 (%) for cross-domain semantic segmentation from Potsdam RGB to Vaihingen IRRG. “Average Results” represent the mean values of IoU and F1 scores across all semantic categories, corresponding to mIoU and mF1, respectively.

Method	Car		Building		Tree		Low Veg		Surface		Average Results
	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	mIoU	mF1
Baseline	8.5	25.8	41.9	50.5	36.2	50.0	24.6	47.1	26.5	34.3	27.5	41.5
MCD	17.1	35.6	57.4	70.2	47.5	68.8	32.7	49.9	58.5	72.6	42.6	59.4
TriADA	27.1	41.5	74.1	81.6	54.8	72.3	40.2	59.7	65.2	77.9	52.3	66.6
ResiDualGAN	28.3	43.0	73.9	81.2	54.6	71.9	41.5	58.8	64.8	76.8	52.6	66.3
CIA-UDA	29.8	44.9	74.6	82.0	55.1	71.5	41.3	58.4	65.9	78.1	53.3	66.9
DAFormer	29.5	44.6	74.2	81.5	55.3	72.1	42.4	59.0	63.7	75.2	53.0	66.5
Swin_Unet	28.4	43.8	70.6	81.4	53.1	70.7	36.3	57.8	64.1	75.5	50.5	65.8
HRDA	30.2	45.7	74.5	82.4	56.7	73.4	43.8	59.1	62.3	74.4	53.5	67.0
RS3Mamba	30.5	45.8	73.0	81.1	56.0	72.5	42.9	58.5	65.8	77.4	53.6	67.1
CACP	29.0	44.0	73.3	81.7	54.2	71.2	41.8	58.6	64.5	76.5	52.6	66.4
CSI	28.7	43.5	72.5	81.2	53.6	70.5	41.0	57.5	63.9	75.8	51.9	65.7
CACC	30.2	44.3	73.8	81.9	54.7	71.6	42.2	58.9	65.0	76.9	53.2	66.7
DGSS	29.6	44.7	74.0	82.0	55.0	71.9	42.5	59.2	65.4	77.3	53.3	67.0
SAF (ours)	31.8	46.8	75.2	83.9	57.9	74.8	44.5	60.8	66.6	79.4	55.2	69.1

presents the comparative results of cross-domain semantic segmentation performance from Potsdam R-G-B to Vaihingen using various state-of-the-art methods. The proposed SAF model demonstrates significant superiority, achieving the highest mean Intersection over Union (mIoU) of 55.2% and mean F1-score (mF1) of 69.1%, outperforming all baseline and advanced models. Specifically, for small objects like “Car”, the SAF achieves an IoU of 31.8% and an F1-score of 46.8%, surpassing HRDA and RS3Mamba, which previously performed well in this category. For large structures such as “Building”, the SAF achieves an IoU of 75.2% and an F1-score of 83.9%, further solidifying its capability to capture intricate semantic patterns. In natural scene categories such as “Tree” and “Low Vegetation”, the SAF demonstrates balanced performance with IoUs of 57.9% and 44.5%, and F1-scores of 74.8% and 60.8%, respectively. For “Impervious Surface”, SAF achieves an IoU of 66.6% and an F1-score of 79.4%, reflecting its ability to generalize well across varying spatial scales and feature distributions.

Compared to competitive methods such as CIA-UDA and DAFormer, the SAF consistently achieves better performance across all semantic categories. Furthermore, when compared with other recent state-of-the-art methods such as CACP, CSI, CACC, and DGSS, the SAF shows clear advantages in both mIoU and mF1 metrics. While CACP and DGSS achieve competitive segmentation in certain categories, their overall mIoU and mF1 scores remain lower than the SAF, indicating limited generalization across diverse semantic categories. For instance, CACP achieves an mIoU of 52.6% and DGSS achieves 53.3%, both lower than the SAF’s 55.2%. Similarly, CSI and CACC achieve mIoUs of 51.9% and 53.2%, respectively, still inferior to the SAF. These improvements can be attributed to the collaborative operation of the SATM, SMM, and DSCS. The SATM enhances cross-domain feature alignment by emphasizing critical semantic structures during style adaptation, the SMM facilitates semantically guided retrieval of historical features to maintain feature consistency under spectral shifts, and the DSCS projects heterogeneous style and semantic representations into a unified latent space, thereby reducing domain-induced discrepancies. Through this synergistic design, the SAF effectively bridges both spatial and spectral gaps, resulting in superior adaptability to diverse feature scales and complex cross-domain variations.

Figure 7 presents the visualization results of different UDA algorithms. Baseline and MCD exhibit significant misclassifications and poorly defined boundaries. While CIA-UDA, DAFormer, Swin_Unet, and RS3Mamba show progressive improvements in feature segmentation, they still struggle with fine-grained details and complex regions. By effectively integrating style transfer and semantic segmentation, the SAF achieves more accurate segmentation, sharper boundary delineation, and higher semantic consistency.

Figure 8 presents the visualization results of different style transfer algorithms for the Potsdam R-G-B to Vaihingen task. It can be observed that CycleGAN introduces significant misalignment in style transformation, leading to incorrect style adaptation in large buildings and vegetation areas. CUT mitigates excessive style deviations in certain regions through its unidirectional consistency constraint but fails to preserve the clear geometric structure of buildings. UNSB performs better in reducing artifacts but still suffers from information loss when dealing with details and small objects. In contrast, the SAF not only achieves a high degree of style alignment with the target domain but also demonstrates superior performance in preserving fine-grained details. For instance, vehicles retain clearer shapes, building contours are better preserved, and vegetation textures closely resemble the real distribution of the target domain.

5. Discussion

5.1. Ablation Study

The ablation study results in

Table 4. Ablation study results of our proposed SAF. Higher mIoU (%) and mF1 (%) indicate better segmentation performance, while lower FID values indicate better style transfer quality. The checkmarks (√) indicate that the corresponding module is included in the configuration for each task.

Task	STGAN	SATM	SMM	DSCS	SSGAN	FID	mIoU	mF1
Vaihingen IR-R-G ↓ Potsdam IR-R-G					√	/	52.1	62.8
	√					115.8	/	/
	√				√	80.6	59.4	72.3
	√	√			√	70.4	61.2	73.4
	√	√	√		√	65.3	62.7	74.6
	√	√	√	√	√	62.1	63.1	75.9
Potsdam R-G-B ↓ Vaihingen IR-R-G					√	/	43.5	58.0
	√					110.5	/	/
	√				√	77.2	51.3	65.2
	√	√			√	71.8	52.4	67.1
	√	√	√		√	64.9	53.9	68.5
	√	√	√	√	√	61.9	55.2	69.1

highlight the contribution of each module in enhancing cross-domain semantic segmentation. Using only STGAN and SSGAN results in low mIoU (52.1%) and mF1 (62.8%) with a high FID (115.8), indicating limitations in segmentation accuracy and image quality. Introducing SATM improves mIoU to 59.4% and mF1 to 72.3%, while reducing FID to 80.6, demonstrating that semantic-aware attention enhances style adaptation by preserving critical object-level structures. The integration of the SMM further refines feature retrieval, raising the mIoU to 61.2% and mF1 to 73.4%, and lowering the FID to 70.4, indicating that semantic-guided memory retrieval enhances cross-domain feature consistency. Incorporating the DSCS strengthens style-semantic alignment, achieving 62.7% mIoU and 74.6% mF1, and reducing FID to 65.3, reflecting the benefits of projecting heterogeneous features into a shared domain-invariant space. The full SAF achieves the best results (mIoU 63.1%, mF1 75.9%, FID 62.1), demonstrating the synergistic effect of integrating the SATM, SMM, and DSCS. This consistent trend is also observed on the Potsdam RGB to Vaihingen IRRG task, validating SAF’s robustness and generalizability across different domain adaptation scenarios.

To assess the efficiency and effectiveness of the SATM, we evaluated its computational overhead and impact on cross-domain segmentation and style transfer performance. Although the SATM introduces a moderate increase in inference time and memory consumption due to the use of five lightweight semantic-aware attention blocks, the additional cost remains acceptable, owing to dimension reduction strategies and compact design. The SATM effectively improves style adaptation, as evidenced by a notable reduction in the FID, by dynamically focusing on semantically critical regions during transformation. In addition, its hierarchical attention design preserves essential semantic structures, leading to consistent gains in mIoU and mF1 across different tasks.

5.2. Impact of Semantic Guidance on Style Transfer

Semantic guidance is primarily achieved through the SATM and further enhanced by the SMM and DSCS. Integrating SATM into the STGAN significantly reduces FID scores, reflecting its ability to dynamically capture key semantic features and align them with visual styles during style transfer. This ensures that the generated images maintain semantic content consistency, even under substantial domain shifts while reducing artifacts and improving overall domain alignment. The inclusion of the SMM further lowers the FID scores by leveraging memory-guided retrieval and the storage of historical style features. By incorporating semantically relevant information, SMM mitigates inconsistencies arising from incomplete or diverse domain-specific data, enhancing the model’s adaptability to complex target domain characteristics. The DSCS reduces the impact of domain-specific variations and enhances the coherence of style and semantic features. The complete SAF achieves the lowest FID scores, demonstrating that semantic guidance not only improves the quality of generated images but also ensures domain consistency.

The ablation study results in Figure 9 illustrate the progressive impact of different modules on style transfer quality. Without semantic guidance, the style transfer process relying solely on the STGAN leads to the loss of semantic details in vegetation and buildings. Introducing the SATM helps preserve the semantic information and maintain color consistency in buildings and roads. The addition of the SSGAN further enhances fine-grained details, such as vehicle structures and building contours, through semantic constraints. By integrating adversarial generation at both the style and semantic levels, the SAF facilitates a more precise and context-aware style adaptation process, achieving superior semantic consistency and visual coherence.

5.3. Impact of Style Transfer on Semantic Segmentation

Style transfer plays a pivotal role in enhancing semantic segmentation performance in cross-domain adaptation by addressing style discrepancies between source and target domains. These discrepancies often impede effective semantic feature extraction and generalization. Integrating style transfer into the SAF, particularly through the STGAN model, ensures that source domain images are transformed to align with the target domain’s visual style while preserving semantic content, thereby enhancing the adaptability of the segmentation network. For challenging small objects like “Car”, the alignment of style features improves both IoU and F1 scores, while for complex structures like “Building”, style transfer mitigates domain inconsistencies, yielding higher accuracy. Additionally, natural scene elements such as “Tree” and “Low Vegetation” benefit from this alignment, with the model showing improved generalization under diverse target domain conditions. By addressing domain discrepancies and aligning visual styles, style transfer optimizes semantic segmentation by enhancing feature consistency, improving adaptability, and ensuring robust performance across diverse target domains. Figure 10 presents the performance of various UDA algorithms in cross-domain semantic segmentation. Compared to TriADA and ResiDualGAN, which do not incorporate style transfer, the SAF demonstrates significantly superior results. In close-up regions such as roads, vegetation, and impervious surfaces, the SAF effectively captures boundaries and accurately predicts categories. By integrating style transfer with semantic segmentation, the SAF enhances feature alignment and achieves more precise pixel-level predictions, further demonstrating its robustness in cross-domain scenarios.

5.4. Cross-Domain Task Analysis

To further validate the effectiveness of the SAF under domain shift conditions, we analyzed the cross-domain segmentation results on the two designed tasks.

On the cross-domain task from Vaihingen IRRG to Potsdam IRRG, although both datasets share near-infrared spectral bands, substantial differences in urban morphology, building scale, vegetation distribution, and seasonal conditions pose significant adaptation challenges. The proposed framework consistently improves the mIoU and mF1 compared to the baseline and other competing methods, demonstrating its capability to align semantic structures across different urban layouts and environmental settings. Notably, the SATM plays a crucial role by preserving key object-level semantics during the style transfer process, enabling more stable feature alignment across domain variations.

For the more challenging cross-domain task from Potsdam RGB to Vaihingen IRRG, which involves both scene layout differences and cross-spectral modality gaps, the SAF achieves stable improvements in segmentation metrics while simultaneously reducing style discrepancies, as reflected by the FID scores. The integration of the SMM and DSCS modules effectively supports semantic-guided retrieval and cross-domain feature alignment, thereby enhancing domain invariance across both spatial and spectral dimensions. These results demonstrate that the SAF not only adapts to variations in spatial structures and object appearances, but also effectively generalizes across different spectral modalities, validating its robustness and flexibility in realistic cross-domain remote sensing segmentation scenarios.

6. Conclusions

In this study, we propose a brain-inspired synergistic adversarial framework (SAF) to address domain shifts in cross-domain semantic segmentation for high-resolution remote sensing imagery. By integrating style transfer and semantic segmentation into a unified adversarial learning framework, the SAF effectively enhances model generalization across different domains. To achieve collaborative optimization between style and semantic features, we introduce three key modules: the SATM, inspired by the brain’s hierarchical feedback mechanism, dynamically captures key semantic features to ensure semantic consistency during style transformation; the SMM, motivated by the hippocampus’ memory retrieval and consolidation functions, stores and retrieves historical style features to enhance adaptation to the target domain; and the DSCS, which aligns style and semantic features within a domain-agnostic latent space to reduce inter-domain style discrepancies while preserving semantic integrity.

Experiments on the ISPRS Vaihingen and Potsdam datasets demonstrate that the SAF outperforms state-of-the-art UDA and style transfer methods. Notably, the SAF effectively addresses both cross-spectral and cross-domain shifts, achieving superior results in adaptation tasks involving Vaihingen IRRG, Potsdam IRRG, and Potsdam RGB. This demonstrates its robustness in mitigating spectral discrepancies and structural variations.

Future work will focus on expanding the SAF to broader cross-domain and cross-spectral scenarios. This study demonstrates the effectiveness of brain-inspired mechanisms in preserving semantic integrity during style transfer while leveraging style adaptation to refine semantic feature learning, providing a promising direction for domain adaptation in remote sensing imagery.