SCDA: A Style and Content Domain Adaptive Semantic Segmentation Method for Remote Sensing Images

Abstract

Due to the differences in imaging methods and acquisition areas, remote sensing datasets can exhibit significant variations in both image style and content. In addition, the ground objects can be quite different in scale even within the same remote sensing image. These differences should be considered in remote sensing image segmentation tasks. Inspired by the recently developed domain generalization model WildNet, we propose a domain adaption framework named “Style and Content Domain Adaptation” (SCDA) for semantic segmentation tasks involving multiple remote sensing datasets with different data distributions. SCDA uses residual style feature transfer (RSFT) in the shallow layer of the baseline network model to enable source domain images to obtain style features from the target domain and reduce the loss of source domain content information. Considering the scale difference of different ground objects in remote sensing images, SCDA uses the projection of the source domain images, the style-transferred source domain images, and the target domain images to construct a multiscale content adaptation learning (MCAL) loss. This enables the model to capture multiscale target domain content information. Experiments show that the proposed method has obvious domain adaptability in remote sensing image segmentation. When performing cross-domain segmentation tasks from VaihingenIRRG to PotsdamIRRG, mIOU is $48.64\%$, and the F1 is $63.11\%$, marking improvements of $1.21\%$ and $0.45\%$, respectively, compared with state-of-the-art methods. When performing cross-domain segmentation tasks from VaihingenIRRG to PotsdamRGB, the mIOU is $44.38\%$, an improvement of $0.77\%$ over the most advanced methods. In summary, SCDA improves the semantic segmentation of remote sensing images through domain adaptation for both style and content. It fully utilizes multiple innovative modules and strategies to enhance the performance and the stability of the model.

1. Introduction

In recent years, computer technology, modern communication technology, and automatic interpretation methods of high-resolution remote sensing images (HRI) have been more and more widely applied to monitor changes on Earth’s surface [1]. Among them, target detection technology [2] can be used to improve national defense security ships, vehicle detection [3], change detection technology that can be used to monitor forest area changes, building changes [4], band selection technology that can monitor specific targets [5], and 3D reconstruction technology that can reconstruct high-precision ground objects and their rich details [6]. The semantic segmentation technology of remote sensing images aims to achieve accurate identification of each pixel, which has great application value and potential. HRI segmentation approaches can be used for land cover classification, road extraction, building extraction, farmland classification, and vegetation identification, and they play a vital role in the fields of urban planning, environmental monitoring, transportation planning, disaster assessment, and land management [7,8,9,10].

In recent years, segmentation models based on convolutional neural networks (CNNs) have achieved remarkable progress in the field of image semantic segmentation tasks [11]. The superior performance of CNN models highly depends on a large amount of labeled training data. However, for semantic segmentation tasks of remote sensing images, this prerequisite is often difficult to meet. HRIs typically have more complex structures than natural images, and the image annotation procedure frequently requires a great deal of domain expert knowledge [12]. Cordts et al. [13] reported that pixel-level annotation of a city overhead optical image takes nearly 90 min. Considering the complexity and high time cost of pixel-level annotation for HRIs, it is necessary to investigate how to segment unlabeled target domain images with a segmentation model trained on labeled source domain samples. Since HRIs are often obtained by different sensors or from different geographical locations, the data distributions in different datasets are usually quite different. Such variation can lead to significant performance decline when a model trained on a source domain dataset is deployed on a target domain dataset. As shown in Figure 1, there are significant differences between HRIs of different datasets. These HRIs captured from different regions by different sensors not only have obvious differences in content but also show significant difference in style (color, texture). In addition, there are scale differences between different object types in HRIs. Therefore, it is necessary to reduce the domain difference from the style and content of HRIs and, at the same time, consider the influence of the scale variations between different ground objects.

Unsupervised Domain Adaptation (UDA) [14,15,16,17,18,19] is a widely used technology to solve the difference in data distribution, which aims to use only labeled source domain samples and unlabeled target domain samples to train the model to achieve the purpose of reducing data discrepancies between domains. According to Xu et al. [14], the current UDA methods can be divided into four categories, namely the generation methods [19], the adversarial methods [15,16], the self-training methods [20], and the hybrid training methods. Our research focuses on the first two categories. Generative methods focus on differences that alter the appearance of an image, such as color, contours, and texture. Inspired by the image-to-image translation task of Generative Adversarial Networks (GANs), Bousmalis et al. [19] uses CycleGAN to generate fake images that approximate the style of the target domain. The goal of these methods [17,18,19] is to generate images that resemble the appearance of the target domain. This approach aims to facilitate the training of high-performance models using images and labels from the source domain, thereby enhancing their consistency with the target domain. The goal of adversarial methods is to extract information at the level of features, pixels, or both and learn domain-invariant features through domain discriminators. Ref. [21] tries to adjust the feature space distribution through a domain classifier. Long et al. [22] proposes a framework to make deep features more transferable by the low-density separation of target domain unlabeled data in deep features and further reduce domain variance by enhancing the statistical power of kernel embedding matching. Most of these classic UDA methods have primarily focused on classification tasks. However, more and more techniques have also been developed for other more specific vision tasks, including the cross-domain semantic segmentation of HRIs.

Currently, in the field of domain adaptive semantic segmentation of HRIs, several methods [23,24,25,26,27,28] have designed cross-domain adaptation frameworks under dynamically changing imaging modes or geographic location changes. Among these methods, GAN-based image style conversion techniques are commonly employed to adapt the network to the target domain. For example, the study in [28] employed DualGAN [29] to translate the source domain image into a target domain for the first time in HRIs cross-domain semantic segmentation. Zhao et al. [23] proposes a geometric consistency (GC) constraint that can be embedded into image conversion. Zhao et al. [30] also proposed an architecture based on DualGAN, which uses an adjustment module inside the network to solve the scale difference of the HRIs dataset and uses residual connections to enhance the stability of the actual converted image. Additionally, some studies focus on leveraging domain discriminators and adversarial training at the feature and category levels to achieve domain adaptation. Liu et al. [26] develops a full-space domain adaptation framework to reduce domain differences in feature distributions by adversarial learning from image space, feature space, and output space. Chen et al. [31] propose a region and category adaptive domain discriminator (RCA-DD), aiming to emphasize the differences in regions and categories during the process of alignment. There are also studies combining the above two methods. For example, Cai et al. [25] proposes an unsupervised domain adaptation method based on bidirectional image translation to take full advantage of the advantages of both domains and overcome the performance issues caused by unidirectional domain adaptation. V2RNet [24] unifies the semantic structure of the source domain with the image style of the target domain by adding a semantic discriminator to the style transfer network SegGAN. Although these works [23,24,25,26,27,28] have achieved excellent results on the target domain, for HRIs datasets, the difference in image style and the difference in image content often exist at the same time. None of the above studies regard HRIs as two parts of style and content or use these two directions to reduce domain differences. In addition, the disturbance of source content caused by style changes and the scale differences in feature categories in HRIs datasets also need to be considered.

Therefore, we refer to the architecture of WildNet [32] in our research. WildNet is a domain generalization method based on style transfer and contrastive learning to simultaneously adapt to the differences in style and content of the dataset. However, WildNet [32] is only studied on ordinary optical images, and it is not used on remote sensing image HRIs. In addition, WildNet also does not take into account the scale differences between different object types in remote sensing image HRIs, such as the size differences of car and road categories. In our work, we adopt the foundational structure of WildNet to address the issue of the network becoming overly specialized to the style and content of a single source domain. At the same time, according to the characteristics of remote sensing image data, we improve and optimize this architecture. Firstly, considering that the source content will be distorted when the target domain style is transferred to the source domain image in WildNet, the improved style transfer module residual style feature transfer (RFST) is proposed, which uses residual connections to enhance the stability of the actual converted content features and prevent the loss of source domain content. Then, to address the problem of scale differences between different types of features in the same HRIs dataset, this study constructed a Multiscale Content Adaptive Loss (MCAL) function based on the content characteristics of the network to train the model to learn content information at multiple scales in the target domain. We refer to this DA framework for remote sensing image segmentation tasks as Style and Content Domain Adaptation (SCDA). The main contributions of this paper are as follows:

(1)

This paper draws on the multiloss function architecture in WildNet to realize domain adaptation with regard to both the style and the content of different images. This domain adaption technology is then introduced into the semantic segmentation tasks of remote sensing images.

(2)

As an improved model based on WildNet, a residual structure is added into the style transfer module, aiming to fully learn the style information of the target domain without losing the content information of the source domain data. Meanwhile, a multiscale model is added at the output end of the model to learn content information at different scales in the target domain.

(3)

In summary, we propose a novel end-to-end domain adaptive semantic segmentation framework SCDA for remote sensing images. The proposed method achieves state-of-the-art performance on cross-domain semantic segmentation tasks between two open-source datasets, Vaihingen and Potsdam. When performing cross-domain segmentation tasks from Vaihingen to Potsdam, mIOU is $48.64\%$ and $44.38\%$, which is an improvement of $1.21\%$ and $0.77\%$ compared with the most advanced methods.

The rest of this paper is organized as follows. Section 2 reviews the related work. Section 3 presents the theory of our proposed framework. Section 4 introduces experimental settings and analyses the experimental results. Finally, conclusions are provided in Section 5.

2. Related Work

In Section 2.1, we introduce some basic knowledge about style transfer and contrastive learning, as well as domain adaptation methods based on style transfer and contrastive learning. In Section 2.2, as the basis of our approach, we introduce the main ideas of WildNet [32].

2.1. Domain Adaptation Techniques Based on Style Transfer and Contrastive Learning

Style transfer aims to transform an image into the style of another image while keeping the original content intact. The application of style transfer methods based on adaptive instance normalization(AdaIN) [18] in domain adaptation has attracted the attention of many researchers. For example, Marsden et al. [33] proposed a style transfer method based on AdaIN, which uses the domain change information of the category and obtains the category-specific target moment by pseudolabeling to convert the style of the labeled source image into the style of the target domain. Figure 2 shows the style transfer network using AdaIN. In the AdaIN [18] module, two types of input are received: content input x and style input y, and the channel-level mean and standard deviation of x are matched to the channel-level mean and standard deviation of y to ensure style similarity. In other words, AdaIN achieves style transfer by changing the data distribution at the feature level, ensuring that the computational and storage overheads are small and easy to implement. Its main idea can be defined as

$$A\mathrm{d}aIN(x,y)=\sigma \left(y\right)\frac{x-\mu \left(x\right)}{\sigma \left(x\right)}+\mu \left(y\right)$$

(1)

where x and y denote the image of the contributed content and the image of the contributed style, respectively, $\sigma \left(x\right)$ and $\mu \left(x\right)$ are the channel-wise mean and the standard deviation of x, and $\sigma \left(y\right)$ and $\mu \left(y\right)$ are the channel-wise mean and the standard deviation of y.

Contrastive learning [34] is a learning strategy that aims to minimize the distance between positive samples in the embedding space while maximizing the distance to negative samples. At present, some work has proposed a domain adaptation framework based on contrastive learning methods. For example, Wang et al. [35] developed a framework based on contrastive learning that accomplishes domain alignment by minimizing the distance between the target domain anchor images and the cross-domain samples of the same category in comparison with the distance from the cross-domain samples of different categories. Among contrastive learning frameworks, SimCLR [36] is an exceptional framework for unsupervised comparison. It uses differently augmented versions of the same image as positive samples, while treating distinct images as negative samples. This encourages the positive samples to cluster closely in the embedding space while keeping the negative samples apart. Therefore, a SimCLR-trained network can spontaneously cluster similar samples. The principle of SimCLR is shown in Figure 3.

Its main idea can be defined as

$${\mathcal{L}}_{i,j}=-log\frac{exp(sim(z_i·z_j)/\tau )}{{\sum }_{k=1}^{2N}{\mathbb{1}}_{[k\ne i]}exp(sim(z_i^s·z_k^t)/\tau )}$$

(2)

where $sim(\mu ,\nu )={\mu }^T\nu /∥\mu ∥∥\nu ∥$, which denotes the dot product between ${\mathcal{L}}_2$ normalized $\mu$ and $\nu$. $(i,j)$ denote a positive pair of examples, and ${\mathbb{1}}_{[k\ne i]}\in \{0,1\}$ is an indicator function evaluating to 1 if $k=i$ and $\tau$ denote a temperature parameter.

2.2. WildNet: A Domain Generalization Model Based on Multiple Loss Functions

The main idea of WildNet is to enable the network to learn domain-general semantic information from actual situations. In general, WildNet contains four modules: Feature Stylization (FS), Style Extension Learning (SEL), Semantic Consistency Regularization (SCR), and Content Extension Learning (CEL). The first three modules together can realize a style adaptive segmentation, and CEL is a technique based on contrastive learning [36,37] which enables content adaptive segmentation. The architecture of WildNet is shown in Figure 4.

In WildNet, the FS method adds an AdaIN [18] layer to the shallow structure of the encoder to adjust the shallow features to obtain the style information of the unknown domain. Refer to Equation (1), which is defined as follows:

$$z_l^{sw}=\sigma \left(z_l^w\right)\frac{z_l^s-\mu \left(z_l^s\right)}{\sigma \left(z_l^s\right)}+\mu \left(z_l^w\right)$$

(3)

where ${\varphi }_l$ denotes the l-th layer of the network, and $z_l$ represent the output features of the l-th layer. In the l-th layer, $z_l^s$ represents source domain style features, and $z_l^w$ represents wild domain style features. The source domain features stylized in the wild domain style, denoted as $z_l^{sw}$, are generated by transferring the wild domain style features $z_l^w$ to the source domain $z_l^s$. $\mu \left(z_l\right)$ and $\sigma \left(z_l\right)$ are the channel-wise mean and the standard deviation of feature $z_l$, respectively.

CEL is based on the contrastive learning strategy [36,38,39], and it consists of two loss functions, namely pixel-wise source content extension loss ${\mathcal{L}}_{SCE}$ and pixel-wise wild content extension loss ${\mathcal{L}}_{WCE}$. ${\mathcal{L}}_{SCE}$ aims to control that source domain content features will not be lost due to style transfer, ${\mathcal{L}}_{WCE}$ trains the network to cluster content containing the same semantic information and distinguish it from content containing other semantic information. They are deformed on the basis of Equation (2) and defined as follows:

$${\mathcal{L}}_{SCE}=-\frac{1}{N_z}\sum _{i=1}^{N_z}log\frac{exp\left(z_i^s·z_i^{sw}/\tau \right)}{exp\left(z_i^s·z_i^{sw}/\tau \right)+{\displaystyle \sum _{j=1}^{N_z}}{\mathbb{1}}_{ij}^{s}exp\left(z_i^s·z_j^{sw}/\tau \right)}$$

(4)

$${\mathcal{L}}_{WCE}=-\frac{1}{N_z}\sum _{i=1}^{N_z}log\frac{exp\left(z_i^s·z_k^w/\tau \right)}{exp\left(z_i^s·z_k^w/\tau \right)+{\displaystyle \sum _{j=1}^{N_z}}{\mathbb{1}}_{ij}^{s}exp\left(z_i^s·z_j^w/\tau \right)}$$

(5)

where $z_i^s$ is the projected content feature of the i-th pixel of the source image, and $z_i^{sw}$ is the projected content feature of the i-th pixel of the wild domain stylized source domain image. $z_k^w$ is a wild content feature selected from the wild content feature dictionary $Q\in {\mathbb{R}}^{C_q\times N_q}$ that is similar to the stylized source domain content feature of the wild domain. ${\mathbb{1}}_{ij}^s$ represents an indicator. When $z_i^{sw}$ and $z_i^s$ are the same, ${\mathbb{1}}_{ij}^s$ is 0. When $z_i^{sw}$ and $z_i^s$ are not the same, ${\mathbb{1}}_{ij}^s$ is 1. $N_z$ represents the number of pixels, and $\tau$ is the temperature parameter.

However, this approach was primarily investigated and adapted for conventional optical images and did not consider the unique characteristics of High-Resolution Images (HRIs) in remote sensing, such as the significant scale difference between categories of ground objects. In order to adapt the multiloss function architecture of WildNet to domain adaptive semantic segmentation of remote sensing images, we propose a learning method that uses a multiscale contrast mechanism to focus on both larger and smaller regions simultaneously.

3. Materials and Methods

In this section, we describe the SCDA method in detail. SCDA is an improved domain adaptive semantic segmentation framework adapted to the data characteristics of HRIs. Figure 5 shows the principle of the proposed SCDA method.

3.1. Preliminaries

In Figure 5, $x^s\in {\mathbb{R}}^{H_s\times W_s\times C_s}$ denotes the source domain image, and $x^t\in {\mathbb{R}}^{H_t\times W_t\times C_t}$ denotes the target domain image, where $H,W,C$, respectively, denote the height, width, and number of channels of the image. $y^s\in {\mathbb{Z}}^{H_s\times W_s\times K}$ denotes the label of the source domain image $x^s$, where K denotes the number of categories. To reduce the impact of scale factors [30], the size of the image should meet the following conditions:

$$\frac{H_s}{H_t}=\frac{W_s}{W_t}$$

(6)

$\varphi$ denotes the semantic segmentation model, which mainly consists of two parts: the feature extractor ($\varphi feat$) and the classifier ($\varphi cls$). The basic semantic segmentation model used in this paper is DeepLab v3 [40], and the encoder structure is ResNet-50 [41]. The model is trained using a cross-entropy loss function:

$${\mathcal{L}}_{ori}=-\frac{1}{HW}\sum _{h=1}^H\sum _{w=1}^W\sum _{k=1}^K{y}_{hwk}^{s}log\left(p^s\right)$$

(7)

$$p^s={\varphi }_{feat}\left({\varphi }_{cls}\left(x^s\right)\right)$$

(8)

$${\varphi }_{cls}\left(z_i\right)=Softmax\left(z^i\right)=\frac{e^{z_i}}{{\displaystyle \sum _{k=1}^K}{e}^{z_{\mathrm{k}}}}$$

(9)

where $y_{hwk}^s$ is a one-hot vector that represents the pixel class label of the source domain image. $p^s$ is the prediction of the network. $\varphi cls$ is the Softmax function. Based on the above segmentation loss using source domain samples for model training, we apply RSFT, MCAL, SEL, and SCR methods during the model training process to achieve style adaptation and content adaptation. SEL and SCR are the native modules, as in the WildNet architecture, while RSFT and MCAL are the improved versions of FS and CEL. In each training iteration, a pair of images, $x^s$ and $x^t$, are randomly selected from the labeled source domain dataset $D_s=\left\{x^s,y^s\right\}$ and the unlabeled target domain dataset $D_t=\left\{x^t\right\}$ as the input to the model. In the feature extractor $\varphi feat$, the source domain image $x^s$ obtains the style features of the target domain image $x^t$ through the RSFT process. The classifier $\varphi cls$ provides the prediction results for both the source domain image and the style-transferred source domain image. The projector $\varphi proj$ outputs the content embedding features of the source domain image, the style-transferred source domain image, and the target domain image, respectively. The model learns the similar features in the source and target domains using SEL, SCR, and MCAL methods. After training, the model is evaluated on the validation set of the unseen target domain T.

3.2. Residual Style Feature Transfer

The study in [42,43] suggests that overfitting to the source domain is often caused by the network overlearning the limited style of the source domain. Moreover, the variation in image styles can have an impact on the semantic segmentation results. The performance of the base network model decreases significantly when the style of the source domain images change after the training process of the model. Zhao et al. [30] use a GAN for image translation operation using residual structure so that the image is transformed into the target domain style without losing source domain information. In the work in Ref. [18], we added a simple residual structure to the FS module based on AdaIN in WildNet to reduce the loss of source domain content information. The structure of the RFST is shown in Figure 6. Therefore, referring to Equation (3), the source domain features after target domain stylization $z_l^{st}$ can be defined as

$$z_l^{st}=\sigma \left(z_l^t\right)\frac{z_l^s-\mu \left(z_l^s\right)}{\sigma \left(z_l^s\right)}+\mu \left(z_l^t\right)+k\times z_l^s$$

(10)

where k is a hyperparameter. ${\varphi }_l$ denotes the l-th layer of the network, and $z_l$ represent the output features of the l-th layer. In the l-th layer, $z_l^s$ represents source domain style features, and $z_l^t$ represents target domain style features. The source domain features stylized in the target domain style, denoted as $z_l^{st}$, are generated by transferring the target domain style features $z_l^t$ to the source domain $z_l^s$. $\mu \left(z_l\right)$ and $\sigma \left(z_l\right)$ are the channel-wise mean and the standard deviation of feature $z_l$, respectively.

Following Equation (10), the distribution of $z_l^s$ is renormalized with the channel-wise statistics of $z_l^t$. The target-stylized feature $z_l^{st}$ is input into layer $l+1$, and $z_{l+1}^{st}={\varphi }_{l+1}\left(z_l^{st}\right)$ is output from the layer. $z_{l+1}^{st}$ can be swapped repeatedly in the style of $z_{l+1}^t$ as

$$z_{l+1}^{st}:=\sigma \left(z_{l+1}^t\right)\frac{z_{l+1}^{st}-\mu \left(z_{l+1}^{st}\right)}{\sigma \left(z_{l+1}^{st}\right)}+\mu \left(z_{l+1}^t\right)+k\times z_l^{s}t$$

(11)

Following this equation, the distribution of $z_l^s$ is renormalized with the channel-wise statistics of $z_l^t$, and the source domain image acquires the target domain style at multiple levels within the network. However, as the network level deepens, the output features are more biased towards semantic information. Therefore, in this paper, RSFT is only applied to the shallow layers of the network.

3.3. Multiscale Content Adaptation Learning

The Equations (4) and (5) solve the problem that the network will learn limited source content by increasing the content diversity within the target domain class in the embedding space. However, remote sensing images have a common problem of large differences in scale between object categories. The above equations can only obtain fixed-scale content information but cannot pay attention to both small-scale and large-scale feature category content information. Therefore, we optimized based on the above method and proposed the MCAL method to solve the problem of large differences in scales of ground object categories. The structure of the MCAL is shown in Figure 7. The specific operation is as follows: a projector $\varphi proj$ is added in parallel with the classifier $\varphi cls$ after the feature extractor $\varphi feat$. This projector $\varphi proj$ maps the pixel-level features extracted by the feature extractor $\varphi feat$ into the embedding space. That is, after the source domain image $x^s$ and target domain image $x^t$ are input into the feature extractor $\varphi feat$, the output source domain feature $z^s$, the target domain feature $z^t$, and the target domain stylized source domain feature $z^{st}$ are input into the projector $\varphi feat$, outputting the source domain projected content feature $z_{proj}^s$, the target domain projected content feature $z_{proj}^t$, and the target domain stylized source domain projected content feature $z_{proj}^{st}$. It should be noted that the projected content feature $z_i^s$ of the i-th pixel corresponding to the source domain image and the projected content feature $z_i^{st}$ of the i-th pixel of the target domain stylized source domain image should have the same semantic information.

In order to obtain common domain content information, the network should be able to cluster projection features containing the same content information and distinguish projection features containing other content information. Inspired by Equation (4), we define the i-th source domain adaptation content loss ${\mathcal{L}}_{SMCAL}$ as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{SMCAL}=-\frac{\alpha }{N_l}\sum _{i=1}^{N_l}log\frac{exp\left(z_i^s·z_i^{st}/\tau \right)}{exp\left(z_i^s·z_i^{st}/\tau \right)+{\displaystyle \sum _{j=1}^{N_l}}{\mathbb{1}}_{ij}^{s}exp\left(z_i^s·z_j^{st}/\tau \right)}\\ \hfill -\frac{(1-\alpha )}{N_s}\sum _{j=1}^{N_s}log\frac{exp\left(z_j^s·z_j^{st}/\tau \right)}{exp\left(z_j^s·z_j^{st}/\tau \right)+{\displaystyle \sum _{i=1}^{N_s}}{\mathbb{1}}_{ji}^{s}exp\left(z_j^s·z_i^{st}/\tau \right)}\end{array}$$

(12)

where ${\mathbb{1}}_{ij}^s$ represents an indicator. When $z_i^{st}$ and $z_i^s$ are the same, ${\mathbb{1}}_{ij}^s$ is 0. When $z_i^{st}$ and $z_i^s$ are not the same, ${\mathbb{1}}_{ij}^s$ is 1, and it is the same as the above when replacing i with j. $N_l$ represents the number of large-scale pixels. $N_s$ represents the number of small-scale pixels. $\tau$ is the temperature parameter. $\alpha \in [0,1]$ is a hyperparameter. Equation (12) encourages the multiscales features $z_i^s$ and $z_i^{st}$ to be closer in the embedding space by reducing the distance between the source domain features and the target domain stylized source domain features in the embedding space and stays away from other negative sample contents.

Based on WildNet in each iteration of training, the projected content features of the target domain $z_{proj}^t$ are filtered at multiple scales and preserved in a dictionary $Q\in {\mathbb{R}}^{C_q\times N_q}$. Here, $C_q$ is the channel number of target domain projected content features, and $N_q$ is the capacity of the dictionary. The dictionary Q stores the content information of multiple scales, which may not exist in the source domain. By learning the information of the effective target domain in the dictionary, the network’s adaptability to the content of the target domain can be enhanced. If domain-general semantic information can be filtered out from the dictionary Q, the network can be more adapted to the target domain. In this work, $z_k^t$ is close to the target domain stylized source domain feature $z_i^{st}$ in the embedding space:

$$\underset{q\in Q}{z_k^t=argmin}\left({∥z_i^{st}-q∥}_2\right)\cup \underset{q\in Q}{argmin}\left({∥z_j^{st}-q∥}_2\right)$$

(13)

By filtering the target domain content information that is similar to the source domain content information from the Equation (5), we can define the pixel-level content adaptation loss function of each pixel on the target domain ${\mathcal{L}}_{TMCAL}$ as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{TMCAL}=-\frac{\alpha }{N_l}\sum _{i=1}^{N_l}log\frac{exp\left(z_i^s·z_k^t/\tau \right)}{exp\left(z_i^s·z_k^t/\tau \right)+{\displaystyle \sum _{j=1}^{N_l}}{\mathbb{1}}_{ij}^{s}exp\left(z_i^s·z_j^t/\tau \right)}\\ \hfill -\frac{1-\alpha }{N_s}\sum _{i=1}^{N_s}log\frac{exp\left(z_i^s·z_k^t/\tau \right)}{exp\left(z_i^s·z_k^t/\tau \right)+{\displaystyle \sum _{j=1}^{N_s}}{\mathbb{1}}_{ij}^{s}exp\left(z_i^s·z_j^t/\tau \right)}\end{array}$$

(14)

Combining the source content adaptation loss in Equation (12) and the target domain content adaptation loss in Equation (12), the CAL loss is defined as follows:

$${\mathcal{L}}_{MCAL}={\mathcal{L}}_{SMCAL}+{\mathcal{L}}_{TMCAL}$$

(15)

4. Experiments and Discussion

4.1. Datasets Description

In the experiment, we utilized two public remote sensing image datasets, Potsdam and Vaihingen, to assess the methodology proposed in this paper. These particular datasets were assembled by the International Society for Photogrammetry and Remote Sensing (ISPRS) [44], and they have been widely recognized and adopted as significant evaluation benchmarks for semantic segmentation tasks that rely on HRIs [28,45,46,47].

Table 1. Synopsis of the characteristics of the Potsdam and Vaihingen datasets.

Description	Datasets
	Vaihingen	Potsdam
Sensor	Leica ADS80	Leica ADS80
Pixel	2000 × 3000	6000 × 6000
GSD	5 cm/pixel	9 cm/pixel
Processed size	512 × 512	896 × 896
Channel compositions	IR-R-G	R-G-B/IR-R-G/IR-R-G-B
Clutter/background samples	0.70%	4.80%
Impervious surface samples	29.30%	29.80%
Car samples	1.40%	1.80%
Tree samples	22.50%	14.50%
Low-vegetation samples	19.30%	21.00%
Buildings samples	26.80%	28.10%

provides a synopsis of the characteristics of the datasets.

The Potsdam dataset serves as a quintessential example of an urban landscape, characterized by expansive building clusters, narrow thoroughfares, and densely organized settlement structures. It encompasses six semantic categories: Clutter/background, Impervious surfaces, Car, Tree, Low vegetation, and Buildings. The Potsdam dataset includes Infrared–Red–Green (IR-R-G) channels, Red–Green–Blue (R-G-B) channels, and Red–Green–Blue–Infrared(R-G-B-IR) channels, among other imaging modes. This article employs the first two imaging modes (IR-R-G/R-G-B). In addition, the Potsdam dataset incorporates 38 True Orthophotos (TOPs), with a resolution of 6000 × 6000 pixels.

The Vaihingen dataset depicts a relatively diminutive village, marked by an abundance of detached edifices and compact multistory constructions. It includes the identical semantic categories as the Potsdam dataset and only contains the Infrared–Red–Green (IR-R-G) channel imaging mode. It includes 33 high-resolution True Orthophotos (VHRTOPs), with sizes ranging from 2000 × 2000 to 2000 × 3000 pixels.

In this paper, images from the Potsdam dataset and the Vaihingen dataset are cut into 896 × 896 and 512 × 512 sized images, respectively, following formula (1). After cutting, the Potsdam dataset totals 1764 images, while the Vaihingen dataset totals 1696 images.

4.2. Experiment Settings

Considering the differences in HRIs as depicted in Figure 8, which are influenced by variations in imaging modes, color saturation, and geographical locations, experiments within this paper are executed in two distinct cross-domain scenarios. The first scenario pertains to cross-domain differences associated with geographical location changes, and the second entails alterations in both geographical location and imaging mode. A detailed overview of the experimental setup can be found in

Table 2. Two different cross-domain scenarios.

Description	Across the Different Region		Across the Different Region and Imaging Sensors
Domain	Source	Target	Source	Target
Location	Vaihingen	Potsdam	Vaihingen	Potsdam
Imaging mode	IR-R-G	IR-R-G	IR-R-G	R-G-B
Labeled	✓		✓

.

For the cross-domain scenario focusing on geographical location changes, the Vaihingen dataset, utilizing the IR-R-G imaging mode, is designated as the source domain, while the Potsdam dataset, operating with the same IR-R-G imaging mode, is identified as the target domain.

For the cross-domain scenario that incorporates changes in both geographical location and imaging mode, the Vaihingen dataset with the IR-R-G imaging mode serves as the source domain, whereas the Potsdam dataset with the R-G-B imaging mode is delineated as the target domain.

The experiment in this paper was implemented utilizing the Pytorch deep learning framework. The operating system used in the experiment is 64-bit Windows 11, with an Intel Xeon(G) W-2223F CPU, 32GB of memory, and an NVIDIA GeForce RTX 3080ti GPU with 12 GB RAM. The total time consumption is about 12 h. The software environment comprised Python 3.8, CUDA 11.3, and cuDNN 8.2. Training and testing were accelerated using the GPU. Specific parameters for this study included a batch size fixed at 2, an initial learning rate of 0.0001, and the employment of the Adam optimizer. The learning rate was adjusted using cosine annealing, and the entire process iterated 100 times.

4.3. Evaluation Metrics

For quantitative evaluation, this paper adopts the Intersection over Union ($IoU$), $F1-scores$, and mean IoU ($mIoU$) metrics, given their comprehensive capacity to assess model performance. These metrics effectively encompass both precision and recall. Therefore, the $IoU$ metric for each category, the $F1-scores$, and the $mIoU$ are utilized in the evaluation of all implemented methodologies.

The $IoU$ is used to measure the overlap between two areas, which is the ratio of the intersection and the union of two areas, and its formula is as follows:

$$IoU=\frac{P_t\cap Y_t}{P_t\cup Y_t}$$

(16)

In the above, $P_t$ represents the pixel-level prediction results on the target domain dataset obtained by the proposed method, while $Y_t$ represents the actual pixel-level labels in the target domain. The $F1-score$ is the harmonic mean of Precision and Recall, and the formula for the $F1-score$ is as follows:

$$F1-score=\frac{2\times Precision\times \mathrm{R}\mathrm{e}call}{Precision\times \mathrm{R}\mathrm{e}call}$$

(17)

4.4. Quantitative Results

In the domain adaptation scenario with geographical location changes, to determine the adaptability of the proposed method in this cross-domain scenario, a comparison was conducted against to advanced domain adaptive semantic segmentation methods, including BiSeNet [48], AdaptSegNet [15], MUCSS [28], and CCDA_LGFA [27]. The former two are UDA methods based on natural images, while the latter two are state-of-the-art methods in the current domain adaptive semantic segmentation methods based on remote sensing images. The segmentation results are shown in

Table 3. Experimental results of cross-domain segmentation under geographical variation (Vaihingen IRRG to Potsdam IRRG).

Methods	Background/ Clutter		Impervious Surface		Car		Tree		Low Vegetation		Buildings		Overall
	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1
Deeplabv3 [40]	9.3	16.86	49.18	65.93	38.51	55.6	7.67	14.24	29.32	45.34	36.96	53.97	28.49	41.99
BiSeNet [48]	29.01	44.97	22.70	36.99	0.69	1.36	41.56	58.71	26.12	41.42	39.61	34.61	23.5	36.34
AdaptSegNet [15]	8.36	15.33	49.55	64.64	40.95	58.11	22.59	36.79	34.43	61.5	48.01	63.41	33.98	49.96
MUCSS [28]	33.48	50.24	50.78	67.35	36.93	54.08	58.69	73.97	40.84	58.00	63.20	77.45	46.36	62.46
CCDA_LGFA [27]	12.31	24.59	64.39	78.59	59.35	75.08	37.55	54.60	47.17	63.27	66.44	79.84	47.87	62.66
Ours	19.68	33.88	66.91	81.17	70.90	83.97	15.23	28.44	48.79	66.58	70.34	84.59	48.64	63.11

. Each column represents the domain adaptive semantic segmentation method, evaluation metrics, each category, and the average of the evaluation metrics. From the results, it is discernible that SCDA enhances the adaptability of the baseline network Deeplabv3 [40] to the above datasets and improves the segmentation effect. At the same time, the $IoU$ and $F1-scores$ of our proposed method are divided into four categories: Impervious surfaces, Cars, Low vegetation, and Buildings, which are slightly higher than the most advanced methods, and the overall segmentation indicators are, respectively, 0.77% and 0.45%, compared with the second-best method CCDA_LGFA [27]. However, the segmentation performance of our method in the two categories of background and trees is not improved, and it is inferior to the CCDA_LGFA [27] and MUCSS [48] methods.

In the domain adaptation scenario where both geographical location and imaging mode are changing dynamically, in order to verify whether the proposed method can be extrapolated to the cross-domain adaptation scenarios in which both geographical location and imaging mode change dynamically, a comparative analysis was conducted against several advanced domain adaptive semantic segmentation methods, including BiSeNet [48], AdaptSegNet [15], MUCSS [28], and CCDA_LGFA [27]. The results are shown in

Table 4. Experimental results of cross-domain segmentation under geographical and imaging mechanism variation (Vaihingen IRRG to Potsdam RGB).

Methods	Clutter/ Background		Impervious Surface		Car		Tree		Low Vegetation		Buildings		Overall
	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1	IoU	F1
Deeplabv3 [40]	6.99	13.04	42.98	60.12	38.01	55.08	0.53	1.06	1.59	3.13	29.09	45.05	19.86	29.58
BiSeNet [48]	23.66	38.26	17.74	30.12	0.99	1.95	32.67	49.24	18.42	31.11	12.64	22.43	17.69	28.85
AdaptSegNet [15]	6.11	11.5	37.66	59.55	42.31	55.95	30.71	45.41	15.1	25.81	54.25	70.31	31.02	44.75
MUCSS [28]	16.09	27.76	47.28	64.21	40.71	57.87	28.96	44.97	43.66	60.79	59.84	74.88	38.06	53.81
CCDA_LGFA [27]	13.27	23.43	57.65	73.14	56.99	72.27	35.87	52.80	29.77	45.88	65.44	79.11	43.17	57.77
Ours	15.76	26.73	63.31	76.02	66.57	80.77	12.63	21.8	40.89	58.22	67.63	81.01	44.38	57.43

. The proposed method emerged as pre-eminent in the segmentation of Impervious surfaces and Cars, thereby attesting that its performance is on par with the most sophisticated cross-domain method and lending additional validation to the effectiveness of our method in semantic segmentation based on HRIs. However, it is inferior to other methods in the four categories of Buildings, Background, Trees, and Low vegetation.

In addition, our model has a parameter count of 40.348M. When given a tensor input of shape (1, 3, 512, 512), the GFLOPs is 138.906G. For a tensor input of shape (1, 3, 896, 896), the GFLOPs increases to 425.397G.

4.5. Visualization Results

Figure 9 and Figure 10, respectively, display the segmentation results of different domain adaptive semantic segmentation methods in the cross-domain scenarios where geographical location changes and both geographical location and imaging mode dynamically change. Each column corresponds to the input image, true labels, segmentation results of Deeplabv3 [40], BiSeNet [48], AdaptSegNet [15], MUCSS [28], CCDA_LGFA [27], and the proposed method in this paper.

SCDA excels in the majority of classes and is generally the best. The most significant issue is the misclassification of the cross-domain class of trees, which leads to poor segmentation results. The reason might be that in the MCAL method, having similar semantic information in the target domain and enhanced source domain content features does not necessarily mean the categories of the two contents are always the same.

4.6. Ablation Study

To ascertain the efficacy of the resized images, the RSFT and MCAL techniques proposed in Section 3, ablation experiments were performed by successively incorporating these three techniques into the baseline. The test results are shown in

Table 5. Ablation study for SCDA. The results are obtained from the task of cross-domain semantic segmentation from Vaihingen IRRG to Potsdam RGB and Vaihingen IRRG to Potsdam IRRG.

Experiment	Vaihingen IRRG to Potsdam RGB		Vaihingen IRRG to Potsdam IRRG
	mIou	F1	mIou	F1
baseline [32]	35.54	52.86	40.25	57.79
resize images	39.32	55.21	45.68	61.56
with RSFT	40.76	55.36	44.98	60.67
with MCAL	42.49	55.92	48.64	63.11
with RSFT and MACL	44.38	57.43	50.59	66.42

. The left columns represent the baseline, resized image, and RSFT and MCAL, respectively. The evaluation metrics use the mean values of $mIoU$ and $F1scores$. As shown in Table 5, in the cross-domain scenario where the geographical location changes (Vaihingen IRRG to Potsdam IRRG), the RSFT and MCAL methods provide the network with the style and content information of the target domain, respectively, which enhances the network’s adaptability to target domain data and significantly improves semantic segmentation performance. Additionally, the “resize images” method involves resizing both the source and target domain images according to their resolution ratios before feeding them into the network. This approach can effectively enhance the network’s adaptability. In cross-domain scenes where both geographical location and imaging methods change, the stylistic differences are significant. The improvement in segmentation performance is the most significant in the MSFT method. After using RSFT alone, the performance of the network drops slightly, indicating that there is overfitting to the source domain data. By adding the MCAL method, the network’s attention to content information at multiple scales in the target domain is increased, and performance is further improved.

Table 6. Evaluation results of RSFT under different hyperparameter settings. The results are obtained from the task of cross-domain semantic segmentation from Vaihingen IRRG to Potsdam RGB and Vaihingen IRRG to Potsdam IRRG.

Hyperparameters Settings		Vaihingen IRRG to Potsdam RGB		Vaihingen IRRG to Potsdam IRRG
		mIou	F1	mIou	F1
RSFT = [1,0,0,0,0]		43.71	53.29	47.15	58.98
RSFT = [1,1,0,0,0]		41.46	52.12	48.96	65.13
RSFT = [1,1,1,0,0]	k = 1.0	44.38	57.43	50.59	66.42
RSFT = [1,1,1,1,0]		43.33	52.51	47.40	59.21
RSFT = [1,1,1,1,1]		42.53	52.26	46.33	58.78
RSFT = [1,1,1,0,0]	k = 0	39.32	55.21	45.68	61.56
	k = 0.5	38.14	53.37	46.30	62.56
	k = 1.0	44.38	57.43	50.59	66.42
	k = 1.5	40.76	54.22	49.02	65.67
	k = 2.0	41.10	55.79	49.43	65.90

shows the effect of the number of RSFTs and the hyperparameter k on domain adaptation performance. In the table, we use a tuple to indicate which layer of ResNet50 uses RSFT. The five elements represent the five layers in ResNet50. When the element is 1, the layer uses RSFT, and when it is 0, it is not used. Experiments show that by using RSFT in the first three layers and k = 1, the network can achieve the best domain adaptation effect, with mIoU being 44.38% and 50.59%, respectively, and F1 being 57.43% and 66.42%, respectively. However, applying RSFT to deeper layers results in a slight decrease in performance. This may be because as layer depth increases, semantic content should be more important than style. The above proof demonstrates the need to appropriately use RSFT in the network to prevent interference with semantic information and help train generalization models by increasing style information in the target domain.

In the

Table 7. Evaluation results of MCAL under different hyperparameter settings. The results are obtained from the task of cross-domain semantic segmentation from Vaihingen IRRG to Potsdam RGB and Vaihingen IRRG to Potsdam IRRG.

Hyperparameter Settings	Vaihingen IRRG to Potsdam IRRG		Vaihingen IRRG to Potsdam RGB
	mIou	F1	mIou	F1
$\alpha$ = 0	48.00	62.80	41.71	52.99
$\alpha$ = 0.5	50.59	66.42	44.38	57.43
$\alpha$ = 1	48.84	63.00	42.06	54.15

, $\alpha =0$ or $\alpha =1$ means only using a single scale (either a large scale or a small scale), and when $\alpha =0.5$, multiple scales are used at the same time. As shown in the table, when using multiple scales, MSAL can provide multiple scales for the network category information to improve the domain adaptation performance of the network.

5. Conclusions

In this paper, we develop a domain adaptive semantic segmentation framework for high-resolution remote sensing images (HRIs). This framework is based on the method utilizing image style transfer and contrastive learning implemented in WildNet, and it is improved according to the characteristics of HRIs, aiming to minimize domain differences between various remote sensing datasets. This enables the transfer of pretrained segmentation models based on source domain remote sensing images to the unlabeled target domain. Firstly, the distribution shift caused by the difference in image style mainly exists in the shallow layer of the network. When this distribution shift is adjusted, the content of the source image will often be lost. SCDA uses the residual style feature transfer (RSFT) method to make the source. The domain image obtains the style information of the target domain image and at the same time reduces the loss of source content information by increasing the source feature information. Secondly, this approach helps prevent the network from overfitting the content of the source domain. Additionally, high-resolution remote sensing images (HRIs) are characterized by significant scale variations across different object categories. We propose a multiscale content adaptive learning (MCAL) method to enable the network to learn the category information of different scales in the target domain that is similar to the source domain, so as to achieve the effect of adapting to the content of the target domain.

SCDA is a simple and effective framework for domain adaptive semantic segmentation. However, SCDA only reduces the domain difference from the content and style of the image, without considering other information of the image, such as layout information, category depth information, etc. Furthermore, SCDA excels at identifying both large-scale and small-scale feature categories, but it performs the segmentation effect poorly on other features. Therefore, how to combine SCDA with research methods that utilize image layout and category depth information, as well as increase the attention of intermediate-scale object categories, is a potential topic for future work.