Learn to Few-Shot Segment Remote Sensing Images from Irrelevant Data

Abstract

Few-shot semantic segmentation (FSS) is committed to segmenting new classes with only a few labels. Generally, FSS assumes that base classes and novel classes belong to the same domain, which limits FSS’s application in a wide range of areas. In particular, since annotation is time-consuming, it is not cost-effective to process remote sensing images using FSS. To address this issue, we designed a feature transformation network (FTNet) for learning to few-shot segment remote sensing images from irrelevant data (FSS-RSI). The main idea is to train networks on irrelevant, already labeled data but inference on remote sensing images. In other words, the training and testing data neither belong to the same domain nor category. The FTNet contains two main modules: a feature transformation module (FTM) and a hierarchical transformer module (HTM). Among them, the FTM transforms features into a domain-agnostic high-level anchor, and the HTM hierarchically enhances matching between support and query features. Moreover, to promote the development of FSS-RSI, we established a new benchmark, which other researchers may use. Our experiments demonstrate that our model outperforms the cutting-edge few-shot semantic segmentation method by 25.39% and 21.31% in the one-shot and five-shot settings, respectively.

1. Introduction

Deep learning-based semantic segmentation is widely used in remote sensing [1,2]. Generally, semantic segmentation provides pixel-level classification for downstream applications, which is a fundamental computer vision task. Many models have been built by adopting fully convolutional networks and have achieved satisfactory results [3,4,5]. On this basis, novel modules such as encoder–decoder [6,7], dilated convolution [8], and atrous spatial pyramid pooling [9] have been proven to be effective. Indeed, pre-trained backbones, such as the ResNet [10] and VGG [11], have been utilized in various semantic segmentation models [8,9,10] for feature extraction, which has gradually become a stereotype. By contrast, VIT [12], SETR [13], and SegFormer [14] divide images into a patch sequence. In these works, transformers were used to extract image features [15,16], and their results surpassed traditional methods to some extent.

However, a large dataset is needed during training, thus limiting semantic segmentation’s application in a broader field. FSS has been proposed [17,18] to solve this limitation. Unlike supervised learning-based methods, FSS requires only a few annotations to segment new classes.

There are no overlap categories between training and inference for FSS [19], which is the main difference between few-shot semantic segmentation and semantic segmentation. Most FSS methods follow meta-learning [20], where episodes are formed by image and label pairs [17,21,22] to mimic few-shot scenes. Currently, FSS is mainly divided into two groups: relation-based methods and metric-based methods. Among them, relation-based methods [18,19,22,23,24] share the same backbone and freeze their parameters during training. The main idea is to design a practical decoder to compare the query and support data. In contrast, metric-based methods [21,25] tend to develop effective encoders to separate foreground and background classes. Furthermore, some works [26,27] bring transformers into FSS tasks with excellent results. As for remote sensing, it is time-consuming to obtain numerous annotated data. Therefore, some works [28,29,30] have aimed to reduce the need for annotations or use semi-supervised methods [31] to handle unknown categories.

Generally, FSS is conducted in a cross-validation manner with four splits [32]. Although there is no class overlap between the training and testing sets, they belong to the same domain. For example, there are 20 categories in PASCAL-5ⁱ [17], but each class’s pixel distribution is similar, called the in-domain dataset. Additionally, although FSS is named “few-shot”, a large, labeled dataset is still needed during training, which is inconvenient for remote sensing. We aim to train such a network on a large but irrelevant dataset and to predict masks on remote sensing images.

This work extends few-shot semantic segmentation to a new task called FSS-RSI. As we know, FSS’s training and testing sets contain different categories within the same domain. By contrast, FSS-RSI’s data differ not only in classes but also in image acquisition sensor and pixel distributions, which belong to irrelevant/cross-domain data.

To achieve the goal of FSS-RSI, the FTNet was designed. The meta-learning method [20] was adopted to train our network. Specifically, the FTNet transforms the support and query features into a domain-agnostic space with the learnable FTM. In this way, the gap between the support data and the query data is narrowed. In addition, the HTM is used to parse the correlations between the support and query features, which fully promotes the fitting capability of the support and query features.

To validate our network and provide convenience for other researchers, we established a new benchmark. The images used came from four different datasets, DeepGlobe [33], Potsdam [34], Vaihingen [35], and AISD [36], which were captured by satellites or drones. All four datasets are typical in remote sensing and contain commonly used categories in engineering. We combined these datasets into an FSS-format dataset and used them as a benchmark for FSS-RSI.

PASCAL-5ⁱ and our benchmark were used for our experiments. The FTNet achieves comparable accuracy to the cutting-edge method on the in-domain dataset. As for FSS-RSI, the FTNet performs at an absolute advantage. The mIoUs in the one-shot and five-shot settings were 25.39% and 21.31%, respectively, higher than the state-of-the-art (SOTA) method.

In summary, our main contributions lie in the following aspects:

• We extend the FSS to FSS-RSI, which aims to utilize irrelevant domain data to guide the segmentation of remote sensing images.
• A new benchmark is proposed. This benchmark may promote the development of FSS-RSI and serve as a tool for researchers.
• We propose an effective network with the FTM and the HTM. Our method significantly outperforms the cutting-edge few-shot semantic segmentation method in the FSS-RSI task.

2. Method

2.1. Problem Setting

Table 1. Differences between SS, FSS, and FSS-RSI.

Task	Data Source	Categories	Example
			Training Pair		Testing Pair
SS	${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}={\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$	${\mathit{C}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\cap {\mathit{C}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}=1$
FSS	${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}={\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$	${\mathit{C}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\cap {\mathit{C}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}=\varnothing$
FSS-RSI	${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\ne {\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$	${\mathit{C}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\cap {\mathit{C}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}=\varnothing$

shows the differences between semantic segmentation (SS), FSS, and FSS-RSI. We define the training and testing data as domains ${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and ${\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ and their semantic categories as ${\mathit{C}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and ${\mathit{C}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$, respectively.

For SS, ${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and ${\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ belonged to the same domain, specifically remote sensing, in our task. The training and testing categories were the same. That is, SS only handles classes that have appeared in training. For FSS, both ${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and ${\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ were derived from remote sensing, but their categories did not overlap. That is, FSS can process classes with no appearance during training. FSS-RSI was the most challenging task, with ${\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and ${\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ originating from different domains. The two domains have different classes and pixel distributions, which we call irrelevant data.

In the FSS-RSI task, episodes [18] were used to mimic few-shot scenes. Each episode consisted of a query set $\mathit{Q}=\left\{\left(I^q,M^q\right)\right\}$ and a support set $\mathit{S}={\left\{\left(I_i^s,M_i^s\right)\right\}}_{i=1}^K$. In our study, $\left(·,·\right)$ represents image pairs consisting of RGB images and corresponding masks. $I^s,I^q\in {ℝ}^{H\times W\times 3}$ denotes the RGB images. $M^{\mathrm{s}},M^q\in {ℝ}^{H\times W}$ represents their masks. K means K pairs of images and masks were used, which we call the K-shot. ${\mathit{S}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and $I^q\in {\mathit{Q}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ are the inputs during training. The proposed network predicts a binary mask to compute loss with $M^q\in {\mathit{Q}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$. In the testing phase, the network predicted a new mask with ${\mathit{S}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ and $I^q\in {\mathit{Q}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ as the inputs. It should be noted that ${\mathit{S}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}{,\mathit{Q}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}\subset {\mathit{D}}_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$, and ${\mathit{S}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}{,\mathit{Q}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}\subset {\mathit{D}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$, respectively.

2.2. Model

The FTNet is designed to deal with FSS-RSI tasks. As shown in Figure 1, the network is built in a meta-learning manner [20]. Specifically, we used ResNet50 [10], which was pre-trained by ImageNet [37], as the backbone and froze its parameters during training. The query and support branches share the same backbone to extract multi-layered features. Furthermore, a prior mask [18] from high-level feature maps was introduced to strengthen the connection between the query and support data. It should be noted that support masks are important for FSS. Therefore, the FTNet adopted a support mask several times to enhance its guidance for the query images. In particular, the FTM is designed to transform the middle query feature, support feature, and prototype into a domain-independent, high-level feature space called the feature anchor. The FTNet achieves better performance when processing FSS-RSI tasks with the FTM. In addition, we input the fused query feature, support feature, and mask into the HTM, which enhanced information fusion within and between features. Figure 1 shows the model architecture of a one-shot structure, which can be easily expanded to a five-shot structure.

2.2.1. Feature Extraction

During the training phase, we froze the backbone’s parameters, which was the strategy employed by other methods [18,19,26]. There are five stages included in ResNet50. The FTNet mainly adopts feature maps for stage 3, stage 4, and stage 5, which are denoted as $f_{s3}$, $f_{s4}$, and $f_{s5}$. In order to enhance the performance of high-level feature maps, PPM [38] was used to refactor stage 5. Thus, we obtained $f_{s6}$ as the following:

$$f_{s6}=\underset{i=\mathrm{1,2},\mathrm{3,4}}{{\mathcal{F}}_{\mathrm{cat}}}\left({\mathcal{U}}^i\left({\mathcal{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}^i\left({\mathcal{F}}_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}^i\left(f_{s5}\right)\right)\right)\right)$$

(1)

where ${\mathcal{F}}_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}$ means the average pooling and ${\mathcal{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}$ denotes the convolution, followed by the BatchNorm [39] and ReLU functions. $\mathcal{U}$ is the upsampling and ${\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}$ represents the concatenation. Pyramids with the sizes 1 × 1, 2 × 2, 3 × 3, and 6 × 6 were used, and $i$ is the level of pyramid. $f_{s4}$, $f_{s5}$, and $f_{s6}$ were resized to be the same as $f_{s3}$.

According to [22,40,41,42], middle-level features contain more semantic information, such as the outline and color. Therefore, the FTNet concatenates $f_{s3}$ and $f_{s4}$ to obtain a middle-level feature map:

$$f_m^s={\mathcal{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}\left({\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}\left(f_{s3}^s,f_{s4}^s\right)\right)\in {ℝ}^{c\times h\times w}$$

(2)

$$f_m^q={\mathcal{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}\left({\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}\left(f_{s3}^q,f_{s4}^q\right)\right)\in {ℝ}^{c\times h\times w}$$

(3)

where $f_m^s$ and $f_m^q$ are the middle-level features of the support and query data. ${\mathcal{F}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}$ and ${\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}$ are the same as in Function (1) with different parameters. Furthermore, we calculated the prototype using the support mask and $f_m^s$ as the following:

$$p^s={\mathcal{F}}_{\mathrm{g}\mathrm{a}\mathrm{p}}\left(f_m^s\odot \mathcal{R}\left(M^s\right)\right)\in {ℝ}^c$$

(4)

where $\odot$ means the Hadamard product, and $\mathcal{R}$ represents the operation to reshape the initial query mask from ${ℝ}^{H\times W}$ to ${ℝ}^{c\times h\times w}$, with the same size as $f_m^s$. ${\mathcal{F}}_{\mathrm{g}\mathrm{a}\mathrm{p}}$ is the global average pooling to reshape the feature map from ${ℝ}^{c\times h\times w}$ to ${ℝ}^{c\times 1}$.

In addition, the prior mask generated by the high-level feature boosts performance in a training-class-insensitive way [18]. We used $f_{s4}$ and $f_{s5}$ to generate prior masks and merge them as the following:

$$M^p={\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}\left(\mathcal{P}\left(f_{s4}\right),\mathcal{P}\left(f_{s5}\right)\right)$$

(5)

where $\mathcal{P}$ denotes the generation of a prior mask $M^p$.

2.2.2. Feature Transformation Module

Motivation. Features extracted by convolutions have an excellent characterization within category and domain. As for FSS-RSI, the parameters learned during training tend to segment the categories that appear during training. Therefore, the FTNet transforms features into a space independent of classes and domains. This strategy reduces the influence of the source domain and training data. Inspired by a task-adaptive feature transformer (TAF) [43], we propose a simple learnable transformation matrix that transforms $f_m^s$, $f_m^q$, and $p^s$ to a domain-agnostic space.

For the feature matrix $\mathit{F}$, the goal was to find a matrix $\mathit{T}$ that transforms $\mathit{F}$ to a domain-independent feature matrix $\mathit{W}$, called the feature anchor, as the following:

$$\mathit{TF}=\mathit{W}$$

(6)

In general, $\mathit{F}$ is a non-square matrix with no inverse. One solution is to calculate the pseudo-inverse [43] of $\mathit{F}$ is ${\mathit{F}}^{+}={\left\{{\mathit{F}}^T\mathit{F}\right\}}^{-1}{\mathit{F}}^T$. Thus, the transformation matrix was obtained as the following:

$$\mathit{T}={\mathit{WF}}^{+}$$

(7)

The parameters of $\mathit{W}$ were initialized randomly and changed with the gradient’s backpropagation; therefore, the matrix $\mathit{T}$ was constantly optimized.

Specifically, for the prototype $p^s$, we obtained $p_{\mathrm{n}\mathrm{e}\mathrm{w}}^s=\mathit{T}{p}^s$. As for $f_m^s$, we needed to transform it as the following:

$$f_m^{s^{\prime }}=\mathcal{R}\left(f_m^s\right)$$

(8)

where $\mathcal{R}\left(·\right)$ represents the reshape operation: ${ℝ}^{c\times h\times w}\to {ℝ}^{c\times (h\times w)}$. We used the transformation matrix to multiply Formula (8) to obtain $f_{m,\mathrm{n}\mathrm{e}\mathrm{w}}^{s^{\prime }}=\mathit{T}{f}_m^{s^{\prime }}$. Furthermore, $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^{s^{\prime }}$ was transformed to the original shape as $f_m^s$, that is,

$$f_{m, \mathrm{new}}^s={\mathcal{R}}^{-1}\left(f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^{s^{\prime }}\right)$$

(9)

where the inverse reshape is included. The same operation was performed for $f_m^q$. Finally, $p_{\mathrm{n}\mathrm{e}\mathrm{w}}^s$, $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^s$, and $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^q$ were obtained. They are domain-agnostic features. In other words, the gap between $p_{\mathrm{n}\mathrm{e}\mathrm{w}}^s$ and $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^q$ and the gap between $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^s$ and $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^q$ were significantly reduced. We provide a more detailed explanation in Appendix A.

We further merged $p_{\mathrm{n}\mathrm{e}\mathrm{w}}^s$, $f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^q$, and $M^p$ to obtain $f_{\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{g}\mathrm{e}}^q$ as the following:

$$f_{\mathrm{merge}}^q={\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}\left(f_{m, \mathrm{n}\mathrm{e}\mathrm{w}}^q,M^p,\mathcal{J}\left(p_{\mathrm{n}\mathrm{e}\mathrm{w}}^s\right)\right)$$

(10)

where $\mathcal{J}\left(·\right)$ is the repeat operation: ${ℝ}^{c\times 1}\to {ℝ}^{c\times h\times w}$. It should be noted that only the parameters of $\mathit{W}$ were learnable. The transformation matrix was calculated directly. This method does not add too many parameters.

2.2.3. Hierarchical Transformer Module

Motivation. A transformer is used in many works to extract features [12,13,14]. It establishes relationships within and between features to mine the connections between image blocks. As illustrated in [27], prototype-based FSS models are committed to providing class-wise clues rather than pixel-wise clues. We adopted self- and cross-attention paradigms to mine deep matching correlations.

To strengthen the support data’s performance, we again used support masks. We define $Q=W^q{f}^q$, $K=W^k\left(f^s\odot M^s\right)$, and $V=W^v\left(f^s\odot M^s\right)$. We followed the usual transformer calculation, which is formulated as the following:

$$Trans\left(Q,K,V\right)=\mathrm{softmax}\left(\frac{Q{K}^T}{\sqrt{d}}\right)V$$

(11)

where $W^q$, $W^k$, and $W^v$ are the learnable parameters, and $d$ is the hidden dimension. Equation (11) is a general form of cross-attention. As shown in Figure 2, when the features $f^s$ in $K$ and $V$ are replaced with $f^V$, Equation (11) represents the self-attention manner, which represents the relationship among query features. The main task of the HTM is to calculate an informative query feature. Thus, we only performed self-attention within the query path in a standard multi-head manner [12]. In addition, a cross-attention layer follows self-attention. Similar to [27], $Q$ was obtained from the query features, and $K$ and $V$ were obtained from the support features. Inspired by the ResNet [10] and SegFormer [14], we design a hierarchical architecture with four scale blocks. Each block contains self- and cross-attention, followed by downsampling. At the end of the HTM process, we concatenated the four blocks after scaling them to the same resolution. In a nutshell, our model extracts abundant information within query features and obtains pixel-wise matching correlations using a cross-attention layer. We demonstrate the role of the HTM in mining this matching relationship, as detailed in Appendix B.

Finally, the FTNet adopts a simple decoder to generate predictions, mainly including stacked convolutional layers and upsampling. Because FSS is a binary classification task, binary cross-entropy loss (BCE) was used to optimize our model, which is formulated as the following:

$$L=\frac{1}{n}\sum _{i=1}^n\mathrm{B}\mathrm{C}\mathrm{E}\left({\mathcal{P}}_i^q,M^q\right)$$

(12)

where $n$ is the number of episodes in each batch and $\mathcal{P}$ is the prediction of the query image.

2.2.4. Extension to K-Shot

Extending our model to K-shot (K > 1) format was straightforward. The K-shot setting means that there are K support pairs in one episode. Specifically, the support pair is $\mathit{S}={\left\{\left(I_i^s,M_i^s\right)\right\}}_{i=1}^K$ and the query pair is still $\mathit{Q}=\left\{\left(I^q,M^q\right)\right\}$. In order not to change the model’s settings, we concatenated the K groups in the channel dimension directly as the following:

$$\mathit{S}=\left\{\left({{\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}\left(I_i^s\right)}_{i=1}^K,{{\mathcal{F}}_{\mathrm{c}\mathrm{a}\mathrm{t}}\left(M_i^s\right)}_{i=1}^K\right)\right\}$$

(13)

Therefore, the FTNet obtained an input with the same structure as a one-shot structure with simple convolutions.

2.3. Benchmark

The FSS-RSI benchmark was derived from four datasets of remote sensing, including DeepGlobe [33], Potsdam [34], Vaihingen [35], and AISD. These datasets differ from the commonly used FSS datasets regarding their pixel distributions and categories.

DeepGlobe [33] consists of natural landscape images taken by satellites. This dataset is annotated into seven categories: unknown, urban, aquatic, agricultural, forested, barren, and rangeland areas. The ground sampling distance was 50 cm. However, only 803 training images were labeled with a size of 2448 × 2448. Fortunately, this dataset was adopted as a tool for FSS-RSI, so DeepGlobe’s training set could meet the need. Specifically, we divided each image into 36 equal blocks and resized them to 400 × 400. The category “unknown” was set as the background. Images that contained only a single category were filtered out. Finally, we obtained 9175 pairs containing six categories. We named this dataset FSS-RSI-DeepGlobe; some samples are shown in Figure 3a.

Potsdam [34] was captured over Potsdam in Germany by aerial cameras. This dataset is annotated into six categories: clutter, tree, low vegetation, building, car, and impervious surface. The ground sampling distance of the images was 5 cm. We removed the category “car” because of the overlap with the source domain used in our work. The buildings in Potsdam are scattered and the category distribution is more balanced. Potsdam contains 38 image patches. The size of all images is 6000 × 6000. Each image was divided into 225 equal pieces. Similar to the DeepGlobe dataset, we removed pairs with a single category. Finally, we obtained 1896 pairs containing five categories. We named this dataset FSS-RSI-Potsdam; some samples are shown in Figure 3b.

Vaihingen [35] was captured over Vaihingen in Germany by aerial cameras and includes five categories (after removing “car”), like Potsdam. The ground sampling distance of the images was 9 cm. Unlike Potsdam, the class distribution is more compact, with dense settlement structures, narrow streets, and large buildings. Vaihingen contains 33 patches of different sizes. We resized the images to 2800 × 2000 pixels and divided each image into 35 equal pieces. We removed images with a single category. As already known, episodes needed to be built in each category. However, the filtered Vaihingen dataset contains only 6 “clutter” samples, which was insufficient to build a rich episode. Thus, images containing the category “clutter” were discarded. Finally, we obtained 308 pairs containing four categories. We named this dataset FSS-RSI-Vaihingen; some images are shown in Figure 3c.

AISD [36] is an aerial image segmentation dataset obtained using the OpenStreetMap [44,45,46] and Google Maps [47]. AISD covers parts of different cities, of which Berlin was selected for our experiment. There are only two categories in AISD: road and building. However, their appearance is very similar within and between the two categories. Thus, we believe AISD is a challenging task for FSS. AISD contains 200 patches of the same size, at 2611 × 2453. We resized the images with 2800 × 2400 pixels and divided each image into 42 equal pieces. We removed images with a single category similar to the other three datasets. Finally, we obtained 5640 pairs containing two categories. We named this dataset FSS-RSI-AISD; some samples are shown in Figure 3d.

Table 2. Details of our benchmark. The FID was calculated between each dataset and PASCAL-5ⁱ.

Dataset	Numbers	Classes	FID
PASCAL-5i	17,125	person, bird, dog, cat, cow, chair, dining table, potted plant, sheep, horse, airplane, bicycle, boat, car, bottle, sofa, tv/monitor bus, motorbike, and train	–
FSS-RSI-DeepGlobe	9175	agricultural, forested, barren, urban, rangeland, and aquatic areas	186.55
FSS-RSI-Potsdam	1896	clutter, tree, low vegetation, building, and impervious surface	151.86
FSS-RSI-Vaihingen	308	tree, low vegetation, building, and impervious surface	328.08
FSS-RSI-AISD	5640	building and road	194.90

provides a detailed description of PASCAL-5ⁱ and our benchmark.

As shown in Figure 4, we further calculated the pixel distribution of each category. The pixel distribution was relatively balanced, except for “urban areas” in FSS-RSI-DeepGlobe. Because of the richness of this dataset, we kept the class “urban areas”.

As shown in Table 2, the Fréchet inception distance (FID) [48] was reported to measure the different data distributions between our benchmark and PASCAL-5ⁱ. The FID is the Fréchet inception distance between the Gaussians obtained from the distributions of two datasets:

$$d^2\left(\left({\mu }_1,c_1\right),\left({\mu }_2,c_2\right)\right)={‖{\mu }_1-{\mu }_2‖}_2^2+Tr\left(c_1+c_2-2{\left(c_1{c}_2\right)}^{\frac{1}{2}}\right)$$

(14)

where $\left({\mu }_1,c_1\right)$ and $\left({\mu }_2,c_2\right)$ are means and covariances of the two distributions, and Tr is the matrix trace. The larger the FID, the greater the difference between the datasets, and vice versa.

As shown in

Table 3. FIDs of different PASCAL-5ⁱ splits.

Split-1	Split-2	FID
Fold0	Fold1 + Fold2 + Fold3	79.47
Fold1	Fold0 + Fold2 + Fold3	47.47
Fold2	Fold0 + Fold1 + Fold3	41.45
Fold3	Fold0 + Fold1 + Fold2	61.48

, the same method was used to calculate the FID within PASCAL-5ⁱ. We followed the strategy of FSS, that is, using a standard cross-training manner. Specifically, the FID was calculated between each fold and the other three folds. Compared to the data within PASCAL-5ⁱ, the distribution gaps between our benchmark and PASCAL-5ⁱ were vast, where the FID was more than twice that of the in-domain data. In particular, the FID of FSS-RSI-Vaihingen was 328.08. Therefore, it is further proven that our benchmark and PASCAL-5ⁱ belong to different domains.

2.4. Experiments

2.4.1. Datasets and Metric

Datasets. PASCAL-5ⁱ is the irrelevant domain training set created from PASCAL VOC 2012 [49] with SDS [50] augmentation. The benchmark we proposed is the testing set in the remote sensing domain.

Metric. The mean intersection over union (mIoU) [19,26] was adopted in our experiment, as a standard metric in semantic segmentation. The IoU is defined as the following:

$$\mathrm{IoU}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{F}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{P}}$$

(15)

where $\mathrm{F}\mathrm{N}$,$\mathrm{F}\mathrm{P}$, $\mathrm{a}\mathrm{n}\mathrm{d} \mathrm{T}\mathrm{P}$ represent the number of false negatives, false positives, and true positives of the predictions, respectively. Furthermore, the mIoU is the average IoU of all categories.

2.4.2. Training and Testing Strategy

We used a generic meta-learning manner for training and testing. That is, each batch contained an episode. Unlike FSS, the entire PASCAL-5ⁱ dataset with all four splits was used as the training data. Indeed, we use a supervised learning strategy and fixed the backbone’s pre-training parameters during training. The Adam optimizer was adopted with a learning rate of 10⁻⁴, and the weight decay was 0.01. Furthermore, the size of the images was reshaped to 400 × 400 pixels, which was followed by random scaling, rotation, and cropping. A mini-batch of 16 was utilized in the experiment. We trained each model on four 2080 Ti GPUs with 50 epochs.

The test was performed on a single GPU. It should be noted that we tested the benchmark with the model trained on the PASCAL-5ⁱ without transfer. The mIoU was calculated for each dataset based on the average of 5 runs with different random seeds. A total of 600 tasks were contained in each run.

3. Result

3.1. Models for Comparison

To prove the performance of the FTNet, we selected several representative FSS models, including RPMMs [23], the PFENet [18], HSNet [24], BAM [19], and HDMNet [26]. Among them, RPMMs and the PFENet are classic prototype-based architectures, especially the PFENet, which is the most similar model to the FTNet. The HSNet uses 4D convolution to push meta-learning-based FSS to new heights. BAM and the HDMNet are cutting-edge in-domain FSS methods based on meta-learning and base-learning. For the RPMMs, PFENet, and HSNet, their released codes were used with the same settings. For BAM and the HDMNet, their meta paths were adopted, as there were no base classes in our benchmark. The testing method was exactly the same as ours.

3.2. Main Results

The results are shown in

Table 4. The mIoUs (%) of different methods experimented on our benchmark. The best results are denoted in bold. Suboptimal results are underlined.

Method	FSS-RSI-DeepGlobe		FSS-RSI-Potsdam		FSS-RSI-Vaihingen		FSS-RSI-AISD		Average
	1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot
PFENet	3.97	2.42	6.26	4.84	12.58	12.29	25.03	25.18	11.96	11.18
RPMMs	10.66	13.17	6.76	7.56	16.24	16.45	25.12	23.86	14.70	15.26
HSNet	31.78	35.38	17.94	19.31	21.78	24.16	24.47	26.41	23.99	26.32
BAM	12.23	26.65	12.53	17.37	17.19	26.98	23.35	27.41	16.33	24.60
HDMNet	16.68	17.57	11.27	23.49	21.80	25.53	21.04	25.74	17.70	23.08
FTNet	41.37	44.57	24.68	25.58	24.75	26.44	29.51	31.14	30.08	31.93

. The mIoU of the FTNet significantly exceeded that of the existing FSS model, including the SOTA model. Specifically, on the FSS-RSI-DeepGlobe dataset, the FTNet outperformed the suboptimal HSNet by 30.18% and 25.98% in the one-shot and five-shot settings, respectively. On the FSS-RSI-Potsdam dataset, the FTNet outperformed the suboptimal method by 37.57% and 8.90% in the one-shot and five-shot settings, respectively. On the FSS-RSI-AISD dataset, the FTNet outperformed the suboptimal methods by 17.48% and 13.61% in the one-shot and five-shot, respectively. In addition, our one-shot result was 13.53% higher than the HDMNet on the FSS-RSI-Vaihingen dataset. But the FTNet obtained a value that was 2.00% lower than BAM in the five-shot setting. For the mean results of all datasets, the mIoU significantly exceeded the suboptimal model, which was 25.39% and 21.31% higher than the HSNet in the one-shot and five-shot settings, respectively.

As can be seen, the PFENet achieved an mIoU that was 10% lower on the FSS-RSI-DeepGlobe and FSS-RSI-Potsdam datasets, especially for the value of only 2.42% in the FSS-RSI-DeepGlobe’s five-shot setting. This proves that the PFENet had no FSS-RSI performance on these two datasets. However, as illustrated in Appendix C, the PFENet’s performance within the domain was good. The same is true for RPMMs on the FSS-RSI-Potsdam dataset. As already known, the HDMNet is a cutting-edge FSS model, even with only its meta branch. The results can be seen in Appendix C. However, for the FSS-RSI task, the HDMNet did not perform well. The obtained mIoUs were 17.70 and 23.08 in the one-shot and five-shot settings, which were 41.16% and 17.56% lower than the FTNet, respectively. Performing slightly worse than our method, the HSNet performed suboptimally on FSS-RSI-DeepGlobe’s one-shot and five-shot settings and on FSS-RSI-Potsdam’s one-shot setting. We found that except for the PFENet, the five-shot results of all models were better than the one-shot results. This phenomenon indicates that when there are more support data, FSS-RSI performs better, which is similar to FSS. This experiment showed that the FTNet achieved the best result in FSS-RSI with absolute advantages over the other cutting-edge FSS methods.

Some qualitative results of our methods are shown in Figure 5. Classes with regular shapes, such as buildings in the FSS-RSI-Potsdam and FSS-RSI-Vaihingen datasets, obtained better results. However, FSS-RSI is challenging for irregular categories, such as trees and low vegetation. In particular, FSS-RSI did not work well for categories with similar appearances, such as the barren and rangeland areas in the FSS-RSI-DeepGlobe dataset and all categories in the FSS-RSI-AISD dataset. These challenging cases also need to be solved using semantic segmentation. Indeed, compared to commonly handled categories, such as cars, people, and animals, it is more difficult to segment remote sensing images. This issue is exactly the intractable part that FSS needs to solve.

4. Discussion

4.1. Ablation Study

To prove the effectiveness of the FTNet, we carried out an ablation study. The mIoU was selected as the metric. Our baseline was that the architecture removed the HTM and FTM. To further justify the effectiveness of our HTM, we adopted a vanilla transformer module (VTM) for comparison. There were four repeat blocks in the VTM without concatenation; each block was the same as the first in the HTM.

Effects of the HTM and FTM. 

Table 5. The mIoUs (%) of the ablation study. The best results are denoted in bold. The baseline is the architecture that removed the FTM and HTM.

Method	FSS-IRS-DeepGlobe		FSS-IRS-Potsdam		FSS-IRS-Vaihingen		Average
	1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot
Baseline	19.29	30.56	11.56	17.84	21.69	21.72	17.51	23.37
Baseline + FTM	31.62	33.80	22.57	20.86	24.73	27.61	26.31	27.42
Baseline + HTM	38.41	43.40	23.03	23.99	23.31	25.41	28.25	30.93
Baseline + VTM	21.23	–	20.27	–	23.16	–	21.55	–
Baseline + HTM + FTM	41.37	44.57	24.68	25.58	24.75	26.44	30.27	32.20

shows the results of the five forms. They were the baseline, adding the FTM, adding the HTM, adding the VTM, and adding both the HTM and FTM. As illustrated, the mIoU of the baseline was similar to that of BAM [19] and the HDMNet [26]. After adding our tailored modules, the FTNet’s performance was significantly boosted.

Compared to the baseline, the mIoUs in the one-shot and five-shot settings were improved by 50.26% and 17.33%, respectively, after adding the FTM. Furthermore, adding the HTM improved the result by 61.34% and 32.35%, respectively. What surprised us the most is that our complete structure with both the FTM and HTM achieved the highest mIoUs, which were 72.87% and 37.78% higher than the baseline in the one-shot and five-shot settings, respectively. We note that adding the FTM obtained a result comparable to our complete architecture in the one-shot setting and a higher result in the five-shot setting on the FSS-IRS-Vaihingen dataset. These results prove the effectiveness of the proposed modules.

As illustrated in the HDMNet [26], a transformer module follows a backbone similar to ours. Unlike the HDMNet, we added the HTM to fuse low- and high-level information after the prior mask and prototype. Thus, our model’s performance was improved. As shown in Table 5, adding the VTM raised the mIoU from 17.51 to 21.55 in the one-shot setting. However, our HTM’s result was 31.09% higher than the VTM’s result in the one-shot setting. Furthermore, our device was out of memory when we trained the architecture with the VTM in the five-shot setting. Therefore, we could not collect the five-shot result using the VTM. The experimental results justify that our HTM is more effective than the VTM.

The visualization results of the five forms are shown in Figure 6. They are the results based on FSS-RSI-DeepGlobe in the one-shot setting. It is important to note that these qualitative results were unstable across different test rounds, and we consider the quantitative results in Table 5 on the entire dataset to be more reliable.

Effects of different classes. We counted the mIoUs of each class on the whole benchmark, and the results are shown in Figure 7. To sum up, the FTNet had an unbalanced accuracy for each category. On the FSS-RSI-DeepGlobe dataset, the FTNet had a higher mIoU for agricultural, forested, and barren but a lower mIoU for the remaining three categories. On the FSS-RSI-Potsdam dataset, the category with the highest mIoU was “building”, which was 156.40% higher than “clutter”. On the FSS-RSI-Vaihingen dataset, the best class was “tree”, which was 206.92% higher than the categories of “impervious surface”. On the FSS-RSI-AISD dataset, the mIoU of “building” was higher. We believe that “building” had a more regular shape relative to “road”, which was more conducive to the prediction of the network.

When combined with the pixel distribution in each category in Figure 4, the segmentation results of FSS-RSI-DeepGlobe and FSS-RSI-Potsdam are independent of the number of pixels. On the FSS-RSI-Vaihingen dataset, the smallest pixel ratio was obtained for “impervious surface”, and the mIoU in this category was also the smallest. On the FSS-RSI-AISD dataset, the ratio of pixels was the opposite of the ratio of mIoUs. We do not have enough evidence to prove that the accuracy was related to the pixel ratio. Balancing the number of pixels in our benchmark and improving the semantic segmentation accuracy of each category will need to be considered in the future.

4.2. Limitations

This work introduces FSS into the field of remote sensing image segmentation. Our model achieved an absolute advantage over other SOTA methods, as our experiment shows. The results prove the effectiveness of our approach. However, we need to note that in the FSS-RSI task, the mIoU was only about 30%, which is still far from actual application needs. On the one hand, this phenomenon is attributable to the fact that FSS-RSI is a very challenging task. The training data and the test data were irrelevant. On the other hand, most of the categories in our benchmark did not have fixed shapes, such as low vegetation, impervious surface, and agriculture, which were difficult even for generic semantic segmentation [5,9]. In addition, FSS-RSI did not work well with categories with similar appearances, such as the barren and rangeland areas in the FSS-RSI-DeepGlobe dataset and all categories in the FSS-RSI-AISD dataset.

Some previous works on remote sensing images using FSS have achieved high accuracy [51,52]. Their training and testing data came from the same remote sensing dataset, which was different from our task. And the categories they contained were common categories with fixed shapes, such as airplanes, ships, or cars. In order to extend FSS-RSI to a broader range of applications, some innovative works need to be proposed. For example, tailored models must be designed for categories with no fixed shape to improve their segmentation accuracy. Moreover, FSS-RSI combined with the usual semantic segmentation, which simultaneously segments novel and known categories, would be promising future work.

5. Conclusions

To address the limitations of FSS for remote sensing, we extended the task to a new field called FSS-RSI. Specifically, we established a novel benchmark for evaluating FSS-RSI, which may be useful for other researchers. Moreover, we propose the FTNet with an FTM and an HTM. The FTM transforms the support feature, query feature, and prototype into a domain-agnostic space called the feature anchor. The HTM establishes abundant matching correlations between the support and query patches. In this way, our model can process remote sensing data with data from irrelevant domains.

Experiments were conducted on PASCAL-5ⁱ and our benchmark. The FTNet achieved comparable accuracy to the cutting-edge methods on the in-domain data but obtained an absolute advantage on the FSS-RSI data. The proposed method outperformed the suboptimal model by 25.39% and 21.31% in the one-shot and five-shot settings, respectively. We hope our method will be helpful for few-shot semantic segmentation for remote sensing. For future work, we will focus on two interesting aspects: (1) designing tailored models to improve the accuracy of the FSS-RSI and (2) dealing with FSS problems in some exceptional cases, such as object occlusion, light changes, and similar appearance.