RPAFormer: Building Extraction with Relative Position Aggregated Transformer

Highlights

What are the main findings?

• Develops a novel pure transformer-based building extraction framework named RPAFormer, which is capable of flexibly adapting to the diverse structure variations of buildings and producing accurate local details in complex scenarios.
• Conducts experiments on public building extraction datasets to verify the effectiveness of RPAFormer. The experimental results demonstrate that RPAFormer achieves a more competitive performance than other state-of-the-art methods.

What are the implications of the main findings?

• Relative Position-aware Self-attention (RPSA) block learns the token dependencies within the local window and can flexibly adapt to the intricate background regions and varied structure patterns of buildings.
• Transformer decoder network consisting of multiple Cross Masked Attention (CMA) blocks fused with the multi-scale features is capable of modeling the relative position dependencies of the buildings.

Abstract

Automatic building extraction plays an important role in various remote sensing applications, such as seismic disaster investigation, seismic hazard risk assessment, urban planning, and photogrammetry. Despite the substantial progress, state-of-the-art building extraction methods are still limited by two issues: (i) existing approaches leverage convolutional layers or non-local self-attention to encode the position-aware dependencies, while they cannot flexibly adapt to the complex background contexts and varied structure patterns of buildings; and (ii) the local details cannot be well preserved by existing hierarchical decoders due to the imperfect feature aggregation, yielding unsatisfactory segmentation outputs in the local adjacent region. To address these issues, we propose Relative Position Aggregated Transformer (RPAFormer), which is capable of modeling the relative position dependencies of buildings and producing accurate local details using a dual attention transformer network. Specifically, we propose a Relative Position-aware Self-attention (RPSA) framework to learn the token dependencies within the local window. A transformer decoder network consisting of multiple Cross Masked Attention (CMA) blocks is also introduced to fuse the multi-scale features. Extensive experiments demonstrate the superior performance of the proposed method and its great promise for real-world engineering deployment.

1. Introduction

Building extraction from high-resolution remote sensing (RS) images is a fundamental topic with various applications, e.g., seismic damage survey and path planning, autonomous driving and Geographic Information Systems (GISs) [1,2,3,4,5]. This task involves the problem of assigning a binary label to each pixel for distinguishing building foregrounds from complex backgrounds and requires generating the delineated building footprints with sharp boundaries for further processing or model simplification.

Traditional building extraction methods [6,7] generally utilize various descriptors, e.g., texture, geometry and spectrum, to model the shapes of buildings. The early approaches used handcrafted features to represent buildings and then typically introduced carefully designed post-processing modules such as boundary detection [8,9], geometric line extraction [10] and multispectral fusion strategies [11,12] for further refinement.

Since the handcrafted descriptors are sensitive to the drastic shape variations, these methods show unsatisfactory generalization capabilities in the presence of irregular structures of buildings. Some studies also adopt multi-source data fusion such as LIDAR points [13] and multispectral imagery [14] for data augmentation, but acquiring the multi-source data is costly, whereas the segmentation performance is still limited due to the imperfect manual fusion in different data domains.

Inspired by the significant advances achieved by convolutional neural networks (CNNs), deep learning-based segmentation methods have become the dominant approach in the field of building extraction in recent years. These methods [15,16,17] employ hierarchical convolution operators to extract the local scale-invariant features, which can effectively exploit the inductive bias by large-scale model training. However, as the CNN-based architecture relies on stacking multiple convolutional layers and blocks in local receptive fields, it fails to capture the long-range relationship between pixels in different regions.

The transformer architecture [18] was firstly proposed in the NLP field. Recently, it has attracted great interest in various computer vision tasks, e.g., object recognition [19], image generation [20,21] and video understanding [22,23,24]. The Visual Transformer [25,26,27,28,29] (ViT) generally splits the input image into fine-grained patches and leverages Multi-head Self-attention modules to model the long-range dependencies of the embedded patches. The self-attention operator captures a larger receptive field and introduces global context-aware dynamics, thus enabling it to complement the CNN backbones by providing beneficial non-local information. Shi et al. [29] proposed Transformer Scale Gate (TSG) to optimize the multi-scale features combination process; the multi-scale feature selection mechanism in TSG enables it to exploit the inner properties of transformer attention maps for fine-grained per-pixel prediction. SETR [30] adopts ViT as a backbone to extract features and utilizes CNN as a decoder for pixel-wise prediction, achieving impressive performance gains compared with conventional CNN architectures. Segmenter [31] discards the convolutional layers and proposes a pure transformer architecture to capture the global context in encoding and decoding modules, which significantly outperforms previous CNN-based approaches.

As shown in Figure 1, affected by the intricate background interference and varied building structural patterns, existing transformer-based approaches can hardly maintain the balance between the global and local inductive bias learning, which limits their effectiveness in challenging remote sensing images. In addition, the most advanced ViTs employ multi-scale feature fusion and an upsampling operator (e.g., bilinear interpolation or deconvolution) in hierarchical decoders for dense per-pixel prediction, while they cannot recover the high-quality local details, leading to unsatisfactory segmentation outputs in the neighboring region.

Figure 1. The challenges of existing building extraction approaches. (a) The structural appearance of the buildings in different positions varies significantly; (b) The decoded features of adjacent buildings in the blue box cannot present high-quality local details for dense per-pixel prediction.

Motivated by the success of the pyramid architecture in CNNs, some works [28,32,33,34] introduce pyramid structures into transformers to learn multi-scale features. To enhance the computation efficiency of ViTs, the local dependencies operation is also taken into account. For example, Swin Transformer [25] uses hierarchical shifted local windows to partition features into non-overlapping local windows while also allowing for cross-window connections, which shows excellent flexibility and computational efficiency to exploit crucial visual cues at different scales. NAT [35] introduces neighborhood attention to localize the receptive field in the neighborhood around each token, thereby capturing local inductive biases without needing extra operations.

However, with the growing complexity of remote sensing images, traditional algorithms are not capable of meeting the high-precision requirements. Recent state-of-the-art methods use Deep Neural Networks (DNNs) for building extraction, which can be divided into two categories: instance segmentation approach and semantic segmentation-based approach.

The semantic segmentation-based approach passes the RS images through a CNN backbone, e.g., FCN [36,37,38,39,40,41,42,43,44] and UNet [45], to obtain coarse building segmentation masks. Meanwhile, the CNN models perform pixel-wise classification to determine whether each position in the RS image belongs to the building or not. For example, Wei et al. [46] proposed the multi-scale aggregation method to obtain fine-grained representations to extract the buildings. ASLNet [16] adopts an adversarial shape learning module to exploit the building shape patterns; a shape discriminator and a shape regularizer are mutually learned to establish discriminative shape representations to guide building segmentation. Guan et al. [47] presented a dual-branch transformer framework for cross-view geo-localization that removes sky areas via semantic segmentation and adopts geometric projection and gradient-aware weighting to align street and satellite features. The latest works [38,39,40] introduce transformer architecture to model the global dependencies; the self-attention operation in transformer can suppress the irrelevant background interference, which shows great advantages in determining the per-pixel belongings for the building extraction task.

On the other hand, the instance segmentation-based approach can detect and segment all building footprints in a top-down manner. Existing works use Mask R-CNN [48,49,50] or CenterMask [51] to predict the instance-level bounding boxes and pixel-level segmentation masks. Then, the segmentation results of all buildings are refined using active contour models [52] or boundary extraction modules [53,54]. Recent advanced approaches such as PolyMapper [55] and frame field learning [56] employ the polygonal representation via graph structure or directional field for building shape regularization, yielding sharper corners and convenient vectorization by polygonal vertex modeling.

Meanwhile, how to effectively fuse the hierarchical features is crucially important for the building extraction task. To leverage discriminative features in different scales, AFNet [57] designs a Scale-feature Attention Module (SFAM) to enhance the low-level features for small objects and strengthen high-level features for large objects, which is capable of preserving semantic visual cues in various resolutions. Zhu et al. [40] design a multiple attending path network to extract high-level semantics and preserve multi-level localization details with serial convolution blocks that can retain the local details using a channel-wise attention module. Chen et al. [58] further combine the transformer backbone and CNN head to fuse features. The channel-wise and spatial-wise attention modules are employed for fine-grained building footprint prediction. BCTNet [59] jointly leverages two parallel branches of the convolutional encoder branch and the transformer encoder branch to extract multi-scale feature maps. It then designs the Concatenation-then-cross-fusion Transformer Block (CCTB) and Adaptive Gating Module (AGM) to integrate the feature representation. Luo et al. [60] adopt SegFormer as the backbone network and fuse very-high-resolution remote sensing images with LiDAR elevation data. By designing cross-modal feature fusion and multi-scale upsampling decoding strategies, the method effectively improves the accuracy of building extraction. However, these feature fusion strategies introduce hyperparameters that require tricky tuning.

In this work, we propose a pure transformer-based architecture named RPAFormer, which can exploit the global-local dependencies and perform feature fusion by simply stacking multiple self-attention blocks in both encoder and decoder networks. Furthermore, it conducts dense prediction in the whole remote sensing images for fast building extraction. Specifically, we propose a Relative Position-aware Self-attention (RPSA) to learn the token dependencies within the local window, which exploits the relative position bias of different tokens for structural relationship encoding. Afterwards, we propose a transformer decoder network consisting of multiple Cross Masked Attention (CMA) blocks to fuse the multi-scale features that is capable of aggregating the high-quality local details for accurate building extraction.

2. RPAFormer

To tackle the aforementioned problems, we propose a Relative Position Aggregated Transformer (RPAFormer), which is capable of modeling the relative position dependencies of buildings and producing accurate local details using a dual attention transformer network.

Our method first splits the remote sensing image into a sequence of patches, which are encoded into the query–key–value structure. Afterwards, we propose a Relative Position-aware Self-attention (RPSA) module to learn the relative position dependencies of the encoded tokens. The correlation similarity between query and key tokens is modified by incorporating relative position biases into self-attention calculations. The modified self-attention modules are gradually stacked so that the neighboring patches can be merged to construct hierarchical representations. By making full use of the positional dependencies encoded by these blocks, the RPSA module can flexibly retrieve the critical position of the query and key maps in the global area, enabling the transformer network to distinguish the easily confused regions more accurately.

Moreover, as existing segmentation approaches often struggle to preserve high-quality local details using hierarchical decoders, we design a novel transformer-based decoder network to fuse the multi-scale features from backbones with those in decoders. Specifically, we design a Cross Masked Attention (CMA) module to assign dynamic weights to local feature aggregation. Then, the skip connection is applied to enhance the correspondences between the encoder and decoder tokens to recover the local details of the buildings. By aggregating the semantic representation with hierarchical decoder blocks, the segmentation results can be significantly boosted.

2.1. Overall Architecture

The overall framework of the proposed RPAFormer is shown in Figure 2. It consists of an encoder network and a decoder network. The encoder network contains four stages, where each stage leverages an RPSA block to learn the relative position dependencies of the encoded tokens. The decoder network fuses the multi-scale features of the encoder and decoder network using CMA blocks and the corresponding shifted window version with skip connection for local feature aggregation.

Figure 2. The pipeline of the proposed RPAFormer, which consists of an encoder network and a decoder network using transformer architecture. The encoder network employs Relative Position-aware Self-attention (RPSA) module to produce global dependencies with hierarchical representation. Then, the features are sent to Cross Masked Attention (CMA) and Shifted Window Cross Masked Attention (SW_CMA) blocks in the decoder network to capture high-quality local details. Here, the LayerNorm (LN) and MLP layers in CMA and SW-CMA blocks are omitted for simplification.

The encoder network takes an image $\mathbf{I} \in {\mathbb{R}}^{\mathit{H} \times \mathit{W} \times 3}$ with height H and width W as input. The image is first split into ${\displaystyle \frac{\mathit{H}}{4}} \times {\displaystyle \frac{\mathit{W}}{4}}$ patches, where the size of each patch is 4 × 4. These patches are flattened into 1-dim vector and projected into C- dim embedding, hereinafter referred to as $\mathit{C}$, and then the embedded features are passed into the RPSA block to obtain embedded patches ${\mathbf{X}}_0 \in {\mathbb{R}}^{{\displaystyle \frac{\mathit{H}}{4}} \times {\displaystyle \frac{\mathit{W}}{4}} \times \mathit{C}}$. The embedded patches are regarded as visual tokens and are sent into the subsequent stages, the output sizes of the following hierarchical representation are ${{\mathbf{X}}_1 \in \mathbb{R}}^{{\displaystyle \frac{\mathit{H}}{8}} \times {\displaystyle \frac{\mathit{W}}{8}} \times 2\mathit{C}}$, ${{\mathit{X}}_2 \in \mathbb{R}}^{{\displaystyle \frac{\mathit{H}}{16}} \times {\displaystyle \frac{\mathit{W}}{16}} \times 4\mathit{C}}$ and ${\mathbf{X}}_3 \in {\mathbb{R}}^{{\displaystyle \frac{\mathit{H}}{32}} \times {\displaystyle \frac{\mathit{W}}{32}} \times 8\mathit{C}}$, respectively.

To effectively leverage the hierarchical features, we design a decoder network employing Cross Masked Attention (CMA) in multiple stages for local feature aggregation. The feature maps learned in the encoder network and the intermediate decoder features are taken as inputs and projected to be a query–key–value structure. Then, Cross Masked Attention is performed on the projected tokens to enhance the feature representation. Afterwards, the corresponding features in both the encoder and decoder network are passed into a Shifted Window CMA (SW_CMA) block, which fuses the dual features with skip connection to capture high-quality local details. Finally, the decoded features in the terminal stage are upsampled to predict the per-pixel categories.

2.2. Encoder Network

To model the long-range dependencies of the hierarchical patches, existing ViTs employ the self-attention module to enlarge the limited receptive field in classical CNNs. The notable Swin Transformer simplifies the global self-attention into local windows and performs cross-window connections to model the interactions across different regions. However, Swin Transformer partitions the feature maps by fixed shifting window, which limits the flexibility to utilize a suitable receptive field in different positions, especially for the ambiguous region or adjacent buildings. To address this issue, we propose Relative Position-aware Self-attention (RPSA) to exploit the relative position-aware dependencies with hierarchical representation.

RPSA is designed to learn relative position dependencies within the encoder network and can be stacked to construct hierarchical representation. The input feature is flattened and then projected to a C- dim embedding as shown in Figure 3.

Figure 3. The workflow of RPAFormer. The left sub-window illustrates the components of one block in the feature encoder. The right one shows the structure of the RPSA block.

The conventional ViTs use Multi-head Self-attention (MHSA) as follows to ensure the computational efficiency:

$$\mathrm{MHSA}(\mathbf{X}) = \mathrm{Concat}({ \mathrm{head}}_0, \dots , {\mathrm{head}}_{\mathit{M}}) {\mathbf{W}}_{\mathit{P}}$$

(1)

where ${\mathrm{head}}_{\mathrm{m}}$ is the output of the m-th attention head, ${\mathbf{W}}_{\mathrm{P}} \in {\mathbb{R}}^{\mathit{N} \times \mathit{C}}$ is the projection matrix, and M is the number of heads. Each head can be calculated via the self-attention operation as:

$$\mathrm{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \mathrm{Softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\mathbf{T}}}{\sqrt{{\mathit{C}}_{\mathit{m}}}}}\right)\mathbf{V}$$

(2)

where Q, K and V represent the query, key and value tokens with the size $\mathit{N} \times {\mathit{C}}_{\mathit{m}}$, ${\mathit{C}}_{\mathit{m}}$ refers to the channel dimension of the m-th head, and ${\mathit{C}}_{\mathit{m}} = \mathit{C}/\mathit{M}$.

The computational burden of self-attention operation is determined by the resolution of the input image. To enhance the computational efficiency, Swin Transformer [25] introduces a shifted window partitioning strategy and performs Multi-head Self-attention within the shifted window. For the arbitrary pixel i, the shifted window self-attention can be formulated as:

$${\mathit{f}}_{\mathit{i}} = {\sum }_{\mathit{j}\in \mathrm{\Omega }}\mathrm{Softmax}\left({\displaystyle \frac{{\mathit{q}}_{\mathit{i}} {\mathit{k}}_{\mathit{j}}}{\sqrt{{\mathit{C}}_{\mathit{m}}}}}\right){\mathit{v}}_{\mathit{j}}$$

(3)

where ${\mathit{f}}_{\mathit{i}}$ represents the output at pixel $\mathit{i}$; ${\mathit{q}}_{\mathit{i}}$, ${\mathit{k}}_{\mathit{j}}$ and ${\mathit{v}}_{\mathit{j}}$ denote the pixel-wise element in Q, K, and V, respectively; and Ω is the corresponding region within the shifted window.

For the building extraction task, the relative position-aware dependencies of the attention map have not been fully exploited in Equation (3), so the retrieved tokens may attend to irrelevant keys/values in the background region. To address this issue, we propose the Relative Position-aware Self-attention (RPSA), which enforces the weights of the query and key tokens to be flexibly reassigned to the target building areas under the guidance of positional bias. Different from MHSA, which performs dot products between queries and keys in neighboring areas, here, we employ high-order polynomial functions [61] to model non-local dependencies among spatial positions. The RPSA is given by:

$${\mathit{f}}_{\mathit{i}} = \mathrm{R}\mathrm{P}\mathrm{S}\mathrm{A}\left(\mathbf{Q}, \mathbf{K}, \mathbf{V}\right) ={\sum }_{\mathit{j}\in \mathrm{\Omega }}\mathrm{Softmax}(\left({\mathit{q}}_{\mathit{i}}\mathit{⨀} {\mathit{k}}_{\mathit{i}}\right){\mathit{r}}_{\mathit{j}-1}^{\mathit{k}} + \left({\mathit{q}}_{\mathit{j}}\mathit{⨀}{\mathit{k}}_{\mathit{j}}\right){\mathit{r}}_{\mathit{j}-\mathit{i}}^{\mathit{q}} + {\mathit{r}}^{\mathit{b}}){\mathit{v}}_{\mathit{j}}$$

(4)

where ⨀ denotes the Hadamard product.

By modifying the dot product of ${\mathit{q}}_{\mathit{i}}$ and ${\mathit{k}}_{\mathit{j}}$ to the Hadamard product, the complexity of the attention map learned by the query/key token is significantly reduced from quadratic to linear. Moreover, the position bias parameters ${\mathit{r}}_{\mathit{j}-1}^{\mathit{k}}$, ${\mathit{r}}_{\mathit{j}-1}^{\mathit{q}}$ and ${\mathit{r}}^{\mathit{b}}$ present relative position embeddings on the attention map, leading RPSA to be more flexible to adjust the weights at different positions.

The RPSA blocks in the encoder network can be stacked to obtain hierarchical representation; here, we adopt four stages to learn the semantic visual representation. Figure 3 illustrates that each stage consists of an RPSA module, two LayerNorm (LN) layers and an MLP layer. For the l-th hierarchical block, the feature map ${\mathbf{X}}_{\mathit{l}}$ is computed by:

$${\mathbf{Z}}_{\mathit{l} - 1}^{\prime } = \mathrm{R}\mathrm{P}\mathrm{S}\mathrm{A}\left(\mathrm{L}\mathrm{N}\right({\mathbf{X}}_{\mathit{l} - 1}^{\prime }\left)\right) + {\mathbf{X}}_{\mathit{l} - 1}^{\prime }$$

(5)

$${\mathbf{X}}_{\mathit{l}}=\mathrm{MLP}\left(\mathrm{LN}\right({\mathbf{Z}}_{\mathit{l} - 1}^{\prime }\left)\right)+{ \mathbf{Z}}_{\mathit{l} - 1}^{\prime }$$

(6)

At the first stage, the embedded patches are merged ${\displaystyle \frac{\mathit{H}}{4}} \times {\displaystyle \frac{\mathit{W}}{4}} \times \mathit{C}$. At the subsequent stages, the resolution is further reduced with the dimension increasing to construct the hierarchical feature sizes of ${\displaystyle \frac{\mathit{H}}{8}} \times {\displaystyle \frac{\mathit{W}}{8}} \times 2\mathit{C}$, ${\displaystyle \frac{\mathit{H}}{16}} \times {\displaystyle \frac{\mathit{W}}{16}} \times 4\mathit{C}$ and ${\displaystyle \frac{\mathit{H}}{32 }}\times {\displaystyle \frac{\mathit{W}}{32}} \times 8\mathit{C}$, respectively.

2.3. Decoder Network

We propose a transformer decoder network to fuse the hierarchical features. The decoder network consists of four stages, and each stage includes a CMA block, an SW_CMA block and an upsample operation. Given the encoded feature map ${\mathbf{X}}_{\mathrm{l}} \in {\mathbb{R}}^{{\mathit{N}}_{\mathit{l}} \times {\mathit{C}}_{\mathit{l}}}$ (here, ${\mathit{N}}_{\mathit{l}}$ = ${\mathit{H}}_{\mathit{l}} \times {\mathit{W}}_{\mathit{l}}$) at the l-th stage, the CMA module leverages the corresponding features in the encoder network and the output features in the last stage as inputs to calculate the Cross Masked Attention. The output token ${\mathit{F}}_{\mathit{l}}$ can be obtained by:

$${\mathit{F}}_{\mathit{l}} = \{\begin{array}{c}\mathit{S}\mathit{o}\mathit{f}\mathit{t}\mathit{m}\mathit{a}\mathit{x}({\mathcal{M}}_{\mathit{l} - 1} + {\mathbf{Q}}_{3 - \mathit{l}}{\mathbf{K}}_{\mathit{l}}^{\mathrm{T}}){\mathbf{V}}_{\mathit{l} }{+ \mathbf{P}}_{\mathit{l} - 1}, \mathit{i}\mathit{f} \mathit{l} > 0\\ \mathit{S}\mathit{o}\mathit{f}\mathit{t}\mathit{m}\mathit{a}\mathit{x}({\mathcal{M}}_{\mathit{l}} + {\mathit{Q}}_3{\mathit{K}}_{\mathit{l}}^{\mathbf{T}}){\mathbf{V}}_{\mathit{l}} + {\mathbf{X}}_3, \mathit{i}\mathit{f} \mathit{l} = 0\end{array}$$

(7)

where ${\mathbf{P}}_{\mathit{l} - 1}$ is the upsampled output in the previous (l-1)-th stage and ${\mathcal{M}}_{\mathit{l}-1}$ denotes the corresponding attention mask, which can be predicted by:

$${\mathcal{M}}_{\mathit{l} - 1}\left(\mathit{x}, \mathit{y}\right) = \{\begin{array}{c}0, \mathit{i}\mathit{f} { \mathbf{M}}_{\mathit{l} - 1}(\mathit{x}, \mathit{y}) > 0\\ -\infty , \mathit{o}\mathit{t}\mathit{h}\mathit{e}\mathit{r}\mathit{w}\mathit{i}\mathit{s}\mathit{e}\end{array}$$

(8)

where ${\mathit{M}}_{\mathit{l} - 1}\left(\mathit{x}, \mathit{y}\right)$ is the binarized mask prediction in the last decoder stage. The query, key and value token in CMA are calculated by:

$${\mathbf{Q}}_{\mathit{l}} ={ \mathbf{X}}_{\mathit{l}}{\mathbf{W}}_{\mathit{l}}^{\mathrm{Q}},{ \mathbf{K}}_{\mathit{l}} ={ \mathbf{P}}_{\mathit{l} - 1}{\mathbf{W}}_{\mathit{l}}^{\mathrm{K}},{ \mathbf{V}}_{\mathit{l}} ={ \mathbf{P}}_{\mathit{l} - 1}{\mathbf{W}}_{\mathit{l}}^{\mathrm{V}}$$

(9)

where ${\mathbf{Q}}_{\mathit{l}}$, ${\mathbf{K}}_{\mathit{l}}$ and ${\mathbf{V}}_{\mathit{l}}$ represent the query, key and value tokens with the size ${\mathit{N}}_{\mathit{l}} \times {\mathit{C}}_{\mathit{l}};$ ${\mathbf{W}}_{\mathit{l}}^{\mathrm{Q}}$, ${\mathbf{W}}_{\mathit{l}}^{\mathrm{K}}$, and ${\mathbf{W}}_{\mathit{l}}^{\mathrm{V}}$ are the weight matrices corresponding to the query, key and value, respectively.

Note that the Cross Masked Attention is estimated by query token ${\mathbf{Q}}_{\mathit{l}}$ from the encoder feature ${\mathbf{X}}_{\mathit{l}}$, whereas the interactive key ${\mathbf{K}}_{\mathit{l}}$ and value ${\mathbf{V}}_{\mathit{l}}$ are projected from ${\mathbf{P}}_{\mathit{l} - 1}$. The upsampled output ${\mathbf{P}}_{\mathit{l} - 1}$ in the last stage can be estimated by:

$${\mathbf{P}}_{\mathit{l} - 1}=\mathrm{U}\mathrm{P}(\mathrm{S}\mathrm{W}\_\mathrm{C}\mathrm{M}\mathrm{A}({\mathbf{X}}_{\mathit{l} - 1},{ \mathbf{F}}_{\mathit{l} - 1}\left)\right)$$

(10)

where the function UP(·) denotes the upsampling operator, and SW_CMA(·) is the shifted window Cross Masked Attention within the local window to improve the computational efficiency.

Afterwards, the upsampled feature P₃ is passed into a 1 × 1 convolutional layer and normalized with the softmax layer to generate binary segmentation results for dense prediction.

On the whole, the CMA and SW_CMA blocks perform masked attention to fuse the dual encoder–decoder features and capture high-quality local details. The CMA blocks eliminate the irrelevant background pixels, which are capable of constructing more discriminative non-local dependencies for pixel-wise prediction. The SW_CMA(·) block employs a residual connection to fully utilize the dual encoder–decoder features, which enhances the correspondences between the encoder and decoder layers in multiple stages.

2.4. Loss Function

In the training phase, we employ the cross-entropy loss to minimize the difference between the ground truth mask and the predicted segmentation output as follows:

$${\mathcal{L}}_{\mathit{s}\mathit{e}\mathit{g}} = -\sum _{\mathit{i}}\left[\widehat{{\mathit{y}}_{\mathit{i}}}\mathit{l}\mathit{o}\mathit{g}{\mathit{y}}_{\mathit{i}} + (1 -\widehat{{ \mathit{y}}_{\mathit{i}}}\left)\mathit{l}\mathit{o}\mathit{g}(1 - {\mathit{y}}_{\mathit{i}}\right)\right]$$

(11)

where ${\mathit{y}}_{\mathit{i}}$ and $\widehat{{\mathit{y}}_{\mathit{i}}}$ denote the predicted label and the ground truth.

Moreover, since the cross-entropy segmentation loss always leads to coarse results, we adopt binary cross-entropy and dice loss to enhance the boundary details. We adopt the Laplacian convolutional kernel to produce the boundary predictions and ground truth boundary labels. The dice loss is also applied to measure the overlap rate and alleviate the class-imbalance problem of foreground/background pixels. The details of the learning loss are:

$${\mathcal{L}}_{\mathit{d}\mathit{e}\mathit{t}\mathit{a}\mathit{i}\mathit{l} }= {\mathcal{L}}_{\mathit{c}\mathit{e}}\left(\mathit{L}\mathit{a}\mathit{p}\right(\widehat{{\mathit{y}}_{\mathit{i}}}), \mathit{L}\mathit{a}\mathit{p}\left({\mathit{y}}_{\mathit{i}}\left)\right) + {\mathcal{L}}_{\mathit{d}\mathit{i}\mathit{c}\mathit{e}}\right(\widehat{{\mathit{y}}_{\mathit{i}}},{ \mathit{y}}_{\mathit{i}})$$

(12)

where Lap(∙) denotes the Laplacian convolutional operation. Therefore, the whole loss function is:

$${\mathcal{L} = \mathcal{L}}_{\mathit{s}\mathit{e}\mathit{g}}+ \mathit{\lambda }{\mathcal{L}}_{\mathit{d}\mathit{e}\mathit{t}\mathit{a}\mathit{i}\mathit{l}}$$

(13)

where $\mathit{\lambda }$ is the hyperparameter balancing the weight of the segmentation loss and detail loss.

3. Experiments

We first provide the implementation details of RPAFormer and conduct comprehensive comparisons with state-of-the-art methods on three benchmark building datasets. We then conduct ablation experiments to analyze each component designed in this model. The visualization results of challenging remote sensing images are presented for further analyses.

3.1. Implementation Setting

We run the proposed RPAFormer with the PyTorch 1.8 toolkit. The hardware environment includes a server with two RTX3090 GPUs and 8 cores of 3.9 GHz CPU. The remote sensing images in all datasets are cropped to 512 × 512 for training and testing. The data augmentation operations include random crop, random horizontal flipping, photometric distortion and mixup, which are used to improve the model’s robustness. During training, we apply the AdamW optimizer using a linear decay learning rate scheduler with a 1500 iterations linear warm-up. The channel dimension $\mathit{C}$ is set to 128. The model follows a ’poly’ LR schedule with factor 1.0, and the learning rate is set to 6 × 10⁻⁵. The hyperparameter $\mathit{\lambda }$ is set to 0.1, and the weight decay is set to 0.01. We use the batch size of 8 per GPU for all the evaluated datasets, and the model is trained with 160 k iterations.

3.2. Evaluation Metrics

To evaluate the performance of different building extraction methods, we adopt the commonly used evaluation metrics in public remote sensing datasets for fair comparison. Four pixel-level metrics including recall (R), precision (P), F1 score and intersection over union (IoU) are employed for comparison. Each can be calculated as:

$$\mathrm{R} = {\displaystyle \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}}$$

(14)

$$\mathrm{P}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(15)

$$\mathrm{F}1=2 \times {\displaystyle \frac{\mathrm{P} \times \mathrm{R}}{\mathrm{P}+\mathrm{R}}}$$

(16)

$$\mathrm{IoU}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}}$$

(17)

where TP, FP, TN and FN represent the pixel-wise predictions of true positive, false positive, true negative, and false negative, respectively.

4. Implementation Results

A quantitative comparison between the RPAFormer and the previous work is presented in Figure 4. The canonical CNN and transformer-based architectures, e.g., UNet [45] and RSP [62], achieve unsatisfactory results due to inadequate modeling of the global context and local details, whereas by exploiting dynamic position-aware dependencies and introducing effective feature fusion, RPAFormer obtains more significant performance gains in both the F1 and IoU metrics.

Figure 4. The F1 and IoU scores obtained by different algorithms on Massachusetts Dataset.

Subsequently, the computational results of RPAFormer are compared with those of other mainstream methods on three primary building datasets, namely, the Massachusetts Dataset, the Inria Aerial Dataset, and the WHU Dataset.

4.1. Massachusetts Dataset

Massachusetts is an aerial dataset collection captured across Boston, which consists of 137 images in the training set, 4 images in the validation set, and 10 images in the test set. Each image, with a size of 1500 × 1500, is annotated with pixel-wise labels. The spatial resolution of the whole dataset is 1.2 m per pixel, covering around 340 km².

In our setting, we compare the performance of our method on the Massachusetts Dataset with 11 methods, including 8 CNN-based methods and 3 transformer-based methods. The metrics of precision, recall, F1 score and IoU are employed for evaluation, and the comparative performance results are shown in Table 1. RPAFormer achieves the best results overall, obtaining a precision of 87.52% and recall of 83.31%. For the IoU score, RPAFormer significantly outperforms the classical CNN based methods such as PSP-Net and JointNet with significant performance gains of 6.51% and 2.74%, respectively. Compared with the state-of-the-art method ASLNet [16], RPAFormer boosts the score by 2.41% in the IoU metric. Compared with ASLNet-SR, RPAFormer is superior to ASLNet-SR in all of the evaluation metrics. Note that ASLNet-SR requires the introduction of the super-resolution technique for image pre-processing, leading to higher evaluation scores in the precision metric. However, RPAFormer can achieve a more competitive performance without utilizing extra pre-processing techniques.

We also compare RPAFormer with the most recent transformer-based models such as Swin Transformer and SegFormer [63]. RPAFormer achieves relative performance gains of 0.62% in the F1 score and 1.21% in the IoU metric against Swin Transformer, respectively. Compared to RSP, RPAFormer obtains performance gains of 1.44% in the IoU metric and 0.53% in the F1 score. We further demonstrate the segmentation results in Figure 5. Note that the Massachusetts Dataset contains large quantities of small buildings with low resolutions. The decoder network in RPAFormer is able to preserve high-quality local details between the buildings; thus, its segmentation performance is superior to Swin Transformer, SegFormer and other CNN-based methods.

Table 1. Quantitative comparison (%) with state-of-the-art algorithms on Massachusetts Dataset.

Method	Precision	Recall	F1	IoU
U-Net [45]	85.53	80.68	83.33	70.99
PSPNet [64]	80.93	81.29	81.11	68.22
ResuNet [65]	86.16	81.25	83.59	71.87
Deeplab V3 [66]	86.75	78.77	82.57	70.32
JointNet [67]	86.21	81.29	83.68	71.99
GMEDN [68]	84.09	77.49	80.63	67.61
ASLNet [16]	85.70	81.17	83.91	72.32
ASLNet-SR [16]	85.92	82.83	84.32	72.95
SegFormer [63]	84.79	77.89	81.19	68.34
Swin [25]	87.33	82.30	84.74	73.52
RSP [62]	87.28	82.51	84.83	73.69
RPAFormer	87.52	83.31	85.36	74.73

Note: Bold numbers indicate the best experimental results.

Table 1. Quantitative comparison (%) with state-of-the-art algorithms on Massachusetts Dataset.

Method	Precision	Recall	F1	IoU
U-Net [45]	85.53	80.68	83.33	70.99
PSPNet [64]	80.93	81.29	81.11	68.22
ResuNet [65]	86.16	81.25	83.59	71.87
Deeplab V3 [66]	86.75	78.77	82.57	70.32
JointNet [67]	86.21	81.29	83.68	71.99
GMEDN [68]	84.09	77.49	80.63	67.61
ASLNet [16]	85.70	81.17	83.91	72.32
ASLNet-SR [16]	85.92	82.83	84.32	72.95
SegFormer [63]	84.79	77.89	81.19	68.34
Swin [25]	87.33	82.30	84.74	73.52
RSP [62]	87.28	82.51	84.83	73.69
RPAFormer	87.52	83.31	85.36	74.73

Note: Bold numbers indicate the best experimental results.

Figure 5. Experimental building extraction results obtained by different segmentation methods on Massachusetts Dataset. (a) Original image, (b) Ground truth, (c) PSPNet, (d) Deeplab V3, (e) SegFormer, (f) Swin Transformer, (g) Ours.

4.2. Inria Aerial Dataset

The Inria Aerial Dataset is a widely used urban buildings extraction dataset for semantic segmentation. There are 360 5000 × 5000 pixel aerial images with a spatial resolution of 0.3 m in this dataset, wherein 180 are presented with ground truth labels. These images are captured from five cities: Austin, Chicago, Kitsap, Tyrol, and Vienna.

Extensive experiments on the Inria Aerial Dataset were conducted to validate RPAFormer, and the quantitative results are summarized in Table 2. As shown in Table 2, RPAFormer achieves superior results in all of the evaluation metrics. It obtains precision and recall scores of 90.19% and 89.13% and F1 and IoU scores of 89.66% and 81.25%, respectively. Compared to BOMSCNet, RPAFormer obtains performance gains of 1.91% for the F1 score and 3.07% for the IoU score. Compared to the most recent ASLNet [16] algorithm, RPAFormer obtains performance gains of 2.16% for the F1 score and 3.12% for the IoU score.

Table 2. Quantitative comparison (%) with the state-of-the-art algorithms on Inria Aerial Dataset.

Method	Precision	Recall	F1	IoU
U-Net [45]	81.76	82.76	82.03	70.03
SegNet [69]	85.47	81.96	83.70	71.10
PSPNet [64]	87.87	84.98	86.40	76.06
MAPNet [40]	88.58	86.04	87.24	77.79
Deeplab V3 [66]	88.08	85.47	86.75	76.61
GMEDN [68]	87.03	81.37	83.88	72.95
MFCNN [70]	88.58	87.91	88.38	79.35
BOMSCNet [71]	87.93	87.58	87.75	78.18
ASLNet-SR [16]	88.98	86.32	87.50	78.13
SegFormer [63]	87.78	87.64	87.71	78.72
Swin [25]	89.37	87.60	88.48	79.34
RSP [62]	89.57	87.58	88.57	79.48
RPAFormer	90.19	89.13	89.66	81.25

Note: Bold numbers indicate the best experimental results.

Furthermore, we also compare RPAFormer with the state-of-the-art transformer-based algorithms, e.g., Swin Transformer [25] and RSP [62]. RPAFormer outperforms other transformer architectures in all of the evaluation metrics. We present the experimental visualization results in Figure 6. Due to the effective non-local operation such as MSA, the segmentation outputs of Swin Transformer and SegFormer are better than PSPNet and DeepLab V3 (results in the third and fourth columns of the figure), but they still fail to obtain accurate results in segmenting irregular buildings and ambiguous regions (results in the last three columns of the figure). By performing relative position aggregation and feature fusion in the transformer framework, RPAFormer achieves more satisfactory results than Swin Transformer and SegFormer.

Figure 6. Experimental building extraction results obtained by different segmentation methods on Inria Aerial Dataset. (a) Original image, (b) Ground truth, (c) PSPNet, (d) Deeplab V3, (e) SegFormer, (f) Swin Transformer, (g) Ours.

4.3. WHU Dataset

The WHU Dataset is collected from aerial images with a 0.075 m spatial resolution covering 450 km² in Christchurch, New Zealand. It consists of 8188 images with a resolution of 512 × 512. The images in this dataset are split into 4736 images in the training set, 1036 in the validation set and 2416 images in the test set. The whole dataset includes about 187,000 buildings with different scales and shapes.

Table 3 reports the comparative results on the WHU Dataset. RPAFormer obtains an F1 score of 95.04% and an IoU score of 90.74%, which is better than the SOTA CNN-based methods such as MA-FCN [46], HRNet V2 [72] and other transformer-based methods. For the precision metric, RPAFormer obtains a corresponding score of 95.38%, which also outperforms other transformer- and CNN-based algorithms. For the recall metric, HRNet achieves the best performance with a score of 95.37%, while RPAFormer ranks second with a score of 94.70%. This is because the remote sensing images in the WHU Dataset are entirely collected from the unique city of Christchurch, New Zealand; thus, the irregular buildings and scenario variations in WHU are rarer than the cases in the Massachusetts and Inria Aerial Datasets. HRNet V2 can effectively exploit the inductive bias of the segmentation instances using the specifically designed high-to-low resolution convolution. We present the visualized segmentation results in Figure 7 and Figure 8. It can be found that RPAFormer is able to correctly separate the buildings even in adjacent boundary regions, leading to higher precision and IoU scores in the quantitative evaluation.

Table 3. Quantitative comparison (%) with state-of-the-art algorithms on WHU Dataset.

Method	Precision	Recall	F1	IoU
U-Net [45]	92.88	93.18	91.61	86.95
Deeplab V3 [66]	92.94	93.92	93.43	87.67
PSPNet [64]	93.74	93.39	93.57	87.91
SegNet [69]	95.14	93.55	94.34	89.28
DANet [73]	94.25	93.93	94.09	88.87
MA-FCN [46]	94.20	94.47	94.34	89.21
HRNet-V2 [72]	94.16	95.37	94.76	90.04
BOMSCNet [71]	95.14	94.50	94.80	90.15
MSTT-Net [74]	-	-	88.20	88.00
SegFormer [63]	94.71	94.41	94.56	89.68
Swin [25]	94.79	93.53	94.16	88.96
RSP [62]	95.05	94.44	94.74	90.01
RPAFormer	95.38	94.70	95.04	90.74

Note: Bold numbers indicate the best experimental results.

Figure 7. Experimental building extraction results obtained by different segmentation methods on WHU Dataset. (a) Original image, (b) Ground truth, (c) PSPNet, (d) Deeplab V3, (e) SegFormer, (f) Swin Transformer, (g) Ours.

Figure 8. Visual comparison of segmentation results obtained by HRNet-V2 and RPAFormer on the WHU Dataset. (a) Original image, (b) Ground truth, (c) HRNet-V2, (d) Ours.

4.4. Ablation Analysis

4.4.1. Ablation Study of Each Component

We conduct an ablation study to validate the effectiveness of each component, and the quantitative evaluation conducted on the Inria Aerial Dataset is presented for fair comparison. In our experiment, we consider three modules including Relative Position Self-Attention (RPSA), Cross Masked Attention (CMA) and Skip Connection (SC). The complete results are shown in Table 4. The pure RPSA learns the relative dependencies of the encoded tokens and obtains scores of 89.05% and 80.27% in terms of F1 and IoU, respectively. The variant approach only with the CMA block in the decoder network achieves a performance of 88.38% and 79.18% for the F1 and IoU metrics. Afterwards, if the RPSA and CMA modules are jointly trained in a uniform framework, the novel approach achieves a performance of 89.39% and 80.81%, respectively. Finally, by adding the skip connection operator, the RPAFormer improves the F1 and IoU scores to 89.66% and 81.25%. It also obtains the highest scores of 90.19% and 89.13% for the relative precision and recall metrics. The results clearly illustrate the effectiveness of the modules adopted in RPAFormer.

Table 4. Ablation study of the RPSA, CMA and SC modules in the proposed method (%).

RPSA	CMA	SC	Precision	Recall	F1	IoU
√			89.74	88.64	89.05	80.27
	√		89.23	87.54	88.38	79.18
		√	88.25	87.34	87.80	78.25
√	√		90.13	88.65	89.39	80.81
√	√	√	90.19	89.13	89.66	81.25

Note: Bold numbers indicate the best experimental results.

4.4.2. Influence of Different Encoder Backbones

To conduct an investigation into the performance changes using different encoder backbones, here, we conduct comparative experiments to analyze the influence of the backbone features. The results are shown in Table 5. We can observe that the discriminative powers of conventional CNN-based backbones, e.g., VGG-19, ResNet-50 and DenseNet, are significantly limited, as they cannot capture the global dependencies to maintain the complete structure of the buildings. Swin Transformer alleviates this issue via the shifted window self-attention blocks; yet, the performance is compromised by the absence of explicit structural position-aware information modeling. Our encoder network with hierarchical RPSA blocks achieves the best performance, which verifies that modeling the relative position-aware dependencies in transformer architecture is crucial for high-quality footprint extraction.

Table 5. Comparative study of the variant methods using different backbones (%).

Method	Precision	Recall	F1	IoU
VGG-19	89.46	82.41	82.89	70.53
ResNet-50	89.14	84.73	86.88	76.80
DenseNet	89.55	83.68	86.36	76.34
Swin-S	89.11	88.60	88.85	79.95
RPAFormer	90.19	89.13	89.66	81.25

Note: Bold numbers indicate the best experimental results.

4.4.3. Comparison with Other Detail Aggregation Methods

To assess the effectiveness of the detail aggregation methods in the proposed framework, in this experiment, we compare the proposed method with other detail aggregation algorithms. Previous works such as RefineNet [75], SRI-Net [76], MAPNet [40], STDC [77] and MAFNN [70] also introduce detail aggregation strategies for building extraction, while the strategies of these trackers are quite different. Specifically, RefineNet, SRI-Net and MAPNet employ the multi-path aggregation strategy, which incorporates long-range residual connections or attention modules to encode low-level information such as edges and corners. STDC designs a Detail Aggregation Module (DAM) that directly takes low-level boundary information as guidance for feature learning. MAFNN uses morphological filtering to refine the extracted building boundaries; thus, it can improve the shape regularity. The comparative results are shown in Table 6. RPAFormer achieves the best performance in all of the metrics. MAFNN obtains the second best performance on the whole, but its morphological post-processing is time-consuming and cannot be learned in end-to-end manner.

Table 6. Comparison with the state-of-the-art algorithms using detail aggregation strategy (%).

Method	Precision	Recall	F1	IoU
RefineNet [75]	85.36	80.26	82.73	70.55
SRI-Net [76]	85.77	81.46	83.56	71.76
MAPNet [40]	88.58	86.04	87.24	77.79
STDC [77]	88.25	87.34	87.80	78.25
MAFNN [70]	88.58	87.91	88.38	79.35
RPAFormer	90.19	89.13	89.66	81.25

Note: Bold numbers indicate the best experimental results.

4.4.4. Ablation Analysis of the Hyperparameter λ

To further validate the rationality of the hyperparameter used to balance segmentation loss and detail loss in Equation (13), we conduct an ablation analysis to justify the impact of the hyperparameter λ on the Inria Aerial Dataset by tuning its value among 0, 0.1, 1 and 10. The results are presented in Table 7. We can see that setting λ = 0.1 obtains the highest performance across all four metrics, which delivers a good balance between segmentation accuracy and detail preservation. Therefore, we adopt 0.1 as the default setting in all subsequent experiments.

Table 7. Ablation analysis of the hyperparameter λ on Inria Aerial Dataset (%).

Setting	Precision	Recall	F1	IoU
$\mathit{\lambda } =0$	89.02	88.32	88.67	79.65
$\mathit{\lambda } =0.1$	90.19	89.13	89.66	81.25
$\mathit{\lambda } =1$	89.56	89.02	89.29	80.65
$\mathit{\lambda } =10$	89.92	88.61	89.26	80.61

Note: Bold numbers indicate the best experimental results.

4.4.5. Ablation Analysis of Performance on Low-Resolution Images and Computational Overhead

To investigate the performance on low-resolution images, we downsample the images of the cropped images in the Inria Aerial Dataset to be 256 × 256 and report the results in Table 8. Although the downsampled inputs lead to performance degradation, our RPAFormer still obtains 88.15 and 86.42 in terms of precision and recall, respectively. Such results verify the robustness of RPAFormer in low-resolution settings.

Table 8. Ablation analysis of performance on low-resolution images and computational overhead (%).

Setting	Precision	Recall	F1	IoU
$256\times 256$	88.15	86.42	87.28	77.43
$512\times 512$	90.19	89.13	89.66	81.25

Note: Bold numbers indicate the best experimental results.

4.5. Statistical Significance Testing

To guarantee that the observed performance enhancements are not attributed to chance alone, additional statistical significance tests are implemented between RPAFormer and other state-of-the-art methods. We conducted statistical significance tests on the WHU Dataset and the Inria Aerial Dataset. The results are provided in Table 9. The experimental results demonstrate consistent predictive performance across both datasets, indicating the robustness of RPAFormer. This stability is especially evident on the Inria Aerial Dataset, which comprises a significantly larger number of training samples.

Table 9. Statistical significance test on WHU Dataset and Inria Aerial Dataset (%).

Dataset	Precision	Recall	F1	IoU
WHU	$94.91 \pm$ 0.41	$94.97 \pm$ 0.26	$94.94 \pm$ 0.09	$90.57 \pm$ 0.21
Inria	$90.06 \pm$ 0.14	$89.21 \pm$ 0.15	$89.63 \pm$ 0.04	$81.21 \pm$ 0.06

Meanwhile, we have systematically calculated the p-values across all evaluation metrics (precision, recall, F1, and IoU) over three independent runs using the independent two-sample t-test. The results are provided in Table 10. Comparing our proposed method with the baseline, the calculated p-values are 0.0003 for precision, 0.0292 for recall, <0.0001 for F1, and <0.0001 for IoU. Since all p-values are strictly less than the standard significance level of p < 0.05 (with F1 and IoU being highly significant at p < 0.001), we can confidently reject the null hypothesis. This proves that the performance improvements yielded by our method are statistically significant and inherently derived from our architectural design, rather than random variance.

Table 10. Statistical significance test results ((Mean ± Std) %) over three independent runs.

Metric	Baseline	RPAFormer	p-Value
Precision	$88.11 \pm$ 0.26	$90.06 \pm$ 0.14	0.0003
Recall	$88.62 \pm$ 0.27	$89.21 \pm$ 0.15	0.0292
F1	$88.36 \pm$ 0.06	$89.63 \pm$ 0.04	<0.0001
IoU	$79.15 \pm$ 0.10	$81.21 \pm$ 0.06	<0.0001

5. Conclusions

In this paper, we propose RPAFormer to extract building footprints with relative position-aware dependencies using a transformer-based encoder–decoder architecture. To achieve this, we propose a novel Relative Position-aware Self-attention (RPSA) to learn the relative dependencies of the encoded tokens. Then, a decoder network with Cross Masked Attention (CMA) blocks is designed for multi-scale feature fusion, enabling the transformer network to focus on learning precise local details. The dual encoder–decoder features are aggregated using skip connection, allowing us to strengthen the correspondence between the encoder and decoder layers in multiple stages. Extensive experiments demonstrate that our method achieves more competitive performance than other state-of- the-art methods.