A Geometric Significance-Aware Deep Mutual Learning Network for Building Extraction from Aerial Images

Abstract

Knowledge-driven building extraction method exhibits a restricted adaptability scope and is vulnerable to external factors that affect its extraction accuracy. On the other hand, data-driven building extraction method lacks interpretability, heavily relies on extensive training data, and may result in extraction outcomes with building boundary blur issues. The integration of pre-existing knowledge with data-driven learning is essential for the intelligent identification and extraction of buildings from high-resolution aerial images. To overcome the limitations of current deep learning building extraction networks in effectively leveraging prior knowledge of aerial images, a geometric significance-aware deep mutual learning network (GSDMLNet) is proposed. Firstly, the GeoSay algorithm is utilized to derive building geometric significance feature maps as prior knowledge and integrate them into the deep learning network to enhance the targeted extraction of building features. Secondly, a bi-directional guidance attention module (BGAM) is developed to facilitate deep mutual learning between the building feature map and the building geometric significance feature map within the dual-branch network. Furthermore, the deployment of an enhanced flow alignment module (FAM++) is utilized to produce high-resolution, robust semantic feature maps with strong interpretability. Ultimately, a multi-objective loss function is crafted to refine the network’s performance. Experimental results demonstrate that the GSDMLNet excels in building extraction tasks within densely populated and diverse urban areas, reducing misidentification of shadow-obscured regions and color-similar terrains lacking building structural features. This approach effectively ensures the precise acquisition of urban building information in aerial images.

1. Introduction

As pivotal vessels of urban data, the categories and spatial arrangement of buildings mirror the patterns of urbanization and growth. The timely acquisition of building information plays a vital role in urban management and serves as a catalyst for the advancement of smart cities [1,2]. High-resolution images provide a wealth of spectral, textural, and spatial information, enabling the extraction of target objects with exceptional accuracy [3]. Compared with other land covers, buildings, as man-made structures, exhibit certain regularities in their geometry, spatial layout, attributes, and topology, which render them well suited for intelligent extraction from high-resolution imagery. Knowledge-driven methods aim to extract prior knowledge of buildings from remote sensing data and represent it through predefined rules, simulating the decision-making processes of domain experts to construct targeted building extraction models. However, these approaches exhibit limited adaptability and are susceptible to external factors, resulting in reduced extraction accuracy. On the other hand, data-driven methods leverage machine learning techniques to autonomously learn building features from images and construct intelligent extraction models based on network structures and data characteristics. Despite their automated nature, these methods often suffer from poor interpretability, reliance on extensive training samples, and issues such as blurred building boundaries. Therefore, the crux of intelligent building extraction from high-resolution aerial images lies in the autonomous assimilation and application of prior knowledge of buildings.

Over the past few years, deep learning semantic segmentation networks have gained recognition as the prevailing approach for building extraction from high-resolution images [4,5,6]. While initially developed for natural images, current deep learning methods for building extraction predominantly concentrate on learning RGB (Red, Green, Blue) semantic features from high-resolution images. Nonetheless, RGB semantic features encounter challenges in accurately capturing the geometric shapes and structural information of buildings, especially in the extraction of intricate building shapes, corners, and other geometric features. Aerial images possess distinct characteristics compared to natural images, encapsulating extensive prior knowledge of buildings, including building boundaries, corners, and spatial relationships. Regrettably, existing deep learning building extraction networks often struggle to effectively harness prior knowledge of aerial images, thereby leading to less precise building extraction outcomes when processing aerial images.

To address the aforementioned limitations, this article proposes a geometric significance-aware deep mutual learning network, named GSDMLNet. The motivation of GSDMLNet is shown in Figure 1, where the incorporation of geometric saliency information regarding buildings serves as crucial prior knowledge within the network. By leveraging both RGB semantic features and Geometric Significance (GS) semantic features in a bi-directional manner, the network enhances its ability to capture the intricate relationship between visual and geometric properties of buildings in remote sensing imagery. GSDMLNet establishes a guidance mechanism between RGB and GS semantic features, strategically leveraging the complementary characteristics of these two feature sets to adaptively learn building features and subsequently refine them through feature fusion, thereby facilitating deep mutual learning building feature extraction. Through the fusion of geometric saliency information and the deep mutual learning mechanism, GSDMLNet effectively harnesses the rich prior knowledge inherent in aerial images, ultimately accentuating target area delineation. The contributions of this article can be summarized as follows:

• This study incorporates the geometric saliency of buildings as prior knowledge within the deep learning network, directing the network’s focus toward the geometrically salient regions of buildings. This strategic integration significantly improves the network’s specificity in learning building features.
• Furthermore, a novel bi-directional guidance attention module (BGAM) is proposed in this study, effectively capturing the interdependency between same-scale feature maps within the RGB and GS semantic branches. The deep mutual learning facilitated by the BGAM enhances the extraction of features within the target area.
• Additionally, an enhanced flow alignment module (FAM++) is introduced to mitigate the occurrence of voids in building feature maps and attain highly semantic information at a heightened resolution. By implementing a gating mechanism inspired by the original FAM design, FAM++ effectively learns the spatial and semantic relationships across adjacent feature layers.
• To enhance the precision of building extraction outcomes, a multi-objective loss function is devised in this study to fine-tune the network. This loss function encompasses the refinement of both the final prediction map and the feature maps derived from the encoder, with the goal of synergistically enhancing the network’s efficiency.

The structure of this paper is outlined below. Section 2 reviews the relevant work on building extraction from aerial images. Section 3 describes our proposed building extraction method. Section 4 describes the datasets, experimental setup parameters, experimental results, and analysis. Finally, the conclusion of this article is in Section 5.

2. Related Works

Currently, there are two main methods for building extraction on a large scale with high frequency. One is a knowledge-driven method, and the other is a data-driven method.

2.1. Knowledge-Driven Building Extraction

Building extraction driven by knowledge leverages the spectral and spatial attributes of buildings within remote sensing imagery to develop models that are capable of interpreting and extracting building features. It mainly includes methods based on region segmentation, feature-based methods, and prior model-based methods.

Region-based methods for building extraction in remote sensing imagery involve the initial partitioning of the image into regions, followed by the extraction of buildings through the integration of texture, geometry, and spectral features. This approach offers a promising solution to mitigate the impulse noise issues associated with pixel-level extraction techniques. Cui et al. [7] combined the Hough transformation with graph search algorithms to further extract buildings from high-resolution images after region segmentation. However, this method may face limitations when dealing with complex scenes due to its inherent constraints. Merabet et al. [8] utilized the watershed segmentation algorithm for initial building extraction, followed by the watershed algorithm and region-merging strategy to further optimize the extraction results.

Feature-based methods for building extraction in images involve the extraction of diverse features, including spectral, shape, texture, edge contour, and spatial features, to effectively identify buildings. The integration of multiple features is essential due to the intricate nature of building structures and colors in images, compensating for the limitations of single-feature-based extraction methods. Notably, research has demonstrated the efficacy of shadow features in assisting building extraction [9,10], with specific emphasis on their role in reducing the error rate of high-rise building extraction [11,12,13]. Hu et al. [14] leveraged the spatial correlation between shadows and buildings to ascertain the buildings’ locations by computing the angle between the shade line and shadow line (SLSLA). Furthermore, the Morphological Building Index (MBI) [15], which encompasses brightness, contrast, shape, and orientation features, has found widespread application in building extraction. Ding et al. [16] introduced an innovative approach for building extraction by fully utilizing MBI, spectral constraints, shadow constraints, and shape constraints. Additionally, some researchers propose building extraction methods based on geometric boundary features, such as prominent straight boundaries or corners of buildings. Zhang et al. [17] proposed a semi-automatic building extraction method based on building corners and orientations, which performs well in extracting regular-shaped buildings. Ning and Lin [18] used building corners and boundary line segments for spatial voting, enhancing the salience of building areas.

The building extraction method based on a prior model involves consolidating prior knowledge of buildings, encompassing spectral information, geometric features, and spatial characteristics, into a structured prior model utilized for building extraction. Typically, in existing research, the prior shape of buildings is integrated into active contour models, Snake models, and their variations to facilitate building extraction [19,20,21].

This approach relies on designed features derived from prior knowledge, making it susceptible to various factors such as the imaging angle of remote sensing data, lighting conditions, building morphology, and shadows. These factors pose constraints on the accuracy of building extraction outcomes and limit the method’s generalizability in large-scale urban settings.

2.2. Data-Driven Building Extraction

The prevalence of data-driven building extraction methodologies enhances precision by refining the multi-scale feature extraction modules, attention mechanism modules, and loss functions within established network architectures. FCN (Fully Convolutional Networks) [22], PSPNet (Pyramid Scene Parsing Network) [23], DeepLab [24], SegNet [25], UNet [26], and other networks are commonly used for building extraction, with some modifications made to these base networks.

UNet and its variants are extensively implemented for the task of building extraction from satellite imagery. Liu et al. [27] used a UNet model enhanced by depth channels to extract building roofs from drone images. They combined this with image analysis methods to identify changes in the entire roof and detect potential illegal constructions. Zheng et al. [28] devised a strategy for building extraction that capitalizes on the UNet as its foundational structure and introduces an innovative Cross-layer Block (CLB) to harmonize features across disparate scales and integrate contextual information from multiple tiers creating a new network called CLNet..

With the introduction of DenseNet [29], a densely connected network extracts feature information and captures buildings in complex scenes. Zhou et al. [30] incorporated dense connections into the UNet network and introduced the concept of deep supervision, resulting in the UNet++ network structure. Tong et al. [31] replaced the convolutional layers in UNet++ with DenseBlocks, reducing the number of network parameters while maintaining the accuracy of building extraction. Zhao et al. [32] incorporated attention modules and multi-task learning into UNet++ to preserve the details of building boundaries.

Some researchers have added building boundary feature extraction branch networks and boundary constraint optimization functions to improve the accuracy of building edge extraction. Guo et al. [33] obtain semantic features and edge features of buildings by constructing a feature decoupling network (FD-Net), with the learning of edge features preserving the integrity of building edge information. Hu et al. [6] introduced a shape preservation model (SPM) and a boundary loss function into the Mask R-CNN, enhancing the network’s ability to extract building boundaries. Zhu et al. [5] proposed a semantic collaboration module (SCM) to fully explore the correlation between the semantic features and edge features of buildings, further enhancing the performance of the network. Yang et al. [34] combined a centroid map forecasting component, a segmentation forecasting component, and a boundary rectification component into an integrated network, thereby augmenting the precision of building boundary detection. Sun et al. [35] proposed a Geographic Spatial Layered Optimization Sampling (GSOS) method based on geographic prior information and sample space location distribution optimization to obtain representative samples. Multiple qualitative and quantitative experiments were conducted, demonstrating the superiority of their method.

While the aforementioned building extraction methods have introduced novel deep learning network models, they exhibit an excessive dependence on training samples and lack incorporation of prior knowledge guidance during model training, leading to a lack of specificity in network targeting.

In knowledge-driven approaches to building extraction, the manual crafting of prior knowledge for buildings is limited and susceptible to interference from external elements such as lighting conditions, posing challenges for large-scale application. Conversely, data-driven building extraction methods overly rely on training samples and do not benefit from the guidance of a priori knowledge, resulting in inadequate network targeting. To address these limitations and enhance the precision and effective integration of prior knowledge in deep learning-based high-resolution remote sensing image building extraction networks, and to meet the demand for accurate building extraction, it is imperative to introduce a deep mutual learning approach for remote sensing image building extraction. This approach considers geometric saliency and aims to mitigate the aforementioned challenges.

3. Methodology

Aiming at the problem of existing deep learning-based building extraction networks lacking effective utilization of prior knowledge of aerial images, this article proposes a geometric significance-aware deep mutual learning network (GSDMLNet) for building extraction from aerial images. The GSDMLNet’s network configuration is depicted in Figure 2, showcasing an encoder-decoder framework. The encoder part includes a dual-branch network and a bi-directional guided attention module (BGAM), while the decoder part includes an improved flow alignment module (FAM++) and a multi-layer perceptron (MLP) layer. GSDMLNet incorporates the geometric saliency of buildings as prior knowledge into the deep learning framework through the dual-branch network. By leveraging deep mutual learning with the BGAM, the network enhances feature extraction within the target region. Additionally, FAM++ is designed to further bolster the network’s capacity to learn building features effectively. The optimization process involves jointly optimizing a multi-objective loss function to enhance the overall performance of the network.

3.1. Acquiring Geometric Significance of Buildings

The geometric attributes of buildings in remote sensing imagery often demonstrate a notable regularity, frequently manifesting as regular rectangles. Buildings exhibit distinct boundaries, with clear, predominantly right-angled corners, as illustrated in Figure 3. This inherent geometric saliency renders buildings easily distinguishable from other land features within remote sensing imagery. To effectively guide the network in discerning crucial regional features, the proposed GSNMLNet leverages remote sensing imagery to derive a geometric saliency feature map specific to buildings. This feature map is subsequently integrated as prior knowledge within the deep learning network.

The intersections within building structures serve as key indicators for accurately defining the geometric characteristics of buildings, with corners being prominent locations for intersection detection. Drawing insights from the distinctive features of building intersections in remote sensing imagery, the GeoSay (Geometric Saliency) algorithm [36] is employed to derive a geometric saliency feature map specific to buildings, as depicted in Figure 4. The confidence and angle information of intersections serve as the first-order saliency ${\omega }_j^1$, while the adjacency relationships of intersections serve as the second-order saliency ${\omega }_j^2$. By fully integrating the first-order and second-order saliency of intersections on remote sensing imagery, a geometric saliency feature map of buildings is obtained, with the calculation formula as follows:

$$GeoSay(p)={\displaystyle \sum _{j\in J}({\omega }_j^1+{\omega }_j^2)\cdot 1_{p\in R_j}}$$

(1)

where $p$ represents each pixel point on the remote sensing image, $J$ represents all intersections obtained by the ASJ algorithm [37] on the remote sensing image, $j$ represents a single intersection point, $R_j$ represents the quadrilateral region formed by a single intersection point and the angle between branches, and $1_{p\in R_j}$ indicates that the pixel value is 1 when the pixel point $p$ is within the range of $R_j$, otherwise it is 0.

$${\omega }_j^1=\rho \cdot P(j\in J_B\left|j_{\theta }\right.)$$

(2)

where $\rho$ represents the confidence of the intersection point, $P(j\in J_B\left|j_{\theta }\right.)$ represents the probability of an intersection point with an angle of $j_{\theta }\in \left[0,\pi \right)$, and $j$ belongs to building intersections. Since most buildings in aerial images are rectangular, $j_{\theta }$ is set to ${\displaystyle \frac{\pi }{2}}$ in this article.

$${\omega }_j^2={\displaystyle \frac{1}{z}}{\displaystyle \sum _{j^{\prime }\in N_j}{e}^{-\frac{‖R_c^j-R_c^{j^{\prime }}‖}{\tau }}\cdot {\omega }_{j^{\prime }}^1}$$

(3)

where $z$ represents the size of the neighborhood, $N_j$ represents the set of intersecting points obtained through the τ-NN algorithm, $j^{\prime }$ represents the intersecting points within $N_j$, $R_c^j$ and $R_c^{j^{\prime }}$ represent the center points of $j$ and $j^{\prime }$ within a range of $R_j$, and $\tau$ represents the normalization of the distance between $R_c^j$ and $R_c^{j^{\prime }}$.

3.2. Dual-Branch Network

The encoder part of the proposed GSDMLNet constructs a dual-branch network, including an RGB semantic branch and a GS semantic branch. The RGB semantic branch and the GS semantic branch have the same structure, consisting of four E modules (refer to Figure 2b), where the E module is mainly inspired by the transformer backbone network. Considering a remote sensing image denoted as $I\in R^{H\times W\times C}$, it is characterized by its dimensions: $H$ for height, $W$ for width, and $C$ for the count of spectral channels. The remote sensing image is input into the RGB semantic branch to obtain four building feature maps at different scales denoted as $\left\{B{F}_1,B{F}_2,B{F}_3,B{F}_4\right\}$, with a size of $\left\{{\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times 32,{\displaystyle \frac{H}{8}}\times {\displaystyle \frac{W}{8}}\times 64,{\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}\times 160,{\displaystyle \frac{H}{32}}\times {\displaystyle \frac{W}{32}}\times 256\right\}$. After applying the GeoSay algorithm, a single-band building geometric saliency feature map is obtained, which is then input into the GS semantic branch to obtain four building feature maps at different scales containing geometric saliency, denoted as $\left\{G{S}_1,G{S}_2,G{S}_3,G{S}_4\right\}$, with a size of $\left\{{\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times 32,{\displaystyle \frac{H}{8}}\times {\displaystyle \frac{W}{8}}\times 64,{\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}\times 160,{\displaystyle \frac{H}{32}}\times {\displaystyle \frac{W}{32}}\times 256\right\}$.

3.3. Bi-Directional Guided Attention Module

To effectively capture the interdependencies of building feature maps at the same scale in the dual branches, a bi-directional guided attention module (BGAM) is devised, as shown in Figure 2d. The deep mutual learning between the two branches can enhance the feature extraction effect in the target area. Firstly, guided by the building feature map $B{F}_i$, the building geometric saliency feature map $G{S}_i$ adaptively learns the features of the corresponding region, preserving the important region features $IG{S}_i$ in $G{S}_i$ that are strongly correlated with $B{F}_i$. Mathematically:

$${\alpha }_i=sigmoid(B{F}_{i}G{S}_i)$$

(4)

$$IG{S}_i={\alpha }_{i}G{S}_i$$

(5)

where $i=1,2,3,4$, $sigmoid(\cdot )$ represents an activation function, and ${\alpha }_i\in \left[0,1\right]$ represents the correlation between the ith building feature map $BF$ and the building geometric saliency feature map $GS$.

Next, guided by the important region feature $IG{S}_i$, the building feature map $B{F}_i$ adaptively learns the features of the corresponding region, obtaining the feature $IB{F}_i$ in $B{F}_i$ that is strongly correlated with $IG{S}_i$. In mathematical terms, the aforementioned procedures can be formulated as:

$${\beta }_i=sigmoid(B{F}_i+IG{S}_i)$$

(6)

$$IB{F}_i={\beta }_{i}B{F}_i$$

(7)

where ${\beta }_i\in \left[0,1\right]$ represents the correlation between the ith building feature map $BF$ and the building geometric saliency feature map $IGS$.

Finally, the adaptively learned features $IG{S}_i$ and $IB{F}_i$ are fused. In order to complement the semantic information of buildings, the residual idea is utilized by adding it to the input $B{F}_i$, resulting in the deep mutually learned building feature map $B{G}_i$. The $B{G}_i$ can be computed as follows:

$$B{G}_i=B{F}_i+Con{v}_{3\times 3}(IB{F}_i\parallel IG{S}_i)$$

(8)

where $\parallel$ represents feature fusion operation along the channel dimension, and $Con{v}_{3\times 3}(\cdot )$ represents convolution operation with a kernel size of $3\times 3$.

3.4. Improved Flow Alignment Module

To obtain high-resolution (high-spatial resolution), strong semantic information building feature maps and reduce the phenomenon of holes within buildings, an enhanced version of the flow alignment module (FAM++) is constructed, as illustrated in Figure 2e. The advancement over the original flow alignment module (FAM) [38], FAM++, incorporates the idea of gating to better learn semantic and spatial information between adjacent feature layers. Firstly, the semantic flow field $\Delta \in R^{H\times W\times 2}$ is computed for adjacent feature layers, with the calculation formula as follows:

$$\Delta =Con{v}_{3\times 3}(Con{v}_{1\times 1}(B{G}_i)\parallel Up(Con{v}_{1\times 1}(B{G^{\prime }}_{i+1})))$$

(9)

where $B{G}_i$ denotes the high-resolution building feature map that contains weak semantic details, $B{G^{\prime }}_{i+1}$ refers to the low-resolution building feature map that is rich in strong semantic content, $Con{v}_{1\times 1}(\cdot )$ signifies a convolution with a kernel size of $1\times 1$, and $Up(\cdot )$ indicates the process of up-sampling.

Subsequently, to accentuate the commonalities in features across adjacent layers, a merging process is applied to these proximate feature layers. The gated feature map $g$ is obtained through gating operation. The calculation formula for the $g$ is as follows:

$$g=Con{v}_G(Con{v}_{3\times 3}(Con{v}_{1\times 1}(B{G}_i)\parallel Up(Con{v}_{1\times 1}(B{G}_{i+1}^{\prime }))))$$

(10)

where $Con{v}_G(\cdot )$ represents the gating operation, which includes one convolutional layer with a kernel size of $1\times 1$ and one sigmoid activation function.

Ultimately, the identical semantic flow field is deployed to individually carry out warping procedures on the neighboring feature strata, culminating in the acquisition of a pair of high-resolution feature maps that are harmonized with semantic data. Then, the gated feature maps $g$ and $1-g$ are used to perform weighted fusion on the two aligned feature maps, resulting in a high-resolution building feature map $B{G}_i^{\prime }$ with strong semantic information. The calculation formula for $B{G}_i^{\prime }$ is as follows:

$$B{G}_i^{\prime }=gCon{v}_{1\times 1}(warp(B{G}_{i+1}^{\prime },\Delta ))+(1-g)warp(B{G}_i,\Delta )$$

(11)

where $warp(\cdot )$ represents the warp operation.

3.5. Simple Multi-Layer Perceptron Module

To diminish the quantity of parameters throughout the entire network, lightweight MLP layers were used in the encoder section, as shown in Figure 2c. Four high-resolution building feature maps with strong semantic information, denoted as $\left\{B{G}_1^{\prime },B{G}_2^{\prime },B{G}_3^{\prime },B{G}_4\right\}$ and each of size $\left\{{\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times 32,{\displaystyle \frac{H}{8}}\times {\displaystyle \frac{W}{8}}\times 64,{\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}\times 160,{\displaystyle \frac{H}{32}}\times {\displaystyle \frac{W}{32}}\times 256\right\}$, are input into the MLP layer for feature fusion, obtaining the final prediction result map denoted as $F_{final}$. The calculation formula is as follows:

$$F_{final}=MLP(B{G}_1^{\prime },B{G}_2^{\prime },B{G}_3^{\prime },B{G}_4)$$

(12)

where $MLP(\cdot )$ represents the MLP layer operation.

3.6. Multi-Objective Loss Function

We have engineered a multi-objective loss function to optimize the network’s performance, aiming to achieve higher precision in building extraction. The loss from the final prediction map, combined with the losses of the four feature maps produced by the encoder, constitutes our multi-objective loss function, denoted as $L$. The formula for calculation is as follows:

$$L=L_{final}+\lambda {\displaystyle \sum _{i=1}^4{L}_i}$$

(13)

where $L_{final}$ represents the loss function of the final prediction map $F_{final}$, $L_i$ represents the loss function of the intermediate feature map $\left\{B{G}_1,B{G}_2,B{G}_3,B{G}_4\right\}$, and $\lambda$ represents the weight coefficient. In this article, to alleviate the effects of imbalanced sampling, dice loss [39] and focal loss [40] are used for $L_{final}$, focal loss is used for $L_i$, and $\lambda$ is set to 1.

4. Experiments

4.1. Datasets

This research evaluated the performance of the GSDMLNet through effectiveness testing on two renowned UAV image datasets that are accessible to the public: the WHU Building Dataset [41], and Massachusetts Building Dataset [42]. The WHU Building Dataset comprises buildings exhibiting small-type discrepancies and relatively regular shapes, thereby presenting less formidable challenges in building extraction. In contrast, the Massachusetts Building Dataset encompasses urban and suburban buildings characterized by dense arrangements, diverse types, and varied shapes, thus markedly heightening the complexity of building extraction. The details of the two datasets are shown in

Table 1. Detailed information of experimental data.

Dataset	Collection Area	Cover Area (km2)	Spatial Resolution (m)	Sample Size	Sample Number (Group)			Number of Buildings
					Train	Val	Test
WHU Building Dataset	Christchurch, New Zealand	450	0.2	512	8460	3626	2782	28,873
Massachusetts Building Dataset	Boston	340	1	1500	137	4	10	-

.

4.2. Implementation Details and Evaluation Metrics

The experimental hardware setup for this study comprises an NVIDIA GeForce RTX 3090 GPU, with the programming environment based on PyTorch 2.1.1. During the training process of GSDMLNet, the batch size was set to 30, with a maximum of 100 iterations. We froze half of the training epochs to accelerate the training speed, where the learning rate during the frozen training period was set to 0.0001, and the learning rate during the unfrozen period was set to 0.00001, utilizing the Adam optimizer to optimize the network parameters. The study utilizes seven metrics for a quantitative assessment of the proposed GSDMLNet and other methodologies for building extraction outcomes: Precision, Recall, F1-Score (F1), Intersection over Union (IoU), Overall Accuracy (OA), Params, and Frames Per Second (FPS).

4.3. Comparison and Analysis

In order to confirm the efficacy of the proposed GSDMLNet in extracting buildings from aerial images, we compared it with five other methods: PSPNet [23], Deeplabv3+ [43], HRNet [44], SegFormer [45], and BEARNet [46]. PSPNet employs a pyramid pooling module to aggregate contextual information from diverse regions, thereby enhancing its capacity to encapsulate global information. Deeplabv3+ leverages spatial pyramid pooling with dilated convolutions to facilitate multi-scale feature extraction, thereby enhancing the precision of semantic segmentation boundaries. HRNet establishes parallel connections between convolutions at different resolutions to facilitate information interaction, ensuring the preservation of reliable high-resolution representations across the network. SegFormer is a network that integrates lightweight multi-layer perceptrons (MLPs) within Transformers ensuring straightforward and efficient functionality. Finally, BEARNet disentangles building bodies and boundaries, guiding the network to independently learn building boundary features and optimize the targeted acquisition of boundary features.

(1) Results on the WHU Building Dataset: Figure 5 illustrates the outcomes of employing diverse methods for building extraction on the WHU Building Dataset. The dataset features a relatively low density of buildings, predominantly presenting regular rectangular shapes, which simplifies the task of building extraction. From Figure 5, it can be observed that SegFormer, BEARNet, and GSDMLNet extract buildings with more complete boundaries, while PSPNet, Deeplabv3+, and HRNet have lower accuracy in building boundary extraction. Furthermore, when buildings in aerial images have colors similar to the ground, the performance of building extraction by PSPNet, Deeplabv3+, HRNet, SegFormer, and BEARNet is poorer. Due to GSDMLNet incorporating geometric saliency of buildings as prior knowledge, this method can accurately extract buildings by relying on the structural features of buildings even when they have colors similar to the ground. For the extraction of large-scale buildings, PSPNet, Deeplabv3+, HRNet, and SegFormer tend to exhibit hole phenomena, while BEARNet utilizes the FAM to improve this. The use of the FAM++ in GSDMLNet preserves the integrity of buildings and reduces the occurrence of holes in large-scale buildings.

The quantitative assessment of various methodologies on the WHU Building Dataset is presented in

Table 2. Quantitative evaluation results of each method on the WHU Building Dataset and Massachusetts Building Dataset.

Data	Method	Precision (%)	Recall (%)	F1 (%)	IoU (%)	OA (%)	Params (M)
WHU Building Dataset	PSPNet	96.10	96.01	96.05	92.54	97.96	2.38
	Deeplabv3+	91.56	95.83	93.65	88.15	96.48	5.81
	HRNet	95.69	93.96	94.82	90.34	97.36	9.64
	SegFormer	97.57	97.24	97.40	95.00	98.63	3.72
	BEARNet	97.70	97.15	97.42	95.03	98.67	19.42
	GSDMLNet	97.89	97.57	97.73	95.61	98.83	6.09
Massachusetts Building Dataset	PSPNet	75.00	73.20	73.09	63.03	89.95	2.38
	Deeplabv3+	72.46	82.18	77.01	64.28	88.13	5.81
	HRNet	78.46	76.21	77.32	66.46	91.23	9.64
	SegFormer	84.54	83.81	83.68	74.63	93.71	3.72
	BEARNet	84.92	85.27	85.09	75.82	93.99	19.42
	GSDMLNet	85.89	86.58	86.23	77.34	94.42	6.09

. For the relatively easy WHU Building Dataset in terms of building extraction, all methods perform well. Among them, GSDMLNet has the highest accuracy evaluation indicators, followed closely by BEARNet. The Precision, Recall, F1, IoU, and OA of GSDMLNet are 97.89%, 97.57%, 97.73%, 95.61%, and 98.83%, respectively. In comparison to the closely ranked BEARNet, which manifests robust quantitative evaluation outcomes overall, GSDMLNet surpasses with superior Precision, Recall, F1 score, IoU, and OA metrics, surpassing by 0.19%, 0.42%, 0.31%, 0.58%, and 0.16%, respectively. Furthermore, GSDMLNet achieves this performance while utilizing 13.33M fewer parameters compared to BEARNet.

(2) Results on the Massachusetts Building Dataset: The outcomes of building extraction employing diverse methods on the Massachusetts Building Dataset are depicted in Figure 6. The dataset is distinguished by a dense distribution of buildings, featuring complex and diverse shapes alongside inconsistent scales, which collectively present a substantial challenge to the task of building extraction. The building extraction results of PSPNet, Deeplabv3+, and HRNet are not satisfactory, while SegFormer, BEARNet, and GSDMLNet produce building results that are close to the ground truth. Buildings extracted by PSPNet and Deeplabv3+ often exhibit merging phenomena, while HRNet has fewer mergers but incomplete building boundaries. For dense small-scale building extraction, BEARNet and GSDMLNet provide more accurate building boundaries compared to SegFormer. In particular, shadows in aerial images interfere with the extraction of building features to some extent. The BGAM of GSDMLNet is capable of lessening the effects of shadow occlusion, enabling the network to dynamically extract building features and subsequently enhance the precision of building extraction.

Table 2 presents the quantitative evaluation results of a variety of methods applied to the Massachusetts Building Dataset. For the challenging Massachusetts Building Dataset, the evaluation metrics of each method are lower compared to the other two datasets. The precision evaluation metrics of GSDMLNet are superior to other methods, with Precision, Recall, F1, IoU, and OA of 85.89%, 86.58%, 86.23%, 77.34%, and 94.42%, respectively. Compared to the second-ranked BEARNet, GSDMLNet outperforms in Precision, Recall, F1, IoU, and OA by 0.97%, 1.31%, 1.14%, 1.52%, and 0.43%, respectively. When compared to SegFormer, which also uses Transformer as the backbone network, GSDMLNet surpasses in Precision, Recall, F1, IoU, and OA by 1.35%, 2.77%, 2.55%, 2.71%, and 0.71%, while GSDMLNet only increases by 2.37M Params compared to SegFormer.

4.4. Sensitivity Analysis of Key Parameters

This article improves the performance of the GSDMLNet by supervising the optimization using a loss function of the final prediction map and intermediate feature map $\left\{B{G}_1,B{G}_2,B{G}_3,B{G}_4\right\}$. To delve into the responsiveness of the hyperparameter λ within the multi-objective loss function, a series of experiments have been executed on both the WHU Building Dataset and the Massachusetts Building Dataset. The values of λ tested range from 0 to 2.0, with increments of 0.5. As shown in

Table 3. Sensitivity analysis of λ on the WHU Building Dataset and Massachusetts Building Dataset.

Data	λ	F1 (%)	IoU (%)	OA (%)
WHU Building Dataset	0	97.69	95.55	98.81
	0.5	97.71	95.56	98.82
	1.0	97.73	95.61	98.83
	1.5	97.72	95.60	98.83
	2.0	97.72	95.57	98.82
Massachusetts Building Dataset	0	85.86	76.84	94.31
	0.5	85.91	76.92	94.35
	1.0	86.23	77.34	94.42
	1.5	85.99	77.03	94.37
	2.0	86.00	77.04	94.37

, compared to the conventional network with only loss constraints on the final prediction map, the network with loss constraints on the intermediate feature map achieved improved building extraction accuracy on three datasets. When the λ value is set to 1.0, the GSDMLNet achieved the highest building extraction accuracy in terms of F1, IoU, and OA on all datasets, indicating that applying loss constraints on the intermediate feature map effectively enhances the network’s performance.

4.5. Comparing with Microsoft’s Building Extraction Method

Between 2014 and 2024, Bing Maps detected 1.4 billion buildings worldwide. The building extraction process was completed in two stages: First, a deep neural network (DNN) was used to identify building features in aerial images, and then the building pixels were converted into polygons. This approach ensured the regularity and completeness of the building extraction results, offering broad application potential. The method achieved a precision of 94.85%, a recall of 79.02%, and an IoU of 65.24% globally. However, extracting building contours in two steps made it difficult to control errors during the semantic segmentation stage. In some regions, the extraction results showed issues such as overlapping buildings, incorrect shapes, and omissions. Analyzing the cause, it is likely that the deep learning-based semantic segmentation process encountered significant errors, such as disconnections, missing segments, and misclassifications, leading to chaotic geometric results. Compared to Microsoft’s method, our approach does exhibit irregular building edges, but in terms of the semantic segmentation stage alone, our method is more stable, with higher recall and IoU scores.

5. Conclusions

In response to the issue of existing deep learning networks for building extraction from remote sensing imagery lacking effective utilization of prior knowledge in aerial images, this article proposes GSDMLNet for building extraction in aerial images. This network incorporates building geometric saliency feature maps as prior knowledge into the network. By utilizing the BGAM and FAM++ to further enhance the network’s learning capabilities for building features, and optimizing with a multi-objective loss function, the accuracy and precision of building extraction are effectively improved. The introduction of building geometric saliency significantly enhances the network’s targeted learning of building features, the BGAM effectively enhances the extraction performance of target regions, and the FAM++ effectively reduces the occurrence of building holes. GSDMLNet demonstrates superior performance on three datasets, achieving the highest precision evaluation metrics, indicating the method’s versatility and strong generalization across different datasets. However, while the multi-objective loss function is beneficial, it may complicate the training process and lead to difficulties in balancing the various loss components.

In future research, we will focus on simplifying the multi-objective loss function to enhance the robustness and flexibility of the method in various scenarios. Additionally, building extraction from imagery may be incomplete or erroneous due to occlusion by shadows or vegetation. To address this, we plan to incorporate more prior knowledge about buildings, leveraging features such as shadows to optimize the extraction process. The extracted building edges may exhibit irregularities, making them difficult to apply directly. By comparing with Microsoft’s building extraction method, which involves semantic segmentation followed by geometric transformation, we will explore using geometric transformation to further improve the regularity of the extraction results, facilitating their practical application.