RegGAN: An End-to-End Network for Building Footprint Generation with Boundary Regularization

Abstract

Accurate and reliable building footprint maps are of great interest in many applications, e.g., urban monitoring, 3D building modeling, and geographical database updating. When compared to traditional methods, the deep-learning-based semantic segmentation networks have largely boosted the performance of building footprint generation. However, they still are not capable of delineating structured building footprints. Most existing studies dealing with this issue are based on two steps, which regularize building boundaries after the semantic segmentation networks are implemented, making the whole pipeline inefficient. To address this, we propose an end-to-end network for the building footprint generation with boundary regularization, which is termed RegGAN. Our method is based on a generative adversarial network (GAN). Specifically, a multiscale discriminator is proposed to distinguish the input between false and true, and a generator is utilized to learn from the discriminator’s response to generate more realistic building footprints. We propose to incorporate regularized loss in the objective function of RegGAN, in order to further enhance sharp building boundaries. The proposed method is evaluated on two datasets with varying spatial resolutions: the INRIA dataset (30 cm/pixel) and the ISPRS dataset (5 cm/pixel). Experimental results show that RegGAN is able to well preserve regular shapes and sharp building boundaries, which outperforms other competitors.

1. Introduction

Building footprint generation is of great interest in remote sensing, which involves numerous applications, e.g., facilitating urban planning, identifying undocumented buildings, and assessing building damage after natural disasters. High-resolution remote sensing imagery, which provides huge opportunities for meaningful geospatial target extraction at a large scale, becomes a fundamental data source for mapping buildings. Nevertheless, reliable and accurate building footprint generation is still challenging because of several factors. On the one hand, the complex and heterogeneous appearance of buildings leads to internal variability. On the other hand, mixed backgrounds and other objects with similar spectral signatures further limit the class separability.

Early methods focus on the design of hand-crafted features that can best depict buildings. For instance, Ref. [1] proposed to utilize spectral and texture features to extract urban buildings from satellite imagery. Nonetheless, the empirical feature design was satisfactory only under specific requirements or on specific data and lacked good generalization capabilities.

Nowadays, convolutional neural networks (CNNs) are extensively utilized for remote sensing tasks [2], as they surpass conventional methods in terms of accuracy of efficiency. CNNs directly learn hierarchical contextual features from raw inputs, which offer greater generalization capabilities for the task of building footprint generation than traditional methods. Although building footprint maps provided by existing CNNs seem to be impressive at a large scale (Figure 1), it is observed that such results are not that perfect when we zoom in (see results from FC-DenseNet [3] in Figure 1). The extracted buildings show irregular shapes, which are far from their exact geometry.

There are also some studies focused on delineating structured building footprint maps from remote sensing data. Their pipeline usually is composed of two steps, where the first step is building segmentation and the second step is building regularization. For the building regularization that aims to refine the semantic segmentation results, early efforts utilized low-level features [4] or manually defined constraints [5] for the optimization of building boundaries. However, such the hand-crafted features and rules utilized in these methods limit their generalization capability. Recently, a two stage method [6] was proposed to regularize building boundaries using deep learning networks. Nevertheless, their boundary regularization results heavily relied on the input building segmentation maps, thus, the potential of this method is restricted. Moreover, the separation of building segmentation and boundary regularization makes the whole pipeline inefficient in large-scale processing. Therefore, we propose RegGAN, which is an end-to-end network to produce regularized building footprint masks directly from remote sensing imagery. In particular, building segmentation results are not required as input data in RegGAN, making our method more efficient and robust. RegGAN is composed of two modules: the generator and the discriminator. More specifically, the generator has two paths. The regularization path takes remote sensing imagery as input and aims at producing regularized building footprints. The ideal input mask is encoded and decoded in the reconstruction path and then is taken as input to the discriminator. The generator is learned with four loss functions: semantic loss, adversarial loss, reconstruction loss, and regularized loss. It is notable that the regularized loss is beneficial to refining building boundaries in our method. Finally, a multiscale discriminator is proposed to distinguish between ideal building masks and regularized ones, which can leverage the information in different scales to improve the final results.

This work’s contributions are threefold.

(1) We propose to combine building mask segmentation and boundary regularization in an end-to-end network. The proposed method can generate building footprint maps directly from remote sensing imagery, and optimize building shapes while preserving semantic correctness.

(2) Our approach is a novel generative adversarial network (GAN). By designing a multiscale discriminator and incorporating the regularized loss in the full objective function, the proposed network can enhance building boundaries to generate structured building footprints.

(3) In terms of mask and boundary accuracy metrics, RegGAN is superior to other competitors in remote sensing imagery with different spatial resolutions.

2. Related Work

A lot of methods have been proposed to address the task of building footprint generation. According to the types of tasks, they can be classified into two types: semantic segmentation and regularization of building footprints.

2.1. Semantic Segmentation of Building Footprints

Semantic segmentation of building footprints can be regarded as a task of semantic segmentation that assigns each pixel in the image with the label of “building” or “non-building”.

Traditional methodologies include four general types: (1) index-based, (2) segmentation-based, (3) geometrical-primitives-based, and (4) classification-based methods. Index-based methods aim to directly extract buildings with a designed feature index. The morphological building index (MBI) [7] is a favored index to depict the built-up presence, which is calculated by a series of multiscale and multidirectional morphological operators. The segmentation-based methods generate building footprints based on image segmentation methods. For example, a region-based level set segmentation method was proposed in [8] for building extraction from remote sensing imagery. Geometrical-primitives-based algorithms delineate buildings based on the geometrical primitives of buildings such as corners or boundaries. In [9], the corner points of buildings were first detected by a Harris corner detector and then connected to construct polygonal representations. The objective of classification-based methods is to implement machine learning classifiers, assigning a label to each pixel. For instance, [10] generated building footprints by utilizing multiple classifiers and fusing them under a hierarchic architecture. However, these early works have a general limitation: a poor generalization due to the use of prior information and hand-crafted features.

Instead of the complex feature engineering exploited in traditional methods, deep learning methods can directly learn features from raw inputs, thus having a better generalization capability. With the introduction of deep learning architectures, recent studies can provide impressive building mapping results by utilizing semantic segmentation networks, which are usually referred to as fully convolutional network (FCN) [11] or encoder–decoder architectures. The commonly used networks are SegNet [12], U-Net [13], FC-DenseNet [3], and HRNet [14]. As a predecessor of semantic segmentation, FCN substitutes the transposed convolutions for fully connected layers, which effectively alters popular classification models to make pixel-level predictions. In [15], FCN was firstly initialized by using a large volume of possibly inaccurate OpenStreetMap building semantic masks and then was refined on a small number of accurately labeled datasets to generate large-scale building footprint maps. Encoder–decoder architectures are another favored variant. In the encoder, the spatial resolution of the image is gradually reduced to efficiently map features, which are then upsampled into a segmentation map with the original resolution in the decoder. A new cascaded multitask loss was introduced in the training of a SegNet model for the semantic segmentation of building footprints in [16]. From experiments conducted in [17], FC-DenseNet has been proved to achieve state-of-the-art performances in datasets with varied spatial resolutions.

2.2. Regularization of Building Footprints

Considering that the predicted building footprints from semantic segmentation networks usually have irregular shapes, the regularization methods aim to obtain refined building shapes with geometric constraints.

Most works adopt a two-stage pipeline, where the semantic segmentation of buildings is performed first, and then the boundary regularization is applied to the generated building footprint maps. Early efforts rely heavily on the heuristic design of features or constraint rules to do the regularization of building footprints. In order to obtain regularized boundaries, morphological filtering was applied in [18] to restrict the area and the number of building corners within a certain range. In [5,19,20], the length of the building boundary and its angle with respect to the principal orientation constraint were taken into consideration in the regularization step. However, it is still difficult for these manually designed regularization strategies to achieve satisfactory results when building shapes are diversified. Moreover, the accuracy metrics even decreased after the regularization algorithm was implemented [20]. Instead of manually defined criteria, a GAN with the combination of three loss functions (regularized loss, reconstruction loss, and adversarial loss) was proposed in [6] for the automatic regularization of the building footprints obtained from an FCN.

Recently, several studies have proposed to learn regularized building boundaries from remote sensing imagery in one framework. One study used deep structured active contours (DSAC) [21], which learn active contour models (ACM) [22] parameterizations with a CNN model. The other research proposed a deep active ray network (DARNet) [23] to learn ACM based on polar coordinates, which can avoid self-intersection. Although they encoded building regularization as a learnable problem, the results still did not strictly adhere to the realistic building boundaries. In addition, the initialization of both methods relied on some instance segmentation methods that were not incorporated into the learning process. Another framework that can directly generate regular building outlines from aerial images is PolygonCNN [24]. This end-to-end learning framework consists of two parts: an FCN for building semantic segmentation and a modified PointNet [25] for building regularization. However, these above methods still have a limitation in that these methods focus on only one building in each patch.

3. Methodology

3.1. Overview of RegGAN

Our goal is to achieve semantic labeling of each pixel with the aim of improving the boundary regularization results simultaneously. Recently, many semantic segmentation networks have achieved great success in this task, as they are able to automatically learn strong feature representations rather than manually engineered features. However, some problems have also emerged, such as irregular shapes, blurred boundaries, and so on. In parallel with semantic segmentation networks, boundary regularization networks, which produce regularized and visually pleasing boundaries, have also been shown to effectively improve semantic segmentation results. To improve the quality of building footprint maps without losing semantic accuracy, we propose to integrate semantic segmentation and boundary regularization in an end-to-end network.

As shown in Figure 2, RegGAN is a GAN network consisting of two modules: the generator G and the discriminator D. G aims to learn regularized building masks. D distinguishes between generated and ideal building masks. G tries to fool D, thus, these two modules compete with each other in a game. By doing so, G and D are capable of coadapting for the generation of the optimal output. The final output from the G is a more realistic version of buildings masks with sharp boundaries and corners.

The generator G has two paths: the regularization path and the reconstruction path. The regularization path aims at learning a function that maps ideal building masks y from the remote sensing imagery z. The regularization is accomplished by an encoder–decoder structure. It takes remote sensing imagery as input and generates the regularized building footprints from the encoder $E_1$ and decoder F. It should be noted that the discriminator does not directly estimate the ideal mask. Instead, the ideal building footprints are encoded and decoded by the encoder $E_2$ and the common decoder F in the reconstruction path. The design of the reconstruction path is for deriving a reconstructed version of y, which contributes to the training of the whole framework on two aspects. On the one hand, since the ideal mask consists of zeros and ones which will be easily distinguished by D, the output of the decoder with real values between zero and one can help to avoid the situation where D wins. On the other hand, the common decoder F is jointly trained with two encoders in the regularized and reconstructed path, thus the representation learning of F can be enhanced.

As the scale information is essential to the remote sensing imagery, a multiscale decision fusion is proposed in the discriminator D. D aims to distinguish if the presented building footprints are regularized masks from the regularization path or the reconstructed version from the reconstruction path. Specifically, D is separated into $D_1$ and $D_2$, which distinguish between ideal masks and regularized ones at two scales, respectively. By combing the decision information of each scale, a more comprehensive understanding of the difference between regularized and reconstructed building footprints can be assured.

Note that the ground reference y is only utilized in the training stage. In the inference stage, the remote sensing imagery z is taken as input through the regularization path (encoder $E_1$ and decoder F) for outputting the final building masks.

3.2. Objective Function of RegGAN

We use an end-to-end training pipeline for the supervised learning of RegGAN. The whole network is trained by the following loss function:

$$\begin{array}{c}\hfill L=L_G+L_D,\end{array}$$

(1)

where $L_G$ and $L_D$ are two loss functions for optimizing G and D, respectively.

G is learned with four loss functions: adversarial loss $L_{GAN}$, reconstruction loss $L_{REC}$, semantic loss $L_{SEG}$, and regularized loss $L_{REG}$, which are expressed as:

$$\begin{array}{c}\hfill L_G=\alpha ·L_{GAN}+\beta ·L_{REC}+\gamma ·L_{SEG}+\delta ·L_{REG},\end{array}$$

(2)

where $\alpha$, $\beta$, $\gamma$, and $\delta$ are hyperparameters to introduce relative weights on each loss.

The adversarial loss learns the mapping function from z to y, motivating $E_1$ and F to generate building masks that are similar to the ideal ones. It is defined as:

$$\begin{array}{c}\hfill L_{GAN}=-{\mathbb{E}}_z\left[D_1\left(F\left(E_1\left(z\right)\right)\right)\right]-{\mathbb{E}}_z\left[D_2\left(D_1\left(F\left(E_1\left(z\right)\right)\right)\right)\right].\end{array}$$

(3)

The output features of $D_1$ are taken as input for $D_2$.

The reconstruction loss ensures the correctness of the reconstructed version of ideal samples from the reconstruction path. This term can be calculated as:

$$\begin{array}{c}\hfill L_{REC}=-{\mathbb{E}}_y[y·log\left(F\left(E_2\left(y\right)\right)\right)].\end{array}$$

(4)

The semantic loss alleviates the information loss in the regularization path, enabling semantic correctness in the regularized building masks, and is defined as:

$$\begin{array}{c}\hfill L_{SEG}=-{\mathbb{E}}_z[y·log\left(F\left(E_1\left(z\right)\right)\right)].\end{array}$$

(5)

Apart from the aforementioned losses, regularized loss is exploited to leverage more information from the images, which is an important loss term to further improve the building regularization results. It consists of two types of loss, i.e., Potts loss [26] $L_{Potts}$ and normalized cut loss [27] $L_{ncut}$:

$$\begin{array}{c}\hfill L_{REG}=L_{Potts}+\zeta ·L_{Ncut},\end{array}$$

(6)

where $\zeta$ is a hyperparameter to add a weight on the $L_{Ncut}$. Note that $L_{Potts}$ and $L_{Ncut}$ are loss functions based on Potts [28] and normalized cut [29] models, respectively, which are graph clustering algorithms for image segmentation. The goal of these two loss terms is to accurately capture fine local details, helping to accurately localize building boundaries. Specifically, $L_{Potts}$ and $L_{Ncut}$ are defined as:

$$\begin{array}{c}\hfill L_{Potts}={\mathbb{E}}_z[\sum _k{S}^{k\top }W(1-S^k)],\end{array}$$

(7)

$$\begin{array}{c}\hfill L_{Ncut}={\mathbb{E}}_z[\sum _k\frac{S^{k\top }\widehat{W}(1-S^k)]}{1^{\top }\widehat{W}{S}^k},\end{array}$$

(8)

where $S=F\left(E_1\left(z\right)\right)$ is a k-way softmax output from the regularization path and $S^k$ represents the kth channel in S. In our case, $k=2$ as we have two classes, i.e., building and non-building. W and $\widehat{W}$ are affinity matrices that measure weights between neighboring pixels and are computed with Gaussian kernels over the RGBXY space (z and S) [27].

$D_1$ and $D_2$ are trained to distinguish from regularized and reconstructed building masks at two scales, and the objective function is defined as:

$$\begin{array}{c}\hfill L_D=-{\mathbb{E}}_y\left[D_1\left(F\left(E_2\left(y\right)\right)\right)\right]+{\mathbb{E}}_z\left[D_1\left(F\left(E_1\left(z\right)\right)\right)\right]\\ \hfill -{\mathbb{E}}_y\left[D_2\left(D_1\left(F\left(E_2\left(y\right)\right)\right)\right)\right]+{\mathbb{E}}_z\left[D_2\left(D_1\left(F\left(E_1\left(z\right)\right)\right)\right)\right]\end{array}$$

(9)

It is worth noting that an approximation of the Wasserstein distance is used in Equation (9) as it helps to stabilize and improve the training and achieves higher quality results [30].

4. Experiments

4.1. Dataset

In our experiment, we used two datasets with varying spatial resolutions, i.e., the ISPRS dataset [31] and the INRIA dataset [32].

(1) ISPRS dataset: The ISPRS dataset consists of 38 tiles of aerial imagery (Figure 3), which were acquired from Potsdam. Each aerial image has a size of $6000\times 6000$ pixels. The ground sampling distance (GSD) is 5 cm/pixel. The corresponding ground truth map includes six land cover classes. In our experiment, aerial images with RGB bands were utilized as remote sensing imagery. For data preparation, we selected the class of building in the original ground reference as the positive class. The training/validation/test split followed [33]. Specifically, the training set had 20 tiles (id: 7-7, 7-9, 7-10, 7-11, 7-12, 6-7, 6-8, 6-9, 6-10, 6-11, 6-12, 5-10, 5-11, 4-11, 4-12, 3-10, 3-11, 3-12, 2-10, 2-12), the validation set had 4 tiles (id: 7-8, 5-11, 4-10, 2-11), and the test set consisted of the remaining 14 tiles.

(2) INRIA dataset: The INRIA dataset [32] comprises images captured by airborne sensors and consists of 360 tiles of aerial images. Each aerial image (Figure 4) has $5000\times 5000$ pixels at a GSD of 30 cm/pixel. In this dataset, only the ground reference data of five regions (Austin, Chicago, Kitsap County, Western Tyrol, and Vienna) are made publicly available, and hence we only conducted experiments on these regions. According to the set up in [16], the data were split into training and validation sets in our research. The training set had 155 images, and 31 images with ids 6-36 were collected from each region, respectively. The validation set had 25 images, and 5 images with ids 1-5 were collected from each region, respectively. The statistics were derived from the validation set.

4.2. Experiment Setup

In this research, we compared our approach with different semantic segmentation and boundary regularization methods. We first trained eight different semantic segmentation networks: FCN-8s [11], U-Net [13], SegNet [12], FC-DenseNet [3], HRNet [14], HA U-Net [34], EPUNet [35], and ESFNet [36]. Afterward, we selected the semantic segmentation network that had relatively better performance on two datasets, in order to provide the preliminary result for the two-stage method [6], a method recently proposed as a boundary regularization network of buildings. Finally, we compared the proposed method and the two-stage method [6].

4.3. Training Details

For network training, all images and ground-truth masks were cropped into small patches with a size of $256\times 256$ pixels. All models were implemented on the PyTorch framework and trained on an NVIDIA Quadro P4000 GPU with 8 GB memory. PyTorch is a framework that provides an efficient and easy way for the development and deployment of machine learning models. For the network training, we used the Adam [37] optimization algorithm, which can minimize error functions by adaptively estimating first-order and second-order moments [37]. The learning rate of Adam was set as 0.0001. The training batch size of all models was set as 4. For RegGAN, we utilized ResNet34 [38] in the encoders E1 and E2. The decoder F had the same architecture as that in the two-stage method [6]. The discriminators $D_1$ and $D_2$ shared the same layer combinations, which included a sequence of convolutional layers and two max-pooling operations. $\alpha$, $\beta$, $\gamma$, and $\delta$ in Equation (2) were empirically set as 0.5, 1, 10, and 100, respectively. $\zeta$ in Equation (6) was empirically set as 0.01.

4.4. Evaluation Metrics

In this research, the performance of different models was evaluated in two aspects. The mask metrics were exploited to assess building masks. The boundary metrics were focused on the measurement of the quality of building boundaries.

4.4.1. Mask Metrics

In our experiments, two metrics including the F1 score and intersection over union (IoU) were chosen to evaluate the accuracy of the predicted semantic masks [39].

$$\begin{array}{c}\hfill \mathrm{F}1\mathrm{score}=\frac{2\times \mathrm{precision}\times \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}},\end{array}$$

(10)

$$\begin{array}{c}\hfill \mathrm{IoU}=\frac{TP}{TP+FP+FN},\end{array}$$

(11)

$$\begin{array}{c}\hfill \mathrm{precision}=\frac{TP}{TP+FP},\end{array}$$

(12)

$$\begin{array}{c}\hfill \mathrm{recall}=\frac{TP}{TP+FN},\end{array}$$

(13)

where $TP$, $FN$, and $FP$ represent the numbers of true positives, false negatives, and false positives.

4.4.2. Boundary Metrics

For the assessment of building boundaries, we selected the F-measure [40] and similarity index metric (SIM) as the two evaluation criteria in our experiments.

In order to derive the F-measure, we first used the Sobel edge operator [41] to extract building boundaries from predicted and ground reference building masks, respectively. The F-measure is defined as the geometric mean of the precision and recall of the extracted boundaries [39].

SIM measures the polygon similarity between the building polygons in two images [42]. For a set of ground reference polygons $G=\{g_1,\dots ,g_N\}$ and predicted building polygons $P=\{p_1,\dots ,p_M\}$, SIM is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathrm{SIM}=\frac{1}{{\sum }_i^N\left|g_i\right|}{\displaystyle \sum _i^N}\left|g_i\right|{\displaystyle \underset{1\le j\le M}{max}}\mathrm{IoU}(g_i,p_j)·\mathrm{GeoSim}(g_i,p_j),\end{array}\end{array}$$

(14)

where IoU$(g_i,p_j)$ and GeoSim$(g_i,p_j)$ are the IoU and geometric similarity between the jth predicted polygon and the ith ground reference polygon. GeoSim [42] measures the turning functions [43] between the ground truth and predicted polygons. GeoSim [42] can be utilized to assess the quality of building boundaries [44].

5. Results

This section presents the comparisons among RegGAN, semantic segmentation networks (i.e., FCN-8s [11], U-Net [13], SegNet [12], FC-DenseNet [3], HRNet [14], HA U-Net [34], EPUNet [35], and ESFNet [36]), and boundary regularization networks (i.e., two-stage method [6]). Their quantitative (

Table 1. Accuracymetrics (%) of different methods on the ISPRS dataset (GSD: 5 cm/pixel).

	Mask		Boundary
Method	F1-Score	IoU	SIM	F-Measure
FCN-8s [11]	81.82	69.23	52.80	18.71
U-Net [13]	85.37	74.48	58.11	19.32
SegNet [12]	87.81	78.28	54.84	17.11
FC-DenseNet [3]	88.34	79.11	58.91	20.76
HRNet [14]	85.82	75.16	55.77	17.96
HA U-Net [34]	88.09	79.00	59.20	20.59
EPUNet [35]	88.52	79.41	58.63	16.77
ESFNet [36]	88.65	80.23	57.76	19.67
Two-stage method [6]	87.86	78.35	64.01	19.56
RegGAN	90.40	82.48	65.94	22.27

and

Table 2. Accuracy metrics (%) of different methods on the INRIA dataset (GSD: 30 cm/pixel).

	Mask		Boundary
Method	F1-Score	IoU	SIM	F-Measure
FCN-8s [11]	84.79	73.60	68.96	27.01
U-Net [13]	84.83	73.66	69.48	28.98
SegNet [12]	84.43	73.05	68.68	28.16
FC-DenseNet [3]	84.66	73.41	67.94	28.96
HRNet [14]	81.52	68.81	66.02	23.75
HA U-Net [34]	84.28	72.82	69.18	26.64
EPUNet [35]	83.90	72.26	68.38	25.21
ESFNet [36]	83.65	71.90	68.35	24.63
Two-stage method [6]	84.59	73.29	69.73	29.56
RegGAN	86.74	76.50	71.44	32.17

) and qualitative results (see Figure 5 and Figure 6) were derived from the ISPRS and INRIA datasets.

Among all naive semantic segmentation networks, FC-DenseNet [3] performed relatively better on both datasets. The superiority in both mask and boundary results can be attributed to the dense block in the FC-DenseNet [3], which has greater feature extraction capability [17]. Therefore, we took the segmentation masks obtained by FC-DenseNet [3] as prior knowledge for the boundary regularization network: the two-stage method [6]. In other words, FC-DenseNet [3] was exploited to perform the initial semantic segmentation of buildings, and then this result was further processed by the regularization network proposed in [6] to generate the structured building footprints. Afterward, we made a further investigation of RegGAN, FC-DenseNet [3], and the two-stage method [6].

We first compared the results against those of semantic segmentation networks. From the statistical results in Table 1, RegGAN not only obtains increments on mask metrics (e.g., above 2% in IoU), but also improves on boundary metrics (e.g., above 6% in SIM). On the INRIA dataset (cf. Table 2), RegGAN surpasses other semantic segmentation networks by more than 3% in IoU. Moreover, the boundary metrics provided by our method are much higher. For instance, RegGAN shows about 3% improvement (71.44% vs. 67.94% in SIM, and 32.17% vs. 28.96% in F-measure) when compared to FC-DenseNet [3]. This confirms that RegGAN is superior to naive semantic segmentation networks. On the one hand, the proposed method can significantly boost performance in both mask and boundaries metrics. On the other hand, RegGAN is able to realize the semantic segmentation and regularization of buildings simultaneously in an end-to-end manner.

The quantitative results obtained by the two-stage method [6] on the ISPRS and INRIA datasets were further compared with that obtained by RegGAN. It was observed from both boundary and mask metrics on both datasets that RegGAN achieved better performance. In particular, on both datasets, the IOU of our approach was increased by more than 3%, and the SIM was improved by more than 1.5%. This demonstrates the effectiveness and robustness of RegGAN for the task of building footprint generation, as our approach can improve boundary regularization results while preserving semantic correctness.

Figure 5 presents a visual comparison among various methods on the ISPRS dataset. The building footprints generated by RegGAN are more accurate and reliable, as they coincide better with the ground reference when compared with the other two methods. The results provided by FC-DenseNet [3] were taken as the input of the two-stage method [6] for boundary regularization. However, if the results provided by FC-DenseNet [3] were not good, this also limited the potential of the two-stage method [6]. For instance, in Figure 5, some building structures are omitted by FC-DenseNet [3] negatively affecting the performance of the two-stage method [6]. In contrast, RegGAN is capable of accurately capturing more geometric details. It can be clearly seen that more refined boundary representations and detailed building structures can be acquired by our approach. This is due to the fact that RegGAN guides the network to initially learn more regular shapes of buildings from the raw images and get rid of results from other methods as preliminary inputs.

The results obtained by the different methods (Figure 6) correspond to the example area selected on the INRIA dataset. This scene belongs to a residential region where nearly connected buildings are detected as large buildings by most naive semantic segmentation networks (e.g., FCN-8s [11], HRNet [14], and HA U-Net [34]). It is notable that in Figure 6j, RegGAN is able to effectively distinguish consecutive buildings. Moreover, building boundaries delineated by the proposed approach are more rectilinear and precise when compared to other methods. This suggests that RegGAN is still promising in such challenging situations.

6. Discussion

6.1. Ablation Study

In RegGAN, regularized loss and a multiscale discriminator are employed to further improve the building regularization results. In order to investigate the effectiveness of both designs, we performed ablation studies with two competitors, i.e., RegGAN (no regularized loss) and RegGAN (no multiscale discriminator), where the regularized loss and multiscale discriminator were removed from the proposed method, respectively. In the RegGAN (no regularized loss), $\delta =0$. For RegGAN (no multiscale discriminator), the $L_D$ in Equation (9) is defined as $L_D=-{\mathbb{E}}_y\left[D_2\left(D_1\left(F\left(E_2\left(y\right)\right)\right)\right)\right]+{\mathbb{E}}_z\left[D_2\left(D_1\left(F\left(E_1\left(z\right)\right)\right)\right)\right]$. That is to say, we only differentiate from regularized and reconstructed building footprints at one scale.

The statistical results are shown in

Table 3. Accuracy Metrics (%) of ablation studies on the ISPRS dataset (GSD: 5 cm/pixel).

	Mask		Boundary
Method	F1-Score	IoU	SIM	F-Measure
RegGAN (no regularized loss)	88.91	80.03	63.40	21.51
RegGAN (no multiscale discriminator)	87.71	78.12	63.29	17.18
RegGAN	90.40	82.48	65.94	22.27

and

Table 4. Accuracy metrics (%) of ablation studies on the INRIA dataset (GSD: 30 cm/pixel).

	Mask		Boundary
Method	F1-Score	IoU	SIM	F-Measure
RegGAN (no regularized loss)	85.60	74.83	69.51	29.20
RegGAN (no multiscale discriminator)	83.56	71.77	69.78	27.49
RegGAN	86.74	76.50	71.44	32.17

, and visual results are illustrated in Figure 7 and Figure 8. As can be seen from the statistical results, the regularized loss and a multiscale discriminator both bring a significant improvement in mask and boundary metrics, leading to a positive influence on the performance of RegGAN. When compared to the RegGAN (no regularized loss), the use of both designs brings a 2.45% improvement in IoU and a 2.54% improvement in SIM on the ISPRS dataset, respectively. On the INRIA dataset, the proposed method with both designs outperforms RegGAN (no multiscale discriminator) by 4.73% in IoU and 1.66% in SIM, respectively.

Figure 7 and Figure 8 show visual comparisons of the segmentation results, which demonstrate that the performance of RegGAN is further boosted by the leverage of the regularized loss and a multiscale discriminator simultaneously. In Figure 7, the methods without these two designs wrongly identify impervious surfaces as buildings. This is because, impervious surfaces have similar appearances to buildings, leading to a misjudgment. In the example area presented in Figure 8, many buildings are occluded by trees and the methods without these two designs are not capable of providing accurate building boundaries. On the contrary, the use of these two designs is able to not only avoid such false alarms but also obtain refined building boundaries.

Thanks to both designs, RegGAN is capable of more effectively leveraging the information to improve results. Specifically, the regularized loss makes use of the RGB information of neighboring pixels, which helps to obtain refined building segmentation masks. The multiscale discriminator takes advantage of decision information at different scales to enhance the semantic correctness of learned masks. This is beneficial for the generator to learn more regularized building footprints.

6.2. Time Efficiency of Different Methods

Apart from the accuracy of the generated building footprint maps, the time efficiency of different methods is also of great interest in practical applications. In this regard, we investigated the training and inference time of the different methods (Figure 9). Specifically, the training time refers to the time cost of one iteration (batch size was 4), while the inference time measures the time cost of a single batch.

Among all naive semantic segmentation networks, FCN-8s [11] and ESFNet [36] are superior to other methods in terms of the time costs for training and inference. Nevertheless, FCN-8s [11] performs much worse than other approaches on the ISPRS dataset (Table 1). Similarly, the accuracy metrics of ESFNet [36] are much lower than those of other methods on the INRIA dataset (Table 2).

Although the two-stage method [6] is able to obtain increments on both datasets in terms of boundary metrics, its time cost for both training and inference is the highest among all methods. One important reason is that it requires the predicted segmentation maps as input. In other words, another semantic segmentation network should be trained separately, leading to low time efficiency. Compared with the two-stage method [6], RegGAN has not only largely improved the mask and boundary accuracy, but also significantly reduced the time cost for both training and inference. This suggests that RegGAN can achieve a good trade-off between computational time and accuracy metrics.

7. Conclusions

In this paper, RegGAN, a new end-to-end semantic segmentation and boundary regularization network, was proposed to generate building footprint maps. RegGAN is a GAN model comprising two modules: a generator and a discriminator. More specifically, two paths in the generator were designed to learn regularized and reconstructed building footprints, respectively. Afterward, a multiscale discriminator was proposed to differentiate between them. In addition, we demonstrated that the regularized loss could boost the performance of the method, as it helped to obtain refined building boundaries. The performance of the RegGAN was validated on two datasets with varying resolutions: the INRIA dataset (30 cm/pixel) and the ISPRS dataset (5 cm/pixel). Experimental results demonstrated that RegGAN had the potential to generate more satisfying building footprints. On the one hand, building boundaries were more refined. On the other hand, false alarms where non-building classes are incorrectly identified as “building”, were alleviated.