Combining Satellite Image Standardization and Self-Supervised Learning to Improve Building Segmentation Accuracy

Highlights

What are the main findings?

• Performing atmospheric correction before pan-sharpening improves the accuracy of building segmentation.
• The two pretext tasks are specifically designed to consider building features in satellite imagery.

What is the implication of the main finding?

• The newly developed multi-task SSL network performs better than existing SSL methods in building segmentation.
• The proposed method works effectively in situations where labeled satellite data is limited and on personal computers.

Abstract

Many research fields, such as urban planning, urban climate and environmental assessment, require information on the distribution of buildings. In this study, we used U-Net to segment buildings from WorldView-3 imagery. To improve the accuracy of building segmentation, we undertook two endeavors. First, we investigated the optimal order of atmospheric correction (AC) and panchromatic sharpening (pan-sharpening) and found that performing AC before pan-sharpening results in higher building segmentation accuracy than after pan-sharpening, increasing the average IoU by 9.4%. Second, we developed a new multi-task self-supervised learning (SSL) network to pre-train VGG19 backbone using 21 unlabeled WorldView images. The new multi-task SSL network includes two pretext tasks specifically designed to take into account the characteristics of buildings in satellite imagery (size, distribution pattern, multispectral, etc.). Performance evaluation shows that U-Net combined with an SSL pre-trained VGG19 backbone improves building segmentation accuracy by 15.3% compared to U-Net combined with a VGG19 backbone trained from scratch. Comparative analysis also shows that the new multi-task SSL network outperforms other existing SSL methods, improving building segmentation accuracy by 3.5–13.7%. Moreover, the proposed method significantly saves computational costs and can effectively work on a personal computer.

1. Introduction

Buildings are important infrastructure for human activities, and understanding their precise distribution can contribute to research in areas such as urban planning, urban climate, and environmental assessment [1,2,3]. It has been demonstrated that combining high-spatial resolution satellite imagery with deep learning approaches enables fast and accurate extraction of buildings [4]. However, two challenges remain in this task.

First, since each satellite image has different atmospheric effects, the reflectance spectra of buildings obtained from different satellite images may differ significantly, which brings great challenges to the process of building segmentation [5,6]. Furthermore, satellite imagery with a panchromatic (PAN) band, such as WorldView-3 imagery, typically requires a panchromatic sharpening (pan-sharpening) process to generate images that provide both clear textural information and sufficient spectral information before building semantic segmentation can be performed [7]. Therefore, the atmospheric correction (AC) and pan-sharpening processes are two important steps of imagery standardization before building segmentation. These processes remove/mitigate environmental influences and allow building segmentation according to differences in land cover type only, thereby increasing the generalizability of the developed model and the reliability of the generated building maps. However, some studies prefer to perform AC before pan-sharpening [8,9,10], while other studies prefer to perform AC after pan-sharpening [11,12]. Considering that atmospheric effects may be different for the PAN band and multispectral (MS) bands, it is necessary to clarify how the order of AC and pan-sharpening processes affects the accuracy of building segmentation.

The second challenge is the amount of labeled satellite data available to train deep learning-based models for building segmentation. Since satellite images are often taken under different conditions (e.g., different atmospheric effects, different locations with different land uses and covers, different seasons, etc.), a large amount of labeled satellite images are required to train deep learning-based models for more accurate building segmentation. This is a very time-consuming and labor-intensive task. To address the issue of limited manually labeled satellite images, self-supervised learning (SSL) approaches have been recently proposed to extract building features from unlabeled satellite images through single or multiple pretext tasks (pre-training a backbone), and their usefulness has also been confirmed [13,14,15]. Compared with other pre-training approaches, there are two main advantages of using SSL: (1) SSL allows us to continuously add new unlabeled satellite images, avoiding the problem of a lack of satellite images in existing datasets (e.g., ImageNet is primarily composed of photographs) and the impact of different object distribution patterns between satellite images and photographs (no dominant objects vs. the presence of dominant objects such as dogs, cats, etc.); (2) SSL can fully capture the characteristics of multiple satellite image bands, not just red, green, and blue bands (RGB). Therefore, the SSL approach is also considered in this study.

Generally, there are two main categories of pretext tasks in SSL: generative learning methods and contrastive learning methods [16,17]. The former can learn low-level features (e.g., color, texture, shape, etc.) by minimizing the reconstruction loss between the reconstructed image and the original input image [16,18,19]. The latter can learn high-level features (e.g., semantic categories, structure of objects, part-whole relationship of objects, etc.) by measuring the contrastive loss between positive pairs or by considering the contrastive loss of positive and negative pairs simultaneously [20,21].

Generative learning methods widely use image inpainting as their main pretext task, allowing the model to infer artificially damaged regions from the surrounding undamaged pixels (the texture of the remaining buildings, etc.) [19]. To implement image inpainting, most studies use a block mask to generate a damaged input image. For example, Li et al. proposed to mask approximately 30% of an unmanned aerial vehicle (UAV) image (256 × 256-pixel, spatial resolution 0.08 m) using a single large block [22]. Chen et al. proposed to use multiple small blocks to mask 5% of a Google image (512 × 512-pixel, spatial resolution 1 m) [23]. However, there are two problems with applying the above block mask strategies to WorldView-3 imagery (spatial resolution 0.3 m after pan-sharpening). First, when 30% of a 512 × 512-pixel WorldView-3 image is masked in one large block, the block size reaches more than 7000 m², which is usually much larger than the average size of a building in Japan (about 100 m², Ministry of Internal Affairs and Communications 2023 [24]). Previous studies have pointed out that to completely mask a building may result in the building being lost and its features cannot be learned during the inpainting process [25]. Second, if the block mask ratio is too small (e.g., 5% as used in [23]), more training epochs will be required to effectively learn features across the entire 512 × 512-pixel WorldView-3 image, which will increase the computational cost. Therefore, to improve building segmentation accuracy, a new image inpainting pretext task specifically designed to consider the characteristics of buildings in pan-sharpened WorldView-3 imagery (e.g., size, distribution pattern) is needed.

Another representative generative learning method is colorful image colorization (CIC), which converts an input color image into a grayscale image and trains a model to reproduce its color version [18]. However, this method only works for three-band imagery (i.e., RGB imagery) and is not applicable to multi-band satellite imagery. On the other hand, previous studies have demonstrated that multi-band images provide more information and more reliable cues than RGB images and can more effectively distinguish building materials from the background [26,27,28,29,30]. Therefore, it is desirable to develop a new pretext task that can fully learn spectral characteristics from multi-band satellite images, which will enable the segmentation model to better distinguish different building materials from the surrounding background and improve the segmentation accuracy.

Consequently, the objectives of this study are as follows:

(1)

To clarify how the order of the AC and pan-sharpening processes affects the accuracy of building segmentation and recommend the most effective standardization method.

(2)

To propose two new pretext tasks that take into account the features of objects distributed in WorldView-3 images.

(3)

To design a novel multi-task SSL network that combines generative learning methods (using two new pretext tasks) and contrastive learning methods to improve the accuracy of building segmentation.

Through this research, it is hoped to address the limitations of existing methods (inconsistent image standardization order and pretext task design unsuitable for WorldView-3 imagery) and develop a robust method that can be used for urban growth monitoring, land use analysis, and disaster assessment under conditions of limited labeled satellite imagery.

2. Materials

In this study, multiband WorldView series satellite images were collected, as shown in Table 1. Dataset I contains three WorldView-3 images (located at Sakaiminato, Sanuki, and Tsukuba, Japan, respectively, Figure 1) with four MS bands (450–510 nm, 510–580 nm, 630–690 nm, and 770–895 nm) and one PAN band (450–800 nm). Buildings in each image were manually labeled using ArcGIS (v10.8.1), resulting in 21,951, 14,828, and 18,389 objects, respectively. Images were then cropped into patches of 512 × 512 pixels, leaving a 20% overlap between adjacent patches to avoid the “stitching problem” [31]. As a result, 1632, 1683, and 1681 patches were obtained in Sakaiminato, Sanuki, and Tsukuba regions, respectively.

Dataset I was used for two purposes: (1) to investigate the optimal combination of AC and pan-sharpening processes, and (2) to evaluate the performance of the newly designed multi-task SSL network. For these purposes, we split the patches of two optional regions (Sakaiminato and Sanuki, Sakaiminato and Tsukuba, or Sanuki and Tsukuba) into training and validation datasets in a ratio of 8:2 and used the patches of the remaining one region as the test dataset. Furthermore, to increase the diversity of the training dataset, all patches in the training dataset were augmented by geometric transformations (Figure 2, [32]), including offline image augmentation (i.e., horizontal flipping, vertical flipping, and both) and online image augmentation (i.e., random scale cropping, and the 4-image mosaicking from [33]). The average test accuracy of the above three combinations was considered as the final result.

Datasets II, III, and IV were used to pre-train VGG19 backbone through SSL. As all collected images were raw data (digital numbers), AC and pan-sharpening processes were performed to generate surface reflectance MS images with a spatial resolution of 30 cm. The same four bands as those in Dataset I were extracted and applied for further analysis. All 21 images in Datasets II, III, and IV were then cropped into patches of 512 × 512 pixels, resulting in a total of 85,636 patches for pre-training VGG19 backbone via SSL. To reduce the risk of overfitting the VGG19 backbone, we kept 5% of the patches as a validation dataset, considering the maximum utilization of patches for training. Additionally, offline image augmentation (i.e., horizontal flipping, vertical flipping, and both) was also performed to increase the diversity of the training dataset.

Table 1. Details of the WorldView (WV) datasets used in this study. A patch is defined as a small region of 512 × 512 pixels cropped from an image. MS: Multispectral, PAN: Panchromatic. This study: image purchased from Maxar Technologies Inc., Westminster, CO, USA.

Dataset No.	Satellite Type	No. of Images	No. of Bands		Resolution (m)		No. of Patches	No. of Objects	Source
			MS	PAN	MS	PAN
Dataset I	WV-3	3	4	1	1.2	0.3	1632	21,951	This study
							(Sakaiminato, Japan)
							1683	14,828
							(Sanuki, Japan)
							1681	18,389
							(Tsukuba, Japan)
Dataset II	WV-3	9	4	1	1.2	0.3	9593	-	This study
							(9 regions in Japan)
Dataset III	WV-3	1	8	1	1.2	0.3	7392	-	[34]
							(Washington, D.C., USA)
Dataset IV	WV-2	2	8	1	2	0.5	68,651	-	[35]
	WV-3	9	8	1	1.2	0.5	(11 regions outside Japan)

Table 1. Details of the WorldView (WV) datasets used in this study. A patch is defined as a small region of 512 × 512 pixels cropped from an image. MS: Multispectral, PAN: Panchromatic. This study: image purchased from Maxar Technologies Inc., Westminster, CO, USA.

Dataset No.	Satellite Type	No. of Images	No. of Bands		Resolution (m)		No. of Patches	No. of Objects	Source
			MS	PAN	MS	PAN
Dataset I	WV-3	3	4	1	1.2	0.3	1632	21,951	This study
							(Sakaiminato, Japan)
							1683	14,828
							(Sanuki, Japan)
							1681	18,389
							(Tsukuba, Japan)
Dataset II	WV-3	9	4	1	1.2	0.3	9593	-	This study
							(9 regions in Japan)
Dataset III	WV-3	1	8	1	1.2	0.3	7392	-	[34]
							(Washington, D.C., USA)
Dataset IV	WV-2	2	8	1	2	0.5	68,651	-	[35]
	WV-3	9	8	1	1.2	0.5	(11 regions outside Japan)

3. Methods

In this study, we use U-Net with a VGG19 backbone to segment buildings. This choice is based on previous work showing that U-Net with a pre-trained VGG19 backbone achieves higher building segmentation accuracy compared to the combinations of U-Net and other backbones (e.g., DarkNet19, ResNet50, DarkNet53) [36]. Furthermore, the main objective of this study is not to investigate the performance of existing backbones, but to design a multi-task SSL network that can properly consider the building distribution characteristics of WorldView-3 images for building segmentation. This is another reason why we only used VGG19 as the backbone in this study.

3.1. Satellite Image Standardization

We designed three experiments to investigate the most effective method for standardizing satellite imagery. They are as follows: (a) only pan-sharpening, (b) first performing AC on the MS and PAN bands, respectively, and then performing pan-sharpening, and (c) vice versa. For AC, two widely used algorithms, the Second Simulation of the Satellite Signal in the Solar Spectrum (6S) [37,38] and the Fast Line-of-Sight Atmospheric Analysis of Spectral Hypercubes (FLAASH) [39], were selected [40,41,42,43].

In pan-sharpening, the component substitution-based Gram–Schmidt method [44] and the color-based Nearest Neighbor Diffusion method (NNDiffuse) [45] are representative and widely used methods [46]. NNDiffuse performs pan-sharpening by assuming that the spectrum of each pixel in the pan-sharpened image is a weighted linear mixture of the spectra of its neighboring superpixels. By treating each spectrum as the smallest operational unit rather than processing each band individually, NNDiffuse can preserve both spatial detail and spectral fidelity [47]. In contrast, the Gram–Schmidt method often results in changes in spectral characteristics after fusion, resulting in degradation of color information [48,49]. Therefore, in this study, we only consider the NNDiffuse method for pan-sharpening.

Based on the above considerations, a total of five image standardization methods were tested in this study (Figure 3). Details are as follows:

(1)

only pan-sharpening is performed, no AC is performed;

(2)

AC is first performed on the MS and PAN bands using 6S, and then pan-sharpening is performed;

(3)

AC is first performed only on the MS bands using FLAASH (FLAASH cannot be applied to PAN band), and then pan-sharpening is performed;

(4)

pan-sharpening is first performed, and then AC is performed on the pan-sharpened MS bands using 6S;

(5)

pan-sharpening is first performed, and then AC is performed on the pan-sharpened MS bands using FLAASH.

3.2. Designing a Multi-Task SSL Network to Pre-Train the VGG19 Backbone

Figure 4 shows the newly designed multi-task SSL network in this study. The new multi-task SSL network includes three pretext tasks: (1) image inpainting (new), (2) image spectrum recovery (new), and (3) contrastive learning (as in previous studies, e.g., [22,23]). Note that image inpainting (Pretext Task 1) and image spectrum recovery (Pretext Task 2) share the same decoder structure (Appendix A,

Table A1. Decoder structure of the new multi-task SSL network. Interpolate: Up-sampling using bilinear interpolation with a scale factor of 2. Conv: Convolutional layer. BN: Batch normalization layer. ReLU: Rectified Linear Unit (ReLU) activation function. Addition: Element-wise addition between the feature map from the decoder and the corresponding encoder layer (skip connection). Linear: Fully connected layer (linear transformation). Average Pooling: Adaptive average pooling to reduce feature maps to a single vector. “-“: no input required.

Pretext Task	No. of Layers	Layer	Input Channels	Output Channels	Output Size	Kernel Size	Stride	Padding
Image Inpainting	1	Interpolate	-	-	-	-	-	-
	2	Conv	512	512	32 × 32	3 × 3	1	1
	3	BN	512	-	-	-	-	-
	4	ReLU	-	-	-	-	-	-
	5	Addition	-	-	-	-	-	-
	6	Interpolate	-	-	-	-	-	-
	7	Conv	512	256	64 × 64	3 × 3	1	1
	8	BN	256	-	-	-	-	-
	9	ReLU	-	-	-	-	-	-
	10	Addition	-	-	-	-	-	-
	11	Interpolate	-	-	-	-	-	-
	12	Conv	256	128	128 × 128	3 × 3	1	1
	13	BN	128	-	-	-	-	-
	14	ReLU	-	-	-	-	-	-
	15	Addition	-	-	-	-	-	-
	16	Interpolate	-	-	-	-	-	-
	17	Conv	128	64	256 × 256	3 × 3	1	1
	18	BN	64	-	-	-	-	-
	19	ReLU	-	-	-	-	-	-
	20	Addition	-	-	-	-	-	-
	21	Interpolate	-	-	-	-	-	-
	22	Conv	64	32	512 × 512	3 × 3	1	1
	23	BN	32	-	-	-	-	-
	24	ReLU	-	-	-	-	-	-
	25	Conv	32	4	512 × 512	3 × 3	1	1
Spectrum Recovery	1	Interpolate	-	-	-	-	-	-
	2	Conv	512	512	32 × 32	3 × 3	1	1
	3	BN	512	-	-	-	-	-
	4	ReLU	-	-	-	-	-	-
	5	Addition	-	-	-	-	-	-
	6	Interpolate	-	-	-	-	-	-
	7	Conv	512	256	64 × 64	3 × 3	1	1
	8	BN	256	-	-	-	-	-
	9	ReLU	-	-	-	-	-	-
	10	Addition	-	-	-	-	-	-
	11	Interpolate	-	-	-	-	-	-
	12	Conv	256	128	128 × 128	3 × 3	1	1
	13	BN	128	-	-	-	-	-
	14	ReLU	-	-	-	-	-	-
	15	Addition	-	-	-	-	-	-
	16	Interpolate	-	-	-	-	-	-
	17	Conv	128	64	256 × 256	3 × 3	1	1
	18	BN	64	-	-	-	-	-
	19	ReLU	-	-	-	-	-	-
	20	Addition	-	-	-	-	-	-
	21	Interpolate	-	-	-	-	-	-
	22	Conv	64	32	512 × 512	3 × 3	1	1
	23	BN	32	-	-	-	-	-
	24	ReLU	-	-	-	-	-	-
	25	Conv	32	4	512 × 512	3 × 3	1	1
Contrastive Learning	1	Average Pooling	-	-	1 × 1	-	-	-
	2	Linear	512	1024	-	-	-	-
	3	ReLU	-	-	-	-	-	-
	4	Linear	1024	512	-	-	-	-

), since they are two similar tasks in which both are treated as recovering a 3D image matrix (channel, height, and width). Details are as follows.

3.2.1. New Pretext Task 1: Image Inpainting Using Multiple Small Blocks

The image inpainting task allows theVGG19 backbone to learn the textural features of objects by restoring masked areas in the input image [19]. To consider the building distribution characteristics of WorldView-3 images, we propose to use multiple small blocks in Pretext Task 1. Specifically, to avoid the problem of buildings being completely masked, we set the size of each small block to 32 × 32 pixels (approximately 92.16 m²) taking into account the average building area (approximately 100 m²). We also conducted an experiment to verify that this choice was appropriate (

Table A2. Accuracy of building segmentation using the SSL network with only Pretext Task 1 under different small block sizes. (A): use the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): use the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): use the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU and OA values are highlighted in bold.

No.	Method	Size of Small Blocks	Mean		(A)		(B)		(C)
			IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	Inpainting with small blocks	16 × 16	0.625	0.953	0.648	0.961	0.636	0.959	0.591	0.938
2		32 × 32	0.628	0.953	0.642	0.959	0.646	0.959	0.596	0.942
3		64 × 64	0.617	0.951	0.655	0.962	0.595	0.949	0.600	0.943

in Appendix A). On the other hand, to consider the balance between the masked and remaining areas, the proportion of small blocks in the image was set to 0.3, as this value was found in tests to yield the best performance in downstream segmentation tasks (

Table A3. Accuracy of building segmentation using the new SSL network with different proportions of small blocks used in Pretext Task 1. (A): use the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): use the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): use the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU and OA values are highlighted in bold.

No.	Method	Proportion of Small Blocks in Pretext Task 1	Mean		(A)		(B)		(C)
			IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	Multiple task-based SSL network (This study)	0.05	0.595	0.946	0.571	0.945	0.620	0.955	0.594	0.940
2		0.15	0.642	0.955	0.661	0.963	0.663	0.962	0.603	0.942
3		0.30	0.663	0.958	0.692	0.968	0.678	0.964	0.620	0.943
4		0.40	0.630	0.953	0.655	0.962	0.609	0.952	0.624	0.946
5		0.50	0.626	0.952	0.638	0.958	0.623	0.954	0.615	0.943
6		0.60	0.621	0.950	0.655	0.962	0.582	0.946	0.625	0.944
7		0.75	0.641	0.955	0.651	0.960	0.652	0.961	0.621	0.945

in Appendix A). Furthermore, these small blocks were set to randomly distributed within each image without overlapping with each other (Figure 4, Pretext Task 1).

Generally, there are three widely used image transformation methods to damage the masked image for each small block (32 × 32 pixels): image rotation (0 degree, 90 degrees, 180 degrees, or 270 degrees), image flipping (horizontal, vertical, horizontal and vertical, or unchanged), and image color jittering (brightness, contrast, saturation, and hue). We tested three image damage strategies to determine which strategy is most effective at learning building texture features. They are as follows: (1) each small block masked area uses different combinations of all three transformation methods (SB1); (2) all small block masked areas use the same combination of all three transformation methods (SB2); (3) all small block masked areas are simply set to zero (SB3).

The loss function for image inpainting ($L_{inpaint}$) is defined as follows:

$$L_{inpaint}={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|I_i-\right.\left.{I^{\prime }}_i\right|·\left|I_i-\right.\left.{I^{″}}_i\right|$$

(1)

where n is the number of images (512 × 512 pixels) in one batch (n = 8), $I_i$ is the original image, ${I^{\prime }}_i$ is the image with pixels in multiple small blocks transformed, and ${I^{″}}_i$ is the image with pixels in multiple small blocks restored (i = 1, 2, … n, where n is the number of images in the batch).

3.2.2. New Pretext Task 2: Image Spectrum Recovery

Unlike RGB imagery, WorldView-3 imagery has multiple bands, which can provide more information to distinguish buildings from other land covers. To take full advantage of this property, we propose a novel pretext task that enables the VGG19 backbone to learn the spectral features of each land cover (especially buildings) by recovering damaged bands in the input image. We performed four experiments to impair the band information, trying to find the most effective way to enable the VGG19 backbone to properly learn the spectral features of each land cover. Details of the four experiments are as follows: (1) randomly select one band and set the value of the selected band to zero (R1Z); (2) randomly select two bands, average their values, and replace the values of the two selected bands with the average value (R2A); (3) randomly select three bands, average their values, and replace the values of the three selected bands with the average value (R3A); (4) average all four bands and replace the values of the four bands with the average value (4A).

The loss function for image spectrum recovery ($L_{spectrum}$) is defined as follows:

$$L_{spectrum}={\displaystyle \frac{1}{n}}\sum _{s=1}^n\left|I_s\right.-\left.{I^{\prime }}_s\right|·\left|I_s- \right.\left.{I^{″}}_s\right|$$

(2)

where n is the number of images (512 × 512 pixels) in one batch (n = 8), $I_s$ is the original image, ${I^{\prime }}_s$ is the image containing the faulty band information, and ${I^{″}}_s$ is the image containing the restored band information (s = 1, 2, … n, where n is the number of images in the batch).

3.2.3. Pretext Task 3: Contrastive Learning

For Pretext Task 3, we used SimCLR, the most representative contrastive learning method that learns high-level features through data augmentation, and a large number of positive and negative sample pairs [22,50,51]. The cosine similarity (sim) between positive and negative pairs were calculated using the method described in [50]:

$$sim(I, I^{\prime })={\displaystyle \frac{{\phi }^T\left(I\right)·\phi \left(I^{\prime }\right)}{{‖\phi \left(I\right)‖}_2·{‖\phi \left(I^{\prime }\right)‖}_2}}$$

(3)

where $I$ is the original image, $I^{\prime }$ is the transformed image, $\phi \left(I\right)$ and $\phi \left(I^{\prime }\right)$ represent the images features extracted by the VGG19 backbone, respectively. The ${‖·‖}_2$ represents the $l-2$ norm.

The loss function for contrastive learning ($L_{contrastive}$) is defined as follows:

$$L_{contrastive}={\displaystyle \frac{1}{n}}\sum _{j=1}^n\left(-log{\displaystyle \frac{exp\left(sim\right(I_j, {I^{\prime }}_j\left)\right)}{{\sum }_{k\ne j}exp\left(sim\right(I_j,I_k\left)\right)}}-log{\displaystyle \frac{exp\left(sim\right(I_j, {I^{\prime }}_j\left)\right)}{{\sum }_{k\ne j}exp\left(sim\right({I^{\prime }}_j,I_k\left)\right)}}\right)$$

(4)

where n is the number of images (512 × 512 pixels) in one batch (n = 8), $(I_j, {I^{\prime }}_j)$ is the positive pair, and $(I_j,I_k)$ and ${(I^{\prime }}_j,I_k)$ are the negative pairs.

3.3. U-Net with SSL Pre-Trained VGG19 Backbone for Building Segmentation

Figure 5 shows the flowchart of the proposed method for building segmentation. All input WorldView images are first processed using the most effective standardization method obtained from Section 3.1. The U-Net with SSL pre-trained VGG19 backbone in Section 3.2 is then fine-tuned using Dataset I. The output is a predicted building distribution map. Dice loss [52] was used as the loss function for building segmentation.

Implementation details for pre-training the VGG19 backbone are as follows: batch size = 8, total number of training epochs = 20, initial learning rate = 0.0001 and decreased to 95% every 2 epochs, and total loss function of SSL = 6 × $L_{inpaint}$ + 13 × $L_{spectrum}$ + $L_{contrastive}$ (the weights for each loss function were set according to the range of loss values observed when each pretext task was used individually, ensuring that no single pretext task dominates the overall SSL loss). Training is stopped just before the loss on the validation dataset starts to increase. Additionally, a random seed was set to ensure experimental reproducibility.

Note that a U-Net with a non- pretrained VGG19 backbone was used to explore the optimal combination of AC and pan-sharpening processes. Implementation details for training the U-Net are as follows: batch size = 16, total number of training epochs = 200, initial learning rate = 0.001 and decreased to 90% every 10 epochs. Training is stopped just before the loss on the validation dataset starts to increase. A random seed was also set to ensure experimental reproducibility. These implementation details are also used to fine-tune the U-Net and SSL pre-trained VGG19 backbone.

3.4. Accuracy Assessment

In this study, the Bhattacharyya Distance (D_B) [53] was used to calculate the similarity between standardized satellite images as shown in the following equation:

$$D_B(H_1,H_2)=-\mathit{ln}\left(BC\left(H_1,H_2\right)\right)=-\mathit{ln}\left(\sum _i\sqrt{H_1\left(i\right)H_2\left(i\right)}\right)$$

(5)

where $H_1$,$H_2$ are the histograms of the two images being compared, $H_1\left(i\right)$ and $H_2\left(i\right)$ represent the probability of occurrence of the i-th value, and $BC\left(H_1,H_2\right)$ represents the Bhattacharyya coefficient.

Furthermore, the results of the building segmentation are evaluated using widely used pixel-level metrics such as Intersection over Union (IoU) and Overall Accuracy (OA), as shown in the following equations:

$$IoU={\displaystyle \frac{TP}{TP+FP+FN}}$$

(6)

$$OA={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(7)

where TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively.

4. Results

4.1. The Most Effective Method for Standardizing Satellite Imagery

Table 2. Histogram similarities between two of the three images of different regions using different image standardization methods. D_B: Bhattacharyya Distance; image (a): Sakaiminato; image (b): Sanuki; image (c): Tsukuba. The lowest D_B values are highlighted in bold and represent the highest histogram similarity.

No.	Method	Mean DB	DB Between Image (a) and Image (b)	DB Between Image (b) and Image (c)	DB Between Image (c) and Image (a)
1	Only NNDiffuse	0.400	0.459	0.486	0.257
2	6S + NNDiffuse	0.153	0.140	0.113	0.207
3	FLAASH + NNDiffuse	0.369	0.295	0.489	0.322
4	NNDiffuse + 6S	0.252	0.225	0.274	0.258
5	NNDiffuse + FLAASH	0.378	0.389	0.464	0.279

shows the histogram similarities (i.e., D_B values) between two of the three images of different regions using different image standardization methods. Overall, the histogram similarity between the two atmospherically corrected WorldView-3 images increased when compared to the images without AC, with the average D_B value decreasing from 0.400 (method 1) to 0.153–0.378 (methods 2–5). Among them, when comparing the histogram similarity of the atmospherically corrected images using 6S and the atmospherically corrected images using FLAASH, the former (methods 2 and 4, average D_B = 0.153–0.252) show higher similarity than the latter (methods 3 and 5, average D_B = 0.369–0.378). Furthermore, performing AC with 6S before pan-sharpening (method 2) shows the highest histogram similarity (average D_B = 0.153).

Table 3. Accuracy evaluation results of building segmentation using five image standardization methods. (A): uses the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): uses the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): uses the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU, OA and variance values are highlighted in bold, and the second highest IoU, OA, and variance values are underlined.

No.	Method	Mean		(A)		(B)		(C)		Variance
		IoU	OA	IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	Only NNDiffuse	0.395	0.862	0.371	0.883	0.508	0.940	0.304	0.762	5.38 × 10−3	1.66 × 10−3
2	6S + NNDiffuse	0.575	0.943	0.505	0.930	0.622	0.956	0.599	0.942	3.46 × 10−3	1.76 × 10−4
3	FLAASH + NNDiffuse	0.420	0.890	0.324	0.865	0.465	0.918	0.470	0.886	5.22 × 10−3	7.03 × 10−4
4	NNDiffuse + 6S	0.531	0.935	0.502	0.929	0.514	0.938	0.576	0.937	2.07 × 10−4	2.22 × 10−5
5	NNDiffuse + FLAASH	0.424	0.889	0.385	0.885	0.511	0.940	0.378	0.843	4.14 × 10−3	9.41 × 10−4

shows the accuracy evaluation results of building segmentation using each image standardization method. Similar to the histogram similarity results, method 2 (performing AC with 6S before pan-sharpening) performed best with the highest average IoU of 0.575 and average OA of 0.943, followed by method 4 (average IoU = 0.531, average OA = 0.935) and methods 3 and 5 (average IoU = 0.420–0.424, average OA = 0.889–0.890). Method 1 (only pan-sharpening, no AC) has the worst performance (average IoU = 0.395, average OA = 0.862). Furthermore, method 2 shows the second smallest accuracy variance between the use of different training and testing datasets, demonstrating its robustness.

The above results not only highlighted the need for AC but also demonstrated the importance of the order of AC and pan-sharpening.

4.2. Building Segmentation Results Using U-Net with SSL Pre-Trained VGG19 Backbone

4.2.1. Performance of Two New Pretext Tasks in SSL

To clarify the effectiveness of each new pretext task in the multi-task SSL network, we first tested the building segmentation performance using U-Net, which applies a VGG19 backbone pre-trained on only one pretext task.

Table 4. Accuracy of building segmentation using SSL network with the Pretext Task 1 only. Three image transformation methods are image rotation, image flipping, and image color jittering. LB: The large block masked area uses all three transformation methods; SB1: Each small block masked area uses different combinations of all three transformation methods; SB2: All small block masked areas use the same combination of all three transformation methods; SB3: All small block masked areas are simply set to zero. (A): use the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): use the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): use the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU and OA values are highlighted in bold.

No.	Image Inpainting Method	Image Damage Strategy	Mean		(A)		(B)		(C)
			IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	One Large Block	LB	0.602	0.949	0.596	0.951	0.610	0.955	0.598	0.943
2	Small Blocks (this study)	SB1	0.591	0.946	0.543	0.940	0.613	0.954	0.617	0.944
3	Small Blocks (this study)	SB2	0.611	0.950	0.637	0.959	0.611	0.953	0.584	0.939
4	Small Blocks (this study)	SB3	0.628	0.953	0.642	0.959	0.646	0.959	0.596	0.942

shows the results of building segmentation using SSL network with the Pretext Task 1 only (Image Inpainting). We found that the simplest image damage strategy (SB3 in Section 3.2.1), used for regions masked with multiple small blocks (i.e., method 4), exhibited the best performance (average IoU = 0.628, average OA = 0.953). However, applying a complex image damage strategy to regions masked with multiple small blocks resulted in poorer performance (method 2, average IoU = 0.591, average OA = 0.946) compared to the use of a single large block (method 1, average IoU = 0.602, average OA = 0.949). Therefore, method 4 (SB3) in Table 4 is recommended for Pretext Task 1.

Table 5. Accuracy of building segmentation using SSL network with the Pretext Task 2 only. CIC: Colorful Image Colorization; ISR: Image Spectrum Recovery; RGB: Red-Green-Blue color space; CIELAB: International Commission on Illumination (CIE) 1976 (L^∗, a^∗, b^∗) color space [54]; R1Z: Randomly select one band and set the value of the selected band to zero; R2A: Randomly select two bands, average their values, and replace the values of the two selected bands with the average value; R3A: Randomly select three bands, average their values, and replace the values of the three selected bands with the average value; 4A: Average all four bands and replace the values of the four bands with the average value. (A): use the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): use the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): use the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU and OA values are highlighted in bold.

No.	Method	Damage Strategy	Mean		(A)		(B)		(C)
			IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	CIC	RGB to CIELAB	0.648	0.957	0.664	0.965	0.670	0.965	0.611	0.942
2	ISR (this study)	R1Z	0.605	0.949	0.602	0.953	0.624	0.955	0.589	0.938
3	ISR (this study)	R2A	0.593	0.947	0.614	0.954	0.596	0.950	0.570	0.937
4	ISR (this study)	R3A	0.606	0.949	0.588	0.950	0.635	0.957	0.594	0.941
5	ISR (this study)	4A	0.652	0.958	0.668	0.964	0.684	0.966	0.604	0.943

shows the results of building segmentation using the SSL network with the Pretext Task 2 only (CIC used in previous studies or Image Spectrum Recovery proposed in this study). We find that image spectrum recovery method 5 (4A in Section 3.2.2) performs best with an average IoU of 0.652 and an average OA of 0.958. Compared to the CIC method (method 1), the building segmentation accuracy was improved by 0.04 in average IoU. However, other image spectrum recovery methods (methods 2–4) perform worse compared to the CIC method (average IoU = 0.593–0.606 vs. 0.648, average OA = 0.947–0.949 vs. 0.957). Therefore, method 5 (4A) in Table 5 is recommended for Pretext Task 2.

Figure 6 shows some examples that demonstrate the effectiveness of the two recommended new pretext tasks compared to the pretext tasks used in previous studies [18,22]. A visual comparison between Figure 6e (IoU = 0.246) and Figure 6i (IoU = 0.905), and Figure 6f (IoU = 0.827) and Figure 6j (IoU = 0.872), shows that the new pretext task 1 successfully removes the road that misclassified by image inpainting pretext task of [22] (see the red pixels in the red circle in Figure 6e) and properly detects some small buildings that were missed by image inpainting pretext task of [22] (see the green pixels in the red circle in Figure 6f). Similarly, visual comparison of Figure 6g (IoU = 0.330) and Figure 6k (IoU = 0.946), and Figure 6h (IoU = 0.525) and Figure 6l (IoU = 0.881), reveals that the new pretext task 2 successfully removes some farmland (see the red circle in Figure 6k) and properly detects almost all buildings with green rooftop (see the red circle in Figure 6l). Both were all misclassified by CIC method of [18] (see the red pixels in the red circle in Figure 6g and the green pixels in the red circle in Figure 6h). These examples highlight the advantages of the two new pretext tasks proposed in this study.

4.2.2. Performance of the New Multi-Task SSL Network

Table 6. Ablation experiments on the pretext tasks of the multi-task SSL network developed in this study. Pretext task 1: image inpainting using multiple small blocks with image damage strategy of SB3; Pretext task 2: image spectrum recovery with damage strategy of 4A; Pretext task 3: SimCLR from [50]. (A): use the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): use the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): use the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU and OA values are highlighted in bold.

No.	Pretext Task 1	Pretext Task 2	Pretext Task 3	Mean		(A)		(B)		(C)
				IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	×	×	×	0.575	0.943	0.505	0.930	0.622	0.956	0.597	0.942
2	√	×	×	0.628	0.953	0.642	0.959	0.646	0.959	0.596	0.942
3	√	√	×	0.636	0.954	0.661	0.963	0.645	0.958	0.604	0.940
4	√	√	√	0.663	0.958	0.692	0.968	0.678	0.964	0.620	0.943

shows the ablation experiments to clarify the importance of each pretext task in the proposed method. For reference, we also show the building segmentation accuracy using U-Net with a VGG19 backbone trained from scratch (i.e., without SSL; Method 1 in the table). It is not surprising that the U-Net combined with SSL performed better than without SSL (average IoU = 0.628–0.663 vs. average IoU = 0.575). Looking at the performance of U-Net combined with SSL, we can see a steady improvement in building segmentation accuracy with each additional pretext task. Overall, the combination of U-Net and the new multi-task SSL network (including all three pretext tasks) improves building segmentation accuracy by 15.3% on average IoU compared to U-Net without SSL. These results demonstrate not only the importance of SSL but also the necessity of each pretext task.

4.3. Comparison with Other SSL Approaches

The new multi-task SSL network proposed in this study was compared with four state-of-the-art SSL methods, including SimCLR [50], MoCo v2 [51], BYOL [55], and PGSSL [23]. The results are shown in

Table 7. Comparison of building segmentation accuracy using different SSL methods. (A): use the images from Sakaiminato and Sanuki as the training dataset, and the image from Tsukuba as the test dataset; (B): use the images from Sakaiminato and Tsukuba as the training dataset, and the image from Sanuki as the test dataset; (C): use the images from Tsukuba and Sanuki as the training dataset, and the image from Sakaiminato as the test dataset. The highest IoU and OA values are highlighted in bold.

No.	SSL Type	Method	Mean		(A)		(B)		(C)
			IoU	OA	IoU	OA	IoU	OA	IoU	OA
1	Contrastive Learning	SimCLR	0.615	0.951	0.669	0.965	0.582	0.948	0.594	0.940
2		MoCo v2	0.583	0.946	0.602	0.952	0.572	0.946	0.576	0.939
3		BYOL	0.589	0.947	0.569	0.945	0.593	0.952	0.606	0.943
4	Multiple Task-based	PGSSL	0.640	0.955	0.608	0.953	0.659	0.962	0.653	0.951
5		This study	0.663	0.958	0.692	0.968	0.678	0.964	0.620	0.943

. U-Net combined with the new multi-task SSL network performed the best with the highest average IoU (0.663) and average OA (0.958) values, followed by U-Net combined with PGSSL (average IoU = 0.640, average OA = 0.955), SimCLR (average IoU = 0.615, average OA = 0.951), BYOL (average IoU = 0.589, average OA = 0.947), and MoCo v2 (average IoU = 0.583, average OA = 0.946).

Figure 7 shows some examples that demonstrate the advantages of the proposed building segmentation method (i.e., the combination of U-Net and the new SSL pre-trained VGG19 backbone, the segmentation accuracy values for each SSL approach are listed in

Table A4. Accuracy of building segmentation (IoU) in regional areas using different SSL approaches. The highest IoU values are highlighted in bold.

Position in Figure 7	SimCLR	MoCo v2	BYOL	PGSSL	This Study
Top	0.744	0.874	0.780	0.886	0.910
Medium	0.754	0.774	0.752	0.790	0.834
Bottom	0.906	0.941	0.905	0.878	0.992

in Appendix A). Misclassification of bare grounds to buildings (see the red pixels in the red circle at the top of Figure 7) is observed in all four compared SSL methods, but is successfully removed by our method (Figure 7f, top). Another problem with the four compared SSL methods is the misclassification of roads and buildings (see the red pixels in the red circles in the second row of Figure 7), and these misclassified roads are not seen in the second row of Figure 7f (using the method proposed in this study). Moreover, all the four compared SSL methods struggled to detect an entire building composed of several small dense objects (e.g., greenhouse, see the green pixels in the red circle in the third row of Figure 7), while this problem was successfully solved by our method (Figure 7f, bottom).

5. Discussion

Previous studies have identified AC and pan-sharpening as two important steps when using satellite imagery for object segmentation [12,56,57]. However, the optimal order of AC and pan-sharpening remains unclear. In this study, we clarified this issue by conducting three experiments. The results clearly show that performing AC before pan-sharpening results in higher building segmentation accuracy than after pan-sharpening (Table 2 and Table 3). This is likely due to the different atmospheric effects on each MS band and the PAN band [58]. It is considered that performing pan-sharpening before AC may change the spectral characteristics of each land cover type and reduce the building segmentation accuracy. Furthermore, the results also show that using 6S for AC before pan-sharpening performed better than using FLAASH for AC. This is mainly because FLAASH cannot perform AC on the PAN band, which results in inconsistent reflectance values between the MS bands (surface reflectance) and the PAN band (top-of-atmosphere reflectance) when pan-sharpening was performed.

SSL is a powerful approach to solve the problem of lack of labeled training data, especially when using regularly archived satellite data for building segmentation [50,51]. A good pretext task is key to enable the deep learning backbone to automatically and properly learn the features of an object [14,15,16]. In this study, we designed two new pretext tasks that enable the VGG19 backbone to learn the texture and spectral features of both foreground (buildings) and background (other land covers) objects.

First, taking into account the spatial resolution of WorldView-3 images (0.3 m after pan-sharpening) and the average size of buildings, we proposed replacing a single large block with multiple small blocks for image inpainting (Pretext Task 1 in Figure 4). This change avoided losing information of small objects during pre-training of the VGG19 backbone [25] and improved the building segmentation accuracy (Table 4 and Figure 6). For example, the misclassification of the road in Figure 6e and the missed detection of the small house in Figure 6f are due to the use of a single large block, which are improved in this study by the small block design (see the red circles in Figure 6i,j). Furthermore, in this study, the total area of multiple small blocks was set to 30% of each 512 × 512-pixel patch. This value lies between two previous studies (75% in [59]; 5% in [23]) that also use multiple blocks for image inpainting in SSL. However, both the 75% and 5% values are considered inappropriate for segmenting buildings from pan-sharpened WorldView-3 imagery. This is because a masking ratio that is too large may result in missing information of many small objects, while a masking ratio that is too small will require more epochs to learn the features of the entire 512 × 512-pixel patch, thereby increasing computational cost. In addition, it should be noted that even when using multiple small blocks, if the image transformation strategy is too complex (e.g., image inpainting method 2 in Table 4), the features of the same object in the same image will have large differences, making it difficult for the VGG19 backbone to learn effective features for each object [59].

Second, considering that WorldView-3 images have multiple bands, we proposed to have the VGG19 backbone learn the multi-spectral features of each land cover (Pretext Task 2 in Figure 4), rather than only learning spectral information from RGB images through CIC. In general, CIC is a suitable approach for RGB images (or photographs) [18]. However, satellite imagery such as WorldView-3 contains extra spectral information that can help to distinguish buildings from other land covers [26,27,60]. This is probably the reason why method 1 (CIC) in Table 5 has worse building segmentation accuracy than method 5 (4A). For example, the incorrectly classified farmlands in Figure 6g and the undetected buildings (with green rooftop) in Figure 6h are due to spectral/color similarity with buildings/vegetation and were improved by the image spectrum recovery designed in this study (see the red circles in Figure 6k,l). Furthermore, the best performance of method 5 (4A) in Table 5 may be due to the low number of epochs performed in this study (i.e., 20), which was limited by the use of a personal computer. To achieve the same building segmentation accuracy as method 5 (4A), methods 2–4 (RIZ, R2A, and R3A in Table 5) need to run more epochs, which increases the computational cost.

The comparison results show that the new multi-task SSL network proposed in this study outperforms other widely used SSL methods (Table 7). This is mainly because we designed two new pretext tasks depending on the features of objects in the WorldView images (e.g., size, number, and reflectance spectra of buildings) as well as the characteristics of the WorldView images themselves (e.g., spatial resolution and multiple bands). Moreover, the new multi-task SSL network integrates generative and contrastive learning methods, allowing it to learn both high- and low-level features of WorldView-3 imagery. In contrast, SimCLR, MoCo v2, and BYOL were originally developed based on ImageNet (i.e., photos), and therefore did not fully take into account the features of objects in satellite images [50,51,55]. Furthermore, they all only used contrastive learning methods that mainly focus on learning high-level features of satellite images. PGSSL was also developed for building segmentation, but because it uses a very small masking ratio (5% of each 512 × 512-pixel patch), it would require a large computational cost to achieve the same building segmentation accuracy as the new multi-task SSL network, making it difficult to effectively work on a personal computer.

There are some limitations when applying the multi-task SSL network proposed in this study. First, in Pretext Task 1, the size of each small block was set based on the average area of building in Japan and the spatial resolution of pan-sharpened WorldView-3 imagery (0.3 m). Therefore, this size needs to be adjusted when applying the proposed method to satellite images with different spatial resolutions or to regions with different average building areas. Second, in this study, we only tested the combination of U-Net with the VGG19 backbone and the proposed multi-task SSL network. Replacing the VGG19 with other backbones such as ResNet [61] or DarkNet [62], or replacing U-Net with other convolutional neural networks (e.g., HRNet [63] and ABCNet [64]) or Transformer-based networks with attention mechanisms (e.g., SegFormer [65] and UNetFormer [66]), could further improve the building segmentation accuracy. Third, all experimental images were acquired during the summer, and half of them (12 images) were acquired from the Japanese region. Therefore, the robustness and generalizability of the proposed SSL network needs to be further improved using WV images from different seasons and regions. Fourth, the proposed SSL network is currently only applicable when the pre-training data are generated from sensors with the same spatial resolution. To increase the number of available pre-training and fine-tuning data, future research could consider multi-scale feature fusion techniques (e.g., atrous spatial pyramid pooling module [67], adaptive spatial feature fusion [68]) or introduce a resolution adaptation mechanism that dynamically adjusts the block size according to the spatial resolution of the input image. Fifth, the spectrum recovery method developed in this study does not fully consider the influence of shadows on the spectral characteristics of buildings. To solve this problem, shadow removal methods [69,70] can be applied first to minimize the influence of shadows before learning spectral features. All these remaining challenges need to be thoroughly investigated in future research.

6. Conclusions

In this study, we first conduct several comparative experiments using three labeled WorldView-3 images and reveal the optimal order of AC and pan-sharpening (i.e., performing AC with 6S before pan-sharpening with NNDiffuse), which improves building segmentation accuracy by 8.3–45.6% compared to other image standardization methods. We then designed two new pretext tasks that specifically consider the object features distributed in WorldView-3 images (e.g., size, distribution pattern, multispectral, etc.), integrated them into the multi-task SSL network, and pre-trained the VGG19 backbone of U-Net with unlabeled WorldView images. This effort further improved building segmentation accuracy by 15.3% compared to a VGG19 backbone trained from scratch. Finally, we compared the new multi-task SSL network with other existing SSL methods and found it to perform the best. Another advantage of the proposed method is that it has low computational cost and can effectively work on a personal computer. By continuing to address the remaining issues, the proposed method can be applied to other satellite images and other regions, providing useful information in fields such as urban planning.