Enhancing Urban Understanding Through Fine-Grained Segmentation of Very-High-Resolution Aerial Imagery

Abstract

Despite the growing availability of very-high-resolution (VHR) remote sensing imagery, extracting fine-grained urban features and materials remains a complex task. Land use/land cover (LULC) maps generated from satellite imagery often fall short in providing the resolution needed for detailed urban studies. While hyperspectral imagery offers rich spectral information ideal for material classification, its complex acquisition process limits its use on aerial platforms such as manned aircraft and unmanned aerial vehicles (UAVs), reducing its feasibility for large-scale urban mapping. This study explores the potential of using only RGB and LiDAR data from VHR aerial imagery as an alternative for urban material classification. We introduce an end-to-end workflow that leverages a multi-head segmentation network to jointly classify roof and ground materials while also segmenting individual roof components. The workflow includes a multi-offset self-ensemble inference strategy optimized for aerial data and a post-processing step based on digital elevation models (DEMs). In addition, we present a systematic method for extracting roof parts as polygons enriched with material attributes. The study is conducted on six cities in Flanders, Belgium, covering 18 material classes—including rare categories such as green roofs, wood, and glass. The results show a 9.88% improvement in mean intersection over union (mIOU) for building and ground segmentation, and a 3.66% increase in mIOU for material segmentation compared to a baseline pyramid attention network (PAN). These findings demonstrate the potential of RGB and LiDAR data for high-resolution material segmentation in urban analysis.

1. Introduction

In remote sensing, land use/land cover (LULC) classification categorizes landscapes based on human activity and environmental features, including agricultural land, forests, urban areas, wasteland, and water bodies. For decades, LULC maps have been derived using satellite imagery, primarily from platforms like Landsat [1], Sentinel [2], and Worldview [3]. Traditionally, these maps have been utilized for change detection, allowing researchers to track landscape alterations by comparing scenes captured over time. More recently, the integration of deep learning (DL) techniques with high-resolution aerial imagery, captured by manned aircrafts and unmanned aerial vehicles (UAVs) using specialized airborne sensors, has facilitated the creation of more detailed and task-specific LULC maps, particularly suited to the complexities of urban environments.

Urban areas pose distinct challenges and offer unique opportunities in geospatial analysis due to their dense, diverse, and heterogeneous composition, typically comprising residential, commercial, and industrial zones, complex transportation networks, and varied infrastructure types and materials. Uncontrolled urban expansion can worsen critical environmental concerns such as the urban heat island (UHI) effect, inefficient energy use, and the spread of impervious surfaces, which disrupt natural drainage patterns and increase the risk of localized flooding. High-resolution LULC data are essential for addressing these challenges, as they enable actionable insights for environmentally sustainable urban planning, climate adaptation, and efficient resource management. For instance, identifying rooftop materials and surface characteristics supports the strategic deployment of solar panels by pinpointing rooftops with suitable materials and orientations for photovoltaic systems. Similarly, detailed mapping of ground surfaces such as asphalt, grass, or permeable pavement can inform efforts to reduce heat retention and enhance green infrastructure. These applications demand spatially precise, material-specific information that extends beyond traditional LULC categories and the limitations of coarse-resolution satellite imagery.

This study aims to address these needs by applying deep learning methods to very-high-resolution (VHR) aerial imagery for fine-grained classification of roof and ground-surface materials in urban settings. The data for this study were sourced from six municipalities across the Flemish region of Belgium (Figure 1). According to recent reports [4] from the Flemish Statistical Authority, approximately 28.7% of Flanders’ surface area was built-up as of 2021, with a marked increase from 24.4% in 2000. The majority of this built-up land comprises residential areas, roads, and associated spaces for human activity. Nonetheless, cropland and grassland still dominate more than half of the region, making it a unique setting where dense urban cores coexist with expansive natural and agricultural areas. This diversity in land use, building typologies, and material surfaces makes Flanders an ideal testbed for developing and evaluating high-resolution urban classification models.

While prior research on aerial imagery has predominantly focused on building footprint extraction or generic land cover classification, this work expands the scope by enabling fine-grained identification of rooftop types, ground materials, and even distinct roof segments using RGB and LiDAR data. Such detailed semantic segmentation is critical for advancing next-generation urban applications in areas such as energy optimization, infrastructure assessment, and environmental monitoring. Despite the increasing availability of VHR aerial data and significant progress in deep learning-based remote sensing, to the best of our knowledge, no previous study has tackled large-scale semantic segmentation of urban material types from RGB and LiDAR—encompassing both roofing and ground surfaces—across multiple cities. Existing approaches often limit themselves to broader tasks, overlooking the material-level diversity that is essential for real-world applications. The ability to distinguish between materials such as asphalt, concrete, tiles, or green roofs is vital for accurate modeling, yet it remains underexplored. This study, therefore, presents what we believe to be the first large-scale effort to perform material-level semantic segmentation of urban environments using VHR aerial imagery.

Contributions. This work offers the following key contributions: (1) the development of a multi-head segmentation network that simultaneously predicts material classes and delineates roof part boundaries; (2) the introduction of a multi-offset self-ensemble inference strategy specifically adapted for aerial imagery to improve prediction consistency; (3) the integration of a postprocessing filter leveraging digital elevation models (DEMs) to refine segmentation results; and (4) an ablation study that systematically evaluates the contribution of each component within the proposed pipeline.

2. Background and Related Works

Aerial image segmentation has become critical for applications in urban planning, environmental monitoring, and disaster management. Recent advances in deep learning, particularly through convolutional neural networks (CNNs) and transformer-based architectures, have significantly improved segmentation accuracy and efficiency for aerial imagery. Fully convolutional networks (FCNs) [5] were foundational in applying CNNs to segmentation by replacing fully connected layers with convolutional ones, thereby retaining spatial information throughout the process. The U-Net [6] architecture introduced a U-shaped encoder–decoder structure with skip connections, which further enhanced segmentation accuracy, especially in cases requiring fine detail. SegNet [7] refined this approach by leveraging pooling indices from the encoder’s max-pooling steps for non-linear upsampling in the decoder, maintaining spatial resolution and capturing image detail effectively. DeepLab [8] extended CNN capabilities by integrating atrous (dilated) convolutions, allowing the network to capture multi-scale contextual information with an expanded receptive field, essential for accurately segmenting complex structures of varying sizes. This architecture also incorporated conditional random fields (CRFs) [9] as a post-processing step to enhance boundary precision by enforcing spatial coherence and continuity in the segmentation outputs. Vision transformers (ViTs) [10] and their variants, such as the Swin transformer [11], utilize self-attention mechanisms to capture long-range dependencies. However, unlike CNNs, transformer networks lack inherent spatial priors, making them highly data-dependent and necessitating large-scale datasets for effective training.

2.1. Advances with Airborne Platforms

The rapid advances in airborne platforms have dramatically enhanced the quality and resolution of aerial imagery, particularly through improvements in ground sampling distance (GSD), now reaching VHR levels such as 9 cm in the Vaihingen  [12] and 5 cm in the Potsdam [13] datasets. This substantial increase in image resolution has broadened the scope of using airborne platforms across diverse fields. In agriculture, VHR imagery enables precise crop classification [14,15,16], crop health monitoring [17,18,19,20,21,22], and crop row detection [23,24,25,26], promoting data-driven precision farming that optimizes yield and resource efficiency. In environmental monitoring, VHR images play a critical role in forest health monitoring [27,28,29], wetland management [30,31], and coastal area surveillance [32,33]. For urban planning, UAV advancements support applications such as traffic monitoring [34,35], land use classification [36,37], green space analysis [38], and urban tree species identification [39]. These capabilities enable more detailed, data-rich analysis across various disciplines.

2.2. VHR Imagery for Predicting Roof Properties

Airborne platforms equipped with VHR sensors capture detailed aerial imagery, enabling precise analysis of roof characteristics crucial for sustainable urban planning. These high-resolution data aid in optimizing building energy efficiency by facilitating effective thermal regulation, reducing energy consumption for heating and cooling, and mitigating the urban heat island effect [40]. VHR imagery also supports efficient solar panel placement by assessing roof orientation with respect to the sun, the shade of trees, surrounding objects, topography, and the atmospheric conditions, thereby maximizing renewable energy capture [41]. Additionally, VHR data can be used to promote the implementation of green roofs, which enhance air quality and urban biodiversity [42]. In architectural planning, accurate roof data from VHR contribute to cohesive design and historical preservation [43]. By integrating detailed roof characteristics into 3D models, urban planners can perform complex simulations and visualizations [44], facilitating informed decisions that bolster urban resilience and improve environmental quality.

2.3. Roof Type Classification

Recent advances in roof type classification have harnessed high-resolution aerial and satellite imagery through machine learning and deep learning techniques, yielding variable accuracy across diverse datasets and architectural styles. To enhance roof classification and mitigate dependence on low-quality digital surface models (DSMs), Partovi et al. [45] utilized pansharpened imagery combined with a pre-trained CNN and a support vector machine (SVM) classifier, achieving high accuracy and efficiency across seven defined roof types. Similarly, Castagno and Atkins [46] integrated CNN models with SVM and decision tree classifiers, demonstrating that the fusion of satellite and LiDAR data significantly enhances roof type identification. Buyukdemircioglu et al. [47] developed a dataset comprising roof types derived from high-resolution (10 cm) orthophotos in Çeşme, Turkey, to classify six distinct roof types using a shallow CNN model. In another approach, Ölçer et al. [48] employed a one-shot learning framework with a Siamese neural network to classify flat, gable, and hip roofs with minimal training examples.

2.4. Roof Material Segmentation

Material segmentation assigns a material label to each pixel in an image, offering more detailed insights into a scene’s physical composition compared to semantic segmentation, which focuses on object or region type rather than material. Cimpoi et al. [49] introduced the Describable Textures Dataset (DTD), which consists of images of patterns collected “in the wild” and categorized according to forty-seven texture attributes, facilitating the study of texture representation in images. Bell et al. [50] introduced the Materials in Context Database (MINC), a large-scale open dataset of material samples, and integrated it with deep learning to achieve material recognition and segmentation in natural images. Xue et al. [51] developed the GTOS (Ground Terrain in Outdoor Scenes) database, comprising over 30,000 images across 40 classes of ground terrain in outdoor environments. More recently, the KITTI-Materials dataset [52] has provided material annotations for urban driving scenes. Despite these advancements, no dataset has been specifically developed for material segmentation in aerial imagery using RGB and/or LiDAR data. Existing methods primarily focus on close-range or ground-level images, limiting their applicability to aerial and broader geospatial contexts.

Jayasinghe et al. [53] found that while roof orientation has minimal influence on indoor temperatures in passive houses located in warm, humid climates, the selection of roofing materials, insulation, and light-colored surfaces can significantly enhance thermal comfort. Prado and Ferreira [54] investigated the albedo of roofing materials in Brazil by employing a spectrophotometer to measure reflectance and assessing the effects on heat gain and surface temperature under solar radiation, aiming to address urban heat island effects and energy consumption. Mendez et al. [55] evaluated the impact of conventional and alternative roofing materials on the quality of harvested rainwater, concluding that all roofing types necessitate treatment to meet EPA standards, with metal, concrete tile, and cool roofs yielding lower concentrations of contaminants compared to shingle and green roofs, which exhibited higher levels of dissolved organic carbon and potential metal contamination.

Lemp and Weidner [56] integrated hyperspectral and laser scanning data to characterize urban roof surfaces, enhancing material classification by utilizing the detailed spectral resolution of hyperspectral data alongside the geometric information from laser scanning. Ilehag et al. [57] studied energy emissions from building roofs impacted by urban heat islands in Perth, Australia, utilizing multispectral, thermal infrared, RGB, and LiDAR data to classify roofing materials (including cement tiles, Colorbond, and Zincalume) through pixel-wise, superpixel-wise, and building-wise segmentation approaches. Trevisiol et al. [58] introduced a semi-automatic, object-oriented methodology using high-resolution multispectral imagery to map buildings and roofing materials in Bologna, Italy. Kim et al. [59] proposed a 43-layer CNN for detecting buildings and classifying roof materials in satellite images, thereby enhancing disaster resilience by identifying and addressing vulnerabilities in roof structures. These methods for roof material identification primarily rely on hyperspectral imaging, with a clear gap in the literature, as no studies have used RGB imagery, despite its potential for cost-effective solutions.

2.5. Building Delineation

Boonpook et al. [60] present a deep learning framework for UAV-based photogrammetry that integrates RGB, Digital Surface Model (DSM), and Vegetation Difference Index (VDVI) bands, improving building extraction accuracy from 93% to 97% in complex urban areas. Farajzadeh et al. [61] apply a U-Net architecture with ResNet to extract building footprints from UAV orthophotos and normalized Digital Surface Models (nDSMs), enhancing precision from 89% to 97% and recall from 77% to 91% by incorporating height information. Pilinja et al. [62] utilize high-resolution UAV imagery and various machine learning algorithms to extract building rooftops, demonstrating improved accuracy with elevation data, benefiting urban planning. Djenaliev et al. [63] use UAV data over Karakol city, Kyrgyzstan, producing a Digital Terrain Model (DTM) and extracting building footprints with 92.4% completeness and 95.2% correctness. Roof-Former  [64] utilizes a vision Transformer to enhance the vectorization of building roof geometries from raster images, achieving improved edge heat map F1-scores (76.2% to 78.1%) while maintaining topological validity. These methods primarily focus on building delineation but overlook the material properties of roof structures and ground materials, both of which are essential for various urban applications.

3. Materials and Methods

3.1. Dataset

The aerial data used in this study were acquired using the Leica CityMapper-2 airborne hybrid sensor system, which integrates nadir and oblique RGB and LiDAR components for urban mapping. For this analysis, only the nadir RGB imagery and LiDAR-derived elevation data were utilized. The nadir-facing RGB camera has a 150-megapixel resolution, equipped with a low-distortion lens and a backside-illuminated CMOS sensor. Mechanical forward-motion compensation (FMC) was applied to minimize motion blur during acquisition. The LiDAR component operates at a 2 MHz pulse repetition frequency with a Multiple Pulses in the Air (MPiA) configuration and uses an oblique scan pattern to ensure a more uniform point distribution across the surveyed area. The resulting point clouds were processed into rasterized digital elevation models, which were spatially aligned with the orthophotos.

The dataset encompasses six cities within the Flemish region of Belgium, covering in total an approximate area of 4.00 km². Each city is represented by three primary files that contribute to a comprehensive data structure: (i) a 3-channel RGB orthophoto, stored in JPEG2000 or GeoTIFF format, offering a high-resolution ground sampling distance (GSD) of 3 cm, allowing for fine-grained spatial details in urban and suburban landscapes; (ii) a Digital Elevation Model (DEM) with a GSD of 25 cm, stored as a single-channel GeoTIFF, providing crucial height information that aids in differentiating between various objects and structures within the urban environment; and (iii) ground truth annotations available in GeoJSON format, where each polygon is associated with a specific class label. The DEM and RGB files complement each other by providing a layered understanding of both visual and topographical features.

As illustrated in Figure 2, sample data from Brugge exhibit the unique layering and detailed annotation setup of the dataset. The RGB orthophoto contains integer values ranging from 0 to 255 per channel, providing clear visual differentiation among materials, while the DEM file supplies float values in meters, crucial for determining object heights and contours. Annotation polygons in the GeoJSON files define specific material regions and are categorized into 18 class labels, which are organized into two primary categories: roof materials and ground materials, as shown in Figure 3. Roof materials include seven classes—roof tiles, bitumen/EPDM, slates, corrugated sheets, zinc/metal, glass, and green roofs—capturing the diversity of roofing types commonly found in the region. Ground materials span eleven classes: grass, permeable materials, tiles, KWS/asphalt, cobblestone, clinker bricks, concrete, water, gravel, border stones, and wood. The annotations were generated by non-expert annotators, and visual inspection ensured annotation consistency across annotators, providing reliable ground truth labels for model training and evaluation.

The dataset is characterized by a significant class imbalance, posing a challenge for deep learning methods. As highlighted in Figure 3, the grass class dominates the area, covering approximately 38% of the total, while other classes such as border stone, glass, gravel, green roof, water, and wood collectively account for less than 5% of the annotated area. This imbalance reflects the natural distribution of surface materials in urban and suburban landscapes. The complex urban landscape of Flanders, with its varied material types and architectural elements, provides an ideal testing ground for evaluating the robustness and adaptability of aerial image segmentation models.

The dataset also includes a strategic data split to maximize training efficiency and model generalization across different urban contexts. The three largest cities—Brugge, Oostende, and Roeselare—comprise 60% of the total data and are allocated for training, ensuring an ample representation of each material class in the learning phase. This allows the network to learn distinctive material characteristics across various scales and perspectives, enhancing its ability to generalize. The smaller cities, Jabbeke, Oostrozebeke, and Wetteren, serve as validation and testing, providing a robust basis for evaluating the model’s performance in less prevalent urban layouts and architectural styles. This setup allows the network to be tested on regions that it has not encountered during training, offering a realistic assessment of its segmentation capabilities across the diverse urban landscape of the Flemish region.

3.2. Methodology

An overview of the workflow is illustrated in Figure 4, detailing the key processing components involved. The workflow is structured around five primary elements: (1) a multi-head convolutional neural network (CNN) generates two sets of probability maps from a single RGB image, (2) a DEM-based filter distinguishes between ground and roof pixels using the elevation data, (3) a multi-offset self-ensemble approach mitigates noise at the edges of image patches and produces smooth probability maps, (4) a clustering algorithm identifies roof components based on the edges of the roof parts raster, and (5) a vectorization algorithm transforms the identified building roof components and ground materials into polygon representations.

3.2.1. Multi-Head CNN

The architecture utilized for this segmentation task, illustrated in Figure 5, is based on a Convolutional Neural Network (CNN) with a multi-head extension for enhanced functionality. CNNs form the core structure of segmentation networks by leveraging convolution layers exclusively, which allows them to handle input images of various sizes and produce segmentation maps that maintain alignment with the input dimensions. Each pixel in the resulting segmentation map thus directly corresponds to a pixel in the input image. The typical CNN structure consists of an encoder network responsible for extracting features from the input image, followed by a decoder network that reconstructs these features into a segmentation map. The encoder typically comprises stacked convolution and pooling layers, while the decoder contains transposed convolution layers that upsample the feature maps to create the final segmentation output.

For this task, the network architecture builds on the pyramid attention network (PAN) [65], which enhances the traditional U-Net architecture with advanced attention mechanisms specifically designed to improve feature extraction. PAN introduces two primary attention modules: feature pyramid attention (FPA) and global attention upsample (GAU). FPA facilitates multi-scale feature extraction by aggregating information across varying resolutions, thereby increasing the network’s sensitivity to fine details in the input data. Meanwhile, GAU incorporates global context into the upsampling process, which ensures spatial coherence across the segmentation output, resulting in smoother and more accurate delineations. This combination improves the extraction of precise dense features compared to dilated convolutions and artificially designed decoder networks.

The multi-head CNN architecture extends the PAN framework by incorporating two separate segmentation heads in the final decoder layer. The first head is tailored for predicting material types across both roof and surface areas, thus providing a nuanced classification of urban material types. The second head is specifically designed to detect the edges of roof parts, allowing for detailed boundary delineation within structural elements. The material classification task benefits from edge detection by using the boundaries identified by the edge detection head to refine the classification results, as the presence of an edge often indicates a transition between different materials. In turn, the edge detection head benefits from material classification, as knowing the material type in an area can help pinpoint where edges are likely to appear, such as at the transition between roof sections of different materials. This multi-head approach not only enhances the granularity of the segmentation results but also improves the overall accuracy of the network in classifying complex urban environments.

Data preprocessing. In this study, the input raster images are typically large (e.g., Brugge covers approximately $1000\times 1000$ m²), making it impractical to feed entire images to the network due to processing constraints. Consequently, the images are divided into smaller patches, each measuring $1024\times 1024$ pixels, corresponding to an area of approximately $37\times 37$ m². Each input patch has dimensions of $1024\times 1024\times 3$, and the network generates two output probability maps per patch: one of size $1024\times 1024\times 18$ for material classes and another of size $1024\times 1024\times 2$ for roof part edges. To ensure efficient processing, image patches and their corresponding annotations are generated dynamically from JPEG2000/GeoTIFF images and GeoJSON files, respectively.

To expand the training dataset and improve the model’s generalization capabilities, a carefully selected set of data augmentation techniques was employed. Due to the orthophoto nature of the dataset, augmentation was limited to transformations appropriate for aerial imagery. Horizontal and vertical flipping were applied to account for orientation variations. HSV augmentation involved shifting the hue (H) channel within $[-18,+18]$ and scaling the saturation (S) and value (V) channels by a factor sampled from $[1.00,1.10]$ or its reciprocal. The resulting values were clipped to ensure they remained within the valid HSV color range, introducing controlled variations in color intensity and brightness. These augmentations were randomly applied in each training epoch, preventing repetitive data exposure and reducing the risk of overfitting, thus enhancing the network’s robustness to variations within the input data. Additionally, to improve the accuracy of roof part segmentation, the ground truth edges were slightly dilated, aiding the network in learning boundary details more effectively.

Network training. The network was trained on an NVIDIA Tesla V100 GPU with 32GB RAM over the course of 300 epochs, utilizing a dataset containing 3156 image patches from three urban areas: Brugge, Oostende, and Roeselare. Each patch measures $1024\times 1024$ pixels, capturing a diverse array of architectural and environmental features from these regions. To optimize performance in material and structural segmentation, a multi-head categorical cross-entropy (CCE) loss function was employed. This loss function extends the conventional CCE by incorporating multiple segmentation heads, each addressing distinct segmentation objectives such as material classification and roof edge detection.

The CCE loss function, $L_{CCE}$, measures the discrepancy between the predicted and actual probability maps for each pixel, as defined as follows:

$$L_{CCE}=-{\displaystyle \frac{1}{N}}\sum _{n=1}^N\sum _{c=1}^C{w}_c\times 1(y_{n,c}\ne ignore\_index)log{\widehat{y}}_{n,c}$$

(1)

Here, N is the total number of pixels in each patch, and C represents the number of target classes, such as different roof and ground materials. The ground truth label for pixel n in class c is denoted as $y_{n,c}$, and ${\widehat{y}}_{n,c}$ represents the network’s predicted probability that pixel n belongs to class c. The $ignore\_index$ term provides flexibility, enabling certain classes to be excluded from the loss calculation, thus allowing for finer control over the training process. The weight parameter $w_c$ allows for class imbalance handling by assigning higher importance to underrepresented classes. The class weights $w_c$ were computed using the following function:

$$w_c={\displaystyle \frac{1}{100·ln(1.02+f_c)}}$$

(2)

where $f_c\in [0,100]$ denotes the frequency (in percentage) of class c. This inverse logarithmic formulation assigns higher weights to less frequent classes during training. The logarithmic scale ensures that the weight decreases smoothly as class frequency increases, while the additive constant $1.02$ prevents numerical instability for near-zero frequencies. The scalar factor 100 normalizes the resulting weights for effective use within the loss function.

In extending the CCE function to a multi-head architecture, a combined loss function, $L_{CCE,mh}$, was formulated by averaging the losses from each segmentation head, weighted to prioritize specific segmentation tasks. This approach enhances the network’s capacity to manage multiple objectives, such as material classification and edge detection, by leveraging contributions from each head. The multi-head CCE loss function is defined as follows:

$$L_{CCE,mh}={\displaystyle \frac{1}{H}}\sum _{h=1}^H{w}_h\times L_{CCE,h}$$

(3)

Here, H represents the number of output heads, each corresponding to a distinct segmentation task. The weight $w_h$ assigned to each head allows for task prioritization based on importance, with $L_{CCE,h}$ denoting the categorical cross-entropy loss for the head h. By averaging the loss functions across all heads, this approach ensures a balanced optimization that accounts for both material segmentation and structural boundary delineation, ultimately enhancing the network’s performance across varied urban scenarios.

This multi-head approach, combined with class-balancing, addresses the class imbalance inherent in real-world urban datasets, where certain materials (e.g., asphalt, vegetation) may dominate over less frequent classes (e.g., roof edges or specific building materials). Adjusting weights for each class and segmentation head enables the network to more effectively identify subtle features in complex urban landscapes, improving material classification and boundary detection precision. The flexibility in class and head weighting strengthens the model’s robustness, allowing it to prioritize the classes and tasks critical to the intended application domain.

3.2.2. Multi-Offset Self-Ensemble Inference

The inference pipeline generates probability maps for individual image patches, which are then fused to produce a cohesive probability map for the entire image. However, direct fusion of these patches can introduce boundary artifacts along neighboring patches, particularly in high-resolution aerial imagery where seamless continuity is crucial. To address this, a multi-offset self-ensemble inference strategy is employed, as shown in Figure 6. This approach applies the network on each patch after shifting it by a set of predefined offsets (e.g., 0, 256, and 512 pixels) along both the height (h) and width (w) axes. For each offset, the network generates a probability map, and these outputs are subsequently averaged to produce a unified probability map for the entire image. The ensemble of multi-offset predictions not only smooths out boundary inconsistencies but also enhances the overall segmentation quality by preserving spatial continuity across patch borders. By leveraging multi-offset self-ensemble inference, the network achieves more accurate and visually consistent segmentation results—a crucial advantage for aerial image analysis where precision and alignment across patch boundaries are vital.

3.2.3. DEM-Based Filter

The DEM-based filter, $\mathcal{F}$, is designed to address misclassifications of ground pixels that are erroneously identified as building classes. This filter operates as a heuristic approach, applying the following conditional formula:

$$\mathcal{F}(\widehat{\mathcal{Y}},\mathcal{D})=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}c\in B\mathrm{and}{d}_{h,w}\le T\hfill \\ {\widehat{y}}_{h,w}^c,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

In this formulation, $\widehat{\mathcal{Y}}\in {\mathbb{R}}^{C\times H\times W}$ represents the predicted probability map generated by the network, while $\mathcal{D}\in {\mathbb{R}}^{H\times W}$ denotes the corresponding digital elevation model (DEM). Here, $1\le c\le C$ corresponds to each class channel, and $1\le h\le H$ and $1\le w\le W$ represent the spatial coordinates within an image patch.

The filter evaluates each pixel to determine if its probability map value should be retained or set to zero, based on the corresponding DEM value. Specifically, for any pixel with an elevation value ($d_{h,w}$) equal to or below a specified threshold (T), the probabilities assigned to building classes (B) are set to zero, indicating that the pixel likely belongs to the ground rather than a building. This approach enhances segmentation accuracy by preventing low-elevation ground pixels from being misclassified as building structures.

3.2.4. Clustering Roof Parts and Vectorization

In semantic segmentation, each pixel is assigned a class independently, often without accounting for spatial relationships among neighboring pixels. This pixel-wise labeling can complicate downstream processing in geographic information systems (GIS). To mitigate this, adjacent pixels belonging to the same class are grouped and converted into polygons, which are then stored in GeoJSON format for ease of GIS integration. Specifically, ground material pixels are vectorized using the polygonization function from the rasterio library (version 1.3.9), while roof material pixels are combined with roof part segmentation outputs to delineate distinct roof sections, each with its associated materials. Algorithm 1 illustrates roof parts’ clustering and vectorization process.

Algorithm 1 Clustering and Vectorization of Roof Parts for GIS Integration

Require: Binary roof edge segmentation $\mathcal{R}$ (0 for non-edges, 1 for roof part edges) and material raster $\mathcal{M}$ Ensure: Polygons representing individual roof parts with associated materials, in GeoJSON format   1: Input: Binary raster $\mathcal{R}$ for roof part edges and material raster $\mathcal{M}$.   2: procedure Roof Part Vectorization($\mathcal{M},\mathcal{R}$)   3:       Step 1: Thinning Roof Edges          ▹ Thin edges to 1-pixel width   4:       Apply Zhang-Suen [66] thinning on $\mathcal{R}$ to reduce roof part edges to 1-pixel width.   5:       Step 2: Inverting Binary Image               ▹ Isolate roof parts   6:       Invert $\mathcal{R}$ to make roof pixels foreground and edges as background.   7:       Step 3: Extracting Contours   8:       Extract contours from the inverted image.   9:       Step 4: Separating Ground from Roof 10:       for each contour $C_i$ in contours do 11:             Extract material values from $\mathcal{M}$ within contour region $C_i$. 12:             Apply mode filter to determine dominant material in $C_i$. 13:             if mode corresponds to roof material then 14:                 Retain contour $C_i$. 15:             else 16:                 Discard contour $C_i$. 17:             end if 18:         end for 19:         Step 5: Vectorization and Storage 20:         Convert retained contours into polygons. 21:         Save polygons in GeoJSON format for GIS integration. 22: end procedure

4. Results

4.1. Evaluation Metrics

This study utilized a comprehensive set of metrics to assess the performance of the end-to-end workflow. These metrics enable the quantification of various aspects of model performance, ranging from pixel-level accuracy to instance-level recognition. The primary evaluation metrics utilized in this research are the Intersection over Union (IOU) and the Panoptic Quality (PQ) score. These metrics provide valuable insights into the precision of segmentation as well as the model’s capability to identify and distinguish individual instances effectively.

4.1.1. $IOU$ Metric for Segmentation

Mean Intersection over Union ($mIOU$). The $mIOU$ metric quantifies segmentation accuracy by calculating the overlap between predicted and ground truth masks, averaged across all classes. It can be defined using true positive (TP), false positive (FP), and false negative (FN) counts:

$$\mathrm{mIOU}={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FP}}_c+{\mathrm{FN}}_c}},$$

(5)

where C is the number of classes, ${\mathrm{TP}}_c$ is the number of true positives for class c, ${\mathrm{FP}}_c$ is the number of false positives, and ${\mathrm{FN}}_c$ is the number of false negatives. This metric provides an overall assessment of segmentation accuracy but treats all types of misclassifications equally.

Mean Similarity-Weighted Intersection over Union ($msIOU$) [67]. The $msIOU$ metric enhances segmentation evaluation in scenarios where classes are visually or semantically similar. It incorporates a similarity score $S(c,c^{\prime })$ to weigh misclassified pixels based on their similarity to the correct class. The metric is defined as follows:

$$\mathrm{msIOU}={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{T{P}_c^S}{T{P}_c^S+F{P}_c^S+F{N}_c^S}},$$

(6)

where $T{P}_c^S$, $F{P}_c^S$, and $F{N}_c^S$ represent the true positives, false positives, and false negatives adjusted by the similarity score. The similarity score $S(c,c^{\prime })$ ranges from 0 to 1, reducing the penalty for misclassifications between visually or semantically similar classes. This refined approach is particularly beneficial for evaluating complex scenes with visually similar materials.

4.1.2. Panoptic Quality Metric for Roof Part Vectorization

The panoptic quality ($PQ$) metric [68] is used to evaluate panoptic segmentation, which integrates semantic segmentation (assigning a class label to each pixel) and instance segmentation (distinguishing individual object instances). It is defined as follows:

$$PQ={\displaystyle \frac{1}{C}}\sum _{c=1}^C\left(\underset{\mathrm{segmentation}\mathrm{quality}\left(\mathrm{SQ}\right)}{\underbrace{\left({\displaystyle \frac{{\sum }_{(p,g)\in T{P}_c}\mathrm{IoU}(p,g)}{|T{P}_c|}}\right)}}\times \underset{\mathrm{recognition}\mathrm{quality}\left(\mathrm{RQ}\right)}{\underbrace{\left({\displaystyle \frac{|T{P}_c|}{|T{P}_c|+\frac{1}{2}|F{P}_c|+\frac{1}{2}\left|F{N}_c\right|}}\right)}}\right),$$

(7)

where $T{P}_c$, $F{P}_c$, and $F{N}_c$ represent true positives, false positives, and false negatives for class c. The first term, segmentation quality ($SQ$), quantifies the precision of a model’s segmentation by computing the mean intersection over union ($IOU$) between the predicted segments and ground truth. Higher $IOU$ values indicate better segmentation performance. The second term, recognition quality ($RQ$), assesses the model’s instance-level detection performance by incorporating true positives, false positives, and false negatives, ensuring accurate instance detection and minimizing misidentifications or missed detections.

The $PQ$ metric is especially valuable for evaluating vectorized roof parts with materials. Segmentation quality ($SQ$) ensures the accurate delineation of roof sections, maintaining precise boundaries and distinguishing between different materials, which is crucial for applications such as energy modeling, structural analysis, and urban planning. Recognition quality ($RQ$) ensures that each roof part is detected as a distinct instance, reducing misclassification and enhancing correct identification. By integrating $SQ$ and $RQ$, the $PQ$ metric provides a thorough evaluation that balances segmentation precision with instance-level accuracy, making it indispensable for complex tasks such as urban modeling and automated building inspections, where both detailed segmentation and accurate material classification are essential.

4.2. Performance Optimization on Validation Data

Model calibration was conducted on the validation dataset to enhance both segmentation and vectorization performance while maintaining computational efficiency. This process involved exploring various network architectures and optimization strategies to balance fine-grained material segmentation, precise boundary delineation, and overall structural consistency. The evaluation focused on segmentation accuracy, class separability, and geometric coherence in vectorization, ensuring the model effectively differentiated buildings from the ground while also capturing roof structures and materials. Systematic adjustments to key parameters improved the model’s robustness in both tasks, leading to better generalization for real-world applications. The final configuration was selected based on the optimal trade-off between segmentation precision, vectorization quality, and computational feasibility.

4.2.1. Single-Head Segmentation Experiments

Table 1. Evaluation of network architectures, backbones, and loss functions for single-head segmentation tasks. Performance is reported as $msIOU$ (%) for building and ground segmentation without material differentiation and $mIOU$ (%) for material-based segmentation. The highest-performing metrics in each experiment subset are highlighted in green.

Loss Function	Architecture	Backbone	w/o Material msIOU(%)	w/ Material mIOU(%)
CCE	UNet	ResNet18	75.66	19.85
		ResNet34	78.09	19.28
		ResNet50	70.73	15.98
		EfficientNet-b4	77.29	22.45
		EfficientNet-b5	76.42	21.60
		EfficientNet-b6	76.92	20.31
CCE	UNet	EfficientNet-b4	77.29	22.45
	FPN		71.49	18.54
	PAN		77.70	23.00
	PSPNet		68.28	15.86
CCE	PAN	EfficientNet-b4	77.70	23.00
Soft dice			75.49	18.42
Combo			77.44	22.89

presents the outcomes of the network selection experiments, where the model was trained using data from Brugge and validated on data from Jabbeke. This cross-validation approach was adopted to reduce computational time while ensuring robust model evaluation. By limiting the dataset used for training, the process efficiently balanced performance assessment with computational resource management. The primary metric for evaluation was the $mIOU$, which measures the model’s ability to accurately segment building and ground materials. Additionally, the $msIOU$ was calculated to evaluate the model’s capacity to distinguish between buildings and the ground, offering a comprehensive view of both material-specific and structural segmentation performance.

The results indicate that PAN, when combined with EfficientNet-b4 and trained using categorical cross-entropy (CCE) loss, achieved the highest $mIOU$ of 23.00%, demonstrating superior effectiveness in fine-grained material segmentation. In contrast, ResNet34 under the UNet architecture, also utilizing CCE, achieved the highest $msIOU$ of 78.09%, reflecting strong performance in differentiating buildings from their surroundings. However, EfficientNet-b4 with PAN offered the most balanced performance across both metrics, achieving an $msIOU$ of 77.70% while maintaining the highest $mIOU$ score. This dual optimization highlights PANet with EfficientNet-b4 as the most robust choice for segmentation tasks, delivering consistent and precise results across both class-specific and similarity-weighted evaluation metrics.

4.2.2. Multi-Head Segmentation Experiments

This section evaluates the impact of various performance optimization techniques and post-processing strategies designed to improve segmentation accuracy.

Table 2. Evaluation of the multi-head model performance on validation data. Performance is reported as $msIOU$ (%) for building and ground segmentation without material differentiation (Head 1) and $mIOU$ (%) for material-based segmentation (Head 1) and building edge segmentation (Head 2). The highest-performing metrics in each experiment subset are highlighted in green.

		Head 1 (Bldg. & Ground)		Head 2 (Bldg.)
		w/o Material msIOU (%)	w/ Material mIOU (%)	Bldg. Edge mIOU (%)
Baseline		76.55	22.65	50.19
+ Edge Weight	1	76.55	22.65	50.19
	10	78.91	25.87	55.14
	25	81.21	25.54	57.47
+ Edge Dilation	1 px.	81.21	25.54	57.47
	5 px.	80.90	24.60	58.31
	11 px.	84.60	25.54	63.83
	15 px.	82.22	22.89	66.63
+ Input Size	1024 × 1024	84.60	25.54	63.83
	2048 × 2048	86.30	26.38	63.69
	4096 × 4096	86.12	26.35	63.57
+ Multi-offset	(0)	86.30	26.38	63.69
	(0, 256)	86.83	26.85	63.61
	(0, 256, 512)	86.64	26.88	63.70
	(0, 256, 512, 1024)	86.98	26.98	63.85
+ DEM Threshold	0.2 m	87.21	27.01	63.70
	0.5 m	87.67	27.08	63.70
	1 m	87.97	27.09	63.70
	2 m	87.92	26.80	63.70

presents a comprehensive overview of the results obtained from the multi-head segmentation model. In the absence of an established baseline for material segmentation, we adopt the vanilla $PAN$ model with EfficientNet-b4 as the baseline for both segmentation and vectorization tasks. This model was selected based on its superior performance in single-head segmentation experiments, where it achieved the highest mean $mIOU$. The model was trained on datasets collected from three distinct regions: Brugge, Roeselare, and Oostende. These cities were specifically chosen for their diverse urban characteristics, which allowed the model to learn a wide range of building and ground materials. To assess the model’s generalization capability, validation was performed using data from Jabbeke, a location not included in the training set. This approach ensures that the model’s performance reflects its ability to handle previously unseen data, thereby enhancing its robustness.

The performance is assessed in terms of $msIOU$ (%) for building and ground segmentation without material differentiation (Head 1) and $mIOU$ (%) for material-based segmentation (Head 1) and building edge segmentation (Head 2). To optimize performance, the model architecture and training process were progressively refined through targeted improvements. These included adjustments to the loss function, modifications to input parameters, and the introduction of postprocessing techniques, with each step building on the previous one. The subsequent sections detail these strategies and analyze their influence on the model’s segmentation accuracy.

Edge weighting. A key strategy to improve segmentation performance involved incorporating edge weights into the model’s loss function, specifically targeting the sparse roof edges that are often underrepresented in training. These edges, being less prevalent than larger regions like roof surfaces or open ground, can be challenging for the model to capture. By assigning greater importance to roof edges during training, edge weighting encouraged the model to focus on these difficult-to-detect areas, enhancing its ability to capture fine structural details. The results showed that increasing the edge weight from 1 to 25 improved both $msIOU$ and $mIOU$ scores, indicating a better model focus on intricate edge features. With an edge weight of 25, the model achieved its highest $msIOU$ of 81.21% for building and ground segmentation. However, the $mIOU$ score at an edge weight of 25 was slightly lower than that at an edge weight of 10, suggesting that further increasing the edge weight beyond 25 negatively impacts material-based segmentation.

Edge dilation. In conjunction with edge weighting, edge dilation was employed to further enhance the model’s ability to segment edges accurately. While edge weighting emphasizes the importance of edges in the loss function, edge dilation focuses on the spatial representation of these edges during training. Narrow edge pixels, often limited to a single pixel, pose a challenge for segmentation models due to their sparse and subtle nature. By widening these edges through dilation, the model is provided with more prominent edge features to learn from, improving its capacity to identify structural boundaries. Increasing the dilation size from 1 to 11 pixels resulted in consistent improvements in both $msIOU$ and $mIOU$ scores, particularly in building edge segmentation (Head 2). The best performance was observed at an 11-pixel dilation, achieving an $msIOU$ of 84.60% for building and ground segmentation and an $mIOU$ of 25.54% for material-based segmentation.

Input size. The effect of input size on segmentation performance was thoroughly examined to determine its role in enhancing model accuracy. Although the input resolution remained constant, increasing the input size provided the model with a larger spatial context, allowing it to capture more comprehensive structural information. Expanding the input size from 1024 × 1024 to 2048 × 2048 pixels led to significant improvements in segmentation performance, with the model achieving an $msIOU$ of 86.30% for building and ground segmentation and an $mIOU$ of 26.38% for material-based segmentation. However, further increasing the input size beyond 2048 × 2048 pixels leads to a slight decrease in performance, suggesting that an optimal input size should be selected for the best results.

Multi-offset self-ensemble. To further improve segmentation consistency, a multi-offset self-ensemble approach was utilized. Single-offset configurations often lead to visible discontinuities along patch borders, which can degrade the overall segmentation quality, as shown in Figure 7. By integrating multiple spatial offsets, these boundary artifacts were significantly reduced, leading to smoother transitions and more coherent segmentation results. Self-ensembling with multiple offsets relies on the idea of overlap averaging, where the same spatial location is predicted multiple times under different patch alignments. When offsets are chosen such that they are evenly spaced and cover the stride range (e.g., 256 pixels), each region in the image is seen by the model from different spatial contexts, reducing bias introduced by fixed receptive field positioning and improving overall robustness. Among the tested configurations, the offset combination of (0, 256, 512, 1024) achieved the highest performance for an input size of 2048 × 2048; however, it required substantially greater computational resources and processing time. Consequently, a more computationally efficient configuration using offsets (0, 256, 512) was adopted, providing an optimal trade-off between segmentation accuracy and resource utilization. This approach achieved an $msIOU$ of 86.64% for building and ground segmentation and an $mIOU$ of 26.88% for material-based segmentation.

DEM-based filter. The adjustment of the DEM threshold was another post-processing strategy explored to optimize segmentation accuracy. As shown in Figure 8, the DEM threshold plays a crucial role in refining segmentation results. Given that the resolution of the DEM is lower than that of the RGB imagery, a threshold of 1 m was chosen to provide a buffer against misclassifications at building edges. This threshold was determined empirically through experiments across a range of candidate values (e.g., 0.2 m, 0.5 m, 1 m, 2 m), and 1 m yielded the best balance between minimizing false positives from low-lying artifacts and preserving true building detections—particularly for small or low-rise structures. The model achieved its highest $msIOU$ of 87.97% and $mIOU$ of 27.09% at this threshold. Higher DEM thresholds, such as 2 m, resulted in a slight drop in performance, indicating that a 1-m threshold is optimal for minimizing misclassification while preserving accurate building edge detection.

While the numerical improvements from the multi-offset self-ensemble and DEM-based filtering appear modest, their contributions to overall segmentation quality are substantial. These strategies significantly enhance the consistency and robustness of segmentation outputs, which is critical for subsequent vectorization processes, thereby improving the overall effectiveness of the segmentation pipeline.

4.2.3. Vectorization Experiments

Table 3. Evaluation of roof vectorization performance on validation data. Performance is reported as $SQ$ (%), $RQ$ (%), and $PQ$ (%) for vectorization without and with material information. The highest-performing $PQ$ metrics are highlighted in green.

	w/o Material			w/ Material
	SQ (%)	RQ (%)	PQ (%)	SQ (%)	RQ (%)	PQ (%)
Baseline	20.19	1.59	0.32	9.97	0.71	0.16
+ Edge Weight & Dilation	66.90	54.88	36.72	68.19	35.27	23.84
+ Input Size	69.95	53.57	37.47	71.01	35.06	24.71
+ Multi-offset	69.92	53.15	37.16	71.31	34.63	24.51
+ DEM Threshold	69.67	54.52	37.99	71.23	35.03	24.77

summarizes the performance metrics for roof vectorization under different configurations. The results indicate a significant improvement in $PQ$ scores, increasing from 0.32% (without material information) and 0.16% (with material information) to 37.99% and 24.77%, respectively, in the optimized configurations. Figure 9 illustrates the qualitative differences in vectorization performance between the conventional single-head segmentation approach and the proposed multi-head segmentation framework. Vectorization derived from single-head segmentation using QGIS struggles to accurately delineate distinct roof structures. In contrast, the multi-head approach addresses these shortcomings by incorporating dedicated segmentation heads for edges and materials. This architectural separation facilitates the precise identification and extraction of individual roof components, yielding detailed and structurally accurate vector representations of complex roof geometries.

4.3. Performance Evaluation on Test Data

Table 4. Quantitative evaluation of segmentation and vectorization performance on test data. Metrics are reported as $msIOU$ (%) for segmentation without material differentiation, $mIOU$ (%) for material-based segmentation, and $PQ$ (%) for building vectorization. The highest-performing metrics are highlighted in green.

	Segmentation (Bldg. & Gnd.)		Vectorization (Bldg.)
	w/o Mtrl msIOU (%)	w/ Mtrl mIOU (%)	w/o Mtrl PQ (%)	w/ Mtrl PQ (%)
Baseline	69.68	22.06	0.32	0.17
+ Edge Weight & Dilation	82.03	25.72	36.93	26.27
+ Input Size	82.58	28.04	37.05	28.50
+ Multi-offset	83.34	28.53	38.37	28.50
+ DEM Threshold	89.19	28.60	38.93	28.70

provides a comprehensive evaluation of segmentation and vectorization performance on the test datasets from Oostrozebeke and Wetteren. The performance metrics are reported as $msIOU$ for segmentation without material differentiation, $mIOU$ for segmentation that includes material differentiation, and $PQ$ for building vectorization. Initially, the baseline model demonstrates relatively low accuracy in both segmentation and vectorization tasks. Specifically, it achieves $msIOU$ and $mIOU$ scores of 69.68% and 22.06%, respectively. The $PQ$ scores for building vectorization are negligible, with values of 0.32% without material differentiation and 0.17% with material differentiation. These results suggest that the baseline model struggles with both the detection of complex segmentation boundaries and the accurate representation of building structures in the vectorized output.

The integration of various performance optimization techniques—including edge weighting, edge dilation, input size augmentation, multi-offset self-ensembling, and DEM-based filtering—leads to significant improvements across all evaluation metrics. The final optimized model achieves $msIOU$ and $mIOU$ scores of 89.19% and 28.60%, respectively, reflecting notable enhancements in segmentation accuracy. Similarly, building vectorization performance peaks with $PQ$ scores of 38.93% without material differentiation and 28.70% with material differentiation. These findings demonstrate the effectiveness of the combined optimization strategies in improving both segmentation and vectorization performance. The qualitative enhancements are further illustrated in Figure 10, which highlights the multi-head model’s improved capability in accurately segmenting urban environments and delineating building structures.

In addition to these evaluations, it is important to assess the computational efficiency of the proposed method. To this end, inference and post-processing times were measured using standardized input data. All experiments were conducted on an NVIDIA Tesla V100 GPU equipped with 32 GB of RAM. The inference time for multi-head segmentation with a single spatial offset, in combination with the DEM filter, on a $2048\times 2048$ image was approximately $0.84$ s. When using three spatial offsets (0, 256, and 512 pixels), the inference time increased to approximately $4.79$ s. This duration includes sequential execution of the segmentation head three times with the DEM filter, averaging the resulting probability maps, and generating the final output raster. Following segmentation, the polygonization of building footprints for a $2048\times 2048$ image required an additional $1.62$ s.

5. Discussion

Figure 11 illustrates several key factors that influence the accuracy of segmentation and vectorization in complex urban environments. A primary challenge arises from unannotated objects, such as vehicles on roads and solar panels on rooftops, which frequently lead to misclassifications during both segmentation and vectorization. These objects, not included in the training labels, are often incorrectly identified as part of the structures or materials they resemble. For example, vehicles, whether parked or in motion, create ambiguity by disrupting the continuity of road and ground surfaces. Similarly, solar panels, which share reflectance properties with roofing materials, are difficult to distinguish, especially when their placement varies across different roof types. As the model lacks training data for these elements, it tends to misclassify the roof areas containing solar panels, leading to boundary delineation errors that carry over into the vectorization stage. This highlights the need for improved annotation strategies and preprocessing techniques to better handle unannotated objects in urban datasets.

Another significant limitation stems from the patch-wise nature of the segmentation process. Although the multi-offset self-ensemble approach enhances boundary smoothness within individual patches, the vectorization algorithm still operates on discrete patches, which can result in discontinuities when structural elements span multiple segments. This issue is particularly seen in roof structures, where edges that extend across patch boundaries may become misaligned or fragmented during vectorization, leading to incoherent representations of the underlying architecture. The problem is further intensified in areas with intricate roof geometries, where maintaining consistent edge alignment across patches is critical for accurate delineation. Addressing this issue requires more adaptive vectorization strategies that enforce global continuity constraints across patches. Techniques such as graph-based optimization or topology-aware merging could enable more coherent representations of buildings while preserving fine structural details.

Class imbalance further complicates segmentation performance, particularly in the accurate identification of smaller or less frequent structures. As shown in Figure 12, dominant classes such as grass, asphalt, and roof tiles are segmented with higher accuracy, while smaller elements like gravel, wood, and glass exhibit significantly higher misclassification rates. This disparity arises from the unequal distribution of training samples, where models tend to favor well-represented classes, often neglecting the minority ones. To mitigate this issue, loss function weighting was implemented based on the training data distribution, but this approach did not yield substantial improvements. Consequently, smaller classes still suffer from lower recall, resulting in frequent omissions or incorrect segmentation.

The overall $mIOU$ for material segmentation remains relatively low, primarily due to challenges related to generalization and class imbalance. Since the model was trained on data from specific cities and tested on entirely unseen cities, its ability to generalize across different urban environments was significantly limited, as highlighted in previous studies on out-of-distribution (OOD) generalization, where even the best-trained models, including transformer-based models, struggled with OOD data [69,70]. To address this limitation, one potential solution is to annotate small subsets of data from the newly tested cities and use these annotations either to fine-tune city-specific models or to combine them into a more generalized model, similar to foundation models that have been trained on diverse datasets. Additionally, incorporating techniques such as few-shot learning and uncertainty-aware learning could enhance the model’s ability to detect and correct errors in underrepresented classes, ultimately leading to more robust segmentation performance and improved overall accuracy.

The current DEM, derived from LiDAR data, has a GSD of 25 cm, which still presents challenges in accurately delineating individual roof segments. Although LiDAR offers high vertical accuracy, the spatial resolution at this GSD remains insufficient for capturing fine-grained roof structures, especially in complex urban environments where roof forms vary significantly over short distances. This limitation restricts the DEM’s usefulness in identifying subtle architectural features or distinguishing between different roofing materials, which often require higher spatial detail. Consequently, the DEM is not included as part of the network input. Instead, it is used during postprocessing to differentiate buildings from ground surfaces, where elevation contrasts are more pronounced and reliable. Improving the spatial resolution of the DEM, either through denser LiDAR point clouds or advanced interpolation techniques, could potentially enhance its utility for both structural and material-based segmentation in future work.

Shadows present in aerial imagery are another important factor influencing segmentation performance. As demonstrated in Figure 11, the presence of shadows does not significantly degrade model accuracy, which can be attributed to the use of HSV-based data augmentation during training. This augmentation strategy introduces variations in brightness, hue, and saturation, allowing the model to better handle changes in illumination and maintain robustness across diverse lighting conditions. However, the degree to which shadows impact performance also depends on the characteristics of the imaging sensors. In cases where the sensors produce images with low dynamic range or poor contrast in shadowed areas, the network may struggle to accurately distinguish object boundaries or material transitions. These observations highlight the importance of both robust augmentation strategies and high-quality image acquisition to ensure reliable performance across varying illumination conditions.

Limitations and Future Work. Future research should explore the use of transformer-based architectures, particularly those capable of capturing long-range dependencies and contextual relationships within large-scale spatial data. Training these models on more diverse and extensive datasets could significantly enhance their robustness, generalization capabilities, and resilience to domain shifts across different urban regions. In parallel, few-shot learning techniques offer a promising direction for addressing challenges related to data scarcity. These approaches could not only improve model adaptability to previously unseen urban environments but also enhance the detection and segmentation of less frequent or rare classes, which are often underrepresented in training data yet critical for comprehensive urban analysis.

Another important area for future exploration involves the development of uncertainty-aware models. By quantifying confidence in segmentation outputs, these models could facilitate more targeted post-processing strategies and allow for better-informed decisions in downstream applications. Such capabilities would be especially valuable in safety-critical or resource-constrained scenarios, where reliability and interpretability are essential. Finally, the rapid advancement of vision-language models opens up exciting possibilities for the geospatial domain. These models enable joint reasoning across visual and textual modalities, which could lead to more intuitive and interactive systems for interpreting remote sensing data, querying urban features, and even guiding human-in-the-loop analysis. Leveraging such multimodal frameworks may pave the way for more accessible, explainable, and intelligent tools for urban understanding.