ConvNeXt with Context-Weighted Deep Superpixels for High-Spatial-Resolution Aerial Image Semantic Segmentation

Abstract

Semantic segmentation of high-spatial-resolution (HSR) aerial imagery is critical for applications such as urban planning and environmental monitoring, yet challenges, including scale variation, intra-class diversity, and inter-class confusion, persist. This study proposes a deep learning framework that integrates convolutional networks (CNNs) with context-enhanced superpixel generation, using ConvNeXt as the backbone for feature extraction. The framework incorporates two key modules, namely, a deep superpixel module (Spixel) and a global context modeling module (GC-module), which synergistically generate context-weighted superpixel embeddings to enhance scene–object relationship modeling, refining local details while maintaining global semantic consistency. The introduced approach achieves mIoU scores of 84.54%, 90.59%, and 64.46% on diverse HSR aerial imagery benchmark datasets (Vaihingen, Potsdam, and UV6K), respectively. Ablation experiments were conducted to further validate the contributions of the global context modeling module and deep superpixel modules, highlighting their synergy in improving segmentation results. This work facilitates precise spatial detail preservation and semantic consistency in HSR aerial imagery interpretation tasks, particularly for small objects and complex land cover classes.

1. Introduction

As a core task in remote sensing image analysis, semantic segmentation of aerial imagery involves categorizing each pixel into a predefined ground object class. This task holds critical importance in urban planning [1,2], ecological investigation [3], disaster evaluation [4], and agricultural monitoring [5,6,7]. The emergence of HSR aerial imagery has provided high-precision spatial decision support for these applications. However, the unique characteristics of HSR imagery introduce challenges to HSR semantic segmentation: ground object categories exhibit significant scale variations, ranging from large-area geographical entities, like vegetation and lakes, to small targets, such as individual trees and vehicles. Additionally, intra-class objects may differ drastically in material and shape, while inter-class objects may be easily confused due to similar color or texture. These complexities make traditional handcrafted feature-based methods inadequate for capturing the fine-grained differences inherent in HSR imagery [8].

Recent advancements in deep learning have demonstrated superior feature representation capabilities, enabling simultaneous feature extraction and segmentation in practice. The deep learning approach to remote sensing semantic segmentation primarily builds upon classic frameworks such as U-Net [9], DeepLab [10,11,12], and PSPNet [13], with adaptations tailored to remote sensing data. For example, multiscale feature fusion techniques (e.g., atrous convolution, pyramid pooling) are employed to capture objects of varying sizes by fusing features from different receptive fields [14,15], mitigating the scale variation issue. Studies have also introduced attention mechanisms, with examples including channel attention (SE module) [16], spatial attention (CBAM) [17], non-local [18], and GCNet [19], to suppress background noise and enhance feature responses in key regions. While CNN-based methods have achieved remarkable progress, their development has reached a bottleneck due to the inherent limitation of receptive field size. In recent years, transformer architectures have emerged as a research hotspot for HSR remote sensing segmentation [20,21]. Vision transformers are adept at capturing long-range dependencies, offering advantages in global context perception. Pure transformer models or hybrid CNN–transformer architectures have shown unique strengths in capturing global consistency across large scenes and modeling complex object relationships [22,23]. Nevertheless, transformer-based methods still face limitations in remote sensing segmentation practice: they still require patch-wise processing when handling HSR aerial imagery, which—similar to CNN-based approaches—compromises the global context to some extent. Additionally, the self-attention mechanism of transformers focuses on global statistical information, causing features of small objects to be easily diluted by large-area background features, resulting in segmentation accuracy of small objects comparable to that of CNNs. Although studies have attempted to alleviate these issues using dynamic attention mechanisms (e.g., sparse attention, local attention) [24,25], the large-scale practical implementation of transformers in remote sensing segmentation still requires further breakthroughs.

This paper presents a newly refined deep learning framework dedicated to HSR aerial imagery semantic segmentation. To address the aforementioned challenges, our framework integrates three main contributions:

(1)

Global context projection: The framework explicitly models scene–object relationships by projecting global contextual information into local feature representations, enhancing the understanding of spatial dependencies across large scenes.

(2)

Context-weighted superpixel embeddings: The framework generates superpixel-level feature embeddings weighted by global context, enabling discriminative representation of region-level semantic information while preserving fine-grained details.

(3)

Superpixel-guided upsampling: The framework incorporates region-level shape priors from superpixels to guide the upsampling process, optimizing edge details and reducing spatial misalignment in segmentation predictions.

Specifically, this framework adopts a transformer-inspired pure convolutional network ConvNeXt [26] as the backbone. We introduce a deep superpixel module that incorporates global context modeling, which models scene–object relationships using global context, generates context-weighted superpixel embeddings, and provides region-level shape priors to optimize edge details in semantic segmentation predictions.

2. Related Works

2.1. General Semantic Segmentation

Classical methods rely on handcrafted feature descriptors for pixel-level representation, whereas deep learning approaches like CNNs learn hierarchical features directly from pixels for pixel-wise classification. The fully convolutional network (FCN) [27] represents a landmark in pixel-wise segmentation, as it pioneered end-to-end dense prediction and set a foundational architectural paradigm for pixel-level tasks. In parallel, U-Net’s symmetric structure and skip connections demonstrate exceptional spatial detail recovery in refinement tasks, serving as a benchmark model in communities such as medical image segmentation and aerial image interpretation [28,29,30,31]. The DeepLab series introduced atrous convolutions to enlarge receptive fields [12], leveraging spatial contextual information to enhance model performance. Similarly, the Pyramid Pooling Module (PPM) integrates pyramid features of different scales to improve global context modeling [13]. While CNN convolutions primarily capture local features, self-attention mechanisms [25,32,33] have been progressively integrated into vision tasks to model long-range dependencies. Building on the achievements of transformers, vision transformer-based methods (e.g., Segformer [34], Swin-Unet [35]) have advanced pixel-level tasks, exhibiting competitive performance in semantic segmentation. Meanwhile, convolutional network structures inspired by the design of transformers, such as ConvNeXt, have also secured a prominent position in the field of semantic segmentation [36,37].

These generic approaches predominantly focus on multiscale context modeling to capture extensive dependencies and enhance scene semantic comprehension. However, HSR aerial imagery poses unique challenges: (1) larger image coverage with diverse object sizes and severe class imbalance hinders explicit modeling of complex ground objects and (2) demands for precise local detail modeling of small objects further intensify task difficulty.

2.2. Semantic Segmentation in Aerial Images

Deep learning-based semantic segmentation techniques have been widely adopted in remote sensing for applications, including building/road extraction [38,39,40], vehicle detection [41,42,43], and land cover classification [44,45,46]. While these applications follow conventional segmentation paradigms, they incorporate adaptations for HSR imagery and specific recognition targets. To address challenges like small-scale objects and inter-class similarity, studies have extracted multiscale features to enhance network performance [47,48], moderately improving segmentation accuracy for objects of varying sizes.

Attention mechanisms are extensively utilized to boost feature representation, typically by strengthening long-range dependencies within feature maps [33,49]. Leveraging contextual information also proves critical [50,51], where integrating attention and multiscale features in CNNs facilitates accurate context modeling. Transformer-based architectures have been further applied to remote sensing segmentation. These methods treat images as patch sequences, flattening patches into vectors and modeling global relationships via self-attention to capture arbitrary-range dependencies [52,53,54]. Such works validate transformers’ applicability to remote sensing tasks.

However, compared to the progressive downsampling in CNNs, the patch embedding process in vision transformers involves implicit downsampling. This leads to the loss of local correlations between pixels within patches, reduced spatial resolution in feature maps, and compromised local detail preservation. Consequently, these limitations pose significant challenges for precisely identifying fine-grained structures (e.g., building outlines and road boundaries) in aerial imagery. In recent years, several studies have incorporated superpixels to leverage object-level information, aiming to refine the edge details of semantic segmentation results [53,55,56]. However, their efficacy remains contingent upon the accuracy of superpixel segmentation, which invariably induces novel semantic ambiguities. Consequently, it remains imperative to further investigate the optimal strategy for integrating these two components.

3. Methods

A schematic diagram of the proposed framework is depicted in Figure 1. A symmetric encoder is employed to process the input image and generate pyramid features. Furthermore, deep features within the multiscale features generate global descriptors, leveraging the global context to compensate for the semantic consistency of the superpixel module. In the decoder, multi-level feature aggregation is performed to generate a prediction map, and segmentation boundary optimization is completed based on the generated superpixels.

3.1. Symmetric Encoder

For the symmetric encoder, ConvNeXt was employed as the backbone network in this work. As a pure convolutional deep network, ConvNeXt achieves performance comparable to vision transformers (e.g., Swin Transformer [20]). It was chosen as the feature extractor due to its structural simplicity and efficiency. ConvNeXt adopts a ResNet-like architecture: its initial “stem” layer utilizes a 4 × 4 convolution kernel and a stride of 4 to perform 4× downsampling on the input image. Four subsequent stages are then configured to extract feature maps (denoted as ${\mathrm{S}}_{\mathrm{i}}|\mathrm{i}=2, 3, 4, 5$), where each stage consists of stacked basic blocks. These blocks comprise modules such as depthwise convolution, layer normalization, and a multilayer perceptron (MLP). Downsampling with a stride of 2 is applied between consecutive stages, resulting in channel dimensions of 128, 256, 512, and 1024 for the four stages, respectively. Detailed specifications of the ConvNeXt network used in this study are provided in

Table A1. The left-path structure based on ConvNeXt in the symmetric encoder.

Name	Kernel	Stride	Input	OutDim
Input
Image				H × W × 3
Stem
Conv0	4 × 4, 128	4	Image	${\displaystyle \frac{1}{4}}\mathrm{H} \times {\displaystyle \frac{1}{4}}\mathrm{W} \times 3$
Encoder feature extractor
Conv_S2	$\left[\begin{array}{c}7 \times 7, 128\\ 1 \times 1, 512\\ 1 \times 1, 128\end{array}\right] \times 3$	1	Conv0	${\displaystyle \frac{1}{4}}\mathrm{H} \times {\displaystyle \frac{1}{4}}\mathrm{W} \times 128$
Conv_S3	$\left[\begin{array}{c}7 \times 7, 256\\ 1 \times 1, 1024\\ 1 \times 1, 256\end{array}\right] \times 3$	2	Conv_S2	${\displaystyle \frac{1}{8}}\mathrm{H} \times {\displaystyle \frac{1}{8}}\mathrm{W} \times 256$
Conv_S4	$\left[\begin{array}{c}7 \times 7, 512\\ 1 \times 1, 2048\\ 1 \times 1, 512\end{array}\right] \times 9$	2	Conv_S3	${\displaystyle \frac{1}{16}}\mathrm{H} \times {\displaystyle \frac{1}{16}}\mathrm{W} \times 512$
Conv_S5	$\left[\begin{array}{c}7 \times 7, 1024\\ 1 \times 1, 4096\\ 1 \times 1, 1024\end{array}\right] \times 3$	2	Conv_S4	${\displaystyle \frac{1}{32}}\mathrm{H} \times {\displaystyle \frac{1}{32}}\mathrm{W} \times 1024$
PPM_feat	Pyramid Pooling Module	1	Conv_S5	${\displaystyle \frac{1}{32}}\mathrm{H} \times {\displaystyle \frac{1}{32}}\mathrm{W} \times 512$

.

Similar to typical symmetric networks, a symmetric pathway was designed to progressively recover multiscale pyramid feature maps (${\mathrm{P}}_{\mathrm{i}}|\mathrm{i}=2, 3, 4, 5$) with restored spatial resolution. The encoder and decoder pathways were connected via skip connections, and the generation process of Pi is described as follows:

$${\mathrm{P}}_{\mathrm{i}}=\mathrm{S}\mathrm{k}\mathrm{i}\mathrm{p}\left({\mathrm{S}}_{\mathrm{i}}\right)+\mathrm{U}\mathrm{p}({\mathrm{P}}_{\mathrm{i}+1},\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}=2),\mathrm{i}=2, 3, 4,$$

(1)

where Skip(∙) denotes a skip connection implemented via a 1 × 1 convolution layer with 256 channels and Up(∙, scale = 2) represents nearest-neighbor upsampling with a scale factor of 2. This structure aggregates deep-layer semantic information and shallow-layer spatial details, facilitating semantic modeling of remote sensing ground objects and recovery of spatial details. Additionally, PPM was introduced following S₅ to obtain the context-enhanced representation P₅ [57]. Finally, the multiscale pyramid feature maps P_i, with 256 channels, were output to the decoder, where feature map P₂ was additionally used to generate deep superpixels, the details of which are elaborated in subsequent sections.

3.2. Deep Superpixel Module with Integrated Contextual Attention

This study builds on our previous work [53], beginning with a review of the deep superpixel (Spixel) generation process. The core concept of deep superpixels involves using reconstruction loss to guide a fully convolutional neural network in generating a pixel–superpixel affinity matrix, which assigns each pixel to its adjacent superpixel grids. Mathematically, this process involves inputting the image $\mathrm{I}\in {\mathcal{R}}^{\mathrm{H}\times \mathrm{W}\times 3}$ into the neural network and outputting the affinity matrix $\mathrm{Q}\in {\mathcal{R}}^{\mathrm{H}\times \mathrm{W}\times 9}$. Following Yang’s work [58], to simplify computation, the predicted affinity matrix Q only considers the membership probabilities of each pixel to its 9 neighboring superpixel grids. Typically, the affinity matrix can cluster pixels into superpixels; conversely, it can also reconstruct pixel attributes from superpixels. Mathematically, this process is described as:

$$\mathrm{f}(\mathrm{s})={\displaystyle \frac{{\sum }_{{\mathrm{p}:\mathrm{s}\in \mathrm{N}}_{\mathrm{p}}}{\mathrm{f}}_{i,j}(\mathrm{p})·{\mathrm{q}}_{i,j}(\mathrm{p},\mathrm{s})}{{\sum }_{{\mathrm{p}:\mathrm{s}\in \mathrm{N}}_{\mathrm{p}}}{\mathrm{q}}_{i,j}(\mathrm{p})}},$$

(2)

$${\mathrm{f}}_{\mathrm{i},\mathrm{j}}^{\prime }\left(\mathrm{p}\right)= {\sum }_{{\mathrm{s}\in \mathrm{N}}_{\mathrm{p}}}\mathrm{f}(\mathrm{s})·{\mathrm{q}}_{i,j}\left(\mathrm{p}\right),$$

(3)

where f(s) denotes the attribute of superpixel S, f_i,j(p) is the attribute of pixel p at coordinates (i, j), ${\mathrm{f}}_{\mathrm{i},\mathrm{j}}^{\prime }\left(\mathrm{p}\right)$ represents the reconstructed attribute of pixel p, N_p is the set of neighboring superpixels of pixel p, q_i,j(p,s) is the probability of pixel p belonging to superpixels, as indicated by the neural network-predicted affinity matrix, and ${\sum }_{{\mathrm{p}:\mathrm{s}\in \mathrm{N}}_{\mathrm{p}}}{\mathrm{q}}_{\mathrm{i},\mathrm{j}}(\mathrm{p})$ is a normalization constant.

Building on the implicit learning of pixel–superpixel grid relationships by deep superpixels, this study proposes using a global context modeling module (GC-module) to model scene–object relationships. Context-weighted superpixel embeddings are employed to generate the affinity matrix Q, thereby improving the local details of semantic segmentation. Specifically, a projection mapping function is applied to feature map S₅ to condense global information into global statistics $\overline{\mathrm{G}}\in {\mathcal{R}}^{1\times 1\times \mathrm{c}}$. This projection mapping function consists of global average pooling (GAP) and a trainable 1 × 1 convolution layer with 256 channels. Subsequently, the sigmoid function is used to convert the global descriptor $\overline{\mathrm{G}}$ into a channel weight vector $\overline{\mathrm{V}}$, which is then applied for channel-aligned weighting with feature map P₂. To avoid content information loss during the re-weighting process, a shortcut connection is introduced to sum up two features, in order to preserve the content of the original feature map as much as possible. Finally, the fused feature map is used to generate the affinity matrix Q. We formulate this procedure as follows:

$$\mathrm{G}\mathrm{A}\mathrm{P}({\mathrm{S}}_5)={\displaystyle \frac{1}{\mathrm{H}\times \mathrm{W}}}{\sum }_{\mathrm{i}=1}^{\mathrm{H}}{\sum }_{\mathrm{j}=1}^{\mathrm{W}}{\mathrm{S}}_5(i,j,c),$$

(4)

$$\overline{\mathrm{G}}=\sigma \left({\mathrm{W}}_{256}·\mathrm{G}\mathrm{A}\mathrm{P}({\mathrm{S}}_5)+\mathrm{b}\right), \overline{\mathrm{V}}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}(\overline{\mathrm{G}}),$$

(5)

$$\mathrm{Q}=\sigma \left({\mathrm{W}}_9·(\left(\overline{\mathrm{V}}·{\mathrm{P}}_2\right)+{\mathrm{P}}_2)+\mathrm{b}\right)$$

(6)

where W_c is a learnable 1 × 1 convolution layer with c channel, b is a bias term, and σ denotes the ReLU activation function.

3.3. Decoder

The decoder stage involves two processes: feature aggregation and superpixel refinement. Multiscale pyramid feature maps (${\mathrm{P}}_{\mathrm{i}}|\mathrm{i}=2, 3, 4, 5$) are aggregated via concatenation. Prior to concatenation, features P₃, P₄, and P₅ are upscaled to a size equivalent to 1/4 of the input image (i.e., spatially aligned with P₂). The aggregated features are followed by a learnable 1 × 1 convolutional layer to produce pixel-level features with 512 channels, which are then fed into a classification head to generate semantic segmentation logits. These logits are restored to the input image size using the superpixel mapping matrix Q. This upsampling process can be interpreted as reconstructing the spatial resolution of pixel-wise semantic segmentation based on the superpixel centers defined in Q. Ultimately, a semantic object segmentation result refined by deep superpixels is obtained.

3.4. Loss Function

Cross-entropy loss L_ce combined with Dice loss L_dice is adopted as the semantic segmentation loss function for this experiment. The former measures the per-pixel classification error, while the latter evaluates the similarity (i.e., overlap) between the model predictions and ground-truth labels, which is tailored for imbalanced datasets. The semantic segmentation loss L_seg is denoted as:

$$L_{ce}=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\sum }_{k=1}^K{y}_i^k\log \left({\widehat{y}}_i^k\right),$$

(7)

$$L_{dice}=1-{\displaystyle \frac{2}{N}}{\sum }_{i=1}^N{\sum }_{k=1}^K{\displaystyle \frac{y_i^k{\widehat{y}}_i^k}{y_i^k+{\widehat{y}}_i^k,}}$$

(8)

$$L_{seg}=L_{ce}+L_{dice},$$

(9)

where the number of samples is denoted by n, ${\widehat{y}}_i^k$ denotes the probability that the i-th prediction belongs to class k, and $y_i^k$ denotes the i-th ground truth.

Additionally, an object loss L_obj is introduced in the deep superpixel branch. Following Yang’s work, the attributes of pixel p include semantic labels sem_i,j(p) and coordinate vectors loc_i,j(p), and the entire reconstruction loss is implemented as:

$$L_{obj}={\sum }_{\mathrm{P}}\mathrm{E}({\mathrm{s}\mathrm{e}\mathrm{m}}_{\mathrm{i},\mathrm{j}}(\mathrm{p}),{\mathrm{s}\mathrm{e}\mathrm{m}}_{\mathrm{i},\mathrm{j}}^{\prime }(\mathrm{p}))+{\displaystyle \frac{\mathrm{m}}{S}}‖{\mathrm{l}\mathrm{o}\mathrm{c}}_{\mathrm{i},\mathrm{j}}(\mathrm{p})-{\mathrm{l}\mathrm{o}\mathrm{c}}_{\mathrm{i},\mathrm{j}}^{\prime }(\mathrm{p})‖$$

(10)

where E(∙) denotes the cross-entropy loss, S is the superpixel sampling interval, and m is empirically set to 0.03.

Finally, the total loss of the model is a weighted sum of the above loss functions, expressed as:

L_total = L_seg + α L_obj,

(11)

where weight factor α is empirically set to 0.5.

4. Experimental Setting

4.1. Datasets

• Vaihingen Dataset.

The Vaihingen dataset is a well-established ISPRS benchmark for urban remote sensing semantic annotation. It includes 33 HSR aerial images (ground sampling distance = 9 cm). Each image contains three multispectral bands and a normalized digital surface model (nDSM). The ground objects are annotated into six categories. Following previous studies [22,23], the dataset was split by allocating 15 images for training, 1 image for validation, and 17 for testing. In order to conduct a fair performance comparison with the results reported by existing methods, the nDSM data were excluded from this experiment.

2.

Potsdam Dataset.

For the ISPRS urban semantic labeling project, the Potsdam dataset offers 38 HSR images (6000 × 6000 pixels) with a 5 cm GSD, covering larger geographical areas than Vaihingen. Each image includes annotations for six object classes. Consistent with existing research [22,23], 23 RGB images were utilized for training, 1 for validation, and 14 for testing, and the nDSM data were also excluded from this experiment.

3.

UV6K Dataset.

This high-resolution urban vehicle segmentation dataset contains 6310 images (1024 × 1024 pixels) with 245,141 annotated vehicle instances [43]. The imagery features a spatial resolution of 0.1 m. As vehicles represent typical small-scale foreground objects in remote sensing tasks—characterized by severe foreground–background imbalance—this dataset effectively evaluates model performance, particularly in addressing class imbalance challenges. Following the publisher’s split, the dataset was divided into training (4100 images), testing (1710 images), and validation (500 images) subsets.

The ISPRS Vaihingen and ISPRS Potsdam datasets are two well-established high-resolution aerial image datasets that have been extensively utilized to evaluate the performance of semantic segmentation models in the remote sensing community [41]. On the other hand, these two datasets are relatively small-scale aerial image datasets. To better assess the model’s performance in small object segmentation, we introduced the UV6K dataset (Figure 2), a large-scale aerial image dataset of urban vehicles. Small vehicles, as representative geospatial objects in foreground–background imbalanced scenarios, serve as critical evaluation targets for testing segmentation robustness under such challenging conditions. Consequently, in this study, we aimed to comprehensively evaluate the segmentation performance of the developed model using three distinct aerial image datasets, ensuring robustness across varying scales and object characteristics.

4.2. Implementation Details

The proposed model was benchmarked against several advanced segmentation methods, including Deeplabv3+ [12] (spatial context modeling), OCRNet [59] (attention mechanism), Segformer (transformer-based architecture), Unetformer [22] (hybrid CNN–transformer design), and ESPNet [53]. Experimental implementation of all models was performed via PyTorch v1.13.1 on an NVIDIA RTX 3090 GPU. A batch size of four was adopted, with the maximum number of epochs set to 30. The Adam optimizer was employed with an initial learning rate of 0.0001 and a weight decay of 0.01, while cosine annealing was used to adjust the learning rate. The images were randomly cropped to 512 × 512 patches for input. Training data augmentation, including horizontal flipping, vertical flipping, and 90-degree rotation, was randomly applied. For inference, non-overlapping sliding windows of 1024 × 1024 pixels were used to process the high-resolution images, and the predicted results were stitched together. All reproduced methods were tested under the same experimental environment. Furthermore, to enable a more comprehensive comparison, experimental results from recent advanced approaches reported in the literature were further incorporated, such as large-model-based segmentation methods (e.g., Mamba [60] and SAM [61]).

Following practices in prior studies, Intersection over Union (IoU) and the F1-score were adopted as accuracy metrics. Mean IoU (mIoU) is the average of IoU across all classes, and the mean F1 (mF1) follows a similar calculation. The formulas for two metrics are provided below:

$$\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{TP}{TP+FP+FN}},$$

(12)

$$\mathrm{F}1={\displaystyle \frac{2\times TP}{2\times TP+FP+FN}},$$

(13)

where TP corresponds to the true positives of a specific class in the confusion matrix, FP represents the false positives, and FN refers to the false negatives.

5. Results and Analysis

5.1. Results and Comparison on the Vaihingen Dataset

Table 1. Experimental results on the Vaihingen test set. Bold values are the best.

Method	Backbone	IoU (%)					mIoU (%)	mF1 (%)
		Impervious Surface	Building	Low Vegetation	Tree	Car
DeepLabv3+ [12]	ResNet-101	73.76	84.48	63.96	77.46	61.29	72.19	83.28
OCRNet [59]	HRNet-48	87.62	92.84	74.55	82.46	82.73	84.04	90.71
SegFormer [34]	Mit-b5	87.46	92.72	72.81	82.13	78.13	82.65	90.00
UNetformer [22]	ResNet-18	85.76	92.78	67.33	83.22	80.75	81.97	89.85
RS3Mamba [63]	R18-MambaT	86.62	93.83	67.84	83.66	81.97	82.78	90.34
SAM-RS [62]	Swin-B	88.20	94.53	69.16	84.53	83.64	84.01	91.08
ESPNet [53]	ConvNeXt-T	88.40	93.63	74.50	82.12	82.94	84.32	91.06
the proposed model	ConvNeXt-T	88.70	93.84	74.69	82.35	83.13	84.54	91.17

shows that the proposed model outperforms seven state-of-the-art methods, with substantial gains in both the mIoU and mF1 metrics on the Vaihingen test set. These results show the effectiveness of our method in capturing object and boundary representations on HSR aerial imagery. It is worth noting that the incorporation of global context information and deep superpixels substantially enhances semantic segmentation performance, with the proposed method achieving mIoU and mF1 values of 84.54% and 91.17%, respectively, outperforming representative methods such as DeepLabv3+, OCRNet, and SegFormer. For example, the proposed method achieves an IoU of 88.7% on impervious and 74.69% for low vegetation, which are improvements of 2.94% and 7.36%, respectively, compared to UNetformer, and 0.5% and 5.53% increments relative to SAM-RS [62], respectively. Furthermore, the car class exhibits an IoU improvement of 1.16% compared to RS³mamba [63] and a comparable IoU for the building class. This is attributed to the inherently normalized shapes and distinct boundary features of these two classes, consistent with the strength of our deep superpixel module in capturing fine-grained object structure. Compared with ESPNet, there is also a certain improvement in the car class, indicating that introducing global context can enhance superpixels.

On the other hand, the tree class, with its blurred boundaries and varying spatial scales, shows moderate but meaningful improvements. For example, our method outperforms DeepLabV3+ and SegFormer in the tree class, achieving 4.89% and 0.22%, respectively, in terms of IoU, but falls behind the advanced RS³Mamba (1.31% IoU) and SAM-RS (2.18% IoU). Overall, the proposed method achieves consistent gains across all classes, maintaining its lead in mIoU and mF1.

Figure 3 presents a visual comparison of the results produced by the existing advanced methods and our approach on the Vaihingen test set. The comparison in the regions of interest (purple boxes) in each scenario shows that the proposed model has a distinct advantage in capturing fine features and maintaining semantic consistency. For the first scenario (a hollow rectangular building), DeepLabv3+ and SegFormer incorrectly classified the roof greening area as “vegetation”, while the proposed model had the most complete segmentation of the “building” category, indicating that the proposed model has better performance in using context relations to solve the semantic ambiguity within objects. In the second scenario (low vegetation and trees), the proposed model was least affected by shadows, and compared to the other models, it could more accurately distinguish “trees” and “low vegetation” pixels, while other models showed boundary fragmentation or loss of small objects. For the third scenario, OCRNet, SegFormer, and the proposed model all successfully identified the cars that were shaded and not labeled in the ground truth. The proposed model further detected sparse low vegetation in the upper part of the image, indicating stronger sensitivity to small and spatially dispersed objects. These visual results are consistent with quantitative indicators, thereby confirming the validity of the presented model in complex scenarios.

5.2. Results and Comparison on the Potsdam Dataset

Table 2. Experimental results on the Potsdam test set. Bold values are the best.

Method	Backbone	IoU (%)					mIoU (%)	mF1 (%)
		Impervious Surface	Building	Low Vegetation	Tree	Car
DeepLabv3+ [12]	ResNet-101	87.27	92.94	76.24	75.2	93.2	84.97	89.88
GCNet [19]	ResNet-50	87.47	93.24	77.04	79.67	91.70	85.82	/
OCRNet [59]	HRNet-48	91.53	96.04	80.7	80.75	95.86	88.98	92.69
SegFormer [34]	Mit-b5	91.85	96.58	82.21	82.21	95.47	89.66	93.11
SBSS-MS [64]	ConvNeXt-T	88.95	94.88	79.42	81.82	93.34	87.68	/
ESPNet [53]	ConvNeXt-T	92.40	97.21	82.59	82.50	95.95	90.13	93.54
the proposed model	ConvNeXt-T	92.63	97.50	83.26	82.95	96.59	90.59	93.69

presents the quantitative results on the Potsdam test set. The proposed model attains a new performance, with an mIoU of 90.59% and an mF1 of 93.69%, surpassing the six competitive comparison methods. It exhibits superior performance across all assessed land cover categories, with notable improvements in the building and car classes, which are characterized by regular geometric structures and high background contrast. Specifically, over SegFormer, the proposed method yields mIoU and mF1 improvements of 0.93% and 0.58%, respectively. The IoU for the building and car classes improves by 0.92% and 1.12%, respectively. While the distinctive edge features of man-made structures are more prominent in very high-resolution aerial imagery, the proposed method’s enhanced ability to capture object boundaries leads to stronger performance.

For complex classes such as low vegetation and trees, the proposed method still demonstrates significant improvements over SBSS-MS and OCRNet. For example, the low vegetation category improves by 3.84% IoU over SBSS-MS and 2.56% IoU over OCRNet, reflecting the ability of our model to handle heterogeneous vegetation distribution through global context-driven object consistency constraints. In addition, the strong urban scene segmentation methods, i.e., SegFormer and OCRNet, also perform slightly worse than our method in the impervious surface category, indicating that the proposed framework better utilizes high-resolution spatial features to solve the fine-grained classification task of complex urban remote sensing scenes. Compared with the previous ESPNet, the proposed model achieves improvements in both overall accuracy and individual categories.

Figure 4 presents visual comparisons of the segmentation results on the Potsdam test set, with purple bounding boxes highlighting regions of interest. The proposed model exhibits precision in recognizing complex ground objects, most obviously characterized by fewer noise artifacts compared to competing methods. For the first scene, the proposed model delineates building boundaries more completely, and there is less confusion in the pixel boundaries of adjacent different objects. In the second and third scenes—where distorted objects increase boundary recognition difficulty—the proposed model demonstrates robustness, maintaining clear class separation even in highly cluttered areas. These visual improvements align with the quantitative metrics in Table 2.

5.3. Results and Comparison on the UV6K Dataset

The object types and difficulty of the UV6K dataset are significantly different from those of the aforementioned datasets.

Table 3. Experimental results on the UV6K test set. Bold values are the best.

Method	Backbone	IoU	F1
DeepLabv3+ [12]	ResNet-101	53.63	69.82
OCRNet [59]	HRNet48	56.79	72.44
Segformer [34]	Mit_b5	63.68	77.81
Farseg++ [43]	ResNet-50	64.40	78.30
ESPNet [53]	ConvNeXt-T	63.28	77.51
the proposed model	ConvNeXt-T	64.46	78.39

demonstrates that our approach achieved competitive performance, with an IoU of 64.46% and an F1-score of 78.39% for the segmentation of a large number of small-scale objects (vehicles), significantly outperforming the baseline methods, such as DeepLabv3+ (IoU/F1: 53.63%/69.82%) and OCRNet (IoU/F1: 56.79%/72.44%). Compared to SegFormer and ESPNet, which performed well on the ISPRS dataset, our method achieved 0.78% and 1.18%, respectively, in terms of IoU, and 0.58% and 0.88% in terms of the F1-score. Notably, our proposed model achieved comparable or even slightly superior performance to Farseg++ (IoU/F1: 64.40%/78.30%), a strong performance method reported by the publisher of this dataset.

Overall, the robust efficacy of the developed method on the different datasets mentioned above demonstrates that the model can be effectively generalized to various remote sensing scenarios beyond standard benchmark datasets.

Figure 5 presents a visualization example from the UV6K dataset. Focusing on the region of interest (the purple box), the proposed model demonstrates high accuracy in identifying small objects, such as the recognition results of the disordered vehicles in the third sub-figure. Such performance places the model at the top among the compared methods. These visual observations align closely with the improvements in F1 and IoU metrics reported in Table 3.

5.4. Ablation Studies

To analyze the influence of individual components on segmentation performance, an ablation study was performed using a baseline ConvNeXt-T segmentation model without the two modules. The results are shown in

Table 4. Ablation studies on the Vaihingen test set.

Method	ConvNeXt-T	GC-Module	Spixel	mIoU (%)	mF1 (%)	Δparams (M)
Baseline	√			83.13	90.37	0
w/global context	√	√		83.62	90.58	1.190
w/deep superpixel	√		√	84.18	90.98	0.005
The proposed model	√	√	√	84.54	91.17	1.195

Note: The symbol “√” indicates that the corresponding module is included in the model architecture for that experimental configuration. All configurations are built upon the baseline model; each row represents a variant where specific modules are either retained (√) or ablated (absence of “√”).

, where the global context modeling module (GC-module) and deep superpixel module (Spixel) were added sequentially, and both modules were finally combined to form the proposed model.

Baseline Model: The baseline model attained an mIoU of 83.13% and an mF1-score of 90.37% without additional modules, serving as a reference for performance comparison.

Effect of the Global Context Module: Adding the GC-module alone improved mIoU by 0.49% and mF1 by 0.21%, with an increase of 1.19 M parameters. This indicates that modeling global contextual dependencies helps capture long-range semantic relationships, enhancing feature discrimination.

Effect of the Superpixel Module: Incorporating the Spixel module alone yielded a more significant improvement, with mIoU increasing by 1.05% and mF1 by 0.61%, while adding 0.005M parameters. This suggests that superpixel-based local structure modeling effectively refines spatial details, reducing false negatives in small object segmentation.

Combined Effect of Both Modules: When both the GC-module and Spixel module were integrated into the proposed model, it achieved optimal performance with an mIoU of 84.54% and an mF1 of 91.17%. The total parameter increase was 1.195 M, indicating synergistic effects between the global context and local structure modeling.

These findings indicate that the proposed modules complement each other: the GC-module enhances semantic coherence at the global level, while the Spixel module improves spatial precision at the local level, collectively leading to excellent semantic segmentation performance.

5.5. Sensitivity Analysis of the Spixel Module

To further investigate the impact of the superpixel sampling interval S (a key parameter for deep superpixels) on model performance, a sensitivity analysis was performed on the Vaihingen test set, with S set to 4, 8, and 16. The evaluation metrics included mIoU and mean F1-score. Smaller S values indicate a more fragmented segmentation, while larger S values indicate a larger average size of the generated superpixels.

The experimental results in Figure 6 show that model performance decreases with increasing S values. Specifically, when S = 4, mIoU and mF1 reach their highest values. As S increases to 8, mIoU and mF1 decrease slightly, to 83.70% and 90.71%, respectively. When S is further increased to 16, mIoU and mF1 continue to decrease to 83.48% and 90.49%, respectively. This trend suggests that a smaller sampling interval can strengthen the model’s ability to capture fine-grained object boundaries by generating more fragmented superpixels, thereby improving segmentation accuracy. Furthermore, when S is set to 4, the deep superpixel module simultaneously applies 4× upsampling for semantic segmentation when generating superpixels (i.e., the upsampling process is considered as a feature reconstruction). In summary, S = 4 is selected as the optimal parameter in this model because it strikes a balance between detail boundary preservation and segmentation performance.

5.6. Model Complexity Analysis

To assess the practicality of the proposed model, four key metrics—floating-point operations (FLOPs), model parameters, memory footprint, and frames per second (FPS)—were used to analyze its computational complexity and resource consumption (

Table 5. Computational complexity analysis and parameter comparison. Results on two 256 × 256 images from the ISPRS Vaihingen dataset using a single NVIDIA GeForce RTX 3090 GPU. Bold values are the best.

Model	Backbone	FLOPs (G)	Parameter (M)	Memory (MB)	FPS (Image/s)	MIoU (%)
DeepLabv3+ [12]	ResNet-101	127.45	60.21	1913	13.58	72.19
OCRNet [59]	HRNet-48	82.74	70.53	1080	9.17	84.04
SegFormer [34]	Mit-b5	3.14	2.55	1185	6.46	82.65
Unetformer [22]	ResNet-18	5.87	11.68	167	2.87	81.97
RS3Mamba [63]	R18-MambaT	28.25	43.32	2332	/	82.78
SAM-RS [62]	Swin-B	50.84	96.14	5176	3.50	84.01
ConvNeXt-U [37]	ConvNeXt	15.61	80.30	669	16.13	60.43
the proposed model	ConvNeXt-T	133.21	83.69	1087	2.99	84.54

). These metrics reflect computational efficiency, scale, memory requirements, and inference speed, respectively, with an optimal model expected to balance high segmentation accuracy, fast inference, and low resource consumption. Table 5 presents a comparative analysis of the proposed model against advanced semantic segmentation models on the Vaihingen test set. The presented model achieves the highest mIoU of 84.54%, outperforming all the compared methods, including OCRNet (84.04%), SAM-RS (84.01%), and RS³Mamba (82.78%). Moreover, the proposed model’s FPS is 2.99, which is on par with the fastest Unetformer (2.87), demonstrating its fast inference speed. However, this performance gain is accompanied by higher computational complexity: the proposed model exhibits the highest FLOPs (133.21 G) and parameter count (83.69 M) among the evaluated models.

In terms of the accuracy–performance trade-off, the proposed model prioritizes segmentation accuracy, achieving a 0.50% mIoU improvement over the second-best model, OCRNet. This gain is particularly notable for fine-grained remote sensing tasks where precise pixel classification is essential, including urban land cover mapping and ecological environment investigation. While its FLOPs and parameters are higher than lightweight models, like SegFormer and UNetformer, the significant accuracy improvement demonstrates that the effort to enhance baseline model performance by introducing additional modules is worthwhile, even with the accompanying risk of increased computational complexity. In terms of memory efficiency, the proposed model demonstrates favorable memory usage: its memory footprint (1087 MB) is comparable to that of OCRNet (1080 MB) and significantly lower than those of SAM-RS (5176 MB) and RS³Mamba (2332 MB). This suggests the model optimizes memory usage while pushing accuracy boundaries, making it more feasible for deployment compared to memory-intensive alternatives. Overall, despite exceeding lightweight models in computational complexity, its superior segmentation performance and moderate memory footprint make it a practical choice for applications requiring advanced aerial image semantic segmentation, such as high-resolution urban mapping and environmental monitoring. This trade-off highlights the model’s suitability for research and industrial settings where the benefits from improved accuracy may outweigh computational costs. Reducing the computational overhead of the proposed model will be a key focus of future work to further enhance its flexibility.

6. Conclusions

This study presents a refined deep learning framework for semantic segmentation of high-resolution aerial images, addressing the key challenges of scale variation and preservation of fine details. By integrating ConvNeXt as the backbone network with modules such as global context enhancement and deep superpixels, this framework achieves outstanding performance on multiple benchmark datasets. The integration of global context modeling enhances semantic consistency, while deep superpixel refinement improves local boundary fidelity, particularly beneficial for small targets and complex land cover categories. Ablation experiments confirm that the synergistic interaction of these two modules improves the model’s accuracy, where the global context module enhances feature discrimination and the superpixel module optimizes spatial details.

Maintaining an optimal balance between accuracy and efficiency, the proposed model offers practical utility for applications including urban mapping and environmental monitoring. Future research will focus on reducing resource consumption, investigating multimodal data fusion, and extending the framework to dynamic remote sensing scenarios. This study advances the frontier of HSR aerial imagery semantic segmentation by providing a robust method that combines global context understanding and local detail refinement.