MultiDecNet: An Ensemble-Based Semantic Segmentation Architecture for Urban Scene Understanding

Abstract

Semantic segmentation is a fundamental task in computer vision that aims to assign a categorical label to each pixel in an image, facilitating dense and detailed scene understanding. This pixel-level classification is especially crucial in autonomous driving, where accurate environmental perception is vital for dependable object detection and safe decision-making. In this study, we propose MultiDecNet, a novel multi-decoder semantic segmentation framework designed to capture both macroscopic scene layouts and fine-grained spatial boundaries in complex urban environments. Drawing inspiration from classical networks, MultiDecNet incorporates a parallel dual-branch decoding strategy that simultaneously leverages the multi-scale context modeling of the Pyramid Pooling Module (PPM) and the structural refinement capabilities of Atrous Spatial Pyramid Pooling (ASPP). To explore the impact of modern backbone representations, we structurally modernize the feature extraction pipeline by introducing the contemporary ConvNeXt convolutional architecture as an alternative to traditional ResNet101 backbones. We extensively evaluate and compare the baseline configurations alongside our proposed MultiDecNet using both ResNet101 and ConvNeXt-Large backbones on the benchmark Cityscapes dataset. The quantitative assessments demonstrate that the MultiDecNet architecture consistently provides highly competitive performance within the scope of this comparative study, with the MultiDecNet-ConvNeXt variant achieving favorable overall scores among the evaluated methods. Furthermore, a granular, class-wise IoU and training dynamics analysis reveals that while traditional networks retain competitive boundaries for localized minority targets, the modern ConvNeXt backbone ensures faster convergence stability and balanced contextual mastery over large-scale driving layouts. Ultimately, these findings offer critical insights into architectural synergy and backbone selection, presenting a robust, scalable, and well-balanced solution for advanced autonomous navigation systems.

1. Introduction

Autonomous driving refers to a vehicle’s ability to navigate independently by perceiving its environment without human intervention. This capability is achieved through the integration of artificial intelligence (AI), a suite of sensors, and other advanced technologies [1]. Within this domain, semantic segmentation plays a critical role in enabling comprehensive environmental perception. Also known as visual scene understanding, semantic segmentation allows autonomous vehicles to interpret their surroundings accurately, thereby supporting informed and safe decision-making [2].

Semantic segmentation involves converting an input image into a pixel-level representation, where each pixel is assigned a label corresponding to a predefined object class. This pixel-wise classification process is fundamental to enabling machines to understand and interact with visual scenes. It is a key technique in computer vision and AI, with widespread applications in autonomous driving, robotic navigation, remote sensing, medical imaging, industrial inspection, satellite image analysis, and precision agriculture [3,4].

In the context of semantic segmentation for autonomous vehicles, object classes such as roads, pedestrians, vehicles, and other surrounding obstacles are identified to facilitate environmental understanding. This process is predominantly carried out using deep learning algorithms, which have recently demonstrated state-of-the-art performance. Accordingly, deep learning models are trained to assign each image pixel to specific categories, such as road, vehicle, pedestrian, and traffic signs, enabling the vehicle to accurately recognize surrounding objects and interpret its environment.

Semantic segmentation for autonomous driving represents a highly complex and challenging task. Below, we outline several key factors that contribute to this complexity:

•

Autonomous driving scenes, which reflect real-world environments, exhibit considerable complexity. In these scenarios, the system typically detects only prominent objects, often classifying numerous smaller or less conspicuous regions as background. Furthermore, the fine-grained structural details of objects are frequently lost or overly smoothed during processing. These challenges can adversely impact the accuracy and reliability of semantic segmentation algorithms.

•

In urban street scenes, certain object categories are prone to misclassification due to visual similarities. The availability of rich contextual information is essential for accurately distinguishing among these classes. Therefore, it is imperative that we develop methods capable of effectively modeling comprehensive contextual dependencies.

•

Variations in viewing conditions, such as nighttime, rainfall, snowfall, or fog, can significantly impact image quality and present substantial challenges for image processing. These changes in illumination and visibility may adversely affect the performance and robustness of semantic segmentation algorithms.

•

In autonomous driving, image capture is generally performed using many sensors, and differences in sensor perspectives can result in alterations in the perceived location and scale of objects in the image. These variations introduce additional complexity to the semantic segmentation process.

•

Real-world applications such as autonomous driving often necessitate real-time processing capabilities. In these systems, any latency or degradation in performance can lead to critical consequences, indicating the importance of highly efficient and reliable semantic segmentation algorithms.

•

Deep learning-based semantic segmentation techniques typically require substantial quantities of annotated data to attain higher levels of accuracy. This requirement leads to extensive and meticulous labeling processes that are both time-consuming and resource-demanding, presenting considerable challenges in data acquisition and model training.

Addressing these challenges necessitates advancements in algorithmic design, hardware capabilities, sensor technologies, and data acquisition strategies. Equally important is the use of large-scale datasets that encompass diverse scenarios and environmental conditions, along with enhancements to data annotation processes. Current research and development initiatives by engineers and scientists seek to enhance the robustness, accuracy, and efficiency of semantic segmentation methods, thereby contributing to the realization of safe and reliable autonomous driving systems.

To effectively address the structural limitations of conventional single-decoder networks, this study introduces a novel multi-decoder semantic segmentation framework, termed MultiDecNet. The core philosophy behind MultiDecNet is to establish an architectural synergy that concurrently exploits the multi-scale contextual aggregation of the Pyramid Pooling Module (PPM) from PSPNet [5] and the dense spatial boundary refinement of the Atrous Spatial Pyramid Pooling (ASPP) module from DeepLabV3+ [6]. By running these two distinct decoding strategies in a parallel dual-branch architecture, MultiDecNet successfully captures both macroscopic scene layouts and fine-grained boundary representations, offering an integrated and robust solution tailored for complex urban environments.

A foundational motivation of this research is to modernize classical segmentation pipelines by exploring the impact of next-generation backbone representations. To achieve this, we structurally replace the traditional ResNet101 architecture [7]—which long served as the standard backbone for classical networks—with the modern ConvNeXt framework [8]. Introduced as a modernized pure convolutional network that synthesizes the structural macro-design principles of Vision Transformers with the inductive biases of CNNs, ConvNeXt offers superior feature extraction capabilities. In our study, we systematically evaluate both ResNet101 and ConvNeXt-Large across the baseline networks and our proposed MultiDecNet, explicitly demonstrating how next-generation backbones influence spatial representation and feature propagation within a multi-decoder layout.

The proposed MultiDecNet framework is comprehensively validated on the Cityscapes benchmark dataset [9], which specializes in dense, real-world urban street scenes. The network’s performance is rigorously quantified using widely accepted evaluation metrics, including mIoU, F1 score, and overall pixel accuracy, ensuring a thorough comparison against standard standalone baselines. Going beyond aggregate scores, this study conducts a highly granular, class-wise IoU analysis to explore how backbone modernization and multi-decoder averaging mitigate the adverse long-tail effects [10] of data imbalance across the 19 evaluation classes. Ultimately, this research contributes to academic literature by providing a scalable, high-performing framework that effectively bridges the gap between traditional architectural paradigms and modern representational backbones for safe autonomous navigation.

2. Related Work

Recently, deep learning techniques have demonstrated state-of-the-art performance in semantic segmentation, resulting in the emergence of various methodologies based on deep neural networks. A seminal contribution to this field is the Fully Convolutional Network (FCN) model, introduced by Long et al. [3], which transformed traditional Convolutional Neural Network (CNN) architectures—initially developed by LeCun and Bengio [11]—into end-to-end, trainable frameworks for segmentation.

A significant drawback of FCN-based methods is that, as feature maps are repeatedly downsampled through convolution and pooling, the output loses detail. As a result, segmentation predictions produced by these models often suffer from low resolution, leading to imprecise and blurred object boundaries.

Several advanced techniques primarily leverage encoder–decoder architectures to address the limitations associated with low-resolution outputs in FCN-based models. Among the most widely adopted are deep convolutional networks such as U-Net [12], SegNet [4], and RefineNet [13]. These architectures aim to recover spatial detail by progressively upsampling feature maps and incorporating contextual information from earlier layers.

Despite these advancements, a significant limitation remains: FCN-based methods generally function with a predefined, fixed-size receptive field. Consequently, objects that are considerably larger or smaller than the designated size for which the method is calibrated are likely to be misidentified or improperly segmented, as there is insufficient surrounding information to accurately interpret them.

To overcome the problems with fixed-size receptive fields in regular convolutional networks, techniques called dilated (or atrous) convolution have been created, like those used in the DeepLab architecture [6,14,15]. These techniques make the receptive field bigger without losing detail, which helps the architecture better understand the context in larger areas of the image.

In addition, multi-scale representation strategies have been employed to enhance the detection and segmentation of objects with varying spatial dimensions. Notable examples include PSPNet [5] and FeatMap-Net [16]. Each of these approaches utilizes distinct mechanisms to effectively capture and integrate information across multiple scales, thereby improving segmentation accuracy for objects of various sizes.

More recently, the paradigm has shifted from traditional CNNs toward Vision Transformer (ViT)-based architectures [17,18,19] to overcome the structural limitations of localized receptive fields. A prominent milestone in this direction is SegFormer [20]. SegFormer integrates a hierarchically structured Transformer encoder with a lightweight All-MLP (Multi-Layer Perceptron) decoder. Unlike conventional Transformers that require positional encodings—which can degrade performance when resolution changes—SegFormer’s positional-encoding-free design renders it highly flexible across different testing resolutions. Furthermore, its hierarchical encoder successfully extracts both high-resolution coarse features and low-resolution fine-grained features without relying on complex and computationally expensive dilated convolutions, establishing a highly efficient and accurate benchmark in modern semantic segmentation.

In addition to purely transformer-based architectures, a recent paradigm shift has introduced universal segmentation frameworks that discard traditional pixel-level classification in favor of mask classification. The pinnacle of this approach is Masked-attention Mask Transformer (Mask2Former) [21]. Mask2Former presents a unified architecture capable of seamlessly addressing semantic, instance, and panoptic segmentation tasks. The core innovation of Mask2Former is its masked attention mechanism, which constrains cross-attention layers to within the predicted mask regions rather than the entire image. This effectively allows the model to extract highly localized and sharp features around object boundaries while significantly reducing computational overhead. By setting new state-of-the-art benchmarks across diverse datasets, Mask2Former demonstrates that a single, mask-attention-guided framework can outperform task-specific, heavily specialized architectures in complex scene parsing.

Aside from pure Transformer and mask-classification models, recent advancements have also revitalized Convolutional Neural Networks (CNNs) by integrating dynamic spatial awareness. A notable example is InternImage [22], which utilizes core blocks operated by deformable convolution (DCNv3). This allows the network to adaptively adjust its receptive field based on the irregular geometric shapes of urban objects, thereby capturing long-range dependencies efficiently without the prohibitive computational costs of standard self-attention mechanisms. Concurrently, the domain has embraced foundational segmentation models such as Meta’s Segment Anything Model (SAM) [23]. SAM leverages a promptable architecture trained on massive datasets, establishing remarkable zero-shot generalization capabilities that can segment arbitrary objects within intricate street-level imagery without task-specific fine-tuning. Collectively, these modern methodologies represent the shifting frontier of semantic segmentation, moving toward adaptive receptive fields and universal generalization.

A major challenge in semantic segmentation is that fully supervised methods require detailed, pixel-level annotations to train the model effectively. Producing such detailed annotations is extremely labor-intensive and time-consuming, which hinders scalability. To address this issue, weakly supervised learning algorithms have been developed. These approaches utilize less detailed forms of supervision, such as bounding boxes in BoxSup [24], image-level labels in Multiple-Instance Learning (MIL) frameworks [25], and scribble annotations in ScribbleSup [26]. The result significantly reduces the annotation burden while still facilitating effective model training.

The primary challenge with weakly supervised methods is achieving accurate segmentation while working with limited and imprecise types of annotations. In this context, Emek Soylu et al. [27] conducted a comprehensive assessment of studies that utilize FCN-based designs alongside weak supervision techniques for semantic segmentation, emphasizing both the advancements and the limitations of these approaches.

While the aforementioned methods have shown considerable success in general semantic segmentation tasks, their performance in complex urban street scenes remains suboptimal. A primary contributing factor is the inherent complexity of these environments, where many objects are subtle, embedded within the background, exhibit overlapping spatial arrangements, and often lack well-defined boundaries. This complexity creates significant challenges for achieving accurate segmentation.

In highly complex visual environments, models primarily optimized for segmenting distinct and isolated objects tend to exhibit reduced performance. For instance, the study by Emek Soylu et al. [27] highlights a marked discrepancy in segmentation accuracy when comparing performance on datasets containing relatively simple and well-defined objects, such as Pascal VOC [28], with that on datasets characterized by intricate urban scenes, such as CamVid [29], Cityscapes [9], KITTI [30] and SIFT Flow [31]. This performance gap underscores the challenges presented by densely populated, visually ambiguous street-level imagery.

The findings emphasize that it is vital to integrate adequate contextual information for effective semantic segmentation in street-level scenes. Consequently, there is an urgent requirement to devise advanced methodologies that can capture and model extensive contextual dependencies, facilitating robust performance in complex and visually dense urban environments.

This paper examines the interpretation of complex urban street scenes in the context of autonomous driving. To achieve this aim, a semantic segmentation application has been developed, enabling the recognition and contextual understanding of various objects frequently encountered in driving environments, such as pedestrians, vehicles, lane markings, traffic signs, and traffic lights. The application employs advanced deep learning techniques to enhance the perception capabilities of autonomous vehicles, thereby facilitating their navigation and decision-making processes.

3. Methods

3.1. ConvNext-Based PSPNet Model

This study is inspired by the PSPNet (Pyramid Scene Parsing Network) architecture, which has shown great results in semantic segmentation tasks by successfully combining wide and detailed context using its pyramid pooling module(Figure 1). Zhao et al. [5] regarded PSPNet as a foundational model that demonstrates high performance in deep learning-based image segmentation. Its success primarily stems from an architectural focus on capturing multi-scale contextual information, which enhances scene understanding by effectively integrating both local and global features. A crucial element of this architecture is the Pyramid Pooling Module (PPM), which aggregates contextual information across four distinct spatial scales. This allows the model to capture a broad environmental context for each pixel, thereby facilitating a more comprehensive interpretation of the scene. The module generates multi-scale scene representations by integrating features extracted from subregions of varying sizes. Coarse levels within the pyramid capture overarching structural patterns across extensive spatial areas, while finer levels preserve detailed, localized information. This layered approach to feature combination enhances the understanding of scenes by blending information from different scales, enabling the model to better grasp both the overall context and details.

PSPNet further integrates Global Average Pooling (GAP) to bolster its ability to capture the global context of a scene. GAP calculates the mean value across all spatial locations within the feature map, resulting in a condensed feature vector that acts as a global representation of the scene. This global feature is then incorporated into the segmentation framework, thereby enhancing the model’s capability to learn extensive contextual relationships throughout the image.

The feature maps taken at different sizes are combined using a method that merges them by channels, creating a single, richer representation. This fusion process combines information from different image sizes, leading to better and more complete segmentation results. Ultimately, PSPNet performs pixel-wise classification by applying a softmax activation function, which models the probability distribution over semantic classes for each pixel. This allows the network to generate detailed and contextually consistent segmentation maps.

Figure 1. Reconstructed PSPNet architecture with ConvNext.

Figure 1. Reconstructed PSPNet architecture with ConvNext.

The model architecture begins with a preprocessing stage, which processes the raw input images prior to model inference. This phase involves essential preparation steps to convert both the raw images and their corresponding segmentation masks into a format compatible with the deep learning framework. In this study, all input images and segmentation masks were rescaled to 512 × 1024 pixels to align with the model’s input requirements. Subsequently, the images and masks were converted into tensor format, enabling efficient processing by the neural network.

The input images in RGB format were first standardized by resizing pixel values to the [0, 1] range, followed by normalization using the widely adopted ImageNet mean and standard deviation values. To preserve categorical class integrity during resizing, the segmentation masks were processed using nearest-neighbor interpolation. These preprocessing procedures were meticulously designed to promote stable model convergence during training and to enhance overall segmentation accuracy.

We use a robust backbone network, such as ResNet or ConvNeXt, for the extraction of deep features. The resulting feature map is then forwarded to PPM. This process aims to gather multi-scale contextual information, thereby enhancing the model’s comprehension of both global and local scene structures. This module captures contextual information from objects and structures of varying scales within the scene by applying average pooling operations with kernel sizes of 1 × 1, 2 × 2, 3 × 3, and 6 × 6. The resultant feature maps are subsequently upsampled to match the original spatial resolution and concatenated, producing a comprehensive feature representation that encapsulates both fine-grained local details and overarching global context. The combined feature representations are improved using several convolutional layers, leading to the creation of a detailed classification map for each pixel. This map is then rescaled to align with the original input image dimensions. Such an architecture enables high-accuracy semantic segmentation, especially in intricate urban environments, by effectively capturing and analyzing scene content across various spatial scales.

3.2. ConvNext-Based DeepLabV3+ Model

Another architecture looked at in this study is DeepLabV3+, selected because it effectively captures information at different scales, keeps detailed object boundaries clear, and reliably provides strong segmentation results. DeepLabV3+ [6] is an advanced extension of the DeepLabV3 architecture [15] specifically designed for semantic image segmentation. It incorporates several significant enhancements that optimize feature extraction and facilitate object edge definition, ultimately resulting in improved segmentation outcomes in complex images.

The DeepLabV3+ architecture uses a system that includes both an encoder and a decoder to get very accurate results in semantic segmentation while keeping important spatial details. The Atrous Spatial Pyramid Pooling (ASPP) module generates multi-scale feature representations during the encoder stage by using atrous convolutions with varying receptive fields. In the decoder stage, deep features from the encoder are integrated with low-level features extracted from earlier layers through skip connections. This approach enhances the reconstruction of fine object boundaries and spatial details. The combined features undergo further refinement using multiple 3 × 3 convolutional layers to enrich detail and ensure spatial consistency. Finally, the output is upsampled using bilinear interpolation to align with the input image resolution, resulting in high-resolution segmentation outputs.

In the initial phase of the model, the input image is processed through a robust backbone network, such as ResNet or ConvNeXt, which extracts essential features (Figure 2). This backbone produces feature maps that are rich in high-level contextual information. Concurrently, features from earlier layers are managed by a separate pathway to preserve crucial details, such as edges and textures. This dual-stream approach allows the model to retain both semantic richness and spatial precision, both of which are vital for achieving accurate segmentation.

Figure 2. DeepLabV3+ architecture restructured with ConvNext.

Figure 2. DeepLabV3+ architecture restructured with ConvNext.

The ASPP module employs parallel atrous convolutions with diverse dilation rates, enabling the effective extraction of both localized features and broader contextual information. This design enables the model to effectively represent both large-scale scene structures and finer object details. Subsequently, the output of the ASPP module undergoes dimensionality reduction via a 1 × 1 convolution to reduce the number of feature channels. The resulting feature map is then upsampled four times to match the spatial resolution of the low-level features, facilitating their effective integration.

Another notable component of the model is the dedicated processing of low-level features extracted from the early layers of the backbone. These features are first passed through a separate branch and reduced to 48 channels via a 1 × 1 convolution to align their dimensionality with the subsequent processing stages. Afterward, they are fused with the high-level contextual representations obtained from the ASPP module. This fusion is followed by a series of 3 × 3 convolutional layers, enabling the model to learn rich, semantically enhanced feature representations that maintain a balance between fine spatial detail and global context.

Subsequently, a 1 × 1 convolutional layer applies a number of output channels equal to the number of target classes to perform pixel-wise classification. Finally, the resulting feature map is upsampled to match the resolution of the input image, producing a dense segmentation map with per-pixel class predictions.

The multi-layered and multi-scale design of the DeepLabV3+ architecture enables highly accurate semantic segmentation, particularly in complex urban environments. The ASPP module enhances contextual understanding by capturing features at multiple spatial scales, while the incorporation of low-level features from early network layers helps preserve fine structural details, especially along object boundaries. This combination contributes to the production of high-quality segmentation outputs. As a result, DeepLabV3+ presents a robust and effective solution for precision-critical applications, such as autonomous driving and intelligent urban infrastructure systems.

3.3. Developed MultiDecNet Model

The initial stage of the proposed architecture involves the processing of the input image. A three-channel RGB image with dimensions of 3 × 512 × 1024 is first passed through a backbone network to extract high-level feature representations. In this study, we use two pre-trained convolutional neural networks—ResNet101 and ConvNeXt—as the main structures for extracting deep features from the input imagery. Following the feature extraction stage, the resulting feature maps are directly routed to the decoding pipeline to preserve the raw hierarchical representations captured by the backbone networks.

The extracted feature representations are concurrently fed into two distinct decoder branches, each employing a different architectural strategy to refine segmentation outputs. The left branch utilizes the PSPNet decoder, which incorporates a PPM to capture multi-scale contextual information by aggregating features from varying spatial extents. This enables a more comprehensive understanding of the scene by emphasizing broader contextual cues. In parallel, the right branch employs the DeepLabV3+ decoder, which leverages an ASPP module. ASPP uses dilated convolutions at different rates to capture detailed spatial information and structural semantic cues at various levels, which helps to better define the boundaries and edges between different classes. Both decoder pathways process the backbone features and generate intermediate segmentation maps independently.

An ensemble strategy was employed to integrate the outputs generated by the two parallel decoder branches. Ensemble methods [32] are well-established techniques designed to enhance predictive performance, robustness, and generalization by aggregating the outputs of multiple networks. This approach capitalizes on the complementary strengths of individual decoding strategies, allowing the ensemble to mitigate the structural limitations of any single decoder by emphasizing the regions at which each model excels. Consequently, the ensemble mechanism contributes to a more accurate and stable final segmentation outcome.

In this study, the results from the PSPNet and DeepLabV3+ decoders are combined using a method called ensemble averaging, where the probability scores for each class from both decoders are averaged to create the final segmentation map. This method represents a straightforward yet effective application of ensemble learning, leveraging the global contextual strengths of PSPNet and the structural boundary refinement capabilities of DeepLabV3+. By combining these complementary characteristics, the ensemble method yields a more robust, stable, and balanced segmentation prediction.

The proposed parallel architecture (Figure 3) allows for a precise delineation of class boundaries while effectively capturing the context of large-scale scenes. This is accomplished through the strategic co-utilization of multiple decoder frameworks that operate simultaneously on the backbone features. By combining these distinct decoding elements, the model significantly improves its overall predictive capability across intricate urban environments. In this regard, the architecture exemplifies a flexible and high-performing framework that synthesizes advanced strategies from contemporary deep learning-based segmentation methods, offering a streamlined and scalable solution for complex semantic segmentation tasks.

Figure 3. Developed “MultiDecNet” structure.

Figure 3. Developed “MultiDecNet” structure.

Loss Function. In semantic segmentation tasks, the choice of an appropriate loss function is critical to guiding the model toward accurate pixel-wise classification. The loss function measures the divergence between the predicted segmentation maps and the corresponding ground truth annotations, serving as the primary objective for optimization. To minimize this divergence, the loss function effectively guides the model’s training process toward enhanced predictive accuracy [12].

This work utilized the ComboLoss function, an integrated formulation that combines Dice Loss [33] and Focal Loss [34], to concurrently tackle the issues of class imbalance and difficult-to-classify instances. This composite loss function leverages the strengths of both components: Dice loss enhances overlap-based accuracy, particularly for imbalanced class distributions, while focal loss emphasizes the learning of difficult samples by downweighting the contributions of well-classified instances. The combined approach facilitates more robust and balanced model optimizations in complex segmentation scenarios.

Dice loss is a similarity-driven loss function specifically designed to mitigate the effects of class imbalance in segmentation tasks. It achieves this goal by equally weighting the contribution of each class, thereby promoting accurate prediction across all classes, including those with relatively fewer instances. This approach stops the model from ignoring smaller or less common classes while it learns, which improves how well it segments all classes. We derive it from the Dice Coefficient. Its formula is as follows:

Dice coefficient.

$$Dicc(p,g)={\displaystyle \frac{2{\sum }_i{p}_i{g}_i}{{\sum }_i{p}_i+{\sum }_i{g}_i}}$$

(1)

where

•

p_i: the predicted value (probability),

•

g_i: the ground truth label (0 or 1),

•

${\sum }_i{p}_i{g}_i:$ the intersection between the predicted value and ground truth label,

•

${\sum }_i{p}_i+{\sum }_i{g}_i:$ the sum representing the union.

Dice loss.

$$Dicc Loss=1-Dicc(p,g)$$

(2)

Focal loss was introduced to enhance the model’s focus on hard-to-classify examples during training by modulating the contribution of individual samples to the overall loss. It achieves this goal by down-weighting the loss assigned to well-classified or easily predicted instances while amplifying the influence of misclassified or ambiguous samples. This targeted focus helps the model learn better features, especially when there are uneven class distributions or difficult data patterns. Its formula is as follows:

$$FL(p_t={-a}_t{(1-p_t)}^{\gamma }\log p_t$$

(3)

where

$p_t:$ the model’s predicted probability for the true class:

$$p_t=\left\{\begin{array}{c} p, if the true label is 1\\ 1-p, if the true label is 0\end{array}\right.$$

(4)

$a_t:$ the weighting factor to balance class importance (optional),

γ: it modulates the rate at which straightforward examples are down-weighted.

The ComboLoss function utilized in this work aims to integrate the complementary advantages of Dice Loss and Focal Loss. This approach concurrently tackles the issue of class imbalance and improves the model’s sensitivity to difficult-to-classify instances during training. This dual-objective approach facilitates more equitable learning across class distributions while enhancing the model’s discriminative ability in intricate segmentation contexts.

Data Augmentation. Deep learning tasks use data augmentation techniques to artificially enhance the variety of the training dataset. This method improves the model’s generalization capabilities and diminishes the risk of overfitting by presenting the network with a wider array of input variables [35]. We generate new training samples by implementing a sequence of geometric and photometric transformations, such as rotation, scaling, flipping, and color modifications, on the original images. These enriched instances enhance the model’s tolerance to varied visual circumstances and elevate performance on unfamiliar data.

During the data augmentation process, both the input images and their corresponding segmentation masks were subjected to identical transformations. This synchronized augmentation ensures the preservation of pixel-wise label alignment, thereby enabling the model to learn accurate spatial correspondences between the image content and its annotated regions. Such consistency is critical for maintaining semantic integrity in segmentation tasks.

4. Results and Discussion

4.1. Hardware and Software Infrastructure

The training and evaluation of the deep learning models in this study were conducted using two distinct computer systems with varying hardware configurations. The system utilized during the development phase was equipped with an Intel Core i7-11700H (Intel Corporation, Santa Clara, CA, USA) processor, 32 GB of RAM, and an NVIDIA RTX 3070 GPU (NVIDIA Corporation, Santa Clara, CA, USA) featuring 8 GB of VRAM. This workstation was primarily employed for architectural design, small-scale model training, hyperparameter tuning, and preliminary testing.

The large-scale training procedures were carried out on a high-performance computing system specifically configured for intensive deep learning tasks. This training machine was equipped with 12 virtual CPUs (vCPUs), 62 GB of RAM, and an NVIDIA A40 GPU with 48 GB of VRAM. The powerful hardware of this system allowed for quick training of deep neural networks on big datasets and helped significantly shorten the training time during the model improvement process.

The core software components and library versions utilized throughout this study are the following:

• Python programming language, version 3.10
• PyTorch deep learning framework, version 2.4.0
• CUDA parallel computing platform, version 12.4.1

This combination of hardware and software facilitated reduced training times, enhanced computational efficiency, and contributed positively to the overall accuracy and reliability of the model outcomes.

4.2. Experimental Setup

To address the severe VRAM [36] and computational constraints associated with training dense semantic segmentation models on consumer-grade GPU infrastructure [37], a controlled experimental protocol was enforced. All evaluated frameworks—including the baseline architectures and the proposed MultiDecNet variations—were trained entirely from scratch for a fixed duration of 50 epochs. This uniform budget ensures a standardized benchmark regarding structural convergence speed and representational capacity under constrained resource regimes. The exhaustive set of hyperparameter configurations utilized to guarantee full reproducibility is detailed in Table 1.

Table 1. Detailed hyperparameter configuration for the compared architectures.

Hyperparameter	PSPNet- ResNet101	PSPNet- ConvNeXt	DeepLabV3+-ResNet101	DeepLabV3+-ConvNeXt	MultiDecNet-ResNet101	MultiDecNet-ConvNeXt
Backbone	ResNet-101	ConvNeXt-Large	ResNet-101	ConvNeXt-Large	ResNet-101	ConvNeXt-Large
Backbone Weights	ImageNet-1K	ImageNet-1K	ImageNet-1K	ImageNet-1K	ImageNet-1K	ImageNet-1K
Num. Classes	19	19	19	19	19	19
Input Size	512 × 1024	512 × 1024	128 × 128	128 × 128	512 × 1024	512 × 512
Batch Size	24	24	24	24	24	24
Epochs	50	50	50	50	50	50
Optimizer	Adam	Adam	Adam	Adam	AdamW	AdamW
Learning Rate	1 × 10−4	1 × 10−4	1 × 10−4	1 × 10−4	1 × 10−4	1 × 10−4
LR Scheduler	—	—	—	—	Poly (p = 0.9)	Poly (p = 0.9)
Loss Function	CE + Lovász (50/50)	CE + Lovász (50/50)	Cross-Entropy	Cross-Entropy	Dice × 2 + Focal × 1	Dice × 2 + Focal × 1

4.3. Cityscapes Dataset

Datasets serve a pivotal role in evaluating and benchmarking the performance of methodologies developed in deep learning and computer vision. They offer a standardized framework for assessing a model’s generalization capabilities and its ability to learn under diverse and complex conditions. In this context, the Cityscapes dataset [9] has been selected due to its extensive and detailed representation of urban environments, making it particularly well-suited for modeling the dynamic variations and structural challenges inherent in autonomous driving scenarios.

The Cityscapes dataset comprises 5000 high-resolution images (2048 × 1024) extracted from video sequences captured by vehicles traversing 50 distinct cities across Germany. These images were collected during daytime under favorable to moderate weather conditions, spanning the spring, summer, and autumn seasons. Following the official benchmark configuration, the dataset is strictly partitioned into 2975 images for training, 500 images for validation, and 1525 images for testing. The dataset originally encompasses 30 semantic classes (refer to Table 2), with pixel-level annotations that assign a precise class label to each pixel. The annotation process utilizes distinct color coding for each semantic category, ensuring that traffic participants—such as individual vehicles and pedestrians—are annotated separately. This approach provides fine-grained class-level information and captures detailed spatial localization of each object within the scene.

Table 2. Groups and classes in the Cityscapes dataset.

Group	Classes
construction	bridge, building, fence, guard rail, tunnel, wall
flat	parking, rail track, road, sidewalk
human	person, rider
nature	terrain, vegetation
object	pole, pole group, traffic light, traffic sign
sky	Sky
vehicle	bicycle, bus, car, caravan, motorcycle, on rails, trailer, truck
void	dynamic, ground, static

To ensure a standardized evaluation, we strictly adhere to the standard benchmarking protocol by utilizing only the 19 core classes designated for training and quantitative evaluation. The remaining 11 categories (such as out-of-boundary regions, rare artifacts, or void pixels) are mapped to an ignore index and systematically excluded from loss optimization. Consequently, the final pixel-wise classification layer of our proposed MultiDecNet is explicitly configured to output 19 channels corresponding to these standard target classes, ensuring a fair and mathematically sound comparison with existing architectures in the literature.

The Cityscapes dataset is divided into 2975 training images, 500 validation images, and 1525 test images. This structured distribution facilitates a balanced and systematic approach to model development, enabling effective training, hyperparameter optimization, and comprehensive performance evaluation. A few representative samples of labeled images from the Cityscapes dataset are illustrated in Figure 4.

Figure 4. An example set of labeled images from the Cityscapes dataset.

Figure 4. An example set of labeled images from the Cityscapes dataset.

4.4. Evaluation Metrics

In this paper, the semantic segmentation performance was quantitatively evaluated using mIoU, F1 score, and accuracy metrics. Since semantic segmentation involves multiple classes, the evaluation looked at both how well the model performed overall and how it did for each individual class.

mIoU. We compute the Mean Intersection over Union (mIoU) by averaging the Intersection over Union (IoU) values for all classes. This metric assesses the degree of correspondence between the predicted segmentation of the model and the actual ground truth labels for each specific class.

$$IoU={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(5)

• True Positive (TP): The quantity of pixels accurately identified as pertaining to a particular class.
• False Positive (FP): The quantity of pixels erroneously classified as belonging to a category when they do not.
• False Negative (FN): The quantity of pixels that authentically belong to a class but were inaccurately classified by the model.

mIoU assigns equal importance to each class, thereby providing a more balanced and equitable evaluation, particularly in scenarios involving imbalanced class distributions.

F1-score. The F1 score is a harmonic mean of two essential evaluation metrics:

• Precision: This measures the proportion of the model’s positive predictions that are genuinely correct.
• Recall: This indicates the proportion of actual positive instances that the model has accurately identified.

$$\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(6)

$$\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(7)

$$F1=2\times \frac{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}$$

(8)

Precision and recall often exhibit a trade-off, where improvements in one may lead to a decline in the other. The F1 score offers a single, balanced measure of general model efficiency by harmonizing these two metrics.

Accuracy. Accuracy is defined as the ratio of correctly classified examples to the total number of instances. While it gives a quick overview of how well the model is doing, accuracy alone is not enough, especially when some classes have many more examples than others, because it does not take into account how the class labels are spread out.

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(9)

4.5. Quantitative Performance Evaluation

To assess the effectiveness of the proposed models in semantic segmentation tasks, a comprehensive quantitative evaluation was conducted. Standard performance metrics, including mIoU, accuracy, and F1 score, were employed to ensure a robust comparison across different model-backbone configurations. These metrics provide a holistic view of model performance, capturing both pixel-level accuracy and class-wise segmentation quality.

Figure 5 and Table 3 present a comparative analysis of the segmentation networks evaluated using these core metrics. Overall, the proposed MultiDecNet architecture demonstrates a consistently favorable performance alignment compared to the baseline PSPNet and DeepLabV3+ frameworks. Within our multi-decoder framework, the modern ConvNeXt backbone achieves the top overall performance, yielding a competitive mIoU of 0.62, a pixel accuracy of 0.91, and an F1 score of 0.68. While this represents a marginal and closely matched numerical edge over the traditional ResNet101 variant—which maintains a remarkably robust performance with a 0.60 mIoU—the findings successfully validate our initial objective of integrating next-generation macro-designs into dense scene parsing pipelines.

Figure 5. Performance analysis of models.

Table 3. Comparison of performance metrics of models.

Model	MIoU	Accuracy	F1
PSPNet-ResNet101	0.52	0.85	0.64
PSPNet-ConvNext	0.54	0.88	0.66
DeepLabV3+-ResNet101	0.53	0.84	0.66
DeepLabV3+-ConvNext	0.54	0.86	0.66
MultiDecNet-ResNet101	0.60	0.87	0.67
MultiDecNet-ConvNext	0.62	0.91	0.68

Figure 6 presents bar chart visualizations of the class-wise IoU values to provide a granular performance breakdown of the proposed MultiDecNet framework across the 19 evaluation categories. In these charts, the horizontal axis corresponds to the semantic segmentation classes (e.g., road, sidewalk, building, car, etc.), while the vertical axis depicts the IoU scores for each class, ranging from 0 to 1. This visualization enables a balanced assessment of how backbone representations influence segmentation accuracy when dealing with severe class imbalances and diverse object geometries.

Figure 6. Class-based IoU values of MultiDecNet models.

The quantitative breakdown reveals highly competitive, closely matched, yet complementary behaviors between the two backbone configurations, with the modern ConvNeXt network ultimately establishing a more robust global foundation. As illustrated in Figure 6, both models successfully maintain strong representational authority over high-frequency background classes such as “vegetation” (0.81 for both) and “sky” (0.81 vs. 0.84). However, the MultiDecNet-ConvNeXt configuration demonstrates a definitive and distinct advantage in major structural, navigationally critical, and spatial categories. Notably, it yields substantially higher IoU scores in key layout classes, achieving an IoU of 0.96 for “road” (compared to 0.90 for ResNet101), 0.74 for “sidewalk” (compared to 0.64), 0.84 for “building” (compared to 0.80), and 0.52 for “wall” (compared to 0.42). This clear superiority indicates that ConvNeXt’s modernized macro-kernels and larger effective receptive fields are exceptionally effective at capturing continuous global layouts, structural coherence, and wide spatial contexts, which represent the vast majority of pixels in urban driving environments.

Conversely, the MultiDecNet-ResNet101 configuration demonstrates a localized resilience, showing a slight numeric edge in delineating smaller, highly isolated, and low-frequency categories. For instance, it marginally outperforms the ConvNeXt variant in dense minority classes, achieving an IoU of 0.25 for “pole” (vs. 0.18), 0.43 for “traffic light” (vs. 0.38), 0.48 for “traffic sign” (vs. 0.44), and 0.44 for “rider” (vs. 0.37).

In conclusion, the granular analysis confirms that while traditional networks like ResNet101 provide a stable baseline for fine-grained minority classes, the ConvNeXt backbone delivers a more powerful, comprehensive, and scalable representational capacity. By excelling in dominant layout classes and providing a more integrated understanding of large-scale scene geometry, MultiDecNet-ConvNeXt proves to be the superior and more progressive architectural choice for complex urban scene parsing, successfully driving the overall framework to its semantic accuracy.

4.6. Qualitative Performance Evaluation

This section provides a detailed look at the segmentation results from the PSPNet, DeepLabV3+, and the new MultiDecNet models, which were all tested on the same complex city scene. Each row in Figure 7 corresponds to a different model and displays four columns: the original input image, the ground truth segmentation, the model’s prediction, and an image overlay. The input image is the original RGB image that serves as input to each segmentation model. The evaluated scene represents a complex urban environment, containing pedestrians and various other city-related objects. Ground truth is a pixel-level semantic annotation map manually labeled by human annotators. It represents the target output that the models aim to learn during training and serves as the reference standard for evaluating segmentation accuracy. Prediction represents the pixel-wise class assignments generated by each model in response to the input image. These outputs directly reflect the model’s semantic understanding and its ability to accurately discriminate between different classes within the scene. The overlay image superimposes the model’s predicted segmentation map onto the original input image, providing a more intuitive visualization of segmentation performance. This representation is particularly useful for identifying errors in complex regions, such as object boundaries and densely clustered areas, where segmentation accuracy is often challenged.

Figure 7. Qualitative performance comparison of ConvNext-based models.

From a qualitative standpoint, all three models exhibit satisfactory performance in recognizing large and commonly represented classes, such as roads, sidewalks, and buildings. Nevertheless, distinct differences become apparent in the precision and coherence of object boundaries, especially for smaller and more complex instances, such as pedestrians.

The PSPNet model demonstrates noticeable segmentation errors in boundary regions and struggles to delineate individual pedestrians accurately, often merging them with the background or nearby objects. Its prediction map exhibits significant noise and lacks clear object separation, suggesting a limited capacity for spatial discrimination.

DeepLabV3+ shows improved delineation of urban elements compared to PSPNet, particularly in pedestrian areas. The structure of human figures is more distinct, and the segmentation boundaries appear more aligned with the ground truth. Nevertheless, minor inaccuracies remain, especially around occluded or densely clustered objects.

The MultiDecNet architecture demonstrates the highest visual fidelity among the three models. Its predictions exhibit superior boundary accuracy and class consistency, particularly in complex regions involving overlapping pedestrians and urban features. The overlay results further confirm the model’s enhanced capability in preserving structural details and reducing segmentation artifacts. The model’s robustness is particularly evident in the accurate segmentation of multiple pedestrians, with clearer contours and less confusion with adjacent classes.

The qualitative results reinforce the quantitative findings by highlighting the higher performance of the proposed MultiDecNet model in handling complex urban scenes. Its multi-decoder architecture contributes to improved boundary localization and better handling of densely packed or occluded objects, making it especially effective in real-world semantic segmentation tasks.

4.7. Epoch-Based Evaluation

To better understand the training behavior and convergence characteristics of the proposed architecture, we monitored the evolution of the mIoU metric throughout the training process. This analysis allows for a deeper evaluation of model stability, learning dynamics, and backbone influence on optimization efficiency.

Figure 8 presents the training progression of the MultiDecNet architecture, comparing the epoch-wise mIoU trajectories of the ConvNeXt and ResNet101 backbone configurations. Both setups exhibit a highly efficient and rapid early-stage optimization phase, achieving sharp performance gains within the first ten epochs. Throughout the training lifecycle, the two configurations demonstrate highly competitive and closely aligned learning paths. The MultiDecNet-ConvNeXt variant establishes a slightly higher initial baseline and maintains a tight, incremental edge over the ResNet101 variant across the training timeline, ultimately converging gracefully to a closely matched final plateau (0.62 mIoU vs. 0.60 mIoU).

Figure 8. Variation in the mIoU of the MultiDecNet model across training epochs.

In terms of optimization stability, minor differences in learning dynamics can be observed. The ResNet101 configuration displays slight performance oscillations during the intermediate epochs (between epochs 15 and 30) before successfully consolidating its parameters. In contrast, the ConvNeXt configuration yields a marginally smoother and steadier learning curve, indicating that its modern architectural attributes—such as layer normalization and optimized patch embeddings—contribute to a slightly more predictable gradient flow. In conclusion, the tight convergence of both curves highlights the structural robustness of the MultiDecNet framework, demonstrating that it can successfully leverage both traditional and next-generation backbones to achieve stable, higher semantic representation with minimal performance variance.

Figure 9 presents a qualitative assessment of MultiDecNet’s semantic segmentation performance, illustrating the progressive visual evolution of its predictions across three key milestones of the training process: at Epoch 1, Epoch 20, and Epoch 50. To facilitate a direct, pixel-level visual comparison, the layout organizes the training progression vertically by epoch, while displaying four critical modalities horizontally across columns: the original input scene (Input), the manual target annotation (Ground truth), the network’s final output mask (Prediction), and a transparency-blended semantic visualization (Overlay).

Figure 9. Qualitative progression of semantic segmentation results across training epochs.

In the first epoch, the model’s predictions are highly coarse and fail to capture fine-grained object boundaries or precise class distinctions. Large homogeneous regions and noticeable misclassifications are evident across the scene, reflecting the framework’s limited representational learning during the early training phase.

By the 20th epoch, substantial visual improvements are observed as the optimization process advances. The segmentation outputs demonstrate a significantly better understanding of the global scene structure, yielding clearer object boundaries and more accurate class assignments, particularly in complex and structurally demanding regions such as poles, vegetation, and traffic signs.

At the final 50th epoch, the predictions demonstrate a strong visual alignment and high fidelity with the ground truth. Fine structures and small foreground objects are delineated with remarkable accuracy, and the structural overlay reveals a precise spatial correspondence between the predicted and actual semantic regions. This progressive refinement visually validates the steady feature convergence, boundary resolution capabilities, and robust spatial-semantic learning efficiency inherent to our proposed multi-decoder topology over time.

5. Conclusions

In this paper, we propose MultiDecNet, a novel multi-decoder semantic segmentation framework engineered to achieve robust, pixel-level classification in complex urban driving imagery. Moving away from standard single-decoder limitations, the architecture leverages an integrated dual-branch inference strategy that unifies the complementary structural strengths of the PPM and ASPP. By dynamically aggregating the multi-scale contextual feature maps extracted from parallel optimization pathways, MultiDecNet successfully combines a broad macro-scale understanding of wide driving environments with the detailed spatial boundary delineation required for safe autonomous navigation.

Furthermore, we systematically investigated and compared the influence of backbone architectures on modern feature representation by employing ConvNeXt and ResNet101 variants. Both baseline frameworks were re-implemented alongside our proposed MultiDecNet to evaluate their structural synergy and training dynamics under uniform hyperparameters on the benchmark Cityscapes dataset. Comprehensive quantitative and qualitative evaluations—encompassing overall metric performance, training convergence stability, and granular class-wise IoU behaviors—were conducted to thoroughly map the operational boundaries of each setup.

The experimental findings demonstrate that the proposed MultiDecNet topology yields highly competitive and robust performance within the scope of this comparative study, with the MultiDecNet-ConvNeXt configuration achieving favorable overall performance and a smoother, more predictable convergence trajectory compared to its ResNet101 counterpart. On the other hand, the granular analysis highlights that the rigid inductive biases of traditional networks offer localized resilience for highly isolated minority classes. By gracefully balancing these complementary properties through parallel decoding branches, MultiDecNet successfully mitigates the adverse effects of long-tail data distributions and error propagation without losing structural layout coherence. Ultimately, this research provides meaningful architectural insights for autonomous perception, validating that the integration of next-generation backbones with multi-decoder topologies offers a promising and scalable alternative for progressive scene parsing.

Recommendations

This study has developed and evaluated deep learning-based models that are capable of achieving high-accuracy semantic segmentation in urban environments. However, there is considerable potential for future improvements aimed at enhancing model generalization and optimizing performance in real-world deployment scenarios.

Firstly, incorporating datasets that encompass a range of environmental conditions—such as varying weather conditions (e.g., snow, rain) and lighting variations (e.g., day and night)—can significantly enhance the robustness of the models. This diversity is particularly critical for improving model reliability in applications highly sensitive to environmental dynamics, such as autonomous driving systems.

Low-latency inference is crucial for the deployment of semantic segmentation models in real-time systems. Consequently, optimizing model architectures for computational speed and efficiency emerges as a key area of research, particularly for use on embedded platforms or in applications constrained by limited hardware resources.

Moreover, due to the significant costs and labor demands associated with data annotation, the advancement of weakly supervised, semi-supervised, and unsupervised learning approaches presents a promising direction. These methodologies facilitate effective model training on large-scale datasets while requiring only a minimal amount of labeled data, thereby providing a scalable and efficient alternative to extensive manual annotations.

In the end, using different learning methods together—by combining various types of data like depth maps, LiDAR data, and RGB images—can greatly improve how accurately we can separate different objects. This combination of different types of data can greatly improve how well we can tell apart classes and understand the context, especially in complicated and changing city environments.