PMS-Net: A Real-Time Instance Segmentation Framework with Large Receptive Fields for Urban Driving Scenes

Abstract

With the rapid development of edge AI chips and autonomous driving algorithms, autonomous driving perception systems are evolving toward higher accuracy, lower latency, and lightweight deployment. This trend places greater demands on real-time instance segmentation algorithms in terms of multi-scale feature representation, spatial detail modeling, and edge deployment efficiency. To address these challenges, this paper proposes PMS-Net (Progressive Multi-Scale Network), a network designed for real-time instance segmentation. PMS-Net adopts a progressive multi-scale feature modeling mechanism that progressively enlarges the receptive field while integrating semantic and fine-grained spatial information across different scales. This enables efficient collaboration between local features and global contextual information, thereby enhancing scale-awareness and feature representation while maintaining a lightweight architecture. In addition, efficient feature encoding and dynamic feature reconstruction are incorporated to further improve spatial alignment, boundary recovery, and semantic continuity for complex scene modeling. Experimental results show that PMS-Net achieves 36.7% Mask mAP50 and 175 FPS on the Cityscapes dataset, outperforming the baseline by 2.9%. Deployment experiments on the NVIDIA Jetson Orin NX platform further demonstrate that PMS-Net achieves 35.2% Mask mAP50 and 98 FPS, improving the baseline by 3.7% and 14.0%, respectively. These results validate the effectiveness and practicality of PMS-Net for real-time edge-deployed autonomous driving applications.

1. Introduction

With the rapid evolution of artificial intelligence (AI), computer vision (CV)—as one of its core application fields—has undergone leapfrog progress, advancing from basic object recognition to high-level scene perception. The maturity of scene perception and understanding technologies can be attributed to breakthroughs and extensions in object detection methods. These research outcomes have been widely adopted in a variety of real-world applications, such as autonomous driving, humanoid robotics, industrial inspection, and automated systems [1].

Autonomous driving refers to the capability of a vehicle to drive partly or entirely without human intervention [2]. To address inconsistencies and terminology confusion in the autonomous driving sector, the Society of Automotive Engineers (SAE) introduced a standardized vocabulary. This framework was formalized in the 2014 publication SAE-J3016, which clearly outlines six levels of driving automation, ranging from Level 0 to Level 5 [3]. Autonomous driving, a quintessential application of advanced computer vision, relies fundamentally on scene perception and instance segmentation. Scene perception constitutes the core prerequisite for autonomous decision-making in self-driving systems, while instance segmentation provides the essential basis for improving perceptual accuracy. Together, these components form the foundation of the autonomous driving perception module. Relevant studies have extensively demonstrated the application value of instance segmentation and scene perception technologies in autonomous driving. Mask R-CNN unified object detection and instance segmentation within a single framework [4], thereby inspiring the development of subsequent algorithms. The SOLO series [5,6] reframed instance segmentation as a location-aware classification task, eliminating dependency on anchor boxes and enabling direct prediction of instance masks. Deep Snake, a real-time contour-based learning method, achieved robust performance across multiple datasets such as Cityscapes, KINS, SBD, and COCO [7]. In 2025, a research team from KAIST introduced a semi-supervised instance segmentation framework that effectively leverages the fine-grained localization capability of the Segment Anything Model (SAM) while circumventing its category-agnostic limitation. This method significantly improved segmentation performance with limited labeled data, attaining state-of-the-art results on the COCO (using 2% labeled data) and Cityscapes (using 10% labeled data) benchmarks [8].

With the increasing application of scene perception and instance segmentation technologies in autonomous driving, and as the industry progresses toward Level 4 and Level 5 automation, perception algorithms are required to achieve higher segmentation accuracy and faster inference speeds. These improvements are essential to meet real-time processing demands in complex dynamic environments. Moreover, the algorithms must be optimized for efficient on-vehicle deployment to support local inference. And with rapid advancements in edge AI chips, mainstream automotive edge computing chips now provide strong computational support and are progressively evolving to meet the computational requirements of Level 4 autonomous driving. The growing demand for high-precision perception algorithms, together with the rapid advancement of edge computing hardware, has promoted the coordinated iteration of algorithm optimization and hardware upgrading. This motivates the exploration of high-performance instance segmentation algorithms and their efficient deployment optimization for vehicle-end scenarios. In response to this need, we propose PMS-Net and an overview of the method is shown in Figure 1. The principal contributions of this paper are as follows:

Figure 1. The overall framework of PMS-Net.

• This study introduces a progressive multi-scale feature modeling mechanism, which combines cross-layer feature retention with progressive multi-scale fusion. It enables effective collaborative modeling of semantic information and fine-grained spatial details across different scales. While maintaining computational efficiency, the proposed mechanism significantly enhances the model’s ability to capture and represent multi-scale objects.
• We design a feature pyramid architecture that incorporates DySample for dynamic upsampling and GCConv for re-parameterized downsampling. This strategy improves feature alignment accuracy and strengthens feature extraction capability in the downsampling branch, effectively alleviating spatial misalignment during multi-scale information interaction, thereby improving the model’s multi-scale modeling capability and representation stability in complex scenarios.
• We applied the proposed PMS-Net to the instance segmentation task in complex urban autonomous driving scenarios. Experimental results on the widely used Cityscapes dataset show that, compared to existing state-of-the-art methods (CNN-based models and the latest YOLO series models) this model demonstrates significant advantages in both detection accuracy and speed. Furthermore, PMS-Net demonstrates excellent deployability on the NVIDIA Jetson Orin NX edge computing platform, achieving an inference rate of 98 FPS and a Mask mAP50 of 35.2%; compared to the baseline model, it achieves a 3.7% improvement in segmentation accuracy and a 14.0% increase in inference speed.

2. Related Work

2.1. Real-Time Instance Segmentation

Instance segmentation, a core task in computer vision, aims to integrate object detection with pixel-level segmentation. By combining the strengths of object localization and fine-grained pixel-wise recognition, it allows for precise delineation of individual objects in an image. This method is widely used in applications such as autonomous driving and intelligent surveillance systems [9]. As instance segmentation technology becomes widely deployed in real-time applications—such as embedded systems and autonomous driving—there is a growing demand from both the market and application fields for algorithmic models that achieve high accuracy while maintaining real-time performance. Consequently, real-time instance segmentation has become a key research focus in computer vision, with its core challenge lying in the effective balance and optimization of speed and precision [10].

Building on advances in object detection, instance segmentation methods have traditionally followed two-stage and single-stage frameworks, with recent research increasingly focusing on single-stage approaches for real-time performance. In 2016, Redmon et al. proposed the first YOLO algorithm [11], which reformulated object detection as a regression problem and enabled end-to-end real-time detection. Later studies extended this framework by adding segmentation branches. In 2019, YOLACT [12] achieved a breakthrough in real-time instance segmentation by efficiently generating instance masks within a single-stage pipeline. With continuous development of the YOLO series, real-time instance segmentation methods have been further improved. YOLACT++ [13] enhances prototype mask generation and post-processing, achieving a mAP of 34.1 at 33.5 FPS on MS COCO. INSTA-YOLO [14] introduces a one-stage end-to-end model that reaches twice the inference speed of contemporary SOTA methods on a GTX-1080 GPU. More recently, GHA-Inst [1] adopts a dual-branch architecture and shows competitive performance on MS COCO 2017 and KINS datasets. PSC-YOLO [15] optimizes the Segment Anything Model concept and demonstrates strong results on urban road datasets. YOLO-LF [16] integrates lightweight convolutions and a Re-parameterized Efficient Layer Aggregation Network to improve feature extraction while reducing parameters and computational cost, achieving fast and accurate detection on the Cityscapes dataset.

Existing real-time instance segmentation methods have evolved from early-stage functional exploration toward achieving a balance between segmentation accuracy and inference efficiency, and further toward lightweight deployment and multi-scenario generalization, gradually forming a relatively comprehensive technical framework. Nevertheless, there remains considerable room for improvement in simultaneously maintaining high segmentation accuracy under complex conditions and satisfying the efficiency requirements of lightweight implementations.

2.2. Lightweight Convolutional Neural Networks

Convolutional neural networks (CNNs) have achieved state-of-the-art performance across a wide range of computer vision tasks. However, as network architectures grow increasingly deeper and wider, both model parameter count and computational complexity escalate exponentially. While such large-scale CNNs deliver high accuracy in object detection—often at the cost of intensive computation—they are inherently unsuitable for edge computing environments, which are projected to serve as the foundational infrastructure for next-generation general-purpose artificial intelligence. Furthermore, empirical evidence indicates that many deep models contain substantial structural and functional redundancy [17]; consequently, the associated computational overhead is not only unnecessary but also detrimental to resource-constrained edge devices and energy-efficient cloud data centers. In this context, the development of lightweight neural network architectures emerges as a strategically important and technically viable solution to reconcile accuracy, efficiency, and deployability. Fanghui Chen et al. classify lightweight neural network methods into two categories: Design Models and Compression Models. Under Design Models, lightweight convolutional design and lightweight network architecture design represent well-established research directions. In contrast, Compression Models—including network pruning, low-rank decomposition, low-bit quantization, and knowledge distillation—are generally more applicable once the network architecture has been fixed [18].

Lightweight convolution serve as the cornerstone of lightweight network design. The central principle involves refining traditional convolution methods to minimize redundant computations and parameters, while maintaining robust feature extraction performance. Representative lightweight convolution primarily include Depthwise Separable Convolutions [19], which were first systematically and comprehensively defined as Depthwise Separable Convolution (DWConv + Pointwise Conv) and applied in the Xception network—one of the most influential lightweight convolution architectures; Group Convolution [20] partitions input and output channels into equal-sized groups, performing standard convolution independently within each group without cross-group communication. Depthwise Separable Convolution can be considered an extreme special case of group convolution, while Pconv also evolved from the same concept; Ghost Convolution aims to produce more feature maps with reduced computational cost [21]; Mixed convolution integrates multiple convolutional operations to achieve lightweight network designs, and a prominent example is GSconv, which employs depthwise separable convolutions in conjunction with traditional convolutions to balance computational efficiency with representational capacity [22].

The fundamental principle of lightweight network architecture design is to rationally organize the overall network topology by eliminating redundant layers and improving feature reuse. This strategy reduces the number of parameters while maintaining effective feature extraction performance. Representative lightweight network architectures mainly comprise the MobileNet series (V1–V4) [23,24,25,26], the ShuffleNet series (V1–V2) [27,28], SqueezeNet [29], and CSPNet [30]. The design of lightweight convolution is often integrated with lightweight network architecture strategies to achieve more efficient convolutional neural networks.

Research on lightweight convolutional neural networks has enhanced the balance between accuracy and efficiency through coordinated architectural design and convolution optimization, thereby facilitating deployment on edge devices. However, existing approaches still face limitations, as excessive model simplification can result in significant accuracy loss. Thus, the development of more efficient architectures and operations to achieve an optimal accuracy–speed trade-off on edge platforms remains a central challenge in ongoing research.

2.3. Structural Re-Parameterization

Structural Re-parameterization is a structural optimization technique designed to balance model accuracy and efficiency by decoupling the training and inference phases. During training, it employs multi-branch architectures to enhance feature re-presentation. At inference, these complex structures are merged into a single streamlined architecture via parameter equivalence transformations, thereby improving performance without introducing extra computational cost or parameter complexity. In 2019, ACNet systematically demonstrated for the first time the efficacy of re-parameterizing asymmetric convolutional structures. During training, the model concurrently learned 1 × 3, 3 × 1, and 3 × 3 convolutional branches, which were equivalently merged into standard 3 × 3 convolutions at inference time, thereby improving the diversity of feature extraction in convolutional kernels [31]. RepVGG further systematized structural re-parameterization by employing a three-branch training architecture consisting of 3 × 3 convolutions, 1 × 1 convolutions, and identity mappings. Through Conv-Batch Normalization (BN) fusion and branch parameter aggregation, the architecture is equivalently transformed into a lightweight, VGG-style single-branch network during inference, achieving a balance between high accuracy and deployment efficiency [32].

Subsequently, studies including DBB [33], RepFC [34], and GCBlock [35] have further diversified the forms of multi-branch architectures. In parallel, structural re-parameterization has been deeply integrated with lightweight design and large-kernel convolutions, emerging as a widely adopted paradigm for improving the inference efficiency of visual models. This methodology is now extensively utilized in real-time vision applications.

3. Methodology

3.1. PMS-Net: A Progressive Multi-Scale Instance Segmentation Framework

To address challenges in instance segmentation for urban road scenes—such as the frequent omission of small objects, severe interference from complex backgrounds, and constraints in edge-device deployment—this paper proposes a lightweight progressive multi-scale instance segmentation network, PMS-Net (Progressive Multi-Scale Network). Guided by three core objectives, namely full utilization of multi-scale information, enhanced feature representation capability, and efficient lightweight deployment, the proposed network significantly improves segmentation performance in complex scenarios while maintaining real-time inference efficiency. The overall architecture is illustrated in Figure 2. The interlayer information of the YOLO11n and PMS-Net backbones is shown in Table 1.

Figure 2. The architecture of the PMS-Net.

Table 1. Comparison of backbone interlayer information of YOLO11n and PMS-Net.

YOLO11n			PMS-Net
Input	Operator	Output	Input	Operator	Output
$640\times 640\times 3$	Conv+BN+SiLU	$320\times 320\times 64$	$640\times 640\times 3$	Conv+BN+SiLU	$320\times 320\times 64$
$320\times 320\times 64$	Conv+BN+SiLU	$160\times 160\times 128$	$320\times 320\times 64$	Conv+BN+SiLU	$160\times 160\times 128$
$160\times 160\times 128$	C3k2	$160\times 160\times 128$	$160\times 160\times 128$	MSFA	$160\times 160\times 256$
$160\times 160\times 128$	Conv+BN+SiLU	$80\times 80\times 256$	$160\times 160\times 256$	GCConv	$80\times 80\times 256$
$80\times 80\times 256$	C3k2	$80\times 80\times 256$	$80\times 80\times 256$	MSFA	$80\times 80\times 512$
$80\times 80\times 256$	Conv+BN+SiLU	$40\times 40\times 512$	$80\times 80\times 512$	GCConv	$40\times 40\times 512$
$40\times 40\times 512$	C3k2	$40\times 40\times 512$	$40\times 40\times 512$	MSFA	$40\times 40\times 512$
$40\times 40\times 512$	Conv+BN+SiLU	$20\times 20\times 512$	$40\times 40\times 512$	GCConv	$20\times 20\times 512$
$20\times 20\times 512$	C3k2	$20\times 20\times 512$	$20\times 20\times 512$	MSFA	$20\times 20\times 1024$
$20\times 20\times 512$	SPPF	$20\times 20\times 1024$	$20\times 20\times 1024$	SPPF	$20\times 20\times 1024$
$20\times 20\times 1024$	C2PSA	$20\times 20\times 1024$	$20\times 20\times 1024$	C2PSA	$20\times 20\times 1024$

Specifically, in the feature extraction stage, a progressive multi-scale feature aggregation strategy is introduced, and a Multi-Scale Feature Aggregation (MSFA) module is designed. Unlike conventional single-scale processing or simple feature concatenation, the proposed module progressively integrates contextual information across different receptive fields during forward propagation, thereby strengthening the modeling of correlations between shallow fine-grained features and deep semantic representations. This design effectively alleviates issues such as weak representation of small objects, loss of edge details, and pronounced scale variation in complex scenes, thereby enhancing the model’s multi-scale perception and feature representation capability.

In the feature compression and spatial information encoding stages, an efficient convolutional strategy based on structural reparameterization is adopted, and a Golden Cudgel Convolution (GCConv) [35] module with enhanced feature interaction capability is introduced. During training, a multi-branch architecture is employed to improve local spatial modeling and channel-wise feature representation, enabling richer and more discriminative feature learning. During inference, structural reparameterization merges multiple branches into a single standard convolution, achieving performance gains without additional computational overhead, thus effectively addressing the limited representational capacity commonly observed in lightweight networks.

Finally, in the feature reconstruction and semantic recovery stage, a lightweight dynamic upsampling strategy, DySample, is introduced. In contrast to traditional fixed interpolation methods, DySample adaptively predicts sampling offsets to dynamically adjust sampling locations, enabling more accurate recovery of object boundaries and spatial structures. This significantly improves spatial alignment, semantic continuity, and fine-grained detail reconstruction in feature maps, thereby enhancing segmentation performance for complex object contours and densely populated scenes.

Through these design improvements, PMS-Net achieves substantial gains in multi-scale feature modeling, lightweight deployment efficiency, and segmentation accuracy in complex environments, providing an effective solution for real-time instance segmentation on edge devices.

3.2. Multi-Scale Feature Aggregation (MSFA) Module

In instance segmentation tasks for complex urban road scenes, models are required to simultaneously achieve accurate object localization, fine-grained boundary delineation, and effective cross-scale semantic modeling. In particular, when objects exhibit significant scale variation, severe background interference, and frequent occlusion, conventional single-scale or shallow local feature modeling methods often fail to capture complex spatial structures adequately. To address these challenges, this paper proposes an incremental multi-scale feature modeling framework, namely the MSFA Module, whose overall architecture is illustrated in Figure 3, with the implementation procedure detailed in Algorithm 1. Within this framework, the MSFA Module functions as the core multi-scale feature modeling unit, enabling hierarchical feature fusion and progressive representation learning from local details to global semantics. To facilitate this process, a fundamental computational component, termed the MSFA Block, is designed to perform feature extraction and multi-scale spatial modeling.

Figure 3. Schematic Diagram of the MSFA Module and GCConv.

Specifically, within the MSFA Block, the input features are first processed by GCConv, which provides efficient feature representation capability. During training, its multi-branch architecture enhances feature interaction across both spatial and channel dimensions, enabling richer contextual representation learning. During inference, structural reparameterization is employed to fuse the multi-branch structure into a standard convolutional operation, thereby improving feature representation quality without introducing additional computational overhead. Subsequently, the extracted features are split along the channel dimension into two branches, denoted as $x_{1\_1}$ and $x_{1\_2}$. Among them, $x_{1\_1}$ preserves the original information flow to maintain low-level detail information and stable gradient propagation, whereas $x_{1\_2}$ is forwarded into the progressive multi-scale spatial modeling pathway. Within this pathway, features undergo progressive spatial modeling through $5\times 5$ and $7\times 7$ depthwise convolutions (DWConv), enabling gradual receptive-field expansion from local regions to broader contextual regions. Since DWConv performs spatial convolution independently within each channel, the use of larger convolution kernels introduces only limited computational cost. Consequently, the proposed design effectively enlarges the receptive field under lightweight constraints while enhancing the network’s capability to model complex structures and multi-scale objects. Finally, feature branches generated at different stages are concatenated and fused, followed by cross-scale information integration using $1\times 1$ convolutions. Residual connections are further incorporated to improve gradient propagation stability and enhance feature reuse efficiency.

Algorithm 1 MSFA Module

1: Input: $X\in {\mathbb{R}}^{C_1\times H\times W}$   2: Output: $Y\in {\mathbb{R}}^{C_2\times H\times W}$   3: $x_0={\mathrm{Conv}}_{1\times 1}\left(X\right)$   4: $(x_{0\_1},x_{0\_2})=\mathrm{chunk}(x_0,2)$   5: $X_{list}=\{x_{0\_1},x_{0\_2}\}$   6: for $i=1$ to n do   7:      $x_1={\mathrm{GCConv}}_{3\times 3}\left(x_{0\_2}\right)$  ▹ MSFA Block begin   8:      $(x_{1\_1},x_{1\_2})=\mathrm{chunk}\left(x_{1\_2}\right)$   9:      $x_2={\mathrm{DWConv}}_{5\times 5}\left(x_{1\_1}\right)$ 10:      $(x_{2\_1},x_{2\_2})=\mathrm{chunk}\left(x_{2\_2}\right)$ 11:      $x_3={\mathrm{DWConv}}_{7\times 7}\left(x_{2\_1}\right)$ 12:      $x_4=\mathrm{Concat}(x_{1\_2},x_{2\_2},x_3)$ 13:      $x_{\mathrm{out}}^{\left(i\right)}={\mathrm{Conv}}_{1\times 1}\left(x_4\right)+x_{0\_2}$ 14:      $X_{list}\leftarrow X_{list}\cup \left\{x_{\mathrm{out}}^{\left(i\right)}\right\}$    ▹ MSFA Block end 15: end for 16: $x_{final}=\mathrm{Concat}\left(X_{list}\right)$ 17: $Y={\mathrm{Conv}}_{1\times 1}\left(x_{final}\right)$ 18: return Y

In summary, DySample allows PMS-Net to adaptively refine sampling positions based on input feature content during the dynamic feature reconstruction stage. While introducing negligible computational overhead, it significantly improves spatial alignment accuracy and boundary representation quality, thereby providing more precise and structurally consistent high-resolution feature representations for downstream mask prediction tasks in instance segmentation.

3.3. Dynamic Upsampling Method DySample

In instance segmentation tasks for complex urban road scenes, the quality of high-resolution feature reconstruction plays a critical role in mask prediction accuracy. In particular, when object boundaries are highly irregular, occlusions are frequent, and scale variations are pronounced, precise spatial alignment and structural recovery become key factors for improving segmentation performance. As an essential bridge between low-resolution semantic representations and high-resolution spatial details, the upsampling process significantly influences the quality of the final feature representation. Conventional rule-based interpolation methods are computationally efficient and suitable for real-time applications; however, they are inherently content-agnostic and lack adaptive modeling of local structural variations and semantic distributions, which often leads to boundary blurring, spatial misalignment, and loss of fine-grained details.

To address spatial misalignment and limited semantic continuity—common limitations of fixed interpolation strategies in the upsampling stage—this work introduces DySample [36], a lightweight dynamic upsampling module (illustrated in Figure 4). DySample implements a sampling-based dynamic upsampling framework: Given an input feature map $X\in {\mathbb{R}}^{C\times H\times W}$, it first generates a sampling set of adaptive sampling coordinates via a dedicated sampling point generator. Subsequently, the original features are resampled using grid_sample according to the predicted sampling coordinates, producing an output feature map $X^{\prime }$ spatial resolution $(sh,sh)$. Specifically, the effective sampling grid s is formulated as $s=G+O$, where G denotes the original sampling grid and O represents the learnable offset field predicted by the generator. To accommodate diverse deployment constraints and accuracy requirements, DySample offers two configurable variants of the sampling point generator: one employing a static scope factor, and another incorporating a dynamic scope factor. Unlike conventional interpolation methods that rely on fixed, content-agnostic sampling grids, DySample through end-to-end optimized sampling offsets. This mechanism allows the network to dynamically refine sampling locations in response to local feature semantics, thereby improving fidelity in boundary delineation and preserving fine-grained structural details.

Figure 4. The network architecture of DySample.

By dynamically adjusting sampling locations, DySample enables the upsampled features to better preserve object contours and structural information, while providing more accurate spatial context modeling during the feature fusion process. In addition, a group-shared offset prediction mechanism is adopted to further enhance the spatial modeling capability of features while maintaining a low parameter count and computational cost. This design effectively improves the quality of mask prediction in instance segmentation tasks without compromising real-time inference efficiency.

4. Experiment and Analysis

4.1. Introduction to the Experimental Dataset

Cityscapes [37]: Cityscapes is a high-resolution, pixel-annotated benchmark dataset specifically designed for urban scene understanding. It is widely adopted in computer vision research, particularly for semantic and instance segmentation tasks. The dataset consists of 5000 high-fidelity images ($2048\times 1024$ pixels), rigorously partitioned into training (2975 images), validation (500 images), and test (1525 images) subsets according to the official protocol. For instance segmentation, Cityscapes provides fine-grained annotations for eight object categories: person, rider, car, truck, bus, train, motorcycle, and bicycle. Each instance is assigned a unique identifier, necessitating models to jointly perform pixel-wise classification and precise instance discrimination—even among objects of the same semantic class. Owing to the inherent complexity of urban environments—including substantial scale variation, frequent partial occlusion, and high object density—the dataset poses stringent requirements on model capabilities in boundary delineation, hierarchical feature representation, and robust multi-scale segmentation. As such, Cityscapes has become a standard evaluation benchmark in urban scene analysis.

4.2. Experimental Conditions and Parameter Configuration

To ensure experimental reproducibility and methodological rigor, all evaluations were conducted under a strictly controlled hardware and software configuration. The experiments were performed on a workstation equipped with an Intel Core i9-14900KF CPU, 64 GB of DDR4 RAM operating at 5600 MHz, and an NVIDIA GeForce RTX 5090 GPU with 32 GB of video memory. The software stack included Python 3.11.9, PyTorch 2.7.0 (compiled with CUDA 12.8), and Torchvision 0.22.0.

All models were trained for a fixed duration of 500 epochs. Input images were uniformly resized to 640 × 640 pixels prior to augmentation. To mitigate overfitting, an early stopping mechanism was implemented with a patience of 100 epochs—monitoring validation loss—and training terminated automatically upon convergence. Stochastic Gradient Descent (SGD) served as the primary optimizer, configured with a batch size of 16. Additional hyperparameters are provided in Table 2. The training loss functions adopted in this study follow the standard YOLO-style loss formulation, consisting of classification loss, bounding box regression loss, and objectness confidence loss.

Table 2. Hyperparameter settings used in training.

Hyperparameter	Value
Initial learning rate	0.01
Learning rate decay factor	0.01
Close mosaic	10
Momentum	0.937
Weight decay	0.0005
Warmup epochs	3.0
Warmup momentum	0.8
Warmup bias learning rate	0.1

4.3. Model Evaluation Metrics

Model performance was evaluated using Mean Average Precision (mAP), the standard evaluation metric for instance segmentation. Following established conventions in the field, we report mAP at two intersection-over-union (IoU) thresholds: mAP50 (IoU = 0.5) and mAP50–95 (mean mAP across IoU thresholds from 0.5 to 0.95 in 0.05 increments). The mAP computation proceeds as follows:

$$\begin{array}{cc}\hfill \mathit{Precision}& ={\displaystyle \frac{TP}{TP+FP}}\times 100\%\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \mathit{Recall}& ={\displaystyle \frac{TP}{TP+FN}}\times 100\%\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \mathit{AP}& ={\int }_0^{1}P\left(R\right)d\left(R\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \mathit{mAP}& ={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}A{P}_i\times 100\%\hfill \end{array}$$

(4)

In the definitions above, a True Positive (TP) denotes an instance correctly classified by the model as belonging to the positive class, given that it is indeed positive; a False Positive (FP) denotes an instance incorrectly classified as positive despite being negative; and a False Negative (FN) denotes an instance incorrectly classified as negative despite being positive. Let n represent the total number of semantic classes, and P(R) denote the precision value attained at a given recall level R [38]. Additionally, to comprehensively evaluate the model’s efficiency in practical applications, metrics such as parameters, floating-point operations per second (FLOPs), and detection speed were introduced. Detection speed is measured by frames per second (FPS).

4.4. Experiment Results and Analysis

4.4.1. MSFA Module Experimental Results and Analysis

The MSFA Module combines progressive multi-scale feature fusion with lightweight DWConv, enabling the network to capture multi-scale features required for instance segmentation while maintaining computational efficiency. To evaluate the contributions of the channel splitting strategy and the DWConv-based multi-scale modeling in MSFA, a systematic controlled study was conducted.

First, we evaluated the influence of channel splitting ratios on segmentation performance by comparing three configurations—0.25, 0.50, and 0.75—while holding all other architectural and training hyperparameters constant. Results are reported in Table 3. Second, to isolate the effect of the GCConv in the MSFA Module, we substituted it with standard convolutions, GSConv, and DWConv, respectively, under identical experimental conditions. Comparative results are also summarized in Table 4.

Table 3. Experimental results on channel splitting ratios.

Split Factor	Box(mAP50	mAP50–95)	Mask(mAP50	mAP50–95)	FPS	Parameters	FLOPs
0.25	45.0%	25.8%	34.8%	17.0%	158	2.89M	11.5G
0.50	47.4%	27.2%	36.7%	17.9%	175	2.89M	11.5G
0.75	45.5%	26.5%	34.9%	16.4%	158	2.91M	11.6G

Table 4. Performance comparison of different convolution operators in MSFA Modules.

Method	Box(mAP50	mAP50–95)	Mask(mAP50	mAP50–95)	FPS	Parameters	FLOPs
Conv	44.7%	26.0%	34.4%	17.0%	175	2.88M	11.5G
GSConv	44.5%	25.5%	34.3%	15.9%	151	2.66M	10.8G
DWConv	42.4%	24.9%	33.7%	16.5%	175	2.43M	9.9G
GCConv	44.9%	26.4%	34.9%	17.1%	189	2.88M	11.5G

As shown in Table 3, the 0.50 channel splitting ratio achieves the best overall performance in detection accuracy, instance segmentation quality (mAP50), and inference efficiency. In particular, segmentation mAP50 increases by 1.9% compared with the 0.25 and 0.75 configurations, and the FPS increased by 10.76%, indicating that a 50% split ratio is the optimal choice, striking an effective balance between feature retention and multiscale context modeling. In this setting, half of the channels bypass convolution and are directly propagated across layers to retain fine-grained spatial information, while the remaining channels undergo hierarchical DWConv processing to capture multi-scale contextual features and improve discrimination across objects of different scales.

In contrast, the 0.25 ratio allocates insufficient capacity to the multi-scale branch, resulting in limited contextual enhancement and degraded performance on small and medium objects. Conversely, the 0.75 ratio overly emphasizes multi-scale modeling while weakening direct spatial feature propagation, which reduces boundary precision and lowers inference speed to 158 FPS. These results indicate that the channel splitting ratio is a key architectural factor for instance segmentation, and the 0.50 configuration achieves the best balance between accuracy, efficiency, and robustness.

Analysis of Table 4 shows that GSConv reduces computational complexity by combining standard convolution with DWConv, but this efficiency-oriented design introduces a performance trade-off: mAP50–95 decreases by 1.1%, and inference speed drops from 175 to 151 FPS, indicating that reduced computation does not necessarily lead to higher practical efficiency. DWConv achieves the lowest parameter count (2.43M) due to channel-wise convolution, yet the lack of cross-channel interaction limits representational capacity, resulting in the lowest mAP50 (33.7%) and making it less suitable for high-accuracy instance segmentation. In contrast, GCConv delivers the best overall performance: its multi-path convolution structure enhances feature representation during training, raising mAP50 to 34.9%, while during inference it is re-parameterized into a single $3\times 3$ convolution, eliminating extra overhead and increasing inference speed from 175 to 189 FPS, thereby improving both accuracy and efficiency without increasing model complexity.

4.4.2. The Impact of Different Upsampling Methods on Model Performance

To evaluate the impact of different upsampling strategies in PMS-Net, a controlled ablation study is conducted by replacing the upsampling operator with four representative methods: Nearest Neighbor, EUCB, CARAFE, and DySample. All experiments use the same network architecture, training protocol, and hyperparameter settings, with the upsampling operator as the only variable. This design enables a fair comparison and isolates the effect of each method on feature reconstruction quality, spatial alignment, and instance segmentation accuracy.

As shown in Table 5, different upsampling strategies produce notable performance differences on the Cityscapes instance segmentation benchmark. Nearest Neighbor, a fixed parameter-free interpolation method, achieves the highest inference speed and lowest computational cost but suffers from the lowest accuracy due to its rigid sampling grid, which limits adaptive spatial localization and weakens boundary representation. In contrast, DySample dynamically predicts sampling offsets to refine spatial alignment and preserve structural continuity in high-resolution features. Compared with Nearest Neighbor, it improves Box mAP50 by 0.6%, Box mAP50–95 by 0.8%, Mask mAP50 by 1.3%, and Mask mAP50–95 by 0.6%, while introducing only minimal computational overhead and maintaining similar inference speed. CARAFE slightly improves Box mAP50–95 by 0.1% and Mask mAP50 by 0.4%, but adaptive kernel generation increases computation and reduces inference speed to 159 FPS. EUCB achieves moderate gains through lightweight convolutions and channel-wise reordering, but its lack of dynamic sampling limits performance on complex boundaries and fine structures.

Table 5. Performance comparison of different Upsampling methods.

Method	Box(mAP50	mAP50–95)	Mask(mAP50	mAP50–95)	FPS	Parameters	FLOPs
Nearest	46.8%	26.4%	35.4%	17.3%	182	2.88M	11.5G
Carafe	46.4%	26.5%	35.8%	17.0%	159	3.02M	11.8G
EUCB	47.0%	27.4%	36.1%	17.8%	179	2.97M	12.0G
DySample	47.4%	27.2%	36.7%	17.9%	175	2.89M	11.5G

Overall, DySample achieves consistent improvements in both detection and segmentation while maintaining high inference efficiency and low computational cost. Its dynamic sampling mechanism enables more accurate reconstruction of high-resolution features, particularly improving boundary fidelity and fine-grained structural details. Compared with CARAFE and EUCB, DySample provides a better balance between segmentation accuracy and efficiency, making it more suitable for practical real-time instance segmentation.

4.4.3. The Performance Comparison with Other Detection Models

In this section, we conduct a systematic comparative experiment between the proposed PMS-Net and other state-of-the-art (SOTA) detectors on the Cityscapes dataset. All comparisons are performed under identical experimental conditions. We comprehensively evaluate the segmentation accuracy of each detector at different IoU thresholds, alongside metrics such as parameters, FLOPs, and inference speed, to fully reflect their performance in instance segmentation tasks. Detailed results are presented in Table 6, with visualizations of selected instance segmentation results shown in Figure 5.

Table 6. Comparison of different instance segmentation algorithms on the Cityscapes.

Method	Box(mAP50	mAP50–95)	Mask(mAP50	mAP50–95)	FPS	Parameters	FLOPs
YOLOv5n	42.4%	24.7%	33.0%	16.1%	161	2.44M	9.7G
RTMDet-Tiny	47.5%	27.1%	36.9%	18.1%	108	5.62M	11.9G
YOLOv8n	41.0%	23.7%	31.4%	14.9%	169	2.94M	10.7G
YOLOv10n	41.5%	24.7%	33.0%	16.0%	178	2.52M	10.4G
YOLO11n	44.8%	25.0%	33.8%	16.6%	169	2.84M	10.2G
YOLO12n	43.6%	25.3%	33.7%	16.5%	115	2.76M	9.7G
PMS-Net	47.4%	27.2%	36.7%	17.9%	175	2.89M	11.5G

Figure 5. Representative instance segmentation results of different detectors on Cityscapes dataset.

The results in the table show that PMS-Net achieves superior overall performance compared with other models in both detection and segmentation tasks. Compared with the YOLO11n, the mAP50 for bounding boxes and masks increases by 2.6% and 2.9%, reaching 47.4% and 36.7%, respectively. These gains indicate that the proposed multi-scale feature fusion strategy more effectively integrates semantic information and spatial details across different scales, improving the model’s ability to represent object structures and boundaries. Meanwhile, PMS-Net maintains strong real-time performance. Although its inference speed is slightly lower than YOLOv10n (178 FPS), it significantly outperforms several structurally complex detectors. These results demonstrate that the proposed method enhances model expressiveness without introducing excessive computational overhead.

Overall, PMS-Net demonstrates strong comprehensive performance on the Cityscapes instance segmentation task. By improving multi-scale feature modeling and feature interaction, the model more effectively captures object structures and boundary information in complex urban scenes, thereby enhancing overall segmentation performance while maintaining favorable computational efficiency.

4.4.4. Ablation Experiment

To quantify the contribution of each component in PMS-Net to the model’s detection performance, we conducted a series of ablation experiments. All experiments are performed on the Cityscapes dataset with identical training and testing settings to ensure fair comparison. A stepwise module introduction strategy is adopted to evaluate both the individual effects of each module and their combined interactions. Six configurations are tested, including the baseline model, three single-module variants, and combinations of multiple modules. The results and corresponding performance metrics are presented in Table 7.

Table 7. Ablation experiment results.

MSFA	GCConv	DySample	Box(mAP50	mAP50–95)	Mask(mAP50	mAP50–95)	FPS	Parameters	FLOPs
			44.8%	25.0%	33.8%	16.6%	169	2.84M	10.2G
✓			44.9%	26.4%	34.9%	17.1%	189	2.88M	11.5G
	✓		44.9%	25.5%	34.1%	16.5%	172	2.84M	10.2G
		✓	44.9%	25.6%	34.3%	16.6%	141	2.85M	10.2G
✓	✓		46.8%	26.4%	35.4%	17.3%	182	2.88M	11.5G
✓	✓	✓	47.4%	27.2%	36.7%	17.9%	175	2.89M	11.5G

Note: The symbol ✓ indicates that the corresponding module is adopted in the model.

To quantify the contribution of each component in PMS-Net to the model’s detection performance, we conducted ablation experiments on Cityscapes, and further supplemented comparative tests on the aerial VisDrone2019 dataset to validate the generalization of our proposed modules. All experiments adopt identical training and testing configurations for fair comparison. A stepwise module introduction strategy is adopted to evaluate both the individual effects of each module and their combined interactions. Six configurations are tested, including the baseline model, three single-module variants, and combinations of multiple modules. The Cityscapes ablation results are presented in Table 7, while cross-dataset comparison results on VisDrone2019 are listed in Table 8.

Table 8. Performance Comparison on VisDrone2019.

VisDrone2019	YOLO11n				PMS-Net
	mAP50	mAP50–95	Parameters	FLOPs	mAP50	mAP50–95	Parameters	FLOPs
pedestrian	23.3%	9.3%	2.58M	6.3G	25.2%	10.0%	2.64M	7.7G
people	12.1%	4.1%			12.9%	4.4%
bicycle	7.0%	2.8%			7.2%	2.8%
car	67.6%	41.7%			69.5%	43.1%
van	31.3%	20.0%			33.8%	21.9%
truck	31.2%	18.6%			32.7%	20.0%
tricycle	13.2%	7.0%			15.3%	8.3%
awning-tricycle	14.3%	7.7%			16.9%	9.0%
bus	51.4%	33.9%			52.7%	35.3%
motor	23.9%	9.4%			26.5%	10.4%
Avg	27.5%	15.5%			29.3%	16.5%

In single-module experiments, introducing MSFA increased Mask mAP50 to 34.9%, Mask mAP50–95 to 17.1%, and FPS to 189, with parameters slightly rising to 2.88M, indicating improved multi-scale feature representation with high efficiency. With GCConv, Mask mAP50 reached 34.1% and FPS increased to 172, demonstrating that re-parameterized multi-path modeling strengthens feature representation during downsampling while maintaining a lightweight design. Replacing the upsampling operator with DySample raised Mask mAP50 to 34.3% with nearly unchanged parameters, showing that dynamic sampling improves feature accuracy in the upsampling stage with minimal overhead.

In multi-module experiments, combining MSFA and GCConv further improved performance, increasing Mask mAP50 to 35.4% while maintaining 182 FPS, indicating complementary advantages between multi-scale modeling and feature enhancement. When MSFA, GCConv, and DySample were jointly integrated, the model achieved the best performance, with Mask mAP50 and Mask mAP50–95 improving by 2.9% and 1.6% over the baseline while maintaining 175 FPS. These results demonstrate that the synergy of multi-scale modeling, dynamic sampling, and enhanced downsampling enables richer contextual information and more precise spatial features, achieving a balanced improvement in segmentation accuracy and inference efficiency.

As shown in Table 8, we use lightweight YOLO11n as the baseline on VisDrone2019. PMS-Net outperforms YOLO11n across all classes, raising average mAP50 and mAP50–95 by 1.8% and 1.0% with negligible extra computation. Consistent performance improvements on aerial small targets align with the ablation conclusions on Cityscapes, proving the three core modules have strong cross-domain generalization.

4.4.5. Edge Deployment and Performance Validation

To evaluate the deployment performance and practical feasibility of the proposed PMS-Net, inference experiments are conducted on the NVIDIA Jetson Orin NX platform and compared with the baseline under identical settings. The evaluation focuses on real-time inference capability, computational efficiency, and runtime stability in urban driving scenarios.

The experimental platform is equipped with an NVIDIA Jetson Orin NX featuring an ARM CPU and an integrated GPU. TensorRT 10.3.0.30 is adopted for deployment with FP16 precision to improve inference efficiency and reduce memory consumption. Taking YOLO11n as the Baseline, and all models use a unified input size of $640\times 640$ with consistent preprocessing and postprocessing pipelines for fair comparison.

As shown in Table 9, relative to the baseline, PMS-Net achieves consistent improvements in both detection and instance segmentation accuracy. Specifically, Box mAP50 and mAP50–95 are improved by 3.0% and 2.4%, while Mask mAP50 and Mask mAP50–95 are improved by 3.7% and 1.6%, respectively. Meanwhile, PMS-Net achieves a higher inference speed of 98 FPS compared to 86 FPS for the baseline, corresponding to an approximately 14.0% improvement in real-time performance. The results demonstrate that PMS-Net can simultaneously enhance segmentation accuracy and inference efficiency on embedded edge devices. This confirms its effectiveness and practicality for real-time instance segmentation in resource-constrained and complex urban driving scenarios.

Table 9. Experimental results on the NVIDIA Jetson Orin NX platform.

Model	Box(mAP50	mAP50–95)	Mask(mAP50	mAP50–95)	FPS
Baseline	42.2%	24.4%	31.5%	15.5%	86
PMS-Net	45.2%	26.8%	35.2%	17.1%	98

5. Conclusions and Future Work

In this study, we propose an efficient real-time instance segmentation network, namely PMS-Net, designed for complex urban driving scenarios. The proposed method focuses on achieving a balance between segmentation accuracy and computational efficiency under multi-scale and real-time constraints. To enhance multi-scale feature representation and overall segmentation performance, a progressive MSFA design is introduced to strengthen the fusion of semantic information and fine-grained spatial details across different scales. Meanwhile, a re-parameterized GCConv is adopted for efficient feature encoding, improving feature representation while maintaining a lightweight inference structure. In addition, a dynamic upsampling method, DySample, is incorporated for dynamic feature reconstruction to enhance spatial alignment and boundary recovery. Together with an optimized feature pyramid structure, the overall design improves multi-scale information propagation efficiency and stability. On the Cityscapes dataset, the proposed method achieves higher segmentation accuracy and faster inference with only a slight increase in model size. This method also supports multi-class segmentation and integrates image archiving and location logging capabilities, rendering it well-suited for autonomous driving applications. Although the model is deployed on edge AI devices, it has not been validated on long-term continuous data streams collected from real autonomous driving scenarios, especially vehicle-mounted camera videos. Critical factors including long-term inference stability and latency jitter remain untested, leading to insufficient supportive evidence for practical in-vehicle deployment. Future work will focus on performing inference on long-term real-world data streams, implementing lightweight acceleration for embedded systems, and collecting more real-world samples to optimize instance segmentation categories.