EGSDK-Net: Edge-Guided Stepwise Dual Kernel Update Network for Panoptic Segmentation

Abstract

In recent years, panoptic segmentation has garnered increasing attention from researchers aiming to better understand scenes in images. Although many excellent studies have been proposed, they share some common unresolved issues. Firstly, panoptic segmentation, as a novel task, is still confined within inherent frameworks. Secondly, the prevalent kernel update strategies do not adequately utilize the information from each stage. To address these two issues, redwe propose an edge-guided stepwise dual kernel update network (EGSDK-Net) for panoptic segmentation; the core components are the real-time edge guidance module and the stepwise dual kernel update module. The first component, after extracting and positioning edge features through an extra branch, applies these features to the normally transmitted feature maps within the network to highlight the edges. The input image is initially processed with the Canny edge detector to generate and store the predicted edge map, which acts as the ground truth for supervising the extracted edge feature map. The stepwise dual kernel update module enhances the utilization of information by allowing each stage to update both its own kernel and that of the subsequent stage, thereby improving the judgment capabilities of the kernels. redEGSDK-Net achieves a PQ of 60.6, representing a 2.19% improvement over RT-K-Net.

1. Introduction

Panoptic segmentation is an important task in computer vision, with many real-world applications, such as autonomous driving, relying on the detailed information it provides. Researchers introduced panoptic segmentation [1] based on semantic segmentation and instance segmentation to achieve a more comprehensive understanding of scenes. The latter two categorize the pixels in the image into “stuff” and “thing” classes, while panoptic segmentation combines the characteristics of both. It requires classifying countable objects in the “thing” classes and classifying amorphous regions in the “stuff” classes. To better accomplish this task, recent state-of-the-art approaches [2,3,4,5,6,7] have employed a unified mask classification method for panoptic segmentation. Among them, RT-K-Net [5] has improved the structure, training, and inference process of K-Net [2], achieving real-time high-performance panoptic segmentation. Nonetheless, we must point out that there are still some clear defects in the existing methods that remain unresolved.

First, panoptic segmentation, as a novel task, is still confined within the framework of traditional segmentation tasks. It can be approximately understood as the classification of all relevant objects in the image, but viewing this task exclusively through the perspectives of “stuff” and “thing” categories has notable limitations. For instance, YOSO [6] introduces a real-time panoptic segmentation framework. By employing dynamic convolutions and feature pyramid aggregation, YOSO achieves competitive performance while reducing computational overhead, making it suitable for real-time applications. Panoptic SwiftNet [7] utilizes scale-equivariant feature extraction, cross-scale upsampling through pyramidal fusion, and boundary-aware learning of pixel-to-instance assignment to achieve fast inference over large input resolutions. Although YOSO [6] and Panoptic SwiftNet [7] demonstrate excellent efficiency and results, they are still limited to the traditional segmentation framework that classifies solely based on the “stuff” and “things” categories. Moreover, some recent segmentation algorithms [8,9] have attempted to incorporate boundary information to improve segmentation performance. Ref. [8] introduced an improved Mask R-CNN algorithm for segmenting photovoltaic hot spots in thermal infrared images. The proposed method enhances the edge features of hot spots using residual neural networks and integrates a feature pyramid structure with an edge-guided approach, improving segmentation accuracy significantly. On the other hand, ref. [9] focused on the fusion of multimodal information in high-resolution satellite and airborne remote sensing images. The authors proposed a multimodal fusion network with edge detection guidance to bridge the semantic gap and improve feature extraction. The method enhances segmentation performance by incorporating spatial boundary information and designing an adaptive fusion block to optimize multi-level feature integration. These two works underscore the importance of boundary information in improving segmentation accuracy. However, existing panoptic segmentation methods [2,3,4,5,6,7,10,11,12,13] have limited utilization of boundary information. Hence, we believe that integrating insights from external domains to address the limitations of the traditional framework could be a promising direction for future research. In addition, regarding the kernel initialization of K-Net [2], it employs two branches for mask and kernel initialization. However, these two branches are not essential, as they introduce unnecessary complexity to the model and deviate from the original design concept of a unified structure. Existing real-time panoptic segmentation methods [10,11,12,13] rely on a dual-branch structure for semantic segmentation and instance segmentation, which results in shortcomings in both accuracy and inference speed. Therefore, RT-K-Net [5] takes it a step further by using a single unified branch for initialization. Yet, we find that while the initialization process has been optimized, the kernel update still maintains the conventional “progressive” approach. When the kernel update moves to the next update step, the information from the previous step does not contribute to the process, which undoubtedly leads to a waste of information and ultimately results in suboptimal mask prediction outcomes. As shown in Figure 1, our method achieves the best PQ on the benchmark dataset Cityscapes, surpassing RT-K-Net with a particularly small increase in time cost, and significantly outperforming other real-time panoptic segmentation algorithms [10,11,12,13]. The information from the earlier step cannot only prevent information from heading in the wrong direction but also contribute to stabilizing the inference and training processes. Making effective use of this information will be crucial for enhancing the performance of panoptic segmentation.

Based on the considerations mentioned above, we propose a real-time edge guidance module and a stepwise dual kernel update module. The former is designed to better align with the requirements of the panoptic segmentation task. Panoptic segmentation classifies almost all target pixels, while the traditional concept of edges also refers to the boundaries between any objects. The two possess a fundamental similarity in their task properties. Distinguishing different objects through edges will enhance the segmentation accuracy in target boundary areas and promote stronger contrast in target regions. To achieve this, we employ a simple structure to extract edge cues. We begin by taking feature maps from the lower stage, followed by extracting edge features using convolutional structures. We apply supervision to these edge features to ensure ongoing optimization. Finally, we utilize both the max pooling layer and the average pooling layer along the channel dimension to improve attention to spatial edge positions and incorporate the edge information into the backbone’s final output to achieve edge-enhanced feature maps. The stepwise dual kernel update module optimizes the kernel update process based on RT-K-Net [5]. Specifically, previous kernel update modules only had sequential connections between the structural stages, which did not fully utilize information from the earlier stage. To this end, we optimize the kernel generation process in the kernel update step. We first generate the kernels for the current and the next stages. Then, the kernel from the current stage is fused with the kernel from the previous stage (if available) and is convolved with the feature maps to generate the mask for the current stage. Meanwhile, the kernel for the next stage is preserved and will be used in the subsequent stage. The combination of the two with RT-K-Net forms our proposed edge-guided stepwise dual kernel update network (EGSDK-Net). Our contributions can be summarized as follows:

(1)

We propose a novel architecture for the field of panoptic segmentation, called EGSDK-Net. It liberates panoptic segmentation from the constraints of semantic and instance segmentation for the first time and offers valuable structural contributions to the field.

(2)

A real-time edge guidance module (RTEGM) is designed. This module not only introduces a novel theory regarding the relationship between edge detection and panoptic segmentation but also enhances segmentation performance through a lightweight structure.

(3)

A stepwise dual kernel update module (SDKUM) is proposed. Considering the limitations of past methods in utilizing system information, SDKUM addresses this issue through a clever design. By using information more effectively, it successfully promotes advancements in model capabilities.

2. Related Work

Panoptic segmentation methods: One of the primary aims of introducing the panoptic segmentation task is to integrate semantic segmentation and instance segmentation tasks. According to the development progress of this task, it can be divided into two main categories: early panoptic segmentation methods [15,16,17,18] and unified architectures for panoptic segmentation [2,3,4,19]. The early methods either generated instance predictions using bounding box proposals [15,16] or created panoptic segmentation through alternative design strategies that did not require proposals [17,18]. In contrast, recent unified architectures for panoptic segmentation directly predict segmentation masks corresponding to their respective classes. The K-Net [2] framework employed a set of learnable kernels for consistent segmentation of instances and semantic categories, proposing a kernel update strategy to resolve the challenges of distinguishing various instances. Mask2Former [3] constructed a universal architecture for image segmentation, which includes a backbone feature extractor, a pixel decoder, and a Transformer decoder, applying masked attention along with multiple optimization improvements. In addition, some methods focus on real-time panoptic segmentation [5,10,11,12,13]. Petrovai et al. [12] tackled panoptic segmentation as a dense classification problem and generated masks for “stuff” classes as well as for each instance of “things” classes. RT-K-Net [5] is the first unified architecture for real-time panoptic segmentation. It is based on K-Net and comprehensively optimizes its structure, training, and inference processes. YOSO [6] also proposes a real-time panoptic segmentation framework. By utilizing dynamic convolutions and feature pyramid aggregation, YOSO achieves competitive performance while reducing computational overhead, making it suitable for real-time applications. Panoptic SwiftNet [7] focuses on efficient multi-scale feature extraction and boundary-aware learning, making it particularly well-suited for large-scale remote sensing applications. Considering the various shortcomings of the aforementioned methods, we propose the edge-guided stepwise dual kernel update network (EGSDK-Net).

Edge-guided methods: Due to edges being considered an additional target attribute, earlier deep learning methods generally did not integrate edge information into the network. Recently, with the growing emphasis on model performance, researchers have begun to recognize that edges can accurately represent the contours of objects in images. Consequently, numerous edge-guided solutions have emerged across various visual tasks [20,21,22,23,24,25]. To address the common oversight of edge information in significant object detection tasks for optical remote sensing images, ERPNet [20] introduced an edge-guided recursive localization network, which features edge-aware position attention units as its core module to emphasize prominent objects in these images. MLEFGN [22] introduced a framework that integrates edge detection, edge guidance, and image denoising into an end-to-end CNN model to address the issues of complex scenes and information loss. Lin et al. [24] proposed an edge-guided generative network model to produce semantically consistent outputs from small images, which provide limited information. EGCNet [25] treated the image stitching synthesis phase as an image blending problem and utilized perceptual edges to guide the network with additional geometric priors, enhancing the preservation of structural consistency. Ref. [8] proposed an improved Mask R-CNN algorithm for photovoltaic hot spot segmentation in thermal infrared images. By introducing an edge-guided feature pyramid structure and a spatial attention module, this method significantly improves segmentation accuracy. Ref. [9] addressed the challenge of multimodal fusion for high-resolution remote sensing images. The proposed method incorporates an edge detection guidance module to fuse multi-level features and spatial boundary information, thus improving the segmentation performance for complex scenes. In the methods mentioned above, refs. [21,23,24] utilized the Canny method [26] to generate edges or to treat its edge predictions as ground truth. Although the Canny method [26] is somewhat old-fashioned, it still provides a favorable balance between efficiency and accuracy, making it a technique worth considering.

3. Method

In this section, we will provide a detailed introduction to the proposed EGSDK-Net from three aspects: the overall architecture, the real-time edge guidance module, and the stepwise dual kernel update module. Since our method is built upon RT-K-Net [5] and K-Net [2], in the section focusing on the stepwise dual kernel update module, we will review the workflows of K-Net and RT-K-Net, while also presenting our improvements for a better understanding.

3.1. Overall Structure

The overall structure of EGSDK-Net is shown in Figure 2. First, we utilize the backbone composed of convolutional layers and RTFormer blocks proposed in RTFormer [27] to obtain the basic feature maps. Since RTFormer [27] employs a dual-branch module starting from stage three, the top layer of the backbone outputs low-resolution and high-resolution feature maps, denoted as $R_5^{low}$ and $R_5^{high}$, respectively. We denote the output of the second stage as $R_2$. Since $R_2$ has a high resolution and is rich in detailed information, we use it as the input for the real-time edge guidance module (RTEGM). The edge feature map extracted and supervised by the RTEGM subsequently affects $R_5^{low}$ through weighted application. The reason for choosing $R_5^{low}$ instead of $R_5^{high}$ is that $R_5^{low}$ retains a significant amount of semantic information but loses many detailed cues. Enhancing its discrimination of target areas using the edge feature map will be more effective. The edge-guided $R_5^{low}$ is then enhanced through DAPPM [28] to improve its ability to perceive multi-scale objects. After upsampling, the output of this process is combined with $R_5^{high}$ and processed through an additional convolutional module to mitigate aliasing effects, yielding the feature map F. The stage that follows the application of the edge feature map represents the neck of the entire model. The output F from the neck is then fed into the head, which consists of the stepwise dual kernel update module (SDKUM), to obtain the model’s mask predictions and class probability predictions. Considering that the kernel convolves the features to create mask predictions, additional supervision for the feature map F is required to ensure accurate mask predictions. Following [5], we use the auxiliary loss function as follows:

$$\begin{array}{c}\hfill L^{aux}={\omega }_{rank}{L}^{rank}+{\omega }_{seg}{L}^{seg}+{\omega }_{disc}{L}^{disc},\end{array}$$

(1)

where ${\omega }_{rank}=0.1$, ${\omega }_{seg}=1.0$, and ${\omega }_{disc}=1.0$ are the balancing factors used in the auxiliary loss function to balance $L^{rank}$, $L^{seg}$, $L^{disc}$, which is consistent with the design in RT-K-Net. $L^{rank}$ represents the mask-ID cross-entropy loss, $L^{seg}$ denotes the cross-entropy loss, and $L^{disc}$ is the contrastive loss function introduced by RT-K-Net [5]. The overall loss function of EGSDK-Net can be formulated as follows:

$$\begin{array}{c}\hfill L_{total}={\omega }_{mask}{L}^{mask}+{\omega }_{dice}{L}^{dice}+{\omega }_{cls}{L}^{cls}+{{\omega }_{edge}{L}^{edge}+L}^{aux},\end{array}$$

(2)

where ${\omega }_{mask}=1.0$, ${\omega }_{dice}=4.0$, ${\omega }_{cls}=2.0$, and ${\omega }_{edge}=1.0$ are also the balancing factors for the overall loss function. ${\omega }_{mask}$, ${\omega }_{dice}$, ${\omega }_{cls}$ are consistent with the design in RT-K-Net, while ${\omega }_{edge}$ is set by us. $L^{mask}$ refers to the binary cross-entropy loss, $L^{dice}$ represents the dice loss, $L^{cls}$ denotes the focal loss, and $L^{edge}$ utilizes the balanced cross-entropy loss function. The applications of $L^{mask}$, $L^{dice}$, and $L^{cls}$ are consistent with those in K-Net [2] and RT-K-Net [5], to improve the guidance of stuff masks and thing masks during training. Moreover, we adopt the same training and inference optimizations, post-processing methods, and instance-based cropping augmentation as RT-K-Net [5]. Please refer to [5] for more information on these steps.

3.2. Real-Time Edge Guidance Module

Edge detection and panoptic segmentation are closely related. Edges outline the contours of the objects to be detected, while panoptic segmentation not only divides the areas of each object but also categorizes the segmented regions. As a result, they naturally exhibit similar attributes. The current architecture for panoptic segmentation typically trains masks and categories separately. This decoupling is more advantageous for the integration of edge detection and panoptic segmentation tasks. A critical question to ponder is how to achieve reliable edges and appropriately incorporate them into the information flow. There is no intention to use complex structures to obtain edges, as their primary role is to guide panoptic segmentation in focusing on the contours of different targets. Meanwhile, the edges obtained must be accurate, which poses challenges regarding the attention to spatial locations and the training optimization process. Based on this consideration, we designed and proposed a real-time edge guidance module (RTEGM).

The structure of the RTEGM is illustrated in Figure 3. The feature map $R_2$ from the lower stages of the backbone serves as the input for the RTEGM. The process of extracting $R_2$ does not involve cross-token interaction, resulting in it being primarily composed of local information with high resolution. This greatly benefits the extraction of edge features. $R_2$ is first passed through a convolutional layer for additional information extraction and transmission. Subsequently, it is fed into the dual-branch structure of the RTEGM. The design of the first branch is aimed at the application of edge features. Inspired by the spatial attention module in CBAM [29], we performed max pooling and average pooling operations along the channel dimension on $R_2$ to generate feature maps $R_{max}$ and $R_{avg}$ at different contextual scales. This section of the design aims to make the module parameters focus more on the positions of edge features after backpropagation. $R_{max}$ and $R_{avg}$ are then aggregated to produce the edge feature descriptor $R_{mix}$, which possesses characteristics from both max pooling and average pooling. This process can be formulated as follows:

$$\begin{array}{c}\hfill R_{mix}=Avgpool\left(R_2\right)+MaxPool\left(R_2\right),\end{array}$$

(3)

where $R_{mix}\in R^{B\times C\times H\times W}$, B is the batch size, C indicates the number of feature channels and H and W are the height and width of $R_5$, respectively. Unlike CBAM [29], which concatenates the pooled results and uses a large $7\times 7$ convolution kernel to compress the channels and obtain a 2D spatial attention map, our module does not use convolution to filter the edge feature maps. Instead, we directly use additive aggregation to preserve all edge position information, avoiding the loss or insufficiency of contour information that could limit the improvement of panoptic segmentation performance. Subsequently, $R_{mix}$ is applied as weights to $R_5^{low}$; this process can be formulated as follows:

$$\begin{array}{c}\hfill R_5^{low}=R_5^{low}\times \left(1+R_{mix}\right).\end{array}$$

(4)

Since all edge positions are retained in the previous step to avoid potential noise interference, we add 1 to $R_{mix}$ to additionally reference the original target region contour positioning information. It is worth mentioning that we do not normalize $R_{mix}$ using the sigmoid function as most methods do. Instead, we use $R_{mix}$ itself as the weight. Since $R_{mix}$ contains the essential information from all channels of each pixel in two forms, allowing its value to accurately reflect the significance of each point related to edge features. Using the sigmoid function for normalization inevitably reduces the differences in response intensity concerning this significance. We believe this leads to a waste of extracted edge information and ultimately results in suboptimal performance in edge-guided panoptic segmentation. After edge enhancement, $R_5^{low}$ participates in the subsequent processes as described in Section 3.1. The other branch of RTEGM mainly supervises and continuously optimizes edge features through backpropagation. This branch consists solely of a $1\times 1$ convolution, functioning as an edge prediction head, which produces a feature map with dimensions $R^{1\times H\times W}$. After activation with the sigmoid function, this output is sent to $L^{edge}$ for supervision and guidance, which helps eliminate potential noise and enhance the quality of edge features. To provide accurate edge guidance without spending too much time in the generation process of edge ground truth, we use the widely used and validated Canny [26] to generate edge predictions for the training images. Although the Canny algorithm is a traditional edge detection method, it effectively classifies edge pixels based on the physical properties of the image and significantly reduces noise through processes such as filtering, non-maximum suppression, and double thresholding, thereby yielding stable and reliable edge detection results. Given its robustness, we consider that using the output of the Canny algorithm as the ground truth for $L^{edge}$ is a valid and practical approach. Backpropagation directly affects the $1\times 1$ convolution and the convolution processing $R_2$ that is connected to it. The benefit of this approach is that backpropagation bypasses $R_{mix}$, allowing the optimization to focus directly on the edge feature map instead of $R_{mix}$, which is more similar to an edge attention map, thus improving edge prediction performance.

3.3. Stepwise Dual Kernel Update Module

The core idea of K-Net [2] is to divide feature pixels into N meaningful groups using learnable convolutional kernels. Because there are no restrictions on the content of the groups, K-Net [2] can perform semantic segmentation, instance segmentation, and panoptic segmentation in the same manner. Initially, K-Net [2] uses N randomly initialized kernels $K_0$ to generate the initial mask prediction from the image features. As depicted in Figure 4a, K-Net [2] initializes this process with two branches: semantic segmentation and instance segmentation. It generates mask predictions $M_{inst}$ and $M_{seg}$ using the corresponding convolutional kernels and image features. However, RT-K-Net [5] argues that this dual-branch structure not only contradicts the idea of a unified architecture but also adds extra computational burden to the model. To address this, they propose simplifying the process by merging the two branches into a single branch to enhance the role of auxiliary semantic loss. The initialization process of RT-K-Net [5] is shown in Figure 4b. First, the RTFormer segmentation head produces a single feature map F. Then, F is convolved by the directly initialized panoptic kernel $K_0$ to generate the initial mask prediction $M_0$. Since the position for adding the auxiliary semantic loss has been removed, RT-K-Net [5] introduces an auxiliary semantic segmentation head that is used only during training and has the same structure as the main initialization head. We believe that RT-K-Net [5] makes impressive contributions to the mask and kernel initialization, so we use initialization methods that are essentially consistent with it, as shown in Figure 4c. However, in the subsequent kernel update process, RT-K-Net [5] uses a procedure that is identical to that of K-Net [2]. The primary reason for the kernel update is that the initial $K_0$ does not have enough discriminative ability, which results in the generated $M_0$ performing poorly. Typically, the kernel update process consists of several stages. However, the current process still applies a sequential method for kernel transmission, where the kernel information updated in the previous stage serves only as input for the next stage, which undoubtedly leads to a waste of information. To address this, we make adjustments to the kernel update content based on RT-K-Net [5], depicted in Figure 5. Our basic steps are generally consistent with those of K-Net [2] and RT-K-Net [5], primarily including the following three steps:

(1) Group Feature Assembling: First, the feature map, F, of the i-th stage is element-wise multiplied by the binarized mask prediction $B_{i-1}$ from the previous stage to form the group feature, as follows:

$$\begin{array}{c}\hfill F^k=\sum _u^H\sum _v^W{B}_{i-1}\left(u,v\right)\ast F\left(u,v\right),\end{array}$$

(5)

where $F^k\in {\mathbb{R}}^{B\times N\times C}$, B is the batch size, N represents the number of groups discussed earlier, which also equals the number of convolution kernels, and C indicates the number of feature channels.

(2) Adaptive Feature Update Since the mask $M_{i-1}$ from the ($i-1$)-th stage may be inaccurate, and there may be mutual interference from noise among the different groups, element-wise multiplication is first performed between $F^k$ and $K_{i-1}$ to derive $F^E$. Two gates, $G^F$ and $G^K$, are learned to evaluate the effects of the kernel and group features on the kernel update process, respectively. Following this, $G^F$ and $G^K$ are applied to weigh the group features and the kernel, completing the adaptive kernel update, as follows:

$$\begin{array}{c}\hfill \stackrel{\sim }{K}=G^F\ast {\phi }_1\left(F^k\right)+G^K\ast {\phi }_2\left(K_{i-1}\right),\end{array}$$

(6)

where ${\phi }_1$ and ${\phi }_2$ are fully connected layers with non-shared weights, followed by layer normalization (LN). Unlike K-Net [2] and RT-K-Net [5], $\tilde{K}\in {\mathbb{R}}^{B\times N\times 2C}$. The purpose of doubling the channel count during the use of fully connected layers is to output two updated kernels in the subsequent steps: one is the current stage kernel $K_i$ and the other is an early prediction of the next stage kernel ${\widehat{K}}_{i+1}$, which is only temporarily retained.

(3) Kernel Interaction The kernel interaction process facilitates the sharing of contextual information among different groups of each kernel. Considering both convenience and effectiveness, it is common to use a combination of multi-head attention and feedforward neural network for kernel interaction, resulting in the updated kernel $K_i$ at the current stage, which can be formulated as follows:

$$\begin{array}{c}\hfill K_i,{\widehat{K}}_{i+1}=Split\left(FFN\left(MHSA\left(\stackrel{\sim }{K}\right)\right)\right).\end{array}$$

(7)

Subsequently, we utilize different feedforward neural networks (FFNs) to generate predictions for both the mask and class probabilities. This can be formulated as follows:

$$\begin{array}{c}\hfill M_i={FFN}_M\left(K_i\right)\ast F;\end{array}$$

(8)

$$\begin{array}{c}\hfill Cl{s}_i={FFN}_C\left(K_i\right)\ast F.\end{array}$$

(9)

Before entering the next repeated update stage, we concatenate the additional predictions from the previous stage for the current stage kernel with $K_i$, formulated as follows:

$$\begin{array}{c}\hfill K_i=FFN\left(\left[K_i,{\widehat{K}}_i\right]\right),\end{array}$$

(10)

where $[.]$ represents the channel concatenation operation. At this point, the process of kernel updating is complete. Our SDKUM is obtained by combining the kernel and mask initialization proposed in RT-K-Net [5] with this kernel updating operation. Our innovation primarily lies in increasing the number of channels in the previous stage and additionally predicting the kernel for the next stage, which provides two advantages compared to conventional methods: (1) The information extracted from the previous stage is directly incorporated into the update step of the next stage, leading to better information utilization within the module. (2) Since the process of generating $K_i$ and ${\widehat{K}}_{i+1}$ involves extensive information exchange, this means that ${\widehat{K}}_{i+1}$ is rich in content from the current stage. Additionally, with the effects of backpropagation in iterations, ${\widehat{K}}_{i+1}$ also exhibits characteristics of the next stage kernel, thereby contributing to the reduction of semantic information gaps between stages. In conclusion, the stepwise dual kernel update module promotes the effective use of stage information with only a slight increase in model complexity, thereby improving the consistency between the stage kernels. Detailed experimental validation will be presented in Section 4.4.

4. Experiments

In this section, we conduct a comprehensive evaluation of the proposed EGSDK-Net, comparing it with state-of-the-art panoptic segmentation methods and performing ablation studies to assess the effectiveness of our algorithm.

4.1. The Dataset and Metrics

We utilize the widely used Cityscapes dataset [14] to study EGSDK-Net. This dataset contains a total of 5000 images at a resolution of $1024\times 2048$, split into 2975 training images, 500 evaluation images, and 1525 test images. It offers fine-grained panoptic labels across 30 categories, with 19 (8 “things” and 11 “stuff” categories) allocated for evaluation. Similar to RT-K-Net [5], we apply data augmentation techniques such as random scaling, random cropping, random adjustments to color and brightness, and random horizontal flipping, where the random cropping size is fixed at $512\times 1024$. Following earlier methods [2,5], we assess panoptic segmentation results using the standard metric: Panoptic Quality (PQ) [1]. PQ is a metric used to measure the similarity between panoptic segmentation predictions and ground truth. It unifies the evaluation criteria for both semantic segmentation and instance segmentation. By independently calculating the PQ for each category and averaging over all categories, PQ becomes insensitive to class imbalance, allowing for a fair evaluation of panoptic segmentation performance. For each category, the unique matching of predicted and ground truth segments is divided into three groups: true positives (TPs), false positives (FPs), and false negatives (FNs), representing matched segment pairs, unmatched predicted segments, and unmatched ground truth segments, respectively. Ref. [1] defines the use of intersection over union (IoU) to determine whether the ground truth segment and the predicted segment match. If $IoU>0.5$, it is considered TP. For a given class c, the formula for IoU is defined as follows:

$$\begin{array}{c}\hfill Io{U}_c=\left|{\displaystyle \frac{P_c\cap G_c}{P_c\cup G_c}}\right|,\end{array}$$

(11)

where $P_c$ and $G_c$ represent all the pixels labeled as class in the predicted and ground truth segments, respectively. Given these three groups, PQ is defined as follows:

$$\begin{array}{c}\hfill PQ={\displaystyle \frac{{\sum }_{\left(p,g\right)\in TP}IoU\left(p,g\right)}{\left|TP\right|+\frac{1}{2}\left|FP\right|+\frac{1}{2}\left|FP\right|}}.\end{array}$$

(12)

Furthermore, to provide a complete evaluation of the model’s performance, we report metrics for the “thing” and “stuff” classes, referred to as ${PQ}_{th}$ and ${PQ}_{st}$, respectively.

4.2. Implementation Details

EGSDK-Net is implemented in the Detectron2 framework using PyTorch [30]. The base variant of RTFormer is utilized, initialized with a pre-trained model from ImageNet [31]. The hyperparameters associated with training are as follows: batch size (32), initial learning rate (0.0002), weight decay (0.05), maximum iterations (90 k), optimizer (AdamW), scheduler (‘poly’), ${\omega }_{mask}$ (1.0), ${\omega }_{dice}$ (4.0), ${\omega }_{rank}$ (0.1), ${\omega }_{cls}$ (2.0), and ${\omega }_{seg}$ (1.0). We incorporate warm-up in the training process, raising the learning rate from 0.000002 to 0.0002 in the first 1000 iterations. The number of prediction masks (convolutional kernels) N is set to 100. Our kernel update frequency matches that of RT-K-Net [5]. All experiments are conducted on a single V100 GPU.

4.3. Comparison Experiment

Table 1. Comparison results between our method and the state-of-the-art panoptic segmentation methods on the Cityscapes validation set. The best results are shown in boldface and the second-best results are underlined.

Method	Backbone	$\mathit{PQ}$	${\mathit{PQ}}_{\mathit{th}}$	${\mathit{PQ}}_{\mathit{st}}$	GPU
MGNet [10]	ResNet-18	55.7	45.3	63.1	TitanRTX
K-Net [2]	ResNet50-FPN	56.9	46.0	64.8	-
Petrovai and Nedevschi [12]	VoVNet2-39	57.3	50.4	62.4	V100
Panoptic-DeepLab [11]	SWideRNet-(0.25,0.25,0.75)	58.4	-	-	V100
Hou et al. [13]	ResNet50-FPN	58.8	52.1	63.7	V100
Panoptic SwiftNet [7]	ResNet-18	55.9	-	-	RTX3090
YOSO. [6]	ResNet50	59.7	51.0	66.1	V100
RT-K-Net [5]	RTFormer	59.3	48.9	66.7	V100
Ours	RTFormer	60.6	51.9	66.8	V100

presents the quantitative comparison results of EGSDK-Net with SOTA methods [5,10,11,12,13] on the Cityscapes dataset [14]. For a fair comparison, we retrain the main reference method of EGSDK-Net in the current experimental environment, while using the official data for the other methods. According to the experimental results, EGSDK-Net demonstrates a clear leading advantage in the comprehensive evaluation of panoptic segmentation. It achieves a PQ of 60.6, leading RT-K-Net [5], the second-ranked method, by 2.19%, while other methods do not even reach a PQ of 59. In the evaluation of “thing” classes, EGSDK-Net ranks second overall, only slightly behind Hou et al. [13] (51.9 vs. 52.1). Notably, the retrained RT-K-Net [5] performs poorly in the “thing” classes, surpassing only MGNet [10]. Although the improvement over RT-K-Net [5] is not very significant, we must point out that the results of RT-K-Net [5] in the “stuff” classes already reach a relatively high value. EGSDK-Net surpassing it numerically is sufficient to confirm its outstanding performance. This result is attributed not only to the strong RTFormer backbone and the optimizations provided by RT-K-Net [5], but also to the enhancements in the kernel update strategy within EGSDK-Net.

4.4. Ablation Study

We first conduct ablation experiments on the core components of EGSDK-Net. To save on training costs, the maximum number of training iterations for the ablation experiments is set to 60 k. The experimental results and visual comparisons are shown in

Table 2. The results of the ablation experiments on the core components of EGSDK-Net.

Method	$\mathit{PQ}$	${\mathit{PQ}}_{\mathit{th}}$	${\mathit{PQ}}_{\mathit{st}}$
Baseline	59.3	48.9	66.7
Baseline w/SDKUM	60.0	50.0	67.3
Ours	60.6	51.9	66.8

and Figure 6. It is evident that inserting SDKUM into the baseline leads to significant improvements in all three metrics, particularly in the “thing” classes, which rise from 48.9 to 50.0. After inserting RTEGM, we obtain the final structure of the proposed EGSDK-Net. It is evident that the scores for PQ and ${PQ}_{th}$ keep increasing, while ${PQ}_{st}$ shows some decline. We believe this may be related to the Canny operator [26]. This operator extracts edges based on the intrinsic properties of the image, and the contours of the “stuff” classes are quite challenging to predict. When the Canny operator [26] inaccurately predicts these contours, it leads to errors in the model, which in turn results in a decline in the evaluation results for the object classes. Given the significant improvement in ${PQ}_{th}$ after adding RTEGM, we believe that the benefits of this module outweigh its drawbacks, making it worthwhile to retain. Additionally, the results in Table 2 support our analysis of performance sources discussed in Section 4.3.

Next, we investigate the internal structure of RTEGM. We set up a baseline by removing the max pooling and average pooling layers, and the results are shown in the first row of

Table 3. The results of the ablation experiments on the internal structure of RTEGM.

Method	$\mathit{PQ}$	${\mathit{PQ}}_{\mathit{th}}$	${\mathit{PQ}}_{\mathit{st}}$
Convs	59.3	48.9	66.7
only MaxPool	59.6	50.0	66.7
only AvgPool	59.4	49.3	66.7
Ours	60.6	51.9	66.8

. Next, we conduct experiments that restore either max pooling or average pooling, as shown in the second and third rows of Table 3. We observe that the enhancement from the max pooling layer is more significant. This may be because the max pooling layer functions similarly to non-maximum suppression, allowing it to eliminate false positive points with relatively lower probability values, thus producing more accurate edge feature maps. In contrast, the improvement from the average pooling layer is less pronounced, and there is a marked decline in the “stuff” classes. We think this remains tied to the difficult-to-predict edges of the “stuff” classes, as the average pooling layer does not filter out unreliable points as effectively as the max pooling layer. On the contrary, the averaging process across the channel dimension leads to more confusion and complexity in predicting the contours of the “stuff” classes, ultimately causing a drop in results. When both pooling methods are applied simultaneously, the results for the three metrics show improvement compared to when they are used separately. This might be due to the combined use of max pooling and average pooling layers, which provides a more thorough evaluation of the edges from two perspectives, thereby leading to better performance in PQ, ${PQ}_{th}$, and ${PQ}_{st}$. In addition, we conduct experiments on RTEGM as shown in

Table 4. The results of the ablation experiments on the internal structure of SDKUM.

Method	$\mathit{PQ}$	${\mathit{PQ}}_{\mathit{th}}$	${\mathit{PQ}}_{\mathit{st}}$
Baseline w/RTEGM	59.7	49.4	67.1
Using Addition for Fusion	59.5	50.0	66.7
Using Multiplication for Fusion	59.7	50.3	66.5
Ours	60.6	51.9	66.8

. We remove the existing structure of RTEGM and its corresponding supervision, instead relying entirely on the non-trainable Canny [26] for edge prediction. The results indicate that the current strategy significantly outperforms the direct use of Canny [26]. While the results of Canny [26] act as the ground truth for the edge module during training, why is the direct application of Canny [26] less effective? We attribute this to the backpropagation process, as the direct integration of Canny [26] remains separate from the entire system, which prevents timely feedback on the conditions of individual images to the network, thereby limiting parameter adjustments and leading to subpar results.

Finally, we study the internal structure of SDKUM. The focus of our experimental exploration is on how to effectively merge the additional updated kernel ${\widehat{K}}_{i+1}$ from the previous stage with the kernel update results of the current stage, as presented in Table 4. The data comparisons from these four rows indicate that concatenation is still the most effective method. We believe that the combination of concatenation with the subsequent fusion significantly enhances the interaction and knowledge integration between ${\widehat{K}}_{i+1}$ and $K_i$. In contrast, both addition and multiplication have their respective shortcomings. The former tends to retain all the information from the two kernel update outputs, which inevitably causes the previous stage to overly influence the current stage, slowing down the update speed and leading to poor results. Element-wise multiplication retains only the coexistent information in the updated results of the two kernels. As the stage increases, the overall volume of information within the module tends to decrease, which naturally leads to unsatisfactory results. In summary, the current combination method within SDKUM is the most effective.

5. Conclusions

Panoptic segmentation is a critical task in the field of computer vision, with widespread applications in areas such as autonomous driving, robotics, and medical imaging. However, despite its significance, panoptic segmentation faces challenges such as the limitations imposed by traditional segmentation frameworks and the suboptimal utilization of information in existing kernel update strategies. This paper presents a novel panoptic segmentation network, EGSDK-Net. It primarily addresses two issues: the constraints of panoptic segmentation within inherent frameworks and the low information utilization of existing kernel update strategies. To address the first issue, we take into account the relationship between edge detection and the panoptic segmentation task and propose a real-time edge guidance module. This module supervises the extracted edge features, which receive additional attention to their spatial positions, using the edges predicted by Canny as ground truth. For the other issue, we introduce a stepwise dual kernel update module. This module facilitates the rational use of information from the earlier stage by performing additional updates and combinations of the kernels from the previous stage to the next. Numerous comparison experiments and ablation studies confirm the validity and effectiveness of the proposed model. In the future, using more advanced edge detection algorithms instead of the existing Canny to generate edge ground truth appears to be a promising direction.