Structure-Prior-Guided Multi-Stage Cross-Modal Collaborative Network for RGB-D Semantic Segmentation

Abstract

Red–green–blue and depth (RGB-D) semantic segmentation combines appearance cues from RGB images with geometric information from depth maps, but sensor noise, missing measurements, and boundary-inconsistent depth responses can introduce conflicting evidence during cross-modal fusion. We propose the Structure-Prior-Guided Network (SPGNet), a dual-branch, multi-stage framework that follows a correction-before-fusion strategy. At each feature scale, SPGNet estimates a learned structure prior from cross-modal agreement and discrepancy. The Cross-Modal Correction Module (CCM) uses this prior to regulate bidirectional information transfer, suppressing unreliable responses while retaining complementary cues. The Dual-branch Enhancement Fusion Module (DEF) then enhances the corrected RGB and depth features and integrates them through shared-representation-guided interaction, after which a lightweight multi-scale decoder produces the segmentation output. Under a unified training and evaluation protocol, SPGNet achieved three-run mean Intersection over Union (mIoU) scores of 50.845% on NYU Depth V2 and 48.457% on SUN RGB-D. Compared with the best reproduced baseline on each dataset, SPGNet improved mean mIoU by 2.111 and 0.899 percentage points, respectively. These results suggest that separating reliability-oriented correction from multimodal fusion can limit the propagation of unreliable cross-modal responses and improve indoor RGB-D semantic segmentation performance.

1. Introduction

Semantic segmentation, which assigns a semantic label to each image pixel, is a fundamental problem in indoor scene understanding and supports applications such as robotic navigation, scene reconstruction, and human–environment interaction [1,2,3]. Red–green–blue (RGB) images provide rich color, texture, and semantic context, but RGB-only parsing is often degraded by weak texture, illumination variation, occlusion, and cluttered indoor layouts. With the increasing availability of depth sensors, RGB and depth (RGB-D) semantic segmentation has become an effective way to improve indoor parsing by introducing geometric information complementary to RGB appearance [4,5,6]. Depth maps describe object contours, spatial layout, and structural discontinuities, which are especially useful for distinguishing regions that are visually similar but geometrically different [7,8].

However, introducing depth information does not automatically lead to better segmentation. The main difficulty lies in both modality heterogeneity and modality reliability. RGB and depth describe the same scene from different physical perspectives: RGB features emphasize photometric appearance, whereas depth features encode geometric structure and relative spatial layout. At the same time, depth maps captured in indoor scenes may contain sensor noise, missing values, local misalignment, and boundary-inconsistent responses [4,9,10]. If these unreliable responses are directly exchanged or fused with RGB features, local conflicts between appearance and geometry can be propagated into the final representation, causing fragmented object regions, blurred boundaries, and semantic leakage across adjacent categories. Figure 1 illustrates this issue through three representative failure modes of Direct Fusion: region fragmentation, boundary leakage, and cross-modal inconsistency. Here, Direct Fusion denotes a simple RGB-D feature integration baseline without the proposed pre-fusion correction. In the region-fragmentation example (left column), Direct Fusion fails to preserve the individual chair regions and the gaps between adjacent objects, whereas SPGNet recovers a more coherent arrangement. In the boundary-leakage example (middle column), Direct Fusion confuses much of the table region and introduces errors near the chair–table boundary; SPGNet produces a cleaner separation between these categories. In the cross-modal-inconsistency example (right column), Direct Fusion yields substantial class confusion between the table and chair regions, whereas SPGNet restores their dominant semantic regions more consistently with the annotation. Although some annotation boundaries are relatively coarse, these region-level observations are consistent with the interpretation that pre-fusion correction can limit the propagation of conflicting cross-modal responses; the controlled ablation in Section 4.4 provides the corresponding quantitative evidence.

Figure 1. Qualitative motivation examples from NYU Depth V2. The three columns illustrate region fragmentation, boundary leakage, and cross-modal inconsistency, respectively. Rows (1)–(4) show the red–green–blue (RGB) images, ground-truth annotations, Direct Fusion predictions, and Structure-Prior-Guided Network (SPGNet) predictions, respectively.

Existing RGB-D semantic segmentation methods have explored this problem from several directions. Early and convolutional neural network (CNN)-based methods commonly adopt dual-branch encoders, residual fusion, gated propagation, context aggregation, or geometry-aware convolution to combine RGB appearance and depth geometry [5,6,9,11]. Recent Transformer-based and hybrid methods further introduce token interaction, cross-modal feature rectification, asymmetric modality modeling, RGB-D representation pretraining, primary-modality guidance, and depth-guided self-attention [10,12,13,14,15,16]. These studies show that RGB and depth should not be treated as two identical inputs, and that their complementary information must be selected and aligned carefully. Although these methods have significantly improved RGB-D feature interaction, reliability modeling is usually coupled with the fusion or interaction process itself. In such designs, unreliable source-modality responses may be injected into the target branch before their local compatibility is explicitly estimated. This motivates a pre-fusion mechanism that first models feature-level RGB–depth agreement and discrepancy, and then uses the resulting structure prior to regulate the direction and strength of cross-modal transfer.

Motivated by this, this paper proposes the Structure-Prior-Guided Network (SPGNet), a dual-branch multi-stage framework for RGB-D semantic segmentation. The key idea of SPGNet is to decouple reliability-oriented cross-modal correction from representation-oriented multimodal fusion. At each feature scale, SPGNet first estimates a learned feature-level structure prior from RGB–depth consistency and discrepancy cues. Instead of relying on externally supervised edge maps or ground-truth structural labels, this prior is learned in the encoded feature space and reflects local cross-modal compatibility. Guided by this prior, the Cross-Modal Correction Module (CCM) performs gated bidirectional correction to regulate cross-modal transfer before fusion. The Dual-branch Enhancement Fusion Module (DEF) then enhances the corrected modality-specific features and integrates them through shared-representation-guided interaction. Finally, a lightweight multi-scale decoder aggregates the fused features for dense prediction.

This separation enables the network to reduce the influence of locally inconsistent modality responses before feature integration, while preserving both appearance semantics and geometric structure for indoor scene parsing. Experiments on NYU Depth V2 and SUN RGB-D show that SPGNet consistently improves over representative reproduced RGB-D segmentation baselines under the unified evaluation protocol, and ablation studies further support the complementary roles of CCM and DEF.

The main contributions of this paper are summarized as follows:

• We propose SPGNet, a multi-stage RGB-D semantic segmentation framework that decouples reliability-oriented cross-modal correction from representation-oriented multimodal fusion.
• We design CCM to estimate a learned feature-level structure prior from RGB–depth consistency and discrepancy cues, and to use this prior for gated bidirectional cross-modal correction before fusion.
• We design DEF to enhance the corrected RGB and depth features and to integrate them through shared-representation-guided interaction, preserving modality-specific cues while forming a unified multimodal representation.

2. Related Work

2.1. CNN-Based RGB-D Fusion and Geometric Modeling Methods

Early related studies explored perceptual organization, multi-task geometry prediction, and explicit RGB-D constraints before deep bimodal fusion became dominant [17,18,19]. Later, RGB-D semantic segmentation methods commonly used convolutional or recurrent dual-branch architectures to combine RGB appearance with depth geometry. LSTM-CF [20] unifies contextual modeling and RGB-D fusion with LSTMs, while FuseNet [5] injects depth features into an RGB encoder–decoder network, showing that geometric information can improve indoor scene parsing. Common and specific feature learning, STD2P-based pooling, graph-based 3D relation modeling, multi-level residual fusion, gated fusion, and residual encoder–decoder designs further enriched RGB-D feature extraction and interaction [6,21,22,23,24,25]. Depth-aware CNN [7] introduces depth-aware operators, and PAD-Net [26] uses auxiliary prediction and distillation to strengthen dense representation learning. These methods demonstrate that depth can complement RGB features, while leaving room for complementary designs that explicitly estimate local source-feature conflicts before cross-modal transfer.

Other CNN-based studies place more emphasis on geometric structure, context, attention, and spatial guidance. GAD [8] distills geometry-aware embeddings and uses geometric propagation for indoor segmentation. ACNet [27] and CANet [28] model RGB-D complementarity through attention-based or collaborative feature interaction. Malleable 2.5D convolution [29] learns receptive fields along the depth axis for RGB-D scene parsing, while SCN [30] and Zig-Zag Network [31] study switchable context and cross-modal interaction for RGB-D segmentation. CEN [32] explores adaptive multimodal fusion through channel exchanging, and SA-Gate [9] performs bidirectional cross-modality propagation with separation-and-aggregation gating.

More recent CNN-based methods further investigate gated collaboration, depth enhancement, efficient encoding, and local structural operators. SGNet [33] and ESANet [34] study spatial guidance and efficient RGB-D encoding, while ShapeConv [11] introduces shape-aware convolution for local geometric structure. MGCNet [35] studies gated collaboration with multilevel feature interaction. PGDENet [36] introduces progressive guided fusion and depth enhancement for indoor scene parsing, SGACNet [37] explores spatial-information-guided adaptive context modeling, PDCNet [38] uses pixel-difference convolution, and DCANet [39] introduces differential convolution attention. These methods strengthen the use of geometric cues and cross-modal interaction from different perspectives. The present work is motivated by a complementary question: when depth features contain unreliable responses or local conflicts with RGB appearance, feature-level structural agreement and discrepancy can be used to model local source-feature reliability before cross-modal transfer. This motivates the pre-fusion correction design used in this work.

2.2. Transformer-Based RGB-D Semantic Segmentation Methods

A number of recent RGB-D segmentation methods adopt Transformer or hybrid architectures to model long-range dependencies and cross-modal correspondences. TransD-Fusion [40] studies Transformer fusion for indoor RGB-D semantic segmentation. TokenFusion [12] identifies uninformative tokens and replaces them with projected cross-modal features. UCTNet [41] introduces uncertainty-aware cross-modal Transformer modeling for indoor RGB-D segmentation. CMX [10] uses a Cross-Modal Feature Rectification Module and a Feature Fusion Module to align and fuse RGB-X features, including RGB-D and other modality combinations. DFormer [14] introduces RGB-D pre-training and RGB-D-specific blocks to improve depth-aware representation learning, and PrimKD [15] uses primary-modality-guided multimodal fusion to improve RGB-D representation learning.

Other recent methods further refine arbitrary-modal robustness, asymmetric design, efficient adaptation, and geometry-aware attention. Beyond RGB-D, arbitrary-modal segmentation [42] studies robust semantic parsing when different auxiliary modalities are available. AsymFormer [13] adopts asymmetric backbones and attention-guided feature selection to balance accuracy and efficiency, while DPLNet [43] keeps the RGB-pretrained backbone fixed and performs multimodal adaptation through prompt and adapter modules. CDMANet [44] models same-scale cross-modal correlations with central-difference mutual attention and uses a composite decoder for cross-scale fusion. DFormerv2 [16] treats depth as a geometric prior for self-attention and explicitly models depth-guided attention weights, and parameter-efficient symmetrical RGB-X modeling [45] further studies scalable multimodal semantic parsing. Together, these studies illustrate different uses of global interaction, modality-aware adaptation, and geometry-related attention mechanisms in RGB-D and broader RGB-X segmentation.

The proposed method is complementary to these studies. Many Transformer-based methods focus on global representation learning, token-level interaction, or fusion-stage module design. Some of them include rectification or token selection; in this subsection, pre-fusion local reliability filtering refers specifically to estimating feature-level agreement and discrepancy before cross-modal transfer. SPGNet focuses on this pre-fusion stage: CCM estimates a structure-prior map from relative edge agreement and feature discrepancy and uses the structure-prior map to filter cross-modal transfer; DEF then enhances and fuses the corrected modality features across stages.

3. Materials and Methods

3.1. Overall Framework

Given an RGB image $I_r\in {\mathbb{R}}^{3\times H\times W}$ and a depth map $I_d\in {\mathbb{R}}^{1\times H\times W}$, the depth channel is replicated to obtain ${\tilde{I}}_d\in {\mathbb{R}}^{3\times H\times W}$. SPGNet predicts

$$Y=\Phi (I_r,{\tilde{I}}_d),$$

(1)

where $Y\in {\mathbb{R}}^{K\times H\times W}$ contains the final logits and K includes the dataset-specific ignored label index. Supervision uses $Y^{gt}$, with unlabeled pixels excluded as defined in Section 3.5.

As shown in Figure 2, SPGNet contains two encoders, four stage-wise Cross-Modal Correction Modules (CCM), four Dual-branch Enhancement Fusion Modules (DEF), and a multi-scale decoder. At each scale, CCM uses a learned structure prior to regulate cross-modal transfer, after which DEF enhances and fuses the corrected features.

Figure 2. Overall architecture of the Structure-Prior-Guided Network (SPGNet). Red–green–blue (RGB) and depth features are extracted by heterogeneous encoders, corrected by the Cross-Modal Correction Module (CCM) at each stage, enhanced and fused by the Dual-branch Enhancement Fusion Module (DEF), and aggregated by the multi-scale decoder.

The RGB and depth encoders use Pyramid Vision Transformer v2 B2 (PVT-v2-B2) and 34-layer Residual Network (ResNet34), respectively, with ImageNet-pretrained weights. This pairing combines long-range RGB context with the local geometric features of ResNet34, whose four stages are aligned to PVT-v2-B2 through lightweight projections. For inputs divisible by 32, stage $i\in \{1,2,3,4\}$ has resolution $(h_i,w_i)=(H/2^{i+1},W/2^{i+1})$ and channels $(64,128,320,512)$, giving

$$F_r^i,F_d^i\in {\mathbb{R}}^{C_i\times h_i\times w_i},(C_1,C_2,C_3,C_4)=(64,128,320,512).$$

(2)

At each stage, the two features are first corrected and then fused:

$$({\widehat{F}}_r^i,{\widehat{F}}_d^i)={\mathrm{CCM}}_i(F_r^i,F_d^i),$$

(3)

$$F_f^i={\mathrm{DEF}}_i({\widehat{F}}_r^i,{\widehat{F}}_d^i),$$

(4)

where ${\widehat{F}}_r^i$ and ${\widehat{F}}_d^i$ are the CCM-processed RGB and depth features, and $F_f^i\in {\mathbb{R}}^{C_i\times h_i\times w_i}$ is the fused feature at stage i. The four fused features are aligned to the first-stage resolution and aggregated by the decoder to obtain $\tilde{Y}\in {\mathbb{R}}^{K\times h_1\times w_1}$:

$$\tilde{Y}=\mathrm{Decoder}(F_f^1,F_f^2,F_f^3,F_f^4).$$

(5)

The decoder output is then upsampled to the input resolution:

$$Y={\mathrm{Up}}_{\mathrm{ori}}\left(\tilde{Y}\right).$$

(6)

Section 3.2 and Section 3.3 describe CCM and DEF, and Section 3.4 specifies the decoder implementation.

3.2. Cross-Modal Correction Module

CCM estimates a structure prior before fusion to regulate bidirectional transfer and limit the propagation of unreliable depth responses, boundary inconsistencies, and RGB–depth misalignment.

Given the RGB feature $F_r$ and depth feature $F_d$ at the same scale, CCM aims to generate the updated bimodal features ${\widehat{F}}_r$ and ${\widehat{F}}_d$:

$$({\widehat{F}}_r,{\widehat{F}}_d)=\mathrm{CCM}(F_r,F_d).$$

(7)

As summarized in Figure 3, CCM constructs a prior from cross-modal consistency and discrepancy, generates branch-specific gates, and then performs bidirectional cross-attention.

Figure 3. Structure of the Cross-Modal Correction Module (CCM): (a) overall architecture; (b) fixed-edge and aligned-feature construction of the structure prior; and (c) branch-specific gate generation from the target and prior-modulated source features.

3.2.1. Structure Prior Generation

Figure 3b maps $F_r,F_d\in {\mathbb{R}}^{C\times h\times w}$ to the structure prior P by modeling both cross-modal agreement and discrepancy in the encoded feature space.

The implementation first extracts a fixed structural response from each modality. Let $\mathcal{S}\left(F\right)$ denote channel averaging followed by convolution with fixed horizontal and vertical Sobel kernels and gradient-magnitude computation. The modality-specific edge responses are

$$E_r=\mathcal{S}\left(F_r\right),E_d=\mathcal{S}\left(F_d\right),$$

(8)

where $E_r,E_d\in {\mathbb{R}}^{1\times h\times w}$. The edge-consistency response is computed explicitly as

$$C_s=1-{\displaystyle \frac{|E_r-E_d|}{E_r+E_d+ϵ}},$$

(9)

with $ϵ={10}^{-6}$. To measure feature discrepancy, the two modalities are independently aligned by bias-free $1\times 1$ convolutions, $A_r={\varphi }_r\left(F_r\right)$ and $A_d={\varphi }_d\left(F_d\right)$, and their channel-averaged absolute difference is computed as

$$D_s={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}|A_{r,c}-A_{d,c}|.$$

(10)

Thus, $C_s,D_s\in {\mathbb{R}}^{1\times h\times w}$. The two edge responses, consistency response, and discrepancy response are concatenated into a four-channel descriptor and mapped to the channel-wise structure prior:

$$P=\sigma \left({\mathcal{F}}_p\left([E_r,E_d,C_s,D_s]\right)\right),$$

(11)

where $[·]$ denotes channel concatenation, ${\mathcal{F}}_p(·)$ comprises two $3\times 3$ Conv–BN–ReLU layers followed by a $1\times 1$ Conv–BN projection, and $P\in {[0,1]}^{C\times h\times w}$. The consistency and discrepancy quantities are explicit responses; the mapping from their four-channel descriptor to P is learned.

3.2.2. Bidirectional Cross-Modal Correction

The prior P then guides symmetric depth-to-RGB and RGB-to-depth correction before fusion.

First, the two input feature streams are normalized to obtain

$${\overline{F}}_r=\mathrm{BN}\left(F_r\right),{\overline{F}}_d=\mathrm{BN}\left(F_d\right).$$

(12)

Taking the update of the RGB branch as an example, this paper treats the RGB feature as the target branch and the depth feature as the source branch, and obtains the following representations through $1\times 1$ convolutional mappings:

$$T_r={\psi }_r\left({\overline{F}}_r\right),S_d={\psi }_d\left({\overline{F}}_d\right).$$

(13)

Here, $T_r,S_d\in {\mathbb{R}}^{C\times h\times w}$ are the target RGB and source depth representations.

Then, the structure-prior map is used to perform structure-aware modulation on the source-branch feature:

$$S_d^{\prime }=S_d\odot (1+{\alpha }_{r}P),$$

(14)

where ⊙ denotes element-wise multiplication, $P\in {[0,1]}^{C\times h\times w}$ is produced by the Sigmoid output described in the previous subsection, and ${\alpha }_r$ is a learnable scalar initialized to 0. Therefore, this modulation term is an identity mapping at the beginning of training.

Because ${\alpha }_r$ is unbounded, explicit attenuation is performed by the following Sigmoid gate rather than by this modulation term.

Next, the target feature, the prior-modulated source feature, and their difference are concatenated and fed into a gating unit to generate a cross-modal gating map, following the gate-generation pattern in Figure 3c:

$$G_r=\sigma \left({\mathcal{F}}_g^r\left([T_r,S_d^{\prime },T_r-S_d^{\prime }]\right)\right),$$

(15)

where $\sigma (·)$ denotes the Sigmoid function, and $G_r\in {[0,1]}^{C\times h\times w}$.

The Sigmoid gate attenuates or preserves the modulated source response; the same construction is used in the reverse direction.

After gating, the filtered cross-modal feature is obtained as

$${\tilde{S}}_d=S_d^{\prime }\odot G_r.$$

(16)

Thus, CCM filters the source response before cross-modal transfer.

Then, lightweight cross-attention is performed by using the RGB feature as the query and the gated depth feature as the key and value. To control the computational cost, average-pooling downsampling is applied only to the key-value branch in the implementation:

$${\tilde{S}}_{d,\downarrow }={\mathrm{AvgPool}}_{r_i}\left({\tilde{S}}_d\right),$$

(17)

where $r_i$ denotes the key-value downsampling rate of CCM at the current stage, and ${\tilde{S}}_{d,\downarrow }\in {\mathbb{R}}^{C\times h_i^{\downarrow }\times w_i^{\downarrow }}$ denotes the pooled key-value feature with spatial size determined by $r_i$. Then,

$$\Delta F_r={\mathrm{CA}}_r\left(Q=T_r,K={\tilde{S}}_{d,\downarrow },V={\tilde{S}}_{d,\downarrow }\right).$$

(18)

Specifically, ${\mathrm{CA}}_r(·)$ generates the query $Q_r$ from $T_r$ and generates the key and value $K_d,V_d$ from ${\tilde{S}}_{d,\downarrow }$:

$$Q_r=W_q^r{T}_r,K_d=W_k^r{\tilde{S}}_{d,\downarrow },V_d=W_v^r{\tilde{S}}_{d,\downarrow },$$

(19)

The $1\times 1$ projections produce $Q_r\in {\mathbb{R}}^{N_i{d}_i\times h\times w}$ and $K_d,V_d\in {\mathbb{R}}^{N_i{d}_i\times h_i^{\downarrow }\times w_i^{\downarrow }}$. After attention over flattened spatial dimensions, an output projection restores $\Delta F_r\in {\mathbb{R}}^{C\times h\times w}$.

The depth update symmetrically treats depth as the target and RGB as the source:

$$T_d={\psi }_d^{\prime }\left({\overline{F}}_d\right),S_r={\psi }_r^{\prime }\left({\overline{F}}_r\right),$$

(20)

$$S_r^{\prime }=S_r\odot (1+{\alpha }_{d}P),$$

(21)

$$G_d=\sigma \left({\mathcal{F}}_g^d\left([T_d,S_r^{\prime },T_d-S_r^{\prime }]\right)\right),$$

(22)

$${\tilde{S}}_r=S_r^{\prime }\odot G_d,$$

(23)

$${\tilde{S}}_{r,\downarrow }={\mathrm{AvgPool}}_{r_i}\left({\tilde{S}}_r\right),$$

(24)

$$\Delta F_d={\mathrm{CA}}_d\left(Q=T_d,K={\tilde{S}}_{r,\downarrow },V={\tilde{S}}_{r,\downarrow }\right).$$

(25)

Here, $T_d,S_r\in {\mathbb{R}}^{C\times h\times w}$, $G_d\in {[0,1]}^{C\times h\times w}$, and ${\alpha }_d$ is initialized to 0. The projection dimensions match the RGB update: $Q_d\in {\mathbb{R}}^{N_i{d}_i\times h\times w}$, $K_r,V_r\in {\mathbb{R}}^{N_i{d}_i\times h_i^{\downarrow }\times w_i^{\downarrow }}$, and $\Delta F_d\in {\mathbb{R}}^{C\times h\times w}$.

Finally, the bimodal features are updated in a residual manner:

$${\widehat{F}}_r=F_r+{\gamma }_r\Delta F_r,{\widehat{F}}_d=F_d+{\gamma }_d\Delta F_d,$$

(26)

where ${\gamma }_r$ and ${\gamma }_d$ are learnable residual coefficients, both initialized to 0.

The zero initialization of ${\alpha }_r$ and ${\alpha }_d$ makes the prior modulation initially an identity operation, while the zero initialization of ${\gamma }_r$ and ${\gamma }_d$ guarantees that the complete CCM initially preserves its two input features. The residual scales then learn how strongly the cross-modal corrections should be introduced before DEF.

3.3. Dual-Branch Enhancement Fusion Module

Given ${\widehat{F}}_r,{\widehat{F}}_d\in {\mathbb{R}}^{C\times h\times w}$, DEF enhances each modality, constructs a shared reference, applies shared-guided cross-attention, and outputs $F_f\in {\mathbb{R}}^{C\times h\times w}$ for the decoder (Figure 4).

Figure 4. Structure of the Dual-branch Enhancement Fusion Module (DEF): (a) overall architecture and (b) Feature Enhancement Unit. The operator legend applies to both panels; ⊕ denotes residual addition and C denotes channel concatenation.

3.3.1. Modality Feature Enhancement

As shown in Figure 4b, DEF first enhances the RGB and depth features independently to preserve their modality-specific characteristics. For an arbitrary input feature $F\in {\mathbb{R}}^{C\times h\times w}$, the feature enhancement unit can be expressed as

$$\mathcal{E}\left(F\right)={\mathcal{F}}_{\mathrm{enh}}\left([{\mathcal{D}}_{3\times 3}\left(\mathcal{P}\left(F\right)\right),{\mathcal{D}}_{5\times 5}\left(\mathcal{P}\left(F\right)\right)]\right)+F,$$

(27)

Here, $\mathcal{P}$ is a $1\times 1$ projection, ${\mathcal{D}}_{3\times 3}$ and ${\mathcal{D}}_{5\times 5}$ are C-channel depthwise separable convolutions, and ${\mathcal{F}}_{\mathrm{enh}}:{\mathbb{R}}^{2C\times h\times w}\to {\mathbb{R}}^{C\times h\times w}$ fuses their concatenated outputs before residual addition.

The two kernels capture complementary local contexts while the residual path preserves the input.

Therefore, the corrected RGB and depth features are enhanced as

$$F_r^{\mathrm{enh}}=\mathcal{E}\left({\widehat{F}}_r\right),F_d^{\mathrm{enh}}=\mathcal{E}\left({\widehat{F}}_d\right).$$

(28)

3.3.2. Shared-Guided Cross-Modal Interaction

A shared reference is constructed directly from the corrected bimodal features:

$$F_m={\mathcal{F}}_m\left([{\widehat{F}}_r,{\widehat{F}}_d]\right),$$

(29)

where ${\mathcal{F}}_m(·):{\mathbb{R}}^{2C\times h\times w}\to {\mathbb{R}}^{C\times h\times w}$ is implemented by a convolutional fusion block, and therefore $F_m\in {\mathbb{R}}^{C\times h\times w}$.

Thus, $F_m$ retains CCM-corrected shared cues, whereas $F_r^{\mathrm{enh}}$ and $F_d^{\mathrm{enh}}$ retain modality-specific responses.

Subsequently, convolutional projections are applied to the enhanced RGB feature, the enhanced depth feature, and the shared fused feature, respectively, and all three outputs are kept in ${\mathbb{R}}^{C\times h\times w}$:

$$R={\mathcal{R}}_r\left(F_r^{\mathrm{enh}}\right),D={\mathcal{R}}_d\left(F_d^{\mathrm{enh}}\right),M={\mathcal{R}}_m\left(F_m\right).$$

(30)

Next, two-branch cross-attention interaction is performed by using the RGB-branch feature and the depth-branch feature as queries, respectively, and the shared fused feature as the key and value:

$$R^{\mathrm{ca}}={\mathrm{SGCA}}_r^{\mathrm{def}}\left(Q=R,K=M_{\downarrow },V=M_{\downarrow }\right)+R,$$

(31)

$$D^{\mathrm{ca}}={\mathrm{SGCA}}_d^{\mathrm{def}}\left(Q=D,K=M_{\downarrow },V=M_{\downarrow }\right)+D,$$

(32)

Only the shared key/value feature is average-pooled: $M_{\downarrow }\in {\mathbb{R}}^{C\times h_i^{\downarrow }\times w_i^{\downarrow }}$ is determined by $r_i^{\prime }$, while the R and D queries retain resolution $h\times w$. The $Q/K/V$ projections use $16N_i$ channels, scaled dot-product attention aggregates V, and an output projection restores C channels, giving $R^{\mathrm{ca}},D^{\mathrm{ca}}\in {\mathbb{R}}^{C\times h\times w}$.

Using M as the common key/value reference aligns both branches without direct RGB–depth attention.

To provide an additional bypass from the corrected DEF inputs, this paper adds the interaction results to the input features of DEF in a residual manner:

$$R^{\prime }=R^{\mathrm{ca}}+{\widehat{F}}_r,D^{\prime }=D^{\mathrm{ca}}+{\widehat{F}}_d.$$

(33)

The query-side residuals in the attention equations and the additional corrected-feature bypasses preserve modality information.

Finally, the output feature at the current scale is obtained through the final fusion head, corresponding to the “FinalFusion Head” in Figure 4a:

$$F_f={\mathcal{F}}_{\mathrm{final}}\left([R^{\prime },D^{\prime }]\right),$$

(34)

where the convolutional head ${\mathcal{F}}_{\mathrm{final}}:{\mathbb{R}}^{2C\times h\times w}\to {\mathbb{R}}^{C\times h\times w}$ produces the decoder input.

3.4. Decoder and Multi-Scale Fusion

The decoder integrates $\{F_f^1,F_f^2,F_f^3,F_f^4\}$, where $F_f^i\in {\mathbb{R}}^{C_i\times h_i\times w_i}$ and $(C_1,C_2,C_3,C_4)=(64,128,320,512)$, using one prediction branch without auxiliary supervision.

First, fused features from the second to fourth stages are upsampled to the first-stage resolution and then concatenated with $F_f^1$ along the channel dimension:

$$F_{ms}=\mathrm{Concat}\left(F_f^1,{\mathrm{Up}}_{2\to 1}\left(F_f^2\right),{\mathrm{Up}}_{3\to 1}\left(F_f^3\right),{\mathrm{Up}}_{4\to 1}\left(F_f^4\right)\right).$$

(35)

Here, ${\mathrm{Up}}_{i\to 1}(·)$ denotes bilinear interpolation that upsamples the feature of the i-th stage to $h_1\times w_1$. Under the stage-resolution relation $h_i=h_1/2^{i-1}$ and $w_i=w_1/2^{i-1}$, the corresponding scale factors are $2,4,8$ for the second to fourth stages, respectively. Therefore, $F_{ms}\in {\mathbb{R}}^{(64+128+320+512)\times h_1\times w_1}={\mathbb{R}}^{1024\times h_1\times w_1}$.

Then, a convolutional fusion layer and a classification layer are used to output logits at the first-stage resolution:

$$\tilde{Y}={\mathcal{F}}_{\mathrm{cls}}\left({\mathcal{F}}_{\mathrm{fuse}}\left(F_{ms}\right)\right).$$

(36)

${\mathcal{F}}_{\mathrm{fuse}}$ compresses the concatenated features and ${\mathcal{F}}_{\mathrm{cls}}$ produces $\tilde{Y}\in {\mathbb{R}}^{K\times h_1\times w_1}$.

Finally, the logits are resized to the original input size $H\times W$ to obtain the final output:

$$Y={\mathrm{Up}}_{\mathrm{ori}}\left(\tilde{Y}\right),$$

(37)

where $Y\in {\mathbb{R}}^{K\times H\times W}$ contains the final logits.

3.5. Loss Function

Let Y be the output logits and $Y^{gt}\in {\{0,1,\dots ,K-1\}}^{H\times W}$ the ground truth, with label 0 ignored. The valid pixels are

$${\Omega }_v=\{p\mid Y_p^{gt}\ne 0\}.$$

(38)

The overall loss function is defined as

$$\mathcal{L}={\mathcal{L}}_{seg},$$

(39)

where ${\mathcal{L}}_{seg}$ denotes the pixel-level cross-entropy loss over valid pixels:

$${\mathcal{L}}_{seg}=-{\displaystyle \frac{1}{|{\Omega }_v|}}\sum _{p\in {\Omega }_v}log{\displaystyle \frac{exp\left(Y_{Y_p^{gt},p}\right)}{{\sum }_{k=0}^{K-1}exp\left(Y_{k,p}\right)}}.$$

(40)

$K=41$ for NYU Depth V2 and $K=38$ for SUN RGB-D, corresponding to 40 and 37 valid classes plus the ignored label. The loss is applied only to the final output.

3.6. Implementation Specification of CCM and DEF

Table 1 summarizes the four-scale tensor dimensions and hyperparameters for a $480\times 640$ input. Unless stated otherwise, CCM and DEF convolutional blocks use Conv–BN–ReLU, while alignment, $Q/K/V$, Sigmoid, and output projections follow the listed rows. The coefficients ${\alpha }_r,{\alpha }_d,{\gamma }_r,{\gamma }_d$ are initialized to 0, making CCM an identity mapping at initialization. The prior hidden dimension is $C_p=max(\lfloor C/2\rfloor ,16)$; the CCM head dimension is $d_i=max(\lfloor C_i/\left(2N_i\right)\rfloor ,16)$, and the DEF head dimension is 16. Attention projections use $N_i{d}_i$ channels in CCM and $16N_i$ in DEF before being mapped back to C channels.

Table 1. Notation, dimensions, and hyper-parameter specifications of the Cross-Modal Correction Module (CCM) and Dual-branch Enhancement Fusion Module (DEF) used in this work. Subscripts r and d denote the red–green–blue (RGB) and depth branches, respectively. $C\in \{64,128,320,512\}$ denotes the output channels of stages 1–4; $h\times w$ denotes the corresponding spatial resolution; $N_i$ denotes the number of attention heads, $d_i$ denotes the CCM channels per head, $d=16$ denotes the fixed DEF channels per head, and $r_i/r_i^{\prime }$ denote the key/value downsampling ratios in cross-attention.

Symbol	Shape/Setting	Description
Inputs and stage features
$I_r,{\tilde{I}}_d$	$3\times H\times W$	RGB/channel-expanded depth input
$F_r^i,F_d^i$	$C_i\times h_i\times w_i$	Stage-i encoder outputs (RGB/depth)
${\widehat{F}}_r^i,{\widehat{F}}_d^i$	$C_i\times h_i\times w_i$	CCM-corrected features
$F_f^i$	$C_i\times h_i\times w_i$	DEF-fused features at stage i
Structure prior generation
$E_r,E_d$	$1\times h\times w$	Fixed Sobel edge magnitudes of channel-averaged RGB/depth features
${\varphi }_r,{\varphi }_d$	$C\to C$	Independent $1\times 1$ convolutional alignment for discrepancy computation
$A_r,A_d$	$C\times h\times w$	Aligned RGB/depth features
$C_s,D_s$	$1\times h\times w$	Edge-consistency and channel-averaged feature-discrepancy responses
$C_p$	$\max (\lfloor C/2\rfloor ,16)$	Hidden channels of prior generator
${\mathcal{F}}_p$	$4\to C_p\to C_p\to C$	Two $3\times 3$ Conv–BN–ReLU layers + $1\times 1$ Conv–BN before Sigmoid
P	$C\times h\times w$	Sigmoid-bounded structure prior
CCM (per branch and per stage)
${\overline{F}}_{r/d}$	$C\times h\times w$	BN-normalized features
${\psi }_r,{\psi }_d,{\psi }_r^{\prime },{\psi }_d^{\prime }$	$C\to C$	$1\times 1$ Conv–BN–ReLU target/source projection
${\alpha }_r,{\alpha }_d$	scalar	Learnable, unconstrained-sign, init=0
$C_g$	$\max (\lfloor C/2\rfloor ,32)$	Hidden channels of branch-specific gate generator
${\mathcal{F}}_g^{r/d}$	$3C\to C_g\to C$	Target/prior-modulated source/difference input; $3\times 3$ Conv–BN–ReLU + $1\times 1$ Conv before Sigmoid
$G_r,G_d$	$C\times h\times w$	Cross-modal gating mask, $\in [0,1]$
${\tilde{S}}_{r/d,\downarrow }$	$C\times \lfloor {\displaystyle \frac{h}{r_i}}\rfloor \times \lfloor {\displaystyle \frac{w}{r_i}}\rfloor$	Avg-pooled source KV feature in CCM
${\mathrm{CA}}_r,{\mathrm{CA}}_d$	$N_i,d_i=max(\lfloor C_i/\left(2N_i\right)\rfloor ,16),r=r_i$	Multi-head cross-attention; Q/K/V dim $=N_i{d}_i$; output projection $\to C_i$
${\gamma }_r,{\gamma }_d$	scalar	Learnable residual scale, init=0
DEF (per stage)
$\mathcal{E}(·)$	$C\to C$	$1\times 1$ pre + DW($3\times 3$/$5\times 5$) + conv fusion block
$F_r^{\mathrm{enh}},F_d^{\mathrm{enh}}$	$C\times h\times w$	Modality-enhanced features
${\mathcal{R}}_r,{\mathcal{R}}_d$	$C\to C$	$3\times 3$ Conv–BN–ReLU projection of enhanced features
${\mathcal{F}}_m$	$2C\to C$	Conv fusion block for shared feature construction
$F_m$	$C\times h\times w$	Shared fusion feature
${\mathcal{R}}_m$	$C\to C$	$3\times 3$ Conv–BN–ReLU projection of shared feature
$M_{\downarrow }$	$C\times \lfloor {\displaystyle \frac{h}{r_i^{\prime }}}\rfloor \times \lfloor {\displaystyle \frac{w}{r_i^{\prime }}}\rfloor$	Avg-pooled shared KV feature in DEF
${\mathrm{SGCA}}_r^{\mathrm{def}},{\mathrm{SGCA}}_d^{\mathrm{def}}$	$N_i,d=16,r=r_i^{\prime }$	Shared-guided cross-attention; Q/K/V dim $=N_{i}d$; output projection $\to C_i$; shared KV
${\mathcal{F}}_{\mathrm{final}}$	$2C\to C$	Conv fusion head ($1\times 1$ Conv–BN–ReLU + $3\times 3$ Conv–BN–ReLU + $1\times 1$ Conv–BN–ReLU)
Stage-wise hyper-parameters $(N_i,r_i)$ for CCM and $(N_i,r_i^{\prime })$ for DEF
Stage 1 ($C=64$)	$(N_i=1,r_i=8,r_i^{\prime }=8)$
Stage 2 ($C=128$)	$(N_i=2,r_i=4,r_i^{\prime }=4)$
Stage 3 ($C=320$)	$(N_i=4,r_i=4,r_i^{\prime }=2)$
Stage 4 ($C=512$)	$(N_i=4,r_i=2,r_i^{\prime }=1)$
Decoder
${\mathcal{F}}_{\mathrm{fuse}}$	${\sum }_i{C}_i\to 128$	$1\times 1$ Conv–BN–ReLU + $3\times 3$ Conv–BN–ReLU
${\mathcal{F}}_{\mathrm{cls}}$	$128\to K$	$1\times 1$ Conv for output logits
${\mathrm{Up}}_{\mathrm{ori}}$	—	Bilinear upsampling to input size $H\times W$

Note: Italics indicate section headings that group related symbols and settings.

3.7. Datasets and Experimental Settings

Experiments are conducted on two public RGB-D indoor semantic segmentation benchmarks, namely NYU Depth V2 [2] and SUN RGB-D [3]. Both datasets provide aligned RGB images, depth maps, and pixel-level semantic annotations, and cover diverse indoor scenes with different sensor characteristics and depth response patterns.

NYU Depth V2 contains 1449 indoor scene images. Following the standard split, 795 images are used for training and 654 images are used for testing. The dataset is evaluated under the NYU40 setting with 40 valid semantic classes. SUN RGB-D contains 10,335 RGB-D samples collected from different sources, including NYU Depth V2, Berkeley B3DO, and SUN3D. Following the official split, 5285 images are used for training and 5050 images are used for testing. In this study, SUN RGB-D is evaluated using 37 valid semantic classes. For both datasets, label 0 denotes unlabeled or void pixels and is excluded from both loss computation and metric computation.

All experiments were implemented in Python 3.9 using PyTorch 2.8.0 and NumPy 1.26.4 and conducted on a workstation equipped with an NVIDIA GeForce RTX 5070 Ti GPU. The optimization settings for SPGNet were based on published configurations in recent RGB-D semantic segmentation studies and adapted to our unified experimental protocol. We used AdamW with an initial learning rate of $6\times {10}^{-5}$ and a weight decay of $0.01$, matching the NYU Depth V2 configuration reported for PrimKD [15]. The use of polynomial learning-rate decay with a power of 0.9 and a warm-up phase was informed by AsymFormer [13]; in our protocol, the warm-up period was set to 5 epochs. To maintain a consistent training length across the reproduced experiments, all models were trained for 100 epochs. The batch sizes were set to 2 for NYU Depth V2 and 4 for SUN RGB-D to fit the available GPU memory. Within each dataset, the same optimization settings were used for SPGNet and the four controlled SUN RGB-D ablation configurations.

The stage-wise hyperparameters of CCM and DEF were specified according to the encoder feature dimensions and the computational cost of spatial attention. For encoder channels of $\{64,128,320,512\}$, the numbers of attention heads were set to $\{1,2,4,4\}$. The key/value downsampling ratios were $\{8,4,4,2\}$ for CCM and $\{8,4,2,1\}$ for DEF. Thus, stronger key/value reduction was applied at high-resolution stages to control the attention cost, whereas lower-resolution stages retained more key/value locations. The DEF per-head dimension was fixed at 16 as a compact implementation choice. The same CCM and DEF configuration was used for both datasets.

The RGB image was used as a three-channel appearance input, while the single-channel depth map was expanded to three channels to match the input structure of the depth encoder. During training, random scale resizing, random horizontal flipping, and random cropping were applied. The random scale was sampled from $\{0.5,0.75,1.0,1.25,1.5,1.75\}$, after which each sample was cropped or padded to $480\times 640$. During testing, the samples were processed at the same spatial resolution. RGB images were normalized using the ImageNet mean and standard deviation. For the depth modality, the depth map was first scaled by $1/255$ and then normalized using mean values of $(0.48,0.48,0.48)$ and standard deviation values of $(0.28,0.28,0.28)$. The network was trained using the standard cross-entropy loss with label 0 ignored, without auxiliary supervision.

The archived metric exports used for the three-run summaries did not retain a common global seed across Python 3.9, NumPy 1.26.4, and PyTorch 2.8.0; consequently, no seed is assigned retrospectively to these reported runs. They are treated as three independent stochastic training runs, and the sample standard deviations summarize their observed run-to-run variability. In SPGNet, the PVT-v2-B2 RGB encoder and ResNet34 depth encoder were initialized through the pretrained weights provided by their  timm  constructors. Newly introduced convolutional, linear, and normalization layers retained the initialization applied by their PyTorch or  timm  constructors, except for the CCM prior-modulation coefficients ${\alpha }_r$, ${\alpha }_d$ and residual coefficients ${\gamma }_r$, ${\gamma }_d$, which were explicitly initialized to zero as described in Section 3.2.

The reproduced comparison methods include ACNet [27], SA-Gate [9], PrimKD [15], SGACNet [37], ShapeConv [11], and AsymFormer [13]. For each dataset, the proposed method and all reproduced baselines were evaluated using the same input resolution, dataset-specific label-processing procedure, training length, and evaluation code.

3.8. Evaluation Metrics

This paper evaluates RGB-D semantic segmentation performance using six metrics: mean Intersection over Union (mIoU), mean F1-score, mean precision, mean recall, Overall Accuracy (OA), and Cohen’s Kappa. mIoU measures the overlap between predicted and ground-truth regions at the class level. Mean precision reflects the average correctness of class-specific positive predictions, mean recall reflects the average coverage of ground-truth pixels across semantic classes, and the mean F1-score balances the two quantities at the class level. OA measures the proportion of correctly classified evaluated pixels, while Kappa further accounts for chance agreement between predictions and ground-truth annotations.

During evaluation, the segmentation head produces $K=41$ logits for NYU Depth V2 (labels 0–40) and $K=38$ logits for SUN RGB-D (labels 0–37). Pixels with label 0, representing void or padded regions, are removed before confusion-matrix accumulation and are ignored by the cross-entropy loss. For each valid class $c\in \{1,\dots ,K-1\}$, IoU, precision, recall, and F1 are computed from the confusion matrix; mIoU, mean precision, mean recall, and mean F1 are uniformly macro-averaged over labels 1–40 for NYU Depth V2 and labels 1–37 for SUN RGB-D. Masks are resized using nearest-neighbor interpolation and undergo the same spatial transformations as their paired RGB and depth inputs. OA and Kappa are computed from the same confusion matrix over the evaluated non-ignored pixels.

NYU Depth V2 and SUN RGB-D contain 40 and 37 valid categories, respectively, under different processed-label protocols. This study uses the complete-class macro averages above as its primary class-balanced comparison and does not claim that SPGNet improves every individual category or that its gains are concentrated in a predefined object-size group. The archived results contain the aggregate metrics reported in Table 2 and Table 3, but not the per-run class-wise confusion matrices or full-resolution predictions required to reconstruct a consistent complete per-class comparison. Moreover, per-class IoU alone does not identify small- versus large-object behavior because the same semantic category can occupy substantially different image areas. A complete class-wise comparison and instance-scale stratification would therefore require renewed inference under a predefined common protocol and are left as limitations of the present evaluation.

Table 2. Quantitative comparison on NYU Depth V2 under the NYU40 processed-label protocol. All metrics are reported as the mean ± sample standard deviation over three independent training runs. mIoU, F1, and OA denote mean Intersection over Union, mean F1-score, and Overall Accuracy, respectively. The best results are in bold.

Method	mIoU (%)	F1 (%)	Precision (%)	Recall (%)	OA (%)	Kappa (%)
ACNet [27]	$44.318\pm 0.382$	$58.985\pm 0.374$	$61.961\pm 1.665$	$58.827\pm 0.689$	$72.897\pm 0.652$	$69.722\pm 0.633$
SA-Gate [9]	$47.278\pm 0.251$	$61.956\pm 0.039$	$64.472\pm 0.958$	$61.605\pm 0.833$	$74.530\pm 0.673$	$71.648\pm 0.706$
PrimKD [15]	$48.461\pm 0.153$	$62.933\pm 0.263$	$65.212\pm 0.224$	$62.438\pm 0.407$	$75.528\pm 0.376$	$72.693\pm 0.391$
SGACNet [37]	$45.218\pm 0.174$	$59.645\pm 0.151$	$63.922\pm 1.021$	$58.434\pm 1.084$	$73.361\pm 0.261$	$70.258\pm 0.246$
ShapeConv [11]	$47.209\pm 0.190$	$61.480\pm 0.220$	$64.600\pm 0.366$	$60.261\pm 0.232$	$74.824\pm 0.768$	$71.944\pm 0.800$
AsymFormer [13]	$48.734\pm 0.210$	$63.304\pm 0.319$	$65.242\pm 0.057$	$62.767\pm 0.668$	$75.127\pm 0.492$	$72.303\pm 0.506$
SPGNet (Ours)	$\mathbf{50.845}\pm \mathbf{0.126}$	$\mathbf{64.985}\pm \mathbf{0.139}$	$\mathbf{68.390}\pm \mathbf{1.331}$	$\mathbf{64.117}\pm \mathbf{0.326}$	$\mathbf{76.497}\pm \mathbf{0.126}$	$\mathbf{73.956}\pm \mathbf{0.073}$

Table 3. Quantitative comparison on SUN RGB-D under the 37-class processed-label protocol. All metrics are reported as the mean ± sample standard deviation over three independent training runs. mIoU, F1, and OA denote mean Intersection over Union, mean F1-score, and Overall Accuracy, respectively. The best results are in bold.

Method	mIoU (%)	F1 (%)	Precision (%)	Recall (%)	OA (%)	Kappa (%)
ACNet [27]	$43.946\pm 0.230$	$58.039\pm 0.152$	$61.135\pm 0.957$	$56.229\pm 0.428$	$79.098\pm 0.112$	$75.381\pm 0.130$
PrimKD [15]	$46.035\pm 0.111$	$60.102\pm 0.316$	$62.635\pm 1.058$	$58.675\pm 0.659$	$80.249\pm 0.173$	$76.770\pm 0.149$
SA-Gate [9]	$45.112\pm 0.125$	$59.370\pm 0.085$	$61.468\pm 0.191$	$58.510\pm 0.271$	$79.874\pm 0.195$	$76.333\pm 0.207$
SGACNet [37]	$44.269\pm 0.158$	$58.256\pm 0.382$	$61.897\pm 1.256$	$57.361\pm 0.931$	$79.312\pm 0.139$	$75.703\pm 0.123$
ShapeConv [11]	$45.385\pm 0.149$	$59.506\pm 0.553$	$63.318\pm 0.554$	$56.905\pm 0.734$	$79.916\pm 0.340$	$76.308\pm 0.411$
AsymFormer [13]	$47.558\pm 0.268$	$61.533\pm 0.313$	$64.139\pm 0.721$	$60.165\pm 0.089$	$80.640\pm 0.133$	$77.273\pm 0.159$
SPGNet (Ours)	$\mathbf{48.457}\pm \mathbf{0.148}$	$\mathbf{62.496}\pm \mathbf{0.076}$	$\mathbf{64.362}\pm \mathbf{0.168}$	$\mathbf{61.257}\pm \mathbf{0.259}$	$\mathbf{81.183}\pm \mathbf{0.149}$	$\mathbf{77.900}\pm \mathbf{0.175}$

Let $M\in {\mathbb{R}}^{K\times K}$ denote the confusion matrix accumulated over the complete test set, where rows correspond to predicted labels and columns correspond to ground-truth labels. For each valid class c, the true-positive, false-positive, and false-negative counts are $T{P}_c=M_{cc}$, $F{P}_c={\sum }_{j\ne c}{M}_{cj}$, and $F{N}_c={\sum }_{i\ne c}{M}_{ic}$, respectively. Class-wise precision, recall, and F1 are then computed as

$$P_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c+ϵ}},R_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c+ϵ}},F{1}_c={\displaystyle \frac{2P_c{R}_c}{P_c+R_c+ϵ}},$$

(41)

where $ϵ={10}^{-10}$ prevents division by zero. Consistent with the implementation, mean precision, mean recall, and mean F1 are macro-averaged over the $K-1$ valid semantic classes:

$$\overline{P}={\displaystyle \frac{1}{K-1}}{\displaystyle \sum _{c=1}^{K-1}}{P}_c,\overline{R}={\displaystyle \frac{1}{K-1}}{\displaystyle \sum _{c=1}^{K-1}}{R}_c,\overline{F1}={\displaystyle \frac{1}{K-1}}{\displaystyle \sum _{c=1}^{K-1}}F{1}_c.$$

(42)

4. Results and Discussion

Figure 5 reports the unsmoothed per-epoch training and validation losses of SPGNet over the complete 100-epoch schedule. On NYU Depth V2, the training loss decreases from 3.656 to 1.764, while the validation loss decreases from 3.540 to 1.851. On SUN RGB-D, the corresponding losses decrease from 3.135 to 1.063 and from 2.686 to 1.300, respectively. For both datasets, the losses decline rapidly during the early epochs and then decrease more gradually, while the validation curves remain stable during the later stage without sustained divergence. These trends indicate stable optimization convergence under the adopted training protocol.

Figure 5. Unsmoothed per-epoch training and validation loss curves of SPGNet over 100 epochs on (a) NYU Depth V2 and (b) SUN RGB-D.

4.1. Results on NYU Depth V2

Table 2 presents the quantitative comparison between the proposed method and representative RGB-D semantic segmentation methods on NYU Depth V2 under the unified training and evaluation protocol. Across three independent runs, SPGNet obtains the highest mIoU, F1, mean precision, mean recall, OA, and Kappa values of $50.845\pm 0.126$%, $64.985\pm 0.139$%, $68.390\pm 1.331$%, $64.117\pm 0.326$%, $76.497\pm 0.126$%, and $73.956\pm 0.073$%, respectively. Compared with AsymFormer [13], the second-best method for mIoU and F1, SPGNet improves mean mIoU and mean F1 by 2.111 and 1.681 percentage points. For OA and Kappa, PrimKD [15] gives the second-best three-run means, and SPGNet exceeds it by 0.969 and 1.263 percentage points, respectively.

The gains are also consistent when compared with other convolutional or gating-based RGB-D fusion methods. Using the three-run means, SPGNet improves mIoU over ShapeConv [11], SA-Gate [9], SGACNet [37], and ACNet [27] by 3.635, 3.566, 5.627, and 6.526 percentage points, respectively. These results suggest that applying structure-prior-guided correction before dual-branch enhancement fusion is beneficial under this evaluation setting, while the magnitude of the improvement remains moderate against the strongest transformer-based baseline.

For the additional component metrics, SPGNet achieves the highest three-run mean precision and mean recall of 68.390% and 64.117%, respectively. Compared with AsymFormer, the strongest baseline on these two metrics, SPGNet improves mean precision by 3.149 percentage points and mean recall by 1.350 percentage points. The simultaneous improvements in precision and recall are consistent with the higher mean F1-score obtained by SPGNet on NYU Depth V2.

4.2. Results on SUN RGB-D

To further evaluate the proposed method on a larger indoor scene benchmark, this paper reports comparative results on SUN RGB-D. Compared with NYU Depth V2, SUN RGB-D contains more samples from heterogeneous sources and therefore introduces greater variation in scene layout, sensor type, and depth quality. Table 3 summarizes the quantitative results under the corresponding SUN RGB-D processed-label protocol and training pipeline.

Across three independent runs, SPGNet achieves the highest mean mIoU and F1 values of $48.457\pm 0.148$% and $62.496\pm 0.076$%, respectively. Compared with AsymFormer [13], the second-best method on these metrics, SPGNet improves mean mIoU and mean F1 by 0.899 and 0.963 percentage points. Compared with ShapeConv [11], the corresponding improvements are 3.072 and 2.991 points. Its mean mIoU improvements over PrimKD [15], SA-Gate [9], SGACNet [37], and ACNet [27] are 2.422, 3.345, 4.188, and 4.511 percentage points, respectively. SPGNet also obtains the highest three-run mean OA and Kappa values of 81.183% and 77.900%, exceeding AsymFormer by 0.544 and 0.628 percentage points, respectively.

For the additional component metrics, SPGNet achieves the highest three-run mean precision and mean recall of 64.362% and 61.257%, respectively. Compared with AsymFormer, SPGNet improves mean precision by 0.223 percentage points and mean recall by 1.092 percentage points. These simultaneous gains are consistent with the higher mean F1-score obtained by SPGNet on SUN RGB-D.

To assess statistical significance, two-sided Welch tests were applied to the three independent runs of SPGNet and AsymFormer. On NYU Depth V2, the mIoU difference of 2.111 percentage points has a $95\%$ confidence interval of 1.682–2.540 ($p<0.001$), and the F1 difference of 1.681 points has a $95\%$ confidence interval of 1.006–2.356 ($p=0.005$). On SUN RGB-D, the corresponding differences are 0.899 points in mIoU ($95\%$ confidence interval: 0.347–1.450; $p=0.013$) and 0.963 points in F1 ($95\%$ confidence interval: 0.237–1.690; $p=0.028$). Thus, the advantage over the strongest reproduced baseline is retained across the three runs on both datasets. Given the small number of runs, these inferential results are interpreted together with the effect sizes and run-wise variability.

The SUN RGB-D results follow the same trend as the NYU Depth V2 results: SPGNet consistently improves over the reproduced baselines, while the margin over the strongest transformer-based baseline remains relatively small. This observation suggests that the proposed correction-and-fusion design is useful across the two evaluated indoor RGB-D benchmarks, but the results should be interpreted together with the following model-scale analysis.

Figure 6 and Figure 7 present qualitative comparisons on NYU Depth V2 and SUN RGB-D, respectively.

Figure 6. Qualitative comparison on NYU Depth V2. From left to right, the columns show the RGB image, depth map, ground truth, ShapeConv, AsymFormer, and SPGNet predictions.

Figure 7. Qualitative comparison on SUN RGB-D. From left to right, the columns show the RGB image, depth map, ground truth, ShapeConv, AsymFormer, and SPGNet predictions.

In the first NYU Depth V2 example, ShapeConv produces fragmented labels on the tall cabinet and around the bed–floor transition, while AsymFormer introduces a small isolated false-positive region above the furniture. SPGNet yields a more contiguous cabinet region and a cleaner bed contour against the wall and floor. In the second and fourth examples, the bed occupies a large portion of the scene and is surrounded by pillows, nearby furniture, and weak wall–floor depth transitions. SPGNet preserves a more complete bed region with less label leakage into the surrounding background. In the dining-room example, the proposed model also maintains a coherent tabletop region and separates much of the adjacent furniture, although thin chair structures and small objects remain difficult for all three methods.

A similar pattern is visible on SUN RGB-D. In the bathroom and bedroom examples, SPGNet better preserves the dominant object extent at the bathtub–wall and bed–background interfaces, where the depth maps contain sharp transitions together with locally noisy or weak responses. In the living-room example, the sofa region remains more spatially coherent, whereas the comparison predictions contain more local boundary irregularities. The final doorway scene illustrates a harder case involving narrow foreground structures and partial occlusion: SPGNet recovers more of the small foreground regions but still exhibits errors near ambiguous boundaries. These observations are consistent with the intended roles of CCM and DEF: CCM can reduce the influence of locally inconsistent cross-modal responses before fusion, while DEF can combine the corrected modalities to refine object extent and boundaries. However, the end-to-end visual comparisons alone do not isolate the causal contribution of either module; that evidence is provided by the controlled module ablation in Section 4.4.

Representative Failure Cases. To complement the qualitative comparisons above, Figure 8 presents two representative failure cases that illustrate recurring error mechanisms under challenging indoor conditions.

Figure 8. Representative failure cases of SPGNet. From left to right, the columns show the RGB image, ground-truth annotation, and SPGNet prediction. Black pixels in the ground truth correspond to ignore index 0 and are excluded from the loss and quantitative evaluation. The top row illustrates a cluttered scene with extensive ignored regions, dense small objects, and occlusion; the bottom row illustrates boundary displacement and loss of thin structures in a dim bathroom scene.

The first case contains densely arranged furniture, small objects, and substantial occlusion. A large part of the visually complex background is assigned to the ignored category in the reference annotation, whereas SPGNet predicts semantic labels for several visible scene elements in that area. Because predictions at ignored pixels are excluded from the quantitative metrics, these colored regions should not be interpreted directly as segmentation errors. Within the labeled regions, however, the prediction remains locally fragmented around overlapping tables, chairs, and background furnishings. This example indicates that densely packed small structures, severe occlusion, and ambiguous interfaces can still cause confusion between adjacent object categories, while also demonstrating how incomplete annotations can complicate qualitative interpretation.

In the second case, SPGNet recovers the dominant semantic regions of the bathroom but loses several fine structural details. The predicted cyan region extends irregularly across the upper boundary of the dark-blue bathtub region, producing local boundary displacement and label leakage. Thin structures in the reference annotation, including the narrow fixture and bathtub rim, are omitted or absorbed into adjacent regions, and the object interfaces on the right are oversmoothed. Weak illumination, reflective or transparent surfaces, and thin metallic structures provide ambiguous appearance and geometric cues. When both modalities contain weak or inconsistent boundary evidence, the learned structure prior may therefore fail to preserve narrow structures or regulate cross-modal transfer accurately near object interfaces. These observations are consistent with the limitations discussed in Section 5 and motivate boundary-aware supervision, explicit uncertainty modeling, and broader failure analysis on held-out data.

The boundary observations in Figure 6, Figure 7 and Figure 8 are qualitative and should not be interpreted as a quantitative boundary-specific advantage. Boundary IoU and Boundary F1 depend on the boundary-extraction operator, matching tolerance or band width, treatment of ignored pixels, and normalization with respect to image resolution. These choices are particularly consequential for annotations containing coarse interfaces or extensive ignored regions. The archived evaluations do not retain full-resolution predictions for all reproduced methods; therefore, a controlled boundary comparison would require renewed inference for every method after fixing one common extraction, tolerance, ignored-pixel, and resolution-normalization protocol. This additional experiment is not included in the present revision, which retains the standard benchmark metrics and limits its boundary claims accordingly.

Visualization of the Learned Structure Prior. To examine the intermediate representation used by CCM, Figure 9 visualizes representative Stage 1 structure-prior maps on NYU Depth V2.

Figure 9. Visualization of learned Stage 1 structure priors generated by the Cross-Modal Correction Module (CCM) on NYU Depth V2. Each row shows the RGB image, the corresponding depth input, and the learned structure-prior response. For visualization, the Stage 1 prior tensor is averaged across channels, bilinearly upsampled to the input resolution, independently linearly scaled between its 1st and 99th percentiles, and shown without spatial smoothing. Brighter colors indicate stronger relative responses within the same map and should not be interpreted as absolute values comparable across samples.

The displayed response maps exhibit spatially organized patterns around prominent scene structures, including the piano and bench contours in the first example, the closet opening and nearby object boundaries in the second, and the shelf–aisle layout in the third. These responses are not identical to either raw RGB texture or depth intensity; instead, they summarize the learned prior constructed from joint consistency and discrepancy cues in the two encoded modalities. The maps therefore provide qualitative evidence that CCM forms a structure-sensitive intermediate representation before feature fusion. This visualization alone does not prove suppression of unreliable information, but, together with the controlled CCM ablation reported in Section 4.4, it supports the interpretation that the learned prior supplies spatial guidance for regulating subsequent cross-modal transfer.

4.3. Robustness Analysis

To isolate robustness to unreliable depth, we additionally evaluated the compared methods on the full NYU Depth V2 test set while keeping the RGB images and ground-truth labels unchanged. Only the normalized depth input was perturbed. Zero-mean Gaussian noise with standard deviations of $0.05$, $0.10$, and $0.20$ was added to the depth tensor. For the misregistration test, only the depth map was translated by 1, 3, or 5 pixels along a cardinal direction selected independently for each test sample, with replicated boundary values. One fixed trained checkpoint per method was evaluated once for every condition; no retraining or test-time adaptation was performed. Table 4 reports mIoU, and the values in parentheses denote the absolute change in percentage points relative to the clean result from the same checkpoint. Because the archived robustness outputs did not retain the random perturbation seed, this table should be interpreted as a single-realization sensitivity analysis rather than a variance estimate over perturbation realizations.

Table 4. Depth-only robustness comparison on NYU Depth V2. RGB images and labels remain unchanged, while only depth is corrupted by Gaussian noise or spatially shifted. Parentheses indicate the mIoU change from the clean result of the same fixed checkpoint; the best score under each condition is in bold.

Perturbation	Level	mIoU (%)
		ACNet	SA-Gate	PrimKD	SGACNet	ShapeConv	AsymFormer	SPGNet
Clean	–	44.35	47.27	48.54	45.27	47.02	48.68	50.78
Gaussian noise	$\sigma =0.05$	44.17 (−0.18)	46.92 (−0.35)	48.21 (−0.33)	45.00 (−0.27)	46.84 (−0.18)	48.67 (−0.01)	50.22 (−0.56)
	$\sigma =0.10$	43.89 (−0.46)	46.60 (−0.67)	47.95 (−0.59)	44.68 (−0.59)	46.08 (−0.94)	48.64 (−0.04)	49.41 (−1.37)
	$\sigma =0.20$	43.41 (−0.94)	45.83 (−1.44)	47.33 (−1.21)	44.39 (−0.88)	43.10 (−3.92)	48.52 (−0.16)	48.64 (−2.14)
Misalignment	1 pixel	44.38 (+0.03)	47.23 (−0.04)	48.53 (−0.01)	45.26 (−0.01)	46.98 (−0.04)	48.68 (+0.00)	50.79 (+0.01)
	3 pixels	44.35 (+0.00)	47.18 (−0.09)	48.52 (−0.02)	45.25 (−0.02)	46.90 (−0.12)	48.66 (−0.02)	50.78 (+0.00)
	5 pixels	44.28 (−0.07)	47.17 (−0.10)	48.43 (−0.11)	45.18 (−0.09)	46.79 (−0.23)	48.63 (−0.05)	50.78 (+0.00)

SPGNet retains the highest absolute mIoU under every evaluated depth perturbation. Its mIoU decreases from 50.78% on clean inputs to 50.22%, 49.41%, and 48.64% as the depth-noise level increases, remaining 0.12 percentage points above AsymFormer at $\sigma =0.20$. Under depth-only shifts of 1, 3, and 5 pixels, SPGNet obtains 50.79%, 50.78%, and 50.78% mIoU, respectively, and exceeds AsymFormer by 2.15 points at 5 pixels. These absolute results show that SPGNet preserves the strongest segmentation performance across the tested depth corruptions. Nevertheless, AsymFormer exhibits smaller relative decreases under the stronger noise settings; therefore, the evidence supports the highest retained absolute accuracy for SPGNet rather than the smallest degradation in every condition. Large contiguous missing-depth regions remain outside the scope of this experiment.

4.4. Ablation Study

To evaluate the contribution of each key component, this paper conducts step-by-step ablation experiments on SUN RGB-D under the unified training setting. All ablation settings use the same heterogeneous dual-branch encoder and differ only in whether CCM and DEF are enabled. The compared settings include Baseline, which uses neither CCM nor DEF; Baseline+CCM, which introduces only the cross-modal correction module; Baseline+DEF, which introduces only the dual-branch enhancement fusion module; and Full, which uses both CCM and DEF. In particular, Baseline+DEF is the fusion-only configuration: DEF remains enabled while CCM is removed, so the dual-modal features are enhanced and fused without preceding structure-prior-guided correction. The PVT-v2-B2 RGB encoder, ResNet34 depth encoder, multi-scale decoder, training schedule, and evaluation protocol are kept fixed across all four configurations. Therefore, the observed performance differences isolate the contributions of CCM and DEF rather than changes in the feature extractors. The quantitative results are reported in Table 5, and a representative visual comparison is shown in Figure 10.

Table 5. Ablation study on SUN RGB-D under the 37-class processed-label protocol.

Setting	CCM	DEF	mIoU (%)	F1 (%)	OA (%)	Kappa (%)
Baseline	–	–	45.395	59.309	79.749	76.239
Baseline+CCM	✓	–	46.375	60.271	80.183	76.755
Baseline+DEF	–	✓	45.817	59.651	79.483	76.009
SPGNet	✓	✓	48.418	62.418	81.217	77.953

Note: Bold values indicate the best results.

Figure 10. Visual ablation comparison on SUN RGB-D. The panels show the RGB image, ground truth, Baseline, Baseline+CCM, Baseline+DEF, and the full SPGNet prediction.

Adding CCM alone improves mIoU from 45.395% to 46.375% (+0.980) and F1 from 59.309% to 60.271% (+0.962). Adding DEF alone improves mIoU to 45.817% (+0.422) and F1 to 59.651% (+0.342), although its OA and Kappa are slightly lower than those of the Baseline. Thus, DEF alone mainly improves overlap- and F1-related metrics on SUN RGB-D, while its effect on pixel-level accuracy is weaker when CCM is absent.

With both modules enabled, the full model achieves 48.418% mIoU, 62.418% F1, 81.217% OA, and 77.953% Kappa. Compared with the Baseline, the gains are 3.023, 3.109, 1.468, and 1.714 percentage points, respectively. More directly, comparing the full model with the fusion-only Baseline+DEF configuration measures the incremental benefit of adding CCM before DEF under otherwise identical settings. Adding pre-fusion correction increases mIoU from 45.817% to 48.418% (+2.601), F1 from 59.651% to 62.418% (+2.767), OA from 79.483% to 81.217% (+1.734), and Kappa from 76.009% to 77.953% (+1.944). On SUN RGB-D under the unified protocol, this controlled comparison shows that applying CCM before DEF improves final segmentation performance relative to the fusion-only configuration within the evaluated architecture. The full model outperforms both single-module variants on all four metrics, indicating that cross-modal correction and dual-branch enhancement fusion provide complementary contributions when used jointly. The visual example in Figure 10 further illustrates that the full model produces more complete object regions than the Baseline in this SUN RGB-D scene.

4.5. Model Complexity Analysis

To better characterize the trade-off between accuracy and model scale across different methods, this section re-profiles the RGB-D semantic segmentation methods appearing in the main quantitative comparisons using the local profiling framework, as shown in Table 6. All models are evaluated in inference mode with batch size 1 at $480\times 640$, using three-channel RGB images and one-channel depth maps; the depth map is repeated internally only when a model interface requires three channels. Parameter counts are computed directly from the instantiated models, and computational costs are reported as FLOPs under the common convention that one multiply–accumulate operation corresponds to two floating-point operations. For SPGNet, no layer is frozen during training and all model parameters are passed to the AdamW optimizer; therefore, its reported parameter count is also its trainable parameter count. End-to-end latency and throughput for every method are measured on an NVIDIA GeForce RTX 5070 Ti GPU under this common input protocol. CUDA synchronization is applied before and after each timed measurement, and timing begins only after 50 warm-up iterations. For each method, the reported latency is averaged over 1500 timed forward passes from three independent measurement runs, and frames per second (FPS) is computed as the reciprocal of the mean per-image latency. Timing covers the model forward pass to full-resolution logits and excludes model construction and host-to-device transfer. Peak allocated GPU memory is measured after warm-up by resetting the CUDA peak-memory statistics for each independent run, and the three run-wise peaks are averaged. Since fvcore does not support every operator used by these models, the FLOP values are partial profiling results and are intended for model-scale reference rather than exact hardware-independent complexity comparison. The profiled backbone column reflects the local adapter instantiation used for complexity measurement, while the mIoU columns follow the main quantitative comparison tables.

Table 6. Locally profiled complexity, runtime, and three-run mean Intersection over Union (mIoU) comparison of red–green–blue and depth (RGB-D) semantic segmentation methods appearing in the main quantitative experiments. Floating-point operations (FLOPs) are reported at $480\times 640$; unsupported operators make the FLOP counts partial profiling results. End-to-end latency and frames per second (FPS) are measured for all methods on an NVIDIA GeForce RTX 5070 Ti with batch size 1 under the common input and timing protocol described in the text. All 122.21 M parameters reported for SPGNet are trainable.

Model	Backbone	Params (M)	FLOPs (G)	Latency (ms)	FPS	mIoU (%)
						NYU Depth V2	SUN RGB-D
ACNet [27]	ResNet-50	116.63	116.11	20.45	48.89	44.318	43.946
SA-Gate [9]	ResNet-50	110.87	181.85	25.00	40.01	47.278	45.112
PrimKD [15]	MiT-B4	140.26	158.96	58.46	17.11	48.461	46.035
SGACNet [37]	R34-NBt1D	30.66	33.74	27.39	36.51	45.218	44.269
ShapeConv [11]	ResNeXt-101	109.98	169.16	28.25	35.40	47.209	45.385
AsymFormer [13]	official	33.06	36.22	13.39	74.66	48.734	47.558
SPGNet (Ours)	PVT-v2-B2 + ResNet34	122.21	99.91	34.64	28.87	50.845	48.457

Note: Bold values indicate the highest mIoU on each dataset.

In terms of parameter count, SPGNet contains a heterogeneous encoder, four-stage CCM, four-stage DEF, and a multi-scale fusion decoder, resulting in 122.21 M trainable parameters. This places it in the relatively large-model group among the locally instantiated models in this table. Its parameter count is higher than those of ACNet, SA-Gate, SGACNet, ShapeConv, and AsymFormer, but lower than that of PrimKD with MiT-B4 under the current local profiling setup.

In terms of computational cost, SPGNet requires 99.91 G partially profiled FLOPs under the local setup. Because unsupported operators differ across architectures, these FLOP values are provided as implementation-specific diagnostics and are not used for strict cross-model ranking. The common runtime measurements show that SPGNet requires 34.64 ms per image (28.87 FPS). It is slower than ACNet, SA-Gate, SGACNet, ShapeConv, and AsymFormer, but faster than PrimKD under the tested hardware and implementation environment. These results provide a hardware-specific indication of practical deployment cost. Combined with the accuracy results in Table 2 and Table 3, SPGNet achieves the highest mIoU on both NYU Depth V2 and SUN RGB-D among the compared methods, while requiring a relatively large parameter count and mid-range measured latency. Therefore, the proposed method should be regarded as an accuracy-oriented model rather than a lightweight design. Future work may further explore lightweight attention projections and channel transformations in CCM and DEF.

To isolate the computational overhead associated with CCM and DEF, the four controlled ablation configurations are also profiled under the same hardware, input resolution, batch size, inference mode, and timing protocol. As reported in Table 7, Baseline+CCM and Baseline+DEF differ from the common Baseline only by the corresponding module, while the full SPGNet enables both modules. This controlled profiling separates module-related changes in parameter count, FLOPs, latency, and peak allocated GPU memory from changes in the encoders, decoder, or input protocol. Runtime and peak-memory values are interpreted per configuration rather than added across rows because kernel execution and memory reuse are not strictly additive.

Table 7. Controlled module-overhead profiling of the Cross-Modal Correction Module (CCM) and Dual-branch Enhancement Fusion Module (DEF). All configurations use identical encoders and decoder and are profiled on an NVIDIA GeForce RTX 5070 Ti with batch size 1 and $3\times 480\times 640$ RGB and repeated-depth inputs.

Configuration	CCM	DEF	Params (M)	FLOPs (G)	Latency (ms)	Peak Memory (GB)
Baseline	–	–	50.82	53.81	14.44	0.45
Baseline+CCM	✓	–	88.13	67.93	23.57	0.69
Baseline+DEF	–	✓	84.91	85.80	24.75	0.66
SPGNet	✓	✓	122.21	99.91	34.64	0.91

5. Conclusions

This paper proposes the Structure-Prior-Guided Network (SPGNet) for RGB-D semantic segmentation, with the aim of improving multimodal feature interaction under cross-modal response inconsistency, unreliable depth-response propagation, and limited multi-scale fusion. SPGNet adopts a heterogeneous dual-encoder architecture to model RGB appearance semantics and depth geometric structures separately, and applies the Cross-Modal Correction Module (CCM) and the Dual-branch Enhancement Fusion Module (DEF) sequentially at four scales. CCM constructs structure priors from RGB–depth consistency and discrepancy cues, and uses these priors for structural modulation, gated selection, and bidirectional cross-modal correction before fusion. DEF further enhances the corrected dual-branch features and performs shared-guided interaction before the multi-scale fusion decoder aggregates features from different levels for pixel-level prediction.

Under the dataset-specific processed-label protocols used in this paper, SPGNet obtains three-run mean mIoU, F1, OA, and Kappa values of 50.845%, 64.985%, 76.497%, and 73.956% on NYU Depth V2 and 48.457%, 62.496%, 81.183%, and 77.900% on SUN RGB-D, respectively. These results are favorable among the compared representative RGB-D segmentation methods in the reported evaluation setting. The ablation experiments further show that, under the same heterogeneous dual-branch encoder, jointly enabling CCM and DEF improves all four metrics on SUN RGB-D, which is consistent with the complementary roles of pre-fusion structural correction and subsequent enhancement fusion. The complexity analysis indicates that these accuracy gains are accompanied by a relatively large parameter count and computational cost, positioning SPGNet as an accuracy-oriented model rather than a lightweight design.

This study has several limitations. First, although the added depth-only Gaussian-noise and spatial-shift experiments evaluate SPGNet under controlled depth corruption while preserving the RGB input, they do not cover large contiguous missing-depth regions. The effects of severe depth dropout and more substantial RGB–depth misregistration therefore require further study. Second, the current experiments are limited to indoor RGB-D datasets; therefore, generalization to outdoor scenes with different sensing ranges, illumination conditions, depth statistics, and class distributions remains unverified. Transferability to other spatially aligned multimodal segmentation settings, such as RGB–thermal or RGB–LiDAR segmentation, also remains to be established. Because CCM and DEF operate on paired stage-wise feature maps rather than raw sensor-specific values, the framework could in principle be adapted by replacing the depth encoder and relearning modality-specific structure priors. However, differences in sensor alignment, noise characteristics, and spatial sparsity require task-specific validation. Third, SPGNet is an accuracy-oriented rather than lightweight model. It contains 122.21 M trainable parameters, and Table 6 reports 99.91 G FLOPs for the profiled portion of the model at an input resolution of $480\times 640$. The parameter count is independent of input resolution, whereas the computational and memory costs increase as the resolution grows. For convolutional components with fixed channel dimensions and kernels, the computational cost grows approximately linearly with the number of spatial locations. In CCM and DEF, the query retains the full spatial resolution at each stage, while the key and value maps are average-pooled by a ratio r along both spatial dimensions. Thus, for a stage feature map with $N=h_i{w}_i$ query locations, the key/value sequence contains approximately $N/r^2$ locations, and the cross-attention score computation and score-matrix memory scale as $\mathcal{O}(N^2/r^2)$, assuming fixed channel dimensions. Consequently, if the input height and width are both increased by a factor of s, convolutional cost grows approximately as $s^2$, whereas this attention term grows approximately as $s^4$ for a fixed r. The stage-wise key/value downsampling ratios used in CCM ($\{8,4,4,2\}$) and DEF ($\{8,4,2,1\}$) reduce, but do not eliminate, this resolution-dependent growth. Finally, scenes involving severe occlusion, weak or ambiguous boundaries, reflective or transparent surfaces, or strong RGB–depth misalignment may remain challenging. The diagnostic examples in Figure 8 concretely illustrate fragmentation under clutter and occlusion, as well as boundary displacement and loss of thin structures under weak and reflective visual conditions. Although controlled depth noise and shift perturbations are now evaluated, these individual scene and sensor failure modes still require more systematic investigation on held-out data. Complete per-class behavior, instance-scale-stratified performance, and boundary-focused accuracy under a predefined extraction and tolerance protocol also remain to be evaluated.

Future work will focus on three directions. First, the parameter overhead introduced by attention projections and multi-channel convolutions in CCM and DEF can be reduced to improve deployment feasibility in real-time or resource-constrained scenarios. This direction will include evaluating lightweight convolutional networks and hierarchical Transformer architectures as alternative depth encoders to characterize the accuracy–efficiency trade-off and the generality of the heterogeneous encoder design. Second, robustness to controlled levels of depth degradation and the generalization ability of structure-prior-guided correction can be evaluated on more indoor and outdoor RGB-D datasets, cross-domain settings, and other spatially aligned multimodal segmentation tasks. Finally, more fine-grained visualization and error-case analyses can be incorporated to better explain the roles of structure priors, gated selection, and enhancement fusion for different scene categories and boundary regions. This direction will use a predefined protocol to report complete per-class results, stratify performance by instance scale and boundary difficulty, and quantify boundary accuracy under explicitly stated extraction operators and tolerance widths.