A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation

Abstract

Off-road semantic segmentation is challenging due to irregular terrain, vegetation clutter, class-level similarity, and ambiguous boundary annotations. Existing decoder designs often rely on compact bottlenecks that oversmooth fine structures or repeated multi-scale fusion that can amplify annotation noise and increase computational cost. To address these limitations, we propose a Cross-Scale Decoder for robust off-road semantic segmentation. The proposed decoder first stabilizes semantic representations through Global–Local Token Refinement (GLTR) on a compact bottleneck lattice. It then selectively incorporates fine-scale structural cues using Boundary-Guided Correction (BGC) and Gated Cross-Scale Interaction (GCS), avoiding dense and repeated feature fusion. In addition, uncertainty-guided class-aware point refinement focuses computation on ambiguous and low-confidence regions. Experiments on standard off-road benchmarks demonstrate that the proposed method improves segmentation accuracy and boundary consistency over existing approaches while maintaining practical inference efficiency.

1. Introduction

Off-road semantic segmentation presents challenges that differ fundamentally from perception in structured urban environments. Irregular terrain, weak spatial organization, and inherently ambiguous semantic boundaries lead to annotations that are thick, inconsistent, and noisy, particularly around class transitions [1,2,3,4,5]. Vegetation clutter and appearance variations further exacerbate boundary ambiguity, making stable and boundary-coherent learning substantially more difficult than in urban scenes.

A key difficulty arises from strong class-level similarity. Many terrain categories share overlapping visual cues in color, texture, and scale, rendering both class identity and boundaries ambiguous in practice. Thin or rare structures, such as narrow traversable gaps or small obstacles, are often occluded and sparsely annotated, and even human annotators may disagree on precise boundaries, resulting in intrinsically weak and inconsistent supervision [6,7,8].

Robust off-road semantic segmentation is therefore critical for field robotics, where semantic perception directly informs downstream navigation and safety decisions under extreme class imbalance and sparse supervision [6,7,9]. In such settings, errors near ambiguous boundaries or rare but safety-critical structures can propagate into unsafe traversal behavior, making robustness to label uncertainty a system-level requirement. Figure 1 illustrates representative off-road scene challenges, including irregular terrain, ambiguous boundaries, strong light reflections, and severe vegetation occlusion.

Despite recent advances, most existing segmentation architectures remain ill-suited to these conditions. Transformer-based and hybrid designs commonly aggregate multi-scale features into compact low-resolution bottlenecks, discarding fine-scale structural cues early in the network [10,11,12]. While dense multi-scale fusion can partially recover lost details, it often amplifies annotation noise, increases computational cost, and introduces train–test mismatches when refinement branches are used only during training [13,14]. Recent foundation segmentation models, such as Segment Anything Model (SAM)-based approaches, demonstrate strong generalization for visually well-defined object boundaries; however, their dense mask generation and boundary sharpening assumptions are often misaligned with off-road scenes dominated by thick, ambiguous transitions and annotation noise [15,16,17].

These limitations reveal a structural mismatch between existing decoder designs and the requirements of off-road perception. Robust segmentation under ambiguous supervision requires explicitly separating semantic consolidation from boundary-guided structural correction, rather than entangling both through repeated dense fusion [18,19,20].

To address this, we propose a Cross-Scale Decoder with Token Refinement (CSTR) for off-road semantic segmentation, designed to separate semantic consolidation from selective structural correction guided by boundary cues. Global–Local Token Refinement (GLTR) is first applied on a compact bottleneck lattice to stabilize global semantic representations under ambiguous supervision. Subsequently, Boundary-Guided Correction (BGC) extracts fine-scale structural cues, which are selectively consulted through a Gated Cross-Scale Interaction (GCS) to mediate semantic–structural information exchange without dense fusion. Within this interaction, uncertainty-guided, class-aware point-wise correction is applied to focus computation on a small set of low-confidence pixels after semantic stabilization and structural interaction. Together, these components preserve boundary geometry and thin structures while maintaining robustness and deployment efficiency in off-road robotic perception.

The novelty of CSTR lies not in treating attention, gating, or point-wise refinement as isolated modules, but in organizing them into a decoupled semantic–structural decoder tailored to off-road segmentation. By first stabilizing a compact semantic bottleneck and then applying a single uncertainty-guided structural correction, CSTR avoids repeated dense fusion that can amplify noisy boundary annotations. This design directly reflects the characteristics of off-road scenes, where terrain transitions are often thick, noisy, and semantically ambiguous rather than crisp object boundaries.

The main contributions of this work are summarized as follows:

• We propose CSTR, which is designed to separate semantic consolidation from selective structural correction guided by boundary cues, enabling robust off-road semantic segmentation under ambiguous and noisy supervision without relying on dense multi-scale fusion.
• We introduce GLTR, which stabilizes semantic representations on a compact bottleneck lattice through global attention and lightweight local refinement, supported by a boundary-band group attention regularizer for noise-robust semantic consolidation near ambiguous transition regions.
• We realize boundary-guided correction through BGC and GCS, where BGC extracts fine-scale structural cues and GCS selectively integrates them with uncertainty-guided, class-aware point-wise correction to recover rare and ambiguous structures while maintaining practical inference efficiency.

2. Related Work

2.1. Off-Road Semantic Segmentation

Semantic segmentation in off-road environments has gained increasing attention due to its importance in field robotics. Unlike urban datasets with well-defined object boundaries and structured layouts, off-road datasets exhibit irregular terrain, vegetation clutter, and significant annotation ambiguity. Class boundaries are often thick, inconsistent, and visually ambiguous, particularly around transitions between similar terrain types [1,2,6,7].

This ambiguity reflects the intrinsic uncertainty of natural environments rather than simple annotation errors. Terrain categories frequently share overlapping visual characteristics, and fine-scale structures are often occluded or sparsely represented. As a result, dense and fully consistent pixel-level ground truth is difficult to obtain in practice, posing fundamental challenges to learning stable and boundary-coherent representations in off-road scenes [6,8,21].

Recent off-road segmentation studies have therefore focused on improving robustness to terrain appearance variation, class imbalance, and weak supervision. However, many methods still inherit decoder structures designed for structured urban scenes, where object boundaries are relatively clearer and semantic categories are more separable. This limits their ability to handle gradual terrain transitions, vegetation-induced occlusions, and rare but safety-critical structures in unstructured outdoor environments.

2.2. Multi-Scale Decoder Architectures

Recent advances in semantic segmentation have been driven by transformer-based and hybrid architectures that aggregate multi-scale features through attention mechanisms [10,11,22,23,24]. For computational efficiency, many designs compress feature representations into compact low-resolution bottlenecks before global reasoning. While effective for capturing long-range context, this strategy discards fine-scale structural cues early in the network [10,11,12].

Once thin structures and local discontinuities are suppressed at the bottleneck stage, they are difficult to recover through downstream refinement. Several approaches attempt to address this limitation through dense multi-scale fusion or skip connections [13,14,18,25]. However, repeated fusion often amplifies annotation noise and significantly increases computational cost, limiting suitability for resource-constrained robotic deployment [26,27,28].

This trade-off is particularly important in off-road perception, where fine-scale details are useful but not always reliable. Directly fusing high-resolution features can improve local sharpness, but it may also propagate noisy texture patterns and uncertain boundary labels into semantic predictions. Therefore, decoder design for off-road segmentation should not simply maximize multi-scale fusion, but should control when and how fine-scale cues are introduced.

2.3. Boundary-Aware Refinement

Prior work has explored boundary-aware losses, affinity-based constraints, and post-processing techniques such as conditional random fields to improve boundary quality [29,30,31,32]. While effective at sharpening predictions, these methods typically operate as auxiliary components and do not address the architectural causes of boundary degradation.

More recent approaches adopt sparse or point-based refinement strategies to selectively correct uncertain regions [20,33,34]. However, many such modules are enabled only during training and disabled at inference to reduce latency, introducing train–test mismatches that can lead to unstable boundary behavior under noisy supervision [19].

In off-road scenes, boundary refinement is further complicated by the fact that many transitions are not thin object contours but broad semantic transition bands. Thus, aggressively sharpening boundaries may be inappropriate when the ground truth itself is uncertain. A more suitable strategy is to stabilize semantic representations first and then selectively refine ambiguous regions using structural cues only when they are informative.

Taken together, prior work suggests that improving off-road segmentation does not primarily require increasing model capacity or repeated dense fusion, but rather rethinking how semantic consolidation and structural correction are integrated within the decoder. In environments with intrinsic class-level ambiguity, a robust solution should preserve fine-scale boundary cues through selective, gated interaction while remaining fully active at inference time, motivating the Cross-Scale Decoder proposed in this work.

3. Methods

3.1. Overview

Given an input image I, our method extracts multi-scale features using a transformer backbone and processes them with a Cross-Scale Decoder. Unlike conventional decoders that rely on repeated dense fusion, the proposed design explicitly constrains when and where structural information is introduced, reflecting the intrinsic ambiguity and noisy supervision of off-road environments. The decoder consolidates global semantics at a compact bottleneck while selectively consulting structural cues only when necessary, thereby explicitly separating semantic consolidation from boundary-guided structural correction.

As shown in Figure 2, the decoder consists of three conceptual components: Global–Local Token Refinement (GLTR) for stabilizing semantic representations, Boundary-Guided Correction (BGC) for extracting boundary-relevant structural cues, and a Gated Cross-Scale Interaction (GCS) that selectively integrates these cues and performs sparse point-wise correction. In the GCS block, sparse point-wise refinement selects top-K uncertain locations and gathers point-level logits $Z_p$ and structural features $F_p$ to predict refined point outputs $P_{\mathrm{ref}}$. For clarity, $T_0,T_1,T_2\in {\mathbb{R}}^{C\times H_0\times W_0}$ denote compact semantic tokens, whereas $Z_s\in {\mathbb{R}}^{G\times H_p\times W_p}$ and $F_s\in {\mathbb{R}}^{C\times H_p\times W_p}$ denote the logits and structural features used for point-wise refinement.

3.2. Global–Local Token Refinement (GLTR)

Global–Local Token Refinement (GLTR), shown in the green block of Figure 2, is designed to stabilize semantic representations before structural correction is applied. In off-road scenes, fine-scale textures around vegetation, gravel, mud, and terrain transitions are often visually complex but semantically unreliable. If such local details are introduced too early, noisy boundary responses can be entangled with global semantics. GLTR therefore first consolidates multi-scale context into a compact bottleneck lattice, allowing the decoder to form a stable semantic representation before consulting boundary-guided structural cues.

Given multi-scale backbone features ${\left\{F_l\right\}}_{l=1}^L$, each feature is projected into a shared embedding space and resized to a common bottleneck resolution. The projected features are then aggregated using dynamically estimated scale weights:

$$T_0={\displaystyle \sum _{l=1}^L}{\alpha }_l·{\varphi }_l\left(F_l\right),{\alpha }_l={\displaystyle \frac{exp\left({\mathbf{w}}_{\alpha }^{\top }\mathcal{G}\left(F_l\right)\right)}{{\sum }_{j=1}^{L}exp\left({\mathbf{w}}_{\alpha }^{\top }\mathcal{G}\left(F_j\right)\right)}},$$

(1)

where ${\varphi }_l(·)$ denotes a scale-specific projection implemented by convolution and interpolation, $\mathcal{G}(·)$ denotes global average pooling, and ${\mathbf{w}}_{\alpha }$ is a learnable projection vector for estimating the semantic importance of each scale. As shown in Equation (1), the bottleneck token $T_0\in {\mathbb{R}}^{C\times H_0\times W_0}$ selectively aggregates multi-scale context while suppressing unreliable high-frequency details before boundary-related cues are introduced, thereby serving as the semantic anchor for subsequent attention and refinement stages.

After bottleneck aggregation, GLTR performs class-aware semantic consolidation using scaled dot-product attention [35]. Specifically, the bottleneck token $T_0$ is projected into key and value representations using learnable projections $W_K$ and $W_V$, respectively, and then flattened into token sequences for attention computation. Let $Q_c\in {\mathbb{R}}^{N_{\mathrm{class}}\times d}$ be learnable class prototypes. The class-aware semantic representation is computed as

$$T_1=\mathrm{Softmax}\left({\displaystyle \frac{Q_c{K}^{\top }}{\sqrt{d_k}}}\right)V.$$

(2)

Here, K and V denote the flattened key and value sequences projected from $T_0$, and the softmax term represents the class-aware attention response between the learnable class prototypes and the bottleneck semantic lattice. As defined in Equation (2), each class prototype queries $T_0$ for relevant semantic evidence, which forms the core semantic consolidation step in GLTR. In off-road environments, where different terrain categories often share similar color and texture patterns, this class-aware querying helps suppress irrelevant background responses and improves semantic separability before boundary refinement.

Although the attention operation strengthens global semantic consistency, it may weaken local spatial continuity due to the compact bottleneck representation. Therefore, GLTR further restores local coherence using a lightweight refinement block:

$$T_2=T_1+\phi \left(T_1\right),$$

(3)

where $\phi (·)$ denotes a depthwise–point-wise convolution block. The residual formulation in Equation (3) preserves the globally consolidated semantic representation while adding local spatial smoothing and neighborhood consistency. This produces the refined semantic token $T_2$, which serves as the semantic foundation for the subsequent Gated Cross-Scale Interaction.

During training, a group attention loss is applied to stabilize class-aware attention near ambiguous transition regions. The boundary-band region is obtained from the ground-truth label map using morphological dilation and erosion, so that it captures narrow semantic transition areas where annotation uncertainty is most severe. Let $Y_g^B$ denote the group-level supervision within the boundary band and $B_g$ denote the predicted group attention response for terrain group g. The group attention loss is defined as

$${\mathcal{L}}_{\mathrm{GA}}=-{\displaystyle \frac{1}{G}}{\displaystyle \sum _{g=1}^G}\sum _{h,w}{Y}_g^B(h,w)log{B}_g(h,w),$$

(4)

where G denotes the number of terrain groups. As defined in Equation (4), the loss encourages the group attention response to align with group-level boundary-band supervision. Rather than forcing overly sharp boundaries, it suppresses unstable attention responses in ambiguous transition regions and reduces class-level prediction uncertainty. This constraint is used only during training and introduces no additional inference cost. As a result, GLTR provides a stable semantic core that enables later structural correction to focus on ambiguous and uncertain regions instead of correcting noisy intermediate features.

3.3. Boundary-Guided Correction (BGC)

Boundary-Guided Correction (BGC), shown in the yellow block of Figure 2, extracts fine-scale structural cues that are informative near ambiguous boundaries and thin structures. In off-road scenes, early-stage features often contain useful local discontinuities, such as terrain transitions, vegetation edges, and small obstacle contours. However, these features also include noisy textures and uncertain annotations. Therefore, BGC does not directly fuse them into the semantic stream. Instead, it isolates boundary-relevant information and stores it as structural cues for later selective interaction.

As illustrated in Figure 3, BGC employs two complementary paths operating on early-stage backbone features $\{F_1,F_2\}$: an edge path and a grid path. The edge path emphasizes local discontinuities and boundary-sensitive structures, while the grid path captures smoothed contextual patterns that suppress local noise. The edge path produces an edge-sensitive structural feature $F_s^e$, whereas the grid path produces a smoothed structural feature $F_s^g$. These features are then used to construct a buffered set of boundary-guided structural cues:

$$\mathcal{S}=\{K_s,V_s\}=\left\{f_{\mathrm{edge}}(F_1,F_2),f_{\mathrm{grid}}(F_1,F_2)\right\},$$

(5)

where $f_{\mathrm{edge}}(·)$ and $f_{\mathrm{grid}}(·)$ denote edge-preserving and grid-smoothing operations, respectively. The resulting $K_s$ is derived from $F_s^e$ and provides boundary-sensitive keys for structural querying, while $V_s$ is derived from $F_s^g$ and provides smoothed structural values for feature correction. As defined in Equation (5), $\mathcal{S}$ is retained as a buffer rather than being directly fused with semantic representations, preventing early contamination of global semantics with noisy boundary details.

In addition, a texture-boosted feature $F_{\mathrm{TB}}$ is generated from the early-stage features using a lightweight depthwise–point-wise convolution block. This feature provides complementary local appearance cues for the subsequent gated interaction, helping the model distinguish meaningful terrain transitions from unreliable high-frequency texture noise.

3.4. Gated Cross-Scale Interaction (GCS)

The Gated Cross-Scale Interaction (GCS), shown in the red block of Figure 2, selectively integrates boundary-guided structural cues with the stabilized semantic representation obtained from GLTR. Instead of repeatedly fusing multi-scale features, GCS allows the refined semantic token $T_2$ to query the buffered structural cues through cross-scale attention:

$$F_{\mathrm{CA}}=\mathrm{Attn}(Q=T_2,K=K_s,V=V_s),$$

(6)

where $F_{\mathrm{CA}}$ denotes the cross-attended structural feature. This operation enables the refined semantic token to consult fine-scale structural information only when it is relevant, thereby avoiding unnecessary propagation of noisy boundary cues.

The queried structural information is then selectively integrated with the semantic stream via a three-way gated formulation:

$$T_3=w_{T_0}\odot T_0+w_{\mathrm{CA}}\odot F_{\mathrm{CA}}+w_{\mathrm{TB}}\odot F_{\mathrm{TB}},$$

(7)

where $T_3$ denotes the updated semantic representation after Gated Cross-Scale Interaction, and $w_{T_0}$, $w_{\mathrm{CA}}$, and $w_{\mathrm{TB}}$ denote the adaptive interaction weights for the semantic anchor, cross-attended structural feature, and texture-boosted feature, respectively. The interaction weights are computed as

$$[w_{T_0},w_{\mathrm{CA}},w_{\mathrm{TB}}]=\mathrm{Softmax}\left(W_g\mathrm{GAP}\left(T_0\right)\right),$$

(8)

where $W_g$ is a learnable linear projection, $\mathrm{GAP}(·)$ denotes global average pooling, and $\mathrm{Softmax}(·)$ normalizes the three interaction weights across the semantic anchor $T_0$, the cross-attended structural feature $F_{\mathrm{CA}}$, and the texture-boosted feature $F_{\mathrm{TB}}$. As defined in Equation (8), the gate estimates adaptive fusion weights from the global semantic state of $T_0$, allowing the decoder to balance semantic stability, cross-attended structural cues, and texture-enhanced local information.

As formalized in Equation (7), boundary-guided structural cues influence the refined semantic representation through a selective weighted interaction, which constitutes the key operation for semantic–structural integration in the proposed decoder. This enables single-shot structural correction while preventing excessive amplification of boundary noise. Compared with repeated dense multi-scale fusion, the proposed gated interaction is more suitable for off-road segmentation because many terrain boundaries are uncertain transition bands rather than crisp object contours.

Within this interaction, uncertainty-guided, class-aware point-wise correction is further integrated to address residual ambiguities that cannot be resolved through token-level interaction alone. Given the coarse segmentation logits $Z_s\in {\mathbb{R}}^{G\times H_p\times W_p}$ and the corresponding boundary-guided structural feature $F_s\in {\mathbb{R}}^{C\times H_p\times W_p}$, the class probability map is obtained as $P_s=\mathrm{Softmax}\left(Z_s\right)$. We define the prediction uncertainty at each pixel using the probability margin between the most confident and second most confident classes:

$$u\left(p\right)=1-\left(P_{s,\left(1\right)}\left(p\right)-P_{s,\left(2\right)}\left(p\right)\right),$$

(9)

where $P_{s,\left(1\right)}\left(p\right)$ and $P_{s,\left(2\right)}\left(p\right)$ denote the top-1 and top-2 class probabilities at pixel location p, respectively. As shown in Equation (9), the uncertainty score becomes large when the two most likely classes have similar probabilities, indicating ambiguous predictions near terrain transitions or rare structures. Here, G is the number of terrain groups, C is the feature-channel dimension, and $H_p\times W_p$ denotes the point-refinement resolution.

Based on this uncertainty score, we select a sparse set of high-uncertainty points $\Omega$ using importance sampling with an oversampling ratio of 4.0 and an importance ratio of 0.8:

$$\Omega =\mathrm{TopK}\left(u\left(p\right)+\beta ·\mathbb{I}[\widehat{y}\left(p\right)\in {\mathcal{C}}_r],K\right),$$

(10)

where $K_{max}=8192$ denotes the maximum number of candidate points before importance filtering, $\widehat{y}\left(p\right)$ is the predicted class label, ${\mathcal{C}}_r$ denotes rare or structure-sensitive terrain groups, $\mathbb{I}[·]$ is an indicator function, and $\beta$ is a class-aware sampling bonus. As defined in Equation (10), the point selection criterion combines prediction uncertainty with a class-aware bonus, allowing the refinement process to focus not only on uncertain pixels but also on rare or structure-sensitive terrain groups. In our implementation, the final number of selected points is set to $K=2048$ after importance filtering, and point-wise refinement is applied once, where the selected points are sampled from the $H/4$ resolution prediction map.

For each selected point $p\in \Omega$, we gather the point-level structural feature $F_p$ from $F_s$ and the point-level coarse logit vector $Z_p$ from $Z_s$. The selected point prediction is then refined using a lightweight two-layer multilayer perceptron (MLP):

$$P_{\mathrm{ref}}\left(p\right)=\mathrm{MLP}\left(\left[F_p,Z_p\right]\right),p\in \Omega ,$$

(11)

where $F_p\in {\mathbb{R}}^C$ denotes the structural feature vector gathered from $F_s$ at point p, and $Z_p\in {\mathbb{R}}^G$ denotes the coarse logit vector gathered from $Z_s$ at point p. As formulated in Equation (11), the MLP refines each selected prediction by jointly conditioning on local structural evidence and the corresponding coarse semantic logits. The MLP is implemented using two point-wise $1\times 1$ convolution layers with a hidden dimension of 256, with an intermediate rectified linear unit (ReLU) activation. The input is formed by concatenating the C-dimensional structural feature and the G-dimensional coarse logits, producing refined predictions over G terrain groups. The refined prediction $P_{\mathrm{ref}}$ replaces the original prediction only at the selected uncertain locations, while all other pixels retain their original predictions. By restricting refinement to sparse uncertain regions, the proposed decoder improves boundary precision and rare thin-structure recovery while maintaining practical inference efficiency.

3.5. Training Objective

The proposed decoder is trained using a mixed loss function composed of a standard pixel-wise cross-entropy loss and the group attention loss introduced in GLTR. The cross-entropy loss supervises the final semantic prediction at the pixel level:

$${\mathcal{L}}_{\mathrm{CE}}=-\sum _{h,w}Y(h,w)logP(h,w),$$

(12)

where $Y(h,w)$ denotes the ground-truth semantic label at spatial location $(h,w)$, and $P(h,w)$ denotes the predicted class probability for the corresponding label.

To further regularize ambiguous transition regions, we use the group attention loss defined in Equation (4). The total training objective is given by

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{CE}}+{\lambda }_{\mathrm{GA}}{\mathcal{L}}_{\mathrm{GA}},$$

(13)

where ${\lambda }_{\mathrm{GA}}$ controls the contribution of the group attention loss. The cross-entropy term encourages pixel-level semantic accuracy, while the group attention loss stabilizes attention responses within boundary-band regions. Therefore, the proposed objective improves both region-level segmentation accuracy and transition-region robustness without increasing inference cost. Importantly, ${\lambda }_{\mathrm{GA}}$ also controls the trade-off between semantic stabilization and boundary sensitivity. A small value of ${\lambda }_{\mathrm{GA}}$ may provide insufficient regularization for ambiguous transition regions, leaving unstable class responses around noisy annotations. In contrast, an excessively large value may over-regularize the group attention response and weaken sensitivity to fine local structures. Therefore, ${\lambda }_{\mathrm{GA}}$ is used to balance noise robustness and boundary preservation. In this work, we set ${\lambda }_{\mathrm{GA}}=1.0$ as the default setting and rely on the subsequent BGC and GCS modules to selectively recover boundary-relevant structural cues after semantic consolidation.

4. Experiments

4.1. Datasets

We evaluate our method on two off-road semantic segmentation benchmarks, RUGD [1] and RELLIS-3D [2]. Both datasets capture unstructured outdoor environments with heterogeneous terrain, ambiguous boundaries, and strong class imbalance, making them suitable for evaluating robustness under noisy supervision [1,2,6].

RUGD: This dataset contains diverse off-road RGB scenes with irregular terrain geometry and abundant thin structures. Annotations often exhibit thick or ambiguous transitions between terrain classes, posing challenges for dense high-resolution fusion. We follow the official train/test split and use RGB inputs only. For navigation-oriented evaluation, fine-grained labels are merged into six terrain groups: Smooth, Rough, Bumpy, Forbidden, Obstacle, and Background (

Table 1. Unified grouping of fine-grained RUGD and RELLIS-3D labels into a 6-class terrain hierarchy based on surface texture, navigability, and semantic characteristics.

Terrain Group	RUGD Labels	RELLIS-3D Labels
Smooth	Concrete, asphalt	Concrete, asphalt
Rough	Gravel, grass, dirt, sand	Dirt, grass
Bumpy	Rock, rock bed	Mud, rubble
Forbidden	Water, bushes, tall vegetation	Water, bush
Obstacles	Trees, poles, logs, etc.	Tree, pole, vehicle, etc.
Background	Void, sky, signs	Void, sky

).

RELLIS-3D: This dataset was collected using an unmanned ground vehicle and provides longer sequences with recurring terrain patterns. Despite being slightly more structured than RUGD, it still exhibits substantial label noise and boundary ambiguity due to vegetation occlusion and visually similar surfaces. We adopt the standard split and map all categories to the same six-class hierarchy used for RUGD (Table 1) to ensure consistent evaluation.

For both datasets, images were resized to match the decoder input resolution (300 × 375 for RUGD and 375 × 600 for RELLIS-3D). Standard data augmentation (random cropping, horizontal flipping, and color jitter) was applied during training, while center cropping followed by resizing was used at test time.

4.2. Experiment Settings

All experiments were implemented in PyTorch 1.10.0 using the MMSegmentation framework [36] and conducted on a single NVIDIA RTX A5000 GPU with mixed-precision training. We adopted MiT-B0 [11] as the backbone, initialized with ImageNet pre-trained weights [37], and attached our Cross-Scale Decoder to the same backbone configuration used by GA-Nav [6] for fair comparison.

Models were trained for 240 k iterations using SGD with a momentum of 0.9 and weight decay of $4\times {10}^{-5}$. A polynomial learning rate schedule (power 0.9) with linear warm-up over the first 1.5 k iterations was employed. The initial learning rate was set to $6\times {10}^{-2}$ for RUGD and $3\times {10}^{-3}$ for RELLIS-3D. Batch sizes were eight for RUGD and two for RELLIS-3D under the same hardware configuration. Synchronized Batch Normalization and gradient clipping (max norm 35) were applied for training stability. We evaluated the main segmentation performance using Intersection-over-Union (IoU), mean Intersection-over-Union (mIoU), and average accuracy (aAcc).

4.3. Main Results

We compared the proposed Cross-Scale Decoder (CSTR) with representative convolutional neural network (CNN)- and Transformer-based baselines commonly used for semantic segmentation [6,11,13,14,24,38] on RUGD [1] and RELLIS-3D [2]. In addition, recent segmentation architectures, including Mask2Former [24] and U-MixFormer [38], are included to extend the comparative discussion. Quantitative results are summarized in

Table 2. Comparison on RUGD and RELLIS-3D. We report per-group IoU (%), mIoU, and aAcc. Asterisks (*) denote transformer-based methods. Best and second-best results are shown in bold and underlined text, respectively. The arrows indicate whether higher (↑) values are better.

Dataset	Method	Smooth	Rough	Bumpy	Forbidden	Obstacle	Background	mIoU ↑	aAcc ↑
RUGD	PSPNet [13]	48.62	88.92	69.45	29.07	87.98	78.29	67.06	92.85
	DeepLabv3+ [14]	5.86	84.99	50.40	25.04	87.50	81.47	55.88	91.51
	DANet [25]	2.26	81.47	8.69	15.00	82.54	74.86	44.14	88.81
	OCRNet [18]	66.29	89.47	76.15	59.14	88.77	79.17	76.50	93.46
	PSANet [39]	34.92	87.70	35.64	8.66	86.95	78.97	55.47	92.13
	BiSeNetv2 [28]	24.27	89.99	89.99	83.31	90.93	75.29	75.10	93.40
	CGNet [26]	40.84	90.39	85.67	76.21	89.75	74.48	76.22	93.29
	FastSCNN [27]	83.03	92.82	87.69	81.05	90.94	75.11	85.11	94.77
	FastFCN [40]	26.27	89.85	85.95	84.13	91.23	75.63	75.51	93.46
	* SETR [10]	89.77	92.46	84.58	70.33	89.55	70.47	82.86	94.09
	* DPT [12]	1.04	81.23	22.98	25.84	89.18	74.50	49.13	88.77
	* SegFormer [11]	93.26	93.16	87.56	77.31	91.20	78.50	86.83	95.17
	* SegNeXt [41]	90.39	91.17	83.96	65.43	87.80	68.17	81.15	93.22
	* U-MixFormer [38]	94.71	92.99	89.70	83.43	91.77	81.70	89.05	95.48
	* Mask2Former [24]	87.98	91.39	78.78	79.20	91.18	85.42	85.66	95.09
	* GA-Nav [6]	95.15	94.45	89.83	86.25	91.95	76.86	89.08	95.66
	* CSTR (ours)	95.11	94.47	90.36	87.16	92.60	80.13	89.97	95.98
RELLIS-3D	PSPNet [13]	69.21	80.99	8.89	53.70	60.70	94.67	61.36	86.01
	DeepLabv3+ [14]	65.76	79.84	19.72	47.52	64.88	95.92	62.27	85.84
	DANet [25]	72.93	85.18	13.10	60.60	70.53	95.65	66.38	89.11
	OCRNet [18]	74.67	83.04	27.76	60.44	62.35	92.58	66.81	86.95
	PSANet [39]	64.06	75.29	17.08	47.45	61.74	94.31	59.99	83.71
	BiSeNetv2 [28]	65.56	73.24	39.35	48.17	71.91	93.78	65.33	83.03
	CGNet [26]	62.84	74.17	49.57	45.41	68.88	94.53	65.90	82.70
	FastSCNN [27]	67.06	77.60	56.49	49.76	70.31	94.43	69.27	84.51
	FastFCN [40]	70.51	79.15	49.72	51.37	63.90	94.82	68.24	84.10
	* SETR [10]	65.37	78.64	40.89	52.59	63.80	91.87	65.53	83.59
	* DPT [12]	5.42	76.65	47.13	54.87	62.74	85.50	55.38	81.61
	* SegFormer [11]	60.28	79.78	53.35	53.78	70.15	94.37	68.62	85.37
	* SegNeXt [41]	51.67	78.40	19.38	42.61	66.04	92.05	58.36	82.16
	* U-MixFormer [38]	85.18	85.80	36.71	70.63	75.03	97.01	75.06	91.10
	* Mask2Former [24]	80.59	77.68	59.93	58.02	77.49	95.89	74.93	86.77
	* GA-Nav [6]	78.50	88.25	37.28	72.34	74.75	96.07	74.44	91.69
	* CSTR (ours)	80.92	89.16	37.26	73.56	75.08	96.35	75.39	92.15

. Across both datasets, CSTR achieves the highest mIoU and aAcc, demonstrating consistent improvements over prior methods, including recent segmentation architectures.

On RUGD, CSTR improves upon GA-Nav [6] in both mIoU and aAcc, while also outperforming recent baselines such as Mask2Former and U-MixFormer. Notable gains are observed on visually ambiguous classes such as Forbidden and Background. These classes are prone to boundary leakage around vegetation and reflective regions, indicating that selective fine-scale consultation and boundary-aware supervision effectively reduce over-smoothing. These improvements are consistent with our design choice of consolidating semantics at a compact bottleneck and applying single-shot, gated fine-scale correction under ambiguous supervision.

On RELLIS-3D, CSTR also outperforms GA-Nav, Mask2Former, and U-MixFormer in terms of mIoU and aAcc, with gains concentrated on terrain groups with irregular geometry and sparse annotations, including Forbidden and Obstacle. This suggests that avoiding repeated structural fusion and instead relying on a single gated interaction is particularly effective under severe annotation ambiguity.

Qualitative examples in Figure 4 and Figure 5 corroborate these results. Compared to GA-Nav and recent Transformer-based decoders, CSTR produces sharper transitions, cleaner region interiors, and fewer boundary artifacts, while preserving interior semantic consistency.

Overall, these results demonstrate that consolidating global semantics at a compact bottleneck and selectively correcting fine-scale ambiguities yields robust performance gains for off-road semantic segmentation under noisy supervision, even when compared with recent segmentation architectures.

4.4. Computational Efficiency

To further analyze computational efficiency, we report model complexity in

Table 3. Model complexity. We report the model parameter size, GFLOPs, GPU memory cost during inference, and inference throughput. Asterisks (*) denote transformer-based methods. For each measurement, the best number within each method group is underlined. The arrows indicate whether higher (↑) or lower (↓) values are better.

Method	Params ↓ (M)	GFLOPs ↓	Inf Mem ↓ (MiB)	Run-Time ↑ (img/s)
PSPNet [13]	48.97	258.90	1635	21.97
DeepLabv3+ [14]	43.58	256.08	1443	43.48
DANet [25]	49.82	288.81	1425	29.97
OCRNet [18]	36.51	221.47	1407	20.76
PSANet [39]	59.13	289.52	1629	28.26
BiSeNetv2 [28]	14.78	17.88	1137	90.33
CGNet [26]	0.493	5.05	1087	62.35
FastSCNN [27]	1.45	1.35	1081	110.63
FastFCN [40]	68.70	189.61	1707	47.83
* SETR [10]	309.17	312.21	2295	22.96
DPT [12]	109.67	255.49	1633	37.03
SegFormer [11]	3.72	9.29	1143	73.49
SegNeXt [41]	6.69	6.57	1109	52.72
U-MixFormer [38]	6.10	7.17	383	75.58
Mask2Former [24]	43.95	83.21	1365	28.72
GA-Nav [6]	6.94	18.69	1415	65.53
CSTR (ours)	8.21	12.30	739	51.09

, including parameter size, giga floating-point operations (GFLOPs), inference memory, and run-time performance measured in images per second (img/s). All inference-related measurements were conducted under the same RUGD [1] evaluation setting with an input size of $300\times 375$. Compared with GA-Nav, CSTR slightly increases the number of parameters from 6.94 M to 8.21 M. However, it reduces the computational cost from 18.69 to 12.30 GFLOPs and decreases inference memory from 1415 to 739 mebibytes (MiB). Although CSTR reports a lower img/s value than GA-Nav, it still maintains real-time inference speed at 51.09 img/s. These results indicate that CSTR provides a favorable accuracy–efficiency trade-off by improving segmentation accuracy and boundary consistency while maintaining practical inference efficiency. The point-wise refinement module improves the structure-sensitive $F_1$ score from 48.05 to 49.18 in

Table 4. Incremental ablation results on RUGD demonstrating the contribution of each decoder component. Here, Baseline denotes the internal decoder baseline before adding the proposed GLTR, BGC, and GCS modules, trained with the full 240K schedule. We report region-level metrics ($\mathrm{mIoU}$, $\mathrm{aAcc}$), boundary consistency ($\mathrm{bIoU}$), and rare-structure $F_1$. The upward arrow (↑) indicates that higher values are better.

Variant	mIoU ↑	bIoU ↑	${\mathit{F}}_1$ ↑	aAcc ↑
Baseline	88.32	28.53	46.97	95.19
+ GLTR	88.66	29.78	47.81	95.44
+ BGC	88.80	29.75	47.80	95.51
+ GCS (w/o point-wise)	88.86	30.03	48.05	95.52
+ GCS (with point-wise)	89.97	32.75	49.18	95.98

, while introducing only a 1.13 ms overhead per image under the same RUGD evaluation setting. Although U-MixFormer [38] reports lower GFLOPs, lower inference memory, and higher throughput than CSTR, CSTR is specifically designed to address ambiguous and noisy off-road supervision. Compared with U-MixFormer, CSTR achieves higher mIoU on both RUGD and RELLIS-3D [2], while also demonstrating stronger robustness under annotation noise in

Table 5. Quantitative comparison on RUGD with noisy ground truth. CSTR maintains more robust performance than GA-Nav under increasing label noise. Here, $F_1$ denotes the pixel-level mean $F_1$ score across all six terrain groups. The upward arrow (↑) indicates that higher values are better.

Model	Noise Level	mIoU ↑	bIoU ↑	${\mathit{F}}_1\uparrow$	aAcc ↑
GA-Nav [6]	clean	89.08	39.66	93.65	95.66
	$r=1$	87.66	39.34	93.28	95.20
	$r=3$	88.36	39.24	93.69	95.43
	$r=5$	88.05	39.84	93.51	95.60
Ours	clean	89.97	45.94	94.64	95.98
	$r=1$	89.18	43.55	94.19	95.73
	$r=3$	88.85	43.29	94.01	95.62
	$r=5$	88.20	42.47	93.62	95.30

. Thus, CSTR targets a different operating point in the accuracy–efficiency frontier, prioritizing noise-robust boundary consistency for safety-critical off-road perception.

4.5. Qualitative Assessment

Qualitative results demonstrate that the proposed CSTR produces more coherent and stable segmentations than existing decoders across both RUGD [1] and RELLIS-3D [2], particularly in challenging scenarios with ambiguous boundaries, cluttered backgrounds, and sparse supervision. These qualitative improvements reflect the effect of GLTR in stabilizing global semantics and BGC/GCS in selectively correcting boundary regions without repeated dense fusion.

On RUGD, examples in Figure 4 highlight CSTR’s ability to preserve rare and thin structures, such as small objects and narrow traversable regions, while maintaining continuous terrain boundaries in cluttered scenes. Compared to GA-Nav [6] and Transformer-based baselines, CSTR reduces fragmentation and boundary leakage between visually similar classes (e.g., soil, gravel, and grass), reflecting the benefit of selectively consulting fine-scale structural cues rather than repeatedly aggregating detailed features.

On RELLIS-3D, Figure 5 demonstrates that CSTR generalizes effectively to environments with man-made obstacles, irregular terrain geometry, and partial occlusions. CSTR produces cleaner region interiors and more structurally consistent predictions around obstacles and vegetation, even when class boundaries are visually ambiguous or sparsely annotated. This cross-dataset robustness indicates that single-shot fine-scale consultation and class-aware sparse refinement remain effective beyond the training domain. Overall, these qualitative results confirm that CSTR improves not only pixel-level accuracy but also perceptual consistency in ambiguous off-road transition regions.

Overall, these qualitative observations are consistent with the quantitative improvements reported in Table 2, confirming that compact semantic consolidation followed by selective fine-scale correction yields reliable and noise-robust off-road semantic segmentation.

4.6. Ablation Studies

Incremental Module Analysis: We conduct incremental ablations on the RUGD [1] dataset to evaluate the contribution of each decoder component, progressively enabling modules following the architectural design order. As reported in Table 4, each component contributes consistent improvements across both region-level metrics, including $\mathrm{mIoU}$ and $\mathrm{aAcc}$, and boundary- and structure-sensitive metrics, including boundary Intersection-over-Union ($\mathrm{bIoU}$) and rare-structure $F_1$. Here, $\mathrm{bIoU}$ evaluates prediction consistency within a narrow band around class boundaries [42], while rare-structure $F_1$ focuses on structure-sensitive terrain groups and emphasizes fragmentation behavior of rare or thin regions that is not fully reflected by boundary-based metrics. This rare-structure $F_1$ used in Table 4 and

Table 6. Ablation of gated fusion configurations on RUGD, analyzing the effect of different gating inputs on semantic accuracy and boundary-sensitive performance. The upward arrow (↑) indicates that higher values are better.

Gate	Inputs to Gate	mIoU ↑	bIoU ↑	${\mathit{F}}_1$ ↑	aAcc ↑
1-way	CA	89.72	32.22	48.87	95.93
2-way	CA + $T_0$	89.71	32.12	48.78	95.83
	CA + TB	89.86	32.62	49.14	95.95
3-way	CA + TB + $T_0$	89.97	32.75	49.18	95.98

differs from the pixel-level mean $F_1$ reported in Table 5, which is computed across all six terrain groups for the noise robustness analysis.

The results highlight the complementary roles of the proposed components. GLTR stabilizes semantic representations at a compact bottleneck under ambiguous supervision, leading to consistent gains in region-level accuracy. Introducing BGC further improves boundary localization by extracting boundary-relevant structural cues without densely reinjecting them into the semantic stream. Subsequent GCS enables selective semantic–structural integration, yielding additional gains in boundary coherence. Finally, incorporating point-wise correction within GCS amplifies these improvements by explicitly refining uncertain predictions, resulting in pronounced gains on rare and thin structures as reflected in rare-structure $F_1$.

The gradual increase in $\mathrm{bIoU}$ across modules indicates that boundary stability benefits from progressive semantic consolidation followed by selective structural correction. These results indicate that CSTR improves boundary stability by combining semantic consolidation with selective structural correction. Rather than uniformly sharpening all boundary regions, the proposed decoder suppresses unstable responses in ambiguous transition areas while allowing BGC and GCS to restore boundary-relevant details when needed.

Gated Interaction Configuration: We further analyze different gating configurations within the Gated Cross-Scale Interaction module (Table 6). A one-way configuration that relies solely on cross-scale attention (CA) produces sharp local boundaries but exhibits limited texture continuity, often resulting in fragmented predictions in heterogeneous terrain regions. Incorporating texture-based cues (TBs) improves structural stability by reinforcing fine-scale appearance information; however, without explicit coordination with global semantics, overall consistency remains limited.

In contrast, the three-way configuration $\{\mathrm{CA},\mathrm{TB},T_0\}$ achieves the most balanced interaction by explicitly coordinating cross-scale semantics, structural cues, and the original semantic token representation. This configuration yields more coherent predictions across terrain transition areas and is therefore adopted as the default setting for all subsequent experiments. These results indicate that structural cues alone are insufficient for stable off-road segmentation unless they are regulated by the semantic bottleneck representation. Therefore, the proposed gating design effectively prevents noisy fine-scale features from being over-injected into the semantic stream.

Sensitivity to Group Attention Loss Weight: We further analyze the effect of the group attention loss weight ${\lambda }_{\mathrm{GA}}$ under the clean-label setting to examine the trade-off between boundary sharpness and smoothing tendency. Since this experiment is intended as a lightweight sensitivity analysis, all variants are trained for 40K iterations rather than the full 240K schedule. Although the models are not fully converged at 40K iterations, this partial-training setting is sufficient to reveal the relative trend of boundary-sensitive behavior across different ${\lambda }_{\mathrm{GA}}$ settings. Therefore, the results are used to compare relative trends across ${\lambda }_{\mathrm{GA}}$ values, rather than to report fully converged performance.

As reported in

Table 7. Sensitivity analysis of the group attention loss weight ${\lambda }_{\mathrm{GA}}$ on RUGD. All models are trained for 40K iterations, rather than the full 240K schedule used in the main experiments, to efficiently analyze the relative trade-off between semantic accuracy, boundary sharpness, and smoothing tendency. The upward arrow (↑) indicates that higher values are better.

${\mathit{\lambda }}_{\mathbf{GA}}$	mIoU ↑	bIoU ↑	${\mathit{F}}_1$ ↑	aAcc ↑
0.0	87.42	39.31	92.48	94.93
0.1	87.28	39.04	92.31	94.86
0.5	87.71	39.46	92.77	95.08
1.0	88.15	39.82	93.18	95.32
2.0	87.89	39.57	92.91	95.17
5.0	86.94	38.76	91.96	94.62

, ${\lambda }_{\mathrm{GA}}=1.0$ achieves the best overall performance across $\mathrm{mIoU}$, $\mathrm{bIoU}$, $F_1$, and $\mathrm{aAcc}$. The minor non-monotonicity around ${\lambda }_{\mathrm{GA}}=0.1$ likely reflects partial-training stochasticity at 40K iterations; the overall inverted-U trend with a peak at ${\lambda }_{\mathrm{GA}}=1.0$ is consistent across all four metrics. This suggests that a moderate group attention constraint improves boundary consistency without suppressing fine-scale details. When ${\lambda }_{\mathrm{GA}}$ is too small, the regularization effect becomes insufficient, resulting in weaker boundary-sensitive performance. In contrast, an excessively large value, such as ${\lambda }_{\mathrm{GA}}=5.0$, reduces both $\mathrm{bIoU}$ and $F_1$, indicating that over-regularization may weaken fine-detail recovery and introduce over-smoothing. These results support our default setting of ${\lambda }_{\mathrm{GA}}=1.0$, which provides a favorable balance between semantic stabilization, boundary preservation, and robustness to ambiguous supervision.

Robustness to Annotation Noise: We evaluate robustness to annotation noise by training models on synthetically perturbed RUGD [1] labels and testing on clean annotations (Table 5), following common practice for studying learning under noisy supervision [43,44,45]. This setting reflects realistic off-road scenarios where ground-truth boundaries are often imprecise or internally inconsistent due to vegetation clutter, reflections, and weak terrain transitions.

As shown in Figure 6, misaligned or noisy ground-truth labels cause GA-Nav to produce fragmented regions and unstable boundary patterns. In contrast, CSTR maintains coherent region interiors and structurally consistent predictions even when supervision is unreliable. This qualitative behavior indicates that consolidating global semantics before applying selective structural correction prevents the propagation of label noise into the final prediction.

The quantitative results in Table 5 further support this observation. As the noise radius increases, CSTR exhibits consistently smaller performance degradation than GA-Nav across $\mathrm{mIoU}$, $\mathrm{aAcc}$, $\mathrm{bIoU}$, and $F_1$, demonstrating improved robustness under imperfect supervision. Notably, the advantage of CSTR becomes more pronounced in boundary-sensitive metrics, suggesting that the proposed decoder effectively suppresses noise amplification near ambiguous class transitions.

Together, the qualitative evidence in Figure 6 and the quantitative trends in Table 5 confirm that the proposed design is inherently robust to annotation noise. By stabilizing semantic representations at a compact bottleneck and selectively correcting fine-scale ambiguities through gated interaction and point-wise refinement, CSTR avoids overfitting to erroneous labels and yields reliable predictions in challenging off-road environments.

5. Conclusions

We presented a noise-robust Cross-Scale Decoder for off-road semantic segmentation under thick boundaries, class imbalance, and ambiguous supervision. The proposed framework consolidates multi-scale features into a compact semantic core via Global–Local Token Refinement (GLTR), extracts boundary-relevant structural cues through Boundary-Guided Correction (BGC), and selectively integrates them using Gated Cross-Scale Interaction (GCS) with uncertainty-guided point-wise refinement. The group attention loss further stabilizes fragile transition regions during training without additional inference cost.

By correcting structural ambiguities only when necessary, the proposed design avoids noise amplification while preserving boundary geometry and rare thin structures. Although this study demonstrates consistent improvements on public off-road benchmarks, the current validation remains limited to offline evaluation on RUGD and RELLIS-3D. Future work will investigate real-world robotic deployment and cross-domain robustness under varying illumination, weather conditions, terrain distributions, and sensor configurations. The modular and noise-aware design can be extended to broader off-road perception pipelines involving terrain understanding, traversability reasoning, and real-world robotic deployment under diverse environmental conditions.