On Segment-Aware Monocular Depth Estimation Using Vision Transformers

Abstract

Monocular Depth Estimation (MDE) infers per-pixel scene geometry from a single RGB image. Despite recent progress, global MDE models often blur depth discontinuities at object boundaries and fail to capture object-level structure. Segment-aware depth estimation addresses this limitation by exploiting semantic segmentation to decompose depth prediction into simpler, class-specific subproblems. In this work, we study semantic-aware MDE in a multi-branch design where each semantic class is handled by a lightweight Vision Transformer (ViT) branch that predicts dense depth for its class while suppressing interference from other regions. We further examine fusion strategies that merge the branch outputs into a single prediction: (i) a learnable cross-attention fusion module that predicts depth from the stack of per-class proposals and masks, and (ii) a parameter-free stitched summation that sums mask-gated outputs. The proposed architecture is simple, scalable, end-to-end trainable, and compatible with arbitrary transformer backbones. Experiments on Virtual KITTI 2, where ground-truth depth and semantic labels are available, show that segment-aware modeling produces sharper depth boundaries and improves standard error metrics compared to a single-branch baseline (AbsRel 0.243→0.152; RMSE 11.952→9.101). Finally, we find that the parameter-free summation matches, and in most cases improves upon, the accuracy of learned fusion while adding no computational overhead.

1. Introduction

Accurate depth estimation from a single image is crucial for autonomous driving, robotics, and 3D scene reconstruction. Modern MDE models rely on convolutional or transformer networks trained on large RGB-D datasets. Most of these models treat depth estimation as a global pixel-wise regression task, which often leads to blurred object boundaries and loss of local detail. Depth and semantics are closely related: boundaries between semantic objects typically coincide with depth discontinuities, and the depth distribution within a segment tends to follow class-specific patterns. Recent work demonstrates that incorporating segmentation cues can improve MDE. For example, SHED uses a bidirectional segment hierarchy to inform depth prediction [1], and Jiao et al. introduce an attention-driven synergy network where semantic features are shared with depth estimation [2]. However, these methods typically learn segmentation jointly with depth or inject semantic cues into a single predictor; comparatively little work studies the simpler alternative of training class-specific depth experts given external masks and then composing their outputs.

This paper investigates a segment-aware architecture for MDE. Given an image, its semantic masks, and the corresponding depth map during training, we train one depth branch per semantic class so that each branch specialises on a narrow semantic domain. At inference time, we assume that semantic masks are provided by an external segmentation system. Crucially, we study how to compose the class-wise depth proposals into a single prediction: (i) a learnable cross-attention fusion module that directly predicts depth from the stack of per-class proposals and masks, and (ii) a parameter-free stitched summation that simply sums the mask-gated per-class predictions (our proposed default, since it adds no computational overhead). An overview of the baseline and the two segment-aware variants is shown in Figure 1.

In this work we focus on settings where such masks are accurate, as in Virtual KITTI 2 [3,4], in order to isolate and quantify the benefit of segment-wise depth decomposition. Our goal is not to compete with the latest large-scale, multi-dataset depth systems, but to answer a more focused architectural question: given the same backbone, optimiser, and training set, does explicit segment-wise decomposition improve depth quality compared to a purely global predictor? To keep this question well-posed, all of our quantitative comparisons are carried out against a single-branch baseline trained under identical data, optimiser, and training recipe (using the same compact encoder–decoder design as each expert branch), rather than against external monocular depth models that benefit from substantially larger backbones and pretraining corpora. In that stronger regime, our segment-aware decomposition would act as a mechanism for combining class-wise experts (with optional learned fusion), but such experiments would conflate the effects of semantics with gains due to model scale and pretraining and are therefore left outside the scope of this paper. The proposed segment-aware design is explicitly backbone-agnostic: in principle, each expert branch could be replaced by any off-the-shelf depth module, including state-of-the-art predictors trained on external data. Our contributions are as follows:

• Segment-aware depth experts with transformers. We propose a multi-branch monocular depth architecture in which each semantic class is handled by a dedicated, compact ViT encoder–decoder expert that receives a mask-gated RGB input and predicts a dense depth proposal for that class.
• Two composition strategies (learned vs. parameter-free). We study how to compose per-class depth proposals into a single prediction by comparing (i) a cross-attention fusion module that directly predicts depth from the stack of per-class proposals and masks, and (ii) a parameter-free stitched summation that sums the mask-gated proposals.
• Controlled study with accuracy–efficiency trade-offs. Under a controlled training protocol (same data, optimiser, and per-branch backbone) on Virtual KITTI 2, we evaluate 3-, 6-, and 11-class schemes and quantify both accuracy and computational cost. We find that most gains come from the segment-wise decomposition, while simple summation is a strong default that is competitive with, and often outperforms, learned fusion without additional overhead.

To the best of our knowledge, the closest prior work that decomposes monocular depth prediction into category-wise experts conditioned on explicit segments is the Semantic Divide-and-Conquer (SDC-Depth) network [5]. SDC-Depth decomposes the scene into panoptic segments and predicts a canonical depth map for each segment; segments of the same category share a decoder, and scale/shift parameters are estimated to reconstruct the global depth. Our approach differs in three key ways: (i) we use precomputed masks rather than predicting segmentation, (ii) we assign a separate encoder–decoder to each class instead of sharing decoders across categories, and (iii) we study both a learnable cross-attention fusion module and a parameter-free summation baseline, rather than relying on category-wise scale/shift regression. PanopticDepth [6], a closely related model, predicts depth per instance using instance-specific convolution kernels and merges them into a panoptic map. In contrast, our model operates on semantic classes (not instances) and composes class-wise outputs via attention or parameter-free summation rather than dynamic convolutions.

2. Related Work

2.1. Monocular Depth Estimation

Deep networks have greatly advanced monocular depth estimation [7]. Early convolutional architectures regress pixel-wise depth from an RGB image; examples include the Deep Ordinal Regression Network (DORN) [8], which casts depth estimation as an ordinal regression problem and discretises the depth range into intervals, and MiDaS [9], which mixes multiple datasets to learn robust, scale-invariant depth predictors. Transformer-based models further improve global context aggregation by modelling long-range dependencies and have recently achieved state-of-the-art results. Self-supervised and self-training methods exploit photometric consistency across stereo pairs or video frames to supervise depth when ground-truth is unavailable, reducing dependence on dense RGB-D datasets. Despite these advances, most global models still treat depth estimation as a purely pixel-wise regression problem and tend to blur object boundaries or under-utilise semantic structure.

2.2. Semantic Guidance for Depth Estimation

Semantic information can provide strong priors for depth estimation [10]. SDC-Depth [5] decomposes each scene into panoptic segments and predicts a canonical, scale- and shift-invariant depth map for each category, reassembling them via learned scale and shift parameters. PanopticDepth [6] proposes a unified framework for depth-aware panoptic segmentation, where dynamic convolution kernels predict instance-wise masks and depth maps, enabling instance-specific context sharing between segmentation and depth. SHED [1] redesigns the vision transformer to use hierarchical segment tokens that are pooled and unpooled across encoder and decoder layers, jointly reasoning about coarse-to-fine semantics and depth. The Semantic-aware Spatial Feature Alignment (SSFA) framework of Li et al. [11] aligns implicit semantic features with depth features and adds a semantic-guided ranking loss, showing that guiding depth with semantics can improve both metric and relative depth accuracy. Unlike these joint or implicit approaches, our method assumes that semantic masks are provided and trains separate depth experts per semantic class, composing their outputs either via cross-attention fusion or via a parameter-free summation.

An early example of explicitly conditioning depth on semantics is the work of Ladický et al. [12], who couple semantic labels and depth in a unified structured prediction framework to encourage semantic and geometric consistency. Their formulation effectively exploits class-conditioned depth priors, anticipating later divide-and-conquer strategies. Our segment-aware architecture can be seen as a modern, end-to-end analogue: we similarly exploit class-wise structure, but replace hand-crafted features and Conditional Random Field (CRF) fusion with transformer-based per-class experts and a modular composition stage (cross-attention fusion or parameter-free summation). More recently, ROIFormer [13] proposes a semantic-aware region-of-interest transformer for self-supervised monocular depth estimation, constraining cross-attention to local regions guided by semantic cues. In contrast, ROIFormer operates in a self-supervised, real-world setting with a single depth predictor whose attention is restricted by semantics, whereas our model allocates distinct transformer branches to each semantic class and fuses their depth proposals at the output level in a fully supervised synthetic setup on Virtual KITTI 2. A detailed quantitative comparison with these methods would require different training regimes and datasets and is therefore beyond the scope of this work.

2.3. Depth Refinement with Masks

Mask-guided depth refinement treats segmentation as a post-processing cue. Kim et al. [14] introduced Layered Depth Refinement with Mask Guidance, a framework that decomposes an initial depth map into foreground and background layers using a binary or soft mask and performs separate inpainting and outpainting operations to refine depth boundaries. Their approach relies on high-quality masks and refines the output of an existing depth estimator rather than learning depth from scratch. In contrast, our method integrates segmentation cues directly into the estimation pipeline, exploiting mask information during both training and inference to produce sharper boundaries and semantically consistent depth predictions.

Saeedan and Roth [15] similarly use panoptic segmentation maps to boost self-supervised monocular depth estimation via panoptic-guided smoothing, alignment, and stereo consistency losses. Their method regularises a single depth network using instance-level masks, while our approach integrates semantic masks at the architectural level by training per-class depth experts and fusing their outputs. We view these panoptic-guided losses as complementary to our segment-aware design, but a detailed comparison on real-world self-supervised benchmarks is outside the scope of this paper, which focuses on fully supervised depth estimation on Virtual KITTI 2.

In summary, prior work has either jointly learned depth and segmentation in a single network, injected semantic features into a global depth predictor, or used semantic and panoptic masks to regularise self-supervised losses. To the best of our knowledge, prior work has not systematically studied this setting with precomputed semantic masks and separate transformer-based depth experts per class under a controlled protocol with the same training recipe and per-branch backbone, while also contrasting learned cross-attention fusion against a parameter-free stitched summation. Figure 1 provides a high-level schematic of the baseline, the segment-aware model with learned fusion, and the segment-aware no-fusion stitched summation.

3. Segment-Aware Depth Estimation

3.1. Preliminaries

Let $I\in {\mathbb{R}}^{3\times H\times W}$ be an RGB image, $D\in {\mathbb{R}}^{1\times H\times W}$ its metric depth map, and $M\in {\{0,1\}}^{N\times H\times W}$ the binary semantic masks, where N is the number of classes. We assume M is exclusive on the set of valid pixels, i.e., at most one class is active per pixel. In our Virtual KITTI 2 setup we treat the Sky class as invalid: Sky pixels are excluded from all losses and metrics, and the corresponding mask entries are set to zero. Hence, for non-Sky pixels we have ${\sum }_{i=1}^N{M}_i\left(p\right)=1$, while for Sky pixels ${\sum }_{i=1}^N{M}_i\left(p\right)=0$. Each class map $M_i$ is a binary mask that activates pixels belonging to that class.

3.2. Per-Class Depth Branches

Each class has a dedicated depth branch built from a small Vision Transformer encoder followed by a convolutional decoder. Given I and $M_i$, we form the masked input $I_i=I\odot M_i$. The encoder processes $I_i$ and outputs token features, which the decoder upsamples back to an $H\times W$ map. We apply a softplus activation to ensure positive depth and obtain an unmasked per-class depth proposal:

$$z_i=\sigma \left(f_i\left(I\odot M_i\right)\right),$$

where $f_i$ denotes the encoder–decoder of branch i and $\sigma$ is the softplus nonlinearity, ensuring positive depths. Each $z_i$ is multiplied by its corresponding mask $M_i$ when composing the stitched depth by summation, so that predictions are gated outside the class region.

3.3. Cross-Attention Fusion

We form a stitched depth map via mask-gated summation of the per-class predictions:

$$z_{\mathrm{sum}}=\sum _{i=1}^N\left(M_i\odot z_i\right).$$

On valid (non-Sky) pixels, where ${\sum }_i{M}_i\left(p\right)=1$, this summation is equivalent to the mask-weighted average and simply selects the unique active class prediction at each pixel. On Sky pixels, all masks are zero and the pixel is ignored by the losses/metrics. Therefore, at semantic boundaries the stitched depth transitions by switching the contributing expert pixel-wise (hard routing); overlaps do not occur and gaps arise only on invalid Sky pixels, which are excluded from training and evaluation.

The learnable fusion module replaces this fixed stitching with a lightweight cross-attention network that directly predicts the final depth map z from ${\{z_i,M_i\}}_{i=1}^N$. We first compute two global summary maps: the mean mask $m_{\mathrm{mean}}={\displaystyle \frac{1}{N}}{\sum }_i{M}_i$ and a scale-sensitive cue given by the per-pixel standard deviation ${\sigma }_z={\mathrm{Std}}_i\left(M_i\odot z_i\right)$ across the masked per-class depth maps. We deliberately restrict these fusion cues to low-dimensional, easily interpretable statistics to keep the fusion module lightweight and to avoid conflating our main contribution (segment-wise depth experts) with an aggressively engineered fusion mechanism. We then patchify these maps by average pooling to a coarse grid of $H_c\times W_c$ patches (patch size $16\times 16$, a fixed non-tuned choice consistent with common ViT practice) and flatten them into tokens, trading off fusion spatial granularity against computational cost.

The query tokens are obtained by linearly projecting the two-channel token sequence $[m_{\mathrm{mean}},{\sigma }_z]$. Keys and values are obtained by concatenating the depth-mask stack $[z_1,M_1,\dots ,z_N,M_N]$, patchifying it in the same way, and projecting the resulting $2N$-channel tokens. A stack of transformer-style cross-attention blocks updates the queries given the fixed keys/values. Finally, the updated query tokens are reshaped back to the $H_c\times W_c$ grid, bilinearly upsampled to $H\times W$, and passed through a small convolutional head to regress the fused depth map:

$$z=\sigma \left(g\left(\mathcal{F}\left({\{z_i,M_i\}}_{i=1}^N\right)\right)\right),$$

where $\mathcal{F}(·)$ denotes the fusion pipeline (patchification and projection of inputs, cross-attention updates, token-to-grid reshaping, and upsampling to $H\times W$). Here, $g(·)$ denotes the convolutional regression head and $\sigma (·)$ is the softplus nonlinearity enforcing positive depth. Unlike residual “refinement” formulations, the fusion module predicts z directly. Exploring alternative aggregation cues (e.g., probabilistic gating or confidence-weighted mixtures) is a natural extension, but we leave it to future work to preserve the controlled scope of this study.

3.4. Losses

Following Eigen et al. [16], we use the scale-invariant log RMSE (SI-RMSE) for metric depth. The global depth loss is SI-RMSE plus a small BerHu term [17] (weight $0.02$); during the first 5 epochs we warm up with ${\ell }_1$ instead of SI-RMSE + BerHu. To encourage within-object smoothness, we add an edge-aware regulariser weighted by image gradients (weight $1\times {10}^{-2}$). During training we sum the global depth loss with an epoch-weighted sum of per-branch SI-RMSE losses computed on each semantic class mask $M_i$ (no BerHu per branch). Sky pixels are excluded by construction (their mask entries are set to zero), and we skip a branch loss if a batch contains no pixels for that class. Because Virtual KITTI 2 provides ground-truth semantic labels and we do not learn a segmentation head, there is no segmentation loss term; the fusion model is trained end-to-end purely from depth supervision. Concretely, the total training objective is

$$\mathcal{L}={\mathcal{L}}_{\mathrm{global}}+W_{\mathrm{seg}}\sum _{i=1}^N{\mathcal{L}}_{\mathrm{SI}}^{\left(i\right)}+{\lambda }_{\mathrm{smooth}}{\mathcal{L}}_{\mathrm{smooth}},$$

where ${\mathcal{L}}_{\mathrm{global}}$ is the global SI-RMSE+BerHu loss on the fused depth, ${\mathcal{L}}_{\mathrm{SI}}^{\left(i\right)}$ is the SI-RMSE for branch i on class mask $M_i$, computed only if the batch contains pixels of class i, $W_{\mathrm{seg}}$ is the epoch-dependent segment loss weight, and ${\lambda }_{\mathrm{smooth}}$ is the edge-aware smoothness weight. In all experiments we use a two-stage schedule for the segment-loss weight: $W_{\mathrm{seg}}=0.25$ for epochs 1–5 (warm-up) and $W_{\mathrm{seg}}=0.15$ for epochs 6–50, applied identically across all variants and class schemes.

3.5. No-Fusion Variant

To isolate the contribution of the cross-attention module, we experiment with a no-fusion model where the per-class depths are summed without refinement. Concretely, we remove the fusion module and treat the stitched summation as the final prediction, i.e., $z=z_{\mathrm{sum}}={\sum }_{i=1}^N(M_i\odot z_i).$

4. Experimental Setup

4.1. Dataset

We evaluate on the Virtual KITTI 2 dataset [3,4], which provides synthetic driving scenes with RGB images, 16-bit depth maps (values in centimetres), and per-pixel semantic labels. Images are resized to 256 × 256, and the dataset is split into 90% training and 10% validation sets (a commonly used split, kept fixed across all variants and not treated as a tunable hyperparameter). Concretely, we recursively collect all matched (RGB, depth, semantic) triplets, shuffle them at the frame level with a fixed seed, and assign the first 90% to training and the remaining 10% to validation (i.e., the split is not scene-disjoint, consistent with our goal of a controlled architectural comparison). Our conclusions are comparative and are drawn under an identical fixed-seed split and training recipe for all variants to isolate architectural differences under the same empirical evaluation protocol, while scene-/sequence-disjoint generalisation is a complementary question beyond the scope of this study. Semantic labels are remapped into 3, 6, and 11 classes according to three segmentation schemes used in our experiments. These schemes capture coarse, mid-level, and fine semantic granularity:

• 3-class: Vegetation = {Tree, Vegetation}; Ground = {Road, Terrain}; Artificial = {Building, Truck, Car, Van, GuardRail, TrafficSign, TrafficLight, Pole, Misc}.
• 6-class: Road = {Road}; Terrain = {Terrain}; Building = {Building}; Vegetation = {Tree, Vegetation}; Object = {GuardRail, TrafficSign, TrafficLight, Pole, Misc}; Vehicle = {Truck, Van, Car}.
• 11-class: Terrain, Tree, Vegetation, Building, Road, GuardRail, TrafficSign, TrafficLight, Pole, Misc, Vehicle = {Truck, Car, Van}.

Due to severe class imbalance among vehicle subclasses (Truck and Van jointly account for less than 2% of pixels), training separate experts for each vehicle category led to unstable Truck predictions. We therefore merge Truck, Car, and Van into a single Vehicle class in the 11-class configuration.

4.2. Model Variants

• Baseline: a single ViT encoder-decoder that predicts depth from RGB only, without access to semantic masks.
• Segment-aware (Fusion): per-class experts (one encoder–decoder per semantic class), followed by a cross-attention fusion module. The fusion module takes the stack of per-class depth maps and their masks as input and directly predicts a fused depth map.
• Segment-aware (No-Fusion)—proposed: per-class experts (one encoder–decoder per semantic class); the class-wise predictions are mask-gated and summed to form the final depth, with no additional fusion or refinement.

All three variants share the same compact encoder–decoder backbone and are trained from scratch on Virtual KITTI 2. This controlled setup ensures that our comparisons isolate the contribution of the segment-aware decomposition and fusion, rather than the choice of backbone or external training data. We intentionally define the baseline as the single expert “unit” and evaluate segment-aware models as parallel compositions of the same unit across classes; thus, the increased parameters/FLOPs are an explicit accuracy–efficiency trade-off (see the complexity analysis in Section 5.2), rather than an uncontrolled confound. For this reason we deliberately refrain from including large pre-trained depth systems as “external baselines”: they would almost certainly achieve lower absolute error on Virtual KITTI 2, but such a comparison would answer a different question (which model has the largest pre-training budget) rather than whether semantics help a fixed backbone.

For closely related semantic-aware pipelines such as SDC-Depth [5] and PanopticDepth [6], a fair quantitative comparison on our split would require retraining their full pipelines under the same compact from-scratch recipe and aligning their additional assumptions (e.g., panoptic/instance inputs, scale-shift reconstruction, or dynamic kernels). Since our goal is to isolate the architectural effect of segment-wise decomposition (experts + composition) rather than to re-benchmark heterogeneous training regimes, we restrict quantitative evaluation to the controlled baseline and variants described above.

4.3. Training Details

All models are trained on Virtual KITTI 2 at 256 × 256 resolution for 50 epochs using the AdamW optimizer with a cosine-annealing learning rate schedule, shared across all variants. We use 50 epochs and a single, standard AdamW + cosine training recipe as fixed (non-tuned) hyperparameters, applied uniformly across all variants to avoid confounding the architectural comparison with optimisation choices. For all configurations we use a batch size of 4. The learning rate is set to $5\times {10}^{-4}$ with weight decay ${10}^{-4}$; the loss schedule follows ${\ell }_1$ warm-up for the first five epochs and the SI-RMSE+BerHu objective thereafter. All results are obtained on a single NVIDIA GPU using these shared hyperparameters. For stability we apply gradient clipping with a maximum norm of 1.0. Each expert branch uses a compact ViT encoder with patch size $16\times 16$, embedding dimension 128, depth 2 transformer blocks, and 4 attention heads (MLP ratio 4 and dropout 0.1), followed by a lightweight convolutional decoder that reshapes tokens to an $H/16\times W/16$ grid, upsamples to $H\times W$ via bilinear interpolation, applies two $3\times 3$ convolutions (128→64→32) with GELU, and uses a final $1\times 1$ head with softplus output to regress positive metric depth. No data augmentation is used beyond resizing inputs to $256\times 256$ (no random crops, flips, or photometric transforms).

4.4. Evaluation Metrics

We evaluate on valid pixels (all classes except Sky). Let $d_i$ and ${\widehat{d}}_i$ denote ground-truth and predicted depth at valid pixel i, and let N be the number of valid pixels. We report the following:

• Absolute Relative Error (AbsRel): $\mathrm{AbsRel}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\displaystyle \frac{|d_i-{\widehat{d}}_i|}{d_i}}$.
• Squared Relative Error (SqRel): $\mathrm{SqRel}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\displaystyle \frac{{(d_i-{\widehat{d}}_i)}^2}{d_i}}$.
• Root Mean Squared Error (RMSE): $\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{(d_i-{\widehat{d}}_i)}^2}$.
• Log RMSE (RMSE_log): ${\mathrm{RMSE}}_{log}=\sqrt{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{(log{d}_i-log{\widehat{d}}_i)}^2}$.
• Threshold accuracy $\left({\delta }_k\right)$: ${\delta }_k={\displaystyle \frac{1}{N}}\left|\left\{i:max\left({\displaystyle \frac{{\widehat{d}}_i}{d_i}},{\displaystyle \frac{d_i}{{\widehat{d}}_i}}\right)<1.{25}^k\right\}\right|,\hspace{1em}k\in \{1,2,3\}$.

5. Results

5.1. Accuracy Results

Across all class schemes, segment-wise decomposition is the dominant source of improvement. As reported in

Table 1. Quantitative results on the Virtual KITTI 2 dataset, evaluated on all classes except Sky. Lower values indicate better error performance; higher values indicate better accuracy. Bold values indicate the best overall result per evaluation metric across all reported models.

Model	Classes	AbsRel	SqRel	RMSE	RMSE_log	${\mathit{\delta }}_1$	${\mathit{\delta }}_2$	${\mathit{\delta }}_3$
Baseline	-	0.243	3.762	11.952	0.297	0.684	0.894	0.957
Segment-aware (Fusion)	3	0.153	3.090	11.153	0.230	0.841	0.945	0.975
Segment-aware (No-Fusion)	3	0.185	2.259	9.547	0.237	0.764	0.926	0.972
Segment-aware (Fusion)	6	0.155	3.146	11.177	0.227	0.844	0.946	0.975
Segment-aware (No-Fusion)	6	0.156	1.908	9.101	0.207	0.809	0.944	0.980
Segment-aware (Fusion)	11	0.162	3.252	11.304	0.235	0.828	0.941	0.974
Segment-aware (No-Fusion)	11	0.152	1.895	9.250	0.204	0.813	0.946	0.981

(validation on Virtual KITTI 2 excluding Sky pixels), both segment-aware variants outperform the single-branch baseline on every metric. When selecting a default model, we prioritise absolute depth accuracy (RMSE, SqRel): the parameter-free stitched summation is consistently the best and also the most efficient. The learned fusion module mainly improves the strict threshold accuracy ${\delta }_1$, indicating better relative agreement on many pixels, but this does not always translate into lower squared absolute error.

Because the baseline does not use semantic masks, it is unaffected by mask perturbations. Under mask noise, both segment-aware variants degrade as expected; stitched summation is more sensitive to hard routing errors, while fusion degrades more gracefully on the reported indicators (

Table 2. Inference-only mask robustness stress test on Virtual KITTI 2 validation (11-class scheme). We perturb semantic masks at test time via morphological erosion/dilation (radius r = 1 px) and random label flips (p = 0.03). Baseline does not use masks and is unchanged. Bold values indicate the best result per evaluation metric for each perturbation condition.

Condition	Model	AbsRel	${\mathit{\delta }}_1$
clean	Baseline	0.2359	0.6889
clean	No-Fusion	0.1419	0.8314
clean	Fusion	0.1597	0.8281
erode (r = 1)	No-Fusion	0.1658	0.8029
erode (r = 1)	Fusion	0.1649	0.8185
dilate (r = 1)	No-Fusion	0.2159	0.7462
dilate (r = 1)	Fusion	0.1842	0.8099
flip (p = 0.03)	No-Fusion	0.2960	0.7522
flip (p = 0.03)	Fusion	0.1778	0.7927

). We observed the same qualitative trend under stronger boundary perturbations (e.g., r = 2).

A class-wise breakdown in Figure 2 supports the same conclusion. Stitched summation is the most reliable composition for absolute depth accuracy: it consistently reduces per-class RMSE and achieves the best global RMSE/SqRel with zero overhead. In contrast, learned fusion offers a modest gain in ${\delta }_1$ but is less stable across categories and can introduce larger errors in specific classes.

Qualitative comparisons are shown in Figure 3, Figure 4 and Figure 5 for the 3-, 6-, and 11-class schemes, respectively. Relative to the baseline, both segment-aware variants tend to preserve sharper depth discontinuities aligned with semantic boundaries, reducing leakage across adjacent regions. The absolute-error maps (computed excluding Sky pixels) show lower error in many areas; however, fusion can occasionally introduce local deviations, consistent with the per-class RMSE trends in Figure 2.

5.2. Model Complexity and Runtime

To characterise the computational cost of the proposed architecture,

Table 3. Model complexity on Virtual KITTI 2 at 256 × 256 resolution. Params denotes the number of trainable parameters and FLOPs are the approximate multiply–add operations per forward pass.

Model	Classes	Params (M)	FLOPs (G)
Baseline	-	0.91	6.88
Segment-aware (Fusion)	3	2.85	38.04
Segment-aware (No-Fusion)	3	1.76	18.50
Segment-aware (Fusion)	6	4.62	56.55
Segment-aware (No-Fusion)	6	3.52	37.01
Segment-aware (Fusion)	11	7.55	87.39
Segment-aware (No-Fusion)	11	6.46	67.85

reports parameter counts and Floating-Point operations (FLOPs) per 256 × 256 image for the baseline and the segment-aware variants. For real-time considerations, we report inference cost in FLOPs (hardware-independent) relative to the single-branch baseline. The results show that model size and FLOPs grow approximately linearly with the number of semantic classes, reflecting the one-branch-per-class design. Even the largest 11-class configuration remains below 8M parameters. Compared to stitched summation, the fusion variant adds additional parameters and computation. In our implementation, most of this cost is driven by the patch-token grid and the attention blocks, while the input projection introduces a smaller dependence on the number of semantic classes N. This quantifies the trade-off: parameter-free summation is the most efficient variant, while learned fusion incurs moderate extra computation that may improve accuracy for some metrics.

6. Discussion

Our experiments isolate a simple architectural question: given accurate semantic masks, does decomposing monocular depth estimation into class-specific experts improve depth quality relative to a single global predictor trained under the same data and optimisation, and is a learned fusion stage necessary? Across all segmentation schemes, the answer to the first part is consistently positive. Both segment-aware variants improve the baseline on the full suite of standard metrics in Table 1, and the qualitative examples in Figure 3, Figure 4 and Figure 5 show noticeably sharper depth discontinuities at semantic boundaries.

Where the gains come from. The per-class design reduces interference between unrelated regions (e.g., vegetation vs. road vs. vehicles) by restricting each branch to a narrower appearance–geometry distribution. This is reflected in the per-class RMSE bars in Figure 2, where stitched summation (No-Fusion) reduces error for most categories compared to the baseline. The qualitative panels further support this interpretation: boundary leakage visible in the baseline is typically reduced when predictions are produced by class-specific experts and gated by the corresponding masks.

Learned fusion vs. stitched summation. The second part of the question is more nuanced. While the fusion module can improve strict threshold accuracy (most clearly ${\delta }_1$), it is not consistently better on absolute error metrics such as RMSE and SqRel. In our controlled setting, the parameter-free stitched summation is the most reliable composition for absolute accuracy: it achieves the lowest RMSE/SqRel for the 3- and 6-class schemes, and the best overall relative/log error profile (AbsRel, SqRel, RMSE_log) for the 11-class scheme (Table 1). This supports the practical recommendation stated in the abstract: stitched summation is a strong default that matches, and in most cases improves upon, learned fusion without adding computational overhead. Our goal is to compare composition rules under a shared objective and fixed recipe, and the same qualitative pattern across the 3/6/11-class schemes supports stitched summation as the most reliable default for absolute-error criteria (RMSE/SqRel) overall, while fusion most consistently benefits ${\delta }_1$.

One plausible reason for this behaviour is that the fusion head is trained to regress depth directly from stacked proposals at a coarse patch resolution and may trade small, widespread improvements in relative agreement for occasional larger deviations in depth magnitude. This pattern is visible in the class-wise trends: fusion can be competitive on boundary-sensitive criteria (${\delta }_1$), yet can yield worse RMSE for specific categories (Figure 2). Importantly, our qualitative visualisations mask Sky consistently and use shared scales within each sample, so the observed differences are not artefacts of per-image normalisation.

Accuracy–efficiency trade-off. The complexity results in Table 3 show that the one-branch-per-class design scales approximately linearly with the number of classes. The stitched summation variant adds no parameters beyond the experts and requires only a mask-gated sum, whereas learned fusion introduces additional attention and convolutional layers. Given that summation is also the most stable option for RMSE/SqRel, this efficiency profile strengthens its role as the default choice when absolute depth accuracy and runtime are the primary constraints.

Limitations and scope. This study uses ground-truth semantic labels from Virtual KITTI 2 to isolate the effect of segment-wise decomposition under accurate masks. In real deployments, masks will be predicted and therefore imperfect; to probe sensitivity to imperfect masks without retraining, we additionally perform an inference-only stress test by perturbing the ground-truth masks at test time via erosion/dilation (r = 1 px) and random label flips (p = 0.03) (Table 2). As expected, boundary misalignment and small-region errors act primarily as hard routing noise for stitched summation (No-Fusion), causing leakage across adjacent classes near object boundaries, while class confusion routes pixels to the wrong expert and introduces larger inconsistencies. The fusion variant is more tolerant to these perturbations because it can blend competing proposals using disagreement cues, though severe mask errors can still degrade performance since the per-class proposals are themselves mask-conditioned. A broader robustness study with predicted masks from external segmenters (e.g., Mask R-CNN or SAM) remains an important next step and is left for future work. We do not evaluate on real-world datasets (e.g., KITTI or NYU-Depth V2) to keep the analysis focused on isolating the effect of segment-wise decomposition under reliable masks; real-world validation with predicted masks is left for future work. In addition, our experts are intentionally lightweight and trained from scratch on a single synthetic dataset. Because the design is backbone-agnostic, a promising direction is to instantiate each branch with stronger depth backbones (including pretrained models) and study the interaction between semantic decomposition and model scale. An additional direction is to constrain learned fusion to operate as a bounded residual correction on top of stitched summation (instead of predicting depth directly), which may reduce occasional large deviations while retaining the strong RMSE/SqRel behaviour and efficiency of summation.

7. Conclusions

We studied segment-aware monocular depth estimation as a controlled architectural intervention: instead of a single global predictor, depth is estimated by lightweight class-specific ViT experts whose outputs are composed into a final depth map. On Virtual KITTI 2, segment-aware modelling consistently improves standard depth metrics over a single-branch baseline trained under the same data, optimiser, and training recipe and produces qualitatively sharper discontinuities at semantic boundaries. Quantitatively, our best segment-aware configuration improves AbsRel from 0.243 to 0.152 and RMSE from 11.952 to 9.101 on Virtual KITTI 2.

A central finding is that the simplest composition rule is also a strong performer. The parameter-free stitched summation of mask-gated expert predictions matches, and in most cases improves upon, a learned cross-attention fusion module on the key absolute error criteria (RMSE, SqRel), while adding no computational overhead. Learned fusion can provide modest gains in strict threshold accuracy (${\delta }_1$), but it is less stable class-wise and does not reliably reduce squared absolute error.

Overall, these results suggest a practical recipe: use segment-wise decomposition to obtain sharper boundaries and better accuracy, and adopt stitched summation as the default fusion strategy; enable learned fusion only when threshold accuracy is the primary target. A key limitation of this study is that it assumes accurate semantic masks and uses a multi-expert design whose compute scales with the number of classes; future work will evaluate robustness under predicted/noisy masks and explore more efficient variants (e.g., shared encoders with class-specific heads), as well as extend the framework to stronger backbones and real-world datasets.