A Graph Laplacian Regularizer from Deep Features for Depth Map Super-Resolution

Abstract

Current depth map sensing technologies capture depth maps at low spatial resolution, rendering serious problems in various applications. In this paper, we propose a single depth map super-resolution method that combines the advantages of model-based methods and deep learning approaches. Specifically, we formulate a linear inverse problem which we solve by introducing a graph Laplacian regularizer. The regularization approach promotes smoothness and preserves the structural details of the observed depth map. We construct the graph Laplacian matrix by deploying latent features obtained from a pretrained deep learning model. The problem is solved with the Alternating Direction Method of Multipliers (ADMM). Experimental results show that the proposed approach outperforms existing optimization-based and deep learning solutions.

1. Introduction

Depth maps play a crucial role in various applications such as autonomous driving [1], 3D reconstruction [2], augmented reality [3], and semantic scene understanding [4]. There are two main sensing technologies to capture depth information, either passive or active. The most common approach for passive technologies is stereo reconstruction, that is, the estimation of the scene depth from two images of the scene by using stereo matching algorithms [5]. The Time-of-Flight (ToF) camera [6] and structure light scanner [7] are two representative depth sensors for real-time active depth sensing. However, due to hardware limitations and sensor noise, captured depth maps often suffer from low resolution and artifacts, posing serious issues for various 3D applications.

Depth map super-resolution (SR) aims to reconstruct high-resolution (HR) depth maps from their low-resolution (LR) counterparts. Similar to image SR, depth map SR is an ill-conditioned inverse problem, requiring additional information to be solved. In general, depth cameras are equipped with an additional RGB sensor that can capture color images with a resolution higher than that of depth maps. Most of the existing works apply guided depth map SR, that is, they employ the HR color counterpart as guidance for the SR task. Although color images are useful for depth enhancement, the edges in color images do not exactly match the edges in depth maps. Moreover, in low-light conditions, the guidance image is usually noisy and may mislead the depth restoration algorithm, making color-assisted approaches less general [8,9].

To this end, some researchers address single-depth map SR by employing methods similar to single-modal image SR [8,9]. Single-modal SR tasks have been addressed by analytical, filtering and data-driven methods. Analytical methods try to solve the inverse problem by introducing various priors [10]. Among them, graph-based regularization is a popular approach [11,12,13,14,15]. Filtering-based methods estimate the depth of pixels by performing a weighted average of local pixels, and the weights are obtained by the affinity calculated from RGB–D image pairs [16,17,18]. Data-driven methods learn mapping functions from LR depth maps to HR depth maps [9,19,20]. With the rapid development of deep neural networks, deep learning methods have achieved state-of-the-art performance in depth map SR [21,22]. Despite their impressive results, DL methods require large computational resources and datasets, which can limit their applicability in real-world scenarios. Model-based deep learning designs are an alternative approach that tries to bridge the gap between deep learning and model-based solutions [15,23,24].

In this paper, we follow a model-based approach for the solution of single depth map SR, which also leverages the representation power of deep neural networks. Specifically, we consider a graphical representation of depth maps and address the depth map SR as an inverse problem. In order to incorporate prior knowledge, promote smoothness, and preserve the structural integrity of the depth maps, we propose a regularization of the problem at a feature level. We extract latent features from LR depth maps using a pretrained neural network model and apply graph regularization by computing a graph Laplacian using the extracted features. The estimated SR depth maps are obtained as a solution to the graph-regularized problem using the Alternating Direction Method of Multipliers (ADMM) algorithm. The experimental results demonstrate the superior performance of the proposed approach against single-modal and guided SR methods.

The remainder of the paper is organized as follows: Section 2 reviews the related work. Section 3 presents the mathematical formulation and the details of the proposed algorithm. Section 4 provides the experimental setup, the datasets, the experiments, and the results of the algorithm. Section 5 investigates the impact of each component of our framework via ablation studies. Finally, Section 6 concludes the paper.

2. Related Work

2.1. Depth Map Super-Resolution

We briefly review three categories of depth map SR methods, that is, analytical or model-based methods, filtering-based methods, and data-driven methods.

Similar to color image SR, analytical approaches for depth map SR address the estimation of the HR depth map from the observed LR counterpart as an inverse problem and introduce various priors [10]. The corresponding optimization methods apply different regularization techniques. Total variation (TV) regularization aims to minimize the gradient magnitude to enforce smoothness. Although the TV constraint helps to distinguish the edge and the noise, it tends to generate “staircasing artifacts” for the smooth region [10]. Recent studies have shown the powerful capability of the graph Laplacian to deal with piecewise smooth signals, and several works have used it to exploit the smoothness of the depth map [11,12,13,14,15].

Filtering-based methods combine both computational efficiency and structure preservation. Tomasi and Manduchi [16] proposed a bilateral filtering method, which smooths images while preserving edges and considers both spatial proximity and intensity similarity. However, the fact that it relies on local neighborhoods limited the ability to capture long-range dependencies. Buades et al. [17] addressed this problem with non-local means (NLMs) filtering, which extends filtering beyond local neighborhoods by leveraging self-similar patches across the entire image. While NLMs filtering enhances noise reduction and texture preservation, it suffers from high computational costs and potential artifacts in complex structures. To improve efficiency, He et al. [18] proposed an edge-aware approach that takes advantage of a high-resolution guidance image to improve depth map super-resolution.

Data-driven solutions are an alternative approach to address SR tasks and include dictionary learning and deep learning methods. Dictionary learning methods rely on sparsity assumptions and try to learn the correlation between the LR and the HR space from a set of training image pairs [9]. Wang et al. [19] introduced a depth map super-resolution technique that integrates multi-directional dictionary learning with autoregressive (AR) modeling. This method involves training multiple dictionaries, each corresponding to specific geometric directions, to effectively represent directional depth patches. Tosic and Drewes [20] proposed a method for learning overcomplete dictionaries that jointly represent image intensity and scene depth. They developed a novel Joint Basis Pursuit (JBP) algorithm to identify related sparse features across these two modalities, allowing for effective depth inpainting and improved 3D scene representation.

Deep learning (DL) methods rely on large amounts of data to find a direct mapping from the LR observations to the HR ground truth. Pioneer DL designs for general image SR include SRCNN [21], which demonstrated the potential of CNN data-driven approaches, ESRGAN [22], which relied on generative adversarial neural networks, RESNET [25] with its variants (RESNET50, RESNET101, etc.), which introduced residual learning, and U-net [26], which showed that skip connections are useful for recovering fine details. State-of-the-art DL models for depth map SR include a multi-scale fusion model proposed in [27] where the multi-scale guided features are obtained by a visual geometry group (VGG)-like neural network, a multi-modal attention-based fusion model [28], a high-frequency guidance network that employs the octave convolution [29], and the model presented in [30], which uses a guidance image only at the training stage. Super-Resolution Graph Attention Networks (SRGATs) [31] introduced a graph-based deep learning framework for single-image SR, utilizing Graph Attention Networks (GATs) to model complex and non-local relationships between pixels. By representing the image as a graph and applying attention mechanisms, the method enhances the reconstruction of high-frequency details, leading to improved visual quality in the outputs.

2.2. Graph-Based Representations

In graph-based representations, we assume that the signal is represented in the form of a weighted undirected graph $\mathcal{G}$, and similarities in the signal are expressed by the edges $\mathcal{E}$ and the respective weights encoded in the adjacency matrix W. Graph-based representations provide a powerful tool in computer vision, as they capture complex image structures and relationships between pixels [14,32]. In image processing, typically, the nodes of the graph correspond to image pixels, and the graph Laplacian is used to express similarity constraints between the pixels. Graph-based techniques have been widely applied in various image processing tasks, including segmentation, denoising, and super-resolution, due to their ability to model spatial dependencies and structural patterns [33]. Spectral graph theory has been leveraged to design effective regularization strategies that enhance feature preservation and structural coherence in reconstructed images. Graph-based regularization has been incorporated both in model-based approaches [32] and deep learning designs [14,15].

Concerning depth map reconstruction, a graph Laplacian model exploiting prior information about the depth image and the corresponding color image was formulated in [11]. Instead of constructing the graph with pixels, the authors of [12] proposed to construct the graph with a group of similar patches. In [13], the edge weight distribution of an area with sharp edges was considered to be a bimodal distribution. A reweighted graph Laplacian regularizer was proposed to preserve sharp edges and promote the bimodal distribution of edge weights. In [15], the graph Laplacian was learned from their data, and the regularized problem was solved by a DL design in an end-to-end manner.

Building upon these foundations, we consider a graph-based representation of the depth maps and compute the graph Laplacian matrix using latent features. Following an approach similar to [15], we obtain the latent representations from a deep neural network model. Unlike [15], where the latent features and the inverse mapping were learned from their data in an end-to-end learning framework, we use a pretrained model to extract the features and estimate the HR depth map by employing the ADMM algorithm to solve the corresponding graph-regularized optimization problem. We compare our results with state-of-the-art ADMM-based methods performing single depth map SR with other types of regularization [34,35]. We also compare our work with state-of-the art guided depth map SR methods that use a neural network as a graph regularizer [15]. The experiments show that our method yielded the best results in terms of the root mean squared error (RMSE) and visual quality.

3. Depth Map Super-Resolution Using the Admm Algorithm and Graph-Based Regularization

We address depth map SR as a graph-based regularized optimization problem. We considered a graphical representation of depth maps, extracted latent features from LR depth maps using a pretrained neural network model, and computed a graph Laplacian using the extracted features. We experimented with different feature extractors. The obtained problem was solved with ADMM.

3.1. Depth Map Super-Resolution as an Inverse Problem

Consider $\mathbf{y}\in {\mathbb{R}}^{N_1{N}_2}$ as the vectorized form of an observed $N_1\times N_2$ depth map (LR) and $\mathbf{x}\in {\mathbb{R}}^{S^2{N}_1{N}_2}$ as the unknown HR depth map, assuming upscaling by a factor S. Then, the degrading process is described by the following equation:

$$\mathbf{y}=A\mathbf{x}+ϵ,$$

(1)

where $A\in {\mathbb{R}}^{N_1{N}_2\times S^2{N}_1{N}_2}$ is the degradation matrix, and $ϵ$ is the additive noise. Since there can be various HR depth maps with a slight difference in camera angle, illumination, material properties, and other variables for a certain LR depth map, (1) is an ill-posed inverse problem [10]. Seeking a unique solution, we need additional prior knowledge which we incorporate into the problem with regularization, that is,

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{min}{\parallel \mathbf{y}-A\mathbf{x}\parallel }_2^2+\lambda R\left(\mathbf{x}\right),$$

(2)

where $R\left(\mathbf{x}\right)$ is a regularization term weighted by $\lambda >0$.

Problem (2) is of the form

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{min}[g\left(\mathbf{x}\right)+h\left(\mathbf{x}\right)],$$

(3)

where $g\left(\mathbf{x}\right)={\parallel \mathbf{y}-A\mathbf{x}\parallel }_2^2$ is the fidelity term that enforces consistency with the observed data, and $h\left(\mathbf{x}\right)=\lambda R\left(\mathbf{x}\right)$ is the regularization term. We can rewrite (3) as follows:

$$\widehat{\mathbf{x}}=arg\underset{\mathbf{x}}{min}[g\left(\mathbf{x}\right)+h\left(\mathbf{z}\right)],\mathrm{s}.\mathrm{t}.\mathbf{x}-\mathbf{z}=0,$$

(4)

and use the ADMM algorithm for its solution [36]. We formulate the augmented Lagrangian as follows:

$${\mathcal{L}}_p(\mathbf{x},\mathbf{z},\mathbf{u})=g\left(\mathbf{x}\right)+h\left(\mathbf{z}\right)+{\mathbf{u}}^{\top }(\mathbf{x}-\mathbf{z})+{\displaystyle \frac{\rho }{2}}\left|\right|\mathbf{x}-{\mathbf{z}\left|\right|}_2,$$

(5)

where $\mathbf{u}$ is the dual variable, and $\rho$ is the augmented Lagrangian parameter [36]. Then, at the k-th iteration of ADMM, we perform the following: (i) We minimize the augmented Lagrangian with respect to the primal variables $\mathbf{x}$ and $\mathbf{z}$. (ii) We update the dual variable $\mathbf{u}$ with the minimizers. By performing these operations, ADMM takes the form of Algorithm 1 [36], where ${\mathrm{prox}}_{\rho g}(·)$ [${\mathrm{prox}}_{\rho h}(·)$] denotes the proximal operator of $\rho g$ ($\rho h$); the arrow ← denotes an assignment operation. The proximal operator can be calculated according to [36] as follows:

$${\mathrm{prox}}_{\rho g}\left(\mathbf{v}\right)=arg\underset{\mathbf{x}}{min}\{\rho g\left(\mathbf{x}\right)+{\displaystyle \frac{1}{2}}{\parallel \mathbf{x}-\mathbf{v}\parallel }_2^2\}.$$

(6)

Algorithm 1 ADMM algorithm

1: Input: ${\mathbf{u}}^0=0$, ${\mathbf{x}}^0$, and $\rho >0$ 2: for  $k=1,2,\dots ,t$  do 3:        ${\mathbf{z}}^k\leftarrow {\mathrm{prox}}_{\rho g}({\mathbf{x}}^{k-1}-{\mathbf{u}}^{k-1})$ 4:        ${\mathbf{x}}^k\leftarrow {\mathrm{prox}}_{\rho h}({\mathbf{z}}^k+{\mathbf{u}}^{k-1})$ 5:        ${\mathbf{u}}^k\leftarrow {\mathbf{u}}^{k-1}+({\mathbf{z}}^k-{\mathbf{x}}^k)$ 6: end for 7: Return:  ${\mathbf{x}}^t$

3.2. Graph-Based Regularization for Depth Map Super-Resolution

In order to promote smooth upscaling and also preserve the structural information, we consider a graphical representation of a depth map and propose a regularizer that encodes the spatial relationships between pixels by employing the graph Laplacian matrix. Assuming that each pixel in the depth map corresponds to a node in a graph and connections between neighboring pixels are represented by edges, the graph Laplacian matrix is given by

$$L=D-W,$$

(7)

where W is the adjacency matrix of the graph, and D is the degree matrix. Therefore, we define a regularizer of the form

$$R\left(\mathbf{x}\right)={\mathbf{x}}^{T}L\mathbf{x}.$$

(8)

We calculate the proximal operator for $h\left(\mathbf{x}\right)=\lambda {\mathbf{x}}^{T}L\mathbf{x}$ by solving the following optimization problem:

$${\mathrm{prox}}_{\rho h}\left(\mathbf{v}\right)=arg\underset{\mathbf{x}}{min}\{\rho h\left(\mathbf{x}\right)+{\displaystyle \frac{1}{2}}{\parallel \mathbf{x}-\mathbf{v}\parallel }_2^2\}.$$

(9)

We minimize (9) by setting the gradient to zero. Then, the calculation of the proximal operator induces to the solution of a quadratic minimization problem, which can be reformulated as a linear system:

$$(\rho I+2\lambda L)\mathbf{x}=\rho \mathbf{v},$$

(10)

where $\rho$ is the penalty parameter for the ADMM algorithm, and I is the identity matrix with shape $S^2{N}_1{N}_2\times S^2{N}_1{N}_2$. This is a linear system of equations that can be efficiently solved using modern numerical methods, such as conjugate gradient descent. The matrix $(\rho I+2\lambda L)$ is sparse, which makes these methods computationally feasible even for large-scale depth map SR tasks.

3.3. Feature-Based Graph Laplacian Matrix

Our approach considers the construction of a Laplacian matrix based on the similarity between pixels on a latent space. Following the approach of recent methods that employ pretrained neural networks [37,38,39,40], we deploy a deep feature extractor to obtain features for every pixel of the depth map and encode the similarity between features using a Gaussian kernel. For each pixel in the depth map, we consider its 4-connected neighbors. Then, the $(i,j)$ element of the adjacency matrix W of the pixels graph contains the weights of the graph edges and is determined as follows:

$$W_{ij}=exp\left(-{\displaystyle \frac{\parallel {\mathbf{f}}_{\mathbf{i}}-{\mathbf{f}}_{\mathbf{j}}{\parallel }_2^2}{{\sigma }^2}}\right),$$

(11)

where ${\mathbf{f}}_{\mathbf{i}}$ and ${\mathbf{f}}_{\mathbf{j}}$ are the feature vectors with size equal to fdim, and $\sigma$ is a scaling parameter to control the sensitivity of the similarity to feature differences. For non-neighboring pixels, we set $W_{ij}=0$.

As a deep feature extractor, we employed a pretrained neural network. Since we wanted to learn per pixel features and our task involved spatial dependencies, we explored different encoder–decoder deep learning models that have been used successfully in dense prediction tasks, that is, tasks that require representations at a pixel level. Specifically, we investigated the use of the following models.

DeepLabv3 [41] is an excellent feature extraction method for pixel-level information and has been deployed in tasks like semantic segmentation. The model captures multi-scale contextual information by applying dilated (atrous) convolutions with different rates and can understand both local and global features. This is important when working with complex structures in depth estimation tasks.

LinkNet [42] is designed to be lightweight while maintaining strong feature extraction performance through residual connections. Although simpler than the rest of the models tested, it has proven effective in segmentation tasks.

PAN [43] combines spatial attention mechanisms with multi-scale feature extraction. The model includes both global context and fine detail, which are valuable components when dealing with predictions like depth map estimation. Its attention mechanisms help refine features at different levels, making it an interesting candidate for our comparisons.

Finally, U-Net [26] has shown an excellent performance in tasks requiring detailed structure preservation. Its skip connections and balanced encoder–decoder architecture allow for rich feature extraction while retaining spatial precision. All the aforementioned models have an encoder–decoder architecture. We obtained a deep feature representation of a considered depth map as the output of the decoder part. Concerning their implementation, all encoders can be deployed with different backbone neural network blocks, e.g., ResNet34, ResNet50, etc. We tested various encoder backbones and ultimately selected ResNet-50 pretrained on ImageNet, as it consistently yielded the best results in our experiments.

The whole process for the construction of the Laplacian matrix is depicted in Figure 1. The feature extractor receives as input a bicubic interpolated depth map and computes features with dimensions equal to fdim $\times S{N}_1\times S{N}_2$, with fdim = 64. After computing the adjacency matrix, the degree matrix D is calculated as

$$D_{ii}=\sum _j{W}_{ij}.$$

(12)

Finally, the Laplacian matrix is obtained from (7).

Figure 2 provides a visual summary of the complete workflow of our proposed depth map super-resolution method. Starting from a low-resolution depth map, we first extract deep feature representations using a pretrained encoder–decoder neural network. These features capture semantic and spatial information, which is used to define the affinity between pixels in a graph structure. The resulting graph adjacency matrix allows us to compute the Laplacian, which acts as a regularizer promoting smoothness and structure preservation. The reconstruction step solves a convex optimization problem using the ADMM algorithm, integrating both the fidelity to the observed depth values and the structural priors encoded in the graph. The final result is a refined, high-resolution depth map.

Our method integrates graph-based regularization directly into the ADMM framework, and unlike learning-based methods that require extensive training on large datasets, our method does not include any task-specific training or fine-tuning. This significantly reduces the need for large-scale training datasets and allows for broader applicability without the overhead of specialized training. Moreover, we do not use complementary information from a guided color HR image; we only use a single depth map as input.

4. Results and Evaluation

In this section, we investigate the performance of the proposed method by providing numerical and visual experimental results. We performed two categories of experiments. The first involved the deployment of different feature extractors which determined the selection of the proposed model. The second involved the evaluation of the proposed framework against state-of-the-art methods.

4.1. Experimental Setup

We conducted experiments on two widely used depth map datasets, namely, DIML and NYUv2. DIML is a dataset which contains both indoor and outdoor real-world depth maps; we focused on the indoor depth maps. NYUv2 also contains indoor depth maps captured from RGB–D sensors. The experiments involved 5030 samples from the DIML dataset and 600 samples from the NYUv2.

For our experiments, we used depth map patches of size $256\times 256$. We also rescaled the input and output depth maps in the range $[0,1]$. For $\times 4$ upsample, the parameters of the proposed ADMM method were set as follows: $\lambda =1.61$, $\rho =0.11$, and $\sigma =2.71$. For the $\times 8$ upsample, we used $\lambda =1.01$, $\rho =0.11$, and $\sigma =3.21$. These parameters were selected through tuning on a validation set. Specifically, we performed a grid search and evaluated performance using RMSE. The chosen values correspond to those that achieved the best performance on the validation set. All the deployed models used RESNET50 as a backbone encoder pretrained on ImageNet. We ran the algorithm for a total of 15 iterations. For numerical evaluation, we used the average root mean squared error (RMSE).

To obtain the features from the deployed feature extractors, we used the Segmentation Models PyTorch, which is a library with pretrained neural networks for image segmentation based on PyTorch (https://segmentation-modelspytorch.readthedocs.io/en/latest/docs/api.html#api (accessed on 29 April 2025)). For the code implementation, we used the Scientific Computational Imaging Code (SCICO) [44], which is a powerful open-source software framework designed for solving imaging and inverse problems using state-of-the-art computational algorithms. The hardware setup we used was the following: cpu Intel i3 12100, 16 GB ram, and MSI Geforce RTX 4060 graphics card.

4.2. Performance Comparison Across Selected Models

We present results of the proposed approach by deploying four well-established deep learning models for feature extraction, that is, U-net [26], DeepLabv3 [41], LinkNet [42], and the Pyramid Attention Network (PAN) [43]. The datasets were processed in the same way for all models to ensure a fair and unbiased comparison.

The numerical results obtained for each model are presented in

Table 1. Comparison of U-Net with other segmentation models on NYUv2 and DIML datasets in terms of RMSE.

	U-Net [26]	DeepLabV3 [41]	LinkNet [42]	PAN [43]
$\times 4$ NYUv2	0.015409	0.015611	0.015600	0.015612
$\times 8$ NYUv2	0.026526	0.027544	0.027555	0.027546
$\times 4$ DIML	0.015288	0.015473	0.015462	0.015473
$\times 8$ DIML	0.025857	0.026851	0.026865	0.026852

. For $\times 4$ upsampling, all models demonstrated a high performance level; however, U-Net consistently outperformed the others. For $\times 8$ upsampling, the difference between U-Net and all other models became noticeable, showcasing that for higher upsampling rates, the rest of the models performed worse.

In Figure 3, we present visual results to compare the reconstructed depth maps for the four tested models at upsampling factors $\times 4$ and $\times 8$. As for the $\times 4$ case, the visual differences between the models are not particularly pronounced. This aligns with the RMSE results, where all the models performed similarly without noticeable differences. However, for $\times 8$ upsampling, the differences became more evident. All other models appeared slightly noisier, less consistent, and with more artifacts, whereas the U-Net model maintained smooth transitions. These visual observations confirm that U-Net is more effective under more challenging upsampling conditions.

The results we obtained reinforced our choice to choose U-Net as the foundational model for our work. While more complex architectures like DeepLabV3 [41] and PAN [43] offer powerful context-aware mechanisms, they did not match the accuracy of the U-Net. Also, although more simplistic architectures like LinkNet [42] could offer lightweight and accurate features, they could not produce results as good as U-Net.

4.3. Comparison with Other Methods

To assess the effectiveness of our method, we compared it against state-of-the-art ADMM-based methods that used different regularizers. Specifically, we compared it with Plug-and-Play (PnP) methods like DnCNN-ADMM [45], which is a method integrating deep learning within ADMM, using the DnCNN denoiser for regularization. We also compared it with BM3D-ADMM [35], which leverages non-local self-similarities to remove noise and enhance textures, and TV-ADMM [34], which enforces smoothness in images by using total variation. We further made a comparison against the guided depth map SR method presented in [15], which applies graph-based regularization using a deep learning framework. Finally, we conducted experimental comparison on the Deep Attentional Guided Image Filtering (DAGF) method [46], which uses attention-based filters to adaptively transfer structural information from a guidance image to a target image, enabling accurate and edge-aware image restoration across tasks like super-resolution and texture removal. For visual comparison, we also included the results from bicubic upsampling.

Table 2. Comparison of methods on NYUv2 and DIML datasets in terms of RMSE.

	Single Depth Map SR				Guided Depth Map SR
	Proposed	DnCNN-ADMM [45]	TV-ADMM [34]	BM3D-ADMM [35]	DAGF [46]	de Lutio et al. [15]
$\times 4$ NYUv2	0.0154	0.0188	0.0232	0.0273	0.0141	0.0203
$\times 8$ NYUv2	0.0265	0.0338	0.0413	0.0666	0.0292	0.0277
$\times 4$ DIML	0.0152	0.0161	0.0212	0.0242	0.0201	0.0127
$\times 8$ DIML	0.0258	0.0303	0.0393	0.0617	0.0309	0.0187

presents average results for $\times 4$ and $\times 8$ upsampling factors. As can be seen, the results indicate that our method outperformed all the single-modal ADMM-based methods used as a baseline. On the NYUv2 dataset, it also outperformed the state-of-the-art DL method presented in [15], which employs a guidance HR color image. Since our method achieved better results than [15], it follows that it also outperformed all other methods used for comparison in [15], that is, DKN [47], FDKN [47], and FDSR [29]. We also observed that in most cases ($\times 8$ upsampling rate on NYUv2 dataset and $\times 4,\times 8$ upsampling rates on DIML dataset) our method outperformed by a higher margin the second state-of-the-art DL method presented in [46], for which we made a comparison, and only in the $\times 4$ upsampling rate on the NYUv2 dataset did it perform worse, but with a small difference. In summary, our method outperformed all existing single-modal ADMM-based methods, whereas we remained competitive and in some cases outperformed the state-of-the-art DL methods that used the RGB image as side information.

Note that our method does not need any adaptation to the data and instead relies on a pretrained network integrated into an ADMM framework. As a result, as seen in

Table 3. Comparison of the considered methods in terms of execution time (seconds). For the deep learning methods [15,46], we only consider testing time. Execution time does not depend on the upsampling factor.

	Proposed	DnCNN-ADMM [45]	TV-ADMM [34]	BM3D-ADMM [35]	DAGF [46]	de Lutio et al. [15]
NYUv2	8.15	1.75	1.21	8.84	0.05	0.11
DIML	8.30	1.3	1.99	9.5	0.09	0.08

, while the per-image execution time may appear higher than that of some deep learning methods (e.g., DAGF [46] and the approach by de Lutio et al. [15]), those reported times only reflect inference and omit the substantial training cost required to obtain the model. In contrast, our method leverages a fixed pretrained model and does not require any retraining, making it more practical and computationally efficient in scenarios where training resources are limited or when applying the method to new domains. On the other hand, compared to the other ADMM variants, our method has a slightly higher per-image processing time compared to TV-ADMM and DnCNN-ADMM due to the computation of the Laplacian matrix that returned better image quality. In comparison to BM3D-ADMM, we have a lower execution time.

While the numerical results provide an objective measure of performance, the visual quality is also a significant aspect for depth map SR. In Figure 4, we present the reconstructed depth maps for $\times 4$ and $\times 8$ upsampling factors and different methods. Specifically, we present (a) the HR color image, (b) the reference HR depth map patch, (c) the corresponding LR patch, the reconstructed HR depth maps obtained with (d) bicubic interpolation, (e) BM3D-ADMM [35], (f) the method of de Lutio et al. [15], (g) DnCNN-ADMM [45], (h) TV-ADMM [34], (j) DAGF [46] and (i) our method.

We also include additional visualizations presented in Figure 5 in the form of error maps, where darker regions (black) indicate low reconstruction error, and brighter colors—ranging from red to orange—highlight areas of higher error. These maps provide further insight into the spatial distribution of reconstruction inaccuracies across different methods. The first three rows correspond to the $\times 4$ upsampling rate, while the last three illustrate results for the $\times 8$ rate. As shown, our method consistently produced the lowest reconstruction error across all scenes, with noticeably fewer high-error regions compared to the other approaches.

As can be seen, our method demonstrates sharper edge preservation and better depth consistency in structured regions and object boundaries. We observe that the TV-ADMM method produces overly smoothed and blurry depth maps that lose lots of information of depth and also quality. BM3D and DnCNN produced better results compared to TV-ADMM, but they also have the same problem, that is, the depth maps are smoother than expected, and they do not have sufficient details. We also observed that BM3D tended to introduce residual noise in texture-heavy regions. The method of De Lutio et al. [15], which uses a guidance HR image, outperformed the other baseline methods, likely due to the additional learned weights enhancing depth variations. However, our method reconstructed the image with finer details and fewer artifacts and provided a good balance between sharpness and smoothness.

5. Ablation Study

In this section, we present the ablation studies we implemented. Specifically, we analyzed the contribution of key components in our proposed graph-based SR framework. Our study initially focused on varying the number of output fdim in U-Net, which affects the capacity of the neural network. Then, we conducted experiments on various number of ADMM iterations to investigate the convergence and also the output of the framework numerically and visually. Finally, we explored the employment of complementary information from a guidance RGB image, which we used as side information.

Experimental Setup and Results

We performed experiments on the NYUv2 and DIML datasets for both $\times 4$ and $\times 8$ upsampling factors, using RMSE as an evaluation metric. As a baseline, we used the single depth map SR framework, deploying a U-Net feature extractor with fdim = 64, and a total of 15 iterations of the ADMM algorithm. In order to test how each of the numbers of fdim values and the number of iterations affect the framework, we altered one variable at a time and kept the others unchanged.

In

Table 4. Results of different number of fdim values of U-Net.

fdim	64	128	256
$\times 4$ NYUv2	0.01539	0.01539	0.01539
$\times 8$ NYUv2	0.02652	0.02654	0.02656
$\times 4$ DIML	0.01528	0.01528	0.01527
$\times 8$ DIML	0.02585	0.02588	0.02590

, we observe that changing the number of output fdim values in U-Net had an insignificant impact on the RMSE across both datasets and scaling factors. The performance remained nearly the same for 64, 128, and 256 fdim values. The model with 64 fdim values achieved in most cases either the best or equivalent results compared to larger values of the number of fdim values. This suggests that increasing the number of output fdim values does not provide any benefit, while it increases the computational and memory demands. Therefore, choosing 64 as the number of fdim values is the most efficient choice.

Next,

Table 5. Comparison of guided vs. single depth map SR.

Side Info	Yes	No
$\times 4$ NYUv2	0.01540	0.01540
$\times 8$ NYUv2	0.02665	0.02652
$\times 4$ DIML	0.01530	0.01528
$\times 8$ DIML	0.02598	0.02585

showcases the effect of an HR guidance color image as side information. The input depth map and the corresponding RGB image were concatenated into a 3D array of size $4\times height\times width$ of the depth map. The concatenated input was passed to the feature extractor, and the graph Laplacian matrix was constructed as described in our proposed framework. The results indicate that including side information offered no measurable improvement, and in some cases, it performed slightly worse. For example, in the $\times 8$ upsampling rate on both NYUv2 and DIML, removing side information led to better RMSE scores. This indicates that the network is able to extract sufficient information from the single-input modality (depth map), and adding the corresponding RGB image as side information introduces more noise rather that useful guidance. As a result, we conclude that side information is unnecessary for this task in our framework.

Finally,

Table 6. Results of different numbers of ADMM iterations.

Iterations	5	10	15	20
$\times 4$ NYUv2	0.01551	0.01540	0.01540	0.01540
$\times 8$ NYUv2	0.02781	0.02678	0.02652	0.02654
$\times 4$ DIML	0.01535	0.01529	0.01528	0.01529
$\times 8$ DIML	0.02717	0.02615	0.02585	0.02583

presents the impact of different numbers of ADMM iterations. We observed a consistent improvement in the RMSE as the number of iterations increased from 5 to 15. However, beyond 15 iterations, the performance became stable, with minimal or no further improvement. This was especially evident in the $\times 4$ NYUv2 case, where the RMSE remained constant from 10 to 20 iterations. Similarly, for DIML at $\times 8$, the difference between 15 and 20 iterations was very small. Therefore, 15 iterations appear to offer the best tradeoff between reconstruction quality and computational cost. Using more iterations increased runtime without clear benefits.

6. Conclusions

In this work, we addressed single depth map SR with a model-based approach incorporating priors of the form of a graph-based regularizer. The obtained optimization problem was solved with the ADMM algorithm. The proposed method relied on a pretrained deep neural network model to capture the latent features of the considered depth maps and estimate the regularizer. The experimental results on two benchmark datasets have shown that the proposed method outperformed other ADMM-based solutions as well as deep learning-based guided SR approaches in terms of both the numerical and visual results.