UniRender: Reconstructing 3D Surfaces from Aerial Images with a Unified Rendering Scheme

Abstract

While recent advances in the field of neural rendering have shown impressive 3D reconstruction performance, it is still a challenge to accurately capture the appearance and geometry of a scene by using neural rendering, especially for remote sensing scenes. This is because both rendering methods, i.e., surface rendering and volume rendering, have their own limitations. Furthermore, when neural rendering is applied to remote sensing scenes, the view sparsity and content complexity that characterize these scenes will severely hinder its performance. In this work, we aim to address these challenges and to make neural rendering techniques available for 3D reconstruction in remote sensing environments. To achieve this, we propose a novel 3D surface reconstruction method called UniRender. UniRender offers three improvements in locating an accurate 3D surface by using neural rendering: (1) unifying surface and volume rendering by employing their strengths while discarding their weaknesses, which enables accurate 3D surface position localization in a coarse-to-fine manner; (2) incorporating photometric consistency constraints during rendering, and utilizing the points reconstructed by structure from motion (SFM) or multi-view stereo (MVS), to constrain reconstructed surfaces, which significantly improves the accuracy of 3D reconstruction; (3) improving the sampling strategy by locating sampling points in the foreground regions where the surface needs to be reconstructed, thus obtaining better detail in the reconstruction results. Extensive experiments demonstrate that UniRender can reconstruct high-quality 3D surfaces in various remote sensing scenes.

1. Introduction

Reconstructing 3D surfaces of a scene from multi-view imagery is one of the key objectives of remote sensing and photogrammetry, as accurate 3D models have a wide range of applications in fields such as urban planning, 3D GPS navigation, and environmental simulation.

Given multi-view input images, traditional surface reconstruction frameworks [1,2,3,4] first use structure-from-motion (SFM) [5] to estimate camera parameters and to reconstruct sparse 3D points.Then, multi-view stereo (MVS) [4,6] is applied, to estimate dense depth maps for each view, and then all the depth maps are fused into a dense point cloud. Finally, the surface reconstruction method, e.g., Poisson surface reconstruction [7], is performed, to reconstruct surface meshes from this point cloud. However, reconstructing complete mesh surfaces from point clouds is challenging, due to the lack of geometric connectivity information. This can lead to incomplete surfaces, particularly in weakly textured or sparse view contexts [8,9]. Additionally, the multi-stage reconstruction pipeline, as opposed to an end-to-end approach, may introduce cumulative errors and noise.

The latest image-based photogrammetry technique, known as neural rendering [8,10,11,12,13,14,15,16,17], employs continuous functions to represent the geometry and appearance of a 3D scene, and it encodes these continuous functions, using neural networks. Additionally, it employs differentiable rendering, to transform the 3D scene into an image, allowing for end-to-end optimization of neural network parameters, by minimizing the disparity between the rendered image and the input image. More specifically, there are two types of rendering manners in neural rendering: surface rendering [8,15,16,17], and volume rendering [13,14,18,19,20,21]. Recent works have tended to use volume rendering to reconstruct 3D surfaces, because it has stronger robustness and is less prone to getting stuck in local minima, compared to surface rendering. However, compared to the impressive performance on indoor datasets, such as DTU [22], high-quality 3D surface reconstruction using volume rendering still remains challenging for remote sensing scenes. A major reason for this dilemma is that the biases that exist in current volume rendering paradigms [13,14,21] are greatly amplified when applied to remote sensing scenes. Furthermore, the view sparsity that characterizes these scenes introduces serious geometric ambiguity.

To address these challenges, and to make neural rendering techniques available for 3D reconstruction in remote sensing, we propose our solution: UniRender, a 3D surface reconstruction method that can achieve high-quality surface reconstruction performance from aerial images. Given a 3D scene described by images, UniRender employs a signed distance field (SDF) [23] to represent the geometry of the scene, and learns the SDF by minimizing the difference between the rendered images and the input images. To achieve precise surface positioning during rendering, we first analyzed the biases that exist in current 3D surface reconstruction methods based on volume rendering [9,13,19,24] and identify the causes of them. Based on this analysis, we designed an approximately unbiased rendering scheme, by naturally unifying surface rendering and volume rendering within a single rendering framework. With such a rendering scheme, UniRender effectively eliminates biases and enables accurate surface position localization in a coarse-to-fine manner. To address the issue of inherent geometric ambiguity in neural rendering, UniRender integrates a geometric constraint mechanism that takes advantage of stereo matching. This mechanism effectively enhances the quality of the reconstructed geometry. Furthermore, to better reconstruct the details of the scene, UniRender adapts a novel spherical sampling algorithm, which can locate the sampling points to the regions that need more attention. In summary, our contribution are as follows:

• A qualitative and quantitative analysis of the biases that exist in the current volume rendering paradigm.
• An approximately unbiased rendering scheme.
• A novel geometric constraint mechanism.
• A novel spherical sampling algorithm.

2. Related Work

In this section, we offer essential background information on the two primary factors involved in learning 3D geometry from multi-view images using neural rendering: the 3D scene representation and the rendering technique connecting it to 2D observations.

2.1. Neural Implicit Representation

As mentioned in Section 1, traditional multi-view 3D reconstruction frameworks [1,2,3,4] primarily rely on explicit representations, such as point clouds or depth maps, for modeling 3D scenes. These representations are plagued by issues like discretization and limited resolution. By contrast, neural implicit representation [23,25,26] has emerged as an effective solution for 3D geometry and appearance modeling. It leverages continuous implicit functions encoded by neural networks, to represent 3D scenes. Due to its continuity and ability to achieve high spatial resolution, this representation has been widely used. As for learning-based methods, the most commonly used forms are the occupancy field [25] and the signed distance field (SDF) [23].

2.2. 3D Reconstruction with Differentiable Rendering

Unlike traditional photogrammetric techniques [3,4,6,27,28], which treat 3D reconstruction as a cross-view feature matching problem, some approaches [8,13,14,17,19,20,21] treat 3D reconstruction as an inverse rendering problem. They try to estimate the underlying 3D geometry by iteratively rendering the 3D geometry and comparing the rendered image to the input images. The key to this is differentiable rendering [17], which allows the flow of error gradients formed by color differences between the rendered and input images to be back-propagated, thus optimizing the underlying 3D geometry. Depending on the rendering method, we divide these works into two categories: surface rendering and volume rendering.

2.2.1. Surface Rendering

Surface rendering methods [8,15,16,17,29], such as IDR [8], utilize an SDF function encoded by a multi-layer perceptron (MLP), to represent the geometry of a scene. This derives the predicted color from the point where the ray intersects with the surface, which serves as the final rendered color. Subsequently, the MLP is optimized, by using a render loss computed from the rendered color and the color from the input images. However, when propagating backward, the gradient exists only at the surface intersection, leading to a confined spatial receptive field. As a result, these methods usually require mask inputs, to provide additional geometric constraints. Otherwise, the optimization will easily fall into local minima and will produce poor geometric reconstruction.

2.2.2. Density-Field-Based Volume Rendering

By contrast, volume rendering methods [13,14,18,19,20,21] derive the predicted color by integrating colors along the entire ray. Compared to surface rendering, which only optimizes color at a single point, volume rendering benefits from a longer-distance spatial receptive field. NeRF [21] is a typical volume rendering method, but is distinguished by its approach, of treating a 3D scene as a combination of a density field and a color radiance field. It approximates these two fields, using a five-dimensional function encoded by a multi-layer perceptron (MLP). This function takes 3D coordinates and view direction as input and outputs corresponding to volume density and color. By minimizing the discrepancy between the color of the input views and the color estimated through volume rendering using the density field, the density field can learn the correct geometry to satisfy the color constraints from the input views.

2.2.3. SDF-Based Volume Rendering

Despite NeRF [21]’s success in the task of novel view synthesis, the surfaces extracted from the density field are usually noisy [18], and it is difficult to determine a suitable density threshold, to extract the surface from the density field. Therefore, subsequent improvements have attempted to represent scene geometry with an SDF function [23] and have introduced an SDF-induced volume density $\sigma =\psi \left(sdf\right)$, to enable SDF-based volume rendering [13,14]. This approach allows for the extraction of smooth surfaces from the zero set of the SDF.

2.3. Issues Existing in Volume Rendering

Numerous experiments [13,14,24,30] have shown that SDF-based volume rendering is highly beneficial for recovering surfaces from 2D images, especially for indoor datasets like DTU [22]. However, achieving high-quality 3D surface reconstruction still remains challenging for remote sensing scenes, because the biases inherent in current volume rendering paradigms [13,14,21] are greatly amplified when applied to remote sensing scenes. Furthermore, the view sparsity that characterizes these scenes introduces serious geometric ambiguity. More specifically, the biases and ambiguities are as follows:

• Color bias: Given a ray, volume rendering directly optimizes the color integration along the ray rather than the surface color where the ray intersects the surface; therefore, the estimated color at the surface, i.e., the surface-rendered color, may deviate from the volume-rendered color. We note this color bias phenomenon is particularly evident in remote sensing scenes, as shown in Figure 1 (2nd row, 2nd column and 3rd column). For methods that rely on color to optimize geometry, color bias will inevitably erode their geometric constraint ability.
• Weight bias: Take the volume rendering method NeuS [13], as an example. This assumes that when approaching the surface along the ray, the weight distribution of color used to compute the color integration will reach its maximum at the position of the zero signed distance field (SDF) value, i.e., the surface position, as shown in Figure 1 (1st row, 2nd column). This assumption often holds for datasets with close-up dense views and simple geometry, such as a DTU dataset [22]. However, things are different on remote sensing datasets, in which the position with the maximum weight value usually deviates from the position with the zero SDF value, as shown in Figure 1 (2nd row, 2nd column). Such a weight bias phenomenon results in a misalignment between color and geometry, and prevents volume rendering from capturing the correct geometry.
• Geometric ambiguity: For most neural rendering pipelines [11,12,13,14], the geometry is only constrained by the color loss from a single view in each iteration, which cannot guarantee consistency across different views in the direction of geometric optimization, thus leading to ambiguity. When the input views become sparse, this ambiguity increases and results in inaccurate geometry. For remote sensing scenes, in which views are often sparse, this inherent ambiguity is undoubtedly a challenging issue.

Therefore, we aimed for a method that could incorporate both the unbiased characteristic of surface rendering and the long-distance spatial receptive field of volume rendering. UNISURF [18] has attempted to unify these two rendering methods. Its strategy is to first take a coarse sampling and to estimate surface points from the sampling points, via root-finding; it then takes a fine sampling around these surface points and performs volume rendering. The sampling interval ${△}_k$ is controlled by the formula: ${△}_k=max({△}_{max}exp(-k,\beta ),$ ${△}_{min})$, where k denotes the iteration number, $\beta$, and where ${△}_{max}$ and ${△}_{min}$ are predefined hyperparameters. However, this manner of setting a sampling interval manually is less robust. Therefore, we propose our solution. In brief, our solution is different from UNISURF [18], as follows:

• UNISURF [18] represents the scene geometry by occupancy values, while we use SDF representation and, thus, produce a higher-quality surface.
• UNISURF [18] makes a transition from volume rendering to surface rendering by gradually narrowing the sampling intervals during optimization. But the sampling interval is manually defined and controlled by hyperparameters, such as iteration steps, while our method spontaneously completes this transition along with the convergence process of the networks.

3. Preliminaries

Given a set of calibrated multi-view images, our goal is to reconstruct 3D surfaces of the scene. As a by-product, realistic views can also be achieved. An overview of our approach is shown in Figure 2. Since we employ an SDF function to represent the geometry of the scene, in this section we review the fundamentals of SDF-based volumetric rendering, so that we can better demonstrate our analysis in later sections.

3.1. Scene Representation

Let the set $\Omega \in {\mathbb{R}}^3$ represent the space occupied by objects like trees and buildings, and let $\mathcal{S}$ represent the boundary of the occupied space, i.e., the earth surface. We model the entire space, including occupied and unoccupied, with the signed distance field (SDF) function f, which maps a spatial position $\mathit{p}\in {\mathbb{R}}^3$ to a scale value,

$$f\left(\mathit{p}\right)={(-1)}^{I_{\Omega }\left(\mathit{p}\right)}\underset{\mathit{q}\in S}{min}{\parallel \mathit{p}-\mathit{q}\parallel }_2,I_{\Omega }\left(\mathit{p}\right)=\left\{\begin{array}{c}1, if\mathit{p}\in \Omega \\ 0, if\mathit{p}\notin \Omega ,\end{array}\right.$$

(1)

where $I_{\Omega }(·)$ is an indicator function of the space $\Omega$. This allows the surface $\mathcal{S}$ of the scene to be represented by the zero-level set of function f,

$$\mathcal{S}=\left\{\mathit{p}\in {\mathbb{R}}^3|f\left(\mathit{p}\right)=0\right\}.$$

(2)

In addition to geometry, the scene appearance is represented by a function c: ${\mathbb{R}}^3\times {\mathbb{R}}^3\to {\mathbb{R}}^3$ that maps a spatial position $\mathit{p}\in {\mathbb{R}}^3$ and a view direction $\mathit{v}\in {\mathbb{R}}^3$ to a color value:

$$c(\mathit{p},\mathit{v})=RGB.$$

(3)

Finally, we use the same network architecture as IDR [8], i.e., different multi-layer perceptrons (MLPs) are used to encode functions f and c. Specifically, functions f and c are encoded by MLP networks $F_{\theta }$ and $C_{\psi }$, respectively, as shown in Figure 2. This architecture facilitates the deconstruction of geometry and appearance.

3.2. Paradigm of Volume Rendering

Given a pixel and a camera position $\mathit{o}$, we denote the ray emitted by the camera and passing through the pixel as $\mathit{p}\left(t\right)=\mathit{o}+t\mathit{v}, t\ge 0$, where $\mathit{v}$ is the unit direction vector of the ray and t is the depth along the ray starting at $\mathit{o}$. With volume rendering, the color observed at the pixel along the direction $\mathit{v}$ can be estimated by color integration,

$$C(\mathit{o},\mathit{v})={\int }_0^{\infty }w\left(t\right)c\left(t\right)dt,$$

(4)

where $c\left(t\right)=c\left(\mathit{p}\right(t),\mathit{v})$ is the color estimated by function c at point $\mathit{p}\left(t\right)$ along the view direction $\mathit{v}$, and $w\left(t\right)=w\left(\mathit{p}\right(t\left)\right)$ is the color weight at point $\mathit{p}\left(t\right)$ along the ray. And $w\left(t\right)$ is computed by the standard volume rendering Formula ([21]),

$$w\left(t\right)=T\left(t\right)\sigma \left(t\right)$$

(5)

$$T\left(t\right)=exp(-{\int }_0^t\sigma \left(u\right)du),$$

(6)

where $\sigma \left(t\right)=\sigma \left(\mathit{p}\right(t\left)\right)$ is the volume density at point $\mathit{p}\left(t\right)$, and where $T\left(t\right)$ is the so-called accumulated transmittance along the ray. For simplicity, we ignore intermediate variables, and we write $c,w,\sigma$ as functions of t.

3.3. SDF-Induced Volume Density

To capture the SDF representation of the scene geometry via volume rendering, it is necessary to build an appropriate connection between the SDF value $f\left(t\right)$ and the volume density $\sigma \left(t\right)$. We employ the same strategy as NeuS [13], i.e., deriving the volume density $\sigma$ from the SDF via

$$\sigma \left(t\right)=\psi \left(f\left(\mathit{p}\right)\right)=max(-\frac{{\varphi }_s\left(f\left(\mathit{p}\left(t\right)\right)\right)}{{\Phi }_s\left(f\left(\mathit{p}\left(t\right)\right)\right)},0),$$

(7)

where ${\Phi }_s\left(x\right)={(1+e^{-sx})}^{-1}$ is a Sigmoid function, and ${\varphi }_s\left(x\right)=\left(s{e}^{-sx}\right)/{(1+e^{-sx})}^2$ is a logistic density distribution and also the derivative of ${\Phi }_s\left(x\right)$, while $s^{-1}$ is a trainable standard deviation that should tend to zero as the networks converge.

In Figure 3, we illustrate the relationship between the SDF value $f\left(t\right)$, the volume density $\sigma \left(t\right)$, and ${\varphi }_s\left(t\right)$ when the ray intersects a single plane. It can be seen that as parameter $s^{-1}$ decreases, the gradient of the volume density near the surface steepens, and ${\varphi }_s\left(x\right)$ contracts towards zero. Since color weight $w\left(t\right)$ differs only by a factor from ${\varphi }_s\left(t\right)$, i.e., $w\left(t\right)=\left|cos\theta \right|{\varphi }_s\left(t\right)$, reducing $s^{-1}$ leads to a more concentrated distribution of the color weight near the surface, where $\theta$ is the angle between the ray and the surface.

4. Method

4.1. Unified Rendering Scheme

As mentioned in Section 2.3, biases in volume rendering will lead to inaccurate geometry. Let us briefly analyze these biases, before describing our solution in detail. Given the sampling points $\mathit{P}=\left\{\mathit{p}\left(t_i\right)|t_n\le t_i\le t_f,i\in \left\{1,2,\dots n\right\}\right\}$ sampled along the ray $\mathit{p}\left(t\right)$ that is passing through a pixel, and the corresponding depth range of the sampling $[t_{near},t_{far}]$, the volume-rendered color of the pixel can be achieved by the discrete counterpart of Formula (4):

$$C_{volume}=C(o,v)=\sum _{i=1}^{n}w\left(t_i\right)c\left(t_i\right).$$

(8)

We assume that the first intersection point of the ray and the surface is $\mathit{p}\left(t^{*}\right)$, i.e., $f\left(\mathit{p}\left(t^{*}\right)\right)=0$. Then, the surface color at $\mathit{p}\left(t^{*}\right)$ along the direction $\mathit{v}$, i.e., the surface rendering color can be expressed as

$$C_{suface}=c\left(t^{*}\right).$$

(9)

With Formulas (8) and (9), the color bias between the volume-rendered color and the surface color can be derived as

$$\begin{array}{cc}\hfill {△}_c& =C_{volume}-C_{surface}\hfill \\ & =\sum _{\genfrac{}{}{0pt}{}{i=1}{i\ne j}}^{n}w\left(t_i\right)c\left(t_i\right)+w\left(t_j\right)c\left(t_j\right)-c\left(t^{*}\right)\hfill \\ & =\sum _{\genfrac{}{}{0pt}{}{i=1}{i\ne j}}^{n}w\left(t_i\right)c\left(t_i\right)+(w\left(t_j\right)-1)c\left(t_j\right)+c\left(t_j\right)-c\left(t^{*}\right)\hfill \\ & ={△}_{weight}+c\left(t_j\right)-c\left(t^{*}\right)\hfill \\ & ={△}_{weight}+{△}_{disc}.\hfill \end{array}$$

(10)

This formula indicates that the color bias ${△}_c$ can be interpreted as the sum of ${△}_{weight}$ and ${△}_{disc}$. Where ${△}_{weight}$ is the weight bias mentioned in Section 2.3 and is caused by an unreasonable weight distribution, ${△}_{disc}$ is a bias introduced by discrete sampling, $t_j$ is the depth of the point $\mathit{p}\left(t_j\right)$, and $\mathit{p}\left(t_j\right)$ is the point closest to the surface point $\mathit{p}\left(t^{*}\right)$ in the point set $\mathit{P}$.

To eliminate these biases, we propose a novel rendering scheme that unifies volume rendering and surface rendering. Specifically, we achieve this through (1) interpolated volume rendering, (2) weight regularization, and (3) unified color constraints.

4.1.1. Interpolated Volume Rendering

To eliminate ${△}_{disc}$, we first start with the camera position $\mathit{o}$ and move along the direction $\mathit{v}$. While moving, we find the first pair of adjacent sampling points that satisfies the condition

$$f\left(\mathit{p}\left(t_i\right)\right)f(\mathit{p}\left(t_{i+1}\right)<0,$$

(11)

and we denote them as $\mathit{p}\left(t_j\right)$ and $\mathit{p}\left(t_{j+1}\right)$, which means the two adjacent sampling points closest to the surface. Given $p\left(t_j\right)$ and $p\left(t_{j+1}\right)$, the first intersection point of the ray and the surface $p\left(t^{*}\right)$ can be approximated as $p\left(\widehat{t^{*}}\right)$ by linear interpolation:

$$\begin{array}{cc}\hfill & \mathit{p}\left(\widehat{t^{*}}\right)=0+t^{*}\hfill \\ \hfill & t^{*}=\frac{f\left(\mathit{p}\left(j\right)\right)t_{j+1}-f\left(\mathit{p}\left(t_{j+1}\right)\right)t_j}{f\left(\mathit{p}\left(t_j\right)\right)t_{j+1}-f\left(\mathit{p}\left(t_{j+1}\right)\right)t_j}.\hfill \end{array}$$

(12)

Subsequently, we add the point $p\left(\widehat{t^{*}}\right)$ to the point set $\mathit{P}$, to obtain a new point set ${\mathit{P}}^{\prime }=\mathit{P}\cup \mathit{p}\left(t^{*}\right)$, and we use ${\mathit{P}}^{\prime }$ to produce the final volume rendering color:

$$C_{volume}^{\prime }=C^{\prime }(\mathit{o},\mathit{v})=\sum _{i=0}^{n}w\left(t_i\right)c\left(t_i\right)+w\left(\widehat{t^{*}}\right)c\left(\widehat{t^{*}}\right).$$

(13)

As $C(t,v)$ is updated to $C^{\prime }(t,v)$, ${△}_c$ will also be updated to

$$\begin{array}{cc}\hfill {△}_c^{\prime }& =C_{volume}^{\prime }-C_{surface}\hfill \\ & =\sum _{i=1}^{n}w\left(t_i\right)c\left(t_i\right)+w\left(\widehat{t^{*}}\right)c\left(t^{*}\right)-c\left(t^{*}\right)\hfill \\ & =\sum _{i=1}^{n}w\left(t_i\right)c\left(t_i\right)+(w\left(\widehat{t^{*}}\right)-1)c\left(t^{*}\right)+c\left(\widehat{t^{*}}\right)-c\left(t^{*}\right)\hfill \\ & ={△}_{weight}^{\prime }+c\left(\widehat{t^{*}}\right)-c\left(t^{*}\right)\hfill \\ & ={△}_{weight}^{\prime }+{△}_{interp}.\hfill \end{array}$$

(14)

This formula indicates that ${△}_{weight}$ and ${△}_{disc}$ are updated to ${△}_{weight}^{\prime }$ and ${△}_{interp}$, respectively, after interpolation, where ${△}_{interp}$ is a bias caused by linear interpolation and is at least two orders of magnitude smaller than ${△}_{disc}$.

4.1.2. Weight Regularization

To alleviate the updated weight bias ${△}_{weight}^{\prime }$, we propose $L_{weight}$, to regularize the weight distribution:

$$L_{weight}=\sum _{i=1}^{n}w\left(t_i\right)\left|t_i-t^{*}\right|.$$

(15)

$L_{weight}$ aims to penalize anomalous weight distributions, such as those that are far from the surface but have large weight values, thus indirectly driving the weight distribution to shrink towards the surface. Theoretically, as $w\left(t\right)$ tends to $\delta (t-\widehat{t^{*}})$, a delta distribution centered at $\widehat{t^{*}}$, ${△}_{weight}^{\prime }$ will tend to zero, and volume rendering will produce the same result as surface rendering.

Incidentally, Mip-NeRF 360 [31] employs a regularization loss function called $L_{dist}$, which is similar to $L_{weight}$. However, $L_{dist}$ is calculated by a double integration. For a ray with n sampling points, $L_{dist}$’s time complexity is $O\left(n^2\right)$, while $L_{weight}$’s is $O\left(n\right)$.

4.1.3. Color Constraints

With $C_{volume}^{\prime }$, the color loss of volume rendering can be defined as

$$L_{vol}=\left|C_{volume}^{\prime }-C_{GT}\right|,$$

(16)

where $C_{GT}$ is the true color value at the pixel through which the ray $\mathit{p}\left(t\right)$ passes.

Based on previous interpolation operations, the surface-rendered color is readily available, i.e., $C_{surface}=c\left(t^{*}\right)$. We use it to produce another color loss:

$$L_{surf}=\left|C_{surface}-C_{GT}\right|.$$

(17)

In fact, $L_{surf}$ not only optimizes the color at the surface, but also implicitly includes the constraint ${\sum }_{i=1}^{n}w\left(t_i\right)>0$.

Ultimately, the color loss of our framework is defined as

$$L_{color}=L_{surf}+L_{vol}.$$

(18)

4.2. Geometric Constraints

To alleviate the geometric ambiguity described in Section 2.3, we introduce (1) photometric consistency constraints [27], and (2) point constraints, to supervise the learned scene geometry on the SDF representation.

4.2.1. Photometric Consistency Constraints

Let the view to be rendered be the reference view $I_r$, and let its neighboring views be the source views $\left\{I_i|i\in \left\{1,\dots ,m\right\}\right\}$. If the ray $\mathit{p}\left(t\right)$ and the pixel $I_r\left(x\right)$ in the reference view it passes through are given, we can find the surface intersection point $\mathit{p}\left(\widehat{t^{*}}\right)$, as described in Section 4.1.1. And, once $\mathit{p}\left(\widehat{t^{*}}\right)$ is found, its projections in the source views $\left\{I_i\left(x_i\right)|i\in \left\{1,\dots ,m\right\}\right\}$ can be located by a homography matrix $H_i$,

$$\begin{array}{cc}\hfill x_i& ={\mathit{H}}_{\mathit{i}}x\hfill \\ \hfill {\mathit{H}}_{\mathit{i}}& ={\mathit{K}}_{\mathit{i}}({\mathit{R}}_{\mathit{i}}{\mathit{R}}_{\mathit{r}}^T-\frac{{\mathit{R}}_{\mathit{i}}({\mathit{R}}_{\mathit{i}}^T{\mathit{t}}_{\mathit{i}}-{\mathit{R}}_{\mathit{r}}^T{\mathit{t}}_{\mathit{r}}){\mathit{n}}^T}{d}){{\mathit{K}}_{\mathit{r}}}^{-1},\hfill \end{array}$$

(19)

where ${\mathit{K}}_{\mathit{r}}$ and ${\mathit{K}}_{\mathit{i}}$, and ${\mathit{R}}_{\mathit{r}}$ and ${\mathit{R}}_{\mathit{i}}$, and ${\mathit{t}}_{\mathit{i}}$ and ${\mathit{t}}_{\mathit{r}}$ are the internal calibration matrices, rotation matrices, and the translation vectors of the views $I_r$ and $I_i$, respectively. Note that $\mathit{n}$ and d come from the plane tangent to the surface at the point $\mathit{p}\left(\widehat{t^{*}}\right)$:

$${\mathit{n}}^T\mathit{p}\left(\widehat{t^{*}}\right)+d=0.$$

(20)

Formula (19) actually describes the relationship between the projections of point $\mathit{p}\left(\widehat{t^{*}}\right)$ in different views. In addition, this relation can be extended to pixel patches. If X and $X_i$ are the patches centered at x and $x_i$, respectively, then we have $X_i=H_{i}X$.

Intuitively, if $\mathit{p}\left(\widehat{t^{*}}\right)$ is indeed a surface point, then its projections in different views should be consistent. Therefore, we construct a loss function, to supervise the surface we find by measuring the photometric consistency between X and $X_i$.

Specifically, we use normalization cross-correlation as the photometric consistency metric between patches, and this can be computed by

$$NCC(X,{\mathit{H}}_{\mathit{i}}X)=\frac{Cov(X,{\mathit{H}}_{\mathit{i}}X)}{\sqrt{Var(X,{\mathit{H}}_{\mathit{i}}X)}},$$

(21)

where $Cov$ and $Var$ denote covariance and variance, respectively. To deal with occlusions, the 4 best $NCC$ scores in $\left\{NCC(X,{\mathit{H}}_{\mathit{i}}X)|i\in \left\{1,\dots ,m\right\}\right\}$ are selected, for calculating the final photometric consistency loss [24]:

$$L_{photo}=\frac{{\sum }_{i=1}^{4}1-NCC(X,{\mathit{H}}_{\mathit{i}}X)}{4}.$$

(22)

4.2.2. Point Constraints

Consider that current neural rendering pipelines require images with known camera poses that are typically estimated from structure-from-motion (SFM) [5]. Actually, SFM also reconstructs sparse 3D points. Although these points inevitably suffer some noise, they still have some accuracy. Meanwhile, since we are using an end-to-end framework, the noise can be corrected, to a certain extent, during optimization. Therefore, we use these reconstructed points ${\mathit{p}}_{rec}$ to supervise f directly. For those scenes with sparse input views, e.g., the LEVIR-NVS scenes described by 21 images, since the time cost of executing MVS is cheap and can be limited to 10 min, we proceed to execute MVS [6] and use its reconstructed points as ${\mathit{p}}_{rec}$:

$$L_{SDF}=\sum _{\mathit{p}\in {\mathit{p}}_{rec}}\frac{1}{N}\left|f\left(\mathit{p}\right)\right|,$$

(23)

where N is the number of points that ${\mathit{p}}_{rec}$ contains. If we assume that any point $\mathit{p}$ in ${\mathit{p}}_{rec}$ is on the real surface, and that the SDF function f does capture the surface correctly, then f should satisfy $f\left(\mathit{p}\right)=0,\mathit{p}\in {\mathit{p}}_{rec}$. Therefore, $L_{SDF}$ encourages f to fit the coordinate distribution of ${\mathit{p}}_{rec}$.

Because the points generated by MVS [6] are oriented, we regard the normals of these points as additional geometric clues, and we use them to constrain the predicted normals computed with automatic differentiation on function f:

$$L_{norm}=\sum _{\mathit{p}\in {\mathit{p}}_{rec}}\frac{1}{N}\left|\frac{\nabla \mathit{p}\left(f\right(\mathit{p}\left)\right)}{{\parallel \nabla \mathit{p}\left(f\left(\mathit{p}\right)\right)\parallel }_2}-\mathit{n}\left(\mathit{p}\right)\right|.$$

(24)

4.3. Spherical Sampling

Because many BlendedMVS scenes contain a lot of background information (such as the sky), modeling the background is necessary. As NeRF++ [32] did, we divide the scene space into an inner-sphere space and an outer space. The inner-sphere space is responsible for modeling the foreground and it is represented by the functions of f and c described in Section 3.1, while the outer space is responsible for modeling the background, and it is represented by a density-based neural field.

When sampling points, previous works first sampled in a sector space and then assigned the sampling points in the inner-sphere space to the function f, and the rest to the density-based neural field. Since the number of sampling points on each ray is fixed, this approach actually reduces the sampling points in the inner space and increases the sampling points in the outer space. This goes against our intention of allocating the main attention to the foreground, because we are primarily concerned with the quality of the surface reconstruction in the inner space. Therefore, we propose to sample directly in the inner space, as shown in Figure 4b. Specifically, we achieve this by finding the depths of the intersections where the ray enters and exits the inner space, as shown in Algorithm 1 and Figure 4c.

Algorithm 1: Spherical sampling algorithm

Input: the camera position $\mathit{o}$, i.e., the starting point of the ray

the view direction $\mathit{v}$ along the ray

the spatial origin position ${\mathit{o}}^{\prime }$

the radius of the inner space r

Output: the depth range to sample along the ray $[t_n,t_f]$

1: Calculate the projection length of the vector $o-o^{\prime }$ in the ray direction: $l=\left|(\mathit{o}-{\mathit{o}}^{\prime })\mathit{v}\right|$.

2: Calculate the length of the right-side BC of the right triangle △ABC: $h={(r^2-l^2)}^{1/2}$.

3: Calculate the length of the right-side CD of the right triangle △ABC: $d={(r^2-h^2)}^{1/2}$.

4: Determine the sampling range along the ray $[t_n,t_f]$: $t_n=l-d,t_f=l+d$.

Implementation Details

Similar to [8,13], the functions of f and c are encoded by an 8-layer and a 4-layer MLP with 256 hidden units, respectively, while the function r is encoded by the same network structure as [8]. We sample 2048 rays per batch. For each ray, we first sample 32 points uniformly and then sample 96 points hierarchically. We train our model for 50k iterations with about 5 h on a single NVIDIA RTX3090 GPU.

The overall loss we use during training is defined as

$$L=L_{color}+\alpha L_{weight}+\beta L_{photo}+L_{SDF}+\gamma L_{norm}+\lambda L_E,$$

(25)

where $L_E$ is the Eikonal regularization [33] for regularizing the gradients of the SDF function. In our experiments, we empirically set $\alpha =0.1$, $\beta =0.2$, and $\gamma =\lambda =0.1$, to control the contribution of each loss to the error gradient. The patch size used to compute $L_{photo}$ was $5\times 5$, which was smaller than the patch size of $11\times 11$ used in most methods, since we found that using a large patch size might cause occlusions in remote sensing scenes.

After training, we use the marching cube algorithm [34] at ${512}^3$ resolution, to extract a triangular mesh from the function f.

5. Experiments

5.1. Exp Setting

5.1.1. Dataset

Our method was evaluated on two datasets: BlendedMVS and LEVIR-NVS. Views of these two datasets are shown in Figure 5.

The BlendedMVS [35] dataset is a large-scale dataset designed to evaluate the performance of 3D reconstruction algorithms. We selected eight scenes captured in UAV view from the low-resolution BlendedMVS dataset, to test our method. The number of images in each scene ranged from 75 to 472, and the resolution of each image was 768 × 576.

The LEVIR-NVS [20] dataset includes 16 remote sensing scenes, and its views resemble satellite views that are taken from high angles. We evaluated our method’s performance on all scenes; each scene contained only 21 RGB images with a resolution of 512 × 512 pixels, thus leading to a limited range of visual field variations.

5.1.2. Baseline

Three typical scene reconstruction solutions were compared to our approach, including the traditional MVS method, COLMAP [6], the neural-volume-rendering-based method, NeuS [13], and the neural-multiplanar-representation-based method, ImMPI [20].

For COLMAP [6], the parameter trim was set to zero, when reconstructing the meshes.

For NeuS [13], we also sampled 2048 rays per batch, and we set the number of iterations to 50k, while leaving other settings unchanged.

For ImMPI [20], we only evaluated its performance on the LEVIR-NVS dataset, because it does not consider the problem of handling scenes that contain background elements, such as sky and vistas, which are common in the BlendedMVS dataset. Additionally, we used two versions of ImMPI [20] during comparison. One version, denoted as ’ImMPI${}^{*}$’, was first pre-trained on WHU [36] and then trained with 11 views. The other version, denoted as ’ImMPI${}^{†}$’, was not pre-trained, but was trained using all 21 images.

5.1.3. Evaluation Metrics

Similar to MVSNet [37], we evaluated the surface reconstruction performance by both the distance metric [22] and the percentage metric [38]. Assuming that ${\mathit{S}}_{\mathbf{1}}$ represented the ground truth points and that ${\mathit{S}}_{\mathbf{2}}$ represented the points sampled from the reconstructed surface, the chamfer distance from ${\mathit{S}}_{\mathbf{1}}$ to ${\mathit{S}}_{\mathbf{2}}$ was considered as the evaluation metric for reconstruction accuracy (Acc):

$$\mathrm{Acc}=d_{CD}({\mathit{S}}_{\mathbf{1}},{\mathit{S}}_{\mathbf{2}})=\frac{1}{\left|S_1\right|}\sum _{\mathbf{x}\in {\mathit{S}}_{\mathbf{1}}}\underset{\mathbf{y}\in {\mathit{S}}_{\mathbf{2}}}{min}{\parallel \mathit{x}-\mathit{y}\parallel }_2.$$

(26)

The charmful distance from ${\mathit{S}}_{\mathbf{2}}$ to ${\mathit{S}}_{\mathbf{1}}$ was considered as the evaluation metric for reconstruction completeness (Comp):

$$\mathrm{Comp}=d_{CD}({\mathit{S}}_{\mathbf{2}},{\mathit{S}}_{\mathbf{1}})=\frac{1}{\left|S_2\right|}\sum _{\mathbf{x}\in {\mathit{S}}_{\mathbf{2}}}\underset{\mathbf{y}\in {\mathit{S}}_{\mathbf{1}}}{min}{\parallel \mathit{x}-\mathit{y}\parallel }_2.$$

(27)

The overall score was defined as the mean of accuracy and completeness. The percentage metric measured accuracy and completeness in a similar manner, by introducing a distance threshold, and it evaluated the overall performance by F-score.

5.2. Results

We first compared the surface reconstruction results on BlendedMVS, which are shown in Figure 6 and

Table 1. Quantitative surface reconstruction results on the BlendedMVS dataset. More details are shown in Appendix A Table A1.

Method	Distance			Threshold = 0.5
	Acc↓	Comp↓	Overall↓	Acc↑	Comp↑	F-Score↑
COLMAP	0.182	0.284	0.233	78.62	77.53	72.33
NeuS	0.334	0.292	0.313	46.37	59.79	50.86
UniRender	0.156	0.098	0.127	82.70	96.86	88.87

. It can be observed that the reconstructed surfaces from COLMAP [6] were noisy, while NeuS [13] struggled with capturing accurate geometric structures using only color constraints. By contrast, UniRender achieved a balance between precise geometry and smooth surfaces.

Then, we performed similar experiments on the LEVIR-NVS dataset, and the results are shown in Figure 7 and

Table 2. Quantitative surface reconstruction results on the LEVIR-NVS dataset. More details are shown in Appendix A Table A2.

Method	Distance			Threshold = 0.5
	Acc↓	Comp↓	Overall↓	Acc↑	Comp↑	F-Score↑
COLMAP	0.202	0.211	0.207	86.79	91.15	88.79
NeuS	0.255	0.258	0.256	62.68	81.43	68.50
UniRender	0.174	0.192	0.183	90.39	95.17	92.50

. In contrast to the BlendedMVS dataset, the scenes in LEVIR-NVS had sparse views, which caused a relative decline in COLMAP [6]’s performance, particularly in marginal regions. NeuS [13] seemed to adapt better to the LEVIR-NVS dataset, but it performed poorly in near-ground regions.

We also show the depth estimation results on the LEVIR-NVS dataset in

Table 3. Quantitative depth estimation results on the LEVIR-NVS dataset. Here, ImMPI ${}^{*}$ was first pre-trained on the WHU dataset and then fine-tuned with 11 views from the LEVIR-NVS dataset. ImMPI ${}^{†}$ was directly trained using all 21 views of the LEVIR-NVS dataset.

	ImMPI ${}^{*}$	ImMPI ${}^{†}$	NeuS	UniRender
MAE↓	0.832	1.027	0.650	0.254

and Figure 8, in which the mean absolute error (MAE) was used as the evaluation metric. It can be seen that both imMPI${}^{*}$ and imMPI${}^{†}$ had a large MAE, but that the imMPI${}^{†}$ without pre-training had the largest MAE. NeuS struggled with scenes, such as the square. And, except for Unirender, all the methods had difficulty estimating accurate depth information for the facades of high-rise buildings.

Given that UNISURF [18] is a work similar to UniRender, we further compared the 3D reconstruction results between them. As illustrated in

Table 4. Quantitative surface reconstruction results of UNISURF and UniRender.

Scene	Distance
	UNISURF			UniRender
	Acc↓	Comp↓	Overall↓	Acc↓	Comp↓	Overall↓
scene1 of BlendedMVS	0.593	0.583	0.587	0.265	0.325	0.295
scene02 of LEVIR-NVS	0.543	0.509	0.526	0.268	0.275	0.271

and Figure 9, it was evident that UNISURF [18] significantly lagged behind UniRender, in both numerical metrics and visualization.

5.3. Discussion

5.3.1. Biases Elimination Verification

To study the effectiveness of our method for bias elimination, we used NeuS [13] as the baseline, and we conducted experiments on the LEVIR-NVS dataset with different settings:

• (a) using a sampling strategy that first sampled 64 points uniformly and then sampled 64 points hierarchically.
• (b) using a sampling strategy that first sampled 32 points uniformly and then sampled 96 points hierarchically.
• (c) replacing the rendering scheme of NeuS [13] with our proposed rendering scheme, and using the same sampling strategy as (b).
• (d) adding all the geometric constraints based on (c).

As the results show in Figure 10, (a) unsurprisingly produced obvious color bias and weight bias during rendering, and outputted a low quality surface with many holes, while (b) showed that simply changing the sampling strategy did not help eliminate biases and that the reconstructed surface remained poor. Gratifyingly, with the unbiased rendering scheme, (c) significantly reduced the biases, while the holes in the reconstructed surface were filled, and buildings’ outlines were improved. Finally, with the geometric constraints, (d) further tended the weight distribution to a delta function, and details such as shrubs were shown on the reconstructed surface.

We also show the SDF distribution and textured surfaces with settings (a,d) in Figure 11. Comparing (a,b) and (c,d), it can be seen that the proposed rendering scheme not only produces reasonable colors at surfaces, but also facilitates the fitting of the SDF distribution to the remote sensing scenes. Because of the remote sensing scenes, we wish the SDF distribution to have only one zero value in the vertical direction.

5.3.2. Optimization of the Parameters $s^{-1}$

As described in Section 3.3, $s^{-1}$ is a trainable parameter, which controls the slope of the density distribution near the surface and also is the standard deviation of the weighting distribution. In addition, it reflects how well the networks fit a scene. When training is complete, the smaller the parameter, the steeper the slope of the density function around the surface, and the more the weight distribution tends towards a delta distribution. This indicates that the network fits the scene better. Figure 12 illustrates the degree to which NeuS [13] and UniRender are optimized for parameter $s^{-1}$.

5.3.3. Ablation Study

To evaluate the effectiveness of the individual modules that we propose, we conducted further ablation studies on the BlendedMVS dataset. We adapted NeuS [13] as the baseline, and we selectively added these modules to it. As shown in Figure 13, the baseline produced severely distorted geometric surfaces with large holes. Model A produced smooth and continuous surfaces, but only approximated the real geometry. Model B showed an effective enhancement to the geometric quality, but it also produced redundant surfaces and holes. Model C yielded a reconstructed surface with no obvious geometric errors. However, Model D provided a more detailed result. In summary:

• The unified rendering scheme tends to produce smooth and continuous surfaces, and improves geometric accuracy to some extent.
• The geometric constraint mechanisms are effective in improving the quality of geometric reconstruction.
• The spherical sampling strategy can facilitate the capture of detail.

One point worth noting is that although Mode A was optimal, in terms of the numerical results in

Table 5. Quantitative results on ablation models.

Method	Unified Rendering	Geometric Constraints	Sphere Sampling	Acc↓	Comp↓	Overall↓
Baseline				0.19959	0.01143	0.10551
Model A	✓		✓	0.08909	0.00666	0.04787
Model B		✓	✓	0.12987	0.00163	0.06575
Model C	✓	✓		0.11727	0.00171	0.05949
Full model	✓	✓	✓	0.10151	0.00154	0.05153

, its visualization results were not as good. A reasonable explanation for this contradiction is that there were plenty of redundant surfaces located on the back of the visible surfaces in all cases, except for Model A, as shown in Figure 14. The redundant surfaces severely reduced the Acc value for all the models, except Model A, while Model A produced a relatively clean surface, thus resulting in a lower overall value.

6. Conclusions and Outlook

6.1. Conclusions

We present UniRender, a novel 3D surface reconstruction method with neural rendering. In this work, we first analyzed the biases in current volume rendering methods, and we developed an approximately bias-free rendering scheme that unifies volume rendering and surface rendering through a natural transition. Then, we introduced several multi-view geometric constraints into the rendering pipeline, to improve its ability for capturing accurate geometry. Finally, a spherical sampling algorithm was proposed, for allocating more attention to the space with which we were concerned. Extensive experiments on different remote sensing datasets showed that UniRender outperforms other types of 3D surface reconstruction methods by a wide margin.

The main significance of this work is that it identifies the biases that exist in the current volume rendering branch and it proposes a unified rendering strategy to address them. This rendering strategy can be seamlessly integrated into other 3D rendering frameworks, offering universal applicability. Additionally, this work introduces new ideas for jointly optimizing multi-view geometric constraints within neural rendering. Finally, extensive experiments demonstrated the effectiveness of the proposed method and they also promoted the application of neural rendering in remote sensing.

6.2. Outlook

Similar to other neural rendering methods, it takes several hours to train our model for each scene. This is because forward inference and backpropagation for each spatial coordinate point by MLPs consume a significant amount of computational power. Some recent advancements in computer vision have tried to expedite the rendering process, through voxel-based architecture [39,40]. Although these techniques were not originally designed for 3D reconstruction and remote sensing scenarios, they hold significant promise of alleviating the current time-consuming challenges.

Moreover, UniRender currently does not have the ability to perform large-scale surface reconstruction. This is because the fidelity of scene reproduction by MLPs with finite parameters decreases as the scene’s scale increases. This is akin to representing a scene with a fixed-size image, where larger scenes result in lower spatial resolution images. Therefore, constructing neural rendering models for large-scale scenes remains a significant challenge. Fortunately, recent approaches, like Mega-NeRF [41] and Block-NeRF [42], have suggested a potential solution: partitioning a large scene into multiple sub-blocks and employing multiple models to process these sub-blocks in parallel.

In summary, building a faster framework with large-scale processing capability is our future endeavor.