Total Differential Photometric Mesh Refinement with Self-Adapted Mesh Denoising

Abstract

Variational mesh refinement is a crucial step in multiview 3D reconstruction. Existing algorithms either focus on recovering mesh details or focus on suppressing noise. Approaches with consideration of both are lacking. To address this limitation, we proposed a new variational mesh refinement method named total differential mesh refinement (TDR), which mainly included two improvements. First, the traditional partial-differential photo-consistency gradient used in the variational mesh refinement method was replaced by the proposed total-differential photo-consistency gradient. With consideration of the photo-consistency correlation between adjacent pixels, our method can make photo-consistency achieve a more effective convergence than traditional approaches. Second, we introduced the bilateral normal filter with a novel self-adaptive mesh denoising strategy into the variational mesh refinement. This strategy maintains a balance between detail preservation and effective denoising via the zero-normalized cross-correlation (ZNCC) map. Various experiments demonstrated that our method is superior to traditional variational mesh refinement approaches in both accuracy and denoising effect. Moreover, compared with the mesh generated by open-source and commercial software (Context Capture), our meshes are more detailed, regular, and smooth.

1. Introduction

A well-established pipeline of image-based 3D reconstruction technology mainly in-cludes structure from motion (SFM), multiview stereo (MVS), surface reconstruction, mesh refinement, and texture. In this pipeline, the surface reconstruction step reconstructs an initial coarse 3D mesh that may lack details and contain noise due to occlusion and non-Lambertian materials. Photometric stereo [1,2] and mesh refinement are popular methods to reconstruct high-quality mesh shapes. Photometric stereo recovers pixel-wise surface normals from a fixed scene under varying shading cues, which are widely used in the industrial field [3,4,5]. The mesh refinement method evolves the initial mesh to fine details using multiview images. In this paper, we mainly talk about the mesh refinement method for large-scale reconstruction. Variational mesh refinement is the most commonly used refinement method, and it improves mesh details and accuracy by iteratively updating all vertex positions to maximize the photo-consistency between images [6]. However, due to its isotropic regularization term, this method tends to smooth sharpened structures and cannot sufficiently remove excessive mesh noises in texture-less and non-Lambertian regions.

To solve this problem, Li et al. [7] used a content-aware mesh denoising approach as a regularization term for mesh refinement, which was effective in suppressing mesh noise while preserving sharp features. However, without the assistance of image information, some distinct errors and noises of the mesh may be wrongly identified as sharp features and preserved.

Other studies aimed to improve the accuracy of the variational mesh refinement: some researchers selected the best image pairs to refine the mesh [8]. Blaha et al. [9] and Romanoni et al. [10] used semantic information to improve mesh accuracy. However, the improvement is limited when there are few images and little semantic information. In addition, existing variational mesh refinement methods calculate the gradient of each pixel independently and do not consider the photo-consistency of neighboring pixels.

In this study, a total differential mesh refinement (TDR) approach was developed to address the abovementioned problems. First, we proposed to use the total-differential photo-consistency gradient (TDPG) to replace the partial-differential photo-consistency gradient (PDPG) calculation in the variational mesh refinement approach. The TDPG considers the influence of the gradient between adjacent pixels so that the photo-consistency can obtain more effective convergence than the PDPG via gradient descent. Second, we incorporated bilateral normal filtering [11] into the variational mesh refinement to improve the denoising and edge-preserving capabilities. A self-adaptive mesh denoising strategy was adopted to balance detail preservation and effective denoising. Specifically, the zero-normalized cross-correlation (ZNCC), which measures the photo-consistency in the image domain, was transformed to mesh vertices to form a ZNCC map indicative of the uncertainty of the mesh vertices. Then, denoising gradients are self-adaptive and weighted depending on their ZNCC. Thus, our method can remove mesh noise while preserving the details of the mesh. An overview of our method is shown in Figure 1.

Our contributions are as follows:

We proposed the TDPG method, which considers the partial derivative of all pixels in the neighborhood, makes the photo-consistency error converge to a low level, and obtains a fine-details mesh model.

We introduced the bilateral normal filtering [11] to the variational mesh refinement and adopted the self-adaptive mesh denoising strategy that utilized a ZNCC map to guide mesh denoising. This strategy enabled effective denoising while preserving mesh details (Section 2.3).

We used photo-consistency information to guide mesh denoising, which provided a new idea for the study of feature-preserving denoising.

2. Methodology

We enhanced the variational mesh refinement from two aspects. First, the total-differential photo-consistency gradient (TDPG) calculation was proposed for more effective convergence of the photo-consistency. Second, bilateral normal filtering was utilized in mesh refinement for mesh denoising, flattening planes, and sharpening edges. In order to avoid the loss of details that may be caused by denoising, we proposed the self-adaptive mesh denoising strategy. In this section, we first briefly introduce the variational mesh refinement approach (Section 2.1). Then, we present our mesh refinement method (TDR), including TDPG calculation (Section 2.2) and self-adaptive mesh denoising (Section 2.3).

2.1. Preliminaries on Variational Mesh Refinement

The variational mesh refinement approach was introduced by Pons et al. [12] and expanded by Vu et al. [13]. This method minimizes the photo-consistency error between image pairs by iteratively refining the mesh vertex. For one image pair $I_p$ and $I_q$, image $I_q$ can be projected onto the mesh $S$ and then reprojected to image $I_p$ to form a predicted image $I_{pq}^S$ [7,14]. The photo-consistency between the predicted image and the reference image measures the correctness of the mesh, and the goal of variational mesh refinement is to minimize this photo-consistency error between all image pairs.

The energy function is expressed as:

$$E\left(S\right)=E_{photo}\left(S\right)+E_{regularization}\left(S\right)$$

(1)

$S$ is the mesh surface, $E\left(S\right)$ is the total energy function, $E_{regularization}\left(S\right)$ enforces the smoothness of the surface. The $E_{photo}\left(S\right)$ is defined as:

$$E_{photo}\left(S\right)=-{\displaystyle \sum }_{p,q}{{\displaystyle \int }}_{{\Omega }_{pq}^S}^{ }{M}_{zncc}\left(I_p,I_{pq}^S\right)\left(x_i\right)d{x}_i$$

(2)

$M_{zncc}\left(I_p,I_{pq}^S\right)\left(x_i\right)$ is the ZNCC measurement between images $I_p$ and $I_{pq}^S$ at pixel $x_i$. ${\Omega }_{pq}^S$ is the map of the reprojection from image $I_p$ to image $I_q$ via the surface.

To minimize $E_{photo}\left(S\right)$, the gradient is calculated using chain rules (see [13]):

$$g_{photo}\left(V\right)=\frac{d{E}_{photo}\left(S\right)}{dV}={\displaystyle \sum }_{p,q}{{\displaystyle \int }}_{{\Omega }_{pq}^S}^{ }\varphi \left(X_i\right)\partial M\left(x_i\right)D{I}_q(x_j)D{\mathsf{\Pi }}_q\left(X_i\right)\frac{d_i}{N^{\mathrm{T}}{d}_i}Nd{x}_i$$

(3)

$V$ represents all mesh vertices, $g_{photo}\left(V\right)$ is the vertices gradient induced by the photo in each iteration. $M$ is the abbreviation for $M_{zncc}\left(I_p,I_{pq}^S\right)$, $\partial M\left(x_i\right)$ is photo-consistency gradient at pixel $x_i$. $X_i$ is the intersection of S and the ray from the camera center of $I_p$ to $x_i$.$\varphi \left(X_i\right)$ is the barycentric coordinate weight of $X_i$. ${\mathsf{\Pi }}_q$ represents the projection from the world to the $I_q$, $x_j={\mathsf{\Pi }}_q\left(X_i\right)$ and $D{I}_q(x_j)$ is image gradient at $x_j$.$D$ denotes the Jacobian matrix of a function. $d_i$ is the vector joining the camera center of $I_p$ and point $X_i$. $N$ is the outward surface normal at $X_i$.

For mesh regularization, this method adds thin plate energy [15] that penalizes mesh bending to prevent excessive bending of the mesh and excessive deviation of the gradient flow.

$$E_{regularization}\left(S\right)={{\displaystyle \int }}_S^{ }(k_1^2+k_2^2)dS$$

(4)

where $k_1$ and $k_2$ are the principal curvatures of the mesh. The linear combination of Laplacian and Bi-Laplacian operators minimizes this energy [15].

2.2. Total-Differential Photo-Consistency Gradient Calculation

A key step in variational mesh refinement is the calculation of the vertex gradient (Equation (3)), which determines how the vertices are updated to minimize photo-consistency error. In the baseline method [13], the photo-consistency gradient ($\partial M$ in (Equation (3))) is calculated separately at each pixel and the potential contradictions between pixel gradients are ignored. In our approach, we proposed to calculate the photo-consistency gradient for each pixel with consideration of the surrounding related pixels. To distinguish those two methods, we denote the baseline method by $\partial M_{PDPG}$ and our method by $\partial M_{TDPG}$. The following describes the differences between the two methods.

ZNCC is a ubiquitously used photo-consistency measurement in variational mesh refinement, and its formula is as follows:

$$M_{zncc}\left(x_i\right)=\frac{1}{n}\left(\frac{{{\displaystyle \sum }}_{m=1}^n(a_m-{\mu }_A)\left(b_m-{\mu }_B\right)}{{\sigma }_A{\sigma }_B}\right),M_{zncc}\left(x_i\right)\in \left(-1,1\right)$$

(5)

where $x_i$ is the position of a pixel in the reference image, $M_{zncc}\left(x_i\right)$ is the ZNCC measurement at $x_i$, while $A$ and $B$ are two image patches of equal size from the reference and predicted images at $x_i$ (see Figure 2). $a_m$ and $b_m$ ($m\in \left(1,n\right)$) denote the m-th pixel in $A$ and $B$, respectively and $n$ is the number of pixels in each image patch; ${\mu }_A$/${\mu }_B$ and ${\sigma }_A/{\sigma }_B$ are the mean and standard deviations of the pixel values in $A$ and $B$, respectively.

$A$ and $B$ are two image patches from the reference and predicted images at $x_i$. The black lines indicate that the two image patches are combined to calculate the ZNCC value. The arrows represent the differential of ZNCC with respect to the pixels in B. The right part of Figure 2 shows that ZNCC is jointly calculated by $a_m$ and $b_m$. To minimize the ZNCC error, the partial differential only changes the center pixel, while the total differential changes all the pixels in $B$.

The ZNCC error is defined as $1-M_{zncc}$. The variational mesh refinement [13] changes the center pixel value to reduce this error, and the partial derivative is calculated [12,14]:

$$\partial M_{PDPG}\left(x_i\right)=\frac{\partial M_{zncc}\left(x_i\right)}{\partial b_{center}}=\frac{1}{n}\left(\frac{a_{center}-{\mu }_A}{{\sigma }_A{\sigma }_B}-M_{zncc}\left(x_i\right)\frac{b_{center}-{\mu }_B}{{\sigma }_B{}^2}\right)$$

(6)

where $\partial M_{PDPG}\left(x_i\right)$ denotes the PDPG at position $x_i$, $a_{center}$/$b_{center}$ is center pixels of image patches $A$ and $B$.

On the one hand, PDPG only calculates the partial derivative of the center pixel. As in Figure 2, the ZNCC is determined by all patch pixels, and the total differential should be considered. On the other hand, the PDPG is individually calculated in each image patch. As in Figure 3, pixel $x_i$ is contained by nine image patches, but the gradient is only determined by the center patch. The gradient calculated in this way is expected to reduce the ZNCC error of $x_i$. However, it may increase the ZNCC error of the neighboring pixels, which is not conducive to the convergence of the photo-consistency on the entire image.

In our method, the photo-consistency gradient of a pixel is jointly determined by all image patches that contain it. Based on this idea, we propose the TDPG calculation method:

$$\partial M_{TDPG}\left(x_i\right)={\displaystyle \sum }_{m=1}^n{W}_g\left(d\left(x_i^{b_m},x_i\right)\right)\frac{\partial M_{zncc}\left(x_i^{b_m}\right)}{\partial b_{center}}$$

(7)

$$\frac{\partial M_{zncc}\left(x_i^{b_m}\right)}{\partial b_{center}}=\frac{1}{n}\left(\frac{a_m-{\mu }_{A_m}}{{\sigma }_{A_m}{\sigma }_{B_m}}-M_{zncc}\left(x_i^{b_m}\right)\frac{b_m-{\mu }_{B_m}}{{\sigma }_{B_m}{}^2}\right)$$

(8)

where $\partial M_{TDPG}\left(x_i\right)$ is the TDPG at $x_i$ and $x_i^{b_m}$ is the position of $b_m$ in the image patch of $x_i$ (see Figure 3a), $d\left(x_i^{b_m},x_i\right)$ is the Euclidean distance between $x_i^{b_m}$ and $x_i$. $W_g\left(d\right)$ is the Gaussian weight. $\frac{\partial M_{zncc}\left(x_i^{b_m}\right)}{\partial b_{center}}$ is the partial derivative of ZNCC at $x_i^{b_m}$ with respect to $b_{center}$. $A_m$ and $B_m$ are two image patches from the reference and predicted images at $x_i^{b_m}$. ${\mu }_{A_m}$/${\mu }_{B_m}$ and ${\sigma }_{A_m}/{\sigma }_{B_m}$ are the mean and standard deviations of the pixel values in $A_m$ and $B_m$, respectively. The equation shows that the gradient of each pixel is jointly determined by the partial derivative of all pixels within the image patch and that pixels closer to $x_i$ have a more significant impact.

Figure 4 shows the difference between the convergence process of the PDPG and TDPG on the entire image. We found that TDPG converges after ~25 iterations while PDPG takes ~35 iterations (Figure 4b). TDPG achieves a more effective convergence in that it considers the partial derivative of all pixels in the neighborhood and increases the area affected by the gradient, which thereby facilitates photo-consistency convergence on the entire image. Furthermore, TDPG also yielded a much lower ZNCC error than PDPG. The local magnification area in Figure 4c,d shows that the TDPG method makes the predicted image closer to the reference image in the iterative process.

2.3. Self-Adaptive Mesh Denoising

Although the photo-consistency gradient improves accuracy and enriches the mesh details, noise and errors in the initial mesh cannot be effectively removed, especially in the texture-less and non-Lambertian regions. The mesh regularization of variational mesh refinement is a combination of the Laplacian and Bi-Laplacian operation [15], an isotropic and one-step mesh denoising method. The one-step property means that the method cannot effectively remove the mesh noise in limited iterations, and the isotropic property makes it hard to retain high-frequency details [11,16]. Therefore, we propose to use the two-step and anisotropic bilateral normal filtering as a regularization term for mesh refinement. However, directly applying the bilateral normal filtering with its strong mesh deformation capability is inappropriate because the small mesh details that are difficult to distinguish from noise will be erased.

We utilized the image ZNCC metric, which indicates the mesh accuracy, to guide mesh denoising. For image pair $I_p$ and $I_q$, $x_i\left(p,q\right)$ is the position of a pixel in image $I_p$. The ray formed by the camera center of $I_p$ and $x_i\left(p,q\right)$ intersects the mesh at the 3D position $X_i\left(p,q\right)$. The ZNCC value at face $f_k$ can be calculated from all the image pairs visible to it (Figure 5):

$$C\left(f_k\right)=\left\{\begin{array}{c}\frac{{{\displaystyle \sum }}_{p,q}{{\displaystyle \sum }}_{X_i\left(p,q\right)\in f_k}{\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)·M_{zncc}\left(x_i\left(p,q\right)\right)}{{{\displaystyle \sum }}_{p,q}{{\displaystyle \sum }}_{X_i\left(p,q\right)\in f_k}{\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)}, {\displaystyle {\displaystyle \sum }_{p,q}}{\displaystyle {\displaystyle \sum }_{X_i\left(p,q\right)\in f_k}}{\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)\ne 0\\ 0, {\displaystyle {\displaystyle \sum }_{p,q}}{\displaystyle {\displaystyle \sum }_{X_i\left(p,q\right)\in f_k}}{\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)=0\end{array}\right.$$

(9)

where $C\left(f_k\right)$ represents the ZNCC value of face $f_k$. ${\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)$ describes whether the 3D point $X_i\left(p,q\right)$ is simultaneously visible by image $I_p$ and image $I_q$. If it is visible, ${\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)=1$; otherwise, ${\alpha }_{p,q}^{vis}\left(X_i\left(p,q\right)\right)=0$. $X_i\left(p,q\right)\in f_k$ denotes $X_i\left(p,q\right)$ is on face $f_k$. $M_{zncc}\left(x_i\left(p,q\right)\right)$ is the ZNCC value at $x_i\left(p,q\right)$. Then, $C\left(f_k\right)$ is transferred from mesh face to mesh vertex to form a ZNCC map:

$$C\left(V_i\right)=\frac{{{\displaystyle \sum }}_{k\in N\left(V_i\right)}A\left(f_k\right)C\left(f_k\right)}{{{\displaystyle \sum }}_{k\in N\left(V_i\right)}A\left(f_k\right)}$$

(10)

in which, $V_i$ is a vertex of the mesh. $C\left(V_i\right)$ represents the ZNCC value of the mesh vertex $V_i$. $N\left(V_i\right)$ is the one-ring face neighborhood of $V_i$. $A\left(f_k\right)$ is the area of the face $f_k$.

The ZNCC maps of different meshes are shown in Figure 6. We found that the noise area in the mesh has a low value of $C\left(V_i\right)$ due to the error of the mesh shape. In addition, in the area with a high $C\left(V_i\right)$, the mesh shape conforms to multiview photo-consistency. We use a $C\left(V_i\right)$-weighted denoising gradient to achieve self-adaptive mesh denoising:

$$g_{regularization}\left(V_i\right)=\left(1-C\left(V_i\right)\right)·g_{bilateral}\left(V_i\right)$$

(11)

$g_{regularization}\left(V_i\right)$ is the regularization gradient at each vertex, and $g_{bilater}\left(V_i\right)$ is the bilateral denoising gradient at each vertex which is described in [11].

It is worth noting that general mesh denoising methods only remove noise based on geometric information. In contrast, this paper adaptively applies a denoising gradient based on the photo-consistency metric, which is conducive to removing significant errors in the initial mesh (see Figure 1).

Finally, our TDR combines the photometric gradient and regularization gradient by $\beta$:

$$g_{TDR}\left(V\right)=g_{photo}^{TDR}\left(V\right)+\beta g_{regularization}\left(V\right)$$

(12)

$g$ is an abbreviation for gradient. $g_{photo}^{TDR}\left(V\right)$ is the photometric gradient of our TDR method in each iteration. $g_{photo}^{TDR}\left(V\right)$ replaces the $\partial M\left(x_i\right)$ as $\partial M_{TDPG}\left(x_i\right)$ in (Equation (3)).

2.4. Initialization and Implementation Details

We implemented the variational mesh refinement method [13], where ray tracing is used to calculate the projection between images, the image patch size is 5 × 5, $\beta$ is set to 0.2. Those parameters are the same as that used in [13]. The Gaussian weights $W_g\left(d\right)$ are normalized according to the image patch size. The mesh denoising scheme is local for the bilateral normal filtering algorithm, and the normal iterations and the vertex iterations are set to 20 and 10, respectively. To solve the nonconvex problem, we utilize the L-BFGS optimization algorithm [17,18]. The parameter setting of the TDR method is the same as the variational mesh refinement. Our algorithm is implemented with C++. All experiments were conducted on a single PC machine with Intel(R) Core(TM) i7-8700 CPU (12-core), 64 GB RAM, and Nvidia GTX 2070 GPU.

We used a variety of mainstream meshes as the initial inputs, including the OpenMVS [19] mesh, which is reconstructed by the built-in surface reconstruction function, the COLMAP [20] mesh, and the CMP-CMVS mesh [21]. We used the OpenMVS mesh as the initial mesh by default, as the mesh faces are uniformly sized, and the scene reconstruction is complete.

3. Experiments

3.1. Dataset and Evaluation Metrics

Dataset: We use six datasets that cover different scenes, including UAV (unmanned air vehicle) scenes, close-range scenes, and simulation scenes. Information about these datasets is given in

Table 1. Introduction to the datasets used in this study.

Dataset	Name	Image Size	Number of Images	Initial Mesh	Image Acquisition
Tanks And Temples	Family	1920 × 1080	153	OpenMVS	Handheld
	Francis	1920 × 1080	302	OpenMVS	Handheld
	Horse	1920 × 1080	151	OpenMVS	Handheld
	Panther	1920 × 1080	314	OpenMVS	Handheld
ETH3D	delivery area	6048 × 4032	44	OpenMVS	Handheld
	facade	6048 × 4032	76	OpenMVS	Handheld
	relief	6048 × 4032	31	OpenMVS	Handheld
	relief 2	6048 × 4032	31	OpenMVS	Handheld
BlendedMVS	UAV_ Scene1	2048 × 1536	77	OpenMVS	Rendered
	UAV_ Scene2	2048 × 1536	125	OpenMVS	Rendered
	UAV_ Scene3	2048 × 1536	75	OpenMVS	Rendered
EPFL	Herz- Jesu-P8	3072 × 2048	8	OpenMVS /COLMAP/ CMPMVS	Handheld
	Fountain-P11	3072 × 2048	11	OpenMVS/ COLMAP/ CMPMVS	Handheld
CG Simulation Dataset	Joyful	1920 × 1080	70	OpenMVS	Rendered
Personal Collection Dataset	House	4592 × 3056	36	OpenMVS	UAV
	Woodcarving	2016 × 4032	146	OpenMVS	Handheld

.

(1)

Tanks And Temples [22] is a benchmark for image-based 3D reconstruction. The image sequences come from video streams. We picked the Family, Francis, Horse, and Panther data for close-range scene evaluation.

(2)

ETH3D [23] is a benchmark for multiview stereo (MVS) evaluation. It provides ultrahigh-resolution images registered to the 3D laser scan point clouds. We picked its facade, delivery_area, relief, and relief_2 data for close-range scene evaluation.

(3)

BlendedMVS [24] is a large-scale simulation MVS dataset. It provides ground-truth meshes and rendered images. We selected four outdoor scenes captured by UAVs, namely, UAV_Scene1, UAV_Scene2, and UAV_Scene3, for UAV scene evaluation.

(4)

Custom simulation dataset. We picked computer graph (CG) mesh models Joyful [25] as ground-truth mesh and utilized the same lighting to render 70 images from fixed perspectives using Blender [26].

(5)

The EPFL dataset [27] provides two ground-truth meshes captured by LIDAR sensors, namely, Herz-Jesu-P8 and Fountain-P11, and provides the images registered with the meshes.

(6)

Personal Collection Dataset. We collected multiview images from the internet and natural scenes for qualitative evaluation.

Evaluation Metrics: Similar to [22,23], we use the shortest distance from a point to the surface to evaluate the precision of a mesh. I is the input mesh to be evaluated, and R is the reference mesh. For a vertex $i\in I$, its distance to the reference mesh is defined as $d_{\mathrm{i}\to \mathrm{R}}$. These distances can be aggregated to define the accuracy of the input mesh I for any distance threshold $d$:

$$P\left(d\right)=100\ast \frac{{{\displaystyle \sum }}_{i\in I}\left[d_{\mathrm{i}\to \mathrm{R}}<d\right]}{\left|I\right|}$$

(13)

where $\left[·\right]$ is the Iverson bracket. $\left|I\right|$ is the number of vertices for mesh $I$.

Similarly, for a reference mesh vertex $r\in R$, its distance to the input mesh is defined as $d_{\mathrm{r}\to \mathrm{I}}$. The completeness of the input mesh for any distance threshold $d$ is defined as:

$$C\left(d\right)=100\ast \frac{{{\displaystyle \sum }}_{r\in R}\left[d_{\mathrm{r}\to \mathrm{I}}<d\right]}{\left|R\right|}$$

(14)

Accuracy and completeness can be combined to calculate the F-score:

$$F\left(d\right)=\frac{2\ast P\left(d\right)\ast C\left(d\right)}{P\left(d\right)\ast C\left(d\right)}$$

(15)

The F-score is the harmonic mean of the accuracy and completeness at threshold $d$. Threshold $d$ varies according to the different scales of the dataset. Moreover, we also use the mean of $d_{\mathrm{i}\to \mathrm{R}}$ of all vertices as the mean-accuracy metric and the mean of $d_{\mathrm{r}\to \mathrm{I}}$ of all vertices as the mean-completeness metric.

3.2. Comparison with the Baseline Method

3.2.1. Performance on the UAV Dataset

In urban scenes, the 3D reconstruction of structures (such as planes and edges) is the critical point. BlendedMVS provides urban meshes that are difficult to capture by a laser scanner fully. We selected the UAV_Scene1-UAV_Scene3 from BlendedMVS for experiment. Figure 7 shows the visual results. Compared with the baseline method [13], the mesh refined by our TDR approach is sharper in the edge region (see the red boxes in Figure 7a) because we utilize the bilateral normal filtering with an edge-preserving effect. In addition, our method obtains a better mesh detail than the baseline method (see the green box in Figure 7b) because our TDPG converges better on photo-consistency. Furthermore, in a limited number of iterations, the Laplace operator used in the baseline method does not remove the undulation in the initial mesh (see the blue box in Figure 7c). In contrast, our method makes the surface flatter due to the bilateral normal filtering with a better denoising ability.

Table 2. Quantitative evaluation of the BlenderMVS dataset. Acc. means accuracy and Compl. represents completeness.

		Initial Mesh	Baseline	TDR
UAV_ Scene1	Acc. [%]	28.11	77.00	81.67
	Compl. [%]	20.30	61.37	63.16
	F1 [%]	23.57	68.30	71.23
	Mean-Acc. [X10-2]	10.89	6.71	5.09
	Mean-Compl. [X10-2]	24.13	15.03	14.13
UAV_ Scene2	Acc. [%]	31.90	72.79	77.71
	Compl. [%]	29.80	90.20	90.77
	F1 [%]	30.81	80.57	83.73
	Mean-Acc. [X10-2]	12.80	8.00	6.70
	Mean-Compl. [X10-2]	42.40	28.30	27.30
UAV_ Scene3	Acc. [%]	29.53	83.98	85.79
	Compl. [%]	28.90	66.85	67.95
	F1 [%]	29.21	74.44	75.84
	Mean-Acc. [X10-2]	10.27	5.37	4.94
	Mean-Compl. [X10-2]	26.80	17.55	17.44

shows the quantitative results of the BlendedMVS dataset. Since the captured images are far from the objects, we set the cutoff distance $d$ as 0.05 m. Table 2 shows that the baseline method and our TDR method both improve the precision of the initial mesh, but the improvement brought by our method is much more evident than that of the baseline method. The error ($d_{\mathrm{i}\to \mathrm{R}}$) distribution of Figure 7 shows that compared with the baseline method, the accuracy improvement brought by our TDR method is reflected in the flat area. This shows that the proposed TDR method has a stronger ability to regularize the mesh than the baseline method.

In summary, the TDR method can reconstruct sharp features and flat planes even under poor initial conditions, making our method very suitable for reconstructing urban scenes.

3.2.2. Performance on the Close-Range Dataset

For the close-range dataset, we test our method on the recently published close-range MVS datasets, i.e., ETH3D and Tanks and Temples.

ETH3D dataset: This dataset has ultrahigh-resolution images and provides the laser scan point cloud with the registered images. We use the Poisson surface reconstruction [28] to obtain the reference mesh. Figure 8 shows the results of our experiment. Comparing the baseline method, our TDR method which utilized TDPG has better mesh details (see the red ellipses in Figure 8). Then, Figure 8c,d show that the initial mesh has substantial noise in texture-less regions. The baseline method using the isotropic denoising method cannot effectively remove this noise and retain sharp edges, while our TDR methods can effectively achieve these effects (see the blue rectangles in Figure 8). The results of the accuracy evaluation are shown in

Table 3. Quantitative evaluation of results on the ETD3D dataset.

		Initial Mesh	Baseline	TDR
delivery_area	Acc. [%]	53.18	53.33	56.98
	Compl. [%]	37.65	41.85	42.95
	F1 [%]	44.09	46.89	48.98
	Mean-Acc. [X10-3]	7.89	7.80	7.24
	Mean-Compl. [X10-3]	51.24	50.90	50.84
facade	Acc. [%]	24.25	34.35	42.78
	Compl. [%]	27.07	38.22	45.15
	F1 [%]	25.58	36.18	43.93
	Mean-Acc. [X10-3]	34.89	30.67	26.76
	Mean-Compl. [X10-3]	15.35	13.91	13.22
relief	Acc. [%]	95.45	95.87	95.97
	Compl. [%]	94.09	95.63	96.79
	F1 [%]	94.77	95.75	96.38
	Mean-Acc. [X10-3]	1.79	16.25	13.51
	Mean-Compl. [X10-3]	2.15	1.76	13.17
relief_2	Acc. [%]	90.48	90.34	92.58
	Compl. [%]	86.72	89.74	90.72
	F1 [%]	88.56	90.04	91.64
	Mean-Acc. [X10-3]	1.94	2.16	1.93
	Mean-Compl. [X10-3]	2.66	2.19	2.13

. Since the captured images are close to objects, we set $d$ as 0.005 m. Table 3 shows that the proposed TDR algorithm achieves the best results in terms of almost all quantitative metrics.

Tanks and Temples dataset: This dataset does not provide image poses that are registered with the reference point cloud. Therefore, the results of this experiment are qualitatively evaluated. Compared with the baseline algorithm, results of the proposed TDR method have finer details (the red boxes in Figure 9), flatter planes (the green boxes in Figure 9), and sharper edges (the blue boxes in Figure 9). This proves the effectiveness of the two improvements in this paper.

3.3. Discussion

3.3.1. Ablation Experiment

We evaluated the effectiveness of the two improvements in TDR in the simulated CG dataset (see Figure 10), and ablation experiments were conducted. We tested four different configurations: (1) w/o TDPG: using PDPG and self-adaptive bilateral regularization. (2) w/o BI: using TDPG and Laplace regularization. (3) w/o ZNCC weighted: using TDPG and bilateral regularization without ZNCC weighted. (4) Full TDR: using TDPG and self-adaptive bilateral regularization. Figure 11 shows the results. There are giant pits in the face of the initial mesh due to a sizeable texture-less area in the rendered images (Figure 11). The meshes using bilateral regularization (b–d) do not show this error. Comparing (b) with (e) in Figure 11, we found that TDPG presents a more detailed and accurate result than PDPG. Comparing (d) with (e), the ZNCC weighted strategy succeeded in preserving mesh details (see the red box in Figure 11d).

3.3.2. The Influence of Initial Meshes

To discuss the impact of initial meshes on the TDR algorithm, the widely used CMPMVS mesh, OpenMVS mesh, and COLMAP mesh models were chosen as initial meshes for comparisons. We also added the baseline method [13] for comparison. The baseline algorithm generated the CMPMVS_Vu mesh, OpenMVS_Vu mesh, and CCOLMAP_Vu mesh. The TDR method generated the CMPMVS_TDR mesh, OpenMVS_TDR mesh, and COLMAP_TDR mesh.

In this section, we use the EPFL dataset [27] for evaluation. This benchmark is designed for mesh evaluation. Therefore, we test our results using the evaluation metric proposed in this benchmark. First, the reference and the input mesh are projected into the same image, and the residual of the depth of each pixel is calculated. Then, the occupancy rates of the residuals from 3$\mathsf{\sigma }$ ($\mathsf{\sigma }$ = 1.1 mm) to 10-times 3$\mathsf{\sigma }$ are counted. Finally, the occupancy rates of the residuals of all images are averaged to obtain the final occupancy distribution histogram and occupancy density map, as shown in Figure 12. In addition, the weighted average of the residual distribution histogram can be drawn to obtain the accuracy metric. The proportions of the part less than 30$\mathsf{\sigma }$ are counted as the completeness metric [27], shown in

Table 4. The accuracy and completeness of all meshes.

	Herz-Jesu-P8			Fountain-P11
	#faces [M]	Acc. [3σ]	Compl. [%]	#faces [M]	Acc. [3σ]	Compl. [%]
CMPMVS	2.76	6.25	50.57	2.47	5.08	53.42
CMPMVS_Vu	1.25	4.30	66.55	1.55	2.90	70.34
CMPMVS_TDR	1.25	3.77	71.83	1.55	2.64	71.70
OpenMVS	1.54	4.04	72.96	1.89	2.42	79.37
OpenMVS_Vu	1.26	3.59	75.62	1.52	2.15	79.52
OpenMVS_TDR	1.26	3.49	75.72	1.53	1.95	80.29
COLMAP	1.14	3.80	71.45	1.51	2.42	74.33
COLMAP_Vu	1.23	3.60	73.77	1.39	2.24	78.25
COLMAP_TDR	1.22	3.32	76.41	1.40	1.99	79.12

.

Table 4 and Figure 13 show that, regardless of the initial mesh, the precision improvement brought by the TDR algorithm is much more significant than that of the baseline algorithm. For both TDR and baseline methods, the accuracy and completeness of the refined meshes are high if the initial meshes are accurate, and vice versa. The reason is that mesh refinement is a nonconvex problem and has a certain dependence on the initial value when using the gradient descent method to solve it.

Figure 13 shows the visual results for Herz-Jesu-P8. Even when handling a very poor initial mesh, TDR still reconstructs a good result (Figure 13a). All initial meshes have considerable noise at the door region (the blue boxes in Figure 13). The TDR method outperforms the baseline method in mesh denoising ability. What is more commendable is that the proposed algorithm with a strong regularization term well retains the details in the human sculpture (the red boxes in Figure 13). This is attributed to the self-adaptive weighted denoising strategy, which effectively combines the photo-consistency gradient and the mesh denoising gradient, with the denoising gradient mainly applied to the noise areas.

In other words, due to the stronger denoising ability, the TDR method can handle a worse initial mesh model.

3.3.3. Running Times Evaluation

In this section, we discuss the running time of our 3D reconstruction system and TDR algorithm on datasets EPFL and Tanks and Temples.

Our 3D reconstruction system comprises SFM, MVS, mesh reconstruction and mesh refinement steps. The SFM step uses COLMAP (CUDA version) with default parameters. The MVS and mesh reconstruction step uses OpenMVS with default parameters. The proposed TDR algorithm is used in the mesh refinement stage with no special performance optimization.

Table 5. The processing times of the 3D reconstruction system.

	Herz-Jesu-P8		Fountain-P11		Family		Francis		Horse		Panther
	Time (s)	Ratio (%)	Time (s)	Ratio (%)	Time (s)	Ratio (%)	Time (s)	Ratio (%)	Time (s)	Ratio (%)	Time (s)	Ratio (%)
SFM	-	-	-	-	472	12	925	19	276	12	1388	22
MVS	120	35	196	38	1009	25	1313	27	575	24	1545	24
Mesh reconstruction	60	18	86	17	222	6	131	3	91	4	331	5
Mesh refinement	159	47	233	45	2335	58	2455	51	1427	60	3149	49

shows the processing times of the 3D reconstruction system. The SFM, MVS, mesh reconstruction, and mesh refinement time ratios are about 16%, 25%, 4%, and 55%, respectively. Although the mesh refinement step evolves the initial mesh to a high quality with fine details, it cost the most time in the 3D reconstruction system.

Table 6. The time consumption of critical steps in the TDR and the baseline method.

	Herz-Jesu-P8		Fountain-P11		Family		Francis		Horse		Panther
	Vu	TDR	Vu	TDR	Vu	TDR	Vu	TDR	Vu	TDR	Vu	TDR
#Vertices (K)	754	754	944	944	895	895	672	672	527	527	1897	1897
#Images pixels (M)	50	50	69	69	317	317	626	626	313	313	651	651
Ray tracing (s)	30	32	43	45	561	389	624	524	416	291	850	810
Compute $\partial M$ (s)	7	85	10	129	231	1391	285	1271	184	759	198	1626
Compute $g_{photo}$ (s)	17	16	26	23	371	257	277	245	211	151	339	316
Compute $g_{regularization}$ (s)	0	3	0	4	0	7	0	5	0	5	0	10
Others (s)	22	24	30	32	435	290	504	410	331	221	433	387
Total (s)	76	159	109	233	1597	2335	1690	2455	1142	1427	1819	3149

shows the time consumption of critical steps in the TDR and the baseline method. The two improvements of our method correspond to the computation of $\partial M$ and $g_{regularization}$, respectively. On the one hand, the PDPG method can use the integral image [29] to accelerate calculations, while the TDPG method cannot. On the other hand, PDPG calculates the partial differential once per pixel, while TDPG calculates the 25 times partial differential for the 5 × 5 image patch. Therefore, the running time of $\partial M$ is 8~13 times that of PDPG in the experimental data. For the regularization item $g_{regularization}$, the baseline method takes less than 1 s on all experimental data. Our TDR method utilizes the bilateral normal filter method, thus increasing the time consumption, but the time ratio does not exceed 2% of the total time. Overall, the running time of our method is 1.5~2 times more than the baseline method due to a time increase for TDPG.

3.4. Comparison with Open Source and Commercial Software

In this section, we compare our method with representative open source software (OpenMVS and COLMAP), commercial software (Context Capture [30]), and the baseline method [13]. Figure 14 visually compares the reconstructed 3D models on the Personal Collection Dataset. OpenMVS can reconstruct the overall shapes, although some fine details are lost (the roof in the House data), and noise is covered on the smooth surface (all enlarged parts in Figure 14). The COLMAP mesh is similar to the OpenMVS mesh, with slightly more details (the roof in the House data) but fails in texture-less regions (the white wall in the House data). The Context Capture mesh also fails in texture-less regions (the white wall in the House data), and it is too smooth, which causes details loss (roof in the House data). The method of [13] enhances the details of the initial mesh to a certain extent (the roof in the House data), but the mesh noise is still not eliminated, and the edges are not sharpened (the white wall in the House data). In contrast, due to the strong denoising ability brought by the bilateral normal filter, our mesh result on the Woodcarving data is the smoothest, and the house edges are the sharpest. At the same time, the TDPG recovers the most details.

4. Conclusions

This study proposed a new mesh refinement approach coupling total differential photometric mesh refinement and self-adapted mesh denoising. On the one hand, traditional PDPG in variational mesh refinement is replaced by TDPG. TDPG considers the neighboring pixels and increases the area affected by the gradient, which results in more effective convergence of the photo-consistency, thus increasing the details and accuracy of the mesh. On the other hand, the self-adaptive denoising strategy provides a framework for image-guided mesh denoising. The intensity of the denoising gradient can be adaptively adjusted according to the multiview ZNCC metric, which facilitates the removal of significant errors in the initial mesh and preserving mesh details. Experiments on different scenes and comparisons with open-source and commercial software were conducted. The refined meshes are evaluated in terms of both accuracy and completeness. Results showed that our method outperformed current variational mesh refinement methods and is comparable and even better than commercial software, and the mesh refined by our method is the most detailed, accurate, and regular. In the future, we plan to run our method on GPU and explore the fusion of subpixel sampling and photometric stereo technology with mesh refinement.