Parametric Blending of Hole Patches Based on Shape Difference

Abstract

A triangular mesh obtained by scanning 3D models typically contains holes. We present an effective technique for filling a hole in a triangular mesh in geometric modeling. Simple triangulation of a hole is refined and remeshed iteratively to generate an initial patch. The generated patch is then enhanced to become a target patch by minimizing the variation of principal curvatures. In discrete approximation, this produces a third-order Laplacian system of sparse symmetric positive definite matrix, and the symmetry can efficiently be used to find the robust solutions to the given Laplacian system. Laplacian smoothing of the target patch is defined as a source patch. The shape difference between two corresponding vertices of the source and the target patches is measured in terms of Euclidean distance and curvature variation. On the basis of the shape difference and a user-specified control parameter, different blending weights are determined for each vertex, and the final patch is generated by blending two patches. We demonstrate the effectiveness of our technique by discussing several examples. The experimental results show that our technique can effectively restore salient geometric features of the original shape.

1. Introduction

Owing to their simplicity and hardware-accelerated rendering facility, triangular meshes have been widely used for representing the shape of three-dimensional (3D) objects in computer graphics and real-time 3D applications. Even though there exist many advanced modeling techniques, it is still a tedious and time-consuming task to create a 3D model from a scratch using triangle mesh. Recent advances in 3D scanning technology have made it easy to obtain the shape of a 3D object and thus triangle meshes have become essential in reverse engineering, such as ergonomic product design and dental CAD. However, the scanned data set cannot be directly used and requires post-processing steps because it contains several artifacts such as holes, gaps, self-intersections, and handles. Among these artifacts, holes are quite common and are caused mainly by occlusions, accessibility limitations, and surface properties [1]. Figure 1a shows a typical artifact (in purple) encountered in a 3D scanning process and a patch filling the hole is shown in Figure 1b.

Figure 1. Accessibility limitation of an oral 3D scanner results in (a) a hole in the scanned model. (b) The artifact is repaired by a hole-filling technique.

Over several decades, various types of hole-filling techniques have been developed for triangular meshes. Most hole-filling algorithms generate a hole patch that is smoothly connected with the hole boundary. However, they do not provide the user with the additional flexibility of controlling the shape of the generated patch. For example, the hole patch in Figure 1b is generated by solving a complex Laplacian system; thus, it is difficult to control the resulting shape as desired.

In this paper, we present a simple and effective technique for generating a hole patch in a triangular mesh and controlling its shape. The overall process of our method is shown in Figure 2. First, a hole is detected in a triangular mesh by searching closed boundary edges (Figure 2a), and simple triangulation fills the hole without additional vertices (Figure 2b). Subsequently, refinement and remeshing are applied iteratively to produce an initial hole patch (Figure 2c), and two different patches are constructed by Laplacian smoothing and minimizing curvature variation (Figure 2d). We select the patch generated by Laplacian smoothing for source patch $P_S$ and the other for target patch $P_T$ and measure the shape difference between them. The final hole patch is generated by blending the source and the target patches on the basis of the shape difference and a user-specified blending parameter. The result of the hole patch generated by using our method is shown in Figure 2e, where shape features, such as concavity and convexity, are easily exaggerated by a user’s control parameter.

Figure 2. Overview of the proposed algorithm. (a) a hole is detected in a triangular mesh by searching closed boundary edges, (b) simple triangulation fills the hole without additional vertices, (c) refinement and remeshing are applied iteratively to produce an initial hole patch, (d) two different patches are constructed by Laplacian smoothing and minimizing curvature variation, (e) the result of the hole patch generated by using our method.

The main contributions of our method can be summarized as follows:

• Robustness: Our method is based on the remeshing followed by fairing technique, which guarantees the robust solutions to symmetric Laplacian system without self-intersections. Therefore, our method can robustly be used for filling holes with arbitrary sizes and shapes.
• Effectiveness: By analyzing the shape difference between the source and the target patches, salient features, such as convexity and concavity, can be exaggerated or reduced in the resulting patch.
• Controllability: Our system provides the user with a shape control parameter. The user can interactively modify the hole patch until the desired shape is obtained.

The remainder of this paper is organized as follows. Section 2 reviews some related recent work on hole filling in triangular meshes. Section 3 details the construction of a source patch and a target patch by minimizing fairness energy. The parametric blending of two patches is then presented in Section 4, and experimental results are presented in Section 5. Finally, we conclude the paper and mention the scope for future work in Section 6.

2. Related Work

Barequet et al. [2] proposed an algorithm for filling holes and gaps in polygon mesh. This algorithm is based on dynamic programming and later became a basic building block for other mesh repair techniques [3,4,5,6]. However, this method produces rough hole patches because the smoothness around the hole boundary is not considered. Pfeifle et al. [7] constructed smooth patches that filled the holes in a piecewise polynomial surface using Delaunay triangulation and refinement. Liepa et al. [4] proposed an umbrella operator for mesh fairing and constructed a hole patch with a density of triangles similar to that of the neighborhood of the hole. However, this method, employed in later works [5,6], requires a long processing time for filling large holes because of its time complexity, $O\left(n^3\right)$.

Chen et al. [8,9] presented a sharpness-dependent hole-filling method. They constructed an initial patch using radial basis interpolation. If sharp features exist around the hole, they enhance these features by applying Bayesian classification [10] and a sharpness-dependent filter [11]. However, it is difficult to select the correct filter parameters to prevent undesired sharp features [12]. Zhao et al. [13] constructed an initial patch filling of highly curved holes using the advancing front mesh (AFM) technique [14] and improved the initial patch by solving the Poisson equation with Dirichlet boundary conditions [15].

Pernot et al. [16] employed Delaunay triangulation to generate an initial hole patch and optimized its shape while satisfying the tangency and curvature blending condition with the surrounding region. However, their method is not suitable for filling highly curved holes. Branch et al. [1] and Xiao et al. [17] sampled neighboring vertices of the hole boundary and created a hole patch using an (RBF) interpolator. These approaches produce plausible patches; however, additional controls, such as bulge and exaggeration are not possible.

Freeform surfaces have been employed for hole filling in polygon meshes. Charles et al. [18] used $C^1$ cubic triangular spline patches for filling holes in a polygon mesh. Li et al. [19] presented a hole-filling method for preserving sharp features. Their method includes locating incomplete feature curves around the hole boundary and generating a polynomial blending curve that smoothly connects feature curves. Then, two split sub-holes are filled by using a Bezier–Lagrange hybrid patch while guaranteeing smoothness around the hole boundary. Wang et al. [12] also focused on preserving features. They proposed an automatic feature-preserving hole-filling framework for reconstructing missing corners and sharp edges. Fortes et al. [20] proposed a surface reconstruction and hole-filling scheme based on a network of parallel and/or orthogonal curves. However, their method is more suitable for CAD models than for scanned triangular meshes.

3. Construction of Hole Patches

This section presents the construction of two hole patches that are used for the source patch and target patch.

3.1. Triangulation of Hole Boundary

Typically, two triangles share an edge, and a boundary edge is shared by a single triangle. A hole is a closed loop consisting of boundary edges. In this study, we employ a half-edge data structure [21] for representing the triangle mesh, and thus holes and their neighboring vertices are easily detected in runtime.

Let ${\mathbf{v}}_i,i=1,2,\cdots ,n$ be the vertices consisting of a hole boundary. By recursively connecting two vertices ${\mathbf{v}}_i$ and ${\mathbf{v}}_{i+2}$ with the smallest angle $\angle {\mathbf{v}}_i{\mathbf{v}}_{i+1}{\mathbf{v}}_{i+2}$, we can triangulate a hole boundary. The triangulation of the hole boundary is shown in Figure 3a. Our method is similar to the AFM method [14] but does not require preprocessing, such as hole contour cleaning [16]. Our triangulation may contain self-intersections for the complex and highly curved hole boundary; however, this problem can be automatically resolved by subsequent incremental remeshing and fairing stages.

Figure 3. Process of incremental remeshing: (a) initial triangulation, (b) first iteration, (c) third iteration, (d) fifth iteration.

3.2. Incremental Remeshing

As shown in Figure 3a, the initial triangulation is coarse and anisotropic. Moreover, the density of the triangles does not fit to the neighborhood of the hole boundary. To resolve this problem and improve its quality, we apply an incremental remeshing technique [3,22,23] to the initial triangulation.

Algorithm 1 shows each step of the incremental remeshing technique. It takes the target edge length l as input and sets $low$ and $high$ values for the lower and upper bounds, respectively. Each edge longer than $high$ is split into two edges by adding a vertex (in Step 2). Edges shorter than $low$ are collapsed by merging two incident vertices unless the collapse does not generate new edges longer than $high$ (in Step 3). In general, a regular triangle mesh has valences of 6 and 4 for interior vertices and boundary vertices, respectively. We apply an edge-flipping operation to improve mesh regularity (in Step 4) and adjust the vertex positions to obtain isotropic triangles (in Step 5). We repeat these steps until all edge lengths are less than the $high$. Self-intersections in initial triangulation can be removed by repeating these steps.

The intermediate results of iteratively applying incremental remeshing steps are shown in Figure 3b–d. As shown in Figure 3d, incremental remeshing generates a plausible hole patch; however, this patch does not reflect geometric information around the hole at all.

Algorithm 1 Incremental remeshing

Require:M: initial mesh, l: target edge length    Ensure:$M^{\prime }$: refined mesh with isotropic triangles.    $low$ = $l\times \frac{4}{5}$, $high$ = $l\times \frac{4}{3}$    SplitEdges($high$)    CollapseEdges($low$, $high$)    FlipEdges()    Relaxation()

3.3. Source and Target Patches

Let $P_0$ be the hole patch generated by incremental remeshing technique. $P_0$ must be further processed to create source and target patches. First, we explain how to construct a target patch $P_T$, which is obtained by minimizing curvature variations on $P_0$. Let $\mathbf{x}:D(\subset R^2)\to R^3$ be a continuous parametric surface. The minimum variation surface can be obtained by minimizing the variation of curvature represented by [24]:

$$E_{\mathrm{CV}}\left(\mathbf{x}\right)=\underset{D}{\int \int }{\left(\frac{\partial {\kappa }_1}{\partial {\mathbf{t}}_1}\right)}^2+{\left(\frac{\partial {\kappa }_2}{\partial {\mathbf{t}}_2}\right)}^{2}dudv,$$

(1)

where ${\kappa }_1$ and ${\kappa }_2$ denote principal curvatures, and ${\mathbf{t}}_1$ and ${\mathbf{t}}_2$ the corresponding principal directions. The corresponding Euler–Lagrange PDE is ${\mathsf{\Delta }}^3\mathbf{x}(u,v)=0$ in D, where $\mathsf{\Delta }$ is the Laplacian operator. By linearlization followed by discretization, based on finite difference or finite element [24,25,26], Equation (1) yields the following Laplace system for a triangle mesh:

$$L^3\left[\begin{array}{c}{\mathbf{x}}_F\\ {\mathbf{x}}_C\end{array}\right]=\left[\begin{array}{c}\mathbf{0}\\ {\mathbf{x}}_C\end{array}\right],$$

(2)

where L denotes the discrete Laplacian matrix, ${\mathbf{x}}_F$ is the vector of the free (unknown) vertex coordinates, and ${\mathbf{x}}_C$ is the vector of the constrained vertex coordinates representing the boundary condition. For a minimum-variation surface, the boundary condition can be realized by fixing the positions of the two rings of the boundary vertices, as shown in Figure 4c. We employ the cotangent discretization [27] for Laplacian matrix L. Because L is sparse and symmetric positive definite, the solution to the Laplace system in Equation (2) can be efficiently found by Cholesky factorization [3].

Figure 4. Hole patches generated by (a) minimizing membrane energy ($L\mathbf{x}=\mathbf{0}$), (b) minimizing thin-plate energy ($L^2\mathbf{x}=\mathbf{0}$), and (c) minimizing curvature variation energy ($L^3\mathbf{x}=\mathbf{0}$).

The target patch $P_T$ constructed by this is shown in Figure 5b. Instead of minimizing the variation of curvature, we can minimize membrane energy or thin-plate energy, each of which produces a patch with a minimum area or minimum sum of curvature. In this study, we take a minimum variation patch as a target patch because it considers the shape of a wider area around a hole than other methods.

Figure 5. Two hole patches to be blended: (a) Source patch $P_S$, and (b) target patch $P_T$.

Next, we create a source patch $P_S$. For this, we apply Laplacian smoothing to $P_T$. Let ${\mathbf{v}}_i$ be the position of the i-th vertex of $P_T$. Then, the new position of ${\widehat{\mathbf{v}}}_i$ can be computed by

$${\widehat{\mathbf{v}}}_i=\frac{1}{\sum w_{ij}}\sum _{{\mathbf{v}}_j\in N_1\left({\mathbf{v}}_i\right)}{w}_{ij}{\mathbf{v}}_j,$$

where $N_1\left({\mathbf{v}}_i\right)$ denotes one ring vertex of ${\mathbf{v}}_i$, and $w_{ij}$ is the edge weight connecting two vertices ${\mathbf{v}}_i$ and ${\mathbf{v}}_j$. Cotangent weights are commonly used and can be computed by [28]

$$w_{ij}=\frac{cot{\alpha }_{ij}+cot{\beta }_{ij}}{2},$$

where ${\alpha }_{ij}$ and ${\beta }_{ij}$ are the angles shown in Figure 6. As shown in Figure 5a, the source patch $P_S$ is generated by applying Laplacian smoothing.

Figure 6. Cotangent weight $w_{ij}$ for Laplacian smoothing.

4. Parametric Blending of Hole Patches

The shape of the target patch might be unsatisfactory even though it contains more geometric details than the source patch. In this case, the user wants to change the shape of the hole patch. However, it is difficult to control the shape of a hole patch because it is determined by the solution to Equation (2).

In this section, we present an effective method for controlling the shape of a hole patch. On the basis of the shape difference between the source and target patches, we assign different weights to each vertex. These weights are used to generate the desired hole patch, where salient features are exaggerated by blending the source and target patches.

4.1. Shape Difference

There exists a one-to-one correspondence of vertices between the source patch $P_S$ and the target patch $P_T$. Let ${\mathbf{v}}_i$ and ${\mathbf{v}}_i^{\prime }$ be the corresponding vertices between the two patches. First, we define the shape difference between ${\mathbf{v}}_i$ and ${\mathbf{v}}_i^{\prime }$ by using discrete mean curvature and Euclidean distance. Let $\mathsf{\Delta }{H}_i=|H_i-H_i^{\prime }|$ and $d_i=\parallel {\mathbf{v}}_i-{\mathbf{v}}_i^{\prime }\parallel$ denote the mean curvature difference and the Euclidean distance between ${\mathbf{v}}_i$ and ${\mathbf{v}}_i^{\prime }$, respectively. Then, the shape difference ${\sigma }_i$ can be computed by

$${\sigma }_i=\sigma ({\mathbf{v}}_i,{\mathbf{v}}_i^{\prime })=(1-\alpha )\frac{\mathsf{\Delta }{H}_i-\mathsf{\Delta }{H}^{min}}{\mathsf{\Delta }{H}^{max}-\mathsf{\Delta }{H}^{min}}+\alpha \frac{d_i-d^{min}}{d^{max}-d^{min}},$$

where $\mathsf{\Delta }{H}^{min}$, and $\mathsf{\Delta }{H}^{max}$ are the minimum and the maximum of curvature difference, respectively, and $d^{min}$ and $d^{max}$ are the Euclidean distances. The shape difference can be controlled by user-specified parameter $\alpha \in [0,1]$. For example, the shape difference is completely determined by the curvature difference if $\alpha =0$ and by the Euclidean distance if $\alpha =1$. In Figure 7, the color-coded shape difference is shown between $P_S$ and $P_T$.

Figure 7. Color-coded shape difference between $P_S$ and $P_T$ for different values of $\alpha$.

4.2. Blending Hole Patches

Let w be a user-specified blending weight $(0\le w\le 1)$. Based on the one-to-one correspondence between $P_S$ and $P_T$, simple linear blending can generate a new patch $P\left(w\right)=(1-w)P_S+w{P}_T$. However, the uniform blending weight w is not sufficient to produce a hole patch with exaggerated geometric features. To overcome this limitation, we replace the uniform blending weight w with the per-vertex blending weight ${\widehat{w}}_i$ by employing the shape difference between two vertices. The per-vertex blending weight ${\widehat{w}}_i$ can then be computed by

$${\widehat{w}}_i=w+\left(\frac{w}{1+e^{-10{\sigma }_i+5}}\right),$$

(3)

where w is the user-specified uniform weight, and ${\sigma }_i$ is the shape difference between two vertices, ${\mathbf{v}}_i$ and ${\mathbf{v}}_i^{\prime }$. In general, large shape differences imply that the corresponding vertices can be a salient feature and need to be exaggerated by a large blending weight. From this observation, Equation (3) produces the blending weights ${\widehat{w}}_i\approx w$ if ${\sigma }_i=0$ and ${\widehat{w}}_i\approx 2w$ if ${\sigma }_i=1$. In Figure 8, the per-vertex blending weights $\widehat{w}$ are shown depending on different shape differences $\sigma$. Finally, each pair $({\mathbf{v}}_i,{\mathbf{v}}_i^{\prime })$ of the corresponding vertices is blended by per-vertex weight ${\widehat{w}}_i$ and produces the resulting hole patch.

Figure 8. Per-vertex blending weight ${\widehat{w}}_i$ is determined by a user-specified blending parameter w and a shape difference ${\sigma }_i$.

5. Experimental Results

The proposed technique was implemented on a PC with an Intel i7-9700F CPU, 16GB main memory, and an NVIDIA RTX 2060 graphics card, using the C++ language. Various 3D models with different hole shapes and sizes were used to demonstrate the effectiveness of the proposed technique.

The hole-filling results for four different models are shown in Figure 9. The first column shows the original models with various holes, and the second column shows the target hole patches ($P_T$) obtained by minimizing curvature variation ($L^3\mathbf{x}=\mathbf{0}$). Note that the target hole patches are also plausible to some extent. However, they lack geometric details in the region of the hole center. As shown in the third column, in Figure 9, our method provides the user with additional control to enhance the quality of the generated hole patch. From the shape analysis between the source patch and the target patch, the scaled blending weight for each vertex is computed, and these weights are used to exaggerate salient geometric features effectively.

Figure 9. Four examples with complex holes. (a) Left: a bunny model; middle: minimizing curvature variation($L^3\mathbf{x}=\mathbf{0}$), right: shape-difference-based blending(our method), $\alpha =0.5,w=1.2$. (b) Left: a bunny model; middle: $L^3\mathbf{x}=\mathbf{0}$; right: our method, $\alpha =0.9,w=3.3$. (c) Left: a molar model; middle: $L^3\mathbf{x}=\mathbf{0}$, right: our method, $\alpha =0.5,w=1.55$. (d): Left: an armadillo model; middle: $L^3\mathbf{x}=\mathbf{0}$; right: our method, $\alpha =0.2,w=1.4$.

As shown in Figure 9a, we considered both the Euclidean distance and curvature difference by setting $\alpha =0.5$, and thus convexity and concavity are exaggerated in the final blending result. As shown in Figure 9b, we mainly considered the Euclidean distance in the shape difference. In this case, the Euclidean distances of the vertices in the boundary region are small, whereas those in the center region are relatively large. Thus, we can obtain the bulging effect in the center region while restoring the shape of the bunny ear. The sunken hole in Figure 9c is quite challenging because there might be large shape variations in the missing region. Compared to the target patch, our method successfully generates a hole patch with sufficient geometric details. In Figure 9d, we exaggerated the bumps in the target patch by considering a greater curvature difference rather than the Euclidean distance.

Our method supports real-time control for hole-patch generation and blending. As the user changes the blending weight w, only the per-vertex blending weight $w_i$ is recomputed and the hole patch is updated by simple linear blending. Because the source and target patches do not need to be updated, they can be constructed in the pre-processing step. Table 1 lists the computation time required to generate the target patch by solving Equation (2) and the source patch by applying Laplacian smoothing to the target patch. For all examples, per-vertex blending was performed within a millisecond. As shown in Figure 10, the computation time linearly increases depending on the number of vertices constituting the hole patch.

Table 1. Number of vertices on hole boundary and int the hole patch, and the computation time (in ms) for generating the patches for each example.

Examples	#Verts. on Hole Boundary	#Verts. of Patch	Computation Time (in ms)
			Preprocessing		Blending
			${\mathit{P}}_{\mathit{T}}$	${\mathit{P}}_{\mathit{S}}$
Figure 9a	206	1532	234	58	<1
Figure 9b	46	106	40	6	<1
Figure 9c	218	334	52	17	<1
Figure 9d	311	102	121	51	<1

Figure 10. Computational time for creating a hole patch for different hole sizes.

6. Conclusions and Future Work

We present an effective technique for filling a hole in a triangular mesh based on shape differences. The target patch is generated by minimizing the curvature variation of the initial hole patch, and the source patch is generated by applying Laplacian smoothing to the target patch. By employing Euclidean distance and curvature difference, we defined the shape difference between two vertices. Based on the shape difference, the per-vertex blending weight is computed, and these weights are used to blend the source patch and target patch. The experimental results show that per-vertex blending weight can exaggerate salient features, such as convexity and concavity, and we demonstrated the effectiveness of our method by comparing our hole-filling results with those of existing methods.

For a large hole, the shape difference in the hole center tends to be quite small; thus, it is difficult to reconstruct the geometric details of a hole center (see Figure 11). To resolve this limitation, we plan to determine how to transfer shape details outside a hole to the generated hole patch.

Figure 11. Limitation of our method caused by small shape differences in the center of a large hole.