Multi-Scale Upsampling GAN Based Hole-Filling Framework for High-Quality 3D Cultural Heritage Artifacts

: With the rapid development of 3D scanners, the cultural heritage artifacts can be stored as a point cloud and displayed through the Internet. However, due to natural and human factors, many cultural relics had some surface damage when excavated. As a result, the holes caused by these damages still exist in the generated point cloud model. This work proposes a multi-scale upsampling GAN (MU-GAN) based framework for completing these holes. Firstly, a 3D mesh model based on the original point cloud is reconstructed, and the method of detecting holes is presented. Secondly, the point cloud patch contains hole regions and is extracted from the point cloud. Then the patch is input into the MU-GAN to generate a high-quality dense point cloud. Finally, the empty areas on the original point cloud are ﬁlled with the generated dense point cloud patches. A series of real-world experiments are conducted on real scan data to demonstrate that the proposed framework can ﬁll the holes of 3D heritage models with grained details. We hope that our work can provide a useful tool for cultural heritage protection.


Introduction
With the emergence and development of 3D scanners and computer graphics technology, the way that cultural heritage artifacts (CHarts) are preserved has changed considerably. For example, CHarts can be restored in digital space and presented through the Internet. The virtual presentation of CHarts can be divided into three steps: obtaining the 3D fragments by 3D scanners, reassembling the 3D fragments, and finally transferring the virtual CHarts to the Internet. However, due to the physical and chemical erosion of the contact surfaces, the CHarts were often broken into fragments when they were discovered and the edges of the discovered fragments were often incomplete, resulting in the existence of holes on the reassembled 3D CHarts. Thus, how to fill these holes effectively and reconstruct high-quality 3D CHarts is an important issue in the field of archeology.
The traditional hole-filling methods are based on the structural information of 3D mesh models [1][2][3][4][5]. These methods can be generally categorized as template-based methods and mesh-based methods. The former methods are suitable when the holes are relatively large, and a similar 3D model from the 3D model database is selected to be the underlying surface and then used to fill the holes [6,7]. An effective template database is very important; the more 3D models of different shapes, the better the hole-filling effect. Sahay et al. [8].
proposed an online 2D depth database; they projected the 3D hole region into a 2D depth map, then searched the best-matched depth map from the depth-based database to complete the 2D hole region, and finally restored the 3D mesh model. Some studies [9][10][11] focus on In this paper, a novel framework, named MU-GAN, is proposed to fill these hole regions of 3D CHarts. Firstly, a detecting hole method is proposed. Secondly, a point cloud patch with hole regions is extracted as the input of the trained MU-GAN to generate a dense and complete point cloud. Finally, the generated point cloud is used to fill the hole regions of the original point cloud. Experimental results demonstrate that our framework can achieve a complete point cloud with high quality. The main contributions of this work can be summarized as follows:

1.
A multi-scale upsampling GAN (MU-GAN) is built to generate a dense point cloud to fill the hole regions.

2.
To make the learned latent feature more robust, multi-scale point clouds are used to complete the incomplete models, which can restore the CHarts with more fine-grained details.

3.
There are no limitations on the size of the input point cloud, indicating that the proposed framework is more suitable for the CHarts with a huge point number to preserve details.

Hole Detection Method
The accurate detection of holes is a prerequisite for completing the 3D CHarts. Due to the physical and chemical erosion of the contact surfaces, the fractured surfaces of the fragments are often not completely consistent as they were just broken, resulting in holes on the reassembled 3D CHarts. The holes can be classified as simple holes and complex holes, as shown in Figure 1. Based on the triangular principle, we proposed an effective 3D mesh hole detection method. The details of the method are presented in Algorithm 1. Appl. Sci. 2022, 12, x FOR PEER REVIEW 3 of 14 hole regions is extracted as the input of the trained MU-GAN to generate a dense and complete point cloud. Finally, the generated point cloud is used to fill the hole regions of the original point cloud. Experimental results demonstrate that our framework can achieve a complete point cloud with high quality. The main contributions of this work can be summarized as follows: 1. A multi-scale upsampling GAN (MU-GAN) is built to generate a dense point cloud to fill the hole regions. 2. To make the learned latent feature more robust, multi-scale point clouds are used to complete the incomplete models, which can restore the CHarts with more fine-grained details. 3. There are no limitations on the size of the input point cloud, indicating that the proposed framework is more suitable for the CHarts with a huge point number to preserve details.

Hole Detection Method
The accurate detection of holes is a prerequisite for completing the 3D CHarts. Due to the physical and chemical erosion of the contact surfaces, the fractured surfaces of the fragments are often not completely consistent as they were just broken, resulting in holes on the reassembled 3D CHarts. The holes can be classified as simple holes and complex holes, as shown in Figure 1. Based on the triangular principle, we proposed an effective 3D mesh hole detection method. The details of the method are presented in Algorithm 1.

Algorithm 1: Hole detection method for 3D Charts
Initialize. The 3D mesh model of the 3D CHart is composed of a set of triangles, and the triangle contains vertices and edges.
Step 1: Traversing the 3d Mesh to get the number of adjacent triangles on each side to form a matrix ( , , ), where represents the side, represents the number of triangles adjacent to the side, and is a Boolean value, which identifies whether the current side has been traversed.
Step 4: Adding boundary vertices to the hole points set =≥ { 1 , 2 , 3 }, and the Boolean value is set to TRUE.
Step 5: Repeating Step 4, until a closed loop which is the identified hole is formed.
Step 6: Repeating Step 2-Step 4 until the matrix does not have hole edges.

Algorithm 1: Hole detection method for 3D Charts
Initialize. The 3D mesh model of the 3D CHart is composed of a set of triangles, and the triangle contains vertices and edges.
Step 1: Traversing the 3d Mesh to get the number of adjacent triangles on each side to form a matrix M(E, N, B), where E represents the side, N represents the number of triangles adjacent to the side, and B is a Boolean value, which identifies whether the current side has been traversed.
Step 2: Find the edge (v 1 , v 2 ) whose N value is 1 in M(E, N, B), which is defined as the hole points set ϕ = {v 1 , v 2 }.
Step 4: Adding boundary vertices to the hole points set ϕ =≥ {v 1 , v 2 , v 3 }, and the Boolean value is set to TRUE.
Step 5: Repeating Step 4, until a closed loop which is the identified hole is formed.
Step 6: Repeating Step 2-Step 4 until the matrix M does not have hole edges. Output: The hole regions of the 3D mesh model. Figure 2 illustrates the detection result based on the proposed method described in Algorithm 1, and the hole region can be detected effectively. Appl. Sci. 2022, 12, x FOR PEER REVIEW 4 of 14 Output: The hole regions of the 3D mesh model. Figure 2 illustrates the detection result based on the proposed method described in Algorithm 1, and the hole region can be detected effectively.

Figure 2.
Hole detection results; the hole is marked by the red line.

MU-GAN Architecture
The hole area can be regarded as a region with sparse distribution of points. The holefilling task can be accomplished by increasing the density of the point cloud in the hole area, which can also be regarded as upsampling. In this section, an MU-GAN is designed based on the PU-Net [38]. As depicted in Figure 3, the architecture contains a generator and a discriminator. The purpose of the generator is to generate a point cloud that is denser than the input point cloud, and the purpose of the discriminator is to determine whether the input point cloud is real data or generated by the generator.
The generator is shown in the upper part of Figure 3. It has three components: the feature extraction module, the feature expansion module, and the point set reconstruction module. For the feature extraction module, the intra-level dense connections method in [39] is applied to extract the features from the input point cloud with data size ( × ), where = 3. The method defines the local neighborhood in feature space, groups the features via KNN based on the feature similarity, and finally refines the features of each group through multi-layer perceptrons (MLPs). After the feature extraction, the size of the input , the original complete point cloud, is converted to ( × ), where denotes the number of extraction features.
The feature expansion module serves the purpose to increase the number of points (from to ). It should be noted that two key steps in the feature expansion module are the up and down modules, which can enable the generator to produce more diverse point distributions and are depicted as "up" and "down" operators in Figure 3. The detailed structure of these two modules is illustrated in Figure 4. More details about these modules can be referenced in [40].
In the point cloud reconstruction module, the 3D coordinate reconstruction of per-point features (generated from the feature expansion module) is carried out by a series of fully connected layers, and the dimension of the features is changed from × to × 3. Finally, the dense point cloud with the size of × 3 is generated. In the training stage, the original complete point cloud is which has 2048 points. The input data consist of three parts, which are 1 (1024 points), 2 (512 points), and 3 (256 points) are the downsampled point cloud from , 1 , and 2 , respectively. In other words, the inputs are different scales of the original point cloud. After using the three-point cloud as input of the generator, three dense generated point clouds can be obtained.
The discriminator is shown at the bottom of Figure 3. The input is the generated dense point cloud. Additionally, the target point cloud with points is downsampled at different

MU-GAN Architecture
The hole area can be regarded as a region with sparse distribution of points. The holefilling task can be accomplished by increasing the density of the point cloud in the hole area, which can also be regarded as upsampling. In this section, an MU-GAN is designed based on the PU-Net [38]. As depicted in Figure 3, the architecture contains a generator and a discriminator. The purpose of the generator is to generate a point cloud that is denser than the input point cloud, and the purpose of the discriminator is to determine whether the input point cloud is real data or generated by the generator.
Appl. Sci. 2022, 12, x FOR PEER REVIEW scales, and then think of , 1 and 2 as ground truth . The output of the discrim is a value between 0 and 1. If the confidence value is closer to 1, it indicates that the inp    The generator is shown in the upper part of Figure 3. It has three components: the feature extraction module, the feature expansion module, and the point set reconstruction module. For the feature extraction module, the intra-level dense connections method in [39] is applied to extract the features from the input point cloud with data size (N × d), where d = 3. The method defines the local neighborhood in feature space, groups the features via KNN based on the feature similarity, and finally refines the features of each group through multi-layer perceptrons (MLPs). After the feature extraction, the size of the input P, the original complete point cloud, is converted to (N × C), where C denotes the number of extraction features.
The feature expansion module serves the purpose to increase the number of points (from N to rN). It should be noted that two key steps in the feature expansion module are the up and down modules, which can enable the generator to produce more diverse point distributions and are depicted as "up" and "down" operators in Figure 3. The detailed structure of these two modules is illustrated in Figure 4. More details about these modules can be referenced in [40].  more realistic with the target distribution. Based on the network proposed in [35], the global features are obtained through a set of fully connected and max-pooling layers. To improve feature integration, a self-attention module is utilized after feature concatenation. Finally, the confidence value is generated by applying a set of MLPs, the max-pooling layer, and the fully connected layer to determine whether the input data are close to the true distribution.  In the point cloud reconstruction module, the 3D coordinate reconstruction of perpoint features (generated from the feature expansion module) is carried out by a series of fully connected layers, and the dimension of the features is changed from rN × C to rN × 3. Finally, the dense point cloud with the size of rN × 3 is generated.

Discriminator
In the training stage, the original complete point cloud is P which has 2048 points. The input data consist of three parts, which are P 1 (1024 points), P 2 (512 points), and P 3 (256 points) are the downsampled point cloud from P, P 1 , and P 2 , respectively. In other words, the inputs are different scales of the original point cloud. After using the three-point cloud as input of the generator, three dense generated point clouds can be obtained.
The discriminator is shown at the bottom of Figure 3. The input is the generated dense point cloud. Additionally, the target point cloud with rN points is downsampled at different scales, and then think of P, P 1 and P 2 as ground truth P gt . The output of the discriminator is a value between 0 and 1. If the confidence value is closer to 1, it indicates that the input is more realistic with the target distribution. Based on the network proposed in [35], the global features are obtained through a set of fully connected and max-pooling layers. To improve feature integration, a self-attention module is utilized after feature concatenation. Finally, the confidence value is generated by applying a set of MLPs, the max-pooling layer, and the fully connected layer to determine whether the input data are close to the true distribution.
To ensure the stability of the MU-GAN, and reduce the difference between the generated point cloud and ground truth, the least-squares loss is used: where D(P) is denoted as the output of discriminator, and D P gt is the confidence value of ground truth. The generator tries to generate "real" samples to confuse the discriminator, and the discriminator distinguishes whether the input data come from the real data or the generated data. The adversarial loss can ensure the realism of the generated point cloud. Since the generator will predict three-point cloud in various scales, our reconstruction loss consists of three terms (d 1 , d 2 , d 3 ). To reduce the distance between the input point cloud and the output point cloud, the minimum matching distance [35] is defined as reconstruction loss: Therefore, the overall loss function for MU-GAN is defined as where α, β and γ are the weights of generator loss, reconstruction loss, and discriminator loss.

MU-GAN-Based Hole FILLING framework
The goal of this work is to take a 3D CHart model with holes as input and produce a complete high-quality model as output. MU-GAN consists of roughly three stages: (1) Hole detection of the mesh model using Algorithm 1, and then transforming the mesh model into a point cloud model; (2) Acquiring k (k = 3) point cloud data (P 1 , P 2 , P 3 ) of different points of an artifact model P and upsampling them respectively. Then extracting the missing regions in the corresponding original point cloud as patches; (3) Merging the to-be-repaired input data and patches to obtain the final point cloud restoration results. The specific repair framework for this study is shown in Algorithm 2.

Algorithm 2: MU-GAN Based Hole Filling Framework
Initialize: Point cloud data of 3D CHart model.
Step 1: Construct the mesh model and locate the hole regions based on Algorithm 1.
Step 2: Convert the mesh model into point cloud data and extract the point cloud P containing the hole area.
Step 3: Generating the downsampled point cloud P 1 , P 2 and P 3 . P 1 , P 2 and P 3 are the downsampled point cloud from P, P 1 and P 2 , separately.
Step 4: Input P 1 , P 2 and P 3 into the trained MU-GAN to obtain dense point cloud P d .
Step 5: Extracting the point cloud corresponding to the hole area in P d as a patch, is denoted as P hole .
Step 6: Merging P hole with incomplete original point cloud to generate the complete point cloud. Output: 3D CHart model with no hole regions.

Dataset and Implementation Detail
Three-dimensional data of the Terracotta Warriors are acquired by using Creaform VIU 718 hand-held 3D scanners. Furthermore, the Terracotta Warriors are unearthed from the K9901 pit of Emperor Qinshihuang's Mausoleum Site Museum. The scan resolution is 0.05 mm, which is conducive to scan speed, but the scanning accuracy is rough. The software Artect is used to automatically or manually align the scanned terracotta warriors to synthesize the entire model. Because the original point cloud has a large number of points, it occupies a very large memory and affects the efficiency of the algorithm. Therefore, the downsampling procedure is adopted to obtain a relatively sparse point cloud (as shown in Figure 5). size the entire model. Because the original point cloud has a large number of points, it occupies a very large memory and affects the efficiency of the algorithm. Therefore, the downsampling procedure is adopted to obtain a relatively sparse point cloud (as shown in Figure 5). }. For example, the input point cloud data with 256 points require upsampling to the 512 target points cloud data. The network is implemented using PyTorch and Python 3.7, and the Adam algorithm [41] is adopted as the optimization function with two parameters of 0.5 and 0.9. Where 0.5 is the exponential decay rate of the first-order moment estimation, 0.9 is the exponential decay rate of the second-order moment estimation. In the training phase, the generator and discriminator are trained alternately, and the initial learning rates are both set to 0.0001. We gradually decrease the learning rate by a decay rate of 0.5 per 50 k iterations until 10 −6 . We employ batch normalization (BN) and RELU activation units at the generator. We trained the network for 120 epochs with a batch size of 28. We set = 0.1, = 100 and = 0.1 for all our experiments, with the hardware of AMD Ryzen7 2700 (2.39 GHz), 64 GB memory, and NVIDIA RTX TITAN.

Evaluation Metrics
In the test phase, the local CHarts with holes are selected as the ground truth on each of the testing models, and the size of the input point cloud is unfixed. We randomly select  There are 8000 point cloud patches (7356 for training and 644 for testing) extracted from 40 whole CHarts to train the network. The original point cloud P is generated by sampling 2048 points from each patch uniformly. The default point number N of each patch is 2048, and the upsampling ratio is 2. Therefore, the input is randomly selected r = 1 2 N, 1 4 N, 1 8 N points from P (see Figure 6), and the target point cloud is N, 1 2 N, 1 4 N . For example, the input point cloud data with 256 points require upsampling to the 512 target points cloud data. The network is implemented using PyTorch and Python 3.7, and the Adam algorithm [41] is adopted as the optimization function with two parameters of 0.5 and 0.9. Where 0.5 is the exponential decay rate of the first-order moment estimation, 0.9 is the exponential decay rate of the second-order moment estimation. In the training phase, the generator and discriminator are trained alternately, and the initial learning rates are both set to 0.0001. We gradually decrease the learning rate by a decay rate of 0.5 per 50 k iterations until 10 −6 . We employ batch normalization (BN) and RELU activation units at the generator. We trained the network for 120 epochs with a batch size of 28. We set α = 0.1, β = 100 and γ = 0.1 for all our experiments, with the hardware of AMD Ryzen7 2700 (2.39 GHz), 64 GB memory, and NVIDIA RTX TITAN.
Appl. Sci. 2022, 12, x FOR PEER REVIEW 7 of 14 size the entire model. Because the original point cloud has a large number of points, it occupies a very large memory and affects the efficiency of the algorithm. Therefore, the downsampling procedure is adopted to obtain a relatively sparse point cloud (as shown in Figure 5). }. For example, the input point cloud data with 256 points require upsampling to the 512 target points cloud data. The network is implemented using PyTorch and Python 3.7, and the Adam algorithm [41] is adopted as the optimization function with two parameters of 0.5 and 0.9. Where 0.5 is the exponential decay rate of the first-order moment estimation, 0.9 is the exponential decay rate of the second-order moment estimation. In the training phase, the generator and discriminator are trained alternately, and the initial learning rates are both set to 0.0001. We gradually decrease the learning rate by a decay rate of 0.5 per 50 k iterations until 10 −6 . We employ batch normalization (BN) and RELU activation units at the generator. We trained the network for 120 epochs with a batch size of 28. We set = 0.1, = 100 and = 0.1 for all our experiments, with the hardware of AMD Ryzen7 2700 (2.39 GHz), 64 GB memory, and NVIDIA RTX TITAN.

Evaluation Metrics
In the test phase, the local CHarts with holes are selected as the ground truth on each of the testing models, and the size of the input point cloud is unfixed. We randomly select

Evaluation Metrics
In the test phase, the local CHarts with holes are selected as the ground truth on each of the testing models, and the size of the input point cloud is unfixed. We randomly select 1 2 N, 1 4 N, 1 8 N points from a hole region patches as testing input. Due to the disorder of discrete point cloud, and to evaluate the completion results against the ground truth, we adopt two standard metrics introduced in [35]: Chamfer distance (CD) and Earth movers distance (EMD). The CD denotes the distance between the generated point cloud P d and the ground truth P gt . The formula is symmetric, and the first part guarantees the minimum distance between the generated point cloud and the real point cloud, and the latter part ensures that the real point cloud is covered by the output point cloud.
EMD is defined as follows: where ϕ is the mapping generated point cloud to ground truth point cloud. EMD minimizes the average distance between the corresponding point. EMD will be higher than CD, because EMD is the point-to-point correspondence, whereas CD can be one to many.

Experimental Results on MU-GAN
To verify the robustness of the proposed network, a small hole in each model from different parts of the complete CHarts model is generated for testing. Figure 7 shows the complete results of the input data. Our network has more reasonable repair results on both the head with more feature points and the body with fewer feature points. Specifically, we extract the hole region patches from the three different scales generated by MU-GAN; the final complete point cloud is generated by merging the hole region patches with the incomplete point cloud. By this method, not only the holes are filled, but also the high-fidelity features of the model are preserved. The final result is not affected by the downsampling at different scales because the combination of different scales of features can effectively retain more features of the original model. } points from a hole region patches as testing input. Due to the disorder of discrete point cloud, and to evaluate the completion results against the ground truth, we adopt two standard metrics introduced in [35]: Chamfer distance (CD) and Earth movers distance (EMD). The CD denotes the distance between the generated point cloud and the ground truth . The formula is symmetric, and the first part guarantees the minimum distance between the generated point cloud and the real point cloud, and the latter part ensures that the real point cloud is covered by the output point cloud.
EMD is defined as follows: where is the mapping generated point cloud to ground truth point cloud. EMD minimizes the average distance between the corresponding point. EMD will be higher than CD, because EMD is the point-to-point correspondence, whereas CD can be one to many.

Experimental Results on MU-GAN
To verify the robustness of the proposed network, a small hole in each model from different parts of the complete CHarts model is generated for testing. Figure 7 shows the complete results of the input data. Our network has more reasonable repair results on both the head with more feature points and the body with fewer feature points. Specifically, we extract the hole region patches from the three different scales generated by MU-GAN; the final complete point cloud is generated by merging the hole region patches with the incomplete point cloud. By this method, not only the holes are filled, but also the high-fidelity features of the model are preserved. The final result is not affected by the downsampling at different scales because the combination of different scales of features can effectively retain more features of the original model. To prove that the MU-GAN can retain the original structure and feature well for different hole sizes, a series of experiments are conducted. As shown in Figure 8, for Output-2, the output is realistic enough for small holes, while the output is not completed for large holes (e.g., rows 2 and 3). To overcome the problem that more information is not available for large holes, the features obtained by the three different scales should be merged to repair the incomplete model. The results show that the method can retain the structure information of the input point cloud well under different missing ratios. The MU-GAN can further improve the high-quality output for filing the missing region from the compact latent feature to low-level features.
To prove that the MU-GAN can retain the original structure and feature well for different hole sizes, a series of experiments are conducted. As shown in Figure 8, for Output-2, the output is realistic enough for small holes, while the output is not completed for large holes (e.g., rows 2 and 3). To overcome the problem that more information is not available for large holes, the features obtained by the three different scales should be merged to repair the incomplete model. The results show that the method can retain the structure information of the input point cloud well under different missing ratios. The MU-GAN can further improve the high-quality output for filing the missing region from the compact latent feature to low-level features. To evaluate the performance of the proposed method and verify the necessity of multiscale, the values of CD and EMD are compared between the single-scale upsampling GAN (SU-GAN) and MU-GAN, respectively. What needs to be explained is that the single scale means the 1 , which is the downsampled two ratios from the original point cloud . As illustrated in Figure 9, the values of CD and EMD are both the lowest when the missing ratio of the input point cloud is 3%. The reason is that the holes are so small that the surface of the point cloud is semantically reasonable. Additionally, we notice that the input point cloud is the best in terms of CD and EMD for 3% missing data. However, the completed point cloud is sufficiently realistic from our visual inspection in Figure 8 (row 1), while they might not be exactly aligned in every detail. As the proportion of missing data increases, our method outperforms the input point cloud and SU-GAN. For the missing ratio up to 15%, the performance of MU-GAN is improved by 34.15% and 22.44% for the CD and EMD, which is compared with that of SU-GAN. It shows that MU-GAN has a good restoration result and strong robustness for the CHarts model with different hole sizes. To evaluate the performance of the proposed method and verify the necessity of multi-scale, the values of CD and EMD are compared between the single-scale upsampling GAN (SU-GAN) and MU-GAN, respectively. What needs to be explained is that the single scale means the P 1 , which is the downsampled two ratios from the original point cloud P. As illustrated in Figure 9, the values of CD and EMD are both the lowest when the missing ratio of the input point cloud is 3%. The reason is that the holes are so small that the surface of the point cloud is semantically reasonable. Additionally, we notice that the input point cloud is the best in terms of CD and EMD for 3% missing data. However, the completed point cloud is sufficiently realistic from our visual inspection in Figure 8 (row 1), while they might not be exactly aligned in every detail. As the proportion of missing data increases, our method outperforms the input point cloud and SU-GAN. For the missing ratio up to 15%, the performance of MU-GAN is improved by 34.15% and 22.44% for the CD and EMD, which is compared with that of SU-GAN. It shows that MU-GAN has a good restoration result and strong robustness for the CHarts model with different hole sizes. In the following, we compare our method to several existing methods [18,37,42] and present both quantitative and qualitative comparisons on the CHarts test dataset. Table 1 represents the results of the quantitative comparison. The CD and EMD values of different parts vary within a small range, sand it verifies the robustness of our model. Compared with other methods, the MU-GAN achieves the lowest EMD and CD. The CD and EMD values of RL-GAN and PU-Net are higher than the traditional method because the input point clouds of RL-GAN have a fixed size of 2048, and their repair results are too sparse and cannot generate dense point clouds. PU-Net is an upsampling network rather than a GAN network, so it cannot patch the holes completely. The effect of the traditional Poisson-based reconstruction is very close to the proposed method, whose generated point clouds are too uniform in every detail, resulting in a higher probability of alignment with the real data. However, the patches of the hole region generated by the traditional method do not conform to the feature distribution around the hole. The CD and EMD distance of the SU-GAN model in Table 1 are slightly higher than the MU-GAN model. It indicates that the generated results of the proposed method are more similar to the real point cloud. Multi-scale can obtain more dense and uniform repair results because the generator can obtain different levels of the geometric and semantic features. Table 1. Quantitative comparison of CD (10 −6 ) and EMD (10 −1 ) among our methods and several existing methods.  Figure 10 shows the qualitative results. We visualize the SU-GAN (e) and MU-GAN (f), where the SU-GAN has the same generator and discriminator structure as the MU-GAN. It demonstrates that simply applying a SU-GAN framework is insufficient for the CHarts completion problem when the hole is large (e.g., Line 1 and Line 2). As shown in Figure 10, our completion results are more consistent with the distribution of the incompleted point cloud and more realistic and denser. All the methods can achieve good repaired results for the small hole (e.g., Line 3). More importantly, we can repair the CHarts with complex structures. Compared with other methods, our method can achieve more completed results for large holes. In the following, we compare our method to several existing methods [18,37,42] and present both quantitative and qualitative comparisons on the CHarts test dataset. Table 1 represents the results of the quantitative comparison. The CD and EMD values of different parts vary within a small range, sand it verifies the robustness of our model. Compared with other methods, the MU-GAN achieves the lowest EMD and CD. The CD and EMD values of RL-GAN and PU-Net are higher than the traditional method because the input point clouds of RL-GAN have a fixed size of 2048, and their repair results are too sparse and cannot generate dense point clouds. PU-Net is an upsampling network rather than a GAN network, so it cannot patch the holes completely. The effect of the traditional Poisson-based reconstruction is very close to the proposed method, whose generated point clouds are too uniform in every detail, resulting in a higher probability of alignment with the real data. However, the patches of the hole region generated by the traditional method do not conform to the feature distribution around the hole. The CD and EMD distance of the SU-GAN model in Table 1 are slightly higher than the MU-GAN model. It indicates that the generated results of the proposed method are more similar to the real point cloud. Multi-scale can obtain more dense and uniform repair results because the generator can obtain different levels of the geometric and semantic features. Table 1. Quantitative comparison of CD (10 −6 ) and EMD (10 −1 ) among our methods and several existing methods.  Figure 10 shows the qualitative results. We visualize the SU-GAN (e) and MU-GAN (f), where the SU-GAN has the same generator and discriminator structure as the MU-GAN. It demonstrates that simply applying a SU-GAN framework is insufficient for the CHarts completion problem when the hole is large (e.g., Line 1 and Line 2). As shown in Figure 10, our completion results are more consistent with the distribution of the incompleted point cloud and more realistic and denser. All the methods can achieve good repaired results for the small hole (e.g., Line 3). More importantly, we can repair the CHarts with complex structures. Compared with other methods, our method can achieve more completed results for large holes. The RL-GAN method can effectively complete 3D models with simple features and fewer points (e.g., airplanes), but due to the limitation of the network input, the downsampling of the original model causes the lack of features, resulting in a mismatch between the generated model and the original model, and the complete hole part is sparse. PU-Net is also an upsampling method like our method. It can increase the number of points in the original point cloud and can fill in smaller holes effectively. However, for larger holes, the increased number of points cannot be extended to the hole area. The method based on the Poisson equation is mainly for the reconstruction of the 3D mesh. The complete result of the model with more complex surface features is too smooth, and the generated points are too uniform to be combined with the original model features. In summary, for the CHarts model with a complex surface structure and more points, our method performs better-completed results, higher spatial integration, and more grained detail. Appl. Sci. 2022, 12,  The RL-GAN method can effectively complete 3D models with simple features and fewer points (e.g., airplanes), but due to the limitation of the network input, the downsampling of the original model causes the lack of features, resulting in a mismatch between the generated model and the original model, and the complete hole part is sparse. PU-Net is also an upsampling method like our method. It can increase the number of points in the original point cloud and can fill in smaller holes effectively. However, for larger holes, the increased number of points cannot be extended to the hole area. The method based on the Poisson equation is mainly for the reconstruction of the 3D mesh. The complete result of the model with more complex surface features is too smooth, and the generated points are too uniform to be combined with the original model features. In summary, for the CHarts model with a complex surface structure and more points, our method performs better-completed results, higher spatial integration, and more grained detail.

Conclusion and Discussion
China is a country with an ancient culture, and how to restore and preserve these massive cultural relics is of great significance. Due to natural corrosion and human factors, many of the CHarts that have been initially restored have holes of varying sizes on their surface. In this work, we present a multi-scale upsampling GAN to repair these holes. The key of MU-GAN is comprised of a generator and a discriminator. The generator generates a dense point cloud based on the input point cloud with different scales, and the discriminator iteratively reduces the gap between the generated data and the real data. The MU-

Conclusions and Discussion
China is a country with an ancient culture, and how to restore and preserve these massive cultural relics is of great significance. Due to natural corrosion and human factors, many of the CHarts that have been initially restored have holes of varying sizes on their surface. In this work, we present a multi-scale upsampling GAN to repair these holes.
The key of MU-GAN is comprised of a generator and a discriminator. The generator generates a dense point cloud based on the input point cloud with different scales, and the discriminator iteratively reduces the gap between the generated data and the real data.
The MU-GAN can obtain both high-level features and low-level features from different scales to achieve high-quality completion results.
Compared with existing methods, the proposed method achieves state-of-the-art completion performance on real scan data, which maintain the original shape of the fine structure information while completing the shape. One interesting thing is that the effect of the traditional surface reconstruction method is very close to the proposed method because the generated point cloud is too uniform in every detail, resulting in a higher probability alignment with the ground truth point cloud. However, the hole region patches generated by the traditional method do not conform to the feature distribution around the hole. The CD and EMD distance of RL-GAN and PU-Net are lower than the traditional method; the reason is the size of the input point cloud of RL-GAN is fixed to 2048, which is too sparse and cannot complete the dense point cloud of CHarts, while the PU-Net is an upsampling network but not GAN network, so it cannot generate the point with good detail information. In addition, there are still some shortcomings in our proposed framework. Figure 11 shows some failure examples. The failure reason is that the hole is too large, and the head is too dense and uniform. The completed point cloud model in the second column is sparse. Additionally, the complete result in the fourth column can only complete the upper part, which misses less of the point cloud; the lower part cannot be completely repaired due to the large hole. Therefore, there is no good performance for the holes with the large missing area. As for future work, we will concentrate on the designing of a method that combines template-based and learning-based methods to encourage the network to fill the large holes. Another direction for future research is the combination of point-based and image-based approaches to apply the complete setup to both geometry and texture details. GAN can obtain both high-level features and low-level features from different scales to achieve high-quality completion results. Compared with existing methods, the proposed method achieves state-of-the-art completion performance on real scan data, which maintain the original shape of the fine structure information while completing the shape. One interesting thing is that the effect of the traditional surface reconstruction method is very close to the proposed method because the generated point cloud is too uniform in every detail, resulting in a higher probability alignment with the ground truth point cloud. However, the hole region patches generated by the traditional method do not conform to the feature distribution around the hole. The CD and EMD distance of RL-GAN and PU-Net are lower than the traditional method; the reason is the size of the input point cloud of RL-GAN is fixed to 2048, which is too sparse and cannot complete the dense point cloud of CHarts, while the PU-Net is an upsampling network but not GAN network, so it cannot generate the point with good detail information. In addition, there are still some shortcomings in our proposed framework. Figure 11 shows some failure examples. The failure reason is that the hole is too large, and the head is too dense and uniform. The completed point cloud model in the second column is sparse. Additionally, the complete result in the fourth column can only complete the upper part, which misses less of the point cloud; the lower part cannot be completely repaired due to the large hole. Therefore, there is no good performance for the holes with the large missing area. As for future work, we will concentrate on the designing of a method that combines template-based and learning-based methods to encourage the network to fill the large holes. Another direction for future research is the combination of point-based and image-based approaches to apply the complete setup to both geometry and texture details. In conclusion, the purpose of the current study is to complete the CHarts effectively, and the experimental demonstrates the practicability and high-fidelity on the CHarts of the proposed framework. We hope this work can provide a useful completion tool for the virtual restoration of three-dimensional cultural heritage artifacts.  In conclusion, the purpose of the current study is to complete the CHarts effectively, and the experimental demonstrates the practicability and high-fidelity on the CHarts of the proposed framework. We hope this work can provide a useful completion tool for the virtual restoration of three-dimensional cultural heritage artifacts.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to that the Terracotta Warriors involve a policy of secrecy over cultural heritage.