Voids Filling of DEM with Multiattention Generative Adversarial Network Model

Abstract

The digital elevation model (DEM) acquired through photogrammetry or LiDAR usually exposes voids due to phenomena such as instrumentation artifact, ground occlusion, etc. For this reason, this paper proposes a multiattention generative adversarial network model to fill the voids. In this model, a multiscale feature fusion generation network is proposed to initially fill the voids, and then a multiattention filling network is proposed to recover the detailed features of the terrain surrounding the void area, and the channel-spatial cropping attention mechanism module is proposed as an enhancement of the network. Spectral normalization is added to each convolution layer in the discriminator network. Finally, the training of the model by a combined loss function, including reconstruction loss and adversarial loss, is optimized. Three groups of experiments with four different types of terrains, hillsides, valleys, ridges and hills, are conducted for validation of the proposed model. The experimental results show that (1) the structural similarity surrounding terrestrial voids in the three types of terrains (i.e., hillside, valley, and ridge) can reach 80–90%, which implies that the DEM accuracy can be improved by at least 10% relative to the traditional interpolation methods (i.e., Kriging, IDW, and Spline), and can reach 57.4%, while other deep learning models (i.e., CE, GL and CR) only reach 43.2%, 17.1% and 11.4% in the hilly areas, respectively. Therefore, it can be concluded that the structural similarity surrounding the terrestrial voids filled using the model proposed in this paper can reach 60–90% upon the types of terrain, such as hillside, valley, ridge, and hill.

1. Introduction

The digital elevation model (DEM) is one of the sources of national geospatial infrastructure [1,2] and has been widely used in many fields, such as topographic mapping [3,4], ocean monitoring [5,6], glacial evolution detection [7], smart cities [8], and global land change [9]. The quality of a DEM is a basis for the applications and is checked during every step of producing the DEM, from the collection of elevations to the interpolation implemented for resampling [10]. For example, it is unavoidable for a DEM to result in voids due to ground occlusion, among others, resulting in a significantly low-quality spatial analysis relevant to terrain [11,12,13]. Therefore, filling a DEM void is critical for both DEM quality and its applications [14,15].

Many researchers have studied the different algorithms for DEM void filling. In the early work, the traditional interpolation methods, such as inverse distance weighted interpolation (IDW) [16] and Kriging [17], have been widely used. These methods are effective in flat and low-lying terrestrial areas, but are easily impacted by topographical undulations [18]. For this reason, some scholars introduced the concept of image processing to fill the DEM voids. The method, called filling and feathering (FF), is popular and also effective; it uses the feathering process to smooth the sudden changes of the edge. Especially when size of the DEM void is small, this method can obtain a very satisfactory result. Grohman et al. proposed a delta surface fill (DSF) method [18], which is an improvement of FF methods, using auxiliary data to fill voids and get smoother data without feathering. Luedeling et al. [19] improved the DSF method and proposed a more promising method based on a triangular irregular network (TIN), which can better deal with the elevation deviation between different data. In addition, Spline and the advanced Spline method (ANUDEM) [20,21,22] have also been applied to the more specific task of filling DEM voids. On the other hand, a few scholars suggest adding some topographic features as auxiliary data for DEM void filling on the basis of the interpolation method, such as [17,23,24]. Ling et al. [25] are one of the first people who proposed to fill voids on the basis of the trend of valleys in satellite images. Hogan et al. [26] proposed using geometric shadow constraints to fill voids. However, the shadow itself is relatively sparse information, resulting in relatively low accuracy. Yue et al. [27] constructed a high-accuracy surface model (HASM) using the basic geometric feature constraints of landmarks and auxiliary data to fill voids. Soon after, Yue et al. [28] applied HASM to fill voids on the XCO2 surface of GOSAT and SCIAMACHY and achieved good results. Yue et al. [29] used ASTER GDEM v2 and ICTESat to fill the SRTM-1 data voids encountered in the original data to generate a high accuracy and high quality of DEM. Although the methods using interpolation techniques can fill DEM voids well, it heavily relies on the DEM’s local features, such as void size, terrestrial slopes, etc. [30].

In recent years, with the rapid development of deep learning (DL), the DL method has been widely applied in many fields, for instance, image repair [31,32,33]. Therefore, many scholars consider the DEM void filling as image repair, and further use the DL to resolve the limitations of interpolation methods. For example, Goodfellow et al. [34] succeeded in generating images with reasonably clear semantic structure by adversarial training of generators and discriminators. Iizuka et al. [35] proposed a network model with dual discriminators based on generative adversarial networks, and the dual discriminator structure can ensure the global-local consistency of image filling while enhancing the stability of training. Liu et al. [36] introduced partial convolution into the model, and the key idea of partial convolution is to divide the feature image into valid and invalid parts, thus improving the accuracy of filling voids. Dong et al. [30] developed a conditional adversarial network (CAN) framework that employs an L1 norm to constrain the CAN training process for filling shuttle radar topography mission (SRTM) void data. Qiu et al. [37] proposed a terrain texture generation network model, which generated a satisfactory fill surface by constructing samples composed of elevation, terrain slope, and relief degree and designing a loss function composed of pixel-wise, contextual and perceptual loss. However, the TTGM also requires a post-processing step that uses Poisson blur to fuse the resulting filled surface with the original data. Dong et al. [38] used terrain shading as a constraint for conditional generation adversarial networks for filling DEM voids. However, this method has high time and labor costs due to use the filling of data based on field surveys and in situ measurements to train the model. Hui et al. [39] proposed a dense multiscale fusion block for expanding the convolutional kernel receptive field and ensuring the denseness of the convolutional kernels, which have a good result.

On the other hand, Jaderberg et al. [40] first proposed a parameter spatial attention model (Spatial Transformer Network, STN) for target classification tasks. Afterward, many scholars have added the attention mechanism module to the network model. Hu et al. [41] proposed enhancing the ability of the model for the extraction of key features to obtain a high accuracy of filling results. Yu et al. [42] proposed a contextual attention mechanism that processes the feature graph in the deep space of the network. It can extract important features in known regions to fill voids. Wang et al. [43] proposed a multiscale attention mechanism, which is a model that can flexibly process background useful information and suppress useless information. Gavriil et al. [44] proposed a Wasserstein generative adversarial network (WGAN) network structure based on the contextual attention mechanism to fill the DEM holes. Zhang et al. [45] proposed a deep generation model (DGM) in which the reconstruction loss and generative adversarial network (GAN) loss are used to assist network training, and combination with the contextual attention layer to fill the void areas. Zhu et al. [46] designed conditional encoder–decoder generative adversarial neural networks (CEDGANs) to fill the voids, which combines the encoder–decoder structure with adversarial learning to get deep representations of sampled spatial data and the interaction of local structural patterns. Li et al. [47] proposed a topographic knowledge-constrained conditional generative adversarial network (TKCGAN) to fill the voids, which uses qualitative topographic knowledge of valleys and ridges to transform new loss functions. The filling result by WGAN has high accuracy and conforms to topographical fluctuations, but requires additional boundary smoothing processing. Although the methods based on attention mechanisms can extract key information in feature maps, most of them have not considered the internal relationship between features, especially global and local features.

With the reviews above, it can be summarized that the problems of filling DEM voids consist of: (1) the existing models cannot achieve high accuracy for the filled DEM voids in complex terrain, i.e., inconsistent similarity structure surrounding the DEM void area; (2) ignoring how the other features surrounding the DEM void in spatial dimension impact the accuracy of the filled DEM void. For this reason, a multiattention generative adversarial networks model for filling DEM voids is proposed in this paper.

2. A Multiattention Generative Adversarial Network Filling Model

The model architecture proposed in this paper is shown in Figure 1. DEM data I_in, which have voids, are first input into a multiscale feature fusion generation network, and the extracted deep features are then reconstructed to approximate the distribution pattern of the original data I_in. The part of the missing area in I_C is added to the missing data I_in, and an initial filling result I_CR is output. Then, the model takes the I_CR input to the multiattention filling network. After a series of convolution and deconvolution operations, it obtains a more refined result with detailed texture. Finally, the reconstruction loss and the adversarial loss are calculated separately. The global-local adversarial network is optimally trained to obtain a high-precision deep network filling model. The detail for the proposed model is given as follows.

2.1. Multiscale Feature Fusion Generation Network

2.1.1. Network Structure Design

The multiscale feature fusion generation network utilizes the basic VGG16 network structure, as shown in G1 of Figure 1. The G1 module takes the original image I_in and the binary mask M as input. The size of the input original image is H × W × C, where H represents the image height, W is the image width, and C is the number of channels of the image. The binary mask matrix has a value of 0 or 1 (where 0 is for the known area and 1 for the void area) and a size of H × W × 1. The missing image is simulated by the original image multiplying the binary mask matrix. The missing image first passes through six convolution layers with a filter size of 3 × 3 pixels², and each convolutional layer is followed by a batch normalization (BN) layer. In particular, the ReLU function is used for all layers except for the last layer of the output network, where the tanh function is used for the activation function. In this stage, the convolutional layer instead of the maximum pooling layer is used for downsampling and only downsamples the image twice, with a final feature map size of (H/4 × W/4 × 256). Then, the multiscale feature fusion module (MSFF) is introduced, which is described in detail in the following subsections.

The MSFF module can expand the receptive field of the convolution kernel to further extract information from the feature map obtained in the previous step and obtain the feature map with the same size of (H/4 × W/4 × 256). Finally, the feature maps are passed through six deconvolution layers with a filter size of 3 × 3 pixels². This reconstructs the feature distribution of the original image by upsampling twice and outputs a fake image I_C with the same size as the original image I_in. Using the following equation for calculation, the initial repaired image I_CR with a size of H × W × C is obtained, i.e.,

$$I_{CR}=I_{in}\ast (1-M)+I_C\ast M$$

(1)

2.1.2. Multiscale Feature Fusion Module

In order to achieve the fill of large void areas, the output pixels within the voids must contain spatial support for the pixels in the known area, as shown in Figure 2. The receptive field is the size of the area where each pixel of the feature map is affected by the convolution of the previous feature map. The larger receptive field means that more comprehensive information of the feature map can be obtained [34]. To this end, Yu et al. [42] used a 7 × 7 pixels²-sized convolutional kernel for feature extraction to achieve an expanded field of perception, but this adds additional parameters. Dilated convolution was first proposed as a convolution idea for image semantic segmentation tasks where downsampling would degrade image resolution and lose information. Many researchers have started to solve this problem using dilated convolution and achieved promising performance [34,48]. However, since the dilated convolution is sparse, many pixels are ignored in performing feature extraction [49]. For this reason, this paper applies the multiscale feature fusion module to the network model [50], as shown in Figure 3. The MSFF module first decreases the number of channels of the feature map x₀ by the first convolution layer, thus reducing the number of model parameters. Then, each of the four branches of the dilated convolution (rate = 2, 4, 6 and 8) is processed. The feature maps x₁, x₂, x₃ and x₄ in the combined computation, except for x₁, are obtained, which is allowed to be output directly to the last layer. The rest of the feature map x_i needs to be downsampled by a convolution kernel of size 3 × 3 pixels². Then, after a series of overlay combination operations, the sparse feature map x_i is used to combine the dense multiscale feature maps. Finally, the dense multiscale feature maps are convolved once and then combined with x₀. The formula for the combination calculation is by

$$y_i=\{\begin{array}{cc}{x}_i,& i=1;\\ K_i(x_{i-1}+x_i),& i=2;\\ K_i(y_{i-1}+x_i)& i=3,4;\end{array}$$

(2)

where x_i is the feature map output from the convolution layer. K_i is the kernel of one of the convolution layers. y_i is the feature map after the overlay combination.

2.2. Multiattention Filling Network

After obtaining the initially filled results using the G1 network, it is necessary to implement the fill of data detail features. The multiattention filling network is proposed to achieve this purpose. Unlike the G1 network, the G2 module is divided into the regular branch and the attention branch. More detailed description is provided below.

2.2.1. Network Structure Designs

The multiattention filling network also adopts the basic network structure of VGG16, but the channel-spatial pruning attention (CSPA) module is applied in the encoding stage of the G2 model, as shown in G2 of Figure 1.

In the multiattention filling network, the initially filled data I_CR are used as the input into the G2 network. The regular branch adopts the same structure as the encoding stage of the G1 network. First, the feature map F₁ is obtained after six convolution layers with a filter size of 3 × 3 pixels². Then, the MSFF module is used to further extract features and obtain the feature map F_M. Two downsampling operations are carried out on the I_CR by six convolutional layers to obtain the feature map F₂ in the attention branch. A multiattention mechanism module is integrated immediately afterward, which is used to capture the local similarities and global dependencies of the feature maps and to output the feature map F_S. Finally, F_M and F_S are merged. Then, after a series of deconvolution operations, the data I_R generated by the multiattention filling network are obtained. The final fill result I_inpaint is obtained by the following formula:

$$I_{inpaint}=I_{in}\ast (1-M)+I_R\ast M$$

(3)

2.2.2. Multiattention Mechanism Module

The multiattention mechanism module takes feature map F₂ and mask M as input. First, this module inputs feature map F₂ to the channel attention module, which can calculate the attention fraction between each feature map on the channel domain to obtain the channel attention Mc and feature map F_C. Then, the F_C and mask M are input to the spatial cropping attention module. The attention score M_S, which indicates the correlation between the pixels on the feature map, is calculated. The final refined feature map F_R is obtained. The overall structure of the module is shown in Figure 4. The detailed structure of the channel attention module is shown in Figure 5, and the detailed structure of the spatial pruning attention module is shown in Figure 6.

The channel attention module takes the feature map F₂ as input and downsamples the feature map using maximum pooling and average pooling [17] to obtain the one-dimensional feature maps F_CM and F_CA. F_CM and F_CA are input to the three fully connected layers and output M_CM and M_CA to calculate the dependency between feature maps. Finally, M_CM and M_CA are combined into the channel dimension, and the channel attention M_C is obtained by using the sigmoid activation function. The feature map F₂ is matrix multiplied with the channel attention M_C to obtain the feature map F_C.

The input of the spatial pruning attention module is the feature map F_C and mask M. Traditional methods use maximum pooling or average pooling to downsample interchannel feature maps, but this method is too “violent” and can easily cause a large amount of feature information loss [39]. Therefore, the feature map F_C is downsampled with four convolutional kernels of different sizes, whose structure is shown in Figure 6.

The spatial pruning attention mechanism obtains the correlation between feature values by matrix multiplication between feature maps. The feature correlation map ρ between the feature maps w₁(F_C) and w₂(F_C) is calculated after convolution, which is calculated by

$$\rho =w_1{(F_c)}^T\otimes w_2(F_c)$$

(4)

where w(∙) is a 1 × 1 pixels² convolution kernel and ⊗ denotes matrix multiplication.

With the calculation above, the pruned feature correlation map ρ_Μ is obtained by multiplying by the downsampled mask M and the feature correlation map ρ. By pruning, the less relevant feature values are removed and features are only focused on void filling, which is calculated with

$${\rho }_M=\rho \odot M$$

(5)

where ⊙ denotes matrix dot multiply.

After obtaining the pruned feature correlation map ρ_Μ, the global association score K_ρ is calculated using the softmax function by

$$K_{\rho }=\frac{\exp ({\rho }_M)}{{\displaystyle {\sum }_i\exp ({\rho }_M^i)}}$$

(6)

Finally, the feature map w₃(F_C) is multiplied by the global correlation score K_ρ to obtain the spatial attention M_S. Then, M_S is multiplied by the feature map F_C to obtain the final output refined feature map F_R.

$$M_s=w_3(F_c)\otimes K_{\rho }$$

(7)

$$F_{\mathrm{R}}=F_c\otimes M_s$$

(8)

2.3. Global-Local Adversarial Network

The global-local adversarial network, consisting of the two parts of the global discriminator and local discriminator, is shown in Figure 7. By adding a local discriminator to the traditional generative adversarial network, it can constrain whether the filling void area is smooth and coherent with the known region. The inputting data of the global discriminator network are the original data and the data after G2 filling. First, five convolutional layers are used for downsampling, where the size of the convolutional kernel is 5 × 5 pixels² and the step strides are 2. Notably, a spectral normalization constraint is applied to each convolutional layer [50]. The extracted features are then reshaped into a vector of length 2048 by a fully connected layer. The local discriminator takes the filled data at the void area and the original data of the void area as input. The feature extraction process is similar to that of the global discriminator, which first passes through four convolutional layers for downsampling. The filter size of the convolutional kernel is 4 × 4 pixels², and the step strides are 2. The extracted features are then reshaped into a vector of length 2048 by a fully connected layer. Finally, the vectors output by the local discriminator and the global discriminator are combined into a vector of length 4096 and input to a fully connected layer. The probability that the output data are true data after the sigmoid activation function needs to be determined. The parameters of the generator and discriminator are updated by computing the gradient of the loss function with the penalty, i.e., the data fill results are discriminated by combining the recognition results of the two discriminators for the data.

2.4. Combined Loss Function

Generally, there is an objective function in the deep learning model. The training process of the network seeks the optimal parameters of a set of networks by solving for the optimal value of the objective function. Unlike other filling models, the Wasserstein distance in WGAN is applied to measure the mapped and original data [51]. In addition, a gradient penalty is added to the adversarial loss of the WGAN. The Wasserstein distance generates a continuously changing gradient to optimize the entire network, and the gradient penalty avoids extremes in the discriminator network parameters. Therefore, the adversarial loss function in this paper can be expressed as:

$$\min \max \{E_{x_r\sim p_r}[D(x_r)]-E_{x_g\sim p_g}[D(x_g)]+\alpha E_{x^{\prime }\sim p_{x^{\prime }}}{[|{\nabla }_{x^{\prime }}D(x^{\prime }){|}_2-1]}^2\}$$

(9)

$$x^{\prime }=\epsilon x_r+(1+\epsilon )x_g$$

(10)

where x_r is the real data and P_r is the probability density distribution of the original data. x_g is the mapped data, and P_g is the probability density distribution of the mapped data. $x^{\prime }$ is a random interpolation sample between x_r and x_g, and $P_{x^{\prime }}$ is the probability density distribution of $x^{\prime }$. $\epsilon$ is a random number obeying a uniform distribution on the interval [0,1]. α is the weighting factor of the penalty term.

The adversarial loss function of the generator is:

$$\min E_{x_g\sim p_g}[-D(x_g)]$$

(11)

The adversarial loss function of the discriminator is:

$$\min \{E_{x_g\sim p_g}[D(x_g)]-E_{x_r\sim p_r}[D(x_r)]+\alpha E_{x^{\prime }\sim p_{x^{\prime }}}{[||{\nabla }_{x^{\prime }}D(x^{\prime })|{|}_2-1]}^2\}$$

(12)

In GAN, to make the generated image have the same features as the real image, it is also necessary to consider regularization at the pixel level. Therefore, reconstruction loss is chosen to optimize the generator. The reconstruction loss uses the L1 or L2 norm to constrain the pixel-level information of both to prevent overfitting [48]. Since the model eventually only needs to consider data from the void area, a binary mask is added to indicate the location of the void area.

$$L_{pix}=\frac{1}{n}{\displaystyle \sum _{i=1}^{n}M\odot \left|\right|x_r^i-x_g^i|{|}^2}$$

(13)

where M is the mask matrix. ⊙ denotes the matrix dot multiplied, and n is the total number of image pixels.

Therefore, the reconstruction loss and the adversarial loss is combined as the model’s objective function, i.e.,

$$L={\lambda }_{pix}{L}_{pix}+{\lambda }_{adv}{L}_{adv}$$

(14)

where L is the objective loss of the model. L_pix is the reconstruction loss, and the L2 norm is used as the measurement distance of the reconstruction loss in this paper. λ_pix and λ_adv are their weighting coefficients.

3. Experiment and Analysis

3.1. Experimental Data and Preprocessing

3.1.1. Datasets

The experimental data in this paper were obtained from the ASTER GDEM V2 dataset (https://search.earthdata.nasa.gov/search/, accessed on 26 October 2021), with a data resolution of 30 m. To test the performance of our filling model, DEMs from four regions of China (see Figure 8), namely, Guangxi Province, Guizhou Province, Heilongjiang Province and Xinjiang Province, were selected as training samples, where the types of terrain included hills, plains and valleys. To reduce the training time, the DEMs are cropped to (64 × 64) size. A total of 10,000 DEM images are obtained, of which 9500 are used as the training dataset and 500 are used as the test set. We randomly generate a mask matrix with sizes ranging from (12 × 12) to (32 × 32) during the training process. The original DEM image and mask matrix are then used as input data to simulate the DEM image with holes. We make 5 mask operations on each DEM image during testing to verify the accuracy of the model. We use the mean error, mean absolute error, root mean square error, peak signal-to-noise ratio and structural similarity index as the evaluation indices of accuracy. The experiments include the repair of images with different areas and different hole sizes. In addition, a comparison analysis with other different repair methods is performed.

3.1.2. Data Preprocessing

To accelerate the convergence of the calculation and stabilize the network against training, the tailored DEM image training set is normalized to the [0, 1] range by normalizing the elevation values of each image before the training starts. Because the activation function of the final output layer of the model is tanh, the value of the output image is in the range of [−1, 1]. It is also necessary to reduce the value of the normalized image to the [−1, 1].

For the DEM image training set, the randomly generated binary masks are used for matrix multiplication, and for simulating DEM images with voids. The size of the mask matrix is the same as the size of the original image, but the values in the mask matrix include only 0 and 1. The mask matrix is multiplied by the original image pixel by pixel. The location multiplied by 0 is the area to be filled, and the location multiplied by 1 is the known area. As shown in Figure 9, this paper uses the product of the original data and the mask matrix to obtain the data with voids. To smoothly connect the filled area with the known area, local pre-fill is used by expanding the filling area to be filled outward by a few pixels, which can help the model fit the features at the edges of the original void area.

To balance the adversarial training between the discriminator and generator, the model is pre-trained with T_C times. The detailed training steps are shown in

Table 1. Training steps of the filling model.

Input: Original DEM, Mask matrix

Output: Initial generation of the data for G1, Refined filling data of G2, Discriminator loss value

Step 1: Dividing the original DEM into a training set, test set and setting the network training hyperparameters.

Step 2: Generating random mask masks for each DEM image to obtain simulated hole DEM images.

Step 3: When i < T do

Sampling m samples from the training set as mini_batchsize and entering the corresponding mask matrix.

for i < TC

Calculating the reconstruction loss values and updating the G1 network using the Adam optimization algorithm.

else:

Inputting the generated image and the original image into the discriminator, calculating the adversarial loss and L2 loss values of the discriminator D, and updating the discriminator parameters.

Finishing TC pretraining, calculating L2 reconstruction loss, adversarial loss, and updating G1, G2 and D networks using Adam optimization until training is completed and saving the model.

end

Step 4: Sampling m sample data from the test set as mini_batchsize, randomly generating mask, inputting to the trained filling model, putting the test data through G1 and G2 networks, respectively, getting the repaired DEM images, and calculating the repair accuracy.

.

This experiment adopts the Adam gradient optimization algorithm [52] to optimize the model, which is an adaptation learning rate optimization algorithm. In this experiment, the hyperparameter learning rate of the Adam optimizer is set to 2.0 × 10⁻⁴, and beta1 is the default value. After several experiments, the coefficients of the model combined loss function were set to λ_pix = 0.99 and λ_adv = 0.01. The model was trained for a total of T = 2000 epochs, where the number of portraits of the rough generative DEM network was T_C = 100 epochs.

3.1.3. Evaluation Metrics

In order to further evaluate the accuracy of filling voids, this paper selects mean error (ME), mean absolute error (MAE), and root mean square error (RMSE) as evaluation indexes, i.e.,

$$ME=\frac{1}{N}{\displaystyle \sum _i^N(y_i-{y^{\prime }}_i)}$$

(15)

$$MAE=\frac{1}{N}{\displaystyle \sum _i^N\left|y_i-{y^{\prime }}_i\right|}$$

(16)

$$RMSE=\sqrt{\frac{1}{N}{\displaystyle \sum _i^N{(y_i-{y^{\prime }}_i)}^2}}$$

(17)

where $y_i$ and $y_i^{\prime }$ represent the elevation value of each pixel of the filling result and the original data, and N is the total number of pixels in the void area of each datum.

In addition, this paper uses peak signal-to-noise ratio (PSNR) [53] and structural similarity (SSIM) [54] for evaluation of the similarity structure of terrain surrounding the void.

$$PSNR=10*lo{g}_{10}\left(\frac{{(2^n-1)}^2}{MSE}\right)$$

(18)

where MSE is mean square error.

$$SSIM(R,F)=\frac{(2{\mu }_R{\mu }_F+c_1)(2{\sigma }_{RF}+c_2)}{({\mu }_R^2+{\mu }_F^2+c_1)({\sigma }_R^2+{\sigma }_F^2+c_2)}$$

(19)

where R is the original data, F is the result data, ${\mu }_R$ is the average gray of R, ${\mu }_{RF}$ is the average gray of F, and ${\sigma }_{RF}$ is the covariance of R and F; ${\sigma }_R$ is the standard deviation of R, ${\sigma }_F$ is the standard deviation of F, and c₁ and c₂ are constants with extremely small values, which are used to prevent the denominator from being zero.

3.2. The Filled Results in Four Test Areas

To verify that the proposed filling model can effectively fill the missing areas in the DEM, four representative areas were selected. The types of landforms are located in mountains or valleys. Then, the void area was randomly dug, and the results are shown in Figure 10. The experimental conclusion is as follows.

Area 1 is located in the northwest of Heilongjiang Province, with relatively gentle terrain and simple topography. The maximum height difference is 169 m (the minimum elevation is 326 m, and the maximum elevation is 495 m). As shown in Figure 10(a1), the void area is located at the hill slope, and the void crosses a large area. From the experimental results, the boundary connection between the filling result and the original DEM is rather smooth. In addition, the characteristics of the void area have been recovered. As shown in

Table 2. Filling accuracy of results.

	ME (m)	MAE (m)	RMSE (m)	PSNR (dB)	SSIM
Area 1	1.37	4.69	5.77	29.33	83.74%
Area 2	4.22	10.13	12.80	33.52	92.93%
Area 3	8.61	12.23	15.79	28.27	82.94%
Area 4	6.06	11.88	15.85	21.71	59.84%

, the value of ME is 1.37 m, and the value of RMSE is 5.77 m, both of which are the smallest errors among the four experimental areas. The value of SSIM is 83.74%. The experiments show that the fill results have small errors and have high similarity to the original DEM results.

Area 2 is located in the southwest of Xinjiang Province, and the topography is relatively simple. The void area is located in a valley, but the area’s topography is highly variable, with a maximum height difference of 607 m (the minimum elevation is 3187 m, and the maximum elevation is 3794 m). As shown in Figure 10(c2), the terrain characteristics filled by the model are basically consistent with the original image, and the boundary connection with the original DEM is smooth. As shown in Table 2, the value of ME is 4.22 m, the value of RMSE is 12.80 m, and the value of SSIM is as high as 92.93%, resulting in a high similarity with the original DEM.

Area 3 is located in southwestern Guizhou Province, and the topography is relatively simple. The void area is located in a mountain, and the maximum height difference in the area is 409 m (the minimum elevation is 1168 m, and the maximum elevation is 1577 m). As shown in Figure 10(c3), the boundary connection between the fill result and the original DEM is smooth, but some detailed features of the terrain are lost simultaneously. As shown in Table 2, the value of ME is 8.61 m, the value of RMSE is 12.23 m, and the value of the structural similarity metrics reaches 82.94%, which is highly similar to the original DEM.

Area 4 is located in southwestern Guangxi Province, the terrain texture characteristics are rather complex, and the surface is not smooth. The void area includes mountain ridges and valleys, and the maximum height difference of the area is 193 m (the minimum elevation is 259 m, and the maximum elevation is 452 m). As shown in Figure 10(c4), the model fills the general topography within the void area of the original image, but it is not accurate in filling in some details of the terrain texture characteristics. From Table 2, it can be seen that the filling results have approximately 60% similarity with the original DEM.

From the filling results of the four experimental regions, it can be concluded that our filling model can effectively fill the DEM voids and smoothly connect the filling area with the surrounding original area without the visual boundary. For some flat terrain areas, the structural similarity metrics of the filling results reached more than 90%. In some complex terrain areas, the similarity metrics of the filling results also reached approximately 60%.

3.3. Comparison Analysis with Traditional Methods

To verify the excellence of our model, the comparison and analysis with three traditional interpolation methods, kriging, IDW, and spline interpolation, is conducted. The results are shown in Figure 11, in which a mountainous region is selected, a binary mask matrix is selected and used as the mask to fake into a DEM with a large area of voids. The maximum elevation difference is 242.47 m (the minimum elevation is 1122.95 m, and the maximum elevation is 1365.42 m). As in Figure 11a, the black rectangular area is the area to be filled. Figure 11b shows the enlargement of the filling results. Figure 11(d1–d4) show the filling results from different interpolation methods, and Figure 12 shows the 3D display of the filling results.

The experimental results show that the traditional interpolation method cannot fill large-area DEM voids; not only is the accuracy of filling low, but some topographic features generated do not match the original DEM. As shown in Figure 11(c1,c2), the filling results of kriging interpolation and inverse distance interpolation are relatively smooth, and the detailed features of the terrain are lost. Although the boundary of the filling result can be smoothly connected to the original DEM, topographic abruptness is generated in the void area. Among these, the mean error of kriging interpolation is 14.54 m, the root mean square error is 27.01 m, and the structural similarity metrics is 74.10%. The mean error of inverse distance interpolation is 19.82 m, the root mean square error is 38.04 m, and the structural similarity metrics is 61.90%. Figure 11(c3) shows that the spline interpolation method generates a relatively blurred filling result with worse terrain detail recovery and edge connection processing. The mean error of the spline interpolation is 0.27 m, the root mean square error is 21.48 m, and the structural similarity metric is 66.18%.

In addition, the model proposed in this paper is better than other traditional interpolation methods. Not only is the filling accuracy higher than that of the conventional interpolation methods, but the filling results in this paper conform to the topographic change, i.e., the filling results have a higher structural similarity with the original DEM. From the visual effect, the filling effect of our method can effectively restore the detailed features of the terrain and smoothly connect the boundary of the void with the original boundary. The MAE is 9.74 m, and the RMSE is 11.90 m. Compared with other interpolation methods, these two types of errors are the smallest. As shown in

Table 3. Accuracy analysis with traditional methods.

	ME (m)	MAE (m)	RMSE (m)	PSNR (dB)	SSIM
Kriging	14.54	20.14	27.01	24.76	74.10%
IDW	19.82	30.21	38.04	21.80	61.90%
Spline	0.27	14.82	21.48	26.77	66.18%
This paper	2.99	9.74	11.90	31.90	84.42%

The bold values are the best values for each column.

, the structural similarity metrics of our method is 84.42%. Although the PSNR values of the filling results using the traditional interpolation method are all above 20, indicating a good filling effect, the filling results do not match the original DEM in terms of visual effects.

3.4. Comparison Analysis with Other Deep Learning Models

In order to further verify the excellence from our model, comparison analysis with other deep learning models, such as the CE model proposed by Pathak et al. [32], the GL model proposed by Iizuka et al. [34], and the CR model proposed by Yu et al. [42], is conducted. The results are shown in Figure 13 and Figure 14. The results for quantitative analysis are depicted in

Table 4. Accuracy analysis with other deep learning models.

	ME (m)	MAE (m)	RMSE (m)	PSNR (dB)	SSIM
CE	3.94	9.64	17.02	20.18	43.05%
GL	10.80	25.97	34.66	16.36	17.07%
CR	7.13	17.19	22.19	20.23	11.42%
This paper	4.20	13.13	16.72	22.69	57.40%

The bold values are the best values for each column.

.

It can be seen from Table 4 that the PSNR values from the GL model are the smallest (16.36 dB), and PSNR values from the CE model and the CR model achieve 20.18 and 20.23 dB, respectively, while the PSNR value from the model proposed in this paper achieves 22.69 dB, which is slightly higher than that of the CE model and the CR model. In addition, as observed from Figure 13(c2,c4) and Figure 14d,f, it can be found that the filling result of DEM from the GL model is the most blurred visually compared to the results of the other three models, which are relatively clear.

Moreover, as observed from Table 4, the SSIM value from the CR model is the lowest out of the four models, which implies that the filling result of DEM only has 11.42% similarity with the original DEM (see Figure 14e), while the SSIM value from the model proposed in this paper achieved only 57.40%, which means that the filling result of DEM has 57.40% similarity with the original DEM (see Figure 14f). Other cases are between them (see Figure 14c,d). Therefore, it can be concluded that the model proposed in this paper has the highest structure similarity with the DEM surrounding the void (i.e., highest SSIM), which means that other models only obtain a relatively lower SSIM than our model does.

3.5. Discussion

3.5.1. Impact Analysis of the Attention Mechanism vs. Filling Accuracy

To verify the effectiveness of the attention mechanism on the model proposed in this paper, the network models with and without the attention mechanism are trained separately. Each model is trained for 1000 epochs, and finally the trained models are used to fill the test data. The results are shown in Figure 15 and

Table 5. Accuracy analysis with other deep learning methods.

	ME (m)	MAE (m)	RMSE (m)	PSNR (dB)	SSIM
Without attention	9.76	13.51	19.90	26.67	59.25%
With attention	2.56	12.50	15.73	28.72	75.57%

The bold values are the best values for each column.

. It can be seen from Figure 15(c1,c2) that rich details of terrains can be achieved after adding the multiattention mechanism. Specifically, the DEM voids filled with the attention mechanism have a greater improvement than those without the attention mechanism (see Table 5). The accuracy of ME is increased by 281.25%. The accuracy of MAE and RMSE is increased by 8% and 26.5%. The accuracy of PSNR and SSIM has increased by 7.7% and 27.54%.

3.5.2. Impact Analysis of the Loss Function Type vs. Filling Accuracy

The proposed model is trained using a combined loss function, including reconstruction loss and adversarial loss. In order to verify the effect of the loss function on the filling accuracy, this paper uses reconstruction loss, adversarial loss and the combined loss as the model objective function to conduct comparative experiments. The results are depicted in Figure 16.

Figure 16 shows that the slope variation in Area 1 is relatively smooth and that the terrain is relatively simple. The effect of training with reconstruction loss and adversarial loss alone or in combined loss has little effect on the filling, and both can fill the voids with high accuracy. In Area 2, the mountain chains are connected. The filling results of the model trained separately using reconstruction loss and adversarial loss are blurred, and the terrain details are not sufficiently recovered. However, the filling results of the model obtained from the combined loss training not only fill many detailed terrain textures but also connect smoothly with the original DEM boundary. The terrain in Area 3 is more complex and contains many detailed terrain features. The experiments show that the filling results of the models trained using reconstruction loss and adversarial loss, are, respectively, very blurry, and many terrestrial details cannot be filled. Area 4 is located in a valley area, where the voids area is located at the junction between the hillside and the flat land, and the intersection is divided more clearly. The experiments show that the results from our model trained through the combined loss function can produce a consistent structural similarity with the original DEM’s. The results of the filling model trained with adversarial loss can recover more texture features of the terrain but produce abrupt changes in elevation with the original DEM, and the filling yields some redundant features. Thus, it can be concluded from the experiments that different loss functions have a great influence on the filling accuracy of the model. In contrast, the combined loss function has strong robustness, enabling the model to recover more terrain features.

3.5.3. Impact Analysis of the Size of DEM Void vs. Filling Accuracy

The accuracy of filling DEM voids replies on the known information surrounding the DEM voids. The farther away from the void, the less the information contributes to the fill; thus, the size of the void largely determines the filling accuracy. In order to study how the size of the DEM void impacts the filling accuracy, this paper randomly generated 70 sets of masks of different sizes from 5 × 5 pixel² to 50 × 50 pixel², which are located in ridges and valleys. The ME, RSME, PSNR, and SSIM are calculated and shown in Figure 17. As observed from Figure 17, it can be found that the values of the PSNR and SSIM are linearly negatively relevant to the sizes of the DEM void, and the ME and RSME are linearly positively relevant to the sizes of the DEM void. Moreover, when the size of the DEM void is less than 20 × 20 pixels², the values of the ME, RSME, PSNR, and SSIM are relatively scattered, which is caused by topographical undulations. When the size of the DEM void is bigger than 20 × 20 pixels², the values of the ME, RSME, PSNR, and SSIM are relatively followed up by a linear increase, which means that the filling accuracy decreases linearly (see Figure 17). Therefore, it can be concluded that the model proposed in this paper can achieve the highest accuracy when the size of the DEM void is less than 20 × 20 pixels², and the SSIM value can achieve 60%, even 80% above.

4. Conclusions

This paper proposes a multiattention generative adversarial network model to fill the DEM voids. The novelties of this paper lie in the following:

(1)

A multiscale feature fusion generation network in the model is proposed, with which the receptive field is enlarged while maintaining the density of the dilated convolution.

(2)

A channel-spatial cropping attention mechanism module is proposed for the multiattention filling network, with which the correlation between the front and back feature maps is enhanced and the global-local dependence of the feature maps is improved.

(3)

To overcome the difficulty of balancing generator and discriminator adversarial training in generative adversarial networks, this paper proposes a global-local adversarial network and uses the spectral normalization on the output of the network layer, enhancing the stability of network training as a result.

Three groups of experiments are conducted to validate the model proposed. The first group of experiments is to verify the accuracy of the model proposed in this paper. The four different types of terrains, hillsides, valleys, ridges and hills, are selected for validation. Experimental results show that the SSIM value from our model in three types of terrains, hillside, valley and ridge, can reach 80–90%. The second group of experiments is to validate the model proposed in this paper through comparison with the traditional interpolation methods, i.e., Kriging, IDW, and Spline. The experimental results show that the structural similarity from the model proposed in this paper can averagely increase by 10% when compared with the traditional four interpolation methods, i.e., Kriging, IDW, and Spline. The third group of experiments is to validate the model proposed in this paper through comparison with other deep learning models, i.e., CE, GL, and CR. The experimental results show that the SSIM value from the model proposed in this paper can reach 57.4%, while the CE, GL and CR only reach 43.05%, 17.07% and 11.42%, in hilly terrain, respectively. In addition, this paper has also discussed and analyzed the impacts of attention mechanism, loss function, and void size versus the filling accuracy of the DEM void. The experimental results from our model show that (1) the filling accuracy of the DEM void is improved by 16% after adding the attention mechanism; (2) the combined loss function when used enables the model to recover more terrain features, and the results of the filled voids have a consistent structural similarity with the original DEM’s; (3) when the void size is less than 20 × 20 pixels², the SSIM value can reach 60–80% in the four test areas. With all of the experimental results and comparison analysis above, it can be concluded that the model proposed in this paper enables the DEM voids to be filled with higher accuracy and higher structural similarity than the traditional interpolation methods (i.e., IDW, Kriging, and Spline) and the deep learning models (i.e., CE, GL, and CR) under different terrestrial conditions.