Thick Cloud Removal in Multi-Temporal Remote Sensing Images via Frequency Spectrum-Modulated Tensor Completion

Abstract

Clouds often contaminate remote sensing images, which leads to missing land feature information and subsequent application degradation. Low-rank tensor completion has shown great potential in the reconstruction of multi-temporal remote sensing images. However, existing methods ignore different low-rank properties in the spatial and temporal dimensions, such that they cannot utilize spatial and temporal information adequately. In this paper, we propose a new frequency spectrum-modulated tensor completion method (FMTC). First, remote sensing images are rearranged as third-order spatial–temporal tensors for each band. Then, Fourier transform (FT) is introduced in the temporal dimension of the rearranged tensor to generate a spatial–frequential tensor. In view of the fact that land features represent low-frequency components and fickle clouds represent high-frequency components in the time domain, we chose adaptive weights for the completion of different low-rank spatial matrixes, according to the frequency spectrum. Then, Invert Fourier Transform (IFT) was implemented. Through this method, the joint low-rank spatial–temporal constraint was achieved. The simulated data experiments demonstrate that FMTC is applicable on different land-cover types and different missing sizes. With real data experiments, we have validated the effectiveness and stability of FMTC for time-series remote sensing image reconstruction. Compared with other algorithms, the performance of FMTC is better in quantitative and qualitative terms, especially when considering the spectral accuracy and temporal continuity.

1. Introduction

Satellite remote sensing images have been widely used in many fields, such as geography, ecology and environment monitoring [1,2], and have become one of the most important means of obtaining information on the Earth’s surface. However, satellite remote sensing images are prone to contamination by clouds, which causes great difficulties in target detection [3,4], identification, feature classification [5,6] and other applications [7,8]. For this reason, remote sensing image inpainting has become an active research area [9]. Many methods have been proposed to deal with the reconstruction of missing areas due to cloud contamination. Depending on the information used, these methods can be classified into three categories: spatial-based, spectral-based and temporal-based.

The principle of spatial-based methods is to fill in the missing areas in remote sensing images by propagating surrounding similar structures and texture information. These methods include image interpolation [10,11], propagated methods [12], compressive sensing methods [13], group-based methods [14] and variation-based methods [15,16]. However, the spatial-based methods can only be used to reconstruct small missing areas. When the area contaminated by clouds is large, these methods are not applicable.

The spectral-based methods aim to reconstruct the missing areas of remote sensing images using the correlation between different bands or different sensors’ images, especially for multispectral images. Among these methods, Aqua MODIS band 6 inpainting is the most typical. For instance, Rakwatin et al. [17] reconstructed the missing data in MODIS band 6 based on the correlation between band 6 and band 7. Roy et al. [18] used information observed by MODIS to predict Landsat ETM images. Since the infrared band can be used to obtain information on the land under thin clouds, Li et al. [19] used the data of the infrared band to reconstruct the cloud-contaminated area of a visual image. In addition, homomorphic filtering [20] and haze-optimized transform (HOT) [21] have been used to deal with thin cloud-contaminated image reconstruction. Since almost no optical band can penetrate the thick clouds, these methods work less well when images are contaminated by thick clouds.

The temporal-based methods use information from time-series remote sensing images in the same regions to reconstruct missing areas [22]. Melgani et al. [23] reconstructed the missing regions via an unsupervised contextual prediction process, which uses local spectral–temporal relations at different times. Similarly, Zhang et al. [24] and Lin et al. [25] reconstructed contaminated areas via the correlation of information from other temporal images. In addition, the multi-temporal dictionary learning algorithm was also used for image reconstruction [26]. These methods only consider information from the temporal dimension, and fail to take advantage of information in other dimensions, such that the reconstructed results will be limited by the availability of cloud-free areas.

To better utilize the information from the spatial, spectral and temporal dimensions for image reconstruction, many studies have used the low-rank tensor completion method [27]. For example, considering the contribution of the spatial, spectral and temporal information in each dimension, Ng et al. [28] proposed an adaptive weighted tensor completion method. Ji et al. [29] proposed a non-local low-rank tensor completion method, which introduced the non-convex approximation of tensor rank and rearranged the fourth-order tensor in groups. Chen et al. [30] considered spatial–spectral total variance regularized low-rank sparsity decomposition for cloud and shadow removal. Chu et al. [31] designed a novel spatial–temporal adaptive tensor completion method to reconstruct the data of cloud-prone regions. Duan et al. [32] proposed a tensor optimization model based on temporal smoothness and sparse representation (TSSTO) for thick cloud and cloud shadow removal in remote sensing images. Lin et al. [33] proposed a robust thick cloud/shadow removal (RTCR) method using coupled tensor factorization to meet the problem arising from an inaccurate mask, and thus reconstruct the multi-temporal information. Low-rank tensor approximation (LRTA) is an emerging technique [34], having gained much attention in the hyperspectral imagery (HSI) restoration community. Liu et al. [35] proposed a multigraph-based low-rank tensor-approximation method for HSI restoration, which integrated geometry-related information into LRTA to constrain a smooth solution of the restored HSIs.

Almost all existing tensor completion methods require the following two steps. First, singular-value threshold decomposition is performed for each mode of the tensor. Second, all tensors are weight-summed. However, for multi-temporal remote sensing images, the low-rank properties in the spatial dimension are different from those in the temporal dimension, which makes the selection of weight difficult. Therefore, in existing tensor completion methods used for remote sensing images, the temporal and spatial information is not utilized reasonably and effectively, which results in reconstructed images that are unclear or have spectral inaccuracy. In order to effectively utilize different low-rank properties in spatial and temporal dimensions, here, we propose a new frequency spectrum-modulated low-rank tensor completion method (FMTC).

FMTC treats time-series remote sensing images as the fourth-order tensor, and considers different low-rank processing approaches in the spatial and temporal dimensions. For the frequency spectrum of time-series remote sensing images, the temporal continuity, correlation and periodicity of land features are reflected in the low-frequency part, while clouds and noise influence the high-frequency part. On this basis, remote sensing images are rearranged as third-order spatial–temporal tensors in each band. The Fourier transform is introduced in the temporal dimension of the spatial–temporal tensor, which is converted to the spatial–frequential tensor. Low-pass filtering and the adaptive weights for each spatial matrix are performed via low-rank processing. After that, IFT is implemented. Through this method, joint spatial–temporal low-rank information is derived for reconstructing the missing areas. The main contributions of this paper can be listed as follows:

• Different orthogonal decompositions are performed on spatial and temporal dimensions of the tensor. The spatial–temporal tensor for each band is transformed to a spatial–frequential tensor by FT. Singular-value decomposition is performed in low-rank matrix completion in the spatial dimension. Through the frequency spectrum-modulating spatial matrix, joint spatial–temporal low-rank information is achieved, and the effects of different spatial–temporal low-rank properties are avoided. Meanwhile, using the property of conjugated symmetry of FT can also reduce the computation cost during the iteration.
• Gaussian low-pass filtering is applied in the frequency spectrum, and spatial low-rank adaptive weights are calculated according to the frequency characteristics of the time domain. Thus, the difficulty in selecting appropriate weights is solved. This scheme can maintain the low-frequency land features and weaken the high-frequency noise caused by clouds.

The rest of this paper is organized as follows: Section 2 introduces the basic idea, the model of tensor completion and the algorithm of this paper. Section 3 shows the results and analysis of simulated and real data experiments. Section 4 gives the conclusion.

2. Methodology

2.1. Spatial–Temporal Low-Rank Tensor Rearrangement

For multi-temporal remote sensing images obtained by satellites with high revisiting frequency, land features slowly change or are invariant over a period of time. Clouds break the temporal continuity of land features. Meanwhile, information on a region contaminated by thick clouds is lost in all bands. Therefore, more effective information for the reconstruction of images is given in the temporal dimension.

The original multi-temporal remote sensing images can be represented as $\mathit{Y}\in {ℝ}^{m\times n\times b\times t}$, where $m$ and $n$ are the spatial sizes of the image, $b$ is the number of bands, and $t$ is the number of time-series images. In order to make use of the information in the temporal dimension, $\mathit{Y}$ is rearranged, as shown in Figure 1. The data of each band are expressed as a spatial–temporal third-order tensor, and the number of tensors is $b$. In this paper, we handle the spatial–temporal tensor for each band separately. For convenience, we denote the spatial–temporal tensor as $\mathit{X}\in {ℝ}^{m\times n\times t}$. The initial spatial–temporal tensor is denoted as ${\mathit{X}}^0$. $\mathbf{\Omega }\in {ℝ}^{m\times n\times t}$ is the cloud mask tensor of ${\mathit{X}}^0$, where ${\mathbf{\Omega }}_{ijk}=1$ marks clear pixels and ${\mathbf{\Omega }}_{ijk}=0$ marks the pixels contaminated by clouds, and $\left(i,j,k\right)$ are the discrete indices of $\left(m,n,t\right)$.

Due to the correlation and continuity of remote sensing data in the spatial and temporal dimensions, the rearranged spatial–temporal tensor $\mathit{X}\in {ℝ}^{m\times n\times t}$ has low-rank properties. The following low-rank tensor completion model [36] can be applied to reconstruct images,

$$\underset{X}{\min }:\mathrm{rank}\left(\mathit{X}\right) s.t.:{\mathit{X}}_{\mathbf{\Omega }}={\mathit{T}}_{\mathbf{\Omega }},$$

(1)

where $\mathit{X}$, $\mathit{T}\in {ℝ}^{m\times n\times t}$, ${\mathit{T}}_{\mathbf{\Omega }}$ is the spatial–temporal tensor, including pixels that are not obscured by clouds. Instead of $\mathrm{rank}\left(\mathit{X}\right)$ of the trace norm, the optimization problem is changed to the following form [27]:

$$\underset{X}{\min }: {\parallel \mathit{X}\parallel }_{*} s.t.:{\mathit{X}}_{\mathbf{\Omega }}={\mathit{T}}_{\mathbf{\Omega }},$$

(2)

where ${\parallel \mathit{X}\parallel }_{*}$ is the trace norm of $\mathit{X}$. The unconstrained version of the problem [27] can be written as

$$\underset{X}{\min }:\frac{\mathsf{\rho }}{2}{\parallel {\mathit{X}}_{\mathbf{\Omega }}-{\mathit{T}}_{\mathbf{\Omega }}\parallel }_F^2+{\parallel \mathit{X}\parallel }_{*}.$$

(3)

where $\rho$ is a pre-set constant, and ${\parallel {\mathit{X}}_{\mathbf{\Omega }}-{\mathit{T}}_{\mathbf{\Omega }}\parallel }_F^2$ is the Frobenius norm of the tensor. In the form of the solution of Equation (3), the “shrinkage” operator $D_{\omega }\left(\mathit{X}\right)$ [37] is introduced into the low-rank processing of $\mathit{X}$, where $\omega$ is the weight used in the calculation of the trace norm for each mode. It is difficult for the spatial–temporal tensor to select appropriate weights for different spatial–temporal low-rank properties. FMTC is proposed to solve the problem.

2.2. Frequency Spectrum-Modulated Tensor Completion

Fourier transform is a method for projecting time-domain signals onto a set of orthogonal trigonometric bases. It is especially suitable for periodic data decomposition and processing. FMTC uses the frequency spectrum in the temporal dimension to adaptively determine the low-rank weight of each spatial slice matrix, which preserves the low-frequency components of the land features in the temporal dimension and limits the high-frequency noise caused by clouds. Then, we can achieve the joint spatial–temporal low-rank constraint.

First, we perform the Fourier transform on the temporal dimension of the rearranged tensor $\mathit{X}\in {ℝ}^{m\times n\times t}$ to generate a spatial–frequential tensor,

$$\widehat{\mathit{X}}=\mathrm{fft}(\mathit{X},[],3).$$

(4)

where [] denotes the transform length, which is the default value in matlab.

Then, for the spatial slice matrix ${\widehat{\mathit{X}}}_i\in {ℝ}^{m\times n}$of the spatial–frequential tensor $\widehat{\mathit{X}}$, $i=1,\cdots ,t$, the singular-value decomposition is performed:

$${\widehat{\mathit{X}}}_i=\overline{\mathit{U}}*\overline{\mathit{S}}*{\overline{\mathit{V}}}^{*}.$$

(5)

where $\overline{\mathit{U}}$, $\overline{\mathit{S}}$, $\overline{\mathit{V}}$ are the matrices of ${\widehat{\mathit{X}}}_i$’s singular-value decomposition [38].

Examples of a rearranged spatial–temporal tensor and the spatial–frequential tensor are shown in Figure 2. The low-frequency part represents slowly changing or constant land features, shown in the first two images in Figure 2b, and the high-frequency part represents significantly changed land features, clouds and noise, as shown in the last two images in Figure 2b. Figure 3b shows the time-series scatter plot of the near-infrared band from 2003 to 2008 for one pixel of Figure 3a, where the value 0 represents contamination by clouds or data invalidation. After removing the influence of clouds, the periodicity of land features that changes with seasons can be seen in the long-term image sequences. Figure 4 shows the Fourier transform spectrum curve of Figure 3. The maximum value represents zero frequency. The two large values denoted by red dots in the low-frequency part are caused by the periodic variation of land features. Noise and clouds appear in the high-frequency part.

According to frequency spectrum characteristics of the spatial–frequential tensor, we proposed the following operations for $\widehat{\mathit{X}}$. First, in order to maintain the low-frequency land features and weaken the high-frequency noise caused by clouds, a Gaussian low-pass filter is applied to $\widehat{\mathit{X}}$ in the frequency spectrum. Second, spatial adaptive weights determined by the frequency spectrum are performed to achieve the joint spatial–temporal low-rank constraint.

According to the properties of Fourier transform, there is a conjugate symmetry in the frequency domain of $\widehat{\mathit{X}}$. The Gaussian filter is designed as follows:

$$f\left(i\right)=e^{\frac{-{(i-1)}^2}{2{\mathsf{\sigma }}^2}},$$

(6)

where $i=1,\cdots ,\frac{t+1}{2}$, $f\left(i+1\right)=f\left(t-i+1\right)$, and $\sigma$ is a pre-set constant.

We define each spatial matrix of $\widehat{\mathit{X}}$ as ${\widehat{\mathit{X}}}_i$, $i=1,\cdots ,t$. Due to ${\widehat{\mathit{X}}}_i$ representing different frequency parts, when the same threshold is applied to perform the spatial low-rank processing of ${\widehat{\mathit{X}}}_i$, it will cause the loss of land feature information. In order to maintain land feature information in the image reconstruction, we choose the adaptive weight ${\omega }_i$ for ${\widehat{\mathit{X}}}_i$ according to the importance of the information from ${\widehat{\mathit{X}}}_1$ to ${\widehat{\mathit{X}}}_t$.

${\widehat{\mathit{X}}}_i$ is defined as the normalized mean value $k_i$ as follows:

$$k_i=\frac{m_i}{{{\displaystyle \sum }}_{i=1}^t{m}_i},$$

(7)

where $m_i$ denotes the mean value of ${\widehat{\mathit{X}}}_i$. A larger $k_i$ means more important information is contained in ${\widehat{\mathit{X}}}_i$, and a smaller ${\omega }_i$ should be used in the low-rank processing. The adaptive weight ${\mathsf{\omega }}_{\mathrm{i}}$ of each spatial matrix is defined as follows:

$${\omega }_i=\frac{\frac{1}{k_i^2}}{\rho \cdot {{\displaystyle \sum }}_{i=1}^t\frac{1}{k_i^2}}.$$

(8)

where $\rho$ is a pre-set constant.

2.3. Model Optimization

We define the augmented Lagrangian equation as follows:

$${\mathit{L}}_{\rho }\left(\mathit{M},\mathit{X},\mathit{B}\right)={\parallel \mathit{M}\parallel }_{*}+\langle \mathit{B},\mathit{M}-\mathit{X}\rangle +\frac{\rho }{2}{\parallel \mathit{M}-\mathit{X}\parallel }_F^2.$$

(9)

where $\mathit{M}$, $\mathit{B}$ and $\mathit{X}$ are updated by the alternating direction method of multipliers (ADMM) [39]. $\mathit{M}$ has a close-form solution:

$${\mathit{M}}^{k+1}=\mathrm{ifft}\left({\mathit{D}}_{{\omega }_i}\left(f\left(i\right)\cdot \mathrm{fft}\left({\mathit{X}}^k+\frac{{\mathit{B}}^k}{\rho }\right)\right)\right).$$

(10)

where $k$ is the iteration time. $\mathrm{ifft}(\cdot )$ is the Inverse Fast Fourier Transform. The adaptive weight ${\omega }_i$ is determined by Equation (8). For $\mathit{X}$, it is updated as follows:

$${\mathit{X}}^{k+1}=\{\begin{array}{c}{\mathit{M}}^{k+1}-\frac{1}{\rho }{\mathit{B}}^k \mathrm{if} {\mathbf{\Omega }}_{ijk}=0;\\ {\mathit{X}}^{\left(0\right)} \mathrm{if} {\mathbf{\Omega }}_{ijk}=1.\hfill \end{array}$$

(11)

For $\mathit{B}$, it is updated as follows:

$${\mathit{B}}^{k+1}={\mathit{B}}^k-\rho \left({\mathit{M}}^{k+1}-{\mathit{X}}^{k+1}\right).$$

(12)

In this paper, the initial $\rho$ is set to ${10}^{-4}$. Later, to accelerate convergence, $\rho$ will be iteratively increased by ${\rho }^{k+1}=t{\rho }^k$, $t\in \left[1.1,1.2\right]$. The solution process of FMTC is shown in Algorithm 1.

Algorithm 1 Optimization of the FMTC method

Input: Original remote sensing data $\mathit{Y}$, parameter $\rho$.

Initialize: ${\mathit{M}}^0={\mathit{B}}^0=0$$, \max \mathrm{iteration} \mathrm{times} K=200$. for$\mathit{i}=\mathbf{0}$to$\mathit{b}$do     Obtain the rearranged spatial–temporal tensor ${\mathit{X}}^0$ for each band.     for $\mathit{k}=\mathbf{0}$ to $K$ do

Update $\mathit{M}$ by Equation (10);

Update $\mathit{X}$ by Equation (11);

Update $\mathit{B}$ by Equation (12);

end for;

repeat until convergence;

end for;

Output: Reconstructed image data $\mathit{X}$.

3. Experimental Results

3.1. Experimental Data

In this part, Landsat Collection 1 L1TP data from 2003 to 2018 in Beijing have been selected as the experiment data, as shown in

Table 1. Experiment data information.

Source	Duration	Resolution	Band	Size
Landsat-5	2003–2011	30 m	1–6	$500\times 500$
Landsat-8	2013–2018	30 m	1–7	$500\times 500$

. The experiment data include five simulated datasets and two real datasets based on different land-cover types and cloud areas, respectively. In order to demonstrate the effectiveness of FMTC, five algorithms are selected for comparison, including HaLRTC [27], AWTC [28], NL-LRTC [29], TVLRSD [30] and ST-Tensor [31].

3.2. Evaluation Indicators

In the simulated experiments, the Peak Signal-to-Noise Ratio (PSNR) [40], Structural Similarity (SSIM) [41] and Spectral Angle Mapper (SAM) [42] are chosen to evaluate the reconstructed images from the spatial and spectral perspectives.

The PSNR is calculated as follows:

$$\mathrm{PSNR}=10\cdot {\mathit{log}}_{10}(\frac{MA{X}^2}{\mathrm{MSE}})=20\cdot {\mathit{log}}_{10}(\frac{MAX}{\sqrt{\mathrm{MSE}}});$$

(13)

$$\mathrm{MSE}=\frac{1}{mn}{\displaystyle \sum }_{i=0}^{m-1}{\displaystyle \sum }_{j=0}^{n-1}{\parallel I\left(i,j\right)-K\left(i,j\right)\parallel }^2.$$

(14)

where $MAX$ is the maximum pixel value of the original image, and $I\left(i,j\right)$ and $K\left(i,j\right)$ denote the $\left(i,j\right)\mathrm{th}$ pixel values of the original image and the reconstructed image, respectively. $m$ and $n$ are the spatial sizes of those images.

The SSIM is calculated as follows:

$$\mathrm{SSIM}\left(x,y\right)=\frac{\left(2{\mathsf{\mu }}_x{\mathsf{\mu }}_y+c_1\right)\left(2{\mathsf{\sigma }}_{xy}+c_2\right)}{\left({\mathsf{\mu }}_x^2+{\mathsf{\mu }}_y^2+c_1\right)\left({\mathsf{\sigma }}_x^2+{\mathsf{\sigma }}_y^2+c_2\right)}.$$

(15)

where ${\mathsf{\mu }}_x$ and ${\mathsf{\mu }}_y$ represent the mean values of the original image and the reconstructed image, respectively. ${\mathsf{\sigma }}_x$ and ${\mathsf{\sigma }}_y$ represent the standard deviations. ${\mathsf{\sigma }}_{xy}$ represents the covariance.

The SAM is calculated as follows:

$$\mathrm{SAM}=\mathit{arccos}(\left(x_1\cdot y_1+\dots +x_n\cdot y_n\right)/\left(\sqrt{x_1^2+\dots +x_n^2}\cdot \sqrt{y_1^2+\dots +y_n^2}\right)).$$

(16)

where $\left(x_1,\cdots ,x_n\right)$ and $\left(y_1,\cdots ,y_n\right)$ denote the original image and reconstructed image of the spectral vectors.

In the real data experiments, the above evaluation indicators cannot be used because there was no reference image. Therefore, the information entropy (IE) [43] and the average gradient (AG) [44] are used to evaluate the reconstructed image. In addition, the effectiveness of the algorithm can be evaluated by the consistency of spatial images, the clarity of feature targets and the continuity of the time-series curve.

Information entropy of the reconstructed image is calculated as follows:

$$\mathrm{IE}=-{\displaystyle \sum }_t^G{p}_t{\mathit{log}}_2{p}_t.$$

(17)

where $p_t$ is the probability of value $t$. $G$ is the maximum pixel value of the reconstructed image.

The average gradient of the reconstructed image is calculated as follows:

$$\mathrm{AG}=\frac{1}{\left(m-1\right)\left(n-1\right)}\times {\displaystyle \sum }_{i=1}^{m-1}{\displaystyle \sum }_{j=1}^{n-1}\sqrt{\frac{{(K\left(i,j\right)-K\left(i+1,j\right))}^2+{(K\left(i,j\right)-K\left(i,j+1\right))}^2}{2}}.$$

(18)

3.3. Simulated Data Experiments

In the simulated experiments, the effect of clouds is simulated by masking a random area on the cloud-free images. In order to demonstrate the effectiveness of FMTC on different land-cover types and missing areas, two simulated experiments were conducted. One is based on datasets 1–4 with four land-cover types, the other is based on dataset 5 with different sizes of missing areas. The information of the datasets is shown in

Table 2. Information on five datasets.

Dataset	Location	Land-Cover Types	Mask Date	Source
Dataset 1	Beijing City Center	Impervious	17 May 2009	Landsat-5
Dataset 2	Yanqing, Beijing	Soil	26 April 2007	Landsat-5
Dataset 3	Huairou, Beijing	Vegetation	2 June 2009	Landsat-5
Dataset 4	Miyun, Beijing	Water	7 May 2011	Landsat-5
Dataset 5	Pinggu, Beijing	Vegetation/Soil/Impervious	18 May 2015	Landsat-8

.

3.3.1. The Experiment Based on Different Land-Cover Types

In this part, Landsat-5 datasets for four different regions in Beijing from 2003 to 2011 were selected, as shown in Table 2. The experiment results are shown in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. Figure 5a, Figure 7a, Figure 9a and Figure 11a are true-color composition images (red—band 3, green—band 2, and blue—band 1). The simulated data have been generated from random missing areas where the pixel values were set to 0, as shown in Figure 5b, Figure 7b, Figure 9b and Figure 11b. Visual comparisons of FMTC with other algorithms are shown in Figure 5c–h, Figure 7c–h, Figure 9c–h and Figure 11c–h. Figure 6, Figure 8, Figure 10 and Figure 12 show enhanced details of the images.

According to the enhanced detail shown in Figure 5, Figure 7, Figure 9 and Figure 11, we can see that the reconstructed results of HaLRTC and AWTC are blurry for four land-cover types because these algorithms only utilize the low-rank information derived from experimental data. The results of NL-LRTC are better than those of the first two algorithms, but still a bit fuzzy, which is due to the inappropriate choice of weights for the different modes of the tensor. TVLRSD can reconstruct the basic structure and details of the images, but the reconstructed details are still not clear. This may be because the matrix decomposition denoising the image and removing clouds makes images smooth and vague, but the details are not reconstructed. Although ST-Tensor maintains much of the detail of the images, the color of the reconstructed areas is inconsistent with that of the surroundings, due to the inappropriate choice of weights in the rearranged spatial–temporal tensor. In contrast, FMTC is able to ensure spatial detail as well as spectral consistency.

A quantitative comparison for the six algorithms applied to four land-cover types is shown in

Table 3. Quantitative comparison of six algorithms applied to four land-cover types.

Land-Cover Type	Indicator	HaLRTC	AWTC	NL-LRTC	TVLRSD	ST-Tensor	FMTC
Impervious	PSNR	51.024	51.954	52.325	53.490	55.201	55.021
	SSIM	0.9926	0.9935	0.9941	0.9958	0.9992	0.9990
	SAM	0.0767	0.0782	0.6989	0.0603	0.0533	0.0530
	Time(s)	161.59	426.37	649.48	592.94	839.64	269.19
Soil	PSNR	38.839	39.235	40.865	41.876	41.914	41.975
	SSIM	0.9941	0.9951	0.9971	0.9983	0.9984	0.9987
	SAM	0.0437	0.0421	0.0326	0.0295	0.0289	0.0271
	Time(s)	186.13	438.46	659.46	526.35	837.09	294.14
Vegetation	PSNR	38.681	39.024	40.216	43.453	43.477	43.492
	SSIM	0.9985	0.9992	0.9993	0.9995	0.9996	0.9997
	Time(s)	168.38	362.47	0.0457	461.96	710.46	352.98
	SAM	0.0591	0.0588	574.18	0.0376	0.0374	0.0358
Water	PSNR	32.076	32.705	38.783	41.684	42.926	43.003
	SSIM	0.9664	0.9696	0.9762	0.9826	0.9874	0.9901
	SAM	0.0402	0.01386	0.0364	0.0358	0.0327	0.0321
	Time(s)	390.34	822.65	776.04	768.35	910.67	431.76

. HaLRTC, AWTC and NL-LRTC obtained poor results for all quantitative indicators. The results of TVLRSD and ST-Tensor are better than those of the above three algorithms. Generally speaking, FMTC can achieve much better results than the other five algorithms. Although the PSNR and SSIM of ST-Tensor seem to perform a little better than FMTC on impervious land, the SAM of ST-Tensor is significantly less effective than FMTC. In addition, FMTC shows outstanding performance on the three other land-cover types for each indicator. It can be seen that the results of the quantitative evaluation are consistent with visual examination. It is worth mentioning that FMTC is far less time-consuming than the other five algorithms.

3.3.2. The Experiment with Different Missing Sizes

In this simulated experiment, dataset 5 was used to demonstrate the effectiveness of FMTC on different missing sizes. The proportions of the missing area are 6.01%, 19.26% and 32.48%, respectively. Figure 13, Figure 14 and Figure 15 are visual presentations of the results for three missing sizes, respectively. Figure 13a, Figure 14a and Figure 15a show true-color composition images (red—band 4, green—band 3, and blue—band 2). Simulated cloudy images are shown in Figure 13b, Figure 14b and Figure 15b. Figure 13c, Figure 14c and Figure 15c show the reconstructed images. Figure 13d–f, Figure 14d–f and Figure 15d–f show, in enhanced detail, the areas in the red boxes from Figure 13a–c, Figure 14a–c and Figure 15a–c, respectively.

Visually, the reconstructed results are consistent with the surroundings, and the proportion of the missing size reaches 32.48%. According to the areas with enhanced detail, the results of reconstruction approach the original images. In Figure 13d, a white road is depicted. In Figure 13f, the white road is reconstructed completely, owing to the time-series information utilized by FMTC. The above results demonstrate the effectiveness of using FMTC with different missing area sizes.

A quantitative comparison of the six algorithms with different missing sizes is shown in

Table 4. Quantitative comparison of six algorithms with different missing sizes.

Missing Size	Indicator	HaLRTC	AWTC	NL-LRTC	TVLRSD	ST-Tensor	FMTC
6.01%	PSNR	39.658	40.675	45.319	48.355	49.521	49.531
	SSIM	0.9927	0.9941	0.9973	0.9984	0.9999	0.9998
	SAM	0.0863	0.0852	0.7126	0.0664	0.0635	0.0625
	Time(s)	191.19	483.61	593.45	563.95	784.55	277.23
19.26%	PSNR	26.208	26.783	37.634	43.639	44.022	44.083
	SSIM	0.9240	0.9336	0.9736	0.9959	0.9980	0.9979
	SAM	0.0924	0.0911	0.6089	0.0477	0.0446	0.0440
	Time(s)	326.74	684.39	715.64	706.97	936.21	386.42
32.48%	PSNR	25.785	26.199	37.599	40.868	42.815	42.844
	SSIM	0.8343	0.8482	0.9157	0.9945	0.9962	0.9964
	SAM	0.1018	0.0993	0.6943	0.0401	0.0374	0.0366
	Time(s)	403.51	704.62	903.49	873.56	1017.55	464.57

. As the proportion of missing size increases, the PSNR and SSIM of the first three algorithms decrease significantly, while the SAM increases. In contrast, the latter three algorithms are less sensitive to changes in missing sizes. Among all the algorithms, FMTC performs the best in most of the evaluation indicators. For different missing sizes, FMTC can obtain satisfactory results.

3.4. Real Data Experiments

In experiments with real data, two datasets were selected, as shown in

Table 5. Real data experiments’ datasets.

Dataset	Location	Duration	Mask Date	Source	Land Cover Type
Dataset 6	Changpin, Beijing	2003–2011	22 May 2005	Landsat-5	impervious/vegetation/soil
Dataset 7	Mentougou, Beijing	2013–2018	21 April 2017	Landsat-8	vegetation/impervious/soil

. For the real data, we used a Landsat quality assessment to mask clouds and cloud shadows. In order to demonstrate the effectiveness of applying FMTC on different data sources, two real data experiments were conducted.

3.4.1. Real Data Experiment 1

In this part, dataset 6 from Landsat-5 for the first real data experiment was selected. The main land cover type is impervious, and the others are vegetation and soil. Figure 16a shows the true-color composition image. Figure 16b shows the masked image. Visual comparisons of FMTC with other algorithms are shown in Figure 16c–h. Figure 17a–h show enhanced details of the areas in the red boxes.

The results of this experiment are similar to those of the simulated data experiment. The reconstructed results of HaLRTC and AWTC are not satisfactory and cannot meet the basic demands of reconstruction. The reconstructed details of TVLRSD and NL-LRTC are not clear, and the result of ST-Tensor is inconsistent with its surroundings. FMTC can not only clarify the reconstructed details, but it also ensures the consistency of the spectrum.

A quantitative comparison is shown in

Table 6. Quantitative comparison of six algorithms applied to dataset 6.

Indicator	HaLRTC	AWTC	NL-LRTC	TVLRSD	ST-Tensor	FMTC
IE	6.8669	6.8756	6.9317	6.9653	6.9470	6.9955
AG	0.0425	0.0438	0.0465	0.0469	0.0464	0.0476

. The IE and AG of FMTC are better than those of other algorithms, indicating that the reconstructed details are richer and clearer. Figure 18 shows the time-series curves of six algorithms applied to one pixel from 2004 to 2006. The value 0 represents that the pixel was contaminated by cloud on that day. The curves of HaLRTC, AWTC and NL-LRTC display values of 0, leading to unsatisfactory results. The curves of TVLRSD and ST-Tensor are better than those of the first three algorithms, while they still show abrupt changes for the cloudy pixels. The curve of FMTC looks more continuous than those of all other algorithms. It can thus better characterize the temporal continuity and periodicity of land features.

3.4.2. Real Data Experiment 2

In this part, dataset 7 from Landsat-8 was selected for use in the second real data experiment. The main land cover type is vegetation, and the others are impervious and soil. Figure 19a is the original image. Figure 19b is the masked image. Visual comparisons of FMTC with other algorithms are shown in Figure 19c–h. Figure 20a–h show enhanced details of the areas in the red boxes.

In terms of the visual effect, the results of this experiment are similar to those from real data experiment 1. As regards the areas of enhanced detail, TVLRSD is not clear enough. The white road, shown in the red box in Figure 20g,h, is not obvious in the reconstructed result of ST-Tensor, while the white road is reconstructed completely by FMTC. A quantitative evaluation is shown in

Table 7. Quantitative comparisons of six algorithms applied to dataset 7.

Indicator	HaLRTC	AWTC	NL-LRTC	TVLRSD	ST-Tensor	FMTC
IE	6.6871	6.6871	6.6881	6.6899	6.6910	6.6913
AG	0.0301	0.0312	0.0325	0.0336	0.0341	0.0341

. The time-series curves of the six algorithms applied to one pixel from 2015 to 2017 are shown in Figure 21. FMTC achieves the best performance, with excellent reconstructions from different data sources.

4. Conclusions

In this paper, we propose a frequency spectrum-modulated tensor completion method for multi-temporal remote sensing image inpainting. We rearrange the original four-dimensional tensor to establish a series of spatial–temporal tensors with low-rank properties. Fourier transform is introduced in the temporal dimension to generate the spatial–frequential tensor. The spatial adaptive weights are determined by the spectral properties of the time-domain signal, which is filtered using the Gaussian low-pass filtering to achieve joint spatial–temporal low-rank results. The experimental results based on both simulated and real datasets are presented to verify that FMTC can reconstruct missing data, starting from different land-cover types and different missing sizes. Compared with the other five methods used for the reconstruction of images, the results of FMTC are mostly better in terms of visual effects and quantitative evaluation. However, this method of image reconstruction still has some limitations. The reconstructed results may be affected by the detection of clouds and cloud shadows. In addition, FMTC encounters some limitations when applied to fickle objects, such as abrupt water.