SAR-to-Optical Image Translation and Cloud Removal Based on Conditional Generative Adversarial Networks: Literature Survey, Taxonomy, Evaluation Indicators, Limits and Future Directions

Abstract

Due to the limitation of optical images that their waves cannot penetrate clouds, such images always suffer from cloud contamination, which causes missing information and limitations for subsequent agricultural applications, among others. Synthetic aperture radar (SAR) is able to provide surface information for all times and all weather. Therefore, translating SAR or fusing SAR and optical images to obtain cloud-free optical-like images are ideal ways to solve the cloud contamination issue. In this paper, we investigate the existing literature and provides two kinds of taxonomies, one based on the type of input and the other on the method used. Meanwhile, in this paper, we analyze the advantages and disadvantages while using different data as input. In the last section, we discuss the limitations of these current methods and propose several possible directions for future studies in this field.

1. Introduction

Free optical imageries such as Sentinel-2 [1], GaoFen 1 (GF-1, GaoFen means high resolution in Chinese) [2], Landsat 8 [3] and Moderate Resolution Imaging Spectroradiometer (MODIS) [4] play essential roles on current agricultural applications, e.g., crop classification [5], cropland monitoring [6], analysis of historical samples [7], and research on arable land-use intensity [8]. However, due to the limitation of wavelength, these optical imageries inevitably suffer from cloud and shadow contamination, which could lead to inefficient earth surface reflectance data needed to conduct subsequent research [5,9].

In order to solve the above problems, researchers have begun to carry out research from two different ideas.

One idea is to fuse multisource [10], multitemporal [11], or multispectral [12] optical imageries. The rationale of this kind of idea is to use cloud-free imageries or bands on the reference phase to recover the missing information on the target phase [13]. The advantage of this method is that we have many multisource optical image data from which to choose. At the same time, each band range of these data is relatively similar, as is the reflectance of ground objects captured, which is more conducive to the collaborative fusion among the data. However, the disadvantage is that it is not effective at capturing reference data if cloud conditions occur continuously [14]. Thus, such methods cannot fundamentally solve the problem of missing information caused by the existence of cloud.

The other idea is to fuse the synthetic aperture radar (SAR) [15] and target optical imageries or use the SAR data only. SAR has a penetration capacity sufficient to capture surface features through cloud [16]. Therefore, in theory, the fusion of SAR and optical data can solve the deficiency of the first method. However, this approach has its limitations. SAR and optical remote sensing are fundamentally different in imaging principles, resulting in different features of some ground features (e.g., roads, playgrounds and airport runways) in spectral reflectance and the SAR backscattering coefficient. Therefore, it is still difficult to establish a mapping relationship between the two types of data on these ground features [17]. In addition, according to these authors’ original goal, we found that this part of the literature can be roughly divided into two categories: the first category is cloud removal on optical remote sensing data, and the second is SAR-to-optical translation. The second type of target was originally intended to facilitate the interpretation of obscure SAR images, but we believe that this type of target can be extended to the production of cloud-free optical remote sensing data. Although, for the second type, cloud removal is not performed on the basis of the original cloud contaminated data, it is similar to the results of the first type, where the cloud-free optical remote sensing data is obtained in the end. The research on cloud removal using SAR data discussed below actually includes these two types.

In addition, in most cases, these two methods will try to select the same space and time according to the target data when selecting reference data [18,19]. In terms of space, the closer the distance, the greater the correlation between ground objects, the greater the distance, and the greater the difference (the First Law of Geography [20]). In terms of time, the closer the interval is, the smaller the change between ground features will be. However, a few studies did not follow the above rules when selecting data. Zhang et al. used the similarity correlation measurement index to measure the correlation of all remote sensing images in the region within a period of time, sorted them according to index level, and selected the image with a high score as the input [21]. The unsupervised Cycle-Consistent Adversarial Networks (CycleGAN) [22] does not need the remote sensing image pairs in the same region. Although it is difficult to obtain the mapping function of styles between two images like supervised learning with this method, it can obtain a mapping relationship between styles through a series of images of two different styles, which is more convenient for the expansion of the dataset [23,24].

We used ‘TI = “cloud removal” OR AK = “cloud removal”’ as retrieving conditions and retrieved all the relevant literature through the Web of Science. At the same time, we used ’(TI = “SAR” OR AK = “SAR”)AND(TI = “cloud removal” OR AK = “cloud removal”)’ as retrieving conditions and retrieved the relevant literature through the Web of Science. On this basis, we further searched for the relevant papers through the citations obtained from this body of literature to expand the number of papers. Finally, the literature on optical data cloud removal based on SAR data were collected and sorted (Figure 1). The figure shows that, in studies on optical remote sensing data cloud removal, the first method of integrating optical remote sensing data starts earlier than the second method, and there are more papers on the first method. It can be seen in Figure 1 that the development of cloud removal methods used for optical data based on SAR started in recent years. Once generative adversarial networks (GANs) were constructed, many scholars tried to use this kind of neural network to study SAR-to-optical translation.

The study of cloud removal using optical remote sensing data alone has taken a long time to develop. So far, there have been a number of relevant literature reviews [25,26,27], which supports the further development of cloud removal research. However, due to the short development time of the research on cloud removal integrating SAR data, no relevant review can be found. The academic paper published by Fuentes Reyes et al. [28] could be regarded as a relevant review, but its content is more focused on the effect of the CycleGAN network in SAR-to-optical translation. Therefore, we hope that this review can indicate the current cloud removal development of optical remote sensing data using SAR data, especially the emerging methods based on deep learning from 2018.

2. Literature Survey

We researched cloud removal methods for optical imageries using SAR published in journals and conferences. We found 26 papers published in 9 international journals (

Table 1. Number of papers published in 9 journals using SAR for cloud removal. The search was conducted on 20 November 2022.

Source Titles	Records	Percentage	Literature
Remote Sensing	9	34.61%	[28,46,47,48,49,50,51,52,53]
IEEE Transactions on Geoscience and Remote Sensing	5	19.23%	[54,55,56,57,58]
IEEE Access	3	11.54%	[17,59,60]
IEEE Geoscience and Remote Sensing Letters	2	7.69%	[61,62]
IEEE Journal of Selected Topics in Applied Earth Observations	2	7.69%	[63,64]
and Remote Sensing
ISPRS Journal of Photogrammetry and Remote Sensing	2	7.69%	[65,66]
ISPRS International Journal of Geo-Information	1	3.85%	[67]
Journal of the Indian Society of Remote Sensing	1	3.85%	[68]
International Journal of Remote Sensing	1	3.85%	[69]
Total	26	100%

), 13 conferences papers [29,30,31,32,33,34,35,36,37,38,39,40,41] and 4 papers [42,43,44,45] published in arXiv (pronounced “archive”—the X represents the Greek letter chi [$\chi$]), which is an open-access repository of electronic preprints (known as e-prints) approved for posting after moderation but not peer review. Among these 26 journal papers, 14 were published in journals including Remote Sensing and IEEE Transactions on Geoscience and Remote Sensing. 84% of the papers were published after 2019, and the total number of papers is not large, which indicates that the research on cloud removal from optical remote sensing data using SAR data is still at a new stage and still has great potential. We divided these papers into five types of methods, as shown in Section 3.1, and the development of these methods, which are published in journals, is shown in Figure 2.

At the same time, in order to help scholars to search for papers with keywords, we carried out text analysis on the 26 papers and extracted the words commonly used in the titles (Figure 3). It can be seen in the figure that different scholars have different named methods for the title of the paper, but the words with the highest frequency are still about cloud removal of optical remote sensing data using SAR data, such as ’SAR’, ’cloud’, ’optical’, ’image’ and ’removal’. There are also the keywords about deep learning, especially generative adversarial networks, such as ’learning’ and ’adversarial’, which indicates that the generative adversarial networks is widely applied in this field. In the following, we will focus on the current situation, opportunities and challenges of using SAR data to decloud optical remote sensing data based on GANs.

We also plotted a literature citation map of jounal papers with VOSViewer, as Figure 4 shows. The total number of papers in the figure is 25, because [66] cannot be searched in the Web of Science Core Collection (WOSCC). The most cited paper is [62], which presents a cloud removal method for reconstructing the missing information in cloud-contaminated regions of a high-resolution optical satellite image using two types of auxiliary images, i.e., a low-resolution optical satellite composite image and a SAR image.

3. Taxonomy

This section classifies a part of the papers based on the methods used and the input data used. It is hoped that this multiangle classification method will enable scholars to have a more diversified understanding of this field.

3.1. The Categories of the Methods

It has been described in the introduction and literature survey that GANs have been widely used in optical remote sensing data decloud research based on SAR data, so we classified the methods into five categories according to GANs: the conditional generative adversarial networks (cGANs), the unsupervised GANs, the convolutional neural networks (CNNs, not GANs), the hybrid CNNs and other methods, as

Table 2. Categories of the existing cloud removal methods of optical remote sensing imagery using SAR. (¹ Names of these methods were extracted from the literature. If the methods were not named in the literature, the words describing the methods’ major characteristics were used to name them here. cGANs: conditional generative adversarial networks. CNNs: convolutional neural networks.)

Category	Methods ${}^1$ and References
cGANs	modified cGAN-based SAR-to-optical translation method [60];
	feature-guided method [59]; MTcGAN [67]; S-CycleGAN [17];
	Synthesis of Multispectral Optical Images method [61]
Unsupervised GANs	CycleGAN [28]
CNNs (not GANs)	DSen2-CR [65]; CMD [54]; SEN12MS-CR-TS [55]; G-FAN [48]
Hybrid CNNs	SFGAN [46]; PLFM [56]
Other methods	Sparse Representation [62];
	cloud removal based on multifrequency SAR [47];
	IHSW [68]; DGL-INJSC [63]

shows.

3.1.1. cGANs

The cGAN [70] is an extension of the original GAN [71], which is made of two adversarial neural network modules, the Generator (G) and the Discriminator (D). G attempts to generate the simulated images in domain Y from the realistic images in domain X to fool D. D is to be trained from realistic images in domain X and Y to discriminate the input data between synthetic and realistic. When training the cGAN model, the input has to be paired SAR/Optical data.

The flowchart of the cGAN is shown in Figure 5. The loss of the G comes from two parts, GANLoss and L1Loss. GANLoss expresses whether the simulated images y’ are able to fool the D (cause the D to judge the simulated images as real images). L1Loss presents the distance between real images y and simulated images y’. The G tries to make these two loss functions’ values close to zero, which means the simulated results are more realistic. Contrarily, the D is in charge of discriminating the real images y as true and the simulated images y’ as false, i.e., DLoss. Similar to the G, the D also tries to make the DLoss function’s value close to zero to improve its own distinguishing capacity.

GANLoss is a loss function that digitizes the process of the Generator and the Discriminator, which is literally the most fundamental loss function in GANs. The objective function for cGAN (${\mathcal{L}}_{cGAN}$(G,D)) could be expressed in Equation (1):

$${\mathcal{L}}_{cGAN}(G,D)={\mathbb{E}}_{x,y\sim p_{data}(x,y)}[logD(x,y)]+{\mathbb{E}}_{x\sim p\left(x\right),z\sim p\left(z\right)}[log(1-D(x,G(x,z)))]$$

(1)

where $\mathbb{E}$ and log are expectation and logarithmic operators, respectively; p is the distribution of images; z is a random noise vector that follows a prior known distribution $p\left(z\right)$, typically uniform or Gaussian. Ordinarily, a L1 norm distance loss (L1Loss) would be added to the objective function of the cGAN to cause the simulated images to be more similar to the realistic images, as shown in Equation (2):

$$G^{*}=arg\underset{G}{min}\underset{D}{max}{\mathcal{L}}_{cGAN}(G,D)+\lambda {\mathcal{L}}_{L1}\left(G\right)$$

(2)

where $\lambda$ is a regularization weight, and ${\mathcal{L}}_{L1}$ is defined as

$${\mathcal{L}}_{L1}\left(G\right)={\mathbb{E}}_{x,y\sim p_{data}(x,y),z\sim p\left(z\right)}{[\parallel y-G(x,z)\parallel }_1].$$

(3)

Some authors [60] have proposed a modified cGAN-based SAR-to-optical translation method that uses the structural similarity index measurement (SSIM) as a loss function to strengthen the connection between the generated image and the target image and produce the generated image with better details. Others [29,33,61,67] have added the channels of cGANs to use the information of the multitemporal paired SAR/Optical data, which improves the quality of the generated image. Some authors [59] have paid more attention to the high-dimension feature extraction with VGG19 [72], which leads to better results. Although S-CycleGAN [17] uses supervised cycle-consistent adversarial networks, the paired SAR/Opitcal data were utilized as input, and a MSELoss was added to measure the similarity between the generated image and the real image, so we considered S-CycleGAN a cGAN method.

3.1.2. Unsupervised GANs

CycleGAN [22] is a typical unsupervised GAN, and its flowchart is shown in Figure 6. In the absence of truth value data corresponding to generated images, it is impossible for us to use these two data to obtain L1 loss and measure the model effect. Zhu believes that, when there are two Generators (G and F), adversarial training can learn mappings G and F that produce outputs identically distributed as target domains Y and X, respectively. However, the premise is that the direction of the mapping must be restricted. CycleGAN mainly includes two Generators (G and F) and two Discriminators (D${}_X$ and D${}_Y$). As shown in Figure 6a, Generator G is mainly responsible for converting image X to image Y, Generator F is mainly responsible for converting image Y to image X, and the two Discriminators are respectively responsible for judging the generated image as true or false. The whole model operation process can be divided into two directions. The first direction is from image X to image Y, as shown in Figure 6b. Generator G will generate the image Y’ based on the image X, and the Discriminator D${}_Y$ will then determine the true or false of the generated image Y’. This process is consistent with the conditional generative adversarial network. Generator F will then continue to generate the image X’ based on the generated image Y’ and finally calculate the loss function using the image X and generated image X ’. The second direction is from image Y to image X, as shown in Figure 6c. The process is similar to the first direction. In Figure 6b,c, it can be found that the calculation of the cycle-consistency loss function only needs image X, generated image X’, image Y, and generated image Y’ and does not need the paired of image X and image Y, so the whole network is an unsupervised model. The formula of cycle consistency loss is shown in Equation (4). The full objective is shown in Equation (5):

$${\mathcal{L}}_{cyc}(G,F)={\mathbb{E}}_{x\sim p_{data}\left(x\right)}{[\parallel F\left(G\left(x\right)\right)-x\parallel }_1]+{\mathbb{E}}_{y\sim p_{data}\left(y\right)}{[\parallel G\left(F\left(y\right)\right)-y\parallel }_1]$$

(4)

$$\mathcal{L}(G,F,D_X,D_Y)={\mathcal{L}}_{GAN}(G,D_Y,X,Y)+{\mathcal{L}}_{GAN}(F,D_X,Y,X)+\lambda {\mathcal{L}}_{cyc}(G,F)$$

(5)

where D${}_X$ and D${}_Y$ are two adversarial discriminators, D${}_X$ aims to distinguish between images X and generated images F(y), D${}_Y$ aims to distinguish between images Y and generated images G(x), and $\lambda$ controls the relative importance of the two objectives. The final aim is shown in Equation (6):

$$G^{*},F^{*}=arg\underset{G,F}{min}\underset{D_X,D_Y}{max}\mathcal{L}(G,F,D_X,D_Y).$$

(6)

In [28], the author tried to use CycleGAN for SAR-to-optical translation (TerraSAR-X satellite to ALOS PRISM satellite and Sentinel-1 to Sentinel-2). All images were from 13 European cities spread over seven counties. The results showed that CycleGAN is able to take advantage of the large datasets acquired by satellite and has potential for SAR-to-optical image translation. Compared to urban areas, CycleGAN is best used for rural or semiurban areas, as it can recover missing information in mountain or water areas well. It is difficult to isolate walls and buildings, which can be blurred or merged into larger elements.

3.1.3. CNNs

The biggest difference between CNNs and the two methods mentioned above lies in whether there is an adversarial network. Of course, the main network structure is also different. To some extent, the adversarial network is actually a way of extracting high-dimensional features, and it ultimately acts upon the network model training through a loss function [54]. However, the ordinary loss function is a fixed formula that will not change, while the adversarial neural network can constantly change the model formula through continuous learning. That is to say, as long as the structure of the main network is designed reasonably, and the design of the loss function can measure the training effect of the model, it is unnecessary to use an adversarial neural network, as was done in [54,65].

In [65], the author used a residual neural network as the main network structure to recover the missing information of Sentinel-2 images by fusing Sentinel-1 images and all bands of the cloud-contaminated Sentinel-2 images. Meanwhile, the author proposed a new loss function, Cloud-Adaptive Regularized Loss (${\mathcal{L}}_{CARL}$), to estimate the recovery accuracy of the original cloud area and the original cloud-free area in the generated image. The formulation is shown in Equation (7):

$${\mathcal{L}}_{CARL}=\frac{{\parallel \mathbf{CSM}\odot (\mathbf{P}-\mathbf{T})+(\mathbf{1}-\mathbf{CSM})\odot (\mathbf{P}-\mathbf{I})\parallel }_1}{N_{tot}}+\lambda \frac{{\parallel \mathbf{P}-\mathbf{T}\parallel }_1}{N_{tot}}$$

(7)

where P, T and I denote the predicted, target and input optical images, respectively. The CSM (cloud-shadow mask) has the same spatial property of the optical images. The pixel value 1 represents the cloud or cloud shadow and pixel value 0 for the cloud-free area. 1 is a matrix with values of 1 and has the same spatial dimensions as the optical images. $N_{tot}$ is the total number of pixels in all bands of the optical images. Compared with the commonly used loss function to calculate the effect of cloud removal, this loss function takes into account two different regional attributes of the image: cloud and cloudless.

3.1.4. Hybrid CNNs

Hybrid CNNs can mainly be divided into two stages, and a typical flowchart is shown in Figure 7. The first stage is the cloudless image simulation stage, and the second stage is to fuse the output of the first stage with the data contaminated by the cloud to obtain the final cloud-free optical remote sensing image data. In this process, adversarial neural network can be used at one stage [46] or both stages [35] or none at all. The advantage of the hybrid network is that it can not only save the cloud-free area information of the original cloud-contaminated data as much as possible but also use the multisource data to recover the information of the cloud area.

In order to train the network model more specifically, one author used two new loss functions [46]. The first is the perceptual loss function, which is used to judge the accuracy of the model output from the high-dimensional feature level. The second is the local loss function, which is used to dynamically judge the data recovery accuracy of the cloud-polluted area. For the perceptual loss function, the author tried to extract feature maps of the fusion result and the ground truth from the 8th layer of VGG16 [73]. The formula is shown as Equation (8):

$${\mathcal{L}}_{perc}={\parallel vg{g}_8\left(F\right)-vg{g}_8\left(GT\right)\parallel }_1$$

(8)

where F stands for the fusion result, and GT stands for the ground truth data. The formula of the local loss function is shown as Equation (9):

$${\mathcal{L}}_{local}={\parallel M·F-M·GT\parallel }_1$$

(9)

where M is the cloud mask, the author adapted MSCFF (multiscale convolutional feature fusion) [74], which is based on deep learning, to dynamically obtain the cloud mask of each training image of each batch.

3.1.5. Other Methods

In addition to the above four types of methods, there are also studies on SAR-to-optical or optical image cloud removal, which are classified as other methods. For example, Sparse Dictionary Learning was used to remove cloud pollution in optical data by using SAR data [62,63]. Eckardt et al. [47] combined the Closest Spectral Fit (CSF) [75] algorithm with multispectral images and multifrequency SAR data to remove the thick cloud of optical images. Fu et al. [34] tried to build a new concept of remote sensing image translation based on image understanding, target transform and representation. Zhang et al. [68] adapted the intensity–hue–saturation and wavelet transform integrated fusion for SAR-to-optical image translation.

3.2. The Categories of the Input

In addition to the method and the loss function used, we consider the input data to be important in the modeling. Therefore, we classified the above-mentioned literature according to the type of input data combination, as shown in

Table 3. Different inputs of the existing cloud removal methods of optical remote sensing imagery using SAR. (¹ Names of these methods were extracted from the literature. If the methods were not named in the literature, the words describing the methods’ major characteristics were used to name them here. cGANs: conditional generative adversarial networks. CNNs: convolutional neural networks.)

Input	Methods ${}^1$ and References
monotemporal SAR	modified cGAN-based SAR-to-optical translation method [60];
	feature-guided method [59]; DGL-INJSC [63];
	CycleGAN [28]; S-CycleGAN [17]; IHSW [68]
monotemporal SAR & corrupted Optical	Sparse Representation [62]; DSen2-CR [65];
	SFGAN [46]; CMD [54]; PLFM [56];
	cloud removal based on multifrequency SAR [47];
	SEN12MS-CR-TS [55]; G-FAN [48]
multi-temporal SAR & Optical	MTcGAN [67];
	Synthesis of Multispectral Optical Images method [61]

. We hope that readers will be able to see the advantages and disadvantages of different input data in this subsection. Moreover, we found several papers [17,55,57,76] on the dataset for the fusion of SAR and optical image data, which are widely cited and used by other papers. In these papers, Sentinel-1 is the most popular source of SAR data. Ground-range-detected (GRD) products acquired in the most frequently available interferometric wide swath (IW) mode are used. They contain VV and VH polarized channels, and the resolution is 10 m.

3.2.1. Mono-Temporal SAR

The first type of input data in the test stage is monotemporal SAR data, which is the commonly used input data in the traditional SAR-to-optical image translation field [39,60]. In the training stage, we often need to provide SAR data and optical remote sensing data in pairs on a single time phase [31]. SAR data is mainly used to provide feature information, while optical remote sensing data is mainly used to calculate the loss function together with the model output. Moreover, although CycleGAN also needs to input SAR and optical remote sensing data, it does not need to have the same spatiotemporal properties [28]. The advantages of using monotemporal SAR data to generate cloud-free optical remote sensing images are that there are fewer restrictions on the acquisition of input data and that the production of the training dataset is relatively simple. At the same time, this method can also increase the amount of S2 data in the time dimension. However, due to the essential imaging differences between radar data and optical data, the reflection information of some ground objects on these two types of data is different, so the accuracy of using SAR data alone to acquire the optical image on some ground objects cannot be guaranteed [17]. At the same time, this method requires a large number of samples for the model to fully learn the conversion relationship between SAR and optical remote sensing data.

3.2.2. Monotemporal SAR and Corrupted Optical Images

The second type of input data is the combination of monotemporal SAR data and cloud contaminated optical remote sensing data [30]. Unlike the first method, corrupted optical data is added in this method, which helps the model to assist the recovery of the target area according to the feature of the adjacent areas of the cloud area; at the same time, the output results can be more consistent with the real situation [65]. However, since the recovery of the target region (especially in the case of thick clouds) mainly relies on the SAR data, the accuracy of the SAR-to-optical translation process still has a great impact on the results, which is similar to the first method [46].

3.2.3. Multitemporal SAR and Optical Images

The last type is multitemporal SAR and optical image data pairs. In this way, the model learns the conversion relationship between SAR and optical remote sensing data at the reference time and then generates the target optical remote sensing image according to the feature of the SAR data at the target time [67]. The advantage of this method is that the main object distribution of the target image can be roughly obtained from the optical image reference time data, and the reflectance of the target image can then be inferred by using the change of the target time and reference time of the SAR data [33]. However, the problem is that the features of ground objects in reference time and target time cannot change much; otherwise, the accuracy cannot be guaranteed.

4. Evaluation Indicators

How the accuracy of a generated cloud-free image is evaluated is also an important issue for the relevant research. In this section, we counted the existing literature one by one and obtained several evaluation indexes that scholars like to use, as shown in

Table 4. Different input of the existing cloud removal methods of optical remote sensing imagery using SAR.

Indicators	References	Frequency
Structural Similarity Index Measurement (SSIM)	[17,46,48,54,55,56,59,60,62,65,67,68]	12
Peak Signal-to-Noise Ratio (PSNR)	[17,48,54,55,56,59,60,62,65,67]	10
Spectral Angle Mapper (SAM)	[46,47,48,55,56,65,68]	7
Root Mean Square Error (RMSE)	[46,55,56,62,65,68]	6
Correlation Coefficient (CC)	[46,48,56,68]	4
Feature Similarity Index Measurement (FSIM)	[17,59]	2
Universal Image Quality Index (UIQI)	[56,68]	2
Mean Absolute Error (MAE)	[48,65]	2
Mean Square Error (MSE)	[59]	1
Degree of Distortion (DD)	[56]	1

.

The most frequently used indicator is Structural Similarity Index Measurement (SSIM) [77]. In terms of image composition, SSIM defines structural information as independent of brightness and contrast, reflecting the object structure property in the scene, and modeling distortion as a combination of brightness, contrast and structure. The mean value, standard deviation and covariance were used to estimate the luminance, contrast, and structural similarity. The formula is shown as Equation (10).

$$SSIM=\frac{(2{\mu }_y{\mu }_{\widehat{y}}+C_1)(2{\sigma }_{y\widehat{y}}+C_2)}{({\mu }_y^2+{\mu }_{\widehat{y}}^2+C_1)({\sigma }_y^2+{\sigma }_{\widehat{y}}^2+C_2)}$$

(10)

where ${\sigma }_{y\widehat{y}}$ is the covariance between the realistic and simulated values, and C${}_1$ and C${}_2$ are the constants used to enhance the stability of SSIM. SSIM ranges from −1 to 1; the closer the SSIM is to 1, the more accurate the simulated image is.

The second indicator is the Peak Signal-to-Noise Ratio (PSNR) [78], which is a traditional image quality assessment (IQA) index. A higher PSNR generally indicates that the image is of higher quality. The formula can be defined as

$$PSNR=10lg\left(\frac{{255}^2}{MSE}\right)$$

(11)

$$MSE=\frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{N}$$

(12)

where $y_i$ and ${\widehat{y}}_i$ are the realistic and simulated values for the ith pixel, respectively, N is the number of pixels, and $MSE$ is the mean square error.

The third indicator is Spectral Angle Mapper (SAM), which was proposed by Kruse et al. [79] in 1993, which regards the spectrum of each pixel in an image as a high-dimensional vector and measures the similarity between the spectra by calculating the included angle between two vectors. The smaller the included angle, the more similar the two spectra are.

The fourth indicator is the Root Mean Square Error (RMSE) [80], which is used to measure the difference between the actual and simulated values. The formula is defined as

$$RMSE=\sqrt{\frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{N}}$$

(13)

where $y_i$ and ${\widehat{y}}_i$ are the realistic and simulated values for the ith pixel, respectively, and N is the number of pixels. The smaller the RMSE is, the more similar the two images are.

In addition to the above four evaluation indicators, scholars also use the following indicators to evaluate accuracy: Correlation Coefficient (CC) [81], Feature Similarity Index Measurement (FSIM) [82], Universal Image Quality Index (UIQI) [83], Mean Absolute Error (MAE) [80], Mean Square Error (MSE) [84] and Degree of Distortion (DD) [56].

5. Limits and Future Directions

On the basis of this literature review, we analyze the problems of existing methods and propose the following six directions for future research.

5.1. Image Pixel Depth

Remote sensing data present the reflectivity information of surface objects, which are recorded continuously from 0 to 1. Considering that the storage of float data will consume a high amount of storage resources, we usually multiply the reflectivity by 10,000 and retain an integer from 1 to 10,000. The storage format of remote sensing data is usually GeoTIFF, but the image input format used for deep learning is usually PNG format, an 8-bit unsigned integer with a pixel depth of 0–255. Therefore, in order to ingest remote sensing data into the deep learning model smoothly, most scholars adopt the method of saving remote sensing images in PNG format.

However, this approach is similar to a lossy compression, with a loss rate of spectral information reaching 97.44%, and most of the spectral information is lost, which will lead to two problems. First, the amount of information obtained by the model is far lower than the amount of information that the data itself can provide, which is not conducive to the model’s feature extraction of spectral information, and a large amount of information can easily be misclassified, which is not conducive to finding the relationship between the backscattering coefficient and spectral reflectance. This may lead to poor cloud removal effect. Second, the final output results of the model can only be images with pixel values of 0–255. These results are acceptable for simple visual inspection, but cannot be used for subsequent quantitative calculations of the model, because the information loss is too large.

In fact, this problem is not limited to the field of cloud removal studied in this paper, but also exists in other fields that use deep learning to study remote sensing data. We do not recommend choosing to lose the accuracy of remote sensing data itself in order to blindly put the data into a deep learning model, which may cause substantial interference to our research. We suggest that the GDAL library be used to modify the data loading module and data output module of the deep learning model, instead of the usual image format of deep learning, which will enable the deep learning model to directly read and generate data with a higher pixel depth.

5.2. Number of Image Bands

The essential reason for this problem is the same as that in Section 5.1, which is caused by the inconsistency between the professional storage format used for remote sensing data and the data reading and writing format commonly used for deep learning. Optical remote sensing data capture the reflectivity information of different electromagnetic bands: one channel of a panchromatic image, multiple channels of multispectral data and dozens or hundreds of hyperspectral data channels. SAR data also have different combinations according to different polarization modes. The PNG data format commonly used for deep learning can only accommodate three channels, so sometimes optical image data and SAR data are forced to delete the remaining channels or repeat an additional channel, so that the channel number of the image data can reach the three channels required by the PNG data format.

When the remaining channels are deleted, some spectral information will be lost, which causes the features that can be learned by deep learning to be deleted artificially during data processing from the beginning, which is not conducive to the training of cloud removal models. In particular, the impact of clouds is different on different bands. Using the fusion of different bands is an effective way to remove thin clouds from images. However, in order to meet the requirements of PNG format, the remaining bands can only be deleted. We suggest that the GDAL library be used to modify the data loading module and data output module of the deep learning model, instead of the usual image format of deep learning with three channels, which will enable the deep learning model to directly read and generate data with proper channel amounts.

5.3. Global Training Datasets

At present, most research involves the use of a small amount of data in the local area for model training. However, for deep learning models such as generative adversarial networks, many training samples are required for the model to learn knowledge, and sufficient verification samples are also required to evaluate the model effect. However, due to the large repetitive workload and low innovation in dataset production, there are still few publicly available and downloadable datasets for SAR and optical remote sensing image fusion. Therefore, the global regional dataset for optical image cloud removal based on SAR data will be important for research in this field. This dataset needs to have the following characteristics:

• it involves multiple regions of interests in the world,
• it has paired SAR and optical data,
• it is a multi-temporal series,
• it is in GeoTIFF format, and
• the target image which is corrupted by clouds and its cloud free image.

First, different regions of the world must be covered as much as possible in the dataset, so that the model has the opportunity to learn features in different terrains and regions. In this way, the generalization ability of the trained model can be stronger. Second, SAR and optical image data must be fused, so the paired datasets are essential. Third, the model must have a chance to learn the characteristics of the time dimension, which is conducive to further improving the accuracy of the model. Fourth, the feature is to ensure that the dataset has a relatively complete information feature, which was mentioned in the previous two sections. Last, there must be available verification data in the process of cloud removal research, so that the model can be continuously trained and optimized.

5.4. Accuracy Verification for Cloud Regions

At present, the vast majority of research is to evaluate the effect of cloud removal for whole scene data, and the evaluation result is the average result of the whole scene image. However, for cloud removal applications, it is key to recover the areas polluted by clouds or cloud shadows, and the original pixel information in the cloud image to be removed can be used in other nonpolluted areas. Therefore, for cloud removal research, there are certain limitations in judging model accuracy from the whole scene image range, and it is necessary to try to conduct targeted accuracy evaluations in the areas polluted by clouds or cloud shadows.

For example,

Table 5. Evaluation of cloud removal results of the whole scene image.

	Method A	Method B
RMSE	0.0067	0.0094
R${}^2$	0.9596	0.9256
KGE	0.9224	0.9256
SSIM	0.9948	0.9943
PSNR	54.4722	54.1575
SAM	0.0869	0.0940

The bold result is the best among the different metrics.

records the precision comparison of whole scene results of two SAR-based cloud removal methods. Method A has obvious advantages, but when we view the local cloud removal effect, as shown in Figure 8, it is obvious that the cloud removal effect of Method B is better, so it is difficult to evaluate the quality of cloud removal only from the evaluation indicators of the whole scene results. Therefore, we need to extract the area to be cloud-removed separately, and we need to calculate the indicators to evaluate the accuracy of the cloud removal effect, which is more objective.

5.5. Auxiliary Data

The input data of most research are SAR and optical image data, but few studies have paid attention to the value of cloud mask data in this field. At present, many optical data are provided with corresponding cloud mask data when downloading, and there are also many open-source cloud detection model algorithms, making cloud mask data easier to obtain. It is a worthwhile idea to input cloud mask data into the model, which also allows the model to obtain cloud distribution in advance, so as to conduct more targeted cloud removal operations.

Furthermore, we can enrich the types of cloud mask data. In addition to cloud and cloudless, we can further distinguish thin clouds, thick clouds, cloud shadow and other areas and cooperate with the loss function so that the results of the model are more satisfactory. Compared to the ${\mathcal{L}}_{CARL}$ (7), the new loss function is able to measure and feed back more information to improve the accuracy. The equation can be defined as

$${\mathcal{L}}_{NEW}=\left(\begin{array}{c}\hfill \frac{{\parallel \mathbf{TCM}\mathbf{1}\odot (\mathbf{P}-\mathbf{T})\parallel }_1}{N_{tot}}+\lambda 1\frac{{\parallel \mathbf{TCM}\mathbf{2}\odot (\mathbf{P}-\mathbf{T})\parallel }_1}{N_{tot}}+\lambda 2\frac{{\parallel \mathbf{SM}\odot (\mathbf{P}-\mathbf{T})\parallel }_1}{N_{tot}}\\ \hfill +\lambda 3\frac{{\parallel \mathbf{CSM}\odot (\mathbf{P}-\mathbf{T})+(\mathbf{1}-\mathbf{CSM})\odot (\mathbf{P}-\mathbf{I})\parallel }_1}{N_{tot}}+\lambda 4\frac{{\parallel \mathbf{P}-\mathbf{T}\parallel }_1}{N_{tot}}\end{array}\right)$$

(14)

where P, T and I denote the predicted, target and input optical images, respectively. The TCM1 (thick cloud mask) has the same spatial property of the optical images. The pixel value 1 represents thick cloud, and the pixel value 0 represents other areas. The TCM2 (thin cloud mask) has the same spatial property of the optical images. The pixel value 1 represents thin cloud, and the pixel value 0 represents other areas. The SM (shadow mask) has the same spatial property of the optical images. The pixel value 1 represents shadow, and the pixel value 0 represents other areas. The CSM (cloud-shadow mask) has the same spatial property of the optical images. The pixel value 1 represents cloud or cloud shadow, and the pixel value 0 represents cloud-free areas. 1 is a matrix with values of 1 and has the same spatial dimensions as the optical images. $N_{tot}$ is the total number of pixels in all bands of the optical images.

5.6. Loss Functions

The current studies use a variety of precision indicators to verify the results from multiple dimensions, as shown in Table 4. However, this can only be used for the final quantitative evaluation of the results and cannot greatly help the optimization of the model. At the same time, most studies use relatively simple L1 loss or L2 loss for the entire space to calculate the loss function. Next, we can try to put these multidimensional indicators, such as SAM, into the model optimization stage (i.e., the loss function calculation) to see whether they can help the model to optimize training from different dimensions.

It is worth noting that, due to the matrix calculation used in the loss function calculation, we need to modify these commonly used precision index calculation formulas so that they conform to the matrix calculation method, which will involve some mathematical knowledge and programming knowledge. Of course, in theory, we can also directly put the precision index calculation formula into the loss function without modification for calculation, but this will greatly increase the training time, and the initial estimation time consumption may be 1000 times higher, so it is not recommended.

6. Conclusions

Through literature retrieval and analysis, in this paper, we find that the research of optical image cloud removal based on SAR data is a valuable research direction. Compared to traditional methods using optical remote sensing data alone to remove clouds, the research on cloud removal integrating SAR data has risen in recent years. After the emergence of deep learning, more scholars began studying cloud removal by SAR and optical data fusion. Therefore, there is a lack of review papers on the research of optical image cloud removal using SAR data. We hope that, through this paper, more relevant research will be carried out to understand the development of this study field and to communicate the advances with more scholars.

In this paper, we present the main contributing journals, keywords, fundamental literature and other research in the field of optical image cloud removal using SAR data. It will be helpful for scholars to search and study the relevant literature in this field. Scholars can read nearly 54% of the literature in this field from two sources: Remote Sensing and IEEE Transactions on Geoscience and Remote Sensing.

In this paper, we summarize the relevant literature from two dimensions: research methods and data input. We classify the research methods into five categories according to GANs: conditional generative adversarial networks (cGANs), unsupervised GANs, convolutional neural networks (CNNs, not GANs), hybrid CNNs and other methods, as Table 2 shows. We provide the general principles and loss functions of these methods to help scholars understand them. At the same time, we describe the input data used by these methods and summarize three types of data input, as shown in Table 3, to help scholars understand the data and provide advance preparation for studying these methods.

Moreover, this paper documents the accuracy verification indicators used in the current mainstream literature, which can help subsequent scholars make appropriate choices. The results show that the most popular indicators are SSIM, PSNR, SAM and RMSE. We believe that these four indicators are good choices for relevant scholars to quantitatively verify the accuracy of their models.

Finally, this paper discusses the key points of the future development of this field in terms of six aspects: image pixel depth, the number of image bands, global training datasets, accuracy verification for cloud regions, auxiliary data and loss functions. We discuss the current problems in these six areas and provide our solutions. We hope these problems and solutions can inspire scholars and promote the development of optical image cloud removal using SAR data.