Deep Internal Learning for Inpainting of Cloud-Affected Regions in Satellite Imagery

Abstract

Cloud cover remains a significant limitation to a broad range of applications relying on optical remote sensing imagery, including crop identification/yield prediction, climate monitoring, and land cover classification. A common approach to cloud removal treats the problem as an inpainting task and imputes optical data in the cloud-affected regions employing either mosaicing historical data or making use of sensing modalities not impacted by cloud obstructions, such as SAR. Recently, deep learning approaches have been explored in these applications; however, the majority of reported solutions rely on external learning practices, i.e., models trained on fixed datasets. Although these models perform well within the context of a particular dataset, a significant risk of spatial and temporal overfitting exists when applied in different locations or at different times. Here, cloud removal was implemented within an internal learning regime through an inpainting technique based on the deep image prior. The approach was evaluated on both a synthetic dataset with an exact ground truth, as well as real samples. The ability to inpaint the cloud-affected regions for varying weather conditions across a whole year with no prior training was demonstrated, and the performance of the approach was characterised.

1. Introduction

Cloud occlusion reduces the temporal availability and, in turn, the usability of optical satellite data, imposing a significant obstacle to applications where frequent sampling of the ground surface is necessary. In the field of precision agriculture, for tasks such as monitoring crop growth [1], crop classification [2], or crop yield prediction [3], gaps in data result in challenges related to the development of accurate models, variability in prediction performance, and the need for the use of data imputation. Open data sources, such as those generated by the EU Copernicus Sentinel missions [4], have been instrumental to the accelerated development of applications, enabling easy and wide access to large sets of high-resolution data at relatively short revisit periods. However, the fundamental limitations imposed by cloud obstruction remain, which can be mitigated either through the development of new models that do not rely exclusively on optical data or by techniques that reconstruct optical imagery for cloud-affected regions.

Reported approaches to cloud removal from optical satellite images can be segmented into a number of strands. Most early developments, before the growth of deep-learning-based approaches, used interpolation-based techniques [5,6,7], filtering [8,9], or the creation of image mosaics or composites [10,11,12,13,14,15]. A number of approaches employing machine learning were reported [16,17,18,19] prior to a notable increase in research to determine the potential of deep neural networks to yield advances in performance [20,21,22,23,24,25,26,27,28,29,30]. Although a subset of the proposed methods attempt to remove clouds using a single image [8,20,21,31,32,33], more common approaches either rely on multi-temporal data to inform the reconstruction [5,7,9,10,11,12,13,14,15,16,17,18,19,25,28,29,34,35,36,37,38,39] or use sensing modalities not impacted by clouds [6,22,23,25,26,27,30,40].

Most deep-learning-based cloud removal methods depend on a form of external training [20,21,22,23,24,25,26,27,28,29,30,41,42], relying on a pre-training phase, during which the network is optimised to produce a desired target output for a fixed dataset. As a consequence, the performance and generalisation of the models depend heavily on the level of representation the fixed dataset provides of the conditions encountered in deployment. The training dataset must be sufficiently diverse to capture the complexity of the real-world application so as to minimise the expectation of loss within the operational scenario. Conversely, the consequence of striving to achieve increased diversity is a trade in performance on individual samples in favour of average accuracy.

Several deep learning approaches [27,30] claim that larger datasets result in neural network models able to learn general and adaptable transformations. However, this assertion is at odds with the fact that in practice, the majority of datasets contain bias and a considerable amount of features that are highly predictive within the dataset, but do not generalise to new samples [43]. Thus, methods that do not rely on a process of optimisation on a fixed number of samples [5,6,7,8,9,10,11,12,13,14,15], but use engineered priors that guide the synthesis process are more attractive. Recently, an approach [29] that combines external learning with prior-based synthesis has been proposed where spatio-temporal representations are reconstructed with a Deep Image Prior (DIP), while an externally trained network enforces the context consistency of the synthesised images. It must be noted, however, that the external network may introduce a bias in a similar fashion to other external learning approaches.

Here, an alternative technique embracing a pure internal learning paradigm that relies on the optimisation of the network on a single sample of interest and thus obviates the need for pre-training on an external dataset is detailed and evaluated. The approach is restricted to model tasks where a significant part of the information required for reconstruction is not obscured in the source sample. Within the satellite imaging context, data are available from multi-temporal and multi-source streams, enabling the application of knowledge arising from solutions to a number of well-known computer vision inverse problems [44,45,46,47] such as image completion, super-resolution, and denoising. To date, only one other reported technique has adopted the proposed methodology [48], employing a sequence-to-sequence internal learning scheme to translate sequences of Synthetic Aperture Radar (SAR) satellite images to sequences of multi-spectral images using a 3D U-Net architecture. However, the approach of [48] is fundamentally different from deep image prior work [44] and the framework proposed here as it feeds the informing SAR signal directly to the reconstruction network instead of employing noise tensor input and reconstructing the informing signal along with the target optical representation. It is not demonstrated why this should be beneficial in the internal learning context. The sequence-to-sequence model does, however, limit the flexibility of the solution as the data are constrained to the same number of SAR and optical samples. Furthermore, the method was evaluated only on three sample sequences. During the review stage of this manuscript, an independent study on the application of the deep image prior has been published [49]. The paper reported on multi-temporal reconstruction and relied on a recent cloud-free reference, evaluated on 10 Landsat-8 and 10 Sentinel-2 patches. Here, the use of the multi-temporal historical average signal or multi-source SAR information is proposed, which avoids issues with overlapping cloud regions. Furthermore, this work performed a sensitivity analysis of the cloud mask size and the synthesis quality.

In this paper, an end-to-end workflow for the removal of cloud cover in Sentinel-2 optical satellite images is presented for precision agriculture applications, where flexibility and adaptation to a target fixed region of interest is a key requirement. Several methods of informing the reconstruction process were tested, including both multi-source and multi-temporal approaches; for the former case, SAR images from Sentinel-1 were used as an informing input.

The proposed solution is an application of the DIP methodology [44], a prominent internal deep learning approach for image synthesis. Furthermore, the contribution of this work involved the analysis and comparison of several processing approaches to the problem, depending on what data sources were used. Evidence is provided that the proposed internal techniques are capable of producing high-quality reconstructions, which was tested on a synthetic dataset with an accessible ground truth, as well as on real images. Finally, unlike other deep learning methods for cloud removal, the solution is readily adaptable with no additional training, be it using top-of-atmosphere or bottom-of-atmosphere hyperspectral or, indeed, a number of other sensor data and irrespective of the preprocessing approach employed.

The Materials and Methods Section (Section 2) provides details on the data preprocessing routine and the creation of the synthetic dataset, followed by a description of the proposed architectures (Section 2.2). Section 3 presents both quantitative results (Section 3.1), where the performance metrics are computed for the entire dataset, as well as qualitative results (Section 3.2). Conclusions are drawn in Section 4.

2. Materials and Methods

In this section, the dataset used for the evaluation experiments is described, followed by a segment detailing the proposed model architectures.

2.1. Supporting Data

The focus was the removal of clouds from optical Sentinel-2 images in the context of crop monitoring. Hence, the dataset created for the development was based on Sentinel-2 data from a Level-2A product, offering bottom-of-atmosphere reflectance measurements, proven to be most appropriate in vegetation monitoring applications as opposed to top-of-atmosphere data. Even so, the internal synthesis approach presented is adaptable as it is based on the content of the target image only and, as a consequence, can be directly applied to other image types, such as the more commonly used Level-1C Sentinel-2 images. In fact, results (Section 3.2) are also reported on the performance of the method with Level-1C samples for comparison with an external model. Finally, the resultant Sentinel-2 images were paired with Sentinel-1 equivalents based on the closest temporal proximity.

The model requires a cloud mask (or cloud and shadow mask), and although a significant number of cloud detection algorithms have been reported to date [50], the goal was to evaluate the performance of the proposed method on the inpainting task alone. To avoid the propagation of errors from the cloud detection models, the inpainting regions were obtained by removing portions of images under clear sky, providing an accessible and precise ground truth.

The dataset comprises two-year temporal coverage measurements for two different regions, one in Scotland, the other in India, each containing approximately 200 samples (data available at https://doi.org/10.5281/zenodo.5897695 (accessed on 9 February 2022)). The extent of the data allows the valid test of the approach over a substantial period of time subject to varying seasonal conditions and across two disparate geographical locations. Every image is effectively a test sample, obviating the need for splitting the data into training and test subsets, since an internal learning approach was employed. All available images for the years 2019 and 2020 were manually assessed to extract a subset of clear sky images to ensure that the samples were clouds-free. All clear sky images from 2019 were used as a source of prior information, and the images from year 2020 were used for the evaluation. Cloud regions were simulated by obtaining plausible cloud masks from Sentinel-2 data of the Scotland region with a coverage area between 10% and 50%, resulting in 17 realistic mask shapes. The combination of clear sky images with each cloud mask yielded the evaluation dataset. The resulting sample counts for both regions are shown in

Table 1. Resulting sample counts for each dataset. The bottom row contains the total count for 17 cloud masks.

Dataset	Scotland	India
Informing Samples (2019)	18	34
Inference Samples (2020)	20	30
Total Task Samples	340	510

.

The raw data are acquired and preprocessed using the analysis-ready data framework described in [51], as shown in Figure 1. The Sentinel-1 SAR data were radiometrically calibrated, speckle filtered, and terrain corrected. The Sentinel-2 multi-spectral bands and cloud masks (opaque cloud and cirrus) from the Level-2A products were re-sampled to a 10 m spatial resolution. Sentinel-1 and Sentinel-2 tiles were then co-located, yielding image pairs of 256 × 256 px. The output pairs were then treated in a manner similar to [27]. Sentinel-1 images were (1) converted to a decibel scale, (2) clipped (VV range: [−25.0, 0], VH range: [−32.5, 0]), and shifted to [0, 2]. Sentinel-2 images were clipped between [0, 10${}^4$] for all optical channels and divided by 2000.

2.2. Architecture

The internal cloud removal architectures proposed here rely on the techniques proposed in [44], broadly referred to as Deep Image Prior (DIP). The key premise of the approach is that signals, especially natural images, can be encoded in the weights of a deep convolutional neural network by overfitting the model to generate the signal of interest for a given constant input. The network weights tend to converge to solutions of lower entropy [44], where all textures in the image are derived from a relatively narrow distribution, rendering the DIP method applicable to several image inverse problems, such as image inpainting, denoising, or super-resolution. DIP enables a significant amount of prior knowledge on natural images to be introduced through the use of a convolutional architecture.

The applications demonstrated in the original DIP work operate on a single image; here, the method was extended to cloud removal by using additional information, viz. involving several images. The additional images can be obtained from a different sensor (multi-source synthesis) or from different points in time (multi-temporal synthesis). The results demonstrated that the DIP approach can be effectively applied to both multi-temporal and multi-source satellite images through the stacking of all frames to create an image representation with more channels. Furthermore, good-quality images were produced even when a disparity between domains occurred, as is the case for optical and SAR sensor data.

A number of DIP-based reconstruction methods have been proposed, depicted in Figure 2. The most direct approach applies the DIP network to the masked hyperspectral image alone, as shown in Figure 2a, in alignment with the default motivation of the technique, where the inpainting is based only on the textures present in the non-masked region, i.e., not obscured by clouds. Two approaches can be used to incorporate an additional data source: either through the optimisation of (i) a residual target output (Figure 2b,c) or (ii) a stack of the target image with the supporting information appended as additional channels (Figure 2d–f). For the residual modes, the network is tasked with computing the difference between either a sample from a different source (to achieve the same number of channels (3) necessary for the multi-source residual, an additional channel containing the VV and VH mean was appended to the SAR representation) (Figure 2b) or a sample from a different time (Figure 2c). The residual component image that the core DIP network was optimised to produce can be expected to exhibit a similarly narrow texture distribution, just as the reconstructed Sentinel-2 image. For the stacked modes (Figure 2d–f), the supporting information could be multi-source (d), multi-temporal (e), or a combination thereof (f). For these cases, the mask was only applied to the cloudy query image so the supporting data from the synthesised region could be backpropagated.

Multi-temporal synthesis requires additional temporal samples to be used at the inference time. This additional source can be static (the same reference used for a number of different predictions) or dynamic, where an appropriate reference is chosen specifically for the synthesised sample (for example, the last clear sky image taken before the cloudy sample). The conducted experiments involved the former, to eliminate the deviation in the reference and allow a consistent architectural assessment. In order to create a static reference, all clear sky images from the year 2019 were averaged. The main purpose of taking a mean over the entire year comprising multiple seasons and weather conditions was to extract the static structural component of a given region. Moreover, taking the mean also had an additional denoising effect, especially for SAR images. The resulting mean representations of the informing samples are shown in Figure 3.

Cloud removal for Sentinel-2 data can be achieved by utilising Sentinel-1 SAR data as the alternative sensing source. The SAR component was incorporated as a 2-channel image (VV, VH) to enable a valid comparison with other cloud removal techniques [27]. Several approaches are possible to generate temporal support data. A source of dynamic time-dependent support can be obtained from the last Sentinel-2 observation; however, there is no guarantee that this image will not contain significant cloud cover. Thus, the last cloud-free (or nearly cloud-free) observation could be used, but that may lead to temporal inconsistencies between the inferred samples and their support counterparts, compromising performance stability. An alternative is to use a mean support sample derived as the average signal from the cloud-free images captured over the preceding year of the inference period. The average Sentinel-1 and Sentinel-2 data will contain an approximation of the static component shared by all images over the year. This way, each inferred sample was biased by the same support signal. Examples of the mean support data are shown in Figure 3.

All the experiments conducted were obtained using an Adam optimiser with exponential decay rates for the first and second moment coefficients ${\beta }_1$ = 0.9, ${\beta }_2$ = 0.999, a stabilising term $ϵ$ = 10${}^{-8}$, and a learning rate of 2 × 10${}^{-2}$. The networks were trained for 4000 steps with a regularisation noise of 10${}^{-1}$. The code for the proposed method is available at https://github.com/cidcom/satellite-cloud-removal-dip (accessed on 9 February 2022). The result of an ablation study is shown in Figure 4, demonstrating the visual quality of the synthesised cloud inpainting for all synthesis modes presented in Figure 2. Two performance metrics were employed for evaluation: Structural Similarity Index (SSIM) [52] as an indicator of structural coherence and Root-Mean-Squared Error (RMSE) as a measure of reflectance error. Tests were carried out by repeating each mode 4 times, and the displayed examples corresponded to the run with the median SSIM, applied to the inpainted region only, unless otherwise stated. The basic scenario with no extra informing data produced results that matched the colour tones of the image, but lacked fine detail. The residual multi-source approach produced poor-quality results and may require additional mechanisms to achieve improved synthesis. The multi-temporal residual mode produced a reasonable quality result, but was constrained to the same number of synthesised channels as the reference representation, unlike the stacked modes, where the output can be adjusted to yield any number of frames and channels. The stacking approaches are presented in the bottom row, leading to a considerable improvement when employing multi-temporal data.

The metrics between two images $I_A$ and $I_B$ (which correspond to a test image and appropriate ground truth image) were computed using the following formulas:

$$\mathrm{RMSE}(I_A,I_B)=\sqrt{\frac{{(I_A-I_B)}^2}{H\times W}}$$

(1)

where H is the height and W is the width of both $I_A$ and $I_B$.

$$\mathrm{SSIM}(x,y)=\frac{1}{|\Pi (I_A,I_B\left)\right|}\sum _{x,y}^{\Pi (I_A,I_B)}\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}$$

(2)

where $x,y$ correspond to individual patches of size 11 × 11 in $I_A$ and $I_B$ (with Gaussian window as described in [52]), respectively, where $\Pi (I_A,I_B)$ is the set of all collocated patches and $|\Pi (I_A,I_B\left)\right|$ the total number of elements in that set. The variables with $\mu$ correspond to the mean value, $\sigma$ to the variance, and ${\sigma }_{xy}$ to the covariance. Standard constant values of $c_1$ and $c_2$ were used for computational stability, as proposed in [52].

3. Results

The quantitative results relied on the dataset described in Section 2, where clear sky samples were reconstructed from artificially masked images. The qualitative results are presented to demonstrate the visual appearance of the synthesised regions.

3.1. Quantitative Results

All synthesis modes were applied to the evaluation dataset containing the two regions of Scotland and India, with performance metrics computed for the image as a whole and for the inpainted region only; the results of the inference are shown in

Table 2. MS—Multi-Source (Sentinel 1), MT—Multi-Temporal, -R—Residual Variants. Optimal performance maximizes SSIM and minimizes RMSE. This occurs for the modes using temporal support data, with the MS-MT mode surpassing all others for both datasets and both for the inpainting region and the whole image.

Dataset			Direct Modes			Stacked Modes
			Basic	MS-R	MT-R	MS	MT	MS-MT
Scotland	Whole	SSIM ↑	0.84	0.73	0.86	0.84	0.88	0.88
		RMSE ↓	0.09	0.13	0.08	0.08	0.07	0.07
	Inpainting	SSIM ↑	0.58	0.33	0.66	0.58	0.73	0.73
		RMSE ↓	0.15	0.23	0.15	0.14	0.12	0.11
India	Whole	SSIM ↑	0.84	0.74	0.84	0.84	0.87	0.87
		RMSE ↓	0.07	0.12	0.08	0.07	0.06	0.05
	Inpainting	SSIM ↑	0.53	0.32	0.57	0.53	0.64	0.66
		RMSE ↓	0.13	0.21	0.15	0.12	0.11	0.09

. The stacked MT and MS-MT variants that utilised multi-temporal data consistently outperformed the other variants for both datasets and for both the full image and inpainted regions (A metric applied to the inpainted region measures the quality of new generated data and, when applied to the whole image, quantifies the distortion introduced by the optimisation process. The distortion can be partially prevented by stitching the uncovered region with the synthesised pixels, but new distortions on the stitch borders may be introduced by the process). This suggests that the inclusion of the multi-temporal component is more beneficial for the considered synthesis framework. Both residual modes exhibited inferior performance compared to the corresponding stacked modes that used equivalent data. Furthermore, the residual modes constrained the number of frames and channels, and thus, the remainder of the evaluation was focused on the stacked synthesis modes.

The evaluation dataset was designed to contain real cloud masks with coverage from a selected range (10–50%), useful for simulating real-world conditions, but does not give an opportunity to validate how the system performance varies depending on the size of an inpainted region. Consequently, another experiment was carried out, where clear sky samples from each of the two datasets were reconstructed for a number of artificial cloud masks with a controlled coverage area. The resulting SSIM and RMSE traces are shown in Figure 5. The inpainting quality appears to be consistent for cloud coverage ratio between 0.1% to 16%, where the inpainted region SSIM is approximately 0.75 for the Scotland and India regions. The SSIM decreased beyond 16% cloud coverage, however, but never fell below 0.6. Naturally, the whole-image SSIM was higher than that of the inpainted region since the whole image contained the noncovered region, which was optimised directly. The visual quality of the reconstruction of selected samples from each dataset for a range of cloud coverage levels is illustrated in Figure 6.

3.2. Qualitative Results

Figure 7 and Figure 8 present a compilation of the results distributed across the full year for the Scottish and Indian regions. It is clearly evident that the synthesis modes adjusted the content of the inpainted region based on the information available in the clear region, and the informing reference acted as a guide for generating the static structure of the image. The Multi-Source (MS) synthesis mode yielded fairly poor structural coherence, indicated by consistently lower SSIM scores, also apparent upon visual inspection. Both MT and MS-MT modes generated images with much higher structural coherence and achieved a comparable level of performance. The main differences between the two modes can be observed in the colours of individual objects in the inpainted region. The MT mode had no recent (i.e., temporally proximate) reference of the inpainted regions and relied only on historical data and on what was available outside of the cloud mask. On the other hand, the MS-MT mode did entail access to temporally proximate SAR data, but the different sensing modality contributed to textural distortions in the reconstruction of the optical data. Similar textural distortions were visible in both MS and MS-MT modes, but were less pronounced in the latter owing to the inclusion of the temporal source. Finally, some samples were more challenging to synthesise. For instance, the sample of 28 December 2020 in Scotland (bottom row of Figure 7) contained a significant amount of snow cover. The image contained a strong bias towards a value of saturated white with low variance, posing a risk of unstable optimisation, which occurred in the MT mode for this particular sample.

The influence of image content on the reconstruction process was accentuated when the two geographical regions were compared. The Scotland region consisted of considerably larger objects, yielding images that were inherently easier to fit as an output of a Convolutional Neural Network (CNN). This was confirmed by the higher performance achieved for the Scotland region in Table 2. The amount of fine structural detail can have a considerable effect on the robustness of each tested synthesis mode. In this case, the information from the SAR sensor, while useful for the large-scale coherence, contained noise and had insufficient resolution, resulting in higher distortions of the inpainted regions, an effect visible in Figure 8, where the MT approach provided more consistent inpaintings than the other two approaches that used SAR data.

The best- and worst-scored samples for both datasets are shown in Figure 9 and Figure 10 for Scotland and India, respectively. Note that for the Scotland region, the snowy samples from 28 December were excluded when selecting extreme results, as the severe saturation of the images naturally yielded a high SSIM for the reconstructions. In Figure 9, the first two rows contain the samples yielding the worst performance, while the bottom two rows contain the best. Interestingly, a degree of overlap between the identified extremes for the individual modes can be identified. For instance, all modes share the same date for the maximum and minimum RMSE scores, as well as for the maximum SSIM score in Figure 10, potentially attributable to a combination of factors such as the specific image content, its relation to the informing reference, and the mask area.

3.3. Application in a Blind Setting

The proposed techniques do not require pre-training and can be readily applied to new settings. To demonstrate this capability, the methods were compared to an existing cloud removal model of DSen2-CR [27]. The pre-trained DSen2-CR uses Level-1C Sentinel-2 images containing a top-of-atmosphere view. A small number of Level-1C top-of-atmosphere cloud-affected images were obtained: three for both Scotland and India. Although the DIP-based methods require cloud masks, it is nevertheless possible to use existing cloud detection algorithms; however, these are likely to produce false negatives (i.e., undetected cloud regions). Leakage of cloud components outside the mask can affect the optimisation process of the proposed synthesis methods by distorting the texture distribution of the image. For that reason, the six images used in the evaluation were manually annotated to ensure no leakage.

Figure 11 and Figure 12 contain a comparison of the three considered synthesis modes (multi-source, multi-temporal, and MS-MT) to an externally trained model of DSen2-CR for Scotland and India, respectively. Generally, DSen2-CR generated clear-sky regions that were very close to the S2 input; yet, dark artefacts were evident in the clouded regions that poorly matched the colour distribution. The MS mode using equivalent input data (with the addition of a cloud mask) did not produce the artefacts in the clouded regions. The modes MT and MS-MT provided a more detailed structure in the synthesised output, but relied on additional information.

4. Conclusions

The key factor differentiating the presented internal learning approach to cloud region inpainting in satellite imagery from solutions relying on external learning is that it permits the development of a flexible model not biased by the dataset. Elimination of the cost and time of pre-training, as well as the sourcing, preparation, and curation of the training dataset are an attractive set of advantages. Furthermore, inference takes in the order of two minutes on a consumer-grade GPU (24GB NVIDIA Titan RTX), acceptable for almost all agriculture applications considering the significantly higher delays already present in satellite processing pipelines. For scenarios where processing of larger areas is required, this pipeline can be fully parallelised on a high-performance computing platform to yield a comparable processing time, assuming that an automatic cloud detection module is employed (more details on the processing time are included in Appendix A). The key advantage is that the proposed approach is modality agnostic and can be readily applied to different signal types with no requirement for a new dataset or additional training, particularly beneficial for settings where labelled data are not available.

The solutions presented adopted several approaches to inform the synthesis process. The results showed that the DIP-based methods handle stacked data of the same modality better than multi-source composites. Consequently, historical optical data yielded higher-quality synthesised samples than the two-band SAR reference, potentially attributable to the distortions present in the employed SAR representation. Potential solutions to mitigate this effect are either to devise alternative architectures or improved SAR representation or preprocessing. The advantage of a multi-temporal reference, observed in the context of the presented developments, aligns with the early cloud removal solutions that also employ historical data of the same modality. In this work, the benefit of the temporal reference was maintained even when the time difference between the synthesised image and the reference was as large as 1 y.

The proposed synthesis modes were evaluated both quantitatively and qualitatively for two geographic locations of Scotland and India on data obtained throughout the year 2020. The multi-temporal and multi-source (MS-MT) mode offers the highest performance of the three evaluated modes with an average SSIM greater than 0.87 and 0.64 for the whole image and inpainting regions, respectively. The Multi-Temporal only mode (MT) offered performance on par with MS-MT. The MS performed significantly worse than the temporal approaches. Furthermore, the MS-MT mode was tested for varying degrees of cloud coverage, exhibiting a stable performance even when more than 16% of pixels were inpainted. All three proposed internal synthesis modes were qualitatively compared to an externally trained network to demonstrate the synthesis quality.

The techniques presented all rely on the application of high-quality cloud masks and thus on a robust cloud detection scheme. Furthermore, the results indicated that the synthesised images can vary depending on the run. While the characteristics of the inpainting task impose a wide range of feasible solutions, further work is required to reduce the distortion and variability of the synthesised outputs.