# Sequence Image Datasets Construction via Deep Convolution Networks

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- We are one of the first attempts to use the advantage of the different convolution networks to conduct spectral transformation for filling in missing areas of sequence images and produce full remote-sensing sequence images.
- We combine the temporal and spatial neighborhood of sequence images to consider the construction of scene-based remote-sensing sequence images. It does not rely on other high-temporal-resolution remote-sensing data, and only construct datasets based on the sequence itself. This provides a new idea for the construction of remote-sensing datasets.

## 2. Materials and Methods

#### 2.1. Experimental Datasets

#### 2.2. Architecture of Networks

#### 2.2.1. AdaFG Network

#### 2.2.2. CLSTM Network

#### 2.2.3. CyGAN Network

#### 2.3. Evaluation Indicator

_{i}is the probability of a pixel with a gray value of i in image, and L is the total number of gray levels (L = 255). According to Shannon’s information theory [33], there is the most information when there is maximum entropy.

## 3. Results and Analysis

#### 3.1. Experimental Strategy

_{1}and t

_{2}represent the month of training image pairs acquired, and t

_{3}represents the month of the reference image acquired. ${f}_{\left[{I}_{{t}_{1}},{I}_{{t}_{2}},{I}_{{t}_{3}}\right]}\left({I}_{{t}_{1}}^{\prime},{I}_{{t}_{2}}^{\prime}\right)$ represents output image with mapping model ${f}_{\left[{I}_{{t}_{1}},{I}_{{t}_{2}},{I}_{{t}_{3}}\right]}$ and input scenes ${I}_{{t}_{1}}^{\prime}$ and ${I}_{{t}_{2}}^{\prime}$. In CLSTM network, the mapping model is expressed as ${f}_{\left[{I}_{{t}_{1}},\dots ,{I}_{{t}_{n}},{I}_{{t}_{n+1}}\right]}$, where [${I}_{{t}_{1}},\dots ,{I}_{{t}_{n}},{I}_{{t}_{n+1}}$] represents training image multi-group, ${I}_{{t}_{1}},\dots ,{I}_{{t}_{n}}$ represent the training images, ${I}_{{t}_{n+1}}$ represents the reference image, t

_{1},…,t

_{n}represent the month of training images acquired, and t

_{n+1}represents the month of the reference image acquired. ${f}_{\left[{I}_{{t}_{1}},\dots ,{I}_{{t}_{n}},{I}_{{t}_{n+1}}\right]}\left({I}_{{t}_{1}}^{\prime},\dots ,{I}_{{t}_{n}}^{\prime}\right)$ represents output image with mapping model ${f}_{\left[{I}_{{t}_{1}},\dots ,{I}_{{t}_{n}},{I}_{{t}_{n+1}}\right]}$ and input scenes ${I}_{{t}_{1}}^{\prime},\dots ,{I}_{{t}_{n}}^{\prime}$. In CyGAN network, the output image is expressed as ${f}_{\left[{I}_{{t}_{1}},{I}_{{t}_{2}}\right]}\left({I}_{{t}_{1}}^{\prime}\right)$, where ${f}_{\left[{I}_{{t}_{1}},{I}_{{t}_{2}}\right]}$ represents a mapping model trained by training image pair [${I}_{{t}_{1}},{I}_{{t}_{2}}$] with reference image ${I}_{{t}_{2}}$. ${I}_{{t}_{1}}^{\prime}$ in the brackets represents the input used to generate the output image.

#### 3.2. Experimental Details

**Hyper-parameters selection:**We used a separate convolution kernel of size 11 in the AdaFG network, used stacks of 3 × 3 CLSTM network layer, and set ${\mu}_{1}=$350, ${\mu}_{2}=$1/32 in CyGAN network. Meanwhile, we used the mean square error (MSE) as the loss function. The optimizer used in the training was Adamax with ${\beta}_{1}$= 0.9, ${\beta}_{2}$= 0.99, and a learning rate of 0.001. Compared to other network optimizers, Adamax could achieve better convergence of the model [34].

**Time Complexity:**We used the python machine learning library to execute the deep convolution networks. To improve computational efficiency, we organized our layer in computer unified device architecture (CUDA). Our deep convolution networks were able to generate a 256 × 256 image block in 4 s. Obtaining the overall scene image (image size 3072 × 5632) took about 18 min under the acceleration of the graphics processing unit (GPU) [35].

#### 3.3. Experimental Results

#### 3.3.1. Generalization of Networks

#### 3.3.2. Single Network Datasets

#### 3.3.3. Multiple Network Datasets

#### 3.3.4. Comparisons Between Single and Multiple Networks Datasets

## 4. Discussions

#### 4.1. Separable Convolution Kernel Sizes

#### 4.2. Stacked CLSTM Network Layers

#### 4.3. Proportional Hyper-Parameters

## 5. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Guo, H.; Wang, L.; Liang, D. Big Earth Data from space: A new engine for Earth science. Sci. Bull.
**2016**, 61, 505–513. [Google Scholar] [CrossRef] - Vatsavai, R.R.; Ganguly, A.; Chandola, V.; Stefanidis, A.; Klasky, S.; Shekhar, S. Spatio-Temporal Data Mining in the Era of Big Spatial Data: Algorithms and Applications. In Proceedings of the 1st ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data, Redondo Beach, CA, USA, 6 November 2012. [Google Scholar]
- Li, R.; Zhang, X.; Liu, B. Review on Filtering and Reconstruction Algorithms of Remote Sensing Time Series Data. J. Remote Sens.
**2009**, 13, 335–341. [Google Scholar] - Seaquist, J.W.; Chappell, A.; Eklundh, L. Exploring and Improving NOAA AVHRR NDVI Image Quality for African Dry-lands. Geosci. Remote Sens. Symp.
**2002**, 4, 2006–2008. [Google Scholar] - Berterretche, M.; Hudak, A.T.; Cohen, W.B.; Maiersperger, T.K.; Gower, S.T.; Dungan, J. Comparison of regression and geostatistical methods for mapping Leaf Area Index (LAI) with Landsat ETM+ data over a boreal forest. Remote Sens. Environ.
**2005**, 96, 49–61. [Google Scholar] [CrossRef] [Green Version] - Bhattacharjee, S.; Mitra, P.; Ghosh, S.K. Spatial Interpolation to Predict Missing Attributes in GIS Using Semantic Kriging. IEEE Trans. Geosci. Remote Sens.
**2013**, 52, 4771–4780. [Google Scholar] [CrossRef] - Zhou, H.; Wang, N.; Huang, Y. Comparison and analysis of remotely sensed time series of reconstruction models at various intervals. J. Geo-Inf. Sci.
**2016**, 18, 1410–1417. [Google Scholar] - Chen, J.; Jönsson, P.; Tamura, M.; Gu, Z.; Matsushita, B.; Eklundh, L. A simple method for reconstructing a high-quality NDVI time-series dataset based on the Savitzky-Golay filter. Remote Sens. Environ.
**2004**, 91, 332–344. [Google Scholar] [CrossRef] - Hua, D.; Yu, H. Objective Evaluation Method for Image Enhancement Quality Based on Visual Information Fidelity. Micro-Comput. Inf.
**2012**, 28, 173–175. [Google Scholar] - Crosson, W.L.; Al-Hamdan, M.Z.; Hemmings, S.N.; Wade, G.M. A daily merged MODIS Aqua-Terra land surface temperature dataset for the conterminous United States. Remote Sens. Environ.
**2012**, 119, 315–324. [Google Scholar] [CrossRef] - Hao, G.; Wu, B.; Zhang, L.; Fu, D.; Li, Y. Application and Analysis of ESTARFM Model in spatio-temporal Variation of Serincuo Lake Area, Tibet (1976–2014). J. Geo-Inf. Sci.
**2016**, 18, 833–846. [Google Scholar] - Peng, J.; Luo, W.; Ning, X.; Zou, Z. Fusion of remote sensing images based on the STARFM model. Cent. South For. Surv. Plan.
**2018**, 37, 32–37. [Google Scholar] [CrossRef] - Wang, Q.; Tang, Y.; Tong, X.; Atkinson, P.M. Virtual image pair-based spatio-temporal fusion. Remote Sens. Environ.
**2020**, 249, 112009. [Google Scholar] [CrossRef] - Xu, W.; Chen, R.; Huang, B.; Zhang, X.; Liu, C. Single Image Super-Resolution Based on Global Dense Feature Fusion Convolutional Network. Sensors
**2019**, 19, 316. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Lu, Y.; Liu, J.; Jin, S. Image Processing on Time Series. Crim. Technol.
**2004**, 2, 41–42. [Google Scholar] [CrossRef] - Yuval, M.; Alonso, I.; Eyal, G.; Tchernov, D.; Loya, Y.; Murillo, A.C.; Treibitz, T. Repeatable Semantic Reef-Mapping through Photogrammetry and Label-Augmentation. Remote Sens.
**2021**, 13, 659. [Google Scholar] [CrossRef] - Kalajdjieski, J.; Zdravevski, E.; Corizzo, R.; Lameski, P.; Kalajdziski, S.; Pires, I.M.; Garcia, N.M.; Trajkovik, V. Air Pollution Prediction with Multi-Modal Data and Deep Neural Networks. Remote Sens.
**2020**, 12, 4142. [Google Scholar] [CrossRef] - Wang, W.; Liu, X.; Mou, X. Data Augmentation and Spectral Structure Features for Limited Samples Hyperspectral Classification. Remote Sens.
**2021**, 13, 547. [Google Scholar] [CrossRef] - Long, G.; Kneip, L.; Alvarez, J.M.; Li, H.; Zhang, X.; Yu, Q. Learning Image Matching by Simply Watching Video. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016. [Google Scholar] [CrossRef]
- Liu, Z.; Yeh, R.; Tang, X.; Liu, Y.; Agarwala, A. Video Frame Synthesis Using Deep Voxel Flow. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017. [Google Scholar] [CrossRef] [Green Version]
- Niklaus, S.; Mai, L.; Liu, F. Video Frame Interpolation via Adaptive Convolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Pan, X.L.; Zhang, Z.; Jiang, T. Research on NDVI prediction method based on convolutional long short-term memory network. Geogr. Inf. World
**2020**, 27, 60–67. [Google Scholar] - Zhu, J.Y.; Park, T.; Isola, P. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; IEEE Computer Society: Washington, DC, USA; pp. 2223–2232. [Google Scholar] [CrossRef] [Green Version]
- Alvarez-Vanhard, E.; Houet, T.; Mony, C.; Lecoq, L.; Corpetti, T. Can UAVs fill the gap between in situ surveys and satellites for habitat mapping? Remote Sens. Environ.
**2020**, 243, 111780. [Google Scholar] [CrossRef] - Dong, C.; Loy, C.C.; He, K.; Tang, X. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell.
**2015**, 38, 295–307. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zeiler, M.D.; Taylor, G.W.; Fergus, R. Adaptive Deconvolutional Networks for Mid and High Level Feature Learning. In Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 2018–2025. [Google Scholar]
- Shi, W.; Caballero, J.; Huszar, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 1874–1883. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006. [Google Scholar]
- Long, J.; Shelhamer, E.; Darrell, T. Fully Convolutional Networks for Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
- Yuan, Z.; Zhou, X.; Yang, T. Hetero-ConvLSTM: A Deep Learning Approach to Traffic Accident Prediction on Heterogeneous Spatio-Temporal Data. In Proceedings of the International Conference on Knowledge Discovery & Data Mining, London, UK, 19–23 August 2018; pp. 984–992. [Google Scholar] [CrossRef]
- Zhao, C.; Chen, P.H. Icon Shape Generation Based on Generative Adversarial Network. Comput. Sci. Appl.
**2020**, 10, 456–463. [Google Scholar] [CrossRef] - Tian, L.; Cao, Y.; He, B.; Zhang, Y.; He, C.; Li, D. Image Enhancement Driven by Object Characteristics and Dense Feature Reuse Network for Ship Target Detection in Remote Sensing Imagery. Remote Sens.
**2021**, 13, 1327. [Google Scholar] [CrossRef] - Lu, C.-G. A generalization of Shannon’s information theory. Int. J. Gen. Syst.
**1999**, 28, 453–490. [Google Scholar] [CrossRef] - Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv
**2014**, arXiv:1412.6980. [Google Scholar] - Collobert, R.; Kavukcuoglu, K.; Farabet, C. Torch7: A MatLab-Like Environment for Machine Learning. In BigLearn, NIPS Workshop; no.EPFL-CONF-192376; Granada, Spain, 1 January 2011; Available online: https://infoscience.epfl.ch/record/192376/ (accessed on 15 October 2020).
- Huang, X.; Shi, J.; Yang, J.; Yao, J. Evaluation of Color Image Quality Based on Mean Square Error and Peak Signal-to-Noise Ratio of Color Difference. Acta Photonica Sin.
**2007**, S1, 295–298. Available online: http://en.cnki.com.cn/Article_en (accessed on 15 October 2020).

**Figure 3.**Overview of AdaFG architecture: (

**a**–

**d**) represent feature extracting part, feature expanding part, separable convolution part, and backpropagation part, respectively.

**Figure 6.**Visual effect of training (${I}_{t}$) and testing (${I}_{t}^{\prime}$ ) images using different deep convolution networks in the first aspect: (

**A**) UAV and (

**B**) Landsat-8 images; (

**a**–

**c**) show the visual effect using AdaFG, CLSTM, and CyGAN network, respectively.

**Figure 7.**Strategy for constructing sequence datasets in the second and third aspect: (

**A**) available UAV images from 2017 to 2019 and Landsat-8 images from 2013 to 2015; (

**B**) one construction strategy with single network; (

**C**) one construction strategy with multiple networks.

**Figure 8.**(

**a**) Visual effect and (

**b**) pixel error between generated result and reference image using different deep convolution networks: (

**A**) UAV and (

**B**) Landsat-8 datasets.

**Figure 9.**Spectral curves between generated result and reference image using different deep convolution networks at different coordinates: (

**A**) UAV and (

**B**) Landsat-8 datasets.

**Figure 10.**Generated results of UAV images using AdaFG network from 2017 to 2019 according to construction strategy in Table 3: existing images and generated results in (

**A**) 2019 sequence, (

**B**) 2018 sequence, and (

**C**) 2017 sequence.

**Figure 11.**Generated results of Landsat-8 images using CyGAN network from 2013 to 2015 according to construction strategy in Table 4: existing images and generated results in (

**A**) 2015 sequence, (

**B**) 2014 sequence, and (

**C**) 2013 sequence.

**Figure 12.**Generated results of UAV images using AdaFG, CLSTM, and CyGAN network from 2017 to 2019 according to construction strategy in Table 5: existing images and generated results in (

**A**) 2019 sequence, (

**B**) 2018 sequence, and (

**C**) 2017 sequence.

**Figure 13.**Generated results of Landsat-8 images using AdaFG, CLSTM, and CyGAN network from 2013 to 2015 according to construction strategy in Table 6: existing images and generated results in (

**A**) 2015 sequence, (

**B**) 2014 sequence, and (

**C**) 2013 sequence.

**Figure 14.**(

**a**) Initial image, (

**b**–

**d**) Visual effect and (

**e**–

**g**) pixel error between the generated results and reference image using separable convolution kernel sizes 11, 13, and 15: (

**A**) UAV and (

**B**) Landsat-8 datasets.

**Figure 15.**(

**a**) Initial image, (

**b**–

**d**) Visual effect and (

**e**–

**g**) pixel error between the generated result and reference image using stacks of 1 × 1, 2 × 2, and 3 × 3 CLSTM network layers: (

**A**) UAV and (

**B**) Landsat-8 datasets.

**Figure 16.**(

**a**) Initial image, (

**b**–

**d**) Visual effect and (

**e**–

**g**) pixel error between the generated result and reference image using different proportional hyper-parameters: (

**A**) UAV and (

**B**) Landsat-8 datasets.

Type | Kernel Size | Stride | Padding | Feature Maps (Filters × W × H) |
---|---|---|---|---|

Input | — | — | — | 2n × 128 × 128 |

Conv-1 | 3 × 3 | 1 × 1 | 1 × 1 | 32 × 128 × 128 |

Pool-1 | 2 × 2 | 2 × 2 | 32 × 64 × 64 | |

Conv-2 | 3 × 3 | 1 × 1 | 64 × 64 × 64 | |

Pool-2 | 2 × 2 | 2 × 2 | 64 × 32 × 32 | |

Conv-3 | 3 × 3 | 1 × 1 | 128 × 32 × 32 | |

Pool-3 | 2 × 2 | 2 × 2 | 128 × 16 × 16 | |

Conv-4 | 3 × 3 | 1 × 1 | 256 × 16 × 16 | |

Pool-4 | 2 × 2 | 2 × 2 | 256 × 8 × 8 | |

Conv-5 | 3 × 3 | 1 × 1 | 512 × 8 × 8 | |

Pool-5 | 2 × 2 | 2 × 2 | 512 × 4 × 4 | |

Deconv-1 | 3 × 3 | 1 × 1 | 512 × 4 × 4 | |

Upsample-1 | 512 × 8 × 8 | |||

Deconv-2 | 256 × 8 × 8 | |||

Upsample-2 | 256 × 16 × 16 | |||

Deconv-3 | 128 × 16 × 16 | |||

Upsample-3 | 128 × 32 × 32 | |||

Deconv-4 | 64 × 32 × 32 | |||

Upsample-4 | 64 × 64 × 64 | |||

Subnet-1 | k × 128 × 128 | |||

Subnet-2 | k × 128 × 128 | |||

Subnet-3 | k × 128 × 128 | |||

Subnet-4 | k × 128 × 128 |

Datasets | Image Names | Image Dates |
---|---|---|

UAV | ${I}_{4}$ | April 2019 |

${I}_{5}$ | May 2019 | |

${I}_{6}$ | June 2019 | |

${I}_{7}$ | July 2019 | |

${I}_{8}$ | August 2019 | |

Landsat-8 | ${I}_{4}^{\prime}$ | April 2014 |

${I}_{7}^{\prime}$ | July 2014 | |

${I}_{2}^{\prime}$ | February 2014 | |

${I}_{8}^{\prime}$ | August 2014 | |

${I}_{12}^{\prime}$ | December 2014 |

Color. | Training Images | Testing Images | Output Images |
---|---|---|---|

Crimson | ${I}_{4},{I}_{5},{I}_{1}$ | ${I}_{4}^{\prime},{I}_{5}^{\prime}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{1}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ |

${I}_{4},{I}_{5},{I}_{3}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{3}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4},{I}_{5},{I}_{6}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4},{I}_{5},{I}_{7}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4},{I}_{5},{I}_{8}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{9}^{\prime}$ | ${I}_{4},{I}_{5}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | |

${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{10}^{\prime}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{10}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | ||

${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{12}^{\prime}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | ||

Red | ${I}_{7}^{\u2033},{I}_{10}^{\u2033},{I}_{11}^{\u2033}$ | ${I}_{7}^{\prime},{I}_{10}^{\prime}$ | ${f}_{\left[{I}_{7}^{\u2033},{I}_{10}^{\u2033},{I}_{11}^{\u2033}\right]}\left({I}_{7}^{\prime},{I}_{10}^{\prime}\right)$ |

Light-red | ${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{11}^{\prime}$ | ${I}_{4},{I}_{5}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{11}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ |

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{1}^{\prime}$ | ${I}_{10}^{\u2033},{I}_{11}^{\u2033}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | |

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{2}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{3}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{4}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{5}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{5}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{6}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{6}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{8}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{9}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ||

${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{12}^{\prime}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ |

Color | Training Images | Testing Images | Output Images |
---|---|---|---|

Dark-green | ${I}_{1}^{\prime},{I}_{2}^{\prime}$ | ${I}_{1}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{1}\right)$ |

${I}_{1}^{\prime},{I}_{8}^{\prime}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{1}\right)$ | ||

${I}_{1}^{\prime},{I}_{12}^{\prime}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{1}\right)$ | ||

${I}_{1},{I}_{3}$ | ${I}_{1}^{\prime}$ | ${f}_{\left[{I}_{1},{I}_{3}\right]}\left({I}_{1}^{\prime}\right)$ | |

${I}_{1},{I}_{5}$ | ${f}_{\left[{I}_{1},{I}_{5}\right]}\left({I}_{1}^{\prime}\right)$ | ||

${I}_{1},{I}_{10}$ | ${f}_{\left[{I}_{1},{I}_{10}\right]}\left({I}_{1}^{\prime}\right)$ | ||

${I}_{12}^{\u2033},{I}_{6}^{\u2033}$ | ${I}_{12}^{\prime}$ | ${f}_{\left[{I}_{12}^{\u2033},{I}_{6}^{\u2033}\right]}\left({I}_{12}^{\prime}\right)$ | |

${I}_{12}^{\u2033},{I}_{9}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\u2033},{I}_{9}^{\u2033}\right]}\left({I}_{12}^{\prime}\right)$ | ||

${I}_{12}^{\u2033},{I}_{11}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\u2033},{I}_{11}^{\u2033}\right]}\left({I}_{12}^{\prime}\right)$ | ||

${I}_{12}^{\prime},{I}_{1}^{\prime}$ | ${I}_{12}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | |

${I}_{12}^{\prime},{I}_{2}^{\prime}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ||

${I}_{12}^{\prime},{I}_{4}^{\prime}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ||

${I}_{12}^{\prime},{I}_{7}^{\prime}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{7}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ||

${I}_{12}^{\prime},{I}_{8}^{\prime}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ||

Light-green | ${I}_{1}^{\prime},{I}_{6}^{\prime}$ | ${I}_{1}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{6}^{\prime}\right]}\left({I}_{1}\right)$ |

${I}_{1}^{\prime},{I}_{9}^{\prime}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{1}\right)$ | ||

${I}_{1}^{\prime},{I}_{11}^{\prime}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{11}^{\prime}\right]}\left({I}_{1}\right)$ | ||

${I}_{12}^{\prime},{I}_{3}^{\prime}$ | ${I}_{12}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | |

${I}_{12}^{\prime},{I}_{10}^{\prime}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{10}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ |

Networks. | Training Images | Testing Images | Output Images |
---|---|---|---|

AdaFG | ${I}_{4},{I}_{5},{I}_{1}$ | ${I}_{4}^{\prime},{I}_{5}^{\prime}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{1}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ |

${I}_{4},{I}_{5},{I}_{3}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{3}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4},{I}_{5},{I}_{6}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4},{I}_{5},{I}_{7}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4},{I}_{5},{I}_{8}$ | ${f}_{\left[{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ||

${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{9}^{\prime}$ | ${I}_{4},{I}_{5}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | |

${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{10}^{\prime}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{10}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | ||

${I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{12}^{\prime}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | ||

${I}_{7}^{\u2033},{I}_{10}^{\u2033},{I}_{11}^{\u2033}$ | ${I}_{7}^{\prime},{I}_{10}^{\prime}$ | ${f}_{\left[{I}_{7}^{\u2033},{I}_{10}^{\u2033},{I}_{11}^{\u2033}\right]}\left({I}_{7}^{\prime},{I}_{10}^{\prime}\right)$ | |

CyGAN | ${I}_{4}^{\prime},{I}_{11}^{\prime}$ | ${I}_{4}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{11}^{\prime}\right]}\left({I}_{4}\right)$ |

${I}_{10}^{\prime},{I}_{12}^{\prime}$ | ${I}_{10}^{\u2033}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{10}^{\u2033}\right)$ | |

CLSTM | ${I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{8}^{\prime},{I}_{9}^{\prime}$ | ${I}_{10}^{\u2033},{I}_{11}^{\u2033}$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ |

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | |||

${I}_{7}^{\prime},{I}_{8}^{\prime},{I}_{9}^{\prime},{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{12}^{\prime}$ ${I}_{1}^{\prime},{I}_{2}^{\prime},{I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{6}^{\prime}$ | ${I}_{7}^{\u2033},{I}_{8}^{\u2033},{I}_{9}^{\u2033},{I}_{10}^{\u2033},{I}_{11}^{\u2033},{I}_{12}^{\u2033}$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | |

${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | |||

${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | |||

${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | |||

${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{5}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | |||

${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{6}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ |

Networks | Training Images | Testing Images | Output Images |
---|---|---|---|

AdaFG | ${I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}$ | ${I}_{4},{I}_{7}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ |

${I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ||

${I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ||

${I}_{4},{I}_{7},{I}_{3}$ | ${I}_{4}^{\prime},{I}_{7}^{\prime}$ | ${f}_{\left[{I}_{4},{I}_{7},{I}_{3}\right]}\left({I}_{4}^{\prime},{I}_{7}^{\prime}\right)$ | |

${I}_{4},{I}_{7},{I}_{5}$ | ${f}_{\left[{I}_{4},{I}_{7},{I}_{5}\right]}\left({I}_{4}^{\prime},{I}_{7}^{\prime}\right)$ | ||

${I}_{4},{I}_{7},{I}_{10}$ | ${f}_{\left[{I}_{4},{I}_{7},{I}_{10}\right]}\left({I}_{4}^{\prime},{I}_{7}^{\prime}\right)$ | ||

CyGAN | ${I}_{12}^{\u2033},{I}_{6}^{\u2033}$ | ${I}_{12}^{\prime}$ | ${f}_{\left[{I}_{12}^{\u2033},{I}_{6}^{\u2033}\right]}\left({I}_{12}^{\prime}\right)$ |

${I}_{12}^{\u2033},{I}_{9}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\u2033},{I}_{9}^{\u2033}\right]}\left({I}_{12}^{\prime}\right)$ | ||

${I}_{12}^{\u2033},{I}_{11}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\u2033},{I}_{11}^{\u2033}\right]}\left({I}_{12}^{\prime}\right)$ | ||

${I}_{1}^{\prime},{I}_{6}^{\prime}$ | ${I}_{1}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{6}^{\prime}\right]}\left({I}_{1}\right)$ | |

${I}_{1}^{\prime},{I}_{9}^{\prime}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{1}\right)$ | ||

${I}_{1}^{\prime},{I}_{11}^{\prime}$ | ${f}_{\left[{I}_{1}^{\prime},{I}_{11}^{\prime}\right]}\left({I}_{1}\right)$ | ||

${I}_{12}^{\prime},{I}_{10}^{\prime}$ | ${I}_{12}^{\u2033}$ | ${f}_{\left[{I}_{12}^{\prime},{I}_{10}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | |

CLSTM | ${I}_{11}^{\prime},{I}_{12}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}$ | ${I}_{11}^{\u2033},{I}_{12}^{\u2033}$ | ${f}_{\left[{I}_{11}^{\prime},{I}_{12}^{\prime},{I}_{7}^{\prime}\right]}\left({I}_{11}^{\u2033},{I}_{12}^{\u2033}\right)$ |

${f}_{\left[{I}_{11}^{\prime},{I}_{12}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{11}^{\u2033},{I}_{12}^{\u2033}\right)$ | |||

${I}_{5}^{\prime},{I}_{6}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}$ ${I}_{1}^{\prime},{I}_{2}^{\prime},{I}_{3}^{\prime},{I}_{4}^{\prime}$ | ${I}_{5}^{\u2033},{I}_{6}^{\u2033},{I}_{7}^{\u2033},{I}_{8}^{\u2033}$ | ${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | |

${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | |||

${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | |||

${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ |

Datasets | Generated Results | Reference Images | Entropy | RMSE (Pixel) |
---|---|---|---|---|

UAV | ${f}_{\left[{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | I_{6} | 3.658 | 1.128 |

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | 3.625 | 1.758 | ||

${f}_{\left[{I}_{4},{I}_{6}\right]}\left({I}_{4}^{\prime}\right)$ | 3.592 | 3.585 | ||

${f}_{\left[{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | I_{7} | 3.355 | 0.952 | |

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | 3.330 | 1.300 | ||

${f}_{\left[{I}_{4},{I}_{7}\right]}\left({I}_{4}^{\prime}\right)$ | 3.305 | 3.077 | ||

${f}_{\left[{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | I_{8} | 3.482 | 0.954 | |

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | 3.455 | 1.421 | ||

${f}_{\left[{I}_{4},{I}_{8}\right]}\left({I}_{4}^{\prime}\right)$ | 3.426 | 3.690 | ||

Landsat-8 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 3.064 | 1.144 |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | 2.975 | 1.537 | ||

${f}_{\left[{I}_{1}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{1}\right)$ | 2.905 | 2.694 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 3.830 | 1.170 | |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | 3.830 | 1.681 | ||

${f}_{\left[{I}_{1}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{1}\right)$ | 3.777 | 5.017 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 2.994 | 0.929 | |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | 2.848 | 1.484 | ||

${f}_{\left[{I}_{1}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{1}\right)$ | 2.796 | 2.629 |

**Table 8.**Quantitative evaluation indicator between the generated results using a single network (AdaFG, CyGAN) and multiple networks.

Datasets | Single Network | Multiple Network | Reference Images | RMSE (Pixel) |
---|---|---|---|---|

UAV | ${f}_{\left[{I}_{4}^{\prime},{I}_{5}^{\prime},{I}_{11}^{\prime}\right]}\left({I}_{4},{I}_{5}\right)$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{11}^{\prime}\right]}\left({I}_{4}\right)$ | ${I}_{11}^{\u2033}$ | 4.953 |

4.702 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{10}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{10}^{\u2033}\right)$ | ${I}_{12}^{\prime}$ | 3.847 | |

3.787 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{9}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${I}_{9}^{\prime}$ | 2.723 | ||

2.638 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${I}_{8}$ | 2.879 | ||

1.933 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{6}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{6}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | ${I}_{6}$ | 3.483 | |

1.228 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{5}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{5}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | ${I}_{5}^{\prime}$ | 2.967 | |

1.304 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | ${I}_{4}^{\prime}$ | 3.100 | |

1.664 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | ${I}_{3}$ | 3.864 | |

2.704 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | ${I}_{2}^{\prime}$ | 3.504 | |

1.580 | ||||

${f}_{\left[{I}_{10}^{\prime},{I}_{11}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{10}^{\u2033},{I}_{11}^{\u2033}\right)$ | ${f}_{\left[{I}_{7}^{\prime},\dots ,{I}_{12}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{7}^{\u2033},\dots ,{I}_{12}^{\u2033}\right)$ | ${I}_{1}$ | 3.188 | |

2.377 | ||||

Landsat-8 | ${f}_{\left[{I}_{1}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{1}\right)$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 2.695 |

1.144 | ||||

${f}_{\left[{I}_{1}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{1}\right)$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 5.017 | |

1.170 | ||||

${f}_{\left[{I}_{1}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{1}\right)$ | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 2.777 | |

0.929 | ||||

${f}_{\left[{I}_{1},{I}_{3}\right]}\left({I}_{1}^{\prime}\right)$ | ${f}_{\left[{I}_{4},{I}_{7},{I}_{3}\right]}\left({I}_{4}^{\prime},{I}_{7}^{\prime}\right)$ | ${I}_{3}$ | 2.632 | |

1.286 | ||||

${f}_{\left[{I}_{1},{I}_{5}\right]}\left({I}_{1}^{\prime}\right)$ | ${f}_{\left[{I}_{4},{I}_{7},{I}_{5}\right]}\left({I}_{4}^{\prime},{I}_{7}^{\prime}\right)$ | ${I}_{5}$ | 3.375 | |

1.411 | ||||

${f}_{\left[{I}_{1},{I}_{10}\right]}\left({I}_{1}^{\prime}\right)$ | ${f}_{\left[{I}_{4},{I}_{7},{I}_{10}\right]}\left({I}_{4}^{\prime},{I}_{7}^{\prime}\right)$ | ${I}_{10}$ | 3.990 | |

1.968 | ||||

${f}_{\left[{I}_{12}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ${f}_{\left[{I}_{11}^{\prime},{I}_{12}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{11}^{\u2033},{I}_{12}^{\u2033}\right)$ | ${I}_{8}^{\prime}$ | 5.022 | |

1.841 | ||||

${f}_{\left[{I}_{12}^{\prime},{I}_{7}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ${f}_{\left[{I}_{11}^{\prime},{I}_{12}^{\prime},{I}_{7}^{\prime}\right]}\left({I}_{11}^{\u2033},{I}_{12}^{\u2033}\right)$ | ${I}_{7}^{\prime}$ | 4.399 | |

1.729 | ||||

${f}_{\left[{I}_{12}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{4}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | ${I}_{4}^{\prime}$ | 2.771 | |

1.313 | ||||

${f}_{\left[{I}_{12}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{3}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | ${I}_{3}$ | 3.367 | |

2.140 | ||||

${f}_{\left[{I}_{12}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | ${I}_{2}^{\prime}$ | 2.606 | |

2.122 | ||||

${f}_{\left[{I}_{12}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{12}^{\u2033}\right)$ | ${f}_{\left[{I}_{5}^{\prime},\dots ,{I}_{8}^{\prime},{I}_{1}^{\prime}\right]}\left({I}_{5}^{\u2033},\dots ,{I}_{8}^{\u2033}\right)$ | ${I}_{1}^{\prime}$ | 2.686 | |

2.629 |

**Table 9.**Quantitative evaluation between the generated result and reference image using different separable convolution kernel sizes.

Datasets | Kernel Sizes | Generated Results | Reference Images | RMSE (Pixel) |

UAV | 11 | ${f}_{\left[{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{6}$ | 1.128 |

${f}_{\left[{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{7}$ | 0.952 | ||

${f}_{\left[{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{8}$ | 0.954 | ||

13 | ${f}_{\left[{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{6}$ | 1.437 | |

${f}_{\left[{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{7}$ | 1.273 | ||

${f}_{\left[{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{8}$ | 1.416 | ||

15 | ${f}_{\left[{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{6}$ | 1.444 | |

${f}_{\left[{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{7}$ | 1.560 | ||

${f}_{\left[{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{8}$ | 1.418 | ||

Landsat-8 | 11 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 1.144 |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 1.170 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 0.929 | ||

13 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 1.689 | |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 2.533 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 1.402 | ||

15 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 1.938 | |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 2.875 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 1.633 |

**Table 10.**Quantitative evaluation between the generated result and reference image using different stacked CLSTM network layers.

Datasets | Stacked Numbers | Generated Results | Reference Images | RMSE (Pixel) |
---|---|---|---|---|

UAV | 1 × 1 | ${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{6}$ | 2.681 |

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{7}$ | 2.096 | ||

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{8}$ | 2.186 | ||

2 × 2 | ${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{6}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{6}$ | 1.976 | |

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{7}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{7}$ | 1.556 | ||

${f}_{\left[{I}_{3},{I}_{4},{I}_{5},{I}_{8}\right]}\left({I}_{3}^{\prime},{I}_{4}^{\prime},{I}_{5}^{\prime}\right)$ | ${I}_{8}$ | 1.617 | ||

3 × 3 | ${I}_{6}$ | 1.758 | ||

${I}_{7}$ | 1.300 | |||

${I}_{8}$ | 1.421 | |||

Landsat-8 | 1 × 1 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 2.265 |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 3.386 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 1.935 | ||

2 × 2 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 1.780 | |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 2.697 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 1.577 | ||

3 × 3 | ${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{2}^{\prime}$ | 1.537 | |

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{8}^{\prime}$ | 1.681 | ||

${f}_{\left[{I}_{4}^{\prime},{I}_{7}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{4},{I}_{7}\right)$ | ${I}_{12}^{\prime}$ | 1.484 |

**Table 11.**Quantitative evaluation between the generated result and reference image using different proportional hyper-parameters.

Datasets | ${\mathit{\mu}}_{1},{\mathit{\mu}}_{2}$ | Generated Results | Reference Images | RMSE (Pixel) |
---|---|---|---|---|

UAV | 250,1/16 | ${f}_{\left[{I}_{4},{I}_{6}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{6}$ | 3.643 |

${f}_{\left[{I}_{4},{I}_{7}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{7}$ | 3.271 | ||

${f}_{\left[{I}_{4},{I}_{8}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{8}$ | 3.836 | ||

350,1/32 | ${f}_{\left[{I}_{4},{I}_{6}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{6}$ | 3.585 | |

${f}_{\left[{I}_{4},{I}_{7}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{7}$ | 3.077 | ||

${f}_{\left[{I}_{4},{I}_{8}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{8}$ | 3.690 | ||

450,1/64 | ${f}_{\left[{I}_{4},{I}_{6}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{6}$ | 3.814 | |

${f}_{\left[{I}_{4},{I}_{7}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{7}$ | 3.263 | ||

${f}_{\left[{I}_{4},{I}_{8}\right]}\left({I}_{4}^{\prime}\right)$ | ${I}_{8}$ | 3.698 | ||

Landsat-8 | 250,1/16 | ${f}_{\left[{I}_{1}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{2}^{\prime}$ | 2.824 |

${f}_{\left[{I}_{1}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{8}^{\prime}$ | 5.708 | ||

${f}_{\left[{I}_{1}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{12}^{\prime}$ | 2.875 | ||

350,1/32 | ${f}_{\left[{I}_{1}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{2}^{\prime}$ | 2.694 | |

${f}_{\left[{I}_{1}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{8}^{\prime}$ | 5.017 | ||

${f}_{\left[{I}_{1}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{12}^{\prime}$ | 2.629 | ||

450,1/64 | ${f}_{\left[{I}_{1}^{\prime},{I}_{2}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{2}^{\prime}$ | 2.695 | |

${f}_{\left[{I}_{1}^{\prime},{I}_{8}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{8}^{\prime}$ | 5.553 | ||

${f}_{\left[{I}_{1}^{\prime},{I}_{12}^{\prime}\right]}\left({I}_{1}\right)$ | ${I}_{12}^{\prime}$ | 2.777 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jin, X.; Tang, P.; Zhang, Z.
Sequence Image Datasets Construction via Deep Convolution Networks. *Remote Sens.* **2021**, *13*, 1853.
https://doi.org/10.3390/rs13091853

**AMA Style**

Jin X, Tang P, Zhang Z.
Sequence Image Datasets Construction via Deep Convolution Networks. *Remote Sensing*. 2021; 13(9):1853.
https://doi.org/10.3390/rs13091853

**Chicago/Turabian Style**

Jin, Xing, Ping Tang, and Zheng Zhang.
2021. "Sequence Image Datasets Construction via Deep Convolution Networks" *Remote Sensing* 13, no. 9: 1853.
https://doi.org/10.3390/rs13091853