# Reconstruction of Multi-Temporal Satellite Imagery by Coupling Variational Segmentation and Radiometric Analysis

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

- single-image reconstruction;
- multi-image reconstruction

- compositing approach: filling data gaps in target image directly with values from other base images, sometimes rescaled using a linear transformation based on global mean values of base and target images;
- nearest-neighbor approach: making use of values in pixels near the one to be inpainted in order to obtain statistics of a reference data-pool from which the reconstructed values are calculated;
- geostatistics approach: using kriging to explicitly take into account spatial correlation to predict reconstructed values and estimation errors;
- segmentation approach: identifying clusters of homogeneous pixels automatically to build reference data-pools from which statistics for calculation of values to be reconstructed are extracted.

## 2. Overview of Selected Standard Algorithms

#### 2.1. Histogram Matching Methods

#### 2.2. Neighborhood Similar Pixel Interpolation Method

#### 2.3. Geostatistical Method

## 3. Mumford–Shah Variational Model for Image Segmentation

## 4. Coupling Variational Segmentation and Radiometric Analysis for Multi-Temporal Images Reconstruction

- variational segmentation of base and target imagery bands;
- composition of integer rounded segmented bands;
- identification and labeling of connected components (clumps) of composite values;
- calculation of cross product of base and target clump labels;
- radiometric reconstruction.

**Step****1**- Mumford–Shah variational segmentation is applied to base and target imagery bands at hand. In this work, the processing was carried out on digital numbers. A unique mask of SLC-off stripes is used for the processing of target bands. The values of model parameters $\alpha $ and $\lambda $ are selected on an empirical base after a few test; when working with Landsat 7 and DNs, one can start with $\alpha =500$ and $\lambda =8$. The same parameter values were used in the processing bands of each single scene, either base or target. Parameter values different between scenes, i.e., acquisition epochs, are conveniently selected to take into account inter-epochs radiometric differences. To verify if different parameter values yield consistent inter-epoch results, i.e., segmented regions of comparable dimension and shape, segmented base and target bands can be either visually or automatically compared. At this stage, the choice of very different parameter values may depend on very high radiometric differences between base and target bands suggesting the need to look for better base imagery to ease the radiometric reconstruction of the damaged target regions. The Mumford–Shah variational segmentation outputs a real valued piece-wise smooth approximation. Segmented values are rounded to restore integer consistency with input DN. Rounding introduces minor changes with respect to typical DN variances in satellite imagery and much less than the amount of noise-reduction that occurred in segmented regions.
**Step****2**- Composition of integer segmented bands is an easy way to classify the outputs of the previous step. In practice, a total of 32 intensity levels per band are used in the composition, resulting in a 15 bit composed image with 32,768 possible values when, for example, the composition of three bands is performed.
**Step****3**- Dealing with integer composed imagery greatly simplifies the identification and labeling of connected components. Otherwise, if real values were processed, a threshold based fuzzy clumping or an even more sophisticated classification algorithm would be necessary to identify and label segmented regions in a suitable way.
**Step****4**- The cross product of base and target clump labels produces all possible combinations of inter-scene clump labels. Each combination found is associated with the values of the base and target clump labels, allowing for returning to the original integer segmentation values.The output of this step serves as a basic information layer for the reconstruction step. The SLC-off strips mask permits to separate regions to be reconstructed from those from which radiometric analysis will be conducted.For each clump following within the regions to be reconstructed, the reconstruction data set is made of all the pixels belonging to every non-corrupted base clump that shares the same intensity level in the base composite map.Given the value of the intensity level in the base composite map, it is possible to go back to the original integer segmentation values to be used as a data set for radiometric reconstruction.If no identical intensity level can be found, the reconstruction database consists of all clumps with an intensity level as close as possible to the input intensity level. The distance measure is the Euclidean distance in the integer segmented band space and a threshold-based proximity criterion adopted to build the reconstruction database.
**Step****5**- Two different criteria were considered for the final radiometric reconstruction.The first approach is based on the HM transformation. The ECDFs of base and target images are computed considering only non-corrupted regions and the parameters of the HM transformation are estimated. For each pixel to be reconstructed, its value in the base image ${x}_{b}$ is read and its cumulative frequency value ${F}_{b}\left({x}_{b}\right)$ is evaluated. The reconstructed value in the target image ${\widehat{x}}_{t}$ is the one that holds the equality ${F}_{t}\left({\widehat{x}}_{t}\right)={F}_{b}\left({x}_{b}\right)$.The second approach is based on ED and Gaussian distribution sampling. For base homogeneous regions, the band covariance matrix is computed. The data set is transformed according to ED to permit statistical sampling from uncorrelated Gaussian distributions with known mean and variance. The sampling is drawn once for each pixel to be reconstructed. For the sets of base non-corrupted regions with a total count below 30 units, ED and Gaussian sampling are not performed and reconstructed values are set to the mean value of the set of base non-corrupted regions.

## 5. Results

#### 5.1. Applications for Self-Validation

#### 5.1.1. Self-Validation Test on a Single Worldview-3 Band

**Band: Panchromatic (P)—segmentation parameters $\mathit{\alpha}$ = 4000, $\mathit{\lambda}$ = 20**

#### 5.1.2. Self-Validation Tests on a Set of Landsat 7 Bands

`LE71920282000332NSG00`scene which does not present SLC-off defects (Data acquired in mid 2000, before SLC failure). The scene that served as base imagery was artificially damaged by applying SLC-off like gaps to produce target imagery. Original and reconstructed data for a couple of three-band subsets are hereafter presented in order to evaluate the performance of the procedure. In the segmentation phase, about ten tests were performed with different values of the parameters ($\alpha $, $\lambda $). In each test reported hereafter, the same values of the parameters ($\alpha $, $\lambda $) were used for segmentation of base and target bands. The selected parameters are those which led to the best results in the tests. For comparison, each test is complemented by the statistics of results obtained using different pairs of parameter values; for these tests, the reconstructed imagery were qualitatively very similar to those reported in this section.

**Band set: Red, Green, Blue (RGB)—segmentation parameters $\mathit{\alpha}$ = 500, $\mathit{\lambda}$ = 8**

**Band set: Near-infrared, Red, Green (NRG)—segmentation parameters $\mathit{\alpha}$ = 500, $\mathit{\lambda}$ = 8**

#### 5.2. Real Case Applications

`LE71920282014306NSG100`affected by SLC-off disturbances and considering the disturbances-free Landsat 7 scene

`LE71920282002257EDC00`as base imagery, see Figure 16.

## 6. Discussion

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Aubert, G.; Kornprobst, P. Mathematical Problems in Image Processing; Springer: New York, NY, USA, 2006. [Google Scholar] [CrossRef][Green Version]
- Zhang, Q.; Yuan, Q.; Zeng, C.; Li, X.; Wei, Y. Missing Data Reconstruction in Remote Sensing Image with a Unified Spatial–Temporal–Spectral Deep Convolutional Neural Network. IEEE Trans. Geosci. Remote Sens.
**2018**, 56, 4274–4288. [Google Scholar] [CrossRef][Green Version] - Zhu, J.; Li, K.; Hao, B. Restoration of remote sensing images based on nonconvex constrained high-order total variation regularization. J. Appl. Remote Sens.
**2019**, 13, 1–17. [Google Scholar] [CrossRef] - Shen, H.; Li, X.; Cheng, Q.; Zeng, C.; Yang, G.; Li, H.; Zhang, L. Missing Information Reconstruction of Remote Sensing Data: A Technical Review. IEEE Geosci. Remote Sens. Mag.
**2015**, 3, 61–85. [Google Scholar] [CrossRef] - Ali, L.; Kasetkasem, T.; Khan, W.; Chanwimaluang, T.; Nakahara, H. Performance evaluation of different inpainting algorithms for remotely sensed images. In Proceedings of the 2017 Third Asian Conference on Defence Technology (ACDT), Phuket, Thailand, 18–20 January 2017; pp. 43–48. [Google Scholar] [CrossRef]
- Dong, J.; Yin, R.; Sun, X.; Li, Q.; Yang, Y.; Qin, X. Inpainting of Remote Sensing SST Images With Deep Convolutional Generative Adversarial Network. IEEE Geosci. Remote Sens. Lett.
**2019**, 16, 173–177. [Google Scholar] [CrossRef] - Chen, J.; Zhu, X.; Vogelmann, J.E.; Gao, F.; Jin, S. A simple and effective method for filling gaps in Landsat ETM+ SLC-off images. Remote Sens. Environ.
**2011**, 115, 1053–1064. [Google Scholar] [CrossRef] - Brooks, E.; Wynne, R.; Thomas, V. Using Window Regression to Gap-Fill Landsat ETM+ Post SLC-Off Data. Remote Sens.
**2018**, 10, 1502. [Google Scholar] [CrossRef][Green Version] - Qureshi, M.A.; Deriche, M.; Beghdadi, A.; Amin, A. A critical survey of state-of-the-art image inpainting quality assessment metrics. J. Vis. Commun. Image Represent.
**2017**, 49, 177–191. [Google Scholar] [CrossRef] - Asare, Y.M.; Forkuo, E.K.; Forkuor, G.; Thiel, M. Evaluation of gap-filling methods for Landsat 7 ETM+ SLC-off image for LULC classification in a heterogeneous landscape of West Africa. Int. J. Remote Sens.
**2020**, 41, 2544–2564. [Google Scholar] [CrossRef] - Mumford, D.; Shah, J. Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems. Commun. Pure Appl. Math.
**1989**, 42, 577–685. [Google Scholar] [CrossRef][Green Version] - Zambelli, P.; Gebbert, S.; Ciolli, M. Pygrass: An Object Oriented Python Application Programming Interface (API) for Geographic Resources Analysis Support System (GRASS) Geographic Information System (GIS). ISPRS Int. J. Geo-Inf.
**2013**, 2, 201–219. [Google Scholar] [CrossRef][Green Version] - Scaramuzza, P.; Micijevic, E.; Gyanesh, C. SLC Gap-Filled Products—Phase One Methodology. Available online: https://www.usgs.gov/media/files/landsat-7-slc-gap-filled-products-phase-one-methodology (accessed on 1 December 2020).
- USGS. SLC Gap-Filled Products—Phase Two Methodology. Available online: https://www.usgs.gov/media/files/landsat-7-slc-gap-filled-products-phase-two-methodology (accessed on 1 December 2020).
- Aghamohamadnia, M.; Abedini, A. A morphology-stitching method to improve Landsat SLC-off images with stripes. Geod. Geodyn.
**2014**, 5, 27–33. [Google Scholar] [CrossRef][Green Version] - Zhang, C.; Li, W.; Travis, D. Gaps-fill of SLC-off Landsat ETM+ satellite image using a geostatistical approach. Int. J. Remote Sens.
**2007**, 28, 5103–5122. [Google Scholar] [CrossRef] - Gaohong, Y.; McCabe, M.F.; Mariethoz, G. Gap-Filling of Landsat 7 Imagery Using the Direct Sampling Method. Remote Sens.
**2016**, 9, 12. [Google Scholar] - De Giorgi, E. Free discontinuity problems in calculus of variations. In Frontiers in Pure and Applied Mathemathics, a Collection of Papers Dedicated to J.L. Lions on the Occasion of His 60th Birthday; Dautray, R., Ed.; North Holland: Amsterdam, The Netherlands, 1991; pp. 55–62. [Google Scholar]
- De Giorgi, E.; Carriero, M.; Leaci, A. Existence theorem for a minimum problem with free discontinuity set. Arch. Ration. Mech. Anal.
**1989**, 108, 195–218. [Google Scholar] [CrossRef] - Ambrosio, L.; Fusco, N.; Pallara, D. Functions of Bounded Variation and Free Discontinuity Problems; Oxford University Press: Oxford, UK, 2000. [Google Scholar]
- Modica, L.; Mortola, S. Un esempio di Gamma-convergenza. Boll. dell’Unione Matatematica Ital.
**1977**, B-14, 285–299. [Google Scholar] - Vitti, A. The Mumford–Shah variational model for image segmentation: An overview of the theory, implementation and use. ISPRS J. Photogramm. Remote Sens.
**2012**, 69, 50–64. [Google Scholar] [CrossRef] - Yeo, I.K.; Johnson, R.A. A New Family of Power Transformations to Improve Normality or Symmetry. Biometrika
**2000**, 87, 954–959. [Google Scholar] [CrossRef] - Box, G.E.P.; Cox, D.R. An Analysis of Transformations. J. R. Stat. Soc. Ser. B (Methodol.)
**1964**, 26, 211–243. [Google Scholar] [CrossRef] - Baetens, L.; Desjardins, C.; Hagolle, O. Validation of Copernicus Sentinel-2 Cloud Masks Obtained from MAJA, Sen2Cor, and FMask Processors Using Reference Cloud Masks Generated with a Supervised Active Learning Procedure. Remote Sens.
**2019**, 11, 433. [Google Scholar] [CrossRef][Green Version] - Zhu, Z.; Woodcock, C.E. Automated cloud, cloud shadow, and snow detection in multitemporal Landsat data: An algorithm designed specifically for monitoring land cover change. Remote Sens. Environ.
**2014**, 152, 217–234. [Google Scholar] [CrossRef] - Owen, T. Visual Reconstruction by Andrew Blake and Andrew Zisserman, The MIT Press, Massachusetts, USA, 1987, 1987 (£22.50). Robotica
**1988**, 6, 166. [Google Scholar] [CrossRef] - Carriero, M.; Leaci, A.; Tomarelli, F. Free gradient discontinuity and image inpainting. J. Math. Sci.
**2012**, 181, 805–819. [Google Scholar] [CrossRef] - Borghi, A.; Cannizzaro, L.; Vitti, A. Advanced techniques for discontinuity detection in GNSS coordinate time-series. An Italian case study. In International Association of Geodesy Symposia; Springer: Berlin, Germany, 2012; Volume 136, pp. 627–634. [Google Scholar] [CrossRef]
- Vitti, A. Sigseg: A tool for the detection of position and velocity discontinuities in geodetic time-series. GPS Solut.
**2012**, 16, 405–410. [Google Scholar] [CrossRef] - Zanetti, M.; Vitti, A. The Blake-Zisserman model for digital surface models segmentation. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2013**, II-5/W2, 355–360. [Google Scholar] [CrossRef][Green Version] - Benciolini, B.; Ruggiero, V.; Vitti, A.; Zanetti, M. Roof planes detection via a second-order variational model. ISPRS J. Photogramm. Remote Sens.
**2018**, 138, 101–120. [Google Scholar] [CrossRef]

**Figure 6.**Panchromatic self-validation test—radiometric reconstructions; B = Base intensity values, R = Reconstructed intensity values. (

**a**) Scatter plot of Panchromatic values—HM based reconstruction; (

**b**) scatter plot of Panchromatic values—ED based reconstruction.

**Figure 8.**Panchromatic band and radiometric reconstructions—Detail 1. (

**a**) WorldView-3 Panchromatic band; (

**b**) HM reconstruction; (

**c**) ED reconstruction.

**Figure 9.**Panchromatic band and radiometric reconstructions—Detail 2. (

**a**) WorldView-3 Panchromatic band; (

**b**) HM reconstruction; (

**c**) ED reconstruction.

**Figure 10.**Base, target, HM, and ED reconstructions of the RGB band set of Landsat 7

`LE71920282000332NSG00`scene. (

**a**) base image; (

**b**) target image; (

**c**) HM reconstruction; (

**d**) ED reconstruction.

**Figure 11.**RGB self-validation test—HM based reconstruction; B = Base intensity values, R = Reconstructed intensity values. (

**a**) scatter plot of red values; (

**b**) scatter plot of green values; (

**c**) scatter plot of blue values.

**Figure 12.**RGB self-validation test—ED based reconstruction; B = Base intensity values, R = Reconstructed intensity values. (

**a**) scatter plot of red values; (

**b**) scatter plot of green values; (

**c**) scatter plot of blue values.

**Figure 13.**Base, target, HM reconstruction of the NRG band set of Landsat 7

`LE71920282000332NSG00`scene. (

**a**) base image; (

**b**) target image; (

**c**) HM reconstruction.

**Figure 14.**NRG self-validation test—HM based reconstruction; B = Base intensity values, R = Reconstructed intensity values. (

**a**) scatter plot of Near-infrared values; (

**b**) scatter plot of Red values; (

**c**) scatter plot of Green values.

**Figure 15.**NRG self-validation test—ED based reconstruction; B = Base intensity values, R = Reconstructed intensity values. (

**a**) scatter plot of near-infrared values; (

**b**) scatter plot of red values; (

**c**) scatter plot of green values.

**Figure 16.**Base and target images from

`LE71920282002257EDC00`and

`LE71920282014306NSG100`Landsat 7 scenes. (

**a**) base image; (

**b**) target image.

**Figure 18.**Scheme for the construction of radiometric data set for reconstruction. (

**a**) base and target images; (

**b**) base and target images with red region boundaries and one region in green; (

**c**) base and target images with red region boundaries and one region in green and with base coherent regions in light green; (

**d**) base and target images with red region boundaries and one region in green, base coherent regions in light green and with target coherent regions in light blue.

**Figure 20.**Target and HM reconstruction (portion of), see Figure 16. (

**a**) target image; (

**b**) HM reconstruction.

**Table 1.**Error characteristics of panchromatic self-validation tests with $(\alpha ,\lambda )=(4000,20)$; mean of intensity differences, variance of intensity differences, coefficient of determination.

HM | ED | |||||
---|---|---|---|---|---|---|

${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | ${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | |

P | $0.002$ | $0.013$ | $0.999$ | $-0.423$ | $256.313$ | $0.932$ |

**Table 2.**Error characteristics of RGB self-validation tests with $(\alpha ,\lambda )=(500,8)$; mean of intensity differences, variance of intensity differences, coefficient of determination.

HM | ED | |||||
---|---|---|---|---|---|---|

${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | ${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | |

R | $-0.013$ | $0.555$ | $0.999$ | $0.037$ | $6.020$ | $0.989$ |

G | $-0.009$ | $0.542$ | $0.999$ | $-0.037$ | $8.196$ | $0.986$ |

B | $-0.008$ | $0.867$ | $0.999$ | $-0.054$ | $18.071$ | $0.996$ |

**Table 3.**Error characteristics of RGB self-validation tests with $(\alpha ,\lambda )=(1200,8)$, reported for comparison with Table 2; mean of intensity differences, variance of intensity differences, and coefficient of determination.

HM | ED | |||||
---|---|---|---|---|---|---|

${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | ${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | |

R | $-0.015$ | $1.542$ | $0.998$ | $-0.054$ | $21.942$ | $0.968$ |

G | $-0.013$ | $1.003$ | $0.998$ | $-0.034$ | $10.039$ | $0.982$ |

B | $-0.012$ | $0.955$ | $0.998$ | $-0.056$ | $7.330$ | $0.969$ |

**Table 4.**Error characteristics of NRG self-validation tests with $(\alpha ,\lambda )=(500,8)$; mean of intensity differences, variance of intensity differences, and coefficient of determination.

HM | ED | |||||
---|---|---|---|---|---|---|

${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | ${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | |

N | $0.022$ | $2.579$ | $0.996$ | $-0.071$ | $26.062$ | $0.947$ |

R | $-0.031$ | $1.774$ | $0.997$ | $0.090$ | $16.773$ | $0.976$ |

G | $0.021$ | $1.080$ | $0.998$ | $0.072$ | $7.456$ | $0.987$ |

**Table 5.**Error characteristics of NRG self-validation tests with $(\alpha ,\lambda )=(1600,12)$, reported for comparison with Table 4; mean of intensity differences, variance of intensity differences, and coefficient of determination.

HM | ED | |||||
---|---|---|---|---|---|---|

${\mu}_{E}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | ${\mathit{\mu}}_{\mathit{E}}$ | ${\mathit{\sigma}}_{\mathit{E}}^{\mathit{2}}$ | ${\mathit{R}}^{\mathit{2}}$ | |

N | $0.010$ | $3.718$ | $0.995$ | $-0.134$ | $62.845$ | $0.910$ |

R | $0.003$ | $2.819$ | $0.996$ | $-0.127$ | $24.728$ | $0.964$ |

G | $0.001$ | $1.790$ | $0.997$ | $-0.102$ | $11.243$ | $0.980$ |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Case, N.; Vitti, A. Reconstruction of Multi-Temporal Satellite Imagery by Coupling Variational Segmentation and Radiometric Analysis. *ISPRS Int. J. Geo-Inf.* **2021**, *10*, 17.
https://doi.org/10.3390/ijgi10010017

**AMA Style**

Case N, Vitti A. Reconstruction of Multi-Temporal Satellite Imagery by Coupling Variational Segmentation and Radiometric Analysis. *ISPRS International Journal of Geo-Information*. 2021; 10(1):17.
https://doi.org/10.3390/ijgi10010017

**Chicago/Turabian Style**

Case, Nicola, and Alfonso Vitti. 2021. "Reconstruction of Multi-Temporal Satellite Imagery by Coupling Variational Segmentation and Radiometric Analysis" *ISPRS International Journal of Geo-Information* 10, no. 1: 17.
https://doi.org/10.3390/ijgi10010017