Multimodal Satellite Image Time Series Analysis Using GAN-Based Domain Translation and Matrix Profile

Abstract

The technological development of the remote sensing domain led to the acquisition of satellite image time series (SITS) for Earth Observation (EO) by a variety of sensors. The variability in terms of the characteristics of the satellite sensors requires the existence of algorithms that allow the integration of multiple modalities and the identification of anomalous spatio-temporal evolutions caused by natural hazards. The unsupervised analysis of multimodal SITS proposed in this paper follows a two-step methodology: (i) inter-modality translation and (ii) the identification of anomalies in a change-detection framework. Inter-modality translation is achieved by means of a Generative Adversarial Network (GAN) architecture, whereas, for the identification of anomalies caused by natural hazards, we adapt the task to a similarity search in SITS. In this regard, we provide an extension of the matrix profile concept, which represents an answer to identifying differences and to discovering novelties in time series. Furthermore, the proposed inter-modality translation allows the usage of standard unsupervised clustering approaches (e.g., K-means using the Dynamic Time Warping measure) for mono-modal SITS analysis. The effectiveness of the proposed methodology is shown in two use-case scenarios, namely flooding and landslide events, for which a joint acquisition of Sentinel-1 and Sentinel-2 images is performed.

1. Introduction

The last few years have witnessed an increased technological development of remote satellite sensors and related derived products, e.g., data cubes [1], leading to an increased interest in developing new tools and methods for unsupervised satellite image time series (SITS) analysis and, more generally, Earth Observation (EO). The analysis of SITS covers not only the extraction of spatial information but reflects the spatio-temporal dynamic evolution of the scene and the objects composing the scene. There is a wide range of applications that benefit from SITS analysis, e.g., agriculture monitoring, monitoring of land resources, estimation of damage caused by natural hazards, and urban development. The open and free data access provided by many EO missions, e.g., the Copernicus program, allows for the collection of dense multimodal SITS. Although the analysis of multimodal SITS data shows great potential, especially when optical data needs to be complemented by Synthetic Aperture Radar (SAR) data, the exploitation of both spatio-temporal data and multimodal data remains a challenging task and requires the development of new deep learning techniques to advance the state-of-the-art in this domain.

Optical remote sensing images have been widely used for disaster assessment due to their ability to provide rich visual information [2,3,4]. Despite their incontestable benefits, the availability of optical remote sensing data represents a challenge for SITS analysis [5]. For instance, if the area under analysis is covered by clouds or severe weather conditions are present, optical sensors are not able to provide a representative area coverage. On the contrary, SAR sensors can deliver data with good coverage, in all-weather and all-time conditions, with minimal restrictions caused by clouds or other weather conditions. However, this comes at the cost of a decreased level of visual information content if compared to optical sensors. Fusing the benefits of rich visual representation offered by optical sensors with the all-weather and all-time coverage provided by SAR sensors, damage assessment caused by natural hazards can be improved.

Although the benefits of using multimodal EO data are undeniable (e.g., enhanced temporal resolution, multiple and complementary information sources, continuous monitoring), the variability in physical properties of the imaging sensors, including the differences in the statistical models associated with the heterogeneous modalities [6], transforms the multimodal SITS analysis task into a challenge. In a multimodal context, SITS analysis methods based on similarity measures, e.g., Dynamic Time Warping (DTW) [5,7,8], are difficult to be applied directly. One option is to acquire a time series of optical irregularly sampled data that spans over a long period of time. In this case, the DTW-based distance can be used to measure the similarity between two temporal series [5]. However, due to the possible unavailability of optical data, this approach cannot be applied in the case of disaster assessment, which requires a fine temporal resolution in order to capture the exact moment of occurrence of the undesired event.

Recently, inter-modality translation has received considerable attention in the remote sensing community. In this sense, several approaches for transforming images acquired by an EO sensor into images that follow the characteristics of another type of EO sensor have been developed for multimodal SITS analysis [9], bi-temporal change detection [3,10,11], and registration of multi-sensor images [12].

Considering the change detection task, monomodal change detection assumes that bi-temporal images are captured over the same area with the same type of sensor. Since the acquisition conditions are the same, change detection can be performed through a direct analysis of the difference image, which, in most cases, results in finding the optimal threshold separating the changed pixels in “change” versus “non-change” classes. This can be achieved using either a Bayesian framework [13] or a multilevel Lloyd-Max quantization technique [14]. On the contrary, multimodal change detection assumes that the bi-temporal image acquisition is performed either by two different satellite sensors (e.g., SAR/optical) or by the same sensor but with different imaging properties (e.g., different spectral bands). As already mentioned, due to the different statistical properties of the image modalities, methods based on similarity measures are difficult to be applied directly, as in the case of monomodal data. A possible solution is transforming the multimodal images into a common space in which the two images share the same statistical properties and in which the difference image can be directly computed by pixel-wise difference. Various methods have been used for the translation of multimodal images, e.g., multidimensional scaling representation [15], forward and backward homogeneous pixel transformations [16], unsupervised image regression [17], fractal projection framework [18], and, more recently, image-to-image translation based on deep neural networks. The fractal projection method described by Mignotte [18] can be included in the category of image-to-image translation methods since the pre-event image is reconstructed in the post-event modality by means of a dictionary of similar locations and affine transformations applied to these areas.

With the emergence of deep learning techniques, deep neural network architectures (e.g., symmetric convolutional coupling networks [19], code-aligned autoencoders [11], and fully connected neural networks [20]) have been used to express the relationships between modalities. Among the first attempts, neural network-based mapping functions were applied to feature representations extracted via stacked denoising autoencoders [20]. In a mixed image translation–common space approach, image translation was performed by aligning code layers of two autoencoders in a common latent space such that the output of an encoder can be used as input for two decoders, i.e., one yields the reconstruction of the image in the original domain, the other yields the transformation in the other domain [11].

A SAR-to-optical image translation methodology was presented as a solution in an attempt to reconstruct optical images and merge the benefits of all-weather and all-time capabilities of SAR sensors and the rich visual content of images captured by optical sensors [3]. Various architectures based on Generative Adversarial Networks (GANs) have been tested, and the experiments showed that CycleGAN [21] and Pix2Pix [22] methods maintained the best performance in terms of peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM), when comparing original optical images and reconstructed versions from SAR images.

In a recent attempt to reconstruct a target image from multi-temporal source images captured in a different modality than the modality of the target, Liu et al. [9] proposed a new temporal information-guided remote sensing image translation (TIRSIT) framework. The TIRSIT framework incorporates temporal information in the image-to-image translation process through a multi-stream guided translation network. In this regard, the extra-knowledge gained from the temporal information guides the learning process and reduces the cross-modal ambiguity in translation.

Although several techniques for analyzing mono-modal SITS have been successfully developed, e.g., Refs. [5,7], approaches for heterogeneous or multimodal SITS analysis remain limited. When dealing with multitemporal data, an end-to-end deep learning framework, called $M^3$ Fusion, was proposed in [23] to solve the land cover mapping task through the fusion of multimodal, multiscale, and multitemporal satellite images. In the context of land-use classification, it proves to be useful to distinguish among semantic classes according to their temporal profiles. $M^3$ Fusion [23] consists of a deep learning architecture that performs a dual information extraction at both spatial and temporal levels in an end-to-end learning process. More precisely, the patch-based feature descriptors are extracted from one SPOT 6/7 Very-High-Resolution (VHR) image via Convolutional Neural Networks (CNN), whilst the temporal information is derived from 34 Sentinel-2 High-Resolution (HR) images via Recurrent Neural Networks (RNN). The extracted features are sent to three classifiers, namely two auxiliary classifiers that work independently on each set of features and one classifier that is applied to the fused set of features. Fusing information at both feature and decision levels, $M^3$ Fusion achieves an overall accuracy rate of almost 91% for 13 semantic classes relative to the SPOT 6/7 image [23].

In this paper, we propose a new method, denoted as GAN-MP, for unsupervised SITS analysis in a multimodal setting. More precisely, considering a SITS composed of images captured by both SAR and optical sensors, an inter-modality translation network can be used to map all images to the optical domain. The translation from one modality to the other modality follows a GAN-based approach built upon the U-Net neural network architecture. An overview of the methodology of multi-source SITS analysis is presented in Figure 1. Once the modality translation is performed, SITS, comprised of original and translated images, can be analyzed by means of traditional techniques, e.g., unsupervised clustering of spatio-temporal evolutions using the DTW distance measure. Inter-modality translation is essential for bringing all the images from the multimodal SITS into the same domain, where known similarity measures can be applied. Apart from the DTW-based analysis of translated SITS, we investigate the usage of a modified matrix profile (MP) algorithm for determining, in a precise manner, the extent of the damage caused by a natural hazard. Initially used for motif discovery, anomaly detection, and clustering in time series, e.g., seismic signals, ECG, EEG, and power usage data [24], we show that the matrix profile concept can be extended to work with image-based time series, in particular, SITS. The rationale is that each temporal sequence in SITS has a particular matrix profile, which allows the discovery of abnormal events, e.g., landslides and flooding, and, thus, yields a general framework for time-series change detection techniques.

The rest of the paper is structured as follows. Section 2 focuses on the use-case scenarios and study area while offering an overview of the proposed methodology for multimodal SITS analysis. Section 3 presents the experimental results for two datasets of mixed Sentinel 1–Sentinel 2 images, capturing a landslide and a flood event. In Section 4, we discuss the results and draw a parallel with state-of-the-art monomodal SITS analysis methods, which are extended toward multimodal SITS analysis. Finally, Section 5 concludes the paper and outlines the main results.

2. Materials and Methods

2.1. Study Areas

Two use-case scenarios are considered for multimodal SITS analysis. In both cases, natural disasters occur, e.g., flooding and landslides. Damage assessment can be achieved by means of unsupervised SITS analysis. However, most of the images acquired by optical sensors (e.g., Sentinel-2) around the considered events are contaminated by clouds or affected by other weather conditions [25]. In order to solve the unavailability of optical images and to determine the exact moment and extent of the damage, the information from optical images (i.e., those that are available) can be complemented by information extracted from images captured by SAR sensors. In this way, visual information content acquired from optical images can guide the information that can be retrieved by means of SAR sensors in order to provide a precise unsupervised SITS analysis.

2.1.1. First Use-Case Scenario: Flood Detection and Monitoring

The SITS covering the flood detection and monitoring use-case scenario is captured in Buhera, Zimbabwe, near the Save River in the South East of Africa. The flooding event occurred between 14 March 2019 and 18 March 2019, after the dramatic footage of tropical cyclone Idai, which inundated numerous villages and destroyed many households, forcing Zimbabwe to introduce a state of disaster on 17 March 2019. The SITS, shown in Figure 2, is composed of Sentinel-1 and Sentinel-2 images and is part of a collection of similar events called SEN12-Flood [26]. The SITS consists of a sequence of 11 images of $512\times 512$ pixels, out of which two are optical images, and the rest are SAR images. For the Sentinel-1 images, the images are available in dual-polarization VV + VH, whilst, for the Sentinel-2 images, the visible (B2, B3, B4) bands are considered. In both cases, the bands/polarizations are characterized by 10 m spatial resolution.

2.1.2. Second Use-Case Scenario: Landslides

For the second use-case scenario, we considered a landslide that occurred on 15 May 2017 in Alunu, Valcea district, Romania. The landslide destroyed several households and farmlands, roads, and also a forest. The sequence of remote sensing images, shown in Figure 3, is comprised of three optical images and six dual-polarized SAR images of $791\times 691$ pixels. For the Sentinel-1 images, the images are available in dual-polarization VV + VH, whilst, for the Sentinel-2 images, the visible (B2, B3, B4) bands are considered. In both cases, the bands/polarizations are characterized by 10 m spatial resolution. Similar to the previous dataset [26], these multimodal and multitemporal images have been downloaded from the Copernicus Open-Access Hub [25] and have been preprocessed (including co-registration, radiometric calibration, Range Doppler Terrain Correction) using the Sentinel Application Platform (SNAP) [27], an open software platform provided by the European Space Agency (ESA).

The information for both use-case scenarios is synthesized in

Table 1. Use-case scenarios.

Event	Location	Monitoring Period	Number of Images in SITS	Image Dimensions
Flooding	BuheraZimbabwe	1 February 2019–17 April 2019	9 Sentinel-12 Sentinel-2	$512\times 512$
Landslide	AlunuRomania	18 December 2016–6 October 2017	6 Sentinel-13 Sentinel-2	$791\times 691$

.

2.2. Methodology Overview

Analyzing multimodal SITS is a challenging task. In this paper, we consider a two-stage SITS analysis. The first stage is based on the inter-modality translation of images in SITS, whereas the second stage focuses on detecting hazardous events using the newly introduced concept of matrix profile.

Including an inter-modality translation technique in the image processing pipeline for SITS analysis represents a solution for aligning the images in SITS to the same modality and, afterward, for using standard time series processing algorithms. As summarized in Figure 1, the proposed methodology is formed upon a GAN-based modality translation that maps the SAR images to the optical domain.

Generative Adversarial Networks, GANs, represent a generative framework for estimating new data through an adversarial process [22,28,29]. Two models are trained, namely a generative model that performs the translation from one domain to the other domain and a discriminative model, whose role is to estimate whether the sample comes from the training data or from the generated data. The procedure can be regarded as a game in which the generator tries to produce fake samples that perfectly resemble true samples and thus deceive the discriminator, whereas the discriminator aims to discover the differences between true and generated data.

2.3. GAN-Based Modality-to-Modality Translation

The goal of modality-to-modality translation is to learn inter-domain mapping functions. In our case, the domains are composed of images acquired by a SAR sensor (domain ${\mathcal{D}}_{\mathrm{SAR}}$) and by an optical sensor (domain ${\mathcal{D}}_{\mathrm{Opt}}$), respectively. In the following, we denote by $x_{\mathrm{SAR}}$ the image acquired by a SAR sensor and by $x_{\mathrm{Opt}}$ the image acquired by an optical sensor.

In this work, a model consisting of two mapping functions $F:{\mathcal{D}}_{\mathrm{SAR}}\to {\mathcal{D}}_{\mathrm{Opt}}$ and $G:{\mathcal{D}}_{\mathrm{Opt}}\to {\mathcal{D}}_{\mathrm{SAR}}$, allows the reconstruction of images from one domain to the other. The two generative functions map one domain into the other and thus, the mappings can be applied either to original images or generated ones. More precisely, F can be applied on original SAR images and also on GAN-generated SAR images (i.e., images obtained from optical images by applying the mapping function G). As shown in Figure 4, in addition to the inter-domain mapping functions, we introduce two discriminators, $D_{\mathrm{SAR}}$ and $D_{\mathrm{Opt}}$, for each domain. The role of the discriminators is to distinguish between fake and generated regions.

Considering the structure of the modality-to-modality translation model presented in Figure 4, the objective is composed of three types of loss functions, namely, translation loss, cycle-consistency loss, and adversarial loss. The loss functions will be detailed in the following subsections.

2.3.1. Translation Loss

Considering the task of inter-modality translation, assume that the same location is captured by two sensors characterized by different physical properties, e.g., SAR and optical sensors. The acquired images are denoted by $x_{\mathrm{SAR}}\in {\mathcal{D}}_{\mathrm{SAR}}$ and $x_{\mathrm{Opt}}\in {\mathcal{D}}_{\mathrm{Opt}}$. The training set is composed by images reflecting various locations for which pairs of images $(x_{\mathrm{SAR}},x_{\mathrm{Opt}})$ are available. A correct translation from one domain to another is achieved if:

$$\begin{array}{cc}\hfill F\left(x_{\mathrm{SAR}}\right)& \approxeq x_{\mathrm{Opt}}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill G\left(x_{\mathrm{Opt}}\right)& \approxeq x_{\mathrm{SAR}}\hfill \end{array}$$

(2)

This constraint is expressed through the translation loss term, which tries to map images composing a $(x_{\mathrm{SAR}},x_{\mathrm{Opt}})$ pair across modalities:

$${\mathcal{L}}_t(F,G)={\mathbb{E}}_{\mathrm{Opt}}\left[{\parallel F\left(x_{\mathrm{SAR}}\right)-x_{\mathrm{Opt}}\parallel }_2^2\right]+{\mathbb{E}}_{\mathrm{SAR}}\left[{\parallel G\left(x_{\mathrm{Opt}}\right)-x_{\mathrm{SAR}}\parallel }_2^2\right]$$

(3)

2.3.2. Cycle-Consistency Loss

In order to prevent a possible contradiction between inter-domain mapping functions F and G, Zhu et al. [21] introduced the cycle-consistency loss. Inspired by this approach, in an inter-domain scenario, we impose:

$$\begin{array}{cc}\hfill G\left(F\left(x_{\mathrm{SAR}}\right)\right)& \approxeq x_{\mathrm{SAR}}\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill F\left(G\left(x_{\mathrm{Opt}}\right)\right)& \approxeq x_{\mathrm{Opt}}\hfill \end{array}$$

(5)

which leads to writing the cycle-consistency loss as:

$${\mathcal{L}}_c(F,G)={\mathbb{E}}_{\mathrm{SAR}}\left[{\parallel G\left(F\left(x_{\mathrm{SAR}}\right)\right)-x_{\mathrm{SAR}}\parallel }_2^2\right]+{\mathbb{E}}_{\mathrm{Opt}}\left[{\parallel F\left(G\left(x_{\mathrm{Opt}}\right)\right)-x_{\mathrm{Opt}}\parallel }_2^2\right]$$

(6)

The cycle-consistency constraint enforces G to act as an inverse mapping for F and vice-versa.

2.3.3. Adversarial Loss

Adversarial networks are discriminative models that learn to determine whether a sample comes from a generated distribution or a true data distribution [28]. Unlike the approach in [28], in this paper, we adopt a least squares loss function for the discriminator instead of the logarithmic one [30,31]. The discriminator should be able to distinguish generated regions (i.e., “fake”) from real regions. We express the adversarial loss as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_d(D_{\mathrm{SAR}},D_{\mathrm{Opt}},F,G)& =\hfill \\ \hfill & {\mathbb{E}}_{\mathrm{SAR},\mathrm{Opt}}\left[{\parallel D_{\mathrm{SAR}}\left({\tilde{x}}_{\mathrm{SAR}}\right)-\mathbf{M}\parallel }_2^2\right]\hfill \\ \hfill & +{\mathbb{E}}_{\mathrm{SAR},\mathrm{Opt}}\left[{\parallel D_{\mathrm{Opt}}\left({\tilde{x}}_{\mathrm{Opt}}\right)-\mathbf{M}\parallel }_2^2\right]\hfill \end{array}$$

(7)

where ${\tilde{x}}_{\mathrm{SAR}}$ and ${\tilde{x}}_{\mathrm{Opt}}$ denote images under CutMix transformations [32]:

$$\begin{array}{cc}\hfill {\tilde{x}}_{\mathrm{SAR}}& =\mathbf{M}\odot x_{\mathrm{SAR}}+(\mathbf{1}-\mathbf{M})\odot G\left(x_{\mathrm{Opt}}\right)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\tilde{x}}_{\mathrm{Opt}}& =\mathbf{M}\odot x_{\mathrm{Opt}}+(\mathbf{1}-\mathbf{M})\odot F\left(x_{\mathrm{SAR}}\right)\hfill \end{array}$$

(9)

with ⊙ representing element-wise multiplication. $\mathbf{M}={\left[M_n\right]}_n$ is a binary mask indicating if the pixel n comes from a real region ($M_n=1$) or a generated region ($M_n=0$). In our setup, CutMix transformations replace rectangular regions from an original image with the corresponding regions taken from the generated image in different mix proportions. This allows a natural augmentation strategy for training the discriminators under limited data, which typically leads to discriminator overfitting [33].

The overall objective is thus:

$$\mathcal{L}(F,G,D_{\mathrm{SAR}},D_{\mathrm{Opt}})={\omega }_t{\mathcal{L}}_t(F,G)+{\omega }_c{\mathcal{L}}_c(F,G)+{\omega }_d{\mathcal{L}}_d(D_{\mathrm{SAR}},D_{\mathrm{Opt}},F,G)$$

(10)

2.3.4. U-Net-Based Inter-Modality Image Translation

In order to achieve inter-domain image translation, we employ a fully convolutional encoder-decoder scheme, more precisely, the popular U-Net architecture, which was first introduced for biomedical image segmentation [34]. Following an information theory-based parallel, the input is passed, during encoding, through a sequence of layers that successively downsamples the input until a bottleneck layer is met. At this point, the process is reversed, and decoding is performed. The bottlenecked information flowing through the decoder part of the “U”-shaped structure is concatenated, at each level, with mirrored information from the encoder’s layers. The information at each layer is mirrored by means of skip connections added between layer i and layer $n-i$, n being the total number of layers. In this manner, low-level information between the input and output images is translated through these direct connections instead of being discarded by the bottlenecked process. Thus, U-Net-based modality translation brings several advantages, i.e., the inherent property of aligning the structure in the input to the structure in the output [22], the increased stability achieved by means of shortcut connections [9], and also its potential of dealing with a different number of spectral bands at input and output. The last property is particularly useful when dealing with SAR-to-optical image translation.

Considering the above-mentioned advantages, the U-Net architecture is adopted for both the generative functions, i.e., F and G, and the discriminative functions, i.e., $D_{\mathrm{SAR}}$ and $D_{\mathrm{Opt}}$. The standard U-Net architecture consists of a contracting path and an expansive path connected through a bridge of convolutional layers [34]. The contracting path is composed of four blocks, each with two $3\times 3$ convolutional layers, a ReLU activation function, and a maxpooling layer. At each downsampling step, the number of channels is doubled, i.e., the number of the output channels is 64 for the first block, 128 for the second block, 256 for the third block, and 512 for the fourth block [34]. The bridge is formed by two $3\times 3$ convolutional layers with 1024 output channels each, followed by a ReLU activation function. The expansive path contains the same number of blocks as the contracting path, each with two $3\times 3$ convolutional layers, followed by a ReLU activation function. The number of the output channels characterizing each block in the expansive path is halved at each step, whereas the connection between successive blocks is characterized by $2\times 2$ up-convolutions. The long skip connections mirror, by concatenation, the information from the blocks in the contracting path to the corresponding blocks in the expansive path. In order to match the number of input and output channels with the requirements of our setup, we slightly modify the standard U-Net architecture. Specifically, if $C_{\mathrm{SAR}}$ and $C_{\mathrm{Opt}}$ represent the number of bands for the SAR and optical images, respectively, the generative U-Net function F has $C_{\mathrm{SAR}}$ input channels and $C_{\mathrm{Opt}}$ output channels, whereas the generative U-Net function G has $C_{\mathrm{Opt}}$ input channels and $C_{\mathrm{SAR}}$ output channels. Similarly, the discriminators follow a U-Net architecture, with $C_{\mathrm{SAR}}$ and $C_{\mathrm{Opt}}$ input channels (i.e., depending on the domain) and one output channel. Since the translation is performed from the SAR domain to the optical domain, the resulting SITS is composed of original and generated optical images, all being characterized by a number of $C_{\mathrm{Opt}}$ spectral bands.

2.4. Identification of Anomalies in a Change Detection Framework for Multimodal SITS

In the previous section, the multimodal SITS has been translated into a SITS formed by original optical images and “fake” optical images, i.e., images obtained from SAR images by applying a modality-to-modality translation. In this way, the resulting SITS are composed of temporal sequences of length N, and each sample in the sequence has $C_{\mathrm{Opt}}$ components (i.e., the number of spectral bands of the optical sensor). N represents the number of images that form the SITS, i.e., original or generated. In the following, we propose an approach toward the identification of abnormal events under a more general framework for multimodal time-series change detection.

2.4.1. Detection of Abnormal Events in SITS

We adopt an all-pairs-similarity-search (or similarity join) approach for the identification of time series differences. All-pairs similarity search algorithms are considered to be very competitive as novelty and anomaly detectors [24,35], whereas in terms of motif and novelty discovery tasks, one of the fastest algorithms used for finding similarity joins between time series subsequences is the matrix profile, described in [24].

Let us consider a time series x of length N, i.e., the time series x is composed of N pixel arrays at the same position in the SITS. The similarity joins are searched in an all-subsequences set of the time series, which is composed of all the subsequences of a given length m, which are obtained by sliding a window over the time series x. Specifically, given a query of length m, the nearest neighbor (NN) is searched among all the possible subsequences of x of length m. In the following, we denote by $x_i^{i+m-1}=\{x_i,x_{i+1},\dots ,x_{i+m-1}\}$ the subsequence of length m starting at sample $x_i$ and by $x_n^{n+m-1}=\{x_n,x_{n+1},\dots ,x_{n+m-1}\}$ the subsequence of length m starting at sample $x_n$. The similarity join set records, in an ordered manner, pairs of the form $(x_i^{i+m-1},x_n^{n+m-1})$, where $x_n^{n+m-1}$ is the nearest neighbor of $x_i^{i+m-1}$ in terms of Euclidian distance.

Apart from being fast in discovering similarity joins, the matrix profile algorithm is parameter-free. With a slight modification of the initial concept of matrix profile [24], we can extend the concept to C-dimensional signals, where C denotes the number of spectral bands in the newly created SITS (i.e., containing original and generated optical images with $C=C_{\mathrm{Opt}}$ spectral bands). More specifically, for each pair of the similarity join set, we store the squared Euclidian distance between the C-dimensional temporal subsequence of length m, $x_i^{i+m-1}$, and the corresponding nearest neighbor, i.e., $x_n^{n+m-1}$:

$$d_i^2=\sqrt{\sum _{c=0}^{C-1}\sum _{k=0}^{m-1}{(x_{i+k}\left[c\right]-x_{n+k}\left[c\right])}^2}$$

(11)

where $x_i\left[c\right]$ denotes the value that corresponds to channel c of sample $x_i$.

The matrix profile accompanies each spatio-temporal sequence in SITS, and a relatively high value in the matrix profile of a sequence corresponds to a difference or anomaly. More precisely, a subsequence in the original temporal sequence that is unique in its shape is characterized by a high value of $d_i$ [24]. Specifically, a time series difference is a subsequence that is most dissimilar to its nearest neighbor subsequence. Therefore, the identification of anomalies, or differences, in a time series x reduces to recording the highest values in the corresponding matrix profile:

$$d_{max}=\underset{i}{max}{d}_i^2$$

(12)

The $d_{max}$ value is an indicator of the relevance of an anomalous event, whereas the examination of the matrix profile reveals the change point or the moment an anomaly occurs. This can be observed in Figure 5, where two time series are shown, one corresponding to a “normal” time series, whilst the second one is extracted from a region where an anomalous event (i.e., landslide) occurred.

2.4.2. Unsupervised Change Detection in a Maximum-Likelihood Estimation Framework

In order to identify the anomalies in SITS, we form a new image from the maximum matrix profile values $d_{max}$ extracted for each pixel. Intuitively, the higher the maximum value $d_{max}$ in the matrix profiles of spatio-temporal sequences, the more significant the difference/anomaly is. In this regard, the optimal threshold that separates an abnormal event from the rest of the spatio-temporal events in SITS can be determined following an Expectation-Maximization (EM)-based thresholding technique [36].

As shown in Figure 6, the histogram over the maximum matrix profile values follows a mixture of two Gaussian distributions, one corresponding to regions containing anomalies, whereas the other to unchanged regions. Thus, assuming that the maximum value $d_{max}$ of a matrix profile that accompanies a spatio-temporal sequence is taken from a mixture of two Gaussian distributions, $\mathcal{N}({\mu }_a,{\sigma }_a^2)$ (i.e., corresponding to regions containing anomalies) and $\mathcal{N}({\mu }_u,{\sigma }_u^2)$ (i.e., corresponding to unchanged regions), we can write the posterior probability as:

$$p\left(d_{max}\right|\theta )={\pi }_a\mathcal{N}({\mu }_a,{\sigma }_a^2)+{\pi }_u\mathcal{N}({\mu }_u,{\sigma }_u^2)$$

(13)

where $\theta =\{{\pi }_a,{\mu }_a,{\sigma }_a,{\pi }_u,{\mu }_u,{\sigma }_u\}$ are the model parameters. Specifically, ${\pi }_a$ and ${\pi }_u$ are the mixture probabilities such that ${\pi }_a+{\pi }_u=1$, whilst $({\mu }_a,{\sigma }_a)$ and $({\mu }_u,{\sigma }_u)$ are the mean and standard deviations that correspond to the two Gaussian distributions.

Following the Maximum-Likelihood Estimation (MLE) framework, the goal is to estimate the model parameters such that the log-likelihood is maximized:

$$L\left(\theta \right)=lnp\left(d_{max}\right|\theta )$$

(14)

The Expectation-Maximization technique allows the estimation of the aforementioned model parameters in an iterative manner (i.e., from iteration t to next iteration $t+1$) [13]:

$$\begin{array}{ccc}\hfill {\pi }_k^{(t+1)}& =& \frac{{\displaystyle \sum _{d_{max}}}{\zeta }_k^{\left(t\right)}\left(d_{max}\right)}{HW}\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill {\mu }_k^{(t+1)}& =& \frac{{\displaystyle \sum _{d_{max}}}{\zeta }_k^{\left(t\right)}\left(d_{max}\right)·d_{max}}{{\displaystyle \sum _{d_{max}}}{\zeta }_k^{\left(t\right)}\left(d_{max}\right)}\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill {\sigma }_k^{(t+1)}& =& {\left(\frac{{\displaystyle \sum _{d_{max}}}{\zeta }_k^{\left(t\right)}\left(d_{max}\right)·{\left(d_{max}-{\mu }_k^{\left(t\right)}\right)}^2}{{\displaystyle \sum _{d_{max}}}{\zeta }_k^{\left(t\right)}\left(d_{max}\right)}\right)}^{\frac{1}{2}}\hfill \end{array}$$

(17)

where $k\in \{a,u\}$, H and W are the height and the width of the images in SITS, and ${\zeta }_k^{\left(t\right)}\left(d_{max}\right)$ values are derived as:

$${\zeta }_k^{\left(t\right)}\left(d_{max}\right)=\frac{{\pi }_k^{\left(t\right)}\mathcal{N}\left(d_{max}\right|{\mu }_k^{\left(t\right)},{\sigma }_k^{\left(t\right)})}{{\sum }_{k^{\prime }\in \{a,u\}}{\pi }_{k^{\prime }}^{\left(t\right)}\mathcal{N}\left(d_{max}\right|{\mu }_{k^{\prime }}^{\left(t\right)},{\sigma }_{k^{\prime }}^{\left(t\right)})}.$$

(18)

After the estimation of the $\theta$ model parameters, the optimal threshold $T_o$ that separates the two classes (i.e., anomaly and unchanged) are determined from the equality:

$${\pi }_a\mathcal{N}\left(T_o\right|{\mu }_a,{\sigma }_a)={\pi }_u\mathcal{N}\left(T_o\right|{\mu }_u,{\sigma }_u)$$

(19)

which as shown in [13] leads to a quadratic equation in $T_o$. The equation has two possible solutions, but only one solution is between ${\mu }_a$ and ${\mu }_u$. This solution represents the optimal threshold that outlines the anomaly in SITS, i.e., if $d_{max}$ is greater than the threshold $T_o$, then $d_{max}$ corresponds to an anomaly.

3. Experiments

3.1. Inter-Modality Translation

The inter-modality translation model was trained on 500 pairs of Sentinel-1 and Sentinel-2 patches with dimensions of $50\times 50$ pixels randomly selected from the datasets. However, since the changes occurring between two different moments of time may distort the learning process, we selected images that were captured at close temporal moments and that were not interfering with the areas affected by the hazardous events. As mentioned, the loss function incorporates a translation loss, a cycle-consistency loss, and an adversarial loss term, with corresponding weights ${\omega }_t=3$, ${\omega }_c=2$, and ${\omega }_d=1$, respectively. In addition, we added a regularization term with a weight decay of 0.001, which controls the magnitude of the network parameters and reduces overfitting. During the training procedure, we used the Stochastic Gradient Descent (SGD) optimization algorithm, with momentum set to 0.9. The model was trained for 250 epochs, with a batch size of 8 and a learning rate of ${10}^{-5}$. In Figure 7 and Figure 8, we present the results of the GAN-based inter-modality translation for the two datasets of Sentinel-1 and Sentinel-2 images described in Section 2.1.

3.2. Performance Evaluation for Multimodal Time-Series Change Detection

In order to detect anomalies caused by natural disasters (e.g., flooding, landslide), we use a time-series change detection framework based on a modified matrix profile concept. Since the detection of a natural disaster at a certain moment of time involves the analysis of a pre-event image and a post-event image, we set the length of the subsequences m to 2. This assumption leads to quantifying whether regions captured at consecutive moments of time experience a major event or not. If a natural disaster occurs, this anomaly will be reflected by the high values in the matrix profile accompanying the corresponding time series in SITS. The optimal threshold in the matrix of maximum matrix profile values is determined via the EM algorithm, whose initialization is performed by using the K-Means algorithm.

The performance of the proposed algorithm, GAN-MP, for multimodal time-series change detection is measured in terms of overall accuracy ($OA$), true negative rate ($TNR$), true positive rate ($TPR$), and F-score, which were computed as:

$$\begin{array}{cc}\hfill OA& =\frac{TP+TN}{TP+TN+FP+FN}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill TNR& =\frac{TN}{TN+FP}\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill TPR& =\frac{TP}{TP+FN}\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill F-\mathrm{score}& =\frac{TP}{TP+\frac{1}{2}·(FP+FN)}\hfill \end{array}$$

(23)

where $TP,TN,FP$, and $FN$ stand for true positives, true negatives, false positives, and false negatives, respectively.

4. Discussion

The projections from the SAR to the optical domain yield a mono-modal SITS, containing both original and generated optical images. Using the U-Net architecture for inter-modality translation in a GAN-based setup, the generated optical images will have the same number of spectral bands and also similar characteristics to the original optical images. The proposed approach for multimodal SITS analysis, GAN-MP, is based on the matrix profile concept and allows the retrieval of anomalies in heterogeneous SITS. In Figure 9a and Figure 10a, the matrix of $d_{max}$ values indicates the regions where an anomaly or hazardous event occurred, in the sense that, as expected, these regions exhibit higher values of $d_{max}$. Comparing the qualitative results presented in Figure 9b and Figure 10b to the corresponding ground-truth images, the EM algorithm effectively delimits the affected regions by determining the optimal threshold that is applied over the $d_{max}$ values. From a quantitative point of view, the performance results are summarized in

Table 2. Performance evaluation and comparisons with other approaches.

Event	Method	Overall Accuracy	True Positive Rate	True Negative Rate	F-Score
Flooding	Proposed GAN-MP	96.59%	92.29%	96.93%	0.79
	DTW-EM [36]	95.26%	56.21%	98.37%	0.64
	ACE-Net [10]	61.64%	65.38%	61.34%	0.20
Landslide	Proposed GAN-MP	97.18%	80.68%	97.46%	0.49
	DTW-EM [36]	93.05%	85.75%	93.18%	0.29
	ACE-Net [10]	80.66%	87.67%	80.53%	0.13

, and, in both cases, the proposed technique achieves an overall accuracy greater than 96%.

Most of the SITS analysis methods, e.g., unsupervised clustering of SITS, are designed for mono-modal remote sensing images. Since the acquisitions in a mono-modal setup are performed using the same sensor, the spatio-temporal evolutions can be clustered using standard methods, e.g., the DTW distance measure can be used in conjunction with the K-Means algorithm, yielding an unsupervised clustering of spatio-temporal evolutions in SITS [5,7]. Moreover, the mono-modal SITS can be analyzed using standard techniques, e.g., the Latent Dirichlet Allocation (LDA) model to extract latent information from the scene or DTW and EM to retrieve similar evolution patterns [36]. In the case of multimodal SITS, the application of these mono-modal SITS analysis techniques is possible if a projection of the images from another modality to the current modality of analysis is performed beforehand.

In Figure 9d and Figure 10d, we show the results obtained by applying the DTW-based K-means clustering method [5,7] on SITS formed by original and translated optical images in both use-case scenarios, e.g., flooding and landslide. The number of clusters has been set using the Elbow method, which determines the optimal number of clusters by analyzing the shape of the cost function (or inertia) [37]. In the case of K-means clustering, the cost function is the sum of within-cluster distances, i.e., distances from points to the closest cluster centroids. Considering that the K-means for SITS clustering uses the DTW distance measure, we replace the Euclidean distance in the cost function with the DTW distance measure. The cost function with respect to the number of clusters is shown in Figure 11. In the case of the flooding event, the Elbow point (i.e., the point after which the decrease in the cost function with the number of clusters becomes slow or linear) is 4, whereas, in the case of the landslide event, the optimal number of clusters is 5. With these parameters set, Figure 9d and Figure 10d represent the DTW-based K-Means clustering for the two use-case scenarios. Although both the flooding and the landslide events form a distinct category of spatio-temporal evolutions, the unsupervised clustering of spatio-temporal evolutions does not provide information regarding either the possible occurrence of an anomaly or the specific class incorporating the anomaly. In this regard, the retrieval of spatio-temporal events that are similar to a given query, e.g., a single spatio-temporal sequence provided by the user, can be performed using the DTW distance measure computed between all the spatio-temporal sequences in the SITS and the given query [36]. The separation between similar and non-similar patterns is obtained using the EM technique, as shown in Figure 9e and Figure 10e [36].

Apart from the above-mentioned methods, we investigate adversarial cyclic encoder networks, which have been recently proposed in [10] for image translation in an unsupervised multimodal change detection framework. During the experiments, we follow the same architecture and setup as in the original paper, whereas the threshold over the distance image is automatically determined via Otsu’s method [10]. Considering the first and last images in SITS as the pre-event and the post-event images, the delimitation of the landslide event is clear in Figure 10f. In the flooding use-case scenario, the delimitation of the event is less accurate.

Finally, Table 2 shows the results achieved by the proposed technique and detailed quantitative comparisons for the two considered multimodal SITS. In both use-case scenarios, the existence of multitemporal information has an important impact over the performance of the algorithms. In this regard, the results show that considering only pre-event and post-event images leads to an increased number of false alarms, e.g., Figure 9f and Figure 10f.

5. Conclusions

This paper describes a new approach toward unsupervised multimodal SITS analysis. SITS composed of images acquired by different sensors are brought into the same modality domain by means of an inter-modality translation technique, which follows a GAN-based architecture. Working in the same domain allows the usage of standard techniques for analyzing spatio-temporal evolutions in SITS, e.g., DTW-based K-means clustering. In order to detect anomalies and analyze the consequences of natural hazards, we introduce a similarity search methodology based on the extension of the matrix profile concept to SITS analysis. Finally, the identification of regions affected by natural hazards is based on an optimal thresholding approach defined in an MLE framework. Experiments were conducted on two datasets that contain multimodal and multitemporal remote sensing images in two use-case scenarios, namely, flooding and landslide events. The experimental results show the effectiveness of the proposed framework in terms of the delimitation of the regions affected by natural hazards, which further allows a better assessment of the damage caused by these natural events. In this regard, the performance in terms of overall accuracy exceeds 96 % in both use-case scenarios. Moreover, the experimental results establish the importance of multitemporal information when dealing with anomaly detection in SITS, whilst the matrix profile concept allows the identification of the change point corresponding to the occurrence of a hazardous event.