Attention to Both Global and Local Features: A Novel Temporal Encoder for Satellite Image Time Series Classification

Abstract

Satellite image time series (SITS) classification is a challenging application concurrently driven by long-term, large-scale, and high spatial-resolution observations acquired by remote sensing satellites. The focus of current SITS classification research is to exploit the richness of temporal information in SITS data. In the literature, self-attention mechanism-based networks, which are capable of capturing global temporal attention, have achieved state-of-the-art results in SITS classification. However, these methods lack attention to local temporal information, which is also significant for SITS classification tasks. To explore the potential of different scales of temporal information in SITS data, a global–local temporal attention encoder (GL-TAE) is proposed in this paper. GL-TAE has two submodules set up in parallel, one of which is a lightweight temporal attention encoder (LTAE) for extracting global temporal attention and the other is lightweight convolution (LConv) for extracting local temporal attention. Compared with methods exploring global-only or local-only temporal features, the proposed GL-TAE can achieve better performance on two public SITS datasets, which proves the effectiveness of hybrid global–local temporal attention features. The experiments also demonstrate that GL-TAE is a lightweight model, which achieves the same performance as other models but with fewer parameters.

1. Introduction

With the availability of free remote sensing satellite images with regular temporal intervals, satellite image time series (SITS) data are becoming increasingly accessible. SITS plays an increasingly important role in land cover mapping due to the fact that single temporal data are unable to distinguish land cover types with similar structural and spectral characteristics and SITS has a wealth of temporal information that characterizes periodic changes in land cover to help us better classify different types of land cover. Many studies have shown that SITS outperforms single temporal data for land cover classification [1,2].

Various approaches have been developed to cope with the SITS classification, and some classical machine learning algorithms used for single temporal can be adapted to SITS classification, such as support vector machine (SVM) [3] and random forest (RF) [4]. References [5,6,7] adopted SVM for Sentinel or Landsat SITS-based cropland mapping. RF is widely used for land cover and crop type mapping in [8,9,10,11] due to its robustness and convenient operability. Although SVM and RF are theoretically interpretable and reliable, they have some limitations that are difficult to break through, the direct one of which is that they cannot handle input data with unequal dimensions. Due to the differences in the revisit periods of remote sensing satellites in different geographical areas, SITS with unequal temporal acquisitions is very common, especially for large area land cover-mapping tasks. One direct strategy is to flatten SITS into vectors with the same dimensions for SVM or RF, but it means that there have to be different degrees of information losses for different time series in SITS. Another drawback of SVM and RF is that they lack the mining of sequential information in SITS. Therefore, methods dedicated to various length time series analyses have received attention from researchers in the field of SITS classification.

One classical method compatible with input data with unequal lengths is dynamic time warp (DTW), which is originally used for time series similarity measures [12]. DTW is an elastic distance measure of time series with unequal lengths and it outperforms many classical distance measures, such as the Manhattan distance, Euclidean distance, and Pearson’s correlation [13]. Reference [14] first introduced DTW for SITS analysis and the results have proven that DTW is able to deal with irregularly sampled time series and allows the comparison of time series with different temporal lengths. DTW is capable of achieving good classification results when there are no sufficient training samples; thus, it is often used in clustering algorithms [15,16]. Reference [17] proposed a time-weighted dynamic time warping (TW-DTW), which has important improvements for the DTW-based method in SITS classification, which includes a temporal weight for characterizing the seasonality of land cover types. Many studies [18,19,20] have investigated the classification performance of TW-DTW compared with DTW, and the results show that the improvement in time weighting is significant. Although DTW-based methods have successfully been adopted in the SITS classification, the time-consuming computation of the DTW distance prevents its utility for big volumes of SITS datasets. Furthermore, this distance-based method performs poorly for SITS classification with many land cover types and unbalanced training samples.

The difficulty of SITS-based land cover classification lies in how to fully exploit the rich temporal information of SITS to generate discriminative features. The aforementioned traditional machine learning methods are insufficient to model temporal features from SITS. In the last decade, with the development of deep learning, many algorithms based on artificial neural networks have been proposed for SITS classification, and deep temporal features extracted from these deep neural networks have proven to be more effective. These deep learning methods are mainly categorized into three types according to which backbone neural network is employed to extract the temporal features: the recurrent neural network (RNN), convolutional neural network (CNN), and transformer with self-attention mechanisms.

RNN is the earliest DL method used for SITS classification [21,22]; it was initially designed to deal with sequence analysis [23] and is well-developed in NLP [24]. RNNs sequentially input each time step of a time series into recurrent cells and they are able to focus on information from a certain forward temporal range when extracting temporal features. Thus, RNNs can be viewed as local temporal attention-based methods. A recurrent cell is the basic unit of a RNN; long short-term memory (LSTM) and a gated recurrent unit (GRU) are the most widely used cells in RNN-based methods for SITS classifications [25,26,27,28]. A unidirectional recurrent cell pays attention to a unidirectional temporal range for each temporal acquisition in SITS when constructing new feature representations, while a bidirectional recurrent cell is capable of combining both forward and backward attention to encode the time series, which is more effective in some cases [29,30,31].

CNN-based methods—different from RNN-based methods—extract temporal features from SITS, focusing on local temporal information with a fixed symmetric range known as the perceptive field of convolution. Reference [32] proposed a temporal convolutional network (TempCNN), which applies multiple layers of 1D vanilla convolution for pixel-based SITS classification. Reference [33] combined 1D temporal convolutions via trainable attention to exploit information from object-based SITS for land cover classification. Reference [34] compared TempCNN and a generic temporal convolutional network (TCN) [35], which stacked causal convolutions with dilatation at powers of 2, and concluded that they could achieve comparable results. Reference [36] investigated more CNN-based methods, in addition to TempCNN, such as Time-CNN [37], multi-channel deep convolutional neural networks (MCDCNNs) [38,39], and InceptionTime [40], for SITS classification.

Local attention tends to capture information from temporal neighborhoods for each position of the time series, but global attention tends to capture information from all time steps for each position in the time series. Recently, with the remarkable achievements of transformer [41] in NLP [42,43,44,45] and CV [46,47,48], methods based on self-attention mechanisms, which extract global attention from sequence data have also been introduced into SITS classification and demonstrated to be superior to RNN/CNN-based methods. In [49], the authors explored the application of a pre-trained transformer in SITS classification. In [50], the authors adopted the encoder architecture of the transformer, in which the stage of the word-embedding was discarded because SITS is in a continuous space of spectral reflectance values, and global maximum pooling was applied to dimensionally reduce the representation of the last encoder layer. A temporal attention encoder (TAE) [51] simplified the architecture of [50] and defined a single master query to encode an entire time series into a single embedding and summarized the global temporal information. A lightweight temporal attention encoder (LTAE) [52] applied multi-head attention along the channel dimension and implemented the master query with query-as-a-parameter, which is more lightweight than TAE and better at understanding the global temporal information from the time series. LTAE was used in conjunction with U-Net [53] for semantic/panoptic segmentation of SITS [54]. All experiments from [50,51,52] demonstrated that such global attention is capable of focusing crucial positions by considering all temporal data within the input time series.

For DL-based SITS classification, attention is the most critical factor affecting the classification performance and existing methods are only capable of capturing single attention. Although global attention is able to automatically capture key phenological periods, it lacks sensitivity to changes in fixed periods, which is accurately captured by local attention. In theory, combining the attention of different sales is able to make classification features more discriminative. Reference [55] investigated a temporal encoder with two branches for SITS classification, in which one CNN branch has local attention and one RNN branch has forward local attention. The results of the experiments have shown that this two-branch architecture outperforms RNN-only and CNN-only architecture, which illustrates that hybrid attention is better than single attention. Thus, combining global attention with local attention has promising prospects from the perspective of improving the performance of SITS classification. For the design of a deep learning model, a straightforward idea is to make the model capable of obtaining both global and local features. LTAE is a state-of-the-art global attention-based and lightweight model. CNN-based methods are more focused on local temporal information than RNN-based methods [32,35] and are not bothered by the problems of vanishing and exploding gradients, where RNNs often occur. A lightweight convolution (LConv) is proposed in [56], which is capable of extracting local information with much fewer parameters. Thus, we propose a global–local temporal attention encoder (GL-TAE) to implement the idea. The key contributions of this paper are as follows.

1.

The proposed GL-TAE is an early attempt to extract hybrid global–local temporal features for SITS classification, adopting self-attention mechanisms for global attention extraction and convolution for local attention extraction.

2.

The proposed GL-TAE is a lightweight model. We selected and set two lightweight submodules (LTAE and LConv) in parallel and extracted global and local temporal features, respectively, in combination with dimensional split strategies, which further reduce the parameter numbers under the same hyperparameter settings.

2. Methodology

2.1. Overview

The overall architecture of our proposed GL-TAE is shown in Figure 1. Let us denote the input time series as x of size $T\times d_{in}$ (T is the number of temporal acquisitions and $d_{in}$ is the number of channels of input data). x is firstly transformed into size $T\times d_{model}$ by a channel feature enhancement layer composed of a 1D convolution with the LayerNorm that is also used in LTAE and TAE. Then it is processed by a positional encoding layer for catching sequential information of the time series according to the following equation:

$$PE(t,i)=sin(t/{\tau }^{2\times \lfloor i/2\rfloor /d_{model}}+\frac{\pi }{2}\times \left(imod2\right))$$

(1)

where t refers to the time step in the time series, i refers to the i-th channel dimension, and $\tau$ is a constant set to 1000. The core module of our proposed encoder is the global–local attention-based temporal feature extractor, which combines LTAE (the latest self-attention mechanism variant for extracting global temporal features) with LConv for extracting local temporal features. In addition, there is a dimension split layer and a dimension concatenation layer before and after the module, respectively, both of which operate on the channel dimension. In this global–local attention extractor, the positional encoded time series is transformed into the $d_{model}$ size. The final multilayer perceptron (MLP) outputs the classification feature of size $d_{out}$.

2.2. Self-Attention Mechanism

Reference [41] proposed the self-attention mechanism and applied it to the neural machine translation. The basic unit of the self-attention mechanism is the scaled dot-product attention (SDPA), which consists of the following steps:

1.

Compute a triplet of the query–key–value $q^t,k^t,v^t$ for each time step t in the time series by three shared linear layers with transformation matrices $W^Q,W^K,W^V$ to the input $x^t$.

2.

Compute the attention mask in which the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. The compatibility function computes the dot products of the query with all keys, divides each by $\sqrt{d_k}$ (dimension of keys), and applies a softmax function to obtain the weights on the values.

3.

Compute the sum of the values weighted by the corresponding attention mask as the output for each time step.

In practice, we compute the output for each time step in the time series simultaneously and we compute Equation (2) as the matrix of the outputs, in which $W^Q\in {\mathbb{R}}^{d_{model}\times d_q}$, $W^K\in {\mathbb{R}}^{d_{model}\times d_k}$, $W^V\in {\mathbb{R}}^{d_{model}\times d_v}$ are projection transformation matrices, $x\in {\mathbb{R}}^{T\times d_{model}}$ is the input time series, and softmax computes Equation (3) to normalize the attention mask (n is the number of weights and equals $T^2$ in Equation (2)).

$$Attention(x,W^Q,W^K,W^V)=softmax(\frac{\left(x{W}^Q\right){\left(x{W}^K\right)}^T}{\sqrt{d_k}})x{W}^V$$

(2)

$$softmax{\left(z\right)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^n{e}^{z_j}},i=1,...,n$$

(3)

Instead of performing a single SDPA to input the time series, multi-head attention (MHA) is proven to be beneficial with the diversity of feature representations, which compute H times SDPA simultaneously and then concatenates the outputs of H heads to yield the final output with projection matrix $W^O\in {\mathbb{R}}^{H{d}_v\times d_{model}}$ as the following equation:

$$\begin{array}{cc}\hfill Multi-head\left(x\right)& =Concat(hea{d}_1,hea{d}_2,...,hea{d}_H)W^O\hfill \\ \hfill wherehea{d}_i& =Attention(x,W_i^Q,W_i^K,W_i^V),i=1,2,...,H\hfill \end{array}$$

(4)

In the processing of SDPA and MHA, all time steps are taken into account for each time step in the time series; with iterative training, the model will automatically assign different weights for time steps in the time series. Thus, the self-attention mechanism is capable of focusing on some crucial temporal positions through global temporal attention.

2.3. Lightweight Self-Attention for Global Attention Extraction

LTAE [52] is a lightweight self-attention mechanism variant in the classification of a remote sensing time series. Two crucial improvements allow LTAE to achieve strong feature extraction capabilities while reducing the number of trainable parameters:

1.

Unlike the vanilla transformer [41], which computes multiple triplets for multi-head attention from the input sequence of size $T\times d_{in}$ and produces an output size of $T\times n_{head}\times d_{out}$ ($n_{head}$ is the number of heads), LTAE’s multi-head attention is applied along the channel dimensions, then the outputs of all heads are concatenated into a vector with the same channels as the input sequence, regardless of the number of heads. Such modified multi-head attention mechanism, while retaining the diversity of feature extraction, significantly reduces the number of trainable parameters.

2.

Reference [52] defines a learnable query, named a master query, for each head in LTAE instead of computing from the input sequence by a linear layer, which further reduces the parameter numbers. The attention masks of each head are defined as the scaled softmax of the dot-product between the keys and the master query. The output of each head is defined as the sum of the corresponding inputs weighed by the attention mask in the temporal dimension.

LTAE’s attention mechanism can be summarized by the following equation for the h-th head, in which $x_h$ refers to the h-th channel group of x, $Q_h$ refers to the h-th learnable master query, and $W_h^K$ refers to the h-th transformation matrix for keys.

$$LTAE{\left(x\right)}_h=softmax(\frac{Q_h{\left(x_h{W}_h^K\right)}^T}{\sqrt{d_k/n_{head}}})x_h$$

(5)

2.4. Lightweight Convolution for Local Attention Extraction

LConv [56] has a fixed context window and it determines the importance of local neighborhood elements with a set of weights that are shared not only along the temporal dimension but also within certain output channels (channel group). Furthermore, weights are normalized across temporal dimensions using a softmax operation and then every entry of the normalized weights is dropped with probability p. A regular convolution requires $d_{model}^2\times kernel\_size$ trainable weights assuming that both input channels and output channels of convolution are $d_{model}$. Nevertheless, LConv significantly reduces trainable parameters to $n_{groups}\times kernel\_size$ ($n_{groups}$ is the number of channel groups in LConv) while maintaining the same local attention extraction capability. We set $n_{groups}=n_{head}$ so LConv could have a diversity of feature extractions similar to multi-head attention in LTAE. LConv computes the following equation for the i-th time step in the time series and output channel c, in which W is the weight of LConv.

$$LConv(x,W,i,c)=\sum _{j=1}^{kernel\_size}{W}_{\lceil \frac{c\times n_{groups}}{d_{model}}\rceil }·x_{i+j-\lceil \frac{kernel\_size+1}{2}\rceil ,c}$$

(6)

2.5. Channel Dimension Split and Concatenation

Instead of feeding the whole input time series into both two submodules, we used a Dim. Split layer to split the positional encoded time series into two parts of size $T\times \frac{d_{model}}{2}$. LTAE produces the global temporal feature of size $\frac{d_{model}}{2}$ and LConv produces the local temporal feature of size $T\times \frac{d_{model}}{2}$. The local temporal feature is then averaged along temporal dimensions into size $\frac{d_{model}}{2}$ and concatenated in a Dim. Concat. layer with the global temporal feature into the $d_{model}$ size.

3. Experiments and Results

In this section, we compare GL-TAE with other state-of-the-art competing methods in two publicly available datasets to evaluate the effectiveness of hybrid global–local attention.

3.1. Datasets

We selected two publicly available datasets for evaluating our proposed method in the experiments. Both datasets are pixel-based and imbalanced. Their data come from different sensors and geographic regions.

(1)

TiSeLaC: The study data come from the datasets provided by the 2017 Time Series Land Cover Classification Challenge (accessed on 20 December 2021) (https://sites.google.com/site/dinoienco/tiselac-time-series-land-cover-classification-challenge). The original data were generated from the level 2A Landsat, 8 images from 23 acquisition dates in 2014 above Reunion Island, shown in Figure 2. The spatial resolution was 30 m and 10 features were selected for each pixel, including the first 7 bands of Landsat-8 (Band1–Band7) and 3 complementary radiometric indices (normalized difference vegetation index, normalized difference water index, and brightness index) computed from original data. The original geographic coordinate information for each pixel was not considered in the experiments. The original dataset was officially divided into the train set and test set; we reintegrated the dataset to form a dataset consisting of 99,687 pixels distributed over 9 classes of land cover; the sample statistics are shown in

Table 1. Sample Statistics of TiSeLaC.

Class Name	Number of Samples	Proportion (%)
Urban areas	20,000	20.06
Other built-up surfaces	3883	3.89
Forests	20,000	20.06
Sparse vegetation	19,398	19.46
Rocks and bare soil	15,530	15.58
Grassland	6817	6.84
Sugarcane crops	9187	9.22
Other crops	1754	1.76
Water	3118	3.13
Total	99,687	100.0

. The ground truth label of each pixel is referenced from the 2012 Corine Land Cover map (https://www.eea.europa.eu/publications/COR0-landcover) and the 2014 local farmers’ graphical land parcel registration results.

(2)

TimeSen2Crop [57]: The data were collected from 15 Sentinel-2 tiles covering Austria, shown in Figure 3, between September 2017 and September 2018m with cloud coverage of less than 80% and atmospherically corrected using the radiative transfer model MODTRAN [58]. The number of acquisition dates of the time series varies for each tile, ranging from 23 to 38. The dataset has 16 crop types and 1.2 million pixels in total, and the sample statistics are shown in

Table 2. Sample Statistics of TimeSen2Crop.

Class Name	Number of Samples	Proportion (%)
Legumes	7951	0.66
Grassland	283,263	23.37
Maize	164,316	13.55
Potato	30,678	2.53
Sunflower	22,787	1.88
Soy	70,884	5.85
Winter Barley	94,061	7.76
Winter Caraway	1472	0.12
Rye	53,694	4.42
Rapeseed	41,901	3.46
Beet	34,064	2.81
Spring Cereals	85,353	7.04
Winter Wheat	132,327	10.92
Winter Triticale	66,448	5.48
Permanent Plantation	46,312	3.82
Other Crops	76,713	6.33
Total	1,212,224	100.0

. For each pixel, there are 9 Sentinel-2 Bands (B2, B3, B4, B5, B6, B7, B8A, B11, B12) and 1 non-spectral band, indicating the condition of the pixel that was not considered in the experiments. The 20-m bands (B5, B6, B7, B8A, B11, B12) were resampled to 10 m and the final spatial resolution for each pixel is 10 m. The ground truth label of each pixel was extracted from the publicly available Austrian crop type map (https://www.data.gv.at/katalog/dataset/e21a731f).

3.2. Competing Methods and Classification Architecture

In order to verify the effectiveness of GL-TAE, the following temporal encoders were selected for comparison: TempCNN [32], TCN [35], LSTM [21], transformer [50], TAE [51], and LTAE [52].

1.

TempCNN: TempCNN [32] has three vanilla convolutional layers that apply convolutions on the temporal dimensions with different kernels of kernel_size = 3. We added the same channel enhancement layer of GL-TAE before the convolutions. Then TempCNN flattens and dimensionally reduces the time series from $T\times last\_kernels$ to $d_{output}$ with a dense layer. However it is unable to handle the time series with variable temporal lengths; therefore, we replaced the flattened-dense layer with the global (temporal) average pooling layer for experiments on the TimeSen2Crop dataset.

2.

TCN: TCN [35], which is different from TempCNN, has three causal convolutional layers with dilation $d=[1,2,4]$ and $kernel\_size=3$, which means TCN has a larger perceptive field than TempCNN along the temporal dimension.

3.

LSTM: We adopted a one-layer LSTM with adjustable hyperparameters $hidden\_size$, and then used the last hidden state and transformed it into the $d_{model}$ size by a dense layer. A channel enhancement layer was also added before the LSTM layer.

4.

Transformer: The architecture in [50] was adopted. In the original transformer, a query–key–value triplet was computed simultaneously for each time step in the time series and the attention mask was obtained by calculating the similarity of the queries and keys. The encoded result was defined as the sum of the values weighted by the attention mask. Reference [50] took the original transformer encoder with positional encoding, multi-head attention, and feedforward networks and used a global maximum average operation to achieve a dimensionality reduction from size $T\times d_{model}$ to $d_{model}$.

5.

TAE: Reference [51] removed the feedforward networks and defined the master query as the mean of queries. The master query was then multiplied with the keys to determine a single attention mask for weighing the input time series.

6.

LTAE: As mentioned in Section 2.4, LTAE is one of the state-of-the-art temporal encoders.

In the experiments, we followed the scheme in [51]: first, we extracted the temporal features and then used a multi-layer perceptron (MLP) as a common classifier. The purpose was to fairly compare the capabilities of different temporal encoders and exclude the influence of using different classifiers in practice.

3.3. Implementation Details

The five-fold cross-validation scheme was implemented to evaluate the performance of each method, and both datasets were divided into training, validation, and test sets at a 3:1:1 ratio. In the training stage, we trained models at a fixed batch size of 128, focal loss [59] as the objective function, and the Adam optimizer with the default setting ($lr={10}^{-3}$, $\beta =(0.9,0.999)$). Focal loss computes the following with $\widehat{p}$ being classification probability and y being the ground truth for binary classification.

$$Focalloss\left(\widehat{p}\right)=\left\{\begin{array}{c}\hfill -{(1-\widehat{p})}^{\gamma }log\left(\widehat{p}\right),ify=1\\ \hfill -{\widehat{p}}^{\gamma }log(1-\widehat{p}),ify=0\end{array}\right.$$

(7)

The quantitative evaluation metrics used were overall accuracy (OA) and mean per-class intersect over union (mIoU). OA is the most commonly used metric for classification evaluation, but it is subject to the accuracy of classes, which take more proportions in the dataset. That is why mIoU was chosen as a classification metric that reflected the performance of a model on the imbalanced dataset. All networks were trained for 1000/100 epochs in each fold on the TiSeLaC/TimeSen2Crop dataset to ensure the convergence of the model; we selected the best mIoU epoch on the validation set and evaluated it on the test set. We adjusted the hyperparameters of each encoder to obtain models with parameters at different orders of magnitude due to the fact that the number of parameters of the model affected the classification performance [51,52]. The hyperparameters adjusted for each encoder and their candidates are listed in

Table 3. Candidates of hyperparameters of different encoders.

Model	Candidates of Hyperparameters	Reference
GL-TAE	$d_{model}=\{128,256,512,1024,2048\},n_{groups}=n_{head}=\{8,16,32,64\},d_k=8,kernel\_size=3,$ $MLP=\left\{\right[128],[128,128],[256,128],[512,128],[1024,256,128],[2048,1024,256,128\left]\right\}$
LTAE	$d_{model}=\{128,256,512,1024,2048\},n_{head}=\{8,16,32,64\},d_k=8,$ $MLP=\left\{\right[128],[128,128],[256,128],[512,128],[1024,256,128],[2048,1024,256,128\left]\right\}$	[52]
TAE	$d_{model}=\{16,32,64,128,256,512,1024\},n_{head}=\{4,8\},d_k=\{8,16\},$ $MLP=\left\{\right[64,128],[128,128],[256,128],[512,128],[1024,128],[2048,128],[4096,128],[8192,256,128\left]\right\}$	[51]
Transformer	$d_{model}=\{32,64,128,256,512\},n_{head}=4,d_k=8,d_{ffn}=\{128,256,512,1024,2048\}$	[50]
TempCNN	$d_{model}=\{32,64,128,256,512\},kernel\_size=3,kernels=\{32,64,128,256,512\},n\_layer=\{2,3\}$	[32]
TCN	$d_{model}=\{32,64,128,256,512\},kernel\_size=3,kernels=\{32,64,128,256,512\},n\_layer=\{2,3\}$	[35]
LSTM	$d_{model}=\{64,128,256,352,512,640,896\},d_{hidden}=\{64,128,256,352,512,640,896\}$	[21]

. The dropout probability p used for training all models was adjusted in the range of [0, 0.5].

3.4. Classification Results

In

Table 4. Performance of GL-TAE and competing models all having approximate 160 K parameters on (a) TiSeLaC/140 K parameters and (b) TimeSen2Crop.

(a) TiSeLaC
Class	GL-TAE	LTAE	TAE	Transformer	TempCNN	TCN	LSTM
N.params	163 K	161 K	164 K	165 K	156 K	160 K	156 K
OA(%)	$\mathbf{93.70}\mathbf{\pm }\mathbf{0.17}$	$93.03\pm 0.20$	$92.81\pm 0.18$	$91.36\pm 0.37$	$93.38\pm 0.24$	$92.93\pm 0.17$	$91.61\pm 0.33$
mIoU(%)	$\mathbf{83.42}\mathbf{\pm }\mathbf{0.42}$	$81.66\pm 0.23$	$81.42\pm 0.53$	$78.01\pm 0.73$	$82.88\pm 0.52$	$81.72\pm 0.29$	$78.20\pm 0.68$
Urban areas	$93.12\pm 0.38$	$91.81\pm 0.57$	$92.33\pm 0.59$	$90.96\pm 0.67$	$\mathbf{93.15}\mathbf{\pm }\mathbf{0.41}$	$91.43\pm 0.73$	$90.54\pm 0.71$
Other built-up surfaces	$79.97\pm 1.17$	$78.75\pm 2.33$	$77.86\pm 2.76$	$74.94\pm 1.51$	$\mathbf{80.67}\mathbf{\pm }\mathbf{1.95}$	$74.53\pm 1.89$	$74.88\pm 1.26$
Forest	$\mathbf{93.46}\mathbf{\pm }\mathbf{0.99}$	$92.84\pm 1.02$	$92.81\pm 0.53$	$90.93\pm 0.85$	$92.92\pm 0.57$	$92.03\pm 0.76$	$91.32\pm 0.74$
Sparse Vegetation	$\mathbf{96.14}\mathbf{\pm }\mathbf{0.47}$	$95.95\pm 0.21$	$95.04\pm 0.61$	$94.39\pm 0.80$	$95.66\pm 0.46$	$94.64\pm 0.63$	$94.72\pm 0.22$
Rocks and bare soil	$96.73\pm 0.27$	$96.32\pm 0.23$	$96.51\pm 0.36$	$95.32\pm 0.48$	$\mathbf{96.83}\mathbf{\pm }\mathbf{0.17}$	$96.07\pm 0.27$	$96.08\pm 0.28$
Grassland	$\mathbf{90.86}\mathbf{\pm }\mathbf{1.58}$	$90.61\pm 1.01$	$89.07\pm 1.18$	$87.26\pm 0.70$	$89.67\pm 1.60$	$89.41\pm 0.55$	$87.59\pm 0.55$
Sugarcane crops	$95.90\pm 0.34$	$95.37\pm 0.69$	$95.81\pm 0.38$	$94.89\pm 0.51$	$\mathbf{96.42}\mathbf{\pm }\mathbf{0.66}$	$95.22\pm 1.28$	$94.62\pm 0.69$
Other crops	$\mathbf{81.23}\mathbf{\pm }\mathbf{2.34}$	$75.81\pm 1.62$	$74.60\pm 3.10$	$69.86\pm 2.59$	$74.48\pm 1.36$	$75.10\pm 5.72$	$71.95\pm 4.41$
Water	$92.05\pm 1.89$	$92.03\pm 1.52$	$91.35\pm 1.26$	$87.03\pm 2.07$	$\mathbf{92.27}\mathbf{\pm }\mathbf{0.93}$	$90.45\pm 2.34$	$87.80\pm 1.55$
(b) TimeSen2Crop
Class	GL-TAE	LTAE	TAE	Transformer	TempCNN	TCN	LSTM
N.params	139 K	137 K	142 K	142 K	137 K	139 K	137 K
OA(%)	$\mathbf{95.54}\mathbf{\pm }\mathbf{0.05}$	$95.35\pm 0.03$	$94.52\pm 0.06$	$94.45\pm 0.21$	$94.77\pm 0.04$	$94.95\pm 0.05$	$93.98\pm 0.04$
mIoU(%)	$\mathbf{90.95}\mathbf{\pm }\mathbf{0.16}$	$90.56\pm 0.09$	$88.41\pm 0.10$	$88.38\pm 0.27$	$89.21\pm 0.13$	$89.46\pm 0.10$	$86.85\pm 0.09$
Legumes	$\mathbf{93.29}\mathbf{\pm }\mathbf{0.69}$	$92.09\pm 0.64$	$90.04\pm 2.40$	$91.21\pm 0.44$	$91.38\pm 1.26$	$92.32\pm 1.03$	$87.76\pm 1.17$
Grassland	$94.67\pm 0.34$	$94.05\pm 0.38$	$93.89\pm 0.48$	$94.38\pm 0.46$	$\mathbf{94.81}\mathbf{\pm }\mathbf{0.21}$	$94.66\pm 0.59$	$94.34\pm 0.45$
Maize	$98.87\pm 0.17$	$98.83\pm 0.10$	$98.57\pm 0.08$	$98.71\pm 0.18$	$98.80\pm 0.07$	$\mathbf{98.91}\mathbf{\pm }\mathbf{0.10}$	$98.45\pm 0.23$
Potato	$95.59\pm 0.92$	$\mathbf{95.85}\mathbf{\pm }\mathbf{0.55}$	$95.24\pm 0.41$	$94.25\pm 1.27$	$94.34\pm 0.87$	$95.47\pm 0.45$	$93.45\pm 1.06$
Sunflower	$97.30\pm 0.62$	$97.42\pm 0.77$	$96.97\pm 0.46$	$97.11\pm 0.47$	$97.22\pm 0.75$	$\mathbf{97.38}\mathbf{\pm }\mathbf{0.58}$	$96.44\pm 1.17$
Soy	$\mathbf{98.84}\mathbf{\pm }\mathbf{0.20}$	$98.75\pm 0.25$	$98.30\pm 0.21$	$98.72\pm 0.18$	$98.53\pm 0.21$	$98.64\pm 0.13$	$97.94\pm 0.30$
Winter Barley	$98.28\pm 0.51$	$\mathbf{98.29}\mathbf{\pm }\mathbf{0.18}$	$97.83\pm 0.71$	$97.83\pm 0.28$	$97.62\pm 0.63$	$97.97\pm 0.35$	$97.50\pm 0.41$
Winter Caraway	$\mathbf{94.57}\mathbf{\pm }\mathbf{1.27}$	$92.81\pm 0.68$	$91.67\pm 2.11$	$88.49\pm 2.46$	$92.37\pm 2.44$	$91.75\pm 2.14$	$89.24\pm 2.81$
Rye	$\mathbf{93.63}\mathbf{\pm }\mathbf{0.51}$	$92.08\pm 0.64$	$89.40\pm 1.07$	$88.13\pm 0.88$	$89.29\pm 1.82$	$90.51\pm 0.56$	$87.54\pm 1.10$
Rapeseed	$\mathbf{99.35}\mathbf{\pm }\mathbf{0.16}$	$99.09\pm 0.17$	$99.05\pm 0.21$	$99.09\pm 0.16$	$99.18\pm 0.19$	$99.25\pm 0.19$	$99.16\pm 0.16$
Beet	$\mathbf{98.99}\mathbf{\pm }\mathbf{0.12}$	$98.81\pm 0.14$	$98.60\pm 0.18$	$98.54\pm 0.38$	$97.92\pm 0.49$	$98.64\pm 0.26$	$98.19\pm 0.50$
Spring Cereals	$\mathbf{96.79}\mathbf{\pm }\mathbf{0.25}$	$96.66\pm 0.36$	$96.01\pm 0.22$	$95.88\pm 0.32$	$96.33\pm 0.15$	$96.34\pm 0.49$	$95.11\pm 0.78$
Winter Wheat	$\mathbf{97.25}\mathbf{\pm }\mathbf{0.33}$	$97.06\pm 0.52$	$96.19\pm 0.54$	$96.47\pm 0.48$	$96.37\pm 0.36$	$96.45\pm 0.24$	$95.72\pm 0.41$
Winter Triticale	$89.78\pm 1.05$	$\mathbf{89.91}\mathbf{\pm }\mathbf{0.74}$	$85.98\pm 1.00$	$87.45\pm 0.48$	$87.17\pm 1.05$	$88.70\pm 0.33$	$85.75\pm 0.93$
Permanent Plantation	$95.77\pm 0.86$	$\mathbf{95.96}\mathbf{\pm }\mathbf{0.62}$	$93.97\pm 1.17$	$94.35\pm 0.61$	$94.35\pm 0.90$	$95.10\pm 1.06$	$94.80\pm 0.94$
Other Crops	$80.60\pm 0.99$	$\mathbf{81.29}\mathbf{\pm }\mathbf{1.60}$	$79.37\pm 1.24$	$75.96\pm 2.29$	$78.61\pm 1.03$	$77.71\pm 1.49$	$74.89\pm 1.60$

, we present the classification results of GL-TAE and competing methods under the condition of having approximately the same parameter numbers, all calculating the mean and standard deviations from a five-fold cross-validation scheme. Two convolutional networks (TempCNN, TCN) achieved comparable classification results compared with three self-attention mechanism-based models (LTAE, TAE, transformer) in two datasets, corroborating the importance of local temporal features. LTAE outperforms TAE and TAE outperforms the transformer, indicating that the setting of the master query and the modified mechanism of multi-head attention along the channel dimension have improvements in the self-attention mechanisms in extracting global temporal features. Our proposed GL-TAE outperformed all competing models in OA by 0.32–2.34% and 0.19–1.56% and mIoU by 1.70–5.41% and 0.39–4.1% on two datasets. Furthermore, GL-TAE achieved the best results in the per-class accuracy in most classes of two datasets. Specifically, GL-TAE yielded the best performances in the forest, sparse vegetation, grassland, and other crops of TiSeLaC, and yielded the best performances in legumes, soy, winter caraway, rye, rapeseed, beet, spring cereals, and winter wheat. For other classes where GL-TAE did not achieve the best results, the differences in the performance between GL-TAE and the best-performing model were negligible. For example, with TiSeLaC, TempCNN achieved the best results in urban areas and rocks and bare soil, which exceed GL-TAE by not more than 0.1%, and in TimeSen2Crop, GL-TAE achieved the best results in potato, winter barley, winter triticale, and permanent plantation, which exceed GL-TAE by 0.01–0.26%. It can be seen that our proposed GL-TAE, which is based on hybrid global–local temporal attention, is better than global-only or local-only attention-based methods for SITS classification under the same level of the model size.

3.5. Comprehensive Comparison with Different Hyperparameter Settings

Figure 4 and Figure 5 show the performances in OA and mIoU of all models under different parameter numbers, ranging from 6 K to 4 M parameters. All models, except TAE, obtained better performances as the parameter numbers increased. Among the competing methods, TempCNN yielded the best results and LTAE yielded a slightly worse performance in TiSeLaC; however, LTAE yielded the best results in TimeSen2Crop—significantly better than TempCNN and TCN. Based on the above results, we argue that LTAE is more robust, and that with convolutional networks, it is hard to tackle larger datasets, even with larger perceptive fields. The vanilla self-attention mechanism method transformer and recurrent network LSTM yielded the worst results of the two datasets. GL-TAE yielded the best performance with parameters at different orders of magnitude on two datasets; this further demonstrates that our proposed hybrid global–local temporal attention-based approach is more robust and effective at extracting temporal features for classification compared to global-only and local-only attention-based methods.

Furthermore, we can conclude from Figure 4 and Figure 5 that smaller GL-TAE models can achieve comparable performances to other models. A GL-TAE with 163 K parameters achieves a similar performance to a TempCNN with 534 K parameters with $83.42\pm 0.42\%$ mIoU, and $93.70\pm 0.17\%$ OA on TiSeLaC. With TimeSen2Crop, a GL-TAE with 53 K parameters yields comparable results with most competing models with 300 K–3 M parameters and a GL-TAE with 440 K parameters yields $91.99\pm 0.29\%$ mIoU and $95.95\pm 0.05\%$ OA, which is slightly worse than a LTAE with 3.5 M parameters.

Overall, the hybrid global–local attention used in our proposed GL-TAE is better for SITS classification than global attention or local attention in terms of temporal feature extraction, and GL-TAE is a lightweight model, which is an advantage in practice for processing massive amounts of SITS data.

4. Discussion

We verified the effectiveness of GL-TAE and its hybrid global–local attention for SITS classification in two datasets in the last section. In this section, we further analyze which attention plays a more important role in the hybrid global–local attention of GL-TAE, whether the $Dim.Split$ strategy is necessary, and what the impacts are of the different values of the hyperparameters in GL-TAE.

4.1. Importance Comparison of Global and Local Attention

Figure 6 and Figure 7 show the results of GL-TAE, LTAE, and LConv on two datasets under different parameter numbers. LTAE or LConv could outperform GL-TAE on the two datasets, which demonstrates the effectiveness of combining LTAE and LConv to extract global and local attention in GL-TAE. LConv outperforms LTAE with relatively fewer parameter numbers on TiSeLaC, but with the parameter numbers increasing, LTAE achieves a better performance than LConv. LTAE outperforms LConv on TimeSen2Crop with an obvious large margin, which is consistent with the conclusion in Section 3.5 that convolutional networks perform worse on large datasets. We can conclude that global attention extracted by LTAE plays a more important role in GL-TAE than local attention extracted by LConv. LTAE or LConv could outperform GL-TAE under any parameter numbers on the two datasets, which demonstrates the effectiveness and stability of hybrid global–local attention extracted by combining LTAE and LConv for SITS classification.

4.2. Necessity of Channel Dimension Split

The original purpose of Dim. Split is to reduce the parameter numbers of GL-TAE under the same hyperparameter settings. If Dim. Split is discarded, we have to double $MLP\left[0\right]$ to meet the consistency of the channel dimension; the channel numbers in each head of LTAE and each group in LConv will also be doubled. Consequently, the parameter numbers will be doubled under the same hyperparameter settings. Figure 8 and Figure 9 show the results of four hyperparameter settings of GL-TAE on the two datasets without Dim. Split, which corresponds to the second–fifth hyperparameter settings with Dim. Split. Obviously, GL-TAEs with Dim. Split exceed GL-TAEs without Dim. Split under the same order of magnitude of parameters. Furthermore, under the same hyperparameter settings, if we discard Dim. Split, the parameter numbers are almost doubled, but the performance is only slightly improved in some cases. For example, a GL-TAE with $d\_model=256$, $n\_head=n\_groups=16$, $d\_k=8,MLP=[256,128]$, and Dim. Split has 53 K parameters and yields $90.20\pm 0.24\%$ mIoU, and $95.22\pm 0.12\%$ OA on TimeSen2Crop. If we discard Dim. Split, a GL-TAE yields $90.48\pm 0.32\%$ mIoU and $95.38\pm 0.09\%$ OA with 103 K parameters. Therefore, Dim. Split is an effective and necessary strategy to feed input into two submodules in GL-TAE, which make the model lightweight, combined with two lightweight submodules, while maintaining comparable performances.

4.3. Impact of Different Values of Hyperparameters

In

Table 5. The influence of four hyperparameters of GL-TAE on the performance of TiSeLaC. ’Underlines’ are default parameter values.

(a) ${\mathit{d}}_{\mathit{model}}$			(b) ${\mathit{n}}_{\mathit{head}}\left({\mathit{n}}_{\mathit{groups}}\right)$
$d_{model}$	N.params	mIoU	$n_{head}\left(n_{groups}\right)$	N.params	mIoU
256	82 K	$81.82\pm 0.62$	8	113 K	$82.73\pm 0.45$
512	163 K	$83.42\pm 0.42$	16	130 K	$82.90\pm 0.30$
1024	488 K	$84.93\pm 0.35$	32	163 K	$83.42\pm 0.42$
2048	2.8 M	$84.97\pm 0.29$	64	229 K	$83.27\pm 0.54$
(c)${\mathit{d}}_{\mathit{k}}$			(d)$\mathit{kernel}\_\mathit{size}$
$d_k$	N.params	mIoU	$kernel\_size$	N.params	mIoU
4	130 K	$83.35\pm 0.50$	3	163 K	$83.42\pm 0.42$
8	163 K	$83.42\pm 0.42$	5	163 K	$83.36\pm 0.44$
16	229 K	$83.29\pm 0.68$	7	163 K	$83.17\pm 0.57$
32	361 K	$83.32\pm 0.58$	9	163 K	$83.13\pm 0.41$

and

Table 6. Influence of four hyperparameters of GL-TAE on the performance of TimeSen2Crop. Underlines are default parameter values.

(a) ${\mathit{d}}_{\mathit{model}}$			(b) ${\mathit{n}}_{\mathit{head}}\left({\mathit{n}}_{\mathit{groups}}\right)$
$d_{model}$	N.params	mIoU	$n_{head}\left(n_{groups}\right)$	N.params	mIoU
256	70 K	$89.48\pm 0.24$	8	89 K	$89.89\pm 0.40$
512	139 K	$90.95\pm 0.16$	16	106 K	$90.79\pm 0.27$
1024	481 K	$91.91\pm 0.12$	32	139 K	$90.95\pm 0.16$
2048	2.7 M	$91.59\pm 0.23$	64	205 K	$90.55\pm 0.15$
(c)${\mathit{d}}_{\mathit{k}}$			(d)$\mathit{kernel}\_\mathit{size}$
$d_k$	N.params	mIoU	$kernel\_size$	N.params	mIoU
4	106 K	$90.85\pm 0.08$	3	139 K	$90.95\pm 0.16$
8	139 K	$90.95\pm 0.16$	5	139 K	$90.98\pm 0.12$
16	205 K	$90.96\pm 0.06$	7	139 K	$90.96\pm 0.15$
32	337 K	$90.98\pm 0.12$	9	139 K	$90.80\pm 0.22$

, we selected four crucial hyperparameters in GL-TAE and investigated the influences of different values for each hyperparameter on the performance of GL-TAE. The selected hyperparameters are the number of channels $d_{model}$, the number of heads of LTAE $n_{head}$ and groups of LConv $n_{groups}$, and the dimension of keys $d_k$ in LTAE and $kernel\_size$ of LConv. For each hyperparameter to be investigated, we only adjusted its value and kept the other hyperparameters as default settings.

The results of the two datasets show that the GL-TAE with a larger number of channels $d_{model}$ had a better performance, We argue that a larger $d_{model}$ has more parameter numbers and richer representation for the time series. We set $n_{head}=n_{groups}$ in GL-TAE and it can be seen from the results that the performance first became better and then worse with $n_{head}\left(n_{groups}\right)$ increasing. We argue that there is a best $n_{head}\left(n_{groups}\right)$ for a fixed $d_{model}$, smaller $n_{head}\left(n_{groups}\right)$ is insufficient to extract the diversity of global and local temporal information, and larger $n_{head}\left(n_{groups}\right)$ leads to each head of LTAE or each group of LConv lacking the representation of temporal information. Additionally, the dimension of keys $d_k$ in LTAE seems to have a limited effect on the performance and the difference caused by $d_k$ is negligible. We note that a larger $kernel\_size$ leads (incrementally) to a slightly worse performance, which indicates $kernel\_size=3$ is expected in practice.

5. Conclusions

In this paper, a novel temporal encoder named GL-TAE, which pays attention to both global and local temporal features, is proposed for remote sensing time series classifications. GL-TAE combines two lightweight submodules, LTAE and LConv, in parallel for extracting global–local attention features. We compared GL-TAE with other state-of-the-art models on two public datasets. GL-TAE has an average improvement of 1.18%/0.87% in OA and 2.77%/2.14% in mIoU on TiSeLaC/TimeSen2Crop with approximately 160 K/140 K parameters in comparison with existing models. We can conclude from the experiments that hybrid global–local temporal attention extraction is better than global-only or local-only attention feature extraction for SITS classification. Meanwhile, one thing noteworthy is that our proposed GL-TAE requires much fewer parameters when achieving the same performance as other models. A GL-TAE with 370 K fewer parameters is able to achieve similar performance to a TempCNN on TiSeLaC, and a GL-TAE with 53 K parameters is capable of achieving comparable results with most existing models with 300 K+ parameters. At this stage, we only focused on the extraction of temporal features from SITS data, but more applications are currently utilizing the spatial–temporal–spectral features for SITS classifications. Further modifications on the proposed GL-TAE to extract spatial–temporal–spectral features can be focused on in future work.