Spectral-Spatial Classification of Hyperspectral Images: Three Tricks and a New Learning Setting

Abstract

Spectral-spatial classification of hyperspectral images has been the subject of many studies in recent years. When there are only a few labeled pixels for training and a skewed class label distribution, this task becomes very challenging because of the increased risk of overfitting when training a classifier. In this paper, we show that in this setting, a convolutional neural network with a single hidden layer can achieve state-of-the-art performance when three tricks are used: a spectral-locality-aware regularization term and smoothing- and label-based data augmentation. The shallow network architecture prevents overfitting in the presence of many features and few training samples. The locality-aware regularization forces neighboring wavelengths to have similar contributions to the features generated during training. The new data augmentation procedure favors the selection of pixels in smaller classes, which is beneficial for skewed class label distributions. The accuracy of the proposed method is assessed on five publicly available hyperspectral images, where it achieves state-of-the-art results. As other spectral-spatial classification methods, we use the entire image (labeled and unlabeled pixels) to infer the class of its unlabeled pixels. To investigate the positive bias induced by the use of the entire image, we propose a new learning setting where unlabeled pixels are not used for building the classifier. Results show the beneficial effect of the proposed tricks also in this setting and substantiate the advantages of using labeled and unlabeled pixels from the image for hyperspectral image classification.

1. Introduction

Hyperspectral images contain rich spectral information coming from contiguous spectral bands. In the spectral domain, pixels are represented by vectors for which each component is a measurement corresponding to specific wavelengths [1]. The length of the vector is equal to the number of spectral bands that the sensor collects. For hyperspectral images, several hundreds of spectral bands of the same scene are typically available, which form the features of a pixel. Current operational imaging systems provide images for various applications, e.g., in ecology, geology and precision agriculture [2].

A relevant task of hyperspectral image processing is classification, which aims at building a classifier using the pixel features in order to assign each pixel to one of a given set of classes [3]. Current state-of-the-art methods take a spectral-spatial approach, meaning that they use neighborhood information of labeled pixels. Spectral-spatial methods are based on diverse techniques, such as Markov random fields [4,5,6], discriminative feature construction [7,8,9,10,11,12], modification and fusion of classifiers [13,14], label propagation, active learning and semi-supervised learning [15,16], the use of external unlabeled data [17] and deep (convolutional) neural networks [18,19,20,21,22,23,24]. Furthermore, object-based methods utilize geometric features of the image extracted by means of segmentation techniques [25,26,27].

These methods achieve excellent performance on benchmark hyperspectral image classification tasks when a large number of labeled pixels for training is provided [18,19]. However, pixel labeling is an expensive task. Therefore, a problem of more practical relevance is to perform hyperspectral image classification with only a few manually-labeled pixels for training. A second problem is the inherent class unbalance of hyperspectral images, where some classes have many pixels, while other classes have only a few.

In this paper, we propose to tackle these problems using a simple shallow Convolutional Neural Network (CNN) and three ‘tricks’: spectral-locality-aware regularization, smoothing-based data augmentation and label-based data augmentation. The shallow architecture is used to prevent overfitting caused by the few labeled pixels and the many features. Locality-aware regularization forces neighboring wavelengths to have similar contributions to the generated features of the neural network. Smoothing-based data augmentation takes advantage of the spectra of neighboring pixels, and label-based data augmentation exploits labels of neighboring pixels in favor of small classes.

Extensive experiments indicate the effectiveness of the proposed method, which achieves comparable or better accuracy performance than existing methods, such as deep neural networks [28], multiple kernel learning [29], probabilistic class structure regularized sparse representation graph [30,31] and low-rank Gabor filtering [12] (see the results in

Table 10. Average classification accuracy reported on other experimental settings used in the literature. We considered the following methods: Low-Rank and Sparse Representation Classifier with a Spectral Consistency Constraint (LRSRC-SCC), Probabilistic Class Structure Regularized Sparse Representation (PCSSR), Multiple Kernel Learning (MKL), Discriminative Low-Rank Gabor Filtering (DLRGF), Kernel Sparse Representation (KSR), Image Fusion and Recursive Filtering (IFRF) and our method (CNN-RSL). The best accuracy for each dataset and % training samples employed is indicated in bold.

Dataset	Train %	Methods
		LRSRC-SCC [30]	PCSSR [31]	MKL [29]	DLRGF [12]	KSR [34]	IFRF [46]	CNN-RSL
Pavia University	1%	94.15$\pm$0.56	—	96.87	—	—	87.88$\pm$1.19*	94.74$\pm$0.25*
Pavia University	5%	—	—	—	—	—	97.58$\pm$1.51	96.70$\pm$0.44*
KSC	1%	—	—	79.80	89.40$\pm$0.88*	—	84.01$\pm$1.76*	90.34$\pm$0.97
KSC	5%	—	88.48	—	98.73$\pm$0.99*	—	93.46$\pm$1.23*	99.79$\pm$0.20
Indian Pines	1%	—	—	—	83.59$\pm$0.81*	—	84.50$\pm$1.24*	86.42$\pm$0.66
Indian Pines	5%	—	—	—	95.16$\pm$0.24*	—	95.31$\pm$0.85*	96.42$\pm$0.24
Indian Pines	10%	95.18$\pm$0.58*	—	—	—	98.47	96.27$\pm$0.34*	97.62$\pm$0.17*
Salinas	1%	—	—	95.31	—	—	96.36$\pm$0.51*	96.93$\pm$0.55
Salinas	5%	—	—	97.98	—	—	98.36$\pm$0.51*	99.03$\pm$0.17

).

Spectral-spatial methods exploit information from neighborhood pixels. Since the training and testing pixels are drawn from the same image, their features are likely to overlap in the spatial domain due to the shared source of information: for instance, [23] employed input patches, the central pixel of which is in the training set, and [12] applied Gabor filters to an L-size neighborhood of training pixels. As a consequence, the resulting learning setting used in spectral-spatial methods has an intrinsic positive bias induced by the overlap between training and test samples. In order to investigate such a bias, we consider also a non-overlapping learning setting, where only the labeled pixels initially selected for training are used for building a classifier.

Related Work

Below, we briefly mention a few selected spectral-spatial approaches and methods for hyperspectral image classification. We refer the reader to [32] for a recent survey of hyperspectral image classification methods.

We can divide methods for hyperspectral image classification into three broad categories: (1) pre-processing-based; (2) end-to-end methods; (3) hybrid methods. Pre-processing-based methods construct features prior to training a classifier. Recent methods in this category include the Discriminative Low-Rank Gabor Filtering (DLRGF) method by [12] for spectral-spatial feature extraction prior to classification, a deep CNN with 2D input patches and R-PCA [20] and a deep stacked auto-encoder with 2D input patches and PCA [33].

End-to-end methods learn features while training a classifier. These methods include (multiple) kernel learning methods, which use kernels to implicitly map the input space into a high dimensional non-linear space (see the recent survey [29]), and sparse representation-based methods, like [30,31,34], which learn a sparse representation of test pixels by the linear combination of a few training samples from a given dictionary, whereas its corresponding sparse representation coefficients encode the class information implicitly. Hybrid methods involve multi-step procedures, which include pre- and/or post-processing steps. For instance, the superpixel-based graphical model by [35] consists of three steps: the superpixel generation using the watershed segmentation algorithm after performing gradient fusion among multiple spectral bands; the superpixel-based graphical model development with the aid of pixel-level attributes; and the loopy belief propagation algorithm applied at the superpixel level. Here, a superpixel is a group of spatially-connected similar pixels. Object-based methods segment an image and simultaneously try to assign to each segment a class [25,26,27].

Methods specifically related to the one we propose are based on convolutional neural networks and data augmentation. Due to the success of convolutional neural networks in image classification, a plethora of CNN-based methods for hyperspectral image classification have been proposed. They differ mainly in the architecture that they use, the specific loss function that is optimized and the representation of the input data, that is as single pixels, patches of pixels, cubes of pixels, etc. Moreover, some CNN-based methods use preprocessing, often PCA, to either build a low dimensional set of non-linear input features or to extract additional information (e.g., edge detection). These methods include [20], a deep CNN with 2D input patches and R-PCA [33], a deep stacked auto-encoder with 2D input patches and PCA [18], a contextual deep CNN [36], a multi-hypothesis prediction [12], a low-rank Gabor filtering method [19], a deep CNN with 1D pixel spectra [23], a deep CNN with 1D pixel spectra, 2D pixel patches or 3D pixel cubes [21], a deep CNN with 1D pixel spectra and [28] a deep CNN with uniform smoothing kernel and 1D pixel spectra. Fortunately, the authors of the latter method shared the source code with us, which we could then use in our comparative experimental analysis.

Data augmentation is used to enhance the performance of deep neural networks for image classification. This approach has also been used in the context of hyperspectral image classification, in deep CNN-based methods. For instance, [37] used blocks of $5\times 5$ pixels as samples and rotated and flipped the resulting training samples to enlarge the training set. In the deep CNN-based method by [38], the number of training samples was augmented four times by mirroring the training samples across the horizontal, vertical and diagonal axes. Our new data augmentation procedure is different because it takes into account the spatial locality of the data.

In [39], it has been observed that the dependence caused by overlap between the training and testing samples may be artificially enhanced by some spatial information processing techniques used in spectral-spatial classification methods, such as spatial filtering and morphological operators. Therefore, the authors introduced an alternative controlled random sampling strategy for spectral-spatial methods to reduce the overlap between training and testing samples and provided a more objective evaluation. However, the proposed strategy uses information on the class distribution, which may not be available in real-life scenarios. The non-overlapping learning setting that we propose overcomes this limitation.

2. Materials and Methods

A hyperspectral image is represented by a three-dimensional matrix of spectral pixels in ${\mathbb{R}}^{H\times W\times M}$, where H is the height, W is the width and M is the number of wavelengths. We denote such an input image by P and the original input image by $P^{\mathrm{orig}}$.

A subset of pixels $I\subseteq H\times W$ from the input image has known class labels. This subset is called the training set, denoted by $(\mathbf{x},\mathbf{y})$. We denote by ${\mathbf{x}}_i$ the i-th pixel of the training set, $1\le i\le \left|I\right|$ and by $y_i$ its label. The rest of the pixels of the image form the test set, denoted as ${\mathbf{x}}^{\mathrm{test}}$. The number of classes is denoted as K, and we will treat labels as binary vectors, so $y_{i,k}=1$ if and only if the i-th pixel belongs to class k.

Our method extends the training set by doing data augmentation. With a slight abuse of notation, we also denote the resulting training set by $\mathbf{x}$ with assigned labels $\mathbf{y}$.

2.1. Data

We consider five groups of hyperspectral images, which are publicly available (http://www.ehu.eus/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_Scenes):

• Pavia Center: obtained with a Reflective Optics System Imaging Spectrometer (ROSIS) sensor during a flight campaign over Pavia, Northern Italy
• Pavia University: scanned using the ROSIS sensor during a flight campaign over Pavia, Northern Italy
• Kennedy Space Center (KSC): obtained with the Airborne/Visible Infrared Imaging Spectrometer (AVIRIS) sensor over the Kennedy Space Center, Florida (USA)
• Indian Pines: scanned using the AVIRIS sensor over the Indian Pines test site in north-western Indiana (USA)
• Salinas: scanned using the AVIRIS sensor over Salinas Valley, California (USA)

As is common practice in hyperspectral image classification, we consider only the foreground pixels. Moreover, for the Indian Pines dataset, as done, e.g., in [12], we discard classes with very few samples and keep the remaining 12 classes (Corn-notill, Corn-min, Corn, Grass/Pasture, Grass/Trees, Hay-windrowed, Soybeans-notill, Soybeans-min, Soybean-clean, Wheat, Woods and Bldg-Grass-Tree-Drives). Characteristics of the images (size, number of features, foreground pixels and considered classes) are given in

Table 1. Description of the hyperspectral images. KSC, Kennedy Space Center.

Image	Size	# Features	# Foreground Pixels	# Classes
Pavia Center	$1096\times 715$	102	148,152	09
Pavia University	$\phantom{0}610\times 340$	103	42,776	09
KSC	$\phantom{0}512\times 614$	176	5211	13
Indian Pines	$\phantom{0}145\times 145$	220	10,062	12
Salinas	$\phantom{0}512\times 217$	224	54,129	16

.

2.2. Learning Settings

In order to build and assess a classifier, pixels of a hyperspectral image are divided into a training (labeled pixels) and a test (unlabeled pixels) set. Different learning settings can be considered, depending on how training and test sets are used for building a classifier. Here, we consider two learning settings: (1) the transductive setting used in spectral-spatial hyperspectral image classification and (2) a new learning setting where only the labeled pixels selected for training are used to build a classifier.

2.2.1. Transductive Learning Setting

Spectral-spatial methods exploit information from neighborhood pixels. Since the training and testing pixels are drawn from the same image, their features are likely to overlap in the spatial domain due to the shared source of information [39]. In this transductive learning setting, a pixel-based random sampling strategy is used to select labeled pixels for training, and unlabeled pixels from the rest of the image can also be used when building a classifier by using information from the neighborhood of training pixels. The overall number of labeled pixels for each dataset is reported in Table 1. Unlabeled (test) pixels for each of the considered hyperspectral images are shown in Figure 1.

2.2.2. Non-Overlapping Learning Setting

The transductive learning setting used in spectral-spatial methods has an intrinsic positive bias due to the use of the neighborhood of each training pixel in the image, resulting in an overlap between training and test samples. In order to investigate such bias, we consider also a non-overlapping learning setting, where only training samples, i.e., the labeled pixels initially selected for training, are used for building a classifier.

In particular, we propose to randomly select a single patch of pixels for each class to use as training data. We use a patch of $7\times 7$ labeled pixels for each class as a training set, which ensures that we have enough training pixels (at most 49) per class.

In general, there is a mismatch between both learning settings used in spectral-spatial hyperspectral image classification and the standard supervised learning. In supervised learning, methods are tested on independent identically distributed (i.i.d.) data. Therefore, in the case of image datasets, methods should be trained on a set of images that are independent of the test set images. Therefore, even if we do not use pixels other than those selected by our sampling procedure, this does not guarantee that the samples in our training set are independent. Hence, our controlled random sampling procedure is not a proper supervised learning setting. Nevertheless, this setting is useful for assessing the performance of methods without the bias caused by overlap between the training and testing samples.

In [39], another controlled random sampling method to select labeled pixels was introduced. The proposed method considers connected component areas in the image, consisting of pixels with equal class. For each such area, pixels are randomly sampled. Each selected pixel and its 8 neighbors form the training set. Pixels in the rest of the image are only used at test time. See [39] (Algorithm 1) for a detailed description of this method. Although this procedure is interesting for assessing the performance of spectral-spatial methods, it is impractical, since one would have to know the class composition of the whole image in order to perform the step ‘selects all unconnected partitions Pin the class c’. Our controlled random sampling procedure overcomes this drawback because it does not use information on the class distribution and selects pixels by randomly sampling a patch for each class.

Algorithm 1: CNN-RSL.

Input:  Hyperspectral image P partitioned into a labeled training set $(\mathbf{x},\mathbf{y})$ and an unlabeled test set ${\mathbf{x}}^{\mathrm{test}}$. Output:  Predicted labels ${\widehat{\mathbf{y}}}^{\mathrm{test}}$ of ${\mathbf{x}}^{\mathrm{test}}$. 1: $(\mathbf{x},\mathbf{y})=\mathrm{Augment}-\mathrm{train}-\mathrm{set}(\mathbf{x},\mathbf{y},P)$ (see Equations (2) and (4), Section 2.6) 2: $\mathcal{M}=\mathrm{Train}-\mathrm{model}(\mathbf{x},\mathbf{y},P)$ (uses loss Equation (3)) 3: ${\widehat{\mathbf{y}}}^{\mathrm{test}}=\mathrm{Predict}-\mathrm{test}-\mathrm{labels}({\mathbf{x}}^{\mathrm{test}},\mathcal{M})$ 4: return${\widehat{\mathbf{y}}}^{\mathrm{test}}$

2.3. The Baseline CNN

The baseline on which we build our method is a Convolutional Neural Network (CNN) with a single hidden convolutional layer (see Figure 2). Unlike in larger CNN architectures, we do not use pooling or fully-connected hidden layers. We chose this simple architecture because of the limited amount of labeled training data and the relatively high number of features. In this context, a simpler architecture has fewer parameters to learn, which reduces the risk of overfitting.

For training, we use the standard L2 regularized cross-entropy loss function,

$$\begin{array}{cc}\hfill L_{\mathrm{CNN}}\left(\mathit{W}\right)=& \underset{\mathrm{Cross}-\mathrm{entropy}\mathrm{loss}}{\underbrace{-\frac{1}{\left|I\right|}\sum _{i=1}^{\left|I\right|}\sum _{k=1}^K{y}_{i,k}log{\widehat{y}}_{i,k}}}+{\lambda }_1\underset{\text{L2}\mathrm{regularization}}{\underbrace{{∥\mathit{W}∥}^2}}.\hfill \end{array}$$

(1)

Here, ${\widehat{y}}_i\equiv {\psi }_2\left({\mathit{w}}_{\mathbf{2}}·{\psi }_1({\mathit{w}}_{\mathbf{1}}\ast {\mathit{x}}_i)\right)$ is the network’s output, ${\psi }_1(-)$ and ${\psi }_2(-)$ are the activation functions and $\mathit{W}=[{\mathit{w}}_{\mathbf{1}},{\mathit{w}}_{\mathbf{2}}]$ are the weights from the input to the single hidden layer (${\mathit{w}}_{\mathbf{1}}$) and from the hidden layer to the output (${\mathit{w}}_{\mathbf{2}}$). K is the number of classes. For the hidden layer, we use a rectified linear activation function ${\psi }_1\left(u\right)=max(0,u)$, and for the prediction layer, we use softmax, ${\psi }_2{\left(\mathbf{u}\right)}_k=exp\left(u_k\right)/{\sum }_{l}exp\left(u_l\right)$.

To learn the weights of the neural network(s), which optimize this loss function, we use the common ‘Glorot’ procedure for initializing the weights [40] and Stochastic Gradient Descent (SGD) [41] for updating them.

The standard parameters of a CNN are:

• learning rate ($\eta$);
• momentum (default value used in experiments: $0.7$);
• number of convolutional kernels ($\#kernels$);
• size of the convolutional kernels (N);
• stride for the convolution (s);
• L2 regularization constant (${\lambda }_1$).

To enhance the robustness of CNN to perturbed versions of the data, we add random noise to copies of the original data, and then, we add these copies to the original data to increase the amount of available training data. In absence of any further knowledge, it is natural to use Gaussian noise.

The new spectrum of a pixel is generated by adding random Gaussian noise to the original wavelengths of the spectrum as follows:

$$P_{ijk}^{\mathrm{noise}}=P_{ijk}+\beta ·{\mathit{ϵ}}_{ijk},$$

(2)

where $P_{ijk}^{\mathrm{noise}}$ is the k-th wavelength of the new $(i,j)$-th spectral pixel, generated by perturbing $P_{ijk}$, with the addition of Gaussian noise ${\mathit{ϵ}}_{ijk}$ having zero mean and unit variance. $\beta$ is a constant term that we fixed at 0.01. This procedure is applied to all the pixels in the training set.

We use this noise-based data augmentation together with the CNN as a baseline. In the following sections, we describe three tricks to enhance CNN by exploiting spectral-spatial locality:

• by constraining weights of the neural network corresponding to nearby wavelengths to assume similar values (see Section 2.4);
• by generating pixels with smoothed spectra from neighbors of labeled pixels (see Section 2.5);
• by propagating the label of a pixel to its neighbors and adding them to the training set (see Section 2.6).

2.4. Trick 1: Locality-Aware Regularization

We add a term to the CNN loss function, which penalizes large differences between values of adjacent weights, as done in [42]. In this way we enforce that neighboring wavelengths have similar contributions to the generated features, thus taking advantage of the spectral-locality of the data. The augmented loss function consists of the regularized cross-entropy loss term plus our regularization term, which constrains nearby weights to assume similar values:

$$\begin{array}{cc}\hfill L\left(\mathit{W}\right)=& \underset{\mathrm{Cross}-\mathrm{entropy}\mathrm{loss}}{\underbrace{-\frac{1}{\left|I\right|}\sum _{i=1}^{\left|I\right|}\sum _{k=1}^K{y}_{i,k}log{\widehat{y}}_{i,k}}}+{\lambda }_1\underset{\text{L2} \mathrm{regularization}}{\underbrace{{∥\mathit{W}∥}^2}}+{\lambda }_2\underset{\text{Locality-aware} \mathrm{regularization}}{\underbrace{{∥{\mathit{w}}_{\mathbf{1}}-\mathrm{shift}\left({\mathit{w}}_{\mathbf{1}}\right)∥}^2}}.\hfill \end{array}$$

(3)

Here, the variables are as in Equation (1). $\mathrm{shift}\left(·\right)$ is an operation that shifts the elements of an array one position to the left, and ${\lambda }_2$ controls the new spectral-locality-aware regularization term.

2.5. Trick 2: Smoothing-Based Data Augmentation

Spectra of nearby pixels are assumed to be related because they are part of an image containing semantically homogeneous components, such as urban or rural areas.

Recent state-of-the-art methods exploit this property in different ways, such as the use of patches to train a (deep) neural network [18,20,33], the generation of discriminatory features using Gabor filters [12,36] or the use of additive Gaussian noise in addition to linear combinations of training pixels [28].

Here, we just use a Gaussian smoothing filter, because of its simplicity and invariance to rotation of the image. This operation has been called spatial smoothing [12,22,43].

Keeping the same notation introduced in Section 2, we generate the smoothed image $P^{\mathrm{smt}}$ as:

$$P_{ijk}^{\mathrm{smt}}=\frac{{\sum }_{i^{\prime }}{\sum }_{j^{\prime }}{P}_{ijk}^{\mathrm{noise}}exp(-\parallel (i,j)-(i^{\prime },j^{\prime }){\parallel }^2/2\sigma )}{{\sum }_{i^{\prime }}{\sum }_{j^{\prime }}exp(-\parallel (i,j)-(i^{\prime },j^{\prime }){\parallel }^2/2\sigma )},$$

(4)

where $P_{ijk}$ is the k-th wavelength of the $(i,j)$-th spectral pixel and $P_{ijk}^{\mathrm{smt}}$ is the new smoothed wavelength.

In practice, the above sum is computed over pixels $(i^{\prime },j^{\prime })$ whose distance from pixel $(i,j)$ is at most $3\sigma$. Figure 3 shows two pixel-spectra of the same class before and after spatial smoothing is applied.

2.6. Trick 3: Label-Based Data Augmentation

We also exploit spectral-spatial locality with data augmentation at the semantic level, by assuming that neighbor pixels are likely to have the same class. According to this assumption, the label of a pixel in the training set can be propagated to its neighbors. The resulting labeled neighbor pixels are inserted into the training set, which becomes larger at the cost of introducing label-noise. Indeed, this data augmentation procedure is likely to add new pixels with an incorrect label, and even copies of the same pixel labeled in different ways. In this way, the network is trained using a training set that contains pixels with uncertainty on their label.

In order to keep the probability that our assumption is wrong as low as possible, we randomly sample only a subset of pixels in the Moore neighborhood of each pixel (consisting of its 8 surrounding pixels).

Furthermore, we can use this augmentation step to tackle the class unbalance, by favoring the selection of pixels in smaller classes. Specifically, for pixel i in the training set with label $y_i$ and for each pixel j in its neighborhood, j is selected with probability:

$$p\left(\mathrm{select}j\right)=1-\frac{C_{y_i}-min\left(C\right)}{max\left(C\right)-min\left(C\right)},$$

(5)

where $C_{y_i}$ is the number of pixels in the training set with label $y_i$, and $C=[C_1,\cdots ,C_K]$ is the vector consisting of the number of pixels of each class. All selected neighbors are added to the (multi-)set I of labeled pixels to give $I^{\mathrm{la}}$. For any $j\in I^{\mathrm{la}}$ that was added with label augmentation, its label will be $y_j^{\mathrm{la}}=y_i$. This selection procedure biases the insertion of more pixels from smaller classes.

In summary, our label-based data augmentation procedure can be described as follows: for each pixel i in the training set,

• find its Moore neighborhood;
• select a subset of pixels in the neighborhood; see Equation (5);
• propagate the label of i to the selected neighbor pixels;
• insert the selected pixels into the training set.

2.7. Incorporating the Tricks into the Baseline CNN: CNN-RSL

The resulting method for hyperspectral image labeling, called CNN-RSL, incorporates into CNN the proposed three tricks: Regularization (R), Smoothing-based data augmentation (S) and Label-based data augmentation (L).

The ‘augment-train-set’ step of Algorithm 1 (also illustrated in Figure 4) consists of the following steps:

• the original image is perturbed with random Gaussian noise Equation (2) (see Section 2.3);
• the resulting image is spatially smoothed (S step);
• label augmentation is applied (L step);
• the spectra for the labeled pixels are selected from the original, noisy and smoothed images; these are combined to form the training set;
• the spectra are rescaled between $[0,1]$, which is a common practice for artificial neural networks. Note that this rescaling retains the original distribution of the features, while helping the CNN training to converge faster.

The resulting algorithm, called CNN-RSL, is summarized in pseudo-code below Algorithm 1.

3. Experiments

In this section, we describe the experiments conducted on the five groups of hyperspectral images. First, we describe the 16 algorithms considered in our experiments. Next, we report the results, which we also compare with published results from existing methods based on different approaches. Finally, we discuss these results.

3.1. Algorithms

We assess the performance of our baseline CNN with all combinations of the proposed tricks:

• R: spectral-locality-aware regularization term (see Section 2.4);
• S: smoothing-based data augmentation (see Section 2.5);
• L: label-based data augmentation (see Section 2.6).

The combination of the CNN with all three tricks yields CNN-RSL, while the other six combinations are: CNN-R (with R), CNN-S (with S), CNN-L (with L), CNN-RS (with R and S), CNN-RL (with R and L) and CNN-SL (with S and L).

Moreover, in order to investigate the effect of these tricks on other types of neural networks, we incorporate S and L also in the following methods:

• SVM-RBF: a support vector machine with the Radial Basis Function (RBF) kernel;
• HL-ELM: a deep convolutional neural network for hyperspectral image labeling for which we were able to retrieve the source code. HL-ELM has two convolutional and two max pooling hidden layers arranged one after the other (see [28]).

3.2. Parameter Setting

The parameters of the resulting neural networks and the range of values used in our experiments are:

• the number of kernels of the convolutional layer, $\#kernels\in \{4,8,16,32\}$;
• the size of kernels of the convolutional layer, $N\in [2,91]$;
• the stride for the convolution, $s\in [1,4]$;
• the parameters in the regularization terms, ${\lambda }_1,{\lambda }_2={10}^n$ for $n\in [-4,4]$;
• the learning rate, $\eta ={10}^n$ for $n\in [-4,-1]$.

In order to tune these parameters, we use the standard Random Grid Search Cross-Validation framework (RGS-CV) [44]. Resulting values of the CNN-RSL parameters are given in

Table 2. Parameter values of CNN-RSL trained with 1% labeled pixels of each class using Random Grid Search Cross-Validation framework (RGS-CV).

Dataset	$\#\mathit{kernels}$	N	s	${\mathit{\lambda }}_1$	${\mathit{\lambda }}_2$	$\mathit{\eta }$	$\mathit{\sigma }$
Pavia Center	32	47	1	0.001	0.1	0.001	2.33
Pavia University	32	35	1	0.010	0.1	0.001	2.33
KSC	16	51	1	0.010	0.1	0.001	3.00
Indian Pines	16	53	1	0.001	0.1	0.001	3.67
Salinas	32	49	1	0.001	0.1	0.001	4.33

.

We also use RGS-CV to select the value of $\sigma$, the parameter of our spatial smoothing procedure, from the set $\{1,1.67,2.33,3,3.67,4.33,5\}$.

Like our neural network model, SVM-RBF has also a few parameters to be tuned:

• the Gaussian exponent constant $\gamma \in {10}^n$ where $n\in [-4,4]$,
• the regularization constant $\mathrm{C}\in {10}^n$ where $n\in [-4,4]$.

For HL-ELM, we use the parameter setting described in [28].

3.3. Results and Discussion

The results of the experiments with few labeled pixels for training are given in

Table 3. Test set classification accuracy on the Pavia Center dataset, when using 1–5% randomly sampled labeled pixels per class for training. We report the mean and standard deviation over 10 runs. We also list the mean and standard deviation of the number of training pixels per class for each training %. The best accuracy for each training set is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

	Train %	1% 165$\pm$211	2% 329$\pm$422	3% 494$\pm$633	4% 658$\pm$845	5% 823$\pm$1056
Method
CNN		97.79$\pm$0.22 *	97.98$\pm$0.20 *	98.14$\pm$0.25 *	98.57$\pm$0.29 *	98.69$\pm$0.25 *
CNN-S		99.23$\pm$0.16 *	99.26$\pm$0.15 *	99.24$\pm$0.12 *	99.28$\pm$0.11 *	99.35$\pm$0.22 *
CNN-L		98.48$\pm$0.18 *	98.81$\pm$0.20 *	99.00$\pm$0.27 *	99.09$\pm$0.14 *	99.20$\pm$0.22 *
CNN-R		98.01$\pm$0.28 *	98.32$\pm$0.25 *	98.41$\pm$0.22 *	98.63$\pm$0.30 *	98.79$\pm$0.22 *
CNN-RS		99.21$\pm$0.88 *	99.28$\pm$0.68 *	99.28$\pm$0.51 *	99.33$\pm$0.58*	99.41$\pm$0.35 *
CNN-SL		98.11$\pm$0.96 *	98.59$\pm$0.85 *	98.84$\pm$0.74 *	98.96$\pm$0.59 *	99.12$\pm$0.54 *
CNN-RL		98.76$\pm$0.39 *	98.86$\pm$0.45 *	98.95$\pm$0.41 *	99.20$\pm$0.43 *	99.34$\pm$0.29 *
CNN-RSL		99.52 ± 0.07	99.67 ± 0.15	99.72 ± 0.04	99.76 ± 0.07	99.82 ± 0.05
SVM-RBF		84.31$\pm$2.51 *	85.41$\pm$1.77 *	85.89$\pm$1.65 *	87.04$\pm$1.83 *	87.19$\pm$1.67 *
SVM-RBF-S		99.05$\pm$0.52 *	99.21$\pm$0.51 *	99.50$\pm$0.34 *	99.60$\pm$0.38 *	99.64$\pm$0.33 *
SVM-RBF-L		90.12$\pm$5.24 *	92.14$\pm$4.26 *	92.25$\pm$1.85 *	92.84$\pm$0.95 *	92.80$\pm$0.82 *
SVM-RBF-SL		98.95$\pm$0.51 *	99.09$\pm$0.57 *	99.25$\pm$0.62 *	99.36$\pm$0.41 *	99.45$\pm$0.49 *
HL-ELM		96.22$\pm$0.08 *	97.27$\pm$0.11 *	97.78$\pm$0.08 *	98.09$\pm$0.11 *	98.22$\pm$0.10 *
HL-ELM-S		98.75$\pm$0.17 *	98.92$\pm$0.18 *	99.17$\pm$0.13 *	99.28$\pm$0.12 *	99.30$\pm$0.09 *
HL-ELM-L		96.32$\pm$0.15 *	97.54$\pm$0.22 *	97.87$\pm$0.17 *	98.12$\pm$0.22 *	98.29$\pm$0.14 *
HL-ELM-SL		99.05$\pm$0.23 *	99.27$\pm$0.25 *	99.40$\pm$0.10 *	99.43$\pm$0.06 *	99.52$\pm$0.07 *

,

Table 4. Test set classification accuracy on the Pavia University dataset, when using 1–5% randomly sampled labeled pixels per class for training. We report the mean and standard deviation over 10 runs. We also list the mean and standard deviation of the number of training pixels per class for each training %. The best accuracy for each training set is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

	Train %	1% 48$\pm$52	2% 95$\pm$104	3% 143$\pm$157	4% 190$\pm$209	5% 238$\pm$261
Method
CNN		88.76$\pm$0.20 *	89.04$\pm$0.26 *	90.16$\pm$0.27 *	91.52$\pm$0.23 *	92.79$\pm$0.21 *
CNN-S		95.01 ± 0.74	95.55$\pm$0.68 *	95.87$\pm$0.59 *	95.98$\pm$0.46 *	96.25$\pm$0.72*
CNN-L		88.77$\pm$0.13 *	89.51$\pm$0.02 *	88.79$\pm$0.49 *	89.05$\pm$0.81 *	89.20$\pm$0.83 *
CNN-R		89.04$\pm$0.51 *	89.86$\pm$0.45 *	90.29$\pm$0.50 *	91.50$\pm$0.36 *	92.65$\pm$0.45 *
CNN-RS		94.25$\pm$0.49*	95.61$\pm$0.52*	95.92$\pm$0.47*	96.15$\pm$0.53*	96.51$\pm$0.29*
CNN-SL		91.12$\pm$1.15 *	92.17$\pm$0.95 *	92.53$\pm$0.87 *	93.10$\pm$0.61 *	93.32$\pm$0.41 *
CNN-RL		88.51$\pm$0.55 *	88.91$\pm$0.58 *	89.13$\pm$0.49 *	89.62$\pm$0.52 *	89.98$\pm$0.77 *
CNN-RSL		94.74$\pm$0.25	96.36 ± 0.68	96.54 ± 0.46	96.65 ± 0.31	96.70 ± 0.44
SVM-RBF		75.96$\pm$2.56 *	74.85$\pm$0.78 *	74.92$\pm$1.41 *	75.43$\pm$1.26 *	75.98$\pm$1.96 *
SVM-RBF-S		89.18$\pm$4.23 *	91.77$\pm$3.59 *	93.20$\pm$3.57 *	93.74$\pm$4.21 *	94.35$\pm$3.68 *
SVM-RBF-L		57.51$\pm$2.67 *	79.06$\pm$1.47 *	82.56$\pm$2.86 *	82.82$\pm$0.86 *	90.51$\pm$1.24 *
SVM-RBF-SL		91.68$\pm$2.28 *	93.89$\pm$1.23 *	95.00$\pm$0.78 *	95.48$\pm$2.59 *	95.56$\pm$0.59 *
HL-ELM		75.77$\pm$2.02 *	78.90$\pm$2.13 *	82.26$\pm$2.55 *	83.27$\pm$2.18 *	86.51$\pm$2.27 *
HL-ELM-S		90.75$\pm$0.75 *	92.91$\pm$0.81 *	93.85$\pm$0.57 *	94.37$\pm$0.26 *	95.23$\pm$0.19 *
HL-ELM-L		75.96$\pm$1.76 *	81.41$\pm$2.31 *	83.98$\pm$1.95 *	85.59$\pm$1.72 *	86.81$\pm$1.63 *
HL-ELM-SL		92.79$\pm$0.91 *	94.74$\pm$0.79 *	95.56$\pm$0.58 *	96.21$\pm$0.36 *	96.34$\pm$0.18*

,

Table 5. Test set classification accuracy on the KSC dataset, when using 1–5% randomly sampled labeled pixels per class for training. We report the mean and standard deviation over 10 runs. We also list the mean and standard deviation of the number of training pixels per class for each training %. The best accuracy for each training set is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

	Train %	1% 4$\pm$2	2% 8$\pm$5	3% 12$\pm$7	4% 16$\pm$9	5% 20$\pm$11
Method
CNN		78.09$\pm$0.99 *	83.98$\pm$0.92 *	85.37$\pm$1.21 *	87.06$\pm$1.25 *	88.27$\pm$1.75 *
CNN-S		84.74$\pm$1.32 *	85.95$\pm$0.78 *	86.48$\pm$0.72 *	88.61$\pm$0.66 *	90.18$\pm$0.52 *
CNN-L		84.65$\pm$1.85 *	88.02$\pm$1.70 *	91.94$\pm$0.99 *	93.58$\pm$0.36 *	94.21$\pm$0.15 *
CNN-R		80.24$\pm$1.44 *	84.00$\pm$0.82 *	85.41$\pm$0.94 *	88.56$\pm$0.81 *	90.19$\pm$0.59 *
CNN-RS		84.95$\pm$1.14 *	85.86$\pm$0.70 *	86.50$\pm$0.85 *	88.95$\pm$0.91 *	90.09$\pm$0.32 *
CNN-SL		87.07$\pm$0.85 *	90.24$\pm$0.78 *	92.85$\pm$0.71 *	95.05$\pm$0.66 *	97.28$\pm$0.53 *
CNN-RL		84.78$\pm$1.83 *	88.53$\pm$0.80 *	92.65$\pm$0.77 *	93.66$\pm$0.80 *	94.09$\pm$0.42 *
CNN-RSL		90.34$\pm$0.97	95.36$\pm$0.02	97.22$\pm$0.37	98.80$\pm$0.60	99.79$\pm$0.20
SVM-RBF		67.85$\pm$1.97 *	76.17$\pm$2.85 *	79.87$\pm$2.59 *	79.17$\pm$3.25 *	82.45$\pm$2.62 *
SVM-RBF-S		89.03$\pm$2.04 *	90.87$\pm$3.16 *	93.51$\pm$3.05 *	95.36$\pm$2.53 *	96.76$\pm$1.98 *
SVM-RBF-L		81.10$\pm$2.81 *	87.14$\pm$1.59 *	87.66$\pm$2.51 *	91.15$\pm$1.95 *	92.47$\pm$2.06 *
SVM-RBF-SL		89.14$\pm$3.15 *	90.74$\pm$1.65 *	91.12$\pm$2.54 *	96.00$\pm$1.56 *	97.38$\pm$0.85 *
HL-ELM		79.21$\pm$0.65 *	81.88$\pm$1.36 *	82.79$\pm$0.95 *	83.87$\pm$1.23 *	86.21$\pm$0.92 *
HL-ELM-S		81.52$\pm$2.42 *	86.95$\pm$1.24 *	90.58$\pm$1.12 *	92.17$\pm$0.87 *	93.46$\pm$0.51 *
HL-ELM-L		79.60$\pm$1.54 *	83.14$\pm$0.98 *	83.56$\pm$0.64 *	85.35$\pm$0.53 *	86.48$\pm$0.45 *
HL-ELM-SL		88.11$\pm$2.52 *	91.31$\pm$1.14 *	93.64$\pm$1.26 *	95.72$\pm$0.82 *	96.77$\pm$0.58 *

,

Table 6. Test set classification accuracy on the Indian Pines dataset, when using 1–5% randomly sampled labeled pixels per class for training. We report the mean and standard deviation over 10 runs. We also list the mean and standard deviation of the number of training pixels per class for each training %. The best accuracy for each training set is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

	Train %	1% 8$\pm$6	2% 17$\pm$12	3% 25$\pm$18	4% 34$\pm$24	5% 42$\pm$30
Method
CNN		54.83$\pm$0.23 *	60.83$\pm$0.87 *	64.56$\pm$0.74 *	67.58$\pm$0.67 *	70.83$\pm$0.77 *
CNN-S		70.93$\pm$0.40 *	78.59$\pm$0.54 *	83.88$\pm$0.65 *	87.77$\pm$0.70 *	90.51$\pm$0.53 *
CNN-L		65.79$\pm$0.51 *	71.83$\pm$0.42 *	76.34$\pm$0.46 *	78.54$\pm$0.27 *	80.36$\pm$0.88 *
CNN-R		56.24$\pm$0.39 *	61.12$\pm$0.45 *	64.89$\pm$0.55 *	67.64$\pm$0.50 *	71.03$\pm$0.83 *
CNN-RS		72.63$\pm$0.38 *	78.63$\pm$0.44 *	84.01$\pm$0.58 *	88.32$\pm$0.57 *	90.83$\pm$0.49 *
CNN-SL		68.11$\pm$0.54 *	72.67$\pm$0.38 *	76.71$\pm$0.53 *	78.76$\pm$0.57 *	80.85$\pm$0.38 *
CNN-RL		66.02$\pm$0.43 *	72.00$\pm$0.53 *	76.14$\pm$0.37 *	78.82$\pm$0.46 *	80.78$\pm$0.61 *
CNN-RSL		86.42$\pm$0.66	92.70$\pm$0.80	94.45$\pm$0.92	96.00$\pm$0.38	96.42$\pm$0.24
SVM-RBF		58.75$\pm$0.49 *	60.58$\pm$0.36 *	61.47$\pm$0.28 *	63.81$\pm$0.47 *	64.45$\pm$0.32 *
SVM-RBF-S		77.23$\pm$2.90 *	83.23$\pm$2.46 *	86.44$\pm$3.01 *	88.91$\pm$2.78 *	89.52$\pm$2.67 *
SVM-RBF-L		65.74$\pm$2.92 *	69.47$\pm$1.43 *	70.38$\pm$1.35 *	77.18$\pm$1.77 *	77.90$\pm$1.59 *
SVM-RBF-SL		85.14$\pm$2.53 *	90.12$\pm$1.91 *	92.95$\pm$1.57 *	93.24$\pm$1.72 *	94.17$\pm$0.86 *
HL-ELM		66.29$\pm$0.51 *	71.69$\pm$1.24 *	74.28$\pm$0.79 *	76.60$\pm$0.64 *	78.04$\pm$0.52 *
HL-ELM-S		73.88$\pm$0.54 *	82.48$\pm$0.96 *	86.51$\pm$0.74 *	88.49$\pm$0.68 *	90.58$\pm$0.55 *
HL-ELM-L		66.34$\pm$0.14 *	73.15$\pm$0.11 *	73.19$\pm$0.21 *	76.85$\pm$0.16 *	78.27$\pm$0.19 *
HL-ELM-SL		82.05$\pm$0.96 *	88.12$\pm$0.54 *	91.39$\pm$0.49 *	93.37$\pm$0.53 *	94.41$\pm$0.45 *

,

Table 7. Test set classification accuracy on the Salinas dataset when using 1–5% randomly sampled labeled pixels per class for training. We report the mean and standard deviation over 10 runs. We also list the mean and standard deviation of the number of training pixels per class for each training %. The best accuracy for each training set is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

	Train %	1% 34$\pm$27	2% 68$\pm$54	3% 101$\pm$81	4% 135$\pm$107	5% 169$\pm$134
Method
CNN		87.13$\pm$0.42 *	88.21$\pm$0.37 *	88.54$\pm$0.22 *	89.46$\pm$0.57 *	91.14$\pm$0.17 *
CNN-S		96.87$\pm$0.35 *	97.05$\pm$0.29 *	97.17$\pm$0.31 *	97.21$\pm$0.17 *	97.52$\pm$0.54 *
CNN-L		87.13$\pm$0.59 *	87.93$\pm$0.52 *	88.29$\pm$0.37 *	88.45$\pm$0.36 *	89.56$\pm$0.75 *
CNN-R		88.19$\pm$0.37 *	89.15$\pm$0.45 *	89.98$\pm$0.40 *	90.56$\pm$0.32 *	91.27$\pm$0.22 *
CNN-RS		94.93$\pm$0.65 *	97.02$\pm$0.51 *	97.14$\pm$0.57 *	97.30$\pm$0.72 *	97.57$\pm$0.60 *
CNN-SL		93.15$\pm$0.84 *	93.96$\pm$0.73 *	94.23$\pm$0.68 *	95.17$\pm$0.52 *	96.25$\pm$0.43 *
CNN-RL		86.82$\pm$0.77 *	87.02$\pm$0.65 *	87.87$\pm$0.60 *	88.02$\pm$0.54 *	88.36$\pm$0.46 *
CNN-RSL		96.93$\pm$0.55	97.16$\pm$0.69	97.68$\pm$0.78	98.21$\pm$0.41	99.03$\pm$0.17
SVM-RBF		72.38$\pm$1.85 *	73.51$\pm$2.67 *	73.78$\pm$2.54 *	74.36$\pm$2.16 *	74.84$\pm$2.04 *
SVM-RBF-S		93.35$\pm$3.41 *	94.68$\pm$2.87 *	96.18$\pm$2.26 *	96.87$\pm$2.59 *	97.16$\pm$1.85 *
SVM-RBF-L		84.29$\pm$2.85 *	85.36$\pm$2.16 *	85.87$\pm$2.14 *	86.68$\pm$2.27 *	87.16$\pm$1.55 *
SVM-RBF-SL		96.23$\pm$0.47 *	96.31$\pm$0.36 *	97.38$\pm$0.52 *	97.47$\pm$0.75 *	97.68$\pm$0.48 *
HL-ELM		87.50$\pm$0.53 *	88.95$\pm$0.74 *	89.38$\pm$0.45 *	91.02$\pm$0.40 *	91.36$\pm$0.25 *
HL-ELM-S		92.16$\pm$0.78 *	94.64$\pm$0.89 *	95.36$\pm$0.45 *	96.17$\pm$0.41 *	96.42$\pm$0.48 *
HL-ELM-L		87.77$\pm$0.56 *	89.56$\pm$0.46 *	90.16$\pm$0.37 *	91.11$\pm$0.32 *	91.87$\pm$0.40 *
HL-ELM-SL		93.51$\pm$0.92 *	95.44$\pm$0.83 *	96.49$\pm$0.35 *	97.41$\pm$0.26 *	97.81$\pm$0.20 *

and

Table 8. Average classification accuracy of test data over 10 runs using 10 randomly sampled labeled pixels per class for training. The best accuracy for each dataset is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

Method	Pavia Center	Pavia University	KSC	Indian Pines	Salinas
CNN	92.21$\pm$0.56 *	85.51$\pm$1.65 *	90.89$\pm$2.83 *	76.75$\pm$0.56 *	88.58$\pm$2.15 *
CNN-S	97.44$\pm$0.38 *	89.12$\pm$1.10 *	93.62$\pm$2.74 *	80.59$\pm$0.10 *	91.31$\pm$0.79 *
CNN-L	96.57$\pm$0.48 *	88.36$\pm$1.39 *	92.91$\pm$2.54 *	79.77$\pm$0.52 *	90.66$\pm$1.42 *
CNN-R	92.84$\pm$0.44 *	86.11$\pm$1.22 *	91.31$\pm$2.14 *	77.50$\pm$0.49 *	89.01$\pm$1.54 *
CNN-RS	97.87$\pm$0.29 *	90.30$\pm$2.11 *	94.41$\pm$2.03 *	81.25$\pm$0.12 *	91.68$\pm$0.84 *
CNN-SL	97.25$\pm$0.41 *	88.67$\pm$1.02 *	93.10$\pm$1.95 *	80.03$\pm$0.46 *	92.81$\pm$0.79 *
CNN-RL	96.88$\pm$0.43 *	88.39$\pm$1.18 *	93.04$\pm$2.11 *	82.92$\pm$0.41 *	91.75$\pm$0.93 *
CNN-RSL	98.65$\pm$0.37	95.76$\pm$1.14	97.85$\pm$1.35	83.96$\pm$0.60	93.01$\pm$1.44
SVM-RBF	90.04$\pm$0.64 *	83.27$\pm$1.80 *	88.71$\pm$2.76 *	76.49$\pm$0.73 *	87.59$\pm$2.81 *
SVM-RBF-S	90.82$\pm$0.72 *	89.49$\pm$1.46 *	80.86$\pm$1.94 *	78.32$\pm$0.91 *	85.19$\pm$1.61 *
SVM-RBF-L	90.65$\pm$0.79 *	89.05$\pm$1.65 *	80.16$\pm$1.88 *	78.02$\pm$1.14 *	84.78$\pm$1.82 *
SVM-RBF-SL	91.08$\pm$0.58 *	89.79$\pm$1.37 *	81.47$\pm$1.85 *	78.96$\pm$0.83 *	85.69$\pm$1.62 *
HL-ELM	90.89$\pm$0.85 *	83.47$\pm$1.92 *	85.10$\pm$2.74 *	76.22$\pm$0.68 *	84.19$\pm$1.69 *
HL-ELM-S	78.44$\pm$0.51 *	88.59$\pm$1.80 *	91.12$\pm$1.74 *	78.33$\pm$0.61 *	84.79$\pm$1.55 *
HL-ELM-L	77.81$\pm$0.61 *	86.19$\pm$1.85 *	90.96$\pm$1.83 *	77.53$\pm$0.75 *	84.58$\pm$1.73 *
HL-ELM-SL	91.20$\pm$0.67 *	90.07$\pm$1.77 *	92.50$\pm$1.53 *	79.31$\pm$0.66 *	85.71$\pm$1.33 *

. CNN-RSL achieved the best performance, with significant improvement over the baselines. On the Pavia University dataset with 1% training data, CNN-S was most effective (95.01 mean accuracy), closely followed by CNN-RSL (94.74). On the Salinas dataset, the improvement in accuracy from CNN to both CNN-RSL and CNN-S was about 10%. In this case, CNN-RSL was slightly better than CNN-S. On the other hand, with 1% training data, on the KSC dataset, the improvement in accuracy from CNN to CNN-RSL was about 12% (from 78.09–90.34), while CNN-S achieved a 84.74 mean accuracy; and on the Indian Pines dataset, the improvement in accuracy from CNN to CNN-RSL was more than 30% (from 54.83–86.42), while CNN-S achieved a 70.93 mean accuracy. Overall, statistical tests showed the superiority of our method when using all tricks. The increase in accuracy with respect to CNN, SVM-RBF and HL-ELM was higher when fewer training pixels were used, since smoothing- and label-based data augmentation were more beneficial in that case. Clearly, these two tricks also helped to improve the performance of SVM-RBF and HL-ELM. As expected, by increasing the number of training pixels per class, the average test accuracy of all methods increased.

Existing methods using a transductive setting, such as [18,19,21], have been shown to achieve very good results on these datasets when 200 labeled pixels for each class are used as the training set. Our method also achieves excellent performance in this setting: it improves significantly over the considered baselines, according to a binomial test for comparing classifiers [45]; see

Table 9. Test set classification accuracy when using 200 randomly sampled labeled pixels per class for training. We report the mean and standard deviation over 10 runs. The best accuracy for each dataset is indicated in bold. With the exception of methods from [18,19], an ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

Method	Pavia Center	Pavia University	KSC	Indian Pines	Salinas
CNN-RSL	99.52$\pm$0.11	97.85$\pm$0.07	99.87$\pm$0.17	97.73$\pm$0.22	98.54$\pm$0.41
SVM-RBF	96.26$\pm$2.38 *	78.31$\pm$9.59 *	92.96$\pm$3.81 *	71.18$\pm$5.56 *	87.36$\pm$2.18 *
SVM-RBF-SL	98.67$\pm$0.14 *	90.02$\pm$0.19 *	98.92$\pm$0.09 *	95.12$\pm$3.58 *	97.29$\pm$2.01 *
Hu et al. [19]	—	92.56	—	—	92.60
Lee & Kwon [18]	—	—	—	92.06	—
HL-ELM-SL [28]	98.04$\pm$0.09 *	95.74$\pm$0.49 *	93.08$\pm$0.66 *	95.40$\pm$0.54 *	97.84$\pm$0.09 *
MH-KELM [36]	—	80.07$\pm$0.01 *	—	91.75$\pm$0.29 *	—

.

Table 10 reports the results of our method and published results of the following state-of-the-art methods based on different approaches: discriminative low-rank Gabor filtering [12], multiple kernel learning [30], kernel sparse representation [34] and probabilistic class structure regularized sparse representation graph [29,31]. Unfortunately, due to the diversity of choices regarding the number of training pixels, it is not possible to completely fill Table 10. CNN-RSL achieved higher accuracy compared to the other considered methods. Only in three cases, namely on the Pavia University dataset with 1 and 5% pixels and the Indian Pines dataset with 10% pixels available for training, [29,34] reported a higher accuracy than CNN-RSL.

The test set accuracies in the non-overlapping setting are reported in

Table 11. Average classification accuracy of test data over 10 runs in the non-overlapping learning setting (see Section 2.2.2). The best accuracy for each dataset is indicated in bold. An ‘*’ means that the best accuracy is significantly better than the accuracy achieved by the corresponding method according to a binomial test for comparing classifiers [45] (p-value < 0.05).

Method	Pavia Center	Pavia University	KSC	Indian Pines	Salinas
CNN	92.68$\pm$2.35 *	51.03$\pm$8.40 *	66.76$\pm$5.58 *	39.10$\pm$5.15 *	75.66$\pm$3.82 *
CNN-RS	93.38$\pm$3.69	52.74$\pm$5.82	74.86$\pm$4.24	49.22$\pm$3.20	77.90$\pm$3.73
SVM-RBF	80.91$\pm$6.52 *	32.43$\pm$11.35 *	65.92$\pm$6.53 *	24.13$\pm$5.99 *	76.89$\pm$3.78 *
SVM-RBF-S	81.46$\pm$3.53 *	48.44$\pm$9.97 *	68.95$\pm$8.23 *	36.48$\pm$10.97 *	78.29$\pm$3.64 *
HL-ELM	84.72$\pm$2.54 *	49.01$\pm$3.97 *	53.14$\pm$4.65 *	43.85$\pm$5.49 *	71.87$\pm$5.07 *
HL-ELM-S	88.82$\pm$1.79 *	50.28$\pm$3.86 *	64.23$\pm$1.81 *	45.12$\pm$4.76 *	75.84$\pm$1.59 *

. In this setting, spatial smoothing (the S trick) can only be used in a limited way: each pixel in the training set was smoothed using only the other pixels in the training set. Label-based data augmentation (the L trick) cannot be applied any longer, since this step would add new pixels from the image to the training set. Spectral-locality-aware regularization (the R trick) can still be used, since it does not involve the use of pixels that are not in the training set. As we can see, also in the non-overlapping setting, smoothing-based data augmentation helped to achieve a higher accuracy for all the methods we used. Unsurprisingly, since we took a single $7\times 7$ patch of pixels per class as the training data, the performance of the all methods was much lower than in the transductive setting reported in Table 3, Table 4, Table 5, Table 6 and Table 7. In particular, if there is a large variation in the spectra of a single class, we will miss this by using only a single patch per class. Since we did not use label-based data augmentation, our reference method here was CNN-RS.

In general, results in the non-overlapping learning setting showed a large decrease in performance compared with that in the transductive learning setting. Nevertheless, in this setting, spectral-locality-aware regularization and data augmentation, the latter used in a very limited form, were still beneficial, with significant increases in accuracy on the KSC and Indian Pines images.

Overall, the results of all experiments substantiated the beneficial effect of the proposed tricks, which we discuss below.

In general, smoothing-based data augmentation (the S trick) introduces spatial locality into each pixel’s spectrum by averaging it with its neighboring pixels’ spectra. Since neighboring pixels are likely to belong to the same area, spatial smoothing makes nearby spectra look more alike and eases the network’s classification task. Smoothing-based data augmentation has the largest impact on the test accuracy, with significant improvements across all datasets, notably on the Indian Pines and Salinas.

Label augmentation (the L trick) had a bigger impact on small classes, which were also the most difficult to classify correctly, especially in a setting with very few training samples. For a large training set, label-based data augmentation may have a decremental effect, which was nevertheless mitigated or neutralized when used in combination with the other components of CNN-RSL. In particular, label-based data augmentation had a clearly beneficial effect for the KSC and Indian Pines datasets, which were the datasets having more classes and fewer pixels. The label augmentation trick tended to balance the classes by selecting more new training samples from smaller classes. In our experiments with 10 labeled pixels per class (see Table 8), the training set was already class balanced. In this case, label augmentation still improved the results, but the difference was not as large as in the experiments with class unbalanced training data. In particular, for the KSC dataset, a 6% gain in accuracy was achieved by the baseline CNN when using 10 labeled samples per class instead of 2% of randomly selected labeled pixels as the training set (from 83.98–90.89%), although selecting 2% of the data results in 10 samples per class on average (see Table 1). On the other hand, with CNN-RSL, the gain in accuracy was only 2.5% (from 95.36–97.85%). This shows that our data augmentation tricks mitigated the negative effect of the class unbalanced distribution of the training set.

Spectral-locality-aware regularization (the R trick) helped to achieve a higher classification accuracy, when used in conjunction with data augmentation, as can be seen by the reduced accuracy of CNN-SL compared to CNN-RSL. Notably, on the Indian Pines image, with only 1% labeled pixels, a gain of almost 20% was achieved when using locality-aware regularization and data augmentation over using only data augmentation. Locality-aware regularization also helped to improve accuracy on the other datasets, although the gain was not as big as for Indian Pines.

We conclude this section with a discussion about the convergence and run time of CNN-RSL. Figure 5 illustrates the convergence behavior of our loss function during the training of CNN-RSL on one of the datasets used in the experiments. To assess convergence, we use early stopping. The training stops when the validation error does not decrease for at least 100 epochs.

The run time of CNN-RSL depends on the number of pixels and on the value of $\sigma$ used for the spatial smoothing. In fact, the necessary time for spatial smoothing is proportional to both $\sigma$ and the number of pixels. This is a disadvantage of CNN-RSL with respect to algorithms that use deep architectures and no spatial smoothing. However, spatial smoothing can be highly parallelized given that each pixel is smoothed independently of the other pixels. Consequently, its running time can be drastically reduced [47]. In Figure 6,we report the running time of CNN-RSL using a single CPU with a 2300-MHz clock speed. The values refer to the time needed for predicting a single pixel, and they also include the preprocessing time.

4. Conclusions

We have introduced a simple method based on convolutional neural networks and data augmentation for spectral-spatial classification of remotely-sensed hyperspectral images.

The main characteristic of our method is its capability to exploit spectral-spatial information at both the data level (through data augmentation) and at the classifier level (through the locality-aware regularization). We proposed two types of data augmentation: smoothing-based, which constructs new pixels from the spectra of neighbors of the labeled pixels, and label-based data augmentation, which expands the training set with neighbors of the labeled pixels. Smoothing-based data augmentation consistently improves the test accuracy of the tested methods, while the contribution of label-based data augmentation is mostly beneficial for datasets with many small classes and skewed class distributions. Furthermore, we modified the loss function by inserting a term to penalize the difference among networks weights corresponding to nearby wavelengths of the spectra.

Both CNNs and data augmentation have been widely used in hyperspectral classification [23]. Therefore, at first, the contribution of the proposed method seems limited. Nevertheless, CNN-RSL differs from previous methods in two main aspects: (1) the considered CNN architecture is a very basic shallow architecture with only one hidden layer, without pooling and fully-connected layers, which is advantageous because it does not need a large amount of data or computational resources for training as deep neural networks do, and it is more robust to overfitting; (2) we perform data augmentation not only with a rather standard smoothing-based technique, but also with a new label-based technique to favor the selection of pixels in smaller classes, which is beneficial when few labeled pixels are available and when the class distribution is skewed.

An advantage of the proposed method is its modularity, which favors qualitative analysis of the contribution of the single tricks, as well as their embedding in other types of neural networks. The results of our extensive comparative analysis demonstrated the usefulness of the method.

Our data augmentation approach uses neighbors of training pixels, that is test samples, when building a classifier. This transductive learning setting is the natural setting for hyperspectral image classification. When no overlap between training and test data is allowed, our label augmentation strategy cannot be used. Nevertheless, a limited form of smoothing data augmentation and the spectral-locality-aware regularization term can still be used. The results of experiments showed a substantial drop in accuracy with the non-overlapping learning setting.

Our approach considers a single image. In future work, we intend to adapt the approach to multiple images. For instance, in a dynamic setting, where time-series spectral images are given in order to study seasonal changes of vegetation species, we intend to develop multi-channel convolutional neural networks with locality-aware regularization to enforce smooth change in time.

To guarantee full reproducibility of all results and to facilitate direct usage of CNN-RSL, the source code of our method is publicly available at https://bitbucket.org/TeslaH2O/cnn_hyperspectral.