HyperLiteNet: Extremely Lightweight Non-Deep Parallel Network for Hyperspectral Image Classification

Abstract

Deep learning (DL) is widely applied in the field of hyperspectral image (HSI) classification and has proved to be an extremely promising research technique. However, the deployment of DL-based HSI classification algorithms in mobile and embedded vision applications tends to be limited by massive parameters, high memory costs, and the complex networks of DL models. In this article, we propose a novel, extremely lightweight, non-deep parallel network (HyperLiteNet) to address these issues. Based on the development trends of hardware devices, the proposed HyperLiteNet replaces the deep network by the parallel structure in terms of fewer sequential computations and lower latency. The parallel structure can extract and optimize the diverse and divergent spatial and spectral features independently. Meanwhile, an elaborately designed feature-interaction module is constructed to acquire and fuse generalized abstract spectral and spatial features in different parallel layers. The lightweight dynamic convolution further compresses the memory of the network to realize flexible spatial feature extraction. Experiments on several real HSI datasets confirm that the proposed HyperLiteNet can efficiently decrease the number of parameters and the execution time as well as achieve better classification performance compared to several recent state-of-the-art algorithms.

1. Introduction

With continuous improvement to the accuracy of image acquisition equipment, the spectral dimension of hyperspectral images (HSIs) has expanded greatly, which provides great discrimination ability for the component analysis of materials and land covers, etc. Therefore, the availability of abundant spectral–spatial information has allowed HSIs to be widely used in various applications, e.g., Earth observation [1], military reconnaissance [2], environmental protection [3], and resource management [4].

Numerous handcrafted features and traditional classifiers have been proposed over the past decades. However, HSI data suffers from the “curse of dimensionality” problem, due to the extremely limited hyperspectral training samples [5]. Therefore, many linear and nonlinear dimension reduction algorithms, such as band selection [6], principal component analysis [7], independent component analysis [8], and the maximum noise fraction [9] have been proposed to address this problem. Such algorithms employ feature optimization to remove redundant features and improve classification performance. Traditional classifiers can be mainly divided into two categories, according to their feature extraction characteristics: (1) spectral-based classification algorithms, such as support vector machines (SVM) [10], K-nearest neighbor [11], multinomial logistic regression [12], and random forest [13]; (2) spectral–spatial-based classification algorithms, e.g., SVM with composite kernels (SVMCK) [14], sparse representation-based classifiers [15], spatial–spectral derivative-aided kernel joint sparse representation [16], and adaptive nonlocal spatial–spectral kernels [17]. However, above-mentioned algorithms usually adopt handcrafted features in terms of the requirements of professional domain knowledge in the HSI classification area.

In contrast, deep learning (DL)-based algorithms demonstrate automatic representative and abstract feature extraction due to the rapid iteration and GPU-based parallel implementation. In the remote sensing field, DL-based algorithms also demonstrate extremely promising results. For instance, the stacked autoencoder [18] demonstrates the effectiveness of DL models in abstract high-level semantic feature extraction. The restricted Boltzmann machine and deep belief network [19] and unsupervised greedy learning algorithms realize the DL of objects. With the development of DL, convolutional neural networks (CNN) are widely used in HSI classification tasks. For example, Zhong et al. proposed the spectral–spatial CNN-based algorithm (SSRN) [20] by combining 3D convolutions and residual connections to realize robust HSI classification results. To refine the feature fusion process, Gao et al. [21] performed multi-scale feature extraction and dense feature fusion to expand the features of different levels and scales. Li et al. [22] proposed a hierarchical homogeneity–attention network to consolidate the position supervision of object regions and accelerate computation by reducing the number of similar operations. In terms of the visual invariance of the CNN, input image visual transformation causes drastic fluctuations in network performance. To address the issue of insufficient spectral–spatial feature extraction, Zhang et al. [23] proposed a spectral–spatial attention fusion with a deformable convolution residual network. The capsule network (CapsNet) was proposed [24] to improve network robustness and preserve more efficient position information that is omitted in traditional CNNs. Then, the deep convolutional capsule neural network (DC-CapsNet) [25] introduced CapsNet in 3D convolution to enhance the robustness of the learned spectral–spatial features. The generative adversarial network (GAN) has recently demonstrated satisfying performance for alleviating the problem caused by an insufficient number of labeled samples of CapsNet. However, it is difficult for GAN to model and preserve the relative positions between features accurately. To eliminate the mode collapse and gradient disappearance problem caused by traditional GANs, Wang et al. [26] developed a dual-channel spectral–spatial fusion capsule GAN (DcCapsGAN) by integrating CapsNet with a GAN. The progress of DL-based algorithms also exposed some challenging issues. With the expansion in complexity of DL networks, though the performance improved dramatically, the requirement of training samples and depths increase proportionally with the complexity of the model and make it difficult to deploy on edge devices. Therefore, realizations of an efficient DL structure with lower time costs and fewer training samples are more requisite for improving the HSI classification performance; it is a more crucial and applicable research task for mobile devices and in-orbit aerospace applications.

Recent research has revealed that neural networks with deep structures typically exhibit higher nonlinearity capabilities and stronger function approximations when automatically learning more abstract features [27]. However, deeper network structures also give rise to the vanishing and exploding gradient problems. Recent studies have indicated that a class of widened ResNets is far superior than their deep counterparts [28]. Shallow network structures with high parallelizations tend to provide fast responses and low latency. In addition, Wu et al. proved that a well-designed shallow neural network can outperform many deep neural networks [29], and recent empirical work has further indicated that obtaining the best accuracy requires a balance of depth and width [30]. However, most of these previous studies only focused on networks with linear and sequential structures. Conversely, in [31], it is found that the performance of a shallow network with a parallel structure is similar to that of the deep network. Therefore, compared to deep networks with more sequential processing and higher latency, the parallel shallow networks are more suitable for the hardware applications and deployment [31].

On the other hand, compressing a well-trained complex model or learning a lightweight model with carefully designed network structures are the current mainstream methods to handle the network efficiency problem caused by CNN-based algorithms. Network operating efficiency is primarily affected by the following four components: the number of parameters, the number of computations, memory access, and memory usage. The main design principle of a lightweight network architecture model can be mainly divided into the convolutional level, the convolution operator level, and the computation method:

(1) In the convolutional level: Redesigning the convolutional level primarily involves small-size convolution kernels, a bottleneck layer, and channel concatenation. For example, SqueezeNet [32] replaces $3\times 3$ convolution with $1\times 1$ convolution, reduces the number of input channels in the squeeze layer, and concatenates two sets of convolution output channels to obtain the target number of output channels to reduce network complexity.

(2) In the convolution operator level: Standard convolution has the inherent properties for global spacial and channel feature extraction. ShuffleNet V1 [33] limits the convolution operation to each group and performs a channel shuffle to promote the flow of information between groups. This is equivalent to decomposing the global channel feature extraction process into a local channel feature extraction and channel shuffle process. Furthermore, the depthwise separable (DW) convolution completely separates the space and channel feature extractions. From the implementation perspective, DW convolution can be considered as a special case of group convolution. The MobileNet family has successfully applied DW convolution and recently made a series of improvements [34,35,36]. Subsequently, different from DW convolution, Ghost [37] performs a one-to-many mapping output of a single feature map. In addition to channel decomposition, standard convolution can be further decomposed in spatial dimensions. For example, Zhang et al. [38] developed a compact convolution module to divide the input feature maps into groups and perform cyclic recursive feature extraction in each group. Compared to the channel-specific and spatially-agnostic nature of convolution, Involution [39] involves a spatially-specific and channel-agnostic operation that is distinct in the spatial extent but shared across channels. However, extremely low computational costs are likely to cause significant performance degradation. To address this problem, MicroNet [40] integrates sparse connectivity into convolution and constructs a dynamic activation function, i.e., dynamic shift max.

(3) In the computation level: Current CNNs primarily rely on multiplication operations, which are more computationally complex than addition operations. To address this issue, AdderNets [41] replaces massive number of multiplication operations in CNNs with less-expensive addition operations to reduce computational costs. Note that most of the aforementioned algorithms are based on the parameter and computation as model design indicators, and the actual acceleration effect is inferior to the theoretical analysis value. Shufflenet V2 [42] deeply analyzes the influence of different elements, e.g., memory access cost. Therefore, in addition to theoretical indicators, more attention should be paid to the actual time costs on equal hardware. The superiority of the lightweight structures has also been confirmed in the HSI classification tasks. Zhang et al. [43] proposed 3-D-LWNet, which has a deeper network structure and exhibits better classification performance than conventional 3D-CNN models. In addition, LWCNN [44] employs spatial–spectral Schroedinger eigenmap (SSSE)-based feature extraction and a dual-scale convolution module. The former reduces dimensions, and the latter address the SSSE features from a 1D vector viewpoint to reduce the number of parameters. Based on the conclusions of a neural architecture search [45], LMAFN is proposed to combine Ghost and ECA modules for further achieving efficient feature extraction performance [46]. LiteDepthwiseNet [47] used 3D depthwise convolution to reduce the number of parameters as well as remove the ReLU layer and batch normalization layer in the original 3D depthwise convolution to improve the overfitting phenomenon of the model on small-sized datasets.

Considering that compact lightweight models still achieved comparable or even better performance with fewer parameters, a lightweight model indicates more effective representative knowledge capacity than traditional DL models. However, blindly reducing model size can easily lead to knowledge degeneration, which contradicts the purpose of the lightweight design. To address this issue, CondConv [48] embeds multiple sets of convolution kernels in the convolutional layer and adaptively combines the weights according to the input, which is equivalent to learning specialized convolutional kernels for each example in terms of increasing network capacity while maintaining inference efficiency. To reorganize multiple sets of static convolution kernels according to the input, DYNet [49] performs a low-redundancy and efficient dynamic convolution to avoid redundant calculations and similar feature generation while maintaining a lightweight structure and efficient inference ability. Moreover, the study finds that dynamic networks can address domain conflicts by aggregating residual matrices and a static convolution matrix [50].

In the HSI classification area, the limited availability of labeled samples hinders the classification accuracy and optimization performance of DL. Therefore, recently, few-shot learning [51], domain adaptation [52], transfer learning [53], and meta learning [54] have been gradually proposed to address this issue. For example, Li et al. [55] proposed DCFSL, which utilizes a conditional adversarial domain-adaptation strategy to overcome domain shift and achieve domain distribution alignment. In addition, AMF-FSL [56] transfers the learned classification capabilities from multiple source data to the target data. To realize the promising performance with limited labeled samples, the above DL methods are generally based on complicated network structures and learning strategies. Therefore, an efficient network model that can obtain effective classification results with limited labeled samples is required. Intuitively, a lightweight model can also mitigate the requirements for the quantity of labeled samples to a certain extent.

Based on the above-mentioned knowledge, and aiming at realizing high efficiency and performance at lower computation cost, a HyperLiteNet is proposed and designed in this paper, which is an extremely lightweight non-deep parallel network. In the proposed HyperLiteNet, a parallel shallow narrow network is constructed to reduce network complexity and execution cost, where a parallel dual-branch structure is employed to enhance the nonlinearity of network and the capability of feature extraction. In addition, the spatial features and spectral features are extracted via a parallel dual-branch structure to, respectively, promote the diversity and divergence of the extracted features. As the main components of the dual-branch structure, lightweight pointwise convolution units and lightweight dynamic convolution units are constructed to further reduce the network’s complexity. In addition, the introduction of dynamic modules enables more elastic mapping with the input sample to generate compact and representative features. The experimental results confirm that the proposed HyperLiteNet outperforms other comparison algorithms with an extremely low number of parameters and high execution efficiency.

The remainder of this paper is organized as follows. Section 2 presents the proposed HyperLiteNet in detail. Experimental validation of the proposed HyperLiteNet compared to existing algorithms, including parameter analysis, classification results, classification efficiency, and accuracies, are presented in Section 3. Some experimental results are discussed in Section 4. Finally, conclusions are presented in Section 5.

2. Proposed Methods

The overall framework of the proposed HyperLiteNet is shown in Figure 1. The proposed HyperLiteNet can be roughly divided into five main components: (1) the parallel interconnection module (PIM); (2) the pointwise convolution branch (PCB) to extract spectral features; (3) the dynamic convolution branch (DCB) to extract spatial features; (4) the feature interconnection module (FIM); (5) the classification module (CM). In summary, the specific spectral and spatial features are extracted independently by the PCB and the DCB, respectively. Then, the spectral–spatial features are fused by the FIM and classified by the CM module. The details of different components are illustrated in the following parts.

2.1. Parallel Interconnection Module (PIM)

The PIM is the primary component of the proposed HyperLiteNet, where the parallel structure contains more nonlinear activation layers than a serial structure at the same depth. The paralleled shallow structure realizes an efficient nonlinear activation ability to extract abstract high-level features through utilizing PCB and DCB structures. With the reduced network depth, the vanishing and exploding gradient problems are mitigated in the PIM. This dual-branch structure is adopted to extract abstract spectral and spatial features independently and in a parallel manner through an extremely shallow network structure. As a result, due to the task decomposition of the PCB and DCB, fewer channels are established by the PIM in each branch than traditional spectral–spatial combination networks. Therefore, the network structure can be efficiently compressed in a compact, parallel manner with a lighter structure and better feature extraction. The structure of the PIM is mainly composed of three stages ($stag{e}_1,stag{e}_2,stag{e}_3$), which extract shallow pixel-level features, mid-level features, and deep high-level features, respectively. The features extracted in each stage are fused at the tail of each stage and then input to the next stage to further increase the discriminativeness of the extracted features. Meanwhile, each stage contains two pointwise convolution (PW) units and one depthwise separable dynamic convolution (DC) unit. In the feature extraction process, the PCB and DCB comprise lightweight PW and DC units, respectively. Therefore, the spectral and spacial feature can be extracted in a parallel manner to accelerate execution efficiency. Further, for combining and fusing the spectral and spatial features at different stages, the FIM is designed to connect each stage to the subsequent stage. The entire structure provides a compact and efficient parallel method to extract and fuse the spectral and spatial features with a small number of parameters and low computational cost.

2.2. Pointwise Convolution Branch (PCB) and Dynamic Convolution Branch (DCB)

2.2.1. Pointwise Convolution Branch (PCB)

As shown in Figure 2a, lightweight PW units are employed in the PCB to extract spectral features. Here, only continuous spectral features are processed at a certain point in the feature map. The PCB is concentrated to effectively extract spectral distinguishing features. The pointwise (PW) convolution is calculated as follows:

$$\begin{array}{cc}\hfill o_{c,ij}& ={\mathbf{w}}_c\otimes {\mathbf{x}}_{ij}\hfill \end{array}$$

(1)

where ${\mathbf{x}}_{ij}\in {\mathbb{R}}^{1\times 1\times C_{in}}$ is the spectral vector at the $(i,j)$ of the input features. ${\mathbf{w}}_c\in {\mathbb{R}}^{1\times 1\times C_{in}}$ is the c-th PW convolution to convolve with ${\mathbf{x}}_{ij}$ for obtaining the output $o_{c,ij}\in {\mathbb{R}}^1$, where $o_{c,ij}$ is the value at $(i,j)$ in the c-th channel of the output. It can be seen from the calculation process that the PW convolution is essentially a weighted summation of the spectral vectors of the input features. The procedure intensifies the channels that contributed more discriminant information with a dynamic learning process, which can indicate more satisfying effects and efficiency than traditional band-selection algorithms.

2.2.2. Dynamic Convolution Branch (DCB)

In traditional convolution, all input samples are treated equally to the convolution kernel, and the parameters of the convolution kernel are determined after the training process in a static network manner. Differing from static networks, dynamic networks attempt to adjust the network structure adaptively and dynamically according to the varied inputs. Therefore, for different inputs, a dynamic network can deform the network structure corresponding to the local areas, which efficiently improves the robustness and knowledge capacity of the network with low latency. The introduction of a dynamic network enables a more elastic mapping. Define the network convolutions in Figure 3a as follows:

$$\begin{array}{cc}\hfill \overline{\mathbf{W}}& =\mathbf{W}+\widehat{\mathbf{W}}\hfill \end{array}$$

(2)

where $\mathbf{W}\in {\mathbb{R}}^{k\times k\times C_{in}\times C_{out}}$ and $\widehat{\mathbf{W}}\in {\mathbb{R}}^{k\times k\times C_{in}\times C_{out}}$ are the static convolution kernel and the dynamic convolution kernel, respectively. $C_{in}$ and $C_{out}$ are the number of input channels and the number of output channels. If $\mathbf{W}$ is ignored, $\overline{\mathbf{W}}$ is equivalent to $\widehat{\mathbf{W}}$.

The structure of the standard dynamic convolution is shown in Figure 3b, where the multi-group parallel standard convolutions extract features independently. Then, the extracted features are adjusted dynamically through each attention dynamic unit. The multi-group features are dynamically fused by a dynamic fusion unit. Essentially, the dynamic fusion unit reorganizes multi-group standard dynamic convolutions with different weights. The standard dynamic kernel can be generated by the following formulas:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{w}}}_c& =\sum _i^g{\alpha }_c^i·{\mathbf{w}}_c^i\hfill \end{array}$$

(3)

where $c=0,\cdots ,C_{out}$ and $i=0,\cdots ,g$ are the output channels and number of static convolution kernel groups, respectively. $\alpha$ represents the importance weights of different sets of static convolution kernels based on the input. ${\mathbf{w}}_c^i\in {\mathbb{R}}^{k\times k\times C_{in}}$ indicates the c-th convolution kernel of the i-th static convolution. Based on the linear combination characteristic of convolution, the dynamic convolution output can be mathematically represented as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\widehat{\mathbf{O}}}_c& ={\widehat{\mathbf{w}}}_c\otimes \mathbf{X}=\sum _i^g({\alpha }_c^i·{\mathbf{w}}_c^i)\otimes \mathbf{X}=\sum _i^g({\alpha }_c^i·{\mathbf{w}}_c^i\otimes \mathbf{X})\hfill \\ & =\sum _i^g({\alpha }_c^i·({\mathbf{w}}_c^i\otimes \mathbf{X}))=\sum _i^g({\alpha }_c^i·{\mathbf{o}}_c^i)\hfill \end{array}\end{array}$$

(4)

where the output ${\widehat{\mathbf{O}}}_c$ of the dynamic convolution kernel can be expressed as a linear combination of the output ${\mathbf{O}}_c^i$ of the static convolution kernel.

Further, for realizing the design of the lightweight structure, we adopt the lightweight DW dynamic convolution as the primary component of the DC unit in Figure 3c. Specifically, the parameter definition is different in the realization of the DW dynamic convolution. ${\mathbf{w}}_c^i\in {\mathbb{R}}^{k\times k\times 1}$ indicates the c-th channel of the i-th static DW convolution due to the characteristic of DW convolution. Compared with the standard dynamic convolution, DW dynamic convolution realizes the single-channel-level dynamic fusion to further decompose and refine the dynamic convolution. In addition, we further analyze the DW dynamic convolution from the perspective of parameters and calculations. Assume the standard convolution ${\mathbf{W}}_S\in {\mathbb{R}}^{k\times k\times C_{in}\times C_{out}}$ and the DW dynamic convolution ${\mathbf{W}}_{DW}\in {\mathbb{R}}^{k\times k\times C_{out}\times C_{out}\times g}$. The PW convolution adjusts the number of channels in the DW dynamic convolution. Ignoring the input adaptive correlation calculation, the reduced computation and parameters effect can be written as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{k\times k\times C_{out}\times s\times s\times g+C_{in}\times C_{out}\times s\times s}{k\times k\times C_{in}\times C_{out}\times s\times s}& \\ \hfill =\frac{g}{C_{in}}+\frac{1}{k^2}\approx \frac{1}{k^2}(C_{in}\gg g)\end{array}\end{array}$$

(5)

where s is the space size of the feature map.

Compared to the traditional standard convolution, the diversity and spatial differences of the spatial characteristics can be extracted flexibly and enhanced by the dynamic module. In addition, the DCB realizes the potential for reducing the number of parameters and computations, which improves the efficiency of the network operations.

2.3. Feature Interconnection Module (FIM) and Classification Module (CM)

The function of FIM is to fuse the spectral and spatial features of the two branches and feed the fused features to the next stage of the network. Here, suppose the input spectral and spatial features in $stag{e}_3$ are defined as ${\mathbf{X}}^{P_6}$ and ${\mathbf{X}}^{D_3}$, respectively. The calculation process of FIM is determined by the following formula:

$$\begin{array}{cc}\hfill {\mathbf{X}}_{FI{M}_3}& ={\mathbf{X}}^{P_6}+{\mathbf{X}}^{D_3}\hfill \end{array}$$

(6)

For the CM, the PW unit progressively reduces the number of channels and prevents the sharp reduction in the number of channels for avoiding the feature information loss. In this module, we utilize cross entropy as the loss function for HSI classification. The cross entropy loss can be represented as:

$$\begin{array}{c}Loss=-\frac{1}{N}\sum _{n=1}^N\sum _{c=1}^C{y}_c^{n}log\left({\widehat{y}}_c^n\right)\end{array}$$

(7)

where C indicates the number of classes, $y_c^n$ and ${\widehat{y}}_c^n$ are the truth and predicted labels, respectively, and N is the number of samples in a minibatch.

3. Experiments and Analysis

In this section, we mainly evaluate our proposed frameworks on three real HSI datasets. The experimental settings and results, including parameters analysis and evaluation metrics are illustrated. Finally, the comparison result with the existing algorithms are analyzed and discussed in this section.

3.1. Datasets Descriptions

The Indian Pines (IN) dataset was captured in 1992 over the Indian Pines agriculture experimental area by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor. The IN dataset has 220 bands with a wavelength range from 0.4 to 2.5 $\mathsf{\mu }$m, and with $145\times 145$ pixels. Note that 20 bands of noisy or water-absorption regions are removed in our experiments. The IN dataset contains 16 different ground-truth classes. The three-band false-color composite of the IN images and the corresponding ground-truth data are shown in Figure 4.

The University of Pavia (UP) dataset was collected in 2001 over the Pavia university campus through the Reflective Optics System Imaging Spectrometer (ROSIS). The UP dataset has 115 bands with a wavelength range from 0.43 to 0.86 $\mathsf{\mu }$m, and with $610\times 340$ pixels. Note that 12 bands of noisy or water-absorption regions are removed in our experiments. The UP dataset contains nine different ground-truth classes. The three-band false-color composite of the IN images and the corresponding ground-truth data are shown in Figure 5.

The Salinas (SA) dataset was gathered over the Salinas Valley in California by the AVIRIS sensor. The SA dataset has 220 bands with a wavelength range from 0.36 to 0.5 $\mathsf{\mu }$m, and with $512\times 217$ pixels. Note that 20 bands of noisy or water-absorption regions are removed in our experiments. The SA dataset contains 16 different ground-truth classes. The three-band false-color composite of IN images and the corresponding ground-truth data are shown in Figure 6.

3.2. Experimental Configuration and Parameter Analysis

3.2.1. Experimental Configuration

The experimental configuration is as follows: an Intel Xeon W-2123 CPU at 3.60 GHz with 64-GB RAM and an NVIDIA GeForce GTX 2080 Ti graphical processing unit (GPU) with 11-GB RAM were used. The software environment includes a 64-bit Windows 10 system and the Pytorch 1.6.0 DL frameworks. Here, the learning rate is set to 0.0001, and the Adam optimizer is used with a batch size of 72. In addition, five labeled samples are randomly selected from each class as a training set for each HSI. To avoid the inclination in random sampling, the evaluated mean value is reported for parameter analysis and classification result comparison with ten random repetitions.

3.2.2. Experimental Parameter Analysis

To evaluate the effectiveness of the proposed HyperLiteNet, we mainly evaluate the effect of the number of kernels (extracted channels), the quantity of parameters, and the training epoch. We first analyze the effect for the number of kernels in each conventional layer. The number of kernels in each layer is selected from $\left\{16,32,48,64,72,96,128\right\}$ for each dataset. The effect analysis results are illustrated in Figure 7a,b; we can notice that the number of 16 kernels, corresponding to the least number of parameters, presents the worst performance for all experimental datasets. The overall accuracy (OA) gradually increased with the increasing number of channels. Compared to 128 channels, we observe from Figure 7a that 16 channels obtain lower OA with the same number of training epochs (4000). It is notable that the increasing number of channels can improve the classification accuracy significantly. However, when the number of convolution kernels exceeds a certain threshold, the performance gradually increases to a stable value. The excessive number of convolution kernels brings additional an parameter burden without beneficial effects on performance. The aforementioned data analysis indicates that the limited model performance and difficult model optimization phenomenon exist in low-parameter models. Therefore, the optimal parameters acquired from the above-mentioned analysis are utilized in the following comparison experiments.

Because of the correlation between time costs and model convergence, we evaluate OA under different numbers of training epochs to achieve a balance between time costs and performance, where training epochs are a varied set from 50 to 4000 with step 200. The analysis results are presented in Figure 7c. We can notice that the proposed model reaches a stable and exceptional OA after 2000 epochs. Moreover, MiniNet achieves acceptable results in 1000 epochs. Considering the necessity of adequate training, 4000 is chosen as the number of training epochs in the following experiment.

3.3. Ablation Study for DW Dynamic Convolution

To verify the proposed effectiveness of the DW dynamic convolution (DC), we compare the proposed DC with standard convolution (SC) ($3\times 3$ convolution kernel is applied), and multi-scale convolution (MSC) ($3\times 3$ and $5\times 5$ convolution kernels are applied). The comparison results are illustrated in

Table 1. Model performance analysis (OA) and parameters (P) with different convulution types. (1) DC: depthwise separable dynamic convolution; (2) SC: standard convolution; (3) MSC: multi-scale convolution.

	Metric	OA	P	OA	P	OA	P
Type
DC		75.75 ± 3.25	59,658	85.89 ± 4.60	51,939	90.70 ± 1.76	59,914
SC		75.19 ± 3.16	153,840	83.96 ± 4.88	146,121	90.35 ± 2.07	154,096
MSC		75.34 ± 3.62	461,616	83.11 ± 3.89	453,897	88.72 ± 4.21	461,872
Dataset		IN		UP		SA

. It is obvious that the DC improves the classification performance while effectively compressing the model size. In addition to quantitative analysis, the t-SNE feature visualization technology is further exhibited to qualitatively analyze the differences between the three convolutions in Figure 8. The blue dotted line indicates the comparison between DC and SC, and the red dotted line represents the comparison between DC and MSC. From Figure 8a–c, we can observe that the dotted line in Figure 8a has a higher concentration and purity ratio. This phenomenon also appeared in Figure 8g–i. The red dotted area in Figure 8d,g exhibits better concentration than Figure 8f,i. Moreover, it is roughly observed that the distinction between the red and green cluster in the blue dotted rectangle region for Figure 8d is inferior to Figure 8e. Meanwhile, in real classification investigations, there are many misclassified samples in these two classes for Figure 8d,f, which further indicates that these two classes are more indistinguishable in real classification tasks. Overall, from the t-SNE feature visualization, as illustrated in Figure 8, the DC achieves a more distinguishable and recognizable feature extraction capability.

3.4. Classification Accuracy and Performance

In this part, we compare the proposed HyperLiteNet with six comparable methods: two traditional methods, SVM and SVMCK, and four deep learning-based methods, SSRN, LMAFN, DcCapsGAN, and DCFSL. The SVM only extracts spectral features, and the SVMCK further introduces spatial features for classification. The SSRN is a classic method using 3D convolution for HSI classification. The DcCapsGAN combines the capsule network and GAN to enhance the model robustness. Based on the lightweight characteristics of the proposed HyperLiteNet, we also compared it with the LMAFN. In order to test the effectiveness of proposed HyperLiteNet under the condition of limited labeled samples, we further compared the proposed HyperLiteNet with the few-shot learning method DCFSL. The quantitative experimental results of different methods are shown in

Table 2. Classification results (%) on the IN dataset (five labeled samples per class). The best performance values are marked in bold.

Class	SVM	SVMCK	SSRN	LMAFN	DcCapsGAN	DCFSL	HyperLiteNet
1	$83.17 \pm 10.32$	$94.63 \pm 3.75$	$45.60 \pm 21.99$	$99.51 \pm 0.98$	$\mathbf{100} \pm \mathbf{0.00}$	$93.17 \pm 9.05$	$99.76 \pm 0.73$
2	$28.09 \pm 9.09$	$51.33 \pm 11.37$	$\mathbf{64.03} \pm \mathbf{11.93}$	$55.85 \pm 11.61$	$50.50 \pm 0.29$	$43.25 \pm 9.14$	$61.33 \pm 15.75$
3	$37.53 \pm 10.10$	$51.25 \pm 6.99$	$63.89 \pm 14.14$	$67.82 \pm 9.63$	$44.36 \pm 0.10$	$53.82 \pm 4.95$	$\mathbf{69.95} \pm \mathbf{7.97}$
4	$59.18 \pm 11.14$	$88.71 \pm 11.50$	$44.11 \pm 15.82$	$90.60 \pm 8.90$	$46.70 \pm 0.54$	$78.19 \pm 18.67$	$\mathbf{93.41} \pm \mathbf{8.93}$
5	$67.91 \pm 9.21$	$64.16 \pm 12.82$	$78.58 \pm 12.99$	$80.94 \pm 7.47$	$65.06 \pm 0.17$	$72.53 \pm 6.42$	$\mathbf{81.72} \pm \mathbf{8.07}$
6	$72.43 \pm 13.84$	$63.03 \pm 15.13$	$90.83 \pm 7.92$	$94.52 \pm 3.56$	$83.77 \pm 0.07$	$85.31 \pm 9.64$	$\mathbf{96.40} \pm \mathbf{2.53}$
7	$90.00 \pm 5.52$	$98.70 \pm 1.99$	$31.90 \pm 14.60$	$\mathbf{100} \pm \mathbf{0.00}$	$\mathbf{100} \pm \mathbf{0.00}$	$98.70 \pm 3.91$	$\mathbf{100} \pm \mathbf{0.00}$
8	$67.51 \pm 15.56$	$96.62 \pm 3.86$	$\mathbf{97.58} \pm \mathbf{3.86}$	$93.78 \pm 9.86$	$80.20 \pm 0.20$	$86.89 \pm 11.06$	$95.88 \pm 7.87$
9	$91.33 \pm 13.35$	$96.00 \pm 8.00$	$25.50 \pm 15.94$	$\mathbf{100} \pm \mathbf{0.00}$	$\mathbf{100} \pm \mathbf{0.00}$	$99.33 \pm 2.00$	$\mathbf{100} \pm \mathbf{0.00}$
10	$53.58 \pm 14.38$	$57.61 \pm 6.27$	$57.57 \pm 9.91$	$\mathbf{69.51} \pm \mathbf{8.70}$	$62.36 \pm 0.22$	$62.53 \pm 7.68$	$\mathbf{71.98} \pm \mathbf{7.57}$
11	$36.12 \pm 15.10$	$53.05 \pm 14.20$	$69.27 \pm 7.80$	$62.73 \pm 11.90$	$65.02 \pm 0.13$	$58.45 \pm 8.90$	$64.12 \pm 12.27$
12	$34.32 \pm 9.08$	$\mathbf{69.46} \pm \mathbf{12.03}$	$51.80 \pm 16.10$	$69.37 \pm 13.83$	$48.58 \pm 0.32$	$45.90 \pm 7.84$	$69.12 \pm 15.96$
13	$88.00 \pm 6.34$	$95.20 \pm 7.96$	$88.56 \pm 15.34$	$\mathbf{99.05} \pm \mathbf{1.60}$	$99.01 \pm 0.00$	$98.40 \pm 3.49$	$98.50 \pm 1.75$
14	$61.27 \pm 12.99$	$71.52 \pm 12.69$	$\mathbf{95.24} \pm \mathbf{5.47}$	$88.13 \pm 10.50$	$94.89 \pm 0.19$	$82.02 \pm 9.45$	$89.61 \pm 9.64$
15	$28.69 \pm 7.84$	$73.78 \pm 10.15$	$65.58 \pm 14.58$	$76.93 \pm 17.21$	$35.61 \pm 0.75$	$76.09 \pm 11.54$	$\mathbf{86.25} \pm \mathbf{12.16}$
16	$89.20 \pm 6.37$	$98.52 \pm 2.16$	$70.83 \pm 18.10$	$\mathbf{99.32} \pm \mathbf{1.04}$	$98.86 \pm 0.24$	$97.95 \pm 4.09$	$98.75 \pm 2.00$
OA	$47.45 \pm 5.32$	$62.74 \pm 5.60$	$67.10 \pm 3.24$	$73.41 \pm 3.95$	$65.57 \pm 0.03$	$64.96 \pm 2.03$	$\mathbf{75.75} \pm \mathbf{3.25}$
AA	$61.77 \pm 3.20$	$77.01 \pm 4.00$	$65.05 \pm 2.96$	$84.26 \pm 1.91$	$73.43 \pm 0.04$	$77.03 \pm 0.86$	$\mathbf{86.05} \pm \mathbf{1.50}$
KP	$41.67 \pm 5.38$	$58.46 \pm 6.05$	$63.00 \pm 3.49$	$70.08 \pm 4.37$	$60.94 \pm 0.03$	$60.65 \pm 2.08$	$\mathbf{72.71} \pm \mathbf{3.50}$
PA	−	−	346,784	161,451	33,521,328	4,270,121	59,658
Flops (M)	−	−	$110.245$	$7.369$	$433.220$	−	$\mathbf{2.920}$
Train(s)	−	−	$436.20$	$281.83$	$2394.88$	$2629.22$	$\mathbf{80.84}$
Test(s)	−	−	$31.38$	$1.43$	$11.51$	$2.21$	$\mathbf{0.24}$

,

Table 3. Classification results (%) on the UP dataset (five labeled samples per class). The best performance values are marked in bold.

Class	SVM	SVMCK	SSRN	LMAFN	DcCapsGAN	DCFSL	HyperLiteNet
1	$49.41 \pm 13.47$	$55.21 \pm 11.76$	$\mathbf{96.30} \pm \mathbf{2.20}$	$75.68 \pm 11.73$	$36.40 \pm 4.74$	$74.94 \pm 17.10$	$86.64 \pm 6.63$
2	$44.45 \pm 14.50$	$56.70 \pm 11.93$	$\mathbf{96.42} \pm \mathbf{3.57}$	$77.49 \pm 14.00$	$74.92 \pm 0.52$	$85.96 \pm 8.44$	$80.62 \pm 12.13$
3	$62.34 \pm 18.77$	$71.77 \pm 8.74$	$61.35 \pm 8.69$	$68.64 \pm 15.32$	$77.94 \pm 1.21$	$62.43 \pm 12.45$	$\mathbf{87.40} \pm \mathbf{7.65}$
4	$86.33 \pm 11.81$	$93.46 \pm 5.50$	$76.77 \pm 18.53$	$96.21 \pm 5.48$	$\mathbf{98.29} \pm \mathbf{0.11}$	$91.62 \pm 5.01$	$92.94 \pm 6.24$
5	$96.19 \pm 6.99$	$99.57 \pm 0.22$	$99.19 \pm 1.20$	$99.45 \pm 0.87$	$\mathbf{100} \pm \mathbf{0.00}$	$99.25 \pm 0.95$	$99.57 \pm 0.79$
6	$52.54 \pm 13.28$	$71.86 \pm 15.90$	$62.92 \pm 12.21$	$79.36 \pm 16.01$	$54.15 \pm 0.38$	$71.07 \pm 10.49$	$\mathbf{88.07} \pm \mathbf{9.58}$
7	$73.50 \pm 21.41$	$94.42 \pm 2.07$	$77.03 \pm 15.96$	$92.33 \pm 7.52$	$98.04 \pm 0.43$	$77.69 \pm 8.81$	$\mathbf{98.29} \pm \mathbf{2.36}$
8	$49.04 \pm 15.44$	$72.46 \pm 12.68$	$74.35 \pm 12.96$	$80.18 \pm 20.70$	$69.86 \pm 2.33$	$68.16 \pm 11.62$	$\mathbf{89.12} \pm \mathbf{11.37}$
9	$93.68 \pm 7.54$	$86.69 \pm 7.78$	$99.25 \pm 0.69$	$97.07 \pm 5.68$	$\mathbf{100} \pm \mathbf{0.00}$	$99.03 \pm 1.41$	$97.29 \pm 4.14$
OA	$54.05 \pm 6.35$	$66.15 \pm 5.95$	$82.03 \pm 3.31$	$80.15 \pm 6.04$	$69.94 \pm 0.90$	$80.66 \pm 4.02$	$\mathbf{85.89} \pm \mathbf{4.60}$
AA	$67.50 \pm 3.37$	$78.97 \pm 3.36$	$82.62 \pm 3.07$	$85.16 \pm 3.29$	$78.84 \pm 0.65$	$81.13 \pm 2.70$	$\mathbf{91.11} \pm \mathbf{2.25}$
KP	$44.90 \pm 5.99$	$58.52 \pm 6.41$	$77.24 \pm 3.76$	$74.87 \pm 6.97$	$62.29 \pm 1.07$	$74.92 \pm 4.81$	$\mathbf{82.03} \pm \mathbf{5.38}$
PA	−	−	199,153	153,060	21,468,326	4,259,294	51,939
Flops (M)	−	−	$54.178$	$7.001$	$342.850$	−	$\mathbf{2.553}$
Train(s)	−	−	$139.12$	$216.60$	$766.54$	$1593.04$	$\mathbf{79.47}$
Test(s)	−	−	$73.67$	$4.78$	$22.17$	$7.11$	$\mathbf{0.70}$

and

Table 4. Classification results (%) on the SA dataset (five labeled samples per class). The best performance values are marked in bold.

Class	SVM	SVMCK	SSRN	LMAFN	DcCapsGAN	DCFSL	HyperLiteNet
1	$97.45 \pm 1.70$	$98.73 \pm 0.87$	$98.49 \pm 4.51$	$97.74 \pm 2.37$	$97.72 \pm 1.11$	$\mathbf{99.25} \pm \mathbf{1.05}$	$98.65 \pm 2.48$
2	$95.89 \pm 2.99$	$93.76 \pm 7.00$	$99.44 \pm 0.88$	$90.74 \pm 12.57$	$\mathbf{99.95} \pm \mathbf{0.00}$	$99.62 \pm 0.30$	$95.92 \pm 10.14$
3	$69.86 \pm 17.07$	$71.02 \pm 17.71$	$96.29 \pm 1.53$	$85.40 \pm 13.74$	$\mathbf{100} \pm \mathbf{0.00}$	$89.48 \pm 13.06$	$91.16 \pm 10.05$
4	$98.42 \pm 2.40$	$99.63 \pm 0.18$	$95.86 \pm 3.51$	$99.83 \pm 0.19$	$\mathbf{99.86} \pm \mathbf{0.00}$	$99.29 \pm 0.48$	$99.73 \pm 0.40$
5	$94.36 \pm 5.00$	$96.61 \pm 2.09$	$\mathbf{97.08} \pm \mathbf{2.28}$	$94.54 \pm 3.57$	$95.57 \pm 0.11$	$91.55 \pm 2.85$	$93.03 \pm 11.65$
6	$99.21 \pm 0.29$	$99.50 \pm 0.12$	$99.78 \pm 0.34$	$99.63 \pm 0.67$	$\mathbf{100} \pm \mathbf{0.00}$	$99.20 \pm 0.79$	$99.57 \pm 0.59$
7	$98.36 \pm 1.50$	$97.59 \pm 2.03$	$99.29 \pm 1.04$	$99.86 \pm 0.10$	$99.83 \pm 0.00$	$99.13 \pm 1.14$	$\mathbf{99.96} \pm \mathbf{0.07}$
8	$58.39 \pm 13.63$	$61.67 \pm 13.65$	$\mathbf{82.17} \pm \mathbf{5.25}$	$63.25 \pm 14.20$	$61.66 \pm 0.52$	$74.96 \pm 11.32$	$78.79 \pm 9.25$
9	$96.28 \pm 0.46$	$95.61 \pm 3.00$	$99.10 \pm 0.68$	$99.57 \pm 0.70$	$\mathbf{99.87} \pm \mathbf{0.00}$	$99.44 \pm 0.73$	$99.67 \pm 0.68$
10	$76.97 \pm 6.46$	$83.15 \pm 6.71$	$\mathbf{90.97} \pm \mathbf{3.72}$	$85.47 \pm 13.79$	$79.71 \pm 0.16$	$84.36 \pm 5.98$	$90.82 \pm 4.19$
11	$93.88 \pm 3.43$	$91.83 \pm 8.65$	$93.08 \pm 2.84$	$99.10 \pm 0.95$	$98.28 \pm 0.09$	$97.38 \pm 2.21$	$\mathbf{99.20} \pm \mathbf{0.77}$
12	$82.39 \pm 19.68$	$93.11 \pm 9.79$	$97.65 \pm 3.94$	$99.53 \pm 0.75$	$\mathbf{99.90} \pm \mathbf{0.04}$	$98.55 \pm 1.82$	$98.69 \pm 1.29$
13	$97.65 \pm 0.75$	$97.86 \pm 0.79$	$92.87 \pm 7.42$	$98.97 \pm 2.27$	$\mathbf{99.74} \pm \mathbf{0.05}$	$99.20 \pm 0.47$	$99.58 \pm 0.38$
14	$89.92 \pm 5.11$	$91.61 \pm 3.66$	$92.73 \pm 10.92$	$98.07 \pm 1.23$	$\mathbf{99.41} \pm \mathbf{0.04}$	$98.48 \pm 1.78$	$97.29 \pm 5.03$
15	$59.13 \pm 16.60$	$66.76 \pm 16.26$	$64.45 \pm 7.39$	$69.89 \pm 17.11$	$38.65 \pm 1.63$	$74.50 \pm 6.53$	$\mathbf{78.39} \pm \mathbf{14.47}$
16	$85.03 \pm 6.05$	$83.17 \pm 5.96$	$\mathbf{99.55} \pm \mathbf{0.63}$	$93.88 \pm 6.07$	$97.71 \pm 0.07$	$87.56 \pm 8.17$	$93.53 \pm 6.70$
OA	$80.55 \pm 2.29$	$82.95 \pm 2.08$	$88.95 \pm 2.03$	$85.51 \pm 2.47$	$82.07 \pm 0.19$	$88.79 \pm 2.48$	$\mathbf{90.70} \pm \mathbf{1.76}$
AA	$87.04 \pm 2.43$	$88.85 \pm 1.89$	$93.67 \pm 0.83$	$92.22 \pm 2.03$	$91.74 \pm 0.14$	$93.25 \pm 1.66$	$\mathbf{94.62} \pm \mathbf{1.13}$
KP	$78.43 \pm 2.53$	$81.09 \pm 2.30$	$87.74 \pm 2.23$	$83.93 \pm 2.71$	$80.17 \pm 0.20$	$87.55 \pm 2.73$	$\mathbf{89.67} \pm \mathbf{1.96}$
PA	−	−	352,928	161,707	34,627,364	4,270,521	59,914
Flops (M)	−	−	$112.178$	$7.382$	$1274.300$	−	$\mathbf{2.933}$
Train(s)	−	−	$398.63$	$206.90$	$2585.84$	$2574.94$	$\mathbf{79.98}$
Test(s)	−	−	$178.85$	$5.50$	$13.59$	$10.29$	$\mathbf{1.20}$

. The metrics include overall accuracy (OA(%)), average accuracy (AA(%)), kappa (KP(%)), the number of parameters (PA), floating point operations (FLOPs (M)), training time (Train(s)), and test time (Test(s)).

From Table 2, Table 3 and Table 4, we can observe that, in most cases, the accuracy of SVM and SVMCK are inferior to DL-based methods. In terms of the spectral–spatial information involved in SVMCK, which presents higher classification accuracy than SVM, data-driven characteristics and nonlinearity fitting of CNNs can approximate any function, which facilitates the extraction of discriminant features. Therefore, most of the DL-based models in this paper perform better than traditional SVM and SVMCK methods. The implementation of 3D convolution is consistent with the continuity of the HSI spectral curve. Therefore, the SSRN obtains relatively stable classification performance on most datasets. Our previous research work presents LMAFN based on a lightweight design, which achieves the best operating efficiency among all comparable DL-based methods. The ingenious combination of the GAN and capsule network promotes the robustness of the DcCapsGAN. However, the huge number of parameters in the capsule network also becomes a significant burden for DcCapsGAN. The DCFSL obtains acceptable numerical indexes with low efficiency based on conditional adversarial domain daptation strategy. Compared with LMAFN, we can observe from Table 2, Table 3 and Table 4 that the proposed HyperLiteNet further compresses the quantity of parameters and greatly improves network efficiency, which is beneficial to the deployment of the HyperLiteNet on edge devices. Meanwhile, the feature extraction capabilities of the HyperLiteNet support satisfactory classification results with limited labeled samples. Generally, the proposed HyperLiteNet achieves the optimal classification accuracy among all comparison methods with the lowest number of parameters and execution costs.

The qualitative comparisons of the classification maps for different algorithms are illustrated in Figure 9, Figure 10 and Figure 11. In our expectation, due to the lack of spatial information, SVM indicates the most random misclassified noise labels on the three datasets. In contrast, other methods that exploit spectral–spatial combination information present smoother classification maps than SVM. In addition, the SSRN and DcCapsGAN have more distinguishable classification boundaries based on more elaborately designed structures involved in the structure, whereas the number of parameters also increased dramatically. We can further observe that the HyperLiteNet produces a smoother classification area compared with other comparison algorithms in the red dotted-line area in Figure 11i. These results further demonstrate that the proposed HyperLiteNet can obtain more satisfactory results with fewer parameters and more efficient execution times, even under a limited training sample size.

As shown in Figure 12, we further evaluate the classification performance with varied training sample sizes for different comparison algorithms. From Figure 12, we can notice that the classification accuracy gradually improves as the number of training samples increases. It is easy to find that the proposed HyperLiteNet exhibits acceptable satisfactory classification results with different sample sizes.

4. Discussion

4.1. Assessment of the Model Design

According to the experimental results of the different numbers of training epochs (Figure 7c), the HyperLiteNet has the characteristic of fast convergence for different datasets during the training process. In addition, Figure 7a,b confirm that the model complexity and model convergence ability are negatively correlated. The smaller the model, the more difficult it is to find the appropriate parameters. Therefore, the key is to achieve a suitable balance between the efficiency and the model size. Many practical applications only focus on inference costs. Therefore, more training time in exchange for less inference time is feasible for small algorithm models, which can facilitate online leaning in open-scenario circumstances. On the other hand, as can be seen from Table 1, the dynamic convolution on the three datasets improves model performance (from 0.35% to 2.78%) while reducing the number of parameters over 60% and 80% when compared to the SC and MSC. Meanwhile, the dynamic characteristic support the stability of the model on different datasets. This conclusion is further demonstrated through the t-SNE visualization in Figure 8.

4.2. Assessment of the Model Performances

As illustrated in Table 2, Table 3 and Table 4, the HyperLiteNet achieves the highest classification OA—$75.75\%$, $85.89\%$, and $90.70\%$—with gains of $2.34\%$, $3.86\%$, and $1.75\%$ over the optimal results in the comparison algorithms on the three experimental datasets under extremely small training sizes. Through an elaborate, lightweight design, the HyperLiteNet reduces the number of parameters by 66%–98% and Flops by 60%–99%, compared to other DL-based comparison algorithms. In terms of training time and testing time, even compared with the fastest LMAFN, the HyperLiteNet still achieves 2.6-times and 4.6-times efficiency improvements. Experiments on three representative datasets indicate that the proposed HyperLiteNet outperforms these state-of-the-art DL-based comparison algorithms with extremely high efficiency and a low number of parameters. However, Figure 9, Figure 10 and Figure 11 show that the classification maps generated by the HyperLiteNet presents oversmoothed results. This phenomenon illustrates that HyperLiteNet achieves efficient classification performance, whereas the lightweight structure of spectral information is apt to result in oversmooth boundaries of classes.

5. Conclusions

At present, small, portable smart devices require low energy consumption and fast response times. This limits the efficiency and size of the DL models. Therefore, it is not advisable to blindly pursue the high complexity of the model in exchange for the improvement of the performance. In this article, an extremely lightweight, non-deep MiniNet model is proposed for HSI classification. The proposed model employs dynamic convolution and a feature interaction module to realize the flexible extraction and interaction of abstract features in different stages, which can further improve the discriminant information of different inputs. Meanwhile, the dual-branch PW and dynamic convolutional structure plays an important role in extracting spectral and spatial features in a hardware-friendly manner through the shallow parallel lightweight structure design, with extremely low numbers of parameters and execution costs.