Hyperspectral Image Classification Using SIFANet: A Dual-Branch Structure Combining CNN and Transformer

Highlights

What are the main findings?

• A novel dual-branch network, SIFANet, was developed, which synergistically integrates CNN and Transformers for enhanced HSI classification.
• The architecture successfully captures both local spectral–spatial features via the CNN branch and long-range global dependencies through the Transformer branch, overcoming the limitations of single-backbone models.

What are the implications of the main findings?

• SIFANet provides a robust solution for HSI classification in complex scenarios, significantly improving accuracy by effectively balancing local texture details and global contextual information.
• The dual-branch fusion strategy offers a scalable framework for hybrid deep learning models, potentially advancing the processing of high-dimensional remote sensing data with limited labeled samples.

Abstract

The hyperspectral image (HSI) is rich in spectral information and has important applications in the field of ground objects classification. However, HSI data have high dimensions and variable spatial–spectral features, which make it difficult for some models to adequately extract the effective features. Recent studies have shown that fusing spatial and spectral features can significantly improve accuracy by exploiting multi-dimensional correlations. Based on this, this article proposes a spectral integration and focused attention network (SIFANet) with a two-branch structure. SIFANet captures the local spatial features and global spectral dependencies through the parallel-designed spatial feature extractor (SFE) and spectral sequence Transformer (SST), respectively. A cross-module attention fusion (CMAF) mechanism dynamically integrates features from both branches before final classification. Experiments on the Salinas dataset and Xiong’an hyperspectral dataset show that the overall accuracy on these two datasets is 99.89% and 99.79%, which is higher than the other models compared. The proposed method also had the lowest standard deviation of category accuracy and optimal computational efficiency metrics, demonstrating robust spatial–spectral feature integration for improved classification.

1. Introduction

The hyperspectral image (HSI) is a three-dimensional data cube acquired by remote sensing platforms (such as aircrafts or satellites) using push-broom or staring imaging techniques [1,2]. It contains two-dimensional spatial images corresponding to multiple spectral bands. Unlike traditional multispectral images, HSIs collect continuous spectral information from hundreds of narrow bands. These bands span the visible, near-infrared, and shortwave infrared regions (400–2500 nm) of the electromagnetic spectrum. As a result, each pixel in the HSI contains reflectance value across hundreds of spectral channels, forming a continuous spectral curve [3]. This enables HSIs to provide rich spatial–spectral information [4]. In recent years, the rapid development of computer technology has greatly advanced HSI classification tasks. HSIs have been widely applied in agricultural production [5,6], city planning [7,8], environmental science [9,10], and other fields.

HSI classification faces challenges from three aspects: data, model training, and application. At the data level, the typical 200–300 spectral bands of HSIs lead to high dimensionality, raising computational complexity [11]. Moreover, the limited availability of ground truth training samples results in data sparsity in the high-dimensional feature space, violating statistical learning assumptions. This leads to the Hughes phenomenon. Studies have shown that when the ratio of feature dimensionality to sample size exceeds a critical threshold, the generalization performance of classifiers such as Support Vector Machine (SVM) and K-Nearest Neighbor (KNN) will sharply decline due to overfitting. In addition, strong inter-band correlations will introduce substantial redundant information to models, which hinders effective learning and training [12]. At the model training level, large training sample sizes demand substantial computational resources, significantly increasing training costs [13]. In practical applications, HSI preprocessing involves complex procedures such as radiometric and atmospheric correction, geometric registration, and manual sample labeling [14]. Additionally, to further improve the quality of hyperspectral data, research on enhanced deep image prior has been applied to unsupervised hyperspectral image super-resolution [15], which effectively enhances the spatial resolution and reduces noise without the need for large-scale training datasets. Furthermore, cross-scene model transfer issues caused by various imaging angles and sensor inconsistencies present significant challenges [16]. These factors will greatly affect the HSI classification performance in practical applications. In response, researchers have proposed and refined various classification algorithms to address these challenges, including traditional machine learning methods, deep learning approaches, and multi-model fusion methods.

Traditional machine learning methods depend on manual feature engineering. The SVM uses kernel functions to map features into high-dimensional space and performs well especially in small-sample scenarios. Random forest (RF), using bagging and random subspaces, effectively handles high-dimensional nonlinear data. The K-Nearest Neighbor (K-NN) algorithm classifies samples by distance-based majority voting. Despite the K-NN being simple, it faces limitations in high-dimensional spaces due to the “curse of dimensionality”, which makes it difficult to achieve satisfactory classification results. Additionally, K-NN is sensitive to parameters and has lower computational efficiency. Champa et al. [17] proposed a hybrid technique which combined tree-based classifiers with feature dimensionality reduction. The results show that decision trees (DTs), extra trees (ETs), and RF perform well in HSI classification.

Compared to traditional methods, deep learning approaches offer better adaptability for HSI classification tasks. Convolutional neural networks (CNNs) automatically extract spatial–spectral features, mimicking biological visual mechanisms [18]. Residual neural networks (ResNet) incorporate residual structures for cross-layer information transfer. This structure effectively mitigates the vanishing gradient problem [19]. Li et al. [20] introduced the deep belief network (DBN) for feature extraction and image classification. Zhao et al. [21] proposed a feature-level fusion classification framework combining CNN with texture features, utilizing HSI and LiDAR data for high classification accuracy. As for sequential spectral data, the recurrent neural network (RNN) captures spectral sequence correlations via recursive feedback mechanisms [22]. Long short-term memory (LSTM) networks use gating units to alleviate long-term dependency issues [23]. Recently, the Transformer has gained popularity in natural language processing (NLP), which improves semantic extraction through global dependency modeling [24]. Inspired by this, researchers have integrated attention mechanisms into HSI classification to enhance accuracy by strengthening feature interactions. Hong et al. [25] proposed the SpectralFormer network, which uses self-attention to model long-range spectral dependencies. However, its unimodal (spectral-only) structure neglects critical spatial information of HSIs, limiting effectiveness. To address this issue, Wang et al. [26] introduced a 3D attention mechanism to capture joint spectral–spatial features. Ding et al. [27] developed a global–local Transformer network (GLT-Net) for joint classification using multi-scale feature fusion. However, inherent architectural constraints limit the performance of single-structure models. Specifically, CNNs rely on local receptive fields, which make them effective at capturing local textures but less capable of modeling long-range spatial dependencies or the continuous sequential correlation of spectral bands. Conversely, while Transformers excel at global dependency modeling via self-attention, they lack the spatial inductive bias (such as translation invariance and locality) inherent in CNNs. This often leads to difficulties in capturing fine-grained local spatial structures and results in quadratic computational complexity when dealing with high-dimensional spectral sequences. Consequently, deep learning models still face two key limitations: they require large amounts of annotated data, which is costly and time-consuming [28], and they involve high computational complexity, hindering practical deployment [29]. To alleviate this, transudative few-shot learning with enhanced spectral–spatial embedding [30] has been proposed to achieve robust classification with minimal labeled samples.

In recent years, multi-model fusion methods have shown great promise in pattern recognition and image classification by leveraging the complementary strengths of different structures. Researchers optimize feature extraction and processing by combining various models. For example, Wei et al. [31] proposed a strategy that uses CNNs for feature extraction, followed by SVM classification. This approach not only improves accuracy but also reduces overfitting and alleviates reliance on large sample datasets. Combining multi-scale feature fusion with gradient boosting decision trees (GBDTs) has also been proven effective. This ensemble approach integrates multiple weak learners to enhance classification stability and accuracy, especially in complex land cover scenarios [32]. To better exploit the complementary nature of spatial and spectral information, researchers have increasingly turned to dual-branch architectures. For example, the Double-Branch Dual-Attention (DBDA) network [33] utilizes two independent branches to capture spatial and spectral features, respectively, enhanced by dual-attention mechanisms. Similarly, the Multiscale Neighborhood Attention Transformer (MSNAT) [34] further refines this by extracting multi-scale features to adapt to varying land cover sizes. To address feature extraction from a frequency perspective, the Dual Frequency Transformer Network (DFTN) [35] was developed. It utilizes a dual-branch frequency domain feature extraction block to simultaneously capture high-frequency local details and low-frequency global variations, demonstrating the effectiveness of frequency domain information in enhancing HSI classification. Alkhatib et al. [36] proposed an attention-based dual-branch network that fuses features from a real-valued neural network (RVNN) and a complex-valued neural network (CVNN). By using the Fourier transformation to extract frequency information, this model enhances HSI classification performance. While these models have significantly improved classification accuracy, they often face challenges such as high parameter redundancy and suboptimal fusion of heterogeneous features from the two branches. Many researchers have focused on efficient spatial–spectral feature extraction. Yang et al. [3] introduced the multi-scale hybrid CNN–attention (MS-Hybrid-A) network, which uses 3D convolutions for spectral–spatial features extraction. Additionally, it supplementary extracts spatial details with 2D convolutions and incorporates a convolutional block attention module (CBAM) to improve classification performance. Liang et al. [37], inspired by the Transformer, proposed HSI-Mixer, which utilizes a hybrid measurement-based linear projection (HMLP) module for deep spectral–spatial feature fusion. Kong et al. [38] developed a co-feature extraction framework that integrates graph embedding with deep learning. They constructed a supervised within-class/between-class hypergraph (SWBH) for spectral feature learning and introduced a random zero-masking strategy to generate augmented labeled samples. It could facilitate CNN-based spatial feature extraction and mitigate overfitting in small-sample settings. Ahmad et al. [39] proposed the spatial morphological Mamba (SMM) and spatial–spectral morphological Mamba (SSMM) networks, which employ depth-wise separable convolutions to implement morphological operations, like erosion and dilation. By leveraging State Space Models (SSMs), these Mamba-based approaches achieve linear computational complexity while effectively modeling long-range dependencies, offering a promising, efficient alternative to standard Transformers for processing high-dimensional spectral sequences. To address the challenge of cross-layer information loss, Chen et al. [40] designed a hybrid pooling attention (HPA) module and a cross-layer feature fusion (CFF) module to preserve crucial information during the propagation process. Gao et al. [41] introduced a plug-and-play adaptive feature fusion (AFF) module that processes multi-layer networks to better utilize spatial and spectral features. Guo et al. [42] proposed an adaptive score-weighting method to fuse features from spatial and spectral branches. Similarly, the concept of reference-based adaptive modulation has been explored in multi-style fusion tasks [43], providing insights into the dynamic adjustment of features across different domains.

In conclusion, traditional machine learning methods such as SVM and RF perform well on high-dimensional data when only a few labeled samples are available. However, they have limited capacity to model nonlinear relationships and thus struggle to process large-scale, high-dimensional data with complex noise. Deep learning methods, by contrast, greatly improve feature representation through hierarchical abstraction. Nevertheless, they require large quantities of high-quality labeled samples and bring drawbacks such as long training times and high computational cost. Furthermore, multi-model fusion methods can mitigate overfitting effectively and reduce dependence on labeled samples. However, they pose challenges in designing compatible modules and incur heavy computational overhead.

The existing research still lacks effective methods to dynamically fuse spatial and spectral features for HSI classification. Furthermore, the number of model parameters and computational overhead remain too high. In the broader field of remote sensing, highly efficient architectures like SFEARNet [44] have successfully combined semantic flow and edge-aware refinement for tasks such as change detection, demonstrating the importance of structural information in efficient network design. Furthermore, the hybrid CNN–Transformer architecture has been widely validated in diverse remote sensing applications, including cloud detection using Landsat/Sentinel data [45], wind speed sensing using Global Navigation Satellite System Reflectometry (GNSS-R) [46], and chlorophyll concentration inversion using SeaWiFS data [47]. These successes in handling complex, multi-modal remote sensing data provide a strong rationale for adopting a hybrid design to capture both local spatial details and global spectral dependencies in HSI classification. Building on the need for efficiency, this article proposes a dual-branch network called the spectral integration and focused attention network (SIFANet). SIFANet is built on a hybrid CNN–Transformer structure and incorporates a channel attention mechanism to emphasize important spectral bands. By optimizing feature extraction and fusion, SIFANet could improve HSI classification performance and enhance classification accuracy to a certain extent.

The main contributions of this article are as follows:

• Efficient feature extraction structure
  This article designs a dual-branch network composed of a spatial feature extractor (SFE) and a spectral sequence Transformer (SST). The SFE is enhanced with residual blocks (RBs) to alleviate the vanishing gradient problem and accelerate convergence. Simultaneously, the SST incorporates a Conv-Former module to improve spectral feature extraction, enabling the efficient and parallel extraction of spatial–spectral features.
• Cross-Module Attention Fusion (CMAF)
  This article introduces a channel attention-based CMAF mechanism to dynamically and adaptively fuse features from different branches, which significantly reduces information loss during the feature integration step.
• Comprehensive HSI classification accuracy assessment indices
  This article develops a novel computation accuracy parameter efficiency (CAPE) index to quantify the computational efficiency of different models. In addition, the proposed evaluation index system (EIS) also includes classification accuracy metrics, confusion category performance, and computational efficiency index, enabling a comprehensive, multidimensional assessment of the model performance.

2. Methodology

The research workflow is illustrated in Figure 1. First, principal component analysis (PCA) was applied to reduce the dimensionality of the HSI data. To retain most of the information while significantly compressing the data, we retained the first few principal components (PCs) based on a cumulative variance contribution rate threshold of >99%. Specifically, for the Salinas dataset (originally 204 bands), the first 7 PCs were retained, preserving 99.12% of the spectral information. For the Xiong’an hyperspectral dataset (originally 256 bands), the first 6 PCs were retained, preserving 99.09% of the information. Subsequently, spatial and spectral feature tensors were generated via a masking operation and then were input into SIFANet. Within SIFANet, spatial and spectral features were captured through SFE and SST branches, respectively. CMAF was employed to achieve the dynamic weighted fusion. Finally, the fused features were fed into the classifier to produce the result. During the validation step, SIFANet was comprehensively evaluated and compared against other comparable models using the EIS.

2.1. SIFANet Structure

The SIFANet adopts a parallel dual-branch structure rather than a serial one to decouple the extraction of spatial and spectral features. While serial structures often lead to the loss of original information during sequential processing, the parallel design allows the SFE and SST to independently capture local textures and global dependencies. This simultaneous processing ensures that the distinctive characteristics of both domains are fully preserved before being dynamically integrated through the CMAF module, thereby maximizing feature complementarity.

2.1.1. Spatial Feature Extraction

SFE branch adopts a “Conv + RB” CNN variant structure. It applies residual learning to improve gradient flow and mitigate the vanishing gradient problem. Additionally, max-pooling downsampling is incorporated to reduce computational complexity and accelerate training.

Figure 2 illustrates the detailed structure of the SFE module. It begins by applying a 3 × 3 convolutional kernel to extract preliminary spatial features from local neighborhoods. To further reduce spatial dimensionality, a max-pooling layer is subsequently applied, formulated as

$${\mathit{X}}_{3\mathit{d}}=f_{MP}\left(f_{Conv}\left(\mathit{X}\right)\right)$$

(1)

where $\mathit{X}$ is the input tensor, ${\mathit{X}}_{3\mathit{d}}$ is the processing result, $f_{Conv}$ and $f_{MP}$ represent the 3 × 3 convolution layer and the max-pooling layer, respectively. $f_{MP}$ uses a 2 × 2 window and a stride of 2 for pooling.

The RB module adopts a two-branch structure to facilitate feature reuse. The main branch consists of two consecutive “Conv + BatchNorm (BN)” layers, which are used to extract deeper spatial features. BN standardizes intermediate feature distributions across mini batches to mitigate the vanishing gradient problem and accelerate model convergence. The residual output is computed as follows:

$${\mathit{X}}_{\mathit{B}\mathit{N}}={\displaystyle \frac{\mathit{X}-\overline{\mathit{X}}}{\sqrt{{\sigma }_X^2+ϵ}}}·\gamma +\beta$$

(2)

where $\mathit{X}$ is the residual block input tensor, ${\mathit{X}}_{\mathit{B}\mathit{N}}$ is the output of the residual block, $\overline{\mathit{X}}$ is the mean of the input tensor, ${\sigma }_X^2$ is the variance of the input data, $ϵ$ is a small constant added for numerical stability, $\gamma$ and $\beta$ are learnable parameters.

Nonlinearity is introduced through the ReLU activation function, The operation is expressed as follows:

$${\mathit{X}}^{\mathit{\prime }}=f_{Max}\left(0,\mathit{X}\right)$$

(3)

where $\mathit{X}$ denotes the input tensor, ${\mathit{X}}^{\mathit{\prime }}$ is the output tensor, and $f_{Max}$ represents the element-wise maximum function used ReLU.

The skip connection employs cross-layer identity mapping to directly propagate the shallow features ${\mathit{X}}_{3\mathit{d}}$ to deeper layers, thereby preserving important spatial information. The fused output is obtained through element-wise feature summation. Denoting the sequence of operations in the main path as $f_{RB}$, and using $\oplus$ to represent summation, the operation can be defined as follows:

$${\mathit{X}}_{\mathit{R}\mathit{B}}={\mathit{X}}_{3\mathit{d}}\oplus f_{RB}\left({\mathit{X}}_{3\mathit{d}}\right)$$

(4)

Finally, at the output stage, an adaptive average pooling layer is applied to adjust the spatial resolution of the feature maps to a fixed size, and the result is denoted as ${\mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}$. This operation ensures consistent output feature dimensions across both processing branches.

2.1.2. Spectral Sequence Transformer

Some land cover categories are difficult to distinguish when only relying on spatial features due to their high spatial similarity. To address this issue, this article proposes an SST branch, which is illustrated in Figure 3.

A fully connected layer processes the input $\mathit{X}$, converting the spectral vector of each pixel into a token embedding. This transformation constructs a sequence of tokens ${\mathit{X}}_{\mathit{t}\mathit{o}\mathit{k}\mathit{e}\mathit{n}}$, forming the primary input sequence for the Transformer. Concurrently, a lower branch generates a learnable positional embedding $\mathit{P}$ designed to encode the position. $\mathit{P}$ is added element-wise to the token sequence to obtain the position-augmented input representation ${\mathit{X}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{c}\mathit{k}\mathit{e}\mathit{d}}$, i.e.,

$${\mathit{X}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{c}\mathit{k}\mathit{e}\mathit{d}}={\mathit{X}}_{\mathit{t}\mathit{o}\mathit{k}\mathit{e}\mathit{n}}\oplus \mathit{P}$$

(5)

The position-augmented spectral sequence ${\mathit{X}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{c}\mathit{k}\mathit{e}\mathit{d}}$ is then fed into the improved Conv-Former encoder module for feature encoding, producing ${\mathit{Z}}_{\mathit{o}\mathit{u}\mathit{t}}$. Finally, a linear layer projects ${\mathit{Z}}_{\mathit{o}\mathit{u}\mathit{t}}$ to match the output dimensions of the SFE branch. The final spectral feature representation is given by

$${\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}={\mathit{W}}_{\mathit{p}}{\mathit{Z}}_{\mathit{o}\mathit{u}\mathit{t}}+{\mathit{b}}_{\mathit{p}}$$

(6)

where ${\mathit{W}}_{\mathit{p}}$ is the weight matrix of the linear layer, ${\mathit{b}}_{\mathit{p}}$ is the bias vector of the linear layer, and ${\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}$ is the spectral feature information output by the SST branch.

This article combined convolution with a Transformer encoder to leverage the advantages of convolution operations and self-attention mechanisms. Inspired by Lin et al. [48], the improved “Conv-Former” structure is shown in Figure 4. We have significantly adapted the module for HSI-specific characteristics. Specifically, the conventional feed-forward network is replaced with a convolutional feed-forward network based on 1 × 1 convolutions, which efficiently reorganizes and compresses high-dimensional spectral channels while preserving spatial information. Batch normalization is employed throughout the module to improve training stability under the small-sample regime commonly encountered in HSI applications. Together, these modifications allow the proposed module to better capture discriminative spatial–spectral representations and make it more suitable for HSI analysis.

The convolution-attention residual module processes features through its dual-branch structure. The main branch extracts features with a convolution-attention block (CAB), while the auxiliary branch preserves the original input features. The output of the module is computed as

$${\mathit{X}}_{\mathit{C}\mathit{A}}=\mathit{X}\oplus f_{CAB}\left(\mathit{X}\right)$$

(7)

where $\mathit{X}$ is the input tensor, ${\mathit{X}}_{\mathit{C}\mathit{A}}$ is the module’s output, and $f_{CAB}$ denotes the operations within the CAB.

The CAB adopts a “Conv + Self-Attention” processing method. It first applies a 3 × 3 convolution to extract local spectral features, then constructs the query ($\mathit{Q}$), key ($\mathit{K}$), and value ($\mathit{V}$) matrices for self-attention (SA) computation. A multi-head self-attention (MHSA) mechanism with $h=4$ is employed to model global contextual relationships, balancing computational costs and model representational capacity. The SA process is defined as

$${\mathit{M}}_{\mathit{h}}=f_{SM}\left({\displaystyle \frac{\mathit{Q}\otimes {\mathit{K}}^{\mathit{T}}}{\sqrt{d_k}}}\right)$$

(8)

$$SA\left(\mathit{Q},\mathit{K},\mathit{V}\right)={\mathit{M}}_{\mathit{h}}\otimes \mathit{V}$$

(9)

where $d_k$ denotes the input sequence dimension, ${\mathit{M}}_{\mathit{h}}$ represents the attention weight matrix, ${\mathit{K}}^{\mathit{T}}$ is the transpose of $\mathit{K}$, $\otimes$ indicates matrix multiplication, and $f_{SM}$ is the SoftMax function.

The MHSA mechanism captures diverse dependencies and hierarchical relationships by integrating multiple self-attention heads [24]. The overall MHSA output is expressed as

$$MHSA\left(\mathit{Q},\mathit{K},\mathit{V}\right)=f_{CAT}\left({SA}_1, {SA}_2,{SA}_3,\dots ,{SA}_h\right)$$

(10)

where ${SA}_i\left(i=\mathrm{1,2},3,\dots ,h\right)$ represents the self-attention result of the $i$-th part. $f_{CAT}$ is the Concat function.

Finally, a 1 × 1 convolution layer is applied to integrate the features obtained from global dependencies, resulting in the refined output ${\mathit{X}}_{\mathit{S}\mathit{A}}$.

The current feed-forward network (CFFN) refines features from the upper layer through the following operation:

$${\mathit{X}}_{\mathit{C}\mathit{F}\mathit{F}\mathit{N}}={\mathit{X}}_{\mathit{C}\mathit{A}}\oplus f_{CFFN}\left({\mathit{X}}_{\mathit{C}\mathit{A}}\right)$$

(11)

where ${\mathit{X}}_{\mathit{C}\mathit{F}\mathit{F}\mathit{N}}$ is the output and $f_{CFFN}$ refers to the transformation process within the current feed-forward network (CFFN).

2.1.3. Cross-Module Attention Fusion

To facilitate cross-modal feature fusion, this article designs the CMAF module, as shown in Figure 5, inspired by the convolutional block attention module (CBAM) [49].

CMAF first constructs a channel attention weight matrix with a shared multi-layer perception (Shared MLP). Specifically, the features ${\mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}$ and ${\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}$ are fed into the Shared MLP, where the input and hidden layers are connected via fully connected networks to enable feature interaction. After being processed by the hidden layer and activated by the ReLU function, the compressed and expanded feature vectors ${\mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}^{\mathrm{\prime }}$ and ${\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}^{\mathrm{\prime }}$ are output through another fully connected layer. Then, ${\mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}^{\mathrm{\prime }}$ and ${\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}^{\mathrm{\prime }}$ are concatenated along the channel dimension, followed by Sigmoid activation to produce the channel attention weight matrix ${\mathit{M}}_{\mathit{c}}$. The process is defined as follows:

$${\mathit{s}}_{\mathit{C}\mathit{N}\mathit{N}}=f_{Relu}\left({\mathit{W}}_0{\otimes \mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}^{\mathit{\prime }}\right)$$

(12)

$${\mathit{s}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}=f_{Relu}\left({\mathit{W}}_0\otimes {\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}^{\mathit{\prime }}\right)$$

(13)

$${\mathit{M}}_{\mathit{c}}=\sigma \left({\mathit{W}}_1\otimes {\mathit{s}}_{\mathit{C}\mathit{N}\mathit{N}}+{\mathit{W}}_1\otimes {\mathit{s}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}\right)$$

(14)

The above Equations (12) and (13) denote the hidden layer processing and ReLU activation, while (14) describes the steps of output and Sigmoid activation. $\sigma$ is the Sigmoid function; ${\mathit{W}}_0$ and ${\mathit{W}}_1$ correspond to the weight matrices of the input layer and output layer, respectively. ${\mathit{s}}_{\mathit{C}\mathit{N}\mathit{N}}$ is the spatial feature information after hidden layer processing, ${\mathit{s}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}$ is the spectral feature information after hidden layer processing, and $f_{Relu}$ is the ReLU function.

After calculating ${\mathit{M}}_{\mathit{c}}$, the original features ${\mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}$ and ${\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}$ are, respectively, channel-weighted with ${\mathit{M}}_{\mathit{c}}$ through matrix multiplication and the weighted results are stacked to obtain the fused feature ${\mathit{F}}_{\mathit{f}\mathit{u}\mathit{s}\mathit{e}\mathit{d}}$.

$${\mathit{F}}_{\mathit{f}\mathit{u}\mathit{s}\mathit{e}\mathit{d}}=\left({\mathit{M}}_{\mathit{c}}\otimes {\mathit{F}}_{\mathit{C}\mathit{N}\mathit{N}}\right)\oplus \left({\mathit{M}}_{\mathit{c}}\otimes {\mathit{F}}_{\mathit{T}\mathit{r}\mathit{a}\mathit{n}\mathit{s}}\right)$$

(15)

Although the proposed CMAF module is inspired by classical channel attention mechanisms such as the Squeeze-and-Excitation Network (SE-Net) [50] and CBAM, it differs fundamentally in both design objective and information flow. SE-Net and CBAM generate channel attention weights based on the statistical information of a single feature branch, focusing on self-channel recalibration. In contrast, CMAF jointly models CNN-based features and Transformer-based features through a shared MLP to generate a unified channel attention matrix.

By leveraging cross-module feature interaction, the attention weights in CMAF dynamically reflect the relative contribution of different channels from heterogeneous representations, enabling adaptive fusion of local spatial features and global contextual features. Therefore, CMAF acts as a dynamic cross-module fusion mechanism rather than a conventional self-attention-based feature enhancement module.

2.2. Space–Spectral Feature Tensor Generation

SIFANet is for multi-modal tensor inputs, with the SFE and SST branches requiring spatial feature tensors and spectral feature tensors, respectively.

For the SFE branch, the input tensor is constructed from multiple image blocks, with each input tensor having the shape ${\mathit{X}\in \mathbb{R}}^{B\times P\times P}$, where $B$ is the batch size and $P$ is the image block size. As shown in Figure 6, the specific operation for obtaining image blocks is as follows:

First, the original image is padded with a margin of $⌊{\displaystyle \frac{\left(P-1\right)}{2}}⌋$ pixels on all sides to ensure that edge pixels are adequately included. Then, a sliding window (also referred to as a mask) of size $P\times P$ is applied, starting from the first valid pixel in the top-left corner. The window moves across the entire image with a stride of 1 pixel, excluding padded and unclassified pixels. For each valid central pixel, a corresponding local neighborhood is extracted to form an image block. Each block is represented as ${{\mathit{X}}_{\mathit{P}\mathit{a}\mathit{t}\mathit{c}\mathit{h}}\in \mathbb{R}}^{P\times P}$, and multiple such blocks are batched together to construct the spatial input tensor.

For the SST branch, as shown in Figure 7, the multi-band spectral reflectance data of each pixel are directly extracted and assembled in batches to construct the spectral feature input tensor ${\mathit{X}\in \mathbb{R}}^{\mathit{B}\times 1\times \mathit{M}}$, where $\mathit{M}$ is the number of bands. It retains the spatial location of pixels while highlighting the discriminative features, avoiding spatial neighborhood interference, and enhancing the sensitivity to subtle spectral differences.

2.3. Evaluation Index System

As illustrated in Figure 8, the proposed EIS incorporates three core modules. The first module utilizes classification accuracy indexes, including class accuracy (CA), overall accuracy (OA), the Kappa coefficient, and standard deviation (S.D.). These indices jointly evaluate the model’s recognition capability of individual categories and its overall classified consistency and stability. The second module specifically addresses the performance evaluation of confusing classes by integrating the precision–recall curve (PRC) and F1-Score into a unified PR–F comprehensive evaluation model. It focuses on comparative assessment of a model’s discriminatory capacity for challenging, spectrally similar categories. The third module evaluates computational efficiency through the novel CAPE index, which provides a quantitative assessment by systematically integrating key factors including the model’s computational demands, parameter count, achieved classification accuracy, and measured stability performance.

2.3.1. Overall Accuracy

OA represents the proportion of correctly classified samples, calculated as

$$OA={\displaystyle \frac{T_c}{N}}$$

(16)

where $T_c$ denotes the number of correctly classified samples and $N$ denotes the total number of samples.

2.3.2. Kappa Coefficient

The Kappa coefficient measures classification effectiveness while accounting for class imbalance. The calculation formula is as follows:

$$p_e={\displaystyle \frac{\sum _{i=1}^m\left(R_i\times C_i\right)}{N^2}}$$

(17)

$$Kappa={\displaystyle \frac{OA-p_e}{1-p_e}}$$

(18)

where $p_e$ is the chance agreement probability under random classification, $m$ denotes the number of categories, $R_i$ denotes the sum of row $i$ in the confusion matrix, $C_i$ denotes the sum of the column $i$ in the confusion matrix, $N$ denotes the total number of samples.

2.3.3. PR–F Comprehensive Evaluation Model

For the easily confusable ground object types, this article utilizes the PR–F comprehensive evaluation model, draws PRC, and statistically reports the F1-Score and AP values for auxiliary evaluation.

The F1-Score is a comprehensive metric used to evaluate a model’s performance by balancing precision and recall. Precision refers to the proportion of correctly predicted positive samples among all samples predicted as positive. Recall measures the proportion of correctly predicted positive samples among all actual positive samples, indicating the model’s ability to identify all relevant cases. These metrics are derived from the confusion matrix: true positives (TPs) represent the number of samples correctly predicted as positive; false positives (FPs) are the samples incorrectly predicted as positive; and false negatives (FNs) are the actual positive samples that the model failed to identify. Precision, recall, and F1-Score are calculated as follows:

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(19)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(20)

$$F1-Score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(21)

The precision–recall curve (PRC) is used to evaluate the performance of a model by illustrating how precision and recall change at different classification thresholds. The closer the model’s PRC is to the upper right of the coordinate system, the better the model’s performance. To quantify overall model performance, the average precision (AP) is calculated as the area under the PRC. The closer the AP value is to 1, the better the model balances precision and recall.

2.3.4. Computational Accuracy Parameter Efficiency

Traditional single metrics such as OA focus solely on performance, overlooking computational burden and offering limited insight into model applicability [51]. To address this limitation, the CAPE index integrates multiple factors: the number of parameters, the number of floating-point operations (FLOPs), the S.D., and OA. It provides a comprehensive quantitative evaluation of a model’s computational efficiency, classification accuracy, and prediction stability. A higher CAPE value indicates superior model performance—simultaneously reflecting higher accuracy, greater stability, optimized efficiency, and balanced computational design.

To unify measurement scales and introduce a penalization mechanism, the number of parameters (P) and FLOPs (F) are logarithmically transformed using $y={\mathit{log}}_{10}x$. Penalization is implemented via the reciprocal:

$$N_P={\displaystyle \frac{1}{{\mathit{log}}_{10}\left(1+P\right)}}$$

(22)

$$N_F={\displaystyle \frac{1}{{\mathit{log}}_{10}\left(1+F\right)}}$$

(23)

here, $N_P$ and $N_F$ represent the normalized values of parameters and FLOPs. When either P or F is large, their corresponding normalized values decrease significantly, penalizing models with excessive computational costs.

The intermediate metric $E$, representing computational burden, is then calculated as the geometric mean of $N_P$ and $N_F$. A lower E score indicates higher computational resource consumption:

$$E=\sqrt{N_P\times N_F}$$

(24)

The final CAPE score is computed by E, OA, and the inverse of the S.D. (${\displaystyle \frac{1}{\sigma }}$):

$$CAPE=E\times OA\times {\displaystyle \frac{1}{\sigma }}$$

(25)

This multiplicative formulation highlights performance differences among models and balances high accuracy with prediction stability. Notably, even models with high OA could receive a low CAPE score if they have a high S.D.

3. Experimental Design and Results

3.1. Dataset

3.1.1. Salinas Dataset

The Salinas dataset (denoted as SL), acquired by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) (developed by NASA’s Jet Propulsion Laboratory, Pasadena, CA, USA) covers a portion of the Salinas Valley in California, USA. The image is 217 pixels × 512 pixels, with a spatial resolution of 3.7 m. It originally comprised 224 spectral bands. After removing bands affected by water absorption, 204 spectral bands remain, spanning the 400–1000 nm wavelength range. This dataset primarily includes 16 categories of crops. The corresponding ground truth map is shown in Figure 9.

3.1.2. Xiong’an Hyperspectral Dataset

The Xiong’an dataset (denoted as XA) was acquired using the Full-spectrum Multi-modal Imaging Spectrometer (manufactured by Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai, China) [52]. It covers an area of approximately 1320 km², including Xiong County, Anxin County, Rongcheng County, and the main water area of Baiyangdian Lake. This article uses a subset of Matiwan Village, sized 3750 × 1580 pixels at 0.5 m spatial resolution, with 256 spectral bands (400–1000 nm). It includes 20 land cover classes, mainly cash crops (Figure 10).

The key characteristics of the two datasets are summarized in

Table 1. Key characteristics of the two datasets.

Dataset	Image Size (pixel)	Spatial Resolution (m)	Wavelength Range (nm)	Bands	Classes
Salinas	217 × 512	3.7	400–1000	204	16
Xiong’an	3750 × 1580	0.5	400–1000	256	20

for a clear comparison.

The Salinas and Xiong’an datasets were selected for their complementary traits: the Salinas dataset, being small-scale with limited samples, tests small-sample learning and performance under scarcity. In contrast, the large-scale Xiong’an dataset assesses generalization ability and computational efficiency. The Salinas dataset, as a benchmark, enables direct comparison with prior works, while Xiong’an, with complex land cover and higher resolution, reflects real-world diversity and tests model adaptability.

To evaluate the performance and robustness of SIFANet under various data constraints, we first conducted a sensitivity analysis by varying the training sample proportions from 1% to 20%. The results, summarized in

Table 2. OA of SIFANet under different training sample proportions. (Unit: %).

Dataset	1%	5%	10%	15%	20%
Salinas	91.02	97.45	98.92	99.34	99.89
Xiong’an	85.30	94.18	97.05	98.42	99.79

, reveal that SIFANet achieves competitive accuracy even with extremely limited training data (e.g., 1% or 5%). As the proportion increases to 20%, the classification performance tends to saturate, indicating that the model has effectively captured the discriminative spectral–spatial features.

The rationale for selecting the 20% ratio as our baseline is twofold. First, high-spectral data typically exhibit strong spatial correlation; using higher ratios (e.g., 30% or 40%) may lead to overfitting on local spatial patterns rather than learning generalized features [53]. Second, the 20% ratio strikes a balance between ensuring deep learning model convergence and simulating real-world scenarios where labeled samples are scarce and expensive to obtain. Based on this analysis, the detailed partitioning of training and testing samples for each dataset is provided in

Table 3. Sample sizes for training and testing in the dataset.

	Salinas Dataset			Xiong’an Hyperspectral Dataset
NO.	Class Name	Training	Test	Class Name	Training	Test
1	Broccoli_green_weeds_1	399	1610	Compound-leaved maple	45,153	180,764
2	Broccoli_green_weeds_2	784	2942	Willow	35,829	144,124
3	Fallow	381	1595	Elm	3067	12,222
4	Fallow_rough_plow	284	1110	Rice	90,871	361,574
5	Fallow_smooth	535	2143	Pagoda tree	94,755	380,562
6	Stubble	798	3161	Fraxinus	34,056	135,125
7	Celery	730	2849	Golden rain tree	4629	18,695
8	Grapes_untrained	2222	9049	Water	33,125	132,610
9	Soil_vinyard_develop	1226	4977	Bare soil	7800	30,611
10	Corn_senesced_green_weeds	626	2652	Rice stubble	38,763	155,348
11	Lettuce_romaine_4wk	229	839	Black locust	1093	4471
12	Lettuce_romaine_5wk	390	1537	Corn	11,675	47,331
13	Lettuce_romaine_6wk	174	742	Pear tree	205,019	820,983
14	Lettuce_romaine_7wk	230	840	Soybean	1437	5711
15	Vinyard_untrained	1463	5805	Poplar	18,290	73,019
16	Vinyard_vertical_trellis	354	1453	Vegetable field	5875	23,138
17				Sparse forest	297	1218
18				Grassland	84,691	337,850
19				Peach tree	13,070	52,552
20				Building	5926	23,781
Total		10,825	43,304		735,421	2,941,689

.

3.2. Experimental Design

This article employs a controlled variable experimental design, varying only the model type to reduce systematic errors and to enhance comparability. All experiments were implemented using the PyTorch (version 2.6.0) framework. For clarity, the specific structural hyperparameters of SIFANet—including kernel sizes, embedding dimensions, and fusion settings—are listed in

Table 4. Detailed configuration of SIFANet architecture.

Module	Parameter/Component	Value/Configuration
SFE Branch	Structure	Conv + MaxPool + Residual Block
	Conv Kernel Size	3 × 3
	Max Pooling	Window 2 × 2, Stride 2
	Residual Block	2 Layers (3 × 3 Conv + BN + ReLU)
SST Branch	Structure	Conv-Former
	Encoder Layers	3
	Attention Heads	4
	Feed-Forward Network	CFFN (1 × 1 Conv)
	Embedding Dimension	64
CMAF	Interaction Mechanism	Shared MLP
	Activation Function	Sigmoid
	Hidden Layer Units	32

. Specifically, the training process consisted of 200 epochs with a batch size of 256 and an initial learning rate of 0.0001. All experimental results reported are an average of 50 independent experiments.

The EIS proposed in this article is used to assess and compare the classification performance of SIFANet with that of SVM, 3D-CNN, and LSTM. Here, SVM is a typical traditional machine learning model, while 3D-CNN and LSTM are deep learning models. Specifically, 3D-CNN can capture local spatial features through convolution. LSTM excels at processing sequence data. By comparing these diverse benchmarks, this article validates the effectiveness of SIFANet for HSI classification tasks.

3.3. Key Parameter Setting Experiments

3.3.1. Image Block Size

Due to constraints in image width and computing resources, the original HSI is partitioned into multiple blocks of identical size, each containing an odd number of pixels. Since block size significantly affects model performance [54], the optimal size was determined experimentally. Holding other parameters constant, this study tested block sizes from 7 × 7 to 25 × 25 using a 3D-CNN on both the Salinas and Xiong’an datasets. OA was used for evaluation, with results averaging over 10 runs, as shown on Figure 11.

Block size nonlinearly influences model performance. Both datasets show a consistent trend: OA first rises with increasing block size, peaks, and then declines. Specifically, smaller blocks (e.g., 7 × 7, 9 × 9) lead to insufficient spatial feature extraction due to limited receptive fields, yielding lower OA. At 15 × 15, OA reaches 98.99% (SL) and 99.72% (XA)—gains of 6.1% and 6.35% over 7 × 7. Although XA peaks at 99.79% with 17 × 17, the 0.07% difference from 15 × 15 is statistically insignificant. Beyond 17 × 17, performance drops due to boundary effects and local feature dilution, likely caused by increased background noise, reduced focus on discriminative features, and edge information loss.

Considering both marginal gains and computational cost, 15 × 15 was selected as optimal. It achieves peak accuracy on the Salinas dataset and near-peak on Xiong’an dataset, with high computational efficiency, and served as the input size for all subsequent experiments.

3.3.2. Optimizer and Loss Function

The loss function quantifies the error between the predictions and ground truth, guiding parameter updates. The optimizer’s role is to iteratively adjust parameters to minimize the loss, thereby enhancing predictive capability.

Common optimizers include stochastic gradient descent (SGD) and adaptive moment estimation (Adam). SGD uses random data subsets for gradient updates, offering higher efficiency but potentially causing oscillations or slow convergence. Adam estimates gradient moments to combine momentum and adaptive learning rates, enabling faster and more stable convergence. For loss functions, squared hinge loss (SHL) strengthens misclassification penalties via a squared term to maximize margins, while categorical cross-entropy (CCE) minimizes the Kullback–Leibler divergence between predicted and true distributions, aligning the model with inherent class distributions.

This article experimentally evaluated combinations of these optimizers and loss functions, with results averaged over 10 runs (Figure 12).

“Adam + CCE” achieved the best OA on both Salinas (99.04%) and Xiong’an (99.42%). For a fixed optimizer, CCE consistently outperformed SHL; for a fixed loss, Adam surpassed SGD. Thus, “Adam + CCE” was selected for subsequent experiments.

3.4. Experiments Results

Based on the experimental setup discussed above, this article compares the classification performance of SVM, 3D-CNN, LSTM, and SIFANet on the Salinas and the Xiong’an hyperspectral datasets. The results are shown in

Table 5. Results of Salinas dataset. (Unit: %).

No.	SVM		3D-CNN		LSTM		SIFANet
1	96.56	−3.44	100.00	0.00	100.00	0.00	100.00
2	91.74	−8.16	100.00	0.10	99.50	−0.40	99.90
3	88.84	−10.97	99.07	−0.74	97.12	−2.69	99.81
4	98.18	−0.56	98.93	0.19	99.04	0.30	98.74
5	83.31	−16.60	98.82	−1.09	99.02	−0.89	99.91
6	99.97	−0.03	99.87	−0.13	100.00	0.00	100.00
7	92.92	−6.94	100.00	0.14	99.93	0.07	99.86
8	64.16	−35.83	99.04	−0.95	80.80	−19.19	99.99
9	86.58	−13.42	99.88	−0.12	99.13	−0.87	100.00
10	69.74	−30.18	99.62	−0.30	96.39	−3.53	99.92
11	0.00	−99.41	100.00	0.59	98.25	−1.16	99.41
12	66.84	−32.70	98.93	−0.61	97.25	−2.29	99.54
13	91.30	−8.56	100.00	0.14	98.30	−1.56	99.86
14	94.24	−5.41	97.89	−1.76	96.40	−3.25	99.65
15	58.87	−41.13	96.35	−3.65	78.02	−21.98	100.00
16	99.01	−0.85	100.00	0.14	99.86	0.00	99.86
OA	79.18	−20.71	99.05	−0.84	92.36	−7.53	99.89
Kappa	76.55	−23.33	98.95	−0.93	91.48	−8.40	99.88
S.D.	25.18	−24.85	1.00	−0.67	6.68	−6.35	0.33

and

Table 6. Results of Xiong’an hyperspectral dataset. (Unit: %).

No.	SVM		3D-CNN		LSTM		SIFANet
1	33.20	−66.52	99.53	−0.19	78.48	−21.24	99.72
2	51.81	−48.10	99.84	−0.07	77.48	−22.43	99.91
3	0.00	−99.92	99.61	−0.31	78.27	−21.65	99.92
4	93.80	−6.19	99.97	−0.02	98.11	−1.88	99.99
5	40.49	−59.40	99.31	−0.58	72.22	−27.67	99.89
6	52.00	−47.94	99.72	−0.22	77.28	−22.66	99.94
7	85.99	−13.99	99.87	−0.11	88.03	−11.95	99.98
8	89.17	−10.82	99.93	−0.06	97.06	−2.93	99.99
9	75.15	−24.84	100.00	0.01	98.61	−1.38	99.99
10	86.08	−13.92	100.00	0.00	98.34	−1.66	100.00
11	0.00	−99.62	97.05	−2.57	54.95	−44.67	99.62
12	1.19	−98.39	98.46	−1.12	69.14	−30.44	99.58
13	53.59	−46.07	99.05	−0.61	74.35	−25.31	99.66
14	0.00	−99.25	99.37	0.12	73.71	−25.54	99.25
15	58.06	−41.50	99.03	−0.53	72.60	−26.96	99.56
16	82.67	−14.93	94.82	−2.78	70.46	−27.14	97.60
17	0.00	−99.32	93.33	−5.99	25.00	−74.32	99.32
18	45.36	−54.45	99.54	−0.27	74.37	−25.44	99.81
19	0.00	−99.78	99.46	−0.32	74.68	−25.10	99.78
20	97.84	−1.60	99.60	0.16	95.87	−3.57	99.44
OA	59.11	40.68	99.42	−0.37	80.19	−19.60	99.79
Kappa	50.80	−48.96	99.33	−0.43	76.83	−22.93	99.76
S.D.	37.09	−36.51	1.95	−1.37	18.54	−17.96	0.58

. These tables present the classification accuracy in the format of “accuracy ± difference from SIFANet”. Different colors are used to distinguish comparison results: orange indicates accuracy lower than SIFANet, blue indicates accuracy better than SIFANet, and black indicates parity. Red highlights easily confused categories. Box plots of classification accuracy are also provided for enhanced visualization (Figure 13).

SIFANet demonstrates superior performance with the OA of 99.89% (SL)/99.79% (XA), outperforming the second-best 3D-CNN model by 0.84% and 0.37%, respectively. The Kappa coefficient of the proposed SIFANet reaches 99.88% (SL)/99.76% (XA), significantly exceeding other models. The standard deviation remains low at 0.33% (SL)/0.58% (XA), indicated strong adaptability and robustness across land cover categories, whereas other models exhibit substantial fluctuations.

Figure 13 shows pronounced performance differences among the models. Ranked by median accuracy, the order is consistently SIFANet > 3D-CNN > LSTM > SVM. SVM exhibits high inter-category variability, with interquartile ranges of 40% (SL) and 100% (XA), and numerous outliers. LSTM outperforms SVM but still shows limitations (e.g., 45% interquartile range on XA). The 3D-CNN maintains relatively compact quartile ranges and high accuracy but has minor outliers. In contrast, SIFANet shows minimal variance, with narrow quartile ranges and peaked distributions, indicating exceptional stability.

Across all land cover categories, SIFANet consistently maintains over 97% accuracy. The SVM model, constrained by linear classification mechanisms, drops to 0% in several categories with 25.18% (SL)/37.09% (XA) standard deviations. Additionally, both LSTM and 3D-CNN fail to fully capture the spatial–spectral information, causing significant decline in specific categories’ accuracy such as classes 8, 14, and 15 (SL) and 11, 16, and 17 (XA), where performance substantially lags behind the proposed SIFANet. Focusing on poorly classified categories, as for class 8 (Grapes_untrained) in Salinas, SVM’s accuracy is merely 64.16%, LSTM reaches only 80.80%, while SIFANet achieves 99.99%. Similarly for class 17 of Xiong’an hyperspectral dataset (Sparse Forest), LSTM accuracy drops to 25.00% and 3D-CNN to 93.33%, whereas SIFANet maintains 99.32% accuracy.

In summary, SIFANet not only sustains high accuracy across all categories but also exhibits minimal fluctuations and stable performance, conclusively demonstrating exceptional hyperspectral feature extraction and discrimination capabilities.

4. Discussion

4.1. Classification Performance and Accuracy Consistency Verification

Some models, although performing well in quantitative metrics such as OA, still have significant discrepancies between these metrics and the actual classification results. This article analyses the classification result maps of all models and selected typical regions with multi-category boundaries for locally enlarged comparisons, which are shown in Figure 14 and Figure 15.

SVM and LSTM, lacking spatial contextual modeling capabilities, display diffusely distributed misclassification. Severe boundary penetration occurs between spectrally similar categories like “Grapes_untrained” and “Vinyard_untrained”, accompanied by noticeable intra-class noise and boundary blurring.

Although 3D-CNN achieves high accuracy (OA > 99.05%), salt-and-pepper noise appears in the detailed views of both datasets, manifesting as single-pixel-scale categorical mutations. This primarily stems from excessive smoothing of local details in deep networks—an irreversible spatial resolution loss during hierarchical abstraction.

In contrast, SIFANet maintains continuous, clear category boundaries and smooth homogeneous regions in the detailed views. The results validate the effectiveness of the model’s structure design. The spatial–spectral dual-branch could extract complementary features, where residual blocks in the SFE branch effectively alleviate vanishing gradients in deep networks. Concurrently, the Conv-Former module enhances spectral sequence modeling capability. Ultimately, the CMAF module achieves the dynamic adaptive fusion of dual-branch features, minimizing information loss during feature transmission.

4.2. Confusing Classes Classification Performance Comparison

Based on the experimental results in Table 5 and Table 6, this article identifies land cover categories where multiple models perform poorly (i.e., confusing categories) and assesses them using the PR–F comprehensive evaluation model. The experimental results are shown in Figure 16 and Figure 17.

The PRCs reveal significant fluctuations in SVM’s accuracy at varying recall thresholds, characterized by pronounced sawtooth patterns. On the large-scale Xiong’an hyperspectral dataset, LSTM exhibits similar abrupt curve variations. Conversely, 3D-CNN and SIFANet display nearly overlapping trajectories on the Salinas dataset. However, as data complexity increases in Xiong’an, SIFANet maintains superior accuracy through its spatial–spectral fusion mechanism, while 3D-CNN shows significant deviation and accuracy degradation—confirming SIFANet’s efficacy in high-difficulty tasks.

F1-Score and AP values validate PRC observations. SIFANet consistently achieves optimal performance in imbalanced datasets, demonstrating robust results: F1-Score > 0.99 and saturated AP values across all Salinas categories, while maintaining > 0.92 F1-Score and >0.95 AP in Xiong’an dataset. Though 3D-CNN performs second best in most scenes, its F1-Score and AP decrease markedly (20.3% and 14.5%, respectively) when processing challenging features like “Sparse Forest” compared to other confusing categories. SVM and LSTM perform notably worse in the Xiong’an dataset with the F1-Score and AP dropping below 0.5 and even reaching zero for categories like “Black locust” and “Sparse Forest”.

It is worth noting that the Salinas dataset suffers from class imbalance, where several land cover categories contain only a limited number of training samples. Such imbalance often leads to classification bias toward majority classes, particularly for models that rely on single-branch feature representations. The proposed CMAF module helps mitigate this issue by performing dynamic cross-module feature fusion between the spatial-oriented SFE branch and the spectral sequence modeling SST branch. By jointly learning channel-wise attention weights from heterogeneous features, CMAF adaptively enhances discriminative cues that are critical for minority classes, which are often characterized by subtle spectral differences or limited spatial support. This dynamic fusion mechanism contributes to more stable precision–recall behavior and the consistently high F1-Score and AP values across confusing and low-sample categories in the Salinas dataset, effectively reducing the model’s bias toward majority classes.

4.3. Evaluation of Model Calculation Efficiency

Pursuing higher accuracy and stability under the same hardware conditions has always been an inevitable trend in model development.

Table 7. Efficiency parameters of each model.

Index		SVM	3D-CNN	LSTM	SIFANet
Parameters		160	58,992	56,080	269,873
FLOPs (×103)		0.12	2005.248	54.016	7979.392
OA	SL (%)	79.18	99.05	92.36	99.89
	XA (%)	59.11	99.42	80.19	99.79
S.D.	SL (%)	25.18	1.00	6.68	0.33
	XA (%)	37.09	1.95	18.54	0.58

summarizes the number of parameters and FLOPs of each model. Inputting the model efficiency parameter data from Table 7 into the CAPE value calculation formula yields the results shown in Figure 18.

SIFANet’s CAPE values of 49.93 (SL) and 27.91 (XA) are significantly higher than those of 3D-CNN (SL: 18.06/XA: 9.29), LSTM (SL: 2.91/XA: 0.91), and SVM (SL: 1.47/XA: 0.75). The results indicate that SIFANet excels in comprehensive performance across multiple dimensions, validating its adaptability advantage in complex hyperspectral data scenarios. Furthermore, the CAPE index provides a pragmatic benchmark for model selection in real-world deployment. In resource-constrained environments such as drones or edge devices, researchers can leverage CAPE to quantify the “accuracy gain per unit of computational cost”. This facilitates the identification of models that strike an optimal balance between inference speed and diagnostic precision.

4.4. Ablation Experiment

It is necessary to conduct ablation experiments to validate the effectiveness of the three core modules—SFE, SST, and CMAF. The results are shown in

Table 8. Ablation experiment results.

No.	SFE	SST	CMAF	SL				XA
				OA		Kappa		OA		Kappa
1	√	√	√	99.89		99.88		99.79		99.76
2	×	√	√	94.33	−5.56	93.12	−6.76	90.61	−9.18	88.23	−11.53
3	√	×	√	99.26	−0.63	99.33	−0.55	99.54	−0.25	99.41	−0.35
4	√	√	×	95.12	−4.77	94.59	−5.29	94.91	−4.88	94.04	−5.72
5	√	×	×	99.15	−0.74	99.05	−0.83	99.51	−0.28	99.41	−0.35
6	×	√	×	91.74	−8.15	90.01	−9.87	89.84	−9.95	88.26	−11.5

. Different colors in the table are used to distinguish accuracy comparison results: orange-marked values indicate relative accuracy lower than the SIFANet benchmark. “√” indicates that this structure is enabled in the experiment, while “×” indicates that it is disabled. Experiment 1 served as the baseline control group, incorporating all SFE, SST, and CMAF modules. The experimental design followed a progressive decoupling approach to analyze module interactions.

Validation of SFE/SST independent effectiveness used Experiment 1 (SFE + SST + CMAF) as the baseline. Performance comparisons with Experiment 2 (SST + CMAF only) and Experiment 3 (SFE + CMAF only) confirmed that adding either SFE or SST significantly enhanced classification accuracy. Notably, the proposed SFE module outperformed the conventional 3D-CNN due to its multi-scale convolution kernels and residual connections that effectively mitigate deep feature degradation.

Comparing Experiments 2/3 (“single module + CMAF”) against Experiments 5/6 (“single module only”) shows higher OA and Kappa coefficients for the former, demonstrating CMAF’s ability to extract valuable information from SFE/SST branches while reducing interference from ineffective features. Contrasting Experiment 1 (CMAF) with Experiment 4 (simply Add fusion) confirms that it could achieve complementary spatial–spectral information, improving OA by 4.77% (SL)/4.88% (XA) and Kappa by 5.29% (SL)/5.72% (XA). This superiority stems from CMAF’s attention-driven gating mechanism. Unlike simple element-wise addition, which treats all features with equal importance and may lead to “feature dilution” or noise propagation, the attention-based strategy dynamically re-weights the feature maps. It prioritizes discriminative spatial–spectral components while effectively suppressing redundant or non-informative features from individual branches. This adaptive fusion ensures a more precise alignment and integration of heterogeneous information.

In summary, each SIFANet module demonstrates scientifically sound design with significant synergistic effects and collective enhancing classification accuracy.

5. Conclusions

This article proposes a dual-branch collaborative network for HSI classification. The SIFANet is based on spectral integration and focused attention mechanisms. Specifically, the SFE and SST modules extract spatial and spectral features parallelly and efficiently. Subsequently, the proposed CMAF module dynamically fuses the feature information from both branches to enhance image detail representation and class discriminability. The proposed network significantly reduces feature loss and effectively relieves the insufficient feature extraction capability, which could learn the correlation of ground objects from multi-dimensional image feature spaces. As a result, SIFANet significantly outperforms the comparative models in accuracy assessment metrics such as CA, S.D., and CAPE.

However, the dual-branch architecture and attention mechanisms increase the computational complexity, which may limit its efficiency in real-time applications. Future work will focus on developing lightweight modules to reduce computational costs and exploring self-supervised learning to improve the model’s robustness in scenarios with limited labeled samples.