Spatial/Spectral-Frequency Adaptive Network for Hyperspectral Image Reconstruction in CASSI

Highlights

What are the main findings?

• We propose SSFAN, a novel dual-branch transformer network that integrates spatial and channel attention with frequency-domain learning (via a dedicated FDT module and loss) for high-fidelity HSI reconstruction in CASSI.
• The introduced channel compression and expansion modules effectively optimize the computational efficiency of the network, enabling superior detail preservation with moderate resource consumption.

What is the implication of the main findings?

• The integrated spatial/spectral-frequency learning framework achieves breakthrough reconstruction accuracy by simultaneously modeling multi-domain features, particularly in recovering fine textures and suppressing noise.
• This work provides a practical and efficient solution for high-quality hyperspectral imaging, balancing performance and complexity for real-world applications.

Abstract

Coded-Aperture Snapshot Spectral Imaging (CASSI) systems acquire 3D spatial–spectral information on dynamic targets by converting 3D hyperspectral images (HSIs) into 2D compressed measurements. Various end-to-end networks have been proposed for HSI reconstruction from these measurements. However, these methods have not explored the frequency-domain information of HSIs. This research presents the spatial/spectral-frequency adaptive network (SSFAN) for CASSI image reconstruction. A frequency-division transformation (FDT) decomposes HSIs into distinct Fourier frequency components, enabling multiscale feature extraction in the frequency domain. The proposed dual-branch architecture consists of a spatial–spectral module (SSM) to preserve spatial–spectral consistency and a frequency division module (FDM) to model inter-frequency dependencies. Channel compression/expansion modules are integrated into the FDM to balance computational efficiency and reconstruction quality. Frequency-division loss supervises feature learning across divided frequency channels. Ablation experiments validate the contributions of each network module. Furthermore, comparison experiments on synthetic and real CASSI datasets demonstrate that SSFAN outperforms state-of-the-art end-to-end methods in reconstruction performance.

1. Introduction

Hyperspectral images (HSIs) have been widely used in various fields such as remote sensing [1,2,3], medical diagnosis [4,5,6], military security [7,8,9], and tracking [10,11] because they have more spectral bands and richer spectral information than monochromatic or RGB images. Recently, snapshot hyperspectral imaging systems [12,13,14,15,16,17,18] have been developed to capture the 3D spatial–spectral data cubes of dynamic targets. Inspired by compressive sensing theory, coded-aperture snapshot spectral imaging (CASSI) systems [16,19,20,21,22] have been developed to achieve hyperspectral data cubes with high spatial resolution. However, reconstructing HSIs from 2D compressed data remains a critical challenge for CASSI systems.

It is evident that reconstructing 3D HSI data from 2D compressed measurements is a highly ill-posed inverse problem [12]. Reconstruction algorithms for CASSI systems are primarily divided into two categories: optimization algorithms and deep learning methods. Traditional optimization algorithms [20,23,24,25,26] leverage hand-crafted priors and rely on iterative algorithms to solve the problem of reconstruction. For example, Liu proposed the decompress snapshot compressive imaging (DeSCI) algorithm by constructing a joint model that integrates [24] the non-local self-similarity of video/hyperspectral frames and the rank minimization approach. Yuan proposed generalized alternating projection (GAP) to address the inverse problem in CASSI imaging by introducing a prior that minimizes the total variance [25]. Zhang et al. proposed dimension-discriminative low-rank tensor recovery (DLTR) to reconstruct HSIs [26]. Yin et al. uses the non-local low-rank tensor (NLRT) prior to design an optimization algorithm [20]. These traditional optimization methods have notable drawbacks, such as slow reconstruction speed and limited generalization capabilities. Algorithm parameters often require manual adjustment. Moreover, the algorithms often suffer from low accuracy when applied to noisy data.

The deep-learning-based methods for CASSI systems are categorized into three main types: plug-and-play (PnP), deep unfolding networks, and end-to-end (E2E) networks. The PnP algorithm [27,28,29,30,31] substitutes the prior module in the iterative process of optimization algorithm with a denoising network. Comparing with traditional optimization algorithms, PnP algorithms significantly improve reconstruction accuracy [32].

Deep unfolding networks [33,34,35,36] are designed on the basis of the iterative frameworks of optimization algorithms. They feature high interpretability and good reconstruction accuracy. Ma et al. proposed a deep tensor alternating direction method of multipliers network (ADMM-Net) [33] for video SCI systems. They achieved this by unfolding the inference iterations into a layerwise structure and designed a deep neural network based on tensor operations for the prior model. Huang proposed a deep unfolding method for the CASSI system based on the maximum a posteriori (MAP) estimation framework, which uses the learned Gaussian scale mixture (GSM) prior [34]. Cai et al. proposed a principled degradation-aware unfolding half-shuffle transformer (DAUHST) by combining the degradation-aware unfolding framework (DAUF) and the half-shuffle transformer (HST) [35]. Li et al. presented a pixel-adaptive deep unfolding transformer (PADUT) for HSI reconstruction in CASSI by incorporating a pixel-adaptive data module and a non-local spectral transformer prior module [36].

E2E networks [37,38,39,40,41,42,43] aim to establish a direct mapping from the compressed measurements of the CASSI system to reconstructed HSIs. These types of methods have exhibited good reconstruction performance. However, existing E2E networks predominantly exploit the spatial–spectral information of HSIs for reconstruction, without considering the spatial-frequency-domain information of the targets.

CASSI systems can be categorized into single-disperser CASSI (SD-CASSI) [16] and dual-disperser CASSI (DD-CASSI) [21,22]. In this study, taking SD-CASSI as an example, we propose a spatial/spectral-frequency adaptive network (SSFAN) for HSI reconstruction from compressive CASSI measurements. The frequency-division transformation (FDT) is proposed to process the frequency information of the feature maps, and a loss function is developed to help the network learn information from different frequency bands of HSIs. Additionally, both channel-dimension and spatial-dimension transformers are introduced to improve image reconstruction.

Our main contributions are as follows:

1. A novel E2E neural network framework, SSFAN, is proposed for the reconstruction of HSIs in a CASSI system. The framework features two branches—a transformer based on spatial attention and a transformer based on channel attention—designed to reconstruct the spatial–spectral and Fourier frequency components of HSIs across different bands.

2. The FDT is designed to obtain distinct Fourier frequency components and is accompanied by FDT loss to supervise the learning of information from different frequency channels.

3. To address the high computational load caused by the FDT, a channel compression module and a channel expansion module are introduced to balance the reconstruction performance and computer resource consumption of the neural network.

The article is organized as follows. Section 2 briefly reviews the existing E2E networks for CASSI and some works on frequency-domain processing. Section 3 describes the image formation model of SD-CASSI. Section 4 introduces a mathematical formulation of the proposed reconstruction method. Section 5 details the proposed two-branch E2E framework, including the network architecture (a spatial–spectral module and a frequency-division module) and the loss function. Section 6 shows the results of the ablation and comparison experiments. Finally, Section 7 presents the conclusion of this study.

2. Related Works

E2E networks are constructed based on three key aspects: the specific network structure, the utilization of system information, and the selection of loss functions. Regarding network structure, the basic structure of an E2E network includes linear layers, a convolution layer [44], and attention layers [45]. These fundamental components are combined to form the residual structure, the transformer structure [46], the UNet structure [47], etc. These diverse architectures are employed in various E2E networks available today [48,49,50,51,52]. Regarding the use of system information to extract features from inputs, many neural networks utilize various transforms, such as the Fourier transform [53], wavelet transform [54], and S-transform [55]. Regarding loss functions, different types are employed, including, among others, mean square error (MSE) [56] and learned perceptual image patch similarity (LPIPS) [57].

2.1. E2E Neural Networks for CASSI Reconstruction

Many E2E networks have been developed for the reconstruction of CASSI data. The $\lambda$-net [37] is a dual-stage generative model specifically designed to reconstruct the desired 3D data from SCI system. Meng et al. proposed spatial–spectral self-attention (TSA) [38] to process each dimension sequentially but in an order-independent manner. Hu et al. proposed a high-resolution dual-domain learning network (HDNet) integrating a residual network and spatial–spectral attention as its core architecture [39]. Cai et al. proposed the mask-guided spectral-wise transformer (MST) [40] by introducing spectral-wise multi-head self-attention (MSA) and a mask-guided mechanism (MM) to boost performance in HSI reconstruction. Cai et al. designed the coarse-to-fine sparse transformer (CST) [42]. This model employs a spectra-aware screening mechanism (SASM) and spectra-aggregation hashing multi-head self-attention (SAH-MSA) for tasks such as coarse patch selection, fine pixel clustering and self-similarity capturing. Luo et al. designed a dual-window multiscale transformer (DWMT) to capture both long-range dependencies and local details for HSI reconstruction in CASSI systems [43].

2.2. Frequency-Domain Processing

Frequency-domain processing has long been an important research area in signal processing. In recent years, with the extensive applications of deep learning across various fields, many methods [58,59,60,61,62,63,64,65,66] have emerged to integrate deep learning with frequency domain analysis. Xu [58] et al. found that deep neural networks often fit target functions from low to high frequencies during the training process, a concept referred to as the F-Principle. Based on these findings, frequency-domain processing techniques can help networks achieve superior performance. Kai et al. found that learning in the frequency domain with selected static channels can achieve higher accuracy than the conventional spatial downsampling approaches [59]. Wei et al. introduced an invariant wavelet transform with multiple parameters in the fractional domain for a watermarking algorithm [60]. Yang et al. utilized Fourier frequency features in super-resolution fields [61]. Jiang et al. proposed focal frequency loss [62], which enables neural networks to adaptively focus on frequency components in a process called frequency-domain learning. Karami et al. proposed a system that recognizes static gestures of alphabetic characters in Persian Sign Language (PSL) by combining a wavelet transform and neural networks [63]. Jamali et al. introduced the Haar wavelet transform in deep CNNs [64] to facilitate effective feature extraction, thereby enhancing the classification accuracy of polarimetric synthetic-aperture radar imagery. Meanwhile, HDNet employed frequency domain learning to improve reconstruction quality. Zhang et al. employed the fast Fourier transform (FFT) during subaperture decomposition to enhance feature extraction for SAR target recognition [65]. Wan et al. proposed a frequency-domain spectral feature module to perform FFT in order to extract multiscale and multi-frequency features, enhancing the discrimination of local features and global patterns in HSI classification [66]. Inspired by the above works, we propose a network module to learn information from the Fourier frequency components of HSIs.

3. Image Formation Model of SD-CASSI

As shown in Figure 1, the SD-CASSI system consists of an objective lens, a coding mask, a disperser, two relay lenses, and a monochromatic sensor. The input HSI cube is imaged by the objective lens and then spatially modulated by the encoding mask. Next, the coded HSI is spectrally sheared by the disperser. Finally, the spatially sheared and coded HSI is captured compressively by the sensor, creating a 2D compressed image.

For an SD-CASSI system, it is assumed that the input 3D HSI cube is ${\mathbf{X}}_0\in {\mathbb{R}}^{C\times H\times W}$, where H and W are the spatial dimensions and C is the spectral dimension. Furthermore, let the coded aperture be ${\mathbf{M}}_0\in {\mathbb{R}}^{H\times W}$. The modulated HSI cube ${\mathbf{X}}_1$ after passing through the coded aperture can be expressed as

$${\mathbf{X}}_1(\lambda ,h,w)={\mathbf{X}}_0(\lambda ,h,w)·{\mathbf{M}}_0(h,w)$$

(1)

where h and w are the spatial indices of the object plane corresponding to the H and W dimensions, respectively, while $\lambda$ is the spectral index for the C dimension. After passing through the relay lenses and the disperser element, the dispersed HSI cube ${\mathbf{X}}_2(\lambda ,m,n)$ is given by

$${\mathbf{X}}_2(\lambda ,m,n)={\mathbf{X}}_1(\lambda ,h,w+d(\lambda -{\lambda }_c))$$

(2)

where m and n are spatial indices of pixels on the sensor plane, d denotes a dispersive element with linear dispersion, and ${\lambda }_c$ denotes the center wavelength (note that this model assumes linear dispersion). The sensor captures ${\mathbf{X}}_2$ and generates a 2D measurement $\mathbf{Y}$, which is given by

$$\mathbf{Y}=\sum _{\lambda =1}^C{\mathbf{X}}_2(\lambda ,:,:)+\mathbf{G}$$

(3)

where $\mathbf{G}\in {\mathbb{R}}^{H\times (W+d(\lambda -{\lambda }_c))}$ is random noise generated during the imaging process.

The core task in reconstructing an HSI is to retrieve the original HSI cube from the measurement Y obtained by the SD-CASSI system.

4. Problem Formulation

As shown in Figure 2, the E2E model for spectral image reconstruction in the CASSI system is a mapping from the HSIs of targets to the system measurements via a neural network. This mapping can be expressed as

$${\mathbf{X}}_{\mathrm{pred}}=\mathcal{F}\left(\mathbf{Y}\right)$$

(4)

where ${\mathbf{X}}_{\mathrm{pred}}$ is the reconstructed HSI, $\mathbf{Y}$ is the measurements captured by the CASSI system, and $\mathcal{F}$ denotes the mapping relationship between them. The E2E networks commonly employed for CASSI reconstruction in recent years are mainly single-branch networks that learn spatial and spectral features.

Currently, these network methods rarely include the learning of spatial-frequency features for data reconstruction in the SD-CASSI system. To incorporate the frequency-domain feature, we propose a two-branch E2E network that extracts and fuses the holistic HSI information and frequency-band-specific details. Before presenting our reconstruction model, we first introduce our frequency-domain processing approach.

4.1. Frequency-Division Transformation

For the spectral images, although the image of each wavelength captures the same scene, the spatial profiles across different wavelengths exhibit both similarities and disparities. The similarities correspond to the low-frequency components of spatial frequency information, while the differences align with high-frequency components represent band-specific detail variations. Notably, existing neural network methods for HSI reconstruction predominantly utilize spatial information, with limited utilization of information in the spatial-frequency domain.

In the present study, to address this gap, we propose frequency-division transformation (FDT) to decompose spectral images into distinct spatial-frequency components, enabling the network to explicitly learn frequency-specific features. The procedure for FDT is shown in Figure 3.

The input, an arbitrary 3D data cube, is represented as $\mathbf{X}\in {\mathbb{R}}^{C\times H\times W}$. First, an FFT is performed on the input $\mathbf{X}$ to calculate the amplitude diagrams $\mathbf{A}\in {\mathbb{R}}^{C\times U\times V}$ and phase diagrams $\mathbf{P}\in {\mathbb{R}}^{C\times U\times V}$:

$$\mathbf{A},\mathbf{P}=\mathrm{FFT}(\mathbf{X})$$

(5)

where U and V are dimensions in the frequency domain, corresponding, respectively, to the spatial dimensions H and W.

Since the Fourier transform of a real-valued image exhibits symmetric property, a symmetric division of its spectrogram is principled and necessary. This approach inherently respects the physical structure of the frequency domain, ensuring that redundant symmetric component pairs are processed cohesively. Consequently, it enhances parameter efficiency and prevents the model from learning inconsistent representations from spectrally identical regions.

Taking the center of the spectrogram as the origin, the entire image domain is defined over $u\in [-{\displaystyle \frac{U}{2}},{\displaystyle \frac{U}{2}}]$, $v\in [-{\displaystyle \frac{V}{2}},{\displaystyle \frac{V}{2}}]$. Suppose that the spectrogram needs to be divided into P segments. By symmetrically dividing both axes, the frequency bands of the positive half-axis of U can be derived as follows: $(0,⌊{\displaystyle \frac{U}{2P}}⌋)$, $(⌊{\displaystyle \frac{U}{2P}}⌋,⌊{\displaystyle \frac{U}{P}}⌋)$, $\dots$, $(⌊{\displaystyle \frac{(P-1)U}{2P}}⌋,{\displaystyle \frac{U}{2}})$. Similarly, the frequency bands of the V-axis can be obtained. Combining these bands from both axes yields a segmented magnitude spectrum $[{\mathbf{a}}^1,{\mathbf{a}}^2,\dots ,{\mathbf{a}}^P]$. This process is defined by the following function:

$$[{\mathbf{a}}^1,{\mathbf{a}}^2,\dots ,{\mathbf{a}}^P]=\mathrm{Divide}\left(\mathbf{A}\right)$$

(6)

Subsequently, the image components at different frequencies are retrieved by applying the inverse fast Fourier transform (IFFT) to the amplitude spectrum components of the phase spectrum, which is given by

$${\mathbf{x}}^p=\mathrm{IFFT}({\mathbf{a}}^p,\mathbf{P})$$

(7)

A total of P component images are generated; they are denoted as $\{{\mathbf{x}}^p\mid p=1,2,\dots ,P\}$. Finally, these P images are concatenated along the channel dimension, resulting in the output data ${\mathbf{X}}_{FDT}\in {\mathbb{R}}^{CP\times H\times W}$. The pseudocode for the FDT operation described above is summarized in Algorithm 1.

Algorithm 1 Frequency-Division Transformation

Input: Data cube $\mathbf{X}\in {\mathbb{R}}^{C\times H\times W}$.  1: Calculate amplitude map $\mathbf{A}\in {\mathbb{R}}^{C\times U\times V}$ and phase map $\mathbf{P}\in {\mathbb{R}}^{C\times U\times V}$ $\mathbf{A},\mathbf{P}=\mathrm{FFT}2(\mathbf{X},dim=H,W)$  2: Divide amplitude map $\mathbf{A}$ into P sub-maps $[{\mathbf{a}}^1,{\mathbf{a}}^2,\dots ,{\mathbf{a}}^P]$  3: for p:1 to P do  4:    ${\mathbf{X}}^p=\mathrm{IFFT}({\mathbf{a}}^p,\mathbf{P})$  5: end for Output: Data cube after division ${\mathbf{X}}_{FDT}\in {\mathbb{R}}^{CP\times H\times W}$

4.2. Reconstruction Model

In the algorithms addressing the restoration problem of the SD-CASSI system, the measurement $\mathbf{Y}$ serves as input, while the prediction value ${\mathbf{X}}_{\mathrm{pred}}$ constitutes the output.

The framework of the reconstruction model is shown in Figure 2. With the feature extraction operation ${\mathcal{F}}_{\mathrm{input}}$, a feature map $\mathbf{Z}\in {\mathbb{R}}^{C\times H\times W}$ can be calculated as

$$\mathbf{Z}={\mathcal{F}}_{\mathrm{input}}\left(\mathbf{Y}\right)$$

(8)

Here, $\mathbf{Z}$ encapsulates all the spatial- and spectral-domain information.

The FDT operation is applied to decompose $\mathbf{Z}$ into spatial-frequency band components as follows:

$$\mathbf{F}=\mathrm{FDT}\left(\mathbf{Z}\right)=[{\mathbf{f}}^1,{\mathbf{f}}^2,\dots ,{\mathbf{f}}^P]$$

(9)

The spatial–spectral module (SSM), denoted as ${\mathcal{F}}_{\mathrm{SSM}}$, is designed to map $\mathbf{Z}$ onto the base spatial and spectral information of the input HSI cube ${\mathbf{X}}_0$.

$${\mathbf{S}}_{\mathrm{pred}}={\mathcal{F}}_{\mathrm{SSM}}\left(\mathbf{Z}\right)$$

(10)

Assume that the true values of different spatial-frequency bands of targets’ HSIs are represented by ${\mathbf{X}}_{\mathrm{FDT}}$. In order to achieve results close to the spatial-frequency components of ${\mathbf{X}}_{\mathrm{FDT}}$, the frequency-division module (FDM), denoted as ${\mathcal{F}}_{\mathrm{FDM}}$, is designed to map ${\mathbf{f}}^P$ to the predicted value ${\mathbf{f}}_{\mathrm{pred}}^p$:

$${\mathbf{f}}_{\mathrm{pred}}^p={\mathcal{F}}_{\mathrm{FDM}}\left({\mathbf{f}}^p\right)$$

(11)

As a result, the ${\mathcal{F}}_{\mathrm{FDM}}$ process generates the predicted ${\mathbf{F}}_{\mathrm{pred}}=[{\mathbf{f}}_{\mathrm{pred}}^1,{\mathbf{f}}_{\mathrm{pred}}^2,\dots ,{\mathbf{f}}_{\mathrm{pred}}^P]$. Finally, the two pieces of information, ${\mathbf{S}}_{\mathrm{pred}}$ and ${\mathbf{F}}_{\mathrm{pred}}$, are fused to obtain the predicted HSI value ${\mathbf{X}}_{\mathrm{pred}}$.

$${\mathbf{X}}_{\mathrm{pred}}={\mathcal{F}}_{\mathrm{fusion}}({\mathbf{S}}_{\mathrm{pred}},{\mathbf{F}}_{\mathrm{pred}})$$

(12)

5. Proposed Method

In this paper, we propose an E2E network for the reconstruction of CASSI. This method restores HSIs from compressed measurements by extracting spatial–spectral and frequency features from different spatial frequency bands. It is named the spatial/spectral-frequency adaptive network (SSFAN).

5.1. Network Framework

The framework of SSFAN is illustrated in Figure 2. The initial part of the network consists of a convolution layer followed by several residual blocks (RBs), which correspond to the function ${\mathcal{F}}_{\mathrm{input}}$ in the reconstruction model. Through this initial processing, the feature map $\mathbf{Z}$ is obtained. The subsequent section splits into two branches. The right branch is designed to learn the spatial–spectral features of HSI, corresponding to the role of the function ${\mathcal{F}}_{\mathrm{SSM}}$ in the reconstruction model. The left branch, FDM, aims to learn the features of different spatial frequency components of HSIs and corresponds to the function of the expression ${\mathcal{F}}_{\mathrm{FDM}}$ in the reconstruction model. When $\mathbf{Z}$ is fed into these two branches, they learn the features of HSIs, resulting in the outputs ${\mathbf{S}}_{\mathrm{pred}}$ and $[{\mathbf{f}}_{\mathrm{pred}}^1,{\mathbf{f}}_{\mathrm{pred}}^2,\dots ,{\mathbf{f}}_{\mathrm{pred}}^P]$, respectively. They are fed into the final part of the network to generate the final reconstructed HSIs. This part consists of several residual blocks that correspond to the ${\mathcal{F}}_{\mathrm{fusion}}$ function.

5.2. Spatial–Spectral Module (SSM)

As shown in Figure 4a, the SSM is designed to implement the ${\mathcal{F}}_{\mathrm{SSM}}$ function. It takes the feature map $\mathbf{Z}$ as input and generates ${\mathbf{S}}_{\mathrm{pred}}$ as output. The number of input channels, $\mathbf{Z}$, increases through a convolution layer. Then, the cube is fed into a U-shaped network constructed with a non-local spatial attention block (NSAB). As illustrated in Figure 5a, the NSAB comprises two layer normalization (LN) layers, a non-local spatial attention (NSA) module, and a feedforward neural network (FNN). The FFN, shown in Figure 5c, consists of two convolution (CONV) layers, a Gaussian error linear unit (GELU), and a depthwise convolution (DWConv).

The architecture of the NSA module is shown in Figure 5a. First, the input ${\mathbf{X}}_{\mathrm{in}}\in {\mathbb{R}}^{C\times H\times W}$ is shifted with $L/2$. Then, the shifted data are divided into several data cubes $[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_n]$, where the size of each cube ${\mathbf{x}}_i$ is $C\times L\times L$$(i=1,2,\dots ,n)$. For each cube ${\mathbf{x}}_i$, three group networks are utilized to transform it into query vectors ${\mathbf{Q}}_i$, key vectors ${\mathbf{K}}_i$, and value vectors ${\mathbf{V}}_i$. These vectors have a size of $h\times LL\times C$, where h represents the number of heads in the multi-head attention mechanisms.

$${\mathbf{Q}}_i={\mathbf{x}}_i{\mathbf{W}}^Q$$

(13)

$${\mathbf{K}}_i={\mathbf{x}}_i{\mathbf{W}}^K$$

(14)

$${\mathbf{V}}_i={\mathbf{x}}_i{\mathbf{W}}^V$$

(15)

Then, the attention output is given by

$$\mathrm{Attention}({\mathbf{Q}}_i,{\mathbf{K}}_i,{\mathbf{V}}_i)={\mathbf{V}}_i\mathrm{Softmax}\left({\displaystyle \frac{{\mathbf{K}}_i^T{\mathbf{Q}}_i}{\beta }}\right)$$

(16)

The attention map has a size of $h\times LL\times LL$. This map can guide the network to learn features of a specific position.

5.3. Frequency-Division Module (FDM)

As shown in Figure 4b, the FDM is designed to implement the function ${\mathcal{F}}_{FDM}$. As shown in Figure 2, the feature map $\mathbf{Z}$ is taken as input, with ${\mathbf{F}}_{\mathrm{pred}}$ generated as output. The FDT-P block transforms the input into spatial-frequency map $\mathbf{F}={\mathbf{f}}^1,{\mathbf{f}}^2,\dots ,{\mathbf{f}}^P$. The FDM includes multiple non-local channel attention blocks (NCABs). The structure of NCABs is detailed in Figure 5b.

The data volume of this feature map is extremely large. The input feature map size is $C\times H\times W$ and the number of parts established by spatial-frequency division is P. Accordingly, the size of ${\mathbf{F}}_{\mathrm{pred}}$ is $CP\times H\times W$. A comparative analysis of chosen values for P is presented in Section 6.1. Assuming $C=28$ and $P=5$, there will be 140 channels to be processed. Obviously, it is difficult to train a network with limited memory to process data with 140 channels using the network. On the other hand, the features extracted from different spatial frequency divisions do not have the same significant impact on the final reconstruction results.

In order to reduce the computation cost, we can reasonably compress the channel dimension using the channel compress module (CCM). A smaller feature map ${\mathbf{X}}_{FDC}\in {\mathbb{R}}^{C^{\prime }\times H\times W}$$(C^{\prime }<CP)$ is obtained. This feature map is processed by the network to retrieve frequency information. Its channel is then expanded back to $CP$ by the channel extension module (CEM).

The CCM consists of three convolution layers and three RELU activation layers. The first convolution layer, with both the input and output channels set to $CP$, is designed to extract features from the output of FDM. The second convolution layer, with input channels of size $CP$ and output channels of size $C^{\prime }$, is designed to compress the channel information. The third convolution layer, featuring $C^{\prime }$ input and output channels, organizes compressed features for subsequent modules. The architecture is designed to approximate a sufficiently complex function to retain the most efficient information from frequency division while minimizing reconstruction loss. The process can be expressed as the following equation:

$$\mathrm{CCM}\left(x\right)=\mathrm{RELU}\left({\mathrm{Conv}}_{C^{\prime }}^{C^{\prime }}\left(\mathrm{RELU}\left({\mathrm{Conv}}_{CP}^{C^{\prime }}\left(\mathrm{RELU}\left({\mathrm{Conv}}_{CP}^{CP}\left(x\right)\right)\right)\right)\right)\right)$$

(17)

The CEM also consists of three convolution layers and three RELU activation layers, with the input and output channel configurations of the convolution layers being exactly the opposite of those in the CCM. Therefore, the process can be expressed by the following equation:

$$\mathrm{CEM}\left(x\right)=\mathrm{RELU}\left({\mathrm{Conv}}_{CP}^{CP}\left(\mathrm{RELU}\left({\mathrm{Conv}}_{C^{\prime }}^{CP}\left(\mathrm{RELU}\left({\mathrm{Conv}}_{C^{\prime }}^{C^{\prime }}\left(x\right)\right)\right)\right)\right)\right)$$

(18)

The NCAB serves as the core learning structure in the FDM. Analogous to the NSAB, it is composed of two LN layers, a feedforward neural network, and a non-local channel attention (NCA) module. There are two reasons for adopting NCA over NSA. First, the FDT concatenates the HSI data of different frequencies along the channel dimension, so that the information of different frequency components is stored in the channel dimension. Second, the spectral data of the same pixel in a hyperspectral image are also encoded within the channel dimension.

As with the NSA module, cubes are generated by shifting and splitting the input data. For each cube, ${\mathbf{Q}}_i$, ${\mathbf{K}}_i$, and ${\mathbf{V}}_i$ are derived via Equations (13)–(15). The attention output is computed using Equation (16). In NCA, unlike NSA, these vectors have a size of $h\times C\times LL$, and the attention map has a size of $h\times C\times C$. The feature map is finally restored to the input image size by fusion operations and a series of residual blocks.

5.4. Loss Function

To ensure that the neural network output aligns closely with the ground truth (GT), we adjust parameters not only by comparing the final output to the GT in the spatial domain but also by comparing the local outputs of a specific module to the GT in the frequency domain. Therefore, the loss function of the network consists of two main components: the base loss and the frequency-division loss.

The mean squared error (MSE) loss is a standard choice for regression tasks, effectively ensuring pixelwise spectral fidelity and promoting the smoothness of reconstructed features. Therefore, we accordingly adopt the $l_2$ loss function (MSE) to calculate each loss term, as expressed in the following equation:

$$Los{s}_{\mathrm{MSE}}={\displaystyle \frac{1}{N}}\sum _{j=1}^N{(x_j-{\widehat{x}}_j)}^2$$

(19)

The basic loss is given by the following equation:

$$Los{s}_{\mathrm{BASE}}=\mathrm{MSE}({\mathbf{X}}_{\mathrm{gt}},{\mathbf{X}}_{\mathrm{pred}})$$

(20)

where ${\mathbf{X}}_{\mathrm{gt}}$ is the GT image and ${\mathbf{X}}_{\mathrm{pred}}$ is the final output of the network.

As shown in Figure 6, the frequency-division loss is calculated using the FDM of the predicted output ($\mathbf{F}{\mathbf{D}}_{\mathrm{pred}}={\mathcal{F}}_{\mathrm{FDM}}\left({\mathbf{X}}_{\mathrm{pred}}\right)$) and the GT frequency component ($\mathbf{F}{\mathbf{D}}_{\mathrm{gt}}={\mathcal{F}}_{\mathrm{FDM}}\left({\mathbf{X}}_{\mathrm{gt}}\right)$), which is given by the following equation:

$$Los{s}_{\mathrm{FDT}}=\mathrm{MSE}(\mathbf{F}{\mathbf{D}}_{\mathrm{pred}},\mathbf{F}{\mathbf{D}}_{\mathrm{gt}})$$

(21)

The final loss function is expressed as follows:

$$Los{s}_{\mathrm{ALL}}=Los{s}_{\mathrm{FDT}}+\eta Los{s}_{\mathrm{BASE}}$$

(22)

where $\eta$ is a weighting coefficient that balances the two loss terms.

6. Experimental Results

In this section, ablation experiments are conducted on the proposed network, and comparison experiments are conducted with multiple algorithms. The proposed method is compared with several E2E networks, $\lambda$-net [37], TSA-Net [38], HDNet [39], MST-L [40], MST++ [41], CST-L [42], and DWMT [43]. The experiments are performed and evaluated on both synthetic datasets and real datasets.

The peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapper (SAM) are employed to evaluate the reconstruction performance of different methods. For the spatial evaluation indicators PSNR and SSIM, we calculate them for each 2D spectral image and then compute the average values across all spectral bands as the final results. For the spectral indicator SAM, the values of each spatial point are calculated and then averaged to generate the final value.

6.1. Experiment Setup

In the simulation experiments, two datasets, namely, CAVE [67] and KAIST [68], are adopted to generate compressed 2D measurements of the SD system based on the model presented in Section 3. Meanwhile, ten scenes from the KAIST dataset are used to generate a test set. In the real experiment, HSIs of five real scenes captured by an SD-CASSI system [38] are used for comparison. The real HSIs consist of 28 spectral channels ranging from 450 nm to 650 nm, with each channel having two-pixel dispersion.

The proposed SSFAN is implemented using PyTorch 2.7.1. The network models for quantitative comparisons and ablation studies are trained with the Adam optimizer [69] and cosine annealing scheme [70] for 500 epochs on an NVIDIA GeForce RTX 4090 (NVIDIA Corporation; Santa Clara, CA, USA). The training uses a batch size of five and input patches of size 256 × 256. The loss weight $\eta$ is set to one. The initial learning rate during training is $2\times {10}^{-4}$.

6.2. Ablation Study

In this section, we conduct several ablation experiments to investigate the hyperparameters of models and the effectiveness of different components in models. The baseline of the model is a small version in which only the number of network layers is reduced.

First, the optimal number of frequency bands in the FDT for the reconstruction results is investigated.

Table 1. Ablation study of the number of frequency bands in the FDT on the simulation dataset; the average PSNR, SSIM and SAM are listed. The optimal results are highlighted in bold within the table.

Number of Frequency Bands	PSNR	SSIM	SAM
3	36.70	0.964	0.097
4	36.82	0.964	0.093
5	37.02	0.966	0.091
6	36.93	0.965	0.092
7	36.98	0.965	0.094

shows the average PSNR, SSIM and SAM values of ten test scenes in the test dataset. Five different numbers of divided bands are tested in the FDT. The best performance is achieved when the frequency band number is set to five.

The optimal channel numbers in the CCM and CEM are then explored. In this experiment, the channel numbers in the CCM and CEM are set to 28, 40, 56, and 64. According to

Table 2. Ablation study of the number of compressed channels in the CCM and CEM on the simulation dataset; the average PSNR, SSIM, SAM, speed (GFLOPs) and parameter count (Params) are listed. The optimal results are highlighted in bold within the table.

Number of Compressed Channels	PSNR	SSIM	SAM	GFLOPs	Params
28	36.73	0.964	0.097	89.46	4.24 M
40	36.91	0.964	0.095	99.29	4.97 M
56	37.02	0.965	0.091	117.28	6.34 M
64	36.39	0.963	0.100	128.44	7.19 M

, the model performance does not improve monotonically with increasing channel number. The best results are obtained when the channel number is set to 56. The ablation study in

Table 3. Ablation study of the CCM and CEM.

Model	GFLOPs	Params
w/o CCE and CEM	304.08	20.82 M
w/ CCE and CEM	117.28	6.34 M

clearly highlights the compression capability of the CCM and CEM. The proposed model requires only 117.28 GFLOPs and 6.34 M parameters, which is a decrease of about 61.4% and 69.5%, respectively, from the baseline model that operates without them.

To verify the effectiveness of the SSM and FDM, module ablation experiments are conducted. The experiments include three scenarios: one using only the SSM, one using only the FDM, and one using both modules.

Table 4. Ablation study for sub-modules of the SFDN on the simulation dataset; the average PSNR, SSIM and SAM are listed. The optimal results are highlighted in bold within the table.

SSM	FDM	PSNR	SSIM	SAM
✓		36.91	0.963	0.100
	✓	36.77	0.965	0.976
✓	✓	37.02	0.965	0.091

summarizes the results. When only the SSM is used, higher PSNR and SAM values but lower SSIM are obtained than when only the FDM is used. When both modules are used, the best overall performance is achieved. These results indicate that the SSM can provide some spatial information and reduce the noise in HSIs, while the FDM can contribute more structural information and spectral information from the image.

The impact of the FDT was also investigated. In order to isolate the effects of the number of channels on the FDT, a comparison is made between the FDT and a convolution layer that can achieve the same channel-number expansion. The results are presented in

Table 5. Ablation study of the FDT and convolution layer on the simulation dataset; the average PSNR, SSIM, and SAM are listed. The optimal results are highlighted in bold within the table.

Operation	PSNR	SSIM	SAM
FDT	37.02	0.965	0.091
Convoluton Layer	36.69	0.965	0.095

. The network employing the FDT achieves a 0.33 dB improvement over the network using the convolution layer. This finding confirms that the FDT can capture more information about HSIs than the conventional option of increasing the number of channels.

6.3. Performance Evaluation on a Simulation Dataset

Table 6. Comparison between the proposed network and open-source E2E methods on ten test scenes (S1–S10). The PSNR, SSIM, SAM, speed (FLOPs), and parameter count (Params) are listed. The optimal results are highlighted in bold within the table.

Methods	Params	GFLOPs	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	Avg
			30.10	28.46	27.73	37.01	26.19	28.64	26.47	26.09	27.50	27.13	28.53
$\lambda$-net	62.64 M	117.98	0.849	0.805	0.870	0.934	0.817	0.853	0.806	0.831	0.826	0.816	0.841
			0.247	0.304	0.272	0.420	0.285	0.454	0.246	0.481	0.277	0.454	0.344
			32.03	31.00	32.25	39.19	29.39	31.44	20.32	29.35	30.01	29.59	31.46
TSA-Net	44.25 M	110.06	0.892	0.858	0.915	0.953	0.884	0.908	0.878	0.888	0.890	0.874	0.894
			0.152	0.181	0.129	0.146	0.117	0.169	0.133	0.199	0.134	0.167	0.153
			35.14	35.67	36.03	42.30	32.69	34.46	33.67	32.48	34.89	32.38	34.97
HDNet	2.37 M	154.76	0.935	0.940	0.943	0.969	0.946	0.952	0.926	0.941	0.942	0.937	0.943
			0.129	0.142	0.097	0.102	0.089	0.120	0.114	0.142	0.111	0.119	0.117
			35.40	35.87	36.51	42.27	32.77	34.80	35.87	32.67	35.39	32.50	35.18
MST-L	2.03 M	28.15	0.941	0.944	0.953	0.973	0.947	0.955	0.925	0.948	0.949	0.941	0.957
			0.123	0.142	0.106	0.130	0.101	0.137	0.110	0.180	0.130	0.146	0.131
			35.80	36.23	37.34	42.63	33.38	35.38	34.35	33.71	36.67	33.38	35.99
MST++	1.33 M	19.42	0.943	0.947	0.957	0.973	0.952	0.957	0.934	0.953	0.953	0.945	0.951
			0.118	0.130	0.080	0.121	0.081	0.115	0.103	0.145	0.100	0.116	0.111
			35.96	36.84	38.16	42.44	33.25	35.72	36.58	34.34	36.51	33.09	36.28
CST-L	3.00 M	27.81	0.949	0.955	0.962	0.975	0.955	0.963	0.944	0.961	0.957	0.945	0.956
			0.116	0.118	0.082	0.099	0.078	0.104	0.099	0.119	0.101	0.102	0.102
			36.54	37.85	38.59	44.66	34.05	36.24	35.29	34.62	37.52	34.05	36.94
DWMT	14.48 M	46.71	0.957	0.963	0.964	0.984	0.962	0.970	0.948	0.967	0.965	0.958	0.964
			0.109	0.114	0.073	0.097	0.071	0.103	0.092	0.119	0.088	0.098	0.096
			37.41	38.85	38.19	43.84	34.85	37.07	36.14	34.80	37.42	33.82	37.24
SSFAN	6.34 M	117.28	0.965	0.971	0.965	0.985	0.969	0.973	0.955	0.969	0.965	0.959	0.968
			0.101	0.103	0.073	0.077	0.064	0.083	0.086	0.097	0.086	0.089	0.086

summarizes PSNR, SSIM, SAM of eight different methods across ten different scenes on a simulation dataset. Compared with $\lambda$-Net, TSA-Net, HDNet, MST-L, MST++, CST-L, and DWMT, the proposed SSFAN demonstrates superior reconstruction performance. Specifically, SSFAN can achieve an average PSNR improvement of 0.96 dB over CST-L. In terms of spectral accuracy, SSFAN reduces the average SAM by 0.016 rad compared to CST-L.

Visualizations of reconstruction results for eight methods in scenes 5 and 7 are illustrated in Figure 7 and Figure 8. For the 28 spectral bands, the 481.5 nm, 544.0 nm, 575.5 nm, and 636.5 nm bands of HSIs are selected, and these wavelength data are converted into images. Additionally, each image includes zoomed-in patches of the regions of interest (ROIs) within white rectangles. Notably, SSFAN reconstructs HSIs with more accurate image structural information and finer details.

The spectral curves and the correlation coefficients of the spectral curves for selected points of scenes 2 and 3 are illustrated in Figure 9. The correlation coefficient corr is given by

$$\mathrm{corr}(\mathbf{X},\mathbf{Y})={\displaystyle \frac{\mathrm{cov}(\mathbf{X},\mathbf{Y})}{{\sigma }_{\mathbf{X}}{\sigma }_{\mathbf{Y}}}}$$

(23)

Here, $\mathrm{cov}$ denotes covariance, and ${\sigma }_{\mathbf{X}}$ and ${\sigma }_{\mathbf{Y}}$ denote the variance of $\mathbf{X}$ and $\mathbf{Y}$. By comparing the reconstructed spectral signatures with those of other methods, we demonstrate that the proposed SSFAN can achieve higher spectral accuracy than other methods.

In terms of visualization comparison, as shown in Figure 8, the textures in the wings of the bird are reconstructed by SSFAN. On the other hand, the structural information of the same region is not clear in the results of other methods. This is attributed to the proposed FDT module that provides information on different spatial frequency bands to the SSFAN network, thereby enhancing the reconstruction ability for structural information of details in images.

6.4. Performance Evaluation with Real CASSI Data

The effectiveness of SSFAN in reconstructing HSIs from real CASSI measurements is also evaluated. Following the same experimental setup as previous studies on HSI reconstruction of real CASSI images [34,38,40,42], SSFAN is trained using both the CAVE and KAIST datasets with the real mask. To investigate the anti-noise ability of the proposed network, 11-bit shot noise is added during the training process.

Figure 10 provides the visualization of four spectral bands of the HSIs reconstructed by different methods. Compared with other E2E methods, the spectral images reconstructed by SSFAN exhibit more spatial details. In the top two rows, corresponding to the short-wave bands, the MST and CST methods produce noisy artifacts on the left of the foreground target, whereas the results of SSFAN show no apparent noise. This indicates that our SSFAN is more effective in CASSI reconstruction. Furthermore, the spectral curves of the green branch (point A) and the red strawberry (point B) in the reconstructed HSIs are illustrated in Figure 11b,d for comparison. The spectral curves obtained by SSFAN match the colors of the selected points better than those generated by other methods.

7. Conclusions

In this paper, we propose a novel E2E hyperspectral image reconstruction framework for CASSI. The network, named SSFAN, integrates frequency-domain processing and cross-dimensional feature learning. The FDT is proposed to decompose HSIs into distinct Fourier frequency components, enabling the network to capture multiscale spatial-frequency dependencies. Leveraging the FDT, the network is designed to reconstruct HSIs by fusing spatial–spectral information and specific frequency features. Specifically, SSM employs spatial transformers to extract local spatial structures and preserve spectral consistency, while FDM utilizes channelwise attention to model the inter-correlations of specific Fourier frequency components. To balance computational efficiency and reconstruction quality, channel compression and expansion modules are incorporated to optimize feature dimensionality during frequency processing. Ablation experiments validate the contribution of specific modules and parameters in reconstruction performance. Extensive experiments on both simulated and real SD-CASSI datasets demonstrate that SSFAN outperforms state-of-the-art E2E methods. Visual comparisons reveal that SSFAN reconstructs fine textures (e.g., bird wings) and suppresses noise more effectively than existing methods, confirming its superiority in preserving spatial detail. Future research will explore the perceptual and structural loss functions to further improve the visual quality of the reconstructed HSIs.