Pixel-Based Long-Wave Infrared Spectral Image Reconstruction Using a Hierarchical Spectral Transformer

Abstract

Long-wave infrared (LWIR) spectral imaging plays a critical role in various applications such as gas monitoring, mineral exploration, and fire detection. Recent advancements in computational spectral imaging, powered by advanced algorithms, have enabled the acquisition of high-quality spectral images in real time, such as with the Uncooled Snapshot Infrared Spectrometer (USIRS). However, the USIRS system faces challenges, particularly a low spectral resolution and large amount of data noise, which can degrade the image quality. Deep learning has emerged as a promising solution to these challenges, as it is particularly effective at handling noisy data and has demonstrated significant success in hyperspectral imaging tasks. Nevertheless, the application of deep learning in LWIR imaging is hindered by the severe scarcity of long-wave hyperspectral image data, which limits the training of robust models. Moreover, existing networks that rely on convolutional layers or attention mechanisms struggle to effectively capture both local and global spectral correlations. To address these limitations, we propose the pixel-based Hierarchical Spectral Transformer (HST), a novel deep learning architecture that learns from publicly available single-pixel long-wave infrared spectral databases. The HST is designed to achieve a high spectral resolution for LWIR spectral image reconstruction, enhancing both the local and global contextual understanding of the spectral data. We evaluated the performance of the proposed method on both simulated and real-world LWIR data, demonstrating the robustness and effectiveness of the HST in improving the spectral resolution and mitigating noise, even with limited data.

1. Introduction

Long-wave infrared (LWIR) or thermal infrared refers to wavelengths ranging between 7 and 14 $\mathsf{\mu }$m. A range of crucial materials, including minerals and gases, have distinctive spectral characteristics within this LWIR range, enabling their identification [1]. Consequently, LWIR spectral imaging plays a crucial role in various applications such as environmental monitoring, climate studies, gas monitoring, mineral exploration, and fire detection [2,3].

In recent years, the computational spectral imaging technique, powered by advanced algorithms, has enabled the acquisition of high-quality spectral images in real time [4]. A prime example is the recently proposed Uncooled Snapshot Infrared Spectrometer (USIRS), which has enabled real-time video imaging in the LWIR range [5,6]. This is achieved by replicating the target scene using optical multi-aperture imaging and assigning filters of different wavelengths for each scene [7], allowing for real-time LWIR spectral video imaging at 8Hz. However, the USIRS presents two significant challenges. Firstly, the spectral resolution is achieved via the band-pass spectral filtering of the multi-aperture scene, creating a trade-off between the spectral and spatial resolution. To preserve an adequate spatial resolution, the spectral resolution is kept relatively low, limiting the USIRS to only 18 spectral channels. Secondly, the uncooled LWIR system, unlike commercial visible cameras that measure reflected sunlight, measures the target scene’s inherent radiation, typically low-intensity blackbody radiation at room temperature. This radiation is highly sensitive to background noise, resulting in a low signal-to-noise ratio (SNR) [8,9]. This further degrades the quality of spectrum acquisition and imaging performance. Therefore, a specifically designed computational algorithm is required to address these challenges.

Over the past decade, deep learning has demonstrated extraordinary robustness to noise [10,11] and high accuracy in hyperspectral image reconstruction and spectroscopy reconstruction [12], making it a promising solution to the challenges faced by USIRS. This method successfully integrates both spatial and spectral information to reduce noise and preserve details in hyperspectral images, leading to significant improvements in image quality [13,14,15,16]. Liu et al. advanced hyperspectral image denoising using a 3D attention denoising network [17]. By incorporating a three-dimensional attention mechanism, they managed to effectively capture intricate patterns and relationships within the hyperspectral data, resulting in superior denoising performance. Zhang et al. enhanced hyperspectral image denoising using a multitask learning approach combined with sparse representation techniques [18].

On the other hand, the field of spectroscopy reconstruction has introduced many pixel-based spectrum reconstruction methods, which mitigate the need for a large number of spectral images to train a neural network [19,20,21,22]. For instance, Wen et al. utilized the Multilayer Perceptron (MLP) to efficiently reconstruct the absolute spectra of various nonluminous samples, thereby developing a miniaturized spectrometer [23]. J. Zhang et al. suggested feeding the initial results of optimization-based methods into the MLP instead of directly feeding the raw measurements [24]. This reconstruction method has shown great potential for spectral recovery of filter-based miniature spectrometers. However, in the task of LWIR spectral reconstruction, existing neural network structures struggle to handle the noise and unique characteristics of the LWIR spectrum. LWIR data can be noisy, requiring neural networks to be robust against such noise to avoid inaccurate reconstructions. Traditional Convolutional Neural Networks (CNNs) [25,26,27,28] often struggle with this task due to their limited receptive fields and inefficient modeling of long-range dependencies [29]. Furthermore, the reconstruction problem is inherently ill posed, indicating that multiple possible solutions often exist for a given set of measurements. This complexity makes it challenging for MLPs [30,31,32] to learn meaningful representations without significant computational overhead and a risk of overfitting [33]. Therefore, there is a demand for bespoke network architectures for LWIR spectral reconstruction. These should be capable of capturing the complex spectral correlations and dependencies that are inherent to the LWIR spectrum.

To address these challenges, we propose the pixel-based Hierarchical Spectral Transformer (HST). First, instead of directly training on LWIR spectral images, we train the pixel-based neural network on public LWIR spectral databases [34]. This strategy reduces the need for high-quality paired data for neural network training. Second, we incorporate an LWIR spectral noise model into the USIRS data simulation pipeline. The noise model serves as a bridge between synthetic data and experimental data, enabling our network, trained on synthetic data, to effectively handle experimental data that are collected by the USIRS. Third, the HST employs a multistage architecture, where each stage progressively refines the spectral features, enhancing the detail and accuracy of the reconstruction. By employing spectral-wise multi-head self-attention [35], the HST can concentrate on the most relevant spectral features, thereby reducing the computational complexity and enhancing the reconstruction quality. This hierarchical approach ensures that both global and local spectral information effectively mitigate the large amount of noise and achieve superior performance in LWIR spectral image reconstruction tasks. Furthermore, while the HST is specifically designed and evaluated for the USIRS system, its versatility allows for it to be applied to other computational spectral imaging systems.

Our contributions are summarized as follows:

• We propose training the neural network on pixel-based LWIR spectra and integrating an LWIR spectral noise model into the USIRS data simulation pipeline. This approach reduces the need for high-quality paired data for training on LWIR spectral images.
• We introduce the Hierarchical Spectral Transformer (HST), designed to effectively learn and preserve both global and local spectral information, thus mitigating the large amount of noise and enhancing the reconstruction accuracy.
• We evaluate our pipeline using both synthetic and experimental data to demonstrate its effectiveness in handling real-world scenarios.

2. Materials and Methods

2.1. The USIRS

The Uncooled Snapshot Infrared Spectrometer (USIRS) is an industrial instrument developed by the Shanghai Institute of Technical Physics of the Chinese Academy of Sciences, as shown in Figure 1a.

The USIRS uses filters and multi-aperture imaging to capture images at various wavelengths, providing a detailed spectral image of the target object. Figure 1b illustrates the optical design of the USIRS. The process begins with a telescopic lens collecting light from the target. This light is then directed to a collimating lens, ensuring that the rays are parallel. Following this, a lens array composed of nine smaller lenses recreates the target scene. The light then travels through a filter array, in which each filter corresponds to a specific waveband, before it finally reaches the focal plane, where images of different wavelengths are created and captured.

The USIRS utilizes two identical optical systems, with the only difference being the wavelength of the filter array. This effectively makes it a dual system with 17 band-pass wavelengths: 7.22 $\mathsf{\mu }\mathrm{m}$, 7.67 $\mathsf{\mu }\mathrm{m}$, 7.96 $\mathsf{\mu }\mathrm{m}$, 8.33 $\mathsf{\mu }\mathrm{m}$, 8.70 $\mathsf{\mu }\mathrm{m}$, 9.07 $\mathsf{\mu }\mathrm{m}$, 9.48 $\mathsf{\mu }\mathrm{m}$, 9.81 $\mathsf{\mu }\mathrm{m}$, 10.18 $\mathsf{\mu }\mathrm{m}$, 10.55 $\mathsf{\mu }\mathrm{m}$, 10.92 $\mathsf{\mu }\mathrm{m}$, 11.29 $\mathsf{\mu }\mathrm{m}$, 11.66 $\mathsf{\mu }\mathrm{m}$, 12.03 $\mathsf{\mu }\mathrm{m}$, 12.40 $\mathsf{\mu }\mathrm{m}$, 12.77 $\mathsf{\mu }\mathrm{m}$, and 13.14 $\mathsf{\mu }\mathrm{m}$.

Figure 1c displays one of the USIRS images for a target scenario involving two individuals and two gas bottles releasing NH3 and SF6, respectively. The spectral curves of the gases marked by the blue box are depicted in Figure 1d. The USIRS does not have a high spectral resolution, given that it only has 17 band-pass spectral channels and the measurement tends to be quite noisy. For further applications, it is crucial to denoise and reconstruct the high-resolution spectrum.

2.2. Imaging and Noise Model

Deep learning has emerged as a promising solution to the problem of hyperspectral reconstruction with large amounts of noise. However, the challenge in using neural networks for long-wave infrared spectral video reconstruction lies in the scarcity of sufficient high-quality data for training. Most existing long-wave infrared hyperspectral datasets consist of single-point spectral data collected through spectrometers. Therefore, we propose to develop the imaging and noise model of the USIRS, allowing us to generate corresponding low-resolution data from high-resolution ones using this model and subsequently train the neural network at the pixel spectrum level.

The signal received by the USIRS is a combination of the target radiation and background radiation:

$$L\left(\lambda \right)=L_t\left(\lambda \right)\tau \left(\lambda \right)+L_p\left(\lambda \right)$$

(1)

Here, $L_t\left(\lambda \right)$ represents the radiation spectrum of the target on the lens of the USIRS. $L\left(\lambda \right)$ is the radiation reaching the detector, $\tau \left(\lambda \right)$ is the spectral response of the imaging system, and $L_p\left(\lambda \right)$ is the background radiation. The radiation reaching the the detector is converted into a current and then quantified as a digital number (DN):

$$DN=\mathcal{P}\left(\varphi L\right)+N_d+N_q$$

(2)

In this equation, $\mathcal{P}$ is the photon noise operator, $\varphi$ is the quantum efficiency of the detector, $N_d$ is the dark current in the circuits, and $N_q$ is the quantization noise. Both the dark currents $N_d$ and $N_q$ quantization noise are assumed to obey the Gaussian distribution, with $\mathcal{N}({\mu }_d,{\sigma }_d)$ and $\mathcal{N}({\mu }_q,{\sigma }_q)$, respectively.

The imaging and noise model provides a pipeline for converting publicly available high-resolution spectral data into synthetic measurements of the USIRS, thus providing paired data for neural network training.

2.3. Pixel-Based Hierarchical Spectral Transformer

We propose the Hierarchical Spectral Transformer (HST), a model that utilizes self-attention to capture long-range dependencies and spectral correlations. This model employs a multistage architecture to progressively refine spectral and spatial features, thereby enhancing the reconstruction accuracy. This approach preserves both global and local spectral information, improving the performance in LWIR spectral image reconstruction tasks.

The network structure of the proposed HST is illustrated in Figure 2.

2.3.1. Positional Encoding

The input of the network is the measured intensity, denoted as $I\in {\mathbb{R}}^l$ in different spectral channels, concatenated with the corresponding positional encoding, denoted as $Pos\in {\mathbb{R}}^{2Ml}$. Each feature’s position, denoted as m, is transformed into a positional embedding via a trigonometric transformation:

$$\begin{array}{cc}\hfill Pos\left(m\right)=& [sin(m\times 2^0),cos(m\times 2^0),\hfill \\ \hfill & sin(m\times 2^1),cos(m\times 2^1),\hfill \\ \hfill & \dots \hfill \\ \hfill & sin(m\times 2^M),cos(m\times 2^M)]\hfill \end{array}$$

(3)

In this equation, M is the maximum encoding dimension, set to 5 in our system. The combined input is as follows:

$$S=I\oplus Pos\in {\mathbb{R}}^{(2M+1)l}$$

(4)

The combined input is fed into the multistage transformer. This process ensures that each position has a unique encoding, allowing the model to comprehend the order of the input data.

2.3.2. Hierarchical Representation

We employ a point-based transformer to process the input intensity. In contrast to standard visual transformers that work with patches, this method divides the input spectral intensity into separate point tokens. These tokens are then processed using multiple transformer blocks, each integrating multi-head self-attention computations.

We construct a hierarchical representation by reducing the token count with the help of merging layers as we progress deeper into the stages. These merging layers combine the features of every two adjacent tokens and apply a linear layer to the combined features. This process results in halving the token count while doubling the channel depth. All these stages collectively yield a hierarchical representation. In the k-th stage, the output of the transformer is

$$S^{\left(k\right)}=Transformer\left\{S^{(k-1)}\right\}$$

(5)

In this equation, $S^{\left(k\right)}\in {\mathbb{R}}^{(2M+1)l/2^k}$ is the output of the k-th stage transformer, and $S^{(k-1)}\in {\mathbb{R}}^{(2M+1)l/2^{(k-1)}}$ is the input of the k-th stage transformer. In addition, $k=1,2,3,\dots$. In the first stage, we define the input of the first stage as $S^{(k-1)}=S$.

In our experiments, we use a 5-stage representation to convert the 17 channels into a single channel but with a feature dimension of 32. In each stage, we repeat the transformer block $L=4$ times.

2.3.3. Attention Mechanism in Transformer

Each stage employs multi-head attention to facilitate token reduction. The attention mechanism allows the model to focus on the most relevant parts of the input data by assigning different weights to different parts of the input.

In the transformer, the input of the k-th stage of the transformer $S^{(k-1)}$ is first linearly transformed by three learnable weight matrices, $W_Q, W_K,$ and $W_V$, to obtain the queries Q, keys K, and values V, respectively. We denote the queries at the i-th position as $Q_i$. And the keys and values follow the same rule. The weight from the i-th and j-th position is computed as follows:

$${\widehat{\alpha }}_{i,j}=\mathrm{softmax}\left({\displaystyle \frac{Q_i{K_j}^T}{h}}\right)$$

(6)

where h is the length of the dimension of queries and keys. The attention at the i-th position is computed as follows:

$$B_i=\sum _j{\widehat{\alpha }}_{i,j}{V}_j$$

(7)

The output is a weighted sum of the values, allowing the model to focus on the most relevant parts of the input data. At the end of this stage, we concatenate the attention $B=\mathrm{concat}(B_1,B_2,\dots ,B_l)$ and further pass the matching features through a linear layer:

$$S^{\left(k\right)}=\mathrm{linear}\left\{B\right\}$$

(8)

In each stage, the attention mechanism can be repeated L times to produce a deep neural network. In this case, the linear layer is applied only when the last attention mechanism outputs.

2.3.4. Implementation Details

Our network training employs the Mean Squared Error (MSE) loss, which is mathematically defined as follows:

$$MSE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(s_i-{\widehat{s}}_i)}^2$$

(9)

In this equation, n represents the total number of data points, $s_i$ is the ground truth spectrum, and ${\widehat{s}}_i$ is the reconstructed spectrum. This formula calculates the average of the squared differences between the ground truth and predicted spectrum. Larger errors are penalized more heavily, pushing the model to reduce significant discrepancies and aim for more accurate predictions. The use of MSE loss enhances the precision and reliability of our network, ensuring consistent and reliable results.

We trained our network with a batch size of 100 for 1000 epochs. The Adam optimizer was used with an initial learning rate of 0.001, which decays by a factor of 0.5 every 100 epochs. The minimum learning rate was set to $1\times {10}^{-5}$. The HST was implemented using PyTorch 1.8, and the training process took approximately 3 h on an NVIDIA 3090 GPU.

3. Results

We first evaluated the effectiveness of our method using synthetic data.

This synthetic dataset, employed for the evaluation, comprises four sections. The first section includes 1289 samples from the ASTER LWIR spectral library, selected for their distinctive LWIR attributes. The ASTER library is a comprehensive collection housing over 2300 spectra of natural and human-made materials. It covers a wide range of materials, and the wavelengths range from 0.4 to 15.4 µm. The library includes the spectra of various substances such as rocks, minerals, lunar soils, terrestrial soils, human-made materials, meteorites, vegetation, snow, and ice. The second section contains 1000 simulated samples of blackbody radiation, ranging from 100K to 1000K, in line with Planck’s law. The third section presents the gas emission spectra of 24 characteristic LWIR gases, including Methane (CH4), Ethylene (C2H4), ammonia (NH3), sulfur hexafluoride (SF6), Cyclohexane (C6H12), Methanol (CH3OH), Acetone (CH3COCH3), trimethylamine ((CH3)3N), Cyclopropane (C3H6), Propylene (C3H6), trans-2-butene (C4H8), Butadiene (C4H6), Ethylene oxide (C2H4O), Dimethylamine (HN(CH3)2), vinyl chloride (C2H3Cl), 1-Butene (C3H8), Acetylene (C2H2), Propyne (C3H4), Dimethyl ether (C2H6O), Acetaldehyde (CH3CHO), Chloroethane (C2H5Cl), Chloromethane (CH3Cl), Methylamine (CH3NH2), and Sulfur dioxide (SO₂). Lastly, the fourth section simulates the radiation spectrum of gas in front of a blackbody, based on the model in [6], and includes 2400 samples.

We began with a qualitative evaluation of the proposed HST’s ability to handle four different spectrum types—minerals, blackbody, gas emission, and gas transmission—and to generate high-resolution spectrum reconstruction, as depicted in Figure 3. The reconstructed data aligned closely with the ground truth across all types, highlighting the effectiveness of our reconstruction method. This figure emphasizes the importance of accurate spectral data replication in applications such as remote sensing, spectroscopy, and material identification.

For a quantitative evaluation of the reconstruction’s performance, we employed general metrics. The correlation coefficient, for example, provides insights into the linear relationship between two spectra:

$$r=\sum _{i=1}^l{\displaystyle \frac{(s_i-\overline{s})(g_i-\overline{g})}{l{\sigma }_s{\sigma }_g}}$$

(10)

where s and g represent the target and ground truth spectra, respectively. $s_i$ and $g_i$ denote the sampled spectra indexed with i. $\overline{s}$ and $\overline{g}$ are the means of s and g, respectively. ${\sigma }_s$ and ${\sigma }_g$ represent the standard deviations of s and g, respectively. Lastly, l refers to the length of each spectrum.

The Root Mean Squared Error (RMSE) is another commonly used metric for comparing the differences between a reconstructed spectrum and the ground truth. The RMSE is the square root of the average of these differences squared. Using s and g as the target and ground truth spectra, respectively, the RMSE is defined as follows:

$$RMSE=\sqrt{{\displaystyle \frac{1}{l}}\sum _{i=1}^l{(s_i-g_i)}^2}$$

(11)

where l is the length of each spectrum vector, ${\sum }_{i=1}^l$ is the sum from $i=1$ to l, $s_i$ is the sampled result for the reconstructed spectrum at index i, and $g_i$ is the sampled result for the ground truth at index i. ${(s_i-g_i)}^2$ is the squared difference between the spectrum and the ground truth at index i. The RMSE is always non-negative, and a lower value is generally more desirable.

The Peak Signal-to-Noise Ratio (PSNR) is a widely used metric in spectrum reconstruction for assessing the quality of a reconstructed spectrum in comparison to the ground truth [36]. Using s and g as the target and ground truth spectra, respectively, the PSNR is defined as follows:

$$PSNR=20·{log}_{10}\left({\displaystyle \frac{MA{X}_s}{\sqrt{MSE}}}\right)$$

(12)

where $MA{X}_s$ is the maximum possible spectrum intensity value. For a normalized spectrum, this is set to 1. $MSE$ is the Mean Squared Error between the target and ground truth spectra and is defined as follows:

$$MSE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(s_i-g_i)}^2$$

(13)

where n is the length of the spectrum, and $s_i$ and $g_i$ are the intensity values of the reconstructed and ground truth spectra, respectively, at the i-th position. A higher PSNR indicates better image quality, indicating less distortion due to noise.

We quantitatively evaluated our proposed methods and compared them with various other methods. In Figure 4, our proposed HST is compared with other neural network methods and interpolation methods for spectrum reconstruction for four materials: kaolin, a 770 K blackbody, vinyl chloride gas emission, and propene gas transmission. The correlation coefficient of the reconstructed spectrum with the ground truth is indicated in the legend. Our proposed HST consistently demonstrated the highest fidelity to the ground truth across all materials. The Linear and Cubic methods offer simple approximations, while deep learning approaches—MLP [32], CNN [26], and Transformer [35]—demonstrate varying degrees of improvement, with the HST model achieving superior performance.

We further assessed the proposed HST and other methods at different noise levels on synthetic data, as shown in

Table 1. Comparison of different methods with various noise levels using synthetic dataset.

	Noise Level	0	0.1	0.2	0.5
Linear interp.	RMSE	0.1133	0.1346	0.1793	0.3532
	Correlation	0.9558	0.9433	0.9134	0.8008
	PSNR	17.54	16.28	13.98	8.16
Cubic interp.	RMSE	0.1139	0.1686	0.2629	0.5953
	Correlation	0.9593	0.9180	0.8420	0.6868
	PSNR	17.93	14.33	10.13	2.75
MLP [32]	RMSE	0.1208	0.1130	0.1767	0.2153
	Correlation	0.9568	0.9599	0.9011	0.8420
	PSNR	18.04	18.35	14.63	12.83
CNN [26]	RMSE	0.0103	0.0233	0.0428	0.1076
	Correlation	0.9989	0.9968	0.9906	0.9565
	PSNR	33.87	29.22	24.53	18.03
Transformer [35]	RMSE	0.0061	0.0238	0.0436	0.1089
	Correlation	0.9994	0.9970	0.9904	0.9563
	PSNR	36.51	29.46	24.45	17.99
HST (ours)	RMSE	0.0059	0.0212	0.0378	0.1074
	Correlation	0.9995	0.9976	0.9929	0.9566
	PSNR	37.16	30.41	25.74	18.02

Red: the best performance. Blue: the second best performance.

. The performances of various interpolation and neural network methods (Linear Interpolation, Cubic Interpolation, MLP, CNN, Transformer, and HST) in spectrum reconstruction using synthetic data across different noise levels (0, 0.1, 0.2, 0.5) were compared. Performance was measured using the Root Mean Squared Error (RMSE), correlation, and Peak Signal-to-Noise Ratio (PSNR). The HST model exhibited the best performance, with the lowest RMSE and highest correlation and PSNR values across almost all noise levels, as highlighted in red. The CNN and Transformer models follow closely, as marked in blue.

We then evaluated our method using experimental data. The experimental data used for the evaluation consist of laboratory experiments with 11 types of gases and field experiments with a blackbody and two gases. The gases in the laboratory experiments included Methane (CH₄), Ethylene (C₂H₄), ammonia (NH₃), sulfur hexafluoride (SF₆), trimethylamine ((CH₃)₃N), Cyclopropane (C₃H₆), Propylene (C₃H₆), trans-2-butene (C₄H₈), butadiene (C₄H₆), Ethylene oxide (C₂H₄O), and vinyl chloride (C₂H₃Cl), while the gases in the field experiments included ammonia (NH₃) and sulfur hexafluoride (SF₆). In the laboratory experiments, a blackbody was used as an ideal background, and a customised gas chamber was placed between the blackbody and prototype with a 0.5 m optical path. Under these conditions, the measured gases were released from gas bottles into the gas cell. During the experiment, the background blackbody temperature was set at 50 °C. The gas temperature, humidity, and pressure in the gas cell were 23.1 °C, 40%, and 1 atm, respectively. The gas charged into the gas cell with 0.5%, 1%, and 2% purity, corresponding to path-integrated concentrations of 2500, 5000, and 10,000 ppm-m, respectively. The path integral concentration can be determined by multiplying the concentration with the gas cell’s optical path.

We initially compared our proposed HST with other methods using laboratory experiments with gas transmission spectra. As illustrated in Figure 5, our proposed HST accurately reconstructed the high-resolution spectrum from the low-resolution input (Linear Interpolation, in solid blue line) for four different types of gas transmission spectra (Cyclopropane, Butadiene, vinyl chloride, and Methane) across a wavelength range of 7 to 14 $\mathsf{\mu }\mathrm{m}$. In comparison to the MLP, CNN, and Transformer methods, our proposed HST effectively captured the details that were lost by other methods, such as the peak at 9.5 $\mathsf{\mu }\mathrm{m}$ for Cyclopropane. Overall, our proposed HST outperforms the baseline networks.

We also quantitatively evaluated the performance of different methods in

Table 2. Quantitative comparison of different methods using experimental laboratory data. The HST method (ours) outperforms all the others, with the lowest RMSE, highest correlation, and highest PSNR, highlighting its superior accuracy in spectral data prediction. The second best performance is achieved by the Transformer method.

Method	Metric	Performance
Linear interp.	RMSE	0.1149
	Correlation	0.9386
	PSNR	18.30
Cubic interp.	RMSE	0.1338
	Correlation	0.9059
	PSNR	16.28
MLP [32]	RMSE	0.1241
	Correlation	0.9248
	PSNR	17.67
CNN [26]	RMSE	0.0437
	Correlation	0.9866
	PSNR	24.87
Transformer [35]	RMSE	0.0422
	Correlation	0.9880
	PSNR	25.32
HST (ours)	RMSE	0.0333
	Correlation	0.9915
	PSNR	26.78

Red: the best performance. Blue: the second best performance.

. This table illustrates the performance of various interpolation and learning methods in terms of three metrics: Root Mean Square Error (RMSE), correlation, and Peak Signal-to-Noise Ratio (PSNR). The Linear and Cubic Interpolation methods show moderate performance, with RMSE values of 0.1149 and 0.1338, respectively. The MLP method offers a slight improvement with an RMSE of 0.1241 and a correlation of 0.9248. The CNN method significantly enhances the performance, achieving an RMSE of 0.0437 and a correlation of 0.9866. Among all the methods, the HST (ours) method demonstrates the best performance, with the lowest RMSE of 0.0333, the highest correlation of 0.9915, and the highest PSNR of 26.78, indicating its superior predictive capabilities. The second best performance is achieved by the Transformer method, marked in blue, with an RMSE of 0.0422 and a correlation of 0.9880. This table emphasizes the effectiveness of our HST method in accurately reconstructing the high-resolution spectrum.

Figure 6 illustrates the evaluation of our method using field experimental data. The experiment’s goal was to test the effectiveness of the spectral imaging in complex scenes. In an open field, we released SF6 and NH3 gases, each with a concentration of 40,000 ppm, marked by red and green boxes, respectively. The images were captured from approximately 100 m away and 15 m above ground. The ground’s temperature, around 35 °C, created a strong thermal contrast against the cooler gas temperature of 22 °C. This contrast helped highlight the gas targets. However, it also introduced significant noise into the raw data, alongside the inherent Poisson noise. Our HST was trained using synthetic data. We then used the HST to reconstruct the high-resolution spectrum from the noisy input intensity for each pixel. We selected three different targets for this process: a road, SF6 gas, and NH3 gas. The middle section of the figure offers a detailed comparison of the input, reconstructed, and reference spectral data for these three materials. The input data appear noisy and distorted when compared to the reference spectrum, primarily due to background and Poisson noise. However, the graphs in this section demonstrate the strong correlation between the input spectra and the network’s reconstruction against the reference spectrum. This emphasizes the network’s high accuracy in preserving spectral features. The displayed quantitative results further highlight the consistency and precision of our reconstructions.

4. Conclusions

The proposed Hierarchical Spectral Transformer presents an innovative solution to challenges in long-wave infrared spectral image reconstruction. By concentrating training on pixel-based LWIR spectra and incorporating an LWIR spectral noise model into the data simulation pipeline, the HST effectively bridges the gap between synthetic and experimental data, reducing the need for high-quality paired data. The multistage architecture of the HST provides a detailed and accurate reconstruction by progressively refining the spectral features and using spectral-wise multi-head self-attention to focus on relevant spectral features. This approach surpasses traditional CNNs, which struggle with noise and the unique characteristics of the LWIR spectrum. Despite the inherent challenge of the nature of the reconstruction problem, the HST demonstrates promising results in both synthetic and experimental data, indicating its potential for real-world applications in environmental monitoring, climate studies, and even carbon neutrality efforts.