Learning a Fully Connected U-Net for Spectrum Reconstruction of Fourier Transform Imaging Spectrometers

Abstract

Fourier transform imaging spectrometers (FTISs) are widely used in global hyperspectral remote sensing due to the advantages of high stability, high throughput, and high spectral resolution. Spectrum reconstruction (SpecR) is a classic problem of FTISs determining the acquired data quality and application potential. However, the state-of-the-art SpecR algorithms were restricted by the length of maximum optical path difference (MOPD) of FTISs and apodization processing, resulting in a decrease in spectral resolution; thus, the applications of FTISs were limited. In this study, a deep learning SpecR method, which directly learned an end-to-end mapping between the interference/spectrum information with limited MOPD and without apodization processing, was proposed. The mapping was represented as a fully connected U-Net (FCUN) that takes the interference fringes as the input and outputs the highly precise spectral curves. We trained the proposed FCUN model using the real spectra and simulated pulse spectra, as well as the corresponding simulated interference curves, and achieved good results. Additionally, the performance of the proposed FCUN on real interference and spectral datasets was explored. The FCUN could obtain similar spectral values compared with the state-of-the-art fast Fourier transform (FFT)-based method with only 150 and 200 points in the interferograms. The proposed method could be able to enhance the resolution of the reconstructed spectra in the case of insufficient MOPD. Moreover, the FCUN performed well in visual quality using noisy interferograms and gained nearly 70% to 80% relative improvement over FFT for the coefficient of mean relative error (MRE). All the results based on simulated and real satellite datasets showed that the reconstructed spectra of the FCUN were more consistent with the ideal spectrum compared with that of the traditional method, with higher PSNR and lower values of spectral angle (SA) and relative spectral quadratic error (RQE).

1. Introduction

Imaging interferometers, also called Fourier transform imaging spectrometers (FTISs), have several superiorities, including high throughput, high wavenumber accuracy, high resolution, lower weight, and smaller size [1,2,3]. Due to these advantages, FTISs have been widely used in remote sensing of the moon, earth, and space [4,5]. With rapid developments in technology, various types of interferometric spectrometers that can obtain tens or even hundreds of hyperspectral data [6] have been established, such as the temporary modulated imaging interferometer proposed by Michelson, the spatial interferometric imaging spectrometer, and the spatially-temporally modulated interferometric imaging spectrometer [7].

Although there are currently a great variety of FTISs, the key part of the FTIS is the interferometer [8,9,10,11]. Figure 1 shows the optical layout of a type of FTIS, a large aperture static imaging spectrometry (LASIS) system. In theory and practice, the interferometer divides the light from the detection object into two coherent beams that will interfere on the optical sensors. By altering the optical path difference (OPD) of two beams of light, we will acquire a set of interference fringes [12]. To obtain the spectral information, the spectrum reconstruction (SpecR) is used to transform interference fringes into spectral information, and the SpecR algorithm determines the quality and the further application of the spectral data [13,14].

The fast Fourier transform (FFT) plays a vital role in the SpecR of FTISs, which correlates spectral profiles with interference fringes [15]. However, the spectrum recovered by FFT using the original interferogram may result in many side lobes. This is mainly because the interferogram cannot be scanned to infinity as the theoretical calculation of FFT, and then the value of interference fringes, out of the scope, is not recorded, which would cause spectrum leakage. The side lobes would reduce the accuracy of the spectral amplitude value. To solve the spectrum leakage of SpecR, many apodization functions (also called window functions) [16,17,18], such as triangle, Hamming, Hanning, and Happ–Genzel algorithms, have been applied to smooth down an interference curve to zero at the end of the recorded region. Even though apodization algorithms can reduce spectrum leakage, the full width at half maximum (FWHM) of the spectrum becomes large, resulting in a decrease in the spectral resolution. Thus, the peaks and valleys are hard to identify in the reconstructed spectrum. In addition, the spectral resolution of the FFT relies on the length of the interference data. However, long interference data cannot be transmitted due to bandwidth limitations in practical remote sensing applications and insufficient sampling points to record interference signals. When the signal is not long enough, it is difficult for FFT-based methods to recover the high precision spectrum [19]. The multiple signal classification (MUSIC) algorithm, which is maturely used in modern spectrum estimation, was introduced in the SpecR process to improve the resolution. Jian et al. [20] introduced the MUSIC algorithm and the autoregressive (AR) [21,22] model for better performance in the spectral recovery of the pulse signal. These algorithms all achieved good spectrum reconstruction in terms of resolution, however, they were not suited for polychromatic SpecR, particularly for low signal-to-noise ratio data. Hence, currently most of the methods for SpecR are still based on FFT [1,3,4].

Recently, deep learning has been well developed in the fields of image processing [23], such as image restoration [24], super-resolution [25], object detection [26], image segmentation [27], image classification [28], and remote sensing [29]. More closely related to our study task, the multilayer fully connected network (FCN) was applied for high and low-level computer vision problems. Although the FCN has the disadvantages of a large network and many parameters when processing image signals, its fully connected structure can well represent the relationship between the interference curve and spectrum information. The key process of SpecR, the Fourier transform, can be regarded as the linear FCN. To date, deep learning techniques have been successfully used for many studies of FTIS, such as medical image classification [30], digital staining [31], and diagnosis of abnormal thyroid function [32]. However, to the best of our knowledge, deep learning techniques have not been used in the SpecR of FTISs.

The transformation of the interference curve to a spectral curve using the Fourier transform can be defined as an end-to-end model, and it is possible for deep learning to achieve significant improvements in computational accuracy for the end-to-end model [33]. Motivated by this fact, we designed the fully connected U-Net (named FCUN) as an end-to-end mapping between interference fringes and spectral profiles. The architecture can extract and fuse multiscale features. The FCUN was fundamentally different from existing SpecR methods without needing too many intermediate steps, such as apodization or autoregression for recovering the spectrum. Furthermore, the intermediate steps were formulated as fully connected layers, which were involved in the optimization. In the proposed method, the entire SpecR pipeline was fully obtained through learning, with little pre/post-processing.

The main contributions of this research are listed as follows:

• A fully connected U-Net was proposed for spectrum reconstruction of FTISs. The network directly learned an end-to-end mapping between the interference/spectrum information.
• We proved that deep learning could improve the performance of spectrum reconstruction of FTISs, and could achieve good quality spectra. This guided the design of the algorithm of SpecR.
• The proposed FCUN method can recover spectra that exceed the physical spectral resolution of the FTISs, within the limited MOPD of interference fringes.
• Two datasets of SpecR were built for the first time and could be applied for further studies.

2. Theoretical Foundation

The traditional data processing of FTISs and limitations were introduced in the first part. Then, the relationship between FTIS data processing and the deep learning method was illustrated. The architecture of a typical U-Net was also described.

2.1. Traditional Data Processing of FTIS

Derived from Fourier transform spectroscopy theory, the interferogram could be acquired by FTIS from the spectrum of the detected target [20]. The interference fringes of the interferogram can be idealized as:

$$I(\Delta )={\displaystyle {\int }_{-\infty }^{+\infty }B(\sigma )}{e}^{j2\pi \sigma \Delta }d\sigma$$

(1)

where B is the spectrum, I is the interferogram, $\Delta$ is the path difference, and $\sigma$ is the wavenumber. The reconstructed spectrum of the detected target is defined as:

$$B(\sigma )={\displaystyle {\int }_{-\infty }^{+\infty }I(\Delta )}{e}^{-j2\pi \sigma \Delta }d\Delta$$

(2)

Equations (1) and (2) are the main relationship between the interferogram and the spectrum, and the fundamental theory of Fourier transform imaging spectrometers (FTISs).

For actual FTISs, the spectral range and the maximum optical path difference (MOPD) are limited [16], so the true relationship of the interferogram and the reconstructed spectrum are listed as follows:

$$I(\Delta )={\displaystyle {\int }_{{\sigma }_1}^{{\sigma }_2}B(\sigma )}{e}^{j2\pi \sigma \Delta }d\sigma ={\displaystyle {\int }_{-\infty }^{+\infty }{T}_B(\sigma )B(\sigma )}{e}^{j2\pi \sigma \Delta }d\sigma$$

(3)

$$B(\sigma )={\displaystyle {\int }_{-L}^{+L}I(\Delta )}{e}^{-j2\pi \sigma \Delta }d\Delta ={\displaystyle {\int }_{-\infty }^{+\infty }{T}_I(\Delta )I(\Delta )}{e}^{-j2\pi \sigma \Delta }d\Delta$$

(4)

where ${\sigma }_1$ and ${\sigma }_2$ represent the range of wavenumbers and L is the maximum optical path difference (MOPD), which determines the FWHM of the distinguishable wavenumber of the instrument [9]. The FWHM of the distinguishable wavenumber is defined as:

$$\delta \sigma =1/2L$$

(5)

The spectral resolution of the instrument is given by:

$$R=\sigma /\delta \sigma$$

(6)

$T_I$ is the truncation function, and the Fourier transform of the truncation function is the instrument line shape. Since the nature of the Fourier transform determines the Gibbs phenomenon of the instrument linear function, the apodization function is applied to suppress the side lobes of Gibbs oscillation. Figure 2 shows the reconstructed spectrum of the laser, the blue line is the spectrum recovered by the original FFT, and the red line is the spectrum reconstructed using triangle apodization and FFT. It is clearly shown that triangle apodization can greatly suppress side lobes; however, the apodization method also brings a decrease in spectral resolution. The traditional SpecR processing of the interferogram [16] mainly includes preprocessing, phase correction, apodization, and FFT. Therefore, the spectral resolution of the hyperspectral data reconstructed by traditional SpecR methods has poor performance.

2.2. Relationship of FTIS Data Processing and Deep Learning Method

The FFT in traditional FTIS data processing can be viewed as a simplified, fully connected, single-layer network, as shown in Figure 2. We considered an interference curve as a vector with a length of l*1. In the FFT-based method, every point of the l*1 vector multiplied a coefficient on the corresponding band and then summed to obtain the spectral information of the band. As described in Equation (4), we can obtain spectral information of different bands by changing $\sigma$. As illustrated in Figure 3a, the FFT solver can be viewed as a simple linear mapping operator and was applied to transform the interference curve to the spectral curve (length is s*1). Figure 3b shows that the multilayer FCN and FFT transform have the same connection mode, but there was a nonlinear activation function in the multilayer FCN. It is possible to add nonlinear factors to increase the expression ability of the model. Significantly, his can acquire compelling results in numerous studies [25,34].

The above discussion shows that the FFT-based SpecR method can be viewed as a kind of FCN (without nonlinear mapping). According to Equation (4), the linear structure will produce side lobes with a limited optical path difference (OPD) [6]. In contrast, the multilayer FCN can be viewed as a nonlinear mapping with a stronger fitting ability than the linear model, and may produce a high-resolution spectrum without side lobes by the nonlinear transform. Motivated by the analogy, the deep learning method (fully connected U-Net) has been introduced to reconstruct the spectrum of FTIS.

2.3. The Architecture of a Typical U-Net

A U-Net is a U-shaped network structure which is widely used in the fields of image segmentation, image classification, image restoration, etc. The U-Net can extract multiscale features by encoder/decoder operating and can fuse features well by a concatenating procedure. The diagram (Figure 4a) of the original U-Net and an example of biomedical image segmentation (Figure 4b) are from Ronneberger’s work [27]. As shown in Figure 4b, the U-Net can identify the small separation borders of the adjacent touching cells and the precise contouring of cells, which indicates that the network can extract multiscale information and can fuse the extracted information well. Therefore, we used this structure for spectrum reconstruction (SpecR).

The process of the original U-Net for biomedical image segmentation is described in pseudocode as shown in Algorithm 1.

Algorithm 1: biomedical image segmentation network U-Net

Input: the pairs of training data, biomedical images Ib and the corresponding segmentation maps Ms

Output: the segmentation Ms’

Step 1: load the training data and test data pairs (Ib, Ms)

Step 2: perform initialization of the network model, and randomly generate parameters θu of the U-Net

Step 3: training processing

For epoch in max number of Epochs

Randomly load (Ib (n), Ms (n)) (n = [1, 2, …, N]), N is the batch size

Calculate Ms’, and use the cross entropy to evaluate the consistency of segmentation results and true values

Use gradient descent method to update the parameters θu, and save θu with the best segmentation result of training data

End

Step 4: the trained parameters θu are used to get the segmentation results of test datasets

3. Proposed Fully Connected U-Net for Spectrum Reconstruction

Our goal is to build a deep learning model that transforms interference curve I into spectral curve F(I), which is as similar as possible to the ground truth high spectral resolution spectrum B. In this work, we chose U-Net as the backbone network because the architecture contains the encoder/decoder paths that can extract multiscale features [27], and the concatenated structure of U-Net can fuse features well [33]. These features, with sufficient information, were the basis for accurate high-resolution spectrum reconstruction.

3.1. The Basic Architecture

Figure 5 and

Table 1. Detailed parameters of the FCUN.

Layer	Input Size	Output Size	Layer	Input Size	Output Size
Individual FC-1	(J,1)	(512,1)	FC upsampling-1	(16,1)	(32,1)
Individual FC-2	(512,1)	(1024,1)	FC upsampling-2	(32,1)	(64,1)
FC downsampling-1	(1024,1)	(512,1)	FC upsampling-3	(64,1)	(128,1)
FC downsampling-2	(512,1)	(256,1)	FC upsampling-4	(128,1)	(256,1)
FC downsampling-3	(256,1)	(128,1)	FC upsampling-5	(256,1)	(512,1)
FC downsampling-4	(128,1)	(64,1)	FC upsampling-6	(512,1)	(1024,1)
FC downsampling-5	(64,1)	(32,1)	Linear transform	(1024,1)	(K,1)
FC downsampling-6	(32,1)	(16,1)

show the schematic diagram and detailed parameters of the FCUN for SpecR. First, two individual fully connected (FC) layers were applied to extract the detailed features from the interference input. As a result, at each layer, the number of extracted features was expanded, which could be viewed as an increase in spectral resolution by extending the length of the interference curve. Then, six fully connected (FC) downsampling layers were used to calculate high-level features in growing numbers on a coarser scale. These features were connected with computed high-resolution features later using upsampling processing. To overcome the problem of gradient explosion and vanishing in the training processing, the residual connection was adopted to fuse upsampling/downsampling features. This type of residual connection in the U-Net structure can best exploit the potential of pairwise operations of different lengths by increasing the number of layers of upsampling/downsampling. There were six levels connected in the proposed network in this paper, and the length of the operational features of each level was half and double that of the previous level in the downsampling and upsampling layers, respectively. In the final layer, a linear fully connected structure was applied to obtain the spectral result. For interference curve of length J as input and K bands of spectrum to be estimated, the model returned predictions as K*1 vectors.

Table 2. Fully connected block diagram of the base architecture.

Layer	Operation	Input	Output
FC upsampling	Linear(n, 2n), Relu()	n*1 vector	(2n)*1 vector
FC downsampling	Linear(n, n/2), Relu()	n*1 vector	(n/2)*1 vector
Individual FC	Linear(n, m), Relu()	n*1 vector	m*1 vector

shows the detailed structure of FC layers. The linear (in_len, out_len) indicates a linear transform to the input vector. In this paper, in_len indicates the size of the input sample and out_len stands for the size of the output sample. In the FC upsampling/downsampling layers and individual FC layer, the linear transform was followed by ReLU activation. It is possible to add more fully connected layers and ReLU activations to improve the nonlinearity. However, this can increase the complexity of the model, and more training time and data are needed. Hence, we used a single fully connected layer in the upsampling/downsampling layers because it has already produced compelling results.

3.2. Loss Function

Learning the end-to-end mapping function F(I) for SpecR requires the estimation of the parameters of the FCUN. This is accomplished by minimizing the loss between the reconstructed spectral curves and the corresponding ground truth spectral curves. Here, the parameters are denoted by θ, the reconstructed spectrum was defined as F(I; θ) or B^’, and the ground truth spectral curves are represented by B. Note that a good setting of the loss function is the key point to solving the parameter estimation. In general, the spectral angle (SA) [3] and the relative spectral quadratic error (RQE) [4] are two efficient indicators for quantitative evaluation of the error of the reconstructed spectrum. They are defined by:

$$SA(B^{(n)},B^{(n)\prime })={\cos }^{-1}\left({\displaystyle \sum _{k=1}^K{B}_k^{(n)}\cdot B_k^{(n)\prime }}/\left(\sqrt{{\displaystyle \sum _{k=1}^K{\left(B_k^{(n)}\right)}^2}}\cdot \sqrt{{\displaystyle \sum _{k=1}^K{\left(B_k^{(n)\prime }\right)}^2}}\right)\right)$$

(7)

$$RQE(B^{(n)},B^{(n)\prime })=\sqrt{{{\displaystyle \sum _{k=1}^K\left(B_k^{(n)}-B_k^{(n)\prime }\right)}}^2}/{\displaystyle \sum _{k=1}^K{B}_k^{(n)}}$$

(8)

where $B^{(n)}$ stands for the ground truth of the spectrum, $B^{(n)\prime }$ is the reconstructed spectrum, and K is the number of spectral bands. Then, the loss functions based on the above indicators are defined as:

$$l_{SA}=\frac{1}{N}{\displaystyle \sum _{n=1}^{N}SA(B^{(n)},B^{(n)\prime })}$$

(9)

$$l_{RQE}={\displaystyle \sum _{n=1}^{N}RQE(B^{(n)},B^{(n)\prime })}$$

(10)

where N is the total number of examples for the SpecR model training. Accordingly, we defined the final loss function as a linear combination of the two above components in Equations (9) and (10).

$$l_F=l_{SA}+\tau l_{REQ}$$

(11)

Using $l_F$ above as the final loss function favors a high degree of consistency of shape and absolute value between the reconstructed spectrum and ground truth. Here, the value of τ was set as 0.5.

3.3. The Process of Generating SpecR Network FCUN

Our SpecR method based on FCUN is realized by training process. First, randomly select the training data and test data from the dataset; the paired interference curves and the corresponding spectral curves are used for training. Second, perform initialization of the network model by setting hyperparameters; the specific values are shown in Section 4.3, and randomly generate parameters θ of the FCUN. Third, train the network and update the parameters θ; after several iterations of parameter updates, the parameter θ is finally obtained.

The process of generating a SpecR network FCUN is illustrated in pseudocode as shown in Algorithm 2.

Algorithm 2: Spectrum reconstruction network FCUN

Input: the pairs of training data, interference curves and the corresponding spectral curves (I, B)

Output: the reconstructed spectral curves described as F(I; θ) or B’

Step 1: load the training data and test data pairs (I, B)

Step 2: perform initialization of the network model, and randomly generate parameters θ of the FCUN

Step 3: training processing

For epoch in max number of Epochs

Randomly load (I(n), B(n)) (n = [1, 2, …, N]), N is the batch size

Calculate B’$,\mathrm{and}\mathrm{use}\mathrm{the}\mathrm{function}{l}_F$ to evaluate the loss

Update parameters θ, and save θ with the best reconstruction result of training data

End

Step 4: the trained parameters θ are applied to reconstruct the spectral curves B’ of test datasets

4. Experiments and Results

4.1. Datasets

There is no dedicated dataset for spectrum reconstruction of FTIS, especially for testing deep learning methods. We evaluated our SpecR network FCUN on two spectrum reconstruction datasets that we built. Both datasets were divided into two parts: interference curves and spectral curves.

In the first dataset, we simulated spectrum reconstruction datasets according to Equation (3). The real collected spectral data and simulated pulse spectrum were used to generate the corresponding interference curves, making the data sample diverse. We produced 13.7% asymmetric interferograms (almost single-sided) for calculations, as asymmetric structures can retain more spectral information at a fixed detector length for interference [19]. The unit OPD that indicates the OPD of two adjacent pixels was set as 206.96 nanometers (nm), which is the same as the visible and near infrared (VNIR) hyperspectral imager (HSI), a kind of FTIS on Chinese HJ-2(A/B) satellites that launched on 27 September 2020, and the MOPD was 45,738.16 nm in the simulation. Note that simulated interferograms were ideally sampled with one sample exactly located, and there was no inconsistency of the detector response and phase error. The spectra ranged from 455.06 nm to 898.73 nm with a resolution of 3.18 nm and 202 bands in total. The simulated pulsed spectrum with 202 bands was generated by adding Gaussian noise on the full zero spectrum. The two examples of simulated data were shown in Figure 6 and Figure 7.

The second dataset was constructed using current operational satellite images acquired by visible and near-infrared FTIS for testing the proposed FCUN for real data. The dataset included 640,000 samples of different locations on the Earth’s surface. One sample contained an interference curve where the length was 256 and a corresponding spectral curve with 100 bands equally spaced range from 450 nm to 920 nm. All interference curves were corrected by preprocessing and phase correction [19]. Considering tests of the spectral resolution of the reconstructed spectrum by different compared SpecR methods, the long side of corrected asymmetric interference curves was cut to different lengths for testing. Because different interference curve lengths indicate different MOPDs, the smaller MOPD resulted in lower spectral resolution according to Equations (5) and (6). If an algorithm can recover interference curves with a small MOPD into the high-resolution spectrum, the method can perform well in SpecR for FTISs. There is an example of real data shown in Figure 8.

4.2. Methods for Comparison

To test the proposed method, we compared the proposed FCUN with the most effective spectrum reconstruction (SpecR) method, the state-of-the-art FFT-based method [19], and the multitaper power spectral density estimate (MPSDE) method proposed by Thomson [35]. The PyTorch package was used for training and testing implementations [36].

4.3. Implementation Details

The proposed SpecR model was trained on a computer with an AMD Ryzen 9 4900HS CPU, 16 GB memory, and an RTX2060 MaxQ GPU with 6 GB GPU memory. During the training, the batch size was set as 2048, and the Adam algorithm was used for parameter optimization. The parameters for the learning rate, β_1, β_2, and epsilon were 0.001, 0.9, 0.999, and 10⁻⁹, respectively. The training and test data were normalized to avoid the problem of overfitting and further strengthen the training, and dropout was set to 0.5 in the training process. Our network was implemented on the PyTorch framework using the GPU, with 2000 training cycles on the first dataset and 1000 training cycles on the second dataset, because the training cycles were enough to produce a stable network. Finally, we saved the model with minimum training loss. The detailed parameters are listed in the

Table 3. Hyper-parameters of FCUN are used in this paper.

Parameter Name	Parameter Setting
Batch size	2048
Optimizer	Adam
Initial learning rate	0.001
β1	0.9
β2	0.999
Epsilon	10−9
Epochs	2000/1000
Dropout	0.5

. In our experiments, the length of the input interference curve for the first dataset was 256, and the lengths for the second data set were 200 and 150, respectively; therefore, we trained three FCUN models for spectrum reconstruction.

4.4. Evaluation

The spectral angle (SA) [3] in Equation (7), the relative spectral quadratic error (RQE) [4] in Equation (8), and peak signal to noise ratio (PSNR) [25] were used to evaluate the spectral shape, magnitude error, and similarity between the reconstructed spectrum and the ground truth of the two test datasets, respectively.

4.4.1. Evaluation of the First Dataset

In the experiments of the first dataset, 640,000 paired examples of interference curves and spectral curves were used for training and 480,000 paired examples were used for testing. The results of SA, RQE, and PSNR of the first test dataset, summarized in

Table 4. Quantitative evaluation of the tested methods.

Algorithm	SA	RQE	PSNR (dB)
FFT	0.0655	0.0056	29.61
MPSDE	1.1169	0.2720	16.07
FCUN	0.3227 × 10−6	0.0672 × 10−6	38.43

, indicated that the proposed FCUN improved the accuracy of the reconstructed spectrum obtained by simulation. Note that the proposed FCUN yielded the lowest mean SA score in the experiments, which indicated that the shape of the reconstructed spectrum was more consistent with the reference value. The proposed FCUN also gained the best RQE score, and thus, the absolute value of the reconstructed spectra showed better agreement with the ideal reference spectra. In addition, the FCUN achieved the highest mean PSNR score, suggesting that the reconstructed results have relatively better perceptual quality. The specific reconstructed examples from the test dataset are shown in Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13.

Figure 9a showed the typical recovered spectra of the wall. It is evident from the plot that the spectrum recovered by FFT and MPSDE had broader oxygen absorption peaks at 761.9 nm than FCUN. It can be proven that the FCUN achieved better performance in the spectral resolution. In addition, the relative errors [37] (Figure 9b) of spectra reconstructed by FCUN, FFT, and MPSDE to the reference spectrum were 0.65%, 5.60%, and 37.47%, respectively. Therefore, the FCUN had a greater potential to obtain accurate intensity values for spectral recovery.

The recovered spectrum of the tree was shown in Figure 10a, and the intensity value of the proposed FCUN nearly overlapped with the reference spectrum with a relative error of 1.24% (Figure 10b). The FFT produced a spectrum with little deviation from the actual reference with a relative error of 12.07% (Figure 10b), and the intensity value of the spectrum recovered by the MPSDE was quite different from the reference spectrum with a relative error of 64.34% (Figure 10b). Thus, the proposed FCUN also had a better performance for SpecR of the tree.

Pulse spectrum reconstruction was also a vital part for measuring the goodness of the algorithms. Figure 11 shows that the peak positions of the spectra recovered by all compared methods were consistent with the reference values. This result indicated that the three methods can perform well in wavelength accuracy. The spectra reconstructed by FCUN (the purple dash-dot line) and MPSD (the orange line) have few side lobes, while the spectrum of FFT (the red line) has side lobes that are approximately 3% of the high peak. This meant that the FFT-based SpecR method did not perform well in terms of the spectral accuracy of the relative value. Noted that the recovered spectral FWHMs of FCUN, MPSDE, and FFT were 2.22 nm, 4.00 nm, and 5.33 nm, respectively. A smaller FWHM value indicated a higher spectral resolution. Hence, the FCUN and MPSDE performed relatively well in terms of spectral resolution and spectral accuracy for one pulse spectrum.

Figure 12 shows the reconstructed spectra of multiple pulses with scattered peaks, which were obtained by the FFT, MPSDE, and FCUN. All methods accomplished the task of spectral reconstruction of the multipulse interference curve in their own way and obtained precise spectral locations of three peaks. As shown by the red line, the reconstructed spectrum obtained through FFT has more side lobes than the other two methods, which indicated that the intensity value of each band would be influenced by the adjacent bands. This would affect the accuracy of subsequent hyperspectral quantification analysis. As shown by the orange line, even though the spectrum recovered by MPSDE had fewer side lobes than the FFT-based method, the intensity value at 719.9 nm was nearly one-third of the spectrum of FFT (the red line), which would alter the spectral properties of intensity. Compared with FFT and MPSDE, the proposed FCUN had a better performance in spectral resolution, because the spectrum of the FCUN (the purple dash-dot line) with a very narrow FWHM is more consistent with the reference spectrum (the blue line) than the other two methods.

The reconstructed results of the multipulse spectrum with very close pulse peaks using the three compared methods are displayed in Figure 13. The FFT and MPSDE can only reconstruct two peaks in the blue dotted box, while the reference spectrum had three closed peaks, which indicated that the resolving power of FFT and MPSDE was not enough to distinguish very close peaks in the spectrum. The recovered spectrum of the FCUN had three peaks in the blue dotted box with accurate pulse positions. Although the intensity of the middle peak in the 642.7 nm band was different from the reference spectrum, FCUN still performed better than FFT and MPSDE.

Based on the comparison of the reconstructed pulse spectra of Figure 11, Figure 12, Figure 13, it was clear that the FCUN achieved better performance than the other two methods.

4.4.2. Evaluation of the Second Dataset

To assess the effectiveness of the proposed method on real data, 440,000 samples were extracted from the second dataset as training samples and 200,000 as test samples, with interference curves of lengths 200 and 150, respectively. The SA and RQE of the reconstructed results of the test samples using real interference curves with different data lengths were shown in

Table 5. Quantitative evaluation of the tested methods for interference curves of different data lengths.

Algorithm	Data Length	SA	RQE	PSNR (dB)
FFT	150	0.1513	0.0180	24.19
	200	0.1393	0.0154	24.81
FCUN	150	0.1379 × 10−6	0.1189 × 10−6	29.61
	200	0.1198 × 10−6	0.1004 × 10−6	34.55

. Because the MPSDE does not perform well in the continuous spectrum as shown in Table 4, we only compared the state-of-the-art FFT and the proposed FCUN. From the table, the spectra reconstructed by FCUN had a lower value in both quantitative evaluation indicators than the FFT with the same data length. Thus, the FCUN had better performance than the FFT in the real SpecR of FTIS. Additionally, with the decrease of the data length ranges from 200 to 150, the SA of FFT increased 8.6%, the RQE of FFT increased 16.9%, and the SA and RQE of FCUN also increased 15.1% and 18.4%, respectively. This also indicated that a larger length of the interference curve can produce a better spectrum. The specific reconstructed examples from the test dataset are shown in Figure 14 and Figure 15.

The reconstructed spectra of nudation in the test dataset and the relative errors to the reference spectrum are shown in Figure 14. The proposed FCUN method provided recovered spectra that were more consistent with the ideal reference spectrum than the spectra produced by FFT (Figure 14a). The consistency rankings of details (the red rectangle box in Figure 14a) with the reference spectrum were the FCUN with a data length of 200, the FCUN with a data length of 150, the FFT with a data length of 200, and the FFT with a data length of 150. In addition, the relative errors (Figure 14b) of the compared methods with different data lengths were consistent with the detailed comparison of Figure 14a. Moreover, the compared methods provided results of peaky spectrum in Figure 15a in agreement with the results in Figure 14a; the reconstructed spectra of FCUN were in closer agreement with the ideal reference spectrum with the minimal relative error (Figure 15b). Hence, the FCUN had better performance than the FFT on the real dataset, and data with longer interference length gave better results in both methods.

4.4.3. Evaluation of the Noise Data

Our proposed method also had good generalization performance on simulated noisy interferograms. Here, different signal-to-noise ratio (SNR) data were generated by adding different levels of Gaussian noise on the original interferograms according to the following Equation:

$$SNR=20\ast {\log }_{10}(mean(I_{org})/\sqrt{MSE})$$

(12)

where I_org denotes the interference curve with DC component, MSE is the variance of the noise, and mean(*) denotes the mean value. According to the different SNRs, we calculated the corresponding MSE and superimposed Gaussian noise with mean value of 0 and variance value of MSE on interferograms.

Figure 16 shows the true color images of the reconstructed hyperspectral data cubes of the first dataset for different signal-to-noise ratio (SNR) [8] interferograms. The outputs of the FFT contained a large amount of noise even though the SNR is 50 dB (Figure 16b). While the FCUN exhibited superior visual performance when the SNR was 40 dB (Figure 16e), there was no obvious noise on the images (Figure 16d,e). Moreover, the details of the sky in the red rectangular box are in the lower left corner of each subfigure, and compared to the FFT method, the proposed FCUN method had less noise in the subimage and the sky was more uniform. Most remarkably, the mean relative error (MRE) [37] of FFT with SNRs of 40 dB and 50 dB were 31.97% and 11.93%, respectively, while the MRE of FCUN with SNRs of 40 dB and 50 dB were 5.28% and 2.68% (Figure 17). The FCUN gained nearly 80% relative improvement over FFT for MRE. This gain is due to the learned representations of the FCUN.

Figure 18 shows the true color building images of the reconstructed hyperspectral data cubes of the first dataset for different SNR interferograms. From Figure 18d, the outputs of the FCUN had a similar visual quality to that of the reference image (Figure 18a). While the visual quality of the recovered results of FFT was poor, some details of the area corresponding to the red rectangular box were lost (Figure 18b,c). As shown in Figure 19, the proposed approach had the lowest relative error for the interferograms with an SNR of 50. The FFT had higher relative errors than the FCUN. This indicated that the nonlinear method had better performance in SpecR than the linear approach.

In the actual imaging system of FTIS, the data will be corrupted by Gaussian (readout) noise and Poisson (shot) noise. In the following experiment, we accounted for both noises. The original data were generated by adding Poisson and Gaussian noise according to light conditions and characteristics of the FTIS on Chinese HJ-2(A/B) satellites. For the FTIS, each photon received by the detector produces a voltage of 4.2 millivolt; under the default imaging conditions, the system saturation digital number (DN) value was 4095 when the system voltage was 2V. So for the FTIS system, one DN value corresponds to 116 photons. In this experiment, we simulated Poisson noise for three different luminance conditions (low luminance, medium luminance, and high luminance). The average DN of the interferograms are about 500, 1000, and 2000, corresponding to the photon numbers of 58000, 116000, and 232000, respectively. The simulation of Gaussian noise was based on variance of dark current of the FTIS, which is equivalent to 97 photons.

Figure 20, Figure 21 and Figure 22 show the true color images of the reconstructed hyperspectral data cubes for the three different luminances. From the three figures, the outputs of the FCUN have a similar visual quality to that of the reference image. The reconstructed results of the FFT had noise-induced color distortion, and some details were obscured by noise. As shown in Figure 23, the relative error of FCUN is smaller than FFT, the corresponding mean relative errors of FFT and FCUN were 3.33% and 0.87%, and the FCUN gained nearly 73.9% relative improvement than the FFT for SpecR of mixed noise (Gaussian and Poisson) data.

Through quantitative and qualitative comparisons, the proposed FCUN can accurately reconstruct the spectral information of Fourier transform imaging spectrometers (FTISs) and has better stability than the compared methods.

4.4.4. Evaluation of the Structure of FCUN

We then examined the sensitivity of the network to different layers and structures. In previous experiments, we set the proposed FCUN with two individual FC layers, six FC downsampling/upsampling layers, and one linear transform layer. Here, we change the number of layers of the FCUN to obtain two other SpecR networks with different sizes. First, we cut the two individual FC layers, and only use four downsampling/upsampling layers for SpecR as a lightweight FCUN. Second, we cut FC downsampling/upsampling layers from six to three of the proposed FCUN to evaluate the performance of layers in U-Net. All the experiments were done using the first dataset, and the hyperparameters were consistent with those in Table 3. The quantitative evaluation of the different structures for testing the dataset was listed in

Table 6. Quantitative evaluation of the different network structure.

Algorithm	SA	RQE	Parameters Size (Mb)
The proposed FCUN	0.3227 × 10−6	0.0672 × 10−6	8.65
Lightweight FCUN (The first)	0.8527 × 10−6	0.3209 × 10−6	0.54
FCUN with 3 layers cut (The second)	0.4298 × 10−6	0.0740 × 10−6	8.56

.

The ranking of reconstruction accuracy of the compared structures is: the proposed FCUN, the FCUN with 3 layers cut, and lightweight FCUN, in order. This suggested that a reasonably larger size of the FCUN could extract richer spectral information, which would lead to better reconstructed results of FTIS. It is important to emphasize that the feature vector upsampled in the beginning for two layers is important for quality improvement of SpecR. However, the size of the model’s parameters and deployment speed also increases with a larger network size. Therefore, the choice of the network parameters should always be a compromise between speed and performance based on the constraint of computing power.

5. Discussion

Some recent research has shown that deep learning techniques can improve the performance of FTISs in many fields [32,33,34] because of the ability of feature extraction and nonlinear transformation. However, as far as we know, the models of deep learning have not been used in the spectrum reconstruction (SpecR) of FTISs. In this study, we first developed a nonlinear model based on deep learning for spectrum reconstruction (SpecR). The above experiments supported our hypothesis that the nonlinear deep learning model can achieve good performance in spectral resolution (Figure 11) and intensity value (Figure 8 and Figure 9) for SpecR. This accords with previous studies that deep learning can improve the performance of FTISs. The SA and RQE of the proposed FCUN are lower than the state-of-the-art methods, and the FCUN can also achieve better visual quality of the reconstructed images (Figure 16, Figure 18, Figure 20, Figure 21, Figure 22). A possible explanation for the better performance of the FCUN might be that the U-Net structure with residual connection is suitabl for extracting and fusing multiscale features of interference curve. Compared with the other network architectures, the FCUN also worked better in SpecR, which proves the effectiveness of the proposed FCUN model to some extent.

The reconstructed result of the multipulse spectrum with very close pulse peaks may be unsatisfactory because of the limited spectral resolution of FCUN. It is possible that the training data is not abundant enough or the structure of FCUN needs further improvement. It could be hypothesized that the performance of the deep learning method for SpecR can be further promoted by increasing the network layers and using a more efficient network structure, such as attention and transformer [38].

Despite promising results of the FCUN for most testing data, further work on analyzing network structure and building up abundant datasets for spectrum reconstruction should be conducted.

6. Conclusions

In this paper, we proposed a novel fully connected U-Net (FCUN) for spectrum reconstruction (SpecR) of Fourier transform imaging spectrometers (FTISs). We first showed that the traditional fast Fourier transform (FFT) SpecR methods can be reformulated into a deep learning algorithm. The presented approach, FCUN, learned an end-to-end mapping between interference curves and reconstructed spectrum, with little pre/post-processing before the model training, and outperformed the competing state-of-the-art methods for testing simulated and real datasets. It is also important to note that the proposed FCUN easily reconstructs a spectrum with higher resolution than the other algorithms, although it produced some distortion when there were many pulses close to each other in the spectrum. Moreover, the FCUN achieved excellent SpecR results for noisy interferograms. We speculate that better performance can be further obtained by exploring more effective structures of the network, larger training datasets, and different training strategies. In addition, our method has the advantage of stability and simplicity and could be further used for SpecR of a wide range of FTISs.