Comparative Study of Spectral Reconstruction Algorithms Based on Asymmetric Influence Factors

Abstract

Spectral reconstruction is an important way to acquire a high-spatial-resolution multispectral image, and the spectral reconstruction algorithm is the key to its implementation. This work conducts a comparative study of spectral reconstruction algorithms under different asymmetric influencing factors to provide an overview of their performance. Seventeen spectral reconstruction algorithms with different mathematical bases were implemented and compared regarding their adaptability to response format, imaging noise, spectra type, exposure change, and the quality of the reconstructed images. Based on the principles and characteristics of the algorithm, qualitative and quantitative statistical analyses of the results were carried out. Results show that most of the current algorithms: (1) are adaptive to raw and image signal processing (ISP) responses for spectral reconstruction, (2) decrease their spectral reconstruction accuracy with an increase in imaging noise, (3) give poor performance in reconstructing smooth spectra using non-smooth spectra, and (4) exhibit different degrees of sensitivity to exposure changes. In addition, the image quality reconstructed using the raw response is superior to the ISP response.

1. Introduction

Spectral reflectance is one of the important features that characterizes the physical and chemical properties of substances and also serves as the fingerprint of color information. Spectral reflectance enables in the situ, non-contact identification of substances and high-fidelity color reproduction. Regarding spectral acquisition, spectrophotometers can only perform spectral measurements at single points, and it is time-consuming and high-cost for current scanning-based multispectral imaging cameras to acquire multispectral images with high spatial resolutions. Spectral reconstruction technology enables the acquisition of high-spatial-resolution multispectral images in a snapshot way, but the imaging accuracy is affected by various factors, among which the spectral reconstruction algorithm is the most significant.

Spectral reconstruction is implemented based on the digital imaging model, wherein under specified imaging conditions, an imaging system captures training samples with a known spectral reflectance. The response of the training samples is extracted and used to construct a mapping model from response to spectral reflectance using spectral reconstruction algorithms. This model is then applied to testing samples under the same imaging conditions to perform spectral reconstruction, resulting in multispectral images of the testing target. This technology supports applications such as high-fidelity art reproduction, dermatology, cultural heritage preservation, remote sensing, agricultural inspection, computer vision, and so on [1,2,3,4,5,6].

With the development of different kinds of spectral reconstruction systems, different spectral reconstruction algorithms have been proposed, ranging from traditional machine learning-based to current deep learning-based [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29], and the complexity and performance of the algorithms keep increasing. Machine learning-based algorithms, based on different mathematical bases such as regression, interpolation, kernel function, principal component analysis, compression sensing, and optimized versions of them, have been widely studied for spectral reconstruction. Some of them have also been used as support techniques for imaging-based spectral measurement in developed industrial testing and analysis equipment such as the VeriVide DigiEye system. In recent years, to meet the demand for low-cost snapshot multispectral imaging, multispectral reconstruction from RGB images based on deep learning has also been studied. Some excellent spectral reconstruction models such as HSCNN, AWAN, HRNet, and MST++ have been forthrightly proposed. However, due to limitations in the size of multispectral datasets, the robustness of these models in practical use has not been fully verified. Recently, Bian et al. developed a spectral reconstruction network (SRNet) together with a broadband multispectral filter array (BMSFA) to realize fast on-chip hyperspectral imaging with a high spatial and spectral resolution, bringing great progress for BMSFA-based snapshot spectral imaging [6]. However, most existing research on spectral reconstruction algorithms is conducted under controlled conditions in the laboratory or using the synthesized RGB images of the public radiance/reflectance datasets. There is no comprehensive comparative study of the existing algorithms, and their adaptability to different influence factors remains unclear.

Unlike in the controlled experiment conditions (such as light source, illumination level, lighting uniformity, imaging parameters, etc.), in the laboratory spectral reconstruction in open environments faces challenges such as varying light sources, uneven illumination, diverse object categories, and different imaging distances. These factors lead to changes in imaging parameters and may introduce different levels of imaging noise under different lighting intensities, making it difficult to ensure the consistency of imaging conditions in spectral modeling and its applications, thereby causing different spectral reconstruction errors. For example, when using a spectral reconstruction model constructed under CIE D65 to reconstruct spectral reflectance under CIE A, significant reconstruction errors result, as shown in Figure 1. Similarly, as shown in Figure 2, if a spectral reconstruction algorithm is not exposure-invariant, even under the same light source, using a model constructed at one specific exposure level (such as Exposure*1) to reconstruct the spectral reflectance of objects at different exposure levels (such as Exposure*K) can cause the reconstructed spectral curves to deviate from the true spectral curves to varying degrees. This deviation prevents the accurate application of reconstructed spectra for object analysis and identification.

Therefore, focusing on the comparison of spectral reconstruction algorithms in their performance for RGB image-based spectral reconstruction and using X-Rite ColorChecker charts (USA, Michigan) and some publicly available multispectral reflectance and radiance image datasets, this study conducted a comparative study of seventeen machine learning-based spectral reconstruction algorithms in two image response formats: camera raw format and ISP format. The adaptability of these algorithms to response format, imaging noise, spectra type, and exposure change were tested. In addition, the quality of the images reconstructed by each algorithm was also compared. The performance of these algorithms was evaluated and compared using the root-mean-square error (RMSE), CIEDE2000 color difference (∆E00), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM).

Qualitative analysis and quantitative comparison of the experimental results were performed based on the principles and characteristics of the algorithm, and statistical analysis was performed using one-way ANOVA and Tukey’s post hoc tests (α = 0.05). The main contributions of this research can be summarized as follows: (1) most existing spectral reconstruction algorithms are suitable for both raw and ISP responses, with better performance achieved in raw format responses; (2) the accuracy of spectral reconstruction decreases as imaging noise increases; (3) reconstruction error increases significantly when non-smooth radiance spectra are used to reconstruct smooth reflectance spectra, indicating the sensitivity of algorithms to the type of spectra (radiance vs. reflectance) employed for training and testing; (4) all existing algorithms exhibit varying degrees of sensitivity to exposure changes in ISP format, whereas only three algorithms remain exposure-invariant in raw format, thus limiting their applicability in open environments; and (5) reconstructed images derived from raw responses demonstrate higher quality compared to those obtained from ISP responses. More discussions on these algorithms regarding each influencing factor can be found in the Section 5 and Section 6.

2. Principles of Spectral Reconstruction

Based on the imaging principles of digital cameras, which assume a linear photoelectric conversion, the camera response can be represented by Equation (1).

$$d_i={\displaystyle {\int }_{\lambda \mathit{min}}^{\lambda \mathit{max}}l\left(\lambda \right)s_i}\left(\lambda \right)r\left(\lambda \right)d\lambda +n_i$$

(1)

where the camera response d_i is related to the channel i (i = 1, 2, 3) of a pixel or sample, λ is the wavelength, ranging from λ_min to λ_max in the visible wavelength, l(λ) is the spectral power distribution of the light source, s_i(λ) is the spectral sensitivity of the ith channel of the camera, r(λ) is the spectral reflectance of a pixel or sample, and n_i is the noise of the digital camera. Ignoring the noise, Equation (1) can be written in matrix notation, as expressed by Equation (2).

$$\mathit{d}=\mathit{Mr}$$

(2)

where d is the response vector, M is the overall spectral sensitivity matrix of the digital camera, which includes the product of l(λ) and si(λ), and r denotes the spectral reflectance vector. For spectral reconstruction, the goal is to reconstruct the high-dimensional reflectance r_rec from the low-dimensional camera response vector d, as indicated by Equation (3):

$${\mathit{r}}_{\mathit{rec}}=\mathit{Qd}$$

(3)

where Q represents the spectral reconstruction model or matrix. Typically, if the overall spectral sensitivity matrix M of the system is accurately calibrated, the spectral reconstruction algorithm can directly reconstruct r_rec. However, since M includes multiple imaging system parameters and is difficult to calibrate accurately, almost all current spectral reconstruction research employs a sample-based training method, as illustrated in Figure 1.

3. Spectral Reconstruction Algorithms

Table 1. The machine learning-based spectral reconstruction algorithms in this study. According to the training method of the spectral reconstruction model, these algorithms are roughly divided into four categories: global training, global-weighted training, local training, and local-weighted training (last column in Table 1), and a brief introduction of the key idea, input, complexity, and pros/cons of each algorithm are provided.

No.	Algorithms	Key Idea	Input	Complexity	Advantages/Disadvantages	Training Method
1	RLS [13]	Non-linear regression: polynomial model + least-square (LS) + regularization	Can be three-/multi-channel response.	Low: global training and global reconstruction	Advantages: high computational efficiency Disadvantages: all training samples contribute equally	Global training
2	Kernel [24]	Non-linear regression: feature boost with Gaussian kernel + ridge regression
3	PLS [16]	Non-linear regression: polynomial model + partial least-square (PLS)
4	PCA [15]	Linear regression: principal component analysis (PCA) + LS			Advantages: better for linear response. Disadvantages: all training samples contribute equally
5	PCAKX [20]	Non-linear regression: polynomial model + PCA + LS	Designed for three-channel response input. Some methods also work with multi-channel response inputs.		Advantages: better for non-linear response Disadvantages: all training samples contribute equally
6	BLS [27]	Non-linear regression: polynomial model + deep feature mapping and reinforcement + ridge regression		Medium: global training and global reconstruction	Advantages: more response features are used Disadvantages: all training samples contribute equally
7	CpS [19]	Linear regression: PCA + iterative threshold (ITH) solver		Medium: global training and sample-wise reconstruction	Advantages: better for linear response. Disadvantages: all training samples are treated equally
8	SC [21]	Linear mapping: sparse dictionary + orthogonal matching pursuit (OMP)
9	RBFSR [18]	Non-linear mapping: interpolation in multi-dimensional spaces based on radial basis function (RBF) network			Advantages: better for non-linear response Disadvantages: all training samples are treated equally
10	SCO [21]	Non-linear mapping: polynomial model + sparse dictionary based on K-SVD + OMP
11	Li [26]	Non-linear mapping: training sample optimization + thin-plate splines radial basis function (TPS-RBF) interpolation
12	wPCA [15]	Non-linear regression: with/without polynomial model + PCA + adaptive global weighting		High: global-weighted training and sample-wise reconstruction	Advantages: adaptive spectral reconstruction Disadvantages: low computational efficiency	Global-weighted training
13	SRLLA [17]	Linear mapping: locally linear approximation (LLA) + weights mapping from response space to spectral space		High: local training-based for sample-wise reconstruction		Local training
14	Zhang [22]	Non-linear regression: polynomial model + color characterization + local training + LS
15	Liang [25]	Non-linear regression: polynomial model + rgb difference + local training + adaptive weighting		High: local-weighted training for sample-wise reconstruction		Local-weighted training
16	Cao [23]	Linear mapping: color difference + local training + adaptive weighting
17	Shen [14]	Linear regression: Wiener estimation + local training + adaptive weighting			Advantages: better for linear response. Disadvantages: low computational efficiency.

shows the methods to determine the spectral reconstruction matrix Q in Equation (3). According to the training method of the spectral reconstruction model, the current spectral reconstruction algorithms were roughly divided into four categories: global training, global-weighted training, local training, and local-weighted training (last column in Table 1) and we briefly introduce the key idea, input, complexity, and pros/cons of each algorithm to facilitate subsequent discussion. Based on the implementation method and the mathematical principles used, the key ideas of the algorithms are can be imperfectly divided into linear regression, non-linear regression, linear mapping, and non-linear mapping. The inputs of each algorithm are briefly summarized as three-channel (for RGB images) and multi-channel (for multi-channel images), and their complexity is classified as low, medium, and high, depending on the training and reconstruction style of the algorithm. Furthermore, the pros/cons of each algorithm are briefly discussed from the aspects of computational efficiency, the contribution of training samples in the training stage, and the adaptability to response format.

For the spectral reconstruction algorithms based on global training, Connah and Hardeberg first applied the polynomial model to spectral reconstruction, demonstrating that polynomial expansion is significantly superior to linear transformation [13]. Xu et al. used a Gaussian kernel interpolation algorithm to solve the transformation matrix for spectral reconstruction [24]. To address the over-fitting issue of the least-squares method, Shen et al. proposed a spectral reconstruction algorithm based on partial least squares [16]. Xiao et al. introduced a two-step spectral reconstruction algorithm based on PCA for skin reflectance measurement [20]. Yang et al. applied a broad learning system to spectral reconstruction [27]. Zhang et al. proposed a spectral reconstruction algorithm based on compressed sensing [19]. Arad et al. also proposed a spectral reconstruction algorithm based on compress sensing, assuming a linear mapping from RGB to spectral space and optimization improvements based on a polynomial model can be integrated into it to enhance the performance [21]. Nguyen et al. used radial basis functions to recover the spectral reflectance of images [18]. Additionally, Li et al. introduced a spectral reconstruction algorithm based on a thin-plate spline radial basis function together with a training sample optimization strategy [26].

For the spectral reconstruction algorithms based on global-weighted training, Agahian et al. used a weighted PCA for spectral reconstruction based on the Euclidean distance between the training and testing samples, achieving better results than traditional PCA methods [15]. For the spectral reconstruction algorithms based on local training, Li et al. introduced a K-nearest-neighbor-weighted spectral reconstruction algorithm based on local linear approximation, but the linearity of the color space mapping to the spectral reflectance space lacked rationality [17]. Zhang et al. developed a spectral reconstruction algorithm based on a combination of CIE XYZ values under multiple light sources [22]. For the spectral reconstruction algorithms based on local-weighted training, Liang et al. proposed an adaptive weighting algorithm in the camera response value space, further improving reconstruction accuracy by obtaining locally optimal training samples [25]. Cao et al. proposed a spectral reconstruction algorithm based on small-color-difference-sample weighting and tested the application effects of four different weighting methods [23]. Shen et al. proposed a Wiener spectral reconstruction algorithm based on adaptive weighting, applied to scanner spectral characterization modeling [14].

Although existing spectral reconstruction algorithms have made significant progress, most related research has been conducted in laboratory environments, without considering the challenges posed by varying light sources and uneven illumination in outdoor or natural open-environment applications. To address this, Khan et al. proposed a spectral reconstruction method for open environments based on light source estimation and correction [18], theoretically solving the issue of inconsistent light sources but failing to handle exposure-level effects. Wen et al. introduced an illumination-independent interpolation reconstruction method [31]. Lin et al. applied root polynomial regression to spectral reconstruction, addressing exposure variation and uneven illumination issues [32], but the algorithm compressed the feature dimensions of the modeling data and limited the reconstruction accuracy. Additionally, based on the theory of invariant spectral features [33], Liang et al. investigated a lightweight reconstruction model using data augmentation and attention mechanisms to address the robustness issues of Lin’s physically plausible spectral reconstruction algorithm [10]. Although these algorithms have achieved preliminary results, there remains a gap for practical use in open environments. More robust spectral reconstruction algorithms based on RGB images and for open-environment applications still need to be developed.

In addition, with the continuous improvement of multispectral image datasets, some progress has been made in spectral reconstruction based on deep learning [6,7,8,9,10,11,12]. In the field of spectral reconstruction using deep learning networks, Shi et al. constructed the residual network HSCNN-R using adaptive residual blocks [7]. Li et al. proposed a new adaptive weighted network AWAN, targeting the dependency between camera priors and intermediate features [8]. However, convolutional neural networks still have limitations in capturing non-local self-similarity and long-range dependencies. Furthermore, Cai et al. introduced a spectral multi-head attention transformer, further improving reconstruction accuracy [9]. Bian et al. proposed a hybrid neural network that combines the core features of transformer and CNN architectures for efficient, high-precision spectral reconstruction of the BMSFA sensor raw measurements [6]. Although existing deep learning-based spectral reconstruction algorithms have achieved superior accuracy, they also face issues with their adaptability to exposure changes [10], limiting their practical application. Moreover, the complexity of the actual deployment and the hardware requirements limit the practical application of these deep learning methods. Therefore, in the current stage, the comparative study and analysis of spectral reconstruction algorithms are focused on the traditional machine learning-based algorithms summarized in Table 1, and the comparative study on the current deep learning-based spectral reconstruction model will conducted separately in the future. Specific experimental settings are presented in the next section.

4. Experimental Methods

4.1. Experiment Settings

To compare the performance of the spectral reconstruction algorithms listed in Table 1, theoretical studies were conducted using the spectral sensitivity functions of a Nikon D7200 digital camera (Japan), CIE D65 standard illuminant (CIE), and the spectral reflectance data of an X-Rite ColorChecker chart (USA). The spectral sensitivity functions of the Nikon D7200 were estimated using the method described by Jiang et al. [30]. The estimated camera sensitivity functions of a Nikon D7200 (Figure 3a) and the relative spectral power distribution of CIE D65 (Figure 3b). Based on the imaging model in Equation (1), the linearized simulated camera raw format response was obtained by adding Gaussian white noise. And by applying a gamma correction with a value of 2.2 to the raw format response, the image signal post-processed (ISP) non-linear format response is produced.

Based on the above experimental settings, a systematically comparative study on the adaptability of these spectral reconstruction algorithms was conducted from five aspects: adaptability to response format, imaging noise, spectra type, exposure levels, and the quality of the reconstructed images. More descriptions of each experiment are as follows:

• Experiment 1: Adaptability to camera response format. The response-format adaptability experiment aims to investigate the performance of the algorithms on two data formats: linear raw format response and non-linear ISP format response. This is because both formats of camera response can be and have already been used for spectral reconstruction studies, but no research on how these spectral reconstruction algorithms are adapted to the response formats was explored previously.
• Experiment 2: Adaptability to imaging noise. The noise adaptability experiment aims to explore the extent to which the algorithms are affected by noise levels. In this study, Gaussian white noise was used, as usual, to simulate the imaging noise during the imaging process, and the calculation of Gaussian white noise is shown in Equation (4). The noise levels were set at 1%, 2%, 3%, 4%, and 5%, corresponding to signal-to-noise ratios (SNRs) of 40 dB, 34 dB, 30.2 dB, 28.0 dB, and 26 dB on the horizontal axis of plots in Section 5.

$$SNR(dB)=20\times {\log }_{10}(A_{signal}/A_{noise})$$

(4)

where SNR is the signal-to-noise ratio, measured in dB, A_signal is the amplitude value of the image, and A_noise is the amplitude of the Gaussian noise.

• Experiment 3: Adaptability to spectra type. The spectra type adaptability experiment aims to compare the impact of the smoothness of the training and testing spectra data on the performance of the algorithms. Many publicly available spectra datasets contain non-smooth radiometric multispectral images, while the actual spectra of natural objects are typically smooth. Using inappropriate types of spectra data may cause the reconstructed spectra to deviate from the ground truth. This issue is not fully considered in the current deep learning and RGB image-based spectral reconstruction studies since the dataset lacks information on light sources.
• Experiment 4: Adaptability to exposure changes. The exposure change adaptability experiment aims to compare the adaptability of existing algorithms to changes in exposure levels. This evaluates whether the reconstructed spectral curves maintain the correct profile and change proportionally when only the exposure level is varied. In the experiment, spectral reconstruction models were trained at an exposure level of 1.0 and tested at exposure levels of 0.5, 0.75, 1.0, 1.5, and 2.0. Using the spectral reconstruction results in an exposure level of 1.0 as a reference, the adaptability of each algorithm to the exposure changes can be analyzed.
• Experiment 5: Quality of reconstructed images. The reconstructed image quality experiment aims to compare the corresponding color image quality of multispectral images reconstructed by each algorithm. In total, 18 different multispectral images were selected from the FOSTER, CAVE, and ARAD-1K datasets (each of six) as test subjects and the corresponding sRGB images were calculated using the CIE D65 illuminant and the CIE1964 standard observer color-matching functions. The image quality metrics were calculated using the reconstructed and ground-truth image.

4.2. Experiment Samples

In experiments 1 to 4, the X-rite ColorChecker SG chart (CCSG) (USA) was utilized as the training sample and the X-rite ColorChecker 24 chart (CC) (USA) as the testing sample for spectral reconstruction. For experiment 3, the non-smooth radiance spectra data of the CCSG and CC were extracted from the ICVL dataset, as shown in Figure 4. In experiment 5, the reconstruction tests were performed on 18 multispectral images selected from three datasets, six from each database. Some of the rendered images in the three databases are shown in Figure 5.

4.3. Evaluation Metrics

In experiments 1 to 4, the spectral reconstruction accuracy was evaluated using the commonly used metrics of spectral root-mean-square error (RMSE) and color difference CIEDE2000 [37] (∆E00). In experiment 5, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used to assess the quality of the reconstructed images. The calculation of RMSE is shown in Equation (5), while the methods for calculating PSNR and SSIM are provided in Equations (6) and (7), respectively.

$$\mathit{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}{({\mathit{r}}_{\mathit{rec}}-\mathit{r})}^T({\mathit{r}}_{\mathit{rec}}-\mathit{r})}$$

(5)

where r_rec represents the reconstructed spectral reflectance, r represents the measured reference spectral reflectance, T represents the operator of transpose, and N is the sampling number of spectral reflectance. With the spectral wavelength ranges from 400 nm to 700 nm in a sampling interval of 10 nm, N is equal to 31. Generally, a smaller RMSE indicates a higher reconstruction accuracy.

$$\mathit{PSNR}\left(\mathrm{dB}\right)=10\cdot \mathrm{lo}{g}_{10}\left({\displaystyle \frac{{\mathrm{MAX}}^2}{\mathrm{MSE}}}\right)$$

(6)

where MAX is the maximum possible pixel value of the image, and MSE is the mean squared error between the original and reconstructed images. A higher PSNR value indicates better image quality.

$$\mathit{SSIM}(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+c_1)(2{\mathsf{\sigma }}_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\mathsf{\sigma }}_x^2+{\mathsf{\sigma }}_y^2+c_2)}}$$

(7)

where x and y are the two images being compared, µ_x and µ_y are the mean luminance values of x and y, ${\sigma }_x^2$ and ${\sigma }_y^2$ are the luminance variances of x and y, σ_xy is the luminance covariance between x and y, and c₁ and c₂ are two constants. The SSIM value ranges from −1 to 1, where a value closer to 1 indicates greater similarity between the two images, a value closer to −1 indicates greater dissimilarity, and a value of 0 indicates no similarity.

5. Experimental Results Analysis and Discussion

In this section, the results of each experiment ware presented through qualitative and quantitative analysis and comparison. For each experiment, the adaptability of the different algorithms to the influencing factors was first qualitatively analyzed according to their error metric distribution, and then their performance in terms of the influence factor regarding their mathematical principles were quantitatively compared in Table 1. Finally, an overall discussion of the results of the five experiments is given, and the sensitivity and adaptability of the algorithm to different influencing factors are summarized in a Table.

5.1. Response-Format Adaptability

In the response-format adaptability experiment, the main purpose is to test the performance of different algorithms when using raw and ISP format responses. At the same time, the spectral reconstruction accuracy of different algorithms in different response formats can also be compared. The spectral reconstruction error distributions of different algorithms in raw and ISP format responses are plotted in Figure 6, and the corresponding errors of RMSE and ΔE00 are summarized in

Table 2. The mean and standard deviation spectral reconstruction error of RMSE and ΔE00 of different algorithms in raw and ISP format responses; the top three best algorithms are highlighted.

Algorithm	raw		ISP
	RMSE	ΔE00	RMSE	ΔE00
Kernel	0.0163/0.0105	0.5259/0.4486	0.0165/0.0122	0.5038/0.4499
Liang	0.0161/0.0142	0.4533/0.4220	0.0169/0.0156	0.4747/0.4164
BLS	0.0208/0.0188	0.5580/0.5484	0.0190/0.0155	0.5028/0.3780
wPCA	0.0196/0.0114	1.5046/1.0196	0.0206/0.0128	1.6219/1.1500
PLS	0.0223/0.0141	0.5823/0.4087	0.0188/0.0139	0.5354/0.3942
RLS	0.0224/0.0138	0.7973/0.5066	0.0189/0.0136	0.7694/0.5757
Zhang	0.0217/0.0159	0.5713/0.4652	0.0210/0.0146	0.5337/0.4166
SRLLA	0.0234/0.0249	0.4898/0.7607	0.0215/0.0164	1.9261/1.0065
RBFSR	0.0268/0.0166	0.8417/0.5434	0.0225/0.0159	0.6044/0.4458
Li	0.0301/0.0173	1.0826/0.5605	0.0309/0.0153	2.7319/1.8303
Cao	0.0314/0.0202	4.7830/3.6885	0.0317/0.0183	4.6750/2.9614
SCO	0.0374/0.0278	1.1062/0.7642	0.0319/0.0263	0.9277/0.6102
Shen	0.0199/0.0167	0.5067/0.4569	0.0506/0.0296	4.6947/4.4730
PCAKX	0.0501/0.0266	9.4742/9.6886	0.0374/0.0182	6.9036/6.7619
PCA	0.0354/0.0209	1.5624/1.3035	0.1102/0.0442	11.3020/5.6128
CpS	0.0384/0.0225	1.5429/0.9795	0.1316/0.0463	12.8078/6.8725
SC	0.0443/0.0284	1.5710/1.0555	0.2180/0.0941	18.6610/10.1245

.

A. Qualitative analysis of the response-format adaptability. From the spectral reconstruction error distribution in Figure 6, it can be seen that most of the tested algorithms (the first twelve) do not show an apparent difference for both RMSE and ΔE00 in raw and ISP responses, illustrating they can adapt to both raw and ISP responses for spectral reconstruction. However, for algorithms of Shen, PCAXK, PCA, CpS, and SC, they show an apparent difference for both RMSE and ΔE00 in two format responses, indicating that they are sensitive to the response format for spectral reconstruction. Among them, the PCAXK algorithm is more suitable for non-linear ISP responses, as it is based on non-linear regression and PCA-based dimensional data reduction in calculating the spectral reconstruction matrix, making it less robust to a linear response. The other four algorithms of Shen, PCA, CpS, and SC are entirely based on linear regression or mapping for spectral reconstruction; therefore, they are more adapted to the linear raw response than the non-linear ISP response.

For the first twelve algorithms that adapted to both raw and ISP response for spectral reconstruction, there are two reasons for their adaptability. Firstly, by expanding the input features (the response) into a high-dimensional space through various feature enhancement functions (such as through polynomial expansion, Gaussian kernel, and so on) and solving it using the least-square algorithm, these algorithms can efficiently solve linear regression problems, including the need to fit non-linear data. These non-linear regression algorithms are Kernel, Liang, BLS, wPCA, PLS, RLS, and Zhang. Secondly, for the rest of the five algorithms of SRLLA, RBFSR, Li, Cao, and SCO, no matter whether they are based on linear or non-linear mapping, they are all inherently implemented based on local mapping, making them well adapted to both the linear raw and the non-linear ISP responses.

In addition, the Li algorithm shows almost no difference in the RMSE in the raw and ISP responses, but shows an obvious difference in the ΔE00 in the raw and ISP responses, indicating that the problem of metamerism is more likely to occur when using the Li algorithm for spectral reconstruction in the ISP format response. In summary, except for the PCAXK in Table 1, the rest of the algorithms are all adapted to the raw response. Therefore, spectral reconstruction based on linear raw format response is recommended.

B. Quantitative analysis of the spectral reconstruction accuracy. Based on the qualitative analysis results above and the spectral reconstruction results in Table 2, the last five algorithms of Shen, PCAXK, PCA, CpS, and SC in Figure 6 show large RMSEs and ΔE00s in ISP responses. Therefore, the quantitative analysis of the spectral reconstruction accuracy of different algorithms is conducted based on the result of the raw response.

Referencing the key ideas of all of the algorithms in Table 1, it is easy to find from Table 2 that for the spectral reconstruction results in raw format response, the non-linear-regression-based algorithms (such as Kernel, Liang, BLS, wPCA, PLS, RLS, and Zhang) and local-linear-regression-based algorithms (such as Shen) show relatively good spectral reconstruction accuracy compared to the mapping-based algorithms (such as SRLLA, RBFSR, Li, Cao, SCO, CpS, and SC). This is because for the regression-based spectral reconstruction algorithms the relationship from enhanced raw response features to the corresponding spectral reflectance is directly modeled without any preset assumption for the spectral reconstruction process. However, for the mapping-based spectral reconstruction algorithms, the regression relationship in the low-dimensional response space is usually linearly mapped to the high-dimensional spectral space and this may not always work efficiently, as Connah and Hardeberg have illustrated that sometimes more than three parameters (which can be regarded as image channels or dimensions of the response features) are required to accurately represent a surface spectral reflectance [13]. In addition, the traditional PCA algorithms only use the original three-channel raw response for spectral reconstruction, making the response features not enough as input to establish a good regression model between camera response and corresponding spectral reflectance, and therefore leading to an inferior spectral reconstruction result.

Considering the overall spectral reconstruction results in both raw and ISP responses, the top three best algorithms are highlighted in Table 2. However, in terms of raw response, the Liang algorithm based on local-weighting spectral reconstruction shows the best spectral reconstruction results, with an RMSE of 0.0161 and a ΔE00 of 0.4533, followed by the Kernel and Shen algorithms. From Table 1, although the Kernel algorithm is a global-training-based algorithm, it has actually implemented training sample selection and weighting via the hyper-parameter in the kernel function, therefore making it inherently a local-weighted algorithm. The Shen algorithm is local-weighted and shows similar spectral reconstruction results to the Liang and Kernel algorithms. Furthermore, although the Cao algorithm belongs to the local-weighted category, its spectral reconstruction accuracy based on mapping is much lower than that of the regression-based algorithm. Moreover, one-way ANOVA and Tukey’s post hoc tests were performed to test the difference between the mean values of the error distributions in Table 2 with an acceptable confidence level of α = 0.05.

In summary, for spectral reconstruction using a raw format response, the non-linear-regression-based algorithms with response feature enhancement are usually superior to the mapping-based and thoroughly linear regression-based algorithms. Optimization strategies such as weighting and local training will improve the accuracy of all the current algorithms, but they bring other problems, such as the sensitivity to the exposure changes in the Section 5.4.

5.2. Imaging Noise Adaptability

The results of the noise adaptability test are shown in Figure 7, where Figure 7a,b display the RMSE of all the algorithms in raw and ISP responses. Figure 7c presents the results of removing the PCAKX algorithms in Figure 7a, and Figure 7d shows the results after removing the SC algorithms in Figure 7b. Since the spectral reconstruction results of ΔE00 show similar trends to RMSE in this experiment, they are not discussed separately. The RMSE of different algorithms in raw and ISP responses are summarized in

Table 3. The mean and standard deviation spectral reconstruction error of RMSE of different algorithms in raw format response; the top three sensitive algorithms are highlighted according to the standard deviation.

Algorithm	Signal-to-Noise Ratio (dB)					Std.
	40	34	30.2	28	26
Kernel	0.0192/0.0092	0.0239/0.0099	0.0296/0.0128	0.0341/0.0156	0.0386/0.0184	0.0078
PLS	0.0236/0.0132	0.0265/0.0135	0.0300/0.0148	0.0328/0.0157	0.0356/0.0165	0.0048
RLS	0.0239/0.0129	0.0268/0.0132	0.0304/0.0146	0.0331/0.0156	0.0359/0.0165	0.0048
wPCA	0.0210/0.0103	0.0243/0.0112	0.0302/0.0142	0.0353/0.0176	0.0399/0.0207	0.0077
Liang	0.0194/0.0119	0.0234/0.0126	0.0298/0.0162	0.0361/0.0209	0.042/0.0251	0.0092
Zhang	0.0226/0.0153	0.0263/0.0155	0.0312/0.0169	0.0349/0.0178	0.0394/0.0198	0.0067
Shen	0.0237/0.0168	0.0281/0.0193	0.031/0.0195	0.0345/0.0214	0.0387/0.0237	0.0058
RBFSR	0.0276/0.0165	0.0293/0.0160	0.0324/0.0172	0.0352/0.0176	0.0389/0.0193	0.0045
SRLLA	0.0209/0.0235	0.0268/0.0218	0.0326/0.0219	0.0386/0.0254	0.0447/0.0316	0.0094
Li	0.0307/0.0170	0.0321/0.0169	0.0344/0.0172	0.0367/0.0177	0.0399/0.0186	0.0037
Cao	0.0324/0.0195	0.0342/0.0199	0.0371/0.0226	0.0385/0.0228	0.0400/0.0220	0.0031
PCA	0.0359/0.0206	0.0371/0.0205	0.0389/0.0209	0.0409/0.0216	0.0435/0.0228	0.0030
CpS	0.0388/0.0232	0.0400/0.0223	0.0423/0.0223	0.0446/0.0225	0.0480/0.0232	0.0037
BLS	0.0235/0.0123	0.0304/0.0148	0.0423/0.0196	0.0536/0.0312	0.0678/0.0462	0.0178
SC	0.0448/0.0279	0.0456/0.0277	0.0471/0.0275	0.0486/0.0275	0.0508/0.0277	0.0024
SCO	0.0402/0.0276	0.0464/0.0290	0.0498/0.0328	0.0531/0.0331	0.0563/0.0346	0.0062
PCAKX	0.0505/0.0249	0.0535/0.0221	0.0611/0.0202	0.0709/0.0224	0.0837/0.0264	0.0136

and

Table 4. The mean and standard deviation spectral reconstruction error of RMSE of different algorithms in ISP format response, the top three sensitive algorithms are highlighted according to the standard deviation.

Algorithm	Signal-to-Noise Ratio (dB)					Std.
	40	34	30.2	28	26
Kernel	0.0200/0.0108	0.0262/0.0120	0.0335/0.0154	0.039/0.0184	0.0456/0.0214	0.0091
PLS	0.0223/0.0134	0.0284/0.0151	0.0364/0.0190	0.0429/0.0218	0.0503/0.0246	0.0100
RLS	0.0225/0.0132	0.0288/0.0149	0.0369/0.0185	0.0432/0.0216	0.0505/0.0245	0.0100
wPCA	0.0226/0.0119	0.027/0.0141	0.0343/0.0183	0.0407/0.0218	0.0483/0.0261	0.0092
Liang	0.0204/0.0125	0.0256/0.0143	0.0339/0.0187	0.041/0.0229	0.0488/0.0280	0.0102
Zhang	0.0229/0.0140	0.0275/0.0145	0.0336/0.0165	0.0397/0.0189	0.0463/0.0219	0.0084
Shen	0.0438/0.0193	0.0477/0.0188	0.0541/0.0237	0.0599/0.0294	0.0668/0.0323	0.0083
RBFSR	0.0248/0.0163	0.0293/0.0179	0.0355/0.0211	0.0414/0.0237	0.0481/0.0267	0.0083
SRLLA	0.0223/0.0139	0.0286/0.0144	0.0391/0.0245	0.0454/0.0273	0.0531/0.0391	0.0111
Li	0.0330/0.0157	0.0540/0.0216	0.0956/0.0441	0.1352/0.0660	0.1548/0.0769	0.0463
Cao	0.0324/0.0195	0.0352/0.0227	0.0405/0.0269	0.0433/0.0273	0.0468/0.0281	0.0052
PCA	0.1115/0.0440	0.1143/0.0431	0.1183/0.0415	0.1215/0.0406	0.1251/0.0400	0.0049
CpS	0.1325/0.0466	0.1341/0.0470	0.1366/0.0475	0.1390/0.0481	0.1427/0.0491	0.0036
BLS	0.0213/0.0133	0.0277/0.0133	0.0363/0.0175	0.0476/0.0289	0.0605/0.0509	0.0140
SC	0.2747/0.0890	0.2765/0.0745	0.2793/0.0731	0.2820/0.0749	0.2939/0.0633	0.0068
SCO	0.0308/0.0212	0.0451/0.0231	0.0431/0.0212	0.0532/0.0305	0.0560/0.0260	0.0088
PCAKX	0.0415/0.0192	0.0484/0.0202	0.0590/0.0244	0.0727/0.0286	0.0797/0.0348	0.0143

.

From a qualitative analysis of the changing trends of RMSE of different algorithms with imaging noise under raw and ISP responses, firstly, it is easily found from Figure 7 that the ranking of spectral reconstruction accuracy of each algorithm under different imaging noises is generally in accordance with the results in the Section 5.1, and furthermore, the spectral reconstruction accuracy in raw response is better than that in ISP response. Secondly, from Figure 7, the algorithms of PCAKX and BLS are significantly affected by the imaging noise in both the raw and ISP format responses, and the Li algorithm is significantly affected by the imaging noise in the ISP format response. Except for the PCAXK and BLS algorithms and the Li algorithm in ISP response, the RMSEs of most of the rest of the algorithms shows a steady upward trend as the imaging noise increases.

From a quantitative comparison of the sensitivity of different algorithms to the imaging noise, the algorithms of BLS and PCAXK are significantly affected by imaging noise, as shown in Table 3 and Table 4. The standard deviations of their RMSEs in raw response reach 0.0136 and 0.0178, and in ISP response reach 0.0143 and 0.0140, which are significantly larger than the other algorithms. This is because the two algorithms use shallow features (such as the basis vector coefficients of PCA, and the augmented features and enhanced features of BLS) as the intermediate parameters to realize the spectral reconstruction through least-square regression, which leads to the errors introduced into the intermediate parameters being further transferred into the final reconstructed spectral response. Additionally, the greater the imaging noise, the more obviously the algorithm will be affected by the noise.

In addition, the sensitivity of the SRLLA algorithm to imaging noise in raw format (with a standard deviation of 0.0094) in the spectral reconstruction based on locally linear approximation is more sensitive to imaging noise than the regression-based algorithms. The sensitivity of the Li algorithm to imaging noise in ISP format (with a very big standard deviation of 0.0463) is because the spectral reconstruction based on thin-plate spline radial basis function (TPS-RBF) interpolation has an over-fitting problem for the gamma-transformed imaging noise data, and the higher the noise level, the more serious the over-fitting problem, resulting in a significant increase in the spectral reconstruction error.

In summary, except for a few algorithms significantly affected by noise as described above, the spectral reconstruction accuracy of the remaining algorithms at different noise levels has basically kept the same ranking as in the Section 5.1. When facing spectral reconstruction applications where there may be large imaging noise, it would better avoid using the above few algorithms that are sensitive to imaging noise. The one-way ANOVA and Tukey’s post hoc tests with an acceptable confidence level of α = 0.05 also confirm the above results and conclusions.

5.3. Spectra Type Adaptability

The spectra type adaptability experiment includes two groups. In the first group, the smooth spectral reflectance is used to reconstruct itself and non-smooth spectral radiance. In the second group, the non-smooth spectral radiance is used to reconstruct itself and smooth spectral reflectance. Figure 8 shows the RMSE and ΔE00 of using smooth spectral reflectance to reconstruct the non-smooth spectral radiance (S-R) and smooth spectral reflectance (S-S).

From Figure 8a,c, most spectral reconstruction algorithms show slightly lower RMSE in S-R than in S-S, but no significant difference in the ∆E00, as well as for the spectral reconstruction errors in ISP response (Figure 8c,d). Except for several algorithms that are not well adapted to the ISP responses (such as PCA, Shen, CpS, and CS, as confirmed in the Section 5.1), the other algorithms all show no significant difference between S-R and S-S in both the RMSE and ∆E00. The detailed experimental results are shown in

Table 5. The mean and standard deviation spectral reconstruction error of RMSE and ΔE00 of different algorithms when using smooth spectral reflectance to reconstruct itself (S-S) and non-smooth spectral radiance (S-R).

Algorithm	Raw				ISP
	RMSE		ΔE00		RMSE		ΔE00
	S-R	S-S	S-R	S-S	S-R	S-S	S-R	S-S
Liang	0.0142/0.0086	0.0161/0.0142	0.5095/0.2779	0.4533/0.4220	0.0135/0.0082	0.0169/0.0156	0.5200/0.2822	0.4747/0.4164
Kernel	0.0150/0.0077	0.0163/0.0105	0.6028/0.2172	0.5259/0.4486	0.0140/0.0072	0.0165/0.0122	0.6681/0.2302	0.5038/0.4499
SRLLA	0.0187/0.0156	0.0177/0.0249	0.8300/0.5844	0.4898/0.7607	0.0184/0.0163	0.0185/0.0164	2.2889/2.2463	1.0335/1.0065
wPCA	0.0134/0.0084	0.0196/0.0114	1.0626/0.7634	0.8527/1.0196	0.0122/0.0074	0.0206/0.0128	1.6587/1.0666	1.6219/1.1500
Shen	0.0168/0.0158	0.0199/0.0167	0.6316/0.3412	0.5067/0.4567	0.2041/0.02351	0.0506/0.0296	28.3724/28.7056	4.6947/4.4730
BLS	0.0150/0.0080	0.0208/0.0188	0.5448/0.2312	0.5580/0.5484	0.0160/0.0093	0.0190/0.0155	0.8052/0.3864	0.5028/0.3780
Zhang	0.0133/0.0078	0.0217/0.0159	0.4691/0.2383	0.5713/0.4652	0.0126/0.0065	0.0210/0.0146	0.4989/0.2448	0.5337/0.4166
PLS	0.0131/0.0074	0.0223/0.0141	0.5098/0.2458	0.5823/0.4087	0.0125/0.0061	0.0188/0.0139	0.5094/0.1926	0.5354/0.3942
RLS	0.0135/0.0074	0.0224/0.0138	0.9290/0.2936	0.7973/0.5066	0.0127/0.0063	0.0189/0.0136	1.0141/0.3668	0.7694/0.5757
RBFSR	0.0117/0.0060	0.0268/0.0166	0.4955/0.2316	0.8417/0.5434	0.0129/0.0064	0.0225/0.0159	0.5210/0.2341	0.6044/0.4458
Li	0.0199/0.0080	0.0301/0.0173	0.9593/0.2984	1.0826/0.5605	0.0241/0.0070	0.0309/0.0153	4.5061/2.0587	2.7319/1.8303
Cao	0.0215/0.0142	0.0314/0.0202	6.7836/4.2144	4.7830/3.6885	0.0206/0.0145	0.0317/0.0183	6.0327/3.4240	4.6750/2.9614
PCA	0.0110/0.0056	0.0354/0.0209	0.5292/0.5096	1.1330/1.3035	0.1381/0.0045	0.1102/0.0442	17.2526/2.1538	11.302/5.6128
SCO	0.0227/0.0127	0.0374/0.0278	1.5759/0.6375	1.1062/0.7642	0.0254/0.0145	0.0319/0.0263	1.7261/1.2408	0.9277/0.6102
CpS	0.0118/0.0053	0.0384/0.0225	0.8151/0.3612	1.5429/0.9795	0.1593/0.0071	0.1316/0.0463	19.8245/2.5185	12.8078/6.8725
SC	0.0126/0.0064	0.0443/0.0284	0.8596/0.4153	1.5710/1.0555	0.2187/0.0230	0.2180/0.0941	25.0608/4.6594	18.661/10.1245
PCAKX	0.0540/0.0172	0.0501/0.0266	14.5312/6.4434	9.4742/9.6886	0.0282/0.0114	0.0374/0.0182	9.0617/4.5841	6.9036/6.7619

.

Based on the above experimental results, it is reasonable to use the smooth spectral reflectance to reconstruct itself and non-smooth spectral radiance in both raw and ISP format responses, which have illustrated the strong generalization performance of the reconstruction model. This is because, for both smooth spectral reflectances and non-smoothed spectral radiances containing light source information, their spectral profiles in the frequency domain are statistically determined by the low-frequency components. Although the non-smoothed spectral radiance has a certain level of high-frequency components, the reconstruction model constructed using smooth spectral reflectance will not fit these high-frequency features, so it can better reconstruct the overall profile of smooth spectral reflectance and non-smooth radiance in both raw and ISP responses. Therefore, the spectral reconstruction errors of S-R and S-S have no apparent difference in both responses. This reason also explains why in the Section 5.2 the spectral reconstruction based on locally linear approximation is more sensitive to imaging noise than the regression-based algorithms.

Figure 9 illustrates the RMSE and ∆E00 when using non-smooth spectral radiance to reconstruct itself (R-R) and smooth spectral reflectance (R-S). Due to the extremely high RMSE of the BLS algorithm for R-S, the corresponding result is excluded in Figure 9a,b. The detailed experimental results are shown in

Table 6. The mean and standard deviation spectral reconstruction error of RMSE and ΔE00 of different algorithms when using non-smooth spectral radiance to reconstruct itself (R-R) and smooth spectral reflectance (R-S).

Algorithm	Raw				ISP
	RMSE		ΔE00		RMSE		ΔE00
	R-R	R-S	R-R	R-S	R-R	R-S	R-R	R-S
Shen	0.0043/0.0030	0.0581/0.0325	0.2524/0.2024	1.6508/1.0985	0.0088/0.0052	0.1495/0.1289	1.2326/0.8177	11.1910/7.3815
PCA	0.0080/0.0036	0.0562/0.0288	0.8476/0.6312	2.2575/1.9007	0.0167/0.0098	0.1573/0.1314	2.9038/1.3914	13.2569/7.9850
Li	0.0080/0.0061	0.0781/0.0471	1.0811/2.4812	3.2295/4.6032	0.0067/0.0045	0.1096/0.0824	7.9234/1.6756	11.8920/3.9810
SC	0.0125/0.0069	0.0847/0.0485	1.7065/0.8522	2.6752/2.8769	0.0481/0.0071	0.1438/0.1028	3.2144/1.3686	11.8977/5.7645
CpS	0.0164/0.0024	0.0816/0.0446	0.3332/0.9694	4.1274/2.1465	0.0252/0.0071	0.1574/0.1271	0.2943/1.3686	1.4073/6.5606
Kernel	0.0052/0.0038	0.1312/0.1815	0.3541/0.2444	7.7535/15.2307	0.0045/0.0031	0.1134/0.1531	0.3054/0.2249	5.5486/7.9424
RBFSR	0.0049/0.0034	0.1608/0.1339	0.5262/0.2719	9.1925/2.7890	0.0045/0.0031	0.0732/0.0431	0.4517/0.2361	3.3014/0.8024
Cao	0.0051/0.0027	0.1871/0.1545	3.2106/1.4715	9.1216/10.3654	0.0038/0.0021	0.1871/0.1544	2.1443/1.0669	9.6109/10.1545
SRLLA	0.0023/0.0034	0.2325/0.2578	0.1193/0.1487	21.5027/2.8165	0.0022/0.0028	0.2010/0.2227	0.1408/0.1602	20.3453/4.2213
wPCA	0.0031/0.0016	0.8111/1.4455	2.1643/0.5431	21.5360/46.6106	0.0024/0.0013	0.1884/0.1991	1.5363/0.4451	21.3644/20.3502
SCO	0.0066/0.0051	0.9293/2.4577	0.2292/0.4097	26.6122/11.0103	0.0054/0.0050	0.1488/0.2158	0.2151/0.3976	10.4624/2.6748
PCAKX	0.0061/0.0035	1.4398/2.1502	0.0436/1.5763	48.6423/40.2874	0.0054/0.0031	0.3374/0.3021	0.0431/1.2455	17.0377/15.0997
RLS	0.0039/0.0024	1.5466/2.4229	0.2250/0.2926	49.8403/43.9839	0.0037/0.0022	0.3320/0.2983	0.2220/0.3199	17.2291/23.6403
PLS	0.0038/0.0025	1.6266/2.7659	0.4560/0.1636	51.1287/49.0977	0.0036/0.0023	0.3466/0.3572	0.4793/0.1480	24.0252/19.5197
Zhang	0.0037/0.0024	1.7099/4.5095	1.9687/0.1561	56.0902/31.8065	0.0036/0.0022	0.2504/0.2159	1.7564/0.1489	27.2713/8.9753
Liang	0.0009/0.0008	1.7199/3.0538	0.8686/0.0008	58.6871/47.8639	0.0009/0.0007	0.4209/0.4721	0.7258/0.0351	25.3280/18.5131
BLS	0.0027/0.0018	25.2453/39.8369	0.1757/0.1226	78.2101/56.2832	0.0023/0.0015	9.7238/19.8966	0.1529/0.0941	44.4552/33.1752

.

It is easy to find out from Figure 9 and Table 6 that when using non-smooth spectral radiance to reconstruct itself (R-R) and smooth spectral reflectance (R-S), no matter which algorithms are used for spectral reconstruction, the spectral reconstruction errors of R-S are always much larger than that of R-R in both format responses. The RMSE of R-R in both format responses is mostly below 0.01, while the results of R-S are all higher than 0.05. The ∆E00 of R-S is also significantly higher than that of R-R in both responses. Therefore, it is not reasonable to use non-smooth spectral radiance to reconstruct the spectral reflectance, which creates a challenge for current studies in spectral reconstruction based on spectral radiance. Likewise, the reason behind the spectral reconstruction results in Figure 9 and Table 6 is that when using non-smooth spectral radiance as training samples, the spectral reconstruction model will over-fit its high-frequency components, resulting in a large difference between the solution space and the smooth spectral reflectance, and the reconstruction results will deviate significantly from the ground truth, thereby greatly increasing the RMSE and ΔE00. So, the spectral reconstruction error of R-S is much higher than that of R-R. The one-way ANOVA and Tukey’s post hoc tests, conducted at a confidence level of α = 0.05, further support the above results and conclusions.

5.4. Exposure Change Adaptability

The spectral reconstruction error of RMSE for exposure adaptability is presented in Figure 10. Figure 10c shows the results in Figure 10a with the BLS algorithm removed, and Figure 10d displays the results in Figure 10b after both the BLS and RBFSR algorithms are removed. The horizontal axis represents various testing exposure levels, with the training exposure level set at 1.0. Detailed experimental errors of RMSE can be found in

Table 7. The mean and standard deviation of the RMSE of different algorithms in different exposure levels in raw response. The exposure adaptability index S of different algorithms is also reported.

Algorithm	ExposureLevels					S
	0.5	0.75	1	1.5	2
PCA	0.0354/0.0208	0.0354/0.0208	0.0354/0.0208	0.0354/0.0208	0.0354/0.0208	0.0000
SC	0.0443/0.0283	0.0443/0.0284	0.0443/0.0284	0.0443/0.0284	0.0443/0.0284	0.0000
CpS	0.0384/0.0222	0.0384/0.0224	0.0384/0.0225	0.0384/0.0225	0.0384/0.0225	0.0000
Li	0.0359/0.0209	0.0309/0.0153	0.0301/0.0172	0.0348/0.0206	0.0385/0.0228	0.0197
Shen	0.0344/0.0331	0.0260/0.0210	0.0199/0.0166	0.0250/0.0175	0.0263/0.0185	0.0321
wPCA	0.0349/0.0214	0.0225/0.0145	0.0196/0.0114	0.0351/0.0233	0.0558/0.0494	0.0699
SCO	0.0565/0.0441	0.0394/0.0308	0.0374/0.0277	0.0442/0.0392	0.0907/0.1341	0.0812
PLS	0.0358/0.0232	0.0276/0.0176	0.0223/0.0141	0.0406/0.0371	0.0981/0.1034	0.1129
RLS	0.0361/0.0231	0.0278/0.0173	0.0224/0.0137	0.0411/0.0377	0.0993/0.1054	0.1147
Zhang	0.0371/0.0245	0.0277/0.0174	0.0217/0.0158	0.0489/0.0518	0.0980/0.1117	0.1249
Liang	0.0362/0.0265	0.0185/0.0138	0.0161/0.0141	0.0472/0.0462	0.1089/0.1302	0.1464
SRLLA	0.0545/0.0548	0.0302/0.0293	0.0177/0.0249	0.0546/0.1646	0.0946/0.2714	0.1631
Cao	0.0625/0.0397	0.0419/0.0304	0.0314/0.0201	0.0773/0.0711	0.1129/0.1024	0.1690
PCAKX	0.1155/0.0673	0.0705/0.0421	0.0501/0.0266	0.0706/0.0659	0.1543/0.1684	0.2105
Kernel	0.0430/0.0297	0.0246/0.0154	0.0163/0.0104	0.0796/0.1537	0.1363/0.1982	0.2183
RBFSR	0.1599/0.0975	0.0860/0.0499	0.0268/0.0166	0.1840/0.1401	0.4600/0.4931	0.7827
BLS	0.0414/0.0256	0.0259/0.0172	0.0208/0.0162	8.9787/44.7085	18.8984/47.7125	27.8612

and

Table 8. The mean and standard deviation RMSE of different algorithms in different exposure levels in ISP response. The exposure adaptability index S of different algorithms is also reported.

Algorithm	ExposureLevels					S
	0.5	0.75	1	1.5	2
PCA	0.2433/0.0531	0.149/0.0429	0.1102/0.0441	0.1011/0.0803	0.1188/0.1133	0.1714
SC	0.4350/0.1314	0.2930/0.1106	0.2180/0.0941	0.1608/0.0661	0.1430/0.0763	0.1598
CpS	0.2923/0.0584	0.1817/0.0547	0.1316/0.0462	0.1066/0.0706	0.1177/0.1039	0.1719
Li	0.0445/0.0212	0.032/0.0133	0.0309/0.0153	0.0489/0.0507	0.0733/0.0868	0.0751
Shen	0.1148/0.0436	0.0748/0.0393	0.0506/0.0296	0.0968/0.0673	0.0968/0.1016	0.1808
wPCA	0.0293/0.0187	0.0204/0.0127	0.0206/0.0128	0.0391/0.0483	0.0629/0.0846	0.0693
SCO	0.0411/0.0275	0.0325/0.0249	0.0319/0.0263	0.0419/0.0502	0.0623/0.0895	0.0502
PLS	0.0334/0.0228	0.0224/0.0151	0.0188/0.3942	0.0336/0.0488	0.0610/0.0856	0.0752
RLS	0.0348/0.0223	0.0228/0.0145	0.0189/0.0135	0.0337/0.0490	0.0611/0.0858	0.0768
Zhang	0.0324/0.0237	0.0219/0.0148	0.0210/0.0145	0.0354/0.0489	0.0661/0.0903	0.0718
Liang	0.0276/0.0236	0.0160/0.0128	0.0169/0.0156	0.0385/0.0499	0.0701/0.0903	0.0846
SRLLA	0.0432/0.0376	0.0381/0.0411	0.0185/0.0163	0.0474/0.0609	0.0711/0.0904	0.1258
Cao	0.0548/0.0372	0.0377/0.0246	0.0317/0.0183	0.0807/0.0727	0.1155/0.1043	0.1619
PCAKX	0.0678/0.0436	0.0481/0.0259	0.0374/0.0182	0.0446/0.0468	0.0701/0.0869	0.0810
Kernel	0.0393/0.0263	0.0209/0.0123	0.0165/0.0122	0.0459/0.0548	0.0792/0.0953	0.1193
RBFSR	0.2462/0.1516	0.1506/0.0918	0.0225/0.0501	0.4303/0.2338	1.0459/0.5179	1.7830
BLS	0.0325/0.0212	0.0224/0.0154	0.0190/0.0159	1.9025/8.5722	3.1017/8.6001	4.9831

. Since the results of ∆E00 are similar to those of RMSE, they are not presented separately. In addition, to qualitatively evaluate the sensitivity of each algorithm to the exposure changing, an exposure adaptability index S was defined, as shown in Equation (8):

$$S={\displaystyle \sum \left|\mathit{RMS}{E}_k-\mathit{RMS}{E}_{1.0}\right|}(k=0.5,0.75,1.5,2.0)$$

(8)

where RMSE_k denotes the RMSE of an algorithm that uses exposure level 1.0 as a training exposure level to reconstruct the spectral reflectance at the testing exposure level of k. For example, RMSE_0.5 denotes the RMSE of an algorithm uses the training exposure level of 1.0 to reconstruct the spectral reflectance at the testing exposure level of 0.5. The exposure adaptability index S of different algorithms is also calculated and reported in Table 7 and Table 8.

From Figure 10a,c, and Table 7, in raw response, it is clear that except for three entirely linear algorithms (PCA, SC, and CpS) that are not sensitive to exposure changes, all the other tested algorithms are sensitive to exposure changes to varying degrees, but the spectral reconstruction accuracy of the PCA, SC, and CpS algorithms is relatively low (refer Table 2). Although Shen, SRLLA, and Cao are also linear regression or mapping algorithms, they are all implemented in a local-training or locally weighted-training form, making the regression or weighting coefficients incorrectly calculated between the training and testing exposure levels. Therefore, when the incorrect regression or weighting coefficients in a low-dimensional response space are linearly mapped to the high-dimensional spectral space, the spectral reflectance of the testing samples is incorrectly reconstructed. For the rest of the algorithms, as they are all implemented based on non-linear regression or mapping, the linear changing of exposure level will not be linear to the changing of the spectral profile, leading to the profile of the reconstructed spectral reflectance deviating from the ground truth.

From Figure 10b,d and Table 8, it is obvious that for the ISP response, all the linear- and non-linear-regression- or mapping-based algorithms are sensitive to exposure changes to varying degrees. This phenomenon depends not only on whether the algorithm itself is linear or non-linear, but also on the difference between the ISP and raw response formats. Since the ISP response is obtained through non-linear processes from the linear raw response, a linear change in the raw response always leads to non-linear change in the ISP response. Therefore, no matter which kind of algorithm is used for spectral reconstruction, they cannot fully adapt to non-linear exposure changes in the ISP response.

In addition, the exposure adaptability index S reflects the adaptability of each algorithm to exposure changes. The larger the value of S, the lower the adaptability of the algorithm to exposure changes and the larger the spectral reconstruction error caused by the exposure difference. Therefore, it is easy to infer from Table 7 and Table 8 that, generally, the linear algorithms are more adapted to exposure changes in the raw response than the non-linear algorithms, and the non-linear algorithms are more adapted to exposure changes in the ISP response than the linear algorithms. The BLS algorithms show extremely poor adaptability to exposure-level changes in both response formats, indicating that its feature calculation module is very sensitive to exposure changes, especially for the conditions when the testing exposure level is higher than the training exposure level. In summary, as discussed in the introduction, although some research has been carried out for the development of exposure-invariant spectral reconstruction algorithms, after testing these algorithms still have performance gaps for practical use. The validity of the analysis was also confirmed by statistical results obtained using one-way ANOVA and Tukey’s post hoc tests at an acceptable confidence level of α = 0.05.

5.5. Reconstructed Image Quality

The ColorChecker SG140 color chart (USA) was used as the training sample to reconstruct 18 multispectral images (as illustrated in Figure 5) selected from the FOSTER, CAVE, and ARAD-1K datasets. The reconstructed spectral images were rendered into sRGB images using the CIE D65 illuminant and the CIE1964 standard observer color-matching functions. The distribution of PSNR and SSIM of the rendered images between the ground-truth and reconstructed images are plotted in Figure 11 and the results are summarized in

Table 9. The mean and standard deviation PSNR and SSIM between ground-truth and reconstructed images by different algorithms.

Algorithm	Raw		ISP
	PSNR	SSIM	PSNR	SSIM
PCAKX	26.6278/3.9749	0.7601/0.1838	24.9300/3.3271	0.8438/0.1575
Shen	28.8704/5.1647	0.9664/0.0128	22.1256/3.5908	0.8853/0.0793
Cao	29.0737/5.1818	0.9270/0.0272	28.0368/2.4314	0.9208/0.0556
wPCA	39.7596/8.5521	0.9920/0.0085	37.4420/6.3432	0.9734/0.0423
PCA	51.5944/9.3514	0.9990/0.0012	16.3155/2.3194	0.8476/0.0782
RLS	51.8559/5.6922	0.9966/0.0027	45.1519/6.1453	0.9842/0.0191
CpS	52.9168/0.2438	0.9986/0.0011	15.4354/1.9688	0.8463/0.0863
SCO	54.2575/6.4555	0.9997/0.0002	43.0375/6.1595	0.9792/0.0376
Kernel	54.2654/17.7578	0.9976/0.0092	56.5979/10.4821	0.9938/0.0162
Li	55.6808/6.1427	0.9997/0.0003	33.4219/5.6659	0.9350/0.0662
BLS	61.0159/30.7393	0.9976/0.0111	49.0386/12.2053	0.9897/0.0224
RBFSR	64.6144/8.0185	0.9999/0.0000	12.0063/8.6015	0.8068/0.0306
SC	70.9589/7.3311	0.9999/0.0002	17.2521/1.4070	0.7946/0.1064
Zhang	82.3579/27.1771	1.0000/0.0000	69.2845/12.1513	0.9991/0.0024
PLS	251.0000/11.7992	1.0000/0.0000	61.8000/10.7390	0.9980/0.0045
SRLLA	275.5787/13.4710	1.0000/0.0000	35.2972/2.6972	0.9736/0.0285
Liang	282.3722/9.3185	1.0000/0.0000	78.0542/16.1780	0.9994/0.0017

.

The quality of the reconstructed image of each algorithm is closely related to their spectral reconstruction accuracy (see Table 2). Overall, the higher the spectral reconstruction accuracy, the better the reconstructed image quality is. Firstly, from Figure 11 and Table 9, except for the PCAXK algorithm, the other algorithms all show better PSNR and SSIM results in raw response than in ISP response. This is similar to the conclusion in the Section 5.1, which states that spectral reconstruction based on raw response is recommended. Secondly, the sensitivity of some algorithms to the response format is still apparent, such as PCAXK, Shen, PCA, CpS, and SC. In addition, the Li, RBFSR, PLS, SRLLA, and Liang algorithms also show some sensitivity to the response format in this experiment. However, under the raw response, they all show better results than most other algorithms.

Moreover, it is not difficult to find from Table 2 and Table 9 that several algorithms that performed well in the Section 5.1, such as Liang, Kernel, PLS, Zhang, BLS, etc., all achieved good results when testing the quality of reconstructed images. Among them, the Liang algorithm shows an RMSE of 0.0161 and 0.0169 in raw and ISP responses in the Section 5.1, and gets the best PSNR and SSIM in both format responses for the reconstructed image quality tests, indicating the advantages in spectral reconstruction. However, as described in the Imaging Noise Adaptability Section and the Exposure Change Adaptability Section, it has a certain sensitivity to noise and exposure changes in raw response, which restricts its application in open environments. This limitation also occurs in other algorithms mentioned above that have given relatively good testing performance. At an acceptable confidence level of α = 0.05, the statistical results of the one-way ANOVA and Tukey’s post hoc test also showed significant differences in the statistical means. Therefore, the application limitations of the existing spectral reconstruction algorithms should be carefully considered and addressed in the future.

6. Discussions

In the above experiments, a comparative analysis of seventeen machine learning-based spectral reconstruction algorithms was conducted in five aspects: response-format adaptability, imaging noise adaptability, spectra type adaptability, exposure change adaptability, and reconstructed image quality. The aim of this study is to test the robustness of current machine learning-based spectral reconstruction algorithms to different influence factors, to provide an overview to the researchers in this field, and at the same time, to present the challenges faced by spectral reconstruction technology in open-environment applications.

To provide a more intuitive and comprehensive inspection of the experimental results, the adaptability of different algorithms to different influencing factors and the quality of the reconstructed images are summarized in

Table 10. The adaptability of different algorithms to different influence factors and the quality of reconstructed images; the best five performance algorithms for each aspect are marked with numbers.

Algorithm	Response-Format Adaptability		Imaging Noise Adaptability		Spectral Type Adaptability		Exposure Change Adaptability		Reconstructed Image Quality
	Raw	ISP	Raw	ISP	S-R	R-S	Raw	ISP	Raw	ISP
RLS	★	★ 4	★ 3	★	★ 4				★	★
Kernel	★ 2	★ 1	★ 1	★ 1	★				★	★ 4
PLS	★	★ 3	★ 2	★	★ 2				★ 3	★ 3
PCA	★		★		★		★ 1		★
PCAKX		★
BLS	★ 5	★ 4			★				★	★ 5
CpS	★		★		★		★ 2		★
SC	★		★		★		★ 3		★ 5
RBFSR	★	★	★	★ 5	★ 1
SCO	★	★	★	★	★				★	★
Li	★	★	★		★				★
wPCA	★ 3	★	★ 4	★ 4	★ 2				★	★
SRLLA	★	★			★				★ 2	★
Zhang	★	★	★	★ 3	★ 3				★ 4	★ 2
Liang	★ 1	★ 2	★ 5	★ 2	★ 5				★ 1	★ 1
Cao	★	★	★	★	★				★	★
Shen	★ 4	★	★	★	★				★

. The cells marked with ‘★’ mean the algorithm is adapts well to changes in this influence factor, and the null cells mean the algorithm does not adapt well to changes in this influence factor or the spectral reconstruction error is too big. The best five performance algorithms in terms of the spectral reconstruction error of RMSE in each aspect and in raw and ISP response are marked with numbers. Together with the category of each algorithm and their key idea, input, complexity, and pros/cons, as well as the experimental results of each part, an overall discussion is given as follows.

When testing the impact of camera response formats on spectral reconstruction, most algorithms show better spectral reconstruction accuracy in raw than in ISP response. This aligns closely with the assumption that spectral reconstruction is based on a linear imaging model. The algorithms of local- and weighted-training-based regression algorithms generally give better spectral reconstruction than the mapping-based algorithms. The reason behind this has been discussed in the Section 5.1, and the problem of these outperformed algorithms lies in their complexity, which decreases the computational efficiency. Some optimization measures such as image segmentation and parallel computing can be used to overcome this problem. In addition, given that linear raw responses are easy to obtain currently, it is recommended that future research on spectral reconstruction technology focus on raw response.

For the testing of imaging noise adaptability, although a few algorithms are greatly affected by noise changes (such as the PCAKX and BLS algorithm in raw response and the Li algorithm in ISP response), the spectral reconstruction errors of most algorithms, no matter what kind of mathematical principal and basis is behind them, show a steady increase as the noise increases. This is consistent with expectations that high imaging noise will degrade the accuracy of the regression models, interpolation tables, and dictionary representations, and decrease the spectral reconstruction accuracy.

In terms of spectra type adaptability, except for PCAXK, all of the other algorithms are adapted to using smooth spectral reflectance to reconstruct non-smooth spectral radiance (S-R), but no one algorithm is adapted to using non-smooth spectra radiance to reconstruct smooth spectral reflectance (R-S). As is explained in the Section 5.3, this is because the spectral characteristics in the frequency domain for smooth spectral reflectance and non-smoothed spectral radiance are low-frequency components. Non-smoothed spectral radiance does have some high-frequency components, but the spectral reconstruction model based on smooth spectral reflectance will not capture the high-frequency details. Therefore, it works well for both itself and non-smooth radiance. However, the spectral reconstruction model based on non-smooth spectra radiance will capture more details of the high-frequency components and fewer details of low-frequency components, and make the model not suitable for the smooth spectral reflectance.

For the most important influencing factor, exposure change, only three completely linear algorithms can adapt to exposure changes in raw response, but with relatively low spectral reconstruction accuracy. All other algorithms are not adaptable to exposure changes in both raw and ISP responses. In an open environment, uneven lighting results in inconsistent exposure in different areas of the same scene, this brings a huge challenge for current spectral reconstruction methods as the outstanding algorithms are all sensitive to exposure changes. Although theoretical studies on spectral reconstruction algorithms for open environments have been conducted, a gap still exists between these theoretical results and their practical applications. Given the increasing demand for spectral reconstruction technology in open environments, it is essential to develop an optimized algorithm that can adapt to changes in exposure for practical use.

The comparison of the quality of reconstructed images reveals that the tested spectral reconstruction algorithms demonstrate superior spectral reconstruction accuracy when using the raw response. As a result, the quality of the sRGB images reconstructed and rendered from the raw response is better than that obtained from the ISP response. This indicates that research on spectral reconstruction should prioritize using the linear raw response.

Finally, from the overall experimental results shown in Table 5, some valuable and instructive conclusions can be obtained from the comparative study, such as: (1) the linear raw response/image should preferred for spectral reconstruction; (2) under the condition that the spectral reconstruction imaging conditions are controllable, some optimized spectral reconstruction algorithms should be selected as much as possible, such as the Liang, Kernel, PLS, RLS, Zhang, and wPCA algorithms; (3) when performing spectral reconstruction in an open environment, a reconstruction algorithm that adapts to exposure changes should be used to ensure the accuracy of the reconstructed spectral profile. However, the spectral reconstruction accuracy of several existing spectral reconstruction algorithms (such as PCA, CpS, and SC) that are adaptable to exposure changes in raw response is too poor. The one-way ANOVA and Tukey’s post hoc test at the acceptable confidence level of α = 0.05 have also validated the analysis and conclusions of this comparative study. In addition, in our previous work [38], we investigated the application of deep learning spectral reconstruction algorithms in open environments and proposed strategies to enhance the model’s adaptability under such conditions. In the future, we will further investigate deep learning-based spectral reconstruction algorithms and conduct specialized exploratory research.

7. Conclusions

In this study, the performance of various machine learning-based spectral reconstruction algorithms was examined, focusing on their adaptability to response formats, imaging noise, types of spectra, changes in exposure, and the quality of the reconstructed images. Qualitative analysis and quantitative statistical comparison of the experimental results were performed based on the principles and characteristics of the algorithm. The experiments demonstrated that most of the current algorithms: (1) are adaptive to raw and ISP responses for spectral reconstruction, (2) decrease their spectral reconstruction accuracy with an increase in the imaging noise, (3) give poor performance in reconstructing smooth spectra using non-smooth spectra, (4) exhibit a different degree of sensitivity to exposure changes, and (5) have a reconstructed image quality superior to the ISP response when using the raw response. Since this comparative study is mainly based on simulation experiments, it is inevitable that there will be some deviation from the real situation, but it has certain reference value for carrying out research on spectral reconstruction. Furthermore, as almost all the tested spectral reconstruction algorithms are exposure-sensitive, the development of advanced algorithms that can deal with uneven exposure in an open environment is promising to advance spectral reconstruction technology from the laboratory to natural scenes.