Fusion Recalibration Method for Addressing Multiplicative and Additive Effects and Peak Shifts in Analytical Chemistry

Abstract

Analytical chemistry relies on the qualitative and quantitative analysis of multivariate data obtained from various measurement techniques. However, artifacts such as missing data, noise, multiplicative and additive effects, and peak shifts can adversely affect the accuracy of chemical measurements. To ensure the validity and accuracy of results, it is crucial to preprocess the data and correct for these artifacts. This paper proposes a fusion recalibration algorithm, called Spectral Offset Recalibration (SOR), that combines the Extended Multiplicative Signal Correction (EMSC) and Correlation-Optimized Warping (COW) algorithms to address both multiplicative and additive effects and peak shifts. The algorithm incorporates prior spectroscopic knowledge to down-weight or disregard spectral regions with strong absorption or significant distortion caused by peak alignment algorithms. Experimental validation on wood NIR datasets and simulated datasets demonstrates the effectiveness of the proposed method. The fusion recalibration approach offers a comprehensive solution for accurate analyses and predictions in analytical chemistry by mitigating the impact of artifacts.

1. Introduction

From the perspective of analytical chemistry, the core content of qualitative and quantitative analysis in chemometrics is the rational and effective use of multivariate data [1,2], which are obtained from a variety of measurement techniques [3]. These techniques can range from small, portable near-infrared (NIR) spectrometers to complex and expensive techniques such as liquid or gas chromatography-mass spectrometry (LC-MS or GC-MS), or even multi-dimensional implementations like GCxGC/MS [4,5]. However, one common problem that arises from these techniques is the presence of artifacts, or unwanted variations, caused by factors such as the measurement modality, instrumental drifts, sample state, and external physical, chemical, and environmental factors.

Correcting for these artifacts is beneficial for accurate analyses and predictions in analytical chemistry. With the advancements in chemometric research, a variety of preprocessing techniques are now available to address these issues. It is crucial for researchers to identify and correct these artifacts through data preprocessing technology before building models or making predictions in order to ensure the validity and accuracy of the results obtained.

Artifacts in chemical measurement data can be broadly categorized into four main categories: missing data, noise, multiplicative and additive effects [6], and peak shifts [7]. Each type of artifact, as seen in Figure 1, has its own specific causes, which can range from human error during measurements to complex interactions between light and the physical structure of the sample. Among them, missing data and noise are isolated events that can be solved by only requiring the individual signal to be pretreated. However, peak shift [8] and multiplicative and additive effects [9] can interact with each other, making them extremely difficult to solve when both issues appear in the spectrum at the same time. In the remainder of this section, the causes and effects of peak shifts and multiplicative and additive effects will be discussed in brief.

Spectral data in the misaligned and multiplicative and additive effects categories are major problems that can negatively impact the accuracy of the data obtained from analytical techniques. To solve these issues and other artifacts, alignment and multiplicative and additive effects corrections are commonly used data preprocessing techniques.

In the realm of diffuse reflectance spectroscopy, the challenge lies in accurately characterizing sample components amidst the significant spectral variations introduced by factors like particle size distribution and light scattering effects [10]. Notably, the influence of scattering on spectra can often overshadow the spectral changes attributed to the sample’s inherent composition. The degree of scattering is intricately linked with variables such as the wavelength of incident light, particle size distribution, and refractive indices [11]. Adhering to the principles of the Beer–Lambert law, alterations in the optical path length within near-infrared spectra yield an overarching scaling factor for the entire absorption spectrum [12]. This phenomenon gives rise to observable shifts, rotations, and secondary or higher-order curvature alterations in the baseline, a phenomenon collectively known as additive and multiplicative effects in spectra. Consequently, the meticulous elimination of these artifacts proves pivotal in ensuring the attainment of precise and dependable analytical outcomes.

For the problem of multiplicative effects, various techniques have been developed to address this issue, such as multiplicative signal correction (MSC) and piecewise multiplicative signal correction, as well as the standard normal variate (SNV) method [13,14]. These methods have been widely adopted in many applications, likely due to their incorporation into standard software systems for multivariate calibration. These data transformation techniques share the characteristic of being relatively straightforward to implement.

In the construction of chemometric models, a crucial prerequisite involves inputting a set of data with consistent wavelengths and uniform peak/trough positions. This implies that all signals must possess identical lengths and their spectral trends should align. However, data acquired through the integration of near-infrared spectroscopy and chromatography-mass spectrometry techniques often struggle to meet this requirement. Similar features in acquired spectral or chromatographic data frequently emerge at different positions along the frequency or retention time axis. This phenomenon is referred to as peak shift. Peak shifts are commonplace in various techniques such as spectroscopy, nuclear magnetic resonance, and mass spectrometry. They can arise due to variations in instrument mechanics, electronic noise, temperature, pressure, viscosity, and pH values [15,16]. According to the research by Jeffrey J and colleagues, a systematic shift in absorbance towards shorter wavelengths is observed in spectra as temperature increases. This transition could potentially be attributed to a reduction in hydrogen bonding with elevated temperatures [17]. The presence of peak shifts introduces additional complexity to data analysis and modeling, thereby constituting another critical consideration in the manipulation of spectroscopic or chromatographic data.

In the case of peak shifts, various techniques have been developed to solve this issue. A study by Khosravi et al. [18,19] evaluated the effectiveness of various alignment methods in achieving alignment objectives. The study found that the alignment methods had varying levels of efficiency in reducing positional errors [20], and the Correlation-Optimized Warping (COW) method was particularly efficient in aligning the middle of the datasets while maintaining their shape [21,22]. COW uses the correlation coefficient as the basis for stretching and alignment, using the correlation coefficient between the two corresponding segments of two NIR spectral series as the measure of alignment quality.

However, when the multiplicative effect and peak shift phenomenon occur simultaneously in the spectrum, the SNV and MSC algorithms, which use multiple linear regression tools to eliminate the multiplicative effect, are unable to address the deviation between the fitted spectral signal and the reference signal caused by the peak shift phenomenon. This shift problem is particularly pronounced at spectral absorption peaks. On the other hand, using the COW and Dynamic Time Warping (DTW) methods, which rely on the correlation coefficient tool to eliminate peak shifts in the spectrum, cannot address the warping deformation of the spectral signal caused by the multiplicative effect, leading to issues such as distortion and warping of the interpolation signal generated by the peak alignment algorithm.

The principles of the MSC and COW methods can be utilized to design a fusion method to solve the above two problems. However, in certain regions of the spectrum where strong absorption from the target analyte or chemical interferences occur, the MSC method may incorrectly attribute this to light scattering, leading to unrealistic constraints on the signal. Additionally, the use of peak alignment algorithms such as COW and DTW, which employ linear interpolation to stretch or compress each segment, can amplify issues related to MSC pretreatment. To address these limitations, we modified the algorithm using the EMSC approach developed by Martens Harald et al. [23,24]. This method incorporates prior spectroscopic knowledge of the correlation coefficient calculated by the COW algorithm to down-weight or disregard spectral regions where chemical constituents strongly absorb or where peak alignment algorithms cause significant distortion. EMSC [25] is an extended MSC algorithm that incorporates a new parameter to account for the physical and chemical phenomena that affect the measured absorbance spectra. In summary, the particular objectives were as follows:

• We have incorporated a new mode in EMSC that specifically eliminates their influence to address the distortion issue caused by the COW and DTW algorithms.
• We propose a fusion recalibration algorithm that combines the EMSC and COW algorithms to address the issues caused by both the multiplicative and additive effects and peak shift phenomenon.

This paper is organized as follows. The details of the proposed method are illustrated in Section 2. In Section 3, extensive experiments on wood NIR datasets and simulated datasets are conducted to verify the effectiveness of the preprocessing method proposed in this paper, and the results are fully discussed in Section 4. Finally, a simple conclusion is given in Section 5.

2. Theories and Methods

In the Introduction, we mentioned that when both multiplicative and additive effect and peak shift artifacts coexist in near-infrared spectra, the COW algorithm and EMSC algorithm may bring unexpected problems to near-infrared spectra. In the following section, we will provide a systematic discussion on this viewpoint by combining the principles of the COW algorithm and EMSC algorithm.

2.1. Correlation-Optimized Warping

The COW algorithm synchronizes similar features in sets of signals using the principle of dynamic programming [19,26,27]. The algorithm designates one as the target series T and the other the sample series P. The sample series P, which is to be aligned with T, has $L_P+1$ elements and total length $L_P$. The unaligned series P is divided into N segments, with length m each. Then, the number of sections N is given by $N=L_P/m$. Each segment, denoted as I, is warped (i.e., stretched or compressed) via linear interpolation so that an aligned time series A with the same length as T is obtained. When there is a difference in the section length between T and P signals, the COW algorithm also provides a method to deal with it. However, since there is almost no such problem in near-infrared spectra, we will not elaborate on this aspect.

To evaluate the quality of alignment for each section, we used the correlation coefficient $\left(\rho \right)$ between the corresponding segments of the two series T and A as the measure. Specifically, we defined the alignment quality $f\left(I\right)$, also known as the benefit function, for each segment (denoted by I) as $f\left(I\right)=\rho \left(IT,IA\right)$, where $IT$ and $IA$ represent the corresponding segments of T and A, respectively. Our objective was to determine the optimal global alignment between the entire series P and T. To achieve this, we needed to identify the best combination of warpings for all segments based on the node positions in the aligned series A. This optimization problem can be formulated as follows:

$$\mathrm{Given}:x_0=0<x_1<\cdots <x_{N-1}<x_N=L_T$$

(1)

$$u_i\in [\Delta -t,\Delta +t];i=0,\cdots ,N-1\mathrm{and}{x}_{i+1}=x_i+m+u_i;i=0,\cdots ,N-1$$

(2)

At this juncture, the warping optimization to be performed on the series can be expressed as follows: $\overline{x}=argma{x}_x\left({\sum }_{i=0}^{N-1}f\left[x_i;x_{i+1}\right]\right)$. Here, $\left[x_i;x_{i+1}\right]$ denotes an interval obtained by segmenting from point $x_i$ to point $x_{i+1}$.

The objective of the algorithm was to determine the optimal alignment quality between two sets of series composed of discrete elements. Consider each group’s arbitrary elements forming an object. This optimization problem can be formulated as follows: Find the best object from a finite set of objects with a size of ($\left(N+1\right)\times \left(L_T+1\right)$). This can be regarded as a combinatorial optimization problem, which can be addressed using dynamic programming. Let the function to be optimized be denoted as F. Initially, all elements in F correspond to negative infinity alignment quality, except for the last element $F\left(N+1,L_T+1\right)$, which is set to 0, signifying alignment between the last points of T and P. During the backward optimization process, each element in F is accumulated based on the gain function $F_{i,x}=max\left(F_{i+1,x+m}+f\left(\left[x;x+m\right]\right)\right),i=1,2,\dots ,N-1$, where $f\left(\left[x;x+m\right]\right)$ denotes the benefit function in the section defined by border points x and $x+m$. The global optimization value can be achieved at $F_{0,0}=max\left({\sum }_{i=0}^{N-1}f\left(\left|x_i;x_{i+1}\right|\right)\right)$.

2.2. Extended Multiplicative Signal Correction

This section will discuss multiplicative and additive effects, as well as how the EMSC method addresses these effects [25,28,29]. We will also delve into the details of the EMSC method [30]. Consider a set of ideal transparent solutions containing J chemical components. According to the Beer–Lambert law, the absorption spectrum of the i sample of the solution, after being influenced by the multiplicative and additive effects, can be expressed as follows:

$$X_{i,chem}=p\sum _{j=1}^J{c}_{i,j}{s}_j,i=1,2,\dots ,I$$

(3)

$$X_i=a_i{X}_{i,chem}+b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i$$

(4)

In Formula (3), the row vector $X_{i,\mathrm{chem}}$ represents the theoretical absorption spectrum of sample i, where p is related to the path length and is a fixed variable in ideal conditions. $c_{i,j}$ denotes the concentration of the j-th chemical component in the i-th sample, and the row vector $s_j$ is used to evaluate the light-absorbing ability of the j-th chemical component, which is primarily related to the type of chemical component. The deviation in the absorption spectrum is expressed in Formula (4), where $a_i$ is a multiplication coefficient associated with the path length of different mixed samples, which can “scale” the entire original spectrum and lead to a multiplication bias in the Beer–Lambert Law. $b_i$ represents the additive coefficient of the spectrum baseline, and the matrix calculation involving the vector of all ones, $1=[1,1,1,\cdots ,1]$, is used for the absorption spectrum deviation. The coefficients $w_i$ and $d_i$ represent the spectral transformation coefficients between $\lambda$ and ${\lambda }^2$. The vectors $\lambda$ and ${\lambda }^2$ denote wavelength and wavelength squared, respectively, and the variable ${\epsilon }_i$ represents random measurement noise.

Let $X_{mean}$ be the average spectrum of the NIR spectra to be processed, also known as the reference spectrum. Substituting the reference spectrum into Equations (3) and (4), then:

$$X_{i,chem}=p\times \sum _{j=1}^J{c}_{i,j}{s}_j=X_{mean}+p\times \Delta c_i,j=1,2,\dots ,J$$

(5)

$$X_i-a_i{X}_{mean}=a_{i}p\times \Delta c_i+b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i,j=1,2,\dots ,J$$

(6)

The coefficients in Equation (6) were obtained using the least squares method. Then, the corrected spectrum of the i-th sample after removing the additional coefficients was calculated and denoted as $X_{i,corrected}$, as shown in Equation (7):

$$X_{i,corrected}=\left(X_i-\left(b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i\right)\times a_i\right)/a_i$$

(7)

2.3. COW Algorithm Challenges with Multiplicative and Additive Effects in Spectral Data

In Section 2.1, it was mentioned that the COW algorithm defines the alignment quality, i.e., the gain function, of each component by calculating the summation of correlation coefficients between different spectral segments. These spectral segments are obtained by uniformly dividing the original near-infrared spectrum into m wavelength bands. The formula for the correlation coefficient can be expressed by Equation (8):

$$\rho =\frac{E\left[xy\right]}{\sqrt{E\left[x^2\right]E\left[y^2\right]}}$$

(8)

If two sets of spectra, $X_i$ and $X_j$, which are affected by multiplicative and additive effects, are input into the correlation coefficient formula, the correlation coefficient is shown below in Equation (9):

$$\begin{array}{cc}\hfill \rho & =\frac{E\left[X_i{X}_j\right]}{\sqrt{E\left[X_i^2\right]E\left[X_j^2\right]}}\hfill \\ \hfill & =\frac{E\left(a_i{X}_{i,chem}+b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i\right)E\left(a_j{X}_{j,chem}+b_{j}I+w_j\lambda +d_j{\lambda }^2+{\epsilon }_j\right)}{\sqrt{E\left[{\left(a_i{X}_{i,chem}+b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i\right)}^2\right]E\left[{\left(a_j{X}_{j,chem}+b_{j}I+w_j\lambda +d_j{\lambda }^2+{\epsilon }_j\right)}^2\right]}}\hfill \end{array}$$

(9)

If we define $intercep{t}_i=\frac{b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i}{a_i}$, then Equation (9) can be reformed as:

$$\begin{array}{cc}\hfill \rho & =\frac{E\left(a_i{X}_{i,chem}+a_i\times intercep{t}_i\right)E\left(a_j{X}_{j,chem}+a_j\times intercep{t}_j\right)}{\sqrt{E\left[{\left(a_i{X}_{i,chem}+a_i\times intercep{t}_i\right)}^2\right]E\left[{\left(a_j{X}_{j,chem}+a_j\times intercep{t}_j\right)}^2\right]}}\hfill \\ \hfill & =\frac{E\left(X_{i,chem}+intercep{t}_i\right)E\left(X_{j,chem}+intercep{t}_j\right)}{\sqrt{E\left[{\left(X_{i,chem}+intercep{t}_i\right)}^2\right]E\left[{\left(X_{j,chem}+intercep{t}_j\right)}^2\right]}}\hfill \end{array}$$

(10)

According to Formula (10), it can be inferred that the multiplicative and additive effects result in an intercept error, thereby affecting the calculation result of the correlation coefficient. When $\frac{d_i}{a_i}$ and $\frac{b_i}{a_i}$ are large, significant errors in the correlation coefficient will appear in the latter half of the near-infrared spectral band, which will affect the optimal result $F_{i,x}$ of the dynamic programming optimization algorithm and cause severe distortion of the spectral band output by the COW algorithm. Figure 2 depicts two near-infrared spectra affected by multiplicative and additive effects. From the graph, it can be observed that distortion in the latter half of the spectra significantly impacts the calculation of the correlation coefficients.

2.4. EMSC Algorithm Challenges with Peak Shifts in Spectral Data

Let us assume that the maximum peak of the spectral mean, $X_{mean}$, is located at $X_{mean,max}$ in the q-th column. Let us take an ideal spectrum, $X_i$, from the sample set. In reality, when there is an offset in the near-infrared spectrum, there exists an offset spectrum, $X_i^{\prime }$, with an offset of $\beta$, which can be represented as $\left[x_1,\dots ,x_{\beta },X_{i,\alpha }^{\prime }\right]$. The length of $X_i^{\prime }$ is the same as that of the ideal spectrum $X_i$, and $X_{i,\alpha }^{\prime }$ is the submatrix formed by the first to $\alpha$ columns of the ideal spectrum $X_i$. The peak of $X_i^{\prime }$ is located at the $(q+\beta )$-th column. Substituting the $X_i^{\prime }$ and $X_{mean}$ spectra into Equations (3) and (4), and rearranging the formula, we obtain:

$$\begin{array}{cc}\hfill X_i^{\prime }& =\left[x_1,\dots ,x_{\beta },X_{i,\alpha }^{\prime }\right]\hfill \\ \hfill & =\left[x_1,\dots ,x_{\beta },a_i{X}_{\left(i,\alpha \right);chem}+b_{i,\alpha }I+w_{i,\alpha }{\lambda }_{\alpha }+d_{i,\alpha }{\lambda }_{\alpha }^2+{\epsilon }_{i,\alpha }\right]\hfill \end{array}$$

(11)

The row vector $X_{\left(i,\alpha \right);chem}$ in Equation (11) is the theoretical absorption spectrum of sample i after being shifted by $\beta$ bands, and can be represented by Equation (3).

$$\begin{array}{c}\hfill X_{\left(i,\alpha \right);chem}=p\times \sum _{j=1}^J{c}_{\left(i,\alpha \right),j}{s}_{j,\alpha },j=1,2,\dots ,J\end{array}$$

(12)

$$\begin{array}{cc}\hfill X_i^{\prime }& =\left[x_1,\dots ,x_{\beta },X_{i,\alpha }^{\prime }\right]\hfill \\ \hfill & =\left[x_1,\dots ,x_{\beta },a_{i}p\times \sum _{j=1}^J{c}_{\left(i,\alpha \right),j}{s}_{j,\alpha }+b_{i,\alpha }I+w_{i,\alpha }{\lambda }_{\alpha }+d_{i,\alpha }{\lambda }_{\alpha }^2+{\epsilon }_{i,\alpha }\right]\hfill \end{array}$$

(13)

Represent the average spectrum $X_{mean}$ by dividing it into blocks by columns:

$$\begin{array}{c}\hfill X_{mean}=\left[x_{1;mean},\dots ,x_{\beta ;mean},X_{\left(i,\alpha \right);mean}\right]\end{array}$$

(14)

$$\begin{array}{cc}\hfill & X_i^{\prime }-{\alpha }_i{X}_{mean}\hfill \\ \hfill & =\left[x_1-{\alpha }_i{x}_{1;mean},\dots ,x_{\beta }-{\alpha }_i{x}_{\beta ;mean},a_{i}p\times \sum _{j=1}^J{c}_{\left(i,\alpha \right),j}{s}_{j,\alpha }+b_{i,\alpha }I+w_{i,\alpha }{\lambda }_{\alpha }+d_{i,\alpha }{\lambda }_{\alpha }^2+{\epsilon }_{i,\alpha }-{\alpha }_i{X}_{\left(i,\alpha \right);mean}\right]\hfill \end{array}$$

(15)

We discovered that, due to spectral shift issues, when using the least squares method to solve, a horizontally shifted error of $\beta$ will appear in $X_i^{\prime }-{\alpha }_i{X}_{mean}$. From an analytical chemistry perspective, we examined the two sets of spectra. The $(\beta +1)$-th band of $X_i^{\prime }$ should correspond to the first band, $x_{1;mean}$, of $X_{mean}$ after repositioning. We focused on analyzing the spectral bands from the q-th column to the $(q+\beta )$-th column. At this point, the spectral slope of $X_i^{\prime }$ was positive, while the spectral slope of $X_{mean}$ was negative. The inconsistency in slopes can cause significant interference in the MSC/SNV calibration results. Furthermore, when dealing with a set of spectra exhibiting peak shifts, the average spectrum $X_{mean}$ will also be subject to significant errors. Figure 3 provides a visual representation of the spectra curves affected by the offset. Based on the visual assessment and comparison depicted in Figure 3, the differences in slope between the two sets of spectra can be evaluated.

2.5. SNV Algorithm Challenges with Multiplicative and Additive Effects in Spectral Data

SNV assumes that multiplicative effects are uniform over the whole spectral range [31,32], which is not always the case, particularly when there is a quadratic term, $cl+d{l}^2$, varying with wavenumber in the near-infrared spectra. As mentioned earlier, SNV introduces artifacts to the spectra. This will now be demonstrated through theoretical derivation.

Let the equation for the spectrum after SNV transformation be represented as Equation (16):

$$x_{i,j}^{SNV}=\frac{(x_{i,j}-\overline{x}i)}{\sqrt{\frac{\sum {j=1}^p{(x_{i,j}-{\overline{x}}_i)}^2}{p-1}}}$$

(16)

where $x_{i,j}^{SNV}$ is the element of the transformed spectrum, $x_{i,j}$ is the corresponding original element of the spectrum i at variable j, ${\overline{x}}_i$ is the mean of spectrum i, and p is the number of variables or wavelengths in the spectrum.

Let the theoretical absorption spectrum unaffected by multiplicative and additive effects be denoted as $x_{i,chem}$. The relationship between the theoretical absorption spectrum and the actual spectrum can be expressed using Equation (17) when the signal is not affected by random measurement noise $w_i\lambda +d_i{\lambda }^2$:

$$x_i=a_i{x}_{i,chem}+b_{i}I+{\epsilon }_i$$

(17)

Substituting Equation (17) into Equation (16), we obtain:

$$\begin{array}{cc}\hfill x_{i,j}^{SNV}& =\frac{\left(x_{i,j}-{\overline{x}}_i\right)}{\sqrt{\frac{{\sum }_{j=1}^p{\left(x_{i,j}-{\overline{x}}_i\right)}^2}{p-1}}}\hfill \\ & =\frac{\left(a_i{x}_{i,j;chem}+b_{i}I+{\epsilon }_i-\left(a_i{\overline{x}}_{i;chem}+b_{i}I+{\epsilon }_i\right)\right)}{\sqrt{\frac{{\sum }_{j=1}^p{\left(a_i{x}_{i,j;chem}+b_{i}I+{\epsilon }_i-\left(a_i{\overline{x}}_{i;chem}+b_{i}I+{\epsilon }_i\right)\right)}^2}{p-1}}}\hfill \\ & =\frac{\left(x_{i,j;chem}-{\overline{x}}_{i;chem}\right)}{\sqrt{\frac{{\sum }_{j=1}^p{\left(x_{i,j;chem}-{\overline{x}}_{i;chem}\right)}^2}{p-1}}}\hfill \end{array}$$

(18)

$a_i$ and $b_{i}I+{\epsilon }_i$ can be fully eliminated, resulting in a perfect correction of the additive and multiplicative effects. However, when the signal is affected by random measurement noise $w_i\lambda +d_i{\lambda }^2$, the relationship between the theoretical absorbance spectrum and the actual spectrum can be represented by Equation (19):

$$x_i=a_i{x}_{i,chem}+b_{i}I+w_i\lambda +d_i{\lambda }^2+{\epsilon }_i$$

(19)

When Equation (19) is substituted into the term $\left(x_{i,j}-{\overline{x}}_i\right)$ of the SNV method, expanding the term $\left(x_{i,j}-{\overline{x}}_i\right)$ yields:

$$\left(a_i{x}_{i,j;chem}+b_{i}I+w_i{\lambda }_i+d_i{\lambda }_i^2+{\epsilon }_i-\left(a_i{\overline{x}}_{i;chem}+b_{i}I+w_i\overline{\lambda }+d_i{\overline{\lambda }}^2+{\epsilon }_i\right)\right)$$

(20)

In Equation (20), $\lambda$ represents wavenumber, and after taking the average, $w_i{\lambda }_i+d_i{\lambda }_i^2$ cannot be balanced and eliminated with $w_i\overline{\lambda }+d_i{\overline{\lambda }}^2$, introducing new artifacts to the preprocessed spectrum.

2.6. First Derivative Algorithm Challenges with Multiplicative and Additive Effects in Spectral Data

The first derivative algorithm is a commonly used preprocessing algorithm for near-infrared spectra. It effectively corrects for multiplicative effects and baseline drift [33,34]. However, it is well-known that the first derivative can amplify the noise in the near-infrared spectral signal. Additionally, due to the widespread use of electronic components in near-infrared spectrometers, the amplification circuitry in the sensor can also amplify high-frequency noise while capturing and processing the near-infrared spectral information of the sample [35]. High-frequency noise, caused by the quantum properties of photons in the sensor, is an unavoidable noise source. Therefore, the first derivative algorithm has inherent limitations when applied to near-infrared spectra preprocessing.

3. Materials and Methods

3.1. Spectral Offset Recalibration Method

Based on the aforementioned analysis, existing spectral preprocessing methods are unable to address the complexity arising from the combination of peak shifts and multiplicative and additive effects. Therefore, this study proposes a new preprocessing method called Spectral Offset Recalibration (SOR). The principle of the SOR method can be summarized as follows:

Considering that the COW peak shift method can calculate a set of value functions based on the signal curve and obtain the calibrated spectral signal using these value functions, we leveraged this characteristic in combination with a prior multiplicative and additive effects correction method such as the MSC algorithm. By first applying the multiplicative and additive effects correction to the original spectral signal and then performing the COW peak shift correction, the benefit function values could effectively mitigate the impact of the multiplicative and additive effects. These benefit function values were then applied to the original spectrum, followed by another round of multiplicative and additive effect recalibration, resulting in a more accurate correction for the combined effect of peak shifts and multiplicative and additive effects on the original spectrum. The workflow of the new algorithm is as follows:

• The matrix M is defined as:
  $$M=\left[\begin{array}{cccc}{r}_1& 1& 1& 1\\ r_2& 1& 2& 4\\ ⋮& ⋮& ⋮& ⋮\\ r_p& 1& p& p^2\end{array}\right]$$

  where $r_1,r_2,\dots ,r_p$ are the coefficients corresponding to the spectral basis functions. This matrix is used in the calculation of the EMSC coefficients and the subsequent coefficient correction in the EMSC method.
• The Pearson product-moment correlation coefficient is used to calculate the degree of correlation between each spectrum in the original spectrum and the rest of the spectral dataset. These correlation coefficients are then summed and normalized to obtain the weight vector, denoted as W, for the EMSC method. The calculation formula for the weight vector is as follows:
  $${\rho }_i={\rho }_{x_i;x_1,\dots ,x_{i-1},x_{i+1},x_p}={\sum }_{j=1}^p\frac{cov\left(x_i,x_j\right)}{{\sigma }_{x_i}{\sigma }_{x_j}}j⊈i$$
  $W={\sum }_{i=1}^p{\rho }_i$
• To calculate the EMSC coefficients, denoted as
  $$\left[\begin{array}{cccc}a& b& c& d\end{array}\right]={\left(MW{M}^T\right)}^{-1}MW{x}^T$$

  where X is a matrix representing the original spectra, with dimensions $N\times L$, where L is the number of spectral data points; ${}^T$ denotes matrix transpose.
• The coefficient-corrected signal can be obtained using the formula:
  $$z=\frac{1}{a}x-\left[\begin{array}{ccc}1& 1& 1\\ 1& 2& 4\\ ⋮& ⋮& ⋮\\ 1& p& p^2\end{array}\right]\left[\begin{array}{c}\frac{b}{a}\\ \frac{c}{a}\\ \frac{d}{a}\end{array}\right]$$

  where:
  • x represents the original spectrum.
  • a, b, c, and d are the EMSC coefficients obtained from the previous step.
  • The matrix represents the matrix of spectral basis functions, with p indicating the number of basis functions used.
• Let us define the alignment quality, also known as the gain function, as the correlation coefficient $\rho$ between the corrected spectrum z and the spectrum average. The gain function is denoted as $f\left(I\right)$ and is measured as $f\left(I\right)=\rho (I_T,I_A)$, where I represents the segment number. The gain function is defined by the formula:
  $$\overline{z}=arg{max}_z\left({\sum }_{i=0}^{N-1}f\left[z_i;z_{i+1}\right]\right)$$
  Here, the i-th segment of the spectrum $\left[z_i;z_{i+1}\right]$ is defined by the boundary points $z_i$ and $z_{i+1}$. Given $z_0=0<z_1<\cdots <z_{N-1}<z_N=L_T$ and $u_i\in [\Delta -t,\Delta +t];i=0,\cdots ,N-1$, where $\Delta$ is a constant and t is the tolerance, we have $z_{i+1}=z_i+m+u_i;i=0,\cdots ,N-1$.
• To correct the original spectrum x based on the optimal gain function, we can follow these steps:
  • Define the boundary points $z_i$ based on the desired segmentation of the spectrum: $z_0=0$, $z_1$, $z_2$, …, $z_{N-1}$, $z_N=L_T$.
  • Determine the values of $u_i$ within the range $[\Delta -t,\Delta +t]$ for each segment, where $u_i$ represents the offset or shift to be applied to the corresponding segment. Here, $\Delta$ is a constant and t is the tolerance.
  • Calculate the corrected spectrum $\overline{x}$ by applying the following equation for each segment i: $\overline{x}=arg{max}_x\left({\sum }_{i=0}^{N-1}f\left[x_i;x_{i+1}\right]\right)$, where $\left[x_i;x_{i+1}\right]$ denotes the section defined by border points $x_i$ and $x_{i+1}$.
  • Repeat the above step for all segments of the spectrum.
  By applying this correction process based on the optimal gain function, the original spectrum x can be effectively aligned and corrected.
• To further process the offset-corrected near-infrared spectra using the EMSC method, the influence of multiplicative scatter effects are mitigated and further processed spectra for analysis or comparison are obtained.

3.2. Processing Chart Summary

This section presents a brief flowchart of the SOR method, where the key steps and their applications discussed in the brief flowchart are described in Section 3.1. The Figure 4 illustrates the SOR method for coefficient correction and alignment of near-infrared spectra, aiming to provide a clear foundation for subsequent content.

3.3. Simulated Spectra

The simulated spectrum was composed of three Gaussian distributions with a total of 700 spectral bands. The weights for these distributions were randomly sampled from a uniform distribution between 0 and 1. Figure 5a illustrates the three Gaussian-shaped spectral peaks.

Next, these three Gaussian distributions were linearly combined to create a simulated spectrum signal. The resulting signal is shown in Figure 5b. The means of the three Gaussian distributions were 40, 80, and 170, with standard deviations of 10, 12, and 4, respectively.

Subsequently, the spectra were multiplied by a scalar, which was randomly drawn from a normal distribution with a mean of 1 and a standard deviation of 0.1. The multiplied spectra were then added with a background structure and white noise. The background structure was at a scale of 0.1% of the signal, while the white noise was at a scale of 0.001% of the signal. The white noise was generated from a normal distribution with a mean of 0. The resulting spectra are shown in Figure 6a.

Finally, an additional term of $cl+d{l}^2$ was added to these spectra and offset noise was added to the spectral curves, where c and d follow normal distributions with zero mean and standard deviations of ${3.7}^{-9}$ and ${2.6}^{-8}$, and l represents the number of spectral bands, respectively. The resulting signal is shown in Figure 6b.

3.4. NIR Spectra of Wood

This dataset comprises near-infrared spectra of solid wood veneers. The near-infrared spectra were acquired using a portable NIRS (InGaAs)-array spectrometer (NIRquest512, Orlando, FL, USA). The spectrometer’s parameters were as follows: the detection range began at 900 nm and ended at 1700 nm, with an average interval of 3.1 nm between spectral bands. The user interface for the near-infrared spectrometer was the SpectraSuite 2.0 (Quadrangle Blvd, Orlando, FL, USA) spectroscopy software platform.

4. Result and Discussion

4.1. Simulated Spectra

4.1.1. Initial Multiplicative and Additive Effect Correction Results Using SOR Method

Figure 7 illustrates the results obtained by applying three preprocessing methods to near-infrared simulated spectra affected by multiplication factors and baseline drift (as shown in Figure 6a). The three methods include the weighted MSC algorithm, with a weight vector of all ones (Figure 7a); the weighted MSC algorithm, with manually set weights (Figure 7b); and the SOR recalibration method (Figure 7c).

In the weighted MSC algorithm shown in Figure 7a (top), the weights were set to a vector of all ones. In Figure 7b (top), the weights were manually set, with zeros in the region covering two chemical peaks (from 0 to 270) and unity in the rest of the spectral range (from 271 to 700). Figure 7c (top) represents the weights obtained through the SOR recalibration method, resulting in a smooth and distinct weight spectrum.

The spectra were processed for Initial Correction using the SOR algorithm based on the all-ones matrix, as shown in Figure 7a. The spectra in the range of 0–270 represent the linear combination of the first two peaks. However, the all-ones matrix yielded the poorest results among the three approaches. Figure 7d presents the spectra processed by the weighted MSC algorithm, where it can be observed that the offset issue persists. This may be attributed to the significant bias introduced by the all-ones weighted MSC matrix.

In contrast, Figure 7b represents the spectra processed using the ideal weight matrix (W) generated based on the simulated spectral characteristics, while the spectra in the range of 271–700 serve as a reference spectrum unaffected by shape effects. Therefore, the weight values were set to 0 for the first two peaks in the range of 0–270 and set to 1 for the peaks in the range of 271–700. The SOR algorithm constructed using this matrix produced the spectra shown in Figure 7b. Significant differences can be observed between these spectra. The two chemical peaks of the ideal spectra exhibit similar changes and have similar maximum values. In practice, for a set of real near-infrared spectra, the variance is often concentrated in specific regions of the spectrum, while the standard MSC transformation applies uniform normalization to all variables.

Figure 7b,c employ the weighted MSC algorithm. In both figures, the weights are close to 0 in the region affected by shape effects (0–271) and nonzero in the region affected only by size effects (271–700). The weighted MSC algorithm utilized these weights to calculate the mean and standard deviation of the spectra, mainly located in the right region. In the ideal case of Figure 7b, it can be directly observed that the algorithm calculated the mean and standard deviation of only the spectra affected by size effects during variable selection, leading to excellent spectral preprocessing results. Figure 7c, on the other hand, determines the values of matrix W based on the correlation coefficients of the near-infrared spectra (correlation coefficient method). When the diagonal values of W corresponding to a specific wavelength are large, the average and standard deviation of the spectral variables for the current wavelength have a high correlation with the results of the SOR algorithm. Both weighting approaches effectively estimate the multiplicative and additive factors, as shown in Figure 7c. Although the spectral preprocessing results in Figure 7c are not as good as the ideal case in Figure 7b, it is because the ideal weight matrix (W) was generated based on the simulated spectral characteristics of the experiment. However, for a set of unknown test samples, the correlation coefficient method used in the results of Figure 7c can achieve near-perfect calibration.

4.1.2. Peak Shift Correction and Recalibration Results Using SOR Method

Building upon the COW alignment theory, we extended our methodology by incorporating an alignment and calibration step to further enhance the quality of preprocessing. After undergoing initial calibration using the SOR algorithm, the simulated spectra were subjected to COW alignment, which involved segmenting the spectra into individual segments and assessing their distortion levels. Subsequently, an objective function was computed, which informed the offset correction of the original spectra.

Figure 8 provides a visual representation of the original spectral segments before and after alignment using COW. The alignment process yielded satisfactory results, aligning peak maximums with positions in the reference spectrum. However, the influence of multiplicative effects introduced some inaccuracies in the aligned signals. Notably, translation induced by the alignment algorithm can lead to deformation in peaks affected by size effects, resulting in concavity issues at peak crests in certain regions (420–500 nm).

Finally, the SOR algorithm’s recalibration step was employed to eliminate both multiplicative and additive effects from the spectrum. After addressing peak shifts and ensuring minimal impact on the original spectrum, the removal of multiplicative and additive effects becomes a straightforward task. Figure 9 showcases the final outcome following the SOR algorithm’s corrective process, exemplifying the successful removal of these effects.

4.1.3. Comparative Analysis of Preprocessing Method Results and Performance

In continuation of the principles outlined in the VSN paper [25], we extended our exploration in Section 3.3 by introducing a unique strategy aimed at overcoming the inherent challenge of quantitatively gauging the effectiveness of preprocessing algorithms when conventional metrics are absent within the realm of near-infrared spectroscopy. Our approach involved the formulation of an innovative methodology centered around the creation of specialized simulated spectra. These simulated spectra were specifically crafted to counteract the limitations of traditional efficacy evaluation. They served as controlled testbeds, intricately tailored to mimic complex scenarios wherein spectral characteristics are intentionally altered, thereby facilitating a comprehensive assessment of preprocessing algorithm performance.

As illustrated in Figure 6, this set of simulated spectra operated within the 0–270 wavenumber range, where two distinct peak groups were scaled by varying coefficients according to the pattern of Linear Combinations of Normal Random Variables. A third peak remained unaltered by multiplicative scaling. Subsequently, drift noise and peak shift errors were introduced to these simulated spectra.

The efficacy of preprocessing algorithms is traditionally challenging to quantify in near-infrared spectroscopy due to the lack of standardized metrics. In lieu of conventional evaluation criteria, we employed this specialized set of simulated spectra as a surrogate for performance assessment. The width of the third peak served as an indicator of algorithmic effectiveness, with narrower peaks signifying superior performance. We evaluated the efficacy of SNV, MSC, EMSC, and the proposed SOR algorithm on this simulated dataset, aiming to discern the preprocessing approach that yielded the most favorable results. Notably, a narrower third peak indicated better algorithmic performance.

Figure 9 depicts the near-infrared spectra processed by the four preprocessing algorithms: SNV-COW, MSC-COW, EMSC-COW, and our proposed SOR algorithm. Evidently, the third peak of the spectra processed by SNV and MSC still exhibited some degree of drift. Notably, the EMSC algorithm seemed to introduce distortions even more pronounced than those observed with SNV. In contrast, the preprocessing results obtained with our proposed SOR algorithm exhibited superior performance in addressing these distortions.

4.2. NIR Spectra of Wood

4.2.1. Initial Multiplicative and Additive Effect Correction Results Using SOR Method

Figure 10 illustrates the processing results of the near-infrared spectra of solid wood panels, which are clearly affected by baseline offset and multiplicative effects. The results of EMSC are shown in Figure 10a, where the baseline and multiplicative effects are significantly reduced. However, due to the impact of spectral peak shift effects, the correction of the peaks at 191 nm and 336 nm is less satisfactory. In Figure 10b, manually selected weights were used to exclude the portions expected to provide information from the EMSC calculations. The weights were set to 0 at the positions of the characteristic peaks and to 1 for the other wavelengths. Similarly, based on the correlation coefficients of the near-infrared spectra, the values of matrix W were determined for the weighted EMSC. The results of the weighted EMSC are shown in Figure 10b. The side effects of EMSC have been eliminated, and the positions of the peaks and valleys in the spectrum are well-separated, while remaining very close in the rest of the wavelength range.

4.2.2. Peak Shift Correction and Recalibration Results Using SOR Method

After the preliminary preprocessing of the NIR spectra of solid wood panels using Initial Correction for the SOR Method, the multiplicative effects in the spectra were effectively corrected, thereby eliminating their influence on the peak shift algorithm. Subsequently, in this study, the average spectrum of this dataset was taken as the reference baseline for the NIR spectra, and the Peak shift Correction algorithm was employed to construct a cost function. This cost function included the optimal distortion of the reference samples. Based on this cost function, the original NIR spectra of uncorrected solid wood panels were aligned. To visualize a portion of the original spectra graph (range: 0–512), Figure 11 serves as an example, demonstrating the effect of the Peak shift Correction algorithm on the spectral peak alignment of the simulated spectral data. The alignment effect is satisfactory, with the peak maximum values aligned to the positions in the reference spectrum.

Satisfactory alignment can be observed for the first and second characteristic peaks of the spectral signal. However, minor inaccuracies were detected near point 190 in the spectrum graph. Some changes in the shape of the absorption peak were observed in the vicinity of this wavelength range. Additionally, the position where the feature peak should be located at 350 was not entirely satisfactory. Nevertheless, these feature peaks at these positions have relatively low intensities and do not belong to the spectral bands that primarily reflect the mechanical properties and other main attributes of the wood. Hence, they can be disregarded. After the initial offset correction of the raw spectra, the MSC algorithm was employed for recalibration, and Figure 12 represents the final recalibration result.

5. Conclusions

In this study, we developed a new algorithm based on the combination of the EMSC and COW algorithms address the complex issues arising from the coupling of additive and multiplicative effects with signal shift in multivariate signals. The application of the new algorithm effectively corrected peak shifts and drift problems in the original spectra, as demonstrated on both simulated and real data.

However, it is important to further emphasize specific issues related to the use of the new algorithm. Firstly, when facing square drift and shape effects in near-infrared spectroscopy, the SNV algorithm exhibited poor data performance after preprocessing, while the first derivative algorithm suffered from high noise levels. The new algorithm abandoned these preprocessing methods targeting individual signals and required the definition of a “learning set” to compute the weight matrix. Nevertheless, this is not a major concern, as such learning sets are typically required in successive stages of model construction.

Furthermore, several aspects need further investigation to achieve a fully automated implementation of the algorithm. One such aspect is the design of parameters in the weight generation function. Although a weight calculation method based on the sigmoid function transformation was proposed, its theoretical support is currently insufficient and requires validation on other datasets, similar to the validation performed on a real dataset of wood spectra. The impact of other tunable parameters in COW may also warrant additional investigation, which can be addressed through theoretical research.

Lastly, it should be noted that the new algorithm does not provide any improvement compared to standard normalization methods when dealing with multivariate data where shape effects are present in all variables and no square offset term exists. In such cases, the combination of the SNV method with the COW offset-correction algorithm can be used as an alternative preprocessing approach to the new algorithm.

In conclusion, the proposed algorithm effectively addresses the complex issues arising from the coupling of additive and multiplicative effects with signal shift in multivariate spectroscopic analysis. However, further research is needed to address specific issues related to the algorithm’s usage and to achieve fully automated implementation. By considering these aspects, the algorithm can be further enhanced and applied to a wide range of spectroscopic datasets, enabling accurate and reliable analysis in various scientific and industrial applications.