Adaptive Kalman Filtering for Compensating External Effects in On-Line Spectroscopic Measurements

Abstract

This study addresses the challenges of real-time spectroscopic sensing in industrial applications, where external factors such as temperature fluctuations, pressure variations, and particle size distribution significantly impact measurement accuracy. Conventional quantitative analytical methods often neglect these dynamic influences, leading to erroneous concentration estimates. To overcome these limitations, we propose an integrated modeling framework that combines a discrete-time process model with a physics-based spectroscopic sensor model, explicitly accounting for the dynamic properties of the system. A key innovation of this work is the development and application of an Adaptive Kalman Filter (AKF) to systematically correct for measurement distortions caused by external disturbances. Unlike conventional filtering techniques, the AKF dynamically adjusts to changing process conditions by leveraging real-time observability analysis, ensuring robustness even in the presence of sensor noise and environmental variability. Furthermore, to address cases where full observability is not achievable, we introduce a reduced-order Adaptive Kalman Filter (rAKF), which optimally estimates concentrations while minimizing computational complexity. A comprehensive series of simulations was conducted to assess the sensitivity of the estimation to variations in external signal type, noise levels, and initial values for parameters and states. The findings of this study demonstrate the superior performance of both AKF and rAKF in comparison to conventional filtering techniques, including the Extended Kalman Filter. The proposed approaches have been shown to enhance the reliability of spectroscopic sensor measurements, enabling more precise real-time estimations that can be used for monitoring and advanced process control strategies in industrial settings.

1. Introduction

Spectroscopic sensors, such as infrared and Raman spectroscopy-based sensors, play a crucial role in both laboratory research and industrial applications, where they are extensively employed for chemical analysis, material characterization, and real-time process monitoring. These sensors provide valuable insights into molecular composition and structural properties, enabling the precise identification and quantification of chemical species in diverse environments [1,2].

In practical applications, spectroscopic measurements are influenced by various physical factors. Changes in sample properties, such as viscosity and temperature, as well as variations in the measurement environment, for instance optical path length, can introduce nonlinearities that distort the spectral response through both additive and multiplicative effects. These distortions may compromise measurement accuracy and reliability. Thus, it is essential to use advanced algorithms to compensate for these unwanted influences and ensure robust data interpretation. In this contribution, we have only considered temperature variations, as this is a common perturbation in the process industry (e.g., in fermentation tanks) and is also easy to recreate for testing. However, the approach can be easily extended to account for other types of perturbations.

Several authors have proposed data-driven approaches for modeling spectral responses without assuming predefined functional dependencies on wavelength. In the context of this work, an overview by [3] examines data-driven methods designed to mitigate the impact of temperature variations on Near-Infrared (NIR) spectra. Notable algorithms include Extended Multiplicative Signal Correction [4], Local and Global Partial Least-Squares calibration strategies [5], Extended Loading Space Standardization, and Systematic Error Prediction [6]. While these data-driven approaches have demonstrated effectiveness, they rely on an informative dataset for calibration, which is typically performed offline. In addition, they lack the flexibility to integrate prior knowledge, limiting their adaptability in dynamic environments.

To overcome these limitations, we propose modeling the spectral response using physics-based spectral methods [7,8]. This approach represents the spectral response through parametric models, formulated as a linear combination of peak functions that carry physicochemical significance. These models inherently account for nonlinear effects, such as peak shifts and deformations, providing a more interpretable and adaptable framework for spectral analysis.

Following the physics-based approach, the integration of the sensor model with the process dynamic model leads to a nonlinear state-space representation, where the output equation is nonlinear and contains unknown parameters. Although Kalman filters are standard algorithms for state estimation in linear time-varying systems, the model in this application does not conform to a standard structure. However, it can be reformulated in a standard form by applying a transformation as suggested in [9] for a continuous-time model. This transformation results in a time-variant linear output map, where the previously unknown parameters appear linearly in the state equation multiplying the inputs, facilitating their estimation within a structured framework. For this structure, ref. [10] presents an approach based on two interconnected Kalman-like filters, and ref. [11] proposes one based on the combination of a Kalman filter and recursive least square (RLS) algorithm. Kalman filters for linear time-varying systems have been shown to exhibit superior convergence speed and capacity to tune the filter dynamic compared to adaptive observers, such as those described in [12].

The convenient structure of the model obtained through state transformation comes at the cost of introducing additional state variables, resulting in an overparameterized model and increased computational effort for estimation. To address this issue, an alternative error prediction approach has been proposed in [13], which avoids the need for state transformation but leads to highly complex recursive equations for computing the sensitivity of the prediction error with respect to states and parameters.

To address the aforementioned limitations, the present contribution extends the author’s earlier work in continuous time [14] by incorporating a discrete-time formulation, more sophisticated and realistic sensor models, and the explicit consideration of process disturbances. Moreover, it builds upon the results presented in [15] by conducting a formal observability analysis and introducing a novel reduced-order estimator designed to decrease the dimensionality of the system and, consequently, reduce computational complexity. In addition, a thorough comparative evaluation of the proposed estimators is provided, benchmarked against the standard Extended Kalman Filter.

The paper is organized as follows: Section 2 describes an integrated discrete-time model for the dynamic aspects of the process and the spectroscopic information. Section 3 presents a standard Extended Kalman Filter and two versions of an Adaptive Kalman Filter for performing online calibration. It also provides a convergence analysis of the AKF, considering the specific structure of the model to shed light on the conditions that need to be fulfilled. Section 4 presents an example that addresses the problem of correcting the effect of an external variable on the mixture of two components. It compares the performance of all filters and performs a sensitivity analysis with respect to external signal, noise level, and initial values for parameters and states. Finally, the last section summarizes the conclusions and outlines future work.

2. Process and Sensor Modeling

Consider a process in which the concentration of each component in the mixture is independently controlled and monitored by a spectroscopic sensor, as shown in Figure 1.

The transport and mixing dynamic is represented by a discrete-time linear model as follows:

$$\begin{array}{ccc}\hfill \mathbf{x}(k+1)& =& \mathbf{A}\mathbf{x}\left(k\right)+\mathbf{B}\mathbf{u}\left(k\right)+{\mathsf{\omega }}_{\mathbf{x}}\left(k\right)\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill \mathbf{w}\left(k\right)& =& \mathbf{C}\mathbf{x}\left(k\right)\hfill \end{array}$$

where $\mathbf{u}\left(k\right)={[u_1\left(k\right),\cdots ,u_m\left(k\right)]}^T$ is the input vector, $\mathbf{x}={[x_1\left(k\right),\cdots ,x_n\left(k\right)]}^T$ is the state vector, and $\mathbf{w}\left(k\right)={[w_1\left(k\right),\cdots ,w_m\left(k\right)]}^T$ denotes the concentration vector. The state disturbance vector, $\mathsf{\omega }\left(k\right)={[{\omega }_1\left(k\right),\cdots ,{\omega }_m\left(k\right)]}^T$, follows an independent Gaussian noise process with zero mean and a covariance matrix ${\Sigma }_{{\omega }_{\mathbf{x}}}={\sigma }_{{\omega }_{\mathbf{x}}}\mathbf{I}$. The dimensions of the state-space model matrices are $\mathbf{A}\in R^{n\times n}$, $\mathbf{B}\in R^{n\times m}$, and $\mathbf{C}\in R^{m\times n}$. The model assumes that $\mathbf{A}$ is Hurwitz and has been validated in previous research [16].

The sensor output, denoted as $y(k,\lambda )\in \mathbb{R}$, represents the absorbance of the mixture and can be written, as described in [15], as

$$\begin{array}{c}\hfill y(k,\lambda )=\sum _{q=1}^{N_p}{\theta }_q{\Psi }_q(\lambda ,v\left(k\right))\mathbf{w}\left(k\right)+\nu (k,\lambda )\end{array}$$

(2)

where ${\Psi }_q(\lambda ,v\left(k\right))$ contains the cross products between the functions modeling the effect of the external signals and the peak function representing the spectral response of the sensor. The vector $\mathbf{w}\left(k\right)={[w_1\left(k\right),\cdots ,w_m\left(k\right)]}^T$ is the concentration vector and $\mathsf{\theta }$ is the vector of unknown calibration factors. The model also considers that the absorbance of different components is influenced by an exogenous variable $v\left(k\right)$. We assume that the output disturbances $\nu (k,\lambda )$ are independent Gaussian noise processes with zero mean and covariance matrix ${\Sigma }_{\nu }\left(\lambda \right)={\sigma }_{\nu }\left(\lambda \right)$. Notice that Equation (2) can also be written as

$$\begin{array}{c}\hfill y(k,\lambda )={\mathsf{\theta }}^T\mathsf{\Psi }(\lambda ,v\left(k\right))\mathbf{w}\left(k\right)+\nu (k,\lambda )\end{array}$$

(3)

where ${\mathsf{\theta }}^T=[{\theta }_1,\cdots ,{\theta }_{Np}]$ and $\mathsf{\Psi }(\lambda ,v\left(k\right))={[{\Psi }_1(\lambda ,v\left(k\right)),\cdots ,{\Psi }_{Np}(\lambda ,v\left(k\right))]}^T$.

Remark 1.

While the sensor model is primarily designed for spectroscopic sensors, its framework can also accommodate sensors in other applications where sensor characteristics depend on external factors and may vary over time. Drifting sensor gains and nonlinear behaviors are common in various tactical sensors. Several Kalman filters have been proposed in this field (see, for instance, [17,18]). The method proposed in this paper could potentially be applied to these contexts, although further research is required.

3. On-Line Correction

The on-line correction methods estimate concentrations based on spectral measurements, even though they are affected by exogenous variables. Thus, the on-line calibration problem can be defined as follows:

Given the process and sensor models, Equations (1) and (2), and the measurements of $\mathbf{u}\left(k\right)$, $y(k,\lambda )$ and $v\left(k\right)$, respectively, the problem is to estimate the concentrations $\mathbf{w}\left(k\right)$ and the calibration factors ${\theta }_i$.

We consider the unknown parameters to be constants perturbed by a zero mean stochastic variable, satisfying the following equation,

$${\theta }_i(k+1)={\theta }_i\left(k\right)+{\omega }_{{\theta }_i}\left(k\right),i=1,\cdots N_p$$

(4)

3.1. Extended Kalman Filter

By augmenting the state vector with the parameters, i.e., ${\mathbf{x}}_e={\left[\mathbf{x}\mathsf{\theta }\right]}^T$, Equations (1), (2) and (4) can be written as follows:

$${\mathbf{x}}_e(k+1)={\mathbf{A}}_e{\mathbf{x}}_e\left(k\right)+{\mathbf{B}}_e\mathbf{u}\left(k\right)+{\mathsf{\omega }}_{{\mathbf{x}}_e}\left(k\right)$$

(5)

where

$${\mathbf{A}}_e=\left[\begin{array}{cc}\mathbf{A}& \mathbf{0}\\ \mathbf{0}& \mathbf{I}\end{array}\right]{\mathbf{B}}_e=\left[\begin{array}{c}\mathbf{B}\\ \mathbf{0}\end{array}\right]$$

(6)

and ${\mathsf{\omega }}_{{\mathbf{x}}_e}\left(k\right)={\left[{\mathsf{\omega }}_{\mathbf{x}}\left(k\right){\mathsf{\omega }}_{\theta }\left(k\right)\right]}^T$.

The recursive equations for the EKF update equation is

$${\widehat{\mathbf{x}}}_e\left(k\right)={\mathbf{A}}_e{\widehat{\mathbf{x}}}_e(k-1)+{\mathbf{B}}_e\mathbf{u}(k-1)+\mathbf{K}(k,\lambda )\mathbf{e}(k,\lambda )$$

(7)

where the estimation error is

$$\begin{array}{cc}\hfill \mathbf{e}(k,\lambda )=& \mathbf{y}(k,\lambda )-\widehat{\mathsf{\theta }}{\left(k\right)}^T\mathsf{\Psi }(\lambda ,v\left(k\right))\mathbf{C}\widehat{\mathbf{x}}\left(k\right)\hfill \end{array}$$

(8)

for all $\lambda \in \Lambda$. The gain matrix $\mathbf{K}(k,\lambda )$ associated to the state is updated as follows:

$$\begin{array}{cc}\hfill \mathbf{P}\left(k\right|k-1)& ={\mathbf{A}}_e\mathbf{P}(k-1|k-1){\mathbf{A}}_e^T+{\mathbf{Q}}_e\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \mathsf{\Sigma }(k,\lambda )& ={\mathbf{C}}_e(k,\lambda )\mathbf{P}\left(k\right|k-1){\mathbf{C}}_e{(k,\lambda )}^T+\mathbf{R}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill \mathbf{K}(k,\lambda )& =\mathbf{P}\left(k\right|k-1){\mathbf{C}}_e{(k,\lambda )}^T\mathsf{\Sigma }{(k,\lambda )}^{-1}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \mathbf{P}\left(k\right|k)& =[\mathbf{I}-\mathbf{K}(k,\lambda ){\mathbf{C}}_e(k,\lambda )]\mathbf{P}\left(k\right|k-1)\hfill \end{array}$$

(12)

where the linearized measurement model is given by ${\mathbf{C}}_e(k,\lambda )=\left[{\mathbf{C}}_x(k,\lambda ){\mathbf{C}}_{\theta }(k,\lambda )\right]$ with

$$\begin{array}{cc}\hfill {\mathbf{C}}_x(k,\lambda )& =\widehat{\mathsf{\theta }}{\left(k\right)}^T\mathsf{\Psi }(\lambda ,v\left(k\right))\mathbf{C}\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathbf{C}}_{\theta }(k,\lambda )& =\mathsf{\Psi }(\lambda ,v\left(k\right))\mathbf{C}\widehat{\mathbf{x}}\left(k\right)\hfill \end{array}$$

(14)

for all $\lambda \in \Lambda$. We note that the wavelength range $\Lambda$ will in practice be discretized, and measurements at different wavelengths should either be stacked into a measurement vector, or the Kalman filter can be updated with each wavelength sequentially. The same applies to the AKF and its variant in the following sections.

Remark 2.

In a stochastic framework, the matrices ${\mathbf{Q}}_e$ and $\mathbf{R}$ represent noise covariance values, capturing the uncertainty associated with system dynamics and measurements. Conversely, in a deterministic approach, where randomness is not considered, these matrices function as tuning parameters. This perspective allows for adjusting ${\mathbf{Q}}_e$ and $\mathbf{R}$ according to specific needs or prior knowledge, treating them as hyperparameters or regularization terms in the estimation process.

Remark 3.

The accuracy of estimated values in the Extended Kalman Filter (EKF) depends on several key factors, such as the precision of linearization, the observability of the system, the nature of noise, initial state conditions, and the underlying system dynamics. Since these elements play a crucial role in the effectiveness of EKF implementation, the specific criteria for achieving convergence will vary based on the particular application.

3.2. Adaptive Kalman Filter

To transform the nonlinear model into a time-variant linear model, it is necessary to define a new variable ${\mathbf{q}}_i\left(k\right)={\theta }_i\left(k\right)\mathbf{x}\left(k\right)$. This approach was proposed by [9] for continuous-time. By multiplying the recursive equation of (1) by ${\theta }_i\left(k\right)$ we derive the following over-parameterized time-variant linear model:

$$\begin{array}{ccc}\hfill {\mathbf{q}}_i(k+1)& =& \mathbf{A}{\mathbf{q}}_i\left(k\right)+{\theta }_i\left(k\right)\mathbf{B}\mathbf{u}\left(k\right)+{\theta }_i\left(k\right){\mathsf{\omega }}_{\mathbf{x}}\left(k\right)+\hfill \\ & & \mathbf{x}(k+1){\omega }_{{\theta }_i}\left(k\right)\hfill \\ \hfill y(k,\lambda )& =& \sum _{i=1}^{N_p}{\Psi }_i(\lambda ,v\left(k\right))\mathbf{C}{\mathbf{q}}_i\left(k\right)+\nu (k,\lambda )\hfill \end{array}$$

(15)

A drawback of this overparameterization is that it neglects certain dependencies between redundant variables. As a result, the initially simple noise properties become more intricate and exhibit correlations. Consequently, noise is not explicitly taken into account in the following equations.

By defining $\mathbf{p}={[\mathbf{x}{\mathbf{q}}_1\cdots {\mathbf{q}}_{N_p}]}^T$ and $\mathsf{\theta }={[{\theta }_1\cdots {\theta }_{N_p}]}^T$, the system Equations (4) and (15) can be written as

$$\begin{array}{ccc}\hfill \left[\begin{array}{c}\mathbf{p}(k+1)\\ \mathsf{\theta }(k+1)\end{array}\right]& =& \left[\begin{array}{cc}{\mathbf{A}}_p& {\mathbf{A}}_{\theta }\left(k\right)\\ \mathbf{0}& \mathbf{I}\end{array}\right]\left[\begin{array}{c}\mathbf{p}\left(k\right)\\ \mathsf{\theta }\left(k\right)\end{array}\right]+\left[\begin{array}{c}{\mathbf{B}}_p\\ \mathbf{0}\end{array}\right]\mathbf{u}\left(k\right)\hfill \\ \hfill y(k,\lambda )& =& {\mathbf{C}}_p(k,\lambda )\mathbf{p}\left(k\right)\hfill \end{array}$$

(16)

where

$${\mathbf{A}}_p=\left[\begin{array}{cccc}\mathbf{A}& 0& 0& 0\\ 0& \mathbf{A}& 0& 0\\ 0& 0& \ddots & 0\\ 0& 0& 0& \mathbf{A}\end{array}\right]{\mathbf{B}}_p=\left[\begin{array}{c}\mathbf{B}\\ 0\\ ⋮\\ 0\end{array}\right]$$

(17)

$${\mathbf{A}}_{\theta }\left(k\right)=\left[\begin{array}{ccc}0& 0& 0\\ \mathbf{B}\mathbf{u}\left(k\right)& 0& 0\\ 0& \ddots & 0\\ 0& 0& \mathbf{B}\mathbf{u}\left(k\right)\end{array}\right]$$

(18)

$${\mathbf{C}}_p(k,\lambda )=\left[\begin{array}{cccc}0& {\Psi }_1(\lambda ,v\left(k\right))\mathbf{C}& \cdots & {\Psi }_{N_p}(\lambda ,v\left(k\right))\mathbf{C}\end{array}\right]$$

(19)

An Adaptive Kalman Filter with distinct error dynamics for the state and parameters can be derived using the algorithm proposed by Zhang [11], resulting in the following equations:

$$\begin{array}{cc}\hfill \widehat{\mathbf{p}}\left(k\right)& ={\mathbf{A}}_p\widehat{\mathbf{p}}(k-1)+{\mathbf{A}}_{\theta }\left(k\right)\widehat{\mathsf{\theta }}(k-1)+{\mathbf{B}}_p\mathbf{u}(k-1)+\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \mathbf{K}(k,\lambda )\mathbf{e}(k,\lambda )+\mathsf{{\rm Y}}\left(k\right)[\widehat{\mathsf{\theta }}\left(k\right)-\widehat{\mathsf{\theta }}(k-1)]\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \widehat{\mathsf{\theta }}\left(k\right)& =\widehat{\mathsf{\theta }}(k-1)+\mathsf{\Gamma }(k,\lambda )\mathbf{e}(k,\lambda )\hfill \end{array}$$

(21)

where the estimation error is

$$\begin{array}{cc}\hfill \mathbf{e}(k,\lambda )=& \mathbf{y}(k,\lambda )-{\mathbf{C}}_p(k,\lambda )[{\mathbf{A}}_p\widehat{\mathbf{p}}(k-1)+{\mathbf{A}}_{\theta }\left(k\right)\widehat{\mathsf{\theta }}(k-1)+{\mathbf{B}}_p\mathbf{u}\left(k\right)].\hfill \end{array}$$

(22)

The state gain matrix $\mathbf{K}(k,\lambda )$ is updated as follows:

$$\begin{array}{cc}\hfill \mathbf{K}(k,\lambda )& =\mathbf{P}\left(k\right|k-1){\mathbf{C}}_p{(k,\lambda )}^T\mathsf{\Sigma }{(k,\lambda )}^{-1}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill \mathbf{P}\left(k\right|k-1)& ={\mathbf{A}}_p\mathbf{P}(k-1|k-1){\mathbf{A}}_p^T+\mathbf{Q}\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill \mathsf{\Sigma }(k,\lambda )& ={\mathbf{C}}_p(k,\lambda )\mathbf{P}\left(k\right|k-1){\mathbf{C}}_p{(k,\lambda )}^T+\mathbf{R}\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill \mathbf{P}\left(k\right|k)& =[\mathbf{I}-\mathbf{K}(k,\lambda ){\mathbf{C}}_p(k,\lambda )]\mathbf{P}\left(k\right|k-1)\hfill \end{array}$$

(26)

Remark 4.

Equations (24)–(26) calculate the matrices involved in determining the state gain matrix, denoted as $\mathbf{K}(k,\lambda )$. The matrices $\mathbf{Q}$ and $\mathbf{R}$ are tuning parameters. Selecting their values can be challenging because, due to the model’s overparameterization, they no longer represent the noise covariances.

The gain matrix $\mathsf{\Gamma }(k,\lambda )$ associated with the parameters update is given by the following recursive equations:

$$\begin{array}{cc}\hfill \mathsf{\Gamma }(k,\lambda )& =\mathbf{S}(k-1)\mathsf{\Omega }{(k,\lambda )}^T\mathsf{\Lambda }(k,\lambda )\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill \mathsf{\Omega }(k,\lambda )& ={\mathbf{C}}_p(k,\lambda ){\mathbf{A}}_p\mathsf{{\rm Y}}(k-1)+{\mathbf{C}}_p(k,\lambda ){\mathbf{A}}_{\theta }\left(k\right)\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \mathsf{\Lambda }(k,\lambda )& ={[\gamma \mathsf{\Sigma }(k,\lambda )+\mathsf{\Omega }\left(k\right)\mathbf{S}(k-1)\mathsf{\Omega }{\left(k\right)}^T]}^{-1}\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill \mathbf{S}\left(k\right)& ={\displaystyle \frac{1}{\gamma }}\mathbf{S}(k-1)-{\displaystyle \frac{1}{\gamma }}\mathbf{S}(k-1)\mathsf{\Omega }{(k,\lambda )}^T\mathsf{\Lambda }(k,\lambda )\mathsf{\Omega }(k,\lambda )\mathbf{S}(k-1)\hfill \end{array}$$

(30)

where $\gamma \in (0,1)$ is a forgetting factor and $\mathsf{{\rm Y}}\left(k\right)$ is an auxiliary variable defined as follows:

$$\begin{array}{cc}\hfill \mathsf{{\rm Y}}\left(k\right)& =[\mathbf{I}-\mathbf{K}(k,\lambda ){\mathbf{C}}_p(k,\lambda )]{\mathbf{A}}_p\mathsf{{\rm Y}}(k-1)+[\mathbf{I}-\mathbf{K}(k,\lambda ){\mathbf{C}}_p(k,\lambda )]{\mathbf{A}}_{\theta }\left(k\right).\hfill \end{array}$$

(31)

Remark 5.

Equations (28)–(31) calculate matrices associated with the parameter gain matrix, $\mathsf{\Gamma }(k,\lambda )$, and the gain matrix related to the estimated parameter variations, $\mathsf{{\rm Y}}\left(k\right)$, as utilized in the state estimation equation. These equations can be interpreted as a recursive least square estimator with a forgetting factor [11].

3.3. Reduced Order Adaptive Kalman Filter

A reduced order AKF (rAKF) can be obtained if the original states $\mathbf{x}$ are not considered in the extended model, i.e., $\mathbf{p}$ just contains a function of the new state variables $\mathbf{q}$, $\mathbf{p}={[{\mathbf{q}}_1\cdots {\mathbf{q}}_{N_p}]}^T$. Hence, the system equations can be written as

$$\begin{array}{ccc}\hfill \mathbf{p}(k+1)& =& {\mathbf{A}}_p\mathbf{p}\left(k\right)+{\mathbf{B}}_{\theta }\left(k\right)\mathsf{\theta }\left(k\right)\hfill \end{array}$$

(32)

$$\begin{array}{ccc}\hfill y(k,\lambda )& =& {\mathbf{C}}_p(k,\lambda )\mathbf{p}\left(k\right)\hfill \end{array}$$

where

$${\mathbf{A}}_p=\left[\begin{array}{cccc}\mathbf{A}& 0& 0& 0\\ 0& \mathbf{A}& 0& 0\\ 0& 0& \ddots & 0\\ 0& 0& 0& \mathbf{A}\end{array}\right]$$

(33)

$${\mathbf{B}}_{\theta }\left(k\right)=\left[\begin{array}{ccc}\mathbf{B}\mathbf{u}\left(k\right)& 0& 0\\ 0& \ddots & 0\\ 0& 0& \mathbf{B}\mathbf{u}\left(k\right)\end{array}\right]$$

(34)

$${\mathbf{C}}_p(k,\lambda )=\left[\begin{array}{ccc}{\Psi }_1(\lambda ,v\left(k\right))\mathbf{C}& \cdots & {\Psi }_{N_p}(\lambda ,v\left(k\right))\mathbf{C}\end{array}\right].$$

(35)

The estimation of $\mathbf{q}$ and $\mathsf{\theta }$ is the obtained by using Equations (20) and (21). The original state estimate can then be derived by solving the following least-squares problem:

$$\underset{\mathbf{x}}{min}{\parallel \mathrm{diag}\left(\widehat{\mathsf{\theta }}\left(k\right)\right)\mathbf{x}-{[{\widehat{\mathbf{q}}}_1\left(k\right)\cdots {\widehat{\mathbf{q}}}_{N_p}\left(k\right)]}^T\parallel }^2.$$

(36)

Thus,

$$\widehat{\mathbf{x}}\left(k\right)={\displaystyle \frac{\mathrm{diag}\left(\widehat{\mathsf{\theta }}\left(k\right)\right)}{\widehat{\mathsf{\theta }}{\left(k\right)}^T\widehat{\mathsf{\theta }}\left(k\right)}}{[{\widehat{\mathbf{q}}}_1\left(k\right)\cdots {\widehat{\mathbf{q}}}_{N_p}\left(k\right)]}^T$$

(37)

3.4. Convergence Properties

The Adaptive Kalman Filters will converge if a set of key assumptions associated to the model structure and persistency of excitation for the input signals, i.e., $\mathbf{u}\left(k\right)$ and $v\left(k\right)$, are satisfied [11].

Assumption A1.

The pairs $[{\mathbf{A}}_p,\mathbf{Q}]$ and $[{\mathbf{A}}_p,{\mathbf{C}}_p(k,\lambda )]$ are uniformly completely controllable and observable, respectively.

Given that ${\mathbf{A}}_p$ is Hurwitz, it is always possible to choose a positive definite matrix $\mathbf{Q}$ so that the pair $[{\mathbf{A}}_p,\mathbf{Q}]$ is uniformly completely controllable, i.e., there are exit constants $c_c\in \mathbb{R}$ and $N_c\in \mathbb{N}$ such that the controllability Grammian satisfies the following:

$${\mathbf{W}}_c(k+N_c,k)\ge c_c\mathbf{I}\forall k\ge 0$$

(38)

On other hand, if $v\left(k\right)$ satisfies the condition that there are exit constants $c_o\in \mathbb{R}$ and $N_o\in \mathbb{N}$ such that observability Grammian satisfies the following:

$${\mathbf{W}}_o(k+N_o,k)\ge c_o{{\mathbf{A}}_p^{N_0}}^T{\mathbf{A}}_p^{N_0}\forall k\ge 0$$

(39)

then the pair $[{\mathbf{A}}_p,{\mathbf{C}}_p(k,\lambda )]$ will be uniformly completely controllable. This condition is crucial for ensuring the parameter convergence, and it implies that the exogenous signal must exhibit a form of persistent excitation.

Remark 6.

Assumption 1 only considers the conditions to ensure the boundedness of $\mathbf{P}\left(k\right|k)$, $\mathsf{\Sigma }(k,\lambda )$, and $\mathbf{K}(k,\lambda )$.

Remark 7.

The interpretation of ${\mathbf{C}}_p(k,\lambda )$ in Assumption 1 is as a matrix of stacked vectors, each representing the values corresponding to a wavelength belonging to the set Λ.

Remark 8.

Notice that for the full state model, in Equation (17), in the case of the AKF, the first elements of ${\mathbf{C}}_p(k,\lambda )$ corresponding to the state vector, $\mathbf{x}$, are zeros, and the matrix ${\mathbf{A}}_p$ has a block-diagonal structure. This structure renders the state variable $\mathbf{x}$ unobservable, as the observability Gramian will have a block of zeros on the diagonal. In contrast, the equations for the rAKF, Equation (32), lead to an observability Gramian with non-zero blocks on its diagonal.

Assumption A2.

The input signal $\mathbf{u}\left(k\right)$ is persistently exciting in the sense that there exists an integer $h>1$ and a constant $\alpha \in R$ such that $\forall k\ge 0$

$$\sum _{l=0}^{h-1}{\mathsf{\Omega }}^T(k+l,\lambda )\mathsf{\Sigma }{(k+l,\lambda )}^{-1}\mathsf{\Omega }(k+l,\lambda )\ge \alpha {\mathbf{I}}_p$$

(40)

where $\mathsf{\Sigma }(k,\lambda )$ and $\mathsf{\Omega }(k,\lambda )$ are defined by Equations (25) and (28), respectively.

Assumption 2 deals with the evolution of ${\mathbf{A}}_{\theta }$ in (32) and imposes conditions on the input signal, $\mathbf{u}\left(k\right)$. This assumption is required to ensure the boundedness of $\mathbf{S}\left(k\right)$. Notice that $\mathsf{\Omega }(k,\lambda )$ is related to the input signal through ${\mathbf{A}}_{\theta }$ in Equations (28) and (31).

Remark 9.

The Assumptions 1 and 2 require that the input and external variable satisfy persistent excitation condition, which in practice is not always satisfied. Recent works have addressed this problem [19,20], and the analysis of these approaches in the context of this application is the subject of future work.

4. Experimental and Simulation Results

This section presents both experimental and simulation-based results to illustrate and evaluate the proposed modeling and state estimation approach. First, the modeling methodology is demonstrated using experimental data. Following this, a simulated case study is introduced to explore the behavior and performance of the proposed state estimation algorithm under a broad range of operating conditions, highlighting its robustness and effectiveness.

To illustrate the modeling approach using experimental data, we consider the water absorbance spectra in the wavelength range of 850 nm to 1050 nm at different temperatures, as shown in Figure 2. The dataset used for this purpose comes from [5].

Following a physics-based approach, the water spectra $\varphi (\lambda ,T)$ can be represented as a linear combination of peak functions ${\psi }_i\left(\lambda \right)$ as follows:

$$\varphi (\lambda ,T)=\sum _{i=1}^5{a}_i\left(T\right){\psi }_i\left(\lambda \right)=\sum _{i=1}^5{a}_i\left(T\right)exp\left(-{\displaystyle \frac{{(\lambda -c_i)}^2}{w_i^2}}\right)$$

(41)

where the peak functions are defined in terms of Gaussian parameters, namely the center $c_i$ and width $w_i$. A two-step approach is used to identify the model parameters and validate this structure against experimental data.

First, for a given temperature, all parameters are estimated by solving a nonlinear least-squares problem fitted to the spectral response. The resulting basis functions, scaled by their respective linear coefficients, are shown in Figure 3.

In the second step, the Gaussian parameters ($c_i$, $w_i$) are kept fixed, and the linear coefficients $a_i\left(T\right)$ are computed for all temperature conditions. These coefficients are then regressed against temperature. The results, depicted in Figure 4, confirm the validity of the assumed model structure—in this case, a linear dependence on temperature. The estimated parameters and corresponding regression results are summarized in

Table 1. Parameters for water absorbance spectra model.

i	ai(T)	ci	wi
1	$15.2452-0.04765T$	688.8	294.58
2	$-0.10288+0.00308T$	9333.9	17.7178
3	$-0.0801+0.01035T$	960.6	15.68581
4	$8.1216+0.03854T$	972.5	50.31
5	$3.5253+0.03411T$	960.6	37.97104

. The maximum approximation error across the entire dataset is less than 1.

After having validated the modeling approach on the basis of real spectral responses, we illustrate the main ideas of the estimator by considering an example of the simulated mixing of two components presented in [15], where $u_1\left(k\right)$ and $u_2\left(k\right)$ are the flow rates of each component, $v\left(k\right)$ is the mixture temperature, and $y(k,\lambda )$ is the measured spectrum.

The discrete state space model, which represents the transport and the mixing of the two components, is defined by the following matrices:

$$\mathbf{A}=\left[\begin{array}{cccc}1.6& -0.8& 0& 0\\ 0.8& 0& 0& 0\\ 0& 0& 1.7& -0.85\\ 0& 0& 0.85& 0\end{array}\right]\mathbf{B}=\left[\begin{array}{cc}1& 0\\ 0& 0\\ 0& 1\\ 0& 0\end{array}\right]$$

(42)

$$\mathbf{C}=\left[\begin{array}{cccc}0& 0.05& 0& 0\\ 0& 0& 0& 0.0265\end{array}\right]$$

(43)

The measured spectrum considers the absorbance spectra of each component and the effect of temperature, i.e.,

$$y(k,\lambda )=w_1\left(k\right){\varphi }_1(\lambda ,v\left(k\right))+w_2\left(k\right){\varphi }_2(\lambda ,v\left(k\right))$$

(44)

where absorbance of each component is modeled as a linear combination of two Gaussian peaks, as illustrated in Figure 5a, and described by the following parameterization:

$$\begin{array}{cc}\hfill {\varphi }_1(\lambda ,v\left(k\right))& ={\theta }_1{\psi }_{11}\left(\lambda \right)+({\theta }_2+{\theta }_{3}v\left(k\right)){\psi }_{12}\left(\lambda \right)\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill {\varphi }_2(\lambda ,v\left(k\right))& =({\theta }_4+{\theta }_{5}v\left(k\right)){\psi }_{21}\left(\lambda \right)+{\theta }_6{\psi }_{22}\left(\lambda \right)\hfill \end{array}$$

where ${\psi }_{ij}\left(\lambda \right)$ represents Gaussian peaks associated to the model of each component i. Note that in this case, the external variable $v\left(k\right)$ influences the absorbance of each component linearly, as illustrated in Figure 5b.

Rearranging terms (44) can be written as Equation (2), where the set of parameters $\{{\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5,{\theta }_6\}$ represents the parameters of the sensor model, which are assumed to be unknown. Their nominal values are $\mathsf{\theta }={[2.2,-1.5,0.15,-0.02,0.1,2.5]}^T$, and vectors ${\Psi }_i(\lambda ,v\left(k\right))$ are as follows:

$$\begin{array}{c}\hfill {\Psi }_1={\left[\begin{array}{c}{\psi }_{11}\left(\lambda \right)\\ 0\end{array}\right]}^T,{\Psi }_2={\left[\begin{array}{c}{\psi }_{12}\left(\lambda \right)\\ 0\end{array}\right]}^T,{\Psi }_3={\left[\begin{array}{c}{\psi }_{12}\left(\lambda \right)v\left(k\right)\\ 0\end{array}\right]}^T\end{array}$$

(46)

$$\begin{array}{c}\hfill {\Psi }_4={\left[\begin{array}{c}0\\ {\psi }_{21}\left(\lambda \right)\end{array}\right]}^T,{\Psi }_5={\left[\begin{array}{c}0\\ {\psi }_{21}\left(\lambda \right)v\left(k\right)\end{array}\right]}^T,{\Psi }_6={\left[\begin{array}{c}0\\ {\psi }_{22}\left(\lambda \right)\end{array}\right]}^T\end{array}$$

The simulation examples initially focus on a base case to highlight the core features of the algorithms under specific conditions. Subsequently, a sensitivity analysis explores the variations in initial parameters, state conditions, and noise levels to further distinguish the algorithms’ performance under different scenarios.

4.1. Base Case

The base case considers zero-mean Gaussian disturbances in both the sensor measurements and the state equations, with variances ${\sigma }_{\nu }=0.5$ and ${\sigma }_{\omega }=0.1$, respectively. It also accounts for step changes in the flow rates of each component and periodic variations in the external variable, as shown in Figure 6. The effect of these changes and the noise on the spectral response can be seen in Figure 7.

In the next simulations for both the EKF and AKF, the initial conditions for the estimated parameters are set to zero and the states are set as ${[100,100,50,50]}^T$. To illustrate the evolution of estimated states and parameters, we consider one realization of state and measurement noise.

For the EKF, we consider the following parameters and initial conditions: the matrix $\mathbf{Q}=0.1\mathbf{I}$ is chosen to be a diagonal matrix, and $\mathbf{P}\left(0\right)={10}^3\mathbf{I}$. As illustrated in Figure 8a, the estimated parameters converge to their true values more rapidly than the state. This observation is further supported by the evolution of the state and parameter error norms shown in Figure 8b.

For the AKF, the matrix $\mathbf{Q}=0.0001\mathbf{I}$ was chosen as a constant diagonal matrix, $\mathbf{P}\left(0\right)={10}^3\mathbf{I}$, and $\lambda =1$. The observer initial conditions for $\mathbf{q}$ are set to zero. The estimated concentrations quickly converge to the real values, although some parameters take longer to stabilize, as shown in Figure 9a. Variability in input variables, such as flow rates and temperature, ensures the convergence of both estimated concentrations and parameters. Furthermore, the state error asymptotically converges to zero, as depicted in Figure 9b.

In adaptive applications, once the algorithm has converged, it is always informative to analyze the structure of the final observer gain; in this case, it is $\mathbf{K}(300,\lambda )$. From Figure 10, two important observations can be drawn: first, the gains associated with the estimation of the state variables $\mathbf{x}$ are zero; second, the wavelength dependencies are linked to the location of the basis functions (see Figure 5a). Zero gains indicate a lack of observability for the associated states, which is consistent with Remark 8.

The evolution of the estimated parameters for the rAKF follows a similar pattern to that of the AKF, as shown in Figure 11a. However, as illustrated in Figure 11a,b, the estimated concentrations exhibit much faster dynamics, driven by the observer design rather than the open-loop system dynamics.

A comparative analysis of the dynamic responses, depicted in Figure 8, Figure 9 and Figure 11, shows that both the AKF and rAKF algorithms are effective in parameter estimation. The rAKF provides faster dynamics for estimated concentrations because of its enhanced observer design. However, the EKF takes longer to converge, with the state convergence being much slower than the parameter convergence.

A comprehensive analysis across multiple realizations enhances the robustness and reliability of the results by capturing the variability introduced by different instances of noise.

Table 2. Performance measures.

Algorithm	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{\theta }}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{\theta }}}$
EKF	2.6827	0.1486	0.8822	0.0778
AKF	2.0344	0.0468	0.5514	0.0376
rAKF	2.4546	0.0888	0.5514	0.0376

summarizes the results for one hundred noise realizations, presenting the mean values and standard deviations of the Root Mean Squared Errors (RMSEs) for both states and parameters. The results indicate that EKF exhibits higher mean values of $RM{S}_x$ and $RM{S}_{\theta }$ compared to the AKF and rAKF. Lower RMSE values generally suggest better performance, indicating that the AKF and rAKF outperform the EKF based on these measures. Although the rAKF initially shows larger errors than the AKF, its convergence is faster. For detailed information about the accuracy of each estimate, refer to the boxplot of the estimated parameters in Figure 12. It is evident that both the AKF and rAKF provide more accurate parameter estimates compared to the EKF.

4.2. Sensitivity Analysis

The following simulations demonstrate the sensitivity of the estimates to variations in the external signal, initial parameters, states, and noise levels. The values of $\mathbf{Q}$, $\mathbf{R}$, and $\mathbf{P}\left(0\right)$ are consistent with those used in the base case.

To analyze the effect of the external signal profiles, two additional profiles are considered. The first profile consists of a constant value of 20 (°C) with step changes occurring at time instant 75, 150, and 225. This profile provides less dynamic information than the one considered in the base case. The second profile is simply a constant value of 20 (°C), which does not provide enough information for estimating the parameters, violating one of the assumptions necessary for convergence.

Based on the results summarized in

Table 3. Performance measures for different external signal profiles.

Signal	Algorithm	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{\theta }}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{\theta }}}$
Steps	EKF	3.3590	0.2089	0.7815	0.0233
	AKF	2.0344	0.0468	0.7179	0.0164
	rAKF	2.4344	0.0864	0.7179	0.0164
Constant	EKF	3.3955	0.2128	1.2370	0.0030
	AKF	2.0344	0.0468	1.2593	0.0022
	rAKF	2.4342	0.0864	1.2593	0.0022

, the comparison of estimation algorithms—the EKF, AKF, and rAKF—under two distinct signal profiles, Steps and Constant, reveals distinct performance characteristics. The Adaptive Kalman Filter (AKF) consistently demonstrates superior performance in both signal scenarios, showing the lowest mean Root Mean Square (RMS) error for the state $\mathbf{x}$ and $\mathsf{\theta }$, coupled with the least variability. This indicates that the AKF provides more accurate and reliable estimates across different conditions. In contrast, the Extended Kalman Filter (EKF) exhibits the highest mean RMS errors and greater variability for both state variables, making it less consistent and accurate compared to the AKF. The reduced order Adaptive Kalman Filter (rAKF) performs better than the EKF but falls short of the AKF, presenting intermediate levels of mean errors and variability. Specifically, while the rAKF matches the AKF’s performance in estimating $\mathsf{\theta }$ under both signal profiles, it does not achieve the same accuracy for the states. A comparative analysis with respect to the results of the more dynamic rich input signal (Table 2) shows that the EKF performs poorly in estimating the state variables and parameters. However, the AKF and rAKF provide similar estimates for the state variables. As expected, all algorithms perform worse in terms of parameter estimates when the signal has fewer time variations. The lack of persistent excitation in the AKF and rAKF mainly affects the parameter estimation.

To analyze the effect of initial conditions, two sets are considered: Initial Condition 1, where the state and parameters are zero, and Initial Condition 2, where $\widehat{\mathbf{x}}\left(0\right)={[3,6,9,8]}^T$ and $\widehat{\mathsf{\theta }}\left(0\right)={[1,1,0.1,0.1,0.1,1]}^T$. The results summarized in

Table 4. Performance measures for different initial conditions.

IC	Algorithm	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{\theta }}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{\theta }}}$
IC 1	EKF	11.6040	0.2106	10.1255	0.5335
	AKF	2.5395	0.0377	0.5514	0.0376
	rAKF	2.2728	0.0620	0.5724	0.0364
IC 2	EKF	6.7165	0.1064	6.0134	0.1754
	AKF	2.5678	0.0373	0.5488	0.0377
	rAKF	2.2728	0.0620	0.5724	0.0364

clearly show that the EKF is highly sensitive to the choice of initial conditions, whereas the AKF and rAKF provide consistent results regardless of initial conditions. From a practical perspective, this robustness in initialization is a significant advantage, simplifying the setup of the estimation algorithms.

Finally,

Table 5. Performance measures for different noise level.

Noise Level	Algorithm	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{x}}}$	${\mathit{\mu }}_{{\mathit{RMS}}_{\mathit{\theta }}}$	${\mathit{\sigma }}_{{\mathit{RMS}}_{\mathit{\theta }}}$
50%	EKF	2.7151	0.2052	0.9501	0.1058
	AKF	2.7035	0.0468	0.5514	0.0376
	rAKF	2.5971	0.1008	0.6104	0.0481
100%	EKF	2.7621	0.2569	1.0244	0.1304
	AKF	2.8338	0.0667	0.6680	0.0577
	rAKF	2.7341	0.1041	0.6700	0.0575

summarizes the results of the algorithms with respect to an increase of 50% and 100% of the noise level considered in the base case. At the 50% noise level, the AKF demonstrates the lowest mean RMS error for the parameters with the smallest variability, indicating superior robustness to moderate noise. The rAKF follows closely, performing slightly better than the EKF in terms of both parameters and state estimation accuracy. The EKF, while reasonably effective, shows higher mean errors and variability.

At the higher noise level of 100%, all algorithms exhibit increased mean RMS errors and variability. The AKF maintains a relatively better performance, although its state error increases compared to the 50% noise scenario. The rAKF shows a performance comparable to the AKF, with a slight increase in errors and variability, but it still outperforms the EKF. The EKF displays the highest mean RMS errors and variability in both state variables under both noise levels. Overall, while all algorithms are affected by increased noise, the AKF and rAKF exhibit better robustness and consistency in estimation performance compared to the EKF.

5. Conclusions

This research introduces a novel discrete-time correction algorithm based on physic-based models and an Adaptive Kalman Filter (AKF) for spectroscopic applications. The AKF is formulated using an extended model of the process and the spectroscopic sensor, enabling the simultaneous estimation of system states and calibration factors while accounting for the influence of external variables. Additionally, an observability analysis of the AKF led to the development of a reduced-order AKF (rAKF), designed to improve computational efficiency by reducing the number of estimated variables.

The results demonstrate that the AKF provides robust and accurate state estimation across various conditions, outperforming both the rAKF and the Extended Kalman Filter (EKF). The rAKF offers a computationally efficient alternative with faster convergence, making it suitable for scenarios where reduced complexity is necessary. In contrast, the EKF exhibits higher estimation errors and variability, confirming the advantages of the proposed approaches over conventional filtering techniques.

These findings highlight the potential of the AKF and rAKF for enhancing spectroscopic sensing, particularly in applications such as Near-Infrared (NIR) and Raman spectroscopy. However, the practical implementation of these filters requires careful consideration of the persistent excitation conditions necessary for ensuring asymptotic convergence. Since real-world applications may not always satisfy these conditions, future research should focus on developing relaxed convergence criteria to ensure stable performance even under limited excitation.

The promising performance of the proposed AKF framework suggests significant opportunities for advancing spectroscopic sensing technologies. Future studies will explore real-time laboratory validations and extend the methodology to broader sensing applications, further refining the capabilities of adaptive filtering in spectroscopic analysis.