Hybrid Process Models in Electrochemical Syntheses under Deep Uncertainty

Abstract

Chemical process engineering and machine learning are merging rapidly, and hybrid process models have shown promising results in process analysis and process design. However, uncertainties in first-principles process models have an adverse effect on extrapolations and inferences based on hybrid process models. Parameter sensitivities are an essential tool to understand better the underlying uncertainty propagation and hybrid system identification challenges. Still, standard parameter sensitivity concepts may fail to address comprehensive parameter uncertainty problems, i.e., deep uncertainty with aleatoric and epistemic contributions. This work shows a highly effective and reproducible sampling strategy to calculate simulation uncertainties and global parameter sensitivities for hybrid process models under deep uncertainty. We demonstrate the workflow with two electrochemical synthesis simulation studies, including the synthesis of furfuryl alcohol and 4-aminophenol. Compared with Monte Carlo reference simulations, the CPU-time was significantly reduced. The general findings of the hybrid model sensitivity studies under deep uncertainty are twofold. First, epistemic uncertainty has a significant effect on uncertainty analysis. Second, the predicted parameter sensitivities of the hybrid process models add value to the interpretation and analysis of the hybrid models themselves but are not suitable for predicting the real process/full first-principles process model’s sensitivities.

1. Introduction

Hybrid modeling allows to include process knowledge via first-principles system equations and, simultaneously, to compensate for missing process information via data-driven machine learning (ML) algorithms. In the current literature, especially in process system engineering, one finds various examples of hybrid modeling strategies. For instance, Nielsen et al. combine a deep neural network predicting particle phenomena kinetics with a first-principles model to simulate particle processes [1]. Focusing on real-time model predictive control aspects of a rectification column, Schafer et al. propose a model reduction framework, which creates meaningful model compartments based on a first-principles model and artificial neural networks to consider the input-output relations within the compartments [2]. In the case of chemical process optimization, in [3], measured operating data are processed via an artificial network and integrated into a flow-sheet simulator of a chemical synthesis problem. Moreover, hybrid models are also an essential key technology to realize Industry 4.0 and digital twin concepts in the pharmaceutical industry and biotechnology for process intensification and monitoring [4,5]. For instance, Krippl et al. highlight the added value of hybrid models for improved prediction of degradation processes in separation technology [6], and Cardillo et al. discuss the relevant role of hybrid models in in silico process development for vaccine manufacturing [7]. As different as the application fields and the degree of hybridization may be, the fundamental problem of model uncertainty applies to all realizations of hybrid modeling, including system identification and interpretation [8,9]. In [10], the important role of design of experiments (DoE) and informative data are critically discussed, and statistical DoE with an intensified concepts for rapid exploration of the design space compared. Alternatively, the direct study of model uncertainties results in model-based DoE. A significant part of the model uncertainties can be attributed to the parameter uncertainties. Predictions based on hybrid models, where these model parameters are an integral part, are also uncertain. Technically, parameter sensitivities determine the degree of uncertainty amplification. In the context of deep uncertainty, however, the fundamental distinction between aleatoric and epistemic uncertainty seems mandatory in first-principles modeling [11,12,13] and ML algorithms [14,15,16,17]. Thus, aleatoric and epistemic uncertainty should be taken into account for uncertainty propagation and parameter sensitivities in the field of hybrid modeling, too. Please note that in the literature, the term deep uncertainty refers to imprecise probabilities and includes not only the individual consideration of aleatoric and epistemic uncertainty but stresses that these are, in fact, uncertainties of uncertainties [18,19]. Thus, imprecise uncertainties lead to the aleatoric and epistemic components, which have to be examined according to their dependence.

The so-called probability-box (p-box, for short) can be implemented to represent deep uncertainty, which has already been shown in various simulation studies recently [18,20,21,22], but not for hybrid process models. Furthermore, algorithms for efficient uncertainty propagation and global sensitivity analysis (GSA) are the subject of ongoing research. In particular, quasi-Monte Carlo methods and Polynomial Chaos Expansion (PCE) dominate the literature, and both approaches have already been successfully applied to p-box problems [23,24]. However, these standard concepts require a high number of (training) samples and computation time for good accuracy, depending on the number of uncertain model parameters and the level of their variance. Thus, the contribution of this work is twofold: First, deep uncertainty (i.e., aleatoric and epistemic uncertainty) is considered in hybrid modeling and system identification to quantify and study the reliability of the provided simulation results. Second, with the so-called Point Estimate Method (PEM), we introduce a highly efficient approach for uncertainty quantification (UQ) and GSA in the field of hybrid modeling. As the case studies address both a classic combination of a first-principles process model with an ML algorithm as well as a novel neural ordinary differential equation problem [25,26,27], this work provides a generic but also a timely blueprint of an efficient UQ and GSA in hybrid modeling under deep uncertainty.

The remainder of the work is structured as follows: In Section 2, the concepts for hybrid modeling, GSA, deep uncertainty, and the PEM are presented. In Section 3, the workflow of an efficient UQ and GSA for two electrochemical synthesis problems is described, implemented, and discussed. The conclusions are summarized in Section 4.

2. Methods

Relevant hybridization strategies, i.e., the combination of first-principles process models and data-driven ML algorithms, are introduced first. Next, the basics of GSA and the concept of imprecise probabilities, including the p-box approach, are summarised. In the last subsection, the PEM for efficient uncertainty propagation and GSA evaluation is presented.

2.1. Hybrid Process Models

In the literature, the standard hybridization architecture is the combination of first-principles models and ML algorithms in serial or parallel structure and combinations thereof [28,29,30,31]; also see Figure 1. For instance, in a sequential hybrid model setting (Figure 1a), simulation results based on first-principles models are post-processed with ML algorithms for process characterization and decision making [32]. Alternatively, unknown kinetics, model parameters, system states, and process components can be reconstructed with so-called soft sensors using AI-powered, data-driven ML concepts, and sparse data might be enriched with ML-insilico data sets for process modeling [29,33,34,35,36]. These reconstructed quantities are transferred to the first-principles process model, which can only be solved or identified with the additional ML-input. In the case of a parallel setting (Figure 1b), the ML algorithm operates in parallel with the first-principles process model to quantify residuals and plant-model mismatches [30], respectively. The serial and parallel architecture primarily aim at pure forward simulations, i.e., for given CPPs (Critical Process Parameters), the CQAs (Critical Quality Attributes) are predicted, ML algorithms are also used for process optimization and control, see Figure 1c. Note that ML can either be directly utilized to solve the underlying optimization problem [34,37,38] or as part of a model-order reduction framework to reduce computational costs, i.e., the ML algorithm is part of the optimization problem itself [2,39].

Alternatively, differential equations reflecting the mathematical process model and process knowledge can be directly fused with ML algorithms (Figure 1d). When using so-called universal neural (ordinary) differential equations or physics-informed neural networks (PINNs), the neural network/ML-part completes regular ordinary differential systems (ODEs) and solves partial differential equation systems, respectively [27,40,41,42,43]. This novel hybridization strategy (i.e., ODEs where a neural network determines the vector field/derivative function), seems especially useful to efficiently replace incomplete process knowledge with learning-based methods; i.e., preserving the given model structure but improving unknown system states interactions and enabling their identification, respectively [41,44,45].

In the field of ML, neural networks play an essential role and range from the simple feed-forward neural networks, multilayer perceptron, recurrent neural networks to deep neural networks. Generally, the ith neural network layer, ${\mathrm{NNL}}_i\left(x\right):{\mathbb{R}}^{d_{i-1}}\to {\mathbb{R}}^{d_i}$, has $N_i$ neurons in the ith layer. Moreover, ${\mathrm{NNL}}_i\left(x\right)$ is specified with the weight matrix, ${\mathbf{W}}^i\in {\mathbb{R}}^{d_i\times d_{i-1}}$, and the bias vector, ${\mathbf{b}}_i\in {\mathbb{R}}^{d_i}$. For instance, a feed-forward neural network reads as:

$$\begin{array}{cc}\hfill \mathrm{input}\mathrm{layer}:& {\mathrm{NNL}}_0\left(x\right)=x\in {\mathbb{R}}^{d_0},\hfill \end{array}$$

(1a)

$$\begin{array}{cc}\hfill \mathrm{hidden}\mathrm{layer}:& {\mathrm{NNL}}_i\left(x\right)=\sigma ({\mathbf{W}}^i{\mathrm{NNL}}_{i-1}\left(x\right)+{\mathbf{b}}_i)\in {\mathbb{R}}^{d_i},\forall 1\le i\le I-1,\hfill \end{array}$$

(1b)

$$\begin{array}{cc}\hfill \mathrm{output}\mathrm{layer}:& {\mathrm{NNL}}_I\left(x\right)={\mathbf{W}}^I{\mathrm{NNL}}_{I-1}\left(x\right)+{\mathbf{b}}_I\in {\mathbb{R}}^{d_I},\hfill \end{array}$$

(1c)

where for activation functions, $\sigma \left(·\right)$, sigmoid or tangent functions are commonly used [46]. In hybrid modeling, the first-principles model and the used ANN affects the uncertainty propagation and sensitivity analysis problem, including deep uncertainty effects.

In what follows, all implementations were coded in Julia [47]. For the studied ANN (Section 3.1) and the neural ODEs (Section 3.2), the Flux.jl [48,49] and the DiffEqFlux.jl [50] Julia library were utilized, respectively. The Adam [51] first-order gradient-based optimization algorithm was used with the L2-norm (sum of squared errors) to train the ANN and the neural ODEs. All simulations were run on a standard desktop machine.

2.2. Global Sensitivity Analysis

Basically, in GSA the model parameters, $\mathbf{p}$, and the model outcomes, $\mathbf{y}$, are considered as random variables, and GSA intends to measure the amount of variance that each parameter, $\mathbf{p}\left[i\right]$, adds to the total variance of the jth model output, ${\sigma }^2\left(\mathbf{y}\left[j\right]\left(t\right)\right)$[52]. The conditional variance reads as ${\underset{-i}{\sigma }}^2\left(\mathbf{y}\left[j\right]\left(t\right)\right|\mathbf{p}\left[i\right])$. Note that the subscript $-i$ implies that the variance is taken over all parameters with the exception of $\mathbf{p}\left[i\right]$. The expected value of the resulting conditional variance is $\underset{i}{E}\left[{\underset{-i}{\sigma }}^2\left(\mathbf{y}\left[j\right]\left(t\right)\right|\mathbf{p}\left[i\right])\right]$, and the subscript notation of $\underset{i}{E}$ indicates that the expected value is only for the parameter $\mathbf{p}\left[i\right]$. The total output variance, ${\sigma }^2\left(\mathbf{y}\left[j\right]\left(t\right)\right)$, can be partitioned in two additive terms [52]. With:

$${\sigma }^2\left(\mathbf{y}\left[j\right]\left(t\right)\right)={\underset{i}{\sigma }}^2\left(\underset{-i}{E}\left[\mathbf{y}\left[j\right]\left(t\right)\right|\mathbf{p}\left[i\right]]\right)+\underset{i}{E}\left[{\underset{-i}{\sigma }}^2\left(\mathbf{y}\left[j\right]\left(t\right)\right|\mathbf{p}\left[i\right])\right],$$

(2)

the GSA (i.e., first-order Sobol’ indices [52]) is defined as:

$$S[j,i]\left(t\right)=\frac{{\underset{i}{\sigma }}^2\left(\underset{-i}{E}\left[\mathbf{y}\left[j\right]\left(t\right)\right|\mathbf{p}\left[i\right]]\right)}{{\sigma }^2\left(\mathbf{y}\left[j\right]\left(t\right)\right)},$$

(3)

with $S\left(\mathbf{p}\right)\in {\mathbf{R}}^{n_y\times n_p}$, and ${\displaystyle \sum _{i=1}^{n_p}}S[j,i]\left(t\right)\le 1,\forall j\in \{1,\dots ,n_y\}$. For more details regarding the basic concepts of variance-based sensitivity measures and notation used, the reader is referred to [52] and references therein. Please note that in what follows, parameter uncertainties were predefined and, in the presented simulation study, are not directly based on experimental data and particular parameter identification methods, respectively. However, the proposed concept can be readily applied to experimental data. The resulting parameter uncertainties in the context of DoE using the inverse of the Fisher information matrix or sample-based (Monte-Carlo) methods [53,54]. Alternatively, parameter uncertainties could be directly quantified by implementing Bayesian parameter estimation techniques [55].

2.3. Deep Uncertainty

In the case of deep uncertainty, which besides aleatoric uncertainty, includes epistemic uncertainty, the standard UQ and GSA concepts can be extended with the parametric p-box approach [18,20,56]. Here, aleatoric uncertainties $\mathit{\xi }$ depend on epistemic uncertainties $\mathit{\theta }$ according to:

$$F_{\mathbf{\Xi }}\left(\mathbf{\xi }\right)=F_{\mathbf{\Xi }}\left(\mathit{\xi }\right|\mathit{\theta }),\mathit{\theta }\in {\mathcal{D}}_{\Theta }\subset {\mathbb{R}}^{n_{\theta }},$$

(4)

where $F_{\mathbf{\Xi }}$ is the cumulative distribution function (CDF) of the random vector $\mathit{\xi }$, the random vector $\mathit{\theta }$ with independent uniform distribution densities is specified by upper and lower bounds, and ${\mathcal{D}}_{\Theta }=[{\theta }_1^l,{\theta }_1^u]\times \dots [{\theta }_{n_{\theta }}^l,{\theta }_{n_{\theta }}^u]$ denotes the feasible domain of $\mathit{\theta }$. Technically, confidence intervals might be also approximated with statistical moments [22] according to:

$$E_{\mathit{\theta }}[E_{\mathit{\xi }}\left[\mathbf{y}\left[j\right]\right]\pm {\beta }_{\xi }{\sigma }_{\mathit{\xi }}^2{\left[\mathbf{y}\left[j\right]\right]}^{0.5}]\pm {\beta }_{\theta }{\sigma }_{\mathit{\theta }}^2{[E_{\xi }\left[\mathbf{y}\left[j\right]\right]\pm {\beta }_{\xi }{\sigma }_{\mathit{\xi }}^2{\left[\mathbf{y}\left[j\right]\right]}^{0.5}]}^{0.5},$$

(5)

where ${\beta }_{\xi }$ and ${\beta }_{\theta }$ determines the conservativeness from the variation of aleatoric $\mathit{\xi }$ and epistemic $\mathit{\theta }$ uncertainty, respectively. In this context, parameter sensitivities in terms of ${\sigma }_{\mathit{\xi }}^2$ and ${\sigma }_{\mathit{\theta }}^2$ can be quantified with Equation (3) and the PEM.

2.4. Point Estimate Method

The basic problem with the GSA calculation, especially under deep uncertainty, is the high computational cost when approximating the underlying integral terms of Equations (2) and (3) with numerical methods, e.g., Gaussian quadrature and Monte Carlo simulations. The PEM, however, has proven advantageous for different engineering applications [57], which include sophisticated process systems engineering and electrochemical processes scenarios [58,59]. Using the nominal parameter vector ${\mathbf{p}}_0$, artificial but deterministic rule-based model parameter vector realizations ${\mathbf{p}}_k$ result in the following parameter vector sample set, ${\mathbf{p}}_k\in \mathcal{O}:=\{{\mathbf{p}}_0,{\mathcal{O}}_1,-{\mathcal{O}}_1,{\mathcal{O}}_2,-{\mathcal{O}}_2,{\mathcal{O}}_3,-{\mathcal{O}}_3\}$ with:

$$\begin{array}{cc}\hfill & {\mathcal{O}}_1:=\{{\mathbf{p}}_0\left[i\right]+\vartheta ,\forall i\in \{1,\dots ,n_p\}\},\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & {\mathcal{O}}_2:=\{{\mathbf{p}}_0\left[(i,j)\right]+[+\vartheta ,+\vartheta ],\forall i,\underset{j>i}{j}\in \{1,\dots ,n_p\}\},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & {\mathcal{O}}_3:=\{{\mathbf{p}}_0\left[(i,j)\right]+[-\vartheta ,+\vartheta ],\forall i,\underset{j>i}{j}\in \{1,\dots ,n_p\}\}.\hfill \end{array}$$

(8)

Note that the total parameter sample number, $n_{PEM}$, scales quadratically with the number of uncertain model parameters:

$$n_{PEM}=2n_p^2+1,$$

(9)

and the expected value, $E\left[·\right]$, of the model response is approximated with:

$$E\left[\mathbf{y}\left(\mathbf{p}\right)\right]\approx \sum _{k=1}^{n_{PEM}}{w}_k\mathbf{y}\left({\mathbf{p}}_k\right).$$

(10)

Here, in case of a standard Gaussian distribution, the permutation parameter and weight factors are defined as $\vartheta =\sqrt{3},w_0=1+\frac{n_p^2-7n_p}{18},w_{1,\dots ,2n_p+1}=\frac{4-n_p}{18},w_{2n_p+2,\dots ,n_{PEM}}=\frac{1}{36}$, respectively. As demonstrated in [60], using the PEM any parametric or non-parametric probability distribution can be incorporated via a (non)linear transformation step.

The variance, ${\sigma }^2\left[·\right]$, can be approximated in the same way:

$${\sigma }^2\left[\mathbf{y}\left(\mathbf{p}\right)\right]\approx \sum _{k=1}^{n_{PEM}}{w}_k{(\mathbf{y}\left({\mathbf{p}}_k\right)-E\left[\mathbf{y}\left(\mathbf{p}\right)\right])}^2.$$

(11)

Note that to consider epistemic uncertainty, an outer loop of uncertainty propagation must be implemented, e.g., the scaling factor $\vartheta$ and weights $w_i$ of the PEM are adapted to uniform probability distributions [57]. Thus, there are $n_{PEM}=4n_{\theta }^2{n}_{\xi }^2+2(n_{\theta }^2+n_{\xi }^2)+1$ total PEM sample points, including aleatoric and epistemic uncertainty. Moreover, due to a nested re-sampling strategy, the global sensitivity matrix (Equation (3)) can be determined highly efficiently with $n_{PEM}$ model simulations, where appropriate subsets, ${\mathcal{P}}_i\subset \mathcal{O}$, are evaluated to calculate $S\left(\mathbf{p}\right[i\left]\right),$$\forall i\in \{1,\dots ,n_p\}$. For more details regarding the general PEM in UQ and GSA, please refer to [58].

The general structure of the PEM-based approach for UQ and GSA in hybrid modeling under deep uncertainty is visualized in Figure 2.

3. Results and Discussion

Two electrochemical synthesis processes, the synthesis of furfuryl alcohol from furfural and the electrochemical reduction of nitrobenzene to 4-aminophenol, are considered in this work. The electrochemical synthesis has advantage of mild operating conditions such as temperature and pressure in comparison to the conventional chemical synthesis processes. Moreover, the reaction rate of electrochemical synthesis processes can be controlled by changing the potential/current applied during the process [61]. The modeling of electrochemical synthesis of furfuryl alcohol and 4-aminophenol are explained in Section 3.1 and Section 3.2, respectively.

3.1. Furfuryl Alcohol with a Serial Hybrid Model

According to Figure 3, the electrochemical synthesis of furfuryl alcohol (FA) from furfural (FF) involves the formation of adsorbed hydrogen (H${}_{\mathrm{ads}}$) from a Volmer reaction, adsorption of furfural (FF${}_{\mathrm{ads}}$) on the electrode surface, and the reaction of H${}_{\mathrm{ads}}$ and FF${}_{\mathrm{ads}}$. In what follows, the bulk concentration of all species are given in mol cm⁻³. To derive a simplistic first-principles process model, the following assumptions are made:

• The hydrogen fraction of adsorbed hydrogen remains the same during the reaction (continuous reaction and negligible hydrogen evolution).
• There is no formation of additional byproducts such as methyl furan, methyltetrahydrofuran, and tetrahydrofurfuryl alcohol.
• The fraction of the surface area available for the reactions does not change over process time.

The governing ordinary differential equation system based on material balances for a continuous operation reads as:

$$\begin{array}{cc}\hfill \frac{dF{F}_{\mathrm{ads}}}{dt}& =C(r_{\mathrm{ads}}-r_{FF}),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill \frac{dFF}{dt}& =\frac{Q_{\mathrm{in}}}{V}F{F}_{\mathrm{in}}-r_{\mathrm{ads}}·a-\frac{Q_{\mathrm{out}}}{V}FF,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \frac{dFA}{dt}& =\frac{Q_{\mathrm{in}}}{V}F{A}_{\mathrm{in}}+r_{FF}·a-\frac{Q_{\mathrm{out}}}{V}FA.\hfill \end{array}$$

(14)

In Equations (13) and (14), $Q_{\mathrm{in}}$ and $Q_{\mathrm{out}}$ refer to the flow rate of input and output, V refers to reactor volume and a refers to effective surface area. The rate of adsorption of FF on the electrode surface is expressed as:

$$r_{\mathrm{ads}}=k_{\mathrm{ads}}·FF·{\theta }_v,$$

(15)

In Equation (15), $k_{\mathrm{ads}}$ and ${\theta }_v$ refer to rate constant for adsorption of FF and vacant sites available for adsorption of FF, respectively. and the rate of conversion of FF to FA is defined as:

$$r_{FF}=k_{FF}·{\theta }_H^2·F{F}_{\mathrm{ads}}.$$

(16)

In Equation (16), $k_{FF}$ refers to rate constant for conversion of furfural to furfuryl alcohol.

The adsorption of hydrogen by the Volmer reaction, ${\theta }_H$, is typically represented by adsorption isotherms such as the Langmuir adsorption isotherm [62]. In this case, adsorption of hydrogen and FF are necessary for the conversion of FF to FA. As the overpotential-dependent data for ${\theta }_H$ is not available in the literature for the system under study, a sigmoid function relating the initial adsorbed hydrogen, ${\theta }_{H0}$, and the overpotential, E, could be used as an empirical model in the simplest case:

$${\theta }_{H_0}=\frac{1}{1+exp(-(s_1+s_2·E))}.$$

(17)

By evaluating the least squares error, the sigmoid function was parameterized; i.e., $s_1=-5.15$ and $s_2=2.55$. For a more precise calculation, the sigmoid function can be replaced with an ANN resulting in a hybrid process model framework. The overpotential-dependent formation of H${}_{\mathrm{ads}}$ is expressed by a two-layer ANN with six and two nodes (Equation (1)), respectively, with tangent functions as activation function, overpotential as the ANN input, and H${}_{\mathrm{ads}}$ as the ANN output.

The hybrid model was trained with values of yield of furfuryl alcohol for overpotentials between 1.4 to $2.7$ $\mathrm{V}$ from literature [63]. The ANN outcome of the overpotential vs. the fraction of surface covered by H${}_{\mathrm{ads}}$ is shown in Figure 4a, and the process model parameters are given in

Table 1. Model parameters and initial value for electrochemical synthesis of furfuryl alcohol.

Parameter	${\mathit{k}}_{\mathbf{ads}}$	${\mathit{k}}_{\mathbf{FF}}$	C	$\mathbf{FF}(\mathit{t}=0)$	$\mathbf{FA}(\mathit{t}=0)$	${\mathit{\theta }}_{\mathit{H}0}$
Value	0.5969	$5.6437\times {10}^{-8}$	$1.6081\times {10}^9$	$0.1\times {10}^{-3}$	0	Equation (17)
Unit	cm s−1	s−1	-	mol cm−3	mol cm−3	-

. The fraction of adsorbed hydrogen increased with as the overpotential changed from 1.4 to $2.7$ $\mathrm{V}$ since there are no other potential dependent reactions in the reaction scheme of furfuryl alcohol synthesis (Figure 4b). The concentration of furfuryl alcohol predicted by the model with the sigmoid function for hydrogen adsorption and the hybrid model were in reasonable agreement with the data from literature for the overpotential between 1.4 to $2.7$ $\mathrm{V}$ (Figure 4b).

The impact of parameter uncertainties on the yield of furfuryl alcohol was studied using the conventional Monte-Carlo approach and the proposed PEM approach. Please note that these model parameters refer to the parameter vector $\mathbf{p}$ in Equations (2), (3) and (6)–(11). The range of uncertain parameters considered are given in

Table 2. Epistemic uncertainties of the furfuryl alcohol synthesis model assuming uniform random variables.

Parameter	Mean	Standard Deviation
$k_{\mathrm{ads}}$	[0.5372, 0.6566]	[0.0537, 0.0656]
$k_{FF}$	[5.07 $\times {10}^{-8}$, 6.20 $\times {10}^{-8}$]	[0.50 $\times {10}^{-8}$, 0.62 $\times {10}^{-8}$]
C	[1.4 $\times {10}^9$, 1.7 $\times {10}^9$]	[1.4 $\times {10}^8$, 1.7 $\times {10}^8$]

for epistemic uncertainty, and for aleatoric uncertainty a standard deviation of 10% of the nominal model parameters (Table 1) are assumed. The PEM approach was able to predict the impact of uncertain parameters on yield of furfuryl alcohol with less number of samples in comparison to the Monte-Carlo approach. For example, the 95%-confidence intervals (CIs) for yield of furfuryl alcohol in presence of deep uncertainty in model parameters is demonstrated in Figure 5a. The impact of uncertainties is less in the range 2.1 to $2.4$ $\mathrm{V}$ since the process reaches almost complete conversion of furfural to furfuryl alcohol. Moreover, the impact of uncertainties were also negligible for low values of overpotential (<1.7 $\mathrm{V}$). However, operating in this range is not desirable since the yield of furfuryl alcohol is also less (<40%).

As shown in Figure 5b, the sensitivity ratio for aleatoric and epistemic uncertainty with respect to the yield of furfuryl alcohol was calculated using the PEM approach. For the range of overpotential studied (1.4 to 2.7 $\mathrm{V}$) and given statistics in Table 2, the added epistemic uncertainty increased the overall uncertainty in the predicted yield coefficients. Please note that the epistemic contribution is minor at 2.1 and 2.4 $\mathrm{V}$ as the process reaches complete conversion of furfural to furfuryl alcohol, and the dynamic of conversion and the increased model parameter uncertainties, respectively, have a negligible effect in consequence (Figure 5b).

3.2. 4-Aminophenol with Neural Differential Equations

4-Aminophenol is an intermediate for the synthesis of pharmaceutical products such as paracetamol and phenacetin [64]. Typically, 4-aminophenol is synthesized through a chemical route by hydrogenation of nitrobenzene or nitration of phenol. On the other hand, 4-aminophenol can be synthesized through an electrochemical route from nitrobenzene (NB) with phenylhydroxylamine (PHA) as an intermediate [65]. The intermediate (PHA) further undergoes chemical conversion to produce 4-aminophenol (4AP) or electrochemical conversion to produce aniline (AN). In contrast to the chemical route, the electrochemical route does not use any additional toxic agents. In this work, 4-aminophenol (4AP) is considered as desired product, and AN is considered as an undesired product [66,67]. The reaction scheme for electrochemical synthesis of 4-aminophenol is given in Figure 6.

Based on Bakshi and Fedkiw [67], the material balance equations for concentration of NB, PHA, 4AP, and AN read as:

$$\begin{array}{cc}\hfill \frac{dNB}{dt}& =-a·r_{NB},\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill \frac{dPHA}{dt}& =a(r_{NB}-r_{AN})-r_{4AP},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \frac{d4AP}{dt}& =r_{4AP},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \frac{dAN}{dt}& =a·r_{AN},\hfill \end{array}$$

(21)

where the rate of conversion of NB to PHA is given as:

$$\begin{array}{cc}\hfill r_{NB}& =k_1·N{B}_s·u_{c1},\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill u_{c1}& =exp(-{\alpha }_1·f·E_c),\hfill \end{array}$$

(23)

$$\begin{array}{c}\hfill f=\frac{F}{RT}.\end{array}$$

(24)

In Equation (23), ${\alpha }_1$ and $E_c$ refer to the charge transfer coefficient for the desired cathodic reaction, and cathode potential, respectively. In Equation (24), R refers to gas constant and T refers to temperature. Similarly, the electrochemical synthesis of AN from PHA is expressed as:

$$\begin{array}{cc}\hfill r_{AN}& =k_2·PH{A}_s·u_{c2},\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill u_{c2}& =exp(-{\alpha }_2·f·E_c).\hfill \end{array}$$

(26)

In Equation (26), ${\alpha }_2$ refers to the charge transfer coefficient for the undesired cathodic reaction. The chemical synthesis of 4AP from PHA is defined as:

$$r_{4AP}=k_3·PHA.$$

(27)

In Equations (22) and (25), $N{B}_s$ and $PH{A}_s$ refers to surface concentration of NB and PHA, respectively. The surface concentration of NB and PHA are calculated from the bulk concentration based on Equations (28) and (29), respectively.

$$NBs=\frac{k_{mNB}·NB}{k_{mPHA}+k_2·u_{c1}},$$

(28)

$$PH{A}_s=\frac{1}{k_{mPHA}+k_2·u_{c2}}\times \left(k_{mPHA}·PHA+k_1·u_{c1}\frac{k_{mNB}·NB}{k_{mNB}+k_1·u_{c1}}\right).$$

(29)

The model parameters and initial values are summarized in

Table 3. Model parameters and dimensionless initial values for electrochemical synthesis of 4-aminophenol.

Parameter	${\mathit{k}}_1$	${\mathit{k}}_2$	${\mathit{k}}_3$	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	a	${\mathit{E}}_{\mathit{c}}$
Value	$6.03\times {10}^{-8}$	$2.71\times {10}^{-8}$	$4.77\times {10}^{-6}$	0.693	0.398	0.02	0.5
Unit	cm s−1	cm s−1	s−1	-	-	/	$\mathrm{V}$
Parameter	${\mathit{k}}_{\mathit{mNB}}$	${\mathit{k}}_{\mathit{mPHA}}$	f	NB (t = 0)	PHA (t = 0)	4AP (t = 0)	AN (t = 0)
Value	$2.84\times {10}^{-3}$	$1.73\times {10}^{-3}$	38.66	1	0	0	0
Unit	cm s−1	cm s−1	V−1	-	-	-	-

. Note that these model parameters refer to the parameter vector $\mathbf{p}$ in Equations (2), (3), and (6)–(11). Please also note that, as in the original study [66], dimensionless concentration profiles were considered. Here, the normalization is done with the NB initial concentration, $NB(t=0)$$=50\times {10}^{-6}$$\mathrm{mol}{\mathrm{cm}}^{-3}$).

In the simulation study below, the original model (Equations (18)–(21)) is used as a reference and to generate simulated measurement data. Furthermore, in the context of hybrid modeling, the following neural ODE problem is considered:

$$\begin{array}{cc}\hfill \frac{dNB}{dt}& =-a·r_{NB}+ANN(NB,PHA,4AP,AN),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill \frac{dPHA}{dt}& =a(r_{NB}-r_{AN})-r_{4AP}+ANN(NB,PHA,4AP,AN),\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \frac{d4AP}{dt}& =ANN(NB,PHA,4AP,AN),\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill \frac{dAN}{dt}& =ANN(NB,PHA,4AP,AN),\hfill \end{array}$$

(33)

where $ANN\left(·\right)$ refers to a multilayer perceptron (MLP) with two hidden layers, eight nodes per hidden layer, input and output layer with 4 nodes (Equation (1)), and tangent function as activation function. Thus, Equations (30)–(33) represent limited process knowledge, i.e., knowledge is embedded in particular in Equations (30) and (31) with the kinetic expressions for $r_{NB}$, $r_{AN}$, and $r_{4AP}$. The neural ODE system, however, also embeds an ANN for learning missing functions (Equations (32) and (33)) or possibly incomplete expressions (Equations (30) and (31)). Please note that the $ANN\left(·\right)$ term could be used exclusively in Equations (30) and (31), i.e., assuming no process knowledge, and, alternatively, additional first-principles dependencies could be added in Equations (32) and (33), i.e., further mechanistic relationships are known.

Figure 7 shows the reference simulations (Equations (18)–(21)), the artificial measurement data calculated from them under an additive measurement noise (${\sigma }_y=$ $0.1$), and the simulations of the neural ODE system (Equations (30)–(33)). For both the training phase (≤5 $\mathrm{h}$) and the prediction phase (>5 $\mathrm{h}$)), the trajectories of the neural ODE system describe the reference values without any noticeable deviations. Despite the excellent fit, it should not be ignored that the simulations of both the reference model and the determined hybrid model are subject to strong uncertainties due to uncertain model parameters, including the initial value of nitrobenzene. Note that model parameters refer to the reference model and its parameters given in Table 3 which are, however, included in the neural ODE system (Equations (30) and (31)). The aleatoric model parameter uncertainties, i.e., a standard deviation of 20% of the nominal model parameters (Table 3), affect the simulations of the reference model and the hybrid model equally.

In Figure 8, the resulting 95%-CIs are illustrated assuming a Gaussian distribution; i.e., symmetric CIs. In case of the hybrid model (Figure 8a), the uncertainties in the simulation are less amplified when compared with the reference model’s CIs (Figure 8b). These differences can be explained by the fact that the interactions and interrelationships of the system states and the model parameters are not included one-to-one in the derived neural ODE system. Accordingly, although the concentration profiles can be correctly predicted deterministically using the neuronal ODEs, a realistic prediction of the uncertainties is not possible with the hybrid model. The predictions of the hybrid model, in turn, are subject to uncertainties, which have to be taken into account in the hybrid-model-based analysis.

The different behavior in uncertainty propagation of the hybrid and first-principles model is also reflected in the parameter sensitivities. In Figure 9, GSA profiles of the hybrid and the first-principles model are summarized. Similar to the uncertainty propagation problem, the model parameters $k_1$, $k_2$, $k_3$, $k_{mNB}$, and $k_{mPHA}$ and the initial condition of nitrobenzene $NB(t=0)$ are considered random variables with statistics given in

Table 4. Epistemic uncertainties of the 4-aminophenol synthesis model assuming uniform random variables.

Parameter	Mean	Standard Deviation
$k_1$	[$4.82\times {10}^{-8}$, $7.23\times {10}^{-8}$ ]	[$0.482\times {10}^{-8}$, $0.723\times {10}^{-8}$ ]
$k_2$	[$2.16\times {10}^{-8}$, $3.25\times {10}^{-8}$ ]	[$0.216\times {10}^{-8}$, $0.325\times {10}^{-8}$ ]
$k_3$	[$3.81\times {10}^{-6}$, $5.72\times {10}^{-6}$ ]	[$0.381\times {10}^{-6}$, $0.572\times {10}^{-6}$]
$k_{mNB}$	[$2.27\times {10}^{-3}$, $3.40\times {10}^{-3}$ ]	[$0.227\times {10}^{-3}$, $0.340\times {10}^{-3}$ ]
$k_{mPHA}$	[$1.38\times {10}^{-3}$, $2.07\times {10}^{-3}$ ]	[$0.138\times {10}^{-3}$, $0.207\times {10}^{-3}$ ]
$NB(t=0)$	[0.8, 1.2]	[0.08, 0.12]

. In general, it can be concluded for both modeling approaches that $k_{mNB}$ as well as $NB(t=0)$ have the largest impact on the simulations, and thus, correspond well with the literature [66]. However, a closer look also reveals that the temporal profiles differ significantly. For example, in the case of the concentration curve of NB, there is a time lag in the sensitivity profiles when the hybrid model (Figure 9a) is compared with the first-principles model (Figure 9b). In Figure 9c,d, even an inversion of the sensitivity profiles can be seen for the PHA trajectory. Also, sensitivity contributions of the other parameters are not correctly reproduced when studying the hybrid model; see Figure 9e–h. Similar to the uncertainty propagation itself, the hybrid model cannot be used to predict parameter sensitivities of the first-principles model and the real process, respectively. These different effects are true not only for aleatoric uncertainty but also under deep uncertainty.

In the context of deep uncertainty, epistemic uncertainty is added to aleatoric uncertainty. As can be seen from Table 4, epistemic uncertainty refers to the model parameters but also the initial value of NB, and thus, take into account higher-level kinetics variations due to degradation effects or different literature values and fluctuations in the reactants, respectively. For simplicity, the total effect is considered in the uncertainty amplification, i.e., the ratio of deep uncertainty (epistemic plus aleatoric) vs aleatoric uncertainty in the simulation results. In Figure 10, a strong uncertainty amplification can be observed for the hybrid model. The epistemic uncertainty affects the PHA prediction most, i.e., the variance values are increased by factor three. In contrast, the first-principles model is less sensitive to deep uncertainty; see Figure 10b. Although all simulation results are again influenced by the epistemic uncertainty, i.e., the uncertainties in the simulations are increased, there is only an increase of 60–65%. Note that when using Equation (5), the individual global sensitivities of the quantities given in Table 4 could be calculated, and their uncertainty fraction caused by deep uncertainty in the simulations also quantified.

3.3. Computing Effort

As discussed in the previous two case studies, good approximation results can be obtained with the PEM in the uncertainty calculation for hybrid modeling; this includes the GSA calculation and the consideration of aleatoric and epistemic uncertainties in the context of deep uncertainty. Also shown above, the PEM is a useful tool to describe and propagate uncertainties in a CPU-friendly way. The low computational cost, of course, depends on two factors: (1) the deterministic sampling scheme (Equations (6)–(8)) and (2) the re-sampling strategy for GSA using appropriate subsets, ${\mathcal{P}}_i\subset \mathcal{O}$, to calculate $S\left(\mathbf{p}\right[i\left]\right),$$\forall i\in \{1,\dots ,n_p\}$ (Equation (3)). Thus, in the example of FA synthesis and three uncertain parameters ($n_{\xi }=3$), a total of 19 sample points are generated in the parameter space, and 19 model simulations must be performed accordingly for further calculation of confidence intervals and GSA assuming aleatoric uncertainty. If for the further deep uncertainty analysis, the epistemic part is added (i.e., six parameters ($n_{\theta }=6$) describing the uncertainties of the uncertainties and the variance of the three considered model parameters), the total simulation effort increases to 1387 simulation runs. Similarly, in the case of the 4AP synthesis and aleatoric uncertainty, starting with six uncertain model parameters ($n_{\xi }=6$) results in 73 sample points and model simulations, respectively. When adding the epistemic part under deep uncertainty ($n_{\theta }=12$), the overall number of model evaluations increase to 21,097 simulations. This is nevertheless a comparatively small computational effort if one compares this with the standard Monte Carlo simulations. In the case of deep uncertainty with the standard double-loop approach for uncertainty quantification, the computational effort is $\ge {10}^5$ simulations-solely for the determination of the confidence intervals. In the case of PEM, however, the generated parameter samples and the model simulations already allow a comprehensive calculation of global parameter sensitivities and to distinguish between aleatoric and epistemic effects on simulation results.

4. Conclusions

In the present work, it has been successfully demonstrated that hybrid modeling approaches can be usefully employed for the mathematical description of electrochemical synthesis problems, including batch and continuous operation. In terms of hybrid modeling, a standard hybrid architecture, i.e., the serial interconnection of a ML algorithm and a first-principles process model, was applied for the synthesis of furfuryl alcohol with good agreement to measurement data. Whereas for the 4-aminophenol synthesis problem, the highly topical hybrid approach of neural ODEs was successfully implemented in a simulation study. Moreover, regardless of the synthesis problem and hybridization strategy, it became clear that uncertainty quantification and parameter sensitivities are an important tool in model-based analysis when using hybrid process models. For example, confidence intervals provide possible safety limits for further process design and parameter sensitivities provide insight for improving model quality but also possible control variables for improving electrochemical synthesis performance.

With the presented case studies, it also became clear that epistemic uncertainty has a very large impact on model reliability, and thus, the issue of deep uncertainty should play an important role for holistic uncertainty analysis. However, this can only be implemented practically provided that powerful algorithms are available for uncertainty propagation and for the quantification of global parameter uncertainties. Once again, the point estimate method has proven to be an interesting and promising approach. The computational cost does not scale exponentially with the dimension of the uncertain parameters/factors under consideration. Moreover, the deterministic sampling allows for easy parallel computation and reproducible results.

For the neural ODEs, a notable effect was observed in the uncertainty propagation and parameter sensitivities. The hybrid model cannot be used to predict parameter sensitivities of the first-principles model and the real process, respectively. Rather, the GSA values must be considered separately for the specific use case. In the context of hybrid modeling, the GSA reveals the influence of the partially known process knowledge and the associated model parameters on the simulations, and thus, represent added value in themselves when interpreting hybrid process models.

This study deliberately considered the influence of model parameters in the context of hybrid modeling. The uncertainties of the meta-parameters of the ML algorithms were neglected. Ultimately, however, both the model parameters and the meta parameters have a major impact in hybrid modeling under deep uncertainty. Systematic consideration of both classes of parameters will be the subject of future research.