The First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Fredholm-Type Neural Integro-Differential Equations: I. Mathematical Framework

Abstract

This work presents the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type” (1st-FASAM-NIDE-F) and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type” (2nd-FASAM-NIDE-F). It is shown that the 1st-FASAM-NIDE-F methodology enables the most efficient computation of exactly-determined first-order sensitivities of decoder response with respect to the optimized NIDE-F parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIDE-F decoder, hidden layers, and encoder. The 2nd-FASAM-NIDE-F methodology enables the computation, with unparalleled efficiency, of the second-order sensitivities of decoder responses with respect to the optimized/trained weights, requiring only as many large-scale computations for solving the 2nd-Level Adjoint Sensitivity System (2nd-LASS) as there are non-zero feature functions of parameters. The application of both the 1st-FASAM-NIDE-F and the 2nd-FASAM-NIDE-F methodologies is illustrated in an accompanying work (Part II) by considering a paradigm heat transfer model.

1. Introduction

The introduction of “Neural Ordinary Differential Equations” (NODE) models by Chen et al. [1] has significantly advanced the versatility and applicability of neural-nets by providing an explicit connection between deep feed-forward neural networks and dynamical systems. By using differential equation solvers to learn dynamics through continuous deep learning models of neural networks, NODE models provide a bridge between modern deep learning and traditional numerical modeling while offering trade-offs between efficiency, memory costs, and accuracy, as demonstrated by various applications [1,2,3,4,5,6,7,8,9]. However, NODE models are limited to describing systems that are instantaneous, since each time-step is determined locally in time, without contributions from the state of the system at other times.

In contradistinction to differential equations, integral equations (IEs) model global spatio-temporal relations, which are learned through an IE-solver (see, e.g., [10]) which samples the domain of integration continuously. Due to their non-local behavior, IE-solvers are suitable for modeling complex dynamics. Zappala et al. [11] have introduced the Neural Integral Equation (NIE) and the Attentional Neural Integral Equation (ANIE), which can be used to infer the spatio-temporal relations that generated the data, thus enabling the continuous learning of non-local dynamics with arbitrary time resolution [11,12]. Often, ordinary and/or partial differential equations can be recast in integral-equation forms that can be solved more efficiently using IE-solvers, as exemplified in [13,14,15].

Zappala et al. [16] have also developed a deep learning method called Neural Integro-Differential Equation (NIDE), which “learns” an integro-differential equation (IDE) whose solution approximates data sampled from given non-local dynamics. The motivation for using NIDE stems from the need to model systems that present spatio-temporal relations which transcend local modeling, as illustrated by the pioneering works of Volterra on population dynamics [17]. Combining the properties of differential and integral equations, IDEs also present properties that are unique to their non-local behavior [18,19,20], with applications in computational biology, physics, engineering, and applied sciences [18,19,20,21,22,23,24,25,26].

All neural-nets are trained by minimizing a user-chosen “loss functional” which aims at representing the discrepancy between the output produced by the respective net’s decoder and a user-chosen “reference solution”. The neural-net is optimized to reproduce the underlying physical system as closely as possible. However, the physical system modeled by a neural-net comprises parameters that stem from measurements and/or computations which are subject to uncertainties. Therefore, even if the neural-net reproduces perfectly the underlying system, the uncertainties inherent in the system’s parameters would propagate to the subsequent results produced by the decoder. Quantifying the uncertainties in the decoder’s response can only be performed if the sensitivities of the decoder’s response with respect to the neural-net’s optimized parameters are known.

In addition to scalar-valued parameters, neural-nets often comprise scalar-valued functions (e.g., correlations, material properties, etc.) of the model’s scalar parameters. Calling such scalar-valued functions as “features of primary model parameters”, Cacuci [27] has recently introduced the “nth-Order Features Adjoint Sensitivity Analysis Methodology for Nonlinear Systems (nth-FASAM-N)”. The nth-FASAM-N enables the most efficient computation of the exact expressions of arbitrarily high-order sensitivities of model responses with respect to the model’s “features”. Subsequently, the sensitivities of the responses with respect to the primary model parameters are determined, analytically and trivially, by using the well-known “chain-rule of differentiation” to obtain the response sensitivities with respect to the model’s features/functions of parameters.

Based on the general framework of the nth-FASAM-N methodology [27], Cacuci has developed specific sensitivity analysis methodologies for NODE-nets, as follows: the “First-Order and Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (1st-FASAM-NODE)” and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (2nd-FASAM-NODE)” [28]. The 1st-FASAM-NODE and the 2nd-FASAM-NODE are pioneering sensitivity analysis methodologies which enable the computation, with unparalleled efficiency, of exactly-determined first-order and, respectively, second-order sensitivities of decoder response with respect to the optimized/trained weights involved in the NODE’s hidden layers, decoder, and encoder.

Two important families of IDEs are the Volterra and the Fredholm equations. In a Volterra IDE, the interval of integration grows linearly during the system’s dynamics, while in a Fredholm IDE the interval of integration is fixed during the dynamic-history of the system, but at any given time instance within this interval, the system depends on the past, present, and future states of the system. By applying the general concepts underlying the nth-FASAM-N methodology [27], Cacuci [29,30] has also developed the general methodologies underlying the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (2nd-FASAM-NIE-F)” and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (2nd-FASAM-NIE-V)”. The 2nd-FASAM-NIE-F encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (1st-FASAM-NIE-F), while the 2nd-FASAM-NIE-V encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (1st-FASAM-NIE-V)”. The 1st-FASAM-NIE-F and 1st-FASAM-NIE-V methodologies, respectively, enable the computation, with unparalleled efficiency, of exactly-determined first-order sensitivities of decoder response with respect to the NIE-parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIE-net. The 2nd-FASAM-NIE-F and 2nd-FASAM-NIE-V methodologies, respectively, enable the computation (with unparalleled efficiency) of exactly-determined second-order sensitivities of decoder response with respect to the NIE-parameters, requiring only as many “large-scale” computations as there are first-order sensitivities with respect to the feature functions.

Generalizing the methodologies presented in [29,30], this work presents the “First- and Second Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type” abbreviated as “1st-FASAM-NIDE-F” and “2nd-FASAM-NIDE-F”, respectively. These methodologies are also based on the general principles underlying the nth-FASAM-N methodology [24]. The 1st-FASAM-NIDE-F is presented in Section 2, while the 2nd-FASAM-NIDE-F is presented in Section 3, in the sequel. The discussion presented in Section 4 concludes this work by highlighting the unparalleled efficiency of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies, respectively, for computing exact first- and second-order sensitivities, respectively, of decoder responses to model parameters in optimized NIE-F networks. The accompanying work [31] presents an illustrative application of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies to a paradigm heat conduction model which admits exact closed-form solutions/expressions for all quantities of interest and is of fundamental importance in many scientific fields [32,33,34,35,36,37].

2. First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (1st-FASAM-NIDE-F)

The mathematical expression of the network of nonlinear Fredholm-type Neural Integro-Differential Equations (NIDE-F) considered in this work generalizes the NIDE-net model introduced in [16] and is represented in component form by the following system of Nth-order integro-differential equations:

$$\begin{array}{l}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]\frac{d^n{h}_i\left(t\right)}{d{t}^n}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t\in \left[t_0,t_f\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.\end{array}$$

(1)

The boundary conditions, imposed at the “initial time” $t=t_0$ and/or “final time” $t=t_f$ on the functions $h_i\left(t\right)$ and their time-derivatives associated with the encoder of the NIDE-F net represented by Equation (1) are represented in operator form as follows:

$$B_j\left[h\left(t\right);f\left(\theta \right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(2)

The quantities appearing in Equations (1) and (2) are defined as follows:

(i)

The real-valued scalar quantities $t$ and $\tau$, $t_0\le t,\tau \le t_f$, are time-like independent variables which parameterize the dynamics of the hidden/latent neuron units. Customarily, the variable $t$ is called the “global time” while the variable $\tau$ is called the “local time”. The initial time-value is denoted as $t_0$ while the stopping time-value is denoted as $t_f$. Thus, the dynamics modeled by Equation (1) depends both on non-local effects, as well as on instantaneous information.

(ii)

The components of the $TH$-dimensional vector-valued function $h\left(t\right)\triangleq {\left[h_1\left(t\right),\dots ,h_{TH}\left(t\right)\right]}^{†}$ represents the hidden/latent neural networks; $TH$ denotes the total number of components of $h\left(t\right)$. In this work, the symbol “$\triangleq$” will be used to denote “is defined as” or, equivalently, “is by definition equal to”. The various vectors will be considered to be column vectors. Typically, vectors will be denoted using bold lower-case letters. The dagger “$†$” symbol will be used to denote “transposition”.

(iii)

The components of the column-vector $\theta \triangleq {\left[{\theta }_1,\dots ,{\theta }_{TW}\right]}^{†}$ represent the “primary” network parameters, namely scalar learnable adjustable parameters/weights, in all of the latent neural-nets, including the encoders(s) and decoder(s), where $TW$ denotes the total number of adjustable parameters/weights.

(iv)

The scalar-valued components $f_i\left(\theta \right)$, $i=1,\dots ,TF$, of the vector-valued function $f\left(\theta \right)\triangleq {\left[f_1\left(\theta \right),\dots ,f_{TF}\left(\theta \right)\right]}^{†}$ represent the “feature/functions of the primary model parameters”. The quantity $TF$ denotes the total number of such feature functions comprised in the NIDE-F. In particular, all of the model parameters that might appear solely in the boundary and/or initial conditions are considered to be included among the components of the vector $\theta$. In general, $f\left(\theta \right)$ is a nonlinear vector-valued function of $\theta$. The total number of feature functions must necessarily be smaller than the total number of primary parameters (weights), i.e., $TF<TW$. When the NIDE-F comprises only primary parameters, it is considered that $f_i\left(\theta \right)\equiv {\theta }_i$ for all $i=1,\dots ,TW\equiv TF$.

(v)

The functions ${\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]$ model the dynamics of the neurons in a latent space where the local time integration occurs, while the functions ${\phi }_{i,j}\left[f\left(\theta \right);t\right]$ map the local space back to the original data space. The functions $g_i\left[h\left(t\right);f\left(\theta \right)\right]$ model additional dynamics in the original data space. In general, these functions are nonlinear in their arguments.

(vi)

The functions $c_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]$ are coefficient-functions, which may depend nonlinearly on the functions $h\left(t\right)$ and $f\left(\theta \right)$, associated with the order, $n=1,\dots ,N$, of the time-derivatives $d^n{h}_i\left(t\right)/d{t}^n$ of the functions $h_i\left(t\right)$.

(vii)

The operators $B_j\left[h\left(t\right);f\left(\theta \right);t\right]$, $j=1,\dots ,BC$, represent boundary conditions associated with the encoder and/or decoder, imposed at $t=t_0$ and/or at $t=t_f$ on the functions $h_i\left(t\right)$ and on their time-derivatives; the quantity “BC” denotes the “total number of boundary conditions”.

Customarily, the NIDE-F net is “trained” by minimizing a user-chosen loss functional representing the discrepancy between a reference solution (“target data”) and the output produced by the NIDE-F decoder. The “training” process produces “optimal” values for the primary parameters $\theta \triangleq {\left[{\theta }_1,\dots ,{\theta }_{TW}\right]}^{†}$, which will be denoted in this work by using the superscript “zero”, as follows: ${\theta }^0\triangleq {\left[{\theta }_1^0,\dots ,{\theta }_{TW}^0\right]}^{†}$. Using these optimal/nominal parameter values to evaluate the NIDE-F net yields the optimal/nominal solution $h^0\left(t,x\right)\triangleq {\left[h_1^0\left(t\right),\dots ,h_{TH}^0\left(t\right)\right]}^{†}$ which will satisfy the following form of Equation (1):

$$\begin{array}{l}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]\frac{d^n{h}_i^0\left(t\right)}{d{t}^n}}=g_i\left[h^0\left(t\right);f\left({\theta }^0\right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left({\theta }^0\right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h^0\left(\tau \right);f\left({\theta }^0\right);\tau \right]}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(3)

subject to the following optimized/trained boundary conditions:

$$B_j\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(4)

After the NIDE-F net is optimized to reproduce the underlying physical system as closely as possible, the subsequent responses of interest are no longer “loss functionals” but become specific functionals of the NIDE-F’s “decoder” output, which can be generally represented by the functional $R\left[h;f\left(\theta \right)\right]$ defined below:

$$R\left[h;f\left(\theta \right)\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h\left(t\right);f\left(\theta \right);t\right]dt}.$$

(5)

The function $D\left[h\left(t\right);f\left(\theta \right);t\right]$ models the decoder. The scalar-valued quantity $R\left[h;f\left(\theta \right)\right]$ is a functional of $h\left(t,x\right)$ and $f\left(\theta \right)$, and represents the NIDE-F’s decoder-response. At the optimal/nominal parameter values, i.e., at $\theta ={\theta }^0$, the decoder response takes on the following formal form:

$$R\left[h^0;f\left({\theta }^0\right)\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]dt}.$$

(6)

The physical system modeled by the NIDE-F net comprises parameters that stem from measurements and/or computations. Consequently, even if the NIDE-F net models perfectly the underlying physical system, the NIDE-F’s optimal weights/parameters are unavoidably afflicted by uncertainties stemming from the parameters underlying the physical system. Hence, it is important to quantify the uncertainties induced in the decoder output, $R\left[h;f\left(\theta \right)\right]$, by the uncertainties that afflict the parameters/weights underlying the physical system modeled by the NIDE-F net. The relative contributions of the uncertainties afflicting the optimal parameters to the total uncertainty in the decoder response are quantified by the sensitivities of the NIDE-F decoder-response with respect to the optimized NIDE-F parameters. The general methodology for computing the first-order sensitivities of the decoder output, $R\left[h;f\left(\theta \right)\right]$, with respect to the components of the feature function $f\left(\theta \right)$, and with respect to the primary model parameters ${\theta }_1,\dots ,{\theta }_{TW}$, will be presented in this Section.

The known nominal values ${\theta }^0$ of the primary model parameters (“weights”) characterizing the NIDE-V net will differ from the true but unknown values $\theta$ of the respective weights by variations denoted as $\delta \theta \triangleq \theta -{\theta }^0$. The variations $\delta \theta \triangleq \theta -{\theta }^0$ will induce corresponding variations $\delta f\triangleq f\left(\theta \right)-f^0$, $f^0\triangleq f\left({\theta }^0\right)$, in the feature functions. The variations $\delta \theta$ and $\delta f$ will induce, through Equation (1), variations $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$ around the nominal/optimal functions $h^0\left(t\right)$. In turn, the variations $\delta f\triangleq f\left(\theta \right)-f^0$ and $v^{\left(1\right)}\left(t;x\right)$ will induce variations $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f;t\right)$ in the NIE decoder’s response.

The “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (1st-FASAM-NIDE-F)” aims at obtaining the exact expressions of the first-order sensitivities (i.e., functional derivatives) of the decoder’s response with respect to the feature function and the primary model parameters, followed by the most efficient computation of these sensitivities. The 1st-FASAM-NIDE-F will be established by applying the same principles as those underlying the 1st-FASAM-N [24] methodology. The fundamental concept for defining the sensitivity of an operator-valued quantity $R\left(x\right)$ with respect to variations $\delta x$ in a neighborhood around the nominal values $x^0$, has been shown in 1981 by Cacuci [38] to be provided by the 1st-order Gateaux- (G-) variation $\delta R\left(x^0;\delta x\right)$ of $R\left(x\right)$, which is defined as follows:

$$\delta R\left(x^0;\delta x\right)\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}\left[R\left(x^0+\epsilon \delta x\right)\right]\right\}}_{\epsilon =0}\triangleq \underset{\epsilon \to 0}{\lim }{\displaystyle \frac{R\left(x^0+\epsilon \delta x\right)-R\left(x^0\right)}{\epsilon }},$$

(7)

for a scalar $\epsilon$ and for arbitrary vectors $\delta x$ in a neighborhood $\left(x^0+\epsilon \delta x\right)$ around $x^0$. When the G-variation $\delta R\left(x^0;\delta x\right)$ is linear in the variation $\delta x$, it can be written in the form $\delta R\left(x^0;\delta x\right)={\left\{\partial R/\partial x\right\}}_{x^0}\delta x$, where ${\left\{\partial R/\partial x\right\}}_{x^0}$ denotes the first-order G-derivative of $R\left(x\right)$ with respect to $x$, evaluated at $x^0$.

Applying the definition provided in Equation (7) to Equation (5) yields the following expression for the first-order G-variation $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$ of the response $R\left[h;f\left(\theta \right)\right]$:

$$\begin{array}{ll}\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)& ={\left\{{\displaystyle \frac{d}{d\epsilon }}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]dt}\right\}}_{\epsilon =0}\\ & =\hspace{0.17em}{\left\{\delta R\left(h^0;f^0;\delta f\right)\right\}}_{dir}+{\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}\hspace{0.17em},\end{array}$$

(8)

where the “direct effect term” arises directly from variations $\delta f$ and is defined as follows:

$${\left\{\delta R\left(h^0;f^0;\delta f\right)\right\}}_{dir}\triangleq {\displaystyle \sum _{i=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt{\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_i}\delta f_i\right\}}_{{\theta }^0}}},$$

(9)

and where the “indirect effect term” arises indirectly, through the variations $v^{\left(1\right)}\left(t\right)$ in the hidden state functions $h\left(t\right)$, is defined as follows:

$${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}\triangleq {\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}{v}_i^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}}.$$

(10)

The direct-effect term can be quantified using the nominal values $\left(h^0;f^0\right)$ but the indirect-effect term can be quantified only after determining the variations $v^{\left(1\right)}\left(t\right)$, which are caused by the variations $\delta f$ through the NIDE-F net defined in Equation (1).

The first-order relationship between the variations $v^{\left(1\right)}\left(t\right)$ and $\delta f$ is obtained from the first-order G-variations of Equations (1) and (2). The first-order G-variations of Equations (1) and (2), respectively, are obtained, by definition, as follows:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f;t\right]\frac{d^n\left[h_i^0\left(t\right)+\epsilon \delta h_i\left(t\right)\right]}{d{t}^n}}\right\}}_{\epsilon =0}\\ ={\left\{{\displaystyle \frac{d{g}_i\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f\right]}{d\epsilon }}\right\}}_{\epsilon =0}\\ +{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left({\theta }^0\right)+\epsilon \delta f;t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h^0\left(\tau \right)+\epsilon v^{\left(1\right)}\left(\tau \right);f\left({\theta }^0\right)+\epsilon \delta f;\tau \right]}\right\}}_{\epsilon =0}.\end{array}$$

(11)

$${\left\{{\displaystyle \frac{d}{d\epsilon }}{B}_j\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f;t\right]\right\}}_{\epsilon =0}=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0;\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(12)

Carrying out the operations indicated in Equations (11) and (12) yields the following NIDE-F net of Fredholm-type for the function $v^{\left(1\right)}\left(t\right)$:

$$\begin{array}{l}{\left\{{\displaystyle \sum _{k=1}^{TH}{\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)+{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h;f\left(\theta \right);t\right]\frac{d^n{v}_i^{\left(1\right)}\left(t\right)}{d{t}^n}}}}}\right\}}_{{\theta }^0}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{{\theta }^0}\\ -{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k\right\}}_{{\theta }^0}},\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(13)

$$\begin{array}{l}{\left\{{\displaystyle \sum _{k=1}^{TH}\frac{\partial B_j\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}}_{{\theta }^0}=-{\left\{{\displaystyle \sum _{k=1}^{TF}\frac{\partial B_j\left[h;f\left(\theta \right);t\right]}{\partial f_k}}\delta f_k\right\}}_{{\theta }^0},\\ at\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}j=1,\dots ,BC;\end{array}$$

(14)

where:

$$\begin{array}{l}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\triangleq -{\displaystyle \sum _{n=1}^N\frac{\partial c_{i,n}\left[f\left(\theta \right);t\right]}{\partial f_k}\frac{d^n{h}_i\left(t\right)}{d{t}^n}}+{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\\ +{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,\dots ,TF.\end{array}$$

(15)

The NIDE-F net represented by Equations (13) and (14) is called [24] the “1st-Level Variational Sensitivity system (1st-LVSS) and its solution, $v^{\left(1\right)}\left(t\right)$ is called [24] the “1st-level variational function”. All of the quantities in Equations (13) and (14) are to be computed at the nominal parameter values, but the respective indication has not been explicitly shown in order to simplify the notation.

It is important to note that the 1st-LVSS is linear in the variational function $v^{\left(1\right)}\left(t\right)$. Therefore, the 1st-LVSS represented by Equation (13) can be written in matrix-vector form as follows:

$$L\left[h;f\left(\theta \right);t\right]v^{\left(1\right)}\left(t\right)=\hspace{0.17em}{Q}^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\delta f\left(\theta \right),$$

(16)

where the $TH\times TF$-dimensional rectangular matrix $Q^{\left(1\right)}\left[h;f\left(\theta \right);t\right]$ comprises as components the quantities $q_{i,k}^{\left(1\right)}\left(h;f\right)$ defined in Equation (15), while the components of the $TH\times TH$ square matrix $L\left[h;f\left(\theta \right);t\right]\triangleq {\left[L_{ik}\right]}_{TH\times TH}$ are operators (algebraic, differential, integral) defined below, for $i,k=1,\dots ,TH$:

$$\begin{array}{l}{L}_{ii}{v}_i^{\left(1\right)}\left(t\right)\triangleq {\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}{v}_i^{\left(1\right)}\left(t\right)}+{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h;f\left(\theta \right);t\right]\frac{d^n{v}_i^{\left(1\right)}\left(t\right)}{d{t}^n}}\\ -{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_i\left(\tau \right)}{v}_i^{\left(1\right)}\left(\tau \right)d\tau }-{\displaystyle \frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{v}_i^{\left(1\right)}\left(t\right)\hspace{0.17em};\end{array}$$

(17)

$$\begin{array}{l}{L}_{ik}{v}_k^{\left(1\right)}\left(t\right)\triangleq {\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}-{\displaystyle \frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}}{v}_k^{\left(1\right)}\left(t\right)\\ -{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}}{v}_k^{\left(1\right)}\left(\tau \right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i,k=1,\dots ,TH.\end{array}$$

(18)

Note that the 1st-LVSS would need to be solved anew for each variation $\delta F_j$, $j=1,\dots ,TF$, in order to determine the corresponding function $v^{\left(1\right)}\left(t\right)$, which is prohibitively expensive computationally if $TF$ is a large number. The need for repeatedly solving the 1st-LVSS can be avoided if the variational function $v^{\left(1\right)}\left(t\right)$ could be eliminated from appearing in the expression of the indirect-effect term defined in Equation (10). This goal can be achieved [24] by expressing the right-side of Equation (10) in terms of the solutions of the “1st-Level Adjoint Sensitivity System (1st-LASS)” to be constructed next. The construction of this 1st-LASS will be performed in a Hilbert space comprising elements of the same form as $v^{\left(1\right)}\left(t\right)\in {\mathrm{H}}_1\left({\Omega }_t\right)$, defined on the domain ${\Omega }_t\triangleq t\in \left[t_0,t_f\right]$. This Hilbert space is endowed with an inner product of two elements ${\chi }^{(1)}\left(t\right)\triangleq {\left[{\chi }_1^{\left(1\right)}\left(t\right),\dots ,{\chi }_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\Omega }_t\right)$ and ${\eta }^{(1)}\left(t\right)\triangleq {\left[{\eta }_1^{\left(1\right)}\left(t\right),\dots ,{\eta }_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\Omega }_t\right)$, denoted as ${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_1$ and defined as follows:

$${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_1\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\left[{\chi }^{(1)}\left(t\right)\right]}^{†}{\eta }^{(1)}\left(t\right)dt}=\hspace{0.17em}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_i^{\left(1\right)}\left(t\right){\eta }_i^{\left(1\right)}\left(t\right)dt}}.$$

(19)

The next step is to construct the inner product of Equation (13) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\Omega }_t\right)$, where the superscript “(1)” indicates “1st-Level”, to obtain the following relationship:

$${〈a^{\left(1\right)}\left(t\right),L\left[h;f\left(\theta \right);t\right]v^{\left(1\right)}\left(t\right)〉}_1=\hspace{0.17em}{〈a^{\left(1\right)}\left(t\right),\hspace{0.17em}{Q}^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\delta f\left(\theta \right)〉}_1.$$

(20)

The terms appearing in Equation (20) are to be computed at the nominal values $\left(h^0;F^0\right)$ but the respective notation has been omitted for simplicity.

Using the definition of the adjoint operator in ${\mathrm{H}}_1\left({\Omega }_t\right)$, the term on the left-side of Equation (20) is integrated by parts and the order of summations is reversed to obtain the following relation:

$${〈a^{\left(1\right)}\left(t\right),L\left[h;f\left(\theta \right);t\right]v^{\left(1\right)}\left(t\right)〉}_1=\hspace{0.17em}{〈v^{\left(1\right)}\left(t\right),A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]a^{\left(1\right)}\left(t\right)〉}_1+P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right),$$

(21)

where the operator $A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\triangleq L^{\ast }\left[h;f\left(\theta \right);t\right]$ denotes the formal adjoint of the operator $L\left[h;f\left(\theta \right);t\right]$ and where $P\left(h;F;v^{\left(1\right)};a^{\left(1\right)}\right)$ represents the scalar-valued bilinear concomitant evaluated on the boundary $t=t_0$ and/or $t=t_f$. Note that the $TH\times TH$ matrix valued operator $A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\triangleq {\left\{A_{ij}^{\left(1\right)}\left[h;f\left(\theta \right);t\right]\right\}}_{TH\times TH}$ acts linearly on the vector $a^{\left(1\right)}\left(t\right)$. The “star” superscript (*) will be used in this work to denote “formal adjoint operator”.

It follows from Equations (20) and (21) that the following relation holds:

$${〈v^{\left(1\right)}\left(t\right),A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]〉}_1={〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right).$$

(22)

The term on the left-side of Equation (22) is now required to represent the indirect effect term defined in Equation (10) by imposing the following relation:

$$A^{\left(1\right)}\left[h;f\left(\theta \right);t\right]a^{\left(1\right)}\left(t\right)={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h\left(t\right)}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{t}_0<t<t_f.$$

(23)

Using Equations (22) and (23) in Equation (10) yields the following expression for the indirect effect term:

$${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}={\left\{{〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\right\}}_{{\theta }^0}.$$

(24)

The boundary conditions accompanying Equation (23) for the function $a^{\left(1\right)}\left(t\right)$ are now chosen at the time values $t=t_f$ and/or $t=t_0$ so as to eliminate all unknown values of the 1st-level variational function $v^{\left(1\right)}\left(t\right)$ from the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ which remain after implementing the initial conditions provided in Equation (2). These boundary conditions for the function $a^{\left(1\right)}\left(t\right)$ can be represented in operator form as follows:

$$B_j^{\ast }\left[a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(25)

The Fredholm-like NIDE net represented by Equations (23) and (25) will be called the “1st-Level Adjoint Sensitivity System” and the solution, $a^{(1)}\left(t\right)$, will be called the “1st-level adjoint sensitivity function”. The 1st-LASS is solved using the nominal/optimal values for the parameters and for the function $h\left(t\right)$ but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of any parameter variations so it needs to be solved just once to obtain the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$ The 1st-LASS is linear in $a^{(1)}\left(t\right)$ but is, in general, nonlinear in $h\left(t;x\right)$.

Adding the result obtained in Equation (24) for the indirect-effect term ${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}$ to the result obtained in Equation (9) for the direct-effect term yields the following expression for the first-order G-differential of the response $R\left[h;f\left(\theta \right)\right]$:

$$\begin{array}{l}\delta R\left(h^0;f^0;v^{\left(1\right)};\delta F\right)={\left\{{〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\right\}}_{{\theta }^0}\\ +{\displaystyle \sum _{i=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt{\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_i}\delta f_i\right\}}_{{\theta }^0}}}\triangleq {\left\{{\displaystyle \sum _{i=1}^{TF}{R}^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\delta f_i}\right\}}_{{\theta }^0},\end{array}$$

(26)

where $R^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq \partial R\left[u\left(x\right);f\left(\theta \right)\right]/\partial f_i$ denotes the first-order sensitivity of the response $R\left[u\left(x\right);f\left(\theta \right)\right]$ with respect to the components $f_i$ of the “feature”. Each sensitivity $R^{\left(1\right)}\left[i;u\left(x\right);a^{\left(1\right)}\left(x\right);f\left(\alpha \right)\right]$ is obtained by identifying the expression that multiplies the corresponding variation $\delta f_i$ and can be represented formally in the following integral form:

$$R^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{S}^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\hspace{0.17em}dt};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TF.$$

(27)

The functions $S^{\left(1\right)}\left[i;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ will be subsequently used for determining the exact expressions of the second-order sensitivities of the response with respect to the components of the feature function $f\left(\theta \right)$ of model parameters.

In the following subsections, the detailed forms of the 1st-LASS are provided for first-order (n = 1) and, respectively, second-order (n = 2) Fredholm-like NIDE.

2.1. First-Order Neural Integral Equations of Fredholm-Type (1st-NIDE-F)

The representation of the first-order $\left(n=1\right)$ neural integral equations of Fredholm-type (1st-NIDE-F) is provided below, for $\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH$:

$$c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em}.$$

(28)

The typical boundary conditions provided at $t=t_0$ (“encoder”) are as follows:

$$h_i\left(t_0\right)=e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em},$$

(29)

where the scalar values $e_i$ are known, albeit imprecisely, since they are considered to stem from experiments and/or computations. Equations (28) and (29) are customarily considered an “initial value (NIDE-F) problem” although the independent variable t could represent some other physical entity (e.g., space, energy, etc.) rather than time.

The 1st-LVSS for the function $v^{\left(1\right)}\left(t\right)$ is obtained by G-differentiating Equations (28) and (29), and has the following particular forms of Equations (13) and (14) for $n=1$:

$$\begin{array}{l}{\left\{c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{{\theta }^0}{\displaystyle \frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}}+{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{{\theta }^0}\\ -{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f\right)\right\}}_{{\theta }^0}\delta f_k},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(30)

$$v_i^{\left(1\right)}\left(t_0\right)=\delta e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em};$$

(31)

where:

$$\begin{array}{l}{q}_{i,k}^{\left(1\right)}\left(h;f\right)\triangleq -{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }\\ +{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,\dots ,TF.\end{array}$$

(32)

The 1st-LASS is constructed by using Equation (19) to form the inner product of Equation (30) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\Omega }_t\right)$ to obtain the following relationship:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_1\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(33)

Examining the structure of the left-side of Equation (33) reveals that the bilinear concomitant will arise from the integration by parts of the first term the on the left-side of Equation (33) to obtain the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}dt}}=P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}}{dt}dt}},\end{array}$$

(34)

where the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ has the following expression, by definition:

$$P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq {\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right]v_i^{\left(1\right)}\left(t_f\right)-a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]v_i^{\left(1\right)}\left(t_0\right)\right\}}.$$

(35)

The second term on the left-side of Equation (33) will be recast in its “adjoint form” by reversing the order of summations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ to an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}dt}}\hspace{0.17em}.$$

(36)

The third term on the left-side of Equation (33) is now recast in its “adjoint form” by reversing the order of summations and integrations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\hspace{0.17em}{a}_i^{\left(1\right)}\left(t\right)}}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}\hspace{0.17em}}{v}_k^{\left(1\right)}\left(\tau \right)d\tau }\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \hspace{0.17em}{v}_i^{\left(1\right)}\left(\tau \right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_i\left(\tau \right)}}}}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right){\phi }_{k,j}\left[f\left(\theta \right);t\right]\hspace{0.17em}}dt}.\end{array}$$

(37)

The fourth term on the left-side of Equation (33) will be recast in its “adjoint form” by reversing the order of summations and integrations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}.$$

(38)

Using the results obtained in Equations (34)–(38) in the left-side of Equation (33) yields the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_1\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ =P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)dt\left\{-\frac{d}{dt}\right.}}\left[a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right]\\ +{\displaystyle \frac{\partial c_1\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\\ \left.-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\right\}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(39)

The relation in Equation (39) is rearranged as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta F_k}}-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)dt\left\{-\frac{d}{dt}\right.}}\left[a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right]+{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}\\ -{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\left.-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\right\}.\end{array}$$

(40)

The term on the right-side of Equation (40) is now required to represent the “indirect-effect” term defined in Equation (10), which is achieved by requiring the components of the function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ to satisfy the following system of first-order NIDE-F equations:

$$\begin{array}{l}-{\displaystyle \frac{d}{dt}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}+{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}\\ -{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\\ ={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.\end{array}$$

(41)

The relation obtained in Equation (41) is the explicit form of the relation provided in Equation (23) for the particular case when $n=1$, i.e., when considering first-order neural integral equations of Fredholm-type (1st-NIDE-F).

The unknown values $v_i\left(t_f\right)$ in the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ in Equation (40) are eliminated by imposing the following final-time conditions:

$$a_i^{\left(1\right)}\left(t_f\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(42)

It follows from Equations (33)–(42) and (31) that the indirect-effect term defined in Equation (10) has the following expression in terms of the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$:

$${\left\{\delta R\left(h;f;a^{(1)}\right)\right\}}_{ind}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]\delta e_i\hspace{0.17em}}.$$

(43)

The first-order NIDE-F obtained in Equations (41) and (42) represents the explicit form for the particular case n = 1 of the 1st-LASS represented, in general, by Equations (23) and (25). To obtain the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$, the 1st-LASS is solved backwards in time (globally) using the nominal/optimal values for the parameters and for the function $h\left(t\right)$ but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of any parameter variations so it needs to be solved just once to obtain the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$ The 1st-LASS is linear in $a^{(1)}\left(t\right)$ but is, in general, nonlinear in $h\left(t;x\right)$.

Using the results obtained in Equations (43) and (9) in Equation (8) yields the following expression for the G-variation $\delta R\left(h^0;F^0;v^{\left(1\right)};\delta f\right)$, which is seen to be linear in the variations $\delta f_j$, $j=1,\dots ,TF$, in the model’s feature functions (induced by variations in the model’s primary parameters) and the variations $\delta e_i$, $i=1,\dots ,TH$ in the decoder’s initial conditions:

$$\begin{array}{l}\delta R\left[h\left(t\right);f;a^{(1)}\left(t\right);\delta f\right]\triangleq {\displaystyle \sum _{j=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\frac{\partial D\left[h\left(t\right);f;t\right]}{\partial F_j}\delta F_j}}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]\delta e_i}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{j=1}^{TF}{q}_{jk}^{\left(1\right)}\left(h;f;t\right)\delta f_j}}\triangleq {\displaystyle \sum _{j=1}^{TF}\frac{\partial R}{\partial f_j}}\delta f_j+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial e_i}}\delta e_i\hspace{0.17em}.\end{array}$$

(44)

The expression in Equation (44) is to be satisfied at the nominal/optimal values for the respective model parameters, but this fact has not been indicated explicitly in order to simplify the notation.

Identifying in Equation (44) the expressions that multiply the variations $\delta e_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the encoder’s initial conditions:

$${\displaystyle \frac{\partial R}{\partial e_i}}=a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\delta \left(t-t_0\right)dt};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.$$

(45)

It is apparent from Equation (45) that the sensitivities $\partial R/\partial e_i$ are functionals of the form predicted in Equation (27). It is also apparent from Equation (45) that the sensitivities $\partial R/\partial e_i$ are proportional to the values of the respective component $a_i^{\left(1\right)}\left(t_0\right)$ of the 1st-level adjoint function evaluated at the initial-time $t=t_0$. This relation provides an independent mechanism for verifying the correctness of solving the 1st-LASS from $t=t_f$ to $t=t_0$ (backwards in time) since the sensitivities $\partial R/\partial e_i$ can be computed independently of the 1st-LASS by using finite differences of appropriately high-order in conjunction with known variations $\delta e_i\hspace{0.17em}$ and the correspondingly induced variations in the decoder response. Special attention needs to be devoted, however, to ensure that the respective finite-difference formula is accurate, which may need several trials with different values chosen for the variation $\delta e_i\hspace{0.17em}$.

It also follows from Equations (44) and (32) that the sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ have the following expressions, written in the form of Equation (27):

$$\partial R/\partial f_j\triangleq R^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{S}_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}\hspace{0.17em}dt;\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF;$$

(46)

where

$$\begin{array}{l}{S}_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}}-{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(f;t;t\right)}{\partial f_j}}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF.\end{array}$$

(47)

The subscript “1” attached to the quantity $S_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ indicates that this quantity refers to a “first-order” NIDE-F net, while the superscript “(1)” indicates that this quantity refers to “first-order” sensitivities.

The sensitivities with respect to the primary model parameters can be obtained by using the result shown in Equation (46) together with the “chain rule” of differentiating compound functions, as follows:

$${\displaystyle \frac{\partial R}{\partial {\theta }_j}}={\displaystyle \sum _{i=1}^{TF}\frac{\partial R}{\partial f_i}\frac{\partial f_i}{\partial {\theta }_j}}\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TW.$$

(48)

When there only model parameters (i.e., there are no feature functions of model parameters), then $f_i\left(\theta \right)\equiv {\theta }_i$ for all $i=1,\dots ,TF\triangleq TW$, and the expression obtained in Equation (46) yields directly the first-order sensitivities $\partial R/\partial {\theta }_j$, for all $j=1,\dots ,TW$. In this case, all of the sensitivities $\partial R/\partial {\theta }_j$, for all $j=1,\dots ,TW$ would be obtained by computing integrals (using quadrature formulas). In contradistinction, when features of parameters can be established, only $TF$ $\left(TF<TW\right)$ integrals would need to be computed (using quadrature formulas) to obtain the $\partial R/\partial F_j$, $j=1,\dots ,TF$; the sensitivities with respect to the model parameters would subsequently be obtained analytically using the chain-rule provided in Equation (48).

Occasionally, the boundary conditions may be provided through a measurement at the boundary $t=t_f$ (“decoder”), as follows:

$$h_i\left(t_f\right)=d_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em},$$

(49)

where the scalar values $d_i$ are known, albeit imprecisely, since they are considered to stem from experiments and/or computations. In such a case, the determination of the first-order sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ follows the same steps as above, yielding the following results:

(i)

The 1st-LASS will become an “initial value problem” comprising Equation (41), subject not the conditions shown in Equation (42), but subject to the following “initial conditions

$$a_i^{\left(1\right)}\left(t_0\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(50)

(ii)

The sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ will have the same formal expressions as in Equation (46) but the components of the 1st-level adjoint function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ will be the solution of Equations (41) and (50).

(iii)

The sensitivities of the response $R\left[h;f\left(\theta \right)\right]$ with respect to boundary conditions at $t=t_f$ will have the following expressions:

$${\displaystyle \frac{\partial R}{\partial d_i}}=a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.$$

(51)

2.2. Second-Order Neural Integral Equations of Fredholm-Type (2nd-NIDE-F)

The representation of the second-order $\left(n=2\right)$ neural integral equations of Fredholm-type (2nd-NIDE-F) is provided below, for $\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH$:

$$\begin{array}{l}{c}_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}+c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em}d\tau \hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.\end{array}$$

(52)

There are several combinations of boundary conditions that can be provided, either for the function $h_i\left(t\right)$ and/or for its first-derivative $d{h}_i\left(t\right)/dt$, $i=1,\dots ,TH$, at either $t=t_0$ (encoder) or at $t=t_f$ (decoder), or a combination thereof. For illustrative purposes, consider that the boundary conditions are as follows:

$$h_i\left(t_0\right)=e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{h}_i\left(t_f\right)=d_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(53)

The 1st-LVSS is obtained by taking the G-variations of Equations (52) and (53) to obtain the following system, comprising the forms taken on for $n=2$ by Equations (13) and (14), respectively:

$$\begin{array}{l}{\left\{c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{{\theta }^0}{\displaystyle \frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}}+{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}\\ +{\left\{c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{{\theta }^0}{\displaystyle \frac{d^2{v}_i^{\left(1\right)}}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{{\theta }^0}\\ -{{\displaystyle \sum _{k=1}^{TH}\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}}_{{\theta }^0}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f\right)\right\}}_{{\theta }^0}\delta f_k},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(54)

$$v_i^{\left(1\right)}\left(t_0\right)=\delta e_i\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{v}_i^{\left(1\right)}\left(t_f\right)=\delta d_i;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em};$$

(55)

where for $i=1,\dots ,TH;\hspace{0.17em}$ and $k=1,\dots ,TF$:

$$\begin{array}{l}{q}_{ik}^{\left(1\right)}\left(h;f\right)\triangleq -{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}-{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}+{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}\\ +{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}}.\end{array}$$

(56)

The 1st-LASS is constructed by using Equation (19) to form the inner product of Equation (54) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\Omega }_t\right)$ to obtain the following relationship:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(57)

Examining the structure of the left-side of Equation (57) reveals that the bilinear concomitant will arise from the integration by parts of the first and third terms the on the left-side of Equation (57), as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h;f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h;f\right]\frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}dt}}=P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i.1}\left[h\left(t\right);f\right]\right\}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d^2}{d{t}^2}\left\{a_i^{\left(1\right)}\left(t\right)c_{i.2}\left[h\left(t\right);f\right]\right\}dt}},\end{array}$$

(58)

where the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ has the following expression:

$$\begin{array}{l}P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq {\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right]v_i^{\left(1\right)}\left(t_f\right)-a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]v_i^{\left(1\right)}\left(t_0\right)\right\}}\\ +{\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,2}\left[h\left(t_f\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t_f\right)}{dt}-a_i^{\left(1\right)}\left(t_0\right)c_{i,2}\left[h\left(t_0\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t_0\right)}{dt}\right\}}\\ -{{\displaystyle \sum _{i=1}^{TH}{v}_i^{\left(1\right)}\left(t_f\right)\left\{a_i^{\left(1\right)}\left(t\right)\frac{d{c}_{i,2}\left[h\left(t\right);f\right]}{dt}+c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}\\ +{{\displaystyle \sum _{i=1}^{TH}{v}_i^{\left(1\right)}\left(t_0\right)\left\{a_i^{\left(1\right)}\left(t\right)\frac{d{c}_{i,2}\left[h\left(t\right);f\right]}{dt}+c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}\end{array}$$

(59)

The remaining terms on the left-side of Equation (57) will be recast into their corresponding “adjoint form” by using the results obtained in Equations (34)–(38). Using these results together with the results obtained in Equations (58) and (59) yields the following expression for the left-side Equation (57):

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ =P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d^2}{d{t}^2}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h\left(t\right);f\right]\right\}dt}}\hspace{0.17em}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}}-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right)f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \hspace{0.17em}{v}_i^{\left(1\right)}\left(\tau \right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_i\left(\tau \right)}}}}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right){\phi }_{k,j}\left[f\left(\theta \right);t\right]\hspace{0.17em}}dt}\end{array}$$

(60)

Using Equation (58) and rearranging the terms on the right-side of Equation (60) yields the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta F_k}}-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)dt\left\{-\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}\right.}}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h\left(t\right);f\right]\right\}\\ +{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}+{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}\\ -\left.{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\right\}.\end{array}$$

(61)

The term on the right-side of Equation (61) is now required to represent the “indirect-effect” term defined in Equation (10), which is achieved by requiring the components of the function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ to satisfy the following 1st-LASS:

$$\begin{array}{l}-{\displaystyle \frac{d}{dt}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left[h\left(t\right);f\right]\right\}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left[h\left(t\right);f\right]\right\}\\ +{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}+{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}\\ -{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left[f\left(\theta \right);\tau \right]\hspace{0.17em}}d\tau }\\ ={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}}.\end{array}$$

(62)

The relation obtained in Equation (62) is the explicit form of the relation provided in Equation (23) for the particular case when $n=2$, i.e., when considering second-order neural integral equations of Fredholm-type (2nd-NIDE-F).

The unknown values involving the function $v_i\left(t\right)$ in the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ defined in Equation (59) are eliminated by imposing the following conditions:

$$a_i^{\left(1\right)}\left(t_0\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{a}_i^{\left(1\right)}\left(t_f\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH\hspace{0.17em}.$$

(63)

It follows from Equations (57)–(63) and (55) that the indirect-effect term defined in Equation (10) has the following expression in terms of the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$:

$${\left\{\delta R\left(h;f;a^{(1)}\right)\right\}}_{ind}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{ik}^{\left(1\right)}\left(h;f;t\right)\delta f_k}-\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right),$$

(64)

where the boundary quantity $\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ contains the known remaining terms after having implemented the known boundary conditions given in Equations (55) and (63), and has the following explicit expression:

$$\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq -{{\displaystyle \sum _{i=1}^{TH}\delta d_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}+{{\displaystyle \sum _{i=1}^{TH}\delta e_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}.$$

(65)

Using the results obtained in Equations (64), (65), (56) and (9) in Equation (8) yields the following expression for the G-variation $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$, which is seen to be linear in the variations $\delta d_i$, $\delta e_i$ ($i=1,\dots ,TH$) and $\delta f_j$ ($j=1,\dots ,TF$):

$$\begin{array}{l}\delta R\left[h\left(t\right);f\left(\theta \right);a^{(1)}\left(t\right);\delta f\right]\triangleq {\displaystyle \sum _{j=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}\delta f_j}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{j=1}^{TF}{q}_{jk}^{\left(1\right)}\left(h;f;t\right)\delta f_j}}+{{\displaystyle \sum _{i=1}^{TH}\delta d_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}\\ -{{\displaystyle \sum _{i=1}^{TH}\delta e_i\left\{c_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}\triangleq {\displaystyle \sum _{j=1}^{TF}\frac{\partial R}{\partial f_j}}\delta f_j+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial d_i}}\delta d_i+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial e_i}}\delta e_i\hspace{0.17em}\hspace{0.17em}.\end{array}$$

(66)

The expression in Equation (66) is to be satisfied at the nominal/optimal values for the respective model parameters, but this fact has not been indicated explicitly in order to simplify the notation.

It also follows from Equations (66) and (56) that the sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ have the following expressions, written in the form of Equation (27):

$$\partial R/\partial f_j\triangleq R^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{S}_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}\hspace{0.17em}dt;\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF;$$

(67)

where

$$\begin{array}{l}{S}_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]\triangleq {\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}}-{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}}\\ -{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{d^2{h}_i\left(t\right)}{d{t}^2}\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}}+{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(f;t;x\right)}{\partial f_j}}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{a}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TF.\end{array}$$

(68)

The subscript “2” attached to the quantity $S_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ indicates that this quantity refers to a “second-order” NIDE-F net, while the superscript “(1)” indicates that this quantity refers to “first-order” sensitivities. As expected, the expression of $S_2^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ reduces to the expression of $S_1^{\left(1\right)}\left[j;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$ when the “second-order NIDE-F net” reduces to the “first-order NIDE-F net” in the case when $c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]\equiv 0$.

Identifying in Equation (66) the expressions that multiply the variations $\delta e_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the encoder’s initial-time conditions:

$${\displaystyle \frac{\partial R}{\partial e_i}}=-{\left\{c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}}\right\}}_{t=t_0}=-{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{c}_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\delta \left(t-t_0\right)dt};\hspace{0.17em}i=1,\dots ,TH.$$

(69)

Identifying in Equation (66) the expressions that multiply the variations $\delta d_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the final-time conditions:

$${\displaystyle \frac{\partial R}{\partial d_i}}=-{\left\{c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}}\right\}}_{t=t_f}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{c}_{i,2}\left[h\left(t\right);f\right]\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\delta \left(t-t_f\right)dt};\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH.$$

(70)

If the boundary conditions imposed on the forward functions $h_i\left(t\right)$ and/or the first-derivatives $d{h}_i\left(t\right)/dt$, $i=1,\dots ,TH$, differ from the illustrative ones selected in Equation (53), then the corresponding boundary conditions for the 1st-level adjoint function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ would also differ from the ones shown in Equation (63), as would be expected. The components of $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ would consequently have different values; therefore, all of the first-order sensitivities $\partial R/\partial f_j$ would have values different from those computed using Equation (68), even though the formal mathematical expressions of the respective sensitivities would remain unchanged. Of course, the sensitivities $\partial R/\partial e_i$ and $\partial R/\partial d_i$ would have expressions that would differ from those in Equations (69) and (70), respectively, if the boundary conditions in Equation (53), and consequently those in Equation (63), were different, since the residual bilinear concomitant $\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ would have a different expression from that shown in Equation (65).

3. Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm Type (2nd-FASAM-NIDE-F)

The second-order sensitivities of the response $R\left[h;f\left(\theta \right)\right]$ defined in Equation (5) will be computed by conceptually using their basic definitions as being the “first-order sensitivities of the first-order sensitivities”. Recall that the generic expression of the first-order sensitivities, $R^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$, $j_1=\hspace{0.17em}1,\dots ,TF$, of the response with respect to the components of the feature function $f\left(\theta \right)$ is provided in Equation (46). It follows that the second-order sensitivities of the response with respect to the components of the feature function will be provided by the first-order G-differential $\delta R^{\left(1\right)}$ of $R^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]$, which is by definition obtained as follows:

$$\begin{array}{l}\delta R^{\left(1\right)}\left[j_1;h^0\left(t\right);a^{\left(1,0\right)}\left(t\right);f^0\left(\theta \right);v^{\left(1\right)}\left(x\right);\delta a^{\left(1\right)}\left(x\right);\delta f\right]\\ \triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}\delta R^{\left(1\right)}\left[j_1;h^0\left(x\right)+\epsilon v^{\left(1\right)}\left(x\right);a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right);f^0+\epsilon \delta f\right]\right\}}_{\epsilon =0}\\ ={\displaystyle \sum _{j_2=1}^{TF}{\left\{\frac{\partial R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f\right]}{\partial f_{j_2}}\right\}}_{{\theta }^0}}\delta f_{j_2}+{\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;v^{\left(1\right)}\left(x\right);\delta a^{\left(1\right)}\left(x\right)\right]\right\}}_{ind},\end{array}$$

(71)

where the indirect-effect term ${\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;v^{\left(1\right)}\left(x\right);\delta a^{\left(1\right)}\left(x\right)\right]\right\}}_{ind}$ comprises all dependencies on the vectors $v^{\left(1\right)}\left(x\right)$ and $\delta a^{\left(1\right)}\left(x\right)$ of variations in the state functions $h\left(t\right)$ and $a^{\left(1\right)}\left(t\right)$, around the respective nominal values denoted as $h^0\left(t\right)$ and $a^{\left(1,0\right)}\left(t\right)$, respectively, which are computed at the nominal parameter values ${\theta }^0$. This indirect-effect term is defined as follows:

$$\begin{array}{l}{\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)}\right]\right\}}_{ind}\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt}{\left\{{\displaystyle \frac{\partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}{\partial u}}{v}^{\left(1\right)}\left(x\right)\right\}}_{{\theta }^0}\\ +{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt}{\left\{{\displaystyle \frac{\partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]}{\partial a^{\left(1\right)}}}\delta a^{\left(1\right)}\left(x\right)\right\}}_{{\theta }^0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF.\end{array}$$

(72)

The variational function $\delta a^{\left(1\right)}\left(x\right)$ is the solution of the system of equations obtained by G-differentiating the 1st-LASS defined in Equations (23) and (25), which is by definition obtained as follows:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]\left[a_i^{\left(1,0\right)}+\epsilon \delta a_i^{\left(1\right)}\right]}\right\}}_{\epsilon =0}\\ ={\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \frac{\partial D\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]}{\partial h_j\left(t\right)}}\right\}}_{\epsilon =0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TH.\end{array}$$

(73)

$${\left\{{\displaystyle \frac{d}{d\epsilon }}{B}_j^{\ast }\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;a^{\left(1,0\right)}+\epsilon \delta a^{\left(1\right)};t\right]\right\}}_{\epsilon =0};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(74)

Carrying out the operations indicated in Equations (73) and (74) yields the following relations:

$$\begin{array}{l}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial }{\partial h_k\left(t\right)}\left[{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left(h;f;t\right)a_i^{\left(1\right)}\left(j_1;t\right)}\right]-\frac{{\partial }^{2}D\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]}{\partial h_k\left(t\right)\partial h_{j_1}\left(t\right)}\right\}}_{{\theta }^0}}{v}_k^{\left(1\right)}\left(t\right)\\ +{\left\{{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left(h;f;t\right)}\right\}}_{{\theta }^0}\delta a_i^{\left(1\right)}\left(j_1;t\right)={\displaystyle \sum _{j_2=1}^{TF}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\hspace{0.17em}}\delta f_{j_2}\left(\theta \right);\hspace{0.17em}\hspace{0.17em}\\ \hspace{0.17em}j=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF.\end{array}$$

(75)

$$\begin{array}{l}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\triangleq {\left\{{\displaystyle \frac{{\partial }^{2}D\left[h^0+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]}{\partial f_{j_2}\left(\theta \right)\partial h_{j_1}\left(t\right)}}\right\}}_{{\theta }^0}\\ -{\left\{{\displaystyle \frac{\partial }{\partial f_{j_2}\left(\theta \right)}}{\displaystyle \sum _{i=1}^{TH}{A}_{ij}^{\left(1\right)}\left(h;f;t\right)a_i^{\left(1\right)}\left(j_1;t\right)}\right\}}_{{\theta }^0};\hspace{0.17em}\hspace{0.17em}j=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1,j_2=1,\dots ,TF.\end{array}$$

(76)

$$\begin{array}{l}{\left\{{\displaystyle \frac{\partial B_j^{\ast }\left(h;f;a^{\left(1\right)}\right)}{\partial h\left(t\right)}}\right\}}_{{\theta }^0}{v}^{\left(1\right)}\left(t\right)+{\left\{{\displaystyle \frac{\partial B_j^{\ast }\left(h;f;a^{\left(1\right)}\right)}{\partial a^{\left(1\right)}\left(t\right)}}\right\}}_{{\theta }^0}\delta a^{\left(1\right)}\left(t\right)+{\left\{{\displaystyle \frac{\partial B_j^{\ast }\left(h;f;a^{\left(1\right)}\right)}{\partial f\left(\theta \right)}}\right\}}_{{\theta }^0}\delta f\left(\theta \right)=0;\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.\end{array}$$

(77)

For subsequent derivations, it is convenient to represent the relations in Equation (75) in matrix-vector form, as follows:

$$V_{21}^{\left(2\right)}\left(u;a^{\left(1\right)};f\right)v^{\left(1\right)}\left(t\right)+V_{22}^{\left(2\right)}\left(u;f\right)\delta a^{\left(1\right)}\left(t\right)=Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right);$$

(78)

where

$$\begin{array}{l}{V}_{21}^{\left(2\right)}\left(h;a^{\left(1\right)};f\right)\triangleq {\left\{{\displaystyle \frac{\partial A^{\left(1\right)}\left(h;a^{\left(1\right)};f\right)}{\partial h\left(t\right)}}-{\displaystyle \frac{{\partial }^{2}D\left(h;a^{\left(1\right)};f\right)}{\partial h\left(t\right)\partial h\left(t\right)}}\right\}}_{{\theta }^0};\\ V_{22}^{\left(2\right)}\left(h;a^{\left(1\right)};f\right)\triangleq {\left\{{\displaystyle \frac{\partial A^{\left(1\right)}\left(h;a^{\left(1\right)};f\right)}{\partial a^{\left(1\right)}\left(t\right)}}\right\}}_{{\theta }^0};\\ Q^{\left(2\right)}\left[j_1;j_2;h;a^{\left(1\right)};f\left(\theta \right)\right]\triangleq {\left\{q_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\right\}}_{TH\times TF};\end{array}$$

(79)

As indicated by Equation (78), the variational functions $v^{\left(1\right)}\left(x\right)$ and $\delta a^{\left(1\right)}\left(x\right)$ are the solutions of the system of matrix equations obtained by concatenating the 1st-LVSS defined by Equations (14) and (16) with Equations (77) and (78). The concatenated system thus obtained will be called the 2nd-Level Variational Sensitivity System (2nd-LVSS) and has the block-matrix form provided below:

$${\left\{V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]V^{\left(2\right)}\left(2;t\right)\right\}}_{{\theta }^0}={\left\{Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]\hspace{0.17em}\right\}}_{{\theta }^0};\hspace{0.17em}\hspace{0.17em}{t}_0<t<t_f;$$

(80)

$${\left\{B_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);V^{\left(2\right)}\left(2;t\right);f;\delta f\right]\right\}}_{{\theta }^0}=0\left[2\right]\triangleq {\left[0,0\right]}^{†};\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}.$$

(81)

To distinguish block-matrices from block-vectors, two bold capital-letters have been used (and will henceforth be used) to denote block-matrices, as in the case of “the second-level variational matrix” $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$. The “2nd-level” is indicated by the superscript “(2)”. The argument “$2\times 2$”, which appears in the list of arguments of $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$, indicates that this matrix is a $2\times 2$-dimensional block-matrix comprising four submatrices, each of dimensions $TD\times TD$. The structure of the block-matrix $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ is provided below:

$$V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq \left(\begin{array}{cc}L\left[h;f\left(\theta \right);t\right]& 0\\ V_{21}^{\left(2\right)}\left(h;a^{\left(1\right)};f;t\right)& V_{22}^{\left(2\right)}\left(h;a^{\left(1\right)};f;t\right)\end{array}\right)$$

(82)

The argument “2” which appears in the list of arguments of the vector $U^{\left(2\right)}\left(2;t\right)$ and of the “variational vector” $V^{\left(2\right)}\left(2;t\right)$ in Equation (80) indicates that each of these vectors is a 2-block column vector, each block comprising a column-vector of dimension $TD$; the vectors $U^{\left(2\right)}\left(2;t\right)$ and $V^{\left(2\right)}\left(2;t\right)$ are defined as follows:

$$U^{\left(2\right)}\left(2;t\right)\triangleq \left(\begin{array}{c}h\left(t\right)\\ a^{\left(1\right)}\left(t\right)\end{array}\right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{V}^{\left(2\right)}\left(2;t\right)\hspace{0.17em}\triangleq \left(\begin{array}{c}{v}^{\left(1\right)}\left(t\right)\\ \delta a^{\left(1\right)}\left(t\right)\end{array}\right).$$

(83)

The 2-block vector $Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]$ is defined as follows:

$$Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]\hspace{0.17em}\triangleq \left(\begin{array}{c}{Q}^{\left(1\right)}\left[h\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)\\ Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)\end{array}\right)\hspace{0.17em};$$

(84)

The 2-block column vector $B_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);V^{\left(2\right)}\left(2;t\right);f;\delta f\right]$ in Equation (81) represents the concatenated boundary/initial conditions provided in Equations (14) and (77), evaluated at the nominal parameter values. The argument “2” in the expression $0\left[2\right]\triangleq {\left[0,0\right]}^{†}$ in Equation (81) indicates that this expression is a two-block column vector comprising two vectors, each of which has $TD$-components, all of which are zero-valued.

The need for solving the 2nd-LVSS is circumvented by deriving an alternative expression for the indirect-effect term ${\left\{\delta R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)}\right]\right\}}_{ind}$ defined in Equation (72), in which the function $V^{\left(2\right)}\left(2;t\right)\hspace{0.17em}$ is replaced by a 2nd-level adjoint function that is independent of variations in the model parameter and state functions. This 2nd-level adjoint function will be the solution of a 2nd-Level Adjoint Sensitivity System (2nd-LASS), which will be constructed by using the same principles as employed for deriving the 1st-LASS. The 2nd-LASS is constructed in a Hilbert space ${\mathrm{H}}_2\left({\Omega }_t\right)$, ${\Omega }_t\triangleq t\in \left[t_0,t_f\right]$, comprising block-vectors having the same structure as $V^{\left(2\right)}\left(2;t\right)$ that can generically be represented as follows: ${\Phi }^{(2)}\left(2;t\right)\triangleq {\left[{\phi }^{(2)}\left(1;t\right),{\phi }^{(2)}\left(2;t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\Omega }_t\right)$, with ${\phi }^{(2)}\left(i;t\right)\triangleq {\left[{\phi }_{i,1}^{\left(2\right)}\left(t\right),\dots {\phi }_{i,j}^{\left(2\right)}\left(t\right),\dots ,{\phi }_{i,TH}^{\left(2\right)}\left(t\right)\right]}^{†}$, for $i=1,2$. The Hilbert space ${\mathrm{H}}_2\left({\Omega }_t\right)$ is endowed with the following inner product of two vectors ${\Phi }^{(2)}\left(2;t\right)\in {\mathrm{H}}_2\left({\Omega }_t\right)$ and ${\Psi }^{(2)}\left(2;t\right)\in {\mathrm{H}}_2\left({\Omega }_t\right)$:

$$\begin{array}{l}{〈{\Psi }^{(2)}\left(2;t\right),{\Phi }^{(2)}\left(2;t\right)〉}_2\triangleq {\displaystyle \sum _{i=1}^2{〈{\psi }^{(2)}\left(i;t\right),{\phi }^{(2)}\left(i;t\right)〉}_1}\\ ={\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_{1,j}^{\left(2\right)}\left(t\right){\eta }_{1,j}^{\left(2\right)}\left(t\right)dt}\hspace{0.17em}}+{\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_{2,j}^{\left(2\right)}\left(t\right){\eta }_{2,j}^{\left(2\right)}\left(t\right)dt}}\hspace{0.17em}.\end{array}$$

(85)

The inner product defined in Equation (85) will be used to construct the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for a 2nd-level adjoint function $A^{(2)}\left(2;t\right)\triangleq {\left[a^{(2)}\left(1;t\right),a^{(2)}\left(2;t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\Omega }_t\right)$, $a_i^{\left(2\right)}\left(t\right)\triangleq {\left[a_{i,1}^{\left(2\right)}\left(t\right),\dots ,a_{i,TH}^{\left(2\right)}\left(t\right)\right]}^{†}$, $i=1,2$, by implementing the following sequence of steps, which are conceptually similar to those implemented in Section 2 for constructing the 1st-FASAM-NIDE-F methodology:

• Using Equation (85), construct the inner product of the yet undetermined function $A^{(2)}\left(2;t\right)\triangleq {\left[a^{(2)}\left(1;t\right),a^{(2)}\left(2;t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\Omega }_t\right)$ with Equation (80) to obtain the following relation:

  $$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;t\right),V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]V^{\left(2\right)}\left(2;t\right)〉}_2\right\}}_{{\theta }^0}\\ ={\left\{{〈A^{(2)}\left(2;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}.\end{array}$$

  (86)
• Use the definition of the operator adjoint to $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ in the Hilbert space ${\mathrm{H}}_2\left({\Omega }_t\right)$ to transform the inner product on the left-side of Equation (86) as follows:

  $$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;t\right),V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]V^{\left(2\right)}\left(2;t\right)〉}_2\right\}}_{{\theta }^0}={\left\{P^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f\right)\right\}}_{{\theta }^0}\\ +{\left\{{〈V^{\left(2\right)}\left(2;t\right),A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]A^{(2)}\left(2;t\right)〉}_2\right\}}_{{\theta }^0},\end{array}$$

  (87)

  where the quantity ${\left\{P^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f\right)\right\}}_{{\theta }^0}$ denotes the corresponding bilinear concomitant on the domain’s boundary, evaluated at the nominal values for the parameters and respective state functions, and where the operator $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq {\left\{V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}^{\ast }$ denotes the formal adjoint of the matrix-valued operator $V{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$, comprising $\left(2\times 2\right)$ block-matrices, each of dimensions $T{D}^2$, having the following block-matrix structure.

  $$A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq {\left(\begin{array}{cc}L& 0\\ V_{21}^{\left(2\right)}& V_{22}^{\left(2\right)}\end{array}\right)}^{\ast }=\left(\begin{array}{cc}{\left\{L^{\ast }\right\}}^{†}& {\left[V_{21}^{\left(2\right)\ast }\right]}^{†}\\ 0& {\left[V_{22}^{\left(2\right)\ast }\right]}^{†}\end{array}\right).$$

  (88)
• Require the inner product on the right-side of Equation (87) to represent the indirect-effect term ${\left\{\delta R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)}\right]\right\}}_{ind}$ defined in Equation (72) by imposing the following relation:

  $${\left\{A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]A^{(2)}\left(2;j_1;t\right)\right\}}_{{\theta }^0}={\left\{Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}_{{\theta }^0},\hspace{0.17em}{j}_1=1,\dots ,TF;$$

  (89)

  where

  $$Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]\triangleq \left(\begin{array}{c}\partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]/\partial u\\ \partial S_1^{\left(1\right)}\left[j_1;h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right)\right]/\partial a^{\left(1\right)}\end{array}\right),\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF.$$

  (90)

Since the source-term on the right-side of Equation (89) is a distinct quantity for each value of the index $j_1$, this index has been added to the list of arguments of the function $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$ in order to emphasize that a distinct function $A^{(2)}\left(2;j_1;t\right)$ will correspond to each index $j_1$. Of course, the adjoint operator $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ that acts on the function $A^{(2)}\left(2;j_1;t\right)$ is independent of the index $j_1$ and could, in principle, be inverted just once and stored for subsequent repeated applications to the $j_1$-dependent source terms $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$ for computing the corresponding functions $A^{(2)}\left(2;j_1;t\right)$.

4

The definition of the function $A^{(2)}\left(2;j_1;t\right)$ is completed by requiring it to satisfy adjoint boundary/initial conditions represented in operator form as follows:

$${\left\{B_A^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);A^{(2)}\left(2;j_1;t\right);f\right]\right\}}_{{\theta }^0}=0\left[2\right];\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1=1,\dots ,TF;\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0.$$

(91)

The boundary/initial conditions represented by Equation (91) are determined imposing the following requirements:

(a)

they must be independent of unknown values of $V^{\left(2\right)}\left(2;t\right)$;

(b)

the substitution of the boundary and/or initial conditions represented by Equations (81) and (91) into the expression of the bilinear concomitant ${\left\{P^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f\right)\right\}}_{{\theta }^0}$ must cause all terms containing unknown boundary/initial values of $V^{\left(2\right)}\left(2;t\right)$ to vanish.

The NIDE-net comprising Equations (89) and (91) is called the “2nd-Level Adjoint Sensitivity System (2nd-LASS)” and its solution, $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$, $\hspace{0.17em}{j}_1=1,\dots ,TF$, is called the “2nd-level adjoint sensitivity function”. The unique properties of the 2nd-LASS will be highlighted in the sequel, below.

Using in Equation (72) the relations defining 2nd-LASS together with the 2nd-LVSS and the relation provided in Equation (87) yields the following alternative expression for the indirect-effect term, involving the 2nd-level adjoint sensitivity function $A^{(2)}\left(2;j_1;x\right)\triangleq {\left[a^{(2)}\left(1;j_1;x\right),a^{(2)}\left(2;j_1;x\right)\right]}^{†}$ instead of the 2nd-level variational function $V^{\left(2\right)}\left(2;t\right)$:

$$\begin{array}{l}{\left\{\delta R^{\left(1\right)}\left[j_1;h;a^{\left(1\right)};f;A^{(2)}\left(2;j_1;x\right)\right]\right\}}_{ind}=-{\left\{{\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)\right\}}_{{\theta }^0}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}\end{array}$$

(92)

where ${\left\{{\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)\right\}}_{{\theta }^0}$ denotes known residual (non-zero) boundary terms which may not have vanished after having used the boundary and/or initial conditions represented by Equations (81) and (91).

Replacing the expression obtained in Equation (92) into Equation (71) yields the following expression:

$$\begin{array}{l}{\left\{\delta R^{\left(1\right)}\left[j_1;U^{\left(2\right)}\left(2;t\right);A^{(2)}\left(2;j_1;t\right);f;\delta f\right]\right\}}_{{\theta }^0}=-{\left\{{\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)\right\}}_{{\theta }^0}\\ +{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}+{\displaystyle \sum _{j_2=1}^{TF}{\left\{\frac{\partial R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f\right]}{\partial f_{j_2}}\right\}}_{{\theta }^0}}\delta f_{j_2}\\ \triangleq {\displaystyle \sum _{j_2=1}^{TF}{\left\{\frac{{\partial }^{2}R\left[h;f\left(\theta \right)\right]}{\partial f_{j_2}\partial f_{j_1}}\right\}}_{{\theta }^0}}\delta f_{j_2}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1=\hspace{0.17em}1,\dots ,TF\hspace{0.17em}.\end{array}$$

(93)

The expressions of the second-order sensitivities ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}$ of the response with respect to the components of the feature function are obtained by performing the following sequence of operations:

(i)

Use Equation (84) to recast the second term on the right-side of Equation (93) as follows:

$$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}={\left\{{〈a^{(2)}\left(1;j_1;t\right),Q^{\left(1\right)}\left[h\left(t\right);f\left(\theta \right);t\right]\delta f〉}_1\right\}}_{{\theta }^0}\\ +{\left\{{〈a^{(2)}\left(2;j_1;t\right),Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)〉}_1\right\}}_{{\theta }^0}.\end{array}$$

(94)

(ii)

Recall that $Q^{\left(1\right)}\left[h\left(t\right);f\left(\theta \right);t\right]\delta f\triangleq {\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k\right\}}_{{\theta }^0}}$, where the quantities $q_{i,k}^{\left(1\right)}\left(h;f;t\right)$ were defined in Equation (15). Recall that $Q^{\left(2\right)}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]\delta f\left(\theta \right)\triangleq {\displaystyle \sum _{j_2=1}^{TF}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)\hspace{0.17em}}\delta f_{j_2}\left(\theta \right)$ where the quantities $q_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)$ were defined in Equation (76). Insert these expressions in Equation (94) to obtain the following relation:

$$\begin{array}{l}{\left\{{〈A^{(2)}\left(2;j_1;t\right),Q_V^{\left(2\right)}\left[2;U^{\left(2\right)}\left(2;t\right);f;\delta f\right]〉}_2\right\}}_{{\theta }^0}={\left\{{\displaystyle \sum _{j_2=1}^{TF}\left[{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)a_{1,i}^{\left(2\right)}\left(j_1;t\right)dt}}\right]}\hspace{0.17em}\delta f_{j_2}\right\}}_{{\theta }^0}\\ +{\left\{{\displaystyle \sum _{j_2=1}^{TF}\left[{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{j,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)a_{2,i}^{\left(2\right)}\left(j_1;t\right)dt}}\right]}\hspace{0.17em}\delta f_{j_2}\right\}}_{{\theta }^0}.\end{array}$$

(95)

(iii)

Insert into Equation (93) the equivalent expression obtained in Equation (95), and subsequently identifying the quantities that multiply the variations $\delta f_{j_2}$, to obtain the following expression for the second-order sensitivities ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}$:

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R\left[h;f\left(\theta \right)\right]}{\partial f_{j_2}\partial f_{j_1}}}={\displaystyle \frac{\partial R^{\left(1\right)}\left[j_1;u;a^{\left(1\right)};f\right]}{\partial f_{j_2}}}-{\displaystyle \frac{\partial {\widehat{P}}^{\left(2\right)}\left(U^{\left(2\right)};A^{(2)};V^{\left(2\right)};f;\delta f\right)}{\partial f_{j_2}}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)a_{1,i}^{\left(2\right)}\left(j_1;t\right)dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{q}_{i,j_2}^{\left(2\right)}\left(j_1;j_2;h;f;t\right)a_{2,i}^{\left(2\right)}\left(j_1;t\right)dt}}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{j}_1,j_2=\hspace{0.17em}1,\dots ,TF\hspace{0.17em}.\end{array}$$

(96)

It is important to note that the 2nd-LASS is independent of variations $\delta f$ and variations $V^{\left(2\right)}\left(2;x\right)$ in the respective state functions. It is also important to note that the ${\left(2\times TD\right)}^2$-dimensional matrix $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ is independent of the index $j_1$. Only the source-term $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$ depends on the index $j_1$. Therefore, the same solver can be used to invert the matrix $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);\hspace{0.17em}f\right]$ in order to solve numerically the 2nd-LASS for each $j_1$-dependent source $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$ in order to obtain the corresponding $j_1$-dependent $2\times TD$-dimensional 2nd-level adjoint function $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$. Computationally, it would be most efficient to store, if possible, the inverse matrix ${\left\{A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}^{-1}$, in order to multiply directly the inverse matrix ${\left\{A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]\right\}}^{-1}$ with the corresponding source term $Q_A^{\left(2\right)}\left[2;j_1;U^{\left(2\right)}\left(2;t\right);f\right]$, for each index $j_1$, in order to obtain the corresponding $j_1$-dependent $2\times TD$-dimensional 2nd-level adjoint function $A^{(2)}\left(2;j_1;t\right)\triangleq {\left[a^{(2)}\left(1;j_1;t\right),a^{(2)}\left(2;j_1;t\right)\right]}^{†}$.

Since the adjoint matrix $A{M}^{\left(2\right)}\left[2\times 2;U^{\left(2\right)}\left(2;t\right);f\right]$ is block-diagonal, solving the 2nd-LASS is equivalent to solving two 1st-LASS, with two different source terms. Thus, the “solvers” and the computer program used for solving the 1st-LASS can also be used for solving the 2nd-LASS. The 2nd-LASS was designated as the “second-level” rather than the “second-order” adjoint sensitivity system, since the 2nd-LASS does not involve any explicit 2nd-order G-derivatives of the operators underlying the original system but involves the inversion of the same operators that need to be inverted for solving the 1st-LASS.

If the 2nd-LASS is solved $TF$-times, the 2nd-order mixed sensitivities ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}$ will be computed twice, in two different ways, in terms of two distinct 2nd-level adjoint functions. Consequently, the symmetry property ${\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_2}\partial f_{j_1}={\partial }^{2}R\left[h;f\left(\theta \right)\right]/\partial f_{j_1}\partial f_{j_2}$ provides an intrinsic (numerical) verification that the 1st-level adjoint function $a^{(1)}\left(x\right)$ and the components of the 2nd-level adjoint function $A^{(2)}\left(2;j_1;x\right)$ and are computed accurately.

The second-order sensitivities of the decoder-response with respect to the optimal weights/parameters ${\theta }_k,\hspace{0.17em}k=1,\dots ,TW$, are obtained analytically by using the chain rule in conjunction with the expressions obtained in Equations (46) and (96), as follows:

$${\displaystyle \frac{{\partial }^{2}R\left[h;f\left(\theta \right)\right]}{\partial {\theta }_k\partial {\theta }_j}}={\displaystyle \frac{\partial }{\partial {\theta }_k}}\left\{{\displaystyle \sum _{i=1}^{TF}\frac{\partial R\left[h;f\left(\theta \right)\right]}{\partial f_i\left(\theta \right)}\frac{\partial f_i\left(\theta \right)}{\partial {\theta }_j}}\right\}\hspace{0.17em},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j,k=1,\dots ,TW.$$

(97)

4. Discussion

The capabilities of the traditional neural-nets [39,40,41] have been significantly generalized by Chen et al. [1] with the introduction of Neural Ordinary Differential Equations (NODE), which provide an explicit connection between deep feed-forward neural networks and dynamical systems. Since NODE models are limited to describing systems that are instantaneous, Zappala et al. [11] have introduced the Neural Integral Equation (NIE) and the Attentional Neural Integral Equation (ANIE), which can be used to infer the spatio-temporal relations that generated the data, thus enabling the continuous learning of non-local dynamics with arbitrary time resolution [11,12]. Subsequently, Zappala et al. [16] have generalized both NODE and NIE/ANIE by developing a deep learning method called Neural Integro-Differential Equation (NIDE), which “learns” an integro-differential equation (IDE) whose solution approximates data sampled from given non-local dynamics. The motivation for using NIDE stems from the need to model systems that present spatio-temporal relations which transcend local modeling, as illustrated by applications in population dynamics, biology, physics, engineering and applied sciences [17,18,19,20,21,22,23,24,25,26].

The physical system modeled by a neural-net comprises parameters and functions of parameters (correlations, material properties, etc.) that stem from measurements and/or computations which are subject to uncertainties. Therefore, even if the neural-net reproduces perfectly the underlying system, the uncertainties inherent in the system’s parameters would propagate to the subsequent results produced by the decoder. Quantifying the uncertainties in the decoder’s response can only be performed if the sensitivities of the decoder’s response with respect to the neural-net’s optimized parameters are known. Based on the general framework of the nth-FASAM-N methodology [27], Cacuci has the “Second-Order Features Adjoint Sensitivity Analysis Methodologies for Neural Ordinary Differential Equations (2nd-FASAM-NODE)” [28], the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (2nd-FASAM-NIE-F)” [29] and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (2nd-FASAM-NIE-V)” [30]. The 2nd-FASAM-NODE encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodologies for Neural Ordinary Differential Equations (1st-FASAM-NODE)”. The 2nd-FASAM-NIE-F encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (1st-FASAM-NIE-F), while the 2nd-FASAM-NIE-V encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (1st-FASAM-NIE-V)”. The 1st-FASAM-NODE, 1st-FASAM-NIE-F and 1st-FASAM-NIE-V methodologies, respectively, enable the computation, with unparalleled efficiency, of exactly-determined first-order and, respectively, second-order sensitivities of decoder response with respect to the optimized/trained weights (and functions thereof) that characterize the network’s hidden layers, decoder, and encoder, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIE-net. The 2nd-FASAM-NODE, 2nd-FASAM-NIE-F and 2nd-FASAM-NIE-V methodologies, respectively, enable the computation (with unparalleled efficiency) of exactly-determined second-order sensitivities of decoder response with respect to the network’s-parameters and functions thereof, requiring only as many “large-scale” computations as there are non-zero first-order sensitivities with respect to the feature functions.

Generalizing the methodologies presented in [28,29,30], this work has introduced the general mathematical frameworks of the First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (1st-FASAM-NIDE-F), and of the Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type (2nd-FASAM-NIDE-F). The 1st-FASAM-NIDE-F methodology requires a single “large-scale” computation, for solving the 1st-Level Adjoint Sensitivity System (1st-LASS) in order to compute all of the first-order sensitivities of the decoder response, regardless of the number of weights/parameters underlying the NIDE-F net. The 2nd-FASAM-NIDE-F methodology requires at most as many “large-scale” computations as there are non-zero feature functions of model parameters. In the accompanying “Part II” [31], the application of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies will be illustrated by means of a paradigm heat conduction model, which admits exact closed-form solutions/expressions for all quantities of interest, including state functions, first-order and second-order sensitivities. This paradigm heat conduction model is of fundamental importance in many scientific fields involving heat transfer [32,33,34,35,36,37].

5. Conclusions

As has been shown in this work, the 1st-FASAM-NIDE-F methodology yields exact expressions of all first-order sensitivities of NIDE-F decoder-responses with respect to the optimized NIDE-F weights/parameters and functions thereof. These exact expressions are integrals involving the first-level adjoint sensitivity function that is obtained by solving the 1st-Level Adjoint Sensitivity System (1st-LASS). Thus, the computation of all first-order sensitivities of the decoder response requires a single “large-scale” computation for solving the 1st-LASS, regardless of the number of feature functions and parameters underlying the NIDE-F net. It has also been shown that the 2nd-FASAM-NIDE-F methodology enables the most efficient computation of the exactly obtained expressions of the second-order sensitivities of NIDE-F decoder-responses with respect to the optimized NIDE-F weights/parameters, involving integrals over the second-level adjoint sensitivity functions. Computing the requisite second-level adjoint sensitivity functions requires at most as many “large-scale” computations (for solving the 2nd-LASS) as there are non-zero feature functions of model parameters. The second-order mixed sensitivities are computed twice, using distinct 2nd-level adjoint functions, thus providing a stringent mechanism for the verification of the accuracy of the computations of all these adjoint functions.

To complement the present work, ongoing work aims at developing the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type (2nd-FASAM-NIDE-V)”, which will also comprise the 1st-FASAM-NIDE-V (“First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type”). This work will parallel the present 2nd-FASAM-NIDE-F and will enable the most efficient computation of exactly-obtained expressions of the first- and second-order sensitivities of NIDE-V decoder-responses with respect to the optimized NIDE-F weights/parameters and functions thereof. This ongoing work will complete the development of second-order sensitivity analysis methodologies for NODEs, NIEs, and NIDEs. Subsequent work is planned for applications of these methodologies to quantify the accuracy of predicted results by models of large-scale systems of interest to engineering, physical, and biological sciences.