The First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Neural Integro-Differential Equations of Volterra Type: Mathematical Framework and Illustrative Application to a Nonlinear Heat Conduction Model

Abstract

This work presents the mathematical frameworks of the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type” (1st-FASAM-NIDE-V) and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type” (2nd-FASAM-NIDE-V). It is shown that the 1st-FASAM-NIDE-V methodology enables the efficient computation of exactly-determined first-order sensitivities of the decoder response with respect to the optimized NIDE-V parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIE-net. The 2nd-FASAM-NIDE-V methodology enables the computation, with unparalleled efficiency, of the second-order sensitivities of decoder responses with respect to the optimized/trained weights involved in the NIDE-V’s decoder, hidden layers, and encoder, requiring only as many “large-scale” computations as there are non-zero first-order sensitivities with respect to the feature functions. These characteristics of the 1st-FASAM-NIDE-V and 2nd-FASAM-NIDE-V are illustrated by considering a nonlinear heat conduction model that admits analytical solutions, enabling the exact verification of the expressions obtained for the first- and second-order sensitivities of NIDE-V decoder responses with respect to the model’s functions of parameters (weights) that characterize the heat conduction model.

1. Introduction

The capabilities of traditional neural nets [1,2,3] have been extended significantly by the introduction of Neural Ordinary Differential Equations (NODE), formulated by Chen et al. [4], which have enabled the use of deep learning for modeling discretely sampled dynamical systems (new [5,6,7,8,9,10,11,12]). Although NODE provide a bridge between modern deep learning and traditional numerical modelling, they are limited to describing systems that are instantaneous, each time-step being determined locally in time, without contributions from the state of the system at other times. On the other hand, integral equations (IE) are suitable for modeling global “long-distance” spatio-temporal relations and can be solved efficiently using IE solvers that sample the domain of integration continuously, as exemplified in [13,14,15,16]. Two important families of IEs are the Volterra and the Fredholm equations. In a Volterra IE, the interval of integration grows linearly during the system’s dynamics. On the other hand, the interval of integration is fixed in a Fredholm IE during the dynamic-history of the system, but at any given time instance within this interval, the system depends on the past, present, and future states of the system. For IEs of the Volterra and Fredholm types, Zappala et al. [17] have presented a general mathematical framework for Neural Integral Equations (NIE) and Attentional Neural Integral Equations (ANIE), which can be used to infer the spatio-temporal relations that generated the data, thus enabling the continuous learning of non-local dynamics with arbitrary time resolution. In particular, the ANIE interprets the self-attention mechanism as the Nystrom method for approximating integrals [18], which enables efficient integration over higher dimensions. Generalizing this work, Zappala et al. [19] have also developed a deep learning method called Neural Integro-Differential Equation (NIDE), which “learns” an integro-differential equation (IDE) whose solution is obtained using standard IDE-solvers [20,21] and approximates data sampled from given non-local dynamics. The motivation for using NIDE stems from the need to model systems that present spatio-temporal relations that transcend local modeling, as illustrated by the pioneering works of Volterra on population dynamics [22]. Combining the properties of differential and integral equations, NIDEs also present properties that are unique to their non-local behavior, with applications in various applied sciences, including physics, engineering, and computational biology [23,24,25,26,27].

The neural net is trained/optimized to reproduce the underlying physical system as closely as possible, by minimizing a “loss functional” that aims at representing the discrepancy between a “reference solution” and the output produced by the respective net’s decoder. However, the physical system modeled by a neural net comprises parameters that stem from measurements and/or computations that are subject to uncertainties. Therefore, even in the idealized/unrealistic case when a neural net would perfectly model the system under consideration, the unavoidable uncertainties inherent in the physical system’s parameters would propagate to the subsequent results of interest (customarily called “responses”) produced by the net’s decoder. The quantification of the uncertainties in the decoder’s responses, which are various functionals of the net’s hidden variables and parameters rather than some “loss functional,” requires the computation of the sensitivities of the decoder’s response with respect to the optimized weights/parameters comprised within the neural net.

Cacuci [28] has developed the “Nth-Order Comprehensive Adjoint Sensitivity Analysis Methodology for Nonlinear Systems (Nth-CASAM-N)”, which enables the efficient computation of the exact expressions of arbitrarily high-order sensitivities of model responses with respect to the model’s parameters. However, physical systems and, therefore, their neural net counterparts comprise not only scalar-valued weights/parameters but often also comprise scalar-valued functions (e.g., correlations, material properties, etc.) of the model’s scalar parameters. It is convenient to refer to such scalar-valued functions as “features of primary model parameters.” Cacuci [29] has recently generalized his Nth- CASAM methodology, introducing the “Nth-Order Features Adjoint Sensitivity Analysis Methodology for Nonlinear Systems (Nth-FASAM-N),” which enables the computation, with unparalleled efficiency, of the exact expressions of arbitrarily high-order sensitivities of model responses with respect to the model’s “features.” Subsequently, the sensitivities of the responses with respect to the primary model parameters are determined, analytically and trivially, by applying the “chain-rule” to the expressions obtained for the response sensitivities with respect to the model’s features/functions of parameters.

Based on the general framework of the Nth-FASAM-N methodology [29], Cacuci has developed specific sensitivity analysis methodologies for NODE-nets, as follows: the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (1st-FASAM-NODE)” [30] and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (2nd-FASAM-NODE)” [31]. The 1st-FASAM-NODE and the 2nd-FASAM-NODE are pioneering sensitivity analysis methodologies that enable the computation, with unparalleled efficiency, of exactly-determined first-order and, respectively, second-order sensitivities of decoder response with respect to the optimized/trained weights involved in the NODE’s decoder, hidden layers, and encoder.

By applying the general concepts underlying the Nth-FASAM-N methodology [29], Cacuci [32] has recently developed the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (2nd-FASAM-NIE-F)”, which encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Fredholm-Type (1st-FASAM-NIE-F). Cacuci [33] has also developed the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (2nd-FASAM-NIE-V)”, which encompasses the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type (1st-FASAM-NIE-V).” The 1st-FASAM-NIE-F and 1st-FASAM-NIE-V methodologies, respectively, enable the computation, with unparalleled efficiency, of exactly-determined first-order sensitivities of decoder response with respect to the NIE-parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIE-net. The 2nd-FASAM-NIE-F and 2nd-FASAM-NIE-V methodologies, respectively, enable the efficient computation of exactly-determined second-order sensitivities of decoder response with respect to the NIE-parameters, requiring only as many “large-scale” computations as there are first-order sensitivities with respect to the feature functions.

Generalizing the 2nd-FASAM-NIE-F methodology, Cacuci [34] has recently developed the “First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Fredholm-Type Neural Integro-Differential Equations” (1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F), respectively, for computing, with unparalleled efficiency, the exact first- and second-order sensitivities, respectively, of decoder responses to model parameters in optimized NIDE-F networks.

This work presents the “First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Neural Integro-Differential Equations of Volterra-Type” (1st- FASAM-NIDE-V and 2nd-FASAM-NIDE-V, respectively), thus generalizing the 2nd-FASAM-NIE-V methodology [33]. The 1st-FASAM-NIDE-V and, respectively, the 2nd-FASAM-NIDE-V enable, for the first time, the most efficient computation of the exact expressions of the first- and, respectively, second-order sensitivities of NIDE-V decoder-responses with respect to the optimized network’s feature functions and weights/parameters of NIDE-V neural nets.

This work is structured as follows: Section 2 presents the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type” (1st-FASAM-NIDE-V). It is shown that the 1st-FASAM-NDIE-V methodology enables the computation, with unparalleled efficiency, of exactly-determined first-order sensitivities of decoder response with respect to the NODE-parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIDE-V net. When “feature functions of parameters “can be identified within the NIDE-V structure, the number of quadratures for computing the first-order sensitivities is smaller than the number of quadratures needed for computing the first-order decoder-response sensitivities directly with respect to the parameters, since the latter can be computed analytically and exactly by using the first-order sensitivities with respect to the feature functions.

Section 3 illustrates the application of the 1st-FASAM-NIDE-V methodology to a paradigm heat conduction model, which has been chosen for the following reasons: (i) the model is nonlinear but admits an equivalent linear model that is related to the original model through the Kirchhoff transformation; (ii) the model can be represented either as a traditional NODE net or a NIDE-V net; (iii) the model admits explicit closed-form solutions for all quantities of interest, including the functions representing the hidden/latent neural networks, the decoder response, and the sensitivities of the decoder’s response to the optimal weights/parameters; (iv) the model enables an exact comparison between the application of the 2nd-FASAM-NIDE-V methodology developed in Section 2 of this work and the application of the 2nd-FASAM-NODE methodology presented in [30,31]. It will be shown that the algebraic computations are considerably simpler in the Kirchhoff-transformed framework (in which the paradigm model can be exactly transformed from a nonlinear into a linear one) than in the conventional “temperature-framework,” in which the model’s nonlinearities cannot be circumvented.

Section 4 presents the mathematical framework of the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra Type” (2nd-FASAM-NIDE-V), which computes the second-order sensitivities most efficiently by conceptually using their basic definitions as being the “first-order sensitivities of the first-order sensitivities.” It is shown that the 2nd-FASAM-NIDE-V yields the exact expressions of the 2nd-order sensitivities and computes them most efficiently by needing only as many “large-scale” computations as there are non-zero 1st-order sensitivities with respect to the net’s feature functions. It is also shown that the mixed 2nd-order sensitivities are computed twice, using distinct 2nd-level adjoint functions, which provides the most inexpensive path for the stringent mutual verification of the accuracy of the computations performed when solving the respective 2nd-LASS for determining the respective 2nd-level adjoint functions.

Section 5 presents the illustrative application of the 2nd-FASAM-NIDE-V methodology to compute second-order sensitivities of the paradigm heat conduction model. The algebraic computations/manipulations are minimized by considering the heat conduction model in the Kirchhoff-transformed framework. All the general features of the 2nd-FASAM-NIDE-V methodology are highlighted using this model.

The discussion presented in Section 6 summarizes and concludes this work, noting that (i) the 1st-FASAM-NIDE-V methodology enables the computation, with unparalleled efficiency, of exactly-determined first-order sensitivities of decoder response with respect to the NODE-parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIDE-V net; and (ii) the 2nd-FASAM-NIDE-V methodology enables the computation (with unparalleled efficiency) of exactly-determined second-order sensitivities of decoder response with respect to the model’s feature functions and parameters, requiring at most as many “large-scale” computations as there are non-zero first-order sensitivities with respect to the feature functions. Future work will examine the theoretical underpinnings and feasibility of adapting algorithms of “backpropagation-type” for computing high-order sensitivities of decoder response with respect to the feature functions of model parameters, aiming at maximizing the efficiency and accuracy of computing sensitivities of orders higher than first-order.

2. First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type (1st-FASAM-NIDE-V)

The major families of IDEs are the Fredholm and the Volterra equations. In a NIDE-F, the interval of integration is fixed during the dynamic-history of the system, but at any given time instance within this interval, the system depends on the past, present, and future states of the system. In contradistinction, in a NIDE-V, the interval of integration grows linearly during the system’s dynamics. Otherwise, the generic mathematical structures of the NIDE-V and the NIDE-F are similar. Therefore, the notation that will be used for developing the 1st-FASAM-NIDE-V sensitivity analysis methodology to be presented in this work will closely resemble the notation used by Cacuci [34] in his development of the 2nd-FASAM-NIDE-F methodology. Using similar notation will also facilitate the comparison between the 2nd-FASAM-NIDE-F methodology [34] and the 2nd-FASAM-NIDE-V methodology to be developed in this work.

The expression of the network of nonlinear neural integro-differential equations of Volterra-type (NIDE-V) considered in this work generalizes the NIE-V net introduced in [33] and is represented in component form by the following system of Nth-order integro-differential equations:

$$\begin{array}{l}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]\frac{d^n{h}_i\left(t\right)}{d{t}^n}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}; i=1,\dots ,TH\hspace{0.17em}.\end{array}$$

(1)

The boundary conditions for the functions $h_i\left(t\right)$ and their time-derivatives associated with the encoder of the NIDE-V net represented by Equation (1) are typically imposed at the “initial time” $t=t_0$ and can be represented generically in operator form as follows:

$$B_j\left[h\left(t\right);f\left(\theta \right);t\right]=0\hspace{0.17em}; t=t_0 ; j=1,\dots ,BC.$$

(2)

The quantities appearing in Equations (1) and (2) are defined as follows:

(i)

The real-valued scalar quantities $t$ and $\tau$, $t_0\le t,\tau \le t_f$ are time-like independent variables that parameterize the dynamics of the hidden/latent neuron units. Customarily, the variable $t$ is called the “global time” while the variable $\tau$ is called the “local time”, both being defined on the domain ${\mathsf{\Omega }}_t\triangleq \left[t_0,t_f\right]$, where the initial time-value is denoted as $t_0$ while the stopping (“final”) time-value is denoted as $t_f$. Thus, the dynamics modeled by Equation (1) involves both instantaneous/local and non-local information.

(ii)

The components of the $TH$-dimensional vector-valued function $h\left(t\right)\triangleq {\left[h_1\left(t\right),\dots ,h_{TH}\left(t\right)\right]}^{†}$ represent the hidden/latent neural networks; $TH$ denotes the total number of components of $h\left(t\right)$. In this work, the symbol “$\triangleq$” will be used to signify “is defined as” or, equivalently, “is by definition equal to.” The various vectors will be considered to be column vectors. The dagger “$†$” symbol will be used to denote “transposition.”

(iii)

The components of the column-vector $\theta \triangleq {\left[{\theta }_1,\dots ,{\theta }_{TW}\right]}^{†}$ represent the “primary parameters,” namely scalar learnable adjustable parameters/weights, in all the latent neural nets, including the encoders(s) and decoder(s). The total number of adjustable parameters/weights is denoted as “$TW$.” The components $f_i\left(\theta \right)$, $i=1,\dots ,TF$, of the vector-valued function $f\left(\theta \right)\triangleq {\left[f_1\left(\theta \right),\dots ,f_{TF}\left(\theta \right)\right]}^{†}$ represent the “feature/functions of the primary model parameters feature.” The quantity $TF$ denotes the total number of such feature functions comprised in the NIDE-V. In general, the components $f_i\left(\theta \right)$ are nonlinear functions of $\theta$. The total number of feature functions must necessarily be smaller than the total number of primary parameters (weights), i.e., $TF<TW$. When the NIDE-F comprises only primary parameters, it is considered that $f_i\left(\theta \right)\equiv {\theta }_i$ for all $i=1,\dots ,TW\equiv TF$.

(iv)

The functions ${\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]$ model the dynamics of the neurons in a latent space where the local time integration occurs, while the functions ${\phi }_{i,j}\left[f\left(\theta \right);t\right]$ map the local space back to the original data space. The functions $g_i\left[h\left(t\right);f\left(\theta \right)\right]$ model additional dynamics in the original data space. In general, these functions are nonlinear in their arguments.

(v)

The functions $c_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]$ are coefficient-functions associated with the order $n=1,\dots ,N$ of the time-derivatives $d^n{h}_i\left(t\right)/d{t}^n$ of the functions $h_i\left(t\right)$. The functions $c_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]$ may depend nonlinearly on the functions $h\left(t\right)$ and $f\left(\theta \right)$.

(vi)

The operators $B_j\left[h\left(t\right);f\left(\theta \right);t\right]$, $j=1,\dots ,BC$, represent boundary conditions imposed at $t=t_0$ and/or at $t=t_f$ on the functions $h_i\left(t\right)$ and on their time-derivatives; the quantity “BC” denotes the “total number of boundary conditions.”

The NIDE-V net is “trained” by minimizing a user-chosen loss functional representing the discrepancy between a reference solution (“target data”) and the output produced by the NIDE-V decoder. The “training” process produces “optimal” values for the primary parameters $\theta \triangleq {\left[{\theta }_1,\dots ,{\theta }_{TW}\right]}^{†}$. These optimal values become the nominal values to be used for subsequent applications/computations using the NIDE-V net. Optimal/nominal values for parameters and all other quantities will be denoted by using the superscript “zero.” Thus, the nominal parameter values will be denoted as ${\theta }^0\triangleq {\left[{\theta }_1^0,\dots ,{\theta }_{TW}^0\right]}^{†}$. Using these optimal/nominal parameter values to evaluate the NIDE-V net yields the optimal/nominal solution $h^0\left(t,x\right)\triangleq {\left[h_1^0\left(t\right),\dots ,h_{TH}^0\left(t\right)\right]}^{†}$, which will formally satisfy the following form of Equation (1):

$$\begin{array}{l}{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]\frac{d^n{h}_i^0\left(t\right)}{d{t}^n}}=g_i\left[h^0\left(t\right);f\left({\theta }^0\right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left({\theta }^0\right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau {\psi }_j\left[h^0\left(\tau \right);f\left({\theta }^0\right);\tau \right]}; i=1,\dots ,TH;\end{array}$$

(3)

subject to the following optimized/trained boundary conditions:

$$B_j\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_0 \hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_f\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(4)

After the NIDE-F net is optimized to reproduce the underlying physical system as closely as possible, the subsequent responses of interest are no longer “loss functionals” but become specific functionals of the NIDE-F’s “decoder” output. Such responses of interest can be generally represented by the functional $R\left[h;f\left(\theta \right)\right]$ defined below:

$$R\left[h;f\left(\theta \right)\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h\left(t\right);f\left(\theta \right);t\right]dt}.$$

(5)

The function $D\left[h\left(t\right);f\left(\theta \right);t\right]$ models the decoder. The scalar-valued quantity $R\left[h;f\left(\theta \right)\right]$ is a functional of $h\left(t,x\right)$ and $f\left(\theta \right)$, and represents the NIDE-V’s decoder-response. At the optimal/nominal parameter values, the decoder response is formally represented using the superscript “zero” as follows:

$$R\left[h^0;f\left({\theta }^0\right)\right]={\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h^0\left(t\right);f\left({\theta }^0\right);t\right]dt}.$$

(6)

The physical system modeled by the NIDE-V net comprises parameters that stem from measurements and/or computations. Consequently, even in an ideal situation when the NIDE-V net perfectly models the underlying physical system, the NIDE-V’s optimal weights/parameters are unavoidably afflicted by uncertainties stemming from the parameters underlying the physical system. Therefore, the known optimal/nominal values ${\theta }^0$ of the primary model parameters (“weights”) characterizing the NIDE-V net will differ from the true but unknown values $\theta$ of the respective weights by variations denoted as $\delta \theta \triangleq \theta -{\theta }^0$. The variations $\delta \theta \triangleq \theta -{\theta }^0$ will induce corresponding variations, $\delta f\triangleq f\left(\theta \right)-f^0$, $f^0\triangleq f\left({\theta }^0\right)$, in the feature functions. The variations $\delta \theta$ and $\delta f$ will induce, through Equation (1), variations $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$ around the nominal/optimal functions $h^0\left(t\right)$. In turn, the variations $\delta f\triangleq f\left(\theta \right)-f^0$ and $v^{\left(1\right)}\left(t;x\right)$ will induce variations $\delta R\left(h^0;F^0;v^{\left(1\right)};\delta f;t\right)$ in the NIDE-V decoder’s response.

The individual uncertainties afflicting the optimal parameters contribute to the total uncertainty in the decoder response $R\left[h;f\left(\theta \right)\right]$. These individual contributions are quantified by the sensitivities (i.e., functional derivatives) of the NIDE-V decoder-response with respect to the optimized NIDE-V parameters. The “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type (1st-FASAM-NIDE-V)” aims at obtaining the exact expressions of the first-order sensitivities of the decoder’s response with respect to the feature function and the primary model parameters, followed by the most efficient computation of these sensitivities. The 1st-FASAM-NIDE-V will be established by applying the same principles as those underlying the general 1st-FASAM-N [28] methodology.

The fundamental concept for defining the first-order sensitivity of a vector-valued operator $N\left(x\right)$ acting on a vector $x$ with respect to variations $\delta x$ in a neighborhood around nominal values denoted as $x^0$ has been shown in 1981 by Cacuci [35] to be provided by the 1st-order Gateaux- (G-) variation $\delta N\left(x^0;\delta x\right)$ of $N\left(x\right)$, which is defined as follows:

$$\delta N\left(x^0;\delta x\right)\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}\left[N\left(x^0+\epsilon \delta x\right)\right]\right\}}_{\epsilon =0}\triangleq \underset{\epsilon \to 0}{\lim }\left\{\left[N\left(x^0+\epsilon \delta x\right)-N\left(x^0\right)\right]/\epsilon \right\},$$

(7)

for a scalar $\epsilon$ and for arbitrary vectors $\delta x$ in a neighborhood $\left(x^0+\epsilon \delta x\right)$ around $x^0$. When the G-variation $\delta N\left(x^0;\delta x\right)$ is linear in the variation $\delta x$, it is called a “G-differential”, which can be written in the form $\delta N\left(x^0;\delta x\right)={\left\{\partial N/\partial x\right\}}_{x^0}\delta x$, where ${\left\{\partial N/\partial x\right\}}_{x^0}$ denotes the first-order G-derivative of $N\left(x\right)$ with respect to $x$, evaluated at $x^0$.

Applying the definition provided in Equation (7) to Equation (5) yields the following expression for the first-order G-variation $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$ of the response $R\left[h;f\left(\theta \right)\right]$:

$$\begin{array}{l}\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)={\left\{{\displaystyle \frac{d}{d\epsilon }}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}D\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;t\right]dt}\right\}}_{\epsilon =0}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\hspace{0.17em}{\left\{\delta R\left(h^0;f^0;\delta f\right)\right\}}_{dir}+{\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}\hspace{0.17em},\end{array}$$

(8)

where the “direct effect term” arises directly from variations $\delta f$ and is defined as follows:

$${\left\{\delta R\left(h^0;f^0;\delta f\right)\right\}}_{dir}\triangleq {\displaystyle \sum _{i=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt{\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_i}\delta f_i\right\}}_{\left(h^0;f^0\right)}}},$$

(9)

and where the “indirect effect term” arises indirectly, through the variations $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$ in the hidden state functions $h\left(t\right)$, and is defined as follows:

$${\left\{\delta R\left(h^0;f^0;v^{\left(1\right)}\right)\right\}}_{ind}\triangleq {\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\left\{\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_i\left(t\right)}{v}_i^{\left(1\right)}\left(t\right)\right\}}_{\left(h^0;f^0\right)}}}.$$

(10)

The direct-effect term can be quantified using the nominal values $\left(h^0;f^0\right)$; the indirect-effect term can be quantified only after having determined the variations $v^{\left(1\right)}\left(t\right)$, which are caused by the variations $\delta f$ through the NIDE-V net defined in Equation (1).

2.1. Nth-Order Neural Integral Equations of Volterra-Type (Nth-NIDE-V)

The first-order relationship between the variations $v^{\left(1\right)}\left(t\right)$ and $\delta f$ is obtained from the first-order G-variations of Equations (1) and (2). It is convenient to rewrite Equation (1) in the following (more compact) form, for $i=1,\dots ,TH$:

$$N_i\left[h\left(t\right);f\left(\theta \right);t\right]=g_i\left[h\left(t\right);f\left(\theta \right)\right]+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em},$$

(11)

where:

$$N_i\left[h\left(t\right);f\left(\theta \right);t\right]\triangleq {\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h\left(t\right);f\left(\theta \right);t\right]\frac{d^n{h}_i\left(t\right)}{d{t}^n}}; i=1,\dots ,TH\hspace{0.17em}.$$

(12)

The first-order G-variations of Equations (1) and (2), respectively, are obtained, by definition, as follows:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d{N}_i\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f;t\right]}{d\epsilon }}\right\}}_{\epsilon =0}={\left\{{\displaystyle \frac{d{g}_i\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f\right]}{d\epsilon }}\right\}}_{\epsilon =0}\\ +{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left({\theta }^0\right)+\epsilon \delta f;t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau {\psi }_j\left[h^0\left(\tau \right)+\epsilon v^{\left(1\right)}\left(\tau \right);f\left({\theta }^0\right)+\epsilon \delta f;\tau \right]}\right\}}_{\epsilon =0}.\end{array}$$

(13)

$${\left\{{\displaystyle \frac{d}{d\epsilon }}{B}_j\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f\left({\theta }^0\right)+\epsilon \delta f;t\right]\right\}}_{\epsilon =0}=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}t= t_f;\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.$$

(14)

Carrying out the operations indicated in Equation (13) yields the following NIDE-V system for the function $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$:

$$\begin{array}{l}{\left\{{\displaystyle \sum _{k=1}^{TH}\frac{\partial N_i\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}}\right\}}_{\left(h^0,f^0\right)}\\ -{{\displaystyle \sum _{k=1}^{TH}\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}}_{\left(h^0,f^0\right)}={\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f\right)\delta f_k},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(15)

$$\begin{array}{l}{\left\{{\displaystyle \sum _{k=1}^{TH}\frac{\partial B_j\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}}_{\left(h^0,f^0\right)}-{\left\{{\displaystyle \sum _{k=1}^{TF}\frac{\partial B_j\left[h;f\left(\theta \right);t\right]}{\partial f_k}}\delta f_k\right\}}_{\left(h^0,f^0\right)}=0,\\ at\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0; t=t_f;j=1,\dots ,BC;\end{array}$$

(16)

where:

$$\begin{array}{l}{q}_{i,k}^{\left(1\right)}\left(h;f\right)\triangleq -{\displaystyle \frac{\partial N_i\left(h;f;t\right)}{\partial f_k}}+{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}+{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}};\hspace{0.5em}i=1,\dots ,TH;\hspace{0.5em}k=1,\dots ,TF.\end{array}$$

(17)

$$\begin{array}{l}{\displaystyle \sum _{k=1}^{TH}\frac{\partial N_i\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}}{v}_k^{\left(1\right)}\left(t\right)\triangleq {\displaystyle \sum _{n=1}^N\frac{d^n{h}_i\left(t\right)}{d{t}^n}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,n}\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}}\\ +{\displaystyle \sum _{n=1}^N{c}_{i,n}\left[h;f\left(\theta \right);t\right]\frac{d^n{v}_i^{\left(1\right)}\left(t\right)}{d{t}^n}}; i=1,\dots ,TH;\end{array}$$

(18)

$${\displaystyle \frac{\partial N_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_k}}\triangleq {\displaystyle \sum _{n=1}^N\frac{\partial c_{i,n}\left[f\left(\theta \right);t\right]}{\partial f_k}\frac{d^n{h}_i\left(t\right)}{d{t}^n}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,\dots ,TF.$$

(19)

The NIDE-V system represented by Equations (15) and (16) is called [28] the “1st-Level Variational Sensitivity system” (1st-LVSS) and its solution, $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$, is called [28] “1st-level variational function.” It is important to note that the 1st-LVSS is linear in the variational function $v^{\left(1\right)}\left(t\right)$, although it remains nonlinear in $h\left(t\right)$, in general. Note also that the 1st-LVSS would need to be solved anew for each variation $\delta f_j$, $j=1,\dots ,TF$, which is prohibitively expensive computationally if $TF$ is a large number.

The 1st-LVSS can formally be written in operator form as follows:

$$L\left(h;f\right)v^{\left(1\right)}=\hspace{0.17em}{Q}^{\left(1\right)}\left(h;f\right)\delta f,$$

(20)

where $Q^{\left(1\right)}\left(h;f\right)$ denotes the $TH\times TF$ rectangular matrix having as elements the quantities $q_{i,k}^{\left(1\right)}\left(h;f;t\right),\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\hspace{0.17em}\hspace{0.17em}k=1,\dots ,TF$, and where the components of the vector-valued operator $L\left(h;f\right)v^{\left(1\right)}\triangleq \left[L_1\left(h;f\right)v^{\left(1\right)},\dots ,L_{TH}\left(h;f\right)v^{\left(1\right)}\right]$ are defined as follows:

$$\begin{array}{l}{L}_i\left(h;f\right)v^{\left(1\right)}\triangleq {\displaystyle \sum _{k=1}^{TH}\frac{\partial N_i\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ -{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau }{\displaystyle \frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}}{v}_k^{\left(1\right)}\left(\tau \right);\hspace{0.5em}i=1,\dots ,TH.\end{array}$$

(21)

The need for repeatedly solving the 1st-LVSS can be avoided if the variational function $v^{\left(1\right)}\left(t\right)$ could be eliminated from appearing in the expression of the indirect-effect term defined in Equation (10). This goal can be achieved by expressing the right-side of Equation (10) in terms of the solutions of the “1st-Level Adjoint Sensitivity System” (1st-LASS), to be constructed next. The construction of this 1st-LASS is performed in a Hilbert space comprising elements of the same form as $v^{\left(1\right)}\left(t\right)\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, defined on the domain ${\mathsf{\Omega }}_t\triangleq t\in \left[t_0,t_f\right]$. This Hilbert space is considered to be endowed with an inner product denoted as ${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_1$, where ${\chi }^{(1)}\left(t\right)\triangleq {\left[{\chi }_1^{\left(1\right)}\left(t\right),\dots ,{\chi }_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$ and ${\eta }^{(1)}\left(t\right)\triangleq {\left[{\eta }_1^{\left(1\right)}\left(t\right),\dots ,{\eta }_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, and defined as follows:

$${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_1\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }^{(1)}\left(t\right)·{\eta }^{(1)}\left(t\right)dt\triangleq }{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\left[{\chi }^{(1)}\left(t\right)\right]}^{†}{\eta }^{(1)}\left(t\right)dt}=\hspace{0.17em}{\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\chi }_i^{\left(1\right)}\left(t\right){\eta }_i^{\left(1\right)}\left(t\right)dt}}.$$

(22)

The next step is to construct the inner product of Equation (15) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, where the superscript “(1)” indicates “1st-Level”, to obtain the following relationship:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{{\displaystyle \sum _{k=1}^{TH}\frac{\partial N_i\left[h;f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(23)

The terms appearing in Equation (23) are to be computed at the nominal values $\left(h^0;f^0\right)$ but the superscript “zero” has been omitted for simplicity. The relation in Equation (23) can formally be written in inner-product form as follows:

$${〈a^{\left(1\right)}\left(t\right),L\left(h;f\right)v^{\left(1\right)}〉}_1=\hspace{0.17em}{〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\left(h;f\right)\delta f〉}_1.$$

(24)

Using the definition of the adjoint operator in ${\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, the term on the left-side of Equation (20) is integrated by parts and the order of summations is reversed to obtain the following relation:

$${〈a^{\left(1\right)}\left(t\right),L\left(h;f\right)v^{\left(1\right)}〉}_1=\hspace{0.17em}{〈v^{\left(1\right)}\left(t\right),L^{*}\left(h;f\right)a^{\left(1\right)}〉}_1+P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right),$$

(25)

where the vector-valued operator $L^{*}\left(h;f;a^{\left(1\right)}\right)$ is the formal adjoint of the vector-valued operator $L\left(h;f;v^{\left(1\right)}\right)$ and where $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ represents the scalar-valued bilinear concomitant evaluated on the boundary $t=t_0$ and $t=t_f$ of ${\mathsf{\Omega }}_t$.

It follows from Equations (20) and (25) that the following relation holds:

$${〈v^{\left(1\right)}\left(t\right),L^{*}\left(h;f\right)a^{\left(1\right)}〉}_1={〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\left(h;f\right)\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right).$$

(26)

The term on the left-side of Equation (26) is now required to represent the indirect effect term defined in Equation (10) by imposing the following relation:

$$L^{*}\left(h;f\right)a^{\left(1\right)}={\displaystyle \frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h\left(t\right)}}.$$

(27)

Using Equations (26) and (27) in Equation (10) yields the following expression for the indirect effect term:

$${\left\{\delta R\left(h^0;F^0;v^{\left(1\right)}\right)\right\}}_{ind}={\left\{{〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\left(h;f\right)\delta f〉}_1-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\right\}}_{\left(h^0;f^0\right)}.$$

(28)

The boundary conditions accompanying Equation (27) for the function $a^{\left(1\right)}\left(t\right)$ are now chosen at the time values $t=t_f$ and/or $t=t_0$ so as to eliminate all unknown values of the 1st-level variational function $v^{\left(1\right)}\left(t\right)$ from the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$, which remain after implementing the initial conditions provided in Equation (2). These boundary conditions for the function $a^{\left(1\right)}\left(t\right)$ can be represented in operator form as follows:

$$B_j^{*}\left[h\left(t\right);a^{\left(1\right)}\left(t\right);f\left(\theta \right);t\right]=0; at t=t_f\hspace{0.5em}and/or\hspace{0.5em}t=t_0\hspace{0.17em};\hspace{0.5em}j=1,\dots ,BC.$$

(29)

The Volterra-like NIDE-V system represented by Equations (27) and (29) will be called the “1st-Level Adjoint Sensitivity System” and the solution, $a^{(1)}\left(t\right)$, will be called the “1st-level adjoint sensitivity function.” The 1st-LASS is solved using the nominal/optimal values for the parameters and for the function $h\left(t\right)$, but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of any parameter variations so it needs to be solved just once to obtain the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$ The 1st-LASS is linear in $a^{(1)}\left(t\right)$ but is, in general, nonlinear in $h\left(t\right)$.

Adding the expression of the indirect-effect term obtained in Equation (28) with the expression of the direct-effect term shown in Equation (9) yields the following expression for the total G-differential $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$:

$$\delta R\left(h;f;a^{\left(1\right)};\delta f\right)={〈a^{\left(1\right)}\left(t\right),Q^{\left(1\right)}\left(h;f\right)\delta f〉}_1+{\displaystyle \sum _{i=1}^{TF}\left(\delta f_i\right){\displaystyle \underset{t_0}{\overset{t_f}{\int }}\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_i}dt}}-\widehat{P}\left(h;f;\delta f\right),$$

(30)

where the quantity $\widehat{P}\left(h;f;\delta f\right)$ denotes the known non-zero boundary terms that might remain after using, in Equation (28), the boundary conditions provided in Equations (16) and (29). The first-order sensitivities of the decoder response with respect to the components $f_i\left(\theta \right)$ of the feature function $f\left(\theta \right)$ are determined by identifying, in Equation (30), the quantities that multiply the variations $\left(\delta f_i\right)$. Consequently, Equation (30) can be represented formally as follows:

$$\delta R\left(h;f;a^{\left(1\right)};\delta f\right)={\displaystyle \sum _{i=1}^{TF}\left(\delta f_i\right)R_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right);\hspace{0.5em}{R}_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right)\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}{D}_i^{\left(1\right)}\left(h;f;a^{\left(1\right)};t\right)dt}}\hspace{0.17em},$$

(31)

where $R_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right)$ denotes the first-order sensitivities, as indicated by the superscript “(1)”, of the decoder response with respect to the components $f_i\left(\theta \right)$ of the feature function $f\left(\theta \right)$. In view of Equation (30), these first-order sensitivities are functionals that can always be represented in the integral form shown in Equation (31).

The sensitivities with respect to the primary model parameters can be obtained by using the result shown in Equation (31) together with the “chain rule” of differentiating compound functions, as follows:

$${\displaystyle \frac{\partial R}{\partial {\theta }_j}}={\displaystyle \sum _{i=1}^{TF}\frac{\partial R}{\partial f_i}\frac{\partial f_i}{\partial {\theta }_j}}\hspace{0.17em},\hspace{1em}j=1,\dots ,TW.$$

(32)

When there are only model parameters (i.e., there are no feature functions of model parameters), then $f_i\left(\theta \right)\equiv {\theta }_i$ for all $i=1,\dots ,TF\triangleq TW$, and the expression obtained in Equation (31) directly yields the first-order sensitivities $\partial R/\partial {\theta }_j$ for all $j=1,\dots ,TW$. In this case, all the sensitivities $\partial R/\partial {\theta }_j$, for all $j=1,\dots ,TW$ would be obtained by computing integrals using quadrature formulas. In contradistinction, when features of parameters can be established, only $TF$ $\left(TF<TW\right)$ integrals would need to be computed (using quadrature formulas) to obtain the $\partial R/\partial F_j$, $j=1,\dots ,TF$; the sensitivities with respect to the model parameters would subsequently be obtained analytically using the chain-rule provided in Equation (32).

In the Section 2.2, below, the general 1st-FASAM-NIDE-V methodology will be particularized for the second-order (n = 2) 2nd-NIDE-V net to illustrate explicitly the form of the bilinear concomitant and the ensuing adjoint boundary conditions. Notably, the 2nd-NIDE-V system subsumes the first-order 1st-NIDE-V as a particular case.

2.2. Second-Order Neural Integral Equations of Volterra-Type (2nd-NIDE-V)

This subsection presents the development of the 1st-FASAM-NIDE-V methodology for the particular case of the Second-Order Neural Integral Equations of Volterra-Type (2nd-NIDE-V). This development will present the explicit expressions of the corresponding bilinear concomitant and ensuing adjoint boundary conditions. The representation of the second-order $\left(n=2\right)$ neural integral equations of Volterra-type (2nd-NIDE-V) is provided below, for $\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH$:

$$\begin{array}{l}{c}_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}+c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}=g_i\left[h\left(t\right);f\left(\theta \right)\right]\\ +{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}\hspace{0.17em}d\tau \hspace{0.17em};\hspace{0.5em}i=1,\dots ,TH.\end{array}$$

(33)

There are several combinations of boundary conditions that can be provided for the function $h_i\left(t\right)$ and/or for its first-derivative $d{h}_i\left(t\right)/dt$, $i=1,\dots ,TH$, either at $t=t_0$ (encoder) or at $t=t_f$ (decoder), or a combination thereof. For illustrative purposes, consider that the boundary conditions are as follows:

$$h_i\left(t_0\right)=e_i\hspace{0.17em}; h_i\left(t_f\right)=d_i\hspace{0.17em}; i=1,\dots ,TH.$$

(34)

The 1st-LVSS is obtained by taking the G-variations of Equations (33) and (34) to obtain the following system, comprising the forms taken on for $n=2$ by Equations (15) and (16), for the variational function $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$:

$$\begin{array}{l}{\left\{c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{\left(h^0,f^0\right)}{\displaystyle \frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}}+{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{\left(h^0,f^0\right)}}\\ +{\left\{c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]\right\}}_{\left(h^0,f^0\right)}{\displaystyle \frac{d^2{v}_i^{\left(1\right)}}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}{\left\{\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}_{\left(h^0,f^0\right)}}\\ -{\left\{{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[f\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h;f\left(\theta \right);\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}\right\}}_{\left(h^0,f^0\right)}\\ -{{\displaystyle \sum _{k=1}^{TH}\left\{\frac{\partial g_i\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)\right\}}}_{\left(h^0,f^0\right)}={\displaystyle \sum _{k=1}^{TF}{\left\{q_{i,k}^{\left(1\right)}\left(h;f\right)\right\}}_{\left(h^0,f^0\right)}\delta f_k},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TH;\end{array}$$

(35)

$$v_i^{\left(1\right)}\left(t_0\right)=\delta e_i\hspace{0.17em};\hspace{0.5em}{v}_i^{\left(1\right)}\left(t_f\right)=\delta d_i;\hspace{0.5em}i=1,\dots ,TH\hspace{0.17em};$$

(36)

where the following definition was used for $q_{ik}^{\left(1\right)}\left(h;f\right)$, $i=1,\dots ,TH;$ $k=1,\dots ,TF$:

$$\begin{array}{l}{q}_{ik}^{\left(1\right)}\left(h;f\right)\triangleq -{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}{\displaystyle \frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}-{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_k}}+{\displaystyle \frac{\partial g_i\left(h;f;t\right)}{\partial f_k}}\\ +{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\phi }_{i,j}\left(f;t\right)}{\partial f_k}}{\displaystyle \underset{t_0}{\overset{t}{\int }}{\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau \frac{\partial {\psi }_j\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_k}}.\end{array}$$

(37)

The 1st-LASS is constructed by forming the inner product of Equation (35) with a vector $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$ to obtain the following relationship:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left[h\left(t\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+c_{i,2}\left[h\left(t\right);f\right]{\displaystyle \frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left[h\left(t\right);f\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left[h\left(\tau \right);f;\tau \right]}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left[h\left(t\right);f;t\right]}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta f_k}}\hspace{0.17em}.\end{array}$$

(38)

Examining the structure of the left-side of Equation (38) reveals that the bilinear concomitant will arise from the integration by parts of the first and third terms the on the left-side of Equation (38), as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}dt}}=P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i.1}\left(h;f\right)\right\}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d^2}{d{t}^2}\left\{a_i^{\left(1\right)}\left(t\right)c_{i.2}\left(h;f\right)\right\}dt}},\end{array}$$

(39)

where the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ has the following expression:

$$\begin{array}{l}P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq {\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,1}\left[h\left(t_f\right);f\right]v_i^{\left(1\right)}\left(t_f\right)-a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left[h\left(t_0\right);f\right]v_i^{\left(1\right)}\left(t_0\right)\right\}}\\ +{\displaystyle \sum _{i=1}^{TH}\left\{a_i^{\left(1\right)}\left(t_f\right)c_{i,2}\left[h\left(t_f\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t_f\right)}{dt}-a_i^{\left(1\right)}\left(t_0\right)c_{i,2}\left[h\left(t_0\right);f\right]\frac{d{v}_i^{\left(1\right)}\left(t_0\right)}{dt}\right\}}\\ -{{\displaystyle \sum _{i=1}^{TH}\left\{v_i^{\left(1\right)}\left(t_f\right)\frac{d}{dt}\left[a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right]\right\}}}_{t=t_f}+{{\displaystyle \sum _{i=1}^{TH}\left\{v_i^{\left(1\right)}\left(t_0\right)\frac{d}{dt}\left[a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right]\right\}}}_{t=t_0}.\end{array}$$

(40)

The second term on the left-side of Equation (38) will be recast in its “adjoint form” by reversing the order of summations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left(h;f\right)}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,1}\left(h;f\right)}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}dt}}\hspace{0.17em}.$$

(41)

The fourth term on the left-side of Equation (38) will be recast in its “adjoint form” by reversing the order of summations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, just as was done for obtaining the relation in Equation (41), to obtain the following relation:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\frac{d^2{h}_i\left(t\right)}{d{t}^2}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left(h;f\right)}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,2}\left(h;f\right)}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}}\hspace{0.17em}.$$

(42)

The fifth term on the left-side of Equation (38) is now recast in its “adjoint form” by “integrating by parts” and reversing the order of summations and integrations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\hspace{0.17em}{a}_i^{\left(1\right)}\left(t\right)}}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t}{\int }}{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left(h;f;\tau \right)}{\partial h_k\left(\tau \right)}\hspace{0.17em}}{v}_k^{\left(1\right)}\left(\tau \right)d\tau }\\ ={\left\{{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t}{\int }}dt\hspace{0.17em}{a}_i^{\left(1\right)}\left(t\right)}}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t}{\int }}{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left(h;f;\tau \right)}{\partial h_k\left(\tau \right)}\hspace{0.17em}}{v}_k^{\left(1\right)}\left(\tau \right)d\tau }\right\}}_{t=t_f}\\ -{\left\{{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t}{\int }}dt\hspace{0.17em}{a}_i^{\left(1\right)}\left(t\right)}}{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t}{\int }}{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left(h;f;\tau \right)}{\partial h_k\left(\tau \right)}\hspace{0.17em}}{v}_k^{\left(1\right)}\left(\tau \right)d\tau }\right\}}_{t=t_0}\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\hspace{0.17em}{v}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left(h;f;t\right)}{\partial h_i\left(t\right)}}}}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left(f;\tau \right)\hspace{0.17em}}d\tau }\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\hspace{0.17em}{v}_i^{\left(1\right)}\left(t\right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left(h;f;t\right)}{\partial h_i\left(t\right)}}}}\hspace{0.17em}{\displaystyle \underset{t}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left(f;\tau \right)\hspace{0.17em}}d\tau }\end{array}$$

(43)

The sixth term on the left-side of Equation (38) will be recast in its “adjoint form” by reversing the order of summations and integrations so as to transform the inner product involving the function $a^{(1)}\left(t\right)$ into an inner product involving the function $v^{\left(1\right)}\left(t\right)$, as follows:

$${\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left(h;f;t\right)}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left(h\left(t\right);f;t\right)}{\partial h_i\left(t\right)}}.$$

(44)

Collecting the results obtained in Equations (39)–(44) yields the following expression for the left-side of Equation (38):

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}\left\{c_{i,1}\left(h;f\right)\frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}+\frac{d{h}_i\left(t\right)}{dt}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,1}\left(h;f\right)}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right.}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+c_{i,2}\left(h;f\right){\displaystyle \frac{d^2{v}_i^{\left(1\right)}\left(t\right)}{d{t}^2}}+{\displaystyle \frac{d^2{h}_i\left(t\right)}{d{t}^2}}{\displaystyle \sum _{k=1}^{TH}\frac{\partial c_{i,2}\left(h;f\right)}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\\ \left.-{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left(f;t\right)}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau }{\displaystyle \sum _{k=1}^{TH}\frac{\partial {\psi }_j\left(h;f;\tau \right)}{\partial h_k\left(\tau \right)}{v}_k^{\left(1\right)}\left(\tau \right)}-{\displaystyle \sum _{k=1}^{TH}\frac{\partial g_i\left(h;f;t\right)}{\partial h_k\left(t\right)}{v}_k^{\left(1\right)}\left(t\right)}\right\}\\ =P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d}{dt}\left[-a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right]dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)\frac{d^2}{d{t}^2}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right\}dt}}\hspace{0.17em}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,1}\left(h;f\right)}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i^{\left(1\right)}\left(t\right)}\frac{\partial c_{i,2}\left(h;f\right)}{\partial h_i\left(t\right)}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}}-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left(h;f;t\right)}{\partial h_i\left(t\right)}}\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \hspace{0.17em}{v}_i^{\left(1\right)}\left(\tau \right){\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left(h;f;\tau \right)}{\partial h_i\left(\tau \right)}}}}\hspace{0.17em}{\displaystyle \underset{t}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right){\phi }_{k,j}\left(f;t\right)\hspace{0.17em}}dt}\end{array}$$

(45)

Using Equation (39) and rearranging the terms on the right-side of Equation (45) yields the following relation:

$$\begin{array}{l}{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{i,k}^{\left(1\right)}\left(h;f;t\right)\delta F_k}}-P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\\ ={\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{v}_i\left(t\right)dt\left\{-\frac{d}{dt}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right\}\right.}}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right\}\\ +{\displaystyle \frac{\partial c_{i,1}\left(h;f\right)}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}+{\displaystyle \frac{\partial c_{i,2}\left(h;f\right)}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}\\ -\left.{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left(h;f;t\right)}{\partial h_i\left(t\right)}}-{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left(h;f;t\right)}{\partial h_i\left(t\right)}}{\displaystyle \underset{t}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left(f;\tau \right)\hspace{0.17em}}d\tau }\right\}.\end{array}$$

(46)

The term on the right-side of Equation (46) is now required to represent the “indirect-effect” term defined in Equation (10), which is achieved by requiring the components of the function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ to satisfy the following 1st-LASS:

$$\begin{array}{l}-{\displaystyle \frac{d}{dt}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right\}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right\}+{\displaystyle \frac{\partial c_{i,1}\left(h;f\right)}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}\\ +{\displaystyle \frac{\partial c_{i,2}\left(h;f\right)}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left(h;f;t\right)}{\partial h_i\left(t\right)}}\\ -{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left(h;f;t\right)}{\partial h_i\left(t\right)}}{\displaystyle \underset{t}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left(f;\tau \right)\hspace{0.17em}}d\tau }={\displaystyle \frac{\partial D\left(h;f;t\right)}{\partial h_i\left(t\right)}}.\end{array}$$

(47)

Recalling from Equation (36) that the values $v_i^{\left(1\right)}\left(t_0\right)$ and $v_i^{\left(1\right)}\left(t_f\right)$, $i=1,\dots ,TH$, are known, it follows that the unknown values involving the function $v_i\left(t\right)$ in the bilinear concomitant $P\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ defined in Equation (40) are eliminated by imposing the following conditions on the components of the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$:

$$a_i^{\left(1\right)}\left(t_0\right)=0\hspace{0.17em};\hspace{0.5em}{a}_i^{\left(1\right)}\left(t_f\right)=0\hspace{0.17em};\hspace{0.5em}i=1,\dots ,TH\hspace{0.17em}.$$

(48)

It follows from Equations (45)–(48) that the indirect-effect term defined in Equation (10) has the following expression in terms of the 1st-level adjoint sensitivity function $a^{(1)}\left(t\right)$:

$${\left\{\delta R\left(h;f;a^{(1)}\right)\right\}}_{ind}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{k=1}^{TF}{q}_{ik}^{\left(1\right)}\left(h;f;t\right)\delta f_k}-\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right),$$

(49)

where the boundary quantity $\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$ contains the remaining terms after having implemented the known boundary conditions given in Equations (36) and (48), and has the following explicit expression:

$$\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)\triangleq -{{\displaystyle \sum _{i=1}^{TH}\delta d_i\left\{c_{i,2}\left(h;f\right)\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}+{{\displaystyle \sum _{i=1}^{TH}\delta e_i\left\{c_{i,2}\left(h;f\right)\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_0}.$$

(50)

Using the results obtained in Equations (9), (37), (49), and (50) in Equation (8) yields the following expression for the G-variation $\delta R\left(h^0;f^0;v^{\left(1\right)};\delta f\right)$, which is seen to be linear in the variations $\delta d_i$, $\delta e_i$ ($i=1,\dots ,TH$) and $\delta f_j$ ($j=1,\dots ,TF$):

$$\begin{array}{l}\delta R\left[h\left(t\right);f\left(\theta \right);a^{(1)}\left(t\right);\delta f\right]\triangleq {\displaystyle \sum _{j=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\frac{\partial D\left(h;f;t\right)}{\partial F_j}\delta f_j}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)dt}{\displaystyle \sum _{j=1}^{TF}{q}_{jk}^{\left(1\right)}\left(h;f;t\right)\delta f_j}}\\ +{{\displaystyle \sum _{i=1}^{TH}\delta d_i\left\{c_{i,2}\left(h;f\right)\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}\right\}}}_{t=t_f}-{{\displaystyle \sum _{i=1}^{TH}\delta e_i\left\{c_{i,2}\left(h;f\right)\frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}-a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left(h;f\right)\right\}}}_{t=t_0}\\ \triangleq {\displaystyle \sum _{j=1}^{TF}\frac{\partial R}{\partial f_j}}\delta f_j+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial d_i}}\delta d_i+{\displaystyle \sum _{i=1}^{TH}\frac{\partial R}{\partial e_i}}\delta e_i\hspace{0.17em}\hspace{0.17em}.\end{array}$$

(51)

The expression in Equation (51) is to be satisfied at the nominal/optimal values for the respective model parameters, but this fact has not been indicated explicitly in order to simplify the notation.

It also follows from Equations (51) and (37) that the sensitivities $\partial R/\partial f_j$ of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the components $f_j\left(\theta \right)$ of the feature function $f\left(\theta \right)$ have the following expressions:

$$\begin{array}{l}{\displaystyle \frac{\partial R\left[h;f\left(\theta \right)\right]}{\partial f_j}}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}dt}-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}dt}\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}\frac{d^2{h}_i\left(t\right)}{d{t}^2}\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}dt}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(h;f;t\right)}{\partial f_j}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}{\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}{\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}\hspace{0.17em}dt{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.5em}j=1,\dots ,TF.\end{array}$$

(52)

Identifying in Equation (51) the expressions that multiply the variations $\delta e_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the encoder’s initial-time conditions:

$${\displaystyle \frac{\partial R}{\partial e_i}}={\left\{-c_{i,2}\left(h;f\right){\displaystyle \frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}}+a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left(h;f\right)\right\}}_{t=t_0};\hspace{0.5em}i=1,\dots ,TH.$$

(53)

Identifying in Equation (51) the expressions that multiply the variations $\delta d_i\hspace{0.17em}$ yields the following expressions for the decoder response sensitivities with respect to the final-time conditions:

$${\displaystyle \frac{\partial R}{\partial d_i}}={\left\{c_{i,2}\left(h;f\right){\displaystyle \frac{d{a}_i^{\left(1\right)}\left(t\right)}{dt}}\right\}}_{t=t_f};\hspace{0.5em}i=1,\dots ,TH.$$

(54)

As a particular case, the 2nd-NIDE-V system considered in Equations (33) and (34) encompasses the 1st-NIDE-V below, which is obtained by setting $c_{i,2}\left(h;f\right)\equiv 0$ and $d_i\equiv 0$, $i=1,\dots ,TH,$ in these equations:

$$c_{i,1}\left[h\left(t\right);F\left(\theta \right)\right]{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}=g_i\left[h\left(t\right);F\left(\theta \right)\right]+{\displaystyle \sum _{j=1}^{TL}{\phi }_{i,j}\left[F\left(\theta \right);t\right]}{\displaystyle \underset{t_0}{\overset{t}{\int }}d\tau {\psi }_j\left[h\left(\tau \right);F\left(\theta \right);\tau \right]}\hspace{0.17em}.$$

(55)

$$h_i\left(t_0\right)=e_i\hspace{0.17em};\hspace{0.5em}i=1,\dots ,TH\hspace{0.17em},$$

(56)

The first-order sensitivities of the decoder with respect to the feature functions and initial conditions for the above 1st-NIDE-V are obtained by setting $c_{i,2}\left(h;f\right)\equiv 0$ and $d_i\equiv 0$, $i=1,\dots ,TH$, in Equations (52)–(54), to obtain the following expressions:

$$\begin{array}{l}{\displaystyle \frac{\partial R\left[h;f\left(\theta \right)\right]}{\partial f_j}}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}dt}-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}dt}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(h;f;t\right)}{\partial f_j}dt}}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}{\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}{\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}\hspace{0.17em}dt{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.5em}j=1,\dots ,TF.\end{array}$$

(57)

$${\displaystyle \frac{\partial R}{\partial e_i}}={\left\{a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left(h;f\right)\right\}}_{t=t_0};\hspace{0.5em}i=1,\dots ,TH.$$

(58)

It is apparent from Equation (58) that, for the 1st-NIDE-V defined by Equations (55) and (56), the sensitivities $\partial R/\partial e_i$ are proportional to the values of the respective component $a_i^{\left(1\right)}\left(t_0\right)$ of the 1st-level adjoint function evaluated at the initial-time $t=t_0$. This relation provides an independent mechanism for verifying the correctness of solving the 1st-LASS from $t=t_f$ to $t=t_0$ (backwards in time) since the sensitivities $\partial R/\partial e_i$ can be computed independently of the 1st-LASS using finite differences of appropriately high-order in conjunction with known variations $\delta e_i\hspace{0.17em}$ and the correspondingly induced variations in the decoder response. Special attention needs to be devoted, however, to ensure that the respective finite-difference formula is accurate, which may need several trials with different values chosen for the variation $\delta e_i\hspace{0.17em}$.

Effects of Alternative Boundary Conditions for the Forward Functions $h_i\left(t\right)$

If the boundary conditions imposed on the forward functions $h_i\left(t\right)$ and/or the first-derivatives $d{h}_i\left(t\right)/dt$, $i=1,\dots ,TH$, differ from the illustrative ones selected in Equation (34) then the corresponding boundary conditions for the 1st-level adjoint function $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ would also differ from the ones shown in Equation (48), as would be expected. The components of $a^{\left(1\right)}\left(t\right)\triangleq {\left[a_1^{\left(1\right)}\left(t\right),\dots ,a_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ would consequently have different values and so all of the first-order sensitivities $\partial R/\partial F_j$ would have values different from those computed using Equation (52), even though the formal mathematical expressions of the respective sensitivities would remain unchanged. Of course, the sensitivities $\partial R/\partial e_i$ and $\partial R/\partial d_i$ would have expressions that would differ from those in Equations (53) and (54), respectively, if the boundary conditions in Equation (34), and consequently those in Equation (48), were different. This is because the residual bilinear concomitant $\widehat{P}\left(h;F;v^{\left(1\right)};a^{\left(1\right)}\right)$ would have a different expression from that shown in Equation (50).

The above considerations will be illustrated next by replacing the boundary conditions considered in Equation (34) by the following typical “initial conditions” for the encoder of the NIDE-V net defined in Equation (33):

$$h_i\left(t_0\right)=e_i\hspace{0.17em}; {\left\{{\displaystyle \frac{d{h}_i\left(t\right)}{dt}}\right\}}_{t=t_0}={\xi }_i\hspace{0.17em}; i=1,\dots ,TH\hspace{0.17em}.$$

(59)

Note that the same symbols, e.g., $h_i\left(t\right)$, $v_i^{\left(1\right)}\left(t\right)$, $a_i^{\left(1\right)}\left(t\right)$, etc., will be used in this “Subsubsection” for the various functions, even though the values of these functions will differ from the corresponding ones in Section 2.2, since the “initial encoder conditions” considered in Equation (59) differ from the “initial/final encoder/decoder conditions” considered in Equation (34).

The G-differentiated expressions of the above “initial conditions”, which are the counterparts of the expressions provided in Equation (36), are as follows:

$$v_i^{\left(1\right)}\left(t_0\right)=\delta e_i\hspace{0.17em};\hspace{0.5em}{\left\{{\displaystyle \frac{d{v}_i^{\left(1\right)}\left(t\right)}{dt}}\right\}}_{t=t_0}=\delta {\xi }_i\hspace{0.17em};\hspace{0.5em}i=1,\dots ,TH\hspace{0.17em}.$$

(60)

For the above boundary conditions, the 1st-LVSS for the function $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}\triangleq {\left[\delta h_1\left(t\right),\dots ,\delta h_{TH}\left(t\right)\right]}^{†}$ comprises Equations (35) and (60). The 1st-LASS corresponding to the 1st-LVSS comprising Equations (35) and (60) is obtained by following the same procedure as detailed in Section 2.2. The main distinction from the case considered in Section 2.2 will be the expression of the residual bilinear concomitant $\widehat{P}\left(h;f;v^{\left(1\right)};a^{\left(1\right)}\right)$. Thus, applying the same sequence of steps as in Section 2.2 to Equations (35) and (60), but omitting the details for brevity, yields the following expressions for the 1st-order sensitivities of the decoder response:

$$\begin{array}{l}{\displaystyle \frac{\partial R\left[h;f\left(\theta \right)\right]}{\partial f_j}}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}\frac{\partial D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j}dt}-{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}\frac{d{h}_i\left(t\right)}{dt}\frac{\partial c_{i,1}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}dt}\\ -{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}\frac{d^2{h}_i\left(t\right)}{d{t}^2}\frac{\partial c_{i,2}\left[h\left(t\right);f\left(\theta \right)\right]}{\partial f_j}dt}+{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)\frac{\partial g_i\left(h;f;t\right)}{\partial f_j}dt}}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}{\displaystyle \sum _{k=1}^{TL}\frac{\partial {\phi }_{i,k}\left(f;t\right)}{\partial f_j}}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]d\tau }}\\ +{\displaystyle \sum _{i=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_i^{\left(1\right)}\left(t\right)}{\displaystyle \sum _{k=1}^{TL}{\phi }_{i,k}\left(f;t\right)}\hspace{0.17em}dt{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau \frac{\partial {\psi }_k\left[h\left(\tau \right);f\left(\theta \right);\tau \right]}{\partial f_j}}};\hspace{0.5em}j=1,\dots ,TF.\end{array}$$

(61)

$${\displaystyle \frac{\partial R}{\partial e_i}}={\left\{a_i^{\left(1\right)}\left(t_0\right)c_{i,1}\left(h;f\right)-{\displaystyle \frac{d}{dt}}\left[a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right]\right\}}_{t=t_0};\hspace{0.5em}i=1,\dots ,TH.$$

(62)

$${\displaystyle \frac{\partial R}{\partial {\xi }_i}}=a_i^{\left(1\right)}\left(t_0\right)c_{i,2}\left[h\left(t_0\right);f\right];\hspace{0.5em}i=1,\dots ,TH.$$

(63)

The 1st-level adjoint function that appears in Equations (61)–(63) is the solution of a 1st-LASS comprising a system of NIDE-V equations that formally resembles the system presented in Equation (47) but having boundary conditions that differ from those in Equation (48), as follows:

$$\begin{array}{l}-{\displaystyle \frac{d}{dt}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,1}\left(h;f\right)\right\}+{\displaystyle \frac{d^2}{d{t}^2}}\left\{a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right\}+{\displaystyle \frac{\partial c_{i,1}\left(h;f\right)}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d{h}_k\left(t\right)}{dt}}\\ +{\displaystyle \frac{\partial c_{i,2}\left(h;f\right)}{\partial h_i\left(t\right)}}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{d^2{h}_k\left(t\right)}{d{t}^2}dt}-{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(t\right)\frac{\partial g_k\left(h;f;t\right)}{\partial h_i\left(t\right)}}\\ -{\displaystyle \sum _{j=1}^{TL}\frac{\partial {\psi }_j\left(h;f;t\right)}{\partial h_i\left(t\right)}}{\displaystyle \underset{t}{\overset{t_f}{\int }}{\displaystyle \sum _{k=1}^{TH}{a}_k^{\left(1\right)}\left(\tau \right){\phi }_{k,j}\left(f;\tau \right)\hspace{0.17em}}d\tau }={\displaystyle \frac{\partial D\left(h;f;t\right)}{\partial h_i\left(t\right)}}.\end{array}$$

(64)

$$a_i^{\left(1\right)}\left(t_f\right)=0\hspace{0.17em};\hspace{0.5em}{\left\{{\displaystyle \frac{d}{dt}}\left[a_i^{\left(1\right)}\left(t\right)c_{i,2}\left(h;f\right)\right]\right\}}_{t=t_f}=0;\hspace{0.5em}i=1,\dots ,TH\hspace{0.17em}.$$

(65)

3. Illustrative Application of the 1st-FASAM-NIDE-V Methodology to a Heat Conduction Model

The application of the 1st-FASAM-NIDE-V methodology will be illustrated in this section by considering a paradigm model of nonlinear heat conduction through a one-dimensional slab of thickness $\mathcal{l}$, made of material having a temperature dependent conductivity $k\left(T\right)$, insulated at one side at a temperature $T_0$. The slab is heated internally by a nonlinear temperature-dependent heat source that is proportional to the power within the slab. There are several reasons for choosing this illustrative paradigm heat conduction model, including:

(i)

Conduction heat transfer is a physical process occurring ubiquitously in energy engineering, being the subject of many textbooks [36,37,38,39,40,41].

(ii)

The illustrative model is nonlinear but admits an equivalent linear model that is related to the original model through the Kirchhoff transformation [40].

(iii)

The illustrative model can be represented either as a traditional NODE system or a NIDE-V system.

(iv)

The illustrative model admits explicit closed-form solutions for all quantities of interest, including the functions representing the hidden/latent neural networks, the decoder response, and the sensitivities of the decoder’s response to the optimal weights/parameters.

(v)

The illustrative model enables an exact comparison between the application of the 2nd-FASAM-NIDE-V methodology developed in Section 2 and the equivalent application of the 2nd-FASAM-NODE methodology developed in [30,31].

The traditional representation of the heat conduction process within the slab described above is by means of the following nonlinear second-order NODE-net:

$$\begin{array}{l}{\displaystyle \frac{d}{dx}}\left[k\left(T\right){\displaystyle \frac{dT\left(x\right)}{dx}}\right]+Q\left[T\left(x\right)+{\displaystyle \frac{\beta }{2}}{T}^2\left(x\right)\right]=0;\hspace{0.5em}0<x<\mathcal{l}\hspace{0.17em};\\ k\left(T\right)\triangleq k_0\left[1+\beta T\left(x\right)\right];\hspace{0.17em}\end{array}$$

(66)

$$T\left(0\right)=T_0\hspace{0.17em};\hspace{0.5em}{\left[{\displaystyle \frac{dT\left(x\right)}{dx}}\right]}_{x=0}=0\hspace{0.17em}.$$

(67)

The quantities $T_0$, $k_0$,$\beta$, and $Q$ in Equation (66) are imprecisely-known model parameters/weights subject to experimental uncertainties; these parameters are considered to be components of the “vector of model parameters” denoted as $\theta \triangleq {\left[T_0,k_0,\beta ,Q\right]}^{†}$. The optimal/nominal values of these parameters are considered to be known and are denoted using a superscript “zero”, as follows: $T_0^0$, $k_0^0$, ${\beta }^0$, $Q^0$, with ${\theta }^0\triangleq {\left[T_0^0,k_0^0,{\beta }^0,Q^0\right]}^{†}$.

Alternatively, the above heat conduction model can be represented by the following non-linear NIDE-V system:

$$\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{dT\left(x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\left[T\left(y\right)+\frac{\beta }{2}{T}^2\left(y\right)\right]dy}=0;\hspace{0.5em}{b}^2\left(\theta \right)\triangleq Q /k_0 ;\hspace{0.5em}0<x<\mathcal{l}\hspace{0.17em};$$

(68)

$$T\left(0\right)=T_0\hspace{0.17em}.$$

(69)

The decoder’s response of interest is chosen to be the “power per unit distance” (also called the “linear power”) developed within the slab, which depends nonlinearly on the temperature and is defined as follows:

$$R={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}k\left(T\right)}{\displaystyle \frac{dT\left(x\right)}{dx}}dx\hspace{0.17em}.$$

(70)

Heat conduction models involving temperature-dependent conductivities, $k\left(T\right)$, are typically solved by using the Kirchhoff transformation defined below:

$$U\left(T\right)={\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{T\left(x\right)}{\int }}k\left(\tau \right)}\hspace{0.17em}d\tau \hspace{0.17em}=T\left(x\right)+{\displaystyle \frac{\beta }{2}}{T}^2\left(x\right).$$

(71)

The NODE-form of the heat conduction model defined by Equations (66) and (67) has the following expression in the Kirchhoff-transformed framework:

$${\displaystyle \frac{d^{2}U\left(x\right)}{d{x}^2}}+b^2\left(\theta \right)U\left(x\right)=0;\hspace{1em}0<x<\mathcal{l}\hspace{0.17em};$$

(72)

$$U\left(0\right)=U_0\left(\theta \right)\triangleq T_0+{\displaystyle \frac{\beta }{2}}{T}_0^2\hspace{0.17em};\hspace{1em}{\left[{\displaystyle \frac{dU\left(x\right)}{dx}}\right]}_{x=0}=0\hspace{0.17em}.$$

(73)

On the other hand, the NIDE-V form of the heat conduction model defined by Equations (68) and (69) has the following form in the Kirchhoff-transformed framework:

$${\displaystyle \frac{dU\left(x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}=0;\hspace{0.5em}0<x<\mathcal{l}\hspace{0.17em};$$

(74)

$$U\left(0\right)=U_0\left(\theta \right)\hspace{0.17em}.$$

(75)

The expression of the decoder response in the Kirchhoff-transformed framework is obtained by using Equation (71) in Equation (70), which yields:

$$R=k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\frac{dU\left(x\right)}{dx}dx}=k_0\left[U\left(\mathcal{l}\right)-U\left(0\right)\right]=k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}U\left(x\right)\delta \left(x-\mathcal{l}\right)dx}-k_0{U}_0\left(\theta \right)\hspace{0.17em}.$$

(76)

Solving either the NODE or the NIDE-V forms of the illustrative heat conduction model, in either the Kirchhoff-transformed framework or the temperature-framework, yields the following exact closed-form expressions, respectively:

$$U\left(x\right)=U_0\left(\theta \right)\cos xb\left(\theta \right);$$

(77)

$$T\left(x\right)=\left[\sqrt{1+2\beta U\left(x\right)}-1\right]/\beta =\left[\sqrt{1+2\beta U_0\left(\theta \right)\cos xb\left(\theta \right)}-1\right]/\beta .$$

(78)

Using Equation (76), the result obtained in Equation (77), or, alternatively, using in Equation (70) the result obtained Equation (78), yields the following closed-form expression for the decoder response:

$$R=k_0{U}_0\left(\theta \right)\left[\cos \mathcal{l}b\left(\theta \right)-1\right]=k_0\left(T_0+{\displaystyle \frac{\beta }{2}}{T}_0^2\right)\left[\cos \mathcal{l}b\left(\theta \right)-1\right].$$

(79)

3.1. Application of the 1st-FASAM-NIDE-V to Compute 1st-Order Sensitivities of the NIDE-V Representation of the Illustrative Heat Conduction Model

The 1st-FASAM-NIDE-V methodology will be applied in this subsection to the paradigm heat conduction model in order to illustrate the path to obtaining the exact expressions of the first-order sensitivities of the decoder response with respect to the model parameters and subsequently compute these sensitivities most efficiently, requiring a single large-scale computation for solving the respective 1st-Level Adjoint Sensitivity System (1st-LASS). The illustrative mathematical derivations will be performed in the temperature-framework formulation (in Section 3.1.1) and also in the Kirchhoff-transformed framework formulation (in Section 3.1.2) in order to illustrate the advantages of using linearizing transformations, such as Kirchhoff’s transformation, whenever possible.

3.1.1. Representation in the NIDE-V Temperature-Framework

The first-order sensitivities of the decoder response defined in Equation (70) are obtained by determining the first-order G-differential of the respective expression, for variations $\delta T$ and $\delta \theta$ around the nominal values ${\theta }^0\triangleq {\left[T_0^0,k_0^0,{\beta }^0,Q^0\right]}^{†}$, which is by definition obtained as follows:

$$\begin{array}{l}\delta R\left(T^0,{\theta }^0;\delta T,\delta \theta \right)\triangleq \left\{{\displaystyle \frac{d}{d\epsilon }}\left(k_0^0+\epsilon \delta k_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\left({\beta }^0+\epsilon \delta \beta \right)\left(T^0+\epsilon \delta T\right)\right]}\right.\\ {\left.\times {\displaystyle \frac{d\left[T^0\left(x\right)+\epsilon \delta T\left(x\right)\right]}{dx}}dx\right\}}_{\epsilon =0}=\left(\delta k_0\right){\left\{{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\beta T\left(x\right)\right]}{\displaystyle \frac{dT\left(x\right)}{dx}}\right\}}_{{\theta }^0}\\ +\left(\delta \beta \right){\left\{k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\frac{dT\left(x\right)}{dx}dx}\right\}}_{{\theta }^0}+{\left\{k_0\beta {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\frac{d}{dx}\left[T\left(x\right)\delta T\left(x\right)\right]dx}\right\}}_{{\theta }^0}\\ =\delta R{\left(T^0,{\theta }^0;\delta \theta \right)}_{dir}+\delta R{\left(T^0,{\theta }^0;\delta T\right)}_{ind},\end{array}$$

(80)

where the direct-effect term $\delta R{\left(T^0,{\theta }^0;\delta \theta \right)}_{dir}$ and indirect-effect term $\delta R{\left(T^0,{\theta }^0;\delta T\right)}_{ind}$, respectively, are defined below:

$$\begin{array}{l}\delta R{\left(T^0,{\theta }^0;\delta \theta \right)}_{dir}\triangleq \left(\delta k_0\right){\left\{{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\beta T\left(x\right)\right]}{\displaystyle \frac{dT\left(x\right)}{dx}}\right\}}_{{\theta }^0}\\ \hspace{1em}+\left(\delta \beta \right){\left\{k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\frac{dT\left(x\right)}{dx}dx}\right\}}_{{\theta }^0}-\left(\delta T_0\right){\left\{T_0{k}_0\beta \right\}}_{{\theta }^0};\end{array}$$

(81)

$$\delta R{\left(T^0,{\theta }^0;\delta T\right)}_{ind}\triangleq {\left\{k_0\beta T\left(\mathcal{l}\right)\delta T\left(\mathcal{l}\right)\right\}}_{{\theta }^0}={\left\{k_0\beta T\left(\mathcal{l}\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\delta \left(x-\mathcal{l}\right)dx}\right\}}_{{\theta }^0}.$$

(82)

The direct-effect term can be computed at this time, but the indirect-effect term can be computed after having determined the variational function $\delta T\left(x\right)$. This variational function is the solution of the 1st-LVSS obtained by G-differentiating Equations (68) and (69). Applying the definition of the G-differential to these equations yields the following relations for the variational function $\delta T\left(x\right)$:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}\left[1+\left({\beta }^0+\epsilon \delta \beta \right)\left(T^0+\epsilon \delta T\right)\right]{\displaystyle \frac{d\left(T^0+\epsilon \delta T\right)}{dx}}\right\}}_{\epsilon =0}\\ +{\left\{{\displaystyle \frac{d}{d\epsilon }}{\left(b^0+\epsilon \delta b\right)}^2{\displaystyle \underset{0}{\overset{x}{\int }}\left[T^0+\epsilon \delta T+\frac{\left({\beta }^0+\epsilon \delta \beta \right)}{2}{\left(T^0+\epsilon \delta T\right)}^2\right]dy}\right\}}_{\epsilon =0}=0; \hspace{0.17em}\end{array}$$

(83)

$${\left\{{\displaystyle \frac{d}{d\epsilon }}{\left[T^0\left(x\right)+\epsilon \delta T\left(x\right)\right]}_{x=0}\right\}}_{\epsilon =0}=\delta T_0.$$

(84)

Carrying out the operations indicated in Equations (83) and (84) yields the following NIDE-V system representing the 1st-LVSS for the variational function $\delta T\left(x\right)$:

$$\begin{array}{l}{L}_1\left(T;\theta \right)\delta T\left(x\right)\triangleq \beta \left[\delta T\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}+T\left(x\right){\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right]+b^2{\displaystyle \underset{0}{\overset{x}{\int }}\left[1+\beta T\left(y\right)\right]}\hspace{0.17em}\delta T\left(y\right)dy\\ =-\left(\delta \beta \right)T\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}-\left(\delta \beta \right){\displaystyle \frac{b^2}{2}}{\displaystyle \underset{0}{\overset{x}{\int }}{T}^2\left(y\right)dy}-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{x}{\int }}\left[T\left(y\right)+\frac{\beta }{2}{T}^2\left(y\right)\right]dy}\hspace{0.17em}; \end{array}$$

(85)

$$\delta T\left(0\right)=\delta T_0\hspace{0.17em}.$$

(86)

The above 1st-LVSS for the variational function $\delta T\left(x\right)$ is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. It is important to note that the operator $L_1\left(T;\theta \right)\delta T\left(x\right)$ acts linearly on $\delta T\left(x\right)$. The 1st-LVSS would need to be solved repeatedly to obtain the variational function $\delta T\left(x\right)$ for every parameter variation of interest. Repeatedly solving the 1st-LVSS can be avoided by expressing the indirect effect term defined in Equation (82) in terms of a “1st-level adjoint sensitivity function” that will be the solution of a 1st-LASS to be constructed next by following the general procedure outlined in Section 2. The inner product appropriate for constructing the 1st-LASS corresponding to the 1st-LVSS defined by Equations (85) and (86) for the single-component variational function $\delta T\left(x\right)$ takes on the following particular form of Equation (22):

$${〈{\chi }^{(1)}\left(x\right),{\eta }^{(1)}\left(x\right)〉}_1\triangleq {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{(1)}\left(x\right){\eta }^{(1)}\left(x\right)dx}.$$

(87)

The construction of the 1st-LASS will be accomplished by implementing the following sequence of steps:

• Use Equation (87) to construct the inner product of Equation (85) with a yet specified function ${\eta }^{\left(1\right)}\left(x\right)$ to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)}{L}_1\left(T;\theta \right)\delta T\left(x\right)={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)\left[\beta \delta T\left(x\right)\frac{dT\left(x\right)}{dx}+\beta T\left(x\right)\frac{d}{dx}\delta T\left(x\right)\right]}\hspace{0.17em}dx\\ +b^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)}{\displaystyle \underset{0}{\overset{x}{\int }}\left[1+\beta T\left(y\right)\right]}\hspace{0.17em}\delta T\left(y\right)dy\hspace{0.17em}dx=q^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right),\end{array}$$

  (88)

  where:

  $$\begin{array}{l}{q}^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right)\triangleq -\left(\delta \beta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\frac{dT\left(x\right)}{dx}dx}\\ -\left(\delta \beta \right){\displaystyle \frac{b^2}{2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx}{\displaystyle \underset{0}{\overset{x}{\int }}{T}^2\left(y\right)dy}-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right){\displaystyle \underset{0}{\overset{x}{\int }}\left[T\left(y\right)+\frac{\beta }{2}{T}^2\left(y\right)\right]dy}dx}.\end{array}$$

  (89)
• Transfer the operations on $\delta T\left(x\right)$ to operations (differentiation and integration) on ${\eta }^{\left(1\right)}\left(x\right)$ in Equation (88) by constructing the formal adjoint operator $L_1^{*}\left(T;\theta \right)$ satisfying the relation

  $${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)}{L}_1\left(T;\theta \right)\delta T\left(x\right)dx={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}{L}_1^{*}\left(T;\theta \right){\eta }^{\left(1\right)}\left(x\right)dx+P_1\left(T;\theta ;\delta T;{\eta }^{\left(1\right)}\right),$$

  (90)

  where $P_1\left(T;\theta ;\delta T;{\eta }^{\left(1\right)}\right)$ denotes the bilinear concomitant comprising boundary terms. The relationship shown in Equation (90) is derived by integrating by parts the second and third terms in the middle expression of Equation (88) to obtain the following relations:

  $$\begin{array}{l}\beta {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\left[\frac{d}{dx}\delta T\left(x\right)\right]}\hspace{0.17em}dx=\beta {\eta }^{\left(1\right)}\left(\mathcal{l}\right)T\left(\mathcal{l}\right)\delta T\left(\mathcal{l}\right)-\\ -\beta {\eta }^{\left(1\right)}\left(0\right)T\left(0\right)\delta T\left(0\right)-\beta {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\frac{d}{dx}\left[{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\right]}\hspace{0.17em}dx\hspace{0.17em}.\end{array}$$

  (91)

  $$\begin{array}{l}{b}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx}{\displaystyle \underset{0}{\overset{x}{\int }}\left[1+\beta T\left(y\right)\right]}\hspace{0.17em}\delta T\left(y\right)dy\hspace{0.17em}=b^2{\left\{\left[{\displaystyle \underset{0}{\overset{x}{\int }}{\eta }^{\left(1\right)}\left(y\right)dy}\right]\times {\displaystyle \underset{0}{\overset{x}{\int }}\left[1+\beta T\left(y\right)\right]}\hspace{0.17em}\delta T\left(y\right)dy\right\}}_{x=0}^{x=\mathcal{l}}\hspace{0.17em}\\ -b^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\beta T\left(x\right)\right]\delta T\left(x\right)dx}{\displaystyle \underset{0}{\overset{x}{\int }}{\eta }^{\left(1\right)}\left(y\right)dy=}\hspace{0.17em}{b}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\left[1+\beta T\left(x\right)\right]}\hspace{0.17em}dx{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(y\right)dy}.\end{array}$$

  (92)
• Insert the results obtained in Equations (91) and (92) into the left-side Equation (88) and collect the terms multiplying the function $\delta T\left(x\right)$ to obtain the relation below:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}{L}_1^{*}\left(T;\theta \right){\eta }^{\left(1\right)}\left(x\right)dx={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\left\{\beta {\eta }^{\left(1\right)}\left(x\right)\frac{dT\left(x\right)}{dx}\right.}\hspace{0.17em}\\ \left.-\beta {\displaystyle \frac{d}{dx}}\left[{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\right]+b^2\left[1+\beta T\left(x\right)\right]{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(y\right)dy}\right\}\hspace{0.17em}dx\\ =q^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right)-\beta {\eta }^{\left(1\right)}\left(\mathcal{l}\right)T\left(\mathcal{l}\right)\delta T\left(\mathcal{l}\right)+\beta {\eta }^{\left(1\right)}\left(0\right)T\left(0\right)\delta T\left(0\right),\end{array}$$

  (93)
• Require the term on the left-side of Equation (93) to represent the indirect-effect term defined in Equation (82), by requiring that the function ${\eta }^{\left(1\right)}\left(x\right)$ satisfy the following NIDE-V net:

  $$\begin{array}{l}{L}_1^{*}\left(T;\theta \right){\eta }^{\left(1\right)}\left(x\right)=\beta {\eta }^{\left(1\right)}\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}-\beta {\displaystyle \frac{d}{dx}}\left[{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\right]\\ +b^2\left[1+\beta T\left(x\right)\right]{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(y\right)dy}=k_0\beta T\left(\mathcal{l}\right)\delta \left(x-\mathcal{l}\right).\end{array}$$

  (94)
• Eliminate the unknown boundary term in Equation (93) by imposing the following boundary condition of the function ${\eta }^{\left(1\right)}\left(x\right)$:

  $${\eta }^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

  (95)

  The NIDE-V system comprising Equations (94) and (95) constitutes the 1st-Level Adjoint Sensitivity System (1st-LASS) for the 1st-level adjoint sensitivity function ${\eta }^{\left(1\right)}\left(x\right)$.
• Use Equations (86), (94), and (95) in Equation (93) to obtain the following expression for the indirect-effect term:

  $$\delta R{\left(T^0,{\theta }^0;\delta T\right)}_{ind}=q^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right)+\left(\delta T_0\right)\beta {\eta }^{\left(1\right)}\left(0\right)T_0\hspace{0.17em}.$$

  (96)
• Adding the expression obtained in Equation (96) with the expression for the direct-effect term defined in Equation (81) and using the definition of the quantity $q^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right)$ provided in Equation (89) yields the following expression for the G-differential $\delta R\left(T^0,{\theta }^0;\delta T,\delta \theta \right)$:

  $$\begin{array}{l}\delta R\left(T^0,{\theta }^0;\delta T,\delta \theta \right)=\left(\delta k_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\beta T\left(x\right)\right]}{\displaystyle \frac{dT\left(x\right)}{dx}}+\left(\delta \beta \right)k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\frac{dT\left(x\right)}{dx}dx}\\ -\left(\delta T_0\right)T_0{k}_0\beta +\left(\delta T_0\right)\beta {\eta }^{\left(1\right)}\left(0\right)T_0\hspace{0.17em}-\left(\delta \beta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\frac{dT\left(x\right)}{dx}dx}\\ -\left(\delta \beta \right){\displaystyle \frac{b^2}{2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx}{\displaystyle \underset{0}{\overset{x}{\int }}{T}^2\left(y\right)dy}-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx{\displaystyle \underset{0}{\overset{x}{\int }}\left[T\left(y\right)+\frac{\beta }{2}{T}^2\left(y\right)\right]dy}}.\end{array}$$

  (97)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• From the definition of the “feature-function” $b^2\left(\theta \right)\triangleq Q /k_0$ provided in Equation (68), it follows that:

  $$\delta b\left(\theta \right)={\displaystyle \frac{\delta Q}{2\sqrt{k_{0}Q}}}-{\displaystyle \frac{\delta k_0}{2k_0}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\hspace{0.17em}.$$

  (98)
• Inserting the expression provided in Equation (98) into Equation (97) and collecting the terms that multiply the respective parameter variations yields the following expressions of the first-order sensitivities of the decoder response with respect to the model parameters:

  $${\displaystyle \frac{\partial R}{\partial T_0}}=-T_0{k}_0\beta +\beta {\eta }^{\left(1\right)}\left(0\right)T_0;$$

  (99)

  $${\displaystyle \frac{\partial R}{\partial \beta }}=k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\frac{dT\left(x\right)}{dx}dx}-{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)T\left(x\right)\frac{dT\left(x\right)}{dx}dx}-{\displaystyle \frac{Q}{2k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx}{\displaystyle \underset{0}{\overset{x}{\int }}{T}^2\left(y\right)dy}\hspace{0.17em};$$

  (100)

  $${\displaystyle \frac{\partial R}{\partial k_0}}=-{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx{\displaystyle \underset{0}{\overset{x}{\int }}\left[T\left(y\right)+\frac{\beta }{2}{T}^2\left(y\right)\right]dy}}$$

  (101)

  $${\displaystyle \frac{\partial R}{\partial Q}}=-{\displaystyle \frac{1}{\sqrt{k_0}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\eta }^{\left(1\right)}\left(x\right)dx{\displaystyle \underset{0}{\overset{x}{\int }}\left[T\left(y\right)+\frac{\beta }{2}{T}^2\left(y\right)\right]dy}}.$$

  (102)

The expressions presented in Equations (99)–(102) can be evaluated after having obtained the 1st-level adjoint sensitivity function ${\eta }^{\left(1\right)}\left(x\right)$ by solving, a single time, the 1st-LASS represented by the NIDE-V system comprising Equations (94) and (95). These computations will not be performed explicitly at this stage because the closed-form expressions of the sensitivities of the decoder response with respect to the model parameters will be obtained by much simpler algebraic manipulations, in the NIDE-V Kirchhoff-transformed framework, as will be presented in Section 3.1.2, below.

3.1.2. Representation in the NIDE-V Kirchhoff-Transformed Framework

In the Kirchhoff-transformed framework, the 1st-order sensitivities of the decoder response defined in Equation (76) are obtained by determining the first-order G-differential, $\delta R\left(U^0,{\theta }^0;\delta U,\delta \theta \right)$, of the respective expression, for variations $\left(\delta U,\delta \theta \right)$ around the nominal values ${\theta }^0\triangleq {\left[T_0^0,k_0^0,{\beta }^0,Q^0\right]}^{†}$, which is by definition obtained as follows:

$$\begin{array}{l}\delta R\left(U^0,{\theta }^0;\delta U,\delta \theta \right)\triangleq \left\{{\displaystyle \frac{d}{d\epsilon }}\left(k_0^0+\epsilon \delta k_0\right)\right.{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[U^0\left(x\right)+\epsilon \delta U\left(x\right)\right]\delta \left(x-\mathcal{l}\right)dx}\\ {\left.-{\displaystyle \frac{d}{d\epsilon }}\left(k_0^0+\epsilon \delta k_0\right)\left(U_0^0+\epsilon \delta U_0\right)\right\}}_{\epsilon =0}=\delta R{\left(U^0,{\theta }^0;\delta \theta \right)}_{dir}+\delta R{\left(U^0,{\theta }^0;\delta U\right)}_{ind},\end{array}$$

(103)

where the direct-effect term $\delta R{\left(U^0,{\theta }^0;\delta \theta \right)}_{dir}$ and indirect-effect term $\delta R{\left(U^0,{\theta }^0;\delta U\right)}_{ind}$ are defined below:

$$\delta R{\left(U^0,{\theta }^0;\delta \theta \right)}_{dir}\triangleq {\left\{\left(\delta k_0\right)U\left(\mathcal{l}\right)-k_0\left(\delta U_0\right)-U_0\left(\delta k_0\right)\right\}}_{{\theta }^0},$$

(104)

$$\delta R{\left(U^0,{\theta }^0;\delta U\right)}_{ind}\triangleq {\left\{k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\delta \left(x-\mathcal{l}\right)dx}\right\}}_{{\theta }^0}.$$

(105)

The direct-effect term can be computed at this time, but the indirect-effect term can only be computed after having determined the variational function $\delta U\left(x\right)$. This variational function is the solution of the 1st-LVSS obtained by G-differentiating the NIDE-V form of the heat conduction model, namely Equations (74) and (75). Applying the definition of the G-differential to these equations yields the following NIDE-V form for the 1st-LVSS to be satisfied by the variational function $\delta U\left(x\right)$:

$${\displaystyle \frac{d}{dx}}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}=-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy};\hspace{0.5em}0<x<\mathcal{l}\hspace{0.17em};$$

(106)

$$\delta U\left(0\right)=\delta U_0\left(\theta \right)\hspace{0.17em}=\left(1+\beta T_0\right)\left(\delta T_0\right)+{\displaystyle \frac{T_0^2}{2}}\left(\delta \beta \right)$$

(107)

The above 1st-LVSS for the variational function $\delta U\left(x\right)$ is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. The 1st-LVSS would need to be solved repeatedly to obtain the variational function $\delta U\left(x\right)$ for every parameter variation of interest. Repeatedly solving the 1st-LVSS can be avoided by expressing the indirect effect term defined in Equation (105) in terms of an adjoint function to be constructed by following the general procedure outlined in Section 2, as follows:

• Use Equation (87) to construct the inner product of Equation (106) with a yet specified function ${\phi }^{\left(1\right)}\left(x\right)$ to obtain the following relation:

  $${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]}\hspace{0.17em}dx=-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}.$$

  (108)
• Integrate by parts the terms on the left-side of Equation (108) to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]}\hspace{0.17em}dx={\phi }^{\left(1\right)}\left(\mathcal{l}\right)\delta U\left(\mathcal{l}\right)-{\phi }^{\left(1\right)}\left(0\right)\delta U\left(0\right)\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\left[-\frac{d{\phi }^{\left(1\right)}\left(x\right)}{dx}+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\end{array}.$$

  (109)
• Require the last term on the right-side of Equation (109) to represent the indirect-effect term defined in Equation (105), by requiring the function ${\phi }^{\left(1\right)}\left(x\right)$ to satisfy the following relation:

  $$-{\displaystyle \frac{d{\phi }^{\left(1\right)}\left(x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}=k_0\delta \left(x-\mathcal{l}\right).$$

  (110)
• In Equation (109), use the boundary condition provided in Equation (107) and eliminate the unknown remaining boundary term by imposing the following boundary condition of the function ${\phi }^{\left(1\right)}\left(x\right)$:

  $${\phi }^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

  (111)

  The NIDE-V net comprising Equations (110) and (111) constitutes the 1st-Level Adjoint Sensitivity System (1st-LASS) for the 1st-level adjoint function ${\phi }^{\left(1\right)}\left(x\right)$. Note that this 1st-LASS is linear in ${\phi }^{\left(1\right)}\left(x\right)$ and is independent of parameter variations, so it needs to be solved only once (as opposed to many times, like in the case of the 1st-LVSS) to obtain the function ${\phi }^{\left(1\right)}\left(x\right)$.
• Use Equations (107), (110), and (111) in Equation (105) to obtain the following expression for the indirect-effect term:

  $$\delta R{\left(U^0,{\theta }^0;\delta U\right)}_{ind}=\left(\delta U_0\right){\phi }^{\left(1\right)}\left(0\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}\hspace{0.17em}.$$

  (112)
• Adding the expression obtained in Equation (112) with the expression for the direct-effect term defined in Equation (104) yields the following expression for the G-differential $\delta R\left(U^0,{\theta }^0;\delta U,\delta \theta \right)$:

  $$\begin{array}{l}\delta R\left(U^0,{\theta }^0;\delta U,\delta \theta \right)=\left(\delta k_0\right)U\left(\mathcal{l}\right)-k_0\left(\delta U_0\right)-U_0\left(\delta k_0\right)\\ \hspace{0.5em}+\left(\delta U_0\right){\phi }^{\left(1\right)}\left(0\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (113)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• Inserting into Equation (113) the result obtained in Equation (98) and identifying in Equation (113) the expressions that multiply the respective parameter variations yields the following expressions for the first-order sensitivities of the decoder response:

  $${\displaystyle \frac{\partial R}{\partial U_0}}={\phi }^{\left(1\right)}\left(0\right)-k_0;$$

  (114)

  $${\displaystyle \frac{\partial R}{\partial Q}}=-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}\hspace{0.17em};$$

  (115)

  $${\displaystyle \frac{\partial R}{\partial k_0}}={\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}+\left[U\left(\mathcal{l}\right)-U_0\right].$$

  (116)

The sensitivities of the decoder response with respect to the model parameters $T_0$ and $\beta$ are obtained by using the “chain rule” of differentiation of the expression provided in Equation (114) in conjunction with the definition of the feature function $U_0\left(\theta \right)$ provided in Equation (73) to obtain the following expressions:

$${\displaystyle \frac{\partial R}{\partial T_0}}={\displaystyle \frac{\partial R}{\partial U_0}}{\displaystyle \frac{\partial U_0}{\partial T_0}}=\left[{\phi }^{\left(1\right)}\left(0\right)-k_0\right]\left(1+\beta T_0\right);$$

(117)

$${\displaystyle \frac{\partial R}{\partial \beta }}={\displaystyle \frac{\partial R}{\partial U_0}}{\displaystyle \frac{\partial U_0}{\partial \beta }}=\left[{\phi }^{\left(1\right)}\left(0\right)-k_0\right]{\displaystyle \frac{T_0^2}{2}};$$

(118)

The sensitivities obtained in Equations (114)–(118) can be computed after having determined the 1st-level adjoint sensitivity function ${\phi }^{\left(1\right)}\left(x\right)$. Solving the 1st-LASS comprising Equations (110) and (111) yields the following expression for the 1st-level adjoint sensitivity function ${\phi }^{\left(1\right)}\left(x\right)$:

$${\phi }^{\left(1\right)}\left(x\right)=k_0\left[1-H\left(x-\mathcal{l}\right)\right]\cos \left[\left(\mathcal{l}-x\right)b\left(\theta \right)\right].$$

(119)

Using in Equations (114)–(118) the expression for ${\phi }^{\left(1\right)}\left(x\right)$ obtained in Equation (119) and the expression for $U\left(x\right)$ obtained in Equation (77) yields the following explicit expressions for the first-order sensitivities of the decoder response:

$${\displaystyle \frac{\partial R}{\partial U_0}}=k_0\left[\cos \mathcal{l}b\left(\theta \right)-1\right]\hspace{0.17em};$$

(120)

$${\displaystyle \frac{\partial R}{\partial Q}}=-{\displaystyle \frac{U_0\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right);$$

(121)

$${\displaystyle \frac{\partial R}{\partial k_0}}={\displaystyle \frac{U_0\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)+U_0\left[\cos b\left(\theta \right)\mathcal{l}-1\right].$$

(122)

$${\displaystyle \frac{\partial R}{\partial T_0}}={\displaystyle \frac{\partial R}{\partial U_0}}{\displaystyle \frac{\partial U_0}{\partial T_0}}=k_0\left[\cos \mathcal{l}b\left(\theta \right)-1\right]\left(1+\beta T_0\right);$$

(123)

$${\displaystyle \frac{\partial R}{\partial \beta }}={\displaystyle \frac{\partial R}{\partial U_0}}{\displaystyle \frac{\partial U_0}{\partial \beta }}=k_0\left[\cos \mathcal{l}b\left(\theta \right)-1\right]{\displaystyle \frac{T_0^2}{2}}\hspace{0.17em};$$

(124)

Comparing the derivations presented in this subsection (which yielded the explicit exact expressions shown in Equations (120)–(124) for all of the first-order sensitivities of the decoder response with respect to the model parameters) with the derivations presented in Section 3.1.2 indicates that the derivations/computations in the Kirchoff-transformed framework (in which the heat conduction equation and the decoder response become linear functions of the dependent variable $U\left(x\right)$) are considerably simpler than in the original temperature-framework, in which both the heat conduction equation and the decoder response are nonlinear functions of the dependent variable $T\left(x\right)$.

3.2. Application of the 1st-FASAM-NODE to Compute 1st-Order Sensitivities of the NODE Representation of the Illustrative Heat Conduction Model

The 1st-FASAM-NODE methodology [30] will be applied, in this subsection, to the paradigm heat conduction model in order to illustrate the path to obtaining the exact expressions of the first-order sensitivities of the decoder response with respect to the model parameters and to subsequently compute these sensitivities most efficiently, requiring a single large-scale computation for solving the corresponding 1st-Level Adjoint Sensitivity System (1st-LASS). The illustrative mathematical derivations will be performed in the temperature-framework (in Section 3.2.1) and also in the Kirchhoff-transformed framework (in Section 3.2.2) in order to illustrate the advantages of using linearizing transformations, such as Kirchhoff’s transformation, whenever possible. Finally, the application of the 1st-FASAM-NODE methodology will be compared to the application of the 1st-FASAM-NIDE-V methodology, which was presented in Section 3.1

3.2.1. Representation in the NODE Temperature- Framework

In the NODE temperature- framework, the first-order G-differential of the response is as defined in Equation (80) but the variational function $\delta T\left(x\right)$ is the solution of the 1st-LVSS obtained by G-differentiating the NODE-form provided in Equations (66) and (67). Applying the definition of the G-differential to these equations yields the following 1st-LVSS for the variational function $\delta T\left(x\right)$:

$$\begin{array}{l}{L}_2\left(T;\theta \right)\delta T\left(x\right)\triangleq {\displaystyle \frac{d}{dx}}\left\{\beta {\displaystyle \frac{dT\left(x\right)}{dx}}\delta T\left(x\right)+\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}\\ +Q\left[1+\beta T\left(x\right)\right]\delta T\left(x\right)=-\left(\delta k_0\right){\displaystyle \frac{d}{dx}}\left[\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{dT\left(x\right)}{dx}}\right]\\ -\left(\delta \beta \right)k_0{\displaystyle \frac{d}{dx}}\left[T\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}\right]-\left(\delta Q\right)\left[T\left(x\right)+{\displaystyle \frac{\beta }{2}}{T}^2\left(x\right)\right]-\left(\delta \beta \right){\displaystyle \frac{Q}{2}}{T}^2\left(x\right);\end{array}$$

(125)

$$\delta T\left(0\right)=\delta T_0\hspace{0.17em};\hspace{1em}{\left[{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right]}_{x=0}=0\hspace{0.17em}.$$

(126)

The above 1st-LVSS for the variational function $\delta T\left(x\right)$ is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. It is important to note that the operator $L_2\left(T;\theta \right)\delta T\left(x\right)$ acts linearly on $\delta T\left(x\right)$. The 1st-LVSS would need to be solved repeatedly to obtain the variational function $\delta T\left(x\right)$ for every parameter variation of interest. Repeatedly solving the 1st-LVSS can be avoided by expressing the indirect effect term defined in Equation (82) in terms of a “1st-level adjoint sensitivity function” that will be the solution of a 1st-LASS to be constructed next by following the general procedure outlined in Section 2. The construction of this 1st-LASS will be accomplished by implementing the following sequence of steps:

• Use Equation (87) to construct the inner product of Equation (125) with a yet specified function ${\chi }^{\left(1\right)}\left(x\right)$ to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{\left(1\right)}\left(x\right)}{L}_2\left(T;\theta \right)\delta T\left(x\right)={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{\left(1\right)}\left(x\right)\frac{d}{dx}\left\{\beta \frac{dT\left(x\right)}{dx}\delta T\left(x\right)+\left[1+\beta T\left(x\right)\right]\frac{d}{dx}\delta T\left(x\right)\right\}}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{\left(1\right)}\left(x\right)\left[Q\left[1+\beta T\left(x\right)\right]\delta T\left(x\right)\right]}\hspace{0.17em}dx=p^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right),\end{array}$$

  (127)

  where:

  $$\begin{array}{l}{p}^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right)\triangleq -\left(\delta k_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{d}{dx}}\left[\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{dT\left(x\right)}{dx}}\right]\\ -\left(\delta \beta \right)k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{d}{dx}}\left[T\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}\right]\\ -\left(\delta Q\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}\left[T\left(x\right)+{\displaystyle \frac{\beta }{2}}{T}^2\left(x\right)\right]-\left(\delta \beta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{Q}{2}}{T}^2\left(x\right)\hspace{0.17em}.\end{array}$$

  (128)
• Transfer the differentiations on $\delta T\left(x\right)$ in Equation (127) to differentiations on ${\chi }^{\left(1\right)}\left(x\right)$ by constructing the formal adjoint operator $L_2^{*}\left(T;\theta \right)$ satisfying the relation

  $${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{\left(1\right)}\left(x\right)}{L}_2\left(T;\theta \right)\delta T\left(x\right)dx={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}{L}_2^{*}\left(T;\theta \right){\chi }^{\left(1\right)}\left(x\right)dx+P_2\left(T;\theta ;\delta T;{\chi }^{\left(1\right)}\right),$$

  (129)

  where $P_2\left(T;\theta ;\delta T;{\chi }^{\left(1\right)}\right)$ denotes the bilinear concomitant comprising boundary terms. The relationship shown in Equation (129) is derived by integrating by parts the first and the second terms on the left-side of Equation (127) so as to transfer the differentiations on $\delta T\left(x\right)$ to differentiations on ${\chi }^{\left(1\right)}\left(x\right)$, to obtain the following relations:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{\left(1\right)}\left(x\right)\frac{d}{dx}\left[\beta \frac{dT\left(x\right)}{dx}\delta T\left(x\right)\right]}\hspace{0.17em}dx={\left\{{\chi }^{\left(1\right)}\left(x\right)\beta {\displaystyle \frac{dT\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}\\ -{\left\{{\chi }^{\left(1\right)}\left(x\right)\beta {\displaystyle \frac{dT\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=0}-\beta {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\frac{dT\left(x\right)}{dx}\frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\hspace{0.17em}dx\hspace{0.17em};\end{array}$$

  (130)

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }^{\left(1\right)}\left(x\right)\frac{d}{dx}\left\{\left[1+\beta T\left(x\right)\right]\frac{d}{dx}\delta T\left(x\right)\right\}}\hspace{0.17em}dx={\left\{{\chi }^{\left(1\right)}\left(x\right)\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}\\ -{\left\{{\chi }^{\left(1\right)}\left(x\right)\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}-{\left\{\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}\\ +{\left\{\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=0}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\frac{d}{dx}\left\{\left[1+\beta T\left(x\right)\right]\frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}\right\}}\hspace{0.17em}dx.\end{array}$$

  (131)
• Insert the results obtained in Equations (130) and (131) into the left-side Equation (127) and collect the terms that multiply the function $\delta T\left(x\right)$ to obtain the relation below:

  $${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)L_2^{*}\left(T;\theta \right){\chi }^{\left(1\right)}}\hspace{0.17em}dx=p^{\left(1\right)}\left(T;{\chi }^{\left(1\right)};\theta ;\delta \theta \right)-P_2\left(T;{\chi }^{\left(1\right)};\theta ;\delta \theta \right),$$

  (132)

  where the operator $L_2^{*}\left(T;\theta \right)$ has the expression shown below:

  $$L_2^{*}\left(T;\theta \right){\chi }^{\left(1\right)}\triangleq -\left[\beta {\displaystyle \frac{dT\left(x\right)}{dx}}\right]{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}+{\displaystyle \frac{d}{dx}}\left\{\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\right\}.$$

  (133)

  and where the bilinear concomitant denoted as $P_2\left(T;\theta ;\delta T;{\chi }^{\left(1\right)}\right)$ in Equation (132) is defined as follows:

  $$\begin{array}{l}{P}_2\left(T;\theta ;\delta T;{\chi }^{\left(1\right)}\right)\triangleq {\left\{{\chi }^{\left(1\right)}\left(x\right)\beta {\displaystyle \frac{dT\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}-{\left\{{\chi }^{\left(1\right)}\left(x\right)\beta {\displaystyle \frac{dT\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=0}\\ +{\left\{{\chi }^{\left(1\right)}\left(x\right)\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}-{\left\{{\chi }^{\left(1\right)}\left(x\right)\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}\\ -{\left\{\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}+{\left\{\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\delta T\left(x\right)\right\}}_{x=0}.\end{array}$$

  (134)
• Require the term on the left-side of Equation (132) to represent the indirect-effect term defined in Equation (82), by requiring the function ${\chi }^{\left(1\right)}\left(x\right)$ to satisfy the following NIDE-V system:

  $$L_2^{*}\left(T;\theta \right){\chi }^{\left(1\right)}=k_0\beta T\left(\mathcal{l}\right)\delta \left(x-\mathcal{l}\right).$$

  (135)
• Eliminate the unknown boundary terms in Equation (132) by imposing the following boundary condition of the function ${\chi }^{\left(1\right)}\left(x\right)$:

  $${\chi }^{\left(1\right)}\left(\mathcal{l}\right)=0;\hspace{0.5em}{\left[{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=\mathcal{l}}=0\hspace{0.17em}.$$

  (136)

  The NIDE-V system comprising Equations (135) and (136) constitutes the 1st-Level Adjoint Sensitivity System (1st-LASS) for the 1st-level adjoint sensitivity function ${\chi }^{\left(1\right)}\left(x\right)$.
• Use Equations (67), (126), and (134)–(136) in Equation (132) to obtain the following expression for the indirect-effect term:

  $$\delta R{\left(T^0,{\theta }^0;\delta T\right)}_{ind}=p^{\left(1\right)}\left(T;{\eta }^{\left(1\right)};\theta ;\delta \theta \right)-\left(\delta T_0\right)\left(1+\beta T_0\right){\left\{{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0}\hspace{0.17em}.$$

  (137)
• Adding the expression obtained in Equation (137) with the expression for the direct-effect term defined in Equation (81) and using the definition of the quantity provided in Equation (128) yields the following expression for the G-differential $\delta R\left(T^0,{\theta }^0;\delta T,\delta \theta \right)$:

  $$\begin{array}{l}\delta R\left(T^0,{\theta }^0;\delta T,\delta \theta \right)=\left(\delta k_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\beta T\left(x\right)\right]}{\displaystyle \frac{dT\left(x\right)}{dx}}+\left(\delta \beta \right)k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\frac{dT\left(x\right)}{dx}dx}\\ -\left(\delta T_0\right)T_0{k}_0\beta -\left(\delta k_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{d}{dx}}\left[\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{dT\left(x\right)}{dx}}\right]\\ -\left(\delta \beta \right)k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{d}{dx}}\left[T\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}\right]-\left(\delta Q\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}\left[T\left(x\right)+{\displaystyle \frac{\beta }{2}}{T}^2\left(x\right)\right]\\ -\left(\delta \beta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{Q}{2}}{T}^2\left(x\right)-\left(\delta T_0\right)\left(1+\beta T_0\right){\left\{{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0}\hspace{0.17em}.\end{array}$$

  (138)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• Inserting the expression provided in Equation (98) into Equation (138) and collecting the terms that multiply the respective parameter variations yields the following expressions for the first-order sensitivities of the decoder response with respect to the model parameters:

  $${\displaystyle \frac{\partial R}{\partial T_0}}=-T_0{k}_0\beta -\left(1+\beta T_0\right){\left\{{\displaystyle \frac{d{\chi }^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0};$$

  (139)

  $${\displaystyle \frac{\partial R}{\partial \beta }}=k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)\frac{dT\left(x\right)}{dx}dx}-k_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{d}{dx}}\left[T\left(x\right){\displaystyle \frac{dT\left(x\right)}{dx}}\right]-{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{Q}{2}}{T}^2\left(x\right);$$

  (140)

  $${\displaystyle \frac{\partial R}{\partial k_0}}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[1+\beta T\left(x\right)\right]}{\displaystyle \frac{dT\left(x\right)}{dx}}-{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}{\displaystyle \frac{d}{dx}}\left[\left[1+\beta T\left(x\right)\right]{\displaystyle \frac{dT\left(x\right)}{dx}}\right]\hspace{0.17em};$$

  (141)

  $${\displaystyle \frac{\partial R}{\partial Q}}=-{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\chi }^{\left(1\right)}\left(x\right)}\left[T\left(x\right)+{\displaystyle \frac{\beta }{2}}{T}^2\left(x\right)\right].$$

  (142)

The expressions presented in Equations (139)–(142) are to be evaluated at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. These expressions can be evaluated after having obtained the 1st-level adjoint sensitivity function ${\chi }^{\left(1\right)}\left(x\right)$ by solving, a single time, the 1st-LASS represented by the NIDE-V net comprising Equations (135) and (136). These computations will not be performed explicitly at this stage because the closed-form expressions of the sensitivities of the decoder response with respect to the model parameters will be obtained by much simpler algebraic manipulations in the NODE Kirchhoff-transformed framework, as will be presented in Section 3.2.2, below.

3.2.2. Representation in the NODE Kirchhoff-Transformed Framework

In the NODE Kirchhoff-transformed framework, the first-order G-differential of the response is as defined in Equations (103) but the variational function $\delta U\left(x\right)$ is the solution of the 1st-LVSS obtained by G-differentiating the NODE-system provided in Equations (72) and (73). Applying the definition of the G-differential to these equations yields the following 1st-LVSS for the variational function $\delta U\left(x\right)$:

$${\displaystyle \frac{d^2}{d{x}^2}}\delta U\left(x\right)+b^2\left(\theta \right)\delta U\left(x\right)=-2\left(\delta b\right)b\left(\theta \right)U\left(x\right);\hspace{0.5em}0<x<\mathcal{l}\hspace{0.17em};$$

(143)

$$\delta U\left(0\right)=\delta U_0;\hspace{0.5em}{\left[{\displaystyle \frac{d}{dx}}\delta U\left(x\right)\right]}_{x=0}=0\hspace{0.17em}.$$

(144)

The above 1st-LVSS for the variational function $\delta U\left(x\right)$ is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. Since the variational function $\delta U\left(x\right)$ depends on parameter variations, its repeated computation by solving the 1st-LVSS for every parameter variation of interest can be avoided by expressing the indirect effect term defined in Equation (105) in terms of an adjoint function to be constructed by following the general procedure outlined in Section 2, as follows:

• Use Equation (87) to construct the inner product of Equation (143) with a yet specified function ${\psi }^{\left(1\right)}\left(x\right)$ to obtain the following relation:

  $${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)\left[\frac{d^2}{d{x}^2}\delta U\left(x\right)+b^2\left(\theta \right)\delta U\left(x\right)\right]dx}=-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)U\left(x\right)dx}.$$

  (145)
• Integrate by parts twice the first term on the left-side of Equation (145) to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)\left[\frac{d^2}{d{x}^2}\delta U\left(x\right)+b^2\delta U\left(x\right)\right]dx}={\psi }^{\left(1\right)}\left(\mathcal{l}\right){\left[{\displaystyle \frac{d}{dx}}\delta U\left(x\right)\right]}_{x=\mathcal{l}}-{\psi }^{\left(1\right)}\left(0\right){\left[{\displaystyle \frac{d}{dx}}\delta U\left(x\right)\right]}_{x=0}\\ -\delta U\left(\mathcal{l}\right){\left[{\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=\mathcal{l}}+\delta U\left(0\right){\left[{\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=0}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\left[\frac{d^2{\psi }^{\left(1\right)}\left(x\right)}{d{x}^2}+b^2{\psi }^{\left(1\right)}\left(x\right)\right]dx}\hspace{0.17em}.\end{array}$$

  (146)
• Require the last term on the right-side of Equation (146) to represent the indirect-effect term defined in Equation (105) by requiring that the function ${\psi }^{\left(1\right)}\left(x\right)$ satisfy the following relation:

  $${\displaystyle \frac{d^2{\psi }^{\left(1\right)}\left(x\right)}{d{x}^2}}+b^2\left(\theta \right){\psi }^{\left(1\right)}\left(x\right)=k_0\delta \left(x-\mathcal{l}\right).$$

  (147)
• Use in Equation (146) the boundary conditions provided in Equation (144) and eliminate the unknown remaining boundary terms by imposing the following boundary conditions of the function ${\psi }^{\left(1\right)}\left(x\right)$:

  $${\psi }^{\left(1\right)}\left(\mathcal{l}\right)=0;\hspace{1em}{\left[{\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=\mathcal{l}}=0.$$

  (148)

  The NODE-type system comprising Equations (147) and (148) constitutes the 1st-Level Adjoint Sensitivity System (1st-LASS) for the 1st-level adjoint sensitivity function ${\psi }^{\left(1\right)}\left(x\right)$.
• Use Equations (144), (145), (147), and (148) in Equation (146) to obtain the following expression for the indirect-effect term:

  $$\delta R{\left(T^0,{\theta }^0;\delta T\right)}_{ind}=-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)U\left(x\right)dx}-\delta U_0{\left[{\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=0}.$$

  (149)
• Add the expression obtained in Equation (149) with the expression for the direct-effect term defined in Equation (104) to obtain the following expression for the G-differential:

  $$\begin{array}{l}\delta R\left(T^0,{\theta }^0;\delta T,\delta \theta \right)=-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)U\left(x\right)dx}-\delta U_0{\left[{\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=0}\\ +{\left\{\left(\delta k_0\right)\left[U\left(\mathcal{l}\right)-U\left(0\right)\right]-k_0\delta U_0\right\}}_{{\theta }^0}.\end{array}$$

  (150)
• Inserting into Equation (150) the result obtained in Equation (98) and identifying in Equation (150) the expressions that multiply the respective parameter variations yields the following expressions for the first-order sensitivities of the decoder response:

  $${\displaystyle \frac{\partial R}{\partial U_0}}=-k_0-{\left[{\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}\right]}_{x=0};$$

  (151)

  $${\displaystyle \frac{\partial R}{\partial Q}}=-{\displaystyle \frac{b\left(\theta \right)}{\sqrt{k_{0}Q}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)U\left(x\right)dx};$$

  (152)

  $${\displaystyle \frac{\partial R}{\partial k_0}}={\displaystyle \frac{b\left(\theta \right)}{k_0}}\sqrt{{\displaystyle \frac{Q}{k_0}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\psi }^{\left(1\right)}\left(x\right)U\left(x\right)dx}+\left[U\left(\mathcal{l}\right)-U_0\right].$$

  (153)

The expressions presented in Equations (151)–(153) are to be evaluated at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. These expressions can be evaluated after having obtained the 1st-level adjoint sensitivity function ${\psi }^{\left(1\right)}\left(x\right)$ by solving, a single time, the 1st-LASS represented by the NODE-net comprising Equations (147) and (148). Solving this 1st-LASS yields the following closed-form expression for the 1st-level adjoint sensitivity function ${\psi }^{\left(1\right)}\left(x\right)$:

$${\psi }^{\left(1\right)}\left(x\right)={\displaystyle \frac{k_0}{b\left(\theta \right)}}\left[1-H\left(x-\mathcal{l}\right)\right]\sin \left(\mathcal{l}-x\right)b\left(\theta \right).$$

(154)

Using in Equations (151)–(153) the expression for ${\psi }^{\left(1\right)}\left(x\right)$ obtained in Equation (154) and the expression for $U\left(x\right)$ obtained in Equation (77) yields the following expressions for the first-order sensitivities of the decoder response:

$${\displaystyle \frac{\partial R}{\partial U_0}}=k_0\left[\cos \mathcal{l}b\left(\theta \right)-1\right]\hspace{0.17em};$$

(155)

$${\displaystyle \frac{\partial R}{\partial Q}}=-{\displaystyle \frac{U_0\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right);$$

(156)

$${\displaystyle \frac{\partial R}{\partial k_0}}={\displaystyle \frac{U_0\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)+U_0\left[\cos b\left(\theta \right)\mathcal{l}-1\right].$$

(157)

3.3. Comparative Discussion: Application of the 1st-FASAM-NIDE-V Versus the 1st-FASAM-NODE

The developments presented in Section 3.1 and Section 3.2 indicate the following conclusions regarding the comparison of applying the 1st-FASAM-NIDE-V versus the 1st-FASAM-NODE for the nonlinear paradigm heat conduction model:

(i)

The illustrative nonlinear heat conduction model can be equivalently framed either as a nonlinear NODE-net in the original temperature-framework or as a linear NIDE-V net in the Kirchhoff-transformed framework. The solutions for the corresponding dependent variables, i.e., $T\left(x\right)$ in the temperature-framework and $U\left(x\right)$ in the Kirchhoff-transformed framework, are equivalent to each other, as expected.

(ii)

The 1st-Level Variational Sensitivity System (1st-LVSS) in the Kirchhoff-transformed framework is also equivalent to the 1st-LVSS in the temperature-framework.

(iii)

On the other hand, the 1st-Level Adjoint Sensitivity System (1st-LASS) in the Kirchhoff-transformed framework is not equivalent to the 1st-LASS in the temperature-framework. Consequently, their respective solutions (1st-level adjoint functions) are not equivalent to each other. For example, comparing the solution ${\phi }^{\left(1\right)}\left(x\right)$ provided in Equation (119) for the 1st-LASS in the NIDE-V Kirchhoff-transformed framework (cf., Equations (110) and (111)) with the solution ${\psi }^{\left(1\right)}\left(x\right)$ provided in Equation (154) for the 1st-LASS in the NODE Kirchhoff-transformed framework (cf., Equations (147) and (148)) reveals the following relations:

$${\displaystyle \frac{d{\psi }^{\left(1\right)}\left(x\right)}{dx}}=-{\phi }^{\left(1\right)}\left(x\right)\hspace{0.17em}.$$

(158)

(iv)

The mathematical derivations/computations in the Kirchhoff-transformed framework, in which the model is linear in the Kirchhoff-transformed dependent variable $U\left(x\right)$, are simpler (as would be intuitively expected) than the mathematical derivations/computations in the temperature-framework, in which the model is nonlinear in dependent variable $T\left(x\right)$.

4. Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra Type (2nd-FASAM-NIDE-V)

The second-order sensitivities of the response $R\left[h;f\left(\theta \right)\right]$ defined in Equation (5) are obtained by computing the first-order sensitivities of the expression $R_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right)$ defined in Equation (31), which represents the ith-first-order sensitivity of the response $R\left[h;f\left(\theta \right)\right]$ with respect to the component $f_i\left(\theta \right)$ of the feature function $f\left(\theta \right)\triangleq {\left[f_1\left(\theta \right),\dots ,f_{TF}\left(\theta \right)\right]}^{†}$. In other words, the second-order sensitivities of $R\left[h;f\left(\theta \right)\right]$ will be computed by conceptually using their basic definitions as being the “first-order sensitivities of the first-order sensitivities” starting from the G-differential of $R_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right)$, which is obtained by definition as follows:

$$\begin{array}{l}for\hspace{0.17em}\hspace{0.17em}i=1,\dots ,TF:\hspace{1em}\delta R_i^{\left(1\right)}\left(h^0;f^0;a^{\left(1,0\right)};v^{\left(1\right)};\delta a^{\left(1\right)};\delta f\right)\\ \triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}\hspace{0.17em}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{D}_i^{\left(1\right)}\left[h^0\left(t\right)+\epsilon v^{\left(1\right)}\left(t\right);f^0+\epsilon \delta f;a^{\left(1,0\right)}+\epsilon \delta a^{\left(1\right)};t\right]dt}\right\}}_{\epsilon =0}\\ =\hspace{0.17em}{\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;a^{\left(1,0\right)};\delta f\right)\right\}}_{dir}+{\left\{R_i^{\left(1\right)}\left(h^0;f^0;a^{\left(1,0\right)};v^{\left(1\right)};\delta a^{\left(1\right)}\right)\right\}}_{ind}\hspace{0.17em},\end{array}$$

(159)

where the “direct effect term” arises directly from variations $\delta f$ and is defined as follows:

$${\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;a^{\left(1,0\right)};\delta f\right)\right\}}_{dir}\triangleq {\displaystyle \sum _{j=1}^{TF}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt{\left\{\frac{\partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]}{\partial f_j}\delta f_j\right\}}_{\left(h^0;a^{\left(1,0\right)};f^0\right)}}},$$

(160)

and where the “indirect effect term” arises indirectly, through the variations $v^{\left(1\right)}\left(t\right)\triangleq {\left[v_1^{\left(1\right)}\left(t\right),\dots ,v_{TH}^{\left(1\right)}\left(t\right)\right]}^{†}$ and $\delta a^{\left(1\right)}\triangleq {\left[\delta a_1\left(t\right),\dots ,\delta a_{TH}\left(t\right)\right]}^{†}$ in the forward and 1st-level adjoint sensitivity functions $h\left(t\right)$ and $a^{\left(1\right)}\left(t\right)$, respectively, and is defined below:

$$\begin{array}{l}{\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;v^{\left(1\right)};\delta a^{\left(1\right)}\right)\right\}}_{ind}\triangleq {\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\left\{\frac{\partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]}{\partial h}{v}^{\left(1\right)}\left(t\right)\right\}}_{\left(h^0;a^{\left(1,0\right)};f^0\right)}}\\ +{\displaystyle \underset{t_0}{\overset{t_f}{\int }}d\tau {\left\{\frac{\partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]}{\partial a^{\left(1\right)}}\delta a^{\left(1\right)}\left(t\right)\right\}}_{\left(h^0;a^{\left(1,0\right)};f^0\right)}}.\end{array}$$

(161)

The direct-effect term ${\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;a^{\left(1,0\right)};\delta f\right)\right\}}_{dir}$ can be computed immediately while the indirect-effect term ${\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;v^{\left(1\right)};\delta a^{\left(1\right)}\right)\right\}}_{ind}$ can be computed only after having determined the vectors $v^{\left(1\right)}\left(x\right)$ and $\delta a^{\left(1\right)}\left(x\right)$. The vector $v^{\left(1\right)}\left(x\right)$ is the solution of the 1st-LVSS defined by Equations (16) and (20). On the other hand, the vector $\delta a^{\left(1\right)}\left(x\right)$ is the solution of the G-differentiated 1st-LASS comprising Equations (27) and (29), which has the formal expression provided below:

$${\left\{V_{21}^{\left(2\right)}{v}^{\left(1\right)}\left(t\right)+V_{22}^{\left(2\right)}\delta a^{\left(1\right)}\left(t\right)\right\}}_{{\theta }^0}={\left\{Q_2^{\left(2\right)}\left(h;f;a^{\left(1\right)}\right)\delta f\right\}}_{{\theta }^0},$$

(162)

$$\begin{array}{l}{\left\{{\displaystyle \frac{\partial B_j^{*}\left(h;a^{\left(1\right)};f;t\right)}{\partial h}}{v}^{\left(1\right)}\left(t\right)+{\displaystyle \frac{\partial B_j^{*}\left(h;a^{\left(1\right)};f;t\right)}{\partial a^{\left(1\right)}}}\delta a^{\left(1\right)}\left(t\right)+{\displaystyle \frac{\partial B_j^{*}\left(h;a^{\left(1\right)};f;t\right)}{\partial f}}\delta f\right\}}_{{\theta }^0}=0,\\ \hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f \hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC.\end{array}$$

(163)

where:

$$V_{21}^{\left(2\right)}\triangleq {\displaystyle \frac{\partial \left[L^{*}\left(h;f\right)a^{\left(1\right)}\right]}{\partial h}}-{\displaystyle \frac{{\partial }^{2}D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial h\partial h}};$$

(164)

$$V_{22}^{\left(2\right)}\left(u;\alpha \right)\triangleq {\displaystyle \frac{\partial \left[L^{*}\left(h;f\right)a^{\left(1\right)}\right]}{\partial a^{\left(1\right)}}};$$

(165)

$$Q_2^{\left(2\right)}\left(h;f;a^{\left(1\right)}\right)\delta f\triangleq {\displaystyle \frac{{\partial }^{2}D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f\partial h\left(t\right)}}\delta f-{\displaystyle \frac{\partial \left[L^{*}\left(h;f\right)a^{\left(1\right)}\right]}{\partial f}}\delta f\hspace{0.17em}.$$

(166)

Note that Equations (162) and (163) are coupled to the 1st-LVSS through the function $v^{\left(1\right)}\left(t\right)$. Therefore, the functions $v^{\left(1\right)}\left(t\right)$ and $\delta a^{\left(1\right)}\left(t\right)$ are determined by solving the “2nd-Level Variational Sensitivity System (2nd-LVSS)” obtained by concatenating the 1st-LVSS to Equations (162) and (163). This 2nd-LVSS can be written formally in operator form as follows:

$$V^{\left(2\right)}{v}^{\left(2\right)}\left(t\right)=Q_V^{\left(2\right)}\left(h;f;a^{\left(1\right)};\delta f\right),$$

(167)

$$B_V^{\left(2\right)}\left(h;a^{\left(1\right)};f;v^{\left(1\right)};\delta a^{\left(1\right)};\delta f\right)=0,\hspace{0.17em}\hspace{0.17em}at\hspace{0.17em}\hspace{0.17em}t=t_f \hspace{0.17em}\hspace{0.17em}and/or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}t=t_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}j=1,\dots ,BC..$$

(168)

where the superscript “(2)” indicates “2nd-level”, where Equation (168) represents the concatenation of all boundary conditions comprised in Equations (16) and (163), and where the matrices and vectors appearing in Equation (167) are defined as follows:

$$V^{\left(2\right)}\triangleq \left(\begin{array}{cc}L& 0\\ V_{21}^{\left(2\right)}& V_{22}^{\left(2\right)}\end{array}\right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{v}^{\left(2\right)}\left(t\right)\triangleq \left(\begin{array}{c}{v}^{\left(1\right)}\left(t\right)\\ \delta a^{\left(1\right)}\left(t\right)\end{array}\right).$$

(169)

$$Q_V^{\left(2\right)}\left(h;f;a^{\left(1\right)};\delta f\right)\hspace{0.17em}\triangleq \left(\begin{array}{c}{Q}^{\left(1\right)}\left(h;f\right)\delta f\\ Q_2^{\left(2\right)}\left(h;f;a^{\left(1\right)}\right)\delta f\end{array}\right);$$

(170)

The relations in Equations (167) and (168) are to be satisfied at the nominal functions and parameter values, ${\theta }^0$, but this fact has not been indicated explicitly in order to simplify the notation.

Solving the 2nd-LVSS requires $T{F}^2$ large-scale computations, which are unrealistic to perform for large-scale systems comprising many parameters. The need for solving the 2nd-LVSS is alleviated by deriving an alternative expression for the indirect-effect term defined in Equation (161), in which the function $v^{\left(2\right)}\left(t\right)$ is replaced by a 2nd-level adjoint function that will not depend on the variations in the model parameter and state functions. This 2nd-level adjoint function will satisfy a 2nd-Level Adjoint Sensitivity System (2nd-LASS), which will be constructed by using the 2nd-LVSS as starting point and following the same principles as underlying the 2nd-FASAM-N [28], in a Hilbert space that will be denoted as ${\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$, ${\mathsf{\Omega }}_t\triangleq t\in \left[t_0,t_f\right]$, and which comprises as elements block-vectors of the same form as $v^{\left(2\right)}\left(t\right)$. Thus, a generic vector in ${\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$, denoted as ${\psi }^{(2)}\left(t\right)\triangleq {\left[\hspace{0.17em}{\psi }_1^{(2)}\left(t\right),{\psi }_2^{(2)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$, comprises two components of the form ${\psi }_1^{(2)}\left(t\right)\triangleq {\left[{\psi }_{1,1}^{\left(2\right)}\left(t\right),\dots ,{\psi }_{1,TH}^{\left(2\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, ${\psi }_2^{(2)}\left(t\right)\triangleq {\left[{\psi }_{2,1}^{\left(2\right)}\left(t\right),\dots ,{\psi }_{2,TH}^{\left(2\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, each of which are $TH$-dimensional column vectors; hence, ${\psi }^{(2)}\left(t\right)$ is a $\left(2\times TH\right)$-dimensional column vector. The inner product of two vectors ${\psi }^{(2)}\left(t\right)\triangleq {\left[\hspace{0.17em}{\psi }_1^{(2)}\left(t\right),{\psi }_2^{(2)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$ and ${\phi }^{(2)}\left(t\right)\triangleq {\left[{\phi }_1^{(2)}\left(t\right),{\phi }_2^{(2)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$ in the Hilbert space ${\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$ will be denoted as ${〈{\psi }^{(2)}\left(t\right),{\phi }^{(2)}\left(t\right)〉}_2$ and defined below:

$${〈{\psi }^{(2)}\left(t\right),{\phi }^{(2)}\left(t\right)〉}_2\triangleq {\displaystyle \sum _{i=1}^2{〈{\psi }_i^{(2)}\left(t\right),{\phi }_i^{(2)}\left(t\right)〉}_1}=\hspace{0.17em}{\displaystyle \sum _{i=1}^2{\displaystyle \sum _{j=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{\psi }_{i,j}^{\left(2\right)}\left(t\right){\phi }_{i,j}^{\left(2\right)}\left(t\right)dt}}}\hspace{0.17em}.$$

(171)

Using the definition of the inner product defined in Equation (171), construct the inner product of Equation (167) with a vector $a^{(2)}\left(t\right)\triangleq {\left[a_1^{(2)}\left(t\right),a_2^{(2)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_2\left({\mathsf{\Omega }}_t\right)$, with $a_1^{(2)}\left(t\right)\triangleq {\left[a_{1,1}^{\left(2\right)}\left(t\right),\dots ,a_{1,TH}^{\left(2\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, $a_2^{(2)}\left(t\right)\triangleq {\left[a_{2,1}^{\left(2\right)}\left(t\right),\dots ,a_{2,TH}^{\left(2\right)}\left(t\right)\right]}^{†}\in {\mathrm{H}}_1\left({\mathsf{\Omega }}_t\right)$, to obtain the following relation:

$${〈a^{(2)}\left(t\right),V^{\left(2\right)}{v}^{\left(2\right)}\left(t\right)〉}_2={〈a^{(2)}\left(t\right),Q_V^{\left(2\right)}\left(h;f;a^{\left(1\right)};\delta f\right)〉}_2$$

(172)

The inner product on the left-side of Equation (172) is now further transformed by using the definition of the adjoint operator to obtain the following relation:

$${〈a^{(2)}\left(t\right),V^{\left(2\right)}{v}^{\left(2\right)}\left(t\right)〉}_2={〈v^{\left(2\right)}\left(t\right),A^{\left(2\right)}{a}^{(2)}\left(t\right)〉}_2+P^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(t\right);f\right),$$

(173)

where $P^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(t\right);f\right)$ denotes the corresponding bilinear concomitant on the domain’s boundary, evaluated at the nominal values for the parameters and respective state functions and where the adjoint operator $A^{\left(2\right)}$ is defined as follows:

$$A^{\left(2\right)}\triangleq {\left[V^{\left(2\right)}\right]}^{*}=\left(\begin{array}{cc}{L}^{*}& {\left[V_{21}^{\left(2\right)*}\right]}^{†}\\ 0& {\left[V_{22}^{\left(2\right)*}\right]}^{†}\end{array}\right).$$

(174)

The definition domain of the adjoint (matrix-valued) operator $A^{\left(2\right)}$ is specified by requiring the function $a^{(2)}\left(t\right)\triangleq {\left[a_1^{(2)}\left(t\right),a_2^{(2)}\left(t\right)\right]}^{†}$ to satisfy adjoint boundary/initial conditions that will be formally denoted in operator form as follows:

$$B_A^{\left(2\right)}\left(v^{\left(2\right)}\left(t\right);a^{(2)}\left(t\right);f\right)=0,\hspace{1em}t\in \partial {\mathsf{\Omega }}_t\hspace{0.17em}.$$

(175)

The 2nd-level adjoint boundary/initial conditions represented by Equation (175) are determined by requiring that: (a) they must be independent of unknown values of $v^{\left(2\right)}\left(t\right)$, and (b) the substitution of the boundary and/or initial conditions represented by Equations (168) and (175) into the expression of $P^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(t\right);f\right)$ must cause all terms containing unknown values of $v^{\left(2\right)}\left(t\right)$ to vanish.

Implementing the 2nd-level (forward and adjoint) boundary/initial conditions, namely Equations (168) and (175), into Equation (173) will transform the later into the following form:

$${〈v^{\left(2\right)}\left(t\right),A^{\left(2\right)}{a}^{(2)}\left(t\right)〉}_2={〈a^{(2)}\left(t\right),V^{\left(2\right)}{v}^{\left(2\right)}\left(t\right)〉}_2-{\widehat{P}}^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(t\right);f;\delta f\right),$$

(176)

where ${\widehat{P}}^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(t\right);f;\delta f\right)$ denotes residual boundary terms that may not have vanished after having implemented the boundary/initial conditions represented by Equations (168) and (175). The right-side of Equation (172) is now used to replace the vector $V^{\left(2\right)}{v}^{\left(2\right)}\left(t\right)$ in the first term on the right-side of Equation (176), thereby obtaining the following relation:

$${〈v^{\left(2\right)}\left(t\right),A^{\left(2\right)}{a}^{(2)}\left(t\right)〉}_2={〈a^{(2)}\left(t\right),Q_V^{\left(2\right)}\left(h;f;a^{\left(1\right)};\delta f\right)〉}_2-{\widehat{P}}^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(t\right);f;\delta f\right).$$

(177)

The definition of the 2nd-level adjoint function $a^{(2)}\left(t\right)\triangleq {\left[a_1^{(2)}\left(t\right),a_2^{(2)}\left(t\right)\right]}^{†}$ is now completed by requiring that the left-side of Equation (177) be the same as the “indirect-effect term” ${\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;v^{\left(1\right)};\delta a^{\left(1\right)}\right)\right\}}_{ind}$ defined by Equation (161), for each index $i=1,\dots ,TF$. Recall that the index $i=1,\dots ,TF$ corresponds to the “ith” first-order sensitivity $R_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right)$. Hence, there will be a total of $TF$ 2nd-level adjoint sensitivity functions of the form $a^{(2)}\left(i;t\right)\triangleq {\left[a_1^{(2)}\left(i;t\right),a_2^{(2)}\left(i;t\right)\right]}^{†}$, $i=1,\dots ,TF$, each such adjoint function corresponding to a specific (i-dependent) indirect-effect term. The left-side of Equation (177) will be identical to the right-side of Equation (161) by requiring that the following relation be satisfied by the 2nd-level adjoint functions (block-vectors) $a^{(2)}\left(i;t\right)\triangleq {\left[a_1^{(2)}\left(i;t\right),a_2^{(2)}\left(i;t\right)\right]}^{†}$, for each $i=1,\dots ,TF$:

$$A^{\left(2\right)}{a}^{(2)}\left(i;t\right)=\left(\begin{array}{c}\partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]/\partial u\\ \partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]/\partial a^{\left(1\right)}\end{array}\right);\hspace{1em}i=1,\dots ,TF.$$

(178)

The boundary conditions to be satisfied by each of the 2nd-level adjoint functions $a^{(2)}\left(i;t\right)\triangleq {\left[a_1^{(2)}\left(i;t\right),a_2^{(2)}\left(i;t\right)\right]}^{†}$ are as represented by Equation (175), namely:

$$B_A^{\left(2\right)}\left(v^{\left(2\right)}\left(t\right);a^{(2)}\left(i;t\right);f\right)=0,\hspace{1em}t\in \partial {\mathsf{\Omega }}_t\hspace{0.17em}.$$

(179)

The system of equations represented by Equations (178) and (179) will be called the 2nd-Level Adjoint Sensitivity System (2nd-LASS); its solution, $a^{(2)}\left(i;t\right)\triangleq {\left[a_1^{(2)}\left(i;t\right),a_2^{(2)}\left(i;t\right)\right]}^{†}$, will be called the 2nd-level adjoint function. The 2nd-LASS is independent of parameter variations. Furthermore, the ${\left(2\times TH\right)}^2$-dimensional matrix $A^{\left(2\right)}$ is independent of the index $i=1,\dots ,TF$. Only the source-term on the right-side of Equation (178) depends on the index $i=1,\dots ,TF$. Therefore, the same solver can be used to invert the matrix $A^{\left(2\right)}$ in order to solve numerically the 2nd-LASS that corresponds to each first-order sensitivity $R_i^{\left(1\right)}\left(h;f;a^{\left(1\right)}\right)$, $i=1,\dots ,TF$. Computationally, it would be efficient to store, if possible, the inverse matrix ${\left[A^{\left(2\right)}\right]}^{-1}$, in order to multiply directly this inverse matrix with the source term corresponding to each index $i=1,\dots ,TF$, in order to obtain the corresponding 2nd-level adjoint function $a^{(2)}\left(i;t\right)\triangleq {\left[a_1^{(2)}\left(i;t\right),a_2^{(2)}\left(i;t\right)\right]}^{†}$.

Using Equations (176)–(179) in Equation (161) yields the following expression for the indirect-effect term ${\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;v^{\left(1\right)};\delta a^{\left(1\right)}\right)\right\}}_{ind}$, for each $i=1,\dots ,TF$:

$${\left\{\delta R_i^{\left(1\right)}\left(h^0;f^0;v^{\left(1\right)};\delta a^{\left(1\right)}\right)\right\}}_{ind}={〈a^{(2)}\left(i;t\right),Q_V^{\left(2\right)}\left(h;f;a^{\left(1\right)};\delta f\right)〉}_2-{\widehat{P}}^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(i;t\right);f;\delta f\right)\hspace{0.17em}.$$

(180)

Adding the expression obtained in Equation (180) for the indirect-effect term together with the expression for the direct-effect term defined in Equation (160) yields the following expression for the total differential defined by Equation (159), for $i=1,\dots ,TF$:

$$\begin{array}{l}\delta R_i^{\left(1\right)}\left(h^0;f^0;a^{\left(1,0\right)};v^{\left(1\right)};\delta a^{\left(1\right)};\delta f\right)={\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt\frac{\partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]}{\partial f\left(\theta \right)}\delta f\left(\theta \right)}\\ +{〈a^{(2)}\left(i;t\right),Q_V^{\left(2\right)}\left(h;f;a^{\left(1\right)};\delta f\right)〉}_2-{\widehat{P}}^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(i;t\right);f;\delta f\right)\hspace{0.17em}\triangleq {\displaystyle \sum _{j=1}^{TF}\frac{{\partial }^{2}R\left[u\left(x\right);\alpha \right]}{\partial f_j\partial f_i}\delta f_j}.\end{array}$$

(181)

The second-order sensitivities ${\partial }^{2}R/\partial f_j\partial f_i$, $i,j=1,\dots ,TF$ are obtained by identifying in Equation (181) the expressions that multiply the variations $\delta f_j$, which yields the following expressions after detailing the second-term on the right side of Equation (181):

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R\left[u\left(x\right);\alpha \right]}{\partial f_j\partial f_i}}={\displaystyle \underset{t_0}{\overset{t_f}{\int }}dt}{\displaystyle \frac{\partial D_i^{\left(1\right)}\left[h\left(t\right);a^{\left(1\right)};f\left(\theta \right);t\right]}{\partial f_j}}-{\displaystyle \frac{\partial {\widehat{P}}^{\left(2\right)}\left(v^{\left(2\right)};a^{(2)}\left(i;t\right);f;\delta f\right)}{\partial f_j}}\\ +{\displaystyle \sum _{m=1}^{TH}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_{1,m}^{\left(2\right)}\left(t\right)q_{m,j}^{\left(1\right)}\left(h;f\right)dt}}+{\displaystyle \underset{t_0}{\overset{t_f}{\int }}{a}_2^{(2)}\left(t\right)·\left[\frac{{\partial }^{2}D\left[h\left(t\right);f\left(\theta \right);t\right]}{\partial f_j\partial h\left(t\right)}-\frac{\partial \left[L^{*}\left(h;f\right)a^{\left(1\right)}\right]}{\partial f_j}\right]\hspace{0.17em}dt}.\end{array}$$

(182)

As Equations (178) and (179) indicate, solving the 2nd-LASS once provides the 2nd-level adjoint function $a^{(2)}\left(i;t\right)$ for each index $i=1,\dots ,TP$. Thus, the exact computation of all of the partial second-order sensitivities, ${\partial }^{2}R\left(\alpha ,u\right)/\partial f_j\partial f_i\hspace{0.17em}\hspace{0.17em}$ $\hspace{0.17em}\hspace{0.17em}i,\hspace{0.17em}j=1,\dots ,TP,$ requires solving the 2nd-LASS at most $TF$-times, amounting to at most $TF$ “large-scale” computations for, rather than at least $O\left(T{W}^2\right)$ large-scale computations as would be required by forward methods. It is important to note that if the 2nd-LASS is solved $TF$-times, the 2nd-order mixed sensitivities ${\partial }^{2}R\left(\alpha ,u\right)/\partial f_j\partial f_i\hspace{0.17em}\hspace{0.17em}$ will be computed twice, in two different ways, in terms of two distinct 2nd-level adjoint functions. Consequently, the symmetry property ${\partial }^{2}R\left(\alpha ,u\right)/\partial f_j\partial f_i={\partial }^{2}R\left(\alpha ,u\right)/\partial f_i\partial f_j$ inherent to the second-order sensitivities provides an intrinsic (numerical) verification that the respective components of the 2nd-level adjoint function $a^{(2)}\left(i;t\right)$, for two distinct values of the index “i”, are computed accurately.

Since the adjoint matrix $A^{\left(2\right)}$ is block-diagonal, solving the 2nd-LASS is equivalent to solving two 1st-LASS, with two different source terms. Thus, the “solvers” and the computer program used for solving the 1st-LASS can also be used for solving the 2nd-LASS. The 2nd-LASS was designated as the “second-level” rather than the “second-order” adjoint sensitivity system, since the 2nd-LASS does not involve any explicit 2nd-order G-derivatives of the operators underlying the original system but involves the inversion of the same operators that needed to be inverted for solving the 1st-LASS. The block-triangular structure of the 2nd-LASS enables full flexibility for prioritizing the computation of the 2nd-order sensitivities. The computation of the 2nd-order sensitivities would logically be prioritized based on the relative magnitudes of the 1st-order sensitivities: the largest relative 1st-order response sensitivity should have the highest priority for computing the corresponding 2nd-order mixed sensitivities; then, the second largest relative 1st-order response sensitivity should be considered next, and so on. The unimportant 2nd-order sensitivities can be deliberately neglected while knowing the error incurred by neglecting them. Computing 2nd-order sensitivities that correspond to vanishing 1st-order sensitivities may also be of interest, since vanishing 1st-order sensitivities may indicate critical points of the response in the phase-space of model parameters.

5. Illustrative Application of the 2nd-FASAM-NIDE-V Methodology to Compute Second-Order Sensitivities of the Heat Conduction Model

The application of the 2nd-FASAM-NIDE-V methodology will be illustrated in this section by continuing the analysis of the heat conduction model considered in Section 3. As highlighted while developing the general 2nd-FASAM-NIDE-V methodology in Section 4, the second-order sensitivities are conceptually determined by considering them to be the “first-order sensitivities of the first-order sensitivities.” To underscore the essential concepts and steps underlying the application of the 2nd-FASAM-NIDE-V methodology, the heat conduction model will be considered in the Kirchhoff-transformed framework, since the algebraic manipulation in this framework are considerably less involved than in the temperature-framework.

5.1. Second-Order Sensitivities of the Heat Conduction Model in the NIDE-V Kirchhoff-Transformed Framework

The expressions of the first-order sensitivities to be considered as “responses that give rise to the second-order sensitivities of the decoder response with respect to the feature functions and model parameters” are presented in Equations (115)–(118).

5.1.1. Determining the Second-Order Sensitivities Stemming from $\partial R/\partial T_0$

The second-order sensitivities stemming from the first-order sensitivity $\partial R/\partial T_0$ are comprised in the G-differential of the expression provided in Equation (117), which yields, by definition, the following relation:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial T_0}}\right)\triangleq {\left\{\left(1+\beta T_0\right){\displaystyle \frac{d}{d\epsilon }}\hspace{0.17em}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[{\phi }^{\left(1,0\right)}\left(x\right)+\epsilon \delta {\phi }^{\left(1\right)}\left(x\right)\right]\delta \left(x\right)dx}\right\}}_{\epsilon =0}\\ +{\phi }^{\left(1\right)}\left(0\right)\left[\left(\delta \beta \right)T_0+\beta \left(\delta T_0\right)\right]-\left(\delta k_0\right)-\left(\delta k_0\right)T_0^0{\beta }^0-\left(\delta T_0\right)k_0^0{\beta }^0-\left(\delta \beta \right)k_0^0{T}_0^0\\ \triangleq \delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind},\end{array}$$

(183)

where the direct-effect term $\delta {\left(\partial R/\partial T_0\right)}_{dir}$ and the indirect-effect term $\delta {\left(\partial R/\partial T_0\right)}_{ind}$, respectively, are defined below:

$$\delta {\left(\partial R/\partial T_0\right)}_{dir}\triangleq {\phi }^{\left(1\right)}\left(0\right)\left[\left(\delta \beta \right)T_0+\beta \left(\delta T_0\right)\right]-\left(\delta k_0\right)-\left(\delta k_0\right)T_0\beta -\left(\delta T_0\right)k_0\beta -\left(\delta \beta \right)k_0{T}_0\hspace{0.17em};$$

(184)

$$\delta {\left(\partial R/\partial T_0\right)}_{ind}\triangleq \left(1+\beta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\delta \left(x\right)dx\hspace{0.17em}.}$$

(185)

The above expressions are to be evaluated at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.

The direct-effect term $\delta {\left(\partial R/\partial T_0\right)}_{dir}$ can be computed immediately, while the indirect-effect term $\delta {\left(\partial R/\partial T_0\right)}_{ind}$ can be computed only after having determined the function $\delta {\phi }^{\left(1\right)}\left(t\right)$, which is the solution of the G-differentiated 1st-LASS comprising Equations (110) and (111). Applying the definition of the G-differential to these equations yields the following NIDE-V system:

$$-{\displaystyle \frac{d}{dx}}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}=\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\hspace{0.17em};$$

(186)

$$\delta {\phi }^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

(187)

The NIDE-V system comprising Equations (186) and (187) constitutes the 2nd-LVSS for the 2nd-level variational function $\delta {\phi }^{\left(1\right)}\left(x\right)$, which depends on parameter variations. Therefore, the 2nd-LVSS would need to be solved repeatedly to determine $\delta {\phi }^{\left(1\right)}\left(x\right)$ for all possible parameter variations of interest. The need for solving the 2nd-LVSS is alleviated by deriving an alternative expression for the indirect-effect term defined in Equation (185), in which the function $\delta {\phi }^{\left(1\right)}\left(x\right)$ is replaced by a 2nd-level adjoint function that is independent of variations in the model parameter and state functions. This 2nd-level adjoint function will satisfy a 2nd-Level Adjoint Sensitivity System (2nd-LASS), which will be constructed by using the 2nd-LVSS as starting point and following the same principles outlined in Section 4. Since the 2nd-LVSS involves a one-component dependent variable (namely: $\delta {\phi }^{\left(1\right)}\left(x\right)$), the corresponding 2nd-level adjoint function will also be a one-component function, which will be denoted $a^{(2)}\left(1;x\right)$, using the notation introduced in Section 4. The superscript “(2)” in $a^{(2)}\left(1;x\right)$ indicates “2nd-level” while the argument “1” indicates that this 2nd-level adjoint function corresponds to the sensitivity $\partial R/\partial T_0$, which is the “first” first-order sensitivity to be considered. The Hilbert space appropriate for constructing the 2nd-LASS for the function $a^{(2)}\left(1;x\right)$ is endowed with the inner product defined in Equation (87). The construction of this 2nd-LASS proceeds as follows:

• Use Equation (87) to construct the inner product of Equation (186) with a yet unspecified function $a^{(2)}\left(1;x\right)$ to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\\ ={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)\left[\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (188)
• Integrate by parts the terms on the left-side of Equation (188) to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx=-a^{(2)}\left(1;\mathcal{l}\right)\delta {\phi }^{\left(1\right)}\left(\mathcal{l}\right)\\ +a^{(2)}\left(1;0\right)\delta {\phi }^{\left(1\right)}\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}{a}^{(2)}\left(1;x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}^{(2)}\left(1;y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\end{array}$$

  (189)
• Require the last term on the right-side of Equation (189) to represent the indirect-effect term defined in Equation (185), by requiring the function $a^{(2)}\left(1;x\right)$ to satisfy the following relation:

  $${\displaystyle \frac{d{a}^{(2)}\left(1;x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}^{(2)}\left(1;y\right)dy}=\left(1+\beta T_0\right)\delta \left(x\right).$$

  (190)
• Use in Equation (189) the boundary condition provided in Equation (187) and eliminate the unknown remaining boundary term by imposing the following boundary condition of the function $a^{(2)}\left(1;x\right)$:

  $$a^{(2)}\left(1;0\right)=0\hspace{0.17em}.$$

  (191)

  The NIDE-V net comprising Equations (190) and (191) constitutes the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for the 2nd-level adjoint function $a^{(2)}\left(1;x\right)$. Note that this 2nd-LASS is linear in $a^{(2)}\left(1;x\right)$ and is independent of parameter variations, so it needs to be solved only once to obtain the function $a^{(2)}\left(1;x\right)$.
• Use Equations (187), (190), and (191) in Equation (189) to obtain the following expression for the indirect-effect term defined in Equation (185):

  $$\delta {\left(\partial R/\partial T_0\right)}_{ind}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)\left[\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.$$

  (192)
• Adding the expression obtained in Equation (192) with the expression for the direct-effect term defined in Equation (184) yields the following expression for the G-differential $\delta \left(\partial R/\partial T_0\right)$:

  $$\begin{array}{l}\delta \left(\partial R/\partial T_0\right)={\phi }^{\left(1\right)}\left(0\right)\left[\left(\delta \beta \right)T_0+\beta \left(\delta T_0\right)\right]-\left(\delta k_0\right)-\left(\delta k_0\right)T_0\beta \hspace{0.17em}-\left(\delta T_0\right)k_0\beta \\ -\left(\delta \beta \right)k_0{T}_0+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)\left[\left(\delta k_0\right)\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (193)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• Inserting into Equation (193) the result obtained in Equation (98) and identifying in Equation (193) the expressions that multiply the respective parameter variations yields the expressions for the following second-order sensitivities of the decoder response:

  $${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}=\beta \left[{\phi }^{\left(1\right)}\left(0\right)-k_0\right]=\beta \left[\cos \mathcal{l}b\left(\theta \right)-1\right];$$

  (194)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial \beta \hspace{0.17em}\partial T_0}}=T_0\left[{\phi }^{\left(1\right)}\left(0\right)-k_0\right]=T_0\left[\cos \mathcal{l}b\left(\theta \right)-1\right];$$

  (195)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial Q\hspace{0.17em}\partial T_0}}=-{\displaystyle \frac{1}{\sqrt{k_0}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)dx{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}}\hspace{0.17em};$$

  (196)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial T_0}}=-1-T_0\beta +a^{(2)}\left(1;\mathcal{l}\right)+{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;x\right)\left[{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.$$

  (197)

The sensitivities obtained in Equations (196) and (197) can be computed after having determined the 1st-level adjoint sensitivity function $a^{(2)}\left(1;x\right)$. Solving the 2nd-LASS comprising Equations (190) and (191) yields the following expression for the 2nd-level adjoint sensitivity function $a^{(2)}\left(1;x\right)$:

$$a^{(2)}\left(1;x\right)=\left(1+\beta T_0\right)H\left(x\right)\cos xb\left(\theta \right).$$

(198)

Inserting into Equation (196) and Equation (197), respectively, the expressions obtained in Equations (198) and (119) and performing the respective integrations yields the following results:

$${\displaystyle \frac{{\partial }^{2}R}{\partial Q\partial T_0}}=-\left(1+\beta T_0\right){\displaystyle \frac{\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right)$$

(199)

$${\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial T_0}}=\left(1+\beta T_0\right)\left[\cos \mathcal{l}b\left(\theta \right)-1+{\displaystyle \frac{\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)\right]\hspace{0.17em}.$$

(200)

5.1.2. Determining the Second-Order Sensitivities Stemming from $\partial R/\partial \beta$

The second-order sensitivities stemming from the first-order sensitivity $\partial R/\partial \beta$ are comprised in the G-differential of the expression provided in Equation (118), which yields, by definition, the following relation:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \beta }}\right)\triangleq {\left\{{\displaystyle \frac{{\left(T_0^0\right)}^2}{2}}{\displaystyle \frac{d}{d\epsilon }}\hspace{0.17em}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[{\phi }^{\left(1,0\right)}\left(x\right)+\epsilon \delta {\phi }^{\left(1\right)}\left(x\right)\right]\delta \left(x\right)dx}\right\}}_{\epsilon =0}+{\phi }^{\left(1\right)}\left(0\right)T_0^0\left(\delta T_0\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\left(\delta k_0\right){\displaystyle \frac{{\left(T_0^0\right)}^2}{2}}-\left(\delta T_0\right)k_0^0{T}_0^0\triangleq \delta {\left({\displaystyle \frac{\partial R}{\partial \beta }}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial \beta }}\right)}_{ind},\end{array}$$

(201)

where the direct-effect term $\delta {\left(\partial R/\partial \beta \right)}_{dir}$ and the indirect-effect term $\delta {\left(\partial R/\partial \beta \right)}_{ind}$, respectively, are defined below:

$$\delta {\left(\partial R/\partial \beta \right)}_{dir}\triangleq {\phi }^{\left(1\right)}\left(0\right)T_0\left(\delta T_0\right)-\left(\delta k_0\right){\displaystyle \frac{T_0^2}{2}}-\left(\delta T_0\right)k_0{T}_0\hspace{0.17em};$$

(202)

$$\delta {\left(\partial R/\partial \beta \right)}_{ind}\triangleq {\displaystyle \frac{T_0^2}{2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\delta \left(x\right)dx\hspace{0.17em}.}$$

(203)

The above expressions are to be evaluated at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. The direct-effect term $\delta {\left(\partial R/\partial \beta \right)}_{dir}$ can be computed immediately while the indirect-effect term $\delta {\left(\partial R/\partial \beta \right)}_{ind}$ can be computed only after having determined the function $\delta {\phi }^{\left(1\right)}\left(x\right)$, which is the solution of the 2nd-LVSS represented by Equations (186) and (187). As before, the need for solving this 2nd-LVSS repeatedly, for every parameter variation of interest, is alleviated by deriving an alternative expression for the indirect-effect term defined in Equation (203), in which the function $\delta {\phi }^{\left(1\right)}\left(x\right)$ is replaced by a 2nd-level adjoint function that is independent of variations in the model parameter and state functions. This 2nd-level adjoint function will also be a one-component function, which will be denoted $a^{(2)}\left(2;x\right)$, using the notation introduced in Section 4. The superscript “(2)” in $a^{(2)}\left(2;x\right)$ indicates “2nd-level” while the argument “2” indicates that this 2nd-level adjoint function corresponds to the sensitivity $\partial R/\partial \beta$, which is the “second” first-order sensitivity to be considered. The Hilbert space appropriate for constructing the 2nd-LASS for the function $a^{(2)}\left(2;x\right)$ is also endowed with the inner product defined in Equation (87). The construction of this 2nd-LASS proceeds along the same steps as outlined in Section 5.1.1, above, as follows:

• Use Equation (87) to construct the inner product of Equation (186) with a yet unspecified function $a^{(2)}\left(2;x\right)$ to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\\ ={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)\left[\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (204)
• Integrate by parts the terms on the left-side of Equation (204) to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx=-a^{(2)}\left(2;\mathcal{l}\right)\delta {\phi }^{\left(1\right)}\left(\mathcal{l}\right)\\ +a^{(2)}\left(2;0\right)\delta {\phi }^{\left(1\right)}\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}{a}^{(2)}\left(2;x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}^{(2)}\left(2;y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\end{array}$$

  (205)
• Require the last term on the right-side of Equation (205) to represent the indirect-effect term defined in Equation (203), by requiring the function $a^{(2)}\left(2;x\right)$ to satisfy the following relation:

  $${\displaystyle \frac{d{a}^{(2)}\left(1;x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}^{(2)}\left(1;y\right)dy}={\displaystyle \frac{T_0^2}{2}}\delta \left(x\right).$$

  (206)
• Use in Equation (205) the boundary condition provided in Equation (187) and eliminate the unknown remaining boundary term by imposing the following boundary condition of the function $a^{(2)}\left(2;x\right)$:

  $$a^{(2)}\left(2;0\right)=0\hspace{0.17em}.$$

  (207)

  The NIDE-V net comprising Equations (206) and (207) constitutes the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for the 2nd-level adjoint function $a^{(2)}\left(2;x\right)$. Note that this 2nd-LASS is linear in $a^{(2)}\left(2;x\right)$ and is independent of parameter variations, so it needs to be solved only once to obtain the function $a^{(2)}\left(2;x\right)$.
• Use Equations (187), (206), and (207) in Equation (205) to obtain the following expression for the indirect-effect term defined in Equation (203):

  $$\delta {\left(\partial R/\partial \beta \right)}_{ind}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)\left[\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.$$

  (208)
• Adding the expression obtained in Equation (208) with the expression for the direct-effect term defined in Equation (202) yields the following expression for the G-differential $\delta \left(\partial R/\partial \beta \right)$:

  $$\begin{array}{l}\delta \left(\partial R/\partial \beta \right)={\phi }^{\left(1\right)}\left(0\right)T_0\left(\delta T_0\right)-\left(\delta k_0\right){\displaystyle \frac{T_0^2}{2}}-\left(\delta T_0\right)k_0{T}_0\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)\left[\left(\delta k_0\right)\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (209)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• Inserting into Equation (209) the result obtained in Equation (98) and identifying in the resulting form of Equation (209) the expressions that multiply the respective parameter variations yields the expressions for the following second-order sensitivities of the decoder response:

  $${\displaystyle \frac{{\partial }^{2}R}{\partial T\partial {\beta }_0}}=T_0\left[{\phi }^{\left(1\right)}\left(0\right)-k_0\right]=T_0\left[\cos \mathcal{l}b\left(\theta \right)-1\right];$$

  (210)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial \beta \hspace{0.17em}\partial \beta }}=0;$$

  (211)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial Q\partial \beta }}=-{\displaystyle \frac{1}{\sqrt{k_0}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)dx{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}}\hspace{0.17em};$$

  (212)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial \beta }}=-{\displaystyle \frac{T_0^2}{2}}+a^{(2)}\left(2;\mathcal{l}\right)+{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;x\right)\left[{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.$$

  (213)

The sensitivities obtained in Equations (196) and (197) can be computed after having determined the 2nd-level adjoint sensitivity function $a^{(2)}\left(1;x\right)$. Solving the 2nd-LASS comprising Equations (190) and (191) yields the following expression for the 2nd-level adjoint sensitivity function $a^{(2)}\left(1;x\right)$:

$$a^{(2)}\left(2;x\right)={\displaystyle \frac{T_0^2}{2}}H\left(x\right)\cos xb\left(\theta \right).$$

(214)

Inserting into Equation (212) and (213), respectively, the expressions obtained in Equations (214) and (119), and performing the respective integrations yields the following results:

$${\displaystyle \frac{{\partial }^{2}R}{\partial Q\partial \beta }}=-{\displaystyle \frac{\mathcal{l}{T}_0^2}{4}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right)\hspace{0.17em};$$

(215)

$${\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial \beta }}={\displaystyle \frac{T_0^2}{2}}\left[\cos \mathcal{l}b\left(\theta \right)-1-{\displaystyle \frac{\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)\right]\hspace{0.17em}.$$

(216)

5.1.3. Determining the Second-Order Sensitivities Stemming from $\partial R/\partial Q$

The second-order sensitivities stemming from the first-order sensitivity $\partial R/\partial Q$ are comprised in the G-differential of the expression provided in Equation (115), which yields, by definition, the following relation:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial Q}}\right)=-{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \frac{1}{k_0^0+\epsilon \delta k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx\left[{\phi }^{\left(1,0\right)}\left(x\right)+\epsilon \delta {\phi }^{\left(1\right)}\left(x\right)\right]{\displaystyle \underset{0}{\overset{x}{\int }}\left[U^0\left(y\right)+\epsilon \delta U\left(y\right)\right]dy}}\right\}}_{\epsilon =0}\\ =\delta {\left({\displaystyle \frac{\partial R}{\partial Q}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial Q}}\right)}_{ind},\end{array}$$

(217)

where the direct-effect term $\delta {\left(\partial R/\partial Q\right)}_{dir}$ and the indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$, respectively, are defined below:

$$\delta {\left(\partial R/\partial Q\right)}_{dir}\triangleq {\displaystyle \frac{\delta k_0}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}$$

(218)

$$\delta {\left(\partial R/\partial Q\right)}_{ind}\triangleq -{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]dx}\hspace{0.17em}.$$

(219)

The above expressions are to be evaluated at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. The direct-effect term $\delta {\left(\partial R/\partial Q\right)}_{dir}$ can be computed immediately while the indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$ can be computed only after having determined the variational functions $\delta {\phi }^{\left(1\right)}\left(x\right)$ and $\delta U\left(x\right)$. The variational function $\delta {\phi }^{\left(1\right)}\left(x\right)$ is the solution of the 2nd-LVSS represented by Equations (186) and (187). On the other hand, the function $\delta U\left(x\right)$ is the solution of the Equations (106) and (107). Altogether, these equations constitute the 2nd-LVSS for the two-component 2nd-level variational function $v^{\left(2\right)}\left(x\right)\triangleq {\left[\delta U\left(x\right),\delta {\phi }^{\left(1\right)}\left(x\right)\right]}^{†}$, cf. Equation (169). As before, the need for solving this 2nd-LVSS repeatedly, for every parameter variation of interest, is circumvented by deriving an alternative expression for the indirect-effect term defined in Equation (203), in which the function $v^{\left(2\right)}\left(x\right)\triangleq {\left[\delta U\left(x\right),\delta {\phi }^{\left(1\right)}\left(x\right)\right]}^{†}$ is replaced by a 2nd-level adjoint function that is independent of variations in the model parameter and state functions. This 2nd-level adjoint function will be a two-component function, which will be as denoted $a^{(2)}\left(3;x\right)\triangleq {\left[a_1^{(2)}\left(3;x\right),a_2^{(2)}\left(3;x\right)\right]}^{†}$, using the notation introduced in Section 4. The superscript “(2)” in $a^{(2)}\left(3;x\right)$ indicates “2nd-level” while the argument “3” indicates that this 2nd-level adjoint function corresponds to the sensitivity $\partial R/\partial Q$, which is the “third” first-order sensitivity to be considered. The Hilbert space appropriate for constructing the 2nd-LASS for the two-component function $a^{(2)}\left(3;x\right)$ is endowed with the particular form (two-component) of the inner product defined in Equation (171). The construction of this 2nd-LASS proceeds along the same steps as outlined in Section 4, as follows:

• Use Equation (171) to construct the inner product of Equations (106) and (186) with a yet unspecified function $a^{(2)}\left(3;x\right)$ to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;x\right)\left[\frac{d}{dx}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\\ =-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;x\right)\left[\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (220)
• Integrate by parts the terms on the left-side of Equation (220) to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;x\right)\left[\frac{d}{dx}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\\ =a_1^{(2)}\left(3;\mathcal{l}\right)\delta U\left(\mathcal{l}\right)-a_1^{(2)}\left(3;0\right)\delta U\left(0\right)-a_2^{(2)}\left(3;\mathcal{l}\right)\delta {\phi }^{\left(1\right)}\left(\mathcal{l}\right)\\ +a_2^{(2)}\left(3;0\right)\delta {\phi }^{\left(1\right)}\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\left[-\frac{d{a}_1^{(2)}\left(3;x\right)}{dx}+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[\frac{d{a}_2^{(2)}\left(3;x\right)}{dx}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}_2^{(2)}\left(3;y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\end{array}$$

  (221)
• The last two terms on the right-side of Equation (221) will be required to represent the indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$ defined in Equation (219). For this purpose, the second expression in the definition of the indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$ needs to be recast in the following form:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx{\phi }^{\left(1\right)}\left(x\right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}}={\left[{\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}{\displaystyle \underset{0}{\overset{x}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]}_0^{\mathcal{l}}\\ -{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)dx\left[{\displaystyle \underset{0}{\overset{x}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)dx}{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}.\end{array}$$

  (222)

  Replacing the result obtained in Equation (222) into Equation (219) yields the following expression indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$:

  $$\delta {\left(\partial R/\partial Q\right)}_{ind}\triangleq -{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(y\right)dy}{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}.$$

  (223)
• Express the indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$ in Equation (223) in terms of the 2nd-level adjoint sensitivity function $a^{(2)}\left(3;x\right)\triangleq {\left[a_1^{(2)}\left(3;x\right),a_2^{(2)}\left(3;x\right)\right]}^{†}$ by requiring this function to satisfy the following relations:

  $$-{\displaystyle \frac{d{a}_1^{(2)}\left(3;x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;y\right)dy}=-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}=-{\displaystyle \frac{1}{b}}\sin b\left(\mathcal{l}-x\right)\hspace{0.17em};$$

  (224)

  $${\displaystyle \frac{d{a}_2^{(2)}\left(3;x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}_2^{(2)}\left(3;y\right)dy}=-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}=-{\displaystyle \frac{U_0}{k_{0}b}}\sin \left(bx\right)\hspace{0.17em}.$$

  (225)
• Use in Equation (221) the boundary conditions provided in Equations (107) and (187), and eliminate the unknown remaining boundary terms by imposing the following boundary conditions of the function $a^{(2)}\left(3;x\right)\triangleq {\left[a_1^{(2)}\left(3;x\right),a_2^{(2)}\left(3;x\right)\right]}^{†}$:

  $$a_1^{(2)}\left(3;\mathcal{l}\right)=0;\hspace{0.5em}{a}_2^{(2)}\left(3;0\right)=0\hspace{0.17em}.$$

  (226)

  The NIDE-V net comprising Equations (224) –(226) constitutes the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for the 2nd-level adjoint function $a^{(2)}\left(3;x\right)$. Note that this 2nd-LASS is linear in $a^{(2)}\left(3;x\right)$ and is independent of parameter variations, so it needs to be solved only once to obtain the function $a^{(2)}\left(3;x\right)$.
• Use in Equation (220) the equations underlying the 2nd-LVSS for the function $v^{\left(2\right)}\left(x\right)\triangleq {\left[\delta U\left(x\right),\delta {\phi }^{\left(1\right)}\left(x\right)\right]}^{†}$ together with the equations underlying the 2nd-LASS for the function $a^{(2)}\left(3;x\right)$ to obtain the following expression for the indirect-effect term defined in Equation (223):

  $$\begin{array}{l}\delta {\left(\partial R/\partial Q\right)}_{ind}=\hspace{0.17em}{a}_1^{(2)}\left(3;0\right)\delta U_0-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ \hspace{1em}\hspace{1em}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;x\right)\left[\delta k_0\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}.\end{array}$$

  (227)
• Adding the expression obtained in Equation (227) with the expression for the direct-effect term defined in Equation (218) yields the following expression for the G-differential $\delta \left(\partial R/\partial U_0\right)$:

  $$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial Q}}\right)={\displaystyle \frac{\delta k_0}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}+a_1^{(2)}\left(3;0\right)\delta U_0+\left(\delta k_0\right)a_1^{(2)}\left(3;\mathcal{l}\right)\\ -2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy+{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}}\right]}\hspace{0.17em}dx.\end{array}$$

  (228)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• Inserting into Equation (228) the result obtained in Equation (98) and identifying in the resulting form of Equation (228) the expressions that multiply the respective parameter variations yields the expressions for the following second-order sensitivities of the decoder response:

  $${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial Q}}=\left(1+\beta T_0\right)a_1^{(2)}\left(3;0\right);$$

  (229)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial \beta \partial Q}}={\displaystyle \frac{{\left(T_0\right)}^2}{2}}{a}_1^{(2)}\left(3;0\right);$$

  (230)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial Q\hspace{0.17em}\partial Q}}=-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]}\hspace{0.17em}dx-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;x\right)\left[{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx};$$

  (231)

  $$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial Q}}={\displaystyle \frac{1}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}+a_1^{(2)}\left(3;\mathcal{l}\right)\\ +{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(3;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]}\hspace{0.17em}dx.+{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(3;x\right)\left[{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx.\end{array}$$

  (232)

The sensitivities obtained in Equations (229)–(232) can be computed after having determined the 2nd-level adjoint sensitivity function $a^{(2)}\left(3;x\right)$. In this particular case, the equations underlying the 2nd-LASS, comprising Equations (224)–(226), for the two components of $a^{(2)}\left(3;x\right)$ are decoupled and can therefore be solved independently of each other. Solving Equations (224) and (226) yields the following expression for the component $a_1^{(2)}\left(3;x\right)$ of the 2nd-level adjoint sensitivity function $a^{(2)}\left(3;x\right)$:

$$a_1^{(2)}\left(3;x\right)={\displaystyle \frac{x-\mathcal{l}}{2b\left(\theta \right)}}\sin \left(\mathcal{l}-x\right)b\left(\theta \right)={\displaystyle \frac{x-\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \left[\left(\mathcal{l}-x\right)\sqrt{{\displaystyle \frac{Q}{k_0}}}\right]\hspace{0.17em};$$

(233)

Solving Equations (225) and (226) yields the following expression for the component $a_2^{(2)}\left(3;x\right)$ of the 2nd-level adjoint sensitivity function $a^{(2)}\left(3;x\right)$:

$$a_2^{(2)}\left(3;x\right)=-{\displaystyle \frac{U_0}{2k_{0}b\left(\theta \right)}}x\sin xb\left(\theta \right)=-{\displaystyle \frac{U_0}{2\sqrt{Q{k}_0}}}x\sin x\sqrt{{\displaystyle \frac{Q}{k_0}}}\hspace{0.17em}.$$

(234)

Inserting into Equations (229)–(232) the expressions of obtained in Equations (233) and (234) together with the previously obtained expressions for the functions $U\left(x\right)$ and ${\phi }^{\left(1\right)}\left(x\right)$, and performing the respective integrations yields the following closed-form expressions for the second-order sensitivities stemming from the first-order sensitivity $\partial R/\partial Q$:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial Q}}=-\left(1+\beta T_0\right){\displaystyle \frac{\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right);$$

(235)

$${\displaystyle \frac{{\partial }^{2}R}{\partial \beta \partial Q}}=-{\displaystyle \frac{{\left(T_0\right)}^2\mathcal{l}}{4}}\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right);$$

(236)

$${\displaystyle \frac{{\partial }^{2}R}{\partial Q\hspace{0.17em}\partial Q}}={\displaystyle \frac{U_0\mathcal{l}}{4Q}}\left[\sqrt{{\displaystyle \frac{k_0}{Q}}}\sin \mathcal{l}b\left(\theta \right)-\mathcal{l}\cos \mathcal{l}b\left(\theta \right)\right]\hspace{0.17em};$$

(237)

$${\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial Q}}={\displaystyle \frac{U_0\mathcal{l}}{4}}\left[\left({\displaystyle \frac{\mathcal{l}}{k_0}}\right)\cos \mathcal{l}b\left(\theta \right)-{\displaystyle \frac{1}{\sqrt{Q{k}_0}}}\sin \mathcal{l}b\left(\theta \right)\right]\hspace{0.17em}.$$

(238)

5.1.4. Determining the Second-Order Sensitivities Stemming from $\partial R/\partial k_0$

The second-order sensitivities stemming from the first-order sensitivity $\partial R/\partial k_0$ are comprised in the G-differential of the expression provided in Equation (116), which yields, by definition, the following relation:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial k_0}}\right)=-{\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \frac{Q^0+\epsilon \delta Q}{{\left(k_0^0+\epsilon \delta k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}dx\left[{\phi }^{\left(1,0\right)}\left(x\right)+\epsilon \delta {\phi }^{\left(1\right)}\left(x\right)\right]{\displaystyle \underset{0}{\overset{x}{\int }}\left[U^0\left(y\right)+\epsilon \delta U\left(y\right)\right]dy}}\right\}}_{\epsilon =0}\\ \hspace{1em} +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}U\left(x\right)\delta \left(x-\mathcal{l}\right)dx}-\delta U_0=\delta {\left({\displaystyle \frac{\partial R}{\partial k_0}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial k_0}}\right)}_{ind},\end{array}$$

(239)

where the direct-effect term $\delta {\left(\partial R/\partial k_0\right)}_{dir}$ and the indirect-effect term $\delta {\left(\partial R/\partial k_0\right)}_{ind}$, respectively, are defined below:

$$\delta {\left(\partial R/\partial k_0\right)}_{dir}\triangleq \left[{\displaystyle \frac{\delta Q}{{\left(k_0\right)}^2}}-2{\displaystyle \frac{Q\left(\delta k_0\right)}{{\left(k_0\right)}^3}}\right]{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}-\delta U_0\hspace{0.17em};$$

(240)

$$\begin{array}{l}\delta {\left(\partial R/\partial k_0\right)}_{ind}\triangleq {\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]dx}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\delta \left(x-\mathcal{l}\right)dx}.\end{array}$$

(241)

The above expressions are to be evaluated at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation. The direct-effect term $\delta {\left(\partial R/\partial k_0\right)}_{dir}$ can be computed immediately while the indirect-effect term $\delta {\left(\partial R/\partial k_0\right)}_{ind}$ can be computed only after having determined the two-component variational function $v^{\left(2\right)}\left(x\right)\triangleq {\left[\delta U\left(x\right),\delta {\phi }^{\left(1\right)}\left(x\right)\right]}^{†}$, which is the solution of the 2nd-LVSS represented by Equations (106), (107), (186), and (187). As before, the need for solving this 2nd-LVSS repeatedly, for every parameter variation of interest, is circumvented by deriving an alternative expression for the indirect-effect term defined in Equation (241), in which the function $v^{\left(2\right)}\left(x\right)\triangleq {\left[\delta U\left(x\right),\delta {\phi }^{\left(1\right)}\left(x\right)\right]}^{†}$ is replaced by a 2nd-level adjoint function that is independent of variations in the model parameter and state functions. This 2nd-level adjoint function will be a two-component function, which will be as denoted $a^{(2)}\left(4;x\right)\triangleq {\left[a_1^{(2)}\left(4;x\right),a_2^{(2)}\left(4;x\right)\right]}^{†}$, using the notation introduced in Section 4. The superscript “(2)” in $a^{(2)}\left(4;x\right)$ indicates “2nd-level” while the argument “4” indicates that this 2nd-level adjoint function corresponds to the sensitivity $\partial R/\partial k_0$, which is the “fourth” first-order sensitivity to be considered. The Hilbert space appropriate for constructing the 2nd-LASS for the two-component function $a^{(2)}\left(4;x\right)$ is endowed with the particular form (two-component) of the inner product defined in Equation (171). The construction of this 2nd-LASS proceeds along the same steps as outlined in Section 5.1.3, as follows:

• Use Equation (171) to construct the inner product of Equations (106) and (186) with a yet unspecified function $a^{(2)}\left(4;x\right)$ to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(4;x\right)\left[\frac{d}{dx}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\\ =-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(4;x\right)\left[\left(\delta k_0\right)\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}\hspace{0.17em}.\end{array}$$

  (242)
• Integrate by parts the terms on the left-side of Equation (242) to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(4;x\right)\left[\frac{d}{dx}\delta U\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}\delta U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[-\frac{d}{dx}\delta {\phi }^{\left(1\right)}\left(x\right)+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(y\right)dy}\right]}\hspace{0.17em}dx\\ =a_1^{(2)}\left(4;\mathcal{l}\right)\delta U\left(\mathcal{l}\right)-a_1^{(2)}\left(4;0\right)\delta U\left(0\right)-a_2^{(2)}\left(4;\mathcal{l}\right)\delta {\phi }^{\left(1\right)}\left(\mathcal{l}\right)\\ +a_2^{(2)}\left(4;0\right)\delta {\phi }^{\left(1\right)}\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\left[-\frac{d{a}_1^{(2)}\left(4;x\right)}{dx}+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(4;y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[\frac{d{a}_2^{(2)}\left(4;x\right)}{dx}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}_2^{(2)}\left(4;y\right)dy}\right]}\hspace{0.17em}dx\hspace{0.17em}.\end{array}$$

  (243)
• The last two terms on the right-side of Equation (243) will be required to represent the indirect-effect term $\delta {\left(\partial R/\partial k_0\right)}_{ind}$ defined in Equation (241). For this purpose, the second expression in the definition of the indirect-effect term $\delta {\left(\partial R/\partial k_0\right)}_{ind}$ needs to be recast into the form presented in Equation (222). Replacing this form into Equation (241) yields the following expression indirect-effect term $\delta {\left(\partial R/\partial Q\right)}_{ind}$:

  $$\begin{array}{l}\delta {\left(\partial R/\partial k_0\right)}_{ind}\triangleq {\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta {\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)dx}{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta U\left(x\right)\delta \left(x-\mathcal{l}\right)dx}.\end{array}$$

  (244)
• Express the indirect-effect term $\delta {\left(\partial R/\partial k_0\right)}_{ind}$ in Equation (244) in terms of the 2nd-level adjoint sensitivity function $a^{(2)}\left(4;x\right)\triangleq {\left[a_1^{(2)}\left(4;x\right),a_2^{(2)}\left(4;x\right)\right]}^{†}$ by requiring this function to satisfy the following relations:

  $$-{\displaystyle \frac{d{a}_1^{(2)}\left(4;x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(4;y\right)dy}={\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}+\delta \left(x-\mathcal{l}\right);$$

  (245)

  $${\displaystyle \frac{d{a}_2^{(2)}\left(4;x\right)}{dx}}+b^2\left(\theta \right){\displaystyle \underset{0}{\overset{x}{\int }}{a}_2^{(2)}\left(4;y\right)dy}={\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\hspace{0.17em}.$$

  (246)
• Use in Equation (243) the boundary conditions provided in Equations (107) and (187), and eliminate the unknown remaining boundary terms by imposing the following boundary conditions of the function $a^{(2)}\left(4;x\right)\triangleq {\left[a_1^{(2)}\left(4;x\right),a_2^{(2)}\left(4;x\right)\right]}^{†}$:

  $$a_1^{(2)}\left(4;\mathcal{l}\right)=0;\hspace{1em}{a}_2^{(2)}\left(4;0\right)=0\hspace{0.17em}.$$

  (247)

  The NIDE-V system comprising Equations (245)–(247) constitutes the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for the 2nd-level adjoint function $a^{(2)}\left(4;x\right)$. This 2nd-LASS is linear in $a^{(2)}\left(4;x\right)$ and is independent of parameter variations, so it needs to be solved only once to obtain the function $a^{(2)}\left(4;x\right)$.
• Use in Equation (243) the equations underlying the 2nd-LVSS for the function $v^{\left(2\right)}\left(x\right)\triangleq {\left[\delta U\left(x\right),\delta {\phi }^{\left(1\right)}\left(x\right)\right]}^{†}$ together with the equations underlying the 2nd-LASS for the function $a^{(2)}\left(4;x\right)$ to obtain the following expression for the indirect-effect term defined in Equation (244):

  $$\begin{array}{l}\delta {\left(\partial R/\partial k_0\right)}_{ind}=\hspace{0.17em}{a}_1^{(2)}\left(4;0\right)\delta U_0-2b\left(\delta b\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]}\hspace{0.17em}dx\\ \hspace{1em}\hspace{1em}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_1^{(2)}\left(4;x\right)\left[\left(\delta k_0\right)\delta \left(x-\mathcal{l}\right)-2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}\right]dx}.\end{array}$$

  (248)
• Adding the expression obtained in Equation (248) with the expression for the direct-effect term defined in Equation (240) yields the following expression for the G-differential $\delta \left(\partial R/\partial U_0\right)$:

  $$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial k_0}}\right)=\left[{\displaystyle \frac{\delta Q}{{\left(k_0\right)}^2}}-2{\displaystyle \frac{Q\left(\delta k_0\right)}{{\left(k_0\right)}^3}}\right]{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}-\delta U_0+a_1^{(2)}\left(4;0\right)\delta U_0\\ -2\left(\delta b\right)b\left(\theta \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy+{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}}\right]}\hspace{0.17em}dx+\left(\delta k_0\right)a_1^{(2)}\left(4;\mathcal{l}\right).\end{array}$$

  (249)

  The above expression is to be satisfied at the nominal parameter values ${\theta }^0$ but this fact has not been indicated explicitly (using the superscript “zero”) in order to simplify the notation.
• Inserting into Equation (249) the result obtained in Equation (98) and identifying in the resulting form of Equation (249) the expressions that multiply the respective parameter variations yields the expressions for the following second-order sensitivities of the decoder response:

  $${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial k_0}}=-\delta U_0+a_1^{(2)}\left(4;0\right)\delta U_0=\left(1+\beta T_0\right)\left[a_1^{(2)}\left(4;0\right)-1\right];$$

  (250)

  $${\displaystyle \frac{{\partial }^{2}R}{\partial \beta \partial k_0}}={\displaystyle \frac{T_0^2}{2}}\left[a_1^{(2)}\left(4;0\right)-1\right]\hspace{0.17em};$$

  (251)

  $$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial Q\hspace{0.17em}\partial k_0}}={\displaystyle \frac{1}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}\\ \hspace{1em}\hspace{1em}-{\displaystyle \frac{1}{k_0}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy+{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}}\right]}\hspace{0.17em}dx;\end{array}$$

  (252)

  $$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial k_0}}=-2{\displaystyle \frac{Q}{{\left(k_0\right)}^3}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy}\right]dx}+a_1^{(2)}\left(4;\mathcal{l}\right)\\ \hspace{1em}\hspace{1em}+{\displaystyle \frac{Q}{{\left(k_0\right)}^2}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}_2^{(2)}\left(4;x\right)\left[{\displaystyle \underset{0}{\overset{x}{\int }}U\left(y\right)dy+{\displaystyle \underset{x}{\overset{\mathcal{l}}{\int }}{\phi }^{\left(1\right)}\left(y\right)dy}}\right]}\hspace{0.17em}dx.\end{array}$$

  (253)

The sensitivities obtained in Equations (250)–(253) can be computed after having determined the 2nd-level adjoint sensitivity function $a^{(2)}\left(4;x\right)$. In this particular case, the equations underlying the 2nd-LASS, comprising Equations (245)–(247), for the two components of $a^{(2)}\left(4;x\right)$ are decoupled and can therefore be solved independently of each other. Solving Equations (245) and (247) yields the following expression for the component $a_1^{(2)}\left(4;x\right)$ of the 2nd-level adjoint sensitivity function $a^{(2)}\left(4;x\right)$:

$$a_1^{(2)}\left(4;x\right)={\displaystyle \frac{Q}{2k_{0}b\left(\theta \right)}}\left(x-\mathcal{l}\right)\sin \left[\left(x-\mathcal{l}\right)b\left(\theta \right)\right]+\left[1-H\left(x-\mathcal{l}\right)\right]\cos \left[\left(\mathcal{l}-x\right)b\left(\theta \right)\right].$$

(254)

Solving Equations (246) and (247) yields the following expression for the component $a_2^{(2)}\left(4;x\right)$ of the 2nd-level adjoint sensitivity function $a^{(2)}\left(4;x\right)$:

$$a_2^{(2)}\left(4;x\right)={\displaystyle \frac{Q{U}_0\left(\theta \right)}{2b\left(\theta \right){\left(k_0\right)}^2}}x\sin \left[xb\left(\theta \right)\right]\hspace{0.17em}.$$

(255)

Inserting into Equations (250)–(253) the expressions of obtained in Equations (254) and (255) together with the previously obtained expressions for the functions $U\left(x\right)$ and ${\phi }^{\left(1\right)}\left(x\right)$, and performing the respective integrations yields the following closed-form expressions for the second-order sensitivities stemming from the first-order sensitivity $\partial R/\partial k_0$:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial k_0}}=\left(1+\beta T_0\right)\left[{\displaystyle \frac{\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)+\cos \mathcal{l}b\left(\theta \right)-1\right]\hspace{0.17em};$$

(256)

$${\displaystyle \frac{{\partial }^{2}R}{\partial \beta \partial k_0}}={\displaystyle \frac{T_0^2}{2}}\left[{\displaystyle \frac{\mathcal{l}}{2}}\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)+\cos \mathcal{l}b\left(\theta \right)-1\right]\hspace{0.17em};$$

(257)

$${\displaystyle \frac{{\partial }^{2}R}{\partial Q\hspace{0.17em}\partial k_0}}={\displaystyle \frac{\mathcal{l}{U}_0\left(\theta \right)}{4}}\left[\left({\displaystyle \frac{\mathcal{l}}{k_0}}\right)\cos \mathcal{l}b\left(\theta \right)-{\displaystyle \frac{1}{\sqrt{Q{k}_0}}}\sin \mathcal{l}b\left(\theta \right)\right];$$

(258)

$${\displaystyle \frac{{\partial }^{2}R}{\partial k_0\partial k_0}}={\displaystyle \frac{\mathcal{l}{U}_0\left(\theta \right)}{4k_0}}\left[\sqrt{{\displaystyle \frac{Q}{k_0}}}\sin \mathcal{l}b\left(\theta \right)-{\displaystyle \frac{Q\mathcal{l}}{k_0}}\cos \mathcal{l}b\left(\theta \right)\right]\hspace{0.17em}.$$

(259)

5.2. Applying the 2nd-FASAM-NIDE-V Methodology to Compute Second-Order Sensitivities: Highlights

When applying the 2nd-FASAM-NIDE-V methodology, the “large-scale” computations occur when solving the 2nd-LASS to determine the respective 2nd-level adjoint function, since solving the 2nd-LASS involves inversion of differential and integral operators. After having obtained the 2nd-level adjoint function, the computation of the actual sensitivities is comparatively trivial, since it involves computing integrals by using quadrature formulas. As shown in Equation (178), the 2nd-LASS is a lower-triangular system of block-matrices for a 2-component 2nd-level adjoint function of the form $a^{(2)}\left(i;t\right)\triangleq {\left[a_1^{(2)}\left(i;t\right),a_2^{(2)}\left(i;t\right)\right]}^{†}$, the components of which are obtained by solving coupled subsystems of equations. Occasionally, however, the 2nd-LASS may be block-diagonal, so the components of the respective 2nd-level adjoint function are decoupled and can be determined independently of each other. This is the case for the 2nd-LASS developed for the 2nd-level adjoint function $a^{(2)}\left(3;x\right)\triangleq {\left[a_1^{(2)}\left(3;x\right),a_2^{(2)}\left(3;x\right)\right]}^{†}$, in Section 5.1.3, and for the 2nd-level adjoint function $a^{(2)}\left(4;x\right)\triangleq {\left[a_1^{(2)}\left(4;x\right),a_2^{(2)}\left(4;x\right)\right]}^{†}$ in Section 5.1.4, respectively. Occasionally, the 2nd-LASS may be “degenerate,” involving a 2nd-level adjoint function comprising a single-component, as has been the case for the functions $a^{(2)}\left(1;x\right)$ and, respectively, $a^{(2)}\left(2;x\right)$, in Section 5.1.1 and Section 5.1.2, respectively.

When applying the 2nd-FASAM-NIDE-V methodology, the 2nd-order unmixed sensitivities are obtained just once, using a distinct 2nd-level adjoint function for each unmixed 2nd-order sensitivity, as shown in Equation (194), Equation (211), Equation (231), and Equation (253), respectively. On the other hand, the 2nd-order mixed sensitivities are obtained twice, using distinct expressions involving distinct 2nd-level adjoint functions. For example, the sensitivity ${\partial }^{2}R/\partial Q\hspace{0.17em}\partial T_0$ is obtained using Equation (196) and/or Equation (229); the sensitivity ${\partial }^{2}R/\partial k_0\partial T_0$ is obtained using Equation (197) and/or Equation (250), the sensitivity ${\partial }^{2}R/\partial Q\hspace{0.17em}\partial k_0$ is obtained using Equation (232) and/or Equation (252); and so on. These mixed 2nd-order sensitivities are obtained as inexpensively as practicable, involving just quadrature formulas, since the needed 2nd-level adjoint functions are already available. Thus, computing the mixed 2nd-order sensitivities twice, using distinct 2nd-level adjoint functions, provides the most inexpensive path for the stringent mutual verification of the accuracy of the computations performed when solving the respective 2nd-LASS for determining the respective 2nd-level adjoint functions.

6. Discussion and Conclusions

This work commenced by introducing the First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Volterra-Type (1st-FASAM-NIDE-V) for computing most efficiently the exact expressions of the first-order sensitivities of NIDE-V decoder-responses with respect to the optimized NIDE-V weights/parameters. The computation of the first-order sensitivities of the decoder response requires a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIE-net. As the comparison presented Section 3.3 has indicated, it is useful to investigate the advantages/disadvantages of applying the 1st-FASAM-NIDE-V versus the 1st-FASAM-NODE, when the system under consideration admits alternative representations as either an ordinary differential system or as an integro-differential Volterra system.

This work has also presented the 2nd-FASAM-NIDE-V, which builds on the 1st-FASAM-NIDE-V by considering the first-order sensitivities as “system responses” and applying the general principles underlying the Nth-CASAM [28] to determine the second-order sensitivities by determining the first-order sensitivities of these “system responses” (thus determining the “first-order sensitivities of the first-order sensitivities”). It has been shown that the 2nd-FASAM-NIDE-V methodology yields the exact expressions of all of the 2nd-order sensitivities and computes them with unparalleled efficiency, needing only as many “large-scale” computations as there are non-zero 1st-order sensitivities. These “large-scale” computations occur when solving the 2nd-LASS that corresponds to the respective non-zero 1st-order sensitivity considered as a “system response.” The unparalleled efficiency of the 2nd-FASAM-NIDE-V methodology stems from the fact that it scales at most linearly with the number of features functions as opposed to scaling exponentially with the number of model parameters, as do all the other (deterministic or statistical) methods. It has been shown that the mixed 2nd-order sensitivities are computed twice, using distinct 2nd-level adjoint functions, which provides the most inexpensive path for the stringent mutual verification of the accuracy of the computations performed when solving the respective 2nd-LASS for determining the respective 2nd-level adjoint functions.

This work has also shown that the existence of a linear analog of a nonlinear system enables significant simplifications and computational savings by performing the sensitivity analysis on the linear analog rather than directly on the nonlinear model. Albeit relatively rare, such instances are worth exploiting.

The mathematical framework of the 2nd-FASAM-NIDE-V can be generalized for computing arbitrarily-high order sensitivities of decoder response with respect to the NIDE-V feature functions and/or model parameters by applying the principles underlying the Nth-FASAM [29], which entails determining the next-order sensitivities as the “first-order sensitivities of the current sensitivities considered as system responses.” Thus, the 3rd-order sensitivities are obtained as the “first-order sensitivities” of the 2nd-order sensitivities considered as “system responses,” and so on. Future work will examine the theoretical underpinnings and feasibility of adapting algorithms of “backpropagation-type” for computing high-order sensitivities of decoder response with respect to feature functions of model parameters, rather than directly with respect to the model’s weights/parameters, in order to maximize the efficiency and accuracy of computing sensitivities of order higher than first-order.