The First- and Second-Order Features Adjoint Sensitivity Analysis Methodologies for Fredholm-Type Neural Integro-Differential Equations: An Illustrative Application to a Heat Transfer Model—Part II

Abstract

This work illustrates the application of the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type” (1st-FASAM-NIDE-F) and the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type” (2nd-FASAM-NIDE-F) to a paradigm heat transfer model. This physically based heat transfer model has been deliberately constructed so that it can be represented either by a neural integro-differential equation of a Fredholm type (NIDE-F) or by a conventional second-order “neural ordinary differential equation (NODE)” while admitting exact closed-form solutions/expressions for all quantities of interest, including state functions and first-order and second-order sensitivities. This heat transfer model enables a detailed comparison of the 1st- and 2nd-FASAM-NIDE-F versus the recently developed 1st- and 2nd-FASAM-NODE methodologies, highlighting the considerations underlying the optimal choice for cases where the neural net of interest is amenable to using either of these methodologies for its sensitivity analysis. It is shown that the 1st-FASAM-NIDE-F methodology enables the most efficient computation of exactly determined first-order sensitivities of the decoder response with respect to the optimized NIDE-F parameters, requiring a single “large-scale” computation for solving the 1st-Level Adjoint Sensitivity System (1st-LASS), regardless of the number of weights/parameters underlying the NIDE-F decoder, hidden layers, and encoder. The 2nd-FASAM-NIDE-F methodology enables the computation, with unparalleled efficiency, of the second-order sensitivities of decoder responses with respect to the optimized/trained weights.

1. Introduction

The mathematical frameworks of the 1st-FASAM-NIDE-F (First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type) and the 2nd-FASAM-NIDE-F (Second Order Features Adjoint Sensitivity Analysis Methodology for Neural Integro-Differential Equations of Fredholm-Type) were introduced in “Part I” [1]. This work (Part II) presents illustrative applications of the 1st-FASAM-NIDE-F and the 2nd-FASAM-NIDE-F methodologies to a paradigm heat conduction model. Heat conduction processes occur in many fields and have been analyzed for a long time, as illustrated in [2,3,4,5,6,7,8,9,10]. The particular illustrative heat conduction model constructed for this work admits exact closed-form solutions/expressions for all quantities of interest, including state functions and first-order and second-order sensitivities. In addition, this illustrative model can be formulated either as a first-order “neural integro-differential equation of Fredholm-type” (NIDE-F [11,12,13,14,15,16]) or as a conventional second-order “neural ordinary differential equation” (“NODE” [17,18,19,20,21,22,23,24]). The mathematical framework of the “Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (2nd-FASAM-NODE)” was recently presented by Cacuci [25]. The 2nd-FASAM-NODE methodology includes the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (1st-FASAM-NODE).” By admitting equivalent formulations, either as a NIDE-F or as a NODE, the illustrative paradigm heat conduction model considered in this work makes it possible to compare the detailed, step-by-step, applications of the 1st-FASAM-NIDE-F versus the 1st-FASAM-NODE methodologies (for computing most efficiently the exact expressions of the first-order sensitivities of the decoder response with respect to the model parameters) and, subsequently, to compare the applications of the 2nd-FASAM-NIDE-F versus the 2nd-FASAM-NODE methodologies (for computing most efficiently the exact expressions of the second-order sensitivities of the decoder response with respect to the model parameters).

Section 2.1, below, presents the application of the 1st-FASAM-NIDE-F methodology for obtaining the exact expressions of the first-order sensitivities of the heat conduction model response with respect to the model parameters, using the model in its NIDE-F form. For comparison, Section 2.2 uses the model in its NODE form and illustrates the application of the 1st-FASAM-NODE methodology [25] for obtaining the exact expressions of the same first-order sensitivities (of the heat conduction model response with respect to the model parameters). Section 2.3 presents a comparative discussion that highlights the similarities and differences between the application of the 1st-FASAM-NIDE-F methodology versus the 1st-FASAM-NODE methodology.

Section 3.1 presents the application of the 2nd-FASAM-NIDE-F methodology to obtain the exact expressions of the second-order response sensitivities to model parameters. The alternative application of the 2nd-FASAM-NODE methodology [25] to obtain the second-order response sensitivities to model parameters is presented in Section 3.2. The discussion presented in Section 4 highlights the unparallelled efficiency of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies for computing first- and second-order sensitivities of model responses to model parameters. Section 5 concludes this work by noting that the heat conduction model analyzed in this work provides a paradigm example that can be followed not only for applications in heat conduction processes but also for applications in all fields involving neural integro-differential equations of the Fredholm type.

2. Illustrative Application of the 1st-FASAM-NIDE-F Versus the 1st-FASAM-NODE Methodologies to a Heat Transfer Model

The application of the 1st-FASAM-NIDE-F methodology [1] will be illustrated in this Section by considering a model of linear steady-state heat conduction through a homogeneous slab of thickness $\mathcal{l}$, having constant thermal conductivity denoted as $k$ and involving a distributed heat source that is proportional to the temperature distribution within the slab; the proportionality constant is denoted as $Q$. The slab is considered to be insulated on one side, which is held at a temperature $T_0$. The temperature distribution within the slab, denoted as $T\left(x\right)$, is thus modeled by the following linear heat conduction equation:

$${\displaystyle \frac{d^{2}T\left(x\right)}{d{x}^2}}+{\displaystyle \frac{Q}{k}}T\left(x\right)=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}T\left(0\right)=T_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{dT\left(x\right)}{dx}}\right\}}_{x=0}=0.$$

(1)

Consider that the model response of interest, denoted as $R\left(T\right)$, is the average temperature within the slab, which is defined as follows:

$$R\left(T\right)\triangleq {\displaystyle \frac{1}{\mathcal{l}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}dx.$$

(2)

The model’s primary parameters $k$, $Q$, and $T_0$ are subject to uncertainties, but their nominal/optimal values, denoted as $k^0,\hspace{0.17em}{Q}^0,\hspace{0.17em}{T}_0^0$, are considered to be known. These parameters are considered to be components of the following (column) “vector of model parameters”:

$$\theta \triangleq {\left(k,Q,T_0\right)}^{†}.$$

(3)

The solution of Equation (1) has the following expression:

$$T\left(x\right)=T_0\cos x\gamma (\theta );\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\gamma (\theta )\triangleq \sqrt{Q/k}\hspace{0.17em}.$$

(4)

The quantity $\gamma (\theta )$ is a “feature function” of the primary model parameters. Using in Equation (2) the result obtained in Equation (4) yields the following closed form expression for the model response:

$$R\left(T\right)\triangleq {\displaystyle \frac{T_0}{\mathcal{l}\gamma (\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(5)

At the nominal parameter values, the nominal value of the temperature distribution and of the average temperature response, respectively, have the following expressions:

$$T^0\left(x\right)=T_0^0\cos x{\gamma }^0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\gamma }^0\triangleq \gamma ({\theta }^0);$$

(6)

$$R^0\left(T^0\right)\triangleq {\displaystyle \frac{T_0^0}{\mathcal{l}{\gamma }^0}}\sin \mathcal{l}{\gamma }^0\hspace{0.17em}.$$

(7)

2.1. Applying the 1st-FASAM-NIDE-F Methodology to Obtain the First-Order Response Sensitivities to the Primary Model Parameters

The heat conduction equation presented in Equation (1) can be recast into the following equivalent NIDE-F form:

$${\displaystyle \frac{dT\left(x\right)}{dx}}+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}dx=T_0\gamma (\theta )\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}T\left(0\right)=T_0\hspace{0.17em}.$$

(8)

The first-order sensitivities of the response $R\left(T\right)$ will be determined by applying the general principles of the 1st-FASAM-NIDE-F methodology presented in [1], commencing by determining the first-order Gateaux- (G-) differential, denoted as $\delta R$, of the response $R\left(T\right)$, which is obtained by applying the definition of the G-differential to the response defined in Equation (2), as follows:

$$\delta R\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \frac{1}{\mathcal{l}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[T\left(x\right)+\epsilon \delta T\left(x\right)\right]}dx\right\}}_{\epsilon =0}={\displaystyle \frac{1}{\mathcal{l}}}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\hspace{0.17em}dx\hspace{0.17em}}.$$

(9)

The variation $\delta T\left(x\right)$ is the solution of the 1st-Level Variational Sensitivity System (1st-LVSS) obtained by G-differentiating Equation (8), which yields the following NIDE-F for arbitrary variations $\delta T\left(x\right)$ and $\delta \gamma (\theta )$ around the nominal values $T^0\left(x\right),\hspace{0.17em}{\theta }^0$:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{d\epsilon }}\left[{\displaystyle \frac{d\left(T^0+\epsilon \delta T\right)}{dx}}+{\left({\gamma }^0+\epsilon \delta \gamma \right)}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(T^0+\epsilon \delta T\right)}dx\right]\right\}}_{\epsilon =0}\\ ={\left\{{\displaystyle \frac{d}{d\epsilon }}\left(T_0^0+\epsilon \delta T_0\right)\left({\gamma }^0+\epsilon \delta \gamma \right)\left[\sin \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-\sin x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]\right\}}_{\epsilon =0};\\ {\left\{{\displaystyle \frac{d}{d\epsilon }}{\left[T^0\left(x\right)+\epsilon \delta T\left(x\right)\right]}_{x=0}\right\}}_{\epsilon =0}=\delta T_0\hspace{0.17em}.\end{array}$$

(10)

Performing the operations indicated in Equation (10) yields the following form for the 1st-LVSS:

$$\begin{array}{l}{\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}dx\right\}}_{\theta ={\theta }^0}=q^{\left(1\right)}\left(x\right),\\ q^{\left(1\right)}\left(x\right)\triangleq {\left\{\left[\left(\delta T_0\right)\gamma (\theta )+\delta \gamma (\theta )T_0\right]\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]\right\}}_{\theta ={\theta }^0}\\ +\delta \gamma (\theta ){\left\{T_0\gamma (\theta )\left[\mathcal{l}\cos \mathcal{l}\gamma (\theta )-x\cos x\gamma (\theta )\right]\right\}}_{\theta ={\theta }^0}-{\left\{2\delta \gamma (\theta )\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}dx\right\}}_{\theta ={\theta }^0};\end{array}$$

(11)

$${\left\{\delta T\left(x\right)\right\}}_{x=0}=\delta T_0\hspace{0.17em}.$$

(12)

For subsequent reference, it is noted that the solution of the above 1st-LVSS has the following expression:

$$\delta T\left(x\right)=\delta T_0\cos x\gamma (\theta )-\delta \gamma (\theta )T_{0}x\sin x\gamma (\theta )\hspace{0.17em}.$$

(13)

The 1st-LVSS would need to be solved repeatedly, using every possible parameter variation, in order to determine the corresponding value of the temperature variation $\delta T\left(x\right)$. These repeated computations can be avoided by eliminating the appearance of the variation $\delta T\left(x\right)$ in Equation (9); this aim can be achieved by deriving an alternative expression for the response variation $\delta R$ that would not involve the variation $\delta T\left(x\right)$. This alternative expression for $\delta R$ will be derived in terms of a first-level adjoint function to be obtained as the solution of the 1st-Level Adjoint Sensitivity System (1st-LASS) to be constructed next by applying the general principles of the 1st-FASAM-NIDE-F Methodology presented in [1]. The construction of this 1st-LASS will be performed in a Hilbert space dented as ${\mathrm{H}}_{\mathrm{1}}\left({\mathrm{\Omega }}_{\mathrm{x}}\right)$, ${\mathrm{\Omega }}_{\mathrm{x}}\triangleq \left\{x\in \left[0,\mathcal{l}\right]\right\}$, and endowed with an inner product of two elements ${\chi }_1^{\left(1\right)}\left(x\right)\in {\mathrm{H}}_{\mathrm{1}}\left({\mathrm{\Omega }}_{\mathrm{x}}\right)$ and ${\eta }_1^{\left(1\right)}\left(x\right)\in {\mathrm{H}}_{\mathrm{1}}\left({\mathrm{\Omega }}_{\mathrm{x}}\right)$, denoted as ${〈{\chi }_1^{\left(1\right)}\left(x\right),{\eta }_1^{\left(1\right)}\left(x\right)〉}_1$ and defined as follows:

$${〈{\chi }_1^{\left(1\right)}\left(x\right),{\eta }_1^{\left(1\right)}\left(x\right)〉}_1\triangleq {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }_1^{\left(1\right)}\left(x\right){\eta }_1^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}.$$

(14)

Forming the inner product of Equation (11) with a yet-undefined function $a^{\left(1\right)}\left(x\right)$ yields the following relation:

$${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}\hspace{0.17em}\left[{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right]dx+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}dx={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)q^{\left(1\right)}\left(x\right)}dx.$$

(15)

The relation obtained in Equation (15) is satisfied at the nominal/optimal parameter values, but this fact has not been explicitly indicated in order to simplify the notation. Integrating by parts the first term on the left-side of Equation (15) and rearranging the second term on the left-side of Equation (15) yields the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}dx{\displaystyle \frac{d}{dx}}\delta T\left(x\right)+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}dx=a^{\left(1\right)}\left(\mathcal{l}\right)\delta T\left(\mathcal{l}\right)\\ -a^{\left(1\right)}\left(0\right)\delta T\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}dx\left\{-{\displaystyle \frac{d{a}^{\left(1\right)}\left(x\right)}{dx}}+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}dx\right\}.\end{array}$$

(16)

The function $a^{\left(1\right)}\left(x\right)$ will now be determined as follows: (i) it is required that the last term on the right-side of Equation (16) be identical to the G-differential $\delta R$ defined in Equation (9), and (ii) the unknown quantity $\delta T\left(\mathcal{l}\right)$ is eliminated in Equation (16). Imposing these requirements yields the following NIDE-F system for the function $a^{\left(1\right)}\left(x\right)$:

$$-{\displaystyle \frac{d{a}^{\left(1\right)}\left(x\right)}{dx}}+{\gamma }^2(\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}dx={\displaystyle \frac{1}{\mathcal{l}}};$$

(17)

$$a^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

(18)

The NIDE-F-net represented by Equations (17) and (18) constitutes the 1st-Level Adjoint Sensitivity System (1st-LASS) for the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$. The 1st-LASS is satisfied at the nominal parameter values but this fact has not been explicitly indicated in order to simplify the notation.

Using Equations (15)–(18) in conjunction with Equation (9) yields the following alternative expression for the G-differential $\delta R$ in terms of $a^{\left(1\right)}\left(x\right)$:

$$\delta R={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)q^{\left(1\right)}\left(x\right)}dx+a^{\left(1\right)}\left(0\right)\delta T_0.$$

(19)

Using, in Equation (19), the expression provided for $q^{\left(1\right)}\left(x\right)$ in Equation (11) and identifying the expressions that multiply the variations $\delta T_0$ and $\delta \gamma (\theta )$ yields the following expressions for the first-order sensitivities of the response with respect to the initial condition $T_0$ and the feature function $\gamma (\theta )$, respectively:

$${\displaystyle \frac{\partial R}{\partial T_0}}=\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]}dx+a^{\left(1\right)}\left(0\right);$$

(20)

$$\begin{array}{l}{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}=T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\sin \mathcal{l}\gamma (\theta )-\sin x\gamma (\theta )\right]}dx\\ +T_0\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\mathcal{l}\cos \mathcal{l}\gamma (\theta )-x\cos x\gamma (\theta )\right]}dx-2\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)}dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}dx.\end{array}$$

(21)

The expressions obtained in Equations (20) and (21) can be evaluated after having determined the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$. Also, these expressions are to be evaluated using the nominal/optimal parameter values, but this fact has not been explicitly indicated in order to simplify the notation. Notably, the 1st-LASS is independent of parameter variations, so it needs to be solved only once to determine $a^{\left(1\right)}\left(x\right)$. The closed-form explicit expression of the solution of the 1st-LASS represented by Equations (17) and (18) is provided below:

$$a^{\left(1\right)}\left(x\right)=-{\displaystyle \frac{2\left(x-\mathcal{l}\right)}{\mathcal{l}\left[2+{\mathcal{l}}^2{\gamma }^2(\theta )\right]}}.$$

(22)

Using the expression obtained in Equation (22) into Equations (20) and (21), respectively, and performing the respective integrations yields the following closed-form expressions:

$${\displaystyle \frac{\partial R}{\partial T_0}}={\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}\gamma (\theta )}};$$

(23)

$${\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}={\displaystyle \frac{T_0}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{T_0}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(24)

As expected, the expressions obtained in Equations (23) and (24) coincide with the expressions that would be obtained by the direct differentiation of the expression for the model response $R\left(T\right)$ obtained in Equation (5) with respect to $T_0$ and $\gamma (\theta )$, respectively. Of course, the closed-form exact expression for the model response in terms of the model’s primary parameters and/or feature functions is not available in practice.

The sensitivities of the model response with respect to the primary parameters are obtained by using the result obtained in Equation (24) in conjunction with the following “chain-rule of differentiation”:

$${\displaystyle \frac{\partial R}{\partial Q}}={\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}{\displaystyle \frac{\partial \gamma (\theta )}{\partial Q}}={\displaystyle \frac{1}{2\sqrt{kQ}}}{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}};$$

(25)

$${\displaystyle \frac{\partial R}{\partial k}}={\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}{\displaystyle \frac{\partial \gamma (\theta )}{\partial k}}=-{\displaystyle \frac{1}{2k}}\sqrt{{\displaystyle \frac{Q}{k}}}{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\hspace{0.17em}.$$

(26)

2.2. Applying the 1st-FASAM-NODE Methodology to Obtain the First-Order Response Sensitivities to the Primary Model Parameters

The traditional form of the heat conduction model provided in Equation (1) is a neural ordinary differential equation (NODE) which can be analyzed directly by using the “First-Order Features Adjoint Sensitivity Analysis Methodology for Neural Ordinary Differential Equations (1st-FASAM-NODE) introduced by Cacuci [1]. The G-differential of Equation (1) yields the following 1st-LVSS in NODE-form satisfied by the temperature variation $\delta T\left(x\right)$:

$${\left\{{\displaystyle \frac{d^2}{d{x}^2}}\delta T\left(x\right)+{\gamma }^2(\theta )\delta T\left(x\right)\right\}}_{\theta ={\theta }^0}=-{\left\{2\delta \gamma (\theta )\gamma (\theta )T\left(x\right)\right\}}_{\theta ={\theta }^0}\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};$$

(27)

$$\delta T\left(0\right)=\delta T_0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}=0.$$

(28)

The 1st-LASS corresponding to the above 1st-LVSS is obtained by implementing the same steps as outlined in the previous Subsection, by using Equation (14) to construct the inner-product of a yet undetermined function $b^{\left(1\right)}\left(x\right)$ with Equation (27) to obtain the following relation:

$${\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}dx\left[{\displaystyle \frac{d^2}{d{x}^2}}\delta T\left(x\right)+{\gamma }^2(\theta )\delta T\left(x\right)\right]=-2\delta \gamma (\theta )\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}T\left(x\right)dx.$$

(29)

The relation obtained in Equation (29) is to be evaluated at the nominal/optimal parameter values, but this fact has not been explicitly indicated in order to simplify the notation.

Integrating by parts the first term on the left-side of Equation (29) yields the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}dx\left[{\displaystyle \frac{d^2}{d{x}^2}}\delta T\left(x\right)+{\gamma }^2(\theta )\delta T\left(x\right)\right]=b^{\left(1\right)}\left(\mathcal{l}\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}-b^{\left(1\right)}\left(0\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}\\ -{\left\{\delta T\left(x\right){\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=\mathcal{l}}+{\left\{\delta T\left(x\right){\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0}+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}\left[{\displaystyle \frac{d^2{b}^{\left(1\right)}\left(x\right)}{d{x}^2}}+{\gamma }^2(\theta )b^{\left(1\right)}\left(x\right)\right]dx.\end{array}$$

(30)

Identifying the last term on the right-side of Equation (30) with the G-differential $\delta R$ provided in Equation (9), using the conditions provided in Equation (28) and eliminating the unknown boundary values on the right-side of Equation (30) yields the following expression for the G-differential in terms of the function $b^{\left(1\right)}\left(x\right)$:

$$\delta R=-2\delta \gamma (\theta )\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}T\left(x\right)dx-\delta T_0{\left\{{\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0},$$

(31)

where the 1st-level adjoint sensitivity function $b^{\left(1\right)}\left(x\right)$ is the solution of the following 1st-Level Adjoint Sensitivity System (1st-LASS):

$${\displaystyle \frac{d^2{b}^{\left(1\right)}\left(x\right)}{d{x}^2}}+{\gamma }^2(\theta )b^{\left(1\right)}\left(x\right)={\displaystyle \frac{1}{\mathcal{l}}};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};$$

(32)

$$b^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=\mathcal{l}}=0.$$

(33)

Identifying the quantities that multiply the variations $\delta T_0$ and $\delta \gamma (\theta )$ in Equation (31) yields the following expressions for the sensitivities of the model response with respect to $T_0$ and $\gamma (\theta )$:

$${\displaystyle \frac{\partial R}{\partial T_0}}=-{\left\{{\displaystyle \frac{d{b}^{\left(1\right)}\left(x\right)}{dx}}\right\}}_{x=0}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}{\delta }^{\prime }\left(x\right)dx;$$

(34)

$${\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}=-2\gamma (\theta ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}T\left(x\right)dx.$$

(35)

The 1st-LASS can be readily solved to obtain the following expression for the 1st-level adjoint sensitivity function $b^{\left(1\right)}\left(x\right)$:

$$b^{\left(1\right)}\left(x\right)={\displaystyle \frac{1-\cos \left(x-\mathcal{l}\right)\gamma (\theta )}{\mathcal{l}{\gamma }^2(\theta )}}.$$

(36)

Using in Equations (34) and (35) the expression for $b^{\left(1\right)}\left(x\right)$ obtained above yields the following expressions:

$${\displaystyle \frac{\partial R}{\partial T_0}}={\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}\gamma (\theta )}};$$

(37)

$${\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}={\displaystyle \frac{T_0}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{T_0}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta ).$$

(38)

All of the results obtained in Equations (29)–(38) are to be evaluated at the nominal parameter values, but this fact has not been explicitly indicated in order to simplify the notation. As expected, the expression obtained in Equation (37) coincides with the expression obtained in Equation (23), while the expression obtained in Equation (38) coincides with the expression obtained in Equation (24).

2.3. Comparison: Applying the 1st-FASAM-NODE Methodology Versus Applying the 1st-FASAM-NIDE-F Methodology

In cases where the model can be expressed equivalently in either NODE or in NIE-F forms, such as shown in Equation (1) or Equation (8), respectively, it is important to highlight the similarities and differences between applying the 1st-FASAM-NODE methodology versus applying the 1st-FASAM-NIDE-F methodology for determining the first-order response sensitivities to the underlying model parameters. Even though the form of the 1st-LVSS produced by the NODE methodology, namely Equations (27) and (28), differs from the form of the 1st-LVSS produced by the NIDE-F methodology, namely Equations (11) and (12), the two forms are equivalent to each other, so the solutions of these 1st-LVSS are identical to each other, having the expression provided in Equation (13).

On the other hand, the 1st-LASS corresponding to the NODE heat conduction model is not equivalent to the 1st-LASS corresponding to the NIDE-F heat conduction model, so that the corresponding 1st-level adjoint sensitivity function $b^{\left(1\right)}\left(x\right)$ for the NODE-model, namely Equation (36), differs from the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$ for the NIDE-F heat conduction model, which is provided in Equation (22). Consequently, the expressions obtained in terms of the respective 1st-level adjoint sensitivity functions of the sensitivities of the model response with respect to the primary parameters and feature function for the NODE-representation, namely Equations (34) and (35), differ from those obtained for the NIDE-F representation, namely Equations (20) and (21). The structure of the 1st-LASS and expressions for sensitivities appear to be simpler in the NODE-representation than in the NIDE-F representation, but the choice of representation/framework will be largely influenced by the neural-net software available to the individual user. Ultimately, the final results for the first-order response sensitivities obtained in Equations (37) and (38) by treating the heat conduction model as a NODE-net, cf. Equation (1), are identical with the final results obtained in Equations (23) and (24) by having treated the heat conduction model as a NIDE-F-net, cf. Equation (8).

3. Illustrative Application of the 2nd-FASAM-NIDE Methodology Versus the 2nd-FASAM-NODE Methodology for Computing the Second-Order Response Sensitivities to Model Features and Parameters

When applying the 2nd-FASAM-NIDE-F methodology, the second-order sensitivities arise from the first-order sensitivities obtained in Equations (20) and (21). The derivations involved when applying the 2nd-FASAM-NIDE-F methodology [1] will be presented in Section 3.1, below. Alternatively, when applying the 2nd-FASAM-NODE methodology [10], the second-order sensitivities arise from the first-order sensitivities obtained in Equations (34) and (35). The derivations for this case will be presented in Section 3.2, below.

3.1. Application of the 2nd-FASAM-NIDE-F Methodology to Obtain the Exact Expressions of the Second-Order Response Sensitivities to Model Parameters

When applying the 2nd-FASAM-NIDE-F methodology [1], the second-order sensitivities arise from the first-order sensitivities obtained in Equations (20) and (21). The second-order sensitivities stemming from Equation (20) are derived in Section 3.1.1, while the second-order sensitivities arising from Equation (21) are derived in Section 3.1.2.

3.1.1. Second-Order Sensitivities Stemming from $\partial R/\partial T_0$ Defined in Equation (20)

The second-order sensitivities arising from the first-order sensitivity expressed by Equation (20) are provided by the G-differential of the respective expression of $\partial R/\partial T_0$, for arbitrary variations around the nominal parameter and function values (indicated by the use of the superscript “zero”). Using in Equation (20) the expression for $a^{\left(1\right)}\left(0\right)$ obtained from Equation (22) and applying the definition of the G-differential to the resulting expression yields the relation below:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial T_0}}\right)\triangleq \delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}={\left\{{\displaystyle \frac{d}{d\epsilon }}2{\left[2+{\mathcal{l}}^2{\left({\gamma }^0+\epsilon \delta \gamma \right)}^2\right]}^{-1}\right\}}_{\epsilon =0}\\ +{\left\{{\displaystyle \frac{d}{d\epsilon }}\left({\gamma }^0+\epsilon \delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]\left[\sin \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-\sin x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]} dx\right\}}_{\epsilon =0};\end{array}$$

(39)

where the expressions for the above direct-effect and, respectively, indirect-effect terms are shown below:

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{dir}=-{\left\{4\gamma {\mathcal{l}}^2\left(\delta \gamma \right){\left(2+{\mathcal{l}}^2{\gamma }^2\right)}^{-2}\right\}}_{{\theta }^0}\\ +\left(\delta \gamma \right){\left\{{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma +\gamma \mathcal{l}\cos \mathcal{l}\gamma -x\gamma \sin x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\right\}}_{{\theta }^0};\end{array}$$

(40)

$$\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}={\left\{\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)\delta a^{\left(1\right)}\left(x\right)}dx\right\}}_{{\theta }^0}$$

(41)

The direct-effect term can be evaluated immediately. The indirect-effect term depends on the variational function $\delta a^{\left(1\right)}\left(x\right)$, which is the solution of the 2nd-LVSS obtained by G-differentiating the 1st-LASS comprising Equations (17) and (18). By definition, the 2nd-LVSS is obtained as follows:

$${\left\{-{\displaystyle \frac{d}{dx}}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]\right\}}_{\epsilon =0}+{\left\{{\left({\gamma }^0+\epsilon \delta \gamma \right)}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]}dx\right\}}_{\epsilon =0}=0;$$

(42)

$$\delta a^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em}.$$

(43)

Performing the operations indicated in Equation (42) yields the following NIDE-F, to be evaluated at the nominal parameter values:

$$-{\displaystyle \frac{d}{dx}}\delta a^{\left(1\right)}\left(x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)dx}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)dx}\hspace{0.17em}.$$

(44)

Since the indirect-effect term only depends on the variational function $\delta a^{\left(1\right)}\left(x\right)$ but does not depend on the variational function $\delta T\left(x\right)$, the relations presented in Equations (43) and (44) constitute the 2nd-LVSS for the function $\delta a^{\left(1\right)}\left(x\right)$, which is dependent on parameter variations and would need to be solved anew for each parameter variation in interest.

The need for computing $\delta a^{\left(1\right)}\left(x\right)$ repeatedly, for every parameter variation in interest, can be avoided by expressing the indirect-effect term defined by Equation (41) in terms of a 2nd-level adjoint sensitivity function that is independent of parameter variations. This adjoint function will be denoted as $a^{\left(2\right)}\left(1;x\right)$, where the argument “1” indicates that this adjoint function corresponds to the first-order sensitivity $\partial R/\partial T_0$, which was chosen in this case to be the “first” 1st-order sensitivity to be considered. The 2nd-LASS to be satisfied by $a^{\left(2\right)}\left(1;x\right)$ will be constructed by applying the 2nd-FASAM-NIDE-F, which commences by forming the inner product of $a^{\left(2\right)}\left(1;x\right)$ with Equation (44), to obtain the following relation:

$$\begin{array}{l}-{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)\left[\frac{d}{dx}\delta a^{\left(1\right)}\left(x\right)\right]d}x+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(y\right)dy}\\ =-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em}.\end{array}$$

(45)

Integrating by parts the first term on the left-side of Equation (45) and reversing the order of integrations in the remaining terms yields the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}{a}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)dx}\right]d}x-a^{\left(2\right)}\left(1;\mathcal{l}\right)\delta a^{\left(1\right)}\left(\mathcal{l}\right)\\ +a^{\left(2\right)}\left(1;0\right)\delta a^{\left(1\right)}\left(0\right)=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em}.\end{array}$$

(46)

The first term on the left-side of Equation (46) is now required to represent the indirect-effect term defined in Equation (41), which yields the relation below:

$${\displaystyle \frac{d}{dx}}{a}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)dx}=\gamma \left(\sin \mathcal{l}\gamma -\sin x\gamma \right).$$

(47)

The unknown quantity $\delta a^{\left(1\right)}\left(0\right)$ is eliminated from Equation (46) by imposing the following condition:

$$a^{\left(2\right)}\left(1;0\right)=0.$$

(48)

Replacing the results obtained in Equations (43), (47) and (48) into Equation (46) yields the following alternative expression for the indirect-effect term:

$$\delta {\left({\displaystyle \frac{\partial R}{\partial T_0}}\right)}_{ind}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right)dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em},$$

(49)

where the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(1;x\right)$ is the solution of the 2nd-Level Adjoint Sensitivity System (2nd-LASS) comprising Equations (47) and (48). The 2nd-LASS is a NIDE-F net that does not depend on parameter variations and needs to be solved once only at the nominal parameter values; its solution, $a^{\left(2\right)}\left(1;x\right)$, is used in Equation (49).

Adding the expressions obtained in Equations (40) and (49) yields the following expression:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial T_0}}\right)=-4\gamma {\mathcal{l}}^2\left(\delta \gamma \right){\left(2+{\mathcal{l}}^2{\gamma }^2\right)}^{-2}\\ +\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left[\sin \mathcal{l}\gamma -\sin x\gamma +\gamma \mathcal{l}\cos \mathcal{l}\gamma -x\gamma \sin x\gamma \right]} dx\\ -2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em}\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta \gamma .\end{array}$$

(50)

It follows from Equation (50) that

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}=0;\\ {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma +\gamma \mathcal{l}\cos \mathcal{l}\gamma -x\gamma \sin x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\\ -2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(1;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em}-4\gamma {\mathcal{l}}^2{\left(2+{\mathcal{l}}^2{\gamma }^2\right)}^{-2}.\end{array}$$

(51)

The 2nd-LASS represented by Equations (47) and (48) can be solved to obtain the following closed-form expression, to be evaluated at the nominal parameter values, for its solution:

$$a^{\left(2\right)}\left(1;x\right)={\displaystyle \frac{{\gamma }^2\mathcal{l}}{1+{\gamma }^2{\mathcal{l}}^2/2}}x+\cos \gamma x-1.$$

(52)

Inserting the above expression for $a^{\left(2\right)}\left(1;x\right)$ into Equation (51) and performing the respective integrations yields the following closed-form expression for the mixed second-order sensitivity:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}={\displaystyle \frac{1}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(53)

The validity of the above expression can be readily verified by taking the appropriate derivative of either of the first-order sensitivities provided in Equations (37) and (38).

3.1.2. Second-Order Sensitivities Stemming from $\partial R/\partial \gamma (\theta )$ Expressed by Equation (21)

The second-order sensitivities arising from the first-order sensitivity $\partial R/\partial \gamma (\theta )$ expressed by Equation (21) are provided by the G-differential of the respective expression for arbitrary variations around the nominal parameter and function values (indicated by the use of the superscript “zero”). By definition, this G-differential is obtained as follows:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)\triangleq \delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{ind}\triangleq \left\{{\displaystyle \frac{d}{d\epsilon }}\left(T_0^0+\epsilon \delta T_0\right)\right.{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]}\hspace{0.17em}\\ {\left.\times \left[\sin \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-\sin x\left({\gamma }^0+\epsilon \delta \gamma \right)\right] dx\right\}}_{\epsilon =0}+\left\{{\displaystyle \frac{d}{d\epsilon }}\left(T_0^0+\epsilon \delta T_0\right)\right.\left({\gamma }^0+\epsilon \delta \gamma \right)\\ {\left.\times {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]\left[\mathcal{l}\cos \mathcal{l}\left({\gamma }^0+\epsilon \delta \gamma \right)-x\cos x\left({\gamma }^0+\epsilon \delta \gamma \right)\right]} dx\right\}}_{\epsilon =0}\\ -2{\left\{{\displaystyle \frac{d}{d\epsilon }}\left({\gamma }^0+\epsilon \delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[a^{\left(1,0\right)}\left(x\right)+\epsilon \delta a^{\left(1\right)}\left(x\right)\right]} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[T^0\left(x\right)+\epsilon \delta T\left(x\right)\right]} dx\right\}}_{\epsilon =0},\end{array}$$

(54)

where the direct-effect and, respectively, indirect-effect terms have the following expressions:

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{dir}=\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}{\gamma }^0-\sin x{\gamma }^0\right)a^{\left(1,0\right)}\left(x\right)} dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}{\gamma }^0-x\cos x{\gamma }^0\right)a^{\left(1,0\right)}\left(x\right)} dx\\ +\left(T_0^0\delta \gamma +{\gamma }^0\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}{\gamma }^0-x\cos x{\gamma }^0\right)a^{\left(1,0\right)}\left(x\right)} dx\\ +T_0^0{\gamma }^0\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(x^2\sin x{\gamma }^0-{\mathcal{l}}^2\sin \mathcal{l}{\gamma }^0\right)}{a}^{\left(1,0\right)}\left(x\right)\hspace{0.17em}dx-2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1,0\right)}\left(x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{T}^0\left(x\right)} dx;\end{array}$$

(55)

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{ind}=-2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1,0\right)}\left(x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{T}^0\left(x\right)} dx-2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1,0\right)}\left(x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)} dx\hspace{0.17em}\\ +T_0^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}{\gamma }^0-\sin x{\gamma }^0\right)\delta a^{\left(1,0\right)}\left(x\right)} dx+T_0^0{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}{\gamma }^0-x\cos x{\gamma }^0\right)\delta a^{\left(1,0\right)}\left(x\right)} dx.\end{array}$$

(56)

The variational function $\delta T\left(x\right)$ is the solution of Equations (11) and (12) while the variational function $\delta a^{\left(1\right)}\left(x\right)$ is the solution of Equations (43) and (44). Altogether, these four equations constitute the 2nd-LVSS for the two-component vector-valued variational function $V^{\left(2\right)}\left(2;x\right)\hspace{0.17em}\triangleq {\left[\delta T\left(x\right),\delta a^{\left(1\right)}\left(x\right)\right]}^{†}$.

The need for repeatedly solving the above 2nd-LVSS for all parameter variations in interest can be circumvented by eliminating the appearance of $V^{\left(2\right)}\left(2;x\right)\hspace{0.17em}\triangleq {\left[\delta T\left(x\right),\delta a^{\left(1\right)}\left(x\right)\right]}^{†}$ in the expression of the indirect-effect term defined in Equation (56). This elimination is achieved by deriving an alternative expression for the indirect-effect term, using the solution of the 2nd-LASS to be constructed as follows:

1.

Consider a two-component vector function denoted as $A^{(2)}\left(2;2;x\right)\triangleq {\left[a^{(2)}\left(1;2;x\right),a^{(2)}\left(2;2;x\right)\right]}^{†}$, where the first argument denotes the component number and the second argument (“2”) indicates that this function will correspond to the “second” first-order sensitivity $\partial R/\partial \gamma (\theta )$. The construction of this 2nd-LASS will be performed in a Hilbert space denoted as ${\mathrm{H}}_{\mathrm{2}}\left({\mathrm{\Omega }}_{\mathrm{2}}\right)$, comprising elements of the same form as $A^{(2)}\left(2;2;x\right)\in {\mathrm{H}}_{\mathrm{2}}\left({\mathrm{\Omega }}_{\mathrm{2}}\right)$, and endowed with the following inner product, denoted as ${〈{\chi }^{(1)}\left(t\right),{\eta }^{(1)}\left(t\right)〉}_2$, between two elements ${\chi }^{(1)}\left(x\right)\triangleq {\left[{\chi }_1^{\left(1\right)}\left(x\right),{\chi }_2^{\left(1\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\mathrm{2}}$ and ${\eta }^{(1)}\left(x\right)\triangleq {\left[{\eta }_1^{\left(1\right)}\left(x\right),{\eta }_2^{\left(1\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\mathrm{2}}$:

$${〈{\chi }^{(1)}\left(x\right),{\eta }^{(1)}\left(x\right)〉}_2\triangleq {\displaystyle \sum _{i=1}^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{\chi }_i^{\left(1\right)}\left(x\right){\eta }_i^{\left(1\right)}\left(x\right) dx}}.$$

(57)

2.

Using the inner product defined in Equation (57), construct the inner product of $A^{(2)}\left(2;2;x\right)\triangleq {\left[a^{(2)}\left(1;2;x\right),a^{(2)}\left(2;2;x\right)\right]}^{†}$ with Equations (11) and (44), respectively, to obtain the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)\frac{d}{dx}\delta T\left(x\right)} dx+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)} dx\\ -{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)\frac{d}{dx}\delta a^{\left(1\right)}\left(x\right)} dx+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right) dx}\\ ={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)} dx-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right) dx}\hspace{0.17em}.\end{array}$$

(58)

3.

Integrate by parts the first and third terms on the left-side of Equation (58) and rearrange the resulting expressions to obtain the following relation:

$$\begin{array}{l}{a}^{(2)}\left(1;2;\mathcal{l}\right)\delta T\left(\mathcal{l}\right)-a^{(2)}\left(1;2;0\right)\delta T\left(0\right)-a^{(2)}\left(2;2;\mathcal{l}\right)\delta a^{\left(1\right)}\left(\mathcal{l}\right)\\ +a^{(2)}\left(2;2;0\right)\delta a^{\left(1\right)}\left(0\right)+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)\left[-\frac{d}{dx}{a}^{(2)}\left(1;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)}\right]} dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta a^{\left(1\right)}\left(x\right)\left[\frac{d}{dx}{a}^{(2)}\left(2;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)}dx\right]} dx\\ ={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)} dx-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right) dx}\hspace{0.17em}.\end{array}$$

(59)

4.

Require the last two terms (involving integrals over $\delta T\left(x\right)$ and $\delta a^{\left(1\right)}\left(x\right)$, respectively) on the left-side of Equation (59) to represent the indirect-effect term defined in Equation (56) by imposing the following relations:

$$-{\displaystyle \frac{d}{dx}}{a}^{(2)}\left(1;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)}=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)} dx\hspace{0.17em};$$

(60)

$$\begin{array}{l}{\displaystyle \frac{d}{dx}}{a}^{(2)}\left(2;2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)} dx\\ +T_0\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)+T_0\gamma \left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right).\end{array}$$

(61)

5.

Eliminate the unknown quantities $\delta T\left(\mathcal{l}\right)$ and $\delta a^{\left(1\right)}\left(0\right)$ on the left-side of Equation (59) by imposing the following boundary conditions:

$$a^{(2)}\left(1;2;\mathcal{l}\right)=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{a}^{(2)}\left(2;2;0\right)=0.$$

(62)

6.

Insert the boundary conditions represented by Equations (12) and (43) into Equation (59) and use the relations underlying the 2nd-LASS to obtain the following expression for the indirect-effect term defined in Equation (56):

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)}_{ind}=a^{(2)}\left(1;2;0\right)\delta T_0+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)} dx\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right) dx}\hspace{0.17em}.\end{array}$$

(63)

Add the expression obtained in Equation (63) to the expression of the direct-effect term provided in Equation (55) to obtain the expression below for the G-differential $\delta \left(\partial R/\partial \gamma (\theta )\right)$:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right)\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}\delta \gamma =a^{(2)}\left(1;2;0\right)\delta T_0+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)q^{\left(1\right)}\left(x\right)} dx\\ -2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right) dx}+\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\\ +\left(T_0\delta \gamma +\gamma \delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\\ +T_0\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(x^2\sin x\gamma -{\mathcal{l}}^2\sin \mathcal{l}\gamma \right)}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx-2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)} dx\hspace{0.17em}.\end{array}$$

(64)

Insert the expression of $q^{\left(1\right)}\left(x\right)$ into the second term on the right-side of Equation (64) and collect the terms multiplying the variations $\delta T_0$ and $\delta \gamma$, respectively, to obtain the following expressions:

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}=a^{(2)}\left(1;2;0\right)+\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)a^{(2)}\left(1;2;x\right)} dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)a^{\left(1\right)}\left(x\right)} dx+\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)} dx;\end{array}$$

(65)

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(1;2;x\right)\left[T_0\left(\sin \mathcal{l}\gamma -\sin x\gamma \right)+T_0\gamma \left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)}dx\right]} dx\\ -2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{(2)}\left(2;2;x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right) dx}+T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\\ -2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)} dx{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}T\left(x\right)} dx\hspace{0.17em}+T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)a^{\left(1\right)}\left(x\right)} dx\\ +T_0\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(x^2\sin x\gamma -{\mathcal{l}}^2\sin \mathcal{l}\gamma \right)}{a}^{\left(1\right)}\left(x\right)\hspace{0.17em}dx.\end{array}$$

(66)

The algebraic manipulations involved in obtaining the closed-form expressions of the second-order sensitivities presented in Equations (65) and (66) are straightforward but involve a large amount of algebra stemming from the fact that the 2nd-LASS involves the two-component 2nd-level adjoint sensitivity function $A^{(2)}\left(2;2;x\right)\triangleq {\left[a^{(2)}\left(1;2;x\right),a^{(2)}\left(2;2;x\right)\right]}^{†}$. The reason for needing such a two-component adjoint function stems from the expression of the first-order sensitivity $\partial R/\partial \gamma (\theta )$ provided in Equation (21), which involves both the original function $T\left(x\right)$ and the 1st-level adjoint sensitivity function $a^{\left(1\right)}\left(x\right)$. A significant amount of algebraic manipulations could be avoided by eliminating the appearance of either $T\left(x\right)$ or $a^{\left(1\right)}\left(x\right)$ in the expression of $\partial R/\partial \gamma (\theta )$. If either of these functions were eliminated from appearing in the expression of $\partial R/\partial \gamma (\theta )$, then the G-differential of $\partial R/\partial \gamma (\theta )$ would depend either just on $\delta a^{\left(1\right)}$ or just on $\delta T$, which are “single-component” (as opposed to “two-components”) variational sensitivity functions. In such a case, the corresponding 2nd-LASS would also comprise just a single-component (as opposed to “two-components”) 2nd-level adjoint sensitivity function. These considerations will be illustrated in the following by using Equation (4) to eliminate the appearance of the function $T\left(x\right)$ in the expression provided in Equation (21) for $\partial R/\partial \gamma (\theta )$, which would consequently take on the following simplified expression:

$${\displaystyle \frac{\partial R}{\partial \gamma }}=T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)} dx.$$

(67)

Applying the definition of the G-differential to Equation (67) yields the following expression:

$$\delta \left\{{\displaystyle \frac{\partial R}{\partial \gamma }}\right\}=\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{dir}+\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind},$$

(68)

where the direct-effect and the indirect-effect terms are defined below:

$$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{dir}=\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)} dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(-{\mathcal{l}}^2\gamma \sin \mathcal{l}\gamma +\mathcal{l}\cos \mathcal{l}\gamma +x^2\gamma \cos x\gamma -x\cos x\gamma -\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)} dx\end{array}$$

(69)

$$\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind}=T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)\delta a^{\left(1\right)}\left(x\right)} dx\hspace{0.17em}.$$

(70)

The appearance in Equation (70) of the variational function $\delta a^{\left(1\right)}\left(x\right)\hspace{0.17em}$ is eliminated by following the same procedure as followed in the foregoing for the indirect-effect term $\delta {\left(\partial R/\partial T_0\right)}_{ind}$. Ultimately, the indirect-effect term $\delta {\left(\partial R/\partial \gamma \right)}_{ind}$ will have the following expression in terms of a 2nd-level adjoint sensitivity function denoted as $a^{\left(2\right)}\left(2;x\right)$:

$$\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind}=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right)dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em},$$

(71)

where the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(2;x\right)$ is the solution of the following 2nd-LASS:

$${\displaystyle \frac{d}{dx}}{a}^{\left(2\right)}\left(2;x\right)+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right)dx}=T_0\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right);$$

(72)

$$a^{\left(2\right)}\left(2;0\right)=0.$$

(73)

Adding the expressions obtained in Equations (69) and (71) yields the following expression for the G-differential $\delta \left\{\partial R/\partial \gamma \right\}$:

$$\begin{array}{l}\delta \left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)=\left(\delta T_0\right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)} dx\\ +\left(\delta \gamma \right)T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(-{\mathcal{l}}^2\gamma \sin \mathcal{l}\gamma +\mathcal{l}\cos \mathcal{l}\gamma +x^2\gamma \cos x\gamma -x\cos x\gamma -\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)} dx\\ -2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\hspace{0.17em}\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}\delta \gamma .\end{array}$$

(74)

It follows from Equation (74) that the respective second-order sensitivities have the following expressions:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(\mathcal{l}\gamma \cos \mathcal{l}\gamma -x\gamma \cos x\gamma -\sin \mathcal{l}\gamma -\sin x\gamma \right)} dx={\displaystyle \frac{1}{\gamma }}\cos \mathcal{l}\gamma -{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2}}\sin \mathcal{l}\gamma \hspace{0.17em}.$$

(75)

$$\begin{array}{l}{\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \partial \gamma }}=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(2\right)}\left(2;x\right) dx}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(y\right)dy}\\ +T_0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{a}^{\left(1\right)}\left(x\right)\left(-{\mathcal{l}}^2\gamma \sin \mathcal{l}\gamma +\mathcal{l}\cos \mathcal{l}\gamma +x^2\gamma \cos x\gamma -x\cos x\gamma -\mathcal{l}\cos \mathcal{l}\gamma -x\cos x\gamma \right)} dx.\end{array}$$

(76)

The mixed second-order sensitivity ${\partial }^{2}R/\partial T_0\delta \gamma$ in Equation (75) does not depend on the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(2;x\right)$ and is therefore evaluated immediately. Solving Equations (72) and (73) yields the following expression, to be evaluated at the nominal parameter values, for the 2nd-level adjoint sensitivity function $a^{\left(2\right)}\left(2;x\right)$:

$$a^{\left(2\right)}\left(2;x\right)=-T_{0}x\sin \gamma x.$$

(77)

Inserting the result obtained in Equation (77) into Equation (76) and performing the respective operations yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial \gamma (\theta )\partial \gamma (\theta )}}=T_0\left(2{\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}{\gamma }^3(\theta )}}-2{\displaystyle \frac{\cos \mathcal{l}\gamma (\theta )}{{\gamma }^2(\theta )}}-{\displaystyle \frac{\mathcal{l}\sin \mathcal{l}\gamma (\theta )}{\gamma (\theta )}}\right).$$

(78)

It is evident from Equations (51) and (65) or, alternatively, Equation (78) that the mixed second-order sensitivity ${\partial }^{2}R/\partial T_0\delta \gamma$ is computed twice, employing distinct expressions involving distinct 2nd-level adjoint sensitivity functions. This mechanism provides a stringent verification of the accuracy of the computation of the respective adjoint sensitivity functions.

In practice, the closed-form analytical expressions of the original functions, such as provided in Equation (4), are seldom available. Nevertheless, if such expressions are available, they can be advantageously used to reduce the amount of computations involved in determining the response sensitivities, as was shown in the foregoing.

3.2. Application of the 2nd-FASAM-NODE Methodology to Obtain the Second-Order Response Sensitivities to Model Parameters

When applying the 2nd-FASAM-NODE methodology [10], the second-order sensitivities arise from the first-order sensitivities obtained in Equations (34) and (35), respectively. The corresponding derivations are presented below, in Section 3.2.1 and Section 3.2.2, respectively.

3.2.1. Second-Order Sensitivities Stemming from $\partial R/\partial T_0$ Defined in Equation (34)

Thus, the second-order sensitivities arising from the first-order sensitivity $\partial R/\partial T_0$ expressed in Equation (34) are provided by its G-differential for arbitrary variations around the nominal parameter and function values (indicated by the use of the superscript “zero”), which is by definition obtained as follows:

$$\delta \left\{{\displaystyle \frac{\partial R}{\partial T_0}}\right\}\triangleq {\left\{{\displaystyle \frac{d}{d\epsilon }}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{\left(1,0\right)}\left(x\right)+\epsilon \delta b^{\left(1\right)}\left(x\right)\right]}{\delta }^{\prime }\left(x\right)dx\right\}}_{\epsilon =0}={\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)}{\delta }^{\prime }\left(x\right) dx,$$

(79)

where ${\delta }^{\prime }\left(x\right)$ denotes the derivative of the Dirac-delta functional. The variational function $\delta b^{\left(1\right)}\left(x\right)$ is the solution of the following 2nd-LVSS, obtained by G-differentiating Equations (32) and (33):

$${\displaystyle \frac{d^2}{d{x}^2}}\delta b^{\left(1\right)}\left(x\right)+{\gamma }^2\delta b^{\left(1\right)}\left(x\right)=-2\gamma \left(\delta \gamma \right)b^{\left(1\right)}\left(x\right);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<x<\mathcal{l};$$

(80)

$$\delta b^{\left(1\right)}\left(\mathcal{l}\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d}{dx}}\left[\delta b^{\left(1\right)}\left(x\right)\right]\right\}}_{x=\mathcal{l}}=0.$$

(81)

The above 2nd-LVSS is to be satisfied by the function $\delta b^{\left(1\right)}\left(x\right)$ at the nominal parameter values, but the superscript “zero” (which has been used to denote this fact) has been omitted to simplify the notation.

The need for repeatedly solving the above 2nd-LVSS for all parameter variations in interest is circumvented by eliminating the appearance of $\delta b^{\left(1\right)}\left(x\right)$ in Equation (79). This aim will be accomplished by expressing $\delta \left\{\partial R/\partial T_0\right\}$ in terms of the solution of the 2nd-LASS to be constructed by considering an adjoint function that will be denoted as $b^{\left(2\right)}\left(1;x\right)$, where the argument “1” indicates that this adjoint function corresponds to the first-order sensitivity $\partial R/\partial T_0$, which is chosen in this case to be the “first” 1st-order sensitivity to be considered. The 2nd-LASS to be satisfied by $b^{\left(2\right)}\left(1;x\right)$ will be constructed by applying the 2nd-FASAM-NODE [10], which commences by using Equation (14) to form the inner product of $b^{\left(2\right)}\left(1;x\right)$ with Equation (80), to obtain the following relation:

$$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)\left[\frac{d^2}{d{x}^2}\delta b^{\left(1\right)}\left(x\right)\right]d}x+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)\delta b^{\left(1\right)}\left(x\right) dx}\\ =-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right) dx}\hspace{0.17em}.\end{array}$$

(82)

Integrating by parts the first term on the left-side of Equation (82) and rearranging the resulting terms yields the following relation:

$$\begin{array}{l}{b}^{\left(2\right)}\left(1;\mathcal{l}\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(\mathcal{l}\right)-b^{\left(2\right)}\left(1;0\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(0\right)-\delta b^{\left(1\right)}\left(\mathcal{l}\right){\displaystyle \frac{d{b}^{\left(2\right)}\left(1;\mathcal{l}\right)}{dx}}+\delta b^{\left(1\right)}\left(0\right){\displaystyle \frac{d{b}^{\left(2\right)}\left(1;0\right)}{dx}}\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)\left[\frac{d^2}{d{x}^2}{b}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{b}^{\left(2\right)}\left(1;x\right)\right]d}x=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right) dx}\hspace{0.17em}.\end{array}$$

(83)

The last term on the left-side of Equation (83) is now required to represent the G-differential defined in Equation (79) to obtain the relation below:

$${\displaystyle \frac{d^2}{d{x}^2}}{b}^{\left(2\right)}\left(1;x\right)+{\gamma }^2{b}^{\left(2\right)}\left(1;x\right)={\delta }^{\prime }\left(x\right).$$

(84)

The unknown boundary terms are eliminated from Equation (83) by imposing the following conditions:

$$b^{\left(2\right)}\left(1;0\right)=0\hspace{0.17em};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d}{dx}}{b}^{\left(2\right)}\left(1;0\right)\right\}}_{x=0}=0.$$

(85)

The system of equations comprising Equations (84) and (85) constitute the 2nd-LASS for the 2nd-level adjoint sensitivity function $b^{\left(2\right)}\left(1;x\right)$.

Replacing the results obtained in Equations (81), (84) and (85) into Equation (83) yields the following alternative expression for the G-differential $\delta \left(\partial R/\partial T_0\right)$:

$$\delta \left({\displaystyle \frac{\partial R}{\partial T_0}}\right)=-2\gamma \left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx}\triangleq {\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}\delta T_0+{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}\delta \gamma .$$

(86)

where the 2nd-level adjoint sensitivity function $b^{\left(2\right)}\left(1;x\right)$ is the solution of the 2nd-Level Adjoint Sensitivity System (2nd-LASS) comprising Equations (84) and (85). The 2nd-LASS is a NODE-net that does not depend on parameter variations and needs to be solved once only at the nominal parameter values. The solution, $b^{\left(2\right)}\left(1;x\right)$, of this 2nd-LASS is used in Equation (86) to determine the respective second-order response sensitivities.

Identifying in Equation (86) the quantities that multiply the respective parameter variations yields the following expressions:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}=-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(2\right)}\left(1;x\right)b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx};\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\displaystyle \frac{{\partial }^{2}R}{\partial T_0\partial T_0}}=0.$$

(87)

Solving the 2nd-LASS represented by Equations (84) and (85) yields the following expression for the 2nd-level adjoint sensitivity function $b^{\left(2\right)}\left(1;x\right)$:

$$b^{\left(2\right)}\left(1;x\right)=H\left(x\right)\cos \gamma x,$$

(88)

where $H\left(x\right)$ denotes the Heaviside functional. Using in Equation (87) the results obtained in Equations (36) and (88) yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma (\theta )}}={\displaystyle \frac{1}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(89)

As expected, the expression obtained in Equation (89) is identical to the expressions obtained in Equations (53) and (75).

3.2.2. Second-Order Sensitivities Stemming from $\partial R/\partial \gamma (\theta )$ Expressed by Equation (35)

The second-order sensitivities arising from the first-order sensitivity $\partial R/\partial \gamma (\theta )$ represented by Equation (35) are obtained from its G-differential for arbitrary variations around the nominal parameter and function values. Thus, applying the definition of the G-differential to Equation (35) yields the following expression:

$$\delta \left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}\triangleq \delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{dir}+\delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{ind}$$

(90)

where the direct-effect and indirect-effect terms have the expressions below:

$$\delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{dir}\triangleq -2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}T\left(x\right) dx;$$

(91)

$$\delta {\left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}}_{ind}\triangleq -2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\delta T\left(x\right) dx-2{\gamma }^0{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)}T\left(x\right) dx.$$

(92)

The indirect-effect term $\delta {\left\{\partial R/\partial \gamma (\theta )\right\}}_{ind}$ will be recast in terms of an alternative expression that will not involve the variational functions $\delta T\left(x\right)$ and $\delta b^{\left(1\right)}\left(x\right)$ by applying the principles of the 2nd-FASAM-NODE [10], which are fundamentally the same as those underlying the 2nd-FASAM-NIDE-F [1], as follows:

• Consider the two-component vector function $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$, where the first argument denotes the component number and the second argument (“2”) indicates that this function will correspond to the “second” first-order sensitivity, namely $\partial R/\partial \gamma (\theta )$
  Using the inner product defined in Equation (57), construct the inner product of $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$ with Equations (27) and (80), respectively, to obtain the following relation, to be satisfied at the nominal parameter values (although the superscript “zero” will be omitted for simplicity):

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(1;2;x\right)\frac{d^2}{d{x}^2}\delta T\left(x\right)} dx+{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(1;2;x\right)\delta T\left(x\right)} dx+{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(2;2;x\right)}{\displaystyle \frac{d^2}{d{x}^2}}\delta b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx\\ +{\gamma }^2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{(2)}\left(2;2;x\right)}\delta b^{\left(1\right)}\left(x\right)\hspace{0.17em}dx=-2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]} dx.\end{array}$$

  (93)
• Integrate by parts the first and third terms on the left-side of Equation (93) and rearrange the resulting expression to obtain the following relation:

  $$\begin{array}{l}{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta T\left(x\right)}\hspace{0.17em}\left[{\displaystyle \frac{d^2{b}^{(2)}\left(1;2;x\right)}{d{x}^2}}+{\gamma }^2{b}^{(2)}\left(1;2;x\right)\right] dx\\ +{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\delta b^{\left(1\right)}\left(x\right)\left[\frac{d^2{b}^{(2)}\left(2;2;x\right)}{d{x}^2}+{\gamma }^2{b}^{(2)}\left(2;2;x\right)\right]} dx\\ +P\left[\delta T\left(x\right),\delta b^{\left(1\right)}\left(x\right),b^{(2)}\left(1;2;x\right);b^{(2)}\left(2;2;x\right)\right]\\ =-2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]} dx.\end{array}$$

  (94)

  where the bilinear concomitant $P\left[\delta T\left(x\right),\delta b^{\left(1\right)}\left(x\right),b^{(2)}\left(1;2;x\right);b^{(2)}\left(2;2;x\right)\right]$ is defined below:

  $$\begin{array}{l}P\left[\delta T\left(x\right),\delta b^{\left(1\right)}\left(x\right),b^{(2)}\left(1;2;x\right);b^{(2)}\left(2;2;x\right)\right]\triangleq b^{(2)}\left(1;2;\mathcal{l}\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=\mathcal{l}}\\ -b^{(2)}\left(1;2;0\right){\left\{{\displaystyle \frac{d}{dx}}\delta T\left(x\right)\right\}}_{x=0}-{\left\{\delta T\left(x\right){\displaystyle \frac{d{b}^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=\mathcal{l}}+{\left\{\delta T\left(x\right){\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0}\\ +b^{(2)}\left(2;2;\mathcal{l}\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(\mathcal{l}\right)-b^{(2)}\left(2;2;0\right){\displaystyle \frac{d}{dx}}\delta b^{\left(1\right)}\left(0\right)\\ -\delta b^{\left(1\right)}\left(\mathcal{l}\right){\displaystyle \frac{d{b}^{(2)}\left(2;2;\mathcal{l}\right)}{dx}}+\delta b^{\left(1\right)}\left(0\right){\displaystyle \frac{d{b}^{(2)}\left(2;2;0\right)}{dx}}.\end{array}$$

  (95)
• Require the first and second terms on the left-side of Equation (94) to represent the indirect-effect term defined in Equation (92) by imposing the following relations:

  $${\displaystyle \frac{d^2{b}^{(2)}\left(1;2;x\right)}{d{x}^2}}+{\gamma }^2{b}^{(2)}\left(1;2;x\right)=-2\gamma b^{\left(1\right)}\left(x\right);$$

  (96)

  $${\displaystyle \frac{d^2{b}^{(2)}\left(2;2;x\right)}{d{x}^2}}+{\gamma }^2{b}^{(2)}\left(2;2;x\right)=-2\gamma T\left(x\right).$$

  (97)
• Eliminate the unknown boundary terms in the expression of the bilinear concomitant defined in Equation (95) by imposing the following boundary conditions:

  $$b^{(2)}\left(1;2;\mathcal{l}\right)=0;\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d{b}^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=\mathcal{l}}=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{b}^{(2)}\left(2;2;0\right)=0;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\left\{{\displaystyle \frac{d{b}^{(2)}\left(2;2;x\right)}{dx}}\right\}}_{x=0}=0\hspace{0.17em}.$$

  (98)
  The system comprising Equations (96)–(98) constitutes the 2nd-LASS for the two-component 2nd-level adjoint sensitivity function $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$.
• Insert the boundary conditions represented by Equations (28) and (81) into Equation (98) and use the relations representing the 2nd-LASS to obtain the following expression for the indirect-effect term defined in Equation (56):

  $$\begin{array}{l}\delta {\left({\displaystyle \frac{\partial R}{\partial \gamma }}\right)}_{ind}=-\delta T_0\hspace{0.17em}{\left\{{\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0}\\ -2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]} dx.\end{array}$$

  (99)

Adding the expression obtained in Equation (99) to the expression of the direct-effect term provided in Equation (91) yields the following expression for the G-differential $\delta \left\{\partial R/\partial \gamma \right\}$:

$$\begin{array}{l}\delta \left\{{\displaystyle \frac{\partial R}{\partial \gamma (\theta )}}\right\}\triangleq -2\left(\delta \gamma \right){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}\hspace{0.17em}T\left(x\right)\hspace{0.17em}dx-\delta T_0\hspace{0.17em}{\left\{{\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0}\\ -2\gamma (\delta \gamma ){\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]} dx.\end{array}$$

(100)

It follows from the expression obtained in Equation (100) that

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma }}=-{\left\{{\displaystyle \frac{b^{(2)}\left(1;2;x\right)}{dx}}\right\}}_{x=0};$$

(101)

$${\displaystyle \frac{{\partial }^{2}R}{\partial \gamma \delta \gamma }}=-2{\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}{b}^{\left(1\right)}\left(x\right)}T\left(x\right) dx-2\gamma {\displaystyle \underset{0}{\overset{\mathcal{l}}{\int }}\left[b^{(2)}\left(1;2;x\right)T\left(x\right)+b^{(2)}\left(2;2;x\right)b^{\left(1\right)}\left(x\right)\right]} dx.$$

(102)

Solving the 2nd-LASS represented by Equations (96)–(98) yields the following expressions for the components of $B^{(2)}\left(2;2;x\right)\triangleq {\left[b^{(2)}\left(1;2;x\right),b^{(2)}\left(2;2;x\right)\right]}^{†}$:

$$\begin{array}{l}{b}^{(2)}\left(1;2;x\right)={\displaystyle \frac{2}{\mathcal{l}{\gamma }^2}}\left[-{\displaystyle \frac{1}{\gamma }}+{\displaystyle \frac{x}{2}}\sin \gamma \left(x-\mathcal{l}\right)-{\displaystyle \frac{1}{4\gamma }}\cos \gamma \left(x-\mathcal{l}\right)\right]\\ \left({\displaystyle \frac{5}{2\mathcal{l}{\gamma }^3}}\sin \gamma \mathcal{l}+{\displaystyle \frac{{\sin }^2\gamma \mathcal{l}}{{\gamma }^2\cos \gamma \mathcal{l}}}-{\displaystyle \frac{1}{{\gamma }^2\cos \gamma \mathcal{l}}}\right)\sin \gamma x+\left({\displaystyle \frac{5}{2\gamma {\mathcal{l}}^3}}\cos \gamma \mathcal{l}+{\displaystyle \frac{\sin \gamma \mathcal{l}}{{\gamma }^2}}\right)\cos \gamma x\hspace{0.17em};\end{array}$$

(103)

$$b^{(2)}\left(2;2;x\right)=-T_{0}x\sin \gamma x\hspace{0.17em}.$$

(104)

Using in Equation (101) the result obtained in Equation (103) yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial T_0\delta \gamma (\theta )}}={\displaystyle \frac{1}{\gamma (\theta )}}\cos \mathcal{l}\gamma (\theta )-{\displaystyle \frac{1}{\mathcal{l}{\gamma }^2(\theta )}}\sin \mathcal{l}\gamma (\theta )\hspace{0.17em}.$$

(105)

As expected, the above expression coincides with the expression obtained, successively, in Equations (53), (75) and (89). Evidently, the expression of the mixed second-order sensitivity ${\partial }^{2}R/\partial T_0\delta \gamma$ can be determined in several distinct ways, using distinct adjoint sensitivity functions, thus providing alternatives for verifying the computational accuracy of the respective adjoint functions, when these functions are computed numerically, as is the case in practice.

Inserting the results obtained in Equations (4), (88), (103) and (104) into Equation (102) and performing the respective integrations yields the following expression:

$${\displaystyle \frac{{\partial }^{2}R}{\partial \gamma (\theta )\partial \gamma (\theta )}}=T_0\left[2{\displaystyle \frac{\sin \mathcal{l}\gamma (\theta )}{\mathcal{l}{\gamma }^3(\theta )}}-2{\displaystyle \frac{\cos \mathcal{l}\gamma (\theta )}{{\gamma }^2(\theta )}}-{\displaystyle \frac{\mathcal{l}\sin \mathcal{l}\gamma (\theta )}{\gamma (\theta )}}\right].$$

(106)

As expected, the above expression coincides with the expression obtained in Equation (78).

4. Discussion

This work has presented the application of the 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies developed in the accompanying “Part I” [1] to an illustrative paradigm heat conduction model. This illustrative model has been chosen because it can be formulated either as a first-order Neural Differential-Integral Equation (NIDE) of the Fredholm type or as a conventional second-order Neural Ordinary Differential Equation (NODE), while admitting exact closed-form solutions/expressions for all quantities of interest, including state functions and first-order and second-order sensitivities. Recall that NODEs [17,18,19,20,21,22,23,24] use differential equation solvers to provide an explicit connection between traditional neural networks [26,27,28] and traditional numerical modeling while offering trade-offs between efficiency, memory costs, and accuracy.

The availability of these alternative formulations, either as a NIDE-F or a NODE, of the illustrative paradigm heat conduction model makes it possible to compare the detailed, step-by-step, applications of the 1st-FASAM-NIDE-F [1] versus the 1st-FASAM-NODE [25] methodologies for computing most efficiently the exact expressions of the first-order sensitivities of the decoder response with respect to the model parameters. In terms of large-scale computations, the 1st-FASAM-NIDE-F and the 1st-FASAM-NODE methodologies are equally efficient conceptually, each requiring a single large-scale computation (for solving the respective 1st-LASS to obtain the respective 1st-level adjoint sensitivity functions) regardless of the number of feature functions of model parameters. The distinctions between using the 1st-FASAM-NIDE-F [1] versus the 1st-FASAM-NODE [25] arise in the fast computations involving quadrature formulas for computing the respective expressions of the integrals that define the respective first-order sensitivities.

The paradigm heat conduction model analyzed in this work has also enabled the exact comparison of applying of the 2nd-FASAM-NIDE-F methodology [1] versus applying the 2nd-FASAM-NODE methodology [25] for computing most efficiently the exact expressions of the second-order sensitivities of the decoder response with respect to the model parameters. As expected, the 2nd-FASAM-NIDE-F and the 2nd-FASAM-NODE methodologies are equally efficient conceptually, each requiring only as many large-scale computations (for solving the respective 2nd-LASS to obtain the respective 2nd-level adjoint sensitivity functions) as there are non-zero first-order sensitivities of the decoder response with respect to the feature functions of model parameters. The distinctions between using the 2nd-FASAM-NIDE-F [1] versus the 2nd-FASAM-NODE [25] arise in the fast computations involving quadrature formulas for computing the respective expressions of the integrals that define the respective second-order sensitivities.

5. Conclusions

The heat conduction model analyzed in this work provides a paradigm example that can be followed for applications in all fields involving neural integro-differential equations of the Fredholm type, highlighting the unsurpassed efficiency of the 2nd-FASAM-NIDE-F methodology for computing exactly obtained expressions of the second-order sensitivities of model responses with respect to model parameters and features thereof. The 2nd-FASAM-NIDE-F encompasses the 1st-FASAM-NIDE-F methodology, which enables the unparalleled efficient computation of exactly obtained expressions of the first-order sensitivities of model responses with respect to model parameters and features thereof. The 1st-FASAM-NIDE-F and 2nd-FASAM-NIDE-F methodologies generalize the recently developed 2nd-FASAM-NIE-F [29] methodology for computing most efficiently exactly obtained expressions of first- and second-order sensitivities of decoder responses associated with Neural Integral Equation of Fredholm-type (NIE-F) with respect to model parameters and features thereof. Mirroring this development, ongoing work aims to generalize the recently developed 2nd-FASAM-NIE-V (“Second-Order Features Adjoint Sensitivity Analysis Methodology for Neural Integral Equations of Volterra-Type”) methodology [30] to enable the most-efficient computation of exactly obtained expressions of the first- and second-order sensitivities of decoder-responses associated with neural integro-differential equations of Volterra-type, which will be designated using the abbreviation/acronym “2nd-FASAM-NIDE-V.” The development of the 2nd-FASAM-NIDE-V will complete the availability of the specialized 2nd-FASAM-NODE, 2nd-FASAM-NIE-F, the 2nd-FASAM-NIE-V, and the 2nd-FASAM-NIDE-F methodologies, which are all derived by particularizing the general-purpose nth-FASAM-N (nth-Order Features Adjoint Sensitivity Analysis Methodology for Nonlinear Systems) methodology [31] for the special neural nets of type NODE, NIE, and NIDE, respectively. For general purpose applications, the nth-FASAM-N [31] remains the sole high-order sensitivity analysis methodology that can compute exactly obtained sensitivities of an arbitrarily high order of model responses with respect to model parameters and features thereof.