Perspective on Predictive Modeling: Current Status, New High-Order Methodology and Outlook for Energy Systems

Abstract

This work presents a perspective on deterministic predictive modeling methodologies, which aim at extracting best-estimate values for model responses and parameters along with reduced predicted uncertainties for these best-estimate values. The two oldest such methodologies are the data-adjustment method, which stems from the nuclear energy field, and the data-assimilation method, which is implemented in the geophysical sciences. Both of these methodologies attempt to minimize, in the least-square sense, a user-defined functional that represents the discrepancies between computed and measured model responses. These two methodologies were briefly reviewed and shown to be inconsistent even to first-order in the sensitivities of the response to the model parameters. In contrast to these methodologies, it was shown that the “maximum entropy”-based predictive modeling methodology (called BERRU-PM) that was developed by the author not only dispenses with the subjective “user-chosen functional to be minimized” but is also inherently amenable to high-order formulations. This inherent potential was illustrated by presenting a novel, higher-order, MaxEnt-based predictive modeling methodology, labelled BERRU-PM-2+, which is complete and exact to second-order sensitivities and moments of both the a priori and posterior distributions of responses and parameters, while explicitly including third- and fourth-order sensitivities and correlations, thus indicating the mechanism for incorporating information of orders higher than second in predictive modeling. The presentation of this new predictive modeling methodology also aims at motivating a widespread application of predictive modeling principles and methodologies in the energy sciences for obtaining best-estimate results with reduced uncertainties.

1. Introduction

The modeling of a physical system includes the following elements:

(a)

a mathematical model comprising equations that relate the system’s independent variables and parameters to the system’s state (i.e., dependent) variables;

(b)

inequality and/or equality constraints that delimit the ranges of the system’s parameters;

(c)

one or several computational results, customarily referred to as “system responses” (or objective functions, or indices of performance), which are computed using the mathematical model; and

(d)

experimentally measured responses, with their respective nominal (mean) values and uncertainties (variances, covariances, skewness, kurtosis, etc.).

The results of measurements and computations are never perfectly accurate but are subject to unavoidable errors. Thus, the results of measurements are inevitably influenced by experimental errors stemming from imperfect instruments or imperfectly known calibration standards. Around any reported experimental value, therefore, there always exists a range of values that may also be plausibly representative of the true but unknown value of the measured quantity. Similarly, computations are subject to errors stemming from uncertain model parameters, initial and boundary conditions, imperfectly known physical processes, imperfectly known physical geometry caused by imperfect material boundaries and, finally, inexact numerical computations. Therefore, knowledge of just the nominal values of experimentally measured or computed quantities is insufficient for determining the reliability of results in practical applications. The quantitative uncertainties accompanying measurements and computations are also needed, along with the respective nominal values.

The uncertainties inherent to experimental and computational results provide the basic motivation for performing quantitative model verification, validation, qualification, and predictive estimation. The activity of “code/model verification” addresses the question “are the equations underlying the mathematical model solved correctly?” The activity of “code/model validation” addresses the question “does the respective model represent reality?” Model validation requires benchmarking the respective model against independently obtained experimental results, including all accompanying uncertainties (computational, experimental, etc.) to quantify the respective model’s accuracy by comparing the computational results produced by the model, including their accompanying statistical characteristics (standard deviations, skewness, kurtosis, and corresponding correlations) with the corresponding experimental results. The activity “code/model qualification” addresses the certification that a proposed simulation and/or design methodology satisfies all performance and safety specifications regarding the system under consideration.

In the author’s view, “predictive modeling” comprises three key elements, namely: model calibration, model extrapolation, and estimation of the validation domain. Model calibration addresses the integration of experimental data for the purpose of updating the parameters underlying the computational model. Important components include the estimation of inconsistencies in the experimental data, and quantification of the biases between model predictions and experimental data. The state-of-the-art of model calibration is fairly well developed, but current methods are still hampered in practice by the significant computational effort required for performing exhaustive (i.e., including all of the sensitivities of model responses to the model’s parameters) model calibration for large-scale models. Reducing the computational effort is paramount, and methods based on adjoint models show great promise in his regard. “Model extrapolation” addresses the quantification of the uncertainties in predictions under new conditions, including both untested regions of the parameter space and higher levels of system complexity in the validation hierarchy. Extrapolation of models and the resulting increase of uncertainty are poorly understood, particularly the estimation of uncertainty that results from nonlinear coupling of two or more physical phenomena that were not coupled in the existing validation database. The quantification of the validation domain underlying the models of interest requires estimation of contours of constant uncertainty in the high-dimensional space that characterizes the application of interest. In practice, this involves the identification of areas where the predictive estimation of uncertainty meets specified requirements for the performance, reliability, or safety of the system of interest. The conceptual and mathematical development of methods for quantifying the validation domain is in an incipient stage.

The earliest activities aimed at extracting best-estimate values for model parameters were initiated in the 1960s [1,2,3] in the nuclear energy field. These activities aimed at evaluating neutron cross sections by using time-independent reactor physics experiments for measuring “integral quantities” (i.e., model responses) such as reaction rates and multiplication factors. These activities reached conceptual maturity under the name of “cross section adjustment” [4,5,6,7,8], which used a least-square procedure, weighted with first-order response sensitivities, for combining uncertainties in the model responses with uncertainties in the corresponding experimental data. The resulting “adjusted” neutron cross sections (model parameters) and their “adjusted” uncertainties were subsequently employed in the respective reactor core model to predict improved model responses (e.g., improved reaction rates and reaction rate ratios, reactor multiplication factors, Doppler coefficients) in an extended application domain (e.g., a new or improved reactor core design). Since the neutron transport (or diffusion) equation is linear in the neutron flux (dependent variable), it admits a corresponding adjoint transport (or diffusion) equation for the adjoint neutron flux, which was in turn used to compute efficiently the first-order response sensitivities, which appeared as weighting functions in the least squares adjustment procedure. The “data-adjustment” methodology can thus be considered to have been the earliest systematic methodology that embodies principles of “predictive modeling”. The principles underlying this data-adjustment methodology for the linear, time-independent nuclear energy systems modeled by the neutron transport equation are briefly summarized in Section 2.1.

A pioneering formalism for performing data adjustment in the context of time-dependent nonlinear system was conceived by Cacuci’s group at Oak Ridge National Laboratory [9], using the adjoint method of sensitivity analysis for nonlinear systems conceived and developed by Cacuci [10], who had formally introduced the concepts of sensitivity analysis by using concepts of nonlinear functional analysis. The data adjustment formalism for time-dependent nonlinear system [9] also relies on the minimization, in the least-square sense, of a user-defined functional; its underlying principles are presented in Section 2.2.

In the late 1980s and during the 1990s, the fundamental concepts underlying “data adjustment” seem to have been rediscovered in the atmospheric and geophysical sciences while developing the so-called “data assimilation” procedure, in that the concepts underlying data assimilation are the same as those underlying the previously developed “data adjustment” procedure, relying on the minimization, in the least-square sense, of a user-defined “cost functional. The principles underlying data assimilation procedures are amply described in the books by Lewis et al. [11], Lahoz et al. [12], Farago et al. [13] and Cacuci et al. [14]. These principles are briefly summarized in Section 2.3, in order to facilitate comparisons with the other procedures that aim at the same goals.

In contradistinction to the methods used for data adjustment and/or assimilation, the BERRU predictive modeling methodology developed by Cacuci [15,16] dispenses with the need to minimize an a priori chosen “cost functional” (usually a quadratic functional that represents the weighted errors between measured and computed responses), by employing the “maximum entropy” (MaxEnt) principle [17] to combine computational and experimental information for obtaining best-estimate predicted mean values for model responses and parameters, together with reduced predicted uncertainties for these best-estimate values. The BERRU predictive modeling methodology [15,16] incorporates first-order sensitivities of model responses with respect to the model parameters, computed using the adjoint method [10].

The conception of the second- and higher-order comprehensive adjoint sensitivity analysis methodology by Cacuci [18,19,20,21] has enabled the efficient computation of exact expression for arbitrarily-high order sensitivities, thus opening the path to further advances of all of the activities which could use such higher-order sensitivities. In particular, the availability of the second- and higher-order model response sensitivities to model parameters has also enabled the extension of the BERRU predictive modeling methodology from first-order to higher-orders. This extended, novel MaxEnt-based predictive modeling formalism, is labeled the “BERRU-PM-2+” methodology because it incorporates not only all of the possible second-order sensitivities and correlations, but also provides the mechanism for incorporating arbitrarily-high-order sensitivities of model responses with respect to model parameters. The novel “BERRU-PM-2+” methodology is presented in Section 3 of this work.

Section 4 concludes this work by presenting a comparative discussion of the deterministic methodologies currently available in the context of their abilities to predict best-estimate mean values and uncertainties (variances/covariances) for calibrating and validating large-scale models of energy-related physical systems.

2. Traditional Least-Square Based Deterministic Predictive Modeling Methodologies

This Section briefly reviews the traditional deterministic methodologies which comprise elements of predictive modeling, as follows:

(i)

The so-called “data-adjustment methodology”, which is perhaps the oldest such methodology in use and was developed in the nuclear energy field. The initial “data-adjustment methodology”, which was developed in the 1960s for the large-scale time-independent linear systems modeled by the linear neutral particle transport equation, is briefly reviewed in Section 2.1. The data-adjustment methodology for time-dependent nonlinear systems, which could be considered to be the predecessor for the so-called “4D-Var” methodology mentioned below, is reviewed in Section 2.2.

(ii)

The so-called “data assimilation” methodology, which is implemented for the large-scale time-dependent systems encountered in the atmospheric and geophysical field, is briefly reviewed in Section 2.3.

2.1. Time-Independent Least-Squares Based Data Adjustment

The latest versions of the “data-adjustment” methodology for the time-independent systems are described in the leading nuclear engineering code-packages SCALE [22] and CRISTAL [23]. This data-adjustment methodology is intended for assimilating experimental information in order to “adjust” (calibrate) model parameters (namely nuclear cross sections) describing linear reactor physics problems modeled by the linear Boltzmann equation. The data-adjustment methodology relies on a “Generalized Linear Least Squares Methodology (GLLSM)” which involves minimizing the following user-defined Lagrangian functional, denoted below as $L\left(y,z\right)$:

$$\underset{y,z}{\min }L\left(y,z\right);L\left(y,z\right)\triangleq {\left(y,z\right)}^{†}\left(\begin{array}{cc}{C}_{mm}& C_{m\alpha }\\ C_{\alpha m}& C_{\alpha \alpha }\end{array}\right)\left(\begin{array}{c}y\\ z\end{array}\right)+2{\lambda }^{†}\left(S_{k}z-y\right).$$

(1)

The quantities appearing in functional $L\left(y,z\right)$ defined in Equation (1) have the following meanings/definitions:

(i)

The (symmetric) relative covariance matrix for the model parameters, $C_{\alpha \alpha }$, is defined as follows:

$$C_{\alpha \alpha }\triangleq \left[\frac{\mathrm{cov}\left({\alpha }_n,{\alpha }_p\right)}{{\alpha }_n{\alpha }_p}\right];n,p=1,\dots ,M,$$

(2)

where ${\alpha }_n$ denotes the nth model parameter (cross sections); $\mathrm{cov}\left({\alpha }_n,{\alpha }_p\right)$ denotes the covariance between the two respective model parameters; $M$ denotes the total number of model parameters.

(ii)

The (symmetric) relative covariance matrix for the measured model responses (effective multiplication factors), $C_{mm}$, is defined as follows:

$$C_{mm}\triangleq \left(\frac{m_i}{k_i}\right)\left[\frac{\mathrm{cov}\left(m_i,m_j\right)}{m_i{m}_j}\right]\left(\frac{m_j}{k_j}\right);i,j=1,\dots ,I,$$

(3)

where $k_i$ denotes the nominal values of the computed effective multiplication factor, using the nominal values of the model parameters (cross sections); $m_i$ denotes the nominal values of the measured effective multiplication factor; $\mathrm{cov}\left(m_i,m_j\right)$ denotes the covariance between two measured effective multiplication factors; $I$ denotes the total number of model responses (measured or computed effective multiplication factors).

(iii)

The rectangular matrix $C_{\alpha m}$ contains as elements the relative covariances between the measured responses and model parameters, and is defined as follows:

$$C_{\alpha m}\triangleq \left[\frac{\mathrm{cov}\left({\alpha }_n,m_i\right)}{{\alpha }_n{m}_i}\right]\left(\frac{m_i}{k_i}\right);n=1,\dots ,M;i=1,\dots ,I;C_{m\alpha }\triangleq C_{\alpha m}^{†}.$$

(4)

(iv)

Each model response, denoted as $k_i\left(\alpha \right)$, is considered to be a linear function of the model parameters, denoted as $\alpha \triangleq {\left({\alpha }_1,\dots ,{\alpha }_M\right)}^{†}$, having the following form:

$$k_i\left(\alpha +\delta \alpha \right)=k_i\left(\alpha \right)\left[1+{\displaystyle \sum _{n=1}^M{S}_n^i\frac{\delta {\alpha }_n}{{\alpha }_n}}\right];i=1,\dots ,I;\delta \alpha \triangleq {\left(\delta {\alpha }_1,\dots ,\delta {\alpha }_M\right)}^{†},$$

(5)

where $\delta {\alpha }_n$ denotes an arbitrary perturbation in the n-th model parameter, and where $S_n^i$ represents the relative sensitivity of the i-th response to the n-th parameter, computed at the nominal parameter values, and is defined as follows:

$$S_n^i\triangleq \frac{\partial k_i}{\partial {\alpha }_n}\frac{{\alpha }_n}{k_i};i=1,\dots ,I;n=1,\dots ,M.$$

(6)

The linear relationship representing the model response as a function of the model parameters is re-written in the following form:

$$y=d+S_{k}z,$$

(7)

where

$$S_k\triangleq \left[S_n^i\right]\triangleq \left[\frac{\partial k_i}{\partial {\alpha }_n}\frac{{\alpha }_n}{k_i}\right];i=1,\dots ,I;n=1,\dots ,M;$$

(8)

$$z\triangleq {\left(z_1,\dots ,z_M\right)}^{†};z_n\triangleq \frac{\delta {\alpha }_n}{{\alpha }_n};$$

(9)

$$d\triangleq {\left(d_1,\dots ,d_I\right)}^{†};d_i\triangleq \frac{k_i\left(\alpha \right)-m_i}{k_i\left(\alpha \right)};$$

(10)

$$y\triangleq {\left(y_1,\dots ,y_I\right)}^{†};y_i\triangleq \frac{k_i\left(\alpha +\delta \alpha \right)-m_i}{k_i\left(\alpha \right)}.$$

(11)

(v)

The (column) vector of Lagrange multipliers $\lambda \triangleq {\left({\lambda }_1,\dots ,{\lambda }_I\right)}^{†}$, which appears in the functional $R\left(y,z\right)$ defined in Equation (1), enforces the linear model defined by Equation (7) as a “hard constraint.”

The “adjusted” parameter and response values are obtained by solving the following equations, which are to be satisfied at the minimum of the functional $R\left(y,z\right)$:

$$\frac{\partial R\left(y,z\right)}{\partial z}=\frac{\partial R\left(y,z\right)}{\partial y}=0.$$

(12)

The solution of Equation (12) yields the following relations for the “adjusted” parameter and response values:

$$z=\left(C_{\alpha m}-C_{\alpha \alpha }{S}_k^{†}\right)C_{dd}^{-1}d,$$

(13)

$$y=\left(C_{mm}-C_{m\alpha }{S}_k^{†}\right)C_{dd}^{-1}d,$$

(14)

$$C_{dd}\triangleq S_k{C}_{\alpha \alpha }{S}_k^{†}+C_{mm}-S_k{C}_{\alpha m}-C_{m\alpha }{S}_k^{†}.$$

(15)

The “adjusted covariance matrix”, $C_{mm}^{adj}$, for the “adjusted” responses (measured or computed) and “the adjusted covariance matrix”, $C_{\alpha \alpha }^{adj}$, for the “adjusted” parameters, respectively, are obtained by using Equations (13) and (14), to obtain the following expressions:

$$C_{mm}^{adj}=C_{mm}-\left(C_{mm}-C_{m\alpha }{S}_k^{†}\right)C_{dd}^{-1}\left(C_{mm}-S_k{C}_{\alpha m}\right),$$

(16)

$$C_{\alpha \alpha }^{adj}=C_{\alpha \alpha }-\left(C_{\alpha m}-C_{\alpha \alpha }{S}_k^{†}\right)C_{dd}^{-1}\left(C_{m\alpha }-S_k{C}_{\alpha \alpha }\right).$$

(17)

Care must be exercised, however, since the indiscriminate incorporation of all seemingly relevant experimental-response data could produce a set of calibrated (adjusted) parameter values that might differ unreasonably much from the corresponding original nominal values and might even fail to improve the agreement between the calculated and measured values of some of the very responses that were used to calibrate the model parameters. When calibrating (adjusting) a library of model parameters, it is tacitly assumed that the given parameters are basically “correct”, in that their true values lie within some reasonable uncertainty of their reported value. However, calibration can be used to increase the accuracy of a model beyond that which is constrained by its reliance on the reported value. The calibration procedure uses additional data (e.g., experimentally measured responses) for improving the parameter values while reducing their uncertainties. Although such additional information induces modifications of the original parameter values, the adjusted parameters are still generally expected to remain consistent with their original nominal values, within the range of their original uncertainties. However, calibration of a library of model parameters by experimental responses which significantly deviate from their respective computed values would significantly modify the resulting adjusted parameters, perhaps even violating the restriction of linearity expressed by Equation (7). Such unlikely adjustments would most probably lead to failure of even reproducing the original experimental responses.

On the other hand, calibrating model parameter by using measured responses that are very close to their respective computed values would cause minimal parameter modifications and a nearly perfect reproduction of the given responses by the calibrated parameters, as would be expected. In such a case, the given responses would be considered as being consistent with the parameter library, in contradistinction to adjustment by inconsistent experimental information, in which case the adjustment (calibration) could fail because of inconsistencies. These considerations clearly underscore the need for using a quantitative indicator to measure the mutual and joint consistency of the information available for model calibration.

2.2. Time-Dependent Least-Squares Based Data Adjustment

The time-dependent least-squares based data-adjustment methodology [9] was developed in the early 1980’s to analyze time-dependent problems of fundamental interest in nuclear engineering (e.g., thermal-hydraulics transient computations and experiments). The underlying concepts are as above and the transient phenomena can be analyzed either globally (“off-line”), when the entire information during the time interval of interest is being analyzed at once or locally, when the data assimilation/adjustment is performed one time-step at a time, as nowadays done in the atmospheric sciences.

The global (“off-line”) methodology can be conveniently summarized by using the notation provided in [16], which considers that the time span of interest for the nonlinear transient system being analyzed is partitioned into $\left(N_t-1\right)$ intervals. At every time instance $\nu$, the quantities $J_{\alpha }^{\nu }$, and $J_r^{\nu }$ are used to denote the ordered set of integers $J_{\alpha }^{\nu }\triangleq \left\{1,\dots ,N_{\alpha }^{\nu }\right\}$ and, respectively, $J_r^{\nu }\triangleq \left\{1,\dots ,N_r^{\nu }\right\}$, where $N_{\alpha }^{\nu }$ and $N_r^{\nu }$ denote the numbers of distinct system parameters and distinct responses, respectively, at time instance $\nu$. The quantity $J_t$ denotes the set $J_t\triangleq \left\{1,\dots ,N_t\right\}$.

At every time instance $\nu$, the imprecisely known system parameters are considered to be variates with mean values denoted as ${\left({\alpha }_i^{\nu }\right)}^0$. The covariances between two parameters ${\alpha }_i^{\nu }$ and ${\alpha }_j^{\mu }$, at two time instances $\mu$ and $\nu$, is denoted as $\mathrm{cov}\left({\alpha }_i^{\nu },{\alpha }_j^{\mu }\right)$. These local covariances are used to construct the following symmetric global parameter covariance matrix:

$$C_{\alpha }\triangleq \left(\begin{array}{l}{C}_{\alpha }^{11}{C}_{\alpha }^{12}\dots \dots \\ C_{\alpha }^{21}{C}_{\alpha }^{22}\dots \dots \\ \dots \dots \dots \dots \\ \dots \dots \dots C_{\alpha }^{N_t{N}_t}\end{array}\right),C_{\alpha }^{\mu \nu }\triangleq \left[\mathrm{cov}\left({\alpha }_i^{\nu },{\alpha }_j^{\mu }\right)\right];i,j=1,\dots ,J_{\alpha }^{\nu };\nu ,\mu =1,\dots ,J_t.$$

(18)

Similarly, the imprecisely known measured responses are characterized by local mean values denoted as $r_i^{\nu }$, at a time instance $\nu$. The covariances between two responses measured at time instances $\mu$ and $\nu$ are denoted as $\mathrm{cov}\left(r_i^{\nu },r_j^{\mu }\right)$. These covariances are used to construct the symmetric global matrix, $C_m$, for measured responses, defined below:

$$C_m\triangleq \left(\begin{array}{l}{C}_m^{11}{C}_m^{12}\dots \dots \\ C_m^{21}{C}_m^{22}\dots \dots \\ \dots \dots \dots \dots \\ \dots \dots \dots C_m^{N_t{N}_t}\end{array}\right);C_m^{\mu \nu }\triangleq \left[\mathrm{cov}\left(r_i^{\nu },r_j^{\mu }\right)\right];i,j=1,\dots ,J_r^{\nu };\nu ,\mu =1,\dots ,J_t.$$

(19)

In the most general case, the measured responses may be correlated to the parameters through local response-parameter covariances, denoted as $\mathrm{cov}\left({\alpha }_i^{\nu },r_j^{\mu }\right)$, between a parameter ${\alpha }_i^{\nu }$ and a response $r_j^{\mu }$, which are used to construct the global rectangular parameter-response covariance matrix, $C_{\alpha r}$, having the following form:

$$C_{\alpha r}\triangleq \left(\begin{array}{l}{C}_{\alpha r}^{11}{C}_{\alpha r}^{12}\dots \dots \\ C_{\alpha r}^{21}{C}_{\alpha r}^{22}\dots \dots \\ \dots \dots \dots \dots \\ \dots \dots \dots C_{\alpha r}^{N_t{N}_t}\end{array}\right);C_{r\alpha }^{\mu \nu }\triangleq \left[\mathrm{cov}\left({\alpha }_i^{\nu },r_j^{\mu }\right)\right];i=1,\dots ,J_{\alpha }^{\nu };j=1,\dots ,J_r^{\nu };\nu ,\mu =1,\dots ,J_t.$$

(20)

At any given time instance $\nu$, a response $r_i^{\nu }$ can be a function of not only the system parameters at time instance $\nu$, but also of the system parameters at all previous time instances $\mu$, $1\le \mu \le \nu$; this means that $r^{\nu }=R^{\nu }\left(p^{\nu }\right)$, where $p^{\nu }\triangleq \left({\alpha }^1,\dots ,{\alpha }^{\mu },\dots ,{\alpha }^{\nu }\right).$ As in the previous Subsection, the computed response is considered to depend linearly on the model parameters, i.e., the computed response is linearized via a functional Taylor-series expansion around the nominal values, $p_0^{\nu }\triangleq \left({\left({\alpha }^0\right)}^1,\dots ,{\left({\alpha }^0\right)}^{\mu },\dots ,{\left({\alpha }^0\right)}^{\nu }\right)$, of the parameters $p^{\nu }$, as follows:

$$r^{\nu }\triangleq R^{\nu }\left(p^{\nu }\right)=R^{\nu }\left(p_0^{\nu }\right)+{\displaystyle \sum _{\mu =1}^{\nu }{S}^{\nu \mu }\left(p_0^{\mu }\right)\left[{\alpha }^{\mu }-{\left({\alpha }^0\right)}^{\mu }\right]}+higherorderterms,\nu \in J_t,$$

(21)

where $R^{\nu }\left(p_0^{\nu }\right)$ denotes the vector of computed responses at a time instance $\nu$, at the nominal parameter values $p_0^{\nu }$, while $S^{\nu \mu }\left(p_0^{\mu }\right)$, $1\le \mu \le \nu$, represents the $\left(J_r^{\nu }\times J_{\alpha }^{\mu }\right)$-dimensional matrix containing the first-order Gateaux-derivatives of the computed responses with respect to the parameters, defined as follows:

$$S^{\nu \mu }\left(p_0^{\mu }\right)\triangleq \left(\begin{array}{ccc}{s}_{11}^{\nu \mu }& \dots & s_{1N}^{\nu \mu }\\ ⋮& s_{in}^{\nu \mu }& ⋮\\ s_{I1}^{\nu \mu }& \cdots & s_{IN}^{\nu \mu }\end{array}\right)\triangleq \left(\begin{array}{ccc}\frac{\partial R_1^n\left(p_0^{\mu }\right)}{\partial {\alpha }_1^{\mu }}& \dots & \frac{\partial R_1^n\left(p_0^{\mu }\right)}{\partial {\alpha }_N^{\mu }}\\ ⋮& \frac{\partial R_i^{\nu }}{\partial {\alpha }_n^{\mu }}& ⋮\\ \frac{\partial R_I^{\nu }\left(p_0^{\mu }\right)}{\partial {\alpha }_1^{\mu }}& \cdots & \frac{\partial R_I^{\nu }\left(p_0^{\mu }\right)}{\partial {\alpha }_N^{\mu }}\end{array}\right),1\le \mu \le \nu .$$

(22)

Since the response $R^{\nu }\left(p_0^{\nu }\right)$ at time instance $\nu$ can depend only on parameters ${\left({\alpha }^0\right)}^{\mu }$ which appear up to the current time instance $\nu$, it follows that $S^{\nu \mu }=0$ when $\mu >\nu$, implying that non-zero terms in the expansion shown in Equation (21) can only occur in the range $1\le \mu \le \nu$. The linear model expressed by Equation (21) can be written in the form

$$r=R\left({\alpha }^0\right)+S\left(\alpha -{\alpha }^0\right)+higherorderterms;$$

(23)

where

$$\begin{array}{l}\alpha \triangleq \left({\alpha }^1,\dots ,{\alpha }^{\mu },\dots ,{\alpha }^{N_t}\right);{\alpha }^{\nu }=\left\{{\alpha }_n^{\nu }|n\in J_{\alpha }^{\nu }\right\},\nu \in J_t;\\ r\triangleq \left(r^1,\dots ,r^{\mu },\dots ,r^{N_t}\right);r^{\nu }=\left\{r_i^{\nu }|i\in J_r^{\nu }\right\},\nu \in J_t;\\ R\left({\alpha }^0\right)\triangleq \left(R^1,\dots ,R^{\mu },\dots ,R^{N_t}\right),\nu \in J_t;\end{array}$$

(24)

$$S\triangleq \left(\begin{array}{ccc}{S}^{11}& \dots & 0\\ ⋮& \ddots & ⋮\\ S^{N_{t}1}& \cdots & S^{N_t{N}_t}\end{array}\right).$$

(25)

The information provided in Equations (18)–(25) is now used to construct the following Lagrangian functional $P\left(z\right)$ to be minimized:

$$P\left(z\right)\triangleq Q\left(z\right)+2{\lambda }^{†}\left[Z\left({\alpha }^0\right)z+d\right]=\min ,atz=z^{be},$$

(26)

where $\lambda =\left({\lambda }^1,\dots ,{\lambda }^{\nu },\dots ,{\lambda }^{N_t}\right)$ denotes the corresponding vector of Lagrange multipliers and where

$$Q\left(z\right)\triangleq z^{†}{C}^{-1}z;z\triangleq \left(\begin{array}{c}\alpha -{\alpha }^0\\ r-r_m\end{array}\right);C\triangleq \left(\begin{array}{cc}{C}_{\alpha }& C_{\alpha r}\\ C_{r\alpha }& C_m\end{array}\right);{\alpha }^0\triangleq {\left[{\left({\alpha }^0\right)}^1,\dots ,{\left({\alpha }^0\right)}^{\mu },\dots ,{\left({\alpha }^0\right)}^{N_t}\right]}^{†};$$

(27)

$$d\triangleq R\left({\alpha }^0\right)-r_m;r_m\triangleq \left(r_m^1,\dots ,r_m^{\mu },\dots ,r_m^{N_t}\right);Z\triangleq \left(S,U\right);U\triangleq \left(\begin{array}{ccc}-I^{11}& \dots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & -I^{N_t{N}_t}\end{array}\right).$$

(28)

In Equation (28), the quantities $I^{jj},j=1,\dots ,N_t$ denote the identity matrices of corresponding dimensions.

Setting $\partial P\left(z\right)/\partial z=0$ and solving the resulting equations yields the following results, which have the same formal structure as shown in Equations (13)–(17):

(i)

calibrated best-estimate parameter values:

$$\begin{array}{l}\left(\begin{array}{c}{\left({\alpha }^{be}\right)}^1\\ ⋮\\ {\left({\alpha }^{be}\right)}^{N_t}\end{array}\right)=\left(\begin{array}{c}{\left({\alpha }^0\right)}^1\\ ⋮\\ {\left({\alpha }^0\right)}^{N_t}\end{array}\right)+\\ \left(\begin{array}{ccc}{C}_{\alpha r}^{11}-C_{\alpha }^{11}{\left(S^{†}\right)}^{11}& \dots & C_{\alpha r}^{1N_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{\alpha }^{1\rho }{\left(S^{†}\right)}^{N_t\rho }}\\ ⋮& \ddots & ⋮\\ C_{\alpha r}^{N_{t}1}-C_{\alpha }^{N_{t}1}{\left(S^{†}\right)}^{11}& \cdots & C_{\alpha r}^{N_t{N}_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{\alpha }^{N_t\rho }{\left(S^{†}\right)}^{N_t\rho }}\end{array}\right)\times \left(\begin{array}{c}{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{K}_d^{1\mathsf{\eta }}{d}^{\mathsf{\eta }}}\\ ⋮\\ {\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{K}_d^{N_t\mathsf{\eta }}{d}^{\mathsf{\eta }}}\end{array}\right),\end{array}$$

(29)

where $K_d^{\nu \mathsf{\eta }}$ denotes the corresponding $\left(\nu ,\mathsf{\eta }\right)$-element of the block-matrix $C_d^{-1}$, where:

$$\begin{array}{l}{C}_d\triangleq \left(\begin{array}{ccc}{C}_d^{11}& \dots & C_d^{1N_t}\\ ⋮& \ddots & ⋮\\ C_d^{N_{t}1}& \cdots & C_d^{N_t{N}_t}\end{array}\right)=\left(\begin{array}{ccc}{C}_{rc}^{11}+C_m^{11}& \dots & C_{rc}^{1N_t}+C_m^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{rc}^{N_{t}1}+C_m^{N_{t}1}& \cdots & C_{rc}^{N_t{N}_t}+C_m^{N_t{N}_t}\end{array}\right)\\ -\left(\begin{array}{ccc}{C}_{r\alpha }^{11}{\left(S^{†}\right)}^{11}+S^{11}{C}_{\alpha r}^{11}& \dots & S^{11}{C}_{\alpha r}^{1N_t}+{\displaystyle \sum _{\rho =1}^{N_t}{C}_{r\alpha }^{1\rho }{\left(S^{†}\right)}^{N_t\rho }}\\ ⋮& \ddots & ⋮\\ C_{r\alpha }^{N_{t}1}{\left(S^{†}\right)}^{11}+{\displaystyle \sum _{\rho =1}^{N_t}{S}^{N_t\rho }{C}_{\alpha r}^{\rho 1}}& \cdots & {\displaystyle \sum _{\rho =1}^{N_t}\left[C_{r\alpha }^{N_t\rho }{\left(S^{†}\right)}^{N_t\rho }+S^{N_t\rho }{C}_{\alpha r}^{\rho N_t}\right]}\end{array}\right),\end{array}$$

(30)

and where the covariance matrix of the computed responses, $C_{rc}$, is defined as follows:

$$C_{rc}\triangleq \left(\begin{array}{ccc}{C}_{rc}^{11}& \dots & C_{rc}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{rc}^{N_{t}1}& \cdots & C_{rc}^{N_t{N}_t}\end{array}\right)=S{C}_{\alpha }{S}^{†};C_{rc}^{\nu \mu }={\displaystyle \sum _{\mathsf{\eta }=1}^{\nu }{\displaystyle \sum _{\rho =1}^{\mu }{S}^{\nu \mathsf{\eta }}{C}_{\alpha }^{\mathsf{\eta }\rho }{\left(S^{\mu \rho }\right)}^{†}}}={\left(C_{rc}^{\mu \nu }\right)}^{†};\nu ,\mu \in J_t.$$

(31)

Written in component form, Equation (29) indicates that the vector ${\left({\alpha }^{be}\right)}^{\nu }$, representing the calibrated best-estimates for the system parameters at a specific time instance $\nu$, takes on the following expression:

$${\left({\alpha }^{be}\right)}^{\nu }={\left({\alpha }^0\right)}^{\nu }+{\displaystyle \sum _{\mu =1}^{N_t}\left\{\left[C_{\alpha r}^{\nu \mu }-{\displaystyle \sum _{\rho =1}^{\mu }{C}_{\alpha }^{\nu \rho }{\left(S^{†}\right)}^{\mu \rho }}\right]\left[{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{K}_d^{\mu \mathsf{\eta }}{d}^{\mathsf{\eta }}}\right]\right\}},\nu \in J_t$$

(32)

(ii)

The calibrated best-estimate covariance matrix, $C_{\alpha }^{be}$, corresponding to the calibrated best-estimates system parameters:

$$\begin{array}{ll}{C}_{\alpha }^{be}& \triangleq \left(\begin{array}{ccc}{\left(C_{\alpha }^{be}\right)}^{11}& \dots & {\left(C_{\alpha }^{be}\right)}^{1N_t}\\ ⋮& \ddots & ⋮\\ {\left(C_{\alpha }^{be}\right)}^{N_{t}1}& \cdots & {\left(C_{\alpha }^{be}\right)}^{N_t{N}_t}\end{array}\right)=\left(\begin{array}{ccc}{C}_{\alpha }^{11}& \dots & C_{\alpha }^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{\alpha }^{N_{t}1}& \cdots & C_{\alpha }^{N_t{N}_t}\end{array}\right)\\ & -\left(\begin{array}{ccc}{C}_{\alpha d}^{11}& \dots & C_{\alpha d}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{\alpha d}^{N_{t}1}& \cdots & C_{\alpha d}^{N_t{N}_t}\end{array}\right)\left(\begin{array}{ccc}{K}_d^{11}& \dots & K_d^{1N_t}\\ ⋮& \ddots & ⋮\\ K_d^{N_{t}1}& \cdots & K_d^{N_t{N}_t}\end{array}\right){\left(\begin{array}{ccc}{C}_{\alpha d}^{11}& \dots & C_{\alpha d}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{\alpha d}^{N_{t}1}& \cdots & C_{\alpha d}^{N_t{N}_t}\end{array}\right)}^{†}\end{array}$$

(33)

where

$$\left(\begin{array}{ccc}{C}_{\alpha d}^{11}& \dots & C_{\alpha d}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{\alpha d}^{N_{t}1}& \cdots & C_{\alpha d}^{N_t{N}_t}\end{array}\right)\triangleq \left(\begin{array}{ccc}{C}_{\alpha r}^{11}-C_{\alpha }^{11}{\left(S^{†}\right)}^{11}& \dots & C_{\alpha r}^{1N_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{\alpha }^{1\rho }{\left(S^{†}\right)}^{N_t\rho }}\\ ⋮& \ddots & ⋮\\ C_{\alpha r}^{N_{t}1}-C_{\alpha }^{N_{t}1}{\left(S^{†}\right)}^{11}& \cdots & C_{\alpha r}^{N_t{N}_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{\alpha }^{N_t\rho }{\left(S^{†}\right)}^{N_t\rho }}\end{array}\right).$$

(34)

The block-matrix expression in Equation (33) can be written in component form, for the calibrated best-estimate parameter covariance matrix ${\left(C_{\alpha }^{be}\right)}^{\nu \mu }$ between two (distinct or not) time instances $\nu ,\mu \in J_t$, as follows

$${\left(C_{\alpha }^{be}\right)}^{\nu \mu }=C_{\alpha }^{\nu \mu }-{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{\displaystyle \sum _{\rho =1}^{N_t}\left[C_{\alpha r}^{\nu \rho }-{\displaystyle \sum _{\pi =1}^{\rho }{C}_{\alpha }^{\nu \pi }{\left(S^{†}\right)}^{\rho \pi }}\right]}}{K}_d^{\rho \mathsf{\eta }}\left[C_{r\alpha }^{\mathsf{\eta }\mu }-{\displaystyle \sum _{\pi =1}^{\mathsf{\eta }}{S}^{\mathsf{\eta }\pi }{C}_{\alpha }^{\pi \mu }}\right].$$

(35)

(iii)

The vector of calibrated best-estimate system responses at all time instances $\nu \in J_t$:

$$\left(\begin{array}{c}{\left(r^{be}\right)}^1\\ ⋮\\ {\left(r^{be}\right)}^{N_t}\end{array}\right)=\left(\begin{array}{c}{\left(r_m\right)}^1\\ ⋮\\ {\left(r_m\right)}^{N_t}\end{array}\right)+\left(\begin{array}{ccc}{C}_m^{11}-C_{r\alpha }^{11}{\left(S^{†}\right)}^{11}& \dots & C_m^{1N_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{r\alpha }^{1\rho }{\left(S^{†}\right)}^{N_t\rho }}\\ ⋮& \ddots & ⋮\\ C_m^{N_{t}1}-C_{r\alpha }^{N_{t}1}{\left(S^{†}\right)}^{11}& \cdots & C_m^{N_t{N}_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{r\alpha }^{N_t\rho }{\left(S^{†}\right)}^{N_t\rho }}\end{array}\right)\left(\begin{array}{c}{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{K}_d^{1\mathsf{\eta }}{d}^{\mathsf{\eta }}}\\ ⋮\\ {\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{K}_d^{N_t\mathsf{\eta }}{d}^{\mathsf{\eta }}}\end{array}\right).$$

(36)

Written in component form, Equation (36) provides the following expression for the vector ${\left(r^{be}\right)}^{\nu }$, of calibrated best-estimates for the responses at a specific time instance $\nu$:

$${\left(r^{be}\right)}^{\nu }={\left(r_m\right)}^{\nu }+{\displaystyle \sum _{\mu =1}^{N_t}\left\{\left[C_x^{\nu \mu }-{\displaystyle \sum _{\rho =1}^{\mu }{C}_{r\alpha }^{\nu \rho }{\left(S^{†}\right)}^{\mu \rho }}\right]\left[{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{K}_d^{\mu \mathsf{\eta }}{d}^{\mathsf{\eta }}}\right]\right\}},\nu \in J_t.$$

(37)

(iv)

The expression of the calibrated best-estimate covariance block-matrix, $C_r^{be}$, for the best-estimate responses:

$$\begin{array}{l}{C}_r^{be}\triangleq \left(\begin{array}{ccc}{\left(C_r^{be}\right)}^{11}& \dots & {\left(C_r^{be}\right)}^{1N_t}\\ ⋮& \ddots & ⋮\\ {\left(C_r^{be}\right)}^{N_{t}1}& \cdots & {\left(C_r^{be}\right)}^{N_t{N}_t}\end{array}\right)=\left(\begin{array}{ccc}{C}_m^{11}& \dots & C_m^{1N_t}\\ ⋮& \ddots & ⋮\\ C_m^{N_{t}1}& \cdots & C_m^{N_t{N}_t}\end{array}\right)\\ -\left(\begin{array}{ccc}{C}_{rd}^{11}& \dots & C_{rd}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{rd}^{N_{t}1}& \cdots & C_{rd}^{N_t{N}_t}\end{array}\right)\left(\begin{array}{ccc}{K}_d^{11}& \dots & K_d^{1N_t}\\ ⋮& \ddots & ⋮\\ K_d^{N_{t}1}& \cdots & K_d^{N_t{N}_t}\end{array}\right){\left(\begin{array}{ccc}{C}_{rd}^{11}& \dots & C_{rd}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{rd}^{N_{t}1}& \cdots & C_{rd}^{N_t{N}_t}\end{array}\right)}^{†},\end{array}$$

(38)

where

$$\left(\begin{array}{ccc}{C}_{rd}^{11}& \dots & C_{rd}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{rd}^{N_{t}1}& \cdots & C_{rd}^{N_t{N}_t}\end{array}\right)=\left(\begin{array}{ccc}{C}_m^{11}-C_{r\alpha }^{11}{\left(S^{†}\right)}^{11}& \dots & C_m^{1N_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{r\alpha }^{1\rho }{\left(S^{†}\right)}^{N_t\rho }}\\ ⋮& \ddots & ⋮\\ C_m^{N_{t}1}-C_{r\alpha }^{N_{t}1}{\left(S^{†}\right)}^{11}& \cdots & C_m^{N_t{N}_t}-{\displaystyle \sum _{\rho =1}^{N_t}{C}_{r\alpha }^{N_t\rho }{\left(S^{†}\right)}^{N_t\rho }}\end{array}\right).$$

(39)

The block-matrix expression given in Equation (38) can be written in component form, where each of the calibrated best-estimate parameter covariance matrix ${\left(C_r^{be}\right)}^{\nu \mu }$ between two (distinct or not) time instances $\nu ,\mu \in J_t$ has the following form:

$${\left(C_r^{be}\right)}^{\nu \mu }=C_m^{\nu \mu }-{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{\displaystyle \sum _{\rho =1}^{N_t}\left[C_m^{\nu \rho }-{\displaystyle \sum _{\pi =1}^{\rho }{C}_{r\alpha }^{\nu \pi }{\left(S^{†}\right)}^{\rho \pi }}\right]}}{K}_d^{\rho \mathsf{\eta }}\left[C_m^{\mathsf{\eta }\mu }-{\displaystyle \sum _{\pi =1}^{\mathsf{\eta }}{S}^{\mathsf{\eta }\pi }{C}_{\alpha r}^{\pi \mu }}\right].$$

(40)

(v)

The best-estimate response-parameter covariance block-matrix $C_{\alpha r}^{be}$:

$$\begin{array}{l}{C}_{r\alpha }^{be}\triangleq \left(\begin{array}{ccc}{\left(C_{r\alpha }^{be}\right)}^{11}& \dots & {\left(C_{r\alpha }^{be}\right)}^{1N_t}\\ ⋮& \ddots & ⋮\\ {\left(C_{r\alpha }^{be}\right)}^{N_{t}1}& \cdots & {\left(C_{r\alpha }^{be}\right)}^{N_t{N}_t}\end{array}\right)=\left(\begin{array}{ccc}{C}_{r\alpha }^{11}& \dots & C_{r\alpha }^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{r\alpha }^{N_{t}1}& \cdots & C_{r\alpha }^{N_t{N}_t}\end{array}\right)\\ -\left(\begin{array}{ccc}{C}_{rd}^{11}& \dots & C_{rd}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{rd}^{N_{t}1}& \cdots & C_{rd}^{N_t{N}_t}\end{array}\right)\left(\begin{array}{ccc}{K}_d^{11}& \dots & K_d^{1N_t}\\ ⋮& \ddots & ⋮\\ K_d^{N_{t}1}& \cdots & K_d^{N_t{N}_t}\end{array}\right){\left(\begin{array}{ccc}{C}_{\alpha d}^{11}& \dots & C_{\alpha d}^{1N_t}\\ ⋮& \ddots & ⋮\\ C_{\alpha d}^{N_{t}1}& \cdots & C_{\alpha d}^{N_t{N}_t}\end{array}\right)}^{†}.\end{array}$$

(41)

Each of the calibrated best-estimate parameter-response covariance matrices ${\left(C_{r\alpha }^{be}\right)}^{\nu \mu }$ for $\nu ,\mu \in J_t$, which appear as block-matrix components of the matrix $C_{r\alpha }^{be}$ in Equation (41) has the following expression:

$${\left(C_{r\alpha }^{be}\right)}^{\nu \mu }=C_{r\alpha }^{\nu \mu }-{\displaystyle \sum _{\mathsf{\eta }=1}^{N_t}{\displaystyle \sum _{\rho =1}^{N_t}\left[C_x^{\nu \rho }-{\displaystyle \sum _{\pi =1}^{\rho }{C}_{r\alpha }^{\nu \pi }{\left(S^{†}\right)}^{\rho \pi }}\right]}}{K}_d^{\rho \mathsf{\eta }}\left[C_{\alpha r}^{\mathsf{\eta }\mu }-{\displaystyle \sum _{\pi =1}^{\mathsf{\eta }}{S}^{\mathsf{\eta }\pi }{C}_{\alpha }^{\pi \mu }}\right].$$

(42)

Computing the calibrated best-estimate quantities by using Equations (32), (35), (37), (40) and (42) is definitely more advantageous in terms of storage requirements than the direct computations of the corresponding full block-matrices. The largest requirement of computational resources arises when inverting the matrix $C_d$. In view of Equation (30), it is important to note that the inverse matrix, $C_d^{-1}$, incorporates simultaneously all of the available information about the system parameters and responses at all time instances (i.e., $\nu =1,2,\dots ,N_t$). In other words, at any time instance $\nu$, $C_d^{-1}$ incorporates information not only from time instances prior to, and at, $\nu$ (i.e., information regarding the “past” and “present” states of the system) but also from time instances posterior to $\nu$ (i.e., information about the “future” states of the system). Therefore, at any specified time instance $\nu$, the calibrated best-estimates parameters ${\left({\alpha }^{be}\right)}^{\nu }$ and responses $r\left({\alpha }^{be}\right)\triangleq r^{be}$ together with the corresponding calibrated best-estimate covariance matrices ${\left(C_{\alpha }^{be}\right)}^{\nu \mu }$, ${\left(C_r^{be}\right)}^{\nu \mu }$, and ${\left(C_{\alpha r}^{be}\right)}^{\nu \mu }$ will also incorporate automatically, through the matrix $C_d^{-1}$, all of the available information about the system parameters and responses at all time instances, i.e., $\left(\nu =1,2,\dots ,N_t\right).$

Note that the mathematical formalism underlying the time-dependent data-adjustment methodology is written in terms of absolute sensitivities and covariances, in contrast to the data adjustment formalism for time-independent systems, which is written in terms of relative sensitivities and covariances. Nevertheless, the two formalisms produce end-results which look formally quite similar.

The actual application of the predictive modeling results presented in Equations (32), (35), (37), (40) and (42) to a physical system characterized by nominal values and uncertainties for model parameters together with the computed and measured responses is straightforward, in principle, although it can become computationally very demanding regarding both data handling and computational speed.

The minimum value, $Q_{\min }$, of the functional $Q\left(z\right)$ has the following expression:

$$Q_{\min }\triangleq Q\left(z^{be}\right)=d^{†}{\left[C_d\left({\alpha }^0\right)\right]}^{-1}d={\chi }^2.$$

(43)

As the above expression indicates, the minimal value, $Q_{\min }\triangleq Q\left(z^{be}\right)$, represents the square of the length of the vector $d$, measuring (in the corresponding metric) the deviations between the experimental and nominally computed responses. The quantity $Q_{\min }\triangleq Q\left(z^{be}\right)$ can be evaluated directly from the given data (i.e., given parameters and responses, together with their original uncertainties) after having inverted the deviation-vector covariance matrix $C_d\left({\alpha }^0\right)$. It is also very important to note that $Q_{\min }\triangleq Q\left(z^{be}\right)$ is independent of calibrating (or adjusting) the original data. As the dimension of $d$ indicates, the number of degrees of freedom characteristic of the calibration under consideration is equal to the number of experimental responses. In the extreme case of absence of experimental responses, no actual calibration takes place since $d=R\left({\alpha }^0\right)$, so that the best-estimate parameter values are just the original nominal values, i.e., ${\left({\alpha }^{be}\right)}^k={\left({\alpha }^0\right)}^k$. An actual calibration (adjustment) occurs only when including at least one experimental response.

In turn, the variate $Q_{\min }$ follows a ${\chi }^2$-distribution with $n$ degrees of freedom, where $n$ denotes the total number of experimental responses considered in the calibration (adjustment) procedure. The quantity $Q_{\min }$ is the “${\chi }^2$ of the calibration (adjustment) at hand” and can be used as an indicator of the agreement between the computed and measured responses, measuring essentially the consistency of the measured responses with the model parameters. A similar quantity exists for the time-independent data adjustment procedure presented previously, in Section 2.1.

In order to facilitate a comparison with the time-dependent data-assimilation methods used in the atmospheric sciences (which are summarized in Section 2.3, below), which customarily involve just two consecutive time-steps, Cacuci [16] has obtained explicit formulas by simplifying the expressions presented in Equations (32), (35), (37), (40) and (42) by including only two consecutive time-steps, namely $\nu =k-1,k;k=\left(1,2,\dots ,N_t\right)$, and by presenting the explicit form of the inverse matrix $K_d^{\nu \mathsf{\eta }}$. The simplified expressions thus obtained are as follows:

(i)

The expressions for the calibrated best-estimate parameter values take on the following particular form of Equation (32) at time node $k$:

$${\left({\alpha }^{be}\right)}^k={\left({\alpha }^0\right)}^k+{\displaystyle \sum _{\mu =k-1}^k\left\{\left[C_{\alpha r}^{k\mu }-{\displaystyle \sum _{\rho =1}^{\mu }{C}_{\alpha }^{k\rho }{\left(S^{†}\right)}^{\mu \rho }}\right]\left[{\displaystyle \sum _{\mathsf{\eta }=k-1}^k{K}_d^{\mu \mathsf{\eta }}{d}^{\mathsf{\eta }}}\right]\right\}},$$

(44)

where

$$\begin{array}{ll}{K}_d^{k-1,k-1}& ={\left[C_d^{k-1,k-1}-C_d^{k-1,k}{\left(C_d^{k,k}\right)}^{-1}{C}_d^{k,k-1}\right]}^{-1}\\ & ={\left(C_d^{k-1,k-1}\right)}^{-1}+{\left(C_d^{k-1,k-1}\right)}^{-1}{C}_d^{k-1,k}{K}_d^{k,k}{C}_d^{k,k-1}{\left(C_d^{k-1,k-1}\right)}^{-1},\end{array}$$

(45)

$$\begin{array}{ll}{K}_d^{k-1,k}& =-{\left(C_d^{k-1,k-1}\right)}^{-1}{C}_d^{k-1,k}{\left[C_d^{k,k}-C_d^{k,k-1}{\left(C_d^{k-1,k-1}\right)}^{-1}{C}_d^{k-1,k}\right]}^{-1}\\ & =-{\left(C_d^{k-1,k-1}\right)}^{-1}{C}_d^{k-1,k}{K}_d^{k,k},\end{array}$$

(46)

$$\begin{array}{ll}{K}_d^{k,k}& ={\left[C_d^{k,k}-C_d^{k,k-1}{\left(C_d^{k-1,k-1}\right)}^{-1}{C}_d^{k-1,k}\right]}^{-1}\\ & ={\left(C_d^{k,k}\right)}^{-1}+{\left(C_d^{k,k}\right)}^{-1}{C}_d^{k,k-1}{K}_d^{k-1,k-1}{C}_d^{k-1,k}{\left(C_d^{k,k}\right)}^{-1},\end{array}$$

(47)

$$\begin{array}{ll}{K}_d^{k,k-1}& =-{\left(C_d^{k,k}\right)}^{-1}{C}_d^{k,k-1}{\left[C_d^{k-1,k-1}-C_d^{k-1,k}{\left(C_d^{k,k}\right)}^{-1}{C}_d^{k,k-1}\right]}^{-1}\\ & =-{\left(C_d^{k,k}\right)}^{-1}{C}_d^{k,k-1}{K}_d^{k-1,k-1}.\end{array}$$

(48)

(ii)

The components ${\left(C_{\alpha }^{be}\right)}^{\nu \mu }$, $\nu ,\mu =k-1,k$, of the calibrated best-estimate covariance matrix, $C_{\alpha }^{be}$, have the following particular form of Equation (35):

$$\begin{array}{l}{\left(C_{\alpha }^{be}\right)}^{\nu \mu }=C_{\alpha }^{\nu \mu }-{\displaystyle \sum _{\mathsf{\eta }=k-1}^k{\displaystyle \sum _{\rho =k-1}^k\left[C_{\alpha r}^{\nu \rho }-{\displaystyle \sum _{\pi =1}^{\rho }{C}_{\alpha }^{\nu \pi }{\left(S^{†}\right)}^{\rho \pi }}\right]}}{K}_d^{\rho \mathsf{\eta }}\left[C_{r\alpha }^{\mathsf{\eta }\mu }-{\displaystyle \sum _{\pi =1}^{\mathsf{\eta }}{S}^{\mathsf{\eta }\pi }{C}_{\alpha }^{\pi \mu }}\right]\\ for\nu =k-1,k;and\mu =k-1,k.\end{array}$$

(49)

(iii)

The vector ${\left(r^{be}\right)}^k$, representing the calibrated best-estimates for the system parameters at a time instance $k$, takes on the following particular form of Equation (37):

$${\left(r^{be}\right)}^k={\left(r_m\right)}^k+{\displaystyle \sum _{\mu =k-1}^k\left\{\left[C_m^{k\mu }-{\displaystyle \sum _{\rho =1}^{\mu }{C}_{r\alpha }^{k\rho }{\left(S^{†}\right)}^{\mu \rho }}\right]\left[{\displaystyle \sum _{\mathsf{\eta }=k-1}^k{K}_d^{\mu \mathsf{\eta }}{d}^{\mathsf{\eta }}}\right]\right\}}.$$

(50)

(iv)

The components ${\left(C_r^{be}\right)}^{\nu \mu }$ of the calibrated best-estimate covariance block-matrix $C_r^{be}$ for the best-estimate responses takes on the following particular form of Equation (38) for $\nu ,\mu =k-1,k$:

$${\left(C_r^{be}\right)}^{\nu \mu }=C_m^{\nu \mu }-{\displaystyle \sum _{\mathsf{\eta }=k-1}^k{\displaystyle \sum _{\rho =k-1}^k\left[C_x^{\nu \rho }-{\displaystyle \sum _{\pi =k-1}^{\rho }{C}_{r\alpha }^{\nu \pi }{\left(S^{†}\right)}^{\rho \pi }}\right]}}{K}_d^{\rho \mathsf{\eta }}\left[C_m^{\mathsf{\eta }\mu }-{\displaystyle \sum _{\pi =k-1}^{\mathsf{\eta }}{S}^{\mathsf{\eta }\pi }{C}_{\alpha r}^{\pi \mu }}\right].$$

(51)

(v)

The matrix-valued components ${\left(C_{\alpha r}^{be}\right)}^{\nu \mu }$, $\nu ,\mu =k-1,k$, of the best-estimate response-parameter covariance matrix $C_{\alpha r}^{be}$ reduce to the following particular forms of Equation (42):

$${\left(C_{r\alpha }^{be}\right)}^{\nu \mu }=C_{r\alpha }^{\nu \mu }-{\displaystyle \sum _{\mathsf{\eta }=k-1}^k{\displaystyle \sum _{\rho =k-1}^k\left[C_m^{\nu \rho }-{\displaystyle \sum _{\pi =k-1}^{\rho }{C}_{r\alpha }^{\nu \pi }{\left(S^{†}\right)}^{\rho \pi }}\right]}}{K}_d^{\rho \mathsf{\eta }}\left[C_{r\alpha }^{\mathsf{\eta }\mu }-{\displaystyle \sum _{\pi =k-1}^{\mathsf{\eta }}{S}^{\mathsf{\eta }\pi }{C}_{\alpha }^{\pi \mu }}\right].$$

(52)

2.3. Least-Squares Based Variational Data Assimilation

The predecessors to data-assimilation methods were called “objective analyses” [24], to distinguish them from the “subjective analyses” in which data was manipulated according to the opinion of experts. Subsequently, the so-called “nudging” or “Newtonian relaxation” methods were introduced [25] to drive the model variables towards the observations by adding empirical forcing terms. The variational principle introduced by Sasaki [26,27] provided the basis for the development of “optimal interpolation,” three-dimensional variational data assimilation,” and “physical space statistical analysis” methods (see, e.g., the books [11,12,13,14]), which were developed independently but were shown by Lorenc [28] to be equivalent.

The introduction of Cacuci’s adjoint method [10] to the atmospheric sciences [29,30,31] has enabled the subsequent development of the modern techniques nowadays called “variational data assimilation” methodology (also called “4D-VAR”; “4D” refers to a framework involving the four independent variables “time plus three space dimensions”). Variational data assimilation aims at determining the model’s evolution “trajectory” which fits “best”—n a “least-square” sense—the observational data over an assimilation time interval by adjusting the initial conditions supplied for advancing (integrating) the forecast model forward in time. The earliest work on developing the 4D-VAR methodology using adjoint functions was by Derber [32,33] and Hoffmann [34], while Navon [35,36,37] has introduced augmented Lagrangian methods for enforcing the conservation of integral invariants. Most of the meteorological centers around the world have since implemented 4D-VAR methods in conjunction with comprehensive numerical weather prediction models [38,39,40].

The so-called strong constraint “variational data assimilation (VDA)” or classical VDA framework assumes that the forecast model perfectly represents the evolution of the actual physical system (atmosphere), and the “best fit” model trajectory is obtained by adjusting only the initial conditions via minimization of a user-defined cost functional, subject to the model equations as strong constraints. However, like any other models of large-scale processes, numerical weather prediction (NWP) models are affected by errors arising from the omission of sub-grid processes; also, the discretization of continuous processes introduces dissipative and dispersion errors. Furthermore, the mathematical modeling of the boundary conditions and forcing terms is imperfect, and most of the physical processes and their interactions in the atmosphere are parameterized. In general, these modeling approximations/errors comprise both systematic (so-called “bias”) and stochastic components which vary in space and time. Customarily, the model error (ME) term is formally introduced as a correction to the time derivatives of model variables and the resulting methodology is called the weak constraint VDA.

Using the notation of [14], the state of the physical system (e.g., atmosphere) can be described by the following evolution equation:

$$\frac{dx(t)}{dt}=M[x(t)]+T[\mathsf{\eta }(t)],x(t_0)=x_0,$$

(53)

where: (i) the vector $x(t)$ represents the state of the atmosphere, with an initial condition $x_0$ at the initial time $t_0$; (ii) the operator $M[.]$ denotes all the mathematical operations involved in the model; (iii) $\mathsf{\eta }\left(t\right)$ represents the model errors; (iv) $T[.]$ represents an operator that maps the space of ME to the space of the model state $x(t)$. If the model state has an associated model error at every grid point, then the operator $T[\cdot ]$ is identically equal to the unit matrix and the dimension of $\mathsf{\eta }\left(t\right)$ is the same as the dimension of the model state $x(t)$. On the other hand, if only some components of the state vector (e.g., the atmosphere in the polar regions) are affected by modeling errors, then the operator $T[\cdot ]$ can be specified in such a way that only those model grid points (in the polar regions) have modeling errors, while (by comparison) the rest of the model states are free of ME’s. Some authors formally/artificially write $\mathsf{\eta }\left(t\right)$ as the sum of a stochastic error (“noise”) component and a “systematic error” component but such a separation is superfluous in practice since both components are anyway estimated based on “expert opinion” rather than based on exact modeling.

The model error is considered to evolve according to the following evolution model:

$$\frac{d\mathsf{\eta }}{dt}=\Phi [\mathsf{\eta }(t),x(t)],\mathsf{\eta }(t_0)={\mathsf{\eta }}_0,$$

(54)

where the rate of growth or decay of $\mathsf{\eta }\left(t\right)$ in time is governed by the particular form of the mapping $\Phi$.

Data-assimilation schemes determine the analyzed atmospheric state as an optimal combination of a-priori background information and observational information, by minimizing the following user-defined functional:

$$J[x(t_0),\mathsf{\eta }(t_0)]\triangleq J_o+J_b+J_{\mathsf{\eta }},$$

(55)

where

$$J_b\triangleq \frac{1}{2}{[x_0-x^b]}^T{B}^{-1}[x_0-x^b],$$

(56)

$$J_o\triangleq \frac{1}{2}{\displaystyle \sum _{i=0}^n}{[H(x_i)-y^o(t_i)]}^T{R}^{-1}[H(x_i)-y^o(t_i)],$$

(57)

$$J_{\mathsf{\eta }}\triangleq \frac{1}{2}{[{\mathsf{\eta }}_0-{\mathsf{\eta }}^b]}^T{Q}^{-1}[{\mathsf{\eta }}_0-{\mathsf{\eta }}^b].$$

(58)

The components of the functionals $J_b$, $J_o$ and $J_{\mathsf{\eta }}$ are defined as follows:

• The functional $J_b$ quantifies, in a least-square sense, the squared differences between the initial state $x_0$ and the background state $x^b$; $B$ denotes the estimated background covariance matrix. The background state $x^b$ provides an initial guess for minimization of $J.$
• The functional $J_o$ quantifies, in a least-square sense, the squared differences between the observed state and initial state at the time instance $t_i$; $R$ denotes the estimated observation covariance matrix and $H(x_i)$ denotes the observation operator at the time instance $t_i$.
• The functional $J_{\mathsf{\eta }}$ quantifies, in a least-square sense, the model errors; $Q$ denotes the covariance matrix of model errors.

The constrained minimization of the functional $J\left(x_0,{\mathsf{\eta }}_0\right)$ subject to the “hard constraints” represented by Equations (53) and (54) is performed by minimizing the unconstrained augmented Lagrangian functional defined below:

$$\begin{array}{c}L(x,\mathsf{\eta },x^{\ast },{\mathsf{\eta }}^{\ast })=J(x_0,{\mathsf{\eta }}_0)+{\displaystyle {\int }_{t_0}^{t_n}}〈x^{\ast }(t),\left\{\frac{dx(t)}{dt}-M[x(t)]-\mathsf{\eta }(t)\right\}〉dt\\ +{\displaystyle {\int }_{t_0}^{t_n}}〈{\mathsf{\eta }}^{\ast }(t),\left\{\frac{d\mathsf{\eta }}{dt}-\Phi [\mathsf{\eta }(t),x(t)]\right\}〉dt,\end{array}$$

(59)

where $x^{\ast }(t)$ and ${\mathsf{\eta }}^{\ast }(t)$ are the Lagrange multiplier vectors corresponding to $x(t)$ and $\mathsf{\eta }(t)$, respectively, while $〈\cdot ,\cdot 〉$ denotes Euclidean inner product in the three-dimensional physical space. The extrema of $L(x,\mathsf{\eta },x^{\ast },{\mathsf{\eta }}^{\ast })$ are the solutions of the following Euler–Lagrange equations:

$$\frac{\partial L}{\partial x}=0,\frac{\partial L}{\partial \mathsf{\eta }}=0,$$

(60)

$$\frac{\partial L}{\partial x^{\ast }}=0,\frac{\partial L}{\partial {\mathsf{\eta }}^{\ast }}=0.$$

(61)

Imposing Equation (61) ensures that the equations describing the evolution of model state and ME shown in Equations (53) and (54) are satisfied. Solving Equation (60) yields the following adjoint equations which describe the evolution of the adjoint variables $x^{\ast }(t)$ and ${\mathsf{\eta }}^{\ast }(t)$:

$$-\frac{d{x}^{\ast }(t)}{dt}={\left[\frac{\partial M}{\partial x}\right]}^T{x}^{\ast }(t)+{\left[\frac{\partial \Phi }{\partial x}\right]}^T{\mathsf{\eta }}^{\ast }(t)+\delta (t-t_i){\displaystyle \sum _{i=0}^n}{\left[\frac{\partial H}{\partial x}\right]}^T{R}^{-1}[H(x_i)-y^o(t_i)];x^{\ast }(t_n)=0,$$

(62)

$$-\frac{d{\mathsf{\eta }}^{\ast }(t)}{dt}={\left[\frac{\partial \Phi }{\partial \mathsf{\eta }}\right]}^T{\mathsf{\eta }}^{\ast }(t)+x^{\ast }(t);{\mathsf{\eta }}^{\ast }(t_n)=0.$$

(63)

As indicated by Equations (62) and (63), the evolution of $x^{\ast }(t)$ is coupled to the evolution of ${\mathsf{\eta }}^{\ast }(t)$ via the operator $\Phi$. Backward integration of the adjoint models represented by Equations (62) and (63) from time $t_n\to t_0$ provides the values of initial adjoint states: $x^{\ast }(t_0)$ and ${\mathsf{\eta }}^{\ast }(t_0).$ Furthermore, the optimality conditions expressed by Equations (60) and (61) also provide the following expressions for the partial gradients of the functional $L(x,\mathsf{\eta },x^{\ast },{\mathsf{\eta }}^{\ast })$ or, equivalently, the functional $J(x_0,{\eta }_0),$ with respect to the initial model state, $x_0$, and the initial ME state, ${\mathsf{\eta }}_0$:

$${\nabla }_{x_0}J={\nabla }_{x_0}{J}_b+{\nabla }_{x_0}{J}_o=B^{-1}[x_0-x^b]+x^{\ast }(t_0);$$

(64)

$${\nabla }_{{\mathsf{\eta }}_0}J={\nabla }_{{\mathsf{\eta }}_0}{J}_{\mathsf{\eta }}+{\nabla }_{{\mathsf{\eta }}_0}{J}_o=Q^{-1}[{\mathsf{\eta }}_0-{\mathsf{\eta }}^b]+{\mathsf{\eta }}^{\ast }(t_0).$$

(65)

If the model’s errors are neglected, Equation (65) would not exist, so the amount of computations would be just half of the computations needed when modeling errors are explicitly taken into account.

3. BERRU-PM-2+: Second-Order-Plus MaxEnt Forward and Inverse Predictive Modeling Methodology

Cacuci [15,16] conceived a predictive modeling methodology which uses the maximum entropy (MaxEnt) principle [17] to combine experimental and computational information in the joint phase-space of responses and model parameters in order to provide optimal mean values for model parameters, as well as reduced uncertainties for the optimally predicted responses and parameters. This methodology has been called the “BERRU-PM” methodology, which is an acronym for “Best-Estimate Results with Reduced Uncertainties: Predictive Modeling”. Using the MaxEnt principle [17], the BERRU-PM methodology eliminates the need for introducing and “minimizing” a user-chosen “cost functional quantifying the discrepancies between measurements and computations”, thus yielding results that are free of subjective user-interferences while generalizing and significantly extending the current dynamic data assimilation procedures. The use of the maximum entropy principle enables the BERRU-PM to consider the model as an “imprecisely known” entity, as opposed to being appended by Lagrange multipliers as a “hard constraint” devoid of model errors. Incorporating correlations, including those between the imprecisely known model parameters and computed model responses, the BERRU-PM also provides a quantitative metric, constructed from sensitivity and covariance matrices, for determining the degree of agreement among the various computational and experimental data while eliminating discrepant information. This Section will introduce the BERRU-PM-2+ predictive modeling methodology, which further generalizes the BERRU-PM methodology provided in [15,16] by including all of the second-order terms in response sensitivities and providing explicitly the mechanism for incorporating not only second-order but arbitrarily high-order (hence the “2+” in the acronym BERRU-PM-2+) sensitivities of responses with respect to parameters, overcoming the “curse of dimensionality” [41].

3.1. Construction of the Second-Order-Accurate MaxEnt Probability Distribution $p_c\left(z\left|z_c,C_c\right.\right)$ of Computational Model Responses and Parameters

The model parameters are denoted as ${\alpha }_i$, $i=1,\dots ,TP$, where $TP$ denotes the “total number of parameters” and are considered, without loss of generality, to be real-valued scalars. For subsequent mathematical operations, it is convenient to consider that these model parameters are the components of the vector of parameters denoted as $\alpha \triangleq {\left({\alpha }_1,\dots ,{\alpha }_{TP}\right)}^{†}\in {ℝ}^{TP}$. In practice, the model parameters are obtained from experimental measurements, which are external to the model and are performed independently of the manner in which the parameters are used in a computational model. Thus, the model parameters are not known exactly; they are subject to uncertainties. Formally, the model parameters can be considered to obey an unknown multivariate probability distribution, denoted as $p_{\alpha }\left(\alpha \right)$, which is defined on a domain (of definition of parameters) denoted as $D_{\alpha }\in {ℝ}^{TP}$ and is normalized such that ${\displaystyle \underset{D_{\alpha }}{\int }{p}_{\alpha }\left(\alpha \right)}d\alpha =1$. Formally, the expected (or mean) value of a model parameter ${\alpha }_j$, denoted as ${\alpha }_j^0\triangleq E_c\left({\alpha }_j\right)$, is defined as follows:

$${\alpha }_j^0\triangleq {\displaystyle \underset{D_{\alpha }}{\int }{\alpha }_j{p}_{\alpha }\left(\alpha \right)}d\alpha ,j=1,\dots ,TP;{\alpha }^0\triangleq {\left({\alpha }_1^0,\dots ,{\alpha }_{TP}^0\right)}^{†}.$$

(66)

The covariance, $\mathrm{cov}\left({\alpha }_i,{\alpha }_j\right)$, of two parameters, ${\alpha }_i$ and ${\alpha }_j$, is formally defined as follows:

$$\mathrm{cov}\left({\alpha }_i,{\alpha }_j\right)\triangleq {\displaystyle \underset{D_{\alpha }}{\int }\delta {\alpha }_i\delta {\alpha }_j{p}_{\alpha }\left(\alpha \right)}d\alpha ;i,j=1,\dots ,TP;\delta {\alpha }_k\triangleq {\alpha }_k-{\alpha }_k^0;k=1,\dots ,TP.$$

(67)

The variance, $\mathrm{var}\left({\alpha }_i\right)$, of a parameter ${\alpha }_i$, is formally defined as the following particular case of Equation (67):

$$\mathrm{var}\left({\alpha }_i\right)\triangleq {\displaystyle \underset{D_{\alpha }}{\int }{\left(\delta {\alpha }_i\right)}^2{p}_{\alpha }\left(\alpha \right)}d\alpha ;i=1,\dots ,TP.$$

(68)

The standard deviation of ${\alpha }_i$ is denoted as ${\sigma }_i$ and is defined as follows: ${\sigma }_i\triangleq \sqrt{\mathrm{var}({\alpha }_i)}$. The definitions of the higher-order correlations among parameters, which are used together with those shown in Equations (66)–(68) to construct the moments of the distribution of computed responses (i.e., results of interest computed using the model), are presented in Appendix A.

A model response is a function (implicit and, occasionally, explicit) of the model parameters and is denoted as $r_k\left(\alpha \right)$. Evidently, the model response is also an imprecisely known quantity, subject to uncertainties stemming from the model parameters, in addition to uncertainties stemming from the numerical computational procedures. A model response thus comprises by both systematic and random errors. For subsequent mathematical operations, it is convenient to consider that the responses $r_k\left(\alpha \right)$, $k=1,\dots ,TR$, where $TR$ denotes the “total number of responses” computed using the model, are the components of the vector of responses denoted as $r\left(\alpha \right)\triangleq {\left[r_1\left(\alpha \right),\dots ,r_{TR}\left(\alpha \right)\right]}^{†}$.

Conceptually, the computational model’s responses can be considered to obey an unknown multivariate probability distribution function, denoted as $p_c\left(r\right)$, which is defined on a domain denoted as $D_r\in {ℝ}^{TR}$ and is normalized such that ${\displaystyle \underset{D_r}{\int }{p}_c\left(r\right)}dr=1$. Furthermore, the parameters and computed responses can be considered to follow a joint probability distribution function denoted as $p_c\left(\alpha ,r\right)\triangleq p_{\alpha }\left(\alpha \right)p_c\left(r\right)$, which is not known, but is formally defined on a computational domain denoted as $D_c=D_{\alpha }\cup D_r$, which comprises the reunion of the domains of definitions of all of the model’s parameters and responses, and is properly normalized, i.e., ${\displaystyle \underset{D_c}{\int }{p}_c\left(\alpha ,r\right)d\alpha dr=1}$. The subscript “c” is used to indicate quantities pertaining to the “computational model”.

In addition to results produced by computational models, in practice there also exist independent measurements of responses and parameters, which are subject to uncertainties which stem from the respective measurement devices and procedures, independently of the computational model. Since such measurements are scalar quantities which are compared to the computational results, it is convenient to consider that the computational responses are also real-valued scalar quantities, to enable direct comparisons with the corresponding measured responses.

Since the system’s response is a function of the system’s parameters, it is assumed that each computed response can be formally expanded in a multivariate Taylor-series around the parameters’ mean values. In particular, the fourth-order Taylor-series of a system response, denoted as $r_k\left(\alpha \right)$, around the expected (or nominal) parameter values ${\alpha }^0$ has the following formal expression:

$$\begin{array}{l}{r}_k\left(\alpha \right)=r_k\left({\alpha }^0\right)+{\displaystyle \sum _{j_1=1}^{TP}{\left\{\frac{\partial r_k\left(\alpha \right)}{\partial {\alpha }_{j_1}}\right\}}_{{\alpha }^0}\delta {\alpha }_{j_1}}+\frac{1}{2}{\displaystyle \sum _{j_1=1}^{TP}{\displaystyle \sum _{j_2=1}^{TP}{\left\{\frac{{\partial }^2{r}_k\left(\alpha \right)}{\partial {\alpha }_{j_1}\partial {\alpha }_{j_2}}\right\}}_{{\alpha }^0}}}\delta {\alpha }_{j_1}\delta {\alpha }_{j_2}\\ +\frac{1}{3!}{\displaystyle \sum _{j_1=1}^{TP}{\displaystyle \sum _{j_2=1}^{TP}{\displaystyle \sum _{j_3=1}^{TP}{\left\{\frac{{\partial }^3{r}_k\left(\alpha \right)}{\partial {\alpha }_{j_1}\partial {\alpha }_{j_2}\partial {\alpha }_{j_3}}\right\}}_{{\alpha }^0}}}}\delta {\alpha }_{j_1}\delta {\alpha }_{j_2}\delta {\alpha }_{j_3}\\ +\frac{1}{4!}{\displaystyle \sum _{j_1=1}^{TP}{\displaystyle \sum _{j_2=1}^{TP}{\displaystyle \sum _{j_3=1}^{TP}{\displaystyle \sum _{j_4=1}^{TP}{\left\{\frac{{\partial }^4{r}_k\left(\alpha \right)}{\partial {\alpha }_{j_1}\partial {\alpha }_{j_2}\partial {\alpha }_{j_3}\partial {\alpha }_{j_4}}\right\}}_{{\alpha }^0}}}}}\delta {\alpha }_{j_1}\delta {\alpha }_{j_2}\delta {\alpha }_{j_3}\delta {\alpha }_{j_4}+{\epsilon }_k.\end{array}$$

(69)

In Equation (69), the quantity $r_k\left({\alpha }^0\right)$ indicates the computed value of the response using the expected/nominal parameter values ${\alpha }^0\triangleq {\left({\alpha }_1^0,\dots ,{\alpha }_{TP}^0\right)}^{†}$, while the notation ${\left\{\right\}}_{{\alpha }^0}$ indicates that the quantities within the braces (i.e., first- and second-order sensitivities of the response with respect to the parameters) are also computed using the expected/nominal parameter values. The quantity ${\epsilon }_k$ in Equation (69) comprises all quantifiable errors in the representation of the computed response as a function of the model parameters, including the truncation errors $O{\left(\delta {\alpha }_j\right)}^5$ of the Taylor-series expansion, possible bias-errors due to incompletely modeled physical phenomena, and possible random errors due to numerical approximations. The radius/domain of convergence of the series in Equation (69) determines the largest values of the parameter variations $\delta {\alpha }_j$ which are admissible before the respective series becomes divergent. In turn, these maximum admissible parameter variations limit, through Equation (67), the largest parameter covariances/standard deviations which can be considered for using the Taylor-expansion for the subsequent purposes of computing moments of the distribution of computed responses.

As is well known, and as indicated by Equation (69), the Taylor-series of a function of $TP$-variables [e.g., $r_k\left(\alpha \right)$] comprises $TP$ 1st-order derivatives, $TP\left(TP+1\right)/2$ distinct second-order derivatives, and so on. The computation by conventional methods of the $n^{th}$-order functional derivatives (called “sensitivities” in the field of sensitivity analysis) of a response with respect to the $TP$-parameters (on which it depends) would require at least $O\left(T{P}^n\right)$ large-scale computations. The exponential increase—with the order of response sensitivities—of the number of large-scale computations needed to determine higher-order sensitivities is the manifestation of the “curse of dimensionality in sensitivity analysis”, by analogy to the expression coined by Belmann [41] to express the difficulty of using “brute-force” grid search when optimizing a function with many input variables. The “n^th-order Comprehensive Adjoint Sensitivity Analysis Methodology for Nonlinear Systems” (n^th-CASAM-N) conceived by Cacuci [20] and the “n^th-order Comprehensive Adjoint Sensitivity Analysis Methodology for Response-Coupled Forward/Adjoint Linear Systems” (n^th-CASAM-L) conceived by Cacuci [21] are currently the only methodologies that enable the exact and efficient computation of arbitrarily high-order sensitivities while overcoming the curse of dimensionality.

Uncertainties in the model’s parameters will evidently give rise to uncertainties in the model responses $r_k\left(\alpha \right)$. The approximate moments of the unknown distribution of $r_k\left(\alpha \right)$ are obtained by using the so-called “propagation of errors” methodology, integrating formally, over the unknown distribution, various expressions involving the truncated Taylor-series expansion of the response provided in Equation (69), as first performed by Tukey [42]. Tukey’s results were generalized to sixth-order by Cacuci [21].

The expectation value, $E\left(r_k\right)$, of a response $r_k\left(\alpha \right)$ is obtained by integrating formally Equation (69) over $p_c\left(\alpha ,r\right)$, i.e.,:

$$E_c\left(r_k\right)\triangleq {\displaystyle \underset{D_c}{\int }{r}_k\left(\alpha \right)p_c\left(\alpha ,r\right)d\alpha dr}.$$

(70)

The approximations of $E\left(r_k\right)$ up to fourth-order in the parameters’ standard deviations are presented in Appendix A.

The expression of the correlation between a computed responses and a parameter variance, which is denoted as $\mathrm{cov}\left({\alpha }_i,r_k\right)$, is defined below

$$\mathrm{cov}\left({\alpha }_i,r_k\right)\triangleq {\displaystyle \underset{D_c}{\int }\delta {\alpha }_i\left[r_k\left(\alpha \right)-E_c\left(r_k\right)\right]p_c\left(\alpha ,r\right)d\alpha dr}.$$

(71)

The approximations of $\mathrm{cov}\left({\alpha }_i,r_k\right)$ up to fourth-order in the parameters’ standard deviations are presented in Appendix A.

The covariance between two responses $r_k$ and $r_{\ell }$ is denoted as $cov\left(r_k,r_{\ell }\right)$ and is defined below:

$$cov\left(r_k,r_{\ell }\right)\triangleq {\displaystyle \underset{D_c}{\int }\left[r_k\left(\alpha \right)-E_c\left(r_k\right)\right]\left[r_{\ell }\left(\alpha \right)-E_c\left(r_{\ell }\right)\right]p_c\left(\alpha ,r\right)d\alpha dr}.$$

(72)

The approximations of $cov\left(r_k,r_{\ell }\right)$ up to fourth-order in in the parameters’ standard deviations are presented in Appendix A.

It is assumed in this work that the only information about the distribution $p_c\left(\alpha ,r\right)$ is provided by the known first- and second-order moments of the joint distribution of model parameters and responses defined by Equations (66), (67), (70), (71) and (72). t is well known that when the underlying distribution $p_c\left(\alpha ,r\right)$ needs to be determined from such incomplete information, the principle of maximum entropy (MaxEnt) originally formulated by Jaynes [17] provides the optimal compatibility with the available information, while simultaneously ensuring minimal spurious information content. According to the MaxEnt principle, the probability density $p_c\left(\alpha ,r\right)$ would satisfy the “available information” without implying any spurious information or hidden assumptions if:

(i)

$p_c\left(\alpha ,r\right)$ maximizes the Shannon [43] information entropy for the computational model, $S_c$, defined below:

$$S_c=-{\displaystyle \underset{D_c}{\int }{p}_c\left(\alpha ,r\right)\ln \left[p_c\left(\alpha ,r\right)\right]d\alpha dr},$$

(73)

(ii)

$p_c\left(\alpha ,r\right)$ satisfies the “moments constraints” defined by Equations (66), (67), (70), (71) and (72);

(iii)

$p_c\left(\alpha ,r\right)$ satisfies the normalization condition:

$${\displaystyle \underset{D_c}{\int }{p}_c\left(\alpha ,r\right)d\alpha dr}=1.$$

(74)

The MaxEnt distribution $p_c\left(\alpha ,r\right)$ is obtained as the solution of the constrained variational problem $\partial H_c\left(p_c\right)/\partial p_c=0$, where the entropy (Lagrangian functional) $H_c\left(p_c\right)$ is defined as follows:

$$\begin{array}{l}{H}_c\left(p_c\right)=-{\displaystyle \underset{D_c}{\int }{p}_c\left(\alpha ,r\right)\ln \left[p_c\left(\alpha ,r\right)\right]d\alpha dr}-{\lambda }_0\left[{\displaystyle \underset{D_c}{\int }{p}_c\left(\alpha ,r\right)d\alpha dr}-1\right]\\ -{\displaystyle \sum _{k=1}^{TR}{\lambda }_k^{\left(1\right)}\left[{\displaystyle \underset{D_c}{\int }{r}_k{p}_c\left(\alpha ,r\right)d\alpha dr-E_c\left(r_k\right)}\right]}-{\displaystyle \sum _{i=1}^{TP}{\lambda }_i^{\left(2\right)}\left[{\displaystyle \underset{D_c}{\int }{\alpha }_i{p}_c\left(\alpha ,r\right)}d\alpha dr-{\alpha }_i^0\right]}\\ -\frac{1}{2}{\displaystyle \sum _{k=1}^{TR}{\displaystyle \sum _{\ell =1}^{TR}{\lambda }_{k\ell }^{\left(11\right)}}\left[{\displaystyle \underset{D_c}{\int }{r}_k{r}_{\ell }{p}_c\left(\alpha ,r\right)d\alpha dr-cov\left(r_k,r_{\ell }\right)-E_c\left(r_k\right)E_c\left(r_{\ell }\right)}\right]}\\ -{\displaystyle \sum _{k=1}^{TR}{\displaystyle \sum _{i=1}^{TP}{\lambda }_{ki}^{\left(12\right)}}\left[{\displaystyle \underset{D_c}{\int }{r}_k{\alpha }_i{p}_c\left(\alpha ,r\right)d\alpha dr-cov\left(r_k,r_{\ell }\right)-E_c\left(r_k\right){\alpha }_i^0}\right]}\\ -\frac{1}{2}{\displaystyle \sum _{i=1}^{TP}{\displaystyle \sum _{j=1}^{TP}{\lambda }_{ij}^{\left(22\right)}}\left[{\displaystyle \underset{D_c}{\int }{\alpha }_i{\alpha }_j{p}_c\left(\alpha ,r\right)d\alpha dr-\mathrm{cov}\left({\alpha }_i,{\alpha }_j\right)-{\alpha }_i^0{\alpha }_j^0}\right]}.\end{array}$$

(75)

In Equation (75), the quantities ${\lambda }_k^{\left(1\right)}$, ${\lambda }_k^{\left(2\right)}$, ${\lambda }_{k\ell }^{\left(11\right)}$,${\lambda }_{k\ell }^{\left(12\right)}$, and ${\lambda }_{k\ell }^{\left(22\right)}$ denote the respective Lagrange multipliers, and the factors $1/2$ are introduced for subsequent computational convenience.

Solving the equation $\partial H\left(p_c\right)/\partial p_c=0$ yields the following expression for the resulting MaxEnt distribution $p_c\left(z\right)$:

$$p_c\left(z\right)=\frac{1}{Z_c\left(b,{\Lambda }_c\right)}\exp \left(-b^{†}z-\frac{1}{2}{z}^{†}{\Lambda }_{c}z\right),$$

(76)

where the subscript “c” indicates quantities related to the computational model and where the various vectors and matrices are defined as follows:

$$z=\left(\begin{array}{c}r\\ \alpha \end{array}\right);b\triangleq \left(\begin{array}{c}{\lambda }^{\left(1\right)}\\ {\lambda }^{\left(2\right)}\end{array}\right);{\lambda }^{\left(1\right)}\triangleq \left(\begin{array}{c}{\lambda }_1^{\left(1\right)}\\ ·\\ {\lambda }_{TR}^{\left(1\right)}\end{array}\right);{\lambda }^{\left(2\right)}\triangleq \left(\begin{array}{c}{\lambda }_1^{\left(2\right)}\\ ·\\ {\lambda }_{TP}^{\left(1\right)}\end{array}\right);$$

(77)

$$\begin{array}{l}{\Lambda }_c\triangleq \left(\begin{array}{cc}{\Lambda }_c^{\left(11\right)}& {\Lambda }_c^{\left(12\right)}\\ {\left[{\Lambda }_c^{\left(12\right)}\right]}^{†}& {\Lambda }_c^{\left(22\right)}\end{array}\right);{\Lambda }_c^{\left(11\right)}\triangleq \left(\begin{array}{ccc}{\lambda }_{11}^{\left(11\right)}& ·& {\lambda }_{1,TR}^{\left(11\right)}\\ ·& {\lambda }_{k\ell }^{\left(11\right)}& ·\\ {\lambda }_{TR,1}^{\left(11\right)}& ·& {\lambda }_{TR,TR}^{\left(11\right)}\end{array}\right);\\ {\Lambda }_c^{\left(12\right)}\triangleq \left(\begin{array}{ccc}{\lambda }_{11}^{\left(12\right)}& ·& {\lambda }_{1,TP}^{\left(12\right)}\\ ·& {\lambda }_{k\ell }^{\left(12\right)}& ·\\ {\lambda }_{TR,1}^{\left(12\right)}& ·& {\lambda }_{TR,TP}^{\left(12\right)}\end{array}\right);{\Lambda }_c^{\left(11\right)}\triangleq \left(\begin{array}{ccc}{\lambda }_{11}^{\left(22\right)}& ·& {\lambda }_{1,TP}^{\left(22\right)}\\ ·& {\lambda }_{k\ell }^{\left(22\right)}& ·\\ {\lambda }_{TP,1}^{\left(22\right)}& ·& {\lambda }_{TP,TP}^{\left(22\right)}\end{array}\right).\end{array}$$

(78)

The normalization constant $Z_c\left(b,{\Lambda }_c\right)$ in Equation (76) is defined as follows:

$$Z_c\left(b,{\Lambda }_c\right)={\displaystyle \underset{D_c}{\int }\exp \left(-b^{†}z-\frac{1}{2}{z}^{†}{\Lambda }_{c}z\right)d}z;dz\equiv d\alpha dr.$$

(79)

In statistical mechanics, the normalization constant $Z_c$ is called the partition function (or sum over states) and carries all of the information available about the possible states of the system, while the MaxEnt distribution $p_c\left(z\right)$ is called the canonical Boltzmann–Gibbs distribution. The integral in Equation (79) can be evaluated explicitly by conservatively extending the computational domain $D_c$ to the entire multidimensional real vector space ${ℝ}^N$, where $N\triangleq TR+TP$, to obtain the following expression:

$$Z_c\left(b,{\Lambda }_c\right)={\displaystyle \underset{{ℝ}^N}{\int }\exp \left(-b^{†}z-\frac{1}{2}{z}^{†}{\Lambda }_{c}z\right)d}z=\frac{{\left(2\pi \right)}^{N/2}}{\sqrt{Det\left({\Lambda }_c\right)}}{e}^{\left(\frac{1}{2}{b}^{†}{\Lambda }_c^{-1}b\right)}.$$

(80)

The Lagrange multipliers are determined in terms of the known information (means and covariances of parameters and responses) by differentiating the “free energy” $F\left(b,C_c\right)\triangleq -\ln Z_c\left(b,C_c\right)$ with respect to the components of the vector $b\triangleq {\left({\lambda }^{\left(1\right)},{\lambda }^{\left(2\right)}\right)}^{†}$ to obtain the following expressions:

$$\frac{\partial F\left(b,C_c\right)}{\partial {\lambda }_k^{\left(1\right)}}=\frac{1}{Z_c}{\displaystyle \underset{D_c}{\int }{r}_k{e}^{\left(-b^{†}z-\frac{1}{2}{z}^{†}{\Lambda }_{c}z\right)}dz={\displaystyle \underset{D_c}{\int }{r}_k{p}_c\left(\alpha ,r\right)dz}=E_c\left(r_k\right)};k=1,\dots ,TR;$$

(81)

$$\frac{\partial F\left(b,C_c\right)}{\partial {\lambda }_i^{\left(2\right)}}=\frac{1}{Z_c}{\displaystyle \underset{D_c}{\int }{\alpha }_i{e}^{\left(-b^{†}z-\frac{1}{2}{z}^{†}{\Lambda }_{c}z\right)}dz={\displaystyle \underset{D_c}{\int }{\alpha }_i{p}_c\left(\alpha ,r\right)}d\alpha dr={\alpha }_i^0};i=1,\dots ,TP.$$

(82)

The results obtained in Equations (81) and (82) can be collectively written in vector-matrix form as follows:

$$\frac{\partial F\left(b,C_c\right)}{\partial b}=z_c;z_c\triangleq {\left[E_c\left(r\right),{\alpha }^0\right]}^{†};E_c\left(r\right)\triangleq {\left[E_c\left(r_1\right),\dots ,E_c\left(r_{TR}\right)\right]}^{†}.$$

(83)

On the other hand, it follows from Equation (80) that

$$\frac{\partial F\left(b,C_c\right)}{\partial b}=-{\Lambda }_c^{-1}b.$$

(84)

The relations obtained in Equations (83) and (84) imply the following relation:

$$b=-{\Lambda }_c{z}_c.$$

(85)

Differentiating a second time the relation provided in Equation (81) or (82) yields the following relations:

$$\begin{array}{l}\frac{{\partial }^{2}F\left(b,C_c\right)}{\partial {\lambda }_j^{\left(1\right)}\partial {\lambda }_k^{\left(1\right)}}=-\frac{1}{Z_c^2}\frac{\partial Z_c}{\partial {\lambda }_j^{\left(1\right)}}{E}_c\left(r_k\right)+\frac{1}{Z_c}{\displaystyle \underset{D_c}{\int }\left(-r_j\right)r_k{e}^{\left(-b^{†}z-\frac{1}{2}{z}^{†}{\Lambda }_{c}z\right)}dz}\\ =E_c\left(r_j\right)E_c\left(r_k\right)-E_c\left(r_j{r}_k\right)\triangleq -\mathrm{cov}\left(r_j,r_k\right);j,k=1,\dots ,TR.\end{array}$$

(86)

$$\frac{{\partial }^{2}F\left(b,C_c\right)}{\partial {\lambda }_i^{\left(2\right)}\partial {\lambda }_k^{\left(1\right)}}=E_c\left(r_k\right){\alpha }_i^0-E_c\left({\alpha }_i{r}_k\right)\triangleq -\mathrm{cov}\left({\alpha }_i,r_k\right);i=1,\dots ,TP;k=1,\dots ,TR;$$

(87)

$$\frac{{\partial }^{2}F\left(b,C_c\right)}{\partial {\lambda }_i^{\left(2\right)}\partial {\lambda }_j^{\left(2\right)}}={\alpha }_i^0{\alpha }_j^0-E_c\left({\alpha }_i{\alpha }_j\right)\triangleq -\mathrm{cov}\left({\alpha }_i,{\alpha }_j\right);i,j=1,\dots ,TP.$$

(88)

The results obtained in Equations (86)–(88) can be collectively written in vector-matrix form as follows:

$$\begin{array}{l}\frac{{\partial }^{2}F\left(b,C_c\right)}{\partial b\partial b}=-C_c;C_c\triangleq \left(\begin{array}{cc}{C}_c^{rr}& C_c^{r\alpha }\\ C_c^{\alpha r}& C^{\alpha \alpha }\end{array}\right);C_c^{rr}\triangleq {\left[\mathrm{cov}\left(r_j,r_k\right)\right]}_{TR\times TR};\\ C_c^{r\alpha }\triangleq {\left[\mathrm{cov}\left(r_k,{\alpha }_i\right)\right]}_{TR\times TP}={\left(C_c^{\alpha r}\right)}^{†};C^{\alpha \alpha }\triangleq {\left[\mathrm{cov}\left({\alpha }_i,{\alpha }_j\right)\right]}_{TP\times TP}.\end{array}$$

(89)

On the other hand, it follows from Equation (84) that

$$\frac{{\partial }^{2}F\left(b,C_c\right)}{\partial b\partial b}=-{\Lambda }_c^{-1}.$$

(90)

The relations obtained in Equations (89) and (90) imply the following relation:

$${\Lambda }_c^{-1}=C_c.$$

(91)

Introducing the results obtained in Equations (84) and (91) into Equations (76) and (80) yields the following expression for the MaxEnt distribution $p_c\left(z\left|z_c,C_c\right.\right)$:

$$p_c\left(z\left|z_c,C_c\right.\right)=\frac{{\left(2\pi \right)}^{-N/2}}{\sqrt{Det\left(C_c\right)}}\exp \left[-\frac{1}{2}{\left(z-z_c\right)}^{†}{C}_c^{-1}\left(z-z_c\right)\right].$$

(92)

3.2. Construction of the Second-Order-Accurate MaxEnt Probability Distribution $p_e\left(z\left|z_e,C_e\right.\right)$ of Experimentally Measured Responses

In addition to information regarding the computational model’s responses and parameters, in practice there also exists experimental information about the responses and model parameters, which are obtained independently of computations. In the most comprehensive setting, there would be independent measurements for the computed responses, for the model parameters and for measured correlations between the model parameters and the measured responses. In this idealized setting, each one of the computed responses would also be measured experimentally, so that information about the mean values and covariances for a total number $TR$ of experimentally measured responses would also be available. The mean values of the experimentally measured responses are denoted as $r_i^e$, $i=1,\dots ,TR$, and the covariances of two measured responses are denoted as $\mathrm{cov}{\left(r_i,r_j\right)}_e$, $i,j=1,\dots ,TR$. The letter “e” is used either as a superscript or a superscript to indicate experimentally measured quantities. Similarly, experimentally-determined values for the expected values ${\alpha }_i^e$, $i=1,\dots ,TP$, and newly measured covariances $\mathrm{cov}{\left({\alpha }_i,{\alpha }_j\right)}_e$ for the model parameters could also be available. In principle, it is also possible to obtain correlations between some measured responses and some model parameters. In the nuclear energy field, for example, an important measured detector response is the fission rate in a material (e.g., reactor fuel) which contains (fissionable) uranium-235. However, the fission cross section of uranium-235 would also be a model parameter in the computation of the fission reaction rate, which would thus be correlated to the response.

Formally, the measured responses and parameters are considered to follow a joint unknown probability distribution function, which can be denoted as $p_e\left(\alpha ,r\right)$ and which is formally defined on a domain denoted as $D_e\in {ℝ}^N;N\triangleq TP+TR$. The means and covariances of the experimentally measured responses can be defined formally as follows:

$$r_i^e\triangleq {\displaystyle \underset{D_e}{\int }{r}_i{p}_e\left(\alpha ,r\right)d}rd\alpha ,i=1,\dots ,TR.$$

(93)

$$\mathrm{cov}{\left(r_i,r_j\right)}_e\triangleq {\displaystyle \underset{D_e}{\int }\left(r_i-r_i^e\right)\left(r_j-r_j^e\right)p_e\left(\alpha ,r\right)d}rd\alpha ;i,j=1,\dots ,TR.$$

(94)

The expected values of the measured responses are considered to constitute the components of a vector denoted as $r_e\triangleq {\left(r_1^e,\dots ,r_{TR}^e\right)}^{†}$. The covariances (i.e., standard deviations and correlations) of the measured responses are considered to be components of the $TR\times TR$-dimensional covariance matrix of measured responses, which are denoted as $C_e^{rr}\triangleq {\left[\mathrm{cov}{\left(r_i,r_j\right)}_e\right]}_{TR\times TR}$. Similarly, the mean values and covariances for the additional measurements of model parameters can be formally represented as follows:

$${\alpha }_j^e\triangleq {\displaystyle \underset{D_e}{\int }{\alpha }_j{p}_e\left(\alpha ,r\right)d}rd\alpha ,j=1,\dots ,TP.$$

(95)

The expected values of the measured parameters are considered to constitute the components of a vector denoted as ${\alpha }_e\triangleq {\left({\alpha }_1^e,\dots ,{\alpha }_{TR}^e\right)}^{†}$. The covariance, $\mathrm{cov}{\left({\alpha }_i,{\alpha }_j\right)}_e$, of two independently/newly measured parameters, ${\alpha }_i$ and ${\alpha }_j$, is defined as follows:

$$\mathrm{cov}{\left({\alpha }_i,{\alpha }_j\right)}_e\triangleq {\displaystyle \underset{D_e}{\int }\delta {\alpha }_i^e\delta {\alpha }_j^e{p}_e\left(\alpha ,r\right)d}rd\alpha ;i,j=1,\dots ,TP;\delta {\alpha }_i^e\triangleq {\alpha }_i-{\alpha }_i^e.$$

(96)

The covariances $\mathrm{cov}{\left({\alpha }_i,{\alpha }_j\right)}_e$ would constitute the components of the covariance matrix, denoted as $C_e^{\alpha \alpha }$, of newly measured parameters. When correlations, denoted as $\mathrm{cov}{\left({\alpha }_i,r_j\right)}_e$, between the measured responses and the model parameters are available, they can formally be considered to be elements of a (very sparse, in practice) rectangular correlation matrix which is denoted as $C_e^{\alpha r}\triangleq {\left[\mathrm{cov}{\left({\alpha }_i,r_j\right)}_e\right]}_{TP\times TR}$, where

$$\mathrm{cov}{\left({\alpha }_i,r_j\right)}_e\triangleq {\displaystyle \underset{D_e}{\int }\left({\alpha }_i-{\alpha }_i^e\right)\left(r_j-r_j^e\right)p_e\left(\alpha ,r\right)d}rd\alpha ;i=1,\dots ,TP;j=1,\dots ,TR.$$

(97)

The MaxEnt principle [17] can now be applied to construct the least informative (and hence, most conservative) distribution using the information provided by the vector of mean experimental values $r_e$ and covariance matrices $C_e^{rr}\triangleq {\left[\mathrm{cov}{\left(r_i,r_j\right)}_e\right]}_{TR\times TR}$ and $C_e^{r\alpha }\triangleq {\left[\mathrm{cov}{\left({\alpha }_i,r_j\right)}_e\right]}_{TP\times TR}$. This MaxEnt distribution for the experimentally measured responses is denoted as $p_e\left(z\left|z_e,C_e\right.\right)$ and is constructed by applying the same steps as those leading to Equation (92), above, to obtain the following expression:

$$p_e\left(z\left|z_e,C_e\right.\right)=\frac{{\left(2\pi \right)}^{-N/2}}{\sqrt{Det\left(C_e\right)}}\exp \left[-\frac{1}{2}{\left(z-z_e\right)}^{†}{C}_e^{-1}\left(z-z_e\right)\right],$$

(98)

where

$$C_e\triangleq \left(\begin{array}{cc}{C}_e^{rr}& C_e^{r\alpha }\\ C_e^{\alpha r}& C_e^{\alpha \alpha }\end{array}\right);z=\left(\begin{array}{c}r\\ \alpha \end{array}\right);z_e=\left(\begin{array}{c}{r}_e\\ {\alpha }_e\end{array}\right).$$

(99)

3.3. Construction of the Complete Second-Order-Accurate Joint Posterior MaxEnt Probability Distbribution of Computed and Measured Responses and Model Parameters

The joint posterior probability distribution of all computed and experimentally measured quantities, which is denoted as $p_p\left(z\left|z_p,C_p\right.\right)$ where the subscript “p” indicates “posterior,” is obtained as the properly normalized product of the distributions $p_e\left(z\left|z_e,C_e\right.\right)$ and $p_c\left(z\left|z_c,C_c\right.\right)$. The posterior probability $p_p\left(z\left|z_p,C_p\right.\right)$ is defined on the domain $D_p$ obtained as the union of the computational and experimental domains, i.e., $D_p\triangleq D_c\cup D_e$. Straightforward, albeit tedious, algebraic computations lead to the following expression for $p_p\left(z\left|z_p,C_p\right.\right)$:

$$p_p\left(z\left|z_p,C_p\right.\right)\triangleq p_c\left(z\left|z_c,C_c\right.\right)p_e\left(z\left|z_e,C_e\right.\right)=K\exp \left[-\frac{1}{2}Q\left(z\left|z_p,C_p\right.\right)\right],$$

(100)

where the normalization constant $K$ and the quadratic form $Q\left(z_p,C_p\right)$ have the following expressions, respectively:

$$K\triangleq {\left(2\pi \right)}^{-N/2}{\left[Det\left(C_c+C_e\right)\right]}^{-1/2}\exp \left[-\frac{1}{2}{\left(z_c-z_e\right)}^{†}{\left(C_c+C_e\right)}^{-1}\left(z_c-z_e\right)\right];$$

(101)

$$Q\left(z\left|z_p,C_p\right.\right)\triangleq {\left(z-z_p\right)}^{†}{C}_p^{-1}\left(z-z_p\right);$$

(102)

$$C_p\triangleq {\left(C_c^{-1}+C_e^{-1}\right)}^{-1}=C_c-C_c{\left(C_c+C_e\right)}^{-1}{C}_c=C_e-C_e{\left(C_c+C_e\right)}^{-1}{C}_e;$$

(103)

$$z_p\triangleq C_p\left(C_c^{-1}{z}_c+C_e^{-1}{z}_e\right).$$

(104)

The expression obtained in Equation (100) provides the exact first-order (mean values) and second-order (variances and co-variances) of the most comprehensive combined distribution of computations and measurements of responses and parameters. In practice, such a comprehensive amount of experimental information is highly unlikely to be available. Furthermore, even if such massive amount of experimental information were available, the inversion of the matrix ${\left(C_c+C_e\right)}^{-1}$ would require massive computational resources for large systems involving many parameters.

3.4. Practical Situation: Only Response Measurements Are Available to Be Assimilated

In practice, the information (mean values and covariances) about the model parameters indicated in Equations (66)–(68) is obtained prior to using this information in a model and, hence, for computing responses using the respective model. Additional experimental information would usually be available only about the responses of interest. Any experimental information that might be available about the model parameters and/or about correlations between some model parameters and some responses would need to be assessed against the similar information already available in the covariance/correlation matrices $C^{\alpha \alpha }$ and/or $C_c^{r\alpha }$, so the respective experimental and computational information could be combined into the appropriate components of these matrices. In practice, therefore, only information that becomes additionally available about measured responses would need to be assimilated explicitly, i.e., only the components of the vector $r_e\triangleq {\left(r_1^e,\dots ,r_{TR}^e\right)}^{†}$ and of the matrix $C_e^{rr}\triangleq {\left[\mathrm{cov}{\left(r_i,r_j\right)}_e\right]}_{TR\times TR}$ would become available for assimilation and predictive modeling. These measured responses would be considered to follow an unknown probability distribution function, denoted as $p_e\left(r\right)$ and defined on a domain denoted as $D_e\in {ℝ}^{TR}$, so that the expressions in Equations (93) and (94) would take on the following forms, respectively:

$$r_i^e\triangleq {\displaystyle \underset{D_e}{\int }{r}_i{p}_e\left(r\right)d}r,i=1,\dots ,TR.$$

(105)

$$\mathrm{cov}{\left(r_i,r_j\right)}_e\triangleq {\displaystyle \underset{D_e}{\int }\left(r_i-r_i^e\right)\left(r_j-r_j^e\right)p_e\left(r\right)d}r;i,j=1,\dots ,TR.$$

(106)

The MaxEnt distribution corresponding to the information provided in Equations (105) and (106) has the following expression:

$$p_e\left(r\left|r_e,C_e^{rr}\right.\right)={\left(2\pi \right)}^{-TR/2}{\left[Det\left(C_e^{rr}\right)\right]}^{-1/2}\exp \left[-\frac{1}{2}{\left(r-r_e\right)}^{†}{\left(C_e^{rr}\right)}^{-1}\left(r-r_e\right)\right].$$

(107)

Furthermore, when only the experimental information represented by the distribution $p_e\left(r\left|r_e,C_e^{rr}\right.\right)$ is available, the posterior joint probability distribution of the computed and measured quantities, which is denoted as $p_p\left(r,\alpha \right)=p_c\left(z\left|z_c,C_c\right.\right)p_e\left(r\left|r_e,C_e^{rr}\right.\right)$ and which is considered to be defined on the corresponding domain $D_p$, takes on the following form:

$$\begin{array}{l}{p}_p\left(r,\alpha \right)=N\exp \left[-\frac{1}{2}Q\left(r,\alpha \right)\right];N\triangleq {\left[{\displaystyle \underset{D_p}{\int }\exp \left[-\frac{1}{2}Q\left(r,\alpha \right)\right]d\alpha dr}\right]}^{-1};\\ Q\left(r,\alpha \right)\triangleq {\left(z-z_c\right)}^{†}{C}_c^{-1}\left(z-z_c\right)+{\left(r-r_e\right)}^{†}{\left(C_e^{rr}\right)}^{-1}\left(r-r_e\right).\end{array}$$

(108)

The posterior distribution represented by Equation (108) is evidently not Gaussian. The posterior mean values for the best-estimate predicted responses and parameters are defined as before, namely:

$$r_{be}\triangleq {\left[{\displaystyle \underset{D_p}{\int }{p}_p\left(\alpha ,r\right)d\alpha dr}\right]}^{-1}{\displaystyle \underset{D_p}{\int }r{p}_p\left(\alpha ,r\right)d\alpha dr};$$

(109)

$${\alpha }_{be}\triangleq {\left[{\displaystyle \underset{D_p}{\int }{p}_p\left(\alpha ,r\right)d\alpha dr}\right]}^{-1}{\displaystyle \underset{D_p}{\int }\alpha p_p\left(\alpha ,r\right)d\alpha dr};$$

(110)

The evaluation of the integrals appearing in Equations (109) and (110) can be performed to a high degree of accuracy, with a controlled error, by employing the saddle-point (Laplace) method [44,45,46]. For a ratio of integrals of the form

$$I={\left[{\displaystyle \int \exp \left[-g\left(z\right)\right]dz}\right]}^{-1}{\displaystyle \int f\left(z\right)\exp \left[-g\left(z\right)\right]dz},$$

(111)

the saddle-point (Laplace) method yields the following result [44,45,46]:

$$I=\widehat{f}-\frac{1}{2}{\widehat{f}}_{i_1}{\widehat{g}}_{i_2{i}_3{i}_4}{\widehat{g}}^{i_1{i}_2}{\widehat{g}}^{i_3{i}_4}+\frac{1}{2}{\widehat{f}}_{j_1{j}_2}{\widehat{g}}^{j_1{j}_2}+O\left({\widehat{f}}_{j_1{j}_2{j}_3}\right)$$

(112)

where:

(i)

the derivative of a function with respect to a component of $z$ is denoted using a subscript, e.g., $f_i\triangleq \partial f/\partial z_i$, $f_{ij}\triangleq {\partial }^{2}f/\partial z_i\partial z_j$, $i,j=1,\dots ,TI$, where $TI$ denotes the total number of independent variables;

(ii)

the superscripts denote the respective component of the inverse Hessian of the respective function, e.g., $f^{ij}$ denotes the $\left(i,j\right)$-element of the inverse Hessian matrix $\left[f_{ij}^{-1}\right]$;

(iii)

an index that appears as a subscript and a superscript implies a summation over all possible values of that index;

(iv)

the “hat” denotes that the respective quantity is evaluated at the saddle point of $\exp \left[-g\left(z\right)\right]$, which is defined as the point at which the gradient $g_z\left(z\right)\triangleq \left(\partial g/\partial z_1,\dots ,\partial g/\partial z_{TI}\right)$ of $g\left(z\right)$ vanishes, i.e., $g_z\left(z\right)=0$.

The saddle-point of $p_p\left(r,\alpha \right)$ is denoted as $\left(r_s,{\alpha }_s\right)$ and is defined by the following relations:

$$\frac{\partial Q\left(r,\alpha \right)}{\partial r}=0,\frac{\partial Q\left(r,\alpha \right)}{\partial \alpha }=0,at\left(r,\alpha \right)=\left(r_s,{\alpha }_s\right).$$

(113)

To obtain the partial gradients (differentials) shown in Equation (113), it is convenient to write the matrix $C_c^{-1}$ in the form $C_c^{-1}=\left(\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& C_{22}\end{array}\right)$, and use this form together with Equations (99) and (83) in Equation (108) to expand the functional $Q\left(r,\alpha \right)$ in the following form:

$$\begin{array}{l}Q\left(r,\alpha \right)={\left[r-E_c\left(r\right)\right]}^{†}{C}_{11}\left[r-E_c\left(r\right)\right]+{\left[r-E_c\left(r\right)\right]}^{†}{C}_{12}\left(\alpha -{\alpha }^0\right)\\ +{\left(\alpha -{\alpha }^0\right)}^{†}{C}_{21}\left[r-E_c\left(r\right)\right]+{\left(\alpha -{\alpha }^0\right)}^{†}{C}_{22}\left(\alpha -{\alpha }^0\right)+{\left(r-r_e\right)}^{†}{\left(C_e^{rr}\right)}^{-1}\left(r-r_e\right).\end{array}$$

(114)

Taking the partial differentials of the expression in Equation (114) yields the following equation at $\left(r,\alpha \right)=\left(r_s,{\alpha }_s\right)$:

$$\left(\begin{array}{c}{r}_s-E_c\left(r\right)\\ {\alpha }_s-{\alpha }^0\end{array}\right)=-\left(\begin{array}{cc}{C}_c^{rr}& C_c^{r\alpha }\\ C_c^{\alpha r}& C^{\alpha \alpha }\end{array}\right)\left(\begin{array}{c}{\left(C_e^{rr}\right)}^{-1}\left(r_s-r_e\right)\\ 0\end{array}\right).$$

(115)

Solving Equation (115) leads to the following expressions for the saddle point coordinates $r_s$ and ${\alpha }_s$:

$$r_s=r_e+C_e^{rr}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}\left[E_c\left(r\right)-r_e\right],$$

(116)

$${\alpha }_s={\alpha }^0-C_c^{\alpha r}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}\left[E_c\left(r\right)-r_e\right].$$

(117)

Using the results obtained in Equations (112), (116) and (117) in Equations (109) and (110), respectively, yields the following results for the best-estimate predicted values for the responses and, respectively, calibrated parameters:

$$r_{be}=r_s=r_e+C_e^{rr}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}\left[E_c\left(r\right)-r_e\right];$$

(118)

$${\alpha }_{be}={\alpha }_s={\alpha }^0-C_c^{\alpha r}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}\left[E_c\left(r\right)-r_e\right].$$

(119)

Since the components of the vector $E_c\left(r\right)$, and the components of the matrices $C_c^{rr}$ and $C_c^{\alpha r}$ can contain arbitrarily high-order response sensitivities to model parameters, as shown in Appendix A, the formulas presented in (118) and (119) generalize all of the previous formulas of this type found in data adjustment/assimilation procedures published to date (which contain at most second-order sensitivities). The best-estimate parameter values are the “calibrated model parameters” which can be used for subsequent computations with the “calibrated model”.

The second-order moments of the posterior distribution $p_p\left(r,\alpha \right)$ comprise the covariances between the best-estimated response, which are denoted as $C_{be}^{rr}$, the covariances between the best-estimate parameters, which are denoted as $C_{\alpha \alpha }^{be}$, and the correlations between the best-estimate parameters and responses, which are denoted as $C_{\alpha r}^{be}$. The expression of the “best-estimate” posterior parameter covariance matrix $C_r^{be}$ for the best-estimate responses $r_{be}$ is derived by using the results given in (118) and (119) to obtain the following expressions:

$$C_{be}^{rr}\triangleq {\displaystyle \underset{D_p}{\int }\left(r-r_{be}\right){\left(r-r_{be}\right)}^{†}{p}_p\left(\alpha ,r\right)d\alpha dr}=C_e^{rr}-C_e^{rr}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}{C}_e^{rr}.$$

(120)

The following important result has been used to obtain the final expression provided in Equation (120):

$$\left[E_c\left(r\right)-r_e\right]{\left[E_c\left(r\right)-r_e\right]}^{†}=\left[r-r_e-r+E_c\left(r\right)\right]{\left[r-r_e-r+E_c\left(r\right)\right]}^{†}=C_e^{rr}+C_c^{rr}.$$

(121)

As indicated in Equation (120), the initial covariance matrix $C_e^{rr}$ for the experimentally measured responses is multiplied by the matrix $\left[I-{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}{C}_e^{rr}\right]$, which means that the variances contained on the diagonal of the best-estimate matrix $C_{be}^{rr}$ are smaller than the experimentally measured variances contained in $C_e^{rr}$. Hence, the incorporation of experimental information reduces the predicted best-estimate response variances in $C_{be}^{rr}$ by comparison to the measured variances contained a priori in $C_e^{rr}$. Since the components of the matrix $C_{be}^{rr}$ contain high-order sensitivities, the formula presented in Equation (120) generalizes all of the previous formulas of this type found in data adjustment/assimilation procedures published to date (which contain at most first-order sensitivities).

The expression of the “best-estimate” posterior parameter covariance matrix $C_{be}^{\alpha \alpha }$ for the best-estimate parameters ${\alpha }_{be}$ is derived by using the result given in Equation (119) to obtain:

$$C_{be}^{\alpha \alpha }\triangleq {\displaystyle \underset{D_p}{\int }\left(\alpha -{\alpha }_{be}\right){\left(\alpha -{\alpha }_{be}\right)}^{†}{p}_p\left(\alpha ,r\right)d\alpha dr}=C^{\alpha \alpha }-C_c^{\alpha r}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}{C}_c^{r\alpha }.$$

(122)

The matrices $C^{\alpha \alpha }$ and $C_c^{\alpha r}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}{C}_c^{r\alpha }$ are symmetric and positive definite. Therefore, the subtraction indicated in Equation (122) implies that the components of the main diagonal of $C_{be}^{\alpha \alpha }$ must have smaller values than the corresponding elements of the main diagonal of $C^{\alpha \alpha }$. In this sense, the combination of computational and experimental information has reduced the best-estimate parameter variances on the diagonal of $C_{be}^{\alpha \alpha }$. Since the components of the matrices $C^{\alpha \alpha }$, $C_c^{\alpha r}$, and $C_c^{rr}$ contain high-order response sensitivities, the formula presented in Equation (122) generalizes all of the previous formulas of this type found in data adjustment/assimilation procedures published to date (which contain at most first-order sensitivities).

The expression of the “best-estimate” posterior parameter correlation matrix $C_{be}^{\alpha r}$ and/or its transpose $C_{be}^{r\alpha }$, for the best-estimate parameters ${\alpha }_{be}$ and best-estimate responses $r_{be}$, are derived by using the results given in Equations (118) and (119) to obtain the following expressions:

$$C_{be}^{\alpha r}\triangleq {\displaystyle \underset{D_p}{\int }\left(\alpha -{\alpha }_{be}\right){\left(r-r_{be}\right)}^{†}{p}_p\left(\alpha ,r\right)d\alpha dr}=C_c^{\alpha r}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}{C}_e^{rr};$$

(123)

$$C_{be}^{r\alpha }\triangleq {\displaystyle \underset{D_p}{\int }\left(r-r_{be}\right){\left(\alpha -{\alpha }_{be}\right)}^{†}{p}_p\left(\alpha ,r\right)d\alpha dr}=C_e^{rr}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}{C}_c^{r\alpha }={\left(C_{be}^{\alpha r}\right)}^{†}.$$

(124)

Since the components of the matrices $C_c^{\alpha r}$ and $C_c^{rr}$ contain high-order sensitivities, the formulas presented in Equations (123) and (124) generalize all of the previous formulas of this type found in data adjustment/assimilation procedures published to date (which contain at most first-order sensitivities).

It is important to note from the results shown in (118)–(124) that the computation of the best estimate parameter and response values, together with their corresponding best-estimate covariance matrices, only requires the computation of ${\left(C_e^{rr}+C_c^{rr}\right)}^{-1}$, which entails the inversion of a matrix of size $TR\times TR$. This is computationally very advantageous, since $TR\ll TP$, i.e., the number of responses is much less than the number of model parameters in the overwhelming majority of practical situations.

The minimum value, $Q_{\min }=Q\left({\alpha }_{be},r_{be}\right)$, of the quadratic form $Q\left(\alpha ,r\right)$, takes on the following expression, which is obtained by using Equation (115):

$$\begin{array}{l}{Q}_{\min }={\chi }^2={\left(\begin{array}{c}{r}_{be}-E_c\left(r\right)\\ {\alpha }_{be}-{\alpha }^0\end{array}\right)}^{†}{\left(\begin{array}{cc}{C}_c^{rr}& C_c^{r\alpha }\\ C_c^{\alpha r}& C^{\alpha \alpha }\end{array}\right)}^{-1}\left(\begin{array}{c}{r}_{be}-E_c\left(r\right)\\ {\alpha }_{be}-{\alpha }^0\end{array}\right)\\ +{\left(r_{be}-r_e\right)}^{†}{\left(C_e^{rr}\right)}^{-1}\left(r_{be}-r_e\right)={\left[E_c\left(r\right)-r_e\right]}^{†}{\left(C_e^{rr}+C_c^{rr}\right)}^{-1}\left[E_c\left(r\right)-r_e\right].\end{array}$$

(125)

As the expression obtained in Equation (125) indicates, the quantity $Q_{\min }$ represents the square of the length of the vector $\left[E_c\left(r\right)-r_e\right]$, measuring (in the corresponding metric) the deviations between the experimental and nominally computed responses. The quantity $Q_{\min }$ can be evaluated directly from the given data (i.e., model parameters and computed and measured responses, together with their original uncertainties) after having computed the matrix ${\left(C_e^{rr}+C_c^{rr}\right)}^{-1}$. As the dimension of the vector $\left[E_c\left(r\right)-r_e\right]$ indicates, the number of degrees of freedom characteristic of the calibration under consideration is equal to the number $TR$ of experimental responses. It is important to note that $Q_{\min }$ is independent of calibrating (or adjusting) the original data. In the extreme case of absence of experimental responses, no actual calibration takes place. An actual calibration (adjustment) occurs only when including at least one experimental response.

The variate $Q_{\min }$ follows a ${\chi }^2$-distribution with $TR$ degrees of freedom, where $TR$ denotes the total number of experimental responses considered in the assimilation/calibration (adjustment) procedure. Thus, the quantity $Q_{\min }$ is the “${\chi }^2$ of the calibration at hand” and can be used as an indicator of the agreement between the computed and measured responses (which depend on the model parameters). Recall that the ${\chi }^2$-distribution is a measure of the deviation of a “true distribution” (in this case, the distribution of experimental responses) from the hypothetic one (in this case, a MaxEnt distribution constructed from the available information regarding mean values and correlations/covariances). For predictive modeling it is important to assess if: (i) the response and data measurements are free of gross errors (blunders such as wrong settings, mistaken readings, etc.), and (ii) the measurements are consistent with the assumptions regarding the respective means, variances, and covariances.

3.5. Characteristics of the BERRU-PM-2+ Methodology: Summary

The end-results of BERRU-PM-2+ methodology introduced in Section 3.4, above, are the expressions presented in Equations (118)−(125). An examination of these expressions highlights the fact that they can be utilized both for “forward/direct predictive modeling” and for “inverse predictive modeling”. The “forward” or “direct problem” solves the “parameter-to-output” mapping that describes the “cause-to-effect” relationship in the physical process being modeled. The “inverse problem” attempts to solve the “output-to-parameters” mapping. In particular, “measurement problems” are “inverse” to the direct problem, in that they seek to determine (from measurements) the properties of the host medium (e.g., composition, geometry, including internal interfaces) or the properties of the source (e.g., strength, location, direction) and/or the size of the medium on its boundaries. Such inverse problems are encountered in fields as diverse as astrophysics (in which one measures the intensity and spectral distribution of light in order to infer properties of starts), nuclear medicine (where radioisotopes are injected into patients and the radiation emitted is used in diagnostics to reconstruct body properties, e.g., tumors), non-destructive fault detection in materials, underground (oil, water) logging, and detection of sensitive materials. Some authors further group such inverse problems into “invasive”, when the interior particle distribution is accessible for measurements, as opposed to “non-invasive” ones, in which only particle distributions on the boundaries of (or exterior to) the medium can be measured.

Inverse problems are fundamentally ill-posed and/or ill-conditioned, unstable to uncertainties in the model parameters and/or the experimental measurements. In particular, inverse problems involving differential operators are ill-posed because the differentiation operator is not continuous with respect to any physically meaningful observation topology. The existence of a solution for an inverse problem is in most cases secured by defining the data space to be the set of solutions to the direct problem. This approach may fail if the data is incomplete, perturbed, or noisy. If the uniqueness of a solution cannot be secured from the given data, additional data and/or a priori knowledge about the solution need to be used to restrict the set of admissible solutions. In particular, stability of the solution is the most difficult to ensure and verify. If an inverse problem fails to be stable, then small round-off errors or noise in the data will amplify to a degree that renders a computed solution useless. The procedures used to solve approximately an ill-posed problem are called “regularization” procedures and are customarily categorized as “explicit” or “implicit”; see, e.g., [47,48,49]. The historically older explicit methods attempt to manipulate the forward model equations in conjunction with measurements in order to estimate explicitly the unknown source and/or other unknown characteristics of the medium. On the other hand, implicit methods combine measurements with repeated solutions of the direct problem obtained with different values of the unknowns, iterating until an a priori selected functional, usually representing the user-defined “goodness of fit” between measurements and direct computations, is reduced to a value deemed to be “acceptable” by the user.

Since the framework of BERRU-PM methodology (including its present extension, BERRU-PM-2+) comprises the combined phase-space of parameters and responses, it can be used for solving both forward/direct and inverse problems. The solution of forward/direct problems is provided by the expression for the predicted best-estimate response given in Equation (116) together with the corresponding reduced predicted uncertainties provided by Equation (120). The remaining expressions for the best-estimate predicted parameters and their reduced predicted uncertainties provided in Equations (117), (122), along with the parameter-response correlations provided in Equation (123) and the “goodness of fit indicator” provided by Equation (125) are additional features which the BERRU-PM methodology provides. Conversely, the solution of “inverse problems” is provided by the expressions for the best-estimate predicted parameters and their reduced predicted uncertainties provided in Equations (117) and (122). The superior accuracy provided by the BERRU-PM methodology (by comparison to the traditional methods based on the minimization of chi-square/least-squares-type functionals for solving inverse problems) has been demonstrated in [16] for the inverse prediction, from detector responses in the presence of counting uncertainties, of the thickness of a homogeneous slab of material containing uniformly distributed gamma-emitting sources. In particualar, it has been shown in [16] that for optically very thin slabs, both the traditional chi-square-minimization methods and the BERRU-PM methodology predict the slab’s thickness accurately. For optically thick slabs, however, the traditional inverse-problem methods based on the minimization of chi-square-type functionals fail to predict the slab’s thickness, while the BERRU-PM methodology correctly predicts the slab’s actual physical thickness when precise experimental results are assimilated, while also predicting the physically correct response within the selected precision criterion.

As shown in Appendix A, the expressions of the moments of the joint distribution of computed responses and model parameters defined in Equations (70)−(72), in terms of the moments of the distribution of model parameters, explicitly include the third- and higher-order sensitivities of responses with respect to the model parameters, as well as the systematic and random errors that would stem from incomplete modeling and numerical truncations. The expressions provided in Appendix A also indicate the path for incorporating explicitly higher-order sensitivities, if available. Since BERRU-PM-2+ mathematical framework introduced in Section 3 of this work enables the use of higher-order sensitivities, it is expected that the BERRU-PM-2+ methodology will improve considerably the already unparalleled efficiency and accuracy of the extant BERRU-PM methodology.

4. Discussion, Conclusions and Outlook

Although they are not identical, the data adjustment and data-assimilation methodologies share several fundamental common features which stem from their common root in least-square minimization procedures, so the expressions of the end-results produced by these methodologies look similar. Neither of these methodologies use the complete first-order information, however. The data-adjustment methodology is incomplete in first-order because it has no provisions for incorporating the first-order correlations between the model parameters and the computed responses, which are defined by Equation (71) and which can be readily computed, as shown in Appendix A. Even though the provision for incorporating the correlations defined in Equation (4) between measured responses and model parameters is extant, this provision has not been used in applications thus far because of lack of actual data.

The applicability of the data-adjustment methodology is also restricted by the domain of applicability of the “generalized linear least-squares methodology” (GLLSM), which requires that: (i) the uncertainty of the calculated response (e.g., neutron multiplication factor) be dominated by uncertainties in the data (e.g., cross sections, neutron multiplicities, fission spectra), having negligeable systematic errors; and (ii) the data uncertainties be sufficiently small for approximating the response (e.g., neutron multiplication factor) by a first-order Taylor-series expansion about the values of the given data. The analysis performed in [50,51] indicated that neither condition is fulfilled in general, since (i) the linear perturbation assumption underlying the GLLSM predictions breaks down for relative standard deviations (in the data) larger than ca. 1%; and (ii) systematic errors larger than ca. 1% also cause the brake-down of the GLLSM predictions, since such systematic errors are not explicitly considered within the GLLSM formalism. Consequently, for real application cases, which are always characterized by systematic computational errors, the work reported in [50,51] indicates that true coverage probabilities of GLLSM confidence intervals for the computational bias may strongly differ from their expected confidence levels, particularly if a chi-square filter is used. The user-defined “cost functional” represented by Equation (1), which is minimized in the least-square sense, cannot be extended in an obvious manner to incorporate higher-order correlations and sensitivities. A fundamentally different type of “cost functional” would need to be introduced for incorporating higher-order correlations and sensitivities, the form of which has not been produced yet in the literature.

The data-assimilation methodology is also incomplete in first-order because it has no provisions for incorporating the first-order correlations between the model parameters and either the computed or the measured responses; specifically, neither the information represented by Equation (71) nor the information represented by Equation (97) can be incorporated in the user-defined cost functional underlying the extant data-assimilation methodologies. On the other hand, the currently most advanced 4D-VAR data-assimilation methodologies (including the extended/ensemble Kalman filter, second-order filter and variants thereof; see, e.g., references [11,12,13,14]) allow the incorporation of some partial second-order information through the so-called “Hessian-vector product” which appears as an additional term in the model dynamics/forecast step and the data assimilation step. Thus, although the most advanced data-assimilation methods can be considered to be “more inclusive”—in the sense of incorporating some partial second-order information—than the most advanced data-adjustment methods, neither of these methodologies are complete regarding the incorporation of all extant first- and/or second-order information, and neither methodology provides a mechanism (even a theoretical one) for incorporating response sensitivities (to model parameters) higher than first-order and/or incorporating correlations among responses and/or parameters higher than second-order. Of course, the predictions of the data adjustment or assimilation methodologies are also limited to first-order mean values and second-order covariances/correlations for parameters or responses.

The first-order (in response sensitivities) BERRU-PM methodology [16] is based on the MaxEnt principle rather than least-squares minimization, thereby eliminating the user-introduced preference (bias) in the definition of the functional “describing the discrepancies between computations and measurements” which is to be minimized. Although the end-products of the BERRU-PM procedure look superficially similar to those produced by the data adjustment and/or data assimilation procedure, the BERRU-PM procedure differs fundamentally from the data adjustment/assimilation procedures, both regarding the actual contents of its end-products and also the potential for further generalization to higher-order, which are inherent to the BERRU-PM methodology [16] but are lacking within the frameworks of the other aforementioned methodologies.

Extending the BERRU-PM methodology [16], the BERRU-PM-2+ methodology presented in Section 3 of this work also employs the MaxEnt principle instead of a user-defined “cost functional to be minimized”. Employing the MaxEnt principle not only eliminates the impact of the user-defined functional which predetermines the outcome of the respective methodology’s predictions, but also makes it possible to incorporate the complete first- and second-order information (sensitivities and correlations among parameters and responses, computed and measured) by considering the complete joint-phase-space of parameters, computed and measured responses. By setting its mathematical framework in this most-inclusive phase-space, the BERRU-PM-2+ methodology can be used for solving both forward/direct problems and inverse problems, not only for applications related to energy-systems but for applications in all fields that combine computational and experimental information to produce best-estimate predictions for results of interest (responses) while also calibrating/improving the models of the underlying physical systems. Furthermore, the novel MaxEnt-based BERRU-PM-2+ methodology introduced in Section 3 explicitly includes the third-and fourth-order response sensitivities to model parameters, and clearly indicates the path for incorporating explicitly higher-order sensitivities, if available. The BERRU-PM-2+ expressions also comprise quantities representing the systematic and random errors that would stem from incomplete modeling and numerical truncations. It is therefore expected that the BERRU-PM-2+ methodology will further improve the already unparalleled efficiency and accuracy of the extant first-order BERRU-PM methodology.

The data-assimilation methodology is widely used in the atmospheric and geophysical sciences. On the other hand, predictive modeling applications to energy-related fields are very few, with the highly notable exception of applications in the nuclear energy field, where the original data-adjustment methodology was conceived. The perspective on the available predictive modeling methodologies presented in this work and, in particular, the introduction in this work of the BERRU-PM-2+ methodology, also aim to motivate a much wider use of predictive modeling in the energy sciences for obtaining best-estimate results with reduced uncertainties. In practice, predictive modeling methodologies should be an integral part of model validation methodologies to be applied prior to model qualification and licensing considerations.

Ongoing work aims at extending the BERRU-PM-2+ methodology to provide information about the joint parameters/responses posterior distribution’s third-order (skewness) and fourth-order (kurtosis) moments. The incorporation of third- and fourth-order information (i.e., high-order response sensitivities along with skewness and kurtosis of parameters) will greatly increase the range of validity of predicted results, as well as reduce the uncertainties in these predictions. In particular, the envisaged new fourth-order (BERRU-PM-4) predictive modeling methodology will significantly impact the activities of model calibration and model validation against experiments. Further work will aim at quantifying the impact of ignorance (lack of knowledge) caused by non-modeled/missed phenomena in computational models of physical/biological systems. It is envisaged to demonstrate the potential impact of this new, high-order, comprehensive predictive modeling methodology by applications to reduce uncertainties in modeling predictions for energy and related systems.