Optimal Estimators of Cross-Partial Derivatives and Surrogates of Functions

Abstract

Computing cross-partial derivatives using fewer model runs is relevant in modeling, such as stochastic approximation, derivative-based ANOVA, exploring complex models, and active subspaces. This paper introduces surrogates of all the cross-partial derivatives of functions by evaluating such functions at N randomized points and using a set of L constraints. Randomized points rely on independent, central, and symmetric variables. The associated estimators, based on $NL$ model runs, reach the optimal rates of convergence (i.e., $\mathcal{O}\left(N^{-1}\right)$), and the biases of our approximations do not suffer from the curse of dimensionality for a wide class of functions. Such results are used for (i) computing the main and upper bounds of sensitivity indices, and (ii) deriving emulators of simulators or surrogates of functions thanks to the derivative-based ANOVA. Simulations are presented to show the accuracy of our emulators and estimators of sensitivity indices. The plug-in estimates of indices using the U-statistics of one sample are numerically much stable.

1. Introduction

Derivatives are relevant in modeling, such as inverse problems, first-order and second-order stochastic approximation methods [1,2,3,4], exploring complex mathematical models or simulators, derivative-based ANOVA (Db-ANOVA) or exact expansions of functions, and active subspaces. First-order derivatives or gradients are sometime available in modeling. Instances are (i) models defined via their rates of change with respect to their inputs; (ii) implicit functions defined via their derivatives [5,6]; (iii) cases listed in [7,8] and the references therein.

In FANOVA and sensitivity analysis user and developer communities (see, e.g., [9,10,11,12,13,14]), screening of input variables and interactions of high-dimensional simulators is often performed before building emulators of such models using Gaussian processes [15,16,17,18,19], polynomial chaos expansions and SS-ANOVA [20,21], or other machine learning approaches. Emulators are fast-evaluated models that better approximate complex and/or too-expansive simulators. Efficient variance-based screening methods rely on the upper bounds of generalized sensitivity indices, including Sobol’ indices (see [14,22,23,24,25,26] for independent inputs and [27,28,29] for non-independent variables). Such upper bounds require the computations of cross-partial derivatives, even for simulators for which these computations are time-demanding or impossible. Also, active subspaces rely on the first-order derivatives for performing dimension reduction and then for approximating complex models [30,31,32,33].

For functions with full interactions, all the cross-partial derivatives are used in the integral representations of the infinitesimal increment of functions [34], and in the unanchored decompositions of functions in the Sobolev space [35]. Recently, such derivatives have become crucial in the Db-ANOVA representation of every smooth function, such as high-dimensional PDE models. Indeed, it is known, in [14], that every smooth function f admits an exact Db-ANOVA decomposition, that is, $\forall \mathbf{x}\in \Omega \subseteq {\mathbb{R}}^d$,

$${\displaystyle f\left(\mathbf{x}\right)=\mathbb{E}\left[f\left({\mathbf{X}}^{\prime }\right)\right]+\sum _{\begin{array}{c}v\subseteq \{1,\dots ,d\}\\ \left|v\right|>0\end{array}}{\mathbb{E}}_{{\mathbf{X}}^{\prime }}\left[{\mathcal{D}}^{\left|v\right|}f\left({\mathbf{X}}^{\prime }\right)\prod _{k\in v}\frac{G_k\left(X_k^{\prime }\right)-1{\mathrm{I}}_{[X_k^{\prime }\ge x_k]}}{g_k\left(X_k^{\prime }\right)}\right],}$$

(1)

where ${\mathcal{D}}^{\left|v\right|}f$ stands for the cross-partial derivative with respect to $x_k$ for any $k\in v$; ${\mathbf{X}}^{\prime }:=\left(X_1^{\prime },\dots ,X_d^{\prime }\right)$ is a random vector of independent variables, supported on an open $\Omega$ with margins $G_j$s and densities $g_j$s (i.e., $X_j^{\prime }\sim G_j,j=1,\dots ,d$).

Computing all the cross-partial derivatives using a few model runs or evaluations of functions is challenging. Direct computations of accurate cross-partial derivatives were considered in [36,37] using the generalization of Richardson’s extrapolation. Such approximations of all the cross-partial derivatives require a number of model runs that strongly depends on the dimensionality (i.e., d). While adjoint-based methods can provide the gradients for some POE/PDE-based models using only one simulation of the adjoint models [38,39,40,41,42,43], note that computing the Hessian matrix requires running d second-order adjoint models in general, provided that such models are available (see [38,44]).

Stochastic perturbations methods or Monte Carlo approaches have been used in stochastic approximations (see, e.g., [1,2,3,4]), including derivative-free optimization (see [2,3,4,45,46,47,48] and the references therein), for computing the gradients and Hessian of functions. Such approaches lead to the estimates of gradients using a number of model runs that can be less than the dimensionality [48,49]. While gradients computations and the convergence analysis are considered in the first-order stochastic approximations, estimators of the Hessian matrices are investigated in the second-order stochastic approximations [2,50,51,52,53]. Most of such approaches rely on the Taylor expansions of functions and randomized kernels and/or a random vector that is uniformly distributed on the unit sphere. Nevertheless, independent variables are used in [48,50,53], and the approaches considered in [53,54] rely on the Stein identity [55]. Note that the upper bounds of the biases of such approximations depend on the dimensionality, except in the work [48] for the gradients only. Moreover, the convergence analysis for more than the second-order cross-partial derivatives are not available according to our knowledge.

Given a smooth function defined on ${\mathbb{R}}^d$, the motivation of this paper consists of proposing new approaches for deriving surrogates of cross-partial derivatives and derivative-based emulators of functions that

• Are simple to use and generic by making use of d independent variables that are symmetrically distributed about zero and a set of constraints;
• Lead to dimension-free upper bounds of the biases related to the approximations of cross-partial derivatives for a wide class of functions;
• Provide estimators of cross-partial derivatives that reach the optimal and parametric rates of convergence;
• Can be used for computing all the cross-partial derivatives and emulators of functions at given points using a small number of model runs.

In this paper, new expressions of cross-partial derivatives of any order are derived in Section 3 by combining the properties of (i) the generalized Richardson extrapolation approach so as to increase the approximations accuracy, and (ii) the Monte Carlo approaches based only on independent random variables that are symmetrically distributed about zero. Such expressions are followed by their order of approximations and biases. We also derive the estimators of such new expressions and their associated mean squared errors, including the rates of convergence for some classes of functions (see Section 3.3). Section 3.4 provides the derivative-based emulators of functions, depending on the strength of the interactions or (equivalently) the cross-partial derivatives, thanks to Equation (1). The strength of the interactions can be assessed using sensitivity analysis. Thus, Section 4 deals with the derivation of new expressions of sensitivity indices and their estimators by making use of the proposed surrogates of cross-partial derivatives. Simulations based on test functions are considered in Section 5 to show the accuracy of our approach, and we conclude this work in Section 6.

2. Preliminary

For an integer $d\in \mathbb{N}\setminus \left\{0\right\}$, let $\mathbf{X}:=(X_1,\dots ,X_d)$ be a random vector of d independent and continuous variables with marginal cumulative distribution functions (CDFs) $F_j,j=1,\dots ,d$ and probability density functions (PDFs) ${\rho }_j,j=1,\dots ,d$.

For a non-empty subset $u\subseteq \{1,\dots ,d\}$, we use $\left|u\right|$ for its cardinality (i.e., the number of elements in u) and $(\sim u):=\{1,\dots ,d\}\setminus u$. Also, we use ${\mathbf{X}}_u:=(X_j,\forall j\in u)$ for a subset of inputs, and we have the partition $\mathbf{X}=({\mathbf{X}}_u,{\mathbf{X}}_{\sim u})$. Assume that:

Assumption 1 

(A1). $\mathbf{X}$ is a random vector of independent variables, supported on Ω.

Working with partial derivatives requires a specific mathematical space. Given an integer $n\in \mathbb{N}\setminus \left\{0\right\}$ and an open set $\Omega \subseteq {\mathbb{R}}^d$, consider a weak partial differentiable function $f:\Omega \to {\mathbb{R}}^n$ [56,57] and a subset $v\subseteq \{1,\dots ,d\}$ with $\left|v\right|>0$. Namely, we use ${\mathcal{D}}^{\left|v\right|}f:=\left({\prod }_{k\in v}\frac{\partial }{\partial x_k}\right)f$ for the $\left|v\right|$-th weak cross-partial derivatives of each component of f with respect to each $x_k$ with $k\in v$.

Likewise, given $\overrightarrow{\mathit{ı}}:=(i_1,\dots ,i_d)\in {\mathbb{N}}^d$, denote ${\mathcal{D}}^{\left(\overrightarrow{\mathit{ı}}\right)}f:=\left({\prod }_{k=1}^d\frac{{\partial }^{i_k}}{\partial x_k}\right)f$ and $1{\mathrm{I}}_v\left(j\right)=1$ if $j\in v$ and zero otherwise. Thus, taking $\overrightarrow{\mathit{v}}:=\left(1{\mathrm{I}}_v\left(1\right),\dots ,1{\mathrm{I}}_v\left(d\right)\right)$ yields ${\mathcal{D}}^{\left|v\right|}f={\mathcal{D}}^{\left(\overrightarrow{\mathit{v}}\right)}f$. Moreover, denote ${\left(\mathbf{x}\right)}^{\overrightarrow{\mathit{ı}}}={\mathbf{x}}^{\overrightarrow{\mathit{ı}}}:={\prod }_{k=1}^d{x}_k^{i_k}$, $\overrightarrow{\mathit{ı}}!:=i_1!\dots i_d!$, and consider the Hölder space of $\alpha$-smooth functions given by $\forall \mathbf{x},\mathbf{y}\in {\mathbb{R}}^d$

$${\displaystyle {\mathcal{H}}_{\alpha }:=\left\{f:{\mathbb{R}}^d\to \mathbb{R}:\left|f\left(\mathbf{x}\right)-\sum _{0\le i_1+\dots +i_d\le \alpha -1}\frac{{\mathcal{D}}^{\left(\overrightarrow{\mathit{ı}}\right)}f\left(\mathbf{y}\right)}{\overrightarrow{\mathit{ı}}!}{\left(\mathbf{x}-\mathbf{y}\right)}^{\overrightarrow{\mathit{ı}}}\right|\le M_{\alpha }{\left|\left|\mathbf{x}-\mathbf{y}\right|\right|}_2^{\alpha }\right\},}$$

with $\alpha \ge 1$, $M_{\alpha }>0$, and ${\mathcal{D}}^{\left(\overrightarrow{\mathit{ı}}\right)}f\left(\mathbf{y}\right)$ as weak cross-partial derivatives. We use ${\left|\left|·\right|\right|}_2$ for the Euclidean norm, $\left|\right|·{\left|\right|}_1$ for the $L_1$-norm, $\mathbb{E}(·)$ for the expectation, and $\mathbb{V}(·)$ for the variance.

3. Surrogates of Cross-Partial Derivatives and New Emulators of Functions

3.1. New Expressions of Cross-Partial Derivatives

This section aims at providing expressions of cross-partial derivatives using the model of interest, and new independent random vectors. We provide approximated expressions of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ for all $u\subseteq \{1,\dots ,d\}$ and the associated orders of approximations.

Given $L,q\in \mathbb{N}\setminus \left\{0\right\}$, consider ${\beta }_{\ell }\in \mathbb{R}$ with $\ell =1,\dots ,L$, $\mathit{h}:=(h_1,\dots ,h_d)\in {\mathbb{R}}_{+}^d$, and denote with $\mathbf{V}:=(V_1,\dots ,V_d)$ as d-dimensional random vectors of independent variables satisfying $\forall j\in \{1,\dots ,d\}$

$$\mathbb{E}\left[V_j\right]=0;\mathbb{E}\left[{\left(V_j\right)}^2\right]={\sigma }^2;\mathbb{E}\left[{\left(V_j\right)}^{2q+1}\right]=0;\mathbb{E}\left[{\left(V_j\right)}^{2q}\right]<+\infty .$$

Random vectors of d independent variables that are symmetrically distributed about zero are instances of $\mathbf{V}$, including the standard Gaussian random vector and symmetric uniform distributions about zero.

• Denote ${\beta }_{\ell }\mathit{h}\mathbf{V}:=({\beta }_{\ell }{h}_1{V}_1;\dots ,{\beta }_{\ell }{h}_d{V}_d)$. The reals ${\beta }_{\ell }$s are used for controlling the order of derivatives (i.e., $\left|u\right|$) we are interested in, while $V_j$s help in selecting one particular derivative of order $\left|u\right|$. Finally, $h_j$s aim at defining a neighborhood of a sample point $\mathbf{x}$ of $\mathbf{X}$ that will be used. Thus, using ${\beta }_{max}:=max\left(|{\beta }_1|,\dots ,|{\beta }_L|\right)$ and keeping in mind the variance of ${\beta }_{\ell }{h}_j{V}_j$, we assume that $\forall j\in \{1,\dots ,d\}$

Assumption 2 

(A2). ${\beta }_{max}{h}_j\sigma \le 1/2or,equivalently,0<{\beta }_{max}{h}_j\left|V_j\right|\le 1for bounded{V}_{j}s$.

Based on the above framework, Theorem 1 provides a new expression of the cross-partial derivatives of f. Recall that $\left|u\right|$ is the cardinality of u and ${\mathcal{D}}^{\left|u\right|}f={\mathcal{D}}^{\left(\overrightarrow{\mathit{u}}\right)}f$.

Theorem 1. 

Consider distinct ${\beta }_{\ell }$s, and assume that $f\in {\mathcal{H}}_{\alpha }$ with $\alpha \ge \left|u\right|+2L$ and (A2) holds. Then, for any $u\subseteq \{1,\dots ,d\}$ with $\left|u\right|>0$, there exists ${\alpha }_{\left|u\right|}\in \{1,\dots ,L\}$ and coefficients $C_1^{\left(\right|u\left|\right)},\dots ,C_L^{\left(\right|u\left|\right)}$ such that

$${\displaystyle {\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)=\sum _{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{x}+{\beta }_{\ell }\mathit{h}\mathbf{V}\right)\prod _{k\in u}\frac{V_k}{\left(h_k{\sigma }^2\right)}\right]+\mathcal{O}\left({\left|\left|\mathbf{h}\right|\right|}_2^{2{\alpha }_{\left|u\right|}}\right).}$$

(2)

Proof. 

The detailed proof is provided in Appendix A.    □

In view of Theorem 1, we are able to compute all the cross-partial derivatives using the same evaluations of functions with the same or different order of approximations, depending on the constraints imposed to determine the coefficients $C_1^{\left(\right|u\left|\right)},\dots ,C_L^{\left(\right|u\left|\right)}$ (see Appendix A). While the setting $L=1,{\beta }_1=1,C_1^{\left(\right|u\left|\right)}=1$ or the constraints ${\sum }_{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r}$; $r=0,\dots ,L-2,\left|u\right|,L-1\le \left|u\right|$ lead to the order $\mathcal{O}\left({\left|\left|\mathbf{h}\right|\right|}_2^2\right)$, one can increase that order up to $\mathcal{O}\left({\left|\left|\mathbf{h}\right|\right|}_2^{2L}\right)$ by using either ${\sum }_{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^{r+\left|u\right|}={\delta }_{0,r};r=0,2,\dots ,2(L-1)$ or the full constraints given by ${\sum }_{\ell =1}^{L+\left|u\right|}{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r};r=0,\dots ,2L+\left|u\right|-1$. The last setting is going to improve the approximations and numerical computations of derivatives. Since increasing the number of constraints requires more evaluations of simulators, and in ANOVA-like decomposition of $f\left(\mathbf{X}\right)$, it is common to neglect the higher-order components or, equivalently, the higher-order cross-partial derivatives thanks to Equation (1), the following parsimony number of constraints may be considered. Given an integer $r^{*}>0$, controlling the partial derivatives of order up to $r^{*}\le L-2$ can be performed using the constraints

$$\left\{\begin{array}{cc}\sum _{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r};r=0,1,\dots ,L-1\hfill & \mathrm{if}\left|u\right|=1,\dots ,r^{*}\hfill \\ \sum _{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r};r=0,\dots ,r^{*},\left|u\right|,\dots ,\left|u\right|+L-r^{*}-2\hfill & \mathrm{otherwise}\hfill \end{array}\right..$$

(3)

Equation (3) gives approximations of all the cross-partial derivatives of $\mathcal{O}\left({\left|\left|\mathbf{h}\right|\right|}_2^{{\alpha }_{\left|u\right|}}\right)$ where $o:=\left\{\begin{array}{cc}L-\left|u\right|& \mathrm{if}1\le \left|u\right|\le r^{*}\hfill \\ L-r^{*}-1& \mathrm{otherwise}\hfill \end{array}\right.$ and ${\alpha }_{\left|u\right|}=o$ if o is even and $o+1$ otherwise. This equation relies on the Vandermonde matrices and the generalized Vandermonde matrices, which ensure the existence and uniqueness of the coefficients for distinct values of ${\beta }_{\ell }$s (i.e., ${\beta }_{{\ell }_1}\ne {\beta }_{{\ell }_2}$) because the determinant is of the form ${\prod }_{1\le {\ell }_1<{\ell }_2\le L^{\prime }}\left({\beta }_{{\ell }_1}-{\beta }_{{\ell }_2}\right)$ (see [58,59] for more details and the inverse of such matrices).

Remark 1. 

When $L=1$, we must have ${\beta }_1=1$, $C_i^{\left(\right|u\left|\right)}=1,\forall u\subseteq \{1,\dots ,d\}$. Thus, the coefficient $C_i^{\left(\right|u\left|\right)}$ does not necessarily depend on $\left|u\right|$. Taking L for an even integer, the following nodes may be considered: $\left\{{\beta }_1,\dots ,{\beta }_L\right\}=\left\{\pm 2^k,k=0,\dots ,\frac{L-2}{2}\right\}$. When L is odd, one may add 0 to the above set. Of course, other possibilities can be considered provided that ${\sum }_{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^{\left|u\right|}=1$.

Remark 2. 

For a given $u_0\subseteq \{1,\dots ,d\}$, if we are only interested in all the ${\left|v\right|}^{th}$ cross-partial derivatives with $v\subseteq u_0$, it is better to set ${\mathit{h}}_{\sim u}=\mathbf{0}$ in Equation (2).

Remark 3. 

Links to other works.

Consider ${\beta }_1=-1,{\beta }_2=1,{\beta }_3=0$, and $V_j\sim \mathcal{N}(0,1)$ with $j=1,\dots ,d$. Using $L=1$ or $L=2$ or $L=3$, our estimators of the first-order and the second-order cross-partial derivatives are very similar to the results obtained in [53].

Using the uniform perturbations and $L=2$ and $L=3$, our estimators of the first-order and the second-order cross-partial derivatives are similar to those provided in [50]. However, we will see later that specific uniform distributions allow for obtaining dimension-free upper bounds of the biases.

3.2. Upper Bounds of Biases

To derive precise biases of our approximations provided in Theorem 1, different structural assumptions on the deterministic functions f and $\mathbf{V}$ are considered. Assume that $f\in {\mathcal{H}}_{\alpha }$ with $\alpha \ge \left|u\right|$ is sufficient to define ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ for any $u\subseteq \{1,\dots ,d\}$. Note that such an assumption does not depend on the dimensionality d. For the sequel of generality, we provide the upper bounds of the biases for any value of L by considering two sets of constraints.

Denote with $\mathbf{R}:=(R_1,\dots ,R_d)$ a d-dimensional random vector of independent variables that are centered about zero and standardized (i.e., $\mathbb{E}\left[R_k^2\right]=1$, $k=1,\dots ,d$), and ${\mathcal{R}}_c$ the set of such random vectors. For any $r\in \mathbb{N}$, define

$${\Gamma }_r:=\sum _{\ell =1}^L\left|C_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r\right|;K_{1,L}:=\underset{\mathbf{R}\in {\mathcal{R}}_c}{inf}\mathbb{E}\left[{\left|\left|{\mathbf{R}}^2\right|\right|}_2^L\prod _{k\in u}{R}_k^2\right]{\Gamma }_{\left|u\right|+2L}.$$

Corollary 1. 

Consider distinct ${\beta }_{\ell }$s and the constraints ${\sum }_{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^{\left|u\right|+r}={\delta }_{0,r}$ with $r=0,2,4,\dots ,2(L-1)$. If $f\in {\mathcal{H}}_{\left|u\right|+2L}$ and (A2) hold, then there is $M_{\left|u\right|+2L}>0$ such that

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\sum _{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{x}+{\beta }_{\ell }\mathit{h}\mathbf{V}\right)\prod _{k\in u}\frac{V_k}{h_k{\sigma }^2}\right]\right|\le {\sigma }^{2L}{M}_{\left|u\right|+2L}{K}_{1,L}{\left|\left|{\mathit{h}}^2\right|\right|}_2^L.}$$

(4)

Moreover, if $V_k\sim \mathcal{U}(-\xi ,\xi )$ with $\xi >0$ and $k=1,\dots ,d$, then

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\sum _{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{x}+{\beta }_{\ell }\mathit{h}\mathbf{V}\right)\prod _{k\in u}\frac{V_k}{h_k{\sigma }^2}\right]\right|\le M_{\left|u\right|+2L}{\xi }^{2L}\left|\right|{\mathit{h}}^2{\left|\right|}_1^L{\Gamma }_{\left|u\right|+2L}.}$$

(5)

Proof. 

See Appendix B for the detailed proof.    □

In view of Corollary 1, one obtains the upper bounds that do not depend on the dimensionality d by choosing $\xi =\sigma =\frac{1}{\sqrt{d}},h_k=h$ for instance. When ${\Gamma }_{\left|u\right|+2L}>1$, the choice $\xi =\sigma =d^{-\frac{1}{2}}{\left({\Gamma }_{\left|u\right|+2L}\right)}^{-\frac{1}{2L}}$ is more appropriate. Corollary 1 provides the results for highly smooth functions. To be able to derive the optimal rates of convergence for a wide class of functions (i.e., ${\mathcal{H}}_{\left|u\right|+1}$), Corollary 2 starts providing the biases for this class of functions under a specific set of constraints. To that end, define

$$K_1:=\underset{\mathbf{R}\in {\mathcal{R}}_c}{inf}\mathbb{E}\left[{\left|\left|\mathbf{R}\right|\right|}_2\prod _{k\in u}{R}_k^2\right].$$

Corollary 2. 

For distinct ${\beta }_{\ell }$s, consider $r^{*}\in \{0,\dots ,|u|-1\}$ and the constraints ${\sum }_{\ell =1}^{L=r^{*}+2}{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r}$ with $r=0,1,\dots ,r^{*},\left|u\right|$. If $f\in {\mathcal{H}}_{\left|u\right|+1}$ and (A2) hold, then there is $M_{\left|u\right|+1}>0$ such that

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\sum _{\ell =1}^{L=2}{C}_{\ell }^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{x}+{\beta }_{\ell }\mathit{h}\mathbf{V}\right)\prod _{k\in u}\frac{V_k}{h_k{\sigma }^2}\right]\right|\le \sigma M_{\left|u\right|+1}{K}_1{\left|\left|\mathit{h}\right|\right|}_2{\Gamma }_{\left|u\right|+1}.}$$

(6)

Moreover, if $V_k\sim \mathcal{U}(-\xi ,\xi )$ with $\xi >0$ and $k=1,\dots ,d$, then

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\sum _{\ell =1}^{L=2}{C}_{\ell }^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{x}+{\beta }_{\ell }\mathit{h}\mathbf{V}\right)\prod _{k\in u}\frac{V_k}{h_k{\sigma }^2}\right]\right|\le \xi M_{\left|u\right|+1}{\left|\right|\mathit{h}\left|\right|}_1{\Gamma }_{\left|u\right|+1}.}$$

(7)

Proof. 

See Appendix C.    □

Note that the upper bounds derived in Corollary 2 depend on $r^{*}\in \{0,\dots ,|u|-1\}$ through $L=r^{*}+2$ and ${\Gamma }_{\left|u\right|+1}={\sum }_{\ell =1}^{L=r^{*}+2}\left|C_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^{\left|u\right|+1}\right|$. Thus, taking $h_k=h$ and $\xi =1/\left(d{\Gamma }_{\left|u\right|+1}\right)$ will give a dimension-free upper bound that does not increase with $r^{*}$. The crucial role and importance of $r^{*}$ is highlighted in Section 3.3.

Remark 4. 

When $r^{*}=0$, which implies that $L=2$, consider ${\beta }_1=1,{\beta }_2=-1$; $C_1^{\left(\right|u\left|\right)}=1/2$; $C_2^{\left(\right|u\left|\right)}=1/2$ when $\left|u\right|$ is even and $C_2^{\left(\right|u\left|\right)}=-1/2$ otherwise. With the above choices, the upper bounds given by Equations (6) and (7) become, respectively,

$$\sigma M_{\left|u\right|+1}{K}_1{\left|\left|\mathit{h}\right|\right|}_2,\xi M_{\left|u\right|+1}{\left|\right|\mathit{h}\left|\right|}_1.$$

We can check that the same results hold when using $L=1$, ${\beta }_1=1$ and $C_1^{\left(\right|u\left|\right)}=1$.

Remark 5. 

It is worth noting that we obtain exact approximations of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ in Corollary 1 for the class of functions described by

$${\mathcal{B}}_0:=\left\{h\in {\mathcal{H}}_{\infty }:{\mathcal{D}}^{\left(\overrightarrow{\mathit{ı}}\right)}h=0,\forall \overrightarrow{\mathit{ı}}\in {\mathbb{N}}^{d}and\left|\right|\overrightarrow{\mathit{ı}}{\left|\right|}_1\ge \left|u\right|+2L\right\}.$$

In general, exact approximations of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ are obtained when $L\to \infty$ for highly smooth functions.

3.3. Convergence Analysis

Given a sample of $\mathbf{V}$, that is, ${\left\{{\mathbf{V}}_i:=\left(V_{i,1},\dots ,V_{i,d}\right)\right\}}_{i=1}^N$ and using Equation (2), the method of moments implies that the estimator of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ is given by

$${\displaystyle \widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right):=\frac{1}{N}\sum _{i=1}^N\sum _{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}f\left(\mathbf{x}+{\beta }_{\ell }\mathit{h}{\mathbf{V}}_i\right)\prod _{k\in u}\frac{V_{i,k}}{\left(h_k{\sigma }^2\right)}.}$$

Statistically, it is common to measure the quality of an estimator using the mean squared error (MSE), including the rates of convergence. The MSEs can also help in determining the optimal value of $\mathit{h}$. Theorem 2 provides such quantities under different assumptions. To that end, define

$$K_{2,r}:=\underset{\mathbf{R}\in {\mathcal{R}}_c}{inf}\mathbb{E}\left[{\left|\left|{\mathbf{R}}^2\right|\right|}_2^r\prod _{k\in u}{R}_k^2\right].$$

Theorem 2. 

For distinct ${\beta }_{\ell }$s, consider $r^{*}\in \mathbb{N}$ and ${\sum }_{\ell =1}^{L=r^{*}+2}{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r}$ with $r=0,1,\dots ,r^{*},\left|u\right|$ and $r^{*}\le \left|u\right|-1$. If $f\in {\mathcal{H}}_{\left|u\right|+1}$ and (A2) hold, then

$$\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le {\sigma }^2{M}_{\left|u\right|+1}^2{K}_1^2{\Gamma }_{\left|u\right|+1}^2{\left|\left|\mathit{h}\right|\right|}_2^2+\frac{M_{r^{*}+1}^2{\Gamma }_{r^{*}+1}^2{K}_{2,r^{*}+1}}{N{\sigma }^{2\left(\right|u|-r^{*}-1)}{\prod }_{k\in u}{h}_k^2}{\left|\left|{\mathit{h}}^2\right|\right|}_2^{r^{*}+1}.$$

(8)

Moreover, if $V_k\sim \mathcal{U}(-\xi ,\xi )$ with $k=1,\dots ,d$, then

$$.\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le M_{\left|u\right|+1}^2{\left|\right|\mathit{h}\left|\right|}_1^2{\xi }^2{\Gamma }_{\left|u\right|+1}^2+\frac{3^{\left|u\right|}{M}_{r^{*}+1}^2{\Gamma }_{r^{*}+1}^2}{N{\xi }^{2\left(\right|u|-r^{*}-1)}{\prod }_{k\in u}{h}_k^2}{\left|\left|\mathit{h}\right|\right|}_2^{2(r^{*}+1)}.$$

(9)

Proof. 

See Appendix D for the detailed proof.    □

Theorem 2 provides the upper bounds of MSEs for the anisotropic case. Using a uniform bandwidth, that is, $h_k=h$, reveals that such upper bounds clearly depend on the dimensionality of the function of interest. Indeed, we can check that the upper bounds of the MSEs provided in Equations (8) and (9) become, respectively,

$${\sigma }^2{M}_{\left|u\right|+1}^2{K}_1^2{\Gamma }_{\left|u\right|+1}^{2}d{h}^2+\frac{M_{r^{*}+1}^2{\Gamma }_{r^{*}+1}^2{K}_{2,r^{*}+1}}{N{\sigma }^{2\left(\right|u|-r^{*}-1)}{h}^{2\left(\right|u|-r^{*}-1)}}{d}^{\frac{r^{*}+1}{2}},$$

$${\xi }^2{M}_{\left|u\right|+1}^2{\Gamma }_{\left|u\right|+1}^2{d}^2{h}^2+\frac{3^{\left|u\right|}{M}_{r^{*}+1}^2{\Gamma }_{r^{*}+1}^2{d}^{r^{*}+1}}{N{\xi }^{2\left(\right|u|-r^{*}-1)}{h}^{2\left(\right|u|-r^{*}-1)}}.$$

By minimizing such upper bounds with respect to h, the optimal rates of convergence of the proposed estimators are derived in Corollary 3.

Corollary 3. 

Under the assumptions made in Theorem 2, if $r^{*}<\left|u\right|-1$, then

$$\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]=\mathcal{O}\left(N^{-\frac{1}{\left|u\right|-r^{*}}}{d}^{1+\frac{\left|u\right|-1}{\left|u\right|-r^{*}}}\right).$$

Proof. 

See Appendix E for the detailed proof.    □

The optimal rates of convergence obtained in Corollary 3 are far away from the parametric ones, and such rates decrease with $\left|u\right|$. Nevertheless, such optimal rates are function of $d^2$ for any $\left|u\right|\ge 2$ using $r^{*}=1$. The maximum rate of convergence that can be reached is $N^{-1/2}{d}^{\frac{\left|u\right|+1}{2}}$ by taking $r^{*}=\left|u\right|-2$.

To derive the optimal and parametric rates of convergence, let us now choose $r^{*}=\left|u\right|-1$ and $h_k=h$ with $k=1,\dots ,d$. Thus, we can see that the second terms of the upper bounds of the MSEs (provided in Theorem 2) are function of $d^{\left|u\right|}$, but they are independent of h. This key observation leads to Corollary 4.

Corollary 4. 

Under the assumptions made in Theorem 2, if $r^{*}=\left|u\right|-1$; $\xi =1/\left(d{\Gamma }_{\left|u\right|+1}\right)$ and $h_k=h\propto N^{-\gamma /2}$ with $\gamma \in [1,2]$, then we have

$$\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]=\mathcal{O}\left(N^{-1}{d}^{\left|u\right|}\right).$$

Proof. 

The proof is straightforward since $h^2\propto N^{-\gamma }$ and $Nh\to \infty$ if $N\to \infty$. □

It is worth noting that the upper bound of the squared bias obtained in Corollary 4 does not depend on the dimensionality thanks to $\xi$. Also, the optimal and parametric rates of convergence are reached by means of $LN=\left(\right|u|+1)N$ model evaluations, and such model runs can still be used for computing ${\mathcal{D}}^{\left|v\right|}f\left(\mathbf{x}\right)$ for every $v\subseteq \{1,\dots ,d\}$ with $\left|v\right|\le \left|u\right|$. Based on the same assumptions, it appears that the results provided in Corollary 4 are much more convenient for $\left|u\right|=1,2$ in higher dimensions, while those obtained in Corollary 3 are well suited for higher dimensions and for higher values of $\left|u\right|\in \{1,\dots ,d\}$.

For highly smooth functions and for large values of $\left|u\right|$ and d, we are able to derive intermediate rates of convergence of the estimator of ${\mathcal{D}}^{\left|u\right|}f$ (see Theorem 3). To that end, consider an integer $L^{\prime }$, $L=r^{*}+L^{\prime }+1$, and denote with $\left[b\right]$ the largest integer that is less than b for any real b.

Theorem 3. 

For an integer $r^{*}\le \left|u\right|-2$, consider ${\sum }_{\ell =1}^L{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r}$$r=0,1,\dots ,r^{*},\left|u\right|,$$\left|u\right|+2,\left|u\right|+4,\dots ,\left|u\right|+2(L^{\prime }-1)$. If $f\in {\mathcal{H}}_{\left|u\right|+2L^{\prime }}$ and (A2) hold, then

$$\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le {\sigma }^{4L^{\prime }}{M}_{\left|u\right|+2L^{\prime }}^2{K}_{1,L^{\prime }}^2{d}^{L^{\prime }}{h}^{4L^{\prime }}+\frac{M_{r^{*}+1}^2{\Gamma }_{r^{*}+1}^2{K}_{2,r^{*}+1}}{N{\sigma }^{2\left(\right|u|-r^{*}-1)}{h}^{2\left(\right|u|-r^{*}-1)}}{d}^{\frac{r^{*}+1}{2}}.$$

Moreover, if $V_k\sim \mathcal{U}(-\xi ,\xi )$ with $\xi >0$ and $k=1,\dots ,d$, then

$$\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le {\xi }^{4L^{\prime }}{M}_{\left|u\right|+2L^{\prime }}^2{\Gamma }_{\left|u\right|+2L^{\prime }}^2{d}^{2L^{\prime }}{h}^{4L^{\prime }}+\frac{3^{\left|u\right|}{M}_{r^{*}+1}^2{\Gamma }_{r^{*}+1}^2{d}^{r^{*}+1}}{N{\xi }^{2\left(\right|u|-r^{*}-1)}{h}^{2\left(\right|u|-r^{*}-1)}}.$$

For a given $0\le {ϵ}_{op}<1$, taking $L^{\prime }=\left[\frac{\left(\right|u|-r^{*}-1)(1-{ϵ}_{op})}{2{ϵ}_{op}}\right]$ leads to

$$\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]=\mathcal{O}\left(N^{-1+{ϵ}_{op}}{d}^{\left|u\right|(1-{ϵ}_{op})}\right).$$

(10)

Proof. 

Detailed proofs are provided in Appendix F.    □

It turns out that the optimal rate of convergence derived in Theorem 3 is a trade-off between the sample size N and the dimensionality d. For instance, when ${ϵ}_{op}=1/2$, the optimal rate becomes $\mathcal{O}\left(N^{-1/2}{d}^{\left|u\right|/2)}\right)$, which improves the rate obtained in Corollary 3, but under different assumptions.

Remark 6. 

Since the bias vanishes for the class of functions ${\mathcal{B}}_0$, taking $\sigma =1/h$ yields $\mathbb{E}\left[{\left(\widehat{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]=\mathcal{O}\left(N^{-1}\right)$ (see Appendix G). Note that such an optimal rate of convergence is dimension-free.

3.4. Derivative-Based Emulators of Smooth Functions

Using Equation (1) and bearing in mind the estimators of the cross-derivatives provided in Section 3.3, this section aims at providing surrogates of smooth functions, also known as emulators. The general expression of the surrogate of f is given below.

Corollary 5. 

For any $\left|u\right|\in \{1,\dots ,d\}$, consider ${\sum }_{\ell =1}^{L=5}{C}_{\ell }^{\left(\right|u\left|\right)}{\beta }_{\ell }^r={\delta }_{\left|u\right|,r}$, $r=0,1,\dots ,$$r^{*}=max\left(\right|u|-1,3),|u|$. Assume that $f\in {\mathcal{H}}_{d+1}$ and (A1) and (A2) hold. Then, an approximation of f at $\mathbf{x}\in \Omega$ is given by

$${\displaystyle \widehat{f}\left(\mathbf{x}\right):=\mathbb{E}\left[f\left({\mathbf{X}}^{\prime }\right)\right]+\sum _{\begin{array}{c}v\subseteq \{1,\dots ,d\}\\ \left|v\right|>0\end{array}}{\mathbb{E}}_{{\mathbf{X}}^{\prime }}\left[\widehat{{\mathcal{D}}^{\left|v\right|}f}\left({\mathbf{X}}^{\prime }\right)\prod _{k\in v}\frac{G_k\left(X_k^{\prime }\right)-1{\mathrm{I}}_{[X_k^{\prime }\ge x_k]}}{g_k\left(X_k^{\prime }\right)}\right]\stackrel{P}{\to }f\left(\mathbf{x}\right).}$$

(11)

The above plug-in estimator is consistent using the law of large numbers. For a given $\mathbf{x}$, it is worth noting that the choice of $G_j$s is arbitrary, provided that such distributions are supported on an open neighborhood of $\mathbf{x}$.

Often, the higher-order cross-partial derivatives or, equivalently, the higher-order interactions among the model inputs almost vanish, leading us to consider the truncated expressions. Given an integer s with $0<s\ll d$ and keeping in mind the ANOVA decomposition, consider the class of functions that admit at most the s-th-order interactions, that is, ${\mathcal{A}}_s:=\left\{h:{\mathbb{R}}^d\to R:h\left(\mathbf{x}\right)={\sum }_{\begin{array}{c}v\subseteq \{1,\dots ,d\}\\ \left|v\right|\le s\end{array}}{h}_v\left({\mathbf{x}}_v\right)\right\}$. Truncating the functional expansion is a standard practice within the ANOVA-community, that is, $s\ll d$ is assumed in higher dimensions [10,12]. For such a class of functions, requiring $f\in {\mathcal{H}}_{{\alpha }_s}$ with ${\alpha }_s\ge s$ is sufficient to derive our results. Thus, the truncated surrogate of f is given by

$${\displaystyle \widehat{f_s}\left(\mathbf{x}\right):=\mathbb{E}\left[f\left({\mathbf{X}}^{\prime }\right)\right]+\sum _{\begin{array}{c}v\subseteq \{1,\dots ,d\}\\ 0<\left|v\right|\le s\end{array}}{\mathbb{E}}_{{\mathbf{X}}^{\prime }}\left[\widehat{{\mathcal{D}}^{\left|v\right|}f}\left({\mathbf{X}}^{\prime }\right)\prod _{k\in v}\frac{G_k\left(X_k^{\prime }\right)-1{\mathrm{I}}_{[X_k^{\prime }\ge x_k]}}{g_k\left(X_k^{\prime }\right)}\right].}$$

Under the assumptions made in Corollary 5, $\widehat{f_s}\left(\mathbf{x}\right)$ reaches the optimal and parametric rate of convergence for the class of functions ${\mathcal{A}}_3$. For instance, taking $s=1$ leads to the first-order emulator of f, which relies only on the gradient information. Thus, $\widehat{f_{s=1}}$ provides accurate estimates of additive models of the form ${\sum }_{j=1}^d{h}_j\left(x_j\right)$, where $h_j$s are given functions. Likewise, $\widehat{f_2}$ allows for incorporating the second-order terms, but it requires the second-order cross-partial derivatives. Thus, it is relevant to find the class of functions ${\mathcal{A}}_s$ which contains the model of interest before building emulators. The following section deals with such issues.

4. Applications: Computing Sensitivity Indices

In high-dimensional settings, reducing the dimension of functions is often achieved by using screening measures, that is, measures that can be used for quickly identifying non-relevant input variables. Screening measures based on the upper bounds of the total sensitivity indices rely on derivatives [14,23,24,25,60,61]. This section aims at providing optimal computations of upper bounds of the total indices, followed by the computations of the main indices using derivatives.

By evaluating the function f given by Equation (1) at a random vector $\mathbf{X}$ using $G_j=F_j$, one obtains a random vector of the model outputs. Generalized sensitivity indices, including Sobol’s indices, rely on the variance–covariance of sensitivity functionals (SFs), which are also random vectors containing the information about the overall contributions of inputs [14,26,62]. The derivative-based expressions of SFs are given below (see [14] for more details). Given $u\subseteq \{1,\dots ,d\}$, the interaction SF of the inputs ${\mathbf{X}}_u$ is given by

$${\displaystyle f_u\left({\mathbf{X}}_u\right):={\mathbb{E}}_{{\mathbf{X}}^{\prime }}\left[{\mathcal{D}}^{\left|u\right|}f\left({\mathbf{X}}^{\prime }\right)\prod _{k\in u}\frac{F_k\left(X_k^{\prime }\right)-1{\mathrm{I}}_{[X_k^{\prime }\ge X_k]}}{{\rho }_k\left(X_k^{\prime }\right)}\right],}$$

and the first-order SF of ${\mathbf{X}}_u$ is given by

$${\displaystyle f_u^{fo}\left({\mathbf{X}}_u\right):=\sum _{\begin{array}{c}v,v\subseteq u\\ \left|v\right|>0\end{array}}{f}_v\left({\mathbf{X}}_v\right).}$$

Likewise, the total-interaction SF of ${\mathbf{X}}_u$ is given by [14]

$${\displaystyle f_u^{sup}\left(\mathbf{X}\right):=\sum _{\begin{array}{c}v\subseteq \{1,\dots ,d\}\\ u\subseteq v\end{array}}{f}_v\left({\mathbf{X}}_v\right)={\mathbb{E}}_{{\mathbf{X}}^{\prime }}\left[{\mathcal{D}}^{\left|u\right|}f\left({\mathbf{X}}_u^{\prime },{\mathbf{X}}_{\sim u}\right)\prod _{k\in u}\frac{F_k\left(X_k^{\prime }\right)-1{\mathrm{I}}_{[X_k^{\prime }\ge X_k]}}{{\rho }_k\left(X_k^{\prime }\right)}\right],}$$

and the total SF of ${\mathbf{X}}_u$ is given as [14]

$${\displaystyle f_u^{tot}\left(\mathbf{X}\right):=\sum _{\begin{array}{c}v\subseteq \{1,\dots ,d\}\\ u\cap v\ne Ø\end{array}}{f}_v\left({\mathbf{X}}_v\right)=\sum _{\begin{array}{c}v,v\subseteq u\\ \left|v\right|>0\end{array}}{\mathbb{E}}_{{\mathbf{X}}^{\prime }}\left[{\mathcal{D}}^{\left|v\right|}f({\mathbf{X}}_u^{\prime },{\mathbf{X}}_{\sim u})\prod _{k\in v}\frac{F_k\left(X_k^{\prime }\right)-1{\mathrm{I}}_{[X_k^{\prime }\ge X_k]}}{{\rho }_k\left(X_k^{\prime }\right)}\right].}$$

For a single input $X_j$, we have $f_j\left(X_j\right)=f_j^{fo}\left(X_j\right)$ and $f_j^{tot}\left(\mathbf{X}\right)=f_j^{sup}\left(\mathbf{X}\right)$. Among similarity measures [28,29], taking the variance–covariances of SFs, that is, ${\Sigma }_u:=\mathbb{V}\left[f_u\left({\mathbf{X}}_u\right)\right]$, ${\Sigma }_u^{sup}:=\mathbb{V}\left[f_u^{sup}\left(\mathbf{X}\right)\right]$ and ${\Sigma }_u^{tot}:=\mathbb{V}\left[f_u^{tot}\left(\mathbf{X}\right)\right]$, leads to [14]

$${\displaystyle {\Sigma }_u=\mathbb{E}\left[{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{X}\right){\mathcal{D}}^{\left|u\right|}f\left({\mathbf{X}}^{\prime }\right)\prod _{k\in u}\frac{F_k\left(\min \left(X_k,X_k^{\prime }\right)\right)-F_k\left(X_k\right)F_k\left(X_k^{\prime }\right)}{{\rho }_k\left(X_k\right){\rho }_k\left(X_k^{\prime }\right)}\right];}$$

(12)

$${\displaystyle {\Sigma }_u\le {\Sigma }_u^{sup}\le {\Sigma }_u^{ub}:=\frac{1}{2^{\left|u\right|}}\mathbb{E}\left[{\left({\mathcal{D}}^{\left|u\right|}f\left(\mathbf{X}\right)\right)}^2\prod _{k\in u}\frac{F_k\left(X_k\right)\left[1-F_k\left(X_k\right)\right]}{{\left({\rho }_k\left(X_k\right)\right)}^2}\right].}$$

(13)

Thus, ${\Sigma }_u^{ub}$ is the upper bound of ${\Sigma }_u^{sup}$. Likewise, ${\Sigma }_j^{ub}$ is the upper bound of ${\Sigma }_j^{tot}$ (i.e., ${\Sigma }_j^{tot}\le {\Sigma }_j^{ub}$), and it can be used for screening the input variables.

To provide new expressions of the screening measures and the main induces in the following proposition, denote with ${\mathbf{V}}^{\prime }$ an i.i.d. copy of $\mathbf{V}$, and assume that

Assumption 3 

(A3).  $f\left(\mathbf{X}\right)$ has finite second-order moments.

Proposition 1. 

Under the assumptions made in Corollary 4, assume that (A1)–(A3) hold. Then,

$$\begin{array}{ccc}\hfill {\Sigma }_u& =& \sum _{\begin{array}{c}{\ell }_1=1\\ {\ell }_2=1\end{array}}^L{C}_{{\ell }_1}^{\left(\right|u\left|\right)}{C}_{{\ell }_2}^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{X}+{\beta }_{{\ell }_1}\mathit{h}\mathbf{V}\right)f\left({\mathbf{X}}^{\prime }+{\beta }_{{\ell }_2}\mathit{h}{\mathbf{V}}^{\prime }\right)\right.\hfill \\ & & \left.\times \prod _{k\in u}\frac{V_k{V}_k^{\prime }}{\left(h_k^2{\sigma }^4\right)}\left(\frac{F_k\left(\min \left(X_k,X_k^{\prime }\right)\right)-F_k\left(X_k\right)F_k\left(X_k^{\prime }\right)}{{\rho }_k\left(X_k\right){\rho }_k\left(X_k^{\prime }\right)}\right)\right]+\mathcal{O}\left({\left|\left|\mathbf{h}\right|\right|}_2^2\right);\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill {\Sigma }_u^{ub}& =& \frac{1}{2^{\left|u\right|}}\sum _{\begin{array}{c}{\ell }_1=1\\ {\ell }_2=1\end{array}}^L{C}_{{\ell }_1}^{\left(\right|u\left|\right)}{C}_{{\ell }_2}^{\left(\right|u\left|\right)}\mathbb{E}\left[f\left(\mathbf{X}+{\beta }_{{\ell }_1}\mathit{h}\mathbf{V}\right)f\left(\mathbf{X}+{\beta }_{{\ell }_2}\mathit{h}{\mathbf{V}}^{\prime }\right)\right.\hfill \\ & & \left.\times \prod _{k\in u}\frac{V_k{V}_k^{\prime }}{h_k^2{\sigma }^4}\left(\frac{F_k\left(X_k\right)\left[1-F_k\left(X_k\right)\right]}{{\left({\rho }_k\left(X_k\right)\right)}^2}\right)\right]+\mathcal{O}\left({\left|\left|\mathbf{h}\right|\right|}_2^2\right).\hfill \end{array}$$

(15)

Proof. 

Bearing in mind Equations (2), (12) and (13), the proof of Proposition 1 is straightforward.    □

The method of moments allows for deriving the estimators of ${\Sigma }_u$ and ${\Sigma }_u^{ub}$ for all $u\subseteq \{1,\dots ,d\}$. For screening inputs of models, we provide the estimators of ${\Sigma }_j$ and ${\Sigma }_j^{ub}$ for any $j\in \{1,\dots ,d\}$. To that end, we are given four independent samples, that is, ${\left\{{\mathbf{X}}_i\right\}}_{i=1}^N$ from $\mathbf{X}$, ${\left\{{\mathbf{X}}_i^{\prime }\right\}}_{i=1}^N$ from ${\mathbf{X}}^{\prime }$, ${\left\{{\mathbf{V}}_i\right\}}_{i=1}^N$ from $\mathbf{V}$, and ${\left\{{\mathbf{V}}_i^{\prime }\right\}}_{i=1}^N$ from ${\mathbf{V}}^{\prime }$. Consistent estimators of ${\Sigma }_j$ and ${\Sigma }_j^{ub}$ are, respectively, given by

$$\begin{array}{ccc}\hfill \widehat{{\Sigma }_j}& :=& \sum _{i=1}^N\sum _{\begin{array}{c}{\ell }_1=1\\ {\ell }_2=1\end{array}}^L\frac{C_{{\ell }_1}^{\left(1\right)}{C}_{{\ell }_2}^{\left(1\right)}}{N}f\left({\mathbf{X}}_i+{\beta }_{{\ell }_1}\mathit{h}{\mathbf{V}}_i\right)f\left({\mathbf{X}}_i^{\prime }+{\beta }_{{\ell }_2}\mathit{h}{\mathbf{V}}_i^{\prime }\right)\frac{V_{i,j}{V}_{i,j}^{\prime }}{h_j^2{\sigma }^4}\hfill \\ & & \left(\frac{F_j\left(\min \left(X_{i,j},X_{i,j}^{\prime }\right)\right)-F_j\left(X_{i,j}\right)F_j\left(X_{i,j}^{\prime }\right)}{{\rho }_j\left(X_{i,j}\right){\rho }_j\left(X_{i,j}^{\prime }\right)}\right);\hfill \end{array}$$

(16)

$${\displaystyle \widehat{{\Sigma }_j^{ub}}:=\sum _{i=1}^N\sum _{\begin{array}{c}{\ell }_1=1\\ {\ell }_2=1\end{array}}^L\frac{C_{{\ell }_1}^{\left(1\right)}{C}_{{\ell }_2}^{\left(1\right)}}{2N}f\left({\mathbf{X}}_i+{\beta }_{{\ell }_1}\mathit{h}{\mathbf{V}}_i\right)f\left({\mathbf{X}}_i+{\beta }_{{\ell }_2}\mathit{h}{\mathbf{V}}_i^{\prime }\right)\frac{V_{i,j}{V}_{i,j}^{\prime }}{h_j^2{\sigma }^4}\left(\frac{F_j\left(X_{i,j}\right)\left[1-F_j\left(X_{i,j}\right)\right]}{{\left({\rho }_j\left(X_{i,j}\right)\right)}^2}\right).}$$

(17)

The above (direct) estimators require $3LN$ model runs for obtaining the estimates for any $j\in \{1,\dots ,d\}$. Additionally to such estimators, we derive the plug-in estimators, which are relevant in the presence of given data about the estimates of the first-order derivatives. To provide such estimators, we denote with ${\left\{\widehat{{\mathcal{D}}^{\left|\right\{j\left\}\right|}f}\left({\mathbf{X}}_i\right)\right\}}_{i=1}^{N_1}$ a sample of $N_1>1$ known or estimates of the first-order derivatives (i.e., $\left|u\right|=1$). Using Equation (2), such estimates are obtained by considering $L=1$ or $L=2$ or $L=3$ and N. Keeping in mind Equations (12) and (13) and the U-statistic theory for one sample, the plug-in estimator of the main index of $X_j$ is given by

$${\displaystyle {\widehat{{\Sigma }_j}}^{\prime }:=\frac{2}{N_1(N_1-1)}\sum _{1\le i_1<i_2\le N_1}\widehat{{\mathcal{D}}^{\left|\right\{j\left\}\right|}f}\left({\mathbf{X}}_{i_1}\right)\widehat{{\mathcal{D}}^{\left|\right\{j\left\}\right|}f}\left({\mathbf{X}}_{i_2}\right)\frac{F_j\left(\min \left(X_{i_1,j},X_{i_2,j}\right)\right)-F_j\left(X_{i_1,j}\right)F_j\left(X_{i_2,j}\right)}{{\rho }_j\left(X_{i_1,j}\right){\rho }_j\left(X_{i_2,j}\right)}.}$$

(18)

Likewise, the plug-in estimator of the upper bound of the total index of $X_j$ is given by

$${\displaystyle {\widehat{{\Sigma }_j^{ub}}}^{\prime }:=\frac{1}{2N_1}\sum _{i=1}^{N_1}{\left[\widehat{{\mathcal{D}}^{\left|\right\{j\left\}\right|}f}\left({\mathbf{X}}_i\right)\right]}^2\frac{F_j\left(X_{i,j}\right)\left[1-F_j\left(X_{i,j}\right)\right]}{{\left[{\rho }_j\left(X_{i,j}\right)\right]}^2}.}$$

(19)

Note that the plug-in estimators are consistent and require a total of $L{N}_1{N}_0$ model runs for computing such indices, where $N_0$ is the number of model runs used for computing the gradient of f at ${\mathbf{X}}_i$.

5. Illustrations: Screening and Emulators of Models

5.1. Test Functions

5.1.1. Ishigami’s Function ($d=3$)

The Ishigami function includes three independent inputs following a uniform distribution on $[-\pi ,\pi ]$, and it is given by

$$f\left(\mathbf{x}\right)=sin\left(x_1\right)+7{sin}^2\left(x_2\right)+0.1x_3^{4}sin\left(x_1\right).$$

The sensitivity indices are $S_1=0.3139$, $S_2=0.4424$, $S_3=0.0$, $S_{T_1}=0.567$, $S_{T_2}=0.442$, and $S_{T_3}=0.243$.

5.1.2. Sobol’s g-Function ($d=10$)

The g-function [63] includes ten independent inputs following a uniform distribution on $[0,1]$, and it is defined as follows:

$$f\left(\mathbf{x}\right)=\prod _{j=1}^{d=10}\frac{|4x_j-2|+a_j}{1+a_j}.$$

Note that such a function is differentiable almost everywhere. According to the values of $\mathbf{a}=(a_j,j=1,2,\dots ,d)$, this function has different properties [23]:

• If $\mathbf{a}={[0,0,6.52,6.52,6.52,6.52,6.52,6.52,6.52,6.52]}^T$, the values of sensitivity indices are $S_1=S_2=0.39$, $S_j=0.0069$, $\forall j>2$, $S_{T_1}=S_{T_2}=0.54$, and $S_{T_j}=0.013$, $\forall j>2$. Thus, this function has a low effective dimension (function of type A), and it belongs to ${\mathcal{A}}_s$ with $s>1$ (see Section 3.4).
• If $\mathbf{a}={[50,50,50,50,50,50,50,50,50,50]}^T$, the first and total indices are given as follows: $S_j=S_{T_j}=0.1$, $\forall j\in \{1,2,\dots ,d\}$. Thus, all inputs are important, but there is no interaction among these inputs. This function has a high effective dimension (function of type B). Note that it belongs to ${\mathcal{A}}_1$.
• If $\mathbf{a}={[0,0,0,0,0,0,0,0,0,0]}^T$, the function belongs to the class of functions with important interactions among inputs. Indeed, we have $S_j=0.02$ and $S_{T_j}=0.27$, $\forall j\in \{1,2,\dots ,d\}$. All the inputs are relevant due to important interactions (function of type C). Then, this function belongs to ${\mathcal{A}}_s$ with $s\ge 2$.

5.2. Numerical Comparisons of Estimators

This section provides a comparison of the direct and plug-in estimators of the main indices and the upper bounds of the total indices using the test functions of Section 5.1. Different total budgets for the model evaluations are considered in this paper, that is, $N=500$, 1000, 1500, 2000, 3000, 5000, 10,000, 15,000, 20,000. In the case of the plug-in estimators, we used $N_0=2d$. For generating different random values, Sobol’s sequence (scrambled = 3) from the R-package randtoolbox [64] is used. We replicated each estimation 30 times by randomly choosing the seed, and the reported results are the average of the 30 estimates.

Figure 1, Figure 2, Figure 3 and Figure 4 show the mean squared errors related to the estimates of the main indices for the Ishigami function, the g-functions of type A, type B, and type C, respectively. Each figure depicts the results for $L=2$ and $L=3$. All the figures show the convergence and accuracy of our estimates using either $L=2$ or $L=3$.

Figure 1. Average of $d=3$ mean squared errors using the Ishigami function (○ the direct estimator (16) and + for the plug-in estimator (18)).

Figure 2. Average of $d=10$ mean squared errors using the g-function of type A (○ the direct estimator (16) and + for the plug-in estimator (18)).

Figure 3. Average of $d=10$ mean squared errors using the g-function of type B (○ the direct estimator (16) and + for the plug-in estimator (18)).

Figure 4. Average of $d=10$ mean squared errors using the g-function of type C (○ the direct estimator (16) and + for the plug-in estimator (18)).

Likewise, Figure 5, Figure 6, Figure 7 and Figure 8 show the mean squared gaps (differences) between the true total index and its estimated upper bound for the Ishigami function, the g-functions of type A, type B, and type C, respectively.

Figure 5. Average of $d=3$ mean squared gaps using the Ishigami function (○ the direct estimator (17) and + for the plug-in estimator (19)).

Figure 6. Average of $d=10$ mean squared gaps using the g-function of type A (○ the direct estimator (17) and + for the plug-in estimator (19)).

Figure 7. Average of $d=10$ mean squared gaps using the g-function of type B (○ the direct estimator (17) and + for the plug-in estimator (19)).

Figure 8. Average of $d=10$ mean squared gaps using the g-function of type C (○ the direct estimator (17) and + for the plug-in estimator (19)).

It turns out that the plug-in estimators outperform the direct ones. Also, increasing the values of L gives the same results. Moreover, the direct estimators associated with $L=1$ fail to provide accuracy estimates (we do not report such results here). On the contrary, the plug-in estimates using $L=1$ are reported in Table 1 for the Ishigami function and in Table 2 for the three types of the g-function. Such results suggest considering $L=1$ or $L=2$ with ${\beta }_1=-1,{\beta }_2=1$ for plug-in estimators when the total budget of model runs is small. For a larger budget of model runs, the direct estimators associated with $L=2$ and ${\beta }_1=-1,{\beta }_2=1$ can be considered as well in practice.

Table 1. Average of 30 estimates of the main indices and upper bounds of total indices for the Ishigami function using the plug-in estimators, $L=1$, and 2000 model runs.

	X1	X2	X3
$S_j$	0.249	0.318	−0.006
$U{B}_j$	1.420	4.872	0.711

Table 2. Average of 30 estimates of the main indices and upper bounds of total indices for the g-functions using the plug-in estimators, $L=1$, and 2000 model runs.

	X1	X2	X3	X4	X5	X6	X7	X8	X9	X10
	Type A
$S_j$	0.330	0.324	0.006	0.005	0.006	0.006	0.005	0.005	0.006	0.005
$U{B}_j$	2.022	2.005	0.045	0.046	0.047	0.047	0.046	0.046	0.046	0.047
	Type B
$S_j$	0.085	0.085	0.085	0.085	0.085	0.085	0.085	0.085	0.085	0.085
$U{B}_j$	0.362	0.363	0.363	0.362	0.363	0.362	0.363	0.362	0.363	0.363
	Type C
$S_j$	0.028	0.028	0.032	0.032	0.035	0.041	0.031	0.030	0.036	0.034
$U{B}_j$	2.033	1.301	1.825	1.605	1.634	1.641	2.216	1.526	1.793	1.503

5.3. Emulations of the g-Function of Type B

Based on the results obtained in Section 5.2 (see Table 2), all the inputs are important in the case of the g-function of type B, meaning that the dimension reduction is not possible. Also, the estimated upper bounds suggest weak interactions among inputs. As expected, our estimated results confirm that the g-function of type B belongs to ${\mathcal{A}}_1$. Thus, an emulator of f based only on the first-order derivatives is sufficient. Using this information, we have derived the emulator of that function (i.e., $\widehat{f_{s=1}}\left(\mathbf{x}\right)$) under the assumptions made in Corollary 5 ($r^{*}=0,L\ge 2$) and using $G_j=F_j$ with $j=1,\dots ,d$. For a given L, we used 300 model runs to build the emulator, and Figure 9 depicts the approximations of that function (called predictions) at the sample points involved in the construction of the emulator. Note that the evaluations of f at such sample points (called observations) are not directly used in the construction of such an emulator. It turns out from Figure 9 that $\widehat{f_{s=1}}$ provides predictions that are in line with the observations, showing the accuracy of our emulator.

Figure 9. Predictions of g-function of type B using the emulator $\widehat{f_{s=1}}$ versus observations using $L=2$ and $L=3$.

6. Conclusions

In this paper, we firstly provided i) stochastic expressions of cross-partial derivatives of any order, followed by their biases, and ii) estimators of such expressions. Our estimators of the $\left|u\right|$-th cross-partial derivatives ($\forall u\subseteq \{1,\dots ,d\}$) reach the parametric rates of convergence (i.e., $\mathsf{O}\left(N^{-1}{d}^{\left|u\right|}\right)$) by means of a set of $L\ge \left|u\right|+1$ constraints for the Hölder space of $\alpha$-smooth functions ${\mathcal{H}}_{\alpha }$ with $\alpha >\left|u\right|$. Moreover, we showed that the upper bounds of the biases of such estimators do not suffer from the curse of dimensionality. Secondly, the proposed surrogates of cross-partial derivatives are used for deriving (i) new derivative-based emulators of simulators or surrogates of models, even when a large number of model inputs contribute to the model outputs, and (ii) new repressions of the main sensitivity indices and the upper bounds of the total sensitivity indices.

Numerical simulations confirmed the accuracy of our approaches for not only screening the input variables, but also for identifying the class of functions that contains our simulator of interest, such as the class of functions with important or no interaction among inputs. This relevant information allows for designing and building the appropriate emulators of functions. In the case of the g-function of type B, our emulator of this function (based only on the first-order derivatives) provided approximations or predictions that are in line with the observations.

For functions with important interactions or, equivalently, for higher-order cross-partial derivatives, further numerical schemes are necessary to increase the numerical accuracy of the computations of such derivatives and predictions. Such perspectives will be investigated in the near future as well as the computations of the total sensitivity indices using the proposed surrogates of derivatives. Moreover, there is a need for a theoretical investigation to expect derivation of the parametric rates of convergence of the above estimators that do not suffer from the course of dimensionality. Working in $\mathbb{C}$ rather than in $\mathbb{R}$ may be helpful.