On Dimension-Free Stochastic Surrogates and Estimators of Cross-Partial Derivatives and the Hessian Matrix

Abstract

This study introduces stochastic surrogates of all the cross-partial derivatives of functions using L evaluations of functions at randomized points. Such randomized points are constructed using the class of ${\mathcal{l}}_p$-spherical distributions or equivalent distributions. For the cross-partial derivatives of a given order $\left|u\right|\in \{2,\dots ,d\}$, the proposed surrogates and the corresponding estimators of cross-partial derivatives enjoy the parametric rate of convergence and dimension-free mean squared errors when $d\ll p$, leading to breaking down the curse of dimensionality. Imposing $p\ll d$ allows to break down the curse of dimensionality for only the cross-partial derivatives of orders given by $\left|u\right|\ll 1+{\displaystyle \frac{d}{2log\left(d\right)}}$. Also, the L-point-based Hessian surrogate and estimator are proposed, including the convergence analysis. A particular choice of p allows to achieve the dimension-free mean squared errors. Analytical examples and simulations have been provided to show the efficiency of such surrogates and estimators.

1. Introduction

First-order, second-order and higher-order derivatives are relevant quantities in functional analysis and modeling. Instances of their applications are listed below: inverse problems, first-order and second-order stochastic optimizations [1,2,3,4], derivative-based sensitivity analysis methods [5,6,7,8,9,10], derivative-based ANOVA (Db-ANOVA) or exact expansions of functions [10,11], and active subspaces [12,13,14,15,16].

Computing the Hessian and higher-order cross-partial derivatives is particularly relevant for (i) Db-ANOVA and for developing efficient emulators of complex models [11], and (ii) the second-order stochastic optimizations. For higher-dimensional models that are time-demanding, it is challenging to compute such derivatives using a few model runs or evaluations. Expecting a number of model runs that is less than the dimensionality d for computing, e.g., all of the first-order, second-order and third-order cross-partial derivatives, requires relying on stochastic perturbations methods or Monte Carlo approaches [1,2,3,4,11,17,18,19,20,21,22,23]. Indeed, dimension-free mean squared errors have been obtained in the work [23] for the estimations of gradients of functions, showing the ability of stochastic approaches to give consistent estimations of gradients, even though the number of model runs is less than the dimensionality.

Concerning related works on stochastic-based estimators of higher-order partial derivatives, the Hessian estimators have been investigated in [2,24,25,26,27], while estimators of cross-partial derivatives have been considered in [11,28]. Some of such approaches rely on the Taylor expansions of functions, randomized kernels and/or random vectors that are uniformly distributed on the unit sphere. Nevertheless, independent random variables are considered in [24,27,28], and approaches developed in the works [27,29] rely directly on the Stein identity [30] rather than the Taylor expansions. Regarding the convergence analysis, it appears in [27] that the root mean squared error of the Hessian estimator is of the order $\mathcal{O}\left(N{d}^{7/2}\right)$ when using three evaluations of functions, and under the assumption of thrice continuous differentiable functions. Under the same conditions, it is shown in [28] that the mean squared errors for the cross-partial derivatives of order $\left|u\right|$ is $\mathcal{O}\left(N{d}^{\left|u\right|}\right)$ with the biases being independent of the dimensionality and $\left|u\right|$ being the cardinality of $u\subseteq \{1,\dots ,d\}$.

So far, dimension-free mean squared errors (MSEs) have not been achieved by the existing estimators of cross-partial derivatives and/or the Hessian, according to our knowledge. The key motivation of this paper consists in addressing the queries about theoretical advantages of randomized schemes over finite difference methods (FDMs) when estimating the Hessian matrix and cross-partial derivatives of any order (i.e., $\left|u\right|\in \{2,\dots ,d\}$). Based on the ${\mathcal{l}}_p$-spherical distributions or equivalent distributions, this study proposes the $L=\left|u\right|+1$-point-based estimators of the cross-partial derivatives of any order $\left|u\right|$ and the Hessian estimator in the framework of the work [28]. For any given order $\left|u\right|$, the proposed estimators, including the Hessian estimator, allow to achieve:

• The dimension-free upper-bounds of the biases;
• The dimension-free MSEs;
• The parametric rate of convergence.

This is acheived by choosing the values of p such that $d\ll p$. Imposing $p\ll d$ also allows to break down the curse of dimensionality, but for only the cross-partial derivatives of orders given by $\left|u\right|\ll 1+{\displaystyle \frac{d}{2log\left(d\right)}}$.

The paper is organized as follows: Section 2 deals with the extension of the L-point-based surrogate of cross-partial derivatives, and the estimators of such surrogates using the ${\mathcal{l}}_p$-spherical distributions or equivalent distribution. A particular set of L constraints is considered to form the L randomized points. Section 3 deals with the convergence analysis of the estimators of the cross-partial derivatives, and the choice of the hyper-parameters (e.g., values of p) that lead to dimension-free MSEs. The particular case of the Hessian matrix is investigated in Section 4. This section provides surrogates of the Hessian matrix and the joint estimators of the Hessian matrix under precise structural assumptions on the deterministic functions. The convergence analysis has been derived, and the values of p and other parameters of the ${\mathcal{l}}_p$-spherical distributions have been provided so as to reach dimension-free MSEs. Section 5 deals with an application of the proposed Hessian estimator, and we conclude this work in Section 6.

General Notation

For an integer $d\in \mathbb{N}\setminus \left\{0\right\}$, let $\mathbf{X}:=(X_1,\dots ,X_d)$ be a random vector of d independent and continuous variables with marginal cumulative distribution functions (CDFs) $F_j$, $j=1,\dots ,d$.

For a non-empty subset $u\subseteq \{1,\dots ,d\}$, denote with $\left|u\right|$ its cardinality and (∼$u):=\{1,\dots ,d\}\setminus u$. Also, ${\mathbf{X}}_u:=(X_j,\forall j\in u)$ stands for a subset of input variables, and the partition $\mathbf{X}=({\mathbf{X}}_u,{\mathbf{X}}_{\sim u})$ holds.

Assumption 1 

(A1). The input variable $\mathbf{X}$ is a random vector of independent variables, supported on an open set $\Omega \subseteq {\mathbb{R}}^d$.

Working with partial derivatives requires using a specific mathematical space. Consider a weak partial differentiable function $f:\Omega \to \mathbb{R}$ [31,32] and a subset $v\subseteq \{1,\dots ,d\}$ with $\left|v\right|>0$. Denote with ${\mathcal{D}}^{\left|v\right|}f:=\left({\prod }_{k\in v}{\displaystyle \frac{\partial }{\partial x_k}}\right)f$ the ${\left|v\right|}^{\mathrm{th}}$ weak cross-partial derivatives of f w.r.t. each $x_k$ for any $k\in v$.

Likewise, given $\overrightarrow{\mathcal{ı}}:=(i_1,\dots ,i_d)\in {\mathbb{N}}^d$, denote ${\mathcal{D}}^{\left(\overrightarrow{\mathcal{ı}}\right)}f:=\left({\prod }_{k=1}^d{\displaystyle \frac{{\partial }^{i_k}}{\partial x_k}}\right)f$ and ${\mathbb{1}}_v\left(j\right)=1$ if $j\in v$ and zero otherwise. Thus, taking $\overrightarrow{\mathit{v}}:=\left({\mathbb{1}}_v\left(1\right),\dots ,{\mathbb{1}}_v\left(d\right)\right)$ yields ${\mathcal{D}}^{\left|v\right|}f={\mathcal{D}}^{\left(\overrightarrow{\mathit{v}}\right)}f$. Moreover, denote ${\left(\mathbf{x}\right)}^{\overrightarrow{\mathcal{ı}}}={\mathbf{x}}^{\overrightarrow{\mathcal{ı}}}:={\prod }_{k=1}^d{x}_k^{i_k}$, $\overrightarrow{\mathcal{ı}}!:=i_1!\dots i_d!$, and consider the Hölder space of $\alpha$-smooth functions given as follows: $\forall \mathbf{x},\mathbf{y}\in {\mathbb{R}}^d$

$${\displaystyle {\mathcal{H}}_{\alpha }:=\left\{f:{\mathbb{R}}^d\to \mathbb{R}:\left|f\left(\mathbf{x}\right)-\sum _{0\le i_1+\dots +i_d\le \alpha -1}\frac{{\mathcal{D}}^{\left(\overrightarrow{\mathcal{ı}}\right)}f\left(\mathbf{y}\right)}{\overrightarrow{\mathcal{ı}}!}{\left(\mathbf{x}-\mathbf{y}\right)}^{\overrightarrow{\mathcal{ı}}}\right|\le M_{\alpha }{\left|\left|\mathbf{x}-\mathbf{y}\right|\right|}_2^{\alpha }\right\},}$$

with $\alpha \ge 1$; $M_{\alpha }>0$ being the Hölder constant and ${\left|\left|·\right|\right|}_2$ being the Euclidean norm.

Denote with $\left|\right|·{\left|\right|}_p$ the ${\mathcal{l}}_p$-norm for any $p\ge 1$; $\left|\right|·{\left|\right|}_s$ the spectral norm and $\left|\right|·{\left|\right|}_F$ the Frobenius norm. The unit ${\mathcal{l}}_p$-ball and ${\mathcal{l}}_p$-sphere are respectively defined by

$${\mathcal{B}}_p:=\left\{\mathbf{x}\in {\mathbb{R}}^d{:\left|\right|\mathbf{x}\left|\right|}_p<1\right\};\partial {\mathcal{B}}_p:=\left\{\mathbf{x}\in {\mathbb{R}}^d{:\left|\right|\mathbf{x}\left|\right|}_p=1\right\}.$$

In what follows, $\mathbb{E}[·]$ and $\mathbb{V}[·]$ stand for the expectation operator and the variance operator, respectively.

2. Generalized Stochastic Surrogates of Cross-Partial Derivatives and Estimators

This section aims at extending the L-point-based surrogates of cross-partial derivatives of functions provided in [11,28] thanks to the wide class of distribution functions, such as the ${\mathcal{l}}_p$-spherical distributions. Precisely, generic, stochastic surrogates and estimators of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ for all $u\subseteq \{1,\dots ,d\}$ based on $L\ge 1$ evaluations of functions are presented.

2.1. Sets of Constraints

Given $L,r^{\ast }\in \mathbb{N}\setminus \left\{0\right\}$, ${\beta }_{\mathcal{l}}\in \mathbb{R}$ with $\mathcal{l}=1,\dots ,L$, consider the set of L constraints given by ${\sum }_{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}{\beta }_{\mathcal{l}}^r={\delta }_{\left|u\right|,r}$ with $r=0,1,\dots ,L-1$ or

$${\displaystyle \left\{\begin{array}{cc}\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}{\beta }_{\mathcal{l}}^r={\delta }_{\left|u\right|,r};r=0,1,\dots ,L-1\hfill & \mathrm{if}\left|u\right|=1,\dots ,r^{\ast }\hfill \\ \sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}{\beta }_{\mathcal{l}}^r={\delta }_{\left|u\right|,r};r=0,\dots ,r^{\ast },\left|u\right|,\dots ,\left|u\right|+L-r^{\ast }-2\hfill & \mathrm{otherwise}\hfill \end{array}\right.,}$$

where $r^{\ast }\le L-2$ and $C_{\mathcal{l}}^{\left(\right|u\left|\right)}$s are unknown constants. For instance, the latter constraints can be used for killing off the derivative terms of order up to $r^{\ast }\le L-2$ and others in the Taylor expansion of $f(\mathbf{x}+h{\beta }_{\mathcal{l}}\mathbf{v})$ with $\mathcal{l}=1,\dots ,L$, $h\in \mathbb{R}$ and $\mathbf{v}\in {\mathbb{R}}^d$ (see [11,23,28]). When $L=1$, ${\beta }_1=C_1^{\left(\right|u\left|\right)}=1$, and it is obvious that none of derivative terms are going to be killed off. Moreover, taking $L=\left|u\right|+1$ and the set of constraints

$${\displaystyle \sum _{\mathcal{l}=1}^{L=\left|u\right|+1}{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}{\beta }_{\mathcal{l}}^r={\delta }_{\left|u\right|,r};r=0,1,\dots ,\left|u\right|,}$$

(1)

are of particular interest for deriving the dimension-free mean squared errors (MSEs) (see Section 3). Also, Equation (1) leads to the existence and uniqueness of the coefficients $C_{\mathcal{l}}^{\left(\right|u\left|\right)}$s for distinct values of ${\beta }_{\mathcal{l}}$s (i.e., ${\beta }_{{\mathcal{l}}_1}\ne {\beta }_{{\mathcal{l}}_2}$). Indeed, Equation (1) is identical to the following system of equations:

$${\displaystyle \left[\begin{array}{cccc}1& 1& \dots & 1\\ {\beta }_1& {\beta }_2& \dots & {\beta }_L\\ ⋮& ⋮& ⋮& ⋮\\ {\beta }_1^{\left|u\right|}& {\beta }_2^{\left|u\right|}& \dots & {\beta }_L^{\left|u\right|}\end{array}\right]\left[\begin{array}{c}{C}_1^{\left(\right|u\left|\right)}\\ ⋮\\ C_L^{\left(\right|u\left|\right)}\end{array}\right]=\left[\begin{array}{c}0\\ ⋮\\ 0\\ 1\end{array}\right].}$$

Such equations involve the Vandermonde matrix, having the determinant of the form ${\prod }_{1\le {\mathcal{l}}_1<{\mathcal{l}}_2\le L}\left({\beta }_{{\mathcal{l}}_1}-{\beta }_{{\mathcal{l}}_2}\right)$. Consequently, $C_{\mathcal{l}}^{\left(\right|u\left|\right)}$s can be easily computed using any software, and the analytical expression of the Vandermonde matrix’s inverse is available in [33,34].

The set of L constraints will lead to the L-point-based surrogates and estimators of the $\left|u\right|$-th order derivatives. It is worth noting that the aforementioned sets of constraints and other constraints considered in [11,28] can contribute to (i) isolate the $\left|u\right|$-th order derivatives, and/or (ii) to improve the order of approximations of derivatives. But, such constraints are not necessary for accomplishing (i), which is illustrated by the case where $L=1$. To isolate the target $\left|u\right|$-th derivative and to obtain its consistent surrogates and estimators, random vectors listed in Section 2.2 are sufficient. However, Equation (1) is technically necessary for deriving the dimension-free mean squared errors (MSEs), and it must be combined with a particular random vector so as to obtain such a result.

2.2. New Insight into L-Point-Based Surrogates of Cross-Partial Derivatives

While Equation (1) will leads to dimension-free MSEs, generic surrogates of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ are considered in this section for the sake of generality. To that end, denote with $\mathbf{W}:=(W_1,\dots ,W_d)$ a d-dimensional random vectors of independent variables satisfying: $\forall j\in \{1,\dots ,d\}$ and $q\in \mathbb{N}$,

$$\mathbb{E}\left[{\left(W_j\right)}^2\right]={\sigma }^2;\mathbb{E}\left[{\left(W_j\right)}^{2q+1}\right]=0;\mathbb{E}\left[{\left(W_j\right)}^{2q}\right]<+\infty .$$

It is shown in [28] (Theorem 1) that for any $u\subseteq \{1,\dots ,d\}$ with $\left|u\right|>0$, there exists ${\alpha }_{\left|u\right|}\in \{1,\dots ,L\}$ and coefficients $C_1^{\left(\right|u\left|\right)},\dots ,C_L^{\left(\right|u\left|\right)}$ such that

$${\displaystyle {\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)=\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}\mathbb{E}\left[\frac{f\left(\mathbf{x}+{\beta }_{\mathcal{l}}\mathit{h}\mathbf{W}\right)}{{\sigma }^{2\left|u\right|}}\prod _{k\in u}\frac{W_k}{h_k}\right]+\mathcal{O}\left({\left|\left|\mathit{h}\right|\right|}_2^{2{\alpha }_{\left|u\right|}}\right),}$$

(2)

with $\mathit{h}:=(h_1,\dots ,h_d)\in {\mathbb{R}}_{+}^d$ being bandwidths and $\mathit{h}\mathbf{W}:=(h_1{W}_1,\dots ,h_d{W}_d)$. To extend Equation (2) (see Corollary 1), consider the wide class of random vectors $\mathbf{V}:=(V_1,\dots ,V_d)$ verifying:

$$\mathbb{E}\left[\prod _{k\in u}{V}_k^2\right]={\sigma }_{\left|u\right|}^2;\mathbb{E}\left[\prod _{k=1}^d{V}_k^{q_k}\right]=0,\mathrm{if},\mathrm{at}\mathrm{least},\mathrm{one}{q}_k\in \mathbb{N}\mathrm{is} \mathrm{odd}.$$

(3)

Examples of random vectors that satisfy (3) are given as follows:

• Independent random variables sharing the properties of $\mathbf{W}$;
• Random vectors that are uniformly distributed on the unit p-sphere [35,36];
• Random vectors that are uniformly distributed over the unit p-ball [37,38,39];
• Random vectors that are p-spherically distributed on ${\mathbb{R}}^d$ [36,40];
• Other new random vectors proposed in [39,41].

Based on Equation (3), Corollary 1 provides an extension of surrogates of ${\mathcal{D}}^{\left(\overrightarrow{\mathit{u}}\right)}f$ using $L\ge 1$ evaluations of functions.

Corollary 1. 

Consider distinct ${\beta }_{\mathcal{l}}$’s, and assume $f\in {\mathcal{H}}_{\alpha }$ with $\alpha \ge \left|u\right|+2L$ and (A1) hold. Then, for any $u\subseteq \{1,\dots ,d\}$ with $\left|u\right|>0$, there exists ${\alpha }_{\left|u\right|}\in \{1,\dots ,L\}$ and coefficients $C_1^{\left(\right|u\left|\right)},\dots ,C_L^{\left(\right|u\left|\right)}$ such that

$${\displaystyle {\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)=\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}\mathbb{E}\left[\frac{f\left(\mathbf{x}+{\beta }_{\mathcal{l}}\mathit{h}\mathbf{V}\right)}{{\sigma }_{\left|u\right|}^2}\prod _{k\in u}\frac{V_k}{h_k}\right]+\mathcal{O}\left({\left|\left|\mathit{h}\right|\right|}_2^{2{\alpha }_{\left|u\right|}}\right).}$$

(4)

Proof. 

It is just an application of Theorem 1 provided in [28], as $\mathbf{W}$ and $\mathbf{V}$ share the same properties needed (i.e., (3)), and here $\mathbb{E}\left[{\prod }_{k\in u}{V}_k^2\right]={\sigma }_{\left|u\right|}^2$. □

From Corollary 1, it becomes possible to compute all the cross-partial derivatives of a given order using the same evaluations of functions with the same or different order of approximations, depending on the choice of constraints or equivalently the coefficients $C_1^{\left(\right|u\left|\right)},\dots ,C_L^{\left(\right|u\left|\right)}$. Moreover, for a given order $\left|u\right|$, one is able to compute all of the $\left|v\right|$-th cross-partial derivatives provided that $\left|v\right|\le \left|u\right|$.

2.3. Estimators of Cross-Partial Derivatives

In view of Corollary 1, the surrogate of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ is given by

$${\displaystyle {\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\approx \widetilde{{\mathcal{D}}^{\left|u\right|}f}\left(\mathbf{x}\right):=\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}\mathbb{E}\left[\frac{f\left(\mathbf{x}+{\beta }_{\mathcal{l}}\mathit{h}\mathbf{V}\right)}{{\sigma }_{\left|u\right|}^2}\prod _{k\in u}\frac{V_k}{h_k}\right].}$$

To give an estimator of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ using the method of moments, consider a sample of $\mathbf{V}$ given by ${\left\{{\mathbf{V}}_i:=\left(V_{i,1},\dots ,V_{i,d}\right)\right\}}_{i=1}^N$ with $N\ge 1$. The estimator of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$ is given by

$${\displaystyle \widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right):=\frac{1}{N{\sigma }_{\left|u\right|}^2}\sum _{i=1}^N\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}\mathit{h}{\mathbf{V}}_i\right)\prod _{k\in u}\frac{V_{i,k}}{h_k}.}$$

(5)

Remark 1. 

Despite $\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)$ being valid for all $L\ge 1$, it is recommend to use the following estimator when $L=1$:

$${\displaystyle \frac{1}{N{\sigma }_{\left|u\right|}^2}}\sum _{i=1}^N\left\{f\left(\mathbf{x}+\mathit{h}{\mathbf{V}}_i\right)-\overline{f\left(\mathbf{x}+\mathit{h}\mathbf{V}\right)}\right\}\prod _{k\in u}{\displaystyle \frac{V_{i,k}}{h_k}},$$

with $\overline{f\left(\mathbf{x}+\mathit{h}\mathbf{V}\right)}:={\displaystyle \frac{1}{N}}{\sum }_{i=1}^{N}f\left(\mathbf{x}+\mathit{h}{\mathbf{V}}_i\right)$.

While the random vectors satisfying Equation (3) allow to derive valid surrogates of ${\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)$, the ${\mathcal{l}}_p$-spherical distributions on ${\mathbb{R}}^d$ is going to be considered in what follows. By denoting with ${\mathcal{S}}_{d,p}$ a class of ${\mathcal{l}}_p$-spherical distributions on ${\mathbb{R}}^d$ [36,40], note that the random vector $\mathbf{V}\sim {\mathcal{S}}_{d,p}$ is symmetrically distributed about zero. Additional properties of $\mathbf{V}$ are available in [36,40], such as $\mathbf{V}$ admits a stochastic representation of the form $\mathbf{V}=R\mathbf{U}$, where $R>0$ is a non-negative random variable, and it is independent of $\mathbf{U}$, which is uniformly distributed on the unit ${\mathcal{l}}_p$-sphere. Remark that $\tilde{\mathbf{V}}:=R\tilde{\mathbf{U}}$ also satisfies (3) when $\tilde{\mathbf{U}}$ is uniformly distributed over the unit ${\mathcal{l}}_p$-ball, and there is a bijection between $\mathbf{V}$ and $\tilde{\mathbf{V}}$. Thus, only $\mathbf{V}$ and $\tilde{\mathbf{V}}$ are going to be considered for the convergence analysis of the proposed estimator $\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)$.

Remark 2. 

For the second-order derivatives (i.e., $\left|u\right|=2$), taking $L=3$ and ${\beta }_1=-1,{\beta }_2=0,{\beta }_3=1$ leads to $C_1^{\left(\right|u\left|\right)}=C_3^{\left(\right|u\left|\right)}=0.5$ and $C_2^{\left(\right|u\left|\right)}=-1$. In addition, using independent variables $\mathbf{V}\sim \mathcal{U}{(-\eta ,\eta )}^d$ with $\eta >0$ allows to retrieve the estimators of the Hessian-off-diagonal terms proposed in [24]. In the same sense, using $\mathbf{V}\sim \mathcal{N}(\mathbf{0},\mathcal{I})$ with $\mathcal{I}\in {\mathbb{R}}^{d\times d}$ being the identity matrix allows to obtain the estimators of the Hessian-off-diagonal terms proposed in [27]. Therefore, such estimators are particular cases of the proposed estimators, and they cannot achieve dimension-free MSEs (see Section 3). Moreover, we will see in Section 4 that the estimators of the Hessian-diagonal terms proposed in this paper differ from those considered in [24,27].

3. Convergence Analysis of Estimators of Cross-Partial Derivatives

The convergence analysis requires precise structural assumptions on the deterministic function f. To focus on the fundamental convergence properties and isolate the dimensional dependence arising solely from the estimation scheme, we now consider the minimal smoothness assumption sufficient for the derivative’s existence and the simplified setting of equal bandwidths.

Assumption 2 

(A2). The function $f\in {\mathcal{H}}_{\alpha }$ with $\alpha =\left|u\right|+1$.

Note that assumption (A2) does not depend on the dimensionality d in general, except when $\left|u\right|$ is close to d. For highly smooth functions, the dimensional dependence arising from (A2) vanishes. Using equal bandwidths ($h_1=\dots =h_d=h$) leads to the following estimators:

$${\displaystyle \widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)=\left\{\begin{array}{cc}\frac{1}{N{\sigma }_{\left|u\right|}^2{h}^{\left|u\right|}}\sum _{i=1}^N\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(\right|u\left|\right)}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h{\mathbf{V}}_i\right)\prod _{k\in u}{V}_{i,k}& \mathrm{if}L>1\\ \frac{1}{N{\sigma }_{\left|u\right|}^2{h}^{\left|u\right|}}\sum _{i=1}^N\left\{f\left(\mathbf{x}+h{\mathbf{V}}_i\right)-\overline{f\left(\mathbf{x}+h\mathbf{V}\right)}\right\}\prod _{k\in u}{V}_{i,k}& \mathrm{if}L=1\end{array}\right.,}$$

(6)

with $\mathbf{V}\sim {\mathcal{S}}_{d,p}$ or $\mathbf{V}=R\tilde{\mathbf{U}}$. When $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, recall that $V_k=R{U}_k$ and

$${\displaystyle {\sigma }_{\left|u\right|}^2=\mathbb{E}\left[\prod _{k\in u}{V}_k^2\right]=\mathbb{E}\left[R^{2\left|u\right|}\right]\mathbb{E}\left[\prod _{k\in u}{U}_k^2\right]=\frac{\mathbb{E}\left[R^{2\left|u\right|}\right]\Gamma \left(d/p\right)}{\Gamma \left((d+2|u\left|\right)/p\right)}{\left[\frac{\Gamma \left(3/p\right)}{\Gamma \left(1/p\right)}\right]}^{\left|u\right|},}$$

(7)

because R and $\mathbf{U}$ are independent, and $\mathbb{E}\left[{\prod }_{k\in u}{U}_k^2\right]={\displaystyle \frac{\Gamma \left(d/p\right)}{\Gamma \left((d+2|u\left|\right)/p\right)}}{\left[{\displaystyle \frac{\Gamma \left(3/p\right)}{\Gamma \left(1/p\right)}}\right]}^{\left|u\right|}$ (see Corollary 2.7 from [36]). For the particular case given by $\left|u\right|=1$ and $\mathbb{E}\left[V_k^2\right]=\mathbb{E}\left[V_1^2\right]=\mathbb{E}\left[U_1^2\right]\mathbb{E}\left[R^2\right]={\sigma }_1^2$, one can deduce $\mathbb{E}\left[R^2\right]={\sigma }_1^2{\displaystyle \frac{\Gamma (1/p)\Gamma (d/p+2/p)}{\Gamma (3/p)\Gamma (d/p)}}$ using Equation (7). Also, the identity (7) implies

$${\displaystyle \mathbb{E}\left[\frac{R^{2\left|u\right|+1}}{{\sigma }_{\left|u\right|}^2}\right]=\frac{\Gamma \left((d+2|u\left|\right)/p\right)}{\Gamma \left(d/p\right)}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^{\left|u\right|}\frac{\mathbb{E}\left[R^{2\left|u\right|+1}\right]}{\mathbb{E}\left[R^{2\left|u\right|}\right]}.}$$

(8)

3.1. Bias Analysis

Under the above framework, Theorem 1 provides the bias of the proposed estimator for a wide class of functions (i.e., $f\in {\mathcal{H}}_{\left|u\right|+1}$). To that purpose, define

$${\displaystyle {\mathsf{F}}_r:=\sum _{\mathcal{l}=1}^{L=\left|u\right|+1}\left|C_{\mathcal{l}}^{\left(\right|u\left|\right)}{\beta }_{\mathcal{l}}^r\right|;\forall r=0,1,\dots ,d+1;}$$

$${\displaystyle K_{1,p,d,\left|u\right|}:={\left\{\frac{\Gamma \left(\right(4\left|u\right|+1)/p)\Gamma (d/p)}{\Gamma (1/p)\Gamma (d/p+4|u|/p)}\right\}}^{1/2}\frac{\Gamma \left((d+2|u\left|\right)/p\right)}{\Gamma \left(d/p\right)}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^{\left|u\right|}.}$$

To explicit the dimensionality involved in $K_{1,p,d,\left|u\right|}$, the following approximations are derived for higher values of either d or p (see Appendix A):

$${\displaystyle K_{1,p,d,\left|u\right|}\approx \left\{\begin{array}{cc}\sqrt{\frac{\Gamma \left(\right(4\left|u\right|+1)/p)}{\Gamma (1/p)}}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^{\left|u\right|}& \mathrm{if}p\ll d\\ \sqrt{\frac{d+4\left|u\right|}{\left(4\right|u|+1)d}}\frac{3^{\left|u\right|}d}{d+2\left|u\right|}\le 3^{\left|u\right|}\sqrt{\frac{d+4d}{(4+1)d}}=3^{\left|u\right|}& \mathrm{if}d\ll p\end{array}\right..}$$

Theorem 1. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, $h_j=h$, $L=\left|u\right|+1$ and Equation (1). Assume that (A1) and (A2) hold. Then, there are $M_{\left|u\right|+1}>0$ and $C_p>0$ such that

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\mathbb{E}\left[\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)\right]\right|\le h\sqrt{d}{M}_{\left|u\right|+1}{\mathsf{F}}_{\left|u\right|+1}{C}_p{K}_{1,p,d,\left|u\right|}\frac{\mathbb{E}\left[R^{2\left|u\right|+1}\right]}{\mathbb{E}\left[R^{2\left|u\right|}\right]}.}$$

(9)

Proof. 

See Appendix A. □

Note that the upper-bound derived in Theorem 1 holds for any random variable R such that $\mathbf{V}=R\mathbf{U}$. Therefore, obtaining the dimension-free upper-bound of the bias requires choosing R such that the ratio ${\displaystyle \frac{\mathbb{E}\left[R^{2\left|u\right|+1}\right]}{\mathbb{E}\left[R^{2\left|u\right|}\right]}}$ is linear w.r.t. ${\sigma }_1$. The trivial choice of R is the Dirac probability measure given by $R\sim {\delta }_{\eta }$, which requires taking

$${\displaystyle {\eta }^2=\mathbb{E}\left[R^2\right]={\sigma }_1^2\frac{\Gamma (1/p)\Gamma (d/p+2/p)}{\Gamma (3/p)\Gamma (d/p)};\mathrm{and}\frac{\mathbb{E}\left[R^{2\left|u\right|+1}\right]}{\mathbb{E}\left[R^{2\left|u\right|}\right]}=\eta .}$$

Also, the uniform distribution of the form $R_0\sim \mathcal{U}(0,\xi )$ will help for obtaining the dimension-free upper-bound of the bias for any $p>0$. Such a choice requires taking

$${\displaystyle \xi :=\sqrt{3\mathbb{E}\left[R^2\right]}=\sqrt{3{\sigma }_1^2\frac{\Gamma (1/p)\Gamma (d/p+2/p)}{\Gamma (3/p)\Gamma (d/p)}};\mathrm{and}\frac{\mathbb{E}\left[R^{2\left|u\right|+1}\right]}{\mathbb{E}\left[R^{2\left|u\right|}\right]}=\frac{2\left|u\right|+1}{2\left|u\right|+2}\xi \le \xi .}$$

(10)

Obviously, other choices of the distributions of R are possible, and the optimal choice is going to be discussed latter. To provide the dimension-free upper-bound in Corollary 2, define

$${\displaystyle K_{2,p,d,\left|u\right|}:=\sqrt{3}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^{\left|u\right|+1/2}\frac{{\Gamma }^{1/2}\left(\right(4\left|u\right|+1)/p)\Gamma \left((d+2|u\left|\right)/p\right){\Gamma }^{1/2}\left((d+2)/p\right)}{\Gamma \left(d/p\right){\Gamma }^{1/2}(1/p){\Gamma }^{1/2}(d/p+4|u|/p)},}$$

when $R=R_0$ and $K_{2,p,d,\left|u\right|}^{\prime }={\displaystyle \frac{\sqrt{3}{K}_{2,p,d,\left|u\right|}}{3}}$ when $R\sim {\delta }_{\eta }$. Being able to derive dimension-free and small values of biases requires exhibiting the dimensionality in the expression of $K_{2,p,d,\left|u\right|}$. For higher values of d with $p\ll d$ or higher values of p with $d\ll p$, $K_{2,p,d,\left|u\right|}$ is approximated as follows:

$${\displaystyle K_{2,p,d,\left|u\right|}⪅\left\{\begin{array}{cc}\sqrt{3}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^{\left|u\right|+1/2}\frac{{\Gamma }^{1/2}\left(\right(4\left|u\right|+1)/p)}{{\Gamma }^{1/2}(1/p)}{\left(\frac{d}{p}\right)}^{1/p}& \mathrm{if}p\ll d\\ 3^{\left|u\right|+1}{\left(\frac{d}{d+2}\right)}^{3/2}& \mathrm{if}d\ll p\end{array}\right..}$$

Corollary 2. 

Under the condition of Theorem 1, assume that $R=R_0$. Then,

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\mathbb{E}\left[\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)\right]\right|\le h\sqrt{d}{M}_{\left|u\right|+1}{\mathsf{F}}_{\left|u\right|+1}{C}_p{K}_{2,p,d,\left|u\right|}{\sigma }_1.}$$

Moreover, if ${\sigma }_1={\left(\sqrt{d}{\mathsf{F}}_{\left|u\right|+1}{C}_p{K}_{2,p,d,\left|u\right|}\right)}^{-1}$, then

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\mathbb{E}\left[\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)\right]\right|\le h{M}_{\left|u\right|+1}.}$$

(11)

Proof. 

See Appendix B. □

When $R=R_0$, it follows directly from Corollary 2 that the practical values of ${\sigma }_1$ given by

$${\displaystyle {\sigma }_1={\left(\sqrt{d}{\mathsf{F}}_{\left|u\right|+1}{K}_{2,p,d,\left|u\right|}\right)}^{-1},}$$

allow to obtain small bias and to reach dimension-free upper-bound of the bias, as the Hölder constant $M_{\left|u\right|+1}$ and $C_p$ from the concentration inequality are not, in general, known. Consequently, approximated values of ${\sigma }_1$ associated with higher values of d or p may be used as well.

Remark 3. 

Taking $R\sim {\delta }_{\eta }$ and ${\sigma }_1={\left(\sqrt{d}{\mathsf{F}}_{\left|u\right|+1}{K}_{2,p,d,\left|u\right|}^{\prime }\right)}^{-1}$ gives the same upper-bound of the bias, that is,

$${\displaystyle \left|{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)-\mathbb{E}\left[\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)\right]\right|\le h{M}_{\left|u\right|+1}{C}_p.}$$

3.2. Mean Squared Errors

Statistically, it is common to measure the quality of an estimator using MSEs and the rates of convergence. The MSEs are also used for determining the optimal value of the bandwidth h. This section aims at deriving such properties in the case of the proposed estimator of the cross-partial derivatives. Theorem 2 provides the MSE of the proposed estimator under the minimal requirement on f. To that end, define

$${\displaystyle K_{3,p,\left|u\right|}:=\sqrt{\frac{4C\Gamma (4/p)}{p{c}_0^{4/p}}}{\left\{\frac{\Gamma \left(\right(4\left|u\right|+1)/p)}{\Gamma (1/p)}\right\}}^{1/2}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^{2\left|u\right|},}$$

with $C,c_0$ being constants from the concentration inequality that depend only on p.

Theorem 2. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, $h_j=h$, $L=\left|u\right|+1$ and Equation (1). Assume that (A1) and (A2) hold. Then,

$$\begin{array}{ccc}\hfill {\displaystyle \mathbb{E}\left[{\left(\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]}& \le & d{h}^2{M}_{\left|u\right|+1}^2{\mathsf{F}}_{\left|u\right|+1}^2{C}_p^2{K}_{1,p,d,\left|u\right|}^2{\displaystyle \frac{{\mathbb{E}}^2\left[R^{2\left|u\right|+1}\right]}{{\mathbb{E}}^2\left[R^{2\left|u\right|}\right]}}\hfill \\ & & +{\displaystyle \frac{M_{\left|u\right|}^2{\mathsf{F}}_{\left|u\right|}^2}{N{d}^{2/p}}}{\displaystyle \frac{K_{3,p,\left|u\right|}{\Gamma }^2\left((d+2|u\left|\right)/p\right)}{{\Gamma }^{1/2}(d/p+4|u|/p){\Gamma }^{3/2}\left(d/p\right)}}{\displaystyle \frac{\mathbb{E}\left[R^{4\left|u\right|}\right]}{{\mathbb{E}}^2\left[R^{2\left|u\right|}\right]}}.\hfill \end{array}$$

(12)

Proof. 

See Appendix C. □

The first right-term of Equation (12) is the squared upper-bound of the bias derived in Theorem 1, while the last term represents the upper-bound of the variance of $\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)$. The good new is that this bound of the variance does not depend on the bandwidth h. Also, it does not depend neither on ${\sigma }_1$ nor ${\sigma }_u$ when $R\sim {\delta }_{\eta }$ because ${\displaystyle \frac{\mathbb{E}\left[R^{4\left|u\right|}\right]}{{\mathbb{E}}^2\left[R^{2\left|u\right|}\right]}}=1$. The same result holds when $R=R_0$, as ${\displaystyle \frac{\mathbb{E}\left[R_0^{4\left|u\right|}\right]}{{\mathbb{E}}^2\left[R_0^{2\left|u\right|}\right]}}={\displaystyle \frac{{\left(2\right|u|+1)}^2}{4\left|u\right|+1}}>1$. Since the upper-bound of the bias can be controlled through ${\sigma }_1$, it is clear that $R\sim {\delta }_{\eta }$ should outperform $R_0$. Concerning the optimal choice of R, denotes with

$${\displaystyle {\mathcal{B}}_r:=\left\{R>0:\mathbb{E}\left[R^2\right]={\sigma }_1^2\frac{\Gamma (1/p)\Gamma (d/p+2/p)}{\Gamma (3/p)\Gamma (d/p)};\mathbb{E}\left[R^{4d}\right]<+\infty \right\};}$$

the set of strictly positive random variables having finite $4d$th-order moment, and define

$${\displaystyle R_{\ast }:=arg\underset{R\in {\mathcal{B}}_r}{min}\frac{\mathbb{E}\left[R^{4\left|u\right|}\right]}{{\mathbb{E}}^2\left[R^{2\left|u\right|}\right]}.}$$

By minimizing the second term of the upper-bound of the MSE with respect to R, the optimal MSE and parametric rate of convergence of the proposed estimator are derived in Corollary 3. For that purpose, consider

$${\displaystyle K_{4,p,d,\left|u\right|}:=\frac{1}{d^{2/p}}\frac{{\Gamma }^2\left((d+2|u\left|\right)/p\right)}{{\Gamma }^{1/2}(d/p+4|u|/p){\Gamma }^{3/2}\left(d/p\right)}⪅\left\{\begin{array}{cc}\frac{d^{2\left(\right|u|-1)/p}}{p^{2\left|u\right|/p}}& \mathrm{if}p\ll d\\ \frac{\sqrt{5}{d}^{-2/p}{d}^2}{{(d+2)}^2}& d\ll p\end{array}\right.,}$$

where the less approximations hold only for higher-values of d or p.

Corollary 3. 

Under the conditions of Theorem 2, let $R=R_{\ast }$. Then, there exists ${\sigma }_1$ such that

$${\displaystyle \mathbb{E}\left[{\left(\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le h^2+\frac{M_{\left|u\right|}^2{\mathsf{F}}_{\left|u\right|}^2}{N}{K}_{3,p,\left|u\right|}{K}_{4,p,d,\left|u\right|}\frac{\mathbb{E}\left[R_{\ast }^{4\left|u\right|}\right]}{{\mathbb{E}}^2\left[R_{\ast }^{2\left|u\right|}\right]};}$$

Moreover, if $h=N^{-\gamma }$ with $\gamma \in [{\displaystyle \frac{1}{2}},1]$, then

$${\displaystyle \mathbb{E}\left[{\left(\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le \mathcal{O}\left(N^{-1}{K}_{4,p,d,\left|u\right|}\right).}$$

Proof. 

See Appendix D. □

For practical issues, Corollary 4 provides precise results by exhibiting the dimensionality in the upper-bound of the MSE.

Corollary 4. 

Under the conditions of Theorem 2, let $h=N^{-\gamma }$ with $\gamma \in [{\displaystyle \frac{1}{2}},1]$; $R\sim {\delta }_{\eta }$ and ${\sigma }_1={\left(\sqrt{d}{\mathsf{F}}_{\left|u\right|+1}{K}_{2,p,d,\left|u\right|}^{\prime }\right)}^{-1}$

$${\displaystyle \mathbb{E}\left[{\left(\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le h^2{M}_{\left|u\right|+1}^2{C}_p^2+\frac{M_{\left|u\right|}^2{\mathsf{F}}_{\left|u\right|}^2}{N}{K}_{3,p,\left|u\right|}{K}_{4,p,d,\left|u\right|}.}$$

Moreover, if $p\ll d$, then

$${\displaystyle \mathbb{E}\left[{\left(\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le \mathcal{O}\left(N^{-1}{d}^{2\left(\right|u|-1)/p}\right);}$$

and if $d\ll p<+\infty$, then

$${\displaystyle \mathbb{E}\left[{\left(\widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)-{\mathcal{D}}^{\left|u\right|}f\left(\mathbf{x}\right)\right)}^2\right]\le \mathcal{O}\left(N^{-1}{d}^{-2/p}\right).}$$

Such results are straightforward. It turns out that the dimension-free MSE and the parametric rate of convergence are reached when taking higher values of p with $d\ll p$. In the case of $p\ll d$, similar results hold by taking the values of p such that $2\left(\right|u|-1)log\left(d\right)\le p\ll d$, when possible. It is clear that it is only possible when $\left|u\right|\ll 1+{\displaystyle \frac{d}{2log\left(d\right)}}$ by imposing $p\ll d$. When such a condition fails, the first option given by $d\ll p$ is recommended, as the dimension-free MSE is technically valid due to $\mathcal{O}\left(N^{-1}{d}^{-2/p}\right)$. But, such results are going to be taken with a particular caution when $\left|u\right|=d$ or when $\left|u\right|$ is too close to d, as the Assumption (A2) made to derive such results depends on the dimensionality. It is to be noted that for highly smooth functions, such a caution is no longer needed, despite L still depending on the dimensionality.

Remark 4. 

The choice $h=N^{-\gamma }$ with $\gamma \in [{\displaystyle \frac{1}{2}},1]$ results from the common practice in nonparametric estimations, which requires $hN\to +\infty$ when $N\to \infty$ and the willing to obtain $h^2\le 1/N$. In the case of derivatives, the bandwidth h should be close to zero by definition, and it must be less than $1/\sqrt{N}$ to obtain the parametric rate of convergence. Thus, $h\in \left[ϵ,1/N\right]$ is recommended with $ϵ\to 0$ being the precision of the computer used, provided that $0<ϵ\le 1/N$. Otherwise $h=ϵ$.

4. Estimator of the Hessian Matrix

Based on the above sections, the estimators of the off-diagonal elements of the Hessian matrix are straightforward by using $\left|u\right|=2$, $L=3$ and the set of constraints

$${\displaystyle \sum _{\mathcal{l}=1}^{L=3}{C}_{\mathcal{l}}^{\left(2\right)}{\beta }_{\mathcal{l}}^r={\delta }_{2,r};r=0,1,2.}$$

(13)

For all $u\subseteq \{1,\dots ,d\}$ with $\left|u\right|=2$, such estimators are given by

$${\displaystyle \widehat{{\mathcal{D}}_N^{\left|u\right|}f}\left(\mathbf{x}\right)=\frac{1}{N{h}^2{\sigma }_2^2}\sum _{i=1}^N\sum _{\mathcal{l}=1}^{L=3}{C}_{\mathcal{l}}^{\left(2\right)}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}\mathit{h}{\mathbf{V}}_i\right)\prod _{k\in u}{V}_{i,k}.}$$

To provide the estimator of the Hessian matrix (noted ${\widehat{H}}_N$) in Corollary 5, consider integers $q_j\in \mathbb{N}$ with $j=1,\dots ,d$, and the matrix

$${\displaystyle {\mathcal{E}}_q:=\left[\begin{array}{ccccc}\mathbb{E}\left[V_1^{2q_1+2}\right]& \mathbb{E}\left[V_2^2{V}_1^{2q_1}\right]& \mathbb{E}\left[V_2^2{V}_1^{2q_1}\right]& \dots & \mathbb{E}\left[V_2^2{V}_1^{2q_1}\right]\\ \mathbb{E}\left[V_2^2{V}_1^{2q_2}\right]& \mathbb{E}\left[V_1^{2q_2+2}\right]& \mathbb{E}\left[V_2^2{V}_1^{2q_2}\right]& \dots & \mathbb{E}\left[V_2^2{V}_1^{2q_2}\right]\\ \mathbb{E}\left[V_2^2{V}_1^{2q_3}\right]& \mathbb{E}\left[V_2^2{V}_1^{2q_3}\right]& \mathbb{E}\left[V_1^{2q_3+2}\right]& \dots & \mathbb{E}\left[V_2^2{V}_1^{2q_3}\right]\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ \mathbb{E}\left[V_2^2{V}_1^{2q_d}\right]& \mathbb{E}\left[V_2^2{V}_1^{2q_d}\right]& \dots & \dots & \mathbb{E}\left[V_1^{2q_d+2}\right]\end{array}\right]\in {\mathbb{R}}^{d\times d}.}$$

Corollary 5. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, $h_j=h$, $L=3$ and $q_j\in \mathbb{N}$. Assume that (A1) and (A2) hold. Then, the estimators of the off-diagonal terms of ${\widehat{H}}_N$ are given by

$${\displaystyle {\left({\widehat{H}}_N\right)}_{k_1{k}_2}\left(\mathbf{x}\right):=\widehat{{\mathcal{D}}_N^{\left|\right\{k_1,k_2\left\}\right|}f}\left(\mathbf{x}\right),\forall k_1,k_2\in \{1,\dots ,d\},k_1\ne k_2.}$$

Moreover, if $q_j>0,\forall j\in \{1,\dots ,d\}$, then the estimators of the diagonal terms are

$${\displaystyle diag\left({\widehat{H}}_N\right):=\frac{2}{N{h}^2}{\mathcal{E}}_q^{-1}\sum _{i=1}^N\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(2\right)}\left[\begin{array}{c}{V}_{i,1}^{2q_1}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h{\mathbf{V}}_i\right)\\ ⋮\\ V_{i,d}^{2q_d}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h\mathbf{V}\right)\end{array}\right].}$$

Proof. 

See Appendix E. □

Remark 5. 

The estimators of the Hessian-diagonal terms provided in Corollary 5 are exactly the unbiased estimators of the surrogates of Hessian-diagonal terms given by (see Appendix E).

$${\displaystyle diag\left(H\right)\approx \frac{2}{h^2}{\mathcal{E}}_q^{-1}\sum _{\mathcal{l}=1}^{L=3}{C}_{\mathcal{l}}^{\left(2\right)}\mathbb{E}\left[\begin{array}{c}{V}_1^{2q_1}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h\mathbf{V}\right)\\ ⋮\\ V_d^{2q_d}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h\mathbf{V}\right)\end{array}\right].}$$

A reasonable choice of $q_j$s is $q_j=1$ for any $j\in \{1,\dots ,d\}$, which yields a simple expression of $diag\left({\widehat{H}}_N\right)$. Indeed, one can check that the diagonal terms of the Hessian estimator become

$${\displaystyle diag\left({\widehat{H}}_N\right):=\frac{2}{N{h}^2}{\mathcal{E}}_1^{-1}\sum _{i=1}^N\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(2\right)}{\mathbf{V}}_i^{2}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h{\mathbf{V}}_i\right),}$$

where ${\mathcal{E}}_1^{-1}$ is known. Keeping in mind the following identities [36]:

$${\displaystyle \mathbb{E}\left[U_1^4\right]=\frac{\Gamma (5/p)\Gamma (d/p)}{\Gamma (1/p)\Gamma \left(\right(d+4)/p)}\mathbb{E}\left[U_2^2{U}_1^2\right]=\frac{{\Gamma }^2(3/p)\Gamma (d/p)}{{\Gamma }^2(1/p)\Gamma ((d+4)/p)},}$$

the expression of ${\mathcal{E}}_1$ is then given by

$${\displaystyle {\mathcal{E}}_1:=\mathbb{E}\left[R^4\right]\mathbb{E}\left[U_2^2{U}_1^2\right]{\mathcal{A}}_1=\frac{{\Gamma }^2(3/p)\Gamma (d/p)}{{\Gamma }^2(1/p)\Gamma ((d+4)/p)}\mathbb{E}\left[R^4\right]{\mathcal{A}}_1,}$$

(14)

with

$${\displaystyle {\mathcal{A}}_1:=\left[\begin{array}{ccccc}\vartheta & 1& 1& \dots & 1\\ 1& \vartheta & 1& \dots & 1\\ 1& 1& \vartheta & \dots & 1\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& 1& \dots & \dots & \vartheta \end{array}\right];\vartheta :=\frac{\mathbb{E}\left[U_1^4\right]}{\mathbb{E}\left[U_2^2{U}_1^2\right]}=\frac{\Gamma (5/p)\Gamma (1/p)}{{\Gamma }^2(3/p)}.}$$

It is to be noted that ${\mathcal{A}}_1$ is invertible, as well as ${\mathcal{E}}_1$. The inverse of ${\mathcal{A}}_1$ is given by

$${\displaystyle {\left({\mathcal{A}}_1^{-1}\right)}_{k_1{k}_2}=\frac{1}{(\nu -1+d)(\nu -1)}\times \left\{\begin{array}{cc}\nu -2+d& \mathrm{if}{k}_1=k_2\\ -1& \mathrm{otherwise}\end{array}\right.,}$$

by applying the Sherman–Morrison formula. Theorem 3 provides the upper-bound of the bias of the Hessian estimator by making use of

$${\displaystyle K_{5,d,p}:=\frac{\Gamma \left(\right(d+4)/p)}{{\Gamma }^2(3/p)\Gamma \left((d+5)/p\right)}\left[\Gamma \left(6/p\right)\Gamma \left(1/p\right)+(d-1)\Gamma \left(5/p\right)\Gamma \left(2/p\right)\right],}$$

$${\displaystyle K_{1,p,d,2}=\frac{\Gamma \left((d+4)/p\right)}{\Gamma \left(d/p\right)}{\left[\frac{\Gamma \left(1/p\right)}{\Gamma \left(3/p\right)}\right]}^2{\left\{\frac{\Gamma (9/p)\Gamma (d/p)}{\Gamma (1/p)\Gamma \left(\right(d+8)/p)}\right\}}^{1/2},}$$

and the spectral norm of ${\mathcal{A}}_1^{-1}$ given by $\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s$.

Theorem 3. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, $h_j=h$, $L=3$; $q_j=1$. Assume that (A1) and (A2) hold. Then, the bias of ${\widehat{H}}_N$ is given by

$${\displaystyle {\left|\left|diag\left(H\left(\mathbf{x}\right)\right)-\mathbb{E}\left[diag\left({\widehat{H}}_N\left(\mathbf{x}\right)\right)\right]\right|\right|}_2\le 2\sqrt{d}h{M}_3{\mathsf{F}}_3{K}_{5,d,p}\frac{\mathbb{E}\left[R^5\right]}{\mathbb{E}\left[R^4\right]}\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s.}$$

$${\displaystyle {\left|\left|H\left(\mathbf{x}\right)-\mathbb{E}\left[{\widehat{H}}_N\left(\mathbf{x}\right)\right]\right|\right|}_F\le \sqrt{d}h{M}_3{\mathsf{F}}_3\frac{\mathbb{E}\left[R^5\right]}{\mathbb{E}\left[R^4\right]}\sqrt{(d^2-d)C_p^2{K}_{1,p,d,2}^2+4K_{5,d,p}^2\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2}.}$$

Proof. 

See Appendix F. □

Concerning the choice of R, taking $R\sim {\delta }_{\eta }$ and the appropriated value of ${\sigma }_1$ allows for obtaining the dimension-free upper-bound of the bias. Define

$${\displaystyle {\sigma }_1^{\ast }:={\left(\sqrt{d}{\mathsf{F}}_3\sqrt{(d^2-d)C_p^2{K}_{1,p,d,2}^2+4K_{5,d,p}^2\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2}\frac{{\Gamma }^{1/2}(1/p){\Gamma }^{1/2}(d/p+2/p)}{{\Gamma }^{1/2}(3/p){\Gamma }^{1/2}(d/p)}\right)}^{-1}.}$$

(15)

Corollary 6. 

Under the conditions of Theorem 3, Let $R\sim {\delta }_{\eta }$. If ${\sigma }_1={\sigma }_1^{\ast }$, then

$${\displaystyle {\left|\left|H\left(\mathbf{x}\right)-\mathbb{E}\left[{\widehat{H}}_N\left(\mathbf{x}\right)\right]\right|\right|}_F\le h{M}_3.}$$

Proof. 

It is a consequence of Theorem 3 using the value of $\eta$. Indeed, based on this choice, the upper-bound of the bias of the Hessian estimator is given by

$${\displaystyle \sqrt{d}h{\sigma }_1{M}_3{\mathsf{F}}_3\sqrt{(d^2-d)C_p^2{K}_{1,p,d,2}^2+4K_{5,d,p}^2\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2}\frac{{\Gamma }^{1/2}(1/p){\Gamma }^{1/2}(d/p+2/p)}{{\Gamma }^{1/2}(3/p){\Gamma }^{1/2}(d/p)}.}$$

□

Based on such a dimension-free upper-bound of the bias, we now need the variance of the vectorized estimator of the Hessian (i.e., $\mathrm{Vec}\left({\widehat{H}}_N\right)$) so as to derive the MSE of the Hessian estimator. To that end, consider

$${\displaystyle K_{6,d,p}:={\left[\frac{{\Gamma }^2(1/p)\Gamma ((d+4)/p)}{{\Gamma }^2(3/p)\Gamma (d/p)}\right]}^2\frac{C_1\Gamma (4/p)}{p{c}_1^{4/p}},}$$

with $c_1,C_1$ being constants that depend only on p, and define

$${\displaystyle K_{7,d,p}:=\frac{K_{3,p,2}{\Gamma }^2\left((d+4)/p\right)}{{\Gamma }^{1/2}(d/p+8/p){\Gamma }^{3/2}\left(d/p\right)}.}$$

Theorem 4. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, $h_j=h$, $L=3$; $q_j=1$. Assume that (A1) and (A2) hold. Then, the MSEs are given by

$$\begin{array}{ccc}\hfill {\displaystyle \mathbb{E}\left[{\left|\left|diag\left(H\left(\mathbf{x}\right)\right)-diag\left({\widehat{H}}_N\left(\mathbf{x}\right)\right)\right|\right|}_2^2\right]}& \le & 4d{h}^2{M}_3^2{\mathsf{F}}_3^2{K}_{5,d,p}^2{\displaystyle \frac{{\mathbb{E}}^2\left[R^5\right]}{{\mathbb{E}}^2\left[R^4\right]}}\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2\hfill \\ & & +{\displaystyle \frac{64M_2^2{\mathsf{F}}_2^2}{N{d}^{2/p}}}{K}_{6,d,p}{\displaystyle \frac{\mathbb{E}\left[R^8\right]}{{\mathbb{E}}^2\left[R^4\right]}}\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2.\hfill \end{array}$$

$$\begin{array}{ccc}\hfill {\displaystyle \mathbb{E}\left[{\left|\left|H\left(\mathbf{x}\right)-{\widehat{H}}_N\left(\mathbf{x}\right)\right|\right|}_F^2\right]}& \le & d{h}^2{M}_3^2{\mathsf{F}}_3^2{\displaystyle \frac{{\mathbb{E}}^2\left[R^5\right]}{{\mathbb{E}}^2\left[R^4\right]}}\left[(d^2-d)C_p^2{K}_{1,2}^2+4K_{5,d,p}^2\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2\right]\hfill \\ & & +{\displaystyle \frac{M_2^2{\mathsf{F}}_2^2}{N{d}^{2/p}}}{\displaystyle \frac{\mathbb{E}\left[R^8\right]}{{\mathbb{E}}^2\left[R^4\right]}}\left[64\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2{K}_{6,d,p}+(d^2-d)K_{7,d,p}\right].\hfill \end{array}$$

Proof. 

See Appendix G. □

The results provided in Theorem 4 can be used for deriving dimension-free MSEs of $diag\left({\widehat{H}}_N\left(\mathbf{x}\right)\right)$ by taking $R\sim {\delta }_{\eta }$ and ${\sigma }_1={\sigma }_1^{\ast }$. Indeed, the following approximations of $K_{6,d,p}$ help for deducing dimension-free MSE of $diag\left({\widehat{H}}_N\left(\mathbf{x}\right)\right)$:

$${\displaystyle K_{6,d,p}={\left[\frac{{\Gamma }^2(1/p)\Gamma ((d+4)/p)}{{\Gamma }^2(3/p)\Gamma (d/p)}\right]}^2\frac{C_1\Gamma (4/p)}{p{c}_1^{4/p}}\approx \frac{{\Gamma }^4(1/p)}{{\Gamma }^4(3/p)}\frac{C_1\Gamma (4/p)}{p^{1+8/p}{c}_1^{4/p}}{d}^{8/p},}$$

for higher-values of d and $p\ll d$, and

$${\displaystyle K_{6,d,p}={\left[\frac{{\Gamma }^2(1/p)\Gamma ((d+4)/p)}{{\Gamma }^2(3/p)\Gamma (d/p)}\right]}^2\frac{C_1\Gamma (4/p)}{p{c}_1^{4/p}}\approx 81\frac{C_1}{4c_1^{4/p}}{\left(\frac{d}{d+4}\right)}^2\le 81\frac{C_1}{4c_1^{4/p}},}$$

when $d\ll p$. But, it is difficult to deduce dimension-free MSEs of the Hessian estimator using Theorem 4. Being able to derive the dimension-free MSE of ${\widehat{H}}_N\left(\mathbf{x}\right)$ requires working directly on the full estimator of the Hessian matrix. By considering the off-diagonal and diagonal terms simultaneously, the Hessian estimator is then given by

$${\displaystyle {\widehat{H}}_N\left(\mathbf{x}\right):=\frac{1}{N{h}^2}\sum _{i=1}^N\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(2\right)}f\left(\mathbf{x}+{\beta }_{\mathcal{l}}h{\mathbf{V}}_i\right){\mathcal{M}}_i,}$$

where the entries of the matrix ${\mathcal{M}}_i$ are

$${\displaystyle {\left({\mathcal{M}}_i\right)}_{k_1{k}_2}:=\left\{\begin{array}{cc}2\sum _{r=1}^d{\left({\mathcal{E}}_1^{-1}\right)}_{k_{1}r}{V}_{i,r}^2& \mathrm{if}{k}_1=k_2\\ V_{i,k_1}{V}_{i,k_2}/{\sigma }_2^2& \mathrm{otherwise}\end{array}\right..}$$

Theorem 5. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$, $h_j=h$, $L=3$; $q_j=1$. Assume that (A1) and (A2) hold. Then,

$$\begin{array}{ccc}\hfill {\displaystyle \mathbb{E}\left[{\left|\left|H\left(\mathbf{x}\right)-{\widehat{H}}_N\left(\mathbf{x}\right)\right|\right|}_F^2\right]}& \le & d{h}^2{M}_3^2{\mathsf{F}}_3^2{\displaystyle \frac{{\mathbb{E}}^2\left[R^5\right]}{{\mathbb{E}}^2\left[R^4\right]}}\left[(d^2-d)C_p^2{K}_{1,2}^2+4K_{5,d,p}^2\left|\right|{\mathcal{A}}_1^{-1}{\left|\right|}_s^2\right]\hfill \\ & & +{\displaystyle \frac{M_2^2{\mathsf{F}}_2^2{C}_2}{N{d}^{2/p}}}{\displaystyle \frac{\mathbb{E}\left[R^8\right]}{{\mathbb{E}}^2\left[R^4\right]}}\Gamma (4/p){\left[{\displaystyle \frac{{\Gamma }^2(1/p)\Gamma ((d+4)/p)}{{\Gamma }^2(3/p)\Gamma (d/p)}}\right]}^2.\hfill \end{array}$$

Proof. 

See Appendix H. □

Corollary 7. 

Consider $\mathbf{V}\sim {\mathcal{S}}_{d,p}$ with $R\sim {\delta }_{\eta }$; $h=N^{-\gamma }$ with $\gamma \in [{\displaystyle \frac{1}{2}},1]$, $L=3$; $q_j=1$. Assume that (A1) and (A2) hold. If ${\sigma }_1={\sigma }_1^{\ast }$, then

$${\displaystyle \mathbb{E}\left[{\left|\left|H\left(\mathbf{x}\right)-{\widehat{H}}_N\left(\mathbf{x}\right)\right|\right|}_F^2\right]\le \mathcal{O}\left(N^{-1}\frac{d^{-2/p}{\Gamma }^2((d+4)/p)}{{\Gamma }^2(d/p)}\right).}$$

Proof. 

It is a direct consequence of Theorem 5 and Corollary 6. □

Remark 6. 

The quantity ${\displaystyle \frac{d^{-2/p}{\Gamma }^2((d+4)/p)}{{\Gamma }^2(d/p)}}$ from Corollary 7 can be approximated as follows:

$${\displaystyle \frac{d^{-2/p}{\Gamma }^2((d+4)/p)}{{\Gamma }^2(d/p)}\approx \left\{\begin{array}{cc}{d}^{6/p}{p}^{-8/p}& ifp\ll d\\ \frac{d^2}{{(d+4)}^2}\le 1& ifd\ll p\end{array}\right..}$$

It turns out that the dimension-free upper-bound of the MSE of ${\widehat{H}}_N\left(\mathbf{x}\right)$ is reached by taking (i) higher values of p, that is, $d\ll p$, or (ii) p such that $6log\left(d\right)\le p\ll d$.

Remark 7. 

When the fourth-order derivatives are not close to zero, we may add one more constraint to increase the accuracy of the second-order moment, leading to $L=4$ and

$${\displaystyle \sum _{\mathcal{l}=1}^{L=4}{C}_{\mathcal{l}}^{\left(2\right)}{\beta }_{\mathcal{l}}^r={\delta }_{2,r};r=0,1,2,4.}$$

5. Applications: Hessian Estimates

5.1. An Analytical Example Using a Quadratic Function

Consider $\mathbf{b}\in {\mathbb{R}}^d$, $c\in \mathbb{R}$, a symmetric matrix $\mathcal{P}\in {\mathbb{R}}^{d\times d}$ and the function $f_0:{\mathbb{R}}^d\to \mathbb{R}$ given by

$$f_0\left(\mathbf{x}\right):={\mathbf{x}}^T\mathcal{P}\mathbf{x}/2+{\mathbf{b}}^T\mathbf{x}+c.$$

Keeping in mind Equation (13) and the fact that

$${\displaystyle \mathbb{E}\left[{\mathbf{V}}^T\mathcal{P}\mathbf{V}{\mathbf{V}}^2\right]=\mathbb{E}\left[{\mathbf{V}}^2\sum _{k=1}^d{\mathcal{P}}_{kk}{V}_k^2\right]={\mathcal{E}}_{1}diag\left(\mathcal{P}\right),}$$

the approximations of the diagonal terms are given by (see Remark 5).

$${\displaystyle diag\left(H\right)\approx \frac{2}{h^2}{\mathcal{E}}_1^{-1}\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(2\right)}\mathbb{E}\left[\begin{array}{c}{V}_1^2{f}_0\left(\mathbf{x}+{\beta }_{\mathcal{l}}h\mathbf{V}\right)\\ ⋮\\ V_d^2{f}_0\left(\mathbf{x}+{\beta }_{\mathcal{l}}h\mathbf{V}\right)\end{array}\right]={\mathcal{E}}_1^{-1}\mathbb{E}\left[\begin{array}{c}{V}_1^2{\mathbf{V}}^T\mathcal{P}\mathbf{V}\\ ⋮\\ V_d^2{\mathbf{V}}^T\mathcal{P}\mathbf{V}\end{array}\right]=diag\left(\mathcal{P}\right).}$$

Concerning the off-diagonal terms, one can write: $\forall k_1,k_2\in \{1,\dots ,d\},k_1\ne k_2$

$$\begin{array}{ccc}\hfill {\displaystyle H_{k_1{k}_2}\left(\mathbf{x}\right)}& \approx & {\displaystyle \frac{1}{h^2{\sigma }_2^2}}\sum _{\mathcal{l}=1}^L{C}_{\mathcal{l}}^{\left(2\right)}\mathbb{E}\left[V_{k_1}{V}_{k_2}{f}_0\left(\mathbf{x}+{\beta }_{\mathcal{l}}h\mathbf{V}\right)\right]={\displaystyle \frac{1}{2{\sigma }_2^2}}\mathbb{E}\left[V_{k_1}{V}_{k_2}{\mathbf{V}}^T\mathcal{P}\mathbf{V}\right]\hfill \\ & =& {\displaystyle \frac{1}{{\sigma }_2^2}}{\mathcal{P}}_{k_1{k}_2}\mathbb{E}\left[V_{k_1}^2{V}_{k_2}^2\right]={\mathcal{P}}_{k_1{k}_2}.\hfill \end{array}$$

5.2. Illustrations

This section provides simulated results using the Rosenbrock function (see [42]) and the synthetic function (see [20]), which are defined as follows: $\forall \mathbf{x}\in {\mathbb{R}}^d$,

$$r\left(\mathbf{x}\right):=\sum _{k=1}^{d-1}\left[{(1-x_k)}^2+100{\left(x_{k+1}-x_k^2\right)}^2\right],$$

$$s\left(\mathbf{x}\right):=\sum _{k=1}^{d/2}\left[M_{2}sin\left(x_{2k-1}\right)+cos\left(x_{2k}\right)\right]+{\displaystyle \frac{M_1-M_2}{400}}{\mathbf{x}}^T{\mathsf{1}}_{200\times 200}\mathbf{x},$$

with $M_1=200,M_2=0.001$ and ${\mathsf{1}}_{200\times 200}\in {\mathbb{R}}^{200\times 200}$ the unit matrix (i.e., its entries are one).

The function Hessian from the R-package numDeriv [43] is used for estimating the Hessian matrix (i.e., H). Such estimates serve as the true Hessian values, and the following error is considered so as to assess the efficiency of the proposed estimators:

$$Err:={\displaystyle \frac{{\left|\left|H_r\left(\mathbf{0}\right)-\widehat{H_r}\left(\mathbf{0}\right)\right|\right|}_F}{{\left|\left|H_r\left(\mathbf{0}\right)\right|\right|}_F}},$$

with $\widehat{H_r}\left(\mathbf{0}\right)$ being the estimated values of the Hessian matrix of r at $\mathbf{0}$. Moreover, the R-package LHS and the R-package greybox are used for generating the values of the d independent p-generalized (standard) Gaussian variables (i.e., $\mathbf{G}:=(G_1,\dots ,G_d$)), which are then used to obtain $V_j=G_j/{\left|\right|\mathbf{G}\left|\right|}_p;j=1,\dots ,d$. Finally, as $\mathbb{E}\left[V_{j_1}{V}_{j_2}\right]=0$ and $\mathbb{E}\left[V_{j_1}{V}_{j_2}\right]$ is practical approximated by ${\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{V}_{i,j_1}{V}_{i,j_2}$, the Gram–Schmidt procedure is applied (when $N>d$) to obtain values of $V_j$s such that ${\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{V}_{i,j_1}{V}_{i,j_2}=0$ for all $j_1,j_2\in \{1,\dots ,d\}$ and $j_1\ne j_2$. Such a procedure aims at obtaining perfect-orthogonal generated values of $V_j$s, and it is only a numerical requirement to speed up the convergence of Hessian estimates.

For higher values of p (i.e., $p\ge 2000$), the corresponding p-generalized (standard) Gaussian distribution is approximated by the uniform distribution $\mathcal{U}(-1,1)$.

For simulation purposes, the values ${\beta }_1=-1,{\beta }_2=0,{\beta }_3=1$ are used, leading to ${\mathsf{F}}_3=1$ (see Remark 2). To assess the impact of h on the estimates, $h=1/\sqrt{N}$, $h=1/N$ and $h={10}^{-4}$ are considered in concordance with Remark 4. Theoretically, the value ${\sigma }_1={\sigma }_1^{\ast }$ given by Equation (15) should be used. But, it depends on unknown constants, leading to adopt a pragmatic choice of ${\sigma }_1=d^{-3/2}{\left({\displaystyle \frac{{\Gamma }^{1/2}\left(1/p\right){\Gamma }^{1/2}\left((d+2)/p\right)}{{\Gamma }^{1/2}\left(3/p\right){\Gamma }^{1/2}\left(d/p\right)}}\right)}^{-1}$, which is an approximation of ${\sigma }_1^{\ast }$.

When $d=10$, imposing $p\ll d=10$ leads to dimension-free MSEs for $\left|u\right|\ll 1+{\displaystyle \frac{d}{2log\left(d\right)}}=3.17147$. The Hessian estimates corresponding to $\left|u\right|=2$ aim at demonstrating the breakdown of the dimension-free property, as $6log\left(10\right)=13$ (see Remark 6). The values $p\gg 10$ ensure the dimension-free property. We have repeated the procedure of the Hessian estimates $R=50$ times, and

Table 1. Average of 50 values of $Err$ for different values of p and N using the Rosenbrock function with $d=10$, $h=1/\sqrt{N}$ and $\mathbf{x}=\mathbf{0}$.

N	$\mathit{p}=1$	$\mathit{p}=2$	$\mathit{p}=14$	$\mathit{p}=15$	$\mathit{p}=1000$	$\mathit{p}=1500$
11	1.189	1.492	2.361	2.422	1.656	1.694
15	1.530	1.375	2.178	2.183	1.970	1.905
20	1.328	1.191	1.865	1.848	2.194	2.168
40	1.122	0.759	1.117	1.237	2.156	2.320
100	0.691	0.459	0.606	0.626	1.624	1.699
500	0.319	0.181	0.222	0.228	0.733	0.789
1000	0.220	0.124	0.150	0.153	0.547	0.587

,

Table 2. Average of 50 values of $Err$ for different values of p and N using the Rosenbrock function with $d=10$, $h=1/N$ and $\mathbf{x}=\mathbf{0}$.

N	$\mathit{p}=1$	$\mathit{p}=2$	$\mathit{p}=14$	$\mathit{p}=15$	$\mathit{p}=1000$	$\mathit{p}=1500$
11	1.280	1.435	2.392	2.287	1.683	1.778
15	1.389	1.323	2.148	2.230	2.012	2.020
20	1.365	1.146	1.886	1.894	2.220	2.227
40	1.109	0.777	1.195	1.193	2.225	2.184
100	0.694	0.447	0.590	0.590	1.624	1.634
500	0.319	0.186	0.225	0.224	0.772	0.796
1000	0.218	0.129	0.152	0.153	0.535	0.567

,

Table 3. Average of 50 values of $Err$ for different values of p and N using the Rosenbrock function with $d=10$, $h={10}^{-4}$ and $\mathbf{x}=\mathbf{0}$.

N	$\mathit{p}=1$	$\mathit{p}=2$	$\mathit{p}=14$	$\mathit{p}=15$	$\mathit{p}=1000$	$\mathit{p}=1500$
11	1.214	1.433	2.314	2.346	1.673	1.729
15	1.436	1.422	2.200	2.159	1.989	2.096
20	1.341	1.177	1.920	1.903	2.231	2.126
40	1.073	0.768	1.150	1.234	2.188	2.305
100	0.665	0.434	0.602	0.589	1.636	1.647
500	0.310	0.179	0.221	0.225	0.733	0.817
1000	0.218	0.126	0.153	0.156	0.542	0.589

,

Table 4. Average of 50 values of $Err$ for different values of p and N using the Rosenbrock function with $d=50$, $h=1/N$ and $\mathbf{x}=\mathbf{0}$.

N	$\mathit{p}=1$	$\mathit{p}=2$	$\mathit{p}=14$	$\mathit{p}=15$	$\mathit{p}=1000$	$\mathit{p}=1500$
11	56.611	56.185	57.549	57.858	57.255	56.891
15	49.101	47.404	48.153	48.401	48.088	48.071
20	42.449	40.580	40.966	40.858	41.381	41.306
40	30.090	28.249	28.835	28.755	28.950	28.780
100	2.206	3.269	7.278	7.376	7.472	7.546
500	1.270	0.963	2.012	2.023	2.081	2.073
1000	0.929	0.546	1.127	1.133	1.181	1.180

and

Table 5. Average of 50 values of $Err$ for different values of p and N using the synthetic function with $d=10$, $h=1/N$ and $\mathbf{x}=\mathbf{0}$.

N	$\mathit{p}=1$	$\mathit{p}=2$	$\mathit{p}=14$	$\mathit{p}=15$	$\mathit{p}=1000$	$\mathit{p}=1500$
11	1.052	1.087	1.316	1.326	1.141	1.139
15	1.200	1.116	1.342	1.342	1.278	1.193
20	1.145	1.145	1.269	1.301	1.261	1.272
40	0.942	0.923	0.998	1.078	1.375	1.421
100	0.637	0.594	0.714	0.711	1.146	1.136
500	0.281	0.266	0.316	0.326	0.603	0.625
1000	0.205	0.193	0.236	0.229	0.424	0.446

report the average values of the error measure $Err$ for different values of p and N.

It turns out that the error values decrease with the sample size (as expected). Moderate values of p and $p=2$ perform better compared to other values of p considered. Based on the functions considered in this paper, it appears that the caution imposed by Assumption (A2) does not affect the Hessian estimates. Table 4 shows that applying the Gram–Schmidt procedure when $N>d$ improves the Hessian estimates. Therefore, it is interesting to investigate the possibility of applying a kind of the Gram–Schmidt procedure when $N<d$.

6. Conclusions

In this paper, we have firstly provided stochastic surrogates and estimators of the $\left|u\right|$-th-order cross-partial derivatives ($\forall u\subseteq \{1,\dots ,d\}$) by making use of $L=\left|u\right|+1$ randomized points and the ${\mathcal{l}}_p$-spherical distributions. The proposed estimators reach the parametric rate of convergence and dimension-free MSEs when $\left|u\right|\ll 1+{\displaystyle \frac{d}{2log\left(d\right)}}$ by imposing $p\ll d$. On the contrary, taking $p\gg d$ always allows to reach dimension-free MSEs, but caution is needed when $\left|u\right|$ is too close to the dimensionality, as Assumption (A2) depends on the dimensionality. For highly smooth functions, dimension-free MSEs hold regardless of Assumption (A2). Secondly, dimension-free MSEs of the Hessian estimators have been provided under the minimal requirement on the function structures. Such interesting theoretical results are followed by an analytical example and simulations based on test cases, which have shown promising results. While the theoretical results help for choosing some hyper-parameters, some numerical issues remain, such as applying a kind of the Gram–Schmidt procedure when $N<d$. Consequently, more numerical investigations are needed in the near future so as to come out with a more efficient numerical tool.