Probabilistic Taylor-Type Expansions of Functions

Abstract

Taylor–Young and Maclaurin series are widely used for approximating smooth functions around a given point. This study investigates a unified stochastic framework for Taylor-type expansions of functions by means of independent random variables. The proposed probabilistic expansions of a function are able to incorporate evaluations of derivatives at different points, leading to a global approach. Exact expansions are obtained for any order of available derivatives, and such Taylor-type expansions enable the statistical inference of the remainder terms. It appears that the traditional Taylor–Young and Maclaurin series are particular cases of the proposed approach thanks to the Dirac probability measure, and guidelines for using Taylor series have been enhanced. Different ways of choosing the optimal distributions of random variables are provided, particularly when truncations are applied. Numerical comparisons are provided as well.

1. Introduction

Taylor’s theorem [1] is one of the most fundamental results in functional analysis. Taylor–Young and Maclaurin series are widely used for approximating smooth functions around a given point, a.k.a. a node. For instance, such series have been used in partial differential equations [2], in stochastic computations of gradients [3,4,5,6,7,8,9], and in statistical analysis of functions [10,11]. Other instances are listed in [12,13,14] and the references therein. Stochastic Taylor expansions or Ito–Taylor expansions have been proposed to expand functions of solutions of stochastic differential equations (see e.g., [15,16]), and a version of probabilistic Taylor’s expansion has been considered in [17,18,19,20,21] by treating the input as a random variable.

Basically, Taylor series consist of linearizing smooth functions using information about the derivatives of such functions at one given node only, which ease derivations of results in the aforementioned scientific fields and others. Linearizing a function at a single node leads to a local approximation, and the quality of such an approximation often decays quickly for the input values that are far away form this node. Such local approximations make it difficult to assess the behavior of numerical schemes and/or algorithms relying on Taylor series when the input values are not close to the node, because the remainder term is often unknown. Such epistemic uncertainties of Taylor series around given nodes have been addressed in [12] using a Gaussian process. The posterior variance of the Gaussian process given derivative data sets serves as a measure of credibility for the Taylor approximations when the input values are far away from the corresponding nodes.

The issues of choosing the unique node arise in some situations, as derivative data sets become available at different input values [22,23], and they can serve as nodes for function approximations. In contrast to one-node-based approximations, it is very interesting to make best use of such derivative data sets for function approximations, particularly when the first-order derivative information is available at some points, and the second-order one is observed at other points. Moreover, reducing the epistemic uncertainty due the remainder term has been considered in [13,14,24] by deriving the optimal Taylor-like formula, which requires evaluating the first-order and second-order derivatives at different nodes. Each node has a specific weight, leading to a kind of discrete random variable.

So far, the aforementioned expansions or approximations of functions have been developed in the spirit of Taylor’s theorem and/or the mean value theorem. In this paper, a new and unified stochastic framework for Taylor-type expansions of smooth functions is proposed by means of independent random variables. The proposed probabilistic expansions of a function are able to incorporate evaluations of derivatives at different points, leading to a global approach. It turns out that the derived nodes for the proposed approach are expectations of the random variables used, and the coefficients used in such expansions involve the moments of derivatives. Also, exact expansions are obtained for any order of available derivatives, and such expansions enable the statistical inference of the remainder terms. Moreover, the proposed approach can be deployed for approximating functions that are not differentiable at a set of countable points. It appears that the traditional Taylor–Young and Maclaurin series are particular cases of the proposed approach by allowing the random variables to follow the Dirac measure. Thus, taking the Dirac probability measure leads to enhanced guidelines for using Taylor series. Different ways of choosing the optimal distributions of random variables are provided when truncations are applied.

The paper is organized as follows: Section 2 provides the generic expansions of functions using the concept of zero-one-ended functions, including cumulative distribution functions. Cumulative distribution functions are used to derive the probabilistic Taylor-type expansions of functions. Links with the Taylor series are established in that section. Section 3 addresses a kind of convergence analysis by deriving the error bounds of truncated expansions at a given order. Such upper-bounds are then used for deriving the optimal cumulative distribution functions. Section 4 provides simulated results comparing the proposed approach to the traditional Taylor expansions using different functions, and we conclude this work in Section 5.

• Set up and general notation

Consider a set $\Omega \subseteq \mathbb{R}$ and a measurable and deterministic function $f:\Omega \to \mathbb{R}$ given by $x\mapsto f\left(x\right)$. Denote with Y a continuous random variable having F as the cumulative distribution function (CDF) and $\rho$ as the probability density function (PDF). Given an integer $p>0$, $Y^{\left(j\right)}$ with $j=1,\dots ,p$ are i.i.d. copies of Y. Also, denote with $\mathbb{E}[·]$ (resp. $\mathbb{V}[·]$) the expectation (resp. variance) operator taken w.r.t. $Y^{\left(j\right)}$ when there is no ambiguity.

To work with a large class of functions, the concept of weakly differentiable functions is needed. Let ${\mathcal{C}}_c^{\infty }\left(\Omega \right)$ be a class of testing functions, that is, any $\varphi \in {\mathcal{C}}_c^{\infty }$ has a compact support included in an open set $\Omega :=(a,b)$ and $\varphi \in {\mathcal{C}}^{\infty }$ [25,26]. For any $\varphi \in {\mathcal{C}}_c^{\infty }$, ${\varphi }^{\prime }$ is its derivative.

Definition 1

([25,26]). Let $h:\Omega \subseteq \mathbb{R}\to \mathbb{R}$.

A function f is said to be weakly differentiable if there exists a locally integrable function h such that for any $\varphi \in {\mathcal{C}}_c^{\infty }$,

$${\displaystyle {\int }_{\Omega }f\left(x\right){\varphi }^{\prime }\left(x\right)dx=-{\int }_{\Omega }h\left(x\right)\varphi \left(x\right)dx.}$$

The function h is the weak derivative of f w.r.t. x, that is, ${\displaystyle \frac{df}{dx}}\left(x\right)=h\left(x\right)$, and it is uniquely defined almost everywhere. Moreover, the distribution theory allows for defining the weak derivatives of f of every order. For a given integer $k>0$, denote with ${\mathcal{D}}^{k}f$ the kth-order weak derivative of f w.r.t. x, and define the following space:

$${\displaystyle W^p:=\left\{f:\Omega \to \mathbb{R}:f\in {\mathcal{C}}^p(\Omega ),{\mathcal{D}}^{k}f\in {\mathcal{L}}^2(\Omega ,F),\forall k\in \{0,1,\dots ,p\}\right\},}$$

(1)

where ${\mathcal{D}}^{k}f\in {\mathcal{L}}^2(\Omega ,F)$ means that ${\mathbb{E}}_{X\sim F}\left[{\left\{{\mathcal{D}}^{k}f\left(X\right)\right\}}^2\right]={\int }_{\Omega }{\left\{{\mathcal{D}}^{k}f\left(x\right)\right\}}^{2}F\left(dx\right)<+\infty$, with F being a CDF.

2. Expansions of a Function Using Other Functions

This section aims at providing expansions of a function $f:\Omega \to \mathbb{R}$ using its available weak derivatives and specific functions. We start with expansions of f based on specific functions (called zero-one-ended functions), followed by (i) the probabilistic expansions of the deterministic f using the CDFs of continuous random variables, and (ii) the links between such expansions and Taylor series.

Given $a\in \mathbb{R}$, $H_a\left(x\right):={1\mathrm{I}}_{[x\ge a]}$ denotes the Heaviside function or indicator function, that is, ${1\mathrm{I}}_{[x\ge a]}=1$ if $x\ge a$ and 0 otherwise. Its distribution derivative is the Dirac delta function or measure ${\delta }_a\left(x\right)$, leading to $H_a^{\prime }\left(x\right)={\delta }_a\left(x\right)$.

Definition 2.

A function $g:\Omega =(a,b)\to R$ is said to be zero-one-ended whenever

$${\displaystyle g\left(a_{+}\right):=\underset{x\to a}{lim}g\left(x\right)=0;g\left(b_{-}\right):=\underset{x\to b}{lim}g\left(x\right)=1.}$$

Obviously, when g has limits at a and b, then $g\left(a_{+}\right)=g\left(a\right)$ and $g\left(b_{-}\right)=g\left(b\right)$. Instances of zero-one-ended functions are $g_{\lambda }\left(x\right):=exp\left(\lambda x\right){1\mathrm{I}}_{\mathbb{R}-}\left(x\right)$ on ${\Omega }_0:={\mathbb{R}}_{-}$ with $\lambda >0$, and the large class of the well-known CDFs of continuous random variables, defined on the supports of such variables. Weak-differentiable and zero-one-ended functions (even negative) enable the expansions of every smooth function.

Assumption 1

(A1). The function f belongs to $W^{p+1}$ given by Equation (1).

Assumption (A1) provides the needful for not only ensuring that f is smooth enough and ${\mathcal{D}}^{k}f$ is integrable w.r.t. the probability measure F, but also for being able to work with finite second-order moments of ${\mathcal{D}}^{k}f$, when evaluating it at a random variable. The global expansion of f alongside the zero-one-ended function g is derived in the following Lemma.

Lemma 1.

Let $g\in W^1$ be a zero-one-ended function on Ω. Assume ${\int }_{\Omega }{\mathcal{D}}^{k}f\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy<\infty ,\forall k\le p$ and (A1) hold. Then, the pth-order expansion of f at $x_0\in \Omega$ is

$$\begin{array}{ccc}\hfill {\displaystyle f\left(x_0\right)}& =& {\int }_{\Omega }f\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy+\sum _{k=1}^p{\int }_{\Omega }{\mathcal{D}}^{k}f\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy{\int }_{{\Omega }^k}\prod _{j=1}^k\left(g\left(x_j\right)-{1\mathrm{I}}_{[x_j\ge x_{j-1}]}\right)d{x}_1\dots d{x}_k\hfill \\ & & +{\int }_{{\Omega }^{p+1}}{\mathcal{D}}^{p+1}f\left(x_{p+1}\right)\prod _{j=1}^{p+1}\left(g\left(x_j\right)-{1\mathrm{I}}_{[x_j\ge x_{j-1}]}\right)d{x}_1\dots d{x}_{p+1}.\hfill \end{array}$$

(2)

Proof. 

We derive this result using the recurrence reasoning. Firstly, the integration by part gives

$${\displaystyle {\int }_{\Omega }f\left(y\right)\frac{dg}{dy}\left(y\right)dy={\left[f\left(y\right)g\left(y\right)\right]}_a^b-{\int }_{\Omega }\mathcal{D}f\left(y\right)g\left(y\right)dy=f\left(b\right)-{\int }_{\Omega }\mathcal{D}f\left(y\right)g\left(y\right)dy.}$$

Adding an integral to the above equation yields the zero-order expansion of f. Indeed,

$$\begin{array}{ccc}\hfill {\displaystyle {\int }_{\Omega }f\left(y\right)\frac{dg}{dy}\left(y\right)dy+{\int }_{\Omega }\mathcal{D}f\left(y\right)\left(g\left(y\right)-{1\mathrm{I}}_{[y\ge x]}\right)dy}& =& f\left(b\right)-{\int }_{\Omega }\mathcal{D}f\left(y\right){1\mathrm{I}}_{[y\ge x]}dy\hfill \\ & =& f\left(b\right)-{\left[f\left(y\right)\right]}_x^b=f\left(x\right).\hfill \end{array}$$

(3)

Secondly, by applying the zero-order expansion to the function $\mathcal{D}f\left(y\right)$, that is,

$${\displaystyle \mathcal{D}f\left(y\right)={\int }_{\Omega }\mathcal{D}f\left(z\right)\frac{dg}{dz}\left(z\right)dz+{\int }_{\Omega }{\mathcal{D}}^{2}f\left(z\right)\left(g\left(z\right)-{1\mathrm{I}}_{[z\ge y]}\right)dz,}$$

and replacing $\mathcal{D}f\left(y\right)$ with the above expression in Equation (3), we obtain the first-order expansion of f given by

$$\begin{array}{ccc}\hfill {\displaystyle f\left(x\right)}& =& {\int }_{\Omega }f\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy+{\int }_{\Omega }\mathcal{D}f\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy{\int }_{\Omega }\left(g\left(y\right)-{1\mathrm{I}}_{[y\ge x]}\right)dy\hfill \\ & & +{\int }_{{\Omega }^2}{\mathcal{D}}^{2}f\left(z\right)\left(g\left(z\right)-{1\mathrm{I}}_{[z\ge y]}\right)\left(g\left(y\right)-{1\mathrm{I}}_{[y\ge x]}\right)dzdy.\hfill \end{array}$$

Thirdly, combining the remainder term of Equation (2) corresponding to the order $p-1$ with the zero-order expansion of ${\mathcal{D}}^{p}f\left(x_p\right)$, that is,

$${\displaystyle {\mathcal{D}}^{p}f\left(x_p\right)={\int }_{\Omega }{\mathcal{D}}^{p}f\left(x_{p+1}\right)\frac{dg}{d{x}_{p+1}}\left(x_{p+1}\right)d{x}_{p+1}+{\int }_{\Omega }{\mathcal{D}}^{p+1}f\left(x_{p+1}\right)\left(g\left(x_{p+1}\right)-{1\mathrm{I}}_{[x_{p+1}\ge x_p]}\right)d{x}_{p+1},}$$

yields

$$\begin{array}{ccc}\hfill {\displaystyle R_p}& :=& {\int }_{{\Omega }^p}{\mathcal{D}}^{p}f\left(x_p\right)\prod _{j=1}^p\left(g\left(x_j\right)-{1\mathrm{I}}_{[x_j\ge x_{j-1}]}\right)d{x}_1\dots d{x}_p\hfill \\ & =& {\int }_{\Omega }{\mathcal{D}}^{p}f\left(x_{p+1}\right){\displaystyle \frac{dg}{d{x}_{p+1}}}\left(x_{p+1}\right)d{x}_{p+1}{\int }_{{\Omega }^p}\prod _{j=1}^p\left(g\left(x_j\right)-{1\mathrm{I}}_{[x_j\ge x_{j-1}]}\right)d{x}_1\dots d{x}_p\hfill \\ & & +{\int }_{{\Omega }^{p+1}}{\mathcal{D}}^{p+1}f\left(x_{p+1}\right)\prod _{j=1}^{p+1}\left(g\left(x_j\right)-{1\mathrm{I}}_{[x_j\ge x_{j-1}]}\right)d{x}_1\dots d{x}_{p+1},\hfill \end{array}$$

and the result holds. □

The identity in Equation (2) is an exact expansion of f at any $x_0\in \Omega$. We then approximate f by omitting the last term of Equation (2). Such an approximation remains exact for polynomials of degrees up to p. In view of Lemma 1, there are many possibilities of choosing g. The natural choice of g in the probability theory is the CDF, as it can bring information about the input or the output space. Another interesting choice of g might rely on regression functions used in statistical modeling. Before taking the CDFs to represent g, let us introduce the moments of f evaluated at a random variable X using its first-order derivative and the exact expansion given by Equation (2). In what follows, denote $f\left(b_{-}\right):={lim}_{x\to b}f\left(x\right)$ and $f\left(a_{+}\right):={lim}_{x\to a}f\left(x\right)$, which become $f\left(a\right)$ and $f\left(b\right)$ when f has limits at such points.

Proposition 1.

Let $X\sim G$ be a random variable having G as the CDF. If $h\in W^1$, Then,

$$\mathbb{E}\left[h\left(X\right)\right]=h\left(b_{-}\right)-{\int }_{\Omega }{\displaystyle \frac{dh}{dy}}\left(y\right)G\left(y\right)dy.$$

Proof. 

For any smooth function h, using Lemma 1 and bearing in mind Fubini’s theorem [27,28], one can write

$$\begin{array}{ccc}\hfill {\displaystyle \mathbb{E}\left[h\left(X\right)\right]}& =& {\int }_{\Omega }h\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy+\underset{{\Omega }^2}{\int \int }{\displaystyle \frac{dh}{dy}}\left(y\right)\left(g\left(y\right)-{1\mathrm{I}}_{y\ge x}\right)dydG\left(x\right)\hfill \\ & =& {\int }_{\Omega }h\left(y\right){\displaystyle \frac{dg}{dy}}\left(y\right)dy+{\int }_{\Omega }{\displaystyle \frac{dh}{dy}}\left(y\right)\left(g\left(y\right)-G\left(y\right)\right)dy\hfill \\ & =& {\left[h\left(y\right)g\left(y\right)\right]}_a^b-{\int }_{\Omega }{\displaystyle \frac{dh}{dy}}\left(y\right)G\left(y\right)dy\hfill \\ & =& \underset{x\to b}{lim}h\left(x\right)-{\int }_{\Omega }{\displaystyle \frac{dh}{dy}}\left(y\right)G\left(y\right)dy,\hfill \end{array}$$

and the result holds. □

Applications of such a result give new identities involving the moments of functions evaluated at random variables. The following corollary provides such identities.

Corollary 1.

Let $X\sim G$ be a random variable having G as the CDF. Assume $f\in W^1$ and $f\left(X\right)$ has finite $p+1$th-order moment. Then, for any $r\le p+1$,

$$\mathbb{E}\left[f^r\left(X\right)\right]=f^r\left(b_{-}\right)-r{\int }_{\Omega }{f}^{r-1}\left(y\right)\mathcal{D}f\left(y\right)G\left(y\right)dy.$$

$$\mathbb{E}\left[{\left(f\left(X\right)-\mathbb{E}\left[f\left(X\right)\right]\right)}^r\right]={\left(f\left(b_{-}\right)-\mathbb{E}\left[f\left(X\right)\right]\right)}^r-r{\int }_{\Omega }{\left(f\left(y\right)-\mathbb{E}\left[f\left(X\right)\right]\right)}^{r-1}\mathcal{D}f\left(y\right)G\left(y\right)dy.$$

Proof. 

The first result holds using $h=f^r$, and the second one using $h={\left(f-\mathbb{E}\left[f\right(X\left)\right]\right)}^r$ (See Proposition 1). □

In view of Corollary 1, the moments of $f\left(X\right)$ do not depend on g, which is used to expand f. It is also shown in [29] that the variance of $f\left(X\right)$ is

$$\mathbb{V}\left[f\left(X\right)\right]={\int }_{{\Omega }^2}\mathcal{D}f\left(y\right)\mathcal{D}f\left(z\right)\left[G\left(min(y,z)\right)-G\left(y\right)G\left(z\right)\right]dydz.$$

2.1. Probabilistic Expansions of Functions

In this section, the expansions of f based on particular zero-one-ended functions, such as CDFs, are considered. Before giving such expansions, the following intermediate results are needed. Denote with F a continuous CDF and $\rho$ the corresponding probability density function, that is, $\mathcal{D}F=\rho$.

Proposition 2.

Let $x\in \Omega$, $\alpha \in \mathbb{R}$ and $Y\sim F$ be a random variable having finite $(p+1)$th-order moment. Then, the following identity holds:

$$\mathbb{E}\left[{(Y-\alpha )}^p{\displaystyle \frac{F\left(Y\right)-{1\mathrm{I}}_{[Y\ge x]}}{\rho \left(Y\right)}}\right]={\displaystyle \frac{1}{p+1}}\left\{{(x-\alpha )}^{p+1}-\mathbb{E}\left[{(Y-\alpha )}^{p+1}\right]\right\}.$$

(4)

Proof. 

Let $B:={\displaystyle \frac{1}{p+1}}\mathbb{E}\left[{(Y-\alpha )}^{p+1}\right]$. It follows from the integration by part that

$$\begin{array}{ccc}\hfill \mathbb{E}\left[{(Y-\alpha )}^p{\displaystyle \frac{F\left(Y\right)-{1\mathrm{I}}_{[Y\ge x]}}{\rho \left(Y\right)}}\right]& =& {\int }_{\Omega }{(y-\alpha )}^{p}F\left(y\right)dy-{\int }_x^b{(y-\alpha )}^{p}dy\hfill \\ & =& {\displaystyle \frac{1}{p+1}}{\left[{(y-\alpha )}^{p+1}F\left(y\right)\right]}_a^b-B-{\displaystyle \frac{1}{p+1}}{\left[{(y-\alpha )}^{p+1}\right]}_x^b\hfill \\ & =& {\displaystyle \frac{1}{p+1}}{(x-\alpha )}^{p+1}-{\displaystyle \frac{1}{p+1}}\mathbb{E}\left[{(Y-\alpha )}^{p+1}\right],\hfill \end{array}$$

and the result holds. □

Lemma 2.

Let $Y^{\left(0\right)}:=x\in \Omega$, $Y\sim F$ be a random variable and ${\left\{Y^{\left(j\right)}\right\}}_{j=1}^{p+1}$ be i.i.d. copies of Y. If Y has finite $(p+1)$th-order moment, then

$$\mathbb{E}\left[\prod _{j=1}^p{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]=\sum _{j=0}^p{\displaystyle \frac{x^j}{j!}}{U}_{p-j},$$

(5)

with the sequences

$${\displaystyle U_0:=1,U_{k+1}=-\sum _{i=1}^{k+1}\frac{\mathbb{E}\left[Y^i\right]}{i!}{U}_{k+1-i}.}$$

(6)

Proof. 

Let us show the result using recurrence reasoning. The first step (i.e., $p=1$) holds using (4) with $\alpha =\mathbb{E}\left[Y\right]$, that is,

$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge x]}}{\rho \left(Y^{\left(1\right)}\right)}}\right]& =& x-\mathbb{E}\left[Y\right].\hfill \end{array}$$

Secondly, for all $q=1,\dots ,p$, suppose that Equation (5) is still valid.

Thirdly, we are going to show that (5) is also true when $q=p+1$. Indeed,

$$\begin{array}{ccc}& & \mathbb{E}\left[\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]\hfill \\ & =& {\mathbb{E}}_{Y^{\left(1\right)}}\left[{\mathbb{E}}_{Y^{\left(2\right)}-Y^{(p+1)}}\left[\prod _{j=2}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge Y^{\left(0\right)}]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)\right]\hfill \\ & =& {\mathbb{E}}_{Y^{\left(1\right)}}\left[{\mathbb{E}}_{Y^{\left(2\right)}-Y^{(p+1)}}\left[\prod _{k=1}^p{\displaystyle \frac{F\left(Y^{(k+1)}\right)-{1\mathrm{I}}_{[Y^{(k+1)}\ge Y^{\left(k\right)}]}}{\rho \left(Y^{(k+1)}\right)}}\right]\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge Y^{\left(0\right)}]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)\right]\hfill \\ & \stackrel{Step2}{=}& \sum _{j=0}^p{\displaystyle \frac{U_{p-j}}{j!}}{\mathbb{E}}_{Y^{\left(1\right)}}\left[Y^{{\left(1\right)}^j}\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge Y^{\left(0\right)}]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)\right]\hfill \\ & \stackrel{\left(4\right)}{=}& \sum _{j=0}^p{\displaystyle \frac{U_{p-j}}{(j+1)!}}{Y}^{{\left(0\right)}^{j+1}}-\sum _{j=0}^p{\displaystyle \frac{U_{p-j}}{(j+1)!}}\mathbb{E}\left[Y^{j+1}\right]\hfill \\ & =& \sum _{j^{\prime }=1}^{p+1}{\displaystyle \frac{U_{p+1-j^{\prime }}}{\left(j^{\prime }\right)!}}{Y}^{\left(0\right)j^{\prime }}+U_{p+1}=\sum _{j^{\prime }=0}^{p+1}{\displaystyle \frac{U_{p+1-j^{\prime }}}{\left(j^{\prime }\right)!}}{Y}^{\left(0\right)j^{\prime }},\hfill \end{array}$$

using the change of variables of the form $j^{\prime }=j+1$. □

It is sometimes very interesting to have the above expression around a constant, such as the expectation of Y. Such a result is provided in Lemma 3.

Lemma 3.

Let $Y^{\left(0\right)}:=x\in \Omega$, $Y\sim F$ be a random variable and ${\left\{Y^{\left(j\right)}\right\}}_{j=1}^{p+1}$ be i.i.d. copies of Y. If Y has finite $(p+1)$th-order moment, then

$$\mathbb{E}\left[\prod _{j=1}^p{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]=\sum _{j=0}^p{V}_{p-j}{\displaystyle \frac{{\left(x-\mathbb{E}\left[Y\right]\right)}^j}{j!}},$$

(7)

with the sequences

$${\displaystyle V_0:=1,V_1:=0,V_{k+1}=-\sum _{i=2}^{k+1}{V}_{k+1-i}\frac{\mathbb{E}\left[{\left(Y-\mathbb{E}\left[Y\right]\right)}^i\right]}{i!},\forall k\ge 1.}$$

(8)

Proof. 

The first step (i.e., $p=1$) holds using (4) with $\alpha =\mathbb{E}\left[Y\right]$, that is,

$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge x]}}{\rho \left(Y^{\left(1\right)}\right)}}\right]& =& x-\mathbb{E}\left[Y\right].\hfill \end{array}$$

Secondly, for all $q=1,\dots ,p$, suppose that Equation (7) is still valid.

Thirdly, we are going to show that (7) is also true when $q=p+1$. Indeed,

$$\begin{array}{ccc}& & \mathbb{E}\left[\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]\hfill \\ & =& {\mathbb{E}}_{Y^{\left(1\right)}}\left[{\mathbb{E}}_{Y^{\left(2\right)}-Y^{(p+1)}}\left[\prod _{j=2}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge Y^{\left(0\right)}]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)\right]\hfill \\ & =& {\mathbb{E}}_{Y^{\left(1\right)}}\left[{\mathbb{E}}_{Y^{\left(2\right)}-Y^{(p+1)}}\left[\prod _{k=1}^p{\displaystyle \frac{F\left(Y^{(k+1)}\right)-{1\mathrm{I}}_{[Y^{(k+1)}\ge Y^{\left(k\right)}]}}{\rho \left(Y^{(k+1)}\right)}}\right]\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge Y^{\left(0\right)}]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)\right]\hfill \\ & \stackrel{Step2}{=}& \sum _{j=0}^p{\displaystyle \frac{V_{p-j}}{j!}}{\mathbb{E}}_{Y^{\left(1\right)}}\left[{\left(Y^{\left(1\right)}-\mathbb{E}\left[Y\right]\right)}^j\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge Y^{\left(0\right)}]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)\right]\hfill \\ & \stackrel{\left(4\right)}{=}& \sum _{j=0}^p{\displaystyle \frac{V_{p-j}}{(j+1)!}}{\left(Y^{\left(0\right)}-\mathbb{E}\left[Y\right]\right)}^{j+1}-\sum _{j=0}^p{\displaystyle \frac{V_{p-j}}{(j+1)!}}\mathbb{E}\left[{(Y-\mathbb{E}\left[Y\right])}^{j+1}\right]\hfill \\ & =& \sum _{j^{\prime }=1}^{p+1}{\displaystyle \frac{V_{p+1-j^{\prime }}}{\left(j^{\prime }\right)!}}{\left(Y^{\left(0\right)}-\mathbb{E}\left[Y\right]\right)}^{j^{\prime }}+V_{p+1}=\sum _{j^{\prime }=0}^{p+1}{\displaystyle \frac{V_{p+1-j^{\prime }}}{\left(j^{\prime }\right)!}}{\left(Y^{\left(0\right)}-\mathbb{E}\left[Y\right]\right)}^{j^{\prime }},\hfill \end{array}$$

using the change of variables $j^{\prime }=j+1$ and the fact that $\mathbb{E}\left[Y-\mathbb{E}\left[Y\right]\right]=0$. □

Now, we have all the elements in hand to provide the Taylor-type expansions of functions based on CDFs of random variables and their moments (see Theorem 1). To that end, consider the sequences $U_k$ and $V_k$ given by (6) and (8), respectively, and define the coefficients

$${\alpha }_j:=\sum _{k=0}^{p-j}{U}_k\mathbb{E}\left[{\mathcal{D}}^{k+j}f\left(Y\right)\right];{\beta }_j:=\sum _{k=0}^{p-j}{V}_k\mathbb{E}\left[{\mathcal{D}}^{k+j}f\left(Y\right)\right];j=0,\dots ,p;$$

and

$$R_p:=\mathbb{E}\left[{\mathcal{D}}^{p}f\left(Y^{\left(p\right)}\right)\prod _{j=1}^p{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right].$$

Theorem 1.

Let $Y^{\left(0\right)}:=x\in \Omega$, $Y\sim F$ be a random variable, ${\left\{Y^{\left(j\right)}\right\}}_{j=1}^{p+1}$ be i.i.d. copies of Y, and assume (A1) holds. Then, the pth-order expansions of f at x are

$$f\left(x\right)=\sum _{j=0}^p{\alpha }_j{\displaystyle \frac{x^j}{j!}}+R_{p+1};$$

(9)

$$f\left(x\right)=\sum _{j=0}^p{\beta }_j{\displaystyle \frac{{\left(x-\mathbb{E}\left[Y\right]\right)}^j}{j!}}+R_{p+1}.$$

(10)

Proof. 

When taking $g=F$, Equation (2) becomes

$$\begin{array}{ccc}\hfill f\left(x\right)& =& \sum _{k=0}^p\mathbb{E}\left[{\mathcal{D}}^{k}f\left(Y\right)\right]\mathbb{E}\left[\prod _{j=1}^k{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]\hfill \\ & & +\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right],\hfill \end{array}$$

with the convention ${\prod }_{j=1}^{k=0}{\tau }_j:=1$. For the first result, using Lemma 2, that is,

$$\mathbb{E}\left[\prod _{j=1}^p{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]=\sum _{j=0}^p{\displaystyle \frac{U_{p-j}}{j!}}{x}^j,$$

we can write

$$\begin{array}{ccc}\hfill f\left(x\right)& =& \sum _{k=0}^p\mathbb{E}\left[{\mathcal{D}}^{k}f\left(Y\right)\right]\sum _{j=0}^k{U}_{k-j}{\displaystyle \frac{x^j}{j!}}+\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]\hfill \\ & =& \sum _{j=0}^p\sum _{k=j}^p\mathbb{E}\left[{\mathcal{D}}^{k}f\left(Y\right)\right]U_{k-j}{\displaystyle \frac{x^j}{j!}}+\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right].\hfill \end{array}$$

The second result holds using the same reasoning thanks to Lemma 3, that is,

$$\mathbb{E}\left[\prod _{j=1}^p{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]=\sum _{j=0}^p{\displaystyle \frac{V_{p-j}}{j!}}{\left(x-\mathbb{E}\left[Y\right]\right)}^j.$$

□

Theorem 1 provides exact and global expansions of smooth functions at any order $p\ge 1$, and it appears that different expansions are possible thanks to the choice of CDFs. For polynomials of degree p, the last terms in Equations (9) and (10) vanish, and exact expansions still hold using the first pth-order weak-derivatives of f. In general, the last terms are commonly known as remainder terms, and the smaller they are, the better the expansions are without such terms, that is,

$$f\left(x\right)\approx {\tilde{f}}_1\left(x\right):=\sum _{j=0}^p{\alpha }_j{\displaystyle \frac{x^j}{j!}};f\left(x\right)\approx {\tilde{f}}_2\left(x\right):=\sum _{j=0}^p{\beta }_j{\displaystyle \frac{{\left(x-\mathbb{E}\left[Y\right]\right)}^j}{j!}}.$$

Instead of using the above approximations, one may consider the exact expansions at order $p-1$, which include the remainder term when only the first pth-order weak derivatives are available. Indeed, the expectation $R_p$ can be computed in that situation, leading to the following estimator:

$$\widehat{f}\left(x\right)=\sum _{j=0}^{p-1}{\beta }_j{\displaystyle \frac{{\left(x-\mathbb{E}\left[Y\right]\right)}^j}{j!}}+{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathcal{D}}^{p}f\left(Y_i^{\left(p\right)}\right)\prod _{j=1}^p{\displaystyle \frac{F\left(Y_i^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y_i^{\left(j\right)}\ge Y_i^{(j-1)}]}}{\rho \left(Y_i^{\left(j\right)}\right)}},$$

where ${\left\{Y_i^{\left(j\right)}\right\}}_{i=1}^N$ is an i.i.d. sample of $Y^{\left(j\right)}$ for any $j\in \{1,\dots ,p\}$.

Additionally, to polynomials of degrees up to p, note that approximations ${\tilde{f}}_1,{\tilde{f}}_2$ of f become exact when

$$R_{p+1}=\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]=0,$$

for any function $f\in W^{p+1}$. The most important question is as follows: can we find or construct different distribution functions F such that $R_{p+1}=0$? Without an answer to that open problem, this study examines the error bounds so as to find a reasonable CDF (see Section 3).

2.2. Links with Taylor–Young and Maclaurin Series

Taylor series are well known and used by different communities for locally approximating functions. In this section, we are going to show in Corollary 2 that such series are particular cases of the expansions of functions derived in Theorem 1. To that end, for any $x_0\in \Omega$, denote with ${\left\{U_m\sim \mathcal{U}\left(x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right)\right\}}_{m\in \mathbb{N}\setminus \left\{0\right\}}$ a sequence of uniformly distributed random variables, having $F_m$ as the CDF. Applying Equation (10) using $F_m$ yields

$$\begin{array}{ccc}\hfill f_m\left(x\right)& :=& \sum _{j=0}^p\sum _{k=0}^{p-j}\mathbb{E}\left[{\mathcal{D}}^{k+j}f\left(U_m\right)\right]V_{k,m}{\displaystyle \frac{{\left(x-x_0\right)}^j}{j!}}\hfill \\ & & +\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(U_m^{(p+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F_m\left(U_m^{\left(j\right)}\right)-{1\mathrm{I}}_{\left[U_m^{\left(j\right)}\ge U_m^{(j-1)}\right]}}{{\rho }_m\left(U_m^{\left(j\right)}\right)}}\right],\hfill \end{array}$$

which is a sequence of functions that expand f as well. But, it is worth noting that $f\left(x\right)=f_m\left(x\right)$ for only $x\in \left[x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right]$ because $F_m$ is a zero-one-ended function on that support.

Corollary 2.

Let $x\in \Omega$, and assume (A1) holds. Then, we have

$$\underset{m\to \infty }{lim}{f}_m\left(x\right)=\sum _{j=0}^p{\mathcal{D}}^{j}f\left(x_0\right){\displaystyle \frac{{\left(x-x_0\right)}^j}{j!}}.$$

Proof. 

The sequence of random variables given by $U_m\sim \mathcal{U}\left(x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right)$ has ${\rho }_m\left(x\right):=m{1\mathrm{I}}_{\left[x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right]}\left(x\right)$ as density for all $m>0$. The density ${\rho }_m$ is also known as the rectangular function, and it converges in distribution toward the Dirac delta measure, that is, ${\rho }_m\left(x\right)\to {\delta }_{x_0}\left(x\right):=\delta (x-x_0)$. The CDF of $U_m$ is given by $F_m\left(x\right):=m\left(x-x_0+{\displaystyle \frac{1}{2m}}\right)$ and $\mathbb{E}\left[U_m\right]=x_0$. Therefore, the sequence given by Equation (8) becomes

$${\displaystyle V_{0,m}:=1,V_{1,m}:=0,V_{k+1,m}=-\sum _{i=2}^{k+1}{V}_{k+1-i,m}\frac{\mathbb{E}\left[{\left(U_m-x_0\right)}^i\right]}{i!},\forall k\ge 1,}$$

and a direct calculus gives ${lim}_{m\to \infty }{V}_{k+1,m}=0$ because for all $i>0$

$$\underset{m\to \infty }{lim}\mathbb{E}\left[{\left(U_m-x_0\right)}^i\right]=\underset{m\to \infty }{lim}{\int }_{\mathbb{R}}{(x-x_0)}^i{\rho }_m\left(x\right)dx={\int }_{\mathbb{R}}{\delta }_{x_0}\left(x\right){(x-x_0)}^{i}dx=0.$$

Moreover, giving the identity

$$\underset{m\to \infty }{lim}\mathbb{E}\left[{\mathcal{D}}^{j}f\left(U_m\right)\right]=\underset{m\to \infty }{lim}{\int }_{\mathbb{R}}{\mathcal{D}}^{j}f\left(x\right){\rho }_m\left(x\right)dx={\int }_{\mathbb{R}}{\mathcal{D}}^{j}f\left(x\right){\delta }_{x_0}\left(x\right)dx={\mathcal{D}}^{j}f\left(x_0\right),$$

one can write the following limit of $f_m$:

$$\begin{array}{ccc}\hfill \underset{m\to \infty }{lim}{f}_m\left(x\right)& =& \sum _{j=0}^p\underset{m\to \infty }{lim}\mathbb{E}\left[{\mathcal{D}}^{j}f\left(U_m\right)\right]{\displaystyle \frac{{\left(x-x_0\right)}^j}{j!}}\hfill \\ & & +\underset{m\to \infty }{lim}\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(U_m^{(p+1)}\right)\prod _{j=1}^{p+1}\left(U_m^{\left(j\right)}-x_0+{\displaystyle \frac{1}{2m}}-{\displaystyle \frac{{1\mathrm{I}}_{[U_m^{\left(j\right)}\ge U_m^{(j-1)}]}}{m}}\right)\right].\hfill \\ & =& \sum _{j=0}^p{\mathcal{D}}^{j}f\left(x_0\right){\displaystyle \frac{{\left(x-x_0\right)}^j}{j!}},\hfill \end{array}$$

because the last limit vanishes, keeping in mind the dominated convergence theorem. Indeed, using $|a+b|\le \left|a\right|+\left|b\right|$ for all real $a,b$ and ${1\mathrm{I}}_A\le {1\mathrm{I}}_B$ for all sets $A\subseteq B$, we have

$$m\left|z^{\left(j\right)}-x_0+{\displaystyle \frac{1}{2m}}-{\displaystyle \frac{{1\mathrm{I}}_{[z^{\left(j\right)}\ge z^{(j-1)}]}}{m}}\right|{1\mathrm{I}}_{\left[x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right]}\left(z^{\left(j\right)}\right)\le 2{1\mathrm{I}}_{\left[x_0-{\displaystyle \frac{1}{2}},x_0+{\displaystyle \frac{1}{2}}\right]}\left(z^{\left(j\right)}\right),$$

and

$$\begin{array}{ccc}& & \left|{\mathcal{D}}^{p+1}f\left(z^{(p+1)}\right)\prod _{j=1}^{p+1}\left(z^{\left(j\right)}-x_0+{\displaystyle \frac{1}{2m}}-{\displaystyle \frac{{1\mathrm{I}}_{[z^{\left(j\right)}\ge z^{(j-1)}]}}{m}}\right)\right|m{1\mathrm{I}}_{\left[x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right]}\left(z^{\left(j\right)}\right)\hfill \\ & \le & \left|{\mathcal{D}}^{p+1}f\left(z^{(p+1)}\right)\right|\prod _{j=1}^{p+1}2{1\mathrm{I}}_{\left[x_0-{\displaystyle \frac{1}{2}},x_0+{\displaystyle \frac{1}{2}}\right]}\left(z^{\left(j\right)}\right).\hfill \end{array}$$

Since the last term is integrable, the dominated convergence theorem allows us to write

$$\begin{array}{ccc}& & \underset{m\to \infty }{lim}\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(z^{(p+1)}\right)\prod _{j=1}^{p+1}\left(U_m^{\left(j\right)}-x_0+{\displaystyle \frac{1}{2m}}-{\displaystyle \frac{{1\mathrm{I}}_{[U_m^{\left(j\right)}\ge U_m^{(j-1)}]}}{m}}\right)\right]\hfill \\ & =& \int \underset{m\to \infty }{lim}{\mathcal{D}}^{p+1}f\left(z^{(p+1)}\right)\prod _{j=1}^{p+1}\left(z^{\left(j\right)}-x_0+{\displaystyle \frac{1}{2m}}-{\displaystyle \frac{{1\mathrm{I}}_{[z^{\left(j\right)}\ge z^{(j-1)}]}}{m}}\right)m{1\mathrm{I}}_{\left[x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right]}\left(z^{\left(j\right)}\right)d{z}^{\left(j\right)}\hfill \\ & =& \int {\mathcal{D}}^{p+1}f\left(z^{(p+1)}\right)\prod _{j=1}^{p+1}\left(z^{\left(j\right)}-x_0\right){\delta }_{x_0}\left(z^{\left(j\right)}\right)d{z}^{\left(j\right)}=0,\hfill \end{array}$$

and the result holds. □

Obviously, Corollary 2 provides the exact expansion of $f\left(x_0\right)$, and good approximations of $f\left(x\right)$ for any $x\in \left[x_0-{\displaystyle \frac{1}{2m}},x_0+{\displaystyle \frac{1}{2m}}\right]$ with $m>1$. Extending such approximations for any $x\in \Omega$ might not give accurate results, except for some particular cases. Indeed, it is common to consider expanding f within different subsets ${\Omega }_i$s of $\Omega$ that form a partition of $\Omega$ when derivative data sets are available within each ${\Omega }_i$.

3. Error Bounds and Choice of Distribution Functions

This section treats the choice of the distribution functions of random variables depending on selected quality-measures of functions expansions, such as integrated or mean errors, mean squared errors (MSEs) and integrated MSEs (IMSEs).

In this section, the input values of a function follow a prescribed distribution function, that is, $X\sim F_0$. Without additional information on X, it is common to use uniform distributions or the Gaussian distribution with a higher variance. In the other case, $F_0$ can be estimated using the well-established methods for density and/or distribution estimations (see e.g., [30,31,32,33]).

Recall that the main issue consists of expecting to find the distribution functions (i.e., F) such that

$$R_{p+1}=\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]=0.$$

In the case where $Y\sim {\delta }_a$ with ${\delta }_a$ being the Dirac measure, the distribution function is given by $F\left(y\right)=1{1\mathrm{I}}_{[y\ge a]}$ and $Y^{\left(j\right)}=a$ almost surely. Therefore, $R_{p+1}=0$ is achieved, and the exact Taylor expansion at that point is obtained. But, the corresponding approximation is only accurate around the node a, as discussed in Section 2.2.

3.1. Zero-Mean Errors

Concerning Equation (10), $R_{p+1}$ can be seen as an error term, and it is common to require the zero-mean error over the input space (i.e., integrated mean) and the smallest integrated variance. To achieve the zero-mean error, Theorem 2 gives the distribution function F of independent variables as well as the corresponding IMSE.

Theorem 2.

Consider the remainder term $R_{p+1}$ of Equations (9) and (10) and assume that (A1) holds. If $F=F_0$, then

$$\mathbb{E}\left[R_{p+1}\left(X\right)\right]=0,$$

$$\mathbb{E}\left[R_{p+1}{\left(X\right)}^2\right]\le {\displaystyle \frac{1}{3}}{\left[\underset{y\in \Omega }{sup}{\displaystyle \frac{F_0\left(y\right)\left[1-F_0\left(y\right)\right]}{{\rho }_0^2\left(y\right)}}\right]}^p\mathbb{E}\left[{\left\{{\displaystyle \frac{{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)}{{\rho }_0\left(Y^{(p+1)}\right)}}\right\}}^2\right].$$

Moreover, if ${\rho }_0\ge ϵ>0$, then

$$\mathbb{E}\left[R_{p+1}{\left(X\right)}^2\right]\le {\displaystyle \frac{1}{3{ϵ}^{2p}}}\mathbb{E}\left[{\left\{{\displaystyle \frac{{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)}{{\rho }_0\left(Y^{(p+1)}\right)}}\right\}}^2\right].$$

Proof. 

As $R_{p+1}\left(X\right)=\mathbb{E}\left[{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right){\prod }_{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right]$ with $Y^{\left(0\right)}=X$, the first result holds using the Fubini theorem [27] and the fact that ${\mathbb{E}}_X\left[{1\mathrm{I}}_{[Y^{\left(1\right)}\ge X]}\right]=F_0\left(Y^{\left(1\right)}\right)$, leading to $F\left(Y^{\left(1\right)}\right)-{\mathbb{E}}_X\left[{1\mathrm{I}}_{[Y^{\left(1\right)}\ge X]}\right]=0$.

Since $F=F_0$ and $\mathbb{E}\left[R_{p+1}\left(X\right)\right]=0$, the integrated MSE is given by

$$\begin{array}{ccc}\hfill {\mathbb{E}}_X\left[R_{p+1}^2\left(X\right)\right]& \le & {\mathbb{E}}_X\left[{\mathbb{E}}^2\left\{\left|{\mathcal{D}}^{p+1}f\left(Y^{(n+1)}\right)\prod _{j=1}^{p+1}{\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right\}\right]\right]\hfill \\ & \le & \mathbb{E}\left[{\left\{{\displaystyle \frac{{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)}{\rho \left(Y^{(p+1)}\right)}}\right\}}^2\right]\mathbb{E}\left[{\rho }^2\left(Y^{(p+1)}\right)\prod _{j=1}^{p+1}{\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2\right]\hfill \\ & \le & {\displaystyle \frac{1}{3}}{\left[\underset{y\in \Omega }{sup}{\displaystyle \frac{F_0\left(y\right)\left[1-F_0\left(y\right)\right]}{{\rho }_0^2\left(y\right)}}\right]}^p\mathbb{E}\left[{\left\{{\displaystyle \frac{{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)}{\rho \left(Y^{(p+1)}\right)}}\right\}}^2\right].\hfill \end{array}$$

Indeed, for the term $\mathbb{E}\left[{\rho }^2\left(Y^{(p+1)}\right){\prod }_{j=1}^{p+1}{\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2\right]$, it is obvious that

$${\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2={\displaystyle \frac{F^2\left(Y^{\left(j\right)}\right)-2F\left(Y^{\left(j\right)}\right){1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}+{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{{\rho }^2\left(Y^{\left(j\right)}\right)}},$$

and when $j=1$ and $j=p+1$, one can see that

$${\mathbb{E}}_X\left[{\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge X]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)}^2\right]={\displaystyle \frac{F^2\left(Y^{\left(1\right)}\right)-2F\left(Y^{\left(1\right)}\right)F_0\left(Y^{\left(1\right)}\right)+F_0\left(Y^{\left(1\right)}\right)}{{\rho }^2\left(Y^{\left(1\right)}\right)}},$$

and

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{Y^{(p+1)}}\left[{\rho }^2\left(Y^{(p+1)}\right){\left({\displaystyle \frac{F\left(Y^{(p+1)}\right)-{1\mathrm{I}}_{[Y^{(p+1)}\ge Y^{\left(p\right)}]}}{\rho \left(Y^{(p+1)}\right)}}\right)}^2\right]& =& {\displaystyle \frac{1}{3}}-\left\{1-F^2\left(Y^{\left(p\right)}\right)\right\}+1-F\left(Y^{\left(p\right)}\right)\hfill \\ & =& {\displaystyle \frac{1}{3}}+F^2\left(Y^{\left(p\right)}\right)-F\left(Y^{\left(p\right)}\right)\le {\displaystyle \frac{1}{3}}.\hfill \end{array}$$

If $F=F_0$

$${\mathbb{E}}_X\left[{\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge X]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)}^2\right]={\displaystyle \frac{F_0\left(Y^{\left(1\right)}\right)\left[1-F_0\left(Y^{\left(1\right)}\right)\right]}{{\rho }_0^2\left(Y^{\left(1\right)}\right)}},$$

and the second result holds by iterating the above procedure corresponding to the expectation

$${\mathbb{E}}_{Y^{(j-1)})}\left[{\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)})}]}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2\right]={\displaystyle \frac{F\left(Y^{\left(j\right)}\right)\left[1-F\left(Y^{\left(j\right)}\right)\right]}{{\rho }^2\left(Y^{\left(j\right)}\right)}},$$

for any $j\in \{2,\dots ,p\}$.

The last result holds because

$${\mathbb{E}}_X\left[{\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge X]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)}^2\right]={\displaystyle \frac{F_0\left(Y^{\left(1\right)}\right)\left[1-F_0\left(Y^{\left(1\right)}\right)\right]}{{\rho }_0^2\left(Y^{\left(1\right)}\right)}}\le {\displaystyle \frac{1}{{\rho }_0^2\left(Y^{\left(1\right)}\right)}}\le {\displaystyle \frac{1}{{ϵ}^2}}.$$

□

3.2. Integrated Mean Squared Errors

This section aims at providing distribution functions that can contribute to minimizing IMSEs, leading to possible optimal distribution functions for the approximation of f. To that end, define

$$K_p:=\mathbb{E}\left[{\left\{{\displaystyle \frac{{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)}{\rho \left(Y^{(p+1)}\right)}}\right\}}^2\right].$$

Theorem 3.

Consider the remainder term $R_{p+1}$ of Equations (9) and (10) and assume that (A1) holds. Then,

$$\mathbb{E}\left[R_{p+1}{\left(X\right)}^2\right]\le {\displaystyle \frac{K_p}{3}}\underset{y\in \Omega }{sup}{\displaystyle \frac{F^2\left(y\right)-2F\left(y\right)F_0\left(y\right)+F_0\left(y\right)}{{\rho }^2\left(y\right)}}{\left[\underset{y\in \Omega }{sup}{\displaystyle \frac{F\left(y\right)\left[1-F\left(y\right)\right]}{{\rho }^2\left(y\right)}}\right]}^{p-1}.$$

Proof. 

It comes out from the proof of Theorem 2 that the integrated MSE is given by

$${\mathbb{E}}_X\left[R_{p+1}^2\left(X\right)\right]\le {\displaystyle \frac{1}{3}}\mathbb{E}\left[\prod _{j=1}^p{\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2\right]\mathbb{E}\left[{\left\{{\displaystyle \frac{{\mathcal{D}}^{p+1}f\left(Y^{(p+1)}\right)}{\rho \left(Y^{(p+1)}\right)}}\right\}}^2\right].$$

Since

$${\mathbb{E}}_X\left[{\left({\displaystyle \frac{F\left(Y^{\left(1\right)}\right)-{1\mathrm{I}}_{[Y^{\left(1\right)}\ge X]}}{\rho \left(Y^{\left(1\right)}\right)}}\right)}^2\right]={\displaystyle \frac{F^2\left(Y^{\left(1\right)}\right)-2F\left(Y^{\left(1\right)}\right)F_0\left(Y^{\left(1\right)}\right)+F_0\left(Y^{\left(1\right)}\right)}{{\rho }^2\left(Y^{\left(1\right)}\right)}},$$

and

$${\mathbb{E}}_{Y^{(j-1)})}\left[{\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)})}]}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2\right]={\displaystyle \frac{F\left(Y^{\left(j\right)}\right)\left[1-F\left(Y^{\left(j\right)}\right)\right]}{{\rho }^2\left(Y^{\left(j\right)}\right)}},$$

for any $j\in \{2,\dots ,p\}$, one can write

$$\begin{array}{ccc}\hfill A_1& :=& \mathbb{E}\left[\prod _{j=1}^p{\left({\displaystyle \frac{F\left(Y^{\left(j\right)}\right)-{1\mathrm{I}}_{[Y^{\left(j\right)}\ge Y^{(j-1)}]}}{\rho \left(Y^{\left(j\right)}\right)}}\right)}^2\right]\hfill \\ & \le & \underset{y\in \Omega }{sup}{\displaystyle \frac{F^2\left(y\right)-2F\left(y\right)F_0\left(y\right)+F_0\left(y\right)}{{\rho }^2\left(y\right)}}{\left[\underset{y\in \Omega }{sup}{\displaystyle \frac{F\left(y\right)\left[1-F\left(y\right)\right]}{{\rho }^2\left(y\right)}}\right]}^{p-1},\hfill \end{array}$$

and the result holds. □

In view of Theorem 3, the ideal, optimal distribution function $F_{op}$ must satisfy the following functional equation:

$$F_{op}^2\left(y\right)-2F_{op}\left(y\right)F_0\left(y\right)+F_0\left(y\right)=0.$$

Unfortunately, there is no solution in $\mathbb{R}$, suggesting working in $\mathbb{C}$ so as to expect finding $F_{op}$. Nevertheless, such a functional equation is interesting as it could lead to finding a zero-one-ended function, which is not necessarily a distribution function (see Lemma 1).

For probabilistic expansions in $\mathbb{R}$ considered in this paper, finding approximated optimal CDF $F_{op}^{*}$ requires using the upper-bound of the IMSE provided in Theorem 3. To derive $F_{op}^{*}$, denote with ${\mathcal{F}}_{\Omega }^c$ the set of continuous CDFs on the support $\Omega$.

Proposition 3.

Under the conditions of Theorem 3, assume that $p\gg 1$. Then, an optimal CDF is given by

$$F_{op}^{*}:=arg\underset{F\in {\mathcal{F}}_{\Omega }^c}{min}\left(\underset{y\in \Omega }{sup}{\displaystyle \frac{F\left(y\right)\left[1-F\left(y\right)\right]}{{\rho }^2\left(y\right)}}\right).$$

Such a result is straightforward. Of course, parametric approaches may be considered as well. Such approaches consist of (i) choosing particular classes of CDFs F belongs to, and (ii) determining the optimal parameters of the corresponding CDFs, that is, the values of the parameters that minimize the upper-bound of the IMSE. Instances of distributions are Gaussian, arcsine, and beta distributions.

To obtain an optimal CDF that depends on $F_0$ requires relying on the CDF of the form $F=F_0^{\tau }$ with $\tau >0$, which leads to the following result.

Proposition 4.

Under the conditions of Theorem 3, assume that $F=F_0^{\tau }$ with $\tau >0$. Then, an optimal CDF is given by $F_{op}^{**}:=F_0^{{\tau }^{*}}$ with

$${\tau }^{*}:=arg\underset{\tau \in {\mathbb{R}}_{+}\setminus \left\{0\right\}}{min}\left\{\underset{y\in \Omega }{sup}{\displaystyle \frac{F_0^2\left(y\right)-2F_0^{3-\tau }\left(y\right)+F_0^{3-2\tau }\left(y\right)}{{\tau }^2{\rho }_0^2\left(y\right)}}{\left(\underset{y\in \Omega }{sup}{\displaystyle \frac{F_0^{2-\tau }\left(y\right)\left[1-F_0^{\tau }\left(y\right)\right]}{{\tau }^2{\rho }_0^2\left(y\right)}}\right)}^{p-1}\right\}.$$

Proof. 

One can see that $F=F_0^{\tau }$ is a CDF and the corresponding density is $\rho =\tau F_0^{\tau -1}{\rho }_0$. □

4. Applications

To assess the qualities of different methods, including the Taylor approach, the following error measure is considered:

$$Err\left(x_{\ell }\right):={\displaystyle \frac{\left|\tilde{f}\left(x_{\ell }\right)-f\left(x_{\ell }\right)\right|}{\left|f\left(x_{\ell }\right)\right|}};\ell =1,\dots ,L,$$

where $f\left(x_{\ell }\right)$ is the exact evaluation of f at $x_{\ell }\in \Omega$ and $\tilde{f}\left(x_{\ell }\right)$ is the corresponding approximation. The values $x_{\ell },\ell =1,\dots ,L=100$ are selected so that they are equally spaced in the bounded domain $\Omega$. Three types of approximations are computed: the traditional Taylor approximation, the proposed approach with and without the remainder term. All of these approximations make use of the same order p of derivatives.

For each function, the R-package pracma is used for performing the Taylor expansion about (i) the middle of the domain $\Omega$, that is, ${\displaystyle \frac{a+b}{2}}$ with $\Omega =(a,b)$, or (ii) the expectation of X, that is, $\mathbb{E}\left[X\right]$ for for infinite domains. The same R-package is used for obtaining derivatives of functions of orders up to $p=3$. Concerning the proposed approach, different CDFs are considered, such as the truncated Gaussian distribution on $\Omega$ with mean ${\displaystyle \frac{a+b}{2}}$ and standard deviation ${10}^{-6}$ (i.e., practically, the Dirac measure); the uniform distribution on $\Omega$; a mixture of Gaussian distributions; and the optimal beta distribution (when possible) according to Propositions 3 and 4. The coefficients ${\beta }_j$s of the proposed probabilistic expansion (see Theorem 1) are computed using the Monte Carlo approach with the sample size $N=1000$, that is,

$${\beta }_j\approx {\displaystyle \frac{1}{N}}\sum _{k=0}^{p-j}\sum _{i=1}^N{V}_k{\mathcal{D}}^{k+j}f\left(Y_i\right).$$

Table 1. Estimated values of ${\beta }_j$s and the nodes for expanding $sin\left(x\right)$ using the proposed approach associated with different CDFs. For the traditional Taylor approach, the coefficients are $f\left(3.14\right),\mathcal{D}f\left(3.14\right),$ and ${\mathcal{D}}^{2}f\left(3.14\right)$.

	Truncated Gaussian CDF	Uniform CDF	Mixture CDF	Taylor
${\beta }_0$	0.0015	−0.068	−0.008	0.0015
${\beta }_1$	−0.999	−0.007	−0.352	−0.999
${\beta }_2$	−0.002	0.026	0.004	−0.001
Node	3.14	3.14	3.223	3.14

and

Table 2. Estimated values of ${\beta }_j$s and the nodes for expanding $cos\left(x\right)$ using the proposed approach associated with different CDFs. For the traditional Taylor approach, the coefficients are $f\left(3.14\right),\mathcal{D}f\left(3.14\right),$ and ${\mathcal{D}}^{2}f\left(3.14\right)$.

	Truncated Gaussian CDF	Uniform CDF	Mixture CDF	Taylor
${\beta }_0$	−0.999	−0.139	−0.667	−0.999
${\beta }_1$	−0.002	−0.015	−0.019	−0.0015
${\beta }_2$	0.999	0.053	0.356	0.999
Node	3.14	3.14	3.125	3.14

report the estimated values of the coefficients ${\beta }_j$s and the nodes $\mathbb{E}\left[Y\right]$ for expanding $sin\left(x\right)$ and $cos\left(x\right)$, respectively. Such values are obtained using only one simulation and without the remainder terms.

It turns out that the coefficients used for the proposed expansions of functions differ from those of the traditional Taylor approach, except for the truncated Gaussian distribution, as expected.

The following figures show the simulated $L=100$ error values (in log 10) for different functions. Each panel of such figures corresponds to a specific CDF, and it shows the error values associated with Taylor’s approximation compared to the proposed approach, that is, the probabilistic Taylor-type without the remainder term (i.e., PTWoR) and/or with the remainder term (i.e., PTWR).

Figure 1 and Figure 2 depict the error values for the functions $sin\left(x\right)$ and $cos\left(x\right)$ when $p=2$, respectively. Similar results are obtained when $p=1$ and 3. It appears that the practical Dirac measure almost exactly provides the well-known Taylor approximations, and the other CDFs (PTWoR) perform well when the input values are far way from the node used.

Figure 3 and Figure 4 depict the error values for the functions $exp\left(x\right)$ and $log(x+1)$ when $p=1$, respectively. Again, the practical Dirac measure exactly provides the well-known Taylor approximations. The proposed PTWR approach performs well compared to PTWoR for the uniform distribution and the beta distribution.

Figure 5 and Figure 6 show the error values for the absolute function $\left|x\right|$ when $p=2$ and 3, respectively. The practical Dirac measure helps to reproduce the Taylor approximations. The PTWoR performs well when $p=3$, while the PTWR associated with the Dirac measure gives the best results compared to PTWoR when $p=2$. It is to be noted that Taylor series fail for this function.

5. Conclusions

A generic expansion of every smooth function has been proposed in this paper using any zero-one-ended function. Special zero-one-ended functions are CDFs, which lead to the unified stochastic framework for Taylor-type expansions of functions, including Taylor series. The proposed probabilistic expansions are a global approach by allowing one to incorporate derivative information at several points. Since different CDFs are possible, it is shown in this paper that the Dirac measure and its CDF is the best CDF for the proposed probabilistic expansions, leading to Taylor series. But, such series are exact at the node used and give accurate results for input values that fall within a neighborhood of that node. A functional equation is proposed in order to derive the best CDF over the whole domain. Without any solution in $\mathbb{R}$, error bounds have been used for selecting the quasi optimal CDF.

The practical Dirac probability measure, such as truncated Gaussian distributions with the smallest variance, is used to recover the well-known Taylor expansion. It happens that some CDFs allow for improving the Taylor approximations. Such results are promising and require more investigations. For instance, different CDFs can be involved in the proposed expansion instead of only one CDF. Significant advantages of the proposed probabilistic expansion over Taylor series should be expected thanks to the best CDFs.