Derivative Formulas and Gradient of Functions with Non-Independent Variables

Abstract

Stochastic characterizations of functions subject to constraints result in treating them as functions with non-independent variables. By using the distribution function or copula of the input variables that comply with such constraints, we derive two types of partial derivatives of functions with non-independent variables (i.e., actual and dependent derivatives) and argue in favor of the latter. Dependent partial derivatives of functions with non-independent variables rely on the dependent Jacobian matrix of non-independent variables, which is also used to define a tensor metric. The differential geometric framework allows us to derive the gradient, Hessian, and Taylor-type expansions of functions with non-independent variables.

1. Introduction

We used to work with models defined through functions that include non-independent variables such as correlated input variables. This is also the case for models defined via functions with independent input variables and equations or inequations connecting such inputs; functions subject to constraints involving input variables and/or the model output. Knowing the key role of partial derivatives at a given point in (i) the mathematical analysis of functions and convergence, (ii) Poincaré inequalities ([1,2]) and equalities ([3,4]), (iii) optimization and active subspaces ([5,6]), (iv) implicit functions ([7,8]) and (v) differential geometry (see, e.g., [9,10,11]), it is interesting and relevant to have formulas that enable the calculations of partial derivatives of functions in the presence of non-independent variables, including the gradients. Of course, such formulas must account for the dependency structures among the model inputs, including the constraints imposed on such inputs.

Actual partial derivatives aim to calculate the partial derivatives of functions, taking into account the relationships between the input variables ([12]). For instance, let us consider the function $f:{\mathbb{R}}^3\to \mathbb{R}$ given by $f(x,y,z)=2x{y}^2{z}^3$ under the constrained equation $h(x,y,z)=0$, where h is any smooth function. Using $f_x^{\prime }$ to represent the formal partial derivative of f with respect to. x, which is the partial derivative of f when considering other inputs as constant or independent, the chain rule yields the following partial derivative:

$$\begin{array}{ccc}\hfill \frac{\partial f}{\partial x}(x,y,z)& =& f_x^{\prime }(x,y,z)+f_y^{\prime }(x,y,z)\frac{\partial y}{\partial x}+f_z^{\prime }(x,y,z)\frac{\partial z}{\partial x}\hfill \\ & =& 2y^2{z}^3+4xy{z}^3\frac{\partial y}{\partial x}+6x{y}^2{z}^2\frac{\partial z}{\partial x}.\hfill \end{array}$$

In probability and statistics, the point $(x,y,z)$ represents a realization or a sample value of the implicit random vector $(X,Y,Z)$. The quantities $\frac{\partial y}{\partial x},\frac{\partial z}{\partial x}$, and $\frac{\partial f}{\partial x}$ do not have a definite meaning when the variables $(X,Y,Z)$ are npn-independent. At this point, determining $\frac{\partial y}{\partial x},\frac{\partial z}{\partial x}$ and $\frac{\partial f}{\partial x}$ is often challenging without supplementary assumptions such as the choice of directions or paths. When $(X,Z)$ or $(X,Y)$ are independent and using the equation $h(x,y,z)=x+2y-z=0$, we can write ([12]):

$${\displaystyle \frac{\partial f}{\partial x}(x,y,z)=\left\{\begin{array}{cc}2y^2{z}^3-2xy{z}^3\hfill & \mathrm{i}\mathrm{f} X \mathrm{a}\mathrm{n}\mathrm{d} Z \mathrm{a}\mathrm{r}\mathrm{e} \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t} \mathrm{o}\mathrm{r} z \mathrm{i}\mathrm{s} \mathrm{b}\mathrm{e}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{h}\mathrm{e}\mathrm{l}\mathrm{d} \mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}\hfill \\ 2y^2{z}^3+6x{y}^2{z}^2\hfill & \mathrm{i}\mathrm{f} X \mathrm{a}\mathrm{n}\mathrm{d} Y \mathrm{a}\mathrm{r}\mathrm{e} \mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t} \mathrm{o}\mathrm{r} y \mathrm{i}\mathrm{s} \mathrm{b}\mathrm{e}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{h}\mathrm{e}\mathrm{l}\mathrm{d} \mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}\hfill \end{array}\right..}$$

It is clear that the actual partial derivatives are not unique, as such derivatives rely on two different paths or assumptions. While each supplementary assumption can make sense in some cases, it cannot always be guaranteed by the constrained equation $h(x,y,z)=0$ in general. Indeed, when all the initial input variables are dependent or correlated, the above partial derivatives are no longer valid, even for a linear function h, and it is worth finding the correct relationship between the input variables and the partial derivatives, including the gradient.

In differential geometry (see, e.g., [9,10,11]), using the differential of the function f, that is,

$$df=2y^2{z}^{3}dx+4xy{z}^{3}dy+6x{y}^2{z}^{2}dz,$$

the gradient of f is defined as the dual of $df$ with respect to a given tensor metric. Obviously, different tensor metrics will yield different gradients of the same function. While the Euclidean metric, given by $\gamma :=d{x}^2+d{y}^2+d{z}^2$, is more appropriate for independent inputs, finding the appropriate metrics is challenging in general. Indeed, the first fundamental form (in differential geometry) requires the Jacobian matrix of the inputs to define the associated tensor metric.

For non-independent variables with F as the joint cumulative distribution function (CDF), the bivariate dependency models ([13]) and multivariate dependency models ([3,14,15]), including the conditional and inverse Rosenblatt transformation ([16,17]), establish formal and analytical relationships among such variables using either CDFs or the corresponding copulas or new distributions that look like and behave like a copula ([18]). A dependency function characterizes the probabilistic dependency structures among these variables. For a d-dimensional random vector of non-independent variables, the dependency models express a subset of $d-1$ variables as a function of independent variables, consisting of the remaining input and new independent variables.

In this paper, we propose a new approach for calculating the partial derivatives of functions, considering the dependency structures among the input variables. Our approach relies on dependency models. By providing known relationships between the dependent inputs (including constraints imposed on inputs or outputs), dependency models can be regarded as global and probabilistic implicit functions (see Section 2.2). Such dependency models are used to determine the dependent Jacobian and the tensor metric for non-independent variables. The contributions of this paper are threefold:

• To provide a generalization of the actual partial derivatives of functions with non-independent variables and establish its limits;
• To introduce the general derivative formulas of functions with non-independent variables (known as dependent partial derivatives) without any additional assumptions;
• To provide the gradient, Hessian, and Taylor-type expansions of functions with non-independent variables that comply with the dependency structures among the input variables.

In Section 2, we first review the dependency models of dependent input variables, including correlated variables. Next, we derive interesting properties of these models regarding the calculus of partial derivatives and probabilistic implicit functions. By coupling dependency functions with the function of interest, we extend the actual partial derivatives of functions with non-independent variables in Section 3. To avoid their drawbacks, the dependent partial derivatives of functions with non-independent variables are provided in Section 4. The gradient and Hessian matrix of these functions are derived in Section 5 using the framework of differential geometry. We provide an application in Section 6 and conclude this work in Section 7.

General Notations

For an integer $d>0$, let $\mathbf{X}:=(X_1,\dots ,X_d)$ be a random vector of continuous variables with F as the joint cumulative distribution function (CDF) (i.e., $\mathbf{X}\sim F$). For any $j\in \{1,\dots ,d\}$, we use $F_{x_j}$ or $F_j$ for the marginal CDF of $X_j$ and $F_j^{-1}$ for its inverse. Also, we use $(\sim j):=(w_1,\dots ,w_{d-1})$ for an arbitrary permutation of $\{1,\dots ,d\}\setminus \left\{j\right\}$ and ${\mathbf{X}}_{\sim j}:=(X_{w_1},\dots ,X_{w_{d-1}})$.

For a function f that includes $\mathbf{X}$ as inputs, we use $f_{x_j}^{\prime }$ for the formal partial derivative of f with respect to $X_j$, considering other inputs as constant or independent of $X_j$, and $\nabla f:=f_{\mathbf{x}}^{\prime }:={\left[f_{x_1}^{\prime },\dots ,f_{x_d}^{\prime }\right]}^T$. We use $\frac{\partial f}{\partial x_j}$ for the partial derivative of f with respect to $X_j$, which takes the dependencies among inputs into account. We also use $\frac{\partial f}{\partial \mathbf{x}}:={\left[\frac{\partial f}{\partial x_1},\dots ,\frac{\partial f}{\partial x_d}\right]}^T\in {\mathbb{R}}^d$. Of course, $\frac{\partial f}{\partial \mathbf{x}}=\nabla f$ for independent inputs.

2. Probabilistic Characterization of Functions with Non-Independent Variables

In probability theory, it is common to treat input variables as random vectors with associated CDFs. For instance, for the inputs that take their values within a known domain or space, the Bayesian framework allows for assigning a joint distribution, known as a prior distribution, to such inputs. Without additional information about the inputs, it is common to use non-informative prior distributions, such as uniform distributions or Gaussian distributions associated with higher values of the variances (see e.g., [19]).

Functions with non-independent variables include many types of models encountered in practice. An example is the models defined via a given function and equations or inequations connecting its inputs. The resulting inputs that comply with such constraints are often dependent or correlated. In the subsequent discussion, we use probability theory to characterize non-independent variables (see Definition 1).

Definition 1.

Consider $n\in \mathbb{N}\setminus \left\{0\right\}$ and a function $f:{\mathbb{R}}^d\to {\mathbb{R}}^n$ that includes $\mathbf{X}\sim F$ as inputs.

Then, f is said to be a function with non-independent variables whenever there exists at least one pair $j_1,j_2\in \{1,\dots ,d\}$ with $j_1\ne j_2$, such that

$$F_{j_1,j_2}(x_{j_1},x_{j_2})\ne F_{j_1}\left(x_{j_1}\right)F_{j_2}\left(x_{j_2}\right).$$

Using $\mathcal{N}\left(\mathbf{0},\Sigma \right)$ for the multivariate normal distribution, we can check that a function that includes $\mathbf{X}\sim \mathcal{N}\left(\mathbf{0},\Sigma \right)$ as inputs, with $\Sigma$ being a non-diagonal covariance matrix, is a member of the class of functions defined by

$${\mathcal{D}}_{d,n}=\left\{f:{\mathbb{R}}^d\to {\mathbb{R}}^n:\mathbf{X}\sim F,F\left(\mathbf{x}\right)\ne \prod _{j=1}^d{F}_j\left(x_j\right);\mathbf{x}\in {\mathbb{R}}^d\right\}.$$

2.1. New Insight into Dependency Functions

In this section, we recall useful results about generic dependency models of non-independent variables (see [3,13,14,15,18]). For a d-dimensional random vector of non-independent variables (i.e., $\mathbf{X}\sim F$), a dependency model of $\mathbf{X}$ consists of expressing a subset of $d-1$ variables (i.e., ${\mathbf{X}}_{\sim j}$) as a function of independent variables, including $X_j$.

Formally, if $\mathbf{X}\sim F$ with $F\left(\mathbf{x}\right)\ne {\prod }_{j=1}^d{F}_j\left(x_j\right)$, then there exists ([3,13,14,15,18]):

(i)

New independent variables $\mathbf{Z}:=\left(Z_{w_1},\dots ,Z_{w_{d-1}}\right)$, which are independent of $X_j$;

(ii)

A dependency function $r_j:{\mathbb{R}}^d\to R^{d-1}$,

such that

$${\mathbf{X}}_{\sim j}\stackrel{d}{=}{r}_j\left(X_j,\mathbf{Z}\right);\mathrm{a}\mathrm{n}\mathrm{d}\left(X_j,{\mathbf{X}}_{\sim j}\right)\stackrel{d}{=}\left(X_j,r_j\left(X_j,\mathbf{Z}\right)\right),$$

(1)

where ${\mathbf{X}}_{\sim j}=:(X_{w_1},\dots ,X_{w_{d-1}})$, and $A\stackrel{d}{=}B$ means that the random variables A and B have the same CDF.

It is worth noting that the dependency model is not unique in general. Uniqueness can be obtained under additional conditions provided in Proposition 1, which enable the inversion of the dependency function $r_j$.

Proposition 1.

Consider a dependency model of the continuous random vector $\mathbf{X}\sim F$, given by ${\mathbf{X}}_{\sim j}=r_j\left(X_j,\mathbf{Z}\right)$, with a prescribed order $\left(w_1,\dots ,w_{d-1}\right)$.

If $X_j$ is the explanatory variable and the distribution of $\mathbf{Z}$ is prescribed, then:

(i) 

The dependency model is uniquely defined;

(ii) 

The dependency model is invertible, and the unique inverse is given by

$$\mathbf{Z}=r_j^{-1}\left({\mathbf{X}}_{\sim j}|X_j\right).$$

(2)

Proof. 

See Appendix A. □

It should be noted that the dependency models (DMs) are vector-valued functions of independent input variables. Thus, DMs facilitate the calculus of partial derivatives, such as the partial derivatives of $\mathbf{X}$ with respect to $X_j$. Moreover, the inverse of a DM avoids working with $\mathbf{Z}$. A natural choice of the order $\left(w_1,\dots ,w_{d-1}\right)$ is $(1,\dots ,j-1,j+1,\dots ,d)$.

2.2. Enhanced Implicit Functions: Dependency Functions

In this section, we provide a probabilistic version of the implicit function using DMs.

Consider $\mathbf{X}\sim F$, a sample value of $\mathbf{X}$ given by $\mathbf{x}\in {\mathbb{R}}^d$, and a function $h:{\mathbb{R}}^d\to {\mathbb{R}}^p$ with $p\le d$ an integer. When connecting the input variables by p compatible equations, that is,

$$h\left(\mathbf{X}\right)=\mathbf{0},$$

the well-known theorem of the implicit function (see Theorem 1 below) states that for each sample value ${\mathbf{x}}^{*}$ satisfying $h\left({\mathbf{x}}^{*}\right)=\mathbf{0}$, a subset of $\mathbf{X}$ can be expressed as a function of the others in the neighborhood of ${\mathbf{x}}^{*}$. To recall this theorem, we use $u\subseteq \{1,\dots ,d\}$ with $\left|u\right|:=card\left(u\right)=d-p$, ${\mathbf{X}}_u:=\left(X_j,\forall j\in u\right)$, ${\mathbf{X}}_{\sim u}:=\left(X_j,\forall j\in \{1,\dots d\}\setminus u\right)$, and $B({\mathbf{x}}_u^{*},r_1)\subseteq {\mathbb{R}}^{d-p}$ (resp. $B({\mathbf{x}}_{\sim u}^{*},r_2)$) for an open ball centered on ${\mathbf{x}}_u^{*}$ (resp. ${\mathbf{x}}_{\sim u}^{*}$) with a radius of $r_1$ (resp. $r_2$). Again, $h_{{\mathbf{x}}_{\sim u}}^{\prime }\left({\mathbf{x}}^{*}\right)$ (resp. $h_{{\mathbf{x}}_u}^{\prime }$) is the formal Jacobian of h with respect to ${\mathbf{x}}_{\sim u}$ (resp. ${\mathbf{x}}_u$).

Theorem 1

(implicit function). Assume that $h\left({\mathbf{x}}^{*}\right)=\mathbf{0}$ and $h_{{\mathbf{x}}_{\sim u}}^{\prime }\left({\mathbf{x}}^{*}\right)$ is invertible. Then, there exists a function $g:B({\mathbf{x}}_u^{*},r_1)\to B({\mathbf{x}}_{\sim u}^{*},r_2)$, such that

$${\mathbf{x}}_{\sim u}=g\left({\mathbf{x}}_u\right);{\mathbf{x}}_{\sim u}^{*}=g\left({\mathbf{x}}_u^{*}\right);\frac{\partial {\mathbf{x}}_{\sim u}}{\partial {\mathbf{x}}_u}\left({\mathbf{x}}_u^{*}\right)=-{\left[h_{{\mathbf{x}}_{\sim u}}^{\prime }\left({\mathbf{x}}^{*}\right)\right]}^{-1}{h}_{{\mathbf{x}}_u}^{\prime }\left({\mathbf{x}}^{*}\right).$$

While Theorem 1 is useful, it turns out that the implicit function theorem provides a local relationship among the variables. It should be noted that the DMs derived in Section 2.1 provide global relationships once the CDFs of the input variables are known. The distribution function of the variables that satisfy the constraints given by $h\left(\mathbf{X}\right)=\mathbf{0}$ is needed to construct the global implicit function. To derive this distribution function, we assume that:

(A1) All the constraints $h\left(\mathbf{X}\right)=\mathbf{0}$ are compatible.

Under (A1), the constraints $h\left(\mathbf{X}\right)=\mathbf{0}$ introduce new dependency structures on the initial CDF F, which matters for our analysis. Probability theory ensures the existence of a distribution function that captures these dependencies.

Proposition 2.

Let $\mathbf{X}\sim F$ and ${\mathbf{X}}^c:=\left\{\mathbf{X}\sim F:h\left(\mathbf{X}\right)=\mathbf{0}\right\}\sim F^c$ be the constrained variables. If (A1) holds, we have

$${\displaystyle \left\{\begin{array}{c}\mathbf{X}\sim F\hfill \\ s.t.h\left(\mathbf{X}\right):=\mathbf{0}\hfill \end{array}\right.\stackrel{d}{=}{\mathbf{X}}^c,}$$

where $\stackrel{d}{=}$ denotes equality in distribution.

Introducing constraints on initial variables leads us to work with constrained variables that follow a new CDF, that is, ${\mathbf{X}}^c\sim F^c$. Some examples of generic and constrained variables ${\mathbf{X}}^c$ and their corresponding distribution functions can be found in [14,15,20]. When analytical derivations of the CDF of ${\mathbf{X}}^c$ are challenging or not possible, a common practice is to fit a distribution function to the observations of ${\mathbf{X}}^c$ using numerical simulations (see [14,21,22,23,24,25] for examples of distribution and density estimations). By using the new distributions of the input variables, Corollary 1 provides the probabilistic version of the implicit function.

Definition 2.

A distribution G is considered to be a degenerate CDF when G is the CDF of the Dirac measure with ${\delta }_a$ as the probability mass function, where $a\in \mathbb{R}$.

Corollary 1

([14,15]). Consider a random vector ${\mathbf{X}}^c:=\left\{\mathbf{X}\sim F:h\left(\mathbf{X}\right)=\mathbf{0}\right\}$ that follows $F^c$ as a CDF. Assume that (A1) holds and $F^c$ is a nondegenerate CDF.

Then, there exists a function $r_j:{\mathbb{R}}^d\to {\mathbb{R}}^{d-1}$ and $d-1$ independent variables $\mathbf{Z}$, such that $\mathbf{Z}$ is independent of $X_j^c$ and

$${\mathbf{X}}_{\sim j}^c=r_j(X_j^c,\mathbf{Z}).$$

Proof. 

This result is the DM for the distribution $F^c$ (see Section 2.1). □

While Corollary 1 gives the explicit function that links ${\mathbf{X}}_{\sim j}$ to $X_j$, we can sometimes extend this result as follows:

$${\mathbf{X}}_{\sim u}^c=r_u({\mathbf{X}}_u^c,{\mathbf{Z}}_u),$$

(3)

where ${\mathbf{Z}}_u$ is a vector of $d-\left|u\right|$ independent variables, ${\mathbf{Z}}_u$ is independent of ${\mathbf{X}}_u^c$, and $r_u:{\mathbb{R}}^d\to {\mathbb{R}}^{d-\left|u\right|}$ (see Section 2.1 and [14]).

Remark 1.

We can easily generalize the above process to handle (i) constrained inequations such as $h\left(\mathbf{X}\right)<\mathbf{0}$ or $h\left(\mathbf{X}\right)>\mathbf{0}$ (see Section 6 and [15]), and (ii) a mixture of constrained equations and inequalities involving different variables.

Remark 2.

For a continuous random vector $\mathbf{X}$ with C as the copula and $F_j$, $j=1,\dots ,d$ as the marginal distributions, an expression of its DM is given by ([3,14])

$$\begin{array}{c}\hfill \begin{array}{ccc}{X}_1& =& F_1^{-1}\left(C_{1|j}^{-1}\left(Z_1|F_j\left(X_j\right)\right)\right)=:r_{j,1}\left(X_j,Z_1\right)\hfill \\ X_2& =& F_2^{-1}\left(C_{2|j,1}^{-1}\left(Z_2|F_j\left(X_j\right),F_1\left(r_{j,1}\left(X_j,Z_1\right)\right)\right)\right)=:r_{j,2}\left(X_j,Z_1,Z_2\right)\hfill \\ ⋮& & \end{array},\end{array}$$

(4)

where $C_{1|j}$ is the conditional copula, $C_{1|j}^{-1}$ is the inverse of $C_{1|j}$, and $r_{j,1}$ and $r_{j,2}$ are real-valued functions.

2.3. Representation of Functions with Non-Independent Variables

In general, a function may include a group of independent variables, as well as groups of non-independent variables such as correlated variables and/or dependent variables. We can then organize these input variables as follows:

(O): the random vector $\mathbf{X}:=(X_1,\dots ,X_d)$ consists of K independent random vector(s) given by $\mathbf{X}=\left({\mathbf{X}}_{{\mathit{\pi }}_1},\dots ,{\mathbf{X}}_{{\mathit{\pi }}_K}\right)$, where the sets ${\mathit{\pi }}_1,\dots ,{\mathit{\pi }}_K$ form a partition of $\{1,\dots ,d\}$. The random vector ${\mathbf{X}}_{{\mathit{\pi }}_{k_1}}:=\left(X_j,\forall j\in {\mathit{\pi }}_{k_1}\right)$ is independent of ${\mathbf{X}}_{{\mathit{\pi }}_{k_2}}:=\left(X_j,\forall j\in {\mathit{\pi }}_{k_2}\right)$ for every pair $k_1,k_2\in \{1,\dots ,K\}$ with $k_1\ne k_2$. Without loss of generality, we use ${\mathbf{X}}_{{\mathit{\pi }}_1}$ for a random vector of $d_1\ge 0$ independent variable(s) and ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with $k\ge 2$ for a random vector of $d_k\ge 2$ dependent variables.

We use $({\pi }_{1,k},\dots ,{\pi }_{d_k,k})$ to denote the ordered permutation of ${\mathit{\pi }}_k$ (i.e., ${\pi }_{1,k}<{\pi }_{2,k}\dots <{\pi }_{d_k,k}$). For any ${\pi }_{j,k}\in {\mathit{\pi }}_k$, we use $X_{{\pi }_{j,k}}$ to refer to an element of ${\mathbf{X}}_{{\mathit{\pi }}_k}$; $(\sim {\pi }_{j,k}):=({\pi }_{1,k},\dots ,{\pi }_{j-1,k},$${\pi }_{j+1,k},\dots ,{\pi }_{d_k,k})$ and ${\mathbf{Z}}_k:=\left(Z_{{\pi }_{\ell ,k}},\forall {\pi }_{\ell ,k}\in (\sim {\pi }_{j,k})\right)$. Keeping in mind the DMs (see Section 2.1), we can represent ${\mathbf{X}}_{{\mathit{\pi }}_k}$ by

$${\mathbf{X}}_{\sim {\pi }_{j,k}}\stackrel{d}{=}{r}_{{\pi }_{j,k}}\left(X_{{\pi }_{j,k}},{\mathbf{Z}}_k\right),\forall k\in \{2,\dots ,K\},$$

(5)

where $r_{{\pi }_{j,k}}(·)=\left(r_{{\pi }_{1,k}}(·),\dots ,r_{{\pi }_{j-1,k}}(·),r_{{\pi }_{j+1,k}}(·),\dots ,r_{{\pi }_{d_k}}(·)\right)$; ${\mathbf{Z}}_k$ is a random vector of $d_k-1$ independent variables; and $X_{{\pi }_{1,k}}$ is independent of ${\mathbf{Z}}_k$. Based on the above DM of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with $k=2,\dots ,K$, let us introduce new functions, that is, $c_{{\pi }_{j,k}}:{\mathbb{R}}^{d_k}\to {\mathbb{R}}^{d_k}$, given by

$c_{{\pi }_{j,k}}(·)=\left(r_{{\pi }_{1,k}}(·),\dots ,r_{{\pi }_{j-1,k}}(·),r_{{\pi }_{j,k}}(·)=X_{{\pi }_{j,k}},r_{{\pi }_{j+1,k}}(·),\dots ,r_{{\pi }_{d_k}}(·)\right)$ and

$${\mathbf{X}}_{{\mathit{\pi }}_k}\stackrel{}{=}{c}_{{\pi }_{j,k}}(X_{{\pi }_{j,k}},{\mathbf{Z}}_k)=:\left(X_{{\pi }_{j,k}},r_{{\pi }_{j,k}}(X_{{\pi }_{j,k}},{\mathbf{Z}}_k)\right).$$

The function $c_{{\pi }_{j,k}}$ maps the independent variables $(X_{{\pi }_{j,k}},{\mathbf{Z}}_k)$ onto ${\mathbf{X}}_{{\mathit{\pi }}_k}$, and the chart

$${\displaystyle \begin{array}{cccccc}& {\mathbb{R}}^d& \stackrel{c}{\to }& {\mathbb{R}}^d& \stackrel{f}{\to }& {\mathbb{R}}^n\\ \hfill & \left(\begin{array}{c}{\mathbf{X}}_{{\mathit{\pi }}_1}\\ X_{{\pi }_{j,2}}\\ {\mathbf{Z}}_2\\ ⋮\\ X_{{\pi }_{j,K}}\\ {\mathbf{Z}}_K\end{array}\right)& \mapsto & \left(\begin{array}{c}{\mathbf{X}}_{{\mathit{\pi }}_1}\\ {\mathbf{X}}_{{\mathit{\pi }}_2}\\ ⋮\\ {\mathbf{X}}_{{\mathit{\pi }}_K}\end{array}\right)=\mathbf{X}& \mapsto & \left(\begin{array}{c}{f}_1\left(\mathbf{X}\right)\\ ⋮\\ f_n\left(\mathbf{X}\right)\end{array}\right)\end{array},}$$

leads to a new representation of functions with non-independent variables. Indeed, composing f with c yields

$$f({\mathbf{X}}_{{\mathit{\pi }}_1},{\mathbf{X}}_2,\dots ,{\mathbf{X}}_{{\mathit{\pi }}_k})\stackrel{d}{=}f\circ c\left({\mathbf{X}}_{{\mathit{\pi }}_1},X_{{\pi }_{j,2}},{\mathbf{Z}}_2,\dots ,X_{{\pi }_{j,K}},{\mathbf{Z}}_K\right),$$

(6)

where $c\left({\mathbf{X}}_{{\mathit{\pi }}_1},X_{{\pi }_{j,2}},{\mathbf{Z}}_2,\dots ,X_{{\pi }_{j,K}},{\mathbf{Z}}_K\right):=\left({\mathbf{X}}_{{\mathit{\pi }}_1},c_{{\pi }_{j,2}}(X_{{\pi }_{j,2}},{\mathbf{Z}}_2),\dots ,c_{{\pi }_{j,K}}(X_{{\pi }_{j,K}},{\mathbf{Z}}_K)\right)$.

The equivalent representation of f, given by (6), relies on the innovation variables $\mathbf{Z}:=({\mathbf{Z}}_2,\dots ,{\mathbf{Z}}_K)$. Recall that for the continuous random vector ${\mathbf{X}}_{{\mathit{\pi }}_k}$, the DM $r_{{\pi }_{j,k}}$, given by (5), is always invertible (see Proposition 1), and, therefore, $c_{{\pi }_{j,k}}$ is also invertible. These inversions are helpful for working with $\mathbf{X}$ only.

3. Actual Partial Derivatives

This section discusses the calculus of partial derivatives of functions with non-independent variables using only one relationship among the inputs, such as the DM given by Equation (5). The usual assumptions made are:

(A2) The joint (resp. marginal) CDF is continuous and has a density function $\rho >0$ on its open support;

(A3) Each component of the dependency function $r_{{\pi }_{j,k}}$ is differentiable with respect to $X_{{\pi }_{j,k}}$;

(A4) Each component of the function f, that is, $f_{\ell }$ with $\ell \in \{1,\dots ,n\}$, is differentiable with respect to each input.

Without loss of generality and for the sake of simplicity, here, we suppose that $n=1$. Namely, we use ${\mathcal{I}}_{d\times d}\in {\mathbb{R}}^{d\times d}$ for the identity matrix, and ${\mathsf{O}}_{d\times d_1}\in {\mathbb{R}}^{d\times d_1}$ for the null matrix. It is common to use $\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_k}}$ for the formal partial derivatives of f with respect to each input of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ (i.e., the derivatives obtained by considering inputs as independent) with $k=1,\dots ,K$. Thus, the formal gradient of f (i.e., the gradient with respect to the Euclidean metric) is given by

$$\nabla f:={\left[\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_1}}^T\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_2}}^T\dots \nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_K}}^T\right]}^T.$$

(7)

Keeping in mind the function $c_{{\pi }_{j,k}}(·)$, the partial derivatives of each component of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{j,k}}$ are given by

$$J^{\left({\pi }_{j,k}\right)}:=\frac{\partial c_{{\pi }_{j,k}}}{\partial x_{{\pi }_{j,k}}}={\left[\frac{\partial X_{{\pi }_{1,k}}}{\partial x_{{\pi }_{j,k}}}\dots \frac{\partial X_{{\pi }_{d_k,k}}}{\partial x_{{\pi }_{j,k}}}\right]}^T={\left[\frac{\partial r_{{\pi }_{1,k}}}{\partial x_{{\pi }_{j,k}}}\dots \underset{j^{\mathrm{th}}\mathrm{position}}{\underbrace{1}}\dots \frac{\partial r_{{\pi }_{d_k,k}}}{\partial x_{{\pi }_{j,k}}}\right]}^T.$$

(8)

We use $J_i^{\left({\pi }_{j,k}\right)}$ for the $i^{th}$ element of $J^{\left({\pi }_{j,k}\right)}$. For instance, $J_j^{\left({\pi }_{j,k}\right)}=1$ and $J_{d_k}^{\left({\pi }_{j,k}\right)}$ represent the partial derivative of $X_{{\pi }_{d_k,k}}$ with respect to $X_{{\pi }_{j,k}}$. It is worth recalling that $J^{\left({\pi }_{j,k}\right)}$ is a vector-valued function of $(X_{{\pi }_{j,k}},{\mathbf{Z}}_k)$, and Lemma 1 expresses $J^{\left({\pi }_{j,k}\right)}$ as a function of ${\mathbf{x}}_{{\mathit{\pi }}_k}$ only.

Lemma 1.

Let ${\mathbf{x}}_{{\mathit{\pi }}_k}$ be a sample value of ${\mathbf{X}}_{{\mathit{\pi }}_k}$. If Assumptions (A2)–(A4) hold, the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{j,k}}$ evaluated at ${\mathbf{x}}_{{\mathit{\pi }}_k}$ are given by

$$J^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right):={\left[\frac{\partial r_{{\pi }_{1,k}}}{\partial x_{{\pi }_{j,k}}}\dots \underset{j^{\mathit{th}}\mathit{position}}{\underbrace{1}}\dots \frac{\partial r_{{\pi }_{d_k,k}}}{\partial x_{{\pi }_{j,k}}}\right]}^T\left(x_{{\pi }_{j,k}},r_{{\pi }_{j,k}}^{-1}\left({\mathbf{x}}_{\sim {\mathit{\pi }}_{j,k}}|x_{{\pi }_{j,k}}\right)\right).$$

(9)

Proof. 

See Appendix B. □

Again, $J_{\ell }^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$ with $\ell \in \{1,\dots ,d_k\}$ represents the ${\ell }^{th}$ component of $J^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$, as provided in Lemma 1. Using these components and the chain rule, Theorem 2 provides the actual partial derivatives of functions with non-independent variables (i.e., $\frac{{\partial }_{a}f}{\partial \mathbf{x}}=:J_{{\mathit{\pi }}_k}^a\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right))$, which are the derivatives obtained by utilizing only one dependency function given by Equation (5).

Theorem 2.

Let $\mathbf{x}\in {\mathbb{R}}^d$ be a sample value of $\mathbf{X}$ and ${\pi }_{j,k}\in {\mathit{\pi }}_k$ with $k=2,\dots ,K$. If Assumptions (A2)–(A4) hold, then:

(i) 

The actual Jacobian matrix of $c_{{\pi }_{j,k}}$ is given by

$$J_{{\mathit{\pi }}_k}^a\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right):=\left[\begin{array}{ccc}\frac{J^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)}{J_1^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)}& \dots & \frac{J^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)}{J_{d_k}^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)}\end{array}\right],\forall k\in \{2,\dots ,K\}.$$

(10)

(ii) 

The actual Jacobian matrix of c or $\frac{{\partial }_a\mathbf{X}}{\partial \mathbf{x}}$ is given by

$$\begin{array}{ccc}\hfill J^a\left(\mathbf{x}\right)& :=& \left[\begin{array}{cccc}{\mathcal{I}}_{d_1\times d_1}& {\mathsf{O}}_{d_1\times d_2}& \dots & {\mathsf{O}}_{d_1\times d_K}\\ {\mathsf{O}}_{d_2\times d_1}& J_{{\mathit{\pi }}_2}^a\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)& \dots & {\mathsf{O}}_{d_2\times d_K}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathsf{O}}_{d_K\times d_1}& {\mathsf{O}}_{d_K\times d_2}& \dots & J_{{\mathit{\pi }}_K}^a\left({\mathbf{x}}_{{\mathit{\pi }}_K}\right)\end{array}\right],\hfill \end{array}$$

(11)

(iii) 

The actual partial derivatives of f are given by

$$\frac{{\partial }_{a}f}{\partial \mathbf{x}}\left(\mathbf{x}\right):=J^a{\left(\mathbf{x}\right)}^T\nabla f\left(\mathbf{x}\right).$$

(12)

Proof. 

see Appendix C. □

The results from Theorem 2 are based on only one dependency function, which uses $X_{{\pi }_{j,k}}$ as the explanatory input. Thus, the actual Jacobian $J^a\left(\mathbf{x}\right)$ and the actual partial derivatives of f provided in (10)–(12) will change with the choice of the explanatory input $X_{{\pi }_{j,k}}$ for every $j\in \{1,\dots ,d_k\}$. All these possibilities are not surprising. Indeed, while no additional explicit assumption is necessary for calculating the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{j,k}}$ (i.e., $J^{\left({\pi }_{j,k}\right)}$), we implicitly keep the other variables fixed when calculating the partial derivative of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$, that is, $\frac{J^{\left({\pi }_{j,k}\right)}}{J_i^{\left({\pi }_{j,k}\right)}}$ for each $i\ne j$. Such an implicit assumption is due to the reciprocal rule used to derive the results (see Appendix C). In general, the components of ${\mathbf{X}}_{\sim {\pi }_{j,k}}$, such as $X_{{\pi }_{i_1,k}}$ and $X_{i_2,k}$, are both functions of $X_{{\pi }_{j,k}}$ and $Z_{{\pi }_{1,k}}$ at the very least. Thus, different possibilities of the actual Jacobians are based on different implicit assumptions, making it challenging to use the actual partial derivatives. Further drawbacks of the actual partial derivatives of f are illustrated in Section 3

Example 1

We consider the function $f(X_1,X_2)=X_1+X_2+X_1{X}_2$, which includes two correlated inputs $\mathbf{X}\sim {\mathcal{N}}_2\left(\mathbf{0},\left[\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right]\right)$. We see that $\nabla f\left(\mathbf{X}\right)=\left[\begin{array}{c}1+X_2\\ 1+X_1\end{array}\right]$. Using the DM of $\mathbf{X}$ given by (see [3,14,15])

$$X_2=\rho X_1+\sqrt{1-{\rho }^2}{Z}_2⟺Z_2=(X_2-\rho X_1)/\sqrt{1-{\rho }^2},$$

the actual Jacobian matrix of c and the actual partial derivatives of f are given by

$$J^a\left(\mathbf{X}\right)=\left[\begin{array}{cc}1& \frac{1}{\rho }\\ \rho & 1\end{array}\right];\frac{{\partial }_{a}f}{\partial \mathbf{x}}\left(\mathbf{X}\right)=\left[\begin{array}{c}1+X_2+\rho (1+X_1)\\ \frac{1+X_2}{\rho }+1+X_1\end{array}\right].$$

When $\rho =1$, both inputs are perfectly correlated, and we have $X_1=X_2$, which also implies that $f(X_1,X_2)=f\left(X_1\right)=2X_1+X_1^2=f\left(X_2\right)=2X_2+X_2^2$. We can check that $\frac{{\partial }_{a}f}{\partial \mathbf{x}}\left(\mathbf{X}\right)=\left[\begin{array}{c}2+2X_1\\ 2+2X_2\end{array}\right]$. However, when $\rho =0$, both inputs are independent, and we should expect the actual partial derivatives to be equal to the formal gradient $\nabla f$, but this is not the case. Moreover, using the second DM, which is given by

$$X_1=\rho X_2+\sqrt{1-{\rho }^2}{Z}_1⟺Z_1=(X_1-\rho X_2)/\sqrt{1-{\rho }^2},$$

it becomes apparent that $J^a\left(\mathbf{X}\right)=\left[\begin{array}{cc}1& \rho \\ \frac{1}{\rho }& 1\end{array}\right];\frac{{\partial }_{a}f}{\partial \mathbf{X}}\left(\mathbf{X}\right)=\left[\begin{array}{c}1+X_2+\frac{1+X_1}{\rho }\\ \rho (1+X_2)+1+X_1\end{array}\right],$ which differs from the previous results. All these drawbacks are due to the implicit assumptions made (e.g., keeping some variables fixed), which can be avoided (see Section 4).

4. Dependent Jacobian and Partial Derivatives

This section aims to derive the first- and second-order partial derivatives of functions with non-independent variables, without relying on any additional assumption, whether explicit or implicit. Essentially, we calculate or compute the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$ using only the dependency function that includes $X_{{\pi }_{i,k}}$ as an explanatory input, which can be expressed as follows:

$${\mathbf{X}}_{\sim {\pi }_{i,k}}=r_{{\pi }_{i,k}}\left(X_{{\pi }_{i,k}},{\mathbf{Z}}_{{\pi }_{i,k}}\right),\forall i=1,\dots ,d_k;\forall k=2,\dots ,K.$$

By using the above dependency function, the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$ are given as follows (see (9)):

$$J^{\left({\pi }_{i,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right):=\frac{\partial {\mathbf{X}}_{{\mathit{\pi }}_k}}{\partial x_{{\pi }_{i,k}}}={\left[\frac{\partial r_{{\pi }_{1,k}}}{\partial x_{{\pi }_{i,k}}}\dots \underset{i^{\mathrm{th}}\mathrm{position}}{\underbrace{1}}\dots \frac{\partial r_{{\pi }_{d_k,k}}}{\partial x_{{\pi }_{i,k}}}\right]}^T\left(x_{{\pi }_{i,k}},r_{{\pi }_{i,k}}^{-1}\left({\mathbf{x}}_{\sim {\pi }_{i,k}}|x_{{\pi }_{i,k}}\right)\right).$$

It should be noted that $J^{\left({\pi }_{i,k}\right)}$ does not require any supplementary assumption, as $X_{{\pi }_{i,k}}$ and ${\mathbf{Z}}_{{\mathit{\pi }}_{i,k}}$ are independent. Thus, $d_k$ different DMs are necessary to derive the dependent Jacobian and partial derivatives of f (see Theorem 3).

Theorem 3.

Let $\mathbf{x}\in {\mathbb{R}}^d$ be a sample value of $\mathbf{X}$, and assume that (A2)–(A4) hold:

(i) 

For all $k\ge 2$, the dependent Jacobian matrix $\frac{\partial {\mathbf{X}}_{{\mathit{\pi }}_k}}{\partial {\mathbf{x}}_{{\mathit{\pi }}_k}}$ is given by

$$J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right):=\left[\begin{array}{ccc}{J}^{\left({\pi }_{1,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)& \dots & J^{\left({\pi }_{d_k,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)\end{array}\right].$$

(13)

(ii) 

The dependent Jacobian matrix $\frac{\partial \mathbf{X}}{\partial \mathbf{x}}$ is given by

$$\begin{array}{ccc}\hfill J^d\left(\mathbf{x}\right)& :=& \left[\begin{array}{cccc}{\mathcal{I}}_{d_1\times d_1}& {\mathsf{O}}_{d_1\times d_2}& \dots & {\mathsf{O}}_{d_1\times d_K}\\ {\mathsf{O}}_{d_2\times d_1}& J_{{\mathit{\pi }}_2}^d\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)& \dots & {\mathsf{O}}_{d_2\times d_K}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathsf{O}}_{d_K\times d_1}& {\mathsf{O}}_{d_K\times d_2}& \dots & J_{{\mathit{\pi }}_K}^d\left({\mathbf{x}}_{{\mathit{\pi }}_K}\right)\end{array}\right],\hfill \end{array}$$

(14)

(iii) 

The partial derivatives of f are given by

$$\frac{\partial f}{\partial \mathbf{x}}\left(\mathbf{x}\right):=J^d{\left(\mathbf{x}\right)}^T\nabla f\left(\mathbf{x}\right),$$

(15)

Proof. 

See Appendix D. □

Although the results from Theorem 3 require different DMs, these results are more comprehensive than the actual partial derivatives because no supplementary assumption is available for each non-independent variable.

To derive the second-order partial derivatives of f, we use $f_{x_i{x}_j}^{{}^{″}}$ to denote the formal cross-partial derivative of f with respect to $x_i$ and $x_j$, and $H_{{\mathit{\pi }}_k}:={\left(f_{x_{j_1}{x}_{j_2}}^{{}^{″}}\right)}_{j_1,j_2\in {\mathit{\pi }}_k}$ to denote the formal or ordinary Hessian matrices of f restricted to ${\mathbf{X}}_{{\mathit{\pi }}_k}$ for $k=1,\dots ,K$. In the same sense, we use $H_{{\mathit{\pi }}_{k_1},{\mathit{\pi }}_{k_2}}:={\left(f_{x_{j_1}{x}_{j_2}}^{{}^{″}}\right)}_{j_1\in {\mathit{\pi }}_{k_1},j_2\in {\mathit{\pi }}_{k_2}}$ to denote the formal cross-Hessian matrix of f restricted to $({\mathbf{X}}_{{\mathit{\pi }}_{k_1}},{\mathbf{X}}_{{\mathit{\pi }}_{k_2}})$ for every pair $k_1,k_2\in \{1,\dots K\}$ with $k_1\ne k_2$. To ensure the existence of the second-order partial derivatives, we assume that:

(A5) The function f is twice (formal) differentiable with respect to each input;

(A6) Every dependency function $r_{{\pi }_{j,k}}$ is twice differentiable with respect to $X_{{\pi }_{j,k}}$.

By considering the $d_k$ DMs of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ (i.e., ${\mathbf{X}}_{\sim {\pi }_{i,k}}=r_{{\pi }_{i,k}}\left(X_{{\pi }_{i,k}},{\mathbf{Z}}_{{\pi }_{i,k}}\right)$, with $i=1,\dots ,d_k$) used to derive the dependent Jacobian, we can write

$$\frac{\partial J^{\left({\pi }_{i,k}\right)}}{\partial x_{{\pi }_{i,k}}}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right):=\frac{{\partial }^2{\mathbf{X}}_{{\mathit{\pi }}_k}}{{\partial }^2{x}_{{\pi }_{i,k}}}={\left[\frac{{\partial }^2{r}_{{\pi }_{1,k}}}{{\partial }^2{x}_{{\pi }_{i,k}}}\dots \underset{i^{\mathrm{th}}\mathrm{position}}{\underbrace{0}}\dots \frac{{\partial }^2{r}_{{\pi }_{d_k,k}}}{{\partial }^2{x}_{{\pi }_{i,k}}}\right]}^T\left(x_{{\pi }_{i,k}},r_{{\pi }_{i,k}}^{-1}\left({\mathbf{x}}_{\sim {\pi }_{i,k}}|x_{{\pi }_{i,k}}\right)\right),$$

for the second partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$. By using $diag\left(\mathbf{x}\right)\in {\mathbb{R}}^{d\times d}$ to represent a diagonal matrix with $\mathbf{x}$ as its diagonal elements and

$$\mathcal{D}{J}_{{\mathit{\pi }}_k}^d\left(\mathbf{x}\right):=diag\left({\left[\frac{\partial J^{\left({\pi }_{1,k}\right)}}{\partial x_{{\pi }_{1,k}}}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right),\dots ,\frac{\partial J^{\left({\pi }_{d_k,k}\right)}}{\partial x_{{\pi }_{d_k,k}}}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)\right]}^T\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_k}}\left(\mathbf{x}\right)\right)J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right),$$

(16)

for all $k\in \{2,\dots ,K\}$, Theorem 4 provides the dependent second-order partial derivatives (i.e., $\frac{{\partial }^{2}f}{{\partial }^2\mathbf{x}}$).

Theorem 4.

Let $\mathbf{x}$ be a sample value of $\mathbf{X}$. If (A2), (A5), and (A6) hold, then

$$\frac{{\partial }^{2}f}{{\partial }^2\mathbf{x}}\left(\mathbf{x}\right):=$$

$$\begin{array}{c}\hfill \left[\begin{array}{cccc}{H}_{{\mathit{\pi }}_1}\left(\mathbf{x}\right)& H_{{\mathit{\pi }}_1,{\mathit{\pi }}_2}\left(\mathbf{x}\right)J_{{\mathit{\pi }}_2}^d\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)& \dots & H_{{\mathit{\pi }}_1,{\mathit{\pi }}_K}\left(\mathbf{x}\right)J_{{\mathit{\pi }}_K}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)\\ & & & \\ J_{{\mathit{\pi }}_2}^d{\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)}^T{H}_{{\mathit{\pi }}_2,{\mathit{\pi }}_1}\left(\mathbf{x}\right)& J_{{\mathit{\pi }}_2}^d{\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)}^T{H}_{{\mathit{\pi }}_2}\left(\mathbf{x}\right)J_{{\mathit{\pi }}_2}^d\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)& \dots & J_{{\mathit{\pi }}_2}^d{\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)}^T{H}_{{\mathit{\pi }}_2,{\mathit{\pi }}_K}\left(\mathbf{x}\right)\\ & +\mathcal{D}{J}_{{\mathit{\pi }}_2}^d\left(\mathbf{x}\right)& & \times J_{{\mathit{\pi }}_K}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)\\ ⋮& ⋮& \ddots & ⋮\\ J_{{\mathit{\pi }}_K}^d{\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)}^T{H}_{{\mathit{\pi }}_K,{\mathit{\pi }}_1}\left(\mathbf{x}\right)& J_{{\mathit{\pi }}_K}^d{\left({\mathbf{x}}_{{\mathit{\pi }}_K}\right)}^T{H}_{{\mathit{\pi }}_K,{\mathit{\pi }}_2}\left(\mathbf{x}\right)J_{{\mathit{\pi }}_2}^d\left({\mathbf{x}}_{{\mathit{\pi }}_2}\right)& \dots & J_{{\mathit{\pi }}_K}^d{\left({\mathbf{x}}_{{\mathit{\pi }}_K}\right)}^T{H}_{{\mathit{\pi }}_K}\left(\mathbf{x}\right)\\ & & & \times J_{{\mathit{\pi }}_K}^d\left({\mathbf{x}}_{{\mathit{\pi }}_K}\right)+\mathcal{D}{J}_{{\mathit{\pi }}_K}^d\left(\mathbf{x}\right)\end{array}\right].\end{array}$$

Proof. 

See Appendix E. □

Example 1 (Revisited)

Since $f(X_1,X_2)=X_1+X_2+X_1{X}_2$ and the DMs of $\mathbf{X}$ are given by

$$X_2=\rho X_1+\sqrt{1-{\rho }^2}{Z}_2⟹Z_2=(X_2-\rho X_1)/\sqrt{1-{\rho }^2},$$

$$X_1=\rho X_2+\sqrt{1-{\rho }^2}{Z}_1⟹Z_1=(X_1-\rho X_2)/\sqrt{1-{\rho }^2},$$

we can check that

$$J^d=\left[\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right];\frac{\partial f}{\partial \mathbf{x}}\left(\mathbf{X}\right)=\left[\begin{array}{c}1+X_2+\rho (1+X_1)\\ \rho (1+X_2)+1+X_1\end{array}\right];\frac{{\partial }^{2}f}{{\partial }^2\mathbf{x}}\left(\mathbf{X}\right)=\left[\begin{array}{cc}2\rho & 1+{\rho }^2\\ 1+{\rho }^2& 2\rho \end{array}\right].$$

For instance, when $\rho =1$, we have $\frac{\partial f}{\partial \mathbf{x}}\left(\mathbf{X}\right)=\frac{{\partial }_{a}f}{\partial \mathbf{x}}\left(\mathbf{X}\right)$, and when $\rho =0$, we have $\frac{\partial f}{\partial \mathbf{X}}\left(\mathbf{X}\right)=\nabla f\left(\mathbf{X}\right)$ and $\frac{{\partial }^{2}f}{{\partial }^2\mathbf{X}}\left(\mathbf{X}\right)=H\left(\mathbf{X}\right)$. Thus, the dependent partial derivatives of f align with the formal gradient and Hessian matrix when the inputs are independent.

5. Expansion of Functions with Non-Independent Variables

Although Section 4 provided the partial derivatives and cross-partial derivatives of f, it is misleading to think that the infinitesimal increment of f, given by $f({\mathbf{X}}_{{\mathit{\pi }}_k}+ϵ{\mathbf{e}}_{{\mathit{\pi }}_{j,k}})-f\left({\mathbf{X}}_{{\mathit{\pi }}_k}\right)$, should result in the individual effect quantified by $\frac{\partial f\left({\mathbf{X}}_{{\mathit{\pi }}_k}\right)}{\partial x_{{\pi }_{j,k}}}ϵ$ with ${\mathbf{e}}_{{\pi }_{j,k}}:={\left[0,\dots ,0,\underset{{\pi }_{j,k}^{\mathrm{th}}\mathrm{position}}{\underbrace{1}},0,\dots ,0\right]}^T$ and $ϵ>0$. Indeed, when moving $X_{{\pi }_{j,k}}$, it leads to partial movements of the other variables, and the effects we observe (i.e., $\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_k}}^T{J}^{\left({\pi }_{j,k}\right)}$) can also be attributed to other variables. The dependency structures of these effects are described by the dependent Jacobian matrix $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$ (see Equation (13)). Therefore, the definition of the gradient and Hessian of f with non-independent variables requires the introduction of a tensor metric or a Riemannian tensor.

In differential geometry, the function of the form

$$\begin{array}{cccc}{c}_{{\pi }_{j,k}}:& {\mathbb{R}}^{d_k}& \to & {\mathbb{R}}^{d_k}\hfill \\ & \left(X_{{\pi }_{j,k}},{\mathbf{Z}}_{{\pi }_{j,k}}\right)& \mapsto & \mathbf{X}:=(X_{{\pi }_{1,k}},\dots ,X_{{\pi }_{j,k}},\dots ,X_{{\pi }_{d_k,k}});{\mathbf{X}}_{\sim {\pi }_{j,k}}\stackrel{}{=}{r}_{{\pi }_{j,k}}\left(X_{{\pi }_{j,k}},{\mathbf{Z}}_{{\pi }_{j,k}}\right)\hfill \end{array},$$

for every ${\pi }_{i,k}\in {\mathit{\pi }}_k$ can be seen as a parametrization of a manifold ${\mathcal{M}}_k$ in ${\mathbb{R}}^{d_k}$. The $d_k$ column entries of the dependent Jacobian matrix $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)\in {\mathbb{R}}^{d_k\times d_k}$ span a local $m_k$-dimensional vector space, also known as the tangent space at ${\mathbf{x}}_{{\mathit{\pi }}_k}$, where $m_k$ is the rank of $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$, indicating the number of linearly independent columns in $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$.

By considering all the K groups of inputs and the corresponding dependent Jacobian matrix $J^d\left(\mathbf{x}\right)$, we can see that the support of the random vector $\mathbf{X}$ forms an m-dimensional manifold $\mathcal{M}$ in ${\mathbb{R}}^d$, where m is the rank of $J^d\left(\mathbf{x}\right)$. When $m\le d$, we work within the tangent space $T{\mathbb{R}}^m$ (or local coordinate system) spanned by the m column entries of $J^d\left(\mathbf{x}\right)$ that are linearly independent. Working in $T{\mathbb{R}}^m$ rather than $T{\mathbb{R}}^d$ ensures that the Riemannian tensor induced by $\mathbf{x}\mapsto \mathbf{x}$ using the dot product is invertible. Since the Riemannian tensor metric is often symmetric, the Moore–Penrose generalized inverse of symmetric matrices ([26,27,28]) allows us to keep working in $T{\mathbb{R}}^d$ in the subsequent discussion. Using the first fundamental form (see, e.g., [9,10,11]), the induced tensor metric is defined as the inner product between the column entries of the dependent Jacobian matrix of the dependency functions, that is,

$$G\left(\mathbf{x}\right):=J^d{\left(\mathbf{x}\right)}^T{J}^d\left(\mathbf{x}\right).$$

(17)

Based on these elements, the gradient and Hessian matrix are provided in Corollary 2. To that end, we use $G^{-1}$ to represent the inverse of the metric G, as given by Equation (17), when $m=d$, or the generalized inverse of G for every $m<d$ ([26,27,28]). For any $k\in \{1,\dots ,d\}$, the Christoffel symbols are defined by ([9,11,29,30])

$${\Gamma }_{ij}^k:=\frac{1}{2}\sum _{\ell =1}^{m=d}{G}_{k\ell }^{-1}\left(\mathbf{x}\right)\left(G_{i\ell ,x_j}^{{}^{\prime }}\left(\mathbf{x}\right)+G_{j\ell ,x_i}^{{}^{\prime }}\left(\mathbf{x}\right)-G_{ij,x_{\ell }}^{{}^{\prime }}\left(\mathbf{x}\right)\right);\forall i,j=1,\dots ,d,$$

where $G_{i\ell ,x_j}^{{}^{\prime }}$ is the formal partial derivative of $G_{i\ell }$ with respect to $x_j$.

Corollary 2.

Let $\mathbf{x}$ be a sample value of $\mathbf{X}$, and assume that (A2) and (A5)–(A6) hold:

(i) The gradient of f is given by

$$grad\left(f\right)\left(\mathbf{x}\right):=G^{-1}\left(\mathbf{x}\right)\nabla f\left(\mathbf{x}\right),$$

(18)

(ii) The Hessian matrix of f is given by

$$Hes{s}_{ij}\left(f\right)\left(\mathbf{x}\right):=f_{x_i{x}_j}^{{}^{″}}\left(\mathbf{x}\right)-\sum _{k=1}^{m=d}{\Gamma }_{ij}^k\left(\mathbf{x}\right)f_{x_k}^{{}^{\prime }}\left(\mathbf{x}\right).$$

(19)

Proof. 

Points (i)–(ii) result from the definition of the gradient and the Hessian matrix within a Riemannian geometric context equipped with the metric G (see [9,10,11,31]). □

Taylor’s expansion is widely used to approximate functions with independent variables. In the subsequent discussion, we are concerned with the approximation of a function with non-independent variables. The Taylor-type expansion of a function with non-independent variables is provided in Corollary 3 using the gradient and Hessian matrix.

Corollary 3.

Let $\mathbf{x}$, ${\mathbf{x}}_0$ be two sample values of $\mathbf{X}$, and assume that (A2) and (A5)–(A6) hold. Then, we have

$$f\left(\mathbf{x}\right)\approx f\left({\mathbf{x}}_0\right)+{\left(\mathbf{x}-{\mathbf{x}}_0\right)}^{T}grad\left(f\right)\left({\mathbf{x}}_0\right)+\frac{1}{2}{\left(\mathbf{x}-{\mathbf{x}}_0\right)}^{T}Hess\left(f\right)\left({\mathbf{x}}_0\right)\left(\mathbf{x}-{\mathbf{x}}_0\right),$$

(20)

provided that $\mathbf{x}$ is close to ${\mathbf{x}}_0$.

Proof. 

The proof is straightforward using the dot product induced by the tensor metric G within the tangent space and considering the Taylor expansion provided in [11]. □

Example 1 (Revisited)

For the function in Example 1, we can check that the tensor metric is

$G\left(\mathbf{X}\right)=\left[\begin{array}{cc}1+{\rho }^2& 2\rho \\ 2\rho & 1+{\rho }^2\end{array}\right];G^{-1}\left(\mathbf{X}\right)=\frac{1}{{({\rho }^2-1)}^2}\left[\begin{array}{cc}1+{\rho }^2& -2\rho \\ -2\rho & 1+{\rho }^2\end{array}\right]$; and the gradient is

$$grad\left(f\right)\left(\mathbf{X}\right)=\frac{1}{{({\rho }^2-1)}^2}\left[\begin{array}{c}{(1-\rho )}^2+X_2(1+{\rho }^2)-2\rho X_1\\ {(1-\rho )}^2+X_1(1+{\rho }^2)-2\rho X_2\end{array}\right],$$

which reduces to $\nabla f\left(\mathbf{X}\right)$ when the variables are independent (i.e., $\rho =0$).

6. Application

In this section, we consider three independent input factors $X_j\sim \mathcal{N}\left(0,1\right)$ with $j=1,2,3$, $\mathbf{X}:=(X_1,X_2,X_3)$, a constant $c\in {\mathbb{R}}_{+}$, and the function

$$f\left(\mathbf{X}\right)=X_1^2+X_2^2+X_3^2.$$

Also, we consider the constraint $f\left(\mathbf{X}\right)\le c$. It is known in [15] (Corollary 4) that the DM of

$${\mathbf{X}}^w:=\left(X_1^w,X_2^w,X_3^w\right):=\left\{X_j\sim \mathcal{N}(0,1),j=1,2,3:X_1^2+X_2^2+X_3^2\le c\right\}$$

is given by

$$X_2^w=R_2{Z}_2\sqrt{c-{\left(X_1^w\right)}^2};X_3^w=R_3{Z}_3\sqrt{c-{\left(X_1^w\right)}^2}\sqrt{1-Z_2^2}.$$

We can then write

$$\frac{\partial X_2^w}{\partial X_1^w}=-X_1^w{R}_2{Z}_2\frac{1}{\sqrt{c-{\left(X_1^w\right)}^2}}=\frac{-X_1^w{X}_2^w}{c-{\left(X_1^w\right)}^2},$$

$$\frac{\partial X_3^w}{\partial X_1^w}=-X_1^w{R}_3{Z}_3\frac{\sqrt{1-Z_2^2}}{\sqrt{c-{\left(X_1^w\right)}^2}}=\frac{-X_1^w{X}_3^w}{c-{\left(X_1^w\right)}^2}.$$

Using the above derivatives and symmetry among the inputs, the dependent Jacobian and the tensor metric are given by

$$J^d\left({\mathbf{X}}^w\right)=\left[\begin{array}{ccc}1& \frac{-X_1^w{X}_2^w}{c-{\left(X_2^w\right)}^2}& \frac{-X_1^w{X}_3^w}{c-{\left(X_3^w\right)}^2}\\ \frac{-X_1^w{X}_2^w}{c-{\left(X_1^w\right)}^2}& 1& \frac{-X_2^w{X}_3^w}{c-{\left(X_3^w\right)}^2}\\ \frac{-X_1^w{X}_3^w}{c-{\left(X_1^w\right)}^2}& \frac{-X_2^w{X}_3^w}{c-{\left(X_2^w\right)}^2}& 1\end{array}\right],G\left({\mathbf{X}}^w\right)={\left[J^d\left({\mathbf{X}}^w\right)\right]}^T{J}^d\left({\mathbf{X}}^w\right).$$

The following partial derivatives of f can be deduced:

$$\frac{\partial f}{d{\mathbf{x}}^w}=\left[\begin{array}{ccc}1& \frac{-X_1^w{X}_2^w}{c-{\left(X_1^w\right)}^2}& \frac{-X_1^w{X}_3^w}{c-{\left(X_1^w\right)}^2}\\ \frac{-X_1^w{X}_2^w}{c-{\left(X_2^w\right)}^2}& 1& \frac{-X_2^w{X}_3^w}{c-{\left(X_2^w\right)}^2}\\ \frac{-X_1^w{X}_3^w}{c-{\left(X_3^w\right)}^2}& \frac{-X_2^w{X}_3^w}{c-{\left(X_3^w\right)}^2}& 1\end{array}\right]\left[\begin{array}{c}2X_1^w\\ 2X_2^w\\ 2X_3^w\end{array}\right]=\left[\begin{array}{c}2X_1^w\left(1-\frac{{\left(X_2^w\right)}^2}{c-{\left(X_1^w\right)}^2}-\frac{{\left(X_3^w\right)}^2}{c-{\left(X_1^w\right)}^2}\right)\\ 2X_2^w\left(1-\frac{{\left(X_1^w\right)}^2}{c-{\left(X_2^w\right)}^2}-\frac{{\left(X_3^w\right)}^2}{c-{\left(X_2^w\right)}^2}\right)\\ 2X_3^w\left(1-\frac{{\left(X_1^w\right)}^2}{c-{\left(X_3^w\right)}^2}-\frac{{\left(X_2^w\right)}^2}{c-{\left(X_3^w\right)}^2}\right)\end{array}\right].$$

For given values of $X_1^w$, $X_2^w$, and $X_3^w$ and as $c\to \infty$, we can see that $\frac{\partial f}{d{x}_1^w}=2X_1^w$, which is exactly the partial derivative of f when the inputs are independent. Note that as $c\to \infty$, the inputs become independent, as the constraint imposed on $\mathbf{X}$ is always satisfied.

Keeping in mind Equation (6), it is worth noting that the partial derivatives of f can be directly derived by making use of an equivalent DM of ${\mathbf{X}}^w$, that is, ${\left(X_2^w\right)}^2=Z_2(c-{\left(X_1^w\right)}^2),{\left(X_3^w\right)}^2=Z_3(c-{\left(X_1^w\right)}^2)(1-Z_2)$, where ${\left(X_1^w\right)}^2\sim B1(c,1/2,2)$, $Z_2\sim Beta(1/2,3/2)$ and $Z_3\sim Beta(1/2,1)$ are independent, with $B1$ representing the beta distribution of the first kind (see [15], Corollaries 2). Indeed, we have

$$f\left({\mathbf{X}}^w\right)={\left(X_1^w\right)}^2(1-Z_2-Z_3(1-Z_2))+c{Z}_2+c{Z}_3(1-Z_2)={\left(X_1^w\right)}^2(1-Z_2)(1-Z_3)+c{Z}_2+c{Z}_3(1-Z_2);$$

and

$$\frac{\partial f}{d{x}_1^w}=2X_1^w(1-Z_2)(1-Z_3)=2X_1^w\frac{c-{\left(X_1^w\right)}^2-{\left(X_2^w\right)}^2-{\left(X_3^w\right)}^2}{c-{\left(X_1^w\right)}^2},$$

because

$$Z_2=\frac{{\left(X_2^w\right)}^2}{c-{\left(X_1^w\right)}^2};1-Z_2=\frac{c-{\left(X_1^w\right)}^2-{\left(X_2^w\right)}^2}{c-{\left(X_1^w\right)}^2};Z_3=\frac{{\left(X_3^w\right)}^2}{c-{\left(X_1^w\right)}^2-{\left(X_2^w\right)}^2}.$$

As a matter of fact, we obtain the same partial derivatives of f, keeping in mind the symmetry among the inputs.

7. Conclusions

A new approach for calculating and computing the partial derivatives, gradient, and Hessian of functions with non-independent variables is proposed and studied in this paper. It relies on (i) dependency functions that model the dependency structures among dependent variables, including correlated variables; (ii) the dependent Jacobian of the dependency functions; and (iii) the tensor metric using the dependent Jacobian. Based on the unique tensor metric due to the first fundamental form, the unique gradient of a function with non-independent variables is provided. Since the so-called dependent partial derivatives and the dependent Jacobian do not require any additional assumption (which is always the case), such derivatives (including the gradient) should be used.

The results obtained depend on the parameters of the distribution function or the density function of non-independent variables. For the values of such parameters that lead to independent variables, the proposed gradient and partial derivatives reduce to the formal gradient or the gradient with respect to the Euclidean metric. In the same sense, the proposed tensor metric reduces to the Euclidean metric using the above values of the parameters of the distribution function.

Using the proposed gradient and Hessian matrix, the Taylor-type expansion of a function with non-independent variables is provided. Although the generalized inverse of a symmetric matrix is used in some cases, more investigation is needed for the gradient calculus when the tensor metric is not invertible. The proposed gradient will be used for (i) the development of the active subspaces of functions with non-independent variables, and (ii) enhancing the optimization of functions subject to constraints.