Generalizing the Balance Heuristic Estimator in Multiple Importance Sampling

Abstract

In this paper, we propose a novel and generic family of multiple importance sampling estimators. We first revisit the celebrated balance heuristic estimator, a widely used Monte Carlo technique for the approximation of intractable integrals. Then, we establish a generalized framework for the combination of samples simulated from multiple proposals. Our approach is based on considering as free parameters both the sampling rates and the combination coefficients, which are the same in the balance heuristics estimator. Thus our novel framework contains the balance heuristic as a particular case. We study the optimal choice of the free parameters in such a way that the variance of the resulting estimator is minimized. A theoretical variance study shows the optimal solution is always better than the balance heuristic estimator (except in degenerate cases where both are the same). We also give sufficient conditions on the parameter values for the new generalized estimator to be better than the balance heuristic estimator, and one necessary and sufficient condition related to ${\chi }^2$ divergence. Using five numerical examples, we first show the gap in the efficiency of both new and classical balance heuristic estimators, for equal sampling and for several state of the art sampling rates. Then, for these five examples, we find the variances for some notable selection of parameters showing that, for the important case of equal count of samples, our new estimator with an optimal selection of parameters outperforms the classical balance heuristic. Finally, new heuristics are introduced that exploit the theoretical findings.

1. Introduction

Multiple importance sampling (MIS) is a Monte Carlo technique widely used in the literature of signal processing, computational statistics, and computer graphics for approximating complicated integrals. In its basic configuration, it works by drawing random samples from several proposal distributions (also called techniques) and weighting them appropriately in such a way that an estimator built with the pairs of weighted samples is consistent. Since the publication of [1], the celebrated balance heuristic estimator has been extensively used in the Monte Carlo literature, with an unprecedented success in the computer graphics industry (Eric Veach has been awarded with several prizes because of his contributions in the MIS literature, where the balance heuristic is arguably the most relevant one). In the balance heuristic method, different samples are simulated from each proposal and the traditional IS weight is assigned to each of them. Unlike the standard IS estimator, all the weighted samples are combined with an extra weighting, in such a way the resulting estimator typically shows a reduced variance. Its superiority in terms of variance with regard to other traditional combination schemes has been recently shown in [2], where a framework is established for sampling and weighting in MIS under equal number of samples per technique. The balance heuristic, also called the deterministic mixture [3], has been widely used in the literature of MIS. Further efficient variance reduction techniques are proposed in [4,5,6] also in the context of MIS, still with equal counts from each technique. Provably better estimators [7] and heuristically better ones [8,9] have been presented that use a different count of samples than equal count for all techniques. In [10], it has been shown the relationship of a better count of samples with generalized weighted means. The balance heuristic is also present in most of successful adaptive IS (AIS) methods, see [11,12,13,14,15], in particular in the case where all techniques are used to simulate the same number of samples. Recently, MIS has returned to the main focus in computer graphics in [16], where it is shown that allowing weights to be negative, the reduction over balance heuristics of the resulting MIS estimator can be higher than the one predicted by Veach bounds [17]; in [18], where one of the sampling techniques is optimized to decrease the overall variance of the resulting MIS estimator; in [19], where the weights are made proportional to the quotients of second moments divided by the variances of the independent techniques; in [20], where MIS is generalized to uncountably infinite sets of techniques.

Interestingly, the balance heuristic has two properties in the assigned weights. First, all techniques appear at the denominator of the weight of a specific technique. Second, they appear in a form of a mixture, with coefficients proportional to the number of samples simulated from each technique. In this paper, we relax this constraint providing a generalized weighting/combining family of estimators that has the balance heuristic as a particular case. First, we show that it is possible to use a specific set of coefficients to decide the amount of samples per technique, and a different set of coefficients to be applied as the importance weight. Second, we study four different cases fixing some of these coefficients (sampling and/or weighting), and we give the optimal solution for the rest of coefficients in such a way the variance of the MIS estimator is minimized. Note that, the novel estimator always outperforms the balance heuristic under the optimal choice of those coefficients. In five numerical examples we show that, under an adequate choice of parameters, the novel estimator outperforms the celebrated balance heuristic.

The rest of the paper is structured as follows. Section 2 revisits the balance heuristic estimator. In Section 3, we propose the new family of estimators that generalizes the balance heuristic. We address five cases of special interest, depending on which parameters are free and the number of samples simulated from each technique. We then, in Section 4, discuss the different singular points in the variances of the estimators considered. In Section 5, we give a necessary and sufficient condition, in terms of ${\chi }^2$ divergence (and its approximation as a Kullback–Leibler divergence), for the variance of the new estimator to be better than the one of balance heuristics. Finally, we conclude the paper with five numerical examples in Section 6, propose some heuristics, and give some conclusions in Section 7.

2. Balance Heuristic Estimator

The goal in IS is usually the estimation of the value of integral $\mu =\int f\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}$. In MIS, $n_i$ samples, ${\left\{{\mathbf{X}}_{i,j}\right\}}_{j=1}^{n_i}$, are simulated from a set of available probability density functions (pdfs), ${\left\{p_i\right\}}_{i=1}^n$. The MIS estimator introduced by Veach and Guibas [1] is given by

$$Z=\sum _{i=1}^n\frac{1}{n_i}\sum _{j=1}^{n_i}{w}_i\left({\mathbf{X}}_{i,j}\right)\frac{f\left({\mathbf{X}}_{i,j}\right)}{p_i\left({\mathbf{X}}_{i,j}\right)},$$

(1)

where $w_i\left(\mathbf{x}\right)$ is a weight function associated to the i-th proposal that fulfills both following conditions. First, the weights must sum up to one in all points of the domain where the value of the function is different from zero, i.e., ${\sum }_{i=1}^n{w}_i\left(\mathbf{x}\right)=1$, $\forall x$ where $f\left(\mathbf{x}\right)\ne 0$, Second, for all x where $p_i\left(\mathbf{x}\right)=0$, then $w_i\left(\mathbf{x}\right)=0$. In this paper, we consider only weighting functions that yield Z unbiased.

The balance heuristic estimator is a particular case of Equation (1) where the weight function is given by

$$w_i\left(\mathbf{x}\right)=\frac{n_i{p}_i\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{n}_k{p}_k\left(\mathbf{x}\right)},$$

(2)

which can be written too as

$$w_i\left(\mathbf{x}\right)=\frac{{\alpha }_i{p}_i\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)},$$

(3)

where $n_i={\alpha }_{i}N$. In this case, the estimator in Equation (1) becomes the balance heuristic or deterministic mixture estimator given by

$$\begin{array}{ccc}\hfill F& =& \sum _{i=1}^n\frac{1}{n_i}\sum _{j=1}^{n_i}\frac{{\alpha }_{i}f\left({\mathbf{X}}_{i,j}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left({\mathbf{X}}_{i,j}\right)}\hfill \end{array}$$

(4)

$$\begin{array}{ccc}& =& \frac{1}{N}\sum _{i=1}^n\sum _{j=1}^{n_i}\frac{f\left({\mathbf{X}}_{i,j}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left({\mathbf{X}}_{i,j}\right)}.\hfill \end{array}$$

(5)

2.1. Interpretation of F and General Notation of the Paper

Note that F is also called deterministic mixture scheme [3] or multi-sample MISestimator [7,17], as opposed to the case where all independent and identically distributed (i.i.d.) samples are from the mixture ${\psi }_{\alpha }={\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)$. The latter alternative is called the randomized balance heuristic or one-sample MIS estimator, and the estimator is denoted by $\mathcal{F}$. This alternative is a subtle variation of F that simply modifies the sampling by simulating the N total samples i.i.d. from the mixture ${\psi }_{\alpha }$ (instead of deterministically choosing the number of samples per proposal). Then, the estimator is the exact same expression of Equation (4). This randomized version $\mathcal{F}$ allows also for a re-interpretation of the deterministic balance heuristic, F, that can be seen as a mixture sampling (note that the mixture is present in the denominator of all importance weights) with variance reduction using stratified sampling (see Appendix 1 of [2] for a detailed discussion). In the randomized version, $n_i$ is then a random variable with expected value ${\alpha }_{i}N$, for all i. In summary, in this paper we refer as deterministic mixture estimator the cases where the number of samples per technique is deterministic, e.g., F, and random mixture estimator when there is i.i.d. sampling from the mixture (and hence $n_i$ is a random variable), e.g., $\mathcal{F}$. We represent the randomized estimators with calligraphic letters. All estimators, unless the opposite is clearly stated, are deterministic mixture estimators.

Finally, we denote estimators with the superindex${}^1$ when they are versions of a specific estimator but with a number of samples normalized to 1, e.g., $F^1$ is the normalized version of F. In other words, even if the estimators require that all the numbers of samples per technique are $n_i\in \mathbb{R}$, we use these normalized estimators to denote the variance normalized to 1 sample, which simplifies the comparison across estimators (for N total samples, the variance of the estimator would be just the variance of $F^1$ divided by N).

In

Table 1. Naming convention for the multiple importance sampling estimators in this paper. We drop the superindex${}^1$ from primary estimators when not strictly necessary.

Z	Generic deterministic (multi-sample) MIS estimator
$Z^1$	Generic deterministic (multi-sample) MIS estimator normalized to one sample
$\mathcal{Z}$	Generic randomized (one-sample) MIS estimator
${\mathcal{Z}}^1$	Generic randomized (one-sample) MIS estimator, for number of samples equal to 1
F	Balance heuristic multi-sample MIS estimator
$F^1$	Balance heuristic multi-sample MIS estimator normalized to one sample
$\mathcal{F}$	Balance heuristic one-sample MIS estimator
${\mathcal{F}}^1$	Balance heuristic one-sample MIS estimator, for number of samples equal to 1
G	Generalized balance heuristic multi-sample MIS estimator
$G^1$	Generalized balance heuristic multi-sample MIS estimator normalized to one sample
$\mathcal{G}$	Generalized balance heuristic one-sample MIS estimator

we show the naming convention used in this paper.

2.2. Rationale

In Theorems 9.2 and 9.4 of [17], the variances of $Z^1$ and its randomized version ${\mathcal{Z}}^1$ are given, i.e., the version where instead of deterministically selecting $n_i$, all the samples are directly simulated from ${\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)$. By subtracting their values we have

$$\begin{array}{c}\hfill V\left[{\mathcal{Z}}^1\right]-V\left[Z^1\right]=\sum _i{\alpha }_i{{\mu }^{\prime }}_i^2-{\mu }^2,\end{array}$$

(6)

where (we use here a notation to be consistent with notation in [7])

$$\begin{array}{c}\hfill {{\mu }^{\prime }}_i=\frac{1}{{\alpha }_i}\int w_i\left(\mathbf{x}\right)f\left(\mathbf{x}\right)d\mathbf{x},\end{array}$$

(7)

and thus

$$\begin{array}{c}\hfill \sum _i{\alpha }_i{{\mu }^{\prime }}_i=\mu .\end{array}$$

(8)

This result generalizes several particular cases derived in [7] and [2], in the context of a variance analysis of MIS estimators. From Equation (6) (see Appendix A) we have that

$$\begin{array}{c}\hfill V\left[Z^1\right]\le V\left[{\mathcal{Z}}^1\right],\end{array}$$

(9)

and equality only happens (apart from the case when both variances $V\left[Z^1\right],V\left[{\mathcal{Z}}^1\right]$ are zero) when for all i all ${\mu }_i^{\prime }$ are equal. One example is given by taking in Equation (7) for all i, $w_i\left(\mathbf{x}\right)=w_i$ constant and ${\alpha }_i=w_i$. We also show in Appendix A that, for the particular case when ${\alpha }_i=1/n$ and $f\left(x\right)\ge 0$, the upper bound for improvement of deterministic versus non-deterministic estimator is given by

$$\begin{array}{c}\hfill V\left[{\mathcal{Z}}^1\right]-V\left[Z^1\right]\le (n-1){\mu }^2,\end{array}$$

(10)

where the bound would be approached when there is an index k such that for all $i\ne k$, ${\mu }_k^{\prime }>>{\mu }_i^{\prime }$. In Theorem 9.4 of [17], it is shown that the optimal weighting functions $w_i\left(\mathbf{x}\right)$ for $\mathcal{Z}$ (i.e., those that minimize their variance $V\left[\mathcal{Z}\right]$), are the balance heuristic weights, Equation (3); therefore, the optimal case is $\mathcal{Z}\equiv \mathcal{F}$, where $\mathcal{F}$ is the random mixture estimator. This is, for any estimator $\mathcal{Z}$, when using the same distribution of samples, also taking into account Equation (9) (see also [7]), it always holds that

$$\begin{array}{c}\hfill V\left[F\right]\le V\left[\mathcal{F}\right]\le V\left[\mathcal{Z}\right].\end{array}$$

(11)

Further, in Theorem 9.2 of [17], it is proved that the estimator that optimizes the second moment of $Z^1$ estimator, this is, $V\left[Z^1\right]+{\sum }_i{\alpha }_i{{\mu }^{\prime }}_i^2$, is the balance heuristic estimator. Thus, it seems clear that for improvement we have to look for a deterministic estimator, which should be a generalization of balance heuristic mixture estimator F. This is presented in the next section.

2.3. Realistic Applications

2.3.1. Global Illumination

The rendering equation [21] tells us that the radiance $L_{out}(x,{\omega }_{out})$ exiting point x in direction ${\omega }_{out}$ is given by

$$L_{out}(x,{\omega }_{out})=L_e(x,{\omega }_{out})+{\int }_{\mathsf{\Omega }}{L}_{in}(x,{\omega }_{in})\rho (x,{\omega }_{in},{\omega }_{out})cos\theta d{\omega }_{in},$$

(12)

where $L_e(x,{\omega }_{out})$ is the emitted radiance, $L_{in}(x,{\omega }_{in})$ is the incoming radiance at x from direction ${\omega }_{in}$, with the integral extended to the hemisphere $\mathsf{\Omega }$ above x, $\theta$ is the angle of the normal at x with ${\omega }_{in}$, and $\rho$ is the bidirectional reflectance distribution function, which gives the fraction of luminance incoming from ${\omega }_{in}$ and scattered in the direction ${\omega }_{out}$. To approximate the integral in (12), multiple importance sampling is applied with pdfs that sample, respectively, the incoming radiance and the bidirectional reflectance.

2.3.2. Bayesian Inference

Multiple importance sampling (MIS) is often applied in Bayesian inference [22]. In this context, the posterior distribution is usually intractable, and computing its moments analytically is impossible; therefore, approximate methods are required. MIS is particularly well suited for cases when the posterior distribution is multimodal or needs to be approximated by a mixture of densities [2].

Typically, a set of observations, $\mathbf{y}\in {\mathbb{R}}^{d_y}$, are available. The inference task consists in estimating probabilistically some hidden parameters and/or latent variables $\mathbf{x}\in {\mathbb{R}}^{d_y}$, which are connected to the observations through a statistical model. The relation between observations and unknown parameters is encoded in the likelihood, $\ell \left(\mathbf{y}\right|\mathbf{x})$, and the prior knowledge on $\mathbf{x}$ is described through the prior distribution $p_0\left(\mathbf{x}\right)$. Through the Bayes’ rule, we can obtain the posterior distribution of the unknowns as

$$\tilde{\pi }\left(\mathbf{x}\right|\mathbf{y})=\frac{\ell \left(\mathbf{y}\right|\mathbf{x})p_0\left(\mathbf{x}\right)}{Z\left(\mathbf{y}\right)},$$

(13)

where $Z\left(\mathbf{y}\right)=\int \ell \left(\mathbf{y}\right|\mathbf{x})p_0\left(\mathbf{x}\right)d\mathbf{x}$ is the marginal likelihood.

In this scenario, we can be interested in computing a moment $h\left(\mathbf{x}\right)$ of the posterior, which is an expectation (hence an integral) of the following form:

$$\mu =E_{\tilde{\pi }}\left[h\left(\mathbf{x}\right)\right]=\int h\left(\mathbf{x}\right)\tilde{\pi }\left(\mathbf{x}\right|\mathbf{y})d\mathbf{x},$$

(14)

where $h:{\mathbb{R}}^{d_x}\to \mathbb{R}$. In connection with the notation of this paper, $f\left(\mathbf{x}\right)=h\left(\mathbf{x}\right)\tilde{\pi }\left(\mathbf{x}\right|\mathbf{y})$. In the context of Bayesian inference, there are several ways to select the techniques, ${\left\{p_i\left(x\right)\right\}}_{i=1}^n$. For instance, they can be deterministically set a priori, e.g., as Laplace approximations at different points [23]. Alternatively, they can be adapted over iterations via iterative stochastic mechanisms [24]. The latter technique is called adaptive importance sampling (AIS) and its literature is vast (see [15] for a recent review).

3. Generalized Multiple Importance Sampling Balance Heuristic Estimator

In classic balance heuristic, both the number of samples ${\{n_i={\alpha }_{i}N\}}_{i=1}^n$ and the weights of the mixture of pdfs, ${\left\{{\alpha }_i\right\}}_{i=1}^n$, are given by the same set of parameters. Let us consider now the estimator of Equation (4), where we relax the dependence between the number of samples $n_i$, and the associated coefficient ${\alpha }_i$, i.e., now $n_i={\beta }_{i}N$, ${\beta }_i>0$, ${\sum }_{i=1}^n{\beta }_i=1$, where in general ${\alpha }_i\ne {\beta }_i$ (otherwise, we recover F). We now define the estimator

$$\begin{array}{ccc}\hfill G& =& \sum _{i=1}^n\frac{{\alpha }_i}{n_i}\sum _{j=1}^{n_i}\frac{f\left({\mathbf{X}}_{i,j}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left({\mathbf{X}}_{i,j}\right)}\hfill \end{array}$$

(15)

$$\begin{array}{ccc}& =& \frac{1}{N}\sum _{i=1}^n\frac{{\alpha }_i}{{\beta }_i}\sum _{j=1}^{n_i}\frac{f\left({\mathbf{X}}_{i,j}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left({\mathbf{X}}_{i,j}\right)}.\hfill \end{array}$$

(16)

Note that G is a particular case of Z, with weights $w_i=\frac{{\alpha }_i{p}_i\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left({\mathbf{X}}_{i,j}\right)}$ in Equation (1). Note also that the balance heuristic F is a particular case of G, i.e., in general we do not impose the restriction of ${\alpha }_i=\frac{n_i}{N}$. Although the allocation of samples will be different for F and G, we remark that both estimators require the same number of simulations and evaluations. Finally, we note that a close examination of (16) allows us to re-interpret the importance weight associated to each sample. The mixture parametrized with ${\{n_i={\alpha }_{i}N\}}_{i=1}^n$ appears in its denominator, but since this mixture does not reflect the sampling procedure (the true mixture proposal in sampling is the one parametrized by ${\{n_i={\beta }_{i}N\}}_{i=1}^n$), the importance weight of a sample simulated from the technique ${\beta }_i$, is corrected by the factor ${\alpha }_i/{\beta }_i$ in order to account for this mismatch.

Estimator G can be rewritten as $G={\sum }_{i=1}^n{\alpha }_i{G}_i$, where

$$\begin{array}{ccc}\hfill G_i& =& \frac{1}{n_i}\sum _{j=1}^{n_i}\frac{f\left({\mathbf{X}}_{i,j}\right))}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left({\mathbf{X}}_{i,j}\right))}.\hfill \end{array}$$

(17)

Note also that G depends of two sets of parameters, ${\left\{{\alpha }_i\right\}}_{i=1}^n and {\left\{{\beta }_i\right\}}_{i=1}^n$. In the particular case where ${\beta }_i={\alpha }_i,\forall i$, the estimator G becomes F. Let us consider the case with $n_i=1$. Then,

$$\begin{array}{ccc}\hfill G_i^{\prime }& =& \frac{f\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)},\hfill \end{array}$$

(18)

with expectation

$$\begin{array}{c}\hfill E\left[G_i^{\prime }\right]=\int \frac{f\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)}d\mathbf{x}\equiv {\mu }_i^{\prime },\end{array}$$

(19)

and variance

$$\begin{array}{ccc}\hfill {{\sigma }^{\prime }}_i^2& =& \int \frac{f^2\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)\right)}^2}d\mathbf{x}-{\left({\mu }_i^{\prime }\right)}^2.\hfill \end{array}$$

(20)

Observe that $E\left[G_i\right]=E\left[G_i^{\prime }\right]={\mu }_i^{\prime }$, and $V\left[G_i\right]=\frac{1}{n_i}V\left[G_i^{\prime }\right]=\frac{1}{n_i}{{\sigma }^{\prime }}_i^2$.

Theorem 1.

For any set of weights ${\left\{{\alpha }_i\right\}}_{i=1}^n$, such as ${\sum }_{i=1}^n{\alpha }_i=1$ and any set of weights ${\left\{{\beta }_i\right\}}_{i=1}^n$, such as ${\sum }_{i=1}^n{\beta }_i=1$, G is an unbiased estimator of μ.

Proof. 

The estimator G is unbiased, since

$$\begin{array}{}\mathrm{(21)}& \hfill E\left[G\right]& =\sum _{i=1}^n{\alpha }_{i}E\left[G_i\right]=\sum _{i=1}^n{\alpha }_i{\mu }_i^{\prime }=\sum _i{\alpha }_i\int \frac{f\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)}d\mathbf{x}\mathrm{(22)}& & =\int \frac{f\left(\mathbf{x}\right){\sum }_{i=1}^n{\alpha }_i{p}_i\left(\mathbf{x}\right)}{{\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)}d\mathbf{x}& & =\int f\left(\mathbf{x}\right)d\mathbf{x}\equiv \mu .\end{array}$$

□

The variance of G is given by

$$\begin{array}{c}\hfill V\left[G\right]=V\left[\sum _{i=1}^n{\alpha }_i{G}_i\right]=\sum _{i=1}^n{\alpha }_i^{2}V\left[G_i\right]=\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{n_i}.\end{array}$$

(23)

For the sake of the theoretical analysis, we define $G^1$, a normalized version of G with $N=1$ (see Section 2.1), with variance

$$\begin{array}{c}\hfill V\left[G^1\right]=\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}.\end{array}$$

(24)

Next we study four special cases of estimator $G^1$.

Remark 1.

We could also consider the one-sample estimator $\mathcal{G}$, randomized version of G. However, $\mathcal{G}$ is a particular case of the general estimator $\mathcal{Z}$, and we have seen in Section 2.2 that the optimal case for $\mathcal{Z}$ is when $\mathcal{Z}\equiv \mathcal{F}$, thus it only makes sense to consider the extension G of the multi-sample estimator F.

Remark 2.

Grittmann et al. estimator [19], can be interpreted as a G estimator where ${\alpha }_i\propto {\beta }_i\frac{m_i^2}{v_i}={\beta }_i\frac{v_i+{\mu }^2}{v_i}$, where $v_i$ is the variance of i independent technique and $m_i^2=v_i+{\mu }^2$ the second moment,

$$v_i=\int \frac{f{\left(\mathbf{x}\right)}^2}{p_i\left(x\right)}d\mathbf{x}.$$

(25)

3.1. Case 1: ${\alpha }_i={\beta }_i$, $\forall i$

In this particular case, the estimator G reverts to F. The variance is

$$\begin{array}{c}\hfill V\left[F^1\right]=\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i^2,\end{array}$$

(26)

by simple substitution in Equation (24). We aim at finding the optimal ${\left\{{\alpha }_i^{*}\right\}}_{i=1}^n$ such the variance of Equation (26) is minimized.

Theorem 2.

The optimal estimator $F^{*}$ in terms of variance is achieved when $\forall j\in \{1,\dots ,n\}$, the following values are equal

$$\begin{array}{c}\hfill {{\sigma }^{\prime }}_j^2+2{{\mu }_j^{\prime }}^2-2\sum _{i=1}^n{\alpha }_i^{*}{\mu }_i^{\prime }\int \frac{f\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)p_j\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k^{*}{p}_k\left(\mathbf{x}\right)\right)}^2}d\mathbf{x}.\end{array}$$

(27)

See Appendix B for a proof.

Compare Equation (27) with the condition for optimal ${\mathcal{F}}^{*}$, which is [9] that the following expression is equal $\forall j\in \{1,\dots ,n\}$,

$$\begin{array}{c}\hfill {{\sigma }^{\prime }}_j^2+{{\mu }_j^{\prime }}^2.\end{array}$$

(28)

The optimal ${\left\{{\alpha }_i^{*}\right\}}_{i=1}^n$ solutions are in general different for F, Equation (27), and for ${\mathcal{F}}^{*}$, Equation (28), although observe that, when for ${\left\{{\alpha }_i^{*}\right\}}_{i=1}^n$ all i the ${\mu }_i^{\prime }$ are equal, then the optimality condition Equation (27) implies that $\forall i\in \{1,\dots ,n\}$ the ${{\sigma }^{\prime }}_i^2$ are equal too, and thus the optimal solutions for F and $\mathcal{F}$ are the same, in concordance with being $V\left[F\right]=V\left[\mathcal{F}\right]$ for this special case, see Section 2.2 and Appendix A.

Another interesting result regarding the ${\mu }_i^{\prime }$ values is the following theorem,

Theorem 3.

If $V\left[F\right(\mathcal{F}\left)\right]=0$ then for all i, ${\mu }_i^{\prime }=\mu$.

Proof. 

As $V\left[F\right]\le V\left[F\right(\mathcal{F}\left)\right]$ then $V\left[F\right]={\sum }_i^n{\alpha }_i{{\sigma }_i^{\prime }}^2=0$, and thus for all i, ${\sigma }_i^{\prime }=0$; however, $V\left[F\right(\mathcal{F}\left)\right]=0$ is obviously a local (and global) minimum for $V\left[F\right(\mathcal{F}\left)\right]$ (a convex function) and thus from Equation (28) for all i, ${{\sigma }^{\prime }}_i^2+{{\mu }_i^{\prime }}^2$ are equal and hence the result. Alternatively, we know that $V\left[F\right]=V\left[F\right(\mathcal{F}\left)\right]$⇔ for all i, all ${\mu }_i^{\prime }=\mu$. □

3.2. Case 2: Fixed ${\left\{{\alpha }_i\right\}}_{i=1}^n$

Consider now that ${\left\{{\alpha }_i\right\}}_{i=1}^n$ are fixed, and hence also ${\left\{{{\sigma }^{\prime }}_i^2\right\}}_{i=1}^n$, are fixed. Applying Cauchy–Schwartz inequality to sequences ${\left\{\sqrt{{\beta }_i}\right\}}_{i=1}^n$ and ${\left\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{\sqrt{{\beta }_i}}\right\}}_{i=1}^n$ we obtain

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i{\sigma }_i^{\prime }\right)}^2\le \left(\sum _{i=1}^n{\beta }_i\right)\left(\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right).\end{array}$$

(29)

As the left term is fixed, equality in Equation (29) will give the minimum of right term, being this term $V\left[G^1\right]$ as ${\sum }_{i=1}^n{\beta }_i=1$; however, equality can only happen when the right term sequences are proportional, this is, for all i, $\sqrt{{\beta }_i}\propto \frac{{\alpha }_i{{\sigma }^{\prime }}_i}{\sqrt{{\beta }_i}}$, and thus the optimal ${\left\{{\beta }_i\right\}}_{i=1}^n$ are given by

$$\begin{array}{c}\hfill {\beta }_i^{*}\propto {\alpha }_i{\sigma }_i^{\prime },i=1,...n,\end{array}$$

(30)

and the optimal (minimum) variance is

$$\begin{array}{c}\hfill V\left[G^{1*}\right]={\left(\sum _{i=1}^n{\alpha }_i{\sigma }_i^{\prime }\right)}^2.\end{array}$$

(31)

Theorem 4.

Given an estimator F with ${\left\{{\alpha }_i\right\}}_{i=1}^n$ values, we can always find a better estimator G by sampling as ${\beta }_i^{*}\propto {\alpha }_i{\sigma }_i^{\prime }$, which is strictly better whenever not all ${{\sigma }^{\prime }}_i$ are equal.

Proof. 

Observe that, by applying Cauchy–Schwartz inequality to the sequences ${\left\{\sqrt{{\alpha }_i}\right\}}_{i=1}^n$ and ${\left\{\sqrt{{\alpha }_i}{{\sigma }^{\prime }}_i\right\}}_{i=1}^n$,

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i{\sigma }_i^{\prime }\right)}^2\le \left(\sum _{i=1}^n{\alpha }_i\right)\left(\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i^2\right),\end{array}$$

(32)

but the left-hand side of inequality is $V\left[G^{1*}\right]$, and as ${\sum }_{i=1}^n{\alpha }_i=1$, the right-hand side is $V\left[F^1\right]$. Hence, for the optimal values ${\left\{{\beta }_i^{*}\right\}}_{i=1}^n$ as in Equation (30), the estimator $G^{*}$ always outperforms the estimator F (When comparing estimators F and G, we consider, unless explicitly stated, the same set of ${\left\{{\alpha }_i\right\}}_{i=1}^n$ values).

Equality in Equation (32) only happens when for all i, $\sqrt{{\alpha }_i}\propto \sqrt{{\alpha }_i}{{\sigma }^{\prime }}_i$, i.e., when all ${{\sigma }^{\prime }}_i$ are equal. In that case we have for all i${\beta }_i^{*}={\alpha }_i$, and we revert to the F estimator. □

Remark 3.

From the inequality $\left({\sum }_{i=1}^n\frac{1}{n}{{\sigma }^{\prime }}_i^2\right)\le n{\left({\sum }_{i=1}^n\frac{1}{n}{\sigma }_i^{\prime }\right)}^2$, the maximum possible acceleration by using the optimal ${\beta }_i^{*}$ values when for all i, ${\alpha }_i=1/n$, is equal to n (as observed in [7]). This acceleration would be approached when there is an index k such that for all $i\ne k$, ${\sigma }_k^{\prime }>>{\sigma }_i^{\prime }$. In general, the more different the ${\sigma }_i^{\prime }$ are, the higher the acceleration.

A particular case of Theorem 4 is when ${\alpha }_i=\frac{1}{n}$, $\forall i$, where $V\left[G^1\right]$ becomes

$$\begin{array}{c}\hfill V\left[G^1\right]=\frac{1}{n^2}\sum _{i=1}^n\frac{{{\sigma }^{\prime }}_i^2}{{\beta }_i}.\end{array}$$

(33)

This case was introduced in Section 4 of [7]. It was shown that this estimator is provably better than F with ${\alpha }_i=1/n,\forall i$ when

$$\begin{array}{c}\hfill {\beta }_i^{*}\propto {\sigma }_i^{\prime },i=1,\dots ,n,\end{array}$$

(34)

which is the optimal case of Equation (33). Examples showing the improvement obtained were also given in [7].

Optimal Efficiency

Let us now take into account that the cost of each sampling technique can be different, as it is usually considered in the literature [25]. Let us denote the cost of sampling technique i as $c_i$. The inverse of efficiency for the estimator G is given by

$$\begin{array}{c}\hfill E_G^{-1}=\left(\sum _{i=1}^n{\beta }_i{c}_i\right)\left(\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right).\end{array}$$

(35)

Note that this quantity represents the total cost multiplied by the variance of the estimator. Using Cauchy–Schwartz inequality with the sequences ${\left\{\sqrt{{\beta }_i{c}_i}\right\}}_{i=1}^n$ and ${\left\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{\sqrt{{\beta }_i}}\right\}}_{i=1}^n$, we obtain

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i{\sigma }_i^{\prime }\sqrt{c_i}\right)}^2\le \left(\sum _{i=1}^n{\beta }_i{c}_i\right)\left(\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right).\end{array}$$

(36)

The optimal sampling rates (for maximizing the efficiency) are those that yield Equation (36) as an equality, which happens when ${\beta }_i^{*}\propto \frac{{\alpha }_i{{\sigma }^{\prime }}_i}{\sqrt{c_i}}$. Observe that, using again the Cauchy–Schwartz theorem, with sequences ${\left\{\sqrt{{\alpha }_i{c}_i}\right\}}_{i=1}^n$ and ${\left\{\sqrt{{\alpha }_i}{{\sigma }^{\prime }}_i\right\}}_{i=1}^n$, we obtain

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i{\sigma }_i^{\prime }\sqrt{c_i}\right)}^2\le \left(\sum _{i=1}^n{\alpha }_i{c}_i\right)\left(\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i^2\right),\end{array}$$

(37)

where the left-hand side is $E_G^{-1}$ with the optimal sampling rates, and the right-hand side is $E_F^{-1}$. Note that equality only happens when for all i, $\sqrt{{\alpha }_i{c}_i}=\sqrt{{\alpha }_i}{{\sigma }^{\prime }}_i$, i.e., $c_i\propto {{\sigma }^{\prime }}_i^2$, where we have ${\beta }_i^{*}={\alpha }_i$ and we revert to the F estimator. This is summarized in the following theorem.

Theorem 5.

Given an estimator F with ${\left\{{\alpha }_i\right\}}_{i=1}^n$ values, and sampling costs ${\left\{c_i\right\}}_{i=1}^n$, we can always find a strictly more efficient estimator $G^{*}$ when for all i, ${\beta }_i^{*}\propto {\alpha }_i\frac{{\sigma }_i^{\prime }}{\sqrt{c_i}}$, and not all $c_i\propto {{\sigma }^{\prime }}_i^2$.

3.3. Case 3: Fixed ${\left\{{\beta }_i\right\}}_{i=1}^n$

Theorem 6.

Consider now a fixed set ${\left\{{\beta }_i\right\}}_{i=1}^n$. The optimal set ${\left\{{\alpha }_i^{*}\right\}}_{i=1}^n$ can be found using Lagrange multipliers with target function

$$\mathsf{\Lambda }({\left\{{\alpha }_i\right\}}_{i=1}^n,\lambda )=\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}+\lambda \left(\sum _{i=1}^n{\alpha }_i-1\right).$$

Observe that the ${{\sigma }^{\prime }}_i^2$ values depend on the ${\left\{{\alpha }_i\right\}}_{i=1}^n$ values. The optimal values are those that obey, for all j, the following expression

$$\begin{array}{cc}\hfill & \frac{{\alpha }_j^{*}{{\sigma }^{\prime }}_j^2}{{\beta }_j}=\sum _{i=1}^n\frac{{\alpha }_i^{*2}}{{\beta }_i}\times \hfill \\ \hfill & \left(\int \frac{f^2\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)p_j\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k^{*}{p}_k\left(\mathbf{x}\right)\right)}^3}d\mathbf{x}-{\mu }_i^{\prime }\int \frac{f\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)p_j\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k^{*}{p}_k\left(\mathbf{x}\right)\right)}^2}d\mathbf{x}\right).\hfill \end{array}$$

(38)

Proof. 

The derivation can be found in the Appendix C. □

Note that in the general case, the optimal ${\alpha }_i^{*}\ne {\beta }_i$.

Moreover, aside from the optimal values ${\left\{{\alpha }_j^{*}\right\}}_{j=1}^n$, we can find cases where $V\left[G^1\right]\le V\left[F^1\right]$. Given ${\left\{{\beta }_i\right\}}_{i=1}^n$, from Theorem 4 we have that if there exist ${\left\{{\alpha }_i\right\}}_{i=1}^n$ such that ${\beta }_i\propto {\alpha }_i{\sigma }_i^{\prime },i=1,...n$, then $V\left[G^1\right]\le V\left[F^1\right]$. From Theorem 5, we have that if there exist ${\left\{{\alpha }_i\right\}}_{i=1}^n$ such that ${\beta }_i\propto {\alpha }_i\frac{{\sigma }_i^{\prime }}{\sqrt{c_i}},\forall i=1,...n$, then the estimator $G^1$ is more efficient than estimator $F^1$.

Observe that from Equation (38), a necessary and sufficient condition for the optimal solution ${\alpha }_i^{*}$ to be such that ${\alpha }_i^{*}={\beta }_i$ for all i is that Equation (39) holds (see Appendix C),

$$\begin{array}{c}\hfill {{\mu }_j^{\prime }}^2=\sum _{i=1}^n{\alpha }_i{\mu }_i^{\prime }\int \frac{f\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)p_j\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k{p}_k\left(\mathbf{x}\right)\right)}^2}d\mathbf{x},\end{array}$$

(39)

which is satisfied when for all i, ${\mu }_i^{\prime }=\mu$. In this particular case, ${\mu }_i^{\prime }=\mu$, $G\equiv F$, and $V\left[G\right]=V\left[F\right]=V\left[\mathcal{F}\right]$. In other words, when choosing a sampling rate of $\left\{{\beta }_i\right\}=\left\{{\alpha }_i^{\star }\right\}$, where $\left\{{\alpha }_i^{\star }\right\}$ are such that for all i, ${\mu }_i^{\prime }=\mu$, the optimal $\left\{{\alpha }_i\right\}$ parameters for G estimator are precisely $\left\{{\alpha }_i^{\star }\right\}$. That is, we can not improve the $F\left(\left\{{\alpha }_i^{\star }\right\}\right)$ estimator with the G estimator with a suitable selection of $\left\{{\beta }_i\right\}$ parameters. Observe that it is similar to the case studied in Section 3.2 for the $\left\{{\alpha }_i\right\}$ that make all ${{\sigma }^{\prime }}_i$ equal.

Equation (39) might have additional solutions to the one (if exists) that makes all ${\mu }_i^{\prime }$ equal, in this case $V\left[G\right]=V\left[F\right]$ too but $V\left[F\right]\ne V\left[\mathcal{F}\right]$.

3.4. Case 4: ${\beta }_i=1/n,\forall i$

In the case when for all i, ${\beta }_i=1/n$, the variance becomes

$$\begin{array}{c}\hfill V\left[G^1\right]=\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{1/n}=n\sum _{i=1}^n{\alpha }_i^2{{\sigma }^{\prime }}_i^2.\end{array}$$

(40)

Note that this is a usual case in the MIS literature strategies [2,4,5,6] and the adaptive IS (AIS) literature [11,12,13,14,15,24], since all the techniques have the same number of counts. By setting in Equation (38) for all i, ${\beta }_i=1/n$, and if we can optimize ${\left\{{\alpha }_j\right\}}_{j=1}^n$, we can find the minimum variance values ${\left\{{\alpha }_j^{*}\right\}}_{j=1}^n$. Thus the minimum variance $V\left[G^{*}\right]$ corresponds, if the solution exists, to the values ${\left\{{\alpha }_j^{*}\right\}}_{j=1}^n$ that satisfy

$$\begin{array}{cc}\hfill & {\alpha }_j^{*}{{\sigma }^{\prime }}_j^2=\sum _{i=1}^n{\alpha }_i^{*2}\hfill \\ \hfill & \times \left(\int \frac{f^2\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)p_j\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k^{*}{p}_k\left(\mathbf{x}\right)\right)}^3}d\mathbf{x}-{\mu }_i^{\prime }\int \frac{f\left(\mathbf{x}\right)p_i\left(\mathbf{x}\right)p_j\left(\mathbf{x}\right)}{{\left({\sum }_{k=1}^n{\alpha }_k^{*}{p}_k\left(\mathbf{x}\right)\right)}^2}d\mathbf{x}\right).\hfill \end{array}$$

(41)

The corresponding variance $V\left[G^{*}\right]$ will be less or equal than the variance of $V\left[G\right]$ for all ${\left\{{\alpha }_j\right\}}_{j=1}^n$, and in particular for ${\alpha }_i=1/n$, where G converts into F as ${\beta }_i={\alpha }_i=1/n$, the classic balance heuristic estimator, thus $V\left[G^{*}(\left\{{\alpha }_i^{*}\right\},\{{\beta }_i=1/n\})\right]\le V\left[F\left(\{{\alpha }_i=1/n\}\right)\right]$. Observe that in general $\{{\alpha }_i^{*}\ne 1/n\}$, because substituting $\{{\alpha }_i^{*}=1/n\}$ in Equation (41) the resulting equation Equation (39) is not satisfied in general.

Comparing $V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]$, $V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right]$ and $V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]$

Let us see first when $V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\le V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right)]$ and viceversa.

Theorem 7.

If the sequences $\left\{{\beta }_i\right\},\{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\}$ are comonotonic, i.e., ${\beta }_i\le {\beta }_j\Rightarrow \frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\le \frac{{\alpha }_j^2{{\sigma }^{\prime }}_j^2}{{\beta }_j}$, then

$$\begin{array}{c}\hfill V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\le V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right].\end{array}$$

(42)

Proof. 

Applying Theorem 9, Corollary 5 from [26],

$$\begin{array}{ccc}\hfill \sum _{i=1}^n\frac{1}{n}\left(\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right)& \le & \sum _{i=1}^n{\beta }_i\left(\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right),\hfill \end{array}$$

(43)

obtaining the inequality in Equation (42). See also [10,27]. □

Observe that equality in Equation (43) happens when for all i, ${\beta }_i\propto {\alpha }_i^2{{\sigma }^{\prime }}_i^2$.

In the same way we can proof the inverse case as follows,

Theorem 8.

If the sequences $\left\{{\beta }_i\right\}$ and $\{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\}$ are countermonotonic, i.e., ${\beta }_i\le {\beta }_j\Rightarrow \frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\ge \frac{{\alpha }_j^2{{\sigma }^{\prime }}_j^2}{{\beta }_j}$, then

$$\begin{array}{c}\hfill V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\ge V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right)].\end{array}$$

(44)

The proof implies for countermonotonicity a change of sign in the inequality in Theorem 9, Corollary 5 from [26].

Let us compare now $V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right]$ and $V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]$.

Theorem 9.

If the sequences $\left\{{\alpha }_i\right\},\{{\alpha }_i{{\sigma }^{\prime }}_i^2\}$ are countermonotonic, i.e., ${\alpha }_i\le {\alpha }_j\Rightarrow {\alpha }_i{{\sigma }^{\prime }}_i^2\ge {\alpha }_j{{\sigma }^{\prime }}_j^2$, then

$$\begin{array}{c}\hfill V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right]\le V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right].\end{array}$$

(45)

Proof. 

The proof is as above, making use of Theorem 9, Corollary 5 from [26],

$$\begin{array}{ccc}\hfill \sum _{i=1}^n\frac{1}{n}\left({\alpha }_i{{\sigma }^{\prime }}_i^2\right)& \ge & \sum _{i=1}^n{\alpha }_i\left({\alpha }_i{{\sigma }^{\prime }}_i^2\right),\hfill \end{array}$$

(46)

obtaining the inequality in Equation (45). □

Observe that equality in Equation (46) happens when for all i, ${\alpha }_i\propto 1/{{\sigma }^{\prime }}_i^2$. In the same way we can prove the reverse case of Theorem 9.

Theorem 10.

If the sequences $\left\{{\alpha }_i\right\},\{{\alpha }_i{{\sigma }^{\prime }}_i^2\}$ are comonotonic, i.e., ${\alpha }_i\le {\alpha }_j\Rightarrow {\alpha }_i{{\sigma }^{\prime }}_i^2\le {\alpha }_j{{\sigma }^{\prime }}_j^2$, then

$$\begin{array}{c}\hfill V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right]\ge V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right].\end{array}$$

(47)

As a direct consequence of Theorems 9 and 10 we have the following corollary:

Corollary 1.

When ${\alpha }_i\propto 1/{{\sigma }^{\prime }}_i^{2+ϵ}$, if $-2\le ϵ\le 0$ then $V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right]\le V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]$ holds, and if $ϵ\ge 0$ then $V\left[G^1(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\right]\ge V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]$ holds.

By setting $ϵ=-1$ in Corollary 1, the products ${\alpha }_i{{\sigma }^{\prime }}_i$ are equal for all i, which is equivalent to saying that ${\alpha }_i{{\sigma }^{\prime }}_i\propto 1/n={\beta }_i$. Observe that this corresponds to the optimal solution $\left\{{\beta }_j^{*}\right\}$ given by Equation (30). Observe now that the inequality for this optimal solution, $V\left[G^1\right]\le V\left[F^1\right]$, can be obtained by successive application of Theorems 7 and 9.

Theorem 11.

If the sequences $\left\{{\alpha }_i\right\},\left\{{\alpha }_i{{\sigma }^{\prime }}_i^2\right\}$ are countermonotonic, and the sequences $\left\{{\beta }_i\right\},\left\{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right\}$ are comonotonic, then

$$\begin{array}{c}\hfill V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\le V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right].\end{array}$$

(48)

Proof. 

The result is obtained by successive application of Theorems 7 and 9. □

Theorem 12.

If the sequences $\left\{{\alpha }_i\right\},\left\{{\alpha }_i{{\sigma }^{\prime }}_i^2\right\}$ are comonotonic, and the sequences $\left\{{\beta }_i\right\},\left\{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\right\}$ are countermonotonic, then

$$\begin{array}{c}\hfill V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\ge V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right].\end{array}$$

(49)

As a direct consequence of Theorems 11 and 12 we have the following corollary:

Corollary 2.

When ${\alpha }_i\propto 1/{{\sigma }^{\prime }}_i^{2+ϵ}$ and ${\beta }_i\propto {\left({\alpha }_i{{\sigma }^{\prime }}_i\right)}^{\gamma }$, if $-2\le ϵ\le 0$ and $0<\gamma \le 2$ then $V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\le V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]$ holds, and if $ϵ\ge 0$ and $\gamma >2$ then $V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\ge V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]$ holds.

For example, take ${\alpha }_i\propto 1/{{\sigma }^{\prime }}_i^2$, and ${\beta }_i\propto {\alpha }_i{{\sigma }^{\prime }}_i\propto 1/{{\sigma }^{\prime }}_i$, then $V\left[G^1(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]\le V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right].$

Finally, observe that we do not have a sufficient condition for $V\left[F^1\left(\left\{{\alpha }_i\right\}\right)\right]\le V\left[F^1\left(\{{\alpha }_i=1/n\}\right)\right]$, only there are heuristics [7,9,10,28].

3.5. Case 5: General Case

Let us consider the global optimum for G estimator, this is, without imposing any constraint to the parameters $\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\}$. For this we should optimize the lagrangian function

$$\mathsf{\Lambda }({\left\{{\alpha }_i\right\}}_{i=1}^n,{\left\{{\beta }_i\right\}}_{i=1}^n,{\lambda }_1,{\lambda }_2)=\sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}+{\lambda }_1\left(\sum _{i=1}^n{\alpha }_i-1\right)+{\lambda }_2\left(\sum _{i=1}^n{\beta }_i-1\right).$$

Differentiating with respect to ${\alpha }_i$,

$$\frac{\partial \mathsf{\Lambda }({\left\{{\alpha }_i\right\}}_{i=1}^n,{\left\{{\beta }_i\right\}}_{i=1}^n,{\lambda }_1,{\lambda }_2)}{{\partial }_{{\alpha }_j}}=0,$$

we obtain Equation (38).

Differentiating with respect to ${\beta }_i$,

$$\frac{\partial \mathsf{\Lambda }({\left\{{\alpha }_i\right\}}_{i=1}^n,{\left\{{\beta }_i\right\}}_{i=1}^n,{\lambda }_1,{\lambda }_2)}{{\partial }_{{\beta }_j}}=0,$$

we obtain Equation (30). Thus, the optimal parameters ${\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$ and ${\left\{{\beta }_i^{\star }\right\}}_{i=1}^n$ will solve for Equations (38) and (30) simultaneously.

Let us consider now the ${\left\{{\alpha }_i^{†}\right\}}_{i=1}^n$, if they exist, such that for all i, ${\sigma }_i^{\prime }$ are equal. Then, for Equation (30) to hold for ${\left\{{\alpha }_i^{†}\right\}}_{i=1}^n$, we have that ${\beta }_i^{†}={\alpha }_i^{†}$ for all $i=1,\dots ,n$. From Equation (38), taking ${\left\{{\beta }_i^{†}\right\}}_{i=1}^n={\left\{{\alpha }_i^{†}\right\}}_{i=1}^n$, Equation (39) has to hold for all ${\alpha }_i^{†}$.

Furthermore, the holding of Equation (39), together with the hypothesis that for all j, ${{\sigma }^{\prime }}_j^2$ are equal, allows Equation (27) to hold, and then ${\left\{{\alpha }_i^{†}\right\}}_{i=1}^n$ is an optimum for F too, $V\left[F\right]=V\left[G\right]$. If in addition, if $\left\{{\alpha }_i^{†}\right\}$ are such that they make all ${\left\{{\mu }^{\prime }\right\}}_{j=1}^n$ equal, then they would be optimal for $\mathcal{F}$ too, $V\left[\mathcal{F}\right]=V\left[F\right]=V\left[G\right]$.

4. Singular Solutions

The variance of $\mathcal{F}$ can be written as [7]

$$\begin{array}{ccc}\hfill V\left[\mathcal{F}\right]& =& \sum _{i=1}^n{\alpha }_i({{\sigma }^{\prime }}_i^2+{{\mu }^{\prime }}_i^2-{\mu }^2)=\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i^2+\sum _{i=1}^n{\alpha }_i{{\mu }^{\prime }}_i^2-{\mu }^2.\hfill \\ \hfill & =& \sum _{i=1}^n{\alpha }_i({{\sigma }^{\prime }}_i^2+{{\mu }^{\prime }}_i^2)-{\mu }^2.\hfill \end{array}$$

(50)

The optimal parameters ${\left\{{\alpha }_i\right\}}_{i=1}^n$ happen when for all i, ${{\sigma }^{\prime }}_i^2+{{\mu }^{\prime }}_i^2$ equal, which gives a local minimum for $V\left[\mathcal{F}\right]$ (in this case a global minimum too, as $V\left[\mathcal{F}\right]$ is convex). Observe that this makes, in the third inequality, all terms factored by $\left\{{\alpha }_i\right\}$ equal, and also corresponds to equality in the Cauchy–Schwartz inequality

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i\sqrt{({{\sigma }^{\prime }}_i^2+{{\mu }^{\prime }}_i^2)}\right)}^2\le \left(\sum _{i=1}^n{\alpha }_i\right)\left(\sum _{i=1}^n{\alpha }_i({{\sigma }^{\prime }}_i^2+{{\mu }^{\prime }}_i^2)\right),\end{array}$$

(51)

with equality only when, for all i, ${{\sigma }^{\prime }}_i^2+{{\mu }^{\prime }}_i^2$ are equal.

If there exist ${\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$ values such that for all i, ${\mu }_i^{\prime }=\mu$ and all ${{\sigma }^{\prime }}_i$ are equal, these values correspond to the global optimum $V\left[{\mathcal{F}}^{\star }\right]=V\left[F^{\star }\right]=V\left[G^{\star }\right]$, and $F^{\star }\equiv G^{\star }$, but $F^{\star }\not\equiv {\mathcal{F}}^{\star }$. Suppose there exist ${\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$ such that for all i, ${\mu }_i^{\prime }=\mu$, then we can not improve the F estimator with a G estimator with a suitable selection of ${\left\{{\beta }_i\right\}}_{i=1}^n$ parameters, because the optimal $\left\{{\alpha }_i\right\}$ parameters for the G estimator when ${\left\{{\beta }_i\right\}}_{i=1}^n={\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$ are precisely $\left\{{\alpha }_i^{\star }\right\}$. This is, the optimal of $V\left[G({\left\{{\alpha }_i\right\}}_{i=1}^n,{\left\{{\beta }_i\right\}}_{i=1}^n={\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n)\right]$ is for ${\left\{{\alpha }_i\right\}}_{i=1}^n={\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$.

Let us see now $V\left[F\right]$:

$$V\left[F\right]=\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i^2.$$

Suppose there exist ${\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$ such that all ${\sigma }_i^{\prime }$ are equal, then we can not improve the F estimator with a suitable selection of ${\left\{{\beta }_i\right\}}_{i=1}^n$ parameters for the G estimator, because the optimal ${\left\{{\beta }_i^{\star }\right\}}_{i=1}^n$ are such that ${\left\{{\beta }_i^{\star }\right\}}_{i=1}^n={\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$. This is, the optimal of $V\left[G({\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n,{\left\{{\beta }_i\right\}}_{i=1}^n)\right]$ is for ${\left\{{\beta }_i\right\}}_{i=1}^n={\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n$. Observe that by Cauchy–Schwartz

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i\right)}^2\le \left(\sum _{i=1}^n{\alpha }_i\right)\left(\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i^2\right),\end{array}$$

(52)

with equality only when, for all i, all the ${{\sigma }^{\prime }}_i^2$ are equal. Observe thus that both the solutions for ${\mu }_i^{\prime }$ equal and for ${{\sigma }^{\prime }}_i^2$ equal are singular solutions for G, where it reverts to F.

Let us consider now $V\left[G\right]$:

$$\begin{array}{c}\hfill V\left[G\right]=\sum _{i=1}^n\frac{{\alpha }_i^2}{{\beta }_i}{{\sigma }^{\prime }}_i^2=\sum _{i=1}^n{\alpha }_i\left(\frac{{\alpha }_i}{{\beta }_i}{{\sigma }^{\prime }}_i^2\right)=\sum _{i=1}^n{\beta }_i\left(\frac{{\alpha }_i^2}{{\beta }_i^2}{{\sigma }^{\prime }}_i^2\right).\end{array}$$

(53)

In the second equality in Equation (53), the solution for all i, $\frac{{\alpha }_i}{{\beta }_i}{{\sigma }^{\prime }}_i^2$ equal, or ${\beta }_i\propto {\alpha }_i{{\sigma }_i^{\prime }}^2$, makes $V\left[G\right]=V\left[F\right]$, because then for all i, ${\alpha }_i{{\sigma }^{\prime }}_i^2=K{\beta }_i$, and substituting in the expressions for $V\left[G\right],V\left[F\right]$ we have $V\left[G\right]=V\left[F\right]=K$, but $G\not\equiv F$. By Cauchy–Schwartz,

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n\sqrt{\frac{{\alpha }_i^3{{\sigma }^{\prime }}_i^2}{{\beta }_i}}\right)}^2\le \left(\sum _{i=1}^n{\alpha }_i\right)\left(\sum _{i=1}^n\frac{{\alpha }_i^2}{{\beta }_i}{{\sigma }^{\prime }}_i^2\right),\end{array}$$

(54)

with equality only when, for all i, $\frac{{\alpha }_i}{{\beta }_i}{{\sigma }^{\prime }}_i^2$ are equal.

In the third equality in Equation (53), the solution for all i, $\frac{{\alpha }_i^2}{{\beta }_i^2}{{\sigma }^{\prime }}_i^2$, equal, or ${\beta }_i\propto {\alpha }_i{\sigma }_i^{\prime }$, gives the optimal solution for $V\left[G\right]$ when $\left\{{\alpha }_i\right\}$ are fixed, Equation (30). By Cauchy–Schwartz,

$$\begin{array}{c}\hfill {\left(\sum _{i=1}^n{\alpha }_i{{\sigma }^{\prime }}_i\right)}^2\le \left(\sum _{i=1}^n{\beta }_i\right)\left(\sum _{i=1}^n\frac{{\alpha }_i^2}{{\beta }_i}{{\sigma }^{\prime }}_i^2\right),\end{array}$$

(55)

with equality only when, for all i, $\frac{{\alpha }_i^2}{{\beta }_i^2}{{\sigma }^{\prime }}_i^2$ are equal.

We summarize in

Table 2. Singular parameter values. ${\left\{{\alpha }_i^{\star }\right\}}_{i=1}^n,{\left\{{\beta }_i^{\star }\right\}}_{i=1}^n$ are solutions of the equations in first column and same row.

for all i, ${{\sigma }_i^{\prime }}^2+{{\mu }^{\prime }}_i^2$ equal	$V\left[\mathcal{F}\left(\left\{{\alpha }_i^{\star }\right\}\right)\right]$ global minimum
for all i, ${{\sigma }_i^{\prime }}^2$ equal and ${{\mu }^{\prime }}_i$ equal	$V\left[\mathcal{F}\left(\left\{{\alpha }_i^{\star }\right\}\right)\right]=V\left[F\left(\left\{{\alpha }_i^{\star }\right\}\right)\right]=V\left[G(\left\{{\alpha }_i^{\star }\right\},\left\{{\beta }_i\right\}=\left\{{\alpha }_i^{\star }\right\})\right]$ global minimum, $F\equiv G$, but $F\not\equiv \mathcal{F}$
for all i, ${\mu }_i^{\prime }$ equal	$V\left[\mathcal{F}\left(\left\{{\alpha }_i^{\star }\right\}\right)\right]=V\left[F\left(\left\{{\alpha }_i^{\star }\right\}\right)\right]$; optimal of $V\left[G(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\}=\left\{{\alpha }_i^{\star }\right\})\right]$ is for $\left\{{\alpha }_i\right\}=\left\{{\alpha }_i^{\star }\right\}$
for all i, ${\sigma }_i^{\prime }$ equal	optimal of $V\left[G(\left\{{\alpha }_i^{\star }\right\},\left\{{\beta }_i\right\})\right]$ is for $\left\{{\beta }_i\right\}=\left\{{\alpha }_i^{\star }\right\}$, $F\equiv G$
for all i, $\frac{{\alpha }_i}{{\beta }_i}{{\sigma }^{\prime }}_i^2$ equal, or ${\beta }_i\propto {\alpha }_i{{\sigma }_i^{\prime }}^2$	$V\left[G(\left\{{\alpha }_i^{\star }\right\},\left\{{\beta }_i^{\star }\right\})\right]=V\left[F\left(\left\{{\alpha }_i^{\star }\right\}\right)\right]$
for all i, $\frac{{\alpha }_i^2}{{\beta }_i^2}{{\sigma }^{\prime }}_i^2$, equal, or ${\beta }_i\propto {\alpha }_i{\sigma }_i^{\prime }$	optimal of $V\left[G(\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\})\right]$ when $\left\{{\alpha }_i\right\}$ are fixed, $\forall \left\{{\alpha }_i\right\},V\left[G(\left\{{\alpha }_i\right\},\left\{{\beta }_i^{\star }\right\})\right]\le V\left[F\left(\left\{{\alpha }_i\right\}\right)\right]$

the different possibilities.

5. Relationship with ${\mathbf{\chi }}^{\mathbf{2}}$ Divergence: A Necessary and Sufficient Condition for $\mathit{V}\left[\mathit{G}\right]\le \mathit{V}\left[\mathit{F}\right]$

The variance of an importance sampling estimator can be written in terms of a ${\chi }^2$ divergence if $f\left(x\right)\ge 0$ [29,30]. As the $\mathcal{F}$ estimator can be seen as an importance sampling estimator with pdf $p\left(\mathbf{x}\right)={\sum }_i^n{p}_i\left(\mathbf{x}\right)$, its variance can be written as

$$\begin{array}{ccc}\hfill V\left[\mathcal{F}\right]& =& \int \frac{f{\left(\mathbf{x}\right)}^2}{{\sum }_i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)}d\mathbf{x}-{\mu }^2\hfill \\ \hfill & =& {\mu }^2\int \frac{\frac{f{\left(\mathbf{x}\right)}^2}{{\mu }^2}}{{\sum }_i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)}d\mathbf{x}-2{\mu }^2\int \frac{f\left(\mathbf{x}\right)}{\mu }d\mathbf{x}+{\mu }^2\int \left(\sum _i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)\right)d\mathbf{x}\hfill \\ \hfill & =& {\mu }^2\int \frac{{\left(\frac{f\left(\mathbf{x}\right)}{\mu }-\left({\sum }_i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)\right)\right)}^2}{{\sum }_i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)}d\mathbf{x}\hfill \\ \hfill & =& {\mu }^2{\chi }^2\left(\frac{f\left(\mathbf{x}\right)}{\mu },\sum _i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)\right).\hfill \end{array}$$

(56)

Observe that from the ${\chi }^2$ expression in Equation (56) it is clear that $V\left[\mathcal{F}\right]=0\iff f\left(\mathbf{x}\right)\propto {\sum }_i^n{\alpha }_i{p}_i\left(\mathbf{x}\right)$ except possibly in a zero measure set. As a consequence, for all i${\mu }_i^{\prime }=\mu$, and thus $V\left[F\right]=V\left[\mathcal{F}\right]=0$. As $V\left[F\right]\le V\left[\mathcal{F}\right]$, we have the result

$$V\left[F\right]=0\iff V\left[\mathcal{F}\right]=0.$$

(57)

A possible generalization of Equation (56) to the $\mathcal{G}$ estimator, the randomized version of G, is given in Appendix D.

Although neither $V\left[F\right]$ nor $V\left[G\right]$ can be written as in Equation (56), we can still relate them to ${\chi }^2$ divergence. In general, ${\sum }_i^n{\alpha }_k{{\sigma }^{\prime }}_k>0$. Note that, if ${\sum }_k^n{\alpha }_k{{\sigma }^{\prime }}_k=0$, then, for all k, ${{\sigma }^{\prime }}_k=0$ and $V\left[F\right]=V\left[G\right]=0$. Thus both $V\left[G\right],V\left[F\right]=0$ or both $V\left[G\right],V\left[F\right]>0$. Let us denote $K={\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k$. Then,

$$\begin{array}{ccc}\hfill V\left[G\right]& =& \sum _{i=1}^n\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{{\beta }_i}\hfill \\ \hfill & =& K^2\sum _{i=1}^n\frac{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{K^2}}{{\beta }_i}+K^2\sum _{i=1}^n{\beta }_i-K^2+K^2\sum _{i=1}^{n}2\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{K}-K^2\sum _{i=1}^{n}2\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{K}\hfill \\ \hfill & =& K^2\left(\sum _{i=1}^n\frac{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{K^2}-2\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{K}{\beta }_i+{\beta }_i^2}{{\beta }_i}\right)-K^2+K^2\sum _{i=1}^{n}2\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{K}\hfill \\ \hfill & =& K^2\left(\sum _{i=1}^n\frac{\frac{{\alpha }_i^2{{\sigma }^{\prime }}_i^2}{K^2}-2\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{K}{\beta }_i+{\beta }_i^2}{{\beta }_i}\right)+K^2\hfill \\ \hfill & =& K^2\left(\sum _{i=1}^n\frac{{(\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{K}-{\beta }_i)}^2}{{\beta }_i}+1\right)\hfill \\ \hfill & =& {\left(\sum _{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k\right)}^2\left({\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\beta }_i\right\}\right)+1\right).\hfill \end{array}$$

(58)

For ${\{{\beta }_i\propto {\alpha }_i{\sigma }^{\prime }\}}_{i=1}^n$ the ${\chi }^2$ divergence value is zero, and $V\left[G\right]={\left({\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k\right)}^2$. This is the optimal of $V\left[G\right]$ when the ${\left\{{\alpha }_i\right\}}_{i=1}^n$ are fixed, as seen in Section 3.2.

Substituting ${\{{\beta }_i={\alpha }_i\}}_{i=1}^n$ in Equation (58) we obtain

$$\begin{array}{c}\hfill V\left[F\right]={\left(\sum _{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k\right)}^2\left({\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\alpha }_i\right\}\right)+1\right).\end{array}$$

(59)

As we have taken ${\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k>0$, we can divide first and last terms of Equations (58) and (59),

$$\begin{array}{c}\hfill V\left[G\right]=V\left[F\right]\frac{{\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\beta }_i\right\}\right)+1}{{\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\alpha }_i\right\}\right)+1}.\end{array}$$

(60)

Hence the following theorem,

Theorem 13.

If $V\left[G\right],V\left[F\right]>0$, then

$$V\left[G\right]\le V\left[F\right]\iff {\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\beta }_i\right\}\right)\le {\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\alpha }_i\right\}\right).$$

Observe that Theorem 13 does not impose any a priori condition on the $\left\{{\alpha }_i\right\},\left\{{\beta }_i\right\}$ parameters and generalizes Theorem 12. In the same way we can generalize Theorems 7–11 with the following corollaries,

Corollary 3.

If $V\left[G\right],V\left[G\left(\{{\beta }_i=1/n\}\right)\right]>0$, then

$$V\left[G\right]\le V\left[G\left(\{{\beta }_i=1/n\}\right]\right)\iff {\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\beta }_i\right\}\right)\le {\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\{1/n\}\right).$$

Corollary 4.

If $V\left[G\left(\{{\beta }_i=1/n\}\right)\right],V\left[F\right]>0$, then

$$V\left[G\left(\{{\beta }_i=1/n\}\right)\right]\le V\left[F\right]\iff {\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\{1/n\}\right)\le {\chi }^2\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\alpha }_i\right\}\right).$$

Remark 4.

Observe that, using the second order ${\chi }^2$ approximation of $KL$ (Kullback–Leibler) distance between two distributions $X_1,X_2$, ${\chi }^2(X_1,X_2)\approx 2KL(X_1,X_2)$ [31], Theorem 13 can be written as

$$\begin{array}{ccc}\hfill V\left[G\right]& \le & V\left[F\right]\iff KL\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\beta }_i\right\}\right)\le KL\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\alpha }_i\right\}\right)\hfill \\ \hfill & \iff & CE\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\beta }_i\right\}\right)\le CE\left(\{\frac{{\alpha }_i{{\sigma }^{\prime }}_i}{{\sum }_{k=1}^n{\alpha }_k{{\sigma }^{\prime }}_k}\},\left\{{\alpha }_i\right\}\right),\hfill \\ \hfill \end{array}$$

where cross entropy $CE(X_1,X_2)=H\left(X_1\right)+KL(X_1,X_2)$, and H stands for entropy.

The same approximation can be used for Corollaries 3 and 4.

6. Numerical Examples

6.1. Efficiency Comparison between F and G Estimators

We compare the efficiencies for the F estimator and the optimal G estimator defined in Theorem 5 with five examples.

Table 3. We show the metric $E_F^{-1}=V\left[F\right]·Cost\left[F\right]$ and $E_G^{-1}=V\left[G\right]·Cost\left[G\right]$, i.e., the inverse of efficiency or product of variance and cost for the F estimator and for the optimal G estimator, using for both the same $\left\{{\alpha }_k\right\}$ values. The optimal efficiencies of G estimator, for each set of $\left\{{\alpha }_k\right\}$ values, are computed using the left-hand side of Equation (37), while the efficiencies of F estimator use the right-hand side of Equation (37). We consider for the five numerical examples using an equal count of samples, count inversely proportional to the variances of independent estimators [8,9], and for the three estimators defined in [28]. The sampling costs are (1, 6.24, 3.28). In Example 5, we present the case with equal costs (1, 1), and different costs (1, 5).

	Ex. 1		Ex. 2		Ex. 3		Ex. 4		Ex. 5		Ex. 5
									costs = (1,1)	costs = (1,1)	costs = (1,5)	costs = (1,5)
Estimator	F	G	F	G	F	G	F	G	F	G	F	G
${\alpha }_k\propto \frac{1}{n}$	102.26	89.40	17.24	15.44	37.47	31.80	98.68	83.78	0.28	0.23	0.83	0.40
${\alpha }_k\propto \frac{1}{c_k{v}_k}$ [8]	49.53	41.29	9.28	8.10	4.03	3.85	300.12	294.85	0.31	0.26	2.76	2.33
${\alpha }_k\propto \frac{1}{c_k{m}_k^2}$ [28]	54.36	46.2	9.82	8.49	3.12	2.49	534.37	449.33	0.20	0.15	1.51	1.00
${\alpha }_k\propto \frac{{\sigma }_{k,eq}}{\sqrt{c_k}}$ [28]	81.43	69.88	13.54	11.67	28.68	23.17	91.01	73.54	1.00	0.98	2.90	2.50
${\alpha }_k\propto \frac{M_{k,eq}}{\sqrt{c_k}}$ [28]	79.73	67.77	13.08	11.35	25.74	20.76	31.77	25.90	0.29	0.24	2.72	2.28

shows the inverse of the efficiencies, $E_F^{-1}=V\left[F\right]·Cost\left[F\right]$ and $E_G^{-1}=V\left[G\right]·Cost\left[G\right]$, i.e., the product of variance and cost for the F estimator and for the optimal G estimator, for these possible sets of ${\left\{{\alpha }_k\right\}}_{k=1}^n$: (i) equal count of samples, $\{{\alpha }_k=1/n\}$; (ii) count of samples inversely proportional to the variances of the independent techniques times the cost of sampling the technique, $\{{\alpha }_k\propto \frac{1}{c_k{v}_k}\}$ [8,9]; and the three balance heuristic estimators defined in [28], which are (iii) count of samples inversely proportional to the second moments of the independent techniques times the cost of sampling the technique, $\{{\alpha }_k\propto \frac{1}{c_k{m}_k^2}\}$; (iv) count of samples proportional to ${\sigma }_{k,eq}=\frac{{\sigma }_k^{\prime }\left({\{{\alpha }_i=1/n\}}_{i=1}^n\right)}{n}$, and inversely proportional to the square root of the cost, $\{{\alpha }_k\propto \frac{{\sigma }_{k,eq}}{\sqrt{c_k}}\}$; and (v) $\{{\alpha }_k\propto \frac{M_{k,eq}}{\sqrt{c_k}}\}$ where $M_{k,eq}=\sqrt{\int \frac{f^2\left(x\right)p_k\left(x\right)}{{\left({\sum }_i{p}_i\left(x\right)\right)}^2}}$.

Below, we describe the five examples. From the results in Table 3, we can see:

• As expected, Theorems 4 and 5 hold for all the cases.
• Examples 1–4 show a general gain in efficiency, around 20%.
• Example 5 with equal costs shows a gain in efficiency from 2% to around 20%.
• Example 5 with different costs shows big gains in efficiency, in particular for equal count of sampling the efficiency is doubled.

We see in all our examples an increase in efficiency when the estimator G is used instead of the estimator F, showing in some cases, as in Example 5, a 50% improvement. Theorems 4 and 5 ensure that estimator G is always better than estimator F if the optimal $\left\{{\beta }_i\right\}$ values are used; however, for equal costs (i.e., comparing only variances), the amount of improvement strongly depends on the example considered, from very small o negligible (see first three rows of

Table 4. Variances of Examples 1–5 for some notable cases. The variables ${{\mu }^{\prime }}_{i,1/n},{{\sigma }^{\prime }}_{i,1/n}$ correspond to $\{{\alpha }_i=\frac{1}{n}\}$. The variables $m_i^2,v_i$ are the second moment and variance of independent technique i, $v_i=m_i^2-{\mu }^2$.

	Estimator	Example 1	Example 2	Example 3	Example 4	Example 5
1	$Optima{l}_{\alpha }V\left[\mathcal{F}\right]$	22.7122	4.1949	0	0	0.0910
2	$Optima{l}_{\alpha }V\left[F\right]$	22.7122	4.1944	0	0	0.0903
3	$Optima{l}_{\alpha ,\beta }V\left[G\right]$	22.7122	4.1932	0	0	0.0601
4	$V\left[F({\alpha }_i=1/n)\right]$	29.1634	4.9175	10.6877	28.1431	0.2771
5	$Optima{l}_{\beta }V\left[G({\alpha }_i=1/n)\right]$	29.0908	4.9069	10.6125	27.9412	0.2264
6	$Optima{l}_{\alpha }V\left[G({\beta }_i=1/n)\right]$	27.1603	4.7265	0	0	0.0653
7	$V\left[F({\alpha }_i=arg\forall i,{\mu }_i^{\prime }\approx \mu )\right]$	22.8216	4.1980	0	0	0.0926
8	$V\left[G({\alpha }_i=arg\forall i,{\alpha }_i{\sigma }_i^{\prime }\approx \mathrm{equal},{\beta }_i=1/n)\right]$	28.4089	4.8313	12.4705	11.2072	0.0734
9	$V\left[G({\alpha }_i\propto \frac{m_i^2}{v_i},{\beta }_i=1/n)\right]$,19]	28.0224	4.8513	3.7955	172.192	0.2832
10	$V\left[G({\alpha }_i\propto \frac{1}{v_i},{\beta }_i=1/n)\right]$	27.4126	4.7897	0.5466	1256.48	0.2943
11	$V\left[G({\alpha }_i\propto \frac{1}{{\sigma }_{i,1/n}^{\prime }},{\beta }_i=1/n)\right]$	28.3977	4.8291	11.95	13.7926	0.0724
12	$V\left[G({\alpha }_i\propto \frac{{{\mu }^{\prime }}_{i,1/n}}{{{\sigma }^{\prime }}_{i,1/n}},{\beta }_i=1/n)\right)]$	27.4426	4.7373	7.8615	6.7895	0.1112
13	$V\left[G({\alpha }_i=arg\forall i,{\mu }_i^{\prime }\approx \mu ,{\beta }_i=1/n)\right]$	41.8791	8.8798	0.0014	0	0.0739

for the first four examples) to an important one (compare Table 3 for Example 5 with equal costs with the first three rows of Table 4 for Example 5). The values are computed for the results in Table 3 and Table 4 by using numerical approximations obtained with Mathematica software. It would also be possible to select them in an adaptive and iterative way. We defer the investigation of these adaptive schemes for a future work.

6.1.1. Example 1

Suppose we want to solve the integral

$$\mu ={\int }_{\frac{3}{2\pi }}^{\pi }\mathbf{x}\left({\mathbf{x}}^2-\frac{\mathbf{x}}{\pi }\right)sin\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}\approx 10.29$$

(61)

by MIS sampling on functions $\mathbf{x}$, $({\mathbf{x}}^2-\frac{\mathbf{x}}{\pi })$, and $sin\left(\mathbf{x}\right)$, respectively. We first find the normalization constants: ${\int }_{\frac{3}{2\pi }}^{\pi }\mathbf{x}\mathrm{d}\mathbf{x}=4.82082$, ${\int }_{\frac{3}{2\pi }}^{\pi }({\mathbf{x}}^2-\frac{\mathbf{x}}{\pi })\mathrm{d}\mathbf{x}=8.76463$, ${\int }_{\frac{3}{2\pi }}^{\pi }sin\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}=1.88816$. The costs for sampling the techniques are (1; 6.24; 3.28).

6.1.2. Example 2

Let us solve the integral

$$\mu ={\int }_{\frac{3}{2\pi }}^{\pi }\left({\mathbf{x}}^2-\frac{\mathbf{x}}{\pi }\right){sin}^2\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}\approx 3.60$$

(62)

using the same functions $\mathbf{x}$, $({\mathbf{x}}^2-\frac{\mathbf{x}}{\pi })$, and $sin\left(\mathbf{x}\right)$ as before.

6.1.3. Example 3

As the third example, let us solve the integral

$$\mu ={\int }_{\frac{3}{2\pi }}^{\pi }\mathbf{x}+\left({\mathbf{x}}^2-\frac{\mathbf{x}}{\pi }\right)+sin\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}\approx 15.47$$

(63)

using the same functions as before. The optimal $\alpha$ values for this example, with zero variance, are $\frac{1}{4.82082+8.76463+1.88816}(4.82082,8.76463,1.88816)$.

6.1.4. Example 4

As the fourth example, consider the integral of the sum of the three pdfs

$$\begin{array}{cc}\hfill \mu & ={\int }_{\frac{3}{2\pi }}^{\pi }30\frac{\mathbf{x}}{4.82082}+30\frac{\left(x^2-\frac{\mathbf{x}}{\pi }\right)}{8.76463}+40\frac{sin\left(\mathbf{x}\right)}{1.88816}\mathrm{d}\mathbf{x}\hfill \\ \hfill & \approx 100.\hfill \end{array}$$

(64)

In this case we know again the optimal (zero variance) $\alpha$ values: $(0.3,0.3,0.4)$. The difference with the previous example is that this case should be most favorable to equal count of samples.

6.1.5. Example 5

As the last example, consider solving the following integral

$$\begin{array}{c}\hfill \mu ={\int }_{0.01}^{\pi /2}\left(\sqrt{\mathbf{x}}+sin\mathbf{x}\right)\mathrm{d}\mathbf{x}\approx 2.31175.\end{array}$$

(65)

by MIS sampling on functions $2-\mathbf{x}$, and ${sin}^2\left(\mathbf{x}\right)$.

6.2. Variances of Examples 1–5 for Some Notable Cases

In Table 4 we present some notable cases for the variance of Examples 1–5. Observe that here we do not consider the cost of sampling. The values are computed for the results in Table 4 with Mathematica software. We have

• In first, second and third row we present the optimal values of $V\left[\mathcal{F}\right]\ge V\left[F\right]\ge V\left[G\right]$, respectively. In the first example those values are equal up to the fourth decimal position. In the second example they are equal up to the second decimal position, in the third and fourth examples they are equal, and in the fifth example the gain of $V\left[G\right]$ with respect to $V\left[F\right]$ is more than 30%.
• In the fourth row we present the values of $V\left[F\right(1/n\left)\right]$, which are compared against several strategies and heuristics in the following rows
• The values in the fifth row for $V\left[G\right]$, that correspond to Equation (30) (or the left-hand side of Equation (37) with all costs equal to 1), although being better than the values for $V\left[F\right]$ in the fourth row, as expected, do not give any significant improvement. As we have seen in Table 3 the improvements for this case come rather from efficiency.
• The values in the sixth row for $V\left[G\right]$, that correspond to the solution of Equation (40), this is, the optimal $\left\{{\alpha }_i\right\}$ for $\{{\beta }_i=1/n\}$, are smaller than the variances of $V\left[F\right]$ for equal sampling rate, fourth row, and much smaller in some of the examples, with variances equal (as in Examples 3 and 4), or less, as in Example 5, than the optimal $V\left[F\right]$.
• Although in general there is no $\left\{{\alpha }_i\right\}$ solution for all ${\mu }_i^{\prime }$ equal, for instance the numerical approximation obtained in Example 1 is ${\mu }_1^{\prime }=10.2624$, ${\mu }_2^{\prime }={\mu }_3^{\prime }=10.2876$, the variances obtained, in the seventh row of Table 4, are for all the examples very near to the optimal value of $V\left[F\right]$. Observe that if these $\left\{{\alpha }_i^{\star }\right\}=arg\forall i,{\mu }_i^{\prime }equal$ values exist, then they are the optimal $\left\{{\alpha }_i\right\}$ values for $V\left[G({\beta }_i={\alpha }_i^{\star })\right]$, i.e., for the sampling rates $\{{\beta }_i={\alpha }_i^{\star }\}$ the G estimator can not improve on the F estimator.
• In row 8 we show the G estimator corresponding ${\alpha }_i{\sigma }_i^{\prime }\approx equal$, with ${\beta }_i=1/n$. It improves on $F(1/n)$ for all examples except the third one, where it scores closely, beating estimators in rows 9 and 10 except for Example 3. It is indeed a theoretical estimator as there is no easy way to solve this equation. However, it can be approximated by the heuristic in line 11, see below. Observe that if $\left\{{\alpha }_i^{\star }\right\}$ are the solutions of for all i, ${\alpha }_i{\sigma }_i^{\prime }$ equal, then the optimal $\left\{{\beta }_i\right\}$ for $G(\left\{{\alpha }_i^{\star }\right\},\left\{{\beta }_i\right\})$ are $\{{\beta }_i=1/n\}$.
• The results in row 9 correspond to the estimator defined in [19], ${\alpha }_i\propto m_i^2/v_i$, where $m_i^2=v_i+{\mu }^2$ are the second moments of the independent techniques, and ${\beta }_i=1/n$. The results are slightly better than $F(1/n)$ for Examples 1 and 2, much better for Example 3, much worse for Example 4 and slightly worse for Example 5. Observe that the $m_i,v_i$ values can be easily approximated using a first batch of samples in a Monte Carlo integration, as it was already performed in [7,9,10].
• In row 10 we introduce the estimator corresponding to ${\alpha }_i\propto 1/v_i$, and ${\beta }_i=1/n$, which gives better results than $F(1/n)$ for Examples 1 and 2, much better for Example 3, worse for Example 5, and much worse for Example 4. This estimator improves on the one in [19] except for Example 4. Observe that these ${\alpha }_i$ weights correspond to the optimal weights in the linear combination of estimators when the sampling rates (${\beta }_i$) are fixed, Equation (13) of [9].
• In row 11 we approximate the estimator in row 8 by using $\left\{{\sigma }_{i,1/n}^{\prime }\right\}$, which corresponds to $\{{\alpha }_i=1/n\}$, to approximate $\left\{{\sigma }_i^{\prime }\right\}$. Observe that $\left\{{\sigma }_{i,1/n}^{\prime }\right\}$ values can be easily obtained with a first batch of samples in a Monte Carlo integration. In all cases, except in Example 3, we improve on $F(1/n)$. The big error on Example 4 of the estimator in lines 9 and 10 is controlled.
• In row 12 we modulate the estimator in row 11 with the ${\mu }_i^{\prime }$ values. Observe that now for all the examples the variance is less than for $F(1/n)$.
• The results in row 13 show that, even if $F\left(\left\{{\alpha }_i\right\}\right)\le F\left(\{1/n\}\right)$, we can not guarantee that $G(\left\{{\alpha }_i\right\},\{{\beta }_i=1/n\})\le F\left(\{1/n\}\right)$.

From the results in Table 4 row 6, we can conclude that there can be room for improvement on $F(1/n)$ estimator using a G estimator, with a suitable selection of ${\alpha }_i$ parameters. Heuristics as in rows 9 and 10 can give much improvement in some cases at the cost of no improvement at all and even a huge variance in other cases, thus they are not robust heuristics. More sophisticated heuristics as in rows 11 and 12, based on the theoretically justified heuristic in row 8, show a robust behavior.

A very promising estimator is the one in row 7, for all i all ${\mu }_i^{\prime }$ equal (to $\mu$), in all five examples the variance is very near the optimal $V\left[F\right]$. Observe that this estimator has a strong theoretical justification, derived from the study of G estimator. The $\left\{{\alpha }_i\right\}$ parameters that approximate for all i, ${\mu }_i^{\prime }=\mu$ could be obtained in an adaptive way; however, a practical way to obtain this estimator should be devised.

7. Conclusions

In this paper, we have proposed a multiple importance sampling estimator that combines samples simulated from different techniques. The novel estimator generalizes the Monte Carlo balance heuristic estimator, widely used in the literature of signal processing, computational statistics, and computer graphics. In particular, this estimator relaxes the connection between the coefficients that select the number of samples per proposal, and the samples that appear in the mixture of techniques at the denominator of the importance weight. This flexibility shows a relevant improvement in terms of variance in the combined estimator with regard to the balance heuristic estimator (which is included as a particular case in the novel estimator). We have studied the optimal choice of the free coefficients in such a way that the variance of the resulting estimator is minimized. In addition, numerical results have shown that the significant gap in terms of variance between both estimators justifies the use of the novel estimator whenever possible. In the particular, but much used case, of equal sampling, our new estimator shows a big potential for improvement. Future work may include the application of this variance-reduction technique in the context of adaptive importance sampling [15] or within MCMC-based methods that include re-weighting [32].