A Review of Representative Points of Statistical Distributions and Their Applications

Abstract

Statistical modeling relies on a diverse range of statistical distributions, encompassing both univariate and multivariate distributions and/or discrete and continuous distributions. In the literature, numerous statistical methods have been proposed to approximate continuous distributions. The most commonly used approach is the use of the empirical distribution which is obtained from a random sample drawn from the distribution. However, it is very likely that the empirical distribution suffers from an accuracy problem when used to approximate the underlying distribution, especially if the sample size is not sufficient. In order to improve statistical inferences, various alternative forms of discrete approximation to the distribution were proposed in the literature. The choice of support points for the discrete approximation, known as Representative Points (RPs), becomes extremely important in terms of distribution approximations. In this paper we give a review of the three main methods for constructing RPs, namely based on the Monte Carlo method, the number-theoretic method (or quasi-Monte Carlo method), and the mean square error method, aiming to introduce such important methods to the statistical or mathematical community. Additional approaches for forming RPs are also briefly discussed. The review focuses on certain critical aspects such as theoretical properties and computational algorithms for constructing RPs. We also address the issue of the application of RPs through studying practical problems and provide evidence of RPs’ advantages over random samples in approximating the distribution.

1. Introduction

Statistics is an art of science involving data collection and data analysis, in which statistical modeling relies on various types of statistical distributions. Such distributions are either discrete or continuous and either univariate or multivariate distributions. For an unknown continuous distribution F in ${\mathbb{R}}^d$, the conventional approach is to approximate F using the empirical distribution of a random sample. The empirical distribution is discrete, consisting of support points from the random sample, with each point contributing equally to the approximation. Because of the accuracy problem resulting from the empirical distribution, we want to construct a discrete distribution $\widehat{F}\in {\mathbb{R}}^d$ that approximates the distribution F while preserving the distribution information as much as possible. Consider a random vector X following a continuous distribution $F\left(\mathit{x}\right)\in {\mathbb{R}}^d$, characterized by a probability density function (pdf) $p\left(\mathit{x}\right)$. In contrast, a discrete random vector Y is characterized by a probability mass function (pmf) $F_Y\left(\mathit{y}\right)$.

$$\begin{array}{c}\hfill \begin{array}{ccccc}\mathit{Y}& {\mathit{y}}_1& {\mathit{y}}_2& \cdots & {\mathit{y}}_k\\ p& p_1& p_2& \cdots & p_k\end{array}\end{array}$$

(1)

where ${\mathit{y}}_1,\dots ,{\mathit{y}}_k$ are support points of Y and $P(\mathit{Y}={\mathit{y}}_i)=p_i>0,i=1,\dots ,k,{\sum }_{i-1}^k{p}_i=1.$ For defining an approximation distribution $F_Y\left(\mathit{y}\right)$ to $F\left(\mathit{x}\right)$ it should satisfy

(i)

$F_Y\left(\mathit{y}\right)$ is a function F;

(ii)

A pre-decided distance between F and $F_Y$ is small;

(iii)

$F_Y^{\left(k\right)}\to F$ in distribution as $k\to \infty$, where k in $F_Y$ is the number of support points of Y. In this case, the support points ${\mathit{y}}_1,\dots ,{\mathit{y}}_k$ are called representative points (RPs). There are several ways to choose an approximation distribution $F_Y$.

1.1. Monte Carlo—RPs

Let $\mathit{X}\sim F(\mathit{x};\mathit{\theta })\in {\mathbb{R}}^d$ be a random vector, where $\mathit{\theta }$ represents the parameters. For instance, for the normal distribution $N(\mu ,{\sigma }^2)$, the parameters are denoted as $\mathit{\theta }=(\mu ,{\sigma }^2)$. In traditional statistics, random samples are utilized to make inferences about the population. Specifically, a collection of independently and identically distributed (iid) random samples, denoted as ${\mathit{x}}_1,\dots ,{\mathit{x}}_k$, are drawn from the population distribution F. The empirical distribution of the random sample is defined as follows:

$$\begin{array}{c}\hfill F_k\left(\mathit{x}\right)=\frac{1}{k}\sum _{i=1}^{k}I\left\{{\mathit{x}}_i\le \mathit{x}\right\},\end{array}$$

(2)

where $I_A$ is the indicator function of A, and the inequalities ${\mathit{x}}_i\le \mathit{x}$ means that $x_{il}\le x_l$ ($1\le i\le k$, $1\le l\le d$) where ${\mathit{x}}_i={(x_{i1},\cdots ,x_{id})}^T$ and $\mathit{x}={(x_1,\cdots ,x_d)}^T$. Many statistical inferences rely on the empirical distribution $F_k\left(\mathit{x}\right)$, which includes various methods such as:

(1)

Parameter estimation (point estimation and confidence interval estimation);

(2)

Density estimation;

(3)

Testing hypothesis, and so on.

The empirical distribution is a discrete distribution with support points ${\mathit{x}}_1,\dots ,{\mathit{x}}_k$ each having the sampling probability $1/k$ and can be considered as an approximation of $F\left(\mathit{x}\right)$ in the sense of consistency, i.e., $F_k\left(\mathit{x}\right)\to F\left(\mathit{x}\right)$ in distribution as $k\to \infty$. In statistical simulation, a set of random samples can be generated by computer software under the Monte Carlo (MC) method. Therefore, we denote random variable ${\mathit{Y}}_{MC}\sim F_k\left(\mathit{y}\right)$ or ${\mathit{Y}}_{MC}\sim F_{MC}^{\left(k\right)}\left(\mathit{y}\right)$ in this paper. The MC methods have been commonly used. For instance, in the case of a normal population $N(\mu ,{\sigma }^2)$ with unknown parameters $\mu$ and ${\sigma }^2$, one can utilize the sample mean $\overline{x}$ and the sample variance $s^2=\frac{1}{n-1}{\sum }_{i=1}^n{(x_i-\overline{x})}^2$ to estimate $\mu$ and ${\sigma }^2$, respectively.

As the empirical distribution can be regarded as an approximation distribution to $F\left(\mathit{x}\right)$, one can therefore take a set of random samples from $F_{MC}$ instead of from F, as suggested by Efron [1], which is called as bootstrap method. The bootstrap method is a resampling technique, where the random sample takes from an approximation distribution $\widehat{F}$. Later, Efron gave a comprehensive study on the theory and application of the bootstrap method.

The MC method has proven to be useful in statistical theory and applications. However, its efficiency is not always good due to the convergence rate of $F_k\left(x\right)\to F\left(x\right)$ in distribution, which is $O_p\left(\frac{1}{\sqrt{k}}\right)$ as $k\to \infty$. The slow convergence leads to unsatisfactory approximations when performing numerical integration using the MC method. While the empirical distribution serves as one approximation to the true distribution F, alternative approaches were proposed in the literature to address this issue.

1.2. Number-Theoretic RPs or Quasi-Monte Carlo RPs

Let us consider the numerical calculation for high-dimension integration in a canonical form

$$\begin{array}{c}\hfill I\left(f\right)={\int }_0^1\cdots {\int }_0^{1}f(y_1,\dots ,y_d)d{y}_1\cdots d{y}_d={\int }_{C^d}f\left(\mathit{y}\right)d\mathit{y},\end{array}$$

where f is a continuous function on $C^d={[0,1]}^d$. Let $\mathcal{Y}=\{{\mathit{y}}_1,\dots ,{\mathit{y}}_k\}$ be a set of k points uniformly scattered on $C^d$. One can use the mean of $\{f\left({\mathit{y}}_i\right),{\mathit{y}}_i\in \mathcal{Y}\}$, denoted by $\overline{f\left(\mathit{y}\right)}=\frac{1}{k}{\sum }_{i=1}^{k}f\left({\mathit{y}}_i\right)$, to approximate $I\left(f\right)$. By the MC method, we can employ a random sample from the uniform distribution $U\left(C^d\right)$. The rate of convergence of $\overline{f\left(\mathit{y}\right)}\to I\left(f\right)$ is $O_p(1/\sqrt{k})$, which is relatively slow but does not depend on the dimensionality d. How to increase the convergence rate is an impotent subject in applications. The number-theoretic methods (NTM) or quasi-Monte Carlo methods (QMC) provide many methods for the construction of $\mathcal{Y}$ such that $\{{\mathit{y}}_1,\dots ,{\mathit{y}}_k\}$ are uniformly scattered on $C^d$, by which it can increase the rate of convergence into $O\left(k^{-1}{(logk)}^d\right)$. For the theory and methodology of NTM/QMC, one can refer to Hua and Wang [2] and Niederreiter [3]. In the earlier study on NTM, many authors employed the star discrepancy as a measure of the uniformity of $\mathcal{Y}$ in $C^d$. The star discrepancy is defined by

$$\begin{array}{c}\hfill D(F,F^{\left(k\right)})=\underset{\mathit{x}\in {\mathbb{R}}^d}{sup}|F\left(\mathit{x}\right)-F^{\left(k\right)}\left(\mathit{x}\right)|,\end{array}$$

where $F\left(\mathit{x}\right)$ is the cdf of $U\left(C^d\right)$ and $F^{\left(k\right)}\left(\mathit{x}\right)$ is the empirical distribution of $\mathcal{Y}$. An optimal $\mathcal{Y}$ has the minimum $D(F,F^{\left(k\right)})$. In this case the points in $\mathcal{Y}$ are called QMC-RPs that are support points of $F^{\left(k\right)}\left(\mathit{y}\right)$ each having the equal probability $1/k$. In this paper we denote $F^{\left(k\right)}$ by $F_{QMC}^{\left(k\right)}$ and random vector ${\mathit{Y}}_{QMC}^{\left(k\right)}\sim F_{QMC}^{\left(k\right)}$. Another popular measure is the $L_p$-distance between $F^{\left(k\right)}$ and F

$$\begin{array}{c}\hfill D_p(F^{\left(k\right)},F)={\left({\int }_{{\mathbb{R}}^d}{|F^{\left(k\right)}\left(\mathit{x}\right)-F\left(\mathit{x}\right)|}^{p}d\mathit{x}\right)}^{1/p}.\end{array}$$

(3)

When F is the uniform distribution on $C^d$, the the $L_p$-distance is called as $L_p$-discrepancy. The star discrepancy is the $L_p$-discrepancy as $p\to \infty$. In the literature a set of $\{{\mathbf{q}}_1,\dots ,{\mathbf{q}}_n\}$ under a certain structure is regarded as a set of quasirandom F-numbers if its discrepancy has an order $O\left(n^{-1+ϵ}\right)$ under the given discrepancy. When F is the uniform distribution on $C^d$ ($U\left(C^d\right)$), the quasirandom F-numbers are called as quasirandom numbers. The reader can refer to Fang and Wang [4] for details. Due to the numerical computation complexity of the $L_p$-discrepancy ($p\ne 2$), the $L_2$-discrepancy has a simple computational formula. There are more uniformity measures in the experimental design such as the centered $L_2$-discrepancy, wrap-around $L_2$-discrepancy and mixture $L_2$-discrepancy (refer to Fang et al. [5]). Fang and Wang [4] and Fang et al. [6] gave a comprehensive study on NTM and applications of NTM in statistical inference, experimental design, geometric probability, and optimization. Pages [7] gave a detailed study on applications of QMC to financial mathematics. Section 6 will introduce some algorithms for the generation of QMC-RPs.

1.3. Mean Square Error—RPs

Another measure to choose discrete approximation distribution to a given continuous distribution is the mean square error (MSE), and the corresponding support points are called MSE-RPs.

Definition 1.

Suppose that a random vector $\mathit{X}\sim F\left(\mathit{x}\right)$ in ${\mathbb{R}}^d$ has a density function $p\left(\mathit{x}\right)$ with finite mean vector and covariance matrix. A set of points of $\Xi =\{{\mathit{\xi }}_j,j=1,\cdots ,k\}$ is called MSE-RPs if it minimizes the mean square error (MSE)

$$\begin{array}{c}\hfill \mathrm{MSE}(\Xi )={\int }_{{\mathbb{R}}^d}\underset{j=1,\cdots ,k}{min}\left|\right|\mathit{x}-{\mathit{\xi }}_j{\left|\right|}^{2}d\mathit{x},\end{array}$$

(4)

where $\left|\right|\mathit{a}\left|\right|$ denotes $l_2$-norm of $\mathit{a}\in {\mathbb{R}}^d$.

Given a set of k points $\{{\mathit{y}}_1,\dots ,{\mathit{y}}_k\}$ in ${\mathbb{R}}^d$ a set of regions are defined by

$$\begin{array}{c}\hfill S_j=\{\mathit{x}:{(\mathit{x}-{\mathit{y}}_j)}^T(\mathit{x}-{\mathit{y}}_j)\le {(\mathit{x}-{\mathit{y}}_h)}^T(\mathit{x}-{\mathit{y}}_h),\forall h\ne j\},j=1,\dots ,k,\end{array}$$

that are called as Voronoi regions, where $S_j$ is the attraction domain of ${\mathit{y}}_j$.

For a univariate distribution $F\left(x\right)$ with pdf $p\left(x\right)$, mean $\mu$ and variance ${\sigma }^2$, its MSE-RPs can be sorted as ${\xi }_1^{\left(k\right)}<{\xi }_2^{\left(k\right)}<\cdots <{\xi }_k^{\left(k\right)}$, and its MSE can be expressed as

$$\begin{array}{c}\hfill \mathrm{MSE}\left(\mathcal{Y}\right)={\int }_{-\infty }^{+\infty }\underset{i}{min}{(x-{\xi }_i^{\left(k\right)})}^{2}p\left(x\right)dx=\sum _{j=1}^k{\int }_{I_j}{(x-{\xi }_i^{\left(k\right)})}^{2}p\left(x\right)dx,\end{array}$$

(5)

where

$$\begin{array}{ccc}\hfill & & I_1=(a_1,a_2),I_2=(a_2,a_3),\dots ,I_k=(a_k,a_{k+1}),\hfill \\ & & a_1=-\infty ,a_i=\left({\xi }_{i-1}^{\left(k\right)}+{\xi }_i^{\left(k\right)}\right)/2,i=2,\cdots ,k,a_{k+1}=\infty .\hfill \end{array}$$

(6)

The corresponding $Y_{MSE}$ has support points ${\xi }_j^{\left(k\right)},1\le j\le k$ with probabilities $P(Y_{MSE}={\xi }_i)=p_i$ where

$$\begin{array}{ccc}\hfill & & p_1={\int }_{-\infty }^{({\xi }_1+{\xi }_2)/2}p\left(x\right)dx,\hfill \\ & & p_i={\int }_{({\xi }_{i-1}+{\xi }_i)/2}^{({\xi }_i+{\xi }_{i+1})/2}p\left(x\right)dx,i=2,\cdots ,k-1,\hfill \\ & & p_k={\int }_{({\xi }_{k-1}+{\xi }_k)/2)}^{\infty }p\left(x\right)dx.\hfill \end{array}$$

(7)

$$\begin{array}{c}\hfill \begin{array}{ccccc}{Y}_{MSE}& {\xi }_1& {\xi }_2& \cdots & {\xi }_k\\ p& p_1& p_2& \cdots & p_k\end{array}\end{array}$$

(8)

Its loss function (LF) is defined by

$$\begin{array}{c}\hfill L\left(\mathcal{Y}\right)=L({\xi }_1,\dots ,{\xi }_k)=\frac{1}{{\sigma }^2}MSE\left(\mathcal{Y}\right).\end{array}$$

(9)

It is known that $0\le L\left(\mathcal{Y}\right)\le 1$. The loss function shows what percentage of $\mathrm{Var}\left(X\right)$ is lost by using $Y_{MSE}$ replacing X.

The concept of MSE-RPs has been motivated by various problems. In the context of the grouping problem, Cox [8] considered the task of condensing observations of a variate into a limited number of groups, where the grouping intervals are selected to retain maximum information. He introduced the concept of mean squared error (MSE) and provided several sets of MSE-RPs for the standard normal distribution. The concept of MSE-RPs is also relevant in data transmission systems, where analog input signals are converted to digital form, transmitted, and then reconstituted as analog signals at the receiver. The problem of optimal quantization of a continuous random variable with a fixed number of levels was precisely defined by Max [9]. In IEEE journals, MSE-RPs are also referred to as “quatizers”.

Fang and He [10] proposed the mathematical problem based on the national Chinese garment standard (refer to Fang [11]). Iyengar and Solomon [12] considered the mathematical problems that arise in the theory of representing a distribution by a few optimally chosen points. Flury [13], Flury [14] studied a project of the Swiss army to replace existing with newly designed protection masks. They used “principal points” for MSE-RPs due to some link between the principal components and MSE-RPs. The MSE-RPs are also applied to select a few “representative” curves from a large collection of curves which is useful for kernel density estimation (see Flury and Tarpey [15]) and for psychiatric studies by Tarpey and Petkova [16]. Furthermore, MSE-RPs can be applied to problems related to the numerical computation of conditional expectations, stochastic differential equations, and stochastic partial differential equations. These applications are often motivated by challenges encountered in the field of finance [7]. There was a special issue of “IEEE Transaction on Information Theory” on vector quantizers in 1982, a very detailed review on quantization by Gray and Neuhoff [17]. There are several monographs on the theory and applications of RPs, for example, Graf and Luschgy [18] “Foundations of Quantization for Probability Distributions” and Pages [7] “Numerical Probability, An Introduction with Applications to Finance”.

The use of different types of representative points (RPs) allows for the construction of diverse approximation distributions to represent the underlying population distribution. By utilizing these approximation distributions, researchers can make more reliable and precise statistical inferences. The objective of this paper is to provide a comprehensive review of various types of RPs and their associated theory, algorithms, and applications. The focus of this review extends to the examination of recent advancements in the field, highlighting the latest developments and emerging trends. This paper aims to offer valuable insights into the current state of the art and provide researchers and practitioners with a deeper understanding of the potential applications and implications of RPs in statistical science. In Section 2, we present a comprehensive list of properties associated with MSE-RPs for univariate distributions. Section 3 focuses on reviewing various algorithms used for generating MSE-RPs for univariate distributions. In Section 4, we compare various types of RPs in terms of their performance in stochastic simulation and resampling. Additionally, we show the consistency of resampling when MSE-RPs. Properties of MSE-RPs for multivariate distributions are reviewed in Section 5, and algorithms for generating QMC-RPs and MSE-RPs for multivariate distributions are introduced in Section 6. QMC-RPs and MSE-RPs have found numerous applications across various domains. In this paper, we focus on selected applications in statistical inference and geometric probability due to space limitations.

2. Properties of MSE-RPs for Univariate Distributions

We collect some properties of MSR-RPs in the literature in this section. These properties can be grouped into different issues. Some properties are only for the univariate distributions, and some are for the multivariate ones. The following results are from many articles, including Fei [19] under the notation in the previous section.

Theorem 1.

Let X be a continuous random variable with pdf $p\left(x\right)$, finite mean μ and variance ${\sigma }^2$. Then we have

$$\begin{array}{ccc}\hfill & & \left(A\right)E\left(Y_{MSE}\right)=E\left(X\right)=\mu ;\hfill \\ & & \left(B\right)E\left(Y_{MSE}^t\right)=E\left(X{Y}_{MSE}^{t-1}\right),t=2,3,\dots ,m-{1,IfE\left(\right|X|}^m)<\infty ;\hfill \\ & & \left(C\right)\mathrm{Var}\left(Y_{MSE}\right)=\mathrm{Var}\left(X\right)-MSE\left(Y_{MSE}^{\left(k\right)}\right);\hfill \\ & & \left(D\right)\mathrm{MSE}\left(Y_{MSE}^{\left(k\right)}\right)\ge \mathrm{MSE}\left(Y_{MSE}^{(k+1)}\right),k>1;\hfill \\ & & \left(E\right)Y_{MSE}^{\left(k\right)}\to Xin distribution ask\to \infty ;\hfill \\ & & \left(F\right)\mathrm{MSE}\left(Y_{MSE}^{\left(k\right)}\right)\to 0,and L\left(Y_{MSE}^{\left(k\right)}\right)\to 0,as k\to \infty .\hfill \end{array}$$

The property (A) can be regarded as “unbiased mean”. The property (C) gives a decomposition of variance of X as

$$\mathrm{Var}\left(X\right)=\mathrm{Var}\left(Y_{MSE}\right)+MSE\left(Y_{MSE}\right).$$

The concept of self-consistent has been used in the clustering analysis and has a close relation with MSE-RPs.

Definition 2.

The set of k points $\{{\mathit{y}}_1,\dots ,{\mathit{y}}_k\}$ in ${\mathbb{R}}^d$ is called self-consistent with respect to the d-variate random vector X and the partition $\{S_j,j=1,\dots ,k\}$ of ${\mathbb{R}}^d$ if

$$\begin{array}{c}\hfill E\left[\mathit{X}\right|\mathit{X}\in S_j]={\mathit{y}}_j,j=1,\dots ,k,\end{array}$$

where the region $S_i$ is the domain of attraction of ${\mathit{y}}_j$.

Tarpey and Flury [20] gave a comprehensive study on the self-consistent and they pointed out

(1)

MSE-RPs are self-consistent with respective to X;

(2)

MSE-RPs have the minimum mean square error among all sets of the self-consistent to X.

2.1. Existence and Uniqueness of MSE-RPs

The existence of MSE-RPs is no problem for any continuous distributions with the first second moments. For the case of $k=1$, the MSE-RP ${\mathit{\xi }}_1$ is the mean of X. This fact indicates that the MSE-RPs can be regarded as an extension of the mean. The MSE-RPs are no analytic formula for most cases of $k\ge 2$, but there are some discoveries on the symmetric distributions. In this paper, the notation $\mathit{X}\stackrel{d}{=}\mathit{Y}$ means two random vectors X and Y have the same distribution.

Definition 3.

A random vector $\mathit{X}\in {\mathbb{R}}^p$ is symmetric to $\mathbf{0}$ if $\mathit{X}\stackrel{d}{=}-\mathit{X}$ and X is symmetric to its mean vector $\mathit{\mu }$ if $\mathit{X}-\mathit{\mu }\stackrel{d}{=}-(\mathit{X}-\mathit{\mu })$.

Theorem 2.

Let $\Xi =\{{\mathbf{\xi }}_j,j=1,\cdots ,k\}$ be a set of MSE-RPs for a symmetric distribution $\mathit{X}\sim F\left(\mathit{x}\right)$ about $\mathbf{0}$, then the set of $\{-{\mathbf{\xi }}_l,\dots ,-{\mathbf{\xi }}_k\}$ is also a set of MSE-RPs. Furthermore, if the set of MSE-RPs for $F\left(x\right)\in \mathbb{R}$ is unique, and its MSE-RPs are sorted as ${\xi }_1<{\xi }_2<\dots ,{\xi }_k$, then

$$\begin{array}{c}\hfill -{\mathbf{\xi }}_i={\mathbf{\xi }}_{k-i+1},i=1,\dots ,\lfloor k/2\rfloor ,\end{array}$$

(10)

where $\lfloor a\rfloor$ is the largest integer not exceeding a.

The following review is for the univariate distribution F with mean $\mu$ and variance ${\sigma }^2$. Sharma [21] pointed out that the MSE-RPs of a symmetric distribution about zero do not need to be symmetric if the set of MSE-RPs is not unique.

Theorem 3.

Let X be a continuous random variable with pdf $p\left(x\right)$, finite mean μ and variance ${\sigma }^2$ and the distribution of X is symmetric about μ. Let $Z=X-\mu$. The two MSE-RPs of X are

$$\begin{array}{c}\hfill {\xi }_1=\mu -E\left(\right|Z\left|\right), and {\xi }_2=\mu +E\left(\right|Z\left|\right),\end{array}$$

and the corresponding MSE is with the related

$$\begin{array}{c}\hfill \mathit{MSE}({\xi }_1,{\xi }_2)=\mathrm{Var}\left(\right|Z\left|\right)=\mathrm{Var}\left(\right|X-\mu \left|\right),\end{array}$$

(11)

if and only if

$$\begin{array}{c}\hfill p\left(\mu \right)E\left(\right|X-\mu \left|\right)\le \frac{1}{2}.\end{array}$$

(12)

This theorem was presented in Flury [13]. If the condition (12) does not hold, Gu and Mathew [22] gave a detailed study on some characterizations of symmetric two MSE-RPs. Their results are listed below.

• Let X be a random variable with density $p\left(x\right)$ symmetric about the mean $\mu$ and continuous at $\mu$; then

  (a)

  If $p\left(\mu \right)E\left(\right|X-\mu \left|\right)<\frac{1}{2}$, then ${\xi }_1=\mu -E\left(\right|Z\left|\right)$ and ${\xi }_2=\mu +E\left(\right|Z\left|\right)$ are 2 MSE-RPs of X;

  (b)

  If $p\left(\mu \right)E\left(\right|X-\mu \left|\right)>\frac{1}{2}$, it implies that the above points do not provide local minimum of MSE.

• They pointed out that Theorem 3 needs to be modified and gave a counterexample about the standard symmetric exponential distribution with pdf $p\left(x\right)=\frac{1}{2}{e}^{-\left|x\right|}$ and mean $\mu =1$. It is easy to find that $p\left(\mu \right)E\left(\right|X-\mu \left|\right)=\frac{1}{2}$, but $\xi =\pm 1$ are MSE-RPs.

More examples are discussed in their article. If two random variables Z and X have the relationship $Z=a+bX$ and MSE-RPs of X are known, then MSE-RPs of Z can be easily obtained (Fang and He [10] and Zoppè [23]).

Theorem 4.

Let $\mathcal{Y}=\{{\xi }_1<\dots <{\xi }_k\}$ be MSE-RPs of X, then $Z=a+bX$ has MSE-RPs of $\{a+b\mathcal{Y}\}=\{a+b{\xi }_1,\dots ,a+b{\xi }_k\}$ with MSE to be $b^2\mathrm{MSE}\left(\mathcal{Y}\right)$.

There are three special families that satisfy the above relationship: the location-scale family ($b>0$), the location family ($b=1$) and the scales family ($b>0,a=0$).

The study on the uniqueness of MSE-RPs is a challenging problem. Fleischer [24] gave a sufficient condition “log-concavity” for the uniqueness of the MSE-RPs. The sufficient conditions in summary:

• The density function $p\left(x\right)$ must be differentiable;
• The derivative ${\left(\log p\left(x\right)\right)}^{\prime }$ must be strictly decreasing.

Trushkin [25] proved that a log-concave probability density function has a unique set of MSE-RPs.

Definition 4.

A continuous random variable X is said to have a log-concave density $p\left(x\right)$ if it satisfies

$$\begin{array}{c}\hfill log\left(p\right(\lambda x+(1-\lambda )y\left)\right)\ge \lambda log\left(p\right(x\left)\right)+(1-\lambda )log\left(p\right(y\left)\right)\end{array}$$

for all $\lambda \in (0,1)$ and all $x,y$ in the support of X.

Log-concavity of the density is a well-known property, which is satisfied by a large number of remarkable distributions including the normal distribution.

Table 1. Some log-concave densities.

Distribution	Kernel of $\mathit{p}\left(\mathit{x}\right)$	Condition
Normal	$exp\left\{-\frac{{(x-\mu )}^2}{2{\sigma }^2}\right\}$	$\sigma >0$
t-distribution	${(1+\frac{1+x^2}{v})}^{-(v+1)/2}$	$v\ge 3$
Gamma	$x^{\alpha -1}{e}^{-\beta x}$	$\alpha \ge 1,\beta >0$
Beta	$x^{a-1}{(1-x)}^{b-1}$	$a\ge 1,b\ge 1$
Logistic	$e^{-\frac{x-\mu }{\sigma }}/{[1+e^{-\frac{x-\mu }{\sigma }}]}^2$	$\sigma >0$
Gumbel	$e^{-\frac{x-\mu }{\sigma }}exp\{-e^{-\frac{x-\mu }{\sigma }}\}$	$\sigma >0$
Weibull	$x^{\alpha -1}{e}^{-\beta x^{\alpha }}$	$\alpha \ge 1,\beta >0$

lists some log-concave densities, where the kernel of $p\left(x\right)$ ignores some constant in $p\left(x\right)$ so that the condition for log-concavity of $p\left(x\right)$ remains the same. The exponential distribution is the case of gamma distribution with $\alpha =1$, and the uniform distribution $U(0,1)$ is the special case of beta distribution with $a=1,b=1$.

Example 1.

A finite mixture of distributions allows for great flexibility in capturing a variety of density shapes. Research into mixture models has a long history. The most cited early publication is Pearson [26], as he used a two-component normal mixture model for a biometric data set. The density of a mixture of two normal distributions, denoted by $X\sim MixN(\alpha ,{\mu }_1,{\sigma }_1^2;{\mu }_2,{\sigma }_2^2)$, is

$$\begin{array}{ccc}\hfill p\left(x\right)& =& \alpha \phi (x;{\mu }_1,{\sigma }_1^2)+(1-\alpha )\phi (x;{\mu }_2,{\sigma }_2^2)\hfill \\ & =& \alpha \frac{1}{\sqrt{2\pi }{\sigma }_1}exp\left\{-\frac{{\left(x-{\mu }_1\right)}^2}{2{\sigma }_1^2}\right\}+(1-\alpha )\frac{1}{\sqrt{2\pi }{\sigma }_2}exp\left\{-\frac{{\left(x-{\mu }_2\right)}^2}{2{\sigma }_2^2}\right\}.\hfill \end{array}$$

Li et al. [27] gave a detailed study on several aspects of the distribution: “unimodal or bimodal”, “measure of disparity of two normals”, and “uniqueness of MSE-RPs”. Generally, the uniqueness of MSE-RPs is not always true, but under some conditions it is true. For example, for a location mixture of two normal densities with ${\sigma }_1={\sigma }_2=\sigma$, a set of MSE-RPs is is unique if $|{\mu }_1-{\mu }_2|\le \sqrt{2}\sigma$ for all $\alpha \in (0,1)$.

2.2. Asymptotic Behavior of MSE-RPs

There are a lot of studies on the asymptotic behavior of MSE-RPs; for example, see Zador [28], Su [29], Graf and Luschgy [18], and Pagès [7]. It is well-known that the distribution tail gives strong inference on statistical inference. According to different standards, there are many kinds of classification methods for statistical distribution. Embrechts et al. [30] classified the distributions based on the convergence rate of pdf $p\left(x\right)$ as $x\to \infty$, and they defined the so-called heavy-tailed distribution and light-tailed distribution, in which the exponential distribution is used as a standard for comparison. The following formal definitions are from Foss et al. [31].

Definition 5.

The univariate random variable X with the distribution function F is said to have a heavy tail if

$$\begin{array}{c}\hfill {\int }_{-\infty }^{+\infty }{e}^{\lambda x}\mathrm{d}F\left(x\right)=\infty ,for all\lambda >0.\end{array}$$

Otherwise, F is said to have a light tail if

$$\begin{array}{c}\hfill {\int }_{-\infty }^{+\infty }{e}^{\lambda x}\mathrm{d}F\left(x\right)<\infty ,for some\lambda >0.\end{array}$$

Obviously, any univariate random variable supported on a bounded interval is light-tailed. In fact, this definition can intuitively reflect that the tail of a heavy-tailed distribution is heavier than the tail of the exponential distribution. Moreover, the long-tailed distribution is an important subclass of heavy-tailed distribution and is more commonly used in applications. The formal definition of a long-tailed distribution was given by Foss et al. [31] as follows.

Definition 6.

The univariate random variable X with distribution function F is said to be long-tailed if

$$\begin{array}{c}\hfill \underset{x\to \infty }{lim}P(X>x+t|X>x)=1for all t>0,\end{array}$$

or equivalently

$$\begin{array}{c}\hfill \underset{x\to \infty }{lim}\frac{\overline{F}(x+t)}{\overline{F}\left(x\right)}=1for all t>0,\end{array}$$

where $\overline{F}\left(x\right)=P(X>x)$.

Xu et al. [32] studied the limiting behavior of the gap between the largest two representative points of a statistical distribution and obtained another kind of classification for the most useful univariate distributions. They illustrate the relationship between RPs and the concepts of doubly truncated mean residual life (DMRL) and mean residual life (MRL), which are widely used in survival analysis. Denote

$$\begin{array}{c}\hfill {\rho }_k={\xi }_k^{\left(k\right)},{\Delta }_i^{\left(k\right)}={\xi }_i^{\left(k\right)}-{\xi }_{i-1}^{\left(k\right)},i=2,\dots ,k.\end{array}$$

They consider three kinds of distributions according to the domain of distribution, i.e., $\mathbb{R}$, ${\mathbb{R}}^{+}$, and finite interval.

Table 2. Limit behavior of ${\Delta }_k^{\left(k\right)}$ for some distributions on $\mathbb{R}$.

Distribution	Limiting Value of ${\mathbf{\Delta }}_{\mathit{k}}^{\left(\mathit{k}\right)}$
Normal distribution	0
Logistic distribution	$2\beta$
Student’s t distribution	$+\infty$

shows limiting value of ${\Delta }_k^{\left(k\right)}$ of the normal, t, and logistic distributions. Their density functions are

$$\begin{array}{ccc}& & p(x;\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi }\sigma }exp\left\{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2\right\},\mu \in \mathbb{R},{\sigma }^2>0;\hfill \\ & & p(x;\alpha ,\beta )=\frac{exp(-(x-\alpha )/\beta )}{\beta {(1+exp(-(x-\alpha )/\beta ))}^2},\alpha \in \mathbb{R},\beta >0;and\hfill \\ & & p(x;\nu )=\frac{\Gamma \left(\frac{\nu +1}{2}\right)}{\sqrt{\nu \pi }\Gamma \left(\frac{\nu }{2}\right)}{\left(1+\frac{x^2}{\nu }\right)}^{-\frac{\nu +1}{2}},\nu >2,\hfill \end{array}$$

respectively. It is surprising that the normal distribution and t distribution have such different behavior, although the normal distribution is the limiting distribution of the student’s t distribution as $\nu \to \infty$.

Table 3. Limit behavior of ${\Delta }_k^{\left(k\right)}$ for some distributions on ${\mathbb{R}}^{+}$.

Distribution	Limiting Value of ${\mathbf{\Delta }}_{\mathit{k}}^{\left(\mathit{k}\right)}$
Weibull distribution ($m>1$)	0
Weibull distribution ($0<m<1$)	$+\infty$
Gamma distribution	$2\beta$
Exponential distribution	$2\beta ,$ for all $n\ge 2$
F-distribution	$+\infty$
Beta prime distribution	$+\infty$
Log-normal distribution	$+\infty$
Inverse Gaussian distribution	$\frac{2{\mu }^2}{\sigma }$

presents limiting value of ${\Delta }_k^{\left(k\right)}$ for many useful distribution on ${\mathbb{R}}^{+}$. These distributions include the Weibull distribution with density

$$\begin{array}{ccc}& & p(x;\sigma ,m)=\frac{m}{\sigma }{x}^{m-1}{e}^{-{(x/\sigma )}^m},\sigma >0,m>1;\hfill \\ & & p(x;\sigma ,m)=\frac{m}{\sigma }{x}^{m-1}{e}^{-{(x/\sigma )}^m},\sigma >0,0<m<1,\hfill \end{array}$$

the Gamma and exponential distributions with respective densities

$$\begin{array}{ccc}& & p(x;\alpha ,\beta )=\frac{1}{{\beta }^{\alpha }\Gamma \left(\alpha \right)}{x}^{\alpha -1}{e}^{-x/\beta },\alpha >0,\beta >0;\hfill \\ & & p(x;\beta )=\frac{1}{\beta }{e}^{-x/\beta },\beta >0;\hfill \end{array}$$

the density of the F-distribution with degrees of freedoms $d_1$ and $d_2$

$$\begin{array}{c}\hfill p(x;d_1,d_2)=\frac{1}{\mathrm{B}(\frac{d_1}{2},\frac{d_2}{2})}{\left(\frac{d_1}{d_2}\right)}^{\frac{d_1}{2}}{x}^{\frac{d_1}{2}-1}{\left(1+\frac{d_1}{d_2}x\right)}^{-\frac{d_1+d_2}{2}},d_1>0,d_2>4;\end{array}$$

the Beta prime distribution with density

$$\begin{array}{c}\hfill p(x;\alpha ,\beta )=\frac{1}{\mathrm{B}(\alpha ,\beta )}\frac{x^{\alpha -1}}{{(1+x)}^{\alpha +\beta }},\alpha >0,\beta >2;\end{array}$$

the lognormal distribution with density

$$\begin{array}{c}\hfill p(x;\mu ,{\sigma }^2)=\frac{1}{x\sigma \sqrt{2\pi }}exp\left(-\frac{{\left(lnx-\mu \right)}^2}{2{\sigma }^2}\right),\mu \in \mathbb{R},\sigma >0;\end{array}$$

and the inverse Gaussian distribution with density

$$\begin{array}{c}\hfill p(x;\mu ,\sigma )=\sqrt{\frac{\sigma }{2\pi x^3}}exp(-\frac{\sigma {(x-\mu )}^2}{2{\mu }^{2}x}),\mu >0,\sigma >0.\end{array}$$

Observing on these results [32] gave Theorem 5.

Theorem 5.

If the univariate random variable X supported on ${\mathbb{R}}^{+}$ is long-tailed, then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\Delta }_n^{\left(n\right)}=+\infty .\end{array}$$

For the distributions on the finite interval $[m,M]$, Xu et al. [32] gave a systematic study including the following result.

Theorem 6.

Suppose that a random variable X has continuous probability density function $p\left(x\right)$ on $[m,M]$ and $\mathbb{E}\left(X^2\right)<+\infty$. Let $m\le {\xi }_1^{\left(k\right)}<...<{\xi }_k^{\left(k\right)}\le M$ be the k MSE-RPs of X. If $p({\xi }_k^{\left(k\right)}-y)$ converges uniformly to $f(M-y)$, $y\in [0,M-m]$, then

$$\begin{array}{c}\hfill \underset{k\to \infty }{lim}{\Delta }_k^{\left(k\right)}=0,\end{array}$$

provided that the above limit exists.

3. Algorithms for Generation of MSE-RPs of Univariate Distributions

Generation of MSE-RPs is very important for applications. This section reviews algorithms for the generation of univariate distributions. To minimize the mean square error (5) is an optimization problem, including some difficulties:

• The objective function is multivariate on simplex $\{{\xi }_1^{\left(k\right)}<{\xi }_2^{\left(k\right)}<\cdots <{\xi }_k^{\left(k\right)}\}$;
• The objective function might be not differentiable in the whole domain;
• The minimum of the objective function is not unique, and the objective function may have more local minimums on the domain.

This kind of problems can not be directly solved by the classical optimization methods (such as the downhill simplex method, quasi-Newton methods, and conjugate gradient methods) for most of distributions.

There are three main different approaches for the generation of RPs:

(a)

Theoretic approach or combining the theoretic approach and computational calculation;

(b)

Applying the k-means method finds approximate RPs, and this approach can be applied to all of univariate and multivariate distributions;

(c)

To solve a system of nonlinear equations.

Approach (a) can be used for very few distributions, such as the uniform distribution in a finite interval. [33] proposed the method for finding MSE-RPs of the exponential and Laplace distributions by combining the theoretic approach and computational calculation.

Approach (b) applies the k-means methods to any continuous univariate and multivariate distributions. The traditional k-means algorithm needs a set of n observations from the underlying distribution $F\in {\mathbb{R}}^d$ and the user needs to cluster those observations into k groups under a loss function $\varphi$. The k-means algorithm begins with k arbitrary centers. Each observation is then assigned to the nearest center, and each center is recomputed as the center of mass of all points assigned to it. These steps (assignment and center calculation) are repeated until the process stabilizes. One can check that the total error $\varphi$ is monotonically decreasing, which ensures that no clustering is repeated during the course of the algorithm. The mean square error (MSE), see Definition 1, has been popularly used as an error $\varphi$.

It seems to us that Polard [34] was the first one to propose this approach. Along this line, Lloyd [35] proposed two trial-and-error methods. This approach is easy to implement, but it needs to choose a good quality initial and a large number of training samples. There are two kinds of k-means algorithms: nonparametric and parametric k-means algorithms. If the population distribution is known, the training samples are from the known population distribution and the corresponding k-means algorithm is parametric; otherwise, the underlying distribution is unknown and the corresponding k-means algorithm is nonparametric. Usually, the parametric k-means algorithm is more accurate for most univariate distributions.

The parametric k-means algorithm

(1)

For given pdf $p\left(x\right)$, the number of RPs: k, and $t=0$, input a set of initial points $y_1^{\left(t\right)}<y_2^{\left(t\right)}<\cdots <y_k^{\left(t\right)}$. Determine a partition of $\mathbb{R}$ as

$$\begin{array}{c}\hfill I_i^{\left(t\right)}=(a_i^{\left(t\right)},a_{i+1}^{\left(t\right)}],i=1,\dots ,k-1,I_k^{\left(t\right)}=(a_{k-1}^{\left(t\right)},a_k^{\left(t\right)}),\end{array}$$

where

$$\begin{array}{c}\hfill a_1^{\left(t\right)}=-\infty ,a_i^{\left(t\right)}=(y_{i-1}^{\left(t\right)}+y_i^{\left(t\right)})/2,i=2,\dots ,k,a_k^{\left(t\right)}=\infty .\end{array}$$

(2)

Calculate probabilities

$$\begin{array}{c}\hfill p_j^{\left(t\right)}={\int }_{I_j^{\left(t\right)}}p\left(x\right)dx,j=1,\dots ,k;\end{array}$$

and the condition mean

$$\begin{array}{c}\hfill y_j^{(t+1)}=\frac{{\int }_{I_j^{\left(t\right)}}xp\left(x\right)dx}{{\int }_{I_j^{\left(t\right)}}p\left(x\right)dx}=\frac{{\int }_{I_j^{\left(t\right)}}xp\left(x\right)dx}{p_j^{\left(t\right)}}.\end{array}$$

(3)

If two sets of $\left\{y_j^{\left(t\right)}\right\}$ and $\left\{y_j^{(t+1)}\right\}$ are identical, the process stops and deliver $\left\{y_j^{\left(t\right)}\right\}$ as MSE-RPs of the distribution with probabilities $\left\{p_j^{\left(t\right)}\right\}$; otherwise, let $t:=t+1$ and go back to Step (1).

Stampfer and Stadlober [36] called this algorithm the self-consistency algorithm as the output set of RPs is self-consistent (not necessarily MSE-RPs).

Approach (c) was proposed by Max [9] and Fang and He [10] based on the traditional optimization for minimizing the mean square error function $Y=f(z_1,\dots ,z_k)$ with respect to $z_1,\dots ,z_k$, where the objective function (5) is differentiable, taking partial derivative of Y with respect to $z_1,\dots ,z_k$ and constructing a system of equations. Its solutions, denoted by $y_1,\dots ,y_k$, might be the global minimum of Y, i.e., MSE-RPs. For $k\ge 3$ there are three kinds of equations in (13), (14), and (15), respectively. Fang and He [10] gave the conditions for the solution to be unique under the normal distribution.

Theorem 7.

Taking partial derivative of Y with respective to to $z_1,\dots ,z_k$, we have three kinds of equations:

1.

For any $z_1>0$, for the equation

$$\begin{array}{c}\hfill (z_1-\mu )F\left(\frac{z_1+z_2}{2}\right)=-\frac{1}{2b}(z_1+z_2)p\left(\frac{z_1+z_2}{2}\right)\end{array}$$

(13)

there exists a solution $z_2$ if and only if $z_1<\mu$.

2.

For given $z_{i-1}>0,i=2,\cdots ,k-1$, for the equation

$$\begin{array}{ccc}\hfill & & (z_i-\mu )[F\left(\frac{z_i+z_{i+1}}{2}\right)-F\left(\frac{z_{i-1}+z_i}{2}\right)]\hfill \\ & & =\frac{1}{2b}(z_{i-1}+z_i)f\left(\frac{z_{i-1}+z_i}{2}\right)-\frac{1}{2b}(z_i+z_{i+1})p\left(\frac{z_i+z_{i+1}}{2}\right)\hfill \end{array}$$

(14)

there exists a solution $z_{i+1}$ when $z_{i-1}<z_{i,i-1}$, where $z_{i,i-1}$ is the $(i-1)$th representative point in the set of MSE-RPs which has $k=i$.

3.

For any $z_{k-1}>0$, for the equation

$$\begin{array}{c}\hfill (z_k-\mu )[1-F\left(\frac{z_{k-1}+z_k}{2}\right)]=\frac{1}{2b}(z_{k-1}+z_k)p\left(\frac{z_{k-1}+z_k}{2}\right)\end{array}$$

(15)

there exists a solution $z_k$.

The Fang–He algorithm has been applied to many univariate distributions. Max [9] and Fang and He [10] obtained sets of MSE-RPs of $N(0,1)$ for $k\le 36$ and $k\le 31$, respectively. Fu [37] applied the Fang–He algorithm to the gamma distribution $\Gamma (\frac{3}{2},1)$ and obtained MSE-RPs for $k\le 20$. Ke et al. [38] gave a more advanced study on MSE-RPs of the gamma distribution. Zhou and Wang [39] studied the t distribution with 10 degrees of freedom and gave MSE-RPs for $k\le 35$. Fei [40] proposed an algorithm for generating MSE-RPs by Newton optimization algorithm. Li et al. [27] gave a detailed study on MSE-RPs of the mixture normal distributions. Fei [41] studied the class of Pearson distributions where the pdf of X has the form of

$$\begin{array}{c}\hfill p\left(x\right)=cexp\left\{-\int \frac{a_0+a_{1}x}{b_0+b_{1}x+b_2{x}^2}dx\right\},\end{array}$$

(16)

where c is the normalized constant and parameters $a_0,a_1,b_0,b_1,b_2$ satisfy the differential equation

$$\begin{array}{c}\hfill \frac{dp\left(x\right)}{dx}=\frac{a_0+a_{1}x}{b_0+b_{1}x+b_2{x}^2}p\left(x\right).\end{array}$$

The class of Pearson distributions includes many useful distributions. For example, type I is the beta distribution; type II is the symmetrical U-shaped curve; type III is the shifted gamma distribution; type V is the shift inverse gamma distribution; type VI is the inverse beta distribution; type VII is the t distribution; type VIII is the power function distribution; type X is the exponential distribution; and type XI is the normal distribution. Fei [41] gave some sufficient conditions for the uniqueness of the solution.

Comparisons of the three kinds of generations of MSE-RPs: The approach (a) obviously is the best but only for a few distributions. The approach (b) can be applied to any continuous univariate and multivariate distributions. For the generation of univariate MSE-RPs, the parametric k-means algorithm does not need the training sample. Many authors have used this algorithm with a good initial set of points. The approach (c) can find the most accurate MSE-RPs of univariate distributions, but it needs a heavy computational calculation if k is larger.

4. Stochastic Simulation and Resampling

Stochastic simulation has played an important role in statistical research and applications. The traditional simulation employs a set of random samples from the underlying distribution by some computational software.

4.1. Estimation of the Mean, Variance, Skewness, and Kurtosis

Fang et al. [42] first proposed a way of simulation to take random samples from $F_{QMC}$ and $F_{MSE}$ instead of the underlying distribution F and found that the statistical simulation and resampling for estimation of the mean, variance, skewness, and kurtosis under $F_{MSE}$ have the best performance. The second is under $F_{QMC}$. Xu et al. [32] gave comparisons above simulation and resampling accuracy for estimation of the mean, variance, skewness, and kurtosis under a lot of distributions: the uniform normal, logistic, t, F, beta prime, gamma, lognormal, and inverse Gaussian distributions. Yang et al. [43] considered more approximation distributions to F in simulation. All of their results show advantages based on $F_{MSE}$ and $F_{QMC}$.

Example 2.

Li et al. [27] gave comparison estimators (mean, variance, skewness, and kurtosis) and density estimators of a mixture of two normal distributions via the four kinds of RPs. Here, the revised MC (RMC) is as follows. Denote distribution $F_{RMC}$ by a sorted random samples $x_1\le \dots \le x_k$ as support points with related probabilities

$$\begin{array}{ccc}\hfill p_1^{\left(RMC\right)}& =& {\int }_{-\infty }^{\frac{x_1+x_2}{2}}p\left(x\right)dx,p_2^{\left(RMC\right)}={\int }_{\frac{x_1+x_2}{2}}^{\frac{x_2+x_3}{2}}p\left(x\right)dx,\hfill \\ \hfill \cdots \\ \hfill p_i^{\left(RMC\right)}& =& {\int }_{\frac{x_{i-1}+x_i}{2}}^{\frac{x_i+x_{i+1}}{2}}p\left(x\right)dx,i=2,\dots ,k-1,p_k^{\left(RMC\right)}={\int }_{\frac{x_{k-1}+x_k}{2}}^{\infty }p\left(x\right)dx.\hfill \end{array}$$

Consider the mixture of two normal distributions $N(-8,3^2)$ and $N(3,4^2)$ with $\alpha =0.6$; that is, its pdf is

$$\begin{array}{c}\hfill p\left(x\right)=\alpha \phi (x;{\mu }_1,{\sigma }_1^2)+(1-\alpha )\phi (x;{\mu }_2,{\sigma }_2^2),\end{array}$$

(17)

where $\phi (x;0,1)$ is the pdf of $N(0,1)$. When $k=15$, the density estimations by the four methods are given in Figure 1, and the $L_2$-distances between $p\left(x\right)$ and $p_Y\left(x\right)$ by the four methods are given in

Table 4. $L_2$-distance between the population density and approximate densities by 4 methods.

k	MC	RMC	QMC	MSE
15	0.0676	0.0453	0.0171	0.0151
30	0.0674	0.0316	0.0129	0.0126
35	0.0615	0.0284	0.0124	0.0122

. Comparing their performances, we can see that MSE > QMC > RMC > MC.

4.2. Consistency

There are many criteria for stochastic simulation/resampling, including unbiasedness, efficiency (minimum variance), and consistency.

Let $\left\{X_1,\dots ,X_n\right\}$ be a random sample of size n from a population distribution F and let $Z_n$ be a random-sample-based estimator of the parameter of interest $\theta$. Denote a random variable of interest as $T\left(X_1,\dots ,X_n;F\right)=n^{1/2}\left(Z_n-\theta \right)$ depending upon the underlying distribution F. Let

$$H_n\left(x\right)=P\left(T(X_1,\dots ,X_n;F)\le x\right)$$

(18)

be the cdf of $n^{1/2}(Z_n-\theta )$. Let $\widehat{F}$ be an approximation distribution of F and let $\left\{Y_1,\dots ,Y_n\right\}$ be a random sample of size n from $\widehat{F}$. A new plug-in estimate of $T\left(X_1,\dots ,X_n;F\right)$ is defined as

$$T\left(Y_1,\dots ,Y_n;\widehat{F}\right)=n^{1/2}({\tilde{Z}}_n-{\tilde{\theta }}_k),$$

where ${\tilde{\theta }}_k$ is a plug-in estimate of $\theta$ under $\widehat{F}$ and ${\tilde{Z}}_n$ is the counterpart of $Z_n$ based on i.i.d. samples $Y_1,\dots ,Y_n$. The sampling distribution of $T\left(Y_1,\dots ,Y_n;\widehat{F}\right)$ is denoted as

$${\tilde{H}}_n^{\left(k\right)}\left(y\right)=P\left(T(Y_1,\dots ,Y_n;\widehat{F})\le y\right),$$

(19)

which is used to approximate the sampling distribution $H_n\left(x\right)$ defined in (18).

In order to show that the proposed resampling method is effective for simulation, we need to demonstrate that ${\tilde{H}}_n^{\left(k\right)}$ converges to $H_n$ and show ${\tilde{H}}_n^{\left(k\right)}$ is closed to $H_n$ in a certain sense. There are several ways to define a distance between $H_n$ and ${\tilde{H}}_n^{\left(k\right)}$. Shao and Tu [44] provided a comprehensive review of the theoretical properties of the consistency of bootstrapping. Xu et al. [45] extended this study when the approximation distribution is chosen by $F_{mse,k}$ (the approximate distribution formed by k MSE-RPs of F). They employ the Kolmogorov metric (20) and the Mallows–Wasserstein metric (21) to measure the distance between $H_n$ and ${\tilde{H}}_n^{\left(k\right)}$. If one of these distances converges to 0, then we say that ${\tilde{H}}_n^{\left(k\right)}$ is consistent with $H_n$.

Definition 7.

Let G and K be two distribution functions in $\mathcal{F}=\left\{\mathit{all}\mathit{distributions}\mathit{on}\mathbb{R}\right\}$. The Kolmogorov’s distance between G and K is

$${\rho }_{\infty }(G,K)=\underset{-\infty <x<\infty }{sup}|G\left(x\right)-K\left(x\right)|.$$

(20)

Definition 8.

For two distributions G and K in ${\mathcal{F}}_2$ where

$${\mathcal{F}}_2=\left\{F\in \mathcal{F}:{\int }_{-\infty }^{\infty }{\left|x\right|}^{2}dF\left(x\right)<\infty \right\},$$

their Mallows–Wasserstein’s distance is

$${\rho }_2(G,K)=\underset{{\mathcal{T}}_{U,V}}{inf}{\left(E{|U-V|}^2\right)}^{1/2},$$

(21)

where ${\mathcal{T}}_{U,V}$ is the collection of all possible joint distributions of the pairs $(U,V)$, whose marginal distributions are G and K, respectively.

The convergence under the Mallows–Wasserstein metric (21) is stronger than the convergence in distribution. Xu et al. [45] consider the consistency of sample mean and sample variance of $F_{mse,k}$. Let $X_1,\dots ,X_n$ be independent observations on F. Denote sample mean as $\overline{X}$, and sample variance as $S^2$. Let $Y_1,\dots ,Y_n$ be independent observations on $F_{mse,k}$. Denote sample mean as $\overline{Y}$ and sample variance as ${\tilde{S}}^2$. The following results are from Xu et al. [45].

Theorem 8.

Under the above notation, the distribution ${\tilde{H}}_n^{\left(k\right)}$ of $n^{1/2}(\overline{Y}-{\tilde{\mu }}_k)$ converges to the distribution $H_n$ of $n^{1/2}(\overline{X}-\mu )$, where ${\tilde{\mu }}_k$ is the mean of $F_{mse,k}$ and μ is the mean of F, i.e.,

(i).

${\rho }_2({\tilde{H}}_n^{\left(k\right)},H_n)\to 0$ as $k\to \infty$;

(ii).

${\rho }_{\infty }({\tilde{H}}_n^{\left(k\right)},H_n)\to 0$ as both $n\to \infty$ and $k\to \infty$.

Corollary 1.

Given a set of k MSE-RPs from F, where $k\in {\mathbb{N}}^{+}$. For any sample size n, we have

$${\rho }_2({\tilde{H}}_n^{\left(k\right)},H_n)\le E^{1/2}\left[{(X-Y_k^{mse})}^2\right].$$

Corollary 2.

Given a set of k MSE-RPs from F, where $k\in {\mathbb{N}}^{+}$. Denote ${\sigma }^2$ as the variance of F and ${\tilde{\sigma }}_k^2$ as the variance of $F_{mse,k}$. Then, we have

$$\underset{n\to \infty }{lim\; sup}{\rho }_{\infty }({\tilde{H}}_n^{\left(k\right)},H_n)\le {\rho }_{\infty }\left(\Phi \left(\frac{x}{\sigma }\right),\Phi \left(\frac{x}{{\tilde{\sigma }}_k}\right)\right),$$

(22)

where Φ is the cumulative distribution function of the standard normal distribution.

The sampling distribution of $n^{1/2}(\overline{Y}-{\tilde{\mu }}_k)$ is a valid estimation of the sampling distribution of $n^{1/2}(\overline{X}-\mu )$ according to Theorem 8. As shown in Corollary 1, the distance between the two sampling distributions under the Mallows–Wasserstein metric is always bounded by the square root of the mean squared error of k MSE-RPs from the underlying distribution F regardless of the sample size n. Moreover, Corollary 2 explains that when $n\to \infty$, the Kolmogorov’s distance between the two sampling distributions is bounded by the distance between two normal distributions with different standard errors. The distance on the right-hand side of (22) is determined by the number of representative points k. We should notice that the Kolmogorov’s distance between two sampling distributions converges to 0 as both n and k increase. In practice, the number of representative points k should be as large as possible.

Theorem 9.

Let $X\sim F$ and $Y_k^{mse}\sim F_{mse,k}$ on the domain $\mathcal{X}$.

(i) 

If a function $g:\mathcal{X}\to \mathbb{R}$ is continuous, then

$$g\left(Y_k^{mse}\right)\to g\left(X\right)in distribution;$$

(ii) 

If g is also bounded, then

$$\underset{k\to \infty }{lim}E\left[g\left(Y_k^{mse}\right)\right]=E\left[g\left(X\right)\right].$$

Specifically, part (i) of Theorem 9 proves that the convergence of MSE-RPs is preserved by continuous functions, and part (ii) shows that the MSE-RPs can be used to estimate the expectation of $g\left(X\right)$ consistently for any continuous and bounded function g.

Theorem 10.

Let $X\sim F$ and $Y_k^{mse}\sim F_{mse,k}$. If there exists some $r\in {\mathbb{N}}^{+}$ and $r\ge 3$ such that $E\left({\left|X\right|}^{r+\epsilon }\right)<\infty$ with some $ϵ>0$, then

(i) 

$Y_k^{mse}$ converges to X in rth raw moment, i.e.,

$$\underset{k\to \infty }{lim}E\left[{\left(Y_k^{mse}\right)}^r\right]=E\left(X^r\right);$$

(ii) 

$Y_k^{mse}$ converges to X in rth absolute central moment, i.e.,

$$\underset{k\to \infty }{lim}E\left({|Y_k^{mse}-\mu |}^r\right)=E\left({|X-\mu |}^r\right),$$

where $\mu =E\left(Y_k^{mse}\right)=E\left(X\right)$.

Theorem 10 is essential for establishing the consistency of the sample rth moment under $F_{mse,k}$, including the sample variance under $F_{mse,k}$.

4.3. Confidence Interval Estimation

Confidence intervals are usually constructed by considering a pivotal quantity $t_n=T(X_1,\dots ,X_n;F)$ whose distribution is known. Suppose $\theta$ is a parameter of interest. Consider a studentized pivot $t_n=({\widehat{\theta }}_n-\theta )/{\widehat{\sigma }}_n$, where ${\widehat{\theta }}_n$ is an estimator of $\theta$ and ${\widehat{\sigma }}_n$ is an estimator of the standard error of ${\widehat{\theta }}_n$. Denote the distribution of $t_n$ as $H_n$. Assume the inverse of $H_n$ is unique, an exact $100(1-\alpha )\%$ confidence interval for $\theta$ is

$$\left[{\widehat{\theta }}_n-{\widehat{\sigma }}_n{H}_n^{-1}(1-\alpha /2),{\widehat{\theta }}_n-{\widehat{\sigma }}_n{H}_n^{-1}(\alpha /2)\right].$$

(23)

In the traditional asymptotic approach, we replace the unknown $H_n$ by its limit H. There are two main disadvantages of the traditional asymptotic approach. It requires the limit H to be derived analytically and explicitly, and it lacks high-order accuracy. The bootstrap method can be applied to obtain an easy-to-use confidence interval with higher accuracy by replacing $H_n$ in (23) with its bootstrap estimator $H_{boot}$. According to numerical studies under some basic experimental settings by Xu et al. [45], the proposed resampling method tends to be more accurate for interval estimation than the bootstrap method.

5. Property of MSE-RPs of Multivariate Distributions

Let $\mathit{X}={(X_1,\dots ,X_d)}^T$ be a random vector with cdf $F(x_1,\dots ,x_d)$ and pdf $p(x_1,\dots ,x_d)$. Assume that X exist finite mean vector $E\left(\mathit{X}\right)=\mathit{\mu }$ and covariance matrix $\mathbf{\Sigma }=\mathrm{Cov}\left(\mathit{X}\right)$. A set of MSE-RPs of X is denoted by $\Xi =\{{\mathit{\xi }}_j,j=1,\cdots ,k\}$ that minimizes the mean square error (MSE) and the corresponding vector is $\mathit{\xi }={\mathit{Y}}_{MSE}$ with Voronoi regions $S_j,j=1,\dots ,k$ and probabilities $p_j=P(\mathit{X}\in S_j),j=1,\dots ,k$ (refer to Definition 1). The following results are from Flury [13,14].

Theorem 11.

Under the above assumption on X we have

1.

When $k=1$ the MSE-RP is given by $\mathit{\xi }=\mathit{\mu }$;

2.

$E\left({\mathit{Y}}_{MSE}\right)=E\left(\mathit{X}\right)={\sum }_{j=1}^k{p}_j{\mathit{\xi }}_j$, i.e., $E\left(\mathit{X}\right)$ is in the convex hull of ${\mathit{\xi }}_1,\dots ,{\mathit{\xi }}_k$;

3.

MSE-RPs are self-consistent and

$$\mathrm{MSE}\left({\mathit{Y}}_{MSE}\right)=tr(\mathbf{\Sigma })-tr\left({\mathbf{\Sigma }}_{\mathbf{\xi }}\right),$$

where ${\mathbf{\Sigma }}_{\mathbf{\xi }}={\sum }_{j=1}^k{p}_j({\mathbf{\xi }}_j-\mathit{\mu }){({\mathbf{\xi }}_j-\mathit{\mu })}^T$ is the covariance matrix of ${\mathit{Y}}_{MSE}$;

4.

The rank of $({\mathbf{\xi }}_1-\mathit{\mu },\dots ,{\mathbf{\xi }}_k-\mathit{\mu })<k$.

Theorems 2 and 4 can be easily extended to the multivariate case, but Theorem 4 needs some change in linear relation for extension below.

Theorem 12.

Let ${\mathit{X}}_1$ and ${\mathit{X}}_2$ be two random vectors in ${\mathbb{R}}^d$ with relation ${\mathit{X}}_2=\mathbf{a}+b\mathbf{H}{\mathit{X}}_1$, where $\mathbf{a}\in {\mathbb{R}}^d$, $b\in \mathbb{R}$ and $\mathbf{H}$ is an orthogonal matrix of order d. We have

(a) 

If $\{{\mathit{y}}_1,\dots ,{\mathit{y}}_k\}$ is a set of self-consistent points of $X_1$, then $\{\mathbf{a}+b\mathbf{H}{\mathit{y}}_1,\dots ,\mathbf{a}+b\mathbf{H}{\mathit{y}}_k\}$ is a set of self-consistent points of ${\mathit{X}}_2$;

(b) 

If $\{{\mathit{\xi }}_1,\dots ,{\mathit{\xi }}_k\}$ is a set of MSE-RPs of ${\mathit{X}}_1$, then $\{\mathbf{a}+b\mathbf{H}{\mathit{\xi }}_1,\dots ,\mathbf{a}+b\mathbf{H}{\mathit{\xi }}_k\}$ is a set of MSE-RPs of ${\mathit{X}}_2$.

There are various kinds of symmetry in multivariate distributions, among which the class of elliptically symmetric distributions is an extension of the multivariate normal distribution and includes so many useful distributions. For a comprehensive study, refer to Fang et al. [46].

Definition 9.

Spherically and elliptically symmetric distributions. A d-dimensional random vector X is said to have an elliptically symmetric distribution (ESD), or elliptical distribution for short, if X has the following stochastic representation (SR)

$$\begin{array}{c}\hfill \mathit{X}\stackrel{d}{=}\mathit{\mu }+R{\mathbf{\Psi }}^{1/2}{\mathit{U}}^{\left(d\right)},\end{array}$$

(24)

where random variable $R\ge 0$ is independent of ${\mathit{U}}^{\left(p\right)}$ that is uniformly distributed on the unit sphere in ${\mathbb{R}}^p$, $\mathit{\mu }\in {\mathbb{R}}^d$, Ψ is a positive definite matrix of order d (not necessary to be $\mathrm{Cov}\left(\mathit{X}\right)$ ), and ${\Psi }^{1/2}$ is the positive definite square root of Ψ. We write $\mathit{X}\sim ES{D}_p(\mathit{\mu },\Psi ,g)$ if X has a density of the form

$$\begin{array}{c}\hfill p\left(\mathit{x}\right)={|\mathbf{\Psi }|}^{1/2}g\left({(\mathit{x}-\mathit{\mu })}^T{\mathbf{\Psi }}^{-1}(\mathit{x}-\mathit{\mu })\right),\end{array}$$

where g is called as the density generator. When $\mathit{\mu }=\mathbf{0},\Psi =\mathit{I}$, X has a spherical distribution with the stochastic representation

$$\begin{array}{c}\hfill \mathit{X}\stackrel{d}{=}R{\mathit{U}}^{\left(d\right)},\end{array}$$

(25)

and write $\mathit{X}\sim S\left(g\right)$, where $g\left({\mathit{x}}^T\mathit{x}\right)$ is the density of X.

In general, an elliptical/spherical distribution is not necessary to have a density. For example, ${\mathit{U}}^{\left(d\right)}$ does not have a density in ${\mathbb{R}}^d$. If the distribution of X is spherical and $P(\mathit{X}=\mathbf{0})=0$, then $\parallel \mathit{X}\parallel \stackrel{d}{=}R$ and $\mathit{X}/\parallel \mathit{X}\parallel \stackrel{d}{=}{\mathit{U}}^{\left(d\right)}$ are independent. It is known that X defined in (24) has a density if and only if R has a density $f(·)$. The relationship between $g(·)$ and $f(·)$ is given by

$$\begin{array}{c}\hfill f\left(r\right)=\frac{2{\pi }^{d/2}}{\Gamma (d/2)}{r}^{d-1}g\left(r^2\right).\end{array}$$

(26)

Table 5. Density function of some useful ECDs.

Type	Density Function $\mathit{g}\left(\mathit{x}\right)$ in ${\mathbb{R}}^{\mathit{d}}$
Kotz Type	$g\left(\mathit{x}\right)=c{\left({\mathit{x}}^{{}^T}\mathit{x}\right)}^{N-1}\exp [-r{\left({\mathit{x}}^{{}^T}\mathit{x}\right)}^s],$$r,s>0,2N+d>2$
Multinormal	$g\left(\mathit{x}\right)=c\exp (-\frac{1}{2}{\mathit{x}}^{{}^T}\mathit{x})$
Person Type VII	$g\left(\mathit{x}\right)=c{(l+{\mathit{x}}^{{}^T}\mathit{x}/s)}^{-N},N>d/2,s>0$
Multivariate t	$g\left(\mathit{x}\right)=c{(l+{\mathit{x}}^{{}^T}\mathit{x}/s)}^{-(d+q)/2},q>0,\mathrm{an} \mathrm{integer},s>0$
Person Type II	$g\left(\mathit{x}\right)=c{(l-{\mathit{x}}^{{}^T}\mathit{x})}^q,q>0$
Logistics	$g\left(\mathit{x}\right)=c\exp (-{\mathit{x}}^{{}^T}\mathit{x})/{[1+\exp (-{\mathit{x}}^{{}^T}\mathit{x})]}^2$

lists some useful subclasses of the elliptical distributions.

Flury [13] is the first one who found some relationship between the principal components and the MSE-RPs of the elliptical distribution in the following theorems.

Theorem 13.

Suppose $\mathit{X}\sim ES{D}_d(\mathit{\mu },\Psi ,g)$ with mean vector $\mathit{\mu }$, covariance matrix Σ that is proportional to Ψ and density generator g. Then, the two MSE-RPs of X have of the form

$$\begin{array}{c}\hfill {\mathbf{\xi }}_1=\mathit{\mu }+{\lambda }_1\mathbf{\gamma },{\mathit{y}}_2=\mathit{\mu }+{\lambda }_2\mathbf{\gamma },\end{array}$$

(27)

where $\mathbf{\gamma }\in {\mathbb{R}}^d$ is the normalized characteristic vector associated with the largest eigenvalue of Σ, and $\{{\lambda }_1,{\lambda }_2\}$ are the $k=2$ MSE-RPs of the univariate random variable ${\mathbf{\gamma }}^T(\mathit{X}-\mathit{\mu })$. If the MSE-RPs are not unique, they can be chosen as the given form.

Tarpey et al. [47] established a theorem, called the principal subspace theorem, which shows that k principal points of an elliptically symmetric distribution lie in the linear subspace spanned by the first several principal components.

Theorem 14.

Let $\mathit{X}\sim EC{D}_d(\mathbf{0},\mathbf{\Sigma },g)$. If a set of k MSE-RPs of X spans a subspace $\mathcal{V}$ of dimension $q(q\le d)$, then Σ has a set of eigenvectors ${\mathit{\beta }}_1,\dots ,{\mathit{\beta }}_d$ with associated ordered eigenvalues ${\lambda }_1\ge {\lambda }_2\cdots \ge {\lambda }_d$ such that $\mathcal{V}$ is spanned by ${\mathit{\beta }}_1,\dots ,{\mathit{\beta }}_q$.

The principal subspace theorem exploresthe set of MSE-RPs of an elliptical distribution that has a close relationship with its principal components. It is why Flury [13] called the MSE-RPs principal points. Tarpey [48] and Yang et al. [43] proposed ways to generate a set of MSE-RPs of elliptical distributions in several subclasses of elliptical distributions and explore more relationships between the principal components and MSE-RPs. Their studies need algorithms for producing MSE-RPs.

Yang et al. [43] consider numerical simulation for estimation of mean vector and covariance matrix of the elliptical distributions and show that both QMC-RPs and MSE-RPs have better performance than MC-RPs. They also studied the distribution of MSE of MC-RPs for univariate distributions and elliptical distributions and pointed out that MSE of MC-RPs can be fitted by the extreme value distribution. For a random sample with a poor MSE value, it does not expect to have a good result based on this set of random samples.

6. Algorithms of Generation for RPs of Multivariate Distributions

There are a lot of methods for generating a random sample from a given multivariate distribution $F\left(x\right)$. Johnson [49] gave a good introduction to various methods. There are two useful methods: conditional decomposition and stochastic representation.

6.1. Conditional Decomposition

The conditional distribution method changes generation for a multivariate distribution into generation for several conditional univariate distributions. Suppose random vector $\mathit{X}=(X_1,\cdots ,X_d)$ has the cdf $F(x_1,\dots ,x_d)$. Let $F_1\left(x\right)$ be the cdf of $X_1$ and let $F_i\left(x_i\right|x_1,\cdots ,x_{i-1})$ be the conditional distribution of $X_i$ given $X_1=x_1,\cdots ,X_{i-1}=x_{i-1}$. It is known in theory of probability

$$\begin{array}{c}\hfill F(x_1,\dots ,x_p)=F_1\left(x\right)F_2\left(x_2\right|x_1)\cdots F_d\left(x_d\right|x_1,\cdots ,x_{d-1}).\end{array}$$

(28)

Note that each of $F_1\left(x\right),F_2\left(x_2\right|x_1)\cdots$ and $F_d\left(x_d\right|x_1,\cdots ,x_{d-1})$ is a univariate (conditional) distributions. We can apply some methods including the inverse transformation method to generate a random sample from these distributions. Denote a set of random samples from these univariate (conditional) distributions by $z_1,\dots ,z_d$, then $z=z_1\cdots z_d$ is a random sample from X. In particular, when $X_1,\cdots ,X_d$ are independent, $F(x_1,\dots ,x_d)={\prod }_{i=1}^d{F}_{X_i}\left(x_i\right)$, where $F_{X_i}\left(x_i\right)$ is the cdf of $X_i$.

6.2. Stochastic Representation

Let $\mathit{X}\sim F\left(\mathit{x}\right),\mathit{X}\in {\mathbb{R}}^d$. Suppose that X has a stochastic representation

$$\begin{array}{c}\hfill \mathit{X}=\mathit{h}\left(\mathit{Y}\right),\mathit{Y}\sim U\left(C^t\right),t\le d,\end{array}$$

where h is a set of continuous functions on $C^t={[0,1]}^t$ and Y follows the uniform distribution on $C^t$. The Monte Carlo simulation can find a random sample $\mathbf{u}=(u_1,\dots ,u_t)$ from $C^t$. Then, $\mathit{h}\left(\mathbf{u}\right)$ is a random sample from $F\left(\mathit{x}\right)$.

The SR method can be extended to generate a set of QMC-RPs and MSE-RPs. The QMC method employ a set of quasirandom numbers on $C^t$, denoted by $\{{\mathbf{c}}_i,i=1,\dots ,n\},{\mathbf{c}}_i\in C^t$. Set ${\mathit{x}}_i=\mathit{h}\left({\mathbf{c}}_i\right),i=1,\dots ,n$. Then, the set of $\left\{{\mathit{x}}_i\right\}$ is called a set of quasirandom F-numbers, which can be regarded as another kind of RPs of $F\left(\mathit{x}\right)$, i.e., NTM-RPs or QMC-RPs.

Generating a set of MSE-RPs is not possible for most multivariate distributions. If we focus on some class of multivariate distributions that are easily generated by MC or QMC, then the generation of MSE-RPs becomes much easier. One method is the LBG Algorithm by the use of k-means method proposed by Linde et al. [50]. The LBG algorithm requests a training sequence $\mathcal{X}=\{{\mathit{x}}_1,\dots ,{\mathit{x}}_N\}$ from the given distribution $F\left(\mathit{x}\right)$ by a Monte Carlo method, where N is much larger than k and k is the number of RPs for $F\left(\mathit{x}\right)$. The next step chooses a set of initial vectors using the same Monte Carlo method and finds the associated Voronoi partition $\left\{{\mathcal{S}}_j\right\}$ by assigning each $x_i$ to the nearest region of the partition $\left\{{\mathcal{S}}_j\right\}$. Then follow the procedure of the k-means algorithm and iteration steps until reach the stopping rule.

Although the LBG algorithm can reach a local optimal output with a non-increasing MSE, Fang et al. [51] pointed out two problems when applying this algorithm:

(a)

The algorithm gives the local optimum and the results are dependent on the initial points;

(b)

The generation of samples of $F\left(\mathit{x}\right)$ and the calculation of MSE are based on the Monte Carlo method, which is less efficient with the convergence rate $O_p\left(n^{-1/2}\right)$.

Fang et al. [51] revised the LBG algorithm by the use of quasirandom F-numbers in producing the set of training samples and the set of initial vectors. They proposed the so-called NTLBG algorithm for the generation of QMC-RPs of an elliptical distribution.

Recall a spherical distribution $S\left(g\right)$ has a SR $\mathit{X}\stackrel{d}{=}R{\mathit{U}}^{\left(d\right)}$ in (25). If we can find a set of quasirandom numbers of the uniform distribution on $U\left(C^d\right)$ and a set of quasirandom numbers of R, their product can produce a set of QMC-RPs of X. An effective algorithm for generating a set of QMC-RPs on $U\left(C^d\right)$ can refer to Fang and Wang [4]. The latter calls this algorithm as TFWW algorithm. It is easy to see that if $\mathit{X}\sim S\left(g\right)$, then $\mathbf{H}\mathit{X}\sim S\left(g\right)$ for any orthogonal matrix $\mathbf{H}$ of order d. Therefore, if $\{{\mathbf{\xi }}_1,\dots ,{\mathbf{\xi }}_k\}$ is a set of MSE-RPs of X, then $\{\mathbf{H}{\mathbf{\xi }}_1,\dots ,\mathbf{H}{\mathbf{\xi }}_k\}$ is also a set of MSE-RPs of X. That means that the set of MSE-RPs for spherical distributions is quite not unique.

6.3. The NTSR Algorithm for the Generation of a Spherical Distribution

• Generate a set of quasirandom numbers $\{{\mathbf{c}}_j=(c_{j1},\cdots ,c_{jd}),j=1,\cdots ,n\}$ on $C^d$.
• Denote the cdf of R by $F_R\left(r\right)$ and let $F_R^{-1}$ be its inverse function. Compute $r_j=F_R^{-1}\left(c_{jd}\right)$, $j=1,\cdots ,n$.
• Generate a set of quasirandom F-numbers $\{{\mathbf{u}}_j,j=1,\cdots ,n\}$ of the uniform distribution on $U^d$ with the first $(d-1)$-components of ${\mathbf{c}}_j$’s via the TFWW algorithm.
• Then $\{{\mathit{y}}_j=r_j{\mathbf{u}}_j,j=1,\cdots ,n\}$ is a set of quasirandom F-numbers or QMC-RPs of the given spherical distribution $F\left(\mathit{x}\right)$.

This algorithm can be easily extended to generation of quasirandom F-numbers or QMC-RPs for elliptical distributions. The NTLBG algorithm has the following steps:

• Step 1. For a given $F\left(\mathit{x}\right)$, generate a set of quasirandom F-number ${\mathit{x}}_1,\cdots ,{\mathit{x}}_N$ as a training sequence by the NTSR algorithm with a large N.
• Step 2. Set t = 0. For a given k, generate a set of quasirandom F-numbers ${\mathcal{Y}}_t=\{{\mathit{y}}_{i1},\cdots ,{\mathit{y}}_{ik}\}$ of $F\left(x\right)$ as an initial set of output vectors.
• Step 3. Form a partition $\left\{S_i^{\left(t\right)}\right\}$ of $\{{\mathit{x}}_j,j=1,\cdots ,N\}$ such that each ${\mathit{x}}_i$ is assigned to the nearest region of the partition, i.e., ${\mathit{x}}_i\in S_m^{\left(t\right)}$ if $\left|\right|{\mathit{x}}_i-{\mathit{y}}_{tm}\left|\right|\le \left|\right|{\mathit{x}}_i-{\mathit{y}}_{tj}\left|\right|$, $j\ne m$.
• Step 4. Calculate the sample conditional means ${\mathit{y}}_i=E\left[\mathit{x}\right|\mathit{x}\in S_i]$ and form a new set of output vector vectors ${\mathcal{Y}}_{t+1}=\{{\mathit{y}}_{t+1,1},\cdots ,{\mathit{y}}_{t+1,k}\}$, where

  $$\begin{array}{c}\hfill {\mathit{y}}_{t+1,m}=\frac{1}{n_{t+1,m}}\sum _{{\mathit{x}}_j\in S_m^{\left(t\right)}}{\mathit{x}}_j\end{array}$$

  (29)

  and $n_{t+1,m}$ is the number of ${\mathit{x}}_j$ falling in $S_m^{\left(t\right)}$. If ${\mathcal{Y}}_{t+1}={\mathcal{Y}}_t$, deliver ${\mathcal{Y}}_t$ as MSE-RPs and go to Step 6 and $n_{t+1,m}/N$ as estimated probability of $P(\mathit{x}\in S_m)$; otherwise go to the next step.
• Step 5. Let $t:=t+1$ and go to Step 3.
• Step 6. Calculate and deliver MSE

  $$MSE\left({\mathcal{Y}}_t\right)={\int }_{{\mathbb{R}}^p}\underset{1\le j\le k}{min}{(\mathit{x}-{\mathit{y}}_{tj})}^T(\mathit{x}-{\mathit{y}}_{tj})dF\left(\mathit{x}\right),$$

  or

  $$MSE\left({\mathcal{Y}}_t\right)=\sum _{i=1}^N\frac{n_{t+1,m}}{N}{({\mathit{x}}_i-{\mathit{y}}_{t,m})}^T({\mathit{x}}_i-{\mathit{y}}_{t,m})I({\mathit{x}}_i\in S_m^{\left(t\right)})$$

  by its estimate based on the training sequence.

The NTLBG algorithm has used in generation QMC-RPs and MSE-RPs for elliptical distributions ([43,52,53]) and the skew-normal distribution in Yang et al. [43].

7. Applications of QMC-RPs and MSE-RPs

It is known that QMC-RPs have been applied to numerical high-dimensional integration [3], experimental design [5], geometric probability [4], optimization [4], option pricing [7], and big data analysis. Due to the limited space, we cite only some applications.

7.1. Statistical Inference

For statistical estimation, Fang et al. [42] proposed three kinds of RPs (MC-RPS, QMC-RPs, and MSE-RPs) in the estimation of the mean, variance, skewness, and kurtosis as well as density function. Their results show a big potential for the application of QMC-RPs and MSE-RPs in statistical estimation and resampling. Their results also show that MSE-RPs have a better performance than QMC-RPs under the normal distribution. Their observation is true for many other distributions. However, Jiang et al. [54] found that $Y_{QMC}$ of the arcsine distribution have the same mean, variance, skewness, and kurtosis of the population, i.e., QMC-RPs are the best among other approximations. These interesting results motivated Zhou and Fang [55] to consider putting more conditions into a new criterion for construction of RPs, i.e., FM criterion and FM-RPs. There are more criteria for the construction of RPs; for example, Mak and Joseph [56] proposed so-called support points vs. energy distance, and Rover and Friede [57] used the Kullback–Leibler divergence and symmetrized KL-divergence to define a kind of discrete approximation to a continuous distribution F(x). The mean square error is a special case of

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^d}\underset{j=1,\cdots ,k}{min}\left|\right|\mathit{x}-{\mathbf{\xi }}_j{\left|\right|}^{r}d\mathit{x}\end{array}$$

(30)

with $r=2$, where $\left|\right|\mathit{a}\left|\right|$ denotes $l_r$-norm of $\mathit{a}\in {\mathbb{R}}^d$, and the corresponding set of RPs is called set of MrE-RPs. There are a lot of studies under this criteria, especially $r=1$ (see Graf and Luschgy [18]). Yu [58] gave some sufficient conditions to the uniqueness of MSE-RPs.

Liang et al. [52] and Wang et al. [53] proposed some statistics for testing multivariate normality based on MSE-RPs. Their numerical comparisons show that the new goodness-of-fit testing statistics can significantly improve the traditional chi-square test.

7.2. Moment Estimation and Maximum Likelihood Method

Let $x_1,\dots ,{\mathit{x}}_n$ be a sample from the population $F(x;\mathit{\theta })$, where $\mathit{\theta }$ is a set of unknown parameters and one needs to find a good estimator $\widehat{\mathit{\theta }}$ based on this sample. For example, for normal distribution $N(\mu ,{\sigma }^2)$, one needs to estimate the $\mathit{\theta }=(\mu ,{\sigma }^2)$. There are many useful methods for estimation, among which the moment estimator (ME) and maximum-likelihood estimator (MLE) are popular. For the ME, it needs to solve a system of nonlinear equations, and the sequential number-theoretic methods for optimization (SNTO for short) proposed by Fang and Wang [4] is a powerful method for solving the corresponding equations. Note that QMC-RPs have played an important role in the SNTO algorithm.

To find the MLE of $\mathit{\theta }$ is an optimization problem where the objective function (the likelihood function of the random sample) may have many local maximums and the traditional optimization methods may get into a local maximum. Fang and Wang [4] and Li and Fang [59] applied SNTO to find MSE for many univariate distributions.

It is well known that the performances of sample moments highly depend on the quality of the random sample. The sample moments will be less reliable if the sample fails to catch the turning points of population density. One way for ME revision is to improve the representativeness of the random sample used for calculating sample moments. Li and Fang [59] proposed a new concept “QMC-data” via the nonparametric estimator of the quantile proposed by Harrell and Davis [60].

Let $\mathit{x}=\left\{x_1,\dots ,x_n\right\}$ be a random sample of size n from $F\left(x\right)$. Denote $X_{\left(1\right)}\le X_{\left(2\right)}\le \cdots \le X_{\left(n\right)}$ as the order statistics and $F^{-1}\left(q\right)$ as the $q^{th}$ population quantile.

• Step 1. Generate n quasirandom numbers $q_i=(2i-1)/2n$, for $i=1,\dots ,n$, which are uniformly scattered on the interval $(0,1)$. For each $q=q_i$ run the following steps.
• Step 2. The quantile estimator of $F^{-1}\left(q\right)$ based on random sample is proposed to be $Q\left(q\right)$:

  $$SQ\left(q\right)=\sum _{i=1}^n{W}_{n,i}{X}_{\left(i\right)},$$

  (31)

  where

  $$\begin{array}{ccc}\hfill W_{n,i}& =& \frac{1}{\beta \left\{\right(n+1)q,(n+1\left)\right(1-q\left)\right\}}{\int }_{(i-1)/n}^{i/n}{y}^{(n+1)q-1}{(1-\mathit{y})}^{(n+1)(1-q)-1}dy\hfill \\ & =& I_{i/n}\{q(n+1),(1-q)(n+1)\}-I_{(i-1)/n}\{q(n+1),(1-q)(n+1)\}\hfill \end{array}$$

  and $I_x(a,b)$ denotes the incomplete beta function.
• Step 3. Let $z_i=SQ\left(q_i\right)$, for $i=1,\dots ,n$, then $\mathit{x}=\left(x_1,\dots ,x_n\right)$ is replaced by $\mathit{z}=\left(z_1,\dots ,z_n\right)$ for moment estimation. The objective function based on the revised sample is

  $$\tilde{Q}\left(\mathit{\theta }\right|\mathit{z})=\sum _{i=1}^5{(M_i\left(\mathit{\theta }\right)-m_i\left(\mathit{z}\right))}^2.$$

  (32)

Li and Fang [59] call z as QMC-data based on which they obtain a better estimator of $\mathit{\theta }=(\alpha ,{\mu }_1,{\sigma }_1^2,{\mu }_2,{\sigma }^2)$ of a mixture of two normal distributions by ME and/or MLE methods. Chen [61], Chen [62] gave a comprehensive study on estimation including convergency rate and condition for establishing the consistency of MLE based on this popularly used distribution. Qi et al. [63] considered using the density estimation for revising location-biased datasets for reducing biased datasets. The kernel estimation of the density function is a nonparametric method that likes the idea of the QMC-Data structure. Wang et al. [64] proposed a new bias-correction numerical method based on SNTO that can reduce the bias and variance of parameter estimators significantly in three types of extreme value distributions. Many case studies can be found in Qi et al. [63], Li and Fang [59], and Wang et al. [64].

7.3. Geometric Probability

Statistical simulation has been widely used for problems in statistics with no analytic solution. For example, let D be a domain in ${\mathbb{R}}^d$ and E be the output of a stochastic process working on D. One needs to find the distribution of the area or volume of E. Statistical simulations need a set of points that are uniformly scattered on D and on any subarea of D, including E. A set of random samples fails to provide such a set of points. Fang and Wang [4] suggested applying QMC-RPs to two real-life case studies where D is not a rectangle.

7.3.1. Case Study I: Area of Intersection between a Fixed Circle and Several Random Circles

Let K be a unit circle with a center at the origin $\mathbf{o}={(0,0)}^T$. Suppose that $K_1,\dots ,K_m$ are the m random circles centered at $\mathbf{o},\dots ,{\mathbf{o}}_m$ with given radii $r_1,\dots ,r_m$, respectively. The centers ${\mathbf{o}}_1,\dots ,{\mathbf{o}}_m$ are independent and ${\mathbf{o}}_j\sim N_2(\mathbf{0},{\sigma }_j^2{\mathit{I}}_2)$, where ${\sigma }_j>0$ are known. Let $S_m=K\cap (K_1\cup \dots \cup K_m)$, the overlapping region of K and the union of the m random circles $K_1,\dots ,K_m$, and denote $A\left(S_m\right)$ the area of $S_m$. The goal of the problem is to find the distribution of $A\left(S_m\right)$. It is not difficult to find the distribution of the overlapping area with a single random circle; however, there is no analytic solution for the cases when $m\ge 2$.

7.3.2. Case Study II: Area of Random Belts with a Fixed Width Covering a Unit Sphere

Let $S^3$ be a unit sphere in ${\mathbb{R}}^3$ centering at the origin ${(0,0,0)}^T$, and it can be covered by random belts with a fixed width. Each belt is symmetric about a unit circle in $S^3$ centered at the origin ${(0,0,0)}^T$. Let $\mathbf{n}\in {\mathbb{R}}^3$ be a normal direction, then the belt on the surface of $S^3$ with thickness $2h$ can be written as

$$\begin{array}{c}\hfill G_h\left(\mathbf{n}\right)=\{\mathit{w}:|{\mathit{w}}^T\mathit{n}|\le h\},\end{array}$$

(33)

and the area of the belt can be computed by $A(G_h\left(\mathit{n}\right)=4\pi h$. The original problem arises in steel rolling and aims to find the distribution of a roller’s lifespan and find some ways to increase the lifespan.

There are no analytic solutions in both case studies. If one considers a statistical simulation, an NT-net on the unit circle K or the unit sphere in ${\mathbb{R}}^3$ is needed. Fang and Wang [4] employed the statistical simulation over NT-net on $S^3$. They found that QMC-RPs can give the best design for sequential belts that has a significantly longer life. Yang et al. [43] gave comparisons on estimation accuracy between QMC-RPs and MSE-RPs for construction of NT-net on the unit circle and $S^3$, respectively, and concluded that both are working perfectly.

8. Concluding Remarks

The bootstrap method, originally proposed by Efron [1], has found wide applications in statistical theory and practice. This method involves drawing random samples from the empirical distribution, which serves as an approximation to the population distribution $F\left(\mathit{x}\right)$. However, due to the inherent randomness of these samples, the bootstrap method has certain limitations. To overcome this, a natural solution is to construct support points called RPs that offer a more representative characterization of the distribution $F\left(\mathit{x}\right)$ compared to random samples.

This paper discusses three types of RPs: MC-RPs, QMC-RPs, and MSE-RPs, along with their respective approximations. Theoretical foundations and practical applications demonstrate that all of these RPs can be effectively and efficiently utilized for statistical inferences, including estimation and hypothesis testing. In many case studies, MSE-RPs and/or QMC-RPs have shown better performance compared to MC-RPs. QMC-RPs have been widely applied in various fields, including numerical integration in high dimensions, financial mathematics, experimental design, and geometric probability. This paper provides a comprehensive review of the theory and applications of MSE-RPs, with particular emphasis on recent developments. MSE-RPs exhibit significant potential for applications in statistics, financial mathematics, and big data analysis.

However, in the theory of MSE- and QMC-RPs, several open questions remain. For instance, although several new RP construction methods have been proposed, these methods still lack solid theoretical justifications and practical applications. Further research is needed to address these gaps and advance the field.

We are creating a website (https://fst.uic.edu.cn/isci_en/index.htm, accessed on 19 June 2023) where readers can access fundamental knowledge about RPs and MSE-RPs for various univariate distributions in the near future. Additionally, we are in the process of incorporating R software that generates MSE-RPs into the website, which will be available soon. While there are existing monographs such as “Foundations of Quantization for Probability Distributions” by Graf and Luschgy [18] and “Numerical Probability, An Introduction with Applications to Finance” by Pages [7], these works do not specifically focus on applications in statistical inference. Therefore, there is a need for a new monograph that covers recent advancements in both theory and applications. This review article can serve as a valuable resource, providing relevant content and establishing connections for a potential new book in this area.