Methods to Counter Self-Selection Bias in Estimations of the Distribution Function and Quantiles

Abstract

Many surveys are performed using non-probability methods such as web surveys, social networks surveys, or opt-in panels. The estimates made from these data sources are usually biased and must be adjusted to make them representative of the target population. Techniques to mitigate this selection bias in non-probability samples often involve calibration, propensity score adjustment, or statistical matching. In this article, we consider the problem of estimating the finite population distribution function in the context of non-probability surveys and show how some methodologies formulated for linear parameters can be adapted to this functional parameter, both theoretically and empirically, thus enhancing the accuracy and efficiency of the estimates made.

1. Introduction

Distribution function estimation is an important topic in survey research. This approach offers valuable benefits in the context of probability surveys and has been the focus of much research attention in recent years. It is especially useful when the underlying goal is to determine the proportion of values of a study variable that are less than or equal to a certain value. For example, knowledge of the distribution function makes it possible to obtain the reliability function, which is commonly used in life data analysis and reliability engineering [1]. Furthermore, the distribution function allows us to examine whether two samples originate from the same population [2].

Additionally, the finite population distribution function can be used to calculate certain parameters, such as population quantiles. In several areas of study [3,4,5], quantiles are of special interest. For example, rates of extreme pediatric obesity are defined as the body mass index at or above the 99th percentile [6]. In another area, that of ozone concentrations, the 5th percentile is a measure of the baseline condition, while the 95th reflects peak concentration levels [7]. In economics, some variables, such as income, have skewed distributions and in this case quantiles provide a more suitable measure of location than the mean [8,9]. Also in this field, quantiles allow us to obtain measures such as the poverty line and the poverty gap [10,11], as well as inequality parameters indicators such as the headcount index, which measures the proportion of individuals classified as poor within a given population [12]. Other analyses of inequality, such as those focusing on wages or income distribution, also require measures based on percentile ratios [9]. In these cases, estimating the distribution function is also more useful than calculating totals and means [13]. In view of these considerations, some studies have focused on the auxiliary population information available at the estimation stage to gain more accurate values for the distribution function and quantiles [14,15,16,17]. One means of incorporating auxiliary information to develop new estimators of the distribution function is to employ the calibration method, which was originally designed to estimate the total population [18]. An extensive body of research has been conducted in this area, and various implementations of the calibration approach have been applied in the probability survey context to obtain estimators of the distribution function and the quantiles [19,20,21,22,23,24,25,26,27]. The use of calibration techniques has also been considered for estimating the distribution function when a probability survey is subject to non-response [28].

As part of the global commitment to fight poverty and social exclusion, many government agencies wish to know the proportion of the population living below the poverty line, in order to monitor the effectiveness of their policies [29]. One way to obtain this information is to conduct probabilistic surveys, based on representative samples of the target population. The aim of survey sampling theory is to maximize the reliability of the estimates thus obtained.

For a sample to be considered probabilistic and therefore valid for drawing inferences regarding the population, it must be selected under the assumption that all the individuals in the target population have a known and non-null probability of inclusion.

In recent years, alternative data sources to probabilistic samples have been considered, such as big data and web surveys. These approaches offer certain advantages over traditional probability sampling: estimates in near real time may be obtained, data access is easier, and data collection costs are lower. In these non-traditional methods, the data generating process is different and the subsequent analysis is based on non-probability samples. Despite the above advantages, this method also presents serious issues, especially the fact that the selection procedure for the units included in the sample is unknown and so the estimates obtained may be biased, since the sample itself does not necessarily provide a valid picture of the entire population. In other words, the sample is potentially exposed to self-selection bias [30,31].

Many studies of survey sampling have been undertaken to reduce selection bias in the methods used to estimate population totals and means, and this research has been reviewed in [32,33,34,35,36,37], among others. The methods considered include inverse probability weighting [38,39], inverse sampling [34], mass imputation [40], doubly robust methods [31], kernel smoothing methods [41], statistical matching combined with propensity score adjustment [42], and calibration combined with propensity score adjustment [39,43]. However, despite the extensive literature available on using calibration techniques to estimate the distribution function and the population mean under conditions of self-selection bias, little attention has been paid to the development of efficient methods for estimating the population distribution function under these conditions.

To address this research gap, we propose a general framework for drawing statistical inferences for the distribution function with non-probability survey samples when auxiliary information is available. We discuss different methods of adjusting for self-selection bias, depending on the type of information available, applying calibration, propensity score, and statistical matching techniques.

The rest of the paper is organized as follows: in Section 2, we review the estimation of the distribution function from probability and non-probability samples, in order to establish the basic framework and the notation employed. In Section 3, we then propose several estimators for the distribution function, based on calibration, propensity score adjustment (PSA) and statistical matching (SM), taking into consideration that the non-smooth nature of the finite population distribution function produces certain complexities, which are resolved in different ways. The properties of the proposed estimators are described in Section 4, after which we present the results obtained from the simulation studies performed with these estimators. In the final section, we summarize the main conclusions drawn and suggest possible lines of further research in this area.

2. Basic Setup for Estimating the Distribution Function

Let U denote a finite population of size N, $U=\left\{1,\dots ,i,\dots ,N\right\}$. Let $s_V$ be a self-selected sample of size $n_V$, self-selected from U. Let y be the variable of interest in the survey estimation. We assume that $y_k$ is known for all sample units. Our goal is to estimate the distribution function $F_y\left(t\right)$ for the study variable y, which can be defined as follows:

$$F_y\left(t\right)=\frac{1}{N}\sum _{k\in U}\Delta (t-y_k)$$

(1)

where $\Delta (·)$ denotes the Heaviside function, given by:

$$\Delta (t-y_k)=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill \mathrm{if}\mathrm{t}\ge {\mathrm{y}}_{\mathrm{k}}\hfill \\ \hfill 0\hfill & \hfill \mathrm{if}\mathrm{t}<{\mathrm{y}}_{\mathrm{k}}.\hfill \end{array}\right.$$

In the absence of auxiliary information, the distribution function $F_y\left(t\right)$ can be estimated by the naive estimator, defined by

$${\widehat{F}}_{YNa}\left(t\right)=\frac{1}{n}\sum _{k\in s_V}\Delta (t-y_k).$$

(2)

If the convenience sample $s_V$ suffers from selection bias, the above estimator will provide biased results.

Let R be an indicator variable of an element being in $s_V$, such that

$$R_k=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill k\in s_V\hfill \\ \hfill 0\hfill & \hfill k\notin s_V.\hfill \end{array}\right.$$

(3)

If we know $\{R_k:k\in U\}$, the error of ${\widehat{F}}_{YNa}\left(t\right)$ will be:

$${\widehat{F}}_{YNa}\left(t\right)-F_y\left(t\right)=\frac{1}{n}\sum _{k\in U}{R}_k\Delta (t-y_k)-\frac{1}{N}\sum _{k\in U}\Delta (t-y_k)=\frac{1}{f}Cov(R,\Delta (t-y))$$

with $f=n/N$ and ${\displaystyle Cov(R,\Delta (t-y))=\frac{1}{N}\sum _{k\in U}(R_k-{\overline{R}}_N)(\Delta (t-y_k)-F_y\left(t\right)),}$ being ${\overline{R}}_N=\frac{1}{N}{\sum }_{k\in U}{R}_k$.

By applying the expectation of the mean difference, we obtain the selection bias of the estimator, as follows:

$$B=E_R({\widehat{F}}_{YNa}\left(t\right)-F_y\left(t\right))=\frac{1}{f}{E}_R\left(Cov(R,\Delta (t-y))\right)$$

where $E_R$ denotes the expectation with respect to the random mechanism for $R_k$.

The mean squared error is obtained by:

$$ECM=\frac{1}{f^2}{E}_R\left(Cov{(R,\Delta (t-y))}^2\right)=\frac{1}{f^2}{E}_R\left(Corr{(R,\Delta (t-y))}^2\right)Var\left(R\right)Var(\Delta (t-y))=$$

$$=E_R\left(Corr{(R,\Delta (t-y))}^2\right)\times \left(\frac{1}{f}-1\right)\times Var(\Delta (t-y))$$

because $Var\left(R\right)=\frac{1}{N}{\sum }_{k\in U}{(R_k-{\overline{R}}_N)}^2=f(1-f).$

Therefore, a non-probability sampling design with $E_R\left(Corr{(R,\Delta (t-y))}^2\right)\ne 0$ means that the analysis results are subject to selection bias. This is the main problem addressed in our study.

3. Proposed Estimators

The key to successful weighting to eliminate bias in self-selection surveys lies in the use of appropriate auxiliary information. To address this question, let us consider J auxiliary variables $x_1,\dots ,x_J$ and let ${\mathbf{x}}_k^{{}^{\prime }}=(x_{1k},\dots ,x_{Jk})$ be the vector of auxiliary variables at unit k.

We distinguish three different cases, called InfoTP, InfoES, and InfoES, depending on the information at hand ([44])

• InfoTP: Only the population vector totals of the auxiliary variables, ${\sum }_U{\mathbf{x}}_k=\mathbf{X}$, are known.
• InfoES: Information is available at the level of a probability sample conducted on the same target population as the non-probability survey, with good coverage and high response rates. The vector of auxiliary variables ${\mathbf{x}}_k$ is known for every unit in this reference sample.
• InfoEP: Information is available at the level of the population U: the vector of auxiliary variables ${\mathbf{x}}_k$ is known for every $k\in U$.

Below, we consider various adjustment methods, depending on the type of information available.

3.1. InfoTP

The calibration method, originally developed by Deville and Särndall [18] for the estimation of totals, can be adapted to estimate the distribution function. This approach enables us to incorporate the auxiliary information available through the auxiliary vector ${\mathbf{x}}_k$ in several ways [19,20,24,25,26,27].

In the case of InfoTP, the calibration can be performed on the totals: given a pseudo-distance $G(.,.)$, and denoting $w_{vk}=N/n_V$, we seek new calibrated weights $w_{kc1}$ that are the solution to the following minimization problem

$$\underset{w_k}{min}\sum _{k\in s_V}G(w_k,w_{vk})$$

(4)

subject to

$$\sum _{k\in s_V}{w}_k{\mathbf{x}}_k=\mathbf{X}.$$

(5)

The resulting calibrated estimator of the distribution function is given by:

$${\widehat{F}}_{Yc1}\left(t\right)=\frac{1}{N}\sum _{k\in s_V}{w}_{kc1}\Delta (t-y_k).$$

(6)

Ref. [18] proposes a family of pseudo-distance $G(.,.)$ with which to develop calibration estimators. One of the distances proposed is the chi-square distance given by

$${\Phi }_s=\sum _{k\in s_V}\frac{{(w_k-w_{vk})}^2}{w_{vk}{q}_k}$$

(7)

where $q_k$ is positive weights that are usually assumed as uniform $1/q_k=1$ although unequal weights $1/q_k$ are sometimes preferred.

The resulting calibrated weights $w_{kc1}$ with the minimization of (7) subject to the conditions (5) are given by:

$$w_{kc1}=w_{vk}+w_{vk}{q}_k\gamma ·{\mathbf{x}}_k$$

(8)

where

$$\gamma ={\left(\mathbf{X}-\sum _{k\in s_V}{w}_{vk}{\mathbf{x}}_k\right)}^T{\left(\sum _{k\in s_V}{w}_{vk}{q}_k{\mathbf{x}}_k{\mathbf{x}}_k^T\right)}^{-1}$$

In the estimation of totals and means, previous research has shown that the exclusive use of calibration fails to eliminate self-selection bias if this approach is not combined with other methods, such as propensity score adjustment (PSA) [39,43]. Thus, in terms of bias reduction, the results of the calibration and PSA combination clearly surpass those obtained with only calibration weighting [43].

In order to incorporate methods such as PSA and to develop new estimators that overcome the problems met with the ${\widehat{F}}_{Yc1}\left(t\right)$ estimator, we consider other scenarios as follows.

3.2. InfoES

Let $s_R$ be a probability sample of size $n_R$ selected from U under a probability sampling design $(s_R,p_R)$ in which ${\pi }_k={\sum }_{s_R\ni k}{p}_R\left(s_R\right)>0$ is the first-order inclusion probability for individual k. The covariates ${\mathbf{x}}_k$ are common to both samples, but we only have measurements of the variable of interest y for the individuals in the convenience sample. The original design weight of the individual k in the reference (probability) sample is denoted by $w_{Rk}=1/{\pi }_k$.

First, we consider a calibration method for reweighting based on the proposal given in [25], calibrating from the pseudo-variable:

$$g_k={\widehat{\beta }}^T{\mathbf{x}}_k\mathrm{for}k=1,2,...N$$

(9)

$$\widehat{\beta }={\left(\sum _{k\in s_V}{\mathbf{x}}_k{\mathbf{x}}_k^T\right)}^{-1}·\sum _{k\in s_V}{\mathbf{x}}_k{y}_k.$$

(10)

The new weights $w_{kc2}$ are obtained by minimizing the chi-square distance (7) subject to the following conditions:

$$\frac{1}{N}\sum _{k\in s_V}{w}_{kc2}\Delta (t_j-g_k)=\frac{1}{N}\sum _{k\in s_R}{w}_{Rk}\Delta (t_j-g_k)j=1,2,\dots ,P$$

(11)

where $t_j$ for $j=1,...P$ are points chosen arbitrarily and where we assume that $t_1<t_2<...<t_P$ and $q_k$ are positive constants.

The resulting calibrated estimator of the distribution function is given by:

$${\widehat{F}}_{Yc2}\left(t\right)=\frac{1}{N}\sum _{k\in s_V}{w}_{kc2}\Delta (t-y_k).$$

(12)

in which the calibrated weights $w_{kc2}$ are given by:

$$w_{kc2}=w_{vk}+w_{vk}{q}_k\frac{\lambda }{N}\Delta ({\mathbf{t}}_P-g_k)$$

(13)

with

$$\lambda =N^2·{\left({\widehat{F}}_{GR}\left({\mathbf{t}}_P\right)-\frac{1}{N}\sum _{k\in s_V}\Delta ({\mathbf{t}}_P-g_k)\right)}^T{\left(\sum _{k\in s_V}{w}_{vk}{q}_k\Delta ({\mathbf{t}}_P-g_k)\Delta {({\mathbf{t}}_P-g_k)}^T\right)}^{-1}$$

and

$$\Delta {({\mathbf{t}}_P-g_k)}^T=\left(\Delta (t_1-g_k),\Delta (t_2-g_k),\dots ,\Delta (t_P-g_k)\right)$$

$${\left({\widehat{F}}_{GR}\left({\mathbf{t}}_P\right)\right)}^T=\left(\frac{1}{N}\sum _{k\in s_R}{w}_{Rk}\Delta (t_1-g_k),\frac{1}{N}\sum _{k\in s_R}{w}_{Rk}\Delta (t_2-g_k),\dots ,\frac{1}{N}\sum _{k\in s_R}{w}_{Rk}\Delta (t_P-g_k)\right)$$

The calibrated weights (13) and the weights $w_{Rk}$ for the samples $s_V$ and $s_R$, respectively, give the same estimates for the distribution function of the pseudo-variable g, when evaluated over the set of points $t_j$.

In the case of InfoES information, the most popular adjustment method in non-probability settings is propensity score adjustment [38,39,43,45,46,47]. This method, developed by [48], can be used to estimate the distribution function, as described below.

Under PSA, it is assumed that each element of U has a probability (propensity) of being selected for $s_V$, which can be formulated as

$${\pi }_k^v=Pr(R_k=1|{\mathbf{x}}_k,y_k)$$

(14)

We assume that the response selection mechanism is ignorable and follows a parametric model:

$${\pi }_k^v=Pr(R_k=1|{\mathbf{x}}_k)=m({\mathbf{x}}_k,\lambda ),$$

(15)

for a known function $m(·)$ with second continuous derivatives with respect to an unknown parameter $\lambda$.

We estimate the propensity scores ${\pi }_k^v$ by using data from both the self-selection and the probability samples. The maximum likelihood estimator (MLE) of ${\pi }_k^v$ is ${\widehat{\pi }}_k^v=m(\widehat{\lambda },{\mathbf{x}}_k)$, where $\widehat{\lambda }$ corresponds to the value of $\lambda$ that maximizes the pseudo-log-likelihood function:

$$\tilde{l}\left(\lambda \right)=\sum _{s_V}log\frac{m(\lambda ,{\mathbf{x}}_k)}{1-m(\lambda ,{\mathbf{x}}_k)}+\sum _{s_R}\frac{1}{{\pi }_k}log(1-m(\lambda ,{\mathbf{x}}_k)).$$

(16)

The resulting propensities can then be used to calculate new weights, $w_k^{PSA}=\frac{1}{{\widehat{\pi }}_k^v}$. Thus, we define the inverse propensity weighting estimator of the distribution function as:

$${\widehat{F}}_{YIPS}\left(t\right)=\frac{1}{N}\sum _{k\in s_V}{w}_k^{PSA}\Delta (t-y_k).$$

(17)

Another PSA-based estimator can be obtained using the weights $w_k^{PSA2}=\frac{1-{\widehat{\pi }}_k^v}{{\widehat{\pi }}_k^v}$ [49]. In this respect, Refs. [39,47] proposed other PSA weights whereby the combined sample ($s_V\cup s_R$) is grouped into g equally-sized strata of similar propensity scores from which an average propensity is calculated for each group.

The estimator (17) can be obtained as a special case of the general framework on inference for the general parameter proposed in [50]. The latter authors present an estimator that uses the propensity score for each individual in the survey weighted by the estimating equation under logistic regression, thus obtaining the asymptotic variance of the estimator.

A third approach to dealing with InfoES information is that of statistical matching, by which imputed values are created for all elements in the probability sample. This method was introduced by [40] and is based on modeling the relationship between $y_k$ and ${\mathbf{x}}_k$, using the self-selected sample $s_V$ to predict $y_k$ for the probability sample. The question then is how to predict the values $y_k$.

To do so, let us assume a working population model $E_m(y/\mathbf{x})=M(\mathbf{x},\beta )$ where $\beta$ is the unknown parameter. We further assume that the population model holds for the sample $s_V$. Using the data from this sample, we can obtain an estimator ${\widehat{\beta }}_v$ which is consistent for $\beta$ under the model assumed. From ${\widehat{\beta }}_v$, we then propose the matching estimator for the distribution function as:

$${\widehat{F}}_{YSM}\left(t\right)=\frac{1}{N}\sum _{k\in s_R}\Delta (t-{\widehat{y}}_k)/{\pi }_k$$

(18)

where ${\widehat{y}}_k=M(\mathbf{x},{\widehat{\beta }}_v)$ is the predicted value of $y_k$ under the above model. The estimator (18) is consistent if the model for the study variable is correctly specified.

A more complex estimator for the distribution function can be constructed using the idea of double robust estimation [31], which is based on the following considerations. Firstly, the propensity score adjusted estimator (17) requires that the propensity score model be correctly specified. Moreover, the imputation-based estimator (18) requires that the working population model be correctly specified. An estimator is called doubly robust if the estimator is consistent whenever one of these two models is correctly specified [51]. Hence, the double robust estimator of the distribution function is defined as:

$${\widehat{F}}_{YDR}\left(t\right)=\frac{1}{N}(\sum _{k\in s_R}\Delta (t-{\widehat{y}}_k)/{\pi }_k+\sum _{k\in s_V}{w}_k^{PSA}(\Delta (t-y_k)-\Delta (t-{\widehat{y}}_k)).$$

(19)

The estimator (19) is double robust because it is consistent if either the model for the participation probabilities or the model for the study variable is correctly specified.

3.3. InfoEP

In the case of InfoEP, an initial possibility is to consider a similar calibrated estimator, based on the proposal given in [25]. The new weights $w_{kc3}$ are obtained by minimizing the chi-square distance 7 subject to the following conditions:

$$\frac{1}{N}\sum _{k\in s_V}{w}_{kc3}\Delta (t_j-g_k)=F_g\left(t_j\right)j=1,2,\dots ,P$$

(20)

where $F_g\left(t_j\right)$ is the finite distribution function of g at the points $t_j,j=1,2,\dots ,P$.

The resulting calibrated estimator of the distribution function is:

$${\widehat{F}}_{Yc3}\left(t\right)=\frac{1}{N}\sum _{k\in s_V}{w}_{kc3}\Delta (t-y_k)$$

(21)

where the calibrated weights $w_{kc3}$ are given by:

$$w_{kc3}=w_{vk}+w_{vk}{q}_k\frac{\theta }{N}\Delta ({\mathbf{t}}_P-g_k)$$

(22)

with

$$\theta =N^2·{\left(F_g\left({\mathbf{t}}_P\right)-\frac{1}{N}\sum _{k\in s_V}\Delta ({\mathbf{t}}_P-g_k)\right)}^T{\left(\sum _{k\in s_V}{w}_{vk}{q}_k\Delta ({\mathbf{t}}_P-g_k)\Delta {({\mathbf{t}}_P-g_k)}^T\right)}^{-1}$$

$${\left(F_g\left({\mathbf{t}}_P\right)\right)}^T=\left(F_g\left(t_1\right),F_g\left(t_2\right),\dots ,F_g\left(t_P\right)\right).$$

${\widehat{F}}_{Yc3}\left(t\right)$ gives perfect estimates for the distribution function of the pseudo-variable g, when evaluated over the set of points $t_j,j=1,2,\dots ,P$.

We define a model-based estimator based on the non-probability sample as

$${\widehat{F}}_{YDR2}\left(t\right)=\frac{1}{N}\left(\sum _{k\in U-s_V}\Delta (t-{\widehat{y}}_k)+\sum _{k\in s_V}\Delta (t-y_k)\right)$$

(23)

and a model-assisted estimator by

$${\widehat{F}}_{YDR3}\left(t\right)=\frac{1}{N}\left(\sum _{k\in U}\Delta (t-{\widehat{y}}_k)+\sum _{k\in s_V}{w}^{PSA}(\Delta (t-y_k)-\Delta (t-{\widehat{y}}_k)\right)$$

(24)

4. Properties of Proposed Estimators

When estimating the distribution function, the estimator considered ${\widehat{F}}_Y\left(t\right)$ should satisfy the following distribution function properties:

(i)

${\widehat{F}}_Y\left(t\right)$ should be continuous on the right;

(ii)

${\widehat{F}}_Y\left(t\right)$ should be monotonically nondecreasing;

(iii)

${\displaystyle \underset{t\to -\infty }{lim}{\widehat{F}}_Y\left(t\right)=0}$;

(iv)

{${\displaystyle \underset{t\to +\infty }{lim}{\widehat{F}}_Y\left(t\right)=1}$.

If an estimator of the distribution function ${\widehat{F}}_Y\left(t\right)$ is a genuine distribution function, i.e., ${\widehat{F}}_Y\left(t\right)$ meets the above conditions, it can be used directly for estimating the quantiles [16]. Specifically, the quantile $\alpha$ can be estimated as:

$${\widehat{Q}}_{\alpha }=inf\{t:{\widehat{F}}_Y\left(t\right)\ge \alpha \}={\widehat{F}}_Y^{-1}\left(\alpha \right)$$

Since the Heaviside function is continuous on the right, it is clear that all the proposed estimators satisfy conditions (i) and (iii).

In general, however, estimator ${\widehat{F}}_{yc1}\left(t\right)$ does not satisfy conditions (ii) or (iv). In order to meet condition (ii), let us consider the specific pseudo-distances $G(.,.)$ that guarantee positive calibrated weights $w_{kc1}>0$. In this respect, Ref. [18] proposed some pseudo-distances which always produce positive weights whilst avoiding extremely large ones. Some of these pseudo-distances may be considered in estimator ${\widehat{F}}_{yc1}\left(t\right)$ in order to satisfy condition (ii). In addition, to meet condition (iv), we can add the constraint:

$$N=\sum _{k\in s_V}{w}_k$$

(25)

to condition (5).

Similarly, conditions (ii) and (iv are not generally met by the estimators ${\widehat{F}}_{Yc2}\left(t\right)$ or ${\widehat{F}}_{Yc3}\left(t\right)$). Regarding condition (ii), and following [25], the weights $w_{kc2}$ and $w_{kc3}$ are always positive if $q_k=c$ for all units in the population. Thus, under the usual uniform choice $1/q_k=1$, both estimators satisfy condition (ii). To meet condition (iv), in the case of estimator ${\widehat{F}}_{Yc2}\left(t\right)$, we can add the constraint (25) to the conditions (11), while, for estimator ${\widehat{F}}_{Yc3}\left(t\right)$, we can take a value $t_P$ that is large enough so $F_g\left(t_P\right)=1$.

The estimator ${\widehat{F}}_{YIPS}\left(t\right)$ based on the weights $w_k^{PSA}$ verifies condition (ii) if weights $w_k^{PSA}\ge 0$, whereas if ${\widehat{F}}_{YIPS}\left(t\right)$ is based on $w_k^{PSA2}$, then it meets condition (ii) when $w_k^{PSA}\le 1$. Consequently, if $0\le w_k^{PSA}\le 1$, the estimator ${\widehat{F}}_{YIPS}\left(t\right)$ based on both $w_k^{PSA}$ and $w_k^{PSA2}$ satisfies condition (ii). Thus, through the model selected to estimate propensities $m(\lambda ,{\mathbf{x}}_k)$, condition (ii) can be met. For example, an extended option in the estimation of propensities is that of the logistic regression model

$$m(\lambda ,{\mathbf{x}}_k)=\frac{exp\left({\lambda }^T{\mathbf{x}}_k\right)}{1+exp\left({\lambda }^T{\mathbf{x}}_{ik}\right)}$$

that verifies the condition $0\le w_k^{PSA}\le 1$. Hence, if we choose this model, condition (ii) is met by ${\widehat{F}}_{YIPS}\left(t\right)$ regardless of whether we use the weight $w_k^{PSA}$ or the weight $w_k^{PSA2}$.

To ensure that condition (iv) is met with the estimator ${\widehat{F}}_{YIPS}\left(t\right)$, it can be divided by the sum of weights, that is, ${\sum }_{k\in s_V}{w}_k^{PSA}$ or ${\sum }_{k\in s_V}{w}_k^{PSA2}$.

Estimator ${\widehat{F}}_{YSM}\left(t\right)$ satisfies condition (ii) but not condition (iv). To ensure the latter, again we can divide ${\widehat{F}}_{YSM}\left(t\right)$ by the sum of its weights, that is, ${\sum }_{s_R}{\pi }_k$.

Finally, whereas the estimator ${\widehat{F}}_{YDR2}\left(t\right)$ satisfies all the conditions, ${\widehat{F}}_{YDR}\left(t\right)$ and ${\widehat{F}}_{YDR3}\left(t\right)$ do not meet conditions (ii) or (iv). Condition (iv) can be met by both ${\widehat{F}}_{YDR}\left(t\right)$ and ${\widehat{F}}_{YDR3}\left(t\right)$ when they are divided by the sum of their respective weights, but these estimators, in general, are not monotonic non-decreasing functions and therefore are not genuine distribution functions. In both cases, we might consider the general procedure described in [16] to obtain a monotonous non-decreasing version of the estimators ${\tilde{F}}_{YDR}\left(t\right)$ and ${\tilde{F}}_{YDR3}\left(t\right)$. However, this procedure always increases the computational cost when estimating quantiles.

5. Simulation Study

In this section, we conduct a Monte Carlo study to compare the efficiency of the estimators presented in Section 3.2. The simulation study was programmed in R and Python. New code was developed to calculate the estimator considered. Python was only used for training and applying the machine learning models in order to take advantage of the package Optuna [52] for hyperparameter optimization. However, R was chosen as the main programming language since the functions wtd.quantile, from the package reldist [53], and qgeneric, from the package flexsurv [54], facilitate the implementation of custom quantiles. To show that the superiority of some estimators depends on the data, we define various setups based on different sampling strategies for the probability and nonprobability samples. In this analysis, only InfoES information is used.

5.1. Data

The dataset used in the simulation was collected between 2011 and 2012, in the Spanish Life Conditions Survey [55]. Using criteria harmonized for all European Union countries, the Living Conditions Survey generates a reference source of statistics on income distribution and social exclusion within Europe. The dataset was filtered to rule out individuals and variables with large quantities of missing data. Following this procedure, the resulting pseudopopulation had a size of $N=28210$.

The following variables were used in the simulation:

- Demographics

• $COM$: 1 if the individual has a computer at home, and 0 otherwise;
• $SEX$: 1 if the individual is male, and 0 otherwise;
• $AGE$: the individual’s age in years;
• $AREAME$: 1 if the individual lives in a medium-density population area, and 0 otherwise;
• $AREALOW$: 1 if the individual lives in a low-density population area, and 0 otherwise.

- Analysis variable

• $INC$: Household expenses in EUR.

Let us consider two setups. In the first, the sampling procedure is the same as that used to select the sample in the Spanish Life Conditions Survey: the probability sample is obtained by stratified cluster sampling, whereby the strata are defined by the NUTS2 regions and the clusters are composed of the households within these regions, extracted with probabilities proportional to the household size. The number of households to be selected, m, is estimated by dividing $n_R$ (the sample size of $s_{R1}$) by the mean household size. For $n_R=2000$, $m=902$. According to this procedure, the final sample size of $s_{R1}$ is $n_{R1}=2003$.

In the second setup, the reference probability sample is drawn by Midzuno sampling with probabilities proportional to the minimum household income necessary for basic subsistence.

To generate the nonprobability sample, $s_V$, the following scenarios were considered:

• SC1: Simple random sampling from the population with $COM=1$
• SC2: Unequal probability sampling from the full pseudopopulation, where the probability of selection for the i-th individual, $p_i$, is given as follows:

  $$p_i=\frac{1}{1+exp(-2COM+0.2SEX+0.01AGE+0.2AREAME+0.4AREALOW)}$$

  (26)
• SC3: Unequal probability sampling from the full pseudopopulation, where the probability of selection for the i-th individual, $p_i$, is given as follows:

  $$p_i={(AGE-1925)}^3/{(1995-1925)}^3$$

  (27)

These participation mechanisms create weights with different models and levels of variability. Figure 1, Figure 2 and Figure 3 show the resulting histogram of propensities.

By this procedure, we obtained nonprobability samples with sizes $n_V=2000,4000$ and 6000.

5.2. Simulation

In each simulation, the following parameters were estimated:

• The quantiles $Q_{0.25}$, $Q_{0.5}$ and $Q_{0.75}.$
• The distribution function $F_y\left(t\right)$ at points $Q_{0.25}$, $Q_{0.5}$ and $Q_{0.75}$.

The following methods for estimating these parameters were compared:

• Naive estimator, using the sample distribution function of the $s_V$ sample to draw inferences.
• The proposed calibrated estimator ${\widehat{F}}_{Yc2}\left(t\right)$ where $t_j$ for $j=1,2,3$ corresponds to $Q_{0.25}$, $Q_{0.5}$, and $Q_{0.75}$.
• The proposed PSA estimator ${\widehat{F}}_{YIPS}\left(t\right)$.
• The proposed SM estimator ${\widehat{F}}_{YSM}\left(t\right)$.
• The proposed DR estimator ${\widehat{F}}_{YDR}\left(t\right)$.

All five demographic variables were considered potential predictors of propensities, and predicted values $y_i$ for both logistic and linear regression models. In addition, a state-of-the-art machine learning method, XGBoost [56], was used as an alternative to these two models in order to evaluate the effect of the method used to estimate propensities and predict values. Refs. [57,58] show that this technique can improve the representativity of self-selection surveys with respect to other prediction methods.

The quantile $\alpha$ is estimated as follows:

$${\widehat{Q}}_{\alpha }=inf\{t:{\widehat{F}}_Y\left(t\right)\ge \alpha \}={\widehat{F}}_Y^{-1}\left(\alpha \right)$$

where ${\widehat{F}}_Y$ is one of the five above estimators of $F_y$.

One thousand simulations were run for each context. The resulting mean bias, standard deviation, and root mean square error were measured in relative numbers to make them comparable across different scenarios. The formulas used for their calculation were:

$$RBias(\%)=\left|\frac{{\sum }_{i=1}^{1000}{\widehat{\theta }}^{\left(i\right)}}{1000}-{\theta }_N\right|·\frac{100}{{\theta }_N}$$

(28)

$$RStandarddeviation(\%)=\sqrt{\frac{{\sum }_{i=1}^{1000}{({\widehat{\theta }}^{\left(i\right)}-\widehat{\overline{\theta }})}^2}{999}}·\frac{100}{{\theta }_N}$$

(29)

$$RMSE(\%)=\sqrt{RBia{s}^2+RS{D}^2}$$

(30)

where ${\widehat{\theta }}^{\left(i\right)}$ is the estimation of a parameter ${\theta }_N$ in the i-th simulation and $\widehat{\overline{\theta }}$ is the mean of the 1000 estimations.

5.3. Results

The relative bias of estimators is shown in

Table 1. Bias (%) for each reference probability sample, parameter, non-probability sampling and size, method and machine learning model (linear/logistic regression or XGBoost).

			Stratified						Proportional
			${\mathit{Q}}_{\mathbf{0}.\mathbf{25}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{75}}$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{25}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{5}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{75}}\right)$	${\mathit{Q}}_{\mathbf{0}.\mathbf{25}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{75}}$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{25}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{5}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{75}}\right)$
SC1 2000	Naive		23.3	17.9	13.4	−32.6	−20.9	−10.0	23.3	17.9	13.4	−32.6	−20.9	−10.0
	Cal	Reg	6.7	12.9	17.7	−0.6	−17.2	−20.7	−10.3	2.0	12.0	40.7	1.4	−5.9
		XGB	0.2	3.6	9.8	7.1	4.2	−0.3	1.1	6.3	9.8	5.5	1.0	−0.4
	PSA	Reg	16.8	14.6	11.1	−23.3	−16.5	−8.2	21.5	18.0	13.9	−30.0	−20.4	−10.1
		XGB	19.6	13.1	10.5	−28.1	−15.6	−8.8	28.4	20.9	12.5	−38.2	−24.4	−10.5
	SM	Reg	−4$\times {10}^8$	25.8	3$\times {10}^8$	−0.6	−17.3	−20.7	−2$\times {10}^9$	−0.8	4$\times {10}^7$	40.7	1.4	−5.9
		XGB	−4.1	−4.2	0.6	7.3	4.1	−0.4	−2.3	−1.2	0.6	5.7	0.9	−0.4
	DR	Reg	−4$\times {10}^8$	25.8	3$\times {10}^8$	−0.6	−17.2	−20.7	−2$\times {10}^9$	−0.9	4$\times {10}^7$	40.7	1.4	−5.9
		XGB	−4.1	−4.3	0.5	7.3	4.2	−0.3	−2.4	−1.2	0.6	5.6	1.0	−0.4
SC1 4000	Naive		23.3	18.1	13.6	−32.7	−21.0	−10.1	23.3	18.1	13.6	−32.7	−21.0	−10.1
	Cal	Reg	−6.3	2.9	13.7	30.6	−1.6	−10.2	−11.2	1.4	11.7	42.9	2.6	−5.2
		XGB	0.2	3.7	9.9	6.9	4.2	−0.4	1.2	6.3	9.9	5.2	1.0	−0.5
	PSA	Reg	16.8	14.6	11.3	−23.4	−16.6	−8.3	21.5	18.0	13.9	−30.0	−20.4	−10.1
		XGB	19.9	14.0	13.0	−28.0	−16.4	−10.4	28.1	22.1	13.7	−37.1	−25.0	−11.4
	SM	Reg	−2$\times {10}^9$	4.3	2$\times {10}^8$	30.6	−1.6	−10.2	−2$\times {10}^9$	−2.3	6$\times {10}^7$	42.9	2.5	−5.2
		XGB	−4.2	−4.2	0.6	7.2	4.1	−0.4	−2.2	−1.2	0.6	5.5	0.9	−0.5
	DR	Reg	−2$\times {10}^9$	4.3	2$\times {10}^8$	30.6	−1.6	−10.2	−2$\times {10}^9$	−2.4	6$\times {10}^7$	42.9	2.6	−5.2
		XGB	−4.1	−4.2	0.6	7.1	4.2	−0.3	−2.3	−1.2	0.6	5.4	1.0	−0.5
SC1 6000	Naive		23.3	18.0	13.5	−32.7	−20.9	−10.0	23.3	18.0	13.5	−32.7	−20.9	−10.0
	Cal	Reg	0.7	8.1	15.8	13.9	−9.9	−15.8	−10.9	1.6	11.7	42.0	2.1	−5.5
		XGB	0.2	3.6	9.7	6.8	4.1	−0.4	1.2	6.2	9.7	5.1	1.0	−0.5
	PSA	Reg	16.8	14.6	11.4	−23.5	−16.5	−8.3	21.4	17.9	13.9	−30.0	−20.3	−10.1
		XGB	20.1	14.6	14.8	−27.6	−17.2	−11.4	27.6	22.5	14.3	−36.0	−25.1	−11.9
	SM	Reg	−7$\times {10}^8$	15.8	1$\times {10}^8$	13.9	−10.0	−15.8	−3$\times {10}^9$	−1.7	9$\times {10}^7$	42.0	2.1	−5.5
		XGB	−4.3	−4.1	0.6	7.0	4.2	−0.4	−2.1	−1.2	0.6	5.4	1.0	−0.5
	DR	Reg	−7$\times {10}^8$	15.8	1$\times {10}^8$	13.9	−9.9	−15.8	−3$\times {10}^9$	−1.8	9$\times {10}^7$	42.0	2.1	−5.5
		XGB	−4.2	−4.2	0.6	7.0	4.2	−0.4	−2.3	−1.3	0.6	5.3	1.0	−0.5
SC2 2000	Naive		12.1	10.2	7.8	−18.2	−11.6	−5.6	12.1	10.2	7.8	−18.2	−11.6	−5.6
	Cal	Reg	−1.4	0.1	5.9	6.7	4.0	−0.4	−0.7	3.1	5.9	5.0	1.1	−0.6
		XGB	−1.5	0.0	5.8	7.1	4.2	−0.3	−0.8	3.2	5.9	5.5	1.0	−0.4
	PSA	Reg	−5.2	−4.3	−2.9	9.5	4.9	2.0	−0.2	0.1	−0.2	0.2	0.1	0.2
		XGB	5.4	2.4	2.5	−8.5	−2.4	−2.1	15.2	10.8	5.9	−22.5	−12.3	−4.7
	SM	Reg	−4.3	−4.0	0.6	6.7	4.0	−0.4	−2.1	−1.1	0.6	5.0	1.0	−0.6
		XGB	−4.1	−4.2	0.5	7.2	4.2	−0.3	−2.3	−1.2	0.6	5.6	0.9	−0.4
	DR	Reg	−4.3	−4.0	0.6	6.7	4.0	−0.4	−2.2	−1.2	0.6	5.0	1.1	−0.6
		XGB	−4.1	−4.3	0.5	7.1	4.2	−0.3	−2.4	−1.2	0.6	5.5	1.0	−0.4
SC2 4000	Naive		11.8	10.1	7.7	−17.8	−11.5	−5.6	11.8	10.1	7.7	−17.8	−11.5	−5.6
	Cal	Reg	−1.4	−0.1	5.8	6.6	4.1	−0.4	−0.7	3.0	5.8	5.0	1.1	−0.5
		XGB	−1.6	−0.2	5.7	7.0	4.2	−0.4	−0.9	3.1	5.8	5.4	1.0	−0.4
	PSA	Reg	−5.4	−4.3	−2.9	9.8	4.9	2.0	−0.5	−0.1	−0.4	0.8	0.2	0.3
		XGB	3.5	2.0	3.2	−5.6	−1.8	−2.7	13.9	10.8	6.4	−20.3	−12.1	−5.2
	SM	Reg	−4.3	−4.0	0.6	6.6	4.0	−0.4	−2.1	−1.1	0.6	4.9	1.1	−0.5
		XGB	−4.1	−4.1	0.6	7.3	4.1	−0.4	−2.2	−1.2	0.6	5.6	0.9	−0.5
	DR	Reg	−4.3	−4.0	0.5	6.6	4.1	−0.4	−2.1	−1.2	0.6	5.0	1.1	−0.5
		XGB	−4.1	−4.2	0.6	7.0	4.2	−0.3	−2.4	−1.2	0.6	5.4	1.0	−0.5
SC2 6000	Naive		11.4	9.9	7.4	−17.3	−11.1	−5.4	11.4	9.9	7.4	−17.3	−11.1	−5.4
	Cal	Reg	−1.5	−0.2	5.6	6.6	4.1	−0.4	−0.7	2.9	5.7	4.9	1.1	−0.5
		XGB	−1.6	−0.3	5.6	7.0	4.2	−0.4	−0.9	3.0	5.6	5.3	1.0	−0.5
	PSA	Reg	−5.3	−4.4	−2.8	9.8	4.9	1.9	−0.6	−0.2	−0.5	0.9	0.3	0.4
		XGB	1.8	1.4	3.6	−3.0	−1.2	−3.1	12.3	10.3	6.5	−18.1	−11.3	−5.1
	SM	Reg	−4.3	−4.0	0.6	6.5	4.1	−0.4	−2.1	−1.1	0.6	4.9	1.1	−0.5
		XGB	−4.2	−4.2	0.6	7.3	4.1	−0.4	−2.2	−1.1	0.6	5.6	1.0	−0.5
	DR	Reg	−4.3	−4.0	0.5	6.6	4.1	−0.4	−2.1	−1.2	0.5	4.9	1.1	−0.5
		XGB	−4.1	−4.2	0.6	7.0	4.2	−0.3	−2.3	−1.2	0.6	5.3	1.1	−0.5
SC3 2000	Naive		9.8	8.9	6.9	−14.2	−10.2	−4.9	9.8	8.9	6.9	−14.2	−10.2	−4.9
	Cal	Reg	−2.1	−0.5	5.3	7.4	4.3	−0.3	−1.3	2.6	5.4	5.5	1.1	−0.6
		XGB	−1.9	−0.5	5.3	7.0	4.2	−0.3	−1.2	2.8	5.3	5.5	1.0	−0.4
	PSA	Reg	1.7	0.7	−0.6	−3.0	−0.6	0.5	3.4	2.3	0.7	−6.1	−2.5	−0.4
		XGB	10.5	5.7	5.3	−14.5	−5.6	−4.0	13.1	9.4	6.6	−17.8	−10.4	−5.0
	SM	Reg	−4.5	−4.4	0.6	7.4	4.3	−0.3	−2.5	−1.2	0.6	5.5	1.1	−0.6
		XGB	−4.1	−4.1	0.6	7.1	4.2	−0.3	−2.4	−1.1	0.6	5.6	1.0	−0.4
	DR	Reg	−4.5	−4.5	0.5	7.4	4.3	−0.3	−2.6	−1.2	0.6	5.5	1.1	−0.6
		XGB	−4.1	−4.2	0.5	7.1	4.2	−0.3	−2.5	−1.2	0.6	5.5	1.1	−0.4
SC3 4000	Naive		10.0	9.0	6.8	−14.3	−10.2	−4.9	10.0	9.0	6.8	−14.3	−10.2	−4.9
	Cal	Reg	−1.9	−0.6	5.2	6.8	4.2	−0.4	−1.1	2.6	5.3	5.0	1.1	−0.5
		XGB	−1.8	−0.6	5.2	6.7	4.2	−0.3	−1.2	2.7	5.2	5.3	1.1	−0.5
	PSA	Reg	1.6	1.0	−0.4	−2.7	−1.0	0.3	3.3	2.7	0.7	−5.8	−2.9	−0.5
		XGB	10.2	5.9	5.8	−14.1	−5.6	−4.6	12.7	9.8	7.2	−16.7	−11.0	−5.5
	SM	Reg	−4.3	−4.2	0.5	6.8	4.1	−0.4	−2.1	−1.1	0.6	5.0	1.1	−0.5
		XGB	−4.2	−4.1	0.5	6.9	4.2	−0.4	−2.4	−1.2	0.6	5.4	1.0	−0.5
	DR	Reg	−4.4	−4.2	0.5	6.8	4.2	−0.4	−2.2	−1.2	0.6	5.0	1.1	−0.5
		XGB	−4.1	−4.2	0.6	6.9	4.2	−0.3	−2.5	−1.2	0.6	5.3	1.1	−0.5
SC3 6000	Naive		10.0	9.0	6.7	−14.4	−10.2	−4.9	10.0	9.0	6.7	−14.4	−10.2	−4.9
	Cal	Reg	−1.8	−0.6	5.2	6.6	4.1	−0.4	−1.1	2.6	5.2	4.9	1.1	−0.5
		XGB	−1.8	−0.6	5.2	6.7	4.2	−0.4	−1.2	2.6	5.2	5.2	1.1	−0.5
	PSA	Reg	1.6	0.9	−0.3	−2.9	-0.8	0.2	3.2	2.5	0.8	−5.8	−2.7	−0.5
		XGB	9.8	6.4	6.5	−13.5	−6.0	−5.3	12.6	10.3	7.7	−16.2	−11.4	−6.0
	SM	Reg	−4.3	−4.0	0.5	6.6	4.1	−0.4	−2.1	−1.2	0.6	4.9	1.1	−0.5
		XGB	−4.2	−4.2	0.6	6.9	4.2	−0.4	−2.3	−1.1	0.6	5.4	1.0	−0.5
	DR	Reg	−4.3	−4.0	0.5	6.6	4.1	−0.4	−2.1	−1.2	0.5	4.9	1.1	−0.5
		XGB	−4.2	−4.2	0.6	6.9	4.2	−0.3	−2.4	−1.1	0.6	5.2	1.1	−0.5

, for all scenarios and sample sizes.

These results show that the performance of the methods is very similar for each of the probability sample selections considered. The following comments refer to the first columns. i.e., those corresponding to the situation in which the probability sample is chosen through a stratified cluster scheme.

The naive estimator for all parameters in Scenario 1, where there is also coverage bias, reflects a very large degree of bias, which is not eliminated by increasing the sample size. The calibration estimator achieves a considerable reduction in the bias when XGBoost is used to predict the values but does not achieve a significant reduction in the bias with linear regression. For some parameters, this bias is even greater than that of the naive estimator.

As expected, in Scenario 1, the PSA-based estimators do not eliminate the self-selection bias, since there is no relationship between the variables of interest and the probability of participation, and the machine learning method used to predict the propensities has little influence. These results are comparable to those reported by [43], who observed that it is important to add covariates related to the study goal in order to make PSA useful.

On the contrary, with the SM method, the ML technique is of determinant importance: the estimators based on linear regression perform very badly, in general, since there is no linear relationship between the values to be predicted and the covariates. However, the XGBoost method works well in the case of nonlinearity and allows us to select the useful covariates in the prediction. A noteworthy finding is the large amount of bias shown by the regression-based estimator for quantiles $Q_{0.25}$ and $Q_{0.75}$, while the version based on XGBoost achieves a very significant error reduction. A very similar pattern of behavior was observed in all cases between the SM and the DR estimators.

With Scenario 2, the estimators present a different behavior pattern. The probability of participation depends on all the covariates, and the PSA method reduces the self-selection bias considerably, in all cases. The ML method has less impact, and the degree of bias reduction achieved is similar in the two methods. Comparable results were obtained with SM and DR, the methods based on calibration. In these cases, the bias reduction in relation to the values obtained with the naive estimator is very large and does not depend on the ML method used. No clear pattern emerged as to which of the methods was the best: for some parameters ($Q_{0.25}$ and $Q_{0.5}$), the calibration method worked better, while for others ($F_y\left(Q_{0.5}\right)$), the PSA achieved the greatest reduction in bias, and in yet others ($Q_{0.75}$) the best estimates were produced by SM and DR.

In Scenario 3, where the probability of participating depends only on the age covariate, the calibration estimators, DR and SM, also performed well, obtaining a good level of bias reduction. The estimator based on PSA with logistic regression was the best of all in this respect, for all parameters. However, when XGBoost was used, this decrease in bias was not observed in some parameters. This may be due to the fact that this ML method is very sensitive to the choice of hyperparameters and in these simulations the default parameters were chosen and no hyperparameter optimization was performed.

Table 2. RMSE (%) for each reference probability sample, parameter, non-probability sampling and size, method, and machine learning model (linear/logistic regression or XGBoost).

			Stratified						Proportional
			${\mathit{Q}}_{\mathbf{0}.\mathbf{25}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{75}}$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{25}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{5}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{75}}\right)$	${\mathit{Q}}_{\mathbf{0}.\mathbf{25}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{75}}$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{25}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{5}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{75}}\right)$
SC1 2000	Naive		23.4	18.0	13.5	32.8	21.0	10.0	23.4	18.0	13.5	32.8	21.0	10.0
	Cal	Reg	24.0	21.8	19.1	54.4	32.2	27.5	21.7	13.4	13.3	59.3	21.6	15.6
		XGB	1.1	3.8	9.9	7.3	4.2	0.4	1.5	6.5	9.9	5.6	1.0	0.5
	PSA	Reg	16.9	14.7	11.2	23.5	16.6	8.3	21.6	18.0	13.9	30.1	20.5	10.1
		XGB	20.3	13.7	11.4	29.0	16.2	9.3	28.7	21.2	12.8	38.7	24.7	10.8
	SM	Reg	9$\times {10}^8$	45.5	7$\times {10}^8$	54.4	32.2	27.5	3$\times {10}^9$	28.4	2$\times {10}^8$	59.3	21.6	15.6
		XGB	4.2	4.3	0.7	7.5	4.2	0.5	2.4	1.3	0.8	5.8	1.0	0.6
	DR	Reg	9$\times {10}^8$	45.4	7$\times {10}^8$	54.4	32.2	27.5	3$\times {10}^9$	28.5	2$\times {10}^8$	59.3	21.6	15.6
		XGB	4.2	4.3	0.7	7.4	4.2	0.4	2.5	1.3	0.7	5.7	1.0	0.6
SC1 4000	Naive		23.4	18.2	13.6	32.8	21.1	10.1	23.4	18.2	13.6	32.8	21.1	10.1
	Cal	Reg	22.3	16.7	15.3	59.4	25.5	19.8	21.4	12.7	13.0	59.6	20.9	14.8
		XGB	0.8	3.7	9.9	7.0	4.2	0.4	1.4	6.4	10.0	5.3	1.0	0.5
	PSA	Reg	16.9	14.7	11.4	23.5	16.6	8.3	21.5	18.0	13.9	30.1	20.4	10.1
		XGB	20.2	14.3	13.5	28.5	16.8	10.6	28.3	22.2	13.8	37.4	25.1	11.5
	SM	Reg	3$\times {10}^9$	35.3	5$\times {10}^8$	59.4	25.5	19.8	3$\times {10}^9$	27.4	3$\times {10}^8$	59.6	20.9	14.8
		XGB	4.3	4.2	0.7	7.3	4.2	0.5	2.3	1.2	0.7	5.7	1.0	0.6
	DR	Reg	3$\times {10}^9$	35.2	5$\times {10}^8$	59.4	25.5	19.8	3$\times {10}^9$	27.4	3$\times {10}^8$	59.6	20.9	14.8
		XGB	4.2	4.2	0.7	7.2	4.2	0.4	2.4	1.3	0.6	5.4	1.0	0.5
SC1 6000	Naive		23.3	18.1	13.5	32.8	20.9	10.0	23.3	18.1	13.5	32.8	20.9	10.0
	Cal	Reg	23.1	19.5	17.4	56.8	29.3	24.2	21.5	13.0	13.0	59.5	21.2	15.1
		XGB	0.6	3.7	9.7	6.8	4.2	0.4	1.3	6.2	9.8	5.2	1.0	0.5
	PSA	Reg	16.9	14.6	11.5	23.5	16.5	8.3	21.4	18.0	13.9	30.1	20.4	10.1
		XGB	20.4	14.8	15.1	27.9	17.4	11.6	27.8	22.6	14.4	36.2	25.2	11.9
	SM	Reg	1$\times {10}^9$	41.0	3$\times {10}^8$	56.8	29.3	24.2	4$\times {10}^9$	27.8	3$\times {10}^8$	59.5	21.2	15.1
		XGB	4.4	4.2	0.7	7.2	4.2	0.5	2.2	1.2	0.7	5.5	1.0	0.6
	DR	Reg	1$\times {10}^9$	41.0	3$\times {10}^8$	56.8	29.3	24.2	4$\times {10}^9$	27.9	3$\times {10}^8$	59.5	21.2	15.1
		XGB	4.2	4.2	0.6	7.1	4.2	0.4	2.4	1.3	0.6	5.3	1.1	0.5
SC2 2000	Naive		12.4	10.4	8.0	18.5	11.8	5.8	12.4	10.4	8.0	18.5	11.8	5.8
	Cal	Reg	1.5	1.0	6.0	6.7	4.0	0.4	0.9	3.2	6.1	5.0	1.1	0.6
		XGB	1.7	1.1	6.0	7.2	4.2	0.4	1.1	3.4	6.0	5.6	1.0	0.5
	PSA	Reg	5.6	4.6	3.2	10.2	5.2	2.2	1.8	1.6	1.3	3.2	1.7	0.8
		XGB	7.3	4.8	4.2	11.5	5.2	3.5	15.8	11.3	6.6	23.6	12.9	5.3
	SM	Reg	4.3	4.0	0.7	6.7	4.0	0.4	2.1	1.1	0.6	5.0	1.1	0.6
		XGB	4.2	4.2	0.7	7.4	4.2	0.5	2.4	1.3	0.7	5.8	1.0	0.5
	DR	Reg	4.3	4.0	0.6	6.7	4.0	0.4	2.2	1.2	0.6	5.0	1.1	0.6
		XGB	4.1	4.3	0.7	7.2	4.2	0.4	2.5	1.3	0.7	5.6	1.0	0.5
SC2 4000	Naive		11.9	10.2	7.8	18.0	11.6	5.6	11.9	10.2	7.8	18.0	11.6	5.6
	Cal	Reg	1.5	0.7	5.8	6.6	4.1	0.4	0.8	3.1	5.9	5.0	1.1	0.5
		XGB	1.6	0.8	5.8	7.1	4.2	0.4	1.0	3.2	5.8	5.4	1.1	0.5
	PSA	Reg	5.5	4.5	3.0	10.1	5.0	2.0	1.2	1.1	0.9	2.3	1.2	0.6
		XGB	5.0	3.9	4.1	8.0	3.9	3.4	14.3	11.0	6.8	21.0	12.4	5.4
	SM	Reg	4.3	4.0	0.6	6.6	4.0	0.4	2.1	1.1	0.6	4.9	1.1	0.5
		XGB	4.2	4.2	0.7	7.4	4.1	0.5	2.3	1.3	0.7	5.7	1.0	0.5
	DR	Reg	4.3	4.0	0.5	6.6	4.1	0.4	2.1	1.2	0.6	5.0	1.1	0.5
		XGB	4.1	4.2	0.6	7.1	4.2	0.4	2.4	1.3	0.6	5.5	1.1	0.5
SC2 6000	Naive		11.5	9.9	7.5	17.4	11.2	5.4	11.5	9.9	7.5	17.4	11.2	5.4
	Cal	Reg	1.5	0.6	5.7	6.6	4.1	0.4	0.8	2.9	5.7	4.9	1.1	0.5
		XGB	1.7	0.6	5.6	7.1	4.2	0.4	1.0	3.0	5.7	5.3	1.1	0.5
	PSA	Reg	5.4	4.4	2.9	10.0	5.0	2.0	1.1	0.9	0.8	2.1	1.0	0.6
		XGB	3.3	3.1	4.1	5.4	2.9	3.5	12.7	10.5	6.7	18.6	11.6	5.3
	SM	Reg	4.3	4.0	0.6	6.6	4.1	0.4	2.1	1.1	0.6	4.9	1.1	0.5
		XGB	4.3	4.2	0.6	7.4	4.1	0.4	2.2	1.1	0.6	5.7	1.0	0.5
	DR	Reg	4.3	4.0	0.5	6.6	4.1	0.4	2.1	1.2	0.5	4.9	1.1	0.5
		XGB	4.2	4.2	0.6	7.0	4.2	0.4	2.3	1.2	0.6	5.3	1.1	0.5
SC3 2000	Naive		10.1	9.1	7.1	14.5	10.4	5.1	10.1	9.1	7.1	14.5	10.4	5.1
	Cal	Reg	2.2	1.1	5.4	7.4	4.3	0.4	1.5	2.8	5.5	5.6	1.1	0.6
		XGB	2.1	1.1	5.4	7.1	4.3	0.4	1.4	2.9	5.5	5.6	1.1	0.5
	PSA	Reg	6.1	4.3	3.3	10.3	5.0	2.3	6.1	4.4	3.2	10.5	5.0	2.2
		XGB	11.7	7.1	6.5	16.3	7.2	4.9	14.2	10.1	7.4	19.6	11.3	5.6
	SM	Reg	4.5	4.5	0.6	7.4	4.3	0.4	2.5	1.2	0.7	5.6	1.1	0.6
		XGB	4.2	4.2	0.8	7.3	4.2	0.5	2.4	1.2	0.8	5.7	1.0	0.6
	DR	Reg	4.5	4.5	0.6	7.4	4.3	0.4	2.6	1.2	0.6	5.6	1.1	0.6
		XGB	4.2	4.2	0.7	7.2	4.2	0.4	2.5	1.2	0.7	5.6	1.1	0.5
SC3 4000	Naive		10.1	9.0	6.9	14.4	10.3	4.9	10.1	9.0	6.9	14.4	10.3	4.9
	Cal	Reg	1.9	0.8	5.3	6.8	4.2	0.4	1.2	2.7	5.3	5.0	1.1	0.5
		XGB	1.9	0.9	5.3	6.8	4.2	0.4	1.3	2.7	5.3	5.3	1.1	0.5
	PSA	Reg	3.9	3.0	2.2	6.8	3.3	1.5	4.6	3.6	2.1	7.9	4.0	1.5
		XGB	10.7	6.7	6.4	15.0	6.5	5.1	13.3	10.2	7.5	17.6	11.4	5.8
	SM	Reg	4.3	4.2	0.6	6.8	4.1	0.4	2.2	1.1	0.6	5.0	1.1	0.5
		XGB	4.2	4.2	0.7	7.1	4.2	0.4	2.4	1.2	0.7	5.5	1.0	0.6
	DR	Reg	4.4	4.2	0.5	6.8	4.2	0.4	2.3	1.2	0.6	5.0	1.1	0.5
		XGB	4.2	4.2	0.6	7.0	4.2	0.4	2.5	1.2	0.6	5.4	1.1	0.5
SC3 6000	Naive		10.0	9.0	6.8	14.5	10.2	4.9	10.0	9.0	6.8	14.5	10.2	4.9
	Cal	Reg	1.8	0.7	5.2	6.6	4.1	0.4	1.1	2.6	5.3	4.9	1.1	0.5
		XGB	1.9	0.8	5.2	6.8	4.2	0.4	1.3	2.7	5.2	5.2	1.1	0.5
	PSA	Reg	3.1	2.4	1.7	5.5	2.6	1.2	4.0	3.1	1.7	7.0	3.5	1.2
		XGB	10.2	6.9	6.9	14.2	6.6	5.5	12.9	10.5	7.9	16.7	11.6	6.2
	SM	Reg	4.3	4.1	0.5	6.6	4.1	0.4	2.1	1.2	0.6	4.9	1.1	0.5
		XGB	4.2	4.2	0.7	7.0	4.2	0.4	2.4	1.1	0.7	5.4	1.1	0.5
	DR	Reg	4.3	4.1	0.5	6.6	4.1	0.4	2.1	1.2	0.5	4.9	1.1	0.5
		XGB	4.2	4.2	0.6	7.0	4.2	0.4	2.5	1.2	0.6	5.3	1.1	0.5

shows the relative RMSE of these estimators, for each scenario.

In Scenario 1, the estimators that use linear or logistic regression are the least efficient, due to the bias that is present. Calibrated SM and DR estimators based on XGBoost improve efficiency by reducing bias. Moreover, the RMSE reduction is very strong in some parameters ($Q_{0.75}$ and $F_y\left(Q_{0.75}\right)$). However, the PSA-based estimates do not produce a significant reduction in RMSE because the propensities cannot be modeled from the covariates.

In Scenarios 2 and 3, all the proposed methods effectively reduce the error in the estimates, with the exception of PSA with XGBoost in some cases, as discussed above.

To determine whether this problem encountered with the XGBoost method in some situations can be resolved with an appropriate choice of hyperparameters, we repeated the simulation using a hyperparameter optimization process based on the Tree-structured Parzen Estimator (TPE) algorithm [59]. In this procedure, the error is estimated by cross-validation on the logistic loss obtained by each possible model over the training data. Accordingly, this process could be replicated in a real-world scenario.

Table 3. Bias (%) including hyperparameter optimization when overfitting.

			Proportional
			${\mathit{Q}}_{\mathbf{0}.\mathbf{25}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{75}}$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{25}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{5}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{75}}\right)$
SC2 2000	Naive		12.1	10.2	7.8	−18.2	−11.6	−5.6
	Cal	Reg	−0.7	3.1	5.9	5.0	1.1	−0.6
		XGB	−0.8	3.2	5.9	5.5	1.0	−0.4
	PSA	Reg	−0.2	0.1	−0.2	0.2	0.1	0.2
		XGB	15.2	10.8	5.9	−22.5	−12.3	−4.7
		XGB (opt)	2.4	2.7	2.1	−4.1	−2.8	−1.4
	SM	Reg	−2.1	−1.1	0.6	5.0	1.0	−0.6
		XGB	−2.3	−1.2	0.6	5.6	0.9	−0.4
	DR	Reg	−2.2	−1.2	0.6	5.0	1.1	−0.6
		XGB	−2.4	−1.2	0.6	5.5	1.0	−0.4

and

Table 4. RMSE (%) including hyperparameter optimization when overfitting.

			Proportional
			${\mathit{Q}}_{\mathbf{0}.\mathbf{25}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{Q}}_{\mathbf{0}.\mathbf{75}}$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{25}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{5}}\right)$	${\mathit{F}}_{\mathit{y}}\left({\mathit{Q}}_{\mathbf{0}.\mathbf{75}}\right)$
SC2 2000	Naive		12.4	10.4	8.0	18.5	11.8	5.8
	Cal	Reg	0.9	3.2	6.1	5.0	1.1	0.6
		XGB	1.1	3.4	6.0	5.6	1.0	0.5
	PSA	Reg	1.8	1.6	1.3	3.2	1.7	0.8
		XGB	15.8	11.3	6.6	23.6	12.9	5.3
		XGB (opt)	3.0	3.2	2.6	5.2	3.4	1.8
	SM	Reg	2.1	1.1	0.6	5.0	1.1	0.6
		XGB	2.4	1.3	0.7	5.8	1.0	0.5
	DR	Reg	2.2	1.2	0.6	5.0	1.1	0.6
		XGB	2.5	1.3	0.7	5.6	1.0	0.5

show the bias and RMSE values for the estimators with this new simulation for Scenario 2 and Setup 2 (the worst scenario for PSA with the default XGBoost method). Similar results were obtained for all other situations, but for reasons of space they are not shown in this paper.

These results clearly show that, by optimizing the hyperparameters, we have considerably reduced the bias and error of the estimators.

6. Discussion

In recent years, the use of survey-based online research has expanded considerably. Web surveys are an attractive option in many fields of sociological investigation due to their low fieldwork costs and rapid data collection. However, this survey mode is also subject to many limitations in terms of accurately representing the target population, and the estimates thus obtained are highly likely to present coverage and/or self-selection bias. Various correction techniques, such as calibration, propensity score adjustment, and statistical matching, have been proposed as a means of reducing or eliminating these forms of bias.

Our paper focuses on the question of estimating the distribution function. This issue is important: the distribution function is a basic statistic underlying many others; for purposes such as assessing and comparing finite populations, it can be more revealing than the use of simple means and totals. Indeed, many previous studies have been undertaken to consider how calibration techniques may be applied to the estimation of the distribution function in the context of a probability survey [19,20,21,22,23,24,25,26,27] and even to overcome the problem of non-response [28]. However, to our knowledge, very few, if any, studies have addressed this issue from the standpoint of a non-probability survey. Accordingly, we analyze the efficiency obtained by certain bias-correction techniques such as calibration, propensity score adjustment, and statistical matching in various situations within a non-probability survey context. In this analysis, we consider the performance of several estimators in terms of reducing self-selection bias, using a representative survey sample as a proxy for the target population. Among the results obtained by the estimators proposed for the distribution function, ${\widehat{F}}_{yc1}\left(t\right)$ needs specific pseudo-distances $G(.,.)$ in order to satisfy condition (ii). The estimator ${\widehat{F}}_{YDR2}\left(t\right)$ is always a genuine distribution function and under favorable conditions, the estimators ${\widehat{F}}_{Yc2}\left(t\right)$ and ${\widehat{F}}_{Yc3}\left(t\right)$ also obtain a genuine distribution function. Moreover, with minor modifications, the estimators ${\widehat{F}}_{YIPS}\left(t\right)$ (under a logistic regression model) and ${\widehat{F}}_{YSM}\left(t\right)$ also satisfy the distribution function conditions. On the other hand, the estimators ${\widehat{F}}_{YDR}\left(t\right)$ and ${\widehat{F}}_{YDR3}\left(t\right)$ are not generally monotonically nondecreasing functions, and therefore when estimating quantiles, an additional process, which increases the computational cost, must be applied. All the estimators included in our proposal can be used under linear and nonlinear models. Self-evidently, ${\widehat{F}}_{YIPS}\left(t\right)$, ${\widehat{F}}_{YSM}\left(t\right)$, ${\widehat{F}}_{YDR}\left(t\right)$, ${\widehat{F}}_{YDR2}\left(t\right)$, and ${\widehat{F}}_{YDR3}\left(t\right)$ are applicable to linear or nonlinear models. While the calibrated estimators ${\widehat{F}}_{Yc2}\left(t\right)$ and ${\widehat{F}}_{Yc3}\left(t\right)$ assume a linear model, due to the pseudo-variable $g_k$, the combination with XGBoost enables them to be used with other models too. Furthermore, ${\widehat{F}}_{Yc2}\left(t\right)$ and ${\widehat{F}}_{Yc3}\left(t\right)$ can cover the nonlinear case through the procedure described in [60]. The behavior of all these estimators is demonstrated through simulation studies.

Although further investigation is needed, our results show that self-selection bias can be greatly reduced by any of the four methods considered, particularly when appropriate covariates and a valid machine learning technique are used, both in estimating propensities and in predicting values. However, our investigation did not enable us to determine which method is best in all situations. Specifically, for each parameter and bias-reduction method, different behavior patterns were obtained. Nevertheless, in general, the calibration method based on XGBoost is fairly efficient in any situation.

Although the methods proposed are shown to be effective in reducing the MSE of quantile and distribution function estimates in various situations, certain limitations exist and must be acknowledged. For the PS-based method, for example, the amount of bias reduction achieved depends on how well the propensity model predictors predict the outcome. If the propensity model is poorly fitted, the PS estimates may even be more biased than naive estimates. This is also the case with estimators based on SM, which need a good model in order to accurately predict the y-values. In addition, for the distribution function, the issue is even more complex; although there is a good linear relationship between y and the covariates x, this relationship is not necessarily transferred to the jump functions $\Delta (t-y)$. In practice, it is often difficult to decide whether the auxiliary variables contain all the components needed to characterize the selection mechanism and the superpopulation model. Therefore, when selecting the covariates and the function of the model, it is essential to use flexible ML techniques.

Finally, the present study does not address the question of the estimation of variance. Plug-in estimators can be used to construct variance estimators from the expression of the asymptotic variance, but the issue is not simple, as the variance depends on the probability of the sample $s_R$ being selected and on the selection mechanism described by the propensity model. In estimating the variance for nonlinear parameters, jackknife and bootstrap techniques [61] might be useful and should be considered in future research in this area.