Optimal Subsampling for Upper Expectation Parametric Regression

Abstract

In classic regression analysis, the error term of the model typically conforms to the requirement of being independent and identically distributed. However, in the realm of big data, it is exceedingly common for the error term to exhibit varying distributions due to discrepancies in data collection timing and sources. In this article, we expand upon and refine the upper expectation parameter regression model, introducing a novel upper expectation loss form, which handles distribution heterogeneity via group-specific ${\mu }_i$, to ensure consistent and efficient parameter estimation. Furthermore, we establish the asymptotic properties of this estimation. To address the challenges posed by big data or the restrictions of privacy, we propose a method utilizing Poisson subsampling to devise a new loss function. Under certain assumptions, this method satisfies the condition of asymptotic normality. Additionally, based on the asymptotic properties, we determine the optimal sampling probability and introduce the optimal subsampling technique. Our sampling method surpasses uniform sampling, which reduces MSE by about $50\%$ compared to uniform sampling, and is straightforward to implement in practical scenarios. Subsequent simulation experiments and real-world examples further demonstrate the effectiveness of our approach.

1. Introduction

Big data usually refers to datasets that are large in different aspects: there can be many observations, many variables, or both. Big data presents new challenges for statistical methods and inference. Excessive data may result in data that cannot be stored and calculated in one machine. In addition, the data may also come from different sources and have inhomogeneities, which would cause failure of traditional statistical methods. Specifically, the question we are interested in is whether it is possible to extract a computationally feasible model that is suitable for data from different times or different sources, or more generally, data with different potential distributions. Our goal is to obtain a robust parameter estimate and a prediction based on the expected upper limit.

Almost all classical statistical models rely on various assumptions, with the most crucial being that the model in question possesses a probability distribution, whether known or unknown. Classical linear expectations and determinant statistics hinge on the certainty of this distribution or model. However, in cases of model heterogeneity, traditional statistical methods may become inapplicable. For instance, the classical maximum likelihood may not exist or be uniquely determined due to the absence of a definitive likelihood function. In addition, the classical least squares estimation is invalid because the parameters are defined by linear expectations. Moreover, classic statistical models, such as linear regression models, may not be well defined, since their identifiability hinges on the average certainty, and the regression function becomes unidentifiable in its absence. Consequently, to attain the objective of statistical inference, it is imperative to devise a novel statistical framework and corresponding methodologies.

In the absence of certainty in distribution, the expectations derived are often nonlinear. Early studies on nonlinear expectations can be traced back to [1] in the realm of robust statistics and [2] in the field of imprecise probability. Over recent decades, theories and methods related to nonlinear expectations have seen substantial development and have gained recognition in application areas such as financial risk measurement and control. Ref. [3] presented a notable instance of nonlinear expectation within the context of backward stochastic differential equations, termed g-expectation. As an extension, ref. [4] introduced g-expectation and its related forms. Within the framework of nonlinear expectations, the most prevalent distribution is the so-called G-normal distribution, which was first put forth by [4]. Moreover, refs. [5,6] established the law of large numbers and the central limit theorem, serving as the theoretical cornerstone for nonlinear expectations.

Addressing statistical problems arising from distribution heterogeneity, ref. [7] examined k-diverse distributions and introduced the upper expectation regression model. Subsequently, ref. [8] advanced a mini-max risk and mini-mean risk regression approach within the context of distribution heterogeneity. Furthermore, ref. [9] introduced the notion of “maximin effects” along with a suitable estimator, evaluating its predictive accuracy from a theoretical perspective in mixture models with either known or unknown group structures. Then, ref. [10] focused on learning models that ensure uniform performance through distributionally robust optimization, incorporating considerations of the worst-case distribution and tail effects.

On the other hand, in the big data era, the rapid proliferation of data introduces fresh obstacles to numerous traditional statistical challenges. Foremost among these is the practical impossibility of utilizing standard computer data storage and analysis techniques. To address this issue, a multitude of statistical and computational methodologies have been devised thus far. The main strategies include subsampling, divide-and-conquer, and online update [11,12,13,14]. In this paper, we primarily consider the method of subsampling.

A central concept of the subsampling approach is to employ nonuniform sampling probabilities, ensuring that data points with higher information content are more likely to be selected. A notable method in this regard is the leverage-score-based subsampling introduced by [15]. Subsequently, ref. [16] suggested an information-driven optimal subsample selection technique specifically for linear models. This technique avoids random sampling and instead selects subsamples deterministically for statistical analysis. Additionally, ref. [17] derived the optimal Poisson subsampling probability for quasi-likelihood estimation and devised a distributed optimal subsampling strategy.

The main contribution of this paper is to improve and develop the upper expectation regression method within the framework of big data or under privacy constraints for models with distribution heterogeneity. This heterogeneity is common in practical applications due to factors such as differences in data sources and environments, which always lead to variations in data distributions, such as the model for the influencing factors of air quality in Section 5. Upper expectation regression differs from classical regression in that it tends to use larger values to predict the response variable and obtain mini-max prediction risk. Unlike the method proposed by [7], we address model heterogeneity by introducing group-specific ${\mu }_i$ values, which allows us to obtain a consistent estimator for $\beta$, thereby avoiding the potential inconsistency issue of beta found in the literature.

Another major contribution is to study the asymptotic theory of mini-max estimates for upper expectation regression under subsampling conditions. And then we provide a method to obtain the optimal subsampling probability based on the asymptotic theory. Furthermore, we employ an effective and robust estimation and prediction method, making sampling more stable and feasible. This is further supported by simulations and real data.

The rest of this paper is organized as follows. The second section briefly reviews the motivation, methods, and theoretical properties of the upper expectation regression. And we improve the upper expectation regression. The subsampling method and asymptotic theory are studied in the third section. Section 4 provides the selection method and specific implementation form of the optimal subsampling probability. In the fifth section, simulation and real data examples are given to prove the effectiveness and feasibility of the proposed method. The proof of the theorem is postponed to the Appendix A.

2. Upper Expectation Regression Model

2.1. Preliminary of Upper Expectation Model

We analyze the given linear regression model, which is expressed as follows:

$$Y={\beta }^{\prime }\mathbf{x}+\epsilon ,$$

(1)

where Y represents a scalar response variable, $\mathbf{x}={(x_1,\dots ,x_p)}^{\prime }$ is a p-dimensional covariate vector, and $\beta ={({\beta }_1,\dots ,{\beta }_p)}^{\prime }$ is a p-dimensional vector comprising unknown parameters. For simplicity, we need the independence assumption. In this way, we need that the conditional expectation of $\epsilon$ with given $\mathbf{x}$ is a constant independent of $\mathbf{x}$. That is,

$$E\left[\epsilon \right|\mathbf{x}]=\mu ,$$

where $\mu$ is a constant when $\mathbf{x}$ is given.

In the classic regression model, it is often assumed that the error $\epsilon$ is an independent and identically distributed random variable. However, in practical applications, due to the different time and sources of data collection, the assumption of independent and identically distributed for error terms may not always hold.

Firstly, we briefly review the k-sample upper expectation linear regression in [7]. The essential difference between this and the classic regression model is that the error term $\epsilon$ has distribution heterogeneity. The possible distribution of the error term forms a set

$$\mathcal{F}=\left\{F_1,\cdots ,F_k\right\},$$

where k is the number of different distributions and it is finite.

Under the framework of sublinear expectations, the distribution of $\epsilon$ can be defined as

$$\mathbb{E}\left[\phi \left(\epsilon \right)\right]=\underset{F\in \mathcal{F}}{sup}{E}_F\left[\phi \left(\epsilon \right)\right],\phi \in C_{l,Lip}.$$

Subsequently, we express the conditional expectations as follows:

$$\overline{\mu }=\mathbb{E}\left[\epsilon \right|\mathbf{x}],\underline{\mu }=-\mathbb{E}[-\epsilon |\mathbf{x}].$$

Given these definitions, we introduce the concept of upper expectation regression:

$$\mathbb{E}\left[Y\right|\mathbf{x}]={\beta }^{\prime }\mathbf{x}+\overline{\mu }.$$

Let ${\left\{\left(y_{ij},x_{ij}\right)\right\}}_{j=1}^n$ be a sample in model (1), where $i=1,\cdots ,k$, which means that the data are divided into k groups. We assume that samples in different groups have different distributions and that samples in the same group have the same distribution. For simplicity, we assume that the number of samples in each group is equal, i.e., $n_1=n_2=\cdots =n_k=n$. In actual situations, the number of samples in each group may be different, but when the number is not much different, we obtain almost the same theoretical results.

In order to achieve the upper expectation loss, we use the empirical version of it. Specifically, the empirical version of the upper expectation loss is

$$Q\left(\beta \right)=\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j=1}^n{\left[y_{ij}-{\beta }^{\prime }{x}_{ij}-\tilde{\mu }\right]}^2.$$

By minimizing the upper expectation loss $Q\left(\beta \right)$, we can obtain the estimator of $\beta$, and we call it the mini-max estimator of $\beta$.

$$\left({\widehat{\beta }}_G,\widehat{\tilde{\mu }}\right)=arg\underset{\beta \in \mathrm{B},\tilde{\mu }\in \mathrm{U}}{min}\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j=1}^n{\left[y_{ij}-{\beta }^{\prime }{x}_{ij}-\tilde{\mu }\right]}^2.$$

We write ${\sigma }_i^2=E\left[{\left({\epsilon }_{ij}-\tilde{\mu }\right)}^2\right]$ and ${\sigma }_{i_{*}}^2={max}_{1\le i\le k}{\sigma }_i^2$.

The following theorem gives the asymptotic normality of the mini-max estimator of $\beta$.

Theorem 1

([7]). We postulate that $E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]$ is a positive definite matrix, with ${\sigma }_{i_{*}}^2$ exceeding ${\sigma }_i^2$ for all $i_{*}\ne i$. As n approaches infinity, it follows that

$$\sqrt{n}\left({\widehat{\beta }}_G-\beta \right)\stackrel{d}{\to }N\left(0,{\sigma }_{i_{*}}^2{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}\right),$$

where $\stackrel{d}{\to }$ denotes convergence in distribution, and $N\left(0,{\sigma }_{i_{*}}^2{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}\right)$ represents the standard normal distribution.

However, the prediction based on $\widehat{\tilde{\mu }}$ may not align with the upper expectation prediction, as $\widehat{\tilde{\mu }}$ might not consistently estimate the upper expectation $\overline{\mu }$. Prior to concluding this section, we propose a two-stage estimation approach to develop a consistent estimator for $\overline{\mu }$ and subsequently formulate a prediction grounded in upper expectation. Utilizing the consistent estimator ${\widehat{\beta }}_G$ derived previously, the second-stage estimator for $\overline{\mu }$ is defined as

$${\widehat{\overline{\mu }}}_S=\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j=1}^n\left(y_{ij}-x_{ij}^{\prime }{\widehat{\beta }}_G\right)$$

Let $c=1-E\left[{\mathbf{x}}^{\prime }\right]{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}E\left[\mathbf{x}\right]$ and

$${\mathsf{\Omega }}^{-1}\left(\mathbf{x}\right)={\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}+{\displaystyle \frac{{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}E\left[\mathbf{x}\right]E\left[{\mathbf{x}}^{\prime }\right]{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}}{c}}$$

We present the following theorem based on these definitions.

Theorem 2

([7]). Given the conditions outlined in Theorem 1, if the sequences $\left\{{\epsilon }_{ij},j=1,\dots ,n\right\}$ and $\left\{{\epsilon }_{sj},j=1,\dots ,n\right\}$ are independent for $i\ne s$, then the second-stage estimator satisfies the asymptotic distribution:

$$\sqrt{n}\left({\widehat{\overline{\mu }}}_S-\overline{\mu }\right)\stackrel{d}{\to }N\left(0,{\sigma }_{*}^2+{\sigma }_{\overline{i}}^{2}E\left[{\mathbf{x}}^{\prime }{\mathsf{\Omega }}^{-1}\left(\mathbf{x}\right)E\left[\mathbf{x}\right]\right]\right),$$

where $\overline{\mu }$ is equivalent to ${\mu }_{\overline{i}}$ for some index $\overline{i}$ within the set $\{1,\dots ,k\}$. Note that $E\left[{\mathbf{x}}^{\prime }\right]E\left[{\mathsf{\Omega }}^{-1}\left(\mathbf{x}\right)\right]E\left[\mathbf{x}\right]$ has been simplified to $E\left[{\mathbf{x}}^{\prime }{\mathsf{\Omega }}^{-1}\left(\mathbf{x}\right)E\left[\mathbf{x}\right]\right]$ for clarity.

The proof of Theorems 1 and 2 can be found in [7].

2.2. Improvement of the Method

We firstly consider the upper expectation loss given in the Section 2.1:

$$\mathbb{E}\left[{\left(Y-{\beta }^{\prime }\mathbf{x}-\tilde{\mu }\right)}^2\right]=\underset{1\le i\le k}{max}{E}_i{\left[Y_i-{\beta }^{\prime }{\mathbf{x}}_i-\tilde{\mu }\right]}^2.$$

In different groups of data, the expectation of the error term in the right of the formula can be different. But in the expectation loss, each group uses the same $\tilde{\mu }$ value. By comparing the loss of each group, the selected data group with the largest expectation loss is likely to be the group for which the expectation of the error term is farthest from $\tilde{\mu }$. This is somewhat unreasonable to some extent.

To solve this problem, we propose a new upper expectation loss

$$\underset{1\le i\le k}{max}{E}_i{\left[Y_i-{\beta }^{\prime }{\mathbf{x}}_i-{\mu }_i\right]}^2,$$

(2)

where ${\mu }_i$ denotes the expectation of the error term of the group i, and $E_i$ denotes the expectation under the distribution of the data in the group i.

Due to the value of ${\mu }_i$ being unknown, in order to proceed smoothly in the following estimation, we can make an a priori estimation of ${\mu }_i$. That is, in each group, we would use simple least squares estimation to approximate the value of ${\mu }_i$ to be ${\widehat{\mu }}_i$, which is

$${\widehat{\mu }}_i=arg\underset{{\mu }_i\in \mathrm{U}}{min}{\displaystyle \frac{1}{n}}\sum _{j=1}^n{\left[y_{ij}-{\beta }^{\prime }{x}_{ij}-{\mu }_i\right]}^2.$$

(3)

In this optimization equation, $\beta$ is also used as a parameter. In other words, there is an estimate of the value of $\beta$, but we do not need an estimate of $\beta$ here, so it is not listed here.

Then, we give a new empirical version of the upper expectation loss:

$$Q_1\left(\beta \right)=\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j=1}^n{\left[y_{ij}-{\beta }^{\prime }{x}_{ij}-{\widehat{\mu }}_i\right]}^2.$$

(4)

By minimizing the upper expectation loss, we can obtain the estimator of $\beta$. We write it as ${\widehat{\beta }}_N$.

For convenience, we use the same notation as in Section 2.1. In this situation, we write ${\sigma }_i^2=E\left[{\left({\epsilon }_{ij}-{\mu }_i\right)}^2\right]$ and ${\sigma }_{i_{*}}^2={max}_{1\le i\le k}{\sigma }_i^2$. The ${\sigma }_i^2$s that appear later are all defined here.

To better express our method, we summarize it in Algorithm 1.

Algorithm 1 The two-stage estimation process for ${\mu }_i$ and $\beta$,

Require:   ${\left\{\left(y_{ij},x_{ij}\right)\right\}}_{j=1}^n$ be a sample in model (1), where $i=1,\cdots ,k$; Ensure:   the estimations ${\widehat{\mu }}_i$ and ${\widehat{\beta }}_N$;    1: for $i=1$ to n do    2:      Use (3) to solve for the parameter estimate ${\widehat{\mu }}_i$;    3: end for    4: Via minimizing the empirical upper expectation loss (4), obtain the estimation ${\widehat{\beta }}_N$;    5: return ${\widehat{\mu }}_i$ and ${\widehat{\beta }}_N$.

Before presenting the asymptotic results, we first make the following assumption about ${\epsilon }_{ij}$ primarily concerning data independence and distributional assumptions.

Assumption 1.

There exists an index decomposition $I_i,i=1,\cdots ,k$ such that when $\left(ij\right)\in I_i$, ${\epsilon }_{i1},\cdots ,{\epsilon }_{i{n}_i}$ are identically distributed with bounded variance, and

$$\sum _{j=1}^n\mathrm{E}{({\epsilon }_{ij}-{\mu }_i)}^{2}1\left\{|{\epsilon }_{ij}-{\mu }_i|>\delta \right\}\to 0,\mathit{every}\delta >0.$$

The following theorem gives the asymptotic normality of the new mini-max estimator of $\beta$.

Theorem 3.

Under Assumption 1 and further assuming $E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]$ is a positive definite matrix, ${\sigma }_{i_{*}}^2>{\sigma }_i^2$ for all $i_{*}\ne i$. Then, when $n\to \infty$, we have

$$\sqrt{n}\left[{\widehat{\beta }}_N-\beta \right]\stackrel{d}{⟶}N\left(0,{\sigma }_{i_{*}}^2{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}\right),$$

where $\stackrel{d}{⟶}$ stands for convergence in distribution, and $N\left(0,{\sigma }_{i_{*}}^2{\left(E\left[\mathbf{x}{\mathbf{x}}^{\prime }\right]\right)}^{-1}\right)$ is a classic normal distribution.

The proof of the theorem is provided in the Appendix A.

3. General Poisson Subsampling

For large datasets, the increase in the amount of the data has caused great difficulties in calculations and storage. In order to solve this problem, we adopt the Poisson subsampling method. We first consider Poisson subsampling in general, that is, we firstly consider the method without specifying the probability of each sample being selected. Then, we provide a sampling method and give a specific implementation method in later chapters.

3.1. Poisson Subsampling Method

We consider the same datasets as those in Section 2.1. Then, let $p_{ij}$ be probability to obtain the jth sample point in the ith group, where $i=1,\cdots ,k$ and $j=1,\cdots ,n$. Let $S_i$ denote the set of observation values and sampling probabilities of the sampled subsample in the ith group. That is,

$$S_i=\{{\delta }_{ij}(x_{ij},y_{ij},p_{ij}),j=1,\cdots ,n\},$$

where ${\delta }_{ij}$ is a random variable with the Bernoulli distribution. We write ${\delta }_{ij}\sim$ Bernoulli$\left(p_{ij}\right)$.

According to the introduction in the previous section, we know that $Q_1\left(\beta \right)$ is the empirical version of the upper expectation loss ${max}_{1\le i\le k}{E}_i{\left[Y_i-{\beta }^{\prime }{\mathbf{x}}_i-{\widehat{\mu }}_i\right]}^2$.

Due to the large amount of data, a natural idea is to perform statistical analysis on the sampled subsets. Specifically, by using the obtained sample set, we define a new weighted upper expectation loss as

$$\underset{1\le i\le k}{max}{E}_i\left[{\displaystyle \frac{\delta }{p}}{\left(Y_i-{\beta }^{\prime }{\mathbf{x}}_i-{\widehat{\mu }}_i\right)}^2\right].$$

Because of the independence of $\delta$ and $(x,Y)$, we know the weighted upper expectation loss is equal to the upper expectation loss. That is,

$$\begin{array}{cc}& \mathbb{E}\left[{\displaystyle \frac{\delta }{p}}{\left(Y-{\beta }^{\prime }\mathbf{x}-\tilde{\mu }\right)}^2\right]=\underset{1\le i\le k}{max}{E}_i\left[{\displaystyle \frac{\delta }{p}}{\left(Y-{\beta }^{\prime }\mathbf{x}-{\widehat{\mu }}_i\right)}^2\right]=\underset{1\le i\le k}{max}{E}_i\left[{\displaystyle \frac{\delta }{p}}\right]E_i\left[{\left(Y-{\beta }^{\prime }\mathbf{x}-{\widehat{\mu }}_i\right)}^2\right]\hfill \\ & =\underset{1\le i\le k}{max}{E}_i\left[{\left(Y-{\beta }^{\prime }\mathbf{x}-{\widehat{\mu }}_i\right)}^2\right]=\mathbb{E}\left[{\left(Y-{\beta }^{\prime }\mathbf{x}-\tilde{\mu }\right)}^2\right].\hfill \end{array}$$

Then, we define the empirical version of the weighted upper expectation loss as

$$Q_1^{*}\left(\beta \right)=\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j\in S_i}{\displaystyle \frac{1}{p_{ij}}}{\left[y_{ij}-{\beta }^{\prime }{x}_{ij}-{\widehat{\mu }}_i\right]}^2=\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j=1}^n{\displaystyle \frac{{\delta }_{ij}}{p_{ij}}}{\left[y_{ij}-{\beta }^{\prime }{x}_{ij}-{\widehat{\mu }}_i\right]}^2.$$

(5)

We can solve the parameter $\beta$ by minimizing the weighted estimation function. It means that

$$\tilde{\beta }=arg\underset{\beta \in \mathrm{B}}{min}{Q}_1^{*}\left(\beta \right).$$

From simple calculations, we obtain

$$Q_1^{*}\left(\beta \right)=\underset{1\le i\le k}{max}{\displaystyle \frac{1}{n}}\sum _{j\in S_i}{\displaystyle \frac{1}{p_{ij}}}\left[y_{ij}^2+{\beta }^{\prime }{x}_{ij}{x}_{ij}^{\prime }\beta +{\widehat{\mu }}_i^2-2{\beta }^{\prime }{y}_{ij}{x}_{ij}-2y_{ij}{\widehat{\mu }}_i+2{\beta }^{\prime }{x}_{ij}{\widehat{\mu }}_i\right].$$

One advantage of Poisson subsampling is that the probability $p_{ij}$ only depends on the dataset ${\left\{(x_{ij},y_{ij})\right\}}_{j=1}^n$. Therefore, the probability $p_{ij}$ can be generated block by block, without using all the data at once. In addition, according to the above formula, we only need to calculate the sum of $y_{ij}^2$, $x_{ij}{x}_{ij}^{\prime }$, $y_{ij}{x}_{ij}$, $y_{ij}$, and $x_{ij}$ in each group. These statistics can be sent to the central machine for calculation, without transmitting the original data of each group. Thereby, it can reduce the time and cost of communication.

3.2. Theoretical Properties

In order to establish asymptotic results of $\tilde{\beta }$, we need the following assumptions.

Assumption 2.

The regression parameter lies in the $l_1$ ball $\mathsf{\Lambda }=\left\{\beta \in {\mathbb{R}}^p:{\parallel \beta \parallel }_1\le C\right\}$, where C is a constant. This means that $\tilde{\beta }$ and ${\widehat{\beta }}_N$ are the inner points of Λ.

Assumption 3.

We assume

$$E_i\left({∥{\mathbf{x}}_i∥}^3\right)<\infty$$

for all $i\in \{1,\cdots ,k\}$.

Assumption 4.

For every $i\in \{1,\cdots ,k\}$, we have

$$E_i\left[{(y_i-{\beta }^{\prime }{\mathbf{x}}_i-{\widehat{\mu }}_i)}^3\right]<\infty .$$

The size of the subsample $r^{*}$ is a random variable, and we have $E\left(r^{*}\right)={\sum }_{i=1}^n{p}_i$. We use r to denote $E\left(r^{*}\right)$. In this article, it is obvious to assume that $r<n$.

Assumption 5.

We assume that ${max}_{j=1,\cdots ,n}{\left(n{p}_j\right)}^{-1}=O_P\left(r^{-1}\right)$.

Assumption 2 is that the neighborhood of the estimated values $\tilde{\beta }$ and ${\widehat{\beta }}_N$ have reasonable properties. Assumption 3 gives several moment conditions for variables. Assumption 4 requires the moment condition of the loss function. Assumptions 3 and 4 are used in the proof of Theorems 4 and 5 as the key moment conditions in the Appendix A. And Assumption 5 puts a limit on the sampling probability of each point.

First, we give the property of $Q_1^{*}\left(\beta \right)$.

Theorem 4.

If the Assumptions 2–5 hold, then we have

$$Q_1^{*}\left(\beta \right)=Q_1\left(\beta \right)+o_p\left(1\right).$$

The proof of Theorem 4 is in the Appendix A. From Theorem 4, we can see that the $Q_1^{*}\left(\beta \right)$ we proposed is reasonable.

Then, we give the asymptotic normality of $\tilde{\beta }$.

Theorem 5.

If Assumptions 2–5 hold, then when $n\to \infty$, $r\to \infty$, we have

$$V^{-1/2}\left(\tilde{\beta }-{\widehat{\beta }}_N\right)\stackrel{d}{⟶}N(0,I).$$

where $V={\mathsf{\Sigma }}_{i_{*}}^{-1}{V}_c{\mathsf{\Sigma }}_{i_{*}}^{-1}$, and

$$V_c={\displaystyle \frac{1}{n^2}}\sum _{j=1}^n{\displaystyle \frac{{\left\{y_{i_{*}j}-{\widehat{\beta }}_N^{\prime }{x}_{i_{*}j}-{\widehat{\mu }}_{i_{*}}\right\}}^2{x}_{i_{*}j}{x}_{i_{*}j}^{\prime }}{p_{i_{*}j}}}-{\displaystyle \frac{1}{n^2}}\sum _{j=1}^n{\left\{y_{i_{*}j}-{\widehat{\beta }}_N^{\prime }{x}_{i_{*}j}-{\widehat{\mu }}_{i_{*}}\right\}}^2{x}_{i_{*}j}{x}_{i_{*}j}^{\prime },$$

$${\mathsf{\Sigma }}_i=n^{-1}\sum _{j=1}^n{x}_{ij}{x}_{ij}^{\prime }.$$

The proof of Theorem 5 is in the Appendix A.

4. Optimal Subsampling Strategies

In this section, we derive the best subsampling probability to obtain a better estimate of ${\widehat{\beta }}_N$. After the theoretical derivation, we give a selection method for actual calculations. We mainly draw on the method of [17].

4.1. Theoretical Method

We would find the optimal subsampling probability according to the result of Theorem 5. That is, we minimize the asymptotic mean square error of $\tilde{\beta }$ approaching ${\widehat{\beta }}_N$. This is equivalent to minimizing $tr\left(V\right)$. This method is called A-optimality in the language of optimal design (See [18]).

Method 1 (A-optimality).

For ease of presentation, define the statistics in the ith group:

$$d_{ij}^{\mathrm{V}}=\left|y_{ij}-{\widehat{\beta }}_N^{\prime }{x}_{ij}-{\widehat{\mu }}_i\right|{∥{\mathsf{\Sigma }}_i^{-1}{x}_{ij}∥}_2,j=1,\cdots ,n.$$

Let $d_{i\left(1\right)}^{\mathrm{V}}\le d_{i\left(2\right)}^{\mathrm{V}}\le \dots \le d_{i\left(n\right)}^{\mathrm{V}}$ denote the order statistics of ${\left\{d_{ij}^{\mathrm{V}}\right\}}_{j=1}^n$. By minimizing the asymptotic mean square error, i.e., minimizing $tr\left(V\right)$, we obtain the sampling probability of the ith group $p_{ij}$ as

$$p_{i\left(j\right)}^{\mathrm{V}}=(r-s){\displaystyle \frac{d_{i\left(j\right)}^{\mathrm{V}}}{{\sum }_{k=1}^{n-s}\left(d_{i\left(k\right)}^{\mathrm{V}}\right)}},1\le j\le n-s,$$

(6)

and we have

$$p_{i\left(j\right)}^{\mathrm{V}}=1,n-s+1\le j\le n,$$

(7)

where

$$s=min\left\{t\mid 0\le t\le r,(r-t)d_{i(n-t)}^{\mathrm{V}}\le \sum _{j=1}^{n-t}{d}_{i\left(j\right)}^{\mathrm{V}}\right\}.$$

(8)

This means that s satisfies

$$(r-s+1)d_{i(n-s+1)}^{\mathrm{V}}\ge \sum _{j=1}^{n-s+1}{d}_{i\left(j\right)}^{\mathrm{V}},(r-s)d_{i(n-s)}^{\mathrm{V}}<\sum _{j=1}^{n-s}{d}_{i\left(j\right)}^{\mathrm{V}}.$$

${\left\{p_{ij}^{\mathrm{V}}\right\}}_{j=1}^n$ can be obtained directly by using the data of each group of covariates and response variables.

We know that the range of sampling probability values is $[0,1]$. Therefore, the sampling probability calculated based on (6) should be less than or equal to 1. As $\left(j\right)$ increases, the sampling probability gets closer to 1. To ensure that the sampling probability remains less than or equal to 1, we introduce a threshold s that satisfies (8). When $\left(j\right)$ exceeds this threshold, we directly set the sampling probability to 1.

However, when calculating ${\left\{p_{ij}^{\mathrm{V}}\right\}}_{j=1}^n$, we need to calculate ${∥{\mathsf{\Sigma }}_i^{-1}{x}_{ij}∥}_2$ for $j=1,\cdots ,n$. It takes $O\left(n{d}^2\right)$ time. To further reduce the calculation time, a simple and natural way is to use the matrix $V_c$ directly, and thus, we do not need to calculate ${∥{\mathsf{\Sigma }}_i^{-1}{x}_{ij}∥}_2$. In this way, we could change the method by minimizing $tr\left(V_c\right)$ to calculate the optimal subsampling probability. This method is called the linear optimality criterion, or L-optimality for short (see [18]).

Below, we describe in detail the process of obtaining optimal subsampling probability by minimizing $tr\left(V_c\right)$.

Method 2 (L-optimality).

For ease of presentation, define the statistics in the ith group:

$$d_{ij}^{\mathrm{Vc}}=\left|y_{ij}-{\widehat{\beta }}_N^{\prime }{x}_{ij}-{\widehat{\mu }}_i\right|{∥x_{ij}∥}_2,j=1,\cdots ,n.$$

Let $d_{i\left(1\right)}^{\mathrm{Vc}}\le d_{i\left(2\right)}^{\mathrm{Vc}}\le \dots \le d_{i\left(n\right)}^{\mathrm{Vc}}$ denote the order statistics of ${\left\{d_{ij}^{\mathrm{Vc}}\right\}}_{j=1}^n$. By minimizing the asymptotic mean square error, i.e., minimizing $tr\left(V_c\right)$, we obtain the sampling probability of the ith group $p_{ij}$ as

$$p_{i\left(j\right)}^{\mathrm{Vc}}=(r-s){\displaystyle \frac{d_{i\left(j\right)}^{\mathrm{Vc}}}{{\sum }_{k=1}^{n-s}\left(d_{i\left(k\right)}^{\mathrm{Vc}}\right)}},1\le j\le n-s,$$

(9)

and we have

$$p_{i\left(j\right)}^{\mathrm{Vc}}=1,n-s+1\le j\le n,$$

(10)

where

$$s=min\left\{t\mid 0\le t\le r,(r-t)d_{i(n-t)}^{\mathrm{Vc}}\le \sum _{j=1}^{n-t}{d}_{i\left(j\right)}^{\mathrm{Vc}}\right\}.$$

This means that s satisfies

$$(r-s+1)d_{i(n-s+1)}^{\mathrm{Vc}}\ge \sum _{j=1}^{n-s+1}{d}_{i\left(j\right)}^{\mathrm{Vc}},(r-s)d_{i(n-s)}^{\mathrm{Vc}}<\sum _{j=1}^{n-s}{d}_{i\left(j\right)}^{\mathrm{Vc}}.$$

4.2. Robust Implementation

In the above calculation, there is ${\displaystyle \frac{{\delta }_{ij}}{p_{ij}}}$ as the weight in the estimation function. Therefore, we consider the data points which satisfy $y_{ij}-{\widehat{\beta }}_N^{\prime }{x}_{ij}-\widehat{{\mu }_i}\approx 0$. In this case, the probability of these data being sampled is quite small, but they may also be sampled. If these data points are sampled, the estimation equation may be sensitive to the data points. This may cause the failure of Methods 1 and 2 in practical applications. In order to make the estimation more stable, we adopt a more robust sampling method to implement Methods 1 and 2 in the previous section.

We use the following subsampling probability:

$$p_{ij}^{sos}=(1-\rho )p_{ij}^{os}+\rho r{n}^{-1},j=1,\cdots ,n,$$

(11)

where $p_{ij}^{os}$ is $p_{ij}^{\mathrm{V}}$ or $p_{ij}^{\mathrm{Vc}}$, and $\rho \in (0,1)$.

${\left\{p_{ij}^{\mathrm{sos}}\right\}}_{j=1}^n$ is the convex combination of ${\left\{p_{ij}^{\mathrm{os}}\right\}}_{j=1}^n$ and the uniform subsampling probability $r{n}^{-1}$. It inherits the advantages of both. Here, $\rho$ is a preference tuning parameter. A smaller $\rho$ results in a more optimal sampling probability, while a larger $\rho$ yields a more robust sampling probability. When $\rho$ is close to 1, the estimation function is not sensitive to data points, so the estimation obtained becomes more stable. On the other hand, no matter what value $\rho$ takes, the order of $p_{ij}^{\mathrm{sos}}$ is consistent with $p_{ij}^{\mathrm{os}}$. Thus, the estimator can still retain the advantages of optimal subsampling.

In practical implementation, the parameters ${\widehat{\beta }}_N$ in $p_{ij}^{\mathrm{os}}$ are unknown. Therefore, we need to estimate the values of ${\widehat{\beta }}_N$ to continue the following calculations.

In order to estimate ${\widehat{\beta }}_N$, we use uniform sampling to obtain an a priori estimate. In detail, we first take a uniform subsample in each group for which the subsample size is $r_0$. Then, we substitute the obtained subsamples into the estimation function to calculate the pilot estimator of ${\widehat{\beta }}_N$. In addition, we also use the ${\widehat{\mu }}_i$ in the calculation of the pilot estimator. We write the pilot estimator as ${\tilde{\beta }}_0$.

In addition, the typical situation is that $r\ll n$ is required in our subsampling setting. In order to ensure that each $p_{ij}^{\mathrm{sos}}\in [0,1]$, we need take the inverses of $p_{ij}^{\mathrm{sos}}\wedge 1$ as weights in the weighted upper expectation loss.

At the end of this section, we summarize the estimation methods under the optimal sampling probability as Algorithm 2, paving the way for the simulations and real-data applications in the next section.

Algorithm 2 The robust subsampling method for $\beta$.

Require:   ${\left\{\left(y_{ij},x_{ij}\right)\right\}}_{j=1}^n$ be a sample in model (1), where $i=1,\cdots ,k$; Ensure:   the estimations $\tilde{\beta }$;   1: for $i=1$ to n do   2:      Use (3) to solve for the parameter estimate ${\widehat{\mu }}_i$;   3: end for   4: Use uniform sampling to get a priori estimate ${\widehat{\beta }}_0$ based on (5);   5: Use (6) and (7) or (9) and (10) to obtain the subsampling probability $p_{ij}^{\mathrm{V}}$ or $p_{ij}^{\mathrm{Vc}}$;   6: Obtain the robust subsampling probability $p_{ij}^{sos}$ according to (11);   7: Compute the estimation $\tilde{\beta }$ according to (5) with the subsampling probability $p_{ij}^{sos}$;   8: return $\tilde{\beta }$.

5. Simulation Study and Real Data Analysis

5.1. Simulation Study

In this part, we give several simulation examples. By comparing the performance of our proposed optimal subsampling methods and uniform sampling method, the rationality and effectiveness of our proposed method are further illustrated.

From the simulations given later, we discover the following conclusions. Whether it is the case where the mean value is certain or the mean value is uncertain, the optimal subsampling method we proposed has significantly better performance than the simple uniform subsampling method. And as the sampling ratio increases, the performance of the method becomes better.

5.1.1. Experiment 1

In the first simulation experiment, we consider the following linear model:

$$Y={\beta }_1{X}_1+{\beta }_2{X}_2+{\beta }_3{X}_3+\epsilon .$$

During the simulation, we examine a scenario where the error term possesses a defined mean, but its variance is uncertain. For this setup, we select regression coefficients as ${\beta }_j=1$ for $j=1,2,3$. Additionally, the predictors $X_j$ are independently and identically distributed as $N(0,1)$ for $j=1,2,3$. The error term $\epsilon$ is presumed to adhere to a normal distribution with a mean of zero and an unknown variance ${\sigma }^2$, thereby forming a generalized regression model accommodating variance uncertainty. During the simulation process, we draw variance values ${\sigma }_i^2$ for i ranging from 1 to k from a uniform distribution spanning from [0, 4]. Following this, we generate error values ${\epsilon }_{ij}$ for j ranging from 1 to n, using normal distributions centered at zero with variances ${\sigma }_i^2$.

We have three subsampling methods. The first method is the simple uniform sampling method. This method serves as a control to reflect the effectiveness and rationality of the method we proposed. The second method is the Method 1 we proposed in Section 4, which is called A-optimality. And the third method is the Method 2 in Section 4, which is called L-optimality. Let $\rho =0.2$ be the preference tuning parameter. We chose $k=10$ and $n=$ 10,000, and we chose that the number of samples per group r was 100, 300, and 500. This implies that by adopting such a practice with varying sample sizes for subsampling, we can compare the impact of different sample sizes r in the method. We sampled the generated data 500 times and compared the performance of these three methods by calculating the deviation and mean square error of the parameter estimates.

The simulation results are reported in

Table 1. The mean and MSE of three methods when $r=100$ in experiment 1, with $k=10$ and $n=$ 10,000.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.072	1.082	1.075
	MSE	7.25 × 10−2	8.05 × 10−2	7.13 × 10−2
A-optimality	mean	1.050	1.052	1.044
	MSE	4.54 × 10−2	4.08 × 10−2	3.92 × 10−2
L-optimality	mean	1.043	1.052	1.056
	MSE	4.57 × 10−2	4.08 × 10−2	4.18 × 10−2

,

Table 2. The mean and MSE of three methods when $r=300$ in experiment 1, with $k=10$ and $n=$ 10,000.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.045	1.037	1.033
	MSE	2.86 × 10−2	2.54 × 10−2	2.35 × 10−2
A-optimality	mean	1.015	1.025	1.028
	MSE	1.51 × 10−2	1.29 × 10−2	1.46 × 10−2
L-optimality	mean	1.015	1.034	1.023
	MSE	1.52 × 10−2	1.53 × 10−2	1.51 × 10−2

and

Table 3. The mean and MSE of three methods when $r=500$ in experiment 1, with $k=10$ and $n=$ 10,000.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.025	1.027	1.028
	MSE	1.50 × 10−2	1.66 × 10−2	1.43 × 10−2
A-optimality	mean	1.014	1.028	1.026
	MSE	8.51 × 10−3	9.68 × 10−3	8.96 × 10−3
L-optimality	mean	1.009	1.024	1.023
	MSE	8.67 × 10−3	9.19 × 10−3	8.59 × 10−3

. The bolded results in the tables correspond to the best-performing method.

In order to further illustrate the effect of our sampling method in the upper expected regression, we changed the experimental settings, which represents stronger heterogeneity. We considered increasing the number of groups k and appropriately reducing the number of samples in each group. We explored the effect of parameter estimation in the new setting. We chose $k=50$ and $n=2000$ as the new experiment settings. And the other experimental conditions remained unchanged.

The simulation results are reported in

Table 4. The mean and MSE of three methods when $r=100$ in experiment 1, with $k=50$ and $n=2000$.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.146	1.034	1.112
	MSE	1.54 × 10−1	1.40 × 10−1	1.46 × 10−1
A-optimality	mean	1.089	1.030	1.104
	MSE	8.02 × 10−2	7.87 × 10−2	9.23 × 10−2
L-optimality	mean	1.091	1.009	1.104
	MSE	8.23 × 10−2	7.99 × 10−2	9.34 × 10−2

,

Table 5. The mean and MSE of three methods when $r=300$ in experiment 1, with $k=50$ and $n=2000$.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.061	1.008	1.064
	MSE	5.18 × 10−2	4.14 × 10−2	4.65 × 10−2
A-optimality	mean	1.040	0.971	1.028
	MSE	2.66 × 10−2	2.29 × 10−2	2.06 × 10−2
L-optimality	mean	1.038	0.965	1.039
	MSE	2.51 × 10−2	2.23 × 10−2	2.43 × 10−2

and

Table 6. The mean and MSE of three methods when $r=500$ in experiment 1, with $k=50$ and $n=2000$.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.052	0.981	1.040
	MSE	2.75 × 10−2	2.61 × 10−2	2.47 × 10−2
A-optimality	mean	1.026	0.958	1.009
	MSE	1.36 × 10−2	1.29 × 10−2	1.00 × 10−2
L-optimality	mean	1.022	0.964	1.012
	MSE	1.33 × 10−2	1.47 × 10−2	8.45 × 10−3

. Based on the entire simulation experiment, we draw the following conclusions:

• By comparing the results of the three methods, it can be observed that our proposed A-optimality and L-optimality approaches significantly outperformed the uniform sampling method in terms of mean squared error. On the other hand, there was little difference in the effectiveness of the mean estimates obtained by the three methods. This demonstrates the effectiveness and stability of our methods, particularly for the method of A-optimality.
• By comparing Table 1, Table 2 and Table 3 or Table 4, Table 5 and Table 6, it can be observed that as the number of sampling instances r increased, the estimation performance of all three methods improved significantly, both in terms of the mean and mean squared error. This underscores the importance of the number of sampling instances.
• By comparing Table 1, Table 2 and Table 3 and Table 4, Table 5 and Table 6, it can be observed that in cases of high heterogeneity, where k is large and n is small, the estimation performance of all three methods deteriorates. However, as the number of sampling instances r increases, this discrepancy diminishes.

5.1.2. Experiment 2

We reconsider the linear model

$$Y={\beta }_1{X}_1+{\beta }_2{X}_2+{\beta }_3{X}_3+\epsilon ,$$

which is the same in form as in Experiment 1. In this model, we consider the situation that the mean and variance of the error are both uncertain. This means that $\epsilon$∼$N\left(\mu ,{\sigma }^2\right)$ have uncertain mean $\mu$ and variance ${\sigma }^2$.

During the simulation, we also specified that $X_j$ follows a normal distribution $N(0,1)$ for $j=1,2,3$. The error terms $\epsilon$ were generated as follows: First, the mean values ${\mu }_i$ and variance values ${\sigma }_i^2$ were drawn from uniform distributions $U(-1,1)$ and $U(0,1)$, respectively. Subsequently, the error values ${\epsilon }_{ij}$ for $j=1,\dots ,k$ were sampled from the normal distribution $N({\mu }_i,{\sigma }_i^2)$. Additionally, we also set $k=10$ and $n=10,000$ or $k=50$ and $n=2000$, as well as r to be 100, 300, or 500.

The simulation results are reported in

Table 7. The mean and MSE of three methods when $r=100$ in experiment 2, with $k=10$ and $n=$ 10,000.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.086	1.063	1.072
	MSE	8.11 × 10−2	6.92 × 10−2	7.30 × 10−2
A-optimality	mean	1.028	1.047	1.046
	MSE	4.23 × 10−2	4.22 × 10−2	4.57 × 10−2
L-optimality	mean	1.042	1.057	1.038
	MSE	4.68 × 10−2	4.61 × 10−2	3.88 × 10−2

,

Table 8. The mean and MSE of three methods when $r=300$ in experiment 2, with $k=10$ and $n=$ 10,000.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.028	1.036	1.038
	MSE	2.61 × 10−2	2.48 × 10−2	2.53 × 10−2
A-optimality	mean	1.008	1.017	1.030
	MSE	1.44 × 10−2	1.44 × 10−2	1.37 × 10−2
L-optimality	mean	1.015	1.021	1.028
	MSE	1.40 × 10−2	1.59 × 10−2	1.64 × 10−2

,

Table 9. The mean and MSE of three methods when $r=500$ in experiment 2, with $k=10$ and $n=$ 10,000.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.010	1.028	1.036
	MSE	1.58 × 10−2	1.60 × 10−2	1.60 × 10−2
A-optimality	mean	1.001	1.014	1.026
	MSE	8.39 × 10−3	8.38 × 10−3	9.72 × 10−3
L-optimality	mean	1.009	1.010	1.018
	MSE	8.86 × 10−3	9.90 × 10−3	7.79 × 10−3

,

Table 10. The mean and MSE of three methods when $r=100$ in experiment 2, with $k=50$ and $n=2000$.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.125	1.055	1.116
	MSE	1.50 × 10−3	1.37 × 10−1	1.53 × 10−1
A-optimality	mean	1.063	1.011	1.088
	MSE	7.69 × 10−2	8.01 × 10−2	7.81 × 10−2
L-optimality	mean	1.085	1.009	1.082
	MSE	8.80 × 10−2	7.02 × 10−2	8.23 × 10−2

,

Table 11. The mean and MSE of three methods when $r=300$ in experiment 2, with $k=50$ and $n=2000$.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.068	1.000	1.057
	MSE	5.44 × 10−2	4.16 × 10−2	4.65 × 10−2
A-optimality	mean	1.032	0.975	1.024
	MSE	2.65 × 10−2	2.54 × 10−2	2.23 × 10−2
L-optimality	mean	1.028	0.973	1.035
	MSE	2.89 × 10−2	2.33 × 10−2	2.31 × 10−2

and

Table 12. The mean and MSE of three methods when $r=500$ in experiment 2, with $k=50$ and $n=2000$.

Method	Statistics	${\mathit{\beta }}_1$	${\mathit{\beta }}_2$	${\mathit{\beta }}_3$
uniform	mean	1.035	0.974	1.028
	MSE	2.98 × 10−2	2.34 × 10−2	2.67 × 10−2
A-optimality	mean	1.013	0.961	1.016
	MSE	1.34 × 10−2	1.42 × 10−2	9.99 × 10−3
L-optimality	mean	1.018	0.962	1.011
	MSE	1.46 × 10−2	1.61 × 10−2	9.50 × 10−3

. The simulation results are similar to those of Experiment 1, which are briefly summarized here:

1.

In summary, by comparing the methods, our proposed A-optimality and L-optimality approaches outperformed the uniform sampling method in terms of mean squared error, while the mean estimates of all three methods showed little difference in effectiveness.

2.

It can be observed that as the number of sampling instances r increased, the estimation performance of all three methods improved significantly.

3.

Upon reviewing Table 7, Table 8, Table 9, Table 10, Table 11 and Table 12, it becomes apparent that when heterogeneity is high, the estimation capabilities of all three methods suffer.

5.2. Real Data Analysis

An increasing number of urban areas are grappling with persistent air pollution, primarily due to fine particulate matter, especially PM2.5. PM2.5 comprises particles suspended in the air with aerodynamic diameters smaller than 2.5 micrometers. These particles are recognized for their impact on visibility, human health, and climate patterns. Epidemiological studies indicate that exposure to PM2.5 can lead to respiratory ailments, severe cardiovascular diseases, and potentially fatal outcomes.

There are many reasons for PM2.5 pollution, including SO₂ concentration, O₃ concentration, temperature, wind speed, etc. The main consideration in this article is the impact of these four factors on PM2.5 concentration. The dataset used in this article was taken from the UCI machine learning knowledge base. Specifically, we obtained the PM2.5 concentration, SO₂ concentration, O₃ concentration, temperature, and wind speed of the ancient city of Beijing from 1 March 2013 to 29 February 2016. Due to the large time span involved in the model, model heterogeneity may arise. Therefore, we considered grouping the data based on time for processing. Then, we considered the linear influence of the covariate on the response variable, and we tested the effect of the subsampling method proposed in this article.

We first centralized the 25,217 data collected over three years, with the PM2.5 concentration after centralization as the response variable, and the central SO₂ concentration, O₃ concentration, temperature, and wind speed as four covariates. These covariates all have an impact on PM2.5 concentrations. For instance, SO₂ can undergo chemical reactions to transform into components of PM2.5, while wind speed affects the concentration of PM2.5 through dispersion and dilution effects. Next, we divided the data into 12 groups. The grouping method was grouped by time. Every three months was a group. This yielded about 2100 data points in each group. The number of data in each group was not necessarily equal.

For the linear regression problem, we considered two different situations. The first case is that the model has only variance uncertainty, that is, the mean value of the error term in each group is equal. The second case is that the model has both variance uncertainty and mean uncertainty. The two sampling methods we proposed performed well in both cases, and they were significantly better than ordinary uniform sampling methods.

Case 1 (Mean certainty model)

We assumed that the model only has variance heterogeneity. In other words, the errors from different group have the same mean. Consider the following model:

$$Y={\beta }_1{X}_1+{\beta }_2{X}_2+{\beta }_3{X}_3+{\beta }_4{X}_4+\epsilon .$$

Our purpose is to estimate the parameters ${\beta }_1$, ${\beta }_2$, ${\beta }_3$, and ${\beta }_4$, which are the same in every group. Use the subsampling methods we proposed above, we obtained the following table that describes the mean prediction error and computational time of each method under different settings. The sample size r in the subsampling of each group was set to 50.

Case 2 (Mean–variance heterogeneity model)

Then, we considered a model with not only mean heterogeneity but also variance heterogeneity in the errors of different groups. Consider the following model:

$$Y={\beta }_1{X}_1+{\beta }_2{X}_2+{\beta }_3{X}_3+{\beta }_4{X}_4+\epsilon .$$

In this model, ${\epsilon }_i\sim N\left({\mu }_i,{\sigma }_i^2\right)$. We also set the sample size in the subsampling of each group to 50. We have the following table that describes the mean prediction error and computational time of each method under different settings.

The results are presented in

Table 13. The computational time and MPE of three methods in mean certainty model when $r=50$.

Method	Uniform	A-Optimality	L-Optimality
computational time	0.67 s	4.05 s	3.13 s
MPE	0.737	0.630	0.632

and

Table 14. The computational and MPE of three methods in mean–variance heterogeneity model when $r=50$.

Method	Uniform	A-Optimality	L-Optimality
computational time	0.68 s	4.03 s	3.10 s
MPE	0.676	0.513	0.550

. Through the actual data analysis of the two cases, we obtained the following conclusions:

• First, in terms of computational time, it can be observed that uniform sampling was significantly the fastest, while A-optimality was the slowest. This is because the sampling probability for uniform sampling is easily obtained, whereas A-optimality requires the calculation of the inverse of the covariance matrix, which is more complex than L-optimality.
• Based on the mean prediction error metric, we found that the A-optimality and L-optimality methods significantly outperformed the uniform sampling method, with the A-optimality method being the best.
• By comparing Table 13 and Table 14, it can be observed that the prediction performance of the mean–variance heterogeneous model was significantly better than that of the variance heterogeneous model, while their computational times were comparable. This suggests that the mean–variance model is more suitable for the real data example.

6. Conclusions

In the background with heterogeneous distributions, the task of statistical modeling and inference remains a significant challenge. In this paper, we improved the k-sample upper expectation regression model in the context of distribution heterogeneity by employing group-specific ${\mu }_i$ values to address the impact of heterogeneity and obtaining consistent estimates. Additionally, we delved into optimal subsampling techniques specifically designed for big data scenarios, overcoming the challenges posed by large datasets and privacy protection through a robust optimal sampling probability. We analyzed the theoretical properties of our method and conducted comprehensive numerical experiments on both simulated and actual datasets to assess its practical effectiveness. Both theoretical analysis and numerical results confirm the validity of our method when dealing with large datasets. Furthermore, real data analysis demonstrates that our method can be applied to various similar fields, such as finance and environmental monitoring.

However, there are still some shortcomings in our method that require subsequent work to address. First, the k-sample assumption necessitates prior knowledge of k-sample groups, which means we must assume that sample clusters with different distributions are already known, which are often derived from experience or other methods. This is a rather stringent requirement. Future work could consider developing methods that can accommodate complete distribution heterogeneity in situations where the grouping is unknown, for example, first by data-driven clustering and then by applying upper expectation regression. Additionally, when k is large, it often implies a smaller sample size in each group, resulting in a high degree of heterogeneity. In such cases, both computational complexity and estimation effectiveness face significant challenges. Overcoming these issues is also a direction for future work. Lastly, the robust optimal sampling probability involves a tune parameter $\rho$, and selecting the optimal $\rho$ to balance robustness and efficiency of the sampling probability is also a research-worthy problem.