Asymptotics of Subsampling for Generalized Linear Regression Models under Unbounded Design

Abstract

The optimal subsampling is an statistical methodology for generalized linear models (GLMs) to make inference quickly about parameter estimation in massive data regression. Existing literature only considers bounded covariates. In this paper, the asymptotic normality of the subsampling M-estimator based on the Fisher information matrix is obtained. Then, we study the asymptotic properties of subsampling estimators of unbounded GLMs with nonnatural links, including conditional asymptotic properties and unconditional asymptotic properties.

1. Introduction

In recent years, the amount of information that people need to process is increasing dramatically. It is of great challenge to directly process massive data for statistical analysis. The divide-and-conquer strategy can mitigate the challenge of directly processing such big data [1], but it still consumes considerable computing resources. As a cheaper alternative in computing, subsampling gains its value in the case of limited computing resources.

To reduce the burden on the machine, the subsampling strategy based on big data has been given more attention in recent years. Ref. [2] proposes simple necessary and sufficient conditions for a convolved subsampling estimator to produce a normal limit that matches the target of bootstrap estimation; Ref. [3] provides an optimally distributed subsampling for maximum quasi-likelihood estimators with massive data; Ref. [4] studies some adaptive optimal subsampling algorithms; and Ref. [5] describes a subdata selection method based on leverage scores which conduct the linear model selection on a small subdata set.

GLM is a kind of statistical model with a wide range of applications such as [6,7,8]. Many subsampling studies are based on GLMs such as [3,9,10]. However, the covariates of the subsampled GLMs in the literature are bounded. When dealing with some big data problems, the size of covariate is not strictly bounded, such as the number of clicks on a web page, which can grow infinitely. This requires the extension of existing theories to the unbounded design. To fill this gap, this paper aims to study asymptotic properties of the subsampled GLMs with unbounded covariates based on empirical process and martingale technology.

Our three contributions are shown as follows: (1) we describe the asymptotic property of subsampled M-estimator using Fisher information matrix; (2) we give the conditional consistency and asymptotic normality of unbounded GLMs subsampling estimator; (3) we provide the unconditional consistency and asymptotic normality of unbounded GLMs subsampling estimator.

The rest of the paper is organized as follows. Section 2 introduces the basic concepts in GLMs and subsampling M-estimation problem. Section 3 presents the asymptotical properties for unbounded GLMs subsampling estimators. Section 4 gives the conclusion and discussion, as well as future research directions. All the technical proofs are collected in the Appendix A.

2. Preliminaries

This section introduces the subsampling M-estimation problem and GLMs.

2.1. Subsampling M-Estimation

Let $\left\{l\right(\mathit{\beta };\mathit{Z})\in \mathbb{R}|\mathit{Z}\in \mathcal{Z}\}$ be a set of loss functions with a finite dimensional convex set $\mathit{\beta }\in \mathsf{\Theta }\subset {\mathbb{R}}^p$, and $U=\{1,2,\dots ,N\}$ be the index of the full large dataset with $\sigma$-algebra ${\mathcal{F}}_N=\sigma ({\mathit{Z}}_1,\dots ,{\mathit{Z}}_N)$, where for each $i\in U,$ the random data point ${\mathit{Z}}_i\in \mathcal{Z}$ (some probability space) is observed. The empirical risk $L_N:\mathsf{\Theta }\to \mathbb{R}$ is given by $L_N\left(\mathit{\beta }\right)=\frac{1}{N}{\sum }_{i\in U}l\left(\mathit{\beta };{\mathit{Z}}_i\right)$.

The goal is to get the solution ${\widehat{\mathit{\beta }}}_N$ to minimize the risk, namely

$${\widehat{\mathit{\beta }}}_N=arg\underset{\mathit{\beta }\in \mathsf{\Theta }}{min}{L}_N\left(\mathit{\beta }\right).$$

(1)

To solve Equation (1), we need ${\widehat{\mathit{\beta }}}_N$ satisfy: $\nabla L_N\left(\mathit{\beta }\right)=\frac{1}{N}{\sum }_{i\in U}\nabla l\left(\mathit{\beta };{\mathit{Z}}_i\right)=\mathbf{0}$, and let ${\mathsf{\Sigma }}_N:={\nabla }^2{L}_N({\widehat{\mathit{\beta }}}_N)$. This is an M-estimation problem; see [11]. For fast solving large-scale estimation in Equation (1), we propose the subsampling M-estimation. Consider an index set $S=\{i_1,i_2,\dots ,i_n\}$ with replacement from U according to the sampling probability ${\left\{{\pi }_i\right\}}_{i=1}^N$ such that ${\sum }_{i=1}^N{\pi }_i=1.$ The subsampling M-estimation problem is to obtain the solution ${\widehat{\mathit{\beta }}}_n$ satisfying

$$\nabla L_n\left(\mathit{\beta }\right)=\mathbf{0}\mathrm{with}{L}_n\left(\mathit{\beta }\right)=\frac{1}{Nn}\sum _{i\in S}\frac{1}{{\pi }_i^{\ast }}l\left(\mathit{\beta };{\mathit{Z}}_i^{\ast }\right),$$

where ${\mathit{Z}}_i^{\ast }$ is the i-th time subsample with replacement and ${\pi }_i^{\ast }$ is the subsampling probability of ${\mathit{Z}}_i^{\ast }$. For example, if ${\mathit{Z}}_1^{\ast }={\mathit{Z}}_1$, then ${\pi }_1^{\ast }={\pi }_1$; if ${\mathit{Z}}_2^{\ast }={\mathit{Z}}_1$, then ${\pi }_2^{\ast }={\pi }_1$. Denote $a_i$ as the number of i-th subsampled data such that ${\sum }_{i\in U}{a}_i=n$. And $L_n\left(\mathit{\beta }\right)$ is constructed by inverse probability weighting skill such that $\mathrm{E}\left[L_n\left(\mathit{\beta }\right)\right|{\mathcal{F}}_N]=L_N\left(\mathit{\beta }\right)$; see [12]. Details about properties of conditional expectation are shown in [13].

2.2. Generalized Linear Models

Let the random variable Y be the distribution of the natural exponential families $P_{\alpha }$ indexed by parameter $\alpha$,

$$P_{\alpha }\left(dy\right)=d{F}_Y\left(y\right)=c\left(y\right)exp\{y\alpha -b\left(\alpha \right)\}\nu \left(dy\right),c\left(y\right)>0,$$

(2)

where $\alpha$ is often referred to as the canonical parameter belonging to its natural space

$$\mathsf{\Lambda }=\{\alpha :\int c(y)exp\{y\alpha \}\nu (dy)<\infty \}.$$

$\nu (·)$ is the Lebesgue measure for continuous distributions (Normal, Gamma) or counting measure for discrete distributions (binomial, Poisson, negative binomial). The $c\left(y\right)$ is free of $\alpha$.

Let ${\left\{(Y_i,{\mathit{X}}_i)\right\}}_{i=1}^N$ be N independent sample data pairs. Here the ${\mathit{X}}_i\in {\mathbb{R}}^p$ is covariates and we assume that the response $Y_i$ follows the distribution of the natural exponential families with the parameter ${\alpha }_i\in \mathsf{\Lambda }$. The covariates ${\mathit{X}}_i:={(x_{i1},\dots ,x_{ip})}^T,(i=1,2,\dots ,N)$ are supposed to be deterministic.

The conditional expectation of $Y_i$ for a given ${\mathit{X}}_i$ is defined as a function of ${\mathit{\beta }}^T{\mathit{X}}_i$ after a transformation by a link function ${\alpha }_i=\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)$. The mean value denoted as ${\mu }_i:=\mathrm{E}\left(Y_i\right)$ is mostly considered for regression.

If ${\alpha }_i={\mathit{\beta }}^T{\mathit{X}}_i$ then we call that $\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)={\mathit{\beta }}^T{\mathit{X}}_i$ is canonical (or natural) link function, and corresponding model is canonical (or natural) GLMs; see Page 32 in [14]. Sometimes the assumption ${\alpha }_i={\mathit{\beta }}^T{\mathit{X}}_i$ is somewhat strong and not very suitable in practice, while nonnatural link GLMs allow more flexible choices for the link function. We can further assume that ${\alpha }_i$ and ${\mathit{\beta }}^T{\mathit{X}}_i$ can be related by a nonnatural link function ${\alpha }_i=\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)$.

Let $f_{\mathit{\beta }}\left(Y_i\right|{\mathit{X}}_i)$ be the joint density function of the i.i.d. data ${\left\{(Y_i,{\mathit{X}}_i)\right\}}_{i=1}^N$ from the exponential family with a link function $\psi (·)$. Then the nonnatural GLMs [15] is defined by

$$Y_i|{\mathit{X}}_i\sim f_{\mathit{\beta }}\left(Y_i\right|{\mathit{X}}_i)=c\left(Y_i\right)exp\left\{Y_i\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)-b\left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)\right)\right\},i=1,2,\dots ,N.$$

(3)

Here is a classic result for the exponential family (3),

$$\mathrm{E}\left(Y_i\right|{\mathit{X}}_i):={\mu }_i=\dot{b}\left({\alpha }_i\right)=\dot{b}\left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)\right)and\mathrm{Var}\left(Y_i\right|{\mathit{X}}_i):=\mathrm{Var}\left(Y_i\right)=\ddot{b}\left({\alpha }_i\right),$$

(4)

where $i=1,2,\dots ,N$; see P280 in [16].

3. Main Results

3.1. Subsampling M-Estimation Problem

In this part we first look at the term ${\mathsf{\Sigma }}_N^{-1}\nabla L_n({\widehat{\mathit{\beta }}}_N)$. Define an independent random vector sequence ${\{{\mathit{\zeta }}_j\}}_{j=1}^N$ and the subsampled ${\{{\mathit{\zeta }}_j^{\ast }\}}_{j=1}^n$, such that each vector $\mathit{\zeta }$ takes the value among ${\{\frac{1}{N{\pi }_i}{\mathsf{\Sigma }}_N^{-1}\nabla l({\widehat{\mathit{\beta }}}_N;{\mathit{Z}}_i)\}}_{i=1}^N,$ and let

$${\mathrm{V}}_M({\widehat{\mathit{\beta }}}_N;n)=\frac{1}{N^{2}n}{\mathsf{\Sigma }}_N^{-1}\left[\sum _{i\in U}\frac{1}{{\pi }_i}\nabla l\left({\widehat{\mathit{\beta }}}_N;{\mathit{Z}}_i\right){\nabla }^{T}l\left({\widehat{\mathit{\beta }}}_N;{\mathit{Z}}_i\right)\right]{\mathsf{\Sigma }}_N^{-1}.$$

From the definition of $\nabla L_N\left(\mathit{\beta }\right)$, we have $\mathrm{E}\left(\mathit{\zeta }\right|{\mathcal{F}}_N)={\mathsf{\Sigma }}^{-1}\nabla L_N\left({\widehat{\mathit{\beta }}}_N\right)=\mathbf{0}$ and $\mathrm{Var}\left(\mathit{\zeta }\right|{\mathcal{F}}_N)$$=n{\mathrm{V}}_M({\widehat{\mathit{\beta }}}_N;n).$ Then we have the asymptotic property of subsampled M-estimator.

Theorem 1.

Suppose that the risk function $L_N\left(\mathit{\beta }\right)$ is twice differentiable and λ-strongly convex over Θ, that is, for $\mathit{\beta }\in \mathsf{\Theta }$, ${\nabla }^2{L}_N\left(\mathit{\beta }\right)\ge \lambda \mathbf{I}$, where ≥ denotes the semidefinite positive ordering; and the sampling-based moment condition,

$$\begin{array}{c}\hfill \frac{1}{N^4}\sum _{i=1}^N\frac{1}{{\pi }_i^3}{∥\nabla l({\widehat{\mathit{\beta }}}_N;{\mathit{Z}}_i)∥}^4=O_P\left(1\right).\end{array}$$

Then we can obtain: As $n\to \infty ,$ conditioning on ${\mathcal{F}}_N$,

$${\mathrm{V}}_M{({\widehat{\mathit{\beta }}}_N;n)}^{-\frac{1}{2}}({\widehat{\mathit{\beta }}}_n-{\widehat{\mathit{\beta }}}_N)\stackrel{d}{\to }N(0,{\mathbf{I}}_p),$$

(5)

where $\stackrel{d}{\to }$ means convergence in distribution.

Theorem 1 reveals that the subsampling M-estimation scheme is theoretically feasible under mild conditions. In addition, the existence of the estimator is given by the Fisher information matrix.

3.2. Conditional Asymptotic Properties of Subsampled GLMs with Unbounded Covariates

The exponential family is very versatile for containing many common light-tail distributions such as binomial, Poisson, negative binomial, normal and Gamma. Along with their attendant convexity properties which leads to finite variance property for log-density, they can serve for a large amount of popular and effective statistical models. It is precisely because of the commonality of these distributions so that we study the subsampling problem for GLMs.

From the loss function introduced in Section 2.1, we set $l(\mathit{\beta };{\mathit{Z}}_i):=-log{f}_{\mathit{\beta }}\left(Y_i\right|{\mathit{X}}_i)$ where $f_{\mathit{\beta }}\left(Y_i\right|{\mathit{X}}_i)$ is defined by Equation (2), then the problem solving the minimum of the loss function is equivalent to solve the maximum of the likelihood function. For simplicity, we assume that $c\left(y\right)=1$, then

$$\nabla l(\mathit{\beta };{\mathit{Z}}_i):=-\frac{\partial log{f}_{\mathit{\beta }}\left(Y_i\right|{\mathit{X}}_i)}{\partial \mathit{\beta }}=-\left[Y_i-\dot{b}\left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)\right)\right]\dot{\psi }\left({\mathit{\beta }}^T{\mathit{X}}_i\right){\mathit{X}}_i$$

with the nonnatural link function ${\alpha }_i=\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)$. We also use this idea in Section 3.3.

More generally, we consider a wider class saying quasi-GLMs, rather than GLMs, which assumes that Equation (4) holds for a certain function $\mu (·)$. Strong consistency and asymptotic normality of quasi maximum likelihood estimate in GLMs with bounded covariates are proved in [17]. For unbounded covariates, adopting the subsampled estimation of GLMs in [9], we calculate the inverse probability weighted estimator of $\mathit{\beta }$ by solving the estimating equation based on the subsampled index set S,

$$-\frac{1}{Nn}\sum _{i\in S}\frac{1}{{\pi }_i^{\ast }}\left[Y_i^{\ast }-\mu \left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\right)\right]\dot{\psi }\left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right){\mathit{X}}_i^{\ast }=\mathbf{0}.$$

where ${\left\{(Y_i^{\ast },{\mathit{X}}_i^{\ast })\right\}}_{i\in S}$ is subsampled data. Equivalently, we have

$$s_n\left(\mathit{\beta }\right)=\sum _{i\in S}\frac{1}{{\pi }_i^{\ast }}\left[Y_i^{\ast }-\mu \left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\right)\right]\dot{\psi }\left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right){\mathit{X}}_i^{\ast }=\mathbf{0}.$$

(6)

Equation (6) is called quasi-GLMs since Equation (4) is given instead of the distribution function.

Let ${\widehat{\mathit{\beta }}}_n$ be the estimator of the real parameter ${\mathit{\beta }}_0$ in subsampled quasi-GLMs and ${\widehat{\mathit{\beta }}}_N$ be the estimator of ${\mathit{\beta }}_0$ in quasi-GLMs with full data. For the unbounded quasi-GLMs with full data, ${\widehat{\mathit{\beta }}}_N$ is asymptotic unbiased with respect to ${\mathit{\beta }}_0$; see [18]. Next, we focus on the asymptotical properties of ${\widehat{\mathit{\beta }}}_n$, as shown in the following theorems.

Theorem 2.

Let ${\left\{(Y_i^{\ast },{\mathit{X}}_i^{\ast })\right\}}_{i\in S}$ be subsampled from i.i.d. full data ${\left\{(Y_i,{\mathit{X}}_i)\right\}}_{i\in U}$. Consider the Equation (4) and (6) where $\psi (·)$ is three times continuously differentiable whose every derivative is bounded, and $b(·)$ is twice continuously differentiable whose every derivative is also bounded. Assume that:

(A.1) 

The range of the unknown parameter $\mathit{\beta }$ is an open subset of ${\mathbb{R}}^p$.

(A.2) 

For any $i\in S$, $\mathrm{E}{sup}_{\mathit{\beta }\in \mathsf{\Theta }}\left[\frac{1}{{\pi }_i^{\ast }}|Y_i^{\ast }-\mu \left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\right)||{\mathcal{F}}_N\right]=O\left(1\right)$.

(A.3) 

For any $\mathit{\beta }\in \mathsf{\Theta }$ and $i\in S$, $0<{inf}_i\phi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\le {sup}_i\phi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)<\infty$, where $\phi \left(t\right)={\left[\dot{\psi }\left(t\right)\right]}^2\ddot{b}\left(\psi \left(t\right)\right)$.

(A.4) 

For any ${\mathit{\beta }}_1\in \mathsf{\Theta }$ and ${\mathit{\beta }}_2\in \mathsf{\Theta }$, there exists a function $\left|m\right({\mathit{X}}_i^{\ast }\left)\right|<\infty$ such that

$$|\phi \left({\mathit{\beta }}_1^T{\mathit{X}}_i^{\ast }\right)-\phi \left({\mathit{\beta }}_2^T{\mathit{X}}_i^{\ast }\right)|\le |m\left({\mathit{X}}_i^{\ast }\right)\left|\right|{\mathit{\beta }}_1^T{\mathit{X}}_i^{\ast }-{\mathit{\beta }}_2^T{\mathit{X}}_i^{\ast }|.$$

(A.5) 

When $n\to \infty ,{max}_{i\in S}{\mathit{X}}_i^{\ast T}{({\mathbf{X}}^{\ast }{\mathbf{X}}^{\ast \mathbf{T}})}^{-1}{\mathit{X}}_i^{\ast }=O\left(n^{-1}\right)$ and ${\lambda }_{min}\left[{\mathbf{X}}^{\ast }{\mathbf{X}}^{\ast \mathbf{T}}\right]\to \infty$, where ${\mathbf{X}}^{\ast }=({\mathit{X}}_1^{\ast },...,{\mathit{X}}_n^{\ast })$ and ${\lambda }_{min}\left[\mathbf{A}\right]$ is the smallest eigenvalue of the matrix $\mathbf{A}$.

(A.6) 

${min}_{i=1,...,N}\left(N{\pi }_i\right)=O\left(1\right)$, ${max}_{i=1,...,N}\left(N{\pi }_i\right)=O\left(1\right)$.

Then ${\widehat{\mathit{\beta }}}_n$ is consistent with ${\widehat{\mathit{\beta }}}_N$, i.e.,

$$\parallel {\widehat{\mathit{\beta }}}_n-{\widehat{\mathit{\beta }}}_N\parallel =o_{P|{\mathcal{F}}_N}\left(1\right)$$

where $o_{P|{\mathcal{F}}_N}\left(1\right)$ means $o\left(1\right)$ conditioning on ${\mathcal{F}}_N$ in probability.

Theorem 3.

Under the conditions in Theorem 2, as $N\to \infty$ and $n\to \infty$, conditional on ${\mathcal{F}}_N$ in probability,

$$\sqrt{n}({\widehat{\mathit{\beta }}}_n-{\widehat{\mathit{\beta }}}_N)\to N(\mathbf{0},{\mathbf{V}}_s),$$

in distribution, where

$$\begin{array}{cc}& {\mathbf{V}}_s={\mathsf{\Sigma }}_N^{-1}{\mathbf{V}}_N{\mathsf{\Sigma }}_N^{-1},\hfill \\ & {\mathsf{\Sigma }}_N=\sum _{i\in U}{a}_i\left[Y_i-\dot{b}(\psi ({\widehat{\mathit{\beta }}}_N^T{\mathit{X}}_i))\right]\ddot{\psi }({\widehat{\mathit{\beta }}}_N^T{\mathit{X}}_i){\mathit{X}}_i{\mathit{X}}_i^T-\sum _{i\in U}{a}_i\ddot{b}(\psi ({\widehat{\mathit{\beta }}}_N^T{\mathit{X}}_i))\dot{\psi }{({\widehat{\mathit{\beta }}}_N^T{\mathit{X}}_i)}^2{\mathit{X}}_i{\mathit{X}}_i^T,\hfill \\ & {\mathbf{V}}_N=\sum _{i\in U}\frac{a_i}{{\pi }_i}{\left[Y_i-\dot{b}(\psi ({\widehat{\mathit{\beta }}}_N^T{\mathit{X}}_i))\right]}^2\dot{\psi }{({\widehat{\mathit{\beta }}}_N^T{\mathit{X}}_i)}^2{\mathit{X}}_i{\mathit{X}}_i^T.\hfill \end{array}$$

In this part, we complete the asymptotic properties without the moment condition of the covariates ${\left\{{\mathit{X}}_i\right\}}_{i=1}^N$ which is used in [9], and that means ${\mathit{X}}_i$’s are unbounded. Here we only provide the theoretical asymptotic results. Furthermore, the subsampling probability can be derived by A-optimal criterion like [10].

3.3. Unconditional Asymptotic Properties of Subsampled GLMs with Unbounded Covariates

In real engineering, the measurement of some response variable data is very expensive, such as superconductor data, deep space exploration data, etc. The accuracy of estimating the target parameters under measurement constraints of responses is a very important issue. Ref. [19] completed the unconditional asymptotic properties of parameter estimation in bounded GLMs with canonical link. But the unbounded GLMs with nonnatural link situation has not been discussed yet.

In this section, we continue to use the notations of Section 3.2. Through the theory of empirical process [11], we obtain the unconditional consistency of ${\widehat{\mathit{\beta }}}_n$ in the following theorem.

Theorem 4.

(Unconditional subsampled consistency) Assume the conditions:

(B.1) 

${\lambda }_{min}\left(\mathrm{E}\mathit{X}{\mathit{X}}^T\right)>0$ where $\mathit{X}$ is the unbounded covariate of GLMs.

(B.2) 

For $\forall u_1,u_2\in [0,1]$,

$$\underset{\mathit{\beta }\in \mathsf{\Theta }\backslash \left\{{\mathit{\beta }}_0\right\}}{inf}\frac{\mathrm{E}\{\ddot{b}\left({\tilde{\psi }}_{u_1}\right)\dot{\psi }{[(1-u_2)\left({\mathit{\beta }}_0^T\mathit{X}\right)+u_2\left({\mathit{\beta }}^T\mathit{X}\right)]}^2{({\mathit{\beta }}^T\mathit{X}-{\mathit{\beta }}_0^T\mathit{X})}^2\}}{\mathrm{E}{({\mathit{\beta }}^T\mathit{X}-{\mathit{\beta }}_0^T\mathit{X})}^2}\ge C_1>0,$$

where ${\tilde{\psi }}_{u_1}=(1-u_1)\psi \left({\mathit{\beta }}_0^T\mathit{X}\right)+u_1\psi \left({\mathit{\beta }}^T\mathit{X}\right)$ and $\ddot{b}(·)$ is the second derivative with respect to $\mathit{\beta }$.

(B.3) 

$${\mathrm{E}}_{{\mathit{\beta }}_0}\underset{\mathit{\beta }\in \mathsf{\Theta }}{sup}\left[\right|Y-\dot{b}\left(\psi \left({\mathit{\beta }}^T\mathit{X}\right)\right){|·|\left|\mathit{X}\right||}^2]<\infty ,$$

where $\dot{b}(·)$ is the first derivative with respect to $\mathit{\beta }$.

(B.4) 

$\psi (·)$ in (3) is twice continuously differentiable and its every derivative has a positive minimum.

(B.5) 

$b(·)$ in (3) is twice continuously differentiable and its every derivative has a positive minimum.

Then $\parallel {\widehat{\mathit{\beta }}}_n-{\mathit{\beta }}_0\parallel =o_P\left(1\right)$.

Theorem 4 directly obtains the unconditional consistency of the subsampling estimator with respect to the true parameters under the unbounded assumption.

To prove the asymptotic normality of ${\widehat{\mathit{\beta }}}_n$ with respect to ${\mathit{\beta }}_0$, we briefly review the subsampled score function in Section 3.2

$$s_n\left(\mathit{\beta }\right)=\sum _{i\in S}\frac{1}{{\pi }_i^{\ast }}\left[Y_i^{\ast }-\mu \left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\right)\right]\dot{\psi }\left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right){\mathit{X}}_i^{\ast }:=\sum _{i\in S}\frac{1}{{\pi }_i^{\ast }}{\varphi }_{\mathit{\beta }}({\mathit{X}}_i^{\ast },Y_i^{\ast }).$$

Next we will apply a multivariate martingale central limit theorem (Lemma 4 in [19]), which is the extension of Theorem A.1 in [20], to show the asymptotic normality of ${\widehat{\mathit{\beta }}}_n$. Let ${\left\{{\mathcal{F}}_{N,i}\right\}}_{i=1}^n$ be a filtration adaptive to the sampling: ${\mathcal{F}}_{N,0}=\sigma ({\mathit{X}}_1^N,Y_1^N);{\mathcal{F}}_{N,1}=\sigma ({\mathit{X}}_1^N,Y_1^N)\vee \sigma \left({\ast }_1\right);\dots ;{\mathcal{F}}_{N,i}=\sigma ({\mathit{X}}_1^N,Y_1^N)\vee \sigma \left({\ast }_1\right)\vee \dots \vee \sigma \left({\ast }_i\right);\dots$, where $\sigma \left({\ast }_i\right)$ is the $\sigma$-algebra generated by ith sampling step. The subsample of size n is assumed to increase with N. By the filtration, we define the martingale

$$\overline{M}:=\sum _{i=1}^n{\overline{M}}_i:=\sum _{i=1}^n\left[\frac{1}{{\pi }_i^{\ast }}{\varphi }_{\mathit{\beta }}({\mathit{X}}_i^{\ast },Y_i^{\ast })-\sum _{j=1}^N{\varphi }_{\mathit{\beta }}({\mathit{X}}_j,Y_j)\right],$$

where ${\left\{{\overline{M}}_i\right\}}_{i=1}^n$ is a martingale difference sequence adapted to ${\left\{{\mathcal{F}}_{N,i}\right\}}_{i=1}^n$. In addition, define $Q:=n{\sum }_{j=1}^N{\varphi }_{\mathit{\beta }}({\mathit{X}}_j,Y_j)$; $T:=s_n\left(\mathit{\beta }\right)=\overline{M}+Q$; ${\xi }_{Ni}:={\mathrm{Var}}^{-1/2}\left(T\right){\overline{M}}_i$ and $B_N:={\mathrm{Var}}^{-1/2}\left(T\right)\mathrm{Var}\left(\overline{M}\right){\mathrm{Var}}^{-1/2}\left(T\right)$, where matrix ${\mathit{A}}^{1/2}$ is the symmetric square root of $\mathit{A}$, i.e., $\mathit{A}={\left({\mathit{A}}^{1/2}\right)}^2$, and ${\mathit{A}}^{-1/2}={\left({\mathit{A}}^{1/2}\right)}^{-1}={\left({\mathit{A}}^{-1}\right)}^{1/2}$. $B_N$ is the variance of ${\mathrm{Var}}^{-1/2}\left(T\right)\overline{M}$.

The following theorem shows the asymptotic normality of the estimator ${\widehat{\mathit{\beta }}}_n$.

Theorem 5.

Assume the conditions,

(C.1) 

$$\mathsf{\Phi }=\mathrm{E}(\nabla s_n\left(\mathit{\beta }\right))=\mathrm{E}\left[-\sum _{i\in S}\frac{1}{{\pi }_i^{\ast }}\dot{\mu }\left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\right){\left[\dot{\psi }\left({\mathit{\beta }}^T{\mathit{X}}_i^{\ast }\right)\right]}^2{\mathit{X}}_i^{\ast }{\mathit{X}}_i^{\ast T}\right]$$

is finite and nonsingular.

(C.2) 

$\mathrm{E}\left\{{\left[\sum _{i\in U}\frac{a_i}{{\pi }_i}\dot{\mu }\left(\psi \left({\mathit{\beta }}^T{\mathit{X}}_i\right)\right){\left[\dot{\psi }\left({\mathit{\beta }}^T{\mathit{X}}_i\right)\right]}^2{X}_{ik}{X}_{ij}\right]}^2\right\}=o_P\left(1\right)$, for $1\le k,j\le p$,where $X_{ik}$ means k-th element of vector ${\mathit{X}}_i$ and $X_{ij}$ means j-th element of vector ${\mathit{X}}_i$.

(C.3) 

$\psi \left(x\right)$ is three-times continuously differentiable for every x with its domain.

(C.4) 

For any $i\in S$, $\left|\right|{\ddot{\varphi }}_{\mathit{\beta }}({\mathit{X}}_i^{\ast },Y_i^{\ast })\left|\right|<\infty$.

(C.5) 

${min}_{i=1,...,N}\left(N{\pi }_i\right)={max}_{i=1,...,N}\left(N{\pi }_i\right)=O\left(1\right)$ and $n/N=o\left(1\right)$.

(C.6) 

$\underset{N\to \infty }{lim}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[\right||{\xi }_{Ni}{\left|\right|}^4]=0$,

(C.7) 

$\underset{N\to \infty }{lim}\mathrm{E}\left[{\left|\left|{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\xi }_{Ni}{\xi }_{Ni}^T\right|{\mathcal{F}}_{N,i-1}]-B_N\right|\right|}^2\right]=0.$

Then

$$\mathrm{Var}{\left(T\right)}^{-1/2}\mathsf{\Phi }({\widehat{\mathit{\beta }}}_n-{\mathit{\beta }}_0)\stackrel{d}{\to }N(\mathbf{0},{\mathbf{I}}_p).$$

Here, we establish the unconditional asymptotic properties of subsampling estimator for unbounded GLMs. The condition $n/N=o\left(1\right)$ ensures that small-scale subsamples also have expected performance, which greatly release the computational cost. We also present the theoretical asymptotic results, which leads to the subsampling probability using the A-optimal criterion in [10].

4. Conclusions and Future Work

In this paper, we derive the asymptotic normality of the subsampling M-estimator by Fisher information. In the unbounded GLMs with nonnatural link function, we separately obtain the conditional and unconditional asymptotic properties of subsampling estimator.

For future study, it is meaningful to apply the sub-Weibull concentration inequalities in [21] to make nonasymptotic inference. The importance sampling is not ideal, since it tends to assign high sampling probability to the observed samples. Hence, effective subsampling methods are considered for GLMs, such as Markov subsampling in [22]. Moreover, high-dimensional methods in [23,24] for subsampling need further studies.