Cross-Validation Model Averaging for Generalized Functional Linear Model

Abstract

Functional data is a common and important type in econometrics and has been easier and easier to collect in the big data era. To improve estimation accuracy and reduce forecast risks with functional data, in this paper, we propose a novel cross-validation model averaging method for generalized functional linear model where the scalar response variable is related to a random function predictor by a link function. We establish asymptotic theoretical result on the optimality of the weights selected by our method when the true model is not in the candidate model set. Our simulations show that the proposed method often performs better than the commonly used model selection and averaging methods. We also apply the proposed method to Beijing second-hand house price data.

1. Introduction

In recent years, functional data have been increasingly popular in many scientific areas. A common question for the functional data is how to quantify the relationship between functional covariates and scalar responses. Functional linear model (FLM) and generalized functional linear model (GFLM) can take account of some associations between the response and the different points in the domain of the functional covariates, and therefore are two useful tools in many studies for functional data. These two models have now been widely used to solve practical problems, such as exploring the relationship between the growth and age in the life sciences, analyzing the weather data in different areas, recognizing the handwriting data, and conducting the diffusion tensor imaging studies. Functional data analysis usually represents functional covariates and coefficient functions by some linear combinations of a set of basis functions, such as a prespecified basis system like B-splines, Fourier and wavelet bases (James 2002), and data-adaptive basis functions from functional principal component analysis (FPCA) (Yao et al. 2005). We are concerned with the GFLM because it can estimate the flexible and nonlinear relationships between the functional covariates and scalar responses for many types of data such as binary response data, Poisson response data, and multivariate discrete response data. See, for example, James (2002), who expanded generalized linear models to generalized functional linear models with the functional principal component methodology and demonstrated that this approach can be performed for linear, logistic and censored regressions in simulations and real data analysis.

In econometrics, the relationship between time series and scalar response is often of interest. We can use GFLM instead of generalized linear model to handle the case where a time series with the dependence at different time points is used as the explanatory variables with dimension toward to infinity. On the other hand, prediction is often the main goal in econometric data analysis. Several approaches have been proposed to select some important principal components in FPCA such as AIC, BIC, and leave-one-out cross-validation (Müller and Stadtmüller 2005). However, as we will demonstrate, the model selection alone, such as AIC, is not an optimal approach for the purpose of estimation and prediction. in one model selected by AIC or BIC may lead to the loss of information from other models. Different models often capture different data characteristics and therefore model averaging generally gets higher estimating or predicting accuracy, which has received extensive attention in recent years.

Model averaging has two research directions: Bayesian Model Averaging (BMA) and Frequentist Model Averaging (FMA). We will focus on the latter in this paper. A key problem with the FMA is the choice of weights assigned to different models. In this regard, various approaches have been developed. See, for example, smoothed AIC, smoothed BIC (Buckland et al. 1997), smoothed FIC (Hjort and Claeskens 2003; Claeskens and Carroll 2007; Zhang and Liang 2011; Zhang et al. 2012; Xu et al. 2014), Adaptive method (Yang 2001), MMA method (Hansen 2007; Wan et al. 2010), OPT method (Liang et al. 2011), JMA method (Hansen and Racine 2012; Zhang et al. 2013), and leave-subject-out cross-validation method (Gao et al. 2016), which apply to independent, or time series, or longitudinal data.

For functional data, some model averaging methods have been studied. Zhu et al. (2018) proposed a model averaging estimator based on Mallows’ criterion for partial functional linear models whose response is a scalar and the predictors are a random vector and some functional variables. Zhang et al. (2018) proposed a Jackknife model averaging for fully functional linear models whose response and predictor are both functional processes. For generalized functional linear model designed for the case where the scalar response is nonlinearly dependent on functional explanatory variables, model averaging is a good alternative to model selection that may lead to instability in variable selection or coefficient estimation caused by randomness of the data collection and so on.

In this article, we consider model averaging methods for GFLM to capture the nonlinear characteristics hidden in the data and to reduce the prediction errors and risks. The contributions of this article are threefold: We first adopt FPCA to reduce the dimensions as it provides a parsimonious representation of functional data, and then present a novel model averaging procedure based on leave-one-out cross-validation criterion (CV). Second, we prove the consistency of parameter estimator under the misspecified model with some mild conditions. The dimension of the parameter can be divergent. Third, we establish the asymptotic optimality of our method in the squared loss sense for generalized linear model with a diverging number of parameters. Our work relaxes the condition that the expectations of estimators need to exist.

The rest of the article is organized as follows. In Section 2, we introduce our proposed model averaging method for GFLM. We then establish the asymptotic property of the proposed method in Section 3. Simulation studies and a real data example of second-hand house price in Beijing are presented in Section 4. Section 5 concludes. Proofs of theoretical results are provided in Appendix A and Appendix B.

2. Model Averaging for Generalized Functional Linear Model

2.1. The Generalized Functional Linear Model

The data we collected for the ith subject or experimental unit are $\left(\left\{X_i\left(t\right),t\in T\right\},y_i\right),i=1,\dots ,n$. We assume these data are generated independently. The predictor variable $X\left(t\right)\left(t\in T\right)$ is a random curve corresponding to a square integrable stochastic process on a real interval T. The response variable is a real-valued random variable that may be continuous or discrete. For example, in a binary regression, one would have $y\in \{0,1\}$.

Suppose that the given link function $g(·)$ is a strictly monotone and twice continuously differentiable function with bounded derivatives and is thus invertible. This assumption is common in generalized linear model. See, for example, (Chen et al. 1999; Müller and Stadtmüller 2005; Ando and Li 2017). Moreover, we assume a variance function ${\sigma }^2(·)$, which is strictly positive with upper bound defined on the range of the link function. The generalized functional linear model or functional quasi-likelihood model is determined by a parameter function $\beta (·)$, which is square integrable on its domain T, in addition to the link function $g(·)$ and the variance function ${\sigma }^2(·)$.

Given a real measure $d\omega$ on T, we define linear predictors

$${\eta }_i=\alpha +\int \beta \left(t\right)X_i\left(t\right)d\omega \left(t\right),i=1,\dots n,$$

and conditional means ${\mu }_i=g\left({\eta }_i\right)$, where $\mathbf{E}\left(y_i|X_i\left(t\right),t\in T\right)={\mu }_i$, and $\mathbf{Var}\left(y_i|X_i\left(t\right),t\in T\right)={\sigma }^2\left({\mu }_i\right)={\tilde{\sigma }}^2\left({\eta }_i\right)$ with the function ${\tilde{\sigma }}^2\left({\eta }_i\right)={\sigma }^2\left(g\left({\eta }_i\right)\right)$. In a generalized functional linear model, the distribution of $y_i$ would be specified with the exponential family. Thus, we should consider a functional quasi-likelihood model

$$y_i=g\left(\alpha +\int \beta \left(t\right)X_i\left(t\right)d\omega \left(t\right)\right)+e_i,i=1,\dots n,$$

(1)

where $\mathbf{E}\left(e_i|X_i\left(t\right),t\in T\right)=0$ and $\mathbf{Var}\left(e_i|X_i\left(t\right),t\in T\right)={\sigma }^2\left({\mu }_i\right)={\tilde{\sigma }}^2\left({\eta }_i\right)$. Note that $\alpha$ is a constant, and the inclusion of an intercept allows us to require $\mathbf{E}\left(X_i\left(t\right)\right)=0$ for all t. We assume the errors $e_i$ are independent with the same variance. It is easy to obtain $\mathbf{E}\left(e_i\right)=0$ and

$$\mathbf{Var}\left(e_i\right)=\mathbf{Var}\left\{\mathbf{E}\left(e_i\right|X_i\left(t\right),t\in T)\right\}+\mathbf{E}\left\{\mathbf{Var}\left(e_i\right|X_i\left(t\right),t\in T)\right\}=\mathbf{E}\left\{{\tilde{\sigma }}^2\left({\eta }_i\right)\right\}={\sigma }^2.$$

Following Müller and Stadtmüller (2005), we choose an orthonormal basis $\left\{{\rho }_j,i=1,2\dots \right\}$ of the function space $L^2\left(d\omega \right)$, that is ${\int }_T{\rho }_j\left(t\right){\rho }_k\left(t\right)d\omega \left(t\right)={\delta }_{jk}$, where ${\delta }_{jk}=0$ for $j\ne k$ and ${\delta }_{jk}=1$ for $j=k$. Then, we can expand the predictor process $X\left(t\right)$ and the parameter function $\beta \left(t\right)$ as

$$X\left(t\right)=\sum _{j=1}^{\infty }{\epsilon }_j{\rho }_j\left(t\right),$$

and

$$\beta \left(t\right)=\sum _{j=1}^{\infty }{\beta }_j{\rho }_j\left(t\right),$$

in the $L^2\left(d\omega \right)$ sense] with random variables ${\epsilon }_j$ and coefficients ${\beta }_j$ given by ${\epsilon }_j=\int X\left(t\right){\rho }_j\left(t\right)d\omega \left(t\right)$ and ${\beta }_j=\int \beta \left(t\right){\rho }_j\left(t\right)d\omega \left(t\right)$, respectively. By the previous assumptions that $X\left(t\right)$ and $\beta \left(t\right)$ are square integrable, we get ${\sum }_{j=1}^{\infty }{\beta }_j^2<\infty$ and ${\sum }_{j=1}^{\infty }\mathbf{E}{\epsilon }_j^2<\infty$.

From the orthonormality of the basis function ${\rho }_j$ and setting

$${\epsilon }_{i,j}=\int X_i\left(t\right){\rho }_j\left(t\right)d\omega \left(t\right),$$

it follows immediately that

$${\eta }_i=\alpha +\int \beta \left(t\right)X_i\left(t\right)d\omega \left(t\right)=\alpha +\sum _{j=1}^{\infty }{\beta }_j{\epsilon }_{i,j}.$$

It will be convenient to work with standardized errors

$$e_i^{\prime }=e_i/\sigma \left({\mu }_i\right)=e_i/\tilde{\sigma }\left({\eta }_i\right),$$

in which $\mathbf{E}\left(e_i^{\prime }\right|X_i\left(t\right))=0$, $\mathbf{E}\left(e_i^{\prime }\right)=0$, and $\mathbf{E}\left(e_i^{\prime 2}\right)=1$. Then, it will be sufficient to consider the following model,

$$y_i=g\left(\alpha +\sum _{j=1}^{\infty }{\beta }_j{\epsilon }_{i,j}\right)+e_i^{\prime }\tilde{\sigma }\left(\alpha +\sum _{j=1}^{\infty }{\beta }_j{\epsilon }_{i,j}\right),i=1,\dots ,n,$$

(2)

where the function $g(·)$ is known.

The number of parameter in model (2) is infinite. We address the difficulty caused by the infinite dimensionality of the predictors by approximating model (2) with a series of models where the number of predictors is truncated at $p=p_n$, and the dimension $p_n$ can be a constant as large as possible with $p_n<n$. A heuristic truncation strategy is as follows. For the ith sample, a p-truncated linear predictor ${\eta }_{i,p}$ is

$${\eta }_{i,p}=\alpha +\sum _{j=1}^p{\beta }_j{\epsilon }_{i,j}.$$

The approximating model we use is

$$y_i=g\left(\alpha +\sum _{j=1}^p{\beta }_j{\epsilon }_{i,j}\right)+e_i^{\prime }\tilde{\sigma }\left(\alpha +\sum _{j=1}^p{\beta }_j{\epsilon }_{i,j}\right),i=1,\dots ,n.$$

Now, we consider the estimation for generalized functional linear model. First, we use FPCA to get a set of orthogonal eigenfunctions as the basis functions in the space $L^2\left(d\omega \right)$. Then, we consider a series of candidate models. The number of candidate models is M. For the mth candidate model, we adopt the first $p_m$ functional principal components to build the approximating model,

$$y_i=g\left({\alpha }^{\left(m\right)}+\sum _{j=1}^{p_m}{\beta }_j^{\left(m\right)}{\epsilon }_{i,j}\right)+e_i^{\prime }\tilde{\sigma }\left({\alpha }^{\left(m\right)}+\sum _{j=1}^{p_m}{\beta }_j^{\left(m\right)}{\epsilon }_{i,j}\right),i=1,\dots ,n.$$

(3)

We assume that $p_1<p_2<\dots <p_M$. That is, the candidate models are nested. Denote ${\epsilon }_{i,0}=1$ and ${\beta }_0^{\left(m\right)}={\alpha }^{\left(m\right)}$, then we estimate the unknown parameter vector ${\beta }^{\left(m\right)}={({\beta }_0^{\left(m\right)},{\beta }_1^{\left(m\right)},\dots ,{\beta }_{p_m}^{\left(m\right)})}^T$ by solving the following estimating or score equation

$$U_{n,m}\left({\beta }^{\left(m\right)}\right)=\frac{1}{n}\sum _{i=1}^n\left[y_i-g\left({\eta }_{i,p_m}\right)\right]\frac{g^{\prime }\left({\eta }_{i,p_m}\right)}{{\sigma }^2\left({\mu }_{i,p_m}\right)}{\epsilon }_{(i,p_m)}=0,$$

(4)

where ${\eta }_{i,p_m}={\sum }_{j=0}^{p_m}{\beta }_j^{\left(m\right)}{\epsilon }_{i,j}$ and ${\epsilon }_{(i,p_m)}={({\epsilon }_{i,0},\dots ,{\epsilon }_{i,p_m})}^T$. Let ${\widehat{\beta }}^{\left(m\right)}$ be the solution of the score equation $U_{n,m}\left({\beta }^{\left(m\right)}\right)=0$, i.e.,

$$U_{n,m}\left({\widehat{\beta }}^{\left(m\right)}\right)=\frac{1}{n}\sum _{i=1}^n[y_i-g\left({\widehat{\eta }}_{i,p_m}\right)]\frac{g^{\prime }\left({\widehat{\eta }}_{i,p_m}\right)}{{\sigma }^2\left(g\left({\widehat{\eta }}_{i,p_m}\right)\right)}{\epsilon }_{(i,p_m)}=0.$$

(5)

2.2. Model Averaging Estimation

For each candidate model, we get the estimator of the unknown parameter vector by (4). Let

$$w\in H_n=\left\{w\in {[0,1]}^M:\sum _{m=1}^M{w}_m=1\right\},$$

then we obtain the model averaging estimator of ${\eta }_i$:

$${\widehat{\eta }}_i\left(w\right)=\sum _{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m},$$

(6)

where ${\widehat{\eta }}_{i,p_m}={\sum }_{j=0}^{p_m}{\widehat{\beta }}_j^{\left(m\right)}{\epsilon }_{ij}$. Thus, a model averaging estimator of the conditional mean ${\mu }_i$ is given by

$${\widehat{\mu }}_i\left(w\right)=g\left(\sum _{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m}\right).$$

(7)

Let ${\tilde{\beta }}_{-j}^{\left(m\right)}$ be the estimator of ${\beta }^{\left(m\right)}$ from (4) without the jth observation, that is,

$$U_{n,m,-j}\left({\beta }^{\left(m\right)}\right)=\frac{1}{n-1}\sum _{i=1,i\ne j}^n\left[y_i-g\left({\eta }_{i,p_m}\right)\right]\frac{g^{\prime }\left({\eta }_{i,p_m}\right)}{{\sigma }^2\left({\mu }_{i,p_m}\right)}{\epsilon }_{(i,p_m)}=0.$$

(8)

For the observation j, the leave-one-out truncated linear estimator of ${\eta }_j$ under the mth model is

$${\tilde{\eta }}_{j,p_m}={\tilde{\beta }}_{-j}^{\left(m\right)T}{\epsilon }_{(j,p_m)},$$

and the leave-one-out model averaging estimator of ${\mu }_j$ is

$${\tilde{\mu }}_j\left(w\right)=g\left({\sum }_{m=1}^M{w}_m{\tilde{\eta }}_{j,p_m}\right).$$

Thus, we propose the following leave-one-out criterion for choosing weights in the model averaging estimator given by (7)

$$CV\left(w\right)=\sum _{i=1}^n{\left(y_i-{\tilde{\mu }}_i\left(w\right)\right)}^2=\sum _{i=1}^n{\left[y_i-g\left({\sum }_{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)\right]}^2.$$

(9)

Let

$$\widehat{w}=arg\underset{w\in H_n}{min}CV\left(w\right)$$

be the weight vector from $CV\left(w\right)$ criterion. Then, plugging $\widehat{w}$ into (7), we obtain the final model averaging estimator ${\widehat{\mu }}_i\left(\widehat{w}\right),i=1,2,\dots ,n$.

3. Asymptotic Property for Model Averaging Estimator

In this section, we will establish the optimal property of cross-validation model averaging for generalized functional linear model. We allow the dimension of each candidate model to be divergent as n tends to ∞.

Notations and Conditions

We denote the first and second derivatives of the function $g(·)$ by $g^{{}^{\prime }}(\dots )$ and $g^{{}^{″\prime }}(\dots )$, respectively, the diagonal matrix A with diagonal elements $a_1,a_2,\dots ,a_q$ by $A=diag(a_1,a_2,\dots ,a_q)$, the minimum singular value of matrix A by ${\lambda }_{min}\left\{A\right\}$, and

$${\lambda }_{n,m}={\lambda }_{min}\left\{\frac{{\epsilon }_n^{\left(m\right)T}{\epsilon }_n^{\left(m\right)}}{n}\right\},$$

with

$${\epsilon }_n^{\left(m\right)}={\left({\epsilon }_{(1,p_m)},{\epsilon }_{(2,p_m)},\dots ,{\epsilon }_{(n,p_m)}\right)}^T.$$

For any ${\beta }^{\left(m\right)}\in {\mathbb{R}}^{p_m+1}$, $n\in {\mathbb{N}}^{+}$, define

$$\begin{array}{c}\hfill U_{n,m}^{\left(m\right)}\left({\beta }^{\left(m\right)}\right)=\frac{1}{n}\sum _{i=1}^n\left[g\left({\beta }^{\left(m\right)T}{\epsilon }_{(i,p_m)}\right)-{\mu }_i\right]\frac{g^{{}^{\prime }}\left({\beta }^{\left(m\right)T}{\epsilon }_{(i,p_m)}\right)}{{\sigma }^2\left(g\left({\beta }^{\left(m\right)T}{\epsilon }_{(i,p_m)}\right)\right)}{\epsilon }_{(i,p_m)}.\end{array}$$

(10)

We assume $\left|g^{\prime }(·)\right|⩽c<\infty$ and $\left|g^{″}(·)\right|⩽c_1<\infty$, and ${\sigma }^2(·)$ is strictly positive with bound $0<d_1⩽{\sigma }^2(·)⩽d_2<\infty$ and $\left|{\sigma }^{2^{\prime }}(·)\right|⩽d_3<\infty$.

Consider the squared loss function

$$L_n\left(w\right)={\parallel \mu -\widehat{\mu }\left(w\right)\parallel }^2,$$

where $\mu ={\left({\mu }_1,{\mu }_2,\dots ,{\mu }_n\right)}^T$ and $\widehat{\mu }\left(w\right)={\left({\widehat{\mu }}_1\left(w\right),{\widehat{\mu }}_2\left(w\right),\dots ,{\widehat{\mu }}_n\left(w\right)\right)}^T$ are the two $n\times 1$ vectors, and ${\parallel ·\parallel }^2$ is Euclidean norm. Denote

$$R_n\left(w\right)=\sum _{i=1}^n{\left[g\left({\eta }_i\right)-g\left(\sum _{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)\right]}^2,$$

and

$${\xi }_n=\underset{w\in H_n}{inf}{R}_n\left(w\right),$$

where ${\beta }_{★}^{\left(m\right)}$ is the pseudo true parameter, which, like Flynn et al. (2013) and Lv and Liu (2014), is defined as the solution to the following score equation,

$$U_{n,m}^{\left(m\right)}\left({\beta }^{\left(m\right)}\right)=0,$$

and is a theoretical target under the mth candidate model with misspecification. We assume that such a solution is existent and ${∥{\beta }_{★}^{\left(m\right)}∥}^2/\left(p_m+1\right)\le C_b<\infty$. ${\xi }_n$ represents the minimal bias between the true model and the final model generated by model averaging, which is an alternative to the risk based on $L_n\left(w\right)$. In this work, we do not require the expectation of $L_n\left(w\right)$ to exist, which is more relaxed than the common requirement on jackknife model averaging methods for generalized linear model. See, for example, Zhang et al. (2016) and Ando and Li (2017). In the following, we assume that $X_i\left(t\right),i=1,2,\dots ,n$ are non-random with ${sup}_i\left|{\eta }_i\right|⩽C_{\eta }<\infty$.

Condition 1.

For some compact set ${\Theta }_m$ in ${\mathbb{R}}^{p_m+1}$,

$$\underset{n\to +\infty }{lim}\mathbf{P}\left\{0\in U_{n,m}\left({\Theta }_m\right)\right\}=1$$

holds.

Condition 2.

(i) $\left\{e_i\right\},i=1,\dots ,n$ are mutually independent.

(ii) $\mathbf{E}{e}_i=0$.

(iii) $C_1={sup}_i\mathbf{E}{e}_i^2<\infty$.

Condition 3.

${sup}_i{∥{\epsilon }_{(i,p_m)}∥}^2/\left(p_m+1\right)⩽C_2<\infty$.

Condition 4.

$\sqrt{n}{p}^2/{\xi }_n\to 0$ with $p=\underset{m}{max}{p}_m$ and $p^4/n=o\left(1\right)$.

Condition 5.

${\sum }_{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2=O_p\left(p_m^4\right)$.

Condition 6.

${\lambda }_{min}\left\{\frac{\partial }{\partial {\beta }^{\left(m\right)}}{U}_{n,m}^{\left(m\right)}\left({\beta }^{\left(m\right)}\right)\right\}⩾C_0>0.$

Condition 1 is a requirement for generalized model to guarantee the existence of solutions to (4). In general, the existence and consistency of roots obtained by solving (4) have to be checked, so we list Condition 1. The similar condition can be found in Balan and Schiopu-Kratina (2005). In the special case where the link function is $g\left(x\right)=x$, the solution of (4) is a generalized least squares estimator of ${\beta }^{\left(m\right)}$ and Condition 1 is easy to satisfy.

Condition 2 is common for generalized linear model. See, for instance, Chen et al. (1999) and Ando and Li (2017). The least squares estimator for linear regression models is strongly consistent under Condition 2. This condition is less restrictive than (A1) of Ando and Li (2017) for proving the optimality of the weight selection procedure.

Condition 3 is similar to (2.3) of Theorem 1 in Chen et al. (1999) and is due to the nonlinearity. A counterexample is given to show that ${\widehat{\beta }}^{\left(m\right)}$ may not be consistent when Condition 3 (i) is dropped in Chen et al. (1999).

Condition 4 means that the speed of ${\xi }_n$ tending to ∞ should be faster than that of $\sqrt{n}{p}^2$. This condition also implies that the true model is not in the candidate model set, which is a condition commonly used for optimal model averaging. It is easy to satisfy when the true model is an infinite dimensional model. This condition is an alternative to Condition C.3 of Zhang et al. (2016) and (A3) of Ando and Li (2017).

Condition 5 implies $n^{-1}{\sum }_{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2=o_p\left(1\right)$ with $p_m^4/n=o\left(1\right)$. By Lemma A3 in the Appendix A and Condition 3, we have

$$\sum _{i=1}^n{\left({\widehat{\eta }}_{i,p_m}-{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2⩽\sum _{i=1}^n{∥{\epsilon }_{(i,p_m)}∥}^2{∥{\widehat{\beta }}^{\left(m\right)}-{\beta }_{★}^{\left(m\right)}∥}^2=O_p\left(p_m^4\right).$$

Then, with the following standard condition for the application of cross-validation,

$$\left|\frac{{\sum }_{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2}{{\sum }_{i=1}^n{\left({\widehat{\eta }}_{i,p_m}-{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2}-1\right|=o_p\left(1\right),$$

which says that as n gets large, the difference between the ordinary and leave-one-out estimators of ${\eta }_i$ under the mth candidate model gets small, it can be seen that

$$\sum _{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2⩽2\sum _{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2+2\sum _{i=1}^n{\left({\widehat{\eta }}_{i,p_m}-{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2=O_p\left(p_m^4\right),$$

which means Condition 5 is reasonable. For the one-parameter natural exponential family models, Ando and Li (2017) showed under some regularity conditions that ${\sum }_{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2=O_p\left(p_m^2/n\right)$ satisfying our Condition 5. For the linear models where $g\left(x\right)=x$ and ${\sigma }^2(·)=1$, ${\sum }_{i=1}^n{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2=O_P\left(p_m^2/n\right)$ under the assumption that ${\epsilon }_{(i,p_m)}^T{\left({\epsilon }_n^{\left(m\right)T}{\epsilon }_n^{\left(m\right)}\right)}^{-1}{\epsilon }_{(i,p_m)}\le c{p}_m/n$ for some constant $c<\infty$, which is commonly used to ensure the asymptotic optimality of cross-validation. See, for example, Condition (5.2) of Li (1987), Condition (5.2) of Andrews (1991), Condition (A.9) of Hansen and Racine (2012), Condition (C.2) of Zhang (2015), and Condition (C.3) of Zhao et al. (2018). In general, our Condition 5 is more relaxed than those in literature for the complex candidate models.

Condition 6 is to ensure that the pseduo true parameter ${\beta }_{★}^{\left(m\right)}$ is unique. The consistency of the estimator of ${\beta }_{★}^{\left(m\right)}$ can also be derived by this condition. See Lemma A3 in the Appendix A. In addition, the one-parameter natural exponential family considered in Theorem 1 of Ando and Li (2017) is an example with

$${\lambda }_{min}\left\{\frac{\partial }{\partial {\beta }^{\left(m\right)}}{U}_{n,m}^{\left(m\right)}\left({\beta }^{\left(m\right)}\right)\right\}={\lambda }_{min}\left\{\frac{1}{n}{\epsilon }_n^{\left(m\right)T}\Gamma \left({\beta }^{\left(m\right)}\right){\epsilon }_n^{\left(m\right)}\right\},$$

where

$$\Gamma \left({\beta }^{\left(m\right)}\right)=diag\left(g^{\prime }\left({\epsilon }_{(1,p_m)}^T{\beta }^{\left(m\right)}\right),g^{\prime }\left({\epsilon }_{(2,p_m)}^T{\beta }^{\left(m\right)}\right),\dots ,g^{\prime }\left({\epsilon }_{(n,p_m)}^T{\beta }^{\left(m\right)}\right)\right).$$

By the commonly used assumption that ${\lambda }_{n,m}⩾c_0>0$ for some constant $c_0<\infty$, and the assumption (4.3) in Ando and Li (2017), this example satisfies Condition 6.

Theorem 1.

Assume that Conditions 1–6 hold, then $\widehat{w}$ is asymptotically optimal in the sense that

$$\frac{L_n\left(\widehat{w}\right)}{{inf}_{w\in H_n}{L}_n\left(w\right)}\stackrel{p}{\to }1,$$

(11)

where $\stackrel{p}{\to }$ means convergence in probability.

Proof. 

See the Appendix B.  □

Remark 1.

When the dimensions of the candidate models are fixed, condition 4 can be relaxed to $n/{\xi }_n^2\to 0.$

Remark 2.

It is easy to see that if we do not require that the weights sum to one, then we can use M instead of 1 as the upper bound of ${\sum }_{m=1}^M{w}_m^2$ in our proof. Thus, all the proofs are still valid for the fixed M. This implies that Theorem 1 remains true if we remove the constraint that the weights sum to one. In addition, as the candidate models are not necessarily nested in the proof, this theorem still holds when the candidate models are non-nested.

4. Numerical Examples

4.1. Simulation I: Fixed Number of Candidate Models

In this section, we conduct simulation experiments to compare the finite sample performance of our model averaging methods and some commonly used model selection and model averaging methods. For model selection, we consider three methods: AIC, BIC, and FPCA. FPCA is an efficient and common method in functional data analysis, which determines the final model by the cumulative contributions of the functional principal components. For model averaging, we consider the following methods, S-AIC (smoothed AIC), S-BIC (smoothed BIC), and our cross-validation model averaging, which is denoted as CV1 if we restrict the sum of weights to be 1 as before, and CV2 if no constraint on the sum of weights is imposed.

The data generating process is as follows: the predictor variable is

$$X_i\left(t\right)=\sum _{j=1}^J{\epsilon }_{i,j}{\rho }_j\left(t\right),$$

and the parameter function is

$$\beta \left(t\right)=\sum _{j=1}^J{\beta }_j{\rho }_j\left(t\right),$$

where ${\rho }_j\left(t\right)$ is a basis function with $t\in [0,1]$, and $j\ge 1$ and J is the number of the basis functions. Here, we use B-spline base and Fourier base. For B-spline base, we choose the order of the basis functions to be 2, and the number of the basis functions to be 20. As for Fourier base, we choose the number of the basis functions as 21 and the first basis to be a constant function.

In our simulation, the following four cases are considered.

Case 1 

For $1\le j\le 10$, ${\beta }_j$ are generated from the standard normal distribution $N(0,1)$; for $10<j\le 20$, ${\beta }_j=0$. The basis functions $\left\{{\rho }_j\left(t\right),t\in [0,1],1\le j\le 20\right\}$ are B-spline functions with parameters as mentioned above.

Case 2 

For $1\le j\le 20$, ${\beta }_j=j^{-2}$. The basis functions $\left\{{\rho }_j\left(t\right),t\in [0,1],1\le j\le 20\right\}$ are B-spline functions with parameters as mentioned above.

Case 3 

For $1\le j\le 11$, ${\beta }_j$ are generated from the standard normal distribution $N(0,1)$; for $11<j\le 21$, ${\beta }_j=0$. The basis functions $\left\{{\rho }_j\left(t\right),t\in [0,1],1\le j\le 21\right\}$ are Fourier functions with parameters as mentioned above.

Case 4 

For $1\le j\le 21$, ${\beta }_j=j^{-2}$. The basis functions $\left\{{\rho }_j\left(t\right),t\in [0,1],1\le j\le 20\right\}$ are Fourier functions with parameters as mentioned above.

We set the term ${\epsilon }_{i,j}$ to be independently generated from $N(0,R^2/j^2)$, where $R=1,2,\dots ,10$. The response variable $y_i$ is generated from binomial distribution $Binomial(p\left(X_i\left(t\right)\right),1)$ with the probability $p\left(X_i\left(t\right)\right)$ being $g\left({\int }_0^1{X}_i\left(t\right)\beta \left(t\right)dt\right)$. We consider three types of link function $g(·)$: logistic link function $exp(·)/(1+exp(·\left)\right)$, Probit link function, and Poisson link function. For the Poisson model, we only consider the simulations with $R=1$ for Cases 1–4.

In the simulation, we use FPCA to obtain the nested candidate models. Each candidate model contains the first $p_m$ principal components. The number of candidate models is 18 for Cases 1–2 and 19 for Cases 3–4. Then we adopt the weighted iterated least squares algorithm which is a common approach in generalized linear model to get the estimates for each model. For the weights, we use the ’fmincon’ function in Matlab to get the solution of CV criterion.

The sample size is set as $n=60,200,500$. We use the 80% data as the training data $\left\{Y_1,X_1\right\}$ with size $n_1$, and the remaining data as the test data $\left\{Y_2,X_2\right\}$ with size $n_2$. Then, we compare the prediction errors. We calculate the prediction accuracy (${∥{\widehat{Y}}_2-Y_2∥}^2/n_2$), fitting accuracy (${∥{\widehat{Y}}_1-Y_1∥}^2/n_1$), predictor coefficient prediction accuracy (${∥{\widehat{\eta }}_{\left(2\right)}-{\eta }_{\left(2\right)}∥}^2/n_2$ ), and predictor coefficient fitting accuracy (${∥{\widehat{\eta }}_{\left(1\right)}-{\eta }_{\left(1\right)}∥}^2/n_1$ ). We repeat this process 1000 times, and then obtain mean, median, and variance of these prediction errors for each method. To save space, we present only the results on the prediction accuracy. The results on the other type accuracies are available from the authors upon request. We only report the results for logistic link function due to space limitations. Other link function results are also available from the authors.

For Case 1, the prediction errors are summarized in Table A1, Table A2 and Table A3. From Table A1, it is seen that with R varying from 1 to 10, the prediction errors are decreasing, because the difference of probability between the two groups (one group whose response is 1 and the other group whose response is 0) becomes larger. Our methods (CV1 and CV2 in the tables) always obtain the minimum error means (Mean in the tables), medians (Median in the tables), and variances (Var in the tables). However, there is no clear tendency between CV1 and CV2, which perform similarly in most of situations. When R is small, BIC is always better than AIC, and S-BIC is always better than S-AIC. This may be due to less parameters being useful for smaller R values, and in this case, a bigger penalty on the number of parameters in the model is preferred. Moreover, when the candidate models differ significantly, AIC or BIC performs similarly to S-AIC or S-BIC, respectively. As R becomes larger, the difference between AIC and BIC or S-AIC and S-BIC becomes smaller. FPCA is always superior to AIC, BIC, S-AIC, and S-BIC, and their differences become larger as R increases. Now, we turn to Table A2 and Table A3. With the sample size n increasing from 60 to 200 and 500, we can see that the prediction errors decrease for each fixed R. The median and variance of prediction errors also become smaller. AIC and BIC behave increasingly similarly. CV1 and CV2 are still the best among all the methods, and followed by FPCA.

For Case 2, the prediction errors are given in Table A4, Table A5 and Table A6. As shown earlier, CV1 and CV2 perform the best, and followed by FPCA. Likewise, S-AIC or S-BIC is better than AIC or BIC, respectively. For Table A4, with R varying from 1 to 10, the prediction errors are decreasing except FPCA method, which gets the minimum at $R=7$ with a small fluctuation. CV1 and CV2 perform equally well for different R values and sample sizes. The difference between AIC and BIC becomes small with the sample size increasing. The similar phenomenon is observed for S-AIC and S-BIC.

For Case 3, the prediction errors are provided in Table A7, Table A8 and Table A9. For $n=60$ (Table A7), CV1 or CV2 is the best when R is between 1 and 5. However, when R is between 6 and 10, the two model selection methods—AIC and BIC—are the best. The similar conclusions can be found in Table A8 with $n=200$ and Table A9 with $n=500$, although in the latter case, CV1 actually performs the best for all of R values. The error rates of all methods become smaller with R increasing from 1 to 6 and then bigger with R varying from 7 to 10.

For Case 4, the prediction errors are presented in Table A10, Table A11 and Table A12. For $n=60$ in Table A10, CV1, CV2, and BIC are the best, and followed by AIC. In this design, S-AIC or S-BIC is not better than AIC or BIC. For $n=200$ in Table A11, BIC is the best, and followed by AIC. For $n=500$ in Table A12, CV1 always performs the best, and followed by BIC.

In summary, for out-of-sample prediction, our methods CV1 and CV2 perform the best in most of cases and have smaller variances and medians of errors. Furthermore, CV1 and CV2 often perform equally well. This indicates that removing the restriction on the sum of weights may not lead to a better model averaging estimates.

4.2. Simulation II: Divergent Number of Candidate Models

We consider the situations where the number of candidate models tends to ∞ as the sample size increases. We set the sample size n to be 200, 400, and 1000, and the the number of candidate models to be $9n/100$ (So M=18,36, and 90 for the three sample sizes). The data generating process is as before: the predictor variable is $X_i\left(t\right)={\sum }_{j=1}^J{\epsilon }_{i,j}{\rho }_j\left(t\right),$ and the parameter function is $\beta \left(t\right)={\sum }_{j=1}^J{\beta }_j{\rho }_j\left(t\right),$ where ${\rho }_j\left(t\right)$ is a 2-order B-spline basis function, $t\in [0,1],j\ge 1$, and $J=n/10$. For $1\le j\le J$, ${\beta }_j=j^{-1/2}$. We set the term ${\epsilon }_{i,j}$ to be independently generated from $N(0,R^2/j^{1/2})$, where $R=1,3,7$. The response variable $y_i$ is generated from binomial distribution with the logistic link.

The candidate models are nested. The algorithms used in the calculations are the same as that described in Section 4.1. For the simulation results, we report the errors of seven methods considered as Section 4.1. From Table A13, Table A14 and Table A15, our methods—CV1 and CV2—perform the best in most of cases, and followed by FPCA, and SAIC. The difference between AIC and BIC, or SAIC and SBIC is decreasing with increasing R.

4.3. Application: Beijing Second-Hand House Price Data

We apply our method to the Beijing second-hand housing transaction price data, which is captured from the internet collected by the Guoxinda Group Corporation. Most of the data pass through the manual check. This data include the second-hand housing prices and the surrounding environment variables of the 2318 residential areas in Beijing. The second-hand housing prices data are monthly data from January 2015 to December 2017 for each residential area.

Our aim is to predict the increase level in house prices in next year. We are concerned about the relationship between price level to rise and the past housing price curves. We use the median of listing online prices of houses in a residential area as the house price for this residential area. We use the price curve of each residential area from January 2015 to December 2016 as a predictor variable. The response variable is a binary variable, which takes 1 if the rising ratio is high, and 0 otherwise. Here, we define the rising ratio for each district as the ratio of the average monthly price in 2017 to the average monthly price in 2016. The 25%, 50%, and 75% quantile ratios are $1.31,1.37,$ and $1.44$, respectively. We focus on the residential areas whose housing prices are rising rapidly, and so if the ratio is higher than 75% quantile ratios of all residential areas, the response variable of this residential area takes 1 as its value, and 0 otherwise. Of the $n=2318$ residential areas, 568 are rising fast, and 1750 are not.

For simplicity, we standardize all the price data. For each group, we plot the housing price trajectories in Figure 1. Failure to visually detect differences between the groups could result from overcrowding of these plots with too many curves, but when displaying fewer curves (lower panels of Figure 1), the same phenomenon remains. With a few exceptions, no clear visual differences between the two groups can be discerned. On the whole, the trajectories of per year from 2015 to 2016 are not much different. Therefore, the discrimination task at hand is difficult.

Figure 1. Predictor trajectories, corresponding to slightly smoothed monthly price curves. The low rising residential areas are in the upper left (a). The high rising residential areas are in upper right (b). Randomly selected profiles from the panels above are shown in the lower panels (c,d) for 20 districts.

We randomly select 75% of all residential areas as the training set with size 1739, and the rest as the testing data with size 579. We use logistic link and B-spline functions to fit the house price curves. The number of the basis functions is 6, and the order of the B-spline basis functions is 2. Then, we adopt functional principal component analysis (Yao et al. 2005) to built the data-adaptive basis functions to reduce the dimension and deal with the correlations in house price time series.

We compare the out-of-sample prediction errors of the seven methods in Section 4. We repeat every method 20 times. The results are summarized in Table 1 and Table 2. It can be observed from the tables that the error of CV1 or CV2 method is lower 10% on average than those of other methods, and overall, CV1 and CV2 behave similarly. As shown in the simulation above, this indicates the constraint that the sum of weights equals 1 makes sense in practical cases. AIC and BIC perform equally well, as both choose the largest model in most cases. We also find that FPCA is better than AIC or BIC. FPCA always selects the smallest model because the cumulative reliability of the first principal component is ~98%. Further, it is clear that the fitting error and prediction error of FPCA are similar. For the other methods, the fitting errors are always a little smaller than the prediction errors.

Table 1. Error of prediction.

Rounds	AIC	BIC	FPCA	S-AIC	S-BIC	CV1	CV2
1	0.301	0.301	0.275	0.301	0.301	0.221	0.221
2	0.292	0.292	0.247	0.292	0.292	0.178	0.176
3	0.290	0.290	0.242	0.290	0.290	0.187	0.187
4	0.280	0.280	0.233	0.280	0.280	0.176	0.174
5	0.276	0.276	0.233	0.276	0.276	0.147	0.149
6	0.316	0.316	0.233	0.316	0.316	0.188	0.188
7	0.269	0.269	0.244	0.269	0.269	0.164	0.164
8	0.294	0.294	0.225	0.294	0.294	0.174	0.174
9	0.316	0.316	0.235	0.316	0.316	0.187	0.187
10	0.282	0.282	0.242	0.282	0.282	0.174	0.173
11	0.292	0.292	0.240	0.292	0.292	0.162	0.162
12	0.285	0.285	0.261	0.285	0.285	0.188	0.188
13	0.282	0.282	0.219	0.282	0.282	0.150	0.149
14	0.264	0.264	0.280	0.264	0.264	0.188	0.188
15	0.282	0.282	0.247	0.282	0.282	0.187	0.187
16	0.295	0.295	0.269	0.295	0.295	0.185	0.185
17	0.328	0.328	0.252	0.328	0.328	0.204	0.202
18	0.301	0.301	0.245	0.301	0.301	0.187	0.187
19	0.278	0.278	0.209	0.278	0.278	0.150	0.150
20	0.311	0.311	0.249	0.311	0.311	0.183	0.183

Table 2. Error of fitting.

Rounds	AIC	BIC	FPCA	S-AIC	S-BIC	CV1	CV2
1	0.287	0.287	0.235	0.287	0.287	0.166	0.165
2	0.289	0.289	0.244	0.289	0.289	0.181	0.180
3	0.290	0.290	0.246	0.290	0.290	0.174	0.173
4	0.293	0.293	0.249	0.293	0.293	0.182	0.182
5	0.296	0.296	0.249	0.296	0.296	0.190	0.190
6	0.285	0.285	0.249	0.285	0.285	0.175	0.175
7	0.297	0.297	0.246	0.297	0.297	0.184	0.183
8	0.292	0.292	0.252	0.292	0.292	0.179	0.179
9	0.283	0.283	0.248	0.283	0.283	0.174	0.173
10	0.291	0.291	0.246	0.291	0.291	0.182	0.181
11	0.291	0.291	0.247	0.291	0.291	0.184	0.186
12	0.294	0.294	0.240	0.294	0.294	0.175	0.175
13	0.293	0.293	0.254	0.293	0.293	0.190	0.187
14	0.295	0.295	0.233	0.295	0.295	0.175	0.175
15	0.293	0.293	0.244	0.293	0.293	0.176	0.177
16	0.288	0.288	0.237	0.288	0.288	0.179	0.178
17	0.282	0.282	0.243	0.282	0.282	0.173	0.173
18	0.290	0.290	0.245	0.290	0.290	0.178	0.177
19	0.294	0.294	0.257	0.294	0.294	0.186	0.187
20	0.285	0.285	0.244	0.285	0.285	0.179	0.179

5. Concluding Remarks

In this paper, we proposed a model averaging approach under the framework of the generalized functional linear model. We showed that the weight chosen by the leave-one-out cross-validation method is asymptotically optimal in the sense of achieving the lowest possible squared error in a class of model averaging estimators. It can be seen from the theoretical proof that our method is also valid for the non-nested candidate model set. Numerical analysis shows that for generalized functional linear model, cross-validation model averaging is a powerful tool for estimation and prediction. A further work is to develop model averaging inference procedures based on generalized functional linear model. In addition, how to combine other covariates into generalized functional linear model is also an interesting problem.

Appendix B.1. Proof of (A11)

Notice that

$$\begin{array}{cc}\hfill \left|{\tilde{L}}_n\left(w\right)-L_n\left(w\right)\right|& =\left|\sum _{i=1}^n{\tilde{\mu }}_i{\left(w\right)}^2-\sum _{i=1}^n{\widehat{\mu }}_i{\left(w\right)}^2+2\sum _{i=1}^n{\mu }_i\left[{\widehat{\mu }}_i\left(w\right)-{\tilde{\mu }}_i\left(w\right)\right]\right|\hfill \\ \hfill & =\left|\sum _{i=1}^n{\left[{\tilde{\mu }}_i\left(w\right)-{\widehat{\mu }}_i\left(w\right)\right]}^2-2\sum _{i=1}^n{\widehat{\mu }}_i{\left(w\right)}^2+2\sum _{i=1}^n{\tilde{\mu }}_i\left(w\right){\widehat{\mu }}_i\left(w\right)+2\sum _{i=1}^n{\mu }_i\left[{\widehat{\mu }}_i\left(w\right)-{\tilde{\mu }}_i\left(w\right)\right]\right|\hfill \\ \hfill & =\left|\sum _{i=1}^n{\left[{\tilde{\mu }}_i\left(w\right)-{\widehat{\mu }}_i\left(w\right)\right]}^2+2\sum _{i=1}^n\left[{\mu }_i-{\widehat{\mu }}_i\left(w\right)\right]\left[{\widehat{\mu }}_i\left(w\right)-{\tilde{\mu }}_i\left(w\right)\right]\right|\hfill \\ \hfill & =\left|{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2+2⟨\mu -\widehat{\mu }\left(w\right),\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)⟩\right|\hfill \\ \hfill & ⩽{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2+2\sqrt{L_n\left(w\right)}∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥.\hfill \end{array}$$

So,

$$\begin{array}{cc}\hfill \left|\frac{{\tilde{L}}_n\left(w\right)}{L_n\left(w\right)}-1\right|& =\frac{\left|{\tilde{L}}_n\left(w\right)-L_n\left(w\right)\right|}{L_n\left(w\right)}\hfill \\ \hfill & ⩽\frac{{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2+2\sqrt{L_n\left(w\right)}∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}{L_n\left(w\right)}\hfill \\ \hfill & =\frac{{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2}{L_n\left(w\right)}+\frac{2∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}{\sqrt{L_n\left(w\right)}}.\hfill \end{array}$$

Therefore, to prove (A11), it suffices to verify

$$\begin{array}{c}\hfill \underset{w\in H_n}{sup}\frac{\parallel \widehat{\mu }\left(w\right)-\tilde{\mu }{\left(w\right)\parallel }^2}{L_n\left(w\right)}\stackrel{p}{\to }0.\end{array}$$

By Lemma A4, we need only to show

$$\begin{array}{c}\hfill \underset{w\in H_n}{sup}\frac{\parallel \widehat{\mu }\left(w\right)-\tilde{\mu }{\left(w\right)\parallel }^2}{R_n\left(w\right)}\stackrel{p}{\to }0.\end{array}$$

(A13)

Let ${\eta }_{i,p_m}^{*}$ be the point between ${\tilde{\eta }}_{i,p_m}$ and ${\widehat{\eta }}_{i,p_m}$. Then, for any $\delta >0$, we have

$$\begin{array}{c}\mathbf{P}\left\{\underset{w\in H_n}{sup}\frac{{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2}{R_n\left(w\right)}>\delta \right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2>\delta {\xi }_n\right\}\hfill \\ =\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\left[{\tilde{\mu }}_i\left(w\right)-{\widehat{\mu }}_i\left(w\right)\right]}^2>\delta {\xi }_n\right\}\hfill \\ =\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\left[g\left(\sum _{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)-g\left(\sum _{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m}\right)\right]}^2>\delta {\xi }_n\right\}\hfill \\ =\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\left[g^{{}^{\prime }}\left({\eta }_{i,p_m}^{*}\right)\left(\sum _{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}-\sum _{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m}\right)\right]}^2>\delta {\xi }_n\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{1⩽i⩽n,w\in H_n}{max}{\left|g^{{}^{\prime }}\left({\eta }_{i,p_m}^{*}\right)\right|}^2\underset{w\in H_n}{sup}\sum _{i=1}^n{\left[\sum _{m=1}^M{w}_m\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)\right]}^2>\delta {\xi }_n\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{1⩽i⩽n,w\in H_n}{max}{\left|g^{{}^{\prime }}\left({\eta }_{i,p_m}^{*}\right)\right|}^2\underset{w\in H_n}{sup}\sum _{i=1}^n\sum _{m=1}^M{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2>\delta {\xi }_n\right\}\hfill \\ =\mathbf{P}\left\{\underset{1⩽i⩽n,w\in H_n}{max}{\left|g^{{}^{\prime }}\left({\eta }_{i,p_m}^{*}\right)\right|}^2\sum _{i=1}^n\sum _{m=1}^M{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2>\delta {\xi }_n\right\},\hfill \end{array}$$

which, together with the assumption that $g(·)$ is a twice continuously differentiable function with bounded derivatives implying ${max}_{1⩽i⩽n,w\in H_n}{\left|g^{{}^{\prime }}\left({\eta }_{i,p_m}^{*}\right)\right|}^2⩽c^2<\infty$, leads to

$$\mathbf{P}\left\{\underset{w\in H_n}{sup}\frac{{∥\widehat{\mu }\left(w\right)-\tilde{\mu }\left(w\right)∥}^2}{R_n\left(w\right)}>\delta \right\}⩽\mathbf{P}\left\{c^2\sum _{i=1}^n\sum _{m=1}^M{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2/{\xi }_n>\delta \right\}.$$

Thus, to prove (A13), it suffices to show

$$\begin{array}{c}\hfill \sum _{i=1}^n\sum _{m=1}^M{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2/{\xi }_n=o_p\left(1\right).\end{array}$$

(A14)

By Condition 5, for fixed M, we obtain

$$\begin{array}{cc}\hfill \sum _{i=1}^n\sum _{m=1}^M{\left({\tilde{\eta }}_{i,p_m}-{\widehat{\eta }}_{i,p_m}\right)}^2& =O_p\left(\sum _{m=1}^M{p}_m^4\right),\hfill \end{array}$$

which, together with Condition 4, leads to (A14), and thus (A13) holds.

Appendix B.2. Proof of (A12)

As

$$\left|⟨e,\mu -\tilde{\mu }\left(w\right)⟩\right|=\left|\sum _{i=1}^n{e}_i\left[g\left({\eta }_i\right)-g\left(\sum _{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)\right]\right|,$$

it is sufficient to show

$$\underset{w\in H_n}{sup}\left|\sum _{i=1}^n{e}_i\left[g\left({\eta }_i\right)-g\left(\sum _{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)\right]\right|/R_n\left(w\right)\stackrel{p}{\to }0.$$

It is readily seen that

$$\begin{array}{cc}\hfill & \underset{w\in H_n}{sup}\frac{\left|{\sum }_{i=1}^n{e}_i\left[g\left({\eta }_i\right)-g\left({\sum }_{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)\right]\right|}{R_n\left(w\right)}\hfill \\ \hfill & ⩽\underset{w\in H_n}{sup}\frac{\left|{\sum }_{i=1}^n{e}_i\left[g\left({\eta }_i\right)-g\left({\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)\right]\right|}{R_n\left(w\right)}\hfill \\ \hfill & +\underset{w\in H_n}{sup}\frac{\left|{\sum }_{i=1}^n{e}_i\left[g\left({\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)-g\left({\sum }_{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m}\right)\right]\right|}{R_n\left(w\right)}\hfill \\ \hfill & +\underset{w\in H_n}{sup}\frac{\left|{\sum }_{i=1}^n{e}_i\left[g\left({\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}\right)-g\left({\sum }_{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)\right]\right|}{R_n\left(w\right)}\hfill \\ \hfill & \triangleq \underset{w\in H_n}{sup}{A}_n^{\left(1\right)}\left(w\right)+\underset{w\in H_n}{sup}{A}_n^{\left(2\right)}\left(w\right)+\underset{w\in H_n}{sup}{A}_n^{\left(3\right)}\left(w\right).\hfill \end{array}$$

Thus, we need only to prove

$$\underset{w\in H_n}{sup}{A}_n^{\left(1\right)}\left(w\right)\stackrel{p}{\to }0,$$

(A15)

$$\underset{w\in H_n}{sup}{A}_n^{\left(2\right)}\left(w\right)\stackrel{p}{\to }0,$$

(A16)

and

$$\underset{w\in H_n}{sup}{A}_n^{\left(3\right)}\left(w\right)\stackrel{p}{\to }0.$$

(A17)

The proof of (A15) is similar to that of Wu (1981). We denote a metric

$$\rho (w,w^{\prime })=∥w-w^{\prime }∥,$$

which is on $H_n$. Let $\left(H_n,\rho \right)$ be a compact metric space. Then $\mathbf{C}\left(H_n\right)$ is the Banach space of real-valued continuous functions on $H_n$ with the supremum norm

$${∥\Delta ∥}_{\infty }=su{p}_{w\in H_n}\left|\Delta \left(w\right)\right|.$$

Let $\mathbf{N}\left(H_n,\rho ,\epsilon \right)$ denote the minimal number of $\rho$-balls of radius less than or equal to $\epsilon$ which cover $H_n$, and set

$$\mathbf{H}\left(H_n,\rho ,\epsilon \right)=log\mathbf{N}\left(H_n,\rho ,\epsilon \right).$$

We let

$$Lip\left(\rho \right)=\left\{\Delta \in \mathbf{C}\left(H_n\right):\Lambda \left(\Delta \right)=\underset{w\ne w^{\prime }\in H_n}{sup}\frac{\left|\Delta \left(w\right)-\Delta \left(w^{\prime }\right)\right|}{\rho (w,w^{\prime })}<\infty \right\},$$

and for $\Delta \in Lip\left(\rho \right)$, we define

$${∥\Delta ∥}_{\rho }=\Lambda \left(\Delta \right)+\left|\Delta \left(w^{*}\right)\right|,$$

where $w^{*}$ is some fixed point in $H_n$.

Recalling that ${\Delta }_i\left(w\right)=g\left({\eta }_i\right)-g\left({\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)$, we have

$$\begin{array}{c}\Lambda \left(\frac{{\Delta }_i}{p+1}\right)\hfill \\ =\underset{w\ne w^{\prime }\in H_n}{sup}\frac{\left|{\Delta }_i\left(w\right)-{\Delta }_i\left(w^{\prime }\right)\right|}{\left(p+1\right)\rho (w,w^{\prime })}\hfill \\ =\underset{w\ne w^{\prime }\in H_n}{sup}\left|g^{\prime }\left({\gamma }_{0,i}\right)\right|\times \frac{\left|{\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\sum }_{m=1}^M{w}_m^{\prime }{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right|}{\left(p+1\right)\rho (w,w^{\prime })}\hfill \\ ⩽c\times \underset{w\ne w^{\prime }\in H_n}{sup}\frac{\left|{\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\sum }_{m=1}^M{w}_m^{\prime }{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right|}{\left(p+1\right)\rho (w,w^{\prime })}\hfill \\ ⩽c\sqrt{{\sum }_{m=1}^M\frac{{\left({\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2}{{\left(p+1\right)}^2}},\hfill \end{array}$$

where ${\gamma }_{0,i}$ is a point between ${\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}$ and ${\sum }_{m=1}^M{w}_m^{\prime }{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}$. From the assumption ${∥{\beta }_{★}^{\left(m\right)}∥}^2/\left(p_m+1\right)⩽C_b<\infty$, and Condition 3, we obtain

$$\underset{i}{sup}\Lambda \left({\Delta }_i\right)⩽C_g<\infty .$$

(A18)

As for $\left|\frac{{\Delta }_i\left(w^{*}\right)}{p+1}\right|$, using Lagrange theorem, we have

$$\begin{array}{cc}\hfill \left|\frac{{\Delta }_i\left(w^{*}\right)}{p+1}\right|& =\frac{1}{p+1}\left|g^{\prime }\left({\zeta }_i\right)\left({\eta }_i-\sum _{m=1}^M{w}_m^{*}{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)\right|\hfill \\ \hfill & ⩽c\sqrt{{\sum }_{m=1}^M\frac{{\left({\eta }_i-{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)}^2}{{\left(p+1\right)}^2}},\hfill \end{array}$$

where ${\zeta }_i$ is a point between ${\eta }_i$ and ${\sum }_{m=1}^M{w}_m^{*}{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}$. Again, by Condition 3, ${∥{\beta }_{★}^{\left(m\right)}∥}^2/\left(p_m+1\right)⩽C_b<\infty$, and the assumption ${sup}_i\left|{\eta }_i\right|\le C_{\eta }<\infty$, we obtain

$$\underset{i}{sup}\left|{\Delta }_i\left(w^{*}\right)\right|⩽\tilde{C}<\infty .$$

(A19)

For (A15), we have

$$\begin{array}{cc}\hfill & \mathbf{P}\left\{\underset{w\in H_n}{sup}{A}_n^{\left(1\right)}\left(w\right)>\delta \right\}\hfill \\ \hfill & ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}\left|\sum _{i=1}^n{e}_i\left[g\left({\eta }_i\right)-g\left(\sum _{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)\right]\right|>\delta {\xi }_n\right\}\hfill \\ \hfill & =\mathbf{P}\left\{\underset{w\in H_n}{sup}\left|\sum _{i=1}^n{e}_i\frac{{\Delta }_i\left(w\right)}{p+1}\right|>\delta \frac{{\xi }_n}{p+1}\right\}\hfill \\ \hfill & ⩽\frac{{\left(p+1\right)}^2\mathbf{E}{\left[{sup}_{w\in H_n}\left|{\sum }_{i=1}^n{e}_i\frac{{\Delta }_i\left(w\right)}{p+1}\right|\right]}^2}{{\delta }^2{\xi }_n^2}\hfill \\ \hfill & =\frac{{\left(p+1\right)}^2\mathbf{E}{∥{\sum }_{i=1}^n{e}_i\frac{{\Delta }_i}{p+1}∥}_{\infty }^2}{{\delta }^2{\xi }_n^2},\hfill \end{array}$$

where $\delta >0$ is an arbitrary constant. Since $H_n$ is a compact subset of ${\mathbb{R}}^M$, and $\rho (w,w^{\prime })$ is the Euclidean metric in ${\mathbb{R}}^M$, (A1) is satisfied. Therefore, by Lemma A1, we see that there is a constant $A<\infty$ such that for all n,

$$\begin{array}{cc}\hfill \mathbf{E}{∥\sum _{i=1}^n{e}_i\frac{{\Delta }_i}{p+1}∥}_{\infty }^2& ⩽A\sum _{i=1}^n\mathbf{E}{∥e_i\frac{{\Delta }_i}{p+1}∥}_{\rho }^2\hfill \\ \hfill & ⩽A\underset{j}{sup}\mathbf{E}{e}_j^2\sum _{i=1}^n{\left[\Lambda \left(\frac{{\Delta }_i}{p+1}\right)+\left|\frac{{\Delta }_i\left(w^{*}\right)}{p+1}\right|\right]}^2\hfill \\ \hfill & ⩽2A\underset{j}{sup}\mathbf{E}{e}_j^2\sum _{i=1}^n\left({\Lambda }^2\left(\frac{{\Delta }_i}{p+1}\right)+{\left|\frac{{\Delta }_i\left(w^{*}\right)}{p+1}\right|}^2\right)\hfill \\ \hfill & =O\left(n\right),\hfill \end{array}$$

where the last equality is because of (A18), (A19) and ${sup}_j\mathbf{E}{e}_j^2<\infty$. Therefore,

$$\begin{array}{c}\hfill \mathbf{P}\left\{\underset{w\in H_n}{sup}{A}_n^{\left(1\right)}\left(w\right)>\delta \right\}\end{array}=O\left(\frac{{\left(p+1\right)}^{2}n}{{\xi }_n^2}\right)\to 0,$$

and (A15) holds.

Denote ${\tilde{\Delta }}_i=g\left({\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)-g\left({\sum }_{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m}\right)$. For (A16), we have

$$\begin{array}{c}\mathbf{P}\left\{\underset{w\in H_n}{sup}{A}_n^{\left(2\right)}\left(w\right)>\delta \right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}{\left|\sum _{i=1}^n{e}_i\left[g\left(\sum _{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}\right)-g\left(\sum _{m=1}^M{w}_m{\widehat{\eta }}_{i,p_m}\right)\right]\right|}^2>{\delta }^2{\xi }_n^2\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}\left|\sum _{i=1}^n{e}_i^2\sum _{i=1}^n{\tilde{\Delta }}_i^2\right|>{\delta }^2{\xi }_n^2\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\tilde{\Delta }}_i^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ =\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\left|g^{\prime }\left(\sum _{m=1}^M{w}_m{\tilde{\eta }}_{i*}^m\right)\sum _{m=1}^M\left[w_m\left({\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\widehat{\eta }}_{i,p_m}\right)\right]\right|}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}\hfill \\ +\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}{c}^2\sum _{i=1}^n{\left|\sum _{m=1}^M\left[w_m\left({\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\widehat{\eta }}_{i,p_m}\right)\right]\right|}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ ⩽\mathbf{P}\left\{c^2\sum _{i=1}^n\sum _{m=1}^M{\left({\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\widehat{\eta }}_{i,p_m}\right)}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ ⩽\mathbf{P}\left\{c^2\sum _{i=1}^n\sum _{m=1}^M{\left({\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\widehat{\eta }}_{i,p_m}\right)}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\frac{p^2{\sum }_{i=1}^n\mathbf{E}{e}_i^2}{{\xi }_n\sqrt{n}}.\hfill \end{array}$$

From Lemma A3 and Condition 3, we see that

$$\sum _{i=1}^n\sum _{m=1}^M{\left({\epsilon }_{(i,p_m)}^T{\beta }_{★}^{\left(m\right)}-{\widehat{\eta }}_{i,p_m}\right)}^2=\sum _{i=1}^n\sum _{m=1}^M{\left[{\epsilon }_{(i,p_m)}^T\left({\beta }_{★}^{\left(m\right)}-{\widehat{\beta }}^{\left(m\right)}\right)\right]}^2=O_p\left(\sum _{m=1}^M{p}_m^4\right).$$

Therefore, ${lim}_{n\to +\infty }\mathbf{P}\left\{{sup}_{w\in H_n}{A}_n^{\left(2\right)}\left(w\right)>\delta \right\}=0$, that is, (A16) is valid.

Write ${\overline{\Delta }}_i=g\left({\sum }_{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}\right)-g\left({\sum }_{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)$. For (A17), we have

$$\begin{array}{c}\mathbf{P}\left\{\underset{w\in H_n}{sup}{A}_n^{\left(3\right)}\left(w\right)>\delta \right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}{\left|\sum _{i=1}^n{e}_i\left[g\left(\sum _{m=1}^M{w}_m{\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}\right)-g\left(\sum _{m=1}^M{w}_m{\tilde{\eta }}_{i,p_m}\right)\right]\right|}^2>{\delta }^2{\xi }_n^2\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}\left|\sum _{i=1}^n{e}_i^2\sum _{i=1}^n{\overline{\Delta }}_i^2\right|>{\delta }^2{\xi }_n^2\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\overline{\Delta }}_i^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ =\mathbf{P}\left\{\underset{w\in H_n}{sup}\sum _{i=1}^n{\left|g^{\prime }\left(\sum _{m=1}^M{w}_m{\eta }_{i,p_m}^{*}\right)\sum _{m=1}^M\left[w_m\left({\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}-{\tilde{\eta }}_{i,p_m}\right)\right]\right|}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}\hfill \\ +\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ ⩽\mathbf{P}\left\{\underset{w\in H_n}{sup}{c}^2\sum _{i=1}^n{\left|\sum _{m=1}^M\left[w_m\left({\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}-{\tilde{\eta }}_{i,p_m}\right)\right]\right|}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ ⩽\mathbf{P}\left\{c^2\sum _{i=1}^n\sum _{m=1}^M{\left({\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}-{\tilde{\eta }}_{i,p_m}\right)}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\mathbf{P}\left\{\sum _{i=1}^n{e}_i^2>{\xi }_n\sqrt{n}/p^2\right\}\hfill \\ ⩽\mathbf{P}\left\{c^2\sum _{i=1}^n\sum _{m=1}^M{\left({\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}-{\tilde{\eta }}_{i,p_m}\right)}^2>{\delta }^2{\xi }_n{p}^2/\sqrt{n}\right\}+\frac{p^2{\sum }_{i=1}^n\mathbf{E}{e}_i^2}{{\xi }_n\sqrt{n}}.\hfill \end{array}$$

From Condition 5, we see that

$$\sum _{i=1}^n\sum _{m=1}^M{\left({\epsilon }_{(i,p_m)}^T{\widehat{\beta }}^{\left(m\right)}-{\tilde{\eta }}_{i,p_m}\right)}^2=O_p\left(\sum _{m=1}^M{p}_m^4\right).$$

Therefore, ${lim}_{n\to +\infty }\mathbf{P}\left\{{sup}_{w\in H_n}{A}_n^{\left(3\right)}\left(w\right)>\delta \right\}=0$, that is, (A17) is valid.