Multivariate Functional Kernel Machine Regression and Sparse Functional Feature Selection

Abstract

Motivated by mobile devices that record data at a high frequency, we propose a new methodological framework for analyzing a semi-parametric regression model that allow us to study a nonlinear relationship between a scalar response and multiple functional predictors in the presence of scalar covariates. Utilizing functional principal component analysis (FPCA) and the least-squares kernel machine method (LSKM), we are able to substantially extend the framework of semi-parametric regression models of scalar responses on scalar predictors by allowing multiple functional predictors to enter the nonlinear model. Regularization is established for feature selection in the setting of reproducing kernel Hilbert spaces. Our method performs simultaneously model fitting and variable selection on functional features. For the implementation, we propose an effective algorithm to solve related optimization problems in that iterations take place between both linear mixed-effects models and a variable selection method (e.g., sparse group lasso). We show algorithmic convergence results and theoretical guarantees for the proposed methodology. We illustrate its performance through simulation experiments and an analysis of accelerometer data.

1. Introduction

Data captured by mobile devices have lately received much attention in the data science community. Such data are typically recorded at a high frequency, giving rise to an ample volume of information at a very fine scale, and thus present many methodological challenges in statistical modeling and data analyses. In this paper, we plan to utilize the strength of the classical kernel machine method that enjoys fast computing speed via the linear mixed-effects model to deal with such high-frequency data using a functional data analysis approach. The motivation for our proposed framework come from data collected from a tri-axis accelerometer. Accelerometers, worn on the hip or wrist as a way of monitoring physical activity, are becoming more and more common [1,2,3,4]. There are several different accelerometers available such as ActiGraph GT3X+ (ActiGraph, Pensacola, FL, USA) and Actical (Phillips Respironics, Bend, OR). Raw accelerometer data are often collected in high-resolution signals with a sampling frequency ranging from 30–100 Hz. The commercial software on these devices provides activity counts (ACs) [2,4], which are calculated from the raw accelerometer data using proprietary algorithms. As an example from our motivating dataset, Figure 1 displays a three-dimensional time series of ACs per minute, each on one axis, from one subject wearing the GT3X+ over a period of 7 days (d).

Oftentimes, different types of summaries of the tri-axis ACs are suggested in the literature as opposed to the utility of all three raw functionals [5,6,7,8]. These summary-data-based approaches may be regarded as a quick and dirty dimension reduction strategy that comes up with summarized data with computationally manageable volumes, which would be then analyzed by existing methods and software. One concern with the use of summarized data would be the loss of potential fine features that can only be captured in data of high resolution. Recently, some researchers have attempted to use the entire functional AC curve through functional data analysis techniques [6,9,10]. Further details on current methods being used to retrieve and interpret accelerometer data can be found in [11]. Our contribution in this paper pertains to a new framework in that tri-axis accelerometer data are used as three-dimensional correlated functional predictors in an association analysis with a potential health outcome such as the Body Mass Index (BMI). The relationship between physical activities and childhood obesity has long been a central interest of public health sciences, and our new scalar-on-functional regression model can provide some new insights into this important scientific problem.

We begin with a brief review of existing functional data models, the least-squares kernel machine model, and different variable selection techniques, which prelude the framework for this paper.

1.1. Functional Regression

There has been much attention in recent years given to functional data analysis (FDA) where either covariates, or response, or both are functional as opposed to scalar in nature [12,13,14,15,16,17]. In this paper, we focused on the methodology that allows us to relate multiple functional covariates to a scalar outcome in a nonlinear way in the presence of other scalar covariates. To proceed, let us introduce some notation. Let $L^2\left(\mathcal{T}\right)$ be the class of square-integrable functions on a compact set $\mathcal{T}$. This is a separable Hilbert space with inner product $<f$,$g>:={\int }_{\mathcal{T}}fg$ for $f,$$g\in L^2\left(\mathcal{T}\right)$. Consider a probability space $(\mathrm{\Omega },\mathcal{F},P)$, where Z denotes a functional random variable that maps into $L^2\left(\mathcal{T}\right)$, namely $Z:\mathrm{\Omega }\mapsto L^2\left(\mathcal{T}\right)$. Define $L^2\left(\mathrm{\Omega }\right):=\{Z:{\left({\int }_{\mathrm{\Omega }}{∥Z∥}^{2}dP\right)}^{\frac{1}{2}}<\infty \}$, where P is a certain probability measure, ${∥Z∥}^2=$ $<Z$, $Z>$, and assume $Z\in L^2\left(\mathrm{\Omega }\right)$ in the rest of this paper. For convenience, we also assume that Z is mean centered, namely $E\left(Z\right)=0$.

The class of functional linear models (FLM) (e.g., [13,14,15]) is proposed to relate a functional covariate Z with a mean-centered scalar outcome y, which is also known as scalar-on-functional regression: $y=<b,Z>+ϵ$, where the error term $ϵ$ is a mean zero random variable uncorrelated with Z. An optimal solution of the unknown functional parameter $b\in L^2\left(\mathcal{T}\right)$ is typically obtained by minimizing the mean-squared error: ${inf}_{b\in L^2\left(\mathcal{T}\right)}E{(y-<b,Z>)}^2$. Moreover, the mean model for the mean-centered scalar y takes the form $E\left(y\right|Z)={\int }_{\mathcal{T}}Z\left(t\right)b\left(t\right)dt$.

As suggested in the literature, we may obtain an optimal estimator of b by expanding functional predictor Z under certain basis functions. In this paper, we focus on the utility of functional principal component analysis (FPCA) to perform the decomposition of the functional Z. By the Karhunen–Loève expansion (e.g., [18,19,20]), we may write $Z\left(t\right)={\sum }_{k=1}^{\infty }\sqrt{{\varsigma }_k}{\xi }_k{\varphi }_k\left(t\right)$, where ${\varsigma }_k>0$ are the eigenvalues, and the loadings are given by ${\xi }_k:=\frac{1}{\sqrt{{\varsigma }_k}}<Z,{\varphi }_k>$. These coefficients satisfy (i) mean zero, $E\left({\xi }_k\right)=0$; (ii) variance one, $E\left({\xi }_k^2\right)=1$; (iii) uncorrelated, $E\left({\xi }_k{\xi }_j\right)=0$ for $k\ne j$. Then, the mean model may be rewritten as follows,

$$E\left(y\right|Z)=\sum _{k=1}^{\infty }{\mathit{\beta }}_k{\xi }_k,$$

(1)

where coefficients ${\beta }_k=<b,\sqrt{{\varsigma }_k}{\varphi }_k>$, $k=1,···$, which are unknown due to the unknown b. Equation (1) presents a linear projection of scalar outcome y on the space spanned by the standardized principal components (PCs) ${\xi }_k$’s of functional predictor Z. On these lines of research, Müller and Yao (2008) proposed a class of functional additive models (FAMs) that extends Equation (1) by allowing a nonparametric form of the projection:

$$E\left(y\right|Z)=\sum _{k=1}^{\infty }{f}_k\left({\xi }_k\right),$$

(2)

where $f_k$ is a fully unspecified nonlinear smooth function to be estimated. It is obvious that Müller and Yao’s extension given in (2) takes an additive model on individual coefficient (or feature) components ${\xi }_k$’s. Regularization is often needed for both (1) and (2) in order to deal with these infinite-dimensional unknowns. One of the challenges concerning regularization for (2) lies in the technical treatment in the functional space. Müller and Yao (2008) [21] proposed truncation (or a hard threshold) of the eigenspace to retain only the leading components that explain the majority of the total variation in Z. Zhu, Yao, and Zhang (2014) [15] proposed another regularization for the functions $f_k$ using the powerful COSSO method [22]. One advantage for this kind of regularization method is that sums of higher-order functional principal components are allowed to be potentially included in the fit model, if they make stronger contributions to the functional relationship than the leading functional principal components. This regularization method [15] begins with an additive model $E\left(y\right|Z)={\sum }_{k=1}^s{f}_k\left({\xi }_k\right),$ where s represents some initial degrees of truncation to specify the total number of additive components to be considered. Then, COSSO helps simultaneously regularize and select important functional components among the s functions $f_k$. Although the above discussion is based on a single functional predictor Z in mind, it is appealing to extend such a framework with multiple functional predictors for a broad range of problems.

When multiple functional predictors, say $Z^1,\dots ,Z^p$, are considered, it is not clear if the above additive model specification remains suitable to handle the complexity, especially a non-additive relationship (e.g., interactions) may be of interest to understand the association between a scalar outcome and multiple functional predictors. In effect, from both the perspectives of theoretical advances and application needs, relaxing the additive relationship is an important task in functional data analysis. Alternatively, there are some methods (e.g., [16,17]) in the literature that do not use the strategy of decomposing Z into its functional components. In this paper, we adopt the framework of kernel machine regression models to extend the methodologies with non-additive relationships between multiple functional predictors and the scalar outcome.

1.2. Least-Squares Kernel Machine

Liu, Lin, and Ghosh (2007) [23] proposed a semi-parametric regression model $y_i={\mathbf{x}}_i^{\top }\mathit{\beta }+h\left({\mathbf{z}}_i\right)+{ϵ}_i$ for subject $i=1,\dots ,n$, where they used the least-squares kernel machine (LSKM) to analyze multidimensional genetic pathways denoted by a vector ${\mathbf{z}}_i$. The key feature of this model is the nonlinear relationship between the outcome $y_i$ and a vector of gene expressions ${\mathbf{z}}_i$, which is characterized by a nonparametric smooth function h. Under the theory of smoothing splines, function h is assumed to lie in a reproducing kernel Hilbert space (RKHS), ${\mathcal{H}}_{\mathcal{K}}$, generated by a positive-definite kernel function $\mathcal{K}(·,·)$. For the ease of exposition, we suppress the bandwidth for the kernel $\mathcal{K}$ in the following discussion. Then, both parameter $\mathit{\beta }$ and function h are estimated by maximizing the scaled penalized likelihood function:

$$J(h,\mathit{\beta })=-\frac{1}{2}\sum _{i=1}^n{\{y_i-{\mathbf{x}}_i^{\top }\mathit{\beta }-h\left({\mathbf{z}}_i\right)\}}^2-\frac{1}{2}{\lambda }_1{∥h∥}_{{\mathcal{H}}_{\mathcal{K}}}^2,$$

(3)

where ${\lambda }_1>0$ is the tuning parameter and ${∥·∥}_{{\mathcal{H}}_{\mathcal{K}}}$ is the norm of the RKHS. For a function $h\in L^2\left({\mathcal{H}}_{\mathcal{K}}\right)$, we have $h(·)={\sum }_{i=1}^n{\mathit{\alpha }}_i\mathcal{K}(·,{\mathbf{z}}_i)$. Then, ${∥h∥}_{{\mathcal{H}}_{\mathcal{K}}}^2={\mathit{\alpha }}^{\top }\mathbf{K}\mathit{\alpha }$, where $\mathbf{K}$ is an $n\times n$ matrix whose $(i,j)$ entry is $\mathcal{K}({\mathbf{z}}_i,{\mathbf{z}}_j)$ and $\mathit{\alpha }={({\alpha }_1,\dots ,{\alpha }_n)}^{\top }$.

It is known in the literature (e.g., [23,24]) that maximizing $J(h,\mathit{\beta })$ in (3) turns out to be equivalent to solving the normal equations from the following linear mixed-effects model (LMM): $\mathbf{Y}=\mathbf{X}\mathit{\beta }+\mathbf{h}+\mathit{ϵ},$ where $\mathbf{h}$ is an $n\times 1$ vector of random effects with distribution $N(\mathbf{0},\tau \mathbf{K})$ and an n-dimensional vector error term $\mathit{ϵ}\sim N(\mathbf{0},{\sigma }^2\mathbf{I})$, with $\tau ={\lambda }_1^{-1}{\sigma }^2>0$. One remarkable advantage of solving (3) through the existing numerical procedure of the LMM is most advocated in the literature [25], where we can determine the smoothing parameter ${\lambda }_1$ as part of the estimation of the variance components of the LMM. Therefore, instead of using cross-validation or other information-based tuning methods on ${\lambda }_1$, we can solve simultaneously for all the model parameters in (3), as shown in [23]. Utilizing this numerical strength of the kernel machine regression model, we propose a semi-parametric regression model by incorporating functional principal components of functional predictors (i.e., the ${\mathbf{z}}_i$) to evaluate a nonlinear relationship of a scalar outcome with multiple functional covariates in a non-additive way. Assuming that function h belongs to an RKHS, we can use existing software packages for solving LMMs to obtain estimates of all model parameters and the smoothing parameter.

1.3. Feature Selection

To deal with high-dimensional functional principal components from functional covariates, we invoked the sparse regularization approach in the kernel machine regression model. Note that for both mean models (1) and (2), one needs to truncate the series from the Karhunen–Loève expansion. Regularization helps reduce from an infinite number of terms to a sum of finite terms. To introduce some notations, here we present a brief review on the group lasso (GL) [26], sparse group lasso (SGL) [27], and non-negative garrote [28]. See also the series of work originated by COSSO [22]. Yuan and Lin (2007) [26] proposed the group lasso, which solves the convex optimization problem: ${min}_{\mathit{\beta }\in {\mathcal{R}}^p}{∥\mathbf{Y}-{\sum }_{\ell =1}^L{\mathbf{X}}^{\ell }{\mathit{\beta }}^{\ell }∥}_2^2+\lambda {\sum }_{\ell =1}^L{∥{\mathit{\beta }}^{\ell }∥}_2$, where L is the total number of groups of covariates and ${\mathbf{X}}^{\ell }$ refers to a subset of covariates associated with group ℓ. Friedman, Hastie, and Tibshirani [27] extended the group lasso to allow within-group sparsity, namely SGL, given as ${min}_{\mathit{\beta }\in R^p}{∥\mathbf{Y}-{\sum }_{\ell =1}^L{\mathbf{X}}^{\ell }{\mathit{\beta }}^{\ell }∥}_2^2+\lambda (1-\delta ){\sum }_{\ell =1}^L{∥{\mathit{\beta }}^{\ell }∥}_2+\lambda \delta {∥\mathit{\beta }∥}_1$, where $\delta \in [0,1]$. The additional ${\ell }_1$-norm penalty term on $\mathit{\beta }$ encourages individual sparsity, while the first penalty targets sparsity at the group level. It is easy to see that group lasso is a special case of the SGL when $\delta =0$.

The non-negative garrote proposed by Breiman (1995) [28] is another useful means of variable selection. It invokes a scaled version of least-squares estimation given by: $\mathrm{a}\mathrm{r}\mathrm{g}{min}_{\mathbf{d}}\frac{1}{2}{∥\mathbf{Y}-\tilde{\mathbf{X}}\mathbf{d}∥}_2^2+\lambda {\sum }_{j=1}^p{d}_j,$ subject to $d_j\ge 0,j=1,\dots ,p$. Here, $\tilde{\mathbf{X}}=({\tilde{\mathbf{x}}}_1,\dots ,{\tilde{\mathbf{x}}}_p)$ is an $n\times p$ matrix with columns ${\tilde{\mathbf{x}}}_j={\mathbf{x}}_j{\widehat{\beta }}_j^{OLS}$, with ${\widehat{\beta }}_j^{OLS}$ being the least-squares estimates from $\mathrm{a}\mathrm{r}\mathrm{g}{min}_{\mathit{\beta }}\frac{1}{2}{∥\mathbf{Y}-\mathbf{X}\mathit{\beta }∥}_2^2$ with no constraints. Obviously, estimate ${\widehat{d}}_j=0$ implies that covariate $x_j$ would be excluded from the fit model. Breiman’s formulation that turns a variable selection problem into a parameter estimation problem will be applied for the development of feature selection on functional principal components in this paper.

This paper is organized as follows. Section 2 introduces our proposed high-dimensional kernel machine regression. Section 3 outlines a simple step-by-step algorithm that is used to implement the sparse estimation method. Section 4 concerns asymptotic properties for our proposed sparse kernel machine regression. Section 5 provides simulation results to examine the performance of our method, with comparisons with existing methods. Section 6 illustrates the proposed method by an association analysis of the relationship between the BMI and functional accelerometer data. Section 7 includes our conclusions. The Appendix A contains some key technical details, including the proofs of the theoretical results, while Appendix B presents a discussion on the model identifiability issue.

2. Model and Estimation

Consider a regression analysis of a scalar outcome y on p functional covariates, $Z^{\ell }$, $\ell =1,\dots ,p$. Let ${\mathbf{z}}_i^{\ell }={({\xi }_1^{\ell },\dots ,{\xi }_{s_{\ell }}^{\ell })}_i^{\top }$ be the $s_{\ell }$-element vector of functional principal component (FPC) features from the $i\mathrm{t}\mathrm{h}$ observation of the ℓth functional covariate $Z^{\ell }$, and let ${\overrightarrow{\mathbf{z}}}_i={[{\left({\mathbf{z}}_i^1\right)}^{\top },\dots ,{\left({\mathbf{z}}_i^p\right)}^{\top }]}^{\top }$ be the grand vector of all FPC features from all p functional covariates for subject i, $i=1,\dots ,n$. Clearly, the set of FPC features from each functional covariate forms a group, and in total, there are p groups with $s={\sum }_{\ell =1}^p{s}_{\ell }$ many FPC features and ${\overrightarrow{\mathbf{z}}}_i\in {\mathcal{R}}^s$. The high dimensionality of FPC features presents the key methodological challenge in the analysis. We consider the following functional kernel machine regression (FKMR) model:

$$y_i={\mathbf{x}}_i^{\top }\mathit{\beta }+h\left({\overrightarrow{\mathbf{z}}}_i\right)+{ϵ}_i,i=1,···,n,$$

(4)

where $\mathit{\beta }\in {\mathcal{R}}^q$ is a set of parameters for the effects of q scalar covariates $\mathbf{x}={(x_1,\dots ,x_q)}^{\top }$, $h\in {\mathcal{H}}_{\mathcal{K}}$ is an s-variate smooth nonparametric function with ${\mathcal{H}}_{\mathcal{K}}$ being the functional space generated by a Mercer kernel$\mathcal{K}$ and error terms ${ϵ}_i\stackrel{iid}{\sim }N(0,{\sigma }^2)$. The FKMR model (4) allows for not only nonlinear, but also non-additive relationships with multiple functional covariates $Z^{\ell }$ via their FPC features, $\ell =1,\dots ,p$, and a scalar outcome, y. The statistical task is to estimate and select important functional covariates that are related to the outcome of interest through regularizing the FPC features within each functional covariate. To proceed, following Beiman’s [28] non-negative garrote method, we here introduce a new s-dimensional scaling vector $\mathit{\gamma }\in {\mathcal{R}}^s$, $\mathit{\gamma }={({\gamma }_1,\dots ,{\gamma }_{s_1},\dots ,{\gamma }_s)}^{\top }$, by which we can set $\mathit{\gamma }\circ {\overrightarrow{\mathbf{z}}}_i={({\gamma }_1{\xi }_1^1,\dots ,{\gamma }_{s_1}{\xi }_{s_1}^1,\dots ,{\gamma }_s{\xi }_{s_p}^p)}_i^{\top }$ a new vector of weighted FPC features by $\mathit{\gamma }$ via the Hadamard product (i.e., elementwise product). Note that $\mathit{\gamma }$ is grouped and denoted by $\mathit{\gamma }={(\left({\mathit{\gamma }}^1\right){}^{\top },\dots ,{\left({\mathit{\gamma }}^p\right)}^{\top })}^{\top }$ where ${\mathit{\gamma }}^{\ell }$ is an $s_{\ell }$-element vector of FPC features ${\mathbf{z}}^{\ell }$ of the $\ell \mathrm{t}\mathrm{h}$ functional covariate $Z^{\ell }$. When the element, say ${\mathit{\gamma }}_j$, is equal to zero, the corresponding FPC feature ${\xi }_j$ will not be selected in the set of important FPCs, and moreover, functional covariate $Z^{\ell }$ is excluded from the FKMR model when the entire vector $\left({\mathit{\gamma }}^{\ell }\right){}^{\top }=0$.

We estimate the unknowns in the FKMR model (4), as well as the scaling parameters $\mathit{\gamma }$ by minimizing the penalized objective function $J_1(h,\mathit{\beta },\mathit{\gamma })$, whose expression is given on the right-hand side of the following Equation (5):

$$\underset{h,\mathit{\beta },\mathit{\gamma }}{min}{J}_1(h,\mathit{\beta },\mathit{\gamma })=\underset{h,\mathit{\beta },\mathit{\gamma }}{min}\frac{1}{2n}\sum _{i=1}^n{\{y_i-{\mathbf{x}}_i^{\top }\mathit{\beta }-h(\mathit{\gamma }\circ {\mathbf{z}}_i)\}}^2+\frac{1}{2}{\lambda }_1{∥h∥}_{{\mathcal{H}}_K}^2+{\lambda }_2\rho (\mathit{\gamma };\delta ),$$

(5)

where ${\lambda }_1>0$ and ${\lambda }_2>0$ are two tuning parameters, and penalty $\rho (\mathit{\gamma };\delta )$ may be specified according to a certain regularization method. For the case of sparse group lasso (SGL), we take $p(\mathit{\gamma };\delta )=(1-\delta ){\sum }_{\ell =1}^p{∥{\mathit{\gamma }}^{\ell }∥}_2+\delta {∥\mathit{\gamma }∥}_1$, $\delta \in [0,1]$. Typically, $\delta$ is predetermined and set to $0.95$ or $0.05$ depending on the trade-off between group and within-group sparsity, while the factor ($1-\delta$) controls the relative group sparsity to individual sparsity of each functional predictor $Z^{\ell }$. Meanwhile, a large tuning parameter for ${\lambda }_2$ would remove a certain group of FPC features from the FKMR model when all elements in the vector ${\mathit{\gamma }}^{\ell }$ are zero. Given $h\in {\mathcal{H}}_{\mathcal{K}}$, an equivalent optimization to the above (5) can be formulated as follows:

$$\begin{array}{c}\hfill \underset{\mathit{\alpha },\mathit{\beta },\mathit{\gamma }}{min}{J}_2(\mathit{\alpha },\mathit{\beta },\mathit{\gamma })=\underset{\mathit{\alpha },\mathit{\beta },\mathit{\gamma }}{min}\frac{1}{2n}\sum _{i=1}^n{\left\{y_i-{\mathbf{x}}_i^{\top }\mathit{\beta }-\sum _{k=1}^n{\alpha }_k\mathcal{K}(\mathit{\gamma }\circ {\overrightarrow{\mathbf{z}}}_i,\mathit{\gamma }\circ {\overrightarrow{\mathbf{z}}}_k)\right\}}^2\\ \hfill +\frac{1}{2}{\lambda }_1{\mathit{\alpha }}^{\top }\mathbf{K}(\mathit{\gamma };Z)\mathit{\alpha }+{\lambda }_2\rho (\mathit{\gamma };\delta ),\end{array}$$

(6)

where $\mathbf{K}(\mathit{\gamma };Z)$ is an $n\times n$ matrix whose $(i,k)$th element is ${\left[\mathbf{K}(\mathit{\gamma };\mathbf{Z})\right]}_{ik}=\mathcal{K}(\mathit{\gamma }\circ {\overrightarrow{\mathbf{z}}}_i,\mathit{\gamma }\circ {\overrightarrow{\mathbf{z}}}_k)$. Lemma 1 below establishes the equivalency of optimization solutions between (5) and (6), which is crucial in our estimation procedure.

Lemma 1.

A solution ($\widehat{h}$, $\widehat{\mathit{\beta }}$, $\widehat{\mathit{\gamma }}$) is a minimizer of (5) if and only if ($\widehat{\mathit{\alpha }}$, $\widehat{\mathit{\beta }}$, $\widehat{\mathit{\gamma }}$) is a minimizer of (6), where $\widehat{h}(\widehat{\mathit{\gamma }}\circ \overrightarrow{\mathbf{z}})={\sum }_{k=1}^n{\widehat{\alpha }}_k\mathcal{K}(\widehat{\mathit{\gamma }}\circ \overrightarrow{\mathbf{z}},\widehat{\mathit{\gamma }}\circ {\overrightarrow{\mathbf{z}}}_k)$.

The proof of Lemma 1 is given in Appendix A.1.

Theorem 1

(Existence of optimizers). If the kernel $\mathcal{K}(·,\mathit{\gamma }\circ \overrightarrow{\mathbf{z}})$ is continuous with respect to $\mathit{\gamma }\in {\mathcal{R}}^s$, then there exists a global minimizer ($\widehat{h}$, $\widehat{\mathit{\beta }}$, $\widehat{\mathit{\gamma }}$) for the optimization problem (5).

The proof of Theorem 1 is given in Appendix A.3. Note that there may exist multiple optimal minimizers for (5); Theorem 1 ensures only the existence of optimal solutions, but provides no guarantees for uniqueness due to the fact that (5) or (6) is a nonlinear and non-convex optimization problem. It is worth noting that in both (5) and (6), we set the bandwidth for the kernel at a fixed value due to the identifiability issue with respect to the scaling parameters $\mathit{\gamma }$. Refer to Appendix B for more detailed discussions on the issue of parameter identifiability.

3. Implementation and Algorithm

We propose an iterative algorithm to implement our proposed estimation procedure in which we require the differentiability of the kernel with respect to the scaling factor $\mathit{\gamma }$ and some additional assumptions presented below in order to ensure algorithmic convergence. One part of the algorithm solving (5) is carried out under fixed $\mathit{\gamma }$, where the resulting minimization problem reduces to the equivalent maximization problem in the least-squares kernel machine (3) with the FPC features, ${\overrightarrow{\mathbf{z}}}_i$, being replaced by $\mathit{\gamma }\circ {\overrightarrow{\mathbf{z}}}_i$. As pointed out in Section 1.2, the step of numerical calculation can be easily executed in the same fashion as the solution from the linear mixed model, including the REML estimation of the smoothing parameter ${\lambda }_1$. The other part of the algorithm is performed under fixed $\mathit{\alpha }$, $\mathit{\beta }$ and ${\lambda }_1$, where we solve the nonlinear and non-convex optimization problem to update estimates of $\mathit{\gamma }$. Lemma 2 below helps us solve for the scaling parameter $\mathit{\gamma }$.

Lemma 2.

For fixed (α, β, ${\lambda }_1$), minimizing (6) over γ is equivalent to minimizing over γ the following objective function:

$$\frac{1}{2n}{∥\mathbf{F}\left(\mathit{\gamma }\right)-\tilde{\mathbf{Y}}∥}_2^2+{\lambda }_2\rho (\mathit{\gamma };\delta ),\mathrm{for}{\lambda }_2>0,$$

(7)

where $\mathbf{F}\left(\mathit{\gamma }\right)=\mathbf{K}(\mathit{\gamma };Z)\mathit{\alpha }$ and $\tilde{\mathbf{Y}}=\mathbf{Y}-\mathbf{X}\mathit{\beta }-\frac{n}{2}{\lambda }_1\mathit{\alpha }$.

The proof of Lemma 2 is given in Appendix A.2. Linearizing the function $\mathbf{F}\left(\mathit{\gamma }\right)$ in (7) leads to an equivalent form:

$$\begin{array}{c}\hfill \underset{\mathit{\gamma }}{min}\frac{1}{2n}{∥\tilde{\mathbf{Y}}-\sum _{\ell =1}^p{\nabla }_{\mathit{\gamma }}{\mathbf{F}}^{(\ell )}\left(\tilde{\mathit{\gamma }}\right){\mathit{\gamma }}^{\ell }∥}_2^2+{\lambda }_2\rho (\mathit{\gamma };\delta ),\end{array}$$

(8)

where $\tilde{\mathbf{Y}}=\left(\mathbf{Y}-\mathbf{X}\mathit{\beta }-\frac{n}{2}{\lambda }_1\mathit{\alpha }\right)-\mathbf{F}\left(\tilde{\mathit{\gamma }}\right)+{\nabla }_{\mathit{\gamma }}\mathbf{F}\left(\tilde{\mathit{\gamma }}\right)\tilde{\mathit{\gamma }}$, with ${\nabla }_{\mathit{\gamma }}\mathbf{F}\left(\tilde{\mathit{\gamma }}\right)$ being the gradient of the function $\mathbf{F}$ with respect to $\mathit{\gamma }$ evaluated at $\tilde{\mathit{\gamma }}$ for some $\tilde{\mathit{\gamma }}$, and ${\nabla }_{\mathit{\gamma }}{\mathbf{F}}^{(\ell )}\left(\tilde{\mathit{\gamma }}\right)$ being the columns of ${\nabla }_{\mathit{\gamma }}\mathbf{F}\left(\tilde{\mathit{\gamma }}\right)$ associated with the ℓth group of ${\mathit{\gamma }}^{\ell }$. This is precisely the form of the standard sparse group regularization problem: ${min}_{\mathit{\beta }\in {\mathcal{R}}^p}\frac{1}{2n}{∥\mathbf{Y}-{\sum }_{\ell =1}^p{\mathbf{X}}^{\ell }{\mathit{\beta }}^{\ell }∥}_2^2+{\lambda }_2\rho (\mathit{\gamma };\delta ).$ This implies that (8) presents a standard sparse group regularization problem with a specific choice of penalty function $\rho (\mathit{\gamma };\delta )$.

The convergence of the above iterative search algorithm for updating $\tilde{\mathit{\gamma }}$ for fixed ($\mathit{\alpha }$, $\mathit{\beta }$, ${\lambda }_1$) can be justified by the proximal Gauss–Newton method [29]. Readers are referred to [30] for details on the proximal Gauss–Newton method. One of the key assumptions of the proximal Gauss–Newton method is the existence of a local minimizer. This condition is satisfied in the above (8). This is because according to Theorem 1, there exists a global minimizer.

Algorithm 1 summarizes these iterative steps, which is showed to satisfy a descent property: $J_2({\mathit{\alpha }}^{(r+1)},{\mathit{\beta }}^{(r+1)},{\mathit{\gamma }}^{(r+1)})$$\le J_2({\mathit{\alpha }}^{\left(r\right)},{\mathit{\beta }}^{\left(r\right)},{\mathit{\gamma }}^{\left(r\right)})$ under the convergence of the proximal Gauss–Newton algorithm for Step 2.2.

Algorithm 1 An iterative algorithm for optimization in FKMR.

1.1 Perform FPCA (e.g., the R package fdapace) to extract the functional component features for the p functional predictors, and store them in a grand vector for each individual subject ${\overrightarrow{\mathbf{z}}}_i=[{\left({\mathbf{z}}_i^1\right)}^{\top },\dots ,{\left({\mathbf{z}}_i^p\right)}^{\top }){]}^{\top }$, $i=1,···,n$; 1.2 Initialize $\mathit{\gamma }$ to be a vector of ones. which translates to mapping the original component scores to itself. Set up a grid of possible tuning parameters for ${\lambda }_1$ and ${\lambda }_2$, respectively. Set the kernel bandwidth parameter, which may depend on ${\lambda }_1$. For each pair of $({\lambda }_1,{\lambda }_2)$ from our grid, perform Steps 2.1–2.3 and 3.1 below. 2.1 At the $(r+1)$-th step in the algorithm, first solve the LSKM problem with fixed (${\mathit{\gamma }}^{\left(r\right)},{\lambda }_1$) (based on a closed-form solution) to update ${\mathit{\beta }}^{(r+1)}$ and ${\mathit{\alpha }}^{(r+1)}$. 2.2 Solve the group regularity problem (8) with fixed $\tilde{\mathit{\gamma }}={\mathit{\gamma }}^{\left(r\right)}$ and fixed (${\mathit{\alpha }}^{(r+1)}$, ${\mathit{\beta }}^{(r+1)}$, ${\lambda }_1$, ${\lambda }_2$) using the $r+1$ updates from the previous iteration. At this step, the proximal Gauss–Newton algorithm produces an update ${\mathit{\gamma }}^{(r+1)}$ at convergence. 2.3 Repeat Steps 2.1–2.2 until convergence. 3.1 Perform cross-validation over all pairs of (${\lambda }_1,{\lambda }_2$) to determine the final $(\mathit{\alpha },\mathit{\beta },\mathit{\gamma })$.

To speed up Algorithm 1, we propose the following operational schemes that avoid setting up the pairs of (${\lambda }_1$,${\lambda }_2$) and performing Step 3.1. Here are a few remarks on the two algorithms. (i) Algorithm 2 depends on good starting values in order to enjoy a fast search. (ii) The main difference between Algorithms 1 and 2 is that ${\lambda }_2$ is fixed in Algorithm 1, while it is changing in Algorithm 2. Some similar algorithms with changing tuning parameters have been proposed in the literature, such as the single index model [31]. (iii) There is no guarantee that both algorithms converge to a global minimizer, and the proximal Gauss–Newton method used in the implementation can only find stationary points. Numerical solvers for the optimization problem in (5) or in (6) indeed remain an open problem in the field of nonlinear and nonconvex optimization.

Algorithm 2 A fast operational scheme of Algorithm 1.

1. Step 2.1 of Algorithm 1 is performed by running the linear mixed model with our initial fixed $\mathit{\gamma }$ from Step 1.2 of Algorithm 1 to obtained updated values of ${\lambda }_1$, $\mathit{\beta }$, and $\mathit{\alpha }$. 2. Step 2.2 is performed with solving the group regularity problem (8) through the Gauss–Newton algorithm using cross-validation-based tuning (e.g., R package oem). 3. Rerun Step 2.1 using the updated $\mathit{\gamma }$ from Step 2.2 to obtain the estimates for $\mathit{\beta }$ and $\mathit{\alpha }$.

4. Theoretical Guarantees

Our theoretical analysis focuses on the finite-sample $L_2$ error bounds for the estimators $(\widehat{h},\widehat{\mathit{\gamma }})$ obtained by (5) or (6). Consequently, we are able to establish the estimation consistency. For simplicity, we set $\mathit{\beta }=\mathbf{0}$ and consider a general setting of random vectors ${\mathbf{z}}_1,\dots ,{\mathbf{z}}_n$ so that the FPC features ${\overrightarrow{\mathbf{z}}}_1,\dots ,{\overrightarrow{\mathbf{z}}}_n$ correspond to a special case. Along similar lines as those of [15,32], the estimation consistency is proven in the case of the SGL penalty function. We define a map $\mathrm{\Gamma }$ with an s-element vector $\mathit{\gamma }\in {\mathcal{R}}^s$, which gives rise to a collection of all scaling map functions: $\mathcal{A}=\{\mathrm{\Gamma }:{\mathcal{R}}^s\mapsto {\mathcal{R}}^s\mid \mathrm{\Gamma }\left(\mathbf{z}\right)=\mathit{\gamma }\circ \mathbf{z},\mathbf{z}\in {\mathcal{R}}^s\mathrm{and}\mathit{\gamma }\in {\mathcal{R}}^s\}$. Since $\mathrm{\Gamma }$ is a linear (and bounded) operator, $\mathcal{A}$ is a real vector space where $(c_1{\mathrm{\Gamma }}_1+c_2{\mathrm{\Gamma }}_2)\left(\mathbf{z}\right)=c_1{\mathrm{\Gamma }}_1\left(\mathbf{z}\right)+c_2{\mathrm{\Gamma }}_2\left(\mathbf{z}\right)$ with any $c_1,c_2\in \mathcal{R}$ and ${\mathrm{\Gamma }}_1,{\mathrm{\Gamma }}_2\in \mathcal{A}$. To perform a group regularization estimation, we define an SGL penalty by a norm on $\mathcal{A}$ for a fixed $\delta \in [0,1]$ as follows:

$${∥\mathrm{\Gamma }∥}_{SGL}=\delta \sum _{\ell =1}^p{∥{\mathit{\gamma }}^{\ell }∥}_2+(1-\delta ){∥\mathit{\gamma }∥}_1.$$

(9)

Consequently, the SGL regularization estimation requires the following constrained optimization:

$$\underset{\mathrm{\Gamma }\in \mathcal{A},h\in {\mathcal{H}}_{\mathcal{K}}}{min}{J}_3(\mathrm{\Gamma },h)=\underset{\mathrm{\Gamma }\in \mathcal{A},h\in {\mathcal{H}}_{\mathcal{K}}}{min}{∥\mathbf{Y}-h\circ \mathrm{\Gamma }∥}_n^2+{\lambda }_1{∥h∥}_{H_K}^2+{\lambda }_2{∥\mathrm{\Gamma }∥}_{SGL},$$

(10)

where ${∥\mathbf{Y}-h\circ \mathrm{\Gamma }∥}_n^2=\frac{1}{n}{\sum }_{i=1}^n{\left\{y_i-(h\circ \mathrm{\Gamma })\left({\mathbf{z}}_i\right)\right\}}^2$. Lemma 3 below provides the essential finite-sample inequalities that lead to the estimation consistency.

Lemma 3

(Basic inequality). Let $\widehat{h}\circ \widehat{\mathrm{\Gamma }}$ be the minimizer of (10). Let $h_0\circ {\mathrm{\Gamma }}_0$ be the true function. Then, we have:

$$J_3(\widehat{\mathrm{\Gamma }},\widehat{h})\le 2{(\mathit{ϵ},\widehat{h}\circ \widehat{\mathrm{\Gamma }}-h_0\circ {\mathrm{\Gamma }}_0)}_n+{\lambda }_1{∥h_0∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{\lambda }_2{∥{\mathrm{\Gamma }}_0∥}_{SGL},$$

(11)

where $2{(\mathit{ϵ},\widehat{h}\circ \widehat{\mathrm{\Gamma }}-h_0\circ {\mathrm{\Gamma }}_0)}_n=\frac{2}{n}{\sum }_{i=1}^n{ϵ}_i\left\{(\widehat{h}\circ \widehat{\mathrm{\Gamma }})\left({\mathbf{z}}_i\right)-(h_0\circ {\mathrm{\Gamma }}_0)\left({\mathbf{z}}_i\right)\right\}$.

We need the following notation before presenting our theoretical guarantees. Let $\mathcal{N}(\delta ,M,P_n)$ denote the minimal $\delta$ covering number of the function set $\mathcal{M}$ under the empirical metric $P_n$ based on the random vectors ${\mathbf{z}}_1,\cdots ,{\mathbf{z}}_n$. Let $N=\mathcal{N}(\delta ,M,P_n)$ be a shorthand notation. This means that there exist functions $m_1,\cdots ,m_N$ (not necessarily in the set $\mathcal{M}$) such that for every function $m\in \mathcal{M}$, there exists a $j\in \{1,\cdots ,N\}$ such that ${∥m-m_j∥}_{P_n}\le \delta$, with ${∥m-m_j∥}_{P_n}:=\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\{m\left({\mathbf{z}}_i\right)-m_j\left({\mathbf{z}}_i\right)\}}^2}$. Define the $\delta$-entropy of $\mathcal{M}$ for the empirical metric, $P_n$, as $H(\delta ,\mathcal{M},P_n):=log\left(\mathcal{N}(\delta ,\mathcal{M},P_n)\right)$. Consider a functional space of the form:

$$\mathcal{B}=\left\{b:=b(h,\mathrm{\Gamma })=\frac{h\circ \mathrm{\Gamma }-h_0\circ {\mathrm{\Gamma }}_0}{{∥h∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{∥h_0∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{∥\mathrm{\Gamma }∥}_{SGL}^2+{∥{\mathrm{\Gamma }}_0∥}_{SGL}^2}|h\in {\mathcal{H}}_{\mathcal{K}},\mathrm{\Gamma }\in \mathcal{A}\right\}.$$

We postulate the following assumptions.

Assumption A1.

The error term $\mathit{ϵ}={({ϵ}_1,\dots ,{ϵ}_n)}^{\top }$ is uniformly sub-Gaussian; that is, for constants $C_1$ and $C_2$,

$$\underset{n\ge 1}{max}\underset{i=1,\cdots ,n}{max}{C}_1^2\left[E\left\{exp\left(\frac{{{ϵ}_i}^2}{C_1^2}\right)\right\}-1\right]\le C_2.$$

Clearly, the moment condition is bounded below from zero.

Assumption A2.

${∥{\mathrm{\Gamma }}_0∥}_{SGL}^2+{∥h_0∥}_{{\mathcal{H}}_{\mathcal{K}}}^2>0$, and the entropy of space $\mathcal{B}$ with respect to the empirical metric $P_n$ is bounded as follows:

$$H(\delta ,\mathcal{B},P_n)\le C_3{\delta }^{-2\psi },$$

where $C_3$ is some constant and $\psi \in (0,1)$.

Assumption A3.

${sup}_{b\in \mathcal{B}}{∥b∥}_{P_n}\le C_4$ for some constant $C_4$.

Theorem 2.

(Consistency) Under Assumptions 1–3 above, if tuning parameters ${\lambda }_1$ and ${\lambda }_2$ satisfy

$${\lambda }_2^{-1}=n^{\frac{1}{1+\psi }}{\left({∥h_0∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{∥{\mathrm{\Gamma }}_0∥}_{SGL}\right)}^{\frac{1-\psi }{1+\psi }},and{\lambda }_1=O_p\left(1\right){\lambda }_2,$$

then we have

$${∥\widehat{h}\circ \widehat{\mathrm{\Gamma }}-h_0\circ {\mathrm{\Gamma }}_0∥}_n=O_p\left(n^{-\frac{1}{2+2\psi }}\right){\left({∥h∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{∥\mathrm{\Gamma }∥}_{SGL}\right)}^{\frac{\psi }{1+\psi }},and$$

(12)

$${∥\widehat{h}∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{∥\widehat{\mathrm{\Gamma }}∥}_{SGL}=O_p\left(1\right)\left({∥h_0∥}_{{\mathcal{H}}_{\mathcal{K}}}^2+{∥{\mathrm{\Gamma }}_0∥}_{SGL}\right).$$

(13)

Theorem 2 implies estimation consistency under the right rates for the two tuning parameters ${\lambda }_1$ and ${\lambda }_2$. Due to the potential identifiability issues explained in detail in Appendix B, although the estimator $(\widehat{h},\widehat{\mathrm{\Gamma }})$ may not be unique, the sum of $\widehat{h}$ and $\widehat{\mathrm{\Gamma }}$ is not too far away from the sum of the true $h_0$ and ${\mathrm{\Gamma }}_0$.

Corollary 1.

If the RKHS, ${\mathcal{H}}_{\mathcal{K}}$, contains differentiable functions $\nabla h\left(\mathbf{z}\right)$ whose norm ${∥\nabla h\left(\mathbf{z}\right)∥}_{{\mathcal{H}}_{\mathcal{K}}}$ is uniformly bounded for all functions $h\in {\mathcal{H}}_{\mathcal{K}}$ and $\mathbf{z}\in R^s$, then Assumption 2 holds when Theorem 2 is replaced by $H(\delta ,{\mathcal{H}}_{\mathcal{K}},P_n)\le C_1{\delta }^{-2\psi },\mathrm{for}\mathrm{all}\delta \ge 0$.

The proofs of Theorem 2 and Corollary 1 are given in Appendix A.4 and Appendix A.5, respectively. Often, when we are only interested in a subset of functions in the RKHS (e.g., functions with norm less than one), we can substitute the full space ${\mathcal{H}}_{\mathcal{K}}$ in Corollary 1 with the subspace of interest. Refer to [15] or [32], where both considered an RKHS (i.e., Sobolev space) with functions of norm less than or equal to one.

5. Simulation Experiments

We performed extensive simulation to investigate the performance of our proposed procedure, including the performance of SGL variable selection and its overall accuracy. Due to the limitations of space, we include results from two simulation experiments in this section, and more results may be found in the first author’s Ph.D. dissertation [30].

5.1. Setup

In the evaluation of the performance accuracy, following [15], we used both quasi-$R^2$ and adjusted quasi-$R^2$ defined as follows:

$$\begin{array}{c}\hfill R_Q^2:=1-\frac{{\sum }_{i=1}^n{(y_i-\widehat{y_i})}^2}{{\sum }_{i=1}^n{(y_i-\overline{y_i})}^2},and{R}_{AQ}^2:=1-\left(1-R_Q^2\right)\left(\frac{n-1}{n-(k+1)}\right).\end{array}$$

The latter is known to be appealing for the comparison of the estimation sparsity. There is another performance metric of interest in addition to model accuracy. Performance in variable selection is summarized in terms of the stability measured by sensitivity and specificity for both functional and variable selections under these simulation experiments. Our algorithm uses existing R packages, including emmreml, kspm, and oem.

Specifically, we designed the following two simulation settings.  

Scenario 1: A single functional predictor with sparsity in the FPC features.

Scenario 2: Multiple functional predictors with sparsity in the functional predictors and with sparsity in the FPC features of important functional predictors.  

Each of these two scenarios would be handled using certain suitable penalty functions to address the designed sparsity; for example, in Scenario 2 we used a two-level variable selection penalty (e.g., SGL) to deal with two types of sparsity in the true model. In all analyses, we used the Gaussian kernel $\mathcal{K}(u,v)=exp\left(-\frac{1}{p}{∥u-v∥}^2\right)$ in our estimation, where p was set as the number of features, which is equivalent to dividing the $\mathit{\gamma }$ vector by $\sqrt{p}$. This scaling parameter may be either estimated or set to the number of features to overcome the identifiability issue according to [33], where theoretical justification was given for the use of the number of features for the bandwidth parameter in the case of the Gaussian kernel.

According to [23], due to the difficulty of the graphical display for the estimated s-dimensional function $h(·)$ of $\mathbf{z}$, we summarized the goodness-of-fit by regressing the true h on the estimated $\widehat{h}$, with both being evaluated at the design points. From this concordance regression analysis, we may measure the goodness-of-fit on $\widehat{h}$ through the average intercepts, slopes, and R-squared (also known as the coefficient of determination) obtained over the number of replications. Clearly, a high-quality fit is reflected by (i) the intercept being close to zero, (ii) the slope being close to one, and (iii) the R-squared being close to one. Moreover, we graphically display the estimated function $\widehat{h}$ by setting all variables equal to 0.5 except the one of interest over a grid of 100 equally spaced points on the interval $[0,1]$. Such visualization of the functional estimation at each margin further facilitates the evaluation of the proposed algorithm in addition to the results obtained from the concordance regression analyses.

In all scenarios, we generated 1000 IID functional paths, of which 750 paths were assigned to the training set and 250 paths were assigned to the test set for an external performance evaluation. It is the test set that we used to display the performance accuracy. We used a one-dimensional covariate $x_i$ to show the flexibility of our model in a semi-parametric setting, with independent copies of $x_i\sim N(0,1)$. We chose the true coefficients in the kernel machine model similar to those given in [23].

5.2. Simulation in Scenario 1

In this simple scenario with a single functional predictor, we simulated data from a model with sparsity in its FPC features. To do so, we generated a single functional predictor based on the first 15 eigenbasis of the Fourier basis functions over the interval $[0,1]$: $Z\left(t\right)={\sum }_{j=1}^{15}\sqrt{{\varsigma }_j}{\xi }_j{\varphi }_j\left(t\right)$. That is, a functional predictor was created as a linear combination of the 15 basis functions, where ${\varphi }_j(·)$ is the $j\mathrm{t}\mathrm{h}$ Fourier basis function, ${\varsigma }_j$ is the jth eigenvalue of Z, and ${\xi }_j$ is the jth FPC feature that is simulated from a normal distribution detailed as follows.

There were 100 sampled points that were first equally spaced in the interval $[0,1]$ and then varied with certain small deviations drawn from $\nu \sim N(0,0.001)$. Set ${\varsigma }_j=45\times 0.{64}^j$ and ${\xi }_j\sim N(0,1)$ independently over $j=1,\dots ,15$. As was done in [17], instead of directly using ${\xi }_j$, we used ${\zeta }_j=\mathrm{\Phi }\left({\xi }_j\right)$, where $\mathrm{\Phi }$ is the CDF of the standard normal. This resulted in $\overrightarrow{\mathbf{z}}={({\zeta }_1,\dots ,{\zeta }_{15})}^{\top }$. We chose the second, ${\zeta }_2$, and ninth, ${\zeta }_9$, features as important features in the following true nonlinear non-additive model:

$$y_i=2x_i+20cos\left(2\pi {\zeta }_{i2}\right)-10sin\left(2\pi {\zeta }_{i9}\right)+{\zeta }_{i2}{\zeta }_{i9}+{ϵ}_i,$$

with ${ϵ}_i\stackrel{iid}{\sim }N(0,1)$. FPCA was performed by the R package PACE [34], producing the estimated FPC scores, $\widehat{{\xi }_j}$, as well as the estimated eigenvalues, $\widehat{{\varsigma }_j}$, which in turn enabled us to compute ${\widehat{\zeta }}_j$, $j=1,\dots ,15$.

We applied both LASSO and MCP penalty functions in our implementation, termed as $FKM{R}_{Lasso}$ and $FKM{R}_{MCP}$, respectively. We compared the results of our method with the standard linear approach with both LASSO and MCP under the assumption of linear functional relationships, as well as the COSSO method for functional additive regression [15] using the R package COSSO [15,34]. Since the COSSO package is built for nonparametric regression (and not partial linear models), we adopted the backfitting strategy and regressed the residuals with our estimated effect of $x_i$ removed.

In addition, we compared our method with an oracle FKMR estimator, called $FKM{R}^{oracle}$, that assumed the full knowledge of the true ${\zeta }_j$ containing two true nonzero signals, ${\zeta }_2$ and ${\zeta }_9$. We also considered two oracle versions of our proposed algorithm, $FKM{R}_{Lasso}^{oracle}$ and $FKM{R}_{MCP}^{oracle}$, both of which used the knowledge of true ${\zeta }_j$ in order to evaluate the performance of the FPCA procedure. This evaluation is important as our proposed procedure can be in principle used in simpler cases that do not involve functional covariates. Note that once we used FPCA to obtain ${\widehat{\zeta }}_j$ features, our algorithm essentially works in a standard regression setting with the sparsity of covariates. Thus, our proposed procedure can be in principle used in simpler cases with scalar covariates. In Scenario 1, due to the highly nonlinear relationships between the FPC features and the outcome, as expected, the naive linear model performed poorly in terms of both model selection and model consistency. The detailed simulation results for Scenario 1 can be found in the first author’s Ph.D. dissertation [30]. In brief, our proposed method worked well in all aspects. In this setting, COSSO also worked well in terms of model fit, but it tended to select noisy features more frequently than our proposed method, leading to more false positives.

5.3. Simulation in Scenario 2

Now, we generated four functional predictors of the form: $Z^{\ell }\left(t\right)={\sum }_{j=1}^9\sqrt{{\varsigma }_j^{\ell }}{\xi }_j^{\ell }{\varphi }_j^{\ell }\left(t\right)$, $\ell =1,\dots ,4,$ where ${\varphi }_j^{\ell }$, ${\varsigma }_j^{\ell }$, and ${\xi }_j^{\ell }$ were set in the same way as those given in Scenario 1. It follows that $\overrightarrow{\mathbf{z}}={({\zeta }_1^1,\dots ,{\zeta }_9^1,\dots ,{\zeta }_1^4,\dots ,{\zeta }_9^4)}^{\top }$, where ${\zeta }_j^{\ell }$ is the jth $\mathrm{\Phi }$-transformed feature for the ℓth functional covariate. Sparsity was specified as follows: the first and second functional covariates, $Z^1$ and $Z^2$, were chosen as important signals in which these transformed FPC features, $\{{\zeta }_1^1,{\zeta }_3^1,{\zeta }_4^1,{\zeta }_2^2,{\zeta }_7^2\}$, are five important features (three features from the $Z^1$ and two features from $Z^2$) that are related to the outcome:

$$\begin{array}{cc}\hfill y_i=& 2x_i+{\zeta }_{i1}^1+{\zeta }_{i3}^1+{\zeta }_{i4}^1+{\zeta }_{i2}^2+{\zeta }_{i7}^2+10cos\left(2\pi {\zeta }_{i1}^1\right)-10{\left({\zeta }_{i2}^2\right)}^2+10{\left({\zeta }_{i7}^2\right)}^2-10{\left({\zeta }_{i3}^1\right)}^2\hfill \\ \hfill & +10exp(-{\zeta }_{i3}^1){\zeta }_{i4}^1-8sin\left(2\pi {\zeta }_{i7}^2\right)cos\left(2\pi {\zeta }_{i3}^1\right)+20{\zeta }_{i1}^1{\zeta }_{i7}^2+{ϵ}_i,i=1,\dots ,n,\hfill \end{array}$$

where ${ϵ}_i\stackrel{iid}{\sim }N(0,1)$. This model specifies both group sparsity (two of the four functional predictors) and within-group sparsity (three of the nine FPC features in $Z^1$ and two of the nine FPC features in $Z^2$). In addition, we specified non-additive relationships in the true model across multiple functional covariates.

We fit the data using the proposed methods, including $FKM{R}_{GMCP}^{oracle}$, $FKM{R}_{Lasso}$, $FKM{R}_{GLasso}$, $FKM{R}_{SGL}$, $FKM{R}_{MCP}$, and $FKM{R}_{GMCP}$, and the results based on 100 replicates are summarized in

Table 1. Goodness-of-fit and the concordance regression for Scenario 2.

Model	${\mathit{R}}_{\mathbf{AQ}}^2$	$\beta$	Reg of h on $\widehat{\mathit{h}}$
			Intercept	Slope	${\mathit{R}}^2$
$FKM{R}_{Lasso}$	0.830	2.00	−0.062	1.01	0.848
$FKM{R}_{GLasso}$	0.937	1.99	−0.055	1.01	0.972
$FKM{R}_{SGL}$	0.928	2.00	−0.051	1.01	0.955
$FKM{R}_{MCP}$	0.835	2.01	−0.062	1.01	0.856
$FKM{R}_{GMCP}$	0.935	1.99	−0.056	1.01	0.970
$FKM{R}_{GMCP}^{oracle}$	0.911	1.99	−0.049	1.01	0.937
COSSO	0.832	–	–	–	–
LM + Lasso	0.453	–	–	–	–
LM + GLasso	0.324	–	–	–	–
LM + SGL	0.450	–	–	–	–
LM + MCP	0.513	–	–	–	–
LM + GMCP	0.307	–	–	–	–

. For comparison, we also fit the simulated data by existing methods, including the linear model (denoted by LM + penalty), COSSO functional additive regression, and the oracle method using the knowledge of true important features in the analysis, as done in the above simulation of Scenario 1. From Table 1 regarding the goodness-of-fit, we see that all of our FKMR estimators outperformed the standard linear estimators in terms of $R_{AQ}^2$ among all of our penalty functions, and they outperformed COSSO for penalties that accounted for group sparsity. In the concordance regression analysis, we see that all intercepts were close to zero, all slopes close to one, and all $R^2$ close to one, indicating a high goodness-of-fit for functional estimation. COSSO tended to perform on par for penalties that did not account for group sparsity (LASSO and MCP). It is evident that using a group sparsity penalty function (SGL, GLasso, and GMCP) clearly outperformed the methods that did not regularize the grouping of covariates (Lasso and MCP). In addition, our FKMR estimators (except $FKM{R}_{Lasso}$) performed as well as the oracle estimator $FKM{R}_{GMCP}^{oracle}$ both in terms of $R_{AQ}^2$ and in terms of our estimate of functional h. The results also indicated that there were little differences between using a concave (MCP or GMCP) penalty function or using a convex (GLasso or SGL) penalty function.

As regards the group sparsity,

Table 2. Sensitivity and specificity of functional selection for Scenario 2.

Model	Selection Frequency
	$\widehat{{\mathbf{Z}}^{\mathbf{1}}}$	$\widehat{{\mathbf{Z}}^{\mathbf{2}}}$	$\widehat{{\mathbf{Z}}^{\mathbf{3}}}$	$\widehat{{\mathbf{Z}}^{\mathbf{4}}}$
$FKM{R}_{Lasso}$	100	100	0	0
$FKM{R}_{GLasso}$	100	100	4	4
$FKM{R}_{SGL}$	100	100	0	0
$FKM{R}_{MCP}$	100	100	0	0
$FKM{R}_{GMCP}$	100	100	3	4
COSSO	100	100	5	6
LM + Lasso	100	100	19	21
LM + GLasso	94	99	7	8
LM + SGL	100	100	19	18
LM + MCP	100	100	20	19
LM + GMCP	93	99	7	8

indicates that the all methods had a high sensitivity of detecting functional signals, while the proposed FKMR methods had better specificity than both sparse linear models and COSSO. Concerning the within-group sparsity, it is interesting to note that a bigger difference was seen in terms of what type of penalty function was being used in feature selection. As shown in

Table 3. FPC feature selection for signal functional $Z^1$ in Scenario 2.

Model	Selection Frequency
	$\widehat{{\mathbf{\zeta }}_{\mathbf{1}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{2}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{3}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{4}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{5}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{6}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{7}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{8}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{9}}^{\mathbf{1}}}$
$FKM{R}_{Lasso}$	100	1	97	0	0	0	0	0	0
$FKM{R}_{GLasso}$	100	100	100	100	100	100	100	100	100
$FKM{R}_{SGL}$	100	21	100	71	26	20	17	16	15
$FKM{R}_{MCP}$	100	1	99	1	0	0	0	0	0
$FKM{R}_{GMCP}$	100	100	100	100	100	100	100	100	100
COSSO	100	2	100	93	1	0	0	1	0
LM + Lasso	100	10	100	100	10	8	7	10	5
LM + GLasso	94	94	94	94	94	94	94	94	94
LM + SGL	100	12	100	100	10	8	8	11	5
LM + MCP	100	10	100	100	9	8	9	7	5
LM + GMCP	93	93	93	93	93	93	93	93	93

and

Table 4. FPC feature selection for signal functional $Z^2$ in Scenario 2.

Model	Selection Frequency
	$\widehat{{\mathbf{\zeta }}_{\mathbf{1}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{2}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{3}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{4}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{5}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{6}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{7}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{8}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{9}}^{\mathbf{2}}}$
$FKM{R}_{Lasso}$	0	3	0	0	0	0	100	0	0
$FKM{R}_{GLasso}$	100	100	100	100	100	100	100	100	100
$FKM{R}_{SGL}$	16	100	14	7	16	23	100	15	7
$FKM{R}_{MCP}$	0	11	0	0	0	1	100	0	0
$FKM{R}_{GMCP}$	100	100	100	100	100	100	100	100	100
COSSO	8	97	5	5	5	15	100	3	3
LM + Lasso	17	100	14	7	16	23	100	15	6
LM + GLasso	99	99	99	99	99	99	99	99	99
LM + SGL	17	100	14	7	16	23	100	15	7
LM + MCP	17	100	13	6	16	23	100	15	8
LM + GMCP	99	99	99	99	99	99	99	99	99

, using a general penalty (e.g., Lasso and MCP) that does not take the grouping structure into account tended to under-select important features within a group. COSSO tended to perform well within group sparsity. Moreover, Figure 2 shows that the FKMR method estimated the five signal functions ($Z^1$ and $Z^2$) well.

6. Data Example

To show the usefulness of our proposed methodology, we analyzed data of 550 children recruited by the ELEMENTS study [35], who had consent to wear an actigraph (ActiGraph GT3X+; ActiGraph LLC. Pensacola, FL, USA). This wearable was to be placed on their non-dominant wrist for five to seven days with no interruption. The actigraph measured tri-axis accelerometer data sampled at 30 Hz, which captured three different directions of a person’s movement. The BMI was the outcome of interest as it is biomarker of obesity. Sex and age were confounding factors used in the analysis. Due to some missing data, our analysis only included children who wore the device properly for 85% or more over the study period, which resulted in 395 participants, consisting of 189 males and 206 females. Other studies such as [36] have excluded days of accelerometer data with more than five percent missing. The mean ± SD BMI of the study cohort was 21.5 ± 4.1. The mean age of the study participants was 14.3 ± 2.1 y. A more detailed description of the dataset used for this paper can be found in [37]. Our primary interest was to see if the BMI is associated with physical activity in the presence of other covariates, specifically sex and age. We preprocessed the activity counts over the 7 d of wear by taking the median in the 1 min epoch over the entire 7 d of wear. For example, since all the participants started wearing the device at 3 p.m., the first data point for each individual was a median of 7 ACs (each for one day) for the 1 min epoch of 3:00–3:01 p.m. This procedure that takes the medians across the minutes from different days has been considered in other applications such as [36]. See Figure 3 as an example of the resulting time series of medians derived from the AC data displayed in Figure 1.

We applied the following five models, labeled as M0–M4 for convenience, to analyze the data with the 24 h median ACs as functional predictors. Let ${\xi }_{ij}^k$ be the ith person’s kth FPC score for functional predictor j.

M0:

Linear model (LM) with only the fixed features: $BM{I}_i\sim {\beta }_0+{\beta }_{1}Ag{e}_i+{\beta }_{2}Se{x}_i;$

M1:

Linear model with SGL penalty (LM+SGL) using the FPCA features: $BM{I}_i\sim {\beta }_0+{\beta }_{1}Ag{e}_i+{\beta }_{2}Se{x}_i+{\sum }_{j=1}^3{\sum }_{k=1}^{s_k}{\beta }_j^k{\xi }_{ij}^k;$

M2:

LSKM using the FPCA features: $BM{I}_i\sim {\beta }_0+{\beta }_{1}Ag{e}_i+{\beta }_{2}Se{x}_i+h\left({\mathbf{z}}_i\right);$

M3:

FKMR model with SGL penalty ($FKM{R}_{SGL}$) using the FPCA features: $BM{I}_i\sim {\beta }_0+{\beta }_{1}Ag{e}_i+{\beta }_{2}Sex+h(\mathit{\gamma }\circ {\mathbf{z}}_i);$

M4:

COSSO using the FPCA features: $res\left(BM{I}_i\right)|{\mathbf{z}}_i\sim {\sum }_{j=1}^3{\sum }_{k=1}^{s_k}{f}_{ij}\left({\xi }_{ij}^k\right).$ In order for a direct application of the COSSO R package, we used residuals $res\left(BM{I}_i\right)=BM{I}_i-\widehat{{\beta }_0}+\widehat{{\beta }_1}Ag{e}_i+\widehat{{\beta }_2}Se{x}_i$ in the COSSO model fit, with ${\widehat{\beta }}_0,{\widehat{\beta }}_1$ and ${\widehat{\beta }}_2$ being the estimates of the coefficients from Model M0.

The BMI and age were mean centered and scaled to be a standard deviation of one, so ${\beta }_0$ was absent in the models. Here are some key findings from the data analyses. First, in terms of the goodness-of-fit,

Table 5. Goodness-of-fit for the five models used in the data analysis.

Model	Adjusted ${\mathit{R}}^2$
M0: LM	0.07
M1: LM + SGL	0.13
M2: LSKM	0.18
M3: $FKM{R}_{SGL}$	0.30
M4: COSSO	0.14

suggests that M3, i.e., our proposed model FKMR with the SGL penalty, gave the best performance, where the adjusted $R^2$ of M3 was nearly twice as big as all the other four models. Second, it is interesting to note that both the COSSO and the $FKM{R}_{SGL}$ did not select the FPC scores associated with the Z-axis. Third, as shown in

Table 6. Axis-specific FPC feature selection.

Model	X-Axis						Y-Axis					Z-Axis
	$\widehat{{\mathbf{\zeta }}_{\mathbf{1}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{2}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{3}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{4}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{5}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{6}}^{\mathbf{1}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{1}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{2}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{3}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{4}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{5}}^{\mathbf{2}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{1}}^{\mathbf{3}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{2}}^{\mathbf{3}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{3}}^{\mathbf{3}}}$	$\widehat{{\mathbf{\zeta }}_{\mathbf{4}}^{\mathbf{3}}}$
$FKM{R}_{SGL}$		✓	✓	✓		✓	✓		✓		✓
COSSO				✓			✓		✓
LM + SGL	✓			✓	✓	✓	✓	✓	✓						✓

, all of the FPC components chosen by COSSO were also chosen by the $FKM{R}_{SGL}$. It is worth noting that the linear model together with the SGL penalty selected the highest number of FPC components, yet performed the worst in terms of the model fit.

7. Conclusions

In this paper, we proposed a method to model the nonlinear relationship between multiple functional predictors and a scalar outcome in the presence of other scalar confounders. We used the FPCA to decompose the functional predictors for feature extraction and used the LSKM framework to model the functional relationship between the outcome and principal components. We developed a simultaneous procedure to select important functional predictors and important features within selected functionals. We proposed a computationally efficient algorithm to implement our regularization method, which was easily programmed in R with the utility of multiple existing R packages. It should be noted that although we focused on functional regression in this paper, the method proposed can be applied to non-functional predictors. In effect, by using functional principal components, we essentially bypassed the infinite-dimensional problem and worked effectively in a non-functional framework with the FPC features. Through simulation and using data from the ELEMENT dataset, we demonstrated how the FKMR estimator outperformed existing methods in terms of both variable selection and model fit. It should be noted that the existing COSSO method did perform well in terms of variable selection, as shown in Section 5.

A technical issue pertains to identifiability limitations with regard to the bandwidth parameter and to the RKHS estimator. To overcome this, we suggested fixing the bandwidth parameter; see the detailed discussion in Section 3. We established key theoretical guarantees for our proposed estimator. In the case where there are multiple proposed estimators (and thus the identifiability issues arise), the established theoretical properties in Section 4 apply to any of those estimators.

Variable section on functional predictors presents many technical challenges, and there are many methodological problems that remain unsolved. This paper demonstrated a possible framework to regularize estimation with a bi-level sparsity of functional group sparsity and within-group sparsity. In the LSKM paper [23], it was briefly mentioned that if the relationship between the scalar outcome and p genetic pathways is additive, we can tweak the model as $y_i={\mathbf{x}}_i^{\top }\beta +h_1\left({\mathbf{z}}_i^1\right)+\cdots +h_p\left({\mathbf{z}}_i^p\right)+{ϵ}_i$ where each $h_j$ belongs to its own RKHS. It is easy to extend our method and algorithms to handle this case. For future research, an extension on longitudinal outcomes may be considered via a mixed-effects model $y_{ij}={\mathbf{x}}_i^{\top }\beta +h\left({\mathbf{z}}_{ij}\right)+{\mathbf{u}}_{ij}^{\top }{v}_i+{ϵ}_{ij}$ where ${\mathbf{u}}_{ij}^{\top }{v}_i$ are the random effects. Other useful extensions to the proposed paradigm would be on the lines of generalized linear models and Cox regression models.