Research Based on High-Dimensional Fused Lasso Partially Linear Model

Abstract

In this paper, a partially linear model based on the fused lasso method is proposed to solve the problem of high correlation between adjacent variables, and then the idea of the two-stage estimation method is used to study the solution of this model. Firstly, the non-parametric part of the partially linear model is estimated using the kernel function method and transforming the semiparametric model into a parametric model. Secondly, the fused lasso regularization term is introduced into the model to construct the least squares parameter estimation based on the fused lasso penalty. Then, due to the non-smooth terms of the model, the subproblems may not have closed-form solutions, so the linearized alternating direction multiplier method (LADMM) is used to solve the model, and the convergence of the algorithm and the asymptotic properties of the model are analyzed. Finally, the applicability of this model was demonstrated through two types of simulation data and practical problems in predicting worker wages.

1. Introduction

With the advent of the era of big data, many fields have generated massive and complex data, such as genetic representations, biomedical image processing, tumor classification, signal processing, financial time series analysis, etc., and the research on the statistical inference of high-dimensional data has become a hot research topic in recent years. Among these, parametric regression modeling is a crucial method to study the statistical relationships between response variables and a set of covariates. However, in practical data analysis, we may not know the true model structure and can use nonparametric regression models to fit the nonlinear relationships of the variables in the data, but when the dimensionality of the covariates is large, such models can create the problem of “dimensional bane”. Therefore, in the 1980s, the semiparametric regression model, which retains the simplicity of parametric regression but has the flexibility of nonparametric regression, was proposed by Engle et al. [1] for the first time and has wider applications. As a common semiparametric regression model, the partially linear model includes three parts: the unknown regression coefficient, unknown nonparametric function, and Gaussian distribution random error. It can be expressed as

$$Y=X^T\beta +g\left(T\right)+\epsilon ,$$

(1)

where $Y={(Y_1,\cdots Y_n)}^T$ is a real-valued response variable, and $\beta \in R^p$ is a p-dimensional unknown parameter vector. $T={(T_1,\cdots T_n)}^T$ is a random variable valued at $[0,1]$, $g(·)$ is a smooth unknown function defined on $[0,1]$, and $\epsilon ={({\epsilon }_1,\cdots ,{\epsilon }_n)}^T$ is a random error.

This model combines the advantages of the multivariate linear model and nonparametric model, so it has been widely welcomed and considered by statisticians and has achieved very fruitful research results in a short period of time. The non-parametric part of the partially linear model can be estimated using spline estimation, kernel functions, etc. The nonparametric part was estimated using an N-W kernel [2], and nonparametric regression values were inserted in the nonlinear orthogonal projection of the nonparametric part to construct estimates of the parametric part. The idea of kernel smoothness was incorporated in partially linear model studies [3], and least squares was used to obtain parameter estimates. The estimation results of the two-stage spline smooth estimation method and the biased regression method for solving the parameter components were investigated [4]; under the smooth parameter is a sequence that tends to zero. The nonparametric components were estimated with penalized regression splines [5], and then robust S-estimators were used to construct the parametric components. The parameter part of the partially linear model can be estimated using regression methods such as least squares. In [6], the authors studied the two-stage algorithm of the partially linear model under orthogonal constraints. The constrained contour least squares estimation of partially linear models with high-dimensional covariates was investigated in [7].

For high-dimensional data, variable selection is a common approach. The term “high-dimensional” data means that the number of variables is much larger than the sample size. At present, there are many methods for the variable selection of high-dimensional linear models, such as LASSO [8,9], adaptive LASSO [10,11], SCAD [12], fused lasso [13,14,15], and so on. There have also been many studies on variable selection in partially linear models. In [16], the authors studied the asymptotic nature of lasso, a high-dimensional partially linear model. A new variable selection method based on gradient learning is proposed by [17]. The study in [18] proposes a partially linear model for ultra-high-dimensional prediction based on the partial correlation between the partial residuals of the response and the predicted value. Adaptive lasso penalty partial least squares estimates are constructed (see [19,20]). However, the problem of partially linear models highly correlated to adjacent variables remains to be studied. Fused lasso is a method proposed by Tibshirani [21] in feature selection for linear models for ordering in a meaningful way and has since been fully developed in linear models [22,23]. In order to better estimate the partially linear model with a high correlation between adjacent variables, the fused lasso partially linear model is proposed, which belongs to the convex optimization problem.

Traditional methods for solving convex optimization problems include the standard interior point method [24] and gradient descent method [25], but these methods are less effective with high-dimensional data. The alternate-direction multiplier method (ADMM) is particularly suitable for solving large-scale sparse optimization problems in statistics [26,27,28], but classical ADMM solving the fused lasso problem will cause one of the subproblems to not have closed-form solutions. Thus, LADMM is proposed by [14], that is, the linearization technique is introduced to linearize one of the subproblems to obtain an approximate solution. Therefore, LADMM is used to solve the fused lasso partially linear model in this paper.

The concrete structure of this paper is as follows. In Section 2, the fused lasso partially linear model is constructed, and the least squares method is used to estimate the parameters, in which the semiparametric model is converted to the parametric model by using the kernel function method. In Section 3, the algorithm of LADMM for solving the high dimensional partially linear model is designed. In Section 4, the convergence of the algorithm is given. In Section 5, the asymptotic property of the model is proved. In Section 6, two types of data are given for numerical simulation, and the applicability of the model is analyzed. In Section 7, the method proposed in this article is applied to the actual problem of worker wages, verifying its effectiveness.

2. Construction of the Fused Lasso Partially Linear Model

In this section, we construct the fused lasso partially linear model for parameter estimation. Firstly, assuming the parameters are known, the nonparametric part is estimated by the kernel function method, the partially linear model is converted into a parametric model, and then the fused lasso partially linear model is constructed.

Firstly, some basic assumptions need to be made [3].

1.

The support set of the kernel function $K(·)$ is $[-1,1]$, and there are constants $0\le Q_1\le Q_2$ satisfying $Q_1\le K\left(u\right)\le Q_2$, $\int K\left(u\right)du=1$, $\int uK\left(u\right)du=0$, $\int u^{2}K\left(u\right)du\ne 0$.

2.

$tr\left(K^{\prime }K\right)=tr\left(K\right)=O_p\left(h^{-1}\right)$.

3.

${∥g∥}_2^2=O_p\left(n{h}^4\right)$, $g(·)$ is the nonparametric term.

4.

$h=O_p\left(n^{-\frac{1}{5}}\right)$, where the bandwidth h is a constant sequence.

Assuming $(Y_1;X_1^T;T_1),\cdots ,(Y_n;X_n^T;T_n)$ is an independent homogeneous sample of the model, denote $X_i={(X_{i1},\cdots X_{ip})}^T$, $\beta ={({\beta }_1,\cdots {\beta }_p)}^T$; then, Equation (1) can be written as

$$Y_i=X_i^T\beta +g\left(T_i\right)+{\epsilon }_{i}i=1,2,\cdots ,n.$$

(2)

Assuming $\beta$ is known, Equation (2) can be written as a nonparametric regression model:

$$Y_i-X_i^T\beta =g\left(T_i\right)+{\epsilon }_{i}i=1,2,\cdots ,n.$$

(3)

The nonparametric term $g(·)$ can be estimated using the kernel function method

$$g_n(t,\beta )=\sum _{j=1}^p{W}_{nj}\left(t\right)(Y_j-X_j^T\beta ),$$

(4)

where $W_{nj}\left(t\right)=K_h(T_j-t)K_h(T_j-t)/{\displaystyle \sum _{k=1}^n}{K}_h(T_k-t)$ is a non-negative weight function satisfying $0\le W_{nj}\left(t\right)\le 1$, ${\displaystyle \sum _{j=1}^n}{W}_{nj}\left(t\right)=1$, $K(·)$ is a non-negative kernel function, and the bandwidth h is a constant sequence that converges to zero. Thus, we have

$$Y_i-X_i^T\beta =\sum _{j=1}^n{W}_{nj}\left(t\right)(Y_j-X_j\beta )+{\epsilon }_i.$$

(5)

Let

$${\widehat{X}}_i=X_i-\sum _{j=1}^n{W}_{nj}\left(T_j\right)X_j,{\widehat{Y}}_i=Y_i-\sum _{j=1}^n{W}_{nj}\left(T_j\right)Y_j,$$

and substituting $g_n(t,\beta )$ into $g\left(T_i\right)$ in Equation (3) yields a parameter estimation model:

$${\widehat{Y}}_i={\widehat{X}}_i\beta +{\widehat{\epsilon }}_{i}i=1,2,\cdots ,n.$$

(6)

For the variable selection method of high-dimensional partially linear models, Ref. [19] defines the least squares estimation of the coefficient vector $\beta$ of the partially linear model based on the adaptive lasso method.

$$\underset{\beta }{min}\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}^2+\lambda \sum _{j=1}^p{w}_j\left|{\beta }_j\right|.$$

This model performs well in variable selection for high-dimensional data. In order to better solve the problem of a high correlation between adjacent variables, based on the least squares idea and the fused lasso variable selection method, the penalized least squares estimate of the regression coefficient $\beta$ is defined as

$$\underset{\beta }{min}\left\{\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}^2+{\lambda }_1{∥\beta ∥}_1+{\lambda }_2\sum _{j=1}^p\left|{\beta }_j-{\beta }_{j-1}\right|\right\},$$

(7)

where ${\lambda }_1>0,{\lambda }_2>0$ is the tuning parameter.

For convenience, write ${\displaystyle \sum _{j=1}^p}\left|{\beta }_j-{\beta }_{j-1}\right|$ as ${∥D\beta ∥}_1$, where D is the matrix of differential operators:

$$D=\left[\begin{array}{c}-1\\ ⋮\\ 0\\ 0\end{array}\begin{array}{c}1\\ ⋮\\ -1\\ 0\end{array}\begin{array}{c}0\\ \ddots \\ \cdots \\ \cdots \end{array}\begin{array}{c}\cdots \\ \ddots \\ 1\\ -1\end{array}\begin{array}{c}0\\ ⋮\\ 0\\ 1\end{array}\right].$$

Therefore, Equation (7) can be written as

$$\underset{\beta }{min}\left\{\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}^2+{\lambda }_1{∥\beta ∥}_1+{\lambda }_2{∥D\beta ∥}_1\right\},$$

(8)

Introducing the auxiliary variable, Equation (8) can be written equivalently:

$$\begin{array}{c}\underset{\beta }{min}\left\{\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}^2+{\lambda }_1{∥\beta ∥}_1+{\lambda }_2{∥y∥}_1\right\},\hfill \\ s.t.D\beta =y\hfill \end{array}$$

(9)

then the solution of Equation (7) is transformed into the solution of constrained optimization problems Equation (9), that is Fused lasso Partially linear model (for short FPLM).

3. The Solution and Algorithm Design of FPLM

In this section, LADMM is used to solve Equation (9), and the algorithm design of LADMM to solve the fused lasso partially linear models is provided.

3.1. LADMM for Solving FPLM

For the optimization problem in Equation (9), the augmented Lagrange multiplier method is used to transform the constrained programming problem into an unconstrained programming problem, and then the augmented Lagrange function is

$$L_{\rho }\left(\beta ,y,\alpha \right)=\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}_2^2+{\lambda }_1{∥\beta ∥}_1+{\lambda }_2{∥y∥}_1-{\alpha }^T\left(D\beta -y\right)+\frac{\rho }{2}{∥D\beta -y∥}_2^2,$$

(10)

where $\rho >0$ is the penalty parameter.

Applying the augmented Lagrange method, the $m+1$ step iteration of Equation (9) is

$$\left\{\begin{array}{c}{\beta }^{m+1}=arg\underset{\beta \in R^p}{min}{L}_{\rho }\left(\beta ,y^m,{\alpha }^m\right),\hfill \\ y^{m+1}=arg\underset{y\in R^{p-1}}{min}{L}_{\rho }\left({\beta }^{m+1},y,{\alpha }^m\right),\hfill \\ {\alpha }^{m+1}={\alpha }^m-\rho \left(D{\beta }^{m+1}-y^{m+1}\right).\hfill \end{array}\right.$$

(11)

A new iteration point $({\beta }^{m+1},y^{m+1},{\alpha }^{m+1})$ can be obtained by solving the subproblem of Equation (11). The subproblems are solved separately as follows.

Consider the $\beta$-subproblem firstly:

$$\begin{array}{cc}\hfill {\beta }^{m+1}& =arg\underset{\beta \in R^p}{min}\left\{\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}_2^2+{\lambda }_1{∥\beta ∥}_1-{\left({\alpha }^m\right)}^T\left(D\beta -y^m\right)+\frac{\rho }{2}{∥D\beta -y^m∥}_2^2\right\}\hfill \\ \hfill & =arg\underset{\beta \in R^p}{min}\left\{\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}_2^2+{\lambda }_1{∥\beta ∥}_1+\frac{\rho }{2}{∥D\beta -y^m-{\alpha }^m/\phantom{{\alpha }^m\rho }\rho ∥}_2^2\right\}\hfill \\ \hfill & =arg\underset{\beta \in R^p}{min}\left\{{\lambda }_1{∥\beta ∥}_1+\frac{1}{2}{∥\tilde{X}\beta -{\tilde{Y}}_m∥}_2^2\right\}.\hfill \end{array}$$

(12)

where $\tilde{X}={\left({\widehat{X}}^T,\sqrt{\rho }{D}^T\right)}^T\in R^{\left(n+p-1\right)\times p}$, ${\tilde{Y}}_m=({\widehat{Y}}^T,\sqrt{\rho }{\left(y^m+{\alpha }^m/\phantom{{\alpha }^m\rho }\rho \right)}^T\in R^{n+p-1}$. Due to the presence of the non-smooth term ${\lambda }_1{∥\beta ∥}_1$ and the non-identity matrix $\tilde{X}$, the $\beta$-subproblem does not have a closed-form solution. Therefore, the approximate solution of $\beta$ can be obtained by linearizing the quadratic term $\frac{1}{2}{∥\tilde{X}\beta -{\tilde{Y}}_m∥}_2^2$ in Equation (12) and expanding this term at ${\beta }^m$ by Taylor, that is,

$$\frac{1}{2}{∥\tilde{X}\beta -{\tilde{Y}}_m∥}_2^2=\frac{1}{2}{∥\tilde{X}{\beta }^m-{\tilde{Y}}_m∥}_2^2+{\tilde{X}}^T{\left(\tilde{X}{\beta }^m-{\tilde{Y}}_m\right)}^T\left(\beta -{\beta }^m\right)+\frac{\upsilon }{2}{∥\beta -{\beta }^m∥}_2^2,$$

where $\upsilon >0$ is the parameter that controls the proximity of ${\beta }^m$. Thus, Equation (12) is

$$\begin{array}{cc}\hfill {\beta }^{m+1}& =arg\underset{\beta \in R^p}{min}\left\{{\lambda }_1{∥\beta ∥}_1+\left({\tilde{X}}^T{\left(\tilde{X}{\beta }^m-{\tilde{Y}}_m\right)}^T\left(\beta -{\beta }^m\right)\right)+\frac{\upsilon }{2}{∥\beta -{\beta }^m∥}_2^2\right\}\hfill \\ \hfill & =arg\underset{\beta \in R^p}{min}\left\{{\lambda }_1{∥\beta ∥}_1+\frac{\upsilon }{2}{∥\beta -{\beta }^m+{\tilde{X}}^T\left(\tilde{X}{\beta }^m-{\tilde{Y}}_m\right)/\phantom{{\tilde{X}}^T\left(\tilde{X}{\beta }^m-{\tilde{Y}}_m\right)\upsilon }\upsilon ∥}_2^2\right\},\hfill \end{array}$$

(13)

To obtain a closed-form solution to the subproblem, the following operator [29] is considered:

$$x^{*}=arg\underset{x}{min}\left\{\lambda {∥x∥}_1+\frac{r}{2}{∥x-\kappa ∥}_2^2\right\},$$

and the closed solution of this operator is

$$x^{*}:=S_{\lambda /\phantom{\lambda r}r}\left(\kappa \right):=sign\left(\kappa \right)\odot max\left\{\left|\kappa \right|-\frac{\lambda }{r},0\right\},$$

where r is a scalar.

Then, the solution of the $\beta$-subproblem can be given:

$$\begin{array}{cc}\hfill {\beta }^{m+1}& =S_{{\lambda }_1/\phantom{{\lambda }_1\upsilon }\upsilon }({\beta }^m-{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_k)/\phantom{{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_k)\upsilon }\upsilon )\hfill \\ \hfill & =sign\left(\gamma \right)\odot max\left\{\left|\gamma \right|-\frac{{\lambda }_1}{\upsilon },0\right\}.\hfill \end{array}$$

(14)

where $\gamma ={\beta }^m-{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_k)/\phantom{{\beta }^m-{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_k)\upsilon }\upsilon .$

For the solution of y-subproblem, it can be obtained through calculation:

$$\begin{array}{cc}\hfill y^{m+1}& =arg\underset{y\in R^{p-1}}{min}\left\{{\lambda }_2{∥y∥}_1-{\left({\alpha }^m\right)}^T\left(D{\beta }^{m+1}-y\right)+\frac{\rho }{2}{∥D{\beta }^{m+1}-y∥}_2^2\right\}\hfill \\ \hfill & =arg\underset{y\in R^{p-1}}{min}\left\{{\lambda }_2{∥y∥}_1+\frac{\rho }{2}{∥y-D{\beta }^{m+1}+{\alpha }^m/\phantom{{\alpha }^m\rho }\rho ∥}_2^2\right\}\hfill \\ \hfill & =S_{{\lambda }_2/\phantom{{\lambda }_2\rho }\rho }(D{\beta }^{m+1}-{\alpha }^m/\phantom{{\alpha }^m\rho }\rho )\hfill \\ \hfill & =sign\left(\eta \right)\odot max\left\{\left|\eta \right|-\frac{{\lambda }_2}{\rho },0\right\}.\hfill \end{array}$$

(15)

where $\eta =D{\beta }^{m+1}-{\alpha }^m/\phantom{{\alpha }^m\rho }\rho$.

Overall, applying LADMM, the solution of the $\beta$-subproblem of Equation (9) can be given by Equation (14), and the solution of the y-subproblem can be given by Equation (15). Then, the $m+1$ iteration of Equation (9) is

$$\left\{\begin{array}{c}{\beta }^{m+1}=S_{{\lambda }_1/\phantom{{\lambda }_1\upsilon }\upsilon }({\beta }^m-{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_k)/\phantom{{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_k)\upsilon }\upsilon ),\hfill \\ y^{m+1}=S_{{\lambda }_2/\phantom{{\lambda }_2\rho }\rho }(D{\beta }^{m+1}-{\alpha }^m/\phantom{{\alpha }^m\rho }\rho ),\hfill \\ {\alpha }^{m+1}={\alpha }^m-\rho \left(D{\beta }^{m+1}-y^{m+1}\right).\hfill \end{array}\right.$$

(16)

3.2. Algorithm Design of LADMM for Solving FPLM

In this section, we provide the algorithm design for solving fused lasso partially linear models using LADMM. In summary, when solving an optimization problem with constraints, the augmented Lagrangian function is constructed firstly, the problem is transformed into an unconstrained problem, and then the solution of the model is obtained by solving the iterative subproblems with LADMM. Therefore, the iterative algorithm for LADMM to solve FPLM is (Algorithm 1).

Algorithm 1 Iterative scheme of LADMM for FPLM.

Input: $X,Y,W,t$ and $tol$. Choose ${\lambda }_1>0$, ${\lambda }_2>0$, $\rho >0$, $\upsilon >\mu ({\widehat{X}}^T\widehat{X}+\rho D^{T}D)$,             Given the initial variables $({\beta }^0,y^0,{\alpha }^0)$. Output: Stop until a certain stopping criterion is met 1: Compute ${\beta }^{m+1}$ by Equation (14), 2: Compute $y^{m+1}$ by Equation (15), 3: Compute ${\alpha }^{m+1}$ by ${\alpha }^{m+1}={\alpha }^m-\rho (D{\beta }^{m+1}-y^{m+1})$, return ${\beta }^{m+1}$.

4. Convergence

In this section, we will use a variational inequality to prove that the algorithm given above converges. The Lagrange function of Equation (9) is

$$L_{\rho }\left(\beta ,y,\alpha \right)=\frac{1}{2}{∥\widehat{Y}-\widehat{X}\beta ∥}_2^2+{\lambda }_1{∥\beta ∥}_1+{\lambda }_2{∥y∥}_1-{\alpha }^T\left(D\beta -y\right),$$

(17)

where $\alpha \in R^{p-1}$ is the Lagrange multiplier.

Solving Equation (9) is equivalent to finding $({\beta }^{*},y^{*},{\alpha }^{*})\in Q^{*}:=R^p\times R^{p-1}\times R^{p-1}$, $f\left({\beta }^{*}\right)\in \partial \left({∥{\beta }^{*}∥}_1\right)$, $g\left(y^{*}\right)\in \partial \left({∥y^{*}∥}_1\right)$, such that the first-order optimality condition of Equation (17) holds:

$$\left\{\begin{array}{c}0={\lambda }_{1}f\left({\beta }^{*}\right)+{\widehat{X}}^T(\widehat{X}{\beta }^{*}-\widehat{Y})-D^T{\alpha }^{*},\hfill \\ 0=g\left(y^{*}\right)+{\alpha }^{*},\hfill \\ 0=D{\beta }^{*}-y^{*}.\hfill \end{array}\right.$$

(18)

where $\partial (·)$ represents the subdifferential operator of the non-smooth convex function. $Q^{*}$ represents all elements in Q that satisfy Equation (18).

Define ${\theta }^{*}=({\beta }^{*},y^{*},{\alpha }^{*})\in Q^{*}$, $f\left(\beta \right)\in \partial \left({∥\beta ∥}_1\right)$, $g\left(y\right)\in \partial \left({∥y∥}_1\right)$; it is possible to write Equation (18) as a variational inequality:

$$VI(Q,F):(\theta -{\theta }^{*})F\left({\theta }^{*}\right)\ge 0\forall \theta \in Q,$$

where

$$\theta =\left(\begin{array}{c}\beta \\ y\\ \alpha \end{array}\right)F\left(\theta \right)=\left(\begin{array}{c}{\lambda }_{1}f\left(\beta \right)+{\widehat{X}}^T(\widehat{X}\beta -\widehat{Y})-D^T\alpha \\ g\left(y\right)+\alpha \\ D\beta -y\end{array}\right).$$

(19)

The following analysis requires the use of positive definite matrices G, which are given by

$$G=\left(\begin{array}{ccc}\upsilon I_p-{\tilde{X}}^T\tilde{X}& 0& 0\\ 0& \rho I_p& 0\\ 0& 0& \frac{1}{\rho }{I}_p\end{array}\right)$$

(20)

where $\tilde{X}={({\widehat{X}}^T,\sqrt{\rho }{D}^T)}^T$. $\upsilon >\mu ({\widehat{X}}^T\widehat{X}+\rho D^{T}D)$ guarantees the positive determinism of G. $\mu (·)$ represents the spectral radius of the matrix.

The following lemma describes the $m+1$-th iteration of the algorithm as the $VI(Q,F)$ problem used to analyze the convergence of the algorithm.

Lemma 1. 

Let $\left\{{\theta }^m\right\}$ be the sequence produced by the algorithm; then, there is

$${(\theta {}^{\prime }-{\theta }^{m+1})}^T(F\left({\theta }^{m+1}\right)+R(y^m-y^{m+1})-G({\theta }^m-{\theta }^{m+1}))\ge 0\forall {\theta }^{{}^{\prime }}\in Q$$

(21)

where

$$R=\left(\begin{array}{c}-\rho D\\ \rho I_{p-1}\\ 0_{p-1}\end{array}\right).$$

Proof. 

By $\tilde{X}={({\widehat{X}}^T,\sqrt{\rho }{D}^T)}^T$ and ${\alpha }^{m+1}={\alpha }^m-\rho (D{\beta }^{m+1}-y^{m+1})$, we have

$$\begin{array}{cc}\hfill {\tilde{X}}^T(\tilde{X}{\beta }^{m+1}-{\tilde{Y}}_m)& =({\widehat{X}}^T\widehat{X}+\rho D^{T}D){\beta }^{m+1}-({\widehat{X}}^T\widehat{Y}+\rho D^T{y}^m+D^T{\alpha }^m)\hfill \\ \hfill & ={\widehat{X}}^T(\widehat{X}{\beta }^{m+1}-\widehat{Y})-D^T{\alpha }^{m+1}-D^T\rho (y^m-y^{m+1}).\hfill \end{array}$$

(22)

From Equations (13) and (15), it can be seen that the iterative scheme of iteration Equation (11) is equivalent to finding $f\left({\beta }^{m+1}\right)\in \partial \left({∥{\beta }^{m+1}∥}_1\right)$, ${\theta }^{m+1}=({\beta }^{m+1},y^{m+1},{\alpha }^{m+1})\in Q$, $g\left(y^{m+1}\right)\in \partial \left({∥y^{m+1}∥}_1\right)$, satisfying

$$\left\{\begin{array}{c}0={\lambda }_{1}f\left({\beta }^{m+1}\right)+{\tilde{X}}^T(\tilde{X}{\beta }^m-{\tilde{Y}}_m)+\upsilon ({\beta }^{m+1}-{\beta }^m),\hfill \\ 0={\lambda }_{2}g\left(y^{m+1}\right)+\rho (y^{m+1}-D{\beta }^{m+1}+{\alpha }^m/\phantom{{\alpha }^m\rho }\rho ),\hfill \\ 0=D{\beta }^{m+1}-y^{m+1}-({\alpha }^m-{\alpha }^{m+1})/\phantom{({\alpha }^m-{\alpha }^{m+1})\rho }\rho .\hfill \end{array}\right.$$

(23)

Combining Equation (22) and the positive definite matrices G, Equation (23) can easily be written as Equation (21). □

The following lemma can be easily derived from Lemma 1.

Lemma 2. 

Let $\left\{{\theta }^m\right\}$ be the sequence produced by the algorithm; then, for any ${\theta }^{*}\in Q^{*}$,

$$\begin{array}{cc}\hfill {({\theta }^m-{\theta }^{*})}^{T}G({\theta }^m-{\theta }^{m+1})\ge & {({\theta }^m-{\theta }^{m+1})}^{T}G({\theta }^m-{\theta }^{m+1})\hfill \\ & -{({\alpha }^m-{\alpha }^{m+1})}^T(y^m-y^{m+1}).\hfill \end{array}$$

(24)

From Lemmas 1 and 2, it can be seen that the sequence is contracted for the solution set $Q^{*}$.

Lemma 3. 

Let $\left\{{\theta }^m\right\}$ be the sequence produced by the algorithm; then, for any ${\theta }^{*}\in Q^{*}$,

$${∥{\theta }^{m+1}-{\theta }^{*}∥}_G^2\le {∥{\theta }^m-{\theta }^{*}∥}_G^2-{∥{\theta }^m-{\theta }^{m+1}∥}_G^2$$

(25)

Lemma 3 indicates that the sequence $\left\{{\theta }^m\right\}$ is contracted for the solution set $Q^{*}$; the following corollary can be obtained from Lemma 3.

Corollary 1. 

Let $\left\{{\theta }^m\right\}$ be the sequence produced by the algorithm; then,

1.

$\underset{m\to \infty }{lim}{∥{\theta }^m-{\theta }^{m+1}∥}_G=0.$

2.

The sequence $\left\{{\theta }^m\right\}$ is bounded.

3.

For any ${\theta }^{*}\in Q^{*}$, the sequence $\left\{{∥{\theta }^m-{\theta }^{*}∥}_G\right\}$ is monotonic and does not increase.

Theorem 1. 

For any $\rho >0$, $\upsilon >\mu ({\widehat{X}}^T\widehat{X}+\rho D^{T}D)$, given the initial point $({\beta }^0,y^0,{\alpha }^0)\in Q$, the sequence $\left\{{\theta }^m=({\beta }^m,y^m,{\alpha }^m)\right\}$ generated by the algorithm converges to ${\theta }^{\infty }=({\beta }^{\infty },y^{\infty },{\alpha }^{\infty })$, where $({\beta }^{\infty },y^{\infty })$ is the solution of Equation (9).

Proof. 

According to Corollary 1, there is

$$\underset{m\to \infty }{lim}{∥{\beta }^m-{\beta }^{m+1}∥}_G=0,\underset{m\to \infty }{lim}{∥y^m-y^{m+1}∥}_G=0,\underset{m\to \infty }{lim}{∥{\alpha }^m-{\alpha }^{m+1}∥}_G=0.$$

Additionally the sequence $\left\{{\theta }^m\right\}$ has at least one cluster point, noted as ${\theta }^{\infty }=({\beta }^{\infty },y^{\infty },{\alpha }^{\infty })$. Let $\left\{{\theta }^{m_i}\right\}$ be the subsequence that converges to ${\theta }^{\infty }$; then, we have

$${\beta }^{m_i}\to {\beta }^{\infty },{\mathrm{y}}^{m_i}\to y^{\infty },{\alpha }^{m_i}\to {\alpha }^{\infty },$$

(26)

$$\underset{i\to \infty }{lim}{∥{\beta }^{m_i}-{\beta }^{m_i+1}∥}_G=0,\underset{i\to \infty }{lim}{∥y^{m_i}-y^{m_i+1}∥}_G=0,\underset{i\to \infty }{lim}{∥{\alpha }^{m_i}-{\alpha }^{m_i+1}∥}_G=0.$$

(27)

It is shown below that the cluster point ${\theta }^{\infty }$ satisfies the optimality condition Equation (18). According to Lemma 1 and Equation (27), for any ${\theta }^{{}^{\prime }}\in Q$,

$$\underset{i\to \infty }{lim}{({\theta }^{{}^{\prime }}-{\theta }^{m_i})}^{T}F\left({\theta }^{m_i}\right)\ge 0,$$

From Equation (26), the above inequality becomes

$${({\theta }^{{}^{\prime }}-{\theta }^{\infty })}^{T}F\left({\theta }^{\infty }\right)\ge 0,\forall {\theta }^{{}^{\prime }}\in Q.$$

so the cluster point ${\theta }^{\infty }$ satisfies Equation (18), that is, ${\theta }^{\infty }\in Q^{*}$. From Corollary 1, for any $m\ge 0$,

$${∥{\theta }^{m+1}-{\theta }^{\infty }∥}_G\le {∥{\theta }^m-{\theta }^{\infty }∥}_G.$$

In summary, the sequence $\left\{{\theta }^m\right\}$ has a unique convergence ${\theta }^{\infty }$, so the sequence $\left\{{\theta }^m\right\}$ converges to ${\theta }^{\infty }$. This proof is completed. □

5. Asymptotic Properties

In this section, consider the asymptotic properties of fused lasso $\widehat{\beta }$ estimators for partially linear model parameters $\beta$.

Theorem 2. 

If ${\lambda }^{\left(N\right)}/\phantom{{\lambda }^{\left(N\right)}\sqrt{N}}\sqrt{N}\to {\lambda }^{\left(0\right)}$, and $V=\underset{N\to \infty }{lim}\left(\frac{1}{N}{\displaystyle \sum _{i=1}^N}{\widehat{X}}_i{{\widehat{X}}_i}^T\right)$ is non-singular, then

$$\sqrt{N}({\beta }^{*}-\beta )\underset{d}{\to }argmin\mathrm{Z}\left(\mathrm{u}\right).$$

where

$$\begin{array}{cc}\hfill Z& \left(u\right)=-2u^{T}M+u^{T}Vu+{\lambda }_1^{\left(0\right)}\sum _{j=1}^p\left\{u_j\mathrm{sgn}\left({\beta }_j\right)I({\beta }_j\ne 0)+\left|u_j\right|I({\beta }_j=0)\right\}\hfill \\ \hfill & +{\lambda }_2^{\left(0\right)}\sum _{j=2}^p\left\{(u_j-u_{j-1})\mathrm{sgn}({\beta }_j-{\beta }_{j-1})I({\beta }_j\ne {\beta }_{j-1})+\left|u_j-u^{j-1}\right|I({\beta }_j={\beta }_{j-1})\right\}.\hfill \end{array}$$

where $M\sim N(0,{\sigma }^{2}V)$.

Proof. 

Define $Z_N\left(u\right)$ by

$$\begin{array}{cc}\hfill Z_N\left(u\right)=& \sum _{i=1}^N\left\{{({\epsilon }_i-u^T{\widehat{X}}_i/\phantom{u^T{\widehat{X}}_i\sqrt{N}}\sqrt{N})}^2-{\epsilon }_i^2\right\}+{\lambda }_1^{\left(N\right)}\sum _{j=1}^p(\left|{\beta }_j+u_j/\phantom{u_j\sqrt{N}}\sqrt{N}\right|-\left|{\beta }_j\right|)\hfill \\ \hfill & +{\lambda }_2^{\left(N\right)}\sum _{j=1}^{p-1}(\left|{\beta }_j-{\beta }_{j-1}+(u_j-u_{j-1})/\phantom{(u_j-u_{j-1})\sqrt{N}}\sqrt{N}\right|-\left|{\beta }_j-{\beta }_{j-1}\right|).\hfill \end{array}$$

where $u={\left\{u_1,\cdots ,u_p\right\}}^T$. We can determine that $Z_N$ is smallest at $\sqrt{N}({\beta }^{*}-\beta )$. Note that

$$\sum _{i=1}^N\left\{{({\epsilon }_i-u^T{\widehat{X}}_i/\phantom{u^T{\widehat{X}}_i\sqrt{N}}\sqrt{N})}^2-{\epsilon }_i^2\right\}\underset{d}{\to }-2u^{T}M+u^{T}Vu,$$

with finite dimensional convergence holding trivially. In addition, we have

$$\begin{array}{c}{\lambda }_1^{\left(N\right)}\sum _{j=1}^p(\left|{\beta }_j+u_j/\phantom{u_j\sqrt{N}}\sqrt{N}\right|-\left|{\beta }_j\right|)\to {\lambda }_1^{\left(0\right)}\sum _{j=1}^p\left\{u_j\mathrm{sgn}\left({\beta }_j\right)I({\beta }_j\ne 0)+\left|u_j\right|I({\beta }_j=0)\right\},\\ {\lambda }_2^{\left(N\right)}\sum _{j=1}^{p-1}\left\{\left|{\beta }_j-{\beta }_{j-1}+(u_j-u_{j-1})/\phantom{(u_j-u_{j-1})\sqrt{N}}\sqrt{N}\right|-\left|{\beta }_j-{\beta }_{j-1}\right|\right\}\hfill \\ \to {\lambda }_2^{\left(0\right)}\sum _{j=2}^p\left\{(u_j-u_{j-1})\mathrm{sgn}({\beta }_j-{\beta }_{j-1})I({\beta }_j\ne {\beta }_{j-1})+\left|u_j-u^{j-1}\right|I({\beta }_j={\beta }_{j-1})\right\},\hfill \end{array}$$

Thus, $Z_N\left(u\right)\underset{d}{\to }Z\left(u\right)$, and because of the convexity of $Z_N\left(u\right)$ and the fact that $Z\left(u\right)$ has a unique minimum,

$$argmin{Z}_N\left(u\right)=\sqrt{N}({\beta }^{*}-\beta )\underset{d}{\to }argmin\mathrm{Z}\left(\mathrm{u}\right).$$

□

Theorem 2 shows that if ${\lambda }^{\left(N\right)}=o\left(\sqrt{N}\right)$, then $\sqrt{N}({\beta }^{*}-\beta )\underset{d}{\to }{V}^{-1}M\sim N(0,{\sigma }^2{V}^{-1})$. If ${\lambda }^{\left(N\right)}=O\left(\sqrt{N}\right)$, then the estimated $\beta$ of the non-zero coefficient ${\beta }_c^{*}$ of ${\beta }_c$ is not $\sqrt{N}$-Consistent.

6. Numerical Experiments

In this section, two kinds of data are used to demonstrate that the model proposed in this paper performs better in high-dimensional data with a high correlation between adjacent variables. The experiments were divided into two groups, one with general high-dimensional data and the other with high-dimensional simulated data with a high correlation between adjacent variables. The LADMM algorithm is used to solve the fused lasso partially linear model, the ADMM algorithm is used to solve the adaptive lasso partially linear model, and numerical comparisons are made to test the validity of the models. All calculations are performed on the Microsoft Windows 10 operating system and MATLAB R2016a.

When estimating the nonparametric part with a weight function, the selection of window width h directly determines the estimation effect of the nonparametric part, which in turn determines the estimation effect of the parametric part. The theoretical optimal value of window width [30] is $h=c{n}^{-1/\phantom{-15}5}$; c is constant, and $c=1/2$ is taken in this paper. The kernel function takes the Gaussian kernel $K\left(u\right)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{u}^2}$.

6.1. Numerical Experimental Results of General High-Dimensional Data

The section deals with general high-dimensional data, denoted as numerical experiment 1. A data set with a sample size n and feature p is generated. X obeys a p-dimensional multivariate normal distribution, that is, $X\sim N(0,\sum ),\sum =0.5^{\left|j-i\right|}$, where i, j are the i-th and j-th components of the covariance, respectively, and parameters are given by $\beta ={(1,2,0.5,-1,0,...,0)}^T$.The random error term $\epsilon \sim N(0,{\sigma }^2)$,where $\sigma$ takes 0.5, 1, 2. The variable t follows a uniform distribution on interval $[0,1]$, that is, $t\sim U(0,1)$. the measurable function takes $g\left(t\right)=cos\left(2\pi t\right)$, and the response variable is generated by $Y=X\beta +g\left(t\right)+\epsilon$. In addition, for the parameter ${\lambda }_1=1,{\lambda }_2=0.1,\rho =1$ in the fused lasso model, the parameter $\lambda =0.1,\rho =0.01$ in the adaptive lasso model, the sample size is set to $n=100$, and the dimension takes a different value.

Simulation experiments were conducted for the Fused Lasso partially linear model and the adaptive Lasso partially linear model using the high-dimensional data with highly correlated to adjacent variables and parameter generated by the above method, respectively, and the main results are shown in

Table 1. Mean squared error processed by the two methods.

n	p	$\mathit{\sigma }$	MSEF	MSEA	$\mathit{\sigma }$	MSEF	MSEA	$\mathit{\sigma }$	MSEF	MSEA
100	180	0.5	0.0025	0.0027	1	0.0050	0.0030	2	0.0101	0.0043
	280		0.0029	0.0023		0.0064	0.0024		0.0134	0.0030
	380		0.0049	0.0024		0.0104	0.0023		0.0215	0.0027
	480		0.0067	0.0021		0.0143	0.0021		0.0296	0.0024
	580		0.0069	0.0024		0.0143	0.0027		0.0294	0.0035
	680		0.0084	0.0021		0.0177	0.0022		0.0365	0.0026
	780		0.0088	0.0021		0.0191	0.0020		0.0399	0.0023
	880		0.0149	0.0024		0.0326	0.0023		0.0683	0.0024
	980		0.0216	0.0022		0.0446	0.0023		0.0913	0.0025
	1080		0.0038	0.0023		0.0085	0.0022		0.0180	0.0025

.

The indicator for evaluating the quality of a result is the mean squared error, that is, $MSE={∥{\beta }^{*}-\beta ∥}^2$, where MSEF represents the mean squared error of the fused lasso method, and MSEA represents the mean squared error of the adaptive lasso method.

The results in Table 1 show that the mean squared error of the fused lasso method is slightly higher than that of the adaptive lasso method under general high-dimensional data, but the mean squared errors of both methods are small. The mean squared error increases with the increase in $\sigma$ but is small overall, indicating that both methods are ideal for processing general high-dimensional data.

6.2. Numerical Experimental Results of High-Dimensional Data Highly Correlated to Adjacent Variables

In this section, simulated data that are highly correlated with adjacent variables will be processed separately using the fusion lasso method and the adaptive lasso method. Firstly, a Gaussian matrix $X_j$ with a unit column norm is randomly generated. In order to generate solutions with sparsity and linear relationships with adjacent variables, the dimension p is divided into 80 groups, 10 of them are randomly chosen, and then sample sets of bases $s=i,i=3,5,7,9,11,13,15$ are randomly selected, respectively. The parameter $\beta$ is generated as follows [14]:

$${\beta }_i=\left\{\begin{array}{c}{\xi }_i(1+\left|c_i\right|),ifi\in S;\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.$$

where $c_i$ follows the $N(0,1)$ distribution, ${\xi }_i$ is randomly selected in the set $\left\{-1,1\right\}$, and the random error term $\sigma$=$0.01,0.5,1$. The settings for variable t, measurable function $g\left(t\right)$, and response variable Y are consistent with the method in numerical experiment 1. To compare the effects of the two methods, take $(n,p,s)=(100,80i,i)$, $i=3,5,7,9,11,13,15$. For parameters $\lambda =0.1,\rho =0.1$ in the adaptive lasso model, and parameters ${\lambda }_1=1,{\lambda }_2=0.01,\rho =1$ in the fused lasso model.

Due to the consistency of kernel function estimation methods for nonparametric parts, only the nonparametric estimation effect at $p=1200,\sigma =1$ is shown here, as shown in Figure 1:

From Figure 1, it can be seen that using kernel functions to estimate nonparametric parts has a better effect.

Simulation experiments were conducted for the fused lasso partially linear model and the adaptive lasso partially linear model using the high-dimensional data and parameters generated by the above methods, respectively, and the main results are shown in

Table 2. Mean squared error processed by the two methods.

n	p	$\mathit{\sigma }$	MSEF	MSEA	$\mathit{\sigma }$	MSEF	MSEA	$\mathit{\sigma }$	MSEF	MSEA
100	240	0.01	0.0047	0.0094	0.5	0.0047	0.0112	1	0.0047	0.0134
	400		0.0047	0.0100		0.0047	0.0112		0.0046	0.0127
	560		0.0044	0.0100		0.0044	0.0109		0.0045	0.0115
	720		0.0045	0.0092		0.0045	0.0110		0.0046	0.0120
	880		0.0045	0.0089		0.0045	0.0094		0.0045	0.0120
	1040		0.0046	0.0085		0.0046	0.0086		0.0047	0.0103
	1200		0.0046	0.00131		0.0046	0.0143		0.0047	0.0157

.

As can be seen from Table 2, the mean squared error of the fused lasso is smaller than that of the adaptive lasso when the neighboring variables are highly correlated, and as the error interference increases, the difference between fused lasso and adaptive lasso becomes more significant. This shows that when dealing with data with a high correlation of adjacent variables, the fused lasso effect is more prominent. Figure 2, Figure 3 and Figure 4 are the parameter estimates when $\sigma =0.01,0.5,1$, respectively.

It can also be seen from Figure 2, Figure 3 and Figure 4 that the fused lasso method performs better than the adaptive lasso in high-dimensional data with high correlation between adjacent variables, which is consistent with the existing conclusions.

In conclusion, the numerical experiments show that the fused lasso partially linear model proposed in this paper has a good fitting effect in general high-dimensional data, and the fitting effect is better than the adaptive lasso in high-dimensional data where the adjacent variables are highly correlated, so it has a certain applicability.

7. Example Analysis

In this section, the method proposed in this paper is applied to the actual data of worker wage analysis provided by the 1985 Current Population Survey (CPS85) [31]. These indicators of data include both quantitative and categorical data, so they are representative. In addition, these data have been studied by other scholars, which facilitates comparison. The data contain 534 observation samples and 11 variables, providing information on worker wages and other attribute of workers, such as sector, occupation, south, race, age, marital status, sex, number of years of education, union, and number of years of work experience. Among these, there is not necessarily a simple linear relationship between number of years of work experience and worker wages, so this is selected as a nonparametric variable $T_i$, and other worker attributes are used as parameter variables $X_i$.

In the experiment, 70 percent of the sample was selected as the training set, and the remaining 30 percent was used as the prediction set for predicting worker wages. The prediction effect is evaluated using median absolute error (MAE) and standard error (SE) as indicators, and the expression is

$$MSE=medain\left\{\left|y_1-y_1^{*}\right|,\left|y_2-y_2^{*}\right|,\cdots ,\left|y_n-y_n^{*}\right|\right\},$$

$$SE=\sqrt{\frac{\sum {(y_i-y_i^{*})}^2}{n}}.$$

The smaller the values of MAE and SE, the better the prediction performance. The results are shown in

Table 3. The SE and MAE of worker wage prediction.

Variable	Description	SE	MAE
Sector	Manufacturing = 1, other = 0	0.1909	0.4813
Occupation	Prof = 1, sales = 1, Service = 1, Management = 1, Other = 0	0.2040	0.7595
South	South = 1, Other = 0	0.1910	0.4880
Race	White = 1, Other = 0	0.1903	0.4800
Age	Age	0.1983	0.5651
Marry	Married = 1, Single = 0	0.1902	0.4944
Sex	Female = 1, Male = 0	0.1901	0.5076
Education	Number of years of education	0.2043	0.6357
Union	Union = 1, Not union = 0	0.1899	0.4916
Total	All variables	0.2445	1.0818

:

The first few rows of Table 3 display the SE and MAE for predicting worker wages based on various worker characteristics, while the last row shows the overall SE and MAE for predicting worker wages using all worker feature variables. From the results in Table 3, it can be seen that the SE and MAE values predicted using a single worker characteristic variable are very small, and when all variables are combined to predict worker wages, despite the correlation between age and marital status, race, and south, the resulting SE and MAE values are still very small. This indicates that the error between the predicted and actual wages of workers is relatively small, so the method proposed in this article is relatively effective.

8. Conclusions

In this paper, a fused lasso partially linear model is proposed to solve the problem of adjacent data being highly correlated, the LADMM algorithm is designed to estimate the parameters of the model, the algorithm framework is presented, the convergence of the algorithm is analyzed, and the asymptotic property of the model is proved. Numerical experiments were conducted using general high-dimensional data and high-dimensional data with high correlation between adjacent variables, and the method proposed in this paper was applied to the practical problem of worker wage prediction. The results showed that the fused lasso partially linear model proposed in this paper has a certain applicability. This model is an extension of the partially linear model, expanding the applicability of the semiparametric model.