Multi-Task Learning for Compositional Data via Sparse Network Lasso

Abstract

Multi-task learning is a statistical methodology that aims to improve the generalization performances of estimation and prediction tasks by sharing common information among multiple tasks. On the other hand, compositional data consist of proportions as components summing to one. Because components of compositional data depend on each other, existing methods for multi-task learning cannot be directly applied to them. In the framework of multi-task learning, a network lasso regularization enables us to consider each sample as a single task and construct different models for each one. In this paper, we propose a multi-task learning method for compositional data using a sparse network lasso. We focus on a symmetric form of the log-contrast model, which is a regression model with compositional covariates. Our proposed method enables us to extract latent clusters and relevant variables for compositional data by considering relationships among samples. The effectiveness of the proposed method is evaluated through simulation studies and application to gut microbiome data. Both results show that the prediction accuracy of our proposed method is better than existing methods when information about relationships among samples is appropriately obtained.

1. Introduction

Multi-task learning is a statistical methodology that assumes a different model for each task and jointly estimates these models. By sharing the common information between them, the generalization performance of estimation and prediction tasks is improved [1]. Multi-task learning has been used in various fields of research, such as computer vision [2], natural language processing [3], and life sciences [4]. In life sciences, the risk factors may vary from patient to patient [5], and a model that is common to all patients cannot sufficiently extract general risk factors. In multi-task learning, each patient can be considered as a single task, and different models are built for each patient to extract both patient-specific and common factors for the disease [6]. Localized lasso [7] is a method that performs multi-task learning using network lasso regularization [8]. By treating each sample as a single task, localized lasso simultaneously performs multi-task learning and clustering in the framework of a regression model.

On the other hand, compositional data, which consist of the individual proportions of a composition, are used in the fields of geology and life sciences for microbiome analysis. Compositional data are constrained to always take positive values summing to one. Due to these constraints, it is difficult to apply existing multi-task learning methods to compositional data. In the field of microbiome analysis, studies on gut microbiomes [9,10] have suggested that there are multiple types of gut microbiome clusters that vary from individual to individual [11]. In the case of such data where multiple clusters may exist, it is difficult to extract sufficient information using existing regression models for compositional data.

In this paper, we propose a multi-task learning method for compositional data, focusing on the network lasso regularization and the symmetric form of the log-contrast model [12], which is a linear regression model with compositional covariates. The symmetric form is extended to the locally symmetric form in which each sample has a different regression coefficient vector. These regression coefficient vectors are clustered by the network lasso regularization. Furthermore, because the dimensionality of features in compositional data has been increasing, in particular in microbiome analysis [13], we use an ${\ell }_1$-regularization [14] to perform variable selection. The advantage of using ${\ell }_1$-regularization is being able to perform variable selection even if the number of parameters exceeds the sample size. In addition, ${\ell }_1$-regularization is formulated by convex optimization, which leads to feasible computation, while classical subset selection is not. The estimation of the parameters included in the model is performed using an estimation algorithm based on the alternating direction method of multipliers [15], because the model includes non-differentiable points in the ${\ell }_1$-regularization term and zero-sum constraints on the parameters. The constructed model includes regularization parameters, which are determined by cross-validation (CV).

The remainder of this paper is organized as follows. Section 2 introduces multi-task learning based on a network lasso. In Section 3, we describe the regression models for compositional data. We propose a multi-task learning method for compositional data and its estimation algorithm in Section 4. In Section 5, we discuss the effectiveness of the proposed method through Monte Carlo simulations. An application to gut microbiome data is presented in Section 6. Finally, Section 7 summarizes this paper and discusses future work.

2. Multi-Task Learning Based on a Network Lasso

Suppose that we have n observed p-dimensional data $\{{\mathit{x}}_i;i=1,\dots ,n\}$ and n observed data for the response variable $\{y_i;i=1,\dots ,n\}$ and that these pairs $\{(y_i,{\mathit{x}}_i),i=1,\dots ,n\}$ are given independently. The graph $R=R^T\in {\mathbb{R}}^{n\times n}$ is also given, where ${\left(R\right)}_{ij}=r_{i,j}\ge 0$ represents the relationship between the sample pair $(y_i,{\mathit{x}}_i)$ and $(y_j,{\mathit{x}}_j)$, and thus the diagonal components are zero.

We consider the following linear regression model:

$$y_i={\mathit{x}}_i^T{\mathit{w}}_i+{ϵ}_i,i=1,\dots ,n,$$

(1)

where ${\mathit{w}}_i={(w_{i1},\dots ,w_{ip})}^T\in {\mathbb{R}}^p$ is the p-dimensional regression coefficient vector for sample ${\mathit{x}}_i$, and ${ϵ}_i$ is an error term distributed as $N(0,{\sigma }^2)$ independently. Note that we exclude the intercept $w_0$ from the model, because we assume the centered response and the standardized explanatory variables. Model (1) comprises a different model for each sample. In classical regression models, the regression coefficient vectors are assumed to be identical (i.e., ${\mathit{w}}_1={\mathit{w}}_2=\cdots ={\mathit{w}}_n$).

For Model (1), we consider the following minimization problem:

$$\underset{{\mathit{w}}_i\in {\mathbb{R}}^p,i=1\dots ,n}{min}\left\{\sum _{i=1}^n{(y_i-{\mathit{x}}_i^T{\mathit{w}}_i)}^2+\lambda \sum _{m>l}^n{r}_{m,l}{\parallel {\mathit{w}}_m-{\mathit{w}}_l\parallel }_2\right\},$$

(2)

where $\lambda (>0)$ is a regularization parameter. The second term in (2) is the network lasso regularization term [8]. For coefficient vectors ${\mathit{w}}_m$ and ${\mathit{w}}_l$, the network lasso regularization term induces ${\mathit{w}}_m={\mathit{w}}_l$. If these vectors are estimated to be the same, then the m-th and l-th samples are interpreted as belonging to the same cluster. In the framework of multi-task learning, the minimization problem (2) considers one sample as one task by setting a coefficient vector for each sample. This allows us to extract the information of the regression coefficient vectors separately for each task. In addition, by clustering the regression coefficient vectors using the network lasso regularization term, we can extract the common information among tasks.

Yamada et al. [7] proposed the localized lasso for minimization problem (2) by adding an ${\ell }_{1,2}$-norm regularization term [16] as follows:

$$\underset{{\mathit{w}}_i\in {\mathbb{R}}^p,i=1,\dots ,n}{min}\left\{\sum _{i=1}^n{(y_i-{\mathit{x}}_i^T{\mathit{w}}_i)}^2+{\lambda }_1\sum _{m>l}^n{r}_{m,l}\parallel {\mathit{w}}_m-{\mathit{w}}_l{\parallel }_2+{\lambda }_2\sum _{i=1}^n{\parallel {\mathit{w}}_i\parallel }_1^2\right\}.$$

(3)

The ${\ell }_{1,2}$-norm regularization term induces group structure and intra-group level sparsity: several regression coefficients in a group are estimated to be zero, but at least one is estimated to be non-zero by squaring over the ${\ell }_1$-norm. In the localized lasso, each regression coefficient vector ${\mathit{w}}_i$ is treated as a group in order to remain ${\mathit{w}}_i\ne \mathbf{0}$. The localized lasso is used for multi-task learning and variable selection.

3. Regression Modeling for Compositional Data

The p-dimensional compositional data $\mathit{x}={(x_1,\dots ,x_p)}^T$ are defined as proportional data in the simplex space:

$${\mathbb{S}}^{p-1}=\left\{(x_1,\dots ,x_p):x_j>0(j=1,\dots ,p),\sum _{j=1}^p{x}_j=1\right\}.$$

(4)

This structure imposes dependence between the features of the compositional data. Thus, statistical methods defined for spaces of real numbers cannot be applied [17]. To overcome this problem, Aitchison and Bacon-Shone [12] proposed the log-contrast model, which is a linear regression model with compositional covariates.

Suppose that we have n observed p-dimensional compositional data $\{{\mathit{x}}_i;i=1,\dots ,n\}$ and n objective variable data $\{y_i;i=1,\dots ,n\}$ and these pairs $\{(y_i,{\mathit{x}}_i),i=1\dots ,n\}$ are given independently. The log-contrast model is represented as follows:

$$y_i=\sum _{j=1}^{p-1}log\frac{x_{ij}}{x_{ip}}{\beta }_j+{ϵ}_i,i=1,\dots ,n,$$

(5)

where $\mathbf{\beta }={({\beta }_1,\dots ,{\beta }_{p-1})}^T\in {\mathbb{R}}^{p-1}$ is a regression coefficient vector. Because the model uses an arbitrary variable as a reference for all other variables, the solution changes depending on the selection of the reference. By introducing ${\beta }_p=-{\sum }_{j=1}^{p-1}{\beta }_j$, the log-contrast model is equivalently expressed in symmetric form as:

$$y_i={\mathit{z}}_i^T\mathbf{\beta }+{ϵ}_i,\mathrm{s}.\mathrm{t}.\sum _{j=1}^p{\beta }_j=0,i=1,\dots ,n,$$

(6)

where ${\mathit{z}}_i={(log{x}_{i1},\dots ,log{x}_{ip})}^T$, and $\mathbf{\beta }={({\beta }_1,\dots ,{\beta }_p)}^T\in {\mathbb{R}}^p$ is a regression coefficient vector. Lin et al. [13] proposed the minimization problem to select relevant variables in symmetric form by adding an ${\ell }_1$-regularization term [14]:

$$\underset{\mathbf{\beta }\in {\mathbb{R}}^p}{min}\left\{\sum _{i=1}^n{(y_i-{\mathit{z}}_i^T\mathbf{\beta })}^2+\lambda {\parallel \mathbf{\beta }\parallel }_1\right\},\mathrm{s}.\mathrm{t}.\sum _{j=1}^p{\beta }_j=0.$$

(7)

Other models that extend this symmetric form of the problem have also been proposed [18,19,20,21].

4. Proposed Method

In this section, we propose a multi-task learning method for compositional data based on the network lasso and the symmetric form of the log-contrast model.

4.1. Model

We consider the locally symmetric form of the log-contrast model:

$$y_i={\mathit{z}}_i^T{\mathit{w}}_i+{ϵ}_i,\mathrm{s}.\mathrm{t}.\sum _{j=1}^p{w}_{ij}=0,i=1,\dots ,n,$$

(8)

where ${\mathit{z}}_i={(log{x}_{i1},\dots ,log{x}_{ip})}^T$, and ${\mathit{w}}_i={(w_{i1},\dots ,w_{ip})}^T$ is the regression coefficient vector for i-th sample of compositional data ${\mathit{x}}_i$. For Model (8), we consider the following minimization problem:

$$\begin{array}{c}\hfill \underset{{\mathit{w}}_i\in {\mathbb{R}}^p,i=1\dots ,n}{min}\left\{\sum _{i=1}^n{(y_i-{\mathit{z}}_i^T{\mathit{w}}_i)}^2+{\lambda }_1\sum _{m>l}^n{r}_{m,l}\parallel {\mathit{w}}_m-{\mathit{w}}_l{\parallel }_2+{\lambda }_2\sum _{i=1}^n{\parallel {\mathit{w}}_i\parallel }_1\right\},\\ \hfill \mathrm{s}.\mathrm{t}.\sum _{j=1}^p{w}_{ij}=0,i=1,\dots ,n,\end{array}$$

(9)

where ${\lambda }_1,{\lambda }_2(>0)$ are regularization parameters. The second term is the network lasso regularization term, which performs the clustering of the regression coefficient vectors. The third term is the ${\ell }_1$-regularization term [14]. This term is interpreted as a special case of the ${\ell }_{1,2}$-regularization term used in Model (3). Unlike the ${\ell }_1$-regularization term, it is difficult to optimize the ${\ell }_{1,2}$-regularization directly, because it does not have a closed form of the updates. To construct the estimation algorithm that performs variable selection and preserves the constraints for regression coefficient vectors simultaneously, we employ the ${\ell }_1$-regularization term. Since variable selection is performed by the ${\ell }_1$-regularization term, we refer to the combination of the second term and the third term as sparse network lasso after sparse group lasso [22].

For Model (8), when a new data point ${\mathit{z}}_{i^{*}}$ is obtained after the estimation, there is no corresponding regression coefficient vector ${\mathit{w}}_{i^{*}}$ for ${\mathit{z}}_{i^{*}}$. Thus, it is necessary to estimate the coefficient vector for predicting the response. Hallac et al. [8] proposed solving the following minimization problem:

$$\underset{{\mathit{w}}_{i^{*}}\in {\mathbb{R}}^p}{min}\left\{\sum _{i=1}^n{r}_{i^{*},i}{\parallel {\mathit{w}}_{i^{*}}-{\widehat{\mathit{w}}}_i\parallel }_2\right\},$$

(10)

where ${\widehat{\mathit{w}}}_i$ is the estimated regression coefficient vector for the i-th sample. This problem is also known as the Weber problem. The solution of this problem is interpreted as the weighted geometric median of ${\widehat{\mathit{w}}}_i$. For our proposed method, we consider solving the constrained Weber problem with the zero-sum constraint in the form:

$$\underset{{\mathit{w}}_{i^{*}}\in {\mathbb{R}}^p}{min}\left\{\sum _{i=1}^n{r}_{i^{*},i}{\parallel {\mathit{w}}_{i^{*}}-{\widehat{\mathit{w}}}_i\parallel }_2\right\},\mathrm{s}.\mathrm{t}.\sum _{j=1}^p{w}_{i^{*}j}=0.$$

(11)

4.2. Estimation Algorithm

To construct the estimation algorithm for the proposed method, we rewrite minimization problem (9) as follows:

$$\begin{array}{c}\hfill \underset{{\mathit{w}}_i,{\mathit{a}}_m,{\mathit{b}}_i\in {\mathbb{R}}^p,i=1\dots ,n}{min}\left\{\sum _{i=1}^n{(y_i-{\mathit{z}}_i^T{\mathit{w}}_i)}^2+{\lambda }_1\sum _{m>l}^n{r}_{m,l}\parallel {\mathit{a}}_{m,l}-{\mathit{a}}_{l,m}{\parallel }_2+{\lambda }_2\sum _{i=1}^n{\parallel {\mathit{b}}_i\parallel }_1\right\},\\ \hfill \mathrm{s}.\mathrm{t}.{\mathit{w}}_m={\mathit{a}}_{m,l},{\mathit{w}}_i={\mathit{b}}_i,{\mathbf{1}}_p^T{\mathit{w}}_i=0,i,m,l=1,\dots ,n,\end{array}$$

(12)

where ${\mathbf{1}}_p$ is the p-dimensional vector of ones. The augmented Lagrangian for (12) is formulated as:

$$\begin{array}{cc}\hfill L{(\Theta ,Q)}_{\Omega }& =\sum _{i=1}^n{(y_i-{\mathit{z}}_i^T{\mathit{w}}_i)}^2\hfill \\ \hfill & +\sum _{m>l}^n\left\{{\lambda }_1{r}_{m,l}{\parallel {\mathit{a}}_{m,l}-{\mathit{a}}_{l,m}\parallel }_2\right.\hfill \\ \hfill & +\left.\frac{\rho }{2}(\parallel {\mathit{w}}_m-{\mathit{a}}_{m,l}+{\mathit{s}}_{m,l}{\parallel }_2^2+\parallel {\mathit{w}}_l-{\mathit{a}}_{l,m}+{\mathit{s}}_{l,m}{\parallel }_2^2)-\frac{\rho }{2}(\parallel {\mathit{s}}_{m,l}{\parallel }_2^2+\parallel {\mathit{s}}_{l,m}{\parallel }_2^2)\right\}\hfill \\ \hfill & +\sum _{i=1}^n\left\{{\lambda }_2\parallel {\mathit{b}}_i{\parallel }_1+{\mathit{t}}_i^T({\mathit{w}}_i-{\mathit{b}}_i)+\frac{\varphi }{2}{\parallel {\mathit{w}}_i-{\mathit{b}}_i\parallel }_2^2\right\}\hfill \\ \hfill & +\sum _{i=1}^n\left\{u_i{\mathbf{1}}_p^T{\mathit{w}}_i+\frac{\psi }{2}{\parallel {\mathbf{1}}_p^T{\mathit{w}}_i\parallel }_2^2\right\},\hfill \end{array}$$

(13)

where ${\mathit{s}}_{m,l},{\mathit{t}}_i,u_i$ are the Lagrange multipliers and $\rho ,\varphi ,\psi (>0)$ are the tuning parameters. For simplicity of notation, the parameters in the model ${\mathit{w}}_i,{\mathit{a}}_{i,j},{\mathit{b}}_i$ are collectively denoted as $\Theta$, the Lagrange multipliers are collectively denoted as Q, and the tuning parameters are collectively denoted as $\Omega$.

The update algorithm for $\Theta$ with the alternating direction method of multipliers (ADMM) is obtained from the following minimization problem:

$$\begin{array}{cc}\hfill {\mathit{w}}^{(k+1)}& =\underset{\mathit{w}}{\arg \min }L{(\mathit{w},{\mathit{a}}^{\left(k\right)},{\mathit{b}}^{\left(k\right)},Q^{\left(k\right)})}_{\Omega },\hfill \\ \hfill {\mathit{a}}^{(k+1)}& =\underset{\mathit{a}}{\arg \min }L{({\mathit{w}}^{(k+1)},\mathit{a},{\mathit{b}}^{\left(k\right)},Q^{\left(k\right)})}_{\Omega },\hfill \\ \hfill {\mathit{b}}^{(k+1)}& =\underset{\mathit{b}}{\arg \min }L{({\mathit{w}}^{(k+1)},{\mathit{a}}^{(k+1)},\mathit{b},Q^{\left(k\right)})}_{\Omega },\hfill \end{array}$$

(14)

where k denotes the repetition number. By repeating the updates (14) and the update for Q, the estimation algorithm for (12) is represented by Algorithm 1. The estimation algorithm for (11) is represented by Algorithm 2 with the update from ADMM. The details of the derivations of the estimation algorithms are presented in Appendices Appendix A and Appendix B.

Algorithm 1 Estimation algorithm for (12) via ADMM

Require: Initialize ${\mathit{w}}^{\left(0\right)},{\mathit{a}}^{\left(0\right)},{\mathit{b}}^{\left(0\right)},{\mathit{s}}^{\left(0\right)},{\mathit{t}}^{\left(0\right)},u^{\left(0\right)}$.  while convergence do     for  $i=1\dots ,n$ do              $$\begin{array}{cc}\hfill {\mathit{w}}_i^{(k+1)}& ={\left\{2{\mathit{z}}_i{\mathit{z}}_i^T+(\rho (n-1)+\varphi )I_p+\psi {\mathbf{1}}_p{\mathbf{1}}_p^T\right\}}^{-1}\hfill \\ \hfill & \left\{2y_i{\mathit{z}}_i+\rho \sum _{m\ne i}^n({\mathit{a}}_{i,m}^{\left(k\right)}-{\mathit{s}}_{i,m}^{\left(k\right)})-{\mathit{t}}_i^{\left(k\right)}+\varphi {\mathit{b}}_i^{\left(k\right)}-u_i^{\left(k\right)}{\mathbf{1}}_p\right\}\hfill \end{array}$$      end for      for $m,l=1,\dots ,n,(m>l)$ do         $\theta =\max \left(1-\frac{{\lambda }_1{r}_{m,l}}{\rho \parallel ({\mathit{w}}_m^{(k+1)}+{\mathit{s}}_{m,l}^{\left(k\right)})-({\mathit{w}}_l^{(k+1)}+{\mathit{s}}_{l,m}^{\left(k\right)}){\parallel }_2},0.5\right)$         ${\mathit{a}}_{m,l}^{(k+1)}=\theta ({\mathit{w}}_m^{(k+1)}+{\mathit{s}}_{m,l}^{\left(k\right)})+(1-\theta )({\mathit{w}}_l^{(k+1)}+{\mathit{s}}_{l,m}^{\left(k\right)})$         ${\mathit{a}}_{l,m}^{(k+1)}=(1-\theta )({\mathit{w}}_m^{(k+1)}+{\mathit{s}}_{m,l}^{\left(k\right)})+\theta ({\mathit{w}}_l^{(k+1)}+{\mathit{s}}_{l,m}^{\left(k\right)})$     end for     for $i=1,\dots ,n,j=1,\dots ,p$ do         $b_{ij}^{(k+1)}=\mathrm{S}(w_{ij}^{(k+1)}+\frac{1}{\varphi }{t}_{ij}^{\left(k\right)},\frac{{\lambda }_2}{\varphi })$     end for     for $m,l=1,\dots ,n,(m\ne l)$ do         ${\mathit{s}}_{m,l}^{(k+1)}={\mathit{s}}_{m,l}^{\left(k\right)}+\rho ({\mathit{w}}_m^{(k+1)}-{\mathit{a}}_{m,l}^{(k+1)})$     end for     for $i=1\dots ,n$ do         ${\mathit{t}}_i^{(k+1)}={\mathit{t}}_i^{\left(k\right)}+\varphi ({\mathit{w}}_i^{(k+1)}-{\mathit{b}}_i^{(k+1)})$         $u_i^{(k+1)}=u_i^{\left(k\right)}+\psi {\mathbf{1}}_p^T{\mathit{w}}_i^{(k+1)}$     end for end while Ensure: ${\mathit{b}}_i,i=1,\dots ,n.$

Algorithm 2 Estimation algorithm for constrained Weber problem (11) via ADMM

Require: Initialize ${\mathit{w}}_{i^{*}}^{\left(0\right)},{\mathit{e}}^{\left(0\right)},{\mathit{u}}^{\left(0\right)},v^{\left(0\right)}$. while convergence do     for $i=1\dots ,n$ do         ${\mathit{e}}_i^{(k+1)}=\min \left(\frac{r_{i^{*},i}}{\mu },{\parallel {\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}-{\widehat{\mathit{w}}}_i\parallel }_2\right)\frac{{\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}-{\widehat{\mathit{w}}}_i}{\parallel {\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}-{\widehat{\mathit{w}}}_i{\parallel }_2}$     end for     ${\mathit{w}}_{i^{*}}^{(k+1)}={(\mu n{I}_p+\eta {\mathbf{1}}_p{\mathbf{1}}_p^T)}^{-1}\left\{\mu {\sum }_{i=1}^n({\mathit{e}}_i^{(k+1)}+\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)})-v^{\left(k\right)}{\mathbf{1}}_p\right\}$     for $i=1\dots ,n$ do         ${\mathit{u}}_i^{(k+1)}={\mathit{u}}_i^{\left(k\right)}+\mu ({\mathit{e}}_i^{(k+1)}-{\mathit{w}}_{i^{*}}^{(k+1)})$     end for     $v^{(k+1)}=v^{\left(k\right)}+\eta {\mathbf{1}}_p^T{{\mathit{w}}_{{\mathit{i}}^{*}}}^{(k+1)}$ end while Ensure:  ${\mathit{w}}_{i^{*}}$

5. Simulation Studies

In this section, we report simulation studies conducted with our proposed method using artificial data.

In our simulations, we generated artificial data from the true model:

$$y_i=\left\{\begin{array}{cc}{\mathit{z}}_i^T{\mathit{w}}_{\left(1\right)}^{*}+{ϵ}_i,\hfill & (i=1,\dots ,40),\hfill \\ {\mathit{z}}_i^T{\mathit{w}}_{\left(2\right)}^{*}+{ϵ}_i,\hfill & (i=41,\dots ,80),\hfill \\ {\mathit{z}}_i^T{\mathit{w}}_{\left(3\right)}^{*}+{ϵ}_i,\hfill & (i=81,\dots ,120),\hfill \end{array}\right.$$

(15)

where ${\mathit{z}}_i={(log{x}_{i1},\dots ,log{x}_{ip})}^T$, ${\mathit{x}}_i={(x_{i1},\dots ,x_{ip})}^T$ is p-dimensional compositional data, ${\mathit{w}}_{\left(1\right)}^{*}$,${\mathit{w}}_{\left(2\right)}^{*}$,${\mathit{w}}_{\left(3\right)}^{*}\in {\mathbb{R}}^p$ are the true regression coefficient vectors, and ${ϵ}_i$ is an error term distributed as $N(0,{\sigma }^2)$ independently. We generated compositional data $\{{\mathit{x}}_i;i=1,\dots ,120\}$ as follows. First, we generated the data $\{{\mathit{c}}_i,i=1,\dots ,120\}$ from the p-dimensional multivariate normal distribution $N_p(\mathbf{\omega },\Sigma )$ independently, where ${\left(\mathbf{\omega }\right)}_j={\omega }_j$, ${(\Sigma )}_{ij}=0.2^{\left|i-j\right|}$, and

$${\omega }_j=\left\{\begin{array}{cc}log\left(0.5p\right),\hfill & (j=1,\dots ,5),\hfill \\ 0,\hfill & (j=6,\dots ,p).\hfill \end{array}\right.$$

(16)

Then, the data $\{{\mathit{c}}_i,i=1,\dots ,120\}$ were converted to the compositional data $\{{\mathit{x}}_i;i=1,\dots ,120\}$ by the following transformation:

$$x_{ij}=\frac{exp\left(c_{ij}\right)}{{\sum }_{k=1}^{p}exp\left(c_{ik}\right)},i=1,\dots ,120,j=1,\dots ,p.$$

(17)

The true regression coefficient vectors were set as:

$$\begin{array}{cc}\hfill {\mathit{w}}_{\left(1\right)}^{*}& ={(1,-0.8,0.6,0,0,-1.5,-0.5,1.2,{\mathbf{0}}_{p-8}^T)}^T,\hfill \\ \hfill {\mathit{w}}_{\left(2\right)}^{*}& ={(0,-0.5,1,1.2,0.1,-1,0,-0.8,{\mathbf{0}}_{p-8}^T)}^T,\hfill \\ \hfill {\mathit{w}}_{\left(3\right)}^{*}& ={(0,0,0,0.8,1,0,-0.8,-1,{\mathbf{0}}_{p-8}^T)}^T.\hfill \end{array}$$

(18)

We also assumed that the graph $R\in {\{0,1\}}^{120\times 120}$ was observed. In the graph, the true value of each element was obtained with probability $P_R$. We calculated MSE as ${\sum }_{i^{*}=1}^{n^{*}}{(y_{i^{*}}-{\mathit{z}}_{i^{*}}^T{\widehat{\mathit{w}}}_{i^{*}})}^2/n^{*}$, dividing the 120 samples into 100 training data and 20 validation data. Here, $n^{*}$ indicates the number of samples for validation data (i.e., 20). The regression coefficient vectors for the validation data were estimated based on the constrained Weber problem (11). To evaluate the effectiveness of our proposed method, it is compared with both Model (7) and the model obtained by removing the zero-sum constraint from minimization problem (9). We refer to the latter two comparison methods as compositional lasso (CL) and sparse network lasso (SNL), respectively. To the best of our knowledge, there are no studies that simultaneously perform regression and clustering on compositional data. Therefore, we compared with the CL and SNL models; CL assumes the situation where the existence of the multiple clusters is not considered, while SNL considers their existence while the nature of the compositional data is ignored.

The regularization parameters were determined by five-fold CV for both the proposed method and the comparison methods. The values of tuning parameters $\rho ,\varphi ,\psi ,\mu ,\eta$ for ADMM were all set to one. We considered several settings: $p=\{30,100,200\}$, $\sigma =\{0.1,0.5,1\}$, $P_R=\{0.99,0.95,0.90,0.80,0.70\}$. We generated 100 datasets and computed the mean and standard deviation of MSE from the 100 repetitions.

Table 1, Table 2 and Table 3 show the results for the mean and standard deviation of MSE for each $\sigma$. The proposed method and SNL show better prediction accuracy than CL in almost settings. The reason for this may be that CL assumes only a single regression coefficient vector and thus fails to capture the true structure, which consists of three clusters. A comparison between the proposed method and SNL shows that the proposed method has higher prediction accuracy than SNL when $P_R=0.99,0.95,$ and $0.90$, whereas SNL shows better results in most cases when $P_R=0.80,0.70$. This means that the proposed method is superior to SNL when the structure of the graph R is similar to the true structure. On the whole, the prediction accuracy deteriorates as $P_R$ decreases for both the proposed method and SNL, but this deterioration is more extreme for the proposed method. For both the proposed method and SNL, which assume multiple regression coefficient vectors, the standard deviation is somewhat large.

Table 1. Mean and deviation of MSE in $\sigma =0.1$ for simulations.

Method	$\mathit{p}=30$	$\mathit{p}=100$	$\mathit{p}=200$
CL	$5.20\left(1.44\right)$	$6.99\left(1.81\right)$	$8.75\left(2.69\right)$
$P_R=0.99$
Proposed	$\mathbf{0}.\mathbf{51}\left(0.58\right)$	$\mathbf{2}.\mathbf{13}\left(2.35\right)$	$\mathbf{2}.\mathbf{70}\left(1.92\right)$
SNL	$2.54\left(0.97\right)$	$3.73\left(1.31\right)$	$6.55\left(3.94\right)$
$P_R=0.95$
Proposed	$\mathbf{2}.\mathbf{74}\left(1.19\right)$	$\mathbf{2}.\mathbf{96}\left(1.33\right)$	$\mathbf{3}.\mathbf{68}\left(2.71\right)$
SNL	$3.19\left(0.98\right)$	$3.87\left(1.35\right)$	$5.26\left(4.15\right)$
$P_R=0.90$
Proposed	$\mathbf{3}.\mathbf{29}\left(1.38\right)$	$\mathbf{3}.\mathbf{80}\left(1.33\right)$	$\mathbf{4}.\mathbf{49}\left(1.81\right)$
SNL	$3.40\left(1.23\right)$	$4.25\left(1.49\right)$	$4.75\left(1.49\right)$
$P_R=0.80$
Proposed	$4.20\left(1.58\right)$	$5.53\left(2.30\right)$	$7.49\left(3.98\right)$
SNL	$\mathbf{3}.\mathbf{87}\left(1.41\right)$	$\mathbf{4}.\mathbf{86}\left(1.52\right)$	$\mathbf{5}.\mathbf{70}\left(2.00\right)$
$P_R=0.70$
Proposed	$\mathbf{4}.\mathbf{13}\left(1.56\right)$	$6.57\left(2.55\right)$	$7.66\left(2.28\right)$
SNL	$4.56\left(1.55\right)$	$\mathbf{5}.\mathbf{63}\left(1.72\right)$	$\mathbf{6}.\mathbf{69}\left(2.25\right)$

Table 2. Mean and deviation of MSE in $\sigma =0.5$ for simulations.

Method	$\mathit{p}=30$	$\mathit{p}=100$	$\mathit{p}=200$
CL	$5.64\left(1.42\right)$	$7.94\left(2.31\right)$	$10.02\left(2.96\right)$
$P_R=0.99$
Proposed	$\mathbf{1}.\mathbf{02}\left(0.75\right)$	$\mathbf{2}.\mathbf{13}\left(1.50\right)$	$\mathbf{3}.\mathbf{07}\left(1.61\right)$
SNL	$2.97\left(1.23\right)$	$3.73\left(1.32\right)$	$5.98\left(3.95\right)$
$P_R=0.95$
Proposed	$\mathbf{3}.\mathbf{05}\left(1.28\right)$	$\mathbf{3}.\mathbf{36}\left(1.05\right)$	$\mathbf{4}.\mathbf{37}\left(3.47\right)$
SNL	$3.50\left(1.20\right)$	$4.19\left(1.35\right)$	$5.18\left(2.54\right)$
$P_R=0.90$
Proposed	$\mathbf{3}.\mathbf{55}\left(1.40\right)$	$4.83\left(1.61\right)$	$\mathbf{5}.\mathbf{13}\left(2.99\right)$
SNL	$3.83\left(1.30\right)$	$\mathbf{4}.\mathbf{43}\left(1.26\right)$	$5.21\left(3.42\right)$
$P_R=0.80$
Proposed	$4.10\left(1.47\right)$	$\mathbf{5}.\mathbf{21}\left(1.89\right)$	$6.70\left(2.51\right)$
SNL	$\mathbf{4}.\mathbf{06}\left(1.35\right)$	$5.32\left(1.88\right)$	$\mathbf{6}.\mathbf{06}\left(1.94\right)$
$P_R=0.70$
Proposed	$\mathbf{4}.\mathbf{33}\left(1.39\right)$	$6.59\left(2.62\right)$	$8.58\left(3.16\right)$
SNL	$4.53\left(1.54\right)$	$\mathbf{5}.\mathbf{71}\left(1.78\right)$	$\mathbf{7}.\mathbf{37}\left(2.24\right)$

Table 3. Mean and deviation of MSE in $\sigma =1$ for simulations.

Method	$\mathit{p}=30$	$\mathit{p}=100$	$\mathit{p}=200$
CL	$6.50\left(1.78\right)$	$8.05\left(2.57\right)$	$10.41\left(3.10\right)$
$P_R=0.99$
Proposed	$\mathbf{2}.\mathbf{34}\left(1.29\right)$	$\mathbf{3}.\mathbf{25}\left(1.87\right)$	$\mathbf{3}.\mathbf{87}\left(1.97\right)$
SNL	$3.94\left(1.47\right)$	$4.98\left(1.67\right)$	$5.90\left(3.12\right)$
$P_R=0.95$
Proposed	$\mathbf{3}.\mathbf{43}\left(1.24\right)$	$\mathbf{3}.\mathbf{96}\left(1.61\right)$	$\mathbf{4}.\mathbf{73}\left(2.35\right)$
SNL	$4.36\left(1.31\right)$	$4.73\left(1.29\right)$	$5.39\left(1.64\right)$
$P_R=0.90$
Proposed	$\mathbf{4}.\mathbf{28}\left(1.55\right)$	$\mathbf{4}.\mathbf{83}\left(1.61\right)$	$\mathbf{5}.\mathbf{09}\left(1.88\right)$
SNL	$4.71\left(1.49\right)$	$5.25\left(2.08\right)$	$5.75\left(2.29\right)$
$P_R=0.80$
Proposed	$5.67\left(1.80\right)$	$6.61\left(2.05\right)$	$7.77\left(3.68\right)$
SNL	$\mathbf{5}.\mathbf{20}\left(1.79\right)$	$\mathbf{6}.\mathbf{06}\left(2.01\right)$	$\mathbf{6}.\mathbf{77}\left(2.07\right)$
$P_R=0.70$
Proposed	$5.73\left(1.87\right)$	$8.21\left(2.71\right)$	$8.86\left(3.57\right)$
SNL	$\mathbf{5}.\mathbf{31}\left(1.84\right)$	$\mathbf{7}.\mathbf{02}\left(2.29\right)$	$\mathbf{7}.\mathbf{97}\left(2.56\right)$

6. Application to Gut Microbiome Data

The gut microbiome affects human health/disease, for example, in terms of obesity. Gut microbiomes may be exposed to inter-individual heterogeneity from environmental factors such as diet as well as from hormonal factors and age [23,24]. In this section, we applied our proposed method to the real dataset reported by [9]. This dataset was obtained from a cross-sectional study of 98 healthy volunteers conducted at the University of Pennsylvania to investigate the connections between long-term dietary patterns and gut microbiome composition. Stool samples were collected from the subjects, and DNA samples were analyzed by 454/Roche pyrosequencing of 16S rRNA gene segments of the V1–V2 region. In the results, the counts for more than 17,000 species-level OTUs were obtained. Demographic data, such as body mass index (BMI), sex, and age, were also obtained.

We used centered BMI as the response and the compositional data of the gut microbiome as the explanatory variable. In order to reduce their number, we used single-linkage clustering based on an available phylogenetic tree to aggregate the OTUs, which is provided as the function tip_glom in the R package “phyloseq” see [25]. In this process, some closely related OTUs defined on the leaf nodes of the phylogenetic tree are aggregated into one OTU when the cophenetic distances between the OTUs are smaller than a certain threshold. We set the threshold at $0.5$. As a result, 166 OTUs were obtained after the aggregation. Since some OTUs had zero counts, making it impossible to take the logarithm, these counts were replaced by the value one before converting them to compositional data.

We computed the graph $R\in {\mathbb{R}}^{98\times 98}$ as follows:

$$R=\frac{S^T+S}{2},{\left(S\right)}_{ij}=\left\{\begin{array}{cc}1\hfill & j\mathrm{-th}\mathrm{sample}\mathrm{is}\mathrm{a} 5\mathrm{-NN}\mathrm{of} i\mathrm{-th}\mathrm{sample}\mathrm{with}{D}_{ij},\hfill \\ 0\hfill & \mathrm{Otherwise},\hfill \end{array}\right.$$

(19)

where $D_{ij}$ is the distance between the i-th and j-th samples. Distance $D_{ij}$ was calculated in the following two ways: (i) Gower distance [26] was calculated using the sex and age data of the subjects. (ii) Log-ratio distance (e.g., see [27]) was calculated using the explanatory variable, as follows:

$$D_{ij}=\sqrt{\sum _{l=1}^p{(z_{il}^c-z_{jl}^c)}^2},$$

(20)

where $z_{ij}^c=log{x}_{ij}-\frac{1}{p}{\sum }_{j=1}^{p}log{x}_{ij}$. Using these two ways of calculating distance, we obtained two different R and estimation results. We refer to these two methods as Proposed (i) and Proposed (ii), respectively. Equation (19) is the same as the one used in Yamada et al. [7] in the application to real datasets.

To evaluate the prediction accuracy of our proposed method, we calculated MSE by randomly splitting the dataset into 90 samples as the training data and eight samples as the validation data. We again used the method of Lin et al. (2014), which is referred to as compositional lasso (CL), as a comparison method. The regularization parameters were determined by five-fold CV for both our proposed method and CL. The mean and standard deviation of MSE were calculated from 100 repetitions.

Table 4 shows the mean and standard deviation of MSE in the real data analysis. We observe that Proposed (i) has the smallest mean and standard deviation of MSE. This indicates that our proposed method captures the inherent structure of the compositional data by providing an appropriate graph R. However, the standard deviation is large even for Proposed (i), which indicates that the prediction accuracy may strongly depend on the assignments of samples to the training data and the validation data.

Table 4. Mean and standard deviation of MSE for real data analysis (100 repetitions).

Value	Proposed (i)	Proposed (ii)	CL
MSE (SD)	$\mathbf{23}.\mathbf{01}\left(16.62\right)$	$31.59\left(22.44\right)$	$30.96\left(23.36\right)$

Table 5 shows the coefficient of determination ${\mathrm{R}}^2$ using leave-one-out cross-validation (LOOCV) for Proposed (i) and CL. The fittings of the observed and predicted BMI are plotted in Figure 1a,b for Proposed (i) and CL, respectively. The horizontal axis represents the centered observed BMI values, and the vertical axis represents the corresponding predicted BMI. As shown, CL does not predict data with centered observed values between $-10$ and $-5$ as being in that interval, whereas Proposed (i) predicts these data correctly to some extent.

Table 5. Coefficients of determination using LOOCV.

Value	Proposed (i)	CL
LOOCV ${\mathrm{R}}^2$	$\mathbf{0}.\mathbf{245}$	$0.083$

Figure 1. Observed and predicted BMI using LOOCV.

Figure 2 shows the regression coefficients estimated by Proposed (i) for all samples, where the regularization parameters are determined by LOOCV. To obtain the results in Figure 2, we used hierarchical clustering to group together similar regression coefficient vectors, setting the number of clusters as five.

Figure 2. Estimated regression coefficients for all samples.

With our proposed method, many of the estimated regression coefficients were not exactly zero but close to zero. Thus, we will treat estimated regression coefficients $|{\widehat{w}}_{ij}|<0.05$ as being exactly zero to simplify the interpretation. Figure 3 shows only those coefficients that satisfy $|{\widehat{w}}_{ij}|\ge 0.05$ in at least one sample, where the corresponding variables are listed in Table 6.

Figure 3. Only the estimated regression coefficients with $|{\widehat{w}}_{ij}|\ge 0.05$ for at least one sample.

Table 6. Variables with estimated regression coefficients $|{\widehat{w}}_{ij}|\ge 0.05$ for at least one sample.

Variable	Kingdom	Phylum	Class	Order	Family	Genus	Species
OTU1	Bacteria
OTU2	Bacteria
OTU3	Bacteria	Bacteroidetes
OTU4	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales
OTU5	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Bacteroidaceae	Bacteroides
OTU6	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Bacteroidaceae	Bacteroides
OTU7	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Bacteroidaceae	Bacteroides
OTU8	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Bacteroidaceae	Bacteroides
OTU9	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Bacteroidaceae	Bacteroides
OTU10	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Bacteroidaceae	Bacteroides
OTU11	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Porphyromonadaceae	Parabacteroides
OTU12	Bacteria	Bacteroidetes	Bacteroidetes	Bacteroidales	Prevotellaceae
OTU13	Bacteria	Firmicutes	Clostridia
OTU14	Bacteria	Firmicutes	Clostridia	Clostridiales
OTU15	Bacteria	Firmicutes	Clostridia	Clostridiales
OTU16	Bacteria	Firmicutes	Clostridia	Clostridiales
OTU17	Bacteria	Firmicutes	Clostridia	Clostridiales
OTU18	Bacteria	Firmicutes	Clostridia	Clostridiales
OTU19	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU20	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU21	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU22	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU23	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU24	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU25	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU26	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU27	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU28	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae
OTU29	Bacteria	Firmicutes	Clostridia	Clostridiales	Lachnospiraceae	Roseburia
OTU30	Bacteria	Firmicutes	Clostridia	Clostridiales	Ruminococcaceae
OTU31	Bacteria	Firmicutes	Clostridia	Clostridiales	Ruminococcaceae
OTU32	Bacteria	Firmicutes	Erysipelotrichia	Erysipelotrichales	Erysipelotrichaceae	Catenibacterium
OTU33	Bacteria	Firmicutes	Erysipelotrichia	Erysipelotrichales	Erysipelotrichaceae	Erysipelotrichaceae. Incertae.Sedis
OTU34	Bacteria	Proteobacteria
OTU35	Bacteria	Proteobacteria	Gammaproteobacteria	Enterobacteriales	Enterobacteriaceae

It is reported that the human gut microbiome can be classified into three clusters, called enterotypes, which are characterized by three predominant genera: Bacteroides, Prevotella, and Ruminococcus [11]. In the dataset, OTUs of genus levels Prevotella and Ruminococcus were aggregated into the OTUs of family levels Prevotellaceae and Ruminococcaceae by the single-linkage clustering. In Figure 3, Bacteroides correspond to OTU5, 6, 7, 8, 9, and 10; Prevotellaceae corresponds to OTU12; and Ruminococcaceae corresponds to OTU30 and 31. For these OTUs, the differences are clear between OTU6, 9, 10 and OTU30, 31 among samples 81–90, in which only Bacteroides are correlated to the response. On the other hand, the differences among samples 65–74 are also indicated, in which only Bacteroides do not affect BMI. These results suggest that Bacteroides, Prevotellaceae, and Ruminococcaceae may have different effects on BMI that are associated with enterotypes. In addition, it is reported that women with a higher abundance of Prevotellaceae are more obese [28]. The regression coefficients of non-zero Prevotellaceae are all positive, and the eight corresponding samples are all females. On the other hand, in OTU29 indicating Roseburia, 9 samples out of 10 are negatively associated with BMI. Roseburia is also reported to be negatively correlated with indicators of body weight [29].

7. Conclusions

We proposed a multi-task learning method for compositional data based on a sparse network lasso and log-contrast model. By imposing a zero-sum constraint on the model corresponding to each sample, we could extract the information of latent clusters in the regression coefficient vectors for compositional data. In the results of simulations, the proposed method worked well when clusters existed for the compositional data and an appropriate graph R was obtained. In a human gut microbiome analysis, our proposed method is better than the existing method in prediction accuracy by considering the heterogeneity from age and sex. In addition, cluster-specific OTUs such as ones related to enterotypes were detected in terms of effects on BMI.

Although our proposed method shrinks some regression coefficients that do not affect response to zero, many coefficients close to zero remain. Furthermore, in both the simulations and human gut microbiome analysis, the prediction accuracy of the proposed method deteriorated significantly when the obtained R did not capture the true structure. Moreover, the standard deviations of MSE were high in almost all cases. We leave these as topics of future research.

Appendix A.1. Update of w

In the update of $\mathit{w}$, we minimize the terms of the augmented Lagrangian (13) depending on ${\mathit{w}}_{\mathit{i}}$ as follows:

$$\begin{array}{c}\hfill {\mathit{w}}_i^{(k+1)}=\underset{{\mathit{w}}_i\in {\mathbb{R}}^p}{\arg \min }\left\{{(y_i-{\mathit{z}}_i^T{\mathit{w}}_i)}^2+\frac{\rho }{2}\sum _{m\ne i}^n{\parallel {\mathit{w}}_i-{\mathit{a}}_{i,m}^{\left(k\right)}+{\mathit{s}}_{i,m}^{\left(k\right)}\parallel }_2^2\right.\\ \hfill \left.+{\mathit{t}}_i^{\left(k\right)T}({\mathit{w}}_i-{\mathit{b}}_i^{\left(k\right)})+\frac{\varphi }{2}\parallel {\mathit{w}}_i-{\mathit{b}}_i^{\left(k\right)}{\parallel }_2^2+u_i^{\left(k\right)}{\mathbf{1}}_p^T{\mathit{w}}_i+\frac{\psi }{2}{\parallel {\mathbf{1}}_p^T{\mathit{w}}_i\parallel }_2^2\right\}.\end{array}$$

(A1)

From $\frac{\partial L}{\partial {\mathit{w}}_i}=\mathbf{0}$, we obtain the update:

$$\begin{array}{cc}\hfill {\mathit{w}}_i^{(k+1)}& ={\left\{2{\mathit{z}}_i{\mathit{z}}_i^T+(\rho (n-1)+\varphi )I_p+\psi {\mathbf{1}}_p{\mathbf{1}}_p^T\right\}}^{-1}\hfill \\ \hfill & \left\{2y_i{\mathit{z}}_i+\rho \sum _{m\ne i}^n({\mathit{a}}_{i,m}^{\left(k\right)}-{\mathit{s}}_{i,m}^{\left(k\right)})-{\mathit{t}}_i^{\left(k\right)}+\varphi {\mathit{b}}_i^{\left(k\right)}-u_i^{\left(k\right)}{\mathbf{1}}_p\right\}.\hfill \end{array}$$

(A2)

Appendix A.2. Update of a

The update of $\mathit{a}$ is obtained by the joint minimization of ${\mathit{a}}_{m,l}$ and ${\mathit{a}}_{l,m}$ as follows:

$$\begin{array}{cc}\hfill {\mathit{a}}_{m,l}^{(k+1)},{\mathit{a}}_{l,m}^{(k+1)}& =\underset{{\mathit{a}}_{m,l},{\mathit{a}}_{l,m}\in {\mathrm{R}}^p}{\arg \min }\left\{{\lambda }_1{r}_{m,l}{\parallel a_{m,l}-a_{l,m}\parallel }_2\right.\hfill \\ & +\left.\frac{\rho }{2}(\parallel {\mathit{w}}_m^{(k+1)}-{\mathit{a}}_{m,l}+{\mathit{s}}_{m,l}^{\left(k\right)}{\parallel }_2^2+\parallel {\mathit{w}}_l^{(k+1)}-{\mathit{a}}_{l,m}+{\mathit{s}}_{l,m}^{\left(k\right)}{\parallel }_2^2)\right\}.\hfill \end{array}$$

(A3)

In [8], the analytical solution was given as:

$$\begin{array}{c}\hfill {\mathit{a}}_{m,l}^{(k+1)}=\theta ({\mathit{w}}_m^{(k+1)}+{\mathit{s}}_{m,l}^{\left(k\right)})+(1-\theta )({\mathit{w}}_l^{(k+1)}+{\mathit{s}}_{l,m}^{\left(k\right)}),\\ \hfill {\mathit{a}}_{l,m}^{(k+1)}=(1-\theta )({\mathit{w}}_m^{(k+1)}+{\mathit{s}}_{m,l}^{\left(k\right)})+\theta ({\mathit{w}}_l^{(k+1)}+{\mathit{s}}_{l,m}^{\left(k\right)}),\end{array}$$

(A4)

where

$$\theta =\max \left(1-\frac{{\lambda }_1{r}_{m,l}}{\rho \parallel ({\mathit{w}}_m^{(k+1)}+{\mathit{s}}_{m,l}^{\left(k\right)})-({\mathit{w}}_l^{(k+1)}+{\mathit{s}}_{l,m}^{\left(k\right)}){\parallel }_2},0.5\right).$$

(A5)

Appendix A.3. Update of b

The update of $\mathit{b}$ is obtained by the following minimization problem:

$${\mathit{b}}_i^{(k+1)}=\underset{{\mathit{b}}_i\in {\mathrm{R}}^p}{\arg \min }\sum _{i=1}^n\left\{{\lambda }_2\parallel {\mathit{b}}_i{\parallel }_1+{\mathit{t}}_i^T({\mathit{w}}_i-{\mathit{b}}_i)+\frac{\varphi }{2}{\parallel {\mathit{w}}_i-{\mathit{b}}_i\parallel }_2^2\right\}.$$

(A6)

Because the minimization problem with respect to ${\mathit{b}}_i$ contains a non-differentiable point in the ${\ell }_1$ norm of ${\mathit{b}}_i$, we consider the subderivative of $|b_{ij}|$. Then, we obtain the update:

$$b_{ij}^{(k+1)}=\mathrm{S}\left(w_{ij}+\frac{t_{i,j}}{\varphi },\frac{{\lambda }_2}{\varphi }\right),j=1,\dots ,p,$$

(A7)

where $\mathrm{S}(·,·)$ is the soft-thresholding operator given by $\mathrm{S}(x,\lambda ):=\mathrm{sign}\left(x\right){\left(\right|x|-\lambda )}_{+}$.

Appendix A.4. Update of Q

The updates for the Lagrange multipliers denoted as Q are obtained by gradient descent as follows:

$$\begin{array}{cc}\hfill {\mathit{s}}_{m,l}^{(k+1)}& ={\mathit{s}}_{m,l}^{\left(k\right)}+\rho ({\mathit{w}}_m^{(k+1)}-{\mathit{a}}_{m,l}^{(k+1)}),m,l=1,\dots ,n(i\ne j),\hfill \\ \hfill {\mathit{t}}_i^{(k+1)}& ={\mathit{t}}_i^{\left(k\right)}+\varphi ({\mathit{w}}_i^{(k+1)}-{\mathit{b}}_i^{(k+1)}),i=1,\dots ,n,\hfill \\ \hfill u_i^{(k+1)}& =u_i^{\left(k\right)}+\psi {\mathbf{1}}_p^T{\mathit{w}}_i^{(k+1)},i=1,\dots ,n.\hfill \end{array}$$

(A8)

Appendix B.1. Update of w i *

In the update of ${\mathit{w}}_{i^{*}}$, we minimize the terms of the augmented Lagrangian (A11) depending on ${\mathit{w}}_{{\mathit{i}}^{*}}$ as follows:

$${\mathit{w}}_{i^{*}}^{(k+1)}=\underset{w_{i^{*}}\in {\mathbb{R}}^p}{\arg \min }\left\{\sum _{i=1}^n\left\{\frac{\mu }{2}{∥{\mathit{w}}_{i^{*}}-{\mathit{e}}_i^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}∥}_2^2\right\}+v^{\left(k\right)}{\mathbf{1}}_p^T{\mathit{w}}_{i^{*}}+\frac{\eta }{2}{\parallel {\mathbf{1}}_p^T{\mathit{w}}_{i^{*}}\parallel }_2^2\right\}.$$

(A12)

From $\frac{\partial L}{\partial {\mathit{w}}_{i^{*}}}=\mathbf{0}$, we obtain the update:

$${\mathit{w}}_{i^{*}}^{(k+1)}={(\mu n{\mathbf{1}}_p+\eta {\mathbf{1}}_p{\mathbf{1}}_p^T)}^{-1}\left\{\mu \sum _{i=1}^n\left({\mathit{e}}_i^{\left(k\right)}+\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}\right)-v^{\left(k\right)}{\mathbf{1}}_p\right\}.$$

(A13)

Appendix B.2. Update of e

In the update of ${\mathit{e}}_i$, we minimize the terms of augmented Lagrangian (A11) depending on ${\mathit{e}}_i$ as follows:

$$\begin{array}{c}\hfill {\mathit{e}}_i^{(k+1)}=\underset{{\mathit{e}}_i\in {\mathbb{R}}^p}{\arg \min }\left\{r_{i^{*},i}{\parallel {\mathit{e}}_i-{\mathit{w}}_{i^{*}}^{(k+1)}\parallel }_2+\frac{\mu }{2}{∥{\mathit{e}}_i-\left({\mathit{w}}_{i^{*}}^{(k+1)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}\right)∥}_2^2\right\},\\ \hfill i=1,\cdots ,n,\end{array}$$

(A14)

The minimization problem (A14) is equivalently expressed as:

$$\begin{array}{c}\hfill {\mathit{e}}_i^{(k+1)}=\underset{{\mathit{e}}_i\in {\mathbb{R}}^p}{\arg \min }\left\{\frac{r_{i^{*},i}}{\mu }{\parallel {\mathit{e}}_i-{\mathit{w}}_{i^{*}}^{\left(k\right)}\parallel }_2+\frac{1}{2}{∥{\mathit{e}}_i-\left({\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}\right)∥}_2^2\right\},\\ \hfill i=1,\cdots ,n.\end{array}$$

(A15)

In [30], because the right-hand side of (A15) is the proximal map of the function $f\left({\mathit{e}}_i\right)=\frac{r_{i^{*},i}}{\rho }{\parallel {\mathit{e}}_i-{\mathit{w}}_{i^{*}}\parallel }_2$, by using Moreau’s decomposition (e.g., [31]), the updates of $\mathit{e}$ are obtained by:

$${\mathit{e}}_i^{(k+1)}=min\left(\frac{r_{i^{*},i}}{\mu },{∥{\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}-{\widehat{\mathit{w}}}_i∥}_2\right)\frac{{\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}-{\widehat{\mathit{w}}}_i}{\parallel {\mathit{w}}_{i^{*}}^{\left(k\right)}-\frac{1}{\mu }{\mathit{u}}_i^{\left(k\right)}-{\widehat{\mathit{w}}}_i{\parallel }_2}.$$

(A16)

Appendix B.3. Update of u and v

The updates for Lagrange multipliers ${\mathit{u}}_i$ and v are obtained by gradient descent as follows:

$$\begin{array}{cc}\hfill {\mathit{u}}_i^{(k+1)}& ={\mathit{u}}_i^{\left(k\right)}+\mu ({\mathit{e}}_i^{(k+1)}-{\mathit{w}}_{i^{*}}^{(k+1)}),i=1,\dots ,n,\hfill \\ \hfill v^{(k+1)}& =v^{\left(k\right)}+\eta {\mathbf{1}}_p^T{{\mathit{w}}_{{\mathit{i}}^{*}}}^{(k+1)}.\hfill \end{array}$$

(A17)