An Efficient Algorithm for Convex Biclustering

Abstract

We consider biclustering that clusters both samples and features and propose efficient convex biclustering procedures. The convex biclustering algorithm (COBRA) procedure solves twice the standard convex clustering problem that contains a non-differentiable function optimization. We instead convert the original optimization problem to a differentiable one and improve another approach based on the augmented Lagrangian method (ALM). Our proposed method combines the basic procedures in the ALM with the accelerated gradient descent method (Nesterov’s accelerated gradient method), which can attain $O(1/k^2)$ convergence rate. It only uses first-order gradient information, and the efficiency is not influenced by the tuning parameter $\lambda$ so much. This advantage allows users to quickly iterate among the various tuning parameters $\lambda$ and explore the resulting changes in the biclustering solutions. The numerical experiments demonstrate that our proposed method has high accuracy and is much faster than the currently known algorithms, even for large-scale problems.

1. Introduction

By clustering, such as k-means clustering [1] and hierarchical clustering [2,3], we usually mean dividing N samples, each consisting of p covariate values, into several categories, where $N,p\ge 1$.

In this paper, we consider biclustering [4] that is an extended notion of clustering. In biclustering, we divide both $\{1,\dots ,N\}$ and $\{1,\dots ,p\}$ based on the data simultaneously. If we are given a data matrix in ${\mathbb{R}}^{N\times p}$, then the rows and columns within the shared group exhibit similar characteristics. For example, given a gene expression data matrix, with genes as columns and samples as rows, the biclustering detects the submatrices, which represent the cooperative behavior of a group of genes corresponding to a group of samples [5]. Figure 1 illustrates an intuitive difference between standard clustering and biclustering. In recent years, biclustering has become a ubiquitous data-mining technique with varied applications, such as text mining, recommendation system, and bioinformatics. A comprehensive survey of biclustering was given by [6,7,8].

Figure 1. While standard clustering divides either rows or columns, the biclustering divides both. (a) Row clustering; (b) column clustering; (c) biclustering.

However, as noted in [6], biclustering is an NP-hard problem. Thus, the results may vary significantly with different initializations. Moreover, some conventional biclustering models suffer from poor performance due to non-convexity, which may return local optimal solutions. In order to avoid such an inconvenience, Chi et al. [9] proposed convex biclustering by reformulating the problem to convex formulations and using the fused lasso [10] concept.

In convex clustering [9,11,12,13], given a data matrix $X\in {\mathbb{R}}^{N\times p}$ and $\lambda >0$, we compute a matrix U of the same size as X. Let $X_{i·}$ and $U_{i·}$ be the i-th rows of X and U, let $X_{·j}$ and $U_{·j}$ be the the j-th columns of X and U, and let ${\parallel X-U\parallel }_F^2$ denote the square sums of the $Np$ elements, and $\parallel U_{i·}-U_{j·}\parallel$, $\parallel U_{·m}-U_{·n}\parallel$ as the ${\ell }_2$ norm of p, N-dimensional vectors, respectively. Convex clustering finds $U\in {\mathbb{R}}^{N\times p}$ that minimizes a weighted sum of ${\parallel X-U\parallel }_F^2$ and $\{\lambda \parallel U_{i·}-U_{j·}{\parallel \}}_{i\ne j}$ (it may be formulated as minimizing a weighted sum of ${\parallel X-U\parallel }_F^2$ and $\{\lambda \parallel U_{·m}-U_{·n}{\parallel \}}_{m\ne n}$). If $X_{i·}$ and $X_{j·}$ share $U_{i·}=U_{j·}$, then they are in the same group w.r.t. $\{1,\cdots ,N\}$. On the other hand, convex biclustering finds $U\in {\mathbb{R}}^{N\times p}$ that minimizes a weighted sum of ${\parallel X-U\parallel }_F^2$ and $\{\lambda \parallel U_{i·}-U_{j·}{\parallel \}}_{i\ne j}$ and $\{\lambda \parallel U_{·m}-U_{·n}{\parallel \}}_{m\ne n}$.

The convex biclustering achieves checker-board-like biclusters by penalizing both the rows and columns of U. When $\lambda$ (the tuning parameter for $\{\parallel U_{i·}-U_{j·}{\parallel \}}_{i\ne j}$ and $\{\parallel U_{·m}-U_{·n}{\parallel \}}_{m\ne n}$) is zero, each $(i,j)$ occupies a unique bicluster $\left\{\right(i,j\left)\right\}$ and $x_{ij}=u_{ij}$ for $i=1,\cdots ,N$ and $j=1,\cdots ,p$ when $X=\left(x_{ij}\right)$ and $U=\left(u_{ij}\right)$. As $\lambda$ increases, the bicluster begins to fuse. For sufficiently large $\lambda$, all $(i,j)$ merge into one single bicluster $\left\{\right(i,j\left)\right|i=1,\dots ,N,j=1,\dots ,p\}$. The convex formulation guarantees a globally optimal solution and demonstrates superior performance to competing approaches. Chi et al. [9] claimed that the convex biclustering performs better than the dynamic tree-cutting algorithm [14] and sparse biclustering algorithm [15] in their experiments.

Nevertheless, despite these advantages, convex biclustering has not yet gained widespread popularity, due to its intensive computation. On the one hand, the main challenge of solving the optimization problem is the two fused penalty terms: indecomposable and non-differentiable. These properties increase the difficulty of solving. Many splitting methods for the indecomposable problem are complicated and create many subproblems to solve; techniques such as the subgradient method for the non-differentiable problem are slow to converge [16,17]. Moreover, it is difficult to find the optimal tuning parameter $\lambda$ because we need to solve optimization problems with a sequence of parameters $\lambda$ and select the one for the specific demand of researchers. Hence, we need to propose a fast way to solve the problems with the sequence of parameters $\lambda$. On the other hand, with the increased demands of biclustering techniques, convex biclustering is faced with large-scale data as the volume and complexity of data grows. Above all, it is necessary to propose an efficient algorithm to solve the convex biclustering problem.

There are limited algorithms for solving the problem in the literature. Chi et al. [9] proposed the convex biclustering algorithm (COBRA) using a Dykstra-like proximal algorithm [18] to solve the convex biclustering problem. Weylandt [19] proposed using alternating direction method of multipliers (ADMM) [20,21] and its variant, generalized ADMM [22], to solve the problem.

However, the COBRA yields subproblems, including the convex clustering problem, which requires expensive computations for large-scale problems due to the high per iteration cost [23,24]. Essentially, COBRA is a splitting method that separately solves a composite optimization problem containing three terms. Additionally, it is sensitive to tuning parameter $\lambda$. Therefore, obtaining the solutions under a wide range of parameters $\lambda$ takes time, which is not feasible for broad applications and different demands for users. ADMM generally solves the problem by breaking it into smaller pieces and updating the variables alternately. Still, at the same time, it also introduces several subproblems which may cost much time. To be more specific, the ADMM proposed by Weylandt [19] requires solving the Sylvester equation in the step of updating the variable U. Hence, the Schur decomposition requires solving the Sylvester equation based on the numerical method from [25], which is complicated and time consuming. Additionally, it is known that ADMM exhibits $O\left(\frac{1}{k}\right)$, where k is the number of iterations, convergence in general [26]. It often takes time to achieve relatively high precision [27], which is not feasible in some highly accurate applications. For example, the gene expression data contain huge information (the feature dimension usually exceeds 1000). However, COBRA and ADMM do not scale well for such large-scale problems. Overall, the above algorithm shows weak performance, which motivates us to combine some current algorithms to efficiently solve the convex biclustering problem like in reference [28].

This paper proposes an efficient algorithm with simple subproblems and a fast convergence rate to solve the convex biclustering problem. Rather than update each variable alternately, like ADMM, we use the augmented Lagrangian method (ALM) to update the primal variables simultaneously. In this way, we can transform the optimization problem to be differentiable, solve the problem via an efficient gradient descent method and further simplify the subproblems. Our proposed method is motivated by the work [29] in which the authors presented a way to convert the augmented Lagrangian function to a composite optimization that can be solved by the proximal gradient method [30]. Using the process twice to handle the two fused penalties $\{\lambda \parallel U_{i·}-U_{j·}{\parallel \}}_{i\ne j}$ and $\{\lambda \parallel U_{·m}-U_{·n}{\parallel \}}_{m\ne n}$, we obtain a differentiable problem from the augmented Lagrangian function. Then, we propose Nesterov’s accelerated gradient method to solve the differentiable problem, which has $O\left(\frac{1}{k^2}\right)$ global convergence rate.

Our main contributions are as follows:

• We propose an efficient algorithm to solve the convex biclustering model for large-scale N and p. The algorithm is a first-order method with simple subproblems. It only requires calculating the matrix multiplications and simple proximal operators, while the ADMM approaches require matrix inversion.
• Our proposed method does not require as much computation, even when the tuning parameter $\lambda$ is large, as the existing approaches do, which means that it is easier to obtain biclustering results simultaneously for several $\lambda$ values.

The remaining parts of this paper are as follows. In Section 2, we provide some preliminaries which are used in the paper and introduce the convex biclustering problem. In Section 3, we illustrate our proposed algorithm for solving the convex biclustering model. After that, we conduct numerical experiments to evaluate the performance of our algorithm in Section 4.

Notation: In this paper, we use ${\left|\right|x\left|\right|}_p$ to denote the ${\ell }_p$ norm of a vector $x\in {\mathbb{R}}^d$, ${\left|\right|x\left|\right|}_p:=({\sum }_{i=1}^d|x_i{{|}^p)}^{\frac{1}{p}}$ for $p\in [1,\infty )$, and ${\left|\right|x\left|\right|}_{\infty }:={max}_i\left|x_i\right|$. For a matrix $X\in {\mathbb{R}}^{p\times q}$, we use ${\left|\right|X\left|\right|}_F$ to denote the Frobenius norm, ${\left|\right|X\left|\right|}_2$ denotes the spectral norm, and ${\left|\right|X\left|\right|}_1:={\sum }_{i=1}^p{\sum }_{j=1}^q\left|x_{ij}\right|$ if not specified.

2. Preliminaries

In this section, we provide the background for understanding the proposed method in Section 3. In particular, we introduce the notions of ADMM, ALM, NAGM, and convex biclustering.

We say that differentiable $f:{\mathbb{R}}^n\to \mathbb{R}$ has a Lipschitz-continuous gradient if there exists $L>0$ (Lipschitz constant) such that

$$\left|\right|\nabla f\left(x\right)-{\nabla f\left(y\right)\left|\right|}_2\le L\left|\right|x-{y\left|\right|}_2,\forall x,y\in {\mathbb{R}}^n.$$

(1)

We define the conjugate of a function $f:{\mathbb{R}}^n\to \mathbb{R}$ by

$$f^{*}\left(y\right)=\underset{x\in \mathrm{dom}f}{sup}(y^{T}x-f\left(x\right)),$$

where $\mathrm{dom}f\subseteq {\mathbb{R}}^n$ is the domain of f, and know that $f^{*}$ is closed ($\{x\in \mathrm{dom}\left(f^{*}\right)|f^{*}\left(x\right)\le \alpha \}$ is a closed set for any $\alpha \in \mathbb{R}$) and convex. It is known that ${\left(f^{*}\right)}^{*}=f$ when f is closed and convex, and that Moreau’s decomposition [31] is available. Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be closed and convex. For any $x\in {\mathbb{R}}^n$ and $\gamma >0$, we have

$$pro{x}_{\gamma f}\left(x\right)+\gamma pro{x}_{{\gamma }^{-1}{f}^{*}}\left({\gamma }^{-1}x\right)=x.$$

(2)

where ${\mathrm{prox}}_f:{\mathbb{R}}^n\to {\mathbb{R}}^n$ is the proximal operator defined by

$${\mathrm{prox}}_f\left(x\right):=arg\underset{y\in {\mathbb{R}}^n}{min}\{\frac{1}{2}\left|\right|x-{y\left|\right|}_2^2+f\left(y\right)\}.$$

(3)

The relation (2) is derived from ${\left(f^{*}\right)}^{*}=f$ and the definition (3) [31].

2.1. ADMM and ALM

In this subsection, we introduce the general optimization procedures of ADMM and ALM.

Let $f,g:{\mathbb{R}}^n\to \mathbb{R}$ and $h:{\mathbb{R}}^p\to \mathbb{R}$ be convex. Assume that f is differentiable, and g and h are not necessarily differentiable. We consider the following optimization problem:

$$\begin{array}{cc}\hfill & \underset{x,y}{min}f\left(x\right)+h\left(y\right)+g\left(x\right)\hfill \\ \hfill & \mathrm{subject}\mathrm{to}Ax=y,\hfill \end{array}$$

(4)

with variables $x\in {\mathbb{R}}^n$ and $y\in {\mathbb{R}}^p$, and matrix $A\in {\mathbb{R}}^{p\times n}$. To this end, we define the augmented Lagrangian function as the following,

$$L_{\nu }(x,y,\lambda ):=f\left(x\right)+g\left(x\right)+h\left(y\right)+⟨\lambda ,Ax-y⟩+\frac{\nu }{2}\left|\right|Ax-{y\left|\right|}_2^2,$$

(5)

where $\nu >0$ is an augmented Lagrangian parameter, and $\lambda \in {\mathbb{R}}^p$ is the Lagrangian multipliers.

ADMM is a general procedure to find the solution to the problem (4) by iterating

$$\begin{array}{cc}\hfill x^{k+1}& :=arg\underset{x}{min}{L}_{\nu }(x,y^k,{\lambda }^k),\hfill \\ \hfill y^{k+1}& :=arg\underset{y}{min}{L}_{\nu }(x^{k+1},y,{\lambda }^k),\hfill \\ \hfill {\lambda }^{k+1}& :={\lambda }^k+\nu (A{x}^{k+1}-y^{k+1}),\hfill \end{array}$$

given the initial values $y^1$ and ${\lambda }^1$.

What we mean by the ALM [32,33,34] is to minimize the augmented Lagrangian function (5) w.r.t. variables x and y simultaneously given a $\lambda$ value, i.e., we iterate the following steps:

$$\begin{array}{cc}\hfill (x^{k+1},y^{k+1})& :=arg\underset{x,y}{min}{L}_{\nu }(x,y,{\lambda }^k),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\lambda }^{k+1}& :={\lambda }^k+\nu (A{x}^{k+1}-y^{k+1}).\hfill \end{array}$$

(7)

Shimmura and Suzuki [29] considered the minimization of the function $\varphi :{\mathbb{R}}^n\to \mathbb{R}$,

$$\varphi \left(x\right):=f\left(x\right)+\underset{y}{min}\{h\left(y\right)+⟨\lambda ,Ax-y⟩+\frac{\nu }{2}\left|\right|Ax-{y\left|\right|}_2^2\},$$

and the non-differentiable function $g\left(x\right)$ over $x\in {\mathbb{R}}^n$, replacing the minimization over $x\in {\mathbb{R}}^n$ and $y\in {\mathbb{R}}^n$ in (6).

Lemma 1

([29], Theorem 1). The function $\varphi \left(x\right)$ is differentiable and its differential is

$$\nabla \varphi \left(x\right)=\nabla f\left(x\right)+A^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu Ax+\lambda )\right).$$

By Lemma 1, the minimization in (6) can be regarded as the composite optimization problem with the differentiable function $\varphi \left(x\right)$ and the non-differentiable function $g\left(x\right)$. Therefore, it is feasible to use the proximal gradient method to update the variable x, such as the fast iterative shrinkage-thresholding algorithm (FISTA) [30].

2.2. Nesterov’s Accelerated Gradient Method

Nesterov [35] proposed a variant of the gradient descent method for Lipschitz differentiable functions. It has $O\left(\frac{1}{k^2}\right)$ convergence rate while the (traditional) gradient descent method has $O\left(\frac{1}{k}\right)$ [35]. Considering the minimization of a convex and differentiable function $F\left(x\right)$, NAGM is described in Algorithm 1 when $\nabla F\left(x\right)$ has a Lipschitz constant L in (1).

Algorithm 1 NAGM.

Input: Lipschitz constant L, initial value $x^0=y^0$, $t^1=1$. While $k<k_{max}$ (until convergence) do   1: $x^{k+1}=y^k-\frac{1}{L}\nabla F\left(y^k\right)$   2: $t^{k+1}=\frac{1+\sqrt{4{t^k}^2+1}}{2}$   3: $y^{k+1}=x^{k+1}+\frac{t^k-1}{t^{k+1}}(x^{k+1}-x^k)$   4: $k=k+1$ End while

Algorithm 1 replaces the gradient descent $y^{k+1}=y^k-\frac{1}{L}\nabla F\left(y^k\right)$ by Steps 1 to 3: Step 1 executes the gradient descent to obtain $x^{k+1}$ from $y^k$, Steps 2 and 3 calculate new $y^{k+1}$ based on the previous $x^k,x^{k+1}$, and then return to the gradient descent in Step 1. NAGM assumes that F is differentiable, while FISTA [30], an accelerated version of ISTA [30], deals with non-differentiable F using the proximal gradient descent (see Table 1).

Table 1. Gradient descent and its modifications.

	Differentiable	Non-Differentiable
Ordinary	Gradient descent	Proximal gradient descent (e.g., ISTA [30])
Accelated	NAGM [35]	FISTA [30]

2.3. Convex Biclustering

We consider the convex biclustering problem in a general setting. Suppose we have a data matrix $X=\left(x_{ij}\right)$ consisting of N observations, $i=1,\dots ,N$, w.r.t. p features $j=1,\dots ,p$. Our task is to assign each observation to one of the non-overlapped row clusters $C_1,\cdots ,C_R\subseteq \{1,\dots ,N\}$ and assign each feature to one of the non-overlapped column clusters $D_1,\cdots ,D_K\subseteq \{1,\dots ,p\}$. We assume that the clusters $C_1,\cdots ,C_R$ and $D_1,\cdots ,D_K$ and the values of R and K are not known a priori.

More precisely, the convex biclustering in this paper is formulated as follows:

$$\underset{U\in {\mathbb{R}}^{N\times p}}{min}\frac{1}{2}\left|\right|X-{U\left|\right|}_F^2+\lambda \left(\sum _{i=1}^{N-1}\sum _{j=i+1}^N{\omega }_{ij}\left|\right|U_{i·}-U_{j·}{\left|\right|}_2+\sum _{m=1}^{p-1}\sum _{n=m+1}^p{\tilde{\omega }}_{mn}\left|\right|U_{·m}-U_{·n}{\left|\right|}_2\right),$$

(8)

where $U_{i·}$ and $U_{·j}$ are the i-th row and j-th column of $U\in {\mathbb{R}}^{N\times p}$. Chi et al. [9] suggested a requirement on the weights selection:

$${\omega }_{ij}:=1_{i,j}^{k}exp\left(-\varphi \parallel x_{i·}-x_{j·}{\parallel }_2^2\right)$$

and

$${\tilde{\omega }}_{mn}:=1_{m,n}^{k}exp\left(-\tilde{\varphi }{\parallel x_{·m}-x_{·b}\parallel }_2^2\right),$$

where $1_{i,j}^k$ is 1 if j belongs to the i’s k-nearest neighbors and 0 otherwise. $1_{m,n}^k$ is defined similarly (the parameter k should be specified beforehand), and $x_{i·}$ and $x_{·j}$ are the i-th row and j-th column of the matrix X. They suggested that the constants $\varphi$ and $\tilde{\varphi }$ are determined so that the sums ${\sum }_{i=1}^{N-1}{\sum }_{j=i+1}^N{\omega }_{ij}$ and ${\sum }_{m=1}^{p-1}{\sum }_{n=m+1}^p{\tilde{\omega }}_{mn}$ are $N^{-1/2}$ and $p^{-1/2}$, respectively.

Chi et al. [9] proposed COBRA to solve the problem (8). Essentially, COBRA solves the standard convex clustering problems of rows and columns alternately,

$$\underset{U\in {\mathbb{R}}^{N\times p}}{min}\frac{1}{2}\left|\right|X-{U\left|\right|}_F^2+\lambda \sum _{i=1}^{N-1}\sum _{j=i+1}^N{\omega }_{ij}\left|\right|U_{i·}-U_{j·}{\left|\right|}_2$$

(9)

and

$$\underset{U\in {\mathbb{R}}^{N\times p}}{min}\frac{1}{2}\left|\right|X-{U\left|\right|}_F^2+\lambda \sum _{m=1}^{p-1}\sum _{n=m+1}^p{\tilde{\omega }}_{mn}\left|\right|U_{·m}-U_{·n}{\left|\right|}_2,$$

(10)

until the solution converges. However, both optimization problems contain a non-differentiable ${\ell }_2$ norm term, and solving the convex clustering problems (9) and (10) take much time [23]. Later, the ADMM-based approaches considered alternating variables procedures and outperformed the COBRA for large parameter $\lambda$ [19].

3. The Proposed Method and Theoretical Analysis

In this section, we first show that the whole terms in the augmented Lagrangian function of (8) can be differentiable w.r.t. U after introducing two dual variables. Therefore, we use NAGM rather than FISTA in [29], where assumes the objective function contains a non-differentiable term.

In order to make the notation clear, we formulate the problem (8) in another way. Let ${ϵ}_1$ and ${ϵ}_2$ be the sets $\left\{(i,j)\right|{\omega }_{ij}>0,i<j\}$ and $\left\{(m,n)\right|{\tilde{\omega }}_{mn}>0,m<n\}$, respectively, and denote the cardinality of a set S by $\left|S\right|$. We define the matrices $C\in {\mathbb{R}}^{|{ϵ}_1|\times n}$ and $D\in {\mathbb{R}}^{p\times |{ϵ}_2|}$ by

$$C_{l,i}=1,C_{l,j}=-1,C_{l,k}=0,k\ne i,j⟺l=(i,j)\in {ϵ}_1,$$

$$C={\left[\begin{array}{cccccc}1& -1& 0& \cdots & 0& 0\\ 1& 0& -1& \cdots & 0& 0\\ ⋮& & & \ddots & & ⋮\\ 0& 0& 0& \cdots & 1& -1\end{array}\right]}_{|{ϵ}_1|\times n},$$

and

$$D_{m,l}=1,D_{n,l}=-1,D_{k,l}=0,k\ne m,n⟺l=(m,n)\in {ϵ}_2,$$

$$D={\left[\begin{array}{cccc}1& 1& \cdots & 0\\ -1& 0& \cdots & 0\\ 0& -1& \cdots & 0\\ ⋮& & \ddots & ⋮\\ 0& 0& \cdots & 1\\ 0& 0& \cdots & -1\end{array}\right]}_{p\times |{ϵ}_2|},$$

respectively.

Then, the optimization problem (8) can be reformulated as follows:

$$\begin{array}{c}\hfill \underset{U\in {\mathbb{R}}^{N\times p}}{min}\frac{1}{2}\left|\right|X-{U\left|\right|}_F^2+\lambda \left(\sum _{l\in {ϵ}_1}{\omega }_l\left|\right|C_{l,·}{U\left|\right|}_2+\sum _{l\in {ϵ}_2}{\tilde{\omega }}_l\left|\right|U{D}_{·,l}{\left|\right|}_2\right).\end{array}$$

(11)

3.1. The ALM Formulation

To implement ALM, we further construct the problem (11) into the following constrained optimization problem by introducing the dual variables $V\in {\mathbb{R}}^{|{ϵ}_1|\times p}$ and $Z\in {\mathbb{R}}^{N\times |{ϵ}_2|}$,

$$\begin{array}{cc}\hfill \underset{U,V,Z}{min}& \frac{1}{2}\left|\right|X-{U\left|\right|}_F^2+\lambda \left(\sum _{l\in {ϵ}_1}{\omega }_l\left|\right|V_l{\left|\right|}_2+\sum _{l\in {ϵ}_2}{\tilde{\omega }}_l\left|\right|Z_l{\left|\right|}_2\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& C_{l,·}U-V_l=0,\forall l\in {ϵ}_1,\hfill \\ \hfill & U{D}_{·,l}-Z_l=0,\forall l\in {ϵ}_2,\hfill \end{array}$$

(12)

where $V_l$ and $Z_l$ are the l-th row and l-th column of V and Z, respectively. If we introduce the following functions,

$$\begin{array}{cc}\hfill f\left(U\right):& =\frac{1}{2}\left|\right|X-{U\left|\right|}_F^2\hfill \\ \hfill h\left(V\right):& =\lambda \sum _{l\in {ϵ}_1}{\omega }_l\left|\right|V_l{\left|\right|}_2\hfill \\ \hfill g\left(Z\right):& =\lambda \sum _{l\in {ϵ}_2}{\tilde{\omega }}_l\left|\right|Z_l{\left|\right|}_2,\hfill \end{array}$$

then the problem in (12) becomes

$$\begin{array}{c}\hfill \underset{U,V,Z}{min}\left\{f\left(U\right)+h\left(V\right)+g\left(Z\right)\right\}.\end{array}$$

(13)

The augmented Lagrangian function of the problem (12) is given by

$$\begin{array}{cc}\hfill L_{\nu }(U,V,Z,{\Lambda }_1,{\Lambda }_2):=& f\left(U\right)+h\left(V\right)+\sum _{l\in {ϵ}_1}⟨{\Lambda }_{1l},C_{l,·}U-V_l⟩+\frac{\nu }{2}\sum _{l\in {ϵ}_1}\left|\right|C_{l,·}U-V_l{\left|\right|}_2^2\hfill \\ \hfill & +g\left(Z\right)+\sum _{l\in {ϵ}_2}⟨{\Lambda }_{2l},U{D}_{·,l}-Z_l⟩+\frac{\nu }{2}\sum _{l\in {ϵ}_2}\left|\right|U{D}_{·,l}-Z_l{\left|\right|}_2^2,\hfill \end{array}$$

(14)

where $\nu >0$ is an augmented Lagrangian penalty, ${\Lambda }_1\in {\mathbb{R}}^{|{ϵ}_1|\times p}$ and ${\Lambda }_2\in {\mathbb{R}}^{N\times |{ϵ}_2|}$ are Lagrangian multipliers, and ${\Lambda }_{1l}$ and ${\Lambda }_{2l}$ are the l-th row and l-th column of ${\Lambda }_1$ and ${\Lambda }_2$, respectively.

Hence, the ALM procedure of the problem (12) consists of the following three steps:

$$\begin{array}{cc}\hfill (U^k,V^k,Z^k)& =arg\underset{U,V,Z}{min}{L}_{\nu }(U,V,Z,{\Lambda }_1^{k-1},{\Lambda }_2^{k-1}),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\Lambda }_{1l}^k& ={\Lambda }_{1l}^{k-1}+\nu (C_{l,·}{U}^k-V_l^k),\forall l\in {ϵ}_1,\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\Lambda }_{2l}^k& ={\Lambda }_{2l}^{k-1}+\nu (C_{l,·}{U}^k-V_l^k),\forall l\in {ϵ}_2.\hfill \end{array}$$

(17)

3.2. The Proposed Method

We construct our proposed method: repeatedly minimizing the augmented Lagrangian function in Equation (15) w.r.t $U,V_l,Z_l$ and updating the Lagrange multipliers in Equations (16) and (17). The whole procedure is summarized in Algorithm 2.

Algorithm 2 Proposed method.

Input: Data X, matrices C and D, Lipschitz constant L calculated by (27), penalties $\lambda$ and $\nu$, initial value ${\Lambda }_1^0,{\Lambda }_2^0$, $Y^1$, $t^1=1$. While $k<k_{max}$ (until convergence) do   1: Calculate the gradient $\nabla F\left(Y^k\right)$ by (20).   2: Update iterate: $U^k\leftarrow Y^k-\frac{1}{L}\nabla F\left(Y^k\right)$.   3: Update iterate: ${\Lambda }_{1l}^k\leftarrow P_{{\mathcal{B}}_l}({\Lambda }_{1l}^{k-1}+\nu C_{l,·}{U}^k)$, $\mathrm{for}l\in {ϵ}_1$, by (24) and (25), where ${\mathcal{B}}_l:={\{y:|\left|y\right||}_2\le \lambda {\omega }_l\}$.   4: Update iterate: ${\Lambda }_{2l}^k\leftarrow P_{{\tilde{\mathcal{B}}}_l}({\Lambda }_{2l}^{k-1}+\nu U^k{D}_{·,l})$, $\mathrm{for}l\in {ϵ}_2$, by (26) and (25), where ${\tilde{\mathcal{B}}}_l:=\{\tilde{y}:\left|\right|\tilde{y}{\left|\right|}_2\le \lambda {\tilde{\omega }}_l\}$.   5: $t^{k+1}=\frac{1+\sqrt{1+4{t^k}^2}}{2}$   6: $Y^{k+1}=U^k+\frac{t^k-1}{t^{k+1}}(U^k-U^{k-1})$   7: $k=k+1$ Output: optimal solution to problem (8), $U^{*}=U^k$.

Step 1: update $\mathit{U}$. In the U-update, if we define the following function in Equation (14),

$$\begin{array}{cc}\hfill F\left(U\right):& =f\left(U\right)+\underset{V,Z}{min}\left\{L(U,V,Z,{\Lambda }_1,{\Lambda }_2)\right\}\hfill \\ \hfill & =f\left(U\right)+\underset{V,Z}{min}\left\{h\left(V\right)+\sum _{l\in {ϵ}_1}⟨{\Lambda }_{1l},C_{l,·}U-V_l⟩+\frac{\nu }{2}\sum _{l\in {ϵ}_1}\left|\right|C_{l,·}U-V_l{\left|\right|}_2^2\right.\hfill \\ \hfill & \left.+g\left(Z\right)+\sum _{l\in {ϵ}_2}⟨{\Lambda }_{2l},U{D}_{·,l}-Z_l⟩+\frac{\nu }{2}\sum _{l\in {ϵ}_2}\left|\right|U{D}_{·,l}-Z_l{\left|\right|}_2^2\right\},\hfill \end{array}$$

(18)

then the update of U in (15) can be written as

$$U^{k+1}:=arg\underset{U}{min}F\left(U^k\right).$$

(19)

We find that (18) is differentiable due to Lemma 1, and obtain the following proposition.

Proposition 1.

The function $F\left(U\right)$ is differentiable with respect to U, and

$${\nabla }_{U}F\left(U\right)=-X+U+C^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu CU+{\Lambda }_1)\right)+\left({\mathrm{prox}}_{\nu g^{*}}(\nu UD+{\Lambda }_2)\right)D^T.$$

(20)

For the proof, see the Appendix A.1.

With Proposition 1, we can use NAGM (Algorithm 1) to update U by solving the differentiable optimization problem (19).

Step 2: update ${\mathit{V}}_{\mathit{l}}$ and ${\Lambda }_{\mathbf{1}}$. By step (15) in the ALM procedure, we must minimize the functions in Equation (18) corresponding to the vector $V_l$ by updating the following,

$$\begin{array}{cc}\hfill V_l^k& =arg\underset{V_l}{min}\left\{\lambda {\omega }_l\left|\right|V_l{\left|\right|}_2+\frac{\nu }{2}\left|\right|V_l{\left|\right|}_2^2-\sum _{l\in {ϵ}_1}⟨{\Lambda }_{1l}^{k-1}+\nu C_{l,·}{U}^k,V_l⟩\right\}\hfill \\ \hfill & =arg\underset{V_l}{min}\left\{\frac{\nu }{2}\left|\right|V_l-(C_{l,·}{U}^k+{\nu }^{-1}{\Lambda }_{1l}^k){\left|\right|}_2^2+h_l\left(V_l\right)\right\}\hfill \\ \hfill & ={\mathrm{prox}}_{h_l/\nu }(C_{l,·}{U}^k+{\nu }^{-1}{\Lambda }_{1l}^{k-1}),\hfill \end{array}$$

where $h_l\left(V_l\right):=\lambda {\omega }_l\left|\right|V_l{\left|\right|}_2$ denotes the l-th term in $h\left(V\right)$.

We substitute the optimal $V_l^k$ in the k-th iteration into step ()

$${\Lambda }_{1l}^k\leftarrow {\Lambda }_{1l}^{k-1}+\nu (C_{l,·}{U}^k-V_l^k),\forall l\in {ϵ}_1,$$

to obtain

$${\Lambda }_{1l}^k\leftarrow {\Lambda }_{1l}^{k-1}+\nu C_{l,·}{U}^k-\nu {\mathrm{prox}}_{h_l/\nu }(C_{l,·}{U}^k+{\nu }^{-1}{\Lambda }_{1l}^{k-1}).$$

(21)

By Moreau’s decomposition (2), we further simplify the update (21) as follows,

$${\Lambda }_{1l}^k\leftarrow {\mathrm{prox}}_{\nu h_l^{*}}({\Lambda }_{1l}^{k-1}+\nu C_{l,·}{U}^k),$$

(22)

which means the updates of $V_l$ and ${\Lambda }_{1l}$ become one update (22). Hence, there is no longer a need to store and compute the variable $V_l$ in the ALM updates, which reduces computational costs.

In the update (22), the conjugate function $h_l^{*}\left(y\right)$ of the ${\ell }_2$ norm is an indicator function ([36], Example 3.26):

$$\begin{array}{c}\hfill h_l^{*}\left(y\right)=\left\{\begin{array}{cc}0,& {\mathrm{if}\left|\right|y\left|\right|}_2\le \lambda {\omega }_l,\\ \infty ,& \mathrm{otherwise}.\end{array}\right.\end{array}$$

(23)

Moreover, the proximal operator of the indicator function (22) is the projection problem ([37], Theorem 6.24):

$${\mathrm{prox}}_{\nu h_l^{*}}(\nu C_{l,·}{U}^k+{\Lambda }_{1l}^{k-1})=P_{{\mathcal{B}}_l}(\nu C_{l,·}{U}^k+{\Lambda }_{1l}^{k-1}),$$

(24)

where ${\mathcal{B}}_l:={\{y:|\left|y\right||}_2\le \lambda {\omega }_l\}$, and the operator $P_{{\mathcal{B}}_l}$ denotes the projection onto the ball ${\mathcal{B}}_l$. It solves the problem $P_{{\mathcal{B}}_l}\left(x\right):=arg{min}_{u\in {\mathcal{B}}_l}\left|\right|u-{x\left|\right|}_2^2$, i.e.,

$$\begin{array}{c}\hfill P_{{\mathcal{B}}_l}\left(x\right)=\left\{\begin{array}{cc}x,& {\mathrm{if}\left|\right|x\left|\right|}_2\le \lambda {\omega }_l,\\ \lambda {\omega }_l,& \mathrm{otherwise}.\end{array}\right.\end{array}$$

(25)

This projection problem completes in $O\left(p\right)$ operations for a p-dimensional vector $x\in {\mathbb{R}}^p$.

Step 3: update Z and ${\Lambda }_2$. Similarly, we can derive the following equations:

$$\begin{array}{cc}\hfill Z_l^{k+1}& =arg\underset{Z_l}{min}\left\{\lambda \widetilde{{\omega }_l}\left|\right|Z_l{\left|\right|}_2+\frac{\nu }{2}\left|\right|Z_l{\left|\right|}_2^2-\sum _{l\in {ϵ}_2}⟨{\Lambda }_{2l}^{k-1}+\nu U^k{D}_{·,l},Z_l⟩\right\}\hfill \\ \hfill & =arg\underset{Z_l}{min}\left\{\frac{\nu }{2}\left|\right|Z_l-(U^k{D}_{·,l}+{\nu }^{-1}{\Lambda }_{2l}^{k-1}){\left|\right|}_2^2+g_l\left(Z_l\right)\right\}\hfill \\ \hfill & ={\mathrm{prox}}_{g_l/\nu }(U^k{D}_{·,l}+{\nu }^{-1}{\Lambda }_{2l}^{k-1})\hfill \end{array}$$

where $g_l\left(Z_l\right):=\lambda \widetilde{{\omega }_l}\left|\right|Z_l{\left|\right|}_2$. Then, the dual variable ${\Lambda }_2$ update becomes

$${\Lambda }_{2l}^k\leftarrow {\mathrm{prox}}_{\nu g_l^{*}}({\Lambda }_{2l}^{k-1}+\nu U^k{D}_{·,l}),\mathrm{for}l\in {ϵ}_2.$$

If we write in the projection operator, then it becomes

$${\Lambda }_{2l}^k\leftarrow P_{{\tilde{\mathcal{B}}}_l}({\Lambda }_{2l}^{k-1}+\nu U^k{D}_{·,l})$$

(26)

where ${\tilde{\mathcal{B}}}_l:=\{\tilde{y}:\left|\right|\tilde{y}{\left|\right|}_2\le \lambda {\tilde{\omega }}_l\}$.

Our proposed method only uses first-order information. Furthermore, we just need to calculate the gradient of the function F and proximal operators in each iteration, where the proximal operators are easy to obtain by solving the projection problem.

3.3. Lipschitz Constant and Convergence Rate

By the following lemma, we know that if we choose $\frac{1}{L}$ as the step size for each iteration in the NAGM, then the convergence rate is, at most, $O\left(\frac{1}{k^2}\right)$.

Lemma 2

([30,35,38]). Let $\left\{U^k\right\}$ as the sequence generated by Algorithm 1, and $U_0$ as an initial value. If we take the step size as $\frac{1}{L}$, then for any $k\ge 1$ we have

$$F\left(U^k\right)-F\left(U^{*}\right)\le \frac{2L\left|\right|U_0-U^{*}{\left|\right|}_F^2}{{(k+1)}^2}.$$

In order to examine the performance of the proposed method, we derive the Lipschitz constant L of ${\nabla }_{U}F\left(U\right)$ as in the following proposition.

Proposition 2.

The Lipschitz constant of ${\nabla }_{U}F\left(U\right)$ is upperbounded by

$$1+\nu {\lambda }_{max}\left(C^{T}C\right)+\nu {\lambda }_{max}\left(D^{T}D\right),$$

(27)

where ${\lambda }_{max}$ denotes the maximum eigenvalue of the corresponding matrix.

Proof. 

By definition in (1) and Proposition 1, we derive the Lipschitz constant as follows,

$$\begin{array}{cc}\hfill \left|\right|{\nabla }_{U}F\left(U_1\right)-{\nabla }_{U}F\left(U_2\right){\left|\right|}_2& =\left|\right|U_1-U_2+C^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu C{U}_1+{\Lambda }_1)\right)-C^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu C{U}_2+{\Lambda }_1)\right)\hfill \\ \hfill & +\left({\mathrm{prox}}_{\nu g^{*}}(\nu U_{1}D+{\Lambda }_2)\right)D^T-\left({\mathrm{prox}}_{\nu g^{*}}(\nu U_{2}D+{\Lambda }_2)\right)D^T{\left|\right|}_2\hfill \\ \hfill & \le \left|\right|U_1-U_2{\left|\right|}_2+\left|\right|C^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu C{U}_1+{\Lambda }_1)\right)-C^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu C{U}_2+{\Lambda }_1)\right){\left|\right|}_2\hfill \\ \hfill & +\left|\right|\left({\mathrm{prox}}_{\nu g^{*}}(\nu U_{1}D+{\Lambda }_2)\right)D^T-\left({\mathrm{prox}}_{\nu g^{*}}(\nu U_{2}D+{\Lambda }_2)\right)D^T{\left|\right|}_2.\hfill \end{array}$$

By the definition of matrix 2-norm and the nonexpansiveness of the proximal operators ([39], Lemma 2.4), we obtain

$$\begin{array}{cc}\hfill \left|\right|{\nabla }_{U}F\left(U_1\right)-{\nabla }_{U}F\left(U_2\right){\left|\right|}_2& \le \left|\right|U_1-U_2{\left|\right|}_2+\sqrt{{\lambda }_{max}\left(C^{T}C\right)}\left|\right|\nu C{U}_1-\nu C{U}_2{\left|\right|}_2+\left|\right|\nu U_{1}D-\nu U_{2}D{\left|\right|}_2\sqrt{{\lambda }_{max}\left(D^{T}D\right)}\hfill \\ \hfill & \le \left|\right|U_1-U_2{\left|\right|}_2+\nu {\lambda }_{max}\left(C^{T}C\right)\left|\right|U_1-U_2{\left|\right|}_2+\nu {\lambda }_{max}\left(D^{T}D\right)\left|\right|U_1-U_2{\left|\right|}_2\hfill \\ \hfill & \le \left(1+\nu {\lambda }_{max}\left(C^{T}C\right)+\nu {\lambda }_{max}\left(D^{T}D\right)\right)\left|\right|U_1-U_2{\left|\right|}_2\hfill \end{array}$$

□

Finally, it should be noted that for the time complexity, the proposed method is less sensitive to the $\lambda$ value than the conventional methods. In fact, the $\lambda$ value affects the proposed method only through the functions $h_l^{*}$ and $g_l^{*}$ that take 0 or ∞ depending on ${\parallel y\parallel }_2\le \lambda {\omega }_l$ and $\parallel \tilde{y}{\parallel }_2\le \lambda {\tilde{\omega }}_l$ in (23).

On the other hand, the COBRA solves two optimization problems, and the ADMM-based methods need to solve the Sylvester equation, which means that all of them are influenced by the $\lambda$ value so much.

4. Experiments

In this section, we show the performance of the proposed approach for estimating and assessing the biclusters by conducting experiments on both synthetic and real datasets. We executed the following algorithms:

• COBRA: Dykstra-like proximal algorithm proposed by Chi et al. [9].
• ADMM: the ADMM proposed by Weylandt [19].
• G-ADMM (generalized ADMM): the modified ADMM presented by Weylandt [19].
• Proposed method: the proposed algorithm showed in Algorithm 2.

They were all implemented by Rcpp on a Macbook Air with 1.6 GHz Intel Core i5 and 8 GB memory. We recorded the wall times for the four algorithms.

4.1. Artificial Data Analysis

We evaluate the performance of the proposed methods on synthetic data in terms of the number of iterations, the execution time, and the clustering quality.

We generate the artificial data $X\in {\mathbb{R}}^{N\times p}$ with a checkerboard bicluster structure similar to the method in [9]. We simulate $X_{ij}\sim N({\mu }_{rc},{\sigma }^2)$ (i.i.d.) as follows, where the indices r and c range in clusters $\{1,\cdots ,R\}$ and $\{1,\cdots ,C\}$, respectively, which means that the number of biclusters is $M:=R\times C$. We assign each $x_{ij}$ randomly belongs to one of those M biclusters. The mean ${\mu }_{rc}$ is chosen uniformly from an equally spaced sequence $\{-10,-9,\cdots ,9,10\}$, and the $\sigma$ is chosen as $1.5$ and $3.0$ for different noise levels.

In our experiments, we consider the following stopping criteria for the four algorithms.

• Relative error:

  $$\frac{\left|\right|U^{k+1}-U^k{\left|\right|}_F}{max\left\{\right|\left|U^k\right|{|}_F,1\}}\le ϵ.$$
• Objective function error:

  $$\left|\right|F\left(U^k\right)-F\left(U^{*}\right){\left|\right|}_F\le ϵ.$$

where $ϵ$ is a given accuracy tolerance. We terminate the algorithm if the above error is smaller than $ϵ$ or the maximum number of iterations exceeds 10,000. We use the relative error for the time comparisons and quality assessment and the objective function error for convergence rate analysis.

4.1.1. Comparisons

We change the sizes of $N,p$ of the data matrix X and the tuning parameter $\lambda$ to test the performance of four algorithms and compare the performance among the algorithms. At first, we compare the execution time with different $\lambda$, ranging from 1 to 2000, and setting $R=4$, $C=4$, $\sigma =1.5$. We obtain the results shown in Figure 2a.

Figure 2. Execution time for various $\lambda$ and p with $N=100$ and $ϵ=1\times {10}^{-6}$. (a) Different $\lambda$, with $p=40$; (b) different p, with $\lambda =1$.

From Figure 2a, we observe that the execution time of the COBRA and G-ADMM increases rapidly as $\lambda$ varies. The execution time of COBRA is the largest when $\lambda >1400$. Therefore, it will take a long time for COBRA to visualize the whole fusion process, particularly the single bicluster case. Our proposed method significantly outperforms the other three algorithms and offers high stability in a wide range of $\lambda$. Due to its low computational time, our proposed method is a preferable choice to visualize the biclusters for various $\lambda$ values when applying biclustering.

Next, we compare the execution time with different p (from 1 to 200). Here, we fix the number of column clusters and row clusters ($C=R=4$). Figure 2b shows that the execution time of the algorithms increases as p grows. The proposed method shows better performance than the other three. In particular, the ADMM and G-ADMM are suffered from the feature dimension p and the computations grow dramatically.

Then, we vary the sample size N from 100 to 1000 and fix the size of the feature $p=40$, with $\lambda =1$, $R=4$, $C=4$, $\sigma =1.5$. Figure 3 shows the execution times of the three algorithms (COBRA, G-ADMM, and Proposed) for each N.

Figure 3. Execution times for each N with $\lambda =1,p=40$ and $ϵ=1\times {10}^{-6}$.

The curves reveal that when the larger the sample size N, the more time the algorithms require. Moreover, the ADMM takes more than 500 s when $N>500$ and takes around 1800 s when $N=1000$, which are much larger than the other three algorithms. Thus, we remove the result of ADMM from the figure. However, our proposed method only takes around 10 s even when $N=1000$, which is six times smaller than G-ADMM.

4.1.2. Assessment

We evaluate the clustering quality by a widely used criterion called the Rand index (RI) [40]. The value of RI ranges from 0 to 1; a higher value shows better performance, and 1 indicates the perfect quality of the clustering. Note that we can obtain the true bicluster labels in the data generation procedure. We generate the matrix data with $N=100$ and $p=100$, and set two noise levels, low ($\sigma =1.5$) and high ($\sigma =3.0$). We compare the clustering quality of our proposed method with ADMM, G-ADMM, and COBRA under different settings. Setting 1: $R=2$, $C=4$, $\sigma =1.5$; Setting 2: $R=4$, $C=4$, $\sigma =1.5$; Setting 3: $R=4$, $C=8$, $\sigma =1.5$; Setting 4: $R=2$, $C=4$, $\sigma =3$; Setting 5: $R=4$, $C=4$, $\sigma =3$; Setting 6: $R=4$, $C=8$, $\sigma =3$.

Table 2 presents the result of the experiment. As the tuning parameter $\lambda$ increases, the biclusters tend to fuse and reduce noise interference in the raw data. While in some cases, for extremely high $\lambda$ such as 10,000, the biclusters may be over-smoothed, and the value of the Rand index is decreased. For example, in the first case (Setting 1: the number of biclusters is $2\times 4$ and $\sigma =1.5$). The Rand index in COBRA, ADMM, and our proposed method shows a similar value in most cases because all the algorithms solve the same model. However, the G-ADMM exhibits the worst performance due to its slow convergence rate, and it cannot converge well when the tuning parameter $\lambda$ is large ($\lambda =5000$ and $\lambda =$ 10,000). Overall, from the results in Table 2, our proposed method shows high accuracy and stability from low to high noise.

Table 2. Assessment result.

Setting	Algorithm	Rand Index
		$\lambda =\mathbf{100}$	$\lambda =\mathbf{1000}$	$\lambda =\mathbf{5000}$	$\lambda =\mathbf{10},\mathbf{000}$
Setting 1	COBRA	0.874	0.875	0.999	0.931
	ADMM	0.872	0.875	0.999	0.931
	G-ADMM	0.874	0.875	0.874	0.872
	Proposed	0.875	0.875	0.999	0.931
Setting 2	COBRA	0.928	0.932	0.994	0.999
	ADMM	0.928	0.935	0.994	0.999
	G-ADMM	0.928	0.934	0.981	0.936
	Proposed	0.928	0.934	0.994	0.999
Setting 3	COBRA	0.959	0.962	0.962	0.999
	ADMM	0.961	0.962	0.962	0.998
	G-ADMM	0.959	0.962	0.967	0.967
	Proposed	0.961	0.962	0.962	0.999
Setting 4	COBRA	0.870	0.870	0.870	0.935
	ADMM	0.870	0.868	0.871	0.933
	G-ADMM	0.870	0.870	0.871	0.871
	Proposed	0.870	0.870	0.871	0.935
Setting 5	COBRA	0.934	0.934	0.934	0.964
	ADMM	0.934	0.932	0.934	0.964
	G-ADMM	0.934	0.934	0.932	0.932
	Proposed	0.934	0.934	0.934	0.964
Setting 6	COBRA	0.960	0.960	0.962	0.962
	ADMM	0.961	0.962	0.962	0.962
	G-ADMM	0.960	0.962	0.962	0.960
	Proposed	0.961	0.962	0.962	0.962

4.2. Real Data Analysis

In this section, we use three different real datasets to demonstrate the performance of our proposed method.

Firstly, we use the presidential speeches dataset preprocessed by Weylandt et al. [41] that contains 75 high-frequency words taken from the significant speeches of the 44 U.S. presidents around the year 2018. We show the heatmaps in Figure 4 under a wide range of tuning parameters $\lambda$ to exhibit the fusion process of biclusters. We set the tolerance $ϵ$ to be $1\times {10}^{-6}$, and use the relative error stopping criterion as described in Section 4.1. The columns represent the different presidents, and the rows represent the different words. When $\lambda =0$, the heatmap is disordered, and there are no distinct subgroups. While we increase the $\lambda$, the biclusters begin to merge. We can further find out the common vocabulary used in some groups of the prime minister’s speeches. Moreover, as shown in Figure 4f, the heatmap clearly shows four biclusters with two subgroups of presidents and two subgroups of words when $\lambda =$ 30,000.

Figure 4. The heatmap results of proposed method implementation on the presidential speeches dataset under a wide range of $\lambda$. (a) $\lambda =0$; (b) $\lambda =1500$; (c) $\lambda =2000$; (d) $\lambda =5000$; (e) $\lambda =$ 15,000; (f) $\lambda =$ 30,000.

Secondly, we compare the computational time of four algorithms for two actual datasets. One is The Cancer Genome Atlas (TCGA) dataset [42], which contains 438 breast cancer patients (samples) and 353 genes (features), and the other one is the diffuse large-B-cell lymphoma (DLBCL) dataset [43] with 3795 genes and 58 patient samples. In DLBCL, there are 32 samples from cured patients and 26 samples from sick individuals among the 58 samples. Furthermore, we extract 500 genes with the highest variances among the original genes.

Figure 5a,b depicts the outcomes of the elapsed time comparison. From the curves, we observe that our proposed approach surpasses the other three methods. In contrast, ADMM shows the worst performance in the DLBCL dataset, and the case of tolerance $ϵ<{10}^{-3}$ requirement in the TCGA dataset.

Figure 5. Plot of $log(F\left(U^k\right)-F\left(U^{*}\right))$ vs. the elapsed time. (a) TCGA; (b) DLBCL.

Lastly, we compare the number of iterations to achieve the specified tolerance of $F\left(U^k\right)-F\left(U^{*}\right)$ and run it on the TCGA and DLBCL datasets. Figure 6a,b reveals that the COBRA algorithm has the fastest convergence rate, whereas the generalized ADMM is the slowest to converge. Our proposed method shows competitive performance in the convergence rate.

Figure 6. Plot of $log(F\left(U^k\right)-F\left(U^{*}\right))$ vs. the number of iterations. (a) TCGA; (b) DLBCL.

Overall, from the above experiment results of the artificial and real datasets, our proposed method has superior computational performance with high accuracy.

5. Discussion

We proposed a method to find a solution to the convex biclustering problem. We found that it outperformed the conventional algorithms, such as COBRA and ADMM-based procedures, in the sense of efficiency. Our proposed method is more efficient than COBRA because the latter should solve two optimization problems containing non-differentiable fused terms in each cycle. Additionally, the proposed method performed better than the ADMM-based procedures because the former is based on [29] and uses the NAGM to update the variable U. However, ADMM spends much more time computing the matrix inverse. Moreover, our proposed method is stable while varying the tuning parameters $\lambda$, which is convenient for us to find the optimal $\lambda$ and visualize the variation of the heatmaps under a wide range of $\lambda$.

As for further improvements, we can use ADMM as a warm start strategy to select an initial value for our proposed method. What is more, according to the fusion process of the heatmap results in Figure 4, it will be meaningful if we can derive the range of tuning parameters $\lambda$ that yield the non-trivial solutions of the convex biclustering with more than one bicluster. Additionally, the proposed method can motivate future work. We can extend the proposed method to solve other clustering problems, such as the sparse singular value decomposition model [44] and the integrative generalized convex clustering model [45].

Appendix A.1. Proof of Proposition 1

First, we define the following two functions,

$$r_1\left(V\right):=h\left(V\right)+\frac{\nu }{2}{\left|\right|V\left|\right|}_F^2,$$

and

$$r_2\left(Z\right):=g\left(Z\right)+\frac{\nu }{2}{\left|\right|Z\left|\right|}_F^2.$$

By the definition of $F\left(U\right)$ in (18),

$$\begin{array}{cc}\hfill F\left(U\right):& =f\left(U\right)+\underset{V,Z}{min}\left\{L(U,V,Z,{\Lambda }_1,{\Lambda }_2)\right\}\hfill \\ \hfill & =f\left(U\right)+\underset{V,Z}{min}\left\{h\left(V\right)+g\left(Z\right)+\sum _l⟨{\Lambda }_{1l},C_{l,·}U-V_l⟩+\frac{\nu }{2}\sum _l\left|\right|C_{l,·}U-V_l{\left|\right|}_2^2\right.\hfill \\ \hfill & \left.+\sum _l⟨{\Lambda }_{2l},U{D}_{·,l}-Z_l⟩+\frac{\nu }{2}\sum _l\left|\right|U{D}_{·,l}-Z_l{\left|\right|}_2^2\right\}\hfill \\ \hfill & =f\left(U\right)+\underset{V}{min}\left\{h\left(V\right)+\frac{\nu }{2}{\left|\right|V\left|\right|}_F^2-\sum _l⟨{\Lambda }_{1l}+\nu C_{l,·}U,V_l⟩\right\}+\frac{\nu }{2}{\left|\right|CU\left|\right|}_F^2+\sum _l⟨{\Lambda }_{1l},C_{l,·}U⟩\hfill \\ \hfill & +\underset{Z}{min}\left\{g\left(Z\right)+\frac{\nu }{2}{\left|\right|Z\left|\right|}_F^2-\sum _l⟨{\Lambda }_{2l}+\nu U{D}_{·,l},Z_l⟩\right\}+\frac{\nu }{2}{\left|\right|UD\left|\right|}_F^2+\sum _l⟨{\Lambda }_{2l},U{D}_{·,l}⟩\hfill \\ \hfill & =-\underset{V}{max}\left\{⟨\nu CU+{\Lambda }_1,V⟩-h\left(V\right)-\frac{\nu }{2}{\left|\right|V\left|\right|}_F^2\right\}+f\left(U\right)+\frac{\nu }{2}{\left|\right|CU\left|\right|}_F^2+\sum _l⟨{\Lambda }_{1l},C_{l,·}U⟩\hfill \\ \hfill & -\underset{Z}{max}\left\{⟨\nu UD+{\Lambda }_2,Z⟩-g\left(Z\right)-\frac{\nu }{2}{\left|\right|Z\left|\right|}_F^2\right\}+\frac{\nu }{2}{\left|\right|UD\left|\right|}_F^2+\sum _l⟨{\Lambda }_{2l},U{D}_{·,l}⟩\hfill \\ \hfill & =f\left(U\right)-r_1^{*}(\nu CU+{\Lambda }_1)-r_2^{*}(\nu UD+{\Lambda }_2)+\frac{\nu }{2}{\left|\right|CU\left|\right|}_F^2+⟨{\Lambda }_1,CU⟩\hfill \\ \hfill & +\frac{\nu }{2}{\left|\right|UD\left|\right|}_F^2+⟨{\Lambda }_2,UD⟩.\hfill \end{array}$$

By Theorem 26.3 in [33], if the function $r:{\mathbb{R}}^p\to \mathbb{R}$ is closed and strongly convex, then we have the differentiable conjugate function $r^{*}\left(v\right)$, and

$$\nabla r^{*}\left(v\right)=arg\underset{u\in {\mathbb{R}}^p}{max}\{⟨u,v⟩-r\left(u\right)\}.$$

Hence, we can derive the following equations,

$$\begin{array}{cc}\hfill \nabla r_1^{*}\left(v\right)& =arg\underset{u}{max}\left\{⟨u,v⟩-h\left(u\right)-\frac{\nu }{2}{\left|\right|u\left|\right|}_F^2\right\}\hfill \\ \hfill & =arg\underset{u}{max}\left\{-\frac{1}{2}{\left|\right|u\left|\right|}_F^2+\frac{1}{\nu }⟨u,v⟩-\frac{1}{\nu }h\left(u\right)\right\}\hfill \\ \hfill & =arg\underset{u}{min}\left\{\frac{1}{2}\left|\right|u-\frac{v}{\nu }{\left|\right|}_F^2+\frac{1}{\nu }h\left(u\right)\right\}\hfill \\ \hfill & ={\mathrm{prox}}_{h/\nu }\left(\frac{v}{\nu }\right).\hfill \end{array}$$

Then, we obtain

$$\nabla r_1^{*}(\nu CU+{\Lambda }_1)=\nu C^T\left({\mathrm{prox}}_{h/\nu }(CU+\frac{{\Lambda }_1}{\nu })\right)$$

and, similarly,

$$\nabla r_2^{*}(\nu UD+{\Lambda }_2)=\nu \left({\mathrm{prox}}_{g/\nu }(UD+\frac{{\Lambda }_2}{\nu })\right)D^T.$$

Next, take the derivative of $F\left(U\right)$ w.r.t U,

$$\begin{array}{cc}\hfill {\nabla }_{U}F\left(U\right)& =\nabla f\left(U\right)-\nabla r_1^{*}(\nu CU+{\Lambda }_1)-\nabla r_2^{*}(\nu UD+{\Lambda }_2)\hfill \\ \hfill & +\nu C^{T}CU+\nu UD{D}^T+C^T{\Lambda }_1+{\Lambda }_2{D}^T\hfill \\ \hfill & =\nabla f\left(U\right)-\nu C^T\left({\mathrm{prox}}_{h/\nu }(CU+{\nu }^{-1}{\Lambda }_1)\right)-\nu \left({\mathrm{prox}}_{g/\nu }(UD+\frac{{\Lambda }_2}{\nu })\right)D^T\hfill \\ \hfill & +\nu C^{T}CU+C^T{\Lambda }_1+\nu UD{D}^T+{\Lambda }_2{D}^T\hfill \\ \hfill & =\nabla f\left(U\right)+C^T\left({\mathrm{prox}}_{\nu h^{*}}(\nu CU+{\Lambda }_1)\right)+\left({\mathrm{prox}}_{\nu g^{*}}(\nu UD+{\Lambda }_2)\right)D^T,\hfill \end{array}$$

and we obtain the last equation by Moreau’s decomposition.