Alternating Direction Method of Multipliers for Generalized Low-Rank Tensor Recovery

Abstract

Low-Rank Tensor Recovery (LRTR), the higher order generalization of Low-Rank Matrix Recovery (LRMR), is especially suitable for analyzing multi-linear data with gross corruptions, outliers and missing values, and it attracts broad attention in the fields of computer vision, machine learning and data mining. This paper considers a generalized model of LRTR and attempts to recover simultaneously the low-rank, the sparse, and the small disturbance components from partial entries of a given data tensor. Specifically, we first describe generalized LRTR as a tensor nuclear norm optimization problem that minimizes a weighted combination of the tensor nuclear norm, the l₁-norm and the Frobenius norm under linear constraints. Then, the technique of Alternating Direction Method of Multipliers (ADMM) is employed to solve the proposed minimization problem. Next, we discuss the weak convergence of the proposed iterative algorithm. Finally, experimental results on synthetic and real-world datasets validate the efficiency and effectiveness of the proposed method.

1. Introduction

In the past decade, the low-rank property of some datasets has been explored skillfully to recover both the low-rank and the sparse components or complete the missing entries. The datasets to be analyzed are usually modeled by matrices and the corresponding recovery technique is named as Low-Rank Matrix Recovery (LRMR). LRMR has received a significant amount of attention in some fields of information science such as computer vision, machine learning, pattern recognition, data mining and linear system identification. There are several appealing types of LRMR including Matrix Completion (MC) [1], Robust Principal Component Analysis (RPCA) [2] and Low-Rank Representation (LRR) [3]. Mathematically, we customarily formulate LRMR as matrix nuclear norm minimization problems that can be effectively solved by several scalable methods. Diverse variants of LRMR derive from the aforementioned three models, for instance, MC with noise (or stable MC) [4], stable RPCA [5,6], incomplete RPCA [7], LRR with missing entries [8], and LRMR based on matrix factorization or tri-factorization [9,10].

The fields of image and signal processing usually require processing large amounts of multi-way data, such as video sequences, functional magnetic resonance imaging sequences and direct-sequence code-division multiple access (DS-CDMA) systems [11]. For the datasets with multi-linear structure, traditional matrix-based data analysis is prone to destroy the spatial and temporal structure, and can be affected by the curse of dimensionality. In contrast, tensor-based data representation can avoid or alleviate the above deficiencies to some extent. Furthermore, tensor decompositions can be used to obtain a low-rank approximation of an investigated data tensor. Two of the most popular tensor decompositions are the Tucker model and the PARAFAC (Parallel Factor Analysis) model, and they can be thought of as the higher order generalizations of Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) [12].

Generally speaking, the traditional tensor decomposition models are very sensitive to missing entries, arbitrary outliers and gross corruptions. On the contrary, Low-Rank Tensor Recovery (LRTR) is very effective to recover simultaneously the low-rank component and the sparse noise, or complete the missing entries. As the higher order generalization of LRMR, LRTR is composed mainly of tensor completion [13] and Multi-linear RPCA (MRPCA) [14]. The task of tensor completion is to recover missing or unsampled entries according to the low-rank structure. The alternating least squares method based on the PARAFAC decomposition was originally proposed to complete missing entries [15]. Subsequently, the Tucker decomposition was applied to the problem of tensor completion [16,17,18]. The multi-linear rank is commonly used to describe the low-rank property, although it is hard to be properly estimated. The tensor nuclear norm generalizes the matrix nuclear norm and becomes a new standard for measuring the low-rank structure of a tensor. Hence, LRTR can usually be boiled down to a tensor nuclear norm minimization problem.

Liu et al. [13] established a tensor nuclear norm minimization model for tensor completion and proposed the Alternating Direction Method of Multipliers (ADMM) for efficiently solving this model. Gandy et al. [19] considered the low-n-rank tensor recovery problem with some linear constraints and developed the Douglas–Rachford splitting technique, a dual variant of the ADMM. To reduce the computational complexity of the tensor nuclear norm minimization problems, Liu et al. [20] introduced the matrix factorization idea into the minimization model and obtained a much smaller matrix nuclear norm minimization problem. Tan et al. [21] transformed tensor completion into a linear least squares problem and proposed a nonlinear Gauss–Seidel method to solve the corresponding optimization problem. In [14], Shi et al. extended RPCA to the case of tensors and presented the MRPCA model. MRPCA, also named robust low-rank tensor recovery [22], was described as a tensor nuclear norm minimization which can be solved efficiently by the ADMM.

This paper studies a generalized model of LRTR. In this model, the investigated data tensor is assumed to be the superposition of a low-rank component, a gross sparse tensor and a small dense error tensor. The Generalized Low-Rank Tensor Recovery (GLRTR) aims mainly to recover the low-rank and the sparse components from partially observed entries. For this purpose, we establish a tensor nuclear norm minimization model for GLRTR. In addition, the ADMM method is adopted to solve the proposed convex model.

The rest of this paper is organized as follows. Section 2 provides mathematical notations and introduces preliminaries on tensor algebra. In Section 3, we review related works on LRTR. We present a generalized LRTR model and develop an efficient iterative algorithm for solving the proposed model in Section 4. Section 5 discusses the weak convergence result on the proposed algorithm. We report the experimental results in Section 6. Finally, Section 7 draws the conclusions.

2. Notations and Preliminaries

This section will briefly introduce mathematical notations and review tensor algebra. We denote tensors by boldface Euclid Math One, e.g., $\mathcal{A}$, matrices by boldface capital letters, e.g., A, vectors by boldface letters, e.g., a, and scalars by italic letters, e.g., a or A. A tensor with order N indicates that its entries can be expressed via N indices. For an Nth-order real tensor $\mathcal{A}\in {\mathsf{ℝ}}^{I_1\times I_2\times \cdots \times I_N}$, its $(i_1,i_2,\cdots ,i_N)$ entry is denoted by $a_{i_1{i}_2\cdots i_N}$.

The n-mode matricization (unfolding) of tensor $\mathcal{A}$, denoted by ${\mathbf{A}}_{(\mathrm{n})}$, is an $I_{\mathrm{n}}\times {\displaystyle {\prod }_{\mathrm{j}\ne \mathrm{n}}{I}_{\mathrm{j}}}$ matrix obtained by rearranging n-mode fibers to be the columns of the resulting matrix. Furthermore, we define an opposite operation “fold” as ${fold}_{\mathrm{n}}({\mathbf{A}}_{(\mathrm{n})})=\mathcal{A}$. Given another tensor $\mathit{ℬ}\in {\mathsf{ℝ}}^{{\mathrm{I}}_1\times {\mathrm{I}}_2\times \cdots \times {\mathrm{I}}_{\mathrm{N}}}$, the inner product between $\mathcal{A}$ and $\mathit{ℬ}$ is defined as $\langle \mathcal{A},\mathit{ℬ}\rangle ={\displaystyle {\sum }_{i_1=1}^{I_1}{\displaystyle {\sum }_{i_2=1}^{I_2}\cdots {\displaystyle {\sum }_{i_N=1}^{I_N}{a}_{i_1{i}_2\cdots i_N}{b}_{i_1{i}_2\cdots i_N}}}}$. Then, the Frobenius norm (or l₂-norm) of tensor $\mathcal{A}$ can be induced by the above inner product, that is, ${\Vert \mathcal{A}\Vert }_{\mathrm{F}}=\sqrt{\langle \mathcal{A},\mathcal{A}\rangle }$. Moreover, we also give other norm definitions of a tensor, which will be needed in the following sections. The l₁-norm of tensor $\mathcal{A}$ is expressed as ${\Vert \mathcal{A}\Vert }_1={\displaystyle {\sum }_{i_1=1}^{I_1}{\displaystyle {\sum }_{i_2=1}^{I_2}\cdots {\displaystyle {\sum }_{i_N=1}^{I_N}\left|a_{i_1{i}_2\cdots i_N}\right|}}}$ and the tensor nuclear norm is ${\Vert \mathcal{A}\Vert }_{\ast }={\displaystyle {\sum }_{n=1}^N{w}_n{\Vert {\mathbf{A}}_{(n)}\Vert }_{\ast }}$, where ${\Vert {\mathbf{A}}_{(n)}\Vert }_{\ast }$, named the nuclear norm of ${\mathbf{A}}_{(n)}$, is the sum of singular values of ${\mathbf{A}}_{(\mathrm{n})},w_n\ge 0$ and ${\displaystyle {\sum }_{\mathrm{n}=1}^{\mathrm{N}}{w}_n}=1$. Due to the fact that the nuclear norm of one matrix is the tightest convex relaxation of the rank function on the unit ball in the spectral norm, it is easy to verify that the tensor nuclear norm ${\Vert \mathcal{A}\Vert }_{\ast }$ is a convex function with respect to $\mathcal{A}$.

The n-mode product of tensor $\mathcal{A}$ by a matrix $\mathbf{U}\in {\mathsf{ℝ}}^{{\mathrm{J}}_{\mathrm{n}}\times {\mathrm{I}}_{\mathrm{n}}}$, denoted as $\mathcal{A}{\times }_{\mathrm{n}}\mathbf{U}$, is an Nth-order tensor with the dimensionality of $I_1\times \cdots \times I_{\mathrm{n}-1}\times J_{\mathrm{n}}\times I_{\mathrm{n}-1}\times \cdots \times I_N$, and its $(i_1,\cdots ,i_{\mathrm{n}-1},\mathrm{j},i_{\mathrm{n}+1}\cdots ,i_N)-th$ entry is calculated as ${\displaystyle {\sum }_{{\mathrm{i}}_{\mathrm{n}}=1}^{{\mathrm{I}}_{\mathrm{n}}}{a}_{i_1{i}_2\cdots i_N}{\mathrm{u}}_{\mathrm{j}{\mathrm{i}}_{\mathrm{n}}}}$. The multi-linear rank of tensor $\mathcal{A}$ is stipulated as a vector $\left(rank({\mathbf{A}}_{(1)}),rank({\mathbf{A}}_{(2)}),\dots ,rank({\mathbf{A}}_{(N)})\right)$, where $rank(\cdot )$ indicates the rank function of a matrix. Tensor $\mathcal{A}$ is low-rank if and only if $rank({\mathbf{A}}_{(n)})\ll I_n$ for some n. Then, the Tucker decomposition of $\mathcal{A}$ is defined as

$$\mathcal{A}\approx \mathcal{C}{\times }_1{\mathbf{U}}^{(1)}{\times }_2{\mathbf{U}}^{(2)}\cdots {\times }_{\mathrm{N}}{\mathbf{U}}^{(\mathrm{N})}\triangleq \mathcal{C}{\times }_{\mathrm{n}=1}^{\mathrm{N}}{\mathbf{U}}^{(\mathrm{n})},$$

(1)

where the core tensor $\mathcal{C}\in {\mathsf{ℝ}}^{{\mathrm{J}}_1\times J_2\times \cdots \times J_N}$, mode matrices ${\mathbf{U}}^{(\mathrm{n})}\in {\mathsf{ℝ}}^{{\mathrm{J}}_{\mathrm{n}}\times {\mathrm{I}}_{\mathrm{n}}}$ and ${\mathrm{J}}_{\mathrm{n}}\le {\mathrm{I}}_{\mathrm{n}}$. The multi-linear rank of a tensor has superiority over other definitions (e.g., the rank used in PARAFAC decomposition) because it is easy to compute. Refer to the survey [12] for further understanding of tensor algebra.

Within the field of low-rank matrix/tensor recovery, two proximal minimization problems are extensively employed:

$$\underset{\mathcal{X}}{\text{min}} \mathsf{\lambda }{\Vert \mathcal{X}\Vert }_1+\frac{1}{2}{\Vert \mathcal{X}-\mathcal{A}\Vert }_{\mathrm{F}}^2,$$

(2)

$$\underset{\mathbf{X}}{\text{min}} \mathsf{\lambda }{\Vert \mathbf{X}\Vert }_{\ast }+\frac{1}{2}{\Vert \mathbf{X}-\mathbf{A}\Vert }_{\mathrm{F}}^2,$$

(3)

where $\mathcal{A}$ and A are a given data tensor and matrix respectively, $\mathsf{\lambda }\ge 0$ is a regularization factor. To address the above two optimization problems, we first define an absolute thresholding operator ${\mathrm{S}}_{\mathsf{\lambda }}(\mathbf{·}):{\mathsf{ℝ}}^{I_1\times I_2\times \cdots \times I_N}\to {\mathsf{ℝ}}^{I_1\times I_2\times \cdots \times I_N}$ as below: ${\left({\mathrm{S}}_{\mathsf{\lambda }}(\mathcal{A})\right)}_{i_1{i}_2\cdots i_N}=\text{max}\left(\left|{\mathrm{a}}_{i_1{i}_2\cdots i_N}\right|-\mathsf{\lambda },0\right)$. It has been proven that problems (2) and (3) have closed-form solutions denoted by ${\mathrm{S}}_{\mathsf{\lambda }}(\mathcal{A})$ [2] and ${\mathrm{T}}_{\mathsf{\lambda }}(\mathbf{A})$ [23] respectively, where ${\mathrm{T}}_{\mathsf{\lambda }}(\mathbf{A})=\mathbf{U}{\mathrm{S}}_{\mathsf{\lambda }}(\mathbf{\Sigma }){\mathbf{V}}^{\mathrm{T}}$ and $\mathbf{U}\mathbf{\Sigma }{\mathbf{V}}^{\mathrm{T}}$ is the singular value decomposition of A.

3. Related Works

Tensor completion and MRPCA are two important and appealing applications of LRTR. In this section, we review the related works on the aforementioned two applications.

As the higher order generalization of matrix completion, tensor completion aims to recover all missing entries with the aid of the low-rank (or approximately low-rank) structure of a data tensor. Although low-rank tensor decompositions are practical in dealing with missing values [15,16,17,18], we have to estimate properly the rank of an incomplete tensor in advance. In the past few years, the matrix nuclear norm minimization model has been extended to the tensor case [13]. Given an incomplete data tensor $\mathcal{D}\in {\mathsf{ℝ}}^{I_1\times I_2\times \cdots \times I_N}$ and a corresponding sampling index set $\Omega \subset [{\mathrm{I}}_1]\times [{\mathrm{I}}_2]\times \cdots \times [{\mathrm{I}}_{\mathrm{N}}]$, we define a linear operator ${\mathrm{P}}_{\Omega }(\mathbf{·}):{\mathsf{ℝ}}^{I_1\times I_2\times \cdots \times I_N}\to {\mathsf{ℝ}}^{I_1\times I_2\times \cdots \times I_N}$ as follows: if $(i_1,i_2,\cdots ,i_N)\in \Omega$, ${\left({\mathrm{P}}_{\Omega }(\mathcal{D})\right)}_{i_1{i}_2\cdots i_N}={\mathrm{d}}_{i_1{i}_2\cdots i_N}$; otherwise, ${\left({\mathrm{P}}_{\Omega }(\mathcal{D})\right)}_{i_1{i}_2\cdots i_N}=0$, where $[{\mathrm{I}}_{\mathrm{n}}]=\{1,2,\dots ,{\mathrm{I}}_{\mathrm{n}}\}$. Then, the original tensor nuclear norm minimization model for tensor completion [13] is described as:

$$\underset{\mathcal{X}}{\text{min}} {\Vert \mathcal{X}\Vert }_{\ast }, \mathrm{s}.\mathrm{t}. {\mathrm{P}}_{\Omega }(\mathcal{X})={\mathrm{P}}_{\Omega }(\mathcal{D}).$$

(4)

To tackle problem (4), Liu et al. [13] developed three different algorithms and demonstrated experimentally that the ADMM is the most efficient algorithm in obtaining a high accuracy solution. If we further take the dense Gaussian noise into consideration, then the stable version of tensor completion is given as below [19]:

$$\underset{\mathcal{X}}{\text{min}} {\Vert \mathcal{X}\Vert }_{\ast }, \mathrm{s}.\mathrm{t}. {\Vert {\mathrm{P}}_{\Omega }(\mathcal{X})-{\mathrm{P}}_{\Omega }(\mathcal{D})\Vert }_{\mathrm{F}}\le \mathsf{\eta },$$

(5)

or its corresponding unconstrained formulation:

$$\underset{\mathcal{X}}{\text{min}} \mathsf{\lambda }{\Vert \mathcal{X}\Vert }_{\ast }+\frac{1}{2}{\Vert {\mathrm{P}}_{\Omega }(\mathcal{X})-{\mathrm{P}}_{\Omega }(\mathcal{D})\Vert }_{\mathrm{F}}^2,$$

(6)

where the regularized parameter $\mathsf{\lambda }(\ge 0)$ is used to balance the low-rankness and the approximate error, and η is a known estimate of the noise level.

In RPCA, a data matrix is decomposed into the sum of a low-rank component and a sparse component, and it is possible to recover simultaneously the two components by principal component pursuit under some suitable assumptions [2]. Shi et al. [14] extended RPCA to the case of tensors and presented the framework of MRPCA, which regards the data tensor $\mathcal{D}$ as the sum of a low-rank tensor $\mathcal{A}$ and a sparse noise term $\mathit{ℰ}$. Mathematically, the low-rank and the sparse components can be simultaneously recovered by solving the following tensor nuclear norm minimization problem:

$$\underset{\mathcal{A},\mathit{ℰ}}{\text{min}} {\Vert \mathcal{A}\Vert }_{\ast }+\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1, \mathrm{s}.\mathrm{t}. \mathcal{D}=\mathcal{A}+\mathit{ℰ}.$$

(7)

In [22], MRPCA is also called as robust low-rank tensor recovery.

4. Generalized Low-Rank Tensor Recovery

Both tensor completion and MRPCA do not consider dense Gaussian noise corruptions. In this section, we investigate the model of Generalized Low-Rank Tensor Recovery (GLRTR) and develop a corresponding iterative scheme.

4.1. Model of GLRTR

The datasets contaminated by Gaussian noise are very universal in practical engineering applications. In view of this, we assume the data tensor $\mathcal{D}$ to be the superposition of the low-rank component $\mathcal{A}$, the large sparse corruption $\mathit{ℰ}$ and the Gaussian noise $\mathcal{G}$. We also consider the case that some entries of $\mathcal{D}$ are missing. To recover simultaneously the above three terms, we establish a convex GLRTR model as follows:

$$\underset{\mathcal{A},ℰ,\mathcal{G}}{\text{min}} {\Vert \mathcal{A}\Vert }_{\ast }+\lambda {\Vert \mathit{ℰ}\Vert }_1+\tau {\Vert \mathcal{G}\Vert }_{\mathrm{F}}^2, \mathrm{s}.\mathrm{t}. {\mathrm{P}}_{\Omega }(\mathcal{D})={\mathrm{P}}_{\Omega }(\mathcal{A}+\mathit{ℰ}+\mathcal{G}),$$

(8)

where the regularization coefficients λ and τ are nonnegative.

If we reinforce the constraints $\mathcal{G}=\mathcal{O}$(or equivalently $\tau \to +\infty$) and $\mathit{ℰ}=\mathcal{O}$ (or equivalently $\mathsf{\lambda }\to +\infty$), then the GLRTR is transformed into the tensor completion model (4), where the zero tensor $\mathcal{O}$ has the same dimensionality as $\mathcal{D}$. If only the constraint $\mathit{ℰ}=\mathcal{O}$ is considered, then the GLRTR is equivalent to Equation (6). Furthermore, if we take $\mathcal{G}=\mathcal{O}$ and $\Omega =[{\mathrm{I}}_1]\times [{\mathrm{I}}_2]\times \cdots \times [{\mathrm{I}}_{\mathrm{N}}]$, then the model of GLRTR becomes the model of MRPCA. In summary, the proposed model is the generalization of the existing LRTR.

For the convenience of using the splitting method, we discard $\mathcal{A}$ and introduce N + 1 auxiliary tensor variables ${\mathit{ℳ}}_1,{\mathit{ℳ}}_2,\dots ,{\mathit{ℳ}}_{\mathrm{N}},\mathcal{X}$, where these auxiliary variables have the same dimensionality as $\mathcal{D}$. Let ${\mathbf{M}}_{\mathrm{n}(\mathrm{n})}$ be the n-mode matricization of ${\mathit{ℳ}}_{\mathrm{n}}$ for each $\mathrm{n}\in [\mathrm{N}]$ and $\mathit{ℳ}=\left\{{\mathit{ℳ}}_1,{\mathit{ℳ}}_2,\dots ,{\mathit{ℳ}}_{\mathrm{N}}\right\}$. Hence, we have the equivalent formulation of Equation (8):

$$\begin{array}{l}\underset{\mathcal{X},ℰ,\mathcal{G},\mathcal{M}}{\text{min}} {\sum }_{\mathrm{N}=1}^{\mathrm{n}} {\mathrm{w}}_{\mathrm{n}}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast }+\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\Vert \mathcal{G}\Vert }_{\mathrm{F}}^2,\\ \mathrm{s}.\mathrm{t}. \mathcal{X}={\mathcal{M}}_{\mathrm{n}}+\mathcal{G}+\mathit{ℰ},{\mathrm{P}}_{\Omega }(\mathcal{X})={\mathrm{P}}_{\Omega }(\mathcal{D}),\mathrm{n}=1,2,\dots ,\mathrm{N}.\end{array}$$

(9)

As a matter of fact, the discarded low-rank tensor $\mathcal{A}$ can be represented by $\mathcal{A}={\sum }_{\mathrm{N}=1}^{\mathrm{n}}{\mathcal{M}}_{\mathrm{n}}/\mathrm{N}$. Now, we explain the low-rankness of $\mathcal{A}$ from two viewpoints. One is ${\Vert \mathcal{A}\Vert }_{\ast }\le {\sum }_{\mathrm{N}=1}^{\mathrm{n}}{\Vert {\mathcal{M}}_{\mathrm{n}}\Vert }_{\ast }/\mathrm{N}\le {\displaystyle \underset{1\le \mathrm{n}\le \mathrm{N}}{\text{max}}}{\Vert {\mathcal{M}}_{\mathrm{n}}\Vert }_{\ast }$. The other is that a better solution to Equation (9) will satisfy ${\mathit{ℳ}}_1\approx {\mathit{ℳ}}_2\approx \dots \approx {\mathit{ℳ}}_{\mathrm{N}}$. These viewpoints illustrate that $\mathcal{A}$ is approximately low-rank along each mode. The aforementioned non-smooth minimization problem is distributed convex. Concretely speaking, the tensor variables can be split into several parts and the objective function is separable across this splitting.

4.2. Optimization Algorithm to GLRTR

As a special splitting method, the ADMM is very efficient to solve a distributed optimization problem with linear equality constraints. It takes the form of the decomposition–coordination procedure and blends the merits of dual decomposition and augmented Lagrangian methods. In this section, we will propose the method of ADMM to solve Equation (9).

We first construct the augmented Lagrangian function of the aforementioned convex optimization problem without considering the constraint ${\mathrm{P}}_{\Omega }(\mathcal{X})={\mathrm{P}}_{\Omega }(\mathcal{D})$:

$$\begin{array}{ll}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},& \mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})= {\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}{\mathrm{w}}_{\mathrm{n}}}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast }+\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\Vert \mathcal{G}\Vert }_{\mathrm{F}}^2+\\ & {\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}\left(\langle {\mathcal{Y}}_{\mathrm{n}},\mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}\rangle +\frac{\mathsf{\mu }}{2}{\Vert \mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}\Vert }_{\mathrm{F}}^2\right)},\end{array}$$

(10)

where μ is a positive numerical constant, ${\mathcal{Y}}_{\mathrm{n}}$ are Lagrangian multiplier tensors and $\mathcal{Y}=\{{\mathcal{Y}}_1,{\mathcal{Y}}_2,\dots ,{\mathcal{Y}}_{\mathrm{N}}\}$. There are totally five blocks of variables in ${\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})$. The ADMM algorithm updates alternatively each block of variables while letting the other blocks of variables be fixed. If $\mathcal{Y}$ is given, one block of variables in $\{\mathcal{X},\mathit{ℳ},\mathcal{G},\mathit{ℰ}\}$ can be updated by minimizing ${\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})$ with respect to one argument. On the contrary, if $\{\mathcal{X},\mathit{ℳ},\mathcal{G},\mathit{ℰ}\}$ are given, $\mathcal{Y}$ is updated by maximizing ${\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})$ with respect to $\mathcal{Y}$. The detailed iterative procedure is outlined as follows:

Computing $\mathcal{X}$. Fix $\left\{\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y}\right\}$ and minimize ${\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})$ with respect to $\mathcal{X}$:

$$\begin{array}{ll}\mathcal{X}:& =\text{arg} \underset{\mathcal{X}}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})\\ & =\text{arg} \underset{\mathcal{X}}{\text{min}} {\displaystyle {\sum }_{\mathrm{n}=1}^{\mathrm{N}}{\Vert \mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\Vert }_{\mathrm{F}}^2}\\ & =\tilde{\mathcal{X}},\end{array}$$

(11)

where $\tilde{\mathcal{X}}={\sum }_{\mathrm{n}=1}^{\mathrm{N}}({\mathit{ℳ}}_{\mathrm{n}}-{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu })/\mathrm{N}+\mathcal{G}+\mathit{ℰ}$. In consideration of the constraint ${\mathrm{P}}_{\Omega }(\mathcal{X})={\mathrm{P}}_{\Omega }(\mathcal{D})$, the final update formulation of $\mathcal{X}$ is revised as

$$\mathcal{X}:={\mathrm{P}}_{\overline{\Omega }}(\tilde{\mathcal{X}})+{\mathrm{P}}_{\Omega }(\mathcal{D}),$$

(12)

where $\overline{\Omega }$ is the complement set of Ω.

Computing $\mathit{ℳ}$. If ${\mathit{ℳ}}_{\mathrm{n}}$ is unknown and other variables are fixed, we update ${\mathit{ℳ}}_{\mathrm{n}}$ by minimizing ${\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})$ with respect to ${\mathit{ℳ}}_{\mathrm{n}}$:

$$\begin{array}{ll}{\mathit{ℳ}}_{\mathrm{n}}:& =\text{arg} \underset{{ℳ}_{\mathrm{n}}}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})\\ & =\text{arg} \underset{{ℳ}_{\mathrm{n}}}{\text{min}} {\mathrm{w}}_{\mathrm{n}}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast }+\frac{\mathsf{\mu }}{2}{\Vert \left(\mathcal{X}-\mathcal{G}-\mathit{ℰ}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)-{\mathit{ℳ}}_{\mathrm{n}}\Vert }_{\mathrm{F}}^2\\ & =\text{arg} \underset{{ℳ}_{\mathrm{n}}}{\text{min}} \frac{{\mathrm{w}}_{\mathrm{n}}}{\mathsf{\mu }}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast }+\frac{1}{2}{\Vert \left({\mathbf{Q}}_{(\mathrm{n})}-{\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\right)-{\mathit{ℳ}}_{\mathrm{n}}\Vert }_{\mathrm{F}}^2\\ & ={\text{fold}}_{\mathrm{n}}\left({\mathrm{T}}_{{\mathrm{w}}_{\mathrm{n}}/\mathsf{\mu }}\right({\mathbf{Q}}_{(\mathrm{n})}\left)\right),\end{array}$$

(13)

where $\mathcal{Q}=\mathcal{X}-\mathcal{G}-\mathit{ℰ}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }$ and ${\mathbf{Q}}_{(\mathrm{n})}$ is the n-mode matricization of $\mathcal{Q}$.

Computing $\mathit{ℰ}$. If $\mathit{ℰ}$ is unknown and other variables are fixed, the calculation procedure of $\mathit{ℰ}$ is given as follows:

$$\begin{array}{ll}\mathit{ℰ}:& =\text{arg} \underset{ℰ}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})\\ & =\text{arg} \underset{ℰ}{\text{min}}\frac{\mathsf{\lambda }}{\mathsf{\mu }\mathrm{N}}{\Vert \mathit{ℰ}\Vert }_1+\frac{1}{2}{\Vert \mathit{ℰ}-{\displaystyle {\sum }_{\mathrm{n}=1}^{\mathrm{N}}\left(\mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)/\mathrm{N}}\Vert }_{\mathrm{F}}^2\\ & ={\mathrm{S}}_{\left(\mathsf{\lambda }/\mathsf{\mu }\mathrm{N}\right)}\left({\displaystyle {\sum }_{\mathrm{n}=1}^{\mathrm{N}}\left(\mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)/\mathrm{N}}\right).\end{array}$$

(14)

Computing $\mathcal{G}$. The update formulation of $\mathcal{G}$ is calculated as

$$\begin{array}{ll}\mathcal{G}:& =\text{arg} \underset{\mathcal{G}}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})\\ & =\text{arg} \underset{\mathcal{G}}{\text{min}}\mathsf{\tau }{\Vert \mathcal{G}\Vert }_{\mathrm{F}}^2+\frac{\mathsf{\mu }\mathrm{N}}{2}{\Vert \mathcal{G}-{\displaystyle {\sum }_{\mathrm{n}=1}^{\mathrm{N}}(\mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathit{ℰ}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu })/\mathrm{N}}\Vert }_{\mathrm{F}}^2\\ & =\frac{\mathsf{\mu }}{2\mathsf{\tau }+\mathsf{\mu }\mathrm{N}}{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}(\mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathit{ℰ}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu })}.\end{array}$$

(15)

Computing $\mathcal{Y}$. If $\{\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G}\}$ are fixed, we can update the Lagrangian multipliers $\mathcal{Y}$ as follows:

$${\mathcal{Y}}_{\mathrm{n}}:={\mathcal{Y}}_{\mathrm{n}}+\mathsf{\mu }\left(\mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}\right), \mathrm{n}=1,2,\dots ,\mathrm{N}.$$

(16)

Once $\left\{\mathcal{X},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y}\right\}$ are updated, we will increase the value of μ by multiplying it with a constant $\mathsf{\rho }(>1)$ during the procedure of iterations. The whole iterative procedure is outlined in Algorithm 1. The stopping condition of Algorithm 1 is set as $\underset{1\le \mathrm{n}\le \mathrm{N}}{\text{max}}{\Vert \mathcal{X}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}\Vert }_{\mathrm{F}}^2<\mathsf{\epsilon }$ or the maximum number of iterations is reached, where ε is a sufficiently small positive number.

Algorithm 1. Solving GLRTR by ADMM

Input: Data tensor $\mathcal{D}$, sampling index set Ω, regularization parameters λ and τ.

Initialize: $\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y},\mathsf{\mu },{\mathsf{\mu }}_{\text{max}}$ and ρ.

Output: $\mathcal{X},\mathit{ℰ},\mathcal{G}$ and $\mathit{ℳ}$.

While not converged do Update $\mathcal{X}$ according to (12). Update ${\mathit{ℳ}}_{\mathrm{n}}$ according to (13), $\mathrm{n}=1,2,\dots ,\mathrm{N}$. Update $\mathit{ℰ}$ according to (14). Update $\mathcal{G}$ according to (15). Update ${\mathcal{Y}}_{\mathrm{n}}$ according to (16), $\mathrm{n}=1,2,\dots ,\mathrm{N}$. Update μ as $\mathsf{\mu }:=\text{min}(\mathsf{\rho }\mathsf{\mu },{\mathsf{\mu }}_{\text{max}})$.

End while

In our implementation, $\mathit{ℳ},\mathit{ℰ},\mathcal{G}$ and $\mathcal{Y}$ are initialized to zeros tensors. The other parameters are set as follows: ${\mathrm{w}}_1={\mathrm{w}}_2=\dots ={\mathrm{w}}_{\mathrm{N}}=1/\mathrm{N},\mathsf{\mu }={10}^{-4},\mathsf{\rho }=1.1, {\mathsf{\mu }}_{\text{max}}={10}^{10}$ and $\mathsf{\epsilon }={10}^{-8}.$ Furthermore, the maximum number of iterations is set to 100.

5. Convergence Analysis

Because the number of block variables in Equation (9) is more than two, it is difficult for us to prove the convergence of Algorithm 1. Nevertheless, the experimental results in the next section demonstrate this algorithm has good convergence behavior. This section will discuss the weak convergence result on our ADMM algorithm.

We consider a special case of Algorithm 1, that is, there is no missing entries. In this case, it holds that $\mathcal{X}=\mathcal{D}$. Thus, Equation (9) is transformed into

$$\begin{array}{l}\underset{ℳ,ℰ,\mathcal{G}}{\text{min}} {\mathrm{w}}_{\mathrm{n}}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast }+\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\Vert \mathcal{G}\Vert }_{\mathrm{F}}^2\\ \mathrm{s}.\mathrm{t}. \mathcal{D}={\mathit{ℳ}}_{\mathrm{n}}+\mathcal{G}+\mathit{ℰ},\mathrm{n}=1,2,\dots ,\mathrm{N}.\end{array}$$

(17)

We can design a corresponding ADMM algorithm by revising Algorithm 1, namely, the update formulation of $\mathcal{X}$ is replaced by $\mathcal{X}=\mathcal{D}$. In fact, the revised algorithm is an inexact version of ADMM. Subsequently, we give the iterative formulations of exact ADMM for Equation (17):

$$\{\begin{array}{l}\mathit{ℳ}:=\text{arg} \underset{ℳ}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{D},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y}),\\ (\mathit{ℰ},\mathcal{G}):=\text{arg} \underset{ℰ,\mathcal{G}}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{D},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y}),\\ {\mathcal{Y}}_{\mathrm{n}}:={\mathcal{Y}}_{\mathrm{n}}+\mathsf{\mu }\left(\mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}\right),\mathrm{n}=1,2,\dots ,\mathrm{N}.\end{array}$$

(18)

In this circumstance, we have the following statement:

Theorem 1. 

Let $\left\{{\mathit{ℳ}}^{(\mathrm{k})},{\mathit{ℰ}}^{(\mathrm{k})},{\mathcal{G}}^{(\mathrm{k})},{\mathcal{Y}}^{(\mathrm{k})}\right\}$ be the sequence produced by (18). If ${\mathrm{L}}_0(\mathcal{D},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})$ has a saddle point with respect to $\left\{\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y}\right\}$, then $\left\{{\mathit{ℳ}}^{(\mathrm{k})},{\mathit{ℰ}}^{(\mathrm{k})},{\mathcal{G}}^{(\mathrm{k})}\right\}$ satisfy that the iterative sequence of objective function in Equation (17) converges to the minimum value.

Proof. 

By matricizing each tensor along the one-mode, the constraints of Equation (17) can be rewritten as a linear equation:

$$\left(\begin{array}{l}{\mathbf{D}}_{(1)}\\ {\mathbf{D}}_{(1)}\\ ⋮\\ {\mathbf{D}}_{(1)}\end{array}\right)=\left(\begin{array}{l}{\mathbf{M}}_{1(1)}\\ {\mathbf{M}}_{2(1)}\\ ⋮\\ {\mathbf{M}}_{\mathrm{N}(1)}\end{array}\right)+\left(\begin{array}{l}{\mathbf{G}}_{(1)}\\ {\mathbf{G}}_{(1)}\\ ⋮\\ {\mathbf{G}}_{(1)}\end{array}\right)+\left(\begin{array}{l}{\mathbf{E}}_{(1)}\\ {\mathbf{E}}_{(1)}\\ ⋮\\ {\mathbf{E}}_{(1)}\end{array}\right),$$

(19)

or equivalently,

$$\left(\begin{array}{l}\mathbf{I}\\ \mathbf{I}\\ ⋮\\ \mathbf{I}\end{array}\right){\mathbf{D}}_{(1)}=\left(\begin{array}{l}{\mathbf{M}}_{1(1)}\\ {\mathbf{M}}_{2(1)}\\ ⋮\\ {\mathbf{M}}_{\mathrm{N}(1)}\end{array}\right)+\left(\begin{array}{l}\mathbf{I}\\ \mathbf{I}\\ ⋮\\ \mathbf{I}\end{array}\right){\mathbf{G}}_{(1)}+\left(\begin{array}{l}\mathbf{I}\\ \mathbf{I}\\ ⋮\\ \mathbf{I}\end{array}\right){\mathbf{E}}_{(1)},$$

(20)

where I is an identity matrix of size I₁ × I₁. If we integrate ${\mathbf{G}}_{(1)}$ and ${\mathbf{E}}_{(1)}$ into one block of variables, then Equation (20) is re-expressed as below:

$$\left(\begin{array}{l}\mathbf{I}\\ \mathbf{I}\\ ⋮\\ \mathbf{I}\end{array}\right){\mathbf{D}}_{(1)}=\left(\begin{array}{l}{\mathbf{M}}_{1(1)}\\ {\mathbf{M}}_{2(1)}\\ ⋮\\ {\mathbf{M}}_{\mathrm{N}(1)}\end{array}\right)+\left(\begin{array}{cc}\begin{array}{l}\mathbf{I}\\ \mathbf{I}\\ ⋮\\ \mathbf{I}\end{array}& \begin{array}{l}\mathbf{I}\\ \mathbf{I}\\ ⋮\\ \mathbf{I}\end{array}\end{array}\right)\left(\begin{array}{l}{\mathbf{G}}_{(1)}\\ {\mathbf{E}}_{(1)}\end{array}\right).$$

(21)

We denote $\mathbf{M}=\left\{{\mathbf{M}}_{1(1)},{\mathbf{M}}_{2(1)},\cdots ,{\mathbf{M}}_{\mathrm{N}(1)}\right\}$ and partition all variables in Equation (17) into two parts in form of matrices: M and $\left({\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)}\right)$. Essentially, $\mathbf{M},{\mathbf{E}}_{(1)}$ and ${\mathbf{G}}_{(1)}$ are the matricizations of $\mathit{ℳ},\mathit{ℰ}$ and $\mathcal{G}$, respectively. Let $\mathrm{g}(\mathbf{M})= {\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}{\mathrm{w}}_{\mathrm{n}}}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast },\mathrm{h}({\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)})=\mathsf{\lambda }{\Vert {\mathbf{E}}_{(1)}\Vert }_1+\mathsf{\tau }{\Vert {\mathbf{G}}_{(1)}\Vert }_{\mathrm{F}}^2$. Then, the objective function in Equation (17) can be expressed as $\mathrm{g}(\mathbf{M})+\mathrm{h}({\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)})$. It is obvious that $\mathrm{g}(\mathbf{M})$ and $\mathrm{h}({\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)})$ are two closed, proper and convex functions. According to the basic convergence result given in [24], we have $\underset{\mathrm{k}\to +\infty }{\text{lim}}\mathrm{g}({\mathbf{M}}^{(\mathrm{k})})\mathrm{+}\mathrm{h}({\mathbf{E}}_{(1)}^{(\mathrm{k})},{\mathbf{G}}_{(1)}^{(\mathrm{k})})={\mathrm{f}}^{\ast }$, where ${\mathrm{f}}^{\ast }$ is the minimum value of $\mathrm{g}(\mathbf{M})+\mathrm{h}({\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)})$ under the linear Constraint (21). This ends the proof. □

In the following, we discuss the detailed iterative procedure of $\left\{\mathbf{M},{\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)}\right\}$ or $\left\{\mathit{ℳ},\mathit{ℰ},\mathcal{G}\right\}$ in Equation (18). It is easy to verify that the update formulation of $\mathit{ℳ}$ in Equation (18) is equivalent to:

$$\begin{array}{l}({\mathbf{M}}_{1(1)},{\mathbf{M}}_{2(1)},\cdots ,{\mathbf{M}}_{\mathrm{N}(1)}):=\text{arg} \underset{\mathbf{M}}{\text{min}}{\mathrm{L}}_{\mathsf{\mu }}(\mathcal{D},\mathit{ℳ},\mathit{ℰ},\mathcal{G},\mathcal{Y})\\ =\text{arg}\underset{({\mathbf{M}}_{1(1)},{\mathbf{M}}_{2(1)},\cdots ,{\mathbf{M}}_{\mathrm{N}(1)})}{\text{min}} {\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}\left({\mathrm{w}}_{\mathrm{n}}{\Vert {\mathbf{M}}_{\mathrm{n}(\mathrm{n})}\Vert }_{\ast }+\frac{\mathsf{\mu }}{2}{\Vert \mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}-\mathcal{G}-\mathit{ℰ}+\frac{1}{\mathsf{\mu }}{\mathcal{Y}}_{\mathrm{n}}\Vert }_{\mathrm{F}}^2\right)}.\end{array}$$

(22)

The block of variables ${\mathbf{M}}_{\mathrm{n}(1)}$ can be solved in parallel because the objective function in Equation (22) is separable with respect to ${\mathbf{M}}_{1(1)},{\mathbf{M}}_{2(1)},\cdots ,{\mathbf{M}}_{\mathrm{N}(1)}$. Hence, the iterative formulation of ${\mathit{ℳ}}_{\mathrm{n}}$ is similar to that of Equation (13).

For fixed M and $\mathcal{Y}$, we can get the optimal block of variables $\left(\mathit{ℰ},\mathcal{G}\right)$ or $({\mathbf{E}}_{(1)},{\mathbf{G}}_{(1)})$ by minimizing $\mathrm{f}\left(\mathit{ℰ},\mathcal{G}\right)$, where $\mathrm{f}\left(\mathit{ℰ},\mathcal{G}\right)=\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\Vert \mathcal{G}\Vert }_{\mathrm{F}}^2+\frac{\mathsf{\mu }}{2}{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}{\Vert \left(\mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)-(\mathcal{G}+\mathit{ℰ})\Vert }_{\mathrm{F}}^2}$. The partial derivative of $\mathrm{f}\left(\mathit{ℰ},\mathcal{G}\right)$ with respect to $\mathcal{G}$ is

$$\frac{\partial \mathrm{f}}{\partial \mathcal{G}}=2\mathsf{\tau }\mathcal{G}+\mathsf{\mu }{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}\left((\mathcal{G}+\mathit{ℰ})-\left(\mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)\right)}.$$

(23)

By letting $\frac{\partial \mathrm{f}}{\partial \mathcal{G}}=0$, we have

$$\mathcal{G}:=\mathsf{\delta }\mathcal{T}-\mathsf{\delta }\mathrm{N}\mathit{ℰ},$$

(24)

where $\mathsf{\delta }=\frac{\mathsf{\mu }}{2\mathsf{\tau }+\mathsf{\mu }\mathrm{N}},\mathcal{T}={\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}\left(\mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)}$. By substituting Equation (24) into $\mathrm{f}\left(\mathit{ℰ},\mathcal{G}\right)$, we have the update formulation of $\mathit{ℰ}$:

$$\begin{array}{l}\mathit{ℰ}:=\text{arg} \underset{ℰ}{\text{min}} \mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\Vert \mathsf{\delta }\mathcal{T}-\mathsf{\delta }\mathrm{N}\mathit{ℰ}\Vert }_{\mathrm{F}}^2+\frac{\mathsf{\mu }}{2}{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}{\Vert \left(\mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }\right)-\mathit{ℰ}-\mathsf{\delta }\mathcal{T}+\mathsf{\delta }\mathrm{N}\mathit{ℰ}\Vert }_{\mathrm{F}}^2}\\ =\text{arg} \underset{ℰ}{\text{min}}\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\mathsf{\delta }}^2{\mathrm{N}}^2{\Vert \mathit{ℰ}-\frac{1}{\mathrm{N}}\mathcal{T}\Vert }_{\mathrm{F}}^2+\frac{\mathsf{\mu }{(1-\mathsf{\delta }\mathrm{N})}^2\mathrm{N}}{2}{\Vert \mathit{ℰ}-\frac{1}{(1-\mathsf{\delta }\mathrm{N})\mathrm{N}}{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{N}}\left(\mathcal{D}-{\mathit{ℳ}}_{\mathrm{n}}+{\mathcal{Y}}_{\mathrm{n}}/\mathsf{\mu }-\mathsf{\delta }\mathcal{T}\right)}\Vert }_{\mathrm{F}}^2\\ =\text{arg} \underset{ℰ}{\text{min}}\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\mathsf{\tau }{\mathsf{\delta }}^2{\mathrm{N}}^2{\Vert \frac{1}{\mathrm{N}}\mathcal{T}-\mathit{ℰ}\Vert }_{\mathrm{F}}^2+\frac{\mathsf{\mu }{(1-\mathsf{\delta }\mathrm{N})}^2\mathrm{N}}{2}{\Vert \mathit{ℰ}-\frac{1}{\mathrm{N}}\mathcal{T}\Vert }_{\mathrm{F}}^2\\ =\text{arg} \underset{ℰ}{\text{min}}\mathsf{\lambda }{\Vert \mathit{ℰ}\Vert }_1+\frac{\mathsf{\tau }{\mathsf{\delta }}^2{\mathrm{N}}^2+\mathsf{\mu }{(1-\mathsf{\delta }\mathrm{N})}^2\mathrm{N}}{2}{\Vert \mathit{ℰ}-\frac{1}{\mathrm{N}}\mathcal{T}\Vert }_{\mathrm{F}}^2\\ ={\mathrm{S}}_{\left(\mathsf{\lambda }/\left(\mathsf{\tau }{\mathsf{\delta }}^2{\mathrm{N}}^2+\mathsf{\mu }{(1-\mathsf{\delta }\mathrm{N})}^2\mathrm{N}\right)\right)}\left(\frac{1}{\mathrm{N}}\mathcal{T}\right).\end{array}$$

(25)

A standard extension of the classic ADMM is to use varying parameter μ for each iteration. The goal of this extension is to improve the convergence and reduce the dependence on the initial choice of μ. In the context of the method of multipliers, this approach is proven to be superlinearly convergent if ${\mathsf{\mu }}^{(\mathrm{k})}\to +\infty$ [25], which inspires us to adopt the non-decreasing sequence $\left\{{\mathsf{\mu }}^{(\mathrm{k})}\right\}$. Furthermore, large values of μ result in a large penalty on violations of primal feasibility and are thus inclined to produce small primal residuals. However, it is difficult to prove the convergence of ADMM [24]. A commonly-used choice for $\left\{{\mathsf{\mu }}^{(\mathrm{k})}\right\}$ is ${\mathsf{\mu }}^{(\mathrm{k}+1)}:=\text{min}(\mathsf{\rho }{\mathsf{\mu }}^{(\mathrm{k})},{\mathsf{\mu }}_{\text{max}})$, where $\mathsf{\rho }>1$ and ${\mathsf{\mu }}_{\text{max}}$ is the upper limit of μ. Due to the fact that ${\mathsf{\mu }}^{(\mathrm{k})}$ is fixed after a finite number of iterations, the corresponding ADMM is convergent according to Theorem 1.

6. Experimental Results

We perform experiments on synthetic data and two real-world video sequences, and validate the feasibility and effectiveness of the proposed method. The experimental results of GLRTR are compared with that of MRPCA, where the missing values in MRCPA are replaced by zeros.

6.1. Synthetic Data

In this subsection, we synthesize data tensors with missing entries. First, we generate an Nth-order low-rank tensor as follows: $\mathcal{A}=\mathcal{C}{\times }_{\mathrm{n}=1}^{\mathrm{N}}{\mathbf{U}}^{(\mathrm{n})},$ with the core tensor $\mathcal{C}\in {\mathsf{ℝ}}^{{\mathrm{J}}_1\times {\mathrm{J}}_2\times \cdots \times {\mathrm{J}}_{\mathrm{N}}}$ and mode matrices ${\mathbf{U}}^{(\mathrm{n})}\in {\mathsf{ℝ}}^{{\mathrm{I}}_{\mathrm{n}}\times {\mathrm{J}}_{\mathrm{n}}}({\mathrm{J}}_{\mathrm{n}}<{\mathrm{I}}_{\mathrm{n}})$. The entries of $\mathcal{C}$ and ${\mathbf{U}}^{(\mathrm{n})}$ are independently drawn from the standard normal distribution. Then, we generate randomly a dense noise tensor $\mathcal{G}\in {\mathsf{ℝ}}^{{\mathrm{I}}_1\times {\mathrm{I}}_2\times \cdots \times {\mathrm{I}}_{\mathrm{N}}}$ whose entries also obey the standard normal distribution. Next, we construct a sparse noise tensor ${\mathrm{P}}_{{\Omega }^{\prime }}(\mathit{ℰ})$, where $\mathit{ℰ}\in {\mathsf{ℝ}}^{{\mathrm{I}}_1\times {\mathrm{I}}_2\times \cdots \times {\mathrm{I}}_{\mathrm{N}}}$ is produced by a uniform distribution on the interval $(-\mathrm{a},\mathrm{a})$ and the index set ${\Omega }^{\prime }$ is produced by uniformly sampling on $[{\mathrm{I}}_1]\times [{\mathrm{I}}_2]\times \cdots \times [{\mathrm{I}}_{\mathrm{N}}]$ with probability p′%. Finally, the generation of the sampling index set Ω is similar to Ω′ and the corresponding sampling rate is set to be p%. Therefore, an incomplete data tensor is synthesized as ${\mathcal{D}}_{\Omega }={\mathrm{P}}_{\Omega }(\mathcal{A}+\mathcal{G}+\mathit{ℰ})$.

For given data tensor ${\mathcal{D}}_{\Omega }$ with missing values, its low-rank component recovered by some method is denoted by $\widehat{\mathcal{A}}$. The Relative Error (RE) is employed to evaluate the recovery performance of the low-rank structure and its definition is given as follows: $\text{RE}={\Vert \widehat{\mathcal{A}}-\mathcal{A}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$. Small relative error means good recovery performance. The experiments are carried out on 50 × 50 × 50 tensors and 20 × 20 × 20 × 20 tensors, respectively. Furthermore, we set a = 500 and ${\mathrm{J}}_1={\mathrm{J}}_2=\cdots ={\mathrm{J}}_{\mathrm{N}}=\mathrm{r}$.

For convenience of comparison, we design three groups of experiments. In the first group of experiments, we only consider the case that there are no missing entries, that is, $\Omega =[{\mathrm{I}}_1]\times [{\mathrm{I}}_2]\times \cdots \times [{\mathrm{I}}_{\mathrm{N}}]$ or p = 100. The values of ${\Vert \mathit{ℰ}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$ and ${\Vert \mathcal{G}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$ are adopted to indicate the Inverse Signal-to-Noise Ratio (ISNR) with respect to the sparse and the Gaussian noise respectively. Three different degrees of sparsity are taken into account, that is, p′ = 5, 10 and 15. In addition, we take $\mathsf{\lambda }=0.025,\mathsf{\tau }=0.02,\mathrm{r}\in \{3,5\}$ for 3rd-order tensors and $\mathsf{\lambda }=0.03,\mathsf{\tau }=0.01,\mathrm{r}\in \{2,4\}$ for 4th-order tensors. For given parameters, we repeat the experiments ten times and report the average results. As a low-rank approximation method for tensors, the Higher-Order SVD (HOSVD) truncation [12] to rank-$({\mathrm{r}}_1,{\mathrm{r}}_2,\dots ,{\mathrm{r}}_{\mathrm{N}})$ is not suitable for gross corruptions owing to the fact that its relative error reaches up to 97% or even 100%. Hence, we do not compare this method with our GLRTR in subsequent experiments. The experimental results are shown in

Table 1. Comparison of experimental results on 50 × 50 × 50 tensors.

(r, p′)	${\Vert \mathit{ℰ}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$ (%)	${\Vert \mathcal{G}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$ (%)	RE of MRPCA (%)	RE of GLRTR (%)
(3, 5)	1253	18.33	9.71 ± 0.98	6.60 ± 0.69
(3, 10)	1769	19.20	8.04 ± 0.84	7.99 ± 0.84
(3, 15)	2218	20.05	9.84 ± 1.97	8.75 ± 1.87
(5, 5)	569	8.79	7.39 ± 0.79	3.74 ± 0.42
(5, 10)	855	9.40	8.55 ± 0.97	4.47 ± 0.52
(5, 15)	1040	9.16	9.13 ± 0.52	5.05 ± 0.31

and

Table 2. Comparison of experimental results on 20 × 20 × 20 × 20 tensors.

(r, p′)	${\Vert \mathit{ℰ}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$ (%)	${\Vert \mathcal{G}\Vert }_{\mathrm{F}}/{\Vert \mathcal{A}\Vert }_{\mathrm{F}}$ (%)	RE of MRPCA (%)	RE of GLRTR (%)
(2, 5)	1722	24.05	29.24 ± 7.95	13.90 ± 4.68
(2, 10)	2823	37.72	48.07 ± 18.47	23.98 ± 9.66
(2, 15)	3052	27.17	44.71 ± 13.66	23.10 ± 8.52
(4, 5)	448	6.97	12.00 ± 1.60	8.96 ± 1.15
(4, 10)	563	6.04	15.12 ± 1.98	6.26 ± 0.87
(4, 15)	758	6.58	24.22 ± 2.61	12.16 ± 2.90

, respectively.

From the above two tables, we have the following observations: (I) Although the values of ISNR on sparse noise are very large, both MRPCA and GLRTR remove efficiently sparse noise to some extent. Meanwhile, large values of ISNR on Gaussian noise are disadvantageous for recovering the low-rank components; (II) GLRTR has better recovery performance than MRPCA. In an average sense, the relative error of GLRTR is 2.68% smaller than that of MRPCA for 50 × 50 × 50 tensors, and 14.17% for 20 × 20 × 20 × 20 tensors; (III) For 3rd-order tensors, GLRTR removes effectively Gaussian noise, and, on average, its relative error is 5.96% smaller than the value of ISNR on Gaussian noise; for 4th-order tensors, GLRTR effectively removes Gaussian noise only in the case r = 2. In summary, GLRTR is more effective than MRPCA in recovering the low-rank components.

The second group of experiments considers four different sampling rates for Ω and one fixed degree of sparsity for Ω′, that is, $\mathrm{p}\in \{30,50,70,90\}$ and p′ = 5. We set τ = 0.02 for both 3rd-order and 4th-order tensors, and choose the superior tradeoff parameter λ for each p. The comparisons of experimental results between MRPCA and GLRTR are shown in

Table 3. Comparison of experimental results on incomplete 50 × 50 × 50 tensors for r = 5.

(p, λ)	(30,0.07)	(50, 0.05)	(70, 0.03)	(90,0.03)
RE of MRPCA (%)	87.81 ± 0.27	77.02 ± 0.46	60.64 ± 1.77	11.49 ± 1.73
RE of GLRTR (%)	11.58 ± 1.72	6.62 ± 0.82	5.51 ± 0.83	4.27 ± 0.47

and

Table 4. Comparison of experimental results on incomplete 20 × 20 × 20 × 20 tensors for r = 4.

(p, λ)	(30, 0.03)	(50, 0.035)	(70, 0.03)	(90, 0.03)
RE of MRPCA (%)	88.38 ± 0.25	78.94 ± 0.34	65.06 ± 0.59	40.03 ± 1.34
RE of GLRTR (%)	62.51 ± 1.72	20.09 ± 2.63	9.14 ± 0.12	8.58 ± 1.11

, respectively. We can see from these two tables that MRPCA is very sensitive to the sampling rate p%, and it hardly recovers the low-rank components. In contrast, GLRTR achieves better recovery performance for 3rd-order tensors. As for 4th-order tensors, it also has smaller relative error when the sampling rate p% is relatively large. These observations show that GLRTR is more robust to missing values than MRPCA.

We will evaluate the sensitivity of GLRTR to the choice of λ and τ in the last group of experiments. For convenience of designing experiments, we only perform experiments on 50 × 50 × 50 tensors and consider the case that p = 100. The values of λ and τ are set according to the following manner: we vary the value of one parameter while letting the other be fixed. In the first case, the parameter τ is chosen as 0.01. Under this circumstance, the relative errors versus different λ of MRPCA and GLRTR are shown in Figure 1. We take λ = 0.01 in the second case and the relative errors versus different τ of GLRTR are shown in Figure 2.

It can be seen from Figure 1 that the relative errors of MRPCA and GLRTR are about 9.90% and 4.62%, respectively, if 0.02 ≤ λ ≤ 0.07, which means the latter has better recovery performance than the former. Furthermore, their relative errors are relatively stable when λ lies within a certain interval. Figure 2 illustrates that the relative error has the tendency to increase monotonically, and it becomes almost stationary when τ ≥ 1. At this moment, the relative errors lie in the interval (0.037, 0.080). This group of experiments implies that, for our synthetic data, GLRTR is not very sensitive to the choice of λ and τ.

6.2. Influence of Noise and Sampling Rate on the Relative Error

This subsection will evaluate the influence of noise and sampling rate on the relative error. For this purpose, we design four groups of experiments and use the synthetic data generated in the same manner as in the previous subsection. For the sake of convenience, we only carry out experiments on 50 × 50 × 50 tensors.

The first group of experiments aims to investigate the influence of noise on the recovery performance. In the data generation process, we only change the manner for generating $\mathcal{G}$, that is, each entry of $\mathcal{G}$ is drawn independently from the normal distribution with mean zero and standard deviation b. Let $\mathrm{r}=5,{\mathrm{p}}^{\prime }=10,\mathrm{p}=100,\mathrm{a}=50\mathrm{i}$ and b = 0.2j, where $\mathrm{i},\mathrm{j}\in \left\{0,1,\dots ,10\right\}$. For different combinations of a and b, the relative errors of GLRTR are shown in Figure 3. We can draw two conclusions from this figure. For given b, the relative error is relatively stable with the increasing of a, which means the relative error is not very sensitive to the magnitude of sparse noise. The relative error monotonically increases with the increasing of Gaussian noise level, which validates that large Gaussian noise is disadvantageous for recovering the low-rank component.

Next, we study the influence of sampling rate p% on the relative error for different r. Set ${\mathrm{p}}^{\prime }=10,\mathrm{a}=500,\mathrm{b}=1,\mathrm{r}=1+2\mathrm{i},\mathrm{i}=1,2,\dots ,14$ and vary the value of p% from 30 to 100 in steps of size 10. For fixed r and p, we obtain the low-rank component according to GLRTR and then plot the relative errors in Figure 4. From the 3-D colored surface in Figure 4, we can see that both r and p have significant influence on the relative error. This observation indicates that small r or large p is conducive to the recovery of low-rank term.

The third group of experiments will validate the robustness of GLRTR to sparse noise. Concretely speaking, we investigate the recovery performance under different ISNR on sparse noise without consideration of Gaussian noise. Set $\mathrm{r}\in \{3,5\},{\mathrm{p}}^{\prime }=10,\mathrm{p}=100,\mathrm{b}=0, \mathrm{a}=2^{\mathrm{i}}\times 500$, $\mathrm{i}=-15,-14,\dots ,10$. The experimental results are shown in Figure 5, where the horizontal and the vertical coordinates represent the ISNR on sparse noise and the relative error, respectively. This figure illustrates that the relative error is less than 4.5% for synthetic 3rd-order tensors, which verifies experimentally that our method is very robust to sparse noise.

In the last group of experiments, we discuss the performance of GLRTR in removing the small dense noise for given large sparse noise. We also propose a combination strategy: GLRTR + HOSVD, that is, GLRTR is followed by HOSVD. The goal of this new method is to improve the denoising performance of GLRTR. Let $\mathrm{r}\in \{3,5\},{\mathrm{p}}^{\prime }=10,\mathrm{p}=100,\mathrm{a}=500$ and $\mathrm{b}=0.1\mathrm{i},\mathrm{i}=0,1,\dots ,50$. Different values for b lead to different ISNR on Gaussian noise. We draw four curves to reflect the relationship between the relative error and ISNR on Gaussian noise, as shown in Figure 6, where the black dashed line is a reference line. We have two observations from this figure. When ISNR on Gaussian noise is larger than 3.5%, GLRTR not only successfully separates the sparse noise to some extent but also effectively removes the Gaussian noise. The GLRTR+HOSVD method has better denoising performance than GLRTR in the presence of large Gaussian noise.

6.3. Applications in Background Modeling

In this subsection, we test our method on two real-world surveillance videos for object detection and background subtraction: Lobby and Bootstrap datasets [26]. For convenience of computation, we only consider the first 200 frames for each dataset and transform the color images into the gray-level images. The resolutions of each image in the Lobby and Bootstrap datasets are 128 × 160 and 120 × 160, respectively. We add Gaussian noise with mean zero and standard deviation 5 to each image. Hence, we obtain two data tensors of order 3 and their sizes are 128 × 160 × 200 and 120 × 160 × 200, respectively. For two given tensors, we execute random sampling on them with a probability of 50%.

Considering the fact that MRPCA fails in recovering the low-rank components on the synthetic data with missing values, we only implement the method of GLRTR on the video datasets. Two tradeoff parameters are set as follows: λ = 0.0072 and τ = 0.001. We can obtain the low-rank, the sparse and the completed components from the incomplete data tensors according to the proposed method. Actually, the low-rank terms are the backgrounds and the sparse noise terms correspond to the foregrounds. The experimental results are partially shown in Figure 7 and Figure 8, respectively, where the missing entries in incomplete images are shown in white. From Figure 7 and Figure 8, we can see that GLRTR can recover efficiently the low-rank images and the sparse noise images. Moreover, we observe from the recovered images that a large proportion of missing entries are effectively completed.

To evaluate the completion performance of GLRTR, we define the Relative Approximate Error (RAE) as $\text{RAE}={\Vert \widehat{\mathcal{D}}-\mathcal{D}\Vert }_{\mathrm{F}}/{\Vert \mathcal{D}\Vert }_{\mathrm{F}}$, where $\mathcal{D}$ is the original video tensor without Gaussian noise corruptions and missing entries, and $\widehat{\mathcal{D}}$ is the approximated term of $\mathcal{D}$. The RAE of the Lobby dataset is 8.94% and that of the Bootstrap dataset is 20.11%. These results demonstrate GLRTR can complete approximately the missing entries to a certain degree. There are two reasons for that the Bootstrap dataset has relatively large RAE: one is its more complex foreground and the other is that the entries of the foreground can not be recovered when they are missing. In summary, GLRTR is robust to gross corruption, Gaussian noise and missing values.

7. Conclusions

In this paper, we investigate a generalized model of LRTR in which large sparse corruption, missing entries and Gaussian noise are taken into account. For the generalized LRTR, we establish an optimization problem that minimizes the weighted combination of the tensor nuclear norm, the l₁ norm and the Frobenius norm. To address this minimization problem, we present an iterative scheme based on the technique of ADMM. The experimental results on synthetic data and real-world video datasets illustrate that the proposed method is efficient and feasible in recovering the low-rank components and completing missing entries. In the future, we will consider the theoretical conditions for exact recoverability and other scalable algorithms.