Correntropy Based Matrix Completion

Abstract

This paper studies the matrix completion problems when the entries are contaminated by non-Gaussian noise or outliers. The proposed approach employs a nonconvex loss function induced by the maximum correntropy criterion. With the help of this loss function, we develop a rank constrained, as well as a nuclear norm regularized model, which is resistant to non-Gaussian noise and outliers. However, its non-convexity also leads to certain difficulties. To tackle this problem, we use the simple iterative soft and hard thresholding strategies. We show that when extending to the general affine rank minimization problems, under proper conditions, certain recoverability results can be obtained for the proposed algorithms. Numerical experiments indicate the improved performance of our proposed approach.

1. Introduction

Arising from a variety of applications such as online recommendation systems [1,2], image inpainting [3,4] and video denoising [5], the matrix completion problem has drawn tremendous and continuous attention over recent years [6,7,8,9,10,11,12]. The matrix completion aims at recovering a low rank matrix from partial observations of its entries [7]. The problem can be mathematically formulated as:

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }\mathrm{rank}(X)\mathrm{s}.\mathrm{t}.X_{ij}=B_{ij},(i,j)\in \Omega ,$$

(1)

where $X,B\in {\mathbb{R}}^{m\times n}$ and $\Omega$ is an index set. Due to the nonconvexity of the rank function $\mathrm{rank}(·)$, solving this minimization problem is NP-hard in general. To obtain a tractable convex relaxation, the nuclear norm heuristic was proposed [7]. Incorporated with the least squares loss, the nuclear norm regularization was proposed to solve (1) when the observed entries are contaminated by Gaussian noise [13,14,15,16]. In real-world applications, datasets might be contaminated by non-Gaussian noise or sparse gross errors, which can appear in both explanatory and response variables. However, it has been well understood that the least squares loss cannot be resistant to non-Gaussian noise or outliers.

To address this problem, some efforts have been made in the literature. Ref. [17] proposed a robust approach by using the least absolute deviation loss. Huber’s criterion was adopted in [18] to introduce robustness into matrix completion. Ref. [19] proposed to use an $L_p$ ($0<p\le 1$) loss to enhance the robustness. However, as explained later, the approaches mentioned above cannot be robust to impulsive errors. In this study, we propose to use the correntropy-induced loss function in matrix completion problems when pursuing robustness.

Correntropy, which serves as a similarity measurement between two random variables, was proposed in [20] within the information-theoretic learning framework developed in [21]. It is shown that in prediction problems, error correntropy is closely related to the error entropy [21]. The correntropy and the induced error criterion have been drawing a great deal of attention in the signal processing and machine learning community. Given two scalar random variables U, V, the correntropy ${\mathcal{V}}_{\sigma }$ between U and V is defined as ${\mathcal{V}}_{\sigma }(U,V)=\mathbb{E}{\mathcal{K}}_{\sigma }(U,V)$ with ${\mathcal{K}}_{\sigma }$ a Gaussian kernel given by ${\mathcal{K}}_{\sigma }(u,v)=\exp \left\{-{(u-v)}^2/{\sigma }^2\right\}$, the scale parameter $\sigma >0$ and $(u,v)$ a realization of $(U,V)$. It is noticed in [20] that the correntropy ${\mathcal{V}}_{\sigma }(U,V)$ can induce a new metric between U and V.

In this study, by employing the correntropy-induced losses, we propose a nonconvex relaxation approach to robust matrix completion. Specifically, we develop two models: one with a rank constraint and the other with a nuclear norm regularization term. To solve them, we propose to use simple, but efficient algorithms. Experiments on synthetic, as well as real data are implemented and show that our methods are effective even for heavily-contaminated datasets. We make the following contributions in this paper:

• In Section 3, we propose a nonconvex relaxation strategy for the robust matrix completion problem, where the robustness benefits from using a robust loss. Based on this loss, a rank constraint, as well as a nuclear norm penalized model is proposed. We also extend the proposed models to deal with the affine rank minimization problem, which includes the matrix completion as a special case.
• In Section 4, we propose to use simple, but effective algorithms to solve the proposed models, which are based on gradient descent and employ the hard/soft shrinkage operators. By verifying the Lipschitz continuity, the convergence of the algorithms can be proven. When extended to affine rank minimization problems, under proper conditions, certain recoverability results are obtained. These results give understandings of this loss function in an algorithmic sense, which is in accordance with and extends our previous work [22].

This paper is organized as follows: In Section 2, we review some existing (robust) matrix completion approaches. In Section 3, we propose our nonconvex relaxation approach. Two algorithms are proposed in Section 4 to solve the proposed models. Theoretical results will be presented in Section 4.1. Experimental results are reported in Section 5. We end this paper in Section 6 with concluding remarks.

2. Related Work and Discussions

In matrix completion, solving the optimization problem in Model (1) is NP-hard, and a usual remedy is to consider the following nuclear norm convex relaxation:

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }{\parallel X\parallel }_{\ast }\mathrm{s}.\mathrm{t}.X_{i,j}=B_{i,j},(i,j)\in \Omega .$$

(2)

Theoretically, it has been demonstrated in [7,8] that under proper assumptions, with an overwhelming probability, one can reconstruct the original matrix. Situations of the matrix completion with noisy entries have been also considered; see, e.g., [6,9]. In the noisy setting, the corresponding observed matrix turns out to be:

$$\begin{array}{c}\hfill B_{\Omega }=X_{\Omega }+E,\end{array}$$

(3)

where $B_{\Omega }$ denotes the projection of B onto $\Omega$, and E refers to the noise. The following two models are frequently adopted to deal with the noisy case:

$$\begin{array}{c}\underset{X\in {\mathbb{R}}^{m\times n}}{\min }\frac{1}{2}{\parallel X_{\Omega }-B_{\Omega }\parallel }_F^2\mathrm{s}.\mathrm{t}.\mathrm{rank}(X)\le R,\hfill \end{array}$$

and its convex relaxed and regularized heuristic:

$$\begin{array}{c}\hfill \underset{X\in {\mathbb{R}}^{m\times n}}{\min }\frac{1}{2}\parallel X_{\Omega }-B_{\Omega }{\parallel }_F^2+\lambda {\parallel X\parallel }_{\ast },\end{array}$$

where $\lambda >0$ is a regularization parameter. Similar theoretical reconstruction results have been also derived in the noiseless case under technical assumptions. Along this line, various approaches have been proposed [14,15,16,23,24]. Among others, Refs. [10,25] interpreted the matrix completion problem as a specific case of the trace regression problem endowed with an entry-wise least squares loss, ${\parallel ·\parallel }_F^2$. In the above-mentioned settings, the noise term E is usually assumed to be Gaussian or sub-Gaussian to ensure the good generalization ability, which certainly excludes the heavily-tailed noise and/or outliers.

Existing Robust Matrix Completion Approaches

It has been well understood that the least squares estimator cannot deal with non-Gaussian noise or outliers. To alleviate this limitation, some efforts have been made.

In a seminal work, Ref. [17] proposed a robust matrix completion approach, in which the model takes the following form:

$$\underset{X,E\in {\mathbb{R}}^{m\times n}}{\min }{\parallel E\parallel }_1+\lambda {\parallel X\parallel }_{\ast }\mathrm{s}.\mathrm{t}.X_{\Omega }+E=B_{\Omega }.$$

(4)

The above model can be further formulated as:

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }\parallel X_{\Omega }-B_{\Omega }{\parallel }_1+\lambda {\parallel X\parallel }_{\ast },$$

where $\lambda >0$ is a regularization parameter. The robustness of the model (4) results from using the least absolute deviation loss (LAD). This model was later applied to the column-wise robust matrix completion problem in [26].

By further decomposing E into $E=E_1+E_2$, where $E_1$ refers to the noise and $E_2$ stands for the outliers, Ref. [18] proposed the following robust reconstruction model:

$$\underset{X,E_2\in {\mathbb{R}}^{m\times n}}{\min }\parallel X_{\Omega }-B_{\Omega }-E_2{\parallel }_F^2+{\lambda \parallel X\parallel }_{\ast }+\gamma {\parallel E_2\parallel }_1,$$

where $\lambda ,\gamma >0$ are regularization parameters. They further showed that the above estimator is equivalent to the one obtained by using Huber’s criterion when evaluating the data-fitting risk. We also note that [19] adopted an $L_p$ ($0<p\le 1$) loss to enhance the robustness.

3. The Proposed Approach

3.1. Our Proposed Nonconvex Relaxation Approach

As stated previously, matrix completion models based on the least squares loss cannot perform well with non-Gaussian noise and/or outliers. Accordingly, robustness can be pursued by using a robust loss as mentioned earlier. Associated with a nuclear norm penalization term, they are essentially regularized M-estimator. However, note that the LAD loss and the $L_p$ loss penalize the small residuals strongly and hence cannot lead to accurate prediction for unobserved entries from the trace regression viewpoint. Moreover, robust statistics reminds us that models based on the above three mentioned loss functions cannot be robust to impulsive errors [27,28]. These limitations encourage us to employ more robust surrogate loss functions to address this problem. In this paper, we present a nonconvex relaxation approach to deal with the matrix completion problem with entries heavily contaminated by noise and/or outliers.

In our study, we propose the robust matrix completion model based on a robust and nonconvex loss, which is defined by:

$${\rho }_{\sigma }(t)={\sigma }^2(1-\exp (-t^2/{\sigma }^2)),$$

with $\sigma >0$ a scale parameter. To give an intuitive impression, plots of loss functions mentioned above are given in Figure 1. As mentioned above, this loss function is induced by the correntropy, which measures the similarity between two random variables [20,21] and has found many successful applications [29,30,31]. Recently, it was shown in [22] that regression with the correntropy-induced losses regresses towards the conditional mean function with a diverging scale parameter $\sigma$ when the sample size goes to infinity. It was also shown in [32] that when the noise variable admits a unique global mode, regression with the correntropy-induced losses regresses towards the conditional mode. As argued in [22,32], learning with correntropy-induced losses can be resistant to non-Gaussian noise and outliers, while ensuring good prediction accuracy simultaneously with properly chosen $\sigma$.

Figure 1. Different losses: least squares, absolute deviation loss (LAD), Huber’s loss and ${\rho }_{\sigma }$ (Welsch loss).

Associated with the ${\rho }_{\sigma }$ loss, our rank-constraint robust matrix completion problem is formulated as:

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }{\ell }_{\sigma }(X)\mathrm{s}.\mathrm{t}.\mathrm{rank}(X)\le R,$$

(5)

where the data-fitting risk ${\ell }_{\sigma }(X)$ is given by:

$$\begin{array}{c}\hfill {\ell }_{\sigma }(X)=\frac{1}{2}\sum _{(i,j)\in \Omega }{\rho }_{\sigma }\left(X_{ij}-B_{ij}\right)=\frac{{\sigma }^2}{2}\sum _{(i,j)\in \Omega }\left(1-\exp \left(-{(X_{ij}-B_{ij})}^2/{\sigma }^2\right)\right).\end{array}$$

The nuclear norm heuristic model takes the following form:

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }{\ell }_{\sigma }(X)+\lambda {\parallel X\parallel }_{\ast },$$

(6)

where $\lambda >0$ is a regularization parameter.

3.2. Affine Rank Minimization Problem

In this part, we will show that our robust matrix completion approach can be extended to deal with the robust affine rank minimization problems.

It is known that the matrix completion problem (1) is a special case of the following affine rank minimization problem:

$$\begin{array}{c}\hfill \underset{X\in {\mathbb{R}}^{m\times n}}{\min }\mathrm{rank}(X)\mathrm{s}.\mathrm{t}.\mathcal{A}(X)=b,\end{array}$$

(7)

where $b\in {\mathbb{R}}^p$ is given, and $\mathcal{A}:{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^p$ is a linear operator defined by:

$$\begin{array}{c}\hfill \mathcal{A}(·):={\left[⟨A^1,·⟩,⟨A^2,·⟩,\dots ,⟨A^p,·⟩\right]}^T,\end{array}$$

where $A^i\in {\mathbb{R}}^{m\times n}$ for each i. Introduced and studied in [33], this problem has drawn much attention in recent years [14,15,16,23]. Note that (7) can be reduced to the matrix completion problem (1) if we set $p=|\Omega |$ (the cardinality of $\Omega$), and let $A^{(i-1)n+j}=e_i(m)e_j{(n)}^T$ for each $(i,j)\in \Omega$, where $e_i(m),i=1,\dots ,m$ and $e_j(n),j=1,\dots ,n$ are the canonical basis vector of ${\mathbb{R}}^m$ and ${\mathbb{R}}^n$, respectively.

In fact, (5) and (6) can be naturally extended to handle cases with noise and outliers of (7). Denote the risk as follows:

$$\begin{array}{c}\hfill {\tilde{\ell }}_{\sigma }(X)=\frac{{\sigma }^2}{2}\sum _{i=1}^p\left(1-\exp \left(-{\left(⟨A^i,X⟩-b_i\right)}^2/{\sigma }^2\right)\right).\end{array}$$

The rank constrained model can be formulated as:

$$\underset{X\in {\mathbb{R}}^{m\times n}}{\min }{\tilde{\ell }}_{\sigma }(X)\mathrm{s}.\mathrm{t}.\mathrm{rank}(X)\le R,$$

(8)

and the nuclear norm regularized heuristic takes the form:

$$\begin{array}{cc}\hfill & \underset{X\in {\mathbb{R}}^{m\times n}}{\min }{\tilde{\ell }}_{\sigma }(X)+\lambda {\parallel X\parallel }_{\ast }.\hfill \end{array}$$

(9)

Referring to computational considerations presented below, we will focus on the more general optimization problems (8) and (9), which can be directly applied to (5) and (6).

4. Algorithms and Analysis

We consider using gradient descent-based algorithms to solve the proposed models. It is usually admitted that gradient descent is not very efficient. However, in our experiments, we find that gradient descent is still efficient, and comparable with some state-of-the-art methods. On the other hand, we present recoverability and convergence rate results for gradient descent applied to the proposed models. Such results and analysis may help us better understand the models and such a nonconvex loss function from the algorithmic aspects.

We first consider gradient descent with hard thresholding for solving (8). The derivation is standard. Denote $S_R:=\{X\in {\mathbb{R}}^{m\times n}|\mathrm{rank}(X)\le R\}$. By the differentiability of ${\ell }_{\sigma }$, when Y is sufficiently close to X, ${\ell }_{\sigma }$ can be approximated by:

$${\ell }_{\sigma }(X)\approx {\ell }_{\sigma }(Y)+⟨\nabla {\ell }_{\sigma }(Y),X-Y⟩+\frac{\alpha }{2}{\parallel X-Y\parallel }_F^2.$$

Here, $\alpha >0$ is a parameter, and $\nabla {\ell }_{\sigma }(Y)$, the gradient of ${\ell }_{\sigma }$ at Y, is equal to:

$$\sum _{i=1}^p\exp \left(-{(⟨A^i,Y⟩-b_i)}^2/{\sigma }^2\right)(⟨A^i,Y⟩-b_i)A^i.$$

(10)

Now, the iterates can be generated as follows:

$$\begin{array}{c}{X}^{(k+1)}=\arg \underset{X\in S_R}{\min }{\ell }_{\sigma }(X^{(k)})+⟨\nabla {\ell }_{\sigma }(X^{(k)}),X-X^{(k)}⟩\hfill \\ +\frac{\alpha }{2}{\parallel X-X^{(k)}\parallel }_F^2\hfill \\ =\arg \underset{X\in S_R}{\min }{\parallel X-Y^{(k+1)}\parallel }_F^2\hfill \end{array}$$

(11)

with:

$$\begin{array}{c}\hfill Y^{(k+1)}=X^{(k)}-{\alpha }^{-1}\nabla {\ell }_{\sigma }(X^{(k)}).\end{array}$$

(12)

We simply write (11) as $X^{(k+1)}={\mathcal{P}}_{S_R}(Y^{(k+1)})$, where ${\mathcal{P}}_{S_R}$ denotes the hard thresholding operator, i.e., the best rank-R approximation to $Y^{(k+1)}$. The algorithm is presented in Algorithm 1.

Algorithm 1 Gradient descent iterative hard thresholding for (8).

Input: linear operator $\mathcal{A}:{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^p$, initial guess $X^{(0)}\in {\mathbb{R}}^{m\times n}$, prescribed rank $R\ge 1$, $\sigma >0$   Output: the recovered matrix $X^{(k+1)}$    while a certain stopping criterion is not satisfied do  1: Choose a fixed step-size ${\alpha }^{-1}>0$.  2: Compute the gradient descent step (12) $$Y^{(k+1)}=X^{(k)}-{\alpha }^{-1}\nabla {\ell }_{\sigma }(X^{(k)}).$$    3: Perform the hard thresholding operator to obtain $$X^{(k+1)}={\mathcal{P}}_{S_R}\left(Y^{(k+1)}\right),$$  and set $k:=k+1$. end while

The algorithm starts from an initial guess $X^{(0)}$ and continues until some stopping criterion is satisfied, e.g., $\parallel X^{(k+1)}-X^{(k)}{\parallel }_F\le ϵ$, where $ϵ$ is a certain given positive number. Indeed, such a stopping criterion makes sense, as Proposition A3 shows that $\parallel X^{(k)}-X^{(k+1)}{\parallel }_F\to 0$. To ensure the convergence, the step-size should satisfy $\alpha >L:={\parallel \mathcal{A}\parallel }_2^2$, where ${\parallel \mathcal{A}\parallel }_2$ denotes the spectral norm of $\mathcal{A}$. For matrix completion, the spectral norm is smaller than one, and thus, we can set $\alpha >1$. In Appendix A, we have shown the Lipschitz continuity of $\nabla {\ell }_{\sigma }(·)$, which is necessary for the convergence of the algorithm. $\alpha$ can also be self-adaptive by using a certain line-search rule. Algorithm 2 is the line-search version of Algorithm 1.

Algorithm 2 Line-search version of Algorithm 1.

Input: linear operator $\mathcal{A}:{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^p$, initial guess $X^{(0)}\in {\mathbb{R}}^{m\times n}$, prescribed rank $R\ge 1$, $\sigma >0$, ${\alpha }^{(0)}>0$, $\delta \in (0,1)$, $\eta >1$ Output: the recovered matrix $X^{(k+1)}$ while a certain stopping criterion is not satisfied do  1: ${\alpha }^{(k+1)}={\alpha }^{(k)}$  repeat   2: $X^{(k+1)}={\mathcal{P}}_{S_R}\left(X^{(k)}-\frac{1}{{\alpha }^{(k+1)}}\nabla {\ell }_{\sigma }(X^{(k)})\right)$   3: ${\alpha }^{(k+1)}:={\alpha }^{(k+1)}\eta$  until ${\ell }_{\sigma }(X^{(k+1)})\le {\ell }_{\sigma }(X^{(k)})-\frac{\delta {\alpha }^{(k+1)}}{2}{\parallel X^{(k+1)}-X^{(k)}\parallel }_F^2$  4: ${\alpha }^{(k+1)}:={\alpha }^{(k+1)}/\eta$,  and set $k:=k+1$. end while

Solving (9) is similar, with only the hard thresholding ${\mathcal{P}}_R$ replaced by the soft thresholding ${\mathcal{S}}_{\tau }$, which can be derived as follows. Denote $Y^{(k+1)}=U\mathrm{diag}\left({\{{\sigma }_i\}}_{1\le i\le r}\right)V^T$ as the SVD of $Y^{(k+1)}$. Then, $S_{\lambda /\alpha }$ is the matrix soft thresholding operator [13,16] defined as $S_{\lambda /\alpha }(Y^{(k+1)})=U\mathrm{diag}\left(\max \{{\sigma }_i-\lambda /\alpha ,0\}\right)V^T.$ Gradient descent-based soft thresholding is summarized in Algorithm 3.

Algorithm 3 Gradient descent iterative soft thresholding for (9).

Input: linear operator $\mathcal{A}:{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^p$, initial guess $X^{(0)}\in {\mathbb{R}}^{m\times n}$, parameter $\lambda >0$, $\sigma >0$ Output: the recovered matrix $X^{(k+1)}$ while a certain stopping criterion is not satisfied do  1: Choose a fixed step-size ${\alpha }^{-1}>0$, or choose it via the line-search rule.  2: Compute $$Y^{(k+1)}=X^{(k)}-{\alpha }^{-1}\nabla {\ell }_{\sigma }(X^{(k)}).$$  3: Perform the soft thresholding operator to obtain $$X^{(k+1)}=S_{\lambda /\alpha }(Y^{(k+1)}),$$  and set $k:=k+1$. end while

4.1. Convergence

With the Lipschitz continuity of $\nabla {\ell }_{\sigma }$ presented in Appendix A, it is a standard routine to show the convergence of Algorithms 1 and 3, i.e., let $\{X^{(k)}\}$ be a sequence generated by Algorithm 1 or 3. Then, every limit point of the sequence is a critical point of the problem. In fact, the results can be enhanced to the statement that “the entire sequence converges to a critical point”, namely one can prove that ${\lim }_{k\to \infty }{X}^{(k)}=X^{\ast }$ where $X^{\ast }$ is a critical point. This can be achieved by verifying the so-called Kurdyka–ojasiewicz (KL) property [34] of the problems (8) and (9). As this is not the main concern of this paper, we omit the verification here.

4.2. Recoverability and Linear Convergence Rate

For affine rank minimization problems, the convergence rate results have been obtained in the literature; see, e.g., [23,24]. However, all the existing results are obtained for algorithms that solve the optimization problems incorporating the least squares loss. In this part, we are concerned with the recoverability and convergence rate of Algorithm 1. These results give the understanding of this loss function from the algorithmic aspect, which is in accordance with and extends our previous work [22].

It has been known that the convergence rate analysis requires the matrix RIPcondition [33]. In our context, instead of using the matrix RIP, we adopt the concept of the matrix scalable restricted isometry property (SRIP) [24].

Definition 1

(SRIP [24]). For any $X\in S_r$, there exist constants ${\nu }_r,{\mu }_r>0$ such that:

$${\nu }_r{\parallel X\parallel }_F\le {\parallel \mathcal{A}(X)\parallel }_F\le {\mu }_r{\parallel X\parallel }_F.$$

Due to the scalability of ${\nu }_r,{\mu }_r$ on the operator $\mathcal{A}$, SRIP is a generalization of the RIP [33] as commented in [24]. We point out that the results of Algorithm 1 for the affine rank minimization problem (8) rely on the SRIP condition. However, in the matrix completion problem (5), this condition cannot be met, since ${\nu }_r$ in this case is zero. Consequently, the results provided below cannot be applied directly to the matrix completion problem (5). However, similar results might be established for (5), if some refined RIP conditions are assumed to hold for the operator $\mathcal{A}$ in the situation of matrix completion [23]. To obtain the convergence rate results, besides the SRIP condition, we also need to make some assumptions.

Assumption 1.

• At the $(k+1)$-th iteration of Algorithm 1, the parameter ${\sigma }^{k+1}$ in the loss function ${\ell }_{\sigma }$ is chosen as:

  $$\begin{array}{c}\hfill {\sigma }^{k+1}=\max \left\{\frac{\parallel \mathcal{A}(X^{(k)}){-b\parallel }_F}{\sqrt{2(1-\beta )}},\widehat{\sigma }\right\},\end{array}$$

  where $\beta \in [0.988,1)$, and $\widehat{\sigma }$ is a positive constant.
• The spectral norm of A is upper bounded as ${\parallel A\parallel }_2^2\le \frac{6}{5}{\nu }_{2R}^2.$

Based on Assumption 1, the following results for Algorithm 1 can be derived.

Theorem 1.

Assume that $\mathcal{A}(X^{\ast })+ϵ=b$, where $X^{\ast }$ is the matrix to be recovered and $\mathrm{rank}(X^{\ast })=R$. Assume that Assumption 1 holds. Let $\{X^{(k)}\}$ be generated by Algorithm 1, with the step-size $\alpha ={\parallel A\parallel }_2^2$. Then

• at iteration $(k+1)$, Algorithm 1 will recover a matrix $X^{k+1}$ satisfying:

  $$\parallel X^{(k+1)}-X^{\ast }{\parallel }_F\le q_1^{k+1}{\parallel X^{(0)}-X^{\ast }\parallel }_F+\frac{2}{1-q_1}\frac{{∥ϵ∥}_F}{{\parallel A\parallel }_2},$$

  where $q_1\in (0.8165,0.9082)$ depending on β.
• If there is no noise or outliers, i.e., $\mathcal{A}(X^{\ast })=b$, then the algorithm converges linearly in the least squares and ${\ell }_{\sigma }$ sense, respectively, i.e.,

  $$\begin{array}{ccc}\hfill \parallel \mathcal{A}(X^{(k+1)}){-b\parallel }^2& \le & q_2{\parallel \mathcal{A}(X^{(k)})-b\parallel }^2,\mathrm{and}\hfill \\ \hfill {\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k+1)})& \le & q_3{\tilde{\ell }}_{{\sigma }^k}(X^{(k)}),\hfill \end{array}$$

  where $q_2\in (0.8,0.9898)$ and $q_3\in (0.2,0.262)$, depending on the choice of β.

The proof of Theorem 1 relies on the following lemmas, which reveal certain properties of the loss function ${\tilde{\ell }}_{\sigma }$.

Lemma 1.

For any $\sigma >0$ and $t\in \mathbb{R}$, it holds:

$$\begin{array}{c}\hfill \frac{{\sigma }^2}{2}\left(1-\exp \left(\frac{-t^2}{{\sigma }^2}\right)\right)\le \frac{t^2}{2}.\end{array}$$

Proof. 

For any $\sigma >0$, let $f(t):=\frac{t^2}{2}-\frac{{\sigma }^2}{2}(1-\exp (\frac{-t^2}{{\sigma }^2}))$. Since $f(t)$ is even, we need to only consider $t\ge 0$. Note that $f^{{}^{\prime }}(t)=t-t\exp (\frac{-t^2}{{\sigma }^2})$, which is nonnegative when $t\ge 0$. Therefore, $f(t)$ is a nondecreasing function on $[0,+\infty )$. On the other hand, $f^{{}^{\prime }}(0)=0$ and $f(t)=0$. Thus, the minimum of $f(t)$ is $f(0)=0$. As a result, $f(t)\ge 0$. This completes the proof. ☐

Lemma 2.

Assuming that $\beta \in [0,1)$, and $0<\delta \le 2(1-\beta )$, it holds:

$$\begin{array}{c}\hfill g(\delta ):=1-\exp (-\delta )-\beta \delta \ge 0.\end{array}$$

Proof. 

Since $\delta >0$, it is not hard to check that $1-\exp (-\delta )\ge \delta -\frac{1}{2}{\delta }^2$. From the range of $\delta$, it follows $\delta -\frac{1}{2}{\delta }^2\ge \beta \delta$. This completes the proof. ☐

Lemma 3.

Given a fixed $t\in \mathbb{R}$, for $\sigma >0$, $h(\sigma ):={\sigma }^2(1-\exp (-t^2/{\sigma }^2))$ is nondecreasing with respect to σ.

Proof. 

It is not hard to check that $h^{\prime }$ is nonnegative on $\sigma >0$. ☐

Proof of Theorem 1.

By the fact that $X^{\ast }$ is rank-R and $X^{(k+1)}$ is the best rank-R approximation to $Y^{(k+1)}$, we have:

$$\begin{array}{cc}& \parallel X^{(k+1)}-X^{\ast }{\parallel }_F\hfill \\ \le & \parallel X^{(k+1)}-Y^{(k+1)}{\parallel }_F+{\parallel Y^{(k+1)}-X^{\ast }\parallel }_F\hfill \\ \le & 2\parallel Y^{(k+1)}-X^{\ast }{\parallel }_F\hfill \\ =& 2\parallel X^{(k)}-X^{\ast }-\frac{1}{\alpha }\nabla {\ell }_{{\sigma }^{k+1}}(X^{(k)}){\parallel }_F.\hfill \end{array}$$

Since:

$$\begin{array}{ccc}\hfill \mathrm{vec}\left(\nabla {\ell }_{{\sigma }^{k+1}}(X^{(k)})\right)& =& A^T\Lambda \left(A\mathrm{vec}(X^{(k)})-b\right)\hfill \\ & =& A^T\Lambda \left(A\mathrm{vec}(X^{(k)}-X^{\ast })-ϵ\right),\hfill \end{array}$$

we know that:

$$\begin{array}{cc}& {∥X^{(k)}-X^{\ast }-\frac{1}{\alpha }\nabla {\ell }_{{\sigma }^{k+1}}(X^{(k)})∥}_F\hfill \\ =& {∥\mathrm{vec}(X^{(k)}-X^{\ast })-\frac{1}{\alpha }{A}^T\Lambda \left(A\mathrm{vec}(X^{(k)}-X^{\ast })-ϵ\right)∥}_F\hfill \\ \le & {∥\mathrm{vec}(X^{(k)}-X^{\ast })-\frac{1}{\alpha }{A}^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })∥}_F\hfill \\ & +\frac{1}{\alpha }{∥A^T\Lambda ϵ∥}_F\hfill \\ \le & {∥\mathrm{vec}(X^{(k)}-X^{\ast })-\frac{1}{\alpha }{A}^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })∥}_F\hfill \\ & +\frac{{∥ϵ∥}_F}{{\parallel A\parallel }_2},\hfill \end{array}$$

where the last inequality follows from:

$$\parallel A^T{\Lambda ϵ\parallel }_F\le {\parallel A\parallel }_2{\parallel \Lambda \parallel }_2{\parallel ϵ\parallel }_F\le {\parallel A\parallel }_2{\parallel ϵ\parallel }_F$$

and the choice of the step-size $\alpha$. It remains to estimate $\parallel \mathrm{vec}(X^{(k)}-X^{\ast })-\frac{1}{\alpha }{A}^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast }){\parallel }_F$. We first see that:

$$\begin{array}{cc}& {∥\mathrm{vec}(X^{(k)}-X^{\ast })-\frac{1}{\alpha }{A}^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })∥}_F^2\hfill \\ =& -\frac{2}{\alpha }⟨\mathrm{vec}(X^{(k)}-X^{\ast }),A^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })⟩\hfill \\ & +\frac{1}{{\alpha }^2}{∥A^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })∥}_F^2+{∥X^{(k)}-X^{\ast }∥}_F^2\hfill \end{array}$$

(13)

To verify our first assertion, it remains to bound the first two terms by means of $\parallel X^{(k)}-X^{\ast }{\parallel }_F^2$. We consider the first term. Denoting $y_i^k=⟨A^i,X^{(k)}-X^{\ast }⟩$, we know that:

$$\begin{array}{cc}& ⟨\mathrm{vec}(X^{(k)}-X^{\ast }),A^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })⟩\hfill \\ =& ⟨A\mathrm{vec}(X^{(k)}-X^{\ast }),\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })⟩\hfill \\ =& \sum _{i=1}^p\exp \left(-{\left(\frac{⟨A^i,X^{(k)}⟩-b_i}{{\sigma }^{k+1}}\right)}^2\right){\left(y_i^k\right)}^2.\hfill \end{array}$$

The choice of ${\sigma }^{k+1}$ tells us that:

$$\exp \left(-{\left(\frac{⟨A^i,X^{(k)}⟩-b_i}{{\sigma }^{k+1}}\right)}^2\right)\ge \exp \left(-2(1-\beta )\right),$$

and consequently:

$$\begin{array}{c}-\frac{2}{\alpha }⟨\mathrm{vec}(X^{(k)}-X^{\ast }),A^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })⟩\hfill \\ \le -\frac{2}{\alpha }\exp \left(-2(1-\beta )\right){\parallel A\mathrm{vec}(X^{(k)}-X^{\ast })\parallel }_F^2.\hfill \end{array}$$

(14)

Then, by the fact that ${\parallel \Lambda \parallel }_2^2\le 1$ and the choice of the step-size $\alpha$, we observe that the second term of (13) can be upper bounded by:

$$\frac{1}{{\alpha }^2}\parallel A^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast }){\parallel }_F^2\le \frac{1}{\alpha }{\parallel A\mathrm{vec}(X^{(k)}-X^{\ast })\parallel }_F^2.$$

(15)

Combining (14) and (15) and denoting $\gamma =1-2\exp \left(-2(1-\beta )\right)$, we come to the following conclusion:

$$\begin{array}{cc}& {∥\mathrm{vec}(X^{(k)}-X^{\ast })-\frac{1}{\alpha }{A}^T\Lambda A\mathrm{vec}(X^{(k)}-X^{\ast })∥}_F^2\hfill \\ \le & {∥X^{(k)}-X^{\ast }∥}_F^2+\frac{\gamma }{\alpha }{∥A\mathrm{vec}(X^{(k)}-X^{\ast })∥}_F^2\hfill \\ \le & {∥X^{(k)}-X^{\ast }∥}_F^2+\frac{\gamma {\nu }_{2R}^2}{\alpha }{∥X^{(k)}-X^{\ast }∥}_F^2,\hfill \end{array}$$

where the last inequality follows from the SRIP condition and the fact that $\gamma <0$ by the range of $\beta$. As a result, we get the following estimation:

$$\begin{array}{ccc}\hfill \parallel X^{(k+1)}-X^{\ast }{\parallel }_F& \le & 2\parallel X^{(k)}-X^{\ast }-\frac{1}{\alpha }\nabla {\ell }_{{\sigma }^{k+1}}(X^{(k)}){\parallel }_F\hfill \\ & \le & 2\sqrt{1+\frac{\gamma {\nu }_{2R}^2}{\alpha }}{\parallel X^{(k)}-X^{\ast }\parallel }_F+2\frac{{∥ϵ∥}_F}{{\parallel A\parallel }_2}\hfill \\ & \le & 2\sqrt{1+\frac{5\gamma }{6}}{\parallel X^{(k)}-X^{\ast }\parallel }_F+2\frac{{∥ϵ∥}_F}{{\parallel A\parallel }_2}\hfill \end{array}$$

(16)

where the last inequality follows from the assumption $\alpha ={\parallel A\parallel }_2^2\le 6/5{\nu }_{2R}^2$. Denote $q_1=2\sqrt{1+\frac{5\gamma }{6}}$. The range of $\beta$ tells us that $q_1\in (0.8165,0.9082).$ Iterating (16), we obtain:

$$\parallel X^{(k+1)}-X^{\ast }{\parallel }_F\le q_1^{k+1}{\parallel X^{(0)}-X^{\ast }\parallel }_F+\frac{2}{1-q_1}\frac{{∥ϵ∥}_F}{{\parallel A\parallel }_2}.$$

Therefore, The first assertion concerning the recoverability is proven.

Suppose there is no noise or outliers, i.e., we have $\mathcal{A}(X^{\ast })=b$. In this case, it follows from (16) that:

$$\begin{array}{c}\hfill \parallel X^{(k+1)}-X^{\ast }{\parallel }_F\le q_1{\parallel X^{(k)}-X^{\ast }\parallel }_F,\end{array}$$

and then, the SRIP condition tells us that:

$$\begin{array}{ccc}\hfill \parallel \mathcal{A}(X^k){-b\parallel }_F^2& \le & {\mu }_{2R}^2{\parallel X^{k+1}-X^{\ast }\parallel }_F^2\hfill \\ & \le & {\mu }_{2R}^2{q}_1^2{\parallel X^{(k)}-X^{\ast }\parallel }_F^2\hfill \\ & \le & {\left(\frac{{\mu }_{2R}}{{\nu }_{2R}}\right)}^2{q}_1^2{\parallel \mathcal{A}(X^k)-b\parallel }_F^2\hfill \\ & \le & \frac{6}{5}{q}_1^2{\parallel \mathcal{A}(X^k)-b\parallel }_F^2,\hfill \end{array}$$

where the last inequality comes from the inequality chain ${\mu }_{2R}^2\le {\parallel A\parallel }_2^2\le 6/5{\nu }_{2R}^2$. Denote $q_2=6q_1^2/5$. Then, $q_2\in (0.8,0.9898)$. Therefore, the algorithm converges linearly to $X^{\ast }$ in the least squares sense.

We now proceed to show the linear convergence in the ${\tilde{\ell }}_{\sigma }$ sense. Following from the inequality $\parallel X^{(k+1)}-Y^{(k+1)}{\parallel }_F^2\le {\parallel X^{\ast }-Y^{(k+1)}\parallel }_F^2$, we obtain:

$$\begin{array}{cc}& \frac{\alpha }{2}{\parallel X^{(k+1)}-X^{(k)}\parallel }_F^2\hfill \\ & +⟨\nabla {\ell }_{{\sigma }^{k+1}}(X^{(k)}),X^{(k+1)}-X^{(k)}⟩\hfill \\ \le & \frac{\alpha }{2}{\parallel X^{(k)}-X^{\ast }\parallel }_F^2+⟨\nabla {\ell }_{{\sigma }^{k+1}}(X^{(k)}),X^{\ast }-X^{(k)}⟩.\hfill \end{array}$$

Combining with Inequality (A1), we see that ${\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k+1)})$ can be upper bounded by:

$${\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)})+\frac{\alpha }{2}{\parallel X^{(k)}-X^{\ast }\parallel }_F^2+⟨\nabla {\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}),X^{\ast }-X^{(k)}⟩.$$

(17)

We need to upper bound $⟨\nabla {\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}),X^{\ast }-X^{(k)}⟩$ and $\frac{\alpha }{2}{\parallel X^{(k)}-X^{\ast }\parallel }_F^2$ in terms of ${\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)})$. We first consider the second term. Under the SRIP condition, we have:

$$\begin{array}{ccc}\hfill \parallel X^{(k)}-X^{\ast }{\parallel }_F^2& \le & \frac{1}{{\nu }_{2R}^2}{\parallel \mathcal{A}(X^{(k)}-X^{\ast })\parallel }_F^2\hfill \\ & =& \frac{1}{{\nu }_{2R}^2}{\parallel \mathcal{A}(X^{(k)})-b\parallel }_F^2.\hfill \end{array}$$

By setting $\delta ={\left(\frac{y_i^k}{{\sigma }^{k+1}}\right)}^2$, we get $\delta \le 2(1-\beta )$. Lemma 2 tells us that:

$$\begin{array}{c}\hfill \beta {(y_i^k)}^2\le {({\sigma }^{k+1})}^2\left(1-\exp \left(-{(y_i^k/{\sigma }^{k+1})}^2\right)\right).\end{array}$$

Summing the above inequalities over i from 1 to p, we have:

$$\begin{array}{cc}& \beta \parallel \mathcal{A}(X^{(k)}){-b\parallel }_F^2\hfill \\ \le & {({\sigma }^{k+1})}^2\sum _{i=1}^p\left(1-\exp \left(-{(y_i^k/{\sigma }^{k+1})}^2\right)\right)\hfill \\ =& 2{\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}).\hfill \end{array}$$

Therefore, $\frac{\alpha }{2}{\parallel X^{(k)}-X^{\ast }\parallel }_F^2$ can be bounded as follows:

$$\begin{array}{ccc}\frac{\alpha }{2}{\parallel X^{(k)}-X^{\ast }\parallel }_F^2\hfill & \le \hfill & \frac{\alpha }{2{\nu }_{2R}^2}{\parallel \mathcal{A}(X^{(k)}-X^{\ast })\parallel }^2\hfill \\ & \le \hfill & \frac{\alpha }{\beta {\nu }_{2R}^2}{\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}).\hfill \end{array}$$

(18)

We proceed to bound $⟨\nabla {\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}),X^{\ast }-X^{(k)}⟩$. It follows from (14) and Lemma 1 that:

$$\begin{array}{c}⟨\nabla {\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}),X^{\ast }-X^{(k)}⟩\hfill \\ \le -\exp (-2(1-\beta ))\parallel \mathcal{A}(X^{(k)}){-b\parallel }^2\hfill \\ \le -2\exp (-2(1-\beta )){\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}).\hfill \end{array}$$

(19)

Combining (17)–(19) together, we get:

$$\begin{array}{cc}& {\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k+1)})\hfill \\ \le & \left(1+\frac{\alpha }{\beta {\nu }_{2R}^2}-2\exp (-2(1-\beta ))\right){\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)})\hfill \\ \le & \left(1+\frac{6}{5\beta }-2\exp (-2(1-\beta ))\right){\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)}),\hfill \end{array}$$

where the last inequality follows from $\alpha \le \frac{6}{5}{\nu }_{2R}^2$.

By Lemma 3, the function ${\sigma }^2(1-\exp (-t^2/{\sigma }^2))$ is nondecreasing with respect to $\sigma >0$. This in connection with the fact that:

$$\begin{array}{cc}& {\sigma }^{k+1}=\max \left\{\frac{\parallel \mathcal{A}(X^{(k)}){-b\parallel }_F}{\sqrt{2(1-\beta )}},\sigma \right\}\hfill \\ \le & {\sigma }^k=\max \left\{\frac{\parallel \mathcal{A}(X^{(k-1)}){-b\parallel }_F}{\sqrt{2(1-\beta )}},\sigma \right\}\hfill \end{array}$$

yields ${\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)})\le {\tilde{\ell }}_{{\sigma }^k}(X^{(k)})$. Let $q_3=1+\frac{6}{5\beta }-2\exp \left(-2(1-\beta )\right)$, and consequently, $q_3\in (0.2,0.2620)$. We thus have:

$${\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k+1)})\le q_3{\tilde{\ell }}_{{\sigma }^{k+1}}(X^{(k)})\le q_3{\tilde{\ell }}_{{\sigma }^k}(X^{(k)}).$$

The proof is now completed. ☐

The above results show that it is possible that Algorithm 1 will find ${\mathcal{X}}^{\ast }$ if the magnitude of the noise is not too large. Moreover, the results also imply that the algorithm is safe when there is no noise.

5. Numerical Experiments

This section presents numerical experiments to illustrate the effectiveness of our methods. Empirical comparisons with other methods are implemented on synthetic and real data contaminated by outliers or non-Gaussian noise.

The following 4 algorithms are implemented. RMC-${\ell }_{\sigma }$-IHTand RMC-${\ell }_{\sigma }$-ISTare denoted as Algorithms 1 and 3 incorporated with the line-search rule, respectively. The approach proposed in [16] is denoted as MC-${\ell }_2$-IST, which is an iterative soft thresholding algorithm based on the least squares loss. The robust approach based on the LAD loss proposed in [17] is denoted by RMC-${\ell }_1$-ADM. Empirically, the $\sigma$ value of ${\ell }_{\sigma }$ is set to be $0.5$; the tuned parameter $\lambda$ of RMC-${\ell }_{\sigma }$-IST and MC-${\ell }_2$-IST is set to $\lambda =\frac{\min \{m,n\}}{10\sqrt{\max \{m,n\}}}$, while for RMC-${\ell }_1$-ADM, $\lambda =1/\sqrt{\max \{m,n\}}$, as suggested in [17]. All the numerical computations are conducted on an Intel i7-3770 CPU desktop computer with 16 GB of RAM. The supporting software is MATLAB R2013a. Some notations used frequently in this section are introduced first in Table 1. Bold number in the tables of this section means that it is the best among the competitors.

Table 1. Notations used in the experiments.

Notations	Descriptions
${\rho }_r$	the ratio of the rank to the dimensionality of a matrix
${\rho }_o$	the ratio of outliers to the number of entries of a matrix
${\rho }_m$	the level of missing entries
$s_n$	the factor of scale of noise

5.1. Evaluation on Synthetic Data

The synthetic datasets are generated in the following way:

• Generating a low rank matrix: We first generate an $m\times n$ matrix with i.i.d. Gaussian entries ∼N(0,1), where $m=n=1000$. Then, a $\lfloor {\rho }_{r}m\rfloor$-rank matrix M is obtained from the above matrix by rank truncation, where ${\rho }_r$ varies from $0.04$–$0.4$.
• Adding outliers: We create a zero matrix $E\in {\mathbb{R}}^{m\times n}$ and uniformly randomly sample ${\rho }_o{m}^2$ entries, where ${\rho }_o$ varies from 0–$0.6$. These entries are randomly drawn from the chi-square distribution, with four degrees of freedom. Multiplied by 10, the matrix E is used as the sparse error matrix.
• Missing entries: ${\rho }_m{m}^2$ of the entries are randomly missing, with ${\rho }_m$ varying between $\{0,10\%,20\%,30\%\}$. Finally, the observed matrix is denoted as $B=P_{\Omega }(M+E)$.

RMC-${\ell }_{\sigma }$-IHT (Algorithm 1), RMC-${\ell }_{\sigma }$-IST (Algorithm 3) and RMC-${\ell }_1$-ADM [17] are implemented respectively on the matrix completion problem with the datasets generated above. For these three algorithms, the same initial guess with the all-zero matrix $X^0=0$ is applied. The stopping criterion is $\parallel X^{(k+1)}-X^{(k)}{\parallel }_F\le {10}^{-3}$, or restrictions on the number of iterations, which is set to be 500. For each tuple $({\rho }_m,{\rho }_r,{\rho }_o)$, we repeat 10 runs. The algorithm is regarded as successful if the relative error of the result $\widehat{X}$ satisfies $\parallel \widehat{X}{-M\parallel }_F/{\parallel M\parallel }_F\le {10}^{-1}$.

Experimental results of RMC-${\ell }_{\sigma }$-IHT (top), RMC-${\ell }_{\sigma }$-IST (middle) and RMC-${\ell }_1$-ADM (bottom) are reported in Figure 2, which are given in terms of phase transition diagrams. In Figure 2, the white zones denote perfect recovery in all the experiments, while the black ones denote failure for all the experiments. In each diagram, the x-axis represents the ratio of rank, i.e., we let ${\rho }_r=\frac{\mathrm{rank}}{m}\in [0.04,0.4]$, and the y-axis represents the level of outliers, i.e., we let ${\rho }_o=\frac{♯\mathrm{outliers}}{m^2}\in [0,0.6]$. The level of missing entries ${\rho }_m$ varies from left to right in each row. As shown in Figure 2, our approach outperforms RMC-${\ell }_1$-ADM when ${\rho }_o$ and ${\rho }_r$ increase. We also observe that RMC-${\ell }_{\sigma }$-IHT performs better than RMC-${\ell }_{\sigma }$-IST when the level of outliers increases, while RMC-${\ell }_{\sigma }$-IST outperforms RMC-${\ell }_{\sigma }$-IHT when the ratio of missing entries increases.

Figure 2. Phase transition diagrams of RMC-${\ell }_{\sigma }$-IHT (Algorithm 1), RMC-${\ell }_{\sigma }$-IST (Algorithm 3) and RMC-${\ell }_1$-ADM [17]. The first row: RMC-${\ell }_{\sigma }$-IHT; the second row: RMC-${\ell }_{\sigma }$-IST; the last row: RMC-${\ell }_1$-ADM. x-axis: ${\rho }_r\in [0.04,0.4]$; y-axis: ${\rho }_o\in [0,0.6]$. From the first column to the last column, ${\rho }_m$ varies from 0–$30\%$.

Comparison of the computational time and the relative error are also reported in Table 2. In this experiment, the level of missing entries ${\rho }_m=\{20\%,30\%\}$, the ratio of rank ${\rho }_r=0.1$ and the level of outliers ${\rho }_o$ varies between $\{0.1,0.15,0.2,0.25,0.3\}$. For each ${\rho }_o$, we randomly generate 20 instances and then average the results. In the table, “time” denotes the CPU time, with the unit being second, and “rel.err” represents the relative error introduced in the previous paragraph. The results also demonstrate the improved performance of our methods in most of the cases on CPU time and relative error, especially for RMC-${\ell }_{\sigma }$-IHT.

Table 2. Comparison of RMC-${\ell }_{\sigma }$-IHT(Algorithm 1), RMC-${\ell }_{\sigma }$-IST(Algorithm 3) and RMC-${\ell }_1$-ADM [17] on CPU time and the relative error on synthetic data. ${\rho }_m=0.3$, ${\rho }_r=0.1$. rel.err, relative error.

${\mathit{\rho }}_{\mathit{m}}$	${\mathit{\rho }}_{\mathit{o}}$	RMC-${\mathit{\ell }}_{\mathit{\sigma }}$-IHT		RMC-${\mathit{\ell }}_{\mathit{\sigma }}$-IST		RMC-${\mathit{\ell }}_1$-ADM
		Algorithm 1		Algorithm 3		[17]
		Time	rel.err	Time	rel.err	Time	rel.err
	0.1	15.43	3.80$\times {10}^{-03}$	20.53	4.55$\times {10}^{-02}$	19.24	2.58$\times {10}^{-06}$
	0.15	15.31	4.40$\times {10}^{-03}$	21.26	4.96$\times {10}^{-02}$	18.32	2.33$\times {10}^{-06}$
	0.2	16.93	5.40$\times {10}^{-03}$	22.95	5.53$\times {10}^{-02}$	48.97	2.82$\times {10}^{-04}$
	0.25	19.04	5.80$\times {10}^{-03}$	26.41	6.23$\times {10}^{-02}$	243.80	1.07$\times {10}^{-01}$
	0.3	27.10	7.00$\times {10}^{-03}$	29.47	7.01$\times {10}^{-02}$	137.99	3.16$\times {10}^{-01}$
0.2	0.35	26.35	8.00$\times {10}^{-03}$	36.03	8.10$\times {10}^{-02}$	99.26	4.86$\times {10}^{-01}$
	0.4	23.91	1.03$\times {10}^{-02}$	37.41	9.41$\times {10}^{-02}$	79.85	6.38$\times {10}^{-01}$
	0.45	29.64	1.24$\times {10}^{-02}$	45.68	1.10$\times {10}^{-01}$	67.45	7.77$\times {10}^{-01}$
	0.5	40.41	1.69$\times {10}^{-02}$	61.39	1.37$\times {10}^{-01}$	60.08	9.52$\times {10}^{-01}$
	0.55	60.28	2.45$\times {10}^{-02}$	103.87	1.80$\times {10}^{-01}$	68.52	1.39$\times {10}^{+00}$
	0.6	102.19	3.69$\times {10}^{-02}$	154.04	2.65$\times {10}^{-01}$	144.37	2.86$\times {10}^{+00}$
	0.1	16.38	5.20$\times {10}^{-03}$	24.14	5.66$\times {10}^{-02}$	24.81	2.86$\times {10}^{-06}$
	0.15	20.14	5.00$\times {10}^{-03}$	23.85	6.41$\times {10}^{-02}$	110.67	8.30$\times {10}^{-03}$
	0.2	22.83	6.00$\times {10}^{-03}$	25.92	7.00$\times {10}^{-02}$	117.91	1.15$\times {10}^{-01}$
	0.25	20.71	7.00$\times {10}^{-03}$	28.93	7.97$\times {10}^{-02}$	118.10	3.08$\times {10}^{-01}$
	0.3	20.77	8.80$\times {10}^{-03}$	32.99	9.21$\times {10}^{-02}$	89.56	4.68$\times {10}^{-01}$
0.3	0.35	21.28	8.20$\times {10}^{-03}$	33.72	9.09$\times {10}^{-02}$	88.73	4.66$\times {10}^{-01}$
	0.4	27.64	1.15$\times {10}^{-02}$	41.53	1.05$\times {10}^{-01}$	75.07	5.98$\times {10}^{-01}$
	0.45	32.38	1.40$\times {10}^{-02}$	48.45	1.23$\times {10}^{-01}$	71.14	7.13$\times {10}^{-01}$
	0.5	44.53	1.68$\times {10}^{-02}$	84.67	1.50$\times {10}^{-01}$	73.63	8.02$\times {10}^{-01}$
	0.55	62.23	2.26$\times {10}^{-02}$	125.48	1.95$\times {10}^{-01}$	78.34	8.84$\times {10}^{-01}$
	0.6	92.14	3.26$\times {10}^{-02}$	241.35	2.78$\times {10}^{-01}$	74.09	1.07$\times {10}^{+00}$

5.2. Image Inpainting and Denoising

One typical application of matrix completion is the image inpainting problem [4]. The datasets and the experiment are conducted as follows:

• We first choose five gray images, named “Baboon”, “Camera Man”, “Lake”, “Lena” and “Pepper” (the size of each image is $512\times 512$), each of which is stored in a matrix M.
• The outliers matrix E is added to each M, where E is generated in the same way as the previous experiment, and the level of outliers ${\rho }_o$ varies among $\{0.3,0.4,0.5,0.6,0.7\}$.
• The ratio of the missing entries is set to $30\%$. RMC-${\ell }_{\sigma }$-IST, RMC-${\ell }_1$-ADM and MC-${\ell }_2$-IST, are tested in this experiment. In addition, we also test the Cauchy loss-based model ${\min }_X{\ell }_c(X)+\lambda {\parallel X\parallel }_{\ast }$, which is denoted as RMC-${\ell }_c$-IST, where:

  $${\ell }_c:=\frac{c^2}{2}\sum _{(i,j)\in \Omega }ln\left(1+{\left(X_{ij}-B_{ij}\right)}^2/c^2\right),$$

  where $c>0$ is a parameter controlling the robustness. Empirically, we set $c=0.15$. Other parameters are set to the same as those of RMC-${\ell }_{\sigma }$-IST. The above model is also solved by soft thresholding similar to Algorithm 3. Note that Cauchy loss has a similar shape as that of Welsch loss and also enjoys the redescending property; such a loss function is also frequently used in the robust statistics literature. The initial guess is $X^0=0$. The stopping criterion is $\parallel X^{(k+1)}-X^{(k)}{\parallel }_F\le {10}^{-2}$, or the iterations exceed 500.

Detailed comparison results in terms of the relative error and CPU time are listed in Table 3, from which one can see the efficiency of our method. Indeed, experimental results show that our method can be terminated within 80 iterations. According to the relative error in Table 3, our method performs the best in almost all cases, followed by RMC-${\ell }_c$-IST. This is not surprising because the Cauchy loss-based model enjoys similar properties as the proposed model. We also observe that the RMC-${\ell }_1$-ADM algorithm cannot deal with situations when images are heavily contaminated by outliers. This illustrates the robustness of our method.

Table 3. Experimental results of RMC-${\ell }_{\sigma }$-IST (Algorithm 3), RMC-${\ell }_1$-ADM [17] and MC-${\ell }_2$-IST [16] on different images with ${\rho }_r=0.1$, ${\rho }_m=0.3$ and ${\rho }_o$ varying from $0.3$ to $0.7$.

${\mathit{\rho }}_{\mathit{o}}$	Images	Baboon		Camera Man		Lake		Lena		Pepper
	Method	Time	rel.err	Time	rel.err	Time	rel.err	Time	rel.err	Time	rel.err
	RMC-${\ell }_{\sigma }$-IST (Algorithm 3)	3.17	1.46$\times {10}^{-02}$	3.55	1.74$\times {10}^{-02}$	3.79	1.61$\times {10}^{-02}$	4.36	2.05$\times {10}^{-02}$	3.80	1.10$\times {10}^{-02}$
0.3	RMC-${\ell }_1$-ADM [17]	32.22	2.86$\times {10}^{-02}$	35.87	4.36$\times {10}^{-02}$	26.74	4.57$\times {10}^{-02}$	20.67	3.98$\times {10}^{-02}$	33.08	2.46$\times {10}^{-02}$
	MC-${\ell }_2$-IST [16]	68.33	4.35$\times {10}^{+00}$	72.44	4.44$\times {10}^{+00}$	68.39	4.14$\times {10}^{+00}$	68.68	4.22$\times {10}^{+00}$	68.38	3.07$\times {10}^{+00}$
	RMC-${\ell }_c$-IST	5.19	1.38$\times {10}^{-02}$	5.60	1.83$\times {10}^{-02}$	5.24	1.70$\times {10}^{-02}$	4.73	2.46$\times {10}^{-02}$	4.36	1.61$\times {10}^{-02}$
	RMC-${\ell }_{\sigma }$-IST (Algorithm 3)	3.76	1.73$\times {10}^{-02}$	3.94	2.15$\times {10}^{-02}$	4.69	1.96$\times {10}^{-02}$	4.58	2.41$\times {10}^{-02}$	4.91	1.42$\times {10}^{-02}$
0.4	RMC-${\ell }_1$-ADM [17]	30.93	3.51$\times {10}^{-02}$	36.76	5.16$\times {10}^{-02}$	26.67	5.48$\times {10}^{-02}$	22.41	4.76$\times {10}^{-02}$	32.18	3.28$\times {10}^{-02}$
	MC-${\ell }_2$-IST [16]	68.51	5.07$\times {10}^{+00}$	68.94	5.08$\times {10}^{+00}$	68.09	4.74$\times {10}^{+00}$	68.84	4.88$\times {10}^{+00}$	68.68	3.54$\times {10}^{+00}$
	RMC-${\ell }_c$-IST	4.88	1.70$\times {10}^{-02}$	5.73	2.37$\times {10}^{-02}$	5.34	2.21$\times {10}^{-02}$	5.39	2.89$\times {10}^{-02}$	5.56	1.87$\times {10}^{-02}$
	RMC-${\ell }_{\sigma }$-IST (Algorithm 3)	4.01	2.13$\times {10}^{-02}$	4.44	2.61$\times {10}^{-02}$	5.29	2.40$\times {10}^{-02}$	5.27	2.76$\times {10}^{-02}$	6.77	1.63$\times {10}^{-02}$
0.5	RMC-${\ell }_1$-ADM [17]	24.95	4.91$\times {10}^{-02}$	27.69	6.57$\times {10}^{-02}$	22.75	6.92$\times {10}^{-02}$	20.74	6.71$\times {10}^{-02}$	26.86	3.98$\times {10}^{-02}$
	MC-${\ell }_2$-IST [16]	68.30	5.56$\times {10}^{+00}$	69.64	5.62$\times {10}^{+00}$	68.71	5.37$\times {10}^{+00}$	68.56	5.44$\times {10}^{+00}$	68.71	3.91$\times {10}^{+00}$
	RMC-${\ell }_c$-IST	6.63	2.18$\times {10}^{-02}$	6.94	2.95$\times {10}^{-02}$	5.84	2.90$\times {10}^{-02}$	6.10	3.32$\times {10}^{-02}$	6.94	2.15$\times {10}^{-02}$
	RMC-${\ell }_{\sigma }$-IST (Algorithm 3)	4.98	2.65$\times {10}^{-02}$	6.36	3.37$\times {10}^{-02}$	7.96	3.11$\times {10}^{-02}$	5.75	3.49$\times {10}^{-02}$	9.52	2.20$\times {10}^{-02}$
0.6	RMC-${\ell }_1$-ADM [17]	15.55	1.41$\times {10}^{-01}$	15.21	1.61$\times {10}^{-01}$	15.23	1.48$\times {10}^{-01}$	15.56	1.38$\times {10}^{-01}$	15.95	9.71$\times {10}^{-02}$
	MC-${\ell }_2$-IST [16]	68.22	6.06$\times {10}^{+00}$	69.93	6.17$\times {10}^{+00}$	68.73	5.77$\times {10}^{+00}$	68.34	5.88$\times {10}^{+00}$	68.51	4.23$\times {10}^{+00}$
	RMC-${\ell }_c$-IST	7.93	2.70$\times {10}^{-02}$	6.08	4.51$\times {10}^{-02}$	8.19	3.22$\times {10}^{-02}$	7.87	3.81$\times {10}^{-02}$	10.36	2.85$\times {10}^{-02}$
	RMC-${\ell }_{\sigma }$-IST (Algorithm 3)	8.74	3.59$\times {10}^{-02}$	11.37	4.41$\times {10}^{-02}$	11.75	4.21$\times {10}^{-02}$	9.59	4.16$\times {10}^{-02}$	19.95	2.69$\times {10}^{-02}$
0.7	RMC-${\ell }_1$-ADM [17]	44.31	1.90$\times {10}^{+00}$	44.63	1.96$\times {10}^{+00}$	45.16	1.81$\times {10}^{+00}$	43.49	1.85$\times {10}^{+00}$	43.88	1.37$\times {10}^{+00}$
	MC-${\ell }_2$-IST [16]	68.54	6.52$\times {10}^{+00}$	68.75	6.59$\times {10}^{+00}$	69.06	6.18$\times {10}^{+00}$	68.41	6.22$\times {10}^{+00}$	68.62	4.52$\times {10}^{+00}$
	RMC-${\ell }_c$-IST	13.12	3.59$\times {10}^{-02}$	23.03	5.03$\times {10}^{-02}$	15.19	4.36$\times {10}^{-02}$	22.95	4.68$\times {10}^{-02}$	14.78	3.86$\times {10}^{-02}$

To better illustrate the robustness of our method empirically, we also attach images recovered by the three methods in Figure 3. For the sake of saving space, we merely list the recovery results for the case ${\rho }_o=0.6$ with $30\%$ missing entries. In Figure 3, the first column represents five original images, namely, “Baboon”, “Camera Man”, “Lake”, “Lena” and “Pepper”. Images in the second column are contaminated images with $60\%$ outliers and $30\%$ missing entries. Recovered results of each image are report in the remaining columns respectively by using RMC-${\ell }_{\sigma }$-IST, RMC-${\ell }_1$-ADM, MC-${\ell }_2$-IST and RMC-${\ell }_c$-IST. One can observe that the images recovered by our method retain most of the important information, followed by RMC-${\ell }_c$-IST.

Figure 3. Comparison of RMC-${\ell }_{\sigma }$-IST, RMC-${\ell }_1$-ADM and MC-${\ell }_2$-IST on different images with $60\%$ outliers and $30\%$ missing entries. (a) The original low rank images; (b) images with $30\%$ missing entries and contaminated by $70\%$ outliers; (c) images recovered by RMC-${\ell }_{\sigma }$-IST (Algorithm 3); (d) images recovered by RMC-${\ell }_1$-ADM [17]; (e) images recovered by MC-${\ell }_2$-IST [16]; (f) images recovered by RMC-${\ell }_c$-IST.

Our next experiment is designed to show the effectiveness of our method in dealing with the non-Gaussian noise. We assume that the entries of the noise matrix E are i.i.d drawn from Student’s t distribution, with three degrees of freedom. We then scale E by a factor $s_n$, and we denote the corresponding $E:=s_n·E$. The noise scale factor $s_n$ varies in $\{0.01,0.05,0.1\}$, and ${\rho }_m$ varies in $\{0.1,0.3,0.5\}$. The results are shown in Table 4, where the image “Building” is used. We list the recovered images in Figure 4 with the case $s_n=0.05$. From the table and the recovered images, we can see that our method also performs well when the image is only contaminated by non-Gaussian noise.

Table 4. Experimental results on the image “Building”, contaminated by non-Gaussian noise with varying ${\rho }_m$ and the noise scale.

${\mathit{s}}_{\mathit{n}}$	${\mathit{\rho }}_{\mathit{m}}$	RMC-${\mathit{\ell }}_{\mathit{\sigma }}$-IST		RMC-${\mathit{\ell }}_1$-ADM		MC-${\mathit{\ell }}_2$-IST
		Algorithm 3		[17]		[16]
		Time	rel.err	Time	rel.err	Time	rel.err
	0.1	0.91	6.70$\times {10}^{-03}$	2.57	1.76$\times {10}^{-02}$	0.59	6.70$\times {10}^{-03}$
0.01	0.3	0.90	9.60$\times {10}^{-03}$	2.40	2.32$\times {10}^{-02}$	0.85	9.60$\times {10}^{-03}$
	0.5	1.05	1.44$\times {10}^{-02}$	2.77	3.24$\times {10}^{-02}$	1.29	1.44$\times {10}^{-02}$
	0.1	1.24	1.58$\times {10}^{-02}$	1.17	2.16$\times {10}^{-02}$	0.82	1.91$\times {10}^{-02}$
0.05	0.3	1.11	2.03$\times {10}^{-02}$	1.37	2.70$\times {10}^{-02}$	1.64	3.63$\times {10}^{-02}$
	0.5	1.32	2.49$\times {10}^{-02}$	2.22	3.61$\times {10}^{-02}$	1.94	2.88$\times {10}^{-02}$
	0.1	2.34	3.31$\times {10}^{-02}$	1.08	3.04$\times {10}^{-02}$	1.35	5.72$\times {10}^{-02}$
0.1	0.3	3.30	3.40$\times {10}^{-02}$	1.44	3.78$\times {10}^{-02}$	2.32	4.28$\times {10}^{-02}$
	0.5	3.70	4.66$\times {10}^{-02}$	2.42	5.53$\times {10}^{-02}$	3.98	1.55$\times {10}^{-01}$

Figure 4. Recovery results of RMC-${\ell }_{\sigma }$-IST (third), RMC-${\ell }_1$-ADM (fourth) and MC-${\ell }_2$-IST (fifth) on the image “Building” contaminated by non-Gaussian noise with $s_n=0.05$ and 30% missing entries.

5.3. Background Subtraction

Background subtraction, also known as foreground detection, is one of the major tasks in computer vision, which aims at detecting changes in image or video sequences and finds application in video surveillance, human motion analysis and human-machine interaction from static cameras [35].

Given a sequence of images, one can cast them into a matrix B by vectorizing each image and then stacking row by row. In many cases, it is reasonable to assume that the background varies little. Consequently, the background forms a low rank matrix M, while the foreground activity is spatially localized and can be seen as the error matrix E. Correspondingly, the image sequence matrix B can be expressed as the sum of a low rank background matrix M and a sparse error matrix E, which represents the activity in the scene.

In practice, it is reasonable to assume that some entries of the image sequence are missing and the images are contaminated by noise or outliers. Therefore, the foreground object detection problem can be formulated as a robust matrix completion problem. Ref. [36] proposed to use the LAD-loss-based matrix completion approach to separate M and E. The data of this experiment were downloaded from http://perception.i2r.a-star.edu.sg/bkmodel/bkindex.html.

Our experiment in this scenario is implemented as follows:

• We choose the sequence named “Restaurant” for our experiment, which consists of 3057 color images. Each image of “Restaurant” is $160\times 120$ in size. From the sequence, we pick 100 continuous images and convert them to gray images to form the original matrix B, which is $100\times 19200$ in size, where each row is a vector converted from an image.
• Two types of non-Gaussian noise are added to B. The first type of noise is drawn from the chi-square distribution, with four degree of freedom; the second type of noise is drawn from Student’s t distribution, with three degrees of freedom. Then, the two types of noise are simultaneously rescaled by $s_n=\{0.01,0.02,0.05\}$. The last $50\%$ of the entries are missing randomly.
• RMC-${\ell }_{\sigma }$-IHT and RMC-${\ell }_1$-ADM are used to deal with this problem. We set $R=1$ in RMC-${\ell }_{\sigma }$-IHT. The initial guess is the zero matrix. The stopping criterion is $\parallel X^{(k+1)}-X^{(k)}{\parallel }_F\le {10}^{-2}$, or the iterations exceed 200.

The running time and relative error are reported in Table 5. From the table, we see that the proposed approach is faster and gives smaller relative errors. To give an intuitive impression, we choose five frames from each image sequence, as shown in Figure 5, from which we can observe that when the image sequences are corrupted by noise ($s_n=0.05$) and missing entries, both of the methods can successfully extract the background and foreground images, and it seems that our method performs better because the details of the background images are recovered well, whereas the LAD-based approach does not seem to perform as well as ours where some details of the background are added to the foreground. It can be also observed that none of the two methods can recover the missing entries in the foreground. In order to achieve this, maybe more effective approaches are needed.

Table 5. Experiment results on “Restaurant” contaminated by non-Gaussian noise and $50\%$ missing entries.

${\mathit{s}}_{\mathit{n}}$	Method	Time	rel.err
0.01	RMC-${\ell }_{\sigma }$-IHT (Algorithm 1)	70.58	9.77$\times {10}^{-02}$
	RMC-${\ell }_1$-ADM [17]	229.88	1.14$\times {10}^{-01}$
0.02	RMC-${\ell }_{\sigma }$-IHT (Algorithm 1)	58.51	9.78$\times {10}^{-02}$
	RMC-${\ell }_1$-ADM [17]	230.24	1.30$\times {10}^{-01}$
0.05	RMC-${\ell }_{\sigma }$-IHT (Algorithm 1)	99.87	1.14$\times {10}^{-01}$
	RMC-${\ell }_1$-ADM [17]	221.60	2.37$\times {10}^{-01}$

Figure 5. Comparison between RMC-${\ell }_{\sigma }$-IHT (Algorithm 1) and RMC-${\ell }_1$-ADM [17] on extracting the image sequence “Restaurant” with ${\rho }_m=50\%$ and contaminated by two types of non-Gaussian noise with $s_n=0.05$. (a) The original image sequence; (b) the image sequence with missing entries and contaminated by noise; (c) background extracted by RMC-${\ell }_{\sigma }$-IHT (Algorithm 1); (d) foreground extracted by RMC-${\ell }_{\sigma }$-IHT (Algorithm 1); (e) background extracted by RMC-${\ell }_1$-ADM [17]; (f) foreground extracted by RMC-${\ell }_1$-ADM [17].

6. Concluding Remarks

The correntropy loss function has been studied in the literature [20,21] and has found many successful applications [29,30,31]. Learning with correntropy-induced losses could be resistant to non-Gaussian noise and outliers while ensuring good prediction accuracy simultaneously with properly chosen parameter $\sigma$. This paper addressed the robust matrix completion problem based on the correntropy loss. The proposed approach was shown to be efficient to deal with non-Gaussian noise and sparse gross errors. The nonconvexity of the proposed approach was due to using the ${\ell }_{\sigma }$ loss. Based on the above approach, we proposed two nonconvex optimization models and extend them to the more general robust affine rank minimization problems. Two gradient-based iterative schemes to solve the nonconvex optimization problems were offered, with convergence rate results being obtained under proper assumptions. It would be interesting to investigate similar convergence and recoverability results for other redescending-type loss functions-based models. Numerical experiments verified the improved performance of our methods, where empirically, the parameter $\sigma$ for ${\ell }_{\sigma }$ is set to $0.5$ and $\lambda$ for the nuclear norm model (6) is $\lambda =\frac{\min \{m,n\}}{10\sqrt{\max \{m,n\}}}$.