Guaranteed Robust Tensor Completion via ∗_L-SVD with Applications to Remote Sensing Data

Abstract

This paper conducts a rigorous analysis for the problem of robust tensor completion, which aims at recovering an unknown three-way tensor from incomplete observations corrupted by gross sparse outliers and small dense noises simultaneously due to various reasons such as sensor dead pixels, communication loss, electromagnetic interferences, cloud shadows, etc. To estimate the underlying tensor, a new penalized least squares estimator is first formulated by exploiting the low rankness of the signal tensor within the framework of tensor ${\ast }_L$-Singular Value Decomposition (${\ast }_L$-SVD) and leveraging the sparse structure of the outlier tensor. Then, an algorithm based on the Alternating Direction Method of Multipliers (ADMM) is designed to compute the estimator in an efficient way. Statistically, the non-asymptotic upper bound on the estimation error is established and further proved to be optimal (up to a log factor) in a minimax sense. Simulation studies on synthetic data demonstrate that the proposed error bound can predict the scaling behavior of the estimation error with problem parameters (i.e., tubal rank of the underlying tensor, sparsity of the outliers, and the number of uncorrupted observations). Both the effectiveness and efficiency of the proposed algorithm are evaluated through experiments for robust completion on seven different types of remote sensing data.

1. Introduction

Despite the broad adoption of advanced sensors in various remote sensing tasks, the quality of data remains a critical issue and can significantly influence the actual performances of the backend applications. Many types of modern remote sensing data in the modality of optical, hyperspectral, multispectral, thermal, Light Detection and Ranging (LiDAR), Synthetic Aperture Radar (SAR), etc., are typically multi-way and can be readily stored, analyzed, and processed by tensor-based models [1,2,3,4,5,6,7]. In some extreme circumstances, the data tensor may encounter missing entries, gross sparse outliers, and small dense noises at the same time, as a result of partial sensor failures, communication errors, occlusion by obstacles, and so on [8,9]. To robustly complete a partially observed data tensor corrupted by outliers and noises, the problem of robust tensor completion arises.

When only a fraction of partially corrupted observations are available, the crucial point of robust tensor completion lies in the assumption that the underlying data tensor is highly redundant such that the main components of it remain only slightly suppressed by missing information, outliers, and noises, and thus can be effectively reconstructed by exploiting the intrinsic redundancy. The tensor low-rankness is an ideal tool to model the redundancy of tensor data, and has gained extensive attention in remote sensing data restoration [5,10,11].

As higher-order extensions of low-rank matrix models [12], low-rank tensor models are typically formulated as minimization problems of the tensor rank function [13]. However, there are multiple definitions of tensor ranks, such as the CP rank [14], Tucker rank [15], TT rank [16], TR rank [17], etc., which focus on low rank structures in the original domains (like the pixel domain of optimal images) [18,19]. Recently, a remarkably different example named the low-tubal-rank tensor model [20,21] was proposed within the algebraic framework of tensor Singular Value Decomposition (t-SVD) [20,22], which captures low-rankness in the frequency domain defined via Discrete Fourier Transform (DFT). As discussed in [18,19,21,23], the low-tubal-rank tensor models are capable to exploit both low-rankness and smoothness of the tensor data, making it quite suitable to analyze and process diverse remote sensing imagery data which are often simultaneously low-rank and smooth [5,10].

Motivated by the advantages of low-tubal-rankness in modeling remote sensing data, we resolve the robust tensor completion problem by utilizing a generalized low-tubal-rank model based on the tensor ${\ast }_L$-Singular Value Decomposition (${\ast }_L$-SVD) [24], which leverages low-rankness in more general transformed domains rather than DFT. What needs to be pointed out is that the ${\ast }_L$-SVD has become a research focus in tensor-based signal processing, computer vision, and machine learning very recently [18,23,25,26]. Regarding the preference of theory in this paper, we only introduce several typical works with statistical analysis as follows. For tensor completion in the noiseless settings, Lu et al. [26] proposed a ${\ast }_L$-SVD-based model which can exactly recover the underlying tensor under mild conditions. For tensor completion from partial observations corrupted by sparse outliers, Song et al. [27] designed a ${\ast }_L$-SVD-based algorithm with exact recovery guarantee. Zhang et al. [25] developed a theoretically guaranteed approach via the ${\ast }_L$-SVD to for tensor completion from Poisson noises. The problem of tensor recovery from noisy linear observations is studied in [18] based the ${\ast }_L$-SVD with guaranteed statistical performance.

In this paper, we focus on statistical guaranteed approaches in a more challenging setting than the aforementioned ${\ast }_L$-SVD-based models, where the underlying signal tensor suffers from missing entries, sparse outliers, and small dense noises simultaneously. Specifically, we resolve the problem of robust tensor completion by formulating a ${\ast }_L$-SVD-based estimator whose estimation error is established and further proved to be minimax optimal (up to a log factor). We propose an algorithm based on Alternating Direction Method of Multipliers (ADMM) [28,29] to compute the estimator and evaluate both the effectiveness and efficiency on seven different types of remote sensing data.

The remainder of this paper proceeds as follows. We first introduce some notation and preliminaries in Section 2. Then, the proposed estimator for robust tensor completion is formulated in Section 3. We compute the estimator by using an ADMM-based algorithm described in Section 4. The statistical performance of the proposed estimator is analyzed in Section 5. Experimental results on both synthetic and real datasets are reported in Section 7. We summarize this paper and discuss future directions briefly in Section 8. The proofs of the theoretical results are given in Appendix A.

2. Preliminaries

In this section, we first introduce some notations and then give a brief introduction to the ${\ast }_L$-SVD framework.

2.1. Notations

Main notations are listed in Table 1. Let $\left[d\right]:=\{1,\dots ,d\}$, $\forall d\in {\mathbb{N}}_{+}$. Let $a\vee b=max\{a,b\}$ and $a\wedge b=min\{a,b\}$, $\forall a,b\in {\mathbb{R}}^{}$. For $i\in \left[d\right],{\mathbf{e}}_i\in {\mathbb{R}}^d$ denotes the standard vector basis whose $i_{th}$ entry is 1 with the others 0. For $(i,j,k)\in \left[d_1\right]\times \left[d_2\right]\times \left[d_3\right]$, the outer product ${\mathbf{e}}_i\circ {\mathbf{e}}_j\circ {\mathbf{e}}_k$ denotes a standard tensor basis in ${\mathbb{R}}^{d_1\times d_2\times d_3}$, whose ${(i,j,k)}_{th}$ entry is 1 with the others 0. For a 3-way tensor, a tube is a vector defined by fixing indices of the first two modes and varying the third one; A slice is a matrix defined by fixing all but two indices. For any set $\Theta$, $|\Theta |$ denotes its cardinality and ${\Theta }^{\perp }$ its complement. Absolute positive constants are denoted by $C,c$, $c_0,\mathrm{etc}$ whose values may vary from line to line. When the field and size of a tensor are not shown explicitly, it is defaulted to be in ${\mathbb{R}}^{d_1\times d_2\times d_3}$. The spectral norm $\parallel ·\parallel$ and nuclear norm ${\parallel ·\parallel }_{\ast }$ of a matrix are the maximum and the sum of the singular values, respectively.

Table 1. List of notations.

Notations	Descriptions	Notations	Descriptions
t	a scaler	$\mathbf{T}$	a matrix
$\mathbf{t}$	a vector	$\mathcal{T}$	a tensor
${\mathcal{L}}^{\ast }$	the true low-rank tensor	$\widehat{\mathcal{L}}$	the estimator of ${\mathcal{L}}^{\ast }$
${\mathcal{S}}^{\ast }$	the true sparse tensor	$\widehat{\mathcal{S}}$	the estimator of ${\mathcal{S}}^{\ast }$
$y_i$	a scalar observation	${\xi }_i$	Gaussian noise
${\mathcal{X}}_i$	a design tensor	N	number of observations
$N_{\iota }$	number of uncorrupted observations	$N_s$	$N-N_{\iota }$
${\Theta }_s$	support of corruption tensor ${\mathcal{S}}^{\ast }$	${\Theta }_s^{\perp }$	complement of ${\Theta }_s$
$\mathfrak{X}(·)$	design operator	${\mathfrak{X}}^{\ast }(·)$	adjoint operator of $\mathfrak{X}(·)$
$\mathbf{L}$	an orthogonal matrix in ${\mathbb{R}}^{d_3\times d_3}$	$L\left(\mathcal{T}\right):=\mathcal{T}{\times }_3\mathbf{L}$	tensor L-transform
$\overline{\mathbf{T}}$	block-diagonal matrix of $L\left(\mathcal{T}\right)$	${\parallel \mathcal{T}\parallel }_{sp}:=\parallel \overline{\mathbf{T}}\parallel$	tensor spectral norm
${\mathcal{T}}_{ijk}$	${(i,j,k)}_{th}$ entry of $\mathcal{T}$	${\parallel \mathcal{T}\parallel }_{\star }:={\parallel \overline{\mathbf{T}}\parallel }_{\ast }$	tubal nuclear norm
$\mathcal{T}(i,j,:)$	${(i,j)}_{th}$ tube of $\mathcal{T}$	${\parallel \mathcal{T}\parallel }_1:={\sum }_{ijk}\left|{\mathcal{T}}_{ijk}\right|$	tensor $l_1$-norm
$\mathcal{T}(:,:,k)$	$k_{th}$ frontal slice of $\mathcal{T}$	${\parallel \mathcal{T}\parallel }_{\mathrm{F}}:=\sqrt{{\sum }_{ijk}{\mathcal{T}}_{ijk}^2}$	tensor F-norm
${\mathbf{T}}^{\left(k\right)}$	$\mathcal{T}(:,:,k)$	${\parallel \mathcal{T}\parallel }_{\infty }:={max}_{ijk}\left|{\mathcal{T}}_{ijk}\right|$	tensor $l_{\infty }$-norm
${\mathbf{T}}_{\left(k\right)}$	mode-k unfolding of $\mathcal{T}$	$⟨\mathcal{A},\mathcal{B}⟩:={\sum }_{ijk}{\mathcal{A}}_{ijk}{\mathcal{B}}_{ijk}$	tensor inner product

2.2. Tensor ${\ast }_L$-Singular Value Decomposition

The tensor ${\ast }_L$-SVD is a generalization of the t-SVD [22]. To get a better understanding of ${\ast }_L$-SVD, we first introduce several basic notions of t-SVD as follows. For any tensor $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, its block circulant matrix $\mathtt{bcirc}\left(\mathcal{T}\right)$ is defined as

$$\begin{array}{c}\hfill \mathtt{bcirc}\left(\mathcal{T}\right):=\left[\begin{array}{cccc}{\mathcal{T}}^{\left(1\right)}& {\mathcal{T}}^{\left(d_3\right)}& \cdots & {\mathcal{T}}^{\left(2\right)}\\ {\mathcal{T}}^{\left(2\right)}& {\mathcal{T}}^{\left(1\right)}& \cdots & {\mathcal{T}}^{\left(3\right)}\\ ⋮& \ddots & \ddots & ⋮\\ {\mathcal{T}}^{\left(d_3\right)}& {\mathcal{T}}^{(d_3-1)}& \cdots & {\mathcal{T}}^{\left(1\right)}\end{array}\right]\end{array}$$

We also define the block vectorization operator and its inverse operator for any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ by:

$$\begin{array}{c}\hfill \mathtt{bvec}\left(\mathcal{T}\right):=\left[\begin{array}{c}{\mathcal{T}}^{\left(1\right)}\\ {\mathcal{T}}^{\left(2\right)}\\ ⋮\\ {\mathcal{T}}^{\left(d_3\right)}\end{array}\right],\mathtt{bvfold}\left(\mathtt{bvec}\left(\mathcal{T}\right)\right)=\mathcal{T}\end{array}$$

Then, based on the operators defined above, we are able to give the definition of the tensor t-product.

Definition 1

(T-product [22]). For any tensors $\mathcal{A}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ and $\mathcal{B}\in {\mathbb{R}}^{d_2\times d_4\times d_3}$, their t-product is a tensor $\mathcal{C}$ of size $d_1\times d_4\times d_3$ computed as follows:

$$\begin{array}{c}\hfill \mathcal{C}=\mathcal{A}\ast \mathcal{B}:=\mathtt{bvfold}\left(\mathtt{bcirc}\left(\mathcal{A}\right)\mathtt{bvec}\left(\mathcal{B}\right)\right)\end{array}$$

If we view the 3-way tensor $\mathcal{C}\in {\mathbb{R}}^{d_1\times d_4\times d_3}$ as a $d_1$-by-$d_4$ “matrix” $\mathcal{C}$ of tubes $\mathcal{C}(i,j,:)\in {\mathbb{R}}^{d_3}$, then the t-product can be analogously conducted like the matrix multiplication by changing scalar multiplication by the circular convolution between the tubes (i.e., vectors), as follows:

$$\begin{array}{c}\hfill \mathcal{C}(i,j,:)=\sum _{k=1}^{d_2}\mathcal{A}(i,k,:)⊛\mathcal{B}(k,j,:)\end{array}$$

(1)

where the symbol ⊛ denotes the circular convolution of two tubes $\mathbf{a},\mathbf{b}\in {\mathbb{R}}^{d_3}$ defined as follows [22]:

$$\begin{array}{c}\hfill {(\mathbf{a}⊛\mathbf{b})}_j=\sum _{k=1}^{d_3}{\mathbf{a}}_k{\mathbf{b}}_{1+(j-k)\mathtt{mod}{d}_3}\end{array}$$

where $\mathtt{mod}(·)$ is the modulus operator. According to the well-known relationship between circular convolution and DFT, the t-product is equivalent to matrix multiplication between all the frontal slices in the Fourier domain [22], i.e., 

$$\begin{array}{c}\hfill \overline{\mathcal{C}}=\overline{\mathcal{A}}\odot \overline{\mathcal{B}}\end{array}$$

(2)

where $\overline{\mathcal{T}}$ denotes the tensor obtained by conducting DFT on all the mode-3 fibers of any tensor $\mathcal{T}$, i.e., 

$$\begin{array}{c}\hfill \overline{\mathcal{T}}=\mathcal{T}{\times }_3{\mathbf{F}}_{d_3}\end{array}$$

(3)

where ${\mathbf{F}}_{d_3}$ is the transform matrix of DFT [22], and ${\times }_3$ denotes the tensor mode-3 product [30].

In [24], Kernfeld et al. extended the t-product to the tensor ${\ast }_L$-product by replacing DFT by any invertible linear transform $L(·)$ induced by a non-singular transformation matrix $\mathbf{L}$, and established the framework of ${\ast }_L$-SVD. In the latest studies, the  transformation matrix $\mathbf{L}$ defining the transform L is restricted to be orthogonal [18,26,31,32] (unitary in [25,27]) for better properties, which is also followed in this paper.

Given any orthogonal matrix  $\mathbf{L}\in {\mathbb{R}}^{d_3\times d_3}$ (though we restrict $\mathbf{L}$ to be orthogonal for simplicy, our analysis still holds with simple extensions for unitary $\mathbf{L}$ [27]), define the associated linear transform $L(·)$ with inverse $L^{-1}(·)$ on any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ as

$$\begin{array}{c}\hfill \overline{\mathcal{T}}=L\left(\mathcal{T}\right):=\mathcal{T}{\times }_3\mathbf{L},and{L}^{-1}\left(\mathcal{T}\right):=\mathcal{T}{\times }_3{\mathbf{L}}^{-1}\end{array}$$

(4)

Definition 2

(Tensor ${\ast }_L$-product [24]). The ${\ast }_L$–product of any $\mathcal{A}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ and $\mathcal{B}\in {\mathbb{R}}^{d_2\times d_4\times d_3}$ under the invertible linear transform L in Equation (4), denoted by $\mathcal{A}{\ast }_L\mathcal{B}$, is defined as the tensor $\mathcal{C}\in {\mathbb{R}}^{d_1\times d_4\times d_3}$ such that $L\left(\mathcal{C}\right)=L\left(\mathcal{A}\right)\odot L\left(\mathcal{B}\right)$.

Definition 3

(${\ast }_L$–block-diagonal matrix [18]). For any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, its ${\ast }_L$–block-diagonal matrix, denoted by $\overline{\mathbf{T}}$, is defined as the block diagonal matrix whose i-th diagonal block is the i-th frontal slice ${\overline{\mathbf{T}}}^{\left(i\right)}$ of $\overline{\mathcal{T}}=L\left(\mathcal{T}\right)$, i.e.,

$$\begin{array}{c}\hfill \overline{\mathbf{T}}:=\mathtt{bdiag}\left(\overline{\mathcal{T}}\right):=\left[\begin{array}{ccc}{\overline{\mathbf{T}}}^{\left(1\right)}& & \\ & \ddots & \\ & & {\overline{\mathbf{T}}}^{\left(d_3\right)}\end{array}\right]\in {\mathbb{R}}^{d_1{d}_3\times d_2{d}_3}\end{array}$$

Based on the notions of tensor ${\ast }_L$–transpose, ${\ast }_L$-identity tensor, ${\ast }_L$-orthogonal tensor, and f-diagonal tensor [24], the ${\ast }_L$–SVD (illustrated in Figure 1) is given.

Figure 1. An illustration of ${\ast }_L$–SVD [18].

Theorem 1

(Tensor ${\ast }_L$–SVD, ${\ast }_L$-tubal rank [24]). Any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ has a tensor ${\ast }_L$–Singular Value Decomposition (${\ast }_L$–SVD) under any L in Equation (4), given as follows

$$\mathcal{T}=\mathcal{U}{\ast }_L\mathcal{D}{\ast }_L{\mathcal{V}}^{\top }$$

(5)

where $\mathcal{U}\in {\mathbb{R}}^{d_1\times d_1\times d_3}$, $\mathcal{V}\in {\mathbb{R}}^{d_2\times d_2\times d_3}$ are ${\ast }_L$-orthogonal, and $\mathcal{D}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ is f-diagonal.

The ${\ast }_L$-tubal rank of $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ is defined as the number of non-zero tubes of $\mathcal{D}$ in its ${\ast }_L$–SVD in Equation (5) i.e.,

$$\begin{array}{c}\hfill r_{tb}\left(\mathcal{T}\right):=\#\left\{i\right|\mathcal{D}(i,i,:)\ne \mathbf{0},i\in [d_1\wedge d_2]\}\end{array}$$

where # counts the number of elements of a given set.

For any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, we have the following equivalence between its ${\ast }_L$-SVD and the matrix SVD of its ${\ast }_L$–block-diagonal matrix $\overline{\mathbf{T}}$:

$$\begin{array}{c}\hfill \mathcal{T}=\mathcal{U}{\ast }_L\mathcal{D}{\ast }_L\mathcal{V}{}^{\top }\iff \overline{\mathbf{T}}=\overline{\mathbf{U}}·\overline{\mathbf{D}}·\overline{\mathbf{V}}{}^{\top }.\end{array}$$

Considering the block diagonal structure of $\overline{\mathbf{T}}$, we define the tensor ${\ast }_L$-multi-rank on its diagonal blocks ${\overline{\mathbf{T}}}^{\left(i\right)}$:

Definition 4

(Tensor ${\ast }_L$–nuclear norm, tensor ${\ast }_L$-spectral norm [26]). The tensor ${\ast }_L$–nuclear norm (${\ast }_L$-TNN) and ${\ast }_L$-spectral norm of any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ under any L in Equation (4) are defined as the matrix nuclear norm and matrix spectral norm of $\overline{\mathbf{T}}$, respectively, i.e.,

$$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{\star }:=\parallel \overline{\mathbf{T}}{\parallel }_{\ast }{,\parallel \mathcal{T}\parallel }_{sp}:=\parallel \overline{\mathbf{T}}\parallel .\end{array}$$

As proved in [26,27], ${\ast }_L$–TNN is the convex envelop of the $l_1$-norm of the ${\ast }_L$–multi-rank in unit tensor ${\ast }_L$-spectral norm ball. Thus, ${\ast }_L$–TNN encourages a low ${\ast }_L$–multi-rank structure which means low-rankness in spectral domain. When the linear transform L represents the DFT (although we restrict the $\mathbf{L}$ in Equation (4) to be orthogonal, we still consider TNN as a special case of ${\ast }_L$–TNN up to constants and real/complex domain) along the 3-rd mode, ${\ast }_L$–TNN and tensor ${\ast }_L$-spectral norm degenerate to the Tubal Nuclear Norm (TNN) and the tensor spectral norm, respectively, up to a constant factor $d_3^{-1}$ [26,33].

3. Robust Tensor Completion

In this section, we will formulate the robust tensor completion problem. The  observation model will be shown first.

3.1. The Observation Model

Consider an underling signal tensor ${\mathcal{L}}^{\ast }\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ which possesses intrinsically low-dimensionality structure characterized by low-tubal-rankness, that is $r_{tb}\left({\mathcal{L}}^{\ast }\right)\ll d_1\wedge d_2$. Suppose we obtain N scalar observations $y_i$ of ${\mathcal{L}}^{\ast }\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ from the noisy observation model:

$$\begin{array}{c}\hfill y_i=⟨{\mathcal{L}}^{\ast }+{\mathcal{S}}^{\ast },{\mathcal{X}}_i⟩+{\xi }_i,\forall i\in \left[N\right],\end{array}$$

(6)

where the tensor ${\mathcal{S}}^{\ast }\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ represents some gross corruptions (e.g., outliers, errors, etc.) additive to the signal ${\mathcal{L}}^{\ast }$ which is element-wisely sparse (the presented theoretical analysis and optimization algorithm can be generalized to more sparsity settings of corruptions (e.g. the tube-wise sparsity [20,34], and slice-wise sparsity [34,35]) by using the tools developed for robust matrix completion in [36] and robust tensor decomposition in [34]; for simplicity, we only consider the most common element-wisely sparse case), ${\xi }_i$’s are random noises sampled i.i.d. from Gaussian distribution $\mathcal{N}(0,{\sigma }^2)$, and ${\mathcal{X}}_i$’s are known random design tensors in ${\mathbb{R}}^{d_1\times d_2\times d_3}$ satisfying the following assumptions:

Assumption A1.

We make two natural assumptions on the design tensors:

I.

All the corrupted positions of ${\mathcal{L}}^{\ast }$ are observed, that is, the (unknown) support ${\Theta }_s=\mathtt{supp}\left({\mathcal{S}}^{\ast }\right):=\left\{(i,j,k)\right|{\mathcal{S}}_{ijk}^{\ast }\ne 0\}$ of the corruption tensor ${\mathcal{S}}^{\ast }$ is fully observed. Formally speaking, there exists an unknown subset  ${\mathsf{X}}_s\subset {\left\{{\mathcal{X}}_i\right\}}_{i=1}^N$ drawn from an (unknown) distribution ${\Pi }_{{\Theta }_s}$ on the set ${\mathsf{X}}_{{\Theta }_s}:=\left\{{\mathbf{e}}_j\circ {\mathbf{e}}_k\circ {\mathbf{e}}_l,\forall (j,k,l)\in {\Theta }_s\right\}$, such that each element in ${\mathsf{X}}_{{\Theta }_s}$ is sampled at least once.

II.

All uncorrupted positions of ${\mathcal{L}}^{\ast }$ are sampled uniformly with replacement for simplicity of exposition. Formally speaking, each element of the set ${\mathsf{X}}_s^{\perp }:={\left\{{\mathcal{X}}_i\right\}}_{i=1}^N\setminus {\mathsf{X}}_s$ is sampled i.i.d. from an uniform distribution ${\Pi }_{{\Theta }_s^{\perp }}$ on the set ${\mathsf{X}}_{{\Theta }_s^{\perp }}:=\left\{{\mathbf{e}}_j\circ {\mathbf{e}}_k\circ {\mathbf{e}}_l,\forall (j,k,l)\in {\Theta }_s^{\perp }\right\}$.

According to the observation model (6), the true tensor ${\mathcal{L}}^{\ast }$ is first corrupted by a sparse tensor ${\mathcal{S}}^{\ast }$ and then sampled to N scalars $\left\{y_i\right\}$ with additive Gaussian noises $\left\{{\xi }_i\right\}$ (see Figure 2). The  corrupted positions of ${\mathcal{L}}^{\ast }$ are further assumed in Assumption A1 to be totally observed with design tensors in ${\mathsf{X}}_s\subset {\left\{{\mathcal{X}}_i\right\}}_{i=1}^N$, and the remaining uncorrupted positions are sampled uniformly through design tensors in ${\mathsf{X}}_s^{\perp }={\left\{{\mathcal{X}}_i\right\}}_{i=1}^N\setminus {\mathsf{X}}_s$.

Figure 2. An illustration of the robust tensor completion problem.

Let $\mathbf{y}=(y_1,\dots ,y_N){}^{\top }\in {\mathbb{R}}^N$ and $\xi =({\xi }_1,\dots ,{\xi }_N){}^{\top }\in {\mathbb{R}}^N$ be the vector of observations and noises, respectively. Define the design operator $\mathfrak{X}:{\mathbb{R}}^{d_1\times d_2\times d_3}\to {\mathbb{R}}^N$ as $\mathfrak{X}(·):=\left(⟨·,{\mathcal{X}}_1⟩,\dots ,⟨·,{\mathcal{X}}_N⟩\right){}^{\top },$ and its adjoint operator ${\mathfrak{X}}^{\ast }\left(\mathbf{z}\right):={\sum }_{i=1}^N{z}_i{\mathcal{X}}_i$ for any $\mathbf{z}\in {\mathbb{R}}^N$. Then the observation model (6) can be rewritten in a compact form

$$\begin{array}{c}\hfill \mathbf{y}=\mathfrak{X}({\mathcal{L}}^{\ast }+{\mathcal{S}}^{\ast })+\xi .\end{array}$$

3.2. The Proposed Estimator

The aim of robust tensor completion is to reconstruct the unknown low-rank ${\mathcal{L}}^{\ast }$ and sparse ${\mathcal{S}}^{\ast }$ from incomplete and noisy measurements ${\left\{({\mathcal{X}}_i,y_i)\right\}}_{i=1}^N$ generated by the observation model (6). It can be treated as a robust extension of tensor completion in [33], and a noisy partial variant of tensor robust PCA [37].

To reconstruct the underlying low-rank tensor ${\mathcal{L}}^{\ast }$ and sparse tensor ${\mathcal{S}}^{\ast }$, it is natural to consider the following minimization model:

$$\begin{array}{c}\hfill \underset{\mathcal{L},\mathcal{S}}{min}\frac{1}{2N}{\parallel \mathbf{y}-\mathfrak{X}(\mathcal{L}+\mathcal{S})\parallel }_2^2+{\lambda }_{\iota }{r}_{tb}\left(\mathcal{L}\right)+{\lambda }_s{\parallel \mathcal{S}\parallel }_1,\end{array}$$

(7)

where we use least squares as the fidelity term for Gaussian noises, the tubal rank as the regularization to impose low-rank structure in $\mathcal{L}$, the tensor $l_0$-(pseudo)norm to regularize $\mathcal{S}$ for sparsity, ${\lambda }_{\iota },{\lambda }_s\ge 0$ are tunable regularization parameters, balancing the regularizations and the fidelity term.

However, general rank and $l_0$-norm minimization is NP-hard [12,38], making it extremely hard to soundly solve Problem (7). For tractable low-rank and sparse optimization, we follow the most common idea to relax the non-convex functions $r_{tb}(·)$ and ${\parallel ·\parallel }_0$ to their convex surrogates, i.e.,  the ${\ast }_L$–tubal nuclear norm ${\parallel ·\parallel }_{\star }$ and the tensor $l_1$-norm ${\parallel ·\parallel }_1$, respectively. Specifically, the following estimator is defined:

$$\begin{array}{c}\hfill (\widehat{\mathcal{L}},\widehat{\mathcal{S}}):=\underset{{\parallel \mathcal{L}\parallel }_{\infty }\le a,{\parallel \mathcal{S}\parallel }_{\infty }\le a}{error}\frac{1}{2N}\parallel \mathbf{y}-\mathfrak{X}(\mathcal{L}+\mathcal{S}){\parallel }_2^2+{\lambda }_{\iota }{\parallel \mathcal{L}\parallel }_{\star }+{\lambda }_s{\parallel \mathcal{S}\parallel }_1,\end{array}$$

(8)

where $a>0$ is a known constant constraining the magnitude of entries in ${\mathcal{L}}^{\ast }$ and ${\mathcal{S}}^{\ast }$. The additional constraint ${\parallel \mathcal{L}\parallel }_{\infty }\le a$ and ${\parallel \mathcal{S}\parallel }_{\infty }\le a$ is very mild since most signals and corruptions are of limited energy in real applications. It can also provide a theoretical benefit to exclude the “spiky” tensors, which is important in controlling the separability of ${\mathcal{L}}^{\ast }$ and ${\mathcal{S}}^{\ast }$. Such “non-spiky” constraints are also imposed in previous literatures [36,39,40], playing a key role in bounding the estimation error.

Then, it is natural to ask the following questions:

Q1:

How to compute the proposed estimator?

Q2:

How well can the proposed estimator estimate ${\mathcal{L}}^{\ast }$ and ${\mathcal{S}}^{\ast }$?

We first discuss Q1 in Section 4 and then answer Q2 in Section 5.

4. Algorithm

In this section, we answer Q1 by designing an algorithm based on ADMM to compute the proposed estimator.

To solve Problem (8), the first step is to introduce auxiliary variables $\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N}$ to deal with the complex couplings between $\mathfrak{X}(·)$, ${\parallel ·\parallel }_2$${\parallel ·\parallel }_{\star }$, ${\parallel ·\parallel }_1$, and ${\parallel ·\parallel }_{\infty }$ as follows:

$$\begin{array}{cc}\hfill \underset{\mathbf{g},\mathcal{L},\mathcal{S},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N}}{min}& \frac{1}{2N}{\parallel \mathbf{g}\parallel }_2^2+{\lambda }_{\iota }{\parallel \mathcal{K}\parallel }_{\star }+{\lambda }_s{\parallel \mathcal{T}\parallel }_1+{\delta }_a^{\infty }\left(\mathcal{M}\right)+{\delta }_a^{\infty }\left(\mathcal{N}\right),\hfill \\ \hfill s.t.& \mathbf{g}=\mathbf{y}-\mathfrak{X}(\mathcal{L}+\mathcal{S}),\mathcal{L}=\mathcal{K}=\mathcal{M},\mathcal{S}=\mathcal{T}=\mathcal{N}\hfill \end{array}$$

(9)

where ${\delta }_a^{\infty }(·)$ is the indicator function of tensor $l_{\infty }$-norm ball defined as follows

$$\begin{array}{c}\hfill {\delta }_a^{\infty }\left(\mathcal{M}\right)=\left\{\begin{array}{cc}& {0\parallel \mathcal{M}\parallel }_{\infty }\le a\hfill \\ & +{\infty \parallel \mathcal{M}\parallel }_{\infty }>a\hfill \end{array}\right.\end{array}$$

We then give the augmented Lagrangian of Equation (9) with Lagrangian multipliers $\mathbf{z}$ and ${\left\{{\mathbf{Z}}_i\right\}}_{i=1}^4$ and penalty parameter $\rho >0$:

$$\begin{array}{cc}\hfill & L_{\rho }(\mathcal{L},\mathcal{S},\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},\mathbf{z},{\mathcal{Z}}_1,{\mathcal{Z}}_2,{\mathcal{Z}}_3,{\mathcal{Z}}_4)\hfill \\ & =\frac{1}{2N}{\parallel \mathbf{g}\parallel }_2^2+{\lambda }_{\iota }{\parallel \mathcal{K}\parallel }_{\star }+{\lambda }_s{\parallel \mathcal{T}\parallel }_1+{\delta }_a^{\infty }\left(\mathcal{M}\right)+{\delta }_a^{\infty }\left(\mathcal{N}\right)\hfill \\ & +⟨\mathbf{z},\mathbf{g}+\mathfrak{X}(\mathcal{L}+\mathcal{S})-\mathbf{y}⟩+\frac{\rho }{2}{\parallel \mathbf{g}+\mathfrak{X}(\mathcal{L}+\mathcal{S})-\mathbf{y}\parallel }_2^2\hfill \\ & +⟨{\mathcal{Z}}_1,\mathcal{L}-\mathcal{K}⟩+\frac{\rho }{2}{\parallel \mathcal{L}-\mathcal{K}\parallel }_{\mathrm{F}}^2+⟨{\mathcal{Z}}_2,\mathcal{L}-\mathcal{M}⟩+\frac{\rho }{2}{\parallel \mathcal{L}-\mathcal{M}\parallel }_{\mathrm{F}}^2\hfill \\ & +⟨{\mathcal{Z}}_3,\mathcal{S}-\mathcal{T}⟩+\frac{\rho }{2}{\parallel \mathcal{S}-\mathcal{T}\parallel }_{\mathrm{F}}^2+⟨{\mathcal{Z}}_4,\mathcal{S}-\mathcal{N}⟩+\frac{\rho }{2}{\parallel \mathcal{S}-\mathcal{N}\parallel }_{\mathrm{F}}^2\hfill \end{array}$$

(10)

Following the framework of standard two-block ADMM [41], we separate the primal variables into two blocks $(\mathcal{L},\mathcal{S})$ and $(\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N})$, and update them alternatively as follows:

Update the first block $(\mathcal{L},\mathcal{S})$: After the t-th iteration, we first update $(\mathcal{L},\mathcal{S})$ by keeping the other variables fixed as follows:

$$\begin{array}{cc}& ({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1})\hfill \\ & =\underset{\mathcal{L},\mathcal{S}}{error}{L}_{\rho }(\mathcal{L},\mathcal{S},{\mathbf{g}}^t,{\mathcal{K}}^t,{\mathcal{T}}^t,{\mathcal{M}}^t,{\mathcal{N}}^t,{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)\hfill \\ & =\underset{\mathcal{L},\mathcal{S}}{error}⟨{\mathbf{z}}^t,{\mathbf{g}}^t+\mathfrak{X}(\mathcal{L}+\mathcal{S})-\mathbf{y}⟩+\frac{\rho }{2}{\parallel {\mathbf{g}}^t+\mathfrak{X}(\mathcal{L}+\mathcal{S})-\mathbf{y}\parallel }_2^2\hfill \\ & +⟨{\mathcal{Z}}_1^t,\mathcal{L}-{\mathcal{K}}^t⟩+\frac{\rho }{2}\parallel \mathcal{L}-{\mathcal{K}}^t{\parallel }_{\mathrm{F}}^2+⟨{\mathcal{Z}}_2^t,\mathcal{L}-{\mathcal{M}}^t⟩+\frac{\rho }{2}{\parallel \mathcal{L}-{\mathcal{M}}^t\parallel }_{\mathrm{F}}^2\hfill \\ & +⟨{\mathcal{Z}}_3^t,\mathcal{S}-{\mathcal{T}}^t⟩+\frac{\rho }{2}\parallel \mathcal{S}-{\mathcal{T}}^t{\parallel }_{\mathrm{F}}^2+⟨{\mathcal{Z}}_4^t,\mathcal{S}-{\mathcal{N}}^t⟩+\frac{\rho }{2}{\parallel \mathcal{S}-{\mathcal{N}}^t\parallel }_{\mathrm{F}}^2\hfill \end{array}$$

(11)

By taking derivatives, respectively, to $\mathcal{L}$ and $\mathcal{S}$ and setting them to zero, we obtain the following system of equations:

$$\begin{array}{cc}\hfill {\mathfrak{X}}^{\ast }{\mathbf{z}}^t+\rho {\mathfrak{X}}^{\ast }\left(\mathfrak{X}(\mathcal{L}+\mathcal{S})+{\mathbf{g}}^t-\mathbf{y}\right)+{\mathcal{Z}}_1^t+\rho (\mathcal{L}-{\mathcal{K}}^t)+{\mathcal{Z}}_2^t+\rho (\mathcal{L}-{\mathcal{M}}^t)& =\mathbf{0}\hfill \\ \hfill {\mathfrak{X}}^{\ast }{\mathbf{z}}^t+\rho {\mathfrak{X}}^{\ast }\left(\mathfrak{X}(\mathcal{L}+\mathcal{S})+{\mathbf{g}}^t-\mathbf{y}\right)+{\mathcal{Z}}_3^t+\rho (\mathcal{S}-{\mathcal{T}}^t)+{\mathcal{Z}}_4^t+\rho (\mathcal{S}-{\mathcal{N}}^t)& =\mathbf{0}\hfill \end{array}$$

(12)

Through solving the system of equations in Equation (12), we obtain

$$\begin{array}{cc}\hfill {\mathcal{L}}^{t+1}& =\frac{1}{4\rho }\left({\mathfrak{X}}^{\ast }\mathfrak{X}{({\mathfrak{X}}^{\ast }\mathfrak{X}+\mathbb{I})}^{-1}(2\mathcal{A}+{\mathcal{B}}_{\iota }+{\mathcal{B}}_s)-2(\mathcal{A}+{\mathcal{B}}_{\iota })\right)\hfill \\ \hfill {\mathcal{S}}^{t+1}& =\frac{1}{4\rho }\left({\mathfrak{X}}^{\ast }\mathfrak{X}{({\mathfrak{X}}^{\ast }\mathfrak{X}+\mathbb{I})}^{-1}(2\mathcal{A}+{\mathcal{B}}_{\iota }+{\mathcal{B}}_s)-2(\mathcal{A}+{\mathcal{B}}_s)\right)\hfill \end{array}$$

(13)

where $\mathbb{I}$ denotes the identity operator, and the intermediate tensors are given by $\mathcal{A}={\mathfrak{X}}^{\ast }({\mathbf{z}}^t+\rho {\mathbf{g}}^t-\mathbf{y})$, ${\mathcal{B}}_{\iota }={\mathcal{Z}}_1^t+{\mathcal{Z}}_2^t-\rho ({\mathcal{K}}^t+{\mathcal{M}}^t)$, and ${\mathcal{B}}_s={\mathcal{Z}}_3^t+{\mathcal{Z}}_4^t-\rho ({\mathcal{T}}^t+{\mathcal{N}}^t)$.  

Update the second block $(\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N})$: According to the special form of the Lagrangian in Equation (10), the variables $\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N}$ in the second block can be updated separately as follows.

We first update $\mathbf{g}$ with fixed $(\mathcal{L},\mathcal{S})$:

$$\begin{array}{cc}\hfill {\mathbf{g}}^{t+1}& =\underset{\mathbf{g}}{error}{L}_{\rho }({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1},\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)\hfill \\ & =\underset{\mathbf{g}}{error}\frac{1}{2N}{\parallel \mathbf{g}\parallel }_2^2+\frac{\rho }{2}{\parallel \mathbf{g}+\mathfrak{X}({\mathcal{L}}^{t+1}+{\mathcal{S}}^{t+1})-\mathbf{y}+{\rho }^{-1}{\mathbf{z}}^t\parallel }_2^2\hfill \\ & =\frac{N\rho }{1+N\rho }\left(\mathbf{y}-\mathfrak{X}({\mathcal{L}}^{t+1}+{\mathcal{S}}^{t+1})-{\rho }^{-1}{\mathbf{z}}^t\right)\hfill \end{array}$$

(14)

We then update $\mathcal{K}$ with fixed $(\mathcal{L},\mathcal{S})$:

$$\begin{array}{cc}\hfill {\mathcal{K}}^{t+1}& =\underset{\mathcal{K}}{error}{L}_{\rho }({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1},\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)\hfill \\ & =\underset{\mathcal{K}}{error}{\lambda }_{\iota }{\parallel \mathcal{K}\parallel }_{\star }+⟨{\mathcal{Z}}_1^t,{\mathcal{L}}^{t+1}-\mathcal{K}⟩+\frac{\rho }{2}{\parallel {\mathcal{L}}^{t+1}-\mathcal{K}\parallel }_{\mathrm{F}}^2\hfill \\ & ={\mathtt{Prox}}_{{\lambda }_{\iota }{\rho }^{-1}}^{{\parallel ·\parallel }_{\star }}({\mathcal{L}}^{t+1}+{\rho }^{-1}{\mathcal{Z}}_1^t),\hfill \end{array}$$

(15)

where ${\mathtt{Prox}}_{{\rho }^{-1}}^{{\parallel ·\parallel }_{\star }}(·)$ is the proximality operator of ${\ast }_L$–TNN given in the following lemma.

Lemma 1

(A modified version of Theorem 3.2 in [26]). Let ${\mathcal{L}}_0\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ be any tensor with ${\ast }_L$–SVD ${\mathcal{L}}_0=\mathcal{U}{\ast }_L\mathcal{D}{\ast }_L\mathcal{V}{}^{\top }$. Then the proximality operator of ${\ast }_L$–TNN at ${\mathcal{L}}_0$ with constant $\tau >0$, defined as ${\mathtt{Prox}}_{\tau }^{{\parallel ·\parallel }_{\star }}\left({\mathcal{L}}_0\right):={error}_{\mathcal{L}}{\tau \parallel \mathcal{L}\parallel }_{\star }+\frac{1}{2}{\parallel \mathcal{L}-{\mathcal{L}}_0\parallel }_{\mathrm{F}}$, can be computed by

$$\begin{array}{c}\hfill {\mathtt{Prox}}_{\tau }^{{\parallel ·\parallel }_{\star }}\left({\mathcal{L}}_0\right)=\mathcal{U}{\ast }_L{\mathcal{D}}_{\tau }{\ast }_L\mathcal{V}{}^{\top }\end{array}$$

(16)

where

$$\begin{array}{c}\hfill {\mathcal{D}}_{\tau }=L^{-1}{(L\left(\mathcal{D}\right)-\tau )}_{+}).\end{array}$$

(17)

where $t_{+}$ denotes the positive part of t, i.e., $t_{+}=max(t,0)$.

We update $\mathcal{T}$ with fixed $(\mathcal{L},\mathcal{S})$:

$$\begin{array}{cc}\hfill {\mathcal{T}}^{t+1}& =\underset{\mathcal{T}}{error}{L}_{\rho }({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1},\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)\hfill \\ & =\underset{\mathcal{T}}{error}{\lambda }_s{\parallel \mathcal{T}\parallel }_1+⟨{\mathcal{Z}}_3^t,{\mathcal{S}}^{t+1}-\mathcal{T}⟩+\frac{\rho }{2}{\parallel {\mathcal{S}}^{t+1}-\mathcal{T}\parallel }_{\mathrm{F}}^2\hfill \\ & ={\mathtt{Prox}}_{{\lambda }_s{\rho }^{-1}}^{{\parallel ·\parallel }_1}({\mathcal{S}}^{t+1}+{\rho }^{-1}{\mathcal{Z}}_3^t),\hfill \end{array}$$

(18)

where ${\mathtt{Prox}}_{\tau }^{{\parallel ·\parallel }_1}\left(\mathcal{T}\right)$ is the proximality operator [19] of the tensor $l_1$-norm at point $\mathcal{T}$ given as ${\mathtt{Prox}}_{\tau }^{{\parallel ·\parallel }_1}\left(\mathcal{T}\right)=\mathtt{sign}\left(\mathcal{T}\right)\odot {\left(\right|\mathcal{T}|-\tau )}_{+}$, where ⊙ denotes the element-wise product.

We then update $\mathcal{M}$ with fixed $(\mathcal{L},\mathcal{S})$:

$$\begin{array}{cc}\hfill {\mathcal{M}}^{t+1}& =\underset{\mathcal{M}}{error}{L}_{\rho }({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1},\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)\hfill \\ & =\underset{\mathcal{M}}{error}{\delta }_a^{\infty }\left(\mathcal{M}\right)+⟨{\mathcal{Z}}_2^t,{\mathcal{L}}^{t+1}-\mathcal{M}⟩+\frac{\rho }{2}{\parallel {\mathcal{L}}^{t+1}-\mathcal{M}\parallel }_{\mathrm{F}}^2\hfill \\ & ={\mathtt{Proj}}_a^{{\parallel ·\parallel }_{\infty }}({\mathcal{L}}^{t+1}+{\rho }^{-1}{\mathcal{Z}}_2^t),\hfill \end{array}$$

(19)

where ${\mathtt{Proj}}_a^{{\parallel ·\parallel }_{\infty }}(·)$ is the projector onto the tensor $l_{\infty }$-norm ball of radius a, which is given by ${\mathtt{Proj}}_a^{{\parallel ·\parallel }_{\infty }}\left(\mathcal{M}\right)=\mathtt{sign}\left(\mathcal{M}\right)\odot min\left(\right|\mathcal{M}|,a)$ [19].

Similarly, we update $\mathcal{N}$ as follows:

$$\begin{array}{cc}\hfill {\mathcal{N}}^{t+1}& =\underset{\mathcal{N}}{error}{L}_{\rho }({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1},\mathbf{g},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)\hfill \\ & =\underset{\mathcal{N}}{error}{\delta }_a^{\infty }\left(\mathcal{N}\right)+⟨{\mathcal{Z}}_4^t,{\mathcal{S}}^{t+1}-\mathcal{N}⟩+\frac{\rho }{2}{\parallel {\mathcal{S}}^{t+1}-\mathcal{N}\parallel }_{\mathrm{F}}^2\hfill \\ & ={\mathtt{Proj}}_a^{{\parallel ·\parallel }_{\infty }}({\mathcal{S}}^{t+1}+{\rho }^{-1}{\mathcal{Z}}_4^t).\hfill \end{array}$$

(20)

Update the dual variables $\mathbf{z}$ and $\left\{{\mathcal{Z}}_i\right\}$: According to the update strategy of dual variables in ADMM [41], the variables $\mathbf{z}$ and $\left\{{\mathcal{Z}}_i\right\}$ can be updated using dual ascent as follows:

$$\begin{array}{cc}\hfill {\mathbf{z}}^{t+1}& ={\mathbf{z}}^t+\rho ({\mathbf{g}}^{t+1}+\mathfrak{X}({\mathcal{L}}^{t+1}+{\mathcal{S}}^{t+1})-\mathbf{y})\hfill \\ \hfill {\mathcal{Z}}_1^{t+1}& ={\mathcal{Z}}_1^{t+1}+\rho ({\mathcal{L}}^{t+1}-{\mathcal{K}}^{t+1})\hfill \\ \hfill {\mathcal{Z}}_2^{t+1}& ={\mathcal{Z}}_2^{t+1}+\rho ({\mathcal{L}}^{t+1}-{\mathcal{M}}^{t+1})\hfill \\ \hfill {\mathcal{Z}}_3^{t+1}& ={\mathcal{Z}}_3^{t+1}+\rho ({\mathcal{S}}^{t+1}-{\mathcal{T}}^{t+1})\hfill \\ \hfill {\mathcal{Z}}_4^{t+1}& ={\mathcal{Z}}_4^{t+1}+\rho ({\mathcal{S}}^{t+1}-{\mathcal{N}}^{t+1})\hfill \end{array}$$

(21)

The algorithm for solving Problem (8) is summarized in Algorithm 1.

Algorithm 1 Solving Problem (8) using ADMM.

Input:  The design tensors $\left\{{\mathcal{X}}_i\right\}$ and observations $\left\{y_i\right\}$, the regularization parameters ${\lambda }_{\iota },{\lambda }_s$, the $l_1$-norm bound a, the penalty parameter $\rho$ of the Lagrangian, the convergence tolerance $\delta$, the maximum iteration number $T_{max}$. 1: Initialize $t=0$, ${\mathbf{g}}^0={\mathbf{z}}^0=\mathbf{0}\in {\mathbb{R}}^N,{\mathcal{L}}^0={\mathcal{S}}^0={\mathcal{K}}^0={\mathcal{T}}^0={\mathcal{M}}^0={\mathcal{N}}^0={\mathcal{Z}}_1^0={\mathcal{Z}}_2^0={\mathcal{Z}}_3^0={\mathcal{Z}}_4^0=\mathbf{0}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ 2: for$t=0,\cdots ,T_{max}$do 3:     Update $({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1})$ by Equation (13); 4:     Update $({\mathbf{g}}^{t+1},{\mathcal{K}}^{t+1},{\mathcal{T}}^{t+1},{\mathcal{M}}^{t+1},{\mathcal{N}}^{t+1})$ by Equations (14)–(20), respectively; 5:     Update $({\mathbf{z}}^{t+1},{\mathcal{Z}}_1^{t+1},{\mathcal{Z}}_2^{t+1},{\mathcal{Z}}_3^{t+1},{\mathcal{Z}}_4^{t+1})$ by Equation (21); 6:     Check the convergence criteria: 7: (i) convergence of primal variables: $$\begin{array}{c}\hfill \parallel {\mathcal{A}}^{t+1}-{\mathcal{A}}^t{\parallel }_{\infty }\le \delta ,\forall \mathcal{A}\in \{\mathbf{g},\mathcal{L},\mathcal{S},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N}\}\end{array}$$ (ii) convergence of constraints: $$\begin{array}{cccc}\hfill max\{\parallel {\mathcal{L}}^{t+1}-{\mathcal{K}}^{t+1}{\parallel }_{\infty },\parallel {\mathcal{L}}^{t+1}-{\mathcal{M}}^{t+1}{\parallel }_{\infty }\}& \le \delta max\{\parallel {\mathcal{S}}^{t+1}-{\mathcal{T}}^{t+1}{\parallel }_{\infty },\parallel {\mathcal{S}}^{t+1}-{\mathcal{N}}^{t+1}{\parallel }_{\infty }\}\hfill & \hfill \le \delta \parallel {\mathbf{g}}^{t+1}+\mathfrak{X}({\mathcal{L}}^{t+1}+{\mathcal{S}}^{t+1})-\mathbf{y}{\parallel }_{\infty }& \le \delta \hfill \end{array}$$ 8: end for Output:  $(\widehat{\mathcal{L}},\widehat{\mathcal{S}})=({\mathcal{L}}^{t+1},{\mathcal{S}}^{t+1})$.

Complexity Analysis: The time complexity of Algorithm 1 is analyzed as follows. Due to the special structures of design tensors $\left\{{\mathcal{X}}_i\right\}$, the operators $\mathfrak{X}$ and ${({\mathfrak{X}}^{\ast }\mathfrak{X}{\mathfrak{X}}^{\ast }\mathfrak{X}+\mathbb{I})}^{-1}$ can be implemented with time cost $O\left(N\right)$ and $O(d_1{d}_2{d}_3+N)$, respectively. The cost of updating $\mathcal{L},\mathcal{S},\mathcal{T},\mathcal{M},\mathcal{N}$ and $\left\{{\mathcal{Z}}_i\right\}$ is $O\left(d_1{d}_2{d}_3\right)$. The main time cost in Algorithm 1 lies in the update of $\mathcal{K}$ which needs the ${\ast }_L$–SVD on $d_1\times d_2\times d_3$ tensors, involving the ${\ast }_L$-transform (costing $O\left(d_1{d}_2{d}_3^2\right)$ in general), and $d_3$ matrix SVDs on $d_1\times d_2$ matrices (costing $O\left(d_1{d}_2{d}_3(d_1\wedge d_2)\right)$). Thus, the one-iteration cost of Algorithm 1 is

$$\begin{array}{c}\hfill O\left(d_1{d}_2{d}_3((d_1\wedge d_2)+d_3)\right)\end{array}$$

(22)

in general, and can be reduced to $O\left(d_1{d}_2{d}_3((d_1\wedge d_2)+log{d}_3)\right)$ for some linear transforms L which have fast implementations (like DFT and DCT).

Convergence Analysis: According to [28], the convergence rate of general ADMM-based algorithms is $O(1/t)$, where t is the iteration number. The convergence analysis of Algorithm 1 is established in Theorem 2.

Theorem 2

(Convergence of Algorithm 1). For any positive constant ρ, if the unaugmented Lagrangian function $L_0(\mathbf{g},\mathcal{L},\mathcal{S},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},\mathbf{z},{\mathcal{Z}}_1,{\mathcal{Z}}_2,{\mathcal{Z}}_3,{\mathcal{Z}}_4)$ has a saddle point, then the iterations $({\mathbf{g}}^t,{\mathcal{L}}^t,{\mathcal{S}}^t,{\mathcal{K}}^t,{\mathcal{T}}^t,{\mathcal{M}}^t,{\mathcal{N}}^t,{\mathbf{z}}^t,{\mathcal{Z}}_1^t,{\mathcal{Z}}_2^t,{\mathcal{Z}}_3^t,{\mathcal{Z}}_4^t)$ in Algorithm 1 satisfy the residual convergence, objective convergence and dual variable convergence (defined in [41]) of Problem (9) as $t\to \infty$.

Proof. 

The key idea is to rewrite Problem (9) into a standard two-block ADMM problem. For notational simplicity, let

$$\begin{array}{c}\hfill f\left(\mathbf{u}\right)=0,g\left(\mathbf{v}\right)=\frac{1}{2N}{\parallel \mathbf{g}\parallel }_2^2+{\lambda }_{\iota }{\parallel \mathcal{K}\parallel }_{\star }+{\lambda }_s{\parallel \mathcal{S}\parallel }_1+{\delta }_a^{\infty }\left(\mathcal{M}\right)+{\delta }_a^{\infty }\left(\mathcal{N}\right),\end{array}$$

with $\mathbf{u},\mathbf{v},\mathbf{w},\mathbf{c}$ and $\mathbf{A}$ defined as follows

$$\begin{array}{cc}& \mathbf{u}=\left[\begin{array}{c}\mathtt{vec}\left(\mathcal{L}\right)\\ \mathtt{vec}\left(\mathcal{S}\right)\end{array}\right]\in {\mathbb{R}}^{2d_1{d}_2{d}_3},\mathbf{v}=\left[\begin{array}{c}\mathbf{g}\\ \mathtt{vec}\left(\mathcal{K}\right)\\ \mathtt{vec}\left(\mathcal{T}\right)\\ \mathtt{vec}\left(\mathcal{M}\right)\\ \mathtt{vec}\left(\mathcal{N}\right)\end{array}\right]\in {\mathbb{R}}^{N+4d_1{d}_2{d}_3},\hfill \\ & \mathbf{w}=\left[\begin{array}{c}\mathbf{z}\\ \mathtt{vec}\left({\mathcal{Z}}_1\right)\\ \mathtt{vec}\left({\mathcal{Z}}_2\right)\\ \mathtt{vec}\left({\mathcal{Z}}_3\right)\\ \mathtt{vec}\left({\mathcal{Z}}_4\right)\end{array}\right]\in {\mathbb{R}}^{N+4d_1{d}_2{d}_3},\mathbf{c}=\left[\begin{array}{c}-\mathbf{y}\\ \mathbf{0}\\ \mathbf{0}\\ \mathbf{0}\\ \mathbf{0}\end{array}\right]\in {\mathbb{R}}^{N+4d_1{d}_2{d}_3},\hfill \end{array}$$

and

$$\begin{array}{c}\hfill \mathbf{A}=\left[\begin{array}{cc}-\mathbf{X}& -\mathbf{X}\\ {\mathbf{I}}_{d_1{d}_2{d}_3}& \mathbf{0}\\ {\mathbf{I}}_{d_1{d}_2{d}_3}& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{d_1{d}_2{d}_3}\\ \mathbf{0}& {\mathbf{I}}_{d_1{d}_2{d}_3}\end{array}\right]\in {\mathbb{R}}^{(N+4d_1{d}_2{d}_3)\times \left(2d_1{d}_2{d}_3\right)},with\mathbf{X}=\left[\begin{array}{c}\mathtt{vec}\left({\mathcal{X}}_1\right){}^{\top }\\ \mathtt{vec}\left({\mathcal{X}}_2\right){}^{\top }\\ ⋮\\ \mathtt{vec}\left({\mathcal{X}}_N\right){}^{\top }\end{array}\right]\in {\mathbb{R}}^{N\times \left(2d_1{d}_2{d}_3\right)},\end{array}$$

(23)

where $\mathtt{vec}(·)$ denotes the operation of tensor vectorization (see [30]).

It can be verified that $f(·)$ and $g(·)$ are closed, proper convex functions. Then, Problem (9) can be re-written as follows:

$$\begin{array}{cc}\hfill \underset{\mathbf{u},\mathbf{v}}{min}& f\left(\mathbf{u}\right)+g\left(\mathbf{v}\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathbf{A}\mathbf{u}-\mathbf{v}=\mathbf{c}.\hfill \end{array}$$

According to the convergence analysis in [41], we have:

$$\begin{array}{cccc}& \mathrm{objective}\mathrm{convergence}:\hfill & \hfill & \underset{t\to \infty }{lim}f\left({\mathbf{u}}^t\right)+g\left({\mathbf{v}}^t\right)=f^{\star }+g^{\star },\hfill \\ & \mathrm{dual}\mathrm{variable}\mathrm{convergence}:\hfill & \hfill & \underset{t\to \infty }{lim}{\mathbf{w}}^t={\mathbf{w}}^{\star },\hfill \\ & \mathrm{constraint}\mathrm{convergence}:\hfill & \hfill & \underset{t\to \infty }{lim}\mathbf{A}{\mathbf{u}}^t-{\mathbf{v}}^t=\mathbf{c},\hfill \end{array}$$

where $f^{\star },g^{\star }$ are the optimal values of $f\left(\mathbf{u}\right)$, $g\left(\mathbf{v}\right)$, respectively. Variable ${\mathbf{w}}^{\star }$ is a dual optimal point defined as:

$$\begin{array}{c}\hfill {\mathbf{w}}^{\star }=\mathbf{w}=\left[\begin{array}{c}{\mathbf{z}}^{\star }\\ \mathtt{vec}\left({\mathcal{Z}}_1^{\star }\right)\\ \mathtt{vec}\left({\mathcal{Z}}_2^{\star }\right)\\ \mathtt{vec}\left({\mathcal{Z}}_3^{\star }\right)\\ \mathtt{vec}\left({\mathcal{Z}}_4^{\star }\right)\end{array}\right]\end{array}$$

where $({\mathbf{z}}^{\star },{\mathcal{Z}}_1^{\star },{\mathcal{Z}}_2^{\star },{\mathcal{Z}}_3^{\star },{\mathcal{Z}}_4^{\star })$ is the component of dual variables in a saddle point

$({\mathbf{g}}^{\star },{\mathcal{L}}^{\star },{\mathcal{S}}^{\star },{\mathcal{K}}^{\star },{\mathcal{T}}^{\star },{\mathcal{M}}^{\star },{\mathcal{N}}^{\star },{\mathbf{z}}^{\star },{\mathcal{Z}}_1^{\star },{\mathcal{Z}}_2^{\star },{\mathcal{Z}}_3^{\star },{\mathcal{Z}}_4^{\star })$ of the unaugmented Lagrangian

$L_0(\mathbf{g},\mathcal{L},\mathcal{S},\mathcal{K},\mathcal{T},\mathcal{M},\mathcal{N},\mathbf{z},{\mathcal{Z}}_1,{\mathcal{Z}}_2,{\mathcal{Z}}_3,{\mathcal{Z}}_4)$. □

5. Statistical Performance

In this section, we answer Q2 by studying the statistical performances of the proposed estimator $(\widehat{\mathcal{L}},\widehat{\mathcal{S}})$. Specifically, the goal is to upper bound the squared F-norm error $\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}^2+{\parallel \widehat{\mathcal{S}}-{\mathcal{S}}^{\ast }\parallel }_{\mathrm{F}}^2$. We will first give an upper bound on the estimation error in a non-asymptotic manner, and then prove that the upper bound is minimax optimal up to a logarithm factor.

5.1. Upper Bound on the Estimation Error

We establish an upper bounds on the estimation error in the following theorem. For notational simplicity, let $N_s=\left|{\mathsf{X}}_s\right|$ and $N_{\iota }=\left|{\mathsf{X}}_s^{\perp }\right|$ denote the number of corrupted and uncorrupted observations of ${\mathcal{L}}^{\ast }$ in the observation model (6), respectively.

Theorem 3

(Upper bounds on the estimation error). If the number of uncorrupted observations in the observation model (6) satisfy

$$\begin{array}{c}\hfill N_{\iota }\ge c_1{d}_1{d}_{3}log(d_1{d}_3+d_2{d}_3){log}^2(d_1+d_2)\end{array}$$

(24)

and regularization parameters in Problem (8) are set by

$$\begin{array}{c}\hfill {\lambda }_{\iota }=c_2(\sigma \vee a)\sqrt{\frac{log(d_1{d}_3+d_2{d}_3)}{d_1\wedge d_2}},and{\lambda }_s=c_3(\sigma \vee a)\frac{log(d_1{d}_3+d_2{d}_3)}{N},\end{array}$$

(25)

then it holds with probability at least $1-c_5{(d_1{d}_3+d_2{d}_3)}^{-1}$ that:

$$\begin{array}{cc}\hfill & \frac{\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}^2+{\parallel \widehat{\mathcal{S}}-{\mathcal{S}}^{\ast }\parallel }_{\mathrm{F}}^2}{d_1{d}_2{d}_3}\hfill \\ & \le C\left(r_{tb}\left({\mathcal{L}}^{\ast }\right)·\frac{({\sigma }^2\vee a^2)(d_1\vee d_2)d_{3}log\tilde{d}}{N_{\iota }}+\frac{N_{s}log(d_1{d}_3+d_2{d}_3)}{N_{\iota }}+{\parallel {\mathcal{S}}^{\ast }\parallel }_0·\frac{a^2}{d_1{d}_2{d}_3}\right).\hfill \end{array}$$

(26)

Theorem 3 implies that, if the noise level $\sigma$ and spikiness level a are fixed, and all the corrupted positions are observed exactly only once (i.e., the number of corrupted observations $N_s={\parallel {\mathcal{S}}^{\ast }\parallel }_0$), then the estimation error in Equation (26) would be bounded by

$$\begin{array}{c}\hfill O\left(r_{tb}\left({\mathcal{L}}^{\ast }\right)·\frac{(d_1\vee d_2)d_{3}log(d_1{d}_3+d_2{d}_3)}{N_{\iota }}+{\parallel {\mathcal{S}}^{\ast }\parallel }_0·\left(\frac{log(d_1{d}_3+d_2{d}_3)}{N_{\iota }}+\frac{1}{d_1{d}_2{d}_3}\right)\right).\end{array}$$

(27)

Note that, the bound in Equation (27) is intuition-consistent: if the underlying tensor ${\mathcal{L}}^{\ast }$ gets more complex (i.e., with higher tubal rank), then the estimation error will be larger; if the corruption tensor ${\mathcal{S}}^{\ast }$ gets denser, then the estimation error will also become larger; if the number of uncorrupted observations $N_{\iota }$ gets larger, then the estimation error will decrease. The scaling behavior of the estimation error in Equation (27) will be verified through experiments on synthetic data in Section 7.1.

Remark 1

(Consistence with prior models for robust low-tubal-rank tensor completion). According to Equation (27), our ${\ast }_L$–SVD-based estimator in Equation (8) allows the tubal rank $r_{tb}\left({\mathcal{L}}^{\ast }\right)$ to take the order $O(d_2/log(d_1{d}_3+d_2{d}_3))$, and the corruption ratio $\parallel {\mathcal{S}}^{\ast }{\parallel }_0/\left(d_1{d}_2{d}_3\right)$ to be $O\left(1\right)$ for approximate estimation with small error. It is slightly better with a logarithm factor than the results for t-SVD-based tensor robust completion model in [8] which allows $r_{tb}\left({\mathcal{L}}^{\ast }\right)=O(d_2/{log}^2(d_1{d}_3+d_2{d}_3))$ and $\parallel {\mathcal{S}}^{\ast }{\parallel }_0/\left(d_1{d}_2{d}_3\right)=O\left(1\right)$.

Remark 2

(Consistence with prior models for noisy low-tubal-rank tensor completion). If $\parallel {\mathcal{S}}^{\ast }{\parallel }_0=0$, i.e., the corruption ${\mathcal{S}}^{\ast }$ vanishes, then we obtain

$$\begin{array}{c}\hfill \frac{\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}^2}{d_1{d}_2{d}_3}=O\left(\frac{r_{tb}\left({\mathcal{L}}^{\ast }\right)(d_1\vee d_2)d_{3}log(d_1{d}_3+d_2{d}_3)}{N}\right)\end{array}$$

which is consistent with the error bound for t-SVD-based noisy tensor completion [42,43,44], and ${\ast }_L$–SVD-based tensor Dantzig Selector in [18].

Remark 3

(Consistence with prior models for robust low-tubal-rank tensor decomposition). In the setting of Robust Tensor Decomposition (RTD) [34], the fully observed model instead of our estimation model in Equation (6) is considered. For the RTD problem, our error bound in Equation (27) is consistent with the t-SVD-based bound for RTD [34] (up to a logarithm factor).

Remark 4

(No exact recovery guarantee). According to Theorem 3, when $\sigma =0$ and $\parallel {\mathcal{S}}^{\ast }{\parallel }_0=0$, i.e., in the noiseless case, the estimation error is upper bounded by $O(a(d_1\vee d_2)d_3{r}_{tb}\left({\mathcal{L}}^{\ast }\right)log\tilde{d}/N)$ which is not zero. Thus, no exact recovery is guaranteed by Theorem 3. It can be seen as a trade-off that we do not assume the low-tubal-rank tensor ${\mathcal{L}}^{\ast }$ to satisfy the tensor incoherent conditions [8,35,37] which essentially ensures the separability between ${\mathcal{L}}^{\ast }$ and ${\mathcal{S}}^{\ast }$.

5.2. A Minimax Lower Bound for the Estimation Error

In Theorem 3, we established the estimation error for Model (8). Then one may ask the complementary questions: how tight is the upper bound? Are there fundamental (model-independent) limits of estimation error in robust tensor completion? In this section, we will answer the questions.

To analyze the optimality of the proposed upper bound in Theorem 3, the minimax lower bounds of the estimation error is established for the tensor pair $({\mathcal{L}}^{\ast },{\mathcal{S}}^{\ast })$ belonging to the class $\mathsf{A}(r,s,a)$ of tensor pairs defined as:

$$\begin{array}{c}\hfill \mathsf{A}(r,s,a):=\left\{(\mathcal{L},\mathcal{S})|r_{tb}\left(\mathcal{L}\right)\le {r,\parallel \mathcal{S}\parallel }_0\le {s,\parallel \mathcal{L}\parallel }_{\infty }<a,{\parallel \mathcal{S}\parallel }_{\infty }\le a\right\}\end{array}$$

(28)

We then define the associated element-wise minimax error as follows

$$\begin{array}{c}\hfill \mathcal{M}\left(\mathsf{A}(r,s,a)\right):=\underset{(\widehat{\mathcal{L}},\widehat{\mathcal{S}})}{inf}\underset{({\mathcal{L}}^{\ast },{\mathcal{S}}^{\ast })\in \mathsf{A}(r,s,a)}{sup}\mathbb{E}\left[\frac{\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}^2+{\parallel \widehat{\mathcal{S}}-{\mathcal{S}}^{\ast }\parallel }_{\mathrm{F}}^2}{d_1{d}_2{d}_3}\right],\end{array}$$

(29)

where the infimum ranges over all pairs of estimators $(\widehat{\mathcal{L}},\widehat{\mathcal{S}})$ , the supremum ranges over all pairs of underlying tensors $({\mathcal{L}}^{\ast },{\mathcal{S}}^{\ast })$ in the given tensor class $\mathsf{A}(r,s,a)$, and the expectation is taken over the design tensors $\left\{{\mathcal{X}}_i\right\}$ and i.i.d. Gaussian noises $\left\{{\xi }_i\right\}$ in the observation model (6). We come up with the following theorem.

Theorem 4

(Minimax lower bound). Suppose the dimensionality $d_1,d_2\ge 2$, the rank and sparsity parameters $r\in [d_1\vee d_2],s\le d_1{d}_2{d}_3/2$, the number of uncorrupted entries $N_{\iota }\ge r{d}_1{d}_3$, and the number of corrupted entries $N_s\le \tau r\tilde{d}$ with a constant $\tau >0$. Then, under Assumption A1, there exist absolute constants $b\in (0,1)$ and $c>0$, such that

$$\begin{array}{c}\hfill \mathcal{M}\left(\mathsf{A}\right(r,s,a\left)\right)\ge b\varphi (N,r,s)\end{array}$$

(30)

where

$$\begin{array}{c}\hfill \varphi (N,r,s):={(\sigma \wedge a)}^2\left(\frac{r(d_1+d_2)d_3+N_s}{N_{\iota }}+\frac{s}{d_1{d}_2{d}_3}\right).\end{array}$$

(31)

The lower bound given in Equation (30) implies that the proposed upper bound in Theorem 3 is optimal (up to a log factor) in the minimax sense for tensors belonging to the set $\mathsf{A}(r,s,a)$. That is to say no estimator can obtain more accurate estimation than our estimator in Equation (8) (up to a log factor) for $({\mathcal{L}}^{\ast },{\mathcal{S}}^{\ast })\in \mathsf{A}(r,s,a)$, thereby showing the optimality of the proposed estimator.

6. Connections and Differences with Previous Works

In this section, we discuss the connections and differences with existing nuclear norm based robust matrix/tensor completion models, where the underlying matrix/tensor suffers from missing values, gross sparse outliers, and small dense noises at the same time.

First, we briefly introduce and analyze the two most related models, i.e., the matrix nuclear norm based model [36] and the sum of mode-wise matrix nuclear norms based model [45] as follows.

(1)

The matrix Nuclear Norm (NN) based model [36]: If the underlying tensor is of 2-way, i.e., a matrix, then the observation model in Equation (6) becomes the setting for robust matrix completion, and the proposed estimator in Equation (8) degenerates to the matrix nuclear norm based estimator in [36]. In both model formulation and statistical analysis, this work can be seen as a 3-way generalization of [36].

Moreover, by conducting robust matrix completion on each frontal slice of a 3-way tensor, we can obtain the matrix nuclear norm based robust tensor completion model as follows:

$$\begin{array}{c}\hfill \underset{\mathcal{L},\mathcal{S}}{min}\frac{1}{2N}{\parallel \mathbf{y}-\mathfrak{X}(\mathcal{L}+\mathcal{S})\parallel }_2^2+{\lambda }_{\iota }\sum _{k=1}^{d_3}(\parallel {\mathbf{L}}^{\left(k\right)}{\parallel }_{\ast }+{\lambda }_s\parallel {\mathbf{S}}^{\left(k\right)}{\parallel }_1)\end{array}$$

(32)

(2)

The Sum of mode-wise matrix Nuclear Norms (SNN) based model [45]: Huang et al. [45] proposed a robust tensor completion model based on the sum of mode-wise nuclear norms deduced by the Tucker decomposition as follows

$$\begin{array}{c}\hfill \underset{\mathcal{L},\mathcal{S}}{min}\frac{1}{2N}{\parallel \mathbf{y}-\mathfrak{X}(\mathcal{L}+\mathcal{S})\parallel }_2^2+\sum _{k=1}^3{\alpha }_k\parallel {\mathbf{L}}_{\left(k\right)}{\parallel }_{\ast }+{\lambda }_s{\parallel \mathcal{S}\parallel }_1,\end{array}$$

(33)

where ${\mathbf{L}}_{\left(k\right)}\in {\mathbb{R}}^{d_i\times {\prod }_{j\ne k}{d}_j}$ is the mode-k matriculation of tensor $\mathcal{L}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, for all $i=1,2,3$.

The main differences between SNN and this work are two-fold: (i) SNN is based on the Tucker decomposition [15], whereas this work is based on the recently proposed tensor ${\ast }_L$-SVD [24]; (ii) the theoretical analysis for SNN cannot guarantee the minimax optimality of the model in [45], whereas this works rigorously proof of the minimax optimality of the proposed estimator is established in Section 5.

Then, we discuss the following related works which can be seen as special cases of this work.

(1)

The robust tensor completion model based on t-SVD [46]: In a short conference presentation [46] (whose first author is the same as this paper), the t-SVD-based robust tensor completion model is studied. As t-SVD can be viewed as a special case of the ${\ast }_L$-SVD (when DFT is used as the transform L), the model in [46] can be a special case of ours.

(2)

The robust tensor recovery models with missing values and sparse outliers [8,27]: In [8,27], the authors considered the robust reconstruction of incomplete tensor polluted by sparse outliers, and proposed t-SVD (or ${\ast }_L$-SVD) based models with theoretical guarantees for exact recovery. As they did not consider small dense noises, their settings are indeed a special case of our observation model (6) when $\mathcal{E}=0$.

(3)

The robust tensor decomposition based on t-SVD [34]: In [34], the authors studied the t-SVD-based robust tensor decomposition, which aims at recovering a tensor corrupted by both gross sparse outliers and small dense noises. Comparing with this work, Ref. [34] can be seen as a special case when there are no missing values.

7. Experiments

In this section, experiments on synthetic datasets will be first conducted to validate the sharpness of the proposed upper bounds in Theorem 3. Then, both effectiveness and efficiency of the proposed algorithm will be demonstrated through experiments on seven different types of remote sensing datasets. All codes are written in Matlab, and all experiments are performed on a Windows 10 laptop with AMD Ryzen 3.0 GHz CPU and 8 GB RAM.

7.1. Sharpness of the Proposed Upper Bound

Sharpness of the proposed upper bounds in Theorem 3 will be validated. Specifically, we will check whether the upper bounds in Equation (27) can reflect the true scaling behavior of the estimation error. As predicted in Equation (27), if the upper bound is “sharp”, then it is expected that the Mean Square Errors (MSE) $(\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}^2+\parallel \widehat{\mathcal{S}}-{\mathcal{S}}^{\ast }{\parallel }_{\mathrm{F}}^2)/\left(d_1{d}_2{d}_3\right)$ will possess a scaling behavior very similar to the upper bound: approximately linear w.r.t the tubal rank of the underlying tensor ${\mathcal{L}}^{\ast }$, the $l_0$-norm of the corruption tensor ${\mathcal{S}}^{\ast }$, and the reciprocal of uncorrupted observation number $N_{\iota }$. We will examine whether this expectation will happen in simulation studies on synthetic datasets.

The synthetic datasets are generated as follows. Similar to [26], we consider three cases of linear transform L with orthogonal matrix $\mathbf{L}$: (1) Discrete Fourier Transform (DFT); (2) Discrete Cosine Transform (DCT) [24]; (3) Random Orthogonal Matrix (ROM) [26]. The underlying low-rank tensor ${\mathcal{L}}^{\ast }\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ with ${\ast }_L$-tubal rank $r^{\ast }$ is generated by ${\mathcal{L}}^{\ast }=\mathcal{P}{\ast }_L\mathcal{Q}$, where $\mathcal{P}\in {\mathbb{R}}^{d_1\times r^{\ast }\times d_3}$ and $\mathcal{Q}\in {\mathbb{R}}^{r^{\ast }\times d_2\times d_3}$ are i.i.d. sampled from $\mathcal{N}(0,1)$. ${\mathcal{L}}^{\ast }$ is then normalized such that $\parallel {\mathcal{L}}^{\ast }{\parallel }_{\infty }=1$. Second, to generate the sparse corruption tensor ${\mathcal{S}}^{\ast }$, we first form ${\mathcal{S}}_0$ with i.i.d. uniform distribution $\mathrm{Uni}(0,1)$ and then uniformly select $\gamma d_1{d}_2{d}_3$ entries. Thus the number of corrupted entries $\parallel {\mathcal{S}}^{\ast }{\parallel }_0=\gamma d_1{d}_2{d}_3$. Third, we uniformly select $N_{\iota }$ elements from the uncorrupted positions of $({\mathcal{L}}^{\ast }+{\mathcal{S}}^{\ast })$. Finally, the noise $\left\{{\xi }_i\right\}$ are sampled from i.i.d. Gaussian $N(0,{\sigma }^2)$ with $\sigma =0.1\parallel {\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}/\sqrt{d_1{d}_2{d}_3}$. We consider f-diagonal tensors with $d_1=d_2=d\in \{80,100,120\}$, $d_3=30$ and tubal rank $r_{tb}\left({\mathcal{L}}^{\ast }\right)\in \{3,6,9,12,15\}$. We choose corruption ratio $\gamma \in \{0.01:0.01:0.1\}$ and uncorrupted observation ratio $N_{\iota }/(d_1{d}_2{d}_3-N_s)\in \{0.4:0.1:0.9\}$. In each setting, the MSE averaged over 30 trials is reported.

In Figure 3, we report the results for $100\times 100\times 30$ tensors when the DFT is adopted as the linear transform L in Equation (4). According to sub-plots (a), (b), and (d) in Figure 3, it can be seen that the MSE scales approximately linearly w.r.t. $r_{tb}\left({\mathcal{L}}^{\ast }\right)$, $\parallel {\mathcal{S}}^{\ast }{\parallel }_0$, and $N_{\iota }^{-1}$. There results accord well with our expectation for the size $100\times 100\times 30$ and linear transform $L=\mathrm{DFT}$. As very similar phenomena are also observed in all the other settings where $d\in \{80,120\}$ and $L\in \{\mathrm{DCT},\mathrm{ROM}\}$, we simply omit them.Thus, it can be verified that the scaling behavior of the estimation error can be approximately predicted by the proposed upper bound in Equation (27).

Figure 3. Plots of the MSE versus the tubal rank $r_{tb}\left({\mathcal{L}}^{\ast }\right)$ of the underlying tensor, the number of corruptions $\parallel {\mathcal{S}}^{\ast }{\parallel }_0$, the number of uncorrupted observations $N_{\iota }$ and its inversion $N_{\iota }^{-1}$ : (a) MSE vs. the tubal rank $r_{tb}\left({\mathcal{L}}^{\ast }\right)$ with fixed corruption level $\parallel {\mathcal{S}}^{\ast }{\parallel }_0=0.03d_1{d}_2{d}_3$ and number of uncorrupted observations $N_{\iota }=0.7d_1{d}_2{d}_3-{\parallel {\mathcal{S}}^{\ast }\parallel }_0$; (b) MSE vs. the number of corruptions $\parallel {\mathcal{S}}^{\ast }{\parallel }_0$ with fixed tubal rank 9 and total observation number $0.7d_1{d}_2{d}_3$; (c) MSE vs. the number of uncorrupted observation $N_{\iota }$ with $r_{tb}\left({\mathcal{L}}^{\ast }\right)=3$ and corruption level $\parallel {\mathcal{S}}^{\ast }{\parallel }_0=0.01d_1{d}_2{d}_3$; (d) MSE vs. $N_{\iota }^{-1}$ with $r_{tb}\left({\mathcal{L}}^{\ast }\right)=3$ and $\parallel {\mathcal{S}}^{\ast }{\parallel }_0=0.01d_1{d}_2{d}_3$.

7.2. Effectiveness and Efficient of the Proposed Algorithm

In this section, we evaluate both the effectiveness and efficiency of the proposed Algorithm 1 by conducting robust tensor completion on seven different types of datasets collected from several remote sensing related applications from Section 7.2.1– Section 7.2.7.

Following [25], we adopted three different transformations L in Equation (4) to define the ${\ast }_L$–TNN: the first two transformations are DFT and DCF (denoted by TNN (DFT) and TNN (DCT), respectively), and the third one named TNN (Data) depends on the given data motived by [27,31]. We first perform SVD on the mode-3 unfolding matrix of ${\mathcal{L}}^{\ast }$ as ${\mathbf{L}}_{\left(3\right)}^{\ast }=\mathbf{U}\mathbf{S}\mathbf{V}{}^{\top }$, and then use $\mathbf{U}{}^{\top }$ as the desired transform matrix in the ${\ast }_L$–product (4). The proposed algorithm is compared with the aforementioned models NN [36] in Equation (32) and SNN [45] in Equation (33) in Section 6. Both Model (32) and Model (33) are solved by using ADMM with implementations by ourselves in Matlab language.

We conduct robust tensor completion on the datasets in Figure 4 with a similar settings as [47]. For a $d_1\times d_2\times d_3$ tensor data ${\mathcal{L}}^{\ast }$ re-scaled by $\parallel {\mathcal{L}}^{\ast }{\parallel }_{\infty }=1$, we choose its support uniformly at random with ratio ${\rho }_{\mathrm{s}}$ and fill in the values with i.i.d. standard Gaussian variables to generate the corruption ${\mathcal{S}}^{\ast }$. Then, we randomly sample the entries of ${\mathcal{L}}^{\ast }+{\mathcal{S}}^{\ast }$ uniformly with observation ratio ${\rho }_{\mathrm{obs}}$. The noises $\left\{{\xi }_i\right\}$ are further generated with i.i.d. zero-mean Gaussian entries whose standard deviation is given by $\sigma =0.05\parallel {\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}/\sqrt{d_1{d}_2{d}_3}$ to generate the observations $\left\{y_i\right\}$. The goal in the experiments is to estimate the underlying signal ${\mathcal{L}}^{\ast }$ from $\left\{y_i\right\}$. The effectiveness of algorithms are measured by the Peaks Signal Noise Ratio (PSNR) and structural similarity (SSIM) [48]. Specifically, the PSNR of an estimator $\widehat{\mathcal{L}}$ is defined as

$$\begin{array}{c}\hfill \mathtt{PSNR}:=10{log}_{10}\left(\frac{d_1{d}_2{d}_3{\parallel {\mathcal{L}}^{\ast }\parallel }_{\infty }^2}{\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\mathrm{F}}^2}\right),\end{array}$$

for the underlying tensor ${\mathcal{L}}^{\ast }\in {\mathbb{R}}^{d_1\times d_2\times d_3}$. The SSIM is computed via

$$\begin{array}{c}\hfill \mathtt{SSIM}:=\frac{(2{\mu }_{{\mathcal{L}}^{\ast }}{\mu }_{\widehat{\mathcal{L}}}+{\left(0.01\overline{\omega }\right)}^2)(2{\sigma }_{{\mathcal{L}}^{\ast },\widehat{\mathcal{L}}}+{\left(0.03\overline{\omega }\right)}^2)}{({\mu }_{{\mathcal{L}}^{\ast }}^2+{\mu }_{\widehat{\mathcal{L}}}^2+{\left(0.01\overline{\omega }\right)}^2)({\sigma }_{{\mathcal{L}}^{\ast }}^2+{\sigma }_{\widehat{\mathcal{L}}}^2+{\left(0.03\overline{\omega }\right)}^2)},\end{array}$$

where ${\mu }_{{\mathcal{L}}^{\ast }},{\mu }_{\widehat{\mathcal{L}}},{\sigma }_{{\mathcal{L}}^{\ast }},{\sigma }_{\widehat{\mathcal{L}}},{\sigma }_{{\mathcal{L}}^{\ast },\widehat{\mathcal{L}}}$ and $\overline{\omega }$ denotes the local means, standard deviation, cross-covariance, and dynamic range of the magnitude of tensors ${\mathcal{L}}^{\ast }$ and $\widehat{\mathcal{L}}$. Larger PSNR and SSIM values indicate higher quality of the estimator $\widehat{\mathcal{L}}$.

Figure 4. The dataset consists of the 85-th frame of all the 21 classes in the UCMerced dataset.

7.2.1. Experiments on an Urban Area Imagery Dataset

Area imagery data processing plays a key role in many remote sensing applications, such as land-use mapping [49]. We adopt the popular area imagery dataset UCMerced [50], which is a 21 class land use image dataset meant for research purposes. The images were manually extracted from large images from the USGS National Map Urban Area Imagery collection for various urban areas around the country. The pixel resolution of this public domain imagery is 1 foot, and each RGB image measures $256\times 256$ pixels. There are 100 images for each class, and we chose the 85-th image to form a dataset of 21 images as shown in Figure 4.

We consider two scenarios by setting $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})\in \{(0.3,0.2),(0.8,0.3)\}$ for the $d\times d\times 3$ images. For NN (Model (32)), we set the regularization parameters ${\lambda }_s={\lambda }_{\iota }/\sqrt{d{\rho }_{\mathrm{obs}}}$ (suggested by [38]), and tune the parameter ${\lambda }_{\iota }$ around $6.5\sigma \sqrt{{\rho }_{\mathrm{obs}}dlog\left(6d\right)}$ (suggested by [51]). For SNN, the parameter ${\lambda }_s$ is tuned in $\{0.01,0.05,0.1,1\}$ for better performance in most cases, and the weight $\alpha$ is set by ${\alpha }_1={\alpha }_2={\lambda }_s\sqrt{3d{\rho }_{\mathrm{obs}}},{\alpha }_3=0.01{\lambda }_s\sqrt{3d{\rho }_{\mathrm{obs}}}$. For Algorithm 1, we tune ${\lambda }_{\iota }$ around $2\sigma \sqrt{3{\rho }_{\mathrm{obs}}dlog\left(6d\right)}$, and let ${\lambda }_s={\lambda }_{\iota }/\sqrt{3d{\rho }_{\mathrm{obs}}}$ for TNN (DFT) and ${\lambda }_s={\lambda }_{\iota }/\sqrt{d{\rho }_{\mathrm{obs}}}$ for TNN (DCT) and TNN (Data). In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds).

We present the PSNR, SSIM values and running time in Figure 5 and Figure 6 for settings of $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})=(0.3,0.2)$ and $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})=(0.8,0.3)$, respectively, for quantitative evalution, with visual examples shown in Figure 7 and Figure 8. It can seen that from Figure 5, Figure 6, Figure 7 and Figure 8 that the proposed TNN (Data) has the highest recovery quality in most cases, and posses a comparative running time as NN. We attribute the promising performance of the proposed algorithm to the extraordinary representation power of the low-tubl-rank models: low-tubal-rankness can exploit both low-rankness and smoothness simultaneously, whereas traditional models like NN and SNN can only exploit low-rankness in the original domain [18].

Figure 5. The PSNR, SSIM values and running time (in seconds) on the UCMerced dataset for the setting $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})=(0.3,0.2)$.

Figure 6. The PSNR, SSIM values and running time (in seconds) on the UCMerced dataset for the setting $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})=(0.8,0.3)$.

Figure 7. The visual examples for five models on UCMerced dataset for the setting $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})=(0.3,0.2)$. (a) The original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

Figure 8. The visual examples for five models on UCMerced dataset for the setting $({\rho }_{\mathrm{obs}},{\rho }_{\mathrm{s}})=(0.8,0.3)$. (a) The original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

7.2.2. Experiments on Hyperspectral Data

Benefit from its fine spectral and spatial resolutions, hyperspectral image processing has been extensively adopted in many remote sensing applications [10,52]. In this section, we conduct robust tensor completion on subsets of the two representative hyperspectral datasets described as follows:

• Indian Pines: This dataset was collected by AVIRIS sensor in 1992 over the Indian Pines test site in North-western Indiana and consists of $145\times 145$ pixels and 224 spectral reflectance bands. We use the first 30 bands in the experiments due to the trade-off between the limitation of computing resources and the efforts for parameter tuning.
• Salinas A: The data were acquired by AVIRIS sensor over the Salinas Valley, California in 1998, and consists of 224 bands over a spectrum range of 400–2500 nm. This dataset has a spatial extent of $86\times 83$ pixels with a resolution of 3.7 m. We use the first 30 bands in the experiments too.

We consider three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3$, ${\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6$, ${\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$ for robust completion of hyper-spectral data. For NN, we set the regularization parameters ${\lambda }_s={\lambda }_{\iota }/\sqrt{{\rho }_{\mathrm{obs}}(d_1\vee d_2)}$ (suggested by [38]), and tune the parameter ${\lambda }_{\iota }$ around $6.5\sigma \sqrt{{\rho }_{\mathrm{obs}}(d_1\vee d_2)log(d_1{d}_3+d_2{d}_3)}$ (suggested by [51]). For SNN, the parameter ${\lambda }_s$ is tuned in $\{0.01,0.05,0.1,1\}$ for better performance in most cases, and we chose the weight $\alpha$ by ${\alpha }_1={\alpha }_2={\alpha }_3={\lambda }_s\sqrt{{\rho }_{\mathrm{obs}}(d_1\vee d_2)d_3}$ (suggested by [47]). For Algorithm 1, we tune the parameter ${\lambda }_{\iota }$ around $2\sigma \sqrt{{\rho }_{\mathrm{obs}}(d_1\vee d_2)d_{3}log(d_1{d}_3+d_2{d}_3)}$, and let ${\lambda }_s={\lambda }_{\iota }/\sqrt{{\rho }_{\mathrm{obs}}(d_1\vee d_2)d_3}$ for TNN (DFT) and ${\lambda }_s={\lambda }_{\iota }/\sqrt{{\rho }_{\mathrm{obs}}(d_1\vee d_2)}$ for TNN (DCT) and TNN (Data). In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds).

For quantitative evalution, we report the PSNR, SSIM values and running time in Table 2 and Table 3 for the Indian Pines and Salinas A datasets, respectively. The visual examples are, respectively, shown in Figure 9 and Figure 10. It can seen that the proposed TNN (Data) has the highest recovery quality in most cases, and has a comparative running time as NN, indicating the effectiveness and efficiency of low-tubal-rank models in comparison with original domain-based models NN and SNN.

Table 2. Quantitative evaluation on the Indian Pines dataset in PSNR, SSIM, and running time of five models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	20.63	25.46	28.49	29.33	30.08
Setting I	SSIM	0.4842	0.7275	0.7619	0.7872	0.8181
	TIME	14.77	40.1	11.17	15.53	13.54
	PSNR	21.95	27.66	29.49	30.17	30.61
Setting II	SSIM	0.5454	0.7864	0.7912	0.8073	0.8296
	TIME	14.18	39.76	11.04	15.23	13.29
	PSNR	22.43	28.22	29.64	30.31	30.87
Setting III	SSIM	0.5534	0.8051	0.7971	0.8139	0.8345
	TIME	14.21	38.88	11.05	15.27	13.43

Table 3. Quantitative evaluation on the Salinas A dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	19.01	26.18	27.1	30.99	32.69
Setting I	SSIM	0.4918	0.837	0.7501	0.8350	0.8774
	TIME	5.57	11.53	4.07	5.73	4.9
	PSNR	20.97	28.79	29.4	32.26	33.3
Setting II	SSIM	0.5806	0.8675	0.8117	0.8645	0.8714
	TIME	5.41	11.36	4.02	5.67	4.79
	PSNR	21.54	29.5	29.73	32.38	33.54
Setting III	SSIM	0.5914	0.8772	0.8208	0.8683	0.8848
	TIME	5.34	11.01	3.98	5.59	4.91

Figure 9. Visual results of robust tensor completion for five models on the 21st bound of Indian Pines dataset. The top, middle, and bottum row corresponds to the Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$, respectively. The sub-plots from (a) to (g): (a) the original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

Figure 10. Visual results of robust tensor completion for five models on the 21st bound of Salinas A dataset. The top, middle, and bottum row corresponds to the Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$, respectively. The sub-plots from (a) to (g): (a) the original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

7.2.3. Experiments on Multispectral Images

Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum, and has become one of the most widely utilized datatype in remote sensing. This section presents simulated experiments on multispectral images. The original data are two multispectral images Beads and Cloth from the Columbia MSI Database (available at http://www1.cs.columbia.edu/CAVE/databases/multispectral accessed on 28 July 2021) containing scenes of a variety of real-world objects. Each MSI is of size 512 × 512 × 31 with intensity range scaled to [0, 1].

We also consider three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,$${\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$ for robust completion of multi-spectral data. We tune the parameters in the same way as Section 7.2.2. In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds).

For quantitative evalution, we report the PSNR, SSIM values and running time in Table 4 and Table 5 for the Beads and Cloth datasets, respectively. The visual examples for the Cloth dataset is shown in Figure 11. We can also find that the proposed TNN (Data) achieves the highest accuracy in most cases, and has a comparative running time as NN, which demonstrates both the effectiveness and efficiency of low-tubal-rank models.

Table 4. Quantitative evaluation on the Beads dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	18.58	18.71	25.11	25.18	27.05
Setting I	SSIM	0.448	0.6208	0.804	0.8203	0.8673
	TIME	309.55	933.12	280	260.46	241.65
	PSNR	20.4	21.35	27.31	27.46	28.9
Setting II	SSIM	0.5406	0.7603	0.8754	0.8894	0.9143
	TIME	302.95	915.57	276.2	268.58	244.5
	PSNR	21.01	22.36	27.96	28.13	29.4
Setting III	SSIM	0.5531	0.7848	0.8803	0.8944	0.9165
	TIME	301.92	922.07	276.99	272.59	244.02

Table 5. Quantitative evaluation on the Cloth dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	21.5	22.79	29.7	30.83	30.77
Setting I	SSIM	0.5054	0.6333	0.8649	0.8883	0.8941
	TIME	308.29	915.43	281.4	264.84	242.63
	PSNR	22.63	24.94	32.32	33.57	33.86
Setting II	SSIM	0.5566	0.7355	0.916	0.9323	0.9391
	TIME	300.3	911.27	273.36	268.46	243.37
	PSNR	22.99	25.78	32.76	34.02	34.39
Setting III	SSIM	0.5652	0.7643	0.9183	0.9342	0.941
	TIME	297.94	910.64	280.51	268.25	246.14

Figure 11. Visual results of robust tensor completion for five models on the 21st bound of Cloth dataset. The top, middle, and bottum row corresponds to the Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$, respectively. The sub-plots from (a) to (g): (a) The original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

7.2.4. Experiments on Point Could Data

With the rapid advances of sensor technology, the emerging point cloud data provide better performance than 2D images in many remote sensing applications due to its flexible and scalable geometric representation [53]. In this section, we also conduct experiments on a dataset (scenario B from http://www.mrt.kit.edu/z/publ/download/velodynetracking/dataset.html, accessed on 28 July 2021) for Unmanned Ground Vehicle (UGV). The dataset contains a sequence of point cloud data acquired from a Velodyne HDL-64E LiDAR. We select 30 frames (Frame Nos. 65-94) from the data sequence. The point cloud data is formatted into two tensors sized $64\times 870\times 30$ representing the distance data (named SenerioB Distance) and the intensity data (named SenerioB Intensity), , respectively.

We also consider three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,$${\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$ for robust completion of point cloud data. We tune the parameters in the same way as Section 7.2.2. In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds). For quantitative evalution, we report the PSNR, SSIM values and running time in Table 6 and Table 7 for the SenerioB Distance and SenerioB Intensity datasets, respectively. We can also find that the proposed TNN (Data) achieves the highest accuracy in most cases, and has a comparative running time as NN, which demonstrates both the effectiveness and efficiency of low-tubal-rank models.

Table 6. Quantitative evaluation on the SenerioB Distance dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	17.55	20.01	23.86	23.86	23.87
Setting I	SSIM	0.468	0.763	0.8732	0.8737	0.8739
	TIME	15.22	186.31	14.49	19.66	15.83
	PSNR	18.57	23.87	25.28	25.31	25.34
Setting II	SSIM	0.551	0.9055	0.9096	0.91	0.9105
	TIME	15.51	189.87	15.55	18.68	16.31
	PSNR	18.98	24.78	25.79	25.83	25.87
Setting III	SSIM	0.5678	0.9197	0.9179	0.9184	0.9189
	TIME	15.03	195	14.8	19.11	15.02

Table 7. Quantitative evaluation on the SenerioB Intensity dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	16.35	20.35	21.28	21.26	21.31
Setting I	SSIM	0.2588	0.7076	0.7114	0.7116	0.7137
	TIME	15.13	188.96	14.18	19.06	15.19
	PSNR	17.09	22.17	22.28	22.32	22.45
Setting II	SSIM	0.3149	0.7889	0.7708	0.7718	0.7781
	TIME	14.64	187.61	14.09	19.1	15.74
	PSNR	17.35	22.48	22.57	22.61	22.79
Setting III	SSIM	0.3331	0.7985	0.7828	0.7836	0.7914
	TIME	14.7	187.66	14.23	19	15.22

7.2.5. Experiments on Aerial Video Data

Aerial videos (or time sequences of images) are broadly used in many computer vision based remote sensing tasks [54]. We experiment on a $180\times 320\times 30$ tensor which consists of the first 30 frames of the Sky dataset (available at http://www.loujing.com/rss-small-target, accessed on 28 July 2021) for small object detection [55].

We also consider three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. We tune the parameters in the same way as Section 7.2.2. In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds). For quantitative evalution, we report the PSNR, SSIM values and running time in Table 8. It is also found that the proposed TNN (Data) achieves the highest accuracy in most cases, and can run as fast as NN.

Table 8. Quantitative evaluation on the Sky dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	21.03	26.74	28.67	28.59	29.74
Setting I	SSIM	0.4875	0.7805	0.708	0.7076	0.788
	TIME	36.1	144.84	28.1	39.85	34.43
	PSNR	22.44	28.8	29.41	29.35	30.48
Setting II	SSIM	0.5715	0.8026	0.7155	0.7147	0.7814
	TIME	34.23	138.52	27.11	38.77	33.74
	PSNR	22.77	28.72	29.55	29.49	30.59
Setting III	SSIM	0.5786	0.7471	0.7324	0.7307	0.7906
	TIME	34.42	139.78	26.97	39.14	33.86

7.2.6. Experiments on Thermal Imaging Data

Thermal infrared data can provide important measurements of surface energy fluxes and temperatures in various remote sensing applications [7]. In this section, we experiment on two infrared datasets as follows:

• The Infraed Detection dataset [56]: this dataset is collected for infrared detection and tracking of dim-small aircraft targets under ground/air background (available at http://www.csdata.org/p/387/, accessed on 28 July 2021). It consists of 22 subsets of infrared image sequences of all aircraft targets. We use the first 30 frames of data3.zip to form a $256\times 256\times 30$ tensor due to the trade-off between the limitation of computing resources and the efforts for parameter tuning.
• The OSU Thermal Database [3]: The sequences were recorded on the Ohio State University campus during the months of February and March 2005, and show several people, some in groups, moving through the scene. We use the first 30 frames of Sequences 1 and form a tensor of size $320\times 240\times 30$.

Similiar to Section 7.2.2, we test in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$, and use the same strategy for parameter tuning. In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds). For quantitative evalution, we report the PSNR, SSIM values and running time in Table 9 and Table 10 for the Infraed Detection and OSU Thermal Database datasets, respectively. The visual examples are, respectively, shown in Figure 12 and Figure 13. It can seen that the proposed TNN (Data) has the highest recovery quality in most cases, and has a comparative running time as NN, showing both effectiveness and efficiency of low-tubal-rank models in comparison with original domain-based models NN and SNN.

Table 9. Quantitative evaluation on the Infraed Detection dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	24.82	30.09	31.94	32.07	32.84
Setting I	SSIM	0.6021	0.8231	0.7408	0.7437	0.7768
	TIME	49.01	215.41	48.97	52.37	46.59
	PSNR	26.58	31.8	32.33	32.43	33.11
Setting II	SSIM	0.6679	0.8414	0.7428	0.7453	0.7724
	TIME	47.55	217.43	50.07	52.9	47.18
	PSNR	26.95	31.9	33.11	33.2	33.81
Setting III	SSIM	0.6682	0.8454	0.7237	0.7265	0.7525
	TIME	48.65	216.42	49.13	52.81	46.94

Table 10. Quantitative evaluation on the OSU Thermal Database in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	15.62	21.5	31.33	31.5	31.51
Setting I	SSIM	0.3402	0.8105	0.9345	0.9347	0.935
	TIME	49	222.49	40.31	49.45	42.22
	PSNR	17.47	29.48	32.88	33.19	33.21
Setting II	SSIM	0.4428	0.9057	0.9427	0.9431	0.9433
	TIME	46.18	197.9	36.14	47.19	41.79
	PSNR	18.17	30.83	33.31	33.71	33.75
Setting III	SSIM	0.468	0.9265	0.9495	0.95	0.9507
	TIME	45.85	200.39	36.39	46.96	41.36

Figure 12. Visual results of robust tensor completion for five models on the 21st bound of Infraed Detection dataset. The top, middle, and bottum row corresponds to the Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$, respectively. The sub-plots from (a) to (g): (a) the original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

Figure 13. Visual results of robust tensor completion for five models on the 21st bound of OSU Thermal Database dataset. The top, middle, and bottum row corresponds to the Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$, respectively. The sub-plots from (a) to (g): (a) the original image; (b) the observed image; (c) image recovered by the matrix nuclear norm (NN) based Model (32); (d) recovered by the sum of mode-wise nuclear norms (SNN) based Model (33); (e) image recovered by TNN (DFT); (f) image recovered by TNN (DCT); (g) image recovered by TNN (Data).

7.2.7. Experiments on SAR Data

Polarimetric synthetic aperture radar (PolSAR) has attracted lots of attention from remote sensing scientists because of its various advantages, e.g., all-weather, all-time, penetrating capability, and multi-polarimetry [57]. In this section, we adopt the PolSAR UAVSAR Change Detection Images dataset. It is a dataset of single-look quad-polarimetric SAR images acquired by the UAVSAR airborne sensor in L-band over an urban area in San Francisco city on 18 September 2009, and May 11, 2015. The dataset #1 have length and width of 200 pixels, and we use the first 30 bands.

We also consider three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,$${\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. We tune the parameters in the same way as Section 7.2.2. In each setting, we test each image for 10 trials and report the averaged PSNR (in db), SSIM and running time (in seconds) in Table 11. It is also found that the proposed TNN (Data) achieves the highest accuracy in most cases, and can run as fast as NN.

Table 11. Quantitative evaluation on the UAVSAR-Dataset1-2015 dataset in PSNR, SSIM, and running time of five tensor completion models for robust tensor completion in three settings, i.e., Setting I $({\rho }_{\mathrm{obs}}=0.3,{\rho }_{\mathrm{s}}=0.2)$, Setting II $({\rho }_{\mathrm{obs}}=0.6,{\rho }_{\mathrm{s}}=0.25)$, and Setting III $({\rho }_{\mathrm{obs}}=0.8,{\rho }_{\mathrm{s}}=0.3)$. The highest PSNR/SSIM, or lowest time (in seconds) is highlighted in bold.

Settings	Metrics	NN	SNN	TNN-DFT	TNN-DCT	TNN-Data
	PSNR	29.14	25.86	26.22	26.5	31.62
Setting I	SSIM	0.8748	0.8797	0.8868	0.8909	0.9438
	TIME	23.54	75.07	17.46	25.24	22.6
	PSNR	31.3	26.71	26.96	27.31	34.28
Setting II	SSIM	0.9044	0.8742	0.9018	0.9059	0.9615
	TIME	23.09	74.75	17.6	24.77	22.59
	PSNR	31.8	26.98	27.14	27.66	35.03
Setting III	SSIM	0.9118	0.8829	0.903	0.9092	0.9649
	TIME	22.88	73.03	17.64	25.22	22.54

8. Conclusions

In this paper, we resolve the challenging robust tensor completion problem by proposing a ${\ast }_L$-SVD-based estimator to robustly reconstruct a low-rank tensor in the presence of missing values, gross outliers, and small noises simultaneously. Specifically, this work can be concluded in the following three aspects:

(1)

Algorithmically, we design an efficient algorithm within the framework of ADMM to efficiently compute the proposed estimator with guaranteed convergence behavior.

(2)

Statistically, we analyze the statistical performance of the proposed estimator by establishing a non-asymptotic upper bound on the estimation error. The proposed upper bound is further proved to be minimax optimal (up to a log factor).

(3)

Experimentally, the correctness of the upper bound is first validated through simulations on synthetic datasets. Then both effectiveness and efficiency of the proposed algorithm are demonstrated by extensive comparisons with state-of-the-art nuclear norm based models (i.e., NN and SNN) on seven different types of remote sensing data.

However, from a critical point of view, the proposed method has the following two limitations:

(1)

The orientational sensitivity of ${\ast }_L$-SVD: Despite the promising empirical performance of the ${\ast }_L$-SVD-based estimator, a typical defect of it is the orientation sensitivity owing to low-rankness strictly defined along the tubal orientation which makes it fail to simultaneously exploit transformed low-rankness in multiple orientations [19,58].

(2)

The difficulty in finding the optimal transform $L(·)$ for ${\ast }_L$-SVD: Although a direct use of fixed transforms (like DFT and DCT) may produce fairish empirical performance, it is still unclear how to find the best optimal transformation $L(·)$ for any certain tensor ${\mathcal{L}}^{\ast }$ when only partial and corrupted observations are available.

According to the above limitations, it is interesting to consider higher-order extensions of the proposed model in an orientation invariant way like [19] and discuss the statistical performance. It is also interesting to consider the data-dependent transformation learning like [31,59]. Another future direction is to consider more efficient solvers of Problem (8) using the factorization strategy or Frank–Wolfe method [47,60,61,62].

Appendix A.1. Additional Notations and Preliminaries

For the ease of exposition, we first list the additive notations often used in the proofs in Table A1.

Table A1. Additional notations in the proofs.

Notations	Descriptions	Notations	Descriptions
${\Delta }^{\iota }:={\mathcal{L}}^{\ast }-\widehat{\mathcal{L}}$	estimation error of ${\mathcal{L}}^{\ast }$	${\Delta }^s:={\mathcal{S}}^{\ast }-\widehat{\mathcal{S}}$	estimation error of ${\mathcal{S}}^{\ast }$
$r^{\ast }:=r_{tb}\left({\mathcal{L}}^{\ast }\right)$	${\ast }_L$-tubal-rank of ${\mathcal{L}}^{\ast }$	$s^{\ast }:={\parallel {\mathcal{S}}^{\ast }\parallel }_0$	sparsity of ${\mathcal{S}}^{\ast }$
$\varrho :=N/N_{\iota }$	inverse uncorrupted ratio	a	$l_{\infty }$-norm bound in Equation (8)
$\tilde{d}$	$(d_1+d_3)d_3$
${\Omega }_{\iota }:=\{i\in \left[N\right]|⟨{\mathcal{X}}_i,{\mathcal{S}}^{\ast }⟩=0\}$	index set of design tensors $\left\{{\mathcal{X}}_i\right\}$ corresponding to uncorrupted entries
${\Omega }_s:=\{i\in \left[N\right]|⟨{\mathcal{X}}_i,{\mathcal{S}}^{\ast }⟩\ne 0\}$	index set of design tensors $\left\{{\mathcal{X}}_i\right\}$ corresponding to corrupted entries
$\mathcal{E}:=\frac{1}{N}{\sum }_{i\in {\Omega }_{\iota }}{\xi }_i{\mathcal{X}}_i$	stochastic tensor defined to lower bound parameters ${\lambda }_{\iota }$ and ${\lambda }_s$
$\mathcal{W}:=\frac{1}{N}{\sum }_{i\in {\Omega }_{\iota }}{\mathcal{X}}_i$	stochastic tensor defined to lower bound parameter ${\lambda }_s$
${\mathcal{R}}_{\Sigma }:=\frac{1}{N_{\iota }}{\sum }_{i\in {\Omega }_{\iota }}{\epsilon }_i{\mathcal{X}}_i$	random tensor defined in bounding $\parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}$ with i.i.d. Rademacher $\left\{{\epsilon }_i\right\}$
${\parallel \mathcal{T}\parallel }_{\Pi }:=\sqrt{{\mathbb{E}}_{{\mathcal{X}}_i}\left[{⟨{\mathcal{X}}_i,{\mathcal{T}}_{{\Theta }_s^{\perp }}⟩}^2\right]}$	expectation of ${⟨{\mathcal{X}}_i,·⟩}^2$ for $i\in {\Omega }_{\iota }$ defined to establish the RSC condition

Table A1. Additional notations in the proofs.

Notations	Descriptions	Notations	Descriptions
${\Delta }^{\iota }:={\mathcal{L}}^{\ast }-\widehat{\mathcal{L}}$	estimation error of ${\mathcal{L}}^{\ast }$	${\Delta }^s:={\mathcal{S}}^{\ast }-\widehat{\mathcal{S}}$	estimation error of ${\mathcal{S}}^{\ast }$
$r^{\ast }:=r_{tb}\left({\mathcal{L}}^{\ast }\right)$	${\ast }_L$-tubal-rank of ${\mathcal{L}}^{\ast }$	$s^{\ast }:={\parallel {\mathcal{S}}^{\ast }\parallel }_0$	sparsity of ${\mathcal{S}}^{\ast }$
$\varrho :=N/N_{\iota }$	inverse uncorrupted ratio	a	$l_{\infty }$-norm bound in Equation (8)
$\tilde{d}$	$(d_1+d_3)d_3$
${\Omega }_{\iota }:=\{i\in \left[N\right]|⟨{\mathcal{X}}_i,{\mathcal{S}}^{\ast }⟩=0\}$	index set of design tensors $\left\{{\mathcal{X}}_i\right\}$ corresponding to uncorrupted entries
${\Omega }_s:=\{i\in \left[N\right]|⟨{\mathcal{X}}_i,{\mathcal{S}}^{\ast }⟩\ne 0\}$	index set of design tensors $\left\{{\mathcal{X}}_i\right\}$ corresponding to corrupted entries
$\mathcal{E}:=\frac{1}{N}{\sum }_{i\in {\Omega }_{\iota }}{\xi }_i{\mathcal{X}}_i$	stochastic tensor defined to lower bound parameters ${\lambda }_{\iota }$ and ${\lambda }_s$
$\mathcal{W}:=\frac{1}{N}{\sum }_{i\in {\Omega }_{\iota }}{\mathcal{X}}_i$	stochastic tensor defined to lower bound parameter ${\lambda }_s$
${\mathcal{R}}_{\Sigma }:=\frac{1}{N_{\iota }}{\sum }_{i\in {\Omega }_{\iota }}{\epsilon }_i{\mathcal{X}}_i$	random tensor defined in bounding $\parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}$ with i.i.d. Rademacher $\left\{{\epsilon }_i\right\}$
${\parallel \mathcal{T}\parallel }_{\Pi }:=\sqrt{{\mathbb{E}}_{{\mathcal{X}}_i}\left[{⟨{\mathcal{X}}_i,{\mathcal{T}}_{{\Theta }_s^{\perp }}⟩}^2\right]}$	expectation of ${⟨{\mathcal{X}}_i,·⟩}^2$ for $i\in {\Omega }_{\iota }$ defined to establish the RSC condition

We then introduce the decomposability of ${\ast }_L$-TNN and tensor $l_1$-norm which plays a key role in the analysis.

Decomposability of ${\ast }_L$-TNN. Suppose ${\mathcal{L}}^{\ast }$ has reduced ${\ast }_L$-SVD as ${\mathcal{L}}^{\ast }=\mathcal{U}{\ast }_L\mathcal{D}{\ast }_L\mathcal{V}{}^{\top }$, where $\mathcal{U}\in {\mathbb{R}}^{d_1\times r^{\ast }\times d_3}$ and $\mathcal{V}\in {\mathbb{R}}^{d_2\times r^{\ast }\times d_3}$ are orthogonal and $\mathcal{D}\in {\mathbb{R}}^{r^{\ast }\times r^{\ast }\times d_3}$ is f-diagonal. Define projectors ${\mathcal{P}}^{\star }(·)$ and ${\mathcal{P}}^{\perp }(·)$ as follows:

$$\begin{array}{cc}& {\mathcal{P}}^{\star }\left(\mathcal{T}\right)=\mathcal{U}{\ast }_L\mathcal{U}{}^{\top }{\ast }_L\mathcal{T}+\mathcal{T}{\ast }_L\mathcal{V}{\ast }_L\mathcal{V}{}^{\top }-\mathcal{U}{\ast }_L\mathcal{U}{}^{\top }{\ast }_L\mathcal{T}{\ast }_L\mathcal{V}{\ast }_L\mathcal{V}{}^{\top },\hfill \\ & {\mathcal{P}}^{\perp }\left(\mathcal{T}\right)=(\mathcal{I}-\mathcal{U}{\ast }_L\mathcal{U}{}^{\top }){\ast }_L\mathcal{T}{\ast }_L(\mathcal{I}-\mathcal{V}{\ast }_L\mathcal{V}{}^{\top }).\hfill \end{array}$$

Then, it can be verified that:

(I).

$\mathcal{T}={\mathcal{P}}^{\star }\left(\mathcal{T}\right)+{\mathcal{P}}^{\perp }\left(\mathcal{T}\right)$, $\forall \mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$;

(II).

$⟨{\mathcal{P}}^{\star }\left(\mathcal{A}\right),{\mathcal{P}}^{\perp }\left(\mathcal{B}\right)⟩=0$, $\forall \mathcal{A},\mathcal{B}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$.

(III).

$r_{tb}\left({\mathcal{P}}^{\star }\left(\mathcal{T}\right)\right)\le 2r_{tb}\left({\mathcal{L}}^{\ast }\right)$, $\forall \mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$.

In the same way to the results in supplementary material of [43], it can also be shown that the following equations hold:

(I).

(Decomposability of ${\ast }_L$–TNN) For any $\mathcal{A},\mathcal{B}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$ satisfying $\mathcal{A}{\ast }_L\mathcal{B}{}^{\top }=\mathbf{0}$, $\mathcal{A}{}^{\top }{\ast }_L\mathcal{B}=\mathbf{0}$,

$$\begin{array}{c}\hfill {\parallel \mathcal{A}+\mathcal{B}\parallel }_{\star }=\parallel {\mathcal{P}}^{\star }\left(\mathcal{A}\right){\parallel }_{\star }+{\parallel {\mathcal{P}}^{\perp }\left(\mathcal{B}\right)\parallel }_{\star }.\end{array}$$

(A1)

(II).

(Norm compatibility inequality) For any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$

$$\begin{array}{c}\hfill {\parallel \mathcal{T}\parallel }_{\star }=\sqrt{d_3{r}_{tb}\left(\mathcal{T}\right)}{\parallel \mathcal{T}\parallel }_{\mathrm{F}}.\end{array}$$

(A2)

Decomposability of tensor $l_1$-norm [63]. Let ${\Omega }_{\mathrm{e}}\subset \left[d_1\right]\times \left[d_2\right]\times \left[d_3\right]$ denote any index set and ${\Omega }_{\mathrm{e}}^{\perp }$ its complement. Then for any tensor $\mathcal{T}$, define two tensors ${\mathcal{T}}_{{\Omega }_{\mathrm{e}}}$ and ${\mathcal{T}}_{{\Omega }_{\mathrm{e}}^{\perp }}$ as follows

$$\begin{array}{c}\hfill {\mathcal{T}}_{{\Omega }_{\mathrm{e}}}(i,j,k):=\left\{\begin{array}{cc}& T_{ijk},(i,j,k)\in {\Omega }_{\mathrm{e}}\hfill \\ & 0,(i,j,k)\in {\Omega }_{\mathrm{e}}^{\perp }\hfill \end{array}\right.,{\mathcal{T}}_{{\Omega }_{\mathrm{e}}^{\perp }}:=\mathcal{T}-{\mathcal{T}}_{{\Omega }_{\mathrm{e}}}.\end{array}$$

(A3)

Then, one has

(I).

(Decomposability of $l_1$-norm) For any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, ${\parallel \mathcal{T}\parallel }_1=\parallel {\mathcal{T}}_{{\Omega }_{\mathrm{e}}}{\parallel }_1+{\parallel {\mathcal{T}}_{{\Omega }_{\mathrm{e}}^{\perp }}\parallel }_1.$

(II).

(Norm compatibility inequality) For any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$, $\parallel {\mathcal{T}}_{{\Omega }_{\mathrm{e}}}{\parallel }_1=\sqrt{|{\Omega }_{\mathrm{e}}|}{\parallel {\mathcal{T}}_{{\Omega }_{\mathrm{e}}}\parallel }_{\mathrm{F}}.$

Appendix A.2. The Proof for Theorem 3

The proof follows the lines of [34,36]. For notational simplicity, we define the following two sets

$$\begin{array}{c}\hfill {\Omega }_{\iota }:=\{i\in \left[N\right]|⟨{\mathcal{X}}_i,{\mathcal{S}}^{\ast }⟩=0\},{\Omega }_s:=\{i\in \left[N\right]|⟨{\mathcal{X}}_i,{\mathcal{S}}^{\ast }⟩\ne 0\}\end{array}$$

(A4)

which denote the index set of design tensors $\left\{{\mathcal{X}}_i\right\}$ corresponding to uncorrupted/corrupted entries, respectively.

Appendix A.2.1. Mainstream of Proving Theorem 3

Proof of Theorem 3.

Let $\mathcal{F}(\mathcal{L},\mathcal{S})=\frac{1}{N}{\sum }_{i=1}^N{(y_i-⟨\mathcal{L}+\mathcal{S},{\mathcal{X}}_i⟩)}^2+{\lambda }_{\iota }{\parallel \mathcal{L}\parallel }_{\star }+{\lambda }_s{\parallel \mathcal{S}\parallel }_1$ for simplicity. Then, according to the optimality of $(\widehat{\mathcal{L}},\widehat{\mathcal{S}})$ to Problem (8), it holds that

$$\begin{array}{c}\hfill \mathcal{F}(\widehat{\mathcal{L}},\widehat{\mathcal{S}})\le \mathcal{F}({\mathcal{L}}^{\ast },{\mathcal{S}}^{\ast })\end{array}$$

(A5)

and

$$\begin{array}{cc}\hfill & \parallel {\Delta }^{\iota }{\parallel }_{\infty }\stackrel{}{=}\parallel \widehat{\mathcal{L}}-{\mathcal{L}}^{\ast }{\parallel }_{\infty }\stackrel{}{\le }\parallel \widehat{\mathcal{L}}{\parallel }_{\infty }+{\parallel {\mathcal{L}}^{\ast }\parallel }_{\infty }\stackrel{}{\le }2a\hfill \\ & \parallel {\Delta }^s{\parallel }_{\infty }\stackrel{}{=}\parallel \widehat{\mathcal{S}}-{\mathcal{S}}^{\ast }{\parallel }_{\infty }\stackrel{}{\le }\parallel \widehat{\mathcal{S}}{\parallel }_{\infty }+{\parallel {\mathcal{S}}^{\ast }\parallel }_{\infty }\stackrel{}{\le }2a\hfill \end{array}$$

(A6)

Equation (A5) indicates that

$$\begin{array}{c}\hfill \frac{1}{N}\sum _{i=1}^N{({\xi }_i+⟨{\Delta }^{\iota }+{\Delta }^s,{\mathcal{X}}_i⟩)}^2+{\lambda }_{\iota }\parallel \widehat{\mathcal{L}}{\parallel }_{\star }+{\lambda }_s\parallel \widehat{\mathcal{S}}{\parallel }_1\le \frac{1}{N}\sum _{i=1}^N{\xi }_i^2+{\lambda }_{\iota }\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }+{\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1\end{array}$$

(A7)

which leads to

$$\begin{array}{cc}\hfill \frac{1}{N}\sum _{i\in {\Omega }_{\iota }}{⟨{\Delta }^{\iota }+{\Delta }^s,{\mathcal{X}}_i⟩}^2& \le \underset{:=\mathbf{I}}{\underbrace{\frac{2}{N}{\sum }_{i\in {\Omega }_s}\left|⟨{\xi }_i{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩\right|-\frac{1}{N}{\sum }_{i\in {\Omega }_s}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2}}\hfill \\ & +\underset{:=\mathbf{I}\mathbf{I}}{\underbrace{2|⟨\mathcal{E},{\Delta }^{\iota }⟩|+{\lambda }_{\iota }(\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-\parallel \widehat{\mathcal{L}}{\parallel }_{\star })}}\hfill \\ & +\underset{:=\mathbf{I}\mathbf{I}\mathbf{I}}{\underbrace{2\left|⟨\mathcal{E},{\Delta }_{{\Theta }_s^{\perp }}^s⟩\right|+{\lambda }_s(\parallel {\mathcal{S}}^{\ast }{\parallel }_1-\parallel \widehat{\mathcal{S}}{\parallel }_1)}}\hfill \end{array}$$

(A8)

where $\mathcal{E}:=\frac{1}{N_{\iota }}{\sum }_{i\in {\Omega }_{\iota }}{\xi }_i{\mathcal{X}}_i$, and the equality $⟨\mathcal{E},{\Delta }^s⟩=⟨\mathcal{E},{\Delta }_{{\Theta }_s^{\perp }}^s⟩$ holds. Now each item in the right hand side of (A8) will be upper bounded separately as follows. Following the idea of [36], the upper bound will be analyzed upon the following event

$$\begin{array}{c}\hfill \mathsf{E}:=\left\{\underset{1\le i\le N}{max}\left|{\xi }_i\right|\le C_{\ast }\sigma \sqrt{log\tilde{d}}\right\}\end{array}$$

(A9)

According to the tail behavior of the maximum in a sub-Gaussian sequence, it holds with an absolute constant $C_{\ast }>0$ such that $\mathbb{P}\left[\mathsf{E}\right]\ge 1-1/\left(2\tilde{d}\right)$.

Bound$\mathbf{I}$. On the event $\mathsf{E}$, we get

$$\begin{array}{c}\hfill \mathbf{I}\le \frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2\le \frac{C{\sigma }^2\left|{\Theta }_s\right|log\tilde{d}}{N}.\end{array}$$

(A10)

Bound$\mathbf{I}\mathbf{I}$. Note that according to the properties of ${\ast }_L$-TNN, we have

$$\begin{array}{cc}\hfill \parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-{\parallel \widehat{\mathcal{L}}\parallel }_{\star }& =\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-{\parallel {\mathcal{L}}^{\ast }-{\Delta }^{\iota }\parallel }_{\star }\hfill \\ & =\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-{\parallel {\mathcal{L}}^{\ast }-{\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right)-{\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\star }\hfill \\ & \le \parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-(\parallel {\mathcal{L}}^{\ast }-{\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }-\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star })\hfill \\ & =\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-(\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }+\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }-\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star })\hfill \\ & =\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star }-{\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right)\parallel }_{\star }\hfill \end{array}$$

Thus, we can bound term $\mathbf{I}\mathbf{I}$ by

$$\begin{array}{c}\hfill \mathbf{I}\mathbf{I}\le {2\parallel \mathcal{E}\parallel }_{sp}{\parallel {\Delta }^{\iota }\parallel }_{\star }+{\lambda }_{\iota }\left(\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star }-{\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right)\parallel }_{\star }\right)\end{array}$$

By letting ${\lambda }_{\iota }\ge 4{\parallel \mathcal{E}\parallel }_{sp}$, it holds that

$$\begin{array}{c}\hfill \mathbf{I}\mathbf{I}\le \frac{3}{2}{\lambda }_{\iota }\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star }\le \frac{3}{2}{\lambda }_{\iota }\sqrt{2r^{\ast }{d}_3}{\parallel {\Delta }^{\iota }\parallel }_{\mathrm{F}}.\end{array}$$

(A11)

Bound$\mathbf{I}\mathbf{I}\mathbf{I}$: Note that since ${\mathcal{S}}_{{\Theta }_s^{\perp }}^{\ast }=\mathbf{0}$, we have ${\Delta }_{{\Theta }_s^{\perp }}^s=-{\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}$, leading to

$$\begin{array}{c}\hfill \mathbf{I}\mathbf{I}\mathbf{I}\le {2\parallel \mathcal{E}\parallel }_{\infty }\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}{\parallel }_1+{\lambda }_s(\parallel {\mathcal{S}}^{\ast }{\parallel }_1-\parallel \widehat{\mathcal{S}}{\parallel }_1{)\le (2\parallel \mathcal{E}\parallel }_{\infty }-{\lambda }_s)\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}{\parallel }_1+{\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1\end{array}$$

Letting ${\lambda }_s\ge 4{\parallel \mathcal{E}\parallel }_{\infty }$ yields

$$\begin{array}{c}\hfill \mathbf{I}\mathbf{I}\mathbf{I}\le {\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1\end{array}$$

(A12)

Thus, putting Equations (A10)–(A12) together, we have the following inequality on the event $\mathsf{E}$:

$$\begin{array}{c}\hfill \frac{1}{N}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2\le \frac{3{\lambda }_{\iota }\sqrt{r^{\ast }{d}_3}}{\sqrt{2}}\parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}+{\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1+\frac{C{\sigma }^2\left|{\Theta }_s\right|log\tilde{d}}{N}\end{array}$$

(A13)

Then, we follow the line of [36] to specify a kind of Restricted Strong Convexity (RSC) for the random sampling operator formed by the design tensors $\left\{{\mathcal{X}}_i\right\}$ on a carefully chosen constrained set. The RSC will show that when the error tensors $({\Delta }^{\iota },{\Delta }^s)$ belong to the constrained set, the following relationship:

$$\begin{array}{c}\hfill \frac{1}{N}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2\ge {\kappa }_0{\parallel {\Delta }^{\iota }+{\Delta }^s\parallel }_{\Pi }^2-\tau ,\end{array}$$

(A14)

holds with an appropriate residual $\tau$ with high probability.

Before explicitly defining the constrained set, we first consider the following set where ${\Delta }^s$ should lie:

$$\begin{array}{c}\hfill \mathsf{B}({\delta }_1,{\delta }_2):=\{\mathcal{B}\in {\mathbb{R}}^{d_1\times d_2\times d_3}{|\parallel \mathcal{B}\parallel }_{\Pi }^2\le {\delta }_1^2{,\parallel \mathcal{B}\parallel }_1\le {\delta }_2\}\end{array}$$

(A15)

with two positive constants ${\delta }_1$ and ${\delta }_2$ whose values will be specified later. We also define the following set of tensor pairs:

$$\begin{array}{c}\hfill \mathsf{D}(r,\kappa ,\beta ):=\left\{{(\mathcal{A},\mathcal{B})|\parallel \mathcal{A}+\mathcal{B}\parallel }_{\Pi }^2\ge {\beta ,\parallel \mathcal{A}+\mathcal{B}\parallel }_{\infty }\le {1,\parallel \mathcal{A}\parallel }_{\star }\le \sqrt{r{d}_3}{\parallel {\mathcal{A}}_{{\Theta }_s^{\perp }}\parallel }_{\mathrm{F}}+\kappa \right\}\end{array}$$

(A16)

We then define the constrained set as the intersection:

$$\begin{array}{c}\hfill \mathsf{D}(r,\kappa ,\beta )\cap \{{\mathbb{R}}^{d_1\times d_2\times d_3}\times \mathsf{B}({\delta }_1,{\delta }_2)\}\end{array}$$

(A17)

To bound the estimation error in Equation (26), we will upper bound $\parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}$ and $\parallel {\Delta }^s{\parallel }_{\mathrm{F}}$ separately.

Note that

$$\begin{array}{c}\hfill \parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}^2=\parallel {\Delta }_{{\Theta }_s}^{\iota }{\parallel }_{\mathrm{F}}^2+\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\mathrm{F}}^2\le \parallel {\Delta }_{{\Theta }_s}^{\iota }{\parallel }_{\mathrm{F}}^2+4a^2|{\Theta }_s^{\perp }|=|{\Theta }_s|\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\Pi }^2+4a^2\left|{\Theta }_s^{\perp }\right|\end{array}$$

(A18)

and similarly

$$\begin{array}{c}\hfill \parallel {\Delta }^s{\parallel }_{\mathrm{F}}^2\le |{\Theta }_s|\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_{\Pi }^2+4a^2\left|{\Theta }_s^{\perp }\right|\end{array}$$

(A19)

We will bound $\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\Pi }^2$ and $\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_{\Pi }^2$ separately. We first bound $\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\Pi }^2$ in what follows.

Case 1: If $\parallel {\Delta }^{\iota }+{\Delta }^s{\parallel }_{\Pi }^2\le 16a^2\sqrt{\frac{128log\tilde{d}}{N_{\iota }}}$, then we use the following inequality

$$\begin{array}{c}\hfill \parallel {\Delta }^{\iota }+{\Delta }^s{\parallel }_{\Pi }^2\ge \frac{1}{2}\parallel {\Delta }^{\iota }{\parallel }_{\Pi }^2-{\parallel {\Delta }^s\parallel }_{\Pi }^2\end{array}$$

(A20)

which holds due to ${(x+y)}^2=x^2+y^2+2xy\ge x^2+y^2+2·x/\sqrt{2}·\sqrt{2}y\ge x^2+y^2-(x^2/2+2y^2)=x^2/2-y^2$.

Thus, we can upper bound $\parallel {\Delta }^{\iota }{\parallel }_{\Pi }$ with an upper bound of $\parallel {\Delta }^s{\parallel }_{\Pi }$ in Lemma A1

$$\begin{array}{c}\hfill \parallel {\Delta }^{\iota }{\parallel }_{\Pi }^2\le 16a^2\sqrt{\frac{128log\tilde{d}}{N_{\iota }}}+{\parallel {\Delta }^s\parallel }_{\Pi }^2\end{array}$$

(A21)

Case 2: Suppose $\parallel {\Delta }^{\iota }+{\Delta }^s{\parallel }_{\Pi }^2\ge 16a^2\sqrt{\frac{128log\tilde{d}}{N_{\iota }}}$. First, according to Lemma A3, it holds on the event $\mathsf{E}$ defined in Equation (A9) that

$$\begin{array}{cc}\hfill \parallel {\Delta }^{\iota }{\parallel }_{\star }& \stackrel{\left(i\right)}{\le }\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }+{\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\star }\hfill \\ & \stackrel{\left(ii\right)}{\le }4{\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\star }+\frac{2N_s}{N{\lambda }_{\iota }}(aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\hfill \\ & \stackrel{\left(iii\right)}{\le }4\sqrt{2r^{\ast }{d}_3}{\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\mathrm{F}}+\frac{2N_s}{N{\lambda }_{\iota }}(aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\hfill \\ & \stackrel{\left(iv\right)}{\le }4\sqrt{2r^{\ast }{d}_3}{\parallel {\Delta }^{\iota }\parallel }_{\mathrm{F}}+\frac{2N_s}{N{\lambda }_{\iota }}(aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\hfill \\ & \stackrel{\left(v\right)}{\le }4\sqrt{2r^{\ast }{d}_3}(\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\mathrm{F}}+\parallel {\Delta }_{{\Theta }_s}^{\iota }{\parallel }_{\mathrm{F}})+\frac{2N_s}{N{\lambda }_{\iota }}(aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\hfill \\ & \stackrel{\left(vi\right)}{\le }\sqrt{32r^{\ast }{d}_3}{\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }\parallel }_{\mathrm{F}}+a\sqrt{128r^{\ast }{d}_3\left|{\Theta }_s\right|}+\frac{2N_s}{N{\lambda }_{\iota }}(aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\hfill \end{array}$$

(A22)

where $\left(i\right)$ holds due to the triangular inequality; $\left(ii\right)$ is a direct consequence of Lemma A3, and the definition of event $\mathsf{E}$; $\left(iii\right)$ holds because $r_{tb}\left({\mathcal{P}}^{\star }\left(\mathcal{T}\right)\right)\le 2r^{\ast }$, and ${\parallel \mathcal{T}\parallel }_{\star }\le \sqrt{r_{tb}\left(\mathcal{T}\right)d_3}{\parallel \mathcal{T}\parallel }_{\mathrm{F}}$ for any $\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}$; $\left(iv\right)$ stems from the inequality $\parallel {\mathcal{P}}^{\star }\left(\mathcal{T}\right){\parallel }_{\mathrm{F}}\le {\parallel \mathcal{T}\parallel }_{\mathrm{F}}$$=\sqrt{\parallel {\mathcal{P}}^{\star }\left(\mathcal{T}\right){\parallel }_{\mathrm{F}}^2+{\parallel {\mathcal{P}}^{\perp }\left(\mathcal{T}\right)\parallel }_{\mathrm{F}}^2}$; $\left(v\right)$ is due to the triangular inequality; $\left(vi\right)$ holds since $\parallel {\Delta }^{\iota }{\parallel }_{\infty }\le 2a$.

Note that according to Lemma A1, we have with probability at least $1-2.5/\tilde{d}$:

$$\begin{array}{c}\hfill \frac{{\Delta }^s}{4a}\in \mathsf{B}\left({\delta }_1,{\delta }_2\right)with{\delta }_1=\frac{\sqrt{{\Delta }_1}}{4a},{\delta }_2=\frac{N_s}{4aN{\lambda }_s}(4a^2+8aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\end{array}$$

(A23)

where ${\Delta }_1$ is defined in Lemma A4.

Together with Equation (A22), we have

$$\begin{array}{c}\hfill \frac{1}{4a}({\Delta }^{\iota },{\Delta }^s)\in \mathsf{D}(r,\kappa ,\beta )\cap ({\mathbb{R}}^{d_1\times d_2\times d_3}\times \mathsf{B}\left({\delta }_1,{\delta }_2\right))\end{array}$$

(A24)

with the following parameters

$$\begin{array}{c}\hfill r=32r^{\ast }and\kappa =2a\sqrt{2r^{\ast }{d}_3\left|{\Theta }_s\right|}+\frac{N_s}{2aN{\lambda }_{\iota }}(aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\end{array}$$

(A25)

Then, according to Lemma A6, it holds with probability at least $1-2/\tilde{d}$ that

$$\begin{array}{c}\hfill \frac{1}{16a^2{N}_{\iota }}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2\ge \frac{1}{32a^2}{\parallel {\Delta }^{\iota }+{\Delta }^s\parallel }_{\Pi }^2-\tau (r,\kappa ,{\delta }_1,{\delta }_2)\end{array}$$

(A26)

where $\tau (r,\kappa ,{\delta }_1,{\delta }_2)$ is defined in Equation (A64).

Recall that Equation (A13) writes:

$$\begin{array}{c}\hfill \frac{1}{N}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2\le \frac{3{\lambda }_{\iota }\sqrt{r^{\ast }{d}_3}}{\sqrt{2}}{\parallel {\Delta }^{\iota }\parallel }_{\mathrm{F}}+N_s(a{\lambda }_s+\frac{C{\sigma }^{2}log\tilde{d}}{N})\end{array}$$

(A27)

Letting $\varrho :=N/N_{\iota }$, we further obtain

$$\begin{array}{cc}\hfill \frac{1}{2}{\parallel {\Delta }^{\iota }+{\Delta }^s\parallel }_{\Pi }^2& \le \frac{3\varrho {\lambda }_{\iota }}{\sqrt{2}}\sqrt{r^{\ast }{d}_3}{\parallel {\Delta }^{\iota }\parallel }_{\mathrm{F}}+C{\tau }^{\prime }\hfill \\ & \le \frac{9}{8}{\varrho }^2{\lambda }_{\iota }^2{r}^{\ast }{d}_3·4d_1{d}_2{d}_3+\frac{\parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}^2}{4d_1{d}_2{d}_3}+C{\tau }^{\prime }\hfill \\ & \le \frac{9}{2}{\varrho }^2{\lambda }_{\iota }^2{r}^{\ast }{d}_1{d}_2{d}_3^2+\frac{\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\mathrm{F}}^2}{4d_1{d}_2{d}_3}+\frac{a^2\left|{\Theta }_s\right|}{d_1{d}_2{d}_3}+C{\tau }^{\prime }\hfill \end{array}$$

(A28)

Thus, by using Equation (A20) and Lemma A1, we have

$$\begin{array}{c}\hfill \frac{\parallel {\Delta }_{{\Theta }_s^{\perp }}^{\iota }{\parallel }_{\mathrm{F}}^2}{d_1{d}_2{d}_3}\le C\left({\varrho }^2{\lambda }_{\iota }^2{r}^{\ast }{d}_1{d}_2{d}_3^2+\frac{a^2\left|{\Theta }_s\right|}{d_1{d}_2{d}_3}+{\tau }^{\prime }\right)\end{array}$$

(A29)

where

$$\begin{array}{c}\hfill {\tau }^{\prime }=\varrho N_s(a{\lambda }_s+\frac{C{\sigma }^{2}log\tilde{d}}{N})+16a^2\tau (r,\kappa ,{\delta }_1,{\delta }_2)\end{array}$$

(A30)

Note that the bound on $\parallel {\Delta }^s{\parallel }_{\Pi }$ is given in Lemma A4, and the values of ${\lambda }_{\iota }$ and ${\lambda }_s$ can be set according to Lemmas A8 and A9, respectively. Then, by putting Equations (A18), (A19) and (A52) together, and using Lemmas A8 and A9 to bound associated norms of the stochastic quantities $\mathcal{E}$, $\mathcal{W}$, and ${\mathcal{R}}_{\Sigma }$ in the error term, we can obtain the bound on $\parallel {\Delta }^{\iota }{\parallel }_{\mathrm{F}}^2+{\parallel {\Delta }^s\parallel }_{\mathrm{F}}^2$ and complete the proof. □

Appendix A.2.2. Lemmas for the Proof of Theorem 3

Lemma A1.

Letting ${\lambda }_s\ge {4(\parallel \mathcal{E}\parallel }_{\infty }+{2a\parallel \mathcal{W}\parallel }_{\infty })$, it holds that

$$\begin{array}{c}\hfill \parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1\le 3{\parallel {\Delta }_{{\Omega }_s}^s\parallel }_1+\frac{1}{N{\lambda }_s}\left(4a^2{N}_s+\sum _{i\in {\Omega }_s}{\xi }_i^2\right)\end{array}$$

(A31)

Proof of Lemma A1.

By the standard condition for optimality over a convex set, it holds that for any feasible $(\mathcal{L},\mathcal{S})$

$$\begin{array}{c}\hfill ⟨(\mathcal{L},\mathcal{S}),\partial \mathcal{F}(\widehat{\mathcal{L}},\widehat{\mathcal{S}})⟩\ge 0\end{array}$$

(A32)

which further leads to

$$\begin{array}{cc}\hfill & -\frac{2}{N}\sum _{i=1}^N(y_i-⟨{\mathcal{X}}_i,\widehat{\mathcal{L}}+\widehat{\mathcal{S}}⟩)⟨{\mathcal{X}}_i,\mathcal{L}+\mathcal{S}-\widehat{\mathcal{L}}-\widehat{\mathcal{S}}⟩\hfill \\ \hfill & +{\lambda }_{\iota }⟨\partial \parallel \widehat{\mathcal{L}}{\parallel }_{\star },\mathcal{L}-\widehat{\mathcal{L}}⟩+{\lambda }_s⟨\partial \parallel \widehat{\mathcal{S}}{\parallel }_1,\mathcal{S}-\widehat{\mathcal{S}}⟩\ge 0.\hfill \end{array}$$

(A33)

Letting $(\mathcal{L},\mathcal{S})\leftarrow (\widehat{\mathcal{L}},{\mathcal{S}}^{\ast })$, we have

$$\begin{array}{c}\hfill -\frac{2}{N}\sum _{i=1}^N(y_i-⟨{\mathcal{X}}_i,\widehat{\mathcal{L}}+\widehat{\mathcal{S}}⟩)⟨{\mathcal{X}}_i,{\Delta }^s⟩+{\lambda }_s⟨\partial \parallel \widehat{\mathcal{S}}{\parallel }_1,{\Delta }^s⟩\ge 0.\end{array}$$

(A34)

Note that

$$\begin{array}{cc}& -\frac{2}{N}\sum _{i=1}^N(y_i-⟨{\mathcal{X}}_i,\widehat{\mathcal{L}}+\widehat{\mathcal{S}}⟩)⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & \stackrel{}{=}-\frac{2}{N}\sum _{i=1}^N({\xi }_i+⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩)⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & =-\frac{2}{N}\sum _{i=1}^N{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2-\frac{2}{N}\sum _{i=1}^N{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^s⟩-\frac{2}{N}\sum _{i=1}^N⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & \stackrel{}{=}-\frac{2}{N}\sum _{i=1}^N{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2-\left(\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^s⟩+\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^s⟩\right)\hfill \\ & -\left(\frac{2}{N}\sum _{i\in {\Omega }_s}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩+\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\right)\hfill \\ & \stackrel{}{=}-\left(\frac{2}{N}\sum _{i=1}^N{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2+\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^s⟩+\frac{2}{N}\sum _{i\in {\Omega }_s}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\right)\hfill \\ & -2⟨\mathcal{E},{\Delta }^s⟩-\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & \stackrel{}{\le }-\left(\frac{2}{N}\sum _{i=1}^N{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2-\frac{1}{N}\sum _{i\in {\Omega }_s}({\xi }_i^2+{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2)-\frac{1}{N}\sum _{i\in {\Omega }_s}({⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}^2+{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2)\right)\hfill \\ & -2⟨\mathcal{E},{\Delta }^s⟩-\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & =\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{1}{N}\sum _{i\in {\Omega }_s}({⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}^2-2⟨\mathcal{E},{\Delta }^s⟩-\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & \stackrel{}{\le }\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}-2⟨\mathcal{E},{\Delta }^s⟩-\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \end{array}$$

(A35)

Thus, we have

$$\begin{array}{cc}& {\lambda }_s⟨\partial \parallel \widehat{\mathcal{S}}{\parallel }_1,\widehat{\mathcal{S}}-{\mathcal{S}}^{\ast }⟩\hfill \\ & \le \frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}-2⟨\mathcal{E},{\Delta }^s⟩-\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & \le \frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}+2\left|⟨\mathcal{E},{\Delta }^s⟩\right|+\frac{2}{N}\left|\sum _{i\in {\Omega }_{\iota }}⟨⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩{\mathcal{X}}_i,{\Delta }^s⟩\right|\hfill \\ & \stackrel{}{\le }\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}+{2\parallel \mathcal{E}\parallel }_{\infty }\parallel {\Delta }^s{\parallel }_1+2\parallel \frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩{\mathcal{X}}_i{\parallel }_{\infty }{\parallel {\Delta }^s\parallel }_1\hfill \\ & \stackrel{}{\le }\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}+{2\parallel \mathcal{E}\parallel }_{\infty }\parallel {\Delta }^s{\parallel }_1+{4a\parallel \mathcal{W}\parallel }_{\infty }{\parallel {\Delta }^s\parallel }_1\hfill \end{array}$$

(A36)

One the other hand, the definition of sub-differential indicates

$$\begin{array}{c}\hfill \parallel {\mathcal{S}}^{\ast }{\parallel }_1-{\parallel \widehat{\mathcal{S}}\parallel }_1\ge ⟨{\mathcal{S}}^{\ast }-\widehat{\mathcal{S}},\partial {\parallel \widehat{\mathcal{S}}\parallel }_1⟩\end{array}$$

(A37)

which implies

$$\begin{array}{c}\hfill {\lambda }_s(\parallel \widehat{\mathcal{S}}{\parallel }_1-\parallel {\mathcal{S}}^{\ast }{\parallel }_1)\le \frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}+{2\parallel \mathcal{E}\parallel }_{\infty }\parallel {\Delta }^s{\parallel }_1+{2\parallel \mathcal{W}\parallel }_{\infty }{\parallel {\Delta }^s\parallel }_1\end{array}$$

Also note that

$$\begin{array}{cc}\hfill \parallel \widehat{\mathcal{S}}{\parallel }_1-{\parallel {\mathcal{S}}^{\ast }\parallel }_1& =\parallel {\mathcal{S}}^{\ast }-{\Delta }^s{\parallel }_1-{\parallel {\mathcal{S}}^{\ast }\parallel }_1\hfill \\ & =\parallel {\mathcal{S}}_{{\Theta }_s}^{\ast }-({\Delta }_{{\Theta }_s^{\perp }}^s+{\Delta }_{{\Theta }_s}^s){\parallel }_1-{\parallel {\mathcal{S}}_{{\Theta }_s}^{\ast }\parallel }_1\hfill \\ & \ge \parallel {\mathcal{S}}_{{\Theta }_s}^{\ast }-{\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1-\parallel {\Delta }_{{\Theta }_s}^s{\parallel }_1-{\parallel {\mathcal{S}}_{{\Theta }_s}^{\ast }\parallel }_1\hfill \\ & =\parallel {\mathcal{S}}_{{\Theta }_s}^{\ast }{\parallel }_1+\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1-\parallel {\Delta }_{{\Theta }_s}^s{\parallel }_1-{\parallel {\mathcal{S}}_{{\Theta }_s}^{\ast }\parallel }_1\hfill \\ & =\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1-{\parallel {\Delta }_{{\Theta }_s}^s\parallel }_1\hfill \end{array}$$

(A38)

which implies

$$\begin{array}{c}\hfill {\lambda }_s(\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1-\parallel {\Delta }_{{\Theta }_s}^s{\parallel }_1)\le \frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}+{2\parallel \mathcal{E}\parallel }_{\infty }\parallel {\Delta }^s{\parallel }_1+{2\parallel \mathcal{W}\parallel }_{\infty }{\parallel {\Delta }^s\parallel }_1\end{array}$$

Since ${\lambda }_s\ge {4(\parallel \mathcal{E}\parallel }_{\infty }+{2a\parallel \mathcal{W}\parallel }_{\infty })$, we have

$$\begin{array}{c}\hfill {\lambda }_s(\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1-\parallel {\Delta }_{{\Theta }_s}^s{\parallel }_1)\le \frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{4a^2\left|{\Theta }_s\right|}{N}+\frac{{\lambda }_s}{2}(\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1+\parallel {\Delta }_{{\Theta }_s}^s{\parallel }_1)\end{array}$$

(A39)

Thus, it holds that

$$\begin{array}{c}\hfill \parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1\le 3\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1+\frac{1}{N{\lambda }_s}(4a^2|{\Theta }_s|+\sum _{i\in {\Omega }_s}{\xi }_i^2)\end{array}$$

(A40)

which complete the proof. □

Lemma A2.

It holds that

$$\begin{array}{c}\hfill \parallel \frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩{\mathcal{X}}_i{\parallel }_{\infty }\le 2a{\parallel \mathcal{W}\parallel }_{\infty }\end{array}$$

(A41)

Proof of Lemma A2.

Note that

$$\begin{array}{cc}\hfill \parallel \frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩{\mathcal{X}}_i{\parallel }_{\infty }& \stackrel{\left(i\right)}{\le }\underset{{\parallel \mathcal{T}\parallel }_1\le 1}{sup}⟨\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩{\mathcal{X}}_i,\mathcal{T}⟩\hfill \\ & \stackrel{}{\le }2a\underset{{\parallel \mathcal{T}\parallel }_1\le 1}{sup}⟨\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}\frac{⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}{2a}{\mathcal{X}}_i,\mathcal{T}⟩\hfill \\ & \stackrel{\left(ii\right)}{\le }2a\underset{\parallel {\mathcal{T}}^{\prime }{\parallel }_1\le 1}{sup}⟨\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{\mathcal{X}}_i,{\mathcal{T}}^{\prime }⟩\hfill \\ & \le {2a\parallel \mathcal{W}\parallel }_{\infty }\hfill \end{array}$$

(A42)

where $\left(i\right)$ hold since ${\parallel ·\parallel }_{\infty }$ is the dual norm of ${\parallel ·\parallel }_1$; $\left(ii\right)$ holds since $|⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩|\le \parallel {\Delta }^{\iota }{\parallel }_{\infty }\le 2a$ and the tensor ${\ell }_1$-norm ${\parallel ·\parallel }_1$ is invariant to changes in sign. □

Lemma A3.

By letting ${\lambda }_{\iota }\ge 4{\parallel \mathcal{E}\parallel }_{sp}$ and ${\lambda }_s\ge {\parallel \mathcal{E}\parallel }_{\infty }$, we have

$$\begin{array}{c}\hfill \parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }\le 3{\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\star }+\frac{2a{\lambda }_s}{{\lambda }_{\iota }}{N}_s+\frac{2}{N{\lambda }_{\iota }}\sum _{i\in {\Omega }_s}{\xi }_i^2\end{array}$$

(A43)

Proof of Lemma A3.

In Equation (A33), letting $(\mathcal{L},\mathcal{S})\leftarrow ({\mathcal{L}}^{\ast },{\mathcal{S}}^{\ast })$, we obtain

$$\begin{array}{c}\hfill -\frac{2}{N}\sum _{i=1}^N(y_i-⟨{\mathcal{X}}_i,\widehat{\mathcal{L}}+\widehat{\mathcal{S}}⟩)⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩+{\lambda }_{\iota }⟨\partial \parallel \widehat{\mathcal{L}}{\parallel }_{\star },{\Delta }^{\iota }⟩+{\lambda }_s⟨\partial \parallel \widehat{\mathcal{S}}{\parallel }_1,{\Delta }^s⟩\ge 0\end{array}$$

(A44)

First, note that

$$\begin{array}{cc}& -\frac{2}{N}\sum _{i=1}^N(y_i-⟨{\mathcal{X}}_i,\widehat{\mathcal{L}}+\widehat{\mathcal{S}}⟩)⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩\hfill \\ & \stackrel{}{=}-\frac{2}{N}\sum _{i=1}^N({\xi }_i+⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩)⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩\hfill \\ & \stackrel{}{=}-\frac{2}{N}\sum _{i=1}^N{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2-\frac{2}{N}\sum _{i=1}^N{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩\hfill \\ & \stackrel{}{=}-\frac{2}{N}\sum _{i=1}^N{⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩}^2-\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩-2⟨\mathcal{E},{\Delta }^{\iota }⟩-2⟨\mathcal{E},{\Delta }_{{\Theta }_s^{\perp }}^s⟩\hfill \end{array}$$

(A45)

Also, we have according to the convexity of ${\parallel ·\parallel }_{\star }$ and ${\parallel ·\parallel }_1$ that

$$\begin{array}{c}\hfill \parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }-\parallel \widehat{\mathcal{L}}{\parallel }_{\star }\ge ⟨{\mathcal{L}}^{\ast }-\widehat{\mathcal{L}},\partial {\parallel \widehat{\mathcal{L}}\parallel }_{\star }⟩,\mathrm{and}\parallel {\mathcal{S}}^{\ast }{\parallel }_1-{\parallel \widehat{\mathcal{S}}\parallel }_1\ge ⟨{\mathcal{S}}^{\ast }-\widehat{\mathcal{S}},\partial {\parallel \widehat{\mathcal{S}}\parallel }_1⟩\end{array}$$

(A46)

Thus, we have

$$\begin{array}{c}\hfill {\lambda }_{\iota }(\parallel \widehat{\mathcal{L}}{\parallel }_{\star }-\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star })+{\lambda }_s(\parallel \widehat{\mathcal{S}}{\parallel }_1-\parallel {\mathcal{S}}^{\ast }{\parallel }_1{)\le 2\parallel \mathcal{E}\parallel }_{sp}\parallel {\Delta }^{\iota }{\parallel }_{\star }+{2\parallel \mathcal{E}\parallel }_{\infty }{\parallel {\Delta }_{{\Theta }_s^{\perp }}^s\parallel }_1+\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2.\end{array}$$

Moreover, it is often used that

$$\begin{array}{c}\hfill \parallel \widehat{\mathcal{L}}{\parallel }_{\star }-\parallel {\mathcal{L}}^{\ast }{\parallel }_{\star }\ge \parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }-{\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\star },\end{array}$$

(A47)

Since we set ${\lambda }_{\iota }\ge 4{\parallel \mathcal{E}\parallel }_{sp}$ and ${\lambda }_s\ge 4{\parallel \mathcal{E}\parallel }_{\infty }$, we have

$$\begin{array}{cc}& {\lambda }_{\iota }(\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }-\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star })+{\lambda }_s(\parallel \widehat{\mathcal{S}}{\parallel }_1-\parallel {\mathcal{S}}^{\ast }{\parallel }_1)\hfill \\ & \le \frac{{\lambda }_{\iota }}{2}(\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star }+\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star })+\frac{{\lambda }_s}{2}{\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}\parallel }_1+\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2\hfill \end{array}$$

(A48)

where we use ${\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}=-{\Delta }_{{\Theta }_s^{\perp }}^s$. It implies

$$\begin{array}{c}\hfill \frac{{\lambda }_{\iota }}{2}\parallel {\mathcal{P}}^{\perp }\left({\Delta }^{\iota }\right){\parallel }_{\star }+{\lambda }_s\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s}{\parallel }_1+\frac{{\lambda }_s}{2}\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}{\parallel }_1\le \frac{3{\lambda }_{\iota }}{2}\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star }+{\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1+\frac{1}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2\end{array}$$

(A49)

Note that, $\parallel {\mathcal{S}}^{\ast }{\parallel }_1\le a{N}_s$. Thus, we have

$$\begin{array}{c}\hfill \parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right){\parallel }_{\star }\le 3{\lambda }_{\iota }{\parallel {\mathcal{P}}^{\star }\left({\Delta }^{\iota }\right)\parallel }_{\star }+\frac{2a{\lambda }_s}{{\lambda }_{\iota }}{N}_s+\frac{2}{N{\lambda }_{\iota }}\sum _{i\in {\Omega }_s}{\xi }_i^2\end{array}$$

(A50)

□

Lemma A4.

If $N_{\iota }\ge \tilde{d}$ and ${\lambda }_s\ge {4(\parallel \mathcal{E}\parallel }_{\infty }+{2a\parallel \mathcal{W}\parallel }_{\infty })$, then on the event $\mathsf{E}$ defined in Equation (A9), we have

$$\begin{array}{c}\hfill \parallel {\Delta }^s{\parallel }_1\le \frac{N_s}{N{\lambda }_s}(4a^2+8aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\end{array}$$

(A51)

and it holds with probability at least $1-2.5/\tilde{d}$ that

$$\begin{array}{cc}\hfill & \parallel {\Delta }^s{\parallel }_{\Pi }\le {\Delta }_1:=\hfill \\ & C(\varrho \frac{2N_s}{N_{\iota }}(4a^2+2aN{\lambda }_{\iota }+C{N}_s{\sigma }^{2}log\tilde{d})+\frac{16a{N}_s}{N{\lambda }_s}(4a^2+8aN{\lambda }_s+C{\sigma }^{2}log\tilde{d}){\mathbb{E}[\parallel \mathcal{E}\parallel }_{\infty }\left]\right)\hfill \end{array}$$

(A52)

Proof of Lemma A4.

We first prove Equation (A51), and then prove Equation (A52).

(I) The proof of Equation (A51): Recall that Lemma A1 implies

$$\begin{array}{c}\hfill \parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1\le 3{\parallel {\Delta }_{{\Omega }_s}^s\parallel }_1+\frac{1}{N{\lambda }_s}\left(4a^2{N}_s+\sum _{i\in {\Omega }_s}{\xi }_i^2\right)\end{array}$$

Then, we have on event $\mathsf{E}$:

$$\begin{array}{cc}\hfill \parallel {\Delta }^s{\parallel }_1=\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1+{\parallel {\Delta }_{{\Theta }_s}^s\parallel }_1& \stackrel{\left(i\right)}{\le }\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1+{\parallel {\Delta }_{{\Omega }_s}^s\parallel }_1\hfill \\ & \stackrel{\left(ii\right)}{\le }4{\parallel {\Delta }_{{\Omega }_s}^s\parallel }_1+\frac{1}{N{\lambda }_s}\left(4a^2{N}_s+\sum _{i\in {\Omega }_s}{\xi }_i^2\right)\hfill \\ & \stackrel{\left(iii\right)}{\le }\frac{N_s}{N{\lambda }_s}(4a^2+8aN{\lambda }_s+C{\sigma }^{2}log\tilde{d})\hfill \end{array}$$

(A53)

where $\left(i\right)$ holds due to Assumption A1.I; $\left(ii\right)$ is a direct use of Lemma A1; $\left(iii\right)$ stems from the facts that $\parallel {\Delta }^s{\parallel }_{\infty }\le 2a$ and the definition of event $\mathsf{E}$ in Equation (A9).

Thus, Equation (A51) is proved.

(II) The proof of Equation (A52): According to the optimality of $(\widehat{\mathcal{L}},\widehat{\mathcal{S}})$ to Problem (8), we have

$$\begin{array}{c}\hfill \mathcal{F}(\widehat{\mathcal{L}},\widehat{\mathcal{S}})\le \mathcal{F}(\widehat{\mathcal{L}},{\mathcal{S}}^{\ast })\end{array}$$

(A54)

which implies

$$\begin{array}{cc}& \frac{1}{N}\sum _{i=1}^{N_{\iota }}{({\xi }_i+⟨{\mathcal{X}}_i,{\Delta }^{\iota }+{\Delta }^s⟩)}^2+{\lambda }_s\parallel \widehat{\mathcal{S}}{\parallel }_1\le \frac{1}{N}\sum _{i=1}^{N_{\iota }}{({\xi }_i+⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩)}^2+{\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1\hfill \end{array}$$

(A55)

which further leads to

$$\begin{array}{c}\hfill \frac{1}{N}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2+\frac{1}{N}\sum _{i\in {\Omega }_s}{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2+\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^s⟩+\frac{2}{N}\sum _{i\in {\Omega }_s}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\\ \hfill +\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }_{{\Theta }_s^{\perp }}^s⟩+2\sum _{i\in {\Omega }_{\iota }}⟨\mathcal{E},{\Delta }_{{\Theta }_s^{\perp }}^s⟩+{\lambda }_s\parallel \widehat{\mathcal{S}}{\parallel }_1\le {\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1\end{array}$$

(A56)

Note that by using $2ab>-(1/2a^2+2b^2)$, we have

$$\begin{array}{cc}& \frac{1}{N}\sum _{i\in {\Omega }_s}{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2+\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i⟨{\mathcal{X}}_i,{\Delta }^s⟩+\frac{2}{N}\sum _{i\in {\Omega }_s}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }^s⟩\hfill \\ & \ge -(\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{2}{N}\sum _{i\in {\Omega }_s}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}^2)\hfill \end{array}$$

Thus on the event $\mathsf{E}$ defined in Equation (A9), we have

$$\begin{array}{cc}\hfill \frac{1}{N}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2& \le \left|\frac{2}{N}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩⟨{\mathcal{X}}_i,{\Delta }_{{\Theta }_s^{\perp }}^s⟩\right|+\left|2\sum _{i\in {\Omega }_{\iota }}⟨\mathcal{E},{\Delta }_{{\Theta }_s^{\perp }}^s⟩\right|\hfill \\ & +{\lambda }_s(\parallel {\mathcal{S}}^{\ast }{\parallel }_1-\parallel \widehat{\mathcal{S}}{\parallel }_1)+\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{2}{N}\sum _{i\in {\Omega }_s}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}^2\hfill \\ & \stackrel{\left(i\right)}{\le }{(4a\parallel \mathcal{W}\parallel }_{\infty }+{2\parallel \mathcal{E}\parallel }_{\infty })\parallel {\Delta }_{{\Theta }_s^{\perp }}^s{\parallel }_1+{\lambda }_s(\parallel {\mathcal{S}}^{\ast }{\parallel }_1-\parallel \widehat{\mathcal{S}}{\parallel }_1)\hfill \\ & +\frac{2}{N}\sum _{i\in {\Omega }_s}{\xi }_i^2+\frac{2}{N}\sum _{i\in {\Omega }_s}{⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}^2\hfill \\ & \stackrel{\left(ii\right)}{\le }\frac{{\lambda }_s}{2}\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}{\parallel }_1+{\lambda }_s(\parallel {\mathcal{S}}^{\ast }{\parallel }_1-\parallel \widehat{\mathcal{S}}{\parallel }_1)+\frac{2}{N}\sum _{i\in {\Omega }_s}({\xi }_i^2+{⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩}^2)\hfill \\ & \stackrel{\left(iii\right)}{\le }{\lambda }_s{\parallel {\mathcal{S}}^{\ast }\parallel }_1+\frac{2}{N}\sum _{i\in {\Omega }_s}({\xi }_i^2+4a^2)\hfill \\ & \stackrel{\left(iv\right)}{\le }\frac{2N_s}{N}(4a^2+2aN{\lambda }_{\iota }+C{N}_s{\sigma }^{2}log\tilde{d})\hfill \end{array}$$

(A57)

where $\left(i\right)$ holds due to Lemma A2; $\left(ii\right)$ holds because ${\lambda }_s\ge {4(2a\parallel \mathcal{W}\parallel }_{\infty }+{\parallel \mathcal{E}\parallel }_{\infty })$, and ${\Delta }_{{\Theta }_s^{\perp }}^s=-{\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}$; $\left(iii\right)$ holds because $\parallel \widehat{\mathcal{S}}{\parallel }_1=\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s}{\parallel }_1+\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}{\parallel }_1\ge {\parallel {\widehat{\mathcal{S}}}_{{\Theta }_s^{\perp }}\parallel }_1$, and $|⟨{\mathcal{X}}_i,{\Delta }^{\iota }⟩|\le \parallel {\Delta }^{\iota }{\parallel }_{\infty }\le 2a$; $\left(iv\right)$ holds as a consequence of $\parallel {\mathcal{S}}^{\ast }{\parallel }_{\infty }\le a$, $|{\Theta }_s|\le N_s$ (due to Assumption A1.I), and the definition of event $\mathsf{E}$.

Now, we discuss the bound of $\parallel {\Delta }^s{\parallel }_{\Pi }$ in two cases.

• Case 1. If $\parallel {\Delta }^s{\parallel }_{\Pi }^2\le \beta =4a^2\sqrt{\frac{128log\tilde{d}}{N_{\iota }}}$, then Equation (A52) holds trivially.
• Case 2. If $\parallel {\Delta }^s{\parallel }_{\Pi }^2\ge 4a^2\sqrt{\frac{128log\tilde{d}}{N_{\iota }}}$, then we have

  $$\begin{array}{c}\hfill \frac{{\Delta }^s}{2a}\in \mathsf{D}\left(\frac{N_s}{2aN{\lambda }_s}(4a^2+8aN{\lambda }_s+C{\sigma }^{2}log\tilde{d}),\delta \right)\end{array}$$

  due to the fact $\parallel {\Delta }^s/\left(2a\right){\parallel }_{\infty }\le 1$ and Equation (A51). Then according to Lemma A5, it holds with probability at least 1 - $1/{\tilde{d}}^2$ that

  $$\begin{array}{c}\hfill \frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,{\Delta }^s⟩}^2\ge \frac{1}{2}\parallel {\Delta }^s{\parallel }_{\Pi }^2-\frac{16a{N}_s}{N{\lambda }_s}(4a^2+8aN{\lambda }_s+C{\sigma }^{2}log\tilde{d}){\mathbb{E}[\parallel \mathcal{E}\parallel }_{\infty }].\end{array}$$

  (A58)
  Combing Equations (A57) and (A58) yields the bound on $\parallel {\Delta }^s{\parallel }_{\Pi }$.

□

Lemma A5.

Define the following set

$$\begin{array}{c}\hfill \mathsf{D}(\delta ,\beta ):=\left\{\mathcal{B}\in {\mathbb{R}}^{d_1\times d_2\times d_3}{|\parallel \mathcal{B}\parallel }_{\infty }\le {1,\parallel \mathcal{B}\parallel }_{\Pi }^2\ge \beta ,{\parallel \mathcal{B}\parallel }_1\le \delta \right\}\end{array}$$

(A59)

Then, it holds with probability at least $1-2/{\tilde{d}}^3$ that

$$\begin{array}{c}\hfill \frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,\mathcal{B}⟩}^2\ge \frac{1}{2}{\parallel \mathcal{B}\parallel }_{\Pi }^2-8\delta \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]\end{array}$$

(A60)

for any $\mathcal{B}\in \mathsf{D}(\delta ,\beta )$.

Proof of Lemma A5.

We prove this lemma using a standard peeling argument. First, define the following

$$\begin{array}{c}\hfill \mathsf{G}:=\left\{\exists \mathcal{B}\in \mathsf{D}(\delta ,\beta )suchthat|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,\mathcal{B}⟩-{\parallel \mathcal{B}\parallel }_{\Pi }^2|\ge \frac{1}{2}{\parallel \mathcal{B}\parallel }_{\Pi }^2+8\delta \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]\right\}\end{array}$$

We partition this set to simpler events with $l\in {\mathbb{N}}_{+}$:

$$\begin{array}{cc}\hfill {\mathsf{G}}_l:=\{\exists \mathcal{B}& \in \mathsf{D}(\delta ,\beta )\cap {\mathsf{C}}^{\prime }\left(t\right)witht\in [{\alpha }^{l-1}\beta ,{\alpha }^l\beta ]\hfill \\ & suchthat|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,\mathcal{B}⟩-{\parallel \mathcal{B}\parallel }_{\Pi }^2|\ge \frac{1}{2}{\parallel \mathcal{B}\parallel }_{\Pi }^2+8\delta \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]\}\hfill \end{array}$$

(A61)

Note that according to Lemma A7, we have with $t\in [{\alpha }^{l-1}\beta ,{\alpha }^l\beta )$:

$$\begin{array}{c}\hfill \mathbb{P}\left[{\mathsf{G}}_l\right]\le \mathbb{P}\left[\underset{\mathcal{B}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}⟨{\mathcal{X}}_i,\mathcal{B}⟩-{\parallel \mathcal{B}\parallel }_{\Pi }^2|\ge \frac{1}{2\alpha }{\alpha }^l\beta +8\delta \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]\right]\le exp(-\frac{n{\left({\alpha }^l\beta \right)}^2}{32{\alpha }^2})\end{array}$$

Thus, we have

$$\begin{array}{cc}\hfill \mathbb{P}\left[\mathsf{G}\right]\le \mathbb{P}\left[\bigcup _{l=1}^{\infty }{\mathsf{G}}_l\right]\le \sum _{l=1}^{\infty }\mathbb{P}\left[{\mathsf{G}}_l\right]& \le \sum _{l=1}^{\infty }exp(-\frac{N_{\iota }{\left({\alpha }^l\beta \right)}^2}{32{\alpha }^2})\hfill \\ & \stackrel{}{=}exp(-\frac{N_{\iota }{\beta }^2}{32})+\sum _{l=2}^{\infty }exp(-\frac{N_{\iota }{\beta }^2}{32}{\alpha }^{2(l-1)})\hfill \\ & \stackrel{\left(i\right)}{\le }exp(-\frac{N_{\iota }{\beta }^2}{32})+\sum _{l=2}^{\infty }exp(-\frac{N_{\iota }{\beta }^2}{32}·2(l-1)log\alpha )\hfill \\ & \stackrel{}{\le }exp(-\frac{N_{\iota }{\beta }^2}{32})+\frac{exp(-\frac{N_{\iota }{\beta }^2}{16}log\alpha )}{1-exp(-\frac{N_{\iota }{\beta }^2}{16}log\alpha )}\hfill \end{array}$$

(A62)

where $\left(i\right)$ is due to $x\ge logx$ for positive x. By setting $\alpha =e$ and recalling the value of $\beta =\sqrt{\frac{128log\tilde{d}}{N_{\iota }}}$, the lemma is proved. □

Lemma A6.

For any $(\mathcal{A},\mathcal{B})\in \mathsf{C}(r,\kappa ,\beta )\cap {\mathbb{R}}^{d_1\times d_2\times d_3}\times \mathsf{B}({\delta }_1,{\delta }_2)$, it holds with probability at least $1-2/\tilde{d}$ that

$$\begin{array}{c}\hfill \frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,\mathcal{A}+\mathcal{B}⟩}^2\ge \frac{1}{2}{\parallel \mathcal{A}+\mathcal{B}\parallel }_{\Pi }^2-\tau (r,\kappa ,{\delta }_1,{\delta }_2)\end{array}$$

(A63)

where

$$\begin{array}{c}\hfill \tau (r,\kappa ,{\delta }_1,{\delta }_2)=4(16\alpha +1)r{d}_3|{\Theta }_s^{\perp }\left|\mathbb{E}\right[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}{]}^2+8\kappa \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}]+8{\delta }_2\mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]+4{\delta }_1^2\end{array}$$

(A64)

Proof of Lemma A6.

The proof is very similar to that of Lemma A5, and we simply omit it. □

Lemma A7.

Define the set

$$\begin{array}{c}\hfill {\mathsf{C}}^{\prime }\left(t\right):=\{\mathcal{T}\in {\mathbb{R}}^{d_1\times d_2\times d_3}{|\parallel \mathcal{T}\parallel }_{\Pi }^2\le {t,\parallel \mathcal{T}\parallel }_{\infty }\le 1\}\end{array}$$

(A65)

and

$$\begin{array}{c}\hfill Z_t:=\underset{\mathcal{T}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}\left|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,\mathcal{T}⟩}^2-{\parallel \mathcal{T}\parallel }_{\Pi }^2\right|.\end{array}$$

(A66)

Then, it holds that

$$\begin{array}{c}\hfill \mathbb{P}[Z_t\ge \mathbb{E}\left[Z_t\right]+\frac{t}{4\alpha }]\le exp(-\frac{N_{\iota }{t}^2}{128{\alpha }^2}),and\mathbb{E}\left[Z_t\right]\le 8\mathbb{E}\left[\underset{\mathcal{T}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}\left|⟨{\mathcal{R}}_{\Sigma },\mathcal{T}⟩\right|\right]\end{array}$$

(A67)

Proof of Lemma A7.

First, we study the tail behavior of $Z_t$ by directly using the Massart’s inequality in Theorem 14.2 of [64]. According to the Massart’s inequality, it holds for any $s>0$

$$\begin{array}{c}\hfill \mathbb{P}[Z_t\ge \mathbb{E}\left[Z_t\right]+s]\le exp(-\frac{N_{\iota }{s}^2}{8})\end{array}$$

(A68)

By letting $s=t/\left(4\alpha \right)$, the first inequality in Equation (A67) is proved.

Then, we will upper bound the expectation of $Z_t$. By standard symmetrization argument [65], we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[Z_t\right]& =\mathbb{E}\left[\underset{\mathcal{T}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}\left|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{⟨{\mathcal{X}}_i,\mathcal{T}⟩}^2-\mathbb{E}{⟨\mathcal{X},\mathcal{T}⟩}^2\right|\right]\hfill \\ & =2\mathbb{E}\left[\underset{\mathcal{T}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}\left|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{\epsilon }_i{⟨{\mathcal{X}}_i,\mathcal{T}⟩}^2\right|\right]\hfill \end{array}$$

(A69)

where ${\epsilon }_i$’s are i.i.d. Randemacher variables. Further, according to the contraction principle [66], it holds that

$$\begin{array}{cc}\hfill \mathbb{E}\left[Z_t\right]& \le 8\mathbb{E}\left[\underset{\mathcal{T}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}\left|\frac{1}{N_{\iota }}\sum _{i\in {\Omega }_{\iota }}{\epsilon }_i⟨{\mathcal{X}}_i,\mathcal{T}⟩\right|\right]=8\mathbb{E}\left[\underset{\mathcal{T}\in {\mathsf{C}}^{\prime }\left(t\right)}{sup}\left|⟨{\mathcal{R}}_{\Sigma },\mathcal{T}⟩\right|\right]\hfill \end{array}$$

(A70)

In the following, we consider the two cases:

Case 1. Consider $\mathcal{T}\in \mathsf{D}(\delta ,\beta )\cap {\mathsf{C}}^{\prime }\left(t\right)$, we have

$$\begin{array}{c}\hfill \mathbb{E}\left[Z_t\right]\le 8\mathbb{E}\left[\underset{\mathcal{T}}{sup}\left|⟨{\mathcal{R}}_{\Sigma },\mathcal{T}⟩\right|\right]\le 8\mathbb{E}\left[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }{\parallel \mathcal{T}\parallel }_1\right]\le 8\delta \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }].\end{array}$$

(A71)

By letting $s=t/\left(2\alpha \right)$ in Equation (A68), we obtain

$$\begin{array}{c}\hfill \mathbb{P}\left[Z_t\ge 8\delta \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]+\frac{t}{2\alpha }\right]\le exp(-\frac{N_{\iota }{t}^2}{32{\alpha }^2})\end{array}$$

(A72)

when $\mathcal{T}\in \mathsf{D}(\delta ,\beta )\cap {\mathsf{C}}^{\prime }\left(t\right)$.

Case 2. Consider $\mathcal{T}=\mathcal{A}+\mathcal{B}$, where $(\mathcal{A},\mathcal{B})\in \mathsf{C}(r,\kappa ,\beta )$, $\mathcal{B}\in \mathsf{B}({\delta }_1,{\delta }_2)$, and ${\parallel \mathcal{T}\parallel }_{\Pi }^2\le t$. The goal in this case is to upper bound

$$\begin{array}{cc}\hfill \mathbb{E}\left[Z_t\right]& \le 8\mathbb{E}\left[\underset{\mathcal{A},\mathcal{B}}{sup}\left|⟨{\mathcal{R}}_{\Sigma },\mathcal{A}+\mathcal{B}⟩\right|\right]\hfill \\ & \stackrel{\left(i\right)}{\le }8\mathbb{E}\left[\underset{\mathcal{A}}{sup}\left|⟨{\mathcal{R}}_{\Sigma },\mathcal{A}⟩\right|\right]+8\mathbb{E}\left[\underset{\mathcal{B}}{sup}\left|⟨{\mathcal{R}}_{\Sigma },\mathcal{B}⟩\right|\right]\hfill \\ & \stackrel{\left(ii\right)}{\le }8\mathbb{E}\left[\underset{\mathcal{A}}{sup}\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}{\parallel \mathcal{A}\parallel }_{\star }\right]+8\mathbb{E}\left[\underset{\mathcal{B}}{sup}\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }{\parallel \mathcal{B}\parallel }_1\right]\hfill \\ & \stackrel{\left(iii\right)}{\le }8\mathbb{E}\left[\underset{\mathcal{A}}{sup}\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}{\parallel \mathcal{A}\parallel }_{\star }\right]+8{\delta }_2\mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]\hfill \end{array}$$

(A73)

where $\left(i\right)$ holds as a property of the sup operation; $\left(ii\right)$ holds due to the definition of dual norm; $\left(iii\right)$ stems from the condition $\mathcal{B}\in \mathsf{B}({\delta }_1,{\delta }_2)$.

It remains to upper bound ${\parallel \mathcal{A}\parallel }_{\star }$ in Equation (A72). First, according to the definition of $\mathsf{C}(r,\kappa ,\beta )$, we have

$$\begin{array}{c}\hfill {\parallel \mathcal{A}\parallel }_{\star }\le \sqrt{r{d}_3}{\parallel {\mathcal{A}}_{{\Theta }_s^{\perp }}\parallel }_{\mathrm{F}}+\kappa \end{array}$$

(A74)

We also have

$$\begin{array}{c}\hfill {\parallel \mathcal{A}\parallel }_{\Pi }\stackrel{\left(i\right)}{\le }{\parallel \mathcal{A}+\mathcal{B}\parallel }_{\Pi }+{\parallel \mathcal{B}\parallel }_{\Pi }\stackrel{\left(ii\right)}{\le }\sqrt{t}+{\delta }_1.\end{array}$$

(A75)

where $\left(i\right)$ holds due to the triangular inequality, and $\left(ii\right)$ is a result of conditions ${\parallel \mathcal{T}\parallel }_{\Pi }^2\le t$ and $\mathcal{B}\in \mathsf{B}({\delta }_1,{\delta }_2)$. Since Assumption A1.II indicates ${\parallel \mathcal{A}\parallel }_{\Pi }^2=|{\Theta }_s^{\perp }{|}^{-1}{\parallel {\mathcal{A}}_{{\Theta }_s^{\perp }}\parallel }_{\mathrm{F}}^2$, combing Equations (A74) and (A75) further yields an upper bound on ${\parallel \mathcal{A}\parallel }_{\star }$ as follows:

$$\begin{array}{c}\hfill {\parallel \mathcal{A}\parallel }_{\star }\le \sqrt{r{d}_3\left|{\Theta }_s^{\perp }\right|}(\sqrt{t}+{\delta }_1)+\kappa \end{array}$$

which further leads to

$$\begin{array}{c}\hfill 8\mathbb{E}\left[\underset{\mathcal{A}}{sup}\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}{\parallel \mathcal{A}\parallel }_{\star }\right]\le 8\mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}](\sqrt{r{d}_3\left|{\Theta }_s^{\perp }\right|}(\sqrt{t}+{\delta }_1)+\kappa )\end{array}$$

(A76)

The application of $2\sqrt{ab}\le a/c+bc$ is also used to further relax the above inequality:

$$\begin{array}{cc}\hfill & 8\mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}\left]\right(\sqrt{r{d}_3\left|{\Theta }_s^{\perp }\right|}\sqrt{t}\le \frac{t}{4\alpha }+64\alpha r{d}_3|{\Theta }_s^{\perp }\left|\mathbb{E}\right[\parallel {\mathcal{R}}_{\Sigma }{{\parallel }_{sp}]}^2\hfill \\ & 8\mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}\left]\right(\sqrt{r{d}_3\left|{\Theta }_s^{\perp }\right|}{\delta }_1\le 4{\delta }_1^2+4r{d}_3|{\Theta }_s^{\perp }\left|\mathbb{E}\right[\parallel {\mathcal{R}}_{\Sigma }{{\parallel }_{sp}]}^2\hfill \end{array}$$

(A77)

Thus, putting things in Equations (A72), (A76) and (A77) together, we obtain

$$\begin{array}{c}\hfill \mathbb{E}\left[Z_t\right]\le \frac{t}{4\alpha }+\underset{=:\tau }{\underbrace{4(16\alpha +1)r{d}_3|{\Theta }_s^{\perp }\left|\mathbb{E}\right[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}{]}^2+8\kappa \mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{sp}]+8{\delta }_2\mathbb{E}[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }]+4{\delta }_1^2}}\end{array}$$

which further gives

$$\begin{array}{c}\hfill \mathbb{P}\left[Z_t\ge \frac{t}{2\alpha }+\tau \right]\le exp(-\frac{N_{\iota }{t}^2}{128})\end{array}$$

(A78)

for $\mathcal{T}=\mathcal{A}+\mathcal{B}$, where $(\mathcal{A},\mathcal{B})\in \mathsf{C}(r,\kappa ,\beta )$, $\mathcal{B}\in \mathsf{B}({\delta }_1,{\delta }_2)$, and ${\parallel \mathcal{T}\parallel }_{\Pi }^2\le t$. □

Lemma A8.

Under Assumption A1, there exists an absolute constant $C>0$ such that the following bounds on the tensor spectral norm of stochastic tensors $\mathcal{E}$ and ${\mathcal{R}}_{\Sigma }$ hold:

(I)

For tensor $\mathcal{E}$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left[{\parallel \mathcal{E}\parallel }_{sp}\le C\sigma max\left\{\sqrt{\frac{t+log\tilde{d}}{\varrho N(d_1\wedge d_2)}}+\frac{log(d_1\wedge d_2)(t+log\tilde{d})}{N}\right\}\right]\le 1-e^{-t}\end{array}$$

(A79)

(II)

For tensor ${\mathcal{R}}_{\Sigma }$, we have

$$\begin{array}{c}\hfill \mathbb{E}\left[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }\right]\le \left(\sqrt{\frac{log\tilde{d}}{N_{\iota }(d_1\wedge d_2)}}+\frac{{log}^2\tilde{d}}{N_{\iota }}\right)\end{array}$$

(A80)

Proof of Lemma A8.

Equation (A79) can be seen as a special case of Lemma 8 in [18] when $k=1$. Equation (A80) can be proved very similarly to Equation (A79) followed by tricks used in proof of Lemma 6 in [51]. We omit the details due to the high similarity. □

Lemma A9.

Under Assumption A1, there exists an absolute constant $C>0$ such that the following bounds on the $l_{\infty }$-norm of stochastic tensors $\mathcal{E}$, $\mathcal{W}$, and ${\mathcal{R}}_{\Sigma }$ hold:

(I) 

For tensor $\mathcal{E}$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left[{\parallel \mathcal{E}\parallel }_{\infty }\le C\sigma \left(\sqrt{\frac{t+log\tilde{d}}{\varrho N{d}_1{d}_2{d}_3}}+\frac{t+logd}{N}\right)\right]\le 1-e^{-t}\end{array}$$

(A81)

$$\begin{array}{c}\hfill \mathbb{E}\left[{\parallel \mathcal{E}\parallel }_{\infty }\right]\le C\sigma \left(\sqrt{\frac{log\tilde{d}}{\varrho N{d}_1{d}_2{d}_3}}+\frac{logd}{N}\right)\end{array}$$

(A82)

(II) 

For tensor $\mathcal{W}$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left[{\parallel \mathcal{W}\parallel }_{\infty }\le C\left(\frac{1}{\varrho d_1{d}_2{d}_3}+\sqrt{\frac{t+log\tilde{d}}{\varrho N{d}_1{d}_2{d}_3}}+\frac{t+log\tilde{d}}{N}\right)\right]\le 1-e^{-t}\end{array}$$

(A83)

$$\begin{array}{c}\hfill \mathbb{E}\left[{\parallel \mathcal{W}\parallel }_{\infty }\right]\le C\left(\frac{1}{\varrho d_1{d}_2{d}_3}+\sqrt{\frac{log\tilde{d}}{\varrho N{d}_1{d}_2{d}_3}}+\frac{log\tilde{d}}{N}\right)\end{array}$$

(A84)

(III) 

For tensor ${\mathcal{R}}_{\Sigma }$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }\le C\left(\sqrt{\frac{t+log\tilde{d}}{N_{\iota }{d}_1{d}_2{d}_3}}+\frac{t+logd}{N_{\iota }}\right)\right]\le 1-e^{-t}\end{array}$$

(A85)

$$\begin{array}{c}\hfill \mathbb{E}\left[\parallel {\mathcal{R}}_{\Sigma }{\parallel }_{\infty }\right]\le C\sigma \left(\sqrt{\frac{log\tilde{d}}{N_{\iota }{d}_1{d}_2{d}_3}}+\frac{log\tilde{d}}{N_{\iota }}\right)\end{array}$$

(A86)

Proof of Lemma A9.

Since this lemma can be straightforwardly proved in the same way as Lemma 10 in [36], we omit the proof. □