Tensor Completion Based on Triple Tubal Nuclear Norm

Many tasks in computer vision suffer from missing values in tensor data, i.e., multi-way data array. The recently proposed tensor tubal nuclear norm (TNN) has shown superiority in imputing missing values in 3D visual data, like color images and videos. However, by interpreting in a circulant way, TNN only exploits tube (often carrying temporal/channel information) redundancy in a circulant way while preserving the row and column (often carrying spatial information) relationship. In this paper, a new tensor norm named the triple tubal nuclear norm (TriTNN) is proposed to simultaneously exploit tube, row and column redundancy in a circulant way by using a weighted sum of three TNNs. Thus, more spatial-temporal information can be mined. Further, a TriTNN-based tensor completion model with an ADMM solver is developed. Experiments on color images, videos and LiDAR datasets show the superiority of the proposed TriTNN against state-of-the-art nuclear norm-based tensor norms.


Introduction
In recent decades, the rapid progress in multi-linear algebra [1] has provided a firm theoretical foundation for many applications in computer vision [2], data mining [3], machine learning [4], signal processing [5], and many other areas.Benefiting from its multi-way nature, the tensor has power against the vector and matrix in exploiting multi-way information in multi-modal data, like color images [6], videos [7], hyper-spectral images [8], functional magnetic resonance imaging [9], traffic volume data [10], etc.In many computer vision tasks, the data, like color images or videos, may be moderately redundant, then it can be interpreted by fewer latent factors [11].The low-rank tensor provides a suitable model for such data [12].The two most well-known low-rank tensor models are the low-CP-rank model [13], which tries to interpret a tensor in the fewest rank-one tensors [14], and the low-Tucker-rank model [15], which seeks a tensor proxy that is simultaneously low-rank along each mode.
In many computer vision applications, like image or video inpainting, one has to tackle the missing values in the observed data tensor due to many circumstances [2,16], including failure of sensors, errors or loss in communication, occlusions or noise in the environment, etc.However, it is obviously unable to fill in the missing entries perfectly since they can take arbitrary values without other priors taken into consideration.The most adopted prior is the low-rank prior assuming the underlying data tensor has low rank.Low-rank tensor completion [2,17] seeks a low-rank tensor to fit the underlying data tensor.It has been a hot research topic due its wide use [18].In low-rank tensor recovery, the rank minimization problem (RMP) is often formulated [2].However, the general rank minimization problem and most tensor problems are NP-hard [19,20].To obtain polynomial-time algorithms, many different tensor rank surrogates have been proposed [2,7,17,[21][22][23][24] to substitute the rank functions in RMP.Surrogates of the tensor CP rank and Tucker rank have been broadly studied [7,17,23,[25][26][27][28][29].
Recently, a novel low-rank tensor model called the low-tubal-rank model was proposed [22,30].The core of it is to model the 3D data as a tensor that has low tubal-rank [31], which is defined through a new tensor singular value decomposition (t-SVD) [1,32].It has been successfully used in modeling multi-way real-world data, such as color images [6], videos [33], seismic data [34], WiFi fingerprint [35], MRI imaging [22], traffic volume data [36], etc.As pointed out in [37], compared with other tensor models, the low-tubal-rank tensor model is superior in capturing a "spatial-shifting" correlation, which is ubiquitous in real-world data arrays.
This paper focuses on low-tubal-rank models for tensor completion.The recently-proposed tensor tubal nuclear norm (TNN) [30] based on t-SVD has shown superiority in imputing missing values in 3D visual data, like color images and videos.Its power lies in exploiting tube (often carrying temporal/channel information) redundancy in a circulant way while preserving the row and column (often carrying spatial information) relationship.A simple and successful variant of TNN, dubbed the twist tubal nuclear norm (t-TNN) [16], instead exploits row redundancy in a circulant way while keeping the tube relationship.However, both of them only exploit one kind of redundancy in a circulant way.In this paper, a new tensor norm dubbed the tensor triple tubal nuclear norm (TriTNN) is proposed to simultaneously exploit the row, column and tube redundancy while preserving the relative tube, row and column relationship.Based on the proposed TriTNN, a tensor completion model is studied and optimized by alternating direction multiplier methods (ADMM) [38].Experimental results on color images, videos and LiDAR datasets demonstrate that the proposed TriTNN has better performances than other state-of-the-art nuclear norm-based tensor norms.
The paper is organized as follows.Some notations and preliminaries are presented in Section 2. The TriTNN is proposed following the introductions of the most related works in Section 3. The problem formulation and the proposed ADMM algorithm are shown in Section 4. Experimental results are reported in Section 5. We conclude this work in Section 6.

Notations and Preliminaries
In this section, the notations and the basic definitions are introduced.

Notations
Vectors are denoted by bold lower case letters, e.g., v ∈ R n , matrices are denoted by bold upper case letters, e.g., M ∈ R n 1 ×n 2 , and tensors are denoted by calligraphy letters, e.g., T ∈ R n 1 ×n 2 ×n 3 .Given a third-order tensor, its fiber is defined as a 1D vector obtained by fixing all indices, but one, and its slice is a 2D matrix defined by fixing all but two indices.Given a 3D tensor T , T ijk denotes the entry with index (i, j, k); T (k) := T (:, :, k) denotes the k-th frontal slice.T denotes the tensor after performing the fast Fourier transformation along the tube fibers of T .Notations dft3(•) and idft3(•) are used to represent the discrete Fourier transformation (DFT) and inverse discrete Fourier transformation (IDFT) along the tube fibers of 3D tensors.
Given a matrix M ∈ R n 1 ×n 2 , its nuclear norm is defined as are the singular values of M in non-ascending order.The inner product between two 3D tensors T 1 , T 2 ∈ R n 1 ×n 2 ×n 3 is defined as T 1 , T 2 := ∑ ijk T 1 (i, j, k)T 2 (i, j, k).The Frobenius norm of a tensor T ∈ R n 1 ×n 2 ×n 3 is defined as
the block diag matrix and its opposite operation: T (1)  . . .
and the block circulant matrix as follows: Based on the five operators defined above, we are able to give the definition of the tensor t-product.
Definition 1 (t-product [1]).Let T 1 ∈ R n 1 ×n 2 ×n 3 and T 2 ∈ R n 2 ×n 4 ×n 3 .The t-product of T 1 and T 2 is a tensor T of size n 1 × n 4 × n 3 : Viewing a 3D tensor T ∈ R n 1 ×n 2 ×n 3 as an n 1 × n 2 matrix of tubes, the tensor t-product is analogous to the matrix multiplication by replacing scalar multiplication with the vector circular convolution between the tubes, as follows: where • denotes the circular convolution [1] between two tube vector x, y ∈ R n 3 defined as: Due to the relationship between the circular convolution and the DFT, the t-product in the original domain is equivalent to matrix multiplication of the frontal slices in the Fourier domain [1], i.e., The tensor transpose, identity tensor, f-diagonal tensor and orthogonal tensor are further defined.
Definition 2 (Tensor transpose [1]).Let T be a tensor of size n 1 × n 2 × n 3 ; then T is the n 2 × n 1 × n 3 tensor obtained by transposing each of the frontal slices and then reversing the order of transposed frontal Slices 2 through n 3 .
Definition 3 (Identity tensor [1]).The identity tensor I ∈ R n 1 ×n 1 ×n 3 is a tensor whose first frontal slice is the n 1 × n 1 identity matrix, and all other frontal slices are zero.A Definition 4 (F-diagonal tensor [1]).A tensor is called f-diagonal if each frontal slice of the tensor is a diagonal matrix.
Definition 5 (Orthogonal tensor [1]).A tensor Q ∈ R n 1 ×n 1 ×n 3 is orthogonal if it satisfies the following relationship: Based on the concepts defined above, the tensor singular value decomposition (t-SVD) and the tensor tubal rank are established as follows.
Definition 6 (Tensor singular value decomposition and tensor tubal-rank [31]).Any tensor T ∈ R n 1 ×n 2 ×n 3 can be decomposed as: where U ∈ R n 1 ×n 1 ×n 3 and V ∈ R n 2 ×n 2 ×n 3 are orthogonal tensors and S is a rectangular f -diagonal tensor of size n 1 × n 2 × n 3 .
The tensor tubal rank of T is defined to be the number of non-zero tubes of S in Equation (4), i.e., r tubal (T ) := ∑ i 1(S(i, i, :) = 0), (5) where 1(•) is an indicator function whose value is one if the input condition is satisfied, and zero otherwise.
The t-SVD is illustrated in Figure 1.It can be computed efficiently by FFT and IFFT in the Fourier domain according to Equation (3).For more details, see [1].

The Triple Tubal Nuclear Norm
In this section, we will define the triple tensor tubal nuclear norm, before which the most related norms, i.e., tubal nuclear norm and twist tubal nuclear norm, are introduced first.

Tubal Nuclear Norm
Based on the preliminaries introduced in Section 2, the tubal nuclear norm is defined as follows: Definition 7 (Tubal nuclear norm [31]).The tubal nuclear norm (TNN) T of a 3D tensor T is defined as the nuclear norm of the block diagonal matrix of T (the Fourier domain version of T ), i.e., From Definition 7, we can compute the TNN of a tensor efficiently through first conducting FFT along the tube direction and summing the nuclear norms of each frontal slice.Given a tensor T ∈ R n 1 ×n 2 ×n 3 , the computation cost of T is O(n 1 n 2 n 3 (n 3 + log n 3 )).Since a block circulant matrix can be block diagonalized through the Fourier transform [1], we obtain: where ⊗ denotes the Kronecker product [14], F n is the n × n discrete Fourier transform matrix and I n is an n × n identity matrix.Note that (F The tubal nuclear norm has been used as a convex relaxation of the tensor tubal-rank for tensor completion, tensor robust principle component analysis (TRPCA) and outlier robust tensor principle component analysis (OR-TPCA) [6,30,36,39].In optimization over TNN, one often needs to compute the proximal operator [40] of TNN defined as: In [3], a closed-form expression of S τ (•) is given as follows: ).For a 3D tensor T ∈ R n 1 ×n 2 ×n 3 with reduced t-SVD T = U * S * V , where U ∈ R n 1 ×r×n 3 and V ∈ R n 2 ×r×n 3 are orthogonal tensors and S ∈ R r×r×n 3 is the f-diagonal tensor of singular tubes, the proximal operator S τ (•) at T 0 can be computed through the following equation:

Twist Tubal Nuclear Norm
The twist tubal nuclear norm [16] is related to a pair of tensor operations named column twist and column squeeze as follows.
Then, we give the definition of twist nuclear norm as follows: Definition 10 (Twist tensor nuclear norm [16]).The twist tensor nuclear norm (t-TNN) based on the t-SVD framework is defined as follows: From the above definition, t-TNN can be computed efficiently by first twisting the original tensor and then computing the TNN.Given a tensor T ∈ R n 1 ×n 2 ×n 3 , the computation cost of ).The t-TNN has attained significant improvement against the TNN in video inpainting [16].Since the t-TNN is essentially a TNN, the proximal operator of it can be derived from the proximal operator of TNN as follows:

A Circular Interpretation of TNN and t-TNN
In this subsection, an illustration of TNN and t-TNN in a circular way [16], which motivates the proposal of TriTNN, will be given.For a tensor T ∈ R n 1 ×n 2 ×n 3 , we define the operation called circulant block matricization of T [41] in the following equation: where circ(x) denotes an n 3 × n 3 matrix with x ∈ R n 3 in the following way: By the permutation operation, there exist two so-called stride permutation matrices [42] P 1 and P 2 , such that the following relationship between circ(T ) and bcirc(T ) holds: Since the matrix nuclear norm is permutation invariant, it holds that [16]: As an example, Figure 3a,b intuitively shows the relationships between the original tensor, the column twist tensor, the block circulant matrix and the circulant block matricization of a tensor T ∈ R 3×3×3 .As illustrated in Subplots (a) and (b) of Figure 3, from the circulant perspective, TNN essentially exploits the tube redundancy in a circulant way while keeping the row and column relationship, and t-TNN essentially exploits the row redundancy in a circulant way while preserving the tube and column relationship [16].In computer vision applications, the row and column of a data tensor (like color images or videos) often carry spatial information, and the tube often carries temporal or channel information.From the computational perspective, FFT is operated along the tube direction to compute TNN, while t-TNN needs FFT along the row direction.4, from the circulant persp essentially exploits the tube redundancy in a circulant way while keeping the row relationship and t-TNN essentially exploits the row redundancy in a circulant way whil the tube and column relationship [17].In computer vision applications, the row and colu tensor (like color images or videos) often carry spatial information and the tube often carr or channel information.From the computational perspective, FFT is operated along the tu to compute TNN while t-TNN needs FFT along the row direction.

The Proposed Row Twist Tubal Nuclear Norm and Triple Tubal Nuclear Norm
As discussed above, TNN and t-TNN need FFT along the the tube and row direction Note that, for real-word visual data, like color images, the row and column carry ho information and they are better to be treated equally.A simple operation similar to the c and column squeeze, called row twist and row squeeze, respectively, are defined.
Definition 11 (Tensor Row Twist and Row Squeeze).Let T ∈ R n 1 ×n 2 ×n 3 , then the row twist exploits the row redundancy in a circulant way while keeping the tube and column relationship and  4, from the circulant essentially exploits the tube redundancy in a circulant way while keeping the relationship and t-TNN essentially exploits the row redundancy in a circulant way the tube and column relationship [17].In computer vision applications, the row and tensor (like color images or videos) often carry spatial information and the tube ofte or channel information.From the computational perspective, FFT is operated along to compute TNN while t-TNN needs FFT along the row direction.

The Proposed Row Twist Tubal Nuclear Norm and Triple Tubal Nuclear Norm
As discussed above, TNN and t-TNN need FFT along the the tube and row dire Note that, for real-word visual data, like color images, the row and column ca information and they are better to be treated equally.A simple operation similar to and column squeeze, called row twist and row squeeze, respectively, are defined.
Definition 11 (Tensor Row Twist and Row Squeeze).Let T ∈ R n 1 ×n 2 ×n 3 , then the row T − → is a tensor of size n 3 × n 2 × n 1 whose horizontal slice T 1 (k, :, :) = T (k) .Correspondin exploits the column redundancy in a circulant way while keeping the tube and row relationship.

The Proposed Row Twist Tubal Nuclear Norm and Triple Tubal Nuclear Norm
As discussed above, TNN and t-TNN need FFT along the tube and row direction, respectively.Note that, for real-word visual data, like color images, the row and column carry homogeneous information, and they are better treated equally.A simple operation similar to the column twist and column squeeze, called row twist and row squeeze, respectively, is defined.Definition 11 (Tensor row twist and row squeeze).Let T ∈ R n 1 ×n 2 ×n 3 , then the row twist tensor T 1 = T − → is a tensor of size n 3 × n 2 × n 1 whose horizontal slice T 1 (k, :, :) = T (k) .Correspondingly, the row squeeze tensor of T 1 , i.e., T = T 1 ← − , can be obtained by the reverse process, i.e., T (k) = RowSqueeze(T 1 (k, :, :)).See Figure 4.
As illustrated in Subplot (c) of Figure 3, from the circulant perspective, rt-TNN essentially exploits the column redundancy in a circulant way while keeping the row and tube relationship.From the computational perspective, FFT is operated along the column direction to compute rt-TNN.The proximal operator of rt-TNN can be derived from the proximal operator of TNN as follows: It should be noted that each of TNN, t-TNN and rt-TNN only exploits one type of redundancy, i.e., the tube, row and column redundancies, in a circulant way.Real-world data may have more than one type of redundancy, and it is beneficial to exploit such a property.To simultaneously exploit the tube, row and column redundancy in a circulant way while keeping other relationships, we simply combine the TNN, t-TNN and rt-TNN to get the triple tubal nuclear norm.

Definition 13 (Triple tubal nuclear norm).
The triple tubal nuclear norm (TriTNN) is defined as a weighted sum of its tubal nuclear norm, column twist tubal nuclear norm and its row twist tubal nuclear norm, i.e., where λ 1 , λ 2 and λ 3 are positive weights satisfying: From the above definition, the computation of T can be divided into computations of TNN, t-TNN and rt-TNN, which has the following computational complexity: Due to the coupling of three tubal nuclear norms, it is very difficult to derive a closed-form expression of the proximal operator of TriTNN.

Problem Formulation
Let L * ∈ R n 1 ×n 2 ×n 3 be the true, but unknown tensor to be completed.Suppose only a small fraction of its entries are observed and the observations are corrupted by small dense noise.Let T ∈ R n 1 ×n 2 ×n 3 denote the observed noisy tensor of L * .Then, we have the following observation model: where E ∈ R n 1 ×n 2 ×n 3 is the noise tensor with element-wisely i.i.d.Gaussian noise, denotes the element-wise multiplication and O ∈ R n 1 ×n 2 ×n 3 denotes the binary tensor whose entry O ijk = 1 if the (i, j, k)-th entry is observed, otherwise O ijk = 0.The goal is to estimate L * given noisy observation T from observation Model (19).We estimate L * by simultaneously exploiting the tube, row and column redundancy in a circular way through minimizing the proposed triple tubal nuclear norm.Specifically, we come up with the following problem: min where parameter > 0 denotes the noise level.The motivation is to recover L * by choosing the tensor with the smallest TriTNN from a hyper-ball in R n 1 ×n 2 ×n 3 with radius .It is well known that Problem (20) in the form of convex minimization with a bounded norm constraint is equivalent to the following unconstrained problem [23]: where τ > 0 is the regularization parameter.

An ADMM Solver to Problem (21)
The alternative direction multiplier method (ADMM) [38] has been extensively used in solving composite convex problems like Problem (21).We will solve Problem (21) by using ADMM in this subsection.
Considering the definition of TriTNN, we introduce auxiliary variables U , V, W ∈ R n 1 ×n 2 ×n 3 and obtain the following constrained problem: First, the augmented Lagrangian of Problem ( 22) is: where Y 1 , Y 2 and Y 3 are Lagrangian multipliers and ρ > 0 is the penalty parameter.
Using the framework of ADMM, we update the variables alternatively by fixing others at the k + 1-th iteration in the following way.
Update L. We update L by fixing the other variables as follows: where denotes element-wise division and I denotes the tensor of all ones.Update U ,V and W. Tensor U is updated as follows: where S • (•) is the proximal operator of TNN at point L k+1 − Y k 1 /ρ with parameter τλ 1 /ρ (see Equation ( 8)).
Tensors V and W are updated similarly to U , )), (26) and: Update Y 1 , Y 2 and Y 3 .Using dual ascending, we update Y 1 , Y 2 and Y 3 as follows: We summarize the algorithm in Algorithm 1 and analyze the computational complexity as follows.
Complexity analysis: The main computational cost in each iteration rests in the singular tube thresholding operator, requiring the computation of FFT, IFFT and SVDs.Therefore, the time complexity in each iteration is: Algorithm 1 Solving Problem (21) using ADMM.

Convergence of Algorithm 1
As Problem (21) has more than two variables, the convergence property of Algorithm 1 cannot be directly obtained from existing results on the convergence of ADMM [38].Thus, we prove its convergence in terms of the objective function in the following theorem.

Theorem 1 (Convergence of Algorithm 1). For any
3 ) in Algorithm 1 satisfy the residual convergence, objective convergence and dual variable convergence of Problem (21).

Proof.
The key idea of the proof is to rewrite Problem (9) into a two-block ADMM problem.Since the RowTwist and ColTwist operations are linear, there exist two matrices P 1 , P 2 ∈ R n 1 n 2 n 3 ×n 1 n 2 n 3 , such that the constraints V = ColTwist(L) and W = RowTwist(L) are equivalent to the vectorization expressions: vec(V ) = P 1 vec(L), and vec(W ) = P 2 vec(L), where vec(•) denotes the operation of tensor vectorization (see [14]).
For notational simplicity, let: and: It is obvious that f (•) and g(•) are closed, proper and convex.Then, Problem ( 21) can be re-written as follows: min According to the convergence analysis in [38], we have: where f * , g * are the optimal values of f (x), g(z), respectively.Variable y * is a dual optimal point defined as:

Differences from Prior Work
First, we show the difference between the proposed model TriTNN and two mostly related models TNN [30] and t-TNN [16].Although all of them are based on the tubal nuclear norm, the main difference lies in that TNN and t-TNN only use information of one orientation, whereas TriTNN uses information of three orientations.Now, we compare the proposed model with Tubal-Alt-Min [37].It is based on tensor tubal rank and employs the tensor factorization strategy for tensor completion.The differences between TriTNN and Tubal-Alt-Min are: (a) TriTNN preserves the low-rank structure by summing three tubal nuclear norms, whereas Tubal-Alt-Min adopts low-rank tensor factorization to characterize the low-rank property of a tensor.In this way, they are two different kinds of models for tensor completion (since Tubal-Alt-Min and the proposed TriTNN are quite different algorithms and the main goal of this paper is to improve upon TNN, we do not compare Tubal-Alt-Min in the experiment section.)(b) Since TriTNN is based on the tubal nuclear norm, it is formulated as a convex optimization problem (21).Benefiting from convexity, each local minimum of Problem ( 21) must be a global minimum.However, Tubal-Alt-Min is formulated as a non-convex optimization problem, thus it may produce sub-optimal solutions.

Experiments
In this section, extensive experiments will be conducted to explore the effectiveness of the proposed Algorithm 1.All the codes are implemented in MATLAB language, and all experiments are carried out in Windows 10 based on an Intel Core(TM) 2.60-GHz CPU with 12 G RAM.
To explore the effectiveness of the proposed TriTNN-based model, we compare with the following nuclear norm-based tensor completion models:

•
The tensor nuclear norm-based model with ADMM solver: high accuracy low-rank tensor completion (HaLRTC, denoted by SNNin this paper) (code available: http://www.cs.rochester.edu/u/jliu/publications.html) [2], The tensor nuclear norm is defined as the weighted sum of nuclear norms of the unfolding matrices along each mode (thus, we denote this model as SNN): where α 1 , α 2 , α 3 are positive parameters and T (i) ∈ R n i ×∏ j =i n i , i = 1, 2, 3, is the unfolding matrix of tensor T ∈ R n 1 ×n 2 ×n 3 along the i-th mode [2].
We conduct tensor completion experiments on color images, videos and a dataset for autonomous vehicle.For an estimation tensor L ∈ R n 1 ×n 2 ×n 3 , its quality is evaluated by the peak signal-to-noise ratio (PSNR) computed by the definition: where L * is the underlying tensor.The higher the PSNR value is, the better the recovery performance will be.

Color Image Inpainting
Color images in row × column × channel are naturally expressed in 3D tensor form.Image inpainting aims at reconstructing a color image from a small fraction of its entries.In this experiment, twelve test images of size 256 × 256 × 3 are used; see Figure 5.Given an image M of size d 1 × d 2 × 3, we randomly sample 30% of its pixels and add i.i.d.Gaussian noise with standard deviation σ = 0.1σ 0 , where σ 0 = M F / √ 3d 1 d 2 is the rescaled magnitude of M.
The weight parameters α of SNN are chosen to satisfy α 1 : α 2 : α 3 = 1 : 1 : 0.01 as suggested in [2].Parameter τ = 8e3 and λ 1 , λ 2 , λ 3 in Problem (22) are chosen to satisfy λ 1 : λ 2 : λ 3 = 1 : 0.01 : 0.01.Parameters of other algorithms are tuned for better performances in most cases.We also employ the structural similarity index measure (SSIM) [43] to measure the quality of inpainted color images.The higher the SSIM value is, the better the inpainting performance will be.Given a color image, we test ten times and report the averaged PSNR and SSIM values.
The inpainting results of five images are shown in Figure 6 for qualitative comparison.We can see that the proposed TriTNN-based model obtains better visual performances.For quantitative comparison, the PSNR and SSIM values on the twelve images of seven algorithms are reported in Figure 7.It can be seen that the proposed TriTNN-based outperforms the competitors in most cases.[21], SquareNN [23], tensor tubal nuclear norm (TNN) [31], t-TNN [16] and the proposed model, respectively.The corresponding PSNR and SSIM values are listed in (k).The highest PSNR and SSIM indicating the best inpainting performance are highlighted in bold.It is suggested to be viewed in color as a pdf file with a 4× zoom in.TriTNN, triple tubal nuclear norm.

Video Inpainting
The video inpainting task aims at imputing the missing pixels of a video.The performance competition is carried out on five widely-used YUVvideos (They are available from https:// sites.google.com/site/subudhibadri/fewhelpfuldownloads):salesman_qcif, silent_qcif, suzie_qcif, tempete_cif and waterfall_cif.Due to the computational limitation, we use the first 30 frames of Y components in each video.This results in three tensors sized 144 × 176 × 32 and two tensors sized 288 × 352 × 32.For each video, we uniformly sample 10% of the entries and conduct the video inpainting experiments.
The qualitative comparison is shown in Figure 8.The PSNR values are reported in Table 1.It can be seen that the TriTNN-based model outperforms the others.The proposed TriTNN has better performances than TNN and t-TNN, since the TriTNN exploits the row, column and tube redundancy simultaneously, whereas TNN or t-TNN only exploit one type of redundancy.The superiority of TriTNN over SNN, LatentNN and SquareNN may be explained by the fact that the circulant block matricization of a tensor coded in TriTNN makes use of more information than directly unfolding along each mode., LatentNN [21], SquareNN [23], TNN [31], t-TNN [16] and the proposed model TriTNN-based model, respectively.It is suggested to be viewed as a pdf file with a 4× zoom in.
Table 1.Quantitative evaluation of algorithms in PSNR values for YUV video inpainting: each video is sampled uniformly with ratio p = 0.1.

A Dataset for Autonomous Driving
Environment perception for autonomous driving has attracted more and more attention in computer vision.In this subsection, experiments on a dataset collected for autonomous driving are performed.
The dataset (a collection of Frame No. 165-No.244 in Scenario B and Scenario B; additional sensor data available at http://www.mrt.kit.edu/z/publ/download/velodynetracking/dataset.html) has 80 frames of gray images and LiDAR point cloud data acquired by a Velodyne HDL-64E LiDAR.The image sequence is resized to be a tensor of size 128 × 256 × 80, and the LiDAR data are resampled, transformed and formatted to be two tensors of size 64 × 436 × 80 representing the distance data and the intensity data, respectively.
Given a tensor to complete T ∈ R n 1 ×n 2 ×n 3 , experiments are carried out with seven different observation settings where the sampling ratio p varies from 0.1-0.7.The observed entries are further corrupted by i.i.d.Gaussian noise with standard deviation σ = 0.2σ 0 , where σ 0 = T F / √ d 1 d 2 d 3 is the normalized magnitude of T .The weight parameters α of SNN are chosen to satisfy α 1 : α 2 : α 3 = 1 : 1 : 1. Parameters τ = 2, ρ = 5e-5 and λ 1 , λ 2 , λ 3 are chosen to satisfy λ 1 : λ 2 : λ 3 = 1 : 1 : 1 for the For the proposed TriTNN, the authors believe it can outperform many nuclear norm-based tensor completion models because more spatial-temporal information is exploited.However, generally speaking, it has the following two drawbacks: 1. Computational inefficiency: Compared to TNN and t-TNN, it is more time-consuming since it involves computing TNN, t-TNN and rt-TNN (see Equation ( 18)). 2. Sample inefficiency: Using the analysis of [23] and [44], to complete an incomplete tensor, TriTNN needs more observations than TNN and t-TNN (limited to the scope of this paper; the authors do not discuss it further).
In the future research, the authors are interested in efficient algorithms like [45] to tackle the problem of computational inefficiency.To decrease the sample complexity of TriTNN, it will be helpful to follow the suggestions in [44] to design new atomic norms like [46].To get better visual completion performances, the authors would like to consider adding smoothness regularization in the model like [47][48][49] and adopting different tensorization methods like [50].It is also helpful for studying new tensor completion models using deep neural networks [51].For potential extensions of TriTNN, the authors would like to explore the p-th order (p > 3) extension [52] and extensions to other discrete transforms other than DFT like [53].

Figure 2 .
Figure 2. The column twist and column squeeze operations.

Figure 3 .
Figure 3.An intuitive illustration of relationships between the original tensor, the column twist tensor, the row twist tensor, the block circulant matrix and the circulant block matricization of a tensor T ∈ R 3×3×3 .Subplots (a-c) show the operations on the column twist tensor, the original tensor and the row twist tensor, respectively.It can be seen that circ(T ) exploits the tube redundancy in a circulant way while keeping the row and column relationship, Algorithms 2018, xx, 1 circ( − → T ) circ(T − → ) As an example, Fig. 4 (a) and (b) intuitively show the relationships between the or the column twist tensor, the block circulant matrix and the circulant block matricizatio T ∈ R 3×3×3 .As illustrated in subplots (a) and (b) of Fig.4, from the circulant persp essentially exploits the tube redundancy in a circulant way while keeping the row relationship and t-TNN essentially exploits the row redundancy in a circulant way whil the tube and column relationship[17].In computer vision applications, the row and colu tensor (like color images or videos) often carry spatial information and the tube often carr or channel information.From the computational perspective, FFT is operated along the tu to compute TNN while t-TNN needs FFT along the row direction.
As an example, Fig. 4 (a) and (b) intuitively show the relationships between t the column twist tensor, the block circulant matrix and the circulant block matric T ∈ R 3×3×3 .As illustrated in subplots (a) and (b) of Fig.

Figure 4 .
Figure 4.The row twist and row squeeze operations.

Figure 5 .
Figure 5. Twelve color images used in the experiments.

Figure 6 .
Figure 6.Examples of color image inpainting.(a) is the observed noisy incomplete image (Obs.) with sampling ratio 0.3 and noise level σ = 0.1σ 0 ; (b-g) show the inpainting results of nuclear norm-based models: SNN [2], latent tensor nuclear norm-based model (LatentNN)[21], SquareNN[23], tensor tubal nuclear norm (TNN)[31], t-TNN[16] and the proposed model, respectively.The corresponding PSNR and SSIM values are listed in (k).The highest PSNR and SSIM indicating the best inpainting performance are highlighted in bold.It is suggested to be viewed in color as a pdf file with a 4× zoom in.TriTNN, triple tubal nuclear norm.

Figure 7 .
Figure 7. Quantitative evaluation of algorithms on color images for Uniform-0.3:the image is sampled uniformly with ratio p = 0.3 and corrupted with noise level σ = 0.1σ 0 .(a) PSNR values; (b) SSIM values.