Color Image Recovery Using Generalized Matrix Completion over Higher-Order Finite Dimensional Algebra

Abstract

To improve the accuracy of color image completion with missing entries, we present a recovery method based on generalized higher-order scalars. We extend the traditional second-order matrix model to a more comprehensive higher-order matrix equivalent, called the “t-matrix” model, which incorporates a pixel neighborhood expansion strategy to characterize the local pixel constraints. This “t-matrix” model is then used to extend some commonly used matrix and tensor completion algorithms to their higher-order versions. We perform extensive experiments on various algorithms using simulated data and publicly available images. The results show that our generalized matrix completion model and the corresponding algorithm compare favorably with their lower-order tensor and conventional matrix counterparts.

1. Introduction

1.1. Background and Related Works

Vectors and matrices are fundamental to data analysis and processing, but it is often a struggle to encapsulate the complex, higher-order structures found in data such as color images, video sequences, and hyperspectral images. These multilinear data structures defy satisfactory representation by traditional vectors and matrices, prompting the use of tensors, i.e., higher-order extensions of vectors and matrices, for more accurate representation.

In real-world scenarios, it is common to find that high dimensional data often have low intrinsic dimensions. This property facilitates several advanced techniques and applications. For example, Floryan et al. [1] reduce data to their intrinsic dimensions, allowing more accurate and low-dimensional dynamical models to capture the essential behavior of high-dimensional systems with low-dimensional features. Li et al. [2] achieve efficiency and robustness by training deep neural networks in low-dimensional spaces without sacrificing performance. Chen et al. [3] use deep learning networks for nonparametric regression on low-dimensional manifolds, emphasizing the adaptability of the networks to low-dimensional geometric structures in data. Xu et al. [4] propose a method that improves high-dimensional data classification through efficient feature extraction and diverse feature fusion, outperforming mainstream ensemble methods.

This notion of a low intrinsic dimension often applies when the data are represented as matrices or tensors. Several methods take advantage of this property. Fu et al. [5] develop a low-rank tensor approximation model for multiview intrinsic subspace clustering that effectively reduces view-specific constraints and improves optimization, with notable success on real-world datasets. Wang et al.’s [6] tensor low-rank and sparse representation method skillfully preserves intrinsic 3D structures in hyperspectral anomaly detection. In addition, Liu et al. [7] comprehensively survey low-rank tensor approximation for hyperspectral image restoration, highlighting state-of-the-art techniques and current challenges in the field. This collective body of work underscores the flexibility and potential of using low-rank approximations to manage and interpret complex, high-dimensional data.

The collection of high-dimensional data can result in the loss of some elements. Low-rank tensor completion (LRTC) addresses this problem by reconstructing the missing components from known data elements. Unlike low-rank matrix completion (LRMC), which relies solely on second-order information, LRTC exploits higher-order information, making the study of low-rank tensor completion techniques an important frontier in many fields.

For example, Liu et al. [8] introduced the Sum of Matricization-based Nuclear Norms (SMNN), which is based on the Tucker rank of the tensor, and formulated three optimization algorithms for tensor completion via SMNN minimization, successfully applying them to visual data completion. Zhang et al. [9,10] developed the tubal nuclear norm (tubal-NN) approach based on the tubal-rank tensor, and designed an algorithm that uses tensor nuclear norm penalization for tensor completion. The algorithm is effective in video recovery and denoising. Lu et al. [11] constructed a tensor completion model using the tensor nuclear norm (TNN). They showed that the TNN is a specific atomic norm, and established a bound for guaranteed low-tubal-rank tensor recovery, thus providing recovery guarantees for tensor completion. Xue et al. [12] extended the tensor completion model to include the tensor truncated nuclear norm (T-TNN), thereby improving its effectiveness in real-world video and image processing.

In addition to these newly defined tensor norms, researchers are now developing other techniques that extend traditional concepts to new applications in tensor completion. For example, Zeng et al. [13] introduced a multimodal nuclear tensor factorization technique, which incorporates low-rank insights and an efficient block successive upper-bound minimization algorithm. The method was applied to tasks such as hyperspectral images, video, and MRI completion, with experimental results confirming its superior performance. Similarly, Wu et al. [14] presented the tensor wheel decomposition method, which characterizes complex interactions with only a few hyperparameters, improving performance on both synthetic and real data. Zhao et al. [15] presented a nonconvex model with a proximal majorization–minimization algorithm for robust low-rank tensor completion, providing theoretical guarantees and demonstrating high efficiency on visual data, including color and multispectral images.

In recommendation systems, Deng et al. [16] applied a meta-learning strategy with low-rank tensor completion for hyperparameter optimization and demonstrated its effectiveness. Nguyen [17] developed a consistency-based framework that emphasizes unit-scale consistency for matrix and tensor completion, with attributes such as fairness and the ability to exploit high-dimensional relationships. Hui et al. [18] integrated social–spatial context into tensor completion for time-aware point-of-interest recommendations, outperforming existing methods.

Tensor completion has also contributed to advance in data mining. Song et al. [19] reviewed recent tensor completion algorithms, examining four perspectives and various applications in data mining. Wu et al. [20] introduced a multiattentional tensor completion network for handling missing entries in road sensor data, demonstrating improved performance. Lee et al.’s [21] sign representable tensor model addresses both low and high rank signals for tensor completion, improving performance on human brain connectivity and topic data mining datasets.

1.2. Contributions and Organization of This Paper

Building on the success of low-rank tensor completion (LRTC) as compared with low-rank matrix completion (LRMC), this paper takes a leap forward by employing a higher-order t-matrix model with high-order circular convolution. This novel method forms a specific generalization of LRMC tailored to multi-way image recovery, including the completion of RGB images with missing entries.

Our model is inspired by the well-regarded completion algorithm proposed by Lu et al. [11]. Our model extends Lu et al.’s algorithm by incorporating a high-order methodology for high-dimensional data. Evaluations of our approach demonstrate competitive recovery performance, with favorable recovery performance compared to existing algorithms.

The practical implications of this work are multifaceted, providing solutions that not only improve visual data completion, but also offer broader applications in areas such as video recovery, data mining, and medical imaging. By integrating our higher-order generalization into existing systems, it is possible to create more efficient and robust mechanisms for handling high-order, high-dimensional data.

The contributions of this research are summarized as follows.

• This research presents a t-matrix model that can extend traditional matrix methods to a higher order. The higher-order algorithm, termed “Higher-Order TNN”, is designed to exploit intricate structures in high-dimensional data and refines classical lower-order algorithms for missing entry recovery of RGB images. Compared to its predecessors, Higher-Order TNN offers significantly improved recovery performance.
• Using the t-matrix model over a finite-dimensional algebra, several image analysis algorithms are extended to a higher order using a novel pixel neighborhood strategy.
• Many constructions in matrix and vector analysis are extended to the t-matrix model. Examples include rank, norm, and inner product. In addition, t-matrix versions of Lagrange multipliers are defined.

The rest of the paper is organized as follows: Section 2 introduces t-matrices, outlining their structure, representation, and possible extension. Section 3 describes the low-rank matrix completion (LRMC) methodology and its higher-order counterparts, explaining the use of t-scalars and t-matrices. Section 4 provides an in-depth exploration of rank considerations, presenting different notions of rank and the concept of higher-order rank. Section 5 details experimental validation and performance analysis, using both simulated random data and real-world datasets such as the Berkeley Segmentation Dataset. Section 6 summarizes the content of this paper and its implications. Appendix A provides further mathematical justification, explaining the mechanism of t-scalars and t-matrices from a unique matrix perspective of representation and operator theory, along with an exploration of the Lagrange multiplier with t-matrix variables.

2. T-Matrices

A t-matrix is a rectangular array composed of elements called t-scalars [22]. Since a t-scalar forms an array in ${\mathbb{C}}^{I_1\times \cdots \times I_N}$, a t-matrix with $D_1$ rows and $D_2$ columns can be represented by a multiway complex array in ${\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$. While various authors [23,24], including Kilmer et al. [10], categorize these t-matrices as tensors, we use the term “t-matrix over t-scalars”. The t-matrix model is used to extend many existing matrix algorithms.

2.1. T-Scalars

A complex array of order N is by definition an element of the set C, where $C\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N}$. Similarly, a real array of order N is an element of the set R, where $R\equiv {\mathbb{R}}^{I_1\times \cdots \times I_N}$. The sets R and C share a commutative ring structure, in which the multiplication is defined by a circular convolution of order N and the addition is entry-wise. Elements of C and R are called t-scalars. In this paper, we focus primarily on C, since R is a subset of C. By further defining the multiplication of a t-scalar with a complex number by conventional scalar multiplication, we can elevate the ring C to a finite dimensional commutative algebra.

Using t-scalars not only allows us to construct novel matrices, but also allows the extension of many classical matrix algorithms into the realm of t-matrix algorithms.

We adopt the notions described by Liao and Maybank [22]. For example, the following definitions are given for t-scalars.

Definition 1

(Addition of T-scalars [22]). Consider two t-scalars $\dot{x},\dot{y}\in {\mathbb{C}}^{I_1\times \cdots \times I_N}$ as order-N arrays of size $I_1\times \cdots \times I_N$. The sum, $\dot{c}=\dot{x}+\dot{y}$, with $\dot{c}\in {\mathbb{C}}^{I_1\times \cdots \times I_N}$, is calculated element-wise,

$${\left(\dot{c}\right)}_{i_1,\dots ,i_N}={\left(\dot{x}\right)}_{i_1,\dots ,i_N}+{\left(\dot{y}\right)}_{i_1,\dots ,i_N}\forall i_1,\dots ,i_N.$$

Definition 2

(Multiplication of T-scalars [22]). Let $\dot{x},\dot{y}\in {\mathbb{C}}^{I_1\times \cdots \times I_N}$ be two t-scalars of size $I_1\times \cdots \times I_N$. Their product is defined as $\dot{c}=\dot{x}\circ \dot{y}$, where $\dot{c}$ results from the order-N circular convolution of $\dot{x}$ and $\dot{y}$. Specifically, we have

$${\left(\dot{c}\right)}_{i_1,\dots ,i_N}=\sum _{(k_1,\dots ,k_N)\in \left[I_1\right]\times \cdots \times \left[I_N\right]}{\left(\dot{x}\right)}_{k_1,\dots ,k_N}·{\left(\dot{y}\right)}_{k_1^{\prime },\dots ,k_N^{\prime }},\forall i_1,\dots ,i_N$$

where $k_n^{\prime }=mod(i_n-k_n,I_n)+1$, for all $n\in \left[N\right]$.

Although order-two t-scalars share the data structure of order-two numerical arrays, they differ from matrices in that their multiplication is commutative. Nevertheless, when a linear transformation, such as the Fourier transform, is applied to a t-scalar, it is convenient to think of the underlying order-two arrays of t-scalars as matrices. This principle extends to higher-order t-scalars as well. When a multilinear transformation is applied to a higher-order t-scalar, the underlying higher-order array of the t-scalar can be treated as a tensor.

We use the notation $tensor\left(\dot{x}\right)$ to elevate the underlying order-N array of $\dot{x}$ to a conventional tensor of identical size and entries. With $tensor\left(\dot{x}\right)$ as the conventional tensor, the multiplication of two t-scalars is given in the Fourier domain by the following theorem.

Theorem 1

(Fourier Transform). Let $\dot{x},\dot{y}\in C$ be two t-scalars with the product $\dot{c}=\dot{x}\circ \dot{y}\in C$. Define $\tilde{x}=F\left(\dot{x}\right)$, $\tilde{y}=F\left(\dot{y}\right)$, and $\tilde{c}\doteq F\left(\dot{c}\right)$ as their respective multilinear Fourier transforms. For any t-scalar $\dot{x}\in C$, its multilinear Fourier transform is given by the following multi-mode multiplication:

$$F\left(\dot{x}\right)\doteq tensor\left(\dot{x}\right){\times }_1{W}_1\cdots {\times }_n{W}_n\cdots {\times }_N{W}_N$$

(1)

where $W_n$ denotes the $I_n\times I_n$ Fourier matrix such that the following equation holds for all $k_1,k_2\in \left[I_n\right]$,

$${\left(W_n\right)}_{k_1,k_2}=exp\left[-2\pi I_n^{-1}\sqrt{-1}(k_1-1)(k_2-1)\right].$$

Consequently, the following Hadamard product holds for all $i_1,\dots ,i_N$:

$${\left(\tilde{c}\right)}_{i_1,\dots ,i_N}={\left(\tilde{x}\right)}_{i_1,\dots ,i_N}·{\left(\tilde{y}\right)}_{i_1,\dots ,i_N}.$$

Definitions 1 and 2 qualify all t-scalars as elements of a commutative ring C. An essential operation in C is the multiplication of elements of C with a scalar. This leads to the following definition.

Definition 3

(Scalar Multiplication [22]). Let $\dot{x}\in C\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N}$ be a t-scalar and λ a complex number. Their multiplication, denoted as $\dot{y}\doteq \lambda ·\dot{x}\in C$, is defined for all $i_1,\dots ,i_N$ as follows:

$${\left(\dot{y}\right)}_{i_1,\dots ,i_N}=\lambda ·{\left(\dot{x}\right)}_{i_1,\dots ,i_N}.$$

With the above definitions as a basis, we have the following proposition about the identity and zero t-scalars in the algebra C.

Proposition 1

(Identity T-scalar and Zero T-scalar [22]). Consider a t-scalar $\dot{e}\in {\mathbb{C}}^{I_1\times \cdots \times I_N}$. In this case, ${\left(\dot{e}\right)}_{i_1,\dots ,i_N}=1$ if $i_1=\cdots =i_N=1$, and ${\left(\dot{e}\right)}_{i_1,\dots ,i_N}=0$ otherwise. Alternatively, if every entry of $\dot{x}$ is 0, we have this t-scalar as the zero t-scalar $\dot{z}$. Note that each entry of the identity t-scalar is 1 in the Fourier domain, while the zero t-scalar remains unchanged in the Fourier domain.

2.2. T-Scalars as Finite-Dimensional Linear Operators

Since every t-scalar in the algebra C defines a commutative finite dimensional linear operator, operator theory allows us to determine the spectrum of any t-scalar $\dot{x}\in C$. The spectrum, or the set of complex eigenvalues of $\dot{x}$, corresponds to the K entries of the Fourier transform $\tilde{x}$ (where $K\doteq I_1·I_2\cdots I_N$), taking multiplicity into account.

The eigenvalues, i.e., Fourier entries, of a t-scalar help to define concepts such as trace, Schatten norm, and singular values of a t-scalar. The following definitions are presented first, before a more detailed exploration of the above concepts later in this paper.

Definition 4

(Conjugate [22]). A unique t-scalar $\dot{y}$ in C is the conjugate of a t-scalar $\dot{x}$ in C if each eigenvalue of $\dot{y}$ is the complex conjugate of the corresponding eigenvalue of $\dot{x}$. The conjugate is denoted by ${\dot{x}}^{*}$.

Definition 5

(Non-negativity [22]). A t-scalar $\dot{x}$ is said to be non-negative if and only if all of its complex eigenvalues (i.e., Fourier entries) are non-negative real numbers.

Definition 5 is crucial because it facilitates the generalization of various concepts of non-negativity, including matrix rank, space dimension, norm, and distance, to non-negative elements in C. We explore these generalizations below, as needed.

Consider any two non-negative t-scalars $\dot{x},\dot{y}$. Their difference is self-conjugate because ${(\dot{x}-\dot{y})}^{*}=\dot{x}-\dot{y}$ holds invariably. The element $\dot{z}\in C$ is the smallest non-negative element. If $\dot{x}-\dot{y}$ is non-negative, a partial order $\dot{x}\ge \dot{y}\ge \dot{z}$ is defined. If neither $\dot{x}-\dot{y}$ nor $\dot{y}-\dot{x}$ is non-negative, $\dot{x}$ and $\dot{y}$ are said to be incomparable.

2.3. T-Matrices

A t-matrix is a rectangular container of t-scalars. Since the underlying data forms of t-scalars are arrays in ${\mathbb{C}}^{I_1\times \cdots \times I_N}$, it is logical to represent the underlying data form of a t-matrix in $C^{D_1\times D_2}$ as a higher-order array in ${\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$. We refer to this higher-order array format as the little-endian representation of a t-matrix. Conversely, some authors may arrange a t-matrix in $C^{D_1\times D_2}$ as an array in ${\mathbb{C}}^{D_1\times D_2\times I_1\times \cdots \times I_N}$ rather than in ${\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$. We call this form the big-endian representation. It is used by Kilmer et al. [10] in their paper. The conversion between the little-endian and big-endian protocols is straightforward. Because of the underlying multiway array structure of t-matrices, some authors refer to these t-matrices as tensors [25], although they differ from the tensors defined in other areas [26].

The operations on t-matrices are analogous to those on traditional matrices. Specifically, if we have a t-matrix in $C^{D_1\times D_2}\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$ and another in $C^{D_2\times D_3}\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N\times D_2\times D_3}$, their multiplication yields a t-matrix in $C^{D_1\times D_3}\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_3}$. Similarly, constructs such as the conjugate transpose and the diagonal matrix can be defined analogously. For a more detailed discussion of these concepts, see [22].

2.4. Singular Value Decomposition of a T-Matrix

To exploit the structure of a t-matrix, it is often decomposed into a product of simpler matrices. In detail, the TSVD (Tensorial SVD) of a t-matrix $\dot{X}\in C^{D_1\times D_2}$ is given by

$$\dot{X}=\dot{U}\circ \dot{S}\circ {\dot{V}}^{*}$$

(2)

where $\dot{U}\in C^{D_1\times D}$, $\dot{S}\in C^{D\times D}$, and $\dot{V}\in C^{D_2\times D}$, where $D\doteq min(D_1,D_2)$. The symbol ${\dot{V}}^{*}$ denotes the conjugate transpose of the t-matrix $\dot{V}$, and $\dot{S}\doteq diag({\dot{\sigma }}_1,\dots ,{\dot{\sigma }}_D)$ is a diagonal t-matrix with non-negative t-scalars as its diagonal entries in non-increasing partial order ${\dot{\sigma }}_1\ge \cdots \ge {\dot{\sigma }}_D\ge \dot{z}$.

In addition, the following generalized orthogonal constraints are available:

$${\dot{U}}^{*}\circ \dot{U}={\dot{V}}^{*}\circ \dot{V}=\dot{I}\doteq diag(\dot{e},\dots ,\dot{e})\in C^{D\times D}\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N\times D\times D}$$

(3)

Here, $\dot{I}$ denotes the identity t-matrix, which has diagonal entries of $\dot{e}$ and off-diagonal entries of $\dot{z}$.

Equation (2) defines the tensorial singular value decomposition (TSVD) of a t-matrix. Numerous methods expound on the computational and operational aspects of TSVD. Among these, one particularly practical method employs the mechanism of spectral slices.

Given a t-matrix $\dot{X}\in C^{D_1\times D_2}$, represented as a little-endian complex array in ${\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$, let $\dot{X}\mapsto tensor\left(\dot{X}\right)$ map this underlying array into a conventional tensor with identical size and entries. Complying with Equation (1), the Fourier transform of $\dot{X}$ is expressed as the following multi-mode multiplication:

$$\tilde{X}\doteq F\left(\dot{X}\right)=tensor\left(\dot{X}\right){\times }_1{W}_1\cdots {\times }_n{W}_n\cdots {\times }_N{W}_N$$

(4)

Since all operations on t-scalars in the Fourier domain are Fourier-entry-wise, the following definition can be used to decompose t-matrices in the Fourier domain and to establish further constructs.

Definition 6

(Spectral Slice [22]). For any t-matrix $\dot{X}\in C^{D_1\times D_2}\equiv {\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$, its Fourier transform $\tilde{X}\in {\mathbb{C}}^{I_1\times \cdots \times I_N\times D_1\times D_2}$, as defined in Equation (4), can be partitioned into K spectral slices (where $K=I_1·I_2\cdots I_N$). Each spectral slice, indexed by $(i_1,\dots ,i_N)$, is a conventional complex matrix denoted by $\tilde{X}(i_1,\dots ,i_N)\in {\mathbb{C}}^{D_1\times D_2}$. The matrix satisfies the following equation for all $i_1,\dots ,i_N$ and $d_1,d_2$:

$${\left(\tilde{X}(i_1,\dots ,i_N)\right)}_{d_1,d_2}={\left(\tilde{X}\right)}_{i_1,\dots ,i_N,d_1,d_2}.$$

(5)

Using spectral slices, various constructs can be introduced. For example, the Tensor Singular Value Decomposition (TSVD) of a t-matrix is obtained by Algorithm 1.

Algorithm 1 Tensorial Singular Value Decomposition via Spectral Slices

1: procedure $[\dot{U},\dot{S},\dot{V}]$ = TSVD($\dot{X}$)  2:     Apply Equation (4) to compute the transform $\tilde{X}$ from $\dot{X}$.  3:     for $(i_1,\dots ,i_N)\in \left[I_1\right]\times \cdots \times \left[I_N\right]$ do  4:        Compute the compact SVD of each spectral slice $\tilde{X}(i_1,\dots ,i_N)=U·S·V^H$.  5:        Store the resulting U, S, and V in $\tilde{U}(i_1,\dots ,i_N)$, $\tilde{S}(i_1,\dots ,i_N)$, and $\tilde{V}(i_1,\dots ,i_N)$.  6:     end for  7:     Apply the inverse transform $F^{-1}$ to each of $\tilde{U}$, $\tilde{S}$, and $\tilde{V}$ to obtain $\dot{U}$, $\dot{S}$, and $\dot{V}$, respectively. 8: end procedure

Note that Algorithm 1 computes the compact TSVD of a tensor $\dot{X}\in C^{D_1\times D_2}$. In this decomposition, $\dot{X}=\dot{U}\circ \dot{S}\circ {\dot{V}}^{*}$, where $\dot{U}\in C^{D_1\times D}$, $\dot{S}\in C^{D\times D}$, and $\dot{V}\in C^{D_2\times D}$, where $D\doteq min(D_1,D_2)$.

Spectral slices facilitate the extension of many conventional algorithms, including the acclaimed Singular Value Thresholding (SVT) algorithm [27,28], which is an integral part of low-rank matrix completion (LRMC) problems. The implementation of Tensorial Singular Value Thresholding on a t-matrix is presented in Algorithm 2.

Algorithm 2 Tensorial Singular Value Thresholding via Spectral Slices

1: procedure $\dot{Y}$ = TSVT($\dot{X},\tau$) where $\tau$ is a small positive constant  2:     Apply Equation (4) to compute the transform $\tilde{X}$ of $\dot{X}$.  3:     for $(i_1,\dots ,i_N)\in \left[I_1\right]\times \cdots \times \left[I_N\right]$ do  4:          Compute the compact SVD on each spectral slice $\tilde{X}(i_1,\dots ,i_N)=U·S·V^H$.  5:          Store the result of the singular value thresholding $U·C_{\tau }\left(S\right)·V^H$ in $\tilde{Y}(i_1,\dots ,i_N)$.  6:     end for  7:     Apply the inverse transform $F^{-1}$ to $\tilde{Y}$ to obtain $\dot{Y}$, the approximation of $\dot{X}$.  8: end procedure

In Algorithm 2, lines 4 and 5, $V^H$ represents the conjugate transpose of V, while $C_{\tau }\left(S\right)$ denotes the soft thresholding of $S=diag({\sigma }_1,\dots ,{\sigma }_D)$. The threshold parameter $\tau$ is applied as $C_{\tau }\left(S\right)=diag({\sigma }_1^{\prime },\dots ,{\sigma }_D^{\prime })$ where ${\sigma }_d^{\prime }=max(0,{\sigma }_d-\tau )$ for $d\in \left[D\right]$.

3. Low-Rank Matrix Completion and Its Generalizations

Besides the generalization of SVD to its higher-order counterpart TSVD, many other matrix algorithms and approaches can be extended analogously over t-scalars. One of them is the so-called low-rank matrix completion (LRMC).

3.1. Low-Rank Matrix Completion

A variant of the matrix completion problem is to determine the minimum rank matrix $X\in {\mathbb{R}}^{D_1\times D_2}$ that matches the desired matrix M for all observed entries within the index set $\Omega$ [28]. This problem can be expressed mathematically as

$$\underset{X}{minimize}rank\left(X\right)\mathrm{subject}\mathrm{to}{\left(X\right)}_{i,j}={\left(M\right)}_{i,j}\forall (i,j)\in \Omega .$$

(6)

Given the NP-hard nature of the above minimization problem, in most practical scenarios the solution to Equation (6) can most likely be reformulated as the solution to the following convex optimization problem:

$$\underset{X,E}{minimize}{\parallel X\parallel }_{*}\mathrm{subject}\mathrm{to}X+E=M,G_{\Omega }\left(E\right)=\mathrm{array}\mathrm{of}\mathrm{zeros}$$

(7)

where ${\parallel X\parallel }_{*}$ denotes the nuclear norm of X and $G_{\Omega }:{\mathbb{R}}^{D_1\times D_2}\to {\mathbb{R}}^{D_1\times D_2}$ is a linear operator, which preserves entries within the set $\Omega$ and sets entries outside of $\Omega$ to zero.

The augmented Lagrange multiplier function for the minimization problem is formalized as follows [28].

$$L(X,E,Y,\tau )={\parallel X\parallel }_{*}+\langle Y,M-X-E\rangle +\frac{1}{2\tau }{\parallel M-X-E\parallel }_F^2$$

(8)

where Y is the dual variable matrix, $\tau >0$, and $\langle Y,M-X-E\rangle$ is the inner product of the matrices Y and $M-X-E$.

The Alternating Direction Method of Multipliers (ADMM) [29,30] can be used to iteratively refine the optimization variables X and E, as described in Algorithm 3. Here, $M\in {\mathbb{R}}^{D_1\times D_2}$, and $\Omega$ represents a random non-empty proper subset of the Cartesian product $\left[D_1\right]\times \left[D_2\right]$. Furthermore, $D_{\tau }$ in line 5 denotes the SVT operator with threshold $\tau$.

3.2. Generalization of Matrix Completion over Higher-Order T-Scalars

Following the application of ADMM to the optimization of Equation (7), many authors have proposed methods to extend the completion process to third-order arrays. For example, using Kilmer et al.’s [10,25] “t-product” model, Lu et al. [11] extend the above completion approach to third-order tensors.

Algorithm 3 ADMM for solving Equation (7)

1: procedure $X_{\mathit{COMP}}$ = MatrixCompletion($M,\Omega$)   2:     Initialization: $k\leftarrow 0$, $Y_0=E_0\leftarrow \mathrm{array}\mathrm{of}\mathrm{zeros}$, $\alpha \leftarrow 0.9$, ${\tau }_0\leftarrow {10}^4$, ${\tau }_{\min }\leftarrow {10}^{-6}$ 3:     Set the missing entries of M, i.e., $(i,j)\in {\Omega }^c$, to zero   4:     while neither convergence nor predefined maximum iterations achieved do  5:          $X_{k+1}\leftarrow \underset{X}{argmin}{\parallel X\parallel }_{*}+\frac{1}{2{\tau }_k}{\parallel X+E_k-M-{\tau }_k·Y_k\parallel }_F^2\equiv D_{{\tau }_k}(M-E_k+{\tau }_k·Y_k)$   6:          $E_{k+1}\leftarrow G_{\overline{\Omega }}(M-X_{k+1}+{\tau }_k·Y_k)$   7:         $Y_{k+1}\leftarrow Y_k+\frac{1}{{\tau }_k}·(M-X_{k+1}-E_{k+1})$   8:         ${\tau }_{k+1}\leftarrow max\left(\alpha ·{\tau }_k,{\tau }_{\min }\right)$ and $k\leftarrow k+1$ 9:     end while  10:     $X_{\mathit{COMP}}\leftarrow X_k$ 11: end procedure

Although they are called tensor algorithms, Lu et al.’s [11] approach and other variants [9,12,31] are essentially matrix completion algorithms operating on first-order t-scalars. However, as noted above, t-scalers can be defined for any order. Since higher-order arrays encapsulate more structural information than their lower-order counterparts in real-world scenarios, we exploit this aspect by increasing the order of the arrays via a pixel neighborhood strategy originally introduced in [32].

Specifically, Figure 1 shows the application of a “$3\times 3$ pixel neighborhood” strategy to increase the order of a $4\times 4$ pixel grayscale image. Note that the figure represents the result of the order-four array as a two-dimensional array of two-dimensional blocks.

The use of this pixel neighborhood strategy results in a fourth-order array of size $3\times 3\times 4\times 4$, which is interpreted as a $4\times 4$ matrix with t-scalars of size $3\times 3$.

Note also that the underlying array format can be chosen in the form of $4\times 4\times 3\times 3$ instead of $3\times 3\times 4\times 4$. We call the $3\times 3\times 4\times 4$ construct the little-endian array format of t-matrices, and the other the big-endian format of t-matrices.

The endian protocol is not an integral part of the algebraic definitions of t-matrices. Several array programming languages, including MATLAB and Python (NumPy), provide convenient tools to facilitate conversion between endian protocols. While Kilmer et al.’s [10,25] “t-product” model uses big-endian, we use little-endian, which has the advantage of aligning the multimode multiplication formulation of the Fourier transform of t-matrices (as in Equation (4)) with that of t-scalars (as in Equation (1)).

Since each pixel value can be elevated to a t-scalar, allowing the conversion of a traditional matrix into a generalized one with higher-order fixed-size arrays as entries, the extension of Algorithm 3 to t-matrix completion is straightforward. The generalization of line 5 in Algorithm 3 can be achieved using TSVD, as described in Algorithm 2. Meanwhile, line 6 in Algorithm 3 is extended by the function $G_{\overline{\Theta }}$, which preserves entries within the enhanced set $\overline{\Theta }$ and sets those outside $\overline{\Theta }$ to the zero t-scalar $\dot{z}$.

We propose a t-matrix completion algorithm for recovering multispectral images with missing values, as described in Algorithm 4, where $M\in {\mathbb{R}}^{D_1\times D_2\times D_3}$, $\Omega$ represents a random non-empty proper subset of the Cartesian product $\left[D_1\right]\times \left[D_2\right]\times \left[D_3\right]$. The result $\Theta$ is a proper subset of $\left[I_1\right]\times \left[I_2\right]\times \left[D_3\right]\times \left[D_1\right]\times \left[D_2\right]$, and the size of t-scalars is $I_1\times I_2\times D_3$, where $I_1,I_2$ are both odd numbers.

Algorithm 4 Higher-Order TNN: ADMM for recovering an image with missing values

1: procedure $X_{\mathit{COMP}}$ = SpectralImageCompletion($M,\Omega$)   2:     Assign the label $-1$ to the missing entries indexed by $\Omega$.   3:      Construct $M_1,M_2,\cdots ,M_{D_3}\in {\mathbb{R}}^{I_1\times I_2\times D_1\times D_2}$ using the $I_1\times I_2$ neighborhood strategy for every frontal slice of M. 4:      Construct $M_{\mathit{UP}}\in {\mathbb{R}}^{I_1\times I_2\times D_1\times D_2\times D_3}$ by aligning $M_1,M_2,\cdots ,M_{D_3}$ along the mode-5 5:      Convert $M_{\mathit{UP}}$ into a t-matrix $\dot{M}\in C^{D_1\times D_2}\equiv {\mathbb{R}}^{I_1\times I_2\times D_3\times D_1\times D_2}$ by permuting the indices of array. 6:     Store the positions of the entries “$-1$” of $\dot{M}$ within $\Theta$.   7:     Initialization: $k\leftarrow 0$, ${\dot{Y}}_0\leftarrow \mathrm{array}\mathrm{of}\mathrm{zeros}$, ${\dot{E}}_0\leftarrow \mathrm{array}\mathrm{of}\mathrm{zeros}$, $\alpha \leftarrow 0.9$, ${\tau }_{\min }\leftarrow {10}^{-6}$ 8:     while neither convergence nor predefined maximum iterations reached do  9:          ${\dot{X}}_{k+1}\leftarrow \mathrm{TSVT}(\dot{M}-{\dot{E}}_k+{\tau }_k·{\dot{Y}}_k,{\tau }_k)$   10:          ${\dot{E}}_{k+1}\leftarrow G_{\overline{\Theta }}(\dot{M}-{\dot{X}}_{k+1}+{\tau }_k·{\dot{Y}}_k)$   11:         ${\dot{Y}}_{k+1}\leftarrow Y_k+\frac{1}{{\tau }_k}(\dot{M}-{\dot{X}}_{k+1}-{\dot{E}}_{k+1})$   12:         ${\tau }_{k+1}\leftarrow max\left(\alpha ·{\tau }_k,{\tau }_{\min }\right)$ and $k\leftarrow k+1$ 13:     end while  14:      Use the row-index-first (MATLAB compliant) protocol to reshape ${\dot{X}}_k$ into an $I_1{I}_2\times D_3{D}_1{D}_2$ matrix, extract the central row, and subsequently reshape it with the row-index-first protocal into an array $X_{\mathit{DOWN}}\in {\mathbb{R}}^{D_3\times D_1\times D_2}$. 15:      Permute the indices of $X_{\mathit{DOWN}}$ to convert it into an array in ${\mathbb{R}}^{D_1\times D_2\times D_3}$.   16:     Adjust the entries of $X_{\mathit{DOWN}}$ to non-negative integers and store the adjusted array $X_{\mathit{DOWN}}$ as $X_{\mathit{COMP}}$, as the recovered multispectral image. 17: end procedure

The lines 3, 4, and 5 of the proposed algorithm up-convert the input multispectral image M, an initial array $D_1\times D_2\times D_3$, into a $D_1\times D_2$ t-matrix $M_{\mathit{UP}}$, with t-scalars of size $I_1\times I_2\times D_3$. Conversely, line 14 down-converts the optimal t-matrix ${\dot{X}}_k$ into a third-order array $X_{\mathit{DOWN}}$.

Algorithm 4 is based on the tensor completion algorithm of Lu et al. [11]. Section 5 of this paper focuses on its empirical validation.

3.3. Computational Complexity

A t-matrix algorithm can be implemented using spectral slices of t-matrices. The number of spectral slices is equal to the dimension of the t-algebra, resulting in a computational complexity that is linearly proportional to the dimension. As a result, t-matrix algorithms often have an increased computational cost. However, each spectral slice operates independently, allowing efficient parallel processing of t-matrix algorithms. This data independence in the Fourier domain facilitates the use of array programming, which is compatible with many programming languages such as MATLAB, R, NumPy, Julia, and Fortran.

4. Rank Considerations

Matrix completion is the recovery of a low-rank matrix given a version of the matrix with some missing entries. Its generalization aims to recover an analogous higher-order t-matrix. However, the rank of a t-matrix is not yet specified, so let us look at some novel rank notions defined for higher-order arrays.

4.1. Tubal Rank and Average Rank

An RGB image consists of three monochromatic channels. Each channel in RGB images from the real world can be adequately approximated by low-rank matrices. However, when viewed as a third-order tensor, the canonical rank of an RGB image, which is defined by the minimal set of rank-one tensor addends, becomes computationally intractable.

Kilmer et al.’s approach is to introduce a novel rank concept, called tubal rank, for a third-order array. Specifically, for a given third-order array $\dot{X}$, with its TSVD defined as $\dot{X}=\dot{U}\circ \dot{S}\circ {\dot{V}}^{*}$, the tubal rank of $\dot{X}$ corresponds to the number of non-zero (i.e., not equal to $\dot{z}$) diagonal t-scalars in $\dot{S}$. Nevertheless, by this definition, a t-matrix of full tubal rank can consist of a full-rank matrix as one of its spectral slices, with all other spectral slices being zero matrices.

To address this problem, Lu et al. [31] define the average of all spectral slice ranks as the “average rank” of a t-matrix. This “average rank” definition is more appropriate than the tubal rank. However, this term is only used for Lu et al.’s generalization of robust component analysis [31], not for the higher-order array recovery problem presented in [11]. The mathematical justification for using this average rank has not been adequately addressed.

4.2. Higher-Order Rank and Its Trace Variant

In addition to the tubal rank of Kilmer et al. and the average rank of Lu et al., another relevant concept is the higher-order rank introduced by Liao and Maybank [22]. Specifically, given a t-matrix $\dot{X}$ with its tensor singular value decomposition (TSVD) $\dot{X}=\dot{U}\circ \dot{S}\circ {\dot{V}}^{*}$, the higher-order rank of $\dot{X}$ is a non-negative t-scalar given by the sum of the diagonal entries of the product ${\dot{S}}^{†}\circ \dot{S}$, where ${\dot{S}}^{†}$ denotes the pseudoinverse of $\dot{S}$.

This rank generalizes the definition of rank for traditional matrices. The pseudo-inverse of a t-matrix can be computed using spectral slices, similar to Algorithms 1 and 2. Specifically, the pseudo-inverse ${\dot{X}}^{†}$ of a t-matrix is defined by assigning the pseudo-inverse of each spectral slice to its corresponding slice in the result. It is not difficult to verify that the pseudo-inverse ${\dot{X}}^{†}$ defined above is equal to the product $\dot{V}\circ {\dot{S}}^{†}\circ {\dot{U}}^{*}$. That is, the equation ${\dot{X}}^{†}=\dot{V}\circ {\dot{S}}^{†}\circ {\dot{U}}^{*}$ holds for any t-matrix $\dot{X}$.

From the previous definition, it is easy to see that the higher-order rank of any t-matrix is a non-negative t-scalar. It can be sorted alongside comparable non-negative counterparts using the partial order introduced in Section 2.2. However, sometimes we prefer a more efficient rank notion similar to the fully ordered ones proposed by Kilmer et al. and Lu et al. as opposed to the partially ordered higher order rank. Reassuringly, the Szpilrajn extension theorem asserts that the partially ordered rank system proposed in [22] can always be extended to a fully ordered rank system.

There are several strategies for transforming the higher-order rank system into its fully ordered counterparts. Considering any higher-order rank of a t-matrix $\dot{X}$, each of its eigenvalues, (i.e., Fourier entries,) is non-negative and the traditional matrix rank of the corresponding spectral slice of $\dot{X}$. Consequently, Kilmer et al.’s tubal rank denotes the maximum value among these spectral points, while Lu et al.’s average rank is equal to their arithmetic mean.

Typically, in most scenarios, the average rank of Lu et al. is considered a superior statistic for a higher-order rank. However, to avoid fractional rank values, we propose using the sum, rather than the arithmetic mean, of the spectral points of a higher-order rank to define its corresponding fully ordered rank. Furthermore, since every t-scalar also functions as a finite-dimensional linear endomorphic operator, the previously defined “sum rank” of a t-matrix is equivalent to the trace of the higher-order rank, so it would be appropriate to formally label it as the “trace rank”.

The fairness of the above definitions can be given by using the representation theory known in the mathematical community. We give a brief discussion of representation theory with its application to justify the above definition in Appendix A.

5. Experiments

This part presents experimental validation and performance analysis of the algorithms.

5.1. Experiments on Simulated Random Data

To evaluate the completion capability of the proposed Higher-Order TNN algorithm, we use simulated random t-matrices. Specifically, we generate two random t-matrices $\dot{P}\in C^{D\times r}$ and $\dot{Q}\in C^{r\times D}$, which are represented as arrays in ${\mathbb{R}}^{I_1\times I_2\times I_3\times D\times r}$ and ${\mathbb{R}}^{I_1\times I_2\times I_3\times r\times D}$, where $I_1\times I_2\times I_3=3\times 3\times 3$ and $r<D$. The parameter r is (with high probability) the tubal rank as defined by Kilmer et al. [10,25]. The real numbers in the underlying arrays of $\dot{P}$ and $\dot{Q}$ are independently sampled from the Gaussian distribution $\mathcal{N}(0,1)$.

The product $\dot{Y}=\dot{P}·\dot{Q}$ gives a random t-matrix in $C^{D\times D}\equiv {\mathbb{R}}^{I_1\times I_2\times I_3\times D\times D}$. The trace rank of $\dot{Y}$ is, with high probability, ${rank}_{\mathrm{trace}}\dot{Y}=I_1{I}_2{I}_3·r$. From the underlying array of $\dot{Y}$, we randomly and uniformly select entries to simulate missing data. The resulting incomplete $\dot{Y}$, with a varying percentage of missing entries, and a rank parameter r for $\dot{Y}$, serve as inputs to the Higher-Order TNN algorithm.

The Higher-Order TNN algorithm produces a t-matrix $\dot{X}\in C^{D\times D}\equiv {\mathbb{R}}^{I_1\times I_2\times I_3\times D\times D}$, which is an estimate of $\dot{Y}$. Let ${\parallel ·\parallel }_F$ be the Frobenius norm. If the Relative Standard Error

$$\mathit{RSE}\doteq \parallel tensor\left(\dot{Y}\right)-tensor\left(\dot{X}\right){\parallel }_F/\parallel tensor\left(\dot{Y}\right){\parallel }_F$$

is less than a threshold, the completion by the Higher-Order TNN algorithm is considered successful.

Figure 2 illustrates the RSE distributions and phase transitions for fifth-order array completions using the proposed Higher-Order TNN. Figure 2a shows the RSE distribution with the parameter $D=40$. Figure 2b shows a phase transition corresponding to $D=50$, where the white and black dots represent successful ($\mathit{RSE}<1\times {10}^{-2}$) and failed completions, respectively. Figure 2a,c show the RSE distributions with $D=40$ and $D=50$ Figure 2b,d shows the phase transitions associated with $D=40$ and $D=50$, respectively.

5.2. Experiments on BSD Color Images

In the following experiments, we use the Berkeley Segmentation Dataset as a benchmark to compare the performance of four related algorithms: tubal-TNN [9], T-TNN [12], TNN [11], and our Higher-Order TNN. Three RGB images, namely “Resort”, “Insect”, and “Seagulls”, are selected for the first experiment. These images are represented as $321\times 481\times 3$ unsigned integer arrays.

To compare the completion performance, we randomly select $70\%$ of the pixel values of each image as “missing” entries. The incomplete images, with missing values set to zero, provide a visual representation in Figure 3, which shows the original complete images alongside their incomplete versions.

We use the three competing algorithms and our Higher-Order TNN, described in Algorithm 4, to obtain a complete RGB image of equal size. The quality of the image completion is quantified by the Peak Signal to Noise Ratio (PSNR), defined as

$$\mathit{PSNR}=10·{log}_{10}\frac{D_1·D_2·D_3}{\parallel X_{\mathit{COMP}}{-M\parallel }_F^2}$$

(9)

where $D_1·D_2·D_3$ is the number of entries in the RGB image. Note that the entries of $X_{\mathit{COMP}}$ and M are scaled such that the largest entry value in each scaled array is 1.

Figure 4 presents visual and quantitative comparisons of the performance of four competing algorithms in completing the observed images: “Resort”, “Insect”, and “Seagulls” from the Berkeley Segmentation Dataset. The proposed Higher-Order TNN outperforms its competitors in terms of PSNRs by at least 1 dB, $1.4$ dB, and $1.6$ dB, respectively.

In our second experiment, we use three different images—“Temple”, “Chapel”, and “Grass-flower”, each with $50\%$ entries randomly missing, to perform completions analogous to the first experiment on the Berkeley images, each with $70\%$ entries randomly missing. Figure 5 shows the original images along with their incomplete versions.

Figure 6 shows the visual and quantitative comparisons of the four related image completion algorithms. Consistent with the results of the first experiment, the Higher-Order TNN significantly outperforms the other algorithms, achieving gains of at least $1.5$ dB, $1.2$ dB, and $2.2$ dB, respectively.

After experiments on 6 RGB images, our research now includes 10 randomly selected RGB images from the Berkeley Segmentation Dataset, with the percentage of missing entries set to increments within $10\%,20\%,\dots ,90\%$.

Figure 7 shows the 10 randomly selected RGB images used for the experiments. Figure 8 shows the PSNR heatmaps for four relevant algorithms: Tubal-NN [9], T-TNN [12], TNN [11], and our newly developed Higher-Order TNN on the images shown in Figure 7.

These results demonstrate the superior performance of the Higher-Order TNN, as it outperforms its counterparts in terms of PSNR.

For a clearer demonstration, Figure 9 shows the PSNR gains of our Higher-Order TNN over tubal-NN, T-TNN, and TNN. It shows that the PSNR gains of Higher-Order TNN over its counterparts are predominantly positive.

6. Conclusions

In this paper, we consider the problem of higher-order array completion using the higher-order t-matrix model. By adopting a consistent solution, we generalize low-rank matrix completion for the recovery of RGB images with missing entries. Our proposed “Higher-Order TNN” method outperforms competitors such as tubal-TNN, T-TNN, and TNN in terms of recovery performance, demonstrating the ability to exploit higher-order relationships within high-dimensional data and showing advantages.

Our solution not only improves visual data completion, but also paves the way for broader applications. Integrating higher-order generalization into existing systems lays the foundation for more robust and efficient handling of high-order, high-dimensional data. Furthermore, any t-matrix algorithm based on spectral slices of t-matrices has a computational complexity linear in the dimension of the t-algebra. Its independent spectral slices enable faster parallel computation using array programming techniques supported by many programming languages.

We demonstrate this with a generalization of Lu et al.’s [11] tensor completion algorithm to its higher-order version by formulating its matrix model over a finite-dimensional algebra. This is achieved through a novel pixel neighborhood strategy.

The study also provides a consistent methodology for exploring various properties of the t-matrix model, including the notions of rank, norm, and inner product. Compared to the existing “t-product” model, our approach offers new insights into t-scalars and t-matrices from a fresh perspective of representation and operator theory. Moreover, the higher-order Lagrange multiplier with t-matrix variables adds to our contributions.

The inclusion of the novel “trace rank”, nuclear norm, Schatten p-norm, and the adaptation of the recent tensor completion algorithm by Lu et al. for higher-order scenarios highlight our results. The experiments emphasize the competitive advantage of our higher-order matrix completion algorithm in RGB image recovery.