Tensor Global Extrapolation Methods Using the n-Mode and the Einstein Products

Abstract

In this paper, we present new Tensor extrapolation methods as generalizations of well known vector, matrix and block extrapolation methods such as polynomial extrapolation methods or $ϵ$-type algorithms. We will define new tensor products that will be used to introduce global tensor extrapolation methods. We discuss the application of these methods to the solution of linear and non linear tensor systems of equations and propose an efficient implementation of these methods via the global-QR decomposition.

1. Introduction

Scalar extrapolation methods have been developed to accelerate the convergence of some sequences of numbers in $\mathbb{R}$ or $\mathbb{C}$. This process consists in transforming a sequence $(s_n)$ converging to s to new ones by applying some transformation $T_k$, $k=1,\dots$ defined as follows

$${\displaystyle T_k:s_n⟶T_k^{(n)}=s_n+\sum _{i=1}^{k-1}{a}_i^{(n)}{g}_i(n),}$$

where $g_i(n)$, $i=1,\dots ,k-1$ is a scalar sequence that defines the method. One of the earlier extrapolation method is the well known Aitken’s ${\Delta }^2$ process [1] defined by

$${\displaystyle T_2^{(n)}=s_n-\frac{\Delta s_n}{{\Delta }^2{s}_n},}$$

where the first and the second forward differences are defined by $\Delta s_n=s_{n+1}-s_n$ and ${\Delta }^2{s}_n=\Delta s_{n+1}-\Delta s_n$ and in that case we have $g_1(n)=\Delta s_n$. Such a process was first proposed in 1926 by Aitken in [2]. It is well known that under some assumptions, the new Aitken sequence $T_2^{(n)}$ will converge faster than $(s_n)$ to the same limit s; see [1] for more details. Vector extrapolation methods have been proposed the last decades and among them are the minimal polynomial extrapolation (MPE) [3,4], the reduced rank extrapolation (RRE) method [5] and the modified minimal polynomial extrapolation (MMPE) [6,7]. For more details, see [1,8,9,10,11,12]. A second class of vector sequence transformations contains the topological $ϵ$-algorithm (TEA) [6]. Applications for solving large linear and nonlinear systems of equations have been considered in [8,13].

Vector extrapolation methods were used in many applications such as google page rank by Golub et al. [14,15] and in other fields such as in statistics [16] or for solving discretized Navier–Stokes problems [17]. In the present paper, we consider tensor sequence transformations and propose new tensor extrapolation methods that generalize the classical vector ones. Using the Einstein product, we define some new tensor products that will allow us to develop the new methods based on orthogonal or oblique projections onto subspaces of small dimensions. It will be shown that when the tensor sequence is generated linearly then the proposed methods are theoretically equivalent to some tensor Krylov subspace methods such as the tensor version of GMRES and Lanczos methods developed recently in [18,19].

The remainder of this paper is organized as follows. In Section 2, we give notations, some basic definitions and properties related to tensors. In Section 3, we introduce the tensor versions of the vector polynomial extrapolation methods namely the Tensor Global Reduced Rank Extrapolation (TG-RRE), the Tensor Global Minimal Polynomial Extrapolation (TG-MPE) and the Tensor Global Modified Minimal Polynomial Extrapolation (TG-MMPE). We also give a Global tensor version of the topological $ϵ$-transformation. Section 4 describes the application of the proposed methods for solving linear and nonlinear tensor system of equations. In Section 5, we introduce efficient implementations via the tensor global-QR decomposition and in the last section, we present some numerical experiments.

2. Preliminaries and Notations

In this section, we briefly review some concepts and notions that are used throughout the paper. A tensor is a multidimensional array of data and a natural extension of scalars, vectors and matrices to a higher order, a scalar is a 0th order tensor, a vector is a 1th-order tensor and a matrix is 2th-order tensor. The tensor order is the number of its indices, which is called modes or ways. For a given N-mode tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$, the notation $x_{i_1,\dots ,i_N}$ (with $1\le i_j\le I_j,j=1,\dots N$) stands for element $\left(i_1,\dots ,i_N\right)$ of the tensor $\mathcal{X}$. Corresponding to a given tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N}$, the notation

$${\mathcal{X}}_{\underset{(N-1)-times}{\underbrace{:,:,\dots ,:}},k}k=1,2,\dots ,I_N$$

denotes a tensor in ${\mathbb{R}}^{I_1\times I_2\times \dots \times I_{N-1}}$ which is obtained by fixing the last index and is called frontal slice. Fibers are the higher-order analogue of matrix rows and columns. A fiber is defined by fixing every index but one. A matrix column is a mode-1 fiber and a matrix row is a mode-2 fiber. Figure 1 shows the frontal, horizontal and lateral slices of a third order tensor and also a mode-3 tube fiber.

The n-mode matrix of a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times I_3\dots \times I_N}$ is denoted by $X_{(n)}$ and arranges the mode-n fibers to be the columns of the resulting matrix $X_{(n)}\in {\mathbb{R}}^{I_n\times (I_1\dots I_3\dots I_{n-1}{I}_{n+1}\dots I_N)}$ with $n=1,\dots ,N.$ We have:

$$X_{(n)}(i_n,j)=x_{i_1,\dots ,i_N}$$

where $j=1+{\sum }_{k=1k\ne n}^N(i_k-1)J_k$ with $J_K={\prod }_{m=1,m\ne n}^{k-1}{I}_m.$

An important operation for a tensor is the tensor-matrix multiplication [20], also known as n-mode product of a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ with a matrix $U\in {\mathbb{R}}^{J\times I_n}$, the n-mode product is a tensor of size $I_1\times I_2\times \dots I_{n-1}\times J\times I_{n+1}\times \dots \times I_N$ whose entries are given by:

$${(\mathcal{X}{\times }_{n}U)}_{i_1,\dots ,i_{n-1},j,i_{n+1,\dots },i_N}=\sum _{i_n=1}^{I_n}{x}_{i_1\dots i_N}{u}_{j{i}_n}$$

The n-mode vector product of a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N}$ with a vector $v\in {\mathbb{R}}^{I_n}$ is denote by $(\mathcal{X}{\tilde{\times }}_{n}v)$ is a tensor of size $I_1\times I_2\times \dots I_{n-1}\times I_{n+1}\times \dots \times I_N$ whose entries are given by:

$${(\mathcal{X}{\tilde{\times }}_{n}v)}_{i_1,\dots ,i_{n-1},i_{n+1,\dots },i_N}=\sum _{i_n=1}^{I_n}{x}_{i_1\dots i_N}{v}_{i_n}$$

We recall the definition and some properties of the Einstein product, see [19].

Definition 1.

Let $\mathcal{A}\in {\mathbb{R}}^{I_1\times \dots \times I_L\times K_1\times \dots \times K_N}$, $\mathcal{B}\in {\mathbb{R}}^{K_1\times \dots \times K_N\times J_1\times \dots \times J_M}$, the Einstein product of the tensors $\mathcal{A}$ and $\mathcal{B}$ is the tensor of size $(I_1\times \dots \times I_L\times J_1\times \dots \times J_M)$ defined as:

$${(\mathcal{A}{\ast }_N\mathcal{B})}_{i_1\dots i_L{j}_1\dots j_M}=\sum _{k_1=1}^{K_1}\sum _{k_2=1}^{K_2}\sum _{k_3=1}^{K_3}\dots \sum _{k_N=1}^{K_N}{a}_{i_1\dots i_L{k}_1\dots k_N}{b}_{k_1\dots k_N{j}_1\dots j_M}$$

Definition 2

([19]). Let $\mathcal{A}=a_{i_1,\dots ,i_N,j_1,\dots ,j_M}\in {\mathbb{R}}^{I_1\times \dots \times I_N\times J_1\times \dots \times J_M}$, $\mathcal{B}=b_{i_1,\dots ,i_M,j_1,\dots ,j_N}\in {\mathbb{R}}^{J_1\times \dots \times J_M\times I_1\times \dots \times I_N}$, if $b_{i_1,\dots ,i_M,j_1,\dots ,j_N}=a_{j_1,\dots ,j_N,i_1,\dots ,i_M}$ then $\mathcal{B}$ is called the transpose of $\mathcal{A}$ and denoted by ${\mathcal{A}}^T$.

A tensor $\mathcal{D}=d_{i_1,\dots ,i_N,j_1,\dots ,j_N}\in {\mathbb{C}}^{I_1\times \dots \times I_N\times I_1\times \dots \times I_N}$ is diagonal if $d_{i_1,\dots ,i_N,j_1,\dots ,j_N}=0$ in the case that the indices $i_1,\dots ,i_N$ are different from $j_1,\dots ,j_N$. If all the entries of tensor are zero, we say this tensor is zero tensor, denoted by $\mathcal{O}$.

The trace of the tensor $\mathcal{A}\in {\mathbb{C}}^{I_1\times \dots \times I_N\times I_1\times \dots \times I_N}$ is given by $tr(\mathcal{A})={\sum }_{i_1\dots i_N}{a}_{i_1\dots i_N{i}_1\dots i_N}$.

The inner product of two tensors of the same order $\mathcal{A},\mathcal{B}\in {\mathbb{R}}^{I_1\times \dots \times I_N\times J_1\times \dots \times J_m}$ is given by

$$\langle \mathcal{A},\mathcal{B}\rangle =tr({\mathcal{B}}^T{\ast }_N\mathcal{A}),$$

(1)

and the associated norm is defined by:

$${\left|\right|\mathcal{A}\left|\right|}^2=tr({\mathcal{A}}^T{\ast }_N\mathcal{A}).$$

Definition 3

([21]). A tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times \dots \times I_N\times I_1\times \dots \times I_N}$ is called the inverse of the square tensor $\mathcal{A}\in {\mathbb{R}}^{I_1\times \dots \times I_N\times I_1\times \dots \times I_N}$ and denoted by ${\mathcal{A}}^{-1}$ if it satisfies:

$$\mathcal{A}{\ast }_N\mathcal{X}=\mathcal{X}{\ast }_N\mathcal{A}=\mathcal{I}$$

where $\mathcal{I}$ the identity tensor is called a unit tensor or identity tensor which all this entries are zero except for the diagonal entries ${\mathcal{I}}_{i_1,\dots ,i_N,i_1,\dots ,i_N}=1$.

In the following, we introduce the new ${⊡}^{(N+M+1)}$ product between two tensors which is a generalization of diamond product introduced in [22].

Definition 4.

The ${⊡}^{(N+M+1)}$ tensor product between two $(M+N+1)$-mode tensors $\mathcal{X}=\left[{\mathcal{X}}_1,\dots ,{\mathcal{X}}_{\ell }\right]\in {\mathbb{R}}^{I_1\times I_2\times I_3\dots \times I_N\times J_1\times J_2\times J_3\dots \times J_M\times \ell }$ and $\mathcal{Y}=\left[{\mathcal{Y}}_1,\dots ,{\mathcal{Y}}_p\right]\in {\mathbb{R}}^{I_1\times \dots \times I_N\times J_1\times \dots \times J_M\times p}$ where ${\mathcal{X}}_i\in {\mathbb{R}}^{I_1\times \dots \times I_N\times J_1\times \dots \times J_M}$ and ${\mathcal{Y}}_j\in {\mathbb{R}}^{I_1\times \dots \times I_N\times J_1\times \dots \times J_M}$, is the $\ell \times p$ matrix whose $(i,j)$ entries, $i=1,\dots ,l$ and $j=1,\dots ,p$, is given by

$${(\mathcal{X}{⊡}^{(N+M+1)}\mathcal{Y})}_{i,j}=\langle {\mathcal{X}}_i,{\mathcal{Y}}_j\rangle =tr({\mathcal{Y}}_j^T{\ast }_N{\mathcal{X}}_i).$$

Next, we give a proposition that will be used later.

Proposition 1.

Assume that $\mathcal{U}=[{\mathcal{U}}_1,\dots ,{\mathcal{U}}_m]\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M\times m}$ is an $(N+M+1)$-mode tensor such that ${\mathcal{U}}_1,\dots ,{\mathcal{U}}_m\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$ and let $y={(y_1,\dots ,y_m)}^T\in {\mathbb{R}}^m$. Then for an arbitrary $(N+M+1)$-mode tensor $\mathcal{Y}=[{\mathcal{Y}}_1,\dots ,{\mathcal{Y}}_m]$ such that ${\mathcal{Y}}_1,\dots ,{\mathcal{Y}}_m\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$, we have:

$$\begin{array}{c}\hfill \mathcal{Y}{⊡}^{(N+M+1)}(\mathcal{U}{\tilde{\times }}_{(N+M+1)}y)=(\mathcal{Y}{⊡}^{(N+M+1)}\mathcal{U})y,\end{array}$$

Proof. 

The proof comes easily from the definition of the two involved tensor products. □

Definition 5.

The set of ($N+M$)-mode tensors ${\mathcal{U}}_1,\dots ,{\mathcal{U}}_m\in {\mathbb{R}}^{I_1\times \dots \times I_N\times J_1\times \dots \times J_M}$ is called orthonormal if

$$\langle {\mathcal{U}}_i,{\mathcal{U}}_j\rangle ={\delta }_{i,j}(=1ifi=jand0elswhere).$$

Remark 1.

Suppose that $\mathcal{U}=[{\mathcal{U}}_1,\dots ,{\mathcal{U}}_m]\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M\times m}$ is an $(N+M+1)$-mode tensor such that ${\mathcal{U}}_1,\dots ,{\mathcal{U}}_m\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$. If the ($M+N$)-mode tensors ${\mathcal{U}}_1,\dots ,{\mathcal{U}}_m$ are orthonormal, then we have:

$$\mathcal{U}{⊡}^{(N+M+1)}\mathcal{U}=I_m.$$

3. Tensor Extrapolation Methods

In this section, we present two classes of new tensor global extrapolation methods. The first class contains the tensor polynomial based methods while the second class is devoted to the global tensor topological $ϵ$-algorithm.

3.1. Tensor Global-Polynomial Extrapolation Methods

Let $({\mathcal{S}}_n)$ be a sequence of tensors of $\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M}$ and consider the transformation $T_k$, $k=1,2,\dots$ from ${\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M}$ onto ${\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M}$ and defined by

$$\begin{array}{cc}\hfill T_k^{(n)}=T_k({\mathcal{S}}_n)& ={\mathcal{S}}_n+\sum _{j=0}^{k-1}{\alpha }_j^{(k)}{\mathcal{G}}_j(n)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & ={\mathcal{S}}_n+{\mathcal{H}}_k(n){\tilde{\times }}_{(N+M+1)}{\alpha }^{(k)}\hfill \end{array}$$

(3)

where ${\alpha }^{(k)}={({\alpha }_0^{(k)},\dots ,{\alpha }_{k-1}^{(k)})}^T\in {\mathbb{R}}^k$ and ${\mathcal{H}}_k(n)=[{\mathcal{G}}_0(n),\dots ,{\mathcal{G}}_{k-1}(n)]$ is the (N + M + 1)-mode tensor defined from the given auxiliary tensor sequences ${({\mathcal{G}}_i(n))}_n\in {\mathbb{R}}^{I_1\times \dots \times I_N\times K_1\times \dots \times K_M};i=0,\dots ,k-1$. We will see later how to choose the vector ${\alpha }^{(k)}\in {\mathbb{R}}^k$.

Let ${\tilde{T}}_k$ denote the new transformation obtained from $(T_k)$ as follows:

$$\begin{array}{cc}\hfill {\tilde{T}}_k({\mathcal{S}}_n)& ={\mathcal{S}}_{n+1}+\sum _{j=0}^{k-1}{\alpha }_j^{(k)}{\mathcal{G}}_j(n+1)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & ={\mathcal{S}}_{n+1}+{\mathcal{H}}_k(n+1){\tilde{\times }}_{(N+M+1)}{\alpha }^{(k)}.\hfill \end{array}$$

(5)

Notice that the scalars ${\alpha }_j^{(k)}$ are the same in the expressions of $T_k({\mathcal{S}}_n)$ and ${\tilde{T}}_k({\mathcal{S}}_n)$. We define now the generalized residual of $T_k^{(n)}$ by

$$\begin{array}{cc}\hfill {\tilde{\mathcal{R}}}_k^n=\tilde{\mathcal{R}}(T_k^{(n)})& ={\tilde{T}}_k({\mathcal{S}}_n)-T_k({\mathcal{S}}_n)\hfill \\ \hfill & =({\mathcal{S}}_{n+1}-{\mathcal{S}}_n)+\sum _{j=0}^{k-1}{\alpha }_j^{(k)}({\mathcal{G}}_j(n+1)-{\mathcal{G}}_j(n))\hfill \\ \hfill & =\Delta {\mathcal{S}}_n+\sum _{j=0}^{k-1}{\alpha }_j^{(k)}(\Delta {\mathcal{G}}_j(n)).\hfill \end{array}$$

Then we get

$$\begin{array}{cc}\hfill \tilde{\mathcal{R}}(T_k^{(n)})& =\Delta {\mathcal{S}}_n+\Delta {\mathcal{H}}_k(n){\tilde{\times }}_{(N+M+1)}{\alpha }^{(k)},\hfill \end{array}$$

(6)

where the first forward differences $\Delta {\mathcal{S}}_n={\mathcal{S}}_{n+1}-{\mathcal{S}}_n$ and $\Delta {\mathcal{H}}_k(n)=[\Delta {\mathcal{G}}_0(n),\dots ,\Delta {\mathcal{G}}_{k-1}(n)]\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M\times k}$. The vector ${\alpha }^{(k)}$ is obtained from the orthogonality relation:

$$\begin{array}{c}\hfill \tilde{\mathcal{R}}(T_k^{(n)})\in {(span\{{\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}\})}^{\perp }\end{array}$$

(7)

where ${\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times K_1\times K_2\times \dots \times K_M}$ are given tensors.

Here, the $span\{{\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}\}$ is the tensor subspace generated by the tensors ${\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}$.

If ${\tilde{\mathbf{H}}}_{k,n}$ and ${\tilde{\mathbf{L}}}_{k,n}$ denote the tensor subspaces ${\tilde{\mathbf{H}}}_{k,n}=span\{\Delta {\mathcal{G}}_0(n),\dots ,\Delta {\mathcal{G}}_{k-1}(n)\}$ and ${\tilde{\mathbf{L}}}_{k,n}=span\{{\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}\}$, then from (6) and (7), the generalized residual satisfies the following relations:

$$\tilde{\mathcal{R}}(T_k^{(n)})-\Delta {\mathcal{S}}_n\in {\tilde{\mathbf{H}}}_{k,n}$$

(8)

and

$$\tilde{\mathcal{R}}(T_k^{(n)})\in {({\tilde{\mathbf{L}}}_{k,n})}^{\perp }.$$

(9)

Let ${\mathcal{L}}_{k,n}=[{\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}]\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M\times k}$, then the relations (8) and (9) could be expressed as follows:

$$\tilde{\mathcal{R}}(T_k^{(n)})-\Delta {\mathcal{S}}_n=\Delta {\mathcal{H}}_k(n){\tilde{\times }}_{(N+M+1)}{\alpha }^{(k)}$$

(10)

and

$${\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\tilde{R}(T_k^{(n)})=0.$$

(11)

Assuming that $({\mathcal{L}}_{k,n}{⊡}^{N+M+1}\Delta {\mathcal{H}}_k(n))$ is nonsingular, the vector ${\alpha }^{(k)}$ appearing in the expression (6) of the generalized residual $\tilde{\mathcal{R}}(T_k^{(n)})$ is given by

$$\begin{array}{c}\hfill {\alpha }^{(k)}=-{({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\Delta {\mathcal{H}}_k(n))}^{-1}({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_n).\end{array}$$

(12)

Therefore, the approximation $T_k^{(n)}$ is computed as follows

$$\begin{array}{c}\hfill T_k^{(n)}={\mathcal{S}}_n+{\mathcal{H}}_k(n){\tilde{\times }}_{(N+M+1)}{\alpha }^{(k)}\end{array}$$

(13)

For the tensor global polynomial extrapolation methods, namely Tensor Global MPE (TG-MPE), Tensor Global RRE (TG-RRE) and Tensor Global MMPE (TG-MMPE), the auxiliary sequences are given as

$${\mathcal{G}}_{\ell }^{(n)}=\Delta {\mathcal{S}}_{(n+\ell )},\ell =0,\dots ,k-1;n\ge 0$$

Let ${\Delta }^i{\mathcal{V}}_k(n)$ be the tensor defined by

$${\Delta }^i{\mathcal{V}}_k(n)=[{\Delta }^i{\mathcal{S}}_n,\dots ,{\Delta }^i{\mathcal{S}}_{n+k-1}]\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M\times k},i=1,2,$$

where ${\Delta }^2$ is the second forward difference ${\Delta }^2{\mathcal{S}}_n=\Delta {\mathcal{S}}_{n+1}-\Delta {\mathcal{S}}_n$.

In this case, using the relation (12) and the fact that ${\mathcal{H}}_k(n)=\Delta {\mathcal{V}}_k(n)$, and $\Delta {\mathcal{H}}_k(n)={\Delta }^2{\mathcal{V}}_k(n)$, the approximations $T_k^{(n)}$ given in (13) can be expressed as:

$$\begin{array}{c}\hfill T_k^{(n)}={\mathcal{S}}_n-\Delta {\mathcal{V}}_k(n){\tilde{\times }}_{(N+M+1)}({({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}{\Delta }^2{\mathcal{V}}_k(n))}^{-1}({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_n)).\end{array}$$

(14)

It is clear that $T_k^{(n)}$ exists and is unique if and only if the square matrix ${\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}{\Delta }^2{\mathcal{V}}_k(n)$ is nonsingular. The generalized residual given in the relation (6) can be expressed as follows:

$$\begin{array}{c}\hfill \tilde{\mathcal{R}}(T_k^{(n)})=\Delta {\mathcal{S}}_n-{\Delta }^2{\mathcal{V}}_k(n){\tilde{\times }}_{(N+M+1)}({({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}{\Delta }^2{\mathcal{V}}_k(n))}^{-1}({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_n)).\end{array}$$

(15)

The choice of the tensors ${\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times K_1\times K_2\times \dots \times K_M}$ required in the orthogonality relation (7) determines the global-polynomial tensor extrapolation method. For the three well known extrapolation polynomial-based methods, we have the following choices

$$\begin{array}{cc}\hfill {\mathcal{Y}}_{\ell }^{(n)}=\Delta {\mathcal{S}}_{n+\ell }& \mathrm{for}\text{TG-MPE},\hfill \\ \hfill {\mathcal{Y}}_{\ell }^{(n)}={\Delta }^2{\mathcal{S}}_{n+\ell }& \mathrm{for}\text{TG-RRE},\hfill \\ \hfill {\mathcal{Y}}_{\ell +1}^{(n)}={\mathcal{Y}}_{n+l}& \mathrm{for}\text{TG-MMPE}.\hfill \end{array}$$

Next, we will propose an efficient implementation of these methods. For this purpose, we first express the approximation $T_k^{(n)}$ given in relation (2) in a different way.

Using the relation $(2)$, the fact that ${\mathcal{G}}_{\ell }^{(n)}=\Delta {\mathcal{S}}_{(n+\ell )},\ell =0,\dots ,k-1;n\ge 0$, and $\tilde{\mathcal{R}}(T_k^{(n)})\in {(span\{{\mathcal{Y}}_0^{(n)},\dots ,{\mathcal{Y}}_{k-1}^{(n)}\})}^{\perp }$, the (TG-RRE), (TG-MPE) and (TG-MMPE) extrapolation methods produce approximations $T_k^{(n)}$ of the form

$$\begin{array}{c}\hfill T_k^{(n)}=\sum _{j=0}^k{\gamma }_j^{(k)}{\mathcal{S}}_{n+j},\end{array}$$

(16)

where

$$\sum _{j=0}^k{\gamma }_j^{(k)}=1\mathrm{and}\sum _{j=0}^k{\eta }_{\ell ,j}{\gamma }_j^{(k)}=00\le \ell <k,$$

(17)

with ${\eta }_{\ell ,j}=\langle {\mathcal{Y}}_{\ell }^{(n)},\Delta {\mathcal{S}}_{n+j}\rangle$.

The system of linear Equation (17) can be written as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\gamma }_0^{(k)}+{\gamma }_1^{(k)}+\dots +{\gamma }_k^{(k)}=1\hfill \\ {\gamma }_0^{(k)}\langle {\mathcal{Y}}_0^{(n)},\Delta {\mathcal{S}}_n\rangle +{\gamma }_1^{(k)}\langle {\mathcal{Y}}_0^{(n)},\Delta {\mathcal{S}}_{n+1}\rangle +\dots +{\gamma }_k^{(k)}\langle {\mathcal{Y}}_0^{(n)},\Delta {\mathcal{S}}_{n+k}\rangle =0\hfill \\ {\gamma }_0^{(k)}\langle {\mathcal{Y}}_1^{(n)},\Delta {\mathcal{S}}_n\rangle +{\gamma }_1^{(k)}\langle {\mathcal{Y}}_1^{(n)},\Delta {\mathcal{S}}_{n+1}\rangle +\dots +{\gamma }_k^{(k)}\langle {\mathcal{Y}}_1^{(n)},\Delta {\mathcal{S}}_{n+k}\rangle =0\hfill \\ \dots \dots \dots \dots \dots \dots \dots \hfill \\ {\gamma }_0^{(k)}\langle {\mathcal{Y}}_{k-1}^{(n)},\Delta {\mathcal{S}}_n\rangle +{\gamma }_1^{(k)}\langle {\mathcal{Y}}_{k-1}^{(n)},\Delta {\mathcal{S}}_{n+1}\rangle +\dots +{\gamma }_k^{(k)}\langle {\mathcal{Y}}_{k-1}^{(n)},\Delta {\mathcal{S}}_{n+k}\rangle =0\hfill \end{array}\right.\end{array}$$

(18)

Let ${\beta }_l^{(k)}={\gamma }_0^{(k)}/{\gamma }_k^{(k)}$ for $0\le l\le k$, then

$$\begin{array}{c}\hfill {\gamma }_l^{(k)}=\frac{{\beta }_l^{(k)}}{{\sum }_{l=0}^k{\beta }_l^{(k)}}\mathrm{for}0\le l<k\mathrm{and}{\beta }_k^{(k)}=1.\end{array}$$

(19)

With these notations, the linear system of Equation (18) becomes

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\beta }_0^{(k)}\langle {\mathcal{Y}}_0^{(n)},\Delta {\mathcal{S}}_n\rangle +\dots +{\beta }_{k-1}^{(k-1)}\langle {\mathcal{Y}}_0^{(n)},\Delta {\mathcal{S}}_{n+k-1}\rangle =-\langle {\mathcal{Y}}_0^{(n)},\Delta {\mathcal{S}}_{n+k}\rangle \hfill \\ \dots \dots \dots \dots \dots \dots \dots \hfill \\ {\beta }_0^{(k)}\langle {\mathcal{Y}}_{k-1}^{(n)},\Delta {\mathcal{S}}_n\rangle +\dots +{\beta }_{k-1}^{(k)}\langle {\mathcal{Y}}_{k-1}^{(n)},\Delta {\mathcal{S}}_{n+k-1}\rangle =-\langle {\mathcal{Y}}_{k-1}^{(n)},\Delta {\mathcal{S}}_{n+k}\rangle .\hfill \end{array}\right.\end{array}$$

(20)

The above system of equations can also be expressed in the following compact form

$$\begin{array}{c}\hfill ({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\Delta {\mathcal{V}}_k(n)){\beta }^{(k)}=-({\mathcal{L}}_{k,n}{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_{n+k})\end{array}$$

(21)

where ${\beta }^{(k)}={[{\beta }_0^{(k)},\dots ,{\beta }_{k-1}^{(k)}]}^T$. Assume now that ${\gamma }_0^{(k)},{\gamma }_1^{(k)},\dots ,{\gamma }_k^{(k)}$ have been calculated and introduce the new variables

$$\begin{array}{c}\hfill {\delta }_0^{(k)}=1-{\gamma }_0^{(k)},{\delta }_j^{(k)}={\delta }_{j-1}^{(k)}-{\gamma }_j^{(k)},1\le j<k\mathrm{and}{\delta }_{k-1}^{(k)}={\gamma }_k^{(k)},\end{array}$$

(22)

then the tensor approximation $T_k^{(n)}$ can be expressed as

$$\begin{array}{c}\hfill T_k^{(n)}={\mathcal{S}}_n+\sum _{j=0}^{k-1}{\delta }_j^{(k)}\Delta {\mathcal{S}}_{n+j}={\mathcal{S}}_n+\Delta {\mathcal{V}}_k(n){\tilde{\times }}_{(N+M+1)}{\delta }^{(k)}\end{array}$$

(23)

where ${\delta }^{(k)}={[{\delta }_0^{(k)},\dots ,{\delta }_{k-1}^{(k)}]}^T$.

3.2. The tensor Global Topological $ϵ$-Transformation

In [6], Brezinski proposed a generalization of the scalar $ϵ$-algorithm for vector sequences called the topological $ϵ$-algorithm (TEA). The matrix case has been introduced by Jbilou and Sadok in ([9]). In this section we define the tensor global topological $ϵ$-transformation (TG-TET).

Let $({\mathcal{S}}_n)$ be a sequence of tensors of $\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M}$ and consider approximations ${\mathcal{E}}_k({\mathcal{S}}_n)={\mathcal{E}}_k^{(n)}$ of the limit of the tensor sequence ${({\mathcal{S}}_n)}_{n\in \mathbb{N}}$ such that

$$\begin{array}{c}\hfill {\mathcal{E}}_k^{(n)}={\mathcal{S}}_n+\sum _{i=1}^k{a}_i^{(n)}\Delta {\mathcal{S}}_{n+i-1},n\ge 0.\end{array}$$

(24)

where $a_i^{(n)}\in \mathbb{R}$, for $i=1,\dots k$. We introduce the new tensor transformation ${\tilde{\mathcal{E}}}_{k,j}^{(n)},j=1,\dots k$ defined by

$$\begin{array}{c}\hfill {\tilde{\mathcal{E}}}_{k,j}^{(n)}={\mathcal{S}}_{n+j}+\sum _{i=1}^k{a}_i^{(n)}\Delta {\mathcal{S}}_{n+i+j-1}j=1,\dots ,k.\end{array}$$

(25)

We set ${\tilde{\mathcal{E}}}_{k,0}^{(n)}={\mathcal{E}}_k^{(n)}$ and define the j-th tensor generalized residual as follows

$$\begin{array}{c}\hfill {\tilde{\mathcal{R}}}_j({\mathcal{E}}_k^{(n)})={\tilde{\mathcal{E}}}_{k,j}^{(n)}-{\tilde{\mathcal{E}}}_{k,j-1}^{(n)}.\end{array}$$

(26)

The coefficients $a_i^{(n)}$ involved in the expression (24) of ${\mathcal{E}}_k^{(n)}$ are computed such that each j-th generalized residual is orthogonal to some chosen tensor $\mathcal{Y}\in {\mathbb{R}}^{I_1\times I_2\dots \times I_N\times K_1\times K_2\dots \times K_M}$, that is

$$\begin{array}{c}\hfill \langle \mathcal{Y},{\tilde{\mathcal{R}}}_j({\mathcal{E}}_k^{(n)})\rangle =0;j=1,\dots ,k.\end{array}$$

(27)

Let $D_{k,n}$ denote the following matrix

$$\begin{array}{c}\hfill D_{k,n}=\left(\begin{array}{cccccc}\langle \mathcal{Y},{\Delta }^2{\mathcal{S}}_n\rangle & .& .& .& .& \langle \mathcal{Y},{\Delta }^2{\mathcal{S}}_{n+k-1}\rangle \\ \langle \mathcal{Y},{\Delta }^2{\mathcal{S}}_{n+1}\rangle & .& .& .& .& \langle \mathcal{Y},{\Delta }^2{\mathcal{S}}_{n+k}\rangle \\ .& .& .& .& .& .\\ .& .& .& .& .& .\\ .& .& .& .& .& .\\ \langle \mathcal{Y},{\Delta }^2{\mathcal{S}}_{n+k-1}\rangle & .& .& .& .& \langle \mathcal{Y},{\Delta }^2{\mathcal{S}}_{n+2k-2}\rangle \end{array}\right).\end{array}$$

(28)

Then, from the orthogonality relation (27), $a^{(n)}={(a_1^{(n)},\dots ,a_k^{(n)})}^T\in {\mathbb{R}}^k$ is given by

$$\begin{array}{c}\hfill a^{(n)}=-D_{k,n}^{-1}{z}^{(n)}\end{array}$$

(29)

where $z^{(n)}={(\langle \mathcal{Y},\Delta {\mathcal{S}}_n\rangle ,\dots ,\langle \mathcal{Y},\Delta {\mathcal{S}}_{n+k-1}\rangle )}^T$. We assume that the matrix $D_{k,n}$ is nonsingular. Hence, the approximation ${\mathcal{E}}_k^{(n)}$ exists, is unique and is expressed as

$$\begin{array}{c}\hfill {\mathcal{E}}_k^{(n)}={\mathcal{S}}_n+\Delta {\mathcal{W}}_k(n){\tilde{\times }}_{(N+M+1)}{\beta }^{(n)}\end{array}$$

(30)

where ${\mathcal{W}}_k(n)=\left[\Delta {\mathcal{S}}_n,\dots ,\Delta {\mathcal{S}}_{n+k-1}\right]$. If the matrix $D_{k,n}$ is singular, then the approximation ${\mathcal{E}}_k^{(n)}$ is not defined and in that case we will have a breakdown of the method. One possibility to overcome this drawback is, instead of computing $a^{(n)}$ by using (29), we can solve the least squares problem

$$\underset{a^{(n)}\in {\mathbb{R}}^k}{min}\parallel D_{k,n}{a}^{(n)}+z^{(n)}\parallel .$$

In the next sections, we will see how these tensor global extrapolation methods could be applied for solving linear and nonlinear tensor equations.

4. Application to Tensor Linear/Non Linear Systems of Equations

4.1. Application to Tensor Linear Systems

Consider the following tensor linear system of equations

$$\begin{array}{c}\hfill \mathcal{A}{\ast }_N\mathcal{X}=\mathcal{B},\end{array}$$

(31)

where $\mathcal{A}$ is a tensor in ${\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times I_1\times I_2\times \dots \times I_N}$, $\mathcal{B}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$ and $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$ is the unknown tensor to be determined. We assume here that the square tensor $\mathcal{A}$ is nonsingular so that (31) has a unique solution ${\mathcal{X}}^{\ast }={\mathcal{A}}^{-1}\star \mathcal{B}$. The purpose here is to apply tensor extrapolation methods to the problem (31) to compute approximate solutions to the solution ${\mathcal{X}}^{\ast }$ of (31). Starting from an initial tensor guess ${\mathcal{S}}_0$, we construct the linear sequence ${({\mathcal{S}}_n)}_n$ as follows

$$\begin{array}{c}\hfill {\mathcal{S}}_{n+1}=(\mathcal{I}-\mathcal{A}){\ast }_N{\mathcal{S}}_n+\mathcal{B}\end{array}$$

(32)

Notice that if the sequence ${({\mathcal{S}}_n)}_n$ is convergent, then its limit $\mathcal{S}={\mathcal{X}}^{\ast }$ is the solution of the system (31).

In [18,19], tensor Krylov subspaces via the Einstein product have been introduced including the tensor GMRES and some tensor based Lanczos methods. The next theorem shows that, when applied to the generated linear sequence (32), the proposed tensor extrapolation methods are Tensor Krylov subspace methods and are mathematically equivalent to some well known Krylov based methods such as the tensor GMRES.

Theorem 1.

When applied to the sequence generated by (32), TG-RRE, and TG-MPE are tensor Krylov subspace methods and are mathematically equivalent to the tensor global GMRES and the tensor global Arnoldi methods, respectively.

Proof. 

Notice first that for tensor linear sequences (32) the generalized residual becomes the true residual. In fact, from (32), we have $\Delta {\mathcal{S}}_n=\mathcal{B}-\mathcal{A}{\ast }_N{\mathcal{S}}_n=\mathcal{R}({\mathcal{S}}_n)$ the residual of the tensor ${\mathcal{S}}_n$. Since ${\Delta }^2{\mathcal{S}}_n=-\mathcal{A}{\ast }_N\Delta {\mathcal{S}}_n$ we have ${\Delta }^2{\mathcal{V}}_k(n)=[{\Delta }^2{\mathcal{S}}_n,\dots ,{\Delta }^2{\mathcal{S}}_{n+k-1}]=-\mathcal{A}{\ast }_N\Delta {\mathcal{V}}_k(n)$ where $\Delta {\mathcal{V}}_k(n)=[\Delta {\mathcal{S}}_n,\dots ,\Delta {\mathcal{S}}_{n+k-1}]$. Consequently using (6) and (32), the generalized residual of the approximation $(T_k^{(n)})$ is the true residual

$$\tilde{\mathcal{R}}(T_k^{(n)})=\mathcal{R}(T_k^{(n)})=\mathcal{B}-\mathcal{A}{\ast }_N{T}_k^{(n)}.$$

For simplicity and unless specified otherwise, we set $n=0$, $T_k^{(0)}=T_k$, ${\mathcal{S}}_0={\mathcal{X}}_0$ the initial guess and drop the index n in all our notations. When applied to the sequence generated by the linear relation (32), the TG-RRE, TG-MMPE and the TG-MPE above produce approximations ${\mathcal{X}}_k=T_k$ such that the corresponding residual ${\mathcal{R}}_k=\mathcal{B}-\mathcal{A}{\ast }_N{T}_k$ satisfies the relations

$$\begin{array}{ccc}\hfill {\mathcal{R}}_k-\Delta {\mathcal{S}}_0& \in & {\tilde{\mathbf{H}}}_k=-\mathcal{A}{\ast }_N{\tilde{\mathbf{V}}}_k\hfill \\ \hfill {\mathcal{R}}_k& \in & {({\tilde{\mathbf{L}}}_k)}^{\perp }\hfill \end{array}$$

where ${\tilde{\mathbf{V}}}_k=span\left\{\Delta {\mathcal{S}}_0,\dots ,\Delta {\mathcal{S}}_{k-1}\right\}$ and ${\tilde{\mathbf{L}}}_k\equiv {\tilde{\mathbf{H}}}_k$ for TG-RRE, ${\tilde{\mathbf{L}}}_k\equiv {\tilde{\mathbf{V}}}_k$ for TG-MPE and ${\tilde{\mathbf{L}}}_k\equiv {\tilde{\mathbf{Y}}}_k=span\left\{{\mathcal{Y}}_0,\dots ,{\mathcal{Y}}_{k-1}\right\}$ for TG-MMPE where ${\mathcal{Y}}_0,\dots ,{\mathcal{Y}}_{k-1}$ are some chosen tensors.

Notice that, since ${\tilde{\mathbf{V}}}_k={\mathbb{K}}_m(\mathcal{A},R_0)$ (the tensor Krylov subspace defined in [18,19]), the extrapolation methods above are tensor global Krylov subspace methods. TG-RRE is an orthogonal projection and is theoretically equivalent to the tensor global GMRES while TG-MPE is oblique projection method and is equivalent to the tensor global Arnoldi method. □

As the TG-RRE is an orthogonal projection method, we also have the classical minimization property for the residual

$${\displaystyle \left|\right|{\mathcal{R}}_{TG-RRE}\left|\right|=\underset{T_k\in {\mathcal{X}}_0+{\mathbb{K}}_m(\mathcal{A},R_0)}{min}\left|\right|\mathcal{B}-\mathcal{A}{\ast }_N{T}_k\left|\right|.}$$

When the linear process (32) is convergent, it is more useful in practice to apply the tensor extrapolation methods after some fixed number of iterations. To save memory, we can also use the algorithm in a cycling mode which means that the iterations are restarted after a fixed number m of iterations. Algorithm 1 is summarized as follows:

Algorithm 1 TG-RRE, TG-MPE and TG-MMPE Algorithms

Step 1. $k=0$ choose ${\mathcal{X}}_0$ and the numbers p and m.

Step 2. Basic iteration

Set $T_0={\mathcal{X}}_0$

${\mathcal{Z}}_0=T_0$

${\mathcal{Z}}_{j+1}=(\mathcal{I}-\mathcal{A}){\ast }_N{\mathcal{Z}}_j+\mathcal{B},j=0,\dots ,p-1$

Step 3. Extrapolation schema

${\mathcal{S}}_0={\mathcal{Z}}_p$

${\mathcal{S}}_{n+1}=(\mathcal{I}-\mathcal{A}){\ast }_N{\mathcal{S}}_n+\mathcal{B},n=0,\dots ,m$

Compute the approximation $T_m^{(0)}$ by TG-RRE, TG-MPE or TG-MMPE.

Set ${\mathcal{X}}_0=T_m^{(0)}$, $k=k+1$ and go to step 2.

4.2. Application to Non Linear Tensor Systems

Consider the nonlinear system of tensor equations

$$\begin{array}{c}\hfill \mathrm{G}(\mathcal{X})=\mathcal{B}\end{array}$$

(33)

with $\mathrm{G}(\mathcal{X})$ an operator from ${\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$ onto ${\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$, and ${\mathcal{X}}^{\ast }\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_N\times J_1\times J_2\times \dots \times J_M}$ the solution of the system of Equation (33). For any arbitrary tensor $\mathcal{X}$, the residual is given by

$$\begin{array}{c}\hfill R(\mathcal{X})=\mathrm{G}(\mathcal{X})-\mathcal{X}\end{array}$$

(34)

Let ${({\mathcal{S}}_n)}_n$ be the sequence of tensors generated from an initial guess ${\mathcal{S}}_0$ as follows

$${\mathcal{S}}_{n+1}=\mathrm{G}({\mathcal{S}}_n),j=0,1\dots .$$

To get approximate solutions to a solution of (33), we apply the tensor extrapolation methods above to the sequence $({\mathcal{S}}_n)$. As for the linear case, the different steps are summarized in the following Algorithm 2.

Algorithm 2 TG-RRE, TG-MPE and TG-MMPE for nonlinear systems

Step 1. $k=0$ choose ${\mathcal{X}}_0$ and the numbers p and m.

Step 2. Basic iteration

Set $T_0={\mathcal{X}}_0$

${\mathcal{W}}_0=T_0$

${\mathcal{W}}_{j+1}=\mathrm{G}({\mathcal{W}}_j)j=0,\dots ,p-1$

Step 3. Extrapolation scheme

${\mathcal{S}}_0={\mathcal{W}}_p$

If $\left|\right|{\mathcal{S}}_1-{\mathcal{S}}_0{\left|\right|}_F<ϵ$ stop

otherwise generate ${\mathcal{S}}_{n+1}=\mathrm{G}({\mathcal{S}}_n),n=0,\dots ,m$

Compute the approximation $T_m^{(0)}$ by TG-RRE, TG-MPE or TG-MMPE.

Set ${\mathcal{X}}_0=T_m^{(0)}$, $k=k+1$ and go to step 2.

Remark 2.

As we stated earlier, when applied to linearly generated tensor sequences, the proposed tensor extrapolation methods produce the same iterates as some well known tensor Krylov subspace methods but differ in the way that the approximations are computed. In the case where the process of generating the sequence ${({\mathcal{S}}_n)}_n$ is not known and what is known is only the terms of this sequences, Krylov-based methods or Newton-type methods could not be used and in that case, extrapolation methods are well come. Such a problem can be found for example in some statistical-problems; see for example [16] for the application of the vector ϵ-algorithm of Wynn [23] in the expectation–maximization (EM) algorithm to find maximum likelihood estimates from incomplete or missing data.

5. The Global-QR Implementation of TG-MPE/TG-RRE

The purpose of this section is to give an efficient implementation of TG-MPE and TG-RRE using a generalisation of the technique based on the QR decomposition and given in [11] for vector MPE and vector RRE methods. We first introduce the Tensor Global-QR decomposition. Let $\mathcal{U}\in {\mathbb{R}}^{I_1\times I_2\times \times \dots I_M\times J_1\times J_2\times \dots \times J_N\times k}$, be an $(M+N+1)$-mode tensor with the column tensors ${\mathcal{U}}_0,\dots ,{\mathcal{U}}_{k-1}\in {\mathbb{R}}^{I_1\times I_2\times \dots \times I_M\times J_1\times J_2\times \dots \times J_N}$. Then, there is an $(M+N+1)$-mode orthogonal tensor $\mathcal{Q}=[{\mathcal{Q}}_0,\dots ,{\mathcal{Q}}_{k-1}]\in {\mathbb{R}}^{I_1\times I_2\times \times \dots I_M\times J_1\times J_2\times \dots \times J_N\times k}$ satisfying $\mathcal{Q}{⊡}^{(N+M+1)}\mathcal{Q}=I_{k\times k}$ and an upper triangular matrix $R\in {\mathbb{R}}^{k\times k}$ such that

$$\begin{array}{c}\hfill \mathcal{U}=\mathcal{Q}{\times }_{(M+N+1)}{R}^T\end{array}$$

(35)

The tensor decomposition (35) will be called the Tensor Global-QR (TG-QR) decomposition of $\mathcal{A}$ and is summarized in the following Algorithm 3.

Algorithm 3 The global-QR decomposition

Given ${\mathcal{U}}_0$, and compute the scalar $r_{00}$ and the tensor ${\mathcal{Q}}_0$ by $r_{0,0}={\langle {\mathcal{U}}_0,{\mathcal{U}}_0\rangle }^{\frac{1}{2}}$ and ${\displaystyle {\mathcal{Q}}_0=\frac{{\mathcal{U}}_0}{r_{00}}}$. For $j=1,\dots ,k$ (a) $\mathcal{W}={\mathcal{U}}_j$ (b) For $i=0,\dots ,j-1$ do $r_{i,j}=\langle {\mathcal{Q}}_i,\mathcal{W}\rangle$; $\mathcal{W}=\mathcal{W}-r_{i,j}{\mathcal{Q}}_i$, End for (c) $r_{j,j}={\langle \mathcal{W},\mathcal{W}\rangle }^{\frac{1}{2}}$ (d) ${\displaystyle {\mathcal{Q}}_j=\frac{\mathcal{W}}{r_{j,j}}}$ (e) End for End

To show that the tensor $\mathcal{Q}$ is orthogonal, we just proceed by induction on k. The coefficients of the matrix R are the $r_{i,j}$’s given by Algorithm 3. Next, we give a proposition to be used.

Proposition 2

([24]). Let $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \times \dots I_N}$, $A\in {\mathbb{R}}^{J_n\times I_n}$ and $y\in {\mathbb{R}}^{J_n}$, then we have

$$\mathcal{X}{\times }_{(n)}A{\tilde{\times }}_{(n)}y=\mathcal{X}{\tilde{\times }}_{(n)}(A^{T}y).$$

For simplicity, we set here $n=0$ and then the approximation $T_k^{(0)}$ (defined earlier in (23)) is given by

$$\begin{array}{c}\hfill T_k^{(0)}={\mathcal{S}}_0+\sum _{j=0}^{k-1}{\delta }_j^{(k)}\Delta {\mathcal{S}}_j={\mathcal{S}}_0+\Delta {\mathcal{V}}_k(0){\tilde{\times }}_{(N+M+1)}{\delta }^{(k)}\end{array}$$

Substituting now ${\mathcal{V}}_k={\mathcal{V}}_k(0)=\mathcal{Q}{\times }_{(N+M+1)}{R}^T$ and using Proposition 2, we get

$$\begin{array}{cc}\hfill T_k^{(0)}& ={\mathcal{S}}_0+\Delta {\mathcal{V}}_k{\tilde{\times }}_{(N+M+1)}{\delta }^{(k)}\hfill \\ \hfill & ={\mathcal{S}}_0+\mathcal{Q}{\tilde{\times }}_{(M+N+1)}(R{\delta }^{(k)}),\hfill \end{array}$$

which gives

$$T_k^{(0)}={\mathcal{S}}_0+\sum _{j=0}^{k-1}{\theta }_j{\mathcal{Q}}_j,$$

(36)

where ${\theta }_j$ is the $(j+1)$ component of the column vector $R{\delta }^{(k)}$; the matrix R and the tensors ${\mathcal{Q}}_j\in {\mathbb{R}}^{I_1\times \dots I_M\times J_1\times \dots \times J_N}$, $j=0,\dots ,k-1$ are given by Algorithm 3. To compute ${\delta }^{(k)}$, we have first to compute ${\beta }^{(k)}={({\beta }_0^{(k)},\dots ,{\beta }_{k-1}^{(k)})}^T$ and use the relations given in (22).

For TG-MPE, the coefficients ${\gamma }_l^{(k)}$’s of the vector ${\gamma }^{(k)}$ are determined by computing the vector ${\beta }^{(k)}$, the solution of the linear system of equations (20)

$$(\Delta {\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{V}}_k){\beta }^{(k)}=-(\Delta {\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_k)$$

(37)

where

$$\begin{array}{cc}\hfill (\Delta {\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{V}}_k)& =\left(\begin{array}{cccccc}\langle \Delta {\mathcal{S}}_0,\Delta {\mathcal{S}}_0\rangle & .& .& .& .& \langle \Delta {\mathcal{S}}_0,\Delta {\mathcal{S}}_{k-1}\rangle \\ \langle \Delta {\mathcal{S}}_1,\Delta {\mathcal{S}}_0\rangle & .& .& .& .& \langle \Delta {\mathcal{S}}_1,\Delta {\mathcal{S}}_{k-1}\rangle \\ .& .& .& .& .& .\\ .& .& .& .& .& .\\ .& .& .& .& .& .\\ \langle \Delta {\mathcal{S}}_{k-1},\Delta {\mathcal{S}}_0\rangle & .& .& .& .& \langle \Delta {\mathcal{S}}_{k-1},\Delta {\mathcal{S}}_{k-1}\rangle \end{array}\right)\in {\mathbb{R}}^{k\times k}\hfill \end{array}$$

(38)

and $(\Delta {\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_k)={(\langle \Delta {\mathcal{S}}_0,\Delta {\mathcal{S}}_k\rangle ,\dots ,\langle \Delta {\mathcal{S}}_{k-1},\Delta {\mathcal{S}}_k\rangle )}^T\in {\mathbb{R}}^k$.

For TG-RRE, ${\beta }^{(k)}$ are determined by solving the linear system of equation

$$({\Delta }^2{\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{V}}_k){\beta }^{(k)}=-({\Delta }^2{\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_k)$$

where

$$\begin{array}{cc}\hfill ({\Delta }^2{\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{V}}_k)& =\left(\begin{array}{cccccc}\langle {\Delta }^2{\mathcal{S}}_0,\Delta {\mathcal{S}}_0\rangle & .& .& .& .& \langle {\Delta }^2{\mathcal{S}}_0,\Delta {\mathcal{S}}_{k-1}\rangle \\ \langle {\Delta }^2{\mathcal{S}}_1,\Delta {\mathcal{S}}_0\rangle & .& .& .& .& \langle {\Delta }^2{\mathcal{S}}_1,\Delta {\mathcal{S}}_{k-1}\rangle \\ .& .& .& .& .& .\\ .& .& .& .& .& .\\ .& .& .& .& .& .\\ \langle {\Delta }^2{\mathcal{S}}_{k-1},\Delta {\mathcal{S}}_0\rangle & .& .& .& .& \langle {\Delta }^2{\mathcal{S}}_{k-1},\Delta {\mathcal{S}}_{k-1}\rangle \end{array}\right)\in {\mathbb{R}}^{k\times k},\hfill \end{array}$$

(39)

and $({\Delta }^2{\mathcal{V}}_k{⊡}^{(N+M+1)}\Delta {\mathcal{S}}_k)={(\langle {\Delta }^2{\mathcal{S}}_0,\Delta {\mathcal{S}}_k\rangle ,\dots ,\langle {\Delta }^2{\mathcal{S}}_{k-1},\Delta {\mathcal{S}}_k\rangle )}^T\in {\mathbb{R}}^k$.

Finally, once ${\beta }^{(k)}$ is computed, the coefficients ${\gamma }_l^{(k)}$’s are given by

$${\gamma }_l^{(k)}=\frac{{\beta }_l^{(k)}}{{\displaystyle \sum _{l=0}^k{\beta }_l^{(k)}}}\mathrm{for}0\le l<k\mathrm{and}{\beta }_k^{(k)}=1.$$

(40)

The following Algorithm 4 summarizes the main steps.

Algorithm 4 Implementation of TG-MPE/TG-RRE via the global-QR decomposition

${\mathcal{S}}_0$ is a given initial guess and k is a fixed index. Apply Algorithm 3 to compute the global-QR decomposition of the tensor $\Delta {\mathcal{V}}_k$. Compute ${\beta }^{(k)}$ by solving the linear system (38) or (39). Compute the coefficients ${\gamma }_j^{(k)}$ from (40). Compute ${\delta }^{(k)}$ from the relation (22) and ${\theta }_j$ is $(j+1)$ component of the column vector $R{\delta }^{(k)}$. Compute $T_k^{(0)}$ by ${\displaystyle T_k^{(0)}={\mathcal{S}}_0+\sum _{j=0}^{k-1}{\theta }_j{\mathcal{Q}}_j}$

As a numerical test, we consider the linear system of tensor equations given by (31) where $\mathcal{A}$ is a random tensor $N=3$, $I_1=I_2=20$, $I_3=10$, $J_1=20$, $J_2=10$ and $J_3=5$ as in [19]. The exact solution is the tensor $\mathcal{X}$ whose elements are all one and the right hand side $\mathcal{B}$ is given by $\mathcal{B}=\mathcal{A}{\ast }_N\mathcal{X}$. The computations are carried out using MATLAB 7.4 with machine epsilon about $2·{10}^{-16}$.

We took $m=10$, $p=1$ and stopped the iteration when the relative error norm of the residual was less than ${10}^{-6}$. Then after $k=6$ iterations, Algorithm 4 for the tensor RRE method, gives the relative residual norm ${\displaystyle \frac{{\mathcal{R}}_k}{{\mathcal{R}}_0}=8.5\times {10}^{-7}}$ where the initial guess is ${\mathcal{X}}_0=\mathcal{O}$. We notice that the convergence of the original sequence $({\mathcal{S}}_n)$ is very slow and at $n=200$, we get a relative residual $\mathcal{R}({\mathcal{S}}_n)/{\mathcal{R}}_0$ of ${10}^{-1}$. Generally, extrapolation methods are more effective than Krylov-based methods for nonlinear problems. More experimental studies and applications in some areas such as mutilinear google pagerank should be considered in the future.

6. Conclusions

In this paper, we introduced new tensor global extrapolation methods to accelerate the convergence of some tensor sequences. The proposed methods are generalisations to the tensor case of some well known vector extrapolation methods such as the reduced rank extrapolation or the topological epsilon algorithm. The new methods were defined as orthogonal or oblique projection processes using the Einstein product and also some new interesting tensor products. We showed how to apply the derived algorithms to tensor linear and nonlinear systems of tensor equations. Application to some interesting problems such as the multilinear page rank is still under investigation.