Exploiting Generalized Cyclic Symmetry to Find Fast Rectangular Matrix Multiplication Algorithms Easier

Abstract

The quest to multiply two large matrices as fast as possible is one that has already intrigued researchers for several decades. However, the ‘optimal’ algorithm for a certain problem size is still not known. The fast matrix multiplication (FMM) problem can be formulated as a non-convex optimization problem—more specifically, as a challenging tensor decomposition problem. In this work, we build upon a state-of-the-art augmented Lagrangian algorithm, which formulates the FMM problem as a constrained least squares problem, by incorporating a new, generalized cyclic symmetric (CS) structure in the decomposition. This structure decreases the number of variables, thereby reducing the large search space and the computational cost per iteration. The constraints are used to find practical solutions, i.e., decompositions with simple coefficients, which yield fast algorithms when implemented in hardware. For the FMM problem, usually a very large number of starting points are necessary to converge to a solution. Extensive numerical experiments for different problem sizes demonstrate that including this structure yields more ‘unique’ practical decompositions for a fixed number of starting points. Uniqueness is defined relative to the known scale and trace invariance transformations that hold for all FMM decompositions. Making it easier to find practical decompositions may lead to the discovery of faster FMM algorithms when used in combination with sufficient computational power. Lastly, we show that the CS structure reduces the cost of multiplying a matrix by itself.

1. Introduction

Matrix multiplication is a fundamental operation in numerical linear algebra, with applications ranging from machine learning to signal processing. Discovering faster algorithms for this task can therefore have a wide-reaching impact. For very large systems, saving a few percent of computation time can be worth millions of dollars. The study of fast matrix multiplication (FMM) began with Strassen’s breakthrough in 1969 [1], and despite decades of progress, many questions remain open. Recently, new algorithms have been discovered using deep reinforcement learning [2], while an augmented Lagrangian (AL) method has been proposed to obtain more stable and efficient decompositions [3]. Existing optimization-based approaches for discovering FMMs [2,4,5,6,7,8] often require significant computational resources and manual tuning, making them difficult to replicate. In this work, we propose a framework that does not require big computational resources and does not require manual tinkering, enabling the discovery of FMM algorithms in a more efficient and reproducible manner.

The fast matrix multiplication (FMM) problem aims for faster ways to multiply large matrices by rewriting the bilinear equations of matrix multiplication as a tensor equation:

$$\begin{array}{cccccc}\hfill \mathbf{Z}(i,j)& =\sum _{k=1}^K\mathbf{X}(i,k)\mathbf{Y}(k,j)\hfill & \hfill & \to \hfill & \hfill {\mathbf{Z}}^{\top }& =\sum _{i,j,k=1}^{I,J,K}{〈{\mathbf{E}}_{ik}^{I\times K},\mathbf{X}〉}_{\mathrm{F}}{〈{\mathbf{E}}_{kj}^{K\times J},\mathbf{Y}〉}_{\mathrm{F}}{\mathbf{E}}_{ji}^{J\times I}\hfill \\ \hfill & \hfill & & \hfill & & =:\left(\sum _{i,j,k=1}^{I,J,K}{\mathbf{E}}_{ik}^{I\times K}\otimes {\mathbf{E}}_{kj}^{K\times J}\otimes {\mathbf{E}}_{ji}^{J\times I}\right){·}_{\mathrm{F}}\left(\mathbf{X},\mathbf{Y}\right),\hfill \end{array}$$

(1)

where $\mathbf{Z}:=\mathbf{X}\mathbf{Y}$ and ${\mathbf{E}}_{ij}^{I\times J}$, for $i=1,\dots ,I$, and $j=1,\dots ,J$, is a basis of matrices in ${\mathbb{R}}^{I\times J}$ such that ${\mathbf{E}}_{ij}^{I\times J}(i_1,j_2)=1$, if $(i_1,j_2)=(i,j)$, and zero otherwise. ‘$\otimes$’ denotes the tensor or outer product, and ${\langle ·\rangle }_{\mathrm{F}}$ and ‘·_F’ indicate Frobenius inner products. A transpose is added in (1) such that the tensor that is defined has additional interesting properties, such as cyclic symmetry (CS), which will be discussed further in Section 2.3.2.

The rank of matrix multiplication is the minimal integer R such that

$$\begin{array}{cc}\hfill {\mathbf{Z}}^{\top }& =\sum _{r=1}^R{〈{\mathbf{U}}_r,\mathbf{X}〉}_{\mathrm{F}}{〈{\mathbf{V}}_r,\mathbf{Y}〉}_{\mathrm{F}}{\mathbf{W}}_r,\hfill \end{array}$$

(2)

for some matrices ${\mathbf{U}}_r\in {\mathbb{R}}^{I\times K}$, ${\mathbf{V}}_r\in {\mathbb{R}}^{K\times J}$, and ${\mathbf{W}}_r\in {\mathbb{R}}^{J\times I}$. It can be shown that minimizing R minimizes the computational complexity of matrix multiplication [9] (Proposition 15.1) and [10]. More specifically, the number of arithmetic operations needed to multiply two matrices in ${\mathbb{R}}^{I\times I}$ is $\mathcal{O}\left(I^{\omega }\right)$, where $\omega :={log}_{I}R$. Since $R<I^3$, it holds that $\omega <3$. Additionally, the exponent is bounded from below by two, since you need at least $I^2$ operations to compute as many elements.

Minimizing R in (2) corresponds to finding a canonical polyadic decomposition (CPD) of the matrix multiplication tensor (MMT) defined implicitly in (1) [1]:

$$\begin{array}{cc}\hfill {\mathcal{T}}_{IKJ}& :=\sum _{i,j,k=1}^{I,J,K}{\mathbf{e}}_{ik}^{IK}\otimes {\mathbf{e}}_{kj}^{KJ}\otimes {\mathbf{e}}_{ji}^{IJ},\hfill \end{array}$$

(3)

where ${\mathcal{T}}_{IKJ}$ has dimensions $IK\times KJ\times IJ$ and ${\mathbf{e}}_{ik}^{IK}:=\mathrm{vec}\left({\mathbf{E}}_{ik}^{I\times J}\right)$, where $\mathrm{vec}\left(·\right)$ denotes vectorization along the columns. An MMT is a sparse tensor consisting of $IJK$ ones and zeros elsewhere:

$$\begin{array}{cc}\hfill {\mathcal{T}}_{IKJ}(i_1+(k_2-1)I,k_1+(j_2-1)K,j_1+(i_2-1)J)& =\left\{\begin{array}{cc}1\hfill & \mathrm{if}{i}_1=i_2,j_1=j_2,k_1=k_2,\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(4)

for all $i_1,i_2=1,\dots ,I$, $j_1,j_2=1,\dots ,J$, and $k_1,k_2=1,\dots ,K$. An MMT thus only depends on the size of the matrices that are multiplied. A polyadic decomposition (PD) of ${\mathcal{T}}_{IKJ}$ of length R decomposes ${\mathcal{T}}_{IKJ}$ into R rank-1 tensors:

$$\begin{array}{c}\hfill {\mathcal{T}}_{IKJ}=\sum _{r=1}^R{\mathbf{u}}_r\otimes {\mathbf{v}}_r\otimes {\mathbf{w}}_r,\end{array}$$

(5)

where ${\mathbf{u}}_r,{\mathbf{v}}_r$, and ${\mathbf{w}}_r$ are vectors in ${\mathbb{R}}^{IK}$, ${\mathbb{R}}^{KJ}$, and ${\mathbb{R}}^{IJ}$, respectively. These vectors can be collected in three so-called factor matrices: $\mathbf{U}:=\left[{\mathbf{u}}_1,\cdots ,{\mathbf{u}}_R\right]$, $\mathbf{V}:=\left[{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_R\right]$, and $\mathbf{W}:=\left[{\mathbf{w}}_1,\cdots ,{\mathbf{w}}_R\right]$. The rank $R^{\ast }$ or $R^{\ast }\left(\mathcal{T}\right)$ denotes the minimal R for which (5) holds. The decomposition is then called canonical (CPD). A PD with R terms is denoted by ${\mathrm{PD}}_R$ and ${\mathrm{PD}}_R\left(\mathcal{T}\right)$ denotes a ${\mathrm{PD}}_R$ of a certain tensor $\mathcal{T}$. Equations (4) and (5) are also known as the Brent equations [11].

Decompositions of MMTs yield base algorithms for FMM of the form

$$\begin{array}{cc}\hfill \mathrm{vec}\left({\mathbf{Z}}^{\top }\right)& =\sum _{r=1}^R\langle {\mathbf{u}}_r,\mathrm{vec}\left(\mathbf{X}\right)\rangle \langle {\mathbf{v}}_r,\mathrm{vec}\left(\mathbf{Y}\right)\rangle {\mathbf{w}}_r,\hfill \end{array}$$

(6)

which is the same equation as (2) with ${\mathbf{u}}_r=\mathrm{vec}\left({\mathbf{U}}_r\right)$, ${\mathbf{v}}_r=\mathrm{vec}\left({\mathbf{V}}_r\right)$, and ${\mathbf{w}}_r=\mathrm{vec}\left({\mathbf{W}}_r\right)$, for $r=1,\dots ,R$. The total number of operations of a base algorithm is higher than for the standard algorithm. However, the active multiplications, i.e., the number of multiplications between (linear combinations of) elements of $\mathbf{X}$ and $\mathbf{Y}$, is reduced from $IJK$ to R. Algorithm (6) becomes faster when applied sufficiently many times recursively to multiply large matrices. This is because the active multiplications determine the asymptotic complexity [12]. In this case, the large matrices, e.g., of size $I^l\times K^l$ and $K^l\times J^l$ for some $l\in \mathbb{N}$, are divided into $I\times K$ and $K\times J$ blocks of equal size, e.g., $I^{l-1}\times K^{l-1}$ and $K^{l-1}\times J^{l-1}$, respectively. All operations in (6) are then directly performed on the blocks. For example, the vectorization operator stacks the blocks in a block vector of length $IK$ and $JK$, respectively, where each element is again a matrix. The inner products are calculated on these block vectors, where each scalar multiplication becomes a scalar element-wise multiplication on all elements of each block. Each multiplication between the inner products becomes another matrix multiplication of smaller dimension and can be further divided into smaller blocks if possible. A more detailed analysis of the recursive application of FMM algorithms is given in Section 2.2.

The rank of ${\mathcal{T}}_{IKJ}$, denoted by $R^{\ast }\left({\mathcal{T}}_{IKJ}\right)$, is not known for most problem parameters I, J, and K, which is one of the main difficulties of the FMM problem. One of the exceptions is ${\mathcal{T}}_{222}$, of which the rank is known to be seven [13] and the problem is fully understood. We call ${\mathrm{PD}}_7\left({\mathcal{T}}_{222}\right)$ discovered by Strassen ${\mathrm{PD}}_{\mathrm{Strassen}}$ [1]. Only lower and upper bounds on the rank exist for most other I, J, and K. For example, it is known that $R^{\ast }\left({\mathcal{T}}_{333}\right)$ lies between 19 [14] and 23 [4,6,7,15]. The upper bound, i.e., the lowest rank of ${\mathcal{T}}_{IKJ}$ known, is denoted by $\tilde{R}\left({\mathcal{T}}_{IKJ}\right)$. The most recent overview table of $\tilde{R}\left({\mathcal{T}}_{IKJ}\right)$ is given in [16] (Table 3). However, since then, there has been some further improvement for $I=K=J=4$ [17]. Note that permuting $(I,J,K)$ does not change the rank. Recently, there has also been an interest in finding complex-valued decompositions of ${\mathcal{T}}_{IKJ}$ [16,18]. However, as the number of standard multiplications to multiply two complex numbers is at least three, the complexity is higher than for real decompositions of ${\mathcal{T}}_{IKJ}$. A recent overview of different techniques to obtain base algorithms for different dimensions is given in [19] (Section 1.1).

To find a ${\mathrm{PD}}_R$ of ${\mathcal{T}}_{IKJ}$, the following nonlinear least squares (NLS) problem in ${\mathbf{x}}^{\top }:=\left[\begin{array}{ccc}\mathrm{vec}{\left(\mathbf{U}\right)}^{\top }& \mathrm{vec}{\left(\mathbf{V}\right)}^{\top }& \mathrm{vec}{\left(\mathbf{W}\right)}^{\top }\end{array}\right]$ is formulated [4,5,7,11,20]:

$$\begin{array}{cc}\hfill & \underset{\mathbf{x}}{min}\underset{=:f\left(\mathbf{x}\right)}{\underbrace{{\displaystyle \frac{1}{2}}{∥{\sum }_{r=1}^R{\mathbf{w}}_r\otimes {\mathbf{v}}_r\otimes {\mathbf{u}}_r-\mathrm{vec}\left({\mathcal{T}}_{IKJ}\right)∥}^2}},\hfill \end{array}$$

(7)

where ‘⊗’ denotes the Kronecker product and $\parallel ·\parallel$ the ${\ell }_2$-norm. Note that ${\mathbf{x}}^{\ast }$ is a ${\mathrm{PD}}_R$ of ${\mathcal{T}}_{IKJ}$ if and only if $f\left({\mathbf{x}}^{\ast }\right)=0$. However, state-of-the-art optimization algorithms are not guaranteed to converge to a global optimum of (7). Furthermore, the convergence can be slow, even for second-order methods. One of the reasons is that (7) is non-convex, and PDs of MMT are not unique and have, furthermore, additional invariances compared with generic PDs [21]. This means that the minima of (7) are non-isolated.

In practice, the rank $R^{\ast }$ is approximated experimentally by the lowest length $\tilde{R}$ for which a solution ${\mathbf{x}}^{\ast }$ with $f\left({\mathbf{x}}^{\ast }\right)=0$ can be obtained using numerical optimization. However, as no globally convergent algorithm for (7) exists, this does not mean that a solution with a lower rank does not exist as long as there is a gap with the theoretical lower bound.

Another difficulty is the existence of border rank PDs (BRPDs) [10,12]. BRPDs are ill-conditioned approximate decompositions of $T_{IKJ}$ with elements that grow to infinity during convergence. The border rank of a particular tensor can be smaller than its canonical rank. This is closely related to the concept of degeneracy [22,23]. The BRPDs of MMTs are, e.g., investigated in [4,24,25,26,27,28]. To eliminate the convergence to BRPDs, constraints are usually added to (7) [3,4,5,8] or the elements are restricted to a discrete set of values [2,6]. The phenomenon of degeneracy also gives rise to regions of very slow convergence in the optimization landscape, informally known as swamps [29,30,31].

Additionally, the number of variables or unknowns grows rapidly with I, J, and K. Therefore, this paper proposes a new structure that can be enforced in PDs of MMTs to reduce the number of variables, which directly reduces the computational complexity of optimization algorithms to solve (7) and shrinks the large search space.

To solve (7), the alternating least squares (ALS) method is frequently used [4,7,20,32]. The local convergence of the ALS method for (7) is known under the assumption that the Hessian is positive definite at the solutions modulo the scaling indeterminacies [33]. This is, however, not satisfied for PDs of MMTs, as they have additional invariances [21]. In [4,5], a constrained optimization problem and corresponding method are proposed to improve the convergence. More specifically, Lagrange multipliers and a quadratic penalty are used for the constraints, and a Levenberg–Marquardt (LM) and an ALS method, respectively, are used to solve the constrained problems. Still, these methods were unable to find ${\mathrm{PD}}_{49}\mathrm{s}$ for $4\times 4$ matrix multiplication, although they are known to exist. Another constrained optimization problem was proposed recently [3]:

$$\underset{\mathbf{x}}{min}f\left(\mathbf{x}\right),\mathrm{s}.\mathrm{t}.h\left(\mathbf{x}\right)=0,l\le \mathbf{x}\le u,$$

(8)

together with an augmented Lagrangian (AL) method to solve it. The bound constraint ensures that the method does not converge to BRPDs. Different equality constraints $h\left(\mathbf{x}\right)$ were proposed to obtain PDs with simple coefficient decompositions. The AL method has as advantage compared with adding quadratic penalty terms to (7) that even outside the neighborhood of an optimum, the constraints are satisfied accurately. New ${\mathrm{PD}}_{49}\left({\mathcal{T}}_{444}\right)\mathrm{s}$, and different new CS PDs are obtained using this method and the stability of existing algorithms is improved [3]. As inner optimization algorithm, the LM method is used, which is a (second-order) damped Gauss–Newton optimization algorithm that optimizes over all factor matrices simultaneously [34].

We use the constrained problem formulation (8) together with the AL method proposed in [3] to find new PDs of ${\mathcal{T}}_{IKJ}$ with the generalized CS structure proposed in this paper. The structure is imposed to find new decompositions more easily by reducing the search space and computational complexity per iteration, which can then, in turn, be used to reduce the complexity of matrix multiplication. Although it is not known whether for all I, J, and K, a CPD of ${\mathcal{T}}_{IKJ}$ with the proposed structure exists, the structure is very flexible, as it is determined by two parameters that can be varied and, for example, can be made smaller to be less restrictive. So far, we have not encountered any CPDs of ${\mathcal{T}}_{IKJ}$ of small dimensions that do not admit this structure for any combination of these two parameters.

1.1. Contribution

A generalization of the CS structure investigated in [7,20,35] to non-square matrix multiplication is proposed for decompositions of MMTs. The advantage of investigating this structure is threefold.

Firstly, this structure reduces the number of variables, thereby shrinking the large search space and speeding up the computation time per iteration when used in an optimization algorithm. We have implemented the structure efficiently in a state-of-the-art all-at-once optimization algorithm for the FMM problem and use a constrained problem formulation to find new practical solutions automatically without manual tinkering. A large number of numerical experiments illustrate that including this structure helps find more practical solutions for a fixed number of starting points.

Secondly, we prove that by exploiting this structure, the cost of multiplying a matrix by itself can be reduced. We give an example with Strassen’s decomposition, which shows that the CS structure enables the speed-up of standard matrix multiplication already from matrices of size $I=J=K=2^8=256$ instead of $2^{10}=1024$.

Lastly, ‘unique’ decompositions of MMTs, based on the well-known invariance transformations [21], are of theoretical interest. Different classes of decompositions have already been investigated before [2,3,7,20,36], but not for rectangular fast matrix multiplication. To determine if two decompositions are equivalent, a non-trivial optimization problem arises. Using the result from [3], where it was shown that the rank of the Jacobian matrix is also invariant under these transformations, we use the different ranks to distinguish ‘unique’ practical decompositions.

1.2. Organization

The paper is organized as follows. Section 2 provides the preliminaries related to the FMM problem. In Section 3, the new CS structure is proposed, and its advantages are discussed. Section 4 presents a large number of numerical experiments for different MMTs, illustrating that incorporating this structure helps to find more (unique) practical decompositions per fixed number of starting points.

2. Preliminaries

This section first discusses the notation and afterwards what is meant by practical decompositions and FMM algorithms. Section 2.3 gives a summary of the most important properties of decompositions of MMTs: invariances, cyclic symmetry, and recursive decompositions.

2.1. Notation

Scalars are denoted by lowercase letters, vectors by bold letters, matrices by bold uppercase letters, and tensors by calligraphic letters, e.g., a, $\mathbf{a}$, $\mathbf{A}$, and $\mathcal{A}$. The ith column of $\mathbf{A}$ is denoted by ${\mathbf{a}}_i$ and the jth element of $\mathbf{a}$ by $a_j$. ‘$\otimes$’ denotes the tensor or outer product, ‘⊗’ the Kronecker product, ∗ the Hadamard or element-wise product, and $\mathrm{vec}\left(·\right)$ the vectorization operator. Vectorization of a tensor is performed such that $\mathrm{vec}\left(\mathcal{T}\right)(i+(j-1)J+(k-1)JK)=\mathcal{T}(i,j,k)$. $\mathbf{I}$ denotes the identity matrix of the appropriate size.

2.2. Practical Algorithms

To multiply larger matrices, e.g., of size $I^l\times K^l$ and $K^l\times J^l$, a base algorithm for $(I,J,K)$ matrix multiplication can be applied recursively, which requires $\mathcal{O}\left(c{R}^l\right)$ floating-point operations. The value of c depends on how many nonzero elements appear in the decomposition. When c and l are sufficiently small and large, respectively, the computational complexity of standard matrix multiplication, i.e., $\mathcal{O}\left({\left(IKJ\right)}^l\right)$ operations, can be lowered. The following result is a generalization of the operation count of Strassen’s decomposition in [1] to general FMM decompositions.

Proposition 1 

(cost fast matrix multiplication). Let $\mathbf{U}$, $\mathbf{V}$, and $\mathbf{W}$ be the factor matrices of a base algorithm for $(I,K,J)$ matrix multiplication. The complexity to use this base algorithm l times recursively to multiply two matrices $X\in {\mathbb{R}}^{I^l\times K^l}$ and $Y\in {\mathbb{R}}^{K^l\times J^l}$ is

$$\begin{array}{c}\hfill \left(1+c_{\mathbf{U}}{\displaystyle \frac{1}{R-IK}}+c_{\mathbf{V}}{\displaystyle \frac{1}{R-KJ}}+c_{\mathbf{W}}{\displaystyle \frac{1}{R-IJ}}\right)R^l-c_{\mathbf{U}}{\displaystyle \frac{{\left(IK\right)}^l}{R-IK}}-c_{\mathbf{V}}{\displaystyle \frac{{\left(KJ\right)}^l}{R-KJ}}-c_{\mathbf{W}}{\displaystyle \frac{{\left(IJ\right)}^l}{R-IJ}},\end{array}$$

where $c_{\mathbf{U}}$ and $c_{\mathbf{V}}$ are the number of additions and scalar multiplications needed to compute the inner products with ${\mathbf{u}}_r$ and ${\mathbf{v}}_r$ in (6), respectively, and $c_{\mathbf{W}}$ is the number of additions and scalar multiplications with the ${\mathbf{w}}_r$.

Proof. 

A base algorithm (6) of rank R of ${\mathcal{T}}_{IKJ}$ can be applied on the $I^{l-1}\times I^{l-1}$ submatrices of $\mathbf{X}$ and the $K^{l-1}\times J^{l-1}$ submatrices of $\mathbf{Y}$, where each active multiplication is then again a multiplication of $\left(I^{l-1},K^{l-1},J^{l-1}\right)$. The cost to multiply $\mathbf{X}$ and $\mathbf{Y}$ can thus be written as

$$\mathrm{cost}\left(I^l,K^l,J^l\right)=R·\mathrm{cost}\left(I^{l-1},K^{l-1},J^{l-1}\right)+c_{\mathbf{U}}·{\left(IK\right)}^{l-1}+c_{\mathbf{V}}·{\left(KJ\right)}^{l-1}+c_{\mathbf{W}}·{\left(IJ\right)}^{l-1}.$$

Applying this formula recursively until $\mathrm{cost}\left(1,1,1\right)=1$, leads to

$$\begin{array}{cc}\hfill \mathrm{cost}\left(I^l,K^l,J^l\right)& =R^l+\sum _{i=1}^l{R}^{i-1}\left(c_{\mathbf{U}}·{\left(IK\right)}^{l-i}+c_{\mathbf{V}}·{\left(KJ\right)}^{l-i}+c_{\mathbf{W}}·{\left(IJ\right)}^{l-i}\right)\hfill \\ \hfill & =R^l+c_{\mathbf{U}}{\left(IK\right)}^{l-1}\sum _{i=0}^{l-1}{\left({\displaystyle \frac{R}{IK}}\right)}^i+c_{\mathbf{V}}{\left(KJ\right)}^{l-1}\sum _{i=0}^{l-1}{\left({\displaystyle \frac{R}{KJ}}\right)}^i+c_{\mathbf{W}}{\left(IJ\right)}^{l-1}\sum _{i=0}^{l-1}{\left({\displaystyle \frac{R}{IJ}}\right)}^i\hfill \\ \hfill & =R^l+c_{\mathbf{U}}{\displaystyle \frac{R^l-{\left(IK\right)}^l}{R-IK}}+c_{\mathbf{V}}{\displaystyle \frac{R^l-{\left(KJ\right)}^l}{R-KJ}}+c_{\mathbf{W}}{\displaystyle \frac{R^l-{\left(IJ\right)}^l}{R-IJ}}\hfill \end{array}$$

□

Remark: The constants $c_{\mathbf{U}}$, $c_{\mathbf{V}}$, and $c_{\mathbf{W}}$ depend implicitly on the parameters I, J, and K in the sense that each factor matrix contains at least $IJK$ and at most $IKR$, $JKR$, and $IJR$ nonzero elements, respectively. Furthermore, we can assume that each factor vector contains at least one nonzero, and the rows of $\mathbf{U}$, $\mathbf{V}$, and $\mathbf{W}$ contain at least I, J, and K nonzeros each, respectively. This can be derived from the specific locations of the ones in the tensor as given in (4). The constants $c_{\mathbf{U}}$, $c_{\mathbf{V}}$, and $c_{\mathbf{W}}$ are directly related to the number of nonzeros, but the exact relation depends on the specific distribution of the nonzeros and their values.

The constants $c_{\mathbf{U}}$, $c_{\mathbf{V}}$, and $c_{\mathbf{W}}$ can be decreased by using sparse PDs with elements that are powers of 2 because multiplication with a power of 2 is not a costly operation when implemented in hardware. Such PDs are called practical because, without small constants, standard matrix multiplication is only improved asymptotically.

A disadvantage of formulation (7) is that the solutions have floating-point elements. Consequently, if a numerical PD of ${\mathcal{T}}_{IKJ}$ is obtained, it has to be transformed to a practical one. The well-known inv-transformations [21] can be used for this purpose [5,8]. However, not all numerical PDs of ${\mathcal{T}}_{IKJ}$ can be transformed to a practical one [36]. Furthermore, this is not an easy process and involves a lot of manual tinkering [5,8]. In [36], this process is called discretization. Another possibility is to add constraints to the optimization problem, such as the following simple equality constraint proposed in [3]:

$$h_{\mathrm{discr}}\left(\mathbf{x}\right):=\mathbf{x}\ast (\mathbf{x}+\mathbf{1})\ast (\mathbf{x}-\mathbf{1}),$$

(9)

where ‘∗’ denotes element-wise multiplication. This constraint forces the elements to the discrete set $\{0,\pm 1\}$.

2.3. Properties of Decompositions of Matrix Multiplication Tensors

This section first briefly discusses the invariances for decompositions of ${\mathcal{T}}_{IKJ}$. Afterwards, other properties such as cyclic symmetry and recursive PDs are discussed in more detail.

2.3.1. Invariance Transformations

Factor matrices are a non-unique representation of a PD. For example, the same tensor is obtained under the permutation of the rank-1 tensors. Additionally, the factor vectors can be scaled as ${\alpha }_r{\mathbf{u}}_r,{\beta }_r{\mathbf{v}}_r,{\displaystyle \frac{1}{{\alpha }_r{\beta }_r}}{\mathbf{w}}_r$, for all $r=1,\dots ,R$, and ${\alpha }_r,{\beta }_r$ in ${\mathbb{R}}_0$.

Decompositions of ${\mathcal{T}}_{IKJ}$ also have other invariances [21]. See, e.g., [3] for an overview. The combination of the trivial transformations and the specific transformations that hold for decompositions of ${\mathcal{T}}_{IKJ}$ is called inv-transformations. They have dimension $I^2+K^2+J^2+2R-3$. Two PDs of ${\mathcal{T}}_{IKJ}$ that can be transformed into one another in this way are called inv-equivalent. All ${\mathrm{PD}}_7\left({\mathcal{T}}_{222}\right)$ are known to be inv-equivalent [13]. Some decompositions of ${\mathcal{T}}_{IKJ}$ also have additional invariances; see, e.g., [3].

The Jacobian matrix can be used to investigate the inv-equivalence of decompositions of MMTs and to discover additional invariances [3]. That is why in Section 4, the size and ranks of the Jacobian matrix at solutions with the proposed structure are given.

2.3.2. Cyclic Symmetry

Because of (3), ${\mathcal{T}}_{IKJ}$ is a structured tensor. More specifically, when $I=J=K$, ${\mathcal{T}}_{IKJ}$ is so-called cyclic symmetric (CS), which means that ${\mathcal{T}}_{III}(k,i,j)={\mathcal{T}}_{III}(j,k,i)={\mathcal{T}}_{III}(i,j,k)$, for $i,j,k=1,\dots ,I^2$. This structure can be exploited in the factor matrices. More specifically, the rank-1 tensors can be forced to either occur in CS pairs or to be symmetric [35]:

$${\mathcal{T}}_{III}=\sum _{s=1}^S{\mathbf{a}}_s\otimes {\mathbf{a}}_s\otimes {\mathbf{a}}_s+\sum _{t=1}^T\left({\mathbf{b}}_t\otimes {\mathbf{c}}_t\otimes {\mathbf{d}}_t+{\mathbf{c}}_t\otimes {\mathbf{d}}_t\otimes {\mathbf{b}}_t+{\mathbf{d}}_t\otimes {\mathbf{b}}_t\otimes {\mathbf{c}}_t\right),$$

where ${\mathbf{a}}_s,{\mathbf{b}}_t,{\mathbf{c}}_t$, and ${\mathbf{d}}_t$ are in ${\mathbb{R}}^{I^2}$, for $s=1,\dots ,S$ and $t=1,\dots ,T$. Here, S and T count the symmetric rank-1 tensors and the CS rank-1 tensors, respectively. The length of a CS PD equals $R=3T+S$. The matrices $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$ can be defined as containing the vectors ${\mathbf{a}}_s,{\mathbf{b}}_t,{\mathbf{c}}_t$, and ${\mathbf{d}}_t$ as columns, and the factor matrices can be parameterized as $\mathbf{U}=\left[\begin{array}{cccc}\mathbf{A}& \mathbf{B}& \mathbf{C}& \mathbf{D}\end{array}\right]$, $\mathbf{V}=\left[\begin{array}{cccc}\mathbf{A}& \mathbf{D}& \mathbf{B}& \mathbf{C}\end{array}\right]$, and $\mathbf{W}=\left[\begin{array}{cccc}\mathbf{A}& \mathbf{C}& \mathbf{D}& \mathbf{B}\end{array}\right]$. The number of variables is only a third of the original value, making this parameterization highly useful for the FMM problem. Furthermore, the cost function can be changed to

$$\begin{array}{cc}\hfill \underset{\mathbf{A},\mathbf{B},\mathbf{C},\mathbf{D}}{min}{\displaystyle \frac{1}{2}}(& \sum _{i=1}^{I^2}\sum _{j=i}^{I^2}\sum _{k=i+1}^{I^2}({\mathcal{T}}_{III}(i,j,k)-\sum _{s=1}^S{a}_{is}{a}_{js}{a}_{ks}\hfill \\ \hfill -& \sum _{t=1}^T\left(b_{it}{d}_{jt}{c}_{kt}+c_{it}{b}_{jt}{d}_{kt}+d_{it}{c}_{jt}{b}_{kt}\right){)}^2\hfill \\ \hfill +& \sum _{i=1}^{I^2}{\left({\mathcal{T}}_{III}(i,i,i)-\sum _{s=1}^S{a}_{is}^3-3\sum _{t=1}^T\left(b_{it}{d}_{it}{c}_{it}\right)\right)}^2),\hfill \end{array}$$

(10)

where $a_{is}:=A(i,s)$, $b_{it}:=B(i,t)$, and similarly for $c_{it}$ and $d_{it}$, which reduces the number of rows in the Jacobian matrix used in NLS optimization methods to solve (10) from $I^6$ to ${\displaystyle \frac{1}{3}}(I^6-I^2)+I^2$. Note that this cost function only includes one element for each CS pair of points because the CS structure already ensures that these elements are equal.

Different practical CS decompositions of rank 7 and 23 of ${\mathcal{T}}_{222}$ and ${\mathcal{T}}_{333}$, respectively, are known [7,20]. They were discovered through an ALS method and analyzed using group theory and algebraic geometry. Concerning Strassen’s decomposition, $S=1$ and $T=2$. Although all ${\mathrm{PD}}_7\left({\mathcal{T}}_{222}\right)\mathrm{s}$ are known to be inv-equivalent, Strassen’s decomposition can be transformed into another practical PD with parameters $(S,T)=(4,1)$. However, this increases the number of nonzeros and is thus not used in practice.

2.4. Decompositions Obtained by Recursion

As mentioned in Section 2.2, an FMM algorithm applies a base algorithm for small I, J, and K to large matrices, e.g., of size $I^l\times K^l$ and $K^l\times J^l$. In the same way, we can also obtain PDs of rank 49 of ${\mathcal{T}}_{444}$ using a ${\mathrm{PD}}_7\left({\mathcal{T}}_{222}\right)$ one time recursively. The ${\mathrm{PD}}_{49}\left({\mathcal{T}}_{444}\right)$ that is obtained using ${\mathrm{PD}}_{\mathrm{Strassen}}$ is called ${\mathrm{PD}}_{\mathrm{Strassen}}^{\mathrm{rec}}$ in the rest of the text. In the following proposition, the formulas to obtain the factor matrices of a recursive PD from the factor matrices of the original PD are given. This result is well known, e.g., from the Supplementary Material of [2] and in [37], and a proof can be found, e.g., in [38] (Section 2.2.3).

Proposition 2 

(Recursive PD). If $\mathbf{U}$, $\mathbf{V}$, and $\mathbf{W}$ are the factor matrices of a ${\mathrm{PD}}_R\left({\mathcal{T}}_{IKJ}\right)$, then the factor matrices ${\mathbf{U}}^{\prime },{\mathbf{V}}^{\prime }$, and ${\mathbf{W}}^{\prime }$, where

$$\begin{array}{ccccc}\hfill {\mathbf{u}}_r^{\prime }& :=\mathrm{vec}\left({\mathbf{U}}_{r_1}\otimes {\mathbf{U}}_{r_2}\right),\hfill & & \hfill {\mathbf{v}}_r^{\prime }& :=\mathrm{vec}\left({\mathbf{V}}_{r_1}\otimes {\mathbf{V}}_{r_2}\right),\hfill \\ \hfill {\mathbf{w}}_r^{\prime }& :=\mathrm{vec}\left({\mathbf{W}}_{r_1}\otimes {\mathbf{W}}_{r_2}\right),\hfill & & \hfill r& :=r_2+(r_1-1)R,\hfill \end{array}$$

where ‘⊗’ denotes the Kronecker product, for $r_1,r_2=1,\dots ,R$, are the factor matrices of a ${\mathrm{PD}}_{R^2}\left({\mathcal{T}}_{I^2{K}^2{J}^2}\right)$.

The following corollary gives the CS parameters and factor matrices of a recursive PD as a function of the original PD.

Corollary 3. 

If $\mathbf{U}$, $\mathbf{V}$, and $\mathbf{W}$ are CS factor matrices of a ${\mathrm{PD}}_R\left({\mathcal{T}}_{III}\right)$ with CS parameters S and T such that $\mathbf{U}=\left[\mathbf{A}\mathbf{B}\mathbf{C}\mathbf{D}\right]$, $\mathbf{V}=\left[\mathbf{A}\mathbf{D}\mathbf{B}\mathbf{C}\right]$, and $\mathbf{W}=\left[\mathbf{A}\mathbf{C}\mathbf{D}\mathbf{B}\right]$, where $\mathbf{A}\in {\mathbb{R}}^{I^2\times S}$ and $\mathbf{B},\mathbf{C},\mathbf{D}\in {\mathbb{R}}^{I^2\times T}$, then the factor matrices ${\mathbf{U}}^{\prime }$, ${\mathbf{V}}^{\prime }$, and ${\mathbf{W}}^{\prime }$ of the recursive decomposition ${\mathrm{PD}}_{R^2}^{\mathrm{rec}}\left({\mathcal{T}}_{I^2{I}^2{I}^2}\right)$ is also CS with parameters $S^{\prime }=S^2$ and $T^{\prime }=T(S+R)$:

$$\begin{array}{cccccccc}\hfill {\mathbf{U}}^{\prime }& :=\left[\begin{array}{c}{\mathbf{A}}^{\prime }{\mathbf{B}}^{\prime }{\mathbf{C}}^{\prime }{\mathbf{D}}^{\prime }\end{array}\right],\hfill & & \hfill {\mathbf{V}}^{\prime }& :=\left[\begin{array}{c}{\mathbf{A}}^{\prime }{\mathbf{D}}^{\prime }{\mathbf{B}}^{\prime }{\mathbf{C}}^{\prime }\end{array}\right],\hfill & & \hfill {\mathbf{W}}^{\prime }& :=\left[\begin{array}{c}{\mathbf{A}}^{\prime }{\mathbf{C}}^{\prime }{\mathbf{D}}^{\prime }{\mathbf{B}}^{\prime }\end{array}\right],\hfill \end{array}$$

where ${\mathbf{A}}^{\prime }\in {\mathbb{R}}^{I^4\times S^{\prime }}$ and ${\mathbf{B}}^{\prime },{\mathbf{C}}^{\prime },{\mathbf{D}}^{\prime }\in {\mathbb{R}}^{I^4\times T^{\prime }}$, and, more specifically,

$$\begin{array}{cccccccc}\hfill {\mathbf{A}}_s^{\prime }& :={\mathbf{A}}_{s_1}\otimes {\mathbf{A}}_{s_2},\hfill & & \hfill s& :=s_2+(s_1-1)S,\hfill & & \hfill s_1,s_2& =1,\dots ,S,\hfill \end{array}$$

and

$$\begin{array}{cccccccc}\hfill {\mathbf{B}}^{\prime }& :=\left[\begin{array}{cc}{\mathbf{B}}_1^{\prime }& {\mathbf{B}}_2^{\prime }\end{array}\right],\hfill & & \hfill {\mathbf{C}}^{\prime }& :=\left[\begin{array}{cc}{\mathbf{C}}_1^{\prime }& {\mathbf{C}}_2^{\prime }\end{array}\right],\hfill & & \hfill {\mathbf{D}}^{\prime }& :=\left[\begin{array}{cc}{\mathbf{D}}_1^{\prime }& {\mathbf{D}}_2^{\prime }\end{array}\right],\hfill \end{array}$$

where

$$\begin{array}{cccccccc}\hfill {\mathbf{B}}_{1,t_1}^{\prime }& :={\mathbf{A}}_{s_1}\otimes {\mathbf{B}}_t,\hfill & & \hfill {\mathbf{C}}_{1,t_1}^{\prime }& :={\mathbf{A}}_{s_1}\otimes {\mathbf{C}}_t,\hfill & & \hfill {\mathbf{D}}_{1,t_1}^{\prime }& :={\mathbf{A}}_{s_1}\otimes {\mathbf{D}}_t,\hfill \\ \hfill {\mathbf{B}}_{2,t_2}^{\prime }& :={\mathbf{B}}_t\otimes {\mathbf{U}}_r,\hfill & & \hfill {\mathbf{C}}_{2,t_2}^{\prime }& :={\mathbf{C}}_t\otimes {\mathbf{W}}_r,\hfill & & \hfill {\mathbf{D}}_{2,t_2}^{\prime }& :={\mathbf{D}}_t\otimes {\mathbf{V}}_r,\hfill \end{array}$$

(11)

where $t_1:=t+(s_1-1)T$ and $t_2:=r+(t-1)R$, for $t=1,\dots ,T$ and $r=1,\dots ,R$.

Proof. 

From Proposition 2 follows that ${\mathbf{U}}_{i^{\prime }}^{\prime }={\mathbf{U}}_{i_1}\otimes {\mathbf{U}}_{i_2}$, ${\mathbf{V}}_{i^{\prime }}^{\prime }={\mathbf{V}}_{i_1}\otimes {\mathbf{V}}_{i_2}$, and ${\mathbf{W}}_{i^{\prime }}^{\prime }={\mathbf{W}}_{i_1}\otimes {\mathbf{W}}_{i_2}$, where $i^{\prime }:=i_2+(i_1-1)R$. Thus, the symmetric part of the recursive PD must satisfy ${\mathbf{U}}_{i_1}={\mathbf{V}}_{i_1}={\mathbf{W}}_{i_1}$ and ${\mathbf{U}}_{i_2}={\mathbf{V}}_{i_2}={\mathbf{W}}_{i_2}$. Consequently, $i_1$ and $i_2$ must be smaller than or equal to S, such that ${\mathbf{U}}_{i_1}={\mathbf{V}}_{i_1}={\mathbf{W}}_{i_1}={\mathbf{A}}_{i_1}$ and ${\mathbf{U}}_{i_2}={\mathbf{V}}_{i_2}={\mathbf{W}}_{i_2}={\mathbf{A}}_{i_2}$. For the other values of $i_1$ and $i_2$, the columns still appear in CS pairs, and they can be rearranged as in (11) to satisfy the CS structure. □

Note that, since ${\mathrm{PD}}_{\mathrm{Strassen}}$ satisfies $S=1$ and $T=2$, the parameters of ${\mathrm{PD}}_{\mathrm{Strassen}}^{\mathrm{rec}}$ are $S=1$ and $T=16$. Another CS ${\mathrm{PD}}_{49}\left({\mathcal{T}}_{444}\right)$ is obtained with $(S,T)=(16,11)$ by using a ${\mathrm{PD}}_7\left({\mathcal{T}}_{222}\right)$ with $(S,T)=(4,1)$ recursively.

3. Generalization Cyclic Symmetry

In this section, a generalization of the CS structure discussed in Section 2.3.2 to the case when I, J, and K are not equal is proposed. We do this to reduce the number of variables and reduce the search space to speed up the convergence. The basis of the idea is the observation that an MMT for non-square matrix multiplication is still CS as long as the indices in (4) are smaller than the smallest dimension. However, because of the interaction of these CS indices with the non-symmetric indices in a CPD of the tensor, imposing this structure on all rank-1 tensors would be too restrictive. Hence, some unstructured terms are added as well. In Section 4, we give experimental results for the maximum number of structured rank-1 terms for different problem sizes. Because, e.g., a PD of ${\mathcal{T}}_{KJI}$ can be obtained from a PD of ${\mathcal{T}}_{IKJ}$, we assume in this chapter that $I\le K\le J$. It is known from (4) that

$$\begin{array}{cc}\hfill & {\mathcal{T}}_{IKJ}(i_1+(k_2-1)I,k_1+(j_2-1)K,j_1+(i_2-1)J)\hfill \\ \hfill & =\sum _{i=1}^r{\mathbf{U}}_i(i_1,k_2){\mathbf{V}}_i(k_1,j_2){\mathbf{W}}_i(j_1,i_2)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{i}_1=i_2,j_1=j_2,k_1=k_2\hfill \\ 0\hfill & \mathrm{else}\hfill \end{array}\right.,\hfill \end{array}$$

for all $i_1,i_2=1,\dots ,I$, $j_1,j_2=1,\dots ,J$, and $k_1,k_2=1,\dots ,K$. Consequently, as long as $i_1,i_2,j_1,j_2,k_1,k_2\le I$, it holds that

$$\begin{array}{cc}\hfill & {\mathcal{T}}_{IKJ}(i_1+(k_2-1)I,k_1+(j_2-1)K,j_1+(i_2-1)J)\hfill \\ \hfill =& {\mathcal{T}}_{IKJ}(k_1+(j_2-1)I,j_1+(i_2-1)K,i_1+(k_2-1)J)\hfill \\ \hfill =& {\mathcal{T}}_{IKJ}(j_1+(i_2-1)I,i_1+(k_2-1)K,k_1+(j_2-1)J).\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill \sum _{r=1}^R{\mathbf{U}}_r(i_1,k_2){\mathbf{V}}_r(k_1,j_2){\mathbf{W}}_r(j_1,i_2)& =\sum _{r=1}^R{\mathbf{U}}_r(k_1,j_2){\mathbf{V}}_r(j_1,i_2){\mathbf{W}}_r(i_1,k_2)\hfill \\ \hfill & =\sum _{r=1}^R{\mathbf{U}}_r(j_1,i_2){\mathbf{V}}_r(i_1,k_2){\mathbf{W}}_r(k_1,j_2),\hfill \end{array}$$

which can be used to include a CS structure in a subpart of the factor matrices. An illustration is shown in Figure 1, where the following submatrices are defined as a function of the position in the factor matrices:

$$\begin{array}{c}\hfill {\mathbf{U}}_r(i_1,k_2)=:\left\{\begin{array}{cc}{\mathbf{A}}_r(i_1,k_2)\hfill & \mathrm{if}{k}_2\le I,r\le S\hfill \\ {\mathbf{B}}_{r-S}(i_1,k_2)\hfill & \mathrm{if}{k}_2\le I,S<r\le S+T\hfill \\ {\mathbf{C}}_{r-S-T}(i_1,k_2)\hfill & \mathrm{if}{k}_2\le I,S+T<r\le S+2T\hfill \\ {\mathbf{D}}_{r-S-2T}(i_1,k_2)\hfill & \mathrm{if}{k}_2\le I,S+2T<r\le R_{\mathrm{CS}}\hfill \\ {\tilde{\mathbf{U}}}_r(i_1,k_2-I)\hfill & \mathrm{if}{k}_2>I,r\le R_{\mathrm{CS}}\hfill \\ {\dot{\mathbf{U}}}_{r-R_{\mathrm{CS}}}(i_1,k_2)\hfill & \mathrm{if}{R}_{\mathrm{CS}}<r\hfill \end{array}\right.,\end{array}$$

where $R_{\mathrm{CS}}:=S+3T$, and for $r=1,\dots ,R$, $i_1=1,\dots ,I$, and $k_2=1,\dots ,K$,

$$\begin{array}{c}\hfill {\mathbf{V}}_r(k_1,j_2)=:\left\{\begin{array}{cc}{\mathbf{A}}_r(k_1,j_2)\hfill & \mathrm{if}{k}_1,j_2\le I,r\le S\hfill \\ {\mathbf{D}}_{r-S}(k_1,j_2)\hfill & \mathrm{if}{k}_1,j_2\le I,S<r\le S+T\hfill \\ {\mathbf{B}}_{r-S-T}(k_1,j_2)\hfill & \mathrm{if}{k}_1,j_2\le I,S+T<r\le S+2T\hfill \\ {\mathbf{C}}_{r-S-2T}(k_1,j_2)\hfill & \mathrm{if}{k}_1,j_2\le I,S+2T<r\le R_{\mathrm{CS}}\hfill \\ {\widehat{\mathbf{V}}}_r(k_1-I,j_2)\hfill & \mathrm{if}{k}_1>I,j_2\le I,r\le R_{\mathrm{CS}}\hfill \\ {\tilde{\mathbf{V}}}_r(k_1,j_2-I)\hfill & \mathrm{if}{j}_2>I,r\le R_{\mathrm{CS}}\hfill \\ {\dot{\mathbf{V}}}_{r-R_{\mathrm{CS}}}(k_1,j_2)\hfill & \mathrm{if}{R}_{\mathrm{CS}}<r\hfill \end{array}\right.,\end{array}$$

for $k_1=1,\dots ,K$, and $j_2=1,\dots ,J$, and

$$\begin{array}{c}\hfill {\mathbf{W}}_r(j_1,i_2)=:\left\{\begin{array}{cc}{\mathbf{A}}_r(j_1,i_2)\hfill & \mathrm{if}{j}_1\le I,r\le S\hfill \\ {\mathbf{C}}_{r-S}(j_1,i_2)\hfill & \mathrm{if}{j}_1\le I,S<r\le S+T\hfill \\ {\mathbf{D}}_{r-S-T}(j_1,i_2)\hfill & \mathrm{if}{j}_1\le I,S+T<r\le S+2T\hfill \\ {\mathbf{B}}_{r-S-2T}(j_1,i_2)\hfill & \mathrm{if}{j}_1\le I,S+2T<r\le R_{\mathrm{CS}}\hfill \\ {\widehat{\mathbf{W}}}_r(j_1-I,i_2)\hfill & \mathrm{if}{j}_1>I,r\le R_{\mathrm{CS}}\hfill \\ {\dot{\mathbf{W}}}_{r-R_{\mathrm{CS}}}(j_1,i_2)\hfill & \mathrm{if}{R}_{\mathrm{CS}}<r\hfill \end{array}\right.,\end{array}$$

for $j_1=1,\dots ,J$, and $i_2=1,\dots ,I$, where ${\mathbf{A}}_s:=\mathrm{reshape}\left({\mathbf{a}}_s,I\times I\right)$, for $s=1,\dots ,S$, ${\mathbf{B}}_t:=\mathrm{reshape}\left({\mathbf{b}}_t,I\times I\right)$, for $t=1,\dots ,T$, and similarly for ${\mathbf{C}}_t$ and ${\mathbf{D}}_t$, and

$$\begin{array}{cc}\hfill {\tilde{\mathbf{U}}}_r& :=\mathrm{reshape}\left(\tilde{\mathbf{U}}(:,r),I\times (K-I)\right),\hfill \\ \hfill {\tilde{\mathbf{V}}}_r& :=\mathrm{reshape}\left(\tilde{\mathbf{V}}(:,r),K\times (J-I)\right),\hfill \\ \hfill {\widehat{\mathbf{V}}}_r& :=\mathrm{reshape}\left(\widehat{\mathbf{V}}(:,r),(K-I)\times I\right),\hfill \\ \hfill {\widehat{\mathbf{W}}}_r& :=\mathrm{reshape}\left(\widehat{\mathbf{W}}(:,r),(J-I)\times I\right),\hfill \end{array}$$

for $r=1,\dots ,R_{\mathrm{CS}}$, and

$$\begin{array}{cc}\hfill {\dot{\mathbf{U}}}_r& :=\mathrm{reshape}\left(\dot{\mathbf{U}}(:,r),I\times K\right),\hfill \\ \hfill {\dot{\mathbf{V}}}_r& :=\mathrm{reshape}\left(\dot{\mathbf{V}}(:,r),K\times J\right),\hfill \\ \hfill {\dot{\mathbf{W}}}_r& :=\mathrm{reshape}\left(\dot{\mathbf{W}}(:,r),J\times I\right),\hfill \end{array}$$

for $r=1,\dots ,(R-R_{\mathrm{CS}})$, and thus $\tilde{\mathbf{U}}\in {\mathbb{R}}^{I(K-I)\times R_{\mathrm{CS}}}$, $\tilde{\mathbf{V}}\in {\mathbb{R}}^{K(J-I)\times R_{\mathrm{CS}}}$, $\widehat{\mathbf{V}}\in {\mathbb{R}}^{(K-I)I\times R_{\mathrm{CS}}}$, $\widehat{\mathbf{W}}\in {\mathbb{R}}^{(J-I)I\times R_{\mathrm{CS}}}$, $\dot{\mathbf{U}}\in {\mathbb{R}}^{IK\times (R-R_{\mathrm{CS}})}$, $\dot{\mathbf{V}}\in {\mathbb{R}}^{KJ\times (R-R_{\mathrm{CS}})}$, and $\dot{\mathbf{W}}\in {\mathbb{R}}^{IJ\times (R-R_{\mathrm{CS}})}$. The proportions in Figure 1 are those of ${\mathcal{T}}_{234}$, $(S,T):=(2,2)$, and $R:=20$. We show in the experiments that such PDs indeed exist. Note that, contrary to the case when I, J, and K are equal, the CS rank $R_{\mathrm{CS}}$ does not equal the rank R, because this would be too restrictive. Also, in the experiments that follow, we did not find PDs for which both are equal. This is likely because of the interaction of the CS part with the parts $\tilde{\mathbf{U}}$, $\tilde{\mathbf{V}}$, $\widehat{\mathbf{V}}$, and $\widehat{\mathbf{W}}$. However, including this structure still reduces the number of parameters from $(IK+KJ+IJ)R$ to $(IK+KJ+IJ)R-2I^2{R}_{\mathrm{CS}}$.

Example 4. 

The optimal CPD (in terms of stability and number of nonzeros) for ${\mathcal{T}}_{223}$ is an extension of Strassen’s decomposition [1,39]:The CS part can clearly be recognized. Remark that the matrices $\tilde{\mathbf{V}}$ and $\widehat{\mathbf{W}}$ are zero in this case. However, this is not always the case, as shown for example in Section 4. Similarly, the optimal ${\mathrm{PD}}_{14}\left({\mathcal{T}}_{224}\right)s$ and ${\mathrm{PD}}_{18}\left({\mathcal{T}}_{225}\right)s$ can also be written with $(S,T)=(1,2)$.

3.1. Computational Complexity Optimization

The most computationally expensive step in a Gauss–Newton-type method, such as the LM method used in Section 4, is solving the linear system for the step, which requires $\mathcal{O}\left({\left(R\left(IK+KJ+IJ\right)\right)}^3\right)$ operations [40]. By incorporating the generalized CS structure, the complexity is directly reduced to $\mathcal{O}\left({\left(R(IK+KJ+IJ)-2I^2{R}_{\mathrm{CS}}\right)}^3\right)$ operations. For square matrix multiplication and when $R_{\mathrm{CS}}=R$, the complexity is reduced by a factor of $3^3$.

3.2. Complexity of Multiplying a Matrix with Itself

When multiplying a matrix with itself, i.e., $\mathbf{Z}:=\mathbf{X}\mathbf{X}$, the CS structure can be exploited to save computations.

Proposition 5. 

Let $\mathbf{U}$, $\mathbf{V}$, and $\mathbf{W}$ be the factor matrices of a base algorithm for $(I,I,I)$ matrix multiplication. When $\mathbf{U}$, $\mathbf{V}$, and $\mathbf{W}$ are CS with parameters S and T, and $R_{\mathrm{CS}}=R$, and $\mathbf{Z}:=\mathbf{X}\mathbf{X}$, with $\mathbf{X}\in {\mathbb{R}}^{I^l\times I^l}$, then the computational cost of applying the base algorithm l times recursively derived in Proposition 1 is reduced to

$$\begin{array}{cc}\hfill {\mathrm{cost}}_{\mathrm{CS}}\left(I^l,I^l,I^l\right)& =2\left(1+{\displaystyle \frac{c_{\mathbf{U}}}{\delta }}\right)R^l-\left(2c_{\mathbf{U}}+\delta \right){\displaystyle \frac{I^{2l}}{\delta }}\hfill \\ \hfill & =\mathrm{cost}\left(I^l,I^l,I^l\right)-c_{\mathbf{U}}\left({\displaystyle \frac{R^l-I^{2l}}{\delta }}\right),\hfill \end{array}$$

where $\delta :=R-I^2>0$ and $c_{\mathbf{U}}$ is the number of additions and scalar multiplications needed to compute the inner products with ${\mathbf{u}}_r$ in (6).

Proof. 

When $\mathbf{Y}=\mathbf{X}$ in the base algorithm (6) and when the decomposition is CS with $R_{\mathrm{CS}}=R$, the inner products with ${\mathbf{v}}_r$ are the same as those with ${\mathbf{u}}_r$ but in a different order:

$$\begin{array}{cc}\hfill \mathrm{vec}\left({\mathbf{Z}}^{\top }\right)& =\sum _{r=1}^R\langle {\mathbf{u}}_r,\mathrm{vec}\left(\mathbf{X}\right)\rangle \langle {\mathbf{v}}_r,\mathrm{vec}\left(\mathbf{X}\right)\rangle {\mathbf{w}}_r\hfill \\ \hfill & =\sum _{s=1}^S\langle {\mathbf{a}}_s,\mathrm{vec}\left(\mathbf{X}\right)\rangle \langle {\mathbf{a}}_s,\mathrm{vec}\left(\mathbf{X}\right)\rangle {\mathbf{a}}_s+\sum _{t=1}^T(\langle {\mathbf{b}}_t,\mathrm{vec}\left(\mathbf{X}\right)\rangle \langle {\mathbf{d}}_t,\mathrm{vec}\left(\mathbf{X}\right)\rangle {\mathbf{c}}_t\dots \hfill \\ \hfill & \langle {\mathbf{c}}_t,\mathrm{vec}\left(\mathbf{X}\right)\rangle \langle {\mathbf{b}}_t,\mathrm{vec}\left(\mathbf{X}\right)\rangle {\mathbf{d}}_t+\langle {\mathbf{d}}_t,\mathrm{vec}\left(\mathbf{X}\right)\rangle \langle {\mathbf{c}}_t,\mathrm{vec}\left(\mathbf{X}\right)\rangle {\mathbf{b}}_t).\hfill \end{array}$$

Consequently, the recursive formula for the cost is

$$\mathrm{cost}\left(I^l,I^l,I^l\right)=R·\mathrm{cost}\left(I^{l-1},I^{l-1},I^{l-1}\right)+c_{\mathbf{U}}·{\left(I^2\right)}^{l-1}+c_{\mathbf{W}}·{\left(I^2\right)}^{l-1}.$$

Because of the CS, the elements in $\mathbf{W}$ are the same as those of $\mathbf{U}$ in a different order. Consequently, $c_{\mathbf{W}}=c_{\mathbf{U}}+R-I^2$. The recursive formula can be expanded in the same way as in the proof of Proposition 1 to obtain:

$$\begin{array}{cc}\hfill {\mathrm{cost}}_{\mathrm{CS}}\left(I^l,I^l,I^l\right)& =\left(1+c_{\mathbf{U}}{\displaystyle \frac{1}{R-I^2}}+c_{\mathbf{W}}{\displaystyle \frac{1}{R-I^2}}\right)R^l-c_{\mathbf{U}}{\displaystyle \frac{I^{2l}}{R-I^2}}-c_{\mathbf{W}}{\displaystyle \frac{I^{2l}}{R-I^2}}\hfill \\ \hfill & =\mathrm{cost}\left(I^l,I^l,I^l\right)-c_{\mathbf{V}}{\displaystyle \frac{R^l}{R-I^2}}+c_{\mathbf{V}}{\displaystyle \frac{I^{2l}}{R-I^2}}\hfill \\ \hfill & =\mathrm{cost}\left(I^l,I^l,I^l\right)-c_{\mathbf{U}}\left({\displaystyle \frac{R^l-I^{2l}}{R-I^2}}\right)\hfill \\ \hfill & =\left(1+\left(2c_{\mathbf{U}}+R-I^2\right){\displaystyle \frac{1}{R-I^2}}\right)R^l-\left(2c_{\mathbf{U}}+R-I^2\right){\displaystyle \frac{I^{2l}}{R-I^2}}\hfill \\ \hfill & =2\left(1+c_{\mathbf{U}}{\displaystyle \frac{1}{R-I^2}}\right)R^l-\left(2c_{\mathbf{U}}+R-I^2\right){\displaystyle \frac{I^{2l}}{R-I^2}}.\hfill \end{array}$$

□

Example 6 

(Complexity Strassen with and without CS). Strassen’s decomposition for $I=2$ has rank 7 and 12 nonzeros in each factor matrix. Thus $c_{\mathbf{U}}=c_{\mathbf{V}}=12-R=5$, and $c_{\mathbf{W}}=12-I^2=8$. The number of operations required as a function of the number of recursive levels l is shown in Figure 2. When exploiting the CS in Strassen’s decomposition, the algorithm improves standard matrix multiplication for the multiplication of a matrix with itself already after eight recursive levels, thus for matrices of size $2^8=256$ or higher. Without CS, there is an improvement over standard matrix multiplication only from size $2^{10}=1024$, and the improvement is less significant.

4. Numerical Experiments

This section presents the results obtained using the AL method [3] to find PDs of MMTs with the generalized CS structure, for various matrix sizes (I, K, J) and different CS parameters S and T. The rank is always chosen as the lowest rank $\tilde{R}$ for which a solution is known to exist in the literature. For the first two experiments, this is also known to be the optimal one [28,41].

The AL method requires upper and lower bounds on the elements in the decomposition: $l\le \mathbf{x}\le u$. We set $u:=-l:=1$ and generate 50 random starting points of magnitude ${10}^{-2}$ using the built-in $\mathtt{randn}$ function in Matlab, for each set of problem parameters:

$${\mathbf{x}}_0:={10}^{-2}·\mathtt{randn}\left((IK+KJ+IJ)R-2I^2{R}_{\mathrm{CS}}\times 1\right).$$

The number of inner iterations of the LM method was set to 50, and the number of outer iterations to 15. The tolerances on the gradient and the constraint were set to ${10}^{-13}$.

When a numerical solution with a cost function value smaller than ${10}^{-12}$ is found, the constraint $h_{\mathrm{discr}}$ from (9), scaled by a factor of 0.1, is added to the optimization problem to convert the numerical solution into a practical one. It is known that not all numerical solutions can be converted into practical ones using inv-transformations [36]. Although we do not use inv-transformations but rather numerical optimization, we also observed that many numerical solutions do not converge to a discrete solution under this constraint, while others converge very quickly. We verified that many of the non-converging numerical solutions also fail to satisfy the necessary conditions for discretizability from [36].

The rank of the Jacobian matrix ${\mathbf{J}}_{\mathrm{CS},\mathrm{gen}}$ of the generalized structure at the discrete solutions is computed. The rank deficiency reflects the dimension of the invariance transformation applicable to the decomposition. Two decompositions with different ranks cannot be transformed into one another using inv-transformations [3]. Thus, the more distinct ranks that are found, the more ‘unique’ solutions are discovered.

The MMTs are ordered according to their best-known rank in

Table 1. The best known ranks for different MMT sizes. The yellow colored problems are tested.

$(\mathit{I},\mathit{K},\mathit{J})$	$\tilde{\mathit{R}}$
(2,2,2)	7
(2,2,3)	11
(2,2,4)	14
(2,3,3)	15
(2,2,5)	18
(2,3,4)	20
(3,3,3)	23
⋮	⋮

. The following paragraphs present the results for sizes between $(2,2,2)$ and $(3,3,3)$, both corresponding to square matrix multiplication tensors. Larger sizes are possible, but at a certain point, the computation time per experiment becomes too large to perform all 50 experiments per parameter set in a reasonable amount of time on a standard computer.

All experiments were performed on a laptop with an AMD Ryzen 7 PRO 7840U processor and 64 $\mathrm{GB}$ of RAM, using Tensorlab 4 beta [42] and Matlab R2023b. The results of the experiments, including the new decompositions, as well as the code, are publicly available (Version 1) (https://github.com/CharlotteVermeylen/ALM_FMM_gen_CS (accessed on 18 September 2025)).

4.1. Problem 1: $(I,K,J)=(2,2,3),R:=R^{\ast }=11$

Table 2. The number of (unique) practical PDs of rank 11 for ${\mathcal{T}}_{223}$ increases significantly for moderate values of $R_{\mathrm{CS}}$. Improvements relative to the unstructured case (bottom row) are highlighted in bold.

${\mathit{R}}_{\mathbf{CS}}$	S	T	$\mathbf{Size}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\right)$	$\mathbf{Rank}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\left({\mathbf{x}}^{\ast }\right)\right)$			# ${\mathbf{x}}^{\ast }$ pract.
				min	max	#
8	2	2	$144\times 112$	92	97	1	1
7	1	2	$144\times 120$	85	90	4	10
	4	1		87	91	4	18
6	0	2	$144\times 128$	91	105	7	13
	3	1		93	100	8	36
5	2	1	$144\times 136$	99	117	8	25
4	1	1	$144\times 144$	105	123	13	30
	4	0		107	118	10	18
3	0	1	$144\times 152$	111	128	11	18
	3	0		113	121	8	29
2	2	0	$144\times 160$	119	127	8	21
1	1	0	$144\times 168$	123	131	7	14
0	0	0	$144\times 176$	125	132	6	15

presents the results for ${\mathrm{PD}}_{11}\left({\mathcal{T}}_{223}\right)\mathrm{s}$ for different combinations of S and T. The rightmost column indicates the number of practical solutions found out of 50 random starting points. The bottom row shows the results without imposing any structure.

The size, minimal and maximal rank, and the number of distinct ranks of ${\mathbf{J}}_{\mathrm{CS},\mathrm{gen}}$ are provided in the three middle columns. For a complete list of all distinct ranks, we refer to the detailed results available online. As expected, the second dimension of the Jacobian matrix and its rank decrease as $R_{\mathrm{CS}}$ increases, due to the reduced parameter space. Also the null-space dimension of the Gauss–Newton Gramian ${\mathbf{J}}_{\mathrm{CS},\mathrm{gen}}^{\top }{\mathbf{J}}_{\mathrm{CS},\mathrm{gen}}$ reduces with $R_{\mathrm{CS}}$.

From Table 2, it is evident that significantly more practical solutions are found for moderate values of S and T compared with the unstructured case (bottom row). Specifically, for $(S,T)=(3,1)$ and $(S,T)=(1,1)$, the number of practical solutions more than doubles. Additionally, for $(S,T)=(1,1)$, the number of unique solutions also doubles. All improvements are highlighted in bold.

4.2. Problem 2: $(I,K,J)=(2,2,4),R:=R^{\ast }=14$

The results for ${\mathrm{PD}}_{14}\left({\mathcal{T}}_{224}\right)$ are shown in

Table 3. The number of (unique) practical PDs of rank 14 for ${\mathcal{T}}_{224}$ increases with $R_{\mathrm{CS}}$ for almost all combinations of S and T. In particular, for $(S,T)=(3,1)$, the number of practical decompositions is four times higher. Improvements relative to the unstructured case are indicated in bold.

${\mathit{R}}_{\mathbf{CS}}$	S	T	$\mathbf{Size}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\right)$	$\mathbf{Rank}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\left({\mathbf{x}}^{\ast }\right)\right)$			# ${\mathbf{x}}^{\ast }$ pract.
				min	max	#
7	1	2	$256\times 224$	178	182	3	9
	4	1		180	186	4	14
6	0	2	$256\times 232$	184	191	5	9
	3	1		184	190	4	21
5	2	1	$256\times 240$	188	201	7	16
4	1	1	$256\times 248$	196	204	7	12
	4	0		196	210	9	12
3	0	1	$256\times 256$	202	209	4	4
	3	0		202	212	6	11
2	2	0	$256\times 264$	214	220	2	4
1	1	0	$256\times 272$	217	218	2	4
0	0	0	$256\times 280$	217	220	3	5

, again for different values of $R_{\mathrm{CS}}$, S, and T. The largest CS rank obtained is $R_{\mathrm{CS}}=7$. The largest number of practical solutions is found for $(S,T)=(3,1)$, which is four times higher than the number without structure ($(S,T)=(0,0)$). Also here, it is evident that including the generalized CS structure leads to more practical PDs and also yields a greater variety of Jacobian ranks, indicating more non-inv-equivalent PDs.

4.3. Problem 3: $(I,K,J)=(2,3,3),R:=\tilde{R}=15$

In

Table 4. The number of (unique) practical PDs of rank 15 of ${\mathcal{T}}_{233}$ increases with $R_{\mathrm{CS}}$ for all values of S and T. Without CS structure, no practical decompositions are found. Improvements w.r.t. the unstructured case (bottom row) are indicated in bold.

${\mathit{R}}_{\mathbf{CS}}$	S	T	$\mathbf{Size}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\right)$	$\mathbf{Rank}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\left({\mathbf{x}}^{\ast }\right)\right)$			# ${\mathbf{x}}^{\ast }$ pract.
				min	max	#
9	6	1	$324\times 243$	218	218	1	1
8	5	1	$324\times 251$	226	226	1	2
7	1	2	$324\times 259$	238	228	1	2
	4	1		230	234	3	5
	0	2		324	234	1	1
6	3	1	$324\times 267$	236	238	2	10
	6	0		238	238	1	1
5	2	1	$324\times 275$	242	242	1	7
4	1	1	$324\times 283$	248	248	1	2
	4	0		250	254	4	6
3	0	1	$324\times 291$	254	254	1	2
	3	0		254	259	4	6
2	2	0	$324\times 299$	258	261	3	4
0	0	0	$324\times 315$	−	−	0	0

, the results for ${\mathcal{T}}_{233}$ at rank 15 are presented. The maximum value of $R_{\mathrm{CS}}$ obtained is 9. The same conclusions as in the previous sections apply; however, generally fewer practical solutions are found because the problem becomes increasingly difficult as I, K, and J grow.

The following decomposition has the same number of nonzeros and stability parameters as the algorithm proposed in [35] but with CS parameters $(S,T)=(1,2)$: In the CS part, Strassen’s decomposition, which also satisfies $(S,T)=(1,2)$, can be recognized. To our knowledge, this is one of the optimal known algorithms for $(I,J,K)=(2,3,3)$, or any permutation of the dimensions.

4.4. Problem 4: $(I,K,J)=(2,2,5),R:=\tilde{R}=18$

Table 5. The number of (unique) practical ${\mathrm{PD}}_{18}\left({\mathcal{T}}_{225}\right)\mathrm{s}$ increases with $R_{\mathrm{CS}}$ for almost all combinations of S and T. For $(S,T)=(3,1)$, ten times more practical solutions are found compared with $(S,T)=(0,0)$. The improvements are indicated in bold.

${\mathit{R}}_{\mathbf{CS}}$	S	T	$\mathbf{Size}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\right)$	$\mathbf{Rank}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\left({\mathbf{x}}^{\ast }\right)\right)$			# ${\mathbf{x}}^{\ast }$ pract.
				min	max	#
9	6	1	$400\times 360$	288	288	1	1
7	1	2	$400\times 376$	292	299	4	4
	4	1		293	301	3	3
6	0	2	$400\times 384$	297	306	3	3
	3	1		299	310	8	10
5	2	1	$400\times 392$	305	312	8	8
4	1	1	$400\times 400$	313	325	4	4
	4	0		312	323	5	5
3	0	1	$400\times 408$	327	327	1	1
	3	0		322	327	4	5
2	2	0	$400\times 416$	330	338	2	2
1	1	0	$400\times 424$	332	332	1	2
0	0	0	$400\times 432$	336	336	1	1

presents the results for ${\mathcal{T}}_{225}$ at rank 18. The maximal CS rank is 9, as was the case in Table 4. Notably, for $(S,T)=(3,1)$, ten times more practical solutions are found compared with the unstructured case.

4.5. Problem 5: $(I,K,J)=(2,3,4),R:=\tilde{R}=20$

Lastly,

Table 6. Practical ${\mathrm{PD}}_{20}\left({\mathcal{T}}_{234}\right)\mathrm{s}$ are found for four combinations of S and T. Without CS structure ($(S,T)=(0,0)$), no practical decompositions are found.

${\mathit{R}}_{\mathbf{CS}}$	S	T	$\mathbf{Size}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\right)$	$\mathbf{Rank}\left({\mathbf{J}}_{\mathbf{CS},\mathbf{gen}}\left({\mathbf{x}}^{\ast }\right)\right)$	# ${\mathbf{x}}^{\ast }$ pract.
8	5	1	$576\times 456$	408	1
7	4	1	$576\times 464$	410	1
5	2	1	$576\times 480$	425	1
4	4	0	$576\times 488$	427	1
0	0	0	$576\times 520$	-	0

presents the results for ${\mathcal{T}}_{234}$ at rank 20. Very few practical solutions are found in this case. Electronic versions of the decompositions are available. When using 1000 starting points for $S=4$ and $T=1$, the most sparse practical PD that we found has 144 nonzeros and stability factors $Q=16$ and $E=57$, which is slightly higher than the decomposition proposed in [8], which has 130 nonzeros and $Q=14$ and $E=35$.

4.6. Remark: Square Matrix Multiplication

Note that the CS rank can also be chosen smaller than R when $I=J=K$. For example, 11 symmetric rank-1 tensors can be enforced within a ${\mathrm{PD}}_{23}\left({\mathcal{T}}_{333}\right)$. The remaining 12 rank-1 tensors do not need to follow any specific structure in this case. To our knowledge, no decomposition with this many symmetric terms has been reported in the literature for ${\mathcal{T}}_{333}$. An example decomposition with this structure is available in electronic form.

5. Conclusions

The proposed generalized cyclic symmetric structure can be used in polyadic decompositions of the fast matrix multiplication (FMM) tensor to reduce the number of parameters. This reduction decreases the size of the search space and shortens the computation time per iteration of an optimization algorithm to find new decompositions. We have shown that the CS structure can be exploited to reduce the computational cost of multiplying a matrix by itself. For several problem parameters—i.e., different matrix multiplication sizes—numerical experiments using a state-of-the-art optimization algorithm demonstrate that incorporating this structure enables significantly more practical solutions to be found for a fixed number of starting points. Although we did not directly improve upon the existing FMM algorithms, the ability to find new decompositions more easily can support further research toward faster FMM algorithms, particularly when combined with sufficient computational resources.