Multidimensional Arrays, Indices and Kronecker Products

Abstract

Much of the algebra that is associated with the Kronecker product of matrices has been rendered in the conventional notation of matrix algebra, which conceals the essential structures of the objects of the analysis. This makes it difficult to establish even the most salient of the results. The problems can be greatly alleviated by adopting an orderly index notation that reveals these structures. This claim is demonstrated by considering a problem that several authors have already addressed without producing a widely accepted solution.

1. Introduction

The algebra of the Kronecker product is essential to multivariate statistical analysis. It is involved in the formulation of multivariate statistical models and in the derivation of their estimating equations. Such derivations commonly require the differentiation of functions of matrices in respect of their matrix arguments. This generates expressions that demand the use of the Kronecker product (see Pollock (1979, 1985) and Magnus (2010) for examples).

The necessary algebraic results have become available in a variety of texts that have arisen over a long period. Early texts on the subject were provided by Balestra (1976), Rogers (1980) and Graham (1981). Other accounts were given by Pollock (1979) and by Magnus and Neudecker (1988). More recently, there have been the texts of Steeb (1997) and of Turkington (2002).

Given that the subject is so well served, one might wonder what scope exists for further contributions. The repository of specialised results is extensive and most of what is needed is available, if looked for diligently. Nevertheless, the subject remains problematic. In many respects, it lacks sufficient transparency to allow easy access to its users, who are liable to call upon it only to satisfy their occasional needs.

Much of the problem lies in the notational difficulty of the subject. This is not primarily a matter of notational complexity. It is a more a result of the use of a notation that conceals the underlying structure of the objects of the analysis. A litany of results has been created that are not readily proven and that are mostly immemorable.

The contention that underlies this paper is that many of the difficulties can be relieved by adopting an appropriate notation involving an explicit use of indices. In the context of a treatise on differential geometry, Cartan (1952) famously decried the confusion that can arise from what he described as une debauche d’indices. He suggested that a clearer understanding of the geometry could be achieved by appealing to abstract concepts. This paper offers a different opinion. An orderly index notation is an indispensable aid to efficient computation. It assists rather that impedes ready access to vector space interpretations.

The continued adherence of econometricians and statisticians to the conventional notation of matrix algebra, when dealing with Kronecker products, is explained by one of their essential purposes, which is to cast complex multivariate models into the formats of simpler models for which there exist well developed theories. An early example of the use of the Kronecker product in econometrics was provided by Zellner (1962) in his treatment of the so-called seemingly unrelated regression equations. His treatment enabled the system of equations to be cast in the format of a single multiple regression equation, which enabled him to exploit the existing results pertaining to such equations.

This paper is concerned primarily with matters of computation. Therefore, the next section considers the alternative ways in which computers handle multidimensional arrays. Section 3 presents an index notation, which is designed to assist in computations, and it provides an analysis of the Kronecker product, which is the matrix of a tensor product with respect to a natural basis.

Section 4, which may be omitted without interrupting the flow of the exposition, describes the formal algebra of the tensor product, and it mentions some of its applications. Section 5 and Section 6 deal with a particular problem concerning multiple Kronecker products, which, it is believed, will serve to illustrate the power of the index notation.

2. The Multidimensional Framework and the Computer

The natural framework for a multi-dimensional array is a set of points in the positive orthant of a multi-dimensional space that have integer-valued co-ordinates. These co-ordinates will serve as the indices of the elements of the array.

The simplest array is a linear array of one dimension, which is described as a vector. Next is a rectangular array of two dimensions, which is a described as a matrix. In three dimensions, there is a cubic array, and the progression continues with hypercubes of increasing dimensions. However, there is an obvious difficulty that limits the dimensions of a practical representation to no more than two. The elements in the interior of a cube are hidden from view, and an object of more than three dimensions is incapable of visualisation.

The only practical recourse for making higher-dimensional arrays accessible and amenable to manipulations is to map them into a linear array or into a matrix. Such a mapping establishes an ordering within the lower-dimensional array of the elements of the higher-dimensional array.

The simplest examples of such mappings are the ways in which a matrix $A=[a_{ij};i=1,2,\dots ,M,j=1,2,\dots ,N]$ of M rows and N columns can be mapped into a vector. The row-major mapping joins successive rows of the matrix in a wide row vector. Here, the index j passes through N values for each increment of the index i. The index that locates the element $a_{ij}$ within the vector is given by $(i-1)N+j$. The alternative column-major mapping joins successive columns of the matrix in a long column vector, and then the index that locates the element $a_{ij}$ is given by $(j-1)M+i$.

These two mappings are commonly employed in assigning the elements of a matrix to contiguous cells of the memory of a computer. Whereas the Pascal and C compilers adopt the row-major principle, the FORTRAN compiler and the MATLAB interpreter adopt the column-major principle.

The space to store a matrix A of order M times N in the memory of a computer is commonly provided by a statement such as

$$A:\mathbf{array}[1,\cdots ,M][1,\cdots ,N].$$

(1)

Regardless of which of the two mappings has been used to store the matrix, its elements can be accessed and displayed readily in either order, and matrix can also be reconstituted and displayed in its original form. Thus, with the help of nested do-loops, the matrix can be displayed via a code such as

$$\begin{array}{c}\mathbf{for}i:=1\mathbf{to}M\mathbf{do}\\ \mathbf{begin}\left\{i\right\}\\ \mathbf{for}j:=1\mathbf{to}N\mathbf{do}\\ \mathbf{begin}\left\{j\right\}\\ Write\left(a\right[i\left]\right[j\left]\right);\\ SkipSpace;\\ \mathbf{end};\left\{j\right\}\\ SkipLine;\\ \mathbf{end};\left\{i\right\}\end{array}$$

(2)

the row-major form would be displayed via

$$\begin{array}{c}\mathbf{for}i:=1\mathbf{to}M\mathbf{do}\\ \mathbf{for}j:=1\mathbf{to}N\mathbf{do}\\ \mathbf{begin}\\ Write\left(a\right[i\left]\right[j\left]\right);\\ SkipSpace;\\ \mathbf{end};\end{array}$$

(3)

and the column-major form would be displayed by reversing the order of the do-loops:

$$\begin{array}{c}\mathbf{for}j:=1\mathbf{to}N\mathbf{do}\\ \mathbf{for}i:=1\mathbf{to}M\mathbf{do}\\ \mathbf{begin}\\ Write\left(a\right[i\left]\right[j\left]\right);\\ SkipLine;\\ \mathbf{end};\end{array}$$

(4)

The extension to arrays of higher dimensions is straightforward. Consider a cubic array

$$A:\mathbf{array}[1,\cdots ,M][1,\cdots ,N][1,\cdots ,P],$$

(5)

in which the elements $a_{ijk}$ are indexed by $i=1,\dots ,M$, $j=1,\dots ,N$, and $k=1,\dots ,P$. On the assumption that the index k in now the most rapidly changing, the row-major mapping would give the element a position corresponding to $(i-1)NP+(j-1)P+k$, whereas, with i as the most rapid index, the column-major mapping would give it the position corresponding to $(k-1)MP+(j-1)P+i$.

It should be noted that, if the initial values of i, j and k are 1, and if $M=N=P=10$, then, in the row-major mapping, it might be declared that k stands for units, j stands for tens and i stands for hundreds. Then, the formula for the position corresponds to how we count in base 10. More generally, if M, N and P were to take different values, then the index would correspond to a mixed radix representation of a number. As it is, with the initial values at one, we are liable to describe $ijk$ within the row-major mapping as the index of a lexicogoraphic or dictionary ordering.

A three-dimensional array has three indices. Two of these indices will serve to define the framework of a matrix and the third index will serve to define the order of the resulting matrices. The indices can be assigned to these roles in six different ways. To keep track of the elements of a multidimensional array, it is clear that an index notation is called for. This will be provided in the next section together with the definitions of a few of the objects that arise most frequently in the processes of statistical computing. These objects are liable to be compounded from objects of lower dimensions. This gives rise to structures that one may be able to exploit in pursuit of computational efficiency.

To achieve computational efficiency, it is necessary to avoid displacing the values that are stored in the memory of the computer. They must be left in place, regardless of the changing nature of the structures to which they may give rise, whether these are vectors or matrices or the products of matrices. Such objects can be created by the manipulation of pointers, which are variables that contain the addresses of the memory locations in which the values are stored.

It is notable that do-loops are implemented by pointers, which accounts for the ease with which their order can be altered within a nested sequence. Thus, whichever system of storage is used, one can rely on do-loops (or pointers) to pluck the elements from the memory in the orders that are required.

3. The Index Notation

The object that will be analysed in detail in subsequent sections of the paper is a threefold Kronecker product postmultuplied by a long vector. It is appropriate already to display this object, which may be denoted by

$$Y^v=(A\otimes B\otimes C)X^v.$$

(6)

Its factors are as follows:

$$\begin{array}{ccc}\hfill A& =\hfill & \hfill \left[a_{jp}\right]=\left(a_{jp}{e}_j^p\right),B=\left[b_{kq}\right]=\left(b_{kq}{e}_k^q\right),\\ \hfill C& =\hfill & \hfill \left[c_{\ell r}\right]=\left(c_{\ell r}{e}_{\ell }^r\right),X^v=\left(x_{pqr}{e}_{pqr}\right).\end{array}$$

(7)

In these definitions, both the brackets $[,]$ and the parentheses $(,)$ are meant to signify that each index is varying over its designated range. The lower limits of the indices are unity, and their upper limits may be denoted by the corresponding capital letters, so that, for example, $j=1,\dots ,J$. The alternative notation for the matrices and for the long vector $X^v$ has been taken from a paper of Pollock (1985). It expresses the matrices as weighted combinations of the corresponding basic matrices. Thus, for example,

$$A=\left(a_{jp}{e}_j^p\right)=\sum _j\sum _p{a}_{jp}{e}_j^p,$$

(8)

where $e_j^p$ is a basic matrix with a unit in the jth row and the pth column and with zeros elsewhere (the parentheses indicate that summations are to be taken in respect of the indices j and p that are associated with the basic matrices and which are repeated within the accompanying scalar elements). The pth column of the matrix A is $\left(a_{jp}{e}_j\right)={\sum }_j{a}_{jp}{e}_j$, whereas its jth row is $\left(a_{jp}{e}^p\right)={\sum }_p{a}_{jp}{e}^p$. Here, as in Equation (8), the summations are over the basis elements, which are vectors in these cases. Summations occur only when there are repeated instances of an index.

There are alternative ways of arranging the elements of the matrix to define different objects. The transpose $A^{\prime }$ of the matrix together with the long column vector $A^c$ and the wide row vector $A^r$, which are described, respectively, as its column-major and row-major forms, may be denoted as follows:

$$\begin{array}{ccc}\hfill A^{\prime }& =\hfill & \hfill {\left(a_{jp}{e}_j^p\right)}^{\prime }=\left(a_{jp}{e}_p^j\right),\\ \hfill A^c& =\hfill & \hfill {\left(a_{jp}{e}_j^p\right)}^c=\left(a_{jp}{e}_{pj}\right),\\ \hfill A^r& =\hfill & \hfill {\left(a_{jp}{e}_j^p\right)}^r=\left(a_{jp}{e}^{jp}\right).\end{array}$$

(9)

Observe that, in the mappings $A\to A^c$ and $A\to A^r$, the migrating index moves ahead of the index or the string of indices that it is joining. In the mappings $A^{\prime }\to A^c$ and $A^{\prime }\to A^r$, the migrating index joins from behind.

Whereas the matrix elements $a_{jp}$ will remain fixed in the computer memory in the order indicated by their indices, the nature of a derived multidimensional object is indicated by the positioning of the indices on the basic matrices or vectors. Thus, $e_j^p$ denotes a basic matrix with a unit in the jth row and the pth column, whereas $e_{pj}$ is a long column vector with a unit in the $\left\{\right(p-1)J+j\}$th position and $e^{jp}$ is a wide row vector with a unit in the $\left\{\right(j-1)P+p\}$th position. Elsewhere, there are only zero-valued elements. For the basis vectors, the composite indices follow a lexicographic ordering, which accords with the conventional definition of a Kronecker product, as will be confirmed at the end of this section.

The translation from the elements that are held in the computer’s memory to the various multidimensional objects should be by virtue of the pointer arithmetic that is associated with the basic arrays $e_j^p$, $e_{pj}$ and $e^{jp}$. These matrices (or vectors) determine the format of the object or its “shape”, which is the MATLAB terminology.

Observe that, in the case of the long column vector $A^c$, the elements are arrayed in the reverse of their lexicographic ordering, with the trailing index p as the leading classifier within the vector. In the case of the long column vector $X^v=\left(x_{pqr}{e}_{pqr}\right)$ of (6), which is the transpose of the wide vector $X^r=\left(x_{pqr}{e}^{pqr}\right)$, the elements follow the lexicographic ordering of the indices $p,q,r$ that are associated with the basic vectors. It will also be necessary to consider the long vector $X^c=\left(x_{pqr}{e}_{rqp}\right)$ in which the elements $x_{pqr}$ are arrayed in reverse lexicographic order, which is the order adopted by the MATLAB program.

The composition of the matrix A with a matrix $D=\left[d_{pr}\right]=\left(d_{pr}{e}_p^r\right)$ may be denoted by

$$AD=\left(a_{jp}{e}_j^p\right)\left(d_{pr}{e}_p^r\right)=\left({\left\{a_{jp}{d}_{pr}\right\}}_p{e}_j^r\right).$$

(10)

The final expression is derived by cancelling the superscripted column index p of the leading matrix with the subscripted row index p in the trailing matrix. The index p that is affixed to the right brace within the product is to emphasise that the elements within the braces are to be summed over the range of the repeated index. It may be omitted if the eyes can be relied on to recognise repeated indices. Thus

$${\left\{a_{jp}{d}_{pr}\right\}}_p=\sum _p{a}_{jp}{d}_{pr},$$

(11)

which is described as a contraction with respect to p.

The transformation ${A^{\prime }}^c=\mathsf{\copyright }{A}^c$ from $A^c=\left(a_{jp}{e}_{pj}\right)$ to ${A^{\prime }}^c=\left(a_{jp}{e}_{jp}\right)$ is effected by the commutation matrix $\mathsf{\copyright }=\left(e_{jp}^{pj}\right)$. Thus

$${A^{\prime }}^c=\left(a_{jp}{e}_{jp}\right)=\left(e_{jp}^{pj}\right)\left(a_{jp}{e}_{pj}\right)=\mathsf{\copyright }{A}^c,$$

(12)

and it will be recognised that ${A^{\prime }}^c={A^r}^{\prime }=A^v$, albeit that these identities do not extend to objects of higher dimensions.

It is easy to seen that © is an orthonormal matrix such that ${\mathsf{\copyright }}^{\prime }={\mathsf{\copyright }}^{-1}$. It has been given various names. Pollock (1979) has called it the tensor commutator, denoted by a capital T inscribed in a circle. Magnus and Neudecker (1979) described it as the commutation matrix, which they denoted by K, and Van Loan (2000) has described it as the shuffle matrix, denoted by S.

The notation for the Kronecker product of matrices is illustrated by the following example:

$$A\otimes B=\left(a_{jp}{e}_j^p\right)\otimes \left(b_{kq}{e}_k^q\right)=\left(\left\{a_{jp}{b}_{kq}\right\}{e}_{jk}^{pq}\right).$$

(13)

Here, the row indices $j,k$ and the column indices $p,q$ associated with the basic matrices both follow lexicographic orderings. The order of the matrices within a Kronecker product can be reversed by applying the tensor commutator to both the rows and the columns. Thus,

$$\begin{array}{ccc}\hfill {\mathsf{\copyright }}_{jk}(A\otimes B){\mathsf{\copyright }}_{pq}^{\prime }& =\hfill & B\otimes A\mathrm{or}\hfill \\ \hfill \left(e_{kj}^{jk}\right)\left(\left\{a_{jp}{b}_{kq}\right\}{e}_{jk}^{pq}\right)\left(e_{pq}^{qp}\right)& =\hfill & \left(\left\{a_{jp}{b}_{kq}\right\}{e}_{kj}^{qp}\right).\hfill \end{array}$$

(14)

The subscripts on the © matrices indicate the indices that are to be commuted.

It is now possible to evaluate the product of (6). This can be expressed in the conventional matrix notation and in the index notation as follows:

$$\begin{array}{ccc}\hfill Y^v& =\hfill & (A\otimes B\otimes C)X^v\mathrm{or}\hfill \\ \hfill \left(y_{jk\ell }{e}_{jk\ell }\right)& =\hfill & \{\left(a_{jp}{e}_j^p\right)\otimes \left(b_{kq}{e}_k^q\right)\otimes \left(c_{\ell r}{e}_{\ell }^r\right)\}\left(x_{pqr}{e}_{pqr}\right)\hfill \\ \hfill & =\hfill & \left(\left\{a_{jp}{b}_{kq}{c}_{\ell r}\right\}{e}_{jk\ell }^{pqr}\right)\left(x_{pqr}{e}_{pqr}\right).\hfill \end{array}$$

(15)

On replacing the row-major vector $X^v$ by the column-major vector $X^c$ and on reversing the order of the matrices, an alternative version of the product is created in the form of

$$\begin{array}{ccc}\hfill Y^c& =\hfill & (C\otimes B\otimes A)X^c\mathrm{or}\hfill \\ \hfill \left(y_{jk\ell }{e}_{\ell kj}\right)& =\hfill & \{\left(c_{\ell r}{e}_{\ell }^r\right)\otimes \left(b_{kq}{e}_k^q\right)\otimes \left(a_{jp}{e}_j^p\right)\}\left(x_{pqr}{e}_{rqp}\right)\hfill \\ \hfill & =\hfill & \left(\left\{a_{jp}{b}_{kq}{c}_{\ell r}\right\}{e}_{\ell kj}^{rqp}\right)\left(x_{pqr}{e}_{rqp}\right).\hfill \end{array}$$

(16)

A series of commutation operations link the two forms. These forms might be regarded as generalisations of the equations

$$\begin{array}{ccc}\hfill Y^v& =\hfill & {\left(AX{B}^{\prime }\right)}^v=(A\otimes B)X^v,\hfill \\ \hfill Y^r& =\hfill & {\left(AX{B}^{\prime }\right)}^r=X^r(A^{\prime }\otimes B^{\prime })\mathrm{and}\hfill \\ \hfill Y^c& =\hfill & {\left(AX{B}^{\prime }\right)}^c=(B\otimes A)X^c,\hfill \end{array}$$

(17)

where $X^v=\left(x_{pq}{e}_{pq}\right)$ and $X^c=\left(x_{pq}{e}_{qp}\right)$, and where the other matrices are as defined previously. The equations of (17) tend to convey the idea that a multiple Kronecker product can be computed via pairwise matrix multiplications. To pursue such a strategy, it is necessary to cast the vectors $X^v,X^r$ or $X^c$ and their transforms into the shapes of matrices.

The various operations of reshaping that might be applied to the column-major vector $X^c=\left(x_{pqr}{e}_{rqp}\right)$ would produce matrices that are denoted as follows:

$$X_p^c=\left(x_{pqr}{e}_p^{rq}\right),X_q^c=\left(x_{pqr}{e}_q^{rp}\right),X_r^c=\left(x_{pqr}{e}_r^{qp}\right).$$

(18)

Similar operations can be applied to the row-major vector $X^v=\left(x_{pqr}{e}_{pqr}\right)$. It is important that the reshaping operations should be effected by manipulating pointers, instead of by moving the elements themselves.

The conversions $X^c\to X_p^c$ and $X^c\to X_r^c$ involve separating the trailing and leading elements, respectively, from the rest of the composite index $rqp$, whereby $e_{rqp}\to e_p^{rq}$ and $e_{rqp}\to e_r^{qp}$. These conversions will prove to be amenable to the reshape operator of MATLAB. To create $X_q^c$ via MATLAB would require a commutation operation to bring the index q to the leading or trailing position. However, the creation of all of these matrices is readily achieved with do-loops that are nested appropriately. The MATLAB reshape operator is considered in detail in Section 6, and examples are provided in an appendix. In a recent paper, Fackler (2019) has show that some efficiency in computing the product of (6) can be gained by exploiting the reshape procedure of that program.

It may be helpful to demonstrate that the definition that has been provided in (13) for the Kronecker product $A\otimes B$ does correspond to the usual definition, which depends on displaying the subscripted elements within the context of their matrices. It is sufficient to consider the product of two square matrices of order 2 as follows:

$$\begin{array}{ccc}\hfill \left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\otimes \left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]& =& \left[\begin{array}{cc}{a}_{11}\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]& a_{12}\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]\\ a_{21}\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]& a_{22}\left[\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]\end{array}\right]\hfill \\ \hfill & =& \left[\begin{array}{cc}\begin{array}{cc}{a}_{11}{b}_{11}& a_{11}{b}_{12}\\ a_{11}{b}_{21}& a_{11}{b}_{22}\end{array}& \begin{array}{cc}{a}_{12}{b}_{11}& a_{12}{b}_{12}\\ a_{12}{b}_{21}& a_{12}{b}_{22}\end{array}\\ \begin{array}{cc}{a}_{21}{b}_{11}& a_{21}{b}_{12}\\ a_{21}{b}_{21}& a_{21}{b}_{22}\end{array}& \begin{array}{cc}{a}_{22}{b}_{11}& a_{22}{b}_{12}\\ a_{22}{b}_{21}& a_{22}{b}_{22}\end{array}\end{array}\right].\hfill \end{array}$$

(19)

It can be seen that the composite row indices $jk$ associated with the elements $a_{jp}{b}_{kq}$ follow the lexicographic sequence $\{11,12,21,22\}$, as do the composite column indices $pq$, which is to say that the conventional definition of a Kronecker product is in accordance with the row-major scheme.

4. Tensor Products and Associated Notations

The majority of econometric texts that consider the Kronecker product make no mention of tensor products, notwithstanding the fact that a Kronecker product may be classified as a tensor product. An exception is the account of Pollock (1985), which defines tensors in the context of multilinear algebra.

There are two ways in which the algebra of tensors may be approached. The first way is via mathematical physics, where vectors are liable to subsist in three-dimensional or four-dimensional spaces. Tensors are used in expounding the theory of electrodynamics via Maxwell’s equations—see Brau (2004)—the theory of special relativity—see Lawden (1967) —and the theory of the elasticity of solids—see Bower (2010). In such contexts, vectors and tensors are liable to acquire substantive connotations.

A second way of approaching tensors and tensor products is via the abstract theory of multilinear algebra that was propounded originally by Bourbaki (1948) and which has been expounded in the more recent texts of Greub (1967) and Marcus (1973). Such expositions take a coordinate-free approach that avoids specifying bases for the vector spaces and which does not preempt the choice of a metric.

The coordinate-free approach is well suited to a physical analysis that regards vectors as geometric objects that exist in space independently of any coordinate system or of its basis vectors. This flexibility is exploited in the theory of special relativity to enable geometric vectors to be viewed within alternative frames of reference. The natural basis is appropriate to an observer at rest within their own frame of reference. However, objects that are in motion relative to the observer require other coordinate systems, which are derived by applying the Lorentz transform to the natural basis. (see Marder (1968) and Susskind and Friedman (2018), for examples).

In the main, data analysts and computer scientists are liable to regard vectors as algebraic objects comprising an ordered set of elements, which are the coordinates of the vector relative to a natural basis consisting of orthogonal vectors of unit length comprised by an identity matrix. Our approach, in this section, is to express the basic definitions in terms of arbitrary vector spaces, denoted by $\mathcal{U}$ and $\mathcal{V}$ etc., and to interpret them immediately in terms of real finite-dimensional coordinate spaces. An n-dimensional real vector space will be denoted by ${\mathcal{R}}_n$.

If $\mathcal{V}$ is a vector space defined over a scalar field $\mathcal{F}$, then the dual space is the space of all linear functionals ${\mathcal{V}}^{*}=\mathcal{L}(\mathcal{V},\mathcal{F})$ mapping from $\mathcal{V}$ to $\mathcal{F}$. Given that it is of the same dimension, the dual space ${\mathcal{V}}^{*}$ is isomorphic to the primal vector space $\mathcal{V}$. The relationship between ${\mathcal{V}}^{*}$ and $\mathcal{V}$ is reflexive or reciprocal, which is to say that the dual of the dual space is the primal space.

The dual of the vector space ${\mathcal{R}}_n$ may be denoted by ${\mathcal{R}}_n^{*}$. The elements of a coordinate vector space ${\mathcal{R}}_n$ are column vectors, which may be denoted by subscripted lowercase symbols, such as $x_1,y_2$, whereas the elements of the corresponding dual space ${\mathcal{R}}_n^{*}$ are row vectors denoted by symbols with superscripts, such as $x^{\prime },x^{*},x^1,y^2$. Lowercase letters without subscripts, such as $x,y$, are liable to be regarded as column vectors. The inner product $x^{\prime }y$ is a linear functional.

If $\mathcal{U}$ and $\mathcal{V}$ are vector spaces defined over a scalar field $\mathcal{F}$, then a mapping $\varphi :\mathcal{U}\times \mathcal{V}\to \mathcal{F}$ from the Cartesian product of the spaces to the scalar field is a bilinear functional if, when $u_1,u_2\in \mathcal{U}$ and $v_1,v_2\in \mathcal{V}$ and when $\alpha ,\beta \in \mathcal{F}$, there are

$$\begin{array}{ccc}\hfill \varphi (\alpha u_1+\beta u_2,v_1)& =\hfill & \alpha \varphi (u_1,v_1)+\beta \varphi (u_2,v_1),\hfill \\ \hfill \varphi (u_1,\alpha v_1+\beta v_2)& =\hfill & \alpha \varphi (u_1,v_1)+\beta \varphi (u_1,v_2).\hfill \end{array}$$

(20)

The set of all such bilinear functionals is a vector space denoted by $\mathcal{L}(\mathcal{U},\mathcal{V};\mathcal{F})$. Familiar examples of the mapping ${\mathcal{R}}_n^{*}\times {\mathcal{R}}_n\to \mathcal{R}$ are as follows:

$$\begin{array}{ccc}\hfill x^{\prime }{E}_{ij}y& =\hfill & \left(x_i{e}^i\right)\left(e_i^j\right)\left(y_j{e}_j\right)=\left(x_i{e}^i\right)\left(y_j{e}_i\right)=x_i{y}_j,\hfill \\ \hfill x^{\prime }{I}_{n}y& =\hfill & \left(x_i{e}^i\right)\left({\delta }_{ij}{e}_i^j\right)\left(y_j{e}_j\right)=\left(x_i{e}^i\right)\left({\delta }_{ij}{y}_j{e}_i\right)={\left\{x_i{y}_i\right\}}_i,\hfill \\ \hfill x^{\prime }Ay& =\hfill & \left(x_i{e}^i\right)\left(a_{ij}{e}_i^j\right)\left(y_j{e}_j\right)={\left\{x_i{a}_{ij}{y}_j\right\}}_{ij},\hfill \end{array}$$

(21)

were $E_{ij}=\left(e_i^j\right)$ denotes a matrix with a unit in the $ij$th position and with zeros elsewhere. The second of these products may be written, more simply, as

$$x^{\prime }y=\left(x_i{e}^i\right)\left(y_i{e}_i\right)={\left\{x_i{y}_i\right\}}_i.$$

(22)

This is a bilinear functional, which can also be considered to be a linear functional, if one of the vectors is regarded as the means by which the other vector is mapped to the field $\mathcal{R}$.

Less familiar are the matrix forms of the mappings ${\mathcal{R}}_n\times {\mathcal{R}}_n\to \mathcal{R}$ and ${\mathcal{R}}_n^{*}\times {\mathcal{R}}_n^{*}\to \mathcal{R}$. Examples of these, in turn, are as follows:

$$A^r(x\otimes y)=\left(a_{ij}{e}^{ij}\right)\left\{\left(x_i{e}_i\right)\otimes \left(y_j{e}_j\right)\right\}=\left(a_{ij}{e}^{ij}\right)\left(x_i{y}_j{e}_{ij}\right)={\left\{a_{ij}{x}_i{y}_j\right\}}_{ij}$$

(23)

and

$$(y^{\prime }\otimes x^{\prime })A^c=\left\{\left(y_j{e}^j\right)\otimes \left(x_i{e}^i\right)\right\}\left(a_{ij}{e}_{ji}\right)=\left(y_j{x}_i{e}^{ji}\right)\left(a_{ij}{e}_{ji}\right)={\left\{y_j{x}_i{a}_{ij}\right\}}_{ij}.$$

(24)

Reference to (17) will show that $A^r(x\otimes y)={\left(x^{\prime }Ay\right)}^r$ and that $(y^{\prime }\otimes x^{\prime })A^c={\left(x^{\prime }Ay\right)}^c$.

It is important to recognise that the vector space $\mathcal{L}(\mathcal{U},\mathcal{V};\mathcal{F})$ represents the means by which vectors $u\in \mathcal{U}$ and $v\in \mathcal{V}$ are mapped into the scalar field $\mathcal{F}$. It should be distinguished from the individual mappings of $\mathcal{U}\times \mathcal{V}\to \mathcal{F}$, each of which entails specific values of u and v. The distinction can be made clear by considering the functional $f=u^{\prime }Av$, where $u={[u_1,\dots ,u_n]}^{\prime }$ and $v={[v_1,\dots ,v_m]}^{\prime }$ are column vectors and $A=\left[a_{ij}\right]$ is a matrix. The pair $(u,v)$ are an element of the Cartesian product $\mathcal{U}\times \mathcal{V}$. The matrix A is an element of $\mathcal{L}(\mathcal{U},\mathcal{V};\mathcal{F})$, and f is a product of a specific mapping of $\mathcal{U}\times \mathcal{V}\to \mathcal{F}$.

A tensor product of vector spaces may be defined in a confusing variety of ways. However, it is appropriate to regard the tensor product $\mathcal{U}\otimes \mathcal{V}$ as unique linear mapping from the set $\mathcal{U}\times \mathcal{V}$. Thus, for example, Shephard (1966) defines it to be the dual of the vector space $\mathcal{L}(\mathcal{U},\mathcal{V};\mathcal{F})$ of bilinear functionals, so that each bilinear functional corresponds to a unique linear functional on $\mathcal{U}\otimes \mathcal{V}$. From the point of view of the real coordinate spaces, this amounts to nothing more than a change of notation, Nevertheless, what is revealed can be insightful.

A distinction must be made between the forms ${\mathcal{U}}^{*}\otimes \mathcal{V}$, ${\mathcal{U}}^{*}\otimes {\mathcal{V}}^{*}$ and $\mathcal{U}\otimes \mathcal{V}$. The first of these is liable to be described as a contravariant product, and it may be noted that ${\mathcal{U}}^{*}\otimes \mathcal{V}=\mathcal{V}\otimes {\mathcal{U}}^{*}$. The others may be described as covariant products, and it should be noted that the order of the two spaces is significant: $\mathcal{U}\otimes \mathcal{V}\ne \mathcal{V}\otimes \mathcal{U}$ and ${\mathcal{U}}^{*}\otimes {\mathcal{V}}^{*}\ne {\mathcal{V}}^{*}\otimes {\mathcal{U}}^{*}$. In the present context, covariance and contravariance denote relationships. However, in mathematical physics and in other contexts, they are regarded as attributes of the vector spaces.

If $x^{\prime }\in {\mathcal{R}}_n^{*}$ and $y\in {\mathcal{R}}_n$, then their contravariant product is an element in ${\mathcal{R}}_n^{*}\otimes {\mathcal{R}}_n$ that has the form

$$x^{\prime }\otimes y=y\otimes x^{\prime }=\left(x_i{e}^i\right)\otimes \left(y_j{e}_j\right)=\left(x_i{y}_j{e}_j^i\right).$$

(25)

The corresponding covariant products in ${\mathcal{R}}_n\otimes {\mathcal{R}}_n$ and ${\mathcal{R}}_n^{*}\otimes {\mathcal{R}}_n^{*}$, respectively. are

$$x\otimes y=\left(x_i{e}_i\right)\otimes \left(y_j{e}_j\right)=\left(x_i{y}_j{e}_{ij}\right)\mathrm{and}{x}^{\prime }\otimes y^{\prime }=\left(x_i{e}^i\right)\otimes \left(y_j{e}^j\right)=\left(x_i{y}_j{e}^{ij}\right).$$

(26)

These are elementary or decomposable products. Examples of non-decomposable tensor products of the three varieties are

$$\begin{array}{ccc}\hfill A& =\hfill & \sum _{ij}{a}_{ij}(e_i\otimes e^j)=\left(a_{ij}{e}_i^j\right),\hfill \\ \hfill A^c& =\hfill & \sum _{ij}{a}_{ij}(e_j\otimes e_i)=\left(a_{ij}{e}_{ji}\right),\hfill \\ \hfill A^r& =\hfill & \sum _{ij}{a}_{ij}(e^i\otimes e^j)=\left(a_{ij}{e}^{ij}\right).\hfill \end{array}$$

(27)

These would become decomposable products if $a_{ij}={\alpha }_i{\beta }_j$ for all $i,j$, albeit that a decomposable product may be the product of two or more non-decomposable products.

It is when coordinates are attributed to physical vectors in 3-dimensional space or in 4-dimensional spacetime that the terms covariant and contravariant come to denote attributes as opposed to reciprocal relationships. The usage can be demonstrated by considering a change of bases. Let $W=[w_1,w_2,\dots ,w_n]$ and $M=[m_1,m_2,\dots ,m_n]$ be alternative bases of a vector space $\mathcal{V}$. Then, there exists transformations $M=WA$ and $W=M{A}^{-1}$ that transform one basis into the other.

Consider a vector expressed as $v=v_1{e}_1+v_2{e}_2+\cdots +v_n{e}_n$ in terms of the natural basis $I_n=[e_1,e_2,\dots ,e_n]$ and, alternatively, as $v=Wx=My$, where $x={[x_1,x_2,\dots ,x_n]}^{\prime }$ and $y={[y_1,y_2,\dots ,y_n]}^{\prime }$ are the coordinate vectors relative to the bases W and M, respectively. Then, since $M=WA$, there are $Wx=WAy$, whence $x=Ay$ and $y=A^{-1}x$. Thus, the bases and the corresponding coordinates transform as follows:

$$\begin{array}{c}W\\ x\end{array}\}\to \{\begin{array}{c}M=WA\\ y=A^{-1}x.\end{array}$$

(28)

Let $A=\left({\lambda }_i{e}_i^i\right)$ be a diagonal matrix. Then, the ith column of the transformed basis M is

$$m_i=\left(m_{ij}{e}_j\right)=\left(w_{ij}{e}_i^j\right)\left({\lambda }_j{e}_j\right)=\left({\left\{w_{ij}{\lambda }_j\right\}}_j{e}_i\right)$$

(29)

and the vector of coordinates relative the basis vectors of M is

$$y=\left(y_i{e}_i\right)=\left({\lambda }_i^{-1}{e}_i^i\right)\left(x_i{e}_i\right)=\left({\lambda }_i^{-1}{x}_i{e}_i\right).$$

(30)

They have a contravariant relationship with the basis vectors. The effect of halving the length of a basis vector will be the doubling of the value of an associated coordinate.

In general, vectors that change their scale inversely to changes in the scale of the reference axes (i.e. the basis vectors) are commonly described as contravariant. Dual vectors that change in the same way as the changes to scale of the reference axes are described as covariant. If $v=v_1{e}_1+v_2{e}_2+\cdots +v_n{e}_n=\left(v_i{e}_i\right)$ denotes a velocity vector, then its coefficients $[v_1,v_2,\dots v_n]$, which have the dimension of distance/time, can said to constitute a contravariant vector. By contrast, a gradient vector of a dimension quantity/distance will be a covariant vector.

In the Einstein notation, which suppresses the summation sign, contravariant components carry upper indices, and the velocity vector will be denoted, typically, by

$$\mathbf{v}=v^1{\mathbf{e}}_1+v^2{\mathbf{e}}_2+\cdots +v^n{\mathbf{e}}_n=v^i{\mathbf{e}}_i,$$

(31)

where ${\mathbf{e}}_i$ denotes a basis vector. The understanding is that, whenever an index is repeated within an expression, there must be a corresponding summation. This easement of the notation, which dispenses with the summation signs, relies on an understanding of the context of the expression.

The notation of the present paper adopts the same summation convention for expressions that are found within parentheses. Thus, $I_n=\left({\delta }_{ij}{e}_i^j\right)$ denotes an identity matrix. However, in the absence of the parentheses, $a_{ij}{e}_i^j$ denotes a matrix with $a_{ij}$ in the ith row and the jth column and with zeros elsewhere. The expression $\left(e_i^j\right)$ from (21) denotes a matrix with a unit in that position and with zeros elsewhere. Since there are no repeated indices, the parentheses do not imply a summation.

It should be noted that, in this notation, scalar elements never carry superscripts. Furthermore, the notation refers invariably to algebraic or coordinate vectors. It does not encompass coordinate-free geometric vectors, which are common in physical contexts.

The diversity of the existing tensor notations is too great to be summarised here. However, numerous references to alternative notations have been given by Harrison and Joseph (2016). Mention should also be made of De Lathauwer et al. (2000) and Bader and Kolda (2006), who have provided accounts that have been widely referenced by computer scientists.

5. The Problem at Hand

In terms of the index notation, the product of (6) may be expressed as

$$\begin{array}{ccc}\hfill (A\otimes B\otimes C)X^v& =\hfill & \left(\left\{a_{jp}{b}_{kq}{c}_{\ell r}\right\}{e}_{jk\ell }^{pqr}\right)\left(x_{pqr}{e}_{pqr}\right)\hfill \\ \hfill & =\hfill & \left({\left\{a_{jp}{b}_{kq}{c}_{\ell r}{x}_{pqr}\right\}}_{pqr}{e}_{jk\ell }\right)=Y^v.\hfill \end{array}$$

(32)

The final expression of the RHS is the product of the post multiplication of $A\otimes B\otimes C$ by $X^v$. It is derived by cancelling the (superscripted) column indices $pqr$, which are associated with the basic matrices within the leading factor, which is the Kronecker product, with the (subscripted) row indices in the trailing factor, which is the vector $X^v$. The indices $pqr$ affixed to the right brace in the penultimate expression emphasise that the elements within the braces are to be summed over the repeated indices. The indices of summation can be inferred from the contents of the braces.

The final expression of (32) suggests various ways of computing the product $Y^v$. One way would be by a single contraction or summation in respect of the composite index $pqr$. In so far as the vector $X^v$ is concerned, the composite index is liable to correspond to a set of adjacent addresses in the memory of the computer.

The number of multiplications involved in forming the Kronecker product is $JP\times KQ\times LR$, which is the product of the number of elements in the constituent matrices. The operation of forming $Y^v$ via a single contraction in respect of the composite index $pqr$ entails $PQR\times JKL$ multiplications and $(PQR-1)\times JKL$ additions.

A more efficient way of computing the product is by successive contractions in respect of the indices r, q and p, separately and successively. This could be described as the divide-and-conquer procedure. Then, a sequence of products, in long-vector form, that would lead from $X^v$ to $Y^v$ are as follows:

$$\begin{array}{ccc}\hfill X^v& =\hfill & \left(x_{pqr}{e}_{pqr}\right),\hfill \\ \hfill W^v& =\hfill & \left(w_{\ell pq}{e}_{\ell pq}\right)=\left({\left\{c_{\ell r}{x}_{pqr}\right\}}_r{e}_{\ell pq}\right),\hfill \\ \hfill V^v& =\hfill & \left(v_{k\ell p}{e}_{k\ell p}\right)=\left({\left\{b_{kq}{w}_{\ell pq}\right\}}_q{e}_{k\ell p}\right)=\left({\left\{b_{kq}{c}_{\ell r}{x}_{pqr}\right\}}_{qr}{e}_{k\ell p}\right),\hfill \\ \hfill Y^v& =\hfill & \left(y_{jk\ell }{e}_{jk\ell }\right)=\left({\left\{a_{jp}{v}_{k\ell p}\right\}}_p{e}_{jk\ell }\right)=\left({\left\{a_{jp}{b}_{kq}{c}_{\ell r}{x}_{pqr}\right\}}_{pqr}{e}_{jk\ell }\right).\hfill \end{array}$$

(33)

Here, the elements of the matrices $C=\left[c_{lr}\right]$, $B=\left[b_{kq}\right]$ and $A=\left[a_{jp}\right]$ are adduced to the product in the course of three successive contractions. Passing from $X^v$ to $W^v$ via the contraction over r requires $R\times LPQ$ multiplications. Passing from $W^v$ to $V^v$ via the contraction over q requires $Q\times KLP$ multiplications. Passing from $V^v$ to $Y^v$ via the contraction over p requires $P\times JKL$ multiplications.

However, there are six different orders in which the contractions can be performed, which correspond to the permutations of the indices $p,q,r,$ which are the column indices of the matrices A, B, and C, respectively. In comparing the efficiency of the alternative procedures, one need only count the number of multiplications, since these are far more expensive than are the additions (the orders of computational complexity are $n^2$ and n, respectively).

When $R=2$, $Q=4$ and $P=3$, with $J=K=L=2$ as before, there are 384 multiplications in the slow procedure. The number of multiplications in the divide-and-conquer procedure depends on the sequence of the contractions. The number of multiplications in the various the divide-and-conquer procedures are listed below with the sequence of contractions denoted by a sequence of matrices, where the order in which they are deployed should be read from right to left (as in a sequence of matrix multiplications):

$$\begin{array}{ccc}A,& B,& C:120\\ B,& C,& A:112\\ C,& A,& B:88\end{array}\begin{array}{ccc}C,& B,& A:96\\ A,& C,& B:96\\ B,& A,& C:128\end{array}$$

(34)

To minimise the computational burden by minimising the number of multiplications within the divide-and-conquer procedure, one should ensure the contractions in respect of p, q, and r are ordered such that numbers of the associated multiplications are declining. When the row orders of the matrices are equal, the most efficient sequence of contractions will be self-evident, and there will be no need to count the number of multiplications.

The calculation of the product of (32) can de accomplished using nested do-loops. Six loops are required that fall into two groups, which comprise the outer indices $j,k,\ell$ and the inner indices $p,q,r$. The former group are the indices of the product vector and the latter group are the indices of the successive contractions. Within each group, the order of the indices may be freely permuted.

However, it might pay to deploy the outer indices in the order listed above, since this will enable the elements of the product to be created on the fly in the order in which they would be printed in a column. Otherwise, the printing of the column would need to be delayed until all of its elements have been created.

To illustrate a code appropriate to such calculations, it is enough to consider the equation $Y^v=(A\otimes B)X^v$, which can be rendered in the index notation as

$$\left(y_{jk}{e}_{jk}\right)=\left(\left\{a_{jp}{b}_{kq}\right\}{e}_{jk}^{pq}\right)\left(x_{pq}{e}_{pq}\right)=\left({\left\{a_{jp}{b}_{kq}{x}_{pq}\right\}}_{pq}{e}_{jk}\right).$$

(35)

The code is as follows:   

$$\begin{array}{c}\mathbf{for}j:=1\mathbf{to}J\mathbf{do}\\ \mathbf{for}k:=1\mathbf{to}K\mathbf{do}\\ \mathbf{begin}\left\{jk\right\}\\ y\left[j\right]\left[k\right]:=0;\\ \mathbf{for}q:=1\mathbf{to}Q\mathbf{do}\\ \mathbf{for}p:=1\mathbf{to}P\mathbf{do}\\ y\left[j\right]\left[k\right]:=y\left[j\right]\left[k\right]+a\left[j\right]\left[p\right]*b\left[k\right]\left[q\right]*x\left[p\right]\left[q\right];\\ \mathbf{end};\left\{jk\right\}\end{array}$$

(36)

The code becomes increasingly prolix with a growing number of indices, each of which gives rise to a nested do-loop. However, such code can be encapsulated within a procedure or a subroutine with a concise interface.

6. Matrix Multiplications and the Reshaping Operations

Many authors have recognised the inefficiency of forming a multiple Kronecker product before postmultiplying it by a vector. They have proposed a variety of methods that rely on pairwise matrix multiplications that are accompanied by the shuffling of the elements of the products in order to facilitate the succeeding matrix multiplications. These operations have been described as the reshaping of the matrices.

A method for solving the equation $Y^v=(A\otimes B\otimes C)X^v$ of (6) for X was proposed by Pereyra and Scherer (1973). A much simplified solution was provided by de Boor (1979), who showed that this could be achieved by a sequence of solutions involving the matrices A, B and C successively.

An indication or the way in which the product (6) might be calculated via a sequence of matrix multiplications was given in a paper on Dyksen (1987), which contained the identity $(A\otimes B)X^v={\left\{B{\left(AX\right)}^{\prime }\right\}}^c$, where $X=\left(x_{pq}{e}_p^q\right)$ and $X^v=\left(x_{pq}{e}_{pq}\right)$, albeit that the necessary vectorisations of the matrices was not indicated.

A paper by Davio (1981) dealt at length with the matter of shuffling the elements of Kronecker product; but it did not relate such operations to pointers. Others, including Benoit et al. (2003), Langville and Stewart (2004) and Dayar and Orhan (2015) have also dealt with such issues. Programs that are aimed at computing the product of (6) have been provided by Constantine and Gleich (2009) and by Kredler (2015).

In a recent paper, Fackler (2019) has proposed that computations involving multiple Kronecker products can be achieved by ordinary matrix multiplications without displacing elements that are stored in the computer memory, as shuffling is liable to do. To illustrate how this method can be implemented in the MATLAB program, we may consider an input vector $X^c$, which follows a column-major scheme. Then, the method, which applies the matrices $A=\left[a_{jp}\right]$, $B=\left[b_{kq}\right]$ and $C=\left[c_{\ell r}\right]$ in succession to $X^c$, entails the following objects,

$$\begin{array}{ccc}\hfill X^c& =\hfill & \left(x_{pqr}{e}_{rqp}\right),\hfill \\ \hfill W^c& =\hfill & \left(w_{qrj}{e}_{jrq}\right)=\left({\left\{a_{jp}{x}_{pqr}\right\}}_p{e}_{jrq}\right),\hfill \\ \hfill V^c& =\hfill & \left(v_{rjk}{e}_{kjr}\right)=\left({\left\{b_{kq}{w}_{qrj}\right\}}_q{e}_{kjr}\right),\hfill \\ \hfill Y^c& =\hfill & \left(y_{jk\ell }{e}_{\ell kj}\right)=\left({\left\{c_{\ell r}{v}_{rjk}\right\}}_r{e}_{\ell kj}\right).\hfill \end{array}$$

(37)

In the process, the basis vector undergo the following transformation:

$$e_{rqp}\to e_{jrq}\to e_{kjr}\to e_{\ell kj}.$$

(38)

It can be seen that, as the indices p, q, r of the contractions drop off the end of the sequence, the new indices j, k, ℓ join at the head.

When the equations of (37) are given their appropriate shapes, they become

$$\begin{array}{ccc}\hfill X_p^c& =\hfill & \left(x_{pqr}{e}_p^{rq}\right),\hfill \\ \hfill W_j^c& =\hfill & \left(w_{qrj}{e}_j^{rq}\right)=\left({\left\{a_{jp}{x}_{pqr}\right\}}_p{e}_j^{rq}\right)=\left(a_{jp}{e}_j^p\right)\left(x_{pqr}{e}_p^{rq}\right),\hfill \\ \hfill V_k^c& =\hfill & \left(v_{rjk}{e}_k^{jr}\right)=\left({\left\{b_{kq}{w}_{qrj}\right\}}_q{e}_k^{jr}\right)=\left(b_{kq}{e}_k^q\right)\left(w_{qrj}{e}_q^{jr}\right),\hfill \\ \hfill Y_{\ell }^c& =\hfill & \left(y_{jk\ell }{e}_{\ell }^{kj}\right)=\left({\left\{c_{\ell r}{v}_{rjk}\right\}}_r{e}_{\ell }^{kj}\right)=\left(c_{\ell r}{e}_{\ell }^r\right)\left(v_{rjk}{e}_r^{kj}\right),\hfill \end{array}$$

(39)

and the sequence of matrix multiplications can be denoted by

$$A{X}_p^c=W_j^c\to B{W}_q^c=V_k^c\to C{V}_r^c=Y_{\ell }^c.$$

(40)

This entails two operations that are amenable to the $\mathtt{reshape}$ procedure:

$$\begin{array}{ccc}\hfill W_j^c& =\hfill & \left(w_{qrj}{e}_j^{rq}\right)\to W^c=\left(w_{qrj}{e}_{jrq}\right)\to W_q^c=\left(w_{qrj}{e}_q^{jr}\right),\hfill \\ \hfill V_k^c& =\hfill & \left(v_{rjk}{e}_k^{jr}\right)\to V^c=\left(v_{rjk}{e}_{kjr}\right)\to V_r^c=\left(v_{rjk}{e}_r^{kj}\right).\hfill \end{array}$$

(41)

Here, the variant matrices $W_j^c,W_q^c$ and $V_k^c,V_r^c$ have shapes that are derived directly from $W^c$ and $V^c$, respectively, which represent what is stored in the computer’s memory. The reshape procedure is explained in detail in the Appendix A.

If it is required to reverse the sequence of matrix multiplications that are applied to the column-major vector $X^c$, then these must become post-multiplications, and the corresponding equations will be

$$\begin{array}{ccc}\hfill {\left(X_r^c\right)}^{\prime }& =\hfill & \left(x_{pqr}{e}_{qp}^r\right),\hfill \\ \hfill {\left(W_{\ell }^c\right)}^{\prime }& =\hfill & \left(w_{\ell pq}{e}_{qp}^{\ell }\right)=\left({\left\{c_{\ell r}{x}_{pqr}\right\}}_r{e}_{qp}^{\ell }\right)=\left(x_{pqr}{e}_{qp}^r\right)\left(c_{\ell r}{e}_r^{\ell }\right),\hfill \\ \hfill {\left(V_k^c\right)}^{\prime }& =\hfill & \left(v_{k\ell p}{e}_{p\ell }^k\right)=\left({\left\{b_{kq}{w}_{\ell pq}\right\}}_q{e}_{p\ell }^k\right)=\left(w_{\ell pq}{e}_{p\ell }^q\right)\left(b_{kq}{e}_q^k\right),\hfill \\ \hfill {\left(Y_j^c\right)}^{\prime }& =\hfill & \left(y_{jk\ell }{e}_{\ell k}^j\right)=\left({\left\{a_{jp}{v}_{k\ell p}\right\}}_p{e}_{\ell k}^j\right)=\left(v_{k\ell p}{e}_{\ell k}^p\right)\left(a_{jp}{e}_p^j\right).\hfill \end{array}$$

(42)

The sequence of matrix multiplications can be denoted by

$${\left(X_r^c\right)}^{\prime }{C}^{\prime }={\left(W_{\ell }^c\right)}^{\prime }\to {\left(W_q^c\right)}^{\prime }{B}^{\prime }={\left(V_k^c\right)}^{\prime }\to {\left(V_p^c\right)}^{\prime }{A}^{\prime }={\left(Y_j^c\right)}^{\prime }.$$

(43)

However, reference to (15) indicates that, if the row-major vector $X^v=\left(x_{pqr}{e}_{pqr}\right)$ is available, then the following sequence of matrix pre-multiplications can be pursued:

$$C{X}_r^v=W_{\ell }^v\to B{W}_q^v=V_k^v\to A{V}_p^v=Y_j^v.$$

(44)

According to the table of (34), which is based on the values $P=3$, $Q=4$, $R=2$ and $J=K=L=2$, neither of the sequences of (40) and (44) lead to a fully efficient computation. A recourse might be to re-order the matrices within the multiple Kronecker product to create the most efficient sequence, which would also require the elements within the vector $Y^c$ to be re-ordered. Whether this would be allowable or convenient is a matter that depends on the substantive nature of the problem that is represented by the equation. However, it is reasonable to suppose that, in most cases, this will be easy to achieve,

Next is the apparent obstacle that the reshape procedure is specific to MATLAB. However, as Fackler (2019) has indicated, it should be straightforward to replicate the procedure in many of the available programming languages, using the arithmetic of pointers. Whereas the method of pairwise multiplication has been described for the case of a column-major scheme, there is no difficulty in adapting to a row-major scheme.

Our preference would be to calculate the product of (6) according to the scheme of (33), or some permutation thereof, that does not require matrix multiplications. However, the intention of this paper has not been to insist on one way of another of conducting the computations, provided that gross inefficiency is avoided. Instead, the aim has been to clarify the nature of the alternative procedures.