First Order and Second Order Learning Algorithms on the Special Orthogonal Group to Compute the SVD of Data Matrices

Abstract

The present paper deals with neural algorithms to learn the singular value decomposition (SVD) of data matrices. The neural algorithms utilized in the present research endeavor were developed by Helmke and Moore (HM) and appear under the form of two continuous-time differential equations over the special orthogonal group of matrices. The purpose of the present paper is to develop and compare different numerical schemes, under the form of two alternating learning rules, to learn the singular value decomposition of large matrices on the basis of the HM learning paradigm. The numerical schemes developed here are both first-order (Euler-like) and second-order (Runge-like). Moreover, a reduced Euler scheme is presented that consists of a single learning rule for one of the factors involved in the SVD. Numerical experiments performed to estimate the optical-flow (which is a component of modern IoT technologies) in real-world video sequences illustrate the features of the novel learning schemes.

1. Introduction

The computation of the singular value decomposition (SVD) of a non-square matrix [1,2,3] plays a key role in a number of applications (see, for instance, [4,5,6,7,8,9,10,11,12,13]); among them, it is worth citing applications in automatic control [14], digital circuit design [15], time-series prediction [16], and image processing [17,18]. Efforts have been devoted to achieving the computation of the SVD of matrices in the artificial neural networks community [19,20,21].

The neural algorithms utilized in the present research work to learn an SVD of a data matrix were developed by Helmke and Moore (HM) in [1]. An HM learning rule appeared under the form of two continuous-time differential equations over the special orthogonal group of matrices. The purpose of the present research is to develop and compare different numerical schemes, under the form of two alternating learning equations, to learn the SVD of large matrices on the basis of the HM learning paradigm. The numerical schemes developed here are both first-order (Euler-like) and second-order (Runge-like). Moreover, a reduced Euler scheme will be presented that consists of a single learning equation for one of the factors involved in the SVD.

The developed methods to learn an SVD of a data matrix are applied to optical-flow estimation. Optical flow estimation is a well-known image-processing operation that allows estimating the motion of portions of an image over a video sequence and has found widespread applications (see, for instance, [22,23,24,25,26,27,28,29]). Optical flow is closely related to motion estimation [30]. Optical flow refers to the change of structured light in an image and captures such change through a velocity vector field. Most of the optical-flow estimation algorithms used in video encoding belong to either the class of block matching algorithms (BMAs) or to the class of pixel recursive algorithms (PRAs) [31].

Optical flow estimation algorithms are essential components in a number of complex Internet of Things (IoT) technologies, as testified by several existing applications, such as intelligent fall detection [32,33], mobile robotics [34], intelligent flight monitoring [35], mobile object tracing [36], automated surveillance [37], smart healthcare [38], solar energy forecasting [39], and in IoT-based analytics in urban space, shops, and retail stores to inform policy makers, shop owners, and the general public about how they interact with the physical space [40]. See also the interesting discussion in [41].

The majority of the current optical-flow estimation methods rely on a block-matching algorithm. The BMA methods are based on the concept of template-matching: it is supposed that a single block in a time-frame has moved solidly to another location in the next time-frame, so the image-block is regarded as a template to be looked for in the subsequent frame. The BMA methods try to evaluate the motion of a block by reducing the number of search locations in the search range and/or by reducing the number of computations at each search location. These algorithms are either ad hoc or are based on the assumption that the error increases monotonically from the best-match location. However, typically, the error surface may exhibit local minima, and the majority of the BMAs get trapped in one of the local minima depending on the starting point and on the search direction. Furthermore, the matching algorithms aim at finding the best match with respect to some selected mismatch (error) measure, but the best match may not represent the true motion [42].

Conversely, standard PRAs try to estimate the motion at each pixel. In the method for optical-flow estimation considered here, based on the paper [42], a methodology similar to the PRA is employed, except that the algorithm operates on a pixel-block basis and finds a single motion-vector for each block. In order to make the method robust in a noisy environment, a total least squares (TLS) estimation approach is invoked. Total least squares may be implemented numerically through singular value decomposition (SVD). Other matrix decomposition techniques, such as eigenvalue decomposition, have been used in the scientific literature [43].

Throughout the paper, we use the following notation: symbol $I_{m,p}$ denotes a pseudo-identity matrix of size $m\times p$ and $I_m=I_{m,m}$, while symbol $0_{m,p}$ denotes a $m\times p$ all-zero matrix, symbol $X^{\top }$ denotes the transpose of the matrix X, and symbol $\mathrm{tr}\left(X\right)$ denotes the trace of the square matrix X, i.e., the sum of its in-diagonal entries. The special orthogonal group is denoted and defined by $\mathrm{SO}\left(m\right):=\{X\in {\mathbb{R}}^{m\times m}|X^{\top }X=I_m,det\left(X\right)=1\}$. For details on this Lie group, see, e.g., [44]. Furthermore, the Frobenius norm of a matrix $X\in {\mathbb{R}}^{p\times p}$ is denoted and defined by ${\parallel X\parallel }_{\mathrm{F}}:=\sqrt{\mathrm{tr}\left(X^{\top }X\right)}$.

2. Helmke–Moore Learning and Advanced Numerical Methods

In the present section, we recall some essential details about the Helmke and Moore (HM) learning algorithm from [1,45]. Moreover, we survey a first-order and two second-order numerical methods to implement a learning paradigm formulated as an initial value problem (IVP, also termed the Cauchy problem) and propose numerical schemes to extend these methods to Riemannian manifolds, of which Lie groups are instances.

There exists a large variety of numerical methods to solve initial value problems. In order to limit the computational burden of the learning rules considered in this paper, we considered a Euler method, a Heun method, and a Runge method (also known as the Runge–Kutta method of the second order). These methods were developed to solve initial values problems over the Euclidean space ${\mathbb{R}}^m$, although they can be adapted to solve initial value problems over smooth manifolds or Lie groups, as the group $\mathrm{SO}\left(m\right)$.

2.1. An HM-Type Neural SVD Learning Algorithm

Denoting as $Z\in {\mathbb{R}}^{m\times p}$ a matrix whose SVD is to be computed and as $r\le min\{m,p\}$ the rank of Z, its singular value decomposition may be written $Z=UD{V}^{\top }$, where $U\in {\mathbb{R}}^{m\times m}$ and $V\in {\mathbb{R}}^{p\times p}$ are (special) orthogonal matrices and D is a pseudo-diagonal matrix that has all-zero values except for the first r diagonal entries, termed singular values. It is easily checked that the columns of U coincide with the eigenvectors of the product $Z{Z}^{\top }$, while V contains the eigenvectors of the self-product $Z^{\top }Z$, with the same eigenvalues.

The Helmke–Moore algorithm is utilized for training, in an unsupervised way, an artificial neural network to learn an SVD of a given rectangular matrix. The HM dynamics arises from the maximization of a specific criterion ${\varphi }_W:\mathrm{SO}\left(m\right)\times \mathrm{SO}\left(m\right)\to \mathbb{R}$ defined as:

$${\varphi }_W(A,B):=2\mathrm{tr}\left(W{A}^{\top }ZB\right),$$

(1)

where $W\in {\mathbb{R}}^{p\times m}$ is a weighting kernel and we assumed that $m⩾p$. The dynamical system, derived as a Riemannian gradient flow on $\mathrm{SO}\left(m\right)\times \mathrm{SO}\left(m\right)$, reads:

$$\left\{\begin{array}{c}H:=A^{\top }ZB,\hfill \\ \dot{A}=A(W^{\top }{H}^{\top }-HW),A\left(0\right)=A_0\in \mathrm{SO}\left(m\right),\hfill \\ \dot{B}=B(WH-H^{\top }{W}^{\top }),B\left(0\right)=B_0\in \mathrm{SO}\left(m\right),\hfill \end{array}\right.$$

(2)

where an over-dot denotes derivation with respect to the time parameter. The weighting matrix W has the structure $\left[W_1{0}_{p,m-p}\right]$, where $W_1\in {\mathbb{R}}^{p\times p}$ must be diagonal with unequal entries on the main diagonal [45].

Since the matrix A has size $m\times m$, the matrix Z has size $m\times p$, and the matrix B has size $p\times p$, then the product matrix H has size $m\times p$.

Whereas the continuous-time versions of the learning algorithms leave the orthogonal group invariant, this is not true for their discrete-time counterparts, which are obtained by employing a numerical integration scheme, unless a suitable integration method is put into effect. In the present case, we may employ a convenient Lie integration method drawn from a manifold-calculus-based integration theory (see, e.g., the contribution [46] and previous reviews in [47,48,49]).

2.2. Euler Method in ${\mathbb{R}}^m$ and Its Extension to $\mathrm{SO}\left(m\right)$

Consider the following initial value problem:

$$\dot{y}=f(t,y),$$

(3)

with $f:\mathbb{R}\times {\mathbb{R}}^m\to {\mathbb{R}}^m$ being a regular function and with the initial condition $y\left(0\right)=y_0\in {\mathbb{R}}^m$. Whenever a closed-form solution to the IVP (3) is out of reach, the simplest numerical scheme to approximate numerically its solution is the forward Euler method described by the iterative rule:

$$y_{n+1}=y_n+hf(t_n,y_n),n⩾0,$$

(4)

with $y_0$ known from the initial condition. As a reference for this method and those invoked in the continuation of the paper, readers might refer to [50]. The constant $h>0$ denotes an integration stepsize and represents the time lapse between each time-node $t_n=nh$ and the next. The key idea behind the above forward Euler method is to estimate the value $y_{n+1}$ by the slope of the vector field $f(t,y)$ at the present node through a linear interpolation across the time lapse h. This Euler method is asymmetric: it makes use only of the value of the vector field f computed in the leftmost point of the interval $[t_n,t_{n+1}]$. The right-hand side of the expression (4) coincides with the zeroth and the first terms of the Taylor series expansion of the function f. The residue owing to truncation (also termed “truncation error”) is of type $\mathcal{O}\left(h^2\right)$; therefore, this method is of order one.

A learning problem on the special orthogonal group is described by the IVP:

$$\dot{Y}=f(t,Y),f:\mathbb{R}\times \mathrm{SO}\left(m\right)\to \mathrm{SO}\left(m\right),$$

(5)

with the initial condition $Y\left(0\right)=Y_0\in \mathrm{SO}\left(m\right)$.

Let us recall that a smooth manifold $\mathbb{M}$ may be endowed with a manifold exponential map and by a parallel transport operator:

• Manifold exponential: The exponential map derived from the geodesic associated with the connection is denoted by $exp:T\mathbb{M}\to \mathbb{M}$. Given $x\in \mathbb{M}$ and $v\in T_x\mathbb{M}$, the exponential of v at a point x is denoted as ${exp}_x\left(v\right)$. On a Euclidean manifold $\mathbb{M}={\mathbb{R}}^m$, ${exp}_x\left(v\right)=x+v$; therefore, an exponential map may be thought of as a generalization of “vector addition” to a curved space.
• Parallel transport along geodesics: The parallel transport map is denoted by $\mathrm{P}:\mathbb{M}\times T\mathbb{M}\to T\mathbb{M}$. Given two points $x,y\in \mathbb{M}$, parallel transport from x to y is denoted as ${\mathrm{P}}^{x\to y}:T_x\mathbb{M}\to T_y\mathbb{M}$. Parallel transport moves a tangent vector between two points along their connecting geodesic arc. Given $x,y\in \mathbb{M}$ and $v\in T_x\mathbb{M}$, the parallel transport of a vector v from a point x to a point y is denoted by ${\mathrm{P}}^{x\to y}\left(v\right)$. Parallel transport does not change the length of a transported vector, namely it is an isometry; moreover, parallel transport does not alter the angle between transported vectors, namely, given $x,y\in \mathbb{M}$ and $a,v\in T_x\mathbb{M}$, it holds that ${\langle {\mathrm{P}}^{x\to y}\left(a\right),{\mathrm{P}}^{x\to y}\left(v\right)\rangle }_y={\langle a,v\rangle }_x$, i.e., we say that parallel transport is a conformal map. Formally, on a Euclidean manifold $\mathbb{M}={\mathbb{R}}^m$, ${\mathrm{P}}^{x\to y}\left(v\right)=v$; therefore, a vector transport may be thought of as a generalization of the familiar geometric notion of the “rigid translation” of vectors to a curved space.

In the case of a special orthogonal group endowed with its canonical metric (inherited by the Euclidean metric), these maps read:

$$\left\{\begin{array}{c}{exp}_X\left(V\right):=X\mathrm{Exp}\left(X^{\top }V\right),(\mathrm{Exponential} \mathrm{map})\hfill \\ {\mathrm{P}}^{X\to Y}\left(V\right):=X\sqrt{X^{\top }Y}{X}^{\top }V\sqrt{X^{\top }Y},(\mathrm{Parallel} \mathrm{transport} \mathrm{operator})\hfill \end{array}\right.$$

(6)

where $X,Y\in \mathrm{SO}\left(m\right)$, $V\in T_X\mathrm{SO}\left(m\right)$, “Exp” denotes a matrix exponential (implemented, for example, by the function expm in MATLAB${}^{\circledR }$), $\sqrt{·}$ denotes a matrix square-root (implemented, for example, by the function sqrtm in MATLAB${}^{\circledR }$). Let us recall that a matrix $V\in T_X\mathrm{SO}\left(m\right)$ may be rewritten as $V=X\mathsf{\Omega }$, with $\mathsf{\Omega }$ skew-symmetric and, therefore, that the following simplified expression for the exponential map may be used:

$${exp}_X\left(X\mathsf{\Omega }\right)=X\mathrm{Exp}\left(\mathsf{\Omega }\right).$$

(7)

A rule of thumb to extend the forward Euler method to smooth manifolds is that the sum between a variable representing a point and a vector field at that point needs to be replaced by the exponential map applied to the point and to the vector field at that point. As a result, a Euler method on a manifold $\mathrm{SO}\left(m\right)$ may be expressed as:

$$Y_{n+1}={exp}_{Y_n}\left(hf(t_n,Y_n)\right),$$

(8)

with $Y_0$ known from the initial condition. Likewise, the original forward Euler method, the method (8), estimates the value of the solution at the temporal node $t_{n+1}$, namely $Y_{n+1}$, as the value of the solution at the node $t_n$ interpolated by the short geodesic arc departing from the current point $Y_n$ toward the direction indicated by the vector field $f(t_n,Y_n)$.

2.3. Heun and Runge Methods in ${\mathbb{R}}^m$ and Their Extension to $\mathrm{SO}\left(m\right)$

The following methods are second-order numerical schemes that may be adopted to solve the Cauchy problem (3).

Heun’s method reads:

$$\left\{\begin{array}{c}{k}_{1,n}:=f(t_n,y_n),\hfill \\ {\tilde{y}}_{n+1}:=y_n+h{k}_{1,n},\hfill \\ k_{2,n}:=f(t_n+h,{\tilde{y}}_{n+1}),\hfill \\ y_{n+1}=y_n+\frac{h}{2}(k_{1,n}+k_{2,n}),\hfill \end{array}\right.$$

(9)

where $h>0$ denotes again a stepsize (or time lapse between two adjacent time nodes). It is worth summarizing an interpretation of the equations that describe the Heun method: The quantity $k_1$ represents an estimation of the value of the vector field f at the leftmost point of the time lapse $[t_n,t_n+h]$, while the quantity $k_2$ represents an estimation of the value of the vector field f at the rightmost point of the time lapse $[t_n,t_n+h]$. Both $k_1$ and $k_2$ are estimations, in that the expression for the quantity $k_{1,n}$ uses the value $y_n$ that is an estimation of the solution of the Cauchy problem obtained in the previous iteration, while the expression for $k_{2,n}$ utilizes an estimation of $y_{n+1}$, indicated by ${\tilde{y}}_{n+1}$, based on a linear interpolation from $y_n$ in the direction $k_{1,n}$. The actual estimation of the solution at $t_n+h$ is obtained as a linear interpolation in a direction computed as the arithmetic average between the directions $k_1$ and $k_2$.

Runge’s method (often denoted as “RK2”) is simpler than Heun’s, although presenting the same precision order, and is expressed as:

$$\left\{\begin{array}{c}{k}_{1,n}:=f(t_n,y_n),\hfill \\ y_{n+1/2}:=y_n+\frac{h}{2}{k}_{1,n},\hfill \\ k_{2,n}:=f(t_n+\frac{h}{2},y_{n+1/2}),\hfill \\ y_{n+1}=y_n+h{k}_{2,n}.\hfill \end{array}\right.$$

(10)

The quantity $k_1$ represents again an estimation of the value of the vector field f at the leftmost point of the time lapse $[t_n,t_n+h]$, while the quantity $k_2$ represents an estimate of the value of the vector field f at the midpoint of the time lapse $[t_n,t_n+h]$. Both $k_1$ and $k_2$ are estimations, rather than being exact evaluations. In particular, the expression for $k_{2,n}$ utilizes an estimation of the exact solution to the Cauchy problem, denoted by ${\tilde{y}}_{n+1/2}$, based on a linear interpolation from $y_n$ in the direction $k_{1,n}$ extended only to half of a whole step. The actual estimation of the solution at $t_n+h$ is obtained as a linear interpolation, from $y_n$ in a direction $k_2$ extended to a whole step.

The above second-order numerical methods may be extended to a smooth manifold by recalling two simple rules-of-thumb:

• The sum between a variable representing a point on a manifold and a vector field at that point are replaced by the exponential map applied to the point and to the vector field at that point.
• The sum between two tangent vectors belonging to two different tangent spaces may be carried out only upon transporting one of the vectors to the tangent space where the other vector lives by means of parallel transport.

By applying the above rules, it is possible to extend the Heun method and the Runge method from the Euclidean space ${\mathbb{R}}^m$ to the manifold $\mathrm{SO}\left(m\right)$.

Heun’s method on $\mathrm{SO}\left(m\right)$ may be expressed as:

$$\left\{\begin{array}{c}{K}_{1,n}:=f(t_n,Y_n)\in T_{Y_n}\mathrm{SO}\left(m\right),\hfill \\ {\tilde{Y}}_{n+1}:={exp}_{Y_n}\left(h{K}_{1,n}\right),\hfill \\ K_{2,n}:=f(t_n+h,{\tilde{Y}}_{n+1})\in T_{{\tilde{Y}}_{n+1}}\mathrm{SO}\left(m\right),\hfill \\ Y_{n+1}={exp}_{Y_n}\left(\frac{h}{2}\left(K_{1,n}+{\mathrm{P}}^{{\tilde{Y}}_{n+1}\to Y_n}\left(K_{2,n}\right)\right)\right).\hfill \end{array}\right.$$

(11)

The above extension of the original Heun method was derived by following faithfully Heun’s concept and by replacing the linear operations with manifold operations. In particular, notice that the mean direction between $K_1$ and $K_2$ cannot be calculated as $\frac{1}{2}(K_1+K_2)$ because $K_1$ and $K_2$ belong to different tangent spaces.

Runge’s method on $\mathrm{SO}\left(m\right)$ may be expressed as:

$$\left\{\begin{array}{c}{K}_{1,n}:=f(t_n,Y_n)\in T_{Y_n}\mathrm{SO}\left(m\right),\hfill \\ Y_{n+1/2}:={exp}_{Y_n}\left(\frac{h}{2}{K}_{1,n}\right),\hfill \\ K_{2,n}:=f(t_n+\frac{h}{2},Y_{n+1/2})\in T_{Y_{n+1/2}}\mathrm{SO}\left(m\right),\hfill \\ Y_{n+1}={exp}_{Y_n}\left(h{\mathrm{P}}^{Y_{n+1/2}\to Y_n}\left(K_{2,n}\right)\right).\hfill \end{array}\right.$$

(12)

The interpretation of the equations that constitute Runge’s method on the manifold $\mathrm{SO}\left(m\right)$ is completely faithful to the original method.

In the following subsection, the forward Euler method, the Heun method, and the Runge method are applied to solve the HM learning system of IVPs (2). This application leads to facing at least two challenges: (1) the HM system is made of two differential equations, which entails the application of each numerical method twice; this gives rise to a non-univocality in the extension of the equations, which will then be presented in different versions; (2) the curved space $\mathrm{SO}\left(m\right)$ may be treated as a smooth Riemannian manifold and as a Lie group; this gives rise to two different ways to design numerical methods, which will be explored and discussed in the following, even by the help of preliminary numerical tests.

2.4. Application of the Euler, Heun, and Runge Method to Solving an HM System

The HM-type learning system to solve is formed by two coupled neural learning equations, namely:

$$\left\{\begin{array}{c}\dot{A}=-A\sigma \left(A^{\top }ZBW\right),\hfill \\ \dot{B}=B\sigma \left(W{A}^{\top }ZB\right),\hfill \end{array}\right.$$

(13)

where the skew-symmetrization operator $\sigma \left(X\right):=X^{\top }-X$ has been introduced and where the initial conditions $A\left(0\right)=A_0$ and $B\left(0\right)=B_0$ have been fixed.

In the following, we shall outline three different classes of numerical methods to tackle the learning problem (13), namely the (forward) Euler class, the (explicit second-order) Heun method, and the (explicit) Runge method (a second-order instance from the general class of Runge–Kutta methods). For the sake of notation compactness, in the following, we shall make use of the twin vector fields:

$$f^A(A,B):=-A\sigma \left(A^{\top }ZBW\right),f^B(A,B):=B\sigma \left(W{A}^{\top }ZB\right).$$

(14)

The numerical schemes adopted in this study are first- and second-order explicit. Implicit methods are overly complex, as they require solving a non-linear problem at every iteration (the computations involved in the implicit methods may be as complex as the problem they were designed to solve). Higher order formulations (such as the fourth-order Runge–Kutta method) are often inappropriate in optimization, because they were designed to provide a highly accurate estimation of the solution to an initial value problem at every step, while in optimization, such accuracy would be out of the scope as the only step that needs accuracy is the final optimization goal.

2.4.1. Forward Euler Method

The Euler method on $\mathrm{SO}\left(m\right)$ outlined in Section 2.2, applied to solve an HM learning system, may be expressed as:

$$\mathbf{Euler}:\left\{\begin{array}{c}{A}_{n+1}={exp}_{A_n}\left(h^A{f}^A(A_n,B_n)\right),\hfill \\ B_{n+1}={exp}_{B_n}\left(h^B{f}^B(A_n,B_n)\right),\hfill \end{array}\right.$$

(15)

with $h^A>0$ and $h^B>0$ being two different learning stepsizes.

2.4.2. Explicit Second-Order Heun Method

The Heun method outlined in Section 2.3, applied to an HM learning problem on $\mathrm{SO}\left(m\right)$, as a manifold, may be expressed as:

$$\mathbf{Heun},\mathbf{Version}\mathbf{1}:\left\{\begin{array}{c}{K}_{1,n}^A:=f^A(A_n,B_n),\hfill \\ {\tilde{A}}_{n+1}:={exp}_{A_n}\left(h^A{K}_{1,n}^A\right),\hfill \\ K_{2,n}^A:=f^A({\tilde{A}}_{n+1},B_n),\hfill \\ A_{n+1}={exp}_{A_n}\left(\frac{h^A}{2}\left(K_{1,n}^A+{\mathrm{P}}^{{\tilde{A}}_{n+1}\to A_n}\left(K_{2,n}^A\right)\right)\right),\hfill \\ K_{1,n}^B:=f^B(A_n,B_n),\hfill \\ {\tilde{B}}_{n+1}:={exp}_{B_n}\left(h^B{K}_{1,n}^B\right),\hfill \\ K_{2,n}^B:=f^B(A_n,{\tilde{B}}_{n+1}),\hfill \\ B_{n+1}={exp}_{B_n}\left(\frac{h^B}{2}\left(K_{1,n}^B+{\mathrm{P}}^{{\tilde{B}}_{n+1}\to B_n}\left(K_{2,n}^B\right)\right)\right).\hfill \end{array}\right.$$

(16)

It is interesting to observe that the same numerical learning algorithm may be recast in a number of slightly different ways by operating some tiny variations on the equations. An alternative version to the original Heun, Version 1 (16) is:

$$\mathbf{Heun},\mathbf{Version}\mathbf{2}:\left\{\begin{array}{c}{K}_{1,n}^A:=f^A(A_n,B_n),\hfill \\ {\tilde{A}}_{n+1}:={exp}_{A_n}\left(h^A{K}_{1,n}^A\right),\hfill \\ K_{2,n}^A:=f^A({\tilde{A}}_{n+1},B_n),\hfill \\ A_{n+1}={exp}_{A_n}\left(\frac{h^A}{2}\left(K_{1,n}^A+{\mathrm{P}}^{{\tilde{A}}_{n+1}\to A_n}\left(K_{2,n}^A\right)\right)\right),\hfill \\ K_{1,n}^B:=f^B(\underline{A_{n+1}},B_n),\hfill \\ {\tilde{B}}_{n+1}:={exp}_{B_n}\left(h^B{K}_{1,n}^B\right),\hfill \\ K_{2,n}^B:=f^B(\underline{A_{n+1}},{\tilde{B}}_{n+1}),\hfill \\ B_{n+1}={exp}_{B_n}\left(\frac{h^B}{2}\left(K_{1,n}^B+{\mathrm{P}}^{{\tilde{B}}_{n+1}\to B_n}\left(K_{2,n}^B\right)\right)\right),\hfill \end{array}\right.$$

(17)

where the differences have been highlighted. Notice that both versions are explicit numerical schemes.

From a computational burden viewpoint, it is important to underline that the above numerical algorithms require the repeated calculation of parallel transport that contributes non-negligibly to increasing the computational complexity of these methods.

As an alternative, the Heun method may be rewritten without using parallel transport by composing the partial steps in a different way over the space $\mathrm{SO}\left(m\right)$, namely:

$$\mathbf{Heun},\mathbf{Version}\mathbf{3}:\left\{\begin{array}{l}{K}_{1,n}^A:=f^A(A_n,B_n),\hfill \\ {\tilde{A}}_{n+1}:={exp}_{A_n}\left(h{K}_{1,n}^A\right),\\ K_{2,n}^A:=f^A({\tilde{A}}_{n+1},B_n),\\ \underline{A_{n+1}=A_n·{\tilde{A}}_{n+1}^{-1}{exp}_{{\tilde{A}}_{n+1}}\left(\frac{h}{2}{K}_{2,n}^A\right)·A_n^{-1}{exp}_{A_n}\left(\frac{h}{2}{K}_{1,n}^A\right),}\\ K_{1,n}^B:=f^B(\underline{A_{n+1}},B_n),\hfill \\ {\tilde{B}}_{n+1}:={exp}_{B_n}\left(h{K}_{1,n}^B\right),\\ K_{2,n}^B:=f^B(\underline{A_{n+1}},{\tilde{B}}_{n+1}),\\ \underline{B_{n+1}=B_n·{\tilde{B}}_{n+1}^{-1}{exp}_{{\tilde{B}}_{n+1}}\left(\frac{h}{2}{K}_{2,n}^B\right)·B_n^{-1}{exp}_{B_n}\left(\frac{h}{2}{K}_{1,n}^B\right).}\end{array}\right.$$

(18)

The differences with the Heun method, Version 1 (16), have been highlighted. It is interesting to observe that, by swapping the order of the steps in $A_{n+1}$ e $B_{n+1}$ ($1\leftrightarrow 2$), we get:

$$\left\{\begin{array}{c}{A}_{n+1}=\cancel{A_n·A_n^{-1}}{exp}_{A_n}\left(\frac{h}{2}{K}_{1,n}^A\right)·{\tilde{A}}_{n+1}^{-1}·{exp}_{{\tilde{A}}_{n+1}}\left(\frac{h}{2}{K}_{2,n}^A\right),\hfill \\ B_{n+1}=\cancel{B_n·B_n^{-1}}{exp}_{B_n}\left(\frac{h}{2}{K}_{1,n}^B\right)·{\tilde{B}}_{n+1}^{-1}·{exp}_{{\tilde{B}}_{n+1}}\left(\frac{h}{2}{K}_{2,n}^B\right),\hfill \end{array}\right.$$

(19)

which is computationally lighter compared to the Heun method, Version 3 (18). It is immediate to verify that Version 3 of the Heun method, further modified by the above equations, will result in being computationally lighter than the versions requiring parallel transport; therefore, the Heun method considered in the following preliminary numerical test will be:

$$\mathbf{Heun},\mathbf{Version}\mathbf{4}:\left\{\begin{array}{c}{K}_{1,n}^A:=f^A(A_n,B_n),\hfill \\ {\tilde{A}}_{n+1}:={exp}_{A_n}\left(h{K}_{1,n}^A\right),\hfill \\ K_{2,n}^A:=f^A({\tilde{A}}_{n+1},B_n),\hfill \\ A_{n+1}={exp}_{A_n}\left(\frac{h}{2}{K}_{1,n}^A\right)·{\tilde{A}}_{n+1}^{-1}·{exp}_{{\tilde{A}}_{n+1}}\left(\frac{h}{2}{K}_{2,n}^A\right),\hfill \\ K_{1,n}^B:=f^B(A_{n+1},B_n),\hfill \\ {\tilde{B}}_{n+1}:={exp}_{B_n}\left(h{K}_{1,n}^B\right),\hfill \\ K_{2,n}^B:=f^B(A_{n+1},{\tilde{B}}_{n+1}),\hfill \\ B_{n+1}={exp}_{B_n}\left(\frac{h}{2}{K}_{1,n}^B\right)·{\tilde{B}}_{n+1}^{-1}·{exp}_{{\tilde{B}}_{n+1}}\left(\frac{h}{2}{K}_{2,n}^B\right),\hfill \end{array}\right.$$

(20)

2.4.3. Explicit Second-Order Runge Method

The Runge method outlined in Section 2.3, applied to an HM learning problem on $\mathrm{SO}\left(m\right)$, as a manifold, may be expressed as:

$$\mathbf{Runge},\mathbf{Version}\mathbf{1}:\left\{\begin{array}{c}{K}_{1,n}^A:=f^A(A_n,B_n),\hfill \\ A_{n+1/2}:={exp}_{A_n}\left(\frac{h^A}{2}{K}_{1,n}^A\right),\hfill \\ K_{2,n}^A:=f^A(A_{n+1/2},B_n),\hfill \\ A_{n+1}={exp}_{A_n}\left(h^A{\mathrm{P}}^{A_{n+1/2}\to A_n}\left(K_{2,n}^A\right)\right),\hfill \\ K_{1,n}^B:=f^B(A_n,B_n),\hfill \\ B_{n+1/2}:={exp}_{B_n}\left(\frac{h^B}{2}{K}_{1,n}^B\right),\hfill \\ K_{2,n}^B:=f^B(A_n,B_{n+1/2}),\hfill \\ B_{n+1}={exp}_{B_n}\left(h^B{\mathrm{P}}^{B_{n+1/2}\to B_n}\left(K_{2,n}^B\right)\right).\hfill \end{array}\right.$$

(21)

The same method may be expressed in a slightly different way, namely by exchanging the order of adaptation of the variables A and B:

$$\mathbf{Runge},\mathbf{Version}\mathbf{2}:\left\{\begin{array}{c}{K}_{1,n}^B:=f^B(A_n,B_n),\hfill \\ B_{n+1/2}:={exp}_{B_n}\left(\frac{h^B}{2}{K}_{1,n}^B\right),\hfill \\ K_{2,n}^B:=f^B(A_n,B_{n+1/2}),\hfill \\ B_{n+1}={exp}_{B_n}\left(h^B{\mathrm{P}}^{B_{n+1/2}\to B_n}\left(K_{2,n}^B\right)\right),\hfill \\ K_{1,n}^A:=f^A(A_n,B_n),\hfill \\ A_{n+1/2}:={exp}_{A_n}\left(\frac{h^A}{2}{K}_{1,n}^A\right),\hfill \\ K_{2,n}^A:=f^A(A_{n+1/2},B_n),\hfill \\ A_{n+1}={exp}_{A_n}\left(h^A{\mathrm{P}}^{A_{n+1/2}\to A_n}\left(K_{2,n}^A\right)\right).\hfill \end{array}\right.$$

(22)

The last version of the HM-type learning systems is obtained by regarding the curved space $\mathrm{SO}\left(m\right)$ as a Lie group with Lie algebra $\mathfrak{so}\left(m\right)$, which leads to the following numerical method:

$$\mathbf{Runge},\mathbf{Version}\mathbf{3}:\left\{\begin{array}{l}{T}_{1,n}^A:=-\sigma \left(A_n^{\top }Z{B}_{n}W\right)\in \mathfrak{so}\left(m\right),\hfill \\ A_{n+1/2}:=A_n\mathrm{Exp}\left(\frac{h}{2}{T}_{1,n}^A\right),\hfill \\ T_{2,n}^A:=-\sigma \left(A_{n+1/2}^{\top }Z{B}_{n}W\right)\in \mathfrak{so}\left(m\right),\hfill \\ \underline{A_{n+1}=A_n\mathrm{Exp}\left(\frac{h}{6}{T}_{1,n}^A+\frac{h}{3}{T}_{2,n}^A\right)},\\ T_{1,n}^B:=\sigma \left(W{A}_n^{\top }Z{B}_n\right)\in \mathfrak{so}\left(m\right),\\ B_{n+1/2}:=B_n\mathrm{Exp}\left(\frac{h}{2}{T}_{1,n}^B\right),\\ T_{2,n}^B:=\sigma \left(W{A}_n^{\top }Z{B}_{n+1/2}\right)\in \mathfrak{so}\left(m\right),\\ \underline{B_{n+1}=B_n\mathrm{Exp}\left(\frac{h}{6}{T}_{1,n}^B+\frac{h}{3}{T}_{2,n}^B\right).}\end{array}\right.$$

(23)

Even this version of the numerical learning scheme does not require using the computationally intensive parallel transport. The highlighted update equations were drawn from the survey [51].

2.5. Preliminary Pyramidal Numerical Tests on the HM Learning Methods

This subsection illustrates the results of a pyramidal comparison of the above learning algorithms in order to find out which algorithm is more convenient from a computational viewpoint, for comparable learning performances. The numerical experiments concerned the computation of the SVD of random matrices Z. The numerical experiments discussed in this subsection were performed on an Intel${}^{\circledR }$ i5-6400T 4-Core CPU, 2.2 GHz clock, 8 GB RAM machine and were coded in a MATLAB${}^{\circledR }$ R2017b environment (we used non-parallel programming in the experiments). In the following experiments, the learning stepsizes $h_A$ and $h_B$ were set to a common value denoted by h. Moreover, we adopted the following criterion to stop the iteration based on the objective function (1):

$${\varphi }_W(A_{n+1},B_{n+1})-{\varphi }_W(A_n,B_n)⩽\tau ,$$

(24)

where $\tau$ denotes a threshold. The size of the unknown-matrix B was set to $p:=3$ in all experiments, and the sub-matrix $W_1$ explained in the Section 2.1 was set to $\mathrm{diag}(3,2,1)$. The size m of the matrix B took values in $\{9,25,225\}$ (these specific values would arise in the computation of the optical-flow of $720\times 720$ pixels frames, as will be shown in Section 4.2). In all experiments, the initial matrices were chosen as $A_0:=I_m$ and $B_0:=I_3$. For comparison purposes, the SVD of a matrix Z was also computed by MATLAB’s SVD engine, whose output triple is denoted by $(U,·,V)$. As objective measures of the quality of the learned SVD factors, we used the following figures of demerit:

$$\left\{\begin{array}{c}{\omega }_A\left(A_n\right):={\parallel A_n^{\top }{A}_n-I_m\parallel }_{\mathrm{F}},(\mathrm{Non}-\mathrm{orthogonality})\hfill \\ {\omega }_B\left(B_n\right):={\parallel B_n^{\top }{B}_n-I_3\parallel }_{\mathrm{F}},\hfill \\ {\epsilon }_A\left(A_n\right):=\parallel |A_n^{\top }U{|-I_m\parallel }_{\mathrm{F}},\left(\mathrm{Discrepancy}\right)\hfill \\ {\epsilon }_B\left(B_n\right):=\parallel |B_n^{\top }V{|-I_3\parallel }_{\mathrm{F}},\hfill \end{array}\right.$$

(25)

where $|·|$ denotes an entry-wise absolute value of a matrix. Both numerical orthogonality and discrepancy between the learned SVD and the MATLAB-computed reference SVD were expected to result in being as close as possible to zero.

The results of the first experiment, which was meant to compare the performances of the Heun method in its Versions 1 and 4, are summarized in

Table 1. Results of a comparison between the runtimes (in seconds) of the Heun method in its Versions 1 and 4.

	Size of A	$\mathit{m}:=9$	$\mathit{m}:=25$	$\mathit{m}:=225$
Method
Heun, Version 1		0.3836	0.3468	34.5559
Heun, Version 4		0.2264	0.1148	9.1531

. The random matrix Z was generated entry-wise as a set of random numbers uniformly distributed in the interval $[0,1]$. This choice corresponded to a normalized pixel luminance, and the uniform distribution was motivated by the fact that a single pixel location may in principle take any luminance value. In the low-dimensional cases, $h:=0.05$, while in the higher dimensional case, $h:=0.005$. In all cases, $\tau :={10}^{-5}$. The acquired runtimes showed that Version 4 was much lighter than Version 1, a phenomenon that became more apparent as the size of the matrix A increased.

A graphical comparison of the values taken by the discrepancies defined in (25) and of the learning criterion (1) during learning is displayed in Figure 1. In this case, $m:=9$. The discrepancy and learning curves showed that, as concerns the learning performances, no meaningful differences between the two versions of the HM-type learning algorithms could be appreciated.

The results of this experiment entailed that the Heun method, Version 4 was the preferable version in the Heun class.

The results of the second experiment, which was meant to compare the performances of the Runge method in its Versions 1 and 3, are summarized in

Table 2. Results of a comparison between the runtimes (in seconds) of the Runge method in its Versions 1 and 3.

	Size of A	$\mathit{m}:=9$	$\mathit{m}:=25$	$\mathit{m}:=225$
Method
Runge, Version 1		0.4536	0.8816	65.1238
Runge, Version 3		0.3645	0.5274	21.7729

. In the low-dimensional cases, $h:=0.05$, while in the higher dimensional case, $h:=0.005$. In all cases, $\tau :={10}^{-5}$. The acquired runtimes showed that Version 3 was lighter than Version 1.

A graphical comparison of the values taken by the discrepancies and of the learning criterion during learning is displayed in the Figure 2. In this case, $m:=9$. The discrepancy and learning curves showed that, as concerns the learning performances, Version 1 converged only slightly more rapidly to the expected solutions.

The results of this experiment entailed that the Runge method, Version 3 was the preferable version in the Runge class.

The results of the last experiment of this subsection, which was meant to compare the performances of the best Runge method and of the best Heun method to the performances of the Euler method, are summarized in

Table 3. Results of a comparison between the runtimes (in seconds) of the Heun method, Version 4, the Runge method, Version 3, and the Euler method.

	Size of A	$\mathit{m}:=9$	$\mathit{m}:=25$	$\mathit{m}:=225$
Method
Heun, Version 4		0.2033	0.3219	20.0415
Runge, Version 3		0.2060	0.3601	26.3659
Forward Euler		0.0740	0.1096	8.5624

. In the low-dimensional cases, $h:=0.05$, while in the higher dimensional case, $h:=0.005$. In all cases, $\tau :={10}^{-5}$. The acquired runtimes revealed that the Euler method was the lightest in terms of computational complexity.

A graphical comparison of the values taken by the discrepancies and of the learning criterion during learning is displayed in Figure 3. In this case, $m:=225$. The discrepancy and learning curves showed that, as concerns the learning performances, the Euler method-based HM learning algorithm and the Heun method-based learning algorithm converged more rapidly to the expected solutions.

A graphical comparison of the values taken by the non-orthogonality figures defined in (25) during learning is displayed in Figure 4. In this case, $m:=225$. The orthogonality curves showed that the Euler method-based HM was unable to keep the same numerical precision of the Heun-based and of the Runge-based HM learning algorithms, the former being a first-order method and the latter two second-order methods. However, all these algorithms kept the non-orthogonality figures at very low values and stabilized after learning completion.

The results of these three experiments revealed that the Euler method to implement a Helmke–Moore learning paradigm guaranteed an excellent trade-off between computational complexity and numerical precision.

3. Reduced Euler Method

In some applications, such as optical-flow estimation recalled in the next Section 4, only the matrix B is relevant. In view of the optical-flow application discussed in the next section, assume that B is a $3\times 3$ factor of the singular value decomposition of a data matrix Z. Notice that, in general, the matrix A has a much bigger size than the matrix B; therefore, it would be certainly convenient to express the matrix A as a function of the matrix B and to express the learning system (2) only in terms of the variable matrix B. The following arguments may be extended easily to the general case that B has different sizes.

3.1. Derivation of a Reduced Euler Method

The HM-type learning system (2) in the form (13) admits, as stable configurations, all pairs $(A,B)\in \mathrm{SO}\left(m\right)\times \mathrm{SO}\left(3\right)$ such that:

$$\left\{\begin{array}{c}\sigma \left(A^{\top }ZBW\right)=0,\hfill \\ \sigma \left(W{A}^{\top }ZB\right)=0.\hfill \end{array}\right.$$

(26)

Therefore, one of these conditions may be used to express the matrix variable A in terms of the matrix variable B. In particular, the first condition, in plain form, reads:

$$W^{\top }{\left(A^{\top }ZB\right)}^{\top }-A^{\top }ZBW=0.$$

(27)

The above equation is equivalent to $\left(W^{\top }{B}^{\top }{Z}^{\top }\right)A-A^{\top }\left(ZBW\right)=0$. Defining $K:=W^{\top }{B}^{\top }{Z}^{\top }$, Equation (27) takes the form:

$$KA-A^{\top }{K}^{\top }=0,$$

(28)

which is to be solved in A. Let us observe that the matrix K has size $9\times 9$ and that its last six rows are zero, while the matrix A has size $9\times 9$ and that its last six columns do not influence the result. Let us partition the matrices K and A as follows:

$$K=\left[\begin{array}{c}{K}_1\\ 0\end{array}\right],A=\left[\begin{array}{cc}{A}_1& A_2\end{array}\right],$$

(29)

with $K_1$ is $3\times 9$, $A_1$ is $9\times 3$, and $A_2$ is $9\times 6$. The matrix product $KA$ exhibits the structure:

$$KA=\left[\begin{array}{c}{K}_1\\ 0\end{array}\right]\left[\begin{array}{cc}{A}_1& A_2\end{array}\right]=\left[\begin{array}{cc}{K}_1{A}_1& K_1{A}_2\\ 0& 0\end{array}\right].$$

(30)

According to the condition (28), the product $KA$ needs to result in a symmetric matrix; hence, the product $K_1{A}_1$ needs to be symmetric, while the product $K_1{A}_2$ needs to be zero. Since the matrix $A_2$ plays no role in the product $KA$, it is computationally convenient to turn down the orthogonality of the matrix A and to set $A_2=0_{m\times n}$, while the columns of the sub-matrix $A_1$ still need to be mutually orthogonal and unit-norm. Since $K_1{A}_1$ is symmetric, let us set $K_1{A}_1=:S$, where S is a $3\times 3$ symmetric matrix to be determined. To determine the unknown matrix S, let us rewrite the last equation by the ansatz $K_1^{\top }=A_{1}S$, from which it follows that:

$$K_1{K}_1\top =S{A}_1^{\top }{A}_{1}S=S^2\Rightarrow S=\sqrt{K_1{K}_1^{\top }},$$

(31)

therefore:

$$K_1^{\top }=A_1\sqrt{K_1{K}_1^{\top }}\Rightarrow A_1=K_1^{\top }{\left(K_1{K}_1^{\top }\right)}^{-1/2}.$$

(32)

The matrix $K_1$ is defined as a sub-matrix of K, which, in turn, is a function of the matrix B (and depends on the weight matrix W and on the data matrix Z). Hence, the function:

$$F\left(B\right):=\left[\begin{array}{cc}{K}_1^{\top }\left(B\right){\left(K_1\left(B\right)K_1^{\top }\left(B\right)\right)}^{-1/2}& 0\end{array}\right],$$

(33)

may be used to reduce the two update rules in the Euler method (15) to the single update rule:

$$\left\{\begin{array}{c}\tilde{f}\left(B\right):=B\sigma \left(W{F}^{\top }\left(B\right)ZBW\right),\hfill \\ B_{n+1}={exp}_{B_n}\left(h\tilde{f}\left(B_n\right)\right),\hfill \end{array}\right.$$

(34)

with $h>0$ and with the only initial condition $B_0$. Notice that the inverse matrix square root may be expressed in terms of the matrix exponential $\mathrm{Exp}$ and the matrix logarithm $\mathrm{Log}$ as $X^{-1/2}=\mathrm{Exp}\left(-\frac{1}{2}\mathrm{Log}\left(X\right)\right)$. The algorithm (34) will be referred to as the reduced Euler method.

3.2. Numerical Comparison of the Euler Method and Its Reduced Version

The results of a comparison of the performances of the forward Euler method (15) and of the reduced Euler method (34) to implement an HM learning paradigm are summarized in

Table 4. Results of a comparison between the runtimes (in seconds) for the forward Euler method and for the reduced Euler method.

	Size of A	$\mathit{m}:=9$	$\mathit{m}:=25$	$\mathit{m}:=225$
Method
Forward Euler		0.2003	0.1190	7.7378
Reduced Euler		0.0833	0.0441	0.4350

. In the low-dimensional cases, $h:=0.05$, while in the higher dimensional case, $h:=0.005$. In all cases, $\tau :={10}^{-5}$. The acquired runtimes showed that the reduced Euler method was way lighter than the original Euler method, a phenomenon that became more apparent as the size m of the $m\times 3$ matrix Z increased.

A graphical comparison of the values taken by the figures of demerit defined in (25) and of the learning criterion (1) during learning is displayed in Figure 5. In this case, $m:=9$. The discrepancy and learning curves showed that, as concerns the learning performances, no meaningful differences between the two versions of the HM-type learning algorithms can be appreciated.

The results of this experiment confirmed that the reduced Euler method was both effective and efficient to implement the HM learning paradigm.

4. Experiments on Optical Flow Estimation by a Helmke–Moore Neural Learning Algorithm

In this section, we recall the notion of optical-flow estimation by a total least squares method from [42] and illustrate the behavior of the devised learning algorithm by means of numerical experiments performed on a recorded video sequence.

4.1. Optical Flow Estimation by Total Least Squares

Let us consider a gray-scale video sequence ${\left\{\mathcal{I}(x,y,q)\right\}}_q$, where $\mathcal{I}$ denotes the scalar image intensity, the pair $(x,y)$ denotes the coordinate pair of any pixel, and q denotes the frame index.

During motion, any pixel moves from frame q and position $(x,y)$ to frame $q+\Delta q$ at position $(x+\Delta x,y+\Delta y)$. The fact that the pixel intensity has moved over the image support may be formally expressed by the optical-flow conservation equation, namely:

$$\mathcal{I}(x-\Delta x,y-\Delta y,q)=\mathcal{I}(x,y,q+\Delta q).$$

(35)

On the basis of the above conservation law, it is possible to estimate the motion of any pixel within the sequence ${\left\{\mathcal{I}(x,y,q)\right\}}_q$. In fact, let us define $\Delta \mathcal{I}(x,y,q):=\mathcal{I}(x,y,q+\Delta q)-\mathcal{I}(x,y,q)$. An application of the Taylor series expansion gives:

$$\Delta \mathcal{I}(x,y,q)=\mathcal{I}(x-\Delta x,y-\Delta y,q)-\mathcal{I}(x,y,q)=-{\mathcal{I}}_x(x,y,q)\Delta x-{\mathcal{I}}_y(x,y,q)\Delta y+\mathrm{residual},$$

(36)

where ${\mathcal{I}}_x$ and ${\mathcal{I}}_y$ denote the partial derivatives of the image function along the vertical and the horizontal direction, respectively, and the term “residual” denotes the sum of higher order terms in the Taylor expansion of the image function and the sum of random disturbances occurring in the recording of a video sequence, while the vertical and horizontal displacements $(\Delta x,\Delta y)$ were supposed small enough for the above truncated Taylor series to represent accurately the optical-flow change. The latter hypothesis was equivalent to assuming slow motion or, equivalently, sufficiently high-rate image sampling.

As mentioned, we made the solid-block motion assumption, namely the above equation was supposed to hold true, with the same values of displacements, for a set of pixels located within the rectangular patch described by $x\in [x_1,x_{N_x}]$ and $y\in [y_1,y_{N_y}]$, where integers $N_x$ and $N_y$ denote the block-size. On the basis of these considerations, it was possible to write the resolving system for any block between frames q and $q+\Delta q$, that is:

$$\begin{array}{ccc}\hfill \mathcal{I}(x_1,y_1,q+\Delta q)-\mathcal{I}(x_1,y_1,q)& =& -{\mathcal{I}}_x(x_1,y_1,q)\Delta x-{\mathcal{I}}_y(x_1,y_1,q)\Delta y,\hfill \\ \hfill \mathcal{I}(x_2,y_1,q+\Delta t)-\mathcal{I}(x_2,y_1,q)& =& -{\mathcal{I}}_x(x_2,y_1,q)\Delta x-{\mathcal{I}}_y(x_2,y_1,q)\Delta y,\hfill \\ \hfill \mathcal{I}(x_3,y_1,q+\Delta q)-\mathcal{I}(x_3,y_1,q)& =& -{\mathcal{I}}_x(x_3,y_1,q)\Delta x-{\mathcal{I}}_y(x_3,y_1,q)\Delta y,\hfill \\ \hfill ⋮\\ \hfill \mathcal{I}(x_1,y_2,t+\Delta q)-\mathcal{I}(x_1,y_2,q)& =& -{\mathcal{I}}_x(x_1,y_2,q)\Delta x-{\mathcal{I}}_y(x_1,y_2,q)\Delta y,\hfill \\ \hfill ⋮\\ \hfill \mathcal{I}(x_{N_x},y_{N_y},q+\Delta q)-\mathcal{I}(x_{N_x},y_{N_y},q)& =& -{\mathcal{I}}_x(x_{N_x},y_{N_y},q)\Delta x-{\mathcal{I}}_y(x_{N_x},y_{N_y},q)\Delta y,\hfill \end{array}$$

where high-order terms were neglected.

By defining a displacement vector $v:={[\Delta x\Delta y]}^{\top }$ and upon defining a matrix $L\in {\mathbb{R}}^{N_x{N}_y\times 2}$ and a vector $c\in {\mathbb{R}}^{N_x{N}_y}$, the above system casts into $Lv=c$. This is an over-determined linear system of $N_x·N_y$ equations in two unknowns, which may be solved by the help of a total least squares technique [52].

The TLS approach to solving an over-determined system of linear equations is based on the singular value decomposition (SVD) technique [53,54]. The resulting procedure is stated in Algorithm 1.

Algorithm 1 Total least squares based algorithm to estimate a displacement vector.

Define the $N_x{N}_y\times 3$ matrix $Z=\left[Lc\right]$; Compute the SVD $(U,D,V)$ of Z, with $V\in$$\mathrm{SO}\left(3\right)$;   Define the block-partition $V=\left[\begin{array}{cc}\cdots & V_{12}\\ \cdots & V_{22}\end{array}\right]$ with $V_{12}\in {\mathbb{R}}^{2\times 1}$;  Estimate a displacement vector v as $-V_{12}{V}_{22}^{-1}$.

It is interesting to underline that, depending on the size of each block, the data matrix Z whose SVD is to be calculated may be large sized. Moreover, for later considerations, it is important to underline that the only factor of an SVD decomposition that plays an effective role in motion estimation is V.

4.2. Numerical Experiments on Optical-Flow Learning

In order to test the capability of the HM learning paradigm in optical-flow estimation, a video was realized and saved in MP4 format. The pre-processing of the images consisted of applying a filter after the subdivision of the frames in sub-windows: every gradient matrix was examined and then the windows, which presented the null gradient, were excluded from further processing in the successive steps as they corresponded to block of pixels that were not subjected to any movement. In the post-processing, after the velocities along the x- and y-axes were calculated by using an HM learning algorithm, their norms were compared to a threshold, and all the windows that presented a velocity norm larger than the threshold were ignored in the optical-flow estimation as they were considered to correspond to noisy pixels. These filters were built to clean the image from noise created by the environment and by the camera, but they also caused a small loss of information, which could be neglected. In the experiments summarized in the present subsection, the following setting was adopted:

• The optical-flow was estimated between Frame 130 and Frame 135.
• The original colored still images drawn from the two considered frames were converted from RGB to gray scale using the luminance formula $0.2989R+0.5870G+0.1140B$ as prescribed by the recommendation ITU-R BT.601-7 [55].
• The threshold to stop iteration was set to $\tau :=0.001$.
• The learning stepsizes were set as $h_A=h_B=h:=0.0005$.
• Given the frame size of $720\times 720$, each frame was subdivided into windows of size $3\times 3$, $5\times 5$, or $15\times 15$ pixels (corresponding to 57,600 matrices of size $9\times 3$ to compute the SVD, 20,736 matrices of size $25\times 3$ to compute the SVD, and to 2304 matrices of size $225\times 3$ to compute the SVD, respectively).
• Computing platform: Intel${}^{\circledR }$ i7-8550U 4-Core CPU, 1.8 GHz clock, 16 GB RAM. Coding platform: MATLAB${}^{\circledR }$ R2018b.

The computational complexity related to the estimation of the optical-flow between two frames depended on the number of sub-windows and on their size; therefore, it was interesting to test the HM learning algorithms in different instances.

With reference to the HM learning paradigm, in the present application, it holds that $m:=N_x·N_y$ (which depends on the size of a image’s window) while $p:=3$ (fixed). Considering the block decomposition $H=\left[\begin{array}{c}{H}_1\\ \cdots \end{array}\right]$, with $H_1$ of size $3\times 3$, the product $WH$ may be simplified as $W_1{H}_1$, while the product $HW$ may be computed as the composite matrix $\left[H{W}_1{0}_{NxNy,NxNy-3}\right]$.

Table 5. Comparison of runtimes (in seconds) to compute the optical-flow between two frames by the forward Euler method and the reduced Euler method.

	Method	Forward Euler Method	Reduced Euler Method	Time Saving (%)
Window Size
$3\times 3$		1833	907	50.52%
$5\times 5$		1177	606	48.51%
$15\times 15$		15,243	241	98.42%

shows the results of a comparison of runtimes taken to estimate the optical-flow between two frames by the forward Euler method-based HM learning paradigm and the reduced Euler method-based algorithm.

It is very interesting to observe how the computational complexity of the forward Euler method increased with the size of the sub-windows, while the complexity of the reduced Euler method showed an opposite trend. The time saving afforded by the reduced Euler method ranged from about $50\%$ for low-dimensional windows to about $98\%$ for high-dimensional windows.

Figure 6 illustrates the results of optical-flow estimation when the windows size was set to $3\times 3$.

Figure 7 illustrates the results of optical-flow estimation when the windows size was set to $5\times 5$.

Figure 8 illustrates the results of optical-flow estimation when the windows size was taken as $15\times 15$.

The estimation obtained through the reduced Euler method seemed cleaner and free of artifacts, which may denote a stronger robustness against noise.

5. Conclusions

The aim of this paper was to present two first-order (forward Euler and reduced Euler) and two second-order (Heun and Runge) numerical learning methods to learn the SVD of large rectangular matrices.

We recalled the Helmke–Moore learning paradigm to estimate the SVD of a given rectangular matrix, which was expressed by two coupled differential equations on the special orthogonal group. Taking the move from classical numerical calculus methods to solve ordinary differential equations, we suggested several numerical schemes to extend these classical methods to the manifold of special orthogonal matrices.

The resulting methods were tested numerically against each other in a pyramidal comparison in order to determine their merits and demerits in terms of computational complexity and effectiveness in computing the SVD. Observing that, for total least-squares-based optical-flow estimation, only one among the three SVD factors was necessary, a further reduced method was developed and tested against its non-reduced counterpart.

The conclusion of the experimental tests was that the estimation obtained through the reduced Euler method looked clean and free of artifacts, which may suggest robustness against noise and that the time saving of the reduced Euler method with respect to the forward Euler method increased noticeably when the size of the sub-windows increased.