Lie Group Methods in Blind Signal Processing

Abstract

This paper deals with the use of Lie group methods to solve optimization problems in blind signal processing (BSP), including Independent Component Analysis (ICA) and Independent Subspace Analysis (ISA). The paper presents the theoretical fundamentals of Lie groups and Lie algebra, the geometry of problems in BSP as well as the basic ideas of optimization techniques based on Lie groups. Optimization algorithms based on the properties of Lie groups are characterized by the fact that during optimization motion, they ensure permanent bonding with a search space. This property is extremely significant in terms of the stability and dynamics of optimization algorithms. The specific geometry of problems such as ICA and ISA along with the search space homogeneity enable the use of optimization techniques based on the properties of the Lie groups $O\left(n\right)$ and $SO\left(n\right)$. An interesting idea is that of optimization motion in one-parameter commutative subalgebras and toral subalgebras that ensure low computational complexity and high-speed algorithms.

1. Introduction

Blind signal processing (BSP) is currently one of the most attractive and fast-growing signal processing areas with solid theoretical foundations and many practical applications. BSP has become a vital research topic in many areas of application, particularly in biomedical engineering, medical imaging, speech and image recognition, communication systems, geophysics, economics, and data analysis. The term “blind processing” originates from the basic feature of these processing methods, i.e., the fact that there is no need to use any training data or a priori knowledge to obtain results. These methods include, among others, Independent Component Analysis (ICA), independent subspace analysis (ISA), sparse component analysis (SCA), nonnegative matrix factorization (NMF), singular value decomposition (SVD), principal component analysis (PCA) and minor component analysis (MCA) as well as the related eigenproblem and invariant subspace problem. Optimization problems of this kind often occur in the context of artificial neural networks, signal processing, pattern recognition, computer vision and numeric [1]. BSP is widely used in biomedical engineering, in technical diagnostics as well as in energy. The work [2] presents the use of SCA to analyze biomedical EEG and fMRI signals proving the effectiveness of this method in the detection of ocular artifacts. The use of SCA in technical diagnostics is presented in [3]. The three-dimensional geometric features-based SCA algorithm was used for compound faults diagnosis of roller bearing. A similar topic was discussed in [4] where NMF was used to extract error signals. The conducted experiments confirmed the effectiveness of these methods in extract the fault features and diagnosis for roller bearing. An interesting use of BSP techniques in energy issues is presented in [5]. Bayesian-optimized bidirectional Long Short-Term Memory (LSTM) method was used for energy disaggregation aiming to identify the individual contribution of appliances in the aggregate electricity load. The use of machine learning techniques as k-means clustering and Support Vector Machine for low-complexity energy disaggregation is presented in [6].

This paper primarily focuses on ICA and ISA problems, which does not, however, limit the applicability of the described methods to other types of problems. The scope of this paper is mainly limited to presenting the geometry of ICA and ISA problems and the application of Lie groups and Lie algebra without providing specific algorithms.

Standard Independent Component Analysis (ICA) consists of the linear transformation of multidimensional data such that the transformed signal components are as much statistically independent as possible. The effectiveness of ICA depends on the correct choice of a cost function and an optimization strategy. Most numerical optimization techniques assume that the model’s parameter space is a usual Euclidean space. In many cases, however, the parameter space has a non-linear structure with its unique non-Euclidean geometry. From a mathematical point of view, the space of search equipped with an inner product takes on the properties of Riemannian manifolds, often with desired algebraic properties [7].

The authors of works in this field take advantage of the specific internal geometry and algebraic properties of models such as the orthogonal group $O\left(n\right)$ or the special orthogonal group $SO\left(n\right)$. Apart from the general group properties, these groups also have the structure of a smooth differential manifold, and thus acquire the Lie group properties and the corresponding Lie algebra. The application of this convenient structure to ICA algorithms is described in [8,9].

From the point of view of standard optimization techniques, in issues of this type one deals with the so-called constrained optimization. The problem of constrained optimization occurs in many issues related to signal processing. In the case of ICA, optimization of this kind consists of looking for extrema of the cost function on the set of matrices satisfying the condition of orthonormal columns ($W^{T}W=I$). However, with standard constrained algorithms one operates in Euclidean space, so in each iterative step the matrix orthogonality is lost. To restore the orthogonality condition, it is necessary to perform orthogonalization in each iterative step (e.g., by the well-known Gram-Schmidt orthogonalization process), which, however, reduces the convergence rate of the algorithms. Other algorithms use the Lagrangian method of optimization by the addition to the cost function of the so-called penalty function to prevent an orthogonality deviation. However, such algorithms are characterized by a low convergence rate and poor quality of the achieved optimum.

If there is a limitation in the form of matrix orthogonality, one can use an alternative method that ensures “locked” with the hyper-surface of orthogonal matrices during optimization motion. This method uses the group structure of a set of orthogonal square matrices which, apart from the properties of a smooth differential manifold, provides the set with the properties of a special structure known as a Lie group.

2. Model Definition (ICA, ISA)

Standard independent components analysis (ICA) consists of estimating a sequence of p statistically independent components (ICs) $s_1,\dots ,s_p$ and the mixing matrix A of dimension $n\times p$ with only a sequence of n observed signals $x_1,\dots ,x_n$. Giving the source signals and observed signals in the form of the source vector $s={\left(s_1,\dots ,s_p\right)}^T$ and the vector of observed (mixed) signals $x={\left(x_1,\dots ,x_n\right)}^T$, where T stands for transposition, the standard linear ICA model takes the form (1):

$$x=As$$

(1)

This assumes that there is no additional noise signal in the observed signal (Figure 1). The ICA model thereby formulated is characterized by a scale and permutation ambiguity, i.e., it is possible to scale (multiply by a given constant) of any source signals $s_i$ and at the same time to divide the i-th column $a_i$ of the mixing matrix $A$ by this constant, while the observed signal $x$ remains unchanged. The same phenomenon will occur at random transposition of any rows of the source vector $s$ (permutation of the source vector $\mathrm{s}$) and the same transposition of the columns of the mixing matrix $A$. It is customary to assume that the source signals have the unit variance ($C_s=\mathrm{E}\left(s{s}^T\right)=I$). In non-negative ICA, it is additionally assumed that the source signals $s_i$ satisfy the condition $s_i\ge 0$ [10,11].

A solution for the ICA problem when $n=p$ consists of finding the demixing (filtration) matrix $Q^T=A^{-1},Q\in Gl\left(n\right)$ where the filtration matrix $Q$ belongs to a general linear group $Gl\left(n\right)$ of non- singular matrices $\det \left(Q\right)\ne 0$. Source signals are obtained via (2):

$$\widehat{s}=Q^{T}x=Q^{T}As$$

(2)

where $\widehat{s}$ is the estimator of a source vector $\mathrm{s}$ (it meets the statistical assumptions for a source signal).

To reduce the computation load in ICA, the pre-processing usually involves performing the whitening of the observed signal to obtain the signal $z=Bx=BAs$, where $B$ is the whitening matrix, with unit variance and the decorrelation $C_z=\mathrm{E}\left(z{z}^T\right)=I$. Assuming that $C_s=I$, we get (3):

$$I=C_z=\mathrm{E}\left(z{z}^T\right)=BA\mathrm{E}\left(s{s}^T\right){\left(BA\right)}^T=BA{\left(BA\right)}^T$$

(3)

From this it follows that ${\left(BA\right)}^T={\left(BA\right)}^{-1}$. Hence, the transformation from $s$ to $z$ takes place via an orthogonal matrix $BA$. Therefore, if $\widehat{s}=W^{T}z=W^{T}BAs=Us$, then the matrix $U=W^{T}BA$ must be an orthogonal matrix (permutation matrix), and thus a new filtering matrix $W$ (after whitening) must also satisfy the orthogonality condition. The whitening of the observed signal therefore simplifies the ICA problem from optimization on the general linear group $Gl\left(n\right)$ (matrices $Q$ only satisfying the invertibility condition $\det \left(Q\right)\ne 0$) to optimization on the special orthogonal group $SO\left(n\right)$ (matrices $W$ satisfying the orthogonality condition $W^{T}W=I$). Both groups are Lie groups at the same time.

Standard ICA is based on the assumption that $n=p$, i.e., the number of source signals $s_i$ is known and equal to the number of observed signals $x_i$. ICA also yields interesting results in a more general case when the number of estimated source signals p is unknown. In this case, it can be $n\ne p$. When $n<p$, i.e., the number of observed signals is smaller than the number of source signals, the problem is known as over complete bases ICA, whereas when $n>p$ it is called under complete bases ICA. This kind of problem can be formally considered to be unconstrained optimization on the Stiefel manifold [12,13]. It is also possible to solve ICA problems for the case $p=1$. This type of problem is often called Single Channel Source Separation [14,15].

Hyvarinen and Hoyer introduced independent subspace analysis (ISA) [16] by omitting the statistical independence condition between extracted source components. The source vector $s$ is composed in $d_k$-tuple $\left(k=1,\dots ,r\right)$, where for a given tuple a statistical dependence between its source signals $s_i$ is allowed, while signals belonging to different tuples are statistically independent. When using the whitening process, the ISA problem boils down to finding orthogonal matrices $W^{T}W=I$ as in standard ICA. However, due to the statistical relationship between the source signals in the tuple, ISA problem optimization cannot be performed on an ordinary Stiefel manifold. It is necessary to introduce a different, more universal manifold allowing for additional symmetries. This manifold is known as a flag manifold.

Traditionally, the ICA model assumes the statistical independence of extracted source signals. It turns out, however, that there are reasons to replace the orthonormality condition with the condition of source signal normality [17]. A precise definition of the ICA problem consists of finding a linear non-orthogonal transformation (of the coordinate system) of multidimensional data such that the transformed data have minimal mutual information. Hyvarinen [18] demonstrated (in an ICA problem) the differences between the use of cost functions based on mutual information and those based on the so-called non-Gaussianity. Achieving maximum de-correlation by maximizing the sum of non-Gaussianity of independent components (ICs) is not necessarily related to the minimization of mutual information (MI). In addition, the orthonormality condition leads to a smaller subset of matrices, which simplifies the optimization process yet may reduce its quality. Orthonormality imposes a greater limitation on the degrees of freedom than normality. In standard ICA, the orthonormality condition of $n\times n$ filtering matrices reduces the number of the degrees of freedom to $\left(n-1\right)/2$, while the normality condition increases the number of free parameters to $n\left(n-1\right)$, which considerably improves the quality of obtained results. A problem of this type can be formally considered to be the unconstrained optimization of an oblique manifold [19,20].

3. Geometry of ICA, ISA and Other BSP Models

The manifolds frequently arise from BSP tasks for a general do not have the group properties. Nevertheless, they are homogenous spaces of the Lie groups. A homogenous space $M$ is a manifold on which the Lie group $G$ acts transitively [21]. This property is fundamental for the considered manifolds because it enables analyzing them as quotient spaces. As mentioned in Section 2, the optimization problem in standard ICA (ISA) boils down to optimizing on the general linear group $Gl\left(n\right)$ (matrices $Q$ only satisfy the invertibility condition $\det \left(Q\right)\ne 0$). The whitening of the observed signal simplifies the ICA problem to optimization on the special orthogonal group $SO\left(n\right)$ (matrices $W$ satisfy the orthogonality condition $W^{T}W=I_n$). In the case of an under complete problem $p<n$, i.e., when the number of extracted ICs is smaller than the number of observed signals, the set of filtering matrices can be treated as an orthogonal Stiefel manifold $St\left(n,p\right)$ defined as the set of orthonormal matrices of dimension $n\times p$ with the form (4):

$$St\left(n,p\right)=\{W=\left(w_1,\dots ,w_p\right)|W\in {\mathbb{R}}^{n\times p},W^{T}W=I_p,rank\left(W\right)=p\}$$

(4)

which can be regarded as the quotient space arising from the orthogonal group.

Lie group $G=O\left(n\right)$ acts transitively on the Stiefel manifold via (5):

$$O\left(n\right)\times St\left(n,p\right)\ni \left(Q,W\right)\to QW\in St\left(n,p\right)$$

(5)

where $Q\in O\left(n\right)$, $W\in St\left(n,p\right)$. It is possible to demonstrate that for two given points $W_1,W_2\in St\left(n,p\right)$ there exists $Q\in O\left(n\right)$ such that $W_2=Q{W}_1$. This means that starting from any point $W_0\in St\left(n,p\right)$ it is possible to reach any point $W\in St\left(n,p\right)$ by the $G$ action. Resorting to group theory terminology, one can say that the entire manifold $St\left(n,p\right)$ is equivalent to the single orbit $G(W_0)$ of a given point $W_0$ where

$$G(W_0)=\left\{W=Q{W}_0|Q\in O\left(n\right),W_0\in St\left(n,p\right)\right\}$$

(6)

The point $W$ on the manifold $St\left(n,p\right)$ can be expressed via a certain point $Q$ on $O\left(n\right)$. The mapping $\pi :Q\to W$ is surjective, i.e., many to one (projective mapping). Redundancy of this mapping is described by so-called the isotropy subgroup $H$ of the point $W_0$. It is a set of matrices that do not change $W_0$

$$H=\left\{N\in O\left(n\right)|N{W}_0=W_0\right\}$$

(7)

The isotropy subgroup $H\in O\left(n\right)$ of the group $O\left(n\right)$ of the point $W_0\in St\left(n,p\right)$ has the form (8):

$$H=\left(W_0,W_0{}_{\perp }\right)\left(\begin{array}{cc}{I}_p& 0\\ 0& O\left(n-p\right)\end{array}\right){\left(W_0,W_0{}_{\perp }\right)}^T$$

(8)

where $W_0{}_{\perp }\in St\left(n,n-p\right)$ is any $n\times \left(n-p\right)$ matrix that satisfies the condition $\left(W_0,W_0{}_{\perp }\right)\in O\left(n\right)$. It is easy to check that the isotropy condition of point $W_0$ is satisified (9):

$$H·W_0=\left(W_0,W_0{}_{\perp }\right)\left(\begin{array}{cc}{I}_p& 0\\ 0& O\left(n-p\right)\end{array}\right){\left(W_0,W_0{}_{\perp }\right)}^T{W}_0=W_0$$

(9)

Choosing $W_0=\left(\begin{array}{c}{I}_p\\ 0_{n-p,p}\end{array}\right)$ the isotropy subgroup $W_0$ is a set $H=\left(\begin{array}{cc}{I}_p& 0\\ 0& O\left(n-p\right)\end{array}\right)$.

In this shot two $n\times n$ orthogonal matrices represent the same point of the Stiefel manifold if their first p columns are identical or equivalently, if they are related by right multiplication of a matrix of the form $\left(\begin{array}{cc}{I}_p& 0\\ 0& O\left(n-p\right)\end{array}\right)$ where $O\left(n-p\right)$ is an orthogonal matrix group of dimension $\left(n-p\right)\times \left(n-p\right)$ [1]. From a mathematical point of view, we say that such representations are in an equivalence relation. All matrices in an equivalence relation form what is called the equivalence class $\left[W\right]$. Thus, the point on the Stiefel manifold is the equivalence class $\left[W\right]$ of $n\times n$ orthogonal matriceswith identical first p columns, while the Stiefel manifold is a quotient space of the form (10):

$$St\left(\mathrm{n},\mathrm{p}\right)\cong O\left(n\right)/O\left(n-p\right)$$

(10)

and specifically as $St\left(n,p\right)\cong O\left(n\right)/H$ where $H=\left(\begin{array}{cc}{I}_p& 0\\ 0& O\left(n-p\right)\end{array}\right)$. However, $H$ is isomorphic to $O\left(n-p\right)$, i.e., $H\cong O\left(n-p\right)$, therefore $St\left(n,p\right)\cong O\left(n\right)/O\left(n-p\right)$.

There are many applications for the problem formulated as the finding of (zero) extreme of a given field defined in a non-Euclidean subspace of dimension $p$ embedded in the Euclidean space ${\mathbb{R}}^n$. This non-Euclidean subspace is known as the Grassmann manifold $Gr\left(n,p;\mathbb{R}\right)$ [22,23]. Grassmann manifolds can be described as an equivalence class of $n\times p$ orthogonal matrices spanning the same p-dimensional subspace $\left[W\right]=\{WO\left(p\right)|W\in St\left(n,p\right)\}$. Therefore, from a theoretical point of view, the Grassmann manifold can be expressed as the quotient space $Gr\left(n,p\right)\cong St\left(n,p\right)/O\left(p\right)$ and given that $St\left(n,p\right)\cong O\left(n\right)/O\left(n-p\right)$, the Grassmann manifold $Gr\left(n,p\right)$ can also be seen as the quotient space $O\left(n\right)/O\left(p\right)\times O\left(n-p\right)$. In this case, the equivalence class $\left[W\right]=\{W\left(\begin{array}{cc}{O}_p& 0\\ 0& O\left(n-p\right)\end{array}\right)|W\in O\left(n\right)\}$ is a set of square n-dimensional orthogonal matrices whose first p columns span the same p-dimensional subspace. Manifolds of this type are used, among others, in invariant subspace analysis, application-driven dimension reduction and subspace tracking [24,25].

When there is a need for a simultaneous (parallel) subspace extraction, as is the case in independent subspace analysis (ISA), one resorts to the concept of generalized flag manifold, which is a manifold consisting of orthogonal subspaces that constitutes a generalization of both Stiefel and Grassmann manifolds [26,27,28]. The generalized flag manifold $Fl\left(\mathrm{n},d_1,\dots ,d_r;\mathbb{R}\right)$ is defined as (11):

$$Fl\left(\mathrm{n},d_1,\dots ,d_r;\mathbb{R}\right)=\{W|W\in {\mathbb{R}}^{n\times p},W^{T}W=I_p,\}$$

(11)

where the orthogonal matrix $W$ takes the form (12):

$$W=\left[W_1,\dots ,W_r\right],W_i=[w_1^i,\dots ,w_{d_i}^i],$$

(12)

where $w_k^i\in {\mathbb{R}}^n,k=1,\dots ,d_r$ for a specified $i=1,\dots ,r$ is a set of orthogonal bases that span subspaces $V_i$. The subspaces $V_i$ are orthogonal relative to each other and satisfy the condition (13):

$$V=V_1\oplus V_2\oplus \dots \oplus V_r\subset {\mathbb{R}}^{n\times p}$$

(13)

Points on the flag manifold are a set of vector spaces $V$ which can be decomposed as (13). If all $d_i\left(1\le i\le r\right)=1$, the manifold $Fl\left(\mathrm{n},d_1,\dots ,d_r;\mathbb{R}\right)$ is reduced to the Stiefel manifold $St\left(n,p\right)$. If $r=1$, it is reduced to the Grassmann manifold $Gr\left(n,p\right)$. It is abbreviated as $Fl\left(n,d\right)$ where $d=(d_1,\dots ,d_r)$. The orthogonal group $O\left(n\right)$ also acts transitively on the manifold $Fl\left(n,d\right)$ via simple matrix multiplication (14):

$$O\left(n\right)\times Fl\left(n,d\right)\ni \left(Q,W\right)\to QW\in Fl\left(n,d\right)$$

(14)

The isotropy subgroup $H\in O\left(n\right)$ of the group $O\left(n\right)$ of the point $W\in Fl\left(n,d\right)$ has the form (15):

$$H=\left(W,W_{\perp }\right)diag\left(R_1,\dots ,R_r,R_{r+1}\right){\left(W,W_{\perp }\right)}^T$$

(15)

where $diag\left(R_1,\dots ,R_r,R_{r+1}\right)$ is a block-diagonal matrix of the form $\left(\begin{array}{cccc}{R}_1& & & 0\\ & \ddots & & ⋮\\ ⋮& & R_r& \\ 0& & & R_{r+1}\end{array}\right)$, $R_k\in O\left(d_k\right),\left(1\le k\le r\right),R_{r+1}\in O\left(n-p\right)$, $W_{\perp }\in St\left(n,n-p\right)$ is any $n\times \left(n-p\right)$ matrix that satisfies the condition $\left[W,W_{\perp }\right]\in O\left(n\right)$. It is easy to check that the isotropy condition of point $W$ is satisified (16):

$$\begin{array}{c}H·W=\left[\mathrm{W}\right]\left(W,W_{\perp }\right)diag\left(R_1,\dots ,R_r,R_{r+1}\right){\left(W,W_{\perp }\right)}^T·W=\left(WR,W_{\perp }{R}_{r+1}\right)\left(W^T{W}_{\perp }^T\right)W\\ =\left(WR{W}^T+W_{\perp }{R}_{r+1}{W}_{\perp }^T\right)W=WR=\left[W\right]\end{array}$$

(16)

where $R=diag\left(R_1,\dots ,R_r\right)$, $WR=\left[W\right]=diag\left(W_i{R}_i\right),i=1,\dots ,r$ is an equivalence class of the point on $Fl\left(n,d\right)$. This means that any two matrices $W_1$ and $W_2$ satisfying the condition $W_2=W_{1}R=\left(W_1,\dots ,W_r\right)diag\left(R_1,\dots ,R_r\right)=diag\left(W_1{R}_1,W_2{R}_2,\dots ,W_r{R}_r\right)$ are identified with the very same point on the manifold $Fl\left(n,d\right)$. Given the above,

$$Fl\left(n,d\right)\cong O\left(n\right)/O\left(d_1\right)\times \dots \times O\left(d_r\right)\times O\left(n-p\right)$$

(17)

As it was already mentioned, the manifold $Fl\left(n,d\right)$ is locally isomorphic to $St\left(n,p\right)$ as a homogenous space when all $d_i\left(1\le i\le r\right)=1$ and to the manifold $Gr\left(n,p\right)$ when $r=1$.

In terms of optimization, the homogeneity of the considered differential manifolds enables the search (optimization motion) in the group $O\left(n\right)$ or $SO\left(n\right)$, and the use of optimization techniques that are well known and adapted to these types of groups. Section 4 presents the basic ideas of optimization methods used in $SO\left(n\right)$ and the concept of toral subalgebra that is characteristic of problems of this type.

4. Lie Group Optimization Methods. One-Parameter Subalgebra and Toral Subalgebra

The idea of a standard optimization procedure based on the Lie groups consists of performing the optimization motion in the Lie algebra space and then mapping exp to find a solution in the Lie group (manifold). The optimization motion in the group $SO\left(n\right)$ starting from the point (matrix) $W_0$ therefore consists of, first, the transition to the Lie algebra $\mathsf{\Omega }=\log W\in \mathfrak{s}\mathfrak{o}\left(n\right)$ via mapping inversely to the exponentiation $\log :={\exp }^{-1}$ of motionin the Lie algebra (performing an operation of addition (of matrices) in the abelian group) in order to obtain a new antisymmetric matrix ${\mathsf{\Omega }}^{\prime }\in \mathfrak{s}\mathfrak{o}\left(n\right)$ and, finally, returning to the Lie group via exponential mapping $W^{\prime }=\exp {\mathsf{\Omega }}^{\prime }\in SO\left(n\right)$. A simple update method using line search procedure relies on finding the search direction in Lie algebra $\mathfrak{s}\mathfrak{o}\left(n\right)$ calculating the gradient of cost function $J$ in Lie algebra space. This gradient must be skew-symmetric (see Appendix A) so (18) [9]:

$${\nabla }_{A}J=({\nabla }_{w}J)W^{\mathrm{T}}-W({\nabla }_{w}J){}^{\mathrm{T}}$$

(18)

Applying the steepest descent procedure with small constant update factor $\mu$ we start from $A=0_n\in \mathfrak{s}\mathfrak{o}\left(n\right)$, move to $B=-\mu {\nabla }_{A}J$, map to $R=\exp \left(B\right)\in SO\left(n\right)$ and finally perform rotating (multiplicative) update $W_{\mathrm{k}+1}=\exp \left(-\mu {\nabla }_{A}J\right)W_k$. This kind of optimization method is called a geodesic flow method [9].

At this point it is necessary to comment on motion in the Lie algebra. In our context, the addition of vectors in the Lie algebra $\mathfrak{s}\mathfrak{o}\left(n\right)$ can only be useful if it is matched by multiplication in the Lie group $SO\left(n\right)$. Then one can write (19):

$$\exp \left(A\right)\exp \left(B\right)=\exp \left(B\right)\exp \left(A\right)=\exp \left(A+B\right)$$

(19)

As was already mentioned, this equation holds true only when the matrices $A$ and $B$ commutate $\left[A,B\right]=0$. This condition is satisfied for all matrices with $\mathfrak{s}\mathfrak{o}\left(2\right)$. When $n\ge 3$, this condition is not satisfied for all matrices in the algebra. When the matrices not commutate (non-abelian Lie algebra), Equation (19) is not satisfied and optimization motion in the Lie algebra (sum A+B) in a direction of e.g., the cost function gradient will not be reflected in the Lie group $\exp \left(A+B\right)\ne \exp \left(A\right)\exp \left(B\right)$. However, taking $\exp \left(A\right)=I_n$, which is tantamount to selecting an initial matrix $A_0=0_n$, this condition will always be satisfied. In this case, $[A_0,B]=0$, and Equation (19) is satisfied too. This is tantamount to motion in the one-parameter Lie algebra. By selecting $A=t\mathsf{\Omega }$ for a random antisymmetric matrix $\mathsf{\Omega }\in \mathfrak{s}\mathfrak{o}\left(n\right)$ and a scalar $\in \mathbb{R}$, all matrices of this form commutate with each other ($A=t_1\mathsf{\Omega },B=t_2\mathsf{\Omega }:\left[A,B\right]=t_1\mathsf{\Omega }{t}_2\mathsf{\Omega }-t_2\mathsf{\Omega }{t}_1\mathsf{\Omega }=0$). A set of such matrices $\mathfrak{s}{\mathfrak{o}}_{\mathsf{\Omega }}\left(n\right)=\{t\mathsf{\Omega }|\mathsf{\Omega }\in \mathfrak{s}\mathfrak{o}\left(n\right),t\in \mathbb{R}\}$ is in itself a Lie algebra known as a one-parameter subalgebra of the Lie algebra $\mathfrak{s}\mathfrak{o}\left(n\right)$. The subalgebra $\mathfrak{s}{\mathfrak{o}}_{\mathsf{\Omega }}\left(n\right)$ is an abelian (commutative) algebra related to the one-parameter subgroup $R\left(t\right)=\exp \left(t\mathsf{\Omega }\right)$. Optimization motion in the subalgebra $\mathfrak{s}{\mathfrak{o}}_{\mathsf{\Omega }}\left(n\right)$ is therefore an equivalent (generalization) to the idea of linear motion in Euclidean space. In this case, the optimization procedure consists of searching for a minimum of the cost function along the subalgebra $\mathfrak{s}{\mathfrak{o}}_{{\mathsf{\Omega }}_1}\left(n\right)$ (for a chosen search direction ${\mathsf{\Omega }}_1$), which corresponds to the search along the subgroup $R\left(t\right)$.

Having found the cost function minimum $\left(R\left(t\right)W_0\right)$, where $W_0$ is a starting point, a new direction of linear searches ${\mathsf{\Omega }}_2$ is selected, and the procedure is repeated until the desired convergence is achieved. Plumblay [8] proposed a modification of the standard procedure described above. This modification consists of moving the point of “origin” of the Lie algebra from a neutral element of the group to point $W$. Due to the group properties $SO\left(n\right)$, it can be written that $W^{\prime }=RW$ for some matrix $R\in SO\left(n\right)$. Moving from the matrix $W=I_{n}W$ to $W^{\prime }=RW$ is therefore equivalent to moving from the identity matrix $I_n$ to the matrix $R$. This procedure consists of moving from the matrix $0_n=\log I_n\in \mathfrak{s}\mathfrak{o}\left(n\right)$ to $\mathsf{\Omega }=\log R\in \mathfrak{s}\mathfrak{o}\left(n\right)$ in the Lie algebra and then returning to the group $SO\left(n\right)$ via the exponential mapping $R=\exp \mathsf{\Omega }$ and, finally, determining $W^{\prime }=RW=\left(\exp \mathsf{\Omega }\right)W\in SO\left(n\right)$. This is equivalent to the concept of optimization motion in the one-parameter abelian subalgebra described above.

The above optimization procedures are computationally expensive due to the necessity of performing (computationally expensive) matrix exponentiation in every iterative step. The representation of antisymmetric matrices in the Jordan canonical form enables the decomposition of optimization movement in the group $SO\left(n\right)$ to commutative rotations in orthogonal planes. Every antisymmetric matrix $\mathsf{\Omega }$ can be presented in a block-diagonal form (for $2m\le n)$ (20):

$$\mathsf{\Omega }=Qdiag\left({\Phi }_1,\dots ,{\Phi }_m,0,\dots ,0\right)Q^t$$

(20)

where $diag\left({\Phi }_1,\dots ,{\Phi }_m,0,\dots ,0\right)$ is a block-diagonal matrix of the form $\left(\begin{array}{cccccc}{\Phi }_1& & \dots & & & 0\\ & \ddots & & & & \\ & & {\Phi }_m& & & ⋮\\ ⋮& & & 0& & \\ & & & & \ddots & \\ 0& & & & & 0\end{array}\right)$, $Q\in SO\left(n\right)$, ${\Phi }_i=\left(\begin{array}{cc}0& {\phi }_i\\ -{\phi }_i& 0\end{array}\right)$ denotes the $2\times 2$ dimensional antisymmetric matrices [29]. This form is known as the Jordan canonical form. Since the relationship $\exp \left(Q^T\mathsf{\Omega }Q\right)=Q^T\exp \left(\mathsf{\Omega }\right)Q$ holds true, the matrix $\mathsf{\Omega }$ can be decomposed into a sum of the form $\mathsf{\Omega }={\mathsf{\Omega }}_1+\dots +{\mathsf{\Omega }}_{\mathrm{m}}$ where ${\mathsf{\Omega }}_{\mathrm{i}}$ is the matrix only containing the i-th Jordan matrix ${\Phi }_i$ and zeros beyond it (21):

$$\mathsf{\Omega }=Qdiag\left({\Phi }_1,0,\dots ,0\right)Q^t+\dots +Qdiag\left(0,\dots ,{\Phi }_m,0,\dots ,0\right)Q^t$$

(21)

The exponentiation of thereby presented matrix $\mathsf{\Omega }$ yields an orthogonal matrix $W$ of the form (22):

$$W=\exp \mathsf{\Omega }=Qdiag\left(R_1,\dots ,R_m,1,\dots ,1\right)Q^t$$

(22)

where $R_i=\left(\begin{array}{cc}cos{\phi }_i& sin{\phi }_i\\ -sin{\phi }_i& cos{\phi }_i\end{array}\right)$ are the $2\times 2$ dimensional rotation matrices. The matrix $W$ can be decomposed into a product of the matrix $W=W_1\dots W_m$ where $W_i$ has the form (23):

$$W_i={\exp \mathsf{\Omega }}_{\mathrm{i}}=Qdiag\left(1,\dots ,1,R_i,1,\dots ,1\right)Q^t$$

(23)

One can notice that the exponentiation of the matrix $\mathsf{\Omega }$ in the Jordan form is reduced to a simple and inexpensive calculation of the functions $sin{\phi }_i$ and $cos{\phi }_i$, which significantly increases the speed of optimization algorithms. The Jordan canonical form of antisymmetric matrix can be obtained via symmetric eigenvalue decomposition [29]. It can be observed that the antisymmetric matrix $\mathsf{\Omega }$ commutates with the symmetric matrix ${\mathsf{\Omega }}^2=-{\mathsf{\Omega }}^{\mathrm{T}}\mathsf{\Omega }$, which means that $\mathsf{\Omega }$ and ${\mathsf{\Omega }}^{\mathrm{T}}\mathsf{\Omega }$ have the same eigenvectors and eigenvalues. The eigenvalues ${\mathsf{\Omega }}^{\mathrm{T}}\mathsf{\Omega }$ occur in pairs corresponding to individual Jordan matrices ${\Phi }_i$.

This form can be visualized as compounding rotations (represented by $W_i$) in mutually orthogonal planes. In addition, the rotation matrices $W_i$ commutate $[W_i,W_j]=0$. The commutation property of the rotation matrix $W_i$ provides the possibility of using the optimization procedure on $SO\left(n\right)$, moving in the Lie algebra $\mathfrak{s}\mathfrak{o}\left(n\right)$.

The case of $SO\left(4\right)$ is interesting from a geometrical point of view. The Jordan canonical form of the antisymmetric matrix $\mathsf{\Omega }$ contains two blocks (matrices) ${\Phi }_i$:

$$\mathsf{\Omega }=Q\left(\begin{array}{cc}\begin{array}{cc}0& {\phi }_1\\ -{\phi }_1& 0\end{array}& \begin{array}{cc}0& 0\\ 0& 0\end{array}\\ \begin{array}{cc}0& 0\\ 0& 0\end{array}& \begin{array}{cc}0& {\phi }_2\\ -{\phi }_2& 0\end{array}\end{array}\right)Q^t$$

(24)

Here, the orthogonal matrix $W$ takes the form (25):

$$W=\exp \mathsf{\Omega }=Q\left(\begin{array}{cc}\begin{array}{cc}cos{\phi }_1& sin{\phi }_1\\ -sin{\phi }_1& cos{\phi }_1\end{array}& \begin{array}{cc}0& 0\\ 0& 0\end{array}\\ \begin{array}{cc}0& 0\\ 0& 0\end{array}& \begin{array}{cc}cos{\phi }_2& sin{\phi }_2\\ -sin{\phi }_2& cos{\phi }_2\end{array}\end{array}\right)Q^t$$

(25)

A visual representation of this case shows rotations in two mutually orthogonal planes, which corresponds to toral geometry (Figure 2).

From the point of view of optimization procedures, the rotation angles ${\phi }_1$ and ${\phi }_2$ should not be free parameters (independent of each other). For the procedure to make sense, the curve over which the search is carried out after a complete rotation (or its multiple) relative to one of the planes of rotation should return to the starting point on the toral surface (Figure 2). This is possible when for some $t$, the relationships $t{\phi }_1=2k_1\pi$ and $t{\phi }_2=2k_2\pi$ are satisfied for the integers $k_1$ and $k_2$. Therefore, the angle of rotation should be described by the relationship $\frac{{\phi }_1}{{\phi }_2}=k_1/k_2$ or ${\phi }_1=a{\phi }_2$, where $a=k_1/k_2$ is a rational number. This concept is naturally transferred to a general case of $SO\left(n\right)$ for $n>4$. The Jordan canonical form represents optimization motion in one-parameter Lie subalgebra $R\left(t\right)=\exp \left(t\mathsf{\Omega }\right)\in \mathfrak{s}{\mathfrak{o}}_{\mathsf{\Omega }}\left(n\right)$ as a rotation in $p$ mutually orthogonal planes, and these rotations are commutative. The geometry of 2-dimensional torus for $SO\left(4\right)$ can also be generalized to the geometry of the $p$-dimensional torus in $SO\left(n\right)$ where $n=2p$ for even $n$ or $n=2p+1$ for odd $n$. This perception of motion in $SO\left(n\right)$ leads to the concept of toral subalgebra $\mathfrak{t}\left(p\right)\subset \mathfrak{s}\mathfrak{o}\left(n\right)$. If we consider a general case of motion on the surface of a $p$-dimensional torus where the angles of rotation ${\phi }_i$ are not interrelated by the above relationship, and individual independent planes of rotation are represented by a set $p$ of commuting matrices ${\mathsf{\Omega }}_{\mathrm{i}}$ ($\mathsf{\Omega }={\displaystyle \sum }_{\mathrm{i}}{\mathsf{\Omega }}_{\mathrm{i}}$). The motion (or rather rotation) on each of the independent planes of rotation can be expressed in the form of a parameterized curve $B_i=t_i{\mathsf{\Omega }}_{\mathrm{i}}$, or actually via its simple exponentiation $W_i=\exp (t_i{\mathsf{\Omega }}_{\mathrm{i}})$. The set of independent parameters $t_i$ that can be identified with the angles of rotation ${\phi }_i$ forms a coordinate system on the toral subalgebra $\mathfrak{t}\left(p\right)$. Compared to the original $n$–dimensional search space $SO\left(n\right)$, the toral subalgebra $\mathfrak{t}\left(p\right)$ is, however, an abelian algebra, which means that motion in this search space is commutative. This ensures the possibility of motion in all directions specified by the coordinates $t_i$ and their sum in the form $B=t_1{\mathsf{\Omega }}_1+\dots +t_p{\mathsf{\Omega }}_{\mathrm{p}}$ will be reflected in the composition of rotations $W=\exp B=\exp \left(t_1{\mathsf{\Omega }}_1\right)·\dots ·\exp \left(t_p{\mathsf{\Omega }}_{\mathrm{p}}\right)$. The optimization procedure based on such a concept consists of decomposing to canonical form a specific antisymmetric matrix $B\in \mathfrak{s}\mathfrak{o}\left(n\right)$ (this can be, for example, a cost function gradient as in the method of steepest descent), and thereby formulating a toral subalgebra. Since the orthogonal matrix $Q$ in the Jordan decomposition (24) is constant, the transition to a new point $W$ in the search space is done by determining $p$ values of the $sin$ and $cos$ functions corresponding to $p$ planes of rotation. After finding in the subalgebra the point that minimizes the cost function, a new antisymmetric matrix $B^{\prime }\in \mathfrak{s}\mathfrak{o}\left(n\right)$ is calculated and again presented in the Jordan canonical form, which establishes a new toral subalgebra. The procedure is repeated until the set minimum cost function is reached. A different problem concerns the determination of the direction of search $B\in \mathfrak{s}\mathfrak{o}\left(n\right)$ and the manner of search along the subalgebra. The selection of directions and the manner of searches depend on the adopted optimization procedure. It can be the steepest descent (SD) method and, in general, geodesic flow, Newton’s method or conjugate gradients. This problem has been extensively studied in [12,22,30].

5. Experimental Results

To illustrate the presented optimization methods on Lie groups, we will first present a rather simply simulation experiment. The purpose of this example is to show how different algorithms work on optimization problems with unitarity constraint. To this end, let us consider the Lie group of complex numbers with the unit module $U\left(1\right)$, which is isomorphic with the $SO\left(2\right)$ group. The unitarity constraint of elements of this group is a unit circle on the complex plane. The cost function we will minimize will be $J\left(z\right)={\left|z+0.3\right|}^2$ with constraint $z{z}^{*}=1$. For optimization, we will use five types of steepest descent (SD) algorithms:

(1)

algorithm SD unconstrained on the Euclidean space,

(2)

algorithm SD on the Euclidean space with constraint restoration,

(3)

algorithm SD on the Euclidean space with penalty function,

(4)

non-geodesic algorithm SD on Riemannian space,

(5)

geodesic algorithm SD on Riemannian space.

In Algorithm (1) update rule has the form $z_{k+1}=z_k-\mu \left(z_k+0.3\right)$ where $\mu$ is the step size. The quantity $\frac{\partial J}{\partial z^{*}}=\left(z_k+0.3\right)$ is the gradient (The gradient of the function defined in the complex space has the form [31]:$\frac{\partial J}{\partial z^{*}}=\frac{1}{2}\left(\frac{\partial J}{\partial \mathrm{Re}\left(z\right)}+\frac{\partial J}{\partial \mathrm{Im}\left(z\right)}\right)$ where $\mathrm{Re}\left(z\right)$ and $\mathrm{Im}\left(z\right)$ is respectively the real and imaginary part of a complex number $z$.) of the cost function $J$ on the Euclidean space. Algorithm (2) uses the same update rule, but after each iteration the unitarity condition is restored in the form $z_{k+1}=\frac{z_{k+1}}{\left|z_{k+1}\right|}$. In Algorithm (3) we used the Lagrange multiplier method. The penalty function of the form ${({\left|z_k\right|}^2-1)}^2$ weighted by a Lagrange parameter $\lambda$ has been added to the initial cost function in order to penalize deviations from unitarity. In this case, update rule is $z_{k+1}=z_k-\mu [(z_k+0.3)+\lambda z_k{({\left|z_k\right|}^2-1)}^2]$. In the case of (4), the algorithm works in the Riemannian space (unit circle) determined by the condition $z{z}^{*}=1$. At each point of $z_k$ the algorithm determines the search direction tangent to the unit circle and after each iteration the obtained point is projected back to the unit circle. In this case, update rule has the form $z_{k+1}=\pi \left(z_k-\mu \left(\frac{\partial J}{\partial z^{*}}-\langle z,\frac{\partial J}{\partial z^{*}}\rangle z\right)\right)=\pi (z_k-\mu [z_k(1-{\left|z_k\right|}^2)-0.3(z_k^2-1)])$ where $\pi$ is the projection operator on the unit circle. In Algorithm (5) we used a multiplicative optimization algorithm on the Lie group described in Section 4. In this case, the update rule has the form $z_{k+1}=\exp \left(0.6\mu i\mathrm{Im}(z_k\right))z_k$ where $\mathrm{Im}(z_k)$ is imaginary part of $z_k$. The starting point of each algorithm is $z_0=\exp \left(i\pi /4\right)$. Figure 3 shows the results of the simulation.

In the point $z_{\left(1\right)}=-0.3$ the cost function reaches its minimum $J\left(z_{\left(1\right)}\right)=0$. However, this is the undesirable minimum determined in the Euclidean space by Algorithm (1) not taking into account the unitarity constraint (Figure 3). The minimum considering this constraint is at the point $z_{min}=-1$ and the value of the cost function reaches its minimum over the Riemannian space, i.e., on the unit circle $J\left(z_{min}\right)=0.49$. Unconstrained SD Algorithm (1) and with penalty function (3) achieve undesirable minimums in points respectively from $z_{\left(1\right)}=-0.3$ and from $z_{\left(3\right)}=-0.5$ while Algorithms (2) (4) and (5) minimum appropriate $z_{min}=-1$. In the case of Algorithms (2) and (4), a characteristic “zig-zag” is associated with lowering the constraint surface and undesirable from the point of view of optimizing properties. Algorithm (4) determines the SD direction tangent to the constraint condition, thus leaving the unit circle in the optimization motion. The resulting point is again projected into a unit circle. Algorithm (5) using the multiplicative update rule on the Lie group (phase rotation) described in this article naturally ensures the condition of unitarity at each step. The optimization movement takes place at each step along the geodesic line. This simple one-dimensional example is only intended to present the idea of algorithms on Lie groups. The following is an example of using these methods on a real signal. As an example of the practical application of optimization methods on Lie groups, we will present a solution to the ICA problem. As the source signals, three speech recordings and a quasi-noisy signal (harmonic signal with high noise content) (Figure 4) with a length of 5000 samples (1.25 s) were used. The source signals were mixed using a four-by-four random mixing matrix. The four observed signals are shown in Figure 4. A $SO\left(4\right)$ group optimization algorithm was used to implement ICA.

For comparison, the INFOMAX algorithm in its original form was also used [32]. Based on visual inspection and listening to the separated components, it can be concluded that ICA results using the INFOMAX algorithm and optimization on the $SO\left(4\right)$ group are good with scale and permutation accuracy. The INFOMAX algorithm with the assumed convergence criterion converges after about 30–40 steps, while the algorithm on the $SO\left(4\right)$ group after about 20 steps. Figure 5 shows the sum of entropy values of separated components depending on the iteration number.

The optimization algorithm on the $SO\left(4\right)$ group converges to $E\left(Y\right)=0.11$ (The entropy value was determined according to an approximate relationship [32]: $E\left(Y\right)=-{\displaystyle \sum }_{i}E\{{\displaystyle \sum }_n\tan \mathrm{h}(y_i)\}+\log \left(\det \left(W\right)\right))$ while the INFOMAX algorithm to $E\left(Y\right)=0.084$. Listening to the results and comparison with the sources confirms the better ICA separation results obtained by the $SO\left(4\right)$ group optimization algorithm.

6. Conclusions

This paper described the application of the Lie group methods for blind signal processing, including ICA and ISA. Theoretical fundamentals of the Lie groups and the Lie algebra as well as the geometry of problems occurring in BSP and basic optimization techniques based on the use of Lie groups are presented. Owing to the specific geometry and algebraic properties of BSP problems, it is possible to use Lie group methods to solve these problems. The homogeneity of search space (parameters) in BSP problems enables the use of optimization techniques based on the Lie group methods for the groups $O\left(n\right)$ and $SO\left(n\right)$. It has been demonstrated that the one-parameter subalgebra $\mathfrak{s}{\mathfrak{o}}_{\mathsf{\Omega }}\left(n\right)$ ensures the convenient property of commutating search directions. In addition, the presentation of an antisymmetric matrix (search direction) in the Jordan canonical form establishes the toral subalgebra $\mathfrak{t}\left(p\right)\subset \mathfrak{s}\mathfrak{o}\left(n\right)$, which—in terms of optimization algorithms—ensures low computational complexity and high process dynamics.