Approximation of Time-Frequency Shift Equivariant Maps by Neural Networks

Abstract

Based on finite-dimensional time-frequency analysis, we study the properties of time-frequency shift equivariant maps that are generally nonlinear. We first establish a one-to-one correspondence between $\mathsf{\Lambda }$-equivariant maps and certain phase-homogeneous functions and also provide a reconstruction formula that expresses $\mathsf{\Lambda }$-equivariant maps in terms of these phase-homogeneous functions, leading to a deeper understanding of the class of $\mathsf{\Lambda }$-equivariant maps. Next, we consider the approximation of $\mathsf{\Lambda }$-equivariant maps by neural networks. In the case where $\mathsf{\Lambda }$ is a cyclic subgroup of order N in ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, we prove that every $\mathsf{\Lambda }$-equivariant map can be approximated by a shallow neural network whose affine linear maps are simply linear combinations of time-frequency shifts by $\mathsf{\Lambda }$. This aligns well with the proven suitability of convolutional neural networks (CNNs) in tasks requiring translation equivariance, particularly in image and signal processing applications.

1. Introduction

Over the past decade, machine learning techniques based on deep neural networks, commonly referred to as deep learning [1], have achieved significant breakthroughs across a wide range of fields, including image recognition [2,3], speech recognition [4], language translation [5,6], and game playing [7], among others. These advancements are largely driven by the availability of increasingly large training datasets and greater computational resources. Another important factor is the development of specialized neural network architectures, including convolutional neural networks [2], residual networks [3], recurrent networks (notably LSTMs [5]), and transformer networks [6].

A common theme in the design of neural network architectures is the necessity to respect the symmetries inherent in the task at hand. For instance, in image classification, the classification result should remain invariant under small translations of the input image, making convolutional neural networks a suitable choice. Likewise, in audio classification [8], the classification result should be invariant to shifts in time or changes in pitch. In principle, a fully connected neural network can learn to respect such symmetries provided that training data are sufficiently given. Nevertheless, architectures that are inherently aligned with these symmetries tend to exhibit improved generalization and thus show better performance.

In mathematical terms, symmetries can be expressed as follows. Let V be a vector space and let $GL\left(V\right)$ be the general linear group of V. For a group G and a map $\rho :G\to GL\left(V\right)$, we say that a map $F:V\to V$ is equivariant under group actions of G (or simply G-equivariant) if $F\rho \left(\lambda \right)=\rho \left(\lambda \right)F$ for all $\lambda \in G$, and invariant under group actions of G (or simply G-invariant) if $F\rho \left(\lambda \right)=F$ for all $\lambda \in G$. We will be focusing on the case where V is a Hilbert space and $\rho \left(\lambda \right)$ is a unitary operator for all $\lambda \in G$. (A Hilbert space is a vector space equipped with an inner product that induces a distance function, making it a complete metric space. Examples of Hilbert spaces include ${\mathbb{R}}^d$ and ${\mathbb{C}}^d$, and Hilbert spaces are often regarded as natural generalizations of signal spaces.)

A particularly important and well-studied example of equivariance involves translations. It is well known that translation-equivariant linear operators are exactly the convolution operators (see, e.g., Section 2.3 of [9], Theorem 4.12 of [10], and Theorem 2.17 of [11]), and that convolutional neural networks (CNNs) are well-suited for approximating these operators. As a natural generalization of CNNs, Cohen and Welling [12] introduced the so-called group equivariant convolutional neural networks (GCNNs), which can handle more general symmetry groups than just translations. Later, Cohen et al. [13] developed a general framework for GCNNs on homogeneous spaces such as ${\mathbb{R}}^d$ and $S^2$, and Yarotsky [14] investigated the approximation of equivariant operators using equivariant neural networks. More recently, Cahill et al. [15] introduced the so-called group-invariant max filters, which are particularly useful for classification tasks involving symmetries, and Balan and Tsoukanis [16,17] constructed stable embeddings on quotient space modulo group action, yielding group-invariant representations via coorbits. Further advances include the work of Huang et al. [18], who designed approximately group-equivariant graph neural networks by focusing on active symmetries, and Blum-Smith and Villar [19], who introduced a method for parameterizing invariant and equivariant functions based on invariant theory. In addition, Wang et al. [20] provided a theoretical analysis of data augmentation and equivariant neural networks applied to non-stationary dynamics forecasting.

In this paper, we are particularly interested in the setting of finite-dimensional time-frequency analysis, which provides a versatile framework for a wide range of signal processing applications, see, e.g., [21,22]. It is known that every linear map from ${\mathbb{C}}^N$ to ${\mathbb{C}}^N$ can be expressed as a linear combination of compositions of translations and modulations (see (3) below). We consider maps $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ that are generally nonlinear and are Λ-equivariant for a given subgroup $\mathsf{\Lambda }$ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, that is, $F\circ \pi (k,\ell )=\pi (k,\ell )\circ F$ for all $(k,\ell )\in \mathsf{\Lambda }$. Here, $\pi (k,\ell ):=M^{\ell }{T}^k$ represents the time-frequency shift by $(k,\ell )$, where $T,M:{\mathbb{C}}^N\to {\mathbb{C}}^N$ are the translation and modulation operators defined as $Tx=(x_{N-1},x_0,x_1,\dots ,x_{N-2})$ and $Mx=({\omega }^0{x}_0,{\omega }^1{x}_1,\cdots ,{\omega }^{N-1}{x}_{N-1})$, $\omega :=e^{2\pi i/N}$, for $x=(x_0,x_1,\cdots ,x_{N-1})\in {\mathbb{C}}^N$, respectively (see Section 2.1 for further details). For any $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ and any nonzero $v\in {\mathbb{C}}^N$, we define $F_v:{\mathbb{C}}^N\to \mathbb{C}$ by $F_v\left(x\right)=〈F\left(x\right),v〉$, $x\in {\mathbb{C}}^N$. For any $\mathsf{\Omega }\subset {\mathbb{Z}}_N$, we say that a function $H:{\mathbb{C}}^N\to \mathbb{C}$ is Ω-phase homogeneous if $H\left(e^{2\pi is/N}x\right)=e^{2\pi is/N}H\left(x\right)$ for all $s\in \mathsf{\Omega }$ and $x\in {\mathbb{C}}^N$.

We first address the properties of the mapping $F\mapsto F_v$ from the space of $\mathsf{\Lambda }$-equivariant functions ${\mathbb{C}}^N\to {\mathbb{C}}^N$ to the space of certain phase homogeneous functions.

Theorem 1

(see Theorem 3 below). Assume that $\mathrm{span}\{\pi (k,\ell )v:(k,\ell )\in \mathsf{\Lambda }\}={\mathbb{C}}^N$ for some subgroup Λ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ and some vector $v\in {\mathbb{C}}^N$. Then, the mapping $F\mapsto F_v$ is an injective map from the space of Λ-equivariant functions ${\mathbb{C}}^N\to {\mathbb{C}}^N$ to the space of ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous functions ${\mathbb{C}}^N\to \mathbb{C}$, where ${\mathsf{\Omega }}_{\mathsf{\Lambda }}:=\{k{\ell }^{\prime }modN:(k,\ell ),(k^{\prime },{\ell }^{\prime })\in \mathsf{\Lambda }\}$. Moreover, if ${\left\{\pi (k,\ell )u\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ is a dual frame of ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ in ${\mathbb{C}}^N$, then a Λ-equivariant function $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ can be expressed as

$$F\left(x\right)=\sum _{(k,\ell )\in \mathsf{\Lambda }}{e}^{-2\pi ik\ell /N}{F}_v\left(\pi (-k,-\ell )x\right)\pi (k,\ell )u.$$

If $\left|\mathsf{\Lambda }\right|=N$, then the mapping $F\mapsto F_v$ is a bijective map from the space of Λ-equivariant functions ${\mathbb{C}}^N\to {\mathbb{C}}^N$ to the space of ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous functions ${\mathbb{C}}^N\to \mathbb{C}$.

We then consider the approximation of $\mathsf{\Lambda }$-equivariant maps. In particular, we show that if $\mathsf{\Lambda }$ is a cyclic subgroup of order N in ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, then every $\mathsf{\Lambda }$-equivariant map can be easily approximated by a shallow neural network whose affine linear maps consist of linear combinations of time-frequency shifts by $\mathsf{\Lambda }$.

Theorem 2

(see Theorem 5 below). Assume that $\sigma :\mathbb{C}\to \mathbb{C}$ is shallow universal and satisfies $\sigma \left(e^{\pi i/N}z\right)=e^{\pi i/N}\sigma \left(z\right)$ for all $z\in \mathbb{C}$. Let $\mathsf{\Lambda }=\left\{\right(0,0),(1,s),\dots ,(N-1,(N-1)s\left)\right\}$ for some $s\in \{0,1,\dots ,N-1\}$. Then, any continuous Λ-equivariant map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ can be approximated (uniformly on compact sets) by a shallow neural network

$$x\mapsto \sum _{j=1}^J{c}_j\sigma (A_{j}x+b_{j}v),$$

where $A_j\in \mathrm{span}\{\pi (k,\ell ):(k,\ell )\in \mathsf{\Lambda }\}$, $b_j\in \mathbb{C}$ for $j=1,\dots ,J$, and $v\in {\mathbb{C}}^N$ satisfies $\pi (k,\ell )v=e^{k\ell \pi i/N}v$ for all $(k,\ell )\in \mathsf{\Lambda }$. Moreover, every map of this form is Λ-equivariant.

In the case $s=0$, i.e., $\mathsf{\Lambda }=\left\{\right(0,0),(1,0),\dots ,(N-1,0\left)\right\}$, the $\mathsf{\Lambda }$-equivariant maps $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ are precisely those that are translation equivariant, meaning that $FT=TF$. Furthermore, if F is linear, then F is just a convolutional map, which can be expressed as a linear combination of $T^k$, $k=0,\dots ,N-1$, or simply as an $N\times N$ circulant matrix. If F is nonlinear, then Theorem 2 shows that F can be approximated by a shallow neural network whose affine linear maps are convolutional maps, i.e., by a shallow convolutional neural network. This agrees with the well-established fact that convolutional neural networks (CNNs) are particularly well-suited for applications involving translation equivariance, especially in image and signal processing.

Organization of the Paper

In Section 2, we begin by reviewing some basic properties of time-frequency shift operators, followed by a discussion on time-frequency group equivariant maps, and then prove our first main result, Theorem 1, which establishes a 1:1 correspondence between $\mathsf{\Lambda }$-equivariant maps and certain phase-homogeneous functions. Section 3 is devoted to the approximation of $\mathsf{\Lambda }$-equivariant maps. We first discuss the embedding of $\mathsf{\Lambda }$ into the Weyl–Heisenberg group, which allows for the use of tools from group representation theory. (The finite Weyl–Heisenberg group ${\mathbb{H}}_N$ is the set ${\mathbb{Z}}_N\times {\mathbb{Z}}_N\times {\mathbb{Z}}_N$ equipped with group operation $(k,\ell ,s)+(k^{\prime },{\ell }^{\prime },s^{\prime }):=(k+k^{\prime },\ell +{\ell }^{\prime },s+s^{\prime }-k{\ell }^{\prime })$. The noncommutativity of ${\mathbb{H}}_N$ plays an important role in finite-dimensional time-frequency analysis; see, e.g., [21,23].) After reviewing key concepts from group representation theory, we consider the case of cyclic subgroups of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, where group representations can be defined directly without embedding into the Weyl–Heisenberg group. Section 3 concludes with the proof of our second main result, Theorem 2, which establishes the approximation of $\mathsf{\Lambda }$-equivariant maps by a shallow neural network whose affine linear maps consist of linear combinations of time-frequency shifts by $\mathsf{\Lambda }$.

2. Time-Frequency Shift Equivariant Maps

2.1. Time-Frequency Shift Operators

We define the translation (time shift) operator $T:{\mathbb{C}}^N\to {\mathbb{C}}^N$ by

$$Tx=(x_{N-1},x_0,x_1,\dots ,x_{N-2}),x=(x_0,x_1,\cdots ,x_{N-1})\in {\mathbb{C}}^N,$$

and the modulation (frequency shift) operator $M:{\mathbb{C}}^N\to {\mathbb{C}}^N$ by

$$Mx=({\omega }^0{x}_0,{\omega }^1{x}_1,\cdots ,{\omega }^{N-1}{x}_{N-1})\mathrm{with}\omega :=e^{2\pi i/N}.$$

These operators are linear unitary operators, which can be represented by $N\times N$ unitary matrices:

$$\begin{array}{c}\hfill T=\left[\begin{array}{cccccc}0& 0& 0& \cdots & 0& 1\\ 1& 0& 0& \cdots & 0& 0\\ 0& 1& 0& \cdots & 0& 0\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & 0& 0\\ 0& 0& 0& \cdots & 1& 0\end{array}\right],M=\left[\begin{array}{cccccc}1& 0& 0& \cdots & 0& 0\\ 0& \omega & 0& \cdots & 0& 0\\ 0& 0& {\omega }^2& \cdots & 0& 0\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & {\omega }^{N-2}& 0\\ 0& 0& 0& \cdots & 0& {\omega }^{N-1}\end{array}\right].\end{array}$$

Note that since $T^N=M^N=I_N$, we have $T^{N+k}=T^k$ and $M^{N+\ell }=M^{\ell }$ for all integers k and ℓ. The time-frequency shift by $(k,\ell )\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N$ is defined by $\pi (k,\ell ):=M^{\ell }{T}^k$. Since $T,M:{\mathbb{C}}^N\to {\mathbb{C}}^N$ are linear unitary operators, the operator $\pi (k,\ell )$ is also linear and unitary.

For a Hilbert space $\mathcal{H}$, we will denote the class of all linear operators on $\mathcal{H}$ by $\mathcal{L}\left(\mathcal{H}\right)$, and the class of all linear unitary operators on $\mathcal{H}$ by $\mathcal{U}\left(\mathcal{H}\right)$.

Proposition 1.

For any $k,\ell =0,\dots ,N-1$, we have

$$M^{\ell }{T}^k={\omega }^{k\ell }{T}^k{M}^{\ell }.$$

(1)

This implies $\left(M^{\ell }{T}^k\right)\left(M^q{T}^p\right)={\omega }^{-kq}{M}^{\ell +q}{T}^{k+p}={\omega }^{\ell p-kq}\left(M^q{T}^p\right)\left(M^{\ell }{T}^k\right)$ for $k,\ell ,p,q=0,\dots ,N-1$, and consequently, the operators $M^{\ell }{T}^k$ and $M^q{T}^p$ commute if and only if $\left[\right(k,\ell ),(p,q\left)\right]:=\ell p-kq$ is a multiple of N. Moreover, for any $k,\ell =0,\dots ,N-1$, we have

$${\left(M^{\ell }{T}^k\right)}^{-1}={\omega }^{-k\ell }{M}^{-\ell }{T}^{-k},$$

(2)

that is, $\pi {(k,\ell )}^{-1}={\omega }^{-k\ell }\pi (-k,-\ell )$.

Proof. 

The relation (1) is easily seen by computation. Using (1), we obtain

$${\left(M^{\ell }{T}^k\right)}^{-1}=T^{-k}{M}^{-\ell }={\omega }^{-k\ell }{M}^{-\ell }{T}^{-k},$$

which is exactly (2). □

Remark 1.

The definition of $[·,·]$ remains unchanged for time-frequency shift operators of the form $T^k{M}^{\ell }$ with $(k,\ell )\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N$. Indeed, (1) implies $\left(T^k{M}^{\ell }\right)\left(T^p{M}^q\right)={\omega }^{\ell p-kq}\left(T^p{M}^q\right)\left(T^k{M}^{\ell }\right)$ for $k,\ell ,p,q=0,\dots ,N-1$, and consequently, the operators $T^k{M}^{\ell }$ and $T^p{M}^q$ commute if and only if $\left[\right(k,\ell ),(p,q\left)\right]=\ell p-kq$ is a multiple of N.

For a subgroup $\mathsf{\Lambda }$ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, its adjoint group is defined by

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}^{\circ }:=& \{(p,q)\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N:\left(M^{\ell }{T}^k\right)\left(M^q{T}^p\right)=\left(M^q{T}^p\right)\left(M^{\ell }{T}^k\right)forall(k,\ell )\in \mathsf{\Lambda }\}\hfill \\ \hfill =& \{(p,q)\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N:\ell p-kq\in N\mathbb{Z}forall(k,\ell )\in \mathsf{\Lambda }\}.\hfill \end{array}$$

Since $\{M^{\ell }{T}^k:k,\ell =0,\dots ,L-1\}$ forms a basis for $\mathcal{L}\left({\mathbb{C}}^N\right)$ (see, e.g., Lemma 1 of [24]), every linear operator $F\in \mathcal{L}\left({\mathbb{C}}^N\right)$ can be expressed as

$$F=\sum _{k,\ell =0}^{L-1}{a}_{k,\ell }{M}^{\ell }{T}^k\mathrm{for}\mathrm{some}{a}_{k,\ell }\in \mathbb{C},k,\ell =0,\dots ,L-1.$$

(3)

If F commutes with $M^{\ell }{T}^k$ for $(k,\ell )\in \mathsf{\Lambda }$, then we must have $a_{k,\ell }=0$ for $(k,\ell )\notin {\mathsf{\Lambda }}^{\circ }$, so that

$$F=\sum _{(k,\ell )\in {\mathsf{\Lambda }}^{\circ }}{a}_{k,\ell }{M}^{\ell }{T}^k.$$

Therefore, the commutant (or centralizer) of $(\pi ,\mathsf{\Lambda })$ (see, e.g., Proposition 4.14 of [25]) is given by

$$\begin{array}{cc}\hfill C(\pi ,\mathsf{\Lambda }):=& \{F\in \mathcal{L}\left({\mathbb{C}}^N\right):F\pi (k,\ell )=\pi (k,\ell )Fforall(k,\ell )\in \mathsf{\Lambda }\}\hfill \\ \hfill =& \mathrm{span}\{\pi (k,\ell ):(k,\ell )\in {\mathsf{\Lambda }}^{\circ }\}.\hfill \end{array}$$

(4)

Remark 2.

For a subgroup Λ of ${\mathbb{Z}}_L\times {\mathbb{Z}}_L$, its adjoint group ${\mathsf{\Lambda }}^{\circ }$ has cardinality $L^2/\left|\mathsf{\Lambda }\right|$. While this fact is somewhat considered folklore, we could not find a suitable reference in the literature, so we provide a short proof of this fact in Appendix A.

2.2. $\mathsf{\Lambda }$-Equivariant Maps

Definition 1.

For any $\mathsf{\Lambda }\subset {\mathbb{Z}}_N\times {\mathbb{Z}}_N$, we say that a map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ is $\mathsf{\Lambda }$-equivariant if

$$F\circ \pi (k,\ell )=\pi (k,\ell )\circ Fforall(k,\ell )\in \mathsf{\Lambda }.$$

(5)

Clearly, the set of $\mathsf{\Lambda }$-equivariant linear maps is precisely $C(\pi ,\mathsf{\Lambda })$, the commutant of $(\pi ,\mathsf{\Lambda })$. According to (4), every $(\pi ,\mathsf{\Lambda })$-equivariant linear map is of the form

$$\sum _{(k,\ell )\in {\mathsf{\Lambda }}^{\circ }}{a}_{k,\ell }{M}^{\ell }{T}^k\mathrm{for}\mathrm{some}{\left\{a_{k,\ell }\right\}}_{(k,\ell )\in {\mathsf{\Lambda }}^{\circ }}\in {\mathbb{C}}^{{\mathsf{\Lambda }}^{\circ }}.$$

Since the case of linear maps is obvious, our consideration of $\mathsf{\Lambda }$-equivariant maps will be focused on nonlinear maps.

We first observe some necessary conditions for $\mathsf{\Lambda }$-equivariance.

Proposition 2.

Let Λ be a subgroup of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, and assume that $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ is Λ-equivariant. If Λ contains $(k,\ell )$ and $(k^{\prime },{\ell }^{\prime })$ with $s=gcd(k{\ell }^{\prime },N)$, then

$$F\left(e^{2\pi is/N}x\right)=e^{2\pi is/N}F\left(x\right)forallx\in {\mathbb{C}}^N.$$

(6)

Proof. 

Note that since $\mathsf{\Lambda }$ is a subgroup, we have $(k+k^{\prime },\ell +{\ell }^{\prime })\in \mathsf{\Lambda }$. Using Proposition 1 and (5), we have

$$\begin{array}{cc}\hfill {\omega }^{-k{\ell }^{\prime }}\pi (k+k^{\prime },\ell +{\ell }^{\prime })F\left(x\right)& =\pi (k,\ell )\pi (k^{\prime },{\ell }^{\prime })F\left(x\right)=F\left(\pi (k,\ell )\pi (k^{\prime },{\ell }^{\prime })x\right)\hfill \\ \hfill & =F\left({\omega }^{-k{\ell }^{\prime }}\pi (k+k^{\prime },\ell +{\ell }^{\prime })x\right)\hfill \\ \hfill & =\pi (k+k^{\prime },\ell +{\ell }^{\prime })F\left({\omega }^{-k^{\prime }\ell }x\right),\hfill \end{array}$$

so that ${\omega }^{-k{\ell }^{\prime }}F\left(x\right)=F\left({\omega }^{-k{\ell }^{\prime }}x\right)$. Since $gcd(k{\ell }^{\prime },N)=s$, there exist some $p,q\in \mathbb{Z}$ with $p(-k^{\prime }\ell )+qN=s$. In fact, we can choose $p\in \{0,\dots ,N-1\}$ such that $p(-k^{\prime }\ell )\equiv smodN$. Then, for any $x\in {\mathbb{C}}^N$, we have

$${\omega }^{s}F\left(x\right)={\left({\omega }^{-k{\ell }^{\prime }}\right)}^{p}F\left(x\right)={\left({\omega }^{-k{\ell }^{\prime }}\right)}^{p-1}F\left({\omega }^{-k{\ell }^{\prime }}x\right)=\dots =F\left({\left({\omega }^{-k{\ell }^{\prime }}\right)}^{p}x\right)=F\left({\omega }^{s}x\right),$$

which is equivalent to (6). □

It is easily seen that for a subgroup $\mathsf{\Lambda }$ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, the set

$${\mathsf{\Omega }}_{\mathsf{\Lambda }}:=\{k{\ell }^{\prime }modN:(k,\ell ),(k^{\prime },{\ell }^{\prime })\in \mathsf{\Lambda }\}$$

forms a subgroup of ${\mathbb{Z}}_N$; in fact, ${\mathsf{\Omega }}_{\mathsf{\Lambda }}=s_0\mathbb{Z}/N\mathbb{Z}$, where $s_0:=min\{gcd(k{\ell }^{\prime },N):(k,\ell ),(k^{\prime },{\ell }^{\prime })\in \mathsf{\Lambda }\}$. This leads to the following definition.

Definition 2.

Let $m,n\in \mathbb{N}$ and $N\in \mathbb{N}$. For any $\mathsf{\Omega }\subset {\mathbb{Z}}_N$, we say that a map $F:{\mathbb{C}}^n\to {\mathbb{C}}^m$ is $\mathsf{\Omega }$-phase homogeneous if

$$F\left(e^{2\pi is/N}x\right)=e^{2\pi is/N}F\left(x\right)foralls\in \mathsf{\Omega },x\in {\mathbb{C}}^n.$$

Definition 3.

For any $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ and any nonzero $v\in {\mathbb{C}}^N$, we define $F_v:{\mathbb{C}}^N\to \mathbb{C}$ by

$$F_v\left(x\right)=〈F\left(x\right),v〉,x\in {\mathbb{C}}^N.$$

We now present our first main theorem, which addresses the properties of the mapping $F\mapsto F_v$. Note that if $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ is $\mathsf{\Lambda }$-equivariant for a subgroup $\mathsf{\Lambda }$ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, then it is ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous by Proposition 2, and so is $F_v$.

Before stating the theorem, we note that $\{\pi (k,\ell )v:(k,\ell )\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N\}$ is a tight frame for ${\mathbb{C}}^N$ whenever $v\ne 0$ (see, e.g., Proposition 2 of [24]). Moreover, there exists a nonzero vector $v\in {\mathbb{C}}^N$ such that every N elements of $\{\pi (k,\ell )v:(k,\ell )\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N\}$ are linearly independent in ${\mathbb{C}}^N$. In fact, such vectors form a dense open set $W_N$ of full measure in ${\mathbb{C}}^N$ (see Theorem 1 of [24]). If $\mathsf{\Lambda }\subset {\mathbb{Z}}_N\times {\mathbb{Z}}_N$ is a set of cardinality at least N, then for any $v\in W_N$ we have $\mathrm{span}\{\pi (k,\ell )v:(k,\ell )\in \mathsf{\Lambda }\}={\mathbb{C}}^N$, in which case ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ forms a frame of ${\mathbb{C}}^N$.

Theorem 3.

Assume that $\mathrm{span}\{\pi (k,\ell )v:(k,\ell )\in \mathsf{\Lambda }\}={\mathbb{C}}^N$ for some subgroup Λ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ and some vector $v\in {\mathbb{C}}^N$. Then, the mapping $F\mapsto F_v$ is an injective map from the space of Λ-equivariant functions ${\mathbb{C}}^N\to {\mathbb{C}}^N$ to the space of ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous functions ${\mathbb{C}}^N\to \mathbb{C}$. Moreover, if ${\left\{\pi (k,\ell )u\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ is a dual frame of ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ in ${\mathbb{C}}^N$, then a Λ-equivariant function $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ can be expressed as

$$F\left(x\right)=\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }{F}_v\left(\pi (-k,-\ell )x\right)\pi (k,\ell )u.$$

(7)

If $\left|\mathsf{\Lambda }\right|=N$, then the mapping $F\mapsto F_v$ is a bijective map from the space of Λ-equivariant functions ${\mathbb{C}}^N\to {\mathbb{C}}^N$ to the space of ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous functions ${\mathbb{C}}^N\to \mathbb{C}$.

Proof. 

To prove the injectivity of $F\mapsto F_v$, suppose that $F_v=H_v$ for some $\mathsf{\Lambda }$-equivariant functions $F,H:{\mathbb{C}}^N\to {\mathbb{C}}^N$. Then, for any $(k,\ell )\in \mathsf{\Lambda }$ and $x\in {\mathbb{C}}^N$, we have

$$\begin{array}{cc}\hfill 〈F\left(x\right),\pi (k,\ell )v〉& =〈\pi {(k,\ell )}^{-1}F\left(x\right),v〉=〈{\omega }^{-k\ell }\pi (-k,-\ell )F\left(x\right),v〉\hfill \\ \hfill & =〈{\omega }^{-k\ell }F\left(\pi (-k,-\ell )x\right),v〉={\omega }^{-k\ell }{F}_v\left(\pi (-k,-\ell )x\right)\hfill \\ \hfill & ={\omega }^{-k\ell }{H}_v\left(\pi (-k,-\ell )x\right)=〈{\omega }^{-k\ell }H\left(\pi (-k,-\ell )x\right),v〉\hfill \\ \hfill & =〈{\omega }^{-k\ell }\pi (-k,-\ell )H\left(x\right),v〉=〈\pi {(k,\ell )}^{-1}H\left(x\right),v〉\hfill \\ \hfill & =〈H\left(x\right),\pi (k,\ell )v〉.\hfill \end{array}$$

Since $\left\{\pi \right(k,\ell )v:(k,\ell )\in \mathsf{\Lambda }\}$ is complete in ${\mathbb{C}}^N$, we obtain that $F\left(x\right)=H\left(x\right)$ for all $x\in {\mathbb{C}}^N$.

Now, let ${\left\{\pi (k,\ell )u\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ be a dual frame of ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ in ${\mathbb{C}}^N$, which means that

$$z=\sum _{(k,\ell )\in \mathsf{\Lambda }}〈z,\pi (k,\ell )v〉\pi (k,\ell )u,z\in {\mathbb{C}}^N.$$

Then, for any $x\in {\mathbb{C}}^N$, we have

$$\begin{array}{cc}\hfill \sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }{F}_v\left(\pi (-k,-\ell )x\right)\pi (k,\ell )u& =\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }〈F\left(\pi \right(-k,-\ell \left)x\right),v〉\pi (k,\ell )u\hfill \\ \hfill & =\sum _{(k,\ell )\in \mathsf{\Lambda }}〈{\omega }^{-k\ell }\pi (-k,-\ell )F\left(x\right),v〉\pi (k,\ell )u\hfill \\ \hfill & =\sum _{(k,\ell )\in \mathsf{\Lambda }}〈\pi {(k,\ell )}^{-1}F\left(x\right),v〉\pi (k,\ell )u\hfill \\ \hfill & =\sum _{(k,\ell )\in \mathsf{\Lambda }}〈F\left(x\right),\pi (k,\ell )v〉\pi (k,\ell )u\hfill \\ \hfill & =F\left(x\right),\hfill \end{array}$$

which establishes (7).

Finally, assume that $\left|\mathsf{\Lambda }\right|=N$. Then, ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ forms a Riesz basis for ${\mathbb{C}}^N$, so there exists a unique dual Riesz basis ${\left\{\pi (k,\ell )u\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ of ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ in ${\mathbb{C}}^N$, which is necessarily biorthogonal to ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ (see, e.g., [26]). To prove the surjectivity of $F\mapsto F_v$, we pick any ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous function $g:{\mathbb{C}}^N\to \mathbb{C}$ and set $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ by

$$F\left(x\right):=\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )x\right)\pi (k,\ell )u,x\in {\mathbb{C}}^N.$$

Then, for any $(p,q)\in \mathsf{\Lambda }$ and $x\in {\mathbb{C}}^N$, we have

$$\begin{array}{cc}\hfill F\left(\pi \right(p,q\left)x\right)& =\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )\pi (p,q)x\right)\pi (k,\ell )u\hfill \\ \hfill & =\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-(k+p)(\ell +q)}g\left(\pi (-k-p,-\ell -q)\pi (p,q)x\right)\pi (k+p,\ell +q)u\hfill \\ \hfill & \stackrel{\left(8\right)}{=}\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-(k+p)(\ell +q)}g\left({\omega }^{-(k+p)q}\pi (-k,-\ell )x\right){\omega }^{p\ell }\pi (p,q)\pi (k,\ell )u\hfill \\ \hfill & =\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-(k+p)(\ell +q)}{\omega }^{-(k+p)q}g\left(\pi (-k,-\ell )x\right){\omega }^{p\ell }\pi (p,q)\pi (k,\ell )u\hfill \\ \hfill & =\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )x\right)\pi (p,q)\pi (k,\ell )u\hfill \\ \hfill & =\pi (p,q)\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )x\right)\pi (k,\ell )u\hfill \\ \hfill & =\pi (p,q)F\left(x\right),\hfill \end{array}$$

which shows that F is $\mathsf{\Lambda }$-equivariant. Since ${\left\{\pi (k,\ell )u\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ and ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ are biorthogonal, it holds for any $x\in {\mathbb{C}}^N$ that

$$\begin{array}{cc}\hfill F_v\left(x\right)& =\langle F\left(x\right),v\rangle =〈\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )x\right)\pi (k,\ell )u,v〉\hfill \\ & =〈\sum _{(k,\ell )\in \mathsf{\Lambda }}g\left({\omega }^{-k\ell }\pi (-k,-\ell )x\right)\pi (k,\ell )u,v〉\hfill \\ & =〈\sum _{(k,\ell )\in \mathsf{\Lambda }}g\left(\pi {(k,\ell )}^{-1}x\right)\pi (k,\ell )u,v〉\hfill \\ & =g\left(x\right).\hfill \end{array}$$

Hence, we conclude that the mapping $F\mapsto F_v$ is also surjective. □

Remark 3.

As one would expect, the mapping $F\mapsto F_v$ is not surjective if $\left|\mathsf{\Lambda }\right|>N$. Indeed, if $\left|\mathsf{\Lambda }\right|>N$ and $\mathrm{span}\{\pi (k,\ell )v:(k,\ell )\in \mathsf{\Lambda }\}={\mathbb{C}}^N$, then there are many dual frames of $\left\{\pi \right(k,\ell )v:(k,\ell )\in \mathsf{\Lambda }\}$ in ${\mathbb{C}}^N$. If $g=F_v$ for some F and v, then for any dual frames ${\left\{\pi (k,\ell )w\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ and ${\left\{\pi (k,\ell )\tilde{w}\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ of ${\left\{\pi (k,\ell )v\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ we have

$$\begin{array}{c}\hfill \sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )x\right)\pi (k,\ell )w=F\left(x\right)=\sum _{(k,\ell )\in \mathsf{\Lambda }}{\omega }^{-k\ell }g\left(\pi (-k,-\ell )x\right)\pi (k,\ell )\tilde{w}\end{array}$$

for all $x\in {\mathbb{C}}^N$ by (7). Certainly, not every ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous function $g:{\mathbb{C}}^N\to \mathbb{C}$ satisfies this property.

3. Approximation of $\mathbf{\Lambda }$-Equivariant Maps

In this section, we consider an approximation of continuous $\mathsf{\Lambda }$-equivariant maps $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ that are generally nonlinear, where $\mathsf{\Lambda }$ is a subgroup of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ and the $\mathsf{\Lambda }$-equivariance is defined by (5). For instance, the map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ given by $F\left(x\right)={\parallel x\parallel }^{p}x$ with $p>0$, is a nonlinear continuous $\mathsf{\Lambda }$-equivariant map.

As seen in Section 2.2 (particularly in Theorem 3 and its proof), working with the time-frequency shift operators $\pi (k,\ell )$, $(k,\ell )\in \mathsf{\Lambda }$, usually requires careful bookkeeping of extra multiplicative phase factors due to the non-commutativity of T and M. (The non-commutativity of T and M can often be frustrating. However, it is precisely this non-commutativity that has given rise to the deep and rich theory of time-frequency analysis [23].) In fact, the map

$${\pi |}_{\mathsf{\Lambda }}:\mathsf{\Lambda }\to \mathcal{U}\left({\mathbb{C}}^N\right),(k,\ell )\mapsto M^{\ell }{T}^k,$$

is generally not a group homomorphism; indeed,

$$\pi (k,\ell )\pi (k^{\prime },{\ell }^{\prime })=e^{-2\pi ik{\ell }^{\prime }/N}\pi (k+k^{\prime },\ell +{\ell }^{\prime })$$

(8)

is equal to $\pi (k+k^{\prime },\ell +{\ell }^{\prime })$ only if $k{\ell }^{\prime }$ is a multiple of N (see Proposition 1). (Although ${\pi |}_{\mathsf{\Lambda }}$ is not a group homomorphism and thus not a group representation, it is often referred to as a projective group representation of G on ${\mathbb{C}}^N$. In general, a map $\rho :G\to \mathcal{U}\left(\mathcal{H}\right)$ is called a projective group representation of G on $\mathcal{H}$ if for each pair of $g_1,g_2\in G$, there exists a unimodular $c(g_1,g_2)\in \mathbb{C}$ such that $\rho \left(g_1{g}_2\right)=c(g_1,g_2)\rho \left(g_1\right)\rho \left(g_2\right)$; see, e.g., [25].) Obviously, the computations involved would be simplified significantly if ${\pi |}_{\mathsf{\Lambda }}$ were a group homomorphism in general. Note that, as mentioned in Section 1, a group homomorphism $\rho :G\to \mathcal{U}\left(\mathcal{H}\right)$ whose images are unitary operators on $\mathcal{H}$ is called a (unitary) group representation of G on $\mathcal{H}$, where G is a group and $\mathcal{H}$ is a separable Hilbert space. Therefore, the map ${\pi |}_{\mathsf{\Lambda }}$ would be a unitary representation if it were a group homomorphism.

In the following, we first discuss a systematic method of avoiding such extra multiplicative phase factors by embedding $\mathsf{\Lambda }\subset {\mathbb{Z}}_N\times {\mathbb{Z}}_N$ into the Weyl–Heisenberg group. After briefly reviewing essential concepts on group representations and neural networks, we consider cyclic subgroups of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, in which case the map ${\pi |}_{\mathsf{\Lambda }}$ can be replaced by a unitary group representation. We show that if $\mathsf{\Lambda }$ is a cyclic subgroup of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, then any $\mathsf{\Lambda }$-equivariant map ${\mathbb{C}}^N\to {\mathbb{C}}^N$ can be approximated with shallow neural networks involving the adjoint group ${\mathsf{\Lambda }}^{\circ }$, which has significantly fewer degrees of freedom compared with standard shallow neural networks.

3.1. Embedding of $\mathsf{\Lambda }$ into the Weyl–Heisenberg Group

To avoid the bookkeeping of extra multiplicative phase factors, we can simply embed the subgroups of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ into the finite Weyl–Heisenberg group ${\mathbb{H}}_N={\mathbb{Z}}_N\times {\mathbb{Z}}_N\times {\mathbb{Z}}_N$, on which group representations can be defined. There exists a group representation $\tau :{\mathbb{H}}_N\to \mathcal{U}\left({\mathbb{C}}^N\right)$, known as the Schrödinger representation, which satisfies $\tau (k,\ell ,0)=\pi (k,\ell )$ for all $(k,\ell )\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N$. In fact, for any subgroup $\mathsf{\Lambda }$ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ and any subgroup $\mathsf{\Omega }$ of ${\mathbb{Z}}_N$ containing ${\mathsf{\Omega }}_{\mathsf{\Lambda }}:=\{k{\ell }^{\prime }modN:(k,\ell ),(k^{\prime },{\ell }^{\prime })\in \mathsf{\Lambda }\}$, the map

$$\tau :\mathsf{\Lambda }\times \mathsf{\Omega }\to \mathcal{U}\left(\mathcal{H}\right),\tau (k,\ell ,s)=e^{2\pi is/N}{M}^{\ell }{T}^k,$$

(9)

is a group representation of $G=\mathsf{\Lambda }\times \mathsf{\Omega }$ on ${\mathbb{C}}^N$, with the group operation on G given by

$$(k,\ell ,s)+(k^{\prime },{\ell }^{\prime },s^{\prime }):=(k+k^{\prime },\ell +{\ell }^{\prime },s+s^{\prime }-k{\ell }^{\prime }).$$

Clearly, we have $\tau (k,\ell ,0)=M^{\ell }{T}^k=\pi (k,\ell )$ for all $(k,\ell )\in \mathsf{\Lambda }$.

It is clear that a map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ is $\mathsf{\Lambda }$-equivariance in the sense of (5) if and only if it is $(\tau ,\mathsf{\Lambda }\times \{0\left\}\right)$-equivariant in the sense of Definition 4. Moreover, in this case, Proposition 2 implies that F is ${\mathsf{\Omega }}_{\mathsf{\Lambda }}$-phase homogeneous, which is equivalent to $F\circ \tau (0,0,s)=\tau (0,0,s)\circ F$ for all $s\in {\mathsf{\Omega }}_{\mathsf{\Lambda }}$. Consequently, we have the following proposition.

Proposition 3.

For any subgroup Λ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ and any $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$, the following are equivalent.

(i) 

F is Λ-equivariant;

(ii) 

F is $(\tau ,\mathsf{\Lambda }\times \{0\left\}\right)$-equivariant;

(iii) 

F is $(\tau ,\mathsf{\Lambda }\times {\mathsf{\Omega }}_{\mathsf{\Lambda }})$-equivariant.

Using the true group representation $\tau$ instead of $\pi |\mathsf{\Lambda }$ allows us to avoid the tedious bookkeeping of extra multiplicative phase factors. Note, however, that $\tau$ requires three input parameters, while $\pi |\mathsf{\Lambda }$ involves only two. In fact, the description of the extra phase factors is simply transferred to the third parameter of $\tau$. Nevertheless, an important advantage of using $\tau$ instead of ${\pi |}_{\mathsf{\Lambda }}$ is that it allows for the use of tools from group representation theory.

3.2. Group Representations and Neural Networks

In this section, we review some concepts and tools from group representation theory and introduce the so-called ♮-transform and its inverse transform for later use. We also review the basic structure of neural networks and the universal approximation theorem.

We assume that G is a finite group, and consider maps of the form $F:\mathcal{H}\to \mathcal{H}$, where $\mathcal{H}$ is a finite-dimensional Hilbert spaces on which a unitary representation $\rho$ of G is defined. This means that for each $\lambda \in G$, the map $\rho \left(\lambda \right):\mathcal{H}\to \mathcal{H}$ is a linear unitary operator, and that $\rho :G\to \mathcal{U}\left(\mathcal{H}\right)$ is a group homomorphism, i.e., $\rho \left({\lambda }_1{\lambda }_2\right)=\rho \left({\lambda }_1\right)\rho \left({\lambda }_2\right)$ for all ${\lambda }_1,{\lambda }_2\in G$. Let us formally state the definition of equivariance and invariance in this setting.

Definition 4

(Equivariance and Invariance). For a group G and a unitary representation ρ of G on a Hilbert space $\mathcal{H}$, we say that a map $F:\mathcal{H}\to \mathcal{H}$ is

• $(\rho ,G)$-equivariant if $F\rho \left(\lambda \right)=\rho \left(\lambda \right)F$ for all $\lambda \in G$;
• $(\rho ,G)$-invariant if $F\rho \left(\lambda \right)=F$ for all $\lambda \in G$.

Note that a $(\rho ,G)$-equivariant/invariant map $F:\mathcal{H}\to \mathcal{H}$ is not necessarily linear or bounded.

Definition 5.

For a group G, the left translation of a vector $x\in {\mathbb{C}}^G$ by $\lambda \in G$ is given by

$$L_{\lambda }x\left(\nu \right):=x\left({\lambda }^{-1}\nu \right)for\nu \in G.$$

In fact, the map $\lambda \mapsto L_{\lambda }$ is a group homomorphism from G to $\mathcal{U}\left({\mathbb{C}}^G\right)$, that is, $L_{{\lambda }_1{\lambda }_2}=L_{{\lambda }_1}{L}_{{\lambda }_2}$ for all ${\lambda }_1,{\lambda }_2\in G$, and therefore, it induces a group representation of G on ${\mathbb{C}}^G$. We say that a map $\mathsf{\Phi }:{\mathbb{C}}^G\to {\mathbb{C}}^G$ is left G-translation equivariant if $\mathsf{\Phi }{L}_{\lambda }=L_{\lambda }\mathsf{\Phi }$ for all $\lambda \in G$.

Definition 6.

Let G be a group and let ρ be a unitary representation of G on a Hilbert space $\mathcal{H}$. Given a window $g\in \mathcal{H}$, the set $\left\{\rho \right(\lambda )g:\lambda \in G\}$ is called the orbit of g under $\rho \left(\lambda \right)$ for $\lambda \in G$. The map $U_g:\mathcal{H}\to {\mathbb{C}}^G$ defined by

$$U_g\left(f\right)={\left\{\langle f,\rho \left(\lambda \right)g\rangle \right\}}_{\lambda \in G}$$

is called the analysis operator of $\left\{\rho \right(\lambda )g:\lambda \in G\}$, and its adjoint operator $U_g^{\ast }:{\mathbb{C}}^G\to \mathcal{H}$ given by

$$U_g^{\ast }\left(x\right)=\sum _{\lambda \in G}{x}_{\lambda }\rho \left(\lambda \right)g$$

is called the synthesis operator of $\left\{\rho \right(\lambda )g:\lambda \in G\}$.

It is easy to check that

$$U_g\rho \left(\lambda \right)=L_{\lambda }{U}_g\mathrm{and}{U}_g^{\ast }{L}_{\lambda }=\rho \left(\lambda \right)U_g^{\ast },\lambda \in G.$$

(10)

We are particularly interested in the case where the orbit of g spans $\mathcal{H}$, that is, $\mathrm{span}\left\{\rho \right(\lambda )g:\lambda \in G\}=\mathcal{H}$. Since $\mathcal{H}$ is finite-dimensional, this implies that $\left\{\rho \right(\lambda )g:\lambda \in G\}$ is a frame for $\mathcal{H}$ and the associated frame operator $S_g:=U_g^{\ast }{U}_g$ is a positive, self-adjoint bounded operator on $\mathcal{H}$. It follows from (10) that $S_g\rho \left(\lambda \right)=\rho \left(\lambda \right)S_g$ and thus $S_g^{-1}\rho \left(\lambda \right)=\rho \left(\lambda \right)S_g^{-1}$ for all $\lambda \in G$. For any $f\in \mathcal{H}$, we have

$$\begin{array}{cc}\hfill f& =S_g^{-1}{S}_{g}f=S_g^{-1}\left(\sum _{\lambda \in G}{U}_{g}f\left(\lambda \right)\rho \left(\lambda \right)g\right)=\sum _{\lambda \in G}{U}_{g}f\left(\lambda \right)S_g^{-1}\left(\rho \left(\lambda \right)g\right)\hfill \\ & =\sum _{\lambda \in G}{U}_{g}f\left(\lambda \right)\rho \left(\lambda \right)S_g^{-1}\left(g\right)=\sum _{\lambda \in G}{U}_{g}f\left(\lambda \right)\rho \left(\lambda \right)g^{\ast }=U_{g^{\ast }}^{\ast }{U}_{g}f,\hfill \end{array}$$

where $g^{\ast }:=S_g^{-1}\left(g\right)\in \mathcal{H}$. This shows that $U_{g^{\ast }}^{\ast }{U}_g$ is the identity operator on $\mathcal{H}$, i.e.,

$$U_{g^{\ast }}^{\ast }{U}_g={Id}_{\mathcal{H}},$$

(11)

and correspondingly, $\{\rho \left(\lambda \right)g^{\ast }:\lambda \in G\}$ is the canonical dual frame of $\left\{\rho \right(\lambda )g:\lambda \in G\}$.

In light of (11), we newly introduce a transform which lifts a map $\mathcal{H}\to \mathcal{H}$ to a map ${\mathbb{C}}^G\to {\mathbb{C}}^G$, and also its inverse transform.

Definition 7.

Let G be a finite group and let ρ be a unitary representation of G on a finite-dimensional Hilbert space $\mathcal{H}$. Assume that $\mathrm{span}\left\{\rho \right(\lambda )g:\lambda \in G\}=\mathcal{H}$, and let $S_g:=U_g^{\ast }{U}_g$ and $g^{\ast }:=S_g^{-1}\left(g\right)$. For any map $F:\mathcal{H}\to \mathcal{H}$, the ♮-transform of F is defined by

$$F^{♮}:=U_g\circ F\circ U_{g^{\ast }}^{\ast }:{\mathbb{C}}^G\to {\mathbb{C}}^G.$$

For any map $\mathsf{\Phi }:{\mathbb{C}}^G\to {\mathbb{C}}^G$, the inverse ♮-transform of Φ is defined by

$${\mathsf{\Phi }}^{-♮}:=U_{g^{\ast }}^{\ast }\circ \mathsf{\Phi }\circ U_g:\mathcal{H}\to \mathcal{H}.$$

As shown in Figure 1, the ♮-transform converts a map $\mathcal{H}\to \mathcal{H}$ into a map ${\mathbb{C}}^G\to {\mathbb{C}}^G$, and the inverse ♮-transform converts a map ${\mathbb{C}}^G\to {\mathbb{C}}^G$ into a map $\mathcal{H}\to \mathcal{H}$.

Proposition 4.

Let G be a finite group, and let ρ be a unitary representation of G on a finite-dimensional Hilbert space $\mathcal{H}$. Assume that $\mathrm{span}\left\{\rho \right(\lambda )g:\lambda \in G\}=\mathcal{H}$, and let $S_g:=U_g^{\ast }{U}_g$ and $g^{\ast }:=S_g^{-1}\left(g\right)$. Then, the following hold.

(i) 

${\left(F^{♮}\right)}^{-♮}=F$ for any map $F:\mathcal{H}\to \mathcal{H}$.

(ii) 

A map $F:\mathcal{H}\to \mathcal{H}$ is continuous if and only if $F^{♮}$ is continuous.

(iii) 

A map $F:\mathcal{H}\to \mathcal{H}$ is $(\rho ,G)$-equivariant if and only if $F^{♮}$ is left G-translation equivariant.

Proof. 

(i) It follows from (11) that ${\left(F^{♮}\right)}^{-♮}=U_{g^{\ast }}^{\ast }\left(U_{g}F{U}_{g^{\ast }}^{\ast }\right)U_g=F$ for any $F:\mathcal{H}\to \mathcal{H}$.

• (ii) Since the maps $U_g:\mathcal{H}\to {\mathbb{C}}^G$ and $U_{g^{\ast }}^{\ast }:{\mathbb{C}}^G\to \mathcal{H}$ are bounded linear operators, the continuity of F implies the continuity of $F^{♮}=U_{g}F{U}_{g^{\ast }}^{\ast }$. Similarly, the continuity of $F^{♮}$ implies the continuity of $F={\left(F^{♮}\right)}^{-♮}=U_{g^{\ast }}^{\ast }{F}^{♮}{U}_g$.
• (iii) It follows from (10) that the G-equivariance of F implies the left G-translation equivariance of $F^{♮}=U_{g}F{U}_{g^{\ast }}^{\ast }$. Similarly, the left G-translation equivariance of $F^{♮}$ implies the G-equivariance of $F={\left(F^{♮}\right)}^{-♮}=U_{g^{\ast }}^{\ast }{F}^{♮}{U}_g$. □

We now provide a brief review of neural networks and the universal approximation theorem.

Let $\mathbb{K}$ be either $\mathbb{R}$ or $\mathbb{C}$. An activation function is a function $\sigma :\mathbb{K}\to \mathbb{K}$ that acts componentwise on vectors; that is, $\sigma (x_1,\dots ,x_n)=\left(\sigma \left(x_1\right),\dots ,\sigma \left(x_n\right)\right)$ for any $(x_1,\dots ,x_n)\in {\mathbb{K}}^n$.

A fully connected feedforward neural network with P hidden layers is given by

$$\mathsf{\Psi }:{\mathbb{K}}^d\to {\mathbb{K}}^n,\mathsf{\Psi }\left(x\right)=R^{\left(P\right)}\circ \left(\sigma \circ R^{(P-1)}\right)\circ \cdots \circ \left(\sigma \circ R^{\left(0\right)}\right),$$

(12)

where $R^{\left(p\right)}:{\mathbb{K}}^{N_p}\to {\mathbb{K}}^{N_{p+1}}$, $x\mapsto A^{\left(p\right)}x+b^{\left(p\right)}$ is affine-linear with $N_0=d$ and $N_{P+1}=n$. Such a function $\mathsf{\Psi }$ is often called a neural network, but we will call it a σ-neural network to specify the activation function employed.

A shallow neural network is a neural network with a single ($P=1$) hidden layer. In particular, a shallow neural network with output dimension $n=1$ is given by

$$\mathsf{\Psi }:{\mathbb{K}}^d\to \mathbb{K},\mathsf{\Psi }\left(x\right)=\sum _{j=1}^J{c}_j\sigma (w_j^{T}x+b_j)\mathrm{with}\mathrm{some}J\in \mathbb{N},c_j,b_j\in \mathbb{K},w_j\in {\mathbb{K}}^d.$$

(13)

Definition 8.

A function $\sigma :\mathbb{K}\to \mathbb{K}$ is called shallow universal if the set of $\mathbb{K}$-valued shallow σ-networks is dense in the set of all continuous functions $f:{\mathbb{K}}^d\to \mathbb{K}$, with respect to locally uniform convergence.

The following theorem, known as the universal approximation theorem, is a fundamental result in the theory of neural networks.

Theorem 4

(The universal approximation theorem; see [27,28,29,30,31] for $\mathbb{K}=\mathbb{R}$, and [32] for $\mathbb{K}=\mathbb{C}$). Let $d\in \mathbb{N}$.

• A function $\sigma :\mathbb{R}\to \mathbb{R}$ is shallow universal if and only if σ is not a polynomial.
• A function $\sigma :\mathbb{C}\to \mathbb{C}$ is shallow universal if and only if σ is not a polyharmonic. Here, a function $\tau :\mathbb{C}\to \mathbb{C}$ is called polyharmonic if there exists $m\in \mathbb{N}$ such that $\tau \in C^{2m}$ in the sense of real variables and ${\Delta }^m\sigma \equiv 0$, where $\Delta ={\displaystyle \frac{{\partial }^2}{\partial x^2}}+{\displaystyle \frac{{\partial }^2}{\partial y^2}}$ is the usual Laplace operator on $\mathbb{C}\cong {\mathbb{R}}^2$.

In 1996, Mhaskar [33] obtained a quantitative result for approximation of $C^n$ functions using shallow networks with smooth activation functions. More recently, Yarotsky [34] derived a quantitative approximation result for deep ReLU networks, where ReLU networks are given by (12) with $\mathbb{K}=\mathbb{R}$ and the ReLU activation function $\sigma :\mathbb{R}\to \mathbb{R}$, $\sigma \left(x\right)=max\{x,0\}$, and “deep” refers to having a large $P\in \mathbb{N}$ in (12). For the case of complex-valued deep neural networks, we refer to [35].

3.3. Cyclic Subgroups $\mathsf{\Lambda }$ of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$

We now consider the case of cyclic subgroups of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, where group representations can be defined directly without embedding into the Weyl–Heisenberg group. The cyclic subgroups of order N in ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ are given by

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_s& =\left\{\right(0,0),(1,s),\dots ,(N-1,(N-1)s\left)\right\}=\langle (1,s)\rangle ,s=0,\dots ,N-1,\hfill \\ \hfill {\mathsf{\Lambda }}_{\infty }& =\left\{\right(0,0),(0,1),\dots ,(0,N-1\left)\right\}=\langle (0,1)\rangle .\hfill \end{array}$$

If N is prime, these are the only nontrivial proper subgroups of ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, but if N is composite, there exist noncyclic subgroups of order N in ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$; for instance, $\{0,2,4\}\times \{0,3\}$ is a noncyclic subgroup of order 6 in ${\mathbb{Z}}_6\times {\mathbb{Z}}_6$. It is easily seen that the adjoint group of ${\mathsf{\Lambda }}_s$ in ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$ is ${\mathsf{\Lambda }}_s$ itself; that is, ${\left({\mathsf{\Lambda }}_s\right)}^{\circ }={\mathsf{\Lambda }}_s$ (see Section 2.1).

We define the map $\rho :{\mathsf{\Lambda }}_s\to \mathcal{U}\left({\mathbb{C}}^N\right)$ by

$$\left(\rho (k,\ell )x\right)\left(n\right)=e^{-k\ell \pi i/N}{e}^{2\pi i\ell n/N}x(n-k),(k,\ell )\in {\mathsf{\Lambda }}_s,x\in {\mathbb{C}}^N.$$

Setting ${\omega }_0:=e^{\pi i/N}$, we may simply write

$$\rho (k,\ell )={\omega }_0^{-k\ell }{M}^{\ell }{T}^k={\omega }_0^{k\ell }{T}^k{M}^{\ell },(k,\ell )\in {\mathsf{\Lambda }}_s.$$

(14)

For any $(k,\ell ),(k^{\prime },{\ell }^{\prime })\in {\mathsf{\Lambda }}_s$, we have

$$\begin{array}{cc}\hfill \rho (k+k^{\prime },\ell +{\ell }^{\prime })& ={\omega }_0^{-(k+\ell )(k^{\prime }+{\ell }^{\prime })}{M}^{\ell +{\ell }^{\prime }}{T}^{k+k^{\prime }}={\omega }_0^{-k\ell -k^{\prime }{\ell }^{\prime }-2k{\ell }^{\prime }}{M}^{\ell +{\ell }^{\prime }}{T}^{k+k^{\prime }}\hfill \\ & \stackrel{\left(8\right)}{=}{\omega }_0^{-k\ell -k^{\prime }{\ell }^{\prime }}{M}^{\ell }{T}^k{M}^{{\ell }^{\prime }}{T}^{k^{\prime }}=\rho (k,\ell )\rho (k^{\prime },{\ell }^{\prime }),\hfill \end{array}$$

where we used the fact that $k^{\prime }\ell =k{\ell }^{\prime }$ for all $(k,\ell ),(k^{\prime },{\ell }^{\prime })\in {\mathsf{\Lambda }}_s$. This shows that $\rho$ is a group homomorphism and thus a unitary group representation of ${\mathsf{\Lambda }}_s$ on ${\mathbb{C}}^N$. Due the symmetry in (14), $\rho$ is called the symmetric representation of ${\mathsf{\Lambda }}_s$ on ${\mathbb{C}}^N$.

Note that for any $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ and $(k,\ell )\in {\mathbb{Z}}_N\times {\mathbb{Z}}_N$, we have $F\pi (k,\ell )=\pi (k,\ell )F$ if and only if $F\rho (k,\ell )=\rho (k,\ell )F$, where we used the relation $\rho (k,\ell )={\omega }_0^{-k\ell }\pi (k,\ell )$ from (14). This implies that a map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ is ${\mathsf{\Lambda }}_s$-equivariant in the sense of Definition 1 if and only if it is $(\rho ,{\mathsf{\Lambda }}_s)$-equivariant in the sense of Definition 4. Importantly, employing $(\rho ,{\mathsf{\Lambda }}_s)$-equivariance in place of ${\mathsf{\Lambda }}_s$-equivariance will allow us to apply the tools from group representation theory described in Section 3.2.

We are interested in approximating ${\mathsf{\Lambda }}_s$-equivariant (or $(\rho ,{\mathsf{\Lambda }}_s)$-equivariant) maps $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ by neural networks. For this, we need to choose a complex-valued activation function $\sigma :\mathbb{C}\to \mathbb{C}$ (see Section 3.2) for the neural networks. Since $\sigma$ acts componentwise on its input, i.e., $(x_1,\dots ,x_N)\mapsto (\sigma \left(x_N\right),\dots ,\sigma \left(x_N\right))$, it clearly commutes with all translations, i.e., $\sigma T=T\sigma$; however, $\sigma$ does not commute with modulations in general. As shown in (14), the representation $\rho$ includes the multiplicative phase factor ${\omega }_0=e^{\pi i/N}$, so we will assume that $\sigma :\mathbb{C}\to \mathbb{C}$ is $e^{\pi i/N}$-phase homogeneous (see Definition 2):

$$\sigma \left(e^{\pi i/N}z\right)=e^{\pi i/N}\sigma \left(z\right),z\in \mathbb{C},$$

which ensures that $\sigma$ commutes with all $\rho (k,\ell )$ and all modulations.

We first need the following lemma. Below, we denote by ${\mathbb{1}}_N:=(1,1,\dots ,1)\in {\mathbb{C}}^N$ the vector whose entries are all equal to 1.

Lemma 1.

Assume that $\sigma :\mathbb{C}\to \mathbb{C}$ is shallow-universal. If a map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ satisfies $FT=TF$, then there exists a shallow convolutional neural network

$$\mathsf{\Psi }:{\mathbb{C}}^N\to {\mathbb{C}}^N,\mathsf{\Psi }\left(x\right)=\sum _{j=1}^J{c}_j\sigma (B_{j}x+b_j{\mathbb{1}}_N),x\in {\mathbb{C}}^N,$$

where $B_j\in \mathrm{span}\{T^k:k=0,\dots ,N-1\}$ and $b_j\in \mathbb{C}$ for $j=1,\dots ,J$, which approximates F uniformly on compact sets in ${\mathbb{C}}^N$.

Proof. 

Using the universal approximation theorem (see Theorem 4), the first output component map $F_0:{\mathbb{C}}^N\to \mathbb{C}$, $x\mapsto \left(Fx\right)\left(0\right)$, can be approximated by a shallow network

$$\psi :{\mathbb{C}}^N\to \mathbb{C},x\mapsto \sum _{j=1}^J{c}_j\sigma (w_j^{\mathrm{T}}x+b_j)$$

with some $J\in \mathbb{N}$, $b_j,c_j\in \mathbb{C}$, $w_j\in {\mathbb{C}}^N$. Note that since $FT=TF$ and since $T^N$ is the identity map on ${\mathbb{C}}^N$, we have $F{T}^n=T^{n}F$ for all $n\in \mathbb{Z}$. This condition provides approximations for other component maps $F_n:{\mathbb{C}}^N\to \mathbb{C}$, $x\mapsto \left(Fx\right)\left(n\right)$, with $n=1,\dots ,N-1$, in terms of $\mathsf{\Psi }$. In fact, we have

$$\left(Fx\right)\left(n\right)=\left(T^{-n}Fx\right)\left(0\right)=\left(F{T}^{-n}x\right)\left(0\right)\approx \psi \left(T^{-n}x\right),x\in {\mathbb{C}}^N,n=1,\dots ,N-1.$$

Consequently, the map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$, $x\mapsto {\left\{\left(Fx\right)\left(n\right)\right\}}_{n=0}^{N-1}$, is approximated by the map $\mathsf{\Psi }:{\mathbb{C}}^N\to {\mathbb{C}}^N$ defined by $\left(\mathsf{\Psi }x\right)\left(n\right)=\psi \left(T^{-n}x\right)$ for $n=0,\dots ,N-1$. For $x,y\in {\mathbb{C}}^N$, let $x\ast y$ be the circular convolution of a and b defined by $(x\ast y)\left(n\right)={\sum }_{k=0}^{N-1}{x}_k{y}_{n-k}$, where x and y are understood as N-periodic sequences on the integers. Then, for any $x\in {\mathbb{C}}^N$ and $n=0,\dots ,N-1$, we have

$$\mathsf{\Psi }\left(T^{-n}x\right)=\sum _{j=1}^J{c}_j\sigma \left((w_j\ast x)\left(n\right)+b_j\right),$$

and therefore, we may write

$$\mathsf{\Psi }:{\mathbb{C}}^N\to {\mathbb{C}}^N,\mathsf{\Psi }\left(x\right)={\left\{\psi \left(T^{-n}x\right)\right\}}_{n=0}^{N-1}={\left\{\sum _{j=1}^J{c}_j\sigma \left((w_j\ast x)\left(n\right)+b_j\right)\right\}}_{n=0}^{N-1}.$$

It is easily seen that every convolutional map ${\mathbb{C}}^N\to {\mathbb{C}}^N$, $x\mapsto w\ast x$, is a linear map, and in fact, a linear combination of $T^k$, $k=0,\dots ,N-1$. Hence, the map $\mathsf{\Psi }:{\mathbb{C}}^N\to {\mathbb{C}}^N$ can be rewritten as

$$\mathsf{\Psi }\left(x\right)=\sum _{j=1}^J{c}_j\sigma (B_{j}x+b_j{\mathbb{1}}_N),x\in {\mathbb{C}}^N,$$

where $B_j\in \mathrm{span}\{T^k:k=0,\dots ,N-1\}$ for $j=1,\dots ,J$. The fact that $\mathsf{\Psi }$ approximates F uniformly on compact sets in ${\mathbb{C}}^N$ follows from the uniform approximation of $F_0$ by $\psi$ on compact sets in $\mathbb{C}$. Finally, we note that $\mathsf{\Psi }$ expressed above is a shallow convolutional neural network described in Section 3.2. This completes the proof. □

Theorem 5.

Assume that $\sigma :\mathbb{C}\to \mathbb{C}$ is shallow universal and satisfies $\sigma \left(e^{\pi i/N}z\right)=e^{\pi i/N}\sigma \left(z\right)$ for all $z\in \mathbb{C}$. Let $\mathsf{\Lambda }={\mathsf{\Lambda }}_s$ for some $s\in \{0,1,\dots ,N-1\}$. Then, any continuous $(\rho ,\mathsf{\Lambda })$-equivariant (or Λ-equivariant) map $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ can be approximated (uniformly on compact sets) by a shallow neural network

$$x\mapsto \sum _{j=1}^J{c}_j\sigma (A_{j}x+b_{j}v),$$

where $A_j\in \mathrm{span}\{\rho (k,\ell ):(k,\ell )\in \mathsf{\Lambda }\}$ and $b_j\in \mathbb{C}$ for $j=1,\dots ,J$, and $v\in {\mathbb{C}}^N$ satisfies $\rho (k,\ell )v=v$ for all $(k,\ell )\in \mathsf{\Lambda }$. Moreover, every map of this form is $(\rho ,\mathsf{\Lambda })$-equivariant (or Λ-equivariant).

Remark 4.

Since $\rho (k,\ell )={\omega }_0^{-k\ell }\pi (k,\ell )$ by (14), we have $\mathrm{span}\left\{\rho \right(k,\ell ):(k,\ell )\in \mathsf{\Lambda }\}=\mathrm{span}\left\{\pi \right(k,\ell ):(k,\ell )\in \mathsf{\Lambda }\}$ for any $\mathsf{\Lambda }\subset {\mathbb{Z}}_N\times {\mathbb{Z}}_N$. On the other hand, the vectors satisfying $\rho (k,\ell )b=b$ can be significantly different from those satisfying $\pi (k,\ell )b=b$.

Proof. 

Since $\mathsf{\Lambda }={\mathsf{\Lambda }}_s$ is cyclic, we order its elements as $(0,0),(1,s),\dots ,(N-1,(N-1\left)s\right)$, and treat ${\mathbb{C}}^{\mathsf{\Lambda }}$ as ${\mathbb{C}}^N$, since ${\mathbb{C}}^{\mathsf{\Lambda }}\simeq {\mathbb{C}}^N$. Then, the operators $U_g:{\mathbb{C}}^N\to {\mathbb{C}}^{\mathsf{\Lambda }}$ and $U_g^{\ast }:{\mathbb{C}}^{\mathsf{\Lambda }}\to {\mathbb{C}}^N$, given in Definition 6, can be represented as the $N\times N$ matrices

$$U_g=\left[\begin{array}{c}{\left(\rho (0,0)g\right)}^{\ast }\\ ⋮\\ {\left(\rho (N-1,(N-1\left)s\right)g\right)}^{\ast }\end{array}\right],U_g^{\ast }=\left[\begin{array}{c}\rho (0,0)g,\dots ,\rho (N-1,(N-1\left)s\right)g\end{array}\right],$$

respectively, where ${(·)}^{\ast }$ denotes the conjugate transpose. Setting $g=(1,0,\dots ,0)\in {\mathbb{C}}^N$, we have

$$U_g^{\ast }=\mathrm{diag}{\left(e^{k^{2}s\pi i/N}\right)}_{k=0}^{N-1}=\left[\begin{array}{cccc}1& 0& \cdots & 0\\ 0& e^{1^{2}s\pi i/N}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & e^{{(N-1)}^{2}s\pi i/N}\end{array}\right],$$

(15)

so that $S_g=U_g^{\ast }{U}_g={\mathrm{Id}}_N$ and $g^{\ast }:=S_g^{-1}g=g$. As a result, the set ${\left\{\rho (k,\ell )g\right\}}_{(k,\ell )\in \mathsf{\Lambda }}$ forms an orthonormal basis for ${\mathbb{C}}^N$.

Note that for any continuous $(\rho ,\mathsf{\Lambda })$-equivariant $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$, the map $F^{♮}:=U_{g}F{U}_g^{\ast }:{\mathbb{C}}^{\mathsf{\Lambda }}\to {\mathbb{C}}^{\mathsf{\Lambda }}$ is continuous and left $\mathsf{\Lambda }$-translation equivariant (see Proposition 4). If F is linear, then $F^{♮}$ is also linear and can be represented as a circulant matrix, equivalently, $F^{♮}={\sum }_{k=0}^{N-1}{c}_k{T}^k:{\mathbb{C}}^{\mathsf{\Lambda }}\to {\mathbb{C}}^{\mathsf{\Lambda }}$ for some $c_0,\dots ,c_{N-1}\in \mathbb{C}$, so that

$$F=U_g^{\ast }\left(U_{g}F{U}_g^{\ast }\right)U_g=U_g^{\ast }{F}^{♮}{U}_g=\sum _{k=0}^{N-1}{c}_k{\left(U_g^{\ast }T{U}_g\right)}^k.$$

Therefore, the commutant of $(\rho ,\mathsf{\Lambda })$ is given by

$$\begin{array}{cc}\hfill C(\rho ,\mathsf{\Lambda }):=& \{F\in \mathcal{L}\left({\mathbb{C}}^N\right):F\rho (k,\ell )=\rho (k,\ell )F\mathrm{for}\mathrm{all}(k,\ell )\in \mathsf{\Lambda }\}\hfill \\ \hfill =& \mathrm{span}\{U_g^{\ast }{T}^k{U}_g:k=0,\dots ,N-1\}.\hfill \end{array}$$

On the other hand, since $\rho (k,\ell )={\omega }_0^{-k\ell }\pi (k,\ell )$ by (14), the commutant of $(\rho ,\mathsf{\Lambda })$ coincides with that of $(\pi ,\mathsf{\Lambda })$, i.e.,

$$C(\rho ,\mathsf{\Lambda })=C(\pi ,\mathsf{\Lambda })\stackrel{\left(4\right)}{=}\mathrm{span}\{\pi (k,\ell ):(k,\ell )\in {\mathsf{\Lambda }}^{\circ }\}=\mathrm{span}\{\rho (k,\ell ):(k,\ell )\in {\mathsf{\Lambda }}^{\circ }\}.$$

Since the adjoint group of $\mathsf{\Lambda }={\mathsf{\Lambda }}_s$ is itself, i.e., ${\mathsf{\Lambda }}^{\circ }=\mathsf{\Lambda }$ (see Section 2.1), we obtain

$$\mathrm{span}\{\rho (k,\ell ):(k,\ell )\in \mathsf{\Lambda }\}=C(\rho ,\mathsf{\Lambda })=\mathrm{span}\{U_g^{\ast }{T}^k{U}_g:k=0,\dots ,N-1\}.$$

(16)

Now, we consider the general case where $F:{\mathbb{C}}^N\to {\mathbb{C}}^N$ is possibly nonlinear. If F is nonlinear, then $F^{♮}=U_{g}F{U}_g^{\ast }:{\mathbb{C}}^{\mathsf{\Lambda }}\to {\mathbb{C}}^{\mathsf{\Lambda }}$ is a nonlinear left $\mathsf{\Lambda }$-translation equivariant map. Since $\mathsf{\Lambda }={\mathsf{\Lambda }}_s=\{(0,0),(1,s),\dots ,(N-1,(N-1)s)\}$ is an additive group and since $\left|\mathsf{\Lambda }\right|=N$ and ${\mathbb{C}}^{\mathsf{\Lambda }}\simeq {\mathbb{C}}^N$, the map $F^{♮}$ can be viewed as a map from ${\mathbb{C}}^N$ to ${\mathbb{C}}^N$. For simplicity, we will abuse notation and write $F^{♮}:{\mathbb{C}}^N\to {\mathbb{C}}^N$ instead of $F^{♮}:{\mathbb{C}}^{\mathsf{\Lambda }}\to {\mathbb{C}}^{\mathsf{\Lambda }}$; thus, the first component of $F^{♮}\left(x\right)\in {\mathbb{C}}^{\mathsf{\Lambda }}(\simeq {\mathbb{C}}^N)$ will be simply denoted by $\left(F^{♮}x\right)\left(0\right)$ instead of $\left(F^{♮}x\right)(0,0)$. Then, the left $\mathsf{\Lambda }$-translation equivariance of $F^{♮}$ can be expressed as $F^{♮}T=T{F}^{♮}$. By applying Lemma 1 to $F^{♮}:{\mathbb{C}}^N\to {\mathbb{C}}^N$, we obtain a shallow convolutional neural network

$$\mathsf{\Psi }:{\mathbb{C}}^N\to {\mathbb{C}}^N,\mathsf{\Psi }\left(x\right)=\sum _{j=1}^J{c}_j\sigma (B_{j}x+b_j{\mathbb{1}}_N),x\in {\mathbb{C}}^N,$$

where $B_j\in \mathrm{span}\{T^k:k=0,\dots ,N-1\}$, and $b_j\in \mathbb{C}$ for $j=1,\dots ,J$, which approximates $F^{♮}$ uniformly on compact sets in ${\mathbb{C}}^N$; that is,

$$F^{♮}\left(x\right)=\left(U_{g}F{U}_g^{\ast }\right)\left(x\right)\approx \mathsf{\Psi }\left(x\right)=\sum _{j=1}^J{c}_j\sigma (B_{j}x+b_j{\mathbb{1}}_N)x\in {\mathbb{C}}^N.$$

By the continuity of the operators $U_g^{\ast }$ and $U_g$, we obtain

$$F\left(x\right)=\left(U_g^{\ast }\left(U_{g}F{U}_g^{\ast }\right)U_g\right)\left(x\right)\approx \sum _{j=1}^J{c}_j{U}_g^{\ast }\sigma (B_j{U}_{g}x+b_j{\mathbb{1}}_N),x\in {\mathbb{C}}^N.$$

Note that since $\sigma \left(e^{\pi i/N}z\right)=e^{\pi i/N}\sigma \left(z\right)$ for all $z\in \mathbb{C}$, the function $\sigma :\mathbb{C}\to \mathbb{C}$ commutes with $U_g^{\ast }$ given by (15), that is, $U_g^{\ast }\sigma =\sigma U_g^{\ast }$. Therefore, we have

$$F\left(x\right)\approx \sum _{j=1}^J{c}_j\sigma (U_g^{\ast }{B}_j{U}_{g}x+b_j{U}_g^{\ast }{\mathbb{1}}_N)=\sum _{j=1}^J{c}_j\sigma (A_{j}x+b_{j}v),x\in {\mathbb{C}}^N,$$

where $A_j:=U_g^{\ast }{B}_j{U}_g\in \mathrm{span}\{\rho (k,\ell ):(k,\ell )\in \mathsf{\Lambda }\}$ by (16), and the vector $v:=U_g^{\ast }{\mathbb{1}}_N\in {\mathbb{C}}^N$ satisfies

$$\rho (k,\ell )v=\rho (k,\ell )U_g^{\ast }{\mathbb{1}}_N\stackrel{\left(4\right)}{=}{U}_g^{\ast }{L}_{\lambda }{\mathbb{1}}_N=U_g^{\ast }{\mathbb{1}}_N=v,(k,\ell )\in \mathsf{\Lambda }.$$

(17)

Finally, we note that for any $(k,\ell )\in \mathsf{\Lambda }$,

$$\begin{array}{cc}\hfill \rho (k,\ell )\left(\sum _{j=1}^J{c}_j\sigma (A_{j}x+b_{j}v)\right)& =\sum _{j=1}^J{c}_j\rho (k,\ell )\sigma (A_{j}x+b_{j}v)\hfill \\ & =\sum _{j=1}^J{c}_j\sigma \left(\rho (k,\ell )A_{j}x+b_j\rho (k,\ell )v\right)\hfill \\ & =\sum _{j=1}^J{c}_j\sigma \left(A_j\rho (k,\ell )x+b_{j}v\right),\hfill \end{array}$$

where we used that $\rho (k,\ell )$ is a linear (unitary) operator commuting with $\sigma$, and that $A_j\in C(\rho ,\mathsf{\Lambda })$ by (16) and $\rho (k,\ell )v=v$ by (17). Therefore, every map of the form $x\mapsto {\sum }_{j=1}^J{c}_j\sigma (A_{j}x+b_{j}v)$ is $(\rho ,\mathsf{\Lambda })$-equivariant. □

Remark 5.

The proof relies on observing (16) and choosing $g\in {\mathbb{C}}^N$ such that $U_g^{\ast }\sigma =\sigma U_g^{\ast }$. To obtain $U_g^{\ast }\sigma =\sigma U_g^{\ast }$, we have chosen $g\in {\mathbb{C}}^N$ so that $U_g^{\ast }$ is a diagonal matrix with exponential entries, and required an appropriate phase-homogeneity on σ so that σ commutes with those exponentials. This technique does not work for ${\mathsf{\Lambda }}_{\infty }$ because $U_g^{\ast }$ cannot be expressed as a diagonal matrix for any $g\in {\mathbb{C}}^N$ in that case.

Example 1.

Let $N=4$ and $s=1$, so that $\mathsf{\Lambda }={\mathsf{\Lambda }}_1=\{(0,0),(1,1),(2,2),(3,3)\}\subset {\mathbb{Z}}_4\times {\mathbb{Z}}_4$. In this case, we have $\omega =e^{2\pi i/4}=i$, ${\omega }_0=e^{\pi i/4}={\displaystyle \frac{1}{\sqrt{2}}}(1+i)$, and $\rho (k,\ell )={\omega }_0^{-k\ell }{M}^{\ell }{T}^k$. Then,

$$\begin{array}{cc}\hfill & \rho (0,0)=I_4,\hfill \\ \hfill & \rho (1,1)={\omega }_0^{-1}MT={\omega }_0^{-1}\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& i& 0& 0\\ 0& 0& -1& 0\\ 0& 0& 0& -i\end{array}\right]\left[\begin{array}{cccc}0& 0& 0& 1\\ 1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]={\omega }_0^{-1}\left[\begin{array}{cccc}0& 0& 0& 1\\ i& 0& 0& 0\\ 0& -1& 0& 0\\ 0& 0& -i& 0\end{array}\right],\hfill \\ \hfill & \rho (2,2)={\omega }_0^{-4}{M}^2{T}^2=-\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& -1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& -1\end{array}\right]\left[\begin{array}{cccc}0& 0& 1& 0\\ 0& 0& 0& 1\\ 1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right]=\left[\begin{array}{cccc}0& 0& -1& 0\\ 0& 0& 0& 1\\ -1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right],\hfill \\ \hfill & \rho (3,3)={\omega }_0^{-9}{M}^2{T}^2={\omega }_0^{-1}\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& -i& 0& 0\\ 0& 0& -1& 0\\ 0& 0& 0& i\end{array}\right]\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\\ 1& 0& 0& 0\end{array}\right]={\omega }_0^{-1}\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& -i& 0\\ 0& 0& 0& -1\\ i& 0& 0& 0\end{array}\right],\hfill \end{array}$$

and $\rho (k,k)\rho (k^{\prime },k^{\prime })=\rho (k+k^{\prime },k+k^{\prime })$ for all $k,k^{\prime }=0,1,2,3$. With $g={(1,0,0,0)}^{\mathrm{T}}$, we have

$$\begin{array}{cc}\hfill & U_g^{\ast }=\left[\begin{array}{cccc}\rho (0,0)g& \rho (1,1)g& \rho (2,2)g& \rho (3,3)g\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& {\omega }_0& 0& 0\\ 0& 0& -1& 0\\ 0& 0& 0& {\omega }_0\end{array}\right],\hfill \\ \hfill & v:=U_g^{\ast }{\mathbb{1}}_4=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& {\omega }_0& 0& 0\\ 0& 0& -1& 0\\ 0& 0& 0& {\omega }_0\end{array}\right]\left[\begin{array}{c}1\\ 1\\ 1\\ 1\end{array}\right]=\left[\begin{array}{c}1\\ {\omega }_0\\ -1\\ {\omega }_0\end{array}\right].\hfill \end{array}$$

It is easy to check that v is invariant under $\rho (0,0)$, $\rho (1,1)$, $\rho (2,2)$, $\rho (3,3)$; that is, $\rho (k,k)v=v$ for all $k=0,1,2,3$. Theorem 5 shows that any Λ-equivariant map $F:{\mathbb{C}}^4\to {\mathbb{C}}^4$ can be approximated (uniformly on compact sets) by functions of the form

$$\begin{array}{c}\hfill x\mapsto \sum _{m=1}^M{c}_m\sigma (A_{m}x+b_{m}v),\end{array}$$

where $A_m\in \mathrm{span}\{\rho (k,k):k=0,1,2,3\}$ and $b_m\in \mathbb{C}$ for $m=1,\dots ,M$. It is worth noting that while ρ is a unitary group representation of $\mathsf{\Lambda }=\left\{\right(0,0),(1,1),(2,2),(3,3\left)\right\}$ on ${\mathbb{C}}^4$, the map ${\pi |}_{\mathsf{\Lambda }}$ given by $\pi (k,\ell )=M^{\ell }{T}^k$ for $(k,\ell )\in \mathsf{\Lambda }$ is not a group representation of Λ on ${\mathbb{C}}^4$, since $\pi (1,1)\pi (1,1)=(-i)\pi (2,2)$ by (8).

4. Discussion

In this paper, we used finite-dimensional time-frequency analysis to investigate the properties of time-frequency shift equivariant maps that are generally nonlinear.

First, we established a one-to-one correspondence between $\mathsf{\Lambda }$-equivariant maps and certain phase-homogeneous functions, accompanied by a reconstruction formula expressing $\mathsf{\Lambda }$-equivariant maps in terms of these functions. This deepens our understanding of the structure of $\mathsf{\Lambda }$-equivariant maps by connecting them to their corresponding phase-homogeneous functions.

Next, we considered the approximation of $\mathsf{\Lambda }$-equivariant maps by neural networks. When $\mathsf{\Lambda }$ is a cyclic subgroup of order N in ${\mathbb{Z}}_N\times {\mathbb{Z}}_N$, we proved that every $\mathsf{\Lambda }$-equivariant map can be approximated by a shallow neural network with affine linear maps formed as linear combinations of time-frequency shifts by $\mathsf{\Lambda }$. For the subgroup $\mathsf{\Lambda }=\langle (1,0)\rangle =\left\{\right(0,0),(1,0),\dots ,(N-1,0\left)\right\}$, the $\mathsf{\Lambda }$-equivariance corresponds to translation equivariance, and our result shows that every translation equivariant map can be approximated by a shallow convolutional neural network, which aligns well with the established effectiveness of convolutional neural networks (CNNs) for applications involving translation equivariance. In this context, our result extends the approximation of translation equivariant maps to general $\mathsf{\Lambda }$-equivariant maps, with potential applications in signal processing.

Finally, we note that the tools used to prove the approximation result (Theorem 2) are applicable in a more general setting than the one described in Section 3.3. In particular, Definitions 6 and 7, and Proposition 4 apply to general unitary representations of arbitrary groups. Therefore, our approach can be adapted to derive similar results for general group-equivariant maps, which we leave as a direction for future research.