SymmetryLens: Unsupervised Symmetry Learning via Locality and Density Preservation

Abstract

We develop a new unsupervised symmetry learning method that starts with raw data and provides the minimal generator of an underlying Lie group of symmetries, together with a symmetry-equivariant representation of the data, which turns the hidden symmetry into an explicit one. The method is able to learn the pixel translation operator from a dataset with only an approximate translation symmetry and can learn quite different types of symmetries that are not apparent to the naked eye. The method is based on the formulation of an information-theoretic loss function that measures both the degree of symmetry of a dataset under a candidate symmetry generator and a proposed notion of locality of the samples, which is coupled to symmetry. We demonstrate that this coupling between symmetry and locality, together with an optimization technique developed for entropy estimation, results in a stable system that provides reproducible results.

1. Introduction

The spectacular success of convolutional neural networks (CNNs) has triggered a wide range of approaches to their generalization. CNNs are equivariant under pixel translations, which is also a property of the information content in natural image data. By using a weight-sharing scheme that respects such an underlying symmetry, it has been possible to introduce a powerful inductive bias in line with the nature of the data, resulting in highly accurate predictive models. How can this approach be generalized?

Translations form a group, and, for those cases where a more general underlying symmetry group is known, a mathematically elegant generalization can be developed in the form of group convolutional networks [1]. However, data often have underlying symmetries that are not known explicitly beforehand. In such cases, a method for discovering the underlying symmetry from data would be highly desirable.

Learning symmetries from data would allow one to develop an efficient weight-sharing scheme as in CNNs, but, even without this practical application, simply being able to discover unknown symmetries is of considerable interest in itself. In many fields, symmetries in data provide deep insights into the nature of the system that produces the data. For example, in physics, continuous symmetries are closely related to conservation laws, and knowledge of the symmetries of a mechanical system allows one to develop both analytical [2] and numerical [3] methods for investigating dynamics. More generally, the discovery of a new symmetry in a dataset would almost surely trigger research into the underlying mechanisms generating that symmetry.

In this paper, we develop a new method for discovering symmetries from raw data. For a dataset that has an underlying unknown symmetry (analogous to translation symmetry) provided by the action of a Lie group, the model provides a generator of the symmetry (analogous to pixel translations). In other words, we develop an unsupervised method, which has no other tasks such as classification or regression: the data are provided to the system without any labels of any sort, and out comes the generator of the symmetry group action. The setup also creates, as a byproduct, a symmetry-based representation of the data, whose importance is forcefully emphasized by Higgins et al. [4] and Anselmi and Patel [5]. This symmetry-based representation can also be used as an adapter between raw data and regular CNNs.

In the context of symmetry learning, one sometimes restricts attention to symmetry transformations that act on samples only through an action on their component indices. For instance, in the case of images, a translated image is obtained by shifting the pixel indices in an appropriate way, without otherwise transforming the data. While such transformations form an important class of symmetry actions, our setting is more general in that we consider a symmetry group of transformations acting directly on samples, and so-called “regular representations” (of which pixel translations are an example) are only a special case of our approach.

Another avenue of generalization relevant to our methodology is the notion of a local symmetry, as opposed to the more commonly encountered notion of global or “rigid” symmetries. The translation symmetry of the underlying distribution of an image dataset does not come from the fact that rigidly translated versions of images are commonly encountered; they are not. One sometimes synthetically creates such samples for purposes of data augmentation, but, in reality, multiple pictures of the same exact setting are rare. What does happen is that real images consist of local building blocks whose locations have a distribution that is approximately invariant under translations (see Figure 1).

Each image consists of the superposition of a number of such building blocks, and the approximate translation symmetry of image data is, in this sense, tied to an underlying local version of the translation symmetry. We ignore the dependence between the locations of different object types. The importance of being able to deal with such a flexible/local notion of symmetry has been emphasized by Bronstein et al. [6] and Anselmi et al. [7].

With this motivation, we seek to develop a symmetry learning methodology that finds transformations that respect the underlying distribution of a dataset of samples and respect a notion of locality that is intertwined with the symmetry.

Before providing an outline of our methodology, let us provide a quick overview of the capabilities of our system. Consider a dataset whose samples are superpositions of local basis signals with locations randomly selected from a uniform distribution over a finite range (see Figure 2). Each localized signal, with possibly a different width and height, is a local building block in the sense described above. The distribution of such samples is approximately invariant under translations (modulo edge effects). The question is, given such a dataset, without any other guidance (e.g., a supervised learning problem on this dataset with ground truth labels that are invariant under the translation symmetry), can one find that the operation of (one-dimensional) pixel translations is a symmetry of this dataset? Our model is, indeed, able to start with a dataset of this form and discover the matrix that represents pixel translations, sending each component of a sample to the next component.

Although this is not a straightforward task, rediscovering the familiar case of (one-dimensional) pixel translations from such a dataset is perhaps the first thing to demand of a method that claims to learn generalized symmetries from raw data in an unsupervised manner. However, our approach allows the system to discover symmetries that are much less obvious. Suppose we take a dataset that is again approximately invariant under pixel translations and apply a fixed permutation to the pixels to obtain a new dataset. The underlying symmetry of this new dataset will be much less obvious to the naked eye: the generator of the symmetry will now be an operator like Pixel 1→Pixel 17→Pixel 12, etc. We demonstrate that our method can discover this symmetry just as well as it discovers pixel translation. Moreover, the model learns a symmetry-based representation as a byproduct, and this representation unscrambles the pixel order to make the hidden symmetry manifest. While we find this highly encouraging, this, too, is a setting where the symmetry operation acts on the pixel indices.

As a final example, consider a symmetry action in the form of frequency shifts, in the sense that the dataset under consideration comprises time-series data that are approximately symmetric under the three-step action of (1) Fourier transformation (more precisely, we use discrete sine transformation I), (2) shifting all the frequency components by the same amount (i.e., a translation in frequency space), and (3) inverse Fourier transformation (see Figure 2). This action clearly is not obtained by simply shuffling around the component indices of the input vectors. Our method learns the matrix that represents the minimal generator of this action as well. In this case, the symmetry-based representation constructed by the model as a byproduct comes out to be almost exactly a discrete Fourier transform matrix. In other words, by simply “looking at” a dataset that has an approximately local symmetry under frequency shifts, the method learns that the appropriate symmetry-based representation for this dataset is a Fourier transform.

In the following sections, we provide the technical details of the method itself, but the fundamental intuition is rather simple, and we next provide a quick description.

Our approach is based on formulating two important properties to generate a symmetry group for a dataset (for the definition of a group in the context of symmetry learning, see, e.g., Helgason [8]), namely invariance and locality. An appropriate loss function that measures the degree to which these properties are satisfied enables us to find the underlying symmetries by gradient-based optimizers. While versions of invariance are often used in the symmetry learning literature, we believe that the inclusion of a locality property is what provides our method the power it has.

Invariance. The first natural characteristic of symmetry is the preservation of the underlying distribution. In other words, the original data and their transformed version under a symmetry map should have approximately the same distribution. In the case of images, this means that the joint distribution of all the pixel activations should be invariant under the global translation of pixels (ignoring boundary effects). For a general symmetry group action (beyond translations), such an invariance may not be obvious to the naked eye, but, once one knows the underlying symmetry, one should be able to confirm the behavior. (In the continuous case, one views data samples as sample functions or realizations of an underlying continuous stochastic process, and a stochastic process with the required invariance property is called a strongly stationary process under the action of the symmetry group. Apart from a mathematical Appendix A, Appendix B, Appendix C, Appendix D, Appendix E and Appendix F, in this paper, we restrict our attention to the discretized version of such a continuous underlying symmetry).

As emphasized by Desai et al. [9], the space of all the density-preserving maps [10] for a multidimensional dataset is large and includes maps that one would not normally want to call symmetries, so simply seeking density- (or distribution-) preserving maps will not necessarily allow one to discover symmetries. This property, nevertheless, is a reasonable characteristic to expect from a symmetry transformation. Implementing a loss function that measures the degree of density preservation will help us to achieve transformations with this property.

Locality. To identify the other fundamental property we will demand of a symmetry group action, we turn back to CNNs for inspiration. A CNN model not only respects the underlying translation symmetry but does so in a local way, in the sense that each filter has a limited spatial extent. This inductive bias, as well, corresponds to an underlying property of image data. As mentioned above, images contain representations of objects, which themselves have locality along the symmetry directions, and local filters do a good job of extracting such local information. While locality and symmetry are distinct properties, they are often coupled, and this coupling often hints at something fundamental about the processes generating the data. In physics, space and time translation symmetries and spacetime locality are intimately coupled, and this locality-symmetry is a fundamental property of quantum field theories describing the standard model of particle physics [11].

We choose to enforce a generalized notion of locality as the second fundamental property that the symmetry group should satisfy. The symmetry directions should be coupled to the locality directions in data. To make this proposal concrete, consider the action of a single local CNN filter. If we apply the same filter to an image and its slightly translated versions, we obtain various scalar representations of the image. These scalars will be strongly dependent random variables that contain similar information when the amount of translation is small, but, with larger translations, the dependence and similarity decrease.

We will require that a general unknown group of symmetries should satisfy a similar locality property. Once again, let us start with the assumption that we have a “local filter”, i.e., a scalar-valued map, this time local along the unknown symmetry direction. Applying this filter to a sample, we obtain a scalar. First transforming the sample by a symmetry transformation and then applying the filter, we obtain another scalar. Assume that the information in the samples is indeed local along the symmetry direction. Then, for “small” transformations, the two scalars should be strongly dependent random variables that encode similar information. For “larger” transformations, the similarity should decrease. A loss function that quantifies this behavior would thus help us to find symmetry groups that have the required coupling with the locality.

In addition to the problem of making the concepts in the scare quotes above precise (which we address in our method sections), this approach also faces an immediate chicken–egg problem: to conduct a search for a set of symmetry transformations in this way, we would first need a scalar representation, a filter, which itself is local along the symmetry direction. But, to find such a filter, we first need to know the set of symmetry transformations (under which the filter will be local).

We solve this problem by a sort of bootstrapping: we seek the local scalar representation and the symmetry group simultaneously and self-consistently using an appropriate loss function that applies to both. As described below, this coupled search actually works, finding the appropriate minimal generators of symmetry, together with a filter that behaves as a “delta function” in the direction of symmetry transformations. As a byproduct, one also obtains a group-equivariant symmetry-based representation of data, which makes the hidden symmetries explicit. See Figure 3 for a visual representation of the method.

From the outset, we emphasize that we do not assume the data space to be a so-called homogeneous space under the action of the Lie group of symmetries. More explicitly, we do not assume that the symmetry group is large enough to map any given point in the sample space to any other point. This would be a highly restrictive assumption, which is not, for example, satisfied by image data.

We find it satisfying that the approach described here actually works, but the details, of course, matter. The dynamics of the optimization of a loss function are complicated, and various seemingly similar implementations of the same intuition can result in widely varying results. The method we detail in the following sections is a specific implementation of the ideas of locality and invariance described above, and it results in a highly accurate and stable system that provides reproducible results.

In summary, the main contributions of the paper are the following:

• The formulation of a new approach to symmetry learning via maps that respect both symmetry and (a proposed notion of) locality, coupled in a natural way. The outputs of the method are both minimal generators of the symmetry and a symmetry-based data representation that, in a sense, makes the hidden symmetry manifest. This representation can be used as input for a regular CNN, allowing the model to work as an adapter between raw data and the whole machinery of CNNs.
• The formulation of an information-theoretic loss function encapsulating the formulated symmetry and locality properties.
• The development of optimization techniques (“time-dependent rank”, see Section 4.3.1) that result in highly robust and reproducible results.
• The demonstration of the symmetry recovery and symmetry-based representation capabilities on quite different sorts of examples, including 1D pixel translation symmetries, a shuffled version of pixel translations, and frequency shifts using dataset dimensionalities as high as 33 (i.e., the relevant symmetry generators with the shape $33\times 33$).

We note that, in the search for symmetry generators, there are no hard-coded simplifications such as sparsity in the symmetry generators or symmetry-based representations. The model indeed needs to learn the relevant matrices without utilizing an underlying simplifying assumption or factorization.

The rest of the paper is structured as follows. We will briefly summarize the current literature in Section 2, while Section 3 includes the theoretical setup motivating our approach. We describe the details of the method in Section 4, and we investigate the performance and robustness of the method in Section 5. Section 6 includes a summary of our contributions, future research directions, and limitations.

2. Related Work

Symmetry learning in neural networks is a rapidly evolving area of research, driven both by the value of symmetries for natural sciences and by the usefulness of symmetry-based inductive biases in model architectures. The current literature involves nuanced definitions for the symmetry learning problem as well as different symmetry learning schemes. The techniques used in symmetry learning can be roughly classified into supervised, self-supervised, and unsupervised learning approaches.

Supervised learning-based approaches combine a supervised problem with a search for symmetry maps that work in harmony with the supervised model in an appropriate sense. Benton et al. [12] use parametrized transformations of input data (such as rotations or affine transformations) for data augmentation and average the predictions over augmented samples to obtain a final prediction that is invariant under the chosen transformations. Using a loss function that encourages the exploration of a range of transformations, they are able to find, e.g., rotations of images as appropriate transformations. Romero and Lohit [13] start with a candidate symmetry group and focus on learning a subgroup of symmetry. Another supervised approach to symmetry learning is via meta-learning. Zhou et al. [14] form many supervised tasks tailored to the symmetry learning problem and use a weight-sharing architecture that is shared between tasks. A cascaded optimization tries to improve the supervised performance by updating both the matrix that determines the weight-sharing and the weights that are used in each task. This way, the model can discover convolutional weight-sharing from 1D translation-invariant data, which is impressive. However, the supervised tasks are a bit unnatural in that the synthetic data generation is conducted by using sampled filters, and the supervised task is to discover the ground truth filters. As symmetries are essential ingredients in physical theories, a wide range of approaches have been tried for symmetry learning in physics-adjacent settings. Craven et al. [15] train a neural network (NN) to approximate a function, and then candidate symmetries are tested by computing the values of the function on inputs transformed by the candidate symmetry and considering the “small transform” asymptotic behavior of the error of the NN. Forestano et al. [16] and Forestano et al. [17] start with specific quadratic forms used in the definition of various Lie groups (as the set of maps that preserve the given form) and then look for linear transformations near identity that leave the form invariant to recover the Lie algebra. Krippendorf and Syvaeri [18] set up a classification problem for predicting the value of a potential function with some underlying symmetries and then use the representations in the embedding layer to search for transformations that relate points that are close to each other in this layer.

Self-supervised approaches adopt auto-regressive setups for the symmetry learning problem. For datasets involving sequential frames, Sohl-Dickstein et al. [19] extract linear transformations relating subsequent time steps in a self-supervised manner, whereas, in the setup proposed by Dehmamy et al. [20], one has access to original data and a transformed version and aims to learn a Lie algebra generator that would relate the two by an approximate group action. In the setting of dynamical systems, Greydanus et al. [21] focus on learning a Hamiltonian, whereas Alet et al. [22] learn conserved quantities, which are both related to underlying symmetries. The self-supervised approach is most useful and interesting in the setting of dynamics of systems, but this also limits its applicability. However, within this setting, the lack of a need for labeled data is an advantage.

Unsupervised approaches to symmetry discovery work with a raw dataset and aim to output maps that represent symmetries of the dataset. A prominent line of research in this setting is based on Generative Adversarial Networks (GANs) [23]. Desai et al. [9] propose using a generator applying candidate symmetry transformations from a parametrized set, such as affine transformations, and a discriminator aiming to distinguish real samples from the transformed versions. The training converges to parameter values that represent symmetries of the dataset, but this approach neither provides the generator of the relevant transformation nor works with a high-dimensional setting without a low-dimensional parametrization for the set of candidate transformations. Yang et al. [24] propose enhancements such as regularization terms preventing identity collapse and encouraging the discovery of multiple Lie algebra generators and sampling the exponential coefficients of the generators from distributions to enable the discovery of subgroups. The main setup is similar in that only low-dimensional parametrized generators are discovered. Yang et al. [25] extend this idea to learn nonlinear group actions by learning a representation where the group action becomes linear simultaneously with the group generator. Except for the autoencoder used for the representation, the setup is similar to the setup proposed by Yang et al. [24]. Tombs and Lester [26] propose an approach that is similar in spirit to the GAN-based symmetry learning, where candidate symmetries are not generated by the model but are provided externally, and a model is trained to discriminate between real and transformed samples. The idea is that, if the model assigns similar probabilities for the sample being real or transformed, then one concludes that the candidate transformation is indeed a symmetry.

Instead of learning symmetry generators explicitly, one can search for a symmetry-equivariant representation directly. Anselmi et al. [7] propose an unsupervised approach for learning such equivariant representations for the case of permutation groups. They demonstrate that the proposed approach can learn equivariant representations for various subgroups of the symmetric group of six objects, acting on a six-dimensional dataset via component permutations. The generated synthetic datasets are completely explicitly symmetric (created by the action of the full group on an initial set of samples).

Among the approaches described above, the requirements for labeled datasets or structured data representations limit the applicability of supervised and self-supervised symmetry learning paradigms. Unsupervised methods offer the widest applicability in principle, but they are demonstrated to work for only up to four-dimensional irreducible representations (and our experiments reported below imply that they perform poorly for higher dimensions). Real-world datasets often involve much higher dimensionalities, which state-of-the-art unsupervised methods cannot deal with.

In this study, we formulate the symmetry learning problem in a representation learning framework where the symmetries are learned via a group convolution map. This makes the symmetry manifest, turning it into a simple translation symmetry, which we believe is unique in the symmetry learning literature. Our setup is able to accurately extract symmetries from datasets with dimensionalities as high as 33 without any hard-wired factorizations, which is by far the highest dimensionality we have seen in the unsupervised approach to symmetry learning. Our approach based on coupling locality and invariance also allows the method to consistently learn the minimal generators, thus enabling one to construct the full symmetry group.

3. Theoretical Setting and Data Model

In this section, we describe a simple setting for data-generating mechanisms that have the symmetry and locality properties described in our introductory discussion. This will both motivate our symmetry learning approach and guide the data generation for our experiments. Readers primarily interested in the description of the method and the results can skip this section in a first reading and peruse Appendix C.1 for information on our datasets.

Our aim is to use the spatial information locality and the translation symmetry of images for inspiration to describe the more general setting of symmetry and locality under a different group of transformations. We first consider the ideal case of continuous data (corresponding to images with “infinite resolution”) and then turn to a discretized version (corresponding to pixelated images). The relevant translation symmetry group for image data is the group of 2-dimensional translations, but we focus on the simpler case of 1-dimensional translations appropriate for “1-dimensional images” (or, more familiarly, time-series data).

3.1. The Continuous Setting

3.1.1. Introduction

A group is a set with an associative binary operation that has an identity element and an inverse for each element. The set of invertible transformations for a wide class of mathematical objects is described under the setting of group theory. The group of 1-dimensional translations is isomorphic to the additive group $\mathbb{R}$, each number representing the translation amount and the identity element corresponding to a translation by 0.

As mentioned in our Introduction in Section 1, images consist of building blocks that are spatially local, and the approximate translation symmetry of an image dataset is borne out of the random distribution of the locations of the building blocks (see Figure 1). In order to formalize the case of 1-dimensional translations, we first model the data-generating mechanisms for “1-dimensional images” as a simple class of stochastic processes and then move on to more general symmetries.

3.1.2. Processes with Symmetry and Locality Under 1-Dimensional Translations

We will think of each 1-dimensional image as a sample function (or a realization) of an underlying stochastic process. Intuitively, a real-valued 1-dimensional stochastic process is a “machine” that provides us a random function $f:\mathbb{R}\to \mathbb{R}$ each time we press a button, using an underlying distribution. A translation-invariant (or stationary) process is one where a sample function $f\left(t\right)$ and all its translated versions $f(t-\tau )$, $\tau \in \mathbb{R}$ are “equally likely”. Of course, it does not make sense to talk about the probability of a single sample function, and rigorously defining probability measures on function spaces is rather tricky. However, the intuitive picture mentioned provides a useful viewpoint that we will utilize below. (In the literature on stochastic processes, a stochastic process is more properly defined in terms of a family of maps from a probability space, and stationarity is commonly defined in terms of the joint distributions of function values $f\left(t_j\right)$ on finite sets ${\left\{t_j\right\}}_{j=1}^N$. For a formal definition of stationary processes, see Doob [27]).

To create a mathematical model for a translation-invariant process with local building blocks, consider a set ${\left\{{\varphi }_i\right\}}_{i\in I}$ of functions that are localized (compactly supported). We would like to think of each ${\varphi }_i$ as representing a type of object. If we shift each ${\varphi }_i$ by an amount ${\tau }_i$ and add up the resulting functions, we obtain a signal $f\left(t\right)$ consisting of local building blocks at various locations, $f\left(t\right)={\sum }_i{\varphi }_i(t-{\tau }_i)$. If we could select each $\tau$ randomly using a uniform distribution, the process of creating such $f\left(t\right)$ would result in a stationary stochastic process.

While it is not possible to have a uniform probability distribution over the whole real line, this is a technical difficulty that can be overcome by, e.g., considering a stochastic process on a “large” circle instead of $\mathbb{R}$ (or using a stationary point process such as the Poisson process to obtain a collection of centers ${\tau }_{ik}$ for each ${\varphi }_i$ with a translation-invariant distribution on $\mathbb{R}$ rather than a single center).

More generally, each ${\varphi }_i$ could have additional parameters ${\lambda }_{ik}$ such as width or amplitude, which one could independently sample from their own distributions. In short, starting with a set of basis signals (“objects types”) ${\varphi }_i$ and sampling the centers and other parameters in an appropriate way obtains a stochastic process that is both symmetric (uniform distribution of centers) and local under translations. Sample functions of such a process are given by

$$f\left(t\right)=\sum _i\left[\sum _k{\varphi }_i(t-{\tau }_{ik};{\lambda }_{ik})\right].$$

(1)

To complete the analogy with (finite-size) images, one can crop each such sample $f\left(t\right)$ to a finite interval. (In real images, opaque objects actually hide each other rather than combining in an additive way, but we ignore such considerations in this simple setting). See the left-hand side of Figure 4 for samples of the kind described here.

3.1.3. Processes with Symmetry and Locality Under a General 1-Dimensional Group Action

We would like to generalize the simple setting above to more general group actions. We will describe an abstract setting first and will then provide concrete examples.

Consider stochastic processes defined on a space $\mathcal{X}$, with the space of sample functions x denoted by $\mathcal{F}\left(\mathcal{X}\right)=\{x:\mathcal{X}\to \mathbb{R}\}$. Suppose we have an Abelian group G (group operation written as +) with a given group action $\rho$ on $\mathcal{F}\left(\mathcal{X}\right)$. In other words, for a sample function $x\in \mathcal{F}\left(\mathcal{X}\right)$ and group element $\tau \in G$, the result of the action $\rho \left(\tau \right)·x$ is another sample function, and $\rho$ satisfies $\rho \left(0\right)=0$, $\rho \left({\tau }_1\right)·\rho \left({\tau }_2\right)=\rho ({\tau }_1+{\tau }_2)$, $\rho (-\tau )={\left[\rho \left(\tau \right)\right]}^{-1}$. We will specialize to the case $\mathcal{X}=\mathbb{R}$ and G isomorphic to $\mathbb{R}$, but much of what we write will apply to more general Abelian Lie groups. We will keep $\rho$ unspecified; importantly, we do not assume that $\rho$ is given by the “regular representation”. In particular, we do not assume $\rho$ acts by simple translations as in $\left(\rho \right(\tau )·x)\left(t\right)=x(t-\tau )$.

Given such an action $\rho$, we would like to describe a process that generates sample functions with invariance and locality properties as above, but, this time, the locality and invariance will be under $\rho$ instead of translations. Taking $\rho$ to be a simple translation for the moment, the following representation of the signal $f\left(t\right)$ of (1) motivates our generalization:

$$\begin{array}{cc}\hfill f\left(t\right)& =\int \delta (t-\tau )f\left(\tau \right)d\tau \hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & =\int (\rho \left(\tau \right)·{\delta }^{\left(\rho \right)})\left(t\right)f\left(\tau \right)d\tau .\hfill \end{array}$$

(3)

Here, in (2), $\delta \left(t\right)$ denotes the usual Dirac delta function, and, in (3), ${\delta }^{\left(\rho \right)}\left(t\right)$ is just another name for it, the notation suggesting that this delta function is “a delta function along the symmetry action $\rho$ of simple translations”. The action $\rho \left(\tau \right)·{\delta }^{\left(\rho \right)}$ by the translation operator $\rho \left(\tau \right)$ provides the shifted delta, $\delta (t-\tau )$.

If we now view the right-hand side of (3) as applying to a general $\rho$ instead of only translations, we obtain the generalization we want. For a given group action $\rho$ for the group G isomorphic to $\mathbb{R}$, assuming one has an appropriate notion of a “delta function ${\delta }^{\left(\rho \right)}$ along $\rho$”, one can create a process that is invariant (stationary) and local under the group action $\rho$ from a process that is invariant and local in the usual sense on $\mathbb{R}$. One obtains samples $f\left(t\right)$ from a process such as (1) on G and transforms each sample to a sample $x\left(t\right)$ on $\mathcal{X}=\mathbb{R}$ via

$$x\left(t\right)=\int (\rho \left(\tau \right)·{\delta }^{\left(\rho \right)})\left(t\right)f\left(\tau \right)d\tau .$$

(4)

Here, $\rho$ no longer represents simple translations, and ${\delta }^{\left(\rho \right)}$ is no longer the usual delta function but is an appropriate delta “function” (or measure, or distribution) defined on $\mathcal{X}$. (Here, the integration measure $d\tau$ on the right-hand side should properly be viewed as the Haar measure on the relevant Lie group, but, in this 1-dimensional setting, we do not lose much by treating it as the Lebesgue measure on $\mathbb{R}$; for higher-dimensional, possibly non-Abelian Lie groups, one will have to be more careful). While we specialize to $\mathcal{X}=\mathbb{R}$ here (and mention a generalization to ${\mathbb{R}}^D$ below), the formalism suggests greater generality.

To provide another concrete example of this abstract setting (in addition to translations), consider the group action of $G\cong \mathbb{R}$ on sample functions given by frequency shifts instead of translations. In other words, let $\rho \left(s\right)$ act on a sample function $x\left(t\right)$ via

$$(\rho \left(s\right)·x)\left(t\right)=e^{-ist}x\left(t\right)$$

(5)

which corresponds to a shift in Fourier space (for simplicity, we take the sample functions in this case to be complex-valued). For this action, an appropriate delta function along $\rho$ is a Fourier basis function ${\delta }_{k_0}^{\left(\rho \right)}\left(t\right)=e^{i{k}_{0}t}$, where $k_0\in \mathbb{R}$. Just as the regular Dirac delta function is sharply localized under translations in the sense that $\delta (t-t_0)$ and $\delta (t-t_0-\tau )$ have zero overlap, the functions ${\delta }_{k_0}^{\left(\rho \right)}\left(x\right)=e^{i{k}_{0}t}$ and $(\rho \left(s\right)·{\delta }_{k_0}^{\left(\rho \right)})\left(t\right)=e^{i(k_0-s)t}$ have zero overlap (e.g., these functions are orthogonal by Fourier analysis). Specializing to the case $k_0=0$, we can construct signals that are local along the action of $\rho$ by using (2) from a signal $f\left(t\right)$ as in (1) defined on the group $G=\mathbb{R}$:

$$x\left(t\right):=\int (\rho \left(s\right)·{\delta }_{\rho })\left(t\right)f\left(s\right)ds=\int e^{-ist}f\left(s\right)ds.$$

(6)

Thus, we obtain the pleasing result that, according to this setup, a signal that is local along the “frequency shift” action $\rho$ of the 1-dimensional group $G\cong \mathbb{R}$ is given by the Fourier transform of a signal ${\mathsf{\Phi }}_0\left(s\right)$ that is local under translations (of s). In other words, to obtain a stochastic process that is invariant and local under frequency shifts, we can start with a spatially local and spatially stationary process on the group $\mathbb{R}$ and take the Fourier transform of each sample.

We will leave an attempt at a rigorous description of the fully general version of this setup for future work and note that, in the general case, the delta function ${\delta }^{\left(\rho \right)}$ on X is closely related to what is called an “approximate identity” in the abstract harmonic analysis literature (see Folland [28] for an introduction). In particular, such a delta will be required to satisfy

$$(\rho \left(s\right)·{\delta }^{\left(\rho \right)})\ast (\rho \left(t\right)·{\delta }^{\left(\rho \right)})=\delta (s-t)$$

(7)

where ∗ denotes the appropriate convolution on $\mathcal{X}$, and the $\delta$ on the RHS is the Dirac delta function on the group $G\cong \mathbb{R}$. This means, in particular, that separately transformed versions of ${\delta }^{\left(\rho \right)}\left(x\right)$ have zero overlap unless the transform parameters are the same.

3.2. The Discrete Setting: Synthetic Data Generation

3.2.1. Discrete Translation Symmetry

To create discrete signals that are local and symmetric under 1-dimensional discrete translations, we follow the procedure of creating a sample out of local basis signals as in (1) and replace t with a discrete index n. The local basis signals ${\varphi }_i\left(t\right)$ and sample functions $x\left(t\right)$ become vectors with components ${\varphi }_{in}$ and $x_n$, respectively.

In principle, the component index n ranges over all integers, but, for data generation purposes, it is restricted to a finite range. The basis signals ${\varphi }_{in}$ we use are Gaussians (parametrized by width and amplitude) and Legendre functions (parametrized by width, amplitude, and order), with the index of the center being sampled uniformly over a finite range. The details of the process are given in Appendix C.1, but, as a quick summary, we use finite-dimensional vectors for each sample vector and use uniform distributions for centers, widths, and amplitudes of local basis signals, including superposition of a few such signals per sample. Finally, we add Gaussian noise on top of each sample for more realistic experimentation. Examples of the resulting samples can be seen in Figure 4 under the title of “Translation”.

3.2.2. General Representations: Behavior Under Orthogonal/Unitary Transformations

Starting with a dataset that is symmetric and local under discrete translations, we can create datasets that are symmetric/local under different group representations by using a discrete version of the transformation (4). Here, we start by proving a transformation property of discrete versions of the symmetry generators and the “Dirac delta along group action” described above and then provide the procedures used in generating synthetic data.

The discrete version $\mathbb{Z}$ of the translation group $G\cong \mathbb{R}$ has a minimal generator that is the pixel (or component) translation operator, which we denote by $1\in \mathbb{Z}$. Similarly, for any representation of the 1-dimensional group $G\cong \mathbb{R}$, there will be a matrix for the minimal generator $\rho \left(1\right)$, which we call $\mathfrak{G}$. With this notation, the discrete version of (4) is

$$\begin{array}{c}\hfill x_n=\sum _p\sum _j{\mathfrak{G}}_{nj}^p{\mathsf{\delta }}_j^{\left(\rho \right)}{f}_p.\end{array}$$

(8)

For samples $f_m$ ($m\in \mathbb{Z}$) with a distribution local and invariant under component shifts, this provides a sample distribution for $x_n$ that is local and invariant under the action of the generator $\mathfrak{G}$.

Now, let us consider the action of an invertible matrix Q on both sides. We obtain

$$\begin{array}{cc}\hfill \sum _n{Q}_{mn}{x}_n& =\sum _{m,j,p}{Q}_{mn}{\mathfrak{G}}_{nj}^p{\mathsf{\delta }}_j^{\left(\rho \right)}{f}_p=\sum _{m,k,p,k,l}{Q}_{mn}{\mathfrak{G}}_{nj}^p{Q}_{jk}^{-1}{Q}_{kl}{\mathsf{\delta }}_l^{\left(\rho \right)}{f}_p\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill x_n^{\prime }& =\sum _l{\mathfrak{G}}_{nl}^{\prime p}{\mathsf{\delta }}_l^{\left({\rho }^{\prime }\right)}{f}_p\hfill \end{array}$$

(10)

where primes denote the transformed version of the objects under Q. Thus, the transformed signal ${\mathbf{x}}^{\prime }$ has the symmetry/locality generator ${\mathfrak{G}}^{\prime }$ given in terms of the old one via a similarity transformation ${\mathfrak{G}}^{\prime }=Q·\mathfrak{G}·Q^{-1}$, and the new “delta along the symmetry” ${\mathsf{\delta }}^{\left({\rho }^{\prime }\right)}$ is given in terms of the old one via ${\mathsf{\delta }}^{\left({\rho }^{\prime }\right)}=Q·{\mathsf{\delta }}^{\left(\rho \right)}$. In other words, starting with symmetry/locality under a given group representation $\rho$ with generator $\mathfrak{G}$, we obtain symmetry/locality under ${\rho }^{\prime }$ with generator ${\mathfrak{G}}^{\prime }$ by using a transformation. If the transformation matrix Q is orthogonal (resp. unitary), then the new generator ${\mathfrak{G}}^{\prime }$ will also be orthogonal (resp. unitary) assuming the old one $\mathfrak{G}$ is so.

To create synthetic data in the case of 1-dimensional symmetry groups described above, we consider three symmetry group actions in our experiments: translations, permuted translations, and frequency shifts. Data samples for each symmetry can be found in Figure 4. In each case, we start with a 33-dimensional dataset that is symmetric/local under component translations $\mathfrak{G}=T$ with ${\left(T\mathbf{x}\right)}_i=x_{i-1}$ (ignoring edge effects; one could use cyclic translations to get rid of them) and apply an appropriate Q. For translations, we take $Q=I$. For frequency shifts, we take $Q=D^{-1}$, where D is the discrete sine transform matrix of type I. This results in the generator $\mathfrak{G}=D^{-1}·T·D$, which shifts the frequency components of the samples. For permuted translations, take $Q=P$, where P is a permutation matrix. Applying P to each raw sample, we obtain samples that are symmetric and local under the powers of the permutation operator $\mathfrak{G}=P·T·P^{-1}$. As can be seen in Figure 4, the symmetry/locality for the permutation and frequency shift cases is not apparent to the naked eye at all.

4. Materials and Methods

In this section, we formulate the learning problem and the loss function and discuss the training loop together with some useful optimization techniques. See Appendix F for information on the computational complexity of our method.

4.1. The Setup: Objects to Learn

Our model will start with a dataset $\mathcal{D}={\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$, where $\mathbf{x}\in {\mathbb{R}}^d$ and n is the number of samples, and will learn an underlying symmetry group representation $\rho \left(s\right)$ and a filter $\mathit{\psi }$ that is local along the symmetry action in the sense suggested in our Introduction, which will be conducted precisely by the loss function below. An appropriate combination of these building blocks will provide a symmetry-based representation given by a matrix L, which we call the group convolution matrix.

4.1.1. Symmetry Generator

We will assume that the symmetry action $\rho \left(s\right)$ is a real unitary representation; in other words, for each s, $\rho \left(s\right)$ will be a $d\times d$ orthogonal matrix, $\rho \left(s\right)\in O\left(d\right)$. We will focus on symmetry actions that can be continuously connected to the identity, which restricts the actions of group elements to those with determinant 1, i.e., elements of $SO\left(d\right)$. While the underlying symmetry action $\rho \left(s\right)$ will be the representation of a 1-dimensional Lie group parametrized by the continuous parameter s, the model will learn a minimal discrete generator $\mathfrak{G}$ of this action appropriate for the dataset at hand. With an appropriate choice of the scale of the s parameter that parametrizes the group, we can write $\mathfrak{G}=\rho (s=1)$, which provides (for any integer s) $\rho \left(s\right)=\rho {\left(1\right)}^s={\mathfrak{G}}^s$. This will allow us to obtain the action of any group element by taking an appropriate power of the generator.

Any element of $SO\left(d\right)$ can be obtained by using the exponential map on an element of the Lie algebra of $SO\left(d\right)$, which is the algebra of $d\times d$ antisymmetric matrices [8]. In particular, our generator $\mathfrak{G}$ should be given as the matrix exponential of an antisymmetric matrix. Instead of explicitly constraining $\mathfrak{G}$ to be an element of $SO\left(d\right)$, we write this matrix in terms of an arbitrary $d\times d$ matrix A via $\mathfrak{G}=exp\left({\displaystyle \frac{A-A^T}{2}}\right)$, where the matrix exponential can be defined in terms of the Taylor series. The optimizer will thus seek an appropriate A instead of a direct search for $\mathfrak{G}$. (To optimize memory usage, one could use a more efficient parametrization of antisymmetric matrices).

4.1.2. The Resolving Filter

We will assume that the resolving filter $\mathit{\psi }$ acts by an inner product, i.e., $\mathit{\psi }$ itself will be given as a d-dimensional (column) vector. This will be closely related to the discrete analog of the delta function ${\delta }^{\left(\rho \right)}$ described in Section 3. We do not use any constraints on this d-dimensional vector; however, we use a normalized version of it in our computations of the relevant dot products.

4.1.3. The Group Convolution Matrix

Following the approach sketched in the Introduction, we will formulate the learning problem in terms of a symmetry-based representation $\mathbf{y}\in {\mathbb{R}}^d$ of the data $\mathbf{x}\in {\mathbb{R}}^d$ given in terms of the generator $\mathfrak{G}$ and the filter $\mathit{\psi }$. For each d-dimensional sample vector $\mathbf{x}$, the components $y_p$ of the symmetry-based representation $\mathbf{y}$ will consist of the application of the filter $\mathit{\psi }$ to transformed versions ${\mathfrak{G}}^p\mathbf{x}$ of the data, $y_p={\sum }_{ij}{\mathit{\psi }}_i{\left({\mathfrak{G}}^T\right)}_{ij}^p{x}_j$, where $p=0,\pm 1,\pm 2,\dots$. Note that this is completely analogous to a CNN, in which case the generator $\mathfrak{G}$ is a pixel translation operator and $\mathit{\psi }$ is a local CNN filter.

These scalar representations $y_p$ of a sample will be the fundamental quantities on which we define the loss function measuring departures from stationarity and locality. In the $\mathbf{y}$ representation, the action of the generator $\mathfrak{G}$ is represented by a simple shift in p, and thus, if $\mathfrak{G}$ is indeed a symmetry generator, the joint distribution of the scalars $y_p$ should be invariant under shifts of p: $p\mapsto p+n$, which is a simple translation symmetry. Similarly, if data are local along the action of $\mathfrak{G}$ and $\mathit{\psi }$ is indeed local along the symmetry direction, $y_p$ and $y_{p^{\prime }}$ should be similar/strongly dependent as random variables if p and $p^{\prime }$ are close and should be approximately independent otherwise (locality). Thus, if the distribution of $\mathbf{x}$ is invariant and local under actions of ${\mathfrak{G}}^s$, the distribution of $\mathbf{y}$ should be invariant and local under simple shifts in the components $y_p$.

While there is no a priori restriction on the range of powers p to use in the representation, in this paper, we choose the dimensionality of the representation $\mathbf{y}$ to be the same as the dimensionality of $\mathbf{x}$. (Our methodology does not rely on this choice, and one could easily consider other ranges for p).

To obtain $\mathbf{y}$ from $\mathbf{x}$ directly via $\mathbf{y}=L\mathbf{x}$, we form the matrix L, which we call the group convolution matrix:

$$L=\left[\begin{array}{c}—{\mathit{\psi }}^T{\left({\mathfrak{G}}^{P_{min}}\right)}^T—\\ —{\mathit{\psi }}^T{\left({\mathfrak{G}}^{P_{min}+1}\right)}^T—\\ ·\\ ·\\ ·\\ —{\mathit{\psi }}^T{\left({\mathfrak{G}}^{P_{max}}\right)}^T—\end{array}\right]$$

(11)

where $P_{min}$ and $P_{max}$ denote the minimum and maximum powers of the generator to be used. In our experiments, we work with odd d and pick $P_{max}={\displaystyle \frac{d-1}{2}}=-P_{min}$.

4.2. The Loss Function

Our loss function will be computed from the transformed version $\mathbf{y}=L\mathbf{x}$ of each batch of vectors $\mathbf{x}$ and will measure the degree to which the distribution of this transformed version is symmetric and local under component translations $y_p\mapsto y_{p+s}$. The loss consists of three building blocks we call stationarity, locality, and information preservation. These are given in terms of the correlation between the components $y_p$ of the $\mathbf{y}$ representation, as well as various entropy and probability density terms for these components. We first describe the pieces of the loss function assuming one has access to estimates of these necessary quantities and will then explain the techniques used for estimating these quantities for each batch.

Note: For notational simplicity below, we use a non-negative indexing for the components $y_p$ of $\mathbf{y}$, i.e., $p=0,1,\dots ,d-1$, and use the notation ${\langle \cdots \rangle }_{\left[\mathrm{conditions}\right]}$ for the average of the expression ⋯ in the angle brackets over index combinations satisfying the conditions.

4.2.1. Stationarity/Uniformity

For true symmetry, we expect the joint probability distribution of the components of $\mathbf{Y}$ to be invariant under simple shifts in components (modulo boundary effects). Since estimating the joint probability density is impractical in high dimensions, we instead use the distributions of individual components and conditional distributions for pairs of components.

We denote the marginal probability density of the random variable $Y_j$ as $p_j$ and the conditional probability density of $y_j$ given $y_i$ by $p_{j|i}$. Our uniformity loss, in terms of these quantities, is given by

$${\mathcal{L}}_{uniformity}={\displaystyle \frac{1}{2}}\left[{〈{\widehat{D}}_{KL}(p_m,p_n)〉}_{\begin{array}{c}m=n\mp 1\end{array}}+{〈{\widehat{D}}_{KL}(p_{i|j},p_{k|l})〉}_{\begin{array}{c}k=i+\Delta \\ l=j+\Delta \\ \Delta =\mp 1\end{array}}\right]$$

(12)

where ${\widehat{D}}_{KL}$ denotes the estimate of the KL divergence (see Appendix A.3 for the estimation procedure). Averages are taken over all possible $m,n$, and $i,j,k,l$ indices satisfying the specified constraints. The marginal probability term provides a first-order proxy for uniformity, and the conditional probability term takes relations between pairs of random variables into account. Overall, this loss function provides a second-order computationally tractable measure of the uniformity of the full distribution under shifts.

4.2.2. Locality

Our locality loss consists of two pieces that we call alignment and resolution. Alignment loss aims to make neighboring pixels have similar information, and resolution aims to make faraway pixels have distinct information.

Alignment We would like successive components of symmetry-based representation to not only have similar distributions (as in the uniformity loss) but to have similar values in each sample. We compute the sample Pearson correlation coefficient ${\rho }_{i,i+1}\in \mathbb{R}$ of the components $y_i$ and $y_{i+1}$ of symmetry-based representation $\mathbf{y}$ for each batch and define the alignment loss ${\mathcal{L}}_{alignment}$ as the average of this quantity over all successive pairs

$${\mathcal{L}}_{alignment}=-{〈{\rho }_{i,i+1}〉}_{i\in [0,d-1)}.$$

(13)

Resolution We would like the components $y_i$ and $y_j$ of $\mathbf{y}$ to contain distinct information when i and j are not close. In the information theory literature, the quantity

$$\begin{array}{c}\hfill C\left(\mathbf{Y}\right)=\sum _{i=0}^{d-1}h\left(Y_i\right)-h\left(\mathbf{Y}\right)\end{array}$$

(14)

called the total correlation measures the degree to which there is shared information between the components of random vector $\mathbf{Y}$; ($h\left(Z\right)$ denotes the differential entropy of the random variable Z). For a local symmetry, only nearby components should share information, and all other pairs of components should be approximately independent, so we expect $C\left(\mathbf{Y}\right)$ to be small and use an approximation to $C\left(\mathbf{Y}\right)$ over each batch as our resolution loss.

As we describe in Section 4.3.1, we compute entropy estimates by using an epoch-dependent rank k. In effect, this increases the number of variables that comprise the entropy estimate from $k=1$ to $k=d$ over the course of the estimation and thus changes the overall scale of the estimated quantity accordingly. To use a normalized scale, we define the resolution loss for each batch to be

$$\begin{array}{c}\hfill {\mathcal{L}}_{resolution}={\displaystyle \frac{1}{d}}\sum _{i=0}^{d-1}h\left(Y_i\right)-{\overline{h}}_k\left(\mathbf{Y}\right)\end{array}$$

(15)

where ${\overline{h}}_k\left(\mathbf{Y}\right)$ is an estimate of the “per rank” contribution to entropy (see Appendix A.2 for exact formula).

4.2.3. Information Preservation

While the correct directions in parameter space reduce the locality and uniformity loss terms described above, there is also a catastrophic solution for L that can minimize these terms: a constant map to a single point. Indeed, we would like to learn a transformation that preserves the information content of the data, and we add a final term to maximize the mutual information between the random vectors corresponding to input $\mathbf{X}$ and the output $\mathbf{Y}$. For a deterministic map, this mutual information can be maximized by simply maximizing the entropy of the output (see the InfoMax principle proposed by Bell and Sejnowski [29]).

As mentioned above, our entropy estimate will use an epoch-dependent rank k, and we use a normalized per-rank version ${\overline{h}}_k\left(\mathbf{Y}\right)$ (see Appendix A.2) of the output entropy as our information preservation loss:

$$\begin{array}{c}\hfill L_{preservation}=-{\overline{h}}_k\left(\mathbf{Y}\right).\end{array}$$

(16)

4.2.4. Total Loss

Merging the overall loss terms, we have

$$\begin{array}{c}\hfill \mathcal{L}={\mathcal{L}}_{alignment}+\alpha {\mathcal{L}}_{resolution}+\beta {\mathcal{L}}_{uniformity}+\gamma {\mathcal{L}}_{preservation}\end{array}$$

(17)

while $\alpha ,\beta ,\gamma \in \mathbb{R}$. Limited experimentation with only one dataset was able to provide a choice for these hyperparameters that ended up working well for all our experiments, making it unnecessary to perform separate tunings for datasets with different dimensionalities and symmetry properties. See Appendix E for a discussion of hyperparameter choice and sensitivity and ablation experiments confirming that all pieces of this loss function indeed contribute to model performance.

4.2.5. On the Coupled Effect of the Alignment and Resolution Terms

In this section, we argue that the combined effect of the alignment and resolution terms in the loss function push the symmetry-based (output) representation towards a (two-sided) Markov process.

Let $\mathbf{Y}$ denote the d-dimensional output representation learned by the model. Using the chain rule for entropy, the joint entropy of $\mathbf{Y}$ can be written as $h\left(\mathbf{Y}\right)={\sum }_{i=1}^{d}h\left(Y_i\right|Y_{<i})$, where ${\mathit{Y}}_{<i}$ denotes the components $\{{\mathit{Y}}_1,{\mathit{Y}}_2,\cdots {\mathit{Y}}_{i-1}\}$ when $i>1$, and, for $i=1$, it simply means no conditioning is applied (in this section, we use non-negative integers for indexing components). We define the following mth-order approximation to the chain rule formula, which only uses local dependencies of window size m:

$$\begin{array}{c}\hfill h^{\left(m\right)}\left(\mathbf{Y}\right)=\sum _{i}h\left(Y_i\right|Y_{[i-1,i-m]})\end{array}$$

(18)

where $Y_{[i-1,i-m]}$ denotes the random variables ${\left\{Y_j\right\}}_{i-m\le j\le m-1}$, and $m=0$ represents the case with no conditioning.

Since conditioning can only reduce entropy [30], we have

$$\begin{array}{c}\hfill h\left(\mathbf{Y}\right)\le h^{(d-1)}\left(\mathbf{Y}\right)\le \cdots \le h^{\left(1\right)}\left(\mathbf{Y}\right)\le h^{\left(0\right)}\left(\mathbf{Y}\right).\end{array}$$

(19)

The “resolution” (total correlation) part of the loss $C\left(\mathbf{Y}\right)=h^{\left(0\right)}\left(\mathbf{Y}\right)-h\left(\mathbf{Y}\right)$ is minimized when all the conditional entropies in this chain are equal, which happens when the distribution of the variables is just the product distribution, i.e., when the $Y_i$ are all independent. However, the “alignment” loss (see Section 4.2.2) aims to maximize the correlation between successive components. Since independent variables have zero correlation, the alignment loss pushes successive components to be dependent. The mutual information $I(Y_i,Y_{i+1})=H\left(Y_{i+1}\right)-H\left(Y_{i+1}\right|Y_i)$ between dependent variables is always positive, so the tendency of the alignment loss is to make $H\left(Y_{i+1}\right)$ strictly greater than $H\left(Y_{i+1}\right|Y_i)$.

In short, the joint effect of the alignment and the resolution terms is to push the $h^{\left(1\right)}\left(\mathbf{Y}\right)$ term in the chain (19) to be strictly less than $h^{\left(0\right)}\left(\mathbf{Y}\right)$ and all the other terms $h^{\left(j\right)}\left(\mathbf{Y}\right)$ and $j\ge 1$ to be equal to each other, i.e.,

$$h\left(\mathbf{Y}\right)=h^{(d-1)}\left(\mathbf{Y}\right)=\cdots =h^{\left(1\right)}\left(\mathbf{Y}\right)<h^{\left(0\right)}\left(\mathbf{Y}\right)$$

(20)

which means that the learned representation $Y_j$ is a Markov process. Repeating the argument in the other direction, we obtain a two-sided Markov process. Of course, the degree to which (20) can, in fact, be satisfied in a given problem is determined by the underlying data distribution.

4.3. Training the Model

Our system is composed of two separate subsystems and training loops. One subsystem seeks the generator $\mathfrak{G}$ of symmetries and the resolving filter $\mathit{\psi }$, which together form the group convolution matrix L, while the other consists of probability estimators. For each batch, densities and entropies of the data transformed by the group convolution matrix are estimated and used in the computation of the loss for symmetry learning. The training for the two subsystems is conducted jointly, with simultaneous gradient updates. See Figure 5 and Figure 6 for a visual representation of the two subsystems, and see Appendix E.1 for the details of the training process.

4.3.1. Training the Group Convolution Layer

We used the ADAM optimizer with learning rate decay for the group convolution layer, while the other parameters were kept at their default values. Since the higher powers computed for the relevant matrices are very sensitive to numerical errors, we preferred smaller learning rates. Numerical values and details can be found in Appendix E.1.

Due to the high dimensionality of the search space of the relevant matrices, accurate identification of true symmetries is challenging. The special initialization and optimization procedures described below help to avoid becoming stuck at the local minimum.

Controlling the rank of the joint entropy. To achieve efficient and stable optimization by first capturing the data’s gross features and then refining them over time, we use a time-dependent rank parameter k for the entropy estimator (A5). To adjust k during training, we use a normalized notion of training time $t_n$, measuring the “amount of gradient flow” via

$$t_n={\displaystyle \frac{{\sum }_{s=1}^n\mathrm{lr}\left(s\right)}{{\sum }_{s=1}^T\mathrm{lr}\left(s\right)}}$$

(21)

where $\mathrm{lr}\left(s\right)$ is the learning rate used at the training step (batch) s, and n and T are the current training step and the total number of training steps, respectively. We control the rank k of the low-rank entropy estimator by setting $k=\mathrm{ceil}(d\times t_n)$ so that, by the end of the training, the rank is at d.

Noise injection to the resolution filter. We initialize the resolving filter with zeros and add Gaussian noise during the early stages of training before computing the loss for each batch,

$$\mathit{\psi }\leftarrow \mathit{\psi }+\mathcal{N}(\mu =0,\sigma =0.1)exp(-t/\tau ).$$

(22)

The amplitude of the noise is set to decay exponentially with a short time constant (of $\tau =10$ epochs). As mentioned previously, to compute the transformed data $\mathbf{y}$, we use a normalized version of $\mathit{\psi }$ at each step: $\mathit{\psi }\leftarrow \mathit{\psi }/{\parallel \mathit{\psi }\parallel }_2$

Initialization of the symmetry generator. We initialize the matrix A used to obtain the symmetry generator $\mathfrak{G}$ via the exponential map $exp\left({\displaystyle \frac{A-A^T}{2}}\right)$ using a normal distribution with $\sigma ={10}^{-3}$ and $\mu =0$ for its entries. This leads to an approximate identity matrix for the initial generator $\mathfrak{G}$. Using smaller standard deviations did not affect the performance; however, significantly larger $\sigma$ values occasionally lead to learning a higher power of the underlying symmetry generator instead of learning the minimal generator.

Padding. We use padding for the symmetry generator and the filter in the sense that the symmetry matrix and the filter have dimensionality that is higher than the dimensionality of the data, but we centrally crop the matrix and the filter before applying them to the data. This is conducted to deal with finite size (edge) effects, and, after experimenting with padding sizes of 6 to 33, we saw that the results are not sensitive to padding size. Working with a cyclic/periodic symmetry would make the padding unnecessary, but this would mean working with a restrictive assumption on the underlying symmetry.

4.3.2. Training Probability Density Estimators

We train probability density estimators based on entropy minimization loss term, which is proposed by Pichler et al. [31]. See Appendix E for the learning rate used.

5. Results

5.1. Results on Synthetic and Real Data

For each dataset listed in

Table A2. Real and synthetic datasets prepared for experiments. The parameters for each basis signal are sampled from a uniform distribution with the indicated ranges.

ID	Signal Type	Invariance	Dimensionality	Parameter Ranges	Size (Samples)
1	Gaussian	Circulant translation	7	Scale: $[0.2,1.0)$ Amplitude: $[0.5,1.5)$	63 K
2	Gaussian	Translation	7	Scale: $[0.2,1.0)$ Amplitude: $[0.5,1.5)$	252 K
3	Mnist slices	Translation	27	——-	1.08 M
4	Gaussian	Translation	33	Scale: $[0.5,5.0)$ Amplitude: $[0.5,1.5)$	1.65 M
5	Legendre ($l=2–3$, $m=1$)	Translation	33	Scale: $[6.0,15.0)$ Amplitude: $[0.5,1.5)$	1.65 M
6	Gaussian	Permuted translation	33	Scale: $[0.5,5.0)$ Amplitude: $[0.5,1.5)$	1.65 M
7	Legendre ($l=2–3$, $m=1$)	Permuted translation	33	Scale: $[6.0,15.0)$ Amplitude: $[0.5,1.5)$	1.65 M
8	Gaussian	Frequency shift	33	Scale: $[0.5,5.0)$ Amplitude: $[0.5,1.5)$	1.65 M
9	Legendre ($l=2–3$, $m=1$)	Frequency shift	33	Scale: $[6.0,15.0)$ Amplitude: $[0.5,1.5)$	1.65 M

, our model was trained to learn the group action generator $\mathfrak{G}$ and the resolving filter $\mathit{\psi }$, which together comprise the group convolution matrix, L, which provides a symmetry-based representation of data. The group generator $\mathfrak{G}$ has a known correct answer in each case (up to a power of $\pm 1$; e.g., both right and left translations are minimal generators of the translation group), which, of course, was not given to the model in any way.

A visual description of the experimental results is provided in Figure 7, Figure 8, Figure 9 and Figure 10, with Figure 7, Figure 8 and Figure 9 containing the results on 33-dimensional synthetic datasets with various symmetries and Figure 10 containing the results on a 27-dimensional real dataset (see Appendix C.2 for details). In each case, we compare the learned generator to the appropriately signed generator of the group. The main points are as follows:

• In each experiment, the learned symmetry generator $\mathfrak{G}$ is indeed very close to the underlying correct generator used in preparing the dataset. See Figure 7a–Figure 10a.
• The group convolution operator L formed by combining the learned symmetry generator $\mathfrak{G}$ with the learned resolving filter $\mathit{\psi }$ as in (11) is approximately equal to the underlying transformation used in generating the dataset (see Figure 7b–Figure 10b). As a result, the matrix L is highly successful in reconstructing the underlying hidden local signals, resulting in a symmetry-based representation (see Figure 11).

Thus, by having access only to raw samples such as those in Figure 4, the model has been able to learn symmetry generators of quite different sorts: pixel translations, pixel shuffles, and frequency shifts were all learned with high accuracy using exactly the same model setup and hyperparameters. The group convolution operator L relates the raw data to a representation where the locality and symmetry are manifest. In the case of frequency shifts, the relevant transformation that completes this is the discrete sine transform (DST). We find it highly satisfying that, simply by looking at data as in Figure 4, the model was able to learn that the DST matrix provides the relevant representation; see Figure 8b.

We emphasize that the same hyperparameters have been used in each case: the learning rates, low-rank entropy estimation scheduling, batch size, etc., were the same for all the experiments. In other words, no fine-tuning was necessary for different data types. The results are also stable in the sense that training the system from scratch results in similar outputs each time. In our various experiments with the chosen settings, only once did we encounter a convergence problem where the training became stuck in an unsatisfactory local minimum.

We repeated each one of the seven experiments four times and reported the cosine similarity in

Table 1. Cosine similarity between the learned and ideal minimal generators.

Symmetry	Gaussian Dataset	Legendre Dataset	MNIST Slices
Translation	$0.999\pm 0.001$	$0.998\pm 0.003$	$0.999\pm 0.001$
Permuted translation	$0.996\pm 0.002$	$0.998\pm 0.001$	-
Frequency space translation	$0.980\pm 0.003$	$0.991\pm 0.003$	-

(cosine similarity here is obtained by treating the generator matrix as a vector, or, equivalently, using the Hilbert–Schmidt (or Frobenius) inner product between the two matrices of the symmetry generator). The error histograms over the entries of the symmetry generator are provided in Figure 7a–Figure 9a.

Some notable properties of the method worth emphasizing are as follows:

Learning the minimal group generator. A dataset with a pixel translation symmetry also has a symmetry under two-pixel translations, three-pixel translations, etc. In fact, each one of these actions could be considered as a generator of a subgroup of the underlying group of symmetries. Some works in the literature have successfully learned symmetry generators from datasets but ended up learning one of many subgroup generators instead of the minimal generator. Our inclusion of locality together with stationarity in the form of alignment and resolution losses has allowed the method to learn the minimal generator in each case.

Learning a symmetry-based representation. In addition to the symmetry generator, our model learns a resolving filter that is local along the symmetry direction. In all our experiments, this resulted in the appropriate “delta function along the symmetry direction” described in Section 3. In the case of simple translation symmetry, the filter has a single nonzero component; in the case of the frequency shift, it is a pure sinusoidal. The convolution operator L obtained by combining the generator and the filter provides a symmetry-based representation of the data, turning the hidden symmetry into a simple translation symmetry. Such a representation is exactly the type of situation where regular CNNs are effective. Thus, for a given supervised learning problem, one can first train our model to learn the underlying symmetry and then transform the data into the symmetry-based representation and feed the resulting form into a regular multi- layer CNN architecture, making the model an adapter between raw data and CNNs. In Appendix B, we prove the relevant equivariance property of the learned representation.

Stability. Our datasets are 33-dimensional, and thus the symmetry generator $\mathfrak{G}$ and the symmetry-based representation matrix L have shapes of $33\times 33$. These dimensions are higher than in many of the works that have a comparable aim to ours (i.e., unsupervised learning of symmetries from raw data [9,24,25]). We emphasize that we do not have any built-in sparsity or factorization enforced on the matrices. The model has to indeed perform the searches over these large spaces. Our loss function involving locality, stationarity, and information preservation has resulted in a highly robust system that reliably avoids local minima and finds the correct symmetries in each of the examples we implemented. We emphasize that none of our datasets are explicitly or cleanly symmetric under the relevant group actions (see Figure 4 for samples). The almost perfect recovery of the symmetry generator from such samples is rather striking. Examples of the robust optimization dynamics can be seen in Figure 12.

5.2. Comparison with Other Unsupervised Symmetry Learning Approaches

The GAN-based methods SymmetryGAN [9] and LieGAN [24], like our method, are set up to learn symmetry transformations starting with raw data in an unsupervised manner. To compare the performance of these methods with ours, we ran experiments with a seven-dimensional version of our translation-invariant Gaussian dataset described above. In order to comply with the setting of [9,24], this time, we set up the dataset so that the translation is circulant (periodic). Our method’s setup can also be extended to cover circulant symmetries, but we attempted the run without this modification and in this sense provided an advantage to the GAN-based methods.

In the end, the best cosine similarity obtained by SymmetryGAN was $0.527$ and that obtained by LieGAN was $0.425$, whereas the cosine similarity for our model was $0.958$. See Figure 13 and Figure 14 for a visualization of the learned generators by each method, and see Appendix D for the details of the comparison.

6. Discussion

Our experiments show that the method described in this paper can uncover the symmetries (generators of group representations) of quite different sorts, starting with raw datasets that are only approximately local/symmetric under a given action of a one-dimensional Abelian Lie group. In addition to symmetry generators, a symmetry-based representation [4] is also learned. The choice of the loss function and the approach used in optimization result in highly stable results. This is the outcome of extensive experimentation with different approaches to the same intuitive ideas described in the Introduction. It is satisfying, and perhaps not surprising, that the current successful methodology is the simplest among the range of approaches we tried.

In this paper, we focused on the action of one-dimensional Abelian Lie groups. More generally, the symmetry group will be multidimensional. The case of translation symmetry in the plane has two generators that commute with each other, and a generalization of the learning problem to such a case will involve adapting the loss function to encode locality and symmetry under the joint actions of those two generators, taking into account possible non-commutation of the latter. More generally, one should consider non-Abelian Lie groups whose generators do not commute with each other.

The datasets used in our experiments were 33-dimensional and thus are not what one would call “low-dimensional” in the context of symmetry learning. However, many real datasets have much higher dimensionalities. Our experiments indicate that our method can work on datasets with three times the dimensionality used in this paper without difficulty; however, for dimensions that are orders of magnitude higher, one will likely require further computational, and possibly methodological, improvements. Such improvements are necessary for, say, typical image datasets.

In this paper, the group actions learned by the model are all linear; i.e., the method learns a representation of the underlying group of symmetries by learning a matrix for each generator. However, more generally, symmetry actions can be nonlinear. By replacing the linear layer that represents the symmetry action with a more general nonlinear architecture it may be possible to apply the philosophy of this paper to the nonlinear case as well, but, of course, the practical problems with optimization and the architectural choices for representing a nonlinear map will require work and experimentation.

In transforming the data using the group convolution map L, we made the choice of having the number of output dimensions equal to the number of input dimensions. This meant setting the number of powers used for the symmetry generator $\mathfrak{G}$ equal to the number of input dimensions. While this is a reasonable choice and is suited to the datasets here, other choices could be appropriate for other settings. We believe it would be worthwhile to experiment with other choices here, possibly by using datasets with cyclic group symmetries whose order is different from the number of dimensions of the datasets they act on.

Finally, let us note that data often have locality directions that are not necessarily symmetry directions. For instance, consider a zeroth-order approximation to the distribution of (say daily) temperatures around Earth. This distribution will have locality under rotations around any axis passing through the center of Earth but will be approximately symmetric only under azimuthal rotations. In other words, temperature varies slowly as you move in any direction, but only similar latitudes will have similar temperature distributions; the poles will not be as hot as the equator. Our model seeks group generators that have both locality and symmetry properties for the dataset at hand, so it is not appropriate for such a dataset. A generalization that would also work for cases like this would be an interesting and worthwhile project.