Symbolic Disentangled Representations for Images

Abstract

The idea of disentangled representations is to reduce the data to a set of generative factors that produce it. Typically, such representations are vectors in latent space, where each coordinate corresponds to one of the generative factors. The object can then be modified by changing the value of a particular coordinate, but it is necessary to determine which coordinate corresponds to the desired generative factor—a difficult task if the vector representation has a high dimension. In this article, we propose ArSyD (Architecture for Symbolic Disentanglement), which represents each generative factor as a vector of the same dimension as the resulting representation. In ArSyD, the object representation is obtained as a superposition of the generative factor vector representations. We call such a representation a symbolic disentangled representation. We use the principles of Hyperdimensional Computing (also known as Vector Symbolic Architectures), where symbols are represented as hypervectors, allowing vector operations on them. Disentanglement is achieved by construction, no additional assumptions about the underlying distributions are made during training, and the model is only trained to reconstruct images in a weakly supervised manner. We study ArSyD on the dSprites and CLEVR datasets and provide a comprehensive analysis of the learned symbolic disentangled representations. ArSyD outperforms BetaVAE and FactorVAE baselines on CLEVR1 paired, achieving an FID of 93.72 compared to 129.68 and 115.61, respectively. It also achieves the best IOU value on dSprites paired, at 98.37, compared to 96.43 and 97.11 for the other baselines. We also propose new disentanglement metrics that allow comparison of methods using latent representations of different dimensions. ArSyD allows us to edit the object properties in a controlled and interpretable way, and the dimensionality of the object property representation coincides with the dimensionality of the object representation itself.

1. Introduction

Good data representation for machine learning algorithms is one of the key success factors for modern approaches. Initially, the construction of good representations consisted of feature engineering, i.e., the manual selection, creation, and generation of such features that allow the model to successfully solve the main problem. Although feature engineering is still used in some areas, current models rely on learning representations from data [1]. This approach has become ubiquitous in computer vision [2], natural language processing [3], processing of complex structures such as graphs [4,5], and others. A good representation can mean a representation with different properties, e.g., proximity of representations for semantically related objects [3], identification of common features in objects [2], preservation of a complex structure with dimensionality reduction, and disentanglement of representations [6,7,8].

In this article, we consider disentangled representations. Following the work in [9], we define disentangled representations as those described by the following three properties: (1) disentanglement, i.e., the representation should factorize (disentangle) the underlying generative factors so that one variable captures at most one factor, (2) completeness, the degree to which each underlying factor is captured by a single variable, (3) informativeness, i.e., the amount of information that a representation captures about the underlying generative factors (to be useful for tasks that require knowledge of the important data attributes, representations must fully capture information about the underlying generative factors). The disentangled representation can potentially improve generalization and explainability in many machine learning tasks: structured scene representation and scene generation [10,11], reinforcement learning [12,13,14], planning [15], reasoning [16,17], and object-centric visual tasks [18,19,20,21,22]. The main research in disentanglement focuses on obtaining representations expressed by a vector, where each coordinate of the vector captures a generative factor [6,7,8], and by changing this coordinate, we can change the property of the represented object. However, disentangled representation learning has also been investigated in object-centric frameworks, such as Slot Attention [23] and its extensions [22,24].

In this work, we propose Architecture for Symbolic Disentanglement (ArSyD), which uses a disentangled representation in which an entire vector of the same dimension as the resulting representation of the object captures a generative factor. The object representation is then obtained as a superposition of the vectors responsible for each of the generative factors. We call such a representation a symbolic disentangled representation. ArSyD is based on the principles of Hyperdimensional Computing (HDC) (also known as Vector Symbolic Architectures) [25,26,27], where symbols are represented by vectors of high and fixed dimension—hypervectors (HVs also known as semantic pointers [28]). In most applications [29,30,31,32,33,34], to represent an object as a vector in high-dimensional latent space (the dimension d is typically greater than 1000), HVs are randomly sampled from this space—these vectors are called seed vectors. However, some approaches learn vectors from data [35,36].

In HDC, as mentioned earlier, the seed vectors that form the description of an object and characterize its features are usually obtained by sampling from a predetermined high-dimensional space [29,33]. They are fixed and are not learned from data. HDC takes advantage of the fact that with an extremely high probability, all seed vectors from high-dimensional spaces are dissimilar to each other (quasi-orthogonal); this is due to the concentration of measure phenomenon [37]. This allows us to reduce the manipulation of the symbols to vector operations. Two main operations in HDC are bundling (addition) and binding (multiplication). The bundling maps a set of HVs to the resulting vector, which is similar to summand vectors but quasi-orthogonal to other seed vectors. Semantically, the bundling represents a set of vectors and, thus, a set of symbols. The binding maps the two HVs to another vector that is dissimilar (quasi-orthogonal) to the summands and other HVs of the vector space. The binding can be interpreted as a representation of an attribute–value pair, i.e., the assignment of a value to a corresponding attribute. Thus, although the resulting representations are expressed as vectors, they also have symbolic properties. HDC uses a distributed representation, where the vector as a whole encodes an object or its property, rather than a separate coordinate of the vector as in a localist representation.

In ArSyD, the vectors representing the feature value of an object are obtained by applying the attention mechanism [38] over a set of fixed seed vectors (a codebook or an item memory in terms of VSA) and the initial representation of the object obtained using the encoder network. The training sample is a pair of objects that match in all but one property. Each object is split into vectors equal to the number of generative factors by an encoder and a generative factor projection network (GF Projection). Next, the feature vectors that differ between the target object and the donor object are exchanged. Then, the target object is reconstructed with a feature vector from the donor object (Figure 1). This weak supervision procedure [39,40] allows the model to learn disentangled representations and edit objects in a controlled manner by manipulating their latent representations for cases where only one object is represented in the scene. For the scenes with multiple objects, we have combined the proposed approach with a Slot Attention model [23], which allows us to obtain object masks. These masks are used to extract a specific object from the scene. After that, we can edit the properties of a concrete object and restore the scenes already taking into account these changes.

Figure 1. The Architecture for Symbolic Disentanglement (ArSyD) uses an encoder network and a Generative Factor (GF) projection to represent the target and donor images as sets of HVs. These HVs correspond to the values of the underlying generative factors in the data. The same encoder is used for both target and donor objects. While most of the values for the generative factors are the same between the donor and target images, one value is different. The Feature Exchange module exchanges the vector from the set of HVs for the target image associated with this different value for a vector from the set of HVs for the donor image. The decoder then reconstructs the target image using the desired value of the generative factor.

Since ArSyD is based on the principles of HDC and uses a distributed representation, it is impossible to use popular disentanglement metrics (such as BetaVAE score [6], DCI disentanglement [9], and others) for localist representations. Therefore, we propose two disentanglement metrics based on the classification of a reconstructed scene—the Disentanglement Modularity Metric (DMM) and the Disentanglement Compactness Metric (DCM)—that allow the evaluation of models regardless of the type of latent representation used. The DMM is responsible for the relationship between the latent unit (coordinate, vector, or other representation) and the generative factor. The DCM is reduced when a change in one unit in the latent space changes the classification of only one feature.

We have demonstrated how ArSyD learns symbolic disentangled representations for objects from modified dSprites [41] and CLEVR datasets [42], and we provide the proof-of-concept results on the realistic CelebA dataset [43]. We also used ArSyD combined with the Slot Attention model to edit scenes from the CLEVR datasets.

Thus, the contributions of this article are:

• An Architecture for Symbolic Disentanglement (ArSyD), which uses a weakly supervised approach to learn symbolic disentangled representations based on a combination of attention over a codebook of fixed vectors and the principles of HDC;
• The learned representations allow us to edit objects in a controlled and interpretable way by manipulating their representations in the latent space;
• A new model based on Slot Attention that allows object-centric and interpretable editing of a scene with multiple objects;
• New disentanglement metrics that allow the comparison of models with different types of latent representations.

2. Disentangled Representation

Most of the methods for learning disentangled representations are based on either the Variational Autoencoder (VAE) [6,8,44,45,46] or on the Adversarial Generative Networks [47] (GAN) [7,48,49].

In these approaches, the disentanglement is achieved by imposing additional constraints on the loss: by introducing a $\beta$ parameter to balance the independence constraints with reconstruction accuracy in BetaVAE [6]; by adding a capacity control objective in [44]; by factorizing the representation distribution in FactorVAE [8]; by decomposing the evidence lower bound in $\beta$-TCVAE [45]; by minimizing the covariance between the latents in DIP-VAE [46]; by maximizing the mutual information in InfoGAN [7]; by introducing a contrastive regularizer in InfoGAN-CR [48]; by adding a mutual information loss to StyleGAN [50] in Info-StyleGAN [49]. Other approaches impose additional restrictions based on group theory on existing VAE models [51], or propagate inductive regulatory bias recursively over the compositional feature space [52], or provide a framework for learning disentangled representations and discovering the latent space [53].

In ArSyD, we do not impose additional specific restrictions on the loss, but we use a structured representation of the object in the latent space using HDC principles and a special learning procedure. We represent each generative factor as a HV of the same dimension as the resulting representation. The object representation is then obtained as a superposition of the vectors responsible for the generative factors.

Works on structured and factorized latent representations in VAEs improves interpretability by introducing hierarchical or grouped latent variables instead of relying purely on independence-promoting losses. Hierarchical VAEs such as NVAE organize information across multiple scales to improve generative modeling [54], while later approaches explore adaptive latent selection and structured priors to identify meaningful latent dimensions automatically [55]. Other methods extend VAEs toward compositional or part-based representations, where latent variables are grouped to reflect semantic structure in the data [56]. Despite these advances, such methods remain fundamentally coordinate-based, with structure emerging through constraints on scalar latent variables.

Works on combining symbolic representations with deep learning focus on integrating structured concepts into neural latent spaces to improve interpretability and reasoning. Neuro-symbolic approaches typically couple neural perception with symbolic reasoning modules or structured knowledge representations, enabling explicit manipulation of high-level concepts while retaining differentiable learning [57,58]. More recent surveys highlight hybrid representation spaces where symbolic variables, logic constraints, or structured abstractions are aligned with continuous embeddings to support reasoning and generalization [59].

In contrast, ArSyD embeds symbolic structure directly into the latent space by representing each generative factor as a full high-dimensional vector. Unlike neuro-symbolic systems that rely on separate symbolic components or external reasoning modules, ArSyD unifies symbolic structure and representation within a single latent space, making compositional semantics an intrinsic property of the representation rather than an externally imposed mechanism.

Recent research on disentangled representation learning has shifted from purely loss-based formulations toward structured, object-centric, and compositional representations, where disentanglement is achieved through architectural priors and weak supervision. Object-centric approaches based on Slot Attention [23] and its extensions have become a central paradigm, representing scenes as sets of object-level latent variables and enabling compositional generalization across environments [22,24,60,61,62].

Thus, the novelty of ArSyD lies in introducing symbolic disentanglement as a representation-level property, providing concrete advantages over both classical and recent approaches. First, representing each factor as a distributed vector avoids the instability and ambiguity of coordinate-based latent spaces, which remain a limitation in modern disentanglement and object-centric models. Second, it enables direct and interpretable control of generative factors without requiring post hoc identification of latent dimensions. Third, the approach naturally supports compositional structure and integrates with object-centric pipelines while avoiding reliance on independence assumptions or auxiliary loss terms. Finally, by decoupling disentanglement from loss engineering, ArSyD avoids the trade-off between reconstruction quality and factor separation observed in VAE-based methods, offering a more stable, interpretable, and structurally grounded alternative to recent disentanglement frameworks.

To evaluate the disentanglement of the representation, special metrics are used that assume a localist representation in the latent space, where each coordinate represents the generative factor. The BetaVae score [6] uses a simple classifier to predict the index of a generative factor that is held fixed while other factors are randomly sampled. This approach is not applicable to distributed representations because individual positions in such representations do not carry all the information about the generative factor. FactorVAE score [8] proposes a similar approach but uses an index of a position with less variance. MIG [45] uses mutual information and also refers to the individual coordinates of the latent representation, making it impossible to use this metric for other representations. SAP score [46] uses classification or linear regression, which also does not work for distributed representations. In this article, we propose two new disentanglement metrics that allow the comparison of models regardless of whether they use a localist or a distributed representation in the latent space.

3. Methods

3.1. Object Representation in Latent Space

In this article, we apply HDC principles to obtain an object representation in the latent space and then map this representation to the raw data. We use a distributed representation and represent each generative factor as an HV with the same dimensionality as a final object representation. This means that the information is distributed across all components, no single HV component has a meaning, i.e., only the whole HV can be interpreted. This differs from the localist representation used by modern disentanglement models, where a single vector component potentially has a meaning. The nature of HVs can be different, such as binary [25], real [63,64,65,66,67], complex [63], or bipolar [68,69]. Also, the exact implementation of the vector operations may vary for different vector spaces while maintaining the computational properties. For a detailed comparison of different VSA implementations, we recommend the reader to refer to [70].

To represent a simple entity in HDC, the seed vector is sampled from the vector space and stored in an item memory. When an object has a complex structure, the resulting representation is obtained by performing vector operations on the seed vectors from the item memory [26]. We illustrate the vector operations defined in HDC with an example from the Holographic Reduced Representation (HRR) [64] that we use in this article. This implementation works with real-valued HVs sampled from the normal distribution $\mathcal{N}{\left(0,{\displaystyle \frac{1}{D}}\right)}^D$ with mean 0 and variance ${\displaystyle \frac{1}{D}}$, and it uses cosine distance as the similarity measure, where D is the vector dimension.

Two main vector operations are bundling (addition) and binding (multiplication).

The addition (+) is implemented as an element-wise sum with normalization:

$$R={\displaystyle \frac{{\sum }_{i=1}^N{V}_i}{N}}.$$

(1)

The resulting vector R is similar to the summand vectors $V_i$ but quasi-orthogonal to other seed vectors. Semantically, bundling represents a set of vectors and, thus, a set of symbols.

The multiplication (⊛) uses a circular convolution:

$$R=V_1⊛V_2={\mathcal{F}}^{-1}\{\mathcal{F}\left\{V_1\right\}\odot \mathcal{F}\left\{V_2\right\}\},$$

(2)

where ⊙ is an element-wise multiplication, $\mathcal{F}$ and ${\mathcal{F}}^{-1}$ are the Fourier transform and the inverse Fourier transform, respectively. The multiplication maps the vectors $V_1$ and $V_2$ to another HV—R. The resulting vector R is dissimilar (quasi-orthogonal) to $V_1$, $V_2$ and other HVs of the vector space. Binding represents an attribute–value pair, i.e., the assignment of a value to a corresponding attribute.

Using these operations, we can represent any object O generated by N underlying generative factors as a set of attribute–value pairs (Figure 2a):

$$\begin{array}{c}O=G_1⊛V_2^{G_1}+G_2⊛V_4^{G_2}+\dots +G_N⊛V_1^{G_N},\end{array}$$

(3)

where $G_i$ is a i-th generative factor, $V_{j_m}^{G_i}$ is a $j_m$-th value of a generative factor $G_i$. $j_m=1\dots k_{G_i}$, where $k_{G_i}$ is the number of values of a generative factor $G_i$.

Figure 2. The process of obtaining a symbolic disentangled representation of an object: (a) usual approach in HDC—a symbolic description is used without grounding in the raw data; (b) approach in ArSyD—the symbolic description of an object is grounded in the raw data to obtain its symbolic disentangled representation.

Thus, the target object O in Figure 1 can be represented as:

$$\begin{array}{c}O=S⊛Cy+C⊛R+Si⊛L+M⊛Me+\\ +C_X⊛X+C_Y⊛Y,\end{array}$$

(4)

where $\left(S\right)hape,\left(C\right)olor,\left(Si\right)ze,\left(M\right)aterial,\left(C\right)oor{d}_X,\left(C\right)oor{d}_Y,\left(Cy\right)linder,\left(R\right)ed,\left(L\right)arge,$ and $\left(Me\right)tal,X,Y$ are HVs for underlying generative factors and their corresponding values. The HVs for the generative factors $G_i$ and the values $V_{k_{G_i}}^{G_i}$ are sampled at the time of model initialization, then fixed and stored in the corresponding item memory (codebook). They do not change during the training and testing phases of the model. The number of sampled HVs $G_i$ corresponds to the number of generative factors, while the number of HVs $V_{k_{G_i}}^{G_i}$ corresponds to the number of values for the respective generative factor $G_i$. Thus, we can obtain an HDC representation of an object according to its symbolic description without grounding in the raw data. This approach has proven effective in many applications, but it requires an explicit algorithm to represent an object by its properties, which makes it difficult to transfer to new domains.

In this article, we propose an architecture that allows HDC representations to be grounded in the raw data (Figure 2b). The first step is to use an encoder to extract an intermediate object representation $O^{\prime }$ from the image I. Next, we use the Generative Factor (GF) projection module to obtain an intermediate representation of the generative factor values $V_i^{\prime }$. We use a single-layer fully connected neural network with D inputs and $ND$ outputs, where N is the number of underlying generative factors for this module. The output of this module is decomposed into N vectors, which are $V_1^{\prime },\dots ,V_N^{\prime }$ vectors of dimension D. We do not use activation functions, nor normalization or regularization for this module. These vectors are then fed into the Generative Factor representation modules, which use the attention mechanism [38] to represent the value vector $V_i^{*}$ as a linear combination of seed vectors from the codebook $V_{j_m}^{G_i}$:

$$\begin{array}{c}{a}_{\ell }=softmax{\left({\displaystyle \frac{V_i^{\prime }{K}_{j_m}}{\sqrt{D}}}\right)}_{j_m},\end{array}$$

(5)

where $softmax{\left(\right)}_{j_m}$—the $j_m$-th component, D—the dimension of a vector space, $K_{j_m}$—the projection of the $j_m$-th seed vector $V_{j_m}^{G_i}$ from the item memory,

$$\begin{array}{c}{V}_i^{*}=a_1{V}_1^{G_i}+a_2{V}_2^{G_i}+\dots +a_{k_{G_i}}{V}_{k_{G_i}}^{G_i}.\end{array}$$

(6)

The main challenge is to bridge the gap between localist and distributed representations. This is necessary to effectively ground distributed HDC representations in the raw data. In ArSyD, this is achieved by using the attention mechanism (Equation (5)). Frozen vectors $V_{k_{G_i}}^{G_i}$ stored in the item memory and initially sampled with respect to the distributed representation are used as value vectors. The trained projections $V_i^{\prime }$ and $K_{j_m}$ make it possible to build a bridge between the localist representations $O^{\prime }$, obtained from the encoder output, and the distributed representations of $V_{k_{G_i}}^{G_i}$, stored in the item memory, without changing the latter. Additional training of the $V_{k_{G_i}}^{G_i}$ vectors leads to a localist representation that violates the original premises.

The resulting value vectors $V_i^{*}$ are multiplied by the corresponding generative factor vectors $G_i$ and summed to produce the vector O, which is a symbolic disentangled representation of an object:

$$\begin{array}{c}O=G_1⊛V_1^{*}+G_2⊛V_2^{*}+\dots +G_N⊛V_N^{*}.\end{array}$$

(7)

The resulting vector O is used to decode the object into an image. The Encoder, GF Projection, and Attentions modules are trainable modules.

3.2. Weakly Supervised Learning

We adopt a weakly supervised learning approach, following prior work [71,72,73,74,75], which has been shown to enable learning disentangled representations [39,40]. We assume that an object O depicted in an image I can be represented as a set of N generative factors $G_i$, each associated with values $V_{j_m}^{G_i}$ (see Equation (3)).

Let $I_t$ and $I_d$ denote the target and donor images, containing the corresponding target and donor objects $O_t$ and $O_d$. We encode these objects into their generative factor representations $G^t=G\left(O_t\right)$ and $G^d=G\left(O_d\right)$. We assume that all generative factors are identical, i.e., $G_i^t=G_i^d$ for all i, and we assume their corresponding values are equal for all indices except for a single index p: $V_{j_m}^{G_i^t}=V_{j_m}^{G_i^d}$ for $i\ne p$. We then exchange the values at index p between the two representations, producing modified representations ${\widehat{O}}_t$ and ${\widehat{O}}_d$. The exchange procedure is summarized in Algorithm 1. After decoding ${\widehat{O}}_t$ into the image ${\widehat{I}}_t$, the result should match the donor image $I_d$. Similarly, after decoding ${\widehat{O}}_d$ into the image ${\widehat{I}}_d$, the result should match the target image $I_t$. This symmetry allows us to define a learning objective based on two reconstructions: comparing ${\widehat{I}}_t$ with $I_d$ and ${\widehat{I}}_d$ with $I_t$.

Algorithm 1 The Exchange Procedure

Require: $G^t=\{V_1^{G^t},\dots ,V_k^{G^t}\}$—target image features, $G^d=\{V_1^{G^d},\dots ,V_k^{G^d}\}$—donor image features, p—exchange index

Ensure: $G^t=\{V_1^{G^t},\dots ,V_p^{G^d},\dots ,V_k^{G^t}\}$—modified target image features, $G^d=\{V_1^{G^d},\dots ,V_p^{G^t},\dots ,V_k^{G^d}\}$—modified donor image features

$exchange\_value=G^t\left[p\right]$

$G^t\left[p\right]=G^d\left[p\right]$

$G^d\left[p\right]=exchange\_value$

3.3. End-to-End Model

For experiments with a single object in the scene, we use the architecture shown in Figure 2b and add a decoder that reconstructs the target and donor images. We train the model using only reconstruction losses, defined as the mean squared error (MSE) between the original and reconstructed images:

$$\begin{array}{c}{L}_{\mathrm{image}}=MSE(I_t,{\tilde{I}}_t)+MSE(I_d,{\tilde{I}}_d),\end{array}$$

(8)

where $I_t$ is the target image, ${\tilde{I}}_t$ is the reconstructed target image, $I_d$ is the donor image, and ${\tilde{I}}_d$ is the reconstructed donor image. The overall end-to-end model for the case of multiple objects in a scene is shown in Figure 3. We use Slot Attention [23] to discover objects in the scene and represent the scene as a collection of individual objects. For the target object $O_t$, we reconstruct its image ${\tilde{I}}_t$ from the corresponding slot S and use this image in the Feature Exchange module.

Figure 3. The ArSyD architecture for the case of multiple objects in a scene. We use Slot Attention to discover objects in the scene and represent the scene as a collection of individual objects. The Feature Exchange module is then used to replace the value of the desired generative factor. The target object and corresponding slot are marked by red dashed box. The model is trained end-to-end using the loss $L_{\mathrm{scene}}$ (Equation (9)).

The Feature Exchange module is implemented similarly to the model shown in Figure 1. However, instead of reconstructing the modified object representation directly after exchanging the generative factor, we use a multilayer perceptron (MLP) to map the modified representation into the slot space, producing a modified slot representation $S^{\prime }$. We then replace the original object slot S with the modified slot $S^{\prime }$ and reconstruct the scene image ${\tilde{I}}_s^{\prime }$. The reconstructed scene therefore contains the object with the modified generative factor.

During training, we use only reconstruction losses (MSE) to enforce image reconstruction quality:

$$\begin{array}{c}{L}_{\mathrm{scene}}=MSE(I_s,{\tilde{I}}_s)+MSE(I_s^{\prime },{\tilde{I}}_s^{\prime })+MSE(I_d,{\tilde{I}}_d),\end{array}$$

(9)

where $I_s$ is the original scene image, ${\tilde{I}}_s$ is the reconstructed original scene image, $I_s^{\prime }$ is the scene image containing the donor object, ${\tilde{I}}_s^{\prime }$ is the reconstructed scene image containing the donor object, $I_d$ is the donor image, and ${\tilde{I}}_d$ is the reconstructed donor image.

We aim to minimize the use of auxiliary supervision; therefore, the model is trained only using original images and their reconstructions. The first term, $MSE(I_s,{\tilde{I}}_s)$, trains the Slot Attention module. The second term, $MSE(I_s^{\prime },{\tilde{I}}_s^{\prime })$, links the Slot Attention, Feature Exchange, and slot-mapping MLP modules. The third term, $MSE(I_d,{\tilde{I}}_d)$, trains the Feature Exchange module.

3.4. Evaluation Metrics

Popular disentanglement metrics, such as the BetaVAE score [6], DCI Disentanglement [9], MIG [45], SAP score [46], and FactorVAE score [8], assume that disentanglement is achieved when each latent vector coordinate captures a single generative factor in the data, i.e., localist representations. Since ArSyD uses distributed representations, these metrics are not suitable for evaluating the disentanglement of the proposed model representations.

When comparing models with different latent representation structures (e.g., localist versus distributed representations), direct comparison using metrics defined on the latent space becomes problematic. Therefore, we propose new metrics that can evaluate disentangled representations independently of how individual generative factors are encoded.

In the proposed approach, the metrics are computed using explicit representations rather than latent representations, e.g., in pixel space. Although our work focuses on images, the proposed evaluation approach can be naturally extended to other modalities, such as text, audio, and others.

We use the term unit of the latent representation (or simply unit) to refer either to a coordinate value in a latent vector for localist representations or to a hypervector (HV) selected from the codebook and associated with a particular generative factor in distributed representations.

The main idea is that a disentangled latent representation should allow controlled modifications of the reconstructed object in pixel space. For example, if we modify the unit responsible for the object color, only the color of the reconstructed object should change. If the representation is not disentangled, changing this unit may also affect other object properties. Conversely, if a latent unit is not associated with any generative factor, modifying it may produce no visible change.

Based on this observation, we define metrics that evaluate disentanglement by classifying reconstructed images in pixel space while modifying individual generative factors in the latent space.

Knowing the number and types of generative factors, we train classifiers that predict the values of these factors from images in pixel space. For a validation image I, we first obtain its latent representation h, reconstruct the image $\tilde{I}=Decoder\left(h\right)$, and predict the values of the generative factors:

$$\widehat{y}=Cl\left(\tilde{I}\right)$$

(10)

These predictions may differ from the ground-truth labels of the original image because of reconstruction artifacts introduced by the decoder. Next, we modify a latent unit c, obtaining a modified representation $h_c^{\prime }$, reconstruct the corresponding image ${\tilde{I}}_c^{\prime }=Decoder\left(h_c^{\prime }\right)$, and predict its generative factors:

$${\widehat{y}}^{\prime }=Cl\left(\widetilde{I^{\prime }}\right)$$

(11)

We then compare the predictions before and after the modification using the Iverson bracket $[{\widehat{y}}_c^{\prime }\ne \widehat{y}]$, which equals 1 if the prediction for a generative factor changes and 0 otherwise.

Importantly, we do not use the ground-truth labels of the original images. This makes the metrics robust to reconstruction errors. For example, if the decoder reconstructs an incorrect shape both before and after modifying the latent representation, this error does not affect the metric values.

Based on [76], we propose two metrics: the Disentanglement Modularity Metric (DMM) and the Disentanglement Compactness Metric (DCM).

The DMM evaluates whether each latent unit is responsible for only one generative factor:

$$\begin{array}{c}DMM={\displaystyle \frac{1}{\left|C\right|}}{\displaystyle \sum _{c\in C}}H\left(\sigma \left({\displaystyle \sum _{v\in V_c}}[{\widehat{y}}_{c,v}^{\prime }\ne \widehat{y}]\right)\right),\end{array}$$

(12)

where C is the set of latent units, $V_c$ is the set of tested values for unit c, $H(·)$ is entropy, and $\sigma (·)$ is the softmax function.

For a perfectly disentangled representation, modifying a latent unit changes only one generative factor. In this case, the resulting probability distribution contains a single dominant value and therefore has minimal entropy.

The DCM evaluates whether each generative factor is encoded by a single latent unit:

$$\begin{array}{c}DCM={\displaystyle \frac{1}{\left|C\right|}}{\displaystyle \sum _{c\in C}}\left|{\displaystyle \sum _{G_i\in G}}{[{\widehat{y}}_c^{\prime }\ne \widehat{y}]}_{G_i}-1\right|,\end{array}$$

(13)

where G is the set of generative factors and ${[·]}_{G_i}$ indicates whether the prediction for factor $G_i$ changes after modifying latent unit c.

The DCM counts how many generative factors change after modifying a latent unit. Ideally, only one factor should change. Unlike previous approaches, we do not normalize by the number of latent units, since such normalization may artificially encourage the use of fewer latent units.

The procedure for computing DMM and DCM is illustrated in Figure 4.

Figure 4. The principle of computing the DMM and DCM metrics is illustrated as a matrix of relations between generative factors and latent units. Columns $G_1,\dots ,G_n$ correspond to the generative factors, while rows correspond to latent units $c\in C$. For each entry (red box), we assign a value of 1 if the prediction for the corresponding generative factor changes after modifying the latent unit, i.e., $[{\widehat{y}}^{\prime }c\ne \widehat{y}]G_i=1$, and 0 otherwise. The row-wise sums are used to compute the DMM metric by measuring how many generative factors are affected when a single latent unit is modified multiple times. The column-wise sums are used to compute the DCM metric by measuring how many latent units influence the same generative factor.

It is worth noting that metric calculations are independent of the classifiers architecture and are based solely on their predictions. Therefore, for complex features, more sophisticated classifier models can be used, including evaluation using visual-linguistic models [77] or fundamental models [78].

3.5. Datasets

In this article, we used datasets generated from dSprites [41] and CLEVR [42] to test the proposed approach. To demonstrate the potential generalizability of our approach to non-synthetic real-world images, we used the CelebA dataset [43].

3.5.1. dSprites Paired

dSprites contains 737,280 procedurally generated 2D shapes (images of size $64\times 64$) with the following generative factors: shape (square, ellipse, heart), scale (6 values), orientation (40 values in $\left[0.2\pi \right]$), x and y coordinates (32 values for each).

Each training example contains two images $x_1,x_2$ with generative factors $(G_1^i,\dots ,G_5^i)$ and a feature exchange vector $e=(e_1,\dots ,e_5)$, where $e_i=1-[G_i^1=G_i^2]$ ($\left[\right]$ is the Iverson bracket). Each of the two images can be a donor for the other (Figure 1), and the feature exchange vector is used in the latent representation stage to exchange the corresponding features.

For our training set, we sampled 100,000 pairs of unique images that differ by the value of one generative factor. At the same time, we excluded from the training set images with $shape=square$ and $x>0.5$ to test ArSyD for compositional generalization, as was done in [79]. The test set was sampled without restrictions and contained 30,000 pairs of unique images that differ by one value of the generative factors and do not match the examples from the training set. We call the resulting dataset dSprites paired. Examples of image pairs from the dSprites paired dataset are shown in Figure 5a.

Figure 5. Examples of image pairs from the training sample of datasets: (a) dSprites paired; (b) CLEVR1 paired; (c) CLEVR5 paired. The target objects in a pair of images differ in the value of one generative factor.

3.5.2. CLEVR Paired

CLEVR contains images of 3D-rendered objects with the following generative factors: shape, size, material type, color, x and y coordinates, and orientation (because the sphere and the cylinder are symmetric in the $XY$ plane, it is impossible to check the result when exchanging the “orientation” feature; therefore, orientation was set to zero for all images and excluded from the list of generative factors).

We have modified the source generation code from the original dataset [42] to create our training and test examples (10,000 and 1000, respectively). In the first split, as in dSprites, each training example contains two images $x_1,x_2$, and we call the feature exchange vector eCLEVR1 paired (Figure 5b) because the images contain one object. The second split additionally contains random objects $(O_1,\dots ,O_k),k\in \{0,\dots ,4\}$ and two scene images $S_1,S_2$ rendered from a set of objects, where $S_i=\{X_i,O_1,\dots ,O_k\}$, called CLEVR5 paired (Figure 5c). Camera position and lighting are random but fixed for all images of the same training example.

We used a version of CLEVR with color/shape conditions (CoGenT) to test for compositional generalization. The training set contains cubes that are gray, blue, brown, or yellow; cylinders that are red, green, purple, or cyan; and spheres that can be any color. In the validation set, the cubes and cylinders have opposite color palettes, and the test set contains all possible combinations. In addition, the size of the generated images is $128\times 128$ (default is $320\times 240$), and there is no restriction on overlapping objects.

3.5.3. CelebA Paired

The CelebA dataset consists of 200k celebrity images and is labeled with 40 binary features describing appearance. Since ArSyD is trained in the weakly supervised mode, we compiled the CelebA paired dataset from paired images, leaving only 7 features, 4 of which were obtained by grouping 11 features from the original label (“Hair” = {“Bald”, “Black_Hair”, “Blond_Hair”, “Brown_Hair”, “Gray_Hair”}, “Beard” = {“No_Beard”, “Goatee”}, “Nose” = {“Big_Nose”, “Pointy_Nose”}, “Hair_Form” = {“Straight_Hair”, “Wavy_Hair”}), and we use 3 features (“Male”, “Young”, and “Pale_Skin”) as is. We kept only those images for which only one value for the grouped features is presented in the image. Thus, we obtained 28K images, of which we made 10k pairs (or 20k images), which is the same as the settings for CLEVR1. We call this version the CelebA paired dataset. We also allowed the images to differ in several generative factors.

3.6. Architecture Details

In the experiments, we usedthe simple convolutional neural network (CNN) encoder and decoder architectures presented in Table 1 and Table 2, respectively. For the K and Q projections in the attention mechanism, we used Linear layers of size 1024 without any nonlinearity.

Table 1. Architecture of the CNN encoder used in the experiments. *,— the layer is used in the models for CLEVR1 paired and CelebA paired.

Layer	Channels	Activation	Params
Conv2D 4 × 4	64	ReLU	stride 2, pad 1
Conv2D 4 × 4	64	ReLU	stride 2, pad 1
Conv2D 4 × 4	64	ReLU	stride 2, pad 1
Conv2D 4 × 4	64	ReLU	stride 2, pad 1
Conv2D 4 × 4 *	64	ReLU	stride 2, pad 1
Linear	1024	ReLU	—
Linear	1024	—	—

Table 2. Architecture of the CNN decoder used in the experiments. *,— the layer is used in the models for CLEVR1 paired and CelebA paired.

Layer	Channels	Activation	Params
Linear	1024	GELU	—
Linear	1024	GELU	—
ConvTranspose2D 4 × 4	64	GELU	stride 2, pad 1
ConvTranspose2D 4 × 4	64	GELU	stride 2, pad 1
ConvTranspose2D 4 × 4	64	GELU	stride 2, pad 1
ConvTranspose2D 4 × 4 *	64	GELU	stride 2, pad 1
ConvTranspose2D 4 × 4	64	Sigmoid	stride 2, pad 1

3.7. Training

We trained ArSyD using the AdamW optimizer [80] with a learning rate of $2.5\times {10}^{-5}$ and other parameters set to default. We also use the learning rate scheduler OneCycleLR [81] with the percentage of the cycle spent increasing the learning rate set to 0.2. For dSprites paired, the number of epochs was 600 with a batch size of 512, and for CLEVR1 paired, and CelebA paired, they were 1000 and 64, respectively.

For CLEVR5, the model is trained end-to-end and uses the Slot Attention module for the Object Discovery task. We trained a model over 2000 epochs, which is equivalent to 300k steps with a batch size of 64. For the Slot Attention module, we used parameters based on the parameters of the original article: number of iterations $T=3$ and number of slots $K=6$, which is 1 more than the maximum number of objects in the image.

It takes about 2.5 h on a single NVIDIA TITAN RTX GPU to train our model on CLEVR1 for 600 epochs, 3.5 days on CLEVR5 for 2000, and about 2 h on dSprites.

3.8. Baselines and Metrics

We compare ArSyD with BetaVAE [6] and FactorVAE [8] as prominent representatives of approaches to obtaining disentangled representations.

Both the DMM and DCM metrics require a pre-trained classifier, for which we used 6 separate pre-trained ResNet-34 [82] models fine-tuned on the CLEVR1 paired dataset, each trained to classify values of a different generative factor.

We select a unit c from the latent representation and vary it with possible values in the latent space. For ArSyD, this is a set of predefined vectors; for BetaVAE and FactorVAE models, it is 32 linearly distributed values in the range of the $\mu \pm 3\sigma$ for this latent coordinate. After changing the latent vector, the original image is reconstructed and classified, then compared to the classification result of the original reconstructed latent vector. Ideally, only one of the coordinates would change during classification. Therefore, when normalizing and applying the softmax function, the vector of change in classifier values should consist of 1 in one coordinate and 0 in the others. This, when interpreted as a probability, gives a minimum entropy.

We also provide quantitative results that evaluate the quality of the reconstruction of objects and scenes, along with the FID metric [83] for the CLEVR paired dataset and the Intersection over Union (IoU) for the dSprites paired dataset. For the preliminary experiments on the CelebA dataset, we provide qualitative results of image reconstruction.

4. Results

In the series of experiments we conducted in this article, we sought to answer the following research questions (RQ).

4.1. RQ1. Is It Possible to Obtain Disentangled Representations Only with HDC?

To test the hypothesis of obtaining disentangled representations using HDC, we trained only the decoder (without feature exchange) on CLEVR1 paired and dSprites paired representations obtained without resorting to the raw data (images) (Figure 2a). The HV in the item memories were sampled at the moment of model initialization and fixed during training. We used the same decoder architectures as in the other experiments.

The results of the reconstruction for the HDC representation, obtained on the CLEVR1 paired and dSprites paired datasets without grounding to the images, are shown in Figure 6. For the CLEVR1 and dSprites datasets, the IOU and FID metrics (for 3 seeds) are 0.994 ± 0.001 and 80.46 ± 1.51, respectively.

Figure 6. Image reconstruction of the CLEVR1 paired and dSprites paired objects from HDC representations without grounding to the images. The results show that images are successfully reconstructed from their HDC representations. However, on CLEVR1 paired, the reconstructed images are blurrier than the original.

4.2. RQ2. Is It Possible to Ground Disentangled Representations Obtained with HDC to Raw Data Using ArSyD?

The answer to this question is the subject of the main set of experiments, in which we show how the proposed approach can be used to obtain disentangled representations based on HDC.

For the dSprites paired dataset, the results are shown in Table 3 (bottom). After training, we tested the possibility of exchanging features between random images (Figure 7). The figure shows the results of the feature exchange for two training examples. On the left side of the image, the top left corner shows the two original images (ellipse and heart). The second row below shows the result of their reconstruction. On the right, each image shows the result of the latent vector decoding after replacing a corresponding feature. It can be seen that the “Shape” feature is correctly replaced, but with a deformation. When “Orientation” and “Scale” are replaced, the main goal is achieved, but the shape of the object is slightly distorted in the case of “Orientation”.

Table 3. For all models, we used the same architectures for the encoder and decoder. All results were computed for 3 seeds. The best result is highlighted in bold. The models were trained for 600 epochs on CLEVR1 paired and 200 epochs on dSprites paired.

Metric	BetaVAE	FactorVAE	ArSyD (Ours)
CLEVR1 paired
FID	$129.68\pm 13.21$	$115.61\pm 2.49$	93.72 ± 19.13
DMM	$0.99\pm 0.12$	0.96 ± 0.27	$1.10\pm 0.04$
DCM	$2.74\pm 0.33$	$4.35\pm 1.48$	1.88 ± 0.03
dSprites paired
IOU(%)	$96.43\pm 0.29$	$97.11\pm 0.32$	98.37 ± 00.26

Figure 7. Image reconstruction of objects with modified generative factor values (yellow frame) for the dSprites paired dataset. The target object (red frame) differs from the donor object (blue frame) in all factor values. The “Shape” feature is correctly replaced but shows some deformation. Replacing “Orientation” and “Scale” achieves the main goal, though “Orientation” slightly distorts the object’s shape.

On the right side of the image, the ellipse features are replaced one by one by the square features. Here, we can see that the “Orientation” feature is strongly related to the “Shape” feature, unlike the “Scale”, “Pos X”, and “Pos Y” features, with which the images are restored relatively well. We explain this by the symmetry features of the figures: the square, the oval, and the heart are symmetric when rotated by $\pi /2$, $\pi$, and $2\pi$, respectively, while the possible rotation angles are in $[0,2\pi ]$.

In addition, the lower left corner of the image shows that the image is correctly reconstructed with the generative combination excluded from the training sample (square with coordinate $x>0.5$). In both examples, the X and Y coordinates are correctly transferred from the donor image.

The results for the CLEVR1 paired dataset are shown in Table 3 (top). ArSyD has achieved a FID metric [83] value of 93.72 ± 19.13. Figure 8 shows the process of exchanging features between objects from the CLEVR1 paired dataset. The feature exchange between random images works well with the “Color”, “Size”, and “Material” properties. In contrast to the dSprites paired dataset, there are problems with reconstruction when changing coordinates. The “Shape” feature exchange works well with respect to reconstructed images. We compare the reconstruction quality of ArSyD and baseline models in Figure 9.

Figure 8. Image reconstruction of objects with modified generative factor values (yellow frame) for the CLEVR1 paired dataset. The target object (red frame) differs from the donor object (blue frame) in all factor values. Feature exchange between random images works well for “Color”, “Size”, and “Material”. Unlike the dSprites paired dataset, changing coordinates causes reconstruction issues. “Shape” exchange, however, reconstructs well.

Figure 9. Examples of image reconstruction for all models from the dSprites paired (left) and CLEVR paired (right) datasets. The results show that the proposed method restores images on dSprites without quality degradation, and for CLEVER, with quality improvement, compared to baseline methods.

4.3. RQ3. Could ArSyD Be Used to Modify a Single Object in a Scene Containing Multiple Objects?

To answer this question, we used a combination of our approach and Slot Attention trained on the CLEVR5 paired dataset. The qualitative results for the CLEVR5 paired dataset are shown in Figure 10. The feature exchange between a single object in the scene generally works, except for the slot mask. Often, the restored slot mask for a modified image takes up more space than the restored object, which leads to problems during reconstruction: if there is a coordinate change, and one object is in front of the other one, the enlarged mask covers the first object. Since the Slot Attention mechanism is applied independently of ArSyD, this problem can potentially be solved using Slot Attention modifications that restore a more accurate object masks [22,84]. In general, the quality of the reconstructed object from the slot is better than the decoding in CLEVR1, which is confirmed by a better FID metric equal to 67.19 ± 1.6. The combination of ArSyD with Slot Attention allows object modification in a scene with multiple objects and correct reconstruction of scene images.

Figure 10. Image reconstruction of scenes with multiple objects from a test sample with modified values of generative factors of one object for the CLEVR5 paired dataset. The results show that although the restored edited image (green frame) degrades in image quality, it still correctly modifies the target object feature (see GT edited for comparison).

4.4. RQ4. How Does the Initialization of HV in the Item Memory Affect Overall Performance?

Since ArSyD uses randomly sampled HV that do not change during training, we tested the stability of the model for the dSprites dataset on 5 seeds with HV dimension D is equal to 1024. We obtained a mean of IoU is equal to $0.981$ with a variance of $0.002$. These results show that the specific set of vectors sampled during the initialization of ArSyD has little effect on the performance of the model.

4.5. RQ5. How Does Changing the HV Dimension D Affect Overall Performance?

An important hyperparameter of the model is the dimension D of the latent space. We tested how changing this value affects the performance of ArSyD for the dSprites paired dataset (Table 4). It can be seen that, in general, the larger the dimension D, the better the performance. In this case, there is a trade-off between the computational complexity and the performance. As the dimension increases above 512, the metric values change insignificantly; therefore, in our work, we used the dimension $D=1024$ for the main experiments.

Table 4. Metrics for the dSprites paired dataset with different latent dimension D. All results are computed for 3 seeds. The best result is highlighted in bold.

D	DMM	DCM	IoU
16	$1.46\pm 0.12$	$1.74\pm 0.02$	$0.948\pm 0.025$
32	$1.36\pm 0.15$	$1.77\pm 0.12$	$0.956\pm 0.018$
64	$1.33\pm 0.13$	$1.65\pm 0.04$	$0.957\pm 0.020$
128	$1.24\pm 0.02$	$1.71\pm 0.07$	$0.968\pm 0.001$
512	0.96 ± 0.07	1.58 ± 0.02	$0.973\pm 0.004$
1024	1.01 ± 0.04	1.49 ± 0.08	$0.984\pm 0.001$
2048	1.03 ± 0.06	1.49 ± 0.03	0.986 ± 0.002

4.6. RQ6. Is It Possible to Replace Multiple Generative Factors at Once with Our Approach?

To answer this question, we performed an experiment with 1–4 simultaneous exchanges and obtained an IoU metric equal to 0.985 ± 0.004.

In Figure 11, we also show the qualitative results of the feature exchange for the case where the target object and the donor objects may differ in all features. This demonstrates the generalizability of the proposed model for the case where only one feature differs.

Figure 11. Examples of image reconstruction of objects from the dSprites paired dataset with modified values of the generative factors. The target object (red frame) may differ from the donor object (blue frame) in all factor values. The results show that several features can be successfully transferred between the target and donor objects.

4.7. RQ7. Could the Proposed Approach Be Generalized to Real-World Data?

To answer this question and to demonstrate the potential generalizability of ArSyD to real-world data, we conducted preliminary experiments on the CelebA dataset. For the experiments, we used the same architecture as for the CLEVR1 dataset. The reconstruction results are shown in Figure 12. The target image (red frame) and the donor image (blue frame) typically differ across multiple generative factors. As illustrated in the second row, features such as “Hair” and “Hair_Form” are transferred effectively, preserving their characteristics in the resulting image. However, it should be noted that the reconstructed images are of insufficient quality. One reason for this is the use of a relatively simple decoder architecture (Table 2). It should also be noted that applying the proposed method to real-world data is an important direction for further research, which we discuss in Section 6.

Figure 12. Examples of image reconstruction from the CelebA paired dataset. The target image (red frame) and the donor image (blue frame) typically differ in several generative factors. The second row shows that features such as “Hair” and “Hair_Form” transfer well between images.

4.8. RQ8. What Information Is Contained in the Resulting Representations of the Values of the Generative Factors?

To answer this question, we reconstructed images from a single feature vector from an item memory (bound to a placeholder value) for the dSprites paired and CLEVR paired datasets. Reconstruction results are shown in Figure 13 and Figure 14. The results show that the vectors of the values of the generative factors do not contain complete information about the object. Of particular interest are the reconstructed images for the generative factors “Pos X” and “Pos Y”, where there is an approximate area on the image in which the object can appear for a given value of that generative factor.

Figure 13. Visualization of image reconstruction from a single feature vector from an item memory (bound to a placeholder value) for the dSprites dataset. It shows that a complete image is not reconstructed from a single vector. This indicates that the vector represents a separate property of the object. This is particularly evident in the reconstruction of the position vectors. For example, on the bottom lines where Pos Y is reconstructed, the line with the fixed position Y is reconstructed.

Figure 14. Visualization of image reconstruction from a single feature vector from an item memory (bound to a placeholder value) for the CLEVR dataset. It shows that a complete image is not reconstructed from a single vector. This indicates that the vector represents a separate property of the object.

4.9. RQ9. How Does Changing the HV Dimension D Affect the Performance of Classifiers Used to Compute DMM and DCM Metrics?

Since the results of the classification by the latent representation of the object are used to compute the proposed metrics, the size of this representation can affect the quality of the classification and thus the accuracy of the metric computation. Therefore, we conducted experiments with varying the dimension of D and computed the classification accuracy for dSprites paired and CLEVR1 paired datasets. The results are shown in Table 5 and Table 6. The results show that in general, the higher the dimensionality D, the higher the classification accuracy. However, for spatial generative factors (orientation and position) with a large number of values, the classification quality is quite low. A possible solution could be to use a stronger classification model.

Table 5. The classification accuracy for different latent space sizes D for each of the generative factors for the dSprites paired dataset. The best result is highlighted in bold.

D	Shape	Scale	Orientation	Pos X	Pos Y
16	0.36 ± 0.08	0.23 ± 0.31	0.09 ± 0.06	0.04 ± 0.03	0.01 ± 0.01
32	0.61 ± 0.01	0.18 ± 0.18	0.11 ± 0.02	0.03 ± 0.02	0.03 ± 0.02
64	0.41 ± 0.12	0.32 ± 0.12	0.17 ± 0.06	0.15 ± 0.04	0.05 ± 0.02
128	0.30 ± 0.09	0.47 ± 0.10	0.36 ± 0.02	0.14 ± 0.02	0.17 ± 0.08
256	0.24 ± 0.19	0.53 ± 0.11	0.37 ± 0.04	0.20 ± 0.04	0.09 ± 0.03
512	0.22 ± 0.09	0.52 ± 0.16	0.36 ± 0.08	0.18 ± 0.06	0.19 ± 0.06
1024	0.52 ± 0.15	0.70 ± 0.07	0.26 ± 0.05	0.33 ± 0.14	0.29 ± 0.06
2048	0.69 ± 0.04	0.60 ± 0.18	0.45 ± 0.02	0.37 ± 0.08	0.39 ± 0.07

Table 6. The classification accuracy for different latent space sizes D for each of the generative factors for the CLEVR1 paired dataset. The best result is highlighted in bold.

D	Shape	Size	Material	Color	Pos X	Pos Y
16	0.52 ± 0.33	0.51 ± 0.06	0.42 ± 0.41	0.27 ± 0.26	0.04 ± 0.02	0.04 ± 0.05
32	0.44 ± 0.08	0.58 ± 0.13	0.55 ± 0.14	0.19 ± 0.08	0.06 ± 0.02	0.09 ± 0.07
64	0.58 ± 0.11	0.73 ± 0.18	0.65 ± 0.24	0.18 ± 0.01	0.08 ± 0.03	0.10 ± 0.06
128	0.54 ± 0.09	0.91 ± 0.03	0.59 ± 0.04	0.45 ± 0.09	0.17 ± 0.05	0.25 ± 0.09
256	0.63 ± 0.06	0.94 ± 0.04	0.71 ± 0.06	0.43 ± 0.06	0.20 ± 0.10	0.33 ± 0.06
512	0.79 ± 0.07	0.96 ± 0.02	0.80 ± 0.04	0.52 ± 0.05	0.28 ± 0.09	0.39 ± 0.12
1024	0.75 ± 0.04	0.98 ± 0.01	0.89 ± 0.07	0.64 ± 0.03	0.37 ± 0.10	0.60 ± 0.03
2048	0.78 ± 0.05	0.96 ± 0.01	0.95 ± 0.01	0.70 ± 0.02	0.65 ± 0.02	0.55 ± 0.10

5. Limitations

To train ArSyD, it is necessary to assume the number of generative factors in the data, since this is a model hyperparameter. This is not a limitation when working with synthetic data, such as game and simulation environments, if the number of generative factors is known in advance. However, when working with realistic data, additional analysis of the data itself may be required to determine which level of the representation hierarchy is worth using. For example, you can describe a person’s face in terms of hairstyles and hair colors, nose shapes, eye colors, and so on, or you can go down the representation hierarchy and describe the shape of a nose, for example, in terms of tangent tilt angles to the line of the nose. The limitations associated with preliminary data analysis are not unique to our approach and are also inherent in other object-centric models, such as those that use a slot representation [23]. The number of slots in such a model is chosen as the maximum possible number of objects in the scene, with or without a background. Although the generalization of the proposed model to the case of realistic data is an interesting direction for further research (Section 6), we nevertheless demonstrate the proof-of-concept results on the CelebA dataset. The preliminary results on the CelebA paired dataset (Figure 12) demonstrate that even when realistic images differ in more than one generative factor (as is typically the case in practice), the proposed method successfully transfers generative factor values from the donor image to the target image.

Since ArSyD is independent of the object extraction method (mask generation) when working with multiple objects, it can be considered separately from the object extraction method when extending the approach to complex real-world scenes. In this case, approaches such as DINOSAUR [85], which allow for fairly accurate (compared to Slot Attention) extraction of object masks in complex realistic scenes, can be used as a model for extracting objects and/or important image elements. Another option is to use unified foundation models for promptable segmentation, such as SAM 3 [78].

The proposed metrics have the following limitations. First, it is necessary to know the number and type of generative factors in advance. Second, it is important to have a labeled dataset on which the classifiers can be trained. This can be problematic for real-world data. Furthermore, since we are using classifiers, their quality and robustness contribute significantly to the metric computation. This is further enhanced by the fact that the training is done on the original data (to avoid bias towards one or the other model being tested), and the classification for the metric is done on the reconstructed images. Another factor is the use of the decoder and its effect on the quality of the reconstruction. A better decoder is more likely to result in better classification quality. This influence is partially eliminated by using the same decoder architecture for different models and by considering relative changes in the prediction rather than absolute changes (we do not compare with the ground truth answers). Despite all these limitations, the proposed metrics are a convenient tool for comparing models that use different latent representations.

6. Discussion

An important issue is the generalization of the ArSyD to real-world data. To do this effectively, two different situations must be considered: one where the set of generative factors is known, and another where it remains unknown. In scenarios where the generative factors for each object in the dataset are known, symbolized as $G^i=G\left(O_i\right)$, where $G^i$ is a disentangled representation of an object $O_i$, we can collect a set of training examples to initially teach the model to recognize these generative factors. If there are N generative factors and two objects $O_1$ and $O_2$ differ by k generative factors, then it is necessary to simultaneously exchange k corresponding generative factors in $G^1$ and $G^2$ to train the model. Due to the significant combinatorial increase in possible combinations, this approach does not require the creation of a large dataset with a large amount of initially labeled data.

In scenarios without using labeled data, it is possible to use pre-trained neural networks, which are good feature extractors and focus on similarity when exchanging generative factors. For example, in the object-centric domain, breakthrough results on real-world data have been achieved in [85] using the training signal from the feature reconstruction of a pre-trained DINO model [86].

It is also worth noting that recent advances in visual–language models (VLMs) and foundation segmentation models enable labeling of realistic images without additional training, relying only on prompted queries. For example, models such as CLIP [87], BLIP [77,88,89], and Segment Anything Model (SAM) [78,90,91] support flexible, prompt-based annotation and segmentation of visual data.

Moreover, these models can help identify distinguishing features between images and, consequently, determine factors that differentiate them. This capability can be leveraged to construct pairs of realistic images with controlled variations, which can then be used to train ArSyD more effectively.

We consider extending our approach to the unlabeled data, where the selection of generative factors to exchange during training is based on a criterion, as future work. This could be a neural network-based criterion trained on a labeled set using examples where we know the differences or a threshold-based criterion selected from examples by a similarity function. We also want to note that in practical situations, not all possible generative factors are usually considered, but a certain reasonable subset. For example, the type of hairstyle may serve as a generative factor but not the generative factor responsible for the location of individual curls. The number of possible generative factors in localist models is also limited by the dimension of the hidden representation, and some of the coordinates may be redundant in these representations. In our approach, it is also possible to sample an initially redundant number of seed vectors, store them in item memory, and, as written above, use one of the possible criteria, e.g., cosine similarity, for the exchange.

Another aspect of ArSyD concerns its use of a fixed codebook and the structured nature of hyperdimensional representations. Since factor representations are grounded in a predefined set of high-dimensional seed vectors, the model benefits from a well-organized and stable representational space, which can improve interpretability and consistency across factors. While this introduces a degree of structure that guides learning, it also defines the representational granularity of the model, which can be particularly suitable for domains with well-defined or compositional factors but may require adaptation for highly continuous variations.

In addition, the overall performance of ArSyD depends on the interaction between the encoder, attention mechanism, and decoder components, which provides flexibility in balancing representation learning and reconstruction quality. This modular design allows the architecture to be adapted to different tasks and data complexities, although optimal performance may depend on appropriate calibration of these components. Overall, this structure supports stable learning of disentangled representations while leaving room for further architectural refinement in more complex settings.

The key component that allows bridging the gap between the localist representation obtained at the output of the encoder and the distributed representation used in HDC is the attention mechanism. The idea of an attention mechanism can be interpreted in terms of HDC. In HDC, a multiset can be represented as a simple set of vectors by summation, given that the necessary vectors appear multiple times in the sum. This is equivalent to multiplying the vector in the sum by a coefficient equal to the number of entries of that vector, or if the sum is normalized to the frequency coefficient, which corresponds to the attention weights in Equation (6). The resulting sum will be closer to the vector that appears more often in the sum.

One of the motivations for developing models that work with HDC is that it uses distributed representations that are highly resistant to noise. This potentially allows HDC models to be implemented on nanoscale hardware, a class of devices that operate at ultra-low voltages because they consist of unreliable, stochastic computational elements. These devices are highly parallel, fabricated at ultra-small scales, consume less power than conventional devices, and are very advantageous for use in wearable devices or mobile robots [92]. The use of localistic representations obtained by models based on VAE or GAN on such devices is impossible due to the fact that feature values are encoded at specific positions of the vector and adding noise to these positions can lead to a drastic changes in the properties of an encoded object.

7. Conclusions

In this article, we propose ArSyD—Architecture for Symbolic Disentanglement—which learns symbolic disentangled representations using a structured representation of an object in latent space through a weakly supervised learning procedure. This representation is based on the principles of Hyperdimensional Computing, and we represent each generative factor as a hypervector of the same dimension as the resulting representation. The object representation is then obtained as a superposition of the hypervectors responsible for the generative factors. In contrast to classical approaches based on VAE or GAN, the learned representations are distributed, i.e., individual coordinates are not interpretable (unlike localist representations). Potentially, this allows us to reduce changing the properties of an object to manipulating its latent representation using vector operations defined in Hyperdimensional Computing. We also propose a way to edit the scene by changing the properties of a single object. This is achieved by combining the Slot Attention model, which allows us to find individual objects in the scene, and the proposed model with symbolic disentangled representations. To compare models that use latent representations of different structures, we propose two metrics that allow us to compare models that use latent representations of different structures: the Disentanglement Modularity Metric and the Disentanglement Compactness Metric. The Disentanglement Modularity Metric refers to the evaluation of whether each latent unit is responsible for encoding only one generative factor. The Disentanglement Compactness Metric measures whether each generative factor is encoded by a single latent unit. The metrics are not applied directly to latent representations but to already reconstructed images. We have shown how ArSyD learns symbolic disentangled representations for objects from modified dSprites and CLEVR datasets, and we provide the proof-of-concept results on the realistic CelebA dataset.