An Empirical Study of Self-Supervised Learning with Wasserstein Distance

Abstract

In this study, we consider the problem of self-supervised learning (SSL) utilizing the 1-Wasserstein distance on a tree structure (a.k.a., Tree-Wasserstein distance (TWD)), where TWD is defined as the L1 distance between two tree-embedded vectors. In SSL methods, the cosine similarity is often utilized as an objective function; however, it has not been well studied when utilizing the Wasserstein distance. Training the Wasserstein distance is numerically challenging. Thus, this study empirically investigates a strategy for optimizing the SSL with the Wasserstein distance and finds a stable training procedure. More specifically, we evaluate the combination of two types of TWD (total variation and ClusterTree) and several probability models, including the softmax function, the ArcFace probability model, and simplicial embedding. We propose a simple yet effective Jeffrey divergence-based regularization method to stabilize optimization. Through empirical experiments on STL10, CIFAR10, CIFAR100, and SVHN, we find that a simple combination of the softmax function and TWD can obtain significantly lower results than the standard SimCLR. Moreover, a simple combination of TWD and SimSiam fails to train the model. We find that the model performance depends on the combination of TWD and probability model, and that the Jeffrey divergence regularization helps in model training. Finally, we show that the appropriate combination of the TWD and probability model outperforms cosine similarity-based representation learning.

1. Introduction

Unsupervised learning is a widely studied topic, and includes autoencoders [1] and variational autoencoders (VAEs) [2]. Self-supervised learning (SSL) algorithms, including SimCLR [3], Bootstrap Your Own Latent (BYOL) [4], MoCo [3,5], SwAV [6], SimSiam [7], and DINO [8], can also be regarded as unsupervised learning methods.

One of the main self-supervised algorithms adopts contrastive learning, in which two data points are systematically generated from a common data source, and lower-dimensional representations are found by maximizing the similarity between the positive pairs while minimizing the similarity between negative pairs. Depending on the context, positive and negative pairs can be defined differently. For example, in SimCLR [3], positive pairs correspond to images generated by independently applying different visual transformations, such as rotation and cropping. In multimodal learning, however, positive pairs are defined as the same examples corresponding in different modalities, such as images and text [9]. The flexibility of formulating positive and negative pairs also makes contrastive learning widely applicable beyond the image domain. This is a powerful pre-training method, because SSL does not require label information and can be trained using several data points.

In addition to contrastive learning-based SSL, non-contrastive approaches, such as BYOL [4], SwAV [6], and SimSiam [7], have been widely used. The fundamental concept of non-contrastive approaches involves the utilization of momentum and/or stop-gradient techniques to prevent mode collapse, as opposed to relying on negative sampling. Many of these approaches employ negative cosine similarity as a loss function. However, a limited number of SSL methods utilize distribution measures, such as cross-entropy, as exemplified by DINO [8], and simplicial embedding [10].

In this paper, leveraging the idea of distribution measures, for the first time we empirically investigate SSL performance using the Wasserstein distance. The Wasserstein distance, a widely adopted optimal transport-based distance for measuring distributional discrepancies, is useful in various machine learning tasks, including generative adversarial networks [11], document classification [12,13], image matching [14], and algorithmic fairness [15,16]. The 1-Wasserstein distance is also known as the earth mover’s distance (EMD) and the word mover’s distance (WMD) [12].

In this study, we consider an SSL framework with a 1-Wasserstein distance under a tree metric (i.e., Tree-Wasserstein distance (TWD)) [17,18]. TWD includes the sliced Wasserstein distance [19,20] and total variation as special cases, and can be represented by the ${\ell }_1$ distance between two vectors. Due to the fact that TWD is given as a non-differentiable function, learning simplicial representations through back-propagation of TWD is challenging. Moreover, because the Wasserstein distance is computed from probability vectors, and several representations of probability vectors exist, it is difficult to determine which is most suitable for SSL training. Hence, we investigate a combination of probability models and the structure of TWD. Specifically, we consider the total variation and ClusterTree for TWD structure and show that the total variation is equivalent to a robust variant of TWD. In terms of the probability representations, we propose the combined use of softmax, an ArcFace-based probability model [21], and simplicial embedding (SEM) [10]. Finally, to stabilize the training, we propose a Jeffrey divergence-based regularization. Through SSL experiments, we find that the standard softmax formulation with back-propagation yields poor results. In particular, the non-contrastive SSL case fails to train the model with a simple combination of the Wasserstein distance and softmax function. For total variation, the ArcFace-based model performs well. By contrast, SEM is suitable for ClusterTree, whereas ArcFace-based models achieve modest performance. Moreover, the proposed regularization significantly outperforms its non-regularized counterparts.

Contribution: The contributions of this study are summarized below:

• We propose to use the tree Wasserstein distance for self-supervised learning including SimCLR and SimSiam for the first time.
• We investigate the combination of probability models and TWD (total variation and ClusterTree). We find that the ArcFace model with prior information is suited for total variation, while SEM [10] is suited for ClusterTree models.
• We propose a robust variant of TWD (RTWD) and show that RTWD is equivalent to total variation.
• We propose the Jeffrey divergence regularization for TWD minimization, and find that the regularization significantly stabilizes training.
• We demonstrate that the combination of TWD and probability models can obtain better performance in self-supervised training for CIFAR10, STL10, and SVHN compared to the cosine similarity in SimCLR experiments, while the performance of CIFAR100 can be improved further in the future.

2. Related Work

The proposed method involves unsupervised representation learning and optimal transport.

Unsupervised Representation Learning: Representation learning is an important research topic in machine learning and involves several methods. The autoencoder [1] and its variational version [2] are widely employed in unsupervised representation learning methods. Current mainstream SSL approaches are based on a cross-view prediction framework [22] and contrastive learning has emerged as a prominent SSL paradigm.

In contrastive learning, a model learns by contrasting positive samples (similar instances) with negative samples (dissimilar instances) using methods such as SimCLR [23]. SimCLR employs data augmentation and similarity metrics to encourage the model to project similar instances close together while pushing dissimilar instances apart. This approach has demonstrated efficacy across various domains, including computer vision and natural language processing, thus enabling learning without explicit labels. SimCLR employs the InfoNCE loss [24]. Subsequently to SimCLR, several alternative approaches have been proposed, including the use of Barlow Twins [25]. The Barlow twin loss function attempts to maximize the correlation between positive pairs while minimizing the cross-correlation with negative samples. Barlow Twins is closely related to the Hilbert–Schmidt independence criterion, a kernel-based independence measure [26,27].

One drawback of SimCLR is its reliance on numerous negative samples. To address this issue, recent research has focused on approaches that eliminate the need for negative sampling, such as BYOL [4], SwAV [6], and DINO [8]. For example, BYOL demonstrates training of representations by minimizing the loss between online and target networks. The target network is formed by maintaining a moving average of the online network parameters, and eliminates the need for negative samples. Surprisingly, BYOL showed favorable results compared with SimCLR. SimSiam, introduced by Chen and He [7], utilizes a Siamese network to train the estimation by fixing one of the networks using a stop gradient.

Both of these approaches concentrate on learning low-dimensional representations with real-valued vector embeddings by employing cosine similarity as a similarity measure in contrastive learning. Recently, Lavoie et al. [10] proposed simplicial embedding (SEM), which involves multiple concatenated softmax functions and learns high-dimensional sparse non-negative representations. This innovation significantly enhances classification accuracy.

All of these approaches employ either a negative cosine similarity or cross-entropy as a loss function. In contrast, use of the Wasserstein distance in this context has not been studied.

Divergence and optimal transport: Measuring the divergence between two probability distributions is a fundamental research problem in machine learning. It has utility for various downstream applications, including document classification [12,13], image matching [14], and algorithmic fairness [15,16]. One widely adopted divergence measure is Kullback–Leibler (KL) divergence [28]. However, KL divergence can diverge to infinity when the supports of the two input probability distributions do not overlap.

The Wasserstein distance, or, as it is known in the computer vision community, EMD, can handle differences in supports between probability distributions. Another key advantages of the Wasserstein distance over KL is that it can identify matches between the data samples. For example, Sarlin et al. [14] proposed SuperGlue, leveraging optimal transport for correspondence determination in local feature sets.

In NLP, Kusner et al. [12] introduced WMD, a Wasserstein distance pioneer in textual similarity tasks that is widely utilized, including for text generation evaluation [29]. Sato et al. [13] further studied the properties of WMD. Another interesting approach is the word rotator distance (WRD) [30], which utilizes the norm of word vectors as a simplicial representation and significantly improves WMD’s performance. However, these methods incur cubic-order computational costs, rendering them unsuitable for extensive distribution-comparison tasks.

The Wasserstein distance can be computed efficiently via linear programming (cubic complexity). However, to speed up EMD and Wasserstein distance computation, Cuturi [31] introduced the Sinkhorn algorithm, which solves the entropic regularized optimization problem and achieves quadratic order Wasserstein distance computation ($O\left({\overline{n}}^2\right)$), where $\overline{n}$ is the number of data points. Moreover, because the optimal solution from the Sinkhorn algorithm can be obtained using an iterative algorithm, it can be easily incorporated into deep learning applications, making optimal transport widely applicable. One limitation of the Sinkhorn algorithm is that it still requires quadratic-time computation, and the final solution depends highly on the regularization parameter.

An alternative approach is the sliced Wasserstein distance (SWD) [19,20], which solves the optimal transport problem within a projected one-dimensional subspace. The algorithm for Wasserstein distance computation over reals essentially applies sorting as a subroutine; thus, SWD offers $O(\overline{n}log\overline{n})$ computation. SWD’s extensions include the generalized sliced Wasserstein distance for multidimensional cases [32]; the max-sliced Wasserstein distance, which determines the optimal transport-enhancing 1D subspace [33,34]; and the subspace-robust Wasserstein distance [35].

The 1-Wasserstein distance with a tree metric (also known as the Tree-Wasserstein Distance (TWD)) is a generalization of the sliced Wasserstein distance and total variation [17,18,36]. The TWD is also known as the UniFrac distance [37] and is assumed to have a phylogenetic tree beforehand. An important property of TWD is that TWD has an analytical solution for the L1 distance of tree-embedded vectors.

Originally, TWD was studied in theoretical computer science, known as the QuadTree algorithm [17]. This has recently been extended by the ML community to include unbalanced TWD [38,39], supervised Wasserstein training [40], tree barycenters [41], robust Wasserstein distance [42], unsupervised tree construction [43], and greedy matching [44]. Moreover, graph-based optimal transport has also been studied recently [45,46] and has been used for many applications including natural language processing [47,48] and single-cell analysis [49].

These approaches focus on approximating the 1-Wasserstein distance through tree construction and often utilize constant-edge weights. In terms of approaches that consider non-uniform edge weights, Backurs et al. [50] introduced FlowTree, amalgamating QuadTree and cost matrix methods, outperforming QuadTree. They guaranteed that QuadTree and FlowTree approximate nearest neighbors by employing the 1-Wasserstein distance. Dey and Zhang [51] proposed an L1-embedding for approximating the 1-Wasserstein distance for persistence diagrams. Finally, Yamada et al. [52] proposed a computationally efficient tree weight estimation technique for TWD and empirically demonstrated that TWD can attain comparable performance to the Wasserstein distance, while achieving computational speeds several orders of magnitude faster than linear programming computation of the Wasserstein distance.

Most existing studies on TWD have focused on tree construction [17,18,40] and edge weight estimation [52]. Frogner et al. [53] and Toyokuni et al. [54] considered utilizing the Wasserstein distance for multi-label classification. These studies focused on supervised learning by employing softmax as the probability model. In this study, we investigate the Wasserstein distance by employing an SSL framework and evaluate various probability models.

3. Background

3.1. Self-Supervised Learning Methods

SimCLR [3]: Given n input vectors ${\left\{{\mathit{x}}_i\right\}}_{i=1}^n$, where ${\mathit{x}}_i\in {\mathbb{R}}^d$, define the data transformation functions ${\mathit{u}}^{\left(1\right)}={\varphi }_1\left(\mathit{x}\right)\in {\mathbb{R}}^d$ and ${\mathit{u}}^{\left(2\right)}={\varphi }_2\left(\mathit{x}\right)\in {\mathbb{R}}^d$. In the context of image applications, ${\mathit{u}}^{\left(1\right)}$ and ${\mathit{u}}^{\left(2\right)}$ can be understood as two image transformations over a given image: translation, rotation, blurring, etc. The neural network model is denoted as ${\mathit{f}}_{\mathit{\theta }}\left(\mathit{u}\right)\in {\mathbb{R}}^{d_{\mathrm{out}}}$, where $\mathit{\theta }$ is a learnable parameter.

SimCLR attempts to train the model by learning features such that ${\mathit{z}}^{\left(1\right)}={\mathit{f}}_{\mathit{\theta }}\left({\mathit{u}}^{\left(1\right)}\right)$ and ${\mathit{z}}^{\left(2\right)}={\mathit{f}}_{\mathit{\theta }}\left({\mathit{u}}^{\left(2\right)}\right)$ are close after the feature mapping, while ensuring that both are distant from the feature map of ${\mathit{u}}^{\prime }$, where ${\mathit{u}}^{\prime }$ is a negative sample generated from a different input image. To this end, InfoNCE loss [24] is employed in the SimCLR model:

$$\begin{array}{cc}\hfill {\ell }_{\mathrm{InfoNCE}}\left({\mathit{z}}_i^{\left(1\right)},{\mathit{z}}_i^{\left(2\right)}\right)& =-log{\displaystyle \frac{exp\left(\mathrm{sim}\left({\mathit{z}}_i^{\left(1\right)},{\mathit{z}}_i^{\left(2\right)}\right)/\tau \right)}{\overline{Z}}}\hfill \\ \hfill & =-\mathrm{sim}({\mathit{z}}_i^{\left(1\right)},{\mathit{z}}_i^{\left(2\right)})/\tau +log\left(\overline{Z}\right),\hfill \end{array}$$

where $\overline{Z}={\sum }_{k=1}^{2R}{\delta }_{k\ne i}exp(\mathrm{sim}({\mathit{z}}_i^{\left(1\right)},{\tilde{\mathit{z}}}_k)/\tau )$ is the normalizer, R is the batch size and $\mathrm{sim}(\mathit{z},{\mathit{z}}^{\prime })$ is a similarity function that takes a higher positive value when $\mathit{z}$ and ${\mathit{z}}^{\prime }$ are similar and a smaller (positive or negative) value when $\mathit{z}$ and ${\mathit{z}}^{\prime }$ are dissimilar. $\tau$ is the temperature parameter, and ${\delta }_{k\ne i}$ is a delta function that takes a value of 1 when $k\ne i$ and 0 otherwise. In contrastive learning, we aim to minimize the InfoNCE loss function. To achieve an optimal solution, we need to maximize the similarity $\mathrm{sim}\left({\mathit{z}}_i^{\left(1\right)},{\mathit{z}}_i^{\left(2\right)}\right)$ while minimizing $log\left(Z\right)$. The first term aims to make ${\mathit{z}}_i^{\left(1\right)}$ and ${\mathit{z}}_i^{\left(2\right)}$ as similar as possible. The second term is a log-sum-exp function, which can be interpreted for small $\tau$ as

$$\begin{array}{cc}\hfill log\left(Z\right)& =log\left(\sum _{k=1}^{2R}{\delta }_{k\ne i}exp(\mathrm{sim}({\mathit{z}}_i^{\left(1\right)},{\tilde{\mathit{z}}}_k)/\tau )\right),\hfill \\ \hfill & \approx \underset{k}{max}\left(\mathrm{sim}({\mathit{z}}_i^{\left(1\right)},{\tilde{\mathit{z}}}_k)\right).\hfill \end{array}$$

By minimizing $log\left(Z\right)$, we can make ${\mathit{z}}_i^{\left(1\right)}$ dissimilar to the negative samples ${\tilde{\mathit{z}}}_k$. Due to the fact that we attempt to minimize the maximum similarity between input ${\mathit{z}}_i$ and its negative samples, we can make ${\mathit{z}}_i$ and its negative samples dissimilar via the second term.

In SimCLR, the parameters are learned by minimizing the InfoNCE loss.

$$\begin{array}{c}\hfill \widehat{\mathit{\theta }}:=\underset{\mathit{\theta }}{\mathrm{argmin}}\sum _{i=1}^n{\ell }_{\mathrm{InfoNCE}}\left({\mathit{f}}_{\mathit{\theta }}\left({\mathit{u}}_i^{\left(1\right)}\right),{\mathit{f}}_{\mathit{\theta }}\left({\mathit{u}}_i^{\left(2\right)}\right)\right).\end{array}$$

SimSiam [7]: SimSiam is a non-contrastive learning method; it does not use negative sampling to prevent mode collapse. In place of negative sampling, SimSiam employs a stop-gradient method. Specifically, the loss function is given by

$$\begin{array}{cc}\hfill & L_{SimSiam}\left(\mathit{\theta }\right)={\displaystyle \frac{1}{2}}{L}_1\left(\mathit{\theta }\right)+{\displaystyle \frac{1}{2}}{L}_2\left(\mathit{\theta }\right),\hfill \\ \hfill & L_1\left(\mathit{\theta }\right)=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{\mathit{h}{\left({\mathit{z}}_i\right)}^{\top }{\overline{\mathit{z}}}_i^{\prime }}{\parallel \mathit{h}\left({\mathit{z}}_i\right){\parallel }_2{\parallel {\overline{\mathit{z}}}_i^{\prime }\parallel }_2}},L_2\left(\mathit{\theta }\right)=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{{\overline{\mathit{z}}}_i^{\top }\mathit{h}\left({\mathit{z}}_i^{\prime }\right)}{\parallel {\overline{\mathit{z}}}_i{\parallel }_2{\parallel \mathit{h}\left({\mathit{z}}_i^{\prime }\right)\parallel }_2}},\hfill \end{array}$$

where $\mathit{h}(·)$ is the MLP head, ${\mathit{z}}_i$ is a latent variable, and ${\overline{\mathit{z}}}_i=\mathrm{StopGradient}\left({\mathit{z}}_i\right)$ is a latent variable with a stop gradient.

3.2. p-Wasserstein Distance

The p-Wasserstein distance between two discrete measures, $\mu ={\sum }_{i=1}^{\overline{n}}{a}_i{\delta }_{{\mathit{x}}_i}$ and ${\mu }^{\prime }={\sum }_{j=1}^{\overline{m}}{a}_j^{\prime }{\delta }_{{\mathit{y}}_j}$ is given by

$$\begin{array}{c}\hfill {\mathcal{W}}_p(\mu ,{\mu }^{\prime })={\left(\underset{\Pi \in U(\mu ,{\mu }^{\prime })}{min}\sum _{i=1}^{\overline{n}}\sum _{j=1}^{\overline{m}}{\pi }_{ij}d{({\mathit{x}}_i,{\mathit{y}}_j)}^p\right)}^{1/p},\end{array}$$

where $U(\mu ,{\mu }^{\prime })$ denotes the set of transport plans and $U(\mu ,{\mu }^{\prime })=\{\Pi \in {\mathbb{R}}_{+}^{\overline{n}\times \overline{m}}:\Pi {\mathit{1}}_{\overline{m}}=\mathit{a},{\Pi }^{\top }{\mathit{1}}_{\overline{n}}={\mathit{a}}^{\prime }\}$. The Wasserstein distance can be computed using a linear program. However, because this includes an optimization problem, the computation of Wasserstein distance for each iteration is computationally expensive.

3.3. 1-Wasserstein Distance with Tree Metric (Tree-Wasserstein Distance)

Another 1-Wasserstein distance is based on trees [17,18]. The 1-Wasserstein distance between two probability distributions $\mu ={\sum }_{i=1}^{N_{\mathrm{leaf}}}{a}_i{\delta }_{{\mathit{x}}_i}$ and ${\mu }^{\prime }={\sum }_{j=1}^{N_{\mathrm{leaf}}}{a}_j^{\prime }{\delta }_{{\mathit{y}}_j}$ with the tree metric is defined as

$$\begin{array}{c}\hfill W_{\mathcal{T}}(\mu ,{\mu }^{\prime })=\underset{\Pi \in U(\mu ,{\mu }^{\prime })}{min}\sum _{i=1}^{N_{\mathrm{leaf}}}\sum _{j=1}^{N_{\mathrm{leaf}}}{\pi }_{ij}{d}_{\mathcal{T}}({\mathit{x}}_i,{\mathit{y}}_j),\end{array}$$

(1)

where $d_{\mathcal{T}}(\mathit{x},\mathit{y})$ is the length of the shortest path between $\mathit{x}$ and $\mathit{y}$ on the tree and $N_{\mathrm{leaf}}$ is the number of leaf nodes. TWD can be further represented by the closed form as follows [18]:

$$\begin{array}{c}\hfill W_{\mathcal{T}}(\mu ,{\mu }^{\prime })=\sum _{e\in E}{w}_e|\mu (\Gamma \left(v_e\right))-{\mu }^{\prime }(\Gamma \left(v_e\right))|,\end{array}$$

(2)

where e is an edge index, $w_e\in {\mathbb{R}}_{+}$ is the edge weight of edge e, $v_e$ is the eth node index, and $\mu (\Gamma \left(v_e\right))$ is the total mass of the subtree with root $v_e$. This closed form solution can be further represented as the L1 distance [40]:

$$\begin{array}{cc}\hfill W_{\mathcal{T}}(\mu ,{\mu }^{\prime })& =\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}\mathit{a}-\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}^{\prime }{\parallel }_1,\hfill \end{array}$$

where $\mathit{B}\in {\{0,1\}}^{N_{\mathrm{node}}\times N_{\mathrm{leaf}}}$ is a tree parameter, ${\left[\mathit{B}\right]}_{i,j}=1$ if node i is the ancestor node of leaf node j and zero otherwise, $N_{\mathrm{node}}$ is the total number of nodes of a tree, and $\mathit{w}\in {\mathbb{R}}_{+}^{N_{\mathrm{node}}}$ is the edge weight.

For illustration, we provide two examples to demonstrate the B matrix by considering a tree with a depth of one and a ClusterTree, as shown in Figure 1. If all edge weights $w_1=w_2=\dots =w_N={\displaystyle \frac{1}{2}}$ in the left panel of Figure 1, then the $\mathit{B}$ matrix is given as $\mathit{B}=\mathit{I}$. By substituting this result into the TWD, we obtain

$$\begin{array}{c}\hfill W_{\mathcal{T}}(\mu ,{\mu }^{\prime })={\displaystyle \frac{1}{2}}\parallel \mathit{a}-{\mathit{a}}^{\prime }{\parallel }_1={\parallel \mathit{a}-{\mathit{a}}^{\prime }\parallel }_{\mathrm{TV}}.\end{array}$$

Figure 1. Left tree corresponds to the total variation if we set the weight as $w_i={\displaystyle \frac{1}{2}},\forall i$. Right tree is a ClusterTree (2 class).

Thus, the total variation is a special case of TWD. In this setting, the shortest-path distance in the tree corresponds to the Hamming distance. Note that Raginsky et al. [55] also assert that the 1-Wasserstein distance with the Hamming metric $d(\mathit{x},\mathit{y})={\delta }_{\mathit{x}\ne \mathit{y}}$ is equivalent to the total variation (Proposition 3.4.1 in Raginsky et al. [55]).

The key advantage of the tree-based approach is that the Wasserstein distance is written in closed form, which is computationally efficient. A chain is included as a special case in the tree. Thus, the widely employed sliced Wasserstein distance is also included as a special case of TWD (Figure 2). Moreover, it has been empirically reported that TWD- and Sinkhorn-based approaches perform similarly in multilabel classification tasks [54].

Figure 2. Tree for sliced Wasserstein distance for $N_{\mathrm{leaf}}=3$. The left figure is a chain and the right figure is the tree representation with internal nodes for the chain ($w_4=w_5=w_6=0$).

4. SSL with 1-Wasserstein Distance

In this section, we first formulate SSL using TWD. We then introduce ArcFace-based probability models and Jeffrey divergence regularization.

4.1. SimCLR with Tree Wasserstein Distance

Let $\mathit{a}$ and ${\mathit{a}}^{\prime }$ be the embedding vectors of $\mathit{x}$ and ${\mathit{x}}^{\prime }$ (i.e., ${\mathit{1}}^{\top }\mathit{a}=1$ and ${\mathit{1}}^{\top }{\mathit{a}}^{\prime }$) with $\mu ={\sum }_j{a}_j{\delta }_{{\mathit{e}}_j}$ and ${\mu }^{\prime }={\sum }_j{a}_j^{\prime }{\delta }_{{\mathit{e}}_j}$, respectively. Here, ${\mathit{e}}_j$ is the virtual embedding corresponding to $a_j$ or $a_j^{\prime }$. $\mathit{e}$ is assumed unavailable in the problem setup. The main idea of this paper is to adopt the negative Wasserstein distance between $\mu$ and ${\mu }^{\prime }$ as the similarity score for SimCLR.

$$\begin{array}{cc}\hfill \mathrm{sim}(\mu ,{\mu }^{\prime })& =-W_{\mathcal{T}}(\mu ,{\mu }^{\prime }).\hfill \end{array}$$

We assume that $\mathit{B}$ and $\mathit{w}$ are given; that is, both the tree structure and weights are known. In particular, we consider the trees shown in Figure 1.

Following the original design of the InfoNCE loss and by substituting the similarity score given by the negative Wasserstein distance, we obtain the following simplified loss function:

$$\begin{array}{cc}\hfill \widehat{\mathit{\theta }}& :=\underset{\mathit{\theta }}{\mathrm{argmin}}\sum _{i=1}^n\left(W_{\mathcal{T}}({\mu }_i^{\left(1\right)},{\mu }_i^{\left(2\right)})/\tau +log\sum _{k=1}^{2N}{\delta }_{k\ne i}exp\left(-W_{\mathcal{T}}({\mu }_i^{\left(1\right)},{\mu }_k^{\left(2\right)})/\tau \right)\right),\hfill \end{array}$$

where $\tau >0$ is the temperature parameter for the InfoNCE loss. Although we mainly focus on the InfoNCE loss, the proposed negative Wasserstein distance as a measure of similarity can be used in other contrastive losses as well, e.g., the Barlow Twins.

4.2. SimSiam with Tree Wasserstein Distance

Here, we consider a combination of SimSiam and TWD. The loss function of the proposed approach is expressed as

$$\begin{array}{cc}\hfill & L_{\mathrm{TWDSimsiam}}\left(\mathit{\theta }\right)={\displaystyle \frac{1}{2}}{L}_1\left(\mathit{\theta }\right)+{\displaystyle \frac{1}{2}}{L}_2\left(\mathit{\theta }\right),\hfill \\ \hfill & L_1\left(\mathit{\theta }\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{W}_{\mathcal{T}}\left({\mu }_i^{\left(1\right)},{\overline{\mu }}_i^{\left(2\right)}\right),L_2\left(\mathit{\theta }\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{W}_{\mathcal{T}}\left({\overline{\mu }}_i^{\left(1\right)},{\mu }_i^{\left(2\right)}\right).\hfill \end{array}$$

The distinction to the original SimSiam is that our formulation employs the Wasserstein distance, whereas the original formulation uses cosine similarity.

4.3. Robust Variant of Tree Wasserstein Distance

In our setup, it is difficult to estimate the tree structure $\mathit{B}$ and edge weight $\mathit{w}$ because the embedding vectors ${\mathit{e}}_1,{\mathit{e}}_2,\dots ,{\mathit{e}}_{d_{\mathrm{out}}}$ are unavailable. To address this problem, we consider a robust estimation of the Wasserstein distance, such as the subspace-robust Wasserstein distance (SRWD) [35], for TWD. The key idea of SRWD is to solve an optimal transport problem in a subspace in which the distance is maximized. In the TWD case, we can consider solving the optimal transport problem for the maximum shortest-path distance. Specifically, for a given $\mathit{B}$, we propose the robust TWD (RTWD) as follows:

$$\begin{array}{c}\hfill \mathrm{RTWD}(\mu ,{\mu }^{\prime })={\displaystyle \frac{1}{2}}\underset{\Pi \in U(\mu ,{\mu }^{\prime })}{min}\underset{\mathit{w}\in \mathcal{B}}{max}\sum _{i=1}^{N_{\mathrm{leafs}}}\sum _{j=1}^{N_{\mathrm{leafs}}}{\pi }_{ij}{d}_{\mathcal{T}}({\mathit{e}}_i,{\mathit{e}}_j),\end{array}$$

where $\mathcal{B}=\{\mathit{w}\in {\mathbb{R}}_{+}^{N_{\mathrm{leaf}}}:{\mathit{B}}^{\top }\mathit{w}=\mathit{1}\mathrm{and}\mathit{w}\ge \mathit{0}\}$, $d_{\mathcal{T}}({\mathit{e}}_i,{\mathit{e}}_j)$ is the shortest-path distance between ${\mathit{e}}_i$ and ${\mathit{e}}_j$, and ${\mathit{e}}_i$ and ${\mathit{e}}_j$ are embedded in a tree $\mathcal{T}$. This constraint implies that the weights of the ancestor node of leaf node j are non-negative and sum to one.

Proposition 1. 

The robust variant of TWD (RTWD) is equivalent to total variation:

$$\begin{array}{c}\hfill \mathrm{RTWD}(\mu ,{\mu }^{\prime })={\parallel \mathit{a}-{\mathit{a}}^{\prime }\parallel }_{\mathrm{TV}},\end{array}$$

where $\parallel \mathit{a}-{\mathit{a}}^{\prime }{\parallel }_{\mathrm{TV}}={\displaystyle \frac{1}{2}}{\parallel \mathit{a}-{\mathit{a}}^{\prime }\parallel }_1$ denotes the total variation.

Proof. 

Let $\mathit{B}\in {\{0,1\}}^{N\times N_{\mathrm{leaf}}}=[{\mathit{b}}_1,{\mathit{b}}_2,\dots ,{\mathit{b}}_{N_{\mathrm{leaf}}}]$ and ${\mathit{b}}_i\in {\{0,1\}}^N$. The shortest-path distance between leaves i and j can be represented as [52]

$$\begin{array}{cc}\hfill d_{\mathcal{T}}({\mathit{e}}_i,{\mathit{e}}_j)& ={\mathit{w}}^{\top }({\mathit{b}}_i+{\mathit{b}}_j-2{\mathit{b}}_i\circ {\mathit{b}}_j).\hfill \end{array}$$

That is, $d_{\mathcal{T}}({\mathit{e}}_i,{\mathit{e}}_j)$ is represented by a linear function with respect to $\mathit{w}$ for a given $\mathit{B}$ and the constraints on $\mathit{w}$ and $\Pi$ are convex. Thus, strong duality holds, and we obtain the following representation using the minimax theorem [56,57]:

$$\begin{array}{cc}\hfill \mathrm{RTWD}(\mu ,{\mu }^{\prime })& ={\displaystyle \frac{1}{2}}\underset{\mathit{w}\mathrm{s}.\mathrm{t}.{\mathit{B}}^{\top }\mathit{w}=\mathit{1}\mathrm{and}\mathit{w}\ge \mathit{0}}{max}\underset{\Pi \in U(\mathit{a},{\mathit{a}}^{\prime })}{min}\sum _{i=1}^{N_{\mathrm{leafs}}}\sum _{j=1}^{N_{\mathrm{leafs}}}{\pi }_{ij}{\mathit{w}}^{\top }({\mathit{b}}_i+{\mathit{b}}_j-2{\mathit{b}}_i\circ {\mathit{b}}_j)\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}\underset{\mathit{w}\mathrm{s}.\mathrm{t}.{\mathit{B}}^{\top }\mathit{w}=\mathit{1}\mathrm{and}\mathit{w}\ge \mathit{0}}{max}{\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime })\parallel }_1,\hfill \end{array}$$

where $\mathrm{TWD}(\mu ,{\mu }^{\prime })={min}_{\Pi \in U(\mathit{a},{\mathit{a}}^{\prime })}{\sum }_{i=1}^{N_{\mathrm{leafs}}}{\sum }_{j=1}^{N_{\mathrm{leafs}}}{\pi }_{ij}{d}_{\mathcal{T}}({\mathit{e}}_i,{\mathit{e}}_j)={\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime })\parallel }_1$.

Without loss of generality, we consider $w_0=0$. First, we rewrite the norm $\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime }){\parallel }_1$ as

$$\begin{array}{c}\hfill \parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime }){\parallel }_1=\sum _{j=1}^N{w}_j\left|\sum _{k\in \left[N_{\mathrm{leafs}}\right],k\in de\left(j\right)}(a_k-a_k^{\prime })\right|,\end{array}$$

where $de\left(j\right)$ denotes the set of descendants of node $j\in \left[N\right]$ (including itself). Using the triangle inequality, we obtain

$$\begin{array}{cc}\hfill \parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime }){\parallel }_1& \le \sum _{j=1}^N{w}_j\sum _{k\in \left[N_{\mathrm{leafs}}\right],k\in de\left(j\right)}|a_k-a_k^{\prime }|\hfill \\ \hfill & =\sum _{k\in \left[N_{\mathrm{leafs}}\right]}|a_k-a_k^{\prime }|\sum _{j\in \left[N\right],j\in pa\left(k\right)}{w}_j,\hfill \end{array}$$

where $pa\left(k\right)$ is the set of ancestors of leaf k (including itself). By rewriting the constraint ${\mathit{B}}^{\top }\mathit{w}=\mathit{1}$ as ${\sum }_{j\in \left[N\right],j\in pa\left(k\right)}{w}_j=1$ for any $k\in \left[N_{\mathrm{leafs}}\right]$, we obtain

$$\begin{array}{c}\hfill \parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime }){\parallel }_1\le \sum _{k\in \left[N_{\mathrm{leafs}}\right]}|a_k-a_k^{\prime }|=\parallel \mathit{a}-{\mathit{a}}^{\prime }{\parallel }_1.\end{array}$$

The latter inequality holds for any weight vector $\mathit{w}$. Therefore, considering the vector such that $w_j=1$ if $j\in \left[N_{\mathrm{leafs}}\right]$ and 0 otherwise, which satisfies the constraint ${\mathit{B}}^{\top }\mathit{w}=\mathit{1}$, we obtain

$$\begin{array}{c}\hfill \parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}(\mathit{a}-{\mathit{a}}^{\prime }){\parallel }_1=\sum _{k=1}^{N_{\mathrm{leafs}}}|a_k-a_k^{\prime }|=\parallel \mathit{a}-{\mathit{a}}^{\prime }{\parallel }_1.\end{array}$$

This completes the proof of the proposition. □

Based on this proposition, RTWD is equivalent to the total variation and does not depend on the tree structure $\mathit{B}$. That is, if we do not have prior information about the tree structure, using the total variation is a reasonable choice.

4.4. Probability Models

In this section, we discuss several choices of probability models for InfoNCE loss and SimSiam loss.

Softmax: The embedded vector with softmax function is given by

$$\begin{array}{c}\hfill {\mathit{a}}_{\mathit{\theta }}\left(\mathit{x}\right)=\mathrm{Softmax}\left({\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)\right),\end{array}$$

where ${\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)$ is a neural network model.

Simplicial Embedding: Lavoie et al. [10] proposed a simple yet efficient simplicial embedding method. Assume that the output dimensionality of a neural network model is $d_{\mathrm{out}}$. Then, SEM applies the softmax function to each V-dimensional vector of ${\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)$, where we have $L=d_{\mathrm{out}}/V$ probability vectors. The ℓth softmax function is thus defined as follows:

$$\begin{array}{cc}\hfill & {\mathit{a}}_{\mathit{\theta }}\left(\mathit{x}\right)={\left[{\mathit{a}}_{\mathit{\theta }}^{\left(1\right)}{\left(\mathit{x}\right)}^{\top },{\mathit{a}}_{\mathit{\theta }}^{\left(2\right)}{\left(\mathit{x}\right)}^{\top },\dots ,{\mathit{a}}_{\mathit{\theta }}^{\left(L\right)}{\left(\mathit{x}\right)}^{\top }\right]}^{\top }\hfill \\ \hfill & \mathrm{with}{\mathit{a}}_{\mathit{\theta }}^{(\ell )}\left(\mathit{x}\right)=\mathrm{Softmax}\left({\mathit{f}}_{\mathit{\theta }}^{(\ell )}\left(\mathit{x}\right)\right)/L,\hfill \end{array}$$

where ${\mathit{f}}_{\mathit{\theta }}^{(\ell )}\left(\mathit{x}\right))\in {\mathbb{R}}^V$ is the ℓ-th block of a neural network model. We normalize the softmax function by L because ${\mathit{a}}_{\mathit{\theta }}\left(\mathit{x}\right)$ must satisfy the sum-to-one constraint. Note that the softmax function can be regarded as a special case of simplicial embedding (where $L=1$). In simplicial embedding, the softmax function is applied separately to each subset of the elements. For example, if $d_{\mathrm{out}}=10$ and $V=5$, the softmax function is applied to each of the two five-dimensional vectors, and the results are then concatenated.

ArcFace model (AF): In comparison to SEM, we propose to employ the ArcFace probability model [21]. The ArcFace models employs cosine similarity in addition to softmax.

$$\begin{array}{c}\hfill {\mathit{a}}_{\mathit{\theta }}\left(\mathit{x}\right)=Softmax\left({\mathit{K}}^{\top }{\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)/\eta \right),\end{array}$$

where $\mathit{K}=[{\mathit{k}}_1,{\mathit{k}}_2,\dots ,{\mathit{k}}_{d_{\mathrm{out}}}]\in {\mathbb{R}}^{d_{\mathrm{out}}\times d_{\mathrm{prob}}}$ is a learning parameter, ${\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)$ is the normalized output of a model (${\mathit{f}}_{\mathit{\theta }}{\left(\mathit{x}\right)}^{\top }{\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)=1$), and $\eta$ is the temperature parameter. Note that AF has a structure similar to that of transformers [58,59]. The key difference from the original notion of attention in transformers is the normalization of the key matrix $\mathit{K}$ and query vector ${\mathit{f}}_{\mathit{\theta }}\left(\mathit{x}\right)$.

AF with Positional Encoding: To the AF model, one can add one more linear layer and then apply the softmax function; then, the output is similar to the standard softmax function. Here, we propose replacing the key matrix with a normalized positional encoding matrix (${\mathit{k}}_i^{\top }{\mathit{k}}_i=1,\forall i$):

$$\begin{array}{cc}\hfill {\mathit{k}}_i& ={\overline{\mathit{k}}}_i/\parallel {\overline{\mathit{k}}}_i{\parallel }_2,\hfill \end{array}$$

where ${\overline{\mathit{k}}}_i^{\left(2j\right)}=sin(i/{10,000}^{2j/d_{\mathrm{out}}})$ and ${\overline{\mathit{k}}}_i^{(2j+1)}=cos(i/{10,000}^{2j/d_{\mathrm{out}}})$.

AF with Discrete Cosine Transform Matrix: Another natural approach would be to utilize an orthogonal matrix as $\mathit{K}$. Therefore, we propose adopting a discrete cosine transform (DCT) [60] matrix as $\mathit{K}$, where DCT is in general used for data compression for images. The DCT matrix is expressed as follows [60]:

$$\begin{array}{c}\hfill {\mathit{k}}_i^{\left(j\right)}=\left\{\begin{array}{cc}1/\sqrt{d_{\mathrm{out}}}\hfill & (i=0)\hfill \\ \sqrt{{\displaystyle \frac{2}{d_{\mathrm{out}}}}}cos{\displaystyle \frac{\pi (2j+1)i}{2d_{\mathrm{out}}}}\hfill & (1\le i\le d_{\mathrm{out}})\hfill \end{array}\right..\end{array}$$

One of the contributions of this study is the finding that combining positional encoding and the DCT matrix with the ArcFace model significantly boosts performance, whereas the standard ArcFace model without these additions performs similarly to the softmax function.

4.5. Jeffrey Divergence Regularization

We empirically observed that optimizing the loss function described above is extremely challenging. In particular, the L1 distance cannot be differentiated at 0. Figure 3b illustrates the learning curve for standard optimization using the softmax function model.

Figure 3. InfoNCE loss and Top-1 (Train) comparisons on STL10 dataset.

To stabilize optimization, we propose including the Jeffrey divergence (JD) as a regularization term. JD is an upper bound of the square of the 1-Wasserstein distance.

Proposition 2. 

For ${\mathit{B}}^{\top }\mathit{w}=\mathit{1}$ and probability vectors ${\mathit{a}}_i$ and ${\mathit{a}}_j$, we have

$$\begin{array}{c}\hfill W_{\mathcal{T}}^2({\mu }_i,{\mu }_j)\le \mathrm{JD}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j),\end{array}$$

where

$$\begin{array}{cc}\hfill \mathrm{JD}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j)& =\mathrm{KL}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j)\hfill \\ \hfill & \phantom{=}+\mathrm{KL}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i)\hfill \end{array}$$

is a Jeffrey divergence.

Proof. 

The following holds if ${\mathit{B}}^{\top }\mathit{w}=\mathit{1}$ with the probability vector $\mathit{a}$ (such that ${\mathit{a}}^{\top }\mathit{1}=1$).

$$\begin{array}{c}\hfill {\mathit{1}}^{\top }\mathrm{diag}\left(\mathit{w}\right)\mathit{B}\mathit{a}=1.\end{array}$$

Then, using Pinsker’s Inequality, we derive the following inequalities:

$$\begin{array}{c}\hfill W_{\mathcal{T}}({\mu }_i,{\mu }_j)={\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i-\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j\parallel }_1\le \sqrt{2\mathrm{KL}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j)},\end{array}$$

and

$$\begin{array}{c}\hfill W_{\mathcal{T}}({\mu }_i,{\mu }_j)={\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j-\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i\parallel }_1\le \sqrt{2\mathrm{KL}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i)},\end{array}$$

Thus,

$$\begin{array}{cc}\hfill W_{\mathcal{T}}^2({\mu }_i,{\mu }_j)& \le \mathrm{KL}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j)+\mathrm{KL}(\mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_j\parallel \mathrm{diag}\left(\mathit{w}\right)\mathit{B}{\mathit{a}}_i)\hfill \end{array}$$

□

This result indicates that minimizing the symmetric KL divergence (i.e., Jeffrey divergence) can minimize the tree-Wasserstein distance. Due to the fact that the Jeffrey divergence is smooth, the computation of the gradient of the upper bound is easier. For presentation, we denote $W_{\mathcal{T}}({\mu }^{\left(1\right)},{\mu }^{\left(2\right)})=W_{\mathcal{T}}({\mathit{a}}^{\left(1\right)},{\mathit{a}}^{\left(2\right)})$.

Note that Frogner et al. [53] considered a multilabel classification problem utilizing the regularized Wasserstein loss. They proposed utilizing Kullback–Leibler divergence-based regularization to stabilize training. We derive the Jeffrey divergence from the TWD, and JD regularization includes a simple KL divergence-based regularization as a special case. Moreover, we propose employing JD regularization for SSL frameworks, which have not been extensively studied.

5. Experiments

This section evaluates SSL methods with different probability models.

5.1. Performance Comparison for SimCLR

For all experiments, we employed the Resnet18 model with an output dimension of ($d_{\mathrm{out}}=256$) and coded all the methods based on a standard SimCLR implementation (https://github.com/sthalles/SimCLR (accessed on 7 July 2023). We used the Adam optimizer and set the learning rate to 0.0003, the weight decay parameter to 1e-4, and temperature $\tau$ to 0.07. For the proposed method, we compared two variants of TWD: total variation and ClusterTree (Figure 1). As part of the model evaluation, we assessed the conventional softmax function, attention model (AF), and simplicial embedding (SEM) [10] and set the temperature parameter $\tau =0.1$ for all experiments. For SEM, we set $L=16$ and $V=16$.

We also evaluated JD regularization, where we set the regularization parameter $\lambda =0.1$ for all experiments. For reference, we compared cosine similarity as a similarity function of SimCLR. For all approaches, we utilized the KNN classifier of the scikit-learn package (https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html (accessed on 7 July 2023)), where the number of nearest neighbor was set to $K=50$. We utilized the L1 distance for Wasserstein distances and cosine similarity for non-probability-based models. All the experiments were computed on A6000 GPUs. We ran all experiments three times and report the average scores.

Figure 3 illustrates the training loss and top-1 accuracy for the three methods: cosine + real-valued embedding, TV + softmax, and TV + AF (DCT). This experiment revealed that the convergence speed of the loss function was nearly identical across all methods. Regarding the training top-1 accuracy, cosine + real-valued embedding achieves the highest accuracy, followed by the softmax function, and AF (DCT) lags. This behavior is expected because real-valued embeddings offer the most flexibility, followed by softmax, with AF models exhibiting the least freedom. For all methods based on the TWD, JD regularization significantly aids the training process, particularly in the case of the softmax function. However, for AF (DCT), the improvement was relatively small. This is likely because AF (DCT) can also be considered a form of regularization.

Table 1 presents the experimental results for the test classification accuracy using KNN. The first observation is that the simple implementation of the conventional softmax function performs poorly (the performance is approximately 10 points lower) compared to cosine similarity. As expected, AF has only one more layer than the simple softmax model, and performs similarly to softmax. Compared to softmax and AF, AF (PE), and AF (DCT) significantly improve the classification accuracy for the total variation and ClusterTree cases. However, for the ClusterTree case, AF (PE) achieves a better classification performance, whereas the AF (DCT) improvement over the softmax model is limited. In the ClusterTree case, SEM significantly improves with the combination of ClusterTree and regularization. One potential reason of the performance improvement on TV + AF (DCT) combination and ClusterTree + SEM is that AF (DCT) utilizes the orthonormal DCT transform of the learned representation, while both SEM and ClusterTree have structures themselves. This means that each element of the final probability vector ${\mathit{a}}_{\mathit{\theta }}$ can be uncorrelated for AF (DCT). As a result, the tree structure may not provide significant information, and the total variation (i.e., each leaf node connected to the root node) might be the best fit for the probability representation. Additionally, the cluster-like structure may conflict with the DCT-based representation. In contrast, SEM has an inherent structure and is computed without the DCT transformation (it learns a sum-to-one vector on subtrees). Therefore, the cluster tree structure and SEM can be a good match.

Table 1. KNN classification result with Resnet18 backbone. In this experiment, we set the number of neighbors as $K=50$ and computed the averaged classification accuracy over three runs. Note that the Wasserstein distance with ($\mathit{B}={\mathit{I}}_{d_{\mathrm{out}}}$) is equivalent to total variation.

Similarity	Prob Model	STL10	CIFAR10	CIFAR100	SVHN
Cosine	N/A	75.77± 0.47	67.39± 0.46	32.06± 0.06	76.35 ± 0.39
	Softmax	70.12 ± 0.04	63.20 ± 0.23	26.88 ± 0.26	74.46 ± 0.62
	SEM	71.33 ± 0.45	61.13 ± 0.56	26.08 ± 0.07	74.28 ± 1.13
	AF (DCT)	72.95 ± 0.31	65.92 ± 0.65	25.96 ± 0.13	76.51± 0.24
TWD (TV)	Softmax	65.54 ± 0.47	59.72 ± 0.39	26.07 ± 0.19	72.67 ± 0.33
	SEM	65.35 ± 0.31	56.56 ± 0.46	24.31 ± 0.43	73.36 ± 1.19
	AF	65.61 ± 0.56	60.92 ± 0.42	26.33 ± 0.42	75.01 ± 0.32
	AF (PE)	71.71 ± 0.17	64.68 ± 0.33	26.38 ± 0.37	76.44 ± 0.45
	AF (DCT)	73.28 ± 0.27	67.03 ± 0.24	25.85 ± 0.39	77.62 ± 0.40
	Softmax + JD	72.64 ± 0.27	67.08 ± 0.14	27.82± 0.22	77.69 ± 0.46
	SEM + JD	71.79 ± 0.92	63.60 ± 0.50	26.14 ± 0.40	75.64 ± 0.44
	AF + JD	72.64 ± 0.37	67.15 ± 0.27	27.45 ± 0.37	78.00 ± 0.15
	AF (PE) + JD	74.47 ± 0.10	67.28 ± 0.65	27.01 ± 0.39	78.12 ± 0.48
	AF (DCT) + JD	76.28± 0.07	68.60± 0.36	26.49 ± 0.24	79.70± 0.23
TWD (Clus)	Softmax	69.15 ± 0.45	62.33 ± 0.40	24.47 ± 0.40	74.87 ± 0.13
	SEM	72.88 ± 0.12	63.82 ± 0.32	22.55 ± 0.28	77.47 ± 0.92
	AF	70.40 ± 0.40	63.28 ± 0.57	24.28 ± 0.15	75.24 ± 0.52
	AF (PE)	72.37 ± 0.28	65.08 ± 0.74	23.33 ± 0.35	76.67 ± 0.26
	AF (DCT)	71.95 ± 0.46	65.89 ± 0.11	21.87 ± 0.19	77.92 ± 0.24
	Softmax + JD	73.52 ± 0.16	66.76 ± 0.29	24.96± 0.07	77.65 ± 0.53
	SEM + JD	75.93± 0.14	67.68± 0.46	22.96 ± 0.28	79.19± 0.53
	AF + JD	73.66 ± 0.23	66.61 ± 0.32	24.55 ± 0.14	77.64 ± 0.19
	AF (PE) + JD	73.92 ± 0.57	67.00 ± 0.13	23.83 ± 0.42	77.87 ± 0.29
	AF (DCT) + JD	74.29 ± 0.30	67.50 ± 0.49	22.89 ± 0.12	78.31 ± 0.72

Overall, the proposed method performs better than cosine similarity without real-valued vector embedding when the number of classes is relatively small (i.e., STL10, CIFAR10, and SVHN). By contrast, the performance of the proposed method degrades for CIFAR100, and the results for ClusterTree are particularly poor. As the Wasserstein distance can be minimized even if it cannot overfit, it is natural for the Wasserstein distance to make mistakes when the number of classes is large. Note that the performances for CIFAR100 with simplicial representation degrade both cosine and TWD loss functions, and the performance degradation seems to come from the softmax operation. Moreover, the total variation is a robust measure and learning with total variation is generally designed to create models that are resilient to noise. In our setting, which involves self-supervised learning, it is likely that similar class representations could become mixed, leading to performance degradation. Since the proposed method performs well on CIFAR-10, we believe this could be the reason for the performance issues on larger datasets. To address this, it may be beneficial to use other types of regularizers or larger deep learning models.

Next, we evaluated the Jeffrey divergence regularization. Surprisingly, simple regularization dramatically improves the classification performance of all the probability models. These results support the idea that the main problem with Wasserstein distance-based representation learning is its numerical instability.

Among the methods, the proposed AF (DCT) + JD with total variation achieves the highest classification accuracy, comparable to the cosine similarity result, and achieves more than 10% improvement from the naive implementation with the softmax function. Moreover, all probability model performances with the cosine similarity combination tend to result in a lower classification error than those with the combination of the TWD and probability models. Based on our empirical study, we propose utilizing TWD (TV) + AF models or TWD (ClusterTree) + SEM for representation-learning tasks in probability-based representation learning.

5.2. Performance Comparison for SimSiam

Next, we evaluated the performance using a non-contrastive setup. For all experiments, we utilized the Resnet18-Cifar-Variant1 model with an output dimension of ($d_{\mathrm{out}}=2048$) and implemented all methods based on a standard SimSiam framework (https://github.com/PatrickHua/SimSiam). The optimization was performed using the SGD optimizer with a base learning rate of 0.03, weight decay parameter of 0.00005, momentum parameter of 0.9, batch size of 512, and a fixed number of epochs set to 800. For the proposed method, we employed the total variation as a loss function, along with the softmax function and ArcFace model (AF). The temperature parameter $\tau$ was set to 0.1 for all experiments. Additionally, we assessed JD regularization with the regularization parameter $\lambda$ set to 0.1 across all experiments. A100 GPUs were used for all experiments, and each experiment was run three times, with the reported results being the average scores.

We compared the proposed methods, TWDSimSiam (softmax + JD) and TWDSimSiam (AF + JD), with the original SimSiam method which employs cosine similarity loss. Upon examination, we observe that learning the total variation with softmax encounters numerical issues, even with JD regularization (See Figure 4a,c). Conversely, the AF + JD combination proved successful in training the models, as shown in Figure 4b,c. One potential reason for the failure of TWD with softmax is that the total variation can easily become zero because the softmax function lacks normalization. For TWDSimSiam (AF + JD), normalization within the AF model prevents convergence to a poor local minimum. From a performance standpoint as shown in Table 2, the utilization of cosine similarity and total variation (TV) yield comparable results. However, a key contribution of this study is the introduction of a practical approach to enhance the model training stability by incorporating Wasserstein distance, specifically through total variation. This discovery has a potential utility in various SSL tasks.

Figure 4. TWD loss for SimSiam models.

Table 2. SimSiam evaluation with CIFAR10 dataset.

Similarity	Probability Model	Linear Classifier
Cosine	N/A	91.13 ± 0.14
TWD (TV)	Softmax + JD	9.99 ± 0.00
	AF (DCT) + JD	90.60 ± 0.02

5.3. Effect of Number of Nearest Neighbors

In this section, we assess the performance of the KNN model by varying the number of nearest neighbors and setting K to 10 or 50. The results for $K=10$ are presented in Table 3, and Table 4 illustrates a comparison of the best models across different nearest neighbor values. Our experiments revealed that utilizing $K=50$ tends to enhance the performance, and the relative order of the results remains consistent, regardless of the number of nearest neighbors.

Table 3. KNN classification result with Resnet18 backbone. In this experiment, we set the number of neighbors as $K=10$ and computed the averaged classification accuracy over three runs. Note that the Wasserstein distance with ($\mathit{B}={\mathit{I}}_{d_{\mathrm{out}}}$) is equivalent to a total variation.

Similarity	Prob Model	STL10	CIFAR10	CIFAR100	SVHN
Cosine	N/A	75.44± 0.21	66.96± 0.45	31.63± 0.25	74.71 ± 0.31
	Softmax	71.25 ± 0.30	63.80 ± 0.48	26.18 ± 0.36	73.06 ± 0.47
	SEM	71.34 ± 0.31	61.26 ± 0.42	25.40 ± 0.06	73.41 ± 0.95
	AF (DCT)	72.15 ± 0.53	65.52 ± 0.45	24.93 ± 0.24	75.68± 0.13
TWD (TV)	Softmax	63.42 ± 0.24	59.03 ± 0.58	24.95 ± 0.31	70.87 ± 0.29
	SEM	63.72 ± 0.17	55.57 ± 0.35	23.40 ± 0.36	71.69 ± 0.75
	AF	63.97 ± 0.05	59.96 ± 0.44	25.29 ± 0.17	73.44 ± 0.35
	AF (PE)	71.04 ± 0.37	64.28 ± 0.14	25.71 ± 0.20	75.70 ± 0.42
	AF (DCT)	72.75 ± 0.11	67.01 ± 0.03	24.95 ± 0.17	76.98 ± 0.44
	Softmax + JD	72.05 ± 0.30	66.61 ± 0.20	26.91 ± 0.19	76.65 ± 0.56
	SEM + JD	70.73 ± 0.89	62.75 ± 0.61	24.83 ± 0.27	74.71 ± 0.43
	AF + JD	71.74 ± 0.19	66.74 ± 0.20	26.68± 0.35	77.10 ± 0.04
	AF (PE) + JD	74.10 ± 0.20	66.82 ± 0.36	26.17 ± 0.00	77.55 ± 0.50
	AF (DCT) + JD	76.24± 0.22	68.62± 0.40	25.70 ± 0.14	79.28± 0.22
TWD (Clust)	Softmax	67.95 ± 0.42	61.59 ± 0.29	23.34 ± 0.26	73.88 ± 0.05
	SEM	72.43 ± 0.11	63.63 ± 0.42	21.29 ± 0.28	77.04 ± 0.77
	AF	69.09 ± 0.05	62.49 ± 0.45	22.56 ± 0.25	74.31 ± 0.40
	AF (PE)	72.08 ± 0.07	64.56 ± 0.31	22.51 ± 0.29	75.98 ± 0.23
	AF (DCT)	71.64 ± 0.15	65.51 ± 0.36	21.04 ± 0.10	77.59 ± 0.25
	Softmax + JD	73.07 ± 0.13	66.38 ± 0.27	23.97± 0.11	76.82 ± 0.50
	SEM + JD	75.50± 0.15	67.44± 0.10	21.90 ± 0.19	78.91± 0.30
	AF + JD	72.70 ± 0.08	66.12 ± 0.26	23.50 ± 0.21	76.92 ± 0.06
	AF (PE) + JD	73.66 ± 0.47	66.58 ± 0.01	22.86 ± 0.02	77.44 ± 0.30
	AF (DCT) + JD	73.79 ± 0.12	67.34 ± 0.38	21.96 ± 0.34	78.00 ± 0.60

Table 4. KNN classification accuracy with different number of neighbors.

Similarity	K	STL10	CIFAR10	CIFAR100	SVHN
TWD (TV)	10	76.24 ± 0.22	68.62 ± 0.40	25.70 ± 0.14	79.28 ± 0.22
	50	76.28 ± 0.07	68.60 ± 0.36	26.49 ± 0.24	79.70 ± 0.23

5.4. Effect of the Regularization Parameter for Jeffrey Divergence

In this experiment, we evaluated model performance by varying the regularization parameter, denoted as $\lambda$. The results indicate a noteworthy improvement in performance with the introduction of regularization parameters. However, as shown in Table 5, it was observed that the performance did not exhibit significant changes across different values of $\lambda$, and setting $\lambda =0.1$ yielded favorable results.

Table 5. KNN classification result with Resnet18 backbone. In this experiment, we set the number of neighbors as $K=50$ and computed the averaged classification accuracy over three runs.

Similarity Function	$\mathit{\lambda }$	STL10	CIFAR10	CIFAR100	SVHN
TWD (TV)	$0.0$	73.28 ± 0.27	67.03 ± 0.24	25.85 ± 0.39	77.62 ± 0.40
	$0.1$	76.28 ± 0.07	68.60± 0.36	26.49± 0.24	79.70 ± 0.23
	$0.2$	77.40 ± 0.17	68.48 ± 0.11	25.59 ± 0.16	79.67 ± 0.26
	$0.3$	77.67± 0.06	68.26 ± 0.51	24.21 ± 0.35	79.91± 0.42

6. Conclusions

This study investigates SSL using TWD. We empirically evaluate several benchmark datasets and find that a simple combination of the softmax function and TWD performs poorly. To address this, we propose simplicial embedding [10] and ArcFace models [21] as probability models. Moreover, to mitigate the intricacies of optimizing TWD, we incorporate an upper bound on the squared 1-Wasserstein distance as a regularization technique. Overall, the combination of ArcFace and DCT outperforms their cosine similarity counterparts. Finally, we find that the combination of TWD (ClusterTree) and SEM yields favorable performance.

There are several potential future directions for our work. Firstly, improving representation learning for larger classes could involve employing larger models and/or introducing new regularization techniques. Secondly, integrating the proposed probability representation into other SSL models such as DINO [8] could enhance our understanding of model performance across different learning tasks. Lastly, while we have empirically studied self-supervised learning with Wasserstein distance, the theoretical properties remain unclear. Therefore, investigating these theoretical properties represents another promising research direction.