Clustering-Based Representation Learning through Output Translation and Its Application to Remote-Sensing Images

Abstract

In supervised deep learning, learning good representations for remote-sensing images (RSI) relies on manual annotations. However, in the area of remote sensing, it is hard to obtain huge amounts of labeled data. Recently, self-supervised learning shows its outstanding capability to learn representations of images, especially the methods of instance discrimination. Comparing methods of instance discrimination, clustering-based methods not only view the transformations of the same image as “positive” samples but also similar images. In this paper, we propose a new clustering-based method for representation learning. We first introduce a quantity to measure representations’ discriminativeness and from which we show that even distribution requires the most discriminative representations. This provides a theoretical insight into why evenly distributing the images works well. We notice that only the even distributions that preserve representations’ neighborhood relations are desirable. Therefore, we develop an algorithm that translates the outputs of a neural network to achieve the goal of evenly distributing the samples while preserving outputs’ neighborhood relations. Extensive experiments have demonstrated that our method can learn representations that are as good as or better than the state of the art approaches, and that our method performs computationally efficiently and robustly on various RSI datasets.

1. Introduction

The self-supervised visual-representation learning paradigm, where “supervision” is created from making use of the intrinsic information contained in the data itself and based on prior knowledge about the world, provides us with an unsupervised way to learn representations of RSI. Recently, methods of instance discrimination [1,2,3,4,5,6] have achieved remarkable progress in unsupervised representation learning. These methods view every sample and its transformations as a “class”, while, in clustering-based methods [7,8], samples grouped to the same cluster are considered as a “class”. In fact, data augmentations are often used in clustering-based methods, and, thus, transformations of the same images are also viewed as the same “class”. However, so far, methods of instance discrimination overall have better performances than clustering-based methods, which seems to imply that clustering-based methods could be further improved.

Degeneracy is a problem that nearly all clustering-related methods need to solve. The usual ways to tackle it is by setting constraints on the clustering process. Whether it is based on maximizing entropy [9,10] or adding regularization [11], all aim to distribute the samples evenly to the clusters. However, even distribution is not a necessary condition for preventing degeneracy; there is a lack of a clear theoretical interpretation for why even distribution works well in these clustering-related methods. On the other hand, clustering should be based on the neighborhood relations of the samples or that of their representations. To avoid degeneracy, the normal clustering process (that clusters should be formed based on the relations amongst the samples) would have to be artificially introduced. It is unclear if approaches used by previous methods to achieve even distribution can completely preserve the neighborhood relations of samples or that of their representations. By closely observing the Sinkhorn–Knopp algorithm used by [8], it is easy to see that the relations between neighboring samples or the distances between the outputs have been changed by the intervention to achieve even distribution.

In this paper, we first introduce a quantity that can be used to measure how discriminative representations are under a generic clustering setting where the clusters are formed based on the neighborhood relations or similarity of an images’ representations. From this quantity, we show that evenly distributing the images into clusters requires the visual representations to be most discriminative. This provides an insight that clearly explains why evenly distributing the samples into clusters worked well in previous related work [8,10]. However, we are only interested in the even distributions that preserve the model’s smoothness, i.e., preserve the neighborhood relations or similarity of the images’ representations, and, thus, we develop an algorithm (The code is avaible at https://github.com/qlilx/OTL that translates the outputs of a convolutional neural network to achieve the goal of evenly distributing the images while, at the same time, preserving the current model’s smoothness. In summary, we make the following contributions:

• A quantity that can be used to measure the discriminativeness of visual representations is introduced and a clear theoretical explanation for evenly distributing images into clusters is given from the viewpoints of this quantity.
• A new efficient and robust clustering-based representation learning algorithm, through translating the outputs of a convolutional neural network, is proposed.
• Our method performs as well as or better than the state-of-the-art approaches and works very well on a variety of RSI datasets.

2. Related Works

Generally, most self-supervised methods can be categorized into four categories: clustering-based methods, contrastive methods, pretext tasks and generative models.

Clustering-based methods learn visual representations by integrating network optimization and clustering. Clustering is a classical approach to unsupervised machine learning [12]. K-means, a standard clustering algorithm, has been applied to DeepCluster [7] for grouping the features extracted from a deep neural network. Subsequently, the cluster assignments were utilized as supervision to update the weights of the network. The method, Anchor Neighborhood Discovery (AND) [13] exploits a class-consistent neighborhood for unsupervised learning, which combines the advantages of both sample-specificity learning and clustering while overcoming their disadvantages. Maximizing the information between the indices of inputs and labels, self-labeling [8] not only avoids the degenerated problem but also creates pseudo-labels for training a deep neural network by a standard cross-entropy loss. Other earlier works of combining clustering and deep learning can be found in [14,15,16,17,18].

The contrastive method [1,2,3,4,5,6] has recently shown remarkable performances on visual-representation learning. The contrastive method is a method of performing instance discrimination, which only considers the transformations of the same image as the “positive” samples, while others are the “negative” samples in the contrastive loss. Recently, some methods combining clustering methods and contrastive methods have appeared. In [19], the nearest neighbors are also considered as the “positive” samples in the contrastive loss, and in [20], both instance-level and cluster-level contrastive losses are introduced. In the past year, contrastive methods have been extensively applied to area of remote sensing [21,22,23,24].

A pretext task uses hand-crafted “supervision” to replace manual labels in supervised learning. These “supervision” pretext tasks, are devised from exploiting the intrinsic information of the unlabeled data. These pretext tasks include predicting context [25], solving jigsaw puzzles [26,27], image rotation and colorization [28,29,30,31], spatio-temporal consistence [32], and so on. The applications of this method to RSI can be found in [33,34,35], which use domain knowledge and temporal prediction to supervise the training.

Generative models. The latent distributions in generative models could be visual representations, due to the fact that they can capture the distributions of input data. These generative models mainly include Boltzmann machines [36,37,38], autoencoders [39] and generative adversarial networks (GAN) [40,41,42,43]. In [44], an improved generative adversarial network was applied to the super-resolution processing of RSI.

3. Method

The goal of this paper is to learn good visual representations for images through clustering them; that is, clustering is not the target but a means to learn visual representation. This section will introduce two essential aspects of good visual representations and, based on them, design our algorithm and training method.

3.1. Discriminativeness of Representations

In the representation feature space, the feature vectors should reveal as much as possible the difference in nonidentical samples; that is, good representations should be discriminative. To compare which representations are more discriminative, we need to be able to measure the discriminativeness of representations. In this section, based on the generic clustering model as shown in Figure 1, we try to find a criterion that can be used to compare the discriminativeness of representations.

Consider a general clustering that groups N images, based on their representations in feature space F, into k clusters by certain methods, for example, k-means. Samples in the same cluster will be assigned the same pseudo-labels that will be applied to standard cross entropy loss to update the network. Due to samples in the same cluster having closer representations and being indiscriminately treated, we could legitimately regard the representations of samples from the same cluster as indistinguishable. If the N representations in space F are very discriminative, there will be few indistinguishable representations.

Therefore, the quantity that can be used to compare the discriminativeness of representations should be able to indicate the representations’ degree of distinguishableness. When we say “distinguishable”, when at least two representations are needed to be compared. Consequently, in this paper, we use the number of distinguishable pairwise comparisons of representations as the quantity to measure the discriminativeness of representations.

In different representation feature spaces, the clustering results will be different. Let $n_i\left(F\right)$ be the number of samples assigned into cluster i in the feature space F, and, thus, $N=n_1\left(F\right)+n_2\left(F\right)+\cdots +n_k\left(F\right)$, where N is the total number of samples. The total number of pairwise comparisons of N representations is $N(N-1)/2$, in which the number of indistinguishable pairwise comparisons is

$$N_{\mathrm{ind}}\left(F\right)=\sum _i^k\left(\begin{array}{c}{n}_i\\ 2\end{array}\right)$$

(1)

and the number of distinguishable pairwise comparisons is

$$N_{\mathrm{dis}}\left(F\right)=\sum _{i=1}^{k-1}\left(n_i\left(F\right)\sum _{j=i+1}^k{n}_j\left(F\right)\right).$$

(2)

Consider representations in two different feature spaces $F_1$ and $F_2$, and perform clustering in these two spaces. If

$$N_{\mathrm{dis}}\left(F_1\right)>N_{\mathrm{dis}}\left(F_2\right),$$

(3)

then the representations in $F_1$ ought to be more discriminative than those in $F_2$ because there are more distinguishable representations in $F_1$. Therefore, the quantity $N_{\mathrm{dis}}\left(F\right)$ can be used to measure the discriminativeness of the N samples’ representations in a feature space F: the larger is $N_{\mathrm{dis}}\left(F\right)$, the more discriminative are the representations. Several simple toy examples are given in Appendix A to elaborate the rationality of this quantification.

From the view points of $N_{\mathrm{dis}}\left(F\right)$, the degeneracy problem in clustering methods is caused by small $N_{\mathrm{dis}}\left(F\right)$. For example, the extreme case that $N_{\mathrm{dis}}\left(F\right)=0$ means that the representations are totally indistinguishable, which corresponds to the case where all samples are assigned into the same cluster. Therefore, increasing $N_{\mathrm{dis}}\left(F\right)$ can prevent the representations from degenerating. However, the question is, when will $N_{\mathrm{dis}}\left(F\right)$ reach the maximum?

Note that $N_{\mathrm{ind}}+N_{\mathrm{dis}}=N(N-1)/2$. Thus, the maximum $N_{\mathrm{dis}}\left(F\right)$ corresponds to the minimal $N_{\mathrm{ind}}\left(F\right)$ due to N and k being constants. Equation (1) simplifies to

$$\begin{array}{c}\hfill N_{\mathrm{ind}}=\frac{1}{2}\left(\sum _i^k{n}_i^2\left(F\right)-N\right)\end{array}$$

(4)

Again, noting N is a constant, $N_{\mathrm{ind}}$ achieves the minimal value when ${\sum }_i^k{n}_i^2\left(F\right)$ is the minimal. According to the inequality of arithmetic and geometric means (AM–GM inequality) [45] and noting $n_i\left(F\right)\ge 0$, ${\sum }_i^k{n}_i^2\left(F\right)$ takes the minimal value if

$$n_i\left(F\right)=\overline{n}\left(F\right)=N/k,\forall i\in \{1,2\cdots ,k\}$$

(5)

which is also the condition for $N_{\mathrm{ind}}$ to reach the maximum. For the cases when $\overline{n}=N/k$ is not an integer, a rigorous proof is given in Appendix B that $N_{\mathrm{ind}}$ reaches the largest value when the standard deviation of the distribution of $n_i\left(F\right)$ is the minimal, i.e., the N samples are distributed as uniformly (evenly) as possible among the k clusters.

Remark 1.

If N nonidentical images are most evenly distributed into k clusters according to their representations in feature space F, then $N_{\mathrm{ind}}\left(F\right)$ must be the maximum.

The fact stated in Remark 1 gives an explicit explanation for why even distribution works well in previous clustering-related methods, e.g., [8]. From the view point of Remark 1: an even distribution corresponds to the most discriminative representations, which can be regarded as the extreme opposite of the degenerated cases. However, for previous works in the literature, the even distribution is simply used as a tool to prevent the model from collapsing. In contrast, in this paper, we provide a fundamental insight that has enabled us to give a clear interpretation for even distribution and use it to develop new techniques to improve the discriminativeness of representations.

Intuitively, “the most discriminative representations give the most uniform distribution” is correct only for special cases. It is important to emphasize that, in this paper, we measure the discriminativeness of representations by $N_{\mathrm{ind}}\left(F\right)$, which gives how many pairs of representations are distinguishable. Again, the rationality of this quantification can be seen from simple examples in Appendix A.

For the given N and k, when $n_i\left(F\right)$ is the most uniform distribution, the assignment of the input images to the k clusters has $N_e$ possible combinations

$$\begin{array}{c}\hfill N_e=\frac{1}{k!}\left(\begin{array}{c}N\\ n_1\left(F\right)\end{array}\right)\prod _{i=2}^k\left(\begin{array}{c}N-\sum _{j=1}^{i-1}{n}_j\left(F\right)\\ n_i\left(F\right)\end{array}\right)\end{array}$$

(6)

This means that the solution to maximal $N_{\mathrm{ind}}\left(F\right)$ is not unique. Therefore, the question now is how we shall approach maximum $N_{\mathrm{ind}}\left(F\right)$ such that it is good for representation learning.

3.2. General-Purpose Prior: Smoothness

Good representation should be discriminative. However, at the end of Section 3.1, we demonstrated that there are many ways to reach evenly distributed pseudo-labels that can be used to improve the discriminativeness of representations. We are only interested in the ones that are good for representation learning. Recall that the basic task of representation learning is to make similar samples to have close representations while different samples have distinct representations. In other words, representation learning is the pursuit of a smooth learning model tht satisfies

$$f\left(x_1\right)\approx f\left(x_2\right)\mathrm{iff}{x}_1\approx x_2,$$

(7)

where f is the mapping function of the model to be learnt. Smoothness is a basic and general requirement of a learning model [46]. Therefore, only the pseudo-labels that are good for improving the smoothness of the model are desirable. Specifically, this requires that similar samples are more likely to be assigned the same pseudo-labels.

Some model architectures have a certain inherent smoothness. For example, randomly initialized AlexNet possesses some degree of smoothness [26]. Therefore, clustering based on the outputs or internal features extracted by the model is likely to group close samples together, thus assigning them with the same pseudo-labels. Applying these pseudo-labels in the standard cross-entropy loss can enhance the smoothness of the model. Besides this, making the representations of transformations of the same image close to each other can also improve the smoothness of the model. Therefore, without any manual annotation, assigning pseudo-labels based on the neighborhood relations of the outputs or the internal representations of the network is a reasonable strategy. This provides us with an important constraint on approaching the even distribution of images in clusters, that is, the neighborhood relations of the outputs or the internal representations of the network must be protected. Recall that evenly assigning the pseudo-labels is necessary because it can not only prevent the model from collapsing, but also improve the discriminativeness of representations.

3.3. Algorithm

This section proposes an effective algorithm that can produce evenly distributed pseudo-labels according to the outputs of the learning model while keeping the smoothness of the current model unaltered. We then use the pseudo-labels and a label-consistent-training (LCT) technique to improve the model’s smoothness.

Model architecture. We consider a general convolutional neural network (CNN) model, as shown in Figure 1, for representation learning. One or several fully connected layers are attached at the end of the CNNs, which will output a k-dimensional vector indicating the assignment of the input samples to the k clusters. Specifically, the sth input images $I\left(s\right)$ is mapped to $X\left(s\right)$ by the convolutional block $\varphi$, i.e., $X\left(s\right)=\varphi \left(I\right(s\left)\right)$. By one or several fully connected layers, the feature $X\left(s\right)$ in D-dimensional space is then mapped to a k-dimensional space: ${\mathbb{R}}^D\to {\mathbb{R}}^k$

$$O\left(s\right)=\left(o_1\left(s\right),o_2\left(s\right),\cdots ,o_k\left(s\right)\right),$$

(8)

where $O\left(s\right)$ is the output of $I\left(s\right)$ without softmax operation (in this paper, all “output“ refers to the final layer of network but the softmax activation on the final layer). The output dimension is set to be the same as the cluster number, such that images could be assigned into k clusters based on the rule: the input $I\left(s\right)$ is assigned to cluster $c_i$ if and only if $o_i\left(s\right)$ is the maximal component:

$$I\left(s\right)\in c_i\mathrm{iff}{o}_i\left(s\right)\ge o_j\left(s\right),\forall j\ne i,i,j=1,2,\cdots ,k.$$

(9)

Clearly, (9) is a winner-takes-all competition extensively used in unsupervised competitive learning [47]. The largest output wins the competition and the rest lose out.

Output translation. In most cases, if we perform clustering as (9) based on outputs (8), most input samples will be assigned into a few clusters. Consider a toy model: assigning eight points in a two-dimensional plane to four clusters based on the quadrant the points are located in. Usually, based on the original location of points, they would not be evenly distributed into the four clusters; for example, the first graph of Figure 2 (one red point represents a point, the boundaries in the diagram are the x and y axes). If the points need to be evenly distributed into the four clusters based on the same rule, the values of their coordinates have to be changed and the question is how to change the values of them. If there is a constraint that their relative distances should be kept (the smoothness requirement), translating them as whole is the most straightforward approach to carry that out, as shown in the second graph of Figure 2. Considering N points in k dimensions and changing the clustering rule to Equation (9), the model then becomes what this paper needs to study and, similarly, the even distribution of pseudo-labels could be reached by translating the outputs in k dimensions.

Therefore, the core task of our algorithm is that of finding the translation T

$$\begin{array}{c}\hfill O^{\prime }\left(s\right)=O\left(s\right)-T\end{array}$$

(10)

such that cluster assignment according to (9) using the new outputs $O^{\prime }\left(s\right)$ gives evenly distributed pseudo-labels. Obviously, T is a k-dimensional vector: $T=(t_1,t_2,\cdots ,t_k)$. To find $t_1,t_2,\cdots ,t_k$, we borrow the basic idea from a competitive learning or self-organising learning algorithm called frequency sensitive competitive learning (FSCL), widely used in the design of optimal vector quantisation [48,49]. If one component (or a direction) “wins” too much in the competition (9), we move all output vectors away from this direction; if one direction “loses” the competition too often, we move all output vectors towards this direction. To perform this, we need to construct each direction’s wining frequency indicator in features space F. We count the number of samples assigned into every cluster $c_i$ according to (9), and set the number as its wining frequency indicator. For convenience, we subtract the mean $\overline{n}\left(F\right)=N/k$ from each cluster to obtain the frequency indicator as

$$\begin{array}{c}\hfill C=\left(\begin{array}{c}{n}_1\left(F\right)-\overline{n}\left(F\right)\\ n_2\left(F\right)-\overline{n}\left(F\right)\\ \cdots \\ n_k\left(F\right)-\overline{n}\left(F\right)\end{array}\right)=\left(\begin{array}{c}{\widehat{n}}_1\left(F\right)\\ {\widehat{n}}_2\left(F\right)\\ \cdots \\ {\widehat{n}}_k\left(F\right)\end{array}\right)\end{array}$$

(11)

where ${\widehat{n}}_i\left(F\right)=n_i\left(F\right)-\overline{n}\left(F\right)$. C can be used as all directions’ winning frequency indicator, e.g., the winning frequency of the first direction is ${\widehat{n}}_1\left(F\right)$. Note that some ${\widehat{n}}_i\left(F\right)$ are positive while others are non-positive. The vector of translation should be proportional to this indicator $T\propto C$. The frequency indicator determines the direction of translation. The magnitude of the translation should be close to that of the output vectors and change with the evenness of the frequency distribution: the more uniform is the distribution, the smaller should be the translation magnitude. Therefore, the translation vector could be set as

$$\begin{array}{c}\hfill T=\alpha \left(\begin{array}{c}{\widehat{n}}_1\left(F\right)\\ {\widehat{n}}_2\left(F\right)\\ \cdots \\ {\widehat{n}}_k\left(F\right)\end{array}\right),\alpha =\frac{\mathrm{std}}{{\mathrm{std}}_{\max }}\left(o_{\max }-o_{\min }\right).\end{array}$$

(12)

where “${\mathrm{std}}_{\max }$” is the maximal standard deviation, which corresponds to the case where all samples fall into the same cluster, while “$\mathrm{std}$” is the standard deviation of the current distribution. $\left(o_{\max }-o_{\min }\right)$ is the difference between the maximal and the minimal components in all outputs vectors, which gives the order of magnitude of $\alpha$. The ratio “$\mathrm{std}/{\mathrm{std}}_{\max }$” indicates the degree of unevenness: unevenness increases from 0 to 1 as “$\mathrm{std}$” changes from 0 to the maximum. Noting that, from Equations (10) to (12), all the quantities are from the outputs, no hyper-parameters are introduced.

To make the distribution of pseudo-labels as uniform as possible, the translation should be repeatedly performed, and $\alpha$ should decrease as iterations of translation increase and, thus, we introduce a decay rate $\beta$ (see Algorithm 1). The pseudo-code of creating evenly distributed pseudo-labels by output translation is given in Algorithm 1.

The translation (10) is just a simple linear transformation, that is, a smooth mapping

$$O^{\prime }\left(s_1\right)\approx O^{\prime }\left(s_2\right)\mathrm{iff}O\left(s_1\right)\approx O\left(s_2\right).$$

(13)

Therefore, translation (10) does not change the smoothness of the current model, that is, if

$$O\left(s_1\right)\approx O\left(s_2\right)\mathrm{iff}I\left(s_1\right)\approx I\left(s_2\right),$$

(14)

then

$$O^{\prime }\left(s_1\right)\approx O^{\prime }\left(s_2\right)\mathrm{iff}I\left(s_1\right)\approx I\left(s_2\right).$$

(15)

More algorithm analysis can be found in Appendix C, where we give a detailed comparison with the Sinkhorn–Knopp algorithm used in [8] in terms of of motivation and technical details, and where we also give a clear mathematical illustration demonstrating how our algorithm preserves the distances of outputs.   

Algorithm 1: Creating even distributed pseudo-labels by output translation

Model update. When the translation of outputs is completed, the sth input sample $I\left(s\right)$ is assigned to cluster $c_i$ according to (9), based on the final translated outputs $O^{\prime }\left(s\right)=(o_1^{\prime }\left(s\right),o_2^{\prime }\left(s\right)\cdots ,o_k^{\prime }\left(s\right))$:

$$\begin{array}{c}\hfill I\left(s\right)\in c_i,y^{\prime }\left(s\right)=(0,0,\cdots ,y_i^{\prime }\left(s\right)=1,\cdots ,0,0)\\ \hfill \mathrm{iff}{o}_i^{\prime }\left(s\right)\ge o_j^{\prime }\left(s\right)\forall j\ne ii,j=1,2,\cdots ,k,\end{array}$$

(16)

where $y^{\prime }\left(t\right)$ is the pseudo-label of $I\left(s\right)$. After reaching uniform labelling, we can update the weights of the network as standard supervised learning via standard cross-entropy loss

$$CE=-\frac{1}{N}\sum _s^N{y}^{\prime }\left(s\right)\log \left(\mathrm{softmax}\left(O\right(s)\right),$$

(17)

which is established from the pseudo-labels $y^{\prime }\left(t\right)$ and the outputs before translations $O\left(t\right)$.

The labels generated by (16) from different versions of the same sample may be not consistent. For example, the grayscale image and the color image of the same sample will give different outputs, which may label the same image differently according to (16). To remove this discrepancy, we propose a label-consistent-training (LCT) method. In this method, we construct the loss function by the sum of the cross entropy losses as:

$$L_{\mathrm{LCT}}=\sum _{a,b}^g\left(-\frac{1}{N}\sum _s^N{y}^{\prime }{\left(s\right)}_a\log \left(\mathrm{softmax}\left(O{\left(s\right)}_b\right)\right)\right).$$

(18)

In the loss above, $a,b$ indicates different transformations of the same images $I\left(s\right)$. If the labels from different versions of the same samples are not consistent, the loss function above would be hard to decrease. Thus, decreasing this loss enables different versions of the same samples to give the same prediction, i.e., the same pseudo-label, which will make the model smoother.

4. Experiments Resluts

In this section, we first give our experiment results on several datasets, which are often used in computer vision, to compare to the state-of-the-art research, and then we conduct experiments on various remote-sensing image datasets.

Recall that our method is an unsupervised representation-learning method and its target is to learn good visual representations. Therefore, the ground-truth labels are never used in the training phase but the evaluation phase, and measuring the performance our method is to measure the performance of representations learned by our method. To evaluate the learned representations, we consider a non-parametric and a parametric classifier: weighted kNN (k-nearest neighbors) and linear classifier [50]. The evaluation process is supervised, that is, the labels are used, but only the CNN layers or the layers before the final layer are retained and frozen. For the weighted kNN evaluation, we take $k=50$, $\sigma =0.1$ and an embedding size of 128. For all experiments, we set decay rate to $\beta =1.5$ and ${\alpha }_0={10}^{-15}$ in Algorithm 1.

4.1. Compare to the State-of-the-Art Approaches

To conveniently compare with other representation-learning methods, in this part our experiments were conducted on a large-scale dataset, ImageNet LSVRC-12 [51] and three smaller scale datasets: CIFAR-10/100 and SVHN. ImageNet LSVRC-12 contains around 1.28 million training pictures of 1000 classes and 50-thousand validation pictures. There are 50,000 training images and 10,000 test images in the CIFAR-10/100 dataset, whose number of classes is 10/100 and pixel resolution is $32\times 32$. SVHN is similar to MNIST (images of digits), which is a real-world image dataset. There are 73,257 images ($32\times 32$) for training and 26,032 images ($32\times 32$) for testing. All the ground truths of these datasets were used only in the evaluations of representations. The implementation details are given in Appendix D.

For fairly comparing to the previous work, we set the same cluster number k for these datasets as [8]. For CIFAR-10, the cluster number was set as 128. Therefore, the mean number of samples in each cluster is $50,000/128$. According to Remark 1, although this number is not an integer, setting it as the target still works: making the number of the samples assigned to each cluster as close as possible to the mean number. The cluster number is set as 128 for SVHN and 512 for CIFAR-100 when experiments were conducted in AlexNet.

Table 1. Evaluations on AlexNet (OTL = our new method). The best results are highlighted by the bold.

	kNN/FC
Method	CIFAR-10	CIFAR-100	SVHN
Supervised	91.9	69.7	96.5
Counting [52]	41.7	15.9	43.4
DeepCluster [7]	62.3	22.7	84.9
Instance [53]	60.3	32.7	79.8
AND [13]	74.8	41.5	90.9
SeLa [8]	77.6	44.2	92.8
ISL [54]	82.8	50.3	91.0
OTL	83.1	53.7	93.8
OTL (Strong augmentations + LCT)	87.3	59.2	95.2
OTL ($k=$ class number)	89.9	60.7	95.4
	Linear Classifier/conv5
Supervised	91.8	71.0	96.1
Counting [52]	50.9	18.2	63.4
DeepCluster [7]	77.9	41.9	92.0
Instance [53]	70.1	39.4	89.3
AND [13]	77.6	47.9	93.7
SeLa [8]	83.4	57.4	94.5
ISL [54]	85.8	60.1	93.9
OTL	84.3	59.2	95.0
OTL (Strong augmentations + LCT)	87.1	63.6	96.0
OTL ($k=$ class number)	90.8	65.6	96.1

gives kNN and linear-classifier evaluations of representations learned from CIFAR-10/100 and SVHN through our method OTL (output translation). In addition, kNN evaluations were performed based on the fully connected layer before the final layer, and all layers except the final layer were frozen. For linear classifier, all the fully connected layers were discarded and the retained CNN layers were frozen. The last CNN layer was then attached by a new randomly initialized fully connected layer whose weights were to be updated by a supervised learning. With simple augmentations, the performances of our method reached the state-of-the-art performances. When strong augmentations and LCT (label consistent training) were employed, the performances of our method outperformed the previous works in the literature. For SVHN, the performance of our method is very close to the performance of supervised learning, especially linear classifier, where the accuracy is only $0.1\%$ lower than the accuracy in supervised learning. When one assumes the actual number of classes in the data is known, and sets k equal to the actual number of classes, this can be regarded as giving the model some prior knowledge. It is seen that such prior knowledge can help improve the performances of our OTL algorithm, especially in the case for linear classifier on SVHN, where the accuracy is the same as that of fully supervised learning.

In

Table 2. Evaluations on ResNet-50 (OTL = Our new method). The best results are highlighted by the bold.

	kNN/FC
Method	CIFAR-10	CIFAR-100	SVHN
Instance [53]	81.8	42.3	92.9
AND [13]	87.6	49.0	93.2
ISL [54]	88.9	58.1	94.5
OTL (ours)	90.0	66.4	95.2
	Linear Classifier/conv5
Instance [53]	85.0	50.1	94.4
AND [13]	90.2	58.2	94.9
SimCRL [6]	90.6	71.6	-
ISL [54]	91.5	65.9	95.2
OTL (ours)	92.0	70.7	95.5

, the experiments were ran on ResNet-50 and for fairly comparing to other methods, $k=128$ was set for all experiments on ResNet-50. Both evaluations, kNN and linear classifier, were conducted on the last CNN layer. Note that both Instance [53], SimCRL [6] and ISL [54] are methods of instance discrimination. This table shows that clustering-based representation-learning methods could learn representations that are as good as that of instance discrimination or even better. Compared to Table 1, we find that the performances on ResNet-50 overall are better than on AlexNet, which is reasonable and predictable.

To demonstrate the effectiveness of our method on a large-scale dataset and the speed of our algorithm in dealing with huge number of images, i.e., over one million, we also conducted experiments on ImageNet with AlexNet. As can be seen from

Table 3. Evaluation on ImageNet. “${}^{*}$” denotes training on larger AlexNet.”3k × 10/1” denotes 3000 clusters and 10/1 heads (10/1 fully connected layers attached at the end of the architecture). The best results are highlighted by the bold.

Classifier	Linear Classifier					kNN
Feature	conv1	conv2	conv3	conv4	conv5	FC
Supervised [50]	19.3	36.3	44.2	48.3	50.5	-
Random [50]	11.6	17.1	16.9	16.3	14.1	3.5
Inpainting [55]	14.1	20.7	21.0	19.8	15.5	-
BiGAN [56]	17.7	24.5	31.0	29.9	28.0	-
Instance retrieval [42]	16.8	26.5	31.8	34.1	35.6	31.3
RotNet [57]	18.8	31.7	38.7	38.2	36.5	-
AND ${}^{*}$ [41]	15.6	27.0	35.9	39.7	37.9	31.3
CMC ${}^{*}$ [39]	18.4	33.5	38.1	40.4	42.6	-
AET ${}^{*}$ [2]	19.3	35.4	44.0	43.6	42.4	-
SeLa [3k × 10] ${}^{*}$ [8]	20.3	32.2	38.6	41.4	39.6	-
ISL [54]	17.3	29.0	38.4	43.3	43.5	38.9
OTL [3k × 1] ${}^{*}$ (ours)	20.6	34.0	41.2	44.6	43.8	36.7

, our method, without strong augmentations and LCT, achieves state-of-the-art performances in evluations of both linear and kNN classifiers. In our model, comparing the representations from the five CNN layers, representations from the last two layers perform better than that of the first three layers.

4.2. Impact of Even Pseudo-Label Distribution on Performances

We have to emphasize, again, that evenly distributing pseudo-labels is in order to make the representations highly discriminative. Besides even distribution, our algorithm can conveniently set various unevenly distributed pseudo-labels, which can be performed by modifying the $\overline{n}\left(F\right)$ in frequency indicator C (11) as the desired distribution.

Unevenly distributed pseudo-label. To investigate the influences of unevenness of pseudo-labels’ distribution, various uneven target distributions of pseudo-labels were set, as shown in Figure 3 (for more details, see Appendix E) for training on CIFAR-10/100. The kNN evaluations for different target distributions are given in

Table 4. CIFAR-10/100: evaluations for unevenly distributed pseudo-labels. The best results are highlighted by the bold.

CIFAR-10
std	0	352.3	524.4	654.4	654.3	857.5
kNN	79.4	78.2	77.7	77.3	76.1	75.4
CIFAR-100
std	0	87.5	130.4	162.8	189.8	213.4
kNN	48.9	48.0	46.5	44.72	45.4	44.9

(these models were trained on AlexNet with fewer epochs than in Table 1 and without strong augmentations and the LCT technique). As one can see from both tables, the accuracy of kNN decreases as evenness decreases. This is consistent with the argument made in the method section that evenly distributed pseudo-labels are good for learning representations.

Unevenly distributed datesets. The standard datasets used in this paper are evenly distributed, i.e., each class contains the same number of images. However, the effectiveness of our method does not depend on the data being evenly distributed. In order to show that, we considered five unevenly distributed datasets which are made from CIFAR-10 by deleting different numbers of images from different classes. As a comparison, five evenly distributed datasets were made, and each has the same number of samples as the corresponding dataset in the five unevenly distributed ones.

In

Table 5. Unevenly VS evenly distributed samples: kNN evaluation.

Dataset	D_100	D_200	D_300	D_400	D_500
Uneven	77.0	76.8	76.0	75.4	73.6
Even	77.4	77.0	76.8	75.8	74.5

(again with fewer epochs and without strong augmentations and the LCT technique), for unevenly distributed “D_100”, we deleted 0, 100, 200, ⋯ 900 images from class 0, 1, 2, ⋯, 9, while evenly distributed “D_100” has the same total number of images but the ground truth is evenly distributed. Similarly, for unevenly distributed “D_200”, we deleted 0, 200, 400, ⋯ 1800 images from class 0 to 9. For unevenly distributed data “D_500”, the number of images in the first class is ten times the number of images in the last class.

We can see from Table 5 that our method is almost unaffected by whether the actual data distribution is even or not. As can be expected, evenly distributed data leads to slightly better performances than that on the unevenly distributed data. This experiment demonstrates that the success of our algorithm does not rely on the even distribution of the datasets.

4.3. Application to Remote-Sensing Images

For all remote-sensing datasets, we divided images into two groups: the training sets and testing sets, that is, we randomly picked around 80% images for training and the others for testing. All training only involved training sets without using any labels. For all datasets, we employed the same backbone, i.e., ResNet-18. All the remote-sensing images were resized to 256 × 256 and then cropped to 224 × 224 and for the strong augmentations we used the strategy from [10] with Augment = 10, n${}_{\mathrm{holes}}$ = 1, and length = 16. In this part, we will apply our method to seven remote-sensing image datasets: HMHR (How to Make High Resolution Remote Sensing Image dataset), EuroSAT, SIRI WHU google, UCMerced LandUse, AID, PatternNet, NWPU RESISC45.

For the very small datasets—HMHR, SIRI WHU google, UCMerced LandUse—we used pretrained models. The training epoch for these three datasets is 64, and we set initial learning rate as 0.03 and drops to 0.003, 0.0006 at epoch 32, 48. For the training without a pretrained model, the total training epoch was set to 300, and learning rate initially was 0.03 which dropped to 0.003 and 0.0006 at epoch 160 and 240. For all the training, the momentum was set to 0.9. For all datasets, the k was set to 128.

This is a dataset that is made from google map via LocalSpece Viewer. In this dataset, there only contains 533 images with five classes: building, farmland, greenbelt, wasteland, water. The resolution and spatial resolution of the remote-sensing images are 256 × 256 and 2.39 m. To prevent the network from overfitting, we used the strong augmentations [10] and the pretrained models trained from other remote-sensing images datasets: PatternNet, NWPU RESISC45 and EuroSAT. All the pretained models were trained through our method discussed in the Methods section and no labels were used in the training. For supervised training, we only considered the pretrained model on PatternNet. From

Table 6. HMHR: How-to-make-high-resolution-remote-sensing-image-dataset.

Method	k	Supervised	OTL (Ours)
Pretrained on		PatternNet	PatternNet	NWPU RESISC45	EuroSAT
kNN	10	82.9	79.0	75.2	72.4
	50	83.8	78.1	76.2	73.3
	100	84.8	78.1	78.1	76.2
	200	83.8	77.1	79.0	75.2
Linear Classifier	-	88.5	85.7	83.8	81.9

, we see that our method works well on very small datasets with few classes. There is only a small gap ($2.8\%$) between supervised method and ours in linear classifier evaluation. Although using different pretrained models has different performances, all of them are effective and perform well.

EuroSAT [58,59] is a dataset that for land use and land cover classification. It is consist of 10 classes with 27,000 labeled images of annual crop, forest, highway, reiver, sealake and so on. The resolution and spatial resolution of these images are 64 × 64 and 10 m.

Table 7. EuroSAT: Land Use and Land Cover Classification with Sentinel-2.

Method	k	Supervised	OTL (Ours)
kNN	10	96.8	94.4
	50	96.6	94.7
	100	96.3	94.6
	200	95.9	94.4
Linear Classifier	-	96.9	95.9

shows that the gap between supervised method and ours is only $1\%$ in Linear classifier evaluation. In kNN evaluation, the performance of our method is also very close to the performance of supervised learning.

The dataset of SIRI WHU google [60] contains 12 classes images with 2400 images. There are 200 images for each class. The class of images includes: agriculture, commercial, harbor, idle land, industrial, meadow, overpass, park, pond, river, water, residential. The pixel and spatial resolution of images in this dataset are $200\times 200$ and 2 m. As can be seen from

Table 8. SIRI WHU google.

Method	k	Supervised	OTL (Ours)
Pretrained on		PatternNet	PatternNet	NWPU RESISC45	EuroSAT
kNN	10	86.9	85.4	89.0	80.8
	50	83.8	82.5	85.6	77.9
	100	83.5	80.4	84.6	76.0
	200	81.3	79.0	83.3	73.8
Linear Classifier	-	89.6	88.6	89.7	85.7

, using the pretained model on NWPU RESISC45 have the best performance which even is better than the that of supervised learning based on pretrained model on PatternNet. Representations learned based on pretrained model on PatternNet perform closely to that of supervised learning.

UCMerced landUse [61] gives the images that were manually extracted from large images from the USGS National Map Urban Area Imagery collection for various urban areas around the country. Each image measures 256 × 256 pixels, with a 0.3 m spatial resolution. There are 21 classes with 2100 images, including airplane, beach, buildings, freeway, golfcourse, tenniscourt and so on and there are 100 images for each class.

Table 9. UCMerced LandUse.

Method	k	Supervised	OTL (Ours)
Pretrained on		PatternNet	PatternNet	NWPU RESISC45	EuroSAT
kNN	10	88.1	86.9	86.2	81.7
	50	87.4	87.9	86.4	82.4
	100	87.1	88.1	86.2	82.6
	200	86.2	88.6	86.2	82.6
Linear Classifier	-	89.6	90.2	89.7	87.2

demonstrates that the performance of our method based on pretrained models on PatternNet and NWPU RESISC45 is very close to supervised learning, and in Linear Classification, our method even perform better.

AID [62] is a aerial image dataset with high resolution, 600 × 600 pixels, and a 0.5–0.8 m spatial resolution, which constitutes collected sample images from Google Earth imagery. There are 30 classes with 10,000 images. This dataset is not evenly distributed and in each class there are about 200 to 400 images. The scene classes include bare land, baseball field, beach, bridge, center, church, dense residential, desert, forest, meadow, medium residential, mountain, park, parking, playground, port, railway station, resort, school and so on. This experiment shows that, for high-resolution remote-sensing images and an unevenly distributed remote-sensing dataset, our method also works well (see

Table 10. AID: a benchmark dataset for performance evaluation of aerial scene classification.

Method	k	Supervised	OTL (Ours)
kNN	10	87.8	83.6
	50	87.6	83.5
	100	86.9	82.6
	200	86.7	81.9
Linear Classifier	-	89.5	85.5

).

PatternNet [63] is a remote-sensing dataset with 30,400 high-resolution images whose resolution is 256 × 256 pixels and spatial resolution is 0.062–4.693 m. These images were collected for remote-sensing image retrieval from Google Earth imagery or via the Google Map API for some US cities. For each of the 38 classes, there are 800 images. From the results in

Table 11. PatternNet.

Method	k	Supervised	OTL (Ours)
kNN	10	96.5	96.4
	50	96.2	96.1
	100	95.7	95.7
	200	95.5	95.6
Linear Classifier	-	96.7	97.4

, we see that, in kNN evaluation, our method performs as well as that of supervised learning; while in linear classifier evaluation, our method has an even better performance.

NWPU RESISC45 [64] is made by Northwestern Polytechnical University (NWPU), which is avaiable for remote-sensing image scene classification (RESISC). It contains 31,500 images in total with a pixel resolution of 256 × 256 and a 0.2–30 m spatial resolution. This dataset cover 45 scene classes with 700 images in each class. These 45 scene classes include baseball diamond, basketball court, bridge, chaparral, church, circular-farmland, cloud, commercial area, dense residential, desert, intersection, island, lake, meadow, medium residential, mobile home park, wetland and so on. The experiment in

Table 12. NWPU RESISC45.

Method	k	Supervised	OTL (Ours)
kNN	10	92.1	89.3
	50	90.6	88.7
	100	89.7	88.3
	200	89.1	87.4
Linear Classifier	-	92.3	89.8

shows that, for a dataset with more classes, our method still has an excellent performance that is very close to performance of supervised learning.

From Table 6, Table 7, Table 8, Table 9, Table 10, Table 11 and Table 12, we see that, for the pixel resolutions of remote-sensing images ranging from $64\times 64$ to $600\times 600$, i.e., from low to high resolution, spatial resolutions ranging from 0.062 m to 30 m, class numbers ranging from 5 to 45, and total numbers of images ranging from 533 to 31,500, our method can learn good representations, which are close to, or even better than, the representations learned from supervised learning. These experiments demonstrate that our method could be widely applied to learn the representations of remote-sensing images.

To investigate the effect of choosing different training and testing samples in the experiments of remote-sensing images, we conduct four more contrast experiments on the dataset SIRI WHU google. In this dataset, there are only 2400 images, and, thus, the random choice of training and testing samples may be quite different. We randomly selected $80\%$ images as training samples and $20\%$ as testing samples four times to construct four different divisions of the dataset SIRI WHU google. The performances on the five different divisions are given in

Table 13. Five different ways to divide the dataset SIRI WHU google.

Dataset	Evaluation Method	k	Supervised	OTL (Ours)
Division 1	kNN	10	86.9	85.4
		50	83.8	82.5
		100	83.5	80.4
		200	81.3	79.0
	Linear Classifier	N/A	89.6	88.6
Division 2	kNN	10	92.5	89.6
		50	91.0	89.4
		100	89.4	87.9
		200	88.3	87.3
	Linear Classifier	N/A	92.6	92.1
Division 3	kNN	10	93.2	88.8
		50	90.6	88.3
		100	88.8	87.3
		200	88.5	87.1
	Linear Classifier	N/A	93.0	92.7
Division 4	kNN	10	93.1	89.8
		50	90.6	89.4
		100	89.4	87.9
		200	88.5	86.3
	Linear Classifier	N/A	92.8	92.5
Division 5	kNN	10	92.3	90.4
		50	90.4	89.0
		100	89.0	88.1
		200	87.9	87.1
	Linear Classifier	N/A	92.8	92.3

. From Table 13, we see that different training and testing samples would affect the performances both on supervised learning and our method. However, in all five cases, using our method, without any labels can always learn good representations for remote-sensing images.

5. Discussion

To study the robustness of our algorithm, we consider how the two parameters, cluster number k and decay rate $\beta$ (see Algorithm 1), affect the performance of our algorithm. For given N, k and ${\alpha }_0$, the decay rate is the unique parameter in our algorithm, and has to be larger than one such that the magnitude of translations will decrease as iterations increase. We conducted two groups of experiments for $\beta$ ranging from 1.5 to 6 and from 5 to 50 in Figure 4, and the results show that our algorithm can effectively create evenly distributed pseudo-labels for these wide ranges of $\beta$ values, indicating our algorithm is insensitive to this free parameter.

In Figure 5, we demonstrate that our method is not only effective but also efficient (it only needs around 20 iterations to converge) in various settings of cluster number k. In

Table 14. CIFAR-10: Evaluations for different k.

k	32	64	128	256	512	1024
$\overline{n}$	1562.5	718.3	390.6	195.3	97.7	48.8
kNN	77.1	78.6	79.4	79.0	78.9	77.8

and

Table 15. CIFAR-100: Evaluations for different k.

k	64	128	256	512	1024	2048
$\overline{n}$	718.3	390.6	195.3	97.7	48.8	24.4
kNN	43.8	46.1	47.8	48.9	49.5	48.9

(without LCT and strong augmentations), we show the kNN evaluations for the representations learned using different k, which demonstrates that our method works well for k ranging from 32 to 2048. When $k=N$, our method would be the same as instance discrimination. Increasing k would make representations more discriminative, while decreasing k would make representations closer. Representation learning requires that representations of different samples to be as distinct as possible while representations of similar samples get closer. Therefore, very large or small k are not considered in this paper: small k would not give the highly discriminative representations while large k would only consider the nearest neighbors as the same class, or even only the transformations of the same sample as instance discrimination does.

The comparisons between our method and the Sinkhorn–Knopp algorithm are given in Figure 6. The Sinkhorn–Knopp algorithm is a classical method which has been widely used, especially in transport problems. For 20 different k values and different $N\times K$ matrices, our method always obtained more uniform pseudo-labels (see Figure 6).

The main advantage of the Sinkhorn–Knopp algorithm is its speed of convergence. For creating pseudo-labels from outputs of ImageNet, the Sinkhorn–Knopp algorithm converges within 2 min [8]. In Figure 7, we give the standard deviations of pseudo-labels for ImageNet and the time for producing them at each epoch by our method. As can been seen from Figure 7a, for all epochs, the pseudo-labels created from our method are very uniform and the average standard deviation is only 1.07. From (b) in Figure 7, we see that our algorithm is very fast to converge even for 1.28 million pictures and 3000 clusters. The average time for approaching even distribution is only 12.68 s on GPU (NVIDIA A100).

6. Conclusions

In this paper, under a clustering model where images are clustered based on the representations of the images, we found a quantity that can be used to compare the discriminativeness of the representations and, based on this quantity, we demonstrated that evenly distributing images into clusters requires their representations to be most discriminative. We developed an algorithm to translate outputs such that pseudo-labels can be created evenly according to representations of images while keeping the smoothness of the model unchanged. Extensive experimental results demonstrate that our method is very effective and can learn representations that are as good as or better than the state-of-the-art methods. We then applied our method to various remote-sensing image datasets and the representations learned by our method are very close to, or even better than, the representations learned by supervised learning.