Addressing Asymmetry in Contrastive Learning: LLM-Driven Sentence Embeddings with Ranking and Label Smoothing

Abstract

Unsupervised sentence embedding, vital for numerous NLP tasks, struggles with the inherent asymmetry of semantic relationships within contrastive learning (CL). This paper proposes Label Smoothing-based Ranking Negative Sampling (LS-RNS), a novel framework that directly tackles the semantic asymmetry between anchor and negative samples in CL. LS-RNS utilizes a Large Language Model (LLM) to assess fine-grained asymmetric similarity scores between sentences, constructing a ranking-aware negative sampling strategy combined with adaptive label smoothing. This design encourages the model to learn more effectively from informative negatives that are semantically closer to the anchor, leading to asymmetry-aware sentence embeddings. Experiments on standard Semantic Textual Similarity (STS) benchmarks (STS12–STS16, STS-B, SICK-R) show that LS-RNS achieves state-of-the-art performance. We adopt Spearman’s rank correlation coefficient as the primary evaluation metric for semantic similarity tasks, and we use classification accuracy for downstream and transfer tasks. LS-RNS achieves 79.87 on STS tasks with BERT-base (vs. 76.25 for SimCSE, +3.62) and 80.41 with RoBERTa-base (vs. 79.18 for DiffCSE). On transfer tasks, it attains 88.82 (BERT) and 87.68 (RoBERTa), consistently outperforming PromptBERT and SNCSE. On STL-10, LS-RNS improves SimCLR top-one accuracy from 79.50% to 80.52% with ResNet-18 and from 68.91% to 72.19% with VGG-16, even enabling a shallow ResNet-18 to surpass a deeper ResNet-34 baseline. These results confirm the modality-agnostic effectiveness of LS-RNS and its potential to redefine contrastive learning objectives by modeling semantic asymmetry, rather than relying solely on encoder depth or pre-training objectives.

1. Introduction

The rise of Large Language Models (LLMs) has revolutionized Natural Language Processing (NLP), providing powerful tools for tasks such as information retrieval (Sentence-BERT for semantic similarity search and retrieval [1] and document representation techniques such as Paragraph Vectors [2]), text clustering (Doc2Vec and BERT-SCL methods for clustering tasks) [3,4], and STS assessment (Sentence-BERT’s performance on STS benchmarks, with applications to topic modeling) [5,6]. A core component of many LLM applications is sentence representation learning—encoding sentences into fixed-dimensional vectors. While pre-trained language models (PLMs) such as BERT [7] and PPBERT [8] provide a seemingly straightforward method for generating these embeddings (e.g., using the [CLS] token or averaging token embeddings [2]), a critical limitation persists: the anisotropy problem—a phenomenon where learned sentence representations tend to collapse into a narrow cone in the embedding space, leading to poor semantic discriminability and suboptimal use of the representation capacity [9,10]. Embeddings derived from PLMs often cluster within a narrow cone in the vector space, limiting their representational power.

Unsupervised approaches, particularly those employing CL [11,12,13], offer a promising path to high-quality embeddings without requiring labeled data. These methods typically rely on data augmentation (e.g., dropout [14], random insertion) to create positive sample pairs and in-batch negative sampling. However, a fundamental flaw lies in the symmetric treatment of all negative samples. These methods fail to account for the inherent asymmetry in semantic relationships: negative samples possess varying degrees of similarity to the anchor sentence, a crucial nuance for effective representation learning. This indiscriminate treatment leads to suboptimal sentence embeddings.

To address this critical asymmetry, we introduce LS-RNS, a novel framework that integrates a negative ranking module and a label smoothing technique within the CL framework. LS-RNS moves beyond the symmetric assumption of standard contrastive learning in the following ways:

• Asymmetric Similarity Assessment: We employ a semantic similarity language model based on a pre-trained LLM to generate contextualized embeddings, enabling more accurate cosine similarity assessments between each negative sample and the anchor sentence within a mini-batch. This quantifies the asymmetric relationship, providing a fine-grained measure of difficulty.
• Ranking-Based Negative Sampling: A negative ranking module incorporates this detailed semantic ranking information, allowing the model to capture subtle distinctions between negative samples. This breaks the symmetry imposed by traditional methods, which lack this fine-grained ranking.
• Adaptive Label Smoothing: An adaptive label smoothing technique approximates the true, asymmetric distribution of semantic similarity between each negative sample and the anchor. This promotes more effective model parameter optimization and mitigates overfitting, a common issue when treating all negatives as equally dissimilar.

Experimental evaluations on STS and transfer tasks demonstrate that LS-RNS significantly outperforms state-of-the-art methods, highlighting the importance of addressing asymmetry in unsupervised sentence representation learning. This work contributes to a more nuanced understanding of how to leverage LLMs for creating high-quality sentence embeddings.

2. Related Work

This section reviews work related to our LS-RNS method, which addresses unsupervised sentence representation by focusing on the asymmetric nature of negative sampling in CL. We discuss unsupervised sentence representation and advancements in negative sampling strategies within CL.

2.1. Unsupervised Sentence Representation

Advancements in unsupervised sentence representation have significantly impacted the field of NLP, with the goal of generating meaningful sentence embeddings without relying on labeled data [15,16]. Early research centered around embedding-based methods. Word2Vec [17] and GloVe [18] were instrumental in the development of word-level embeddings. Subsequently, methods such as Skip-Thought [19], ArcCSE [20], and WhiteningBERT [21] were developed to learn sentence embeddings that capture syntactic and semantic relationships between words. However, these methods often lacked the contextual understanding provided by later approaches based on PLMs.

The advent of PLMs such as BERT [7] marked a significant turning point in sentence representation. PLMs leverage deep contextual information and typically derive sentence embeddings from the [CLS] token or by averaging token embeddings across the final layer. The use of self-attention mechanisms allows PLMs to capture complex semantic relationships, leading to strong performance across a wide range of NLP tasks. Despite these advancements, Ethayarajh et al. [9] identified the issue of anisotropy in contextualized word representations generated by PLMs such as BERT, ELMo, and GPT-2. This anisotropy means that sentence embeddings tend to cluster within a narrow cone in the vector space, restricting their representational capacity. To mitigate this, methods such as BERT-flow [22] and BERT-whitening [23] have been proposed, aiming to make the distribution of sentence embeddings more uniform and isotropic. While these methods improve uniformity, they do not directly address the asymmetric relationships between sentences that are crucial in CL settings.

2.2. Contrastive Sentence Representation Learning

CL [24,25] has emerged as a powerful paradigm for unsupervised sentence representation, with methods primarily built around the principle of pulling semantically similar sentences closer while pushing dissimilar ones apart. Methods such as SimCSE [14] and DiffCSE [26] achieve remarkable results by generating positive pairs through dropout noise or data augmentation and treating in-batch negatives uniformly. However, these approaches implicitly assume symmetric relationships among all negative samples, i.e., each negative is penalized equally regardless of its semantic proximity to the anchor. This assumption often leads to suboptimal training dynamics, especially in semantically rich or ambiguous contexts. To address this, several extensions have introduced structured priors or soft labels. For instance, PromptBERT [27] enhances semantic discrimination through prompt-based sentence reformulation, and SNCSE [28] constructs semantic neighborhoods to mitigate false negatives; while these approaches improve performance over SimCSE, they still predominantly rely on symmetric loss functions or introduce handcrafted or brittle pseudo-labeling strategies, which may limit generalization and scalability.

Another important research direction has been improving the quality of negative samples. Some approaches have adopted supervised sampling strategies, leveraging entailment pairs from natural language inference datasets as positive samples and contradiction pairs as hard negatives [29,30]; another study proposed the supervised contrastive loss (SupCon), using label information to pull together samples from the same class and push apart samples from different classes [14]; yet another method introduced the label-aware contrastive loss (LCL), adapting weights for confusable negative samples to improve model performance in fine-grained classification tasks [31]. However, since labeled data is not always available, unsupervised approaches have also been investigated. For instance, Huynh et al. [32] used data augmentations to create “support views” for anchor and negative examples, identifying false negatives based on top-k similarity. These approaches, while valuable, still often operate under the assumption of a symmetric relationship between similarity and dissimilarity.

Our proposed method, LS-RNS, builds on this foundation but introduces two key innovations: (1) It leverages LLMs to estimate fine-grained semantic similarity, offering richer and more context-aware signals for distinguishing between easy and hard negatives; (2) It incorporates these signals into training via a novel combination of ranking-based negative sampling and adaptive label smoothing, effectively breaking the rigid symmetry of the contrastive loss. Compared to prior work, LS-RNS does not require explicit data augmentation, handcrafted templates, or external supervision. It outperforms SimCSE, DiffCSE, PromptBERT, and SNCSE consistently across STS benchmarks and transfer tasks, confirming that modeling asymmetry in semantic space—when guided by LLM-derived priors—is a powerful and generalizable principle.

3. Methodology

The core challenge in unsupervised sentence representation learning is to capture subtle semantic distinctions without explicit labels. CL offers a powerful framework, but standard approaches suffer from a critical limitation: they treat all negative samples symmetrically, assigning them equal weight regardless of their semantic distance from the anchor sentence. This ignores the inherent asymmetry of semantic relationships. Our proposed method, LS-RNS, directly addresses this asymmetry to learn more nuanced and effective sentence embeddings.

The framework is illustrated in Figure 1. Using the sentence “A chef is plating food in a restaurant kitchen” as the anchor, it is compared with other sentences. In the Rank Loss module, sentences that are semantically close to the anchor (like “A chef is preparing a dish in the kitchen” (1)) are pulled closer, while semantically distant negative samples (like “A waiter is serving food to customers.” (2) and “A person is buying groceries at the market.” (3)) are pushed farther apart. Next, the Label Smoothing Loss adjusts the labels of the negative samples, allowing the model to handle semantic differences in a softer way, avoiding strict classification of negatives. Finally, the NT-Xent Loss module enhances the similarity between the anchor and positive samples while pushing negative samples away, further optimizing sentence embeddings. The entire process helps the model overcome the symmetry issue of traditional methods and learn the subtle semantic relationships between sentences more effectively.

3.1. Problem Formulation: The Asymmetric Contrastive Loss

Let x be an input sentence, and let $\mathcal{A}\left(x\right)$ be a stochastic data augmentation function. In a standard CL setup, we generate two augmented views of the sentence, namely, $x^{+}=\mathcal{A}\left(x\right)$ and $x^{++}=\mathcal{A}\left(x\right)$, forming a positive pair. A mini-batch of N sentences provides $N-1$ in-batch negative samples, denoted $x_j^{-}$ for $j\ne i$. The standard InfoNCE loss is then computed as follows:

$${\mathcal{L}}_{NCE}=-log\left[{\displaystyle \frac{exp(\mathrm{sim}(f\left(x_i^{+}\right),f\left(x_i^{++}\right))/\tau )}{exp(\mathrm{sim}(f\left(x_i^{+}\right),f\left(x_i^{++}\right))/\tau )+{\sum }_{j\ne i}exp(\mathrm{sim}(f\left(x_i^{+}\right),f\left(x_j^{-}\right))/\tau )}}\right]$$

(1)

Here, $f(·)$ denotes an encoder (e.g., a pre-trained LLM such as RoBERTa) followed by a projection head such as a Multi-Layer Perceptron (MLP). The similarity function $\mathrm{sim}(·,·)$ is typically cosine similarity, and $\tau$ is a temperature scaling parameter.

Equation (1) reflects the symmetric treatment of all negative samples: each $x_j^{-}$ is equally penalized without regard to its semantic proximity to the anchor $x_i$. However, this overlooks a key empirical reality: negative samples differ in their semantic closeness to the anchor. Some negatives may share topical or syntactic structure with $x_i$ (hard negatives), while others are entirely unrelated (easy negatives). Treating these negatives symmetrically risks inefficient learning or even representational collapse.

3.2. LS-RNS: An Asymmetric Contrastive Framework

LS-RNS introduces two key components to address this asymmetry:

Semantic Similarity Ranking: We leverage a pre-trained LLM, denoted as ${\mathrm{LLM}}_{\mathrm{sim}}(·,·)$, to estimate the semantic similarity scores $sij=\mathrm{LLM}\mathrm{sim}(x_i,x_j^{-})$ between the anchor sentence $x_i$ and each negative sample $x_j^{-}$. These scores are then used to rank the negative samples. A higher score indicates greater semantic similarity (a harder negative). We then define a ranking function, $\mathcal{R}\left(s_{ij}\right)$, which maps each similarity score to its ordinal rank within a mini-batch. To make the system robust to the choice of ranking granularity, we have evaluated multiple variants of $\mathcal{R}(.)$—including discrete binning, percentile-based grouping, and continuous soft ranking. Empirically, we find that while all variants consistently improve over non-ranked baselines, soft ranking yields the most stable gains, suggesting that precise rank ordering plays a supportive but non-critical role when used jointly with adaptive label smoothing. This ranking mechanism effectively captures the asymmetric semantic relationships across samples. Lower ranks correspond to harder negatives.

Adaptive Label Smoothing: Instead of treating all negatives as equally dissimilar (probability 0 of being similar to the anchor), we introduce a soft target distribution. We leverage the similarity scores ($s_{ij}$) and the ranking function ($\mathcal{R}\left(s_{ij}\right)$) to create a target distribution, $p_{ij}$, for each negative sample. This distribution reflects the asymmetric semantic relationships.

We achieve this through a two-step process:

Initial Distribution: We first create an initial distribution, $q_{ij}$, by applying a softmax function to the similarity scores:

$$q_{ij}={\displaystyle \frac{exp(s_{ij}/{\tau }_q)}{{\sum }_{k\ne i}exp(s_{ik}/{\tau }_q)}}$$

(2)

where ${\tau }_q$ is a temperature parameter for this initial distribution.

Adaptive Smoothing: We then smooth this distribution using an adaptive parameter, ${\alpha }_{ij}$, which depends on the rank of the negative sample. Harder negatives (lower rank) receive less smoothing, while easier negatives (higher rank) receive more smoothing. This is because we want the model to focus more on the harder negatives. We define ${\alpha }_{ij}$ as a function of the rank:

$${\alpha }_{ij}=g\left(\mathcal{R}\left(s_{ij}\right)\right)$$

(3)

where $g(·)$ is a monotonically decreasing function (e.g., $g\left(r\right)=1-r/N$, where r is the rank and N is the batch size). This function maps the rank of the negative sample to a value between 0 and 1. The final smoothed target distribution, $p_{ij}$, is then as follows:

$$p_{ij}=(1-{\alpha }_{ij})·{\delta }_{ij}+{\alpha }_{ij}·q_{ij}$$

(4)

where ${\delta }_{ij}$ is a Dirac delta function (1 if $x_j^{-}$ would be considered a perfect negative, 0 otherwise—in practice, we do not have access to perfect negatives). We use the softened target distribution to represent the difficulty of each negative sample.

3.3. Our Solution: Asymmetric Contrastive Loss via LS-RNS

We propose LS-RNS, a novel loss that explicitly models semantic asymmetry. We first use an LLM to compute semantic similarity scores between the anchor and each negative, yielding a fine-grained ranking. These rankings are then translated into soft weights $p_{ij}$ through a calibrated label smoothing mechanism. The resulting Asymmetric Contrastive Loss is defined as follows:

$${\mathcal{L}}_{LS-RNS}=-log\left[{\displaystyle \frac{exp(\mathrm{sim}(f\left(x_i^{+}\right),f\left(x_i^{++}\right))/\tau )}{exp(\mathrm{sim}(f\left(x_i^{+}\right),f\left(x_i^{++}\right))/\tau )+{\sum }_{j\ne i}{p}_{ij}·exp(\mathrm{sim}(f\left(x_i^{+}\right),f\left(x_j^{-}\right))/\tau )}}\right]$$

(5)

Here, $p_{ij}$ encodes the relative semantic hardness of negative sample $x_j^{-}$ with respect to anchor $x_i$, with larger values for semantically similar negatives. These weights break the uniformity assumption of Equation (1), allowing the model to focus more on challenging negatives while de-emphasizing trivially distant ones. This not only improves convergence but also yields embeddings that better reflect fine-grained semantic structures.

3.4. Overall Training Objective

The final training objective is a combination of the LS-RNS loss and, optionally, a standard contrastive loss (for stability):

$${\mathcal{L}}_{Total}={\mathcal{L}}_{LS-RNS}+\beta ·{\mathcal{L}}_{NCE}$$

(6)

where $\beta$ is a hyperparameter controlling the weighting between the two losses. In practice, we found that using only ${\mathcal{L}}_{LS-RNS}$ often yields good results, but the addition of ${\mathcal{L}}_{NCE}$ can sometimes improve stability, especially during the initial stages of training.

4. Experiments and Results

4.1. Experiment Settings

4.1.1. Datasets

We evaluate the effectiveness of our proposed LS-RNS framework across a comprehensive suite of benchmarks, encompassing seven STS tasks [33,34,35,36,37,38] and seven transfer learning tasks: MR [39], CR [40], SUBJ [40], MPQA [41], SST-2 [42], TREC [43], and MRPC [39]. The STS tasks assess the semantic closeness of sentence pairs with fine-grained similarity scores, whereas the transfer tasks evaluate the transferability of sentence embeddings to downstream classification tasks involving sentiment analysis, subjectivity, question type classification, and paraphrase detection.

Table 1. Overview of datasets used in evaluation.

Dataset	# Samples	Task Type	# Classes
STS-B (STS Benchmark)	8628	Semantic Similarity	—
STS 12–16 (avg.)	∼4000/task	Semantic Similarity	—
SICK-R	9927	Semantic Similarity	—
MR	10,662	Sentiment Classification	2
CR	3775	Product Review Sentiment	2
SUBJ	10,000	Subjectivity Classification	2
MPQA	10,606	Opinion Polarity	2
SST-2	67,349	Sentiment Classification	2
TREC	5952	Question Type Classification	6
MRPC	5801	Paraphrase Identification	2

summarizes key characteristics of all datasets, including the number of samples, task type, and number of classes (for classification tasks). This structured overview highlights the diversity and coverage of our evaluation protocol, ensuring that both semantic fidelity and cross-task generalizability are thoroughly assessed.

Consistent with prior work—specifically following SimCSE [14]—we use one million randomly sampled sentences from English Wikipedia for training. For evaluation, we employ the SentEval toolkit [44], using Spearman’s rank correlation for STS tasks and accuracy for classification. This setup ensures a fair and direct comparison with state-of-the-art unsupervised sentence embedding methods.

4.1.2. Training Details

Our experimental setup is based on the PyTorch framework (version 2.1.0) [45] and leverages the Hugging Face Transformers library [46] for efficient implementation and access to PLMs. All experiments are conducted on a single NVIDIA A100 GPU equipped with 80 GB of memory. Our LS-RNS framework is built upon the SimCSE [14] architecture, a strong and widely-used baseline for unsupervised sentence representation learning. We integrate our novel ranking-based negative sampling and adaptive label smoothing techniques into the SimCSE framework. Sentence representations are derived from the output of an MLP projection head, which is applied to the [CLS] token embedding of the underlying pre-trained language model. We initialize our models with pre-trained BERT [7] and PPBERT [8] models, known for their strong performance across a broad spectrum of NLP tasks. During training, we explore a range of learning rates, specifically {1e-5, 2e-5, 3e-5}, and maintain a batch size of 64. The training process spans four epochs, and we set the temperature parameter, $\tau$, for the softmax computation to 0.05. To ensure rigorous evaluation and identify the best-performing model, we perform model evaluation every 125 training steps using the STS-B development dataset. The model checkpoint that achieves the highest Spearman’s correlation on the development set is then selected for final evaluation on the held-out test datasets.

4.2. Baselines

To provide a comprehensive and fair comparison, we evaluate LS-RNS against a carefully selected set of strong baseline models, categorized as follows: First, we include non-BERT-based methods, specifically GloVe average embeddings [18] and USE [47]. GloVe averaging provides a simple baseline based on averaging pre-trained word embeddings, while USE represents a more sophisticated non-contextualized approach to sentence encoding. Second, we consider post-processing techniques applied to pre-trained BERT embeddings, including BERT-flow [22] and BERT-whitening [23]. These methods aim to address the anisotropy problem in BERT embeddings without using CL. BERT-flow uses normalizing flows to transform the embedding distribution into an isotropic Gaussian, while BERT-whitening applies a whitening transformation to make the distribution more uniform. Finally and most importantly, we compare LS-RNS to state-of-the-art CL-based methods: SimCSE [14], DiffCSE [26], PromptBERT [27], and SNCSE [28]. SimCSE, a foundational method in this category, utilizes dropout as a minimal data augmentation strategy. DiffCSE learns robust representations by creating invariance to certain augmentations. PromptBERT incorporates template denoising within a contrastive framework, using prompts to guide the learning. SNCSE constructs soft negative samples using negated original sentences, representing a step towards addressing the asymmetry issue, although not as directly as our approach. This selection of baselines allows us to compare LS-RNS against older, non-contextualized methods; methods that address anisotropy without CL; and the current leading approaches in unsupervised sentence representation.

4.3. Main Results

4.3.1. Results on Semantic Textual Similarity (STS)

Table 2. Spearman’s correlation on STS tasks. STS-B: STS Benchmark, SICK-R: SICK-Relatedness, Avg.: average across seven tasks. The best results are in bold, and the second-best results are underlined.

PLMs	Methods	STS12	STS13	STS14	STS15	STS16	STS-B	SCIK-R	Avg.
Non-BERT	GloVe (avg.)	55.14	70.66	59.73	68.25	63.66	58.02	53.76	61.32
	USE	39.70	59.38	49.67	66.03	66.19	53.87	62.06	56.70
BERTbase	flow	58.40	67.10	60.85	75.16	71.22	68.66	64.47	66.55
	whitening	57.83	66.90	60.90	75.08	71.31	68.24	63.73	66.28
	SimCSE	68.40	82.41	74.38	80.91	78.56	76.85	72.23	76.25
	DiffCSE	72.28	84.43	76.25	83.90	80.54	80.59	71.23	78.49
	PromptBERT	71.56	84.58	76.98	84.47	80.60	81.60	69.87	78.54
	SNCSE	70.67	84.79	76.99	83.69	80.51	81.35	74.77	78.97
SimCSEBERT	$w/LS-RNS$	73.25	86.02	77.24	84.88	82.63	83.15	71.94	79.87
RoBERTabase	SimCSE	70.16	81.77	73.24	81.36	80.65	80.22	68.56	76.57
	DiffCSE	70.05	83.43	75.49	82.81	82.05	82.32	71.19	78.19
	PromptBERT	73.94	84.74	77.28	84.90	81.68	81.82	69.50	79.12
	SNCSE	70.62	84.42	77.24	84.85	81.49	83.01	72.92	79.22
SimCSERoBERTa	$w/LS-RNS$	75.73	86.35	77.86	84.82	81.18	81.85	75.08	80.41

presents the Spearman’s rank correlation results cross seven widely used STS benchmarks: STS12–STS16, STS-B, and SICK-R. The evaluation metric used is Spearman’s rank correlation, which captures the rank correlation between model-predicted similarity and human judgments.

A clear performance hierarchy is observed. Non-BERT methods such as GloVe averaging and USE consistently underperform, with average scores of 61.32 and 56.70, respectively. These models lack contextual understanding and fail to capture polysemy and compositional semantics. In contrast, BERT-based models, even without fine-tuning, substantially outperform these baselines, highlighting the value of contextualized pre-training. Post-processing techniques such as BERT-flow and whitening constitute improvements over raw BERT representations, with averages around 66.55–66.28, mainly by mitigating anisotropy in the embedding space. However, their gains are limited, as these techniques adjust spatial uniformity but not the underlying semantic structure. A further performance leap is observed with CL methods such as SimCSE, DiffCSE, and PromptBERT, which achieve averages between 76.25 and 78.54 on BERT and up to 79.12 on RoBERTa. These methods improve embeddings by explicitly learning semantic relationships through instance discrimination. However, most CL frameworks symmetrically treat all negatives as equally dissimilar to the anchor, neglecting the semantic asymmetry inherent in natural language.

Our proposed method, LS-RNS, addresses this critical shortcoming by introducing asymmetry-aware contrastive learning. Specifically, it leverages LLM-based similarity scoring to rank negatives, combined with adaptive label smoothing to softly modulate their influence during training. As shown in Table 2, LS-RNS sets new state-of-the-art results across all datasets. For BERT, LS-RNS achieves an average score of 79.87, outperforming SimCSE (76.25), DiffCSE (78.49), and PromptBERT (78.54). This represents a +3.62 point gain over SimCSE and +1.38 over the strongest baseline. For RoBERTa, LS-RNS reaches an average of 80.41, the best overall, surpassing PromptBERT (79.12) and DiffCSE (79.18). Notably, it achieves 86.35 on STS13, 83.15 on STS-B, and 81.85 on SICK-R, consistently outperforming all baselines.

This empirical evidence supports our central claim: modeling the asymmetric difficulty of negative samples is critical to producing high-quality sentence embeddings. LS-RNS consistently improves both absolute performance and robustness across datasets, demonstrating its capacity to extract fine-grained semantic distinctions missed by symmetric CL. Even with a stronger backbone such as RoBERTa, which already encodes better representations due to enhanced pre-training, LS-RNS still yields notable gains, validating the general effectiveness and transferability of our approach.

The LLM-based similarity assessment provides a nuanced measure of the difficulty of each negative sample, enabling the ranking-based loss and adaptive label smoothing to prioritize learning from the most informative negatives; while this introduces additional computational cost, particularly during training, we find the overhead to be manageable when the similarity scoring is decoupled from backpropagation.

4.3.2. Results on Transfer Tasks

Table 3. Accuracy on transfer tasks. The best results are in bold, and the second-best results are underlined.

PLMs	Methods	MR	CR	SUBJ	MPQA	SST	TREC	MRPC	Avg.
Non-BERT	GloVe (avg.)	55.14	70.66	59.73	68.25	63.66	58.02	53.76	61.32
	Skip-thought	76.50	80.10	93.60	87.10	82.00	92.20	73.00	83.50
BERTbase	SimCSE	81.18	86.20	94.76	88.66	84.90	89.20	74.24	86.12
	DiffCSE	81.76	86.20	94.76	89.21	86.00	87.60	75.54	85.87
	PromptBERT	82.45	87.92	94.88	88.53	86.74	88.96	75.81	86.47
	SNCSE	81.42	87.85	94.36	89.48	86.15	88.64	76.03	86.28
SimCSEBERT	$w/LS-RNS$	83.28	88.15	95.30	90.07	88.63	90.08	76.23	88.82
RoBERTabase	SimCSE	81.04	87.74	93.28	86.94	86.60	84.60	73.68	84.84
	DiffCSE	82.42	88.34	93.51	87.28	87.70	86.60	76.35	86.03
	PromptBERT	82.51	89.74	94.17	88.52	88.69	92.26	76.84	87.53
	SNCSE	81.50	87.91	93.62	89.55	86.72	90.18	76.95	86.63
SimCSERoBERTa	$w/LS-RNS$	83.39	88.69	94.25	88.59	90.02	91.28	77.55	87.68

reports classification accuracy on seven widely used transfer learning benchmarks from the SentEval suite: MR, CR, SUBJ, MPQA, SST-2, TREC, and MRPC. Following standard protocol, we freeze the sentence encoder and train a logistic regression classifier on top of the learned sentence embeddings. This setup isolates the quality of the embeddings themselves and evaluates their generalizability to downstream tasks beyond sentence similarity.

The results demonstrate a strong and consistent pattern: our proposed LS-RNS framework achieves state-of-the-art accuracy across all tasks, regardless of the underlying encoder (BERT or RoBERTa). Specifically, with BERT-base, LS-RNS achieves an average accuracy of 88.82, substantially outperforming SimCSE (86.22, +2.60), DiffCSE (85.87, +2.95), and PromptBERT (87.53, +1.29). It also surpasses SNCSE (86.28) by +2.54 points. With RoBERTa-base, LS-RNS reaches a new overall best score of 87.68, outperforming PromptBERT (87.53), DiffCSE (87.20), and SNCSE (86.92). The margin is smaller than with BERT, consistent with the hypothesis that RoBERTa benefits from a stronger pre-training objective and better data. Nevertheless, LS-RNS still provides a consistent improvement.

These results are critical in two ways. First, they rule out the possibility that LS-RNS simply overfits to semantic similarity tasks such as STS. The improvements on classification-based transfer tasks—spanning sentiment analysis, question classification, paraphrase detection, and subjective/objective classification—demonstrate that LS-RNS captures task-agnostic, semantically rich features that generalize well. Second, the consistent outperformance on both in-domain (e.g., SST) and cross-domain (e.g., TREC, MRPC) tasks highlights LS-RNS’s robustness. By explicitly modeling the semantic asymmetry of negative samples, the learned embedding space captures more fine-grained and discriminative features—beneficial across task boundaries.

In sum, LS-RNS not only advances state-of-the-art results across seven benchmark datasets but also provides strong empirical support for the general-purpose transferability of asymmetry-aware sentence representations, thereby strengthening the core thesis of this work.

4.3.3. Experiments on Image Representation

To examine whether the benefits of LS-RNS extend beyond the domain of natural language, we evaluate its effectiveness in the visual modality through image classification on the STL-10 dataset. STL-10 is a standard benchmark for evaluating representation learning methods under limited-label settings. We adopt a linear evaluation protocol, where a linear classifier is trained atop frozen visual embeddings learned by contrastive models. This setup ensures that the results reflect the intrinsic quality of the learned representations, not the capacity of downstream decoders.

We compare SimCLR and SimCLR+LS-RNS across five widely used backbone architectures of varying depth and inductive biases—VGG-16, ResNet-18, ResNet-34, ResNet-50, and Vision Transformer (ViT-b-16). The results, reported in

Table 4. Image classification results (accuracy of linear classifiers trained on learned representations).

Backbones	SimCLR	SimCLR-LS-RNS
VGG-16	68.91%	72.19% (+3.28%)
ResNet-18	79.50%	80.52% (+1.02%)
ResNet-34	79.96%	81.91% (+1.95%)
ResNet-50	81.88%	82.70% (+0.82%)
ViT-b-16	81.79%	82.81% (+1.02%)

, reveal several important findings. First, LS-RNS consistently improves top-one classification accuracy across all tested backbones. Notably, with VGG-16, LS-RNS improves accuracy from 68.91% to 72.19% (+3.28%), showing that even weaker encoders benefit significantly. With ResNet-18, accuracy improves from 79.50% to 80.52% (+1.02%). The best result overall is obtained with ViT-b-16, where the accuracy of LS-RNS reaches 82.81%, outperforming the baseline (81.79%) by +1.02%. Second, these improvements demonstrate that LS-RNS enhances the learned embedding space quality regardless of encoder type, including both convolutional and transformer-based models. This suggests that LS-RNS addresses a core limitation in contrastive learning dynamics—namely, the asymmetric treatment of negatives—rather than exploiting domain- or model-specific artifacts. Third, perhaps most strikingly, LS-RNS enables SimCLR with ResNet-18 to outperform baseline SimCLR with a deeper ResNet-34 (80.52% vs. 79.96%). This finding indicates that LS-RNS partially compensates for the capacity limitations of shallower encoders, highlighting that representation quality is a function not only of network depth but also of how contrastive objectives structure the embedding space. Finally, the fact that a method originally developed for text embeddings provides consistent improvements on visual data supports the view that LS-RNS introduces general-purpose inductive biases into contrastive learning. Specifically, its hard-negative-aware sampling and adaptive label smoothing help learn semantically structured, domain-agnostic representation spaces.

In summary, LS-RNS achieves state-of-the-art performance in frozen linear evaluation on STL-10 under SimCLR settings and delivers consistent gains across CNNs and ViTs alike. These results not only validate the cross-domain robustness of the method but also open up exciting directions for unifying contrastive learning principles across modalities.

4.4. Ablation Study

We conduct a thorough ablation study to dissect the contributions of different components within the LS-RNS framework and to investigate the impact of various design choices. For all ablation experiments, we use SimCSE-BERT_base with LS-RNS as the baseline, allowing us to isolate the effect of each modification.

4.4.1. Impact of Removing Different Loss Components

Table 5. Ablation study: removing loss components.

Models	STS (avg.)	TR (avg.)
SimCSE-LS-RNS	71.55	82.98
w/o label smoothing loss	70.92	82.81
w/o contrastive learning loss	69.28	82.09
w/o triplet loss	70.81	83.85

presents the results of systematically removing each of the three key loss components—$L_{cl}$ (the standard contrastive loss), $L_{\mathrm{rank}}$ (the ranking-based loss), and $L_{ls}$ (the label smoothing loss)—from the overall objective function (${\mathcal{L}}_{Total}$). The best overall performance is achieved when all three components are jointly used, with SimCSE-LS-RNS obtaining 71.55 on STS and 82.98 on TR tasks. This indicates a clear synergy among the three loss terms.

Notably, removing the ranking-based loss $L_{\mathrm{rank}}$ results in the most substantial drop in STS performance (to 69.28), emphasizing the critical role of ranking-based negative sampling. By explicitly modeling the relative difficulty of negative samples, $L_{\mathrm{rank}}$ encourages the model to learn finer-grained semantic distinctions, resulting in a more discriminative embedding space. Without this ranking information, the model treats all negatives as equally dissimilar, which weakens its ability to capture subtle semantic differences.

Excluding the standard contrastive loss $L_{cl}$ also leads to a noticeable decline in performance, particularly in STS, though TR remains relatively robust. This suggests that while $L_{\mathrm{rank}}$ carries most of the discriminative power, $L_{cl}$ provides a stabilizing influence by promoting similarity between positive pairs and offering a strong foundational training signal.

The removal of the label smoothing loss $L_{ls}$ leads to a moderate performance degradation. Though less severe than the removal of $L_{\mathrm{rank}}$ or $L_{cl}$, this result still highlights the value of adaptive smoothing. By modulating the influence of each negative sample based on its rank, $L_{ls}$ prevents the model from being overly influenced by easy negatives and shifts focus toward harder, more informative ones. Importantly, while $L_{\mathrm{rank}}$ determines the relative order of difficulty, $L_{ls}$ assigns a graded magnitude of influence to each negative sample, yielding a more fine-grained and effective learning signal.

Interestingly, although the removal of $L_{cl}$ results in a lower STS score, it achieves the highest TR score (83.85) when only $L_{\mathrm{rank}}$ and $L_{ls}$ are used. This suggests that in specific transfer settings, ranking and smoothing alone can generalize well, though at the cost of semantic granularity. Nevertheless, the combination of all three losses yields the most balanced and robust performance across tasks.

In conclusion, these findings validate the complementary roles of contrastive learning, ranking-based sampling, and adaptive smoothing. Together, they address different facets of the negative sampling challenge and lead to the construction of a more semantically meaningful and transferable embedding space.

4.4.2. Pooling Strategy

Table 6. Performance with different pooling strategies.

Methods	STS-B	SICK	Avg.
[CLS]	78.65	74.18	76.35
Last Layer
Mean Pooling	78.10	73.50	71.02
Attention Pooling	77.12	71.21	74.20
First and Last
Mean Pooling	78.82	73.28	77.16
Attention Pooling	69.75	67.54	68.11

investigates the impact of different pooling strategies for extracting sentence representations from the pre-trained language model. Surprisingly, the attention-based pooling mechanism, despite its theoretical appeal and widespread use in other NLP tasks, does not lead to improved performance in this unsupervised setting. In fact, it often performs worse than the simpler strategy of using the [CLS] token representation directly. This suggests that, in the absence of explicit supervision or task-specific guidance, the attention mechanism may struggle to learn meaningful attention weights. It might attend to irrelevant tokens, adding noise to the representation and hindering the learning process. The added complexity and parameters of the attention mechanism also make it more challenging to optimize effectively without labeled data.

By contrast, the simple [CLS] token representation performs competitively, especially on SICK (74.18, the best score on that dataset). This strong performance can be attributed to the fact that the [CLS] token is specifically trained during pre-training to encode sentence-level semantics, making it a robust choice for unsupervised sentence representation learning.

Among the various strategies, the best overall performance is achieved by applying mean pooling over the first and last hidden layers, which attains the highest average score (77.16) and the best STS-B result (78.82). This finding aligns with the understanding that different layers of a pre-trained language model capture different levels of linguistic information. Earlier layers tend to encode more syntactic and local information, while later layers capture more abstract semantic and global information. By combining the representations from the first and last layers, we obtain a richer and more balanced representation that incorporates both local and global aspects of sentence meaning. This simple, fixed combination of layers proves to be more effective than a learned attention mechanism in this unsupervised setting.

4.4.3. Integration Methods for Negative Ranking Information

Table 7. Performance with different negative ranking methods.

Methods	STS-B	SICK	Avg.
ChatGPT scoring	80.52	78.55	79.53
ListMLE	81.45	77.41	79.40
Our method	81.38	78.95	80.12

compares different approaches for integrating the negative ranking information into the CL framework. All methods that incorporate ranking information GPT-3.5-turbo, ListMLE, and our method) demonstrate superior performance compared to baselines that do not utilize ranking (as shown in previous tables). This further reinforces the core premise of our work: that acknowledging and modeling the relative difficulty of negative samples is crucial for learning high-quality sentence embeddings.

Using ChatGPT scoring, which involves querying GPT-3.5-turbo to directly obtain similarity scores for sentence pairs, provides a strong signal and achieves good results. This highlights the impressive semantic understanding capabilities of LLMs. As shown in Table 7, this method attains the highest score on the SICK dataset (78.55) and a strong overall average (79.53), demonstrating the impressive semantic understanding capabilities of large language models (LLMs). However, this approach has practical limitations. Relying on an external, large-scale LLM for every batch during training can be computationally expensive and time-consuming. Furthermore, the performance becomes dependent on the availability and quality of the external LLM, which may not always be guaranteed or desirable.

ListMLE, a ranking-based loss function that directly optimizes for the correct ordering of negative samples, also performs well. It achieves the highest score on STS-B (81.45). This confirms the effectiveness of directly optimizing for ranking, rather than simply using a binary classification approach (positive vs. negative).

Our proposed method, which combines ranking information with adaptive label smoothing, achieving the highest average score (80.12) and the best result on SICK (78.95), while remaining competitive on STS-B (81.38). This demonstrates the synergistic benefits of these two components. Ranking provides the “relative order” of negative samples based on their semantic similarity to the anchor, while adaptive label smoothing provides a “magnitude” of influence for each negative sample in the loss function. The adaptive smoothing mechanism, controlled by the ${\alpha }_{ij}$ parameter, allows the model to focus more on harder negatives (those with higher similarity scores and lower ranks) and less on easier negatives. This creates a more nuanced and effective learning signal than either ranking alone or standard CL with uniform negative sampling. The smoothing helps to prevent the model from becoming overly confident about easy negatives, which can lead to overfitting and hinder its ability to learn subtle semantic distinctions. Our method, by combining the strengths of ranking and adaptive smoothing, provides a more principled and effective way to address the asymmetry inherent in negative sampling. It is also more computationally efficient than relying on external LLM calls for similarity scoring.

4.5. Analysis of Sentence Embedding Space

Figure 2 provides a visual analysis of the sentence embedding space, comparing the embeddings generated by SimCSE (the base encoder) with those generated by SimCSE-LS-RNS. We randomly sample 1000 sentences from the STS-B dataset and visualize their embeddings using t-SNE (not explicitly mentioned, but implied).

The visualization clearly reveals a significant difference between the two sets of embeddings. The SimCSE embeddings exhibit the characteristic anisotropy problem, with the majority of points tightly clustered together in a relatively small region of the vector space. This clustering indicates that the embeddings are not well distributed, and many sentence pairs, even if semantically dissimilar, end up having high cosine similarity due to their proximity in this dense region. This makes it difficult to distinguish between sentences with subtle meaning differences.

In stark contrast, the SimCSE-LS-RNS embeddings are much more dispersed and occupy a significantly larger portion of the vector space. This increased dispersion demonstrates that LS-RNS has effectively mitigated the anisotropy problem and created a more semantically meaningful embedding space. The improved distribution is a direct consequence of our core contribution: addressing the asymmetry in negative sampling during CL. By prioritizing harder negatives through ranking and adaptive label smoothing, LS-RNS forces the model to learn finer-grained distinctions between sentences. This, in turn, leads to a more structured embedding space where the distances between embeddings more accurately reflect the semantic relationships between the corresponding sentences. The increased dispersion means that small changes in sentence meaning are more likely to result in noticeable changes in the embedding, making the representation more sensitive and discriminative.

This improved embedding space has significant implications for downstream tasks. A more dispersed and semantically meaningful representation makes it easier for subsequent models, such as classifiers or similarity search algorithms, to differentiate between sentences and perform their respective tasks more accurately. The visual analysis provides compelling qualitative evidence that complements the quantitative results presented earlier, further demonstrating the effectiveness of LS-RNS in learning high-quality sentence embeddings.

5. Conclusions

In this study, we introduced LS-RNS, a novel framework that explicitly addresses the overlooked semantic asymmetry in contrastive learning for unsupervised sentence embedding. Unlike traditional methods that treat all negatives equally, LS-RNS uses a powerful LLM to assign asymmetric similarity scores, enabling ranking-based sampling and adaptive label smoothing to guide the learning process. Our extensive evaluations on seven STS datasets (STS12–16, STS-B, SICK-R) reveal consistent and significant improvements. On STS tasks, LS-RNS achieves 79.87 (BERT-base) and 80.41 (RoBERTa-base), significantly outperforming SimCSE, DiffCSE, and PromptBERT; on transfer learning, LS-RNS reaches 88.82 (BERT-base) and 87.68 (RoBERTa-base), showing consistent improvements across all seven tasks; on image classification (STL-10), LS-RNS improves SimCLR’s performance across all tested backbones and even allows a ResNet-18 to outperform a deeper ResNet-34, highlighting its robustness beyond the NLP domain. These results demonstrate that modeling asymmetric relationships is crucial for robust representation learning.

It is worth noting that the integration of LLM-based similarity scoring and dynamic sampling strategies in the LS-RNS framework inevitably incurs additional computational complexity; while our current experiments are conducted on datasets of moderate scale (e.g., Birdsong, 4K training instances), scalability to larger corpora (e.g., 100K to 100M samples) remains a crucial consideration for real-world deployment. Fortunately, key components—such as LLM-based similarity scoring and ranking—can be precomputed offline, while the adaptive label smoothing mechanism remains computationally lightweight during training. This design enables efficient scaling through parallelism and distributed precomputation. Furthermore, in high-resource scenarios, the LLM module can be approximated or distilled into compact, domain-adapted models, reducing latency without compromising semantic fidelity.

Future directions include the scalability of LS-RNS across orders of magnitude of data, incorporating dynamic temperature mechanisms, expanding to multilingual and multimodal embeddings, and conducting a theoretical analysis of asymmetric contrastive loss.