SciRep: A Ranking-Aware Representation Model for Scientific Text

Abstract

Existing scientific text representation methods based on contrastive learning typically adopt a binary classification paradigm of positive and negative samples, which struggles to capture the complex, hierarchical semantic similarity relationships inherent in scientific texts. To address this, we propose SciRep, a novel two-stage ranking distillation framework. In the first stage, we distill knowledge from a large language model to a medium-scale representation model using generated ranking samples; in the second stage, a multi-teacher strategy further transfers fine-grained ranking capability to a lightweight model. Evaluated on a scientific literature semantic embedding benchmark comprising three tasks, SciRep outperforms the strongest baseline by 11.3% in terms of Average Rank and also achieves the highest Mean Reciprocal Rank scores across all three tasks. These results demonstrate that the proposed ranking-aware distillation mechanism significantly enhances scientific text representation quality while maintaining efficient inference, offering a more effective contrastive learning method for domain-specific retrieval tasks.

1. Introduction

The exponential growth of the scientific literature has created an urgent demand for efficient information retrieval and knowledge discovery. In scientific practice, obtaining high-quality representations of scientific texts is a crucial foundation for tasks such as literature retrieval, recommendation systems, and knowledge organization. Various deep learning methods, including CNNs [1], RNNs [2], and Transformers [3], have been applied to text representation, with pre-trained language models (PLMs) like BERT [4] and RoBERTa [5] becoming the mainstream approach. However, the anisotropy issue [6] inherent in PLMs limits representation quality—meaning that sentence embeddings are confined to a narrow conical region in the vector space rather than spreading uniformly, which makes semantically distinct sentences appear artificially similar and reduces discriminative power. To address this, contrastive learning [7] has been introduced and has become the dominant optimization strategy for general text representation.

Nevertheless, existing representation methods still fail to fully meet the customized needs of scientific texts. Recent years have witnessed the emergence of general-purpose embedding models such as Jina-Embeddings [8,9] and NvEmbed-V2 [10], which achieve remarkable performance through contrastive learning on large-scale corpora. However, these models are not specifically designed for scientific texts. Scientific texts often contain a large number of technical terms and complex expressions [11], presenting significant challenges to semantic understanding and requiring more refined semantic differentiation capabilities. Meanwhile, approaches such as SemCSE [12] have been optimized specifically for scientific text representation, yet they still adopt the conventional contrastive learning paradigm that constructs only positive and negative samples, pulling the anchor toward the positive and pushing it away from the negative. However, this simple binary classification fails to capture fine-grained semantic differences [13], a limitation that is especially pronounced in scientific text scenarios where semantic similarity spans a broad spectrum beyond a simple positive–negative dichotomy.

To address this gap, this paper focuses on enhancing the fine-grained semantic differentiation ability of representation models for academic texts. In recent years, large language models (LLMs) have demonstrated outstanding performance in semantic similarity calculation due to their powerful contextual understanding. However, directly using LLMs for academic text retrieval faces an efficiency bottleneck, as they require pairwise similarity calculations that lead to quadratic time complexity. Therefore, we explore distilling the semantic differentiation capabilities of LLMs into lightweight representation models, aiming to maintain high performance while improving inference efficiency.

To address the challenges mentioned above, we propose a ranking-aware representation model for scientific text (SciRep), which leverages a two-stage ranking distillation method to enable lightweight encoders to learn fine-grained similarity ranking relationships between sentences. Specifically, we introduce a ranking distillation loss function based on traditional contrastive learning framework, transferring the fine-grained semantic discrimination capabilities of LLMs to lightweight representation models. The process is divided into two stages: (1) Coarse-grained Distillation Stage. We first leverage an LLM to automatically generate multiple text samples with varying degrees of semantic similarity to a given anchor sentence, thereby constructing a ranked sample set with multi-level semantic differences. We then transfer the LLM’s ranking-aware signals into a medium-sized representation model (e.g., BERT-large) by optimizing a novel ranking-aware distillation loss, thereby enhancing the model’s capability to capture fine-grained semantic hierarchies. (2) Fine-grained Distillation Stage. Using a multi-teacher distillation strategy, we combine the similarity score distributions generated by the medium-sized representation model for sentences in a batch with the ranking samples provided by the LLM to further distill fine-grained ranking ability into a smaller lightweight model (e.g., BERT-base). This method significantly improves inference efficiency while maintaining high-precision semantic representation performance.

We conduct experiments on the benchmark proposed by Brinner et al. [12], which includes multiple tasks for scientific literature semantic embeddings. The results demonstrate that the proposed method achieves significant improvements over existing baseline models, validating the effectiveness of the ranking-aware distillation mechanism in enhancing the quality of scientific text representations.

2. Related Work

2.1. Text Representation in General Domain

Text representation learning aims to map natural language into dense vectors to support semantic understanding and retrieval tasks. Early methods relied on bag-of-words models or static word embeddings such as Word2Vec [14] and GloVe [15], which capture lexical co-occurrence information but struggle to effectively represent contextual semantics and sentence structure. With the rise in Transformer architecture, pre-trained language models like BERT and RoBERTa learn context-aware word embeddings through self-supervised tasks such as masked language modeling on large corpora, significantly enhancing semantic understanding. However, research indicates that sentence representations directly obtained from pre-trained models (e.g., [CLS] vectors or word vector averaging) exhibit anisotropy, where the vectors cluster in narrow conical regions in space, limiting their representational capacity and discriminative power. To improve sentence-level representations, supervised methods such as Sentence-BERT [16] fine-tune on labeled data like natural language inference, yielding task-specific sentence embeddings. However, these methods rely on large amounts of labeled data, which limits their scalability.

Recently, unsupervised contrastive learning has become the mainstream paradigm in sentence representation learning. The core idea is to construct positive and negative sample pairs, bringing semantically similar sentences closer together and pushing semantically dissimilar sentences apart, thereby learning discriminative representations on unlabeled data. The key to contrastive learning methods lies in the strategy for constructing positive and negative samples. Early methods generate positive samples through data augmentation techniques such as back-translation and word order rearrangement. SimCSE [7] introduces a simple yet effective strategy: it performs two forward passes of the same sentence with different dropout masks, using these as positive sample pairs and combining them with within-batch negative samples for contrastive learning. This method achieves significant improvements in several semantic similarity tasks and becomes the baseline for subsequent research. Later, researchers improve the contrastive learning framework from multiple perspectives. For example, DiffCSE [17] introduces differential contrastive learning to enhance the model’s sensitivity to semantic changes. ArcCSE [18] improves the discriminative power of sentence pairs through angular space modeling. EASE [19] generates positive samples by matching sentences with corresponding relevant entities. DCLR [20] focuses on the impact of negative sample quality on learning. Additionally, SynCSE [21] tries to generate positive and negative samples using large language models to increase the quantity and richness of the training corpus.

Despite these advances, most contrastive learning methods classify samples as either “positive” or “negative” and fail to capture fine-grained ranking information, such as distinguishing between highly similar, moderately similar, and dissimilar sentences. This limitation is particularly important in practical scenarios such as retrieval and recommendation. RankCSE [13] builds on this by introducing ranking consistency and ranking distillation mechanisms, learning sentence representations that preserve semantic ranking relationships through list-level ranking objectives, further enhancing performance. However, RankCSE is designed for general-domain sentences and does not address the unique characteristics of academic texts, such as dense terminology, rigorous logical structures, and cross-disciplinary semantic relationships. Moreover, general-domain models are rarely evaluated on scientific benchmarks, making their transferability to academic text representation unsubstantiated.

2.2. Scientific Document Representation

To address the structured nature and dense terminology of academic texts, researchers have developed various domain-specific representation methods. Early work primarily relied on word frequency statistics and topic models, which, while interpretable, had limited representational capacity. With the rise in pre-trained language models, supervised learning methods based on citation networks have gradually become the mainstream. These methods treat citation relationships between papers as weak supervision signals for semantic similarity and use contrastive learning to bring papers with citation relationships closer in the vector space. Representation models such as SciNCL [22] and SPECTER2 [23] have achieved notable success in academic document retrieval and recommendation tasks.

However, citation relationships do not serve as reliable semantic indicators. In practice, citations may arise from various motivations—such as acknowledging foundational work, methodological comparison, or scholarly convention—rather than reflecting true textual similarity. Consequently, relevant but uncited studies often remain disconnected in the representation space. This noisy supervision limits the model’s capacity to capture nuanced semantics, especially in interdisciplinary or emerging fields. To build high-quality semantic representations, recent research trends have shifted away from reliance on citation networks and instead leverage large language models (LLMs) to directly understand and generate semantic information related to academic texts as supervision signals. SemCSE [12] exemplifies this approach by leveraging LLMs to generate semantic paraphrases of academic abstracts, which are then used as positive pairs in contrastive learning. This design encourages the model to focus on intrinsic semantic content rather than superficial citation patterns. Nevertheless, SemCSE still treats semantic similarity as a binary concept (positive vs. negative), failing to capture the continuous gradient of similarity between sentences.

In summary, existing scientific text representation methods suffer from three limitations: (1) they rely on binary similarity supervision that ignores fine-grained ranking distinctions; (2) citation-based methods inherit noisy signals from non-semantic citation motives; and (3) LLMs offer strong semantic understanding but are computationally prohibitive for large-scale retrieval. To address these gaps, we propose SciRep, a ranking-aware representation model with a two-stage distillation framework. SciRep learns hierarchical ranking relationships (similar > weakly similar > unrelated) from LLM-generated samples, avoids citation noise by using semantic-only supervision, and transfers fine-grained ranking knowledge to a lightweight SciBERT encoder via multi-teacher distillation. To our knowledge, this is the first approach that jointly tackles binary supervision, citation noise, and efficiency–performance trade-offs in scientific text representation.

3. Methodology

This study is experimental in nature, aimed at evaluating the effectiveness of the proposed ranking distillation framework through quantitative benchmarking on scientific text representation tasks. Our proposed approach SciRep consists of two stages: coarse-grained distillation and fine-grained distillation. In Stage 1, the LLM acts as the teacher to distill hard ranking knowledge into a medium-scale representation model. In Stage 2, the text similarity distribution from the medium-scale representation model (as soft labels) and hard ranking signals from the LLM jointly guide the training of a lightweight representation model, achieving a balance between inference efficiency and performance. The two-stage training process will be discussed in this section.

3.1. Stage 1: Coarse-Grained Distillation (LLM → Medium-Scale Model)

As illustrated in Figure 1, in the first stage, we employ Llama-3-8B [24] as the teacher model to transfer its semantic ranking capability to a medium-scale representation model (e.g., BERT-large). The training data is sampled from the SciRepEval benchmark [25], comprising $N=\mathrm{350,000}$ title–abstract pairs, denoted as $\mathcal{D}={\left\{(T_i,A_i)\right\}}_{i=1}^N$. To construct hierarchical semantic supervision signals, we prompt Llama-3-8B to generate three semantically differentiated variants for each original abstract $A_i$, a highly similar abstract $A_i^{+}$, a weakly similar abstract $A_i^{w+}$, and an unrelated abstract $A_i^{-}$, with the specific prompts used detailed in

Table 1. Prompt for generating abstract variants with hierarchical semantics.

Task Definition
You are a research assistant specialized in generating scientifically rigorous text variants. Given an original scientific abstract, produce three distinct versions with the following specified levels of semantic similarity.
(a) High Semantic Consistency Generate a complete academic abstract that preserves the original research question, methodology, results, and conclusions exactly. Only paraphrase the wording, adjust sentence structures, or change syntactic ordering.
(b) Weak Similarity Generate a complete academic abstract within the same research domain, but alter at least one of the following: the specific research method, the detailed problem formulation, or the direction of conclusions. The topic should remain related but the content must differ meaningfully.
(c) Unrelated Version Generate a complete academic abstract from a completely different research domain (e.g., if the original is from computer science, generate an abstract from materials science, biomedical engineering, or ecology). There should be no topical overlap with the original.
Output Format Requirements - Each version must be a complete and formally structured academic abstract. - Maintain academic rigor and structural integrity in all outputs. - Output the three abstracts sequentially in the order specified (High Semantic Consistency, Weak Similarity, Unrelated Version).

. To ensure the semantic consistency of the generated samples, we introduce a secondary validation mechanism: Llama-3-8B is prompted to re-evaluate its own generated samples. If the secondary judgment contradicts the intended category (e.g., a sample intended as “weakly similar” is judged as “unrelated”), the sample is labeled as a hard negative and incorporated into the training negative set to increase discriminative difficulty. From each original pair, we derive two training instances, a title-anchored instance $(T_i,A_i^{+},A_i^{w+},A_i^{-})$ and an abstract-anchored instance $(A_i,A_i^{+},A_i^{w+},A_i^{-})$, thereby enabling the simultaneous learning of hierarchical semantics across both symmetric task (title–abstract) and asymmetric task (abstract–abstract) text pairs.

During training, we design a hierarchical contrastive ranking loss that combines contrastive learning and ranking distillation signals. Let a training batch be ${\left\{(q_i,p_i^{+},p_i^{w+},p_i^{-})\right\}}_{i=1}^n$, where $q_i$ denotes the anchor text (either the title or the original abstract), and $p_i^{+},p_i^{w+},p_i^{-}$ represent the positive, weak-positive, and negative texts, respectively. The encoder $f_{\theta }$ (BERT-large) maps each text to a normalized embedding vector: ${\mathbf{q}}_i=f_{\theta }\left(q_i\right)$, ${\mathbf{p}}_i^{+}=f_{\theta }\left(p_i^{+}\right)$, ${\mathbf{p}}_i^{w+}=f_{\theta }\left(p_i^{w+}\right)$, ${\mathbf{p}}_i^{-}=f_{\theta }\left(p_i^{-}\right)$. Similarity is measured by cosine similarity $s(\mathbf{x},\mathbf{y})={\mathbf{x}}^{\top }\mathbf{y}/(\parallel \mathbf{x}\parallel \parallel \mathbf{y}\parallel )$. The contrastive learning loss ${\mathcal{L}}_{\mathrm{contrast}}$ follows the InfoNCE formulation:

$${\mathcal{L}}_{\mathrm{contrast}}=-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}log{\displaystyle \frac{e^{s({\mathbf{q}}_i,{\mathbf{p}}_i^{+})/\tau }}{e^{s({\mathbf{q}}_i,{\mathbf{p}}_i^{+})/\tau }+{\sum }_{k\in {\mathcal{N}}_i}{e}^{s({\mathbf{q}}_i,\mathbf{k})/\tau }}},$$

(1)

where the negative set ${\mathcal{N}}_i$ includes the generated negative sample ${\mathbf{p}}_i^{-}$, hard negative samples ${\mathbf{n}}_i^{\mathrm{hard}}$, positive samples from other anchors ${\left\{{\mathbf{p}}_j^{+}\right\}}_{j\ne i}$, and embeddings of other anchors ${\left\{{\mathbf{q}}_j\right\}}_{j\ne i}$. The ranking distillation loss ${\mathcal{L}}_{\mathrm{rank}}^{\mathrm{LLM}}$ adopts the ListMLE [26] formulation to enforce the similarity ordering “similar > weakly similar > unrelated”:

$${\mathcal{L}}_{\mathrm{rank}}^{\mathrm{LLM}}=-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}log\left({\displaystyle \frac{e^{s({\mathbf{q}}_i,{\mathbf{p}}_i^{+})/\tau }}{e^{s({\mathbf{q}}_i,{\mathbf{p}}_i^{+})/\tau }+Z_i}}·{\displaystyle \frac{e^{s({\mathbf{q}}_i,{\mathbf{p}}_i^{w+})/\tau }}{Z_i}}\right),$$

(2)

with

$$Z_i=e^{s({\mathbf{q}}_i,{\mathbf{p}}_i^{-})/\tau }+\sum _{j\ne i}\left[e^{s({\mathbf{q}}_i,{\mathbf{q}}_j)/\tau }+e^{s({\mathbf{q}}_i,{\mathbf{p}}_j^{+})/\tau }+e^{s({\mathbf{q}}_i,{\mathbf{p}}_j^{w+})/\tau }\right].$$

(3)

The overall training objective is as follows:

$${\mathcal{L}}_{\mathrm{stage}1}={\mathcal{L}}_{\mathrm{contrast}}+{\mathcal{L}}_{\mathrm{rank}}^{\mathrm{LLM}},$$

(4)

with temperature parameter $\tau =0.07$. This joint objective enables the BERT-large model to learn hierarchical semantic ranking capability from the LLM-generated ranking samples, obtaining preliminary semantic discrimination and ranking representation ability.

3.2. Stage 2: Fine-Grained Distillation (LLM and Medium-Scale Model → Lightweight Model)

After completing the first stage of training, we obtain a medium-scale model (BERT-large) with hierarchical ranking capability as the teacher model $f_T(·)$. The goal of the second stage is to further transfer the fine-grained ranking knowledge from this teacher model to a more parameter-efficient lightweight student model (e.g., BERT-base) $f_S(·)$. As illustrated in Figure 2, this stage employs a multi-teacher distillation strategy, utilizing both the ranking samples generated by the LLM from the first stage and the similarity distribution from the teacher model as supervision signals.

The rationale for adopting a multi-teacher strategy is twofold. First, the LLM-generated ranking samples provide hard ranking constraints (i.e., the explicit ordering “similar > weakly similar > unrelated”), which enforce a coarse-grained discriminative boundary. Second, the intermediate teacher model (BERT-large) provides soft similarity distributions over in-batch samples, capturing fine-grained relational structures that are not fully expressed by hard labels alone.

The training loss in the second stage consists of three components. To maintain the basic semantic discriminative ability of the student model, we retain the contrastive learning loss ${\mathcal{L}}_{\mathrm{contrast}}$ from Stage 1 (as shown in Equation (1)), but its computation is based on the student model $f_S$. Simultaneously, we continue to use the three-level ranking samples generated by the LLM in Stage 1, employing the same ListMLE loss ${\mathcal{L}}_{\mathrm{rank}}^{\mathrm{LLM}}$ as in Stage 1 (as shown in Equation (2)), forcing the student model to learn the “similar > weakly similar > unrelated” ranking relationship.

To transfer the fine-grained similarity knowledge from the teacher model to the student model, we introduce a teacher–student similarity distribution alignment loss. For a training batch of size N, for each anchor text $q_i$, we compute its similarity scores with all samples in the batch (including itself). The similarity score vector computed by the teacher model $f_T$ (BERT-large) for anchor $q_i$ is defined as follows:

$${\mathbf{S}}_i^T=\left[s_{i1}^T,s_{i2}^T,\dots ,s_{iN}^T\right],$$

(5)

where $s_{ij}^T={\displaystyle \frac{f_T{\left(q_i\right)}^{\top }{f}_T\left(q_j\right)}{\parallel f_T\left(q_i\right)\parallel \parallel f_T\left(q_j\right)\parallel }}$ denotes the cosine similarity between anchor $q_i$ and the j-th sample $q_j$ in the batch. The similarity score vector computed by the student model $f_S$ (BERT-base) for the same anchor is as follows:

$${\mathbf{S}}_i^S=\left[s_{i1}^S,s_{i2}^S,\dots ,s_{iN}^S\right],$$

(6)

where $s_{ij}^S={\displaystyle \frac{f_S{\left(q_i\right)}^{\top }{f}_S\left(q_j\right)}{\parallel f_S\left(q_i\right)\parallel \parallel f_S\left(q_j\right)\parallel }}$.

To compare the similarity distributions of the two models, we convert the similarity score vectors into probability distributions. Using temperature parameter ${\tau }_t$, we apply softmax normalization to the teacher similarity scores to obtain the teacher distribution $P_i^T=\sigma ({\mathbf{S}}_i^T/{\tau }_t)$, where $\sigma$ denotes the softmax function. Similarly, using temperature parameter ${\tau }_s$, we normalize the student similarity scores to obtain the student distribution $P_i^S=\sigma ({\mathbf{S}}_i^S/{\tau }_s)$. We use Jensen–Shannon (JS) divergence to measure the difference between the teacher and student distributions. JS divergence has good symmetry and boundedness (ranging from 0 to 1), making it suitable for aligning two probability distributions. The loss function minimizing this difference is as follows:

$${\mathcal{L}}_{\mathrm{align}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\mathrm{JS}(P_i^T\parallel P_i^S),$$

(7)

where JS divergence is defined as $\mathrm{JS}(P\parallel Q)={\displaystyle \frac{1}{2}}[\mathrm{KL}(P\parallel M)+\mathrm{KL}(Q\parallel M)]$, $M={\displaystyle \frac{P+Q}{2}}$, and KL divergence $\mathrm{KL}(P\parallel Q)={\sum }_{j=1}^N{P}_{j}log(P_j/Q_j)$. In our experiments, we set the temperature parameters ${\tau }_t={\tau }_s=0.07$, ensuring both models use the same temperature scaling.

The final overall loss for the second stage is the sum of the three components:

$${\mathcal{L}}_{\mathrm{stage}2}={\mathcal{L}}_{\mathrm{contrast}}+{\mathcal{L}}_{\mathrm{rank}}^{\mathrm{LLM}}+{\mathcal{L}}_{\mathrm{align}}.$$

(8)

This strategy combines three-level hard ranking constraints with teacher soft distribution alignment, fully leveraging the hierarchical ranking samples generated by the LLM and the fine-grained similarity knowledge from the teacher model to achieve effective knowledge transfer from the medium-scale teacher model to the lightweight student model.

4. Experimental Settings

4.1. Evaluation Benchmark

We use the scientific text semantic evaluation benchmark proposed by Brinner et al. [12] This benchmark is designed to assess whether embedding models capture the semantic content of scientific texts, rather than relying on citation signals. Unlike citation-based evaluations such as SciRepEval, it focuses on a model’s understanding of the actual meaning of scientific text, instead of potentially noisy cues such as citation networks, author identity, or citation counts. The benchmark includes three tasks:

• Title–Abstract Matching: This task evaluates whether a model maps the title and abstract of the same paper to nearby regions in the embedding space.
• Abstract Segmentation Consistency: This task evaluates whether a model recognizes the semantic consistency between two segments of the same abstract, such as the first half and the second half.
• Query Matching: This task evaluates whether a model matches a user query to relevant papers.

4.2. Baselines

To evaluate our approach, we compare the proposed model with several state-of-the-art text representation models. Specifically, we consider the following specialized models for scientific documents:

• SciBERT: A Transformer-based model pre-trained on a large corpus of scientific text, following the BERT-base architecture.
• SPECTER2: A document-level embedding model built upon SciBERT, pre-trained on the Semantic Scholar corpus using a citation-informed triplet loss to generate general-purpose vector representations of scientific papers.
• SciNCL: A document-level embedding model also based on SciBERT, which employs neighborhood contrastive learning guided by citation graph embeddings to produce semantically aware representations of scientific papers.
• SemCSE: An unsupervised sentence embedding model based on SciDeBERTa, which leverages LLM-generated summaries of scientific abstracts to construct semantically related anchor-positive pairs.

We also make comparisons against several advanced general-purpose representation models, such as SimCSE, SSCL [27], RankCSE, jina-embeddings-v2 [8], jina-embeddings-v3 [9], and NvEmbed-V2 [10].

4.3. Implementation Details

Given the widespread use of SciBERT in scientific text representation, we use it as the backbone network. Since SciBERT is a continuation of BERT-base training and does not have a corresponding BERT-large version, we use BERT-large as the student model in the first stage and one of the teacher models in the second stage. During model training, we employ the Adam optimizer and set the input batch size to 64. To obtain sentence representations, we perform average pooling on the embeddings of all tokens and directly extract the vector of the [CLS] token. The experiments are conducted on an NVIDIA A800 GPU with 80 GB of memory. During training, validation is performed every 1000 batches. We implement early stopping: training is terminated if the validation score does not improve for 15 consecutive epochs. The distance metric between vectors is cosine similarity. To control computational costs, we use the relatively smaller parameter language model Llama-3-8B to generate hierarchical semantic samples. The generation uses temperature $T=0.7$, top-$p=0.95$, and a maximum output length of 600 tokens.

4.4. Evaluation Metrics

To evaluate the performance of models on scientific text representation, we employ average rank as the main metric. Given a query sample (e.g., a title), the model retrieves candidates from a pool of M items (where $M=6000$ in our experiments) based on the cosine similarity between their embeddings and ranks the candidates in descending order of similarity. Since each query corresponds to exactly one ground-truth positive candidate in our benchmark, average rank directly reflects how well the model ranks this sole relevant item at the top of the candidate list. Formally, average rank is defined as the mean position of the correct match across all queries. The optimal value is 1 (indicating that the correct match is always ranked first), and the worst possible value is M.

In parallel, we adopt the Mean Reciprocal Rank (MRR) as a complementary metric. While Average Rank provides an intuitive global assessment by directly reporting the mean position of the correct match, it is sensitive to outlier rankings—a single query with an excessively large rank can disproportionately skew the mean. The MRR mitigates this limitation by focusing on the ranking quality at the top positions, which is particularly critical for real-world retrieval scenarios where users typically only examine the first few results. Formally, the MRR is defined as the average of the reciprocal ranks of the correct matches across all queries: $\mathrm{MRR}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\displaystyle \frac{1}{{\mathrm{rank}}_i}}$, where ${\mathrm{rank}}_i$ denotes the position of the correct candidate for the i-th query. A higher MRR indicates better performance, with the optimal value being 1 when the correct match is always ranked first. By reporting both Average Rank and MRR, we provide a comprehensive evaluation that captures both overall ranking quality and Top-K retrieval utility.

5. Main Results

5.1. Overall Performance Analysis

As shown in

Table 2. The experimental comparison results of Average Rank. Lower scores (closer to 1) indicate better model performance. The Average Score is the mean of the three preceding metrics. The best scores for each metric are highlighted in bold.

Method	Title-Abstr.	Abstr.-Segments	Query	Average Score
SciBERT	807.74	214.37	213.45	411.85
SPECTER2	10.25	12.23	2.18	8.22
SciNCL	5.68	7.35	2.29	5.11
SemCSE	2.47	2.68	1.23	2.13
SimCSE	24.58	43.91	8.78	25.76
SSCL	23.19	44.02	6.82	24.68
RankCSE	21.37	40.42	5.61	22.47
jina(emb.)-v2	3.29	8.77	1.29	4.45
jina(emb.)-v3	3.45	6.96	1.01	3.81
NvEmbed-V2	3.38	3.84	1.02	2.75
SciRep (Ours)	2.10	2.23	1.33	1.89

, our proposed SciRep model achieves an average score of 1.89, which is the best among all methods. Compared to the second best semantic contrastive learning method SemCSE, SciRep improves overall performance by approximately 11.3%. When compared to the general-purpose retrieval model NvEmbed-V2 (2.75), the improvement reaches 31.3%. These results indicate that our proposed two-stage ranking distillation framework effectively enhances model performance on scientific text understanding tasks. Additionally, the experiment reveals another notable trend: models that have been specifically trained or utilize domain knowledge on scientific texts (e.g., SPECTER2, SciNCL, SemCSE) generally outperform general-purpose contrastive learning methods not optimized for the scientific domain (e.g., SimCSE, SSCL, RankCSE). This demonstrates that domain adaptation for scientific texts is necessary and effective. Furthermore, although general-purpose retrieval models such as jina and NvEmbed-V2 are not fine-tuned specifically on scientific datasets, their pre-training corpora are extensive and cover a wide range of topics, including a significant proportion of scientific texts. Nevertheless, SciRep still significantly outperforms these models, indicating that our proposed two-stage knowledge distillation method can effectively enhance the semantic representation quality of the model, even when trained on a limited amount of scientific data. The MRR results presented in

Table 3. The experimental comparison results of Mean Reciprocal Rank (MRR). Higher scores (closer to 1) indicate better model performance. The Average Score is the mean of the three preceding metrics. The best scores for each metric are highlighted in bold.

Method	Title-Abstr.	Abstr.-Segments	Query	Average Score
SciBERT	0.33	0.67	0.22	0.41
SPECTER2	0.76	0.80	0.93	0.83
SciNCL	0.82	0.83	0.86	0.84
SemCSE	0.94	0.91	0.97	0.94
SimCSE	0.67	0.60	0.67	0.65
SSCL	0.69	0.65	0.68	0.67
RankCSE	0.72	0.68	0.73	0.71
jina(emb.)-v2	0.90	0.73	0.92	0.85
jina(emb.)-v3	0.92	0.85	0.94	0.91
NvEmbed-V2	0.94	0.88	0.95	0.92
SciRep (Ours)	0.96	0.93	0.96	0.95

corroborate the findings from Average Rank. SciRep achieves the highest average MRR of 0.95, outperforming SemCSE (0.94), NvEmbed-V2 (0.92), and jina-v3 (0.91). The consistency between Average Rank and MRR rankings confirms that our conclusions are robust and not biased by outlier-sensitive metrics.

5.2. Task-Specific Performance Analysis

The performance on various sub-tasks further reveals the capabilities of the model:

(1)

Title–Abstract Matching: SciRep achieves the best score of 2.10, outperforming SemCSE (2.47) by approximately 15.0%. This indicates that SciRep effectively captures semantic correspondences between different levels of granularity (title and abstract) of the same research work. Some general retrieval models (e.g., jina-v3 with a score of 3.45) also perform well on this task, reflecting its similarity to general document retrieval. Overall, the leading performance of SciRep demonstrates its precision in semantic alignment within the scientific domain. This finding is further supported by the MRR results in Table 3, where SciRep achieves the highest score of 0.96, outperforming SemCSE (0.94) and NvEmbed-V2 (0.94).

(2)

Abstract Segmentation Consistency: SciRep significantly outperforms with a score of 2.23, improving by about 16.8% over SemCSE (2.68) and by about 41.9% over NvEmbed-V2 (3.84). This task requires the model to infer overall meaning from incomplete text segments, demanding deep semantic understanding. The outstanding performance of SciRep on this task validates that the distillation method effectively captures complex semantic relationships between segments. The MRR results reinforce this conclusion, with SciRep achieving a score of 0.93, substantially higher than those of SemCSE (0.91) and NvEmbed-V2 (0.88).

(3)

Query Matching: The retrieval-optimized model jina-v3 achieves the best result (1.01), demonstrating its strong capability in query–document matching. Although SciRep does not achieve the top score on this task, its result of 1.33 remains competitive. This may reflect differences in how tasks are sensitive to data distribution and training objectives: SciRep’s training data primarily consists of titles and abstracts, whereas the queries and papers in the query matching task differ in distribution from the training data. As a result, the performance gap highlights the advantage held by general-purpose retrieval models that benefit from pre-training on large-scale, diverse datasets. Nevertheless, SciRep’s competitive performance further demonstrates the effectiveness of our proposed method in learning semantic representations for scientific texts from limited domain-specific data.

6. Ablation Study

To evaluate the contributions of each component in the proposed method, we conduct comprehensive ablation studies. Our two-stage distillation framework consists of three core modules: (1) Base Contrastive Learning module (BaseCL), which follows the supervised training paradigm of SimCSE and fine-tunes the model using triplets constructed from LLM-generated positive and negative samples; (2) LLM-based Ranking Distillation module (Rank-LLM), which transfers ranking knowledge through hierarchical semantic samples generated by the LLM, combined with the ListMLE loss function; and (3) Similarity Distribution Alignment module (SDA), which aligns the similarity score distributions between the teacher model (BERT-large) and the student model (SciBERT-base) using Jensen–Shannon divergence. We analyze the effectiveness of each component by progressively adding modules to assess their incremental impact, with the experimental results shown in

Table 4. An ablation study on the SciRep method.

Method	Title-Abstr.	Abstr.-Segments	Query	Average Score
SciBERT	807.74	214.37	213.45	411.85
+BaseCL	5.29	5.65	3.12	4.69
+Rank-LLM	2.35	2.54	1.37	2.09
+SDA	2.42	2.59	1.39	2.13
SciRep (Ours)	2.10	2.23	1.33	1.89

. Notably, we evaluate the individual contributions of Rank-LLM and SDA by incorporating each module separately on top of the BaseCL foundation. By leveraging BaseCL, we establish a stable semantic baseline, ensuring that any observed improvements from Rank-LLM or SDA are directly attributed to their respective ranking or distribution alignment mechanisms, rather than compensating for deficiencies in the basic representation capabilities.

Building upon SciBERT, we first observe that after incorporating the BaseCL module, the average score significantly decreases to 4.69, indicating that the basic supervised contrastive learning mechanism substantially improves performance. This result verifies the effectiveness of using LLM-generated samples for supervised contrastive learning, especially for the “Title–Abstract” matching task, where the improvement is the most pronounced. Upon further adding the Rank-LLM module on top of BaseCL, the average score continues to drop to 2.09, yielding additional performance gains compared to using BaseCL alone. This demonstrates that the hierarchical ranking knowledge provided by the LLM effectively enhances the model’s semantic discriminability; notably, in the “Abstract-Segments” consistency task (score reduced from 5.65 to 2.54), this module proves to be particularly valuable for recognizing internal semantic structures within texts. Subsequently, introducing the SDA module alone leads to an average score of 2.13, which, although still better than using only BaseCL, shows a smaller improvement compared to the Rank-LLM module. This suggests that while similarity distillation based on distribution alignment contributes positively, its impact is relatively limited. Finally, the complete model SciRep, integrating all three modules, achieves the best performance with an average score of 1.89, significantly outperforming any single- or dual-module combination. This confirms the strong complementarity among the three modules: BaseCL establishes the semantic foundation, Rank-LLM introduces hard ranking signals, and SDA facilitates fine-grained knowledge transfer via distribution alignment. The synergistic collaboration of three modules collectively enhances the model’s comprehensive capability across scientific text representation tasks.

The discrete ranking labels from the LLM (hard signals) and the continuous similarity distributions from the teacher model (soft signals) are not guaranteed to be perfectly aligned. For instance, a sample labeled as weakly similar by the LLM may receive a relatively high similarity score from the teacher model. Nevertheless, such inconsistencies are mitigated by two factors in our framework. First, the Stage 1 distillation pre-trains the teacher model to approximate the LLM’s ranking preference, thereby reducing systematic disagreement. Second, the joint objective ${\mathcal{L}}_{\mathrm{stage}2}$ does not enforce strict consistency between the two signals; instead, the student model learns to balance them via gradient optimization. As evidenced by the ablation study in Table 3, the full model achieves better performance than variants relying solely on either hard or soft supervision, suggesting that the two signals are generally complementary.

6.1. Analysis

6.1.1. Evaluation of Sample Synthesis Quality

In the synthesizing phase of the samples, we use the Llama-3-8B language model to generate semantically relevant candidate samples for each anchor sentence, primarily based on its task adaptability and data stability. The scientific abstract generation task is characterized by high conciseness and structure, and Llama-3-8B is capable of maintaining semantic consistency with the original text in a continuation-style generation while also avoiding the high cost issues associated with larger models. Additionally, the results show that the biases in its generation (e.g., subtle factual differences) are systematic and do not disrupt the semantic coherence of the training sample pairs. To evaluate generation quality, we randomly selected 15% of the samples for manual assessment. The samples were rated on fluency (grammar and naturalness), relevance (semantic matching), and diversity (content novelty) on a scale from 1 to 5. The final average score was 4.52, with a Kappa consistency coefficient of 0.73, demonstrating that the synthetic data is both of high quality and reliable, fully meeting the requirements for contrastive learning training.

6.1.2. Effectiveness of Different Ranking Loss Functions

To explore the effectiveness of different ranking loss functions in knowledge distillation, we compare the ListMLE loss function used in this paper with the widely used ListNet [28] in information retrieval. Although both are list-level ranking losses, they differ in optimization goals and modeling approaches: ListMLE performs end-to-end modeling by maximizing the likelihood of the complete ranked list, while ListNet focuses on learning the top-one probability distribution in the ranking. The experimental results show that ListMLE slightly outperforms ListNet in terms of average score, though the difference is not significant. Specifically, on key tasks such as title–abstract matching and query matching, ListMLE achieves an average performance improvement of 1.2% to 2.8% over ListNet. This result suggests that the effectiveness of list-level ranking supervision itself may be more important than the choice of a specific loss function. Whether through likelihood modeling or top-one probability optimization, as long as the complete ranking information provided by LLMs is effectively utilized, the model’s semantic discrimination ability can be significantly improved.

It is noteworthy that both list-level loss functions significantly outperform traditional binary contrastive loss. This validates the fundamental advantage of ranking supervision in knowledge distillation: ranking learning establishes a flexible, definable semantic relationship structure, rather than a fixed binary similarity relationship. This characteristic allows researchers in different domains to design corresponding ranking criteria based on their specific research needs. For example, LLMs can be instructed to rank samples based on the relevance to the research question or according to the similarity of methodologies. By designing different prompts to guide LLMs in generating ranked samples that meet specific needs, effective domain-customized semantic knowledge transfer to lightweight models can be achieved.

6.1.3. Analysis of Computational Complexity

The ranking distillation method proposed in this paper involves different design trade-offs in terms of time complexity compared to unsupervised contrastive learning methods like SimCSE. SimCSE constructs positive sample pairs through the forward propagation of a single model and dropout noise, making the training process efficient and requiring no external supervised data. In contrast, the method in this paper introduces additional computational overhead, mainly due to the preprocessing step of generating hierarchical ranking samples with LLMs and the extra training epochs in the two-stage distillation.

However, these costs are manageable and yield clear benefits in engineering practice. First, generating samples with LLMs is a fully offline preprocessing step that only needs to be performed once when constructing the training set, and the generated dataset can be reused permanently. Second, the first-stage distillation to a medium-sized representation model is also a one-time training process, and the resulting model can serve as a fixed teacher model for subsequent distillation. Therefore, the ongoing time cost mainly comes from the second-stage distillation to the lightweight model, which takes about 3 times the training duration of SimCSE due to the need to integrate dual-teacher ranking knowledge.

The key advantage lies in the inference phase: once training is completed, the lightweight student model performs inference in exactly the same way as the standard encoder, requiring only a single forward pass to obtain sentence embeddings, without the need to call LLMs or perform any additional computation. This means that with a limited one-time training cost, we achieve significant improvements in semantic ranking performance while maintaining the efficient inference speed of the lightweight model.

7. Conclusions

In this study, we proposed SciRep, a ranking-aware representation model for scientific text that employs a two-stage ranking distillation framework to capture fine-grained semantic differences. Our experimental results provide quantitative evidence supporting the effectiveness of this approach. Specifically, SciRep achieves an average score of 1.89, outperforming the strongest baseline by 11.3%. The MRR evaluation further confirms the effectiveness of our approach, with SciRep achieving the highest average MRR of 0.95 across all tasks. In task-specific evaluations, SciRep achieves the best performance on title–abstract matching and Abstract Segmentation Consistency. Ablation studies further demonstrate the complementary contributions of the three core components: the contrastive learning framework establishes the semantic foundation, the LLM-based ranking distillation introduces hard ranking signals, and the multi-teacher collaborative distillation facilitates fine-grained knowledge transfer. These results collectively confirm that the proposed ranking-aware distillation mechanism significantly enhances scientific text representation quality while maintaining efficient inference.

8. Limitations

Despite the promising results, several limitations of this study should be acknowledged. First, the two-stage distillation process introduces additional computational overhead during training. While the lightweight student model achieves efficient inference, the training phase is considerably more time-consuming compared to standard contrastive learning methods such as SimCSE. Second, the use of Llama-3-8B as the teacher model may introduce potential biases, as the generated hierarchical samples inevitably reflect the linguistic patterns and semantic preferences of the underlying LLM, which may not always align with human judgment in specialized scientific domains. Third, our evaluation is primarily conducted on English scientific abstracts, and the generalizability of SciRep to other languages remains to be validated.

9. Future Work

Building on these limitations, future research will focus on three specific directions. First, to reduce training overhead, we plan to explore more efficient distillation strategies, such as one-stage direct distillation from LLMs to lightweight models or the use of smaller teacher models to balance performance and computational cost. Second, to quantitatively measure and mitigate potential biases from the LLM teacher, we plan to: (i) compute the agreement between multiple LLMs (e.g., Mistral-7B, Qwen-7B, and Gemma-7B) on the same ranking generation task, thereby quantifying the degree of model-specific bias; (ii) explore ensemble distillation methods that leverage ranking aggregation from multiple LLMs to generate more robust and less biased supervisory signals; and (iii), to assess domain generalizability, we will extend our evaluation to multilingual scientific corpora and other document types. Finally, we plan to deploy the trained SciRep model in real-world scientific literature retrieval and recommendation systems to validate its practical utility beyond benchmark evaluations.