LLM-Guided Weighted Contrastive Learning with Topic-Aware Masking for Efficient Domain Adaptation: A Case Study on Pulp-Era Science Fiction

Abstract

Domain adaptation of pre-trained language models remains challenging, especially for specialized text collections that include distinct vocabularies and unique semantic structures. Existing contrastive learning methods frequently rely on generic masking techniques and coarse-grained similarity measures, which limit their ability to capture fine-grained, domain-specific linguistic nuances. This paper proposes an enhanced domain adaptation framework by integrating weighted contrastive learning guided by large language model (LLM) feedback and a novel topic-aware masking strategy. Specifically, topic modeling is utilized to systematically identify semantically crucial domain-specific terms, enabling the creation of meaningful contrastive pairs through three targeted masking strategies: single-keyword, multiple-keyword, and partial-keyword masking. Each masked sentence undergoes LLM-guided reconstruction, accompanied by graduated similarity assessments that serve as continuous, fine-grained supervision signals. Experiments conducted on an early 20th-century science fiction corpus demonstrate that the proposed approach consistently outperforms existing baselines, such as SimCSE and DiffCSE, across multiple linguistic probing tasks within the newly introduced SF-ProbeEval benchmark. Furthermore, the proposed method achieves these performance improvements with significantly reduced computational requirements, highlighting its practical applicability for efficient and interpretable adaptation of language models to specialized domains.

1. Introduction

Domain adaptation remains a fundamental challenge in natural language processing, particularly when pre-trained language models encounter specialized text collections with distinctive vocabularies, stylistic conventions, and semantics structures that diverge significantly from general-domain training data [1,2,3]. While transformer-based language models such as BERT and RoBERTa demonstrate outstanding performance on standard NLP benchmarks, their effectiveness often notably decreases within specialized domains, where precise understanding of domain-specific linguistic nuances is crucial [2,4]. This performance gap becomes particularly critical in scenarios where traditional fine-tuning methods require substantial computational resources or where domain-specific training data is limited [5,6].

Traditional domain adaptation methods often demand significant computational power and extensive annotated or unlabeled datasets, rendering them impractical for many specialized applications [3,7]. Domain-adaptive pre-training methods, although effective, also incur substantial computational overhead and require considerable domain-specific data, limiting their real-world applicability [8,9]. Therefore, there exists a clear need for more efficient, interpretable, and effective domain adaptation methodologies suitable for resource-constrained deployment scenarios.

Recently, contrastive learning has emerged as a promising strategy for learning robust text representations with minimal manual annotations [10,11,12,13]. Methods such as SimCSE leverage contrastive objectives to cluster semantically similar representations while differentiating dissimilar ones, significantly improving sentence embedding quality [10]. Furthermore, recent advances integrating feedback from large language models (LLMs) have introduced nuanced, graduated similarity signals, further enhancing contrastive learning outcomes [14]. However, existing contrastive learning frameworks frequently rely on generic masking strategies, uniformly masking tokens without considering their semantic importance, potentially overlooking critical domain-specific semantic cues [6,15].

The central challenge lies in the mismatch between generic masking strategies optimized for general domains and the precise semantic structures intrinsic to specialized text corpora. While existing LLM-based methods provide valuable fine-grained supervision [16,17], their generic augmentation often neglects critical domain-specific knowledge that could significantly boost adaptation performance. Moreover, graduated similarity feedback derived from LLMs remains underexplored when systematically combined with domain-aware masking strategies [18,19].

To address these limitations, this paper proposes a novel domain adaptation framework that integrates LLM-guided weighted contrastive learning with a topic-aware masking strategy explicitly tailored for specialized textual domains. The proposed method leverages topic modeling techniques [20] to systematically identify domain-critical terms, guiding precise masking strategies through three targeted approaches: single-keyword masking for precise semantic distinctions, multiple-keyword masking for comprehensive contextual understanding, and partial-keyword masking for nuanced linguistic relationships. These masking strategies ensure meaningful augmentation by preserving critical semantic integrity while systematically introducing controlled semantic variations guided by LLM feedback.

The effectiveness of the proposed framework is evaluated on an early 20th-century science fiction corpus, a specialized domain characterized by unique vocabularies and intricate linguistic patterns. A comprehensive empirical evaluation conducted using the newly developed SF-ProbeEval benchmark demonstrates consistent performance improvements compared to state-of-the-art contrastive learning baselines, including SimCSE and DiffCSE, while achieving these improvements with significantly reduced computational overhead.

The primary contributions of this work are as follows:

• Domain-aware weighted contrastive learning framework: Introduction of a systematic topic modeling-based masking approach combined with graduated LLM similarity feedback, significantly outperforming generic masking strategies while maintaining computational efficiency through minimal training requirements (2 epochs vs. traditional 10–40 epochs);
• SF-ProbeEval benchmark: Development of the first specialized benchmark for early 20th-century science fiction text understanding, comprising five linguistically targeted probing tasks that address the gap in domain-specific evaluation frameworks for historical literary corpora;
• Comprehensive empirical validation: Rigorous experimental evaluation demonstrating consistent improvements across multiple linguistic metrics with practical applicability for specialized text processing scenarios where traditional fine-tuning is resource-intensive or impractical.

This research advances efficient and interpretable domain adaptation methodologies, particularly emphasizing computational efficiency, representation quality, and practical deployment considerations for electronics and technology applications where specialized text understanding is critical.

2. Related Works

2.1. Mathematical Formulation of Domain Adaptation

Domain adaptation for language models can be formalized as learning a mapping function $f_{\theta }$: ${\mathcal{X}}_s\to {\mathcal{X}}_t$, where ${\mathcal{X}}_s$ represents the source domain (general text) and ${\mathcal{X}}_t$ represents the target domain (specialized text). The objective is to minimize the domain discrepancy while preserving semantic relationships:

$$\underset{\theta }{min}{\mathcal{L}}_{\mathrm{task}}\left(\theta \right)+\lambda {\mathcal{L}}_{\mathrm{domain}}\left(\theta \right)$$

(1)

where ${\mathcal{L}}_{\mathrm{task}}$ represents the task-specific loss and ${\mathcal{L}}_{\mathrm{domain}}$ captures domain alignment.

The theoretical foundation of domain adaptation rests on understanding the relationship between source and target domain distributions. Classical domain adaptation theory establishes that target error can be bounded by source error plus a domain discrepancy term [21]:

$${ϵ}_t\left(h\right)\le {ϵ}_s\left(h\right)+{\displaystyle \frac{1}{2}}{d}_{\mathcal{H}\Delta \mathcal{H}}({\mathcal{D}}_s,{\mathcal{D}}_t)+\lambda$$

(2)

where $d_{\mathcal{H}\Delta \mathcal{H}}$ measures domain discrepancy and $\lambda$ represents the combined error of the ideal joint hypothesis. This upper bound provides a theoretical guarantee: when domain discrepancy is minimized and source error is low, target performance should improve.

However, this optimistic bound assumes that source and target label distributions are aligned. Zhao et al. rigorously demonstrated a fundamental limitation by establishing a complementary lower bound [21]:

$${ϵ}_t\left(h\right)\ge {\displaystyle \frac{1}{2}}|P_s(Y=1)-P_t(Y=1)|-{ϵ}_s\left(h\right)$$

(3)

This lower bound reveals that when source and target label priors differ significantly (i.e., $|P_s(Y=1)-P_t(Y=1)|$ is large), blindly enforcing feature invariance can paradoxically increase target error rather than reduce it. The gap between Equations (2) and (3) illustrates a critical insight: successful domain adaptation requires not only minimizing domain discrepancy (as suggested by Equation (2)) but also accounting for label distribution shifts (as warned by Equation (3)).

This theoretical tension has profound implications for specialized domain adaptation. In contexts such as early 20th-century science fiction texts, where both feature distributions and semantic label structures differ substantially from modern general corpora, naive invariance-based approaches guided solely by Equation (2) may fail as predicted by Equation (3).

These insights led to sophisticated divergence measures like the Margin Disparity Discrepancy (MDD) [22] and f-divergence frameworks [23]. Recent advances address large domain gaps through gradual adaptation [24], open-set scenarios with novel classes [25], and information-theoretic bounds [26]. These theoretical developments establish mathematical foundations that directly inform sophisticated adaptation strategies, including the proposed weighted contrastive learning approach that simultaneously addresses feature alignment and semantic relationship preservation through graduated similarity supervision.

2.2. Domain Adaptation for Specialized Text Collections

Pre-trained models exhibit performance degradation on specialized texts due to unique vocabulary, domain-specific syntax, and specialized semantic relationships that differ significantly from general training corpora [1]. Domain-adaptive pre-training (DAPT) through continued masked language modeling on target corpora remains the primary strategy, with demonstrated improvements across diverse domains [3]. However, DAPT approaches often require substantial computational resources and extensive domain-specific corpora, making them impractical for many specialized applications.

Specialized text processing has witnessed methodological advances addressing resource constraints. Contextualized Construct Representations combine psychometric expertise with transformers for Classical Chinese analysis, employing supervised contrastive learning to address data scarcity [27]. The Adapt–Retrieve–Revise framework achieves substantial gains in legal domains by integrating model adaptation with LLM-based evidence assessment [28].

Purpose-built models also show domain-specific superiority but require extensive resources. For example, MacBERTh, trained on English texts from 1450 to 1950, outperforms adapted variants on period-specific tasks [29], though such models demand substantial computational investment and large domain-specific corpora.

Despite these advances, existing domain adaptation approaches face limitations when applied to specialized text collections. Traditional fine-tuning methods require extensive computational resources, while generic contrastive learning approaches fail to capture domain-specific semantic nuances. The challenge lies in developing efficient adaptation strategies that leverage domain-specific knowledge without demanding prohibitive computational overhead.

2.3. Contrastive Learning and Domain-Aware Augmentation

Contrastive learning has emerged as the dominant paradigm for learning sentence representations. SimCSE established foundational principles using dropout noise for positive pair generation, achieving strong performance with minimal supervision [10]. Building upon this, DiffCSE introduced conditional masking with Replaced Token Detection (RTD), combining contrastive objectives with token-level discrimination [13]. These methods demonstrated that sophisticated augmentation strategies—from adversarial attacks [11] to document structure exploitation [12]—substantially enhance representation quality.

Recent advances incorporating large language model feedback have further evolved the field. CLAIF leverages LLMs to provide graduated similarity scores $y_i\in [0,1]$ instead of binary labels, though it abandons the contrastive framework for regression objectives [14]. Contemporary methods like SimCSE++ [30] address technical limitations such as dropout contamination, while prompt-based approaches [31] and whitening techniques [32] explore alternative supervision mechanisms.

However, existing contrastive methods predominantly employ generic augmentation strategies that treat all tokens uniformly [15,33]. Random masking, token shuffling, and syntactic perturbations assume equal importance across tokens, potentially obscuring domain-critical semantic relationships. This limitation becomes particularly acute in specialized domains where certain terms carry disproportionate semantic weight. While LLM-feedback methods offer valuable supervision through synthetic generation [34,35] and constitutional AI approaches [36], their generic masking fails to exploit domain-specific knowledge.

Topic modeling techniques like BERTopic [20] provide systematic identification of semantically important terms within specialized corpora, yet their integration with contrastive learning remains unexplored. This gap motivates the domain-aware approach that combines topic modeling-based masking with LLM-guided similarity assessment, ensuring that domain-critical vocabulary receives appropriate attention during adaptation. By explicitly targeting semantically important terms identified through topic analysis, the proposed framework addresses the fundamental mismatch between generic augmentation and the distinctive semantic structures of specialized text collections.

3. Methodology

3.1. Problem Formulation

Given a specialized domain corpus $\mathcal{D}=\{x_1,x_2,\dots ,x_n\}$ where each $x_i$ represents a sentence, the objective is to learn domain-adapted sentence representations $h_i=f_{\theta }\left(x_i\right)$ that capture specialized semantic relationships. This research formulates domain adaptation as an optimization problem where the model parameters are learned through weighted contrastive learning:

$${\theta }^{*}=arg\underset{\theta }{min}\sum _{i=1}^N{\mathcal{L}}_{\mathrm{weighted}}(h_i,h_i^{+},y_i;\theta )$$

(4)

where $h_i^{+}$ represents the LLM-enhanced positive pair and $y_i\in [0,5]$ denotes the graduated similarity score provided by large language model assessment. This formulation differs from traditional contrastive learning by incorporating continuous supervision signals rather than binary similarity judgments, enabling the more nuanced semantic relationship learning essential for specialized domain adaptation.

3.2. Domain-Specific Dataset Construction

This research employs a curated collection of early 20th-century science fiction texts to demonstrate the effectiveness of the proposed framework under challenging adaptation scenarios. The dataset comprises 531 documentswith approximately 18,618 text segments and an estimated 35,000 distinct sentences after comprehensive preprocessing and quality control. The corpus represents a domain characterized by specialized vocabulary, distinctive stylistic conventions, and domain-specific semantic relationships, making it an ideal testbed for evaluating domain adaptation methodologies under resource-constrained conditions.

To ensure reproducibility, this study documents the complete preprocessing pipeline applied prior to topic modeling and contrastive pair generation. Preprocessing is employed primarily to construct the domain lexicon for topic modeling, while the raw sentences used for contrastive training retain their original orthography and punctuation to preserve domain-specific features.

Text segmentation: Stories were segmented into paragraphs using blank-line delimiters (regex pattern: \n\s*\n+), thereby preserving natural discourse boundaries. Paragraphs shorter than ten words were discarded to avoid fragments.

Text normalization (for topic modeling only): For constructing the domain lexicon via BERTopic, the following operations were applied: (i) lowercase conversion and stopword removal using NLTK English stopwords, (ii) lemmatization with WordNetLemmatizer, (iii) minimum word-length filtering (three characters), (iv) enforcement of a minimum alphabetic ratio (70% per paragraph), (v) contraction expansion (e.g., “won’t” → “will not”), and (vi) hyphen normalization for archaic forms (e.g., “to-morrow” → “tomorrow”). These normalizations do not affect the sentences used in contrastive training, thereby ensuring authentic domain representation.

Deduplication: During pair construction, duplicate pair IDs were resolved by retaining only the instance with the highest hybrid similarity score, which combines surface-level and semantic similarity metrics. This method ensures the most representative version of each semantic unit while eliminating redundant variations.

Quality filtering: Several criteria were applied to safeguard the integrity of training pairs: (i) removal of texts containing residual <mask> tokens from failed infilling, (ii) exclusion of low-confidence generations based on configurable infill log-probability thresholds, (iii) enforcement of hybrid similarity score thresholds to remove semantically mismatched pairs, and (iv) length filtering (4–180 tokens) to maintain consistency for model training.

This preprocessing pipeline produces a rigorously curated corpus that balances authenticity with analytical tractability. By retaining original orthography and punctuation in contrastive training data while applying normalization solely for topic modeling, the approach preserves the distinctive stylistic and lexical features of the early science fiction domain. The resulting collection provides a robust foundation for subsequent stages of experimentation—similarity assessment, contrastive pair construction, and model training—ensuring both reproducibility and domain fidelity.

3.3. Language Model Architectures

Two established transformer-based encoders serve as foundation models for the adaptation framework: BERT-base (uncased) and RoBERTa-base. The selection of these architectures demonstrates the model-agnostic nature of the proposed approach, ensuring that the framework’s effectiveness extends across different pre-trained language model architectures. The methodology encompasses two main stages as illustrated in Figure 1: LLM-guided contrastive pair generation and weighted contrastive learning optimization.

3.4. LLM-Guided Contrastive Pair Generation

The framework employs a sophisticated sentence pair generation strategy that leverages domain-specific masking techniques to create meaningful contrastive examples. The process begins with intelligent keyword identification through topic modeling analysis, enabling systematic masking of domain-critical terms that carry significant semantic weight.

3.4.1. Domain-Aware Masking Strategy

Traditional contrastive learning employs random masking with uniform probability. This research introduces domain-aware masking based on topic modeling results. Let ${\mathcal{V}}_d=\{v_1,v_2,\dots ,v_k\}$ represent domain-critical vocabulary identified through BERTopic analysis. The masking probability for token t is defined as

$$P_{\mathrm{mask}}\left(t\right)=\{\begin{array}{cc}\alpha ·\mathrm{importance}\left(t\right)\hfill & \mathrm{if} t\in {\mathcal{V}}_d\hfill \\ \beta \hfill & \mathrm{otherwise}\hfill \end{array}$$

(5)

where $\mathrm{importance}\left(t\right)$ is derived from c-TF-IDF scores, and $\alpha >\beta$ to prioritize domain-critical terms. This formulation ensures that semantically important vocabulary receives preferential attention during augmentation, addressing the limitation of generic masking strategies that treat all tokens equally.

Three masking strategies are implemented to capture different levels of semantic variation:

• Single-keyword masking: $M_1\left(x\right)=\mathrm{mask}(x,\mathrm{sample}({\mathcal{V}}_d,1))$ targets individual domain-critical terms.
• Multiple-keyword masking: $M_2\left(x\right)=\mathrm{mask}(x,\mathrm{sample}({\mathcal{V}}_d,k))$, where $k\sim \mathrm{Uniform}(2,4)$ for broader contextual variation.
• Partial-keyword masking: $M_3\left(x\right)=\mathrm{partial}\_\mathrm{mask}(x,{\mathcal{V}}_d)$ for compound terms (e.g., “space-ships” → “<mask>-ships”).

3.4.2. LLM-Based Sentence Generation and Scoring

A key innovation of the proposed framework lies in the design of domain-specific prompts that guide LLM-based augmentation. Unlike generic approaches that apply universal templates across domains, these prompts are deliberately constructed to preserve the distinctive linguistic characteristics of specialized texts. Generic prompting would compromise domain fidelity—for example, by replacing period-specific terms such as “aether” with modern expressions like “Wi-Fi,” or by substituting culturally embedded terms such as “ray-gun” with trivialized alternatives like “laser pointer.” Such distortions would undermine the semantic integrity of early twentieth-century science fiction, producing anachronistic substitutions that yield training pairs which mislead, rather than enhance, the model’s domain understanding.

The domain-aware prompting strategy addresses this challenge by explicitly instructing the LLM to preserve four critical features characteristic of pulp-era science fiction: (1) the core narrative meaning within speculative contexts, (2) the retention of period-appropriate scientific devices and technological concepts, (3) the consistency of temporal and spatial settings distinctive of pulp-era narratives, and (4) fidelity to domain-critical keywords identified through topic modeling. This targeted approach ensures that when a token such as “rocket” is masked in a 1920s text, the model generates period-appropriate alternatives such as “projectile” or “torpedo,” rather than anachronistic terms like “spacecraft” or “shuttle.”

For each masked sentence $h^{\prime }$, GPT-4.1-mini generates an infilled version through

$$h^{+}=G_{\mathrm{LLM}}(h^{\prime },{\mathrm{prompt}}_{\mathrm{infill}})$$

(6)

The infilling prompt is designed to preserve domain-specific characteristics:

“Your task is to rewrite the following early 20th-century science fiction sentence by replacing each <mask> token with the most appropriate and coherent word to complete the sentence grammatically while preserving the style and meaning. If you cannot fill in the blanks, just say None as your answer.”

This prompt ensures that replacements maintain period-appropriate vocabulary. For instance, when “apparatus” is masked, the model generates “equipment” (similarity score: 5), preserving the technical terminology of the era rather than introducing modern jargon.

The same LLM then provides graduated similarity assessment:

$$y=S_{\mathrm{LLM}}(h,h^{+},{\mathrm{prompt}}_{\mathrm{score}})\in [0,5]$$

(7)

The scoring prompt evaluates semantic preservation across multiple domain-specific dimensions:

“Your task is to rate the similarity of two sentences drawn from early 20th-century science fiction prose. Return a single floating-point score between 0 and 5 inclusive. Consider four aspects when judging similarity: 1. preservation of the core meaning, 2. retention or plausible substitution of SF devices, 3. consistency of temporal, spatial, and technological background, 4. fidelity to key SF topic words.”

The scoring rubric ranges from 5 (fully equivalent with all details preserved) to 0 (no substantive overlap), providing fine-grained supervision that captures semantic nuances. For example, replacing “information” with “technology” in the context of “giving to the Hans” yields a score of 4, while replacing “solar system” with “harbor” drops to 2, reflecting the loss of domain-specific meaning.

This domain-aware prompting strategy represents a key methodological contribution. By explicitly encoding domain knowledge into both generation and evaluation, we ensure that contrastive pairs maintain semantic validity within the specialized domain. The approach creates training triplets $(h,h^{+},y)$ where the similarity scores reflect genuine domain-specific semantic relationships rather than generic surface similarity. This targeted design directly contributes to the strong performance on semantic tasks (Word Contents: 80.86% for RoBERTa), confirming that domain-specific prompting successfully preserves critical semantic information. Full prompt templates with complete scoring rubrics are provided in Appendix A.1 for reproducibility.

3.5. Weighted Contrastive Learning Framework

As shown in Figure 1B, the weighted contrastive learning process utilizes the generated sentence pairs to enhance domain-sensitive representations. The framework addresses fundamental limitations in existing contrastive learning approaches while building upon their structural advantages. Unlike SimCSE [10]’s reliance on generic dropout noise as the sole augmentation mechanism, this research introduces domain-aware masking strategies that systematically target vocabulary elements with high semantic significance within specialized domains. Where SimCSE employs identical sentences with different dropout masks as positive pairs, the proposed approach generates semantically varied but related sentences through intelligent masking and LLM-guided reconstruction.

The framework integrates the dual-objective structure of DiffCSE [13] with graduated supervision mechanisms inspired by CLAIF [14] while addressing critical limitations in both approaches. DiffCSE employs binary token-level supervision without fine-grained similarity guidance, while CLAIF abandons the contrastive learning structure entirely in favor of regression-based objectives. The proposed method preserves contrastive learning’s comparative advantages while incorporating continuous LLM-derived supervision.

Let $(h_i,h_i^{+},y_i)$ represent the embedding of an anchor sentence, its LLM-processed variant, and the associated similarity score. The training objective combines two complementary losses:

Weighted contrastive loss:

$$\mathcal{L}\mathrm{contrast}=-{\displaystyle \frac{1}{N}}\sum {i=1}^N{y}_{i}log{\displaystyle \frac{e^{\mathrm{sim}(h_i,h_i^{+})/\tau }}{{\sum }_{j=1}^N{e}^{\mathrm{sim}(h_i,h_j^{+})/\tau }}}$$

(8)

where N represents the batch size, $\mathrm{sim}(·,·)$ denotes the cosine similarity function, $\tau$ is the temperature parameter, and $y_i\in [0,5]$ is the LLM-generated similarity score that serves as a continuous weight rather than a binary supervision signal. This weighted contrastive objective differs from SimCSE’s uniform treatment of positive pairs by incorporating graduated similarity scores $y_i$ as continuous supervision weights. Unlike CLAIF’s approach of replacing contrastive learning with MSE regression ($\mathcal{L}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{(cos(h_i,h_i^{+})-y_i)}^2$), this formulation maintains the comparative learning structure while providing nuanced guidance through LLM-generated similarity assessments.

Replaced Token Detection (RTD) loss: Following DiffCSE, the framework incorporates a Replaced Token Detection loss to enhance the model’s sensitivity to semantic variations at the token level. The RTD mechanism operates through a three-stage process: (1) domain-aware masking of the original sentence x to create $x^{\prime }$, (2) LLM-based reconstruction to generate $x^{″}$, and (3) token-level discrimination to identify which positions contain original versus generated tokens.

For a sentence of length T, the RTD loss is formulated as

$${\mathcal{L}}_{\mathrm{RTD}}^x=\sum _{t=1}^T\left(-⊮\left(x_{\left(t\right)}^{″}=x_{\left(t\right)}\right) logD \left(x^{″},\mathbf{h},t\right)-⊮\left(x_{\left(t\right)}^{″}\ne x_{\left(t\right)}\right) log \left(1-D \left(x^{″},\mathbf{h},t\right)\right)\right)$$

(9)

where

• T denotes the sentence length (number of tokens);
• $x_{\left(t\right)}$ represents the t-th token in the original sentence x;
• $x_{\left(t\right)}^{″}$ represents the t-th token in the LLM-generated sentence $x^{″}$;
• $⊮(·)$ is the indicator function that returns 1 if the condition is true, 0 otherwise;
• $D(x^{″},\mathbf{h},t)\in [0,1]$ is the discriminator’s predicted probability that the token at position t has been replaced;
• $\mathbf{h}$ is the sentence-level embedding that provides contextual information to the discriminator.

The RTD objective functions as a conditional binary classification task at each token position. When the generated token matches the original ($x_{\left(t\right)}^{″}=x_{\left(t\right)}$), the loss encourages the discriminator to output a low probability (predicting “not replaced”). Conversely, when the tokens differ ($x_{\left(t\right)}^{″}\ne x_{\left(t\right)}$), the loss encourages a high probability (predicting “replaced”). This token-level supervision forces the sentence encoder to produce representations $\mathbf{h}$ that contain sufficient semantic information for accurate token-level discrimination.

The batch-level RTD loss aggregates individual sentence losses:

$${\mathcal{L}}_{\mathrm{RTD}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathcal{L}}_{\mathrm{RTD}}^{x_i}$$

(10)

Combined training objective:

The final training objective combines both losses with a weighting coefficient $\lambda$:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{contrast}}+\lambda ·{\mathcal{L}}_{\mathrm{RTD}}$$

(11)

where $\lambda$ balances the relative importance of sentence-level contrastive learning and token-level discrimination. The dual-objective design ensures that the sentence encoder learns representations that are both semantically coherent at the global level (through weighted contrastive learning) and sensitive to fine-grained lexical variations (through RTD supervision).

This architectural innovation addresses key limitations in existing approaches: SimCSE’s inability to distinguish between different levels of semantic similarity, DiffCSE’s reliance on binary token supervision, and CLAIF’s abandonment of comparative learning structures. The integration of graduated LLM feedback as similarity weights, rather than regression targets, preserves the contrastive learning framework’s inherent advantages while providing more informative supervision than traditional binary approaches. The RTD component ensures comprehensive supervision at both the sentence and token levels, enabling the framework to capture both global semantic relationships and fine-grained lexical variations essential for specialized domain understanding.

3.6. Implementation Details

The framework is optimized for computational efficiency while maintaining adaptation effectiveness. Training employs minimal epoch requirements (2 epochs for both BERT and RoBERTa) with carefully tuned hyperparameters following established contrastive learning configurations [13]. The learning rates are set to $7\times {10}^{-6}$ for BERT-base and $1\times {10}^{-5}$ for RoBERTa-base, with masking ratios of 0.30 and 0.20, respectively. The principal hyperparameters used for domain adaptation training are summarized in

Table 1. Principal hyperparameters for domain adaptation training.

Encoder	Learning Rate	Masking Ratio	$\mathit{\lambda }$	Epochs	Batch Size
BERT-base	7 $\times {10}^{-6}$	0.30	0.005	2	64
RoBERTa-base	1 $\times {10}^{-5}$	0.20	0.005	2	64

.

To ensure fairness and reproducibility, the temperature is fixed at $\tau =0.05$, following common practice in contrastive sentence embedding research (e.g., SimCSE; DiffCSE) [10,13]. This study deliberately refrains from tuning $\tau$ for either the proposed method or the baselines in order to ensure that (i) all systems are compared under identical conditions without conferring a tuning advantage, (ii) experimental settings remain aligned with widely adopted prior work, and (iii) observed improvements can be attributed to the core contributions—weighted contrastive learning with LLM-guided feedback—rather than temperature optimization. This decision also preserves the two-epoch compute budget adopted in this study.

The batch size is fixed at 64 to balance computational efficiency with training stability. All experiments are conducted on a single NVIDIA A100 GPU, thereby demonstrating the framework’s feasibility for deployment under resource-constrained conditions. Reproducibility is further supported through fixed random seeds (seed = 42) and standardized experimental configurations.

4. Experimental Evaluation

This section evaluates the proposed weighted contrastive learning framework through comprehensive linguistic analysis and comparative assessment against established domain adaptation methods. The evaluation emphasizes practical deployment considerations, computational efficiency, and performance gains achievable under resource-constrained scenarios typical of industrial applications.

4.1. SF-ProbeEval: A Domain-Specific Probing Benchmark

To evaluate embedding quality in early science fiction texts, this research constructs SF-ProbeEval, a specialized benchmark comprising five probing tasks tailored to pulp-era linguistic patterns. Each task contains 1000 test items derived from the specialized domain corpus, capturing the unique challenges of historical science fiction text understanding. SF-ProbeEval provides the first standardized benchmark for assessing sentence embedding quality in early 20th-century science fiction texts, addressing the gap in domain-specific evaluation frameworks for historical literary corpora.

SF-ProbeEval construction follows a two-stage process: automated generation using rule-based algorithms and domain-specific linguistic patterns, followed by expert validation. Three experts from the Department of English Language and Literature reviewed and refined the automatically generated dataset, focusing on linguistic accuracy, difficulty appropriateness, and elimination of generation artifacts. Each test item was evaluated independently by at least two reviewers, with disagreements resolved through consensus-building.

Evaluation protocol: 

Table 2. SF-ProbeEval: a probing dataset for early 20th-century science fiction text understanding.

Task	Description
Word Contents	Identify science fiction terminology and archaic vocabulary, evaluating adaptation to period-specific lexicons including scientific devices, astronomical terms, and technological concepts from pulp-era narratives.
Tree Depth	Predict syntactic complexity levels in vintage prose, assessing understanding of elaborate sentence constructions characteristic of 1920s–1930s science fiction writing styles.
BShift (Bigram Shift)	Detect local syntactic perturbations in period-appropriate word sequences, measuring sensitivity to historical word order patterns and archaic grammatical structures.
SOMO (Semantic Odd Man Out)	Identify semantic anomalies within science fiction contexts, evaluating understanding of genre-specific relationships including scientific speculation and technological innovation concepts.
Coord_Inv (Coordinate Inversion)	Detect structural modifications in complex vintage sentences, assessing comprehension of elaborate discourse patterns typical of early science fiction literary style.

summarizes the SF-ProbeEval tasks employed in this study.

Baseline comparisons: The framework is evaluated against standard contrastive learning approaches to isolate the contribution of AI-guided feedback mechanisms. Four model variants are compared using both BERT-base and RoBERTa-base architectures: (1) unadapted pre-trained encoders serving as domain adaptation baselines, (2) SimCSE fine-tuning with standard positive sampling strategies, (3) DiffCSE fine-tuning with conditional augmentation methods, and (4) the proposed weighted contrastive framework incorporating graduated similarity feedback while maintaining identical computational constraints and training configurations.

4.2. Performance Analysis

Table 3. Performance comparison across different domain adaptation approaches on SF-ProbeEval benchmark. Results demonstrate the practical effectiveness of AI-guided feedback under resource-constrained deployment scenarios.

Model	Method	Word Contents (%)	Tree Depth (%)	BShift (%)	SOMO (%)	Coord_Inv (%)	Avg (%)
BERT	BERT-base	47.14	19.72	67.50	63.33	79.80	55.50
	SimCSE	71.43	17.78	64.17	59.50	69.87	56.55
	DiffCSE	73.71	13.33	62.67	61.67	67.17	55.71
	Proposed	79.71	18.33	69.67	67.00	80.64	63.07
RoBERTa	RoBERTa-base	56.00	20.00	67.17	55.17	72.90	54.25
	SimCSE	74.86	13.89	48.33	60.67	48.99	49.35
	DiffCSE	78.29	15.28	53.33	58.33	51.52	51.35
	Proposed	80.86	13.33	66.67	56.50	74.07	58.29

Note: Results in bold correspond to the best-performing method for each encoder.

presents comprehensive performance comparisons on SF-ProbeEval, demonstrating the practical effectiveness of the proposed framework. The results show consistent and meaningful improvements achieved through AI-guided feedback integration, particularly significant given the resource-constrained adaptation scenario and minimal computational requirements.

For BERT-based implementations, the proposed framework achieves substantial performance gains with a 7.57% average improvement across evaluation metrics, representing a notable advancement in domain adaptation effectiveness. The framework reaches 63.07% average performance, significantly outperforming BERT-base (55.50%), SimCSE (56.55%), and DiffCSE (55.71%) baselines. The framework shows exceptional performance in domain-specific vocabulary understanding (Word Contents: 79.71%), outperforming DiffCSE (73.71%). These improvements are particularly significant considering the minimal computational requirements—achieved with only two training epochs and single GPU deployment, making the approach viable for resource-constrained enterprise environments.

The gains are especially pronounced in tasks critical for specialized domain understanding: word order sensitivity (BShift: 69.67% vs. 67.50% baseline) and coordinate structure comprehension (Coord_Inv: 80.64% vs. 79.80% baseline). These improvements directly translate to enhanced performance in practical applications requiring precise semantic understanding of complex specialized texts.

RoBERTa-based experiments demonstrate similar effectiveness, with the framework delivering a 4.04% average improvement over baseline approaches. Performance reaches a 58.29% average across tasks, substantially outperforming RoBERTa-base (54.25%), SimCSE (49.35%), and DiffCSE (51.35%). The framework shows remarkable improvements in domain-specific vocabulary understanding (Word Contents: 80.86% vs. 56.00% baseline) and structural linguistic analysis (Coord_Inv: 74.07% vs. 72.90% baseline).

Significantly, the results reveal critical limitations in standard contrastive methods when applied to specialized domains. Traditional approaches (SimCSE and DiffCSE) occasionally underperform compared to unmodified pre-trained models, suggesting that naive contrastive augmentation strategies may introduce noise rather than meaningful supervision in domain-specific contexts. This phenomenon underscores the importance of intelligent, domain-aware adaptation approaches such as the proposed LLM-guided framework.

The consistent performance improvements across both transformer architectures demonstrate the framework’s model-agnostic effectiveness and broad applicability. These results are particularly valuable for practitioners operating under resource constraints, where even modest improvements translate to significant operational advantages in production environments. The computational efficiency requirements—minimal hardware demands with substantial performance gains—make this approach practical for enterprise deployment scenarios where extensive fine-tuning is economically unfeasible.

4.3. Embedding Quality Assessment

To evaluate the semantic organization capabilities of the proposed framework, this research employs t-distributed stochastic neighbor embedding (t-SNE) [37] visualization analysis. The assessment focuses on the top five semantic clusters identified through automated topic modeling, comprising 1962 text segments representing diverse domain-specific content patterns. Sentence embeddings generated by BERT-base, RoBERTa-base, and their corresponding adapted models using the proposed framework are reduced using t-SNE with perplexity = 30 and random state = 42.

Figure 2 illustrates the progressive improvements in semantic clustering quality achieved through the proposed adaptation framework. Both baseline models (top row) exhibit substantial semantic overlap between different content categories, with RoBERTa-base showing particularly diffuse cluster boundaries and BERT-base achieving only partial separation. In contrast, the proposed weighted contrastive framework (bottom row) demonstrates superior semantic discriminability across both architectures, with tight, well-separated clusters and minimal inter-category interference.

The improvements are particularly pronounced for three critical domain-specific categories: Neptunian Cataclysm (blue circles), Scientific Crime Trial (orange squares), and Carnivorous Creatures (green triangles). While these clusters show significant overlap in the baseline models—especially in RoBERTa where they form an indistinct central mass—the proposed framework achieves clear delineation with distinct cluster cores and well-defined boundaries. This enhanced separation indicates the framework’s effectiveness in capturing fine-grained semantic distinctions essential for specialized domain understanding.

The visualization results confirm that AI-guided feedback mechanisms effectively enhance semantic organization within specialized domain contexts. This improvement in embedding quality translates directly to better performance in downstream applications requiring precise semantic understanding, such as information retrieval, content categorization, and similarity-based recommendation systems. For enterprise applications, this enhanced semantic organization can significantly improve system accuracy and user satisfaction in domain-specific deployment scenarios where precise content understanding is critical for operational effectiveness.

4.4. Ablation Study

Table 4. Controlled comparison of contrastive learning methodologies using identical specialized domain training data. Results isolate the contribution of AI-guided feedback mechanisms.

Model	Method	Word Contents (%)	Tree Depth (%)	BShift (%)	SOMO (%)	Coord_Inv (%)	Avg (%)
BERT	SimCSE	69.43	16.94	64.17	61.33	75.93	57.56
	DiffCSE	66.57	18.61	70.17	66.00	78.96	60.06
	Proposed	79.71	18.33	69.67	67.00	80.64	63.07
RoBERTa	SimCSE	71.14	20.83	47.50	47.00	59.43	49.18
	DiffCSE	72.00	18.89	52.00	48.17	66.16	51.44
	Proposed	80.86	13.33	66.67	56.50	74.07	58.29

provides controlled comparison results using identical specialized domain training data, isolating the contribution of different contrastive learning methodologies across both BERT and RoBERTa architectures.

The proposed weighted contrastive framework achieves substantial improvements across both architectures when all methods utilize identical training data and computational resources. For BERT, the framework reaches 63.07% average performance, outperforming SimCSE (57.56%) and DiffCSE (60.06%) by 5.51% and 3.01%, respectively. RoBERTa shows even more pronounced gains with a 58.29% average performance, exceeding SimCSE (49.18%) and DiffCSE (51.44%) by 9.11% and 6.85%.

Domain vocabulary understanding (Word Contents) demonstrates the most consistent improvements across both models: BERT achieves 79.71% (vs. 69.43% SimCSE, 66.57% DiffCSE) and RoBERTa reaches 80.86% (vs. 71.14% SimCSE, 72.00% DiffCSE). This indicates that graduated similarity scores from LLMs effectively capture domain-specific semantic nuances that binary supervision approaches miss.

Notably, while baseline methods show architectural sensitivity—SimCSE and DiffCSE perform inconsistently across models—the proposed framework maintains robust performance gains regardless of the underlying architecture. RoBERTa’s baseline methods particularly struggle with syntactic tasks (BShift: 47.50% SimCSE, 52.00% DiffCSE), which the proposed framework substantially improves to 66.67%, demonstrating the value of AI-guided feedback for addressing architectural weaknesses.

These controlled comparisons confirm that performance improvements stem from the AI-generated feedback mechanism rather than dataset characteristics or training configurations, validating the framework’s effectiveness for practical domain adaptation where semantic understanding is critical.

5. Discussion

5.1. Effectiveness of AI-Generated Feedback

This research demonstrates the technical effectiveness of integrating graduated similarity feedback from large language models into contrastive learning frameworks. The fine-grained supervision provided by GPT-4.1-mini consistently improves embedding quality across multiple evaluation metrics, representing a methodological advancement over traditional binary contrastive approaches. The graduated similarity assessment addresses a fundamental limitation in contrastive learning by providing continuous rather than discrete supervision signals, enabling more sophisticated representation learning that captures subtle semantic distinctions essential for specialized domain understanding. The framework’s success across both BERT and RoBERTa architectures, with average improvements of 5–7% and statistically significant micro-average gains ($p<0.001$), validates the robustness of this approach for practical domain adaptation scenarios.

5.2. Computational Efficiency and Resource Constraints

The framework achieves training efficiency by requiring only 2 epochs, compared to 10–40 for typical domain adaptation. However, this benefit comes with quantifiable preprocessing costs from GPT-4.1-mini usage: each sentence requires two API calls (mask infilling and similarity scoring), totaling approximately 70,000 calls for the 35,000-sentence corpus. At current rates (USD 0.15 per 1M input tokens, USD 0.60 per 1 M output tokens), this amounts to roughly USD 2 for the dataset—negligible at research scale but potentially substantial for larger deployments (e.g., approximately USD 600 for 10 M sentences, thousands in USD for 100 M+ sentences).

Beyond direct costs, LLM dependency introduces operational constraints. Typical API latencies (0.5–2 s per call) make real-time adaptation impractical; sensitive domains may be unable to send proprietary data to external services; and model version changes can affect score consistency. These trade-offs highlight the importance of considering total cost of ownership alongside accuracy improvements. Potential mitigations include local LLM deployment (e.g., LLaMA-2-7B, Mistral-7B), selective LLM scoring for high-value examples, and knowledge-distillation pipelines to eliminate runtime LLM dependency while preserving performance gains.

5.3. Framework Design and Transferability

While this framework has been validated on early 20th-century science fiction, its design principles are applicable to other specialized domains characterized by distinctive vocabularies, limited training data, and domain-specific semantic structures. The core components—topic modeling for vocabulary identification, targeted masking strategies, and LLM-guided supervision—are domain-agnostic and can be adapted with minor adjustments.

In technical documentation, masking strategies should preserve hierarchical terminology and acronym–definition relationships. For legal texts, prompts require refinement to maintain statutory precision and cross-reference integrity, while the medical literature demands special handling of clinical terminology and dosage expressions. In each case, topic modeling (e.g., BERTopic) can automatically extract domain-critical vocabulary, and LLM prompts can be instantiated with domain-specific placeholders such as substituting “science fiction devices” with “legal precedents” or “clinical procedures.”

Cross-domain adaptation can follow a reproducible procedure. BERTopic parameters are tuned according to domain characteristics: min_topic_size may be increased (e.g., 10→50) for dense technical vocabularies, whereas nr_topics may be reduced (e.g., 30→10–15) for narrow corpora. Masking ratios are adjusted inversely with vocabulary density, typically around 0.15–0.20 for dense legal texts and 0.25–0.30 for narrative domains. The RTD loss weight $\lambda$ is varied based on lexical diversity, with higher settings (0.01) for highly formulaic corpora and lower settings (0.005) for creative texts. Although this study fixed $\tau =0.05$ following standard practice in contrastive sentence embeddings (e.g., SimCSE; DiffCSE) and did not tune it to preserve fairness and reproducibility under the two-epoch compute budget, future work should evaluate sensitivity through a small grid (e.g., $0.02,0.05,0.10$) under the same budget. These adjustments can be determined efficiently through pilot runs of 500–1000 sentences, preserving the framework’s emphasis on computational efficiency.

The consistent improvements across BERT and RoBERTa (5–7% average gains) demonstrate the framework’s architecture independence and suggest robust transferability. At the same time, domains with strongly formulaic language, such as patent documents, may require additional syntactic constraints beyond semantic similarity. Future evaluations should therefore validate the parameter adjustment strategies across technical, legal, and medical corpora and test whether the two-epoch regime remains sufficient in domains with greater structural complexity than literary texts.

5.4. Limitations and Future Research Directions

This study acknowledges several limitations that suggest directions for future work. First, while the proposed framework yields statistically significant improvements on domain-critical tasks, the average gains are modest (5–7%). The accompanying significance analyses (paired McNemar exact tests with Holm–Bonferroni correction) indicate that micro-average improvements are statistically reliable for both BERT and RoBERTa ($p<0.001$), whereas per-task differences show mixed significance after correction (see Appendix D for detailed results). Notably, syntactic parsing tasks (Tree Depth) show minimal improvements, suggesting that current domain-aware masking strategies may not adequately capture hierarchical linguistic structures. Future work should incorporate syntax-aware augmentation that preserves dependency heads and constituency boundaries during masking, potentially combining semantic similarity with explicit syntactic supervision.

Second, validation with three independent experts on 250 sentence pairs revealed important insights about LLM-generated supervision quality. Expert annotations showed high inter-rater reliability (ICC(2,k) = 0.975; Krippendorff’s ${\alpha }_{\mathrm{interval}}=0.93$), and comparisons with GPT-4.1-mini scores revealed moderate-to-strong correlations (Pearson $r=0.61$, Spearman $\rho =0.64$) with a systematic underestimation bias (−0.35 points on average, RMSE = 1.34). To address concerns about the relatively small validation set, Appendix E reports 95% confidence intervals for the correlation coefficients as well as detailed agreement statistics. These analyses indicate that the sample of 250 pairs provides statistically informative evidence consistent with prior validation practices in contrastive learning (e.g., SimCSE; DiffCSE) while also underscoring the need for larger-scale validation in future work.

Third, the evaluation’s deliberate focus on a single specialized domain enables controlled methodological assessment but limits broader generalizability claims. Systematic cross-domain evaluation—testing models adapted on science fiction against heterogeneous targets such as technical documentation or legal texts—would provide stronger evidence of robustness. Both zero-shot and few-shot transfer regimes merit investigation, as do combinations with domain-adaptive pretraining baselines. Future work should also expand cost–benefit analyses to include comprehensive metrics (GPU hours, API costs, deployment latency) versus accuracy gains, providing clearer guidance for practical deployment decisions.

6. Conclusions

This research contributes a computationally efficient methodology for domain adaptation that integrates graduated LLM feedback into weighted contrastive learning. The framework achieves consistent improvements across linguistic evaluation tasks while requiring only 2 training epochs—a significant reduction from traditional approaches requiring 10–40 epochs—demonstrating that intelligent supervision can substitute for extensive computation.

Experimental validation on early 20th-century science fiction texts reveals both strengths and limitations. The framework excels at domain vocabulary understanding (Word Contents: 80.86% for RoBERTa) and coordinate structure comprehension (Coord_Inv: 80.64% for BERT), with statistically significant micro-average improvements ($p<0.001$) despite modest absolute gains (5–7%). Validation against human expert annotations confirms that LLM-generated similarity scores, though exhibiting systematic underestimation bias ($-0.23$ points), provide sufficiently informative supervision to outperform traditional binary contrastive approaches.

Current limitations point to clear improvement paths. Syntactic parsing performance remains suboptimal, requiring targeted enhancements through syntax-aware masking that preserves dependency structures, auxiliary parsing objectives, and potentially tree-structured attention mechanisms. The single-domain evaluation, while enabling controlled methodological assessment, necessitates systematic cross-domain transfer experiments to establish broader applicability. Additionally, reliance on external LLM APIs introduces operational constraints that merit investigation of open-source alternatives and knowledge distillation strategies for production deployment.

Future work will pursue two critical extensions to enhance robustness and applicability of the framework. The first involves the implementation of dynamic confidence-aware weighting, whereby sample contributions reflect not only graded similarity but also score reliability, estimated through self-consistency checks across multiple LLM passes, calibration on expert-annotated subsets, and entropy-based uncertainty quantification. This approach retains the advantages of contrastive learning while mitigating noise from uncertain supervision. The second extension concerns the development of low-resource prompt tuning techniques designed to reduce manual design costs and improve cross-domain portability. Potential directions include soft prompt embeddings, few-shot prompt induction from minimal seed pairs, and prompt distillation from larger to smaller models. These extensions aim to address fundamental challenges in the practical deployment of domain adaptation systems.

Comprehensive robustness analyses examining temperature parameter sensitivity, masking ratio variations, and confidence intervals for human validation results will further strengthen the framework’s reliability. The complete implementation, including preprocessing pipelines, domain-specific prompts, calibration scripts, and cross-domain transfer protocols, will be publicly released to facilitate reproducibility and accelerate research in resource-constrained domain adaptation.

Despite these limitations, the proposed approach establishes the viability of AI-assisted contrastive learning for specialized domain adaptation, offering a practical pathway for extending modern NLP capabilities to understudied text collections. By explicitly balancing adaptation quality with computational efficiency and transparently characterizing both benefits and trade-offs, the framework provides a foundation for future research in resource-constrained scenarios where traditional fine-tuning remains impractical. The proposed extensions for confidence-aware weighting and low-resource prompt tuning offer concrete paths toward more robust, cost-effective, and broadly applicable domain adaptation methodologies.