Skip to Content
Applied SciencesApplied Sciences
  • Article
  • Open Access

19 May 2026

16 Pages

Cross-Model Deepfake Text Detection with XLM-RoBERTa: A Strongly Generalizable Multi-LLM Training Strategy

and
1
Department of Computer Engineering, Malatya Turgut Ozal University, Malatya 44210, Türkiye
2
Department of Computer Engineering, Firat University, Elazig 23119, Türkiye
*
Author to whom correspondence should be addressed.
This article belongs to the Special Issue Future Applications of Large Language Models

Abstract

The rapid advancement of Large Language Models (LLMs) has significantly complicated the distinction between AI-generated and human-written texts. This challenge becomes particularly pronounced in formal and structurally constrained texts, such as academic writing. In this study, a deep learning approach based on the XLM-RoBERTa architecture is proposed for detecting deepfake (DF) texts, with a focus on achieving strong generalization capability within the academic domain. A large-scale dataset comprising 63,000 human-written and AI-generated texts (from Llama-3.1, Gemma-2, Qwen-2.5, Phi-3, Falcon, and Mistral) was constructed. The proposed multi-model data strategy is designed to encourage the model to learn structural and stylistic distinctions between human and AI-generated texts, rather than memorizing model-specific stylistic patterns, thereby reducing false positive rates, particularly for formal human-written content. To analyze the model’s learning behavior, no preprocessing was applied to the training data. The model was evaluated on two independent test sets (preprocessed and non-preprocessed), neither of which was seen during training. Experimental results show that the model achieves an F1-score of 99.76% on the validation set, while maintaining 93.42% accuracy and 94.67% recall on the preprocessed (Zero-Artifact) test set. These findings indicate that the model relies on inherent linguistic and structural patterns of AI-generated text in formal context, rather than dataset-specific superficial artifacts, suggesting improved robustness and generalization in academic integrity applications.

1. Introduction

The rapid development and widespread adoption of Large Language Models (LLMs) have significantly enhanced the quality of Artificial Intelligence (AI)-generated text [1,2]. The exceptional performance of these models in complex tasks such as text summarization, machine translation, and natural language generation has led to their extensive use across various domains [3,4]. Despite this advancements in generation quality, language models still exhibit tendencies to produce misinformation and hallucinated content. This raises significant concerns about the accuracy and reliability of information in AI-generated text. In domains where intellectual originality is critical, such as scientific publishing, academic peer review, and education, the unregulated use of synthetic text introduces substantial risks, including plagiarism and violations of academic integrity. Consequently, the reliable and accurate detection of AI-generated text has become a critical research challenge in the modern digital landscape.
Although numerous transformer-based models have been introduced in the literature for detecting AI-generated text, these systems exhibit notable limitations in dynamic, complex and structured scenarios. In many existing approaches, statistical features such as perplexity and sentence length variability play a key role in text classification. The tendency of AI models to generate more structured and lower-variance text is often considered a distinguishing signal between human-written and synthetic content.
However, this approach introduces a significant challenge, particularly in formal and structured text domains such as academic writing. Human-authored texts, including scientific articles that follow predefined structures and inherently exhibit low-variance linguistic characteristics, may be misclassified as AI-generated by existing detection systems [5,6]. Another major limitation lies in the limited out-of-distribution (OOD) generalization ability of current detectors. As the architectures of modern language models continue to evolve and diversify, many detection models tend to overfit to patterns specific to particular generative models during training. As a result, their performance often degrades when evaluated on previously unseen model architectures [7,8]. This indicates that detection systems may develop structural biases against formal human writing and may rely on dataset-specific superficial cues rather than capturing underlying linguistic features.
To address these stylistic and structural limitations in the literature, this study proposes a methodological framework utilizing the XLM-RoBERTa [9] architecture as a classification backbone, with a focus on improving generalization capability within formal and academic discourse. The primary objective is to reduce the false positive bias exhibited by existing detection models toward formal human-written texts. To this end, a “Combined Dataset” comprising a total of 63,000 samples was constructed, consisting of human-written texts and outputs from six different artificial intelligence models (Llama-3.1 [2], Gemma-2 [10], Qwen-2.5 [11], Mistral [12], Phi-3 [13], Falcon [14]) in a fully balanced distribution (31,500 human, 31,500 AI). This multi-model data strategy aims to limit overfitting to stylistic features specific to individual generative models, enable the framework to learn structural distinctions between human and AI-generated text, and improve generalization across diverse model architectures.
In the second stage, a controlled ablation framework was designed specifically for a test set of 1200 samples to evaluate the model’s true detection capability and to highlight the issue of source data contamination, which is often overlooked in the literature. Prior to synthetic generation, the source texts were preprocessed to remove structural inconsistencies, thereby limiting the likelihood of language models reproducing superficial artifacts and enabling a more controlled evaluation setting based on cleaner data. The model was evaluated in this artifact-free environment, applied only to the test set, and achieved an accuracy of 93.42% and a recall of 94.67%. These results suggest that the model has learned consistent linguistic patterns characteristic of AI-generated text in structured contexts, rather than relying on superficial data artifacts for classification. In this context, the primary contributions of this study can be summarized as follows:
  • Addressing the False Positive Paradox in Academic Writing: To mitigate the tendency to misclassify formal and structured human-written texts—such as academic articles—as AI-generated, we propose a balanced and diverse training data strategy consisting of 63,000 samples. This approach aims to enable detection systems to make more reliable and balanced decisions specifically when evaluating human-authored academic content.
  • Enhancing Out-of-Distribution (OOD) Robustness and Cross-Model Generalization: By incorporating outputs from six different large language models (Llama-3.1, Gemma-2, Qwen-2.5, Mistral, Phi-3, and Falcon) into the training process, the model is exposed to diverse generative distributions. This data-centric approach aims to limit overfitting to model-specific patterns and to improve generalization across unseen architectures within formal text domain.
  • Evaluating Memorization Tendencies via Data Purification: The model’s reliance on superficial data artifacts is examined through a controlled data purification (ablation) strategy applied exclusively to the test set. The results suggest that the proposed approach leverages consistent linguistic patterns characteristic of AI-generated text within academic discourse, providing a foundation for more resilient detection mechanisms.

3. Materials and Methods

In this section, the two-stage deep learning training and evaluation framework is presented. While leveraging a conventional XLM-RoBERTa architecture as its classification backbone, the primary focus of this methodology lies in its strategic data curation and rigorous evaluation design, developed to address the challenges of overfitting and the “False Positive” paradox in formal and structured human texts. The experimental design is structured to limit the model’s tendency to overfit to specific data distributions or superficial structural noise. Accordingly, the process consists of a large-scale initial training phase involving 60,000 samples, followed by an analysis of the observed overfitting behavior, and a final adaptation stage in which a strategic dataset of 3000 samples is incorporated to improve cross-architecture generalization capability. The effectiveness of this data-centric strategy in capturing consistent stylistic markers, rather than relying on superficial cues, is evaluated in a specialized test setting largely purified of synthetic artifacts. The overall architecture of this two-stage training and evaluation pipeline is illustrated in Figure 1.
Figure 1. Overall architecture of the proposed model.
The model training architecture and data flow diagram presented in Figure 1 consist of three main components:
Initial Training and Overfitting Analysis: In the first stage, “Dataset A,” consisting of 60,000 samples and designed to capture domain-specific linguistic features, is used. The evaluation results from the initial training phase reveal that the model exhibits a tendency to overfit to the underlying data distribution.
Strategic Data Integration and Adaptation Training: To mitigate the effects of overfitting, “Dataset B,” comprising 3000 out-of-distribution (OOD) samples, is incorporated into the training process. The XLM-RoBERTa architecture is then retrained on the combined dataset—consisting of 63,000 samples in total—to improve its ability to generalize across diverse model architectures within the academic domain.
Zero-Artifact Test Set and Evaluation: The performance of the final model is evaluated on an isolated test set that undergoes extensive preprocessing to remove data artifacts (e.g., HTML tags, formatting inconsistencies). This evaluation setting is designed to assess the model’s structural resilience based on consistent stylistic patterns rather than relying on superficial, dataset-specific cues.

3.1. Dataset Preparation

3.1.1. Construction of the Initial Dataset (Dataset A)

Dataset A, designed to enable the model to learn inherent linguistic features of formal text, was constructed as a fully balanced dataset (30,000 human-written and 30,000 AI-generated samples). To reduce the tendency of deep learning models to misclassify formal and structured human-written texts—such as academic articles—as synthetic content (false positives), scientific sources were primarily used for the human-written component. In this context, the “arxiv-summarization” [20] dataset, available on the Hugging Face platform and containing approximately 210,000 academic articles, was selected as the primary source. From this dataset, 30,000 human-written texts were sampled. The 30,000 synthetic texts forming the AI component of Dataset A were generated using outputs from three modern large language models (Llama-3.1, Gemma-2, and Qwen-2.5) to mitigate the commonly reported issue of single-model overfitting. A total of 10,000 samples were obtained from each model, ensuring a balanced dataset aligned with the human-written texts. This data collection and balancing strategy establishes a training environment that enables the XLM-RoBERTa model to learn generalizable linguistic patterns characteristic of AI-generated text, rather than relying on content-specific features. Dataset A is illustrated in Figure 2a.
Figure 2. Distribution of the datasets: (a) Initial Training Set (Dataset A), (b) Strategic Adaptation Set (Dataset B), (c) Final Consolidated “Large Mixture” Training Set, and (d) Zero-Artifact Test Set.

3.1.2. Strategic Adaptation Set (Dataset B) and Out-of-Distribution (OOD) Integration

Following the initial training on Dataset A, the model was observed to exhibit limitations when evaluated on next-generation architectures not included in the training set, indicating an out-of-distribution (OOD) generalization issue. To address this limitation and enhance the model’s robustness and adaptability, Dataset B, consisting of 3000 samples, was constructed.
This strategic adaptation set is composed of 1500 human-written texts [21] and 1500 AI-generated samples. The synthetic component includes outputs from three OOD models (Mistral, Phi-3, and Falcon), with 500 samples collected from each model. Dataset B is illustrated in Figure 2b.
This “Large Mixture,” comprising a total of 63,000 samples, was randomly shuffled to ensure a balanced distribution, and the model was retrained on this combined dataset. This approach aims to limit the model’s tendency to overfit to patterns specific to individual generative models and to improve its generalization capability across diverse architectures. The “Large Mixture” dataset is illustrated in Figure 2c.

3.1.3. Zero-Artifact Test Set and Evaluation Setting

To provide a transparent evaluation of the cross-model detection performance and generalization capability of the proposed XLM-RoBERTa model, a dedicated “Zero-Artifact Test Set” consisting of 1200 samples—fully independent of the training process—was constructed. To minimize algorithmic bias during evaluation, the test set was designed as a balanced dataset comprising 600 human-written and 600 AI-generated texts (50–50%).
The AI component of the test set includes synthetic texts sampled equally (100 per model) from six modern language models examined in this study (Llama-3.1, Gemma-2, Qwen-2.5, Mistral, Phi-3, and Falcon). This homogeneous distribution enables a more balanced assessment of the model’s performance across both architectures encountered during training and those considered out-of-distribution (OOD).
Prior to the final evaluation, all texts in the test set were subjected to a comprehensive preprocessing pipeline. Data artifacts that are commonly exploited by detection models in the literature and that can bias evaluation results (such as HTML tags, irregular spacing, markdown formatting, and prompt leakage) were largely removed. This zero-artifact approach provides a more controlled evaluation setting for assessing the model’s resilience against structural variations based on inherent linguistic markers rather than superficial structural noise. The “Zero-Artifact Test Set” used for evaluation is illustrated in Figure 2d.

3.1.4. Technical Parameters and Implementation Strategy for Data Generation

To ensure methodological transparency and high reproducibility, the data generation and curation pipeline were executed under a rigorous set of technical parameters and validation protocols.
The synthesis of academic content was performed using the Ollama framework and the Hugging Face Transformers library. To optimize VRAM utilization while preserving generative quality, Large Language Models (LLMs) were deployed using 4-bit quantization (BitsAndBytes) with bfloat16 compute precision.
The generation process was governed by distinct hyperparameter configurations tailored to the training and evaluation objectives:
  • Context and Prediction Limits: For the primary training distribution, a large context window (num_ctx: 16,384) was maintained to accommodate the structural complexity of full academic papers. The maximum output was constrained to 800 tokens (num_predict) for training samples and 250 tokens (max_new_tokens) for the adaptation and test sets.
  • Stochastic vs. Deterministic Decoding: A temperature-variant strategy was applied to evaluate model resilience. The training datasets (A and B) utilized a stochastic decoding approach with Temperature (T) values between 0.7 and 0.8 and Top-p (Nucleus Sampling) of 0.9. Conversely, the “Zero-Artifact” test set was generated using a high-stability configuration with T = 0.3, prioritizing linguistic consistency over variability to represent a more challenging detection scenario.
A role-based “Expert Academic Researcher” system prompt was implemented to enforce stylistic uniformity. In the strategic adaptation (Dataset B) and evaluation phases, a Title-to-Abstract generation strategy was used, where models were provided only with the scientific title of the paper. Technical constraints were embedded directly into the prompt to prohibit:
  • Introductory Artifacts: Phrases such as “Certainly, here is the abstract” were suppressed at the source.
  • Structural Noise: The use of bullet points, numbered lists, and multi-paragraph formatting was strictly avoided to ensure a continuous, formal academic prose structure.
  • Length Bias: A dynamic word-count matching strategy (capped at 400 words for training and 150–200 words for testing) was applied to prevent the detector from learning simple length-based discriminators.

3.2. XLM-RoBERTa Architecture

In this study, the XLM-RoBERTa architecture is employed as the primary classification backbone for detecting synthetic text. XLM-RoBERTa is an extension of the transformer-based RoBERTa architecture, pre-trained on large-scale multilingual data. The model is trained using the masked language modeling (MLM) objective and is capable of learning diverse linguistic structures and contextual patterns.
XLM-RoBERTa is a large-scale language model trained on the 2.5 TB multilingual CommonCrawl dataset, covering approximately 100 languages. This extensive training corpus enables the model to learn rich representations not only at the word level but also across sentence-level semantic relationships and syntactic structures. The underlying RoBERTa architecture incorporates several key improvements over BERT, including the removal of the Next Sentence Prediction (NSP) objective, training with larger batch sizes, and the use of dynamic masking.
The XLM-RoBERTa-base configuration is employed in this study. This architecture consists of 12 transformer encoder layers, each comprising a 768-dimensional hidden representation and 12 multi-head attention heads. The attention score, which forms the basis of this mechanism, is computed using the Query (Q), Key (K), and Value (V) matrices, as defined in Equation (1):
A t t e n t i o n Q , K , V = s o f t m a x Q K V d k V
here, d k , denotes the dimensionality of the key vectors. With approximately 270 million trainable parameters, this architecture enables the effective modeling of complex contextual relationships within text. Furthermore, the model can process input sequences of up to 512 tokens.
To adapt the contextual representations learned by the pre-trained XLM-RoBERTa model to a binary classification task, a task-specific classification head is integrated into the architecture. The special <s> (start-of-sentence) token, added to the beginning of each input sequence, captures the contextual representation of the entire text. The classification process is carried out as follows:
  • Representation: The 768-dimensional contextual vector corresponding to the <s> token from the final transformer layer is used as the feature representation of the entire input sequence.
  • Regularization: To mitigate overfitting, a dropout layer with a rate of 0.1 is applied to this vector.
  • Linear Transformation: The resulting representation is fed into a linear layer that projects it from a 768-dimensional space to a 2-dimensional output space (Human/AI classes).
  • Activation: In the output layer, the softmax function is applied to convert the raw class scores (logits) into a probability distribution:
P y = i | x = e z i ∑ j = 1 C e z j
In this equation, z i denotes the predicted class score for class i , while C represents the total number of classes, with C = 2.
To ensure compatibility with the softmax activation function used in the classification head, the cross-entropy loss function (which minimizes the discrepancy between the predicted probability distribution and the ground truth labels) is employed during training. The loss value L, defined over N samples and C classes (with C = 2), is computed as follows:
L = − 1 N ∑ i = 1 N ∑ c = 1 C y i , c   l o g ( y ^ i , c )  
During training, the model was fine-tuned on the “Large Mixture” dataset (comprising 63,000 samples) while preserving its pre-trained weights. This approach aims to adapt the model to the task of deepfake text detection while retaining its previously learned linguistic representations.

4. Results

In this section, the training performance of the XLM-RoBERTa model on the expanded dataset of 63,000 samples and the final results obtained through the proposed framework on the “Zero-Artifact” test set (consisting of 1200 samples) are presented.
The fine-tuning process was limited to 3 epochs to mitigate the risk of overfitting. During training, a learning rate of 2 × 10 − 5 and a mini-batch size of 16 were used. Examination of the training process indicates that the model exhibits a stable learning behavior. The training loss, which was approximately 0.42 at the end of the first epoch, decreased to 0.12 by the end of the third epoch.
In addition, the model achieved an accuracy of 95.8% on the validation set, which corresponds to 10% of the training data. This result suggests that the model effectively captured the underlying patterns in the dataset.
To address potential concerns regarding the model’s dependency on superficial data artifacts, we conducted a rigorous ablation study on the test set. We evaluated the performance under three conditions: (1) Raw Text with original formatting, (2) Without Intro Artifacts where common introductory phrases were removed, and (3) Lowercase Only where all casing information was eliminated. The obtained metrics are presented in Table 1.
Table 1. Performance metrics of the model on the 1200-sample test set obtained via ablation.
An examination of Table 1 indicates that the model maintains a consistent performance across the evaluated testing scenarios. Notably, the removal of introductory artifacts did not lead to a decrease in accuracy, which remained stable at 93.41%. Furthermore, converting the text to lowercase resulted in a marginal change of only 0.17%. These results provide evidence that the proposed approach is largely independent of superficial formatting markers and effectively captures inherent stylistic and structural markers characteristic of academic discourse. The consistency of the false positive rate (7.83% or 47/600) across scenarios further suggests a degree of robustness in detecting AI-generated text beyond simple stylistic artifacts.
Table 2 presents a comparative performance analysis of the fine-tuned XLM-RoBERTa approach alongside current statistical and supervised detection methods. To ensure the objectivity of the evaluations, the testing process was conducted on a balanced Out-of-Distribution (OOD) test set consisting of 1200 samples. This dataset comprises 600 human-written academic texts and 600 AI-generated samples produced by several recent large language models, specifically Falcon, Phi-3, Mistral, Qwen-2.5, Gemma-2, and Llama-3.1 (100 samples each).
Table 2. Comparative performance analysis of the proposed model against other detectors (Accuracy%).
Based on the findings, the proposed framework appears to exhibit a notably stable performance across the different tested scenarios. The model achieved a baseline accuracy of 93.41% on raw texts while maintaining a relatively low false positive rate of 7.83% (47/600). Furthermore, the model’s performance remained consistent (93.24–93.41%) during the stages involving structural modifications. This outcome suggests that the methodology captures consistent linguistic patterns specific to academic writing, rather than relying on structural or superficial formatting cues.
In contrast, a significant degree of variability was observed in the performance of zero-shot statistical models such as DetectGPT, Fast-DetectGPT, and Binoculars. While their baseline accuracies ranged between 64.38% and 68.39%, they exhibited considerably higher ethical risks, with false positive rates recorded at 26.67%, 33.00%, and 64.33%, respectively. These models also seemed to be more sensitive to minor structural changes, such as lowercase conversion or the removal of introductory segments. Specifically, the decline in Binoculars’ accuracy during structural modifications might point toward a potential dependency of these methods on specific linguistic patterns found at the beginning of academic texts, which often results in a higher rate of false accusations. Furthermore, general-purpose supervised detectors, namely Hello-ChatGPT and OpenAI-RoBERTa, yielded results near the level of random chance (approximately 50–51%) on this specialized academic test set. Although these models showed lower FPRs, this was primarily attributed to an extreme class bias toward human-text predictions, effectively rendering them incapable of reliably identifying AI-generated academic content. These results suggest that models trained on general web content may encounter difficulties in distinguishing the unique terminological and structural complexities of academic discourse.
The findings indicate that domain-specific fine-tuning may offer a more consistent alternative to statistical or general-purpose approaches, particularly by reducing the risk of false accusations by approximately four to eight times compared to the baseline statistical methods evaluated.
The confusion matrix, constructed to visualize the framework’s prediction performance on the 1200-sample test set and to analyze class-wise error patterns, is presented in Figure 3.
Figure 3. Confusion matrix of the “Zero-Artifact” test set.
An examination of Figure 3 indicates that the model demonstrates a balanced performance in distinguishing between the two classes (Human and AI). The analysis based on the confusion matrix is summarized as follows:
Out of 600 synthetic texts, 568 (94.67%) were correctly classified as “AI (1)”. The remaining 32 instances misclassified as “Human” indicate that, while the model achieves a high detection rate for synthetic content, a limited number of errors still occur.
Similarly, 553 out of 600 human-written texts (92.17%) were correctly classified as “Human (0)”. However, 47 instances were misclassified as false positives. This may be attributed to the high level of formality and structural regularity of academic texts in the dataset, which can partially resemble the generative patterns of large language models.
Overall, the confusion matrix results demonstrate that the model achieves high performance in detecting synthetic texts, while maintaining a balanced level of accuracy in classifying human-written content.
To further analyze the model’s learning dynamics and generalization behavior during training, the training and validation loss curves are presented in Figure 4.
Figure 4. Training and validation loss curves of the model.
An examination of Figure 4 indicates that the training loss exhibited a rapid decline from an initial value of approximately 0.8 and gradually decreased as training progressed, stabilizing at a low level by the end of the third epoch.
The validation loss maintained a consistently low and stable trend throughout training, suggesting that the model effectively learned without overfitting to the training data. Moreover, the diminishing fluctuations in the training loss over time indicate that the model progressively adapted to the underlying linguistic patterns in the dataset.
To evaluate the discriminative power of the proposed approach between classes, a Receiver Operating Characteristic (ROC) curve analysis was conducted. The ROC curve is illustrated in Figure 5.
Figure 5. ROC curve illustrating the model’s performance.
The ROC curve presented in Figure 5 shows that the framework clearly deviates from the random guessing line and follows a trajectory close to the top-left corner. The Area Under the Curve (AUC) was calculated as 0.9810.
This high AUC value indicates that the model achieves strong discriminative performance in distinguishing between human-written and AI-generated texts. Furthermore, this result demonstrates that the model attains a high true positive rate while maintaining a low false positive rate.

5. Discussion

The findings of this study demonstrate that the proposed methodological framework leveraging a conventional XLM-RoBERTa backbone achieves high discriminative performance, particularly for content written in formal and structured academic language. The “Large Mixture” training strategy employed in this study effectively limits the model’s tendency to overfit to specific generative architectures and contributes to more balanced performance across diverse generation styles. In particular, the results obtained in the “Zero-Artifact” test setting indicate that the strategy relies on structural patterns and stylistic markers inherent to academic discourse rather than superficial data artifacts.
The consistent performance exhibited across different architectures, such as Llama, Mistral, and Falcon, suggests that the proposed approach possesses a strong cross-architecture generalization capability within the context of direct LLM outputs. Notably, the approach’s ability to accurately distinguish human-written academic texts—which often share structural similarities with AI-generated content—emerges as a key finding of this study. This indicates that the model bases its decisions on domain-specific contextual features specific to academic writing rather than relying solely on surface-level metrics.
However, the sustained stability of such detection performance may be subject to challenges related to distributional shift. As suggested by Pozzi et al. [22], exposure bias during the training or distillation of Large Language Models (LLMs) could potentially lead to constrained generative distributions, which detection systems might risk overfitting to. To address this potential risk, the current methodology seeks to integrate a degree of stochastic diversity into the data generation phase through the inclusion of a spectrum of decoding configurations, ranging from high-stability outputs (T = 0.3) to more varied distributions (T = 0.8). Furthermore, the utilization of models deployed with 4-bit quantization (BitsAndBytes) may introduce a layer of structural noise into the training distribution, potentially approximating conditions encountered in resource-constrained real-world deployments. These measures are intended to facilitate the identification of relatively invariant stylistic markers of AI authorship, which may contribute to enhanced resilience against the evolving nature of generative distributions.
Furthermore, the high recall values indicate the framework’s strong capacity to detect direct synthetic texts. The low false negative rate suggests that the likelihood of synthetic content going undetected remains limited. However, the rapid advancement in the generative capabilities of AI models necessitates the continuous updating of such detection systems. The emergence of more advanced and human-like generative models makes the ongoing re-evaluation of detection mechanisms essential.
It is important to note that the effectiveness of current detection mechanisms can be significantly challenged by adversarial techniques. As discussed by Sadasivan et al. [8] (2023), many AI text detectors face substantial performance degradation when subjected to paraphrasing or human-in-the-loop editing. While our strategy demonstrates high discriminative power on the presented dataset, its robustness against high-entropy paraphrasing or collaborative human-AI writing remains a limitation of the current study.
From a future research perspective, leveraging the multilingual capabilities of XLM-RoBERTa to evaluate the proposed method across different languages presents a promising research direction. In particular, the morphological richness of agglutinative languages such as Turkish may provide new opportunities for synthetic text detection. Furthermore, incorporating evaluation against proprietary closed-source systems and advanced adversarial testing—such as evaluating the model against various paraphrasing attacks—is a critical next step to enhance the system’s reliability in more complex, real-world adversarial scenarios.

6. Conclusions

In this study, we presented a data-centric framework for AI-generated academic text detection using XLM-RoBERTa as the classification backbone. Rather than introducing a novel detection architecture, the primary objective of this work was to investigate how training diversity and artifact-controlled evaluation influence cross-model robustness in academic AI-text detection. The findings demonstrate that strategically curated multi-model training distributions, combined with a rigorous “Zero-Artifact” evaluation setting, substantially improve the detector’s ability to generalize across diverse large language model architectures while maintaining balanced performance on formal human-written academic texts. The proposed “Large Mixture” strategy exposed the model to outputs from multiple contemporary LLM families, enabling the framework to learn shared linguistic and structural characteristics of AI-generated academic discourse instead of memorizing model-specific stylistic signatures. Experimental results obtained on the independent Zero-Artifact test set demonstrated that the proposed framework achieves strong and stable discriminative capability, reaching 93.41% accuracy and an AUC score of 0.9810 even after the removal of superficial formatting cues and structural artifacts. These findings suggest that the model primarily relies on relatively stable stylistic and contextual markers inherent to synthetic academic writing rather than dataset-specific leakage patterns. A particularly important outcome of this study is the reduction in false positive behavior on highly formal and structured human-authored texts, which remains a major limitation of many existing detection systems. The results indicate that incorporating formal academic writing into a balanced and diverse training distribution can substantially improve robustness against the “false positive paradox” reported in prior literature. Overall, the study highlights that robust academic AI-text detection may depend more on principled dataset design, diversity-aware training strategies, and controlled evaluation methodologies than on major architectural innovation alone. These findings provide evidence that data-centric generalization strategies constitute a practical and scalable direction for improving reliability in academic integrity applications. Nevertheless, the current framework was evaluated primarily on direct LLM generations. As emphasized in prior research, adversarial paraphrasing, human-AI collaborative editing, and increasingly human-like generation capabilities continue to pose significant challenges for future detection systems. Accordingly, future work should focus on multilingual evaluation, paraphrase-resilient detection mechanisms, and robustness against advanced rewriting attacks in more realistic deployment scenarios.

Author Contributions

Conceptualization, E.Ö. and İ.Ö.; data curation, E.Ö. and İ.Ö.; investigation, İ.Ö.; methodology, E.Ö. and İ.Ö.; supervision, E.Ö. All authors declare equal and joint responsibility for the study. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

All relevant data are fully available within the manuscript without restriction. Deepfake-Text-Summarization-Detection-Dataset, arxiv-abstract-2021, arxiv-summarization data that support this study’s conclusions are publicly available online: https://github.com/Connsas/Deepfake-Text-Summarization-Detection-Dataset, accessed on 1 May 2026. https://huggingface.co/datasets/gfissore/arxiv-abstracts-2021, accessed on 1 May 2026. https://huggingface.co/datasets/ccdv/arxiv-summarization, accessed on 1 May 2026.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Kasneci, E.; Sessler, K.; Küchemann, S.; Bannert, M.; Dementieva, D.; Fischer, F.; Gasser, U.; Groh, G.; Günnemann, S.; Hüllermeier, E.; et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 2023, 103, 102274. [Google Scholar] [CrossRef] [Scilit]
  2. Sam, K. Llama 3.1: An in-depth analysis of the next-generation large language model. SSRN Electron. J. 2024, 6139407. [Google Scholar] [CrossRef] [Scilit]
  3. Crothers, E.N.; Japkowicz, N.; Viktor, H.L. Machine-generated text: A comprehensive survey of threat models and detection methods. IEEE Access 2023, 11, 70977–71002. [Google Scholar] [CrossRef] [Scilit]
  4. Zhang, Y.; Jin, H.; Meng, D.; Wang, J.; Tan, J. A comprehensive survey on automatic text summarization with exploration of LLM-based methods. Neurocomputing 2026, 663, 131928. [Google Scholar] [CrossRef] [Scilit]
  5. Li, Y.; Li, Q.; Cui, L.; Bi, W.; Wang, Z.; Wang, L.; Yang, L.; Shi, S.; Zhang, Y. MAGE: Machine-generated text detection in the wild. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, 11–16 August 2024; pp. 36–53. [Google Scholar] [CrossRef] [Scilit]
  6. Mitchell, E.; Lee, Y.; Khazatsky, A.; Manning, C.D.; Finn, C. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA, 23–29 July 2023; pp. 24950–24962. [Google Scholar]
  7. Liang, W.; Yuksekgonul, M.; Mao, Y.; Wu, E.; Zou, J. GPT detectors are biased against non-native English writers. Patterns 2023, 4, 100779. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  8. Sadasivan, V.S.; Kumar, A.; Balasubramanian, S.; Wang, W.; Feizi, S. Can AI-generated text be reliably detected? In Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria, 7–11 May 2024. [Google Scholar]
  9. Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; Stoyanov, V. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 5–10 July 2020; pp. 8440–8451. [Google Scholar]
  10. Gemma Team; Riviere, M.; Pathak, S.; Sessa, P.G.; Hardin, C.; Bhupatiraju, S.; Hussenot, L.; Mesnard, T.; Shahriari, B.; Ramé, A.; et al. Gemma 2: Improving open language models at a practical size. arXiv 2024, arXiv:2408.00118. [Google Scholar] [CrossRef] [Scilit]
  11. Qwen Team; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; et al. Qwen2.5 technical report. arXiv 2024, arXiv:2412.15115. [Google Scholar] [CrossRef] [Scilit]
  12. Jiang, A.Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D.S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. Mistral 7B. arXiv 2023, arXiv:2310.06825. [Google Scholar] [CrossRef] [Scilit]
  13. Abdin, M.; Aneja, J.; Awadalla, H.; Awadallah, A.; Awan, A.A.; Bach, N.; Bahree, A.; Bakhtiari, A.; Bao, J.; Behl, H.; et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv 2024, arXiv:2404.14219. [Google Scholar] [CrossRef] [Scilit]
  14. Almazrouei, E.; Alobeidli, H.; Alshamsi, A.; Cappelli, A.; Cojocaru, R.; Debbah, M.; Goffinet, É.; Hesslow, D.; Launay, J.; Malartic, Q.; et al. The Falcon series of open language models. arXiv 2023, arXiv:2311.16867. [Google Scholar] [CrossRef] [Scilit]
  15. Guo, B.; Zhang, X.; Wang, Z.; Jiang, M.; Nie, J.; Ding, Y.; Yue, J.; Wu, Y. How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection. arXiv 2023, arXiv:2301.07597. [Google Scholar] [CrossRef] [Scilit]
  16. Borile, C.; Abrate, C. How to generalize the detection of AI-generated text: Confounding neurons. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP, Suzhou, China, 4–9 November 2025; pp. 25461–25476. [Google Scholar]
  17. Guo, X.; Zhang, S.; He, Y.; Zhang, T.; Feng, W.; Huang, H.; Ma, C. Detective: Detecting AI-generated text via multi-level contrastive learning. Adv. Neural Inf. Process. Syst. 2024, 37, 88320–88347. [Google Scholar]
  18. Pu, J.; Sarwar, Z.; Abdullah, S.M.; Rehman, A.; Kim, Y.; Bhattacharya, P.; Javed, M.; Viswanath, B. Deepfake text detection: Limitations and opportunities. In Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA, 22–25 May 2023; pp. 1613–1630. [Google Scholar]
  19. Emi, B.N.; Spero, M.; Masrour, E. Pangram at GenAI Detection Task 3: An active learning approach to machine-generated text detection. In Proceedings of the 1st Workshop on GenAI Content Detection, Abu Dhabi, United Arab Emirates, 19 January 2025; pp. 347–351. [Google Scholar]
  20. Cohan, A.; Dernoncourt, F.; Kim, D.S.; Bui, T.; Kim, S.; Chang, W.; Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA, 1–6 June 2018; Volume 2, pp. 615–621. [Google Scholar] [CrossRef] [Scilit]
  21. Clement, C.B.; Bierbaum, M.; O’Keeffe, K.P.; Alemi, A.A. On the use of arXiv as a dataset. arXiv 2019, arXiv:1905.00075. [Google Scholar] [CrossRef] [Scilit]
  22. Pozzi, A.; Incremona, A.; Tessera, D.; Toti, D. Mitigating exposure bias in large language model distillation: An imitation learning approach. Neural Comput. Appl. 2025, 37, 12013–12029. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.