Detection of Fake News in Romanian: LLM-Based Approaches to COVID-19 Misinformation

Abstract

The spread of misinformation during the COVID-19 pandemic raised widespread concerns about public health communication and media reliability. In this study, we focus on these issues as they manifested in Romanian-language media and employ Large Language Models (LLMs) to classify misinformation, with a particular focus on super-narratives—broad thematic categories that capture recurring patterns and ideological framings commonly found in pandemic-related fake news, such as anti-vaccination discourse, conspiracy theories, or geopolitical blame. While some of the categories reflect global trends, others are shaped by the Romanian cultural and political context. We introduce a novel dataset of fake news centered on COVID-19 misinformation in the Romanian geopolitical context, comprising both annotated and unannotated articles. We experimented with multiple LLMs using zero-shot, few-shot, supervised, and semi-supervised learning strategies, achieving the best results with an LLaMA 3.1 8B model and semi-supervised learning, which yielded an F1-score of 78.81%. Experimental evaluations compared this approach to traditional Machine Learning classifiers augmented with morphosyntactic features. Results show that semi-supervised learning substantially improved classification results in both binary and multi-class settings. Our findings highlight the effectiveness of semi-supervised adaptation in low-resource, domain-specific contexts, as well as the necessity of enabling real-time misinformation tracking and enhancing transparency through claim-level explainability and fact-based counterarguments.

1. Introduction

In an era when information spreads rapidly across digital platforms, distinguishing fact from fiction has become an increasingly challenging task. The widespread dissemination of fake news and misinformation has significant consequences, as it influences public perceptions, disrupts political discourse, and contributes to public misunderstanding of critical issues such as health and security. As Tandoc Jr et al. [1] emphasized, fake news operates at the intersection of journalism and misinformation, mimicking legitimate news to mislead audiences and thereby shaping narratives with potentially far-reaching societal effects. While traditional fact-checking mechanisms play a vital role in countering misinformation, their scalability remains limited by the volume and speed at which false information circulates online. Misinformation detection becomes increasingly difficult in low-resource languages, for which there are still significantly fewer linguistic datasets and AI-driven detection tools than there are for high-resource languages, such as English. Studies indicate that the lack of annotated datasets and pretrained models in these languages hinders the development of robust misinformation-detection systems [2].

The integration of Artificial Intelligence (AI), particularly Large Language Models (LLMs), has significantly advanced efforts to address misinformation. These models, trained on vast amounts of textual data, can recognize patterns, assess credibility, and classify information with increasing accuracy. However, their effectiveness and their capacity to capture the nuances of misinformation narratives are constrained by the availability of high-quality, language-specific datasets [3]. While significant advancements have been made in detection of fake news in English, adapting LLMs to languages with complex morphologies and limited training resources, such as Romanian, presents unique computational and linguistic challenges. Research suggests that language-specific adaptation, data scarcity, and syntactic complexity hinder the direct transfer of LLM-based misinformation detection from English to low-resource languages [4].

Furthermore, there has been no comprehensive benchmarking of Romanian-trained LLMs for misinformation detection. Recent models, such as RoLlama 2 and 3, RoMistral, and RoGemma [5], have demonstrated strong performance in general Romanian NLP tasks; however, their effectiveness in classifying misinformation narratives has not been systematically assessed. This gap is particularly concerning given that misinformation narratives in Romania are heavily influenced by geopolitical, historical, and ideological factors, making them distinct from those in English-language contexts. Additionally, existing research has primarily focused on structured misinformation, such as anti-vaccine claims and health-related falsehoods, which often follow predictable linguistic patterns [6]. However, conspiratorial and politically motivated misinformation presents greater classification challenges, as it frequently combines factual statements with misleading rhetoric, making detection more complex. Additionally, most approaches to detection of fake news focus on binary classification, aiming to label news content as either true or false. However, this overlooks the variety of thematic patterns, which we refer to as super-narratives, that often characterize misinformation, particularly in the context of the COVID-19 pandemic. These include recurring framings such as anti-vaccination rhetoric, conspiracy theories, or narratives assigning geopolitical blame, which may reflect global trends or be shaped by local sociopolitical factors, as in the Romanian context. Current AI-driven efforts at detection of misinformation lack a nuanced understanding of these variations, limiting their capability to adapt to evolving misinformation strategies.

Given these challenges, this study aims to bridge the gap in Romanian-language detection of fake news by evaluating the performance of state-of-the-art LLMs in classifying misinformation across multiple categories. Specifically, the study seeks to accomplish the following:

• Introduce a dataset of 627 manually curated and annotated articles for classification of false information that reflects the cultural and linguistic adaptation of COVID-19-related narratives in the Romanian context;
• Provide a complementary dataset of 7950 unannotated articles from the same domain and period, intended to support semi-supervised learning approaches;
• Assess the effectiveness of both Romanian-trained LLMs and multilingual LLMs in distinguishing real news from various forms of misinformation;
• Analyze detection difficulties across different misinformation types, particularly comparing structured narratives (e.g., vaccine misinformation) to unstructured narratives (e.g., conspiracy theories, political misinformation);
• Examine the impact of fine-tuning strategies on improving model performance, evaluating both supervised and semi-supervised learning approaches.

Our dataset with both annotated and unannotated news articles is published on Huggingface at https://huggingface.co/datasets/upb-nlp/ro_fake_news (accessed on 7 August 2025). We have released all training and validation code under an open-source license, and they are available online at https://github.com/upb-nlp/Ro-FakeNews (accessed on 7 August 2025).

This research contributes to advancing AI-driven misinformation detection in low-resource languages by addressing these objectives. The findings provide valuable insights for fact-checking organizations, policymakers, and AI researchers developing scalable and context-aware strategies to mitigate misinformation. The study emphasizes the importance of language-specific approaches in NLP, underscoring that effective misinformation detection must consider linguistic and cultural specificities rather than rely on one-size-fits-all, uniform solutions.

2. Related Work

Methods to detect fake news are generally classified into two main categories: knowledge-based methods and style-based methods. Knowledge-based methods verify the accuracy of claims by cross-referencing external data sources, for example, using knowledge graphs or fact-checking databases [7]. However, these approaches face limitations when reliable external data are unavailable. Style-based methods, on the other hand, analyze linguistic, syntactic, and stylistic features of the text itself to identify deception, making them adaptable to various misinformation contexts [8,9].

Classical Machine Learning (ML) approaches, such as Support Vector Machines (SVM), Decision Trees, and Naïve Bayes classifiers, rely on statistical features for fake news classification. These models analyze textual attributes, including lexical diversity and syntactic complexity, to distinguish between true and false news. Castillo et al. [10] argued that ML models, including SVM, can effectively classify credible and non-credible news by leveraging a combination of linguistic and social network features. A key advancement in this field was the CSI Model (Capture, Score, Integrate) by Ruchansky et al. [11], which incorporated three essential modules: temporal representation of news, user-behavior analysis, and final content classification. Similarly, Reis et al. [12] integrated sentiment analysis into detection frameworks, highlighting how emotionally charged content plays a crucial role in manipulating public perception.

Recent research has focused on deep learning-based methods that can automatically extract high-level textual patterns. Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNNs) have been widely employed for detecting fake news. For instance, Ma et al. [13] showed that LSTMs effectively model sequential dependencies in news articles, achieving high classification accuracy. Sastrawan et al. [14] took a similar approach, experimenting with CNN, ResNet, and a bidirectional LSTM and also achieving a high accuracy. Transformer-based models, such as BERT and RoBERTa, have further improved classification performance by capturing nuanced contextual relationships. These models have been successfully adapted for stance classification, credibility assessment, and the detection of misinformation. Raza and Ding [15] employed Transformer-based models, namely BART [16], to identify false social media posts and assign a credibility level to different users. The authors utilized both the news articles and the reactions of people on social media to these articles, achieving a high benchmark for detection on their datasets, NELA-GT-2019 [17] and Fakeddit [18]. Ma et al. [19] employed LLMs, namely GPT 3.5 and LLama 2 [20], to extract topics and semantic information from news articles, then represented the relationships between news, entities, and topics as a graph. They used this graph to classify news as either fake or real, and their approach exhibited strong improvements over previous methods.

Given the increasing use of images, videos, and social media metadata in the propagation of fake news, researchers have developed multimodal detection systems that integrate textual and visual features. Zhou et al. [21] proposed a similarity-aware model that independently encodes text and image features (e.g., using Text-CNN and image encoders), computes a cross-modal similarity score to capture semantic consistency, and jointly learns from both modalities and their mismatch to classify news as fake or real. An approach using data from PolitiFact and GossipCop outperformed text-only and image-only baselines, improving accuracy by between 2 and 8%. Segura-Bedmar and Alonso-Bartolome [22] investigated the detection of multimodal false information in social posts using a broader multi-class classification with six categories: true, misleading content, manipulated content, false connection, imposter content, and satire. They used multiple deep learning models (e.g., BERT [23], CNN, and BiLSTM), achieving good performance, especially considering the broader classification task. As fake news continues to evolve, hybrid approaches integrating linguistic, behavioral, and multimodal data hold promise for enhancing the detection of fake news in diverse linguistic contexts, including Romanian.

When considering fake news in the Romanian socio-demographic context, Buzea et al. [24] developed a system for the detection of fake news in Romanian using a custom dataset of over 38,000 articles, comparing classical ML and deep learning models. The authors found that a CNN-based model achieved the best performance, with an accuracy of over 98.2%, outperforming both traditional methods and a Romanian BERT variant. The study also examined sentiment and irony features to gain a deeper understanding of linguistic patterns in fake news. Moisi et al. [25] compared classical machine learning models and Transformer-based approaches for detection of fake news in Romanian using the FakeRom dataset and a newly collected dataset. Multilingual BERT achieved the best accuracy (96.5%), with SVM also performing well (94.6%) on the combined data.

3. Method

This section is organized into several subsections. We begin by defining fake news from both linguistic and psychological perspectives, providing context in which the dataset can be introduced. We then detail the data-collection process and the construction of the dataset. Following this, we present our experimental framework, starting with a baseline that combines traditional machine learning methods and linguistic features. We then introduce the LLMs we used and outline their evaluation procedures. Finally, we describe the preprocessing and training steps specific to the LLM experiments, including tokenization, prompt design, and approach-specific considerations.

As a broad overview, we evaluated the LLM models in all possible configurations to determine the best accuracy achievable by prompting alone (zero- or few-shot). Afterwards, we performed supervised fine-tuning (SFT) on the smaller models, excluding the 70B models due to hardware limitations. In this phase, we explored all available hyperparameter configurations, excluding the few-shot options. This process ensured that we performed a comprehensive search for the best-performing configurations. Next, with the configurations that had the best fine-tuning performance, we conducted experiments using semi-supervised learning (SSL).

3.1. Defining Fake News

Fake news is broadly understood as false or misleading content designed to mimic journalism that is often spread for political, financial, or ideological motives [26,27]. However, defining fake news is challenging due to overlapping terms like misinformation (i.e., false but unintentionally spread information) and disinformation (i.e., deliberately fabricated falsehoods) [28]. At its core, the term ’fake news’ refers to fabricated or distorted information that is deliberately presented in the format and style of legitimate journalism, with the intent to mislead audiences by borrowing the appearance of credible reporting [29]. Fake news articles typically use sensationalist headlines, emotionally charged language, and misleading narratives to capture attention and influence public perception [30]. Unlike opinion pieces or satirical content, which are often framed as non-factual or entertainment, fake news articles intentionally blur the distinction between truth and falsehood [31].

Terian [32] provided a broader conceptualization of fake news, arguing that its defining feature is its perlocutionary effect (i.e., the intended impact on the audience), not its factual inaccuracy. From this perspective, fake news is successful when it persuades, misleads, or incites action, regardless of the creator’s intent. This definition highlights the audience’s role in shaping the effectiveness of fake news, emphasizing psychological susceptibility and cognitive biases as key factors in its dissemination [33].

The dissemination of fake news is often driven by cognitive and psychological factors that make individuals susceptible to misinformation. Psychological influences can be broadly categorized into cognitive biases, emotional manipulation, and social influences [34]. Cognitive biases greatly impact how individuals process and accept information. Key biases contributing to the spread of fake news include confirmation bias, the illusory truth effect, negativity bias, anchoring bias, and the availability heuristic [35]. Confirmation bias leads individuals to favor information that aligns with their pre-existing beliefs, making them more likely to accept fake news that reinforces their worldview [36]. The illusory truth effect describes how repeated exposure to false information increases its perceived truthfulness, even when individuals are aware of its falsehood [33]. Negativity bias causes people to believe and share negative information more readily, as it is perceived as more urgent or important [37]. Anchoring bias causes individuals to rely too heavily on the first piece of information they encounter, which thereby shapes their perception of subsequent news. Lastly, the availability heuristic leads individuals to judge the likelihood of an event based on how easily examples come to mind, contributing to an overestimation of the frequency of events described in fake news [38].

3.2. Dataset

To ensure a comprehensive and methodologically robust dataset, we focused on the most prominent and widely circulated fake news narratives related to the COVID-19 pandemic within Romanian media. As such, we constructed our dataset using the following methodology: using four primary keywords (i.e., “COVID-19,” “COVID 19,” “Corona Virus,” and “Coronavirus”), we retrieved articles from BuzzSumo [39], a widely used content-aggregation tool that ranks articles based on social-media engagement metrics; this approach yielded an initial dataset of 10,000 articles. These results included both reliable journalism and misinformation, ranked according to the total number of likes, shares, and comments. The articles spanned from 1 March 2020 to 16 February 2021, with a peak of activity between March and July 2020. The most frequently referenced websites included both mainstream and alternative media outlets with varying degrees of credibility. Established platforms such as Libertatea, Digi24, and Adevărul had a strong presence, but so did more sensationalist outlets like Stiripesurse and Antena 3, raising questions about how editorial biases and clickbait tactics influenced pandemic coverage. Prioritizing engagement-based selection was a strategic decision, as it ensured that the dataset reflected the most influential narratives—those that had reached and impacted large audiences, as opposed to content that remained obscure.

While BuzzSumo provided a useful starting point for identifying high-impact content, it did not distinguish between real news and misinformation. To refine the dataset, a manual verification process was conducted based on well-established criteria from misinformation research [27,40]. Articles were categorized as fake news if they met at least one of the following criteria:

• False or misleading claims contradicting scientific evidence or official health guidelines.
• Sensationalist or alarmist language designed to provoke emotional reactions rather than inform.
• Lack of credible sources, with writers often relying on anonymous experts or conspiracy theorists.
• Mimicry of journalistic conventions; using fabricated statistics, pseudo-scientific jargon, or misleading headlines to enhance credibility.

Beyond initial thematic markers, this study identified a total of nine dominant super-narratives that structured the COVID-19-related misinformation ecosystem. Each super-narrative contains multiple sub-narratives, contributing to the ideological diversity and persuasive depth of pandemic-related fake news:

• Vaccination Narratives—These sub-narratives challenge the safety, efficacy, or legitimacy of vaccines;
• Accusations of Authoritarianism and Dystopia—Public health measures are recast as precursors to totalitarian control;
• Criticism of National and International Actors—Discredits political and institutional responses;
• Criticism of Restrictions—Opposes mitigation efforts such as lockdowns and mask mandates;
• Geopolitical Narratives—Link the pandemic to global rivalries and blame those rivalries for negative effects;
• Minimization of COVID-19 Impact—Downplays the pandemic’s severity;
• Health-Related Falsehoods and Cultural Claims;
• Conspiracy Theories—Suggest orchestrated manipulation by hidden elites;
• Satire—Though minimal in the dataset, some content was framed using irony or exaggeration, making intent and impact difficult to assess.

To gain deeper insights into what distinguishes fake news from credible journalism, a control dataset of 255 real news articles was compiled. These articles were selected based on their credibility, adherence to journalistic standards, and reliance on official sources such as the World Health Organization (WHO) and the Romanian Ministry of Health. The majority of the selected articles were from established media outlets (e.g., Libertatea, MediaFax, Digi 24, Adevarul, G4 Media), but we identified veridical information also from smaller outlets (e.g., Capital, RoVestea, Descopera.ro). All annotations were performed by a qualified linguist, who examined the lexical and semantic characteristics of fake news media and cross-referenced them with official sources. The process was conducted rigorously and with close attention to detail, with annotation choices informed by the annotator’s expertise.

This comparative approach enables a detailed analysis of the linguistic and rhetorical differences between fake and real news. By contrasting lexical choices, syntactic structures, and persuasive strategies, the study identifies key elements that contribute to the persuasive power of misinformation and explores why certain super-narratives gain traction despite lacking factual accuracy. This classification builds upon existing frameworks, with its unique contribution lying in the cultural and linguistic adaptation of narratives to the Romanian context. This ensures that misinformation is classified thematically and contextually, taking into account Romania’s sociopolitical history, cultural beliefs, and patterns of public trust.

The final dataset consisted of 627 articles, of which 372 were categorized as various types of fake news and 255 were classified as real news; see Figure 1. Each fake news article was annotated with a super-narrative and a sub-narrative based on its content, except articles under the “Satire” super-narrative, which were not assigned a sub-narrative. The real news articles were annotated only with the “Real News” super-narrative. We will continue to refer to the super-narratives as classes and the sub-narratives as subclasses. The dataset was partitioned into training (60%), validation (20%), and testing (20%) sets using a stratified splitting method based on class labels. This stratification was performed to ensure that each subset maintained the same class distribution as the original dataset.

To further improve this dataset, we also filtered the remaining collected articles using the normalized Levenshtein distance to ensure that soft duplicates, especially those related to the annotated subset, were eliminated. With the term ’soft duplicates,’ we refer to slightly modified versions of the same articles. While not annotated, these datasets can still provide valuable information through techniques such as semi-supervised learning, where a model utilizes both annotated and unannotated data to enhance its performance on a task. As such, we introduce an additional 7950 unannotated samples that can be used alongside the previous 627 annotated ones. This sample of 7950 articles is what remained after removal of duplicates and soft duplicates based on the articles’ titles and content. With the term ‘duplicates,’ we refer to perfect duplicates, whereas with the term ’soft duplicates,’ we refer to duplicates that are not perfect, but have small discrepancies that can be easily explained through the crawling process. To identify soft duplicates, we used the normalized Levenshtein distance for each article pair [41]. These can aid the learning process by allowing the use of semi-supervised techniques.

3.3. Baseline: Classical ML Based on Linguistic Features

First, we investigated how a classical ML model would fare in both binary and multi-class classification settings, establishing a baseline for subsequent comparisons.

To achieve this, we considered ReaderBench, an advanced open-source multilingual textual-analysis platform that provides unified access to over 200 textual complexity indices. As an NLP-driven framework, ReaderBench facilitates a multi-layered analysis of linguistic structures, enabling a fine-grained examination of how language operates in the service of persuasion and manipulation [42]. Linguistic features play a critical role in the identification of fake news, as evidenced in [43,44]. Considering this, ReaderBench enabled us to investigate fake news discourse across multiple linguistic dimensions, from surface-level features to deeper semantic and discourse structures using Transformer-based approaches. Readerbench offers textual indices for surface-level complexity, syntactic complexity, morphological complexity, word-level features, and discourse-level features.

In such tasks, SVM, XGBoost, or Random Forest are often selected as baselines because they achieve good performance with modest hardware requirements [43,44]. We tried all three methods for the classical ML model, and we evaluated them with three different feature sets: (a) using all the indices, (b) applying PCA to reduce the feature set to keep 95% of the total variance, (c) using the Mann–Whitney U test to keep only the significant features. Since the first two variants do not require further explanation, we will provide additional details only for the third. The Mann-Whitney U test determines whether two samples are from the same distribution. Using the training set, we evaluated, for each textual index, the likelihood that samples from the fake class and samples from the true class are from the same distribution. As such, we used as input features only those indices that the test suggested (according to the p-value) were from different distributions.

3.4. Tokenization, Prompt Structure and Hyperparameters

Going forward to the LLM-based approaches, to enable a model to process text, the text must first be tokenized. Tokenization is a process that transforms the text into a sequence of vocabulary tokens, each represented by a unique numerical identifier. However, news articles are often lengthy, as illustrated in Figure 2 and

Table 1. Token statistics in the annotated set.

Count	Mean	St. Dev	Min	25%	50%	75%	Max
290	1243.23	1409.20	41.00	509.50	840.50	1448.25	12,177.00

. Given that LLMs have a maximum input sequence of 4096 tokens, it can be challenging to input entire articles into the model, particularly in few-shot learning contexts. Moreover, it is essential to include the necessary information in the prompt to ensure that the model clearly understands the task expectations.

Due to the 4096-token input limit, we needed to split the articles and skip certain parts to prevent the prompt from exceeding this maximum length. Several pruning strategies were employed to fit the body of the article within the token limit, excluding the space reserved for the task description and headline of the articles, which we always want to include:

• Start–include as much as possible from the start;
• End–include as much as possible to the end;
• 1/2-1/2–split the remaining space in half from the beginning and half from the end;
• 1/3-2/3–split the remaining space into three parts, one from the beginning and two from the end;
• 2/3-1/3–split the remaining space into three parts, two from the beginning and one from the end.

For our few-shot experiments, the model requires several examples accompanied by their respective class and subclass labels to enhance its understanding of the task. We experimented with various few-shot settings: zero-shot, one-shot, three-shot, and “all”, where the latter setting involved providing the model with all the examples from the precomputed list (a total of 10 examples). These configurations allowed us to evaluate the model’s performance across different levels of contextual guidance.

We adopted the following methodology to allocate the appropriate token length to each news article. For all few-shot variants, we first subtracted the tokenized length of the task description from the 4096-token limit. We then divided the remaining token space by the total number of articles included in the prompt (i.e., two articles for one-shot, four articles for three-shot, and eleven articles for the “all” setting, including the current article). However, we applied the following token limits based on the few-shot setting:

• For one-shot and three-shot variants: Each few-shot example could use up to 768 tokens, while the current article received the remainder of the allocated space.
• For the “all” setting: Each few-shot example was allocated a fixed length of 200 tokens, while the current article received the remainder of the available space.

This approach ensured consistent token distribution while maximizing the content provided to the model across different few-shot configurations.

We also experimented with providing descriptions for each class and its subclasses. As such, the total combination of possible hyperparameter settings was forty: five from the truncation strategies multiplied by four from the few-shot strategies, multiplied by two from including or excluding descriptions. Considering all these options, the model would have received one of the following four prompts:

• In the zero-shot setting:

  ∘

  [Task Introduction] [Classes w/o Class Description] [Task Description] [Output Example] [Input]

  ∘

  [Task Introduction] [Classes w/Class Description] [Task Description] [Output Example] [Input]

• In the few-shot setting:

  ∘

  [Task Introduction] [Classes w/o Class Description] [Task Description] [Output Example] [Few-Shot Examples] [Input]

  ∘

  [Task Introduction] [Classes w/Class Description] [Task Description] [Output Example] [Few-Shot Examples] [Input]

Additionally, all our experiments were run with a fixed seed (i.e., 42), and for all predictions, we set the temperature to 0, ensuring deterministic outputs for a given input. All of our training experiments were conducted on a single NVIDIA A100 with 80 GB of VRAM, for a maximum of 10 epochs. The actual number of epochs varies depending on the experiment, as we used early stopping on the validation loss to detect when an experiment was not improving for more than three epochs. Each experiment saved the best checkpoint according to the validation loss. We used the AdamW8Bit optimizer with a learning rate of $5\times {10}^{-6}$ and a cosine scheduler with a minimum learning rate of $5\times {10}^{-7}$ (i.e., the learning rate gradually decreases following a cosine curve down to the minimum). Initially, for the fine-tuning experiments, we investigated multiple learning rates, but this offered the most desired learning curve.

3.5. Models and Scoring

Since distinguishing between real news and fake news is challenging, especially when it also requires identifying the type of fake news, we aimed to evaluate the performance of state-of-the-art LLMS on our dataset. Towards this end, we chose two LLMs specially designed for the Romanian language (RoLlama 3.1 and RoMistral [5]), as well as their original counterparts. These have shown superior performance on Romanian tasks for zero-shot, few-shot, and fine-tuning settings compared to their original versions [45,46], as outlined by Masala et al. [5]. We also aimed to use a larger Llama model, specifically Llama 3.1 70B. However, due to the hardware requirements of a 70B model, we were able to employ it only in the zero-shot setting. To provide a fairer comparison, we opted for the newer Llama 3.3 70B instead. We compared the performance of the LLMs both against each other and against the baseline classical ML model.

We used the F1-score with macro and weighted-average options for the evaluation metrics. The macro option assigns equal importance to each class, while the weighted option assigns an importance proportional to the number of samples of that class in the dataset. For the initial experiment, we evaluated four models across all 40 variations to identify the optimal configuration for each model type. Following this evaluation, we proceeded to train the models using both supervised and semi-supervised learning approaches.

To facilitate easy comparison between the LLM models and the baseline model without requiring additional resources, we simply consider any class except “Real News” as “Fake News” for the binary comparison, instead of retraining our best LLM model with a prompt for binary classification.

3.6. Zero-Shot, Few-Shot and Supervised Fine-Tuning

The zero-shot experiments do not require additional explanation, as they consisted solely of running the pretrained models, using the prompt variants, and observing the best performance that can be obtained.

The few-shot experiments require additional information, namely, how the few-shot examples were chosen. To facilitate the retrieval of the few-shot examples, we encoded all the articles (i.e., title and body) using a sentence-embedding Transformer model (i.e., “paraphrase-multilingual-mpnet-base-v2”) [47] and computed the cosine similarity between each sample and all samples within the training set. Although few-shot samples are often selected randomly, we employed a deterministic selection strategy based on their semantic similarity or “closeness” to each target sample. Specifically, for each sample, we identified and stored the most similar examples from each class in a dedicated list. As a result, whenever few-shot samples were required for model input, we could retrieve them directly from each sample’s precomputed list. This approach enables the consistent and efficient selection of relevant examples while maintaining flexibility for both random and deterministic sampling methods.

For the supervised fine-tuning experiments, in an approach similar to that used in the zero-shot experiment, we did not provide examples in the prompt; however, the rest of the hyperparameters remained unchanged. Additionally, we introduced new hyperparameters, including the learning rate and its scheduler, but aside from these, the methodology remains straightforward.

3.7. Semi-Supervised Learning

For this approach, we employed an adaptation of the FixMatch algorithm [48], traditionally used for images, modified for text data.

In FixMatch, a training example is augmented in two different ways. The input sample is first subjected to a weak augmentation, resulting in a prediction that will be used to determine a pseudo-label, provided it exceeds a predefined confidence threshold. Additionally, the same sample undergoes a strong augmentation, which serves as a consistency-regularization mechanism to align the prediction with that of the weakly augmented counterpart. To adapt this to our scenario, we introduced two main modifications.

For the augmentations, we employed Easy Data Augmentation (EDA) [49] with word-level granularity, processing 10% of words for the weak augmentation and 20% for the strong one. We experimented with back-translation for strong augmentations. Unfortunately, we encountered several significant challenges, including notably poor translation accuracy and substantially reduced training speed resulting from the back-translation process; consequently, we proceeded with the experiments using only EDA.

Semi-supervised learning typically relies on encoder-based models with classification heads, enabling consistency regularization via direct cross-entropy between pseudo-labels and model outputs. Adding a classification head to LLMs could enable this mechanism, but it would conflict with their decoder-based architecture and native generation paradigm. As such, we modified the approach as follows. We generate a classification response from the model using the weakly augmented example and compute the probability of the generated sequence. If this probability exceeds the predefined threshold, the sample is retained for use in computing the unsupervised loss. This loss is computed with a new sample that incorporates the strongly augmented input with the previously generated classes concatenated at the end, mimicking the SFT format. We can then compute the loss on the newly created sample. We experimented with several confidence thresholds for pseudo-labeling: 0.8, 0.85, 0.9, 0.925, and 0.95.

4. Results

We begin by presenting the results of the LLM-based models, as they represent the core focus of our study. The baseline results are discussed at the end of the section to provide a clear and structured comparison to the strongest LLM configuration. For a detailed exploration of the impact of each hyperparameter, please refer to Appendix A.

The results of our LLM experiments are summarized in

Table 2. Results super-narratives.

Scenario	Super-Narrative	RoMistral	Llama 3.1 8B	RoLlama 3.1 8B	Llama 3.3 70B
Few-Shot	Vaccine-related narratives	0.00%	10.00%	8.33%	50.00%
	Claims of authoritarianism and dystopia	19.05%	80.00%	0.00%	28.57%
	Conspiracy theories	66.67%	30.77%	46.15%	69.57%
	Criticism of restrictions	50.00%	53.33%	50.00%	66.66%
	Criticism of the national and international actors	57.14%	0.00%	0.00%	0.00%
	Downplaying COVID-19	0.00%	61.54%	44.44%	62.50%
	Geo-politics	57.14%	44.44%	0.00%	57.14%
	Health-related narratives	50.00%	43.33%	61.90%	82.05%
	Satire	66.67%	26.67%	26.67%	70.00%
	Real news	63.16%	33.85%	59.74%	76.29%
	F1 Weighted	62.24%	39.36%	48.21%	70.49%
	F1 Macro	45.34%	38.39%	29.72%	56.28%
	F1 Micro	58.73%	38.89%	46.03%	68.25%
SFT	Vaccine-related narratives	80.00%	80.00%	80.00%	-
	Claims of authoritarianism and dystopia	0.00%	40.00%	33.33%	-
	Conspiracy theories	62.50%	42.86%	53.33%	-
	Criticism of restrictions	50.00%	44.44%	44.44%	-
	Criticism of the national and international actors	0.00%	0.00%	0.00%	-
	Downplaying COVID-19	58.82%	52.63%	55.81%	-
	Geo-politics	57.14%	85.71%	85.71%	-
	Health-related narratives	80.00%	85.71%	72.73%	-
	Satire	89.66%	84.62%	85.71%	-
	Real news	80.37%	81.08%	77.67%	-
	F1 Weighted	71.88%	71.96%	69.62%	-
	F1 Macro	55.85%	59.71%	58.88%	-
	F1 Micro	73.02%	73.02%	69.84%	-
SSL	Vaccine-related narratives	50.00%	80.00%	80.00%	-
	Claims of authoritarianism and dystopia	40.00%	40.00%	33.33%	-
	Conspiracy theories	62.50%	58.82%	40.00%	-
	Criticism of restrictions	40.00%	60.00%	44.44%	-
	Criticism of the national and international actors	0.00%	0.00%	0.00%	-
	Downplaying COVID-19	60.60%	63.15%	58.82%	-
	Geo-politics	80.00%	75.00%	75.00%	-
	Health-related narratives	83.33%	92.30%	85.00%	-
	Satire	91.67%	91.67%	91.67%	-
	Real news	84.68%	86.53%	82.71%	-
	F1 Weighted	75.31%	78.81%	71.74%	-
	F1 Macro	59.27%	64.74%	58.10%	-
	F1 Micro	76.19%	79.36%	73.01%	-

Bold marks the best results. While the weighted, macro, and micro F1 rows are self-explanatory, the values in the rest of the rows are per-class F1 scores.

. One of the most consistent findings in this study is the high accuracy with which vaccine misinformation was classified. SFT and SSL models perform exceptionally well in this category, with scores exceeding 80% across multiple architectures. The models’ capability to classify vaccine misinformation with high accuracy suggests that these narratives exhibit a rigid linguistic structure, making them highly detectable by ML models. The discourse surrounding vaccine misinformation often relies on repetitive lexical patterns, including highly emotive language, pseudo-scientific terminology, and alarmist framing.

The consistently high classification scores (i.e., above 80% across multiple models) indicate that these syntactic and semantic patterns are well-learned by LLMs, suggesting that vaccine misinformation narratives rely on lexical and grammatical predictability rather than nuanced persuasion techniques. Even when slight classification variations, such as SSL RoMistral’s score of 50%, occur, the overall performance suggests that semi-supervised learning reinforces a model’s ability to detect structured misinformation discourse, despite minor inconsistencies.

Compared to vaccine misinformation, conspiracy theories exhibit a more heterogeneous linguistic structure, characterized by intertextuality, ambiguity, and discourse flexibility. Conspiratorial narratives frequently blend factual elements with speculation, constructing a linguistic landscape that resists rigid categorization. The classification challenge stems from the interplay of hedging, evidentiality, and speculative language, which allow conspiracy discourse to mimic critical inquiry while subtly advancing misleading claims.

For instance, conspiracy theories commonly employ modal verbs and epistemic markers to create plausible deniability (e.g., “Some experts believe that the government may be covering up the real cause of the pandemic”). Unlike vaccine misinformation, which asserts claims with epistemic certainty, conspiracy narratives weave possibilities rather than absolute statements, making them less predictable in structure and thus harder to classify.

Political misinformation poses the greatest linguistic challenge due to its reliance on ideological framing, metaphorical constructs, and the selective omission of context, rather than outright falsehoods. Unlike vaccine misinformation, which often employs scientific-sounding jargon, or conspiracy theories, which lean on speculative reasoning, political misinformation operates at a discursive level, utilizing rhetorical persuasion, framing devices, and strategic ambiguity. The generally low scores in classifying authoritarianism narratives highlight how supervised and semi-supervised models struggle with misinformation that seamlessly integrates with ideological discourse. By contrast, Few-Shot Llama 8B achieves much higher scores of 80%, suggesting that pretrained models have internalized certain linguistic markers associated with political rhetoric, and that their performance could be reduced if they were trained without sufficient examples. However, the high variability in classification within this category suggests that political misinformation is not a monolithic construct, but rather an evolving linguistic phenomenon that depends on cultural, historical, and contextual interpretations.

The consistent misclassification of articles for the “Criticism of national and international actors” super-narrative likely stems from rhetorical similarities with the “Downplaying COVID-19” class, as evidenced in Figure 3. Although conceptually distinct, both categories often employ similar discursive strategies, such as questioning the credibility of authorities, using skeptical or delegitimizing language, and framing public health measures as exaggerated or politically motivated. These shared linguistic patterns may have led the models to conflate the two, especially given the small number of training examples for the “Criticism” super-narrative.

The results from the SSL experiments indicate moderate performance improvements (6%) compared to the supervised models. This suggests that exposure to a larger set of diverse examples, even without annotations, contributes meaningfully to the model’s learning process. However, even SSL struggles to differentiate between persuasive political critique and misleading rhetoric, highlighting the need for models to integrate discourse-pragmatic elements such as argument structure, presupposition, and pragmatic implicature to improve classification accuracy.

Beyond misinformation detection, our study reveals a fundamental challenge in distinguishing between real news and misinformation, particularly in cases where rhetorical strategies overlap. While real news adheres to journalistic conventions such as factual verification, attribution, and balanced argumentation, it may still contain linguistic markers that resemble misinformation, particularly in political reporting and editorial pieces. Misclassification tends to occur when real news incorporates sensationalist framing, emotionally charged lexis, or speculative phrasing, blurring the boundary between factual reporting and misinformation discourse.

Returning to a more general perspective, in our experiments, the best-performing model was Llama 3.1 8B in the semi-supervised setting. Considering the mistakes highlighted in the confusion matrix shown in Figure 3, it is important to note that the majority of the model’s errors involve false-information classes being confused with other false-information classes (e.g., conspiracy theories that equate authoritarianism and dystopia, or downplaying the severity of COVID-19). These classes have similar linguistic structures and rhetoric, which can explain the confusion of the model. However, it is undeniable that the confusion between false information and genuine news remains minimal. This is a promising finding, as the ability to discern whether information is accurate or misleading is more critical than the ability to precisely categorize the specific type of misinformation presented in a fake news article.

The results of the comparison of our best model with the classical ML model using linguistic features are summarized in

Table 3. Comparison of best model with classical ML model using linguistic features.

Setting	Super-Narrative	SVM All	SVM + PCA	SVM + MW	SSL Llama 3.1 8B
Binary	Fake News	74.70%	75.13%	70.73%	90.54%
	Real News	51.16%	10.91%	45.45%	86.54%
	F1 Weighted	65.17%	49.13%	60.50%	88.92%
	F1 Macro	62.93%	43.02%	58.09%	88.54%
	F1 Micro	66.66%	61.11%	61.90%	88.88%
Multi-Class	Vaccine-related narratives	0.00%	0.00%	33.33%	80.00%
	Claims of authoritarianism and dystopia	0.00%	33.33%	0.00%	40.00%
	Conspiracy theories	11.76%	10.53%	10.00%	58.82%
	Criticism of restrictions	0.00%	0.00%	0.00%	60.00%
	Criticism of the national and international actors	0.00%	0.00%	0.00%	0.00%
	Downplaying COVID-19	23.53%	23.53%	40.00%	63.15%
	Geo-politics	0.00%	0.00%	22.22%	75.00%
	Health-related narratives	34.15%	22.22%	38.89%	92.30%
	Satire	78.26%	80.00%	78.57%	91.67%
	Real news	55.67%	55.77%	47.52%	86.53%
	F1 Weighted	39.86%	38.89%	40.54%	78.81%
	F1 Macro	20.34%	22.54%	27.05%	64.74%
	F1 Micro	38.10%	38.89%	40.48%	79.36%

Bold marks the best results. While the weighted, macro, and micro F1 rows are self-explanatory, the values in the rest of the rows are per-class F1 scores.

. Out of the three classical ML methods, SVM obtained the best results by a small margin, achieving an F1 score of 65.17% for all features and an F1 score of 60.50% after filtering with Mann–Whitney scores. In contrast, the weighted F1 scores for binary classification using Random Forest were 65.17% for the version using all linguistic features and 59.84% for the one using Mann–Whitney, resulting in a decrease of approximately 0.15%, while XGBoost (xgboost python package, version 3.0.4) achieved a weighted F1 of 62.64% with all features and a 62.86% after the Mann–Whitney filtering, resulting in a decrease of approximately 0.13%. Consequently, we present only the results obtained with SVM; nonetheless, all of the classical ML methods are far behind our semi-supervised learning approach, especially in the multi-class setting. While hyperparameter tuning could certainly improve performance, we consider it highly improbable that it would surpass the LLM. The textual indices alone are insufficient for distinguishing between false and true information, as evidenced by its substantially lower performance compared to the SSL LLaMA model. This performance gap is observed in the binary classification setting, with a decrease of approximately 30%, and is even more pronounced in the multi-class setting, where performance is reduced by around 39%. The overlap in rhetorical style and lexical choices, particularly in politically charged or editorial content, limits the discriminative power of surface-level textual indices. Accurate classification, therefore, requires a deeper understanding of contextual cues and factual grounding. This further reinforces the idea that relying solely on linguistic features and statistical patterns in the text is insufficient for reliably distinguishing between real and false information, especially in the absence of a detailed analysis identifying the linguistic features that most reliably differentiate real news from fake news rhetoric.

Lastly,

Table 4. Results sub-narratives.

Scenario	Model	F1 Weighted	F1 Macro	F1 Micro
Few-Shot	RoMistral	58.51%	36.21%	55.56%
	Llama 8B	33.94%	31.51%	30.95%
	RoLlama 8B	44.62%	31.55%	42.06%
	Llama 70B	61.67%	38.18%	61.11%
SFT	RoMistral	65.16%	35.22%	68.25%
	Llama 8B	65.59%	46.09%	69.05%
	RoLlama 8B	64.18%	38.60%	65.87%
SSL	RoMistral	71.35%	43.79%	73.81%
	Llama 8B	73.73%	50.96%	76.19%
	RoLlama 8B	67.18%	41.01%	69.84%

Bold marks the best results.

presents a summary of our best model for classifying sub-narratives; as such, it is dependent on the accuracy of predicting the correct super-narrative. While the accuracy is, on average, approximately 7% lower than that achieved in super-narrative classification, the results remain strong. These findings support the conclusion that a semi-supervised approach applied to curated data can yield substantial improvements in performance in classifying fake news.

5. Discussion

The findings of this study highlight both significant progress and remaining challenges in misinformation classification. The high accuracy in detecting vaccine misinformation demonstrates that AI models are becoming highly effective at identifying structured misinformation. Since vaccine-related falsehoods follow repetitive linguistic structures, models can recognize these claims with increasing precision. The strong performance of SFT and SSL models in this category suggests that structured misinformation is well-suited for current AI-driven detection methods. This success indicates that future efforts at misinformation detection should prioritize categories of structured misinformation first, ensuring that AI models can quickly and reliably flag well-documented falsehoods before tackling more complex forms of misinformation.

Beyond misinformation detection, a crucial consideration is ensuring that real, factual news is accurately classified. AI-based misinformation detection must differentiate between misleading narratives and legitimate journalism, preventing false positives that could undermine media credibility. The findings suggest that misclassification occurs when real news includes speculative framing or discusses controversial topics that resemble misinformation narratives. This is particularly relevant in political news, where the distinction between factual reporting, editorial perspectives, and misinformation is often subtle.

To improve classification of real news, future models should incorporate a refined understanding of journalistic discourse, including fact-based reporting conventions, citation practices, and editorial standards. This kind of judgment often depends on background knowledge, verification against external sources, or recognition of source credibility—elements that are not readily captured through unsupervised or purely language-based models. As such, improved performance may depend on supervised approaches trained on curated datasets that include factual annotations, or on models exposed to large-scale, diverse examples that enable them to internalize distinctions beyond language form alone. Training models on high-quality, fact-checked news sources will help ensure greater accuracy in distinguishing between factual reporting and misleading claims that rely on similar linguistic structures. These improvements will ensure that misinformation-detection systems are not only efficient at identifying falsehoods but also robust in recognizing credible journalism, thereby maintaining public trust in AI-assisted fact-checking tools.

While semi-supervised learning has demonstrated clear advantages, further refinements in contextual understanding, real-time discourse tracking, and improved classification of political misinformation are necessary. The ability to integrate real-time misinformation updates, track narrative shifts, and distinguish between legitimate and misleading rhetoric will be essential in developing next-generation misinformation-detection models. Ultimately, this study reinforces that misinformation detection is improving, particularly for categories of structured misinformation, such as vaccine-related falsehoods. However, as misinformation evolves, detection models must also evolve, incorporating continuous learning mechanisms, deeper contextual understanding, and improved classification of real news. The findings suggest that future misinformation-detection frameworks should prioritize adaptability, ensuring long-term reliability in identifying both misinformation and credible journalism.

Consequently, we believe it is important to recognize the limitations of our study. We believe that the main limitations can be categorized into three primary areas. The first is related to language and cultural specificity. Our analysis focused on rhetorical patterns within Romanian-language content, using a dataset specifically curated and annotated for this linguistic and cultural context. As a result, the generalizability of our approach to other languages is likely limited. Adapting the methodology would require a comparable process of data collection and annotation tailored to the target language and cultural setting. While it is plausible that narrative structures across languages may be similar, this assumption has not been investigated.

Second, a key limitation of the current implementation is its offline nature. Given the dynamic and continuously evolving nature of news content, the approach lacks mechanisms for real-time updating and integration of emerging information. To address this, a more effective solution would involve embedding our training methodology within an online system that can identify new topics and evolving patterns in misinformation.

Finally, current misinformation-detection models often focus solely on classification (i.e., labeling content as true or false). However, effective misinformation mitigation requires more granular analysis, specifically, the identification of the precise claims within an article that are false, along with evidence-based counterarguments. Without this level of transparency, users may be reluctant to trust automated predictions, even when model accuracy is high. Therefore, even if our approach to identifying false information is objectively useful, it is undeniable that enhancing the interpretability and explainability of the system is mandatory.

An additional noteworthy finding is that the original Llama 3.1 model achieved higher performance with more specialized fine-tuning (e.g., SFT and SSL scenarios). While the RoLlama model surpassed Llama 3.1 in zero- and few-shot evaluations, its relative performance declined in both SFT and SSL settings. This is rather unexpected, considering the results of the few-shot experiments, as well as those reported by Masala et al. [5], where Romanian-adapted models generally exhibited superior outcomes after fine-tuning compared to their base versions. One plausible explanation is that although RoLlama is linguistically and culturally better aligned with Romanian, the detection of complex forms of misinformation in our dataset may rely more heavily on general-world knowledge and reasoning capabilities that are better preserved in the original Llama 3.1 model due to its large-scale multilingual pretraining. In contrast, the RoModels, while initialized from the same base checkpoint, did not undergo additional pretraining on Romanian corpora and were exposed to Romanian only through instruction fine-tuning and Direct Preference Optimization [50]. This limited exposure may have improved task-specific alignment, but at the cost of weakening broader semantic representations. Additionally, while RoLlama achieved higher weighted and micro F1 scores, its macro F1 score was lower. RoLlama was more effective at predicting the majority class (real news, which dominates the dataset), but less consistent on smaller, individual classes of fake news. These results highlight a potential trade-off between linguistic localization through instruction tuning and the retention of more generalizable, cross-lingual capabilities, which are crucial for handling complex and imbalanced classification tasks.

The challenges outlined in our study become even more pressing in light of recent developments in social-media regulation. The decision by Meta (formerly Facebook) to reduce its reliance on third-party fact-checkers raises profound questions about the future of misinformation detection in digital spaces, particularly in environments where AI-driven moderation will play an increasingly dominant role. Our findings underscore that LLMs, despite their improvements, are not yet equipped to fully replace human fact-checking, particularly in categories that require deep contextual understanding, discourse awareness, and analysis of ideological framing, such as misinformation.

Ultimately, our study underscores that as misinformation-detection models continue to evolve, their effectiveness will depend not only on improvements in machine learning but also on their integration within a broader ecosystem of fact-checking, editorial review, and public literacy initiatives. The shift away from human fact-checking presents both risks and opportunities: while AI models may provide scalability, they must also be continuously refined to ensure that misinformation narratives are not misclassified as legitimate discourse, and vice versa. As digital information environments evolve, research on misinformation detection should strive towards a balance between automation and interpretability, ensuring that classification systems maintain both accuracy and contextual relevance.

6. Conclusions

This study presents a comprehensive framework for classifying COVID-19-related misinformation in Romanian-language media, emphasizing the value of combining curated datasets with supervised and semi-supervised learning approaches. Our results indicate that categories of structured misinformation, such as vaccine-related narratives, can be accurately detected by LLMs, especially when these tools are enhanced through semi-supervised learning. The best-performing configuration, using SSL with LLaMA 3.1 8B, which achieved a precision of 79.31%, a recall of 79.36%, a weighted F1 score of 78.81%, a macro F1 score of 64.36%, and a micro F1 score of 79.36% when using the set seed 42, and when trained with five random seeds, it achieved an average weighted F1 of 74.01% ± 0.0413 (95% CI), which reflected significant outperformance of traditional ML baselines based on linguistic features, particularly in both binary and multi-class classification tasks.

While classification performance was strong in most super-narratives, political and conspiratorial misinformation proved more challenging due to their rhetorical variability and contextual dependence. Additionally, differentiating between real news and misinformation remains a non-trivial task when linguistic features overlap.

Future work should prioritize four key directions to advance the effectiveness and applicability of misinformation-detection systems. First, expanding the linguistic and cultural coverage of training data and model architectures is essential for improving generalizability across diverse languages and regional contexts. This involves not only multilingual data collection, but also culturally aware annotation guidelines and language-specific adaptations in model design. Second, the development of online, adaptive models is crucial for enabling real-time detection of emerging misinformation. Such models should incorporate continual learning mechanisms and robust drift detection to dynamically adjust to evolving narratives without extensive retraining. Third, incorporating claim-level explainability and generating fact-based counterarguments can significantly enhance model transparency and user trust. This includes integrating interpretable components (e.g., attention visualization, rationales) and leveraging external knowledge sources (such as verified databases or retrieval-augmented generation) to provide grounded, human-interpretable justifications for predictions. Fourth, further investigation is needed into the types of information that can be reliably extracted from linguistic indices for distinguishing between real and fake news. This involves identifying which linguistic markers—such as lexical choices, syntactic patterns, or stylistic cues—are most informative for this task and clarifying the specific kinds of signals these indices capture. Understanding what aspects of language are effectively represented in these markers is essential for determining their role in misinformation-detection pipelines and for identifying where additional sources of information or complementary modeling approaches are required.