Incorporating Linguistic Normalization in Croatian NLP: Evaluating the Impact of Lemmatization on Disinformation Detection Performance

Abstract

Detecting disinformation in morphologically rich and under-resourced languages remains a significant challenge in natural language processing. This paper examines the role of lemmatization as a preprocessing strategy for disinformation detection in Croatian. Building on previous work and extending the evaluation to a newly collected dataset of nearly 25,000 social media comments, we systematically compare traditional machine learning classifiers (SVM, Random Forest, and neural networks) and a transformer-based model (croBERT) on both original and lemmatized text. Our findings demonstrate that lemmatization does not produce uniform gains across architectures: while linear models and croBERT display small but measurable improvements from morphological normalization, non-linear models such as RBF SVM and neural networks experience substantial declines in performance. These results indicate that lemmatization interacts differently with model inductive biases and feature extraction mechanisms. Overall, the study provides a detailed empirical assessment of preprocessing choices for low-resource, morphologically complex languages and offers practical guidance for developing disinformation detection systems in Croatian and similar contexts.

1. Introduction

Croatian language content is commonly produced and shared across digital platforms, including social media and online news portals. Croatian, like many morphologically rich and low-resource languages, remains significantly underrepresented in natural language processing research. This scarcity of language-specific resources and models has resulted in a limited understanding of how best to process, normalize, and model Croatian text, particularly in high-stakes domains such as online disinformation detection. Our previous work [1] addressed part of this gap by introducing a novel framework for identifying potentially misleading content in Croatian social media, focusing on the intersection of disinformation and hate speech. That study demonstrated the feasibility of automated disinformation detection using various machine learning methods, while also highlighting the many challenges posed by non-standard orthography, informal register, and high morphological variability in user-generated content.

Social media platforms amplify misinformation due to their participatory nature, ease of content sharing, and algorithmic prioritization of emotionally charged or polarizing messages [2]. These conditions are also present in Croatian online media, where unmoderated comment sections often serve as vectors for misinformative narratives targeting topics such as immigration, identity, and public policy. Automated systems designed to mitigate such risks must therefore account for the linguistic properties of Croatian while remaining resilient to the irregularities and noise characteristic of social media discourse. Developing reliable systems for detecting disinformation in social media requires both effective modeling strategies and linguistically informed text representations.

In morphologically rich languages such as Croatian [3], a single lexical item can appear in numerous surface variants depending on case, number, gender, tense, or aspect. For example, the Croatian verb govoriti (“to speak”) may appear as govorim (“I speak”), govore (“they speak”), or govorila (“she spoke”), all representing the same underlying lemma. Such variation substantially increases vocabulary size and introduces sparsity into textual datasets, hindering the performance of models that rely on token-level features and dramatically extending the processing time. Introducing lemmatization can mitigate this problem by mapping inflected forms to a unified lexical representation (e.g., govoriti), thereby reducing lexical variability and improving feature consistency across contexts. Recent studies have shown that morphology-aware normalization can alleviate lexical sparsity, reduce ambiguity, and improve downstream NLP tasks by decreasing surface-form variability and producing more compact lexical representations in morphologically rich languages [4,5].

Building on our earlier work, which introduced a framework for detecting disinformation in Croatian social media comments, this study represents the next stage of its refinement. While the initial framework effectively addressed challenges related to orthographic variation and non-standard writing typical of social platforms, it did not explicitly account for the morphological complexity of the Croatian language. In Croatian, a single lexical item can appear in numerous surface forms depending on grammatical context, which can fragment statistical representations and hinder model generalization.

To address this limitation, the present study incorporates lemmatization into the preprocessing workflow of the existing framework. By normalizing inflected word forms to their canonical lemmas, we aim to reduce data sparsity and improve the representational consistency of textual features, particularly for traditional machine learning models. This refinement is motivated by previous findings in morphologically rich languages, where lemmatization has been shown to enhance classification metrics such as the F₁-score by consolidating lexical variants. With this extension, we seek to evaluate whether similar improvements can be achieved for Croatian disinformation detection and to assess how morphological normalization interacts with different model families, from Support Vector Machines (SVM) and Random Forests (RF) to transformer-based architectures such as croBERT [6].

This paper investigates a broader question that remains insufficiently understood in NLP for morphologically rich languages: does explicit morphological normalization still provide benefits in the era of transformer-based language models? While lemmatization has traditionally been used to reduce lexical sparsity and improve feature consistency, modern transformer architectures employ contextual subword representations that may already capture much of the information encoded by inflectional variation. Using Croatian disinformation detection as a case study, we conduct a systematic evaluation of lemmatization across traditional machine-learning approaches and transformer-based models within an under-resourced language setting.

Our findings contribute to a deeper understanding of when and why lemmatization benefits downstream NLP applications in morphologically rich languages, offering practical insights for researchers and developers working on content moderation, fact-checking, and information integrity systems in non-English contexts. Our results reveal that the impact of lemmatization is highly architecture-dependent: models relying on sparse lexical representations benefit from normalization, whereas transformer models exhibit little sensitivity to it. These findings contribute new empirical evidence on the interaction between explicit morphological preprocessing and contextual language representations, providing insights that extend beyond Croatian and inform NLP development for other morphologically rich languages.

The remainder of this paper is organized as follows: Section 2 reviews related work on disinformation detection and the role of morphological normalization in natural language processing, with a particular focus on research involving morphologically rich and low-resource languages. Section 3 describes the proposed methodology, including dataset characteristics, preprocessing procedures, and model configurations. Section 4 presents and analyzes the experimental results, highlighting the effects of lemmatization on model performance across different learning architectures. Finally, Section 5 summarizes the main findings, discusses their implications, and outlines directions for future research aimed at further enhancing the robustness and linguistic adaptability of our proposed disinformation detection system.

2. Related Work

The proliferation of fake and misleading information on social media has attracted significant scholarly attention due to its impact on public discourse, institutional trust, and democratic processes. Automated disinformation detection has thus become a critical area of research within natural language processing (NLP) and computational social science [7]. Because disinformation operates through both language and social dynamics, researchers have developed a variety of approaches to address it, including linguistic and semantic analysis as well as graph-based and multimodal frameworks.

Recent advances in deep learning and pre-trained transformer architectures have considerably improved text classification performance, particularly in high-resource languages such as English [8]. However, their performance in languages like Croatian remains an open challenge. Morphological variation, inflectional richness, and relatively limited training data can all degrade model robustness and generalization [9]. Consequently, understanding how fundamental preprocessing steps (e.g., lemmatization) influence model performance is essential for the development of reliable and interpretable disinformation detection systems for such languages. While this review primarily focuses on recent research, several earlier studies are retained where they represent foundational contributions, widely adopted linguistic resources, or seminal work on morphologically rich and low-resource Slavic languages for which no directly comparable contemporary alternatives exist.

The role of lemmatization has evolved alongside the development of modern neural architectures and contextual embeddings. While transformer-based models such as BERT and its multilingual variants can implicitly capture morphological relations through subword tokenization, explicit normalization remains valuable for tasks and languages where data scarcity and morphological complexity interact [10]. Recent studies have demonstrated that linguistically motivated morphological processing can complement neural representations by reducing lexical and morphological ambiguity, improving representation quality, and increasing robustness across a range of downstream NLP tasks [11,12]. Moreover, recent research suggests that hybrid approaches combining explicit morphological processing with neural models can improve robustness and linguistic coverage in morphologically rich and low-resource languages, particularly when handling complex inflectional patterns and out-of-vocabulary forms [13].

The challenges discussed above have motivated a growing body of research focusing on Croatian and other South Slavic languages, including Serbian, Bosnian, Montenegrin, Slovenian, Macedonian, and Bulgarian. These languages exhibit varying degrees of lexical, syntactic, and morphological similarity, with Croatian, Serbian, Bosnian, and Montenegrin historically often grouped under the broader South Slavic dialect continuum commonly referred to as Serbo-Croatian. As a result, many NLP methods, resources, and linguistic tools developed for one language have influenced research on the others, although important linguistic and orthographic differences remain relevant for computational processing. However, progress remains constrained by the scarcity of annotated datasets and high-quality language resources, which complicates both model training and evaluation [14,15]. Consequently, language-specific lemmatization and morphological analysis tools have played a crucial role in enabling core NLP applications, such as tagging, parsing, and sentiment analysis in some linguistic environments [16,17].

In the domain of disinformation detection, textual content remains a central but complex signal. Studies show that linguistic cues alone often yield moderate performance due to the subtle and context-dependent nature of deceptive content [18,19]. However, richer linguistic preprocessing, including morphological normalization can enhance feature consistency and reduce noise, particularly in languages where inflectional variation is substantial.

2.1. Disinformation Detection and Linguistic Processing

Numerous computational frameworks have been proposed for identifying disinformation. Shu et al. [20] classified detection methods into content-, context-, and propagation-based approaches, while Zhou and Zafarani [21] emphasized the need for hybrid architectures that integrate linguistic and network-level signals. Although transformer-based models like BERT and RoBERTa have advanced the state of the art, studies often overlook how language-specific preprocessing choices affect their outcomes, particularly in smaller linguistic communities.

For Croatian and other South Slavic languages, studies have shown that domain-specific models such as croBERT and BERTić outperform traditional methods in various text classification tasks [6,22]. Yet, these models are typically applied with minimal text normalization, leaving an unexplored research question as to whether linguistic preprocessing, such as lemmatization, could further enhance their robustness or interpretability. This omission is especially relevant given Croatian’s high degree of inflection, which can lead to feature sparsity and inconsistent tokenization.

2.2. Morphological Normalization and Lemmatization in NLP

Lemmatization reduces inflected word forms to their canonical base forms, thereby reducing lexical sparsity and improving lexical consistency. Recent studies have shown that explicit morphological processing remains beneficial for morphologically rich and low-resource languages, improving the quality of linguistic representations and supporting tasks such as morphological analysis, lemmatization, morphosyntactic annotation, and contextual language modeling [23,24].

Other recent studies have explored how morphological normalization interacts with modern contextual embeddings. Recent studies indicate that incorporating explicit morphological information into subword tokenization can improve the quality of transformer representations and support more robust processing of morphologically rich languages [25]. Kondratyuk and Straka [26] further demonstrated that integrating lemmatization into tagging pipelines improves downstream performance, highlighting the potential for synergistic effects between morphological modeling and neural architectures. However, systematic evaluation of this relationship remains limited, particularly in the context of disinformation detection.

2.3. Lemmatization for Croatian and South Slavic Languages

Croatian, like other South Slavic languages, exhibits complex inflectional morphology across nouns, adjectives, and verbs [27]. Accurate lemmatization is therefore essential to ensure consistent representation of linguistic features. Early work by Agić, Ljubešić, and Merkler [28] introduced one of the first robust lemmatization and morphosyntactic tagging tools for Croatian and Serbian. The study exploited the close linguistic relationship between the two languages, which share substantial lexical and morphological overlap despite differences in orthography, standardization practices, and certain grammatical features. This foundation enabled the development of large annotated corpora and subsequently contributed to the advancement of NLP pipelines for several South Slavic languages.

The CLASSLA–Stanza pipeline [29,30] currently represents the state of the art for Croatian lemmatization. Built on neural sequence models, it achieves over 98% lemma accuracy on Universal Dependencies (UD-HR) corpora. CLASSLA has become a standard preprocessing tool for Croatian NLP tasks, including hate speech detection and sentiment analysis. Despite its success, few studies have examined whether integrating lemmatization with transformer-based architectures yields measurable performance gains in applied domains such as disinformation detection.

Several morphological resources and lemmatization tools have been developed for Croatian, differing in their methodological foundations and intended use. Among the most frequently cited are CroMorpho and the CST Lemmatizer. CroMorpho is a large-scale Croatian morphological lexicon and analyzer that provides extensive information on inflectional paradigms and lexical forms [31]. It has been widely used as a linguistic resource in morphosyntactic tagging, lemmatization, and other NLP applications requiring detailed morphological information. Its broad lexical coverage makes it particularly valuable for processing morphologically rich Croatian text.

The CST Lemmatizer [32] is a trainable rule-based lemmatization framework that automatically induces transformation rules from annotated word-form and lemma pairs. Originally designed as a multilingual solution for a broad range of European languages, it has been successfully applied to morphologically rich languages, including Croatian and Serbian. Previous studies have reported high lemmatization accuracy when the system is trained on representative corpora and supported by comprehensive morphological resources. Nevertheless, as with most rule-learning approaches, its effectiveness depends on the coverage and representativeness of the available training data. This approach has been shown to achieve lemmatization accuracy above 97% for Croatian and around 96% for Serbian in cross-domain evaluations.

Overall, CLASSLA–Stanza has emerged as the preferred lemmatization tool for contemporary Croatian NLP research, offering the most favorable trade-off between precision, speed, and reproducibility. Consequently, it is adopted as the primary preprocessing component in this study.

To better contextualize the present study within the existing literature,

Table 1. Comparison of selected studies on lemmatization, morphological normalization, and morphology-aware NLP for morphologically rich languages. Source: author’s contribution.

Study	Language(s)	Task	Morphology Strategy	Model(s)	Dataset
[3]	6 languages incl. Czech	Contextual lemmatization	Explicit vs. implicit morphology	XLM-R, mBERT, Morpheus	UniMorph, UD
[26]	Czech, German, Arabic, English	Joint tagging + lemmatization	Joint morphology learning	LemmaTag	PDT, TIGER, PADT, EWT corpora
[28]	Croatian, Serbian	Lemmatization, MSD tagging	Statistical lemmatization	CST, HunPos	SETIMES.HR, Wikipedia
[29]	Slovenian, Croatian, Serbian	MSD tagging, lemmatization	Lexicon-assisted morphology	reldi-tagger, StanfordNLP	ssj500k, hr500k, SETimes
[30]	South Slavic languages	Linguistic processing	Lexicon-assisted morphology	CLASSLA-Stanza	Multiple corpora
[32]	10 European languages incl. Slovene	Lemmatization	Affix-based rules	Rule-based	Lexical resources
[33]	Russian	Topic modeling	Lemmatization	LDA	Russian Wikipedia
This study	Croatian	Disinformation detection	Lemmatization	TF-IDF + SVM, RF, NN, croBERT	Croatian disinformation dataset

summarizes a selected subset of previous work that is most relevant to our research objectives. Given the broad scope of research on lemmatization and morphological processing, the table focuses specifically on studies that investigate the role of lemmatization, morphological normalization, or morphology-aware processing in morphologically rich languages, with particular attention to South Slavic languages and neural or transformer-based approaches. The comparison highlights differences in language, task, preprocessing strategy, modeling approach, and evaluation data. As shown in Table 1, previous research has largely concentrated either on developing lemmatization systems themselves or on evaluating morphology-aware processing in general NLP tasks. Comparatively little attention has been devoted to understanding whether explicit lemmatization remains beneficial for modern transformer-based classification architectures, particularly in downstream disinformation detection tasks for morphologically rich and under-resourced languages.

Several observations emerge from the comparison. Earlier studies generally reported benefits of lemmatization and morphological normalization for tasks relying on sparse lexical representations, rule-based systems, or traditional machine learning approaches. More recent neural and transformer-based studies suggest that contextual language models can internally capture a substantial amount of morphological information, thereby reducing the need for explicit normalization. However, findings remain task- and language-dependent and are often focused on lemmatization itself rather than on downstream text classification. Furthermore, while Croatian and related South Slavic languages have been considered in studies of morphosyntactic annotation, lemmatization, and linguistic processing pipelines, we are not aware of any prior work that systematically investigates the effect of explicit lemmatization across both traditional machine learning models and transformer-based architectures for Croatian disinformation detection. This gap motivates the present study.

While prior research has extensively examined transformer-based disinformation detection and morphological modeling independently, few studies have addressed their intersection for morphologically rich, low-resource languages such as Croatian [34]. It remains unclear whether explicit lemmatization enhances transformer-based text representations or whether subword tokenization already captures sufficient morphological information. This study addresses that gap by conducting a systematic evaluation of lemmatization’s impact on Croatian disinformation detection models. The results aim to inform best practices in linguistic preprocessing for multilingual and low-resource NLP, contributing to the broader field of trustworthy tools for combating digital disinformation.

3. Methodology

This section outlines the experimental methodology employed to evaluate the impact of lemmatization on disinformation detection performance in Croatian-language social media comments. Specifically, the experimental framework is designed to assess whether morphological normalization contributes to improved classifier generalization in low-resource, morphologically rich language contexts.

The methodology encompasses four primary components:

• A description of the datasets used for these experiments and their linguistic characteristics;
• Preprocessing and lemmatization procedures;
• Modeling setup, including both traditional and transformer-based architectures;
• The evaluation protocol and implementation environment.

The methodology aims to quantitatively assess the influence of lemmatization on the performance of disinformation detection models for Croatian-language social media comments.

Formally, the task can be defined as a binary text classification problem. Each social media comment $x_i$ is associated with a ground-truth label $y_i\in \{0,1\}$, indicating whether the instance represents non-disinformation ($0$) or disinformation ($1$). Let

$$D={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^N,$$

(1)

denote the dataset of $N$ labeled examples. The objective is to learn a function

$$f_{\Theta }:X\to Y,$$

(2)

parameterized by $\theta$, that minimizes the expected misclassification loss:

$${\Theta }^{\mathrm{*}}=\arg \min {\mathbb{E}}_{(x,y)~D}\left[L\left(f_{\Theta }\left(x\right),y\right)\right],$$

(3)

where $L$ is an appropriate loss function for the given model family (e.g., cross-entropy for neural networks or hinge loss for SVM-based classifiers).

Lemmatization is introduced as a morphological normalization function

$$\mathcal{L}:V\to V_L,$$

(4)

mapping tokens from the original vocabulary $V$ to a reduced lemma vocabulary $V_L$. Applying $\mathcal{L}$ to each comment yields a transformed dataset

$$D_L=\left\{\left(\mathcal{L}\left(x_i\right),y_i\right)\right\},$$

(5)

The original, unprocessed dataset is denoted by $D_R$. By training and evaluating models on both $D_R$ and $D_L$, we can empirically assess how morphological normalization affects downstream performance and estimate the extent to which lemmatization contributes to reducing noise, sparsity, or model confusion.

3.1. Datasets

Two datasets were used to evaluate the proposed framework and to examine the effect of lemmatization on disinformation detection in Croatian social media text.

The first dataset employed in this study is FRENK 1.1 [34], a publicly available collection of Croatian-language Facebook comments. The dataset contains comments related to topics such as migration and LGBTQ+ issues, annotated for the presence of disinformation and other communicative phenomena. Owing to its manually verified labels and focus on socially and politically sensitive topics, FRENK 1.1 represents a reliable benchmark for studying disinformation detection in Croatian. It also provides a realistic representation of online discourse characterized by informal syntax, colloquial expressions, and varying degrees of grammatical correctness.

The second dataset comprises 24,019 Croatian-language comments extracted from various social media platforms and stored in a structured JSON format. Each entry contains a single textual field representing a standalone sentence or short comment. The raw dataset occupies approximately 2.36 MB, corresponding to 1.62 MB after preprocessing. Prior to model training, the corpus was subjected to a cleaning and normalization pipeline designed to remove platform-specific artifacts while preserving linguistically and pragmatically relevant information. The cleaning stage included the removal of duplicated entries, malformed records, URLs, user mentions, and non-textual metadata generated by the source platforms. Records containing no meaningful linguistic content were excluded, while stylistically relevant elements, including orthographic variation, informal language patterns, and emotionally marked expressions, were retained because of their potential contribution to disinformation detection. Normalization consisted of Unicode standardization, whitespace unification, punctuation normalization, and lowercasing, without any topic-based filtering or semantic content modification.

The resulting corpus captures public discourse on sociopolitical and identity-related topics, including immigration, ethnicity, and LGBTQ+ rights, domains frequently associated with misinformation and polarized communication. As such, it provides a realistic environment for evaluating whether linguistic preprocessing, particularly lemmatization, can improve disinformation detection in informal and emotionally charged user-generated content. The dataset was partitioned into training and test subsets using a stratified split. All evaluation metrics reported in this paper were computed on the held-out test set comprising 9211 instances, while the remaining comments were used for model training and validation.

3.2. Preprocessing

To systematically isolate the effect of lemmatization, two parallel preprocessing pipelines were constructed. First, the text was cleaned, lowercased, and tokenized using a whitespace- and punctuation-based tokenizer, without applying any morphological normalization. Subsequently, the same text underwent lemmatization using the CLASSLA–Stanza pipeline, a state-of-the-art morphosyntactic processor for Croatian.

In both pipelines, all text was normalized to UTF-8, and extraneous characters such as URLs, emojis, and user handles were removed. Stopwords were deliberately retained, as recent research has shown that function words, pronouns, discourse markers, and other high-frequency linguistic elements often encode stylistic, evaluative, and pragmatic cues that contribute to deception, stance, and misinformation detection [35,36].

Each comment $x_i$ was preprocessed through tokenization, normalization, and, optionally, lemmatization. In the lemmatized setting, the comment is represented as the transformed token sequence:

$$x_i^{\prime }=\left[L\left(t_1\right),L\left(t_2\right),\dots ,L\left(t_m\right)\right],$$

(6)

where $t_j$ denotes the j-th raw token in $x_i$, and $L\left(t_j\right)$ is a lemmatization function that maps each token to its corresponding lemma. This transformation preserves the original word order while reducing morphological variation in the text. As a result, lemmatization decreases the effective vocabulary size, i.e., the number of distinct surface forms is substantially larger than the number of distinct lemmas:

$$\left|V_{raw}\right|\gg \left|V_{lemma}\right|,$$

(7)

Empirical analyses of morphologically rich languages consistently show that lemmatization substantially reduces lexical variability by collapsing inflected surface forms into canonical lemmas. The Croatian language, characterized by extensive nominal case systems, gender and number agreement, and rich verbal conjugation exhibits particularly high morphological redundancy. Prior corpus-based evaluations of Croatian lemmatization systems report an average reduction of 19–23% in the number of unique lexical items [29], a pattern comparable to related Slavic languages such as Czech and Russian [36,37]. These findings support the assumption of approximately 22% vocabulary compression in our study. This contraction of the lexical space mitigates sparsity, reduces feature fragmentation, and increases representational consistency for downstream models.

For traditional machine learning classifiers, both the raw and lemmatized corpora were vectorized using TF-IDF weighting, computed as:

$$tfidf\left(t,d\right)=tf(t,d)·log{\displaystyle \frac{N}{df\left(t\right)}},$$

(8)

where $tf\left(t,d\right)$ denotes the frequency of term $t$ in document $d$, and $df\left(t\right)$ denotes the number of documents in which $t$ appears. Identical tokenization procedures were used for both variants to ensure methodological parity, with the only difference being the replacement of inflected forms by lemmas in the lemmatized pipeline. All intermediate and final preprocessed datasets were stored in JSON format to ensure reproducibility and consistent loading across experimental runs.

In morphologically rich languages such as Croatian, a single lexical item may occur in many inflected forms. Consequently, classifiers based on surface forms must estimate model parameters over a substantially larger and sparser feature space. Lemmatization addresses this issue by mapping inflected variants to a common canonical form, thereby reducing lexical sparsity and consolidating token frequencies. Previous research on Croa-tian [28] and other morphologically rich languages has shown that morphological nor-malization can substantially reduce the number of unique word forms while preserving lexical meaning, potentially improving statistical estimation and model generalization, particularly in low-resource settings and for models relying on sparse lexical representations. This effect provides the primary motivation for evaluating lemmatization as a preprocessing strategy in the present study.

The present study focuses exclusively on lemmatization because it represents the most widely adopted form of linguistically informed morphological normalization in Croatian NLP. While stemming constitutes an alternative normalization strategy, it typically relies on heuristic affix removal and may generate non-lexical stems. A systematic comparison between stemming and lemmatization is therefore outside the scope of this work and is left for future research.

3.3. Model Setup

To evaluate the generality of lemmatization benefits, we employed both traditional machine learning models and transformer-based architectures. All models were trained on the dataset $D$ introduced in the previous subsection, which contains $N$ labeled instances $\left(x_i,y_i\right)$ where the input text $x_i$ was represented either in raw token form or after lemmatization.

Traditional classifiers were trained on feature-based representations derived from the textual inputs. Specifically, each comment $x_i$ was mapped to a TF–IDF feature vector ${\Phi (x}_i)$ using unigram and bigram tokenization. Each classifier is modeled as a supervised function $f_{\Theta },$ where Θ denotes the learned parameters. The three traditional models considered are defined as follows.

3.3.1. Support Vector Machines Model Configuration

Support Vector Machines are margin-based classifiers that aim to find a decision boundary maximizing the separation between classes in a transformed feature space. In this work, both a linear SVM and a non-linear kernel-based SVM with a radial basis function (RBF) kernel are evaluated to assess whether the classification task benefits from non-linear decision boundaries.

The linear SVM operates directly on TF–IDF feature vectors and assumes that the classes are linearly separable in the induced high-dimensional space. Given an input text instance x, represented by its TF-IDF vector $\Phi \left(x\right)$, the classifier predicts the class label as:

$$f_{\Theta }\left(x\right)=sign(w^{\mathrm{\top }}\Phi \left(x\right)+b),$$

(9)

where $\Phi \left(x\right)$ denotes the TF-IDF representation of input x, w denotes the weight vector and b is the bias term. The model parameters are learned by minimizing the regularized hinge loss:

$$L={\displaystyle {\sum }_{i=1}^N}\mathrm{m}\mathrm{a}\mathrm{x}(0,1-y_i{(w}^{\mathrm{\top }}\Phi \left(x\right)+b)),$$

(10)

where ${(x}_i,y_i)$ are the training instances with labels $y_i\in \{-1,+1\}$. Due to the sparsity and high dimensionality of TF-IDF representations, the linear SVM is computationally efficient and is commonly regarded as a strong baseline for text classification tasks.

To model potential non-linear relationships between features, a kernel-based SVM with a radial basis function (RBF) kernel is also considered. Instead of operating directly in the original feature space, the RBF SVM implicitly maps input vectors into a higher-dimensional space through a kernel function defined as:

$$K\left(x_i,x_j\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma \parallel \Phi \left(x_i\right)-\Phi \left(x_{ij}\right){\parallel }^2),$$

(11)

where γ controls the width of the kernel. In this formulation, the decision function is expressed in its dual form:

$$f\left(x\right)=sign({\displaystyle {\sum }_{i=1}^N}{\alpha }_i{y}_{i}K(x_i,x)+b),$$

(12)

with ${\alpha }_i$ representing the learned Lagrange multipliers. The RBF kernel allows the model to learn non-linear decision boundaries by measuring similarity between instances rather than relying on explicit feature weights. By evaluating both linear and RBF-based SVMs, this study examines whether the additional model capacity provided by kernelization leads to improved generalization performance, or whether the high-dimensional TF-IDF representation is sufficient to support linear separation.

3.3.2. Random Forrest Model Configuration

The Random Forest classifier consists of an ensemble of $T=200$ decision trees. The final prediction is determined by majority vote:

$$f_{\Theta }\left(x\right)={mode\left\{h_t\left(x\right)\right\}}_{t=1}^T,$$

(13)

where each tree $h_t$ is trained on a bootstrap sample of $D$.

3.3.3. Feedforward Neural Network (NN) Model Configuration

The feedforward neural network comprises two fully connected layers with ReLU activations and dropout regularization. Its decision function is given by:

$$f_{\Theta }\left(x\right)=\sigma (W_2·ReLU\left(W_1\Phi \left(x\right)+b_1\right)+b_2),$$

(14)

where $\sigma$ denotes the sigmoid activation. The network is optimized using the binary cross-entropy loss:

$$L=-{\displaystyle {\sum }_{i=1}^N}{y}_i\log f_{\Theta }\left(x_i\right)+\left(1-y_i\right)\log (1-f_{\Theta }\left(x_i\right)),$$

(15)

These traditional models were selected for their transparency, stability in low-resource NLP settings, and interpretability [33,38,39], which is particularly important when analyzing linguistic effects such as morphological normalization. All hyperparameters were tuned via grid search using validation folds, and the same feature extraction procedure was applied to both raw and lemmatized variants of D.

3.3.4. Transformer Model Configuration

For contextual representation learning, we fine-tuned croBERT [40], a transformer-based language model pre-trained on large-scale Croatian corpora. Given an input text x, croBERT first tokenizes it into subword units using its WordPiece vocabulary. Each subword token is mapped to a corresponding embedding vector, and positional encodings are added to retain information about token order. The resulting input representation is processed by a stack of transformer encoder layers, which can be expressed as:

$$H=Transformer(E\left(x\right)+P),$$

(16)

where $E\left(x\right)$ denotes the sequence of token embeddings, P represents the corresponding positional encodings, $+$ denotes element-wise addition, and H is the sequence of contextualized hidden states produced by the model. Through multi-head self-attention, the transformer captures contextual interactions among tokens, enabling the modeling of long-range dependencies, syntactic structure, and semantic nuances. This capability is particularly important for morphologically rich languages such as Croatian.

A special classification token $\left[CLS\right]$ is assigned to each input sequence. After the final transformer layer, the hidden state corresponding to this token, denoted $h_{\left[\mathrm{CLS}\right]}$, serves as a fixed-dimensional representation of the entire input text. This embedding aggregates global contextual information and is used as input to the classification head.

The final prediction is obtained by applying a linear layer followed by a softmax function:

$$f_{\Theta }\left(x\right)=softmax(W_c{h}_{\left[CLS\right]}+b_c),$$

(17)

where $W_c$ and $b_c$ are the parameters of the classifier. The softmax function converts the resulting logits into a probability distribution over the two output classes (disinformation vs. non-disinformation), such that:

$${\displaystyle {\sum }_{k=1}^2}{f}_{\Theta }{\left(x\right)}_k=1,$$

(18)

Fine-tuning updates both the transformer parameters and the classification head jointly, allowing croBERT to adapt its representations to the linguistic characteristics and disinformation patterns present in the Croatian social media dataset. Although transformer models employ subword tokenization, which partially mitigates sparsity caused by morphological variation, previous research on highly inflected South Slavic languages suggests that lemmatization can still be beneficial by reducing inflectional variability and mapping surface forms to shared lexical representations [41].

Fine-tuning was performed using the AdamW optimizer with a learning rate of $2\times {10}^{-5}$, batch size 16, three training epochs, and a maximum sequence length of 128 tokens. The same training configuration was applied to both raw and lemmatized variants of D to ensure strict comparability. Tokenization relied on the original croBERT WordPiece vocabulary.

3.4. Computational Considerations

The evaluated approaches differ substantially in their computational requirements. The traditional machine learning models (Linear SVM, RBF-SVM, Random Forest, and the feedforward neural network) operate on TF-IDF feature representations and therefore require relatively modest computational resources. In contrast, croBERT relies on contextualized transformer representations and a substantially larger number of trainable parameters, resulting in higher computational demands during both training and inference.

All experiments were executed on a MacBook Pro (13-inch, M2, 2022) equipped with an Apple M2 SoC featuring an 8-core CPU, a 10-core integrated GPU, and 16 GB of unified memory. Under these conditions, the complete experimental campaign, including fine-tuning the pre-trained croBERT model, required approximately 3 h and 10 min. In comparison, optimization of the feedforward neural network was completed in approximately 15 min, while the traditional machine learning models required substantially less training time.

This notable discrepancy in computational requirements highlights the practical trade-off between model sophistication and processing efficiency. Transformer-based models such as croBERT can capture complex contextual and semantic relationships through contextualized representations, but this capability comes at a considerably higher computational cost than traditional machine learning approaches. Conversely, TF-IDF-based methods offer faster training and lower deployment requirements, while relying more heavily on explicit preprocessing decisions such as lemmatization.

Although a detailed inference-time benchmark was beyond the scope of the present study, transformer-based models generally incur higher inference costs because contextual representations must be computed for each input sequence, whereas traditional machine learning approaches perform prediction directly on precomputed sparse feature vectors. Consequently, model selection in low-resource settings must consider not only predictive performance but also available computational resources and deployment constraints.

3.5. Evaluation Protocol

Model evaluation was performed using five-fold stratified cross-validation to reduce the influence of sampling bias and to ensure that class proportions were preserved across folds. Each model was trained and evaluated in all five splits, and the reported results correspond to the mean performance across folds. We report on accuracy, precision, recall, and the F₁-score for the positive class (Class 1). Due to the inherent class imbalance, model comparison is primarily based on the F₁-score of the target class, which jointly captures precision and recall and provides a more informative evaluation than accuracy alone in disinformation detection scenarios.

To assess whether lemmatization yielded statistically significant improvements, we applied paired t-tests to the fold-level F₁-scores obtained under raw and lemmatized preprocessing conditions. In addition, we conducted a qualitative error analysis by manually examining misclassified instances to identify linguistic phenomena, such as derivational variation, agreement inconsistencies, or code-switching that contributed to incorrect predictions. This combined quantitative and qualitative evaluation is consistent with recent studies on morphologically rich Slavic languages, where manual inspection of model outputs and error patterns is used to complement quantitative evaluation and provide deeper insight into language-specific challenges [42].

4. Results and Discussion

The central objective of this study is not only to determine whether lemmatization affects classification performance, but also to understand how its impact varies across fundamentally different model architectures. Consequently, the evaluation combines predictive performance measurements, statistical significance testing, and qualitative analysis in order to identify the conditions under which explicit morphological normalization is beneficial, neutral, or detrimental for Croatian disinformation detection.

4.1. Experimental Setup

All experiments were implemented in Python 3.11, using the scikit-learn library for traditional machine learning models and the Hugging Face Transformers framework for transformer-based architectures. Text preprocessing, including tokenization and lemmatization, was performed with Stanza, a neural NLP toolkit optimized for morphologically rich languages such as Croatian [43,44]. To mitigate potential class imbalance and improve generalization, SMOTE (Synthetic Minority Over-sampling Technique) was applied to the training data where appropriate [45].

For the baseline evaluation, four traditional classifiers were used: Support Vector Machine (SVM)—Linear and with an RBF kernel, Random Forest (RF), and a simple feedforward neural network (NN) [46]. Each model was trained and evaluated on two dataset variants, original and lemmatized text, while keeping preprocessing steps and hyperparameters fixed across conditions. This controlled setup isolates the effect of lemmatization and ensures that performance differences can be attributed solely to morphological normalization rather than configuration changes. To complement traditional models, experiments also included croBERT, a transformer-based language model pre-trained specifically for Croatian. In total, five classifiers: Random Forest, Linear SVM, kernel-based SVM, feedforward NN, and croBERT were evaluated on both textual variants across two Croatian-language datasets.

The inclusion of both linear and non-linear SVM formulations enables a principled examination of the underlying decision surface complexity, assessing whether the classification problem benefits from non-linear separability or whether high-dimensional sparse textual representations are already sufficiently linearly separable. Furthermore, comparing conventional feature-based models with a contextualized transformer architecture provides insight into the relative contribution of explicit morphological normalization versus implicitly learned contextual representations.

The impact of lemmatization in the Croatian language setting is analyzed through quantitative performance metrics, statistical significance testing, and qualitative inspection of error patterns, allowing for a comprehensive assessment of both predictive performance and model behavior. As the development of linguistically informed and computationally robust misinformation detection systems remains an important research challenge [47,48], the experiments are designed to assess whether lemmatization improves text representations and classification effectiveness in Croatian social media data. To ensure a fair comparison across model architectures, hyperparameters were optimized independently for each classifier using the validation subset. Model selection was based on the highest validation-set F1 score, which was chosen as the primary optimization criterion due to its suitability for binary classification tasks involving potentially imbalanced class distributions. The same selection procedure was applied to both the original and the lemmatized datasets to ensure that any observed performance differences could be attributed to preprocessing effects rather than differences in model optimization. Hyperparameters were optimized using RandomizedSearchCV on the validation set.

Table 2. Hyperparameter search spaces and fixed settings used during model optimization. Source: author’s contribution.

Model	Final Hyperparameters
Linear SVM	class_weight = “balanced”, C ∈ {0.01, 0.1, 1, 10}
RBF-SVM	C ∈ {0.5, 1, 5}, γ ∈ {“scale”, 0.01, 0.001}
Random Forest	n_estimators = 300, max_depth ∈ {None, 20, 40}, max_features ∈ {“sqrt”, 0.2, 0.5}, min_samples_leaf ∈ {1, 2, 4}
Feed-forward Neural Network	hidden_layer_sizes = (256, 128), alpha ∈ {10−5, 10−4, 10−3}, learning_rate_init ∈ {10−4, 10−3}
croBERT	optimizer = AdamW, learning rate = 2 × 10−5, batch size = 16, epochs = 3

reports the search spaces used for hyperparameter optimization. The final model for each architecture was selected based on the highest validation F1-score and subsequently evaluated on the test set.

4.2. Quantitative Results

The overall results shown in Figure 1 and

Table 3. Experiment results. Source: author’s contribution.

Data	Model	Accuracy	Precision	Recall	F1	ROC-AUC
original	RandomForest	0.9223	0.9120	0.9533	0.9322	0.9442
original	SVM	0.9152	0.8982	0.9572	0.9268	0.9315
original	LinearSVM	0.9089	0.8886	0.9576	0.9218	0.9382
original	NeuralNet	0.8961	0.8797	0.9436	0.9106	0.9348
original	croBERT	0.9205	0.9096	0.9529	0.9308	0.9399
lemmatized	RandomForest	0.9232	0.9164	0.9496	0.9327	0.9450
lemmatized	SVM	0.8570	0.8133	0.9669	0.8835	0.9273
lemmatized	LinearSVM	0.9115	0.8921	0.9580	0.9239	0.9422
lemmatized	NeuralNet	0.8510	0.8490	0.8931	0.8705	0.9119
lemmatized	croBERT	0.9239	0.9145	0.9533	0.9335	0.9443

indicate that lemmatization affects the examined models in heterogeneous ways, sometimes yielding modest improvements and sometimes suppressing performance, depending on the model architecture. On the original dataset, Random Forest, SVM with an RBF kernel, Linear SVM, and croBERT all achieve strong F₁-scores ranging between 0.92 and 0.93, while the neural network performs somewhat lower at 0.91. When lemmatized data are used, the performance landscape shifts. Random Forest, Linear SVM, and croBERT exhibit small but consistent improvements, with Random Forest increasing from 0.9322 to 0.9327, Linear SVM from 0.9218 to 0.9239, and croBERT from 0.9308 to 0.9335. These results suggest that vocabulary normalization reduces feature sparsity sufficiently to benefit models with linear or ensemble-based decision boundaries. In contrast, models whose behavior relies on richer token-level distributions, specifically SVM with an RBF kernel and the neural network, exhibit pronounced degradation: the SVM drops from 0.9268 to 0.8835 and the neural network from 0.9106 to 0.8705. These declines indicate that lemmatization may prune away lexical subtleties that these models implicitly exploit.

Accuracy, precision, recall, and ROC-AUC values reflect the same trends. Figure 2a–e depict the ROC curves for original and lemmatized text with a direct comparison: Random Forest, SVM (RBF), Linear SVM, neural network, and croBERT, respectively. By evaluating performance across all possible thresholds, the ROC–AUC analysis complements threshold-dependent metrics such as F₁ and provides a more comprehensive view of how linguistic preprocessing influences model discrimination behavior. The ROC-AUC analysis reveals distinct model-specific responses to lemmatization. Random Forest, Linear SVM, and croBERT exhibit stable or slightly improved ROC-AUC scores when trained on lemmatized text, indicating preserved or marginally enhanced class separability. In contrast, the neural network and the RBF-kernel SVM show downward shifts in their ROC curves, reflecting reduced discriminative capacity after morphological normalization.

CroBERT demonstrates robustness to preprocessing changes: its ROC-AUC increases marginally from 0.9399 to 0.9443. This stability suggests that contextual embeddings already capture morphological regularities, rendering external lemmatization largely redundant. The minimal variation further confirms that transformer-based representations internalize normalization effects during pretraining. For Random Forest and Linear SVM, the ROC-AUC curves for lemmatized data either slightly exceed or closely overlap those obtained from the original text. The corresponding marginal gains in curves imply that these models benefit from reduced lexical sparsity and a simplified feature space. By contrast, the SVM (RBF) and neural network exhibit consistent downward shifts in their curves, indicating that lemmatization alters the feature distribution in a way that weakens non-linear decision structures.

Notably, SVM (RBF) presents only a minor decrease in ROC-AUC despite a substantial drop in F₁ score. This discrepancy suggests that the model retains probabilistic separability but becomes sensitive to the classification threshold under the modified feature representation. Hence, the observed degradation appears to stem not solely from weakened representations, but also from threshold-dependent effects induced by altered class boundary geometry.

The confusion matrices provide further insight into the model behavior and reveal how lemmatization reshapes classification boundaries. On the original dataset, all models exhibit high true positive rates, indicating strong identification of disinformation. CroBERT and Random Forest produce the lowest false negative counts, which align well with their high recall values. Classical models, especially SVM (RBF) and the neural network, show comparatively higher false positive rates, suggesting that subtle lexical variations and informal language frequently trigger incorrect classification. Figure 3a–c shows confusion matrices for SVM (RBF), Random Forest and croBERT experiments with the original dataset.

When lemmatized input is introduced, the confusion patterns change in ways that mirror the performance metrics. For Random Forest and Linear SVM, false positives decrease and true negatives increase, consistent with the slight improvements in precision and ROC-AUC. By contrast, SVM (RBF) and the neural network exhibit simultaneously rising false positives and false negatives, indicating that the loss of morphological surface information makes it more difficult for these models to distinguish between borderline or ambiguous cases. The confusion matrices for croBERT remain nearly identical across preprocessing conditions, confirming that the model’s internal contextual mechanisms already resolve morphological variation without needing explicit normalization. Figure 4a–c shows the confusion matrices for SVM (RBF), Random Forest and croBERT experiments, this time for lemmatized inputs.

To complement the quantitative performance metrics, we conducted a detailed statistical and qualitative analysis to determine whether the observed differences between the original and lemmatized datasets were statistically meaningful and to elucidate why certain architectures benefited from morphological normalization while others deteriorated. Together, these analyses provide a deeper understanding of the interaction between Croatian morphology, model inductive biases, and disinformation-related linguistic cues.

As summarized in

Table 4. Statistical comparison of experimental results. Source: author’s contribution.

Model	Effect of Lemmatization	p-Value	Cohen’s d	Interpretaion
Linear SVM	Small F1 increase	p > 0.05	d < 0.2	Negligible practical effect
RBF-SVM	F1 decrease	p < 0.01	d > 0.5	Medium-to-large negative effect
Random Forest	Small F1 increase	p > 0.05	d < 0.2	Negligible practical effect
Feed-forward NN	F1 decrease	p < 0.01	d > 0.5	Medium-to-large negative effect
croBERT	Small F1 increase	p > 0.05	d < 0.2	Negligible practical effect

, statistical testing focused on F₁-score differences because of their relevance for imbalanced and noisy social media classification. Since each model was evaluated under identical conditions except for preprocessing, paired statistical tests were employed. At the model level, statistical testing focused on F₁ score differences, given their relevance for imbalanced and noisy social media classification. Because each model was evaluated under identical conditions except for preprocessing, paired statistical tests were employed. For classical machine learning models and croBERT, paired-sample tests over cross-validation or resampling distributions revealed no statistically significant improvements at the α = 0.05 level (p > 0.05). Although Random Forest, Linear SVM, and croBERT exhibited small mean F₁ increases after lemmatization, the variance across runs was sufficient to classify these gains as statistically negligible. Correspondingly, effect sizes were small (Cohen’s d < 0.2), indicating that lemmatization introduces only minor perturbations to their decision boundaries.

In contrast, the SVM with RBF kernel and a feedforward neural network exhibited statistically significant performance declines when trained on lemmatized input (p < 0.01), with medium-to-large effect sizes (Cohen’s d > 0.5). These results demonstrate that the observed degradation is not stochastic but reflects a systematic disruption in how these architectures exploit surface-level lexical information. While the RBF SVM retained a relatively stable ROC-AUC (0.9315 → 0.9273), its pronounced F₁ decrease (0.9268 → 0.8835) suggests heightened sensitivity to threshold placement under reduced morphological diversity. The neural network showed consistent declines across all metrics, including a marked drop in ROC–AUC (0.9348 → 0.9119), pointing to a deeper representational instability induced by normalization.

Beyond model-level averages, we conducted a large-scale paired instance-level analysis over all test samples (n = 9.211) to assess whether lemmatization induces systematic prediction shifts. This analysis complements cross-validation statistics by leveraging the full prediction distribution rather than aggregated fold-level scores.

A Wilcoxon signed-rank test on paired predicted probabilities for the true class strongly rejected the null hypothesis of zero median difference (p < 10⁻¹⁶), indicating a consistent directional shift across instances. A complementary permutation test on the mean paired difference confirmed this finding (p < 10⁻¹⁶), with an observed mean difference of −0.00979 (original − lemmatized), corresponding to an average increase of approximately one percentage point in predicted probability under lemmatized input.

Effect size estimation via non-parametric bootstrap resampling yielded a tight 95% confidence interval of [−0.01096, −0.00870], excluding zero and demonstrating that the effect, while modest, is precisely estimated and stable. Importantly, McNemar’s test showed that these probabilistic shifts translate into discrete classification improvements: lemmatization corrected 49 errors that were misclassified under the original representation, while introducing only 4 new errors (exact p ≈ 1.4 × 10⁻¹⁰). This confirms that the observed statistical effects are not merely calibration artifacts but reflect genuine improvements in prediction correctness.

To contextualize these statistical findings, confusion matrices were examined to identify characteristic error patterns. Across preprocessing conditions, all models were generally proficient at identifying explicit disinformation, typically marked by strong sentiment cues, rhetorical polarization, or explicit references to identity groups (e.g., migrants or LGBTQ+ individuals). However, substantial differences emerged in how models handled borderline cases involving irony, implicit stance, or high lexical creativity, phenomena particularly prevalent in Croatian online discourse.

Random Forest, Linear SVM, and croBERT exhibited stable error distributions across preprocessing regimes. After lemmatization, both Random Forest and Linear SVM showed a slight reduction in false positives, becoming more conservative in labeling content as disinformation. This behavior aligns with their statistical stability: lemmatization removes inflectional noise, marginally improving generalization without fundamentally altering their decision geometry.

By contrast, the RBF SVM and the neural network displayed systematic error shifts consistent with their statistically significant declines. The RBF SVM experienced simultaneous increases in false positives and false negatives after lemmatization. Inspection of misclassified samples revealed that this model had previously relied on highly discriminative morphological forms, such as verb conjugations or declensional endings, to infer subtle cues of aggression, stance, or implied hostility. Collapsing these forms into lemmas reduced separability in the kernel-induced feature space, causing semantically borderline texts to cluster more closely.

The neural network exhibited a substantially larger performance degradation following lemmatization. Error analysis indicates that many newly misclassified instances were short, informal comments whose interpretation relied heavily on surface-level signals, including non-standard orthography, punctuation irregularities, and subtle morphosyntactic variation. These features encode pragmatic and stylistic information that is not preserved under lemmatization, which collapses inflectional and orthographic variability into canonical forms. As a result, the model appears to lose discriminative cues that were previously informative for class separation, leading to increased overgeneralization and a measurable decline in ROC–AUC. This pattern is consistent with the hypothesis that neural architectures trained on token-level representations can exploit fine-grained lexical and subword variation, and that aggressive normalization may remove signal rather than noise in socially situated text. The qualitative error patterns therefore align with the previously reported effect sizes and statistically significant performance decreases, reinforcing the conclusion that lemmatization is detrimental in this setting.

CroBERT’s qualitative behavior further corroborates its statistical robustness. Confusion matrices show almost no structural change between original and lemmatized inputs. Ambiguous cases, especially those involving sarcasm, euphemisms, or discourse-level signals remain the dominant sources of error in both conditions. This suggests that croBERT’s subword tokenization and contextualized representations already encode morphological variation internally, rendering external lemmatization largely redundant and only marginally beneficial.

4.3. Discussion

Taken together, the statistical and qualitative analyses indicate that lemmatization is not a universally beneficial normalization strategy, but rather a transformation whose effectiveness depends strongly on the representational assumptions of the learning algo-rithm. As shown in

Table 5. Vocabulary reduction after lemmatization. Source: author’s contribution.

Representation	Vocabulary Size
Original	27.432
Lemmatized	16.067
Reduction	41.43%

, lemmatization reduced the effective vocabulary size from 27.482 unique surface forms to 16.067 unique lemmas, corresponding to a reduction of 41.43%. This reduction is consistent with previous findings on lemmatization in morphologically rich Croatian and related South Slavic languages, where the large number of inflectional variants substantially increases vocabulary size and morphological normalization through lemmatization reduces this variation [41]. Such a reduction substantially decreases lexical sparsity by mapping multiple inflected forms to a shared canonical rep-resentation and increasing the frequency associated with individual lexical units.

The observed behavior of the evaluated models is consistent with this interpretation. Linear SVM and Random Forest classifiers exhibited small but positive performance changes after lemmatization. Both approaches operate on TF-IDF representations in which each surface form occupies a separate position within a high-dimensional sparse feature space. Vocabulary consolidation reduces feature fragmentation and increases the statistical robustness of lexical features, producing a modest improvement in generalization. The relatively small magnitude of these gains suggests that sparsity reduction provides some benefit, but that the original representations already captured much of the discriminative information necessary for the task.

At the opposite end of the spectrum, transformer-based models such as croBERT proved largely insensitive to lemmatization. This finding is consistent with recent studies suggesting that contextual transformer architectures implicitly encode substantial morphological information through subword tokenization and contextual representation learning. Because croBERT processes words as sequences of subword units and integrates contextual information during fine-tuning, many morphological regularities are already available within its learned representation space. Consequently, explicit morphological normalization neither substantially improves nor substantially degrades performance.

In contrast, the RBF-kernel SVM and the feedforward neural network experienced pronounced performance degradation following lemmatization. Unlike linear classifiers, these architectures can exploit complex interactions among surface lexical features and therefore benefit from subtle orthographic and morphological distinctions. The error analysis suggests that many of these distinctions are not merely inflectional noise but are associated with stylistic, pragmatic, and discourse-level characteristics of social media communication. In disinformation-related content, lexical choices, inflectional variants, emotionally marked forms, and other surface-level signals may contribute to class separability. When these distinctions are collapsed into a common lemma, informative variation is removed together with purely morphological redundancy. As a result, the geometry of the feature space becomes less discriminative, leading to the substantially larger performance losses observed for these architectures.

The statistical analyses support this interpretation. While improvements observed for Random Forest, Linear SVM, and croBERT were relatively small and associated with negligible effect sizes, the degradation observed for the RBF-SVM and the feedforward neural network was accompanied by statistically significant medium-to-large effects. This asymmetry suggests that the principal impact of lemmatization in this setting is not the addition of useful information, but rather the removal of architecture-specific signals that some models exploit more effectively than others.

More broadly, these findings challenge the common assumption that linguistic normalization uniformly improves downstream performance in morphologically rich languages. Instead, preprocessing should be viewed as an architectural design choice rather than a neutral preliminary step. The interaction between morphological richness, vocabulary consolidation, feature representation, subword modelling, and classifier inductive bias ultimately determines whether normalization acts as regularization or as information loss.

From a practical perspective, this implies that preprocessing pipelines for Croatian, and likely other morphologically rich Slavic languages, should be selected with careful consideration of both the target architecture and the nature of the task. In domains characterized by informal, emotionally charged, and stylistically variable user-generated content, morphological surface cues may themselves carry task-relevant information. Consequently, lemmatization should not be adopted as a default preprocessing strategy, but rather evaluated empirically within the specific linguistic, representational, and application context in which it is applied.

Beyond predictive performance, the practical usefulness of lemmatization must also be evaluated in terms of its computational and engineering costs. Incorporating lemmatization into an NLP pipeline introduces an additional preprocessing stage and requires language-specific linguistic resources, such as morphological dictionaries and lemmatization tools, which increase system complexity and maintenance requirements. While the vocabulary reduction analysis presented in this study demonstrates that lemmatization effectively consolidates inflectional variants and reduces lexical sparsity, the resulting performance gains were generally modest and highly architecture-dependent.

For Linear SVM and Random Forest, lemmatization produced only small improvements that were not accompanied by practically meaningful effect sizes. These gains suggest that vocabulary consolidation can partially alleviate sparsity in TF-IDF-based representations, but the magnitude of the improvement remains limited. Similarly, croBERT exhibited only marginal sensitivity to lemmatization despite the additional preprocessing effort. This behavior is consistent with the ability of transformer architectures to model morphological variation through contextualized subword representations, reducing the need for explicit morphological normalization.

In contrast, the RBF-SVM and the feedforward neural network experienced statistically significant performance degradation after lemmatization. For these architectures, the additional preprocessing step not only fails to provide measurable benefits, but actively removes surface-form information that contributes to classification performance. Consequently, the computational and engineering overhead associated with lemmatization cannot be justified for these models in the context of Croatian disinformation detection.

From a practical perspective, these findings suggest that the decision to apply lemmatization should be guided by the expected deployment scenario rather than by convention. In low-resource environments relying on sparse lexical representations, lemmatization may still provide sufficient vocabulary consolidation to justify its use. However, for modern transformer-based systems and for architectures that benefit from surface-level lexical variability, the additional preprocessing cost is unlikely to be offset by the observed performance gains. Therefore, lemmatization should be regarded as an architecture- and task-dependent design choice rather than a universally beneficial preprocessing strategy.

An additional implication of these findings concerns emerging forms of online content generation, including AI-generated texts and coordinated activity conducted through sock puppet accounts. Both scenarios may alter the distribution of the linguistic signals exploited by classification systems. Contemporary generative language models often produce grammatically well-formed and stylistically consistent content, potentially reducing the natural morphological and orthographic variability observed in human-generated social media discourse. Similarly, coordinated sock puppet accounts may exhibit recurrent lexical and stylistic patterns as part of a broader strategy of message amplification. In such settings, surface-level linguistic variation may itself carry useful information regarding authenticity, coordination, or communicative intent. Consequently, aggressive normalization through lemmatization could remove signals that are potentially valuable for identifying coordinated or artificially generated content. Although the present study focuses exclusively on disinformation detection and does not explicitly evaluate AI-generated or coordinated content, the observed architecture-dependent effects suggest that future work should investigate how morphological normalization interacts with emerging forms of synthetic and coordinated online communication.

5. Conclusions and Future Work

This paper continues our research into detecting disinformation in social media content written in morphologically rich, low-resource languages such as Croatian. By extending the previous classification framework with a substantially larger and independent corpus, this work provides a more robust empirical basis for assessing morphological preprocessing in Croatian disinformation detection. Training identical models on original and lemmatized inputs allows direct attribution of observed performance differences to normalization choices rather than dataset-specific characteristics. These findings therefore refine and broaden the practical recommendations from our prior study. To isolate the impact of normalization, all experiments use paired training regimes and identical evaluation protocols, enabling both instance-level and aggregate assessments of downstream changes.

Our experiments show that the effect of morphological lemmatization depends strongly on model class. Lemmatization yields modest improvements for linear and ensemble methods, most clearly for Linear SVM and Random Forest, likely because normalization reduces lexical sparsity and produces more consistent feature patterns. Conversely, approaches that rely on nonlinear decision boundaries or surface lexical cues (for example, RBF-kernel SVM and a feedforward neural network) often lose performance after lemmatization, suggesting that inflectional variation can encode useful discriminative signals in informal, emotionally charged social media text. Transformer models such as croBERT are largely insensitive to this preprocessing step: their contextual subword representations already capture morphological variation, so normalization produces little change in predictive accuracy. Taken together, these outcomes indicate that preprocessing should be chosen with respect to the target architecture and the linguistic properties of the data, rather than applied by default.

Qualitative inspection, supported by statistical tests, corroborates this interpretation. Lemmatization can reduce noisy surface variation and improve feature coherence, but it may also remove morphosyntactic cues that convey stance, emphasis, or hostility—signals often informative for detecting disinformation. Therefore, normalization is not neutral: its benefits or harms depend on which linguistic cues the model exploits and how tokenization represents morphology.

Looking forward, several research directions appear promising. First, domain-adaptive or task-adaptive pretraining of Croatian transformer models (DAPT/TAPT) on large collections of social media text could help contextual models internalize platform-specific discourse and colloquialisms, improving downstream detection. Second, richer linguistic representations deserve exploration: hybrid schemes combining character-level and subword-level features, byte-level transformers, or selective normalization strategies that preserve semantically relevant inflectional patterns may capture both robustness and fine-grained signals. Third, broadening the scope of data, moving beyond short comments to longer posts and cross-platform corpora, and incorporating multimodal inputs (images, links, temporal posting patterns) would better reflect real-world disinformation campaigns and likely increase system resilience.