Irony and Sarcasm Detection in Turkish Texts: A Comparative Study of Transformer-Based Models and Ensemble Learning

Abstract

Irony and sarcasm are forms of expression that emphasize the inconsistency between what is said and what is meant. Correctly classifying such expressions is an important text mining problem, especially on user-centered platforms such as social media. Due to the increasing prevalence of implicit expressions, this topic has become a significant area of research in Natural Language Processing (NLP). However, the simultaneous detection of ironic and sarcastic expressions is highly challenging, as both types of implicit sentiments often convey closely related meanings. To address the detection of irony and sarcasm, this study compares the performance of transformer-based models and an ensemble learning method on Turkish texts, using five textual datasets—monogram, bigram, trigram, quadrigram, and omnigram—that share the same textual content but differ in context length. To improve classification performance, an ensemble learning approach based on the Artificial Rabbit Optimization (ARO) algorithm was implemented, combining the outputs of the models to produce final predictions. The experimental results indicate that as the context width of the datasets increases, the models achieve better predictions, leading to improvements across all performance metrics. The ensemble learning method outperformed individual models in all metrics, with performance increasing as the context expanded, achieving the highest success in the omnigram dataset with 76.71% accuracy, 74.64% precision, 73.29% sensitivity, and 73.96% F-Score. This study demonstrates that both model architecture and data structure are decisive factors in text classification performance, showing that community methods can make significant contributions to the effectiveness of deep learning solutions in low-resource languages.

1. Introduction

Humanity lives in an age where data flows every second around the world and only a small fraction of this data can be understood using manual methods [1]. With the advent of the internet, large amounts of textual data have become available [2,3]. While numerical data within this information can be visualized and analyzed using specific statistical tools, the same is not possible when it comes to texts [1].

As technology advances, people have increasingly more opportunities for interaction, ranging from simple text messages and message boards to more engaging and immersive channels such as images and videos [2]. It is becoming increasingly difficult to absorb the mass of information that exists in sources such as websites, news, blogs, books, scientific articles and social media [4]. Due to the inadequacy of human capacity to process large volumes of text data, Natural Language Processing (NLP) is used to read, interpret and understand such data in a much larger volume than human intelligence can perceive at a time in order to make such data more insightful and quickly processable [1]. Therefore, over the past few decades, text summarization has emerged as a popular field of research, aiming to simplify and make the process of obtaining more efficient [4]. NLP is used to overcome many NLP problems, especially sentence classification, sentence-pair classification, asking and answering questions, and named entity-relationship classification [1].

Oral [5,6] or written [7,8] communication is not a simple process. On the contrary, it requires sharing a common experience and history, as well as being able to extract information beyond contextual meaning. Many communicative acts imply information that is not grammatically expressed in order to decode the full meaning: if the listener is unable to extract this information, it means the communicative process is incomplete [6]. These texts contain content with intense figures of speech [9,10,11]. Figurative language includes concepts such as irony, sarcasm, satire, parody, etc. [12]. However, human understanding of figurative language is a difficult matter even using automated text and data mining systems [13] because the stunning feature of sarcastic expressions such as irony and sarcasm is that they make it difficult to bridge the gap between the literal meaning and the intended meaning. For the detection of irony and sarcasm, the fundamental problem in the context of such expressions is to identify the contradiction between what is said and what is meant [14]. As sarcasm is prevalent on social media, online forums and review websites, it is critical that models attempting to analyze online texts are able to detect and understand it. Without this ability, the nature of sarcasm can interfere with the effectiveness of certain types of tasks [15].

Sarcastic expressions are defined as the use of words that carry a meaning different than the speaker’s true intention, expressing the explicit opposite of what they say in order to hurt a person’s feelings or criticize them humorously. The main purpose of sarcasm is to implicitly say derogatory and upsetting words through the use of comments that are made to hurt the feelings of others or to attack something in a humorous way and mean the opposite of what is said; with this definition, sarcasm is very important in our daily lives [16]. Sarcasm is an offensive type of irony used with a cutting tone [17]. Across the world, there is a noticeable increasing trend in the incorporation of sarcasm into everyday life. This trend can be attributed to the frequent use of sarcasm in everyday life, but more specifically to social media and the internet [18].

Ironic expressions are the different expression of a situation than the actual feeling. It can also be expressed as the opposite situation when there is an expectation. In conceptual terms, irony is a sarcastic and indirect expression technique in which sarcasm is evident and contains different meanings. The most common use of irony is to express something that is not serious as if it were serious [19]. Irony, in a simple definition, is the creative use of language [20] and is a form of communication often used to express contempt or ridicule, where the speaker conveys a message that is the opposite of its true meaning, usually with the intention of ridiculing or belittling a specific target [21].

Irony and sarcasm are prominent features of human language, deeply rooted in nuanced expression and social interaction. Both phenomena involve saying something that contradicts the intended meaning, but they are used differently and carry different implications. Context plays an indispensable role in the perception and understanding of both irony and sarcasm. Rohanian et al. point to the complex nature of situational irony, which involves not conforming to expectations and then eliciting emotional responses [22]. Sarcasm and irony are sometimes thought to be interchangeable. However, Sulis et al. distinguish between them, stating that sarcasm is usually more overtly aggressive and sharp in tone, while irony can have a more subtle, more sarcastic quality [10]. Sarcasm is a form of irony that occurs when there is a discrepancy between the actual meaning of a text or expression and its intended meaning. This inconsistency typically manifests as dislike, disdain, or contempt [15].

The use of sarcasm and sarcastic expression, such as irony and sarcasm, is present everywhere, in facial expressions, gestures and even in texts. It is subjective, based on a situation or circumstances, people, language and even one’s culture; it is often a combination of positive and negative expressions [23]. For this, the need to understand the context and to identify emotions in this context has emerged [14]. In this direction, such analyses have recently attracted the attention of computer scientists [20].

The concepts of irony and sarcasm are used as an umbrella term that includes sarcasm and sarcastic communication. In the literature, since there is scientific evidence that sarcasm tends to be more aggressive than irony [24,25,26], expressions that tend to be harsher, hurtful, aggressive are considered as sarcasm, while expressions in which what is said and what is intended to be said are opposite to each other [27] are considered as irony.

As machine learning models become increasingly popular for natural language processing applications, there is a growing need for these models to be able to recognize sarcasm in online texts [15]. At this point, although it has limitations in its current form [13], NLP emerges as a field with serious solutions to this problem [1]. Most of these applications involve the use of natural language and research to date has shown successful results in tackling such tasks [28]. However, there has not been exactly the same success for the figurative arts because understanding figurative language is something in which even human intelligence has difficulty [29,30,31,32,33]. One of the communicative phenomena that best represents this problem is the identification of irony [21,34,35,36,37,38,39,40,41,42,43] and sarcasm [18,21,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50].

A significant part of the data of the texts containing such mocking, satirical, sarcastic expressions that people produce or face are encountered in social media environments [51,52,53]. Various studies have used Twitter, Reddit and Facebook for sarcasm detection [27,54,55,56]. Identifying mocking behaviors such as irony and sarcasm in the field of online social networks such as Facebook, Twitter, Instagram, surveys, etc. has become an important responsibility as they affect social and personal relationships [14]. To this end, significant efforts have been made to develop effective computational methods that can be used to detect expressions such as irony and sarcasm in various contexts, including social media [28,57,58], news [59,60,61] and public speeches [62]. However, it has been observed that there is a significant lack of research focusing on the analysis of figurative expressions such as irony and sarcasm in sources that contain content on specific topics, such as column texts. In this regard, there is a noticeable gap in new approaches and research studies aimed at analyzing satirical content such as irony and sarcasm in column texts.

Sentiment analysis is of vital importance in areas such as decision-making, data mining, opinion mining, and information management, and it is widely used across various domains—including telecommunications, medical, environmental, and political fields—as well as in multiple languages [63,64,65,66]. Since social media allows individuals to share their personal opinions on various topics in real time to express themselves to others, there exists a vast amount of raw data that needs to be analyzed. Consequently, manually classifying this data becomes infeasible [20].

Due to the prevalence of sarcasm on social media and the difficulty machines face in detecting it, the topic has attracted significant attention in the field of NLP [21]. Irony and sarcasm detection is a significant computational challenge in NLP and is essential for better understanding, as it serves as an interface for effective communication between machines and humans [14]. Sarcasm detection is an interesting yet challenging task when it comes to text, as the absence of tone, expression, or gestures can result in a discrepancy between what is said and what is actually meant. A sarcastic text may appear humorous or harsh, critical or praising, but all such instances potentially convey sarcasm in some form [67,68]. Sarcasm is an implicit emotional state and requires additional information such as context and multimodality for more accurate detection at the text level [16]. Furthermore, studies in the literature are generally conducted in English, meaning that the results obtained fall into a language classification that has a different linguistic structure to Turkish [20,69], which, compared to other languages on the internet, has limited resources [4].

Studies in the literature have focused on concepts that can be categorized as (i) sarcasm, (ii) irony, (iii) simile, (iv) metaphor, (v) satire, (vi) exaggeration and (vii) humour, depending on the datasets and analysis methods of the studies [28]. Due to the ambiguity of ironic and sarcastic expressions in textual data, they are difficult to detect. Research into the detection of sarcasm and irony has spanned more than a decade, but recent progress has been made. The existing studies have typically focused on languages such as English [40,70,71,72,73,74,75,76], Arabic [76,77], Chinese [78,79] and other languages [71,80,81,82,83,84,85,86,87]. However, research focusing simultaneously on irony and sarcasm expressions in Turkish column texts offering political, economic, cultural, and social analyses has not been encountered. In studies on irony and sarcasm in the Turkish language [88,89,90,91] a binary classification process has been carried out, as in other languages. The similarities between ironic and sarcastic expressions make it difficult to identify them simultaneously. This study aims to identify ironic and sarcastic expressions simultaneously.

When the studies are analyzed in terms of dataset, it is seen that datasets containing short sentences obtained from web platforms such as Twitter, Reddit and Amazon are used [2,9,92,93,94,95,96,97,98]. There are also datasets consisting of a few sentences obtained through web platforms [11,32,99,100,101]. Datasets obtained through web platforms are usually labelled with hashtag field control. There are also manual labelling methods. The existing studies are presented in

Table 1. Existing sarcasm and irony identification studies.

Reference	Methodology	Highlights
[102]	Rule Based	The algorithm employs two modules: semi supervised pattern acquisition for identifying sarcastic patterns that serve as features for a classifier, and a classification stage that classifies each sentence to a sarcastic class. To assign a score to new examples in the test set, we use a k-nearest neighbors strategy.
[103]	Rule Based	After detecting sarcastic tweets, a multi-rule-based approach was used to classify them into four sarcasm types: polite, rude, raging, and deadpan. This approach relies on predefined rules and classifies the tweets based on lexical features, syntactic features, semantic features, and neutral tone with implicit contradiction.
[104]	Machine Learning	A supervised machine learning approach was used for sarcasm detection; the Naive Bayes classification algorithm was applied with different feature categories such as content words, function words, POS-based sarcastic patterns, and their combinations.
[105]	Machine Learning	To detect sarcasm in customer tweets, supervised learning algorithms using feature sets combining function words and content words were applied with Naive Bayes and maximum entropy classifiers.
[106]	Machine Learning	Human vs. machine learning classification performance has been compared under varying amounts of contextual information, and machine performance has been evaluated on balanced and unbalanced, and manually labelled and automatically retrieved datasets.
[107]	Machine Learning	The method uses Naive Bayes classification on a feature space augmented using domain adaptation technique. Features are extracted uniformly from both domains and mapped into source, target, and general versions before training.
[108]	Deep Learning	The study employs a Dual-Channel Network (DC-Net) that separates the input text into emotional words and the remaining context, encodes these parts separately through the literal and implied channels, and identifies sarcasm in the analyzer module by detecting emotional contradictions.
[101]	Deep Learning	In the study, a hybrid deep learning approach was employed that combines features extracted from a soft attention–based bidirectional Long Short-Term Memory (BiLSTM) layer utilizing GloVe word embeddings with a convolutional neural network.
[109]	Deep Learning	A deep learning model has been developed that performs sarcasm classification by utilizing a multi-head self-attention–based BiLSTM network, which takes both automatically learned and handcrafted features as input.
[110]	Deep Learning	The proposed approach is a deep learning method that performs sarcasm classification using a Recurrent neural network (RNN)- Long Short-Term Memory (LSTM) model after applying word embedding to the preprocessed dataset. The model cleans and encodes the training dataset with label and integer encoding, constructs a word vector matrix, feeds this matrix into the LSTM to record the weights, and calculates the accuracy using the test dataset.
[111]	Deep Learning	The study identifies sarcasm in textual data by training and evaluating Convolutional Neural Networks (CNN), RNN, and a hybrid combination of these models, while systematically analyzing the effects of training data size, number of epochs, dropout rates, and different word embeddings on classification performance using the large-scale Reddit corpus.
[112]	Ensemble Learning	The study detects sarcasm in tweets through a two-phase approach: first extracting sentiment and punctuation features followed by chi-square feature selection, and second combining the top 200 TF-IDF features with the selected sentiment and punctuation features, then applying Support Vector Machine in the first phase and a voting classifier in the second phase for classification.
[113]	Ensemble Learning	The study detects sarcasm by leveraging positive sentiment attached to negative situations through two ensemble-based approaches—a voted ensemble classifier and a random forest classifier—trained on a corpus generated via a seeding algorithm, while also incorporating a pragmatic classifier to detect emoticon-based sarcasm.
[114]	Transformer-Based	The study detects sarcasm using the Adversarial and Auxiliary Features-Aware BERT (AAFAB) model, which encodes sentences with Bidirectional Encoder Representations from Transformers (BERT) contextual embeddings, combines them with manually extracted auxiliary features, and applies adversarial training by adding perturbations to input word embeddings for improved generalization.
[115]	Transformer-Based	The study identifies sarcasm in social media conversation texts using the BERT model and compares its performance with alternative approaches on Twitter and Reddit datasets with combined context-response and isolated response texts.
[116]	Transformer-Based	In the study, sarcasm detection was performed by encoding the context and the response through separate BERT layers, processing the response with a BiLSTM, summarizing the context using convolution and BiLSTM, and finally classifying the output through a multi-channel CNN and a fully connected layer.
[117]	Transformer-Based	In the study, LSTM, Gated Recurrent Unit (GRU), and Transformer models were applied to detect sarcasm in Twitter posts, and the best performance was achieved through an ensemble combination of BERT, RoBERTa, XLNet, RoBERTa-large, and ALBERT models.
[118]	Transformer-Based	In the study, BERT and aspect-based sentiment analysis approaches were employed to extract the relationship between the contextual dialogue sequence and the response, and to determine whether the response is sarcastic or not.

in terms of methodology as rule-based approaches, machine learning approaches, deep learning-based approaches, ensemble learning approaches, and transformer-based approaches. A review of the existing studies reveals that the data is predominantly collected from social media platforms, typically consisting of short sentences and retrieved using hashtag-based search methods. It is considered that the nature of these datasets may contribute to the seemingly high performance of the proposed methodologies. Upon examining the employed approaches, transformer-based methods have been observed to yield more successful outcomes. The literature largely focuses on single-label classification tasks, particularly targeting phenomena such as irony or sarcasm.

2. Materials and Methods

In this section, information about the methods used in this study and the dataset used will be presented. In this study, the classification processes were performed with a computing cluster with Nvidia V100 GPU and Xeon 6148 2.40 GHz processor in TÜBİTAK ULAKBİM, High Performance and Grid Computing Centre, Ankara, Turkey. The performance of the models was measured by 10-fold cross-validation and train-split strategy of the dataset used in this study. After the model results were obtained using both 10-fold cross-validation and the train-test split technique, comparisons were conducted based on standard machine learning evaluation metrics. Figure 1 presents a step-by-step representation of the proposed approach. In this study, a high-complexity three-class dataset was utilized, and transformer-based Turkish fine-tuned models were employed in conjunction with an optimization algorithm designed to determine the optimal weighting of these models. The resulting outcomes were analyzed according to standard machine learning evaluation metrics.

2.1. Dataset

The dataset used in this study was derived from column texts, manually annotated, and categorized into three classes: irony, sarcasm, and normal. The dataset consists of paragraphs containing multiple sentences, with 429, 354, and 526 instances belonging to irony, sarcasm, and normal classes, respectively. The column texts were obtained from publicly accessible online newspaper websites such as www.milliyet.com.tr (access date: 7 November 2025) [119] and www.cumhuriyet.com.tr (access date: 7 November 2025) [120]. During this study, a total of 3000 column texts written by different authors were examined.

In this study, to investigate the influence of contextual information on the target class, several datasets were constructed based on the number of sentences included. The labelling of the dataset occurs in four stages: the artificial intelligence assistant performs pre-labelling, the human user reviews the labels and filters them, the linguist processes the labelled examples, and any incorrect labels are corrected. In the first stage, due to the time and resource costs of traditional data labelling, AI-supported labelling comes to the fore. Examples of labelling types include zero-shot labelling, few-shot labelling, and instruction-based labelling. Instruction-based labelling is also referred to in the literature as instruction-based fine-tuning or supervised fine-tuning. This fine-tuning is used in example labelling by large language models trained to predict the next word or generate information, following specific instructions set by users. The method consists of instruction and output sections and relies on supervised fine-tuning over specific examples. This ensures that the fine-tuned language model produces the desired output within certain constraints, in line with the instructions [121] Due to the size of the texts to be examined and analysed, the instruction-based labelling method was preferred. An assistant named SarcaTurkGPT, created using a large language model (OpenAI GPT-4o) by a linguist, was used to detect expressions containing irony and sarcasm. During the development of the assistant, the definitions of irony and sarcasm were provided, their differences were highlighted, and the assistant was designed to analyze user-submitted texts and return sentences that potentially contain irony or sarcasm. The temperature parameter was set to 0.5 to ensure more balanced and consistent responses, while the other parameters were kept at their default values (top_p: 1, frequency_penalty: 0, and presence_penalty: 0). The assistant analyses the column texts provided to it and ensures pre-labelling by flagging texts that may contain irony and sarcasm. In the second stage, the examples pre-labelled by SarcaTurkGPT are checked by a human user and marked as belonging to the appropriate class. In the third stage, after the labelling process by SarcaTurkGPT and human users, the examples are checked by the linguist. In the fourth stage, if there are errors in the labelling of the examples checked by the linguist, they are re-examined and labelled by the human user according to the linguist’s recommendations.

The monogram dataset contains only the ironic, sarcastic, or neutral sentence itself. The bigram dataset includes the target sentence along with the immediately preceding sentence. The trigram, quadrigram, and omnigram datasets incorporate two, three, and all preceding sentences, respectively, prior to the target sentence. This configuration enables the examination of how the sentences preceding the ironic or sarcastic sentence affect classification performance. The dataset architecture is presented in Figure 2.

Examples of the bigram and trigram datasets prepared for use in this study are presented in Table 2.

Table 2. Example of the bigram and trigram datasets used in this study.

Class	Monogram Sample	Bigram Sample	Trigram Sample	Paragraph
Irony	Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (So you too can be as happy as a Dane.)	Yumuşak bir aydınlatma, hoş kokulu mumlar, lezzetli yemekler, samimi sohbetler, sıcak tutan yumuşak giysiler… Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (Soft lighting, fragrant candles, delicious food, heartfelt conversations, warm and cozy clothes… So you too can be as happy as a Dane.)	Üstelik yapılması gerekenler hiç de zor sayılmazdı. Yumuşak bir aydınlatma, hoş kokulu mumlar, lezzetli yemekler, samimi sohbetler, sıcak tutan yumuşak giysiler… Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (Moreover, what needed to be done was not difficult at all. Soft lighting, fragrant candles, delicious food, heartfelt conversations, warm and cozy clothes… So you too can be as happy as a Dane.)	Kitabı çok satanlar listesine taşıyan ve Türkçe dahil pek çok dilde yayınlanıp popülerleşmesini sağlayan da kuşkusuz insanların bu mutluluk formüllerini öğrenme ihtiyacıydı. Üstelik yapılması gerekenler hiç de zor sayılmazdı. Yumuşak bir aydınlatma, hoş kokulu mumlar, lezzetli yemekler, samimi sohbetler, sıcak tutan yumuşak giysiler… Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (What propelled the book onto bestseller lists and led to its publication and popularity in many languages, including Turkish, was undoubtedly people’s need to learn these formulas for happiness. Moreover, what needed to be done was not difficult at all. Soft lighting, fragrant candles, delicious food, heartfelt conversations, warm and cozy clothes… So you too can be as happy as a Dane.)
Sarcasm	Ne de olsa “Alamancı” onun bakışı bir başka… (After all, as an “Alamancı”, her perspective is different…)	Bir de dizileri uzun tutuyorlar, kısa yapsınlar! diyordu, o sevimli Türkçesiyle. Ne de olsa “Alamancı” onun bakışı bir başka… (Also, they make the TV series too long—they should make them shorter!” After all, as an “Alamancı”, her perspective is different…)	Üç yıldır Türkiye’deyim, ilk defa bir ödül alıyorum. Bir de dizileri uzun tutuyorlar, kısa yapsınlar! diyordu, o sevimli Türkçesiyle. Ne de olsa “Alamancı” onun bakışı bir başka… (“I’ve been in Turkey for three years, and this is the first time I’m receiving an award. Also, they make the TV series too long—they should make them shorter!” After all, as an “Alamancı”, her perspective is different…)	Oysa “Hürrem Sultan” Meryem Uzerli’nin dediği başka. Üç yıldır Türkiye’deyim, ilk defa bir ödül alıyorum. Bir de dizileri uzun tutuyorlar, kısa yapsınlar! diyordu, o sevimli Türkçesiyle. Ne de olsa “Alamancı” onun bakışı bir başka… (Yet what “Hürrem Sultan” Meryem Uzerli says is different. She was saying, in her cute Turkish: “I’ve been in Turkey for three years, and this is the first time I’m receiving an award. Also, they make the TV series too long—they should make them shorter!” After all, as an “Alamancı”, her perspective is different…)
Normal	Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)	Her bir rating altın değil pırlanta değerinde. Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (Every single rating point is not just worth gold, but diamonds. It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)	Her gün rating karnesi alan onlarca yapım şirketi kıyasıya mücadele ediyorlar. Her bir rating altın değil pırlanta değerinde. Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (Dozens of production companies, each receiving daily ratings reports, compete fiercely. Every single rating point is not just worth gold, but diamonds. It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)	Şaşkınlık verici çünkü Türkiye’de rekabetin en sert ve en görünür olduğu yerlerden biri dizi sektörü. Her gün rating karnesi alan onlarca yapım şirketi kıyasıya mücadele ediyorlar. Her bir rating altın değil pırlanta değerinde. Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (It is astonishing, because one of the places where competition is the toughest and most visible in Turkey is the TV series sector. Dozens of production companies, each receiving daily ratings reports, compete fiercely. Every single rating point is not just worth gold, but diamonds. It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)

The basic statistics of the dataset used this study according to classes are presented in Figure 3.

Table 2. Example of the bigram and trigram datasets used in this study.

Class	Monogram Sample	Bigram Sample	Trigram Sample	Paragraph
Irony	Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (So you too can be as happy as a Dane.)	Yumuşak bir aydınlatma, hoş kokulu mumlar, lezzetli yemekler, samimi sohbetler, sıcak tutan yumuşak giysiler… Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (Soft lighting, fragrant candles, delicious food, heartfelt conversations, warm and cozy clothes… So you too can be as happy as a Dane.)	Üstelik yapılması gerekenler hiç de zor sayılmazdı. Yumuşak bir aydınlatma, hoş kokulu mumlar, lezzetli yemekler, samimi sohbetler, sıcak tutan yumuşak giysiler… Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (Moreover, what needed to be done was not difficult at all. Soft lighting, fragrant candles, delicious food, heartfelt conversations, warm and cozy clothes… So you too can be as happy as a Dane.)	Kitabı çok satanlar listesine taşıyan ve Türkçe dahil pek çok dilde yayınlanıp popülerleşmesini sağlayan da kuşkusuz insanların bu mutluluk formüllerini öğrenme ihtiyacıydı. Üstelik yapılması gerekenler hiç de zor sayılmazdı. Yumuşak bir aydınlatma, hoş kokulu mumlar, lezzetli yemekler, samimi sohbetler, sıcak tutan yumuşak giysiler… Böylece siz de bir Danimarkalı kadar mutlu olabilirsiniz. (What propelled the book onto bestseller lists and led to its publication and popularity in many languages, including Turkish, was undoubtedly people’s need to learn these formulas for happiness. Moreover, what needed to be done was not difficult at all. Soft lighting, fragrant candles, delicious food, heartfelt conversations, warm and cozy clothes… So you too can be as happy as a Dane.)
Sarcasm	Ne de olsa “Alamancı” onun bakışı bir başka… (After all, as an “Alamancı”, her perspective is different…)	Bir de dizileri uzun tutuyorlar, kısa yapsınlar! diyordu, o sevimli Türkçesiyle. Ne de olsa “Alamancı” onun bakışı bir başka… (Also, they make the TV series too long—they should make them shorter!” After all, as an “Alamancı”, her perspective is different…)	Üç yıldır Türkiye’deyim, ilk defa bir ödül alıyorum. Bir de dizileri uzun tutuyorlar, kısa yapsınlar! diyordu, o sevimli Türkçesiyle. Ne de olsa “Alamancı” onun bakışı bir başka… (“I’ve been in Turkey for three years, and this is the first time I’m receiving an award. Also, they make the TV series too long—they should make them shorter!” After all, as an “Alamancı”, her perspective is different…)	Oysa “Hürrem Sultan” Meryem Uzerli’nin dediği başka. Üç yıldır Türkiye’deyim, ilk defa bir ödül alıyorum. Bir de dizileri uzun tutuyorlar, kısa yapsınlar! diyordu, o sevimli Türkçesiyle. Ne de olsa “Alamancı” onun bakışı bir başka… (Yet what “Hürrem Sultan” Meryem Uzerli says is different. She was saying, in her cute Turkish: “I’ve been in Turkey for three years, and this is the first time I’m receiving an award. Also, they make the TV series too long—they should make them shorter!” After all, as an “Alamancı”, her perspective is different…)
Normal	Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)	Her bir rating altın değil pırlanta değerinde. Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (Every single rating point is not just worth gold, but diamonds. It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)	Her gün rating karnesi alan onlarca yapım şirketi kıyasıya mücadele ediyorlar. Her bir rating altın değil pırlanta değerinde. Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (Dozens of production companies, each receiving daily ratings reports, compete fiercely. Every single rating point is not just worth gold, but diamonds. It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)	Şaşkınlık verici çünkü Türkiye’de rekabetin en sert ve en görünür olduğu yerlerden biri dizi sektörü. Her gün rating karnesi alan onlarca yapım şirketi kıyasıya mücadele ediyorlar. Her bir rating altın değil pırlanta değerinde. Kanalların ve yönetimlerinin geleceğini belirliyor, yapım şirketlerinin milyonlarca dolar kâr veya zarar etmesine neden oluyor. (It is astonishing, because one of the places where competition is the toughest and most visible in Turkey is the TV series sector. Dozens of production companies, each receiving daily ratings reports, compete fiercely. Every single rating point is not just worth gold, but diamonds. It determines the future of TV channels and their management, and causes production companies to make millions of dollars in profit or loss.)

The basic statistics of the dataset used this study according to classes are presented in Figure 3.

Figure 3. Average and Median Distributions of Words by Classes.

Figure 3. Average and Median Distributions of Words by Classes.

The thematic distribution statistics of the dataset used in this study are presented in Figure 4. The class distributions are provided under the headings of politics, economy, health, cultural, and others, with the examples predominantly derived from politically themed column texts.

2.2. Modeling

This section will provide information regarding the methods used in this study.

2.2.1. Model Selection

BERTurk, ConvBERTurk, DistilBERTurk, RoBERTaTurk and ELECTRATurk, which are transformer-based models fine-tuned with Turkish language texts, were used in this study. In addition, the Artificial Rabbit Optimization Algorithm (ARO) was used to optimise the weights of these models for ensemble prediction. During the training of the transformer-based models, the learning rate was set to 2 × 10⁻⁵, the batch size to 8, and the maximum sequence length to 512. The number of training epochs was dynamically adjusted according to the learning process, typically ranging between 15 and 20.

BERTurk

Bidirectional Encoder Representations from Transformers (BERT) is a model introduced by Google in 2018 that revolutionized the field of NLP [122] One of its main features is its two-way understanding of words. In this way, it produces effective results in meaning transfer by better understanding the context. The model shows high success rates in various tasks such as text classification, sentiment analysis, and question-answer systems. The pre-training and fine-tuning processes of the BERT model are presented in Figure 5. BERTurk is an NLP model with Turkish language understanding and processing capabilities [123]. This model, based on the BERT architecture, has been specially developed to effectively analyze the language structures and features of Turkish. The agglutinative structure of Turkish requires a special approach due to the formation of words with roots and affixes. In this context, BERTurk has been trained in accordance with the morphological and syntactic features of the language. Trained with a large corpus of 35 GB of Turkish text containing about 4 billion tokens, the model successfully understands different language usages and contexts thanks to the diversity of the datasets.

ConvBERTurk

ConvBERT is a language model developed in the field of NLP by combining the BERT architecture with conventional layers [124]. This model aims to provide more meaningful and effective language processing capabilities by combining contextual and local features. The structure of the ConvBERT model is presented in Figure 6. ConvBERTurk is a model developed for Turkish NLP with Turkish comprehension and processing capabilities [123].

DistilBERTurk

DistilBERT is a lightened version of the BERT model [125]. Trained with ‘knowledge distillation’ techniques, this model runs faster and consumes fewer resources, providing efficiency in NLP applications. DistilBERT offers significant advantages in terms of performance and speed by reducing the number of parameters while maintaining the basic building blocks of BERT. In the training process, it benefits from BERT’s large datasets and develops a lighter structure without losing its basic capabilities by learning from the BERT model in the ‘teacher’ position [126]. The architectural structure of the model is presented in Figure 7. In this study, the DistilBERTurk model trained with the Turkish dataset was used.

ELECTRATurk

Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA) is a pre-trained NLP model developed in 2020 [128]. ELECTRA aims to combine the speed and efficiency of models such as BERT with high performance and has excelled in numerous NLP tasks [129]. The model is based on two main components: Generator and Discriminator. The Generator is used to replace certain words, while the Discriminator analyses these replaced words and tries to make correct predictions. Thanks to this mechanism, ELECTRA works on a task called “Replaced Token Detection” (RTD) instead of “Masked Language Modelling” (MLM). This process improves the training efficiency of the language model and is particularly advantageous for learning syntactic contexts [130]. In the training process of ELECTRA, the model not only understands the data, but also the context of the data and makes accurate word predictions. Research shows that ELECTRA has a distinct advantage over traditional BERT training methods and learns more information with less computational power [131]. The structure used by the model is presented in Figure 8. The ELECTRATurk model, which was trained with the Turkish dataset and prepared for Turkish NLP, was used in this study.

RoBERTaTurk

RoBERTa (A Robustly Optimized BERT Pretraining Approach) is an advanced transformer-based bidirectional training mechanism that has shown significant success in the field of NLP by improving the pretraining procedure of BERT [132]. This model offers improvements such as training with longer and larger datasets, removing the next sentence prediction (NSP) target, working with longer sequences and using dynamic masking. RoBERTa aims to achieve higher performance on various NLP tasks by making systematic changes to the BERT architecture and training method. It has also achieved the best results on standard NLP benchmarks such as GLUE, SQuAD and RACE. In this study, the RoBERTaTurk model trained with Turkish dataset and adapted for Turkish NLP is used.

Artificial Rabbits Optimization Algorithm

In the work presented by [133] a new bio-inspired meta-heuristic algorithm called ‘Artificial Rabbits Optimization’ (ARO) is proposed to solve engineering optimization problems. The algorithm is inspired by the survival strategies of rabbits in nature: foraging away from their burrows to avoid predators (‘detour foraging’) and choosing a random burrow to hide from enemies (‘random hiding’). By modelling these mechanisms, the algorithm significantly improves the global and local search capabilities and performs well in solving engineering problems. The flowchart of the algorithm adapted for ensemble learning is presented in Figure 9.

2.2.2. Ensemble Learning Model

BERTurk, ConvBERTurk, DistilBERTurk, RoBERTaTurk and ELECTRATurk, which are transformer-based models fine-tuned with Turkish language texts, were used in this study. In addition, the Artificial Rabbit Optimization Algorithm (ARO) was used to optimize the weights of these models for ensemble prediction.

2.3. Evaluation Metrics

An example with a positive label in the dataset is predicted as positive, which is referred to as a True Positive (TP). When considering the irony class, it means that an expression that is ironic is also predicted as ironic by the model. An example that is labelled as negative but is predicted as positive is referred to as a False Positive (FP). This is the situation where an expression that is not actually ironic is incorrectly labelled as ironic by the model. When an example that is positive is predicted to be negative, this is referred to as a False Negative (FN). This is when an expression that is ironic is evaluated by the model as not being ironic. When an example that is negative is predicted to be negative, this is referred to as a True Negative (TN). This is when an expression that is not actually ironic is evaluated by the model as not being ironic. The same evaluation method is applied to the sarcasm and normal classes as well; for each class, TP, FP, FN, and TN are defined similarly based on whether the model correctly or incorrectly predicts the corresponding label. To evaluate the results of this study, accuracy, precision, recall and F-score metrics used in similar tasks in the literature were used. A three-class classification was performed, and all evaluation metrics were calculated over these three classes. Related metrics can be seen in Equations (1)–(4). The Cross-Entropy Loss function, as presented in Equation (5), was utilized during model training. In this formulation, C represents the total number of classes, y_i denotes the ground-truth label for the i-th class, and p_i corresponds to the predicted probability of the i-th class. This loss function guides the model toward correct classification by minimizing the divergence between the predicted probability distribution and the actual class labels.

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\left(\mathrm{F}\mathrm{P}\right)+\left(\mathrm{F}\mathrm{N}\right)}}$$

(1)

$$Precision={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(2)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(3)

$$\mathrm{F}-\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

(4)

$$\mathrm{L}={\sum }_{i=1}^C{y}_i\mathit{log}\left(p_i\right)$$

(5)

3. Results and Discussion

In this study, five different pre-trained models—BERTurk, ConvBERTurk, DistilBERTurk, ELECTRATurk, and RoBERTaTurk—were fine-tuned using a custom-designed dataset and evaluated for a text classification task. Additionally, an Artificial Rabbit Optimization (ARO) method was employed to combine the outputs of these models, resulting in a final prediction that leverages the strengths of the individual models. To achieve the optimal performance of the ensemble learning approach, the number of rabbits and iterations in the ARO method were tuned using the grid search optimization technique.

This study conducted various analyses on monogram, bigram, trigram, quadrigram, and omnigram data set architectures. Accordingly, experimental studies were carried out exclusively using these two pieces of architecture. The classification performance of the fine-tuned models was comprehensively evaluated using Accuracy, Precision, Recall, and F-Score metrics, providing a detailed assessment of their effectiveness in the given task. The fine-tuned models were first evaluated using the 10-fold cross-validation method, and the corresponding metric values are presented in

Table 3. Comparative Evaluation of Transformer-Based Models for Turkish Language Tasks.

Model	Dataset	Accuracy (%)	Precision (%)	Recall (%)	F-Score (%)
BERTurk	monogram	65.63	63.19	61.89	62.58
ConvBERTurk		60.89	59.56	58.19	58.88
DistilBERTurk		57.45	55.61	54.89	55.22
ELECTRATurk		59.59	55.30	56.56	55.87
RoBERTaTurk		54.32	50.93	50.72	50.80
BERTurk	bigram	66.72	64.72	63.5	64.12
ConvBERTurk		65.3	64.39	62.00	63.18
DistilBERTurk		60.91	57.81	57.52	57.66
ELECTRATurk		65.53	64.08	61.93	62.99
RoBERTaTurk		61.24	60.99	56.83	58.84
BERTurk	trigram	67.42	64.73	64.32	64.52
ConvBERTurk		65.62	61.58	61.97	61.77
DistilBERTurk		62.71	60.23	59.63	59.93
ELECTRATurk		66.17	63.74	62.96	63.35
RoBERTaTurk		61.16	56.84	56.61	56.72
BERTurk	quadrigram	67.07	65.08	63.99	64.50
ConvBERTurk		68.76	66.29	65.19	65.75
DistilBERTurk		66.39	62.70	62.40	62.57
ELECTRATurk		67.00	62.22	63.72	62.93
RoBERTaTurk		61.11	57.62	56.51	57.05
BERTurk	omnigram	70.28	68.35	67.40	67.90
ConvBERTurk		70.74	67.99	67.74	67.84
DistilBERTurk		66.47	63.22	63.20	63.22
ELECTRATurk		67.76	65.05	64.87	64.96
RoBERTaTurk		61.88	52.00	57.03	54.39

.

The results in Table 3 show the comparative classification performances of Transformer-based models in the Turkish language when used with context width dataset architectures. When the results of the dataset were analyzed, BERTurk was the most successful model according to performance metrics, achieving an accuracy value of 65.63% on the monogram dataset, 66.72% on the bigram dataset, and 67.42% on the trigram dataset. ConvBERTurk was the most successful model, achieving an accuracy value of 68.76% on the quadrigram dataset and an accuracy value of 70.74% on the omnigram dataset. The accuracy, precision, sensitivity, and F-Score metric values achieved by the BERTurk model on the monogram dataset and the metric values achieved by the ConvBERTurk model on the omnigram dataset increased by 7.79% for the accuracy metric, 7.60% for the precision metric, 9.45% for the sensitivity metric, and 8.41% for the F-Score metric, respectively. and 8.41% for the F-score metric. This performance increase demonstrates the positive effect of contextual history on classification in detecting irony and sarcasm. A consistent performance increase is observed across the most successful models, from the most basic monogram dataset architecture to the omnigram dataset.

Based on the results obtained from the monogram dataset, the most successful model was BERTurk, achieving 65.63% accuracy, 63.19% precision, 61.89% recall, and an F-score of 62.58%. In contrast, RoBERTaTurk ranked lowest in terms of performance, with 54.32% accuracy, 50.93% precision, 50.72% recall, and an F-score of 50.80%. Based on the results obtained from the bigram dataset, the most successful model was BERTurk, achieving 66.72% accuracy, 64.72% precision, 63.50% recall, and an F-score of 64.10%. In contrast, RoBERTaTurk ranked lowest in terms of performance, with 61.24% accuracy, 60.99% precision, 56,83% recall, and an F-score of 58.84%. Based on the results obtained from the trigram dataset, the most successful model was BERTurk, achieving 67.42% accuracy, 64.73% precision, 64.32% recall, and an F-score of 64.52%. In contrast, RoBERTaTurk ranked lowest in terms of performance, with 61.16% accuracy, 56.84% precision, 56.61% recall, and an F-score of 56.72%. Based on the results obtained from the quadrigram dataset, the most successful model was ConvBERTurk, achieving 68.76% accuracy, 66.29% precision, 65.19% recall, and an F-score of 65.75%. In contrast, RoBERTaTurk ranked lowest in terms of performance, with 61.11% accuracy, 57.62% precision, 56.51% recall, and an F-score of 57.05%. Based on the results obtained from the omnigram dataset, the most successful model was ConvBERTurk, achieving 70.74% accuracy, 67.99% precision, 67.74% recall, and an F-score of 67.84%. In contrast, RoBERTaTurk ranked lowest in terms of performance, with 61.88% accuracy, 52.00% precision, 57.03% recall, and an F-score of 54.39%.

The metric values achieved by the two most successful models, BERTurk and ConvBERTurk, on the datasets are presented in

Table 4. A Comparative Evaluation of the Two Most Successful Models, BERTurk and ConvBERTurk, Across the Datasets.

Model	Dataset	Accuracy (%)	Precision (%)	Recall (%)	F-Score (%)
BERTurk	monogram	65.63	63.19	61.89	62.58
BERTurk	bigram	66.72	64.72	63.50	64.10
BERTurk	trigram	67.42	64.73	64.32	64.52
BERTurk	quadrigram	67.07	65.08	63.99	64.50
BERTurk	omnigram	7028	68.35	67.40	67.90
ConvBERTurk	monogram	60.89	59.56	58.19	58.88
ConvBERTurk	bigram	65.30	64.39	62.00	63.17
ConvBERTurk	trigram	65.62	61.58	61.97	61.77
ConvBERTurk	quadrigram	68.76	66.29	65.19	65.75
ConvBERTurk	omnigram	70.74	67.99	67.74	67.84

. Although the BERTurk model did not achieve a performance increase when transitioning from the trigram dataset to the quadrigram dataset, an increase in metrics was observed with the expansion of context. The metric values obtained in the omnigram dataset show an increase of 7.09% for the accuracy metric, 8.16% for the precision metric, 8.90% for the sensitivity metric, and 8.50% for the F-Score metric compared to the monogram dataset. The ConvBERTurk model achieved performance improvements at every step with the expansion of context between the monogram and omnigram datasets. It was observed that the metrics achieved in the omnigram dataset increased by 16.19% for the accuracy metric, 14.16% for the precision metric, 16.41% for the sensitivity metric, and 15.22% for the F-Score metric compared to the monogram dataset.

When examining the impact of the dataset architecture, it is observed that the BERTurk and ConvBERTurk models, when evaluated on the monogram and omnigram datasets, yield improvements in performance metrics. Although the RoBERTaTurk model showed the lowest performance across all datasets, its performance increased as the context of the dataset increased. The findings indicate that both the model architecture and the preferred dataset type play a decisive role in classification performance. Performance comparisons of monogram and omnigram dataset architectures are presented in Figure 10 and Figure 11.

Secondly, after splitting the dataset into training and testing sets, model-specific weights were determined via ARO based on their predictions, and a final prediction was generated by capitalizing on the strengths of the individual models. The metric values computed for the predictions made through ensemble learning are presented in

Table 5. Comparative Evaluation of Transformer-Based Models and the Proposed Ensemble Learning Method for Turkish Language Tasks.

Model	Dataset	Accuracy (%)	Precision (%)	Recall (%)	F-Score (%)	Weights
BERTurk	monogram	58.02	54.55	54.62	54.58	0.2916
ConvBERTurk		54.20	60.31	57.97	59.12	0.0671
DistilBERTurk		61.45	53.58	51.80	52.48	0.0086
ELECTRATurk		56.49	40.83	55.62	47.02	0.6122
RoBERTaTurk		61.45	52.48	49.71	51.06	0.0205
Average		58.32	52.75	53.94	52.85	-
Ensemble Model		65.25	63.75	60.55	62.12	-
BERTurk	bigram	61.07	56.26	56.50	56.38	0.0525
ConvBERTurk		62.21	59.26	58.45	58.85	0.0369
DistilBERTurk		57.63	52.68	53.22	52.95	0.0601
ELECTRATurk		64.89	59.17	59.70	59.43	0.4594
RoBERTaTurk		56.49	58.91	54.60	56.68	0.3912
Average		60.45	57.25	56.49	56.85	-
Ensemble Model		69.84	66.42	65.18	65.80	-
BERTurk	trigram	62.50	61.68	60.44	61.06	0.0434
ConvBERTurk		65.27	61.51	61.12	61.31	0.0761
DistilBERTurk		64.50	64.32	61.58	62.93	0.4505
ELECTRATurk		66.41	63.04	62.66	62.85	0.4267
RoBERTaTurk		59.24	58.65	57.03	57.83	0.0032
Average		63.58	61.84	60.56	61.19	-
Ensemble Model		71.37	69.78	67.70	68.73	-
BERTurk	quadrigram	64.12	59.90	59.60	59.75	0.1992
ConvBERTurk		61.83	62.20	61.70	61.95	0.2988
DistilBERTurk		66.03	62.95	63.00	62.98	0.3102
ELECTRATurk		66.79	59.77	60.30	60.03	0.0991
RoBERTaTurk		65.27	57.29	56.80	57.04	0.0927
Average		64.80	60.42	60.28	60.35
Ensemble Model		72.13	70.78	67.81	69.27	-
BERTurk	omnigram	69.85	66.98	66.31	66.64	0.0639
ConvBERTurk		71.76	68.32	68.31	68.32	0.4383
DistilBERTurk		66.41	61.10	60.64	60.87	0.0346
ELECTRATurk		70.23	69.24	65.76	67.46	0.4601
RoBERTaTurk		58.40	50.29	52.29	51.27	0.0031
Average		67.33	63.18	62.66	62.91	-
Ensemble Model		76.71	74.64	73.29	73.96	-

.

When Table 5 is examined, it is seen that the Ensemble approach reached the highest values in all metrics in all datasets. In the monogram dataset, the results obtained from the models were presented. According to these results, the ensemble learning model outperformed the others, achieving 65.25% accuracy, 63.75% precision, 60.55% recall, and an F-Score of 62.12%. Compared to the second-best performing model, DistilBERTurk demonstrated improvements of 6.19% in accuracy, 18.97% in precision, 16.87% in recall, and 18.38% in F-Score, respectively. When examining the average performance metrics of the transformer-based models, the ensemble learning approach yielded superior results, with increases of 11.86% in accuracy, 20.87% in precision, 12.27% in recall, and 17.54% in F-Score compared to the average values. In the ensemble learning method, the ARO algorithm was employed, where the number of rabbits was searched within the range of 50 to 1000, and the model was trained using 50 rabbits and 50 iterations.

In the bigram dataset, the results obtained from the models were presented. According to these results, the ensemble learning model outperformed the others, achieving 69.84% accuracy, 66.42% precision, 65.18% recall, and an F-Score of 65.80%. Compared to the second-best performing model, ELECTRATurk, it demonstrated improvements of 7.64% in accuracy, 12.25% in precision, 9.18% in recall, and 10.72% in F-Score, respectively. When examining the average performance metrics of the transformer-based models, the ensemble learning approach yielded superior results, with increases of 15.55% in accuracy, 16.03% in precision, 15.38% in recall, and 15.74% in F-Score compared to the average values. In the ensemble learning method, the ARO algorithm was employed using 130 rabbits and 50 iterations.

In the trigram dataset, the results obtained from the models were presented. According to these results, the ensemble learning model outperformed the others, achieving 71.37% accuracy, 69.78% precision, 67.70% recall, and an F-Score of 68.73%. Compared to the second-best performing model, ConvBERTurk demonstrated improvements of 9.34% in accuracy, 13.45% in precision, 10.77% in recall, and 12.10% in F-Score, respectively. When examining the average performance metrics of the transformer-based models, the ensemble learning approach yielded superior results, with increases of 12.25% in accuracy, 12.84% in precision, 11.79% in recall, and 12.32% in F-Score compared to the average values. In the ensemble learning method, the ARO algorithm was employed using 190 rabbits and 50 iterations.

In the quadigram dataset, the results obtained from the models were presented. According to these results, the ensemble learning model outperformed the others, achieving 72.13% accuracy, 70.78% precision, 67.81% recall, and an F-Score of 69.27%. Compared to the second-best performing model, ELECTRATurk, it demonstrated improvements of 8.00% in accuracy, 18.41% in precision, 12.45% in recall, and 15.39% in F-Score, respectively. When examining the average performance metrics of the transformer-based models, the ensemble learning approach yielded superior results, with increases of 11.31% in accuracy, 17.15% in precision, 12.49% in recall, and 14.78% in F-Score compared to the average values. In the ensemble learning method, the ARO algorithm was employed using 610 rabbits and 50 iterations.

In the omnigram dataset, the results obtained from the models were presented. According to these results, the ensemble learning model achieved the best performance, with 76.71% accuracy, 74.64% precision, 73.29% recall, and an F-Score of 73.96%. Compared to the second-best performing model, ConvBERTurk, it showed improvements of 6.90% in accuracy, 9.25% in precision, 7.29% in recall, and 8.25% in F-Score, respectively. Based on the average performance metric values of the transformer-based models, the ensemble learning approach produced superior results, outperforming the averages by 13.93% in accuracy, 18.14% in precision, 16.96% in recall, and 17.56% in F-Score. In the ensemble learning method, the ARO algorithm was employed using 390 rabbits and 50 iterations.

As shown in Figure 11, the ensemble learning method achieved an accuracy of 66.41% on the monogram dataset, and it is observed that as the contextual width increased in the subsequent datasets, the obtained accuracy performance also improved. The increasing trend in performance reached its highest level in the omnigram dataset, which represents the dataset with the widest contextual range, achieving 76.71% accuracy. Similarly, when the results of the monogram and omnigram datasets are compared, a noticeable improvement in performance is observed across the other evaluation metrics as well.

4. Conclusions

In this study, we evaluate the performance of five different pre-trained Transformer-based models—BERTurk, ConvBERTurk, DistilBERTurk, ELECTRATurk, and RoBERTaTurk—on a Turkish text classification task and test these models on specially prepared bigram and trigram datasets with fine-tuning. In the experimental process, an ARO-based ensemble learning approach that combines the outputs of the models to improve classification performance was also applied.

When the results obtained by the models on the datasets using the cross-validation technique are examined, it is observed that as the contextual width increased—from the monogram dataset with the narrowest context to the omnigram dataset with the widest context—the accuracy, precision, recall, and F-score metrics respectively improved from 65.63%, 63.19%, 61.89%, and 62.58% to 70.74%, 67.99%, 67.74%, and 67.84%. Among the models, BERTurk achieved the best performance on the monogram, bigram, and trigram datasets, while ConvBERTurk performed best on the quadrigram and omnigram datasets. Across all datasets, the RoBERTaTurk model ranked last in terms of performance. The BERTurk model achieved 65.63% accuracy on the monogram dataset and 70.28% on the omnigram dataset. Similarly, in terms of accuracy, ConvBERTurk improved from 60.89% to 70.74%, DistilBERTurk from 57.45% to 66.47%, ELECTRATurk from 59.59% to 67.76%, and RoBERTaTurk from 54.32% to 61.66%, all demonstrating an upward trend in performance. In general, as the context expanded from bigram to trigram and quadrigram datasets, the models exhibited either a limited performance improvement or results similar to those of the previous dataset. However, in the omnigram dataset, which possesses the widest contextual range, all models showed a distinct increase in performance.

When the results obtained by the transformer-based models on the datasets were examined according to the 80:20 training–test split, it was observed that as the contextual width of the datasets increased, the performance metrics of the individual models also improved. This performance improvement of the individual models was reflected in the ensemble learning method, which combined their predictions to produce the final outcomes. As the performance of the individual models increased, the ensemble learning method achieved 65.25% accuracy on the monogram dataset with the narrowest context, and 76.71% accuracy on the omnigram dataset with the widest context. The obtained results indicate that the ensemble learning approach successfully and effectively integrated the predictions of individual models, achieving higher overall performance across all datasets compared to any single model.

The ensemble learning method, which enhances the performance of individual models, may pose challenges for real-time applications due to its requirements for extensive computational resources, processing time, and hardware capacity, as it involves the training and performance evaluation of all models.

Overall, this study demonstrates that both the type of model employed, and the architecture of the dataset play a decisive role in text classification performance, and that when these two factors are supported by ensemble methods such as ARO, performance can be significantly enhanced. The findings indicate that, particularly in low-resource languages such as Turkish, the joint consideration of model selection, data structure, and ensemble strategies is essential when developing deep learning-based solutions.