Multi-Task Classification of Hebrew News Articles: A Comparative Study of Classical ML and BERT Models in a Morphologically Rich, Low-Resource Setting

Abstract

The automated classification of Hebrew, a morphologically rich language (MRL), presents unique challenges, particularly when high-quality labeled data are scarce. This study investigates the interplay between handcrafted feature engineering and transformer-based representations in a low-resource news classification setting (n = 306). We evaluate a multi-task classification across four distinct dimensions: domain, sentiment, gender, and source. Our methodology employs an extensive feature space of 2149 stylistic and content-based attributes, optimized through a systematic Hill-Climbing selection process. We contrast five classical machine learning architectures with five BERT-based models, integrating five oversampling strategies to mitigate class imbalance. The results reveal that in scenarios of extreme data scarcity, the performance gap between deep learning and optimized classical ML becomes marginal, with stylistic features providing critical stability and interpretability. This study contributes a curated Hebrew news dataset and establishes a robust benchmark, demonstrating that linguistically aware feature engineering remains a vital component for MRL processing when large-scale fine-tuning is impractical.

1. Introduction

Advancements in technology have led to a deluge of news articles, the majority of which remain unlabeled. Consequently, there is a critical need for automated systems capable of accurately classifying news articles. Recent studies have explored diverse approaches to news classification, each providing unique insights and methodologies. One prominent technique involves leveraging machine learning (ML) and deep learning (DL) methods, which have proved effective in processing and classifying text data (e.g., [1,2]). Researchers have explored the integration of natural language processing (NLP) techniques, such as topic modeling [3] and sentiment analysis [4,5], to enhance the accuracy and granularity of news classification. Another important avenue is classifying news types (e.g., [6]).

Numerous news websites publish a large volume of news articles online daily. These articles can be classified based on a wide variety of factors, such as the author’s profile (e.g., age group, gender, and political affiliation), domain, sentiment intensity, and source website.

In this study, we classify Hebrew news articles according to their domain, sentiment, author gender, and source. We constructed a dataset of news articles downloaded from three news websites (see Section 3).

We hypothesize that text classification (TC) models applied to supervised news datasets can successfully perform classification across labels such as gender, domain, website, and sentiment. The research’s key contributions and novelties include:

(1) The construction of a new labeled dataset of online news articles written in Hebrew, a language for which studies on news classification and relevant datasets are scarce. Furthermore, Hebrew possesses a significantly richer morphology than English [7,8]. Hebrew’s complex morphology, characterized by frequent prefixation and suffixation, significantly increases the vocabulary size and poses substantial challenges for standard tokenization [9]. The inherent complexity of Hebrew morphology, where grammatical particles are frequently attached to base words, necessitates specialized representations to capture semantic nuances in tasks like sentiment analysis, which might be obscured by standard sub-word tokenization [10]. While several pre-trained models for Hebrew have emerged, such as AlephBERT [11], their performance on highly specialized or small-scale news datasets remains a subject of investigation, particularly when compared to traditional feature engineering.

(2) This dataset covers news articles for four classification tasks: domain, sentiment, gender, and website.

(3) It performs an extensive and systematic set of experiments across various TC models on datasets using five classical ML methods, three preprocessing methods, five oversampling techniques, parameter tuning, 32 engineered feature sets (17 content-based and 15 style-based), heuristic and hill-climbing methods, and five DL methods.

(4) The research offers specific findings on which preprocessing methods, oversampling techniques, BERT models, AI models, and feature set combinations improve classification results for the different tasks in Hebrew. For instance, it highlights that classical ML methods combined with robust feature engineering outperformed DL models in the sentiment and gender classification tasks for this dataset.

The rest of this paper is organized as follows: Section 2 introduces previous news classification systems and several supervised news datasets. Section 3 provides a visual overview of the entire experimental pipeline, presents the supervised dataset we constructed for this research, and details the traditional supervised ML methods we applied for baseline needs. Section 4 presents the preprocessing methods, oversampling techniques, parameter tuning, the five different BERT models, the rigorous evaluation framework (including repeated cross-validation and statistical testing), and the analysis of the experimental results. Finally, Section 5 summarizes the main findings, presents the conclusions, and suggests expanded future research directions.

2. Previous Related Systems

Text classification (TC) is a foundational task in Natural Language Processing (NLP) that aims at assigning predefined categories to unstructured text [12]. While TC is well-established for languages like English, morphologically rich languages (MRLs) such as Hebrew can present unique computational challenges [9].

Kaur and Bajaj [13] theoretically discussed the different steps involved in news classification: data collection, preprocessing, feature selection, and application of several classification methods such as naive Bayes (NB), artificial neural networks, decision trees, support vector machines (SVM), and k-nearest neighbors (KNN).

HaCohen-Kerner et al. [14] analyzed the influence of preprocessing on TC across several benchmark corpora using a bag-of-words representation. The authors concluded that in general, across all processed datasets, there was always at least one combination of basic preprocessing methods that improved accuracy when using a bag-of-words representation.

Kiesel et al. [15] organized the SemEval-2019 Task 4: Hyperpartisan News Detection, where several teams competed to develop the best model for detecting hyperpartisan news articles written in English. Two different datasets were involved: the by-article dataset, where each article is labeled manually, and the by-publisher dataset, where the label was done at the publisher level. The best-performing system for the by-article dataset was the system of Bertha von Suttner [16], which used an ensemble of convolutional neural networks (CNNs) trained on ELMo sentence embeddings, achieving an accuracy of 82.2%. For the meta-classification task performed on the by-article dataset, the best systems were Fernando Pessa [17] and Spider Jerusalem [18], which obtained an accuracy of 89.9% by assembling the predictions of individual systems. For the by-publisher dataset, the best system was Tintin [19], achieving an accuracy of 70.6% using SVC and LR.

Barua et al. [20] focused on multi-class sports news classification in the Bengali language. The authors developed the BNeC corpus, containing 43,306 sports news documents, and evaluated six ML models: LR, SVC, decision trees (DT), multinomial naive Bayes (MNB), RF, and KNN. The authors extracted TF-IDF features for various n-gram combinations. They classified the news articles into four categories: cricket, football, tennis, and athletics. The best weighted f1-score (0.976) was obtained using SVC with a combination of word unigrams, bigrams, and trigrams. Notably, no resampling technique was employed.

Minaee et al. [12] reviewed over 150 DL models developed for various text classification tasks, including news classification, sentiment analysis, topic classification, question answering, and natural language inference. The authors discussed the strengths and limitations of different architectures, such as attention mechanisms, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers. They compared their performance on popular text classification datasets.

Ahmed and Ahmed [21] presented a single-class text classifier for online news categorized by domain, consisting of eight categories: entertainment, crime, politics, business, world news, sports, media, and technology. The authors used an imbalanced dataset (politics: 44%, entertainment: 21%, business: 10%, and the other categories between 3% and 7%) that contains around 75k news headlines with their content and category from seven various websites with their tags for the years 2012–2018. The authors applied 4 ML methods: naive Bayes (NB), k-nearest neighbors (KNN), support vector machine (SVM), and Logistic Regression (LR). The NB ML method obtained the best F-score and accuracy results, 0.90 and 0.93, respectively.

Chen et al. [22] proposed a classification model of long-text Chinese News based on BERT and Local Feature Convolutional Network (LFCN). Their approach involved summarizing long texts using a Dynamic LEAD-n algorithm, extracting local salient features using a local feature convolution operation, and fusing these features for classification. Their LFCN model outperformed several baseline DL models: CNN, ERNIE, BERT, BERT-wwm, BERT-wwm-ext, RoBERTa-wwm-ext, MacBERT, and ChineseBERT. Their LFCN model obtained accuracies of 96.2% on the MCNews dataset (containing 15,200 online news texts divided into nine categories) and 99.0% on the THUCNews dataset (containing 65,000 news texts evenly distributed across ten categories).

Daud et al. [3] presented a topic classification of online news articles using optimized ML models. The task was to classify online news articles written in English into four domain categories. Their balanced dataset comprised 4180 domestic news, 4180 political news, 4180 international news, and 4180 top news. They compared six ML methods: Random Forest (RF), SVM, Stochastic Gradient Descent (SGD), KNN, LR, and NB. The authors performed text preprocessing methods and employed TF-IDF vectorization with dimensionality reduction. The optimized SVM model obtained the highest accuracy result (85.16%).

These studies have demonstrated the diversity of approaches and datasets employed in the news classification domain. While classical ML models (e.g., LR, SVM, and RF) have shown promising results, DL models, particularly those based on attention mechanisms and transformers, have attracted attention in recent years. Ensemble and meta-classification methods have also been explored to leverage the strengths of multiple models and improve performance. Table 1 presents a summary of related news classification datasets and systems.

Table 1. Summary of related news classification datasets and systems.

Source & Dataset	Language/Dataset	Task (# Classes)	Size (Train/Test/Total)	Distribution	Best Method	Best Result
Kiesel [15] [Art.]	English (SemEval-19)	Hyperpartisan (2)	645/628/1273	Balanced (priv. test) Imalanced (pub. test)	CNN & ELMo	Acc. 82.2% Meta-Class. Acc. 89.9%
Kiesel [15] [Pub.]	English (SemEval-19)	Hyperpartisan (2)	600k/4k/754k	Balanced	SVC & LR	Acc. 70.6%
Barua [20] [BNeC]	Bengali (BNeC)	Sports Domain (4)	N/A/N/A/43,306	Imbalanced	SVC	F1-score: 97.6%
Chen [22] [MC]	Chinese (MCNews)	News (9)	10.6k/4.5k/15.2k	Imbalanced	LFCN	Acc. 96.2%
Chen [22] [THUC]	Chinese (THUCNews)	News (10)	50k/10k/65k	Balanced	LFCN	Acc. 99.0%
Daud [3] [Reut.]	English (Reuters)	News (4)	11.7k/5k/16,720	Balanced	Tuned SVC	Acc. 85.16%

Table 1 can be summarized according to the following criteria.

Applied ML Methods: The studies applied various ML methods, including the Local Feature Convolutional Network proposed by [22] and classic ML methods [3,20].

Preprocessing: Some studies did not apply any preprocessing techniques [20,22], while others performed various preprocessing steps. These included punctuation mark removal, stop word elimination, tokenization, stemming, and lowercase conversion [3].

Features: Some researchers utilized word N-grams with TF-IDF [3,20].

Classification: The classification tasks varied depending on the dataset and research objective. Kiesel et al. [15] focused on binary hyperpartisan/non-hyperpartisan article classification, Barua et al. [20] classified sports news articles into four categories (cricket, football, tennis, and athletics), Chen et al. [22] performed multi-class classification into nine categories (sports, business, house, education, technology, game, entertainment, military, and auto), Daud et al. [3] classified news articles into four broad categories (domestic, politics, international, and top news).

Results: The results depended on the tested languages, datasets, applied ML methods, and features. The sport classification [20] reached a 97.6% f1-score, with most models scoring over 90%. In the news classification tasks by Chen et al. [22], the LFCN model achieved the highest accuracy scores of 99.0% on the THUCNews dataset and 96.2% on the MCNews dataset; these accuracy scores were 1.1% and 2.2% higher, respectively, compared to the baseline models used in the study. The accuracy in the news classification task described in Daud et al. [3] varied from 64.35% to 85.16%. For the hyperpartisan classification performed by Kiesel et al. [15]), the accuracy of the by-article dataset varied from 46.2% to 82.2%, while in the meta-classification experiments, the accuracy was over 85%. For the by-publisher dataset, the highest accuracy was 70.6%.

3. Construction of a Supervised Dataset and the Experimental Setup

Before detailing the dataset construction and the specific modeling configurations, Figure 1 illustrates the overall experimental pipeline utilized in this study. The conceptual architecture contrasts the classical machine learning workflow with the deep learning approach, both culminating in a rigorous evaluation framework.

Figure 1. Systematic overview of the experimental pipeline for Hebrew news classification. The architecture contrasts two main methodologies: a classical machine learning path featuring extensive handcrafted feature engineering, hill-climbing selection, and oversampling; and against a deep learning path utilizing dedicated Hebrew BERT models. Both approaches are evaluated through a rigorous repeated five-fold cross-validation framework, followed by statistical significance testing.

Our dataset comprised news articles written in Hebrew that were downloaded from three Israeli news websites:

• Arutz 7 (https://www.inn.co.il/, accessed on 21 March 2026), “Arutz Sheva (‘Channel 7’), also known in English as Israel National News, is an Israeli media network identifying with religious Zionism. It offers online news articles in Hebrew, English, and Russian and livestreaming radio, video, and free podcasts.” (https://en.wikipedia.org/wiki/Arutz_Sheva, accessed on 21 March 2026).
• N12 (https://www.n12.co.il/, accessed on 21 March 2026) is an Israeli news website and app operated by the Keshet Broadcasting news company.
• Ynet (https://www.ynet.co.il/, accessed on 21 March 2026), is one of the major Israeli news and general content websites and is the online outlet for the Yedioth Ahronot newspaper; however, most of Ynet’s content is original work, published exclusively on the website and written by an independent staff (https://en.wikipedia.org/wiki/Ynet, accessed on 21 March 2026).

The collected news documents belonged to three domains: economy, politics, and sports. Each news item in the dataset comprised the complete text, including the title and subtitle, the author’s name, and the author’s gender. The constructed dataset excludes images, videos, and audio, even where they were present in the original article.

Each news document has a sentiment value (+1, 0, or −1). Three computer science students who were native Hebrew speakers determined the sentiment values. Each student annotated a unique set of news documents; thus, each news document was classified by only one student.

As mentioned before, this research will analyze and classify Hebrew news articles by labels such as gender, domain, website, and sentiment. Table 2 presents the document distribution in the dataset by domain, website, and gender.

Table 2. Document distribution in the dataset by domain, website, and gender.

Website	Economy		Politics		Sports		Total
	Male	Female	Male	Female	Male	Female
Arutz 7	17	17	17	17	17	17	102
N12	17	17	17	17	17	17	102
Ynet	17	17	17	17	17	17	102
Total	102		102		102		306

Table 2 shows that each class in each website/domain/gender contains an equal number of articles. In total, the dataset contained 306 news documents. This dataset was divided into training and test subsets by randomly splitting the 17 documents in each class into a subclass of size 12 for the training set and a subclass of size five for the testing set, for 216 documents in the training set (~71%) and 90 documents in the testing set (~29%).

It can also be seen from Table 2 that all the labels, apart from sentiment (which is not mentioned in Table 1), are entirely balanced:

• The number of documents written by female authors equaled the number of documents written by male authors.
• The number of documents about politics was equal to the number of documents about sports and the number of documents about economics.
• The number of documents from Arutz 7 was equal to the number of documents from N12, which was equal to the number of documents from Ynet.

Table 3 presents the sentiment distribution across the entire dataset.

Table 3. Sentiment distribution across the entire dataset.

Sentiment	−1	0	1	Total
# of documents	41	217	48	306
% of documents	13.40%	70.91%	15.69%	100%

As shown in Table 3, the dataset was imbalanced from the viewpoint of the sentiment label. Approximately 71% of the dataset was labeled as 0 (no sentiment), approximately 15.69% as having a positive sentiment, and approximately 13.40% as having a negative sentiment value.

At the beginning of this study for baseline results, we applied five classical ML methods to train and test the models: LR (Logistic Regression), RF (Random Forest), SVC (Support Vector Classifier), MNB (Multinomial naive Bayes), and MLP (Multilayer Perceptron).

LR [23] is a linear model explicitly designed for classification tasks. In this model, the output value is expressed as a linear combination of input values. LR uses a sigmoid function to model the probability of “success” in a single trial of an experiment with only two possible outcomes.

SVC is a variation in the support vector machine (SVM) ML method [24] implemented in SciKit-Learn. SVC utilizes LibSVM (Library for support vector machines) [25], a high-speed implementation of the SVM method. SVM, a supervised ML method, categorizes vectors in a feature space into one of two sets based on training data. It accomplishes classification by constructing an N-dimensional hyperplane dividing the two sets in the original feature space or a higher-dimensional kernel space.

RF is an ensemble learning method for classification and regression [26]. RF constructs numerous decision trees during training, employing the “bagging” (bootstrap aggregating) concept and randomly selecting features to form a forest of decision trees. The final result for each case is made by a vote of the results of all the decision trees in the forest for the discussed case.

MLP [27] is a deep artificial neural network based on a network of computational units known as perceptrons, interconnected in a feed-forward manner. The network comprises layers of perceptrons, each having directed connections to neurons in the subsequent layer. Typically, these units apply a sigmoid function to their input and feed the next layer with their output.

MNB [28] is a probabilistic generative ML method, serving as a version of naive Bayes commonly applied in text classification tasks. MNB is based on Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features, given the class variable’s value. In MNB, each document is treated as a collection of words, with the order of words considered irrelevant.

The news documents used for training and testing were randomly chosen with a constant seed (of 42) to achieve reproducibility and comparability.

4. Experimental Results and Their Analysis

4.1. Baseline Results

To produce the baseline results for each dataset, we chose to work with the N words (i.e., word unigrams) with the highest TF-IDF values in the relevant training sets. To prevent overfitting during the learning process, we worked with words that met the condition df ≥ 3, where df is the number of documents in the training set containing the discussed word. The variable N varied from 500 to 3000 in steps of 500 because 3141 words met the condition df ≥ 3 in the training set. Then, we trained and tested using five classical supervised ML methods. This process was performed for each of the four labels associated with each document: domain, sentiment, gender, and website. The highest value in each Table is in bold. Suppose there is more than one highest value. In such cases, we selected the one obtained with the lowest value of N. Table 4 presents the complete baseline results: sentiment (f1-score) and domain, gender, and website (accuracy) using word unigrams exclusively. The highest baseline result for each classification type is highlighted in bold.

Table 4. Baseline results: sentiment (f1-score) and domain, gender, and website (accuracy) using word unigrams. The highest value in each sub-Table is in bold.

Domain
# of Words	LR	RF	SVC	MNB	MLP
500	95.556	88.889	91.111	94.444	94.444
1000	94.444	92.222	94.444	95.556	95.556
1500	96.667	87.778	95.556	95.556	95.556
2000	96.667	90	95.556	95.556	96.667
2500	96.667	87.778	95.556	95.556	96.667
3000	96.667	90	95.556	95.556	96.667
Sentiment
# of Words	LR	RF	SVC	MNB	MLP
500	60.573	63.004	60.573	60.573	69.177
1000	60.573	65.476	60.573	60.573	66.418
1500	60.573	63.041	60.573	60.573	67.538
2000	60.573	60.573	60.573	60.573	66.578
2500	60.573	60.573	60.573	60.573	66.207
3000	60.573	63.041	60.573	60.573	64.09
Gender
# of Words	LR	RF	SVC	MNB	MLP
500	56.667	55.556	56.667	56.667	60
1000	62.222	54.444	64.444	64.444	63.333
1500	61.111	54.444	66.667	63.333	62.222
2000	61.111	58.889	64.444	66.667	65.556
2500	62.222	63.333	65.556	64.444	66.667
3000	63.333	57.778	66.667	68.889	70
Website
# of Words	LR	RF	SVC	MNB	MLP
500	58.889	73.333	61.111	61.111	58.889
1000	64.444	68.889	65.556	63.333	67.778
1500	64.444	70	64.444	62.222	65.556
2000	66.667	70	63.333	63.333	70
2500	66.667	66.667	63.333	62.222	70
3000	65.556	74.444	66.667	64.444	66.667

The baseline accuracy for the domain label reached a very high level of 96.667%. The other three baseline results are relatively low. MLP obtained the best results in two classification tasks: an f1-score of 69.117% for the sentiment tasks and an accuracy of 70% for the gender task. LR obtained the best baseline accuracy result, 96.667%, for the domain classification, and RF obtained the best baseline accuracy result, 74.444%, for the website classification.

4.2. Character N-Grams

The character N-gram is a contiguous sequence of N characters from a given sample of text or speech. The character N-Grams TF-IDF model calculates the TF-IDF value for each character N-gram in each document.

Our experiments work with character N-gram sequences that meet the condition df ≥ 3, where df is the number of documents containing a specific character N-gram in the training set. To avoid overfitting, we did not utilize N-gram sequences with df < 3. Table 5 presents the highest results for each label using character N-grams with df ≥ 3.

Table 5. Highest results using character N-grams (df ≥ 3).

Label	Measure	N-Gram	#	Classifier	Highest Baseline Result
Domain	Accuracy	3	6764	LR	97.778
Sentiment	F1_score	3	6764	MLP	66.927
Gender	Accuracy	5	23,237	MNB	72.222
Website	Accuracy	3	6764	RF	76.667

The results of Table 5 using character N-grams show improvements in three out of four classification tasks compared to the results of Table 4 using word unigrams, including domain: 96.667 to 97.778, gender: 70 to 72.222, and website: from 74.444 to 76.667. Only for the sentiment task, the highest baseline using word unigrams) 69.177) was higher than the highest baseline using character N-grams (66.927).

4.3. Combined Experiment—Word Unigrams and Character N-Grams

In this stage, for each label, using the five classical ML methods mentioned above, we performed experiments in which we combined the features that gave the highest result using word unigrams and the features that gave the highest result using character N-grams. Table 6 presents the classification results using character N-grams combined with TF-IDF values. The highest value for each label is in bold.

Table 6. Classification results using character N-grams with TF-IDF.

Label	Domain	Sentiment	Gender	Website
Measure	Accuracy	F1_score	Accuracy	Accuracy
# of word unigrams	1500	500	3000	3000
N-gram characters	6764 3-grams	6764 3-grams	23,237 5-grams	6764 3-grams
LR	97.778	60.573	66.667	72.222
RF	92.222	60.573	65.556	72.222
SVC	97.778	60.573	65.556	68.889
MNB	95.556	60.573	70	60
MLP	96.667	70.036	71.111	74.444

The comparison of Table 6’s results to those of Table 4 and Table 5 shows that there was an improvement only for the sentiment task, where the combination of 500-word unigrams and 23,237 char 5-grams improved the f1-score from the result in Table 5 (69.177) to 70.036 (Table 7).

Table 7. Constructed feature sets.

#	Feature Set	# of Features	#	Feature Set	# of Features
1	Limiters	59	17	Defense	68
2	Intensifiers	47	18	Positive Words	135
3	Time	261	19	Negative Words	136
4	Extended Time	319	20	Strong Positive	31
5	Pronouns	38	21	Strong Negative	29
6	First person	20	22	Quantifiers	26
7	Second person	18	23	Conjunctions	17
8	Third person	40	24	Question Words	21
9	Singular First-Person Pronouns	22	25	Prepositions	24
10	Possessive pronouns	18	26	Modals	11
11	Political	119	27	Auxiliary Verbs	13
12	Specific Political Names	58	28	Quantitative	6
13	Economy	107	29	Orthographic	34
14	Sport	138	30	Titles	202
15	Society	74	31	Subtitle	279
16	Judaism	58	32	4-Gram words	20

4.4. Preprocessing

We implemented three preprocessing techniques for Hebrew:

• S—Stop word removal: removes the stop words in Hebrew from the text documents according to a list of 418 stop words that we constructed based on [29], e.g., “של” (of), “משום” (because), and “ו” (“and”).
• P—Punctuation marks removal: removes all the special characters from the text except those within a word (e.g., which is placed inside the word “יו”ר”, which means “יושב ראש” (Chairman)). The 34 special characters are: !”#$%&’()*+,-./:;<=>?@[\]^_{|}~•₪
• Replacing acronyms with their complete forms: replace all acronyms in the text documents with their complete forms (e.g., convert “יחב”ל” to “יחסים בינלאומיים” (international relations)).

These three functions have eight combinations: no preprocessing (the baseline result), S, P, A, SP, SA, PA, and PSA. We performed each combination of the preprocessing techniques on the text of each article, using the combination of the ML method and the number of word unigrams that achieves the best baseline for each classification task. The analysis of the results showed no improvements for three classification tasks: domain, gender, and website. Only for the sentiment classification, when using P (Punctuation marks removal), there was an improvement of 1.33 in the f1-score from the baseline using word unigrams 69.177 to the result of 70.507. This result was also better than the highest sentiment results in Table 5 and Table 6 (results using character N-grams combined or uncombined with TF-IDF).

4.5. Feature Sets

Table 7 details the 32 feature sets we constructed.

Table 7 contains 17 content-based feature sets and 15 style-based feature sets. The content-based features carry specific semantic meaning and provide information about the subject matter. The style-based features carry information about the structural and grammatical elements that help organize and present the content. In the following two paragraphs, we have given English words for some of the feature sets as examples of features; however, it should be remembered that our feature sets contain Hebrew words, not English words.

The 17 content-based feature sets are: (#3) a list of words related to time (e.g., today, yesterday); (#4) an extended list of words related to time; (#11) political words; (#12) specific political names; (#13) economy words; (#14) sport words; (#15) society words; (#16) Judaism words; (#17) defense words; (#18) positive words (e.g., good, nice); (#19) negative words (e.g., bad, ugly); (#20) strong positive words (e.g., fantastic); (#21) negative words (e.g., worst); (#28) quantitative; (#30) words appear in titles; (#31) words appear in subtitles; and (#32) all 4-gram words that appear in at least three documents in the training set.

The 15 style-based feature sets are: (#1) limiters (e.g., only, just, merely); (#2) intensifiers (e.g., very, really, significantly); (#5) pronouns (e.g., I, you, he, she, it, they); (#6) first-person pronouns (e.g., I, we, us); (#7) second-person pronouns (e.g., you, your); (#8) third-person pronouns (e.g., he, she, it, his, her); (#9) singular first-person pronouns (e.g., I, my, me); (#10) possessive pronouns (e.g., my, your, their); (#22) quantifiers (e.g., few, many, some); (#23) conjunctions (e.g., and, but, however, or); (#24) question words (e.g., what, where, when, who, which); (#25) prepositions (e.g., at, in, ont); (#26) modals (e.g., can, must, should, would); (#27) auxiliary verbs (e.g., be, had, has, have, do); and (#29) orthographic features (e.g., punctuation signs, capitalization).

Some feature sets introduced in Table 7 are explained in more detail below. HaCohen-Kerner and Badash [30] constructed essential lists of positive and negative sentiment words in Hebrew. Based on these two lists, we developed four extended lists for this research: “Positive words,” “Negative words,” “Strong Positive,” and “Strong Negative.” The quantitative feature set referred to the numerical characteristics of the document. This feature set included six features: the total number of words; the total number of characters; the number of sentences in the document; the average number of words per sentence; the average number of characters per sentence; and the average number of characters per word. The orthographic feature set referred to the relative frequency of punctuation marks and special characters. The titles feature set contained all the words that appeared in more than one different title in the training set. The subtitle feature set contained all the words that appeared in more than two different subtitles in the training set. The 4-gram words feature referred to all the sequences of four words that appeared in at least three documents in the training set (there are 20 such sequences).

4.6. Hill-Climbing

One should experiment with every possible combination of feature sets to determine which combination worked best for a given classification task. The number of distinct k-subsets on a set of n elements is ${\displaystyle \frac{n!}{k!\left(n-k\right)!}}$, where a k-subset is a subset of n elements that contains precisely k elements. This is one of the known properties of binomial coefficients.

The following hill-climbing model was proposed to avoid this combinatorial explosion for non-small values of n, where a series of feature sets were added using a hill-climbing search to improve a specific classification, with each addition increasing the accuracy of the classification.

In other words, every feature set was tested separately during the first stage. Out of n possible feature sets, the best feature set has been chosen. In the second step, every possible pairing of two feature sets was examined, one of which was the set selected in the first stage. In other words, the second step involved checking (n − 1) possible feature sets. The procedure will continue if the optimal combination of two feature sets outperforms the optimal single feature set. This procedure is carried out iteratively until no more progress can be made.

This hill-climbing model tests were as follows:

$$n + (n - 1) + \dots + 1 = {\displaystyle \sum _{k=n}^{1}k}= {\displaystyle \frac{n(n+1)}{2}}$$

(1)

as combinations of feature sets. In other words, rather than

O(2ⁿ),

(2)

the complexity of this heuristic model is

O(n²)

(3)

This study has 32 feature sets (i.e., n = 32); therefore, instead of checking 4,294,967,296 combinations, it is sufficient to check a maximum of 1024 combinations.

For each label, we run the hill-climbing process five times, one for each ML classification algorithm, for 20 different hill-climbing experiments. Hill climbing achieved optimal performance when using a suitable preprocessing combination for each label. Table 8 presents the hill-climbing optimization results across all classification tasks. The best-performing result in each column is highlighted in bold.

Table 8. Hill-Climbing Optimization Results Across All Classification Tasks.

Label	Domain	Sentiment	Gender	Website
Measure	Accuracy	F1_score	Accuracy	Accuracy
Baseline	96.667	69.177	66.667	73.333
Best Preprocessing Score	96.667	70.507	66.667	73.333
Best Preprocessing Combination	/	P	/	/
LR	81.111	66.207	72.222	74.444
RF	92.222	69.828	72.222	83.333
SVC	81.111	69.092	63.333	75.556
MNB	84.444	63.004	67.778	70
MLP	86.667	67.096	66.667	78.889

The analysis of Table 8 shows:

• Domain: The result of the best ML algorithm, RF (92.222%), was lower than the baseline result (96.667%), which was already very high.
• Sentiment: 69.828—Using RF, there was a slight improvement, less than 1% from the baseline (69.177%).
• Gender: 72.222%—The improvement from the baseline (66.667%) using RF and LR was about 5.5%.
• Website: 83.333%—Using RF, the improvement from the baseline (73.333%) was 10%.

4.7. Treatment of the Imbalanced Dataset from the Sentiment Viewpoint

We handled the dataset’s imbalances from the viewpoint of the sentiment label by resampling the training set. Weiss et al. [31] showed that oversampling tended to outperform downsampling on smaller datasets to handle class imbalance. Using its limited training data, downsampling can discard potentially valuable minority class examples. In contrast, oversampling augments the minority class to balance the dataset without the loss of information. Therefore, to improve the sentiment classification, we tested five oversampling techniques.

• SMOTE—Synthetic Minority Over-sampling Technique [32]: This technique selected two or more similar instances, using a distance measure, and perturbed an instance one attribute at a time by a random amount within the difference to the neighboring instances.
• RandomOverSampler (ROS)—Random Over-sampling [33]: Duplicate samples from the smaller classes are randomly selected until their size equals that of the more significant class.
• SMOTEN—Synthetic Minority Over-sampling Technique for Nominal and Continuous Data [33]: SMOTEN is similar to SMOTE but can be used for datasets with both categorical and continuous features. It can create synthetic samples by using the k-nearest neighbors algorithm.
• BorderlineSMOTE—Borderline Synthetic Minority Over-sampling Technique [34]: This technique can generate synthetic samples in the vicinity of borderline instances, which are minority class samples that are close to the majority class samples. It can be helpful when the minority class is not well separated from the majority class.
• ADASYN—Adaptive Synthetic Sampling [35]: ADASYN can generate more synthetic samples near the minority class instances that are difficult to learn and fewer samples near those that are easy to learn. It aims to balance the class distribution and improve the model’s performance on the minority class.

After performing each of the first four resampling techniques, the sentiment label in the training set was entirely balanced; however, ADASYN did not guarantee complete balance in the training set. Table 9 presents the top sentiment performance across resampling techniques performed on the original dataset.

Table 9. Top sentiment performance across resampling techniques.

Resampling Technique	# of Words	Classifier	F1_Score
SMOTE	1500	LR	71.291
ROS	2000	LR	73.533
SMOTEN	1000	MNB	64.7
BorderlineSMOTE	2500	LR	69.209
ADASYN	1500	LR	67.832

The results in SMOTE, BorderlineSMOTE, and ROS improved the baseline for sentiment, while ROS performed the best.

In the next step, we ran the hill-climbing algorithm on the sentiment label again, including the preprocessing P, across every resampling technique. Table 10 presents the hill-climbing results for sentiment classification after resampling.

Table 10. Hill-climbing results for sentiment classification post-resampling.

Resampling Technique	LR	RF	SVC	MNB	MLP
SMOTE	70.536	76.996	73.349	73.012	74.576
ROS	71.571	73.687	78.354	76.734	73.31
SMOTEN	68.521	70.236	74.809	70.57	68.737
BorderlineSMOTE	73.178	77.295	73.552	72.755	77.251
ADASYN	69.601	76.343	74.627	71.559	73.316

The hill climbing for sentiment, including the preprocessing P, using SVC, after performing ROS, scored 78.354. It was the highest score we had achieved for sentiment so far. It was better than the result of hill climbing without resampling (69.828) at 8.5. Compared to the baseline, there was a relatively significant improvement from 69.177 to 78.354.

Heuristics: “Sentiment P ROS” meant that we applied the sentiment classification using P, which meant “punctuation marks removal,” and ROS, which meant “random over-sampling.”

Domain: We selected 23 feature sets. These included sets chosen during the hill-climbing process, as well as feature sets that, despite not being selected initially, achieved high classification scores individually. The 23 feature sets were: (31, 16, 27, 14, 25, 30, 13, 23, 15, 24, 20, 22, 10, 17, 32, 19, 26, 1, 4, 11, 12, 18, 29). Fourteen of them were content-based feature sets, and nine were style-based. We tested all the combinations that contained between 15 and 21 feature sets out of the 23 feature sets. Using RF, five combinations got a score of 94.444, and 86 got 93.333 for 21 combinations. Table 11 summarizes the best feature combinations for the domain classification task using RF.

Table 11. Best combinations for domain classification using RF.

Score	# of Feature Sets	# of Features	Feature Sets
94.444	16	1385	(4, 10, 11, 13, 14, 15, 16, 19, 20, 23, 24, 25, 26, 27, 31, 32)
94.444	15	1486	(4, 10, 13, 14, 15, 16, 17, 18, 22, 23, 24, 26, 27, 30, 31)
94.444	15	1491	(4, 10, 13, 14, 15, 16, 17, 19, 20, 23, 26, 27, 30, 31, 32)
94.444	15	1531	(4, 11, 12, 13, 14, 15, 16, 17, 23, 24, 25, 27, 29, 30, 31)
94.444	17	1579	(1, 4, 10, 13, 14, 15, 16, 17, 19, 22, 23, 24, 25, 27, 30, 31, 32)

Sentiment: We chose 19 feature sets. These feature sets were chosen during the hill climbing for the discussed label, and feature sets that were not selected in the hill climbing but obtained good classification scores for the discussed label when we classified only using one feature set. The 19 feature sets were: (9, 16, 26, 6, 18, 14, 20, 21, 8, 22, 2, 3, 4, 5, 29, 30, 31, 27, 24). Nine out of them were content-based feature sets, and ten were style-based.

We tested all the combinations that contained between 8 and 14 feature sets out of the 19 feature sets. We used SVC, which gave the best result in the hill climbing for sentiment. We used P as the preprocessing, which gave the best results for sentiment, and the RandomOverSampler as the oversampling technique, which performed the best in the hill climbing. Table 12 presents the best combinations for the sentiment classification using SVC ROS P.

Table 12. Best combinations for sentiment classification using SVC ROS P.

Score	# of Feature Sets	# of Features	Feature Sets
78.354	8	468	(6, 8, 9, 14, 16, 18, 21, 22)
78.354	9	499	(6, 8, 9, 14, 16, 18, 20, 21, 22)
78.354	11	557	(2, 6, 8, 9, 14, 16, 18, 20, 21, 22, 26)

Gender: Using hill climbing, both LR and RF obtained a result of 72.222. We performed the heuristic process twice, once for LR and once for RF.

We chose 15 feature sets for LR. These feature sets were chosen during the hill climbing for the discussed label, and feature sets that were not selected in the hill climbing but obtained good classification scores for the discussed label when we classify only using one feature set. The 15 feature sets were: (24, 14, 10, 12, 26, 21, 20, 15, 17, 1, 13, 25, 27, 29, 31). Eight out of them were content-based feature sets, and seven were style-based. We tested all the combinations that contained between 6 and 12 feature sets out of the 15 feature sets. We found six combinations that scored 72.222.

We chose 20 feature sets for RF. These feature sets were chosen during the hill climbing for the discussed label, and feature sets that were not selected in the hill climbing but obtained good classification scores for the discussed label when we classified only using one feature set. The 20 feature sets were: (27, 31, 14, 24, 26, 22, 29, 19, 17, 13, 1, 2, 6, 8, 25, 23, 32, 12, 4, 5). Eight out of them were content-based feature sets, and 12 were style-based. We tested all the combinations that contained between 8 and 14 feature sets out of the 20 feature sets. We found three combinations that scored 75.556 and 11 combinations that scored 74.444. Table 13 presents the best combinations for gender classification using RF.

Table 13. Best combinations for gender classification using RF.

Score	# of Feature Sets	# of Features	Feature Sets
75.556	10	646	(2, 6, 8, 12, 13, 14, 17, 19, 24, 26)
75.556	10	696	(1, 8, 12, 14, 22, 23, 24, 25, 29, 31)
75.556	11	748	(2, 6, 13, 14, 17, 24, 25, 26, 27, 31, 32)

Website: We chose 18 feature sets. These feature sets were chosen during the hill climbing for the discussed label, and feature sets that were not selected in the hill climbing but obtained good classification scores for the discussed label when we classified only using one feature set. The 18 feature sets were: (1, 4, 5, 7, 9, 11, 13, 14, 17, 18, 20, 21, 25, 27, 28, 29, 30, 31). Eleven out of them were content-based feature sets, and seven were style-based. We tested all the combinations that contained between 8 and 14 feature sets out of the 18 feature sets. We found two combinations that scored 86.667 and 12 combinations that scored 85.556.

Table 14 presents the best combinations for website classification using RF.

Table 14. Best combinations for website classification using RF.

Score	# of Feature Sets	# of Features	Feature Sets
86.667	10	906	(1, 5, 11, 13, 20, 25, 27, 29, 30, 31)
86.667	10	1190	(4, 7, 11, 13, 17, 20, 27, 29, 30, 31)

• Parameter Tuning

We applied an extensive number of parameter-tuning combinations for all the labels; however, the parameter tuning experiments did not improve performance on the classification tasks. This was likely due to overfitting.

Aggressive tuning tended to increase model complexity and flexibility, introducing selection bias towards inflated training accuracy, as discussed by Cawley and Talbot [36] and Bergstra and Bengio [37]. This overfitting led to models performing worse on new test data despite achieving higher training performance.

Additionally, due to the small size of our dataset, aggressive parameter tuning likely resulted in even more overfitted brittle models that could not generalize past the peculiarities and noise within the limited training data. As Brain & Webb [38] and Banko & Brill [39] note, it was a common issue that small datasets tended to capture spurious chance correlations that failed to transfer to new data. Extensive parameter tuning resulted in model over-optimization relative to the noise within the small training set, thereby hindering the models’ ability to robustly generalize to the test set.

Table 15 summarizes the baseline, hill-climbing, and heuristic results across all classification tasks.

Table 15. Baseline, hill-climbing, and heuristic results across all classification types.

Algorithm	Label	Domain	Sentiment POS	Gender	Website
	Measure	accuracy	F1_score	Accuracy	Accuracy
Baseline	Classifier	LR	MLP	MLP	RF
	# of words	1500	500	3000	3000
	Result	96.667	69.177	70	74.444
Hill Climbing (HC)	Classifier	RF	SVC	RF/LR	RF
	# of feature sets	18	11	11/9	11
	# of features	1058	557	442/892	1302
	Result	92.222	78.354	72.222	83.333
Heuristics	Classifier	RF	SVC	RF	RF
	# of feature sets	16	8	10	10
	# of features	1385	468	646	906
	Result	94.444	78.354	75.556	86.667

The analysis of Table 15 shows the following main findings:

• Domain: The heuristic method did not improve the accuracy compared to the baseline, which was already very high.
• Sentiment: The heuristic method yielded the same result as the hill climbing, decreasing the number of features by 89 (from 557 to 468) and the number of feature sets by 3 (from 11 to 8). The improvement from the baseline was more than 9%.
• Gender: The hill climbing improved the baseline by 2.2%, and the heuristic method improved the hill climbing by 3.3%, resulting in a total of 5.5% difference between the heuristic method and the baseline.
• Website: The heuristic method improved the baseline by 12.2%.

• BERT

Recent advances in DL can provide promising new avenues for exploration. In particular, the pre-trained language model, BERT (Bidirectional Encoder Representations from Transformers), has shown state-of-the-art performance on many NLP tasks. We conducted a massive experiment by applying powerful BERT models.

Tuning strategies were essential to improving the adaptability and accuracy of our models.

• Variation in optimizer learning rates: Adjusting optimizer learning rates was essential to balance rapid convergence and precise learning. Higher rates expedited learning, potentially overcoming local minimums, while lower rates ensured accurate but slowed convergence.
• Exploration of different BERT models: During the tuning process, we delved into different BERT architectures, including language-specific variants optimized for Hebrew within our dataset. The BERT models we tested:
  • bert-base-uncased (https://huggingface.co/bert-base-uncased, accessed on 21 March 2026): A pre-trained model on the English language using a masked language modeling (MLM) objective.
  • onlplab/alephbert-base (https://huggingface.co/onlplab/alephbert-base, accessed on 21 March 2026): A language model for Hebrew trained on the Hebrew Wikipedia, OSCAR, and Hebrew tweets collected from the Twitter sample stream.
  • avichr/heBERT (https://huggingface.co/avichr/heBERT_sentiment_analysis, accessed on 21 March 2026): A Hebrew pre-trained BERT for polarity analysis and emotion recognition trained on the Hebrew Wikipedia, OSCAR, and more.
  • dicta-il/dictabert (https://huggingface.co/dicta-il/dictabert, accessed on 21 March 2026): A general language model for Hebrew.
  • imvladikon/alephbertgimmel-base-512 (https://huggingface.co/imvladikon/alephbertgimmel-base-512, accessed on 21 March 2026): A modern Hebrew pre-trained BERT model with a 128k token vocabulary.
• Oversampling techniques for sentiment classification.
• Test preprocessing combinations.

• Domain and Website

The domain and website classification tasks were relatively straightforward processes. We swiftly attained impressive results by employing various BERT models, experimenting with different optimizers, and tuning the number of epochs. We achieved 96.667% accuracy for the domain task using the avichr/heBERT model, ‘adam’ as the optimizer, and four epochs. For the website task, we achieved 88.889% accuracy with four epochs by using the avichr/heBERT model and ‘adam’ as the optimizer.

• Sentiment and Gender

Our exploration of sentiment and gender classification faced substantial hurdles. Despite thorough experimentation with diverse BERT models, learning rate adjustments, extensive preprocessing methods, and oversampling methods for sentiment, we could not achieve significant improvements. We were far from the performance of our ML models.

This could point to the limitations of current state-of-the-art models, indicating that there might be specific contextual or domain-related nuances that these models cannot capture effectively. Additionally, the inability to achieve higher performance underscored the importance of exploring alternative feature engineering methods, considering diverse linguistic and contextual aspects that might influence the classification. Furthermore, it highlighted the significance of continuous research and innovation in natural language processing to address complex classification tasks effectively, paving the way for future advancements in understanding and accurately predicting these labels from textual data. Figure 2, Figure 3 and Figure 4 present the results for the main stages of the sentiment, website, and gender classification tasks, respectively. We did not present the results for the domain classification task because the highest accuracy was obtained relatively early using character N-grams with df ≥ 3.

Figure 2. F1-scores for the main stages of the sentiment classification task.

Figure 3. Accuracy results for the main stages of the website classification task.

Figure 4. Accuracy results for the main stages of the gender classification task.

For the sentiment classification task (Figure 2), the best f1-score (78.354) was obtained by preprocessing and the heuristic method, as well as by preprocessing and the hill-climbing method. This result was higher than the result (76.144) obtained by the BERT method.

For the website classification task (Figure 3), the set accuracy result (88.889) was obtained by the BERT method.

For the gender classification task (Figure 4), the heuristic method obtained the best accuracy result (75.556). This was higher than the results obtained by the hill-climbing and BERT methods.

To rigorously evaluate the predictive performance of the classification models, a robust experimental framework was employed utilizing a five-fold cross-validation procedure repeated 20 times, yielding a comprehensive set of 100 evaluation metrics for each configuration. This setup facilitated a systematic comparison between standard baseline classifiers and top-performing enhanced models across multiple text classification tasks. Although the enhanced algorithmic architecture achieved notably superior peak performance scores in isolated iterations, an analysis of the aggregate cross-validation statistics revealed substantial variance and significant distributional overlap between the competing models. To formally determine whether these performance differences were statistically significant, we conducted a paired T-test on the resulting performance distributions. The analysis yielded a p-value that exceeded the conventional threshold for statistical significance (p > 0.05). Consequently, we failed to reject the null hypothesis, indicating that the isolated performance gains were likely attributable to stochastic data partitioning rather than a consistent, generalizable algorithmic advantage.

5. Summary, Conclusions, and Future Research

In this research, we constructed a dataset of news articles written in Hebrew. Each article was labeled with four labels: domain, sentiment, gender, and website. We performed extensive and systematic experiments evaluating various TC models, incorporating hill-climbing and heuristic approaches. We defined various feature sets (content-based and style-based). We applied five ML methods, three preprocessing methods, parameter tuning, and five different BERT models.

We computed baseline results across five classical ML algorithms (LR, RF, SVC, MNB, and MLP) for each classification task (domain, sentiment, gender, and website) using bag-of-words (BOW) according to their TF-IDF values. To avoid overfitting, we chose words that met the condition df ≥ 3, where df is the number of documents in the training set containing the discussed word. Using character N-grams with df ≥ 3, we improved three baseline results. Combinations of the BOW and character N-grams did not yield further improvements. We also tested the effect of all possible combinations of three preprocessing methods (stopword removal, punctuation mark removal, and opening acronyms) on the best-performing results. Only for the sentiment classification did removing the punctuation marks lead to a slight improvement.

We built 32 feature sets (17 content-based and 15 style-based) containing 2149 features and ran hill climbing on these feature sets. The hill-climbing method achieved better results for three labels: sentiment, gender, and website. Utilizing resampling techniques effectively tackled the imbalance of the sentiment dataset. Among these, RandomOverSampler demonstrated superior performance, leading to substantial improvement. Running hill climbing with SVC and RandomOverSampler for the sentiment class marked a remarkable enhancement, underscoring the significance of strategic resampling. Then, we ran the heuristic method that improved the results for two labels: gender and website. The BERT model improved the result for the website task. In the sentiment and gender tasks, the heuristic and hill-climbing methods obtained higher results than the BERT method. The reasons for this phenomenon for these tasks might include: (a) The dataset size, as BERT models were pre-trained on large text corpora; when fine-tuning on a small dataset, BERT may not have enough data to generalize effectively. (b) The domain-specific data: BERT’s pretraining data may not cover the specific domain of a small dataset. (c) Our results aligned with the observations of Tsarfaty et al. [9] regarding the ‘morphological gap’ in Hebrew NLP; in low-resource scenarios, handcrafted features can effectively bypass the segmentation errors that often plague transformer-based models when fine-tuned on limited data.

The best result for the sentiment classification task was an f1-score of 78.354 obtained by the heuristic method with preprocessing and the hill climbing with preprocessing methods, while the highest baseline was 69.177. The best accuracy result, 88.889 for the website classification task, was obtained by the BERT method, while the highest baseline was 74.444. The best accuracy result, 75.556 for the gender classification task, was obtained by the heuristic method, while the highest baseline was 70. The best accuracy result, 97.778 for the domain task, was obtained using character N-grams with df ≥ 3, while the highest baseline was 96.667. The improvements were relatively significant for three tasks: around 9.2 for the sentiment task, around 13.3 for the website task, around 5.6 for the gender task, and only around 1.1 for the domain task, where the initial baseline for this task was already very high. For two tasks (sentiment and gender), results achieved via classical ML methods outperformed the DL models. This finding was repeated in previous gender classification tasks (e.g., [40]). In three tasks (sentiment, gender, and website), the number of content-based feature sets was quite similar to the number of style-based feature sets, indicating the importance of both feature sets.

The primary contributions and innovations of this research include: (1) the development of a labeled dataset of online news articles written in Hebrew; (2) the execution of a comprehensive and systematic series of experiments across various TC models utilizing five ML methods, three preprocessing techniques, five oversampling strategies, parameter optimization, and five DL methods; and (3) the presentation of several specific findings that illustrate which combinations of preprocessing methods, oversampling techniques, BERT models, hill-climbing and heuristic models, and feature set combinations enhance classification outcomes.

The limitations of the study were as follows: (1) the dataset was relatively small, (2) the performance of the DL methods yielded low results in several tasks, and (3) several content-based feature sets pertained specifically to the examined domains of economy, politics, and sports, rendering them unsuitable for detecting other types of news.

Future research will focus on expanding the experimental scope of this work to more clearly position the proposed approach within the broader state of the art. First, we aim to evaluate the models on larger, more diverse datasets. To improve generalizability, this could include broadening the corpus to encompass a wider range of media ecosystems and social media content, as well as evaluating model performance on naturally skewed, real-world class distributions rather than perfectly balanced datasets.

Second, to strengthen the dataset’s reliability, future data collection phases will involve multiple human annotators, allowing for the formal measurement of inter-annotator agreement (e.g., Cohen’s Kappa or f1-agreement scores) to ensure rigorous label validity.

Third, we intend to expand our comparative baselines. We plan to compare our handcrafted feature approach against ‘lighter’ yet robust modern baselines, such as FastText and static word embeddings (e.g., Word2Vec and GloVe) specifically tailored for Hebrew. Furthermore, we will include rigorous benchmarking against state-of-the-art, large-scale multilingual transformer baselines. This will provide a more nuanced understanding of the trade-offs between traditional feature engineering, various non-Transformer-based architectures, and global transformer models in low-resource settings.

Fourth, we plan to enhance feature representation by exploring additional engineering strategies; this includes the integration of extracted keywords [41,42], various stylistic feature sets [43], and word and character n-grams, as well as skip-gram configurations for both words and characters [44].

Fifth, the experimental framework will be extended to incorporate sophisticated preprocessing techniques, such as expansion of abbreviation disambiguation [45,46] and word lemmatization. Specifically for the Hebrew language, future work should explore the integration of specialized morphological analyzers, such as Yap [47], to further refine feature representation within deep learning architectures. By addressing these dimensions—scale, linguistic breadth, annotation rigor, and granular feature engineering—we expect to achieve a more robust validation of the algorithmic advantages in complex text classification tasks.