Evaluating Computational Approaches for Harmful Content Analysis: Promise, Pitfalls and Tools for Responsible Research

Abstract

This manuscript develops and demonstrates a practical framework for evaluating automated classifiers used in communication research, using harmful language detection as an illustrative case. We combine (a) a structured review of documentation practices for 27 publicly available classifiers and their associated annotation processes with (b) a cross-dataset evaluation that re-tests each model beyond its original training context. Across 27 datasets, we extract and compare reporting on construct definitions, annotator instructions, and inter-annotator agreement, and we quantify generalization by applying each model to multiple out-of-domain test sets. We also benchmark a contemporary large language model (GPT-5) under a consistent prompting protocol to illustrate how LLM-based classification compares to fine-tuned classifiers. Results show that documentation is uneven and often insufficient for theory-driven measurement, inter-annotator agreement varies widely across datasets, and cross-dataset performance frequently drops substantially relative to within-dataset evaluations. Building on these findings and existing validation guidance, we provide a reusable checklist and decision flow to help researchers select, justify, and report classifier-based measures in ways that support transparency and cumulative science. Recommendations for researchers, reviewers, and journal editors stress aligning model selection with standards of validity, reliability, and transparency.

1. Introduction

As social media increasingly serves as central spaces for sharing opinions and debating contentious societal issues, concerns about their use to disseminate harmful content have also grown [1,2]. Exposure to online hate is linked to increased anxiety, depression, and post-traumatic stress among targeted users [3], and to hate crime activity [4]. Extremist violence can also trigger spikes in hate speech on Twitter/X, including content that advocates violence [5]. These stakes have made computational methods central for analyzing large volumes of user-generated content and identifying harmful expressions such as hate speech and offensive language [2,6,7].

As computational methods become routine in communication research, scholars and editors increasingly expect model-based coding to meet the reliability and validity standards long associated with human content analysis, yet validation and reporting remain inconsistent across text-as-data studies [8] and are often under-specified for construct alignment, evaluation design, and transfer across contexts in approaches like topic modeling, framing, and sentiment analysis [9,10,11]. To support credible inference, we treat transparent documentation of dataset provenance and labeling conditions as a pre-requisite [12] and center evaluation rather than constant model development [13,14], responding to concerns that harmful content research often prioritizes new architectures over generalization, validation, and bias [6,15].

Existing tools for evaluating harmful language classifiers include systematic dataset reviews [16], surveys of detection methods [17], and model repositories such as Hugging Face Hub [18]. These frameworks assess model performance through accuracy and F1 scores computed on the same dataset the model was trained on. This approach has known limitations: models frequently overfit to their training data, and cross-dataset evaluation is widely recommended to better reflect real-world performance [17,19]. We build on this by additionally evaluating whether annotators were trained and instructed consistently and whether the construct being measured aligns with how hate, offense, and threat are defined in communication research [20,21].

Our framework audits annotator instructions, recruitment, and qualifications to assess dataset-level construct validity and the operationalization of “hate,” “offense,” and “incivility” [11,12,22,23], and it foregrounds inter-annotator agreement by comparing reported reliability metrics to align ML practice with established communication measurement standards [8,24,25]. We also implement multi-dataset evaluation to avoid over-fitting conclusions to a narrow set of benchmarks and social contexts [6,26], addressing known limits in target and context coverage [27] and recommendations for validation across heterogeneous settings [9,11]. Finally, we strengthen replicability by documenting data retrieval when only post IDs are available [28], responding to broader reproducibility critiques [12,22], and we operationalize these principles as a practical, bias-aware evaluation framework with minimal reporting requirements and decision rules, demonstrated in harmful language detection where definitional and annotation heterogeneity can change what models effectively measure [6,8,9,12,22,27].

Empirically, we evaluate 27 publicly available models across five categories—hate speech, offensive language, bullying, racism, and threats—drawing on 27 training datasets (9 of them that we rebuilt) and retesting each model across additional datasets to quantify out-of-domain performance [11,29,30]. We also compare ChatGPT-5 labeling under a fixed prompting protocol to illustrate when LLM-based classification can and cannot substitute for task-specific models, given evidence of systematic classification bias in related settings [7,22]. Finally, we situate model selection within a theory-driven measurement approach, emphasizing that theory should guide both what is measured and how it is validated [14], and we close with recommendations for researchers, reviewers, and editors as AI tools continue to evolve [25].

1.1. Literature Review

As harmful content like hate speech, racism, threats, and bullying continues to grow on social media, its psychological and social consequences become increasingly severe. Exposure to such content can lead to anxiety, depression, and post-traumatic stress [31], while also creating hostile online environments that silence marginalized voices and discourage participation. Algorithms that prioritize engagement often amplify divisive posts, boosting extremism [32]. This can erode civil discourse and deepen social divides, normalizing prejudice and reinforcing harmful stereotypes [33]. Over time, this silencing of targeted communities weakens democratic participation and reduces the diversity of voices in public discussions [34].

The effects of online hate content are not limited to digital spaces; they shape real-world attitudes, politics, and violence [5]. Normalized hate speech can legitimize exclusionary ideologies and fuel support for xenophobic policies, especially during politically charged moments. For instance, Müller and Schwarz [4] found that anti-refugee sentiment on Facebook in Germany correlated with increased attacks against refugees, especially in far-right-leaning regions. Scholars using computational methods have uncovered the patterns and impact of such content: hate speech often delegitimizes protest movements [4], spreads within ideological echo chambers, and is masked through humor in memes [35]. At the same time, detecting nuanced toxic expressions remains a major challenge, especially in dataset development.

Increasing access to social media data and concerns about content misuse have prompted the adoption of computational methods for content analysis. These computational techniques label social media posts with predefined tags [17]. Borrowed from other disciplines, mainly computer sciences, computational methods raise concerns regarding the accuracy standards and whether they meet the expectations that communication researchers have followed for more than half a century. Content analysis principles are widely taught in communication research methods globally and need no detailed introduction; Berelson [20] nicely summarizes them as “the objective, systematic, and quantitative description of the manifest content of communication.” In essence, it involves extracting meaning from large volumes of content, yielding distinct labels for statistical analysis. Thus, using computational methods to label social media content shares objectives with traditional content analysis. However, standards in communication disciplines may diverge from those in computational approaches. We will first examine computational learning models to establish a foundation for assessing their quality. Next, we propose evaluating machine learning models against established communication science standards, focusing on construct validity, methods-specific external validity or cross-dataset consistency, internal validity, and intercoder reliability.

1.1.1. Computational Language Models: Background

Researchers can label previously unseen texts with predefined categories using multiple approaches, including classic supervised learning, human-in-the-loop workflows, and more recent neural language models. In this study, we focus on two widely used families of approaches: (1) pretrained language models (PLMs) that are typically fine-tuned for supervised classification, and (2) large language models (LLMs) that can be used for classification through prompting. We focus on these families because they dominate publicly available “off-the-shelf” solutions and are increasingly adopted in computational communication research.

1.1.2. Pretrained Language Models (PLMs)

PLMs, such as BERT (Bidirectional Encoder Representations from Transformers), are models trained on large amounts of unlabeled text to capture language patterns [36]. PLMs are then fine-tuned on specific, labeled datasets to classify content (e.g., hate speech) with high accuracy. This approach involves building upon a general language understanding to address a specialized task. In practice, PLMs are adapted to a classification task by training (often called fine-tuning) on labeled examples, producing a model that maps text inputs to class probabilities [36]. Fine-tuning can improve task performance when labeled data are informative and the target domain matches the deployment setting, but gains are not guaranteed and depend on data quality, label definitions, and domain shift.

Training datasets: The Data. The most common approach is to gather social media data directly from social media platforms like Twitter or Reddit [26,37,38], using publicly available Application Programming Interfaces (APIs) or using already prepared and cleaned datasets made available by other researchers. In other cases, training datasets are generated by large language models, such as ChatGPT, which produce specific content, such as offensive posts.

Training Datasets: Labeling. Models are trained on datasets considered the “accurate” classification of data into predefined labels. Labels can be broad, like “harassment” or “bullying,” or specific, such as “racist slur,” “threat,” or “derogatory language” [2]. Labeling this data is challenging, as concepts of “hate” or “toxicity” differ across cultures, languages, and contexts [26]. Most training dataset creators rely on crowdsourcing platforms like Amazon Mechanical Turk or Appen, or professional labeling services to annotate large volumes of content [37]. In these setups, hundreds of annotators tag each post as hateful, offensive, threatening, or none of the above. The schema varies: some datasets use binary classifications (“hate” vs. “not hate”), others adopt multiclass schemes (“racism,” “sexism,” “neutral”), and some implement hierarchical structures (“offensive” vs. “offensive with targeted hate” vs. “offensive without targeted hate”) [23,38]. Labeling content is subjective, prompting researchers to use inter-annotator agreement metrics like Cohen’s kappa or Krippendorff’s alpha to assess label consistency [17]. In such cases, domain experts, such as sociologists or linguists, may revise the annotation scheme or address contentious examples. Some research groups adopt iterative labeling, starting with crowd-based labels, identifying inconsistencies, and resolving them with expert annotators who clarify subtle distinctions [38].

The Model Training Process. Training itself is an iterative process [36]. A model begins with randomly initialized parameters, essentially, random guesses about how to transform an input (e.g., a social media post) into an output (i.e., a label, such as “racist” or “not racist”). It then refines these parameters by comparing its predictions to the correct “answer” in the dataset.

Evaluating the Accuracy of Models. Harmful speech detection models often report high accuracy based on in-sample testing, but these figures rarely reflect real-world performance [17]. Models frequently overfit to their training data and struggle with unseen content, especially when trained on narrow communities or slur sets, with accuracy dropping to 75–80% on new data [39]. Cross-domain and multi-dataset evaluations are recommended to improve generalization [2,17]. The lack of cross-platform testing raises concerns about generalizability, reinforcing the need for broader validation across different platforms and user groups [2].

Transparency. Evaluating computational models for content classification relies on transparency regarding dataset creation, annotator instructions, and model training. Davidson et al. [26] highlight the risks of keyword-based sampling in hate speech datasets, while Mathew et al. [1] demonstrate the importance of explainable annotations. Open-sourcing code along with labeling criteria and dataset details like source platforms and linguistic diversity enhances trustworthy evaluation [40]. As hateful language evolves swiftly, this openness is vital. Models like SOSNet [41] illustrate how design interpretability can be structured.

Three procedural decisions apply uniformly across all cross-dataset evaluations and are documented here for transparency. First, where the size of the negative class substantially exceeded the positive class, the negative class was down sampled to match the positive class size, ensuring that model performance metrics are not inflated by class imbalance. Second, BiasCorp does not contain a discrete racism label; binary labels were derived using the confidence-weighted scoring method described in the original dataset paper, whereby a weighted bias score above 1 is treated as positive (racially biased) and a score of 1 or below as negative. Bias scores range from 1 to 5, and confidence scores from 1 to 10 per annotator, with weights applied proportionally. Third, the Jigsaw Unintended Bias in Toxicity dataset provides continuous toxicity scores ranging from 0 to 1; following the binarization threshold recommended in the dataset’s official documentation, scores of 0.5 or above are treated as positive (toxic/threat) and scores below 0.5 as negative. In the Supplementary Materials, we present a Label Mapping Map for each target task, specifying which original dataset labels are coded as “positive,” which as “negative,” and which are excluded.

1.1.3. Large Language Models (LLMs)

LLMs are generative models trained to predict and generate text. Beyond pretraining on large corpora, many contemporary LLMs are instruction-tuned and aligned with human feedback to follow natural-language prompts. For classification, researchers can prompt an LLM to assign labels (zero-shot or few-shot) or to produce structured outputs, which can reduce the need for task-specific training data but introduces sensitivity to prompt design, model safety filters, and hidden training data [14]. A large body of work has studied prompting and instruction-following for classification; in this manuscript, we treat GPT-5 as a transparent baseline under a fixed prompting protocol rather than as a substitute for domain-specific validation.

1.1.4. Evaluation Framework: Validity, Reliability, and Transferability

We treat deep computational methods, PLMs and LLMs, that are used for automated content classification as measurement devices whose validity must be demonstrated rather than assumed [42]. We organize evaluation around three questions that map directly onto common social-scientific validity concerns and recent validation reviews [8,43].

Construct validity asks whether the labels correspond to the intended concept and whether the operational definition is documented (e.g., label definitions, examples, annotator instructions, and data provenance; [12]). Annotation reliability (i.e., human intercoder reliability) asks whether humans could apply the coding scheme consistently (e.g., Krippendorff’s alpha or related agreement statistics; [11]). External validity, in this context, concerns whether performance generalizes across time, platforms, languages, communities, or sampling frames, particularly when the model is applied beyond its training context [22,30]. Recognizing that external validity is more frequently perceived in terms of sample-to-population generality, we refer to this type of validity for this reason as “cross-dataset consistency.” Recent work further suggests that instruction-based variants can improve robustness against certain training-data biases, reinforcing the value of explicit instructions as part of the measurement instrument [44]. See Figure 1.

2. Materials and Methods

This study aims to assist communication scholars in evaluating and selecting computational models for classifying harmful content, such as hate speech, racism, threats, bullying, and offensive language. The methodology was designed to mirror what researchers might encounter when selecting or deploying a model using publicly available resources. Three core aspects of the methodology are described here: model selection, training dataset selection, and cross-dataset testing. Links to models and datasets are on the online Supplementary Materials section.

2.1. Model Selection

We followed a multi-criteria selection process to identify widely used models for five harmful language tasks (hate speech, offensive language, bullying/harassment, racism, and threats). We began with the Hugging Face model hub and the model papers cited in their documentation, then cross-checked task popularity using download counts and citations to prioritize models that are actively used in research and practice. In total, 27 models were identified and examined: 7 hate, 7 threat, 7 offensive language, 4 bullying, and 2 racism models. For each model we recorded the architecture, reported training data, label mapping, and any published evaluation details.

Types of models. Most selected models are BERT-family encoders fine-tuned for classification, but the set also includes alternative transformer variants and models trained with different pretraining or fine-tuning regimes to capture common choices available to researchers.

2.2. Training Dataset Selection

Datasets were sourced primarily from the Hugging Face Datasets hub and from the original papers linked in model documentation. When multiple datasets existed for a task, we prioritized those that are (a) publicly accessible, (b) accompanied by a paper or technical report, and (c) commonly used in benchmark comparisons. We extracted metadata about platform, language, sampling, label definitions, and availability of annotation documentation. Because many datasets contain multiple labels, some datasets contribute to more than one task category. For example, TDavidson’s corpus includes hate and offensive labels, enabling separate operationalizations depending on label mapping decisions.

2.3. Documentation Extraction and Coding (RQ1–RQ2)

For RQ1–RQ2, we created a structured extraction form aligned with prior validation guidance [12,43]. For each model and dataset, we recorded: (1) how the construct was defined (if at all), (2) the annotation protocol (instructions, examples, adjudication), (3) annotator characteristics and training, and (4) any reported inter-annotator agreement (e.g., Krippendorff’s alpha). When papers did not report information, we coded the item as not reported rather than inferring it. A second author audited a subset of extracted fields against the original sources to reduce transcription error.

2.4. Cross-Dataset Testing

To evaluate cross-dataset consistency (RQ3), we tested each model not only on its original benchmark dataset (when available) but also on multiple alternative labeled datasets that operationalize similar constructs. We standardized text preprocessing and label mapping across datasets to the extent possible, then computed performance on held-out splits without additional fine-tuning. This design approximates the common research scenario in which a classifier trained on one platform or population is applied to another; for example, a model trained on Twitter may be used to code YouTube comments or forum posts. Such domain shifts are known to reduce performance and can introduce systematic error [6,30]. We report accuracy and macro-averaged F1 for each model–dataset pairing and summarize generalization as the difference between within-dataset and out-of-domain performance. Where feasible, we complement these summaries with qualitative error inspection to identify recurrent failure modes (e.g., implicit hate, sarcasm, or contextual slurs).

2.5. GPT-5 Benchmark (LLM Prompting)

To contextualize results for researchers considering LLM-based labeling, we evaluated GPT-5 on the same dataset splits using a fixed prompt and the same label set used for each task. Inference was executed via OpenAI’s Batch API to ensure identical prompts and settings across items. We constrained outputs to a single label from the task’s label vocabulary and used deterministic decoding to reduce run-to-run variation. GPT-5 is not treated as ground truth; rather, it serves as a comparable, prompt-based classifier whose strengths and limitations can be contrasted with fine-tuned PLMs [7].

3. Results

RQ1: How do published harmful language classifiers document construct validity (concept definitions, labeling instructions, and training data provenance)?

Only 19 of the 27 models disclosed their training datasets. The level of transparency regarding the process of making and evaluating these datasets varies.

Table 1. Models and their respective training datasets *.

	Model Name	Dataset Name	Dataset Information	Annotator Guidelines	Inter-Annotator Agreement **	Final Label Decision
Hate	hatexplain	HateXplain	250 Amazon MTurk 20 k tweets and GAB comments	Definition examples	K = 0.46 (not acceptable)	Majority
	facebook_roberta	DynaHate	Not mentioned 40 k tweets and comments	Definition	K = 0.52 (not acceptable)	Majority
	dehatebert_ non-English	TDavidson	Appen 25 k tweets	Definition	A = 0.92 (reliable)	Majority
	distilroberta_of-fensive_hate	Badmatr11x	Not mentioned 57 k tweets	Not mentioned	Not mentioned	Not mentioned
	roberta_large_hate	TweetHate	Amazon MTurk 7 k tweets	Not mentioned	K = 0.53 (not acceptable)	Agreement of at least 2 of 5 annotators
Racism	deberta	TDavidson	Appen 25 k tweets	Definition	A = 0.92 (reliable)	Majority
Bullying	lstm_bert cyberbully-ing_bert	Kaggle Cyberbullying Dataset	Not mentioned 47 k tweets	Not mentioned	Not mentioned	Not mentioned
Offensive	distilroberta_olid twitter_roberta offensive_speech_ detector	OLID	Appen 14,100 tweets	Test Set	F = 0.83 (al- most perfect)	Majority
	hatexplain	HateXplain	250 Amazon MTurk 20 k tweets and GAB comments	Definition examples	K = 0.46 (not acceptable)	Majority
	distilroberta_ offensive_hate	Badmatr11x	Not mentioned 57 k tweets	Not mentioned	Not mentioned	Not mentioned
	llama	TDavidson	Appen 25 k tweets	Definition	A = 0.92 (reliable)	Majority
Threat	ensemble lstm_glove nb_svm lstm_rnn detoxify	Toxic Comment Classification	Not mentioned 160 k tweets	Not mentioned	Not mentioned	Not mentioned
	detoxify_unbiased	Jigsaw Unintended Toxicity Bias	Appen 1.05 M comments	Not mentioned	Not mentioned	Majority

* Of the 27 models, 8 were not included in this table, as they did not disclose their training datasets. These are: hatebert (hate), distilbert_hate (hate), distilbert_cyberbully (Bully), fair_cyberbully (Bully), xlm_r_racismo (Racism), bert_threatening (Threat), and arash_bert (Offensive). The remaining 19 are listed above. ** Accuracy levels are inconsistent across metrics. We use language associated wit beach metrics, as indicated in its related article. K = Krippendorff’s alpha [21], F = Fleiss’ Kappa [45], C = Cohen Kappa [45], ICC = Intraclass Correlation Coefficient [46], A = Appen Platform’s score.

details the findings regarding annotators, inter-annotator agreement, and the label selection process. Where data was unavailable, it is indicated by “not mentioned”. It should be noted that models, such as detoxify, were trained to classify posts into more than one label, such as toxic, obscene, insult, threat, and identity_hate. In Table 1 below, we repeat these models, as they reflect the multiple labels this study explores. However, aggregate data for models are reported for unique models, with no replications. The same applies to training datasets, which often provide multiple labels for training multiple models.

Overall, we find that different models for the same label (e.g., Hate) were often trained on the same datasets. Both the bullying models whose training datasets are accessible were trained on the same dataset; 3 of the 7 offensive language models were trained on one dataset, and 5 of the 7 Threat models were trained on the same dataset. This is a common practice because benchmark datasets offer standardized, high-quality resources that enable consistent evaluation, reproducibility, and accelerate model training by removing the overhead of data collection [16]. However, this practice also highlights even further the importance of being able to evaluate, or at least better understand, the training models.

In contrast to traditional manual content analysis, where just a few well-trained human coders label the datasets, the process of labeling was much different. Of the total 11 models with at least some information about the training datasets, 10 used crowdsourcing platforms, such as Amazon MTurk or Appen, to manually label the posts. The number of annotators (i.e., “human coders”) is only mentioned in one model, hatexplain. The annotator guidelines across hate, bullying, racism, offensive, and threat detection datasets vary significantly in depth and transparency, as related to the training of the annotators. To systematically evaluate them, we categorize the guidelines into six types: Definition, Examples, Test Set, Spans, and Not Mentioned. The “Definition” category indicates that datasets provided annotators with explicit definitions of hate or bias and their different forms. Six of the models utilized datasets that indicated that a definition was given to annotators. However, only four disclosed the definition, all relying on the same training dataset. The definition provided in the TDavidson dataset, for instance, is “A language that is used to express hatred towards a targeted group or is intended to be derogatory, to humiliate, or to insult the members of the group.”

The “Examples” category indicates that annotators were given examples of labeled content, such as hate speech or racism, as part of their guidelines. Only the HateXplain dataset provided multiple examples with classification, target community, and span annotations to help the annotators understand the task. The “Spans” category, also seen only in HateXplain, indicates that instructions were given to annotators by marking specific portions of the text to highlight where hate is expressed. In other words, rather than providing examples of hateful content, for instance, this approach to training shows annotators how a specific type of content, as hate or racism, is being manifested in text. The hatexplain model relies on the HateXplain training dataset, which provides a sample of posts with spans that experts highlight as a reference for annotators.

Models with the “Test Set” note in the table suggest that a sample of pre-labeled posts was used to evaluate and select annotators before the complete annotation task. These posts are labeled by experts and serve as a benchmark to assess annotator reliability. In particular, 3 models in this study rely on the OLID dataset, which incorporates such a test set and imposes a minimum accuracy threshold. Only annotators who meet this threshold on the test set are allowed to proceed with labeling the full dataset, ensuring higher annotation quality. However, 16 of the 27 models rely on 10 datasets that fall under the “Not mentioned” group, where annotator instruction details are absent, reducing transparency and limiting interpretability.

3.1. Additional Training Datasets

Dataset Information. The limited amount of training data used for these models and the scarcity of information about them prompted us to explore additional publicly available datasets for training. Specifically, for bullying, offensive language, and racism, the scarcity of training datasets led us to identify supplementary datasets for cross-evaluation. This approach aligns with prior research, which emphasizes that cross-dataset evaluation is essential for assessing model generalizability across domains, particularly in detecting social bias and harmful content [17]. We identified an additional 18 training datasets: 4 for Racism, 4 for Offensive Language, 6 for Bullying, 3 for Hate Speech, and 2 for Threats (one dataset included both hate and offensive labels). A total of 4 datasets—BiasCorp, OffensiveLang, MMHS150k, and Implicit Hate Corpus—used Amazon Mechanical Turk annotators to assess bias and confidence scores. Three datasets—HASOC, Online Harassment Research, and New-Wave-Hate—are annotated by their authors. The Large-Scale Hate Speech dataset contains both hate and offensive labels and was labeled by 20 graduate and undergraduate students. One racism dataset, Are You Racist Or Am I Seeing Things, is labeled by Appen annotators. The other 7 datasets did not disclose their annotator sources or any other information about the process.

Annotator Information. Of the 18 additional datasets, 4 use Amazon Mechanical Turk annotators. The Large-Scale Hate Speech dataset uses 20 graduate and undergraduate student annotators. The authors of three datasets, including HASOC, Automated Cyberbullying Detection, and New-Wave-Hate, developed their own codebooks and annotated them themselves. The Are You Racist or Am I Seeing Things dataset employed Appen annotators, while Cyberbullying Twitter personally contacted 17 annotators. However, detailed information about them is not publicly disclosed. The Automated Cyberbullying Detection dataset incorporated pre-labeled aggressive/non-aggressive messages from prior work [47]. Seven of the 18 additional datasets did not disclose the source of their annotators.

Other labeling procedures. The Automated Cyberbullying Detection dataset combines pre-labeled aggressive/non-aggressive messages from Elsafoury et al. [47] with synthetically generated user data (e.g., age, gender, student grade) to simulate user interactions. Final bully/not-bully labels are determined by integrating aggression labels with three computed factors: intent to harm, peerness, and repetition. The Offensive-Lang dataset employs a novel strategy: generating posts with ChatGPT and labeling them manually via Amazon Mechanical Turk. See

Table 2. Additional training datasets.

Label	Dataset Name	Dataset & Annotators Information	Annotator Guidelines	Inter-annotator Agreement *	Final Label Decision
Hate	Large-Scale Hate Speech	20 Student annotators 68.5 k tweets	Definition Examples	K = 0.395 (not acceptable)	Unanimous
	Implicit Hate Corpus	Amazon MTurk 19 k tweets	Definition Examples Test	ICC = 0.616 (moderate)	Majority
	New-Wave-Hate	Authors (Amazon MTurk -> authors)	Definition Codebook	F = 0.84 (almost perfect)	Single
Racism	BiasCorp	Amazon MTurk 45 k comments	Not mentioned	Not mentioned	Not mentioned
	Are You Racist, Or Am I Seeing Things	Appen, Experts 7 k tweets	Test	F = 0.57 (moderate)	Majority
	MMHS150k	Amazon MTurk annotators 150 k tweets	Definition Examples	Not mentioned	Majority
	Twitter Racism Dataset	Not mentioned 13.5 k tweets	Not mentioned	Not mentioned	Not mentioned
Bully	Cyberbullying Twitter	17 annotators personally contacted 62.5 k tweets	Not mentioned	K = 0.67 (tentatively acceptable)	Majority
	Online Harassment Dataset	Not mentioned. 8.2 k tweets	Not mentioned	Not mentioned	Not mentioned
	Kaggle Insults Dataset	Not mentioned 2 k comments	Not mentioned	Not mentioned	Not mentioned
	Bullying Traces Dataset	Not mentioned 1.7 k tweets	Not mentioned	Not mentioned	Not mentioned
	Automated Cyberbullying Detection	Pre-labeled. 15 k messages	Definition Codebook	Not mentioned	Threshold
	Online Harassment Research	Authors 35 k tweets	Definition	C = 0.84 (almost perfect)	Majority
Offensive	OffensiveLang	Amazon MTurk annotators 8270 posts generated by ChatGPT	Definition	C = 0.54 (moderate)	Majority
	Multimodal Meme Dataset	Not mentioned 743 meme content	Examples Codebook	F = 0.4–0.5 (moderate)	Single
	HASOC	Authors 6 k posts	Definition	% of agreement: 72% (acceptable)	Not mentioned
	Large-Scale Hate Speech	20 Student annotators 68.5 k tweets	Definition Examples	K = 0.395 (not acceptable)	Unanimous
Threat	Combined Toxicity Profanity	Not mentioned 710 k comments	Not mentioned	Not mentioned	Not mentioned. Comments can have multiple labels.
	Suspicious Tweets	Not mentioned 59 k tweets	Not mentioned	Not mentioned	Not mentioned

* K = Krippendorff’s alpha [21], F = Fleiss’ Kappa [45], C = Cohen Kappa [45], ICC = Intraclass Correlation Coefficient [46], A = Appen Platform’s score.

.

RQ2: What evidence is provided about annotation reliability in the datasets used to train these classifiers (e.g., inter-annotator agreement), and how is it reported?

Across datasets, evidence on annotation reliability is uneven and often difficult to interpret because papers differ in what they report. Some datasets provide detailed coder instructions and report formal agreement statistics, whereas others provide only minimal descriptions of the labeling workflow or omit agreement entirely. This variability matters because classifier performance is bounded by label quality: When labels are noisy or ambiguous, high predictive accuracy may reflect replication of noise rather than valid measurement [25].

Types of reported reliability. When agreement was reported, Krippendorff’s alpha was the most common statistic, alongside percent agreement and occasional use of Cohen’s kappa or Fleiss’ kappa. However, many papers did not specify key details needed for comparability, such as the unit of analysis, whether agreement was computed before or after adjudication, and how many annotators coded each item.

Final labels and adjudication. Even when agreement statistics were available, many datasets used majority voting or a final adjudication step to produce a single “gold” label for training. This practice can increase label consistency but can also mask systematic disagreements about borderline cases, which are often theoretically consequential in harmful language research [27].

RQ3: How well do these classifiers generalize when applied to alternative labeled datasets beyond their original training data (out-of-domain testing)?

We selected multiple training datasets for each harmful content category, including, where available, the datasets originally associated with the selected models, as well as widely used, stand-alone datasets that contained the same labels.

3.2. Hate Speech Detection Models

Training accuracies for these models range from 69% to over 94%, but cross-dataset performance often fluctuates, indicating challenges with cross-dataset consistency. For example, distilroberta_offensive_hate achieves a high accuracy of 96% on its training dataset, Badmatr11x, but drops to around 56–91% on other datasets. Our cross-dataset analysis reveals that the distilroberta_offensive_hate and roberta_large_offensive_hate models achieved the highest average accuracy (0.74, F1 = 0.72; 0.73, F1 = 0.73), respectively. These values are significantly lower than the values (0.97, F1 = 0.97; and 0.99, F1 = 0.99, respectively). See

Table 3. Hate label results *.

Dataset Model	Dyna-Hate	HateX-Plain	Badmatr11x	TweetHate	TDavidson	Large-Scale Hate Speech	Implicit Hate Corpus	New-Wave-Hate	Average
	0.75	0.55						0.69	0.63
facebook_roberta	(0.76)	(0.55)	0.62 (0.71)	0.73 (0.74)	0.25 (0.33)	0.85 (0.90)	0.62 (0.59)	(0.68)	(0.66)
	0.54	0.84						0.60	0.76
hatexplain	(0.43)	(0.84)	0.92 (0.90)	0.73 (0.63)	0.93 (0.92)	0.98 (0.97)	0.51 (0.37)	(0.47)	(0.70)
distilroberta_offen-	0.56	0.65						0.63	0.74
sive_hate	(0.50)	(0.65)	0.96 (0.96)	0.69 (0.66)	0.97 (0.97)	0.91 (0.94)	0.53 (0.52)	(0.57)	(0.72)
	0.56	0.70						0.72	0.73
roberta_large_hate	(0.50)	(0.70)	0.77 (0.82)	0.98 (0.98)	0.59 (0.69)	0.96 (0.96)	0.56 (0.48)	(0.68)	(0.73)
dehate-	0.60	0.55						0.57	0.62
bert_mono_english	(0.57)	(0.56)	0.63 (0.72)	0.71 (0.72)	0.76 (0.73)	0.90 (0.93)	0.60 (0.55)	(0.57)	(0.64)
	0.56	0.54						0.60	0.61
hatebert	(0.50)	(0.52)	0.50 (0.46)	0.75 (0.71)	0.39 (0.39)	0.94 (0.56)	0.60 (0.55)	(0.58)	(0.53)
	0.50	0.50						0.54	0.56
distilbert_hate	(0.49)	(0.47)	0.59 (0.42)	0.57 (0.50)	0.67 (0.42)	0.71 (0.43)	0.48 (0.45)	(0.50)	(0.46)
	0.81	0.79						0.74	0.75
ChatGPT-5	(0.81)	(0.79)	0.69 (0.66)	0.69 (0.64)	0.89 (0.85)	0.64 (0.58)	0.73 (0.73)	(0.70)	(0.72)

* Italicized text indicates results from the training dataset used to train the model. Bold text marks the best-performing model for each dataset. Each row shows the model’s accuracy across all datasets, with F1 scores in parentheses.

.

3.3. Threat Detection Models

This section presents the performance of seven threat detection models across four benchmark datasets, evaluated using accuracy and F1 scores. Among the models, lstm_glove achieved the highest average accuracy (0.73). detoxify_unbiased showed strong F1 performance (0.61), particularly on datasets with nuanced toxicity like Jigsaw and Combined Toxicity Profanity. In contrast, ensemble and nb_svm models exhibited significantly lower F1 scores, indicating limited effectiveness in identifying threats beyond surface-level cues. See

Table 4. Threat label results *.

Dataset Model	Toxic Comments Dataset	Jigsaw Unintended Toxicity Bias	Combined Toxicity Profanity	Suspicious Tweets	Average
ensemble	0.54 (0.42)	0.60 (0.01)	0.61 (0.03)	0.60 (<0.01)	0.59 (0.11)
lstm_glove	0.90 (0.90)	0.70 (0.43)	0.68 (0.49)	0.62 (0.25)	0.73 (0.52)
nb_svm	0.89 (0.88)	0.60 (0.02)	0.65 (0.22)	0.60 (<0.01)	0.69 (0.28)
bert_threatening	0.81 (0.76)	0.62 (0.08)	0.66 (0.26)	0.60 (<0.01)	0.67 (0.28)
lstm_rnn	0.75 (0.67)	0.63 (0.14)	0.66 (0.27)	0.60 (<0.01)	0.66 (0.27)
detoxify	0.95 (0.95)	0.58 (0.29)	0.97 (0.03)	<0.001 (<0.001)	0.63 (0.32)
detoxify_unbi-ased	0.82 (0.78)	0.76 (0.68)	0.80 (0.76)	<0.001 (<0.001)	0.60 (0.56)
ChatGPT-5	0.92 (0.92)	0.72 (0.64)	0.75 (0.70)	0.60 (0.38)	0.75 (0.72)

* Italicized text indicates results from the training dataset used to train the model. Bold text marks the best-performing model for each dataset. Each row shows the model’s accuracy across all datasets, with F1 scores in parentheses.

.

3.4. Bully Detection Models

On average, the cyberbullying_bert model demonstrated the highest performance, with a mean accuracy of 0.71 and an F1 score of 0.55 across all datasets. The distil-bert_cyberbully model consistently performed the lowest, yielding a mean accuracy of 0.40 and an F1 score of 0.36. The highest individual performance was achieved by the lstm_bert model on the Kaggle Cyberbullying Dataset, with an accuracy of 0.95. The lowest recorded performance was by the distilbert_cyberbully model on the Online Harassment Dataset, with an accuracy score of 0.24. See

Table 5. Bully label results *.

Model Dataset	Kaggle Cyberbullying Dataset	Automated Cyberbullying Detection	Bullying Traces Dataset	Online Harassment Dataset	Cyberbul- Lying Twitter	Kaggle Insults Dataset	Online Harassment Corpus	Average
lstm_bert	0.95 (0.95)	0.65 (0.63)	0.75 (0.54)	0.93 (0.64)	0.53 (0.40)	0.64 (0.57)	0.47 (0.46)	0.70 (0.60)
fair_cyber-bully	0.61 (0.55)	0.81 (0.76)	0.64 (0.61)	0.80 (0.56)	0.47 (0.38)	0.72 (0.69)	0.53 (0.51)	0.65 (0.58)
distilbert_ cyberbully	0.40 (0.34)	0.46 (0.44)	0.41 (0.41)	0.24 (0.21)	0.45 (0.35)	0.44 (0.37)	0.42 (0.41)	0.40 (0.36)
cyberbully-ing_bert	0.86 (0.86)	0.64 (0.48)	0.69 (0.50)	0.80 (0.50)	0.72 (0.47)	0.55 (0.54)	0.69 (0.48)	0.71 (0.55)
ChatGPT-5	0.50 (0.48)	0.83 (0.80)	0.54 (0.48)	0.96 (0.79)	0.64 (0.48)	0.81 (0.80)	0.54 (0.53)	0.75 (0.66)

* Italicized text indicates results from the training dataset used to train the model. Bold text marks the best-performing model for each dataset. Each row shows the model’s accuracy across all datasets, with F1 scores in parentheses.

.

3.5. Offensive Speech Detection Models

The hatexplain model demonstrated the highest overall average performance across all datasets, achieving an accuracy of 0.73 and an F1-score of 0.62. While the distilrob-erta_offensive_hate model had the lowest average F1-score, it achieved the single best performance with an accuracy of 0.97 and an F1-score of 0.95 on the TDavidson dataset. The analysis also revealed that F1-scores were consistently lower than accuracy scores for most models, indicating a common challenge in striking a balance between precision and recall. See

Table 6. Offensive label results *.

Dataset Model	OLID	Offensive-Lang	TDavidson	HASOC	Large-Scale Hate Speech	Multimodal Meme Dataset	Average
arash_bert	0.90 (0.90)	0.30 (0.29)	0.79 (0.72)	0.59 (0.48)	0.86 (0.68)	0.61 (0.56)	0.67 (0.60)
distilroberta_olid	0.86 (0.77)	0.26 (0.26)	0.79 (0.72)	0.68 (0.51)	0.89 (0.83)	0.61 (0.56)	0.68 (0.61)
twitter_roberta	0.86 (0.78)	0.28 (0.26)	0.87 (0.80)	0.62 (0.49)	0.88 (0.82)	0.60 (0.54)	0.69 (0.61)
offensive_speech_de-tector	0.75 (0.71)	0.26 (0.26)	0.75 (0.83)	0.70 (0.52)	0.91 (0.85)	0.61 (0.52)	0.66 (0.61)
hatexplain	0.75 (0.68)	0.28 (0.19)	0.88 (0.93)	0.87 (0.53)	0.98 (0.97)	0.59 (0.40)	0.73 (0.62)
llama	0.73 (0.66)	0.27 (0.26)	0.90 (0.88)	0.68 (0.50)	0.87 (0.76)	0.62 (0.60)	0.68 (0.61)
distilroberta_ offensive_hate	0.73 (0.61)	0.22 (0.19)	0.97 (0.95)	0.82 (0.42)	0.89 (0.78)	0.59 (0.38)	0.70 (0.57)
ChatGPT-5	0.72 (0.61)	0.48 (0.47)	0.84 (0.83)	0.51 (0.51)	0.72 (0.72)	0.51 (0.49)	0.52 (0.49)

* Italicized text indicates results from the training dataset used to train the model. Bold text marks the best-performing model for each dataset. Each row shows the model’s accuracy across all datasets, with F1 scores in parentheses.

.

3.6. Racism Detection Models

Racism models like deberta are trained on a mix of datasets, including TDavidson, but even with reported training accuracies up to 70%, their cross-dataset F1 scores drop to 40–60%. xlm_r_racismo has comparable performance to deberta, but deberta has higher F1 scores. See

Table 7. Racism Label Results *.

Model Dataset	Are You Racist, Or Am I Seeing Things?	BiasCorp	MMHS150k	Twitter Racism Dataset	Average
deberta	0.62 (0.60)	0.52 (0.52)	0.50 (0.48)	0.61 (0.41)	0.56 (0.50)
xlm_r_rac-ismo	0.63 (0.58)	0.51 (0.32)	0.51 (0.38)	0.63 (0.46)	0.57 (0.43)
ChatGPT-5	0.84 (0.82)	0.53 (0.53)	0.63 (0.63)	0.84 (0.83)	0.71 (0.70)

* Italicized text indicates results from the training dataset used to train the model. Bold text marks the best-performing model for each dataset. Each row shows the model’s accuracy across all datasets, with F1 scores in parentheses.

.

4. Discussion

This study proposed and implemented a structured process for evaluating machine learning models and generative AI tools used to label harmful content, positioning automated classification measurement instruments rather than purely technical tools [8,42,43]. By following the steps outlined here, researchers can assemble the information needed to justify model selection in ways that align with long-standing standards of validity, reliability, and transparency in content analysis [11,20,21]. While this study presents empirical findings on model performance across datasets, the broader contribution lies in reframing model choice as a methodological decision embedded in theory, construct definition, and research design [14,43].

We started with Figure 1 above, which presents the evaluation workflow. Based on this study, we now document the key decision thresholds applied at each stage to support replication and guide researchers adopting this framework. At the label selection stage, majority voting among 3 annotators was the most common final label decision method across datasets, used in most of the 27 datasets examined; supplementary approaches included unanimous agreement among 5 annotators with curator adjudication for ties, agreement of at least 2 of 5 annotators, and confidence-weighted aggregation for continuous-score datasets (Jigsaw: threshold ≥ 0. 0.5; BiasCorp: threshold > 1). Researchers selecting datasets should note the label decision method, as single-annotator labels or majority votes with low inter-annotator agreement may introduce noise into cross-valuation results. At the inter-annotator agreement stage, reported Krippendorff’s α values across all datasets in this study ranged from 0.39 to 0.84. Following the benchmarks established in content analysis [45], values below 0.40 are considered poor, and datasets relying on them should be flagged as higher-risk; values between 0.40 and 0.60 indicate moderate agreement and warrant interpretive caution; values above 0.60 are generally considered acceptable for scholarly use. These thresholds are domain-dependent for tasks involving subjective or implicit content (e.g., implicit hate, racism), which tend to yield lower agreement and may require adjusted expectations. For the ChatGPT-5 evaluation, inference was run via OpenAI’s Batch API using the v 1/responses endpoint with GPT-5.1, reasoning effort set to low, verbosity set to minimal, and maximum output tokens capped at 128 per post; these settings are reported to ensure the prompting protocol is fully reproducible. Finally, for cross-evaluation stability, datasets with fewer than 500 labeled positive class instances may yield unstable F1 estimates; results from such datasets should be interpreted with caution and flagged in reporting. This study addresses recent methodological critiques by evaluating the model rather than just developing it, with an emphasis on documentation quality, annotator agreement, testing across datasets, and procedural transparency [8,9,12,13]. Across all harmful content categories, we find substantial variation in how datasets report construct definitions, annotator preparation, coding procedures, and inter-annotator reliability. Although this unevenness does not necessarily reflect poor-quality work, it limits comparability and leaves important measurement properties only partially visible to researchers using these classifiers [11,21,42].

These gaps matter because classifiers inherit the assumptions embedded in their training data. Many datasets provide limited conceptual definitions for labels such as hate or racism, relying more on examples or crowdsourced judgments than on sustained coder training. Prior research shows that such judgments are shaped by annotator background and context, meaning that labels are socially situated rather than purely objective [48,49,50]. For communication research, this underscores the need to treat classifiers as measurement instruments whose construct validity must be made explicit and justified, rather than as neutral tools [14,20,21].

Our cross-dataset evaluation further shows that performance varies across labels and domains. Hate speech, offensive language, and threat models generally perform better than racism models, which are less stable across settings, likely because racism is especially context-dependent and vulnerable to bias in annotation and training data [27,39,51]. Models built to detect multiple harms also tend to underperform compared with models optimized for a single construct, suggesting limits to broad, one-size-fits-all approaches. At the same time, cross-dataset testing is constrained by differences in how datasets define labels, so performance reflects both model generalization and variation in construct operationalization. Even so, this kind of testing remains essential because it more closely approximates real-world research use, where models are often applied beyond their original training context [11,30].

The comparison between PLM classifiers and GPT-5 highlights a related tradeoff. PLM-based systems often provide more visible documentation of training and evaluation, whereas LLM-based classification offers flexibility but adds opacity regarding training data, alignment, and prompt sensitivity [12,52]. Although GPT-5 can be competitive under a consistent prompting protocol, recent research suggests that LLM-based classification may also introduce systematic biases across contexts and targets [7,22]. Overall, our findings support a central conclusion: automated classifiers in communication research should be treated as socially and methodologically embedded measurement instruments, requiring explicit attention to construct definition, annotation reliability, and cross-context validation. The framework proposed here is intended to make those considerations more visible, systematic, and usable in future research [8,42].

4.1. Implications for Researchers, Reviewers, and Editors

Treat classifiers as measurement instruments. Automated classifiers should be treated as measurement tools rather than purely technical systems. Researchers should specify what construct a model measure, how it is operationalized in the training data, and how this aligns with theory.

Be transparent. Studies should report the origin of training data, annotation procedures, annotator instructions, and any inter-annotator agreement, since these practices shape how constructs such as hate or racism are defined.

Do not rely only on reported accuracy. Reporting benchmark accuracy alone says little about performance in new contexts; cross-dataset evaluation on data that resemble the target context is necessary to assess generalizability.

Align model choice with theory. Broad “harmful content” categories can obscure conceptually distinct constructs, so model labels should match theoretical distinctions such as racism, incivility, or threats.

PLM or LLM is a methodological choice. PLM classifiers may offer clearer documentation of training and evaluation, while LLM labeling provides flexibility but introduces opacity and prompt sensitivity; in both cases, researchers should document operational definitions and validate performance in the intended context.

Overall, these implications suggest that as automated classifiers become routine, communication science’s established standards for construct clarity, coder reliability, and cross-dataset consistency provide a foundation for their responsible and cumulative use [8,11,21].

4.2. Limitations and Further Research

One limitation of this study is its focus on harmful language detection; while this domain is consequential and methodologically challenging, the evaluation framework is intended to generalize to other classifier-based measures in communication research. A second limitation is that our LLM benchmark uses a single, fixed prompt designed for replicability; alternative prompts, few-shot examples, or different decoding settings may yield different performance. Future work should therefore treat prompt design as part of the measurement instrument and evaluate robustness across prompt variants. Finally, future research should expand bias and error-impact analyses, including how misclassification propagates into downstream statistical inference and how error-correction methods can be incorporated into standard workflows [53]. Importantly, the current design cannot differentiate between “generalization failure” and “construct shift,” because the cross-dataset tests used in this paper mostly involve heterogeneous datasets from different sources and annotation protocols, and performance declines may therefore arise from construct bias rather than from a failure of generalization. Furthermore, there is a potential ChatGPT leakage risk; the long-established training datasets have been included in the ChatGPT pretraining corpus, as they are publicly available. Future research using LLMs should validate findings on “recently collected or unpublished data. Lastly, another limitation of this study is that it primarily focuses on PLMs and uses only one LLM as a reference for comprehensive PLM analysis. We recognize that different LLMs and prompts could lead to different classifications, and this should be explored in future research.