Cyberbullying Detection Based on Hybrid Neural Networks and Multi-Feature Fusion

Abstract

Cyberbullying demonstrates notable metaphorical and contextual traits, characterized by a high-dimensional sparse semantic space and dynamic evolution. Pre-trained models utilize extensive textual data for learning and employ transformer-based word vector generation techniques to accurately capture intricate semantics and nuanced syntax in text. However, although a single pre-trained model demonstrates strong performance in contextual modeling, it still faces challenges including inadequate feature representation and limited generalization capability in classifying cyberbullying texts. This study proposes a cyberbullying detection model employing BERT-BiGRU-CNN (BBGC) to address this issue. The BBGC model initially employs BERT to produce word embeddings, subsequently inputs them into a BiGRU layer to acquire sequence features, and finally utilizes a CNN for the extraction of local features. The features derived from BERT, BiGRU, and CNN are integrated, followed by the application of the softmax function to yield the final outcome of cyberbullying detection. Experimental findings indicate that the BBGC fusion model surpasses individual pre-trained models in the task of detecting cyberbullying text. Furthermore, in comparison to hybrid neural network models utilizing RoBERTa, ALBERT, DistilBERT and other pre-trained models, the BBGC model demonstrates considerable advantages.

1. Introduction

Cyberbullying constitutes a significant social problem, especially among adolescents [1,2], and may result in psychological trauma, depression, and, in extreme instances, suicide. Cyberbullying encompasses actions such as disseminating rumors, posting defamatory, discriminatory, and hateful remarks, engaging in malicious harassment, and executing personal assaults. These manifestations of online violence have emerged as a significant concern in everyday social interactions [3,4,5]. If cyberbullying behaviors are not swiftly addressed, they can result in significant repercussions for the victims. Adolescent cyberbullying can be prevented through a variety of strategies [6,7]. Consequently, the significance of cyberbullying detection technology has escalated, with machine learning and deep learning methodologies being extensively utilized and demonstrated to be effective [8].

Initially machine learning techniques for detecting cyberbullying text encompassed Naive Bayes (NB) [9], Support Vector Machines (SVMs) [10], and Random Forest (RF) [11,12]. However, the fundamental bottleneck in traditional machine learning methods lies in their inherent inability to directly comprehend and process the semantics and structure of human language. These approaches—such as Naïve Bayes and Support Vector Machines—typically require converting text into discrete feature vectors based on bag-of-words representations. This transformation discards crucial linguistic information such as word order, syntax, and contextual dependencies, and fails to capture lexical semantic relationships (e.g., synonymy and polysemy). As a result, they are incapable of modeling the sequential nature and deeper semantics of language, leading to models that are computationally inefficient, difficult to generalize, and ultimately constrained by a performance ceiling that cannot be surmounted.

As deep learning has evolved, classification techniques utilizing neural network models have garnered significant acknowledgment among researchers, including Convolutional Neural Networks (CNNs) [13], Recurrent Neural Networks (RNNs) [14,15], Long Short-Term Memory (LSTM) [16], Bidirectional Long Short-Term Memory (BiLSTM) [17], Attention mechanisms [18], Bidirectional Gated Recurrent Units (BiGRUs) [19], and Graph Convolutional Networks (GNNs) [20]. Nevertheless, these neural network models employ static word embedding generation methods, which typically necessitate a substantial amount of computational resources to manage word embedding and generate word vectors. Furthermore, they are incapable of accurately managing synonyms or polysemy. This static word embedding generation method is unable to comprehend semantic and syntactic information due to its inability to account for the diversity of word meanings and changes in context. It is particularly inadequate for long text tasks that necessitate the capture of long-distance syntactic or semantic information. The advent of pre-trained models has markedly enhanced word embedding and word vector generation, exemplified by BERT (Bidirectional Encoder Representations from Transformers) [21], RoBERTa (Robustly optimized BERT approach) [22], ALBERT (A Lite BERT) [23], and DistilBERT (A Distilled Version of BERT) [24]. The pre-trained model is a transformer-based pre-trained language model capable of processing both the left and right context simultaneously. Its bidirectional encoder incorporates all preceding and following contextual information when interpreting a given word. This architecture enables it to directly compute the relationships between any two words in a sentence. Consequently, when fine-tuned for specific downstream tasks, the model can achieve strong performance without requiring massive amounts of labeled data, resulting in significant savings in both time and cost.

Nevertheless, such models still exhibit limitations in focusing on specific local information. For instance, the attention mechanism may become dispersed across numerous words that are only generally important for the task, rather than being highly concentrated on a critical local segment. When handling tasks requiring strong local constraints—such as determining whether an idiom is used correctly or identifying a compact phrasal structure—the models may fail to capture the tight, conventionalized relationships between adjacent words as adeptly as a specifically designed CNN. Additionally, strongly suggestive local keywords or phrases (e.g., the nuanced connotations in sentiment polarity) may be diluted by the model’s broader contextual understanding. The primary research inquiry of this study is the integration of pre-trained models with other neural networks to leverage the complementary benefits of multiple models.

The unique insight behind our proposed BBGC architecture lies in its design of a hierarchical and complementary feature-processing pipeline, which systematically addresses the multi-level challenges in text classification involving semantics, sequential dependencies, and local patterns. Specifically, BERT serves as the semantic foundation, resolving lexical ambiguities and providing rich contextualized embedding. Building upon this, BiGRU efficiently models bidirectional long-range dependencies in the text, capturing sentence-level logical flow. At the top, the CNN acts as a “detector” of critical local patterns, extracting the most discriminative phrase-level features from semantically and sequentially enhanced representations. This combination is not merely a stack of components; the core idea is to allow each layer to focus on its most proficient level of abstraction (e.g., semantics, structure and patterns), enabling later layers to operate more effectively on the refined outputs of preceding ones. Experiments demonstrate that this design facilitates synergistic complementarity and, in the task of cyberbullying text classification, captures the decisive features of text more comprehensively than single models or other dual-component combinations.

Many current studies are focused on developing model methods for online bullying text classification using neural networks or machine learning. Research on the integration of pre-trained models and neural networks, or deeper hybrid neural network models, remains underexplored. To fill this gap, the purpose of this research is to investigate model methods with stronger generalization capabilities for multi-class or binary classification of online bullying texts using a combination of pre-trained models and neural networks. To accomplish this goal, the study will address the following issues and provide the following contributions:

• By combining pre-trained models with neural networks, the study effectively optimizes the limitations of a single neural network in handling text classification tasks, such as insufficient processing of complex semantics, syntactic structures, difficulties in handling long sequence dependencies, and poor generalization ability.
• A comparison of hybrid neural network models combining BERT with pre-trained models such as RoBERTa, ALBERT, and DistilBERT revealed the performance differences of various pre-trained models in different neural network architectures when handling online bullying text type classification and online bullying detection tasks.
• The hybrid neural network model integrates multiple neural network architectures, adjusting different components and hyperparameters to evaluate their effectiveness in cyberbullying text classification, aiming to identify the best-performing model.

The structure of this paper is as follows: Section 1 is the introduction. Section 2 describes the current research status of this study. Section 3 introduces the structure of the hybrid neural network model and dataset. Section 4 compares the classification performance of the proposed model with other hybrid neural network models. Section 5 is a discussion of recent related studies. Section 6 provides the conclusion and future outlook.

2. Related Research

In recent years, hybrid neural network models have garnered considerable attention in the study of cyberbullying text classification tasks. Paul et al. [25] conducted a comparison of the fine-tuned BERT model against several models, including CNN and LSTM. The proposed BERT model attained an F1-Score of 0.96 in cyberbullying text classification, though it exhibited suboptimal performance with cross-category comments. Obaid et al. [26] developed a four-layer LSTM multi-layer model for the detection of cyberbullying, integrating fuzzy logic to assess the severity of bullying. The model’s accuracy, recall, and F1-Score were, respectively, 0.9367, 0.9362, and 0.9364, suggesting that the overall performance requires improvement. Kumar et al. [27] proposed the Bi-GRU Attention-CapsNet (Bi-GAC) hybrid model, using ELMo as input, combining BiGRU with CapsNet, and incorporating the Attention mechanism. This model outperforms CNN and other models in the binary classification task of social media cyberbullying detection; however, it is inapplicable to multi-class tasks. Bensoltane et al. [28] employed the BERT-BiGRU model for binary classification on Arabic datasets, attaining commendable performance; however, it remains suboptimal in multi-class tasks. Daraghmi et al. [29] constructed a CNN-BiLSTM-GRU hybrid model with stacked word embeddings, which outperforms single models in Arabic text binary classification but does not address the multi-label classification scenario. Khan et al. [30] proposed HCovBi-Caps, a novel deep learning model based on Convolutional Neural Networks, Bidirectional Gated Recurrent Units, and Capsule Networks, designed for hate speech classification. The model achieves optimal performance on imbalanced datasets and demonstrates significant superiority over existing methods.

Alqahtani et al. [31] employed an ensemble machine learning methodology, integrating N-gram and TF-IDF feature extraction techniques, and utilized three multi-class models, including Decision Trees, to address multi-label cyberbullying texts. The findings indicated that the ensemble model surpassed conventional machine learning techniques; however, it was less effective than multi-classifiers utilizing pre-trained models. Talpur et al. [32] employed PMI-SO text preprocessing in conjunction with SVM and NB algorithms to execute multi-class classification and ascertain the severity levels of tweets. Nevertheless, the training efficacy of artificial neural network models was constrained due to significantly imbalanced positive class data, which contained fewer than 500 instances of a single class. Kamal et al. [33] proposed a deep learning model named BiCapsHate, which employs a five-layer deep neural network architecture. The model effectively captures multi-dimensional contextual information from text sequences in both forward and backward directions through capsule networks. By innovatively incorporating a rich set of manually crafted shallow and deep auxiliary features, including the Hatebase lexicon, the model significantly enhances its perceptual capabilities. Experimental results demonstrate that BiCapsHate outperforms various lightweight BERT-based methods across multiple balanced and imbalanced datasets.

Furthermore, extensive investigations are ongoing in the study of cyberbullying text classification and detection utilizing alternative pre-trained models. Ogunleye et al. [34] and associates utilized a multilingual variant of XLM-RoBERTa for cyberbullying detection, exhibiting enhanced efficacy relative to conventional machine learning techniques such as SVM and RF, in addition to the BiLSTM-attention model. Nonetheless, their research remains confined to binary classification tasks. Umer et al. [35] and his team amalgamated the RoBERTa model with GLOVE word-embedding features derived via Principal Component Analysis (PCA) and incorporated the LIME module. This methodology surpassed current machine learning, deep learning, and transformer-based models regarding accuracy, F1-Score, and additional metrics. Tripathy et al. [36] implemented a three-class classification for cyberbullying texts using ALBERT. The model, based on ALBERT’s text feature representation, outperformed traditional methods such as Word2vec and CNN. However, it still lagged behind hybrid neural networks. Saranyanath et al. [37] proposed an ensemble method combining DistilBERT and SVM, leveraging DistilBERT’s inherent tokenization benefits and effective feature extraction capabilities to enhance the accuracy of cyberbullying detection significantly. Khan et al. [38] proposed a BiCHAT model, employing a novel deep learning architecture that integrates bidirectional LSTM, deep CNN, and hierarchical attention mechanisms. Designed specifically for tweet representation learning and hate speech detection, it demonstrates superior performance across three benchmark Twitter datasets.

Despite the aforementioned studies having achieved certain results in cyberbullying text processing, their performance remains suboptimal, with each method exhibiting distinct limitations. For instance, most current approaches focus solely on single-language analysis and are incapable of detecting or filtering content across multiple languages. Currently, no method has achieved consistently high performance in both of the following aspects: (1) accurately detecting whether a text constitutes cyberbullying (coarse-grained judgment); (2) accurately classifying the specific type of cyberbullying within the detected texts (fine-grained analysis). Moreover, most successes have been limited to unimodal scenarios and have not yet incorporated multimodal data—a direction we aim to explore in subsequent work. Therefore, this study proposes a hybrid neural network model named BERT-BiGRU-CNN (BBGC), which integrates the pre-trained BERT model with a Bidirectional Gated Recurrent Unit (BiGRU) and Convolutional Neural Network (CNN) to more effectively address cyberbullying texts. By systematically investigating the interrelationships among pre-trained models, neural networks, and their integration mechanisms, this research aims to enhance the classification and detection performance of cyberbullying texts.

3. Method

3.1. Data Preprocessing

The cyberbullying text encompasses two datasets: the multi-class cyberbullying dataset and the binary classification dataset for cyberbullying detection, both obtained from the Kaggle platform, accessed on 1 January 2026, (https://www.kaggle.com/datasets/andrewmvd/cyberbullying-classification and https://www.kaggle.com/datasets/gbiamgaurav/cyberbullying-detection), corresponding to the cyberbullying_tweets and aggression_parsed_dataset datasets, respectively. The original multi-class dataset comprised over 47,000 entries. To identify potential labeling errors, we trained a lightweight BERT classifier on the original training set and used it to make predictions on the same training set. Samples where the model’s prediction confidence was ≥0.95 and the predicted label differed from the original annotation were filtered out and independently reviewed by two annotators. Near-duplicate samples were removed between the training and test sets during this process to ensure unbiased evaluation of the test set. After manual removal of mislabeled categories and duplicate entries, the dataset was refined to 44,243 valid records. The binary-class detection dataset consists of approximately 100,000 data records. To ensure rigorous evaluation and prevent information leakage during the validation process, each dataset was partitioned into three disjointed subsets: training set (70%), validation set (15%), and test set (15%). All splits were stratified by class labels to preserve the original class distribution and ensure fairness. This partitioning ensures that no overlapping data exists between the training and testing sets, and the final model is evaluated only once on the held-out test set. The validation set was used solely for hyperparameter tuning and model selection; the test set was held out exclusively for final evaluation.

During the training phase, we begin with the final cleaned dataset. First, a fixed random seed is employed to partition the data into training, test and validation sets. The test set remains completely isolated and is not accessed at any stage during the model development and hyperparameter tuning process. For each hyperparameter configuration, a new model was trained from scratch on the training set. Early stopping was applied based on training loss, and the model at the stopping epoch was evaluated on the held-out test set only once at the end of tuning. No configuration selection was performed based on test set performance; all reported test results are from the final model with fixed hyperparameters. To facilitate training, the categories in the multi-class cyberbullying dataset—age, race, gender, religion, and non-cyberbullying—were designated as 0, 1, 2, 3, and 4, respectively. In the binary classification dataset for cyberbullying detection, the labels were 0 and 1, with 0 indicating the absence of bullying-related content and 1 indicating the presence of bullying-related content. Table 1 and Table 2 display a selection of data exhibiting characteristics of cyberbullying. Table 3 shows the distribution of the classification dataset for cyberbullying detection.

Table 1. Part of the data of the cyberbullying classification.

Lable	Text Content
0	He also caused a kid to drop out of high school and made another girl a scapegoat for all of his problems. I feel for the kid that was bullied and tormented…but that ends the day he brings a gun to school.
1	Listen here you dumb bitch next time you decide to call me a nigger kids be in around us ain’t gonna stop me from beat in the fuck out of you.
2	He talked about gay people, trans people, and rape in a couple jokes. one would think he was being gross…but no.
3	You have destroyed your life. Welcome to the world of Islamic Terrorism.
4	In short we shouldn’t let girls who bullied us in high school go to nursing school.

Table 2. Part of the data of the cyberbullying detection.

Lable	Text Content
1	Bitches love to go Bali with everyone but their bitch????
0	Here myself factor decide dog defense a free program another of peace.
0	Don’t worry about it! Come on when you can and we will always be happy to see you! In the meantime, take care of yourself!
1	Do not cede ground to this bitch motherfucker that knows full well what you meant.

Table 3. The classification distribution of both datasets.

Class	Total	Train	Test
Multi	44,243	35,394	8849
Binary	99,991	79,993	19,998

3.2. Pre-Trained Model Module

The BERT model’s training process comprises two essential phases: pretraining and fine-tuning. The pretraining phase comprises two fundamental tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). MLM randomly masks approximately 15% of the words in the input sequence and trains the model to predict these masked words, encouraging the model to fully consider contextual information and effectively capture the bidirectional semantics of words during training. The NSP aims to comprehend the logical sequence and interconnections among sentences. By providing sentence pairs, the model ascertains whether the second sentence is a continuation of the first, thereby enhancing its understanding of sentence relationships and context. The core objective of the fine-tuning stage is to adapt the pretrained BERT model to specific downstream tasks, typically by freezing most parameters and only updating and optimizing a small number of parameters. The basic architecture of BERT is a multi-layer bidirectional transformer encoder, consisting of 12 stacked encoder units. Each encoder is equipped with 12 self-attention heads, and the feed-forward layer contains 768 hidden units. As shown in Figure 1, the sentence “how could you just walk away from the person that at one time” is split into individual words, and the output representation of each word is formed by adding three vectors: word embeddings, segment embeddings, and position embeddings. Word embeddings convert words into vectors; segment embeddings are used to distinguish sentence pairs; position embeddings compensate for the transformer’s lack of sequence order handling. The hidden state corresponding to the [CLS] token serves as the overall representation of the sentence, while [SEP] is used for padding or separating sentences. The combined representation formed by adding these three embedding vectors incorporates word semantics, contextual information, and positional information within the sentence.

Figure 1. Word vector transformation representation.

3.3. Bidirectional Gated Recurrent Unit Module

BiGRU is an improved structure of Recurrent Neural Networks (RNNs) that incorporates bidirectionality and gating. In the sequence, BiGRU processes information in both forward and backward orientations, addressing the gradient vanishing problem encountered by traditional RNNs in long sequences with the use of gating mechanisms. BiGRU is primarily composed of two gates: the update gate and the reset gate. The update gate regulates the degree to which the current state preserves information from the prior state, whereas the reset gate dictates the impact of the previous state on the computation of the present state. In comparison to LSTM, the GRU architecture is more straightforward and necessitates fewer parameters. Figure 2 shows the network structure of BiGRU, it is composed of an input layer, an output layer, a forward hidden layer, and a backward hidden layer. The input data is simultaneously transmitted to both the forward and backward hidden layers at each time step by the input layer. In other words, the data is transmitted to two GRU networks that are traveling in opposite directions. Both GRUs determine the output sequence of the output layer. Let ${\mathrm{x}}_1,{\mathrm{x}}_2,{\mathrm{x}}_3,\dots ,{\mathrm{x}}_{\mathrm{n}}$ represent the input vector at the specified time step. After being processed by both the forward and backward GRU units, the output ${\mathrm{h}}_1,{\mathrm{h}}_2,{\mathrm{h}}_3,\dots ,{\mathrm{h}}_{\mathrm{n}}$ is obtained by performing a weighted sum of the forward and backward hidden states. Based on a specific time phase $\mathrm{t}$, let ${\mathrm{x}}_{\mathrm{t}}$ represent the input vector at that time $\mathrm{t}$, and the calculation formulas of the BiGRU network are as follows:

$$Z_t=\sigma \left(W_z{x}_t+U_z{h}_{t-1}+b_z\right)$$

(1)

$$R_t=\sigma \left(W_r{x}_t+U_r{h}_{t-1}+b_r\right)$$

(2)

$${\tilde{H}}_t=tanh\left(W_h{x}_t+U_h\left(r_t\circ h_{t-1}\right)+b_h\right)$$

(3)

$$H_t=z_t\circ H_{t-1}+\left(1-z_t\right)\circ {\tilde{H}}_t$$

(4)

Figure 2. BiGRU network structure diagram.

In Equations (1)–(4), ${\mathrm{Z}}_{\mathrm{t}}$ and ${\mathrm{R}}_{\mathrm{t}}$ denote the update gate and the reset gate, respectively; ${\tilde{\mathrm{H}}}_{\mathrm{t}}$ represent the candidate hidden state; ${\mathrm{H}}_{\mathrm{t}}$ and ${\mathrm{H}}_{\mathrm{t}-1}$, respectively, represent the hidden states at time $\mathrm{t}$ and $\mathrm{t}-1$; $\mathrm{W}$ and $\mathrm{U}$ denote the weights and $\mathrm{t}$ denotes bias; $\sigma$ is the represent activation function; and ∘ denotes element-wise multiplication. BiGRU’s output, which integrates the hidden states from the forward and backward directions, is usually expressed as

$${\overrightarrow{H}}_t=GRU\left(x_t,{\overrightarrow{H}}_{t-1}\right)$$

(5)

$${\overleftarrow{H}}_t=GRU\left(x_t,{\overleftarrow{H}}_{t-1}\right)$$

(6)

$$H_t=f\left(W_{{\overrightarrow{H}}_t}{\overrightarrow{H}}_t+W_{{\overleftarrow{H}}_t}{\overleftarrow{H}}_t+b_t\right)$$

(7)

In Equations (5)–(7), ${\overrightarrow{\mathrm{H}}}_{\mathrm{t}}$ and ${\overleftarrow{\mathrm{H}}}_{\mathrm{t}}$ are the forward and backward hidden states, respectively, at time $\mathrm{t}$; ${\mathrm{W}}_{{\overleftarrow{\mathrm{H}}}_{\mathrm{t}}}$ and ${\mathrm{W}}_{{\overleftarrow{\mathrm{H}}}_{\mathrm{t}}}$ are the weights for the forward and backward hidden states, respectively, at time $\mathrm{t}$; and ${\mathrm{b}}_{\mathrm{t}}$ is the bias for the hidden state at time $\mathrm{t}$.

3.4. Convolutional Neural Network Module

The BBGC model’s CNN layer comprises convolutional layers, pooling layers, and fully connected layers. The convolutional layer executes convolution operations to extract local features and produce feature maps, employing nonlinear activation functions (such as ReLU) to incorporate nonlinearity into the model. The pooling layer reduces the dimensionality of the feature maps through the max pooling operation, and the output is subsequently flattened prior to input into the fully connected layer. The output is ultimately converted into a probability distribution across categories through the application of the Softmax activation function, resulting in the final classification outcome.

For the input feature map $\mathrm{I}$ and the convolution kernel $\mathrm{k}$, the convolution operation and the max pooling operation can be expressed as

$$O\left(i,j\right)=\sum _m\sum _{n}I\left(i+m,j+n\right)·k\left(m,n\right)$$

(8)

$$O\left(i,j\right)=\underset{m,n}{max}I\left(i·S+m,j·S+n\right)$$

(9)

In Equation (8), $\mathrm{O}\left(\mathrm{i},\mathrm{j}\right)$ is the value of the output feature map at position $\left(\mathrm{i},\mathrm{j}\right)$ and $\mathrm{m}$, $\mathrm{n}$ are the dimensions of the convolution kernel. In Equation (9), $\mathrm{O}\left(\mathrm{i},\mathrm{j}\right)$ is the value of the output feature map at position $\left(\mathrm{i},\mathrm{j}\right)$, $\mathrm{S}$ is the stride and $\mathrm{m}$, $\mathrm{n}$ are the indices within the pooling window.

3.5. Model Architecture

As shown in Figure 3, the proposed model BBGC consists of the following core components connected sequentially: a BERT encoding layer, a BiGRU contextual encoding layer, and a CNN local feature extraction layer. The detailed designs of each component and the feature fusion process will be elaborated below:

Figure 3. BERT-BiGRU-CNN model structure diagram.

(1) The BERT layer in the diagram accepts text input ${\mathrm{W}}_1,{\mathrm{W}}_2,{\mathrm{W}}_3,{\mathrm{W}}_4,\dots ,{\mathrm{W}}_{\mathrm{n}}$, which is formatted as [CLS] + tokens + [SEP], and generates the corresponding Token ID, Segment ID, and Position ID tensors. These tensors are fed into the BERT model. We extract the hidden states from all token positions in the final layer of BERT as output, denoted as $\mathrm{H}\_\mathrm{bert}\in {\mathbb{R}}^{B\times L\times D\_\mathrm{bert}}$, where B is the batch size, L is the sequence length, and $D_{\_\mathrm{bert}}$ is the hidden dimension of BERT (e.g., 768). This tensor contains rich contextual semantic information.

(2) Subsequently, $\mathrm{H}\_\mathrm{bert}$ is fed into a bidirectional GRU layer to further capture bidirectional long-term dependencies in the sequence. Before being input to the bidirectional GRU network, the output features from the BERT layer retain long-range dependencies in the text. The bidirectional GRU process sequences information through its forward and backward modules, enabling the capture of contextual information from both previous and subsequent time steps at each position. This layer outputs the concatenation of the forward and backward hidden states at each time step, resulting in a context-enhanced sequence representation $\mathrm{H}\_\mathrm{bigru}\in {\mathbb{R}}^{B\times L\times D_{\_\mathrm{bigru}}}$, where ${\mathrm{D}}_{\_\mathrm{bigru}}$ is twice the hidden dimension of the BiGRU. The forward and backward outputs of the bidirectional GRU are then concatenated and passed to the CNN layer.

(3) The BERT-BiGRU layer passes the output sequence to the CNN layer, which employs multiple convolutional kernels to extract local features, treating $\mathrm{H}\_\mathrm{bigru}$ as a one-dimensional sequence with $\mathrm{D}\_\mathrm{bigru}$ channels. At this stage, three parallel one-dimensional convolutional layers with different kernel widths are used to capture local phrase patterns at varying granularities. The output of each convolutional layer is processed through a ReLU activation function and a 1D max-pooling layer, then concatenated and flattened into a fixed-length feature vector $\mathrm{v}\_\mathrm{cnn}\in {\mathbb{R}}^{D\_\mathrm{cnn}}$.

(4) During max pooling, the output sequence ${\mathrm{H}}_{\_\mathrm{bigru}}$ of the BiGRU is pooled to obtain a vector ${\mathrm{v}}_{\_\mathrm{bigru}\_\mathrm{pool}}\in {\mathbb{R}}^{D_{\_\mathrm{bigru}}}$, which summarizes the contextual information of the entire sequence. The output of the pooling layer is leveled and fed into the fully connected layer. The input and weight combination that is linear in nature is the primary assignment of the fully linked layer. The fully connected layer performs nonlinear transformation of the linear combination through activation functions. First, the input vector is multiplied by the weight matrix with a bias term added, expressed as $\mathrm{z}=\mathrm{Wx}+\mathrm{b}$. The linear output is then transformed nonlinearly via an activation function: $\mathrm{a}=\mathrm{F}\left(\mathrm{z}\right)$. The activated output is passed as input to the subsequent layer. Feature fusion involves concatenating the BiGRU pooled vector $\mathrm{v}\_\mathrm{bigru}\_\mathrm{pool}$ with the locally extracted feature vector $\mathrm{v}\_\mathrm{cnn}$ from the CNN, forming the final fused feature vector $\mathrm{v}\_\mathrm{fused}=\mathrm{concat}(\mathrm{v}\_\mathrm{bigru}\_\mathrm{pool},\mathrm{v}\_\mathrm{cnn})\in {\mathbb{R}}^{(D\_\mathrm{bigru}+D_{\_\mathrm{cnn}})}$, which integrates the contextual semantics from BERT and BiGRU with the local pattern features from the CNN. Finally, the fused feature vector $\mathrm{v}\_\mathrm{fused}$ is passed through a fully connected layer with Dropout regularization, followed by a Softmax function to output the probability distribution across all classes. After processing through all fully connected layers, the network generates predicted probabilities for each class, completing the entire pipeline from feature extraction to classification decision.

Assuming the input vector is $\mathrm{x}$, the weight matrix is $\mathrm{W}$, and the bias is $\mathrm{b}$, the output vector $\mathrm{y}$ can be expressed as

$$Y=W·x+b$$

(10)

$$F\left(z\right)={\displaystyle \frac{e^z-e^{-z}}{e^z+e^{-z}}}$$

(11)

$$\sigma \left(z_i\right)={\displaystyle \frac{e^{z_i}}{{\sum }_j{e}^{z_i}}}$$

(12)

In Equation (10), the weight matrix $\mathrm{W}$ has dimension $\left({\mathrm{n}}_{\mathrm{out}},{\mathrm{n}}_{\mathrm{in}}\right)$, where ${\mathrm{n}}_{\mathrm{out}}$ is the quantity of output units and ${\mathrm{n}}_{\mathrm{in}}$ is the number of input units, while the bias vector $\mathrm{b}$ has a dimension of ${\mathrm{n}}_{\mathrm{out}}$. The output of the fully connected layer generates the final classification result through a Softmax classifier. Equation (11) represents the activation function. The Softmax function converts a vector into a probability distribution, as shown in Equation (12). Here, $\mathrm{z}$ is the input vector, ${\mathrm{z}}_{\mathrm{i}}$ is the number $\mathrm{i}$ element of the input vector, $\mathrm{e}$ is the natural exponential, and the denominator is the sum of the exponentials of all input elements. The output values of Softmax $\sigma \left({\mathrm{z}}_{\mathrm{i}}\right)$ range from 0 to 1, and the sum of all output values equals 1.

4. Experimental Comparison and Results Analysis

4.1. Evaluation Indicators

The evaluation index is an important part of the evaluation data and model. Accuracy, Precision, Recall, and F1 (F1-Score) were used as evaluation indicators, with F1 (F1-Score) being the main evaluation index. For the multi-class classification task, we report the macro-averaged Precision, Recall, and F1-Score to equally evaluate the model’s performance across each class. The calculation formulas are as follows: Formulas (13)–(16).

$$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(13)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(14)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(15)

$$F1\text{-}score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}={\displaystyle \frac{2\times TP}{2\times TP+FP+TN}}$$

(16)

The various parameters used to calculate the above metrics are as follows:

True Positive (TP): The number of instances where the model predicts the positive class and the actual label is also positive.

False Positive (FP): The number of instances where the model predicts the positive class but the actual label is negative.

True Negative (TN): The number of instances where the model predicts the negative class and the actual label is also negative.

False Negative (FN): The number of instances where the model predicts the negative class but the actual label is positive.

4.2. Comparison and Analysis of Pre-Trained Models

Pre-trained models such BERT, RoBERTa, ALBERT, and DistilBERT excel in managing diverse semantics and extracting linguistic features during text processing. In contrast to BERT, RoBERTa employs dynamic masking instead of static masking, enabling it to more effectively capture contextual information. ALBERT diminishes the parameter count by implementing parameter sharing across various layers and employing factorized embedding methodologies. DistilBERT, a streamlined variant of BERT, employs knowledge distillation and typically comprises 60% of BERT’s architectural layers, thereby substantially diminishing computational resource requirements during training.

This study conducted evaluation experiments on the task of cyberbullying text detection in order to more accurately assess these pre-trained models. Table 4 and Table 5 display the comparison data for binary classification and multi-class classification, respectively. The F1-Scores of 0.8306 and 0.9481 for multi-class and binary classification, respectively, demonstrate that ALBERT, which reduces the number of parameters, does not perform well, while DistilBERT, which simplifies the architecture layers, achieves the best performance. In comparison to the original BERT, RoBERTa has a significant advantage, with F1-Scores of 0.8177 and 0.9361 for multi-class and binary classification, respectively. These scores are 0.0129 and 0.0120 lower than those of DistilBERT. It is clear from the comparison that each pre-trained model possesses its own unique characteristics when it comes to detecting cyberbullying text, and the original BERT does not possess an advantage. RoBERTa and DistilBERT, on the other hand, exhibit superior performance.

Table 4. Comparison of different pre-trained models in cyberbullying multi-classifications.

Model	Accuracy	Precision	Recall	F1-Score
BERT	0.7876	0.7895	0.7895	0.7820
RoBERTa	0.8130	0.8100	0.8097	0.8177
ALBERT	0.6239	0.6246	0.6219	0.6047
DistilBERT	0.8453	0.8425	0.8334	0.8306

The bolded fonts merely serve as highlighting, indicating that this method has the best overall performance in this table. The same as follow.

Table 5. Comparison of different pre-trained models in cyberbullying binary classification.

Model	Accuracy	Precision	Recall	F1-Score
BERT	0.8232	0.8222	0.8380	0.8194
RoBERTa	0.9366	0.9369	0.9365	0.9361
ALBERT	0.8240	0.8239	0.8242	0.8228
DistilBERT	0.9485	0.9490	0.9482	0.9481

Both BiGRU and BiLSTM are bidirectional RNN models that are capable of transmitting information both forward and backward. They not only solve the vanishing gradient problem of traditional RNNs and effectively capture long-term dependencies, but they also improve the model’s generalization ability and sequential modeling capacity in classification tasks. A comparison experiment was conducted, as shown in Table 4 and Table 5, to compare the single and trained models. This section will further integrate BiGRU and BiLSTM based on the aforementioned results and assess the fusion effect of the hybrid neural network model in the multi-class and binary classification of cyberbullying. The experimental data are presented in Table 6 and Table 7. Compared to the single pre-trained models, the hybrid neural network models exhibited improved classification performance, with more consistent results, all falling within the range of 0.92 to 0.94. The most substantial performance improvement was demonstrated by ALBERT when it was integrated with BiGRU and BiLSTM. Despite the fact that its classification performance remained at its lowest point, there was a discernible enhancement in comparison to the single model. Furthermore, it is particularly important to note that the fusion effect of the original BERT model was more pronounced. The most favorable outcomes were achieved by both BERT-BiGRU and BERT-BiLSTM. This suggests that the hybrid neural network model, which is based on the original BERT, demonstrates superior overall performance in the detection of cyberbullying text, despite the fact that RoBERTa, ALBERT, and DistilBERT are enhanced versions of BERT.

Table 6. Comparison of different pre-trained models based on BiGRU and BiLSTM in cyberbullying multi-classifications.

Model	Accuracy	Precision	Recall	F1-Score
RoBERTa–BiLSTM	0.9326	0.9319	0.9305	0.9303
ALBERT–BiLSTM	0.9226	0.9221	0.9226	0.9211
DistilBERT–BiLSTM	0.9323	0.9312	0.9315	0.9295
BERT–BiLSTM	0.9370	0.9375	0.9363	0.9356
RoBERTa–BiGRU	0.9352	0.9349	0.9344	0.9342
ALBERT–BiGRU	0.9312	0.9313	0.9304	0.9329
DistilBERT–BiGRU	0.9356	0.9356	0.9355	0.9354
BERT–BiGRU	0.9397	0.9399	0.9399	0.9387

Table 7. Comparison of different pre-trained models based on BiGRU and BiLSTM in cyberbullying binary classification.

Model	Accuracy	Precision	Recall	F1-Score
RoBERTa–BiLSTM	0.9740	0.9740	0.9741	0.9739
ALBERT–BiLSTM	0.9719	0.9719	0.9718	0.9718
DistilBERT–BiLSTM	0.9733	0.9733	0.9731	0.9732
BERT–BiLSTM	0.9828	0.9828	0.9828	0.9827
RoBERTa–BiGRU	0.9832	0.9832	0.9831	0.9831
ALBERT–BiGRU	0.9720	0.9718	0.9721	0.9719
DistilBERT–BiGRU	0.9827	0.9830	0.9824	0.9826
BERT–BiGRU	0.9845	0.9844	0.9845	0.9844

Subsequent to evaluating the fusion effects of pre-trained models, the next phase involves examining the performance disparities between BiGRU and BiLSTM. A comparison was conducted for multi-class classification of cyberbullying (Table 6) and binary classification (Table 7). The combination of pre-trained BERT, RoBERTa, ALBERT, and DistilBERT with BiGRU surpassed the combination with BiLSTM regarding Accuracy, Precision, Recall, and F1-Score. This suggests that the integration of pre-trained models with BiGRU offers a superior advantage in the domain of cyberbullying detection.

4.3. Comparison and Analysis of Hybrid Neural Network Models

Upon integrating BiGRU or BiLSTM, which possess superior sequence modeling capabilities, into the BERT model, its performance exceeds that of alternative pre-trained models, with the BiGRU-based model outperforming the BiLSTM variant. To investigate the impact of hybrid neural networks on cyberbullying speech classification, Convolutional Neural Networks (CNNs) capable of extracting local feature information are incorporated into the aforementioned models to develop a hybrid neural network model based on pre-trained architectures. The experimental findings presented in Table 8 and Table 9 elucidate this clearly. The model exhibits enhancements in metrics such as Accuracy, Precision, Recall, and F1-Score when the CNN is further integrated with BERT-BiGRU and BERT-BiLSTM. This suggests that the model’s capacity to identify local patterns is significantly improved by the inclusion of a CNN, thereby enhancing its overall performance. It is important to mention that the fusion with BiGRU continues to provide a significant advantage in the majority of cases; however, ALBERT-BiGRU-CNN exhibits slightly inferior performance compared to ALBERT-BiLSTM-CNN.

Table 8. Comparison of hybrid neural networks based on BiGRU and BiLSTM in cyberbullying multi-classifications.

Model	Accuracy	Precision	Recall	F1-Score
RoBERTa–BiLSTM–CNN	0.9398	0.9396	0.9393	0.9381
ALBERT–BiLSTM–CNN	0.9365	0.9355	0.9373	0.9360
DistilBERT–BiLSTM–CNN	0.9382	0.9386	0.9390	0.9378
BERT–BiLSTM–CNN	0.9416	0.9417	0.9416	0.9403
RoBERTa–BiGRU–CNN	0.9400	0.9407	0.9390	0.9393
ALBERT–BiGRU–CNN	0.9354	0.9327	0.9322	0.9313
DistilBERT–BiGRU–CNN	0.9402	0.9403	0.9395	0.9388
BERT–BiGRU–CNN	0.9444	0.9446	0.9438	0.9433

Table 9. Comparison of hybrid neural networks based on BiGRU and BiLSTM in cyberbullying binary classification.

Model	Accuracy	Precision	Recall	F1-Score
RoBERTa–BiLSTM–CNN	0.9902	0.9902	0.9900	0.9900
ALBERT–BiLSTM–CNN	0.9910	0.9911	0.9909	0.9909
DistilBERT–BiLSTM–CNN	0.9912	0.9912	0.9911	0.9911
BERT–BiLSTM–CNN	0.9947	0.9947	0.9946	0.9946
RoBERTa–BiGRU–CNN	0.9930	0.9930	0.9929	0.9928
ALBERT–BiGRU–CNN	0.9877	0.9875	0.9877	0.9875
DistilBERT–BiGRU–CNN	0.9937	0.9937	0.9936	0.9936
BERT–BiGRU–CNN	0.9955	0.9955	0.9954	0.9956

The Attention mechanism also possesses sequence modeling capabilities. The primary objective of this section’s experiment is to evaluate the performance disparities between BiGRU, BiLSTM, and Attention in the detection of cyberbullying. Previously, it was determined that the original BERT offers more benefits when incorporated with neural network models through a comparison of the data in Table 6, Table 7, Table 8 and Table 9. Consequently, in order to facilitate a lucid comparison, only the experiments on BERT-BiGRU-CNN, BERT-BiLSTM-CNN, and BERT-Attention-CNN were conducted. It is evident from Table 10 and Table 11 that the Attention mechanism does not possess an advantage in the detection of cyberbullying, as it scores lower than BiGRU and BiLSTM in both multi-class and binary classification tasks.

Table 10. Hybrid neural networks based on BiGRU, BiLSTM and Attention in cyberbullying multi-classifications comparison.

Model	Accuracy	Precision	Recall	F1-Score
BERT–BiLSTM–CNN	0.9416	0.9417	0.9416	0.9403
BERT–Attention–CNN	0.9348	0.9340	0.9341	0.9327
BERT–BiGRU–CNN	0.9444	0.9446	0.9438	0.9433

Table 11. Hybrid neural networks based on BiGRU, BiLSTM and Attention in cyberbullying binary classification comparison.

Model	Accuracy	Precision	Recall	F1-Score
BERT–BiLSTM–CNN	0.9947	0.9947	0.9946	0.9946
BERT–Attention–CNN	0.9925	0.9926	0.9923	0.9924
BERT–BiGRU–CNN	0.9955	0.9955	0.9954	0.9956

To enhance the analysis of the classification efficacy of BiGRU, BiLSTM, and Attention within each subclass, the confusion matrix is employed for a more comprehensive analysis. Figure 4 illustrates the comparison of the multi-class confusion matrix for the cyberbullying detection models presented in Table 10. BERT-BiLSTM-CNN exhibits marginally superior correct predictions for class 4 (1486) relative to the BBGC model (1466); however, the overall correct predictions total 6861, which is inferior to BBGC’s 6891. BERT-Attention-CNN also exhibits a greater number of accurate predictions for class 4 (1483) in comparison to BBGC; its overall correct predictions (6823) remain inferior to those of BBGC. Figure 5 illustrates the comparison of the binary classification confusion matrix for cyberbullying detection, as shown in Table 11. The findings indicate that BBGC achieved a total of 12,870 correct predictions, whereas BERT-BiLSTM-CNN and BERT-Attention-CNN recorded 12,860 and 12,832 correct predictions, respectively. Despite BERT-BiLSTM-CNN achieving a greater quantity of accurate predictions for label 1, its predictive accuracy for label 0 is inferior, resulting in an overall performance that is not as favorable as BBGC. The BBGC model exhibits superior discrimination capability in cyberbullying detection tasks.

Figure 4. Multi-class classification confusion matrices of the models.

Figure 5. Binary classification confusion matrices of the models.

To verify the superior performance of BERT in hybrid models, it is further compared with mainstream word embedding models that generate static word vectors, such as GloVe, Word2Vec, and FastText. Given that BiGRU-CNN has exhibited outstanding performance in the experiments detailed in Table 8, Table 9, Table 10 and Table 11, the experimental procedure is streamlined by amalgamating BiGRU-CNN with four pre-trained models—BERT, GloVe, Word2Vec, and FastText—for additional evaluation.

The performances of Glove-BiGRU-CNN and W2V-BiGRU-CNN are approximately equivalent, as both are inferior to that of FastText-BiGRU-CNN and BERT-BiGRU-CNN, as indicated by the experimental data presented in Table 12 and Table 13. This may be due to the fact that Word2Vec concentrates on local context information, whereas GloVe models solely rely on global co-occurrence information. This simpler modeling approach has specific limitations. FastText enhances its capacity to handle out-of-vocabulary words by integrating subword (n-gram) information with Word2Vec. In terms of cyberbullying detection performance, the Accuracy, Precision, Recall, and F1-Scores of FastText-BiGRU-CNN are higher than those of the Glove and W2V hybrid models. However, there is still a significant gap when compared to BERT. In both binary and multi-class classification experiments, BERT-BiGRU-CNN yields the most favorable outcomes. All metrics in the binary classification task exceed 0.99, while the Accuracy, Precision, Recall, and F1-Scores all exceed 0.94 in the multi-class task.

Table 12. Comparison of hybrid neural networks based on different word embedding models in multi-classifications of cyberbullying.

Model	Accuracy	Precision	Recall	F1-Score
Glove– BiGRU–CNN	0.9044	0.9043	0.9027	0.9017
W2V–BiGRU–CNN	0.9094	0.9091	0.9081	0.9068
FastText–BiGRU–CNN	0.9245	0.9244	0.9218	0.9223
BERT–BiGRU–CNN	0.9444	0.9446	0.9438	0.9433

Table 13. Comparison of hybrid neural networks based on different word embedding models in binary classification of cyberbullying.

Model	Accuracy	Precision	Recall	F1-Score
Glove–BiGRU–CNN	0.9923	0.9922	0.9922	0.9921
W2V–BiGRU–CNN	0.9879	0.9877	0.9878	0.9877
FastText–BiGRU–CNN	0.9935	0.9934	0.9933	0.9933
BERT–BiGRU–CNN	0.9955	0.9955	0.9954	0.9956

4.4. Hyperparameter Comparison and Analysis of BBGC Model

Hyperparameter design has a significant impact on model performance. The specific values of the experimental parameter design are shown in Table 14. To assess the efficacy of the BERT-BiGRU-CNN model, hyperparameter optimization for multi-class cyberbullying detection was performed, incorporating various configurations of hyperparameters, including Hidden_sizes and Kernel_sizes. Table 15 illustrates the variation in Hidden_size while maintaining other hyperparameters constant. When the Hidden_size is minimal, the model may inadequately capture the intricate features of the input data, resulting in underfitting. Conversely, a large Hidden_size enables the model to learn additional features and more intricate patterns, yet it may also result in overfitting. Consequently, choosing an optimal Hidden_size can markedly enhance the model’s efficacy. In this series of experiments, a Hidden_size of 256 yielded the most favorable outcomes. Table 16 similarly demonstrates the variation in Kernel_size while maintaining other parameters constant. Analogous to the modifications in Hidden_size, diminutive convolution kernels can solely encapsulate highly localized features, whereas larger convolution kernels may result in heightened model complexity, consequently elevating the likelihood of overfitting. The experiment necessitates the utilization of convolution kernels of varying sizes to more effectively capture multi-scale feature information. In this series of experiments, kernel sizes of [5, 6, 7] attained optimal performance.

Table 14. Experimental parameter design.

Hyperparameter	Value	Hyperparameter	Value
Hidden_size	256	Optimizer	Adam
Dropout	0.2	Max_length	300
Learning_rate	0.001	Batch_size	128
Epochs	30	Kernel_sizes	[5, 6, 7]

Table 15. Comparison of different Hidden_sizes in the BBGC model for multi-class cyberbullying classification.

Hidden Sizes	Kernel Sizes	Dropout	Num Layers	Accuracy	Precision	Recall	F1-Score
128	[5, 6, 7]	0.2	2	0.9376	0.9372	0.9381	0.9363
256	[5, 6, 7]	0.2	2	0.9444	0.9446	0.9438	0.9433
512	[5, 6, 7]	0.2	2	0.9407	0.9423	0.9405	0.9397
768	[5, 6, 7]	0.2	2	0.9400	0.9417	0.9390	0.9393

Table 16. Comparison of different Kernel_sizes in the BBGC model for multi-class cyberbullying classification.

Hidden Sizes	Kernel Sizes	Dropout	Num Layers	Accuracy	Precision	Recall	F1-Score
256	[3, 4, 5]	0.2	2	0.9386	0.9383	0.9366	0.9363
256	[4, 5, 6]	0.2	2	0.9372	0.9374	0.9377	0.9355
256	[5, 6, 7]	0.2	2	0.9444	0.9446	0.9438	0.9433
256	[6, 7, 8]	0.2	2	0.9393	0.9402	0.9383	0.9379
256	[7, 8, 9]	0.2	2	0.9387	0.9364	0.9365	0.9351

4.5. Ablation Experiment

Ablation studies are conducted to validate the design rationality of each module in the model and determine whether every component contributes positively to the final performance. These studies enhance model transparency and interpretability, thereby improving computational efficiency and reducing resource consumption. The ablation experiment design is as follows:

• Remove the BERT and CNN components, and use randomly initialized embedding layers to represent text inputs. Independently compare the BiLSTM and BiGRU models to verify which model is more suitable for long texts and better captures long-range dependencies.
• Remove the CNN layer and compare the performance of pre-trained models combined with BiLSTM and BiGRU neural networks to examine whether the pre-trained models enhance BiLSTM and BiGRU in the absence of local feature extraction by the CNN.
• Replace BiLSTM and BiGRU separately in the hybrid neural network model to evaluate which neural network is more effective for cyberbullying text classification.

From Table 17, it can be concluded that the overall performance of the BiLSTM model is higher than that of BiGRU, indicating that BiLSTM has better capability than BiGRU in capturing long-range dependencies in long texts. To examine the impact of incorporating the BERT pre-trained model on BiLSTM and BiGRU after removing the CNN layer from the hybrid neural network model, BERT-BiGRU demonstrated an overall performance surpassing that of BERT-BiLSTM. To further validate the performance of BiLSTM and BiGRU, CNN layers were added to both models. The results showed that, with the addition of the CNN, the BiGRU model outperformed the BiLSTM model in all aspects, indicating that the inclusion of the CNN indeed enhances the model’s ability to capture local features. Although the standalone BiLSTM model exhibits higher overall performance than BiGRU, suggesting its significant advantage in capturing long-range dependencies, its individual performance remains insufficiently satisfactory and is lower than that of the hybrid neural network incorporating pre-trained models.

Table 17. The ablation experiment of multi-classification cyberbullying.

Model	Accuracy	Precision	Recall	F1-Score
BiLSTM	0.9288	0.9275	0.9272	0.9262
BiGRU	0.9115	0.9106	0.9097	0.9087
BERT–BiLSTM	0.9370	0.9375	0.9363	0.9356
BERT–BiGRU	0.9397	0.9399	0.9399	0.9387
BERT–BiLSTM–CNN	0.9416	0.9417	0.9416	0.9403
BERT–BiGRU–CNN	0.9444	0.9446	0.9438	0.9433

With the integration of BiGRU and CNN with pre-trained models, the overall performance of the BBGC model surpasses that of BBLC. This is attributed to the more concise parameters and structure of GRU compared to LSTM. When handling complex tasks or long sequences—particularly in the context of highly diverse and voluminous cyberbullying text data—reducing computational complexity may contribute to enhancing the model’s overall performance.

5. Discussion

To validate the superior efficacy of the fusion model BERT-BiGRU-CNN in cyberbullying detection tasks, we conducted a comparative analysis with contemporary mainstream fusion models in both multi-class and binary classification experiments. On comparable tasks, our method achieves results competitive with state-of-the-art approaches. Table 18 and Table 19 present the performance reported in previous studies under their respective experimental setups alongside our method’s results. Due to differences in datasets, preprocessing, and data splits, these figures should not be compared directly or quantitatively; they are provided only as contextual reference. The advantages of our method are further validated through ablation studies and controlled comparative experiments. In the experiments presented in Table 18, we compare it with FAEO-ECNN [39] and BERT-SVM [40]. The FAEO-ECNN model, introduced by Murshed et al., is a five-class classification experiment on RW-CB-Twitter cyberbullying texts, attaining Accuracy, Recall, and F1-Scores of 0.9189, 0.9132, and 0.9156, respectively. The BERT-SVM model, developed by Aggarwal et al., is a hybrid model that amalgamates BERT, SVM, and grid search methodologies, incorporating SHAP technology to improve the interpretability of its predictions. The BERT-SVM model exhibits elevated computational complexity; however, its experimental outcomes are suboptimal, yielding Accuracy, Recall, and F1-Score values of 0.9000, 0.9000, 0.9020, and 0.9000, respectively.

Table 18. Comparison of the proposed model with existing methods in cyberbullying multi-classifications.

Method	Class	Accuracy	Precision	Recall	F1-Score
FAEO-ECNN [39]	5	0.9189	0.9228	0.9132	0.9156
BERT-SVM [40]	5	0.9000	0.9000 (avg)	0.9020 (avg)	0.9000 (avg)
Ours	5	0.9444	0.9446	0.9438	0.9433

Table 19. Comparison of the proposed model with existing methods in binary classification of cyberbullying detection.

Method	Accuracy	Precision	Recall	F1-Score
Keras+LSTM [26]	0.9367	-	0.9362	0.9364
CNN-BiLSTM-GRU [29]	0.9883	0.9815	0.9955	0.9813
RoBERTa+GLOVE+PCA [31]	0.9800	0.9700	0.9800	0.9700
BERT+ensemble stacking learning [41]	0.9740	0.9500	0.9200	0.9640
GNN [42]	0.9278	0.9651	0.8015	0.8757
BERT+SBiGRU-BCO [43]	0.9912	0.9473	0.9745	0.9391
Ours	0.9955	0.9954	0.9954	0.9956

In the binary classification task of cyberbullying detection, Obaid et al. [26] introduced a multi-layer model architecture for the purpose. The text, once embedded with Keras, is fed into a four-layer LSTM for classification, subsequently employing fuzzy logic to assess the severity of various forms of bullying. Daraghmi et al. [29] combined CNN, BiLSTM, and GRU neural networks to create a hybrid CNN-BiLSTM-GRU model for the detection of Arabic cyberbullying. Umer et al. [35] employed the RoBERTa model, integrating global word vector (GLOVE) embeddings derived via Principal Component Analysis (PCA), and subsequently incorporated the LIME module for the detection of cyberbullying. Muneer et al. [41] integrated the BERT model with an ensemble stacking learning framework, whereas Nurfiqri et al. [42] utilized Graph Neural Networks for the detection of cyberbullying. Mali et al. [43] employed the BERT model as the foundational classifier and executed SBiGRU-BCO methodology for identifying cyberbullying behavior. The data in Table 19 of this experiment indicates that BERT-BiGRU-CNN excels in the binary classification task of cyberbullying detection.

6. Conclusions

Cyberbullying text encompasses intricate semantics and linguistic attributes, with terminology associated with bullying frequently displaying sophisticated language and semantic qualities. A solitary neural network model may find it challenging to adequately encapsulate these intricate semantics and linguistic characteristics. The processing capacity of a single model is somewhat constrained regarding syntactic structure and long-sequence dependencies. Pretrained models, utilizing extensive text data pretraining and techniques such as bidirectional modeling, masked language models, positional encoding, and multi-layer transformers, exhibit robust proficiency in revealing nuanced semantic meanings, dissecting intricate syntactic structures, and managing long-sequence dependencies. Consequently, our research integrates pre-trained models with other neural networks. Through iterative optimization of model architecture and parameters, this approach effectively compensates for the limitations inherent in both standalone pre-trained models and individual neural network models.

Drawing on the analysis above, this article presents the BERT-BiGRU-CNN multi-feature fusion hybrid neural network model for cyberbullying text categorization. To obtain word vectors rich in semantic information, the text is first entered into the BERT model for preprocessing. The BiGRU layer then receives the text vectors to learn sentence sequence characteristics. The CNN convolutional layer pulls local features after creating the feature sequence. The output features of BiGRU and the CNN are finally combined and when they are input into a fully connected layer and pass through the Softmax activation function, the final prediction result is produced. The BERT-BiGRU-CNN model exhibits the highest performance as a result of two primary factors, as evidenced by comparative experiments. One advantage of this approach is its ability to effectively capture the subtle syntactic features and rich semantic nuances of language by learning from large-scale text data, which is facilitated by advanced pretraining and fine-tuning techniques. Conversely, the BiGRU-enhanced model possesses a robust capacity to comprehend sequential data, whereas the CNN network optimizes the model’s overall performance by recognizing local patterns, thereby offering a substantial advantage when managing intricate data.

We acknowledge that cyberbullying detection models may harbor inherent biases. For instance, the model might generate false positives for non-malignant slang used within specific subcultural groups. To ensure fairness, future work will incorporate bias auditing of model predictions to identify and mitigate performance disparities based on demographic factors. The dataset used in this study has undergone rigorous anonymization, with all personally identifiable information removed. In any future practical deployment, we would also adhere to privacy-by-design principles, collecting and processing only the data strictly necessary for detection.

Certainly, this research is not without limitations. Although the model has achieved commendable performance in English text, it has not yet been extended to multilingual scenarios for detecting abusive content and emerging online terminology—an area still under exploration. This limitation arises from three aspects: the model relies on English pre-trained resources (often lacking for other languages); its architecture is optimized for English syntax (e.g., SVO word order and space-based tokenization), making BiGRU less adaptable to dependency relations in SOV/VSO languages and CNNs challenged by non-spaced languages (e.g., Chinese); and cyberbullying expressions are culturally specific, leading to performance degradation if directly transferred. These challenges define our future research direction. Therefore, we will not only further investigate the capacity of the BBGC model to handle nuanced expressions prevalent in cyberbullying (e.g., sarcasm, abbreviations, and emojis), but also explore in greater depth how pre-trained models such as RoBERTa, ALBERT, and DistilBERT can be optimized to enhance their performance across diverse textual tasks. Among adolescents, cyberbullying is a serious social problem that can lead to psychological trauma, depression, and even suicide. Thus, the capacity to automatically identify and categorize harmful texts can aid in prompt intervention, safeguarding victims’ safety and mental well-being.