Research on Enhanced Chinese Text Classification Through Feature Confusion and Hierarchical Perception

Abstract

Conventional text classification models experience inadequate contextual information extraction, restricted classification Accuracy, and significant text information loss. This article presents an improved Chinese text classification model that incorporates feature obfuscation and hierarchical perception. A text sequence channel is established, employing the RoFormer model to increase the positional attributes of word embeddings. Subsequently, multi-scale window convolutions acquire multidimensional feature information within the sequence. An enhanced confusion technique ultimately improves the model’s resilience to interference and finalizes feature confusion fusion. Evaluation on the Waimai, ChnSentiCorp, and COLD datasets indicates that this model attains enhanced accuracy and F1 scores relative to previous benchmark models, confirming its efficacy and relevance.

1. Introduction

Text classification is a core task in natural language processing, aiming to assign documents to predefined categories based on their content. Due to the diversity of text structures and the complexity of content, traditional machine learning methods can no longer meet classification demands. In contrast, deep learning methods can automatically derive higher-level and more abstract semantic representations through multi-layered semantic operations. Their strong generalization capabilities enable broader applications across domains such as spam detection [1], sentiment analysis [2], public opinion analysis [3], topic classification [4], and news categorization [5].

With the advancement of neural networks, text classification has achieved new breakthroughs. Most classification models based on the CNN architecture [6] train their models to learn local information within windows. They then extract features by sliding these local windows to perform category classification. However, CNN models also reveal certain limitations, as they neglect the dependency between local features. Therefore, this paper proposes a fusion model that integrates hierarchical perception and feature mixing (TCTM) from this perspective. The research content is as follows:

(1)

The pre-trained RoFormer model is employed to perform word embedding on text data, representing textual information as a dense matrix in a low-dimensional space to extract high-level semantic features.

(2)

An optimized multi-scale convolutional neural network is utilized to enhance the relational representation of character information, thereby obtaining richer global contextual features.

(3)

The captured global contextual features are combined with the initial RoFormer sentence features through an optimized blending method for text classification.

2. Related Work

Text classification, as one of the core tasks in natural language processing, aims to automatically assign textual documents to predefined categories. Its research paradigm has undergone a significant evolution from traditional machine learning to deep learning alongside advancements in artificial intelligence technology, achieving breakthrough leaps in model capabilities and feature representation.

The primary methods for extracting textual features through machine learning include k-nearest neighbors, support vector machines, and random forests. Jones et al. [7] utilized the term frequency–inverse document frequency (TF-IDF) index to vectorize textual data, subsequently combining logistic regression and support vector machines for modeling. Ruan et al. [8] proposed an approach to enhance the Naive Bayes classifier by refining the distance correlation coefficient. Unlike traditional statistical measures based on the cumulative distribution function of random vectors, this method evaluates joint random vector correlation by describing the distance between the product of joint feature functions and marginal feature functions. Hindi et al. [9] proposed a novel approach based on the Inverse Specified Class Distance Metric (ISCDM) and Value Distance Metric by refining the KNN model, making it more suitable for documents represented as vectors via word frequency. Recent studies have found that Extreme Gradient Boosting [10] and Light Gradient Boosting Machine [11] demonstrate outstanding performance. However, the overall classification effectiveness of machine learning remains relatively low. This is because traditional machine learning relies on shallow feature extraction, neglecting the relationships between words and between sentences. It lacks sufficient understanding of the underlying semantics, structure, sequence, and context of text, resulting in poor handling and generalization capabilities for high-dimensional data and limited representational power of the models.

Compared to traditional machine learning, deep learning can utilize its own network architecture to learn and extract data features. Examples include convolutional neural networks, recurrent neural networks [12], and pre-trained models.

CNNs were initially proposed for image classification due to their ability to extract image features through convolutional filters. CNNs perform multi-kernel convolutional operations across multiple sequence blocks. Consequently, neural networks have found applications in numerous NLP problems. Building upon CNN architecture, Kim proposed TextCNN, an unbiased model of convolutional neural networks. It can better identify distinguishing phrases in max-pooling layers through a single convolutional layer and learn hyperparameters beyond word vectors by keeping word vectors static. Building upon this, Alexis et al. [13] investigated issues with deep networks in text classification tasks and proposed the character-level deep convolutional neural network VDCNN; Johnson et al. [14] introduced a model for text classification tasks based on two-view semi-supervised learning. Building upon this, Johnson et al. [15] introduced Deep Pyramid Convolutional Neural Networks for Text Categorization, enhancing computational accuracy by increasing network depth. DPCNN features a simpler architecture than Residual Networks [16]. Furthermore, researchers adopted graph neural network optimization. Kipf et al. [17] proposed the Graph Convolutional Network model based on graph convolutional neural networks. Subsequent researchers studied and developed variants of the GCN model, particularly for text classification tasks, introducing models such as TextGCN [18], D-GCN [19], and GCNI [20].

Recurrent neural networks are commonly used to capture the evolutionary direction of sequences through recursive computations. Within deep learning, gated recurrent units and long short-term memory models are particularly prevalent. To overcome issues such as gradient explosion or vanishing in RNNs, Koutnik et al. [21] proposed the Clock Work RNN (CW-RNN), which employs a clock frequency to optimize RNN performance. Building upon unidirectional RNNs, Schuster et al. [22] introduced bidirectional recurrent neural networks. This model captures information from both past and future directions, enhancing accuracy in text classification tasks. Wang et al. [23] introduced a capsule-structured RNN model that demonstrated strong performance in sentiment classification tasks. Building upon this, Zhou [24] noted that BiLSTM fails to accurately measure the importance of each word. Consequently, they combined BiLSTM with an attention mechanism for text classification tasks, extracting contextual semantic information from both preceding and subsequent perspectives.

In 2019, Devlin et al. [25] introduced the BERT model. By jointly adjusting the context across all layers to pre-train deep bidirectional representations, it efficiently learned global semantics and achieved state-of-the-art performance across multiple NLP evaluation tasks, thereby ushering in the era of pre-trained models for NLP. Subsequent research focused on optimizing BERT. Building upon it, ROBERTa [26] introduced improvements through dynamic masking, generating masking patterns sequentially and feeding them into the model. ALBERT [27] reduced BERT’s parameters by shortening token lengths and sharing parameters across all encoders, enabling cross-layer parameter sharing. Dai et al. [28] trained three Chinese BERT models using standard character-level masking, whole-word masking, and a combination of both. Dict-BERT [29] enhanced language model pre-training by leveraging definitions of rare words from dictionaries.

3. Methods

The overall structure of the model is primarily constructed based on two branches. The first branch consists of an embedding layer and raw sentence vectors, primarily performing comprehensive semantic feature extraction from sentences. The second branch comprises an embedding layer and a multi-layer, multi-scale CNN, performing global contextual feature extraction. The overall architecture is shown in Figure 1.

3.1. Word Embedding Layer

RoFormer [30] is an enhanced Transformer model featuring rotational position embeddings. Specifically, Rotary Position Embedding (RoPE) encodes absolute positions using rotation matrices while explicitly incorporating relative position dependencies into the self-attention formulation. The core idea is to encode relative positions by multiplying context embeddings using rotation matrices based on absolute positions. It can be proven that the inner product of context embeddings depends solely on relative positions.

RoPE was designed to address the limitation in Transformers whereby positional encoding cannot represent relative relationships. By embedding positional information into the query and key vectors and implementing it as a rotation matrix, the attention scores are directly dependent on the relative positions between tokens. This allows the attention mechanism to naturally capture distances between tokens, thereby enhancing generalization capabilities for long sequences. The core idea is to construct a rotation matrix, enabling relative position encoding on top of existing absolute position encoding, as detailed in Equation (1).

$$\left[f\left(q,m\right),f\left(k,n\right)\right]=g\left(q,k,m-n\right)$$

(1)

Suppose there exists a function g such that the dot product of vectors generated by absolute position information q and k contains the relative position m − n between them. The specific operation is computed using rotation matrices, as shown in Equation (2).

$$f\left(q,m\right)=\left(\begin{array}{cc}\cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{array}\right)\left(\begin{array}{c}{q}_0\\ q_1\end{array}\right)$$

(2)

Since matrix multiplication represents a spatial transformation mapping one matrix to another, and the aforementioned matrices perform rotation operations on q, the corresponding k undergoes identical rotational operations. Subsequently, relative position information is expressed through matrix multiplication, as detailed in Equation (3).

$$\mathrm{g}=f{(q,m)}^{T}f(k,n)=(q_0 q_1)\left(\begin{array}{cc}\cos ((m-n)\theta )& -\sin ((m-n)\theta )\\ \sin ((m-n)\theta )& \cos ((m-n)\theta )\end{array}\right)\left(\begin{array}{c}{k}_1\\ k_2\end{array}\right)$$

(3)

By multiplying matrices, the attention calculation automatically incorporates the relative positional relationship between m and n, enabling a comprehensive integration of word semantics and positional information. This approach captures more word features, thereby better addressing the issue of polysemy.

3.2. Multi-Scale Convolutional Layer

The character feature H outputs from RoFormer are fed into CNN networks of varying scales. This study employs convolutions using convolutional kernels of sizes 1 × 1024, 3 × 1024, and 5 × 1024 to capture global contextual information at different scales, as shown in Figure 2.

The specific implementation formulas are as shown in Equations (4)–(6).

$$y_i=f(W_r{H}_{i;i+r-1}+b),r=1,3,5$$

(4)

$$Y^r=y_1^r\oplus y_2^r\oplus \cdots \oplus y_{L-r+1}^r$$

(5)

$$Y=[Y_1^1\oplus Y_1^3\oplus Y_1^5]$$

(6)

In the above equation, y_i represents the information extracted by the i-th convolution kernel of size r, f denotes the Tanh activation function, W_r indicates the weight matrix corresponding to kernel r, H_i:i+r−1 denotes the feature values within the window from i to i + r − 1, and b is the bias parameter. $Y^r$ represents the feature information obtained using the r-sized convolution kernel, while $Y$ denotes the global information obtained by concatenating the results from different scales along the first dimension.

The next step employs max pooling operations and fully connected layers to prevent overfitting while extracting the final deep textual features as shown in Equations (7) and (8).

$$V_{3r}=\max (Y_1,Y_2,Y_3,\dots ,Y_{seq})$$

(7)

$$V_{emdedding}=f(W{V}_{3r}+b)$$

(8)

In Formula (7), max pooling is applied to the second dimension of vector $Y$ to obtain the most prominent features extracted by each convolution kernel. A fully connected layer is then used to align the output dimension with the embedding layer dimension.

3.3. Feature Fusion Layer

Sentence vector outputs represent the entire sentence text to a certain extent, enabling fairer integration of character information within sentences. However, they lack contextual interaction. To address this, this study combines sentence vector outputs with convolutional outputs, thereby highlighting key word features and enhancing text classification bias. The specific formulas are as shown in Equations (9)–(11).

$$F=CL{S}_{out}\odot Y_{emdedding}$$

(9)

$$Out=Concat[\mathsf{\partial }\ast F;(1-\mathsf{\partial })\ast F·P]$$

(10)

$$P_{i,j}=\left\{\begin{array}{cc}1,& \mathrm{if} j=\sigma (i)\\ 0,& \mathrm{otherwise}\end{array}\right.$$

(11)

Formula (9) represents the bitwise multiplication of the sentence vector output CLS and the convolutional output $Y$, which better preserves detailed information and enhances the model’s nonlinear modeling capability. In Formula (10), $\mathsf{\partial }$ denotes the confusion coefficient following a Beta distribution, and P represents a matrix. In Formula (11), $\mathit{P}\in \{\mathrm{0,1}{\}}^{N\times N}$ is a randomly generated permutation matrix serving function. $\sigma$ is a random index shuffling function. This formula is used to implement random sample generation within batches. By applying sample mixing output [31], the model gains enhanced robustness and resistance to interference. The random shuffling and confusion processes are shown in detail in Figure 3, below.

Figure 3 illustrates the detailed workflow of the random shuffling and confusion mechanism, which consists of three key steps:

(1)

Random Shuffling: Based on the fused features F generated by Equation (9), we first extract the corresponding sample indices. The torch.randperm function is then employed to randomly permute these indices, yielding the shuffled feature representation F’ (as depicted in the top-right corner of Figure 3).

(2)

Adaptive Confusion: A mixing coefficient ∂, sampled from a Beta distribution, is introduced to perform linear interpolation. Specifically, the original fused features F are weighted by ∂, while the shuffled features F’ are weighted by the complementary coefficient (1 − ∂). These weighted components are then summed to achieve adaptive feature confusion.

(3)

Feature Concatenation: The resulting confused feature vectors are concatenated to construct the final enhanced representation, preparing the input for the subsequent fully connected layers for classification.

3.4. Loss Function

Label smoothing [32] is an optimization technique employed in deep learning tasks. In recent years, label smoothing has proven effective for deep learning applications. By reducing the predicted probability of the target class while increasing the predicted probabilities of other classes, it enhances the model’s tolerance to errors and effectively suppresses overfitting. Simultaneously, label smoothing increases inter-class distances while reducing intra-class distances in classification tasks. This makes the clusters of different categories more compact, thereby improving model generalization.

The specific implementation involves converting hard labels into soft labels with uniformly distributed categories, as shown in Equations (12) and (13).

$${\widehat{y}}_i=y_{\mathrm{hot}}(1-\alpha ){\displaystyle \frac{\alpha }{K}}$$

(12)

$${\widehat{y}}_i=\left\{\begin{array}{cc}1-\alpha & i=\mathrm{target}\\ \alpha /K& i\ne \mathrm{target}\end{array}\right.$$

(13)

In the above equation, K represents the total number of categories, and the smoothing coefficient $\propto$ denotes the probability reduction value for the predicted label, set to 0.1 in this paper. Equation (13) is a simplified form of Equation (12). Finally, the soft labels are substituted for the original y_i and applied to the cross-entropy loss function to optimize the model parameters, as shown in Equation (14).

$$\mathrm{CELoss}(y,p)={\displaystyle \sum _{i=1}^K-}{y}_i\log p_i$$

(14)

4. Experiments

4.1. Dataset Selection and Characteristics

The datasets used in this experiment are Waimai_10k, ChnSentiCorp, and COLD. The specific experimental data are shown in

Table 1. Dataset.

Dataset	Train	Val	Test	Domain
Waimai_10k	9589	1199	1199	Food Delivery Reviews
ChnSentiCorp	9600	1200	1200	Hotel Reviews
COLD	25,726	6431	5323	Offensive Language Detection

.

Waimai_10k: Extracted from Meituan Takeout reviews, comprising 9589 training examples, 1199 validation examples, and 1199 test examples, totaling 11,987 reviews across two categories: 4000 positive reviews and 7987 negative reviews. The average sentence length is 24.92 characters, as shown in Figure 4.

ChnSentiCorp: An online hotel review dataset comprising 9600 training examples, 1200 validation examples, and 1200 test examples, totaling 12,000 reviews. It features two categories: 6000 positive and 6000 negative reviews. The average sentence length is 103 characters, as shown in Figure 5.

COLD: A Chinese offensive language evaluation dataset covering topics such as race, gender, and region. The dataset contains 37,480 statements across two categories: 18,041 offensive language statements and 19,439 safe language statements. The average sentence length is 47.77 characters, as shown in Figure 6.

4.2. Evaluation Metrics

The experiment employs Accuracy, Precision, Recall, and F1-measure as evaluation metrics. Accuracy represents the proportion of correct samples in the predicted sample set relative to the total sample population. Precision measures the proportion of samples where both the predicted and actual results are positive, out of all samples predicted as positive. Recall measures the proportion of samples where both the predicted and actual results are positive, out of all samples that are actually positive. The F1-measure represents the weighted average of Recall and Accuracy, as shown in Equations (15)–(18).

$$Accurary={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(15)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(16)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(17)

$$F_1={\displaystyle \frac{precision\times recall\times 2}{precision+recall}}$$

(18)

Among these, TP, TN, FP, and FN represent different classification outcomes, with specific classifications detailed in

Table 2. Confusion matrix.

Sample True Class	Model Prediction Category
	True	False
True	TP	FN
False	FP	TN

.

4.3. Experimental Environment and Preprocessing

Table 3. Experimental environment.

Environment Component	Configuration
Operating System	Ubuntu20.04
CPU	Intel Xeon Platinum 8358P
GPU	RTX 3090
RAM	90 GB
Programming Language	Python 3.8
Framework	PyTorch 1.10.0

shows the experimental environment of this study. All experiments were implemented using Python 3.8 (Python Software Foundation, Wilmington, DE, USA) and PyTorch 1.10.0 (Meta Platforms Inc., Menlo Park, CA, USA) on Ubuntu 20.04 (Canonical Ltd., London, UK), with an NVIDIA GeForce RTX 3090 GPU (NVIDIA Corporation, Santa Clara, CA, USA).

The hyperparameter configuration of the proposed model is detailed in

Table 4. Parameter settings.

Parameter	Waimai_10k	ChnSentiCorp	COLD
Batch size	16	16	16
max_len	256	256	256
lr	2 × 10−5	2 × 10−5	2 × 10−5
Word vector dimension	1024	1024	1024
eval_step	500	500	500
Optimizer	AdamW	AdamW	AdamW
Epoch	7	7	3
Convolution kernel size	1, 3, 5	1, 3, 5	1, 3, 5
$\mathsf{\partial }$	0.9	0.9	0.9
out_channels	256	256	256
Dropout	0.3	0.3	0.3

. Specifically, the dropout rate is set to 0.3, and the GELU activation function is employed to accelerate model convergence. The optimization objective utilizes label-smoothed cross-entropy loss, with a fixed batch size of 16. Given variations in dataset sizes, the number of training epochs was adjusted accordingly. Furthermore, the eval_step parameter was configured to control the frequency of performance evaluation (i.e., the interval of steps between assessments). Detailed specifications are presented in Table 4.

We employed the RoFormer-Large configuration for our experiments. The specific architectural hyperparameters were set as follows: 24 hidden layers (num_hidden_layers), a hidden size of 1024 (hidden_size), 16 attention heads (num_attention_heads), and the GELU activation function (hidden_act).

During the data preprocessing phase, the raw datasets are stored in a text + ‘\t’ + label format, where the text content and the class label are separated by a tab character. Subsequently, an adaptive truncation strategy is applied based on a predefined maximum sequence length (max_len). Specifically, the algorithm first checks whether the text length is less than or equal to max_len−2. If this condition is met, the original text is preserved in its entirety. Conversely, if the text exceeds this limit, it is truncated to exactly max_len−2 characters. Rationale: The subtraction of 2 is intentional to reserve space for the special [CLS] and [SEP] tokens required by pre-trained models. This ensures that the final input sequence strictly adheres to the model’s length constraints while enabling proper semantic interpretation.

4.4. Comparative Model

TextCNN: The Text Convolutional Neural Network is a convolutional neural network architecture specifically designed for processing text data.

FastText [33]: This model is an efficient text classification and representation library and framework developed by Facebook’s AI Research team.

DPCNN: The Deep Pyramid Convolutional Neural Network is a convolutional neural network architecture specifically designed for text classification tasks.

BERT: This model is a powerful pre-trained language representation model proposed by Google researchers in 2018. It significantly advanced the field by achieving state-of-the-art results across various natural language processing tasks.

BERT-CNN-LSTM [34]: This model utilizes BERT as the word embedding model. First, TextCNN extracts semantic features from the text, and then LSTM extracts feature information from the TextCNN output.

BERT-BiLSTM-ATT [35]: This model also employs BERT as the word embedding model. BiLSTM extracts semantic features from the text, and an attention mechanism is used to weight these features.

BERT-CBLBGA [36]: This model utilizes BERT as the word embedding model, employing TextCNN, BiLSTM, and BiGRU to extract global and local semantic features from text, and combines these through self-attention for feature fusion.

ALBERT: This is a variant of the BERT model designed to reduce model size and computational requirements while maintaining performance. This model was proposed by Google Research in 2019.

RoBERTa: This is a variant of the BERT (Bidirectional Encoder Representations from Transformers) model proposed by Facebook AI researchers in 2019. By addressing specific limitations and applying additional training techniques, RoBERTa aims to enhance BERT’s performance.

Ernie [37]: This is a language representation model developed by Baidu Research. It significantly improves language understanding capabilities by integrating external knowledge resources such as knowledge graphs or structured databases.

XLNet [38]: This is a state-of-the-art language representation model proposed by Google researchers in 2019. Based on the Transformer architecture, it introduces several innovative concepts to overcome limitations of previous models.

MacBert [39]: This model effectively bridges the gap between pre-training and fine-tuning phases by adopting a similar word tokenization mechanism, significantly improving performance on downstream tasks.

AlDCBAT [40]: This proposes a dual-channel hybrid framework integrating ALBERT, CNN, and BiGRU. Designed specifically for Tibetan sentiment analysis, the model mitigates semantic information loss and demonstrates improved classification accuracy compared to single-channel baselines.

OSLCFit [41]: This proposes a cohesive feature extraction architecture that organically integrates CNN and Bi-LSTM networks. By employing a unified optimizer and fine-tuning embeddings on review data, the model effectively tailors representations for sentiment polarity classification.

KSCB [42]: This model establishes an end-to-end framework integrating K-means++-SMOTE with CNN-Bi-LSTM. By restructuring corpora based on imbalance ratios, the model achieves adaptive data distribution optimization, demonstrating superior robustness compared to standard deep learning baselines.

CG-BERT [43]: This model integrates graph convolutions with BERT to model aspect-context dependencies, significantly enhancing aspect-level sentiment classification performance in low-resource scenarios.

MF-SDAM [44]: This model fuses syntactic dependency, BERT, TextCNN, and Att-BiLSTM via multi-feature fusion to enhance sentiment classification accuracy and robustness.

4.5. Results and Analysis

4.5.1. Comparison with Other Models

The results of the model comparison experiment are shown in

Table 5. Model comparison results.

Model	ChnSentiCorp		COLD		Waimai_10k
	Acc (%)	F1 (%)	Acc (%)	F1 (%)	Acc (%)	F1 (%)
TextCNN	84.00	84.00	74.15	74.09	87.56	85.40
FastText	86.92	86.90	74.96	75.21	89.93	88.83
DPCNN	84.75	82.75	76.57	76.22	88.93	87.64
BERT-CNN-LSTM	-	-	-	-	89.65	88.35
BERT	92.30	92.29	81.23	81.39	88.98	87.75
BERT-BiLSTM-ATT	-	-	-	-	89.23	87.71
Albert-base	89.10	89.09	79.56	80.12	87.40	86.80
Albert-large	91.47	91.52	80.24	79.83	-	-
RoBerta	92.64	92.64	81.62	81.83	-	-
MacBERT	92.45	92.45	81.36	81.29	-	-
ERNIE	93.58	93.58	81.29	81.46	-	-
Xlnet	94.75	94.75	81.57	81.76	91.42	90.50
AlDCBAT	-	-	-	-	89.07	87.46
OSLCFit	-	-	-	-	89.01	90.57
KSCB	-	-	-	-	89.62	89.51
CG-BERT	-	-	-	-	89.01	78.34
MF-SDAM	95.00	95.00	-	-	91.49	90.36
Our	95.94	95.91	82.13	82.10	91.50	91.48

Note: Bold values indicate the best performance compared to the baseline models.

.

The results of Table 5 have the following indications:

(1)

Across three datasets, the Acc score improvement over traditional deep learning models ranged from 9.02% to 11.94%, 5.56% to 7.98%, and 1.57% to 3.94%. Similarly, the F1 score improvements were 9.04–13.16%, 5.88–8.01%, and 2.65–6.08%.

(2)

Across the three datasets, compared to pre-trained models, the Accuracy improvement ranged from 1.19% to 6.84%, 0.51% to 2.57%, and 2.52% to 4.1%. Similarly, the F1 score improvement ranged from 1.16% to 6.82%, 0.27% to 2.27%, and 3.73% to 4.68%. Compared to traditional deep learning models, pre-trained models can learn deeper semantic meanings. However, the proposed model further improves performance over pre-trained models, indicating that our approach uncovers local fine-grained features overlooked by pre-trained models, thereby enhancing the model’s resolution when processing complex texts.

(3)

On the Waimai_10k dataset, compared to the AlDCBAT model, Accuracy and F1 scores improved by 2.43% and 4.02%, respectively; compared to the OSLCFit model, Accuracy and F1 scores improved by 2.49% and 0.91%, respectively. The results demonstrate that our method achieves further performance improvements compared to the AlDCBAT and OSLCFit models, which also employ convolutional networks. This is attributed to the high divergence and colloquial, informal expressions characteristic of the Waimai_10k dataset. Our proposed approach utilizes multi-scale convolutions to capture local keywords and long-range semantic dependencies. Through a multi-granularity feature fusion mechanism, it significantly enhances the feature fitting capability for non-standardized text compared to conventional convolutional networks.

(4)

On the Waimai_10k dataset, compared to the KSCB model, the Accuracy and F1 scores improved by 1.88% and 1.97%, respectively. Results indicate that while KSCB effectively adjusts data distribution by generating synthetic corpora, this sample-generation strategy may introduce latent semantic noise into the feature space. In contrast, the proposed method achieves synergistic collaboration among its modules, comprehensively analyzing text semantics across multiple receptive fields and granularities. Concurrently, its deeply integrated feature confusion mechanism effectively suppresses overfitting risks caused by noisy data, balancing multidimensional feature extraction with model robustness.

(5)

On the ChnSentiCorp dataset, our method achieves an Accuracy of 95.94% and an F1 score of 95.91%, surpassing MF-SDAM, which records 95% for both metrics. This demonstrates that integrating RoFormer-based positional enhancement with multi-scale convolution more effectively captures long-range dependencies and local semantic nuances in Chinese text. In the short-text, high-noise scenario of Waimai_10k, our approach attains 91.5% accuracy and 91.48% F1, significantly outperforming CG-BERT, which yields an F1 score of 78.34%, and slightly exceeding MF-SDAM. Notably, the substantial improvement in F1 score underscores the efficacy of our feature confusion mechanism in suppressing noise interference and sharpening decision boundaries, thereby confirming the model’s enhanced robustness and generalization capability, particularly under class imbalance conditions.

4.5.2. Ablation Experiment

• Overall Analysis

To better explore the effective structure of the model, ablation experiments were conducted on three datasets. The model is introduced as follows:

Base: Uses only the RoFormer pre-trained model.

Hierarchy-aware: Incorporates multi-scale convolutional modules.

Feature fusion: Utilizes feature fusion modules.

Specific experimental results are shown in

Table 6. Model ChnSentiCorp ablation experiment.

Model	ChnSentiCorp
	Acc (%)	F1 (%)
Base	94.93	94.92
Base + Hierarchy-aware	95.16	95.15
Base + Hierarchy-aware + Feature fusion (Addition, no confusion)	95.47	95.47
Base + Hierarchy-aware + Feature fusion (Addition, confusion)	95.86	95.86
Base + Hierarchy-aware + Feature fusion (Multiplication, no confusion)	95.86	95.86
Base + Hierarchy-aware + Feature fusion (Multiplication, confusion—our model)	95.94	95.91

Note: Bold values indicate the best performance compared to the baseline models.

,

Table 7. Model COLD ablation experiment.

Model	COLD
	Acc (%)	F1 (%)
Base	80.95	80.77
Base + Hierarchy-aware	81.49	81.27
Base + Hierarchy-aware + Feature fusion (Addition, no confusion)	81.88	81.64
Base + Hierarchy-aware + Feature fusion (Addition, confusion)	81.79	81.48
Base + Hierarchy-aware + Feature fusion (Multiplication, no confusion)	81.83	81.53
Base + Hierarchy-aware + Feature fusion (Multiplication, confusion—our model)	82.13	82.10

Note: Bold values indicate the best performance compared to the baseline models.

and

Table 8. Model Waimai_10k ablation experiment.

Model	Waimai_10k
	Acc (%)	F1 (%)
Base	90.83	89.23
Base + Hierarchy-aware	90.91	89.73
Base + Hierarchy-aware + Feature fusion (Addition, no confusion)	91.33	90.05
Base + Hierarchy-aware + Feature fusion (Addition, confusion)	90.91	89.54
Base + Hierarchy-aware + Feature fusion (Multiplication, no confusion)	91.08	89.85
Base + Hierarchy-aware + Feature fusion (Multiplication, confusion—our model)	91.50	91.48

Note: Bold values indicate the best performance compared to the baseline models.

.

The results in Table 6, Table 7 and Table 8 are analyzed as follows:

(1)

To validate the role of the hierarchical perception module in extracting local semantic features, a multi-scale convolution module was introduced based on the Base model. Experimental results show that after adding this module, the model outperformed the baseline model across all three datasets. Specifically, on the ChnSentiCorp dataset, Acc improved from 94.93% to 95.16%; on the COLD dataset, it increased from 80.95% to 81.49%; and on the Waimai_10k dataset, it rose from 90.83% to 90.91%. This significant improvement demonstrates that the multi-scale convolutional network effectively captures local phrase features, complementing the global features of the pre-trained model.

(2)

Having validated the Hierarchy-Aware Module, we further investigated the necessity of the Feature Fusion Module. By integrating BERT sentence vectors with the outputs of multi-scale convolutions, model performance was further enhanced. Experimental data show that compared to models using only hierarchical perception, introducing feature fusion increased ChnSentiCorp Accuracy by 0.31%, COLD by 0.39%, and Waimai_10k by 0.42%. This proves that simple feature concatenation is insufficient to fully utilize information, while explicit feature fusion mechanisms can more effectively integrate global and local information to improve classification accuracy.

(3)

To determine the optimal feature interaction method, experiments compared “additive” and “multiplicative” fusion strategies. While the additive strategy performed well on some metrics, the multiplicative strategy demonstrated greater potential for deep interactions among high-dimensional features. Specifically, on the ChnSentiCorp dataset, the “multiplication, no confusion” strategy achieved an Accuracy of 95.86%, significantly outperforming the “addition, no confusion” strategy’s Accuracy of 95.47%. This suggests that the multiplication operation may function similarly to gating or attention mechanisms within the feature space, suppressing noise while amplifying key feature representations. This provides a superior feature foundation for subsequent confusion mechanisms.

(4)

Finally, experiments validate the ultimate contribution of the proposed obfuscation mechanism to the model. Comparing the “multiplication, no obfuscation” approach with the final model (ours—multiplication, obfuscation), it can be seen that incorporating the obfuscation mechanism achieves optimal performance across all datasets. For the more complex COLD and Waimai_10k datasets, in particular, the Accuracy improved to 82.13% and 91.50%, respectively, with significant gains in F1 scores. This confirms that the confusion mechanism, as an effective regularization approach, successfully prevents model overfitting on training data while substantially enhancing generalization capability and robustness across diverse domain data.

2.

Category Accuracy Analysis

The aforementioned ablation experiments have demonstrated the effectiveness of the proposed module. To more precisely investigate the improvement of the proposed model on classification results, category accuracy validation was conducted on three datasets. The specific results are shown in

Table 9. Model ChnSentiCorp category metric.

Model	ChnSentiCorp
	Negative			Positive
	P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)
Base	93.15	97.26	95.16	96.97	92.48	94.67
Base + Hierarchy-aware	93.44	97.41	95.38	97.15	92.80	94.92
Base + Hierarchy-aware + Feature fusion (Addition, no confusion)	93.98	97.41	95.67	97.17	93.44	95.26
Base + Hierarchy-aware + Feature fusion (Addition, confusion)	95.76	96.20	95.98	95.98	95.52	95.74
Base + Hierarchy-aware + Feature fusion (Multiplication, no confusion)	94.28	97.87	96.04	97.66	93.76	95.67
Base + Hierarchy-aware + Feature fusion (Multiplication, confusion—our model)	95.49	96.65	96.07	96.43	95.20	95.81

Note: Bold values indicate the best performance compared to the baseline models.

,

Table 10. Model COLD category metric.

Model	COLD
	Safe			Offensive
	P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)
Base	92.11	74.87	82.60	70.17	90.22	78.94
Base + Hierarchy-aware	91.48	76.49	83.31	71.29	89.13	79.22
Base + Hierarchy-aware + Feature fusion (Addition, no confusion)	91.33	77.36	83.77	71.98	88.79	79.51
Base + Hierarchy-aware + Feature fusion (Addition, confusion)	90.16	78.42	83.88	72.52	86.94	79.08
Base + Hierarchy-aware + Feature fusion (Multiplication, no confusion)	90.34	78.29	83.89	72.47	87.23	79.17
Base + Hierarchy-aware + Feature fusion (Multiplication, confusion—our model)	90.28	78.91	84.22	73.00	87.04	79.41

Note: Bold values indicate the best performance compared to the baseline models.

and

Table 11. Model Waimai_10k category metric.

Model	Waimai_10k
	Negative			Positive
	P (%)	R (%)	F1 (%)	P (%)	R (%)	F1 (%)
Base	90.02	97.00	93.38	92.89	78.50	85.09
Base + Hierarchy-aware	92.81	93.62	93.21	87.02	85.50	86.25
Base + Hierarchy-aware + Feature fusion (Addition, no confusion)	91.92	95.37	93.61	90.00	83.25	86.49
Base + Hierarchy-aware + Feature fusion (Addition, confusion)	91.37	95.37	93.33	89.86	82.00	85.75
Base + Hierarchy-aware + Feature fusion (Multiplication, no confusion)	92.41	94.37	93.38	88.25	84.50	86.33
Base + Hierarchy-aware + Feature fusion (Multiplication, confusion—our model)	91.64	96.00	93.77	91.16	82.50	86.61

Note: Bold values indicate the best performance compared to the baseline models.

.

Table 9, Table 10 and Table 11 present the following analysis results:

(1)

After introducing the hierarchical perception module on top of the Base model, the model’s ability to capture fine-grained semantics is significantly enhanced, with particularly noticeable improvements in Recall. Experimental data reveals that the pure Transformer architecture exhibits omissions when processing certain specific categories. Taking the Waimai_10k dataset as an example, the Base model achieves only 78.50% Recall for positive categories. However, after incorporating the hierarchical perception module, this metric rapidly jumps to 85.50%, representing a 7-percentage-point increase. This demonstrates that the multi-scale convolutional network effectively extracts key local phrase features, compensating for the pre-trained model’s shortcomings in local semantic perception and substantially reducing sample omissions.

(2)

When validating the feature fusion module, experiments compared “additive” and “multiplicative” strategies (both under no confusion settings). Data indicates that the multiplicative strategy promotes deeper interactions between features. On the ChnSentiCorp dataset, comparing the “additive, no confusion” model with the “multiplicative, no confusion” model, the latter achieved higher F1 scores across both categories: F1 for the Negative category increased from 95.67% to 96.04%, and F1 for the positive category rose from 95.26% to 95.67%. This suggests that the multiplication operation may function similarly to attention gating, enhancing core semantic features while suppressing noise, thereby achieving a better balance between Precision and Recall.

(3)

After determining “multiplication” as the optimal fusion method, introducing a random index shuffling mechanism (our model) further enhances the model’s robustness in complex contexts. This is particularly evident in the more challenging COLD dataset. Compared to “Multiplication without Scrambling” and “Our,” the F1 score for the Safe category increased from 83.89% to 84.22%, while the F1 score for the Offensive category rose from 79.17% to 79.41%. The confusion mechanism effectively regularizes the model by increasing training sample diversity, enabling stronger generalization capabilities when encountering semantically ambiguous or borderline samples.

(4)

Overall, the Base model often exhibited an imbalance between Precision and Recall metrics, whereas our model achieved high performance across multiple metrics through modular integration. For instance, in the positive category of ChnSentiCorp, the Base model exhibits relatively low Recall (92.48%), whereas our model maintains high Precision (96.43%) while significantly boosting Recall to 95.20%, ultimately achieving an optimal F1 score of 95.81%. In the Negative category of Waimai_10k, our model achieves an impressive 96.00% Recall and 93.77% F1 score. This confirms that the proposed architecture effectively optimizes prediction distributions for specific categories, demonstrating significant advantages in fine-grained classification tasks.

4.5.3. Parameter Sensitivity Experiment

• Multi-scale parameters

To determine the optimal hyperparameter settings for the multi-scale convolutional module and investigate the contribution of local semantic features at different granularities to classification performance, we compared various combinations of convolutional kernel sizes. The kernel size dictates the receptive field range for extracting local context. While keeping other parameters constant, we tested multiple combinations. The experimental results are shown in Figure 7, Figure 8 and Figure 9.

Results in Figure 7, Figure 8 and Figure 9 indicate the following:

(1)

For the Acc metric, the ChnSentiCorp dataset achieved optimal performance of 96.10% when the convolution kernel value was (1, 3), with the lowest performance of 95.86% at value 1. In the COLD dataset, performance peaked at 82.26% with kernel values (1, 3) and dropped to 81.70% at value 1. In the Waimai_10k dataset, performance reached its best at 91.50% with kernel values (1, 3, 5) and 1, while the lowest performance was 91.41% at values (1, 3).

(2)

In the Waimai_10k dataset, multi-scale processing did not yield significant improvements. This is because food delivery reviews are typically very short, with most core semantics often contained within just 1–2 characters or words. Consequently, a single scale is sufficient to capture the vast majority of critical sentiment features.

(3)

On the semantically ambiguous COLD and ChnSentiCorp datasets, models using Kernel = 1 alone performed poorly. Introducing the dual-scale structure (1, 3) rapidly enhanced the model’s understanding of phrase-level semantics, improving performance by 0.56% and 0.24%, respectively. This demonstrates that convolutional kernels of different scales form effective complementarity in the feature space, jointly constructing a more comprehensive semantic view.

2.

Confusing Parameters

To further evaluate the robustness of the proposed model and investigate the impact of data augmentation intensity on model performance within the confusion mechanism, we conducted experiments on hyperparameters. Their values determine the mixed proportion distribution when generating virtual samples. Different values were set across three scenarios, with the results shown in Figure 10, Figure 11 and Figure 12.

The results from Figure 10, Figure 11 and Figure 12 indicate the following:

(1)

In the ChnSentiCorp dataset, performance peaked at 96.25% when the threshold was set to 0.3 and reached its lowest point at 95.55% when set to 0.5. Overall, the average Accuracy and F1 scores reached 95.98% and 95.97%, respectively, outperforming all comparison models in the experiment. Additionally, the range of variation for both Accuracy and F1 metrics was 0.70%, indicating low fluctuation.

(2)

In the COLD dataset, performance peaked at 82.71% with a value of 0.6 and reached its lowest point at 82.13% with a value of 0.9. Overall, the average Accuracy and F1 scores both reached 82.36%, outperforming the comparison models in the experiment. Additionally, the range of variation for both Accuracy and F1 scores was low at 0.58% and 0.61%, respectively.

(3)

In the Waimai_10k dataset, performance peaked at 91.66% with a value of 0.1 and reached its lowest point at 90.75% with a value of 0.6. Overall, the average Accuracy and F1 scores both reached 91.06%, outperforming the comparison models in the experiment. Additionally, the range of Accuracy results was 0.91%, indicating a low fluctuation range.

(4)

The differing extremes across these three datasets stem from distinct semantic space distributions: The ChnSentiCorp dataset primarily consists of reviews from domains like hotels and books. Texts are typically of moderate length with relatively standardized syntax, clear sentiment word distribution, and minimal noise, allowing smaller parameter values to serve as regularization. The COLD dataset contains aggressive social media discourse, often featuring ambiguous semantics, irony, slang, or non-standard expressions. Parameter fine-tuning has limited impact here, demonstrating high robustness. The Waimai_10k dataset is characterized by extremely short texts with sparse semantic features, making it highly sensitive to parameter values.

In summary, through fine-grained parameter optimization, the proposed model fully leverages the advantages of feature masking and multi-scale convolutional networks to maximize classification accuracy, validating the method’s effectiveness and superiority. Concurrently, it is evident that the sparser the text’s features, the more sensitive the model becomes to parameters, necessitating smaller values; conversely, for more complex semantic contexts and richer features, larger parameter values should be explored.

4.5.4. Efficiency and Performance Trade-Off Analysis

To comprehensively evaluate the practical viability of our proposed method, we conducted a rigorous comparative analysis against the RoFormer-large baseline across three datasets. The evaluation encompasses five key metrics: Accuracy, F1-score, Training Time, Inference Time, and Model Size. The detailed results are presented in

Table 12. Comprehensive comparison of performance and efficiency.

Datasets	Acc (%)		F1 (%)		Training Time (s)		Inference Time (s)		Model Size (MB)
	Baseline	Our	Baseline	Our	Baseline	Our	Baseline	Our	Baseline	Our
ChnSentiCorp	94.93	95.94 (+1.01%)	94.92	95.91 (+0.99%)	1820.1	2030.3 (+210.2 s)	10.26	10.40 (+0.14 s)	1198	1214 (+16 M)
COLD	80.95	82.13 (+1.18%)	80.77	82.10 (+1.33%)	1445.96	1559.71 (+113.75 s)	23.97	24.04 (+0.07 s)	1198	1214 (+16 M)
Waimai_10k	90.83	91.5 (+0.67%)	89.23	91.48 (+2.25%)	820.46	916.75 (+96.29 s)	4.64	4.68 (+0.04 s)	1198	1214 (+16 M)

Note: Bold values indicate the best performance compared to the baseline models.

.

The proposed method achieves substantial improvements in classification performance compared to the baseline model, with an average gain of 0.95% in Accuracy and 1.52% in F1-score across all datasets. Furthermore, regarding efficiency, although training time increases by approximately 10%, inference latency remains negligible with an increase of less than 1.5%, and the model size grows only marginally by 16 MB. This demonstrates that our architectural enhancements deliver significant performance gains while introducing minimal computational overhead.

5. Conclusions and Future Work

To address the challenges of insufficient information extraction and suboptimal utilization of semantic features in text classification, we propose a feature-blending and hierarchical-aware classification approach. This model leverages RoFormer for character-level modeling to generate comprehensive word vector representations and incorporates a Multi-Scale Convolutional Network to capture both local and high-order features in depth. By employing our proposed feature-blending strategy, the model effectively enhances text representation learning and robustness against interference.

Extensive experiments on three benchmark datasets further validate the superiority of our method. Specifically, the model achieved Accuracy scores of 95.94% on ChnSentiCorp, 82.13% on COLD, and 91.50% on Waimai_10k. These results significantly outperform existing baselines, establishing a new benchmark for relevant research.

Future work will explore integrating knowledge graphs and multimodal methodologies to incorporate richer foundational data, thereby further improving classification efficacy and generalization capabilities across multi-domain texts.