DCANet: Diffusion-Coded Attention Network for Cross-Domain Semantic Noise Mitigation and Multi-Scale Context Fusion

Abstract

Neural language models have achieved remarkable progress in semantic representation learning. However, cross-domain representation learning still suffers from prominent semantic noise propagation issues. Existing methods still face challenges in cross-domain semantic modeling, including limited robustness across different semantic granularities, difficulty in separating transferable semantics from task-irrelevant semantic interference, and insufficient adaptability to specialized scenarios. These issues may reduce feature discriminability in fine-grained semantic tasks and complex application settings. To address these problems, we propose the Diffusion-Coded Attention Network (DCANet), a novel cross-domain representation learning architecture with three synergistic core modules: a multi-granular parallel diffusion masking mechanism for cross-scale context fusion via stochastic path activation, an implicit semantic encoder that distills domain-invariant patterns into adaptive bias codes via shared latent manifolds, and a self-correcting attention topology realizing dynamic semantic purification via closed-loop interactions between local features and global bias states. Extensive evaluations are conducted on nine well-recognized benchmark datasets to verify DCANet’s effectiveness and reliability. Experimental results show that DCANet attains state-of-the-art results on the majority of the benchmark datasets, with significant accuracy improvements on text classification and sentiment analysis tasks.

1. Introduction

Recent breakthroughs in neural language models have revolutionized the acquisition of semantic representations across diverse natural language processing tasks. The paradigm of transferring open-domain embeddings to downstream applications has demonstrated remarkable success, which is fundamentally based on the capacity to distill task adaptive text representations through low-dimensional vector mapping. Although traditional approaches like Word2Vec and GloVe established foundational word-level embedding frameworks [1,2,3], subsequent architectures employing CNNs and RNNs advanced contextual feature extraction—CNNs specializing in localized pattern recognition [4,5] and RNNs modeling sequential dependencies [6,7,8]. Nevertheless, these methods exhibit inherent limitations in resolving fine-grained semantic distinctions and cross-domain noise propagation [9,10,11]. Furthermore, as the role of new technologies continues to evolve, multi-scale context fusion remains a cornerstone for advancing language understanding in the next decade [12]. To address these inherent limitations in fine-grained semantic modeling and cross-domain noise suppression, our proposed DCANet leverages a multi-granularity diffusion mechanism to extract abstract information from text at different granularities.

Self-attention mechanisms have fundamentally transformed the dynamic modeling of syntactic and semantic relationships. Building on this foundation, innovations such as multilevel attention–CNN hybrids [5,7,8,13], contextual lexicon-enhanced LSTM architectures [14], and label-specific attention with gating mechanisms [15] have advanced tasks demanding nuanced semantic discrimination. However, traditional attention weighting strategies are limited by the semantic noise present in open-domain embeddings, especially when managing domain-shifted lexical semantics. Among them, domain drift is a distribution-level concept that describes the overall data distribution changes between the source domain and the target domain; semantic noise focuses on the sample or representation level, referring to persistent task-irrelevant residual interference within a single representation, which can occur even when the input data distribution remains unchanged. This propagation of noise reduces feature discriminability and ultimately impairs the precision of the model on fine-grained semantic tasks. In contrast, the DCANet combines a random diffusion graph generated by a diffusion masking with a global bias code output by a latent mechanism to decouple domain-specific noise from transferable semantics.

Although recent text representation models have achieved substantial progress, their performance may still degrade when open-domain semantic representations are transferred to downstream tasks with different contextual scopes and domain characteristics. In particular, representation learning may be affected by semantic noise, which in this work refers to task-irrelevant or misleading semantic components contained in transferred semantic embeddings. Such interference may weaken feature discriminability and reduce model precision in fine-grained semantic tasks. In addition, many existing architectures mainly rely on predefined local windows, fixed hierarchical structures, or single-scope attention patterns, which may limit their robustness across texts with substantially different semantic granularities and context spans. To address these challenges—weak fine-grained modeling, excessive open-domain noise, and insufficient semantic self-purification—we propose the Diffusion-Coded Attention Network (DCANet), which unifies global semantic bias correction with localized feature learning.

DCANet consists of the following three main modules: First, the Multi-granularity Parallel Diffusion Masking (MPDM) module generates semantic graphs through random path activation, thereby achieving cross-scale context fusion and alleviating the problems of fine-grained capture and global-local imbalance. Secondly, the Implicit Semantic Encoder (ISE) module extracts domain-invariant patterns as adaptive bias codes through a shared latent manifold, thereby separating noise from useful semantics and mitigating open-domain noise and domain bias. Third, the Self-Correcting Attention Topology (SAT) module achieves dynamic semantic purification by associating local features with the global implicit state, thus compensating for the lack of semantic self-purification. Comprehensive evaluations across nine benchmark datasets show that DCANet outperforms most state-of-the-art baselines and achieves state-of-the-art performance on three of the five main comparison datasets.

The main contributions of this work are as follows.

• Cross-Domain Noise Mitigation: DCANet introduces the first framework that unifies stochastic graph diffusion, global bias coding, and self-correcting attention for cross-domain semantic noise mitigation in NLP via the proposed MPDM and ISE, effectively separating domain-specific noise from transferable semantic patterns;
• Multi-Scale Context Integration: In the MPDM module, the diffusion masking mechanism enables simultaneous preservation of local semantic coherence and global distributional trends through density adaptive graph construction;
• Dynamic Semantic Purification: The shared manifold structure between the ISE and SAT constructs a closed-loop self-regulating attention mechanism where global semantic priors continuously refine local feature weighting, achieving progressive noise suppression.

2. Related Work

2.1. Self-Attention

The attention mechanism has achieved remarkable results in NLP tasks such as text classification and sentiment analysis, becoming a key foundational component. Recently, researchers have adopted attention-only frameworks instead of traditional CNNs/RNNs achieving significant success. In cross-domain sentiment analysis, He et al. [16] show that models tend to rely on domain-specific lexical features, which can lead to inconsistent predictions even when the underlying semantics remain unchanged. Similarly, Liu et al. [17] demonstrate through counterfactual experiments that altering surface-level tokens—while preserving the core semantic meaning—can significantly affect model outputs, indicating the presence of non-semantic interference in representations. These findings in cross-domain learning and representation learning suggest that text representations contain task-irrelevant or misleading semantic components, which correspond to semantic noise. However, addressing semantic noise on open-domain datasets requires better integration of attention mechanisms to improve contextual information computation.

To efficiently compute text context semantics, soft attention is applied to input sequences, calculating context semantic vectors as weighted sums of all hidden states to improve accuracy [18]. Liu [19] proposed a hybrid model combining multi-stage attention with TCN and CNN, which improves parallelism via TCN and integrates attention into CNN layers to emphasize discriminative features, boosting accuracy. To tackle incomplete information, Chen [20] designed an architecture combining semantic priors with deep attention residual groups to infer missing information. This general attention architecture, as shown in Figure 1, is applicable to various models and applications.

Previous research has primarily focused on improving the accuracy of semantic information computation by layering multiple attention levels and combining multiple model structures. In contrast, this study introduces a novel model architecture that seeks to integrate global contextual information into text representations, thereby achieving more effective semantic computation of text. While existing research has largely relied on layering diverse attention mechanisms and integrating complex structures to refine semantic computation, our method leverages task-specific contextual cues to focus on pertinent text features, resulting in richer semantic embeddings. This approach provides the model with a comprehensive and detailed semantic understanding of text data.

Existing text representation models encode semantics at different predefined granularities. For example, HAN [21] constructs document representations through a fixed word-level and sentence-level hierarchy, while Wang [22] models text by measuring the compatibility between embedded words and labels in a shared space. For longer contexts, Longformer [23] introduces an attention pattern that combines local windowed attention with task-motivated global attention. These representative studies suggest that semantic granularity and context scope are important modeling variables in text representation learning. Accordingly, in this paper we avoid overgeneralizing the limitations of prior work and instead describe the challenge more cautiously as robustness across heterogeneous semantic granularities and context spans.

2.2. Latent Code

Latent code is a fundamental concept in deep learning, particularly in generative models such as Generative Adversarial Networks (GANs) [15,24,25]. It denotes the compressed, low-dimensional representation of input data, capturing essential features and patterns while discarding irrelevant or redundant information. This latent representation underlies many advanced techniques in the generation, processing, and understanding of data features [26]. Latent code is derived via an encoding process, typically executed using a neural network encoder. This encoder maps high-dimensional input data to a low-dimensional latent space, where each point corresponds to a unique latent code. The latent space created captures the inherent structures and variations present in the input data in a more compact and meaningful form [20,27,28]. The core principle of latent code is its ability to disentangle the factors of variation within the data [29]. Based on the classical theoretical framework of latent representation disentanglement, the statistical independence between domain-specific and domain-invariant latent components on the shared manifold ensures that transferable semantics and domain noise do not overlap at the distribution level, thus forming a sufficient theoretical condition for semantic separation. Meanwhile, orthogonal initialization of linear layers effectively avoids latent space dimensional collapse and maintains the one-to-one correspondence between latent dimensions and independent semantic factors. These two conditions work together to make it possible to separate noise from semantic representations. The purpose of the latent code is to represent data such that individual dimensions or groups of dimensions correspond to different and interpretable generative factors. This separation enhances control over the generated output, enabling targeted modifications to specific data aspects. In GANs, the random vectors traditionally used as input can be viewed as a simple form of latent code [30]. However, recent advancements focus on leveraging more complex latent representations to enhance the consistency, geometric structure, and diversity of generated images. Models incorporating latent vectors facilitate fine-grained control over generated visual features, addressing issues like poor consistency and reduced geometric coherence observed in earlier methods, as shown in Figure 2.

These models often assume a level of independence between the components described by different layers in the generation process. However, by utilizing latent variable control, they can adjust and unify overall styles and features across layers, fostering information propagation between global latent features and model-specific features. The latent data is rich in global information and actively participates in the computation of each attention layer, ensuring feature coherence and consistency throughout the entire generation process.

The latent code is widely used in deep learning, particularly in generative models. Its disentangled representation enables targeted attribute manipulation. For example, SALAD [31] proposed a skeleton-aware latent diffusion model for text-driven action generation, demonstrating the disentangled operational capabilities of latent encoding in the text domain. Fu et al. [32] exploit the role of latent encoding in distilling essential features from noise; by filtering superficial wording to capture core emotional intent, the model generates high-quality emotion-supportive dialogues. By mapping discrete text and speech tokens into a continuous latent space, Ma et al. [33] demonstrate the potential of continuous latent encoding for data augmentation and style interpolation in sequential data processing. In conclusion, latent encoding is crucial in deep learning: it captures core features and removes redundant information, making it a valuable tool for data generation, manipulation, and understanding.

Previous research on latent codes has mostly focused on the image domain. This paper proposes a latent code method for text semantic computation. This method uses trainable parameters to learn global text semantic information and integrate it into the text semantic computation process. This method provides richer semantic information for models on text data.

3. Proposed Method

3.1. Overall Architecture of DCANet

DCANet is a neural network designed to better capture dependencies between words and sentences, as well as global semantic information. The overall structure of the model is depicted in Figure 3. Upon feeding the data into the model, the MPDM module processes it to capture multilevel semantic information across different information densities, which ultimately generates a text representation with abundant semantic details. In the ISE module, text representation is utilized to calculate compensatory information for the current input data under global semantic information. Subsequently, the SAT module combines the outputs of the MPDM module and the ISE module for fusion calculation, introducing the global context information learned by the latent code into the attention calculation process, enhancing the model’s accuracy in capturing the semantic information of the text. Then, the output of the SAT module is calculated through the output layer to obtain the predicted results of the input data. It should be noted that the ISE module is not designed as an independent prediction branch. Instead, it serves as an auxiliary latent-code generation mechanism that provides global semantic guidance for the SAT module.

3.2. Multi-Granular Parallel Diffusion Masking

To address the noise problem in cross-domain representation caused by uneven density of contextual information in different scenarios, we propose the Multi-granularity Parallel Diffusion Masking (MPDM) mechanism integrated with a stochastic graph diffusion strategy. For the given embedding dimension d and sequence length l of the input data matrix X, the MPDM leverages stochastic graph diffusion processes to generate a mask matrix $M=(m_1,m_2,\dots ,m_k)$ with the same dimension to perform a diffusion-based random mask on the input matrix X. Specifically, the parameters of the mask matrix M follow a Bernoulli distribution $m_i\sim Bernoulli(1-p)$, where $n=1$ indicates that the parameter is randomly sampled only once, and p represents the masking probability derived from the diffusion dynamics, and the p is derived from the diffusion forward process, defined as $p=1-{\beta }_t$, where ${\beta }_t$ represents the diffusion noise schedule that increases with the iteration step t, consistent with the noise addition rule of standard diffusion models. This makes the masking process a discrete approximation of the continuous stochastic diffusion process, where each mask step corresponds to a noise injection step in diffusion models, as shown in Equation (1).

$$X_t=X_{t-1}\odot M$$

(1)

The process of diffusion masking is shown in Figure 4. By introducing the mask matrix M, features can be extracted from data matrices with different information at different time steps. Compared to extracting features from the original data, extracting features from data matrices with different information levels enables us to capture more holistic information about the data and enhances the robustness of the model.

In order to efficiently obtain input data with varying degrees of masking at different time steps, this work adopts a synchronous processing approach to simultaneously apply masks with different diffusion intensities to the original data, as shown in Figure 4b. Adjust the mask matrix by setting the parameters $m_i$ in the mask matrix $M_t=(m_1,m_2,\dots ,m_k)$ to follow a binomial distribution $m_i\sim B(n,p_t)$, where the masking probability $p_t$ of parameter $m_i$ changes as the number of mask iterations increases, as shown in Equation (2).

$$p_t=\alpha ({\displaystyle \frac{T-t}{T}})+\beta$$

(2)

where $\alpha$ is a hyperparameter representing the threshold of the masking probability, $\beta$ is another hyperparameter typically set to a small value to prevent a masking probability of 0, T denotes the total number of mask iterations, and t denotes the current mask step. Furthermore, the linear masking probability dynamics employed in Equation (2) ensure that the masking strength evolves smoothly and steadily across iteration steps. This guarantees stable parallel extraction of multi-scale contextual features and prevents instability in feature learning caused by abrupt changes in probability. Meanwhile, the linear design is computationally efficient and introduces no additional complexity, which aligns well with the objective of balancing performance and efficiency in cross-domain semantic processing. In contrast, nonlinear schedules often lead to excessive masking in early steps or insufficient masking in later steps for text context fusion tasks. After this step, the process of obtaining the data matrix $X_t$ in step t is no longer dependent on the previous time step, as shown in Equation (3).

$$X_t=X{M}_t$$

(3)

On this basis, the intrinsic connection between the diffusion masking and the standard stochastic diffusion process is illustrated. The masking probability $p_t$ is defined by Equation (2), which decreases monotonically with the time step t and is consistent with the noise scheduling rule of the standard diffusion process. Taking the expectation of the diffusion masking operation yields $E\left[X_t\right]=p_{t}X$, and its variation trend matches that of the standard stochastic diffusion process, indicating that the multi-granularity parallel diffusion masking is a discretized approximation of the continuous diffusion process. The relation between MPDM and diffusion goes beyond a structural analogy. Specifically, the masking process defines a diffusion-inspired stochastic perturbation on semantic representations, and the resulting expectation dynamics $E\left[X_t\right]=p_{t}X$ provide a probabilistic correspondence to the standard forward diffusion process.

3.3. Implicit Semantic Encoder

We propose an Implicit Semantic Encoder (ISE) structure that utilizes latent code to guide feature computation, aiming to achieve a top-down, holistic-to-part understanding of objects in cognitive tasks from a global perspective. Such an implicit modeling paradigm aligns with the principles of continuous verification in zero-trust architectures [34], because implicit feature management can be more effective than explicit, static methods for capturing complex distributions. To achieve this goal, this paper adopts a layer weight sharing method, in which the weights of the linear layer used for feature inference in final layers of the model are shared with the linear layer used to guide model feature computation with latent code. Similar to models with an RNN-like reusable network structure, our model accumulates input data at different time steps using the same network structure to obtain an output vector that represents the semantic information of all input data. The layer weight sharing enables the introduction of information that describes the global context of the task and represents the overall feature distribution space of the data from final layers to the linear layers in early part of the model, allowing for a holistic perspective in data computation during feature extraction. It is important to note that although the weights of the two linear layers are the same, their biases are independent, thus allowing the two linear layers to be considered as separate layers.

The process of obtaining the latent code of the model through weight sharing layers is as follows: Activate the global hidden context information $y^{\prime }$ represented by the weight sharing layer using a fully filled matrix $X^{same}$ the same dimension as the data matrix, as shown in Equation (4), where $W_f$ denotes the shared weights from the weight sharing linear layer, $X^{same}$ represents a fully populated matrix of the same size and shape as the input data X, and b represents the bias term. Here, $X_{same}\in {\mathbb{R}}^{d\times l}$ is a predefined task-agnostic global feature matrix, whose elements follow a standard normal distribution $N(0,1)$ with orthogonal normalization and remain fixed during training to capture the invariant semantic distribution of the dataset.

$$y^{\prime }=W_f{X}^{same}+b$$

(4)

After obtaining the global hidden contextual information $y^{\prime }$, the difference between $y^{\prime }$ and the elements of the data matrix X is calculated. This difference represents the semantic gap between the global prior and the local sample features, through which the compensatory information, latent code, is calculated for the input data under the global hidden context. The computation process is illustrated in Equation (5).

$$Latentcode=softmax(y^{\prime }-normalize\left(x\right))$$

(5)

It should be emphasized that, in the proposed framework, the discrepancy between the global hidden semantic reference $y^{\prime }$ and the normalized input representation is not introduced as a standalone universal benchmark metric for semantic noise. Instead, it serves as an operational characterization of representation-level semantic interference within the model. In this sense, the latent code in Equation (5) can be interpreted as an adaptive correction signal derived from the discrepancy between local input semantics and global task-relevant semantic guidance. This design allows the model to compensate for residual semantic interference during downstream semantic computation, rather than treating such interference as a dataset-level shift.

In order to align the values of the input data X and the global contextual information $y^{\prime }$ to the same dimension, normalization is applied to the input data X to align $y^{\prime }$. We then compute the divergence between $y^{\prime }$ and the normalized X. Subsequently, a Softmax operation transforms this difference into a latent code in the form of a probability distribution, characterizing the probabilistic intensity of compensatory correction required for each feature dimension, a latent code is derived to direct the attention computation over X.

3.4. Self-Correcting Attention Topology

In the previous discussion, we propose a method for obtaining global hidden semantic information through weight sharing structures, as well as the corresponding latent code derived from the combination of global hidden semantic information and the data matrix. To facilitate the use of the latent code for guiding the model’s attention computation on the data matrix, we propose the Self-Correcting Attention Topology (SAT) structure as shown in Figure 5.

In contrast to the self-attention calculation method, the SAT module takes two main inputs: the original input data X and the compensatory information latent code. The original input data X is transformed by a trainable weight matrix to obtain $Q_X$, which participates in the attention computation. Similarly, the latent code is transformed by two trainable weight matrices to obtain $K_l$ and $V_l$, which also contribute to the attention computation process. The calculation method of SAT module is given by Equations (6) and (7). Where $F_a$ denotes the attention score function, and d represents the dimension of the query vector $Q_X$. The output of the SAT module is obtained by combining the information after attention computation with the original input data X and passing it through a linear layer. Different from standard attention, the closed-loop self-correcting mechanism of the SAT module is formally established on the basis of iterative interaction between global semantic priors and local attention computation. Given the local feature input X and the initial global latent encoding $L_0$ generated by the ISE module, the mechanism performs cyclic updates in a closed-loop manner: the attention output dynamically optimizes the global latent encoding, and the updated latent encoding in turn guides the subsequent attention weight calculation. Formally, at each iteration step t, the latent encoding is updated as $L_{t+1}=\mathrm{Norm}(L_t+\gamma ·A_t)$, where $A_t$ denotes the attention output at step t and $\gamma$ is a learnable feedback coefficient. The updated latent encoding $L_{t+1}$ is then used to generate new $K_l^{(t+1)}$ and $V_l^{(t+1)}$ for the next round of attention computation, and this iterative process continues until convergence. After T rounds of closed-loop iteration, the final attention result $A_T$ is output. Through such multi-round closed-loop iteration, the module achieves adaptive alignment between global semantic guidance and local feature extraction, forming a self-correcting closed-loop system.

$$F_a(K_l,Q_X)={\displaystyle \frac{Q_X{K}_l^T}{\sqrt{d}}}$$

(6)

$$A(X,L)=softmax\left(F_a(K_l,Q_X)\right)V_l$$

(7)

4. Experiment

4.1. Dataset

Experiments were conducted on nine datasets in total: five for main comparison (

Table 1. Introduction to five publicly datasets.

Dataset	Classes	Train	Test	Maximum Sequence Length
AG News	4	120k	7.6k	135
Yelp P.	2	560k	38k	1104
Yelp F.	5	650k	50k	1175
Yah.A.	10	1400k	60k	3998
DBP.	14	560k	70k	1302

) and four for ablation studies (

Table 2. Introduction to ablation experiment dataset.

Dataset	Language	Classes	Total Numbers	Positive Samples	Negative Samples
hotel review unbalanced	Chinese	2	7766	5322	2444
hotel review balanced	Chinese	2	6000	3000	3000
Weibo review	Chinese	2	119,988	59,993	59,995
IMDB movie reviews	English	2	50,000	25,000	25,000

). To assess the effectiveness of the model, this study conducted experiments on five publicly available datasets, namely, AG News, Yelp Full Review (Yelp F.), Yelp Polarity Review (Yelp P.), DBPedia (DBP.), and Yahoo! Answers (Yah.A.). A detailed description of these datasets is provided in Table 1.

The data supporting the findings of this study is publicly available in the Torchtext library. Torchtext is a Python library for natural language processing that provides the ability to load text data and download corresponding data through the name of the dataset.

4.2. Baseline Models

We select five state-of-the-art baseline models for comparison with our DCANet.

LBCNN [9]: LBCNN, a label-based convolutional neural network, can capture the importance of individual words in text sequences based on labels. Additionally, it identifies the most influential semantic features within word vectors.

CWC [10]: The model applies capsule networks to relationship modeling between word embeddings and introduces a novel routing algorithm based on k-means clustering theory to fully explore the relationships among word embeddings.

SLCNN [13]: SLCNN represents documents as three-dimensional tensors within the network, allowing for the comprehensive utilization of positional information in text sentences. This design enables the model to extract additional features by analyzing neighboring sentences.

LEAM [35]: This model introduces an attention framework to measure the compatibility between text sequences and labels, thereby facilitating the assessment of embedding compatibility.

DeBERTa-v3 [36]: This model improves pre-training efficiency and downstream language understanding performance through ELECTRA-style replaced token detection and gradient-disentangled embedding sharing.

4.3. Parameter Settings

In the experimental model, the learning rate was set to 0.0001, the batch size was set to 128, the hidden size was set to 300, the Dropout was set to 0.5, and the number of training epochs was set to 30. The dimension of word embeddings was set to 300. The kernel size of all involved CNNs was set to 3. Word embeddings are initialized with Bert-base-chinese, and jieba is used for the segmentation of all Chinese datasets. Linear layers in MPDM adopt a 2-layer stack with ReLU activation and Layer Normalization hidden dim = 300; the weight-sharing linear layer in ISE uses GELU activation and Batch Normalization hidden dim = 300, all aligned with the word embedding dimension. The Adam method was used as the optimizer for the model. Additionally, most experiments were conducted on an NVIDIA GeForce RTX 2080 GPU platform, which has 8 GB of memory.

4.4. Experimental Results and Analysis

To validate the effectiveness of the proposed DCANet, the model was evaluated on five benchmark datasets, and comparative experiments were conducted against five baseline models. The classification accuracies of the different models on the five datasets are reported in

Table 3. Classification accuracy on five standard benchmarks.

Model	AG News (%)	Yelp P. (%)	Yelp F. (%)	Yah.A. (%)	DBP. (%)
LEAM	92.45	95.31	64.09	77.42	99.02
LBCNN	92.90	95.82	64.38	74.89	99.21
CWC	92.39	96.48	65.85	73.85	98.72
SLCNN	91.26	96.01	64.46	70.88	98.76
DeBERTa-v3	95.31	97.01	66.93	78.00	99.29
DCANet (Ours)	93.86	97.03	67.48	77.30	99.31

Bold text indicates the best performance in this column of metrics.

.

As shown in Table 3, after including the recent Transformer-based baseline DeBERTa-v3, DCANet still demonstrates strong competitiveness across the five benchmark datasets. In particular, DCANet achieves the highest accuracies on Yelp Polarity (97.03%), Yelp Full (67.48%), and DBPedia (99.31%), outperforming the strongest competing baseline by 0.02%, 0.55%, and 0.02%, respectively. Meanwhile, DeBERTa-v3 attains the best results on AG News and Yahoo! Answers, with accuracies of 95.31% and 78.00%, respectively. These results suggest that DCANet remains a competitive architecture and continues to show advantages on several benchmark datasets even when compared with a strong recent pre-trained Transformer-based model.

4.5. Ablation Experiments

In order to further verify the effectiveness of the proposed DCANet architecture, we conducted experimental comparisons between the DCANet model and models using CNN, RNN, and Attention architectures. At the same time, ablation experiments were conducted on the DCANet model to verify the effectiveness of each structure. The experiments conducted were conducted on four datasets, including the English IMDB film review dataset, the balanced Chinese hotel review dataset, the unbalanced Chinese hotel review dataset, and the Chinese Weibo review dataset. These datasets were chosen for their inclusion of both Chinese and English texts, the presence of balanced and imbalanced datasets, varying sizes of datasets, and the distinction between datasets with shorter average text lengths (e.g., hotel and Weibo reviews) and datasets with longer average text lengths (IMDB movie reviews with an average length of 294). For further details regarding the datasets, please refer to Table 2.

The hotel review dataset is compiled by Tan Songbo from Ctrip.com. The Weibo comment dataset comes from Sina Weibo, which is collected online. The IMDB movie review dataset can be downloaded from kaggle. Unlike the data source that comes with keras, kaggle’s IMDB dataset provides a CSV file which can better divide the training set, test set, and data preprocessing in the experiment.

The classification accuracies of the models in the ablation experiments on four datasets is shown in

Table 4. Experiment results of ablation experiment.

Model	Hotel Review Unbalanced (%)	Hotel Review Balanced (%)	IMDB (%)	Weibo Review (%)
RCNN	84.81	87.81	90.80	98.32
Transformer	84.94	88.17	93.60	97.69
TextCNN	84.04	88.31	89.20	97.92
LSTM	84.68	81.47	93.10	96.38
Bi-LSTM	80.79	87.47	91.10	97.81
BiLSTM-attention	83.27	88.65	93.80	98.31
DCANet-only-MPDM	80.03	80.59	90.56	94.30
DCANet-only-SAT	84.46	88.29	93.28	97.43
DCANet	85.46	89.37	94.61	98.80

Bold text indicates the best performance in this column of metrics.

. As shown, the proposed DCANet achieves superior results on these four datasets. Compared to baseline models, including TextCNN, LSTM, Bi-LSTM, BiLSTM-attention, RCNN, and Transformer, the DCANet shows superior performance on the unbalanced Chinese hotel review dataset. In contrast, it also demonstrates robust performance in Chinese hotel ratings, English IMDB movie reviews, and Chinese Weibo comments. The DCANet achieves the highest accuracies across all datasets, reaching 85.46%, 89.37%, 94.61%, and 98.80% respectively. In particular, the DCANet demonstrates the most remarkable improvement on the IMDB dataset. As shown in Table 4, the ablated DCANet variants with only the MPDM module or only the SAT module showed poor performance in various datasets. This indicates that compared to mature CNN, RNN, and attention based architectures, simple module design may not improve the performance. However, the DCANet used in combination with the MPDM and SAT modules performs better than these models, indicating that the model structure designed in this study has certain advantages in calculating text semantics.

It should be emphasized that the current ablation design follows the functional structure of the proposed model. The MPDM module can operate as an independent feature extraction branch, and therefore its isolated contribution can be examined directly. By contrast, the ISE module is not designed as an independently complete prediction branch; instead, it generates latent compensatory information that is used by the SAT module during attention computation. Therefore, the ablation study focuses on the independently meaningful MPDM and SAT branches. It should be noted that the current ablation study is intended to evaluate the effectiveness of the major functional modules of DCANet, rather than to provide a dedicated sensitivity analysis of the diffusion-related hyperparameters. In particular, although the number of diffusion iterations T is an important parameter in MPDM, its effect is directly coupled with $\alpha$ and $\beta$ through the masking probability formulation. Therefore, a meaningful analysis of these parameters would require a joint multi-parameter study rather than a simple one-factor comparison. For this reason, we do not include an additional sensitivity analysis of T, $\alpha$, and $\beta$ in the present work, and leave this issue for future investigation.

As shown in Figure 6, compared to baseline models that utilize CNN-based architectures to calculate textual semantic information, the DCANet achieves the best results on all four datasets. Among these datasets, the CNN-based models perform the best on the Weibo review dataset, which can be attributed to the shorter length of texts in this dataset. In comparison to the other datasets, the CNN-based models exhibit better performance in computing semantic information for short texts.

As shown in Figure 7, compared to baseline models that utilize RNN-based architectures for computing textual semantic information, the DCANet exhibits the highest accuracy improvement on the hotel review balanced dataset. The reason for this can be attributed to the fact that Chinese texts rely more on contextual semantic relationships compared to English texts. Additionally, the computational difficulty of capturing contextual semantic relationships in long texts is greater. Consequently, the performance of RNN-based models is slightly lower on the hotel review dataset compared to the Weibo review dataset. On the other hand, the DCANet’s diffusion mask method possesses a stronger capability to compute textual contextual information. Thus, it demonstrates a significant improvement in performance on this type of dataset.

As shown in Figure 8, compared to the baseline model that uses attention-based architecture to calculate text semantic information, the DCANet shows the highest accuracy improvement on the IMDB dataset. The reason for this is that compared to other datasets, the length of text in the IMDB dataset is longer, and compared to traditional attention architectures, the SAT method combining the DCANet with diffusion masking method has stronger ability to calculate text context information. Therefore, it significantly improves the performance of such datasets.

In the experiment, we examined the model’s attention to individual words within a certain range and the attention between words during the calculation process. The results are shown in Figure 9.

Figure 9a represents the attention computation of each word in the original attention calculation, while Figure 9b represents the attention computation of each word in the SAT calculation. Since attention is primarily used to modify the input information in the early computation process of the model, it can be observed from the figure that the modification method strengthens the attention on key words while not providing additional attention to non-key words. When combined with the subsequent neural network feature extraction process, the entire feature computation process involves selective feature extraction. This selective extraction continuously enhances key features while relatively suppressing non-key features. For practical datasets, computing key features allows for accurate classification of a large number of samples in the dataset. However, for a small number of samples with fewer key features, the model may not effectively process them. This study is based on the semantic space of the entire problem, using SAT to enhance the non-key features of the input data, thereby improving the feature extraction ability of the model. Our goal is to ensure effective representation and computation of key and non-key features.

Figure 10 illustrates the mutual attention between words in the traditional self-attention computation Figure 10a and the DCANet Figure 10b. In the traditional attention computation, the emphasis is on the attention relationship between key words in the input data. The attention relationship graph obtained through DCANet compensation reveals that attention effectively compensates for the attention between non-key words and key words. This demonstrates the effective guidance of global contextual information on attention computation and confirms the integration of global contextual information into the input data.

4.6. Computational Cost Analysis

While DCANet achieves strong predictive performance, its multi-branch diffusion masking and latent-code-guided attention may introduce additional computational overhead compared with a standard Transformer. To clarify this issue, we further compare DCANet with the Transformer baseline used in the ablation study on two representative datasets, namely IMDb and Weibo, which correspond to relatively longer-text and shorter-text settings, respectively.

For a fair comparison, the same training hyperparameters as in Section 4.3 were retained. The runtime and memory statistics were collected under PyTorch 2.3.0, Python 3.12 (Ubuntu 22.04), CUDA 12.1, one RTX 3080 Ti GPU with 12 GB memory, 12 vCPU Intel(R) Xeon(R) Silver 4214R CPU @ 2.40 GHz, and 90 GB system memory.

Table 5. Computational cost comparison between DCANet and the Transformer baseline on representative short- and long-text datasets.

Dataset	Model	Test Acc	Params (M)	Peak GPU (GB)	Epoch (s)	Test Infer (s)
IMDb	Transformer	0.9369	66.96	1.391	120.49	10.310
IMDb	DCANet	0.9461	72.94	1.640	152.80	12.60
Weibo	Transformer	0.9769	102.27	2.334	315.93	33.327
Weibo	DCANet	0.9880	116.18	2.89	421.35	42.52

reports the test accuracy together with several computational cost indicators, including the number of trainable parameters, peak GPU memory, training time per epoch, and full-test inference time. Compared with the Transformer baseline, DCANet improves the accuracy from 0.9369 to 0.9461 on IMDB and from 0.9769 to 0.9880 on Weibo. In return, DCANet increases the number of trainable parameters, peak GPU memory, and runtime on both datasets.

These results suggest that the proposed model incurs a moderate but practically manageable computational overhead while yielding consistent accuracy gains. This trade-off is reasonable because the additional cost mainly comes from the MPDM module, which performs parallel diffusion masking at multiple information densities, and the SAT module, which further integrates latent-code-guided attention.

5. Conclusions

We hereby propose a novel cross-domain representation learning method, designated as DCANet, which addresses the noise and challenges involved in pre-training word vectors on open-domain datasets and semantic computation. The cross-domain refers to the scenario where open-domain pre-trained embeddings are transferred to downstream tasks with different domain characteristics, and the model mitigates semantic noise in this process. A formal cross-domain transfer experimental setup will be explored in future work. The model employs the MPDM module, which generates a semantic graph with varying density through random path activation to capture potential contextual semantic information in the text at different semantic distances, thereby enhancing the fine-grained semantic information in the data. By integrating the SAT module and ISE module, the DCANet comprehensively models global semantic information and alleviates semantic biases caused by semantic noise during word vector pre-training on open-domain datasets. Consequently, the DCANet significantly enriches the semantic richness of the data and improves the accuracy of semantic computation. Rigorous experiments on nine publicly available benchmark datasets verify the superiority and practicality of the DCANet. The current study has two main limitations: the ablation analysis is primarily accuracy-based, and the experimental setting still entails non-negligible computational overhead. Future work will focus, firstly, on conducting a more systematic investigation into hyperparameter sensitivity and, second, on reducing experimental costs.