A Multimodal Fake News Detection Model Based on Bidirectional Semantic Enhancement and Adversarial Network Under Web3.0

Abstract

Web3.0 aims to foster a trustworthy environment enabling user trust and content verifiability. However, the proliferation of fake news undermines this trust and disrupts social ecosystems, making the effective alignment of visual-textual semantics and accurate content verification a pivotal challenge. Existing methods still struggle with deep cross-modal interaction and the adaptive calibration of discrepancies. To address this, we introduce the Bidirectional Semantic Enhancement and Adversarial Network (BSEAN). BSEAN first extracts features using large pre-trained models: a hybrid encoder for text and the Swin Transformer for images. It then employs a Bidirectional Modality Mapping Network, governed by cycle consistency, to achieve preliminary semantic alignment. Building on this, a Semantic Enhancement and Calibration Network explores inter-modal dependencies and quantifies semantic deviations to enhance discriminative capability. Finally, a Dual Adversarial Learning framework bolsters event generalization and representation consistency through adversarial training with event and modality discriminators. Experiments on public Weibo and Twitter datasets validate BSEAN’s superior performance across all metrics, demonstrating its efficacy in tackling the complex challenges of deep cross-modal interaction and dynamic modality calibration within Web3.0 social networks.

1. Introduction

The proliferation of social media since the advent of the Web 2.0 era has substantially facilitated information dissemination, transforming users from passive consumers into active content creators and distributors [1]. Social media platforms such as Weibo and Twitter, characterized by their immediacy, interactivity, and diverse content modalities (including text, images, and videos), have significantly expanded the scope and influence of information propagation. This convenience, however, has also enabled the rapid creation and widespread circulation of fake news. A sophisticated example is presented in Figure 1, which shows a social media post claiming a polar bear escaped during transport in Kazakhstan. As analyzed in Figure 1b, a stark semantic conflict exists between the textual narrative, which implies danger and urgency (“escaped”), and the visual scene, which depicts unnatural calmness. The polar bear’s relaxed, human-like posture and the police officers’ non-urgent, observational stance contradict the expected behavior of both the animal and the authorities in such a crisis. An effective detection system must possess a deeper reasoning capability to assess behavioral plausibility, situational context, and potential physical inconsistencies, which is the central challenge this paper aims to address.

Fake news [2] is formally defined as deliberately fabricated information intended to achieve specific objectives—such as generating web traffic, manipulating public opinion, or securing financial gain—and whose falsity can be verified by reliable external sources [3,4]. This type of information is often highly deceptive, propagates at an accelerated rate, and can affect a large user base within a short period [5]. In the Web3.0 era, the internet is evolving from an “Internet of Information” to an “Internet of Value.” This paradigm establishes an underlying infrastructure for trust and value transfer through blockchain technology. Its core principles are leveraged to enhance cybersecurity and data integrity, forming a foundational layer of trust for the new web [6], while cryptocurrencies and token economies facilitate the direct and efficient circulation of value, enabling direct incentives for producers (users). However, the rapid proliferation of generative AI within this new landscape has significantly lowered the barrier to entry for fabricating sophisticated fake videos, images, and text. Technologies such as deepfakes are capable of fundamentally disrupting human perception. In recent years, negative incidents stemming from fake news have become increasingly prevalent, both domestically and internationally. Such misinformation not only compromises the content security of social networks and erodes the bedrock of social trust but also undermines the credibility of governments and related industries, and can even foment social discord and disrupt critical socio-political processes [7,8,9]. A notable case in 2024 saw a Hong Kong-based company incur a fraudulent loss of HKD 200 million due to a deepfake video, highlighting the severe threat posed to personal and corporate financial security. Consequently, in the Web3.0 era, the development of more accurate multimodal fake news detection techniques is of paramount practical importance for sanitizing the online environment, bolstering social trust, and preserving the integrity of the information ecosystem, which requires not only verifying on-chain provenance but also scrutinizing the semantic consistency of the content itself.

Current research on fake news detection is predominantly centered on content-based approaches. These methods aim to extract discriminative features from the textual and visual elements of a news item to assess its authenticity. The quality and representativeness of these features are, thus, decisive factors in the final detection performance. Early research efforts were largely focused on the analysis of unimodal information. For textual content, deep learning models, particularly Recurrent Neural Networks (RNNs) and their variants, have been widely applied [10,11]. These models typically convert text into vector representations using pretrained word embeddings and then employ RNNs to capture contextual dependencies and deep semantic information within the text sequences for classification. In this context, large pretrained language models (PLMs) like BERT often function as powerful text feature extractors [12,13]. A common practice is to utilize the output of BERT’s final layer as the definitive textual representation. While straightforward, this approach may not fully exploit the hierarchical semantic information encoded in the model’s intermediate layers, which can be essential for a comprehensive understanding of complex or intentionally misleading narratives.

For visual content in news, unimodal detection methods centered on Convolutional Neural Networks (CNNs) have also been developed [14,15]. These approaches typically leverage CNN architectures (e.g., VGG) pretrained on large-scale image datasets to extract visual features [16], which are subsequently employed to ascertain image authenticity or its relevance to a specific event. Notably, the explicit features extracted by pretrained models, whether text- or image-based, can be influenced to some extent by the training data distribution. Specific types of fake news, such as those concerning sensitive domains or exhibiting particular emotional biases [17,18], may present certain distributional characteristics within datasets. If a model overrelies on these superficial, distribution-related features, its generalization capabilities when encountering novel, differently distributed data may be compromised.

Although unimodal methods have provided valuable insights and technical approaches for fake news detection within their respective domains, the evolution of information dissemination formats has seen purveyors of fake news increasingly employ multimodal strategies, such as combining text and images, to enhance deceptiveness and virality. They may exploit subtle inconsistencies between text and images, misleading cues from visual elements, or ambiguities arising from their combination to achieve deceptive ends. In such scenarios, relying solely on the analysis of a single modality struggles to comprehensively capture this cross-modal complexity and potential deceptiveness, thereby exposing the inherent limitations of unimodal detection methods.

To overcome the deficiencies of unimodal approaches, multimodal fake news detection has emerged, aiming to enhance detection accuracy and robustness by comprehensively analyzing multiple information sources, such as text and images. Although multimodal methods demonstrate superior potential compared to their unimodal counterparts, existing research still confronts several key challenges:

• Insufficient Cross-Modal Semantic Alignment and Deep Interaction: When processing features from different modalities [19,20], many multimodal models involve a fusion operation. However, simplistic concatenation or shallow interaction mechanisms are often insufficient to bridge the semantic gap between text and images and fail to adequately explore deep cross-modal semantic correlations. Even with the introduction of attention mechanisms [21,22], the efficacy of interaction remains constrained without effective initial alignment of features within a shared semantic space.
• Lack of Calibration Capability for Cross-Modal Semantic Discrepancies: In real-world multimodal news, varying degrees of semantic discrepancy, or even outright contradiction, can exist between text and images. Most current models lack a mechanism to dynamically assess and calibrate these discrepancies. Consequently, they cannot adaptively adjust the contribution of each modality based on the degree of visual-textual consistency, rendering them susceptible to interference from inconsistent or misleading information.
• Need for Enhanced Robustness and Generalization in Feature Representations: Fake news encompasses diverse event themes and manipulation tactics. If a model overfits to specific patterns prevalent in the training data, its generalization ability will be limited. This restricts its adaptability to varying event contexts and differences in modal information quality, and it hinders the learning of stable, discriminative features for identifying fake news.

To address the aforementioned challenges, this paper introduces the Bidirectional Semantic Enhancement and Adversarial Network (BSEAN), a novel model for multimodal fake news detection. BSEAN is designed to systematically enhance the model’s joint understanding of visual-textual information and its capability to identify inconsistencies through a multistage process. In the feature extraction stage, BSEAN employs the BERT pre-trained model to extract multi-level semantic features from text and integrates Text-CNN to enhance local short-text information. For the visual modality, the Swin Transformer pre-trained model is utilized to extract regional features from images.

The effective verification of consistency between textual and visual information is crucial in discerning multimodal fake news. To emulate the human cognitive process of cross-modal verification, which involves comparing content and identifying discrepancies, BSEAN operates as follows. First, the Bidirectional Modality Mapping Network establishes a preliminary alignment between textual and visual features in a shared semantic space via bidirectional mapping, aiming to capture richer cross-modal semantic correlations. Subsequently, the Semantic Enhancement and Calibration Network is employed. This network first deepens the model’s understanding through semantic interaction, then quantifies the deviation between modalities and dynamically recalibrates their contributions to mitigate potential conflicts. Furthermore, an adversarial learning strategy integrated within the Dual Adversarial Learning framework enhances the model’s adaptability across diverse event scenarios.

The main contributions of this paper can be summarized as follows:

• We designed a bidirectional modality mapping method that, by promoting cross-modal feature transformation, aims to achieve effective alignment of textual and visual features within a shared semantic space, providing a high-quality semantic foundation for subsequent interaction;
• We designed a unified Semantic Enhancement and Calibration Network (SECN). This network deeply explores cross-modal relationships through a semantic interaction mechanism and adaptively adjusts modal contributions via a deviation calibration strategy that dynamically adjusts modal contributions based on perceived semantic differences, enables a refined understanding of cross-modal information and robust adaptation to potential semantic conflicts, thereby enhancing the model’s ability to discern semantic inconsistencies;
• Through extensive comparative experiments and ablation studies against several state-of-the-art benchmark models on public Weibo and Twitter multimodal fake news datasets, the proposed BSEAN model demonstrates significant performance improvements across key evaluation metrics, validating its effectiveness in the task of multimodal fake news detection.

2. Related Work

This section systematically reviews the relevant research progress in the field of fake news detection, primarily delineating advancements from two perspectives: unimodal information processing and multimodal information analysis.

2.1. Unimodal Detection Methods

In the early era of the internet, news was predominantly presented in textual form; consequently, a substantial body of research focused on extracting effective cues from textual content to discern fake news. Prior to the application of deep learning techniques in this domain, researchers commonly relied on feature engineering methods [23]. These methods involved extracting statistical or linguistic features such as Term Frequency-Inverse Document Frequency (TF-IDF), N-grams, and sentiment lexicon scores [24,25], and subsequently employing traditional machine learning classifiers like Support Vector Machines (SVMs), Logistic Regression, and Decision Trees for veracity assessment. While these methods achieved a degree of success, the feature extraction process was time-consuming, labor-intensive, and heavily reliant on domain expertise, often failing to capture the deeper semantic information within texts.

The advent of deep learning precipitated a paradigm shift. Ma et al. [10] pioneered the application of Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs), to the task of fake news detection. These models demonstrated the ability to automatically learn context-dependent semantic representations from text sequences, significantly enhancing detection performance. Recognizing that veracity assessment often necessitates background knowledge extending beyond the text itself, some studies began to explore effective ways to incorporate external knowledge into models. For instance, Dun et al. [26] linked entities within news text to knowledge graphs and utilized attention mechanisms to fuse contextually relevant information associated with these entities, thereby augmenting the model’s comprehension capabilities. Xu et al. [27] focused on the internal structure of evidence texts, employing graph neural networks to capture long-range dependencies between evidence snippets and, thus, identify and eliminate redundant or irrelevant evidential information. Liao et al. [28] designed a multi-step evidence retrieval and enhancement framework that simulates human online fact-checking behaviors, iteratively retrieving and filtering relevant evidence to aid judgment. More recently, following the remarkable success of Large Language Models (LLMs), research has also begun to investigate leveraging their powerful text understanding and knowledge reasoning capabilities to improve the accuracy of fake news detection [29].

Concurrently, given the increasingly significant role of images in the propagation of fake news—for instance, through tampering, splicing, or association with misleading text—image-based fake news detection methods have also garnered attention. Such approaches typically address the problem from two perspectives: first, detecting traces of manipulation within the image itself, and second, analyzing whether the semantic content of the image aligns with common sense or the accompanying textual descriptions. For example, Qi et al. [30] proposed a method combining spatial domain features for semantic understanding with frequency domain features for detecting tampering traces. Other works directly utilize Convolutional Neural Networks (CNNs), such as VGG and ResNet, pre-trained on large-scale image datasets, to extract deep visual features for subsequent classification tasks [31]. However, relying solely on unimodal information—be it textual or visual—for fake news detection inherently faces limitations. Perpetrators of fake news often skillfully exploit the combination and manipulation of information across different modalities, making it difficult for single-modality analysis to reveal the inherent deceptiveness.

2.2. Multimodal Detection Methods

With the increasing prevalence of multimodal content such as images and text on social media, leveraging multi-modal information for fake news detection has become a mainstream research trend and an unavoidable necessity. Compared to unimodal approaches, multimodal methods can integrate cues from diverse information sources, analyzing news content from a more comprehensive perspective and, thus, possessing the potential to more accurately capture complementary information and potential inconsistencies across modalities.

Early multimodal methods typically employed relatively direct feature-level fusion strategies. Specifically, they would first independently extract features using unimodal feature extractors [32,33]—such as RNN/BERT for text and CNNs for images—and then merge these heterogeneous features into a unified representation through simple operations like feature concatenation, element-wise addition, or averaging, before finally feeding this representation into a classifier for judgment. For instance, Wang et al. [16] used Text-CNN to extract textual features and, after concatenating them with image features, also introduced an event discrimination sub-task to learn shared feature representations across events. The work by Singhal et al. [34] demonstrated that employing powerful pre-trained language models (e.g., BERT) as text encoders can significantly enhance detection performance due to their rich prior knowledge, even without relying on complex fusion mechanisms or auxiliary tasks. However, such simplistic “early fusion” or “late fusion” strategies often struggle to fully exploit the complex intrinsic dependencies and subtle semantic interactions between features from different modalities. This can lead to information redundancy or allow noise from one modality to adversely affect the overall judgment, thereby limiting further improvements in model performance.

To overcome the limitations of simple fusion strategies, researchers began to explore more sophisticated cross-modal interaction mechanisms, among which the attention mechanism has gained widespread favor due to its ability to dynamically assign weights to different information segments and capture long-range dependencies. The core idea is to allow information from one modality, such as keywords in text, to guide the model’s focus towards specific, relevant regions or aspects in another modality, like an image, thereby achieving selective enhancement and alignment of cross-modal information. For example, Jin et al. [35] proposed an RNN model incorporating an attention mechanism, which, after fusing textual and social context features, utilized linear attention to enhance the representation of image features. Wu et al. [36] drew inspiration from human reading habits to design a multimodal cross-attention network that simulates the process of humans alternately attending to text and images, learning cross-modal dependencies via a cross-attention mechanism. Qi et al. [37] introduced an attention-based multimodal fusion network aimed at facilitating higher-level information ex-change at both the lexical and visual object levels. These methods, through various forms of attention mechanisms, have indeed promoted cross-modal information interaction and fusion to a certain extent.

Simultaneously, considering that a common characteristic of fake news is the inconsistency or weak correlation between textual descriptions and accompanying visual content, many research efforts have framed the fake news detection problem as a task of measuring and judging visual-textual consistency. The core of such methods lies in assessing whether the information conveyed by the textual and visual modalities points to the same fact or event. For instance, Shang et al. [38] constructed an object-aware visual encoder to extract salient object features from news images and utilized principles from Generative Adversarial Networks (GANs) to evaluate the consistency between visual and textual features, thereby determining if the image matched the textual content. Xue et al. [39] judged consistency by calculating the similarity between overall semantic representations of images and text. Zhou et al. [40] also identified potential inconsistencies by comparing the differences between latent features extracted from images and text. These approaches directly address cross-modal semantic contradictions or deviations and exhibit good discriminative power for certain types of fake news.

Building on attention mechanisms, the Transformer architecture has advanced multimodal fusion by capturing long-range dependencies and fine-grained cross-modal interactions via self-attention, which dynamically weights contextual information across modalities—critical for social media data with complex text–image–user interaction relationships. Recent work on attention-guided Transformers further enhances such capabilities: Selvam et al. [41] leveraged a unified framework integrating Transformer-based NLP and graph neural networks to process multimodal features (textual semantics and social network structures) for fake account identification, demonstrating the effective alignment of heterogeneous information and improved discriminative power for deceptive patterns in social media. These models represent a shift from heuristic fusion to end-to-end hierarchical attention modeling, strengthening adaptability to complex fake content scenarios.

Despite the significant advancements in multimodal fake news detection, existing research still confronts several critical challenges in achieving ideal detection performance. Firstly, effective cross-modal semantic alignment and deep interaction remain a core technical bottleneck. Although many methods employ attention mechanisms, if features from different modalities are not effectively represented in a shared or well-aligned semantic space prior to interaction, the efficacy of subsequent interactions is often constrained, making it difficult to truly capture deep, fine-grained semantic correlations and subtle inconsistencies between modalities. Secondly, current models generally lack the capability for adaptive calibration of cross-modal semantic discrepancies. In real-world news data, the semantic relationship between text and images is complex and varied, potentially exhibiting partial consistency, complete inconsistency, or even deliberately fabricated contradictions. Most models struggle to dynamically and adaptively adjust the contribution weights of each modality in the final decision-making process based on the degree of visual-textual consistency in specific samples, rendering them susceptible to interference from noise or misleading information. Furthermore, the generalization ability of models and their robustness to specific event themes and modalities require further improvement to ensure stable detection performance when faced with diverse and evolving fake news.While our approach leverages established components such as pre-trained encoders and cross-attention, its primary novelty lies in their synergistic integration into a multi-stage verification framework designed to emulate human-like reasoning. Unlike prior works that often rely on direct fusion or shallow interaction, BSEAN introduces a dedicated network to explicitly model and adaptively calibrate the semantic discrepancies between modalities. The specifics of this calibration mechanism, which builds upon and refines earlier alignment and gating concepts, will be detailed in the subsequent sections.

3. Methodology

To address the challenges delineated above, the BSEAN model proposed in this paper aims to construct a more comprehensive framework for multimodal fake news detection. Initially, the model leverages large-scale pre-trained models for multimodal feature extraction. Next, a Bidirectional Modality Mapping Network (BMMN) utilizes cycle-consistency constraints to achieve robust cross-modal feature alignment and mutual mapping, thereby establishing a foundational basis for subsequent deep interaction. Building upon this, we introduce the Semantic Enhancement and Calibration Network (SECN) to facilitate deep interaction and adaptive fusion. The SECN operates in two stages. First, the Semantic Interaction Enhancement (SIE) stage uses self-attention and bidirectional cross-attention mechanisms for deep visual-textual semantic interaction. Second, the Semantic Deviation Calibration (SDC) stage adaptively adjusts the contribution of each modality based on their semantic deviation, effectively handling potential inconsistencies. Finally, a Dual Adversarial Learning (DAL) framework is integrated to enhance the model’s event generalization capability and the consistency of modal representations, thus offering a more holistic and generalizable solution for multimodal fake news detection. The overall architecture of the proposed BSEAN model is illustrated in Figure 2. This section will provide a detailed description of the model from the following aspects.

3.1. Feature Extraction

Multimodal news encompasses both textual and visual modalities. To comprehensively capture the deep semantic information within text, this paper employs a text encoding method that combines a pre-trained language model with local feature enhancement. Specifically, the BERT pre-trained model is used to extract three hierarchical levels of features, which are then augmented by Text-CNN to enhance local short-text features. For the image encoder, the Swin Transformer [42] pre-trained model, renowned for its out-standing performance in visual tasks, is utilized.

3.1.1. Text Encoder

Firstly, multi-level feature extraction is performed using the $BERT$ model. Given an input text $C=\{c_1,c_2,\dots ,c_n\}$, the 12-layer encoder of the BERT model is employed to obtain semantic features at different levels of abstraction:

$$f_{\theta }\left(C\right)=\{t_1,t_2,\dots ,t_n\}=BERT\left(C\right)$$

(1)

Let $t_i\in R^{d_t}$ represent the embedding vector for the $i-th$ word, where $d_t$ is the embedding dimension. To integrate shallow syntactic and deep semantic information, the outputs from BERT’s layers are grouped and summed to generate three sets of multi-level features, corresponding to shallow, intermediate, and deep features, respectively:

$$t_i^0=\sum _{j=1}^4{f}_{\theta }{\left(C\right)}_{ji},t_i^1=\sum _{j=5}^8{f}_{\theta }{\left(C\right)}_{ji},t_i^2=\sum _{j=9}^{12}{f}_{\theta }{\left(C\right)}_{ji}$$

(2)

Subsequently, for each feature set $t^k\in R^{n\times d_t}$, multi-scale convolutional kernels are applied to extract local semantics, and max-pooling is used to aggregate key information: $h_k=MaxPool\left(ReLU\left(Conv(t^k,W_k)\right)\right)$. Here, $W_k$ represents convolutional kernels of varying window sizes. Finally, these multi-level features are fused through a concatenation operation to form the final textual representation:

$$F_t=Concat(h_0,h_1,h_2)$$

(3)

3.1.2. Image Encoder

The Swin Transformer adopted in this paper is an efficient architecture for visual tasks. By introducing hierarchical feature maps and a shifted window multi-head self-attention mechanism, it not only ensures the capture of global semantics but also preserves detailed information while reducing computational complexity. An input image I is encoded by the Swin-T, outputting an image region feature matrix: $V=Swin-T\left(I\right)\in R^{M\times d_v}$, where M is the number of image regions and $d_v$ is the feature dimension. To match the textual feature space, a fully connected layer is employed to project the visual features into a unified dimension d, thereby achieving effective alignment with the textual features. The finally obtained projected features are denoted as $F_v\in R^{d_v}$.

3.2. Bidirectional Modality Mapping Network

To achieve efficient knowledge transfer between textual and visual modalities, we designed a Bidirectional Modality Mapping Network (BMMN). This network employs a multi-head attention mechanism to realize bidirectional mapping between modalities. Concurrently, a cycle consistency constraint strategy is introduced to ensure that the mapped features maintain semantic consistency and stability. This module is illustrated in Figure 3.

3.2.1. Cross-Modal Bidirectional Feature Mapping

Given the textual features $F_t$ and visual features $F_v$, cross-modal mapping is achieved through Text-to-Visual (T2V) and Visual-to-Text (V2T) transformations. For T2V mapping, the textual features $F_t$ serve as the Query, while the visual features $F_v$ act as the Key and Value. This is computed as follows:

$$A_{t\to v}^u=tanh\left({\displaystyle \frac{\left[Q_t^u\right]{\left[K_v^u\right]}^T}{\sqrt{d/h}}}\right)$$

(4)

$$F_{t\to v}=\mathrm{T}2\mathrm{V}\left(F_t\right)\triangleq Concat(\left[H_t^1\right],\dots ,\left[H_t^h\right]),\left[H_t^u\right]=A_{t\to v}^u\left[V_v^u\right]$$

(5)

where h is the number of attention heads, and $\left[Q_t^u\right]$,$\left[K_v^u\right]$,$\left[V_v^u\right]$ represent the linearly transformed feature matrices. Similarly, the Visual-to-Text (V2T) mapping process generates $F_{v\to t}$. Finally, the dimensions of the mapped features are aligned through normalization.

3.2.2. Cycle Consistency and Regularization Optimization

To ensure that the mapped features retain the semantic information of their original modalities, a cycle consistency constraint is introduced. This involves reconstructing the features $F_{t\to v}$ and $F_{v\to t}$ back to their original modalities, yielding ${\widehat{F}}_t=\mathrm{V}2\mathrm{T}\left(F_{t\to v}\right),$${\widehat{F}}_v=\mathrm{T}2\mathrm{V}\left(F_{v\to t}\right)$, respectively. The cycle consistency loss encourages consistency between the features before and after mapping. Furthermore, to prevent attention over-fitting, entropy regularization is applied to the attention matrices, promoting diversity in the attention distribution. The influence of this regularization is controlled by corresponding hyperparameters: ${\lambda }_{vt}$,${\lambda }_{tv}$,${\lambda }_h$. The overall loss for this module is formulated as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{BMMN}}& ={\lambda }_{vt}·\mathrm{MSE}({\widehat{F}}_t,F_t)+{\lambda }_{tv}·\mathrm{MSE}({\widehat{F}}_v,F_v)+{\lambda }_h·\hfill \\ \hfill & \left({\displaystyle \frac{1}{h}}\sum _{u=1}^{h}H\left(A_{t\to v}^u\right)+{\displaystyle \frac{1}{h}}\sum _{u=1}^{h}H\left(A_{v\to t}^u\right)\right)+{\lambda }_{tv}·\mathrm{MSE}(F_{t\to v},F_{v\to t})\hfill \end{array}$$

(6)

3.3. Semantic Enhancement and Calibration Network

To capture cross-modal semantic discrepancies, enhance local semantic features between text and images, and optimize feature representations, we have designed a cross-modal semantic information enhancement network, the structure of which is illustrated in Figure 4. This network progresses from an information fusion stage within a self-attention sub-layer to a cross-attention sub-layer, yielding cross-modally semantically enhanced features. These features are then calibrated based on semantic discrepancies to produce the final fused representation.

3.3.1. Semantic Interaction Enhancement

This module aims to capture intra-modal semantic relationships and cross-modal interaction information through self-attention and cross-attention mechanisms. Its structure, as depicted on the left side of Figure 4 (the attention mechanism module), involves mapping an input feature $D_1$ to the Q (Query) space, and another input feature $D_2$ to the K (Key) and V (Value) spaces. These are then propagated forward through a multi-head attention mechanism and a feed-forward neural network, with each component connected via a residual link and a normalization module. The process can be represented as

$$\begin{array}{c}Q=W^Q{D}_1,K=W^K{D}_2,V=W^V{D}_2\\ D_1^{*}=\mathrm{Norm}(MH(Q,K,V)+D_1)\\ D_{Att}=\mathrm{Norm}(FNN\left(D_1^{*}\right)+D_1^{*})\end{array}$$

(7)

where $MH$ denotes the multi-head attention operation, $FNN$ represents the feed-forward neural network, and $D_{Att}$ is the output of this module.

Prior to cross-modal interaction, semantic information from different perspectives is first integrated by fusing the original features with their corresponding mapped features. The textual branch comprises the original textual features $F_t$ and the cross-modally mapped features $F_{t\to v}$. Similarly, the visual branch consists of the original visual features $F_v$ and the cross-modally mapped features $F_{v\to t}$. The process is as follows:

$$\begin{array}{cc}\hfill T_C& =Concat(F_t,F_{t\to v})\hfill \\ \hfill V_C& =Concat(F_v,F_{v\to t})\hfill \end{array}$$

(8)

where the sequence length of the concatenated features is the sum of the sequence lengths of the individual features.

Subsequently, $T_C$ and $V_C$ are fed into separate self-attention modules, where intra-modal contextual enhancement is performed via the self-attention mechanism:

$$\begin{array}{cc}\hfill T_{CA}& ={\mathrm{SelfAtt}}_T\left(T_C\right)\hfill \\ \hfill V_{CA}& ={\mathrm{SelfAtt}}_V\left(V_C\right)\hfill \end{array}$$

(9)

To achieve deep cross-modal information learning, a cross-attention layer is employed to strengthen the model’s profound understanding and alignment of local and global semantics between visual and textual features. Specifically, for the textual branch, its self-attention enhanced features $T_{CA}$ are used as the Query to attend to the visual semantic information contained within the enhanced features of the visual branch $V_{CA}$ (serving as Key and Value). This generates a more comprehensive textual representation infused with visual context. Under the guidance of visual features, the textual features learn and integrate relevant visual scene and object information. Symmetrically, the same process occurs in the visual branch: visual features, guided by textual features, learn and associate with relevant descriptive or conceptual textual information. Through this bidirectional, mutually guided cross-attention mechanism, the model can effectively capture and model complex correspondences and complementary information between images and text. This process is as follows:

$$\begin{array}{c}\hfill T^{*}=CrossAtt{t}_{V\to T}(T_{CA},V_{CA})\\ \hfill V^{*}=CrossAtt{t}_{T\to V}(V_{CA},T_{CA})\end{array}$$

(10)

3.3.2. Semantic Deviation Calibration

To mitigate inherent semantic differences between modalities, incomplete informational correspondence, or even deliberately fabricated semantic contradictions, this paper introduces a Semantic Deviation Calibration (SDC) module. This module adaptively evaluates, adjusts, and calibrates the multimodal features obtained after deep interaction, based on perceived semantic deviations.

The SDC module first needs to assess whether a semantic deviation exists between the enhanced textual features $T^{*}$ and the enhanced visual features $V^{*}$. We quantify this deviation by calculating the cosine distance between them:

$$\delta =1-Cosine(T^{*},V^{*})$$

(11)

Based on the quantified semantic deviation $\delta$, the model learns to generate gating weights $g_t$ and $g_v$ for the textual and visual modalities, respectively. These weights dynamically adjust the contribution of each modal feature in the subsequent fusion process according to the degree of deviation. Specifically, modal information exhibiting greater deviation is appropriately suppressed, while information with higher consistency is preserved or enhanced, thereby achieving adaptive feature calibration:

$$g_m=\sigma (W_{c,m}·\delta +b_{c,m}),m\in \{t,v\}$$

(12)

$$\widehat{T}=g_t·T^{*},\widehat{V}=g_v·V^{*}$$

(13)

Finally, the adaptively calibrated textual features $\widehat{T}$ and visual features $\widehat{V}$ are deeply fused under the explicit guidance of the gating weights, which are dynamically generated based on the semantic deviation, to produce a unified multimodal representation M.

3.4. Dual Adversarial Learning

To enhance the model’s discriminative capability concerning event relevance and modality consistency, we construct an adversarial learning framework between a fake news discriminator and an event-modality consistency discriminator. Firstly, the fake news discriminator incorporates a Softmax layer following a fully connected layer to distinguish between true and fake news. Subsequently, the event-modality consistency discriminator engages in adversarial training through a Gradient Reversal Layer (GRL) to make the event features and modality features indistinguishable by their respective discriminators. The fake news discriminator and its loss function are represented as

$$P=Softmax(W\ast M+b)$$

(14)

$${\mathcal{L}}_r=-\left[y_{r}log{P}_r+(1-y_r)log(1-P_r)\right]$$

(15)

where $P_r$ is the predicted probability of the news being “true”, and $y_r$ is the true label. Similarly, the loss functions for the event discriminator and the modality consistency discriminator are denoted as ${\mathcal{L}}_e$ and ${\mathcal{L}}_m$, respectively. With the introduction of the GRL, the feature extractor is trained to become insensitive to event labels, thereby compelling the event discriminator to generate feature distributions that are indistinguishable across different event categories. This aims to mitigate the model’s reliance on specific event scenarios. Concurrently, the modality consistency discriminator aligns the cross-modal distributions of textual and visual features in the latent space, thereby suppressing modal bias. The final objective function is formulated as

$${\mathcal{L}}_{final}=L_r+\lambda L_e+\gamma L_m$$

(16)

where the parameters $\lambda$ and $\gamma$ are used to balance the adversarial losses from the event and modality consistency tasks with the primary fake news detection task, and take negative values. During adversarial training, the fake news discriminator aims to minimize its own loss, while simultaneously maximizing the losses of the event and modality consistency discriminators.

4. Experiments

4.1. Datasets

To validate the performance of the BSEAN model, this paper utilizes two datasets collected from different social media platforms: Weibo and Twitter.

The Weibo dataset, originally collected and curated by Jin et al. [35], comprises confirmed fake news items verified by Sina Weibo’s official rumor debunking platform between May 2012 and January 2016. These news items primarily originated from posts by ordinary users and were authenticated by a panel of experts. The authentic news items in this dataset were sourced from official reports by Xinhua News Agency, an authoritative Chinese news organization.

The Twitter dataset, constructed by Boididou et al. [43], is frequently employed for multimedia information credibility verification tasks on platforms like MediaEval. It focuses on identifying fake posts on social media and includes a development set for model training (approximately 5000 news items, covering over 10 rumor events) and a test set for evaluation (approximately 2000 news items, encompassing over 30 rumor events).

During the data preprocessing phase, to ensure the rigor of the experiments and the quality of the data, we removed text-only posts, duplicate images, data with broken links, and excessively low-quality images from both datasets. Textual content underwent standardization, including the removal of punctuation, numbers, special characters, and stopwords or overly short words. Through these preprocessing steps, we obtained refined visual-textual datasets suitable for training and evaluating our model.

4.2. Experimental Setup

In the feature extraction stage, we employed a pre-trained BERT model to extract textual features. Its 12-layer outputs were integrated into three sets of multi-level features. These were subsequently processed by a Text-CNN network containing three kernel sizes, 3, 4, and 5 (each with 128 filters), ultimately generating a 256-dimensional textual representation. For the visual modality, a pre-trained Swin-T model was used to process input images, and the output regional features were projected to 256 dimensions via a fully connected layer.

Regarding the model architecture, the multi-head attention mechanisms in all modules were configured with eight attention heads, each having a dimension of 32. For the key hyperparameters in our proposed modules, we set the loss weights in the Bidirectional Modality Mapping Network (BMMN) to ${\lambda }_{vt}=2.0$ and ${\lambda }_{tv}=1.0$, with an entropy regularization weight of ${\lambda }_h=0.1$. In the DAL framework, the weights for the event and modality adversarial losses were set to $\lambda =-0.1$ and $\gamma =-0.1$, respectively. During the training process, we utilized the AdamW optimizer with an initial learning rate set to $2\times {10}^{-5}$ and a batch size of 64. The model was trained for a maximum of 100 epochs. To prevent overfitting and select the best model, we incorporated an early stopping mechanism with a patience of 10 epochs, monitoring the F1-score on the validation set. All experiments were implemented using the PyTorch 2.3.0 framework and conducted on NVIDIA GeForce RTX 3090 GPU with 24 GB of memory.

4.3. Evaluation Metrics

To comprehensively evaluate the proposed method, the experiments utilized Accuracy, Precision, Recall, and F1-score to assess the model’s performance. These metrics were also computed separately for true news and fake news.

$$\begin{array}{c}\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\end{array}$$

(17)

$$\begin{array}{c}\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}\end{array}$$

(18)

$$\begin{array}{c}\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}\end{array}$$

(19)

$$\begin{array}{c}\mathrm{F}1={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}\end{array}$$

(20)

where $TP$ (true positives) is the number of fake news items correctly classified as fake; $TN$ (true negatives) is the number of true news items correctly classified as true; $FP$ (false positives) is the number of true news items incorrectly classified as fake; and $FN$ (false negatives) is the number of fake news items incorrectly classified as true.

4.4. Comparative Experiments

To validate the effectiveness of the proposed BSEAN model, this paper compares it with several classic model approaches in the field of fake news detection. The baseline models include the following:

Visual-Only: employs only the pre-trained Swin Transformer model to extract image features, followed by a simple classification.

Text-Only: utilizes only the pre-trained BERT model and Text-CNN to extract textual features, followed by a simple classification.

CLIP [44]: leverages the powerful visual and text encoders of CLIP to extract deep semantic features from the image and text, respectively. Subsequently, the features from both modalities are concatenated and passed through a classification layer to determine the news veracity.

EANN [16]: extracts textual and visual features using Text-CNN and VGG-19, respectively. These features are concatenated and fed into a fake news classifier. It incorporates an adversarial network to learn event-agnostic multimodal features for fake news detection.

MVAE [19]: extracts textual and visual features using Bi-LSTM and VGG-19. After concatenation, a Variational Auto-encoder (VAE) learns a shared multimodal representation to capture complex cross-modal relationships.

MCNN [39]: combines ELA (Error Level Analysis) algorithm and ResNet50 to extract image tampering and semantic features. BERT and Bi-GRU are used for textual feature ex-traction. An attention mechanism explores the overall news features.

ENRoBERTa [31]: achieves cross-modal information complementarity through efficient bimodal fusion and targeted preprocessing (ELA). Cross-modal heterogeneous data undergo a deep feature fusion module to complete the final detection task.

MCAN [36]: extracts spatial domain features using VGG-19 and textual features using BERT. It also incorporates DCT (Discrete Cosine Transform) to extract frequency domain features. Self-attention and co-attention modules are used to fuse multimodal information.

The comparative experimental results are presented in

Table 1. Comparative experimental results.

Dataset	Methods	Accuracy	Macro-F1	Fake News				True News
				Precision	Recall	F1		Precision	Recall	F1
Weibo	Visual-Only	0.528	0.528	0.540	0.520	0.530		0.515	0.538	0.526
	Text-Only	0.661	0.658	0.685	0.610	0.645		0.630	0.715	0.670
	CLIP	0.771	0.771	0.769	0.778	0.773		0.773	0.764	0.768
	EANN	0.782	0.780	0.827	0.697	0.756		0.752	0.863	0.804
	MVAE	0.824	0.823	0.854	0.769	0.809		0.802	0.875	0.837
	MCNN	0.846	0.845	0.809	0.857	0.832		0.879	0.837	0.858
	ENROBERTa	0.812	0.799	0.851	0.784	0.816		0.744	0.826	0.782
	MCAN	0.899	0.899	0.913	0.889	0.901		0.884	0.909	0.897
	BSEAN(Ours)	0.901	0.905	0.925	0.870	0.897		0.890	0.935	0.912
Twitter	Visual-Only	0.534	0.534	0.550	0.480	0.513		0.520	0.595	0.555
	Text-Only	0.592	0.591	0.610	0.530	0.567		0.575	0.660	0.615
	CLIP	0.788	0.785	0.770	0.760	0.765		0.790	0.820	0.805
	EANN	0.648	0.639	0.810	0.498	0.617		0.584	0.759	0.660
	MVAE	0.745	0.744	0.801	0.719	0.758		0.689	0.777	0.730
	MCNN	0.784	0.784	0.778	0.781	0.779		0.790	0.787	0.788
	ENROBERTa	0.853	0.849	0.821	0.943	0.877		0.913	0.745	0.820
	MCAN	0.809	0.809	0.889	0.765	0.822		0.732	0.871	0.795
	BSEAN(Ours)	0.883	0.876	0.905	0.845	0.874		0.911	0.846	0.877

. Observation and analysis of these results yield the following findings:

• The model proposed in this paper outperforms most other baseline models across the evaluation metrics. In terms of accuracy, our model achieves improvements on both the Weibo and Twitter datasets compared to the other models, with overall accuracies of 90.1% and 88.3%, respectively. Furthermore, the F1-scores indicate that our proposed model demonstrates stable discriminative ability for fake news detection. Although the improvement in accuracy over some baseline models, such as MCAN on Weibo, is marginal, a deeper analysis reveals the distinct advantages of BSEAN. As shown in Table 1, BSEAN achieves the highest Macro-F1 score, which indicates a more balanced performance across both the real and fake news categories. Furthermore, as we will introduce in Section 4.7, the calibration of our model exhibits a lower Expected Calibration Error (ECE) score, which means its prediction confidences are more reliable.
• The data in the table show that fake news detection tasks relying solely on either textual or visual unimodal information perform poorly, as unimodal approaches struggle to capture rich semantic information. Moreover, detection tasks relying solely on the textual modality perform better than those relying solely on the visual modality. This is likely because the information provided by the textual modality is often more complete for semantic understanding than that directly obtainable from the visual modality.
• In the context of multimodal fake news detection: The CLIP-based model demonstrates the potential of pre-trained vision-language models, yet its results also suggest a potential limitation of relying solely on powerful general-purpose representations. EANN, through its event adversarial neural network, fuses textual and visual features and excels at mining explicit and implicit common features across events. However, the removal of specific event features can diminish its identification capability. MVAE’s multimodal variational autoencoder generates fused features via a self-supervised loss, enhancing generalization. Nevertheless, its simple fusion of modal information and lack of sophisticated interaction lead to suboptimal performance, and its sensitivity to hyperparameters further degrades overall efficacy. MCNN improves detection performance by analyzing social media features and the consistency of multimodal data. However, this method does not fully exploit the potential synergistic value of information from different modalities, a limitation that constrains the effective utilization of fused features by the model. ENRoBERTa achieves cross-modal information complementarity through efficient feature extraction and contrastive learning, effectively improving detection accuracy. However, the depth of feature fusion for complex modal interaction scenarios needs further enhancement. MCAN deeply fuses multimodal features through a multi-layer co-attention stack. This approach learns interdependencies between cross-modal features, underscoring the importance of capturing cross-modal semantics.

4.5. Ablation Study Analysis

This study explores the effectiveness of each component of the BSEAN model by evaluating performance after the removal of individual components. The following variants were created:

w/o BMMN: In this variant, the Bidirectional Modality Mapping Network (BMMN) module was removed. This means that textual and visual features, before entering the Semantic Interaction Enhancement module, no longer undergo explicit cross-modal mapping and cycle consistency constraints.

w/o SIE: The attention mechanisms within this module were removed. Instead, the input features were directly concatenated and simply fused, then fed into the Semantic Deviation Calibration module. Furthermore, to investigate the effectiveness of the bidirectional cross-attention mechanism within the SIE module, several variants with common attention mechanisms were designed to observe changes in model performance: Self-Only, Cross-Only, and Self-Cross.

w/o SDC: In this variant, the Semantic Deviation Calibration (SDC) module was removed. This implies that the deeply interacted features obtained after semantic interaction enhancement are directly concatenated without undergoing quantification of semantic deviation and adaptive adjustment of gating weights.

w/o DAL: This variant removed the adversarial learning mechanism from the discriminators. The final loss function no longer includes the adversarial terms from the event relevance discriminator and the modality consistency discriminator, retaining only the loss from the fake news discriminator.

The experimental results are shown in Figure 5. It can be observed that removing any module leads to a decline in model performance.

Removing the BMMN module results in a significant performance degradation. This indicates that effective initial semantic alignment is crucial for subsequent deep cross-modal interaction, directly impacting the ability to capture rich cross-modal semantic correlations. The model can more effectively align and fuse features from different modalities, as BMMN’s bidirectional mapping and cycle consistency constraint mechanisms effectively mitigate cross-modal heterogeneity. This allows the model to better learn modality-agnostic semantic representations, laying a solid foundation for subsequent deep semantic interaction.

Removing the SIE module or using its variants with other attention mechanisms causes a substantial drop in model performance, particularly in the F1-score. This corroborates our assertion regarding the lack of deep cross-modal interaction in current methods. The SIE module, through its cross-modal interaction attention mechanism, effectively deepens the understanding of intra-modal contexts and promotes the deep fusion of cross-modal information. It enables textual features to “attend to” and absorb visual semantics from image scenes, and vice versa, thereby generating more discriminative cross-modal features. This deep, attention-driven interaction is key to understanding potential contradictions and correlations in multimodal fake news, significantly enhancing the model’s ability to discriminate between true and fake news. Ablation study results show performance after removing various modules used in the model, and for the SIE module, use different attention mechanism variants.

The absence of the SDC module leads to a decline in model performance, especially when handling samples with incompletely consistent modal information. This highlights the effectiveness of SDC in addressing inherent cross-modal semantic discrepancies. By quantifying the deviation through cosine distance between textual and visual features and adaptively adjusting the contribution of each modal feature based on this deviation, SDC can suppress modal features that are less informative or contradictory while enhancing highly consistent information. This prevents noise or misleading information from a single modality from negatively impacting the final discriminative outcome.

After removing the DAL module, all performance metrics decreased. The event relevance discriminator, by compelling the feature extractor to be insensitive to event labels, helps the model learn more generalizable fake news features rather than overfitting to specific event scenarios. Simultaneously, the modality consistency discriminator effectively suppresses modal bias by aligning the distributions of textual and visual features in the latent space. This dual adversarial mechanism works synergistically to significantly enhance the model’s generalization ability and robustness, enabling it to exhibit strong-er discriminative power when faced with diverse fake news.

4.6. Quantitative Analysis of SDC Module Weights

To meticulously evaluate the effectiveness of the adaptive gating mechanism within the SDC module, we conducted a hyperparameter sensitivity experiment. We established a series of fixed gating weights (ranging from 0.1 to 0.9 with a step of 0.1), which replaced the dynamically calculated gating weights based on semantic deviation in the SDC module. The experimental results are illustrated in Figure 6. The following conclusions can be observed and analyzed:

When the fixed gating weight g approaches 0.1 or 0.9, the model’s accuracy and F1-score exhibit a sharp decline. This indicates that the excessive suppression of modal features leads to the filtering or loss of substantial critical semantic information, causing the model to face information scarcity during the fusion stage. In such scenarios, the model is almost unable to effectively utilize visual-textual features for fake news discrimination, and its performance may even be inferior to that of baseline models relying solely on a single modality, as even unimodal information is inappropriately attenuated. This validates the importance of moderately preserving and utilizing modal features.

As the fixed weight g increases, the model’s performance (accuracy and F1-score) initially rises, then tends to plateau, and, eventually, may slightly decrease. For a fixed gating weight, peak performance is reached when g is between 0.6 and 0.8. This point represents the best performance achievable by a static gating mechanism on this dataset: It suppresses noise and inconsistent information to some extent while retaining sufficient effective features. However, even this optimal fixed weight yields a performance significantly lower than that achieved by the complete BSEAN model. This clearly reveals the inherent limitation of a fixed-weight mechanism—it is a global, static decision incapable of making fine-grained adjustments based on the cross-modal semantic deviations of different samples.

In stark contrast to the performance curves under all fixed weights, the SDC module employing adaptive weights consistently maintains the highest accuracy and F1-score throughout the experiment, with its performance point exceeding the peak achievable by any fixed weight. BSEAN’s adaptive gating mechanism dynamically generates gating weights for textual and visual features based on the specific semantic deviation (i.e., the degree of cross-modal consistency or conflict) calculated from the visual-textual features of each input sample. This implies the following:

• When visual-textual information is highly consistent and mutually supportive, the weight values tend to be higher, ensuring that this strong, consistent evidence is fully utilized.
• When significant deviations or even contradictions exist between visual and textual information, the weight values tend to be lower to appropriately suppress this potential noise or conflicting information, preventing it from misleading the model’s discrimination.
• This context-aware dynamic adjustment capability enables the model to simulate the human cognitive process of “weighing” and “filtering” evidence when processing multimodal information, thereby achieving effective fusion of heterogeneous information.

4.7. Investigation into Model Reliability via Calibration Analysis

While accuracy and F1-score are pivotal for assessing discriminative power, the trustworthiness of a model’s predictions is equally critical for its deployment in high-stakes domains like fake news detection. For this purpose, we employ the Expected Calibration Error (ECE) metric. A model is considered well-calibrated if its predicted confidence aligns with its empirical accuracy, and a lower ECE score indicates better calibration. This analysis aims to explore how the architectural components of BSEAN contribute to the trustworthiness of its predictions.

The results, presented in Figure 7, detail the calibration performance of BSEAN and its ablated variants. Our full BSEAN model achieved ECE scores of 3.4% on the Weibo dataset and 4.2% on the Twitter dataset. Among the tested configurations, the complete model exhibits the most favorable calibration properties, suggesting that its predictive probabilities are more reliable than those of its incomplete counterparts.

The primary finding from this analysis emerges from the ablation study. A significant increase in ECE was observed when the Semantic Deviation Calibration (SDC) module was removed (w/o SDC), reaching 0.076 on Weibo and 0.085 on Twitter. This provides a strong indication that the SDC module plays a crucial role in improving the model’s calibration. By designedly addressing semantic conflicts between modalities, the SDC module appears to help regulate the model’s confidence, leading to more dependable probability estimates, especially in ambiguous cases. Similarly, the removal of the DAL and BMMN modules also resulted in poorer calibration, suggesting that generalization and initial feature alignment are contributing factors to overall model reliability.

In summary, this investigation into model calibration suggests that the architectural choices within BSEAN, particularly the inclusion of the SDC module, not only enhance its discriminative accuracy but also contribute positively to the reliability of its predictions. While further research involving broader cross-model calibration comparisons is warranted, these findings position BSEAN as a promising approach that considers both performance and trustworthiness.

4.8. Analysis of Model Boundaries

To provide a more granular analysis of our model’s performance and address the identification of specific failure patterns, we conducted a detailed error analysis. This involves both a quantitative breakdown of classification results via confusion matrices and a qualitative investigation into the root causes of misclassifications.

Figure 8 presents the confusion matrices of the BSEAN model on the Weibo and Twitter test sets. These visualizations offer a clear breakdown of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).

The matrices reveal a key insight: On both datasets, the number of false negatives (FN) is significantly higher than the number of false positives (FP). This indicates that the primary challenge for BSEAN is not in misidentifying real news, but rather in failing to detect sophisticated or nuanced fake news. This underscores the difficulty of the task and motivates a deeper analysis of these “missed” fakes. To understand the practical boundaries of our model, we manually reviewed a sample of the false negative cases and identified two predominant failure patterns:

Difficulty with Satirical and Ironic Content:A significant portion of the false negatives involved satire, where the literal meaning of the text is inconsistent with its pragmatic, deceptive intent. For example, a post might pair a mundane image with text like, “Following the latest city policy, our skies have never been clearer,” intending to sarcastically critique pollution. BSEAN, which excels at finding direct semantic contradictions, struggles to capture this type of high-level linguistic nuance, thus misclassifying the post as real.

Out-of-Context Image-Text Pairs: Another common failure pattern occurs when a real, unaltered image is repurposed with a misleading textual narrative. For instance, a well-documented, emotive photograph of two children from one country can be falsely re-contextualized as depicting victims of a natural disaster in another. Our model can confirm that the image is authentic and the text is grammatically sound, and may even find a thematic link. However, it lacks the external world knowledge or fact-checking capability to verify the spatio-temporal context, leading to an incorrect classification.

5. Case Study

To provide a concrete understanding of BSEAN’s capabilities and limitations, we present three distinct case studies in Figure 9. We visualize the model’s attention via heatmaps and report its classification confidence, linking these outputs to our quantitative analyses.

From these cases, it can be observed that the model extracts key information from both visual and textual modalities to understand the semantic associations between them. Focusing solely on either modality can easily lead to misclassification. The success cases in Figure 9a,b demonstrate the model’s core strength in detecting various forms of semantic deviation.

In the first case in Figure 9a, the model is faced with deep associations between the visual of lenticular clouds and a conflicting textual narrative (e.g., describing it as a Cedar tree cloud). This suggests that the model learns to identify discrepancies between the sentiment or narrative conveyed by the text (describing a “spectacle” with incorrect terms) and the factual content of the image. The model’s attention visualization supports this, as it focuses on the core visual element to assess its semantic alignment with the text, correctly classifying the post as Fake (confidence: 94.7%). This sensitivity enables the effective identification of superficially consistent but factually conflicting information.

Similarly, in the second case Figure 9b, the model scrutinizes the semantic consistency between the actions attributed to the entity by the text (HIDING, scared) and the static, inanimate nature of the image (the Statue of Liberty). When the textual description conflicts with the inherent attributes of the visual content, the model accurately identifies this semantic deviation and effectively discerns the false information arising from the image–text discrepancy, leading to a high-confidence Fake classification (confidence: 91.2%).

The third case in Figure 9c serves as a crucial example of an ‘out-of-context’ failure pattern, clearly defining the model’s operational boundary. The post combines an emotive image of two children with a false narrative about the “#NepalEarthquake”. BSEAN misclassifies this post as Real, but with a very low confidence of 56.93%. This insightful result reveals that the model correctly identifies the high semantic harmony between the visual and textual content. However, the extremely low confidence signals the model’s “uncertainty” when faced with a challenge beyond its scope (external fact-checking). It demonstrates that while BSEAN excels at detecting internal semantic inconsistencies, it lacks the external knowledge for provenance verification, a clear direction for future work.

6. Discussion

Our study, while demonstrating the efficacy of BSEAN on established benchmarks, operates within a rapidly evolving information ecosystem. It is, therefore, pertinent to discuss the broader context of our model’s capabilities, particularly concerning emerging challenges such as AI-Generated Content (AIGC) and adversarial threats.

A key consideration for any contemporary detection model is its applicability to AIGC. Although the datasets utilized in this work predate the widespread use of generative AI, the architectural design of BSEAN is inherently suited to addressing this challenge. AIGC often exhibits subtle cross-modal inconsistencies—for instance, a synthetic image may lack perfect semantic coherence with a detailed textual narrative. BSEAN’s core mechanism, the Semantic Enhancement and Calibration Network (SECN), is engineered precisely to identify such fine-grained semantic deviations. By focusing on deep logical congruence rather than superficial features, our model possesses a foundational capability to detect the subtle incongruities common in AI-generated fakes, positioning it as a strong candidate for future validation on AIGC-centric datasets.

Furthermore, ensuring model robustness is paramount for real-world deployment. While this study did not include explicit adversarial attack evaluations, the Dual Adversarial Learning (DAL) framework is an integral component designed to bolster intrinsic resilience. By promoting the learning of event-agnostic features, the DAL module mitigates the risk of overfitting to spurious correlations within the training data. This process of learning disentangled representations enhances the model’s stability against distribution shifts and certain data perturbations. This design choice serves as a foundational step towards building a more adversarially robust system, providing a solid basis for future investigations into specialized defense mechanisms against targeted white-box and black-box attacks. Similarly, another critical dimension of robustness for real-world deployment is the model’s cross-lingual generalizability. While our study provides a strong validation on major English and Chinese benchmarks, assessing its efficacy across more diverse cultural and linguistic contexts remains a crucial direction for future work.

7. Conclusions

In this paper, we addressed the pressing challenge of multimodal fake news detection within the Web3.0 ecosystem by proposing a novel framework, the Bidirectional Semantic Enhancement and Adversarial Network (BSEAN). Our primary contribution lies in a multi-stage approach that emulates human-like verification by first achieving robust cross-modal alignment, then deeply exploring semantic inconsistencies via a dedicated enhancement and calibration mechanism, and, finally, learning generalizable features through a dual adversarial strategy. Extensive experiments on the Weibo and Twitter datasets demonstrate that BSEAN significantly outperforms existing state-of-the-art methods, validating its effectiveness in identifying subtle cross-modal semantic conflicts. While this work establishes a strong foundation for content-based analysis, future iterations could further align with the Web3.0 paradigm by integrating on-chain provenance data for a more holistic and trustworthy detection framework.