User Comment-Guided Cross-Modal Attention for Interpretable Multimodal Fake News Detection

Abstract

In order to address the pressing challenge posed by the proliferation of fake news in the digital age, we emphasize its profound and harmful impact on societal structures, including the misguidance of public opinion, the erosion of social trust, and the exacerbation of social polarization. Current fake news detection methods are largely limited to superficial text analysis or basic text–image integration, which face significant limitations in accurately identifying deceptive information. To bridge this gap, we propose the UC-CMAF framework, which comprehensively integrates news text, images, and user comments through an adaptive co-attention fusion mechanism. The UC-CMAF workflow consists of four key subprocesses: multimodal feature extraction, cross-modal adaptive collaborative attention fusion of news text and images, cross-modal attention fusion of user comments with news text and images, and finally, input of fusion features into a fake news detector. Specifically, we introduce multi-head cross-modal attention heatmaps and comment importance visualizations to provide interpretability support for the model’s predictions, revealing key semantic areas and user perspectives that influence judgments. Through the cross-modal adaptive collaborative attention mechanism, UC-CMAF achieves deep semantic alignment between news text and images and uses social signals from user comments to build an enhanced credibility evaluation path, offering a new paradigm for interpretable fake information detection. Experimental results demonstrate that UC-CMAF consistently outperforms 15 baseline models across two benchmark datasets, achieving F1 Scores of 0.894 and 0.909. These results validate the effectiveness of its adaptive cross-modal attention mechanism and the incorporation of user comments in enhancing both detection accuracy and interpretability.

1. Introduction

In the era of digital information explosion, the widespread dissemination of fake news has become an urgent social challenge, threatening not only public discourse and democratic processes but also individual decision-making [1,2]. Particularly, multimodal fake news, through strategically combining manipulated text, images, and user interactions, spreads significantly faster than pure textual misinformation and achieves higher engagement rates, posing severe challenges to traditional detection methods [3,4]. Therefore, developing effective fake news detection and mitigation methods has become a critical research direction in the fields of artificial intelligence and machine learning [5,6].

In recent years, with the widespread application of multimodal learning in information retrieval, its advantages of leveraging multiple data sources for complementary information have become increasingly prominent [7,8]. From image–text matching to video captioning and sentiment analysis, multimodal methods have significantly improved task performance by extracting and fusing features from different modalities [9,10]. In this context, extending multimodal learning to the fake news detection domain, particularly utilizing user comments as a key modality for analysis to improve accuracy in identifying fake news through social signals, holds significant research value and practical importance [11,12].

Although fake news detection has developed various methods based on content, context, and social signals [13,14], these methods are typically limited to superficial textual analysis or basic text–image integration, encountering significant obstacles in precisely locating deceptive content [15,16]. Existing methods often fail to comprehensively capture the nuances and complexities of fake news, primarily manifested in several aspects: sophisticated fake news often employs subtle linguistic techniques that are difficult to detect through traditional textual analysis; visual content is frequently manipulated or taken out of context to support false narratives; fake news creators exploit cognitive biases and emotional triggers to enhance credibility; valuable insights provided by user comments and reactions are often underutilized in current detection systems [17,18].

To address these challenges, this work tackles three fundamental research problems: the effective integration of user comments as a primary modality rather than auxiliary information in multimodal fake news detection; the design of cross-modal attention mechanisms capable of capturing fine-grained semantic relationships between news content and social validation signals; and the development of interpretable detection systems that provide transparent decision-making processes for practical deployment scenarios. These problems arise from the limitations of existing methods that treat user comments as supplementary data and lack the ability to capture nuanced cross-modal interactions while providing explainable detection decisions.

To bridge these gaps, we propose the UC-CMAF framework, which fundamentally transforms the role of user comments from auxiliary information to a primary modality while introducing novel adaptive cross-modal attention mechanisms for comprehensive fake news detection. Our research objectives are threefold: first, to develop a framework that effectively harnesses collective intelligence and social validation signals embedded in user comments; second, to design sophisticated cross-modal attention mechanisms that capture intricate semantic relationships between textual claims, visual evidence, and user feedback; and third, to ensure interpretable detection outcomes through comprehensive attention visualizations and hierarchical importance scoring, thereby enabling transparency and facilitating practical deployment in real-world fact-checking scenarios.

The main contributions of this paper include three aspects. First, we propose UC-CMAF, an innovative multimodal fusion framework that successfully integrates three different information modalities—news text, images, and user comments—for fake news detection. Second, UC-CMAF integrates an adaptive co-attention fusion mechanism that can dynamically adjust weights between different modalities, achieving efficient cross-modal information fusion. Finally, we conducted extensive experimental validation on multiple datasets, demonstrating the excellent performance of the UC-CMAF model in fake news detection tasks, and further validated the importance of each component in the model through ablation studies.

While effectively identifying fake news, improving the interpretability of detection decisions is of great significance for building public trust and supporting human intervention [19,20]. With the rapid development of Explainable Artificial Intelligence (XAI), incorporating its principles into multimodal misinformation identification tasks can achieve the synergistic goal of “visual explanation + social context explanation” [21,22]. Specifically, fake news detection not only requires high accuracy but also needs the model to answer “why” certain information is judged as false, which relies on the joint support of cross-modal semantic consistency and verifiable signals from user comments [23,24].

2. Review of Fake News Detection Approaches and Multimodal Learning Methods

2.1. Content-Based Fake News Detection

Content-based approaches primarily identify false information by analyzing intrinsic characteristics of news articles, including linguistic patterns, semantic inconsistencies, and visual manipulations. Early research relied on handcrafted features such as lexical diversity, syntactic patterns, readability metrics, and writing style analysis [25,26,27]. Perez et al. [25] proposed a classification method based on linguistic features, identifying fake news through lexical complexity and syntactic structure analysis. Zhou et al. [26] further extended this approach by constructing comprehensive feature sets incorporating psycholinguistic and rhetorical characteristics. While these traditional methods provided baseline detection capabilities, they often failed to capture sophisticated manipulation techniques employed in modern fake news.

With the advancement of deep learning technologies, neural network approaches have demonstrated superior performance in capturing complex semantic patterns. Wang et al. [28] proposed a CNN-based text classification method that effectively extracted local textual features. Yang et al. [8] employed Long Short-Term Memory (LSTM) networks to handle temporal dependencies in sequential data, achieving significant improvements in fake news detection tasks. Khattar et al. [16] developed MVAE (Multimodal Variational Autoencoder) that learns joint representations of text and images, enabling the model to capture latent correlations between different modalities.

Recent multimodal approaches have explored various fusion strategies to integrate textual and visual information. Wang et al. [15] proposed EANN (Event Adversarial Neural Network) that combines adversarial training with multimodal feature extraction, significantly improving generalization capabilities across different domains and events. Zhou et al. [29] designed multimodal fusion networks that dynamically balance the importance of different modalities through attention mechanisms. However, these methods often fail to capture subtle inconsistencies characteristic of complex fake news and lack the ability to leverage external validation signals from user communities.

2.2. Context-Based Fake News Detection

Context-based approaches identify false information by examining the social environment surrounding news propagation, including user characteristics, propagation patterns, temporal dynamics, and social network structures. These methods recognize that fake news often exhibits distinctive behavioral patterns during propagation through social networks [30,31].

Social network analysis research has revealed characteristic propagation patterns of fake news. Vosoughi et al. [32] analyzed large-scale Twitter data and found that false news spreads faster, farther, and deeper than true news. Jin et al. [33] further demonstrated that fake news exhibits higher engagement rates and distinctive user interaction patterns. Tacchini et al. [34] showed that false news tends to propagate within echo chambers and filter bubbles, forming identifiable network topological features.

Graph-based methods have shown unique advantages in modeling complex relationships. Guo et al. [35] proposed Hierarchical Social Attention (HSA) networks that utilize hierarchical social network structures and user comments for rumor identification. Liu et al. [36] developed graph convolutional networks capable of simultaneously processing complex relationships among users, content, and propagation patterns. Ma et al. [37] designed recurrent neural networks that model temporal evolution characteristics of rumors by constructing propagation tree structures. Additionally, tensor factorization with sparse and graph regularization has been applied to capture intricate social network structures for fake news detection [38].

Recent research has focused more on deep integration of user comments and social signals. Shu et al. [39] proposed the dEFEND framework that uses user comments as external knowledge to enhance detection performance. Shang et al. [40] developed DGExplain, introducing a dual-generative framework that combines content analysis with comment-based explanations, providing both detection capabilities and interpretable insights. However, existing methods typically treat comments as auxiliary information rather than primary modalities, representing an important research gap.

2.3. Attention Mechanisms in Multimodal Learning

Attention mechanisms have revolutionized multimodal learning by enabling models to focus on relevant information from different modalities while suppressing noise inputs [41,42]. Bahdanau et al. [41] initially introduced attention mechanisms for machine translation tasks, allowing models to dynamically attend to relevant parts of input sequences when generating outputs.

Cross-modal attention mechanisms enable identification of correspondences and relationships between different data types [43,44]. Xu et al. [45] proposed visual attention models capable of precisely aligning textual descriptions with image regions. Anderson et al. [46] developed bottom-up attention mechanisms that extract visual features through object detection techniques, significantly improving performance on vision–language tasks. Yu et al. [47] designed collaborative attention mechanisms that facilitate bidirectional information flow between modalities, enabling each modality to mutually enhance representational capabilities.

The emergence of Transformer architectures has further advanced attention mechanism capabilities. Vaswani et al. [42] proposed self-attention mechanisms capable of modeling long-range dependencies. Building upon this, Lu et al. [48] developed ViLBERT and Li et al. [49] proposed VisualBERT, achieving breakthrough progress in multimodal understanding tasks. Chen et al. [50] unified multiple vision–language pre-training tasks with UNITER, demonstrating the powerful potential of Transformers in multimodal learning.

Despite the significant success of attention mechanisms in multimodal learning, their application in fake news detection, particularly integrating user comments as primary modalities, remains limited, providing important opportunities for improving detection accuracy and robustness.

2.4. User Comment Analysis for Credibility Assessment

User comments provide valuable information for news credibility assessment through mechanisms including collective intelligence, crowdsourced fact-checking, and early warning signals [51,52]. Castillo et al. [30] conducted pioneering research showing that user replies on Twitter contain rich credibility cues that effectively assist rumor detection. Kumar et al. [53] further validated the phenomenon that domain experts, fact-checkers, and informed citizens in comment sections can quickly identify and expose misinformation.

Temporal dynamics of comments also provide important detection signals. Kwon et al. [31] found that comment patterns for legitimate news typically show predictable correlations with news cycles and public interest, while fake news often exhibits artificial engagement patterns or abnormal temporal distributions. Zhao et al. [54] confirmed the crucial role of early comments in rumor detection through Weibo data analysis.

Semantic analysis of comments reveals additional credibility indicators. Popat et al. [55] discovered that comments on legitimate news tend to focus on substantive discussions about reported events, while comments on fake news often exhibit characteristics such as emotional manipulation, topic deflection, or coordinated messaging campaigns. Enayet et al. [56] further analyzed the impact of stance detection and sentiment orientation in comments on credibility assessment.

However, comment analysis faces numerous challenges requiring effective solutions: noise filtering, relevance assessment, temporal dynamics modeling, and scalability issues [4,57]. Existing research still lacks sufficient exploration of comment information potential, particularly in cross-modal fusion and real-time processing capabilities.

2.5. Explainable AI in Misinformation Detection

Explainable Artificial Intelligence (XAI) aims to enhance transparency and interpretability of model decisions, playing crucial roles in safety-critical applications [20,58]. In fake news detection, Ribeiro et al. [22] proposed the LIME method that explains black-box model predictions through local approximation. Lundberg et al. [23] developed the SHAP framework that quantifies feature contributions to predictions based on game theory principles.

Research on explainability for multimodal fake news detection is gradually emerging. Antol et al. [59] demonstrated model attention focus when processing vision–language tasks through attention visualization. Selvaraju et al. [60] proposed Grad-CAM techniques that generate class activation maps to explain convolutional neural network decision processes. For fake news detection, Shang et al. [40] provided interpretable detection evidence for COVID-19 information detection through the DGExplain generative explanation framework.

However, existing explainability methods primarily focus on single modalities or simple multimodal fusion, lacking deep interpretability capabilities for social signals such as user comments. Furthermore, cross-modal consistency analysis and interpretable mining of community intelligence remain important directions to be explored [61,62].

Our proposed UC-CMAF framework introduces cross-modal interpretability mechanisms that not only display attention distributions of image–text consistency differences but also reveal evidence signal sources of collective intelligence through user comment importance scoring, achieving truly multimodal interpretable detection pathways.

3. Methodology

This study employs a quantitative experimental design to develop and evaluate a novel multimodal fake news detection framework. Our research design systematically addresses three specific objectives: (1) developing the UC-CMAF architecture that elevates user comments to primary modality status, (2) implementing adaptive cross-modal attention mechanisms for capturing semantic relationships between heterogeneous data sources, and (3) ensuring interpretable detection outcomes through comprehensive attention visualization and hierarchical importance scoring mechanisms.

3.1. Problem Statement

In this paper, our focus is on determining the authenticity of news articles circulating on social media platforms. Given a multimodal news article P composed of text T, images $I=\{I_1,I_2,\dots ,I_m\}$, and associated user comments $C=\{C_1,C_2,\dots ,C_n\}$, our goal is to predict the authenticity label $y\in \{0,1\}$, where $y=0$ indicates real news and $y=1$ indicates fake news.

The challenge lies in effectively modeling the complex relationships between these heterogeneous modalities while filtering noise and irrelevant information that may confound the detection process. Our approach must consider the following:

• Multimodal Consistency: Analyzing whether textual claims are supported by visual evidence.
• Social Validation: Leveraging user comments to identify credibility signals and fact-checking information.
• Temporal Dynamics: Considering the timing and sequence of content publication and user responses.
• Noise Resilience: Filtering irrelevant content, spam comments, and manipulative elements.

3.2. Overall Framework

Our model UC-CMAF can be represented in the architectural Figure 1, including four key sub-processes: multimodal feature extraction, cross-modal adaptive collaborative attention fusion for news text and images, cross-modal attention fusion of user comments with news text and images, and fake news detection.

The UC-CMAF framework consists of four integrated components working synergistically to achieve accurate fake news detection. The framework begins with text and image encoding networks that leverage state-of-the-art pre-trained models to extract rich semantic representations from textual and visual modalities of the provided news content. These encoded representations are then processed through the cross-modal adaptive collaborative attention (CMA) component, which implements sophisticated attention mechanisms that capture fine-grained relationships between textual and visual content while dynamically adjusting attention weights based on content characteristics. The User Comment Integration (UCI/UCT) component introduces specialized networks to process and integrate user comments with news content, enabling the model to leverage social signals and collective intelligence for authenticity assessment. Finally, the fake news detector serves as the aggregation component that combines all multimodal features and produces authenticity predictions with associated confidence scores, completing the comprehensive framework for detecting fake news through the synergistic integration of content analysis and social validation signals.

3.3. Multimodal Feature Extraction

As described in the problem statement, our model accepts multimodal news input $P=\{T,I,C\}$, where T represents textual content, I represents visual content, and C represents corresponding user comments. Text and comment inputs are encoded using a shared pre-trained BERT model to ensure consistent semantic representations. Images are processed using a ResNet-50 backbone, with final convolutional outputs adjusted to match the dimension of text embeddings through 2D convolution layers. This alignment facilitates effective interaction across modalities in the subsequent attention fusion stage.

3.3.1. Text Encoding Network

For textual content, we leverage BERT (Bidirectional Encoder Representations from Transformers), a pre-trained language model developed by Google that understands the contextual meaning of words in sentences by analyzing text bidirectionally, enabling it to capture nuanced semantic relationships and linguistic patterns essential for fake news detection. BERT generates features containing word semantics and linguistic context. Given text content T, it is modeled as a word sequence $T=\{w_1,w_2,\dots ,w_m\}$, where m represents the number of words in the article text.

$$H_T=\mathrm{BERT}\left(T\right)=\{h_1,h_2,\dots ,h_m\}$$

(1)

where $h_i\in {\mathbb{R}}^{d_t}$ represents the hidden state output of the corresponding token in BERT, and $d_t$ represents the dimension of text embeddings.

3.3.2. Image Encoding Network

For visual content I, we adopt ResNet-50 (Residual Network with 50 layers), a deep convolutional neural network that excels at image classification and feature extraction by using residual connections to effectively train very deep networks, making it particularly suitable for extracting comprehensive visual features from news images that may contain manipulated or misleading content. Considering that a piece of content may contain multiple images, we process each image independently and then aggregate the features.

$$F_V=\{f_1,f_2,\dots ,f_m\}\mathrm{where}{f}_j=\mathrm{ResNet}50\left(I_j\right)$$

(2)

where $f_j\in {\mathbb{R}}^{d_v}$ represents the feature vector of the j-th image, and $d_v$ represents the dimension of image embeddings (typically 2048 for ResNet50).

To ensure dimensional consistency across modalities, we incorporate a 2D convolutional layer that adapts the embedding dimension from $d_v$ to $d_t$:

$$F_{V\_adapted}=\mathrm{Conv}2\mathrm{D}\left(F_V\right){\mathbb{R}}^{d_v}\to {\mathbb{R}}^{d_t}$$

(3)

where Conv2D represents a 2D convolutional layer, a fundamental component in deep learning that applies filters to detect visual patterns and features in images.

3.3.3. Comment Encoding Network

User comments are processed through the same BERT architecture as text encoding, ensuring semantic representation consistency across textual modalities.

$$F_C=\{c_1,c_2,\dots ,c_n\}\mathrm{where}{c}_i=\mathrm{BERT}\left(C_i\right)$$

(4)

where $c_i\in {\mathbb{R}}^{d_t}$ represents the encoded representation of the i-th comment, and n represents the total number of comments.

3.4. Cross-Modal Adaptive Collaborative Attention Fusion for Text and Images

This component directly addresses our first research objective by implementing sophisticated attention mechanisms that treat user comments as equal partners with textual and visual content, moving beyond auxiliary data treatment to primary modality integration.

3.4.1. Attention Distribution Generation

To generate an attention distribution of visual features guided by textual features, we input the extracted text features $H_T$ and image features $F_V$ into a single-layer neural network followed by a softmax function:

$$\begin{array}{cc}\hfill \rho & =\tanh (W_H{H}_T\oplus (W_V{F}_V+b_V))\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill \alpha & =\mathrm{softmax}(W_{\alpha }\rho +b_{\alpha })\hfill \end{array}$$

(6)

where $W_H,W_V\in {\mathbb{R}}^{r\times d}$ and $W_{\alpha }$ are learnable parameters, $b_V$ and $b_{\alpha }$ are bias terms, and ⊕ represents element-wise addition with broadcasting. Where softmax is a mathematical function that converts a vector of numbers into a probability distribution, ensuring all attention weights sum to one.

3.4.2. Visual–Text Feature Alignment

The visual vector related to text features can be obtained by using the attention distribution as weights:

$$\widehat{v}=\sum _i{\alpha }_i·f_i$$

(7)

This weighted aggregation allows the model to focus on visual regions most relevant to textual content, enabling detection of inconsistencies between claimed events and visual evidence.

3.4.3. Bidirectional Attention Mechanism

We also compute text features guided by visual features to achieve bidirectional information flow:

$$\begin{array}{cc}\hfill \mu & =\tanh (W_V{F}_V\oplus (W_H{H}_T+b_H))\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \beta & =\mathrm{softmax}(W_{\beta }\mu +b_{\beta })\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \widehat{h}& =\sum _j{\beta }_j·h_j\hfill \end{array}$$

(10)

where $W_V,W_H\in {\mathbb{R}}^{r\times d}$, and $b_H,b_{\beta }$ are bias terms,${\mu }_j\in {\mathbb{R}}^r$ is the j-th column of $\mu \in {\mathbb{R}}^{r\times m}$, $W_{\beta }\in {\mathbb{R}}^{1\times r}$, $b_{\beta }\in \mathbb{R}$, and m is the number of text tokens. This process implements bidirectional attention by computing text features guided by visual information. It generates attention weights to determine text token importance based on visual guidance and produces visually guided text representations, enabling the model to identify textual content most relevant to the visual information.

3.4.4. Adaptive Fusion Strategy

After obtaining attention distributions $\alpha$ and $\beta$, we compute the mutually guided representation $\delta$ between text and visual modalities:

$$\delta =\widehat{v}\oplus \widehat{h}$$

(11)

where $\delta \in {\mathbb{R}}^{d\times n}$ represents the fused representation capturing text and visual semantics and their cross-modal dependencies.

3.5. Fusion of User Comments and News Content

The UCI and UCT components fulfill our second research objective by establishing bidirectional semantic relationships between social validation signals and news content, enabling comprehensive credibility assessment through collective intelligence.

3.5.1. User Comment and Image Fusion (UCI)

We adopt a cross-attention mechanism to fuse news images and user comments, enabling the model to identify visual elements that users are discussing or questioning:

The comment-to-image attention mechanism involves several sequential transformations, each serving a specific purpose in establishing cross-modal relationships.

The query transformation extracts attention queries from comment features:

$$\begin{array}{c}\hfill Q_{CI}={\mathrm{Linear}}_Q\left(F_C\right)\end{array}$$

(12)

where $Q_{CI}$ represents the query matrix derived from comments, $F_C$ denotes the comment feature representations, and ${\mathrm{Linear}}_Q$ is a learnable linear transformation layer that projects comment features into the query space.

The key transformation maps image features to attention keys:

$$\begin{array}{c}\hfill K_{CI}={\mathrm{Linear}}_K\left(F_V\right)\end{array}$$

(13)

where $K_{CI}$ is the key matrix generated from visual content, $F_V$ represents the visual feature representations, and ${\mathrm{Linear}}_K$ is a learnable linear transformation that projects image features into the key space.

The value transformation converts image features into attention values:

$$\begin{array}{c}\hfill V_{CI}={\mathrm{Linear}}_V\left(F_V\right)\end{array}$$

(14)

where $V_{CI}$ denotes the value matrix from visual features, $F_V$ represents the same visual feature representations, and ${\mathrm{Linear}}_V$ is a learnable linear transformation that projects image features into the value space.

The attention weight computation applies scaled dot-product attention:

$$\begin{array}{c}\hfill A_{CI}=\mathrm{softmax}\left({\displaystyle \frac{Q_{CI}{K}_{CI}^T}{\sqrt{d_k}}}\right)\end{array}$$

(15)

where $A_{CI}$ represents the normalized attention weights, $Q_{CI}{K}_{CI}^T$ computes the dot product between queries and keys, $d_k$ is the dimensionality of the key vectors used for scaling, and softmax ensures attention weights sum to one.

The final attended feature computation aggregates visual information based on attention weights:

$$\begin{array}{c}\hfill F_{CI}=A_{CI}{V}_{CI}\end{array}$$

(16)

where $F_{CI}$ represents the final comment-informed visual features, $A_{CI}$ are the attention weights, and $V_{CI}$ are the value representations that are weighted and aggregated.

3.5.2. User Comment and Text Fusion (UCT)

Since user comments often contain rich textual information that can supplement or refute the main news text, we implement a sophisticated fusion mechanism to capture comment-to-text relationships.

The query transformation extracts attention queries from comment features:

$$\begin{array}{c}\hfill Q_{CT}={\mathrm{Linear}}_Q\left(F_C\right)\end{array}$$

(17)

where $Q_{CT}$ represents the query matrix derived from comment features, $F_C$ denotes the comment feature representations, and ${\mathrm{Linear}}_Q$ is a learnable linear transformation layer that projects comment features into the query space for text attention.

The key transformation maps news text features to attention keys:

$$\begin{array}{c}\hfill K_{CT}={\mathrm{Linear}}_K\left(H_T\right)\end{array}$$

(18)

where $K_{CT}$ is the key matrix generated from textual content, $H_T$ represents the news text feature representations, and ${\mathrm{Linear}}_K$ is a learnable linear transformation that projects text features into the key space.

The value transformation converts text features into attention values:

$$\begin{array}{c}\hfill V_{CT}={\mathrm{Linear}}_V\left(H_T\right)\end{array}$$

(19)

where $V_{CT}$ denotes the value matrix from textual features, $H_T$ represents the same news text feature representations, and ${\mathrm{Linear}}_V$ is a learnable linear transformation that projects text features into the value space.

The attention weight computation applies scaled dot-product attention:

$$\begin{array}{c}\hfill A_{CT}=\mathrm{softmax}\left({\displaystyle \frac{Q_{CT}{K}_{CT}^T}{\sqrt{d_k}}}\right)\end{array}$$

(20)

where $A_{CT}$ represents the normalized attention weights, $Q_{CT}{K}_{CT}^T$ computes the dot product between comment queries and text keys, $d_k$ is the dimensionality of the key vectors used for scaling, and softmax ensures attention weights sum to one.

The final attended feature computation aggregates textual information based on attention weights:

$$\begin{array}{c}\hfill F_{CT}=A_{CT}{V}_{CT}\end{array}$$

(21)

where $F_{CT}$ represents the final comment-informed textual features, $A_{CT}$ are the attention weights, and $V_{CT}$ are the value representations that are weighted and aggregated to capture relevant textual information guided by comment content.

Algorithm 1 achieves deep fusion of user comments with news content, including two main branches:

UCI Branch (Comment–Image Fusion): Adopts cross-attention mechanism, using comments as queries and image features as keys and values. This design enables the model to identify specific visual elements discussed or questioned by users in comments, such as inconsistencies in images, traces of photo manipulation, or visual content that does not match textual descriptions.

UCT Branch (Comment–Text Fusion): Similarly uses the cross-attention mechanism, but with text features as key-value pairs. This enables the model to capture supplementary, corrective, or refuting information from comments regarding text content.

Aggregation Mechanism: Combined with the importance weights obtained from the first algorithm, comment features are weighted and aggregated to ensure that high-quality comments play a greater role in the fusion process. Finally, the aggregated comment representation is combined with original text features through feature concatenation.

Algorithm 1 User comment fusion algorithm.

Require:  Comment features $F_C$, image features $F_V$, text features $H_T$, importance weights $S_{importance}$ Ensure:  Comment-image fusion features $F_{CI}$, comment-text fusion features $F_{TC}$   1: // UCI: Comment-Image Fusion   2: $Q_{CI}\leftarrow {\mathrm{Linear}}_Q\left(F_C\right)$                                                                                                     ▹ Comment queries   3: $K_{CI}\leftarrow {\mathrm{Linear}}_K\left(F_V\right)$, $V_{CI}\leftarrow {\mathrm{Linear}}_V\left(F_V\right)$                                                            ▹ Image keys and values   4: $A_{CI}\leftarrow \mathrm{softmax}\left({\displaystyle \frac{Q_{CI}{K}_{CI}^T}{\sqrt{d_k}}}\right)$                                                                                           ▹ Attention weights   5: $F_{CI}\leftarrow A_{CI}{V}_{CI}$                                                                                                                 ▹ Weighted fusion   6: // UCT: Comment-Text Fusion   7: $Q_{CT}\leftarrow {\mathrm{Linear}}_Q\left(F_C\right)$                                                                                                    ▹ Comment queries   8: $K_{CT}\leftarrow {\mathrm{Linear}}_K\left(H_T\right)$, $V_{CT}\leftarrow {\mathrm{Linear}}_V\left(H_T\right)$                                                            ▹ Text keys and values   9: $A_{CT}\leftarrow \mathrm{softmax}\left({\displaystyle \frac{Q_{CT}{K}_{CT}^T}{\sqrt{d_k}}}\right)$                                                                                        ▹ Attention weights 10: $F_{CT}\leftarrow A_{CT}{V}_{CT}$                                                                                                               ▹ Weighted fusion 11: // Aggregated Comment Representation 12: $C_{agg}\leftarrow {\sum }_{i=1}^n{S}_{importance}\left(c_i\right)·c_i$                                                                                ▹ Importance weighting 13: $F_{TC}\leftarrow \mathrm{Concat}(H_T,C_{agg})$                                                                                      ▹ Feature concatenation 14: return $F_{CI}$, $F_{TC}$

Algorithm 1 establishes a comprehensive framework for integrating user comments with multimodal news content. The UCI branch enables the model to identify visual elements that users discuss or question, while the UCT branch captures supplementary or contradictory information from comments regarding textual content. The importance weighting mechanism ensures that high-quality, credible comments play a more significant role in the detection process. This dual-branch design fully utilizes the diversity of user comments, considering both the verification of visual content by comments and the supplementation and correction of textual content.

3.5.3. Hierarchical Comment Importance Scoring

Considering that not all comments are equally informative for credibility assessment, we implement a hierarchical scoring mechanism to evaluate comment importance across multiple dimensions.

The relevance score measures semantic similarity between comments and news text:

$$\begin{array}{c}\hfill S_{rel}\left(c_i\right)=\mathrm{cosine\_similarity} (c_i,H_T)\end{array}$$

(22)

where $S_{rel}\left(c_i\right)$ represents the relevance score for comment $c_i$, $c_i$ denotes the i-th comment representation, $H_T$ represents the news text feature representation, and cosine_similarity computes the normalized dot product between comment and text embeddings.

The credibility score evaluates trustworthiness indicators within comments:

$$\begin{array}{c}\hfill S_{cred}\left(c_i\right)=\mathrm{CredibilityScorer}\left(c_i\right)\end{array}$$

(23)

where $S_{cred}\left(c_i\right)$ represents the credibility score for comment $c_i$, $c_i$ denotes the comment content, and CredibilityScorer is a specialized neural network that analyzes linguistic patterns, factual consistency, and other credibility indicators within the comment text.

The temporal importance score accounts for comment recency:

$$\begin{array}{c}\hfill S_{temp}\left(c_i\right)=\exp (-\lambda ·(t_{current}-t_{comment}))\end{array}$$

(24)

where $S_{temp}\left(c_i\right)$ represents the temporal importance score for comment $c_i$, $\lambda$ is a decay parameter controlling the rate of temporal degradation, $t_{current}$ denotes the current timestamp, $t_{comment}$ represents the comment posting time, and the exponential function ensures recent comments receive higher weights.

The overall importance score combines all three dimensions:

$$\begin{array}{c}\hfill S_{importance}\left(c_i\right)=\alpha ·S_{rel}\left(c_i\right)+\beta ·S_{cred}\left(c_i\right)+\gamma ·S_{temp}\left(c_i\right)\end{array}$$

(25)

where $S_{importance}\left(c_i\right)$ represents the final importance score for comment $c_i$; $\alpha$, $\beta$, and $\gamma$ are learnable weighting parameters that balance the contribution of relevance, credibility, and temporal factors, respectively; and the linear combination produces a comprehensive measure of comment informativeness for fake news detection.

The core idea of Algorithm 2 is to assign an importance weight to each user comment to identify the most valuable comments for fake news detection. The algorithm provides a systematic approach to evaluating comment quality across multiple dimensions. By combining relevance, credibility, and temporal factors, this algorithm automatically identifies the most valuable comments for fake news detection, focusing the model’s attention on expert opinions, fact-checking information, and timely responses while filtering out noise and irrelevant content.

Relevance Score ($S_{rel}$): By calculating the cosine similarity between comment features and news text features, it measures the degree of relevance between comments and main content. Comments with high relevance typically contain direct discussions or questioning of news content.

Credibility Score ($S_{cred}$): Through a specialized credibility scorer, it identifies credibility indicators in comments such as fact-checking information, source citations, and professional knowledge. These comments often come from users with professional backgrounds or contain verifiable information.

Temporal Importance ($S_{temp}$): An exponential decay function is used to model the publication time of comments. Early comments usually contain more original information and immediate reactions, making them more valuable for detecting fake news.

Algorithm 2 User comment importance scoring algorithm.

Require:  Comment set $F_C=\{c_1,c_2,\dots ,c_n\}$, text features $H_T$, hyperparameters $\alpha ,\beta ,\gamma ,\lambda$ Ensure:  Importance weight set $S_{importance}=\{s_1,s_2,\dots ,s_n\}$   1: for  $i=1$  n  do   2:        $S_{rel}\left(c_i\right)\leftarrow \mathrm{cosine\_similarity}(c_i,H_T)$                                                                                     ▹ Relevance score   3:        $S_{cred}\left(c_i\right)\leftarrow \mathrm{CredibilityScorer}\left(c_i\right)$                                                                                        ▹ Credibility score   4:        $S_{temp}\left(c_i\right)\leftarrow \exp (-\lambda ·(t_{current}-t_{comment}\left(c_i\right)))$                                                                       ▹ Temporal decay   5:        $s_i\leftarrow \alpha ·S_{rel}\left(c_i\right)+\beta ·S_{cred}\left(c_i\right)+\gamma ·S_{temp}\left(c_i\right)$                                                                        ▹ Weighted fusion   6: end for   7: Normalize $S_{importance}$ such that ${\sum }_{i=1}^n{s}_i=1$   8: return  $S_{importance}$

The final importance score is obtained through weighted linear combination and normalized to ensure that the sum of all comment weights equals 1. This design enables the model to automatically identify and focus on the most informative comments.

3.5.4. Aggregated Comment Representation

We compute the aggregated comment attention for text features:

$$\begin{array}{cc}\hfill C_{agg}& =\sum _i(S_{importance}\left(c_i\right)·c_i)\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill F_{TC}& =\mathrm{Concat}(H_T,C_{agg})\hfill \end{array}$$

(27)

This aggregation ensures that the most informative comments receive greater weight in the final representation. This process aggregates user comments based on their importance scores and integrates them with news text features. The weighted aggregation ensures that comments with higher credibility, relevance, and temporal importance contribute more significantly to the final multimodal representation, while the concatenation creates a unified feature vector that captures both original news content and filtered social feedback signals.

3.6. Fake News Detection

The detection framework achieves our third research objective through multi-level feature integration and interpretable prediction mechanisms that provide transparent decision-making processes.

3.6.1. Multi-Level Feature Integration

We combine features from different fusion paths to create a comprehensive multimodal representation that captures both content-based and social signal-based information.

The cross-modal attention features represent text–image fusion:

$$\begin{array}{c}\hfill F_{CMA}=\delta \end{array}$$

(28)

where $F_{CMA}$ represents the fused features from the cross-modal attention network, and $\delta$ denotes the integrated text–image representations that capture semantic correspondences between textual and visual content.

The comment–image fusion features capture visual information guided by user comments:

$$\begin{array}{c}\hfill F_{UCI}=F_{CI}\end{array}$$

(29)

where $F_{UCI}$ represents the User Comment Integration features for visual content, and $F_{CI}$ denotes the comment-informed visual features obtained from the comment-to-image attention mechanism.

The comment–text fusion features capture textual information guided by user comments:

$$\begin{array}{c}\hfill F_{UCT}=F_{TC}\end{array}$$

(30)

where $F_{UCT}$ represents the User Comment Integration features for textual content, and $F_{TC}$ denotes the comment-informed textual features obtained from the comment-to-text attention mechanism.

The global feature representation concatenates all fusion pathways:

$$\begin{array}{c}\hfill F_{global}=\mathrm{Concat}\left([F_{CMA},F_{UCI},F_{UCT}]\right)\end{array}$$

(31)

where $F_{global}$ represents the comprehensive feature vector, Concat denotes the concatenation operation, and the bracketed terms represent the feature vectors from cross-modal attention, comment–image fusion, and comment–text fusion, respectively.

3.6.2. Multi-Branch Prediction

Both the cross-modal attention network (CMA) and User Comment Integration networks (UCI/UCT) predict probability distributions for fake news classification through separate prediction branches.

The primary prediction utilizes cross-modal content features:

$$\begin{array}{c}\hfill y_1={\mathrm{Classifier}}_{CMA}\left(F_{CMA}\right)\end{array}$$

(32)

where $y_1$ represents the primary prediction probability distribution, ${\mathrm{Classifier}}_{CMA}$ denotes a specialized neural network classifier, and $F_{CMA}$ represents the cross-modal attention features that capture content-based evidence.

The secondary prediction leverages comment integration features:

$$\begin{array}{c}\hfill y_2={\mathrm{Classifier}}_{UCI\_UCT}\left(\mathrm{Concat}\left([F_{UCI},F_{UCT}]\right)\right)\end{array}$$

(33)

where $y_2$ represents the secondary prediction probability distribution, ${\mathrm{Classifier}}_{UCI\_UCT}$ denotes another specialized classifier, and the input combines both comment–image and comment–text fusion features to capture social signal-based evidence.

The final prediction combines both branches through weighted averaging:

$$\begin{array}{c}\hfill y_{final}=\lambda ·y_1+(1-\lambda )·y_2\end{array}$$

(34)

where $y_{final}$ represents the final prediction probability distribution, $\lambda$ is a learnable weight parameter that balances the contributions of content-based and comment-based evidence, $y_1$ represents the content-focused prediction, and $y_2$ represents the social signal-focused prediction.

3.6.3. Training Objectives

We adopt multiple loss functions to optimize different aspects of the model:

Primary Classification Loss:

$${\mathcal{L}}_{cls}=\mathrm{CrossEntropy}(y_{true},y_{final})$$

(35)

Auxiliary Classification Loss:

$${\mathcal{L}}_{aux}=\mathrm{CrossEntropy}(y_{true},y_1)+\mathrm{CrossEntropy}(y_{true},y_2)$$

(36)

Attention Regularization Loss:

$${\mathcal{L}}_{att}=\parallel A_{CI}{\parallel }_F+{\parallel A_{CT}\parallel }_F$$

(37)

Total Training Loss:

$${\mathcal{L}}_{total}={\mathcal{L}}_{cls}+{\alpha }_{aux}·{\mathcal{L}}_{aux}+{\alpha }_{att}·{\mathcal{L}}_{att}$$

(38)

where ${\alpha }_{aux}$ and ${\alpha }_{att}$ are hyperparameters controlling the relative importance of auxiliary objectives.

Algorithm 3 implements a multi-objective training strategy that balances primary classification performance with auxiliary learning goals. The comprehensive loss function ensures that each model component learns effective representations independently while maintaining overall system coherence, leading to robust performance across different types of misinformation scenarios.The algorithm designs multi-level loss functions to achieve comprehensive model optimization:

Primary Classification Loss (${\mathcal{L}}_{cls}$): Uses cross-entropy loss to train the final binary classification prediction, which is the main optimization objective of the model and directly relates to the accuracy of fake news detection.

Auxiliary Classification Loss (${\mathcal{L}}_{aux}$): Provides supervised training for cross-modal branches and comment integration branches separately. This design ensures that each branch can independently learn effective feature representations, avoiding over-dependence of one branch on others and improving model robustness.

Attention Regularization Loss (${\mathcal{L}}_{att}$): Regularizes attention matrices through the Frobenius norm to prevent attention weights from being overly concentrated on a few positions, promoting the model to learn more balanced and generalizable attention distributions.

Algorithm 3 Multi-objective training algorithm.

Require:  Ground truth labels $y_{true}$, primary prediction $y_1$, auxiliary prediction $y_2$, final prediction $y_{final}$, attention matrices $A_{CI},A_{CT}$, hyperparameters ${\alpha }_{aux},{\alpha }_{att}$ Ensure:  Total loss ${\mathcal{L}}_{total}$   1: // Primary Classification Loss   2: ${\mathcal{L}}_{cls}\leftarrow \mathrm{CrossEntropy}(y_{true},y_{final})$   3: // Auxiliary Classification Loss   4: ${\mathcal{L}}_{aux1}\leftarrow \mathrm{CrossEntropy}(y_{true},y_1)$                                                                                                                                                                           ▹ Cross-modal branch   5: ${\mathcal{L}}_{aux2}\leftarrow \mathrm{CrossEntropy}(y_{true},y_2)$                                                                                                                                                            ▹ Comment integration branch   6: ${\mathcal{L}}_{aux}\leftarrow {\mathcal{L}}_{aux1}+{\mathcal{L}}_{aux2}$   7: // Attention Regularization Loss   8: ${\mathcal{L}}_{att\_CI}\leftarrow {\parallel A_{CI}\parallel }_F$                                                                                                                                                                ▹ Comment-image attention regularization   9: ${\mathcal{L}}_{att\_CT}\leftarrow {\parallel A_{CT}\parallel }_F$                                                                                                                                                                   ▹ Comment-text attention regularization 10: ${\mathcal{L}}_{att}\leftarrow {\mathcal{L}}_{att\_CI}+{\mathcal{L}}_{att\_CT}$ 11: // Total Loss Computation 12: ${\mathcal{L}}_{total}\leftarrow {\mathcal{L}}_{cls}+{\alpha }_{aux}·{\mathcal{L}}_{aux}+{\alpha }_{att}·{\mathcal{L}}_{att}$ 13: return  ${\mathcal{L}}_{total}$

Through weighted combination of these three types of losses, the algorithm achieves a balance between main task performance, branch independence, and attention distribution quality. Hyperparameters ${\alpha }_{aux}$ and ${\alpha }_{att}$ allow adjustment of the relative importance of different loss terms according to specific datasets and task requirements, providing flexibility for model optimization.

4. Materials and Methods

4.1. Datasets

We evaluate our UC-CMAF framework on multiple benchmark datasets to demonstrate its generalizability and robustness across different domains and contexts (

Table 1. Dataset statistics and characteristics.

Metric	ReCOVery [63]	MMCoVaR [64]
Total Articles	2029	2593
Fake News	1364	1635
Real News	665	958
Images	1675	2357
Comments	140,820	Rich
Focus	COVID-19 General	COVID-19 Vaccine

).

ReCOVery Dataset: The ReCOVery dataset [63] contains 2029 COVID-19 related news articles collected from January 2020 to May 2020, including 1364 fake news articles and 665 real news articles. The dataset includes 1675 associated images and 140,820 user comments, making it ideal for evaluating multimodal fake news detection with social signals.

MMCoVaR Dataset: The MMCoVaR (Multimodal COVID-19 Vaccine focused Repository) dataset [64] comprises 2593 news articles (1635 fake and 958 real) with 2357 associated images. This dataset focuses specifically on COVID-19 vaccine misinformation and includes rich comment data for social signal analysis.

4.2. Implementation Details

To thoroughly evaluate the effectiveness of our UC-CMAF framework, we adopt a structured experimental pipeline. This includes data preprocessing, multimodal feature extraction from news content (text and image) and user comments, cross-modal fusion using attention mechanisms, and model training with evaluation on benchmark datasets. Each step is designed with reproducibility and clarity in mind, and detailed settings are provided to ensure transparent experimental procedures.

Our multimodal embedding method employs carefully selected pre-trained models and architectures for feature extraction and fusion. For text processing, we utilize the pre-trained BERT-base-uncased model to extract 768-dimensional text features, with maximum sequence lengths set to 512 tokens for news articles and 128 tokens for comments, using a vocabulary of 30,522 tokens. Visual processing adopts ResNet-50 for feature extraction, reducing 2048-dimensional features to 768 dimensions through 2D convolution to maintain dimensional consistency. Image preprocessing includes resizing to 224 × 224 pixels and normalization using ImageNet statistics, with data augmentation applied during training through random horizontal flipping and color jittering. For comment processing, we select up to 50 comments per article (20 for the MMCoVaR dataset), with comments filtered by minimum length of 10 characters and maximum length of 280 characters, while preserving the temporal order of all comments. For user comments, we perform a multi-step preprocessing pipeline: (1) removal of non-linguistic content such as URLs, emojis, and special characters; (2) filtering of comments that are too short or too long; (3) BERT-based tokenization and normalization; and (4) retention of timestamp metadata for use in the temporal importance weighting module. This ensures that only informative and high-quality comments are incorporated into the model. The model is trained using the Adam optimizer with a learning rate of 0.001, a batch size of 64, and for 200 epochs. These settings are selected via cross-validation to ensure a balance between convergence speed and generalization. Xavier initialization is applied to stabilize training dynamics. For the ReCOVery and MMCoVaR datasets, we randomly sample up to 50 and 20 comments per article, respectively, padding with empty comments when necessary. The fusion weight parameter $\lambda$ for integrating user reviews is set to 0.8 based on experimental validation.

4.3. Evaluation Metrics

We adopt comprehensive evaluation metrics to assess model performance across multiple dimensions: accuracy measures overall classification correctness, precision evaluates the true positive rate which is important for minimizing false accusations, recall assesses sensitivity to actual fake news instances, and F1 Score provides the harmonic mean of precision and recall to balance both metrics.

4.4. Baseline Methods

We compare UC-CMAF against state-of-the-art methods from different categories to demonstrate its effectiveness:

Traditional Machine Learning Approaches:

• SVM-based [30]: Support Vector Machine utilizing TF-IDF features for text-based classification.
• Random Forest [25]: Ensemble method employing handcrafted linguistic and statistical features.

Deep Learning Baselines:

• BERT-only [65]: Fine-tuned BERT model for text classification without multimodal integration.
• ResNet-only [66]: CNN-based image classification using ResNet-50 architecture.
• Simple Concatenation [28]: Basic feature concatenation without attention mechanisms or sophisticated fusion strategies.

Multimodal Methods:

• HSA [35]: Hierarchical Social Attention for rumor detection through social networks and user comments.
• ExFaux [67]: A weakly supervised approach to explainable fauxtography detection.
• dEFEND [39]: Explainable fake news detection leveraging news–comment correlations with co-attention mechanisms.
• BTIC [68]: Supervised contrastive learning for multimodal unreliable news detection in COVID-19 pandemic.
• SAFE [26]: Similarity-Aware multimodal Fake nEws detection with cross-modal attention.
• HMCAN [69]: Hierarchical multimodal contextual attention network for fake news detection.
• SpotFake [10]: Framework leveraging pre-trained BERT and VGG-19 for Tweet verification.
• MVAE [16]: Multimodal Variational AutoEncoder generating representations through dual-modal VAE.
• DGExplain [40]: A duo-generative approach to explainable multimodal COVID-19 misinformation detection.

5. Results and Discussion

5.1. Overall Performance Comparison

Our comprehensive evaluation demonstrates that UC-CMAF achieves superior performance across multiple dimensions while maintaining high interpretability standards.

Table 2. Performance comparison of main data collection methods.

Method	ReCOVery				MMCoVaR
	Accuracy	Precision	Recall	F1	Accuracy	Precision	Recall	F1
SVM-Based	0.695	0.678	0.681	0.679	0.712	0.698	0.703	0.700
Random Forest	0.734	0.719	0.728	0.723	0.748	0.731	0.742	0.736
BERT-only	0.821	0.809	0.798	0.803	0.834	0.823	0.812	0.817
ResNet-only	0.663	0.652	0.671	0.661	0.678	0.665	0.683	0.674
Simple Concatenation	0.756	0.742	0.751	0.746	0.769	0.753	0.762	0.757
HSA	0.779	0.737	0.736	0.736	0.803	0.782	0.785	0.784
ExFaux	0.763	0.719	0.695	0.704	0.769	0.784	0.694	0.707
dEFEND	0.856	0.826	0.813	0.823	0.856	0.847	0.831	0.838
BTIC	0.763	0.719	0.695	0.704	0.829	0.823	0.791	0.803
SAFE	0.831	0.803	0.789	0.795	0.788	0.773	0.749	0.757
EANN	0.847	0.816	0.834	0.824	0.833	0.819	0.810	0.814
SpotFake	0.681	0.637	0.650	0.641	0.699	0.670	0.620	0.623
MVAE	0.825	0.813	0.755	0.774	0.815	0.805	0.834	0.808
DGExplain	0.897	0.890	0.861	0.873	0.895	0.896	0.871	0.881
UC-CMAF	0.927	0.916	0.872	0.894	0.931	0.939	0.881	0.909

and Figure 2 present comprehensive performance comparisons between our proposed UC-CMAF and various baseline methods on both the ReCOVery and MMCoVaR datasets, with evaluations using precision, recall, and F1 Score metrics across 15 representative multimodal fake news detection methods. UC-CMAF consistently achieves the best performance across all evaluation metrics, demonstrating its superiority in multimodal fake news detection. Specifically, on the ReCOVery dataset, UC-CMAF achieves an accuracy of 0.927, precision of 0.916, recall of 0.872, and an F1 Score of 0.894, outperforming all other models and surpassing the strongest baseline DGExplain (F1: 0.873) by 2.1%. On the more challenging MMCoVaR dataset, UC-CMAF further improves its performance with the highest precision of 0.939, recall of 0.881, and F1 Score of 0.909, exceeding DGExplain (F1: 0.881) by a margin of 2.8%. In contrast, traditional machine learning methods such as SVM-Based and Random Forest perform significantly worse with F1 Scores of only 0.679/0.700 and 0.723/0.736, respectively, while single-modal methods like BERT-only show relatively stable results (F1: 0.803 and 0.817) but still fall short of UC-CMAF, and unimodal baselines such as ResNet-only achieve even lower F1 Scores (0.661 on ReCOVery; 0.674 on MMCoVaR), underscoring the limitations of relying solely on either text or image modalities. Some multimodal methods such as dEFEND and EANN (F1: 0.823/0.838 and 0.824/0.814, respectively) also underperform in comparison, and several approaches including BTIC, SAFE, and ExFaux exhibit noticeable fluctuations in performance across datasets, suggesting limited generalizability. UC-CMAF maintains strong and consistent performance on both datasets, validating its robustness and ability to generalize across different scenarios with different semantic and stylistic characteristics. These findings confirm that UC-CMAF’s unified cross-modal attention mechanism enables it to effectively model fine-grained interactions between textual and visual modalities, leading to superior detection performance in complex multimodal fake news detection tasks, with the visual annotations clearly demonstrating the consistent superiority of the UC-CMAF approach over all baselines.

Figure 3 presents the accuracy and F1 Score trends of 15 representative models on the ReCOVery and MMCoVaR datasets. Overall, models that leverage multimodal fusion strategies consistently outperform unimodal baselines such as BERT-only and ResNet-only, demonstrating the importance of integrating textual and visual cues in fake news detection. On both datasets, the proposed UC-CMAF model shows clear superiority over all other methods. Specifically, UC-CMAF achieves the highest accuracy and F1 score on both datasets, with values of 0.927 and 0.894 on ReCOVery, and 0.931 and 0.909 on MMCoVaR, respectively. These results indicate not only strong predictive power but also robust generalization across different distributions of misinformation. Close competitors such as DGExplain, MVAE, and dEFEND also exhibit relatively strong performance, but their F1 Scores remain 2–3% lower than that of UC-CMAF, underscoring the effectiveness of our proposed cross-modal attention fusion strategy. Additionally, traditional machine learning approaches like SVM-Based and Random Forest display limited performance, further validating the advantages of deep, modality-aware architectures.

Figure 4 provides a comprehensive visualization of the performance metrics for 15 representative models across the ReCOVery and MMCoVaR datasets. Each heatmap illustrates four standard evaluation metrics—accuracy, precision, recall, and F1 Score—for every method. On the ReCOVery dataset, our proposed UC-CMAF model clearly outperforms all baselines across all metrics, with an accuracy of 0.894, precision of 0.927, recall of 0.916, and an F1 Score of 0.872. This demonstrates the model’s ability to effectively capture fine-grained cross-modal interactions and maintain semantic consistency in multimodal news content. Similarly, on the MMCoVaR dataset, UC-CMAF once again leads in every metric, achieving an accuracy of 0.909, precision of 0.931, recall of 0.939, and F1 Score of 0.881. The consistently high scores across datasets of different characteristics confirm the model’s strong generalization capability. By contrast, traditional machine learning approaches such as SVM-Based and Random Forest perform noticeably worse, particularly in recall and F1 Score. Unimodal baselines like BERT-only and ResNet-only also lag behind, reaffirming the necessity of multimodal fusion. Mid-tier multimodal methods like DGExplain and dEFEND perform competitively but still fall short of UC-CMAF, especially in recall on MMCoVaR. Overall, these heatmaps highlight UC-CMAF’s superior performance and robustness in multimodal fake news detection.

These empirical improvements across all evaluation metrics confirm that the proposed adaptive fusion mechanism effectively captures and leverages complementary signals from text, images, and user comments.

5.2. Analysis of UC-CMAF Components

-CMA (w/o Cross-Modal Attention): This component serves as the core mechanism for connecting information from different modalities, responsible for learning the interactive relationships among text, images, and user comments. The cross-modal attention mechanism dynamically fuses multimodal features through attention weights, enabling the model to capture semantic associations across modalities, identify consistency and contradictions between different information sources, thereby providing crucial judgment basis for fake news detection.

-UCI (w/o Comment–Image Fusion): This component specifically handles the fusion process between user comments and image content. Through deep learning techniques, it combines users’ textual comments with visual information, capturing users’ emotional tendencies and semantic understanding of image content. This fusion mechanism can identify users’ judgments about image authenticity in their comments, as well as the degree of matching between images and comment content.

-UCT (w/o Comment–Text Fusion): This component is responsible for effectively fusing user comments with original text content. Through contrastive learning or attention mechanisms, it identifies consistency, contradictions, or complementary information between user comments and the main text, mining users’ collective judgment on text content credibility and providing verification information from the user perspective to the model.

-Comments (w/o User Comments): This component collects and processes feedback information from users, which typically contains rich subjective judgments and emotional expressions. The user comment module can capture collective wisdom in social media environments, identifying users’ questioning, support, or neutral attitudes toward content authenticity, providing important social verification signals for the model.

-Images (Text + Comments Only): This component is responsible for processing and analyzing visual content, extracting image features and identifying visual clues that may indicate misleading or false information. The image module analyzes technical characteristics, semantic content, and potential tampering traces in images through computer vision techniques, providing visual evidence support for multimodal fake news detection.

-Adaptive Weights: This component dynamically adjusts the importance weights of various modalities and components according to the characteristics of different samples, implementing personalized multimodal information fusion strategies. The adaptive weight mechanism can automatically optimize the contribution of different information sources based on input data characteristics, improving the model’s adaptability and accuracy when facing different types of fake news.

The results of the ablation study (

Table 3. Ablation study results.

Removed Components	ReCOVery				MMCoVaR
	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1
None (Full Model)	0.927	0.916	0.872	0.894	0.931	0.939	0.881	0.909
-CMA (w/o Cross-Modal Attention)	0.902	0.883	0.836	0.856	0.914	0.903	0.862	0.887
-UCI (w/o Comment–Image Fusion)	0.895	0.846	0.858	0.851	0.901	0.888	0.861	0.881
-UCT (w/o Comment–Text Fusion)	0.898	0.884	0.811	0.853	0.902	0.895	0.861	0.883
-Comments (w/o User Comments)	0.883	0.867	0.823	0.844	0.886	0.874	0.845	0.859
-Images (Text + Comments Only)	0.908	0.895	0.851	0.872	0.919	0.911	0.867	0.888
-Adaptive Weights	0.912	0.901	0.847	0.873	0.923	0.916	0.871	0.893

) indicate that the complete versions of both models exhibit excellent performance, with MMCoVaR (accuracy: 0.931; F1 Score: 0.909) slightly outperforming ReCOVery (accuracy: 0.927; F1 Score: 0.894) across all metrics, demonstrating the superior effectiveness of its architectural design. An analysis of the importance of each component reveals that the user comment module is the most critical in both models, with its removal leading to the largest decline in F1 Score (approximately 0.050 for both ReCOVery and MMCoVaR), indicating that user comments contain vital information that traditional text and image analysis struggle to capture and underscoring the irreplaceable value of subjective judgment and collective intelligence in detecting misinformation. The cross-modal attention mechanism and the User Comment Integration components (UCI/UCT) rank second in importance, highlighting the limitations of unimodal information and demonstrating that multimodal fusion significantly enhances detection capabilities, with effective integration across different modalities being essential for model success. In contrast, the removal of the image module has a relatively smaller impact on performance (F1 Score drops by approximately 0.02), but it remains non-negligible, suggesting that images play a supplementary and validating role by providing visual cues that are not expressed in text. Although the adaptive weighting mechanism has the smallest impact when removed, it still contributes to stable performance improvements, reflecting the value of dynamic weight adjustment in optimizing information fusion. Overall, MMCoVaR demonstrates greater robustness across all ablation experiments, while ReCOVery shows higher dependency on its key components, with the strong reliance of both models on user comments highlighting the critical role of user feedback in misinformation detection within social media environments.

Figure 5 presents the impact of module ablations on model performance across the ReCOVery and MMCoVaR datasets, where Figure 5a shows the changes in model accuracy when individual components are removed, clearly demonstrating that the removal of the cross-modal attention module (-CMA) and the user comments module (-Comments) results in substantial accuracy drops, highlighting their importance in capturing multimodal associations and integrating user feedback, while the removal of the adaptive weight component leads to relatively minor impact on accuracy. Figure 5b illustrates the corresponding F1 Score variations under the same ablation settings, with the full model achieving the highest F1 Score, while the removal of the CMA, UCI, and comment modules causes notable declines, reaffirming their critical roles in fake news detection, particularly when the cross-modal attention (-CMA) module is removed, as the F1 degradation is especially pronounced, indicating its pivotal contribution to balancing precision and recall in the final predictions.

Figure 6 presents radar charts illustrating the ablation study results on the ReCOVery and MMCoVaR datasets, clearly validating the effectiveness and synergy of the proposed model components. The full model consistently achieves the highest scores across all four metrics—accuracy, precision, recall, and F1—demonstrating the overall advantage of our multimodal fusion strategy. Notably, removing the cross-modal attention (CMA) component leads to a significant performance drop, indicating its crucial role in modeling the interactions among text, images, and user comments. Similarly, the exclusion of the user comment module results in considerable degradation, highlighting the importance of leveraging social feedback signals in fake news detection. Furthermore, the removal of the comment–image (UCI) and comment–text (UCT) fusion components also negatively impacts performance, underscoring the value of semantic alignment between user perspectives and content. Overall, the proposed model exhibits strong robustness and accuracy in detecting fake news, benefiting from fine-grained multimodal integration and an adaptive weighting mechanism tailored to different input scenarios.

The ablation study clearly demonstrates that each component of UC-CMAF, particularly the user comment module and cross-modal attention, plays a vital role in achieving high detection accuracy, thereby justifying their inclusion in the final architecture.

5.3. Attention Mechanism Analysis

To validate the effectiveness and rationale behind our attention mechanism design, we compare several attention variants and analyze the trade-off between performance and computational complexity through visualizations. As illustrated in Figure 7 and

Table 4. Comparison of attention mechanisms.

Attention Variant	ReCOVery F1	CoVaR F1	Complexity	Description
No Attention	0.844	0.859	$\mathcal{O}\left(n\right)$	Simple direct connection
Single-Head	0.867	0.881	$\mathcal{O}\left(n^2\right)$	Basic attention mechanism
Multi-Head	0.873	0.896	$\mathcal{O}\left(n^2\right)$	Standard multi-head attention
Cross-Modal	0.889	0.903	$\mathcal{O}\left(nm\right)$	Cross-modal attention only
Adaptive Attention (Ours)	0.894	0.909	$\mathcal{O}\left(nm\right)$	Our integrated adaptive attention mechanism

, the F1 Scores on both ReCOVery and CoVaR tasks consistently improve from models with no attention to single-head, multi-head, and cross-modal attention, culminating in the highest scores achieved by our proposed adaptive attention. The performance–complexity scatter plot demonstrates that although adaptive attention has a computational complexity of $\mathcal{O}\left(nm\right)$, it significantly outperforms other variants in terms of average F1 Score, achieving the best balance between effectiveness and efficiency. Additionally, the bar–line combination plot presents both the performance metrics and encoded complexity levels, further highlighting that adaptive attention offers superior performance without incurring additional computational overhead. These results collectively confirm the advantages of our design and the practical value of integrating adaptive attention in our architecture.

5.4. Weight Parameter Sensitivity

As shown in Figure 8, the introduction of the weight $\lambda$ significantly improves the performance of the model, particularly in terms of accuracy and F1 Score. By observing the performance changes at different $\lambda$ values, it is evident that both ReCOVery and MMCoVaR show significant improvement as $\lambda$ increases from 0 to 0.8. In particular, at $\lambda =0.8$, both models achieve higher accuracy and F1 Scores, and their performance remains relatively stable in this region. Furthermore, MMCoVaR exhibits stronger generalization ability at higher $\lambda$ values, especially at $\lambda =0.8$, where its accuracy reaches 0.931 and F1 Score reaches 0.909, surpassing the performance of the ReCOVery model. Based on these results, selecting a $\lambda$ value of around 0.8 as the weight ensures that we maintain high performance while avoiding performance degradation when the parameter is too high.

5.5. Impact of Comment Selection Strategy

Figure 9 and

Table 5. Comment selection strategy analysis.

Selection Strategy	ReCOVery Accuracy	MMCoVaR Accuracy	Processing Time
Random Selection	0.902	0.915	156 ms
Time Order	0.908	0.921	143 ms
By Length	0.911	0.923	167 ms
By Engagement	0.918	0.926	189 ms
importance Score	0.927	0.931	203 ms

illustrate the impact of various comment selection strategies on model accuracy and processing time. As shown, the importance score strategy achieves the highest accuracy for both ReCOVery (92.7%) and MMCoVaR (93.1%), demonstrating the effectiveness of using a hierarchical importance mechanism. Although this approach incurs a higher processing time (203 ms), the trade-off is justified by its superior predictive performance. In contrast, simpler strategies such as random or time order selection offer faster processing times but result in lower accuracy. This highlights the importance of intelligent comment selection for optimizing both performance and computational efficiency in misinformation detection tasks. Our hierarchical importance scoring mechanism provides the best performance, despite a slight increase in computational cost.

5.6. Impact of Number of Comments

Figure 10 illustrates the trend of the ReCOVery model’s accuracy and training time as the number of annotated comments increases. It can be observed that the accuracy steadily improves from 0.918 with 10 comments to 0.928 with 100 comments, indicating that additional human feedback contributes positively to model performance. However, the training time also increases significantly, especially beyond 50 comments, where the duration grows from 3.2 to 4.8 h. This suggests a nonlinear relationship between annotation volume and computational cost.

Figure 11 provides a more comprehensive comparison, showing the performance of both the ReCOVery and MMCoVaR models. MMCoVaR consistently achieves higher accuracy across all annotation levels, with the performance gap widening as more comments are incorporated, demonstrating MMCoVaR’s superior ability to leverage annotated information. In terms of resource consumption, both memory usage and training time increase with the number of comments. Nonetheless, the overall resource usage remains within acceptable bounds, indicating that MMCoVaR achieves a favorable balance between accuracy and computational efficiency.

To evaluate the practical deployment feasibility of UC-CMAF, we conducted comprehensive computational cost analysis covering both training and inference scenarios (

Table 6. Impact of number of comments.

Number of Comments	ReCOVery Accuracy	MMCoVaR Accuracy	Memory Usage	Training Time
10	0.918	0.924	1.8 GB	2.1 h
20	0.923	0.928	2.1 GB	2.4 h
30	0.925	0.930	2.3 GB	2.7 h
50	0.927	0.931	2.9 GB	3.2 h
100	0.928	0.932	4.1 GB	4.8 h

).

Inference Performance Metrics: Our full UC-CMAF model with 50 comments requires an average inference time of 203 ms per sample. The memory footprint during inference reaches 2.9 GB, primarily allocated for storing comment embeddings and attention matrices across multiple fusion components.

Baseline Computational Comparison: While UC-CMAF introduces computational overhead compared to simpler baselines (BERT-only: 45 ms; Simple Concatenation: 78 ms), this additional cost is justified by substantial performance improvements. The 2.6× increase in inference time corresponds to a 9.1% improvement in F1-score on ReCOVery dataset, demonstrating favorable cost–performance trade-offs.

Scalability Bottleneck Analysis: Profiling analysis reveals that cross-modal attention computation constitutes the primary computational bottleneck, accounting for approximately 60% of total inference time. The comment–image fusion (UCI) and comment–text fusion (UCT) modules contribute 35% and 25% respectively to the overall computational load. Memory consumption scales linearly with comment volume, with each additional comment requiring approximately 58 MB.

Deployment Optimization Strategies: For resource-constrained environments, we recommend adaptive comment sampling (reducing to 20 comments) which maintains 99.7% of full model accuracy while achieving 30% speed improvement. Additionally, batch processing techniques can improve throughput by 40% for large-scale deployment scenarios.

5.7. Attention Visualization

We analyze the semantic alignment between text and images using the cross-modal attention (CMA) module. As shown in Figure 12, the model significantly focuses on the “data chart” region of the image when processing text tokens related to semantics such as “vaccine”, “efficacy”, and “95%”. The highest attention weight reaches 0.90, indicating that the model effectively associates quantitative information like vaccine efficacy with the data visualization in the image. Simultaneously, for words like “clinical” and “trial”, the model mainly focuses on the “lab scene” region, with attention weights of 0.80 and 0.70, respectively, further validating the model’s effectiveness in understanding medical experiment scenarios. Overall, this module demonstrates strong cross-modal semantic alignment capabilities, which help enhance the model’s performance in medical text–image tasks.

As shown in Figure 13, UC-CMAF assigns high weights of $0.75$, $0.80$, and $0.70$ to Evidence-Doubt, Procedure-Doubt, and Judgment-Support on their logically corresponding dimensions of “Evidence Sufficiency”, “Procedure Legitimacy”, and “Scientific Accuracy”, respectively, while cross-dimension weights remain between $0.05$ and $0.15$, indicating effective suppression of semantic interference. At the same time, Emotional-Tendency distributes evenly across all four dimensions with weights between $0.20$ and $0.30$ (with $0.30$ on both “Evidence Sufficiency” and “Scientific Accuracy”), ensuring comprehensive coverage of emotional context. In the stacked bar chart on the right, 95% of the weight for the “Evidence Sufficiency” dimension is contributed by Evidence-Doubt ($0.75$) and Emotional-Tendency ($0.30$); for “Procedure Legitimacy”, Procedure-Doubt ($0.80$) and Emotional-Tendency ($0.20$) comprise the entire weight; “Scientific Accuracy” is dominated by Judgment-Support ($0.70$) and Emotional-Tendency ($0.30$); and “Data Reliability” is primarily composed of Emotional-Tendency ($0.20$) and Judgment-Support ($0.15$), with other categories contributing less than $0.10$. This distribution both validates UC-CMAF’s high discriminative power in fine-grained intent classification and demonstrates its ability to deeply parse subjective emotional cues.

As shown in the attention alignment in Figure 14, the UCT module demonstrates highly focused comment–text alignment: Comment 1 places a weight of $0.75$ on “efficacy” (all other tokens receive weights below $0.10$); Comment 2 attends most strongly to “95%” with $0.70$, and retains minor attention on “efficacy” ($0.15$) and “phase 3” ($0.05$); Comment 3 concentrates on “phase 3” with $0.60$, while assigning $0.10$ each to “clinical” and “efficacy”. Correspondingly, UC-CMAF assigns the highest importance score of $0.92$ to Comment 2, with Comment 1 and Comment 3 scoring $0.85$ and $0.78$, respectively, perfectly mirroring the UCT attention distributions. These results demonstrate that UCT can precisely align user comments with key textual terms, and that UC-CMAF can effectively discriminate comment priority via quantified importance, validating the model’s alignment accuracy and discriminative power in multi-comment, multi-text scenarios.

5.8. Robustness Analysis

To comprehensively evaluate the stability and reliability of our model in practical applications, we designed robustness testing experiments from two dimensions: noise tolerance and adversarial attack resistance. The noise tolerance test employs four typical noise types (text synthesis, image Gaussian blur, comment injection, and combined noise) to assess model performance degradation under different intensities from 10 dB to 50 dB. The adversarial attack test designed four attack strategies (text perturbation, image adversarial, comment manipulation, and multimodal attack) and validated the effectiveness of the UC-CMAF defense mechanism.

(1) Noise Tolerance Evaluation

The results presented in

Table 7. Robustness against different noise types.

Noise Type	Clean Acc.	10 dB	20 dB	30 dB	50 dB
Text Synthesis	0.927	0.921	0.913	0.904	0.887
Image Gaussian Blur	0.927	0.924	0.920	0.914	0.896
Comment Injection	0.927	0.923	0.918	0.912	0.891
Combined Noise	0.927	0.918	0.906	0.892	0.868

demonstrate that our system exhibits good robustness against various noise types, maintaining satisfactory performance even under enhanced noise conditions. Under extreme 50 dB noise environments, image Gaussian blur has the minimal impact (only 4.7% degradation), while combined noise shows the most significant effect (6.4% degradation), indicating that the superposition effect of multi-source noise causes greater impact on system performance. Overall, performance degradation under all noise types is controlled within acceptable ranges, proving that the model possesses good anti-interference capabilities.

(2) Adversarial Attack Resistance

UC-CMAF demonstrates strong resistance against adversarial attacks, particularly excelling in defending against multimodal attack methods, where single-modal attacks can be identified through inconsistency detection with other modalities. The experimental results presented in

Table 8. Performance under adversarial attacks.

Attack Method	Success Rate	Accuracy Drop	Defense Effectiveness
Text Perturbation	12.3%	−3.2%	87.7%
Image Adversarial	8.7%	−2.1%	91.3%
Comment Manipulation	15.6%	−4.1%	84.4%
Multimodal Attack	18.9%	−5.3%	81.1%

show that image adversarial attacks have the lowest original success rate (8.7%), and after defense, the system accuracy can still maintain 91.3%, demonstrating the best defense effectiveness. In contrast, multimodal coordinated attacks represent the most challenging attack method with an original success rate of 18.9%, but the UC-CMAF defense mechanism can still control it within an acceptable range, achieving 81.1% accuracy after defense.

To comprehensively evaluate the robustness of the model under complex environmental conditions, Figure 15a, Figure 15b, and Figure 15c respectively illustrate the model’s accuracy under varying noise intensities, performance degradation under different adversarial attack strategies, and a heatmap showing key robustness metrics for each attack type.

Figure 15a presents the accuracy trend under different types of noise, with the x-axis representing noise intensity (in dB) and the y-axis representing classification accuracy. It can be observed that as the noise intensity increases, the model performance degrades gradually, but the overall decline is relatively small, indicating strong noise robustness. Among the noise types, the model shows the highest tolerance to image Gaussian blur, while combined noise (multi-source) has the most significant impact.

Figure 15b displays, in the form of a stacked bar chart, the attack success rate, accuracy drop, and defense effectiveness under four types of adversarial attacks. The results show that the model performs most robustly against image-based adversarial attacks, maintaining over 91% accuracy after defense. Although multimodal attacks have the highest success rate (18.9%), the UC-CMAF defense mechanism still effectively limits performance degradation, achieving a defense effectiveness of 81.1%.

Figure 15c shows a heatmap of the relationships between attack types and key robustness metrics. The color intensity indicates the metric value, with darker shades representing better performance. It is evident that different attack strategies affect system performance differently: image-based attacks show the best robustness, while multimodal attacks pose the greatest challenge. The heatmap confirms the trends observed in the previous figures and further highlights the UC-CMAF model’s potential in defending against complex adversarial threats.

6. Conclusions

6.1. Summary and Conclusions

This paper introduces UC-CMAF (User Comment-Guided Cross-Modal Attention Fusion), a novel framework designed for interpretable multimodal fake news detection in the digital era. Our primary contribution lies in developing a pioneering fusion architecture that effectively integrates news text, images, and user comments through adaptive cross-modal attention mechanisms. Critically, we treat user comments as a primary modality rather than auxiliary data, recognizing their essential role in understanding the broader social context of information dissemination.

The experimental evaluation demonstrates UC-CMAF’s superior performance across multiple benchmark datasets. Our framework achieves remarkable results with 92.7% accuracy and 89.4% F1 Score on the ReCOVery dataset, and 93.1% accuracy with 90.9% F1 Score on MMCoVaR, significantly outperforming existing state-of-the-art methods. Beyond performance metrics, UC-CMAF addresses the critical demand for explainable AI through comprehensive multi-head attention heatmaps and comment importance visualizations, while maintaining robust performance against adversarial attacks with defense effectiveness rates ranging from 81.1% to 91.3%.

Our comprehensive ablation studies provide compelling evidence that user comments contain unique contextual and social information that extends far beyond traditional content-based analysis capabilities. This finding validates our architectural decision to elevate user comments to primary modality status and underscores the importance of incorporating social validation signals into misinformation detection systems. UC-CMAF’s consistent superior performance across diverse datasets, coupled with its robust interpretability mechanisms and strong adversarial resistance, represents a significant advancement in the field of multimodal fake news detection.

The framework successfully demonstrates how AI systems can effectively combine rigorous content analysis with social validation while maintaining comprehensive interpretability—a critical requirement for deployment in real-world scenarios where trust and transparency are paramount. This work establishes several important directions for future research: (1) development of real-time processing capabilities for immediate misinformation detection, (2) cross-platform generalization to handle diverse social media environments, and (3) implementation of adaptive mechanisms capable of responding to emerging manipulation techniques.

By successfully integrating social context with sophisticated multimodal analysis, UC-CMAF offers a new paradigm for developing robust, explainable verification systems capable of combating increasingly sophisticated misinformation campaigns. As digital misinformation continues to evolve in complexity and scale, frameworks like UC-CMAF provide essential foundations for maintaining information integrity while preserving the transparency and accountability necessary for fostering public trust in AI-assisted fact-checking systems.

6.2. Limitations and Future Directions

While UC-CMAF demonstrates strong performance across multiple datasets, several limitations warrant discussion for future research directions.

Both ReCOVery and MMCoVaR are centered around COVID-19-related misinformation, which may introduce topic-specific linguistic, visual, and social biases. For instance, common text tokens (e.g., “vaccine,” “clinical trial”) and visual patterns (e.g., medical diagrams or bar charts) may dominate feature learning, while user comments tend to reflect highly polarized or emotionally charged discourse. As a result, the model may learn domain-specific representations that do not generalize well to topics such as politics, finance, or AI-generated content. However, the modular design of UC-CMAF—including separate CMA, UCI, and UCT components—allows for efficient adaptation to new domains via fine-tuning without altering the core architecture. Future work should evaluate the framework on a broader range of datasets, such as PolitiFact or GossipCop, to systematically assess its robustness across diverse semantic and visual distributions.

Comment Availability Dependency: Our framework’s effectiveness is fundamentally tied to the presence and quality of user comments. In scenarios where comments are sparse—such as newly published articles, niche topics, or platforms with limited user engagement—the model’s performance may degrade as it relies more heavily on traditional text–image analysis. Future work should explore hybrid approaches that can gracefully handle comment-scarce environments.

Cross-linguistic and Cross-platform Adaptability: The current evaluation focuses primarily on English-language content from specific social media platforms. Extending to multilingual scenarios presents challenges, including (1) linguistic variations in commenting behavior across cultures, (2) different misinformation propagation patterns in various linguistic communities, and (3) platform-specific user interaction mechanisms. Future research should investigate transfer learning approaches and culturally aware adaptation strategies.

Temporal Dynamics: Our model treats comment collections as relatively static, but real-world comment threads evolve continuously. The temporal importance scoring mechanism may need enhancement to better capture the dynamic nature of evolving misinformation narratives and user response patterns.

Computational Scalability: As demonstrated in our analysis, processing large numbers of comments significantly increases computational overhead. While our importance scoring helps mitigate this issue, real-time deployment scenarios may require further optimization strategies or adaptive sampling techniques.

Adversarial Robustness: Although our framework shows resistance to various attack types, sophisticated adversaries might develop coordinated comment manipulation strategies specifically targeting our importance scoring mechanism, necessitating ongoing defensive improvements.

Multimodal Perception and Social Signal Integration: Although UC-CMAF is evaluated empirically, its design is inspired by cognitive principles of multimodal perception and social reasoning. Human perception tends to integrate semantically consistent signals across modalities while filtering out irrelevant inputs—an idea reflected in our bidirectional adaptive attention mechanism, which selectively enhances aligned text–image regions and dynamically adjusts fusion weights based on contextual relevance. In addition, our use of user comments draws on the notion that social discourse conveys distributed credibility signals; comments often express skepticism, support, or counter-evidence that complement or challenge the source content. By leveraging these social cues through dedicated attention pathways, UC-CMAF models a form of collective reasoning, enhancing its effectiveness in detecting multimodal misinformation.

These limitations provide clear directions for future research, including real-time processing optimization, multilingual adaptation, and enhanced adversarial robustness mechanisms.