A BERT-Based Multimodal Framework for Enhanced Fake News Detection Using Text and Image Data Fusion

Abstract

The spread of fake news on social media is complicated by the fact that fake information spreads extremely fast in both textual and visual formats. Traditional approaches to the detection of fake news focus mainly on text and image features, thereby missing valuable information contained within images and texts. In response to this, we propose a multimodal fake news detection method based on BERT, with an extension to text combined with the extracted text from images through Optical Character Recognition (OCR). Here, we consider extending feature analysis with BERT_base_uncased to process inputs for retrieving relevant text from images and determining a confidence score that suggests the probability of the news being authentic. We report extensive experimental results on the ISOT, WELFAKE, TRUTHSEEKER, and ISOT_WELFAKE_TRUTHSEEKER datasets. Our proposed model demonstrates better generalization on the TRUTHSEEKER dataset with an accuracy of 99.97%, achieving substantial improvements over existing methods with an F1-score of 0.98. Experimental results indicate a potential accuracy increment of +3.35% compared to the latest baselines. These results highlight the potential of our approach to serve as a strong resource for automatic fake news detection by effectively integrating both text and visual data streams. Findings suggest that using diverse datasets enhances the resilience of detection systems against misinformation strategies.

1. Introduction

In recent years, the proliferation of fake news on social media has emerged as a serious challenge to public opinion, democratic processes, and societal trust. This challenge is intensified by the increasing reliance on digital platforms as the primary news source, making the public more vulnerable to misinformation [1]. According to a recent survey, 54% of U.S. adults obtain news from social media platforms, with Facebook and YouTube being the most preferred, followed by Instagram, TikTok, and X (formerly Twitter) [2]. In the same study, 67% of respondents identified Facebook, 65% social media in general, and 60% the broader internet as key sources of fake news exposure—significantly more than traditional outlets like television and print media [3].

This convergence of news consumption and misinformation exposure underscores the need for advanced detection mechanisms tailored to social media environments. Social media content is inherently multimodal, often combining text, images, and videos, which complicates the detection of fake news. In response, we propose a novel multimodal detection framework based on BERT (Bidirectional Encoder Representations from Transformers) to address the evolving nature of misinformation on social platforms [4].

While social media has democratized access to information, it has also fueled the spread of fake news—defined as deliberately misleading information—which poses risks to democracy, public health, and social cohesion [5,6]. Platforms like X, Facebook, and Instagram enable users to instantly share content, often without editorial oversight. This lack of gatekeeping facilitates misinformation campaigns, as seen in the 2016 U.S. election and during the COVID-19 pandemic, where online content influenced both public behavior and policy decisions [7,8].

Traditional fake news detection techniques rely on linguistic features such as sentiment, lexical diversity, and syntax, analyzed through machine learning [9]. While these methods are effective to some extent, they struggle to capture the complexity of media-rich environments. Visual content can enhance deception, and the integration of text and imagery forms multimodal messages that require joint analysis to be properly understood.

The societal consequences of fake news are extensive. It can influence voter decisions [10], erode institutional trust, and promote public health misinformation [11]. The virality of such content—often spreading faster than factual news—further amplifies its impact [12].

Text-only detection systems, despite leveraging methods such as term frequency–inverse document frequency (TF-IDF), part-of-speech tagging, and named entity recognition (NER) [13], remain limited in scope. Fake news creators continuously evolve their language and strategies to evade detection. Moreover, the growing use of images and videos in posts necessitates systems that analyze both textual and visual cues simultaneously.

Advances in deep learning have introduced powerful tools to address these challenges. BERT has revolutionized NLP by capturing contextual word representations, enhancing the accuracy of text understanding [14]. In multimodal contexts, the presence of inconsistencies between textual and visual content often signals deception [15], highlighting the need for integrated analysis of both modalities to improve detection performance.

This paper introduces a novel methodology for detecting fake news on social media using a multimodal BERT-based framework. Our approach incorporates both linguistic analysis and image-derived textual content, further enhanced by a cross-attention fusion mechanism. The major contributions of this work are:

• BERT’s power in analyzing the text via social media as an accurate tool for fake news.
• Development of a new framework that combines text and image features for multimodal fake news detection.
• The effectiveness and generalizability of this system will be tested using actual datasets from real-world social media platforms.

This research, therefore, contributes to the success of making firmer and more accurate detection systems for fake news. Fake news detection systems can protect online discourse and increase trust in credible news sources within the public while reducing its adverse effects on society.

The following sections review the existing literature on fake news detection, describe the proposed BERT-based multimodal framework in detail, and present our evaluation results using benchmark datasets.

2. Literature Review

2.1. Early Approaches

The early research work on fake news detection was done by using linguistic analysis based on traditional machine learning techniques. Therefore, in the early stages, researchers used different linguistic and stylistic features to identify such deceptive content in news articles. Such methods mostly relied on keyword-based analysis syntactic structures and sentiment analysis for determining true and false information. However, the early attempts had issues with the sophistication and nuance of fake news, especially when social media became the first port of call for most news sources [16].

The bad news further leveraged other forms of multimedia aspects of images, videos on social media platforms, such as, X, Facebook, Instagram, Reddit, Tiktok, as it used those for gaining an illusionary aspect of credibility. A new manner in which the news would be consumed compelled a researcher to look at the methods with both visual-textual data in a multimodal approach [17].

Early experiments on multimodal fake news detection used simple feature concatenation, combining text and image features with low-level fusion techniques. Though promising, the techniques were not sophisticated enough to exploit the complementarity of textual and visual information. Thus, the challenge was to find models that would effectively exploit the interactions between different modalities to improve detection accuracy [18].

The focus in this era was still on the development of textual analysis of news reports as the early social media applications were text intensive [19]. Examples include the use of models such as SVM and Naive Bayes for the purpose of classifying text using features like n-grams and TF-IDF scores [20]. Most of these models, however, were incapable of describing complex relationships in language modelling and hence led to false positives and negatives in testing results.

The late 2010s were a period of immense progress in fake news detection, with deep learning techniques being the driving force behind it. CNNs had become the go-to modeling choice for visual information and RNNs and their variants had proven to be highly effective for sequential text modeling [21].

This helped in partly alleviating the previously discussed bottlenecks. Still, further problems continued to emerge within efficient textual and visual fusion. Different techniques involving fusion schemes, like the attention mechanisms, have been proposed. Attention can make models weigh information across various features and enable selective focus on the significant details that could be used in determining whether some piece of news is fake or not [22].

During this stage, the region saw shifting towards multimodal data combination to improve the performance of detection. The researchers proposed models based on the fusion of text and image features using attention-based techniques that improved the classification performance. However, the issue is the lack of one unified framework for integrating them was a bottleneck for further progress, since models could not fully exploit the complementary capabilities of textual and visual data [23].

With CNNs came the turn for analyzing visual data. That unlocked a floodgate of new and unprecedented features and complex patterns being drawn from images [24]. This approach helps significantly in identifying manipulated images and their prevalence in the narratives of fake news; meanwhile, RNN improves temporal analysis of language beyond the static textual evaluation perspective. At this point, research showed the limitation of traditional feature extraction techniques but placed more emphasis on dynamic models that understand how fast fake news evolves. Deep learning techniques, with their ability to learn and generalize from vast amounts of data, presented a promising direction.

2.2. Bert and the Rise of Multimodal Approaches

The BERT model’s emergence in 2018 sparked new developments with the integration of the contextual and semantic information from the text it received, enabling more productive extraction of features from textual sources, further becoming vital in fake news detection purposes [25]. During this period, research seemed to explore the possibility of BERT, from its transformer architecture, manages to capture complex interactions that lie between text and image data making it another promising candidate for multimodal incorporation [26].

Several works used BERT to extract text features and pre-trained visual models, such as VGG-19, for image analysis [27]. As BERT performed well at capturing textual semantics, expanding its capability to effectively capture visual data was a challenge, which has led to developing complex fusion techniques that can unlock the full potential of the complementarity between textual and visual information [28]. These advancements employ the self-attention mechanism of transformers to determine which features are most relevant for text and images, hence advancing the ability to identify inconsistencies and manipulations [29].

However, the question remains how to effectively combine textual and visual information. Different fusion techniques, for example, hierarchical attention networks and contrastive learning, tried to address the question of modality interaction. These methods allow models to learn to pay more attention to important features regarding fake news detection while providing weights for others [30].

2.3. Advances in Multimodal Fusion and Bert

The idea was to build on these earlier attempts to form even more complicated multimodal fusion techniques capable of truly exploiting both the complementary nature of textual and visual data. To that effect, strategies began to emerge that improved multimodal integration with the use of attention mechanisms, contrastive learning, or cross-attention networks [31,32]. For example, in [33], a new method was proposed that relied on BERT for textual and visual feature extraction with a cross-attention network to effectively fuse the two modalities.

The scientists proposed a contrastive learning framework in another experiment. They employed the BERT for text-based analysis and ResNet model [34] for extracting features from images. The contrasts of text and images made use of contrastive loss to obtain higher performance at fake news detection. Moreover, the research involved the exploration of the hierarchical encoding networks for the extraction of the rich hierarchical semantics of textual content. Such models could catch subtle relationships between words and phrases by introducing BERT as a backbone for extracting textual features to improve fake news detection performance [22].

Hence, detection accuracy can be maximized by incorporating different types of fusion strategies. A set of strategies has been discussed and utilized by various researchers such as attention-based mechanisms [35], contrastive learning, and cross-attention networks, for capturing inter-modality interactions and representing multimodal data effectively [36].

Multimodal data raises major concerns about capturing complex relations between text and images effectively. Researchers have come up with innovative techniques, namely, entity-enhanced fusion frameworks and multimodal consistency networks, which can counter these challenges. Advanced mechanisms in fusion allow for feature integration of both text and image, capture intricate modality relationships, and lead to better performance in detecting news [37].

Recent breakthroughs in fusion techniques and integration of cross-modalities have showcased that BERT can successfully be applied to both visual and textual data [38]. Leveraging the transformer architecture found within BERT, researchers could obtain complex interactions between modalities to yield more accurate and more robust fake news detection models [39]. Such approaches show better performances compared to the traditional methods thus representing the possibility of the BERT in improving multilayer integration and precision of detection [40,41].

Generative Adversarial Networks (GANs) [42] have been extensively used for tasks that involve multimodal data, because they can learn and generate complex distributions. For the detection of fake news, GANs have been used to generate multimodal data and augment the training of models by overcoming issues related to data imbalance or generating adversarial examples to make robust model evaluation possible. GAN-based approaches typically consist of a generator, which generates synthetic samples, and a discriminator, which distinguishes between real and generated samples. Although effective, such methods often require substantial computational resources and risk introducing artifacts or biases during synthetic data generation, which can negatively impact downstream performance.

In contrast, the approach proposed in this paper uses a cross-attention mechanism and multimodal feature integration to directly model interactions and discrepancies across different modalities, such as text and images. Unlike GANs, this approach does not involve synthetic data generation but instead focuses on enhancing feature representation through attention mechanisms. This allows the model to dynamically prioritize critical features without the computational overhead associated with adversarial training. In addition, the employment of the BERT model in the proposed framework enables a strong understanding of the textual data, which helps complement the cross-modal interactions.

While GANs are particularly strong in data augmentation and adversarial testing, the proposed methodology is designed with real-world applications in mind, where interpretability, computational efficiency, and direct multimodal feature integration are valued. This makes the proposed approach a complementary alternative to generative models like GANs in the domain of multimodal fake news detection.

Even with all these advances, this multimodal fake news detection still encounters many problems. One of the toughest ones is the multimodal availability of large datasets as only a richly representative multimodal dataset lets the models operate efficiently. Moreover, fake news is dynamic; news content or formats keep being developed daily that act as a test to the models for relevance over time [41].

Another challenge is model interpretability or explainability. As detection systems for fake news complexify, researchers must therefore develop methods that enable the users to understand and have trust in model decisions. This spans techniques for visualizing attention mechanisms and feature importance besides framework ideas to evaluate model performance and reliability [43].

2.4. Comparison with State-of-the-Art (Sota) Approaches

Most existing SoTA approaches are restricted by the unavailability of large, richly representative multimodal datasets, which usually hinders their performance and generalizability. Our work overcomes this limitation by incorporating textual data extracted from images directly into a single BERT-based framework. This approach exploits a combination of publicly available datasets and image-text extraction pipelines, thereby making a much larger multimodal corpus available. Thus, our method reduces the data scarcity challenge by fusing text and images effectively into a unified model. BERT’s contextual embeddings allow the model to adapt more easily to new linguistic variations and emerging social media trends. Furthermore, our pipeline supports periodic retraining or fine-tuning with newly collected data to continue relevance and robustness against evolving fake news formats.

Table 1. Comparative analysis of state-of-the-art approaches with the proposed approach.

Author	Subject	Model/Approach	Dataset	Features	Evaluation Metrics	Findings	Limitations
[16]	Multimodal fake news detection	Multimodal system with BERT and VGG-16	Real-world dataset	Textual, visual, semantic	Accuracy, F1-score	Multimodal approach outperforms BERT baseline by 4.19%	Limited exploration of visual features
[17]	Multimodal fake news detection	SpotFake multimodal framework	Twitter and Weibo	Textual, visual	Accuracy	Outperformed state-of-the-art on Twitter and Weibo datasets	No sub-task consideration
[18]	Attention mechanism in fake news detection	Scaled dot-product attention with BERT and VGG-19	Twitter	Textual, visual	Accuracy	Improved accuracy over state-of-the-art by 3.1%	Challenges in fusing modalities
[20]	CNN-based multimodal detection	Multimodal framework on Fakeddit	Fakeddit	Text and image data	Accuracy	Multimodal CNN achieved 87% accuracy	Limited impact on some categories
[21]	Domain adaptation in multimodal detection	BDANN with BERT and VGG-19	Twitter and Weibo	Textual, visual	Accuracy	Outperformed state-of-the-art models	Noisy images in dataset
[22]	CNN-based multimodal detection	Multimodal approach with CNN	Fakeddit	Text and image data	Accuracy	Multimodal CNN outperforms text-only models	Limited dataset scope
[23]	Capsule networks for fake news detection	CapsNet with BERT	Politifact and Gossipcop	Textual, visual	Accuracy	CapsNet with BERT shows high accuracy	Focus on specific datasets
[25]	Autoencoder for fake news detection	BERT and VGG19-based autoencoder	MediaEval2015 and Weibo	Textual, visual	F-score	Competitive with state-of-the-art on Weibo	Complexity of multimodal features
[26]	CNN-based multimodal detection	Multimodal approach with CNN	Fakeddit	Text and image data	Accuracy	Text and image combination enhances performance	Limited dataset scope
[27]	Neural network for multimodal detection	Multimodal system with neural network	Three standard collections	Textual, visual, semantic	Accuracy	Improved detection with neural network approach	Focus on specific datasets
[28]	Variational autoencoder for fake news	Multimodal variational autoencoder	MediaEval2015 and Weibo	Textual, visual	F-score	High F-score achieved	Complexity of multimodal features
[29]	Cross-attention in multimodal detection	Multi-level multi-modal cross-attention network	Weibo and PHEME	Textual, visual	Accuracy	Advantageous performance over state-of-the-art	Complex relationships not fully captured
[30]	Survey of DL techniques in detection	Survey of DL techniques	Various multimodal datasets	Textual, visual, multimodal	Various metrics	Highlighted limitations and future directions	Limited exploration of datasets
[31]	Similarity and transformer-based detection	Fake News Revealer with BERT	Social media datasets	Textual, visual	Accuracy	Novel mechanism improves detection	Requires further testing
[32]	Adversarial fusion in fake news detection	Attention adversarial fusion method with BERT	Chinese public dataset	Textual, visual	F1-score	5% improvement in F1	Specific to Chinese dataset
[33]	Hierarchical attention network	Hierarchical multi-modal contextual attention network	Various datasets	Textual, visual	Accuracy	State-of-the-art performance	Hierarchical semantics challenge
[34]	Transformer-based multimodal fusion	TGA with transformers	Various datasets	Textual, visual	Effectiveness	Improved performance with feature similarity	Limited exploration of text-image interaction
[22]	CNN-based multimodal detection	Multimodal approach with CNN	Fakeddit	Text and image data	Accuracy	Images enhance detection accuracy	Limited to Fakeddit dataset
[35]	Entity-enhanced fusion for detection	Entity-enhanced multimodal fusion	Various multimodal datasets	Visual entities, text	Performance metrics	Superior to state-of-the-art models	Focus on visual entities
[36]	Survey on multimodal detection	Survey on multimodal fake news detection	Survey	Textual, visual, multimodal	Survey analysis	Discusses state-of-the-art and future directions	Survey highlights lack of large-scale datasets
[41]	Dataset collection challenges	Various multimodal datasets	-	Textual, visual, dynamic data	Dataset diversity	Challenges in acquiring diverse datasets	Lack of large-scale datasets
[43]	Explainability in AI	Explainable AI datasets	-	Textual, visual, explainable	Explainability metrics	Importance of explainability in complex models	Need for better explainability techniques
[38]	BERT’s role in multimodal detection	BERT-integrated datasets	-	Textual, visual, multimodal	Accuracy, performance	BERT improves multimodal detection accuracy	Limited visual feature exploration
[39]	Fusion techniques in BERT	Advanced fusion datasets	-	Textual, visual, semantic	Accuracy, fusion effectiveness	Enhanced performance with advanced fusion	Challenges in integrating fusion techniques
[40]	Model interpretability and trust	Interpretability datasets	-	Textual, visual, interpretability	Trustworthiness, explainability	Importance of trust in model decision-making	Complexity in model explanation
[41]	Unified BERT for real-time analysis	Unified text-image datasets	-	Textual, visual, unified	Real-time accuracy	Unified BERT model enhances efficiency	Challenges in real-time processing

presents a comparative analysis of recent state-of-the-art multimodal fake news detection approaches and their key findings.

3. Dataset

For this research, the ISOT Fake News Detection Dataset was utilized. This well-curated dataset is specifically designed for binary text classification tasks in fake news detection and is publicly available on Kaggle [44]. Renowned for its robustness and comprehensiveness, the ISOT dataset provides a reliable foundation for developing and evaluating emotion-aware fake news detection models.

The dataset consists of a total of 44,919 news articles, divided into two classes: 21,417 labeled as “true” and 23,502 labeled as “fake”. This balanced distribution allows the model to learn from a diverse range of linguistic styles and rhetorical patterns characteristic of both authentic and deceptive content [45].

To ensure generalizability and robustness, the dataset was split into training, testing, and validation sets. Specifically, 50% of the samples were used for training, 20% for testing, and the remaining 30% for validation. This stratified split supports effective learning while providing a strong benchmark for performance evaluation on unseen samples [46].

The data is provided in two primary files: fake.csv and true.csv, each containing respective news articles. This format simplifies preprocessing and integration into the model pipeline. Given its source and detailed compilation by Appen, the dataset serves as a solid basis for building a sophisticated emotion-aware fake news detection system. Its focus on emotional content in relation to article veracity provides valuable insight into the psychological and linguistic dynamics of misinformation [47].

4. Problem Statement

Since they provide easy avenues to distribute information or misinformation at incredibly fast paces, the various social media platforms present a fertile area for this sort of misinformation. Traditionally, since such presentations may be “multi-modal”—containing images or video alongside text —detection mechanisms that center on just textual analysis prove inadequate. This limitation points towards the need for an approach that can integrate both text and visual information within a single framework. Accordingly, this paper aims at utilizing the linguistic depth of BERT in conjunction with more sophisticated multimodal fusion techniques to detect fake news accurately. By incorporating both text and images into its framework in real-time, this framework aspires to produce a strong, reliable confidence score robustly measuring the authenticity of content posted on social media.

4.1. Computational Environment

All the training and scripting were done completely within Google Colab’s GPU environment, which happened to be the NVIDIA Tesla K80 with 12 GB of VRAM. It supported large datasets and complex deep-learning models as they required more computations. Local computations were achieved by a laptop with a 10th-generation Intel Core i7 processor, 16 GB of RAM, and a 256 GB SSD. This configuration allowed quick development cycles, initial data processing, and performance tests without cloud resources [48].

4.2. Data Collection

Three publicly available datasets—ISOT [49], WELFAKE [50], and TRUTHSEEKER [51]—were utilized to ensure a comprehensive evaluation of misinformation detection across diverse sources. Additionally, a concatenated dataset, ISOT_WELFAKE_TRUTHSEEKER, was created by merging the three datasets to provide a broader representation of real and fake news variations.

Each dataset underwent text cleaning, tokenization, and normalization, preserving key linguistic and stylistic features. For image-based text, Optical Character Recognition (OCR) techniques were employed to extract embedded textual content, ensuring robust cross-modal feature alignment.

To maintain authenticity, punctuation and errors in the fake news texts were preserved in the dataset. Most of the records fall between 2016 and 2017, detailing metadata such as title, text, publication date, and type of article. Each sample gathered was timestamped with metadata that included source platform details, date of publication, among other data, which, in turn, facilitated subsequent analytical processing.

$$D=\{t_i,i_i\mid t_i\in T,i_i\in I\}$$

(1)

where T represents textual data from tweets, and I depicts image data from posts on Instagram.

4.3. Preprocessing

Preprocessing included several critical steps aimed at normalizing and enhancing the data up to its proper analysis.

Text Cleaning: Applied advanced natural language processing techniques to remove noise such as URLs, special characters, and emojis. This applied the use of regular expressions and natural language toolkits in standardizing text for processing. Pre-image processing entailed improving image quality. This was enhanced by improving contrast and eliminating noise, among other factors. That way, if the images contained text, the same would be easily readable during the OCR extraction.

$$\begin{array}{cc}\hfill T_{\mathrm{clean}}& =\mathrm{clean}\left(T\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill I_{\mathrm{prep}}& =\mathrm{preprocess}\left(I\right)\hfill \end{array}$$

(3)

For $I_{\mathrm{prep}}$ a preprocessed image ready for OCR, and a clean text.

4.4. Feature Extraction

-

BERT for Text Analysis: Pass the cleaned text data $T_{\mathrm{clean}}$ through the BERT model for deep contextual embeddings. Those embeddings would represent subtle nuances and semantic relationships in the text to be understood for the communication of the message.

-

Text Extraction by OCR: OCR technology was applied on the preprocessed images $I_{\mathrm{prep}}$ and the text content buried in them was extracted [52]. The text data extracted through OCR was found useful as a supplement especially when images contained headlines or news textual information

$$\begin{array}{cc}\hfill E_{\mathrm{text}}& =\mathrm{BERT}\left(T_{\mathrm{clean}}\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill T_{\mathrm{ocr}}& =\mathrm{OCR}\left(I_{\mathrm{prep}}\right)\hfill \end{array}$$

(5)

where $E_{\mathrm{text}}$ refers to the embedded textual features and $T_{\mathrm{ocr}}$ to the text extracted from images.

4.5. Multimodal Integration

At this stage, both image-based text features and text-based text features are combined to constitute an all-inclusive feature set.

Feature Concatenation: Concatenate $E_{\mathrm{text}}$ and OCR extracted text $T_{\mathrm{ocr}}$. The feature concatenation allowed for the combining strengths of textual and visual-based text features to build up a robust dataset for analysis.

Cross-Attention Mechanism: An attention mechanism was used to look for important features across the concatenated dataset [53]. This was the most powerful mechanism, in that it enhanced the ability of the model to classify news articles as genuine versus fake news, specifically targeting discrepancies and anomalies in all modalities.

$$\begin{array}{cc}\hfill F_{\mathrm{enhanced}}& =\mathrm{CrossAttention}\left(F_{\mathrm{integrated}}\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill P_{\mathrm{fake}}& =\mathrm{Classifier}\left(F_{\mathrm{enhanced}}\right)\hfill \end{array}$$

(7)

This is where the role of Cross-Attention Mechanism in the proposed method would come in handy; they identify critical features within a concatenated multimodal dataset, highlighting the crucial difference between modalities. They are efficient to highlight mismatches or differences between input modalities; this becomes important to make a differentiation between genuine and fake news.

It is based on computing attention scores across all the modalities. Specifically, given the integrated feature set $F_{\mathrm{integrated}}$, the cross-attention mechanism generates enhanced features $F_{\mathrm{enhanced}}$ by:

• Query-Key-Value Construction:
  Each modality contributes to the construction of the query, key, and value matrices:

  $$\begin{array}{c}\hfill Q=W_Q{F}_{\mathrm{integrated}}\end{array}$$

  (8)

  $$\begin{array}{c}\hfill K=W_K{F}_{\mathrm{integrated}}\end{array}$$

  (9)

  $$\begin{array}{c}\hfill V=W_V{F}_{\mathrm{integrated}}\end{array}$$

  (10)

  where $W_Q$, $W_K$, and $W_V$ are learnable weight matrices.
• Attention Score Computation:
  The attention scores are calculated as:

  $$A=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)$$

  (11)

  where $d_k$ is the dimensionality of the key vectors, and the softmax function ensures that the scores are normalized.
• Weighted Aggregation of Features:
  The attention scores are then used to weight the value vectors:

  $$F_{\mathrm{enhanced}}=A·V$$

  (12)
• Integration with the Classifier:
  The enhanced features $F_{\mathrm{enhanced}}$ are passed to the classifier for final prediction:

  $$P_{\mathrm{fake}}=\mathrm{Classifier}\left(F_{\mathrm{enhanced}}\right)$$

  (13)

  where $P_{\mathrm{fake}}$ represents the probability of the content being fake.

This mechanism allows the model to dynamically focus on the most relevant aspects of the input data, enabling it to detect subtle inconsistencies across modalities. The model’s ability to leverage the cross-attention mechanism makes it capable of superior performance in identifying fake news with a higher degree of accuracy and robustness.

4.6. Classification

Using the improved integrated features $F_{\mathrm{enhanced}}$, a more complex classification model was applied using a Neural Network Classifier. A deep neural network was employed that sequentially processes features from raw input through multiple layers of refinement to produce an outcome.

The classification process is based on Probability Estimation and Thresholding, where the output of the neural network represents the likelihood of the news being fake. Specifically, the network outputs a probability score, $P_{\mathrm{fake}}$, indicating the likelihood of fake news. A classification decision is then made using a predefined threshold, TT, such that:

$$\mathrm{Class}=\left\{\begin{array}{cc}\mathrm{Fake},\hfill & P_{\mathrm{fake}}\ge T\hfill \\ \mathrm{Real},\hfill & P_{\mathrm{fake}}<T\hfill \end{array}\right.$$

(14)

This threshold-based approach ensures that news items are labeled as fake if the probability score meets or exceeds the threshold, and as real otherwise [54]. This method improves interpretability and reliability in the classification process, leveraging refined features and advanced neural network capabilities.

4.7. Confidence Score Calculation

In this study, the “Confidence Score” is a direct, interpretable measure of the model’s probability outputs. Specifically, if $P_{\mathrm{fake}}$ denotes the model’s predicted probability that a piece of content is fake, then we define:

$$\begin{array}{cc}\hfill \mathrm{Confidence}{\mathrm{Score}}_{\mathrm{fake}}& =P_{\mathrm{fake}}\times 100\%\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill \mathrm{Confidence}{\mathrm{Score}}_{\mathrm{real}}& =(1-P_{\mathrm{fake}})\times 100\%\hfill \end{array}$$

(16)

This transformation of the raw probability into a percentage enables end-users to clearly gauge how likely a piece of content is to be fake or real. For instance, if $P_{\mathrm{fake}}=0.75$, then Confidence Scorefake = 75% and Confidence Scorereal = 25%.

This simple and transparent metric is crucial for enhancing the interpretability of the model’s predictions and guiding informed decision-making.

4.8. Block Diagram

A detailed methodology flow diagram outlines a very comprehensive scheme of detection with data integration from X, Instagram, and other such social media platforms. Then, from data to steps of preprocessing comprising text cleaning, image processing, and noise removal, major features get extracted using BERT for text and OCR for the image text and provided to be analyzed semantically. These features are then combined into a multimodal representation through a multimodal integration process that utilizes cross-attention mechanisms and concatenation to create a single fused representation. Finally, this multimodal classifier will classify the news as real or fake, and the classification confidence score quantifies how reliable the prediction was. Figure 1 illustrates the overall methodology workflow of the proposed model.

5. Experimental Setup and Results

This work focuses on multimodal fake news detection and reports a significant advance by using pre-trained transformer-based models, particularly BERT. This paper tests a BERT-based multimodal approach to fake news detection across four different datasets: ISOT, WELFAKE, TRUTHSEEKER, and ISOT_WELFAKE_TRUTHSEEKER.

Results are systematically analyzed to emphasize the effectiveness of our approach. The pre-trained model, BERT-Base-Uncased, is renowned for its robust NLP capabilities and thus serves as the backbone of our model. Using multiple transformer layers that employ self-attention mechanisms, BERT can capture contextual relationships between words effectively. The model is further enhanced by fine-tuning it with textual and embedded image text, thereby allowing it to differentiate between authentic and deceptive news.

The architecture of the BERT_base_uncased model used in our study is illustrated in Figure 2.

Input Layer: This will also have tokenized text sequences for the input to the model. Now, each token is represented as a numerical ID and padded to a certain length for uniformity in the input sequence. Embedding Layer: The tokens are converted in the dense vector representation as the inputs. These include the meaning and syntax of the word. Encoder Layers: These are the different layers the input sequences undergo. Each encoder layer consists of multi-head self-attention mechanisms and a feedforward neural network. In this context, the self-attention mechanism allows the model to focus on some words within a sequence depending on how much each word interacts with its neighbor. Pooling Layer: This layer of the encoder, towards the end, pools to create a fixed size of the input sequence generally employing mean pooling or max pooling operation. Classifier Layer: The pooled representation is then passed through a fully connected layer with softmax activation to predict the probability of each input sequence belonging to either the fake or real class.

This paper utilized the BERT-Base-Uncased model with 12 transformer layers, 768-dimensional hidden states, and 12 attention heads. The number of parameters is about 109 million, meaning the capacity of the model in learning complex representations of input data.

The key parameters of the BERT_base_uncased model are summarized in Figure 3.

TFBertMainLayer: This is the core of the model; it is a pre-trained BERT language model with an input sequence to take out suitable features.

Dropout: A dropout layer randomly drops the neurons during training to prevent overfitting.

Classifier: A dense layer with 1538 units followed by a softmax function to classify.

The model has 109,483,778 parameters that are all trainable. That shows the model has a very large capacity to learn those complex patterns in the data.

To achieve generalizability and robustness of the model, we’ve created training, testing, and validation sets 80:10:10 ratio. Such a stratified split will enable proper training of the model, while also providing a good benchmark for the evaluation of model performance on unseen data [46]. The datasets are divided mainly into two files: fake.csv and true.csv, which contain all the relevant articles. Such a structure allows for easy preprocessing and integration into the model development pipeline [47].

Figure 4 is an example of ISOT data where each news article has a label as either ‘fake’ or ‘true’. In this representation, ‘0’ is labeled for fake news and ‘1’ is labeled for true news. The columns here include the title of the article, the text body, the subject category, the date of publication, and the corresponding label. The labeling is important because the model needs to learn how to differentiate “True” from “Fake” news content.

Figure 5 below shows that the WELFake dataset is a large aggregation intended to make the machine learning models, tasked with classifying news articles as either real or fake, more generalizable. The dataset has a total of 72,134 news articles, with 35,028 labeled as authentic and 37,106 as deceptive. In an attempt to reduce the effects of bias from a single dataset, it aggregates data from multiple sources: Kaggle, McIntire, Reuters, and BuzzFeed Political. Each article is divided into four distinct columns: a serial number, a headline (Title), the main text (Text), and the corresponding binary label (Label: 0 for fake, 1 for real). This dataset presents a wide-ranging textual manifestation of misinformation and permits an extended framework in assessing the effectiveness of models designed to detect fake news.

Figure 6 presents the TruthSeeker2023 dataset, recognized as one of the most extensive benchmark datasets for the identification of fake news in the realm of social media, particularly on Twitter, encompassing over 180,000 labeled tweets. These tweets were collected from Twitter posts that pertain to both genuine and fabricated news occurrences derived from the PolitiFact dataset. The dataset underwent a three-factor active learning verification process, which was predicated on crowdsourced labeling conducted by 456 expert annotators through the platform Amazon Mechanical Turk. In addition to that, the dataset provides three additional social media metrics, which include Bot Score, Credibility Score, and Influence Score. Using the features from tweets with the user metadata, the dataset can support deep learning detection methods and traditional machine learning methods. The dataset has been rigorously tested with different models of BERT, event detection approaches focusing on clustering, and authenticity of the X posts with regards to content features of its originators and spreaders.

5.1. Model Training and Validation

The datasets were split into training, validation, and test sets to provide a robust framework for evaluation. The split ratios were adjusted to balance data availability against model generalization. The training phase on all datasets showed high accuracy; the variations in generalization ability were reflected in the validation and test phases.

The ISOT Dataset presented near perfection, with 99.7 percent accuracy for all training, validation, and test phases. WELFAKE Dataset generalizes moderately at  94% accuracy, which implies that the dataset has more diverse or challenging samples. TRUTHSEEKER Dataset reached 98% accuracy, balancing model complexity with real-world applicability (see

Table 2. Model Performance Metrics Across Training, Validation, and Test Phases.

Dataset	Training Accuracy	Validation Accuracy	Test Accuracy
ISOT	0.9997	1.0000	0.9999
WELFAKE	0.94	0.94	0.94
TRUTHSEEKER	0.98	0.98	0.98
ISOT_WELFAKE _TRUTHSEEKER	0.80	0.96	0.98

).

ISOT_WELFAKE_TRUTHSEEKER Dataset had the most diverse input, with 80% training accuracy but 96–98% validation and test accuracy, which implies the model adapted well to varied patterns. This was similar to the results observed in previous works [55], where the BERT models were trained on structured datasets that were good for classification but lacked domain adaptation and broader generalization.

The training and evaluation steps of the BERT model are summarized in Algorithm 1.

Algorithm 1 BERT Model Training and Evaluation Steps

Input: Dataset D containing real and fake news    Output: Trained BERT model, best model checkpoint, and evaluation metrics   1: Initialize $D\ne \varnothing$   2: real_news←LoadDataset(D)   3: fake_news←LoadDataset(D)   4: fake_news[’label’]← 0   5: real_news[’label’]← 1   6: data←ConcatenateAndShuffle(fake_news, real_news)   7: SaveCSV(data)   8: Define function preprocess_text(text)   9:    text ← ReplaceNonAlphabetical(text) 10:    text ← ToLowerCase(text) 11:    text ← ApplyStemming(text) 12: End function 13: data[’text_preprocessed’]←preprocess_text(data[’text’]) 14: tokenizer←LoadTokenizer(’bert-base-uncased’) 15: (train_data, val_data, test_data) ← SplitDataset(data) 16: model←LoadBERTModel(’bert-base-uncased’) 17: CompileModel(model, optimizer=adam, loss=binary_crossentropy) 18: history←TrainModel(model, train_data, val_data) 19: accuracy←EvaluateModel(model, test_data) 20: if accuracy > max_accuracy then 21:    SaveBestModel(model) 22: end if 23: PlotMetrics(history, ’accuracy’) 24: PlotMetrics(history, ’loss’) 25: predictions←Predict(model, test_data) 26: PlotConfusionMatrix(predictions, test_data[’label’]) 27: PlotROCCurve(predictions, test_data[’label’]) 28: PlotPrecisionRecallCurve(predictions, test_data[’label’]) 29: End

The inference and deployment steps of the BERT model are described in Algorithm 2.

Algorithm 2 BERT Model Inference and Deployment

Input: Best-performing BERT model from Algorithm 1, new unlabeled dataset $D^{\prime }$   Output: Predicted labels (real/fake) and evaluation metrics on $D^{\prime }$   1: Initialize $D^{\prime }\ne \varnothing$   2: best_model←LoadBestModel(path_to_saved_model)   3: new_data←LoadDataset($D^{\prime }$)   4: Define function preprocess_text(text)   5:    text ← ReplaceNonAlphabetical(text)   6:    text ← ToLowerCase(text)   7:    text ← ApplyStemming(text)   8: End function   9: new_data[’text_preprocessed’]←preprocess_text(new_data[’text’]) 10: new_inputs←tokenizer(new_data[’text_preprocessed’]) 11: predictions←Predict(best_model, new_inputs) 12: labels←AssignLabels(predictions, threshold=0.5) 13: SaveResults(new_data, labels, ’inference_results.csv’) 14: if ground_truth_labels$\in D^{\prime }$ then 15:       accuracy_new ← CalculateAccuracy(labels, ground_truth_labels) 16:       PlotConfusionMatrix(labels, ground_truth_labels) 17:       PlotROCCurve(predictions, ground_truth_labels) 18:       PlotPrecisionRecallCurve(predictions, ground_truth_labels) 19: end if 20: Deploy best_model to production environment if performance is satisfactory 21: End

5.2. Classification Report

The classification report in Table 2 shows that there are significant differences in the performance of the models when tested on the four datasets, highlighting the influence of dataset architecture, variability, and complexity on the accuracy of classification. The ISOT dataset has an impressive overall accuracy rate of 99.25%, with high precision and recall metrics for both authentic and fraudulent news. The marginally reduced recall observed for Class 0 (fake news) indicates that some misleading articles closely resemble genuine news formats, resulting in instances of misclassification. The WELFAKE dataset shows a minor decline in accuracy, recorded at 97.95%, and significant reduction in recall for Class 0 (0.91). This shows that it has trouble effectively identifying the different and less well-organized characteristics of fake news found in this dataset. A more extensive dataset that encompasses multiple news sources is a more linguistically and contextually diverse range, making classification harder.

The TRUTHSEEKER dataset has outstanding classification performance with 99.99% accuracy, precision, recall, and F1-scores at 1.00. This would mean the existence of an almost optimal model; however, it raises questions about the possibility of biases in the used dataset or overfitting to the well-organized nature of false information in social media as well as the credibility scores introduced into the overall dataset. Artificially well-differentiated patterns may have low generalizability when applied to real-world settings, where the complexity of misinformation is higher. On the other hand, in the case of the ISOT_WELFAKE_TRUTHSEEKER dataset’s accuracy, due to its inclusion of multiple sources and increased variability, falls drastically to 80.67%.

The recall-precision difference is especially observed about Class 1 (actual news); the model struggles to generalize to different styles and contexts for both actual and constructed news. This is in accordance with other related studies that note that a boost in diversity within the dataset is associated with lower classification effectiveness but increases generalizability. In conclusion, although organized datasets such as ISOT and TRUTHSEEKER produce elevated levels of accuracy, more varied datasets such as ISOT_WELFAKE_TRUTHSEEKER offer a more authentic evaluation of model efficacy in real-world misinformation detection contexts.

The classification metrics, including precision, recall, and F1-scores for both fake and real news detection, are detailed in

Table 3. Classification Report Detailing Precision, Recall, and F1-Scores for Fake and Real News Detection.

Dataset	Precision (Class 0)	Recall (Class 0)	F1-Score (Class 0)	Precision (Class 1)	Recall (Class 1)	F1-Score (Class 1)	Overall Accuracy
ISOT	0.99	0.97	0.98	0.95	0.97	0.98	99.25%
WELFAKE	0.96	0.91	0.93	0.92	0.96	0.94	97.95%
TRUTHSEEKER	1.00	1.00	1.00	1.00	1.00	1.00	99.99%
ISOT_WELFAKE_ TRUTHSEEKER	0.98	0.99	0.98	0.96	0.91	0.94	80.67%

.

Compared to earlier approaches [56], which demonstrated large discrepancies between real and fake news detection, our model showcases a much more balanced performance across datasets. The relatively uniform precision-recall balance highlights its suitability for real-world deployment, particularly in automated content moderation and misinformation filtering systems.

5.3. Model Accuracy, Loss, Confusion Matrix

Figure 7 shows accuracy curves for all cases which reveals that ISOT and TRUTHSEEKER maintain a stable increasing trend, while WELFAKE experiences small fluctuations related to the natural fluctuation in the dataset. Loss curves prove that although ISOT has training loss close to zero, ISOT_WELFAKE_TRUTHSEEKER is slightly larger with stable validation loss, which can be a benefit for practical applications [57].

Model Loss: Figure 8 plots the curves of training and validation losses of all datasets. The training loss curves smoothly decrease as the model is learning how to make fewer mistakes. The curve of validation loss follows the non-monotonic character with a decrease, after which there is a little increase, followed by a new decrease. Such a behaviour can be taken for indication that the model has been faced with more complicated samples during validation, challenging its generalization capability. Such trends have been documented in related studies by Liu et al. [58].

ISOT shows Practically no loss, hence confirming overfitting. WELfake’s validation loss rises, indicating it is not generalizing well. Truthseeker dataset has a good balance between training and validation loss. While the ISOT_WELFAKE_TRUTHSEEKER loss is higher, the model is still stable.

Learning Rate Scheduler: Figure 8 Depicts the gradual learning rate decay across each epoch is a critical component in fine-tuning the model’s parameters. The model will converge faster at the early stages of training if a higher learning rate is initially employed. Then, with the learning rate decreased, the model refines its weight adjustments to obtain higher accuracy and stability. This adaptive learning rate strategy prevents abrupt updates that may cause a loss in convergence, hence leading to the better performance metrics reported in the results [59].

Precision-Recall Curve: Figure 9 plots The Precision–Recall Curve in the figure illustrates the model’s performance in distinguishing fake from real news. The x-axis represents recall—the proportion of true positives correctly identified—while the y-axis shows precision, indicating the proportion of correct positive predictions. A curve that remains close to the top-right corner signifies that the model maintains high precision even as recall increases, which reflects its effectiveness in minimizing both false positives and false negatives. This balance is crucial in fake news detection, where correctly identifying misinformation while avoiding false alarms is essential [60].

Confusion Matrix: Figure 10 presents the confusion matrix for the performance between classes on the model. Misclassifications are relatively very low, with 122 false positives and 115 false negatives out of an 8979-prediction total. In addition, balanced misclassification across the classes reflects the strength of the model, particularly in keeping low false positives that might be critical to the reliability of fake news detection systems [61].

Receiver Operating Characteristic (ROC) Curve: The ROC curve in Figure 11 shows that the indicates that the model has discriminative power; the area under the curve approaches 1.0. This nearly perfect AUC means that the model perfectly distinguishes between real and fake news, achieving a high true positive rate with a minimal false positive rate. It becomes particularly relevant in comparison with baseline models since it presents a much larger improvement of classification accuracy-a result in agreement with the results obtained.

5.4. Image News Classification Results

The results from OCR-based image classification indicate that the model confidence and performance are different from each other when compared to various datasets. For example, ISOT has an unusually high level of confidence at 99.98% real about the specific news headline. This indicates that the textual structures obtained from images are highly correlated with the linguistic patterns found in the dataset. In this regard, TRUTHSEEKER also tags the same news headline as true with a somewhat lesser confidence level of 97.87% that suggests the dataset may have more stylistic or contextual variations in news article structures, thereby reducing the classification’s certainty by a small margin.

On the other hand, the ISOT-WELFAKE-TRUTHSEEKER dataset shows immense difficulties in the classification accuracy of the newspaper-style content. This results in almost equal probabilities for the real and fake classifications (52.21% real, 47.79% fake) and indicates inherent difficulty in making a distinction in cases where the OCR-extracted text may be distorted, an outdated writing style, or vague phrasing, which necessitates better pre-processing techniques and strategies for domain adaptation to improve the robustness of OCR when handling different news formats. The results are consistent with previous studies concerning the detection of multimodal misinformation, highlighting the necessity for dataset-specific adjustments to enhance classification efficacy among diverse news outlets [62,63,64].

These findings affirm that each module contributes meaningfully to model performance. OCR text from images enhances the contextual integrity of posts containing visual misinformation. The cross-attention mechanism ensures robust interaction across modalities, and BERT remains critical in capturing nuanced linguistic patterns compared to simpler encoders.

Table 4. Cross-dataset evaluation results: Input images, extracted news content, and model predictions across datasets.

Dataset	Input Image/Source	Extracted News	Text Output
ISOT		Stock Price Levels Stabilized
WELFAKE	https://www.kaggle.com/datasets/saurabhshahane/fake-news-classification/data, accessed on 1 May 2024	Pakistan Rangers attack Indian border
TRUTHSEEKER	https://doi.org/10.1109/TCSS.2023.3322303, accessed on 1 May 2024	Pakistan Rangers attack Indian border
ISOT-WELFAKE-TRUTHSEEKER		Stock Price Levels Stabilized

summarizes the results of the cross-dataset evaluation, demonstrating the model’s ability to generalize across different news domains.

Table 5. Performance Comparison of Multimodal Models on FND Datasets.

Model	Dataset	Accuracy	Precision	Recall	F1-Score	Features
Multimodal system with BERT and VGG-16	Real-world dataset	0.76	-	-	0.7955	Textual, visual, semantic
BERTweet	Twitter	0.812	0.813	0.874	0.843	Textual, visual
Multimodal framework on Fakeddit	Fakeddit dataset	91.94%	93.43%	93.07%	93%	Text and image data
Multimodal CNN	Fakeddit	0.87	0.88	0.86	0.87	Text and image data
TGA with transformers	Weibo & Twitter	0.969	0.886	0.922	0.925	Textual, visual
Proposed Model	ISOT FND	97.36	0.97	0.97	0.97	Textual, visual

presents a comparative analysis of various multimodal models on fake news detection (FND) datasets, highlighting the performance of our proposed approach.

Table 6. Ablation Study: Performance Comparison of Model Variants.

Configuration	Accuracy	F1-Score
Full Model (BERT + OCR + Cross-Attention)	99.99%	0.98
Without OCR	96.2%	0.94
Without Cross-Attention	97.5%	0.96
Replace BERT with LSTM	93.8%	0.91

summarizes the ablation study, showing the impact of removing each component on the model’s performance.

5.5. Advanced Critical Discussion

Comparing a number of multimodal datasets for fake news highlights important issues concerning the advantages and disadvantages that are embedded in each one of them. Although the ISOT and WELFake datasets show high accuracy and confidence in classification, their structured nature may limit their use in real-world misinformation scenarios [63]. The datasets are mainly news articles, which makes them effective in detecting fake news in traditional media but less reliable in addressing the dynamic and ever-changing nature of misinformation on social media. In contrast, the ISOT-WELFAKE-TRUTHSEEKER dataset is at once a challenging and an excellent opportunity for improved detection of fake news. The “Stock Price Levels Stabilized” classification resulted in 52.21% authentic and 47.79% fraudulent, which was almost evenly split. While this result may be seen as a weakness of the model, it really speaks to the dataset’s ability to classify nuanced cases where fake news becomes increasingly sophisticated and harder to detect.

Conversely, the TruthSeeker dataset is marked by its high and dynamic benchmark properties. Its integration of a range of sources such as tweets, bot-generated content, and human-verified claims presents a more challenging yet realistic environment for the identification of fake news by models [56].

The dataset effectively captures the contextual nuances of misinformation spread on digital platforms, thereby enabling models to generalize better to real-world scenarios. Furthermore, its integration of credibility and influence scores adds an additional layer of interpretability, which is particularly valuable for advanced machine learning approaches that incorporate user behavior analysis. Therefore, even though, all these datasets contribute to explaining different aspects of fake news detection, TruthSeeker is the most holistic and practically useful dataset. Its ability to challenge models while simultaneously allowing increased classification confidence underlines its utility in the design of adaptive and robust systems for fake news detection. Future work would involve enriching these datasets through the incorporation of multimodal features, including user engagement patterns, sentiment analysis, and deepfake detection, to produce more effective frameworks for misinformation detection [64].

5.6. Future Directions in Model Development

There are ample scopes and opportunities for the further development of fake news detection systems towards being more scalable, adaptable, and real-time deployable, especially for misinformation detection across the different datasets employed.

One of the primary areas to be explored further would be on creating an AI-based application that can receive all types of news-textual, photo-based, video-based-as inputs to classify it as a real or false. It would utilize pre-trained NLP models coupled with computer vision techniques that analyze textual as well as visual inputs and then fine-tune the results on multimodal deep learning frameworks. This application, by implementing mechanisms such as real-time data verification through cross-checking with reliable sources and integrity analysis of metadata, would be able to offer users instant, reliable assessments regarding the credibility of news.

Another crucial way forward would be the embedding of fake news detection models in social media through browser extensions, APIs, or embedded AI-driven moderation tools. The AI agents may reside within the real-time social media environment and thus continue to monitor and flag the possibly misleading content being posted in real time.

Such a system would need robust streaming data pipelines and federated learning approaches to ensure the model stays responsive to the changing misinformation patterns while preserving user privacy and platform integrity.

Another significant area of research agenda would be the expansion of cross-modal capabilities in addressing artificially created misinformation, including deepfakes and edited images. Using the idea of GANs for anomaly detection, future work will take this further using multi-agent AI frameworks.

Besides technological advances, future research should also pay attention to ethics and policy implications that arise when machine-generated misinformation is detected. This includes the model’s fairness, transparency, and explainability and the necessity to address potential biases that may evolve within AI-assisted moderation mechanisms. Collaboration with fact-checking organizations, lawmakers, and media outlets will be key to the design of sensible and effective deployment strategies for AI-based detection of misinformation in real-world settings.

With these improvements in place, future research can help in making more sophisticated and context-sensitive artificial intelligence systems that could be ethically responsible enough to better address the global problem of misinformation in digital spaces.

6. Conclusions

In this study, we introduce a BERT-based multimodal framework that utilizes both textual and image-based content to effectively detect misinformation with high accuracy and robustness. The comparative analysis across four datasets—ISOT, WELFAKE, TRUTHSEEKER, and ISOT_WELFAKE_TRUTHSEEKER—demonstrates the model’s adaptability across varying data distributions. Notably, the model achieves near-perfect classification on the ISOT dataset while exhibiting more generalized performance on the TRUTHSEEKER dataset, which encompasses a broader spectrum of real-world misinformation patterns. These findings highlight the critical role of dataset diversity in enhancing the model’s generalization ability, reinforcing the necessity of diverse data sources in developing scalable and reliable misinformation detection systems.

The results of this study provide clear evidence of the benefits of integrating textual and visual data, demonstrating that multimodal approaches enhance misinformation detection by capturing both textual semantics and embedded text within images. The classification reports, confusion matrices, and evaluation metrics reveal that while ISOT has minimal misclassifications, its overfitting tendencies limit its applicability in broader contexts. In contrast, the TRUTHSEEKER dataset offers a more challenging yet realistic evaluation setting, leading to improved model adaptability in detecting misinformation across dynamic digital environments. These findings align with prior research advocating for cross-platform training data to mitigate overfitting and improve real-world performance.

Despite the strong performance, certain limitations remain, particularly in cross-modal feature alignment and handling ambiguous or complex misinformation formats. The study also highlights that OCR-based classification of news headlines introduces variability due to text extraction from images, necessitating further refinement in text preprocessing and context preservation mechanisms. Additionally, scalability and real-time efficiency present significant challenges for deploying this system in practical applications such as social media content moderation and misinformation detection tools.

Future research should focus on enhancing model interpretability, optimizing attention mechanisms for multimodal integration, and expanding datasets to include emerging misinformation formats, such as deepfake videos and synthetic media. The development of real-time AI agents for misinformation detection, integrated into social media platforms or independent fact-checking applications, represents a promising direction that could significantly enhance digital media integrity.

Ultimately, this study underscores the importance of dataset diversity, multimodal integration, and scalable AI-driven solutions in combating fake news. While all datasets provided valuable insights, the TRUTHSEEKER dataset emerges as the most practical benchmark, as it best reflects the complexities of real-world misinformation. These findings establish a solid foundation for advancing AI-driven misinformation detection and emphasize the need for continued interdisciplinary research to refine automated verification systems for public discourse and digital media analysis.