Skip to Content
You are currently on the new version of our website. Access the old version .
Future InternetFuture Internet
  • Article
  • Open Access

23 January 2026

19 Pages

Video Deepfake Detection Based on Multimodality Semantic Consistency Fusion

,
,
,
and
School of Computer Science and Artificial Intelligence, Liaoning Normal University, No. 1 Liushu South Street, Ganjingzi District, Dalian 116081, China
*
Author to whom correspondence should be addressed.

Abstract

Deepfake detection in video data typically relies on mining deep embedded representations across multiple modalities to obtain discriminative fused features and thereby improve detection accuracy. However, existing approaches predominantly focus on how to exploit complementary information across modalities to ensure effective fusion, while often overlooking the impact of noise and interference present in the data. For instance, issues such as small objects, blurring, and occlusions in the visual modality can disrupt the semantic consistency of the fused features. To address this, we propose a Multimodality Semantic Consistency Fusion model for video forgery detection. The model introduces a semantic consistency gating mechanism to enhance the embedding of semantically aligned information across modalities, thereby improving the discriminability of the fused representations. Furthermore, we incorporate an event-level weakly supervised loss to strengthen the global semantic discrimination of the video data. Extensive experiments on standard video forgery detection benchmarks demonstrate the effectiveness of the proposed method, achieving superior performance in both forgery event detection and localization compared to state-of-the-art approaches.

1. Introduction

Face detection technology enables identity recognition by extracting personalized facial features and has been widely applied in fake news detection, identity authentication, and digital copyright protection [1,2]. However, advances in deep learning and generative models have facilitated face image forgery techniques, such as identity replacement, attribute manipulation, and face synthesis, which introduce false information and pose serious threats to security, privacy, and trust [3]. Consequently, accurate and reliable face forgery detection has become increasingly important. Early forgery detection methods relied on handcrafted features and binary classification [4,5], but their dependence on low level cues limited generalization as forgery techniques evolved. Deep learning based approaches, particularly Convolutional Neural Networks, were subsequently introduced to enhance semantic representation and discriminative capability. For example, Dang et al. [6] combined Visual Geometry Group (VGG) based architectures with texture feature extraction to capture multi scale forgery traces, demonstrating the effectiveness of Convolutional Neural Networks (CNNs) in this task [7]. As a result, deep learning based methods have become the dominant paradigm for face forgery detection [8,9,10]. Nevertheless, conventional CNN-based models often lack adaptive feature weighting and fine grained localization, which limits their ability to detect increasingly subtle and localized manipulations. To address this issue, attention mechanisms have been introduced to emphasize discriminative regions and enhance feature representation. Zhao et al. [11] proposed a multi attention framework that improves local forgery perception through attention guided pooling. However, modeling long range spatial dependencies among facial components remains challenging. Vision Transformer-based methods overcome this limitation by leveraging self attention to capture global relationships between facial regions, improving robustness to background interference and pose variations [12].
Most existing face forgery detection methods focus on static images and show limited effectiveness on time sequential video data, despite video being the dominant medium in modern multimedia communication. Early video based approaches exploited manipulation artifacts such as affine inconsistencies or interpolation errors [13], but their robustness degrades under high compression, high quality forgeries, and complex motion. Subsequent deep learning methods incorporated temporal cues, including blinking inconsistencies [14] and frame level face analysis [15], yet they remained restricted to visual information. With the rise of multimodality misinformation, cross modal forgery detection has gained increasing attention. Contrastive Language-Image Pre-training (CLIP)-based frameworks [16] and evidence driven approaches [17] introduced image text alignment for detecting context distorted content. More recently, the HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) framework [18] leveraged contrastive learning and modality aware cross attention to enable fine grained multimodality forgery detection and localization. However, existing multimodality methods often emphasize unimodal feature extraction and lack sufficient cross modal interaction. As a result, detection performance degrades when semantic cues in a single modality are corrupted by occlusion, noise, or small targets in video data.
To address these issues, this paper proposes a multimodality forgery detection network architecture based on semantic consistency. By constructing a semantic consistency gating module, the interference from misleading information is mitigated, ensuring the robustness and accuracy of detection. First, the image and text data are processed through image and text encoders to obtain feature representations, which are then deeply interacted through a multimodality cross-attention feature fusion module to generate cross-fused image features enhanced by textual information. Second, the semantic consistency gating module is constructed to generate gating signals by calculating the semantic similarity between different modalities, adjusting the feature fusion weights. This results in semantically consistent cross fused image features and text embedding features. Finally, hierarchical operation inference is performed, and class loss functions are constructed to enable strong supervision learning in video frames and contrastive learning in across different modalities, allowing multimodality information to be detected and localized for forgery.
The main contributions of this paper are as follows:
  • To address the issue of unclear semantic expression of fused features caused by single-modality information forgery in video forgery detection, we propose a forgery detection model with semantic consistency gating. This model strengthens the consistency discrimination representation of multimodality features under the same semantics by using semantic gating, improving the accuracy of video data forgery detection.
  • Our model ensures the subsequent discriminability of fused features by enhancing the representation of single-modality image features (such as small targets, occlusion, and blur) and the representation of ambiguous semantic information in single-modality text features. It avoids the negative impact of sudden event samples in video data on continuous semantic detection, thereby improving the robustness of forgery detection.
  • The proposed model achieves better detection accuracy in video forgery detection compared to popular multimodality detection models on publicly available video forgery datasets.

3. Methodology

Video-based forgery detection models typically perform frame wise extraction and fusion of deep features from multiple modalities to detect forgery events under the supervision of labeled data. However, such models generally overlook the fact that interference or noise present in a specific modality can degrade the discriminative capability of subsequently fused multimodal features, as illustrated in Figure 1. To address this issue, we propose a multimodal semantic consistency aware fusion detection model, which introduces a consistency gating mechanism to enhance feature representations with coherent semantic labels and improved discriminability. The overall network architecture is shown in Figure 2, and the proposed model is composed of the following three main components.
Figure 1. Examples of ambiguous semantic information in the single modality. The content within the red squares in all sub-images represents the forged parts.
Figure 2. The model architecture diagram.

3.1. Multimodality Feature Fusion Representation

The data studied in this work consist of video sequences. For efficient processing and computation, the videos are sampled to obtain single-frame image samples containing only human subjects, along with their corresponding textual information. The input sample set x i ∈ X , i = 1 , … , N , it includes both image and text data, Let x i v 1 be the image data, x i v 2 is text data. Y is the label set corresponding to the multimodal samples y i ∈ Y , i = 1 , … , N , in the context of multimodal information representation constraints, there are x i v 1 → y i , x i v 2 → y i .
The image encoder is used to obtain the embedded representation of samples in the image modality. First, the input image data x i v 1 is segmented into a series of unfolded image patches. These image patches are projected into the embedding space through a linear layer to obtain the image patch embedding projection E . To preserve the spatial positional information between the image patches, positional embeddings E p o s are added to the patches. For ease of subsequent classification prediction, a learnable embedding vector for the classification category, x c l s v 1 is inserted at the beginning of the vector sequence. Moreover, all the embedding vectors are summed to form the input z 0 v 1 to the image encoder. The specific operations are as follows:
z 0 v 1 = x C L S v 1 ; x p 1 E ; x p 2 E ; … ; x p N E + E p o s
Second, z 0 v 1 is transformed into feature representations through the Encoder as input. In this paper, through the multi-head self-attention mechanism, z 0 v 1 is mapped to multiple different representation spaces, obtaining local semantic relationships and structural representations. First, three different linear transformation layers generate the query, key, and value matrices, and through matrix multiplication of z 0 v 1 with the weight matrices W Q ( v 1 ) , W K ( v 1 ) , and W v ( v 1 ) , it is mapped to the query, key, and value matrices. Then, the query, key, and value matrices are divided into multiple heads (i.e., multiple subspaces), each head having different linear transformation parameters. Each head independently computes attention and generates an output that is weighted by attention. The formula is: h e a d i v 1 = A t t e n t i o n W i Q v 1 Q v 1 , W i K v 1 K v 1 , W i V v 1 V v 1 , where i represents the subspaces of the quer, key, and value matrices. Concatenate the outputs of all attention heads as h e a d ( v 1 ) = c o n c a t h e a d 1 v 1 , … , h e a d h v 1 , and through a linear transformation, a final representation is formed:
M u l t i H e a d Q v 1 , K v 1 , V v 1 = W o ⊙ h e a d ( v 1 )
where W o is the output projection matrix.
Finally, M u l t i H e a d Q v 1 , K v 1 , V v 1 undergoes feature transformation through a multi-layer perceptron (MLP) consisting of two linear layers and the GELU activation function. After batch normalization, this results in a more target-focused, structured image latent feature representation z v 1 .
Meanwhile, a text encoder is used to obtain the embedding representation of samples in the text modality. The input text data x i v 2 first passes through an embedding module consisting of Token Embeddings (TE), Segment Embeddings (SE), and Position Embeddings (PE). TE is designed to better capture the context of the entire sentence when processing text. A [CLS] token is added at the beginning of the first sentence, and then each word token in the input text is converted into a fixed-dimensional vector E t o k v 2 of shape V s , D v 2 through a lookup table for word vectors. Here, V s represents the vocabulary size and D v 2 represents the vector dimension. SE is used to differentiate different paragraphs or sentences in the text. An additional embedding vector E s e g v 2 is embedded for each token based on its position in a specific sentence or paragraph. PE provides positional information about words within the sentence by adding a vector E p o s v 2 to each token that is related to its position in the sentence. All the embedding vectors are summed up and used as the input z 0 v 2 for the text encoder.
z 0 v 2 = E t o k v 2 + E s e g v 2 + E p o s v 2
Second, z 0 v 2 is transformed into feature representations through the Encoder as input. In this paper, through the multi-head self-attention mechanism, z 0 v 2 is mapped to multiple different representation spaces, obtaining local semantic relationships and structural representations. First, three different linear transformation layers generate the query, key, and value matrices, and through matrix multiplication of z 0 v 2 with the weight matrices W Q ( v 2 ) , W K ( v 2 ) , and W v ( v 2 ) , it is mapped to the query, key, and value matrices. Then, the query, key, and value matrices are divided into multiple heads (i.e., multiple subspaces), each head having different linear transformation parameters. Each head independently computes attention and generates an output that is weighted by attention. The formula is: h e a d ( v 2 ) = c o n c a t h e a d 1 v 2 , … , h e a d h v 2 , where j represents the subspaces of the query, key, and value matrices. Concatenate the outputs of all attention heads as h e a d ( v 2 ) = c o n c a t h e a d 1 v 2 , … , h e a d h v 2 , and through a linear transformation, a final representation is formed:
M u l t i H e a d Q v 2 , K v 2 , V v 2 = W o ⊙ h e a d ( v 2 )
where W o is the output projection matrix.
Then, M u l t i H e a d Q v 2 , K v 2 , V v 2 is input into a feed-forward neural network layer module consisting of two linear transformations and a ReLU activation function. The corresponding mathematical formulation is as follows:
z 1 v 2 = W 2 ⊙ max 0 , W 1 ⊙ M u l t i H e a d Q v 2 , K v 2 , V v 2 + b 1 + b 2
Finally, z 1 v 2 is input into a pooling layer, from which the [CLS] token z c l s v 2 is extracted from the input hidden states. The output is then processed through a linear layer and a Tanh activation function to obtain a more targeted, structured latent feature representation z v 2 .

3.2. Semantically Consistent Gated Fusion of Feature Representations

In multimodality forgery detection, existing models typically use a single-stream interaction where the text embedding feature z v 2 serves as the query interacting with the image embedding feature z v 1 , capturing correlations between different modalities. The fused features after interaction are then directly used for forgery detection and localization based on their correlation. This approach allows for global interaction between multimodal features and simplifies the process of interaction and fusion. However, when any modality contains small-scale targets or interfering information, the detection performance may be adversely affected. For example, at a certain time step, occlusions or other visual degradations in an image may lead to an incorrect forgery judgment, which in turn propagates erroneous signals to the textual modality. Such interference can ultimately result in inaccurate detection outcomes. In multimodal forgery detection, the fusion of visual and textual features is a critical step, as the accuracy of forgery recognition is highly dependent on the quality of multimodal feature fusion. Semantic consistency across different modalities plays a vital role in effective cross modal interaction and fusion. Motivated by this observation, we propose a consistency-gated fusion mechanism, which enforces semantic-consistency-aware supervision at each time step to guide cross-modal feature interaction and fusion. This mechanism effectively prevents the degradation of discriminative power in fused representations caused by ambiguous or noisy information from individual modalities, thereby improving the overall accuracy of subsequent forgery detection.
First, we take the image embedding feature z v 1 and the text embedding feature z v 2 as inputs and perform deep interaction through a multimodal cross-attention feature fusion module. The text feature z v 2 is passed through linear layers to generate the query feature z q v 2 , while the image feature z v 1 is passed through different linear layers to generate the key feature z k v 1 and the value feature z v v 1 , with all feature dimensions having size D. In this paper, we compute the correlation between z q v 2 and z k v 1 to obtain the global consistency information between text and image. The specific operations are as follows:
W = s o f t m a x z q v 2 · z k v 2 / D
The softmax operation is used to normalize the correlation strength between different modal features z q v 2 and z k v 1 . Meanwhile, the correlation strength matrix w is applied to the image feature z v v 1 , resulting in the cross-fused image feature representation aided by the text information. The process is as follows:
z c o v 1 = W T ⊙ z v v 1 ⊕ z v v 1
here, ⊕ represents the residual operation.
Second, we construct a semantic consistency gating module. By calculating the semantic similarity between different modalities, we generate gating signals to adjust the weight of feature fusion. If the similarity between the embedded features of different modalities is higher, the fusion weight will also be higher. This helps to avoid interference in subsequent decision-making when there is unclear information in one modality. Therefore, the semantic consistency gating ensures that only semantically consistent features are fused, facilitating the model to learn more discriminative features and improving the accuracy of feature fusion.
This is done by calculating the cosine similarity between the cross-fused image features z c o v 1 and the text embedding features z v 2 .
s i m = cos z c o v 1 , z ( v 2 ) = z c o v 1 · z ( v 2 ) z c o v 1 z ( v 2 )
In this case, z c o v 1 and z ( v 2 ) , respectively, represent the norms of the cross-fused image features z c o v 1 and the text embedding features z v 2 .
Based on the semantic consistency score, a sigmoid-based gating function is adopted to map similarity measurements into adaptive gating coefficients for feature fusion. This mechanism assigns higher weights to semantically consistent modalities and lower weights to inconsistent ones, effectively improving the discriminability of the fused multimodal features. The details are formulated as follows:
g = s i g m o d s i m = 1 1 + e s i m
Then, based on the gating signal, the cross fused image feature z v 1 and the text embedding feature z v 2 are weighted and fused to obtain semantically consistent cross fused image features and text embedding features.
z g cov 1 = g · z cov 1
z g v 2 = 1 − g · z v 2
Finally, this paper introduces a residual operation to enhance the global semantic-related features of both modalities, preventing information loss caused by the gating operation described above. The specific operation is as follows:
z g cov 1 = z g cov 1 ⊕ z cov 1
z g v 2 = z g v 2 ⊕ z v 2
At the same time, the global semantic consistency features from both modalities are concatenated. This allows for the acquisition of complementary multimodal fused features while avoiding noise from any individual modality. The specific operation is as follows:
z g = c o n c a t z g cov 1 , z g v 2

3.3. End-to-End Supervised Training

To enable a thorough exploration of interactions among multimodal information for forgery detection and localization, this work is inspired by the DGM4 framework [18] and jointly constructs a forgery sample loss and a classification loss. This design facilitates fully supervised learning of frame level local information in videos, while simultaneously incorporating a cross modal contrastive learning objective.
First, we align the semantics of image and text embeddings through a tampering-aware contrastive loss L MAC . By introducing the contrastive loss function, we maximize the distance between image-text pairs that exhibit semantic inconsistency, thereby enhancing discriminability during the training process. The contrastive loss from image to text is constructed as follows:
L i 2 t z i v 1 , z i v 2 + , z i v 2 − = − E p I , T log exp S z i v 1 , z i v 2 + / τ ∑ k = 1 k exp S z i v 1 , z k v 2 − / τ
The expression describes that τ is the temperature hyperparameter, z k v 2 − = { z 1 ( v 2 ) − , … , z k ( v 2 ) − } is a set of negative text samples that do not match x i v 1 , and S · represents the similarity calculation. To maintain the consistency of feature representations between textual and image information with the same semantic content, the contrastive loss from text to image is constructed in the same manner as follows:
L t 2 i z i v 2 , z i v 1 + , z i v 1 − = − E p I , T log exp S z i v 2 , z i v 1 + / τ ∑ k = 1 k exp S z i v 2 , z k v 1 − / τ
z k v 1 − = z 1 v 1 − , … , z k v 1 − is a set of negative text samples that do not match z i v 1 . To ensure that data information is not lost within a single modality, we further perform intra-modality contrastive learning in both modalities. The specific approach is as follows:
L i 2 i z i v 1 , z i v 1 + , z i v 1 − = − E p I , T log exp S z i v 1 , z i v 1 + / τ ∑ k = 1 k exp S z i v 1 , z k v 1 − / τ
L t 2 t z i v 2 , z i v 2 + , z i v 2 − = − E p I , T log exp S z i v 2 , z i v 2 + / τ ∑ k = 1 k exp S z i v 2 , z k v 2 − / τ
To maintain single-modality information while achieving multimodality collaborative learning, we combine all the losses into the following tampering-aware contrastive loss L MAC :
L M A C = 1 4 [ L i 2 t z i ( v 1 ) , z i ( v 2 ) + , z i ( v 2 ) − + L t 2 i z i ( v 2 ) , z i ( v 1 ) + , z i ( v 1 ) − + L i 2 i z i ( v 1 ) , z i ( v 1 ) + , z i ( v 1 ) − + L t 2 t z i ( v 2 ) , z i ( v 2 ) + , z i ( v 2 ) − ]
Second, we locate the manipulated image regions by searching for local patches that are inconsistent with the text embeddings. We perform cross-attention between the image and text embeddings to obtain patch embeddings that capture both the image and text relevance. Let u c l s represent the [CLS] token of both the text and image, and u p a t = u 1 , … , u N represent the N image patch embeddings interacting with the text embedding. To aggregate the spatial information of the local patches, we introduce an [AGG] token and add it to the sequence of local patch embeddings u p a t . By using the self-attention mechanism to process the local patch embedding sequence that includes the [AGG] token, the [AGG] token can aggregate the information from other patches, resulting in an aggregated embedding that contains rich spatial information, as shown below:
u a g g = A t t e n t i o n A G G , u p a t , u p a t
Unlike directly using the [CLS] token for bounding box (bbox) prediction, we follow the approach in [18] by employing a BBox detector D v . The input u a g g is fed into the detector, and combined with the generalized IoU loss [32], we compute the image processing localization loss as follows:
L I M G = E p ( I , T ) sigmod D v ( u a g g ) − z b o x + L I o U sigmod D v ( u a g g ) − z b o x
z b o x is the annotation vector of the facial bounding boxes saved in the data.
Since the multimodality complementary fusion feature z g not only fully explores the contextual information of the text but also interacts thoroughly with the image features, it is suitable for locating tampered text labels. We use the label detector D t to predict the label of each token in z g and calculate the cross-entropy loss to construct the operation localization loss as follows:
L T M G = E p I , T H D t z g , y t o k
where H · is the cross-entropy function, and y t o k is the label for text swap tampering. Then, based on the multimodality information z g generated by the semantic consistency gating, the binary classification loss for distinguishing between real and fake is as follows:
L B I C = E p I , T H C b z g , y b i n
where y b i n is the binary classification label, and C b is the binary classifier.
Finally, by combining all the above losses, hierarchical manipulation reasoning is performed to optimize the framework:
L = L MAC + L I M G + L T M G + L B I C

4. Experiments

4.1. Dataset Description and Experimental Setup

The proposed model’s effectiveness is validated on the popular multimodality sequence dataset DGM4. The specific dataset description and experimental setup used in this paper are as follows.

Dataset

The DGM4 multimodality dataset, proposed by Rui Shao et al. for multimodality detection, contains two modalities: vision and text. It was collected from 230,000 news samples, including 77,426 real image-text pairs and 152,574 tampered pairs. The tampering methods include: Image tampering: The tampered pairs include 66,722 face-swapping operations and 56,411 face-attribute operations. Text tampering: 43,546 text-swapping operations and 18,588 text-attribute operations. Multimodality tampering: 1/3 of the tampered images and 1/2 of the tampered texts were combined to form 32,693 mixed tampered pairs. The specific description of the dataset is shown in Table 1.
Table 1. DGM4 Dataset.
The challenge in implementing accurate anomaly detection for this dataset lies in the difficulty of achieving comprehensive forgery detection using high-precision models for individual modalities. As shown in Figure 3, where only text exhibits swapped-forgery behavior, focusing solely on image information would render the detection ineffective. In Figure 4, the text represents real news, while the image contains small-scale, subtle facial image forgery. If only the text information is considered, the detection would fail.
Figure 3. Examples of multimodal samples of fake instances in different situations. (a) Text tampering, image real. (b) Text real, small forged area in image.
Figure 4. In the figure, the left side shows the detection results of DGM4, while the right side shows our detection results. The red boxes represent the ground truth, and the blue boxes represent the model’s detection results. (a) Instance1, (b) Instance2, (c) Instance3, (d) Instance4, (e) Instance5, (f) Instance6, (g) Instance7, (h) Instance8. The content within the red squares in all sub-images represents the forged parts.

4.2. Experimental Parameter Settings

In this experiment, for ease of processing and computation, the image size is set to 256 × 256 pixels, and the length of the text content is padded or truncated to 50 tokens. This study leverages pre-trained single-modal feature representations from large models to obtain more accurate feature descriptions, including the VIT model for image processing and the BERT model for text processing. The pre-trained weights are sourced from ALBEF _4M [33]. The model training in this paper uses the AdamW optimizer with a weight decay rate of 0.02. During training, the learning rate gradually increases from the initial value to 1 × 10 − 4 and then decays to 1 × 10 − 6 using a cosine scheduling strategy. The number of training iterations is set to 100.
All experiments in this study are conducted on the following hardware: Nvidia RTX A4000 GPU cluster. The primary software environment includes Python 3.9, CUDA 11.2, Hugging Face Transformers 4.8.1, and the computer vision library OpenCV 4.5.5.62.

Experimental Evaluation Metrics

To ensure the fairness of the experiment, we use twelve metrics to evaluate the proposed model and the comparison methods. The following metrics are divided into several parts:
A.
For binary classification (real vs. fake), we evaluate Accuracy (ACC), the Area Under the ROC Curve (AUC), and Equal Error Rate (EER).
B.
For multi-label classification, we evaluate Mean Average Precision (MAP), Mean F1 per Class (CF1), and Overall F1 (OF1).
C.
For tampered image localization, we evaluate the Mean Intersection over Union (IoUmean) as well as IoU at thresholds of 0.5 (IoU50) and 0.75 (IoU75).
D.
For tampered text localization, we evaluate Precision, Recall, and F1 score.
A detailed description is shown in Table 2.
Table 2. Experimental Evaluation Metrics.

4.3. Experimental Results and Analysis

4.3.1. Comparison and Analysis of Multimodality Recognition Methods

In this section, we will present a performance comparison between other SOTA multimodality forgery detection methods and our approach on the DGM4 dataset, including CLIP [34], VILT [35], DGM4 [18], VIKI [28], UFAFormer [29], ASAP [30]. CLIP is one of the most popular dual-stream methods, which utilizes contrastive learning to embed images and text into the same vector space, enabling cross-modal retrieval and understanding. VILT is a lightweight multimodality model that fuses image and text information within the same Transformer framework for cross-modal understanding and interaction. The DGM4 model performs forgery detection based on interacting multimodality information from shallow to deep layers, achieving detection and localization of multimodality media tampering.VIKI enhances detection performance by leveraging large language models to enrich fused feature representations. UFAFormer captures continuous temporal dynamics in video data by incorporating frequency-domain modalities, thereby exploiting contextual relationships to improve detection accuracy. ASAP mitigates interference from multimodal noise and irrelevant content through cross-modal semantic alignment. We have compiled the experimental results of the comparison methods across different tasks, as shown in Table 3. As indicated in the table, our method achieves the best results across all tasks compared to other models.
Table 3. Comparison of Multimodality Forgery Detection Methods (Bold as the best result).
In the binary forgery detection task, the zero-shot performance of CLIP is highly dependent on textual prompts, as its training data are collected from the internet and may suffer from label bias, cultural bias, and data imbalance. Inaccurate or incomplete prompt design can, therefore, lead to significant performance degradation. Similarly, the VIKI model incorporates a pretrained natural language large model and thus encounters issues comparable to those of CLIP, resulting in reduced recognition accuracy under multimodal fusion. The VLIT model places greater emphasis on textual modality modeling, while its capability to represent visual information is relatively limited, which may impair its performance in visually dominant tasks. UFAFORMER introduces frequency-domain information to enrich feature representations and consequently achieves improved recognition accuracy compared to other models. However, it does not explicitly account for noise interference across different modalities, and its performance remains inferior to that of ASAP and the proposed method. The DGM4 framework fails to effectively handle interference introduced by multimodal information when semantic cues in a single modality are ambiguous, which can lead to erroneous detection results. ASAP mitigates noise interference through cross-modal semantic alignment and achieves the highest binary classification accuracy, yet it shows limited effectiveness when dealing with small-scale or blurred visual targets. By contrast, the proposed model introduces a multimodal semantic consistency gating mechanism to reinforce target feature representations, thereby achieving the best fine-grained recognition performance.
To further compare and analyze the effectiveness and efficiency of the proposed model, we report the parameter counts and computational metrics of both the DGM4 model and our proposed model in Table 4. Since our model employs a multimodal fusion approach with semantic consistency gating, it achieves reduced parameter complexity compared to DGM4 while maintaining high recognition accuracy.
Table 4. Comparative results of the computational efficiency and parameter size.

4.3.2. Single-Modality Deepfake Detection Methods: Comparison and Analysis

The multimodality deepfake detection method proposed in this paper can obtain more discriminative feature representations from different single modalities. Therefore, we further evaluate the detection performance when only single-modality data is available and compare and analyze it with popular single-modal methods. For the image modality, we compare our approach with two state-of-the-art (SOTA) deepfake detection methods. For the text modality, we compare it with two widely used sequence tagging methods and binary classification methods from the field of Natural Language Processing (NLP). As shown in Table 5, for deepfake detection in a single image modality, the proposed model achieves the optimal detection results. This is due to the fact that during training, the model learns discriminative information from images under the auxiliary supervision of other modalities. As a result, our model not only achieves the highest accuracy in binary classification tasks at the image level but also obtains the best detection region results in image-level forgery localization. The TS [36] method, which extracts high-frequency features through frequency-domain transformation and enhances them to highlight anomalies in forged images, may perform poorly when there is natural high-frequency noise (e.g., camera noise, compression artifacts) in real images, or when high-frequency information from low-quality images (such as low resolution or high compression) is lost or distorted. Additionally, MAT [11] introduces multiple attention modules to capture local anomalies in forged images or videos. However, due to the attention mechanism, the areas of focus may not align with the actual forgery traces. Therefore, it performs poorly in low-quality images or when forged images simulate real image features with finer details.
Table 5. Comparison of Single-modality Image Detection Methods.
Similarly, in the text modality, certain forged texts may use semantic embellishments or contextual disguises, making text based detection methods difficult to identify accurately. In this paper, we also present the recognition results of our method compared with popular methods in this modality, as shown in Table 6. Compared to other text detection methods, our model improves its detection capability by adding a semantic consistency supervision learning process between different modalities based on a pre-trained text recognition model, strengthening the overall detection of potential text forgeries. The BERT [37] model extracts semantic information through text feature extraction, but it is highly dependent on data quality. Thus, BERT struggles to extract semantic features or generalize when handling low quality text (e.g., grammatical errors, semantic ambiguity, spelling mistakes) or small scale datasets. As a result, BERT fails to accurately identify forged samples in low-quality or small scale datasets. Additionally, the LUKE [38] model constructs entity information in text through entity aware self-attention mechanisms and entity embeddings, but it is highly dependent on entity-annotated data and struggles with tasks lacking entity annotated data or with unseen or cross domain data. Therefore, compared to our approach, where the model learns discriminative information from text under the auxiliary supervision of other modalities during training, these two text detection methods achieve inferior results.
Table 6. Comparison of Single-modality Text Detection Methods.
The results validate that our approach, under the multimodality training framework, not only effectively integrates multimodality information but also achieves excellent detection and localization performance in each individual modality. In the image modality comparison, our method outperforms existing SOTA methods in both detection accuracy and localization. In the text modality comparison, our method achieves higher accuracy in both sequence labeling and binary classification tasks, indicating that multimodality training can effectively enhance the semantic understanding capability of text modalities. In conclusion, the multimodality deepfake detection method, by integrating information from multiple modalities such as images and text, can effectively overcome the limitations of single-modal methods and provide more comprehensive and accurate forgery detection capabilities.

4.3.3. Analysis of Experimental Results on the Effectiveness of Multi-Modal Fusion Features for Detection

To explore the impact of multimodality correlation on model performance in greater depth, we designed a series of ablation experiments. By separately retaining inputs and network components related to the image (Ours-Image) or text (Ours-Text) modalities, we evaluated the contribution of each modality to the final detection performance. The results are presented in Table 7 and Table 8. The experimental outcomes show that the complete multimodality fusion model significantly outperforms the ablated versions using a single modality, with a particularly noticeable advantage in the text modality. This finding underscores the critical role of cross-modal feature interactions in improving detection accuracy. Specifically, when the model relies solely on the image or text modality, the lack of complementary information from the other modality limits the model’s detection capability. This limitation becomes more apparent in cases where the image suffers from occlusions or incomplete information, further highlighting the drawbacks of single-modal detection. By effectively fusing both the image and text modalities, our model not only compensates for the limitations of a single modality but also extracts additional complementary information between modalities, thus, enhancing the accuracy of deepfake detection. Furthermore, the experimental results show that the text modality plays a more significant role in manipulation detection and localization tasks. This may be due to the inherent advantages of text information in handling semantic understanding and contextual relationships, providing richer discriminative features for the model. Therefore, how to better utilize text modality information during multimodality fusion is a key factor in improving model performance. In conclusion, the effectiveness of multimodality fusion features is not only reflected in the integration of the strengths of different modalities but also in the ability to establish effective interaction mechanisms between modalities. This allows the model to maintain high detection accuracy in complex scenarios. This study provides important theoretical foundations and practical guidance for the design and optimization of future multimodality fusion models.
Table 7. Ablation Study on the Image Modality.
Table 8. Ablation Study on the Text Modality.

4.3.4. Analysis of Experimental Results in Special Environments

In this section of the experiment, we conduct an in-depth analysis of the key role of multimodality information in forgery detection through visual results shown in Figure 4. Specifically, we highlight how the introduction of multimodality information may sometimes interfere with the subsequent discriminator model, especially in certain special scenarios. For example, when the tampered image has a small scale or when there are occlusions or other complex environments, traditional methods often struggle to accurately identify the forged areas. However, by incorporating semantic consistency across different modalities, our model demonstrates a significant advantage in the multimodality fusion-based forgery detection process. Specifically, Figure 4a,b,e,f show forged images of the face-swapping type. Although the clarity of these images is low, our model is still able to effectively identify the forged regions. Figure 4c,d,g,h display forged images of the facial attribute manipulation type. Among them, Figure 4c,f have small-scale manipulations, while Figure 4d,h contain partially occluded forged areas. In these scenarios, other methods failed to detect the problems in the images, but our method, by combining text information and comparing semantic consistency, successfully detected the forged regions. The experimental results indicate that our method, by constructing a semantic consistency gating mechanism, can better capture the correlations between different modalities. This mechanism directly utilizes the features after interactive fusion for forgery detection and localization, thereby achieving higher accuracy in multimodality forgery detection tasks.

4.4. Ablation Study

To verify the effectiveness of the introduced module, we visualized and compare the results of the proposed model with and without the consistency gating mechanism. When image and text features are fused solely through cross-attention, forged regions are not sufficiently emphasized, particularly in cases where the target objects are small in scale or the images contain substantial noise. By contrast, the proposed model employs a consistency gating mechanism to reinforce multimodal features with aligned semantics, preventing small-scale or blurred targets in the visual modality from being overlooked. At the same time, it suppresses the interference caused by background noise surrounding the forged regions, enabling more accurate localization of manipulated areas. As illustrated in Figure 5, we visualized several representative image pairs, from which it can be observed that the proposed model, benefiting from the consistency gating mechanism, achieves more precise localization and enhancement of forged targets.
Figure 5. Visualization comparison of full attention weight matrices before and after semantic consistency gating.

5. Conclusions

To address the issue that interference in any single modality can degrade the discriminative performance of fused features in multimodality video forgery detection, this paper proposes a Semantic Consistency Fusion approach. Specifically, the method constructs a semantic consistency gating mechanism based on the semantic similarity of high-dimensional embedded features, which enhances the discriminability of the embeddings and mitigates the adverse impact of noisy information on feature fusion. Experimental results on standard benchmark datasets demonstrate that the proposed method achieves promising detection performance compared to popular baseline models. However, the current method relies heavily on fully supervised labels for training, which are often expensive and difficult to obtain in real-world scenarios. This constraint limits the scalability and practical deployment of the model. Therefore, future work will focus on developing weakly supervised learning strategies for video forgery detection in the absence of fully annotated data.

Author Contributions

F.S. was responsible for the overall framework design of the paper and other related work; X.G. undertook the implementation of key technologies, model verification, and the drafting of the paper’s initial manuscript, among other tasks; T.Z. completed the experimental verification and visualization of the model and other relevant work; Y.L. was in charge of data collection and preprocessing and other related work; J.Z. undertook the implementation of the basic model process, revision of the paper, and overall supervision throughout the research. All authors have read and agreed to the published version of the manuscript.

Funding

National Natural Science Foundation of China Youth Fund (No. 61902165), Liaoning Provincial Department of Education Fund (No. JYTMS20231040).

Data Availability Statement

The datasets adopted in this study are all publicly available datasets.

Acknowledgments

This work is in part supported by National Natural Science Foundation of China Youth Fund (No. 61902165), Liaoning Provincial Department of Education Fund (No. JYTMS20231040).

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Gao, Y.; Wei, F.; Bao, J.; Gu, S.; Chen, D.; Wen, F.; Lian, Z. High-fidelity and arbitrary face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 16115–16124. [Google Scholar]
  2. Ma, L.; Yang, P.; Xu, Y.; Yang, Z.; Li, P.; Huang, H. Deep learning technology for face forgery detection: A survey. Neurocomputing 2025, 618, 129055. [Google Scholar] [CrossRef] [Scilit]
  3. Zheng, L.; Zhang, Y.; Thing, V.L.L. A survey on image tampering and its detection in real-world photos. J. Vis. Commun. Image Represent. 2019, 58, 380–399. [Google Scholar] [CrossRef] [Scilit]
  4. Nguyen, H.H.; Yamagishi, J.; Echizen, I. Capsule-forensics: Using capsule networks to detect forged images and videos. In Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 2307–2311. [Google Scholar]
  5. Dang, H.; Liu, F.; Stehouwer, J.; Liu, X.; Jain, A.K. On the detection of digital face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 5781–5790. [Google Scholar]
  6. Dang, L.M.; Hassan, S.I.; Im, S.; Lee, J.; Lee, S.; Moon, H. Deep learning based computer generated face identification using convolutional neural network. Appl. Sci. 2018, 8, 2610. [Google Scholar] [CrossRef] [Scilit]
  7. Masi, I.; Wu, Y.; Hassner, T.; Natarajan, P.; Medioni, G. Deep face recognition: A survey. In Proceedings of the 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Parana, Brazil, 29 October 2018–1 November 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 471–478. [Google Scholar] [CrossRef] [Scilit]
  8. Ismail, A.; Elpeltagy, M.; Zaki, M.S.; Hamed, K.H.; Wu, S. A new deep learning-based methodology for video deepfake detection using XGBoost. Sensors 2021, 21, 5413. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  9. Tariq, S.; Lee, S.; Kim, H.; Woo, Y.S.; Woo, S.S. Detecting both machine and human created fake face images in the wild. In Proceedings of the MPS’18: Proceedings of the 2nd International Workshop on Multimedia Privacy and Security, Toronto, ON, Canada, 15–19 October 2018; Association for Computing Machinery: New York, NY, USA, 2018; pp. 81–87. [Google Scholar] [CrossRef] [Scilit]
  10. Nhu, T.D.; Na, I.S.; Kim, S.H. Forensics face detection from GANs using convolutional neural network. In Proceedings of the International Symposium on Information Technology Convergence 2018 (ISITC), Jeju, Republic of Korea, 17–19 October 2018; ISITC: Bridgewater, NJ, USA, 2018; pp. 92–95. [Google Scholar]
  11. Zhao, H.; Zhou, W.; Chen, D.; Wei, T.; Zhang, W.; Yu, N. Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, 19–25 June 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 2185–2194. [Google Scholar]
  12. Xue, F.; Wang, Q.; Guo, G. TransFER: Learning relation-aware facial expression representations with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 3601–3610. [Google Scholar]
  13. Li, Y.; Lyu, S. Exposing DeepFake videos by detecting face warping artifacts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 16–17 June 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 46–52. [Google Scholar]
  14. Jung, T.; Kim, S.; Kim, K. DeepVision: Deepfakes detection using human eye blinking pattern. IEEE Access 2020, 8, 83144–83154. [Google Scholar] [CrossRef] [Scilit]
  15. Sabir, E.; Cheng, J.; Jaiswal, A.; AbdAlmageed, W.; Masi, I.; Natarajan, P. Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. arXiv 2019, arXiv:1905.00582. [Google Scholar] [CrossRef] [Scilit]
  16. Luo, G.; Darrell, T.; Rohrbach, A. NewsCLIP-pings: Automatic generation of out-of-context multimodal media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic, 7–11 November 2021; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 4351–4360. [Google Scholar]
  17. Abdelnabi, S.; Hasan, R.; Fritz, M. Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 14920–14929. [Google Scholar] [CrossRef] [Scilit]
  18. Shao, R.; Wu, T.; Liu, Z.; Yang, T.; Xu, Y.; Yuan, K. DGM4: Detecting and grounding multi-modal media manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 21786–21795. [Google Scholar]
  19. Xuan, X.; Peng, B.; Wang, W.; Dong, J.; Su, J. On the generalization of GAN image forensics. In Computer Vision—ACCV 2019; Springer International Publishing: Cham, Switzerland, 2019; pp. 234–250. [Google Scholar] [CrossRef] [Scilit]
  20. Zhou, P.; Han, X.; Morariu, V.I.; Davis, L.S. Two-stream neural networks for tampered face detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1831–1839. [Google Scholar] [CrossRef] [Scilit]
  21. Hsu, C.-C.; Zhuang, Y.-X.; Lee, C.-Y. Deep Fake Image Detection Based on Pairwise Learning. Appl. Sci. 2020, 10, 370. [Google Scholar] [CrossRef] [Scilit]
  22. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 2818–2826. [Google Scholar] [CrossRef] [Scilit]
  23. Li, Y.; Chang, M.-C.; Lyu, S. In ictu oculi: Exposing AI generated fake face videos by detecting eye blinking. In Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, China, 11–13 December 2018; IEEE: Piscataway, NJ, USA, 2019; pp. 1–7. [Google Scholar] [CrossRef] [Scilit]
  24. Mittal, T.; Bhattacharya, U.; Chandra, R.; Bera, A.; Manocha, D. Emotions don’t lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM International Conference on Multimedia (MM ‘20), Seattle, WA, USA, 12–16 October 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 2823–2832. [Google Scholar] [CrossRef] [Scilit]
  25. Güera, D.; Delp, E.J. Deepfake Video Detection Using Recurrent Neural Networks. In Proceedings of the 15th EEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New Zealand, 27–30 November 2018; IEEE: Piscataway, NJ, USA, 2019; pp. 1–6. [Google Scholar]
  26. Shi, C.; Zhang, T.; Li, J.; Wei, Q.; Zhang, L. Multi-modality complementary learning network with cross-modality interaction and adaptive fusion for face forgery detection. Eng. Appl. Artif. Intell. 2025, 162, 112373. [Google Scholar] [CrossRef] [Scilit]
  27. Khattar, D.; Goud, J.S.; Gupta, M.; Varma, V. MVAE: Multimodal variational autoencoder for fake news detection. In Proceedings of the World Wide Web Conference (WWW), San Francisco, CA, USA, 13–17 May 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 2915–2921. [Google Scholar] [CrossRef] [Scilit]
  28. Li, Q.; Gao, M.; Zhang, G.; Sun, H.; Gao, S.; Cheng, H. Towards multimodal disinformation detection by vision-language knowledge interaction. Inf. Fusion 2024, 102, 102037. [Google Scholar] [CrossRef] [Scilit]
  29. Liu, H.; Tan, Z.; Chen, Q.; Wei, Y.; Zhao, Y.; Wang, J. Unified frequency-assisted transformer framework for detecting and grounding multi-modal manipulation. Int. J. Comput. Vis. 2025, 133, 1392–1409. [Google Scholar] [CrossRef] [Scilit]
  30. Zhang, Z.; Wang, Y.; Cheng, L.; Zhong, Z.; Guo, D.; Wang, M. ASAP: Advancing semantic alignment promotes multi-modal manipulation detecting and grounding. arXiv 2024, arXiv:2412.12718. [Google Scholar] [CrossRef] [Scilit]
  31. Wang, J.; Liu, B.; Miao, C.; Zhao, Z.; Zhuang, W.; Chu, Q.; Yu, N. Exploiting modality-specific features for multi-modal manipulation detection and grounding. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 14–19 April 2024; IEEE: Piscataway, NJ, USA, 2024; pp. 8060–8064. [Google Scholar] [CrossRef] [Scilit]
  32. Rezatofighi, H.R.; Tsoi, N.; Gwak, J.; Sadeghian, A.; Reid, I.; Savarese, S. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–19 June 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 658–666. [Google Scholar]
  33. Li, J.; Selvaraju, R.R.; Gotmare, A.D.; Joty, S.; Xiong, C.; Hoi, S.C.H. Align before fuse: Vision and language representation learning with momentum distillation. Adv. Neural Inf. Process. Syst. 2021, 34, 9694–9705. [Google Scholar]
  34. Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), Online, 18–24 July 2021; PMLR: Cambridge, MA, USA, 2021; pp. 8748–8763. [Google Scholar]
  35. Kim, W.; Son, B.; Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In Proceedings of the International Conference on Machine Learning (ICML), Online, 18–24 July 2021; PMLR: Cambridge, MA, USA, 2021; pp. 5583–5594. [Google Scholar]
  36. Luo, Y.; Zhang, Y.; Yan, J.; Liu, W. Generalizing face forgery detection with high-frequency features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 16312–16321. [Google Scholar] [CrossRef] [Scilit]
  37. Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, 2–7 June 2019; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 4171–4186. [Google Scholar]
  38. Yamada, I.; Asai, A.; Shindo, H.; Takeda, H.; Matsumoto, Y. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16–20 November 2020; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 6442–6454. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.