RFEA: Reliable Forensic Evidence Aggregation for Deepfake Video Detection

Abstract

Deepfake video detection can benefit from multiple forensic cues, but the resulting responses are not always equally reliable or directly comparable. Moreover, informative forgery evidence may be concentrated in only part of a video. Consequently, indiscriminate fusion can allow unreliable sources to interfere with useful evidence, amplify differences in response scales, and dilute informative temporal regions. We propose Reliability-Filtered Evidence Aggregation (RFEA), a framework for converting heterogeneous forensic responses into robust video predictions without retraining or modifying the upstream detector. RFEA restricts fusion to a retained set of evidence sources, maps their responses into a comparable relative space through rank normalization based on the empirical cumulative distribution function (ECDF), and concentrates the final decision on frames with strong forensic evidence. These operations address source instability, response scale differences, and temporal variation in evidence strength. Experiments on five public deepfake video datasets show that indiscriminate fusion reduces the average area under the receiver operating characteristic curve (AUC) at the video level from 92.05% to 85.58%, demonstrating that additional forensic evidence is not necessarily beneficial when source reliability is ignored. RFEA instead achieves average AUC and average precision (AP) values of 92.68% and 95.16%, respectively. Compared with the same base response using the same temporal aggregation strategy, RFEA improves AUC and AP by 0.63 and 1.16 percentage points. A complementary replicate-level macro paired-bootstrap analysis further supports these positive aggregate gains. These results show that effective forensic evidence aggregation depends not only on the amount of available evidence, but also on its reliability, comparability, and temporal distribution.

1. Introduction

Deep generative models have advanced rapidly, accompanied by increasingly sophisticated face manipulation techniques. Existing methods can alter facial identities and attributes in videos and generate highly realistic facial content. However, deepfakes can also be misused for identity impersonation, opinion manipulation, and reputational attacks, posing an increasing threat to the trustworthiness of digital media. Therefore, automated deepfake video detection has become an important research problem.

Existing approaches mainly exploit forensic differences between authentic and manipulated content. Early studies primarily focused on single-frame detection, typically employing convolutional neural networks (CNNs) to capture texture anomalies, boundary artifacts, and spatial inconsistencies [1,2]. Subsequent studies further investigated geometric transformation artifacts and frequency-domain anomalies [3,4]. However, these low-level traces are often dependent on specific generation methods and can be altered by video compression, resolution changes, and other post-processing operations. More recent studies have shown that deepfake detectors may overfit to method-specific or forgery-specific patterns, resulting in degraded generalization when training and testing distributions differ [5,6,7]. Consequently, the robustness of single-frame detectors to unseen data domains remains limited.

Videos additionally provide continuous dynamic information, motivating extensive research on temporal and spatiotemporal cues for deepfake detection. Fully Temporal Convolution Network (FTCN) and Spatial-Temporal Inconsistency Learning (STIL) model temporal or spatiotemporal consistency across consecutive frames [8,9], whereas LipForensics and ID-Reveal exploit higher-level cues related to lip motion, identity characteristics, and natural behavioral patterns [10,11]. More recent studies have continued to improve cross-domain deepfake video detection by modeling temporal changes in latent representations, general spatiotemporal forgery artifacts, and vulnerability-aware spatiotemporal patterns [12,13,14]. These approaches broaden the range of forensic information available for deepfake video detection. Nevertheless, they primarily focus on learning more discriminative spatial, temporal, or spatiotemporal representations, while the joint treatment of evidence-source reliability, cross-source response alignment, and temporally concentrated evidence at the response-aggregation stage remains less explicitly studied.

Existing detectors can exploit diverse forensic cues, including spatial artifacts, frequency-domain anomalies, high-frequency traces, and blending-related artifacts [15,16,17]. Although these cues can provide complementary forensic information, the frame-level responses produced from different evidence sources are not necessarily equally reliable. Their discriminative behavior may vary with manipulation types, compression conditions, and domain shifts: some responses preserve relatively stable real–fake discrimination across datasets, whereas others are more sensitive to domain-specific artifacts. Moreover, heterogeneous evidence sources may produce responses with substantially different numerical scales, while informative forgery evidence is often concentrated in only a subset of video frames.

Directly combining these heterogeneous responses may, therefore, introduce three problems. First, unstable evidence sources can contaminate otherwise reliable responses. Second, cross-source differences in response scales may cause certain sources to exert a disproportionate influence on the fused evidence. Third, because informative forgery evidence is temporally non-uniform, aggregating all frames equally may dilute locally strong evidence, whereas relying on a single maximum response can make the video-level decision sensitive to isolated anomalous peaks. Consequently, incorporating more forensic cues does not necessarily lead to better video-level detection. Effective aggregation instead requires controlling which evidence sources participate in fusion, aligning heterogeneous responses onto a comparable evidence space, and concentrating the video-level decision on informative temporal evidence.

To address these issues, we propose Reliability-Filtered Evidence Aggregation (RFEA), a score-level framework that aggregates heterogeneous frame-level forensic responses into a robust video-level score without retraining or modifying the upstream detector. RFEA consists of three core mechanisms. Source-level reliability filtering restricts fusion to a retained evidence set, reducing interference from unstable responses. Relative evidence alignment applies rank normalization to map the retained responses into a comparable relative evidence space, mitigating cross-source scale discrepancies while preserving their discriminative ordering. Temporal evidence concentration aggregates a fixed proportion of high-evidence frames, reducing both weak-frame dilution and sensitivity to isolated response peaks. Together, these mechanisms address evidence-source reliability, cross-source comparability, and temporal non-uniformity in video-level evidence aggregation.

The main contributions of this work are summarized as follows:

1.

We propose Reliability-Filtered Evidence Aggregation (RFEA), a score-level framework for converting heterogeneous frame-level forensic responses into robust video-level predictions without retraining or modifying the upstream detector.

2.

We introduce source-level reliability filtering, relative evidence alignment, and temporal evidence concentration to address evidence-source instability, cross-source response-scale discrepancies, and temporally non-uniform forensic evidence, respectively.

3.

We conduct extensive evaluations on five public deepfake video datasets. The results show that RFEA improves the five-dataset average video-level performance over the same base response and naive fusion strategies, while paired bootstrap analysis further supports the stability of the observed gains.

2. Related Work

2.1. Image-Level Deepfake Detection

Image-level deepfake detection is typically formulated as a frame-level classification problem, where a cropped face image or video frame is classified as either real or fake. Large-scale datasets such as FaceForensics++ and Celeb-DF provide diverse manipulated samples for detector development and evaluation [2,18], while benchmarks such as DeepfakeBench further promote standardized preprocessing and evaluation protocols [19]. Early image-level methods commonly relied on CNN-based classifiers to learn spatial artifacts directly from red–green–blue (RGB) face images [1]. Face X-Ray subsequently exploited blending-boundary artifacts introduced when a manipulated face is composited into a target image [20], whereas Self-Blended Images (SBI) synthesized generic blending artifacts from authentic face images to reduce dependence on specific forgery techniques [21].

Subsequent image-level detectors have exploited richer forensic cues from multi-scale, frequency-domain, and facial-structure perspectives. Multi-Modal Multi-Scale Transformers (M2TR) captures subtle local inconsistencies at multiple spatial scales and complements RGB information with frequency-domain evidence [22]. AUNet models relationships among facial action units to identify structural inconsistencies in manipulated faces [23], while Space–Frequency Interactive Convolution (SFIConv) jointly models spatial and frequency-domain forgery traces [24]. Multi-Attentional Deepfake Detection employs multiple attention heads to localize discriminative facial regions and enhances subtle texture artifacts [25], whereas Spatial-Phase Shallow Learning (SPSL) combines spatial information with phase-spectrum representations to capture forgery traces in the frequency domain [26]. More recent studies have explicitly focused on cross-domain generalization. Uncovering Common Features (UCF) separates common forgery features from method-specific patterns [5], Latent Space Data Augmentation (LSDA) expands forgery diversity through latent-space augmentation [6], and Discrepancy Deepfake Detector (D³) investigates discrepancy learning under multi-generator training [7]. Forensics Adapter further adapts Contrastive Language–Image Pre-training (CLIP) for generalizable deepfake detection while retaining knowledge from large-scale pretraining [27]. eXplainable and eXtendable Deepfake Detection (X2-DFD) integrates specialized forensic feature detectors to supplement forgery-related information within an explainable and extendable detection framework [28].

2.2. Video-Level Deepfake Detection

Unlike single-frame detection, video-level deepfake detection can additionally exploit temporal information and has, therefore, motivated extensive research on temporal and spatiotemporal inconsistencies. Early studies used optical flow between adjacent frames to characterize abnormal motion patterns in manipulated videos [29]. Ganiyusufoglu et al. showed that spatiotemporal features can capture temporal abnormalities that are more transferable across manipulation techniques than purely spatial characteristics [30]. FTCN models short- and long-term temporal dependencies to exploit temporal incoherence [8], whereas AltFreezing balances the learning of spatial and temporal artifacts through alternating optimization [31]. Natural Consistency (NACO) learns natural consistency representations from authentic face videos [32], while the extended Thumbnail Layout framework (TALL++) reorganizes video clips into thumbnail layouts for efficient spatiotemporal modeling [33].

Recent studies have further explored more transferable or higher-level video representations. Choi et al. model temporal variations in style latent representations for cross-dataset detection [12], Yan et al. combine video-level blending with spatiotemporal adapter tuning [13], and FakeSTormer performs vulnerability-aware spatiotemporal learning to improve generalization to unseen manipulation domains [14]. Beyond explicit temporal artifacts, other approaches exploit physiological, semantic, or behavioral consistency. FakeCatcher extracts facial physiological signals [34], Yu et al. combine visual appearance with remote photoplethysmography cues [35], and LipForensics models natural lip motion through lip-reading pretraining [10]. RealForensics and UniForensics further learn natural facial dynamics or general video representations from authentic videos to improve robustness to unseen manipulations [36,37].

Beyond representation learning, score-level fusion has also been investigated for deepfake detection. Concas et al. systematically analyze different fusion rules for combining complementary detector outputs and examine their effects on detection generalization [38]. This line of work demonstrates the potential of combining multiple detector responses, but mainly focuses on generic fusion strategies rather than jointly considering differences in source reliability, response scales, and temporal evidence distribution.

Most existing video-level approaches improve detection by learning more discriminative or transferable temporal, spatiotemporal, physiological, or behavioral representations, while prior score-fusion studies primarily investigate how multiple detector outputs can be combined. Because these detector-level methods generally differ in training data, temporal sampling, and representation-learning protocols, their published results are not treated here as controlled score-fusion baselines. RFEA instead focuses on the reliability of response aggregation. Given heterogeneous forensic responses produced by an existing detector, it restricts fusion to retained evidence sources, aligns their responses in a comparable relative evidence space, and concentrates informative temporal evidence to obtain the final video score. In this sense, RFEA complements representation-learning approaches while addressing reliability, comparability, and temporal concentration within forensic evidence aggregation.

3. Methodology

3.1. Multi-Source Forensic Evidence Aggregation Formulation

Given a video $V_i={\left\{x_{i,t}\right\}}_{t=1}^{T_i}$, where $x_{i,t}$ denotes the t-th sampled frame and $T_i$ is the number of sampled frames, an upstream deepfake detector produces a frame-level discriminative response $s_{i,t}^0\in [0,1]$. We index the base response by $c=0$ and the responses associated with M auxiliary forensic cues by $c=1,\dots ,M$, yielding heterogeneous frame-level forensic responses ${\left\{s_{i,t}^c\right\}}_{c=0}^M$. The objective is to aggregate these responses into a robust video-level score.

The key challenge is not simply how to combine multiple responses, but how to account for their heterogeneity. Ideally, each evidence source should preserve the relative strength of the underlying frame-level forensic evidence. Under cross-domain conditions, however, different evidence sources can exhibit unequal discriminative reliability and substantially different numerical scales because of variations in manipulation mechanisms, compression conditions, generation methods, and other distribution shifts. Meanwhile, informative forgery evidence is often temporally non-uniform within a video. Direct aggregation, therefore, implicitly assumes that all evidence sources are sufficiently reliable, mutually comparable, and uniformly informative over time, assumptions that do not generally hold across data distributions.

To provide a conceptual formulation of source reliability, we consider the observation model in Equation (1).

$$s_{i,t}^c=g_c\left(z_{i,t}\right)+{ϵ}_{i,t}^c$$

(1)

where $z_{i,t}$ denotes the latent forensic evidence strength, $g_c(·)$ is the source-specific response mapping, and ${ϵ}_{i,t}^c$ represents perturbations induced by noise or domain shift. When $g_c(·)$ approximately preserves a monotonically increasing relationship with $z_{i,t}$, the resulting response retains the relative ordering of the underlying forensic evidence. When this relationship becomes weak or unstable across domains, the response can instead provide unreliable or misleading evidence.

This formulation motivates a ranking-based interpretation of source reliability. For two frames a and b, a reliable evidence source should tend to preserve their relative latent evidence strengths; that is, the sign of $s_a^c-s_b^c$ should agree with that of $z_a-z_b$. Persistent ranking disagreement indicates that the corresponding source may disturb the discriminative ordering after fusion. RFEA, therefore, does not aim to incorporate as many evidence sources as possible, but instead performs aggregation over a reliability-filtered source set while preserving the relative discriminative structure of the retained responses.

Based on this formulation, RFEA decomposes video-level evidence aggregation into three operations. As illustrated in Figure 1, source-level reliability filtering first restricts fusion to a retained evidence set. Relative evidence alignment then maps the retained responses into a comparable relative evidence space and performs frame-level fusion. Finally, temporal evidence concentration aggregates high-evidence frames to obtain the video-level score.

Figure 1. Overview of Reliability-Filtered Evidence Aggregation (RFEA). Heterogeneous frame-level forensic responses are first restricted to a retained evidence set through source-level reliability filtering. The retained responses are then rank-normalized and fused in a relative evidence space. Temporal evidence concentration subsequently aggregates high-evidence frames to obtain the final video-level score.

3.2. Source-Level Reliability Filtering and Relative Evidence Alignment

Let $\mathcal{C}=\{0,1,\dots ,M\}$ denote the set of available evidence sources, where $c=0$ corresponds to the base response and $c=1,\dots ,M$ correspond to auxiliary forensic responses. We denote the retained auxiliary-source set by ${\mathcal{E}}_r\subseteq \{1,\dots ,M\}$ and define the evidence set participating in subsequent fusion as ${\mathcal{C}}_r=\left\{0\right\}\cup {\mathcal{E}}_r$. The base response is retained as the primary detection evidence, whereas only auxiliary sources included in ${\mathcal{E}}_r$ participate in the subsequent alignment and fusion stages. RFEA applies source-level reliability filtering by restricting aggregation to this retained source set. The fixed retained-source configuration used in our experiments is specified in Section 4.1. In the current formulation, ${\mathcal{C}}_r$ is treated as a fixed source-configuration input rather than being estimated automatically or adaptively at inference time.

This source-level restriction is motivated by the propagation of unreliable responses during fusion. If a source whose response ordering is poorly aligned with the underlying forensic evidence is incorporated directly, its ranking disagreement can disturb the discriminative structure already preserved by more reliable responses. Source-level reliability filtering, therefore, limits the set of participating responses before cross-source alignment, rather than indiscriminately combining all available forensic cues.

Even within ${\mathcal{C}}_r$, however, raw responses from different evidence sources may exhibit substantially different distributional shapes and numerical scales. Direct averaging in the raw response space can consequently make the fused evidence disproportionately influenced by source-specific score ranges rather than by relative discriminative strength. RFEA addresses this problem through relative evidence alignment.

For an evidence source c, let ${\left\{s_n^c\right\}}_{n=1}^N$ denote an unlabeled reference pool of N frame-level responses used to characterize its empirical response distribution. The aligned response is obtained using the empirical cumulative distribution function (ECDF) in Equation (2).

$${\widehat{s}}_{i,t}^c={\mathrm{\Phi }}_c\left(s_{i,t}^c\right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}\mathbb{I}\left[s_n^c\le s_{i,t}^c\right]$$

(2)

where $\mathbb{I}[·]$ denotes the indicator function and ${\mathrm{\Phi }}_c(·)$ is the source-specific ECDF. Equation (2) maps each raw response to its relative position within the empirical distribution of the corresponding evidence source. Since ${\mathrm{\Phi }}_c(·)$ is monotone non-decreasing, the transformation does not invert the within-source ordering of responses, although tied values may share the same relative position. It, therefore, retains the available ordinal structure of each source while reducing sensitivity to heterogeneous numerical scales and calibration ranges. In our experiments, the unlabeled reference pool is constructed separately for each evaluated dataset from its frame-level responses, without using real/fake labels. This corresponds to a label-free transductive normalization setting, since the empirical response distribution of the evaluation batch is available when constructing the relative mapping. Operationally, tied responses are assigned a common tie-aware empirical percentile position; the reference-pool sizes for the evaluated datasets are reported in Section 4.1.

After relative evidence alignment, the retained responses are fused according to Equation (3).

$$f_{i,t}={\displaystyle \frac{1}{|{\mathcal{C}}_r|}}\sum _{c\in {\mathcal{C}}_r}{\widehat{s}}_{i,t}^c$$

(3)

where $f_{i,t}$ denotes the fused frame-level evidence response. Rather than averaging heterogeneous raw values directly, Equation (3) integrates the base and retained auxiliary responses after they have been mapped into the common relative evidence space.

The effect of multi-source fusion can be further interpreted using a simple noisy-observation model. We represent each aligned response according to Equation (4).

$${\widehat{s}}_{i,t}^c=z_{i,t}^{\prime }+b_c+{\eta }_{i,t}^c$$

(4)

where $z_{i,t}^{\prime }$ denotes the normalized latent evidence strength, $b_c$ represents a source-specific systematic bias, and ${\eta }_{i,t}^c$ is a zero-mean perturbation. Source-level reliability filtering is intended to prevent sources with unstable response mappings or substantial systematic deviations from participating indiscriminately in fusion. In the general case, the perturbations from different retained sources may be statistically dependent. Treating $z_{i,t}^{\prime }$ and the source-specific bias terms as fixed, the conditional variance of the fused response can be written as Equation (5).

$$Var\left(f_{i,t}\mid z_{i,t}^{\prime }\right)={\displaystyle \frac{1}{|{\mathcal{C}}_r{|}^2}}\left[\sum _{c\in {\mathcal{C}}_r}{\sigma }_c^2+2\sum _{\begin{array}{c}c,d\in {\mathcal{C}}_r\\ c<d\end{array}}Cov\left({\eta }_{i,t}^c,{\eta }_{i,t}^d\right)\right].$$

(5)

where ${\sigma }_c^2$ denotes the variance of the source-specific perturbation, defined as ${\sigma }_c^2=Var\left({\eta }_{i,t}^c\right)$. When the retained-source perturbations are approximately independent, the covariance terms vanish and Equation (5) reduces to the illustrative special case as Equation (6).

$$Var\left(f_{i,t}\mid z_{i,t}^{\prime }\right)={\displaystyle \frac{1}{|{\mathcal{C}}_r{|}^2}}\sum _{c\in {\mathcal{C}}_r}{\sigma }_c^2$$

(6)

Equations (5) and (6) show that the effect of averaging depends on both source-specific perturbation variances and their cross-source dependence; Equation (6) represents the simplified independent-perturbation case. Source-level reliability filtering limits interference from unreliable sources, while relative evidence alignment reduces cross-source scale mismatch before fusion. Their combination yields the fused frame-level evidence response used for subsequent temporal evidence concentration.

3.3. Temporal Evidence Concentration

After frame-level fusion, each video is represented by the fused response sequence ${\left\{f_{i,t}\right\}}_{t=1}^{T_i}$. These frame-level responses must be further aggregated into a single video-level score. The key observation is that informative forgery evidence is often temporally non-uniform. Some frames may contain strong manipulation traces, whereas others provide only weak evidence because of blur, occlusion, pose variation, compression, or limited visibility of manipulated regions.

We illustrate this effect using a simple two-group model. Suppose that a proportion $\pi$ of frames in a forged video contain strong forgery evidence. Let ${\mu }_1$ and ${\mu }_0$ denote the mean fused responses of strong- and weak-evidence frames, respectively, with $\mathrm{\Delta }={\mu }_1-{\mu }_0>0$. Under full-frame averaging, the expected response gain over the weak-evidence background is given in Equation (7).

$$\mathbb{E}\left[S_i^{\mathrm{Mean}}\right]-{\mu }_0=\pi \mathrm{\Delta }$$

(7)

Equation (7) shows that when $\pi$ is small, the contribution of strong-evidence frames is diluted by the larger number of weak-evidence frames. In contrast, max aggregation preserves the strongest response but makes the video-level score entirely dependent on a single frame, thereby increasing sensitivity to isolated anomalous peaks. Temporal aggregation, therefore, needs to preserve concentrated high-evidence responses without relying exclusively on a single extreme observation.

RFEA implements temporal evidence concentration through fixed-ratio high-evidence frame aggregation. For video $V_i$, we select $K_i=max(1,\lceil \alpha T_i\rceil )$ frames, where $\alpha$ denotes the aggregation ratio. Let ${\mathcal{T}}_i^K$ denote the index set corresponding to the $K_i$ largest values in ${\left\{f_{i,t}\right\}}_{t=1}^{T_i}$.

The final video-level score is defined in Equation (8).

$$S_i^{\mathrm{RFEA}}={\displaystyle \frac{1}{K_i}}\sum _{t\in {\mathcal{T}}_i^K}{f}_{i,t}$$

(8)

When $\alpha <1$, Equation (8) concentrates the video-level decision on frames with relatively strong forensic evidence rather than weighting all frames equally. Compared with full-frame averaging, this reduces dilution from weak-evidence frames. Compared with max aggregation, averaging over the selected set assigns each selected response a direct weight of $1/K_i$, reducing the dependence of the final score on any single response peak.

In the main experiments, we set $\alpha =0.1$, corresponding to aggregation over the top 10% highest-evidence frames, and further evaluate neighboring aggregation ratios in the sensitivity analysis. The same value is used globally for all evaluated datasets and videos and is not adapted on a per-dataset or per-sample basis. The neighboring ratios are reported only as a sensitivity analysis; if $\alpha$ is to be tuned in another setting, it should be selected using data kept separate from the final test set. For ranking-based metrics such as the area under the receiver operating characteristic curve (AUC) and average precision (AP), the continuous video-level score $S_i^{\mathrm{RFEA}}$ in Equation (8) is used directly. Best F1 is obtained by threshold scanning as described in Section 4.1, whereas the binary labels used in the qualitative case visualizations are obtained with a fixed score threshold of 0.5. RFEA, thus, converts the fused frame-level evidence responses into the final video-level score through temporal evidence concentration.

4. Experiments

4.1. Experimental Setup

Datasets. We evaluate RFEA on five public deepfake video detection datasets: UADFV [39], Celeb-DF [18], FaceForensics++ [2], the Deepfake Detection Challenge Preview (DFDCP) [40], and the Deepfake Detection Challenge (DFDC) [41]. These datasets cover early small-scale benchmarks, high-quality face manipulations, and large-scale videos collected under more complex data distributions. We report both frame-level and video-level results, with video-level detection being the primary focus of this work. Following the official X2-DFD [28] protocol, FaceForensics++ serves as the primary training-domain dataset for the upstream detector, whereas UADFV, Celeb-DF, DFDCP, and DFDC provide external-domain evaluations.

Table 1 reports the actual evaluation scale used in our experiments. The number of evaluated frames refers to frames retained after video sampling and face cropping for which valid forensic responses are available, rather than all frames contained in the original videos. The evaluated videos, frame sampling, and face preprocessing follow the official X2-DFD/DeepfakeBench evaluation protocol; RFEA does not introduce any additional dataset-specific video or frame selection. For FaceForensics++, we follow the X2-DFD setting with c23 compression and the standard Deepfakes, Face2Face, FaceSwap, and NeuralTextures manipulation types. DFDCP is the preview release of the Deepfake Detection Challenge dataset, whereas DFDC is the subsequently released large-scale version. Although both belong to the same dataset family, they differ substantially in scale and data distribution and are, therefore, evaluated separately.

Table 1. Evaluation scale of the five deepfake video datasets. DFDCP denotes Deepfake Detection Challenge Preview, and DFDC denotes Deepfake Detection Challenge.

Dataset	Real Videos	Fake Videos	Evaluated Frames	Task
UADFV	49	49	3099	Frame/Video
Celeb-DF	178	340	16,420	Frame/Video
FaceForensics++	140	280	13,436	Frame/Video
DFDCP	230	422	17,162	Frame/Video
DFDC	2315	2389	132,116	Frame/Video

Evaluation metrics. We evaluate detection performance using four metrics: area under the receiver operating characteristic curve (AUC), average precision (AP), equal error rate (EER), and Best F1. Treating fake samples as the positive class, let $\tau$ denote a decision threshold, and let ${\mathrm{TP}}_{\tau }$, ${\mathrm{FP}}_{\tau }$, ${\mathrm{TN}}_{\tau }$, and ${\mathrm{FN}}_{\tau }$ denote the corresponding numbers of true positives, false positives, true negatives, and false negatives. The true-positive rate and false-positive rate are defined as ${\mathrm{TPR}}_{\tau }={\mathrm{TP}}_{\tau }/({\mathrm{TP}}_{\tau }+{\mathrm{FN}}_{\tau })$ and ${\mathrm{FPR}}_{\tau }={\mathrm{FP}}_{\tau }/({\mathrm{FP}}_{\tau }+{\mathrm{TN}}_{\tau })$, respectively.

AUC measures the threshold-independent ranking ability of the detector by integrating the receiver operating characteristic curve over all operating points, as defined in Equation (9).

$$\mathrm{AUC}={\int }_0^1\mathrm{TPR}\left(\mathrm{FPR}\right)d\mathrm{FPR}$$

(9)

A higher AUC indicates a stronger ability to rank fake samples above real samples independently of a specific decision threshold. AP summarizes the precision–recall relationship and is particularly informative under class imbalance. Let ${\mathrm{P}}_k$ and ${\mathrm{R}}_k$ denote precision and recall at the k-th operating point after samples are ranked by their detection scores. AP is computed according to Equation (10).

$$\mathrm{AP}=\sum _k\left({\mathrm{R}}_k-{\mathrm{R}}_{k-1}\right){\mathrm{P}}_k$$

(10)

where precision and recall are given by ${\mathrm{P}}_{\tau }={\mathrm{TP}}_{\tau }/({\mathrm{TP}}_{\tau }+{\mathrm{FP}}_{\tau })$ and ${\mathrm{R}}_{\tau }={\mathrm{TP}}_{\tau }/({\mathrm{TP}}_{\tau }+{\mathrm{FN}}_{\tau })$. EER characterizes the operating point at which the false-positive rate and false-negative rate are equal. With ${\mathrm{FNR}}_{\tau }={\mathrm{FN}}_{\tau }/({\mathrm{TP}}_{\tau }+{\mathrm{FN}}_{\tau })=1-{\mathrm{TPR}}_{\tau }$, EER is defined in Equation (11).

$$\mathrm{EER}={\mathrm{FPR}}_{{\tau }^{*}}={\mathrm{FNR}}_{{\tau }^{*}},{\tau }^{*}:{\mathrm{FPR}}_{{\tau }^{*}}={\mathrm{FNR}}_{{\tau }^{*}}$$

(11)

A lower EER indicates better separation between real and fake samples. Finally, the F1 score at threshold $\tau$ is the harmonic mean of precision and recall. We report Best F1 by scanning the decision threshold and selecting the maximum F1 value, as defined in Equation (12).

$$\mathrm{BestF}1=\underset{\tau }{max}{\displaystyle \frac{2{\mathrm{P}}_{\tau }{\mathrm{R}}_{\tau }}{{\mathrm{P}}_{\tau }+{\mathrm{R}}_{\tau }}}$$

(12)

Frame-level metrics are computed over all evaluated face frames, whereas video-level metrics are computed from one aggregated score per video. Frame-level results are used to examine the discriminative quality of the fused frame-level evidence responses, while video-level results evaluate whether these responses can be reliably aggregated into video-level scores. Because Best F1 explicitly depends on threshold selection, we primarily use AUC, AP, and EER to discuss performance across datasets and report Best F1 as a complementary threshold-dependent metric. The fixed real/fake labels used only in the qualitative case visualizations are obtained with a score threshold of 0.5.

Implementation protocol. We use frame-level forensic responses reproduced with X2-DFD [28] as the upstream detector outputs. We follow the official X2-DFD training and evaluation protocol; in particular, the upstream detector uses FaceForensics++ with c23 compression as its primary training domain, together with the forensic-expert configuration defined in X2-DFD. All RFEA experiments operate on the same set of saved frame-level responses. Given these precomputed responses, RFEA does not retrain or modify the upstream detector, introduce an additional learnable model or model-inference pass, or add learnable parameters. It operates solely as a score-level post-processing framework that performs source-level reliability filtering, relative evidence alignment, and temporal evidence concentration.

The upstream X2-DFD responses are reproduced on a Linux server equipped with an NVIDIA RTX 4090 graphics processing unit (GPU) using Python 3.10.20 and PyTorch 2.5.1+cu121. We follow the same face-frame preprocessing pipeline for all evaluated samples. RFEA subsequently processes the saved frame-level responses directly. For fair comparison, all experimental settings use the same test videos, sampled frames, face crops, and evaluation scripts.

We use the same fixed retained-source configuration across all evaluated datasets. The base response is always retained as the primary evidence source, while the blending-artifact cue is retained as the auxiliary source participating in RFEA. The diffusion-artifact cue is used for source diagnosis and naive-fusion comparison but is excluded from the final RFEA fusion. The retained-source configuration is not adapted on a per-dataset or per-sample basis during evaluation. Accordingly, the current implementation treats the retained source set as a predefined global configuration rather than an automatically estimated inference-time variable. Rank normalization estimates relative positions from unlabeled frame-level response distributions and does not use real/fake labels. The reference distribution is constructed separately for each dataset, with $N=3099$, $16,420$, $13,436$, $17,162$, and $132,116$ frame-level responses for UADFV, Celeb-DF, FaceForensics++, DFDCP, and DFDC, respectively. This is a label-free transductive normalization setting because the unlabeled evaluation response distribution is available when constructing the relative mapping. In the main video-level comparison, all settings except Base-Max use the same top-10% aggregation ratio so that differences primarily reflect evidence-source selection and fusion rather than different temporal aggregation settings.

Compared settings. To isolate the effects of source-level reliability filtering, relative evidence alignment, and temporal evidence concentration, we consider the following settings. Base denotes the frame-level response produced by X2-DFD. Blending-artifact Cue (BC) denotes the auxiliary forensic response associated with face blending boundaries and compositing artifacts, whereas Diffusion-artifact Cue (DC) denotes the auxiliary response associated with diffusion-generation artifacts. Base, BC, and DC are stored as distinct response quantities: Base is the final X2-DFD detector response under the dual-expert configuration, whereas BC and DC are the corresponding raw expert responses supplied to X2-DFD. They should, therefore, not be interpreted as a numerical decomposition of the Base score. Naive-All directly fuses Base, BC, and DC in the raw response space and is used to examine the effect of indiscriminately incorporating all available sources. Raw-Selected Fusion (RSF) fuses only Base and BC in the raw response space without rank normalization, thereby isolating the effect of source-level reliability filtering. RFEA applies relative evidence alignment and frame-level fusion over the same retained evidence set.

For video-level evaluation, Base-Max uses the maximum Base response as the video-level score. Base-T10 applies top-10% aggregation to Base and, therefore, isolates the effect of temporal evidence concentration from multi-source fusion. BC-T10 and DC-T10 evaluate the video-level discriminative behavior of the two auxiliary evidence sources individually. Naive-All-T10 applies the same temporal aggregation after indiscriminate raw-space fusion, whereas RSF-T10 applies it after source-level reliability filtering without relative evidence alignment. RFEA-T10 denotes the complete video-level method combining source-level reliability filtering, relative evidence alignment, and temporal evidence concentration.

4.2. Comparative Results Across Datasets

We evaluate RFEA at both the frame and video levels. Frame-level results assess whether source-level reliability filtering and relative evidence alignment produce more discriminative fused evidence responses, whereas video-level results evaluate whether these responses can be further converted into robust video-level scores through temporal evidence concentration.

The frame-level results across the five evaluated datasets are reported in Table 2. The two auxiliary evidence sources exhibit markedly different behavior across datasets. BC provides competitive auxiliary discriminative information across several datasets, although it does not independently outperform Base under every data distribution. This observation indicates that the value of an auxiliary source does not require it to outperform the base response on every dataset; rather, it should provide sufficiently stable and complementary discriminative information for subsequent fusion. In contrast, DC performs close to chance level on several datasets and falls substantially below chance on UADFV, showing that its response ordering is not consistently aligned with the real–fake discrimination objective across domains.

Table 2. Frame-level detection results across the five evaluated datasets (%). Best average results are highlighted in bold. Upward arrows indicate higher-is-better metrics, whereas downward arrows indicate lower-is-better metrics.

	UADFV				Celeb-DF				FaceForensics++				DFDCP				DFDC				Average
Method	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑
Base	97.83	98.67	3.94	97.12	86.08	92.51	22.20	84.17	89.05	94.70	19.25	86.02	87.36	93.11	21.16	85.53	81.25	82.15	26.16	75.68	88.32	92.23	18.54	85.70
BC	97.91	98.46	5.36	94.69	87.75	93.22	20.56	85.58	86.63	93.02	21.75	85.31	86.19	92.49	22.66	84.93	83.27	85.97	25.04	76.60	88.35	92.63	19.07	85.42
DC	32.17	36.33	64.89	66.71	50.17	63.26	49.75	79.35	51.21	64.81	48.80	80.00	48.25	66.47	51.75	79.24	47.21	49.43	51.86	68.52	45.80	56.06	53.41	74.76
Naive-All	95.97	94.52	7.42	93.32	80.04	88.70	28.86	82.94	83.73	90.78	25.61	84.65	82.81	89.78	26.61	84.19	77.78	79.62	30.35	73.88	84.07	88.68	23.77	83.79
RSF	97.61	98.40	5.29	94.80	86.58	93.05	21.94	84.24	87.74	93.62	19.22	86.06	87.02	92.83	21.18	85.62	82.75	85.72	25.57	75.98	88.34	92.72	18.64	85.34
RFEA	97.98	98.71	4.45	96.03	87.71	93.48	20.60	85.56	88.58	94.31	19.51	86.24	87.64	93.25	21.47	85.93	83.29	85.81	24.99	76.75	89.04	93.11	18.21	86.10

The fusion results further show that increasing the number of evidence sources does not necessarily improve detection. Naive-All reduces the five-dataset average AUC from 88.32% for Base to 84.07% and increases EER from 18.54% to 23.77%, indicating that an unstable source can disrupt otherwise discriminative responses. After DC is excluded, RSF recovers the average AUC to 88.34%, demonstrating the importance of source-level reliability filtering. Applying relative evidence alignment over the same retained source set further increases the average AUC to 89.04% and AP to 93.11%, while reducing EER to 18.21%. RFEA consequently achieves the best five-dataset average across all four frame-level metrics.

Table 3 reports the video-level results, which more directly evaluate the primary objective of RFEA. Replacing Base-Max with Base-T10 increases the average AUC from 91.38% to 92.05% and reduces EER from 15.70% to 14.62%. This comparison shows that relying exclusively on a single maximum-response frame is suboptimal; aggregating a set of high-evidence frames can preserve informative responses while reducing sensitivity to isolated peaks.

Table 3. Video-level detection results across the five evaluated datasets (%). Best average results are highlighted in bold. Upward arrows indicate higher-is-better metrics, whereas downward arrows indicate lower-is-better metrics.

	UADFV				Celeb-DF				FaceForensics++				DFDCP				DFDC				Average
Method	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑
Base-Max	99.67	99.71	2.04	98.97	93.92	95.34	13.51	90.34	91.09	95.73	17.86	89.48	88.99	93.79	20.86	86.95	83.25	81.51	24.23	77.43	91.38	93.22	15.70	88.63
Base-T10	99.79	99.81	2.04	98.97	94.18	96.78	13.36	90.54	91.93	96.25	14.46	90.39	90.21	94.30	19.95	88.07	84.15	82.77	23.28	78.08	92.05	93.98	14.62	89.21
BC-T10	99.63	99.66	2.04	97.96	95.53	97.60	12.36	92.84	88.87	94.22	17.86	87.21	89.39	94.06	18.25	86.76	86.03	87.93	23.11	78.11	91.89	94.69	14.72	88.57
DC-T10	27.30	37.92	69.39	66.67	48.68	61.76	51.73	79.25	46.62	64.42	52.14	80.00	51.70	65.50	49.55	78.58	46.13	47.49	52.87	67.36	44.09	55.42	55.13	74.37
Naive-All-T10	91.17	87.08	13.27	93.20	89.02	94.40	18.53	86.49	83.80	91.32	23.39	83.13	85.12	91.07	23.13	83.62	78.78	80.22	28.68	72.72	85.58	88.82	21.40	83.83
RSF-T10	99.75	99.78	2.04	98.97	95.44	97.57	12.36	92.63	89.17	94.41	18.04	87.86	90.19	94.38	18.25	87.65	86.40	88.12	22.85	78.56	92.19	94.85	14.71	89.13
RFEA-T10	99.79	99.81	2.04	98.97	95.42	97.52	11.93	92.31	91.26	95.77	15.00	89.84	90.73	94.84	18.71	88.13	86.21	87.85	22.47	78.37	92.68	95.16	14.03	89.52

The effect of source-level reliability filtering is more pronounced at the video level. Naive-All-T10 achieves an average AUC of 85.58%, substantially below the 92.05% obtained by Base-T10, showing that indiscriminately incorporating additional sources can introduce substantial interference. Excluding DC increases the average AUC to 92.19% for RSF-T10, supporting the role of source-level reliability filtering. With the same retained source set, RFEA-T10 further increases the average AUC to 92.68% and AP to 95.16%, while reducing EER to 14.03%. These comparisons indicate that the final improvement cannot be attributed to temporal aggregation alone, but arises from the combined effects of source-level reliability filtering, relative evidence alignment, and temporal evidence concentration. At the individual-dataset level, however, the improvement is not uniform across all benchmarks. Relative to Base-T10, RFEA-T10 provides a modest five-dataset average AUC gain of 0.63 percentage points, while the relative ordering of the compared settings varies across individual datasets.

We further examine representative video-level decisions to illustrate how different aggregation strategies affect individual samples. Because RFEA operates on forensic responses rather than spatial localization maps, the qualitative analysis focuses on changes in video-level scores and prediction outcomes.

Figure 2 illustrates two representative behaviors. In Figure 2a, Base and Naive-All assign high forgery scores to real videos and produce false positives, whereas RFEA reduces the contribution of unstable evidence and recovers the correct real predictions. In Figure 2b, Naive-All suppresses the forgery evidence of several fake videos sufficiently to yield incorrect predictions, whereas RFEA preserves stronger discriminative evidence and maintains the correct fake predictions. These cases qualitatively illustrate how reliability-aware response aggregation can improve the robustness of individual video-level decisions.

Figure 2. Representative video-level decision cases. All scores are obtained using top-10% temporal aggregation. Green denotes a correct prediction and red denotes an incorrect prediction.

Figure 3 shows the frame-level response trajectories of Base, Naive-All, and RFEA. Informative forgery evidence is concentrated in limited temporal regions rather than being uniformly distributed throughout the video. Temporal evidence concentration emphasizes these high-evidence regions while averaging over multiple selected frames, thereby reducing both full-frame dilution and excessive dependence on a single response peak.

Figure 3. Temporal response trajectories and top-10% evidence selection. Dots mark the frames included in the top-10% aggregation set.

4.3. Ablation and Diagnostic Analyses

Evidence-source and fusion-protocol ablation. We first examine the contributions of source-level reliability filtering and relative evidence alignment using the five-dataset average results in Table 4. DC performs poorly at both the frame and video levels, indicating that it is unsuitable as a retained evidence source under the current protocol. Naive-All substantially degrades relative to Base, demonstrating that indiscriminate fusion can corrupt otherwise discriminative responses. Removing DC recovers performance in RSF, which isolates the effect of source-level reliability filtering. With the same retained source set, RFEA further applies relative evidence alignment and achieves the best average performance at both evaluation levels. The comparison between RSF and RFEA, therefore, isolates the additional contribution of rank-normalized alignment beyond source filtering alone. Because Base and BC are both generated within the X2-DFD pipeline, we additionally examine their empirical dependence. Across the five evaluated datasets, their Pearson correlations range from 0.57 to 0.80 and their Spearman correlations range from 0.88 to 0.93, confirming that the two responses are statistically related rather than independent.

Table 4. Ablation of evidence sources and fusion protocols (%). Upward arrows indicate higher-is-better metrics, whereas downward arrows indicate lower-is-better metrics. Boldface identifies the proposed RFEA setting and the best average results.

	Frame Level				Video Level
Setting	AUC↑	AP↑	EER↓	Best F1↑	AUC↑	AP↑	EER↓	Best F1↑
Base	88.32	92.23	18.54	85.70	92.05	93.98	14.62	89.21
BC	88.35	92.63	19.07	85.42	91.89	94.69	14.72	88.57
DC	45.80	56.06	53.41	74.76	44.09	55.42	55.13	74.37
Naive-All	84.07	88.68	23.77	83.79	85.58	88.82	21.40	83.83
RSF	88.34	92.72	18.64	85.34	92.19	94.85	14.71	89.13
RFEA	89.04	93.11	18.21	86.10	92.68	95.16	14.03	89.52

Expert-cue source diagnosis. We further diagnose the discriminative behavior of different expert-cue configurations on Celeb-DF. No Expert (NE) uses no auxiliary expert cue; Blending Only (BO) introduces only the blending-related cue; Diffusion Only (DO) introduces only the diffusion-related cue; and Dual Experts (DE) uses both cues. This experiment is intended specifically to diagnose the reliability differences between expert-cue sources. Its protocol differs from the final score-level aggregation protocol of RFEA and should, therefore, not be interpreted as a direct component ablation of the complete framework. In particular, DE and Base correspond to the same dual-expert X2-DFD configuration, whereas BC and DC denote the separately retained raw expert responses; the identical rounded DE and Base metrics, therefore, do not imply that Base is a numerical combination of BC and DC.

As shown in Table 5, BO substantially outperforms NE and DO across the evaluated metrics, indicating that the blending-related cue provides considerably stronger auxiliary discriminative information under this diagnostic protocol. DO provides little improvement over NE, while DE does not outperform BO. These results further support the observation that introducing additional expert cues does not necessarily improve detection when their discriminative reliability differs. We further examine whether the weak DC performance can be explained simply by a globally inverted score direction. Replacing DC with $1-\mathrm{DC}$ increases the five-dataset average video-level AUC from 44.09% to 53.25%; however, the inverted response remains close to chance on Celeb-DF, FaceForensics++, DFDCP, and DFDC. Thus, the cross-domain instability of DC cannot be explained by a single global polarity reversal.

Table 5. Diagnostic analysis of expert-cue sources on Celeb-DF (%). Upward arrows indicate higher-is-better metrics, whereas downward arrows indicate lower-is-better metrics.

Setting	AUC ↑	AP ↑	EER ↓	Best F1 ↑
NE	55.00	71.59	47.40	79.39
BO	86.67	92.59	22.04	85.23
DO	54.34	68.44	46.72	79.42
DE	86.08	92.51	22.20	84.17

Temporal aggregation sensitivity. We further examine the sensitivity of temporal evidence concentration to different aggregation strategies. We compare full-frame averaging (Mean), max aggregation (Max), top-5% (T5), top-10% (T10), and top-20% (T20) high-evidence-frame aggregation, as well as median aggregation over the top-10% frames (Med-T10).

Table 6 reveals three consistent observations. First, T5, T10, and T20 all outperform Mean, supporting the assumption that informative forgery evidence is temporally non-uniform and can be diluted by full-frame averaging. Second, the three top-ratio strategies also outperform Max, indicating that aggregating a set of high-evidence frames is more robust than relying on a single response peak. Third, the performance differences among T5, T10, and T20 are small, suggesting limited sensitivity to the exact aggregation ratio. These results indicate limited sensitivity to the exact fixed ratio around the default T10 operating point used in the main experiments.

Table 6. Sensitivity analysis of temporal evidence concentration (%). Upward arrows indicate metrics for which higher values are preferred, whereas downward arrows indicate metrics for which lower values are preferred. Boldface indicates the best result for each metric.

Aggregation	AUC ↑	AP ↑	EER ↓	Best F1 ↑
Mean	91.92	94.50	15.47	88.93
Max	92.24	94.87	15.06	89.09
T5	92.49	95.03	14.67	89.30
T10	92.68	95.16	14.03	89.52
T20	92.64	95.11	14.29	89.42
Med-T10	92.52	95.00	14.22	89.13

Paired bootstrap analysis. To assess the robustness of the observed video-level improvements to sample variation, we conduct paired bootstrap analysis using videos as the resampling unit. We perform 1000 bootstrap resamples and report the performance gain of RFEA-T10 over each comparator, together with the corresponding 95% confidence interval (CI) and bootstrap win rate. Because AUC and AP are the primary ranking-based metrics, the analysis focuses on these two measures. For Table 7, paired bootstrap statistics are first computed independently within each benchmark, and the reported five-dataset entries summarize the corresponding benchmark-level quantities by macro-averaging.

Table 7. Macro-averaged benchmark-wise paired bootstrap summaries of video-level improvements.

Comparison	Metric	Gain	CI Lower	CI Upper	Win (%)	Resamples
RFEA-T10 vs. Base-Max	AUC	+1.30	+0.56	+2.16	84.74	1000
RFEA-T10 vs. Base-Max	AP	+1.86	+1.03	+2.87	79.62	1000
RFEA-T10 vs. Base-T10	AUC	+0.63	+0.07	+1.20	62.44	1000
RFEA-T10 vs. Base-T10	AP	+1.16	+0.62	+1.67	63.46	1000
RFEA-T10 vs. Naive-All-T10	AUC	+7.10	+4.39	+9.99	100.00	1000
RFEA-T10 vs. Naive-All-T10	AP	+6.34	+3.38	+9.85	100.00	1000
RFEA-T10 vs. RSF-T10	AUC	+0.49	+0.09	+0.98	58.48	1000
RFEA-T10 vs. RSF-T10	AP	+0.31	−0.02	+0.71	56.68	1000
RFEA-T10 vs. BC-T10	AUC	+0.79	+0.24	+1.41	79.72	1000
RFEA-T10 vs. BC-T10	AP	+0.46	+0.06	+0.96	69.04	1000

Table 7 shows that the macro-averaged lower and upper bounds of the benchmark-wise 95% confidence intervals are positive for the AUC and AP gains of RFEA-T10 over Base-Max, Base-T10, and Naive-All-T10. The particularly large positive intervals relative to Naive-All-T10 further support the degradation observed when unstable evidence is incorporated indiscriminately. Relative to RSF-T10, the AUC gain remains positive with a 95% CI of [0.09, 0.98], whereas the AP interval of [−0.02, 0.71] slightly crosses zero. Thus, after the dominant unstable source has been removed, relative evidence alignment provides a smaller additional improvement, while the AP gain should be interpreted more cautiously. Overall, the bootstrap results support the stability of the principal video-level improvements without implying uniform gains for every metric and comparison.

To complement the dataset-wise bootstrap summaries in Table 7, we additionally perform a replicate-level macro paired bootstrap in which the five benchmark-level metric differences are macro-averaged within each bootstrap replicate. Under this aggregation, RFEA-T10 versus Base-T10 retains the same AUC and AP point gains of 0.63 and 1.16 percentage points, with 95% confidence intervals of [0.36, 0.89] and [0.86, 1.41], respectively; both corresponding bootstrap win rates are 100%. This complementary analysis leads to the same aggregate conclusion while deriving the confidence interval and win rate from one common macro-bootstrap distribution.

Additional controlled score-level comparisons. Following the reviewer-requested comparison settings, we additionally evaluate conventional non-parametric fusion rules and alternative response normalization strategies using the same retained evidence sources and top-10% temporal aggregation protocol. Table 8 reports the five-dataset average video-level results.

Table 8. Additional controlled score-level comparisons using the five-dataset average video-level performance (%). Upward arrows indicate higher-is-better metrics, whereas downward arrows indicate lower-is-better metrics. Boldface identifies the RFEA setting and the best average results.

Setting	AUC ↑	AP ↑	EER ↓	Best F1 ↑
Concas-Average (Raw Mean)	92.19	94.85	14.71	89.13
Concas-Bayesian	92.55	94.96	14.17	89.47
Concas-Product	92.06	94.80	14.63	88.93
Concas-Maximum	92.10	94.12	14.62	89.21
Concas-Minimum	91.91	94.72	14.72	88.78
Min–max + Mean	92.19	94.85	14.71	89.13
Z-score + Mean	92.22	94.87	14.60	89.25
RFEA	92.68	95.16	14.03	89.52

Among the conventional non-parametric fusion rules, Bayesian fusion is the strongest competitor and reaches 92.55% AUC and 94.96% AP. Alternative min–max and z-score normalization remain close to raw averaging, whereas the percentile-rank alignment used by RFEA reaches 92.68% AUC and 95.16% AP. These controlled comparisons provide additional evidence that the observed average improvement is not specific to a single naive averaging baseline.

Additional cross-architecture and stronger-source analysis. To examine whether the aggregation behavior is specific to X2-DFD, we additionally use an independently pretrained Xception detector on Celeb-DF, DFDCP, and DFDC while keeping the evaluated frames fixed. The results are reported in Table 9.

Table 9. Additional video-level analysis with an independently pretrained Xception detector (%). All settings use top-10% temporal aggregation.

	Celeb-DF		DFDCP		DFDC		Average
Setting	AUC	AP	AUC	AP	AUC	AP	AUC	AP
Xception	78.74	87.29	74.54	82.79	72.19	74.91	75.16	81.66
Xception + BC-Raw	94.73	97.28	87.47	93.26	83.66	86.33	88.62	92.29
Xception + BC-Rank	94.27	96.94	87.17	92.78	83.29	85.34	88.25	91.69
X2-DFD	94.18	96.78	90.21	94.30	84.15	82.77	89.51	91.29
X2-DFD + Xception-Raw	91.81	95.60	85.51	91.32	79.76	81.72	85.69	89.54
X2-DFD + Xception-Rank	93.62	96.54	89.09	93.83	82.68	83.84	88.46	91.41

With Xception as the alternative upstream response, the three-dataset average AUC/AP increases from 75.16%/81.66% for Xception alone to 88.25%/91.69% for the rank-aligned Xception+BC setting. The corresponding raw fusion reaches 88.62%/92.29%, indicating that the framework can be instantiated with a substantially different upstream detector, while the benefit of relative alignment itself can depend on the particular source combination.

Xception also provides a stronger additional-source test than DC: despite achieving a standalone average AUC of 75.16%, directly averaging Xception with X2-DFD reduces AUC from 89.51% to 85.69%. Rank alignment recovers the same-source result to 88.46% AUC and 91.41% AP, improving over raw fusion by 2.77 and 1.86 percentage points, respectively, while not exceeding the stronger X2-DFD response alone.

Computational overhead. We further report the computational overhead associated with the RFEA evaluation over 182,233 frame responses from 6392 videos. As summarized in Table 10, RFEA requires no additional training, model-inference pass, or learnable parameters. The recorded top-k video aggregation time is 0.1780 s, while the rank-normalized frame-fusion stage is reported as 0.0000 s at the timing precision used by the evaluation script. The total recorded evaluation time is 1.2208 s, of which 1.0429 s is spent on metric computation. Since metric computation is evaluation bookkeeping rather than part of the RFEA decision rule, it is reported separately and should not be interpreted as deployment-time processing overhead. Component runtimes are independently rounded to four decimal places and, therefore, may not sum exactly to the reported total.

Table 10. Recorded runtime breakdown for RFEA evaluation on the five-dataset evaluation set.

Component	Training	Extra Model Inference	Extra Parameters	Runtime (s)
Rank-normalized frame fusion	No	No	0	0.0000
Top-k video aggregation	No	No	0	0.1780
Metric computation	No	No	0	1.0429
Total recorded evaluation	No	No	0	1.2208

4.4. Limitations

RFEA is a score-level evidence aggregation framework rather than a retrained deepfake detector. It can filter, align, and aggregate forensic responses that are already available, but it cannot generate new visual forensic representations. Consequently, when all retained evidence sources provide weak or incorrect discriminative responses, RFEA cannot recover forensic information that is absent from the upstream detector outputs.

As illustrated in Figure 4, some real videos still receive high forgery scores after RFEA and are consequently misclassified. These cases demonstrate that the effectiveness of score-level aggregation remains bounded by the quality of the input forensic responses. When the retained sources fail simultaneously, response aggregation cannot substitute for improved representation learning in the upstream detector.

Figure 4. Representative failure cases of RFEA. Green shading indicates correct predictions, whereas red shading indicates incorrect predictions.

RFEA also adopts a fixed retained-source configuration rather than dynamically estimating the most appropriate source set for each sample. This design avoids sample-wise source adaptation during evaluation, but it limits the ability of the aggregation framework to accommodate variations in manipulation mechanisms, compression conditions, and generation methods. Future work may investigate adaptive source-reliability estimation using an independent calibration set or unlabeled confidence information. The current relative-alignment implementation constructs its reference distribution from the unlabeled evaluation responses of each benchmark and is, therefore, label-free but transductive rather than strictly online inductive. Temporal evidence concentration also uses a fixed global aggregation ratio; although the sensitivity analysis shows limited variation around this setting, sample-adaptive temporal selection may provide greater flexibility when the amount of informative evidence varies substantially across videos.

Finally, the diffusion-artifact response used in the current setting does not preserve a stable positive discriminative relationship across the evaluated datasets and is, therefore, excluded from the retained evidence set used by RFEA. If more robust diffusion-related forensic responses become available, or if their response direction and cross-domain calibration can be improved, the RFEA formulation can naturally accommodate them as additional evidence sources.

5. Conclusions

This work investigated reliable video-level aggregation of heterogeneous frame-level forensic responses for deepfake detection across multiple datasets. Different evidence sources can exhibit unequal discriminative reliability and numerical scales, while informative forgery evidence is often temporally non-uniform. To address these issues, we proposed Reliability-Filtered Evidence Aggregation (RFEA), which combines source-level reliability filtering, relative evidence alignment, and temporal evidence concentration without retraining or modifying the upstream detector.

Experiments on five public datasets show that indiscriminately incorporating all available evidence sources can substantially degrade both frame-level and video-level detection, whereas restricting fusion to a retained source set recovers much of the discriminative performance. Relative evidence alignment provides further improvement over the same retained source set, while aggregating a proportion of high-evidence frames consistently outperforms both full-frame averaging and single-frame max aggregation in the temporal sensitivity analysis. RFEA-T10 achieves average video-level AUC and AP of 92.68% and 95.16%, respectively. Relative to Base-T10, the improvements are 0.63 and 1.16 percentage points. A complementary replicate-level macro paired-bootstrap analysis yields 95% confidence intervals of [0.36, 0.89] and [0.86, 1.41] for the corresponding AUC and AP gains, respectively.

These results indicate that effective multi-source forensic aggregation depends not only on the number of available evidence sources, but also on the reliability of the participating sources, their cross-source comparability, and the temporal concentration of informative responses. RFEA provides a score-level post-processing solution to these issues, but its performance remains bounded by the discriminative quality of the upstream responses and by the current fixed retained-source configuration. Future work will investigate adaptive source-reliability estimation and dynamic evidence selection without relying on test labels.