Protecting Digital Identities: Deepfake Face Detection Using Dual-Decoder U-Net Semantic Segmentation

Abstract

Deepfake content forgery compromises the integrity of digital media and the protection of personal identity, making its detection essential for preserving trust and enabling effective forensic analysis. Most deepfake detection approaches focus on global classification with a binary decision, which is inadequate for precise localization of manipulated regions. This limitation becomes particularly evident under image processing distortions. This paper proposes a dual-decoder architecture for the detection and segmentation of original and deepfake facial manipulations. Unlike conventional single-decoder segmentation models, the proposed approach introduces two decoding branches that learn complementary feature representations of authentic and forgery facial textures. In addition, attention mechanism modules are incorporated to refine encoder features based on decoder context, introducing adaptive feature selection during reconstruction. This architectural design reduces feature interference during reconstruction and enhances the localization of subtle inconsistencies introduced by deepfake manipulations. This approach generates complementary masks for real and forged regions, providing more precise boundary delineation. Experimental results highlight the robustness of the proposed method under image processing distortions, achieving intersection over union (IoU) scores of 0.9387 for real faces and 0.9254 for deepfake segmentation. These results underscore the effectiveness of the dual-decoder architecture in accurately detecting and localizing deepfake facial manipulations.

1. Introduction

The distribution of manipulated digital content via social media platforms, particularly deepfakes, generates challenges to personal identity and media integrity. Deepfakes modify images or videos by adding, replacing, or removing people or objects and manipulating facial regions to compromise the authenticity of the content. These manipulations contribute to disinformation, fraud, and malicious image forgery. For this reason, it is necessary to generate an accurate method for detection and precise localization, which are essential for verifying authenticity and enabling interpretable forensic analysis [1,2,3,4,5,6].

The detection of manipulated facial content is essential for verifying authenticity, preventing the dissemination of forged images, and protecting individual identity by accurately identifying authentic and altered regions. In this context, faces are the primary targets of deepfake manipulation and subtle manipulations are difficult to detect, particularly combined with common image processing distortions such as compression, scaling, cropping, noise, and blurring. In this context, deep learning models can identify features between real and manipulated faces, achieving a robust performance across different manipulation techniques; meanwhile, global binary classification methods lack the spatial resolution to localize manipulations, limiting interpretability and forensic reliability [7,8].

The segmentation of deepfake and authentic faces in digital images provides valuable information for the detection of image forgery. In this context, neural networks have demonstrated the ability to accurately recognize manipulated facial regions, even in the presence of common image distortions such as compression, scaling, noise, or blurring. Recent studies have explored diverse neural network architectures to improve the segmentation of deepfake faces; however, most approaches focus primarily on the localization of forged regions. Several relevant methods have been proposed in the literature. Chang et al. [9] introduced NA-VGG, a network based on the VGG16 model combined with Spatial Rich Model (SRM) features to extract residual noise, enabling the detection of subtle forgeries. Yu et al. [10] employed the SegNet architecture as a separable convolutional neural network (CNN) with ensemble learning for image segmentation. Wang et al. [11] proposed a facial semantic segmentation approach combined with binary classification using ASPP with Xception modules and a DeepLabV3-based decoder, improving segmentation accuracy through contextual information. Nirkin et al. [12] employed a dual-network approach combining facial segmentation and context recognition, enhancing detection performance. Abdulla [13] used a DenseNet121 for image segmentation, focusing on local feature extraction. Wang et al. [14] introduced a two-stage method using residual spatial maps and a weighted fusion strategy with spatial attention. Sekar et al. [15] combined feature extraction using Gabor filters, ResNet-50, and attention mechanisms. Jagam et al. [16] focused on fast training from a CNN with eight layers, which was developed to detect deepfakes. Kurniawan [17] focused on real-time deepfake detection using a YOLOv11. Additionally, Alsolai et al. [18] combined Vision Transformers and temporal modeling with long short-term memory (LSTM) and attention mechanisms to improve detection in video sequences. Gupta et al. [19] proposed a framework combining attention-enhanced convolutional networks with quantum components to improve feature representation and model efficiency. Their approach highlights the importance of focusing on the manipulation of facial regions and improving generalization across datasets.

Most deepfake detection research methodologies realize a binary classification, determining whether an image or video is real or fake. To address these limitations, we propose a dual-decoder neural network architecture that separates feature representations of authentic and manipulated facial regions within a unified architecture. One decoder emphasizes the localization of forged regions at the pixel level, while the other focuses on segmenting authentic facial areas. The features from both classes are processed in both branches, and the network reduces interference between real and manipulated features. In addition, attention modules are incorporated to selectively refine encoder features based on decoder context to enhance boundary precision and detection of subtle deepfake artifacts. This approach generates complementary masks for real and manipulated regions, providing a structured representation to improve forensic analysis and automated verification tasks that also maintains robustness under challenging image conditions. The main contributions consist of the following:

• The dual-decoder architecture enhances the localization of manipulated regions by separating specific features, increasing the robustness against common image distortions, including compression, scaling, cropping, noise, and blurring.
• The proposed architecture generates complementary masks, improving interpretability and providing pixel-level localization of both authentic and forged facial regions.
• The dual-decoder architecture employs attention mechanisms to improve boundary precision by detecting subtle manipulations that conventional single-decoder segmentation models and global classifiers often fail to capture.
• The unified segmentation framework enables multitask learning while maintaining high accuracy for both authentic and manipulated regions.
• The dual-decoder outputs provide structured masks that support subsequent forensic applications, including automated verification, forgery localization, and confidence estimation.

The next section reviews related work to provide context for the proposed methodology and highlight existing approaches for detecting and segmenting authentic and deepfake faces.

2. Materials and Methods

2.1. Materials

The OpenForensics dataset [20] was employed for training and validation, comprising 41,194 images, with 20% allocated for validation. The OpenForensics dataset includes labels for authentic and manipulated facial regions. The dataset consists of two subsets: a Train set containing undistorted images and a Test Challenge set including images affected by various digital distortions, such as noise addition, filtering, brightness variations, grayscale conversion, and cropping. Additionally, the proposed neural network was evaluated using 6028 undistorted images from the Test set and 6670 distorted images from the Test Challenge set.

The algorithm was implemented in Python 3.11.5 using the PyTorch 2.1.0+cu118 framework and executed on an NVIDIA GTX 4050 GPU and an Intel Core i7-155H processor (4.8 GHz) running Windows 11. This configuration ensured efficient processing during model training and evaluation.

2.2. Proposed Dual-Decoder for Authentic and Manipulated Deepfake Face Segmentation

The proposed neural network learns discriminative visual patterns to accurately detect and segment forged regions, enabling explicit localization of regions of interest in both authentic and manipulated faces. This enhances image analysis by providing spatial information, in contrast to most of the conventional approaches that are limited to image-level classification.

The designed architecture enhances the robustness of the segmentation against common image processing distortions. The overall architecture of the proposed method is illustrated in Figure 1.

Figure 1. Dual-decoder neural network.

The proposed architecture is designed with a dual-decoder structure for the segmentation of authentic and manipulated facial regions [21]. Each branch learns complementary feature representations to capture consistent facial patterns and subtle inconsistencies introduced by manipulation.

One branch focuses on modeling the structures and textures of authentic faces, such as smooth boundaries and natural appearance. In parallel, the second branch detects anomalous patterns, like inconsistencies introduced during face manipulation processes. This complementary learning strategy improves the segmentation performance. The learning of each branch is based on pixel annotations provided by the OpenForensics dataset, which include labels for authentic and manipulated facial regions.

This strategy enhances the segmentation performance and robustness in challenging scenarios such as partial manipulations, boundary overlaps, and ambiguous regions near hair, accessories, and occlusions.

Furthermore, residual blocks are incorporated into the encoder to facilitate feature propagation and stabilize training, while attention mechanisms are introduced in the decoder to refine feature selection during the upsampling stages. These mechanisms emphasize relevant spatial regions, enhancing segmentation accuracy. The detailed configuration of the neural network is presented in Table 1.

Table 1. Configuration of the layers from the proposed U-Net with dual-decoder.

Stage	Layer	Neurons	Description
Encoder	Residual Block 64	64	Extracts low-level spatial features
	Residual Block 128	128	Learn image textures
	Residual Block 256	256	Identify facial features
	Residual Block 512	512	Detects inconsistencies in facial coherence across the image
	Residual Block 1024	1024	Generate feature maps with global and semantic features
	MaxPooling	---	Reduce tensor dimensions
Decoder branch 1	5 Up Conv Blocks	1024 → 64	Upsample with the attention mechanism and skip connections (increase the predicted mask resolution)
	CNN 3	3	Generate the predicted mask from deepfake segmentation
	Activation	Sigmoid	Pixel probability map
Decoder branch 2	5 Up Conv Blocks	1024 → 64	Upsample with the attention mechanism and skip connections (increase the predicted mask resolution)
	CNN 3	3	Generate the predicted mask from the real face segmentation
	Activation	Sigmoid	Pixel probability map

The neural network is trained using pairs of input images $\mathcal{X}=\{X_n{\}}_{n=1}^N$ and their corresponding masks $\mathcal{Y}=\{M_n^r,M_n^f{\}}_{n=1}^N$, where $M_n^r\in {\mathbb{R}}^{H\times W}$ and $M_n^f\in {\mathbb{R}}^{H\times W}$ contain the authentic and manipulated regions for the neural network learning. Each pixel location $(i,j)$ encodes the belonging class. Given an input image $X\in {\mathbb{R}}^{H\times W\times 3}$, the proposed model learns the following mapping (1):

$$F:{ℝ}^{H\times W\times 3}\to {{ℝ}_R}^{H\times W},{{ℝ}_F}^{H\times W}$$

(1)

where the neural network simultaneously predicts two segmentation masks, which are presented as follows (2):

$$({\stackrel{⌢}{M}}_R,{\stackrel{⌢}{M}}_F)=F(X)$$

(2)

where ${\widehat{M}}_R$ represents the predicted mask for authentic regions, while ${\widehat{M}}_F$ corresponds to the predicted mask for manipulated regions. This complementary representation models facial consistency and the other emphasizes the detection of anomalous patterns.

The encoder employs residual blocks (Resblock, Figure 2), which facilitate the learning of hierarchical features by mitigating the vanishing gradient problem and improving feature propagation [22,23]. The residual blocks consist of a shortcut connection with a convolution kernel of 1 × 1 to match the input and the output feature map size. In addition, two CNNs with a kernel of 3 × 3 are employed, and each block uses a stride of 1 and padding of 1 to preserve the feature maps. A batch normalization is applied to stabilize the training of the neural network, followed by a ReLU activation function.

Figure 2. Residual block diagram.

The operation of a residual block $y_r$ can be expressed as follows (3):

$$y_r=X_{fm}+F(X_{fm},W)$$

(3)

where $X_{fm}$ represents the input feature maps, and $F(X_{fm},W)$ denotes the residual feature maps learned by the convolutional layers, with $W$ corresponding to the linear projection induced by the $1\times 1$ convolution in the shortcut block.

Therefore, each decoder begins with a transposed convolutional layer (TCNN) to increase resolution of the feature maps. The transposed convolution produces an upsample feature map. This operation can be interpreted as an upsampling process that refines the identification of face feature regions. The transposed convolution can be formally expressed as follows (4):

$$X_{TCNN}=W_t\odot X_{fm}$$

(4)

where $W_t$ denotes the kernel associated with the transposed convolution, and $\odot$ represents the transposed convolution operation. This operation is employed to reconstruct spatial details from feature maps. The transposed convolution at the beginning of each decoding stage ensures an appropriate spatial scale. Subsequently, the UpConv blocks are incorporated, and they employ attention mechanisms to refine the feature information for the predicted mask (Figure 3).

Figure 3. (a) UpConv block, (b) attention mechanism.

The UpConv operation can be expressed as follows (5):

$$y_{up}=Concat(X_e,X_{at})$$

(5)

where X_e represents the input feature maps from the encoder skip connection upsample layer, X_d is the output of the TCNN and X_at is the output of the attention mechanism.

The attention mechanism is relevant for deepfake detection, where the forgeries are subtle. Conventional skip connections propagate encoder features without discrimination, introducing noise. In contrast, the attention module enables adaptive feature selection, prioritizing boundary detection for the localization of image manipulations that are difficult to identify using standard encoder–decoder architectures. First, both feature maps $X_E$ and $X_D$ are transform by δ, which consists of 1 × 1 convolutions, to project them into a common feature space. This can be expressed as follows (6):

$$z=\mathrm{ReLu}(\delta (X_E)+ \delta (X_D\left)\right)$$

(6)

Subsequently, a scalar attention coefficient map is generated by the following sigmoid activation (7):

$$\alpha =sigmoid(\mathsf{\Psi }(z))$$

(7)

where $\mathsf{\psi }$ denotes the layers operations. Finally, the attention map modulates the encoder features through the following (8):

$$X_{at}=\alpha \times X_E$$

(8)

The resulting feature map $X_{at}$ suppresses features before being fused with the decoder. The dual-decoder realizes a feature disentanglement. Each decoder generates predicted masks using encoder representations (9) and (10), which are as follows:

$${\stackrel{⌢}{M}}_R=D_R(c_{1,}{c}_{2,}{c}_{3,}{c}_{4,}{c}_{5,})$$

(9)

$${\stackrel{⌢}{M}}_F=D_F(c_{1,}{c}_{2,}{c}_{3,}{c}_{4,}{c}_{5,})$$

(10)

This design focuses on each decoder branch learning complementary representations. Specifically, one branch focuses on modeling natural texture and structural consistency associated with authentic faces, while the other emphasizes image inconsistencies and localized distortions from manipulated regions. Through the separation of these objectives, the proposed architecture reduces feature interference between classes, enhancing the localization of facial manipulations that are difficult to capture using conventional single-decoder architectures.

In this context single-decoder architectures have demonstrated strong performance in image segmentation tasks, particularly in identifying regions of interest in manipulated images [24,25]. However, conventional designs may present some limitations for segmenting classes with similar visual patterns, such as authentic and deepfake faces. To overcome this limitation, the architecture with a dual-decoder is employed to learn independent features and avoid interference between classes [26,27,28,29], improving the precision of authentic and deepfake face segmentation.

To optimize network training, a set of hyperparameters and training strategies were used, as defined in Table 2.

Table 2. Training hyperparameters.

Epochs	Learning Rate	Batch Size	Optimizer
300	0.001	32	ADAM

The proposed loss function is designed for the segmentation of subtle, original facial and deepfake manipulation classes, where accuracy and boundary precision are critical. The employed composite loss combines the Binary Cross-Entropy (BCE), Dice, and intersection over union (IoU) losses with a boundary consistency term to optimize pixel classification, region overlap, and boundary localization.

The BCE loss ensures stable pixel classification between manipulated and authentic regions, which can be depicted as follows (11):

$$L_{BCE}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N[y_i\log ({\widehat{y}}_i)+(1+y_i)\log (1-{\widehat{y}}_i)]}+\varsigma digit$$

(11)

where $y_i$ is the ground truth label, ${\widehat{y}}_i$ is the predicted mask, and N represents the total pixels in the image. The term Çdigit represents the scheduling of the learning rate and the regularization coefficient. Both parameters are periodically adjusted every 30 epochs to improve training stability and feature discrimination.

Specifically, the learning rate is updated to refine the optimization process, while the regularization coefficient is progressively reduced as an optimization learning strategy. The model initially learns robust feature representations under stronger regularization and later focuses on finer adjustments.

The update interval of 30 epochs was selected to balance training stability and adaptability, providing sufficient time for the model to learn the most important patterns before applying further adjustments. The learning rate decayed by a factor of 0.005 every 30 epochs. This strategy stabilizes gradient updates and improves the discrimination between manipulated and authentic regions at the local level.

The Dice Loss ($L_{Dice}$) evaluates the overlap between the predicted mask and the ground truth mask. It is defined as follows (12):

$$L_{Dice}=1-{\displaystyle \frac{2{\displaystyle \sum _{i=1}^N{\widehat{y}}_i{y}_i}}{{\displaystyle \sum _{i=1}^N{\widehat{y}}_i}+{\displaystyle \sum _{i=1}^N{y}_i}}}$$

(12)

This term effectively addresses class imbalance and improves the segmentation of subtle manipulated regions. The IoU Loss ($L_{IoU}$) measures the ratio between the intersection (∩) and the union (∪) of the predicted and ground truth masks. It is depicted as follows (13):

$$L_{IoU}=1-{\displaystyle \frac{{\widehat{y}}_i\cap y_i}{{\widehat{y}}_i\cup y}}=1-{\displaystyle \frac{{\displaystyle \sum _{i=1}^N{\widehat{y}}_i{y}_i}}{{\displaystyle \sum _{i=1}^N{\widehat{y}}_i+{\displaystyle \sum _{i=1}^N{y}_i-{\displaystyle \sum _{i=1}^N{\widehat{y}}_i{y}_i}}}}}$$

(13)

where $L_{IoU}$ directly penalizes the proportion of overlap relative to the total union, making it a widely used metric for object segmentation. To enhance boundary localization, the $L_{boundary}$ is introduce, which is a consistency term based on spatial gradients and can be depicted as follows (14):

$$L_{boundary}=\nabla {\stackrel{⌢}{M}}_R\times \nabla {\stackrel{⌢}{M}}_F$$

(14)

This term enhances boundary delineation between authentic and manipulated regions by incorporating edge information derived from image gradients into the learning process. Since image gradients capture intensity variations corresponding to structural transitions, they enable more accurate extraction of edge information for precise segmentation of both classes, especially near region boundaries. As a result, it improves convergence behavior and strengthens structural transitions in the localization of deepfake forgeries. The boundary loss is incorporated as an additional component in the overall loss function.

The total loss is expressed as (15), where the loss function includes the boundary consistency term. This loss function reduces feature interference and improves the localization of subtle deepfake manipulations, and is depicted as follows:

$$L_{total}=L_{authentic}+L_{manipulated}+\chi L_{boundary}$$

(15)

where the loss for each branch is defined as (16), where each decoder is optimized independently.

$$L_i=\epsilon L_{BCE}+\delta L_{Dice}+\kappa L_{IoU} i\in \{authentic,manipulated\}$$

(16)

To ensure a stable convergence, all weights were set to 1 ($\epsilon =1, \delta =1, \kappa =1, \chi =1$) during the initial training (100 epochs). However, the fine-tunning stage improved the IoU metric, and so the weights were adjusted to $\epsilon$ $=0.6, \delta =1.5, \kappa =1.5, \chi =0.8$. The selection and impact of these values are further justified through the ablation study.

The proposed method combines a dual-decoder architecture with a composite loss function to achieve accurate and robust segmentation of authentic and manipulated facial regions. The loss function optimize pixel segmentation accuracy, region overlapping, and boundary consistency to captures subtle deepfake forgeries.

3. Results

The proposed dual-decoder architecture was evaluated through a series of experiments to validate its effectiveness for the segmentation of authentic and deepfake face manipulations. The experiments assessed the robustness under standard conditions and the robustness of the method against common image processing distortions, including JPEG compression, scaling, cropping, noise addition, and blurring. The results demonstrate that the proposed neural network identifies manipulated regions under challenging conditions.

3.1. Quantitative Performance Evaluation Under Image Distortions

The proposed segmentation method was evaluated using mean Dice score, mean intersection over union (IoU), accuracy, mean F1 score, and Area Under the ROC Curve (AUC), providing a comprehensive assessment of pixel-level classification, region overlap, and discrimination capability.

The Dice score measures the overlap between predicted and ground truth masks and is defined as follows (17):

$$Dice={\displaystyle \frac{2|Predicted\cap GroundTruth|}{\left|Predicted\right|+\left|GroundTruth\right|}}={\displaystyle \frac{2Tp}{2Tp+Fp+Fn}}$$

(17)

where Tp represents the true positives, which are the pixels correctly classified; the false positives Fp are pixels incorrectly classified; the true negatives Tn are the pixels that are not manipulated and are correctly identified as non-manipulated; and finally, the false negatives Fn are pixels that belong to a class but were missed by the prediction.

The IoU evaluates the ratio of the overlapping area between the predicted mask and the ground truth and indicates a segmentation accuracy (18).

$$IoU={\displaystyle \frac{|Predicted\cap GroundTruth|}{|Predicted\cup GroundTruth|}}={\displaystyle \frac{Tp}{Tp+Fp+Fn}}$$

(18)

Additionally, accuracy, F1-score, and AUC are computed. The F1-score reflects the balance between precision and recall. The accuracy evaluates the overall pixel classification, and AUC measures the discrimination between authentic and manipulated regions

Table 3 presents the robustness of the proposed dual-decoder neural network for the segmentation of authentic and deepfake facial regions under various image distortions. The F1-score is adopted as a key indicator of segmentation balance, as it captures the relation between precision and recall.

Table 3. Performance of the proposed dual-decoder neural network for authentic and deepfake face segmentation.

Image Distortion	Region of Interest	Dice	IoU	F1-Score	Accuracy	AUC
No distortion	Real Deepfake	0.9598 0.9533	0.9315 0.9201	0.9596 0.9542	0.9967 0.9973	0.9476 0.9435
JPEG Compression QF = 70	Real Deepfake	0.9597 0.9517	0.9317 0.9212	0.9599 0.9542	0.9967 0.9973	0.9475 0.9435
JPEG Compression QF = 30	Real Deepfake	0.9592 0.9501	0.9300 0.9195	0.9583 0.9510	0.9967 0.9630	0.9462 0.9422
Salt and pepper noise δ = 20%	Real Deepfake	0.9542 0.9449	0.9240 0.9108	0.9454 0.9456	0.9839 0.9866	0.9423 0.9355
Salt and pepper noise δ = 10%	Real Deepfake	0.9292 0.9135	0.9286 0.9133	0.9492 0.9537	0.9917 0.9928	0.9415 0.9395
Gaussian noise σ = 0.9	Real Deepfake	0.9185 0.9010	0.8793 0.8522	0.9195 0.9092	0.9954 0.9965	0.9005 0.8930
Gaussian noise σ = 0.3	Real Deepfake	0.9506 0.9400	0.9190 0.9044	0.9510 0.9409	0.9966 0.9972	0.9391 0.9300
Blurring kernel 5 × 5	Real Deepfake	0.9591 0.9529	0.9390 0.9107	0.9594 0.9538	0.9966 0.9973	0.9499 0.9443
Blurring kernel 7 × 7	Real Deepfake	0.9276 0.9409	0.9167 0.9277	0.9518 0.9418	0.9965 0.9972	0.9480 0.9360
Gamma correction γ = 1.5	Real Deepfake	0.9276 0.9405	0.9269 0.9175	0.9578 0.9414	0.9965 0.9972	0.9475 0.9358
Gamma correction γ = 0.7	Real Deepfake	0.9689 0.9624	0.9383 0.9198	0.9591 0.9633	0.9966 0.9973	0.9403 0.9459
Histogram equalization	Real Deepfake	0.9478 0.9382	0.9021 0.8904	0.9480 0.9392	0.9953 0.9964	0.9200 0.9384
Random distortion	Real Deepfake	0.9387 0.9254	0.9088 0.8928	0.9388 0.9259	0.9932 0.9944	0.9496 0.9364

To evaluate the robustness of the proposed method, controlled distortion experiments were conducted. Each distortion type was applied independently to the same set of test images to ensure a fair comparison across conditions. In the case of random distortion, the transformation was applied following the protocol defined in the Test Challenge image set from the OpenForensics dataset benchmark, where distortions are randomly assigned. This setting was preserved to ensure consistency with the original evaluation framework.

Table 3 demonstrates the robustness of the proposed method under common image processing distortions. The results demonstrate that the model segmentation performance across authentic and deepfake regions is consistent. The performance remains stable under JPEG compression with different quality factors QF (QF = 70 and QF = 30).

Under Gaussian noise ($\sigma =0.9$), the Dice score decreases. This behavior suggests that manipulated features are more sensitive to distortions that affect boundary consistency. Similarly, salt and pepper noise ($\delta =20\mathrm{\%}$) introduces noticeable degradation, although the model maintains a consistent performance.

In contrast, the method shows a high robustness against image blurring (kernel $5\times 5$) and gamma correction. Additionally, accuracy and AUC values remain consistent (above 0.98 and ~0.94, respectively), demonstrating the robustness of the model across different conditions.

These results indicate that the proposed architecture effectively learns robust and complementary feature representations for an accurate localization of manipulated regions, maintaining a consistent performance against image distortions. Additionally, Figure 4 shows the distribution of IoU scores for both real and deepfake face segmentation.

Figure 4. Distribution of the IoU scores for real face and deepfake segmentation.

Figure 4 illustrates the distribution of IoU values for the predicted segmentation masks. The results show low variance, particularly for deepfake regions, indicating stable and consistent segmentation performance. In contrast, slightly higher variability is observed for real face segmentation, with lower minimum IoU values (around 0.90), which may be attributed to the greater complexity and variability of authentic facial textures.

3.2. Qualitative Evaluation of Deepfake Segmentation

Table 4 presents a visual comparison between the ground truth masks and the predictions generated by the proposed architecture. The results depict that each decoder branch efficiently identifies patterns from the corresponding class, even under for subtle manipulations.

Table 4. Qualitive comparison.

Original image
Deepfake face ground truth
Deepfake face Prediction
Original face ground truth
Original face prediction

Table 4 presents the qualitative results from the segmentation performance of the proposed model under undistorted conditions. The comparison between the ground truth and the predicted masks shows a high similarity. Additionally, the results exhibit accurate boundaries detection, with minimal misclassification. Overall, Table 5 demonstrates that the proposed dual-decoder architecture achieves precise segmentation under standard conditions, consistent with the quantitative results.

Table 5. Cross-dataset validation of the proposed method.

Train Dataset	Test Dataset	Accuracy	AUC
OpenForensics	OpenForensics	99%	94%
OpenForensics	FaceForensics++	96%	81%

The model was trained on OpenForensics and directly evaluated on FaceForensics++ without fine-tuning. All results were obtained using images without additional distortions.

3.3. Cross-Dataset Validation

To evaluate the generalization capability of the proposed method, an additional experiment was conducted by validating the trained neural network on a different dataset without applying any fine-tuning. Specifically, the FaceForensics++ dataset was used to assess the performance of the trained model under cross-dataset conditions. All evaluations were performed using images without additional distortions.

As expected, Table 5 shows a decrease in performance when the model is evaluated on the FaceForensics++ dataset. This can be attributed to the fact that the model was trained on the OpenForensics dataset, which consists of image data with more evident forgeries and precise annotations for authentic and manipulated facial regions, facilitating spatial patterns learning. In contrast, FaceForensics++ includes more subtle manipulations and focuses on video facial modifications, such as expression changes and identity alterations. These differences in data distribution impact on the performance of the neural network generalization.

Despite this, the proposed method maintains competitive performance, suggesting that the learned latent representations capture features that do not depend on a specific dataset.

3.4. Comparative Analysis of Neural Network Architectures

The proposed method is compared against other neural network architectures by analyzing its segmentation accuracy. Figure 5 presents the accuracy comparison, while Figure 6 shows the corresponding inference time for each model.

Figure 5. Neural network performance comparison.

Figure 6. Inference processing time.

The results in Figure 5 indicate that the proposed dual-decoder neural network outperforms the evaluated architectures in terms of segmentation accuracy. This improvement can be attributed to the separation of authentic and deepfake regions, ensuring that it learns specialized patterns to discriminate feature representations. In contrast, conventional architectures focus on single decoding, which mixes feature representations and leads to less precise localization of manipulated regions.

In addition, the computational efficiency of the proposed architecture was evaluated. As shown in Figure 6, the proposed model achieves lower inference time compared to other neural networks. This computational efficiency is achieved through the encoder feature sharing design of the dual-decoder architecture, which enables effective feature reuse while avoiding unnecessary computational overhead.

3.5. Comparison with State-of-the-Art Methods

The proposed method is compared with other approaches reported in the literature, using the performance metrics provided by their respective authors.

The results in Table 6 show that the proposed method achieves high robustness and efficiency, outperforming or matching several existing approaches. These results highlight its effectiveness for the segmentation of precise, localized manipulated facial regions.

Table 6. Comparative analysis of the proposed and existing method.

Author	Methodology	Dataset	Performance
Wang et al. [11]	Facial semantic segmentation combined with a based neural network DeepLabV3 model.	Celeb-DF	Acc: 92.51%
Nirkin et al. [12]	The detection is based on two neural networks: (1) for facial segmentation and (2) for context recognition to enhance the binary classifier.	FaceForensics +and Celeb-DF-v2	AUC: 99.7 (Face Forensics) AUC: 66.0 (Celeb-DF-v2)
Sekar et al. [15]	Facial regions were detected with global features extraction using ResNet-50 with OLSTM.	FaceForensics, DFDC, Celeb-DF, Wild Deepfake	Acc: 99.78% (Face Forensics) 94.65% (Celeb-DF), AUC: 99 (Face Forensics++), 94 (Celeb-DF),
Kurniawan et al. [17]	YOLOv11 with C3K2 blocks and C2PSA modules for real and deepfake face detection.	EC2-Deepfake Dataset y Deepfake Dataset	Recall: 95%
Alsolai et al. [18]	Guardian-AI: feature extraction CNN with LSTM and multi-head attention.	CDDB (proprietary Deepfake Detection Benchmark).	Recall: 95.6% F1-score: 95.9%
Proposed Method	Real and Fake faces segmentation with a dual-branch U-Net model including residual blocks	Open RL FaceForensics	Accuracy: 99% (Open RL) AUC: 94 (Open RL) Recall: 98% (Open RL) Accuracy: 96% (FaceForensics) AUC: 81 (FaceForensics) Recall: 81% (FaceForensics)

Many existing approaches focus on binary deepfake detection without providing spatial localization or are limited to detecting deepfakes without identifying the manipulated regions. In contrast, the proposed method performs pixel segmentation of authentic and manipulated facial regions, explicitly separating the two classes. This provides a visual analysis of facial content and improves the identification of subtle deepfakes.

The model must simultaneously segment both authentic and manipulated regions, introducing more complex feature learning. If the approach were restricted to only deepfake segmentation, a higher performance could be achieved by focusing on features of manipulated areas. However, the proposed dual-decoder architecture is designed to provide a more complete image analysis.

However, a performance decrease is observed when the model is evaluated on the FaceForensics dataset, where accuracy and AUC decrease to 96% and 81%, respectively. This reduction can be attributed to the nature of the dataset, which consists of video sequences. The proposed method focus on images, and it does not correctly identify model temporal features on consecutive frames.

3.6. Ablation Study

To analyze the contribution of the proposed architecture, this section evaluates the impact of both the loss function components and the network structure. First, the effect of the composite loss function is assessed by removing individual terms, in order to analyze their impact on segmentation accuracy.

Table 7 demonstrates the effect of each component of the proposed loss function on deepfake segmentation. Removing the BCE term results in a significant performance decrease, highlighting its importance in stabilizing pixel-wise classification during segmentation. Excluding the IoU term reduces the overlap refinement between the predicted masks and the ground truth. In addition, removing the Dice component increases errors in imbalanced classes, negatively affecting the segmentation of small or subtle regions. Finally, eliminating the boundary term leads to errors in edge detection, since this component contributes to the generation of a precise localization of manipulated facial regions. Overall, this analysis demonstrates that each component improves the deepfake segmentation performance of the proposed model.

Table 7. Ablation study of loss function components.

Loss Function Elements				F1-Score		Dice		IoU
Dice	IoU	BCE	Boundary	Deepfake	Authentic	Deepfake	Authentic	Deepfake	Authentic
✔	✔	✔	✔	0.9542	0.9596	0.9201	0.9315	0.9533	0.9598
✗	✔	✔	✔	0.9153	0.9294	0.9130	0.9292	0.8695	0.8903
✔	✗	✔	✔	0.9030	0.9149	0.9420	0.9147	0.8572	0.8718
✔	✔	✗	✔	0.8434	0.8454	0.8434	0.8554	0.7802	0.7965
✔	✔	✔	✗	0.9172	0.9075	0.9075	0.8748	0.8596	0.8748

Additionally, the architectural design is analyzed by comparing the proposed model with different configurations: a standard U-Net architecture, a dual-decoder network, and the proposed dual-decoder architecture with attention modules. This comparison highlights the role of attention mechanisms in enhancing feature representation and improving the localization and accuracy of segmentation.

Table 8 compares different architectural configurations of the neural network. The U-Net model achieves the lowest performance for multitask segmentation. On the other hand, the dual-decoder architecture improves the results by separating feature learning for authentic and deepfake regions, allowing each branch to specialize and enhancing overall segmentation accuracy. Additionally, the proposed dual-decoder architecture with attention mechanisms achieves the best segmentation accuracy due to the attention mechanisms, which improve feature selection by focusing on the relevant patterns. This enables the detection of subtle patterns and enhances deepfake segmentation performance.

Table 8. Abalation study for different neural networks architectures.

Model	IoU		Dice		F1-Score
	Deepfake	Authentic	Deepfake	Authentic	Deepfake	Authentic
U-Net	0.8362	0.8639	0.8872	0.8972	0.8972	0.9172
Dual-decoder	0.9199	0.9113	0.9497	0.9432	0.9501	0.9342
Proposed dual-decoder with attention mechanisms	0.9313	0.9201	0.9501	0.9515	0.9542	0.9596

All models were evaluated under the same training and testing conditions.

Table 9 presents the ablation study conducted on the gain parameters $(ϵ,\delta ,\kappa ,\chi )$. The baseline configuration set the gains equal to one (1,1,1,1). Therefore, the fine-tuning adjusts the contribution of this component.

Table 9. Ablation study of loss function gains.

Gain	Deepfake IoU	Authentic IoU	Deepfake AUC	Authentic AUC
All equal $\epsilon =1, \delta =1, \kappa =1, \chi =1$	0.8876	0.8826	0.8873	0.9143
$\mathrm{B}\mathrm{a}\mathrm{l}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}$ $\epsilon =1, \delta =0.25, \kappa =0.25, \chi =0.25$	0.8629	0.8743	0.9025	0.9054
Skewed $\epsilon =0.3, \delta =1, \kappa =1, \chi =0.5$	0.8614	0.8800	0.8966	0.9076
$\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{d}$ $\epsilon$  $=0.6, \delta =1.5, \kappa =1.5, \chi =0.8$	0.9201	0.9435	0.9435	0.9476

The baseline configuration, where all gains are equally set to one, provides a balanced reference. However, this configuration does not make for an efficient boundary segmentation. The balanced configuration decreases IoU performance, particularly for deepfake regions, suggesting that these components affect spatial localization. Similarly, the skewed configuration indicates a deficient balance between region accuracy and boundary refinement.

In contrast, the proposed configuration $\left(ϵ=0.6, \delta =1.5, \kappa =1.5, \chi =0.8\right)$ achieves the best overall performance, with significant improvements in both deepfake and authentic IoUs, as well as AUC scores. These results demonstrate that increasing the contribution of terms related to structural consistency and boundary information generates an accurate segmentation of manipulated and authentic regions.

The ablation study confirms that the optimal performance is obtained when the gain parameters are fine-tuned from the initial equal setting, enabling a more effective balance between detection sensitivity and segmentation precision.

4. Discussion

Figure 5 shows that the proposed dual branch architecture achieves higher accuracy compared with other neural network architectures. This improvement can be attributed to the dual-decoder design, which learns specific patterns from authentic and deepfake facial regions.

In contrast, pretrained neural networks exhibit lower accuracy, because a subset of layers is fine-tuned, limiting feature learning. In this context, the proposed architecture is trained end-to-end, resulting in more stable and discriminative feature representations for accurate segmentation.

Table 3 demonstrates the robustness of the proposed architecture in identifying patterns across facial regions, effectively differentiating authentic and deepfake faces, even under image distortions. Each decoder branch is dedicated to modeling features associated with its corresponding class, improving the robustness of the segmentation process.

Additional experiments conducted on the FaceForensics dataset decrease the performance when processing video sequences, highlighting a limitation of the current architecture. This behavior suggests that the model, designed for image-based analysis, does not explicitly capture temporal dependencies across frames.

The ablation study demonstrates the contribution of the elements from the loss function on the neural network performance. The removal of this elements may generate a decrease in the segmentation accuracy. In addition, the architectural comparison shows that the incorporation of attention mechanisms enhances feature representation for deepfake segmentation.

These findings indicate that incorporating temporal modeling strategies, such as recurrent networks or temporal modules, could improve performance when using video-based datasets. Furthermore, task-specific architectures could be explored to optimize performance for scenarios, such as deepfake-only segmentation or video-based analysis. Training the model directly on FaceForensics++ data could also improve its generalization capability in such conditions.

5. Conclusions

This paper presented a dual-decoder neural network architecture for the segmentation of authentic and deepfake facial regions. The proposed method realizes a precise localization of manipulated areas by generating complementary masks for forged faces and original faces, improving interpretability compared to traditional classification approaches.

The results demonstrate the effectiveness of the proposed neural network architecture for the segmentation of authentic and deepfake facial regions. The model shows a high robustness against common image processing distortions introduced during deepfake generation.

The main contribution of this work lies in the integration of a dual-decoder structure with attention mechanisms, which enhances feature representation and reduces interference between classes during reconstruction. Additionally, the proposed method maintains stable performance under image processing distortions, including JPEG compression, Gaussian noise, salt and pepper noise, blurring, illumination variations, histogram equalization, and random perturbations. This robustness is attributed to the dual-decoder design, where each branch learns distinct features. Another advantage of the proposed architecture lies in its ability to independently segment authentic and manipulated regions, enabling detailed image analysis and precise localization of deepfake artifacts.

As future work, the incorporation of temporal modeling is proposed to extend the architecture to video sequences, improving the detection of manipulated regions across frames. Additionally, the integration of attention mechanisms in the temporal domain could further enhance the identification of subtle patterns in dynamic scenarios. Moreover, a demographic analysis will be conducted to investigate the impact of different factors and facial attributes on model performance, as these variations may influence manipulation patterns and detection behavior.