Audio Deepfake Detection Using Dual-Branch CNN with Shared Weights

Abstract

The detection of audio deepfakes has emerged as a significant problem in the field of voice biometrics systems, aiming to distinguish real human voices from those generated by Artificial Intelligence (AI). With synthetic voice becoming increasingly high-quality, it is more likely that such a voice will be abused for illicit purposes like identity theft and impersonation. The dual-branch CNN with shared weights architecture presented here is augmented with self-attention modules to detect audio deepfakes with greater efficiency. Convolutional operations and dual branches are used to extract complex characteristics from raw audio signals in our module in order to directly compare the unprocessed original audio with the modified audio. Afterward, residual connections improve network performance. Designed alongside these fundamental layers, self-attention modules are trained in a layered manner to detect multi-headed attention within audio frames. This feature helps the network distinguish between original and modified audio and improves feature extraction compared with the standard method. A range of audio modifications have been analyzed to assess the effectiveness of the method, and comprehensive testing across all possible audio manipulation situations has been conducted on the Controlled Singing Voice Deepfake Detection Challenge (CtrSVDD) dataset to assess its resilience. Both deep learning (DL) and machine learning (ML) models were outperformed by the proposed dual-branch CNN with shared weights. With an accuracy of 97.26%, precision of 99%, recall of 99.27%, and an F1 score of 98.88%, this model has achieved a remarkable performance.

1. Introduction

Various sectors have experienced significant advancements and challenges as deepfake technology has become widespread, including entertainment, social media, and finance. Taken together, these examples demonstrate how synthetic media—digitally created or manipulated audio, video, or images using Artificial Intelligence—not only manipulates public opinion but also amplifies the psychological and social repercussions of war. Media content has also increased globally due to the growth of social media. Initially, deepfakes were used for manipulated visual media, but they have been adapted to audio with advances in technology. Deepfakes were employed in these cases to undermine societal trust, misinformation, and alter the geopolitical histories [1].

Due to such maneuvers, the mental and social consequences of war are increased, and people worry about how synthetic media influences the views of people. There have been equally alarming consequences in the financial sector due to audio deepfakes. The attackers successfully exploited deepfake technology to mimic a company CEO’s voice and obtained approximately €220,000 from employees [2]. This incident highlights the vulnerability of voice-driven systems, particularly when voice authentication is essential. Moreover, these cases prove that conventional methods of security are not sufficient to safeguard against new threats. Audio deepfakes are increasingly posing a threat to the credibility of political figures, celebrities and organizations. It is not uncommon for audio recordings to be manipulated to fabricate controversial statements or to create false evidence in legal and reputational disputes [3]. For example, former President Barack Obama is depicted in a widely circulated deepfake making offensive remarks in a fabricated public service announcement [4,5]. As a result, digital communications are less trustworthy, and it is more challenging to verify the authenticity of information. It has become even more challenging with the availability of deepfake tools.

Modern audio deepfake programs do not need any technical knowledge to be used and hence they can be used widely. Voice-cloning services offer a democratic platform to create a misleading media by producing believable synthetic voices with only several seconds of input information [6]. As a result of this connectivity, there has been an increase in phishing scams, fake interviews, and fraudulent customer service inquiries. In both an ethical and societal context, these incidents have profound implications. As well as compromising individual and organizational security, audio deepfakes threaten societal cohesion and democratic trust [7].

A strong, multidisciplinary solution is required, which engages researchers, policymakers and technology developers. Several deep learning advances have made it possible to develop robust models for diverse classification tasks, such as disease diagnosis and tweet detection [8]. CNN-based models that are optimized with the gray wolf optimizer (GWO), for example, have been shown to classify complex visual patterns, such as monkeypox skin lesions, with high accuracy. Transformer-based architectures, as demonstrated in [9], can process different data modalities, e.g., toxic Arabic tweets, with remarkable accuracy. Specifically, we use these methodologies to derive domain-specific features and optimize our models. In addition, innovative frameworks integrating machine learning with cloud and blockchain technologies have been proposed for efficient, secure classification, such as in cancer diagnostics [10]. Medical imaging and text analysis have been the focus of such techniques, but audio processing is yet to be explored.

By combining self-attention modules with dual-branch CNN with shared weights architectures, audio deepfake detection is addressed. A traditional detection approach encompasses both machine learning and deep learning [11]. There are still a number of challenges facing existing methods, such as the inability to generalize across different audio domains and handle a wide range of synthetic manipulations. Common deep learning approaches are the Long Short-Term Memory (LSTM) network and CNN [12]. Although these techniques work, they usually have several drawbacks, such as time-consuming preprocessing, complicated feature engineering, and generalization difficulties. The goal of this study is to develop an audio deepfake detection model that reduces the need for preprocessing and achieves higher accuracy while still remaining scalable.

By using stronger features extracted from the input speech and reducing the model size, the detection mechanism is enhanced. Our objective is to develop a model that is able to detect manipulated audio signals efficiently by focusing on minimal preprocessing. As an added benefit, the model’s architecture will capture subtle changes in audio data that may indicate tampering, further improving detection accuracy. This paper makes a huge step forward in the field of audio deepfakes. The focus on feature extraction and minimizing preprocessing makes our model better suited to handling a wide range of deepfake scenarios, ensuring both robustness and efficiency.

2. Related Work

In this section, we will introduce you to all the latest techniques for detecting audio deepfakes, including imitations and synthetics. This needs a plethora of studies, which are based on the selection of datasets, the selection of models, the accuracy that is reported, and the general contributions to the field, and not just simply dividing the existing approaches into ML and DL.

Pal et al. (2018) [13] provided proof of the effectiveness and generalization of a fabricated speaking detection scheme to resist spoofing attacks. The classifiers developed by the proposed method on the proposed Squeeze-and-Excitation networks (SENet) were very effective with an EER of 0.43%. Exceptional results were achieved on the ASVspoof 2015 dataset with an overall EER of 0.05% and an improvement of 75.69% compared to the State-of-the-Art method. It is limited to ASVspoof 2015; it has not been tested on newer datasets or real-world scenarios.

Singh and Singh (2021) [14] voice-converted and synthesized 419,426 audio files to examine. The authors classified the double (real versus fake) and multiclass (specific attack types) and obtained a number of spectral features with models like CNN, WaveNets, Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs). Using CNN, WaveNet, LSTMs, and GRUs, the false positive rate (FPR) is 0.0216, 0.0260, 0.0302, and 0.0358 respectively. TTS systems have been tested on a limited number of samples (704); there is potential for overfitting in this dataset.

Using a Wav2vec 2 feature extraction model pre-trained by Zhang et al. (2023a) [15], a speech aspect fusion section is proposed aimed at classifying speech. Information from multiple timeframes and dimensions is integrated into the presented module to enhance detection accuracy. Among the key features are raw waveforms handled by the Wav2vec 2.0 (W2V2) approach. Based on an LA dataset, the approach had an EER of 1.18% and on a DF dataset, it had an EER of 2.62%. There is a high level of computational complexity involved in Wav2vec 2, and the fusion module is not fully explained.

Xue et al. (2022) [16] proposed a deepfake audio detection algorithm using an ASVspoof 2019 LA dataset, which combines fundamental frequency (F0), actual and imaginary spectrogram characters. The model extracted discriminative features by using subbands of F0 and phase information for the purpose of detecting deepfakes. A subband that contains most of the F0 data was used for feature extraction, and the modeling of disjoint subbands was implemented separately in both the real spectrogram and the imaginary spectrogram. The presented classifiers based on SENet achieved an extremely low EER of 0.43%. It has a complex architecture and has not been validated across multiple datasets.

The LA and Wild datasets were used by Wang et al. (2024) [17] to detect audio deepfakes. The Multi-scale Permutation Entropy (MPE) is the new method for pattern discovery that utilizes the active features of audio waves at singular time measures to capture a complex and dynamic characteristic. This implementation consisted of two types of features: MPE combined with LFCC and, optionally, W2V2. Combined with MPE, the Equal Error Rate (EER) obtained in ASVspoof2019 LA was 1.94%, which is much lower than the EER obtained when using LFCC alone. The combination of features needs to be optimized for limited ablation studies.

Iqbal et al. (2022) [18] investigated audio deepfake detection using the FoR dataset. They preprocessed audio data, extracted features such as MFCC and spectral roll off, and reduced the feature set using PCA. Six machine learning classifiers were tested. SVM achieved 97.57% accuracy, while XGB reached 92.60%. The recommended system outperformed the line in terms of EER. Based on the FoR dataset, classical machine learning may be unable to detect complex patterns.

Bhatia et al. (2022) [19] tackled the challenge of detecting synthesized audio, highlighting an importance of audio authentication as synthetic technologies advance. To address this challenge, they developed a method using diverse features and tested several models, including ML tools and DL architectures such as VGG16 and a custom CNN. As a result, these models achieved accuracy rates of 99.83% and 99.99%. Training and evaluation were conducted using a dataset of 12,890 five-second audio samples, comprising 8140 human and 4750 AI-generated speech samples from various TTS models. The results may not be generalizable to other languages because they are language-specific.

Doan et al. (2023b) [20] introduced the BTS-E structure to detect deepfake audios by analyzing correlations among breathing, talking, and silence in audio clips. Their approach used Gaussian Mixture Models (GMMs), which are probabilistic models for representing normally distributed data, to segment sounds into silence, breathing, and talking.

They then extracted positional correlation encoding for artificial audio detection. These characters were technically integrated with existing countermeasures such as RawNet2, a deep neural network classifier for audio deepfake detection. BTS-E improved performance, reducing the Equal Error Rate (EER)—a metric indicating where false acceptance and false rejection rates are equal—by up to 46% compared to the baseline. There are concerns regarding reproducibility of the dataset.

Ren et al. (2021) [21] proposed the Sinc-SENet scheme, an approach for speech deepfake detection which integrates Sinc convolution, a convolutional layer using parameterized Sinc functions to extract frequency features, and a squeeze-excitation module that adaptively recalibrates channel-wise feature responses. This method focuses on efficient feature extraction and channel-wise recalibration. It achieved an average Equal Error Rate (EER) of 7.23% on the ASVspoof 2019 LA dataset. There is a moderate EER compared to SOTA, and the architecture needs to be optimized.

Kawa et al. (2022) [22] developed SpecRNet for deepfake audio detection using a WaveFake dataset. In their approach, audio is first preprocessed into Linear Frequency Cepstral Coefficients and spectrograms before training SpecRNet, which incorporates residual blocks, FMS attention, and bidirectional GRU layers. As a result, SpecRNet achieved an EER of 0.1549 and an AUC of 99.9941, closely matching LCNN’s performance. It is limited to WaveFake dataset; it has not been tested on newer datasets or real-world scenarios. Xie et al. (2023a) [23] tested their method on three audio datasets and checked how it worked with different languages and noisy backgrounds. Their ASDG model combined LCNN and Bi-LSTM features and used adversarial domain discrimination and triplet mining to tell authentic speech from false. In clean conditions, ASDG reached an Equal Error Rate of 5.70%, which was better than the baseline models. EER is higher in noisy conditions and the training pipeline is more complex.

Liu et al. (2024) [24] proposed MSCR-ADD, a framework that detects mono-to-binaural audio deepfakes using multi-space channel representation learning. To evaluate their method, the authors considered the ASVspoof2019, ASVspoof2021, and VSDC datasets. The approach notably utilizes specialized encoders to extract channel-specific, invariant, and differential features. As a result, MSCR-ADD achieved an Equal Error Rate (EER) of 0.38% on the VSDC dataset. The company is specialized in the evaluation of binaural audio; mono audio evaluations are limited.

To enhance the security of AVS for spoofing attacks, Lei et al. (2020) [25] created the new Siamese CNN with the Gaussian probability feature. Both CQCC and LFCC use features that are extracted from the ASVspoof 2019 database. On the evaluation set, the Siamese CNN and LFCC system achieved 38.14% and 32.06% higher improvement over the baseline (LFCC + GMM), respectively, for the min-TDCF and the EER. A complex training requirement is dependent on the engineering of the feature.

Ma et al. (2023) [26] proposed an end-to-end anti-spoofing model using the ConvNeXt architecture without requiring any features to be extracted beforehand. They introduce multi-scale convolution layers and a channel attention module to enhance feature extraction besides emphasizing the greatest critical elements of audio. It performed with an EER of 0.64% on the LA evaluation set, as well as with good generalization; it performed with EERs of 0.86% and 1.18% on the ASVspoof 2015 dataset. The application potential of this technology is limited due to its computationally intensive nature.

3. Methods

The main components of our proposed system are presented in this section. Figure 1 illustrates these components which are described in detail in the following subsections. Two identical branches of the proposed dual-branch CNN with shared weights handle separate input audio samples, preparing for comparison. An architecture pipeline can be described as follows:

Figure 1. Illustrates the proposed system.

3.1. Data Analysis

It is a method for gathering and analyzing accurate information from different sources to identify trends, evaluate probabilities, etc., to evaluate possible results. We used the CtrSVDD [27] dataset in this search. An enhanced controllability, diversity, and data openness benchmark dataset for Controlled Singing Voice Deepfake Detection (CtrSVDD) that they believe could accelerate research towards SVDD. The synthesis pipeline is managed end-to-end, including specific details about the source and target datasets and how deepfakes are created. A total of 188,486 deepfake song clips (260.34 h) were generated and compared with 32,312 bonafide song clips (47.64 h) from 164 singers with an average length of 5.02 s in order to achieve diversity.

Data openness is enabled by the inclusion of 220,798 mono vocal clips recorded at 16 kHz in a total of 307.98 h in the CtrSVDD dataset. In order to provide a comprehensive evaluation landscape, they include 14 deepfake systems across both SVS and SVC. It was important to select models from open-source toolkits, train them on publicly available singing benchmarks, and then apply them to real singing vocals, with the exception of A14, which was evaluated using a commercial system. Source datasets and deepfake methods allocation are illustrated in Figure 1.

3.2. Data Cleaning

In order to understand and utilize raw data, it needs to be transformed into a usable format. Typically, real-life datasets are incomplete and formatted inconsistently. It is possible to resolve this problem by preprocessing the data in order to make it more efficient and complete for analysis.

As a result, the model is able to detect audio signals directly, thus simplifying the detection process in comparison with traditional methods that require extensive preprocessing and analysis of features.

3.3. Feature Generation

The goal is to convert raw audio signals into numerical representations that effectively distinguish genuine speech from synthetically generated speech (deepfake). Among the mainstream approaches, there are three major categories: handcrafted, deep learning, and hybrid/fusion which are explained in Table 1.

Table 1. Feature generation methods.

Feature	Methods	Core Idea	Advantages	Limitations
Traditional Hand-crafted	MFCC, LFCC, CQCC	Signal processing-based; extracts human-interpretable acoustic attributes (spectral/cepstral domains).	Computationally efficient, highly interpretable, robust against specific attack types (e.g., replay).	Weaker generalization; may fail to capture traces left by the latest generative AI models.
Deep Learning (SSL)	Wav2vec 2.0, WavLM	Leverages self-supervised learning on massive unlabeled audio data to automatically extract high-level, abstract representations.	Strong representational power, excellent generalization, adapts well to unseen attacks.	Computationally expensive, lacks interpretability (“black box” problem).
Hybrid/Fusion	Fusion of MFCC + WavLM features	Combines the precision of hand-crafted features with the high-level semantics of deep features.	Superior overall performance, highly robust, captures subtle and multi-modal forgery artifacts.	High system complexity and difficult to design optimally.

3.4. Feature Selection

Feature selection is crucial when building a fake audio detection system for removing noisy features and keeping only those relevant to the system.

We use the EER method. This step is essential to determine the number of features in a dataset. We choose features with the lowest EERs as shown in Table 2.

Table 2. Feature selection using EER.

Method	Features	EER
RNN	Mel-Spectrogram	8.18
Siamese CNN using Gaussian	LFCC CQCC	3.78
CNN	CQCC	9.61
Res2Net	CQT Features	6.02
Spec-ResNet	MFCC, CQCC, STFT	4.75
dual-branch CNN with shared weights	MFCC	2.95

A secondary metric is t-DCF, which provides additional insight into the real-world costs of detection errors. While EER is the primary performance metric, it also provides insight into how detection errors are measured daily. In practical applications, models with lower t-DCF values are more suitable for deployment due to their higher accuracy and lower risk of costly errors, as shown in Table 3.

Table 3. Feature selection using t-DCF.

Method	Features	t-DCF
RNN	Mel-Spectrogram	0.233
Siamese CNN using Gaussian	LFCC CQCC	0.378
CNN	CQCC	0.093
Res2Net	CQT Features	0.1569
Spec-ResNet	MFCC, CQCC, STFT	0.110
dual-branch CNN with shared weights	MFCC	0.045

3.5. Feature Extraction

Mel-Frequency Cepstral Coefficients (MFCCs) are widely used features in tasks like speech recognition and deepfake detection. These features come from a mel-scale triangular filter that reflects how humans perceive sound by adding more filters in the lower frequency range, as shown in Algorithm 1. After extraction from the audio, normalizing and reshaping the MFCCs make them effective inputs for the dual-branch CNN with the shared weights model, as shown in Figure 2.

Figure 2. The MFCC algorithm.

Algorithm 1: MFCC Algorithm

Input: Dataset (D) containing audio samples from CtrSVDD. Output: MFCC features. Step one: Begin { Step two: Load voice from resource. Step three: Pre-emphasis (High-Frequency Boost). Framing (Slicing into Short). Step four: Fast Fourier Transform (FFT)—Time to Frequency. // Mel Filter Bank (Mimicking Human Hearing). Logarithm (Log)—Perceiving Loudness Step five: Discrete Cosine Transform (DCT)—Compression          } //End Algorithm

3.6. Dual-Branch CNN with Shared Weights

To ensure consistency in feature extraction, each dual-branch CNN with a shared weights branch follows the same architecture. A description of the configuration can be found in Table 4 and Table 5.

Table 4. Parameter of the proposed dual-branch CNN with shared weights layer.

Layer	Configuration	Activation Function
Input l	MFCC feature	-
Conv 1	filters = 32, kernel size = (3,3), stride = 1	Relu
Max pooling 1	Size = (2,2)	-
Conv 2	filters = 64, kernel size = (3,3), strides = 1	Relu
Max pooling 2	Size = (2,2)	-
Conv 3	filters = 128, kernel size = (3,3), strides = 1	Relu
Residual connection	Skip connect between Conv 2 and 3	-
Self-attention	Multi-head = 8	Sigmoid
Dropout	Rate = 0.5	-
Fully connect	Neuro = 129	Relu

Table 5. Key parameter of the proposed work.

Parameter	Value
Sampling Rate	16 kHz
Window Length	25 ms
Hop Length	10 ms
Pre-emphasis Coefficient	0.97
Number of Mel Filters	40
Frequency Range	20 Hz–8000 Hz
Normalization	Z-score (per-frame)

The dual-branch CNN with shared weights consists of two identical CNNs with the same architecture as the previous section, but without fully connected layers. The baseline system calculates MFCC features separately and uses them as input to the convolutional layer as shown in Figure 3. Each branch of a CNN is trained simultaneously on a training dataset, and the two embedding vectors are of the same dimension, as shown in Algorithm 2.

Figure 3. The dual-branch CNN with shared weights architecture.

After concatenating these two vectors, we input them into the self-attention module and a residual connection, followed by a fully connected layer with Sigmoid and dropout.

3.7. Self-Attention Mechanisms and Residual Connections

In this way, the network is able to capture long-range dependencies and improve feature representations by focusing on different parts of the audio signal. Using attention maps, we can understand how self-attention identifies critical areas in audio signals, as shown in Figure 4. While extracting features, the model focuses on temporal and spectral regions. Natural speech patterns are emphasized by the attention mechanism for real audio signals.

Figure 4. Self-attention mechanisms and residual connection.

False audio signals, on the other hand, are detected by the attention mechanism when anomalies are detected in the frequency distribution and time transitions of the signal. As can be seen in these visualizations, self-attention makes it possible for the model to capture subtle differences, improving the ability of the model to extract features for classification significantly.

It is important to include residual connections when training deeper architectures so that information and gradients can flow more easily. By focusing on important spectral and temporal regions, self-attention improves feature learning. There is a higher gradient flow when residual connections are present, which leads to more stable training. As a result of dropout regularization, overfitting cannot occur, thus ensuring robust generalization. To evaluate the contribution of individual components to the model’s performance, we conducted ablation studies by systematically removing the self-attention modules and the Sigmoid function. The results demonstrate the importance of each component.

To evaluate the contribution of individual components to the model’s performance, we conducted ablation studies by systematically removing the self-attention modules and the Sigmoid function.

• Without Self-Attention Modules: the EER increased to 3.87%, emphasizing the critical role of self-attention in capturing long-range dependencies and enhancing feature extraction.
• Without Sigmoid: the EER increased to 3.45%, highlighting the necessity of this customized contrastive loss function for effective discrimination between real and fake audio.

Algorithm 2: Proposed work algorithm

Input: Dataset (D) containing audio samples; Training Epochs (E) Output: Trained Model; Classification Result (Real/Fake) Training Phase: 1. Initialize Model: -Dual-branch CNN with shared weights branches with identical architectures. -Loss Function: binary—crossentropy. 2. For each epoch e in E: a. Sample from dataset D. b. Preprocess audio signals A: -Features_A ← Extract_MFCC(A) c. Feature Extraction through dual-branch CNN with shared weights: -Embedding_A ← Process Features_A through Branch_1. Output: Evaluation Metric. Begin {       Step one: Load data.       Step two: Import key libraries. //ex: Dropout, Adam, Dense.        Step three: Reshape the data.             //Shape =(input-vector-shape), type=’float32’,           Dense=1024, activation=relu.       Step four: Normalize the data. //x-train, x-test, y-train, y-test.       Step five: Define the model function.       //input layer = Dense (32, activation=relu)       Combined1 = Dropout (0.3)       hidden layer1 = Dense (64, activation=relu)       combined2= Dropout (0.3)       hidden layer2 = Dense (128, activation=relu)       residual_input = Dense (128, activation=relu)       attention_output = (num_heads= 8)       output layer = Dense (1, activation= sigmoid)      model= (optimizer=Adam; learning_rate=1 × 10−4; loss=binary_crossentropy;      metrics= accuracy         Step six:  Run the model.      //model. Fit (x-train, y-train, validation data= (x-val, y-val), epochs=80, batch      size=512).      Step seven: Evaluation Matrix. // Precision, Recall, F-score // End Algorithm

4. Results

In this section, the experimental results are presented using some performance metrics. Furthermore, we have highlighted the performance based on the chosen features from previous studies on the same dataset.

4.1. The Technical Requirement

4.1.1. Hardware

The computer has a perfect effect on program implementation. There should be suitability between the computer’s specifications to be implemented. We apply the suggested system with the following specification:

• Macintosh platform having a 2 GHz Quad-Core Intel Core i5 processor with 16 GB memory.

4.1.2. Software

The software used to implement the system has an effect on the results. The following specifications have also been implemented to the software using this system, as shown in Table 6.

Table 6. Specification used to implement the suggested system.

S/H	Properties
System	Macintosh platform
Processor Speed	2 GHz
Processor Name	Quad-Core Intel Core i5
Memory	16 GB
Programming Tool	VS code
Programming Language	Python (PyCharm 3.10)

4.2. Dataset

We used the Controlled Singing Voice Deepfake Detection (CtrSVDD) benchmark dataset in this search. It has been curated to increase controllability, diversity, and openness, thereby accelerating research on SVDD. The entire synthesis pipeline is managed end-to-end, including the source and target datasets and the exact method used for deepfake generation. The CtrSVDD, 188,486 deepfake song clips (260.34 h), were generated and compared with 32,312 bonafide song clips (47.64 h) from 164 singers with an average length of 5.02 s and a sample rate 16 kHz in order to achieve diversity. Data openness is supported by CC BY-NC-ND 4.0 licenses for all datasets. In the case of deepfake clips, the generation results are based on open-source methods such as 7 SVS and 7 SVC, as well as those of a collaborating company, which allows us to distribute the data under the terms mentioned above, as illustrated in Section 3.

Figure 5 demonstrates the main parts of the CtrSVDD dataset and the number of audios in each part.

Figure 5. Main component of the CtrSVDD dataset.

Table 7 summarizes the size, number of audios, no. of AI used to synthesize this audio and no. of datasets used of each part of the CtrSVDD dataset.

Table 7. Analyzing the parts of CtrSVDD dataset.

CtrSVDD Dataset Parts	Size	No. of Audio	No. Of AI Used	Dataset
Train	7.9 GB	84,010	A01, A02, A03, A04, A05, A06, A07, A08	M4singer, Ofuton, jvsmusic, Oniku, Kiritan, Opencpop
Train.txt	5.0 MB	84,010	A01, A02, A03, A04, A05, A06, A07, A08	M4singer, Ofuton, jvsmusic, Oniku, Kiritan, Opencpop
Development	3.4 GB	43,101	A01, A02, A03, A04, A05, A06, A07, A08	M4singer, Ofuton, jvsmusic, Oniku, Kiritan, Opencpop
Dev.txt	2.6 MB	43,101	A01, A02, A03, A04, A05, A06, A07, A08	M4singer, Ofuton, jvsmusic, Oniku, Kiritan, Opencpop
Test	9.0 GB	92,769	A09, A10, A11, A12, A13, A14	M4singer, Kising, ACESinger
Test.txt	6.3 MB	92,769	A09, A10, A11, A12, A13, A14	M4singer, Kising, ACESinger
Total	20.18 GB	219,880	14	8

We presented our experimental work and explained each step in the proposed system:

An optimal learning rate, batch size (mini-batch size), and dropout rate were determined using grid search to train the network. When the maximum number of epochs has been reached, or the validation loss has not improved after 10 epochs, the training process stops. In order to conduct the grid search, the following parameter ranges were used:

• Learning rate: 1 × 10⁻⁴ To 1 × 10^−2.
• Batch size: 32–64.
• Dropout rate: 0.1–0.3.

According to the tuning results, the following hyperparameters were chosen:

• Learning rate: 1 × 10⁻³.
• Batch size: 64.
• Dropout rate: 0.3.

The training strategies used in this project were carefully designed to promote stability during training and to help the model achieve the best possible results.

• Firstly, the Adam optimizer was chosen because it can acclimate the learning rate for each parameter, which helps the model learn more efficiently.

For the optimizer’s momentum parameters, setting the values to 0.9 and 0.999 is recommended. These values help the optimizer update the model weights more effectively during training.

2.

Secondly, the learning rate was managed using the ReduceLROnPlateau scheduler, which automatically lowers the learning rate when the model’s efficiency on the validation set stops improving.

If validation loss does not improve for five epochs, the learning rate is halved. This helps the model continue learning and avoid suboptimal solutions.

A minimum learning rate is also set to avoid reducing the learning rate too much, which helps maintain stability during the fine-tuning phase.

Regularization techniques were also applied to further improve the model’s performance.

To reduce the risk of overfitting and manage the complexity of the model, L2 regularization was used with a weight decay value of 1 × 10⁻⁴.

Batch normalization was implemented to normalize the activations across layers, which donates to more stable and consistent training.

Dropout with a rate of 0.3 was applied to the dense layers to improve the model’s ability to generalize to new data.

3.

Finally, training was stopped if there was no improvement in the validation loss for ten consecutive epochs, which helped to prevent unnecessary training and reduce the risk of overfitting.

4.3. Preprocessing

The method reduces the amount of preprocessing and feature engineering required by allowing the model to work directly with raw audio signals, thereby simplifying detection.

4.4. Feature Selection

While EER is the primary metric for comparing performance, as shown in Table 8, T-DCF provides additional insight into the real-world costs of detection errors. The dual-branch CNN with shared weights (0.045) is a model with a lower t-DCF value, as shown in Table 9, demonstrating both improved accuracy and lower error rates in practical applications.

Table 8. Comparison with results for CtrSVDD in terms of EER.

Method	Feature Extraction	EER
CNN	CQCC	9.52%
RNN	Mel-Spectrogram	8.73%
CRNN	Spectrogram	5.41%
Res2Net	CQT	6.20%
Spec-ResNet	MFCC, CQCC, STFT	5.47%
TEResNet	Global 2D-DCT	6.48%
Siamese CNN using Gaussian	LFCC CQCC	3.93%
Dual-branch CNN with shared weights	MFCC	2.93%

Table 9. Comparison with results for CtrSVDD in terms of t-DCF.

Method	Feature Extraction	t-DCF
CNN	CQCC	0.232
RNN	Mel-Spectrogram	0.215
CRNN	Spectrogram	0.132
Res2Net	CQT	0.1596
Spec-ResNet	MFCC, CQCC, STFT	0.101
TEResNet	Global 2D-DCT	0.159
Siamese CNN using Gaussian	LFCC CQCC	0.095
Dual-branch CNN with shared weights	MFCC	0.043

4.5. Pattern Discovery

Based on EER and t-DCF, the MFCCs have the lowest values, so we used them for feature extraction.

An MFCC is a Mel-Frequency Cepstral Coefficient. An audio signal’s short-term power spectrum is represented compactly to mimic the human ear’s nonlinear frequency perception.

To calculate MFCCs, follow this method:

• The signal needs to be divided into short time windows (for example, 25 ms).
• The power spectrum should be calculated using FFT.
• Make a Mel-filter bank, a bank of triangular filters at different frequencies (logarithmic at higher frequencies, linear around 1 kHz). This reflects how the human brain perceives equal distance between frequency regions.
• The filter bank energies should be expressed in logarithmic form.
• Decorrelate and compress log-energy values (usually 13–40 coefficients) using the Discrete Cosine Transform (DCT).
• The MFCC vector for each frame must contain the first N coefficients (e.g., 40).

Our method involves discarding fine pitch and phase data from the voice to obtain spectral envelopes (one per time frame). Our model (CNN + Attention on 40x64 MFCCs) is able to capture the temporal dynamics of the voice without consuming a large amount of memory. In order to detect synthetic speech, MFCCs provide a compact, noise-proof, perceptually relevant representation of the spectral envelope.

As a result of using 40 MFCCs, good spectral detail is achieved, computational efficiency is improved, and powerful dynamic features can be recognized, all of which contribute to the accurate detection of deepfakes in real situations.

4.6. Learning Algorithm

After finding the MFCC features between audios and labels, we enter these columns into CNN learning models with some properties, as illustrated in Section 3.

Self-Attention Modules Contribution

These results underscore the essential role of these components in achieving the observed high performance. Our experiments further evaluated combinations of these features, affirming that both elements are necessary to achieve State-of-the-Art accuracy in deepfake detection.

The layer-wise feature map analysis provides insights into how the features evolve through the network. In the early layers, the model captures basic spectral and temporal patterns, as seen in the highly scattered feature maps. Intermediate layers refine these patterns, with some regions becoming more pronounced, indicating the extraction of higher-level representations.

In the final self-attention-enhanced layers, the feature maps show focused regions that correspond to key discriminative characteristics of real and fake audio samples. This refinement through the layers highlights the critical role of self-attention in isolating anomalies and improving classification accuracy. The progressive enhancement of feature representations directly contributes to the model’s superior performance, as evidenced by metrics such as EER and t-DCF.

Furthermore, we examined several combinations to evaluate the relative contributions of self-attention modules and Sigmoid. Overall, our experiments show that both elements are necessary to achieve state-of-the-art deepfake detection accuracy. MFCCs are a very robust way to represent the content of an audio signal, but they lack fine-grained temporal information.

Future work should explore combining them with complementary features, such as spectral features or MFCC deltas, to make the model better at capturing temporally evolving inconsistencies.

That is challenging on the competition’s CSV, which has real-time audios and does not exist in training or testing CSV.

After we learn the training CSV, we do testing on the competition CSV.

The best result was accuracy of 0.9757 and validation-loss function of 0.0128 because the first ten epochs were overfitting.

Based on Table 10, Table 11, Table 12 and Table 13, the dual-branch CNN with shared weights demonstrates superior performance across all evaluation metrics.

Table 10. Summary of the execution time.

Model	Execution Time
Preprocessing	45 s
MFCC	20 min
Dual-branch CNN with shared weights	150 min
Totally	170.45 min

Table 11. Confusion Matrix of the proposed model on CtrSVDD.

Metric	Accuracy
Accuracy	97.27%
Precision	99%
Recall	99.26%
F1 score	98.88%
ROC-AUC	98%
t-DCF	0.043%
EER	2.93

Table 12. Confusion Matrix of the proposed model on ASVspoof19.

Metric	Accuracy
Accuracy	97.27%
Precision	99%
Recall	99.26%
F1 score	98.88%
ROC-AUC	98%
t-DCF	0.043%
EER	2.93

Table 13. Comparison with features on ASVspoof 2019 dataset.

Features	CtrSVDD EER	CtrSVDD t-DCF
Spec-Resnet	5.47	0.101
CNN	9.52	0.232
RNN	8.73	0.215
CRNN	5.41	0.132
CQCC +GMM	9.06	0.342
CQCC-Resnet	8.98	00.289
MFCC-Resnet	8.72	0.367
LFCC +GMM	8.46	0.290
LFCC-LCNN	5.04	0.109
FFT-LCNN	4.51	0.119
FFT-LCGRNN	3.00	0.089
LFCC-MFCC	4.29	0.044
LFCC-Siamese CNN	4.98	0.095
RW-Resnet	2.45	0.073
MFCC	2.93	0.048

As a result of the architecture’s enhanced feature extraction capabilities, self-attention modules, and residual connections, the model can generalize to various audio manipulations with improved performance.

On the CtrSVDD dataset, we evaluated the proposed dual-branch CNN with shared weights against several baseline models. In Table 14, each model is summarized according to its accuracy. Using accuracy as the primary metric, the comparison table shows the performance of various audio deepfake detection models.

Table 14. Performance of various audio deepfake detection models.

Method	CtrSVDD Accuracy	EER
RNN	86	8.73
Continual learning (RWM)	94.53	4.36
Traditional CNN	84	9.52
BiLSTM-CNN hybrid	90.22	5.41
Enhanced Spectro-temporal models	89	5.93
Ensemble of spectrogram models	83	5.04
Proposed (SpecRNet)	89	5.47
Proposed (SENet)	78.23	8.72
Transformer-based model	92.1	3.00
Proposed dual-branch CNN with shared weights	97.27	2.93

This section examines these methods and highlights their strengths and limitations:

This model divides the dataset into K equal-sized subsets, called folds, so we can evaluate the learning model’s performance using K-Fold Cross Validation. K-fold is used to train the model, and the last fold is used for testing. An estimate of the generalization ability of the model is then obtained by averaging the model’s performance over K iterations, as shown in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11. Therefore, we used k = 5, and the result was as shown in Table 15.

Figure 6. K-fold results on the CtrSVDD dataset.

Figure 7. Confusion matrix when k = 1.

Figure 8. Confusion matrix when k = 2.

Figure 9. Confusion matrix when k = 3.

Figure 10. Confusion matrix when k = 4.

Figure 11. Confusion matrix when k = 5.

Table 15. The result of dual-branch CNN with shared weights + K-Fold.

K. No	Accuracy
K = 1	97.99%
K = 2	98.14%
K = 3	97.81%
K = 4	97.63%
K = 5	97.69%

5. Discussion

System implementation was performed on a Macintosh platform with a dual-core Intel Core i5 processor and 16 GB of memory, illustrated in Table 10.

The VS code was used. As shown in Table 10 preprocessing takes 45 s on average, and calculating MFCC features of audios takes 20 min. The test and the training of dual-branch CNN with shared weights take 150 min, respectively. The proposed approach outperforms conventional machine learning and deep learning models on all metrics, as shown in Table 10. The detection system achieves 97.26% accuracy, 99% precision, 99.27% recall, and an F1-score of 98.88%, which indicates its effectiveness in real-world detection situations. The ROC-AUC score of 98% achieved further guarantee of the model.

Detection costs and error rates are high for methods utilizing traditional features like CQCC and simple models like GMM. Especially when artificial audio is generated using advanced synthesis techniques, these approaches are not as capable of detecting subtle differences between fake and real speech. In the case of deep learning methods (ResNet, CNN, RNN), performance is significantly reduced as a result of the deep learning architectures (ResNet, CNN, LCNN, CRNN).

This demonstrates that these models can capture more sophisticated audio patterns. These results are achieved with FFT-LCGRNN and CRNN architectures, which combine temporal and spatial information. The proposed dual-branch CNN with shared weights achieves impressive results by leveraging Siamese networks and handcrafted features such as MFCC. When used for deepfake detection, the dual-branch CNN with shared weights learns discriminative embeddings that make it easier for it to differentiate between real and fake audio. As a secondary metric, t-DCF can provide more insight into real-world detection errors than EER alone.

Low-t-DCF models, such as the proposed dual-branch CNN with shared weights (0.043), are better at predicting errors in practical applications, which makes them more useful for deployment. With low EER and t-DCF values, models such as FFT-LCGRNN, RW-ResNet, and the proposed enhanced dual-branch CNN with shared weights are most effective at detecting audio deepfakes. Compared to earlier models, these models take full advantage of advanced architectures that capture both temporal and frequency domain information. On the CtrSVDD dataset, we compared the performance of our proposed dual-branch CNN with shared weights with several baseline models.

Using ML (Random Forest) to detect audio deepfakes, we achieved 89%. In addition, we used another ML method, but it did not perform well.

It displays good baseline performance but is limited in its ability to capture temporal dependencies in audio data. According to the low accuracy, it may not be able to resist sophisticated audio deepfakes created with advanced synthesis techniques. Though RNNs are designed to cope with sequential data, they are only slightly more efficient than traditional CNNs. In addition, they are not capable of extracting and processing complex frequency–time patterns, which are crucial for detecting audio deepfakes.

Combining both temporal (via BiLSTM) and spatial (via CNN) features greatly enhances the performance of this hybrid model. In this case, deep learning architectures are combined to handle multifaceted data with superior performance. Transformers allow the audio input to be focused on critical regions by introducing attention mechanisms. In deepfake audio data, the model is capable of processing complex patterns with high accuracy.

Using multiple models in an ensemble improves generalization. As a consequence of the slightly lower accuracy compared to transformer-based and BiLSTM-CNN hybrid models, it could be argued that the limitations of an individual model may continue to propagate within ensembles without sophisticated integration mechanisms. A combination of spectral and temporal dynamics is successfully integrated in this model, which results in higher accuracy than a standalone CNN or RNN. While slightly less accurate than BiLSTM-CNN hybrids, it remains possible to improve their handling of intricate audio manipulations.

By continuously learning, the model adapts to new deepfake patterns, maintaining high accuracy over time. In real-world applications, adaptability is crucial to success.

The proposed model, which uses a dual-branch CNN with a shared weights architecture enhanced with self-attention, achieves the highest accuracy. Feature extraction and discrimination between real and fake audio are made easier through these innovations, making it the most reliable method evaluated here.

Because of its capability to balance computational efficiency and detection accuracy, it is a practical solution for real-world implementations. As a result, the results demonstrate that traditional methods such as CNNs and RNNs may not be able to handle sophisticated audio manipulations, whereas modern methods such as hybrid CNNs based on BiLSTM and transformers can handle them with great success. Based on self-attention mechanisms, the proposed enhanced dual-branch CNN with shared weights provides superior accuracy compared to all other methods. As a result of these studies, the evolution of audio deepfake detection technologies is paving the way for more robust, efficient solutions to combat the misuse of synthetic media.

6. Conclusions

Due to social media’s increasing popularity, more and more people are consuming news via social media instead of traditional media. In addition, social media has been used to spread deepfake audios, which can negatively impact individual users and society. As part of this research, we reviewed the existing literature on the characterization and detection of fake audios. Researchers attempted different approaches to build audio deepfake detection systems using machine learning and deep learning approaches.

This work focused on the competitive studies that used the CtrSVDD dataset. In addition, it highlighted these works’ strengths and weaknesses. From studying the existing works, it is concluded that the proposed approaches fail to build a system that can detect deepfake audios efficiently.

With an enhanced dual-branch CNN with a shared weights model, we present a technique for detecting audio deepfakes. By minimizing preprocessing requirements and improving feature extraction to achieve superior detection performance, the proposed method addresses several limitations of existing techniques. With its ability to process audio data in both the time and frequency domains, the model achieves significant accuracy gains over traditional and modern deep learning models.

Consequently, it provides researchers and practitioners with a reliable early detection tool for audio deepfakes. Among the many advantages offered by the dual-branch CNN with a shared weights architecture are increased detection accuracy, efficient feature extraction, and reduced preprocessing requirements.

Another strength of this model is its ability to generalize well, so it can handle many different types of audio manipulations, even those it has not seen before. Because of this, the method is promising for real-world uses such as security, protecting media integrity, and preventing fraud, where it is important to detect deepfakes quickly and accurately.

In the future, it will be important to make the model more resilient and to test how well it works with other types of deepfakes, such as video. The model should also be tried on larger and more varied datasets to see if it can scale up. Before using it in real situations, we need to check how complex it is to run. As synthetic media technologies keep changing quickly, these steps are necessary to keep deepfake detection systems effective.