Deepfake Speech Detection Using Perceptual Pathological Features Related to Timbral Attributes and Deep Learning

Abstract

The detection of deepfake speech has become a significant research area due to rapid advancements in generative AI for speech synthesis. These technologies pose significant security risks in applications such as biometric authentication, voice-controlled systems, and automatic speaker verification (ASV) systems. Therefore, enhancing the detection capabilities of such applications is essential to mitigate potential threats. This study investigates perceptual speech-pathological features, which are commonly used to evaluate the unnaturalness of voice disorders in clinical settings, as potential indicators for detecting deepfake speech. Specifically, the timbral attributes of hardness, depth, brightness, roughness, sharpness, warmth, boominess, and reverberation are examined. The analysis reveals that these attributes provide meaningful distinctions between genuine and synthetic speech. Furthermore, the detection performance is enhanced by extending the dimensional representation of timbral attributes, enabling a more comprehensive characterization of the speech signal. This paper proposes a method that combines two models: one utilizing the different dimensions of speech-pathological features with a deep neural network (DNN), and another employing a gammatone filterbank model that simulates the auditory processing mechanism of the human cochlea with ResNet-18 architecture, improving deepfake speech detection. The proposed method is evaluated on the Automatic Speaker Verification Spoofing and Countermeasures Challenge (ASVspoof) 2019 dataset. Experimental results demonstrate that the proposed approach outperforms baseline models in terms of Equal Error Rate (EER), achieving an EER of $5.93$%.

1. Introduction

Recent in developments Artificial Intelligence (AI) are transforming many aspects of our daily lives. With its remarkable capabilities, AI offers numerous benefits in diverse fields such as healthcare, finance, transportation, and entertainment. In the field of speech, AI has driven transformative innovations, offering numerous advantages and applications such as speech recognition, speech synthesis, and text-to-speech (TTS). Despite the numerous advantages offered by AI speech, the misuse of AI technologies, such as deepfake speech, poses a significant threat to economies and societies around the world. As a case in point, as highlighted by a report from AP News, ahead of the New Hampshire primary election, deepfake AI technology was used in robocalls that falsely attributed messages to US President Joe Biden, creating confusion and concern during the campaign [1]. This underscores the growing threat of AI-generated misinformation in political campaigns, which can undermine trust in political and social systems. Thus, security systems that utilize speech as a form of access, such as automatic speaker verification systems, have an inherent need for strong mechanisms related to the detection and countering of deepfake speech.

Several methods have been proposed for detecting deepfake speech [2,3,4,5]. These methods primarily focus on two main aspects: the exploration of various acoustic features [3,6,7] and the development of efficient classifiers [8,9,10].

The first aspect emphasizes utilizing and enhancing acoustic features. Various acoustic features have been employed to detect deepfake speech, such as spectrograms [11], mel-spectrograms [12], linear-frequency cepstral coefficients (LFCCs) [13], constant-Q cepstral coefficients (CQCCs) [14], constant-Q transform (CQT) [15], and mel-frequency cepstral coefficients (MFCCs) [16]. These features are represented using phase, power spectrum, and cepstral coefficients. However, they provide high-dimensional representations.

The second aspect employs a variety of classifiers, such as Gaussian mixture models (GMMs) [17], deep neural networks [18], convolutional neural networks (CNNs) [19], recurrent neural networks (RNNs) [20], long short-term memory (LSTM) [21], bidirectional LSTM (Bi-LSTM) [22], and residual neural networks (ResNets) [23].

Acoustic speech-pathological features are widely used to detect voice disorders [24,25,26] and have also been introduced to identify unnatural characteristics in synthesized speech [27] and audio [28]. These acoustic speech-pathological features effectively distinguish between genuine and deepfake speech [29,30].

Perceptual speech-pathological features have long been used to evaluate voice quality in disordered speech [31,32,33,34,35]. Recently, a limited number of studies have explored the applicability of such speech-pathological features for detecting deepfake speech, including our own earlier investigations [36,37]. These prior studies primarily focused on demonstrating the feasibility of using perceptual pathological features through global or utterance-level representations.

However, deepfake speech often differs from genuine speech not only in global perceptual characteristics but also in localized temporal inconsistencies, such as unstable timbral behavior across short segments, which is largely obscured by global averaging. To date, the segmental dynamics and temporal variability of perceptual speech-pathological features have not been systematically analyzed for deepfake speech detection, representing a critical gap in the literature.

In this study, we extend prior work by shifting from utterance-level feature aggregation to a segmental frame-wise analysis of perceptual speech-pathological features. This enables the modeling of temporal instability patterns that more closely reflect how human listeners perceive unnaturalness in synthetic speech. Furthermore, we conduct a comprehensive feature-level analysis and ablation study to identify which perceptual attributes contribute most significantly to deepfake discrimination. Finally, we propose a dual-path detection framework that integrates segmental perceptual features with an auditory-inspired gammatone filterbank model, exploiting their complementary perceptual dimensions through score-level fusion.

2. Perceptual Speech-Pathological Features

This research focuses on timbre as the primary perceptual attribute, given its vital role in music and audio cognition [38,39]. Timbre conveys essential emotional and perceptual information, making it invaluable across various areas of audio processing.

In more detail, timbre represents a complex range of auditory qualities that together determine the character or nature of a sound. It typically consists of various spectral and harmonic components that contribute distinctive characteristics to a sound [40]. Timbre is defined as the characteristic that allows listeners to distinguish between two sounds with identical pitch, loudness, and duration. Despite similarities in these aspects, unique timbral qualities enable the perception of distinct auditory experiences [41].

Several studies have investigated the modeling of timbre through psychoacoustic frameworks, focusing on the development of objective metrics for various timbral attributes [42,43,44]. In this study, we employ an extraction algorithm developed by the Audio Commons Project that has been extensively applied in psychoacoustic research [45]. This model expresses eight timbral characteristics—hardness, depth, brightness, roughness, sharpness, warmth, and boominess—and rates them from 0 to 100, while reverberation rate is represented by a value of 0 or 1 [45]. These timbral features are described in detail below.

2.1. Hardness

The perception of sound hardness is largely influenced by two key elements: loudness and harshness. A sound described as harsh suggests a disruption within its audio core, specifically within the 2 kHz to 5 kHz frequency range, where human hearing is most sensitive. Hardness, therefore, functions as an indicator of a sound’s desirability or lack thereof, and the connection between its loudness and frequency range can be analyzed based on human auditory perception.

Four key parameters are suitable for predicting hardness: the mean of the spectral centroid over time, the time-weighted average of the spectral centroid as it varies over time, the mean of the time-varying spectral level, and the slope of the spectral level [44].

2.2. Depth

Depth can refer to either a timbral or spatial attribute and has been reported to be associated with a marked low-frequency component [46]. In this study, we developed a depth model to calculate the depth metric by performing linear regression with three main elements: lower spectral centroid (LSP), lower ratio, and low-frequency limit (LSL):

$$\begin{array}{c}\hfill \mathrm{LSP}={\displaystyle \frac{{\sum }_{\mathrm{k}\left(30\mathrm{Hz}\right)}^{\mathrm{k}\left(200\mathrm{Hz}\right)}\mathit{f}\left(\mathit{k}\right)\mathit{x}\left(\mathit{k}\right)}{{\sum }_{\mathrm{k}\left(30\mathrm{Hz}\right)}^{\mathrm{k}\left(200\mathrm{Hz}\right)}\mathit{x}\left(\mathit{k}\right)}},\end{array}$$

(1)

where $k\left(\omega \right)$ is the bin number relating to frequency $\omega$, $f\left(k\right)$ is the frequency of the k$th$ bin, and $x\left(k\right)$ is the magnitude of the k$th$ bin. The mean lower spectral centroid is then calculated across all frames:

$$\begin{array}{c}\hfill \mathrm{Lower}\mathrm{ratio}={\displaystyle \frac{{\sum }_{\mathrm{k}\left(30\mathrm{Hz}\right)}^{\mathrm{k}\left(200\mathrm{Hz}\right)}\mathit{f}\left(\mathit{k}\right)\mathit{x}\left(\mathit{k}\right)}{{\sum }_{\mathrm{k}\left(30\mathrm{Hz}\right)}^{\mathit{k}\left(\mathrm{Nyquist}\right)}\mathit{x}\left(\mathit{k}\right)}},\end{array}$$

(2)

where k(Nyquist) represents the frequency bin corresponding to the Nyquist frequency. The mean lower ratio is subsequently computed across all frames as

$$\begin{array}{c}\hfill \mathrm{LSL}={\displaystyle \frac{{\sum }_{\mathrm{k}\left(30\mathrm{Hz}\right)}^{\mathrm{k}\left(200\mathrm{Hz}\right)}\mathit{x}\left(\mathit{k}\right)}{{\sum }_{\mathrm{k}\left(30\mathrm{Hz}\right)}^{\mathit{k}\left(\mathrm{Nyquist}\right)}\mathit{x}\left(\mathit{k}\right)}}.\end{array}$$

(3)

2.3. Brightness

Brightness has been explored in considerable detail, with studies demonstrating that the spectral centroid strongly correlates with perceived brightness [47,48]. In a recent study, Pearce examined existing models and formulated a novel brightness model that integrates both a variant of the spectral centroid and the spectral energy ratio [49]. We apply this model in the current study. The calculation of brightness involves two metrics, one of which is the frequency-limited spectral centroid (FLSC) designed to focus on frequencies exceeding 3 kHz, and the other is the ratio:

$$\begin{array}{c}\hfill \mathrm{FLSC}={\displaystyle \frac{{\sum }_{k\left(3\mathrm{kHz}\right)}^{k\left(\mathrm{Nyquist}\right)}\mathit{f}\left(\mathit{k}\right)\mathit{x}\left(\mathit{k}\right)}{{\sum }_{k\left(3\mathrm{kHz}\right)}^{k\left(\mathrm{Nyquist}\right)}\mathit{x}\left(\mathit{k}\right)}},\end{array}$$

(4)

and

$$\begin{array}{c}\hfill \mathrm{Ratio}={\displaystyle \frac{{\sum }_{\mathrm{k}\left(3\mathrm{kHz}\right)}^{\mathit{k}\left(\mathrm{Nyquist}\right)}\mathit{x}\left(\mathit{k}\right)}{{\sum }_{\mathrm{k}\left(20\mathrm{Hz}\right)}^{\mathit{k}\left(\mathrm{Nyquist}\right)}\mathit{x}\left(\mathit{k}\right)}},\end{array}$$

(5)

where $k\left(\omega \right)$ represents the bin number corresponding to a frequency $\omega$, $f\left(k\right)$ represents the frequency linked to the k-th bin, $x\left(k\right)$ denotes the magnitude of the k-th bin, and $k\left(N_{\mathrm{yquist}}\right)$ stands for the bin number corresponding to the Nyquist frequency. Finally, linear regression is performed to derive findings for brightness:

$$\mathrm{Brightness}=-25.8699+64.0127\left(0.44{log}_{10}\left(\mathrm{FLSC}\right)+{log}_{10}\left(\mathrm{Ratio}\right)\right)$$

(6)

2.4. Roughness

Roughness causes a buzzing sensation and creates a harsh perception. Rough sounds are perceived based on amplitude fluctuations in the voice modulated at between 16 and 80 Hz. The roughness model is built based on three variables derived from a study using a sinusoidal model approach. The first two variables, which correspond to the amplitude of two sinusoidal components, are the dependence of roughness on intensity and the dependence of roughness on the degree of amplitude fluctuation [50]. Roughness is calculated as

$$\mathrm{Roughness}=0.5X^{0.1}{Y}^{3.11}Z,$$

(7)

where

$$\begin{array}{c}\hfill X=A_{min}\times A_{max},\end{array}$$

(8)

$$\begin{array}{c}\hfill Y={\displaystyle \frac{2A_{min}}{A_{min}+A_{max}}},\end{array}$$

(9)

$$\begin{array}{c}\hfill Z=e^{(-3.5s(f_{max}-f_{min}))}-e^{(-5.75s(f_{max}-f_{min}))},\end{array}$$

(10)

and

$$\begin{array}{c}\hfill s={\displaystyle \frac{0.24}{0.0207f_{min}+18.96}}.\end{array}$$

(11)

$A_{max}$ and $A_{min}$ indicate the maximum and minimum magnitudes of the peak pairs, respectively, and $f_{max}$ and $f_{min}$ correspond to the maximum and minimum frequencies of the two peaks.

2.5. Warmth

Relatively few studies have explored the perception of warmth and the various factors that contribute to it. One example is the work of Sorensen, who found that the acoustic characteristics of concert venues, including their shape and the way sound reflects, significantly affect how warm orchestral music sounds to listeners [51]. Specific features of a room, such as its size, the duration of sound echoes, and the location of reflections, were shown to enhance the sense of liveliness and fullness in the lower frequencies of music, which are closely associated with the perception of warmth.

Bromham explored how different audio effect processing techniques influence warmth [52]. The study highlighted the importance of the bass ratio (BR) metric, which measures the distribution of energy in the low-frequency spectrum. Changes to BR, implemented through methods like equalization, were found to significantly impact the perceived warmth of the audio.

In this study, the warmth model was developed using an iterative modeling technique. Various subsets of low-level timbral characteristics were selected, and a multilinear regression approach was applied [46].

2.6. Sharpness

Sharpness is a metric associated with the perception of acute or piercing sensations, which becomes more pronounced as the central frequency shifts to a higher range. Based on this observation, Zwicker et al. introduced the concept of “acum”, which is defined as the unit of sharpness for narrowband noise centered at 1000 Hz and with a loudness level of 60 phons [53]. Sharpness is calculated using a model based on loudness density and is measured across the Bark scale. The resulting sharpness is typically represented as

$$\mathrm{Sharpness}=0.11{\displaystyle \frac{{\int }_0^{24Bark}{D}^{\prime }\left(z\right)g\left(z\right)dz}{{\int }_0^{24Bark}{D}^{\prime }dz}},$$

(12)

where 24Bark refers to the 24 critical bands of human hearing. $D^{\prime }\left(z\right)$ denotes the density of loudness within the critical-band rate z. Loudness, a fundamental characteristic of auditory perception, is measured in phons. The function $g\left(z\right)$ serves as the weighting factor for sharpness at a given rate. Psychoacoustic studies have determined that this weighting factor is $1.0$ for frequencies up to $3000\mathrm{Hz}$, rising sharply to $4.0$ for frequencies beyond this range.

2.7. Boominess

Boominess is evaluated to quantify the booming sensation through a method that involves summing the power of the 1/3 octave band signals [54]. As booming is commonly perceived as a low-frequency vibration, it is represented by the ratio of the loudness below 280 Hz to the overall bandwidth. The degree of booming is calculated as the booming index (BI), a quantitative measure for assessing and characterizing the boominess of sound, as

$$\mathrm{BI}=\mathrm{PS}\times (S_l/S_t),$$

(13)

where “PS” represents the power summation of the 1/3 octave band signals, $S_t$ denotes the loudness of the total band, and $S_l$ refers to the loudness of the band below 280 Hz.

2.8. Reverberation

Reverberation is a prolonged sound that continuously persists even after the source has already stopped. Reverberation may be caused by the reflection of multiple sounds from the environment. The term reverberation refers to the acoustic fading of an audio signal and encompasses various standard measures. Among these, the most prevalent is Reverberation Time 60 (RT60), which gauges the duration of decay. Although RT60 is often assessed in concert halls and similar settings, estimating it from recordings poses challenges. In 2015, the IEEE conducted the Acoustic Characterization of Environments (ACE) challenge to attempt blind estimation of RT60 and direct-to-reverberant ratio in recorded speech signals [55]. The RT60 values are provided as inputs to a linear regression model, which produces binary outputs, where 0 denotes the absence of reverberation and 1 denotes the presence of reverberation.

2.9. Gammatone Filterbank

The gammatone filterbank is related to human auditory perception and serves as a key component of auditory signal processing, designed to replicate the time-frequency analysis performed by the basilar membrane of the human cochlea [56]. It consists of a series of filters, each tuned to a specific frequency range, reflecting the frequency-selective properties of different regions along the basilar membrane. The most frequently used gammatone filterbank models are the cochlear model [57] and the auditory filterbank model based on Equivalent Rectangular Bandwidth ERB [58]. Figure 1 visually represents the frequency responses generated by the gammatone filterbank.

Figure 1. Frequency response of gammatone filterbank.

In this study, the speech signal duration was fixed at 4 s with a sampling rate of 16 k $\mathrm{Hz}$. To ensure uniformity, signals shorter than 4 s were looped from the beginning, while longer signals were truncated to match the specified duration. The gammatone filterbank employed in the analysis consists of 32 filters, with a minimum center frequency of 60 $\mathrm{Hz}$ and a maximum center frequency of 1200 $\mathrm{Hz}$. As a result, the extracted feature representation has a dimensionality of $32\times 401$.

3. Proposed Method

3.1. Feature Analyses

This section explores the potential of speech-pathological features for differentiating between genuine and deepfake speech. Figure 2 compares the raw waveforms of genuine (top) and fake (bottom) speech signals. The figure shows that the fake speech signal appears to be mixed with noise compared to the genuine speech signal. Figure 3 presents box plots illustrating the differences in each pathological feature when distinguishing between genuine and fake speech signals, based only on their average values from the Audio Commons Project [45]. In this chart, 1000 genuine and 1000 fake speech signals were randomly selected from the ASVspoof 2019 dataset, as summarized in Table 1 and Table 2. Each speech signal was standardized to a duration of 4 s with a sample rate of 16 k $\mathrm{Hz}$. To ensure consistency, signals shorter than 4 s were looped from the beginning, while those exceeding 4 s were cropped accordingly. The results indicate that hardness and warmth are particularly effective in differentiating genuine speech from deepfake speech. Additionally, depth, sharpness, and boominess provide moderate utility in this distinction. However, features such as brightness, roughness, and reverberation may be less effective in separating genuine speech from deepfake signals. The reverberation feature cannot be represented using a box plot because its output is binary, taking a value of either 1 (indicating the presence of reverberation) or 0 (indicating its absence).

Figure 2. A comparison between two segments extracted from genuine and fake speech waveforms.

Figure 3. Box plots of speech-pathological features derived from 1000 signals of both genuine (green) and fake (red) speech: (a) hardness, (b) depth, (c) brightness, (d) roughness, (e) warmth, (f) sharpness, (g) boominess, and (h) a bar chart of reverberation.

3.2. Segmental Frames of Analysis

Even though the average of speech-pathological features can help distinguish between genuine and deepfake speech, it may be insufficient. For example, if the difference between genuine and fake speech pertains to consistency, where approximately 70–80% of the speech is consistent but the remaining segments exhibit significant fluctuations, the average value becomes less meaningful. Therefore, rather than computing the average of speech-pathological features from the entire speech signal, as is typically done, we propose extracting these features using a segmented frame analysis approach.

The procedure for extracting perceptual speech-pathological features through segmented frame analysis is depicted in Figure 4. This process begins with receiving a speech signal, which is subsequently divided into M frames. The perceptual speech-pathological features are then extracted from each individual frame. The extraction continues sequentially from the first frame to the M-th frame, meaning the total number of perceptual speech-pathological features corresponds to the total number of frames processed.

Figure 4. Segmental frames of analysis of speech-pathological features.

Table 4 shows the effectiveness of the segmented frame analysis of perceptual speech-pathological features with Multi-Layer Perceptron Neural Network (MLP). Here, each feature is derived on a frame-by-frame basis, using a window frame of 1 s with an overlap of 50%. The data in frame 1 is fed into the feature extraction process, where feature values are obtained and stored in an array at position 1. This feature extraction process is then repeated for frames 2, 3, and so on until the last frame, which in this case is the 7th frame. Consequently, for each feature, the data for a single voice is located at position 7. The total number of segmented frames depends on the duration of the speech and the window frame size.

3.3. Classifier

There are three classifiers applied in this study.

3.3.1. Multi-Layer Perceptron Neural Network

The MLP used in this study consists of three layers: an input layer, a hidden layer, and a single-node output layer. The dimensions of the input and hidden layers are determined by the size of the input feature set. As shown in Table 3, the network configuration for individual features, from hardness to reverberation, is $(1, 1, 1)$, whereas the combination of all features results in an $(8, 8, 1)$ architecture. Similarly, in Table 4, the dimensions for individual features are $(7, 7, 1)$, while the combined feature set excluding reverberation uses a $(49, 49, 1)$ configuration. Reverberation is excluded from this combined set because it lacks efficacy in distinguishing between fake and genuine samples. For the analysis in Table 5, the dimensions are $(42, 42, 1)$. The hidden layer is activated using the Rectified Linear Unit (ReLU) function, while the output layer employs the sigmoid function. The classifier was trained for up to 100 epochs with a learning rate of $0.0001$ and a batch size of 128. A dropout rate of $0.2$ was applied to the final layers to mitigate overfitting. Binary cross-entropy served as the loss function, and the Adam optimizer was utilized. The MLP model architecture employed in this study is illustrated in Figure 5.

Figure 5. MLP architecture.

3.3.2. DNN

The DNN architecture used in this study consists of eight layers: an input layer, six hidden layers, and a single-node output layer. The hidden layers employ the ReLU activation function, while the output layer uses the sigmoid activation function. The input layer configuration depends on the input dimensions; for example, as shown in Table 6, a 1 s frame size results in a dimension of 49, while frame sizes of 500 ms and 250 ms result in dimensions of 105 and 217, respectively. The first and second hidden layers consist of 128 nodes each, while the third, fourth, fifth, and sixth hidden layers contain 64, 32, 16, and 8 nodes, respectively. A dropout rate of $0.2$ is applied to the first through third hidden layers to reduce overfitting. The structure of the DNN model applied in this study is shown in Figure 6. The training configurations are similar to those of the MLP model.

Figure 6. DNN architecture.

3.3.3. ResNet-18

A ResNet, employed as a classifier, represents a highly effective deep neural network architecture designed to mitigate the vanishing gradient problem, where gradients become excessively small during backpropagation [59]. Its robust performance has led to its widespread adoption in various audio and speech signal processing applications [60]. The function of the residual block, which integrates an intermediate input with the output of a sequence of convolutional layers, is formulated as

$$\mathbf{y}=\mathcal{F}\left(\mathbf{x}\right)+\mathbf{x},$$

(14)

where $\mathbf{x}$ and $\mathbf{y}$ are the input and output of the preceding layer, respectively, and $\mathcal{F}\left(\mathbf{x}\right)$ denotes the residual mapping consisting of multiple convolutional layers. Residual blocks are implemented across several layers ranging in number from 10 to over 100, with each layer containing a specific number of residual blocks. However, incorporating an excessive number of features was deemed unnecessary for this study. Therefore, the ResNet-18 model, consisting of 18 residual layers, was selected as the classifier. We conducted the training process over 100 epochs with a learning rate of $0.0001$ and a batch size of 32, utilizing the Adam optimizer. A dropout rate of $0.2$ is utilized for the final layers to reduce overfitting. Binary cross-entropy was adopted as the loss function to measure the discrepancy between the predicted and target values. The output score was derived from the activation of the “fake” node in the final fully connected layer, prior to applying the sigmoid function.

The hyperparameter settings for MLP, DNN, and ResNet-18 are summarized in Table 1. The implementation was conducted on a computer with an NVIDIA GeForce RTX 3090 graphics card with 24 GB of memory, an Intel Core i9-10920X CPU operating at $3.50$ GHz, and 128 GB of system memory.

Table 1. Hyperparameters of the classifiers.

Hyperparameter	MLP	DNN	ResNet-18
Layers	3	8	18
Epoch	100	100	100
Batch size	128	128	32
Optimizer	Adam	Adam	Adam
Learning rate	0.0001	0.0001	0.0001
Dropout rate	0.2	0.2	0.2

Table 1. Hyperparameters of the classifiers.

Hyperparameter	MLP	DNN	ResNet-18
Layers	3	8	18
Epoch	100	100	100
Batch size	128	128	32
Optimizer	Adam	Adam	Adam
Learning rate	0.0001	0.0001	0.0001
Dropout rate	0.2	0.2	0.2

4. Evaluation

This section presents the details of the datasets and evaluation metrics, followed by a comprehensive discussion of the results and findings.

4.1. Dataset and Metrics

The dataset and metrics are explained in the following sections.

4.1.1. Dataset

We used the dataset of the ASVspoof 2019 challenge to evaluate the performance of the proposed method. The ASVspoof 2019 dataset is a widely utilized benchmark for evaluating the resilience of automatic speaker verification (ASV) systems against spoofing attacks, specifically in distinguishing between bonafide (authentic) and spoofed (fake) speech. The Logical Access (LA) subset of ASVspoof 2019 [61,62] is divided into three parts: the training subset (approximately $20.85$%), which includes 8 male and 12 female speakers, contributing 2580 bonafide and 22,800 spoofed speech samples; the development subset (approximately $19.81$%), which features 4 male and 6 female speakers, with 2548 bonafide and 22,296 spoofed samples; and the evaluation subset (approximately $59.34$%), which expands to 21 male and 27 female speakers, offering 7355 bonafide and 63,882 spoofed samples, as shown in Table 2. This dataset provides a rich and diverse resource for advancing research in anti-spoofing technologies, enabling the development, validation, and evaluation of robust ASV systems.

Table 2. Number of utterances in the ASVspoof 2019 dataset.

Dataset	Splits	Number of Utterances
		Genuine	Spoofed	Total
ASVspoof 2019	Training	2580	22,800	25,380
	Development	2548	22,296	24,844
	Evaluation	7355	63,882	71,237

Table 2. Number of utterances in the ASVspoof 2019 dataset.

Dataset	Splits	Number of Utterances
		Genuine	Spoofed	Total
ASVspoof 2019	Training	2580	22,800	25,380
	Development	2548	22,296	24,844
	Evaluation	7355	63,882	71,237

4.1.2. Evaluation Metrics

Accuracy, F1-score, and EER are used to evaluate the proposed method.

Accuracy

Accuracy measures the proportion of correctly classified instances to the total number of instances:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}},$$

(15)

where TP is True Positive, TN is True Negative, FP is False Positive, and FN is False Negative.

F1-Score

The F1-score is the harmonic mean of Precision and Recall:

$$\mathrm{F}1\text{-}\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}},$$

(16)

where

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}},\mathrm{and}\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}.$$

Equal Error Rate (EER)

Equal Error Rate (EER) is the operating point at which the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). This is achieved by adjusting a decision threshold, which determines the boundary for classifying inputs as positive or negative:

$$\mathrm{EER}=\mathrm{FAR}\left(\mathrm{threshold}\right)=\mathrm{FRR}\left(\mathrm{threshold}\right),$$

(17)

where

$$\mathrm{FAR}={\displaystyle \frac{\mathrm{False}\mathrm{Accepts}}{\mathrm{Total}\mathrm{Negative}\mathrm{Instances}}},$$

$$\mathrm{FRR}={\displaystyle \frac{\mathrm{False}\mathrm{Rejects}}{\mathrm{Total}\mathrm{Positive}\mathrm{Instances}}}.$$

The threshold directly influences the rates of FAR and FRR. A lower threshold may reduce the FRR but increase the FAR, and vice versa. The EER is determined at the specific threshold where both rates are equal, providing a single metric to evaluate the balance between these errors in a system.

4.2. Results and Discussion

Table 3 highlights the performance of various averages of timbral features applied with a neural network on the ASVspoof 2019 development set, measured in terms of accuracy, F1-score, and EER. Among the features, depth achieves the highest accuracy of 80%, while brightness records the best F1-score of 87.96%. Hardness shows the lowest EER of 35.70%. Some features, such as warmth and boominess, exhibit balanced performance across metrics, whereas sharpness and roughness perform relatively poorly. Interestingly, reverberation achieves a high F1-score of 86.70% but suffers from a high EER of 48.23%.

These results indicate that individual timbral features capture different and complementary perceptual cues, with no single feature consistently outperforming others across all evaluation metrics. Combining all features leads to significant improvements, resulting in the best overall performance with an 82.03% accuracy, an F1-score of 89.89%, and the lowest EER of 32.37%, demonstrating the effectiveness of feature aggregation for improved spoof detection. While this performance remains below recent state-of-the-art deep feature-based systems, it establishes a meaningful perceptual baseline, which influences the advanced modeling strategies presented later.

Table 4 presents the results of using segmental frames to analyze perceptual speech-pathological features with neural networks on the ASVspoof 2019 development set. Among the individual features, sharpness achieves the best performance with an EER of 33.08%, followed closely by boominess and hardness. In contrast, reverberation fails entirely, with 0% accuracy and F1-score and an EER of 50%, indicating its ineffectiveness in this context.

Compared to average-based feature representations, the segmental analysis improves discrimination for most timbral features by capturing localized temporal variations that are characteristic of spoofed speech. In particular, sharpness and boominess benefit from frame-wise modeling, suggesting that temporal fluctuations in spectral emphasis are informative cues for imitation detection.

Table 3. Results of applying an average of timbral features with an MLP [63] on the development set of ASVspoof 2019.

Timbral Feature	Accuracy (%)	F1-Score (%)	EER (%)
Hardness	73.10	83.44	35.70
Depth	80.00	82.45	48.20
Brightness	78.91	87.96	47.68
Roughness	66.72	79.29	46.70
Warmth	72.61	83.20	39.29
Sharpness	63.41	76.58	48.03
Boominess	66.49	78.56	40.18
Reverberation	77.38	86.70	48.23
Combining all features	82.03	89.89	32.37

Table 3. Results of applying an average of timbral features with an MLP [63] on the development set of ASVspoof 2019.

Timbral Feature	Accuracy (%)	F1-Score (%)	EER (%)
Hardness	73.10	83.44	35.70
Depth	80.00	82.45	48.20
Brightness	78.91	87.96	47.68
Roughness	66.72	79.29	46.70
Warmth	72.61	83.20	39.29
Sharpness	63.41	76.58	48.03
Boominess	66.49	78.56	40.18
Reverberation	77.38	86.70	48.23
Combining all features	82.03	89.89	32.37

Table 4. Results of using segmental frames of analysis of perceptual speech-pathological features with an MLP [63] on the development set of ASVspoof 2019.

Timbral Feature	Accuracy (%)	F1-Score (%)	EER (%)
Hardness	89.77	94.61	37.56
Depth	85.25	91.97	48.82
Brightness	87.21	93.11	43.70
Roughness	88.40	93.81	43.09
Warmth	87.38	93.24	52.51
Sharpness	89.65	95.54	33.08
Boominess	89.48	94.40	33.50
Reverberation	0.00	0.00	50.00
Combining 7 features (except Reverberation)	93.98	96.70	13.82

Table 4. Results of using segmental frames of analysis of perceptual speech-pathological features with an MLP [63] on the development set of ASVspoof 2019.

Timbral Feature	Accuracy (%)	F1-Score (%)	EER (%)
Hardness	89.77	94.61	37.56
Depth	85.25	91.97	48.82
Brightness	87.21	93.11	43.70
Roughness	88.40	93.81	43.09
Warmth	87.38	93.24	52.51
Sharpness	89.65	95.54	33.08
Boominess	89.48	94.40	33.50
Reverberation	0.00	0.00	50.00
Combining 7 features (except Reverberation)	93.98	96.70	13.82

When all features are combined and fed to the MLP, the model achieves the highest overall performance, with 93.98% accuracy, a 96.70% F1-score, and the lowest EER of 13.82%, highlighting the advantage of integrating features for spoof detection. This improvement confirms the complementary nature of perceptual features in the segmental domain and supports the effectiveness of feature fusion prior to more advanced modeling strategies.

Table 5 reports the results of an ablation study evaluating the contribution of individual speech-pathological features to the performance of an MLP model applied to the ASVspoof 2019 development set. Among all the features, depth emerges as the most crucial, as its exclusion leads to the lowest accuracy of $90.09$% and a notable increase in EER of $14.42$%. Similarly, removing sharpness significantly affects performance, resulting in the lowest F1-score of $95.86$% and the highest EER of $18.60$%, highlighting its importance in maintaining precision. Features like roughness and warmth also contribute substantially, with their exclusion increasing EER to $18.29$% and $17.43$%, respectively, whereas features such as brightness and hardness show relatively minor impacts when excluded. The study uses an MLP model, demonstrating its sensitivity to specific features for detecting spoofed audio. The results underscore that certain features are more integral to the model’s performance, with the exclusion of key features like depth and sharpness causing pronounced drops, whereas the model is more resilient to the removal of others. This highlights the importance of feature selection in optimizing neural network-based audio spoof detection.

Table 5. Ablation study of segmental frames of analysis of perceptual speech-pathological features with an MLP [63] on development set of ASVspoof 2019.

Excluded Feature	Accuracy (%)	F1-Score (%)	EER (%)
Hardness	93.28	96.33	13.82
Depth	90.09	96.23	14.42
Brightness	93.33	96.35	13.85
Roughness	92.72	96.04	18.29
Warmth	92.66	96.00	17.43
Sharpness	92.39	95.86	18.60
Boominess	92.85	96.10	15.95

Table 5. Ablation study of segmental frames of analysis of perceptual speech-pathological features with an MLP [63] on development set of ASVspoof 2019.

Excluded Feature	Accuracy (%)	F1-Score (%)	EER (%)
Hardness	93.28	96.33	13.82
Depth	90.09	96.23	14.42
Brightness	93.33	96.35	13.85
Roughness	92.72	96.04	18.29
Warmth	92.66	96.00	17.43
Sharpness	92.39	95.86	18.60
Boominess	92.85	96.10	15.95

Table 6 evaluates the impact of different segmental frame sizes for analyzing speech-pathological features for spoof detection on the evaluation set of ASVspoof 2019. Larger frames, such as 1 s, capture more comprehensive feature information, achieving the highest accuracy of $89.17$% and F1-score of $97.10$%. Shorter frames, like 500 ms, strike a balance, yielding the lowest EER of $17.47$% while maintaining a competitive accuracy of $88.34$% and an F1-score of $93.60$%. The smallest frame, 250 ms, offers finer temporal resolution but performs slightly worse, with an EER of $19.08$% and a moderate F1-score of $93.77$%.

These results highlight that the DNN model adapts to varying frame sizes with consistent performance across metrics. Longer frames, such as 1 s, enhance feature extraction and maximize accuracy and precision, whereas shorter frames, like 500 ms, provide a trade-off by reducing EER, possibly due to improved ability to capture fine-grained temporal variations within each segment. These findings suggest that the choice of frame size is crucial for optimizing model performance based on application requirements, balancing accuracy and error rates effectively.

To investigate further, we evaluated various methods using different features, classifiers, and frame dimensions on the ASVspoof 2019 evaluation set, as shown in Table 7. Initially, all seven timbral features with a frame size of 1 s and a dimension of $7\times 7$ were analyzed using a DNN, achieving an accuracy of $89.17$%, an F1-score of $94.15$%, and an EER of $17.94$%. To refine the approach, the three best-performing features—hardness, depth, and sharpness—were selected with the same frame size but reduced dimensions of $3\times 7$, resulting in a slightly higher accuracy of $89.5$% but an increased EER of $20.99$%, indicating a trade-off in feature reduction. Additionally, using a gammatone filterbank with larger dimensions of $32\times 401$ and a ResNet-18 classifier significantly improved performance, yielding an accuracy of $96.20$%, an F1-score of $97.88$%, and an EER of $6.63$%. Finally, the proposed method combines a gammatone filterbank with three timbral features, utilizing their respective models, ResNet-18 and a DNN, through score fusion, as shown in Figure 7. In this strategy, the gammatone filterbank with ResNet-18 serves as the primary model, while the timbral features with the DNN constitute the secondary model. If the prediction score exceeds a threshold of $0.5$, it is directly considered the final decision. However, if the score falls below the threshold, the final score is determined using a weighted average of 70% from the primary model and 30% from the secondary model. The results indicate that the proposed approach achieved a superior performance, with an accuracy of $96.78$%, an F1-score of $98.20$%, and a lowest EER of $5.93$%, thus demonstrating the effectiveness of integrating advanced classifiers with refined feature representations.

Figure 7. The proposed method, combining (1) three pathological segment features with DNN and (2) gammatone filterbank with ResNet-18, through score fusion.

Table 6. Results of using segmental frames of analysis of perceptual speech-pathological features with DNN with different frame sizes on the evaluation set of ASVspoof 2019.

Frame Size	Dimension	Accuracy (%)	F1-Score (%)	EER (%)
1 s	7 × 7	89.17	97.10	17.94
500 ms	7 × 15	88.34	93.60	17.47
250 ms	7 × 31	88.70	93.77	19.08

Table 6. Results of using segmental frames of analysis of perceptual speech-pathological features with DNN with different frame sizes on the evaluation set of ASVspoof 2019.

Frame Size	Dimension	Accuracy (%)	F1-Score (%)	EER (%)
1 s	7 × 7	89.17	97.10	17.94
500 ms	7 × 15	88.34	93.60	17.47
250 ms	7 × 31	88.70	93.77	19.08

Table 7. Comparison of seven timbral features, three selected timbral features, and the gammatone filterbank.

No	Feature Set	Model	Accuracy (%)	F1-Score (%)	EER (%)
1	Seven Timbral ($7\times 7$)	DNN	89.17	94.15	17.94
2	Hardness + Depth + Sharpness ($3\times 7$)	DNN	89.50	94.14	20.99
3	Gammatone filterbank ($32\times 401$)	ResNet-18	96.22	97.88	6.63
4	Proposed Method	Score Fusion Using DNN and ResNet-18	96.78	98.20	5.93

Table 7. Comparison of seven timbral features, three selected timbral features, and the gammatone filterbank.

No	Feature Set	Model	Accuracy (%)	F1-Score (%)	EER (%)
1	Seven Timbral ($7\times 7$)	DNN	89.17	94.15	17.94
2	Hardness + Depth + Sharpness ($3\times 7$)	DNN	89.50	94.14	20.99
3	Gammatone filterbank ($32\times 401$)	ResNet-18	96.22	97.88	6.63
4	Proposed Method	Score Fusion Using DNN and ResNet-18	96.78	98.20	5.93

4.3. Comparison with ASVspoof 2019 Evaluation Method

The comparison with related works in Table 8 shows that the proposed method achieves competitive performance on the ASVspoof 2019 dataset, particularly when compared with classical and recent deep learning-based baselines. With an EER of 1.33% on the development set and 5.93% on the evaluation set, the proposed approach consistently outperforms traditional cepstral feature-based systems such as CQCC- and LFCC-GMM models, as well as several transformer-based and CNN-based methods evaluated by the same benchmark. Although some state-of-the-art approaches leveraging high-dimensional spectral representations achieve comparable or slightly lower EERs, the proposed method offers a distinct advantage by relying on perceptually motivated, low-dimensional timbral features. This comparison highlights that incorporating auditory-inspired representations, together with segmental-wise analysis and hybrid modeling, provides a robust and interpretable alternative for deepfake speech detection, complementing existing state-of-the-art systems rather than replacing them.

Table 8. Comparison of the proposed method with related works.

Method	Features	Models	Dev (EER %)	Eval (EER %)
[64]	CQCC	GMM	0.43	9.57
[64]	LFCC	GMM	2.71	8.09
[65]	Multi-feature approaches Learning-based	Transformer	–	24.27
[66]	Log spectrogram	Transformer	–	17.51
[67]	First digit distribution	Transformer	–	16.37
[68]	Spectrogram images	Constant-vision transformer	–	11.02
[69]	eCQCC	DNN	0.00	11.08
[70]	Spectrogram	CNN	0.00	9.57
[71]	STM (ERB FB)	LCNN	0.02	8.33
[72]	STFT	VGG-Transformer	0.06	8.29
[73]	Logspec	SENet34	–	6.70
[4]	CQCC	Siamese CNN	0.15	8.75
Proposed method	Timbral segmental-wise with Gammatone filterbank	DNN and ResNet-18	1.33	5.93

Table 8. Comparison of the proposed method with related works.

Method	Features	Models	Dev (EER %)	Eval (EER %)
[64]	CQCC	GMM	0.43	9.57
[64]	LFCC	GMM	2.71	8.09
[65]	Multi-feature approaches Learning-based	Transformer	–	24.27
[66]	Log spectrogram	Transformer	–	17.51
[67]	First digit distribution	Transformer	–	16.37
[68]	Spectrogram images	Constant-vision transformer	–	11.02
[69]	eCQCC	DNN	0.00	11.08
[70]	Spectrogram	CNN	0.00	9.57
[71]	STM (ERB FB)	LCNN	0.02	8.33
[72]	STFT	VGG-Transformer	0.06	8.29
[73]	Logspec	SENet34	–	6.70
[4]	CQCC	Siamese CNN	0.15	8.75
Proposed method	Timbral segmental-wise with Gammatone filterbank	DNN and ResNet-18	1.33	5.93

5. Conclusions

This study tackled the critical challenge of detecting deepfake speech by emphasizing the segmental frame-wise analysis of speech-pathological features, such as timbral attributes, to identify nuanced and localized differences between genuine and synthetic speech. By integrating these perceptual attributes with a gammatone filterbank that simulated human auditory processing, the proposed method significantly enhanced detection performance. Experimental results on the ASVspoof 2019 dataset validated the effectiveness of this approach, with the combination of ResNet-18 and DNN models achieving an Equal Error Rate (EER) of $5.93$%, outperforming baseline methods. These findings highlight the importance of segmental analysis in improving the robustness of deepfake detection systems.

Despite the promising results, several limitations remain. Some perceptual features, particularly reverberation, showed limited effectiveness in their current binary formulation. In addition, the frame-wise extraction of perceptual timbral features increases computational overhead, which may hinder real-time or resource-constrained deployment. Future work will address these limitations by extending experiments to cross-corpus, multilingual, and real-world datasets, refining perceptual feature representations, and incorporating additional auditory-inspired features. Moreover, lightweight architectures and adaptive fusion strategies will be explored to reduce computational cost while maintaining detection performance, thereby improving robustness and practical applicability.