MFF-Net: Multi-Feature Fusion Network for Speech Deception Detection

Abstract

Speech deception detection often suffers from overly narrow feature representations and limited robustness under changing recording conditions. To address these limitations, we propose MFF-Net, a multi-feature fusion network with three modules. The first module uses a Gated-BiLSTM with attention to model sequences of acoustic-prosodic (ACO) and nonlinear dynamic (NLD) features. The second module uses a hybrid Transformer–CNN branch on Mel spectrograms, with a Levenberg–Marquardt (LM) refinement stage for the Mel-branch classifier. The third module fuses the two branches via cross-attention before a softmax decision. On the English Real-Life Trial Deception Database (RTDD), MFF-Net achieves 79.22% accuracy and 77.69% F1. On our Chinese Lie Dataset (CLD), it reaches 78.29% accuracy and 79.13% F1, outperforming strong baselines. Together with speaker-aware evaluation and clarified training objectives, the results support MFF-Net as a reproducible approach for deception detection across complementary corpora.

1. Introduction

Deception is common in social interaction, yet humans often judge truthfulness intuitively, which invites errors with serious consequences. Lying can erode trust and fairness across judicial proceedings, public safety, commerce, and personal relationships. Voice-based deception detection offers a non-contact way to complement human judgment by exploiting acoustic and prosodic cues, with applications in interrogation, screening, verification, and trust assessment [1,2,3].

In recent years, with advancements in artificial intelligence and machine learning, speech deception detection technology, as a non-invasive deceptive detection method, has gained widespread attention and achieved remarkable progress. This technology’s development relies on interdisciplinary research, including psychology, acoustics, and computational linguistics. The theoretical basis stems from the discovery that lying is often accompanied by physiological changes, which can manifest in features of the speech signal, such as fluctuations in acoustic features like pitch, fundamental frequency, speech rate, and sound intensity [4]. Furthermore, psychological studies also indicate a close correlation between emotional fluctuations and deceptive behaviors, providing theoretical support for emotion recognition in speech deception detection.

Technologically, deception detection can be divided into traditional feature extraction methods and deep learning techniques. Early speech deception detection primarily relied on handcrafted acoustic features, using machine learning methods like Support Vector Machines (SVM) to analyze changes in these features for lie identification. However, with the development of deep learning and neural networks, especially the widespread application of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), the accuracy of speech deception detection has significantly improved. Among them, Long Short-Term Memory (LSTM) networks have unique advantages in processing time-series data, capturing dynamic feature changes in speech signals, thereby enhancing the ability to identify deceptive behavior [5].

Additionally, multimodal analysis has gradually become a research hotspot in speech deception detection. This technology not only relies on speech signals but also combines other physiological or behavioral features (such as facial expressions, eye movements, and body gestures) [6], further improving deception detection accuracy through the fusion of multimodal data. This research is based on the core hypothesis that deceptive behavior is typically accompanied by a series of comprehensive physiological, emotional, and behavioral reactions, thus enabling more accurate deception identification through multi-dimensional signal capture.

Despite progress in various aspects, speech deception detection technology still faces numerous challenges in practical applications. Firstly, due to individual differences in speech data, speech features vary greatly among different speakers, leading to insufficient model generalization across populations. Secondly, in complex real-world environments, factors like noise, echo, or environmental interference adversely affect the analysis accuracy of speech signals. Furthermore, traditional methods mainly focus on static prosodic features of speech; single and fixed features may affect the behavior of the detected object. Aiming at these problems, this paper proposes a Multi-Feature Fusion-Based Network (MFF-Net), which consists of three modules. The first module combines the Gated-BiLSTM with an Attention mechanism. This module processes static prosodic features and nonlinear dynamic features. Static prosodic features encompass various speech qualities like intonation, volume, and speech rate, often used to identify the speaker’s psychological state and are fundamental information in speech-related studies. Nonlinear dynamic features originate from the nonlinear dynamic characteristics of speech signals, possess time-series properties, and are less sensitive to noise. The second module integrates the LM optimization algorithm into the Transformer-CNN structure, replacing traditional optimizers for faster convergence and avoiding local optima, thus enhancing the original model’s performance. The final module performs cross-attention-based feature fusion on the two types of high-level sequential features obtained from the previous steps, forming a complete collective that allows complementary information from different feature types. Experiments were conducted on both the public English judicial scenario dataset RTDD and a self-constructed Chinese deception dataset CLD. Results show that MFF-Net demonstrates better accuracy and generalization in both scenarios. The main contributions of this paper include:

1.

Innovatively proposing a multi-feature fusion-based speech deception detection network, which deeply integrates traditional handcrafted features with Mel-spectrogram features extracted by deep learning methods, balancing the static characteristics, dynamic nonlinear characteristics, and deep time-frequency patterns of speech for complementary advantages.

2.

Designing a heterogeneous feature adaptive processing strategy, namely using the Gated-BiLSTM-Attention module to parse the sequential patterns of handcrafted features, and utilizing the Transformer-CNN-LM module to mine the spatiotemporal features of Mel-spectrograms.

3.

Constructing a high-quality Chinese deception dataset (CLD) in a laboratory setting. This dataset employs both intentional lying and interactive lying paradigms with a standardized collection process and a fine-grained annotation strategy combining sentence-level and key information points, providing an important resource for studying Chinese speech features in controlled environments and effectively complementing datasets in other languages.

4.

Evaluating MFF-Net on two complementary corpora—English courtroom-style RTDD clips and laboratory-collected Mandarin CLD—under speaker-aware splits, where MFF-Net consistently outperforms strong baselines.

2. Related Work

Speech deception detection technology, as a frontier field intersecting computational psychology and audio signal processing, has evolved from shallow models relying on handcrafted features to data-driven deep learning methods, and further to the integration of multimodal information. This section systematically reviews related research, categorizing it into three types based on technical paradigms: traditional method-based lie detection, deep learning-based lie detection, and multimodal-based lie detection.

2.1. Traditional Method-Based Lie Detection

The core paradigm of traditional handcrafted feature-based lie detection involves extracting predefined acoustic features from speech signals and classifying them using statistical models. The development of this path is closely centered around the evolution of feature engineering and the promotion of public datasets. The scientific basis stems from Ekman’s “Emotional Leakage Theory”, which hypothesizes that deceptive behavior triggers involuntary physiological leaks, including changes in voice [7]. Guided by this theory, early research systematically explored acoustic cues related to deception. For example, Hirschberg et al. showed significant differences in prosodic features like fundamental frequency, speech rate, and utterance duration between deceptive and non-deceptive speech [8]. Besides prosodic features, voice quality features stemming from microtremors of vocal organ muscles (e.g., jitter, shimmer) and nonlinear dynamic features (e.g., fractal dimension, recurrence quantification analysis features) have also proven to be effective discriminative indicators [9]. For instance, the 18-dimensional nonlinear dynamic features proposed by Professor Zhao Heming’s team at Soochow University achieved a recognition rate of 70.7% in deception detection tasks [10]. Furthermore, researchers have explored more diverse acoustic features, such as phase-based features [11] and formant features reflecting vocal tract tuning changes [12]. Research standardization benefited from the construction of large-scale lie databases, such as the Columbia-SRI-Colorado (CSC) database built by Hirschberg et al., which achieved about 70% accuracy using acoustic-prosodic and lexical features [8]. Subsequently, the Interspeech 2016 ComParE Deception Sub-Challenge and its provided DSD database further promoted research standardization. Levitan et al. achieved about 69.4% accuracy on the test set by fusing the acoustic baseline feature set and text emotion features [13]. In automatic speaker verification anti-spoofing, large-scale shared tasks such as ASVspoof popularized strong cue-based baselines (e.g., MFCC/CQCC paired with GMMs or SVMs) and shaped subsequent deep model comparisons [14]. To improve model performance, subsequent research focused on optimizing feature selection and model fusion, such as using recursive feature elimination algorithms to select feature subsets [15], or introducing ensemble learning models like Gradient Boosting Decision Trees [16]. Recently, researchers have also attempted to optimize traditional features from the perspective of noise robustness. For example, Wang et al. proposed an improved noise-invariant feature extraction method that enhanced detection stability in noisy environments [17]. However, the fundamental limitation of traditional methods lies in their performance, heavily relying on expert-designed features, making it difficult to capture subtle, high-order nonlinear patterns of deception, and their generalization ability is limited when facing complex environmental noise and unknown speakers [18].

2.2. Deep Learning-Based Lie Detection

The rise of deep learning technology has fundamentally changed the research paradigm of speech deception detection, shifting it from reliance on handcrafted features to end-to-end representation learning. Its evolution path can be traced from feature learning and temporal modeling to current pre-training and generalization. Early research focused on using deep learning models to learn more robust and deeper representations from handcrafted features or raw signals. For example, Zhou et al. proposed a model based on the K-SVD algorithm and Deep Belief Networks, obtaining deep features through sparse coding [19]; Fu et al. used Denoising Autoencoders to reduce the dimensionality and reconstruct high-dimensional handcrafted features, improving model robustness with limited labeled samples [20]. With the development of sequence modeling techniques, the research focus shifted to end-to-end architectures capable of directly capturing the temporal dynamics of speech. Recurrent Neural Networks and their variants, like Long Short-Term Memory networks, were widely adopted. For instance, Fang et al. combined Autoencoders, Bidirectional LSTM, and an attention mechanism to construct a semi-supervised model focusing on key speech segments [21]. In the ASVspoof 2017 and subsequent challenges, deep models based on CNN, Residual Networks, and RNN comprehensively outperformed the traditional GMM baseline, establishing the dominance of deep learning [14]. Simultaneously, more network architectures were introduced, such as the Convolutional LSTM network studied by Sahidullah et al. for replay attack detection [22]. In recent years, self-supervised pre-trained models based on the Transformer architecture have become mainstream. For example, Liu et al. fine-tuned the Wav2Vec 2.0 model, pre-trained on large-scale unlabeled data, for speech spoofing detection tasks, achieving excellent performance and proving the effectiveness of learning universal speech representations [23]. To more effectively model the temporal dynamics of speech signals, spatiotemporal convolutional architectures like S3D networks have also been applied to audio spoofing detection [24]. Current cutting-edge research aims to address challenges like data scarcity and unknown attacks, actively exploring the use of contrastive learning to leverage unlabeled data [25], and adopting paradigms like one-class learning to enhance model generalization robustness against unknown spoofing types [26]. Meanwhile, research on model efficiency and interpretability is also deepening. For example, Zhang et al. proposed a lightweight multi-scale temporal convolutional network that significantly reduced computational overhead while maintaining high accuracy [27], and defensive research against adversarial attacks is gaining attention [28].

Critical perspective. Self-supervised front ends (e.g., Wav2Vec 2.0 [23]) excel when large unlabeled corpora are available and the target domain matches pre-training, but they can be data-hungry and less transparent for courtroom-style deception cues that are often studied with curated paralinguistic descriptors. Likewise, spoofing-focused benchmarks (e.g., ASVspoof [14]) emphasize attack types and metrics that are not always aligned with human deceptive speech in interviews or trials. These gaps motivate architectures that explicitly retain interpretable handcrafted cues while still exploiting deep time–frequency representations.

2.3. Multimodal-Based Lie Detection

Deception is a complex psychophysiological behavior whose clues are simultaneously reflected in multiple channels such as speech, facial expressions, eye movements, posture, and physiological signals [7]. Therefore, fusing multimodal information for joint analysis is considered an inevitable approach to achieving high-accuracy, high-robustness deception detection. Early research validated the effectiveness of fusing information from speech and facial micro-expressions [29]. With the development of deep learning, fusion methods based on attention mechanisms have become mainstream. For example, Gallardo-Antolin and Montero proposed a multimodal LSTM framework based on an attention mechanism to dynamically weight and fuse information from gaze behavior and speech, effectively improving detection performance [30]. The core of current research lies in achieving deep cross-modal interaction. Cross-modal Transformer architectures, with their powerful global attention mechanism, can model fine-grained alignment and dependency relationships between modalities. The model proposed by Chen et al. achieved deep fusion of audio, visual, and textual information through a cross-attention module [31]. To efficiently utilize large-scale pre-trained foundation models, parameter-efficient fine-tuning techniques are widely adopted, requiring training only a small number of parameters to adapt the model to downstream multimodal deception detection tasks, significantly reducing computational costs [32]. Furthermore, exploring more advanced fusion strategies is also a focus, such as graph neural network-based multimodal fusion methods [33] and multi-level hybrid feature fusion networks [34]. The vigorous development of this field benefits from the establishment of high-quality multimodal datasets, such as DOLOS [35] and SEUMLD [36], which provide critical support for training and evaluating complex models. Currently, multi-task learning and robust fusion strategies insensitive to modal missingness are important research trends aimed at further improving the practicality of systems in real complex scenarios [37]. Additionally, the latest research begins to explore more interpretable fusion mechanisms, such as visualizing attention weights to understand model decision basis [38], and exploring the fusion of physiological signals (e.g., ECG, GSR) with behavioral modalities to uncover deeper stress and cognitive load clues [39].

Although speech deception detection technology has made significant progress, existing research still faces a series of common challenges and limitations. Firstly, traditional machine learning-based methods, while having the advantage of strong interpretability, face performance bottlenecks due to heavy reliance on expert-experience-driven feature engineering, making it difficult to adaptively capture the complex and subtle nonlinear patterns in deceptive behavior, with limited generalization ability [7,8,9,10,11,12,13]. Secondly, deep learning-based methods, through end-to-end learning, break through the limitations of feature design, but their performance heavily depends on large-scale, high-quality annotated data, and the scarcity of deception data restricts their development [14,15,16,17]. Finally, multimodal fusion-based methods can theoretically provide more comprehensive clues, but their development is constrained by the lack of high-quality, scenario-specific multimodal datasets; meanwhile, how to design efficient fusion architectures to fully exploit the complementary and synergistic information between modalities, rather than simply concatenating information, and reducing the computational and deployment costs of models, remains a core problem to be solved [24,25,26,27]. Addressing the above limitations, especially the effectiveness of feature representation and model generalization ability, the MFF-Net network proposed in this paper focuses on the deep fusion of traditional handcrafted features and deep features, aiming to combine the clear physical meaning of traditional features with the powerful representational capacity of deep learning to improve the detection performance and robustness of the model under different deception scenarios. The research in this paper constitutes a beneficial exploration and supplement to the limitations of existing speech deception detection technologies.

3. Materials and Methods

3.1. Design Rationale and Comparison to Alternatives

Speech deception cues are expressed across complementary representations: high-level prosodic statistics and nonlinear dynamics (ACO/NLD) offer interpretable, domain-established descriptors, whereas Mel time–frequency patterns capture finer texture that CNNs/Transformers can exploit. A single-stream end-to-end model (e.g., waveform CNN or spectrogram Transformer alone) can underuse the inductive biases that have proven useful in deception and paralinguistics benchmarks, while only using handcrafted functionals discards non-local spectro-temporal regularities. We therefore adopt a dual-branch design: (i) a sequence model with gating and attention for ACO+NLD, following evidence that temporal weighting helps deception-related segments [21,40]; (ii) a parallel Transformer–CNN branch for Mel features to combine local convolutional structure with long-range attention [23,24]; and (iii) cross-attention fusion instead of fixed concatenation or voting, so each modality can query the other before the final decision. Simpler fusion rules are included later as ablations (hard voting) to quantify the gain from learned interaction.

This paper designs a speech deception detection network based on multi-feature fusion, effectively combining traditional acoustic features with deep learning methods to mine acoustic, dynamic, and time-frequency discriminative information related to deception from speech signals. It contains three modules, as shown in Figure 1. The Gated-BiLSTM-Attention module performs context-aware sequence modeling on acoustic and nonlinear dynamic features, enhancing the model’s ability to capture temporal evolution patterns in speech. The Transformer-CNN-LM hybrid module jointly mines local structures and global temporal dependencies from Mel-spectrograms, improving the discriminative power and robustness of feature representation. In the final multi-feature fusion module, a cross-attention based feature fusion mechanism is designed to achieve deep interaction of heterogeneous features, rather than simple weighted fusion, and the Softmax classifier is used to complete the classification decision between truth and deception. Before describing each module in detail, the following definitions are given:

Let $X=\{{\mathcal{X}}_A,{\mathcal{X}}_N,{\mathcal{X}}_M\}$ represent a multi-feature input sample, where ${\mathcal{X}}_A$, ${\mathcal{X}}_N$, ${\mathcal{X}}_M$ denote the acoustic-prosodic features, nonlinear dynamic features, and Mel-spectrogram features extracted from the speech signal, respectively. The feature data sequence can be represented as ${\mathcal{X}}_i=\{{\mathcal{X}}_{i,1},{\mathcal{X}}_{i,2},\dots ,{\mathcal{X}}_{i,n}\}$, where $i\in \{A,N,M\}$, and n represents the feature dimension. The goal is to predict the category of the sample based on its deceptive content. The data label is $y\in \{y_0,y_1\}$, where $y_0$ represents a lie, and $y_1$ represents the truth. Finally, the embedded feature vectors ${\mathcal{F}}_X$ for the three fused features are obtained, where $X\in \{A,N,M\}$. The predicted label $\overline{y}$ is obtained as follows:

$$\overline{y}=\mathcal{M}(F_A,F_N,F_M)$$

(1)

where $\mathcal{M}$ is the fusion function, and $y\in \{y_0,y_1\}$.

3.2. The Gated-BiLSTM-Attention Module

The design of this module is inspired by the Convolutional Bidirectional Long Short-Term Memory network (CovBiLSTM) in the literature [40]. This paper selects emotionally representative prosodic features from prosodic features, including Mel-Frequency Cepstral Coefficients (MFCC), Bark Band Energy, and pitch, among others. Additionally, this paper uses the box-counting method to extract 15-dimensional nonlinear dynamic features from speech and concatenates them with the prosodic features. The core function is to perform deep sequential pattern parsing on the handcrafted acoustic-prosodic features (ACO) and nonlinear dynamic features (NLD), thereby extracting high-level sequential features related to deception. The module structure is innovatively enhanced based on the BiLSTM-Attention framework by introducing a secondary gating mechanism to improve the model’s ability to capture key information.

Prosodic features reflect the static characteristics of the speech signal over short time intervals (20 ms) [41], generally analyzed using short-time analysis methods. This paper uses the OPENSmile toolkit to extract the INTERSPEECH 2016 ComParE Challenge and eGeMAPSvO2 feature sets at the functionals level. The resulting integrated feature set contains a large number of static features, specifically including low-level descriptors (LLDs) such as loudness, pitch, energy distribution, and spectral bands, as well as various statistical functions used to characterize the dynamic changes of LLDs, including five categories: central tendency, dispersion, distribution shape, order statistics, and regression analysis, totaling 6461 static features.

In the analysis of nonlinear dynamic features of speech signals, the Box-Counting Method is a classic computational method based on fractal geometry, used to quantify the complexity and irregularity of signal waveforms [42]. This method analyzes the self-similarity characteristics of the speech signal at different scales to extract its Fractal Dimension, thereby revealing the implicit nonlinear dynamic behavior in the signal. Its core idea is to cover the signal waveform with grids of different sizes, count the minimum number of boxes required for coverage, and then estimate the fractal dimension through a power-law relationship. The specific method is as follows:

First, long speech segments need to be divided into short-term speech signals and converted into time-series waveforms; this step also requires preprocessing like denoising and normalization to eliminate effects like baseline drift and high-frequency interference. Second, the waveform is treated as a curve in a two-dimensional plane (time-amplitude), covered by a grid of squares with side length $\epsilon$. Then, gradually reduce the box size $\epsilon$ (e.g., sequentially take $\epsilon =2^{-k},k=1,2,\dots$), and count the minimum number of boxes $N\left(\epsilon \right)$ required to cover the curve at each scale. According to fractal theory, $N\left(\epsilon \right)$ and $\epsilon$ satisfy a power-law relationship: $N\left(\epsilon \right)\propto {\epsilon }^{-D}$. After taking the logarithm of both, use linear regression to fit the data points of $logN\left(\epsilon \right)$ versus $log(1/\epsilon )$; the slope of the resulting line is the estimated value of the fractal dimension D. The calculation formula is:

$$D=\underset{\epsilon \to 0}{lim}{\displaystyle \frac{log\left(N\right(\epsilon \left)\right)}{log(1/\epsilon )}}$$

(2)

The fractal dimension D obtained by the box-counting method directly reflects the nonlinear dynamic characteristics of the speech signal. A higher D value indicates a more complex and irregular signal waveform, possibly corresponding to nonlinear phenomena like turbulence and vortices in speech. The fractal dimension describes the complexity and self-similarity dimension of the system’s motion trajectory in phase space, capable of capturing the chaotic characteristics caused by physical mechanisms such as glottal airflow and vocal tract wall vibrations during speech production, effectively complementing linear features (like formants). Finally, this paper extracts 15 nonlinear features including fractal features, Lyapunov exponent, and K-entropy. Compared to the emotional cues captured by prosodic features, NLD features focus more on cues related to cognition, memory, and strategic communication, enhancing the system’s ability to model nonlinear dynamic behavior.

The combined features obtained using the above method are dimensionality-reduced through a max-pooling layer and then input to the Gated-BiLSTM. The “memory” of LSTM is also called cells. These cells determine which previous information and state need to be retained or remembered and which need to be erased, effectively preserving relevant information from much earlier times. Meanwhile, through the three unique gate structures of LSTM: forget gate, input gate, and output gate, the cell state is protected and controlled. The forget gate decides what information to discard from the cell state, the input gate determines what new information to store in the cell state, and the output gate decides which part of the cell state will be output. The formulas are as follows:

$$f_t=\sigma \left(W_f·\left[h_{t-1},x_t\right]+b_f\right)$$

(3)

$$C_t=f_t\ast C_{t-1}+i_t\ast {\tilde{C}}_t$$

(4)

$$o_t=\sigma \left(W_o\left[h_{t-1},x_t\right]+b_o\right)$$

(5)

where $h_{t-1}$ and $x_t$ represent the output from the previous time step and the input at the current time step, $\sigma$ is the activation function, W and b are the corresponding weights and biases, ${\tilde{C}}_t$ represents the new candidate value vector, and $o_t$ represents the output gate controlling the output of the current hidden state. To overcome the limitations of traditional BiLSTM in regulating information flow in deep networks, this paper introduces a gating enhancement mechanism. This mechanism adds an information control gate on top of the standard LSTM’s forget, input, and output gates to achieve fine-grained regulation of cell state updates. The final hidden state is:

$$h_t=o_t\odot tanh\left(C_t\right)$$

(6)

where ⊙ denotes element-wise multiplication.

Given that the primary task of this module is to evaluate and focus on the most discriminative segments from the temporal sequences of acoustic-prosodic and nonlinear dynamic (NLD) features—rather than to model long-range contextual dependencies—a context-based additive attention mechanism is judiciously employed for the weighted fusion of the Gated-BiLSTM outputs. This choice is motivated by the mechanism’s computational efficiency and directness in highlighting critical information.

First, the attention score for each time step is calculated. A learnable context vector interacts with a non-linear projection of the hidden state to produce an unnormalized score:

$$e_t=v^T·tanh\left(W_a·h_t+b_a\right)$$

(7)

where $v^T$ is a learnable weight vector, $W_a$ is a weight matrix, and $b_a$ is a bias term. The attention scores are then normalized across all time steps using the Softmax function to obtain the attention weights ${\alpha }_t$:

$${\alpha }_t={\displaystyle \frac{e^{e_t}}{{\sum }_{j=1}^T{e}^{e_j}}}$$

(8)

Subsequently, a fixed-dimensional context vector r is obtained by performing a weighted sum of the hidden states of all time steps:

$$r=\sum _{t=1}^T{\alpha }_t{h}_t$$

(9)

Finally, this module outputs the weighted feature vector $F_{AN}$. This vector not only integrates the temporal dynamic information from the prosodic feature sequence $F_A$ and the NLD feature sequence $F_N$ but also, due to the gating enhancement and the attention mechanism, accentuates the subtle variations in key segments of speech deception. Thereby, it forms a higher-level feature representation with stronger discriminative power, providing a solid foundation for subsequent multi-feature fusion and classification.

The entire module combines the fine-grained modeling capability of Gated-BiLSTM for long-range dependencies with the focus ability of the attention mechanism on key information, effectively enhancing the discriminativity and robustness of the feature representation. The process of this module is shown in Figure 2.

3.3. Transformer-CNN-LM Module

For a normal signal $X\left(t\right)$, we typically seek to understand it from both time and frequency domain perspectives. In the feature extraction process of speech signals, the Mel-spectrogram is a key time-frequency representation method. Its calculation process is shown in Figure 3. First, pre-emphasis is applied to the original speech signal to compensate for the attenuation of high-frequency components during transmission, thereby boosting high-frequency energy and making the overall spectrum flatter. Subsequently, the pre-emphasized signal is segmented into consecutive short-time frames, and a window function (e.g., Hamming window) is applied to each frame to reduce spectral leakage caused by signal truncation. Next, the Fast Fourier Transform (FFT) is performed on each framed, windowed signal, converting it from the time domain to the frequency domain. This process is also known as the Short-Time Fourier Transform (STFT), mathematically expressed as:

$$X(t,f)={\int }_{-\infty }^{\infty }x\left(\tau \right)w(t-\tau )e^{-j2\pi f\tau }d\tau$$

(10)

where $x\left(t\right)$ is the original signal, $w(t-\tau )$ is the window function, t represents time, and f represents frequency. The STFT converts the signal from the time domain to the frequency domain, revealing the energy distribution of the signal across different frequency components, thus characterizing the oscillation properties of the signal, which is crucial and beneficial for subsequent analysis. To simulate the nonlinear perception characteristics of the human auditory system to frequency, the linear spectrum obtained from the STFT is passed through a Mel-scale triangular filter bank. This filter bank is equally spaced on the Mel frequency scale, with higher resolution in low-frequency regions and lower resolution in high-frequency regions, making it more consistent with the human auditory mechanism. Applying a logarithmic operation to the output energy of the filter bank compresses the dynamic range, further approximating the human perception of sound intensity, ultimately yielding the Mel-spectrogram. In the experiments of this paper, the librosa library was used to extract Mel-spectrograms with a sampling rate of 48 kHz, a Hamming window length of 512, and an FFT window size of 1024. To enhance model robustness, additive white Gaussian noise (AWGN) was introduced during the training phase to augment the original dataset, increasing data diversity and thereby improving accuracy and generalization ability in complex environments. Figure 4 shows a comparison of the linear spectrograms and Mel-spectrograms of truthful and deceptive speech, where the horizontal axis represents time, the vertical axis represents frequency, and the color depth represents energy intensity. By comparison, it can be observed that the spectrograms of speech deception often exhibit stronger energy distribution in high-frequency regions, which is a distinguishable difference from truthful speech.

To achieve high-precision mining of deception-related features from Mel spectrograms, this paper designs a parallel hybrid model that possesses both local feature extraction and global temporal modeling capabilities, and introduces an advanced second-order optimization algorithm to enhance its nonlinear fitting performance. The overall structure of the model is shown in Figure 5, mainly comprising three core parts: the Convolutional Neural Network module, the Transformer encoder module, and the optimization strategy based on the Levenberg-Marquardt algorithm.

First, the input Mel-spectrogram is fed in parallel to the CNN module and the Transformer module. In the CNN branch, the data flows through four consecutive convolutional blocks. Each convolutional block performs a convolution operation:

$$Z^l=W^l\ast X^l+b^l$$

(11)

This is followed by the introduction of non-linearity through batch normalization and the ReLU activation function, and the spatial dimensions are compressed by a max-pooling operation, ultimately resulting in a flattened convolutional embedding $h_{\mathrm{cnn}}\in {\mathbb{R}}^{256}$.

$${\overline{Z}}^{\left(l\right)}=\gamma ·{\displaystyle \frac{z^l-{\mu }_B}{\sqrt{{\sigma }^2}+\epsilon }}+\beta$$

(12)

In the Transformer branch, the spectrogram first undergoes max-pooling for dimensionality compression and adjustment, obtaining a serialized feature representation $X_{\mathrm{pool}}$, which is then fed into a multi-layer encoder. The encoder utilizes a multi-head self-attention mechanism:

$$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}\left({\mathrm{head}}_1,\dots ,{\mathrm{head}}_4\right)W^0$$

(13)

to dynamically compute global temporal dependencies, and undergoes deep nonlinear transformation through a feedforward neural network:

$$FFN\left(x\right)=W_2·ReLU\left(W_{1}X+b_1\right)+b_2$$

(14)

supplemented by layer normalization and residual connections to ensure training stability. Finally, a 64-dimensional Transformer embedding feature $h_{\mathrm{trans}}\in {\mathbb{R}}^{64}$ is obtained by average pooling over the time dimension. At this point, the model obtains $F_C$ representing local spatial patterns and $F_T$ representing long-term contextual relationships, respectively.

Subsequently, the feature fusion layer concatenates $F_C$ and $F_T$ to form a unified deep feature fed into the subsequent fully connected layers for classification decision. To optimize the parameters of this critical step, this paper innovatively introduces the LM algorithm. The LM algorithm is an efficient optimization algorithm specifically designed for solving nonlinear least squares problems. It adaptively adjusts the damping factor, skillfully combining the fast convergence of the Gauss-Newton method with the global stability of gradient descent. Considering the huge computational cost of the LM algorithm due to the need to calculate the Jacobian matrix, this paper adopts a two-stage training strategy. First, the Adam optimizer is used to pre-train the entire Transformer-CNN feature extraction backbone to obtain stable initial parameters. Subsequently, during the optimization phase, the backbone network parameters are frozen, and the LM algorithm is applied only to the top-level classifier. The core is to define the residual $\gamma$:

$$\gamma =y_{\mathrm{true}}-y_{\mathrm{pred}}$$

(15)

and update the parameters by solving the linear system:

$$\left(J^{T}J+\lambda I\right)\Delta W=-J^T\gamma$$

(16)

where J is the Jacobian matrix of the residuals with respect to the top-layer parameters W, and $\lambda$ is the damping factor. This algorithm can adaptively switch between gradient descent and the Gauss-Newton method, achieving superlinear convergence near the optimal solution, thereby accurately learning the complex mapping from the fused features to the final deception judgment result, effectively improving the model’s convergence speed and generalization performance.

During the LM refinement stage only (Mel-branch top fully connected layers, with the CNN/Transformer backbone frozen), we treat the binary targets as continuous scores in $[0,1]$ and minimize a mean squared error (MSE) objective so that the Levenberg–Marquardt update matches its standard nonlinear least-squares formulation:

$$L_{\mathrm{MSE}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_i-{\widehat{y}}_i\right)}^2$$

(17)

where N is the batch size, $y_i$ denotes the true label of the i-th sample, and ${\widehat{y}}_i$ represents the corresponding predicted value before thresholding. The LM algorithm iteratively minimizes this loss to refine the Mel-branch head parameters W.

Implementation (LM head in PyTorch). The Levenberg–Marquardt refinement is implemented in PyTorch 2.1.2. After freezing all CNN and Transformer parameters, we collect residuals $\gamma =y_{\mathrm{true}}-y_{\mathrm{pred}}$ from the last one to two fully connected (nn.Linear) layers of the Mel branch (the head that maps fused CNN/Transformer embeddings to scalar scores in $[0,1]$). We use PyTorch’s automatic differentiation to form the Jacobian–vector products needed for each LM step (equivalently, a Gauss–Newton damped update on those head weights only), iterating until convergence criteria on $\parallel \Delta W\parallel$ or on the MSE over the current mini-batch. The backbone remains non-trainable during this stage; no LM update is applied to the cross-attention fusion or the ACO/NLD branch.

Clarification (training objective for the full model). The Transformer–CNN Mel backbone is first pre-trained with Adam as in

Table 1. Training and model configuration. CE + SGD refers to the main end-to-end classifier; LM + MSE applies only to the Mel-branch head after backbone pre-training (see text). ${⠀}^{†}$ All sub-segments derived from the same original RTDD courtroom statement (same speaker instance in that trial) are assigned to the same fold, so training and test sets do not share speakers across partitions.

Parameter	Value
Audio Sampling Rate	48,000 Hz
Training Epochs	400
Batch Size	32
Backbone pre-training (Mel branch)	Adam optimizer
Main optimizer (global CE training)	SGD
Main classification loss	Cross-entropy (softmax, two classes)
LM refinement (optional head tuning)	Levenberg–Marquardt with MSE
Validation	Leave-one-speaker-out; leave-one-source trial-out ${⠀}^{†}$
Gated-BiLSTM Hidden Size	128
Gated-BiLSTM Dropout	0.4
CNN Block Channels	16-32-64-64
Transformer Encoder Layers	4
Transformer Attention Heads	4
Transformer FFN Dimension	512
Fusion FC Input Dim	320
Fusion FC Output Dim	2

, then frozen. The Gated-BiLSTM–Attention branch, the cross-attention fusion module, and the final softmax classifier are trained with stochastic gradient descent (SGD) and cross-entropy (CE) on the two-way softmax output. Thus, CE governs the main end-to-end classification path, whereas MSE appears only in the LM polishing step on the Mel-branch head described above—not as the global loss for MFF-Net.

In summary, this module ensures the comprehensiveness of feature extraction through the parallel structure of Transformer and CNN, and achieves efficient and precise optimization of model parameters by applying the LM algorithm at the key decision layer, jointly providing strong technical support for the deception detection task. The high-level representation of the Mel-spectrogram features $F_M$ output by this module will be fused with the acoustic features via cross-attention.

3.4. Multi-Feature Fusion Module Based on Cross-Attention

Acoustic-prosodic features (ACO) primarily characterize the prosody, voice quality, and spectral statistics of speech. Nonlinear dynamic features (NLD) reflect the chaotic dynamic behavior of speech production. Mel-spectrogram features provide global pattern information from a time-frequency perspective. These three types of features have significant differences in representation granularity, physical meaning, and data structure, constituting a typical heterogeneous feature fusion problem. Traditional multi-feature fusion in speech deception detection often relies on simple weighted summation or feature concatenation. Such methods can introduce a large amount of redundant information and struggle to fully explore the nonlinear correlations and deep complementarity between heterogeneous features. Targeting the characteristic that these three types of features, though from the same source, have distinct representational perspectives, this paper designs a deep multi-feature fusion module based on a cross-attention mechanism. This mechanism allows each type of feature to actively query relevant information in the feature space of the others, rather than being passively fused. For example, when acoustic features detect abnormal fluctuations in fundamental frequency, they can automatically enhance the attention to corresponding frequency band features in the Mel-spectrogram, achieving synergistic enhancement between features. This mechanism enables fine-grained feature fusion through bidirectional interaction and adaptive weight allocation between features.

Specifically, the high-level representation of acoustic nonlinear dynamic features $F_{AN}\in {\mathbb{R}}^{d_1}$ output by the Gated-BiLSTM-Attention module and the high-level representation of Mel-spectrogram features $F_M\in {\mathbb{R}}^{d_2}$ extracted by the Transformer-CNN-LM module are taken as inputs. First, they are mapped to a unified hidden dimension d through independent linear projection layers:

$$F_{AN}^{\left(\mathrm{proj}\right)}=F_{AN}{W}_{\mathrm{proj}}^{AN}+b_{\mathrm{proj}}^{AN}$$

(18)

$$F_M^{\left(\mathrm{proj}\right)}=F_M{W}_{\mathrm{proj}}^M+b_{\mathrm{proj}}^M$$

(19)

where $W_{\mathrm{proj}}^{AN},W_{\mathrm{proj}}^M\in {\mathbb{R}}^{d\times d}$ are projection weights, ensuring feature dimensionality consistency and laying the foundation for subsequent attention calculation. For the attention flow from acoustic features to Mel-spectrum:

$$Q_{AN}=F_{AN}^{\left(\mathrm{proj}\right)}{W}_q,K_M=F_M^{\left(\mathrm{proj}\right)}{W}_k,V_M=F_M^{\left(\mathrm{proj}\right)}{W}_v$$

(20)

$$Q_M=F_M^{\left(\mathrm{proj}\right)}{W}_q^{\prime },K_{AN}=F_{AN}^{\left(\mathrm{proj}\right)}{W}_k^{\prime },V_{AN}=F_{AN}^{\left(\mathrm{proj}\right)}{W}_v^{\prime }$$

(21)

where $W_q,W_k,W_v,W_q^{\prime },W_k^{\prime },W_v^{\prime }\in {\mathbb{R}}^{d\times d}$ are learnable parameter matrices. Then, the bidirectional cross-attention weights are calculated:

$${Attn}_{AN}=Softmax\left({\displaystyle \frac{Q_{AN}{K}_M^{\top }}{\sqrt{d}}}+M_{\mathrm{bias}}\right)V_M$$

(22)

$${\mathrm{Attn}}_M=Softmax\left({\displaystyle \frac{Q_M{K}_{AN}^{\top }}{\sqrt{d}}}+M_{\mathrm{bias}}^{\prime }\right)V_{AN}$$

(23)

where $M_{\mathrm{bias}},M_{\mathrm{bias}}^{\prime }$ are optional attention bias matrices for introducing prior knowledge. ${\mathrm{Attn}}_{AN}$ represents the context-enhanced representation obtained by acoustic features attending to Mel-spectrogram features, and ${\mathrm{Attn}}_M$ is the representation enhanced for Mel-spectrogram features by acoustic features. This design enables the model to dynamically capture cross-feature correlations such as “abnormal fundamental frequency and formant shifts”. To fully utilize the attention outputs, residual connections and a gating mechanism are employed:

$$G_{AN}=\sigma \left(W_g\left[{Attn}_{AN}\mid \mid F_{AN}^{\left(\mathrm{proj}\right)}\right]+b_g\right)$$

(24)

$$F_{AN}^{\prime }=LayerNorm\left(F_{AN}^{\left(\mathrm{proj}\right)}+G_{AN}\odot {\mathrm{Attn}}_{AN}\right)$$

(25)

where $W_g\in {\mathbb{R}}^{d\times d}$ is a learnable parameter. $F_M^{\prime }$ is obtained similarly. Finally, a unified representation is generated through a hierarchical fusion strategy:

$$F_{ANM}=\mathrm{FFN}\left(\left[F_{AN}^{\prime }\left|\right|F_M^{\prime }\right]\right)$$

(26)

where $\left|\right|$ denotes the concatenation operation, and FFN is a two-layer feedforward network. The fused feature vector $F_{ANM}$ will be input to the Softmax classifier. The mapping from the feature space to the category space is completed through the formula $\overline{y}=argmax\left(Softmax\left(W_c{F}_{ANM}+b_c\right)\right)$, ultimately achieving binary classification of lies and truth. This fusion mechanism effectively addresses the representation gap problem of heterogeneous features, providing a unified feature representation with stronger discriminative power for deception detection.

4. Experiments

4.1. Experiments Setup

All experiments were conducted on a system equipped with a 13th Gen Intel^® Core i9-13900HX CPU (2.20 GHz), 16 GB of RAM, and an NVIDIA GeForce RTX 4060 GPU. The software environment utilized PyTorch 2.1.2 with CUDA 11.8. The detailed training and model hyperparameters are listed in Table 1.

Hyperparameter selection. The key architectural choices (Gated-BiLSTM width 128, dropout 0.4, four Transformer layers with four heads and feed-forward size 512) were fixed after a coarse grid search on CLD validation folds over learning rates $\{1\times {10}^{-3},5\times {10}^{-4},1\times {10}^{-4}\}$ and hidden widths $\{64,128\}$. Other settings follow the reference implementations of CovBiLSTM [40] and the hierarchical attention baseline [43].

Speaker-aware splits. On CLD, we use leave-one-speaker-out (LOSO) cross-validation: all clips from a held-out speaker are reserved for testing while the remaining 45 speakers are used for training, so clips from the same participant never appear in both partitions. RTDD was expanded from 121 verified trial statements into 3205 shorter clips; to mitigate optimistic bias, folds are formed at the source-statement level (all clips originating from the same original RTDD utterance stay together), which prevents the same courtroom speaker instance from appearing in both training and test sets within a fold.

Checkpoint selection for deep models. Deep networks are sensitive to random initialization and mini-batch order. Under the experimental budget of this work, each deep baseline and MFF-Net was trained once per dataset fold using the protocol in Table 1. Subsequent experiments report test-set accuracy together with precision/recall/F1 evaluated at the checkpoint that achieved the highest validation accuracy during that single training run—i.e., a validation-best checkpoint. This uses the same selection principle as multi-seed studies that pick, for each seed, the validation-optimal state before reading off test performance, but here it applies to only one realized trajectory per model and fold: we do not average over or fabricate additional random seeds, and we do not report mean ± standard deviation across reinitializations. Systematic multi-seed aggregation is left for future work.

4.2. Dataset

In speech deception detection research, speech data from real judicial scenarios and laboratory-controlled scenarios have significant differences. Speech in real scenarios (e.g., court trials) is often accompanied by environmental noise, multi-person interaction, and complex conversational structures. Deceptive behavior is often premeditated, with stronger emotional masking ability, and speech features are relatively natural but mixed with more interference. In laboratory scenarios, the speech acquisition environment is clean, deception tasks are usually structured, individuals are in a relatively isolated state, and the pressure to lie is more concentrated, potentially exhibiting more obvious unnatural pauses, fundamental frequency fluctuations, and other features in the speech. Existing high-quality speech deception data mostly come from real scenarios, such as RTDD used in this paper [44]. To comprehensively evaluate the generalization ability of MFF-Net in different contexts, this paper also constructed a Chinese deception dataset (CLD), which induces purer and more controllable speech deception samples by simulating single-person narrative tasks in a controlled environment. The two datasets are introduced below.

4.2.1. The Real-Life Trial Deception Database (RTDD)

This paper uses RTDD for deception detection experiments in real judicial scenarios. This database contains audio and video recordings from real court trial processes; this paper uses only the audio data. All statement segments have been subsequently verified by judicial procedures and are clearly labeled as “True Statement” or “Deceptive Statement,” possessing high ecological validity. The recordings were collected in the courtroom. Although there is some environmental noise, the overall speech intelligibility is high, reflecting the actual conditions of speech deception detection in real judicial environments.

The original database contains 121 trial statement segments (61 lies, 60 truths), each with a corresponding truth/false label. To adapt to model training needs, this paper first performed silence removal and sentence segmentation on the original long audio, and then segmented and annotated based on the transcript and truth labels, ultimately constructing a subset of 3205 valid speech samples, including 1624 truthful statement samples and 1581 deceptive statement samples. The specific distribution of the dataset is shown in

Table 2. RTDD clip counts after segmentation. Reading guide: defendants and witnesses contribute similar numbers of truthful vs. deceptive clips.

Speaker Type	Number of Truths	Number of Lies	Duration (s)
Defendant	892	875	12.4
Witness	732	706	11.8
Total	1624	1581	12.1

.

This database not only provides high-quality English deception data but also serves as an important foundation for studying multimodal features (such as acoustic characteristics, speech disfluencies, etc.) in judicial environments. Feature extraction and model training conducted on this basis validate the effectiveness of the proposed MFF-Net in English scenarios.

4.2.2. The Chinese Lie Dataset (CLD)

This paper constructs a Chinese lie collection process based on both intentional lying and interactive lying paradigms. A total of 46 participants were involved, including 30 students and 16 working professionals. All participants were aged between 20 and 30, with 23 males and 23 females, and all used Mandarin for the experiment. The main experiment consisted of two parts. The first part was “Intentional Lying.” First, participants were given three topics (“My University,” “My Best Friend,” “My Teacher”) and asked to write about them, requiring the content to be truthful. During the experiment, one theme was randomly selected first, and the participant was asked to describe the written content without notes; secondly, one of the remaining two themes was selected, and the participant was required to lie about the key information of the written content, and was informed that they would receive an additional participant reward if the lie was not detected by the experimental instrument. The second part was “Interactive Lying.” First, participants needed to fill out a series of questionnaire questions, including but not limited to likes/dislikes about certain things, past experiences in certain aspects; the questionnaire only required answering “Yes” or “No.” During the experiment, professional recorders guided and recorded the process. Participants were paired up, taking on the different roles of “deception detector” and “liar.” The “liar” was required to state or lie about the answers in the form, requiring lying about approximately half of the content; rewards would be deducted if requirements were not met or if contradictions arose. The “detector” needed to ask questions based on the form filled by the “liar,” and judge whether the “liar” was lying; questions could be repeated, asked in random order, etc., to judge more accurately. The correctness of the “detector”’s judgment also affected the reward.

The recording process was conducted in a professional recording studio. All recording equipment was strictly adjusted and tested before the formal experiment to ensure accuracy and standardization. Audio annotation and segmentation were performed manually using Praat 6.4.62 and Adobe Audition 26.0 tools. The processors also received professional, standardized training, following unified audio processing guidelines. The segmentation strategy combined sentence-level segmentation and pause-based segmentation to closely match the actual meaning and intent. In the “Intentional Lying” off-script narration part, processors segmented each complete sentence. In the fabrication part of “Intentional Lying,” processors recorded the segments where the participant changed key information. For example, if a participant wrote “My best friend is A” but said “My best friend is B” during the experiment, since only “B” is the lied part, subtle vocal differences might not necessarily occur when saying other parts; thus, processors only recorded the segment where “B” was uttered to emphasize the specificity of speech during lying. In the “Interactive Lying” part, processors only recorded the segments where the “liar” answered “Yes” or “No” during the experiment. Short silent pauses for breath were allowed during the experiment but had to be within a specified time threshold; if exceeded, the intermediate silent segment was labeled “NOISE.” Similarly, non-speech segments like coughs, sneezes, or slips of the tongue were labeled “NOISE” and were removed during post-processing, leaving only clear and valid speech segments. Finally, CLD collected a total of 3566 speech segments from 46 speakers. The basic content of the dataset is shown in

Table 3. CLD segment counts by paradigm and label. Reading guide: interactive lying contributes the largest number of clips per class.

Collecting Scene	Truth/Lie	Speaker Num	Segments Num
Intentional lying1	Truth	46	846
Intentional lying2	Lie	46	283
Interactive lying1	Truth	46	1231
Interactive lying2	Lie	46	1206

. All data are stored in WAV format and divided using the same standard. During recording, it was found that compared to truthful speech, most people’s extracted speech segments were longer when lying, with more unnatural pauses and more blurred pronunciation.

Ethics, consent, and annotation quality control. All CLD participants provided written informed consent after receiving an explanation of the recording procedures, reward rules, and data handling policy. Personally identifiable material was excluded from prompts, and audio files were pseudonymized before feature extraction. Annotators were trained on a common segmentation manual; ambiguous cases were discussed among annotators until consensus was reached. CLD was assembled as an in-house institutional dataset; prospective approval from an independent ethics committee (IRB) with a formal protocol number was not obtained. The end-of-manuscript Institutional Review Board statement describes this governance context and the protections that were applied nonetheless.

Confidentiality and data release. CLD contains sensitive and confidentiality-classified content under institutional policies. Therefore, the raw audio and participant-linked metadata cannot be made publicly available in the short term. To support reproducibility within these constraints, we provide detailed acquisition, segmentation, and annotation protocols in the manuscript; aggregated statistics and non-identifying methodological descriptions are given in the tables. Requests for collaboration or auditor-level access must follow institutional approval procedures.

In deception-related research, data collection methods are crucial. The intentional lying paradigm has unique advantages in the form of experimental tasks. In contrast, most studies collecting data from participants’ spontaneous speech are variants of the Trier Social Stress Test, which requires free speech or storytelling on fixed topics to induce different stress levels. In this experiment, since participants prepared their speech content in advance, emotional fluctuations caused by the nervousness of impromptu speech were avoided. Meanwhile, the lied speech, being influenced by the prepared truth, is closer to real-life lies, helping to obtain more authentic and reliable deception data. In the interactive lying paradigm, deception detection tasks are set up by assigning different roles to participants, such as “liar” and “detector.” Similar settings can be found in many studies. Compared to game-based or simulated crime paradigms, the interactive lying paradigm has a simpler experimental setup. Participants only answer yes or no; lying and lie detection are achieved through Q&A interactions, which helps highlight key behaviors and reactions in deception. The entire data collection process was conducted in a closed indoor recording setting, reducing noise interference. The speech segment processing was also unified and standardized, improving audio data quality. The combination of two speech segmentation methods is more in line with semantics and people’s cognition of lies, while improving the accuracy of speech data, paving the way for enhancing model performance.

4.3. Ablation Studies

To systematically evaluate the contribution of different features and modules in the proposed MFF-Net, this section designs multiple sets of ablation experiments. The experiments are conducted on CLD, using Accuracy, Precision, Recall, and F1-score as evaluation metrics to comprehensively measure model performance. These metrics are all calculated based on True Positives ($TP$), True Negatives ($TN$), False Positives ($FP$), and False Negatives ($FN$) from the confusion matrix. Their mathematical definitions are as follows:

Accuracy measures the model’s ability to correctly classify the overall samples, calculated as:

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(27)

Precision focuses on the reliability of the model’s predictions as positive examples, calculated as:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(28)

Recall measures the completeness of the model’s identification of true positive examples, calculated as:

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(29)

Given that Precision and Recall are often mutually constrained, this study uses the F1-score as a core evaluation metric. The F1-score is the harmonic mean of Precision and Recall, particularly suitable for handling datasets that may have class imbalance. It is calculated as:

$$\mathrm{F}1-\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(30)

To systematically evaluate the effectiveness of different features and their combinations in the speech deception detection task, a series of ablation experiments was conducted using a consistent LSTM baseline model. The performance of single features and various feature combinations is summarized in

Table 4. Single- and multi-feature ablations with a fixed LSTM baseline (%). Reading guide: Mel alone is strongest among singles, yet combining all three yields the best score, indicating complementarity.

ACO	NLD	Mel	Accuracy	Precision	Recall	F1-Score
✓	×	×	62.37	63.28	62.05	63.11
×	✓	×	57.27	51.37	58.29	59.98
×	×	✓	67.31	65.44	66.28	67.29
✓	✓	×	63.75	63.01	62.89	64.37
✓	×	✓	67.32	66.28	66.97	66.98
×	✓	✓	66.73	65.33	65.47	66.68
✓	✓	✓	69.79	68.23	67.36	68.01

.

Analysis of Table 4 yields several key findings. First, among the single features, the Mel-spectrogram feature achieves the best performance (Accuracy: 67.31%, F1-score: 67.29%), indicating its high reliability in discriminating deceptive speech when used independently. The acoustic-prosodic (ACO) features show moderate performance. In contrast, the nonlinear dynamic (NLD) features perform the weakest on their own, which is attributable to the inherently subtle and low signal-to-noise-ratio characteristics of the nonlinear dynamics in speech.

Second, the combination of any two features consistently outperforms the weaker single feature involved, demonstrating complementary information between feature types. Crucially, although NLD features perform poorly in isolation, their integration with either ACO or Mel features leads to a stable performance gain (e.g., ACO + NLD achieves higher accuracy than either ACO or NLD alone). This strongly suggests that NLD features contain complementary information related to the psychophysiological state of deception, which is not captured by the ACO or Mel features.

Finally, the fusion of all three features (ACO + NLD + Mel) achieves the best overall performance (Accuracy: 69.79%, F1-score: 68.01%). This result validates that multi-feature fusion can more comprehensively represent the multi-dimensional acoustic manifestations of deceptive behavior, thereby enhancing the robustness of the detection system. These findings provide a solid experimental foundation for the subsequent design of the sophisticated multi-feature fusion architecture (MFF-Net).

To further validate the effectiveness of the proposed MFF-Net, ablation experiments were conducted on key modules. The complete model was decomposed into different sub-modules (Gated-BiLSTM, Gated-BiLSTM-Attention, Transformer-CNN-LM) and fed with different feature combinations. The results are shown in

Table 5. Module ablation on CLD (%). Reading guide: progressively adding attention and the Mel branch increases accuracy, with the full MFF-Net row showing the benefit of cross-attention fusion.

Module	Feature	Accuracy	Precision	Recall	F1-Score
Gated-BiLSTM	ACO	72.69	67.28	68.39	67.88
Gated-BiLSTM	ACO + NLD	73.35	70.77	73.22	70.34
Gated-BiLSTM-Attention	ACO	74.33	69.51	66.29	68.33
Gated-BiLSTM-Attention	ACO + NLD	74.68	70.37	70.65	71.39
Transformer-CNN-LM	Mel	76.56	72.34	73.15	73.28
MFF-Net (Ours)	All	78.29	79.16	78.48	79.13

.

Analysis of Table 5 leads to the conclusion that the complete MFF-Net (with “All” features as input) achieves optimal performance across all metrics, with an accuracy of 78.29%, significantly higher than any sub-model using partial features. This fully validates the effectiveness of the cross-attention-based multi-feature fusion strategy adopted in this paper, as different features provide complementary information. Comparing the performance of BiLSTM and BiLSTM-Attention when processing ACO or ACO + NLD features, it can be seen that introducing the attention mechanism improves model performance, indicating that the attention mechanism helps the model focus on more discriminative temporal segments. The Transformer-CNN module, using only Mel features, achieves an accuracy of 76.56%, indicating that this parallel structure can simultaneously capture local details and global temporal dependencies in Mel-spectrograms, demonstrating strong feature extraction capability. Ultimately, MFF-Net processes acoustic features (ACO + NLD) through the Gated-BiLSTM-Attention module, processes Mel-spectrogram features (Mel) through the Transformer-CNN-LM module, and performs interactive fusion in the multi-feature fusion module, achieving comprehensive improvement in accuracy, precision, recall, and F1-score. Particularly, the high recall rate of 78.48% indicates that the network can effectively reduce missed detections, which is of great value in practical applications. The ablation experiments have validated the rationality of the model design in this paper from both the feature and model structure dimensions.

4.4. Experimental Comparison on RTDD

To validate the effectiveness of MFF-Net on English deception datasets, this paper conducted systematic comparative experiments on RTDD. As shown in Table 6, this paper selected Random Forest (RF) and Deep Feedforward Network (DFNN) as baseline models, while also comparing performance with advanced networks such as Relevance Vector Machine (RVM) [1], Convolutional Bidirectional Long Short-Term Memory network (CovBiLSTM) [40], and the hierarchical attention architecture of Chou et al. [43] adapted to our acoustic(+NLD) inputs as HAN.

Table 6. Comparative results on RTDD (%). RF/DFNN/RVM are deterministic given features; for deep models, metrics are evaluated at the validation-best checkpoint of the single reported training run per fold (see Experiments setup).

Algorithm	Features	Fusion Type	Accuracy	Precision	Recall	F1-Score
RF	ACO		59.84	61.31	59.24	63.16
DFNN	ACO		61.26	63.23	61.06	62.17
RVM [1]	ACO		65.25	64.19	63.31	65.14
RVM [1]	ACO + NLD	Hard Voting	67.34	66.85	66.21	65.19
RVM [1]	ACO + NLD	Soft Voting	68.42	67.39	67.46	70.15
CovBiLSTM [40]	ACO		73.85	71.91	72.17	74.97
HAN [43]	ACO		70.21	70.36	72.75	70.03
HAN [43]	ACO + NLD	Hard Voting	71.17	73.21	73.27	71.41
HAN [43]	ACO + NLD	Soft Voting	73.31	73.89	74.21	72.74
MFF-Net (Ours)	ACO + NLD + Mel	Hard Voting	75.28	77.21	75.23	74.29
MFF-Net (Ours)	ACO + NLD + Mel	Cross-Attention	79.22	80.18	78.32	77.69

Table 6. Comparative results on RTDD (%). RF/DFNN/RVM are deterministic given features; for deep models, metrics are evaluated at the validation-best checkpoint of the single reported training run per fold (see Experiments setup).

Algorithm	Features	Fusion Type	Accuracy	Precision	Recall	F1-Score
RF	ACO		59.84	61.31	59.24	63.16
DFNN	ACO		61.26	63.23	61.06	62.17
RVM [1]	ACO		65.25	64.19	63.31	65.14
RVM [1]	ACO + NLD	Hard Voting	67.34	66.85	66.21	65.19
RVM [1]	ACO + NLD	Soft Voting	68.42	67.39	67.46	70.15
CovBiLSTM [40]	ACO		73.85	71.91	72.17	74.97
HAN [43]	ACO		70.21	70.36	72.75	70.03
HAN [43]	ACO + NLD	Hard Voting	71.17	73.21	73.27	71.41
HAN [43]	ACO + NLD	Soft Voting	73.31	73.89	74.21	72.74
MFF-Net (Ours)	ACO + NLD + Mel	Hard Voting	75.28	77.21	75.23	74.29
MFF-Net (Ours)	ACO + NLD + Mel	Cross-Attention	79.22	80.18	78.32	77.69

The results in Table 6 clearly demonstrate a progressive performance hierarchy among the algorithms. Firstly, traditional machine learning methods (RF, DFNN) exhibit the weakest performance, indicating their limited representational capacity for the complex, high-dimensional features in speech deception. RVM, which incorporates a Bayesian framework, shows improvement but reaches a clear performance ceiling. Deep learning models, represented by CovBiLSTM and HAN, achieve a significant performance leap through automatic feature learning, confirming the advantage of deep networks in capturing non-linear patterns and temporal dependencies in deceptive speech. Within similar models, the soft voting fusion strategy generally outperforms hard voting (e.g., comparisons for RVM and HAN), as soft voting, based on probability weighting, retains more confidence information, enabling a more refined decision fusion. CovBiLSTM achieves the best performance among all baselines (73.85%) using only ACO features, benefiting from CNN’s powerful local feature extraction and BiLSTM’s effective modeling of long-term temporal contexts.

The proposed MFF-Net achieves superiority on multiple levels. First, even with the same hard voting fusion strategy, MFF-Net (75.28%), by introducing Mel-spectrogram features and employing a more powerful Gated-BiLSTM and Transformer encoder combination as the feature extraction backbone, already outperforms CovBiLSTM using only ACO features (73.85%) and HAN using ACO + NLD features (73.31%). This proves that multi-feature complementarity and the improved network architecture can extract more discriminative representations. Ultimately, when employing the cross-attention fusion mechanism, MFF-Net achieves the optimal performance (79.22%). Compared to the best baseline CovBiLSTM, accuracy is improved by 5.37 percentage points. This leap is primarily attributed to the cross-attention mechanism’s ability to dynamically and discriminatively model the interactions and contribution weights among different feature modalities (ACO, NLD, Mel), achieving deep information fusion rather than simple voting or concatenation. This allows for a more thorough exploitation of the synergistic discriminative cues among multi-modal features.

4.5. Experimental Comparison on CLD

Most existing speech deception detection methods are tested on English deception datasets. To validate the effectiveness of MFF-Net on Chinese data, this paper conducted corresponding comparative experiments on CLD. The comparative experimental results on CLD, shown in Table 7, indicate that although MFF-Net still maintains a leading position, its overall performance is slightly lower compared to RTDD. This phenomenon is closely related to the richer and more subtle characteristics of Chinese deception, sample distribution properties, and data quality.

Table 7. Comparative results on CLD (%). Deep models: validation-best checkpoint within one training run per fold, as in Table 6.

Algorithm	Features	Fusion Type	Accuracy	Precision	Recall	F1-Score
RF	ACO		59.31	60.18	61.37	60.71
DFNN	ACO		61.33	62.76	60.71	63.95
RVM [1]	ACO		63.21	60.63	63.21	62.83
RVM [1]	ACO + NLD	Hard Voting	66.26	67.35	64.19	63.62
RVM [1]	ACO + NLD	Soft Voting	68.92	68.15	67.39	66.15
CovBiLSTM [40]	ACO		74.80	74.65	74.21	73.68
HAN [43]	ACO		65.19	66.07	62.98	62.13
HAN [43]	ACO + NLD	Hard Voting	68.65	69.48	67.92	68.94
HAN [43]	ACO + NLD	Soft Voting	69.09	70.47	69.92	71.56
MFF-Net (Ours)	ACO + NLD + Mel	Hard Voting	76.97	75.01	77.63	75.59
MFF-Net (Ours)	ACO + NLD + Mel	Cross-Attention	78.29	79.16	78.48	79.13

Table 7. Comparative results on CLD (%). Deep models: validation-best checkpoint within one training run per fold, as in Table 6.

Algorithm	Features	Fusion Type	Accuracy	Precision	Recall	F1-Score
RF	ACO		59.31	60.18	61.37	60.71
DFNN	ACO		61.33	62.76	60.71	63.95
RVM [1]	ACO		63.21	60.63	63.21	62.83
RVM [1]	ACO + NLD	Hard Voting	66.26	67.35	64.19	63.62
RVM [1]	ACO + NLD	Soft Voting	68.92	68.15	67.39	66.15
CovBiLSTM [40]	ACO		74.80	74.65	74.21	73.68
HAN [43]	ACO		65.19	66.07	62.98	62.13
HAN [43]	ACO + NLD	Hard Voting	68.65	69.48	67.92	68.94
HAN [43]	ACO + NLD	Soft Voting	69.09	70.47	69.92	71.56
MFF-Net (Ours)	ACO + NLD + Mel	Hard Voting	76.97	75.01	77.63	75.59
MFF-Net (Ours)	ACO + NLD + Mel	Cross-Attention	78.29	79.16	78.48	79.13

On the Chinese CLD dataset, the performance ranking of the models is analogous to that on RTDD, but the overall metric values are somewhat lower. CovBiLSTM remains a strong baseline (74.80%), while the HAN model performs relatively weaker on this dataset, which may be related to its greater reliance on text-level attention mechanisms, whereas acoustic features might be more universal in speech deception tasks. The two variants of MFF-Net also securely occupy the top two positions, with the cross-attention fusion version ultimately achieving the best performance (78.29%), representing a 3.49 percentage point improvement over the best baseline CovBiLSTM.

It is noteworthy that MFF-Net’s performance on CLD (78.29%) is slightly lower than on RTDD (79.22%). This discrepancy may stem from the following reasons: First, inherent differences in dataset difficulty and characteristics. Chinese deception may involve more complex intonation, rhythm variations, and cultural-context dependencies. These subtle nuances pose greater challenges for feature extraction and model generalization. Second, class imbalance issues. As mentioned, truthful samples constitute 58.24% of CLD. This imbalance may cause the model to slightly overfit the majority class, affecting recall for the minority class (deception). Finally, data scale and quality. The English RTDD dataset might be larger, have more consistent annotations, or contain less noise, providing a more stable foundation for model learning. Despite these challenges, MFF-Net still achieves a significant performance lead on the more challenging CLD dataset. This strongly demonstrates the robustness and generalization capability of its model architecture. The cross-attention mechanism can adaptively focus on the most relevant combinations of features for deceptive speech across different languages and cultural contexts, thereby maintaining superior performance even when faced with variations in data distribution and inherent challenges.

5. Discussion

5.1. Interpretation

MFF-Net improves over strong baselines because it lets interpretable sequence cues (ACO/NLD) and deep time–frequency cues (Mel) interact before the softmax decision. Hard-voting ablations already surpass single-feature systems, while cross-attention yields an additional gain by learning which temporal regions of each branch should be emphasized for a given clip. This pattern aligns with the deception literature, where both global prosodic trends and local irregularities carry complementary information [8,22].

5.2. Computational Cost and Deployability

Training used consumer hardware (RTX 4060, 16 GB RAM). The dual-branch structure is heavier than a single-stream CNN, but it remains modest compared with very large self-supervised speech encoders [24]. Inference is dominated by feature extraction (OPENSmile/librosa) followed by two moderate-size branches and a shallow fusion head, which is compatible with offline batch screening rather than sub-10 ms streaming use cases. Further compression (pruning, branch distillation) is left to future work.

5.3. Robustness to Noise and Compression

Real deployments may involve telephony codecs, packet loss, or background noise. While we apply light AWGN augmentation during Mel-branch training, we do not claim robustness to strong codec cascades or very low SNR without additional testing. NLD features and narrow-band Mel statistics can degrade under aggressive compression; systematic evaluation with ITU-T codec simulations and controlled SNR sweeps is an important next step [18].

5.4. Limitations

Dataset scale remains moderate: RTDD clips are numerous but originate from a finite set of trials, and CLD is laboratory-controlled Mandarin speech from young adults, which may not capture cultural or stylistic variation in deceptive behavior elsewhere. Self-reported and task-induced lies also differ from high-stakes forensic lies. Finally, speaker-aware splits reduce optimism but do not remove all domain shift between studios and courtrooms. Separately, CLD was built under internal institutional governance without a numbered independent IRB/ethics-committee protocol; while informed consent and pseudonymization were applied, this differs from fully IRB-registered human-subjects studies and may affect comparability with stricter regulatory settings.

5.5. Future Work

Future directions include (i) codec/noise stress tests with calibrated metrics; (ii) semi-supervised or self-supervised front ends fine-tuned on deception corpora; (iii) distilling MFF-Net for edge deployment; and (iv) further documenting CLD procedures while respecting confidentiality constraints (public release of raw CLD audio is not planned in the short term).

6. Conclusions

To address limitations of single-representation detectors and to study Mandarin laboratory speech alongside English courtroom clips, this paper proposes an innovative network named MFF-Net. The core of this network lies in the fusion of acoustic-prosodic features (ACO), nonlinear dynamic features (NLD), and Mel-spectrogram features (Mel). It combines a network structure integrating the Gated-BiLSTM-Attention module, the Transformer-CNN-LM module, and the multi-feature fusion module to effectively mine and adaptively fuse multi-dimensional information related to deception in speech signals. Through extensive experiments on the English deception dataset (RTDD) and a self-built Chinese deception dataset (CLD), the effectiveness of the proposed method was validated. Ablation experiment results showed that the three types of features—ACO, NLD, and Mel—do contain complementary information, and their fusion can bring significant performance improvements. Meanwhile, each module of the proposed network contributed positively to the final performance. Comparative experiments with various mainstream algorithms further demonstrated that MFF-Net achieves leading advantages in key metrics such as accuracy, precision, recall, and F1-score, performing notably well on CLD. In summary, this work shows that coupling handcrafted prosodic/dynamic descriptors with a deep Mel branch and learned cross-modal interaction improves deception detection on both RTDD and CLD under speaker-aware evaluation. The approach is not restricted to Chinese speech—CLD simply complements RTDD—but broader claims about noise, compression, or cross-cultural lying require further targeted experiments. Future work will stress-test codecs and SNR, explore lighter deployments, and enrich methodological documentation; CLD raw audio remains confidential and is not publicly released in the short term.