Comparison of Modern Deep Learning Models for Speaker Verification

Abstract

This research presents an extensive comparative analysis of a selection of popular deep speaker embedding models, namely WavLM, TitaNet, ECAPA, and PyAnnote, applied in speaker verification tasks. The study employs a specially curated dataset, specifically designed to mirror the real-world operating conditions of voice models as accurately as possible. This dataset includes short, non-English statements gathered from interviews on a popular online video platform. The dataset features a wide range of speakers, with 33 males and 17 females, making a total of 50 unique voices. These speakers vary in age from 20 to 70 years old. This variety helps in thoroughly testing speaker verification models. This dataset is especially useful for research on speaker verification with short recordings. It consists of 10 clips for each person, each clip being no longer than 10 s, adding up to 500 recordings in total. The total length of all recordings is about 1 h and 30 min, which averages to roughly 100 s for each speaker. This dataset is a valuable tool for research in speaker verification, particularly for studies involving short audio clips. The performance of these models is evaluated using common biometric metrics such as false acceptance rate (FAR), false rejection rate (FRR), equal error rate (EER) and detection cost function (DCF). The results reveal that the TitaNet and ECAPA models stand out by presenting the lowest EER (1.91% and 1.71%, respectively) and thus exhibiting higher discriminative features, ensuring, on the one hand, a reduction in intra-class distance (the same speaker), and, on the other hand, maximizing the distance between different speaker embeddings. This analysis also highlights the ECAPA model’s advantageous balance of performance and efficiency, achieving an inference time of 69.43 milliseconds, slightly longer than the PyAnnote models. This study not only compares the performance of models but also provides a comparative analysis of respective model embeddings, offering insights into their strengths and weaknesses. The presented findings serve as a foundation for guiding future research in speaker verification, especially in the context of short audio samples or limited data. This may be particularly relevant for applications requiring quick and accurate speaker identification from short voice clips.

1. Introduction

Speaker verification is one of the important research areas in fast-growing modern machine learning applications, known as Speaker Recognition (SR). Two other SR sub-fields are speaker identification and speaker diarization [1,2].

Speaker verification aims to determine whether the declared identity of the speaker is true or false. During speaker verification, the system typically compares the provided voice sample with a reference voiceprint (template) associated with the declared identity; this is known as a one-to-one mapping. Eventually, the system makes a decision indicating whether the speaker is verified (matches the provided identity) or not verified (does not match the provided identity). In contrast to verification, the task of speaker identification is to determine the identity of the speaker from a set of possible speakers. In the speaker identification process, the system compares a given voice sample with voiceprints from a database of known speakers (one-to-many mapping). Then, the system attempts to identify the speaker by selecting the closest match from the database.

The main goal of speaker diarization is to divide the input audio recording into homogeneous segments corresponding to individual speakers. Traditionally, speaker diarization involves both segmentation (detecting speaker changes in an audio recording) and clustering (grouping speech segments corresponding to specific speakers based on voice characteristics). Unlike speaker verification and identification, speaker diarization can be approached using unsupervised learning [3].

Although this paper is dedicated to studying several state-of-the-art deep speaker embedding models primarily in the context of verification, the results may also have implications for the broader field of speaker recognition.

Nowadays, speaker verification has become a crucial task in many applications, such as security, authentication, surveillance, forensics, multi-speaker tracking, personalized user interfaces, voice assistants, door access control systems and others [4,5,6,7]. Nonetheless, the most important capability of speaker verification is the ability to identify and distinguish between different speakers accurately to be able to pinpoint a specific speaker.

Speech is a time-varying signal that conveys information on multiple levels, including acoustic, semantic, emotional and anatomical. The inherent variability of speech, coupled with the fact that even the same individual cannot say a phrase identically twice, makes its analysis a complex challenge. This complexity poses significant challenges in designing speaker recognition and verification systems.

Early speaker verification systems were constrained by the limitations of statistical methods, which resulted in restricted accuracy. It was only with the advent of machine learning that a significant improvement in performance was achieved [2].

In recent years, numerous new models have emerged in the field of NLP; however, the dominant trend in speech processing revolves around models based on speaker embedding technology. The essence of the speaker embeddings technique is to map the essential features of a speaker’s voice into a numerical (vector) representation of constant dimensions in a multidimensional space [8,9]. Over the subsequent years, the concept of speaker embeddings has been continuously developed, primarily aiming to measure and enhance performance under variable recording or processing conditions [10].

Speaker embeddings offer several advantages compared to traditional speech processing techniques, including greater compactness of voice representation, resilience to noise and recording condition changes and improved discrimination leading to enhanced performance in speech recognition tasks. Currently, speaker embedding technology can be applied to various tasks, including speaker recognition, verification, identification, diarization, tracking different speakers in a recording and detecting the presence of specific keywords [1,11].

Advanced characteristics of speaker embeddings are achieved through training models on extensive voice datasets, allowing the capture of unique voice features crucial for reliable speaker recognition and verification. Modern models vary in architecture, methods of calculating embeddings, length of embeddings and numerous parameters [1,12]. A key area of interest is comparing the performance of these models in real-world scenarios, including their adaptability and portability to other languages.

This study presents a comparison of state-of-the-art models for speaker verification tasks, focusing on models that utilize speaker embeddings. These models offer notable advantages, such as scalability [13], allowing the addition of new users without the need for retraining or fine-tuning. This feature is particularly valuable in applications with constantly growing and changing user bases, where the ability to seamlessly integrate new users without the computational overhead of updating the model is crucial. Moreover, this scalability ensures that the model’s performance remains consistent and reliable, making these models ideal for large-scale application [14]. The models’ performance was compared using a custom dataset comprising 50 speakers. The experiment’s source code and reproducibility details can be found in the following repository: [15].

The primary contributions of the paper include (1) an extensive comparison of state-of-the-art models such as WavLM, TitaNet, Ecapa and Pyannote, with an emphasis on their performance in speaker verification tasks; (2) the utilization of a newly compiled dataset, featuring non-English speech samples of Ukrainian politicians, for evaluating these models; and (3) a thorough analysis of the embeddings produced by each model, providing insights into their strengths, weaknesses and potential applications in real-world scenarios.

The organization of the paper is as follows: Section 1 introduces the context and significance of speaker verification, Section 2 reviews related works and establishes the background for the research, Section 3 outlines the aims of the research and the rationale behind the selection of the models and dataset, Section 4 provides a detailed overview of the models under study, Section 5 describes the experimental setup and methodology, and, finally, Section 6 presents the findings and conclusions.

2. Related Works

The number of publications on Automatic Speech Recognition (ASR), including speaker verification, is rapidly growing [11]. Real-world applications necessitate the development of automatic speaker verification systems, which would be capable of using short arbitrary (text-independent) utterances that might be captured in diverse settings, including time and equipment. These variables intensify the challenge of identifying specific unique voice characteristics of a speaker, which are inherently subject to variability [16,17,18].

Numerous studies have proposed innovative approaches to enhance speaker verification by exploring various neural network architectures and processing methods, in many cases utilizing open or self-generated datasets [19,20,21,22]. For instance, paper [20] presents a novel network with a hierarchical processing method, where information about intonation as a high-level function is combined with low-level embedding functions. This improves the accuracy of the system, which is confirmed by experimental results on the VoxCeleb1 test data. Paper [21] investigates the relationship between visual speech cues, like lip movements, and audio speech, leading to the introduction of a cross-modal speech co-learning paradigm.

Recognizing speakers based on non-English language utterances presents another significant challenge. Studies show that speaker verification performance declines when there is a language mismatch between training and testing data. To improve cross-language invariance, ref. [23] proposed a method of unsupervised adversarial discriminative domain adaptation. In this approach, the representative speech data from one language is adapted to align with the source domain of another language. Taking it a step further, the authors of [24] introduced the Siamese SpeakerNet network, able to verify speakers without being influenced by language, gender, or age differences and demonstrated superior performance compared to existing methods.

In this study, we concentrate on open and accessible pre-trained speech models for speaker verification. Following a thorough analysis, we selected the following four state-of-the-art speech models for further investigation: Pyannote, WavLM, TitaNet and ECAPA [25,26,27,28].

Several publications have discussed the results and capabilities of these language models. For instance, ref. [26] describes the WavLM speaker embedding model as versatile and suitable for a range of tasks, including speaker verification, speech recognition and diarization. The authors of WavLM assert that their model outperforms others, specifically ECAPA, in speaker verification tasks on the VoxCeleb dataset.

Studies [3,29] present the results of utilizing the Pyannote and ECAPA models in diarization tasks rather than verification tasks.

The authors of [30] aim to review the recent progress in speaker embedding development and to perform an experimental benchmark experimental comparison among the state-of-the-art speaker representations for a speaker verification task. This study uses the VoxCeleb1 dataset for the evaluation of these models. The dataset used in the conducted experiment in this study is in the English language. This study performs training as well as evaluation, and they compare the EER metrics for various types of speaker embedding types, such as x-vectors, d-vectors, r-vectors and others. This study does not consider the inference time nor delves into why these embeddings perform well or not.

Table 1. Experiment results of the study in [30].

Embedding Method	EER (%)
i-vector	10.73
d-vector	6.44
x-vector	3.98
x-vector (E-TDNN)	3.76
x-vector (F-TDNN)	3.53
r-vector (ResNet)	3.18
r-vector (Res2Net)	2.71

presents the results of the experiments of [30], including the EER scores.

In [31], the authors aim to compare the speaker verification performance for children’s and adult’s speech. To achieve this, they use the GMM-based speaker verification system and PF-STAR dataset, which contains children’s English speech utterances.

Our study evaluates the effectiveness of current speaker verification models without additional modifications, emphasizing that the dataset size is not a critical factor for analysis [32,33,34]. This approach is distinct from scenarios where extensive data are essential for training purposes [35,36,37]. We also explore the challenges posed by using brief recordings from real-world environments and examine the performances of these models with Ukrainian speakers, despite the primary training of the models being on English utterances only.

3. Aim of Research

The objective of this study is to identify the most effective state-of-the-art speaker embedding models for verifying speakers, with a special focus on international voices. We assess the models’ ability to verify speakers using a limited set of short utterances, each no longer than 10 s and exclusively in Ukrainian. These utterances were collected from open sources that fall under the definition of “in the wild”. The findings aim to provide a comprehensive guide for speaker verification and offer insights for selecting appropriate speaker embedding models in both research and commercial environments.

4. Models Overview

The models that are used for speaker embedding extraction use different approaches and different neural network architectures to achieve their results and performance. However, these models follow the same generalized architecture, which is presented in Figure 1.

This generalized speaker embedding model architecture consists of three main blocks: feature extraction, deep neural network and speaker labels classifier. A speaker embedding is the feature vector extracted from the output of the deep neural network, with the intention to be used as the input for the speaker classifier at the training stage. The training stage is structured to ensure that the speaker embeddings for different speaker classes are positioned as far apart as possible within a vector space. In

Table 2. Comparison of the speaker embedding model parameters.

Model	Speaker Embedding Dimension	Training Dataset	Architecture
PyAnnote	512	VoxCeleb	X-vector with SincNet
WavLM	256	LibriSpeech	Transformer
TitaNet	192	VoxCeleb, NIST SRE, Fisher, LibriSpeech	ContextNet with channel attention pooling
Ecapa-TDNN	192	VoxCeleb	Improved TDNN

, a comparison of the tested models is presented.

4.1. WavLM

WavLM is a new pre-trained model that leverages self-supervised learning within the HuBERT framework, prioritizing both spoken content modeling and speaker identity retention. This model is pre-trained on large-scale unlabeled data and is able to enhance task performance, diminish the need for data labeling and ease the task-specific adaptation process. The WavLM model learns not only the ASR information by masked speech prediction but also the knowledge of non-ASR tasks by speech denoising modeling. The authors optimize the model structure and training data of HuBERT and wav2vec2. The addition of the gated relative position bias to the Transformer structure as the backbone improves model performance for ASR while keeping almost the same number of parameters and training speed. They also propose an utterance mixing training strategy, where additional overlapped utterances are created unsupervised and incorporated during model training to improve speaker discrimination. The WavLM was pre-trained on a large amount of unlabeled data consisting of 94k h of public audio data (including Libri-Light, GigaSpeech and VoxPopuli datasets). WavLM achieves state-of-the-art performance on the SUPERB benchmark and significant improvements for various speech processing tasks on their representative benchmarks [26].

The WavLM architecture uses the Transformer model as a backbone. It contains a convolutional feature encoder and a Transformer decoder. To improve the model, a gated relative position bias [38] is encoded based on the offset between the “key” and “query” in the Transformer self-attention mechanism. Compared with the convolutional relative position embedding in wav2vec2 and HuBERT, the gates take the content into consideration and adaptively adjust the relative position bias by conditioning on the current speech content.

4.2. PyAnnote

PyAnnote is a framework which provides various modules for speaker diarization, including an embedding module, which is used for speaker embedding extraction [25,39]. PyAnnote speaker embedding model is based on the canonical x-vectors TDNN-base architecture [40] with filter banks replaced by trainable SincNet features [41]. PyAnnote provides pre-trained PyTorch models which share the same generic PyanNet base architecture.

PyAnnote’s speaker embedding model uses a 512-unit-wide and 3-recurrent-layer deep network, which relies on x-vector-like statistical temporal pooling and has been trained on short 500 ms audio chunks. The speaker embedding model was trained on the VoxCeleb dataset.

4.3. TitaNet

TitaNet is a model for a speaker representation extraction, which employs 1D depth-wise convolutions with Squeeze-and-Excitation (SE) layers that integrate global context, and a channel attention-based statistics pooling layer that maps variable-length utterances to fixed-length embeddings (t-vectors) [42]. This model achieves state-of-the-art results in the speaker verification task [27,43]. Authors of TitaNet use the encoder of the ContextNet model as a top-level feature extractor and feed the output to the attentive pooling layer. This layer computes attention features across channel dimensions to capture the time-independent utterance-level speaker representations. The output speaker representation has a fixed size of 192. The training dataset for TitaNet consists of the data from VoxCeleb1 and VoxCeleb2, NIST SRE, Switchboard-Cellular1 and Switchboard-Cellular2, Fisher and LibriSpeech. Combined, these datasets consist of approximately 4.8 M utterances from 16.6K speakers with a total duration of 3.3K h. Additionally, the training process is enhanced with RIR impulse corpora, speed perturbation and spectral augmentation.

4.4. ECAPA

The Ecapa-TDNN model improves the existing x-vector speaker representation extraction architecture with multiple enhancements inspired by the recent advancements in face verification. The authors introduce the Squeeze-and-Excitation blocks into the 1-D Res2Net modules to explicitly model channel interdependencies. An SE block expands the temporal context of the frame layer by rescaling the channels according to the global properties of the recording [42]. Authors also aggregate and propagate features of different hierarchical levels to leverage the ability of the neural networks to learn these hierarchical features. They also improve the statistics pooling module with channel-dependent frame attention, which enables the network to focus on different subsets of frames while estimating statistics. The proposed architecture outperforms state-of-the-art TDNN-based systems on the VoxCeleb test datasets [28]. The Ecapa-TDNN model is trained on the VoxCeleb2 dataset, which includes approximately 5.9K unique speakers. The authors also apply augmentations to the training data audio files, and, as a result, generate six samples per utterance. The aggregations include babble, noise, reverberation, increased and decreased tempo and the effects of using different Opus codec compression settings. The output speaker representation has a fixed size of 192 elements.

5. Experiment Setup

In this study, a custom-created dataset of non-English short utterances was used to evaluate speaker verification models based on speaker embeddings. Comprising speech recordings from 50 Ukrainian politicians (50 unique voices, 33 male and 17 female) sourced from YouTube interviews, the dataset features 10 clips per individual, each up to 10 s long. This amounts to a total of 500 recordings, collectively spanning 1 h and 30 min (about 100 s per speaker). Every entry within the dataset consists of single-channel (mono) audio sampled at 16 kHz, recorded in the .wav format. Additionally, it is worth noting that the age range of the speakers spans from 20 to 70. The dataset is designed to be diverse, including speakers of different genders and ages, to provide a comprehensive evaluation of the speaker verification models. This dataset is an important resource for research in speaker verification and in the field of speaker verification with short-duration recordings.

In order to evaluate the performance of the speaker verification models, a test dataset of 4500 pairs of recordings was created. Half of these pairs consist of recordings of the same speaker, while the other half consists of recordings of different speakers. This allows for a comprehensive evaluation of the models’ ability to accurately identify a speaker and differentiate them from others.

The dataset is organized into 50 distinct classes or speakers, each corresponding to a different speaker. Within the designated directory for each class, there are 10 audio files containing the respective speakers’ voices. The structural layout of the dataset is illustrated in Figure 2.

The test set, derived from the dataset depicted in Figure 2, is specifically tailored for evaluating speaker verification performance. In this restructured dataset, each speaker class yields 90 audio file pairs. These pairs are evenly divided into two subsets: one containing 45 intra-class pairs, where each audio file is matched with another from the same speaker, and another with 45 inter-class pairs, formed by pairing a random audio file from the target class with one from a different, randomly chosen class. The composition and structure of the test set are detailed in Figure 3.

To evaluate and compare each of the speaker verification models’ performance, the Equal Error Rate (EER), False Acceptance Rate (FAR) and False Rejection Rate (FRR) were used, which are popular performance metrics in biometric systems. The False Acceptance Rate (FAR) represents the proportion of incorrect acceptances, indicating the likelihood that an unauthorized speaker is mistakenly verified. The False Rejection Rate (FRR) indicates the proportion of incorrect rejections, reflecting the frequency with which an authorized speaker is erroneously denied. The Equal Error Rate (EER) metric is a measure of the error rate of the system where the threshold is modified in such a way that the False Acceptance Rate and False Rejection Rate are equal [44], providing a single value to summarize the overall trade-off between the FAR and FRR values.

The experiment is conducted in the following steps for each of the pairs in the collected dataset. Both audio files of a pair are passed to the speaker verification model as an input, where the model outputs an embedding. Both output embeddings (which correspond to the input audio files) are compared using the cosine similarity function, which is a number between 0 and 1. Then, based on the retrieved similarity and threshold, a decision is made: if similarity is equal to or greater than the threshold, the decision is to accept (the speakers in the input audio files are the same person), but if it is lower than the threshold, the decision is to reject (the speakers in the input audio files are not the same person). After this process is repeated for each of the pairs in the dataset, the predicted decisions are compared against the ground truth and the metrics are calculated. In Figure 4, the experiment structure is displayed.

All experiments were run on the following hardware: a virtual cloud server (powered by RunPod) with an A4500 GPU with 20 GB of VRAM.

The experiment results are presented in

Table 3. Calculated metrics in the conducted experiment.

Model	FAR (%)	FRR (%)	EER (%)	DCF	Inference Time (ms)
PyAnnote	3.78	3.82	3.8	0.259	49.44 ± 11.97
WavLM-Base-SV	12.53	12.4	12.47	0.445	91.39 ± 12.85
WavLM-Base-Plus-SV	10.84	10.93	10.88	0.407	92.25 ± 17.01
TitaNet-Large	1.91	1.91	1.91	0.138	110.18 ± 8.59
Ecapa	1.73	1.68	1.71	0.139	69.43 ± 8.06

.

From the experiment results, it can be seen that out of the five selected speaker verification models, TitaNet and Ecapa models provide superior speaker verification performance, achieving the lowest rates for FAR, FRR and EER. Pyannote ranks third in performance. In contrast, both variants of the WavLM model have worse results compared to the others. To delve deeper into these results, we computed additional parameters such as embedding distance between different classes and embedding distance between the same class, and the Detection Cost Function (DCF) [45], which combines false acceptance and rejection rates into a single cost metric, reflecting the overall effectiveness of the system. These findings are detailed in

Table 4. Calculated speaker embedding parameters.

Model	The Distance between Samples of Different Classes	The Distance between Samples of the Same Class
PyAnnote	1.298 ± 0.071	0.941 ± 0.117
WavLM-Base-SV	0.654 ± 0.238	0.286 ± 0.075
WavLM-Base-Plus-SV	0.684 ± 0.254	0.286 ± 0.073
TitaNet-Large	1.246 ± 0.084	0.773 ± 0.121
Ecapa	1.293 ± 0.081	0.827 ± 0.119

.

Further analysis of the results shows that the better-performing models, namely TitaNet and Ecapa, exhibit greater disparities between intra-class (same class) and inter-class (different class) distances compared to the less effective models, such as WavLM. Notably, the standard deviation for the different-class distances with the WavLM model is much higher than the other models. These observations explain why the TitaNet and Ecapa models performed so well and why the WavLM model did not. This variability implies that embeddings from WavLM are more dispersed, complicating the accurate differentiation between speakers. Figure 5 visualizes some of the speaker embeddings, offering a clearer perspective on the experimental findings.

The prior observations are further confirmed by the visualizations. It can be seen that the embeddings of the better-performing models (TitaNet, Ecapa) are more tightly clustered among the speakers, which facilitates more accurate speaker verification. In contrast, the WavLM model produces more scattered embeddings, resulting in less distinct groupings, and, consequently, less reliable verification.

6. Conclusions

In summary, our study provides a comprehensive comparison of modern deep learning models for speaker verification based on speaker embeddings on a custom-created dataset. The findings highlight the robust capabilities of speaker embeddings in this domain and provide valuable insights for further research in this field. Out of all the tested speaker verification models, the TitaNet and Ecapa models emerged as superior, with the lowest Equal Error Rates of 1.91% and 1.71%, respectively. While Pyannote trailed with a higher EER of 3.8%, WavLM resulted in an EER of 10.88%. The capability of speaker embedding models to generate distinctly separable embeddings plays a crucial role in a speaker verification system.

A critical aspect of our study was the application of large models to small datasets, specifically for testing purposes. This approach, not intended for training, was instrumental in assessing the models’ efficiency and accuracy in real-world scenarios where data availability (of both language-related datasets and particular speaker samples) can be limited.

The model with the shortest per-sample inference time was Pyannote (49.44 ms), while the longest one was the TitaNet (110.18 ms). Notably, the Ecapa model provides the best combination of short inference time (69.43 ms), which is a little bit longer than PyAnnote, coupled with superior accuracy as reflected by the lowest Equal Error Rate among the models evaluated. This balance is especially relevant in the context of applying large models to smaller datasets for testing, demonstrating the Ecapa model’s adaptability and efficiency in varied data environments.