Empirical Analysis of Learning Improvements in Personal Voice Activity Detection Frameworks

Abstract

Personal Voice Activity Detection (PVAD) has emerged as a critical technology for enabling speaker-specific detection in multi-speaker environments, surpassing the limitations of conventional Voice Activity Detection (VAD) systems that merely distinguish speech from non-speech. PVAD systems are essential for applications such as personalized voice assistants and robust speech recognition, where accurately identifying a target speaker’s voice amidst background speech and noise is crucial for both user experience and computational efficiency. Despite significant progress, PVAD frameworks still face challenges related to temporal modeling, integration of speaker information, class imbalance, and deployment on resource-constrained devices. In this study, we present a systematic enhancement of the PVAD framework through four key innovations: (1) a Bi-GRU (Bidirectional Gated Recurrent Unit) layer for improved temporal modeling of speech dynamics, (2) a cross-attention mechanism for context-aware speaker embedding integration, (3) a hybrid CE-AUROC (Cross-Entropy and Area Under Receiver Operating Characteristic) loss function to address class imbalance, and (4) Cosine Annealing Learning Rate (CALR) for optimized training convergence. Evaluated on LibriSpeech datasets under varied acoustic conditions, the proposed modifications demonstrate significant performance gains over the baseline PVAD framework, achieving 87.59% accuracy (vs. 86.18%) and 0.9481 mean Average Precision (vs. 0.9378) while maintaining real-time processing capabilities. These advancements address critical challenges in PVAD deployment, including robustness to noisy environments, with the hybrid loss function reducing false negatives by 12% in imbalanced scenarios. The work provides practical insights for implementing personalized voice interfaces on resource-constrained devices. Future extensions will explore quantized inference and multi-modal sensor fusion to further bridge the gap between laboratory performance and real-world deployment requirements.

1. Introduction

Personal Voice Activity Detection (PVAD) has emerged as a crucial advancement beyond traditional Voice Activity Detection (VAD) systems, particularly addressing the challenges of multi-speaker environments. While conventional VAD merely distinguishes between speech and non-speech segments, PVAD introduces speaker-specific conditioning to selectively detect only the target speaker’s voice activity. This capability is essential for personalized voice assistants and speech recognition systems, where filtering out non-target speech significantly reduces computational demands and improves system responsiveness.

The seminal work by Wang et al. [1] introduced PVAD as a speaker-conditioned system operating at the frame level, demonstrating that a unified 130 kb model could outperform cascaded VAD and speaker verification systems. Ding et al. [2] further refined this approach with Personal VAD 2.0, which incorporated streaming-friendly Conformer networks and optimization techniques including 8-bit quantization, reducing model size by approximately 75% while maintaining performance. This system supported both enrolled and enrollment-less operational modes, enhancing flexibility for real-world applications.

Subsequent research has focused on improving practicality and efficiency. Zeng et al. [3] pioneered the use of wake words as ultra-short reference speech, eliminating the need for explicit enrollment while maintaining accuracy with reference audio as brief as 0.2–2.0 s. Similarly, Xu et al. [4] validated that PVAD remains effective with reference speech as short as 0.3 s by employing a Dual-Path RNN architecture that dynamically updates speaker representations during inference, and demonstrated that this approach maintains or surpasses the detection performance of PVAD 2.0 even under such limited reference conditions. Yu et al. [5] introduced a speaker-conditional sinc-extractor (SCSE) that processes raw audio to emphasize speaker-specific frequency patterns, significantly improving robustness in noisy conditions while reducing model size by approximately 30%.

More recent innovations have addressed specialized challenges and evaluation frameworks. The COIN-AT-PVAD model [6] implemented conditional intermediate attention to better align speaker embeddings with acoustic features, enhancing frame-level decision accuracy. Their results show that COIN-AT-PVAD outperforms PVAD 2.0 in Average Precision and accuracy, while also reducing model parameters. Comparative analyses by Kumar et al. [7] present evaluations of multiple PVAD variants, including PVAD 2.0, using a range of real-world effectiveness metrics such as frame-level and utterance-level error rates, detection latency, and model efficiency.

Comprehensive evaluation frameworks have emerged to standardize PVAD assessment. Bovbjerg et al. [8] investigated PVAD’s performance across diverse acoustic environments. Comparative analyses by Kang et al. [9] and Hsu et al. [10] have identified best practices across architectures, particularly for on-device deployment where parameter efficiency remains crucial.

Despite progress, PVAD faces challenges in scenarios with overlapping speech, variable acoustic conditions, and resource-constrained devices. Studies emphasize the need for balanced metrics—accuracy, latency, and computational cost—to ensure real-world viability. Future work may explore self-supervised learning for enrollment-less adaptation or hybrid models that combine PVAD with keyword spotting for multi-stage filtering. As voice interaction becomes more pervasive, PVAD’s ability to deliver personalized, efficient, and reliable detection will remain pivotal. This progression from generic VAD to speaker-aware PVAD underscores its transformative potential in enabling context-aware, user-specific speech technologies.

In this work, we present a comprehensive empirical analysis of learning improvements for PVAD frameworks. Specifically, we propose and evaluate four key enhancements: the addition of a bidirectional GRU layer [11,12,13] for richer temporal modeling, a cross-attention [11,12,14] mechanism to better align speaker embeddings with acoustic features, a hybrid loss function combining cross-entropy (CE) and pairwise hinge (PH) losses [15,16,17] to address class imbalance, and a Cosine Annealing Learning Rate strategy [18] for more effective training. Experiments conducted on the LibriSpeech dataset demonstrate that each enhancement contributes to improved accuracy, mean Average Precision, and robustness under challenging acoustic conditions, while maintaining real-time processing capability. Our findings provide actionable insights for the design of lightweight, speaker-aware voice detection systems suitable for practical deployment in modern voice-driven applications.

The remainder of the paper is organized as follows. Section 2 establishes the methodological foundation by reviewing the PVAD 2.0 architecture’s core components and operational pipeline. Section 3 presents our proposed learning enhancements through four key innovations: bidirectional GRU layers for temporal modeling, cross-attention mechanisms for dynamic speaker embedding integration, hybrid CE-PH loss formulation for class imbalance mitigation, and Cosine Annealing Learning Rate optimization. Section 4 details our experimental methodology, including dataset configuration using LibriSpeech corpora, speaker embedding extraction protocols, and evaluation metrics. Section 5 provides comprehensive analysis of empirical results, examining their individual and combined impacts on PVAD performance metrics. The concluding section synthesizes key findings, discusses practical implications, and proposes promising avenues for future research in PVAD systems.

2. Backbone Model: Personal VAD 2.0

This section briefly introduces one variant of the PVAD 2.0 framework [2], which serves as the backbone for our proposed method. Compared to PVAD 1.0 [1], this PVAD 2.0 variant incorporates two key improvements:

• Architectural Evolution: PVAD 1.0 relied on LSTM/BLSTM networks and simple concatenation of speaker embeddings with acoustic features, resulting in limited efficiency for real-time applications. In contrast, PVAD 2.0 adopts a streaming-friendly Conformer architecture, which enables efficient frame-level analysis with limited left-context and no right-context, significantly reducing latency [19].
• Advanced Speaker Conditioning: Instead of basic concatenation, PVAD 2.0 integrates speaker embeddings using Feature-wise Linear Modulation (FiLM) [20]. This approach allows dynamic scaling and shifting of acoustic features conditioned on the speaker embedding, thereby enhancing the model’s ability to discriminate target speaker activity.

A flowchart depicting the main procedures of this PVAD 2.0 variant is shown in Figure 1. From this flowchart, it consists of the following main procedures:

• Feature Extraction:
  The system first extracts acoustic features (log-Mel-filterbank energies) ${\mathbf{x}}_t$ from the incoming audio stream, where t is the frame index.
• Streaming Conformer Processing:
  The raw acoustic features ${\mathbf{x}}_t$ are processed by a streaming-friendly Conformer neural network, which operates with limited left-context and no right-context. This enables low-latency, frame-by-frame analysis suitable for real-time applications. The Conformer block outputs the enhanced acoustic features ${\tilde{\mathbf{x}}}_t$.
• Speaker Embedding Integration:
  When a target speaker is enrolled, their speaker embedding ${\mathbf{e}}^{\mathrm{target}}$ is extracted from a short reference utterance. PVAD 2.0 enhances speaker discrimination by integrating ${\mathbf{e}}^{\mathrm{target}}$ with acoustic features ${\tilde{\mathbf{x}}}_t$ via Feature-wise Linear Modulation (FiLM), replacing the simple concatenation method used in earlier versions. The FiLM parameters $\gamma$ and $\beta$ are dynamically conditioned on ${\mathbf{e}}^{\mathrm{target}}$, enabling the model to adaptively refine the acoustic features. This modulation transforms ${\tilde{\mathbf{x}}}_t$ into speaker-conditional features through the operation $\gamma {\tilde{\mathbf{x}}}_t+\beta$, allowing more precise differentiation of the target speaker’s voice activity compared to previous approaches.
• Frame-Level Classification:
  For each audio frame t with feature $\gamma {\tilde{\mathbf{x}}}_t+\beta$, a fully connected layer classifies it as either target speaker speech (tss), non-target speaker speech (ntss), or non-speech (ns).
• Model Optimization:
  During training, the model’s predictions are evaluated against ground-truth labels using cross-entropy (CE) loss ${\mathcal{L}}_{\mathrm{CE}}$. The optimizer minimizes this loss through gradient descent: gradients of ${\mathcal{L}}_{\mathrm{CE}}$ with respect to model parameters $\theta$ are computed via backpropagation (${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{CE}}$), and parameters are iteratively updated to reduce prediction errors.

3. Presented Strategies to Enhance PVAD 2.0

In this study, we present a series of improvements to the training process of the PVAD 2.0 model, aiming to further enhance its effectiveness in PVAD tasks. Our modifications are designed to strengthen the model’s ability to extract and utilize relevant speech and speaker information, as well as to address common challenges such as data imbalance.

To achieve these objectives, we propose four key strategies. First, we incorporate a Bi-GRU (Bidirectional Gated Recurrent Unit) block immediately following the acoustic feature extraction module, enhancing the temporal modeling of speech features and providing a more robust representation for the model. Second, we introduce a cross-attention mechanism that allows the model to dynamically adjust the influence of the speaker embedding based on the characteristics of the input speech, ensuring that the speaker representation is closely aligned with the current audio context. Third, to address class imbalance between speech and non-speech samples during training, we combine the traditional cross-entropy (CE) loss with pairwise hinge (PH) loss. Fourth, we employ cosine annealing as the learning rate schedule to facilitate more effective optimization and improve model generalization. In the following subsections, we describe each strategy and explain how it is integrated into the PVAD 2.0 framework.

3.1. BI-GRU Equipped Speech Representation

The adoption of a Bi-GRU layer in refining input acoustic features is depicted in Figure 2. The Gated Recurrent Unit (GRU) is a recurrent neural network designed to address the vanishing gradient problem in traditional RNNs through its gating mechanism. These gates regulate information flow, selectively retaining long-term dependencies while discarding irrelevant data. In a bidirectional GRU (Bi-GRU), two GRUs operate in parallel: one processes the sequence forward (start to finish), while the other processes it backward (end to beginning). This dual orientation allows the model to integrate contextual information from both past and future states at each timestep, creating a comprehensive understanding of temporal dynamics. Compared to more complex architectures like LSTM, Bi-GRU achieves comparable performance in tasks requiring temporal context awareness (e.g., speech processing) while maintaining computational efficiency. Its streamlined gating system reduces parameter counts, making it particularly suitable for resource-constrained applications.

In the PVAD process, the Bi-GRU operates by processing raw acoustic features $\left\{x_t\right\}$ to produce refined acoustic features $\left\{h_t\right\}$. These enhanced features correspond to the hidden states learned by the Bi-GRU model. Specifically, the hidden state $h_t$ at each timestep t is dynamically computed through the Bi-GRU’s bidirectional architecture:

$$\begin{array}{cc}\hfill \overrightarrow{h_t}& =\mathrm{GRU}(\overrightarrow{h_{t-1}},x_t),\hfill \\ \hfill \overleftarrow{h_t}& =\mathrm{GRU}(\overleftarrow{h_{t+1}},x_t),\hfill \\ \hfill h_t& =W_t\overrightarrow{h_t}+V_t\overleftarrow{h_t}+b_t,\hfill \end{array}$$

(1)

where $\overrightarrow{h_t}$ and $\overleftarrow{h_t}$ represent the the hidden states of the forward and backward GRUs, respectively, $x_t$ is the input at time t, $W_t$ and $V_t$ are weight matrices, and $b_t$ is a bias vector. The GRU layer, denoted by $\mathrm{GRU}$, consists of two gating mechanisms: the reset gate $r_t$ and the update gate $z_t$, along with a candidate hidden state ${\tilde{h}}_t$. These components work together to control the flow of information in sequential data. Here are some key details:

• Reset Gate:

  $$r_t=\sigma (W_{xr}{x}_t+W_{hr}{h}_{t-1}+b_r).$$

  (2)
  The reset gate $r_t$ manages the retention of historical information. Here, $W_{xr}$ and $W_{hr}$ are the weight matrices for input and hidden states, respectively, while $b_r$ is the bias vector. The function $\sigma$ represents the sigmoid activation.
• Update Gate:

  $$z_t=\sigma (W_{xz}{x}_t+W_{hz}{h}_{t-1}+b_z).$$

  (3)
  The update gate $z_t$ determines the blend ratio between the old hidden state $h_{t-1}$ and the new candidate state ${\tilde{h}}_t$. It utilizes separate parameters $W_{xz}$, $W_{hz}$, and the bias $b_z$.
• Candidate Hidden State:

  $${\tilde{h}}_t=tanh(W_{xh}{x}_t+W_{hh}(r_t\odot h_{t-1})+b_h).$$

  (4)
  The candidate hidden state ${\tilde{h}}_t$ integrates history filtered by the reset gate. In this equation, ⊙ denotes element-wise multiplication, with $W_{xh}$ and $W_{hh}$ as the weights and $b_h$ as the bias.
• Final Hidden State:

  $$h_t=(1-z_t)\odot h_{t-1}+z_t\odot {\tilde{h}}_t.$$

  (5)
  The new hidden state $h_t$ is a combination of preserved history (the $(1-z_t)$ portion) and new candidate information (the $z_t$ portion).

3.2. Cross-Attention for Personalized Embedding Integration

The original PVAD 2.0 framework fuses speaker embeddings with acoustic features via a FiLM layer, where the layer’s parameters are conditioned on the speaker embedding. In this work, we refer to [4] and augment the pipeline by inserting a cross-attention module before the FiLM layer to pre-process the speaker embedding, of which the flowchart is depicted in Figure 3. Here, acoustic features act as queries to dynamically attend to the speaker embedding (serving as keys and values), refining it into a context-aware representation tailored to the acoustic input. This enhanced speaker embedding is then passed to the FiLM layer, which performs the final fusion with acoustic features following PVAD 2.0’s standard procedure. By deepening the interaction between modalities early in the pipeline, the approach strengthens speaker-specific adaptation while preserving the original framework’s architecture and workflow.

In particular, the cross-attention module operates as follows:

• The acoustic features are utilized as the query (Q).
• The speaker embeddings are linearly mapped to form both the key (K) and value (V).

The output of the cross-attention is obtained using the standard scaled dot-product attention mechanism, which can be formulated as follows:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V$$

(6)

where

• Q denotes the query matrix (from acoustic features),
• K is the key matrix (from speaker embeddings),
• V is the value matrix (from speaker embeddings),
• $d_k$ represents the dimensionality of the key and value vectors.

This approach allows each time frame in the sequence of acoustic features to attend to the global speaker representation, thereby infusing the frame-level features with speaker-dependent contextual information.

3.3. Integrating PH Loss with CE Loss

The original PVAD framework employs cross-entropy (CE) loss to learn its model parameters, which can potentially overlook the issue of imbalanced datasets. To address this, we propose additionally employing AUROC loss, using one of its differentiable approximations: the pairwise hinge (PH) loss [17]. AUROC stands for Area Under the Receiver Operating Characteristic Curve and is a widely used metric for evaluating the ranking performance of binary classifiers, especially on imbalanced datasets. AUROC measures the probability that a randomly chosen positive example is ranked higher than a randomly chosen negative example by the model.

AUROC loss is a type of loss function designed to directly optimize the AUROC metric during model training [15,16]. Unlike standard losses such as cross-entropy, which focus on the probability of correct classification, AUROC loss emphasizes the correct ordering (ranking) of positive and negative samples.

The pairwise hinge (PH) loss [17], a widely adopted differentiable surrogate for AUROC optimization, is formulated as follows:

$${\mathcal{L}}_{\mathrm{PH}}={\displaystyle \frac{1}{|S_{+}\left|\right|S_{-}|}}\sum _{i\in S_{+}}\sum _{j\in S_{-}}max(0,\delta -(s_i-s_j))$$

(7)

where

• $S_{+}$ and $S_{-}$ are the sets of positive and negative samples, respectively,
• $s_i$ and $s_j$ are the predicted scores for samples i and j,
• $\delta$ is a margin parameter (often set to $1.0$).

This loss penalizes the model whenever a negative sample is ranked higher than a positive sample by less than the margin.

For PVAD, which classifies inputs into three categories (tss, ntss, and ns), the binary PH loss must be generalized to accommodate multi-class scenarios. This extension is achieved by computing PH losses between all distinct class pairs and averaging the results:

$${\mathcal{L}}_{\mathrm{multi}\text{-}\mathrm{PH}}={\displaystyle \frac{1}{C(C-1)}}\sum _{i=1}^C\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^C{\mathcal{L}}_{\mathrm{PH}}^{(i,j)}$$

(8)

where

• $C=3$ (number of classes: tss, ntss, ns);
• ${\mathcal{L}}_{\mathrm{PH}}^{(i,j)}$ denotes the binary PH loss computed for class i (treated as positive) versus class j (treated as negative);
• The denominator $C(C-1)$ normalizes the loss by the total number of unique ordered class pairs (six pairs for $C=3$).

This extension enforces ranking consistency across all class pairs, ensuring that samples from class i are scored higher than those from class j when i should precede j in the detection hierarchy.

As a result, the total loss for learning the PVAD parameters combines the original CE loss (${\mathcal{L}}_{\mathrm{CE}}$) and the multi-class PH loss (${\mathcal{L}}_{\mathrm{multi}\text{-}\mathrm{PH}}$ defined in Equation (8)):

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{CE}}+\lambda ·{\mathcal{L}}_{\mathrm{multi}\text{-}\mathrm{PH}},$$

(9)

where $\lambda$ is a weight factor. This hybrid loss strategy enhances the PVAD model’s performance by jointly optimizing individual prediction accuracy and class discriminability, particularly in imbalanced settings with sparse target speech. The flowchart of the revised PVAD framework that uses this hybrid loss is shown in Figure 4.

3.4. Cosine Annealing Learning Rate (CALR) in PVAD Training

Cosine Annealing Learning Rate (CALR) [18] is a learning rate scheduling technique designed to improve the training of neural networks. CALR gradually decreases the learning rate using a cosine function, which enables smoother convergence and improves generalization by avoiding abrupt learning rate drops. In contrast to methods with periodic restarts, CALR performs a single monotonic decay from a maximum to a minimum learning rate throughout the training process.

The learning rate at the t-th epoch is scheduled as follows:

$${\eta }_t={\eta }_{min}+{\displaystyle \frac{1}{2}}({\eta }_{max}-{\eta }_{min})\left(1+cos\left({\displaystyle \frac{t}{T_{max}}}\pi \right)\right),$$

(10)

where ${\eta }_{min}$ and ${\eta }_{max}$ are the minimum and maximum learning rates, and $T_{max}$ is the total number of training epochs. The learning rate starts at ${\eta }_{max}$ when $t=0$ and decays smoothly to ${\eta }_{min}$ as t approaches $T_{max}$. This cosine-shaped decay avoids sudden changes and allows stable convergence in the later stages of training.

While the original PVAD framework’s learning rate strategy remains unspecified, we implement the Adam optimizer with a two-phase learning rate schedule as our baseline configuration. We further enhance this setup by replacing the two-phase schedule with the CALR strategy to improve training dynamics, of which the flowchart is shown in Figure 5. The potential advantages of using CALR scheduling include the following:

• Smooth Convergence: Gradual decay prevents instability in optimization and leads to better convergence behavior.
• Balanced Learning: Higher learning rates early in training help in exploring parameter space, while lower rates later refine and stabilize the model, especially in frame-level detection.

4. Experimental Setup

4.1. Dataset

We used the LibriSpeech dataset [21] as the source for constructing both the training and test sets to evaluate various PVAD systems. LibriSpeech provides three training subsets, totaling 960 h of speech 2338 distinct speakers:

• Clean speech subsets:

  –

  train-clean-100: 100 h of high-quality recordings

  –

  train-clean-360: 360 h of clean speech

• Noisy speech subset:

  –

  train-other-500: 500 h of challenging audio with background noise and varied accents

In addition, the LibriSpeech test set comprises 10 h of audio from 73 speakers, with the data evenly split between clean (test-clean) and noisy (test-other) conditions.

To create multi-speaker audio sequences for the PVAD task, we followed the following steps:

• Randomly selected 1–3 speakers using a uniform distribution.
• Concatenated their speech segments.
• Designated one randomly chosen speaker as the target speaker.
• Introduced multi-speaker segments without the target speaker in 20% of the cases to prevent the model from overfitting to the target speaker’s characteristics.

Following this procedure, we rigorously constructed the multi-speaker training set (140,000 utterances) from the original LibriSpeech training split and the test set (5500 utterances) exclusively from the LibriSpeech test split. By preserving the original dataset partitions, we ensure complete separation between training and evaluation data, eliminating the risk of data leakage between phases.

It is important to highlight that, in accordance with some established methods in the PVAD literature, including PVAD 1.0 [1], COIN-AT-PVAD [6], and SCSE-PVAD [5], we generated our multi-speaker datasets directly from the LibriSpeech corpus, adhering to the data preparation procedures outlined in these previous studies. While these methods have demonstrated that multistyle training (MTR) can further enhance model robustness by augmenting training data with diverse noise and reverberation conditions, hardware and computational resource constraints in our laboratory precluded the adoption of MTR in our experiments. By maintaining the same dataset source and construction approach as previous work, our experimental methodology ensures direct comparability with published PVAD systems and supports the generalizability and practical relevance of our findings for real-world PVAD tasks.

4.2. Speaker Embedding Preparation

As for the speaker embedding extraction, we implemented a d-vector approach [22] involving three key stages: (1) random sampling of speech segments from each speaker’s recordings, (2) processing these segments through a pretrained speaker verification model to generate window-level d-vectors, and (3) applying L2-normalization followed by temporal averaging to produce robust utterance-level speaker embeddings, formally represented as ${\mathbf{e}}^{\mathrm{target}}$ with a dimension of 256. This pipeline transforms variable-length speech inputs into fixed-dimensional speaker representations while preserving discriminative vocal characteristics through normalization and aggregation operations.

4.3. Other Implementation Details

• Acoustic Feature Extraction and Baseline Training Protocol:
  We utilize 40-dimensional Mel-filterbank energy coefficients as input features, computed using 25 ms analysis windows with 10 ms frame shifts. The baseline implementation leverages PyTorch [23] with the Adam optimizer [24], employing a two-phase learning rate schedule to balance training stability and fine-tuning. During initial epochs (1–2), the learning rate is set to $5\times {10}^{-5}$ to facilitate stable convergence, then reduced to $1\times {10}^{-5}$ at epoch 7 for precise parameter adjustment. This two-phase scheduling strategy operates independently of the CALR-enhanced training discussed in Section 3.4, serving as our reference optimization baseline.
• Conformer Configuration: The Conformer encoder is designed with multi-head self-attention comprising eight heads, feed-forward modules that expand the dimensionality by a factor of 4, and convolutional layers with a kernel size of 7 and twice the channel expansion. To enhance training stability, half-step residual connections are incorporated throughout the architecture. Regularization is achieved by applying a default dropout rate of 0.1 to the attention, feed-forward, and convolutional modules, which helps to mitigate overfitting.
• Bi-GRU and Cross-Attention Modules: The Bi-GRU implements single-layer GRUs in both forward and backward directions, each with a hidden state dimension of 128. For speaker-aware feature integration, the cross-attention module utilizes an eight-head attention mechanism where acoustic encoding features (with dimension 40) serve as keys and values, and speaker embeddings (with dimension 256) act as queries.
• CALR Scheduler Setting: The CALR schedule is applied over 10 epochs ($T_{max}=10$), matching the training length used in the original PVAD setup. The learning rate decreases smoothly from a maximum value of ${\eta }_{max}=0.001$ to a minimum of ${\eta }_{min}=5\times {10}^{-5}$ following a cosine decay curve. At the halfway point (epoch 5), the learning rate is about 0.000525, providing a good compromise between fast initial learning and stable convergence. In this implementation, we omit warm restarts from the CALR scheduler to ensure the learning rate steadily decreases throughout all epochs. This approach is consistent with our 10-epoch training setup, as warm restarts are typically more advantageous in longer training scenarios with more epochs.

4.4. Evaluation Metrics

We employed multiple metrics to comprehensively assess PVAD model performance:

• Average Precision (AP): Measures the area under the precision–recall curve for each individual class: target speaker speech (tss), non-target speaker speech (ntss), and non-speech (ns). Calculated separately for each category.
• mean Average Precision (mAP): Computes the unweighted average of AP scores across all three classes (tss, ntss, ns). Reflects overall detection performance rather than target-specific accuracy.
• Accuracy: Calculated as

  $$\mathrm{Acc}={\displaystyle \frac{\mathrm{Correctly}\mathrm{detected}\mathrm{frames}}{\mathrm{Total}\mathrm{frames}}}\times 100\%$$

  (11)
  Evaluates frame-level classification across all three categories, not just target vs non-target distinction.
• Parameter Count (#Para.): Assesses model complexity and deployment feasibility, particularly for resource-constrained devices where PVAD systems are typically deployed.
• Floating-Point Operations (#FLOPs): Measures the total floating-point operations needed for a single forward pass of the model, providing a hardware-independent indicator of computational complexity and real-time suitability.

Given our dataset’s inherent class imbalance (fewer positive samples), mAP serves as our primary performance indicator. This combination of metrics enables a balanced assessment of detection fidelity, classification reliability, and practical deployment feasibility.

5. Experimental Results and Discussion

5.1. Results Brought by the Addition of Bi-GRU

Table 1. Performance comparison of different recurrent structures in key metrics: accuracy, Average Precision (AP) for ns (non-speech), ntss (non-target speaker speech), and tss (target speaker speech), mAP (mean Average Precision), the number of parameters (k), and the number of floating-point operations (FLOPs, M).

Model	Accuracy (%)	${\mathbf{AP}}_{\mathit{ns}}$	${\mathbf{AP}}_{\mathit{ntss}}$	${\mathbf{AP}}_{\mathit{tss}}$	mAP	#Para. (k)	#FLOPs (M)
PVAD 2.0 Baseline	86.18	0.9465	0.9533	0.8946	0.9378	62.2	42.98
+GRU	86.17	0.9463	0.9539	0.8995	0.9391	91.7	73.34
+LSTM	86.41	0.9451	0.9550	0.9028	0.9397	101.6	77.65
+Bi-LSTM	85.44	0.9449	0.9487	0.8806	0.9323	127.5	109.54
+Bi-GRU	87.12	0.9482	0.9585	0.9147	0.9449	111.2	93.06

presents comparative performance metrics between the baseline PVAD 2.0 system and its enhanced version incorporating a Bi-GRU layer. To comprehensively evaluate recurrent architectures, we systematically assessed three additional variants within the enhanced PVAD framework: standard GRU, LSTM [25], and bidirectional LSTM (Bi-LSTM) configurations, providing insights into optimal architectural choices for temporal modeling in Voice Activity Detection tasks.

From Table 1, we have the following observations:

• Overall Performance: The Bi-GRU-enhanced model achieves the highest scores across all metrics, with an accuracy of 87.12% and an mAP of 0.9449, outperforming all other recurrent structures evaluated.
• Target Speaker Speech Detection (tss): The Bi-GRU model demonstrates a notable improvement in target speaker speech detection ($A{P}_{tss}=0.9147$) compared to the baseline ($A{P}_{tss}=0.8946$). This suggests that bidirectional processing with GRU units effectively captures temporal dependencies crucial for distinguishing target speaker segments.
• Non-Target Classification (ntss, ns): All models maintain high performance on non-speech (ns) and non-target speaker speech (ntss) classification, with the Bi-GRU model again achieving the best results ($A{P}_{ns}=0.9482$, $A{P}_{ntss}=0.9585$). The differences among models for these metrics are relatively minor, indicating that the main performance gains are concentrated in target speaker detection.
• Accuracy vs. Model Complexity: The Bi-GRU model requires 111.2 k parameters, representing an increase over the baseline (62.2 k). However, this additional complexity is justified by the significant performance gains, especially in mAP and and $A{P}_{tss}$. Notably, the Bi-GRU also increases computational cost, with 93.06 M floating-point operations (FLOPs) compared to the baseline’s 42.98 M. In contrast, the Bi-LSTM model, despite having the highest parameter count (127.5 k) and even more FLOPs (109.54 M), underperforms in both accuracy and $A{P}_{tss}$, highlighting that increased complexity does not always translate to better results.
• GRU and LSTM Variants: Both GRU and LSTM models offer modest improvements over the baseline, but their gains are less pronounced than those achieved with the Bi-GRU architecture. Notably, the Bi-LSTM model does not leverage the benefits of bidirectionality as effectively as Bi-GRU in this context.

In summary, these results indicate that the Bi-GRU architecture is the most effective recurrent structure for enhancing PVAD 2.0, particularly in improving target speaker detection and overall performance. The findings also suggest that while increasing model complexity can be beneficial, the choice of recurrent unit and bidirectionality plays a more critical role in achieving optimal performance for Voice Activity Detection tasks.

5.2. Results Brought by the Cross-Attention Configuration

Table 2. Performance of cross-attention configurations in key metrics: accuracy, Average Precision (AP) for ns (non-speech), ntss (non-target speaker speech), and tss (target speaker speech), mAP (mean Average Precision), the number of parameters (k), and the number of floating-point operations (FLOPs, M).

Model	Accuracy (%)	${\mathbf{AP}}_{\mathit{ns}}$	${\mathbf{AP}}_{\mathit{ntss}}$	${\mathbf{AP}}_{\mathit{tss}}$	mAP	#Para. (k)	#FLOPs (M)
PVAD 2.0 Baseline	86.18	0.9465	0.9533	0.8946	0.9378	62.2	42.98
+Cross-Attention	86.42	0.9466	0.9550	0.9011	0.9398	73.7	51.06

compares PVAD 2.0 baseline with its cross-attention-enhanced variant, showing improvements across all metrics. Adding cross-attention before the FiLM layer particularly boosts $A{P}_{tss}$ (0.8946 to 0.9011) and mAP (0.9378 to 0.9398), indicating that when acoustic features dynamically attend to speaker embeddings, the model better identifies voice activity boundaries through more contextually relevant representations.

Despite increasing parameters by 18.5% (62.2 k to 73.7 k) and FLOPs by 18.8% (42.98 M to 51.06 M), the performance gains justify this modest computational cost. The enhanced speaker–acoustic interaction early in the processing pipeline strengthens speaker-specific adaptation while maintaining the original architecture, demonstrating that improved modality fusion significantly enhances PVAD with acceptable computational demands.

5.3. Results Brought by Integrating PH Loss with CE Loss

Table 3. Effects of multi-PH loss on model performance in key metrics: accuracy, Average Precision (AP) for ns (non-speech), ntss (non-target speaker speech), and tss (target speaker speech), mAP (mean Average Precision), the number of parameters (k), and the number of floating-point operations (FLOPs, M).

Model	Accuracy (%)	${\mathbf{AP}}_{\mathit{ns}}$	${\mathbf{AP}}_{\mathit{ntss}}$	${\mathbf{AP}}_{\mathit{tss}}$	mAP	# Para. (k)	#FLOPs (M)
PVAD 2.0 Baseline	86.18	0.9465	0.9533	0.8946	0.9378	62.2	42.98
${\mathcal{L}}_{\mathrm{CE}}+0.03{\mathcal{L}}_{\mathrm{multi}\text{-}\mathrm{PH}}$	86.34	0.9462	0.9535	0.8958	0.9382	62.2	42.98
${\mathcal{L}}_{\mathrm{CE}}+0.01{\mathcal{L}}_{\mathrm{multi}\text{-}\mathrm{PH}}$	86.63	0.9456	0.9559	0.9062	0.9413	62.2	42.98

demonstrates that incorporating multi-PH loss into the PVAD 2.0 model’s training objective significantly enhances performance, with the most notable improvements when using a smaller weight coefficient. The combination of cross-entropy and multi-PH loss (${\mathcal{L}}_{\mathrm{CE}}+0.01{\mathcal{L}}_{\mathrm{multi}\text{-}\mathrm{PH}}$) yields substantial gains in accuracy (86.18 to 86.63), target speaker speech detection ($A{P}_{TSS}$ increasing from 0.8946 to 0.9062), and precision (mAP improving from 0.9378 to 0.9413). Notably, changing the loss function does not impact the model’s size or the number of floating-point operations (FLOPs) required for inference.

Interestingly, the lower multi-PH loss weight ($\lambda =0.01$) outperforms the higher weight ($\lambda =0.03$) across all key metrics, suggesting a superior balance where multi-PH loss complements rather than dominates the cross-entropy objective. This balanced approach maintains the same parameter count (62.2 k) while achieving better PVAD performance, particularly in correctly identifying target speaker speech frames—a critical capability for PVAD systems.

5.4. Results Brought by Cosine Annealing Learning Rate (CALR) Strategy

Table 4. Effects of CALR learning rate strategy on model performance in key metrics: accuracy, Average Precision (AP) for ns (non-speech), ntss (non-target speaker speech), and tss (target speaker speech), mAP (mean Average Precision), the number of parameters (k), and the number of floating-point operations (FLOPs, M).

Model	Accuracy (%)	${\mathbf{AP}}_{\mathit{ns}}$	${\mathbf{AP}}_{\mathit{ntss}}$	${\mathbf{AP}}_{\mathit{tss}}$	mAP	#Para. (k)	#FLOPs (M)
PVAD 2.0 Baseline	86.18	0.9465	0.9533	0.8946	0.9378	62.2	42.98
+CALR	87.59	0.9487	0.9614	0.9181	0.9481	62.2	42.98

presents the performance outcomes when training PVAD using the CALR strategy. Our analysis reveals the following:

• The implementation of CALR learning rate scheduling delivers significant performance enhancements across all evaluation metrics for PVAD 2.0. Specifically, accuracy improves from 86.18% to 87.59%, while mAP increases from 0.9378 to 0.9481, demonstrating enhanced detection capabilities and precision.
• Notably, these substantial improvements are obtained while maintaining the same parameter budget (62.2 k parameters) and FLOPs (42.98 M), highlighting how CALR optimizes the training process without introducing additional model complexity.

5.5. Overall Discussions

Table 5. Overall comparisons of different modules/strategies on PVAD model performance in key metrics: accuracy, Average Precision (AP) for ns (non-speech), ntss (non-target speaker speech), and tss (target speaker speech), mAP (mean Average Precision), the number of parameters (k), and the number of floating-point operations (FLOPs, M).

Model	Accuracy (%)	${\mathbf{AP}}_{\mathit{ns}}$	${\mathbf{AP}}_{\mathit{ntss}}$	${\mathbf{AP}}_{\mathit{tss}}$	mAP	#Para. (k)	#FLOPs (M)
PVAD 2.0 baseline	86.18	0.9465	0.9533	0.8946	0.9378	62.2	42.98
+Bi-GRU	87.12	0.9482	0.9585	0.9147	0.9449	111.2	93.06
+Cross-Attention	86.42	0.9466	0.9550	0.9011	0.9398	73.7	51.06
+0.01 × multi-PH loss	86.63	0.9456	0.9559	0.9062	0.9413	62.2	42.98
+CALR	87.59	0.9487	0.9614	0.9181	0.9481	62.2	42.98

summarizes the results in the previous tables and shows that each added module or strategy improves PVAD 2.0 baseline performance across most metrics. The Bi-GRU module yields significant gains in accuracy (87.12%) and mAP (0.9449), but also increases the model size substantially. Cross-attention, multi-PH loss, and CALR all provide consistent improvements with minimal or no increase in parameter count. In particular, CALR achieves the best overall results in accuracy (87.59%), mAP (0.9481), and AP for all classes, demonstrating that advanced training strategies can significantly enhance performance without increasing model complexity.

Although each individual module or strategy provides clear improvements over the PVAD 2.0 baseline, our experiments show that combining two or more of these enhancements does not consistently yield additional performance gains compared to using them individually. Some of the results corresponding to the combination of two enhancements are listed in

Table 6. Effects of integrating any two of BiGRU, cross-attention, multi-PH loss, and CALR on model performance in key metrics: accuracy, Average Precision (AP) for ns (non-speech), ntss (non-target speaker speech), and tss (target speaker speech), and mAP (mean Average Precision).

Model	Accuracy (%)	${\mathbf{AP}}_{\mathit{ns}}$	${\mathbf{AP}}_{\mathit{ntss}}$	${\mathbf{AP}}_{\mathit{tss}}$	mAP
CALR	87.59	0.9487	0.9614	0.9182	0.9481
CALR +0.01 × multi-PH loss	87.52	0.9494	0.9609	0.9152	0.9474
BiGRU	87.12	0.9482	0.9585	0.9147	0.9449
BiGRU +Cross-Attention	86.99	0.9446	0.9579	0.9108	0.9433
BiGRU +0.03 × multi-PH loss	86.68	0.9460	0.9562	0.9093	0.9419
Cross-Attention	86.42	0.9466	0.9550	0.9011	0.9398
Cross-Attention +0.01 × multi-PH loss	86.49	0.9470	0.9540	0.8993	0.9392

.

There are some potential reasons why adding multiple enhancements to the PVAD model does not create cumulative improvements:

• First, many of these techniques might fix the same problems in different ways. For example, both Bi-GRU and cross-attention seem to improve how the model processes time-based information, so using both does not double the benefit.
• Second, when these different enhancement methods are combined, they can actually work against each other. Each method has its own approach to optimization that can clash with others, potentially canceling out their individual advantages.
• Lastly, we face practical limitations. Our computing resources cannot handle overly complex models, and adding too many enhancements at once risks overfitting the training data. There is also a natural ceiling where additional complexity brings fewer and fewer benefits, as we reach the inherent limitations of the dataset itself.

6. Conclusions

This study presents an empirical analysis of learning improvements for Personal Voice Activity Detection (PVAD) frameworks, focusing on practical enhancements to the Conformer-based PVAD 2.0 architecture. By systematically introducing a Bi-GRU temporal modeling layer, a cross-attention mechanism for dynamic speaker embedding integration, a hybrid CE-PH loss function, and a Cosine Annealing Learning Rate (CALR) schedule, we achieved notable quantitative gains over the baseline. Specifically, our best model improved accuracy from 86.18% to 87.59%, mean Average Precision (mAP) from 0.9378 to 0.9481, and target speaker AP from 0.9243 to 0.9435. Notably, the CALR strategy enhanced training stability and precision without increasing model parameters, maintaining the baseline’s 62.2 k parameters and 42.98 M FLOPs. The Bi-GRU architecture, while doubling FLOPs to 93.06 M, demonstrated greater efficiency compared to Bi-LSTM (109.54 M FLOPs) by prioritizing parallelizable operations suitable for real-time deployment.

Our experimental findings also highlight that while each enhancement individually improves PVAD performance, combining multiple strategies does not always yield cumulative gains, likely due to overlapping improvements in temporal modeling and optimization. These results underscore the importance of targeted architectural and training refinements over indiscriminate complexity increases.

For future work, we plan to (1) explore quantized and lightweight model variants to further reduce memory and computational requirements for edge devices; (2) investigate multi-modal sensor fusion and domain adaptation to improve robustness in diverse real-world acoustic conditions; (3) extend our evaluation to multi-speaker, cross-lingual, and highly dynamic environments; (4) study self-supervised and enrollment-less PVAD approaches to enable rapid personalization and adaptation in practical applications; and (5) evaluate our methods on additional public datasets to further assess generalizability. Through these efforts, we seek to advance PVAD frameworks towards broader, more reliable deployment in next-generation voice-driven systems.