Enhancing Audio–Visual Synchronization and Spatiotemporal Expressiveness for Talking Face Generation

Abstract

Talking face generation aims to produce high-fidelity, temporally coherent videos of speakers with synchronized lip movements aligned to input audio. neural radiance fields (NeRF) are widely adopted due to their realistic modeling capabilities. However, existing NeRF-based approaches face several challenges. First, background noise often disrupts lip synchronization, making it difficult to align lip movements accurately with audio signals, especially when training data are temporally constrained. Furthermore, these methods suffer from spatiotemporal inconsistency, which manifests in two ways: temporally, unreliable audio signals lead to flickering lip movements, undermining coherence; spatially, the lack of facial structure constraints reduces realism and hinders training efficiency. To address these issues, we propose a NeRF-based method that enhances audio–visual synchronization and SpatioTemporal expressiveness for talking face generation (AVIST). Specifically, we enhance the saliency of human speech in audio using audio event features, effectively suppressing background noise interference during training and inference to improve lip-sync accuracy. Additionally, we introduce a feedback mechanism that incorporates lip features from preceding frames to stabilize current lip movements, mitigating temporal instability. Finally, we integrate facial depth supervision to expedite network training and enhance spatial consistency, resulting in more realistic face rendering. Extensive experiments on mainstream datasets demonstrate that AVIST achieves state-of-the-art performance in lip synchronization, spatiotemporal stability, and overall visual fidelity.

1. Introduction

Talking face generation (TFG) aims to produce high-fidelity speaker videos with synchronized lip movements [1], serving as a cornerstone for applications in film production [2], digital avatar creation, and virtual assistants [3]. With the proliferation of Extended Reality technologies, such as Virtual Reality (VR) [4] and Augmented Reality (AR), TFG has emerged as a crucial technology to enable more life-like and immersive virtual interaction experiences. Compared with traditional Computer Graphics (CG) rendering pipelines, neural-rendered talking faces offer greater realism and immersion, especially in VR-based remote communication. In such scenarios, headset occlusion prevents capturing facial images through conventional cameras, making speech-driven 3D digital humans a crucial medium for semantic communication [5].

Recent advancements in TFG can be broadly categorized into two primary approaches: (1) 2D generative models, and (2) 3D reconstruction methods, which include multi-stage neural radiance field (NeRF)-based techniques, end-to-end NeRF-based strategies, and Gaussian splatting-based methods. Among them, 2D generative models [6,7,8] offer the advantage of requiring only minimal fine-tuning for character-specific images but often struggle to preserve geometric details, leading to artifacts such as facial flickering or anatomical misalignment. Multi-stage NeRF-based methods [9,10] utilize intermediate representations such as facial landmarks and rely on multi-step inference. Although these approaches are effective, they are susceptible to cumulative errors and impose significant computational overhead. In contrast, end-to-end NeRF-based methods have gained prominence due to their streamlined training processes, superior inference efficiency, and avoidance of error propagation. Leveraging NeRF’s capability for photorealistic 3D modeling, these approaches excel in rendering fine-grained spatial details and maintaining character identity. However, despite these advantages, challenges remain in deploying such methods effectively in real-world scenarios.

A critical issue in existing end-to-end NeRF-based methods is their vulnerability to background noise interference during both training and inference. In real-world datasets, as shown in Figure 1, audio inputs often contain non-speech elements such as environmental noise or silence, which can lead to mismatched audio–visual labels. During training, this mismatch disrupts the model’s ability to establish reliable speech-to-lip correlations. Compounding this, during inference, the model’s sensitivity to noisy audio can result in anomalous lip movements that erroneously respond to irrelevant acoustic signals, undermining lip synchronization accuracy and naturalness.

Figure 1. Due to insufficiently clean speaker video data, mismatched audio–visual labels between background sounds and the character’s mouth shape are formed, which affects the lip-synchronization accuracy of the model’s inference.

Beyond noise interference, these methods also face limitations in spatiotemporal expressiveness, which compromises their ability to generate coherent and realistic talking face videos. Temporally, the reliance on implicit temporal information from speech audio poses significant challenges. Audio instability, such as weak vocal signals or brief pauses, disrupts the generation of smooth and consistent lip movements, resulting in temporal flickering and disjointed transitions between frames. Spatially, the absence of explicit geometric constraints hinders the accurate reconstruction of facial structures. Without additional supervision, NeRF-based models struggle to capture fine-grained facial details, leading to artifacts and degraded spatial consistency.

To address these limitations, we propose AVIST, a NeRF-based framework characterized by seamless audio–visual synchronization and strong spatiotemporal coherence. To mitigate background noise, we incorporate an audio event detection model. By isolating human speech, this approach encourages the model to focus on the intrinsic relationship between speech content and lip motion, suppressing noise-induced artifacts and spurious movements during inference. Unlike conventional signal-level denoising, our semantic-level gating mechanism explicitly models which audio segments should be associated with visual supervision. These strategies can naturally coexist: low-level denoising improves signal quality, while our semantic gating further refines alignment by suppressing acoustically present but semantically irrelevant signals, enabling stable lip motion in low-SNR scenarios. In addition to audio enhancement, we introduce a lightweight lip-shape feedback mechanism to alleviate temporal inconsistencies. By integrating encoded lip geometry from the preceding frame into the current generation process, this design provides an explicit structural constraint that enforces frame-to-frame continuity. This mechanism effectively reduces flickering artifacts and ensures smooth transitions. Unlike parameter-heavy recurrent architectures (e.g., RNNs or GRUs), our approach efficiently prevents abrupt changes between adjacent frames, better matching the requirements of talking-face generation. Finally, to improve spatial consistency and accelerate convergence, we introduce depth supervision derived from a 3D Morphable Model (3DMM) [11]. Utilizing approximate facial depth labels generated through a depth renderer, we impose geometric constraints on the learning process. The inclusion of adaptive scale depth loss further enhances the network’s ability to capture fine-grained facial structures, resulting in more realistic reconstructions and reduced training time.

Experimental evaluations demonstrate that AVIST significantly outperforms existing methods across multiple benchmarks. By addressing the intertwined challenges of noise interference, temporal instability, and spatial inconsistency, our approach achieves state-of-the-art results in lip synchronization accuracy, temporal smoothness, and visual realism. Specifically, in terms of generating quality, our method achieves a PSNR improvement of 1.908 compared to the baseline. For lip synchronization, the LSE-C and LSE-D metrics show respective increases of 1.002 and 0.822. These results underscore the robustness and practicality of AVIST, facilitating more reliable applications in talking face generation. Our main contributions can be summarized as follows:

• We propose AVIST, an end-to-end framework designed to improve the reliability of talking face generation under unconstrained audio–visual conditions.
• To address the issue of mismatched audio–visual labels caused by noisy or low signal-to-noise ratio training data, we integrate an audio event module, which enhances the model’s focus on meaningful speech elements during training and inference, significantly improving lip synchronization accuracy while mitigating errors caused by irrelevant background sounds.
• To enhance spatiotemporal expressiveness, we introduce a lip shape feedback mechanism that ensures temporal continuity by constraining the generation of each frame with motion cues from the previous frame. Furthermore, we employ a scale-adaptive depth loss with 3DMM-based facial depth supervision to improve spatial consistency and rendering quality, reducing artifacts and structural instability.
• Qualitative and quantitative experiments demonstrate the effectiveness of our approach, achieving state-of-the-art levels in both final rendering quality and lip-sync performance.

2. Related Works

2.1. 2D-Based Talking Face Generation

Talking face generation (TFG) aims to generate high-quality personalized talking videos driven by human speech while ensuring audio–lip synchronization. In 2D-based approaches, generative adversarial models (GAN) [6,7,8,12,13,14,15,16,17] or diffusion models [18,19,20] have been widely used to model the relationship between audio and lip movements. These methods typically require only a few reference images or a single image for generating talking videos, allowing for minimal speaker-specific fine-tuning and even enabling one-shot learning. Few-shot TFG [6,12,21] uses a small set of images as references to generate lip movements that synchronize with the input audio. One-shot methods [22,23] aim to extract identity information from a single image and generate target videos with synchronized lip movements, facial expressions, and head motions. While early 2D-based methods focused primarily on improving lip synchronization, later works [22,23,24] extended this by introducing facial expression coefficients and pose features, making the generated talking faces more controllable. However, despite these advancements, 2D-based methods suffer from several limitations. The most notable of these is the lack of consistency in spatial geometric features, particularly when handling head pose control and preserving identity across different poses [12,25]. When a small number of reference images is provided, large pose variations can result in significant identity inconsistencies, further amplified by the lack of explicit 3D spatial constraints.

2.2. 3D-Based Talking Face Generation

Three-dimensional-based talking face generation has emerged as a solution to the limitations of 2D methods, providing more accurate and realistic representations of faces in a 3D space. Compared with traditional CG-based facial animation pipelines [11], which rely on explicit geometric modeling, rigging, and carefully designed blendshape systems, neural rendering approaches enable more flexible and data-driven facial motion synthesis while significantly reducing manual intervention, and often achieve higher rendering fidelity and visual realism when modeling real human faces. With its high-fidelity novel view synthesis capabilities, NeRF [26] has shown remarkable effectiveness in the field of multi-view scene reconstruction. It achieves photo-realistic generation results without the need for explicit scene modeling. Combined with deformable NeRF, many works [27,28,29,30,31,32] have established the basic pipeline for audio-driven talking face synthesis. They can achieve neural dynamic modeling of an individual with just 3-5 min of a specific speaker video without requiring pre-training on large-scale datasets.

ADNeRF [27] encodes the speech information and inserts it into the spatial location information of the line-of-sight sampling points, then predicts the appearance information of the sampling points through a large-scale Multilayer Perceptron (MLP) [33], and obtains the final portrait which matches the input audio through volume rendering. Due to the slow convergence and inference speed inherent in training large-scale MLP networks, the original NeRF approach is inadequate for meeting the real-time rendering requirements of talking face images.

Then, drawing inspiration from instant-ngp’s [34] spatial voxel segmentation and hash encoding strategies, some studies [29,30] use the Tri-Plane Hash Representation as spatial position encoding and employ small-scale MLP decoding to obtain corresponding spatial representations, significantly improving training and inference speed. Other studies [10] have employed a multi-stage training strategy that utilizes facial landmarks as intermediate representations. This approach constructs a flow from audio to facial landmarks, which subsequently drives NeRF’s rendering of facial images. Compared to multi-stage talking face generation methods [9,10], end-to-end approaches are more efficient and avoid cumulative errors. To achieve more precise lip synchronization effects, some works [32] model the mouth separately using additional NeRF networks and obtain generated images through mixed rendering. However, this requires more memory and computation. Furthermore, they all ignore the role of enhancing the human vocal component of the audio in improving lip synchronization.

Existing NeRF-based methods improve audio–lip synchronization through specialized architectural constraints. For example, AE-NeRF [28] employs an audio-aware aggregation mechanism to disentangle audio-related facial regions and stabilize motion, while SyncTalk [30] utilizes dedicated controllers and stabilizers to enforce explicit alignment and maintain identity consistency. In contrast to these approaches that focus on feature fusion or alignment controllers, we target the practical challenge of robustness against noisy, unconstrained audio inputs. By integrating semantic-level audio event modeling and temporal feedback, our framework suppresses acoustic disturbances and ensures stable, semantically meaningful lip dynamics without altering the fundamental NeRF representation.

Another area of development within 3D-based methods involves Gaussian Splatting [35], which has shown promise in 3D face modeling and rendering. Gaussian avatars [36] reconstruct digital humans by binding Gaussians to a standard facial model. However, these methods lack speech modality inputs and are limited to transferring expressions and motions from other facial images. Subsequently, some studies [37,38,39,40] associated the displacement and deformation of Gaussians with audio signals. They enable the displacement of local Gaussians driven by audio information, ultimately rendering facial images with synchronized lip shapes. However, due to the explicit nature of the Gaussian Splatting representation, artifacts may emerge during the rendering process typically caused by outlier Gaussians.

While both NeRF-based and Gaussian-based methods show strong potential in 3D talking face generation, they still face challenges related to spatiotemporal instability and occasional artifacts. These issues highlight the need for further improvements in both model robustness and rendering quality.

2.3. Depth-Guided Neural Radiance Fields

Several studies [41,42,43,44] have explored that depth information plays a crucial role in NeRF training by significantly improving the spatial consistency and rendering quality of generated 3D scenes. Traditional NeRF models rely on multi-view images to synthesize novel views, but the absence of explicit spatial information often leads to depth blur and geometric inconsistencies in the generated outputs. Depth supervision, by introducing depth information, helps the model better understand the spatial structure of the scene, accelerating convergence and improving rendering precision. These studies use sparse feature points generated during the Structure from Motion (SFM) [45] process as relative depth ground truth and establish sparse depth supervision, which enables faster training speeds and superior rendering effects in novel views. Additionally, existing work [43] utilizes depth completion networks to further improve depth labels and achieve superior performance.

However, despite the success of depth supervision in static scene reconstruction, its application in dynamic scenarios, such as talking face generation, remains challenging due to the lack of sufficient depth data. The scarcity of RGBD speaker datasets and the inability to obtain sparse landmarks through SFM contribute to a lack of effective depth supervision for training. To address this, in AVIST, we introduce a 3D Morphable Model (3DMM) [11]-based depth renderer to generate relative depth labels for the face, providing effective depth supervision for the model training. This approach enhances rendering quality and accelerates network training.

3. Methodology

3.1. Preliminaries

In this subsection, we introduce the preliminaries of NeRF reconstruction and the specific problem setting of our task.

NeRF reconstruction maps a spatial sample point $\mathbf{p}=(x,y,z)$ and viewing direction $\mathbf{d}$ to its corresponding color $\mathbf{c}$ and volumetric density $\rho$. This process is defined as $F:(\mathbf{p},\mathbf{d})\to (\mathbf{c},\rho )$. The final pixel color $\widehat{\mathrm{C}}$ for a camera ray $\mathbf{r}\left(t\right)=\mathbf{o}+t\mathbf{d}$ is synthesized using volume rendering [26] as follows:

$$\widehat{\mathrm{C}}\left(\mathbf{r}\right)=\sum _{k=1}^K{w}_k{\mathbf{c}}_k,w_k=T_k(1-\exp (-{\rho }_k{\delta }_k)),$$

(1)

where $w_k$ is the integration weight for the k-th sampling point, ${\delta }_k$ is the interval between adjacent samples, and $T_k=\exp (-{\sum }_{i=1}^{k-1}{\rho }_i{\delta }_i)$ represents the accumulated transmittance, and $t_k$ denotes the depth value of the k-th sample point along the ray $\mathbf{r}$.

For the task of dynamic talking face generation, the static NeRF model must be extended to handle temporal control. We incorporate a set of dynamic control factors m into the mapping function:

$$F:(\mathbf{p},\mathbf{d};m)\to (\mathbf{c},\rho ),$$

(2)

where m encapsulates driving signals such as speech audio and blinking intensity. This formulation allows for the precise control of detailed facial expressions and lip movements while maintaining the spatial integrity of the 3D representation.

3.2. Overview

Having established the problem setup, we now provide an overview of our method. As illustrated in Figure 2, during the generation of the n-th frame, the core neural network for talking face reconstruction, denoted as ${\Pi }_{\varphi }$, is a lightweight MLP that takes five types of inputs. Among these, the spatial position $(x,y,z)$ and viewing direction d are static inputs dependent solely on camera poses, which in turn determine the rendered face’s viewing angle. The remaining inputs are dynamic facial control features: (1) blink information to regulate blinking intensity, (2) audio features to control lip movements, and (3) prior-frame lip shape information for maintaining temporal coherence.

Figure 2. Our method follows the complete process depicted in the figure. Multimodal inputs, including eye blink features $f_{eye}^n$, tri-plane position encoding $f_{H3}^n$, audio-enhanced features $f_{spc}^n$, and previous frame lip shape $f_{lip}^{n-1}$, are fused by regional attention networks ${\mathrm{\Psi }}_{\chi }$ and passed through an MLP ${\Pi }_{\varphi }$ to generate color $c^{\ast }$ and density values ${\rho }^{\ast }$, which are then rendered through volume rendering into the depth map and current frame.

To improve the efficiency of network training and inference, we encode the spatial position $(x,y,z)$ using a Tri-plane hashing method [29], yielding spatial position features $f_{H3}^n$. The blink feature $f_{eye}^n$ is extracted from ground-truth images during training using the OpenFace framework [46], which provides blink amplitude values. For audio, we employ a dual-branch fusion strategy to extract enhanced audio-driven features $f_{spc}^n$. Finally, the lip shape from the previous frame, encoded as $f_{lip}^{n-1}$, is used as a guiding input for the current frame. The overall process can be represented as

$${\rho }^{\ast },c^{\ast }={\Pi }_{\varphi }\left(f_{input}^n,d\right),$$

(3)

$$f_{input}^n=f_{H3}^n\oplus {\mathrm{\Psi }}_{au}\left(f_{spc}^n\right)\oplus {\mathrm{\Psi }}_{eye}\left(f_{eye}^n\right)\oplus {\mathrm{\Psi }}_{lip}\left(f_{lip}^{n-1}\right)\in {\mathbb{R}}^{B\times (36+32+1+20)},$$

(4)

where ⊕ denotes the concatenation operation, and ${\mathrm{\Psi }}_{\chi }(\ast )$ represents the Region Attention Network [29]. Since different facial regions respond differently to control factors (e.g., the mouth region is more sensitive to audio inputs), we use this region attention network to weight each control input as follows:

$${\mathrm{\Psi }}_{\chi }\left(f\right)=f\odot {\mathrm{MLP}}_{\chi }\left(f_{H3}^n\right).$$

(5)

Specifically, each control signal is processed by a dedicated attention MLP with a two-layer hidden structure, thereby avoiding cross-region interference. The outputs ${\rho }^{\ast }$ and $c^{\ast }$ represent the density and color values at the sampled points, respectively. The final image is synthesized using volume rendering [26].

In Section 3.3, we introduce the principles of the audio event enhancement module and explain how dual-branch audio features are fused as enhanced control inputs. In Section 3.4, we present the lip shape feedback mechanism, which ensures temporal continuity in lip shape generation. Finally, in Section 3.5, we describe how 3DMM-based facial reconstruction is employed to obtain depth labels, further improving rendering quality.

3.3. Audio Event Enhancement

To address the issue of mismatched audio-video pairs caused by noise in audio data from non-curated, in-the-wild datasets (e.g., daily talking videos collected from social media platforms), which degrades lip synchronization performance, we propose an Audio Event (AE) enhancement module. This module isolates and amplifies human voice components, thereby improving the alignment between audio and lip movements.

As illustrated in Figure 2, the input audio is segmented into frames, and for the n-th frame, we adopt a dual-branch structure to extract audio features. First, we utilize a pre-trained DeepSpeech network [47] to extract audio semantic features $f_{au}^n\in {\mathbb{R}}^{8\times 29\times 16}$. These features capture the core semantic information of the audio and serve as the primary control signals for lip motion. Second, to evaluate the reliability of the human voice components within the audio segment, we use a pre-trained audio event detector, YamNet [48], to extract audio event features $f_{en}^n\in {\mathbb{R}}^{10}$, which represent the audio’s component composition.

We then fuse $f_{en}^n$ and $f_{au}^n$ through a Linear Attention Fusion mechanism to obtain refined human voice semantic features $f_{spc}^n\in {\mathbb{R}}^{8\times 29\times 16}$, ensuring purer voice-driven signals for lip synchronization. The fusion process is formulated as

$$f_{spc}^n=f_{au}^n\odot \mathrm{Sigmoid}\left({\Gamma }_{\theta }\left(f_{en}^n\right)\right),$$

(6)

where ${\Gamma }_{\theta }$ represents a linear layer, and ⊙ denotes element-wise multiplication.

3.4. Lip Shape Feedback Mechanism

To ensure temporal coherence, relying solely on audio information is often insufficient for continuous lip reconstruction. Building on the effectiveness of facial landmarks in 3D generation [9,10], we propose the lip shape feedback (LSF) mechanism to utilize prior frame information for current frame inference.

As illustrated in Figure 3, for each generated frame, a Lip Encoder captures and encodes the rendered lip landmarks into a feature vector $f_{lip}^{n-1}$. To ensure robustness against facial size variances and coordinate deviations, we extract lip landmarks using a pre-trained detector [49] and normalize them into a scale-invariant structural space:

$$f_{lip}^{n-1}=E_{lip}\left({\displaystyle \frac{lmk{s}^{n-1}-{\overline{lmks}}^{n-1}}{W_{face}}}\right),$$

(7)

where $lmks\in {\mathbb{R}}^{20}$ denotes the landmark coordinates, $\overline{lmks}$ is the mouth center, and $W_{face}$ is the facial width used for standardizing the scale.

Figure 3. Lip Encoder: For the previous frame output, lip landmarks are extracted, normalized in the structural space, and then encoded through the MLP network $E_{lip}$.

During training, we incorporate landmarks from ground-truth images to prevent error accumulation. In the inference stage, the model recursively utilizes the landmarks from the previously rendered frame as input, effectively reducing flickering artifacts and ensuring smooth transitions even under weak audio signals.

3.5. Depth Supervision from 3DMM

Training NeRF with only RGB supervision often leads to structural uncertainty, particularly when training data is limited to sparse viewing angles. To address this, we introduce additional depth information to provide spatial constraints, which optimizes facial rendering performance and accelerates convergence.

The depth rendering process is analogous to the RGB volume rendering in Equation (1). We approximate the surface distance $\widehat{\mathrm{D}}\left(\mathbf{r}\right)$ at which a ray terminates, reflecting the face’s depth as follows:

$$\widehat{\mathrm{D}}\left(\mathbf{r}\right)=\sum _{k=1}^{\mathbf{K}}{w}_k{t}_k.$$

(8)

Similar to color loss, differentiable depth loss contributes to updating NeRF gradients [41]. Such supervision is crucial not only for the convergence of the network but also for enhancing the precision of the scene representation and the final rendering quality.

Obtaining precise facial depth labels from monocular RGB video is challenging; thus, we construct a depth renderer using 3DMM [11]. As illustrated in Figure 4, we first fit the facial pose $\mathbf{P}$, shape, and expression coefficients from image $\mathbf{I}$. The facial contour model $\mathbf{M}$ is then parsed from these coefficients using the FLAME 2023 [50] base. To ensure accurate mouth interior depth, we augment the model by binding a rigid teeth model to the lip positions. During rasterization, the model vertices are transformed into 2D screen space, where depth labels $\mathrm{D}$ are extracted from the z-buffer relative to the camera:

$$\mathrm{D}=Proj(\mathbf{M},\mathbf{P}).$$

(9)

The corresponding depth loss ${\mathcal{L}}_{\mathrm{depth}}$ is formulated as

$${\mathcal{L}}_{\mathrm{depth}}=\sum _{m\in {\mathrm{M}}_{\mathrm{face}}}{∥\zeta {\widehat{\mathrm{D}}}_m-{\mathrm{D}}_m∥}_2^2,$$

(10)

where ${\mathrm{M}}_{\mathrm{face}}$ is the valid facial depth mask and $\zeta$ is a learnable scale factor designed to align the linear scales of the inferred depth $\widehat{\mathrm{D}}$ and the ground-truth depth $\mathrm{D}$.

Figure 4. The diagram above illustrates the depth renderer, utilized to provide depth supervision during the training of NeRF. After parameterizing facial modeling using 3DMM, the projected depth of the model relative to the camera is used as the depth label for the face.

3.6. Loss Function

Regarding the training strategy, we adopted a multi-stage training approach. In the coarse training phase, the loss function used is

$${\mathcal{L}}_{coarse}=\sum _{\mathrm{i}\in I_{face}}{∥C_{\mathrm{i}}-{\widehat{C}}_{\mathrm{i}}∥}_2^2+\alpha {\mathcal{L}}_{\mathrm{depth}},$$

(11)

where $\alpha$ is set to 0.02. In the fine-tuning phase, we utilize both Mean Squared Error (MSE) loss and Learned Perceptual Image Patch Similarity (LPIPS) [51] loss, specifically focusing on the mouth region like previous studies [29], and we randomly define a patch $\mathcal{P}$:

$${\mathcal{L}}_{fine}=\sum _{\mathrm{i}\in \mathcal{P}}{∥C_{\mathrm{i}}-{\widehat{C}}_{\mathrm{i}}∥}_2^2+\lambda \times LPIPS(\widehat{\mathcal{P}},\mathcal{P}),$$

(12)

where $\lambda$ is configured to 0.001.

4. Experiment

4.1. Implementation Details

To ensure that the training speaker video produces one-to-one audio–visual matching pairs, we set the input video frame rate to 25 fps. During the data preprocessing stage, we utilized a 3DMM depth renderer to obtain depth labels. Additionally, to avoid non-rigid twisting of the neck region, which introduces extra reconstruction errors, we extract the head region using a pre-trained avatar segmentation network from [52]. We extract facial landmarks by [49], which are labeled to facilitate the training of the lip shape feedback mechanism, thereby mitigating cumulative errors that may arise from feedback formation. We also utilized the pre-trained YamNet [48] for speech event detection. Since the audio event detection is less frequent in the time series than the DeepSpeech [47] audio feature, in order to complete the alignment in the time series, we interpolated and upsampled this audio event feature to obtain a sequence of the same length as the audio–visual pair. The network architecture was constructed using PyTorch 3.10, and training, as well as subsequent experiments, were conducted on a single RTX3090 GPU using the AdamW optimizer [53] whose parameters are ${\beta }_1$ = 0; ${\beta }_2$ = 0.99. In practical training, as described in the methodology, we adopt a two-stage training strategy. In the first stage, a global geometric constraint incorporating depth supervision is used to accelerate the convergence of facial structure. In the second stage, the model is further refined using RGB and perceptual losses to enhance fine details. For a single-identity training set consisting of approximately 5 min of video (about 7000 frames) at a resolution of around $450\times 450$, the model is trained for $15+5$ epochs, with a GPU memory consumption of about 2.2 GB and a total training time of approximately 1.8 h.

4.2. Dataset

Unlike 2D-based generative methods that rely on large-scale, multi-identity datasets such as CREMA-D [54] and GRID [55], NeRF-based talking face generation typically requires only 3–5 min of identity-specific video for model training. To evaluate our approach, we conduct quantitative comparisons using the Obama dataset from ADNeRF [52] and speaker videos curated from the HDTF [56] dataset. The Obama sequence consists of a 5 min video at a resolution of 450 × 450 pixels and a frame rate of 25 FPS. Following standard protocols [52], the video is partitioned into a 4.5 min training-validation set and a 30 s test set. To further assess the generalization capability of AVIST, we performed additional evaluations on other collected video sequences. These videos were preprocessed to retain only active speaking segments, with the subject centered and front-facing at a 450 × 450 resolution. Notably, to address potential privacy and infringement concerns regarding the publication of public figure avatars, our qualitative visualizations are primarily derived from a private dataset captured via a mobile device (iPhone). The resolution, frame rate, and dataset splitting for this private data are consistent with the settings used in [52].

4.3. Evaluation Metrics

To evaluate the performance of our method comprehensively, we employ several metrics across four dimensions: image fidelity, audio–visual synchronization, perceptual quality, and computational efficiency.

Image Fidelity: Consistent with standard 3D reconstruction tasks, we use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) to assess the pixel-level and structural fidelity of the synthesized frames relative to the ground truth.

Audio–Visual Synchronization: Following [21], we utilize Lip Sync Error Distance (LSE-D) and Confidence (LSE-C) to quantify the alignment between audio signals and lip movements. Additionally, Facial Landmark Distance (LMD) is computed to evaluate the overall consistency of facial expressions.

Perceptual Quality: To assess the visual realism and distribution similarity, we employ Fréchet Inception Distance (FID) [57] and Learned Perceptual Image Patch Similarity (LPIPS) [51]. While FID measures the statistical distance between real and generated image distributions, LPIPS calculates the perceptual distance between deep feature representations to reflect human visual judgment.

System Efficiency: Given the requirements for practical deployment, we report the training Time per identity, the rendering speed (FPS) during inference, and the end-to-end Latency(the time elapsed from audio input to the generation of a single frame).

4.4. Comparative Experiment

4.4.1. Qualitative Results

To objectively compare the performance against baselines, we replicated various methods, as shown in Figure 5 where our method achieved better results in terms of lip edge clarity and lip synchronization. Furthermore, due to the integration of audio events, individuals tended to close their mouths better in silent conditions, avoiding white residue between the lips.

Figure 5. The visualization results of reconstruction for the same video clip, comparing our proposed method with relevant models, including ADNeRF [52], RadNeRF [58], ERNeRF [29], gsTalker [38] and GsTalker [39]. We present a frame-by-frame comparison against the ground truth (GT) across different phonemes. It can be observed that our method achieves remarkable improvements in the reconstruction of lip and tooth structures, and exhibits superior performance in high-frequency texture recovery as well as lip shape consistency with the GT.

4.4.2. Quantitative Results

Our quantitative comparison results are presented in Table 1, indicating that our method achieved optimal or near-optimal results across all metrics. In terms of image quality metrics, our approach demonstrates superior generation quality compared to both 2D-based methods such as Wav2lip [21] and DINet [6] and methods based on NeRF or 3D Gaussians. Taking the Obama dataset as an example, we achieved improvements of 1.908 and 0.0094 in the PSNR and LPIPS metrics, respectively, which reflect the quality of image reconstruction. In terms of facial structure restoration accuracy, the LMD and SSIM metrics showed respective improvements of 0.358 and 0.0072 compared to other methods. This advantage is evident in datasets like the Obama dataset and the HDTF dataset, primarily due to the precise modeling of individuals enabled by depth supervision.

Table 1. The quantitative results of the head reconstruction. ↑ and ↓ denote that higher and lower values represent better performance, respectively. We highlight best and second-best results, and the blue item represents the optimal method achieved using 3D-based approaches.

Dataset	Methods	PSNR ↑	LPIPS ↓	LMD ↓	FID ↓	SSIM ↑	LSE-C ↑	LSE-D ↓	Time ↓	FPS ↑	Latency ↓	Category
Obama [52]	Wav2lip [21]	31.1620	0.0624	5.1265	15.9862	0.8464	9.182	6.1254	-	-	-	2D
	DINet [6]	30.1894	0.0293	3.7625	15.6412	0.8504	6.165	7.1203	-	-	-	2D
	ADNeRF [52]	29.2536	0.0956	4.4891	12.6215	0.8631	5.203	9.3241	17 h	0.11	9.5s	3D
	RadNeRF [58]	33.4519	0.0526	2.8421	8.5614	0.8931	5.268	9.7562	3.5 h	29	34 ms	3D
	ERNeRF [29]	34.0125	0.0283	3.7451	9.2645	0.9276	5.763	8.6348	2 h	30	33 ms	3D
	SyncTalk [30]	33.3512	0.0291	3.5012	8.6510	0.8850	5.652	8.6135	2 h	30	35 ms	3D
	gsTalker [38]	29.1356	0.1023	4.6702	8.9230	0.8726	4.925	10.3271	2.5 h	52	25 ms	3D
	GsTalker [39]	32.1732	0.0874	3.4752	7.1284	0.8683	5.201	9.3213	2.2 h	49	27 ms	3D
	Ours	35.9206	0.0189	2.4838	4.8879	0.9348	6.765	7.8126	1.8 h	22	45 ms	3D
HDTF [56]	Wav2lip [21]	29.1251	0.1243	4.2381	13.6942	0.8361	8.032	7.4162	-	-	-	2D
	DINet [6]	30.9421	0.0375	5.9421	12.0142	0.8714	6.165	7.1203	-	-	-	2D
	ADNeRF [52]	28.3249	0.0812	5.7644	10.3405	0.8145	4.215	10.2416	17 h	0.11	9.5 s	3D
	RadNeRF [58]	31.2051	0.0913	5.7936	11.3497	0.8796	4.967	9.4610	3.5 h	29	34 ms	3D
	ERNeRF [29]	30.6234	0.0412	3.6427	7.2163	0.8863	5.380	8.3561	2 h	31	33 ms	3D
	SyncTalk [30]	29.0134	0.0814	4.1253	8.4322	0.8850	5.912	8.5672	2 h	30	35 ms	3D
	gsTalker [38]	29.0326	0.0932	4.7856	10.3270	0.8581	4.301	10.3230	2.5 h	52	25 ms	3D
	GsTalker [39]	31.3853	0.0532	4.6832	9.0490	0.8622	5.721	8.5472	2.2 h	49	27 ms	3D
	Ours	32.5843	0.0239	3.1286	8.0143	0.8962	6.031	8.1126	1.8 h	23	45 ms	3D

In terms of lip synchronization accuracy, as shown in Table 1, 2D-based methods such as Wav2Lip [21] and DINet [6] achieve higher lip synchronization scores (LSE-C and LSE-D) than 3D-based approaches. This is mainly due to their large-scale pre-training on multi-speaker datasets, which enables strong and robust phoneme-to-lip alignment. In contrast, NeRF-based methods, including ours, are trained in an identity-specific manner using only a few minutes of target speaker data, which inherently limits absolute performance on audio–visual synchronization metrics. Despite this setting mismatch, our method consistently outperforms all existing 3D-based approaches on lip-related metrics such as LMD and LSE, indicating more accurate and stable mouth motion modeling within the 3D paradigm. We include comparisons with 2D methods to provide a reference for relative lip synchronization performance, highlighting the gap between pre-trained 2D models and identity-specific 3D reconstruction methods, while emphasizing the complementary strengths of 3D approaches in preserving speaker-dependent articulation patterns.

Additionally, the incorporation of facial depth supervision during the coarse training phase expedites the construction of facial NeRF, effectively reducing training time while maintaining high-quality facial reconstruction and rendering.

4.5. Ablation Study

4.5.1. Ablation Study on Audio Event Module

As shown in Table 2, we integrated different modules into the ERNeRF-based baseline [29]. The performance metrics related to lip synchronization were significantly improved after the incorporation of the AE module. In addition, we observe that introducing the AE module alone leads to a slight decrease in image quality as measured by PSNR, dropping from 31.682 to 30.352. However, the improvements on lip-related metrics are consistent with our design objectives. When combined with depth supervision, which is specifically introduced to enhance visual quality, the overall performance becomes optimal across both visual fidelity and lip synchronization metrics.

Table 2. The table below presents the ablation experiments concerning the audio event enhancement, lip shape feedback mechanism, and depth supervision, showing the results after adding certain modules. ✓ indicates the strategy is used, while × indicates the strategy is removed. ↑ and ↓ denote that higher and lower values represent better performance, respectively. We highlight best and second-best results.

Depth	LSF	AE	PSNR ↑	LPIPS ↓	LMD ↓	FID ↓	SSIM ↑	LSE-C ↑	LSE-D ↓
ine ×	×	×	31.682	0.0596	6.6345	8.3264	0.8312	5.401	9.6146
✓	×	×	34.0948	0.0124	2.6821	5.6624	0.9185	5.253	10.6321
×	✓	×	33.1276	0.0201	3.4216	5.9026	0.8926	6.215	9.2143
×	×	✓	30.3518	0.0768	5.8421	6.1684	0.8612	6.632	8.6143
✓	✓	✓	35.9206	0.0189	2.4838	4.8819	0.9348	6.765	8.6241

To validate the effectiveness of our audio event module under complex audio, we simulate the training and inference process of data collection in complex scenarios by mixing original human voices and urban background sounds. After setting different noise ratios, we train our method with and without AE for the same number of rounds to compare the lip synchronization results. As illustrated in Figure 6a, our method maintains robust lip-synchronization even as noise levels increase and the signal-to-noise ratio (SNR) decreases. The integration of the AE module strengthens the network’s focus on vocal components during training, fostering more precise audio–visual alignment and effectively filtering out acoustic interference. Furthermore, the AE module improves PSNR metrics under noisy conditions. This enhancement is attributed to superior data alignment, which prevents the generation of blurred lip artifacts and consequently elevates overall reconstruction quality. Figure 6b illustrates the inference results using noisy audio. The dual-branch audio feature module effectively integrates vocal component information into the extracted features. Consequently, our method accurately responds to human speech while maintaining a stable, stationary, or closed mouth state during periods of background noise, thereby ensuring robust lip synchronization even in adverse acoustic environments.

Figure 6. Robustness and sensitivity analysis of the audio event (AE) enhancement module. Quantitative results (a) show that the AE module significantly maintains PSNR and LSE-C stability as background noise levels increase. Qualitative comparisons (b) demonstrate the model’s ability to accurately discriminate human speech from ambient noise, ensuring stable lip motion even in noisy environments.

4.5.2. Ablation Study on Lip Shape Feedback

To verify the contribution of the lip shape feedback (LSF) module, we analyze the experimental results in Table 2 by comparing groups with/without LSF while fixing other modules’ states: When both the Depth and AE modules are disabled, the model without LSF achieves a PSNR of 31.6820, LMD of 6.6345, and LSE-C of 5.401, while enabling LSF increases the PSNR to 33.1276 ($+1.4456$), reduces the LMD to 3.4216 ($-3.2129$), and raises the LSE-C to 6.215 ($+0.814$), confirming that the LSF module can independently enhance lip reconstruction quality and reduce synchronization errors. When the Depth and AE modules are enabled, adding the LSF module further pushes the model to achieve the optimal comprehensive performance in the table, which demonstrates that the LSF module can synergize with other modules to amplify their complementary effects.

4.5.3. Ablation Study on Depth Supervision

The Table 2 is the ablation experiment of depth supervision. From the experimental results, it can be observed that there is an improvement in the quality of head image generation after adding depth supervision. Also, the depth supervision improves the reconstruction quality of the face. Figure 7a shows the performance of the model on the validation set at different epochs during the training process, where the model has a higher upper limit of performance after using depth supervision and can continue to maintain a stable performance. In addition, Figure 7b is the facial reconstruction surface geometry diagram after completing the training. We can see that due to the supervision of depth information, our facial contours are clearer and more accurate, proving that depth supervision effectively improves the geometric consistency of the face. However, the efficacy of this supervision is intrinsically linked to the quality of the 3DMM-derived depth maps. Potential inaccuracies in the fitting process may introduce biased supervision signals, which could theoretically degrade performance. To evaluate the robustness of our framework against such structured geometric biases, we conduct a sensitivity analysis by perturbing the ground-truth depth maps with a synthetic tilted-plane bias:

$$\mathrm{\Delta }D(x,y)=\gamma (a\tilde{x}+b\tilde{y}),$$

(13)

where $\tilde{x},\tilde{y}$ are normalized coordinates, $a,b$ are random coefficients, and $\gamma$ scales the perturbation. Quantitative results in Table 3 indicate that while second-stage RGB refinement can mitigate mild biases, larger perturbations significantly degrade performance. This analysis underscores both the robustness and the inherent dependency of our framework on the quality of 3DMM-based depth supervision.

Figure 7. Effect of depth supervision on (a) model convergence (PSNR) and (b) spatial structural accuracy. Depth constraints significantly accelerate convergence and result in smoother facial geometry with reduced structural errors compared to RGB-only training.

Table 3. Impact of depth bias on the results, ↑ and ↓ denote that higher and lower values represent better performance, respectively. We highlight best results.

Setting ($\mathit{\gamma }$)	PSNR ↑	LPIPS ↓	SSIM ↑	LSE-C ↑	LSE-D ↓
0.0	35.921	0.018	0.9348	6.765	7.812
0.05	35.817	0.019	0.9312	6.720	7.835
0.1	35.121	0.020	0.9291	6.631	7.903
0.3	32.483	0.028	0.9020	5.801	9.459
0.6	28.591	0.053	0.8521	5.218	10.095

4.5.4. Weights Sensitivity Analysis

We conduct a parameter sensitivity analysis to evaluate the impact of different parameter settings on the performance of our model. The parameters analyzed are $\alpha$ and $\lambda$ in the loss function, with various combinations tested. The results are summarized in Table 4. In summary, $\alpha$ = 0.02 and $\lambda$ = 0.001 generally perform best across most metrics.

Table 4. Results of the hyperparameter sensitivity experiment concerning the loss function weights. ↑ and ↓ denote that higher and lower values represent better performance, respectively. We highlight best and second-best results.

Setting ($\mathit{\alpha }$/$\mathit{\lambda }$)	PSNR ↑	LPIPS ↓	SSIM ↑	LSE-C ↑	LSE-D ↓
0.02/0.001	35.921	0.018	0.9348	6.765	7.812
0.01/0.001	34.953	0.018	0.9218	6.631	7.795
0.01/0.005	35.019	0.023	0.9031	6.103	7.961
0.02/0.005	35.564	0.021	0.9147	6.516	8.221
0.05/0.01	34.591	0.022	0.9143	6.314	8.021
0.05/0.001	35.614	0.021	0.9130	6.453	7.920

4.5.5. Architectural Sensitivity Analysis

In the architectural design of our model, to investigate appropriate feature fusion ratios and the contribution of different feature components, we further analyze the impact of varying lip feedback signal dimensionalities on the final performance. In addition, we examine whether applying proper normalization to lip coordinates affects the overall results. As shown in Table 5, when the dimensionality of the lip feedback signal is too low, the model fails to effectively capture the lip configuration of the previous frame, resulting in a limited benefit from the feedback mechanism. Conversely, excessively high dimensionality introduces overly strong constraints, which may dominate the current audio-driven signal and reduce flexibility. The best performance is achieved when the feedback dimensionality is set to 20. Moreover, appropriate normalization of lip coordinates effectively removes biases caused by coordinate systems and facial shape variations, leading to further performance improvements.

Table 5. Performance comparison under different lip feature designs, ✓ indicates the strategy is used, while × indicates the strategy is removed. ↑ and ↓ denote that higher and lower values represent better performance, respectively. The best results are highlighted in bold.

Setting		PSNR ↑	LPIPS ↓	SSIM ↑	LSE-C ↑	LSE-D ↓
Dimension	8	35.621	0.019	0.9303	6.691	7.936
	20	35.921	0.018	0.9348	6.765	7.812
	32	34.907	0.021	0.9041	6.312	8.230
Norm	✓	35.921	0.018	0.9348	6.765	7.812
	×	33.725	0.023	0.9074	6.201	8.132

5. Discussion

5.1. Limitations

Despite its advancements, AVIST has several limitations that provide directions for future research. While our AE module enhances robustness against environmental noise, the framework still maintains a certain dependency on the underlying speech clarity; consequently, extreme audio distortion or sparse vocal signals may still hinder precise lip-sync results. Furthermore, the requirement of 3–5 min of target speaker video for training creates an identity-specific bottleneck, which restricts the model’s zero-shot generalization to unseen individuals without further fine-tuning. There are also potential generalization issues regarding facial geometries and speaking styles, as our 3DMM-based depth supervision assumes a standard facial topology that may struggle to accommodate extreme features, such as heavy facial hair, or highly idiosyncratic articulation patterns. Finally, while our method achieves a competitive 22 FPS, it still lags behind explicit 3DGS-based approaches like gsTalker [38] in terms of inference speed, potentially affecting its deployment in ultra-low-latency scenarios.

5.2. Future Work

To address these limitations, we propose the following directions: Combining the rendering speed of 3D Gaussian Splatting with NeRF’s implicit representation could balance efficiency and fidelity. For example, using 3DGS for coarse rendering and NeRF for fine details. Exploring few-shot or meta-learning techniques to generalize across unseen speakers with minimal training data. Furthermore, we plan to investigate the use of generative models, such as diffusion models, to synthesize high-quality and identity-preserving training samples from limited data. This could effectively augment the dataset in low-resource scenarios and further improve speaker reconstruction under few-shot settings.

6. Conclusions

In conclusion, we present a novel NeRF-based talking face generation framework that significantly enhances audio–visual synchronization and spatiotemporal expressiveness. By integrating audio event information, our method improves the correlation between human speech and lip movements, enabling more accurate lip synchronization and greater consistency in complex audio conditions. Additionally, the lip shape feedback module ensures stability and coherence across consecutive frames through temporal modeling. Our depth renderer further refines facial geometry learning by providing depth labels for model training. Experimental results, both qualitative and quantitative, show that our framework excels in both head reconstruction rendering quality and audio–visual consistency, demonstrating its potential for advancing digital avatar realism.