MAVAGEN: Multimodal Avatar Generation Framework for Personalized Human–Computer Interaction

Abstract

Digital-avatar systems still provide limited control over emotionally expressive behavior in human–computer interaction, especially in Large Language Model (LLM)-based chatbots and virtual assistants with personalized visual embodiments. To address this problem, we propose Multimodal Avatar Generation (MAVAGEN), a multimodal avatar generation framework for synthesizing upper-body digital avatars with personalized appearance and controllable emotional expression. The user specifies the desired gender and age, as well as provides a short text input from which the target emotional state is inferred. MAVAGEN then retrieves an identity image from the HaGRIDv2-1M corpus and generates an avatar clip with synchronized facial expressions, hand gestures, and expressive speech. The framework uses the following six feature streams: textual features, emotion-distribution features, landmark-based pose features, depth-geometry features, RGB-appearance features, and acoustic features. In a quantitative evaluation against recent human animation methods, MAVAGEN achieves the best overall avatar quality, with FID 48.20, FVD 592.00, SSIM 0.741, Sync-C 7.40, HKC 0.929, HKV 25.30, CSIM 0.563, and EmoAcc 0.88. Ablation results show that emotion and acoustic features contribute most to emotional agreement, while landmark-based pose and depth features improve geometric and motion stability. These results support the practical use of MAVAGEN in personalized LLM-based assistants and other emotion-sensitive interactive systems.

1. Introduction

Emotionally expressive digital avatars are becoming increasingly important in human–computer interaction, especially in socially sensitive domains such as mental health support, education, and assistive technologies [1,2,3], where LLM-based chatbots and virtual assistants may particularly benefit from personalized visual embodiments. In such settings, natural interaction depends not only on semantic content but also on the coordinated expression of emotion through facial movements, gestures, and speech. Effective avatar generation therefore requires multimodal modeling of emotional states while accounting for personal factors that shape how emotion is expressed and perceived [4,5,6]. These considerations create a practical need for methods that move beyond static visual identity and support both personalized appearance and controllable emotional expression.

Despite recent progress, existing multimodal affective systems still face important limitations. Many rely primarily on text [7,8] or text paired with audio [9], while underutilizing essential non-verbal cues such as facial expressions, gestures, and paralinguistic speech features. At the same time, multimodal integration efforts [2,10] often depend on large supervised corpora, which are expensive to annotate and difficult to adapt to diverse emotional [11] and therapeutic [12] settings. From the standpoint of avatar generation, a further challenge is the limited controllability of the generated behavior as follows: many systems can model emotional content, but they still struggle to synthesize avatars that express a desired emotion in a clear and coordinated manner. Users interacting with digital characters may expect emotional nuance and responsiveness that current systems still cannot provide reliably [13].

Recent methods for avatar generation and animation offer promising building blocks, but none fully addresses this problem. InstructAvatar [14] enables text-guided control over emotional states and physical movements, while VQTalker [15], CtrlAvatar [16], and MoEE [17] improve facial animation and audio-driven expressiveness. Other methods, including GraphAvatar [18], PERSE [19], HRAvatar [20], GPAvatar [21], and VLOGGER [22], focus on high-quality avatar representation and synthesis. Nevertheless, existing approaches often prioritize only part of the overall generation problem as follows: some emphasize facial motion, others audio-driven animation, and still others focus on geometry or rendering quality. As a result, avatar synthesis that coordinates facial expressions, gestures, and speech remains insufficiently explored, especially when the avatar must be conditioned on both the desired demographic attributes and a target emotion.

To address these challenges, we propose the MAVAGEN framework for personalized upper-body avatar synthesis with controllable emotional expression in human–computer interaction scenarios involving LLM-based chatbots and virtual assistants. The user specifies the desired avatar attributes, specifically gender and age, as well as provides a short text input from which the target emotional state is inferred. Based on these inputs, MAVAGEN retrieves a visually appropriate identity image from the HaGRIDv2-1M corpus and generates an avatar clip with synchronized facial expressions, hand gestures, and expressive speech. The framework uses the following six feature streams: textual features, emotion-distribution features, landmark-based pose features, depth-geometry features, RGB-appearance features, and acoustic features. In this way, MAVAGEN combines personalized appearance with controllable emotional expression within a unified diffusion-based pipeline. Our main contributions are as follows:

• We propose MAVAGEN, a novel multimodal avatar generation framework for synthesizing personalized upper-body digital avatars with controllable emotional expression.
• We introduce an attribute-conditioned multimodal generation pipeline that integrates desired gender and age attributes with textual features, emotion-distribution features, landmark-based pose features, depth-geometry features, RGB-appearance features, and acoustic features within a unified diffusion-based architecture.
• A quantitative evaluation shows that MAVAGEN achieves the best overall avatar quality among the evaluated human animation methods.
• We introduce a novel EmoAcc measure that quantifies the agreement between the target emotion specified for the avatar and the emotion expressed by the generated avatar.

2. Related Work

Recent work on avatar generation relevant to our setting spans three closely connected directions as follows: multimodal emotion modeling, emotion-aware avatar animation, and personalized, controllable avatar synthesis. Collectively, these lines of research address how emotion can be inferred, represented, and rendered through coordinated visual and acoustic behavior. In this section, we review representative advances in each direction and highlight the remaining gap between emotion understanding and the generation of personalized, expressive avatars.

2.1. Emotion-Aware Avatar Generation and Animation

A growing body of work addresses the generation of expressive avatars directly. InstructAvatar [14] enables text-guided control over both emotional states and body motion, while VQTalker [15] introduces facial motion tokenization for multilingual talking avatars. MoEE [17] uses a mixture of emotion experts for audio-driven portrait animation, VLOGGER [22] extends multimodal diffusion to full-body avatar synthesis, and weakly supervised emotion-transition learning has also been explored for 3D co-speech gesture generation [23]. Related audio-to-video work further highlights the importance of aligning motion and speech during generation [24]. Other methods emphasize avatar representation quality and rendering fidelity as follows: GraphAvatar [18] uses graph neural networks for lightweight 3D Gaussian avatars, whereas SVE-NeRF [25] and relightable neural avatars [26] focus on high-fidelity, expression-conditioned head reconstruction and rendering. These methods provide important building blocks for expressive avatar synthesis, but they often emphasize facial animation, audio-driven motion, or rendering quality in isolation, rather than jointly modeling coordinated affect across the face, hands, upper body, and speech.

2.2. Multimodal Emotion Modeling and Affective Interaction

Multimodal affect modeling has advanced significantly in recent years, particularly in systems that integrate textual, acoustic, and visual cues to better capture emotional dynamics. Representative examples include Multimodal Fused Graph Convolutional Network (MMGCN) [27], which uses graph convolutional networks to fuse multimodal conversational signals, and COntextualized Graph Neural Network based Multimodal Emotion recognitioN (COGMEN) [28], which models contextual dependencies more explicitly. Joyful [29] combines joint modality fusion with graph contrastive learning, while Teacher-leading Multimodal fusion network for ERC (TelME) [30] transfers affective cues from text to non-verbal modalities through cross-modal knowledge distillation. Dynamic affect modeling has been further improved by Dialog and Event Relation-aware Graph Convolutional neural Network (DER-GCN) [31]. In parallel, more specialized vision-language approaches such as Cross-modal Emotion-aware Prompting (CEPrompt) [32] and Human Expression-Sensitive Prompting (HESP) [33] improve sensitivity to fine-grained visual expression cues, while OmniVox [34] and ExpLLM [35] explore broader multimodal and reasoning-based affect understanding. Despite these advances, most such methods focus on emotion recognition or interpretation rather than synthesizing emotionally expressive avatar behavior, and they rarely provide unified control over facial motion, gestures, and speech.

2.3. Personalized and Controllable Avatar Synthesis

Recent work has also emphasized personalization and explicit controllability. CtrlAvatar [16] uses disentangled invertible networks for controllable emotional animation, while Hierarchically Controlled Deformable Gaussians [36] enable fine-grained expression synthesis in talking heads. The Multiple Feature Refining Network (MFRN) [37] improves emotion distribution learning through iterative feature refinement. PERSE [19] generates personalized 3D avatars from a single image, and HRAvatar [20] alongside GPAvatar [21] advance high-quality Gaussian-based avatar representation. Closely related work on photorealistic and efficient avatar rendering includes FlashAvatar [38], HERA [39], and real-time Gaussian human avatars [40]. At the same time, Zero-1-to-A [41] and DAGSM [42] explore zero-shot animatable avatars and disentangled generative modeling with geometric guidance. Although these methods move toward more personalized and controllable synthesis, they usually address only part of the overall problem as follows: identity personalization, emotional control, geometric fidelity, or motion transfer. A unified framework that combines attribute-conditioned avatar appearance with explicit emotion conditioning and synchronized multimodal behavior remains comparatively underexplored.

3. Methods

The proposed MAVAGEN framework consists of the following two main components: an LLM-based chatbot that interacts with the user and an avatar generation module that synthesizes a personalized digital avatar from the collected inputs. In this section, we describe the assistant interface, avatar image retrieval, multimodal feature extraction, and multimodal fusion model.

The pipeline of MAVAGEN is shown in Figure 1. An assistant interface first requests the desired avatar attributes, specifically gender and age. The user then provides a short free-form message from which a pre-trained emotion recognition model estimates a probability distribution over categorical emotions. The same message is forwarded to the LLM-based chatbot, which generates a response that is subsequently expressed by the avatar. In parallel, a corresponding identity image is retrieved from a large corpus and processed using monocular depth estimation and landmark extraction to extract facial, hand, and upper-body keypoints. The chatbot response is converted into expressive speech using an emotion-conditioned Text-to-Speech (TTS) model driven by the predicted emotion distribution. Representations derived from the chatbot response text, the emotion-distribution vector, image, depth map, landmarks, and generated speech are processed by modality-specific encoders. The image and depth latents are combined with stochastic noise and processed by the ReferenceNet [43] model, after which the aligned multimodal embeddings are passed to the Denoising U-Net [44] model to synthesize an avatar clip with synchronized facial expressions, hand gestures, and expressive speech. The resulting clip is returned to the user as the final output. Overall, the pipeline provides a controllable way to generate avatar clips from multimodal inputs.

3.1. LLM-Based Chatbot

The interaction begins with an assistant interface, which in our implementation is realized as an LLM-based assistant (Qwen2.5-7B-Instruct (https://huggingface.co/Qwen/Qwen2.5-7B-Instruct, accessed on 31 March 2026)). The assistant requests the avatar attributes used to condition the avatar’s appearance in subsequent stages. Specifically, it first asks the user to specify gender and age. The user then responds in a standardized format, providing the attribute specification $\mathbf{a}=(G,Y)$, e.g., “Generate a Y-year-old [G] avatar”, where G denotes the desired gender and Y denotes the desired age in years. After that, the assistant continues the dialogue with a follow-up prompt such as “How can I help you today?”

The user’s free-form message is forwarded to the text emotion recognition block, where it is used to estimate the target emotional state that conditions avatar generation. The same message is also provided to the LLM-based assistant, which generates a textual response for subsequent avatar generation stages. The collected gender and age attributes are forwarded to the avatar image retrieval block, where they guide the retrieval of a visually appropriate identity image from the HaGRIDv2-1M [45] (https://github.com/hukenovs/hagrid/tree/Hagrid_v2-1M, accessed on 31 March 2026) corpus by matching the requested gender and the exact requested age whenever available, or, otherwise, the nearest available age. This setup aligns the synthesized avatar with the requested attributes while enabling emotion-conditioned generation.

3.2. Avatar Image Retrieval

Based on the requested avatar attributes $\mathbf{a}=(G,Y)$, the avatar image retrieval block selects a retrieved identity image $I^{\ast }\in \mathcal{I}$ from the HaGRIDv2-1M [45] corpus, where $\mathcal{I}$ denotes the full image pool, G denotes the desired gender, and Y denotes the desired age in years. This corpus was selected for this task because it provides a large and diverse pool of identity images together with the metadata required for attribute-conditioned retrieval.

To retrieve a visually appropriate identity image, we first restrict the candidate pool to images with the requested gender as follows:

$${\mathcal{I}}_G=\{I\in \mathcal{I}\mid g\left(I\right)=G\},$$

(1)

where $g\left(I\right)$ denotes the gender metadata associated with image I.

Within this gender-matched subset, we search for the closest available age to the requested age Y. Let $y\left(I\right)$ denote the age metadata associated with image I. We define the minimum age difference as

$${\delta }^{\ast }=\underset{I\in {\mathcal{I}}_G}{min}|y\left(I\right)-Y|.$$

(2)

If an image with the exact requested age is available, then ${\delta }^{\ast }=0$; otherwise, the retrieval procedure falls back to the closest available age within the same gender subset.

We then define the set of best-matching identity-image candidates associated with the requested attributes $\mathbf{a}$ as

$${\mathcal{I}}^{\ast }\left(\mathbf{a}\right)=\{I\in {\mathcal{I}}_G\mid |y\left(I\right)-Y|={\delta }^{\ast }\}.$$

(3)

Finally, the retrieved identity image is sampled uniformly from this set as follows:

$$I^{\ast }\sim \mathrm{Unif}\left({\mathcal{I}}^{\ast }\left(\mathbf{a}\right)\right).$$

(4)

This retrieval procedure ensures that the selected identity image matches the requested gender and corresponds to either the exact requested age or, when an exact match is unavailable, the nearest available age.

3.3. Multimodal Feature Extraction

3.3.1. Landmark Features

The goal of this stage is to obtain a compact geometric description of the avatar’s face, hands, and upper body that can be used by the trainable Landmark Encoder and, later, by the diffusion model. As illustrated by the “Finding landmarks” block in Figure 1, we apply a frozen holistic landmark detector directly to the retrieved identity image $I^{\ast }$.

Holistic landmark detection. For landmark extraction, we use the MediaPipe Holistic Landmarker (https://ai.google.dev/edge/mediapipe/solutions/vision/holistic_landmarker, accessed on 31 March 2026). Given the preprocessed RGB image $I^{\ast }$, the detector predicts 2D keypoints for the face [46,47], both hands [48], and the upper body [49]. We denote the resulting landmark configuration by

$$L\in {\mathbb{R}}^{N_L\times 2},$$

(5)

where $N_L$ is the total number of detected keypoints and each row of L contains the $(x,y)$ coordinates of one keypoint. All coordinates are returned in image pixels and are converted to the $[0, 1]$ range by dividing them by the image width and height. The detector also provides a visibility flag for each keypoint, which is stored as a binary mask

$${\mathbf{m}}_L\in {\{0,1\}}^{N_L}.$$

(6)

Normalization and vectorization. To reduce sensitivity to global translation and scale, the landmark configuration is re-centered at a torso reference joint and normalized by the distance between the left and right shoulders. Let $\tilde{L}\in {\mathbb{R}}^{N_L\times 2}$ denote the resulting normalized landmark coordinates. The normalized coordinates and the visibility mask are then concatenated and flattened into a fixed-length vector as follows:

$$\tilde{\mathbf{l}}=\mathrm{vec}\left(\tilde{L},{\mathbf{m}}_L\right)\in {\mathbb{R}}^{d_L^{\mathrm{in}}},$$

(7)

where $\mathrm{vec}(·)$ denotes concatenation followed by vectorization, and $d_L^{\mathrm{in}}$ is the dimensionality of the resulting landmark input vector.

This vector serves as the input to the trainable Landmark Encoder $E_L$. In our implementation, $E_L$ is realized as a Mamba2-based model [50], which produces the landmark-based pose feature representation

$${\mathbf{z}}_L=E_L\left(\tilde{\mathbf{l}}\right)\in {\mathbb{R}}^{d_L},$$

(8)

where $d_L$ is the dimensionality of the landmark-based pose representation.

3.3.2. Appearance Features and Depth Features

In addition to 2D landmarks, the avatar generator relies on the following two complementary visual signals extracted from the retrieved identity image $I^{\ast }$: an RGB image and a depth map [51]. Both signals are processed by frozen modules and correspond to the “Image encoder”, “Image-to-depth”, and “Depth encoder” blocks in Figure 1.

To obtain a depth map from a single RGB image, we use the pre-trained Marigold image-to-depth [52] (https://github.com/prs-eth/marigold, accessed on 31 March 2026) model. Given the retrieved identity image $I^{\ast }$, the model predicts the corresponding depth map

$$D^{\ast }=M_{\mathrm{depth}}\left(I^{\ast }\right),$$

(9)

where $M_{\mathrm{depth}}$ denotes the frozen image-to-depth model.

The RGB image $I^{\ast }$ and the corresponding depth map $D^{\ast }$ are then encoded using a pre-trained Variational Autoencoder (VAE) [53] to obtain compact latent visual representations. Both inputs are standardized using the official VAE preprocessing pipeline, which includes resizing, cropping, and normalization.

During training, Gaussian noise [54] is added to the latent visual representations to regularize the conditioning process. The resulting RGB and depth latents are then passed to the trainable ReferenceNet [43] backbone. ReferenceNet acts as an intermediate visual-conditioning module between the static avatar reference and the generative diffusion backbone: it transforms the latent appearance and geometry cues extracted from the selected image into spatial feature maps, denoted by ${\mathbf{z}}_I$ and ${\mathbf{z}}_D$, that preserve identity-specific texture, coarse structural layout, and view-consistent geometry. These feature maps are subsequently injected into the Denoising U-Net as visual conditioning signals, allowing the diffusion model to preserve the reference appearance throughout generation rather than relying solely on compact global embeddings.

The appearance feature maps ${\mathbf{z}}_I$ and depth feature maps ${\mathbf{z}}_D$ provide the avatar generator with complementary stylistic and geometric information as follows: ${\mathbf{z}}_D$ stabilizes head pose and camera geometry, while ${\mathbf{z}}_I$ maintains visual identity consistency across the generated video.

3.3.3. Emotion Features and Text Features

The user input (see Figure 1) is used to estimate the target emotional state that conditions avatar generation, while the response generated by the LLM-based chatbot provides the semantic content to be expressed by the avatar. In this subsection, we describe how these two texts are processed to obtain an emotion probability vector and a contextual text embedding.

Emotion-aware text encoder. Let U denote the user input, and let R denote the textual response generated by the LLM-based chatbot. The text-processing stage uses a pre-trained RoBERTa-based model (https://huggingface.co/michellejieli/emotion_text_classifier, accessed on 31 March 2026) to process these two texts and produce two outputs as follows:

$${\mathbf{z}}_E\in {\mathbb{R}}^K,{\displaystyle \sum _{k=1}^K}{z}_{E,k}=1,$$

(10)

where ${\mathbf{z}}_E$ is the emotion probability vector over K basic emotions obtained from U via the classifier head and a softmax layer, and

$${\mathbf{z}}_T\in {\mathbb{R}}^{d_T},$$

(11)

where ${\mathbf{z}}_T$ is a contextual text embedding computed from R as the pooled representation of the final Transformer layer, and $d_T$ is the dimensionality of the text embedding. These outputs correspond to the “Text emotion recognition” and “Text encoder” blocks in Figure 1. The model parameters are kept frozen and are not updated during training or evaluation.

Feature usage. The emotion vector ${\mathbf{z}}_E$ derived from the user input U is forwarded to two components of the pipeline. First, it conditions the emotion-conditioned TTS block, shaping the prosody of the waveform generated from the chatbot response (Section 3.3.4). Second, it is passed directly to the Denoising U-Net [44] as part of the overall conditioning signal (Section 3.4). The embedding ${\mathbf{z}}_T$ derived from the chatbot response R forms the textual feature stream in the multimodal representation and encodes the lexical content and discourse-level semantics of the text to be spoken by the avatar, influencing the communicative intent of the generated avatar behavior.

3.3.4. Acoustic Features

The acoustic module of the pipeline converts the chatbot response and the emotion probability vector inferred from the user input into expressive speech and then extracts a compact acoustic representation. This module corresponds to the “Text-to-speech with emotion” and “Acoustic encoder” blocks in Figure 1 and consists entirely of frozen components.

Emotion-conditioned text-to-speech. As described in Section 3.3.3, emotion recognition applied to the user input U produces an emotion probability vector ${\mathbf{z}}_E\in {\mathbb{R}}^K$ over K basic emotions. This vector is used to control an emotion-conditioned DIA-TTS model (https://huggingface.co/nari-labs/Dia-1.6B, accessed on 31 March 2026). The model receives the triplet $(R,{\mathbf{z}}_E,\mathbf{a})$, where $\mathbf{a}$ is the avatar attribute specification introduced above, and it generates a waveform $s\left(t\right)\in \mathbb{R}$ sampled at a fixed rate of 16 kHz. The emotion probability vector ${\mathbf{z}}_E$ modulates prosodic characteristics such as intensity, speaking rate, and pitch range, allowing the synthesized speech to reflect mixtures of basic emotions rather than a single discrete label.

Acoustic feature extraction. To obtain a compact representation of the generated speech, the waveform $s\left(t\right)$ is processed by a frozen acoustic encoder inspired by Open Whisper-style Speech Model Connectionist Temporal Classification (OWSM-CTC) [55]. First, $s\left(t\right)$ is converted into a log-mel spectrogram, which is then passed through a stack of convolutional and Transformer layers to produce a sequence of frame-level embeddings. These embeddings are average-pooled over time, yielding the acoustic feature representation

$${\mathbf{z}}_A\in {\mathbb{R}}^{d_A},$$

(12)

where $d_A$ is the dimensionality of the acoustic feature representation. The vector ${\mathbf{z}}_A$ captures articulatory dynamics, rhythm, and prosodic patterns that are important for lip motion, jaw movements, and subtle head and body synchrony with the audio. It forms the acoustic feature stream in the proposed multimodal fusion model and is fed directly to the Denoising U-Net [44] as part of the conditioning signal.

3.4. Multimodal Fusion Model

The multimodal fusion stage is illustrated in the lower part of Figure 1. At this stage, all feature streams extracted in the previous subsections are used to condition a trainable Denoising U-Net [44] with temporal attention [56], which operates in the latent space of a VAE. During training, this module learns to generate temporally coherent avatar videos from multimodal conditioning signals in a standard diffusion noise-prediction setting; during inference, iterative denoising produces the final avatar clip. The resulting latent video is then decoded to obtain the final “Digital avatar” clip.

The frozen feature extractors described above, together with the trainable modules, jointly form the multimodal fusion model of the avatar generator. In MAVAGEN, the conditioning signal is formed from the following six feature streams extracted from the multimodal inputs introduced above: (i) textual features ${\mathbf{z}}_T\in {\mathbb{R}}^{d_T}$, encoding semantic content; (ii) emotion-distribution features ${\mathbf{z}}_E\in {\mathbb{R}}^K$, represented by an emotion probability vector and encoding the target emotion; (iii) landmark-based pose features ${\mathbf{z}}_L\in {\mathbb{R}}^{d_L}$; (iv) acoustic features ${\mathbf{z}}_A\in {\mathbb{R}}^{d_A}$, encoding prosodic information; (v) RGB-appearance feature maps ${\mathbf{z}}_I$; and (vi) depth-geometry feature maps ${\mathbf{z}}_D$. The textual, emotion, landmark-based pose, and acoustic streams are represented as compact vectors, whereas ${\mathbf{z}}_I$ and ${\mathbf{z}}_D$ are spatial feature maps produced by the visual-conditioning branch.

Denoising U-Net with temporal attention. The fusion backbone, depicted as “Denoising U-Net” in Figure 1, is a U-Net-style [57] network that processes ${\mathbf{x}}_t$ as a sequence of latent feature maps along the temporal dimension, where ${\mathbf{x}}_t$ denotes the noisy latent representation of the avatar video at diffusion step t. Convolutional blocks operate on each frame spatially, while dedicated “Temporal Attention” blocks (gray modules between the colored bars in the figure) perform self-attention [58] across frames to enforce smooth and coherent motion. A diffusion time embedding is injected into the network so that its behavior adapts to the current diffusion noise level.

Multimodal conditioning of the diffusion model. In the full MAVAGEN configuration, the Denoising U-Net $D_{\theta }$ receives the noisy latent video ${\mathbf{x}}_t$, the diffusion step t, the vector-valued conditioning signals ${\mathbf{z}}_T$, ${\mathbf{z}}_E$, ${\mathbf{z}}_L$, and ${\mathbf{z}}_A$, and the spatial visual-conditioning feature maps ${\mathbf{z}}_I$ and ${\mathbf{z}}_D$. The network predicts the diffusion noise as

$${\widehat{\mathsf{ϵ}}}_t=D_{\theta }\left({\mathbf{x}}_t,t,{\mathbf{z}}_T,{\mathbf{z}}_E,{\mathbf{z}}_L,{\mathbf{z}}_A,{\mathbf{z}}_I,{\mathbf{z}}_D\right).$$

(13)

The vector-valued conditioning signals are projected to a common conditioning space and injected through cross-attention and feature-wise modulation, whereas the appearance and depth feature maps are injected into multiple layers of the U-Net as spatial visual conditioning signals. In this way, the latent video is jointly conditioned on textual content, target emotion, landmark-based pose, visual appearance, depth geometry, and prosody, resulting in avatar clips that are semantically appropriate, emotionally aligned, and visually coherent.

Decoding into a digital avatar. After S denoising steps, the final latent sequence ${\widehat{\mathbf{x}}}_0$ represents a clean, temporally coherent avatar video in the VAE latent space. The frozen VAE decoder [44], depicted as “Variational Autoencoder Decoder” in Figure 1, maps this sequence back to RGB frames, producing the final “Digital avatar” clip. This clip preserves the requested visual appearance, exhibits synchronized facial expressions, hand gestures, and expressive speech, and constitutes the final generated avatar clip returned by the system.

4. Experiments and Results

4.1. Research Corpus

All experiments are conducted on the HaGRIDv2-1M hand gesture corpus [45], a large-scale extension of the HaGRID corpus containing 1,086,158 FullHD RGB images of 65,977 adult subjects performing 33 gesture classes (including many emotional gestures) and a no_gesture class. The data are collected predominantly indoors under diverse backgrounds, lighting conditions, and camera-to-subject distances, which lead to substantial variation in appearance and hand poses. In our study, HaGRIDv2-1M serves solely as the source of avatar images. For each avatar request, an identity image is retrieved from the training split using the attribute-conditioned retrieval procedure described in Section 3.2 and used as a static identity reference. This reference is then processed by the landmark, depth, and image encoders to condition the generation of the digital avatar clip. These retrieved identity images are further used to construct a synthetic set of avatar clips for training and quantitative evaluation of the avatar generator.

4.2. Experimental Setup

All avatar-generator models are trained on a synthetic set of avatar clips built from static identity images retrieved from the HaGRIDv2-1M training split at a spatial resolution of $512\times 512$ pixels and a frame rate of 25 fps. We use the AdamW optimizer [59] with weight decay $0.01$, an initial learning rate of $1\times {10}^{-4}$, a cosine annealing schedule [60], and a linear warm-up during the first 5% of training steps. The Denoising U-Net is trained with a standard noise-prediction loss in the VAE latent space, and stochastic weight averaging is applied at regular intervals. At inference time, all models use the same deterministic sampler with $S=30$ diffusion steps. Inference was performed on a workstation equipped with an NVIDIA RTX 4090 GPU and an AMD Ryzen 9 5950X CPU. Generating one 5 s avatar clip at 25 fps, corresponding to 125 output frames, takes approximately 500 s. This corresponds to an effective inference speed of about 0.25 frames per second, while 25 fps denotes the playback frame rate of the generated video rather than real-time computational throughput. The resulting avatar clips are cached and used for all quantitative evaluations of the avatar generator.

4.3. Performance Measures

To evaluate the quality of the generated avatars, we compute the standard image- and video-generation metrics. Overall frame realism is measured with Fréchet Inception Distance (FID) [61] between real and generated frames, while temporal coherence is assessed with Fréchet Video Distance (FVD) [61], computed in the feature space of a pre-trained action-recognition network. Frame-wise structural fidelity between real and generated video frames is quantified with Structural Similarity Index Measure (SSIM) [62] and Peak Signal-to-Noise Ratio (PSNR). Following EchoMimicV2 [44], we additionally report Expression FID (E-FID), which measures the distance between real and generated videos in a facial-expression embedding space, an audio-visual synchrony distance (Sync-D), and the following three motion-consistency metrics: Head-Keypoint Consistency (HKC), Head-Keypoint Variance (HKV), and Cosine Similarity of Motion Features (CSIM). Since the primary role of the avatar is to convey emotion and synchronize with speech, we also report an audio-visual synchrony score (Sync-C) [63], obtained from a SyncNet-style model, as well as EmoAcc as an auxiliary task-specific measure of emotional agreement. To complement the automatic metrics with human perception, we additionally conducted a subjective Mean Opinion Score (MOS) evaluation. Generated videos from the compared methods were presented to 21 independent participants without method labels, and the order of videos was randomized separately for each participant. Each video was rated on a 10-point scale according to overall generation quality, including speech synchronization with facial and body movements, consistency with the reference image, naturalness of motion, and visible visual artifacts. The final MOS was reported as mean ± standard deviation across participants and evaluated clips for each method.

For EmoAcc, the target emotion is defined as the dominant class obtained by applying $argmax$ to the emotion probability vector ${\mathbf{z}}_E$. A multimodal emotion classifier (MASAI [64]) is then applied to each generated clip together with its corresponding textual response. For clip i, this classifier predicts an emotion label ${\widehat{y}}_i$ by jointly modeling facial, acoustic, and linguistic features. Formally, the target label is defined as

$$y_i^{\ast }=arg\underset{1\le k\le K}{max}{z}_{E,k}^{\left(i\right)},$$

(14)

and the binary correctness variable is defined as

$$c_i=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\widehat{y}}_i=y_i^{\ast },\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(15)

The final metric is computed as

$$\mathrm{EmoAcc}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{c}_i,$$

(16)

where $z_{E,k}^{\left(i\right)}$ denotes the k-th component of the emotion vector for clip i, and N is the number of evaluated clips.

4.4. Loss Function

The avatar generator is trained in the VAE latent space using a standard diffusion-style noise-prediction objective. At each diffusion step t, the Denoising U-Net $D_{\theta }$ receives the noisy latent video representation together with the multimodal conditioning signals described in Section 3.4 and predicts the injected Gaussian noise ${\widehat{\mathsf{ϵ}}}_t$. The training objective is the mean-squared error between the predicted and true noise as follows:

$${\mathcal{L}}_{\mathrm{diff}}=\mathbb{E}\left[{∥{\mathsf{ϵ}}_t-{\widehat{\mathsf{ϵ}}}_t∥}_2^2\right].$$

(17)

Here, ${\mathsf{ϵ}}_t$ denotes the Gaussian noise injected at diffusion step t, ${\widehat{\mathsf{ϵ}}}_t$ denotes the noise predicted by $D_{\theta }$, and $\mathbb{E}[·]$ denotes expectation over the training samples, diffusion steps, and noise realizations. All frozen modules remain fixed during training, while the trainable components of the generator are optimized jointly through this diffusion loss.

4.5. Results and Ablation Study

In this subsection, we report the quantitative results of MAVAGEN and analyze the contribution of individual feature streams. We evaluate the quality of the generated avatar videos against existing human animation methods and then study ablations over the five non-appearance feature streams (textual features, emotion-distribution features, landmark-based pose features, depth-geometry features, and acoustic features; the RGB image is always kept as the base modality).

Avatar generation performance measures and ablations. 

Table 1. Quantitative and subjective comparison of MAVAGEN with existing human animation methods. Lower FID/FVD/E-FID/Sync-D and higher SSIM/PSNR/Sync-C/HKC/HKV/CSIM/EmoAcc/MOS indicate better performance. EmoAcc is an auxiliary, task-specific metric and is only defined for MAVAGEN among the compared methods. MOS is reported as mean ± standard deviation on a 10-point scale.

Methods	FID ↓	FVD ↓	SSIM ↑	PSNR ↑	E-FID ↓	Sync-D ↓	Sync-C ↑	HKC ↑	HKV ↑	CSIM ↑	EmoAcc ↑	MOS ↑
AnimateAnyone [43]	60.10	1030.12	0.726	20.40	3.900	14.10	0.950	0.805	23.70	0.380	–	$2.64\pm 1.44$
MimicMotion [65]	55.20	635.40	0.705	19.10	2.700	8.10	1.450	0.902	24.70	0.520	–	$3.59\pm 2.20$
EchoMimicV2 [44]	50.10	605.30	0.736	21.90	2.240	7.10	7.150	0.921	25.20	0.555	–	$5.26\pm 2.16$
MAVAGEN (ours)	48.20	592.00	0.741	21.95	2.250	6.85	7.40	0.929	25.30	0.563	0.88	6.97 ± 2.35

summarizes the objective comparison of the avatar generator against existing human animation methods, while

Table 2. Ablation results for MAVAGEN together with the straightforward baseline. EmoAcc is an auxiliary, task-specific metric.

Methods	FID ↓	FVD ↓	SSIM ↑	PSNR ↑	E-FID ↓	Sync-D ↓	Sync-C ↑	HKC ↑	HKV ↑	CSIM ↑	EmoAcc ↑
w/o text features	48.90	600.00	0.738	21.80	2.300	6.98	7.25	0.925	25.05	0.559	0.84
w/o emotion vector	49.90	612.00	0.734	21.70	2.330	7.12	7.02	0.918	24.75	0.552	0.80
w/o landmark-based pose	49.10	605.00	0.737	21.75	2.310	7.00	7.15	0.922	24.90	0.556	0.83
w/o depth geometry	48.60	597.00	0.740	21.85	2.280	6.93	7.33	0.927	25.15	0.561	0.86
w/o acoustic features	50.10	617.00	0.735	21.65	2.350	7.18	6.88	0.915	24.55	0.548	0.78
w/o text + emotion	50.50	622.00	0.731	21.55	2.380	7.24	6.78	0.910	24.35	0.545	0.75
w/o text + landmarks	49.90	614.00	0.734	21.65	2.340	7.14	6.93	0.914	24.50	0.547	0.79
w/o text + depth	49.40	610.00	0.736	21.70	2.320	7.07	7.03	0.918	24.65	0.549	0.81
w/o text + audio	50.80	627.00	0.730	21.50	2.400	7.28	6.68	0.908	24.25	0.542	0.73
w/o emotion + landmarks	50.30	620.00	0.732	21.60	2.370	7.19	6.83	0.912	24.40	0.544	0.77
w/o emotion + depth	50.00	616.00	0.734	21.65	2.350	7.13	6.90	0.914	24.50	0.546	0.78
w/o emotion + audio	51.30	630.00	0.728	21.45	2.420	7.32	6.58	0.906	24.15	0.540	0.70
w/o landmarks + depth	49.70	612.00	0.735	21.68	2.330	7.09	6.98	0.916	24.55	0.548	0.80
w/o landmarks + audio	51.00	625.00	0.729	21.50	2.390	7.26	6.73	0.909	24.30	0.543	0.72
w/o depth + audio	50.60	623.00	0.732	21.58	2.360	7.21	6.80	0.911	24.40	0.545	0.74
straightforward baseline	52.80	640.00	0.668	19.40	2.800	7.30	6.70	0.828	22.70	0.510	–

reports the single-stream and pairwise ablation results together with a straightforward baseline. We follow the evaluation protocol of EchoMimicV2 [44] and compare MAVAGEN with AnimateAnyone [43], MimicMotion [65], and EchoMimicV2, reporting FID, FVD, SSIM, PSNR, E-FID, two audio-visual synchrony metrics (Sync-D, Sync-C), head-keypoint metrics (HKC, HKV), CSIM, the auxiliary, task-specific metric Emotion-preservation Accuracy (EmoAcc), and the subjective MOS. Lower FID/FVD/E-FID/Sync-D and higher SSIM/PSNR/Sync-C/HKC/HKV/CSIM/EmoAcc/MOS indicate better performance.

Compared to AnimateAnyone and MimicMotion, MAVAGEN substantially improves frame- and sequence-level visual quality (as indicated by the lower FID/FVD and higher SSIM), while remaining competitive with EchoMimicV2. The full MAVAGEN model achieves better FID and FVD than EchoMimicV2 as well as slightly higher SSIM and synchrony scores (Sync-C), at the cost of a marginally higher E-FID and essentially unchanged PSNR. In addition, MAVAGEN achieves an EmoAcc of 0.88, indicating strong agreement between the target emotion and the emotion expressed by the generated avatar. The MOS results in Figure 2 further indicate that MAVAGEN achieves the highest subjective perceived quality among the compared methods. Some representative generated videos and qualitative comparisons are available on the project webpage: https://smil-spcras.github.io/MAVAGEN/, accessed on 31 March 2026.

Rows labeled “w/o…” in Table 2 correspond to ablations of the five non-appearance feature streams. Removing the explicit emotion feature stream or acoustic features leads to the largest drops in Sync-C and EmoAcc, highlighting the central role of prosody and explicit emotion control. Landmark-based pose features and depth-geometry features also contribute noticeably by improving HKC/HKV and stabilizing CSIM, while their removal causes a moderate degradation in visual metrics. Pairwise ablations further confirm the complementary roles of textual features, emotion-distribution features, and acoustic features as follows: jointly dropping emotion-distribution features and acoustic features yields the lowest EmoAcc and the weakest synchrony, whereas combinations that keep at least one of these signals degrade more gracefully. The last row reports the straightforward baseline for reference. In all variants, the RGB-appearance feature stream is retained and serves as the base appearance cue.

5. Discussion

The experiments show that MAVAGEN improves both the visual and emotional quality of the generated avatar clips relative to recent human animation methods, while maintaining temporal coherence and audio-visual synchrony. In particular, MAVAGEN achieves competitive or improved results in terms of FID, FVD, SSIM, PSNR, E-FID, Sync-D, Sync-C, HKC, HKV, and CSIM, while also yielding a high EmoAcc value as an auxiliary measure of emotional agreement. These results indicate that multimodal conditioning is an effective mechanism for generating avatar clips that are visually plausible, temporally stable, and emotionally aligned with the target emotion, which is particularly important for personalized LLM-based chatbots and virtual assistants operating in emotion-sensitive human–computer interaction scenarios.

The comparison in Table 1 shows that MAVAGEN is competitive with state-of-the-art human animation methods such as AnimateAnyone [43], MimicMotion [65], and EchoMimicV2 [44], achieving a favorable trade-off between visual quality, synchrony, and motion consistency, while also yielding strong emotional agreement, as measured by the auxiliary EmoAcc metric. The ablation study in Table 2 indicates that not all feature streams contribute equally to these gains. The emotion probability vector ${\mathbf{z}}_E$ and the acoustic feature vector ${\mathbf{z}}_A$ are the most influential, as follows: removing either causes the largest drops in Sync-C, motion-consistency metrics, and EmoAcc, highlighting the central roles of prosody and explicit emotion control in avatar generation. Landmark-based pose features and RGB-appearance features mainly improve perceived naturalness and motion stability through richer non-verbal behavior and identity consistency, while depth-geometry features primarily stabilize head pose and geometry with a smaller but still positive effect on the performance metrics. Taken together, the pairwise ablations suggest that emotion-distribution features and acoustic features are the main drivers of emotional agreement and synchrony, while textual features remain complementary and landmark-based pose features and depth-geometry features primarily support motion stability and visual coherence. In this sense, MAVAGEN can be viewed as a modular avatar-generation framework in which each feature stream has a clear functional role, making the framework suitable for practical use in personalized LLM-based chatbots and other emotion-sensitive interactive systems.

The proposed method has several limitations. The avatar pool is restricted to HaGRIDv2-1M, which may introduce demographic and contextual biases; the emotion representation is limited to a small set of basic categories; and the current system conditions the avatar’s appearance only on gender and age attributes. In addition, MAVAGEN focuses on text-conditioned emotional speaking behavior and does not explicitly model natural listening or idle states, such as eye contact, gaze shifts, head nods, or backchannels. Addressing these behaviors would require real-time multimodal analysis of the interlocutor and more advanced dialogue-level behavior control, which is beyond the scope of the present study. Moreover, although the system uses a probability distribution over basic emotions, the current speech generation stage selects the maximum-probability emotion rather than synthesizing blended emotional speech, since the employed text-to-speech model does not explicitly support compound emotional states. Generating an avatar clip for each request remains computationally expensive, and the method alone does not guaranty safe or clinically appropriate behavior in sensitive domains. These constraints motivate future work on richer emotion representations (e.g., continuous valence-arousal spaces and compound emotional states), explicit modeling of listening and idle behaviors, more expressive text-to-speech models, more efficient diffusion backbones, broader evaluations across diverse user populations and application domains, and more detailed analyses of safety and bias in real-world emotion-sensitive applications.

6. Conclusions

In this work, we introduced MAVAGEN, a multimodal avatar generation framework for synthesizing personalized upper-body digital avatars with controllable emotional expression. The framework combines six feature streams, textual features, emotion-distribution features, landmark-based pose features, depth-geometry features, RGB-appearance features, and acoustic features, and it is designed for personalized human–computer interaction scenarios involving LLM-based chatbots and virtual assistants.

The evaluation results show that MAVAGEN achieves the strongest overall performance across most reported metrics. In terms of frame- and sequence-level realism, MAVAGEN attains an FID of 48.20 and an FVD of 592.00, as well as the highest SSIM of 0.741 and a competitive PSNR of 21.95. Expression fidelity remains on par with the strongest baseline (an E-FID of 2.25 vs. 2.24 for EchoMimicV2), while audio-visual synchrony and motion consistency are slightly improved: Sync-D decreases to 6.85, Sync-C increases to 7.40, HKC and HKV reach values of 0.929 and 25.30, and CSIM grows to 0.563. As an auxiliary, task-specific metric, EmoAcc reaches a value of 0.88 for the full model, indicating strong agreement between the target emotion and the emotion expressed by the generated avatar.

The ablation study confirms the contribution of each non-appearance feature stream. Removing only one feature stream leads to moderate degradation, with the strongest effects observed for the emotion-distribution features and acoustic features (e.g., EmoAcc drops from 0.88 to 0.80 and 0.78, and Sync-C drops from 7.40 to 7.02 and 6.88). Depth-geometry features and landmark-based pose features mainly stabilize geometry and motion (HKC/HKV and CSIM decrease when they are removed), while textual features support fine-grained control over expression and synchrony. Pairwise ablations further highlight the complementary roles of these signals: the combination without both emotion-distribution and acoustic feature streams yields the lowest EmoAcc (0.70) and one of the weakest synchrony scores (Sync-C 6.58), while configurations that retain at least one of these feature streams degrade more gently across all metrics. The subjective MOS evaluation also shows that MAVAGEN receives the highest perceived quality score among the compared methods.

In general, the results indicate that MAVAGEN effectively leverages these multimodal signals to improve visual quality, temporal coherence, audio-visual synchrony, and emotion accuracy in avatar generation, and that all five non-appearance feature streams contribute meaningfully to the final performance. These findings support the practical use of MAVAGEN in personalized LLM-based chatbots and other emotion-sensitive interactive systems. Future work will focus on extending MAVAGEN in several directions. First, we plan to move beyond basic discrete emotion categories toward richer affective representations, including continuous valence-arousal spaces and compound emotional states. Second, we plan to incorporate natural listening and idle-state behaviors, such as eye contact, gaze shifts, head nods, and neutral waiting dynamics, which are essential for realistic interactive conversations. Finally, we will improve computational efficiency for real-time interaction and scale evaluations to more diverse user populations, avatar attributes, and interactive settings.