SSL-MEPR: A Semi-Supervised Multi-Task Cross-Domain Learning Framework for Multimodal Emotion and Personality Recognition

Abstract

The growing demand for personalized human–computer interaction calls for methods that jointly model emotional states and personality traits. However, large-scale multimodal corpora annotated for both tasks are still lacking. This challenge stems from integrating diverse, task-specific corpora with divergent modality informativeness and domain characteristics. To address it, we propose SSL-MEPR, a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition, which enables cross-task knowledge transfer without jointly labeled data. SSL-MEPR employs a three-stage strategy, progressively integrating unimodal single-task, unimodal multi-task, and multimodal multi-task models. Key innovations include Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks, which enable robust fusion and effective use of semi-labeled data via a modified GradNorm method. Evaluated on MOSEI (emotion) and FIv2 (personality), SSL-MEPR achieves a mean Weighted Accuracy (mWACC) of 70.26 and a mean Accuracy (mACC) of 92.88 in single-task cross-domain settings, outperforming state-of-the-art methods. Multi-task learning reveals domain-induced misalignment in modality informativeness but still uncovers consistent psychological patterns: sadness correlates with lower personality trait scores, while happiness aligns with higher ones. This work establishes a new paradigm for extracting cross-task psychological knowledge from disjoint multimodal corpora, demonstrating that semi-supervised multi-task cross-domain learning can bridge annotation gaps while preserving theoretically grounded emotion–personality relationships.

1. Introduction

Effective interaction between humans and Intelligent Systems (ISs) depends on accurately recognizing both short-term emotions [1,2,3] and long-term personality traits (PTs) [4,5,6]. Consequently, robust Emotion Recognition (ER) and Personality Traits Assessment (PTA) capabilities have become crucial for building adaptive and context-aware systems [7,8]. Emotions and PTs are inherently interconnected [9,10]. PTs define the basic emotional background and determine the tendency to certain emotional responses, while repetitive and persistent emotions over time adjust PTs [11]. For instance, a PT such as Neuroticism demonstrably modulates emotional reactivity to negative stimuli [12], while individuals with high Extraversion tend to express their emotions in a more positive way [13]. Understanding and leveraging this interplay requires not only advanced Machine Learning (ML), but also principled knowledge extraction across psychological constructs [14]. Despite this connection ER [15,16,17,18] and PTA [19,20] tasks are typically addressed independently. Such isolated methods limit the interaction between these two tasks, which negatively affects the robustness and generalizability of the model and foregoes opportunities for cross-task knowledge transfer [21].

In order to use emotions and PTs together, we need annotated corpora covering both psychological states; however, no such fully aligned corpora exist to date. Consequently, researchers should combine multimodal corpora, which introduce significant challenges: some corpora are annotated by text modality [22,23], where emotional cues vary across contexts and video modalities suffer from noise (e.g., multiple speakers and background noise), while other corpora [12,24] are annotated by video that captures predominantly non-verbal behavior. As a result, single-task methods typically exploit only a pair of modalities. For example, text and audio are used for ER [3,25], and audio and video are applied for PTA [26]. Because corpora rely on different annotation protocols, the dominant modality is often different for ER and PTA. This discrepancy can produce conflicting gradients during joint training and degrade multi-task performance.

Crucially, the modality–task biases induced by these annotation protocols often contradict the true informativeness of the modalities when it comes to psychological signals. While emotions are predominantly conveyed through non-verbal channels, such as facial expressions [27,28,29] and vocal prosody [30], even when the verbal content is neutral or misleading [31], PTs manifest through stable linguistic styles and long-term behavioral patterns [32]. This misalignment is further aggravated by limitations of current multimodal methods [20,33,34]. Despite fusing audio, video, and text, state-of-the-art (SOTA) methods [35,36] often over-rely on the dominant modality in the source domain, leading to poor generalization when modality reliability shifts in new domains [37]. Moreover, existing methods suffer from three key shortcomings: (i) they are predominantly single-task, ignoring the interplay of emotions and PTs; (ii) they degrade significantly on semi-labeled or cross-domain data, limiting real-world applicability [37,38]; and (iii) they inadequately exploit intermodal interactions, often employing simplistic or late-level modality fusion mechanisms [25]. These limitations hinder cross-task knowledge transfer and degrade generalization.

Given these mismatched annotations and divergent modality relevance across tasks and domains, and the architectural limitations of related work, direct supervised fusion becomes unreliable. This makes semi-supervised [39,40] multi-task cross-domain learning [41] a necessary strategy for effective integration and cross-task knowledge transfer. However, semi-supervised learning alone does not resolve the annotation-induced modality biases inherited from the source corpora. In standard fusion architectures, these biases can still dominate the shared representation and limit cross-task transfer. This motivates the development of architectural designs that explicitly control the interactions between modalities and tasks.

To address these issues, we propose a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition (MEPR), which combines unimodal and multimodal processing within a coherent multi-task cross-domain architecture. Our framework integrates robust modality-specific feature extractors (Contrastive Language—Image Pre-training (CLIP) [42], Wav2Vec2.0 [43], BAAI General Embeddings (BGE)-en [44]) and temporal models (transformer [45] and Mamba [46]). It also includes four novel components to effectively combine heterogeneous and cross-domain data: (i) a Graph Attention Fusion layer to align and fuse modality-specific features by modeling their interdependencies through learned attention weights; (ii) a Task-Specific Query-based Multi-Head Cross-Attention Fusion layer to dynamically integrate cross-modal representations guided by emotion- or personality-specific queries, enabling task-aware feature interaction; (iii) predict projectors to map unimodal predictions into shared latent spaces for fusion; and (iv) guide banks to store and refine task-specific prototypes, thereby stabilizing and guiding multimodal learning across domains. Together, these components support two key abilities. First, they allow knowledge extraction across different modalities. Second, they facilitate bidirectional knowledge transfer between ER and PTA. As a result, the model can learn both general and task-specific representations. These representations capture the psychological relationship between transient affective states and stable PTs. Additionally, we introduce a three-stage learning scheme. Initially trained unimodal single-task single-domain models are refined through cross-attention mechanisms to enhance joint predictions for emotions and PTs. In the final stage, unimodal features and predictions from the multi-task cross-domain models are integrated into a unified multimodal multi-task cross-domain model. To enable semi-supervised learning (SSL), task-specific thresholds are applied to generate pseudo-labels for unlabeled data. The contributions of semi-supervised and supervised losses are weighed during training. To do this, we modify the GradNorm [47] method. Specifically, we address the following research questions:

• RQ1: Can semi-supervised multi-task cross-domain learning effectively bridge two large-scale corpora annotated for different psychological tasks (emotion vs. personality) in the absence of joint labels?
• RQ2: To what extent does cross-domain transfer improve or hinder performance, given divergent modality informativeness across annotation protocols?
• RQ3: Does the model leverage cross-task features during joint inference, and how does this reflect the underlying emotion-personality relationship?

While previous work has focused on improving accuracy in isolated tasks or domains, we take a different learning strategy. We see the lack of joint annotations as an opportunity, rather than a limitation. Our goal is to develop a learning paradigm that models cross-task psychological dependencies. The proposed components (Graph Attention Fusion, Task-Specific Query-based Multi-Head Cross-Attention Fusion, predict projectors, and guide banks) are not just architectural modules. They are mechanisms for encoding, preserving, and transferring task-specific knowledge across modalities and domains. The main contributions of this article are as follows:

• A novel learning paradigm for joint emotion–personality modeling under label scarcity and domain shift, grounded in semi-supervised cross-task knowledge transfer.
• A novel unified semi-supervised multi-task cross-domain learning framework for MEPR, explicitly designed for joint knowledge extraction across modalities, tasks and domains.
• A three-stage learning strategy to model intra- and inter-domain interactions across modalities, from unimodal to multimodal data.
• Novel components to effectively combine heterogeneous and cross-domain data, including (i) a Graph Attention Fusion layer; (ii) a Task-Specific Query-based Multi-Head Cross-Attention Fusion layer; (iii) predict projectors; and (iv) guide banks, which jointly enable structured knowledge extraction from multimodal and multi-task signals.
• Comprehensive evaluations conducted on benchmark corpora, CMU Multimodal Opinion Sentiment and Emotion Intensity (MOSEI) and ChaLearn First Impressions v2 (FIv2), demonstrating improvements over prior SOTA results.

The rest of this article is structured as follows. In Section 2, we describe SOTA Multimodal Emotion Recognition (MER), PTA, and multi-task MEPR methods. Section 3 provides a detailed explanation of the framework architecture and learning strategy. Section 4 outlines the research corpora, performance metrics, and experimental results. Section 5 discusses the advantages, limitations, and computational cost of the proposed framework, as well as its applicability in ISs and research questions. Finally, Section 6 summarizes the key findings and outlines future research directions.

2. Related Work

This section provides a brief review of SOTA methods employed for MER and PTA as well as SSL methods. Transformers, Large Language Models (LLMs) and hybrid architectures combining transformers and deep Convolutional Neural Networks (CNNs) or Long Short-TermMemory networks (LSTMs) are widely used in multimodal affective and personality computing. However, most existing methods are limited to a single task or domain, lack interaction between tasks, and require large amounts of labeled data. Therefore, the proposed SSL-MEPR framework aims to address these limitations and domain shift in MER and PTA.

2.1. State-of-the-Art Methods for Multimodal Emotion Recognition

Since multimodal information enhances ER compared to unimodal information, recent MER research has increasingly focused on multimodal methods involving text, audio, and video [48,49,50,51]. However, many of them produce imprecise predictions, suffer from limited training data, or insufficiently leverage certain modalities. To ensure balanced contributions from each modality, it is necessary to address multimodal heterogeneity [35]. For instance, Li et al. [48] proposed Decoupled Multimodal Distillation (DMD), which improves feature discrimination by separating modalities into irrelevant and exclusive subspaces via self-regression. A key limitation of DMD, however, is its failure to exploit intramodal interactions. The development of effective methods is hindered by the annotation protocols in existing corpora [25,52], which employ either multi-class classification (a single label per sample) or multi-label binary classification (multiple labels per sample). For example, Zhang et al. [53] introduced versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), a multi-label-based method that improves multimodal representations and promotes modality-specific diversity. However, this method is trained and evaluated on a single corpus, leading to overfitting and poor generalization. To address this issue, Ryumina et al. [38] proposed multi-corpus learning with context-independent gated attention for combining multimodal features and improving performance on unseen data. He et al. [54] employed an adversarial network and a label-based contrastive learning method to enhance the audio–textual representations of modality-specific emotional features. Also, Yin et al. [55] used a Knowledge-Guided Cross-Attention mechanism to mitigate the affective distribution shift in visual data. Zhao et al. [56] proposed a network that enables source-free cross-corpus speech ER by leveraging emotion-aware contrastive learning to adapt a pre-trained model to an unlabeled target corpus to enhance feature discriminability. On the contrary, Gao et al. [57] developed a transformer-based architecture to learn a shared feature distribution for both source and target speech domains. However, most cross-corpus research is limited to a single modality, as difficulty arises in keeping a correlation between modalities within more than one corpus.

In recent years, transformers have been widely used in MER, successfully integrating relevant information from different modalities [25,36,58,59]. Modality-specific transformer-based feature extractors, such as Bidirectional Encoder Representations from Transformers (BERT) [60], the Robustly optimized BERT approach (RoBERTa) [61], Wav2Vec2.0 [43], and Vision Transformer (ViT) (https://huggingface.co/google/vit-base-patch16-224, accessed on 3 December 2025), are applied to improve MER performance. Farhadipour et al. [36] combined these pre-trained extractors with a hybrid CNN–transformer-based model using local attention to highlight relevant temporal information. In contrast, Deng et al. [25] employed Cross-Modal Multi-Head Attention to aggregate audio and text modalities.

The fusion of modalities in SOTA MER methods is not limited to transformer-based architectures. Liu et al. [62] proposed a multimodal fusion method in which acoustic and facial features are unified using the chi-square test. Similarly, Hosseini et al. [49] used the chi-square test to select informative features and exclude redundant or noisy ones. LLMs [63,64,65] have recently shown strong performance in MER. The novel task of ER combined with a Multimodal Large Language Model (MLLM), employing non-facial body language, acoustic, and linguistic features, was introduced by [66]. Chandraumakantham et al. [63] proposed an LLM-based method for early multimodal fusion. After converting audio and video to text and performing augmentation, the method used multimodal text representations to generate prompts for an LLM.

Table 1 shows a comparative overview of MER methods. SOTA methods in MER leverage multimodal fusion (e.g., transformers, DMD, and Motion, Image, Speech, and Text (MIST)) to overcome unimodal limitations, yet face challenges in modality balance and generalization. Hybrid architectures (CNN–transformers) and LLM-enhanced methods now dominate, employing advanced mechanisms such as Cross-Modal Attention and Dynamic Fusion to improve MER performance.

Table 1. Overview of recent MER methods. Abbreviations: CMA (Cross-Modal Attention), MHA (Multi-Head Attention), FLF (Feature-Level type of Fusion), HCME (Hierarchical Cross-Modal Encoder), LGD (Label-Guided Decoder), DMR (Diffusion-based Modality Recovery), GKD (Graph-based Knowledge Distillation), CMGFN (Cross-Modal Gated Fusion Network), and MMTCDM (Multimodal Multi-Task Cross-Domain Model).

Year	Method	Features					Modality Fusion	Learning Type	Learning Domain	Corpora Used
		Linguistic	Acoustic	Facial	Body	Scene
2022	Zhang et al. [53]	GloVe, Transformer	COVAREP, Transformer	Facet, Transformer	–	–	FLF, HCME, LGD	Supervised	Single-domain	MOSEI
2023	Wang et al. [52]	BERT	COVAREP	Facet	–	–	FLF, DMR	Supervised	Single domain	MOSI, MOSEI
2023	Li et al. [48]	GloVe, BERT	COVAREP	Facet	–	–	FLF, GKD	Supervised	Single-domain	IEMOCAP, MOSEI, AFEW, MOSI
2024	Chandraumakantham et al. [63]	DistilRoBERTa	OpenSmile	PyFeat	–	–	FLF, LLM	Supervised	Single-domain	RAVDESS, BAUM-1
2024	Hosseini et al. [49]	BiLSTM	CNN-LSTM	Inception-ResNet-v2	–	–	FLF, DNN	Supervised	Single-domain	IEMOCAP
2024	Peng et al. [35]	GloVe, Transformer	COVAREP, Transformer	Facet, Transformer	–	–	FLF, CRLN	Supervised	Single-domain	MOSEI, M3ED
2024	Deng et al. [25]	300-dim embeddings, Transformer	MFCC, Transformer	–	–	–	FLF, CMA, MHA	Supervised	Single-domain	MELD
2025	Ryumina et al. [38]	RoBERTa	Wav2Vec2.0	ResNet-50	–	–	FLF, CMGFN	Supervised	Cross-domain	MOSEI, MELD, IEMOCAP, AFEW
2025	Li et al. [66]	LLaMa	Qwen2-Audio	–	Video Q-Former	Qwen2.5-VL	FLF, LLM	Supervised	Single-domain	DEEMO-NFBL, DEEMO-MER
2025	Farhadipour et al. [36]	RoBERTa	Wav2Vec2.0	FacialNet, BiLSTM	–	CNN, Transformer	FLF, CNN, Transformer	Supervised	Single-domain	MELD
2025	He et al. [54]	BERT	HuBERT	–	–	–	FLF, CNN, Transformer	Supervised	Cross-domain	IEMOCAP, MELD
2025	Ours (SSL-MEPR)	BGE-en, Mamba, Transformer	Wav2Vec2.0, Mamba, Transformer	CLIP, Mamba, Transformer	CLIP, Mamba, Transformer	CLIP, Mamba, Transformer	MMTCDM	Semi-supervised	Cross-domain	MOSEI, FIv2

2.2. State-of-the-Art Methods for Personality Traits Assessment

PTA has gained popularity as ISs increasingly adopt personalized interactions [67]. While ER facilitates such systems to adapt their intonation, personalization allows interactions to be tailored to individual personality. The Big Five [68,69,70] model is a widely known framework describing five traits, Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, also known as the OCEAN model [12]. To evaluate PTs, multiple modalities are typically combined, with visual features playing a central role and often decomposed into distinct visual modalities such as face, body, scene, and gestures. For example, Aslan et al. [71] employed scene, facial, acoustic, and linguistic features. They combined deep CNNs [72] and an LSTM [73] network to model spatial and temporal aspects of the data, demonstrating effectiveness in PTA. This highlights that CNNs and LSTMs are widely used Deep Learning (DL) methods commonly applied in both MER and PTA [74].

Hybrid architectures have also been used to improve the accuracy of PTA. For example, Zhao et al. [74] combined CNN, Bidirectional LSTM network (BiLSTM) [75], and transformer models with decision-level fusion to extract detailed spatio-temporal representations from audio–visual inputs. Wang et al. [32] presented a Graph Relational Network (GRN) to model intra- and intermodal relationships among scene features extracted from video data. Kong et al. [26] used a feature-level fusion module as a key component to analyze visual and audio modalities. They incorporated a cross-attention mechanism to suppress redundant signals and emphasize highly discriminative features.

Ryumina et al. [4] proposed the Gated Siamese Fusion Network (GSFN) instead of a conventional CNN, employing the fusion of hand-crafted and deep features across face, audio, and text modalities. Experimental results showed that pairwise comparison of the PTs of two individuals improved the overall performance of the method. Agrawal et al. [76] presented Forced Attention (FAt), a transformer-based architecture designed to address challenges related to limited data and background noise. The model improved performance across diverse tasks and inputs while requiring fewer computational and memory resources.

Table 2 shows a comparative overview of the PTA methods. Transformers have shown higher-performing results for PTA than LSTMs [76]. For example, FAt achieved outstanding results on several corpora, including UDIVA [24] and FIv2 [12]. Nevertheless, hybrid DL methods enhance the robustness of the overall multimodal network, as well as each single modality.

Table 2. Overview of recent PTA methods. Abbreviations: FLF (Feature-Level type of Fusion), PLF (Probability-Level type of Fusion), DLFS (Decision-Level Fusion strategy), FAt-CA (Forced Attention transformer Cross-Attention), GSFN (Gated Siamese Fusion Network), MAFFN (Multimodal Attention Feature Fusion Network), STEs (Spatio-Temporal Encodings), MMTCDM (Multimodal Multi-Task Cross-Domain Model), and Swin (Video Swin Transformer).

Year	Method	Features					Modality Fusion	Learning Type	Learning Domain	Corpora Used
		Linguistic	Acoustic	Facial	Body	Scene
2021	Aslan et al. [71]	ELMo-FCNN	VGGish-LSTM	ResNetV2-101, LSTM	–	ResNetV2-101, LSTM	FLF, Attention Module	Supervised	Single-domain	FIv2
2022	Agrawal et al. [76]	BERT	VGGish	R(2+1)D, STEs	LCRNet	R(2+1)D, STEs	FLF, Transformer, LSTM	Supervised	Single-domain	FIv2, MuPTA
2023	Zhao et al. [74]	–	VGGish	VGG-Face	–	VGG-Face	FLF, PLF, DLFS	Supervised	Single-domain	FIv2
2023	Gan et al. [77]	Personality descriptors + CLIP	–	CLIP	–	–	FLF, Regression Model	Supervised	Single-domain	FIv2
2023	Agrawal et al. [78]	XLM-RoBERTa	Trill-Distilled	R(2+1)D, Swin	–	R(2+1)D, Swin	PLF, FAt-CA	Supervised	Single-domain	FIv2, UDIVA
2024	Ryumina et al. [4]	BERT, BiLSTM, LIWC, ReBiLSTM	VGG-16, FCNN, OpenSmile, LSTM	Emo-AffectNet, LSTM, Geometric Features, LSTM	–	–	FLF, GSFN	Supervised	Single-domain	FIv2, MuPTA
2025	Kong et al. [26]	–	MFCC	EfficientFace	–	–	FLF, Feature Concatenation, Attention Module	Supervised	Single-domain	FIv2
2025	Wang et al. [32]	CLIP	Wav2CLIP	–	–	LSTM	FLF, MAFFN	Supervised	Single-domain	FIv2, UDIVA
2025	Ours (SSL-MEPR)	BGE-en, Mamba, Transformer	Wav2Vec2.0, Mamba, Transformer	CLIP, Mamba, Transformer	CLIP, Mamba, Transformer	CLIP, Mamba, Transformer	MMTCDM	Semi-supervised	Cross-domain	MOSEI, FIv2

2.3. State-of-the-Art Multi-Task Emotion and Personality Recognition

Some studies have introduced SOTA multi-task methods for several affective state recognition [7,79,80]. However, only a limited number of studies explore the interaction between ER and PTA. Therefore, the question remains crucial. Li et al. [79] highlighted the direct influence of emotion on personality. The authors applied the Sigmoid function and a cross-entropy loss to develop a multi-task learning framework that simultaneously predicts both PTs and emotions from text. M. Talaat et al. [7] proposed an enhanced multi-task model for unimodal Emotion and Personality Recognition (EPR) based on text data from social media. For the multiclass classification task, Gray Wolf Optimization (GWO) [81] was applied for hyperparameter optimization, while a group variant of GWO was used for feature selection.

Gao et al. [13] presented a Temporal Condition Network (TCN) in which personality features are integrated with Hidden-unit BERT (HuBERT)-based [82] acoustic features for speech ER. Li et al. [83] proposed methods based on transformer and emotional Wav2Vec2.0 models to infer PTs from speech. Several studies are grounded in psychological findings suggesting that human emotions are influenced by PTs. For example, Wen et al. [84] proposed a method designed to address personality-affected emotion generation via personality-driven mood transition modeling. To extract relevant information, the pre-trained BERT was fine-tuned, and a token-level attention mechanism based on Valence, Arousal, and Dominance (VAD) embeddings was constructed.

PTA has been extensively explored, guided by emotion analyses [11,85]. To improve the accuracy of PTA, Liu et al. [86] employed the Graph Convolution method. The method performed classification using two feature types: psychological language knowledge and sentiment semantics. In particular, Ryumina et al. [4] integrated emotional features from face, audio, and text modalities to enhance PTA by matching features between two people using a GSFN.

Several studies have explored physiological signals. For example, Pant et al. [87] proposed Physiological dataset for Multimodal Emotion Recognition (PhyMER), a method for ER based on physiological responses, with PTs used as contextual information. The study also leveraged the video modality for ER and introduced an annotation tool to label elicited affective states.

The relationship between PTs and emotions has been widely studied via some methods that examine emotional features for PTA (e.g., Ryumina et al. [4], Bao et al. [11], Wang et al. [85], and Liu et al. [86]) and others have investigated the inverse direction of influence (e.g., Wen et al. [84]). However, existing multi-task methods are predominantly unimodal, relying solely on text [7,79] or audio [13,83], thus limiting their relevance for real-world applications, such as affective computing, mental health monitoring, or personalized recommendation systems. A multimodal cross-domain learning method could address these limitations and improve practical applications.

2.4. State-of-the-Art Methods Based on Semi-Supervised Learning

SSL effectively integrates supervised and unsupervised learning methods to address the key challenges of both techniques and improve the overall model performance [88]. The aim of this task is to train models using both a small amount of labeled data and a large amount of unlabeled data.

SSL acts as a key point in recent research covering ER [89], PTA [90], and other mental states [91]. However these studies primarily focus on unimodal methods, such as text [90], video [92] and physiological signals [89].

Some SSL-based methods aim to address the challenges of label ambiguity and heavy annotation cost. For example, Lian et al. [93] proposed a unified semi-supervised method, where intramodal and cross-modal interactive modules jointly exploit labeled and unlabeled data through combined reconstruction–classification losses. Alameer et al. [89] employed adversarial modules, such as the domain discriminator and the subject discriminator, implemented with a Gradient Reversal Layer (GRL) and cross-entropy loss.

Several existing works have leveraged SSL models to eliminate noise. For instance, Zhu et al. [90] proposed contrastive learning to exploit unlabeled data through an augmented Graph Neural Network (GNN). Ouali et al. [88] leveraged unlabeled data through cross-consistency regularization to effectively exploit low-density regions in the latent feature space.

Despite the significant contribution of the above-mentioned methods to the performance of SSL, they are not fully effective. SOTA methods do not consider limitations such as the correlation between labels or apply only a single modality, covering one task along with similar data domains.

3. Proposed Framework

Figure 1 illustrates the proposed semi-supervised multi-task cross-domain learning framework for MEPR. This framework produces multi-task predictions from heterogeneous input modalities, including face, body, scene, audio, and text. It enables knowledge extraction across modalities and psychological constructs by learning representations that encode both task-specific cues and cross-task dependencies. The framework operates under a multi-task learning paradigm, where both unimodal and multimodal representations are used to capture both modality-specific and cross-domain interactions. At the same time, it reveals shared and task-specific patterns that reflect the relationship between transient emotions and stable PTs.

Figure 1. The pipeline of the semi-supervised multi-task cross-domain learning framework for MEPR. The flame icon represents the use of trained models.

In the proposed framework, unimodal input signals are first processed using specialized Unimodal Multi-Task Cross-Domain Models. These models are trained in a cross-domain setting, allowing them to extract modality-specific features and generate predictions for each modality separately. The extracted features and modality-specific predictions are subsequently fused through effective integration and learning from heterogeneous data across different domains. The final output includes emotion predictions (Neutral, Anger, Disgust, Fear, Happiness, Sadness, and Surprise) along with PT scores (Openness, Conscientiousness, Extraversion, Agreeableness, and Non-Neuroticism).

From an ML standpoint, jointly modeling ER and PTA is justified by their inherent interconnection. This implies that shared or interactively refined representations can improve generalization, especially when there is limited label data or when the domain shifts. Rather than using hard parameter sharing, which assumes identical feature relevance, our framework allows the model to preserve task-specific features while transferring complementary information between tasks. This aligns with the psychological structures underlying these tasks and is consistent with transfer learning and knowledge distillation principles. In the proposed framework, both task-specific and shared parameters adapt to capture task-specific representations within a unified latent space. Subsequent sections provide a detailed explanation of each model component.

3.1. Video-Based Models

Video information is divided into three regions of interest: face, body, and scene. For each region, a dedicated temporal model is built for subsequent multimodal fusion. All regions are extracted from video sequences by uniformly sampling 30 evenly spaced frames from each video.

For each frame, face and body regions are detected using the MediaPipe (https://github.com/google-ai-edge/mediapipe, accessed on 3 December 2025) and the YOLOv13 (https://github.com/iMoonLab/yolov13, accessed on 3 December 2025) models, respectively. Videos that do not contain any faces or bodies are excluded from further analysis to maintain data integrity.

We employ three distinct static models for visual feature extraction. The EmoNeXt model [94] (https://github.com/yelboudouri/EmoNeXt, accessed on 3 December 2025), based on the ConvNeXt [95] architecture, is specifically optimized to capture fine-grained emotional features. Among the available versions of EmoNeXt, we use the base version, which achieves an optimal balance between computational cost and representational performance. This model generates 1024-dimensional feature embeddings from the input face regions.

The CLIP model [42] (https://huggingface.co/openai/clip-vit-base-patch32, accessed on 3 December 2025), based on the ViT architecture with Base-32 patch size, is a multimodal model trained on 400 million image–text pairs for cross-modal alignment. Unlike traditional vision models, CLIP learns visual representations by contrasting paired images and text in a shared embedding space, allowing it to capture semantically rich features transferable to downstream tasks. This model generates 512-dimensional feature embeddings.

The ResNet-50 model [96] (https://huggingface.co/microsoft/resnet-50, accessed on 3 December 2025), based on a 50-layer convolutional architecture with residual connections, is pre-trained on ImageNet for 1000-class object recognition (e.g., person, jeans, and chair). This model produces 2048-dimensional feature embeddings that capture robust visual patterns of human bodies and scene elements, complementing our research tasks.

The extracted static features are temporally aggregated and processed using temporal models, such as transformer [45] and Mamba [46]. The transformer model uses self-attention mechanisms to capture global temporal dependencies, while Mamba employs selective state spaces and input-dependent gating for efficient long-sequence modeling. This allows it to achieve linear-time complexity, compared to the quadratic scaling of the transformer.

3.2. Audio-Based Models

We employ two specialized audio embedding models to capture paralinguistic and emotional cues from speech signals. The Wav2Vec2.0 model (https://huggingface.co/audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim, accessed on 3 December 2025) is fine-tuned for ER on the MSP-Podcast corpus. By leveraging its transformer-based encoder and context-aware representations, the model effectively captures subtle prosodic features (e.g., pitch dynamics and speech rate) and spectral patterns (e.g., harmonics).

In contrast, the Extended HuBERT (ExHuBERT) model [37] (https://huggingface.co/amiriparian/ExHuBERT, accessed on 3 December 2025) is fine-tuned through multi-corpus and multi-task settings using 37 benchmark emotion corpora, spanning acted and spontaneous speech across diverse languages and domains. This model captures speaker-invariant emotional features through its masked prediction objective and multi-corpora regularization. Both models produce 1024-dimensional embeddings that are particularly robust to acoustic variability in natural speech.

Consistent with the video modality, we use two temporal models (transformer and Mamba) to model sequential patterns and temporal dependencies in speech. To ensure consistency across research corpora with varying sampling rates and durations, all audio signals are resampled to 16 kHz. During batch formation, we apply zero-padding to the extracted embeddings so they match the length of the longest sequence in the batch.

3.3. Text-Based Models

We use two different text embedding models to extract linguistic features. The Jina-v3 model [97] (https://huggingface.co/jinaai/jina-embeddings-v3, accessed on 3 December 2025), based on the XLM-RoBERTa architecture (https://huggingface.co/jinaai/xlm-roberta-flash-implementation, accessed on 3 December 2025), is a multilingual transformer that has been pre-trained on 30 languages using contrastive learning. This model generates 1024-dimensional embeddings. These embeddings capture cross-lingual semantic relationships through the 24-layer bidirectional attention mechanism.

In contrast, the monolingual BGE-en model (https://huggingface.co/BAAI/bge-large-en, accessed on 3 December 2025) is based on a BERT-like architecture refined through three-stage learning: (i) masked language modeling pre-training, (ii) contrastive learning with hard negative mining, and (iii) multi-task fine-tuning on labeled corpora. The model produces compact 384-dimensional embeddings optimized for semantic retrieval tasks.

Similar to the other modalities, we employ two temporal models (transformer and Mamba) to capture contextual information in the utterances and model task-specific patterns. The utterance length varies across samples in the research corpora. To address this variability, we apply zero-padding to all embeddings during batch formation, standardizing them to the length of the longest sequence within each batch.

3.4. Unimodal Multi-Task Cross-Domain Model

The pipeline of the proposed Unimodal Multi-Task Cross-Domain Model for unimodal EPR is illustrated in Figure 2. This model consists of three main components: an emotion-specific branch, a personality-specific branch, and a multi-task cross-domain fusion branch. Our proposed model is trained in a two-stage setting. In the first stage, we independently train two unimodal models: one for ER ($EM$) and one for PTA ($PT$). These single-domain models are designed to learn rich domain-specific temporal representations and produce initial task-specific predictions. In the second stage, the parameters of the unimodal single-task single-domain models are frozen, and their learned representations are used in a unimodal multi-task cross-domain fusion model, which performs attention-based interaction between the two domains. This separation allows each model to specialize in its respective task.

Figure 2. Pipeline of the unimodal multi-task cross-domain model for unimodal EPR. The flame and snowflake icons represent the use of trained and frozen models, respectively.

Stage 1: Unimodal Single-Task Single-Domain Models. The input to the unimodal single-task single-domain model is unimodal signals corresponding to different modalities (face, body, scene, audio, and text). Each modality undergoes its own pre-processing (e.g., face or body region detection, audio resampling, or text tokenization). This is followed by feature extraction using pre-trained encoders (including EmoNeXt, CLIP, Wav2Vec2.0, and Jina-v3). The model subsequently processes temporally aligned sequences of emotion (${\mathbf{X}}^{EM}\in {\mathbb{R}}^{Bs\times T\times D^E}$) and personality (${\mathbf{X}}^{PT}\in {\mathbb{R}}^{Bs\times T\times D^E}$) features, where $Bs$, T, and $D^E$ denote the batch size, sequence length, and encoder-derived feature dimension, respectively. These inputs are then passed to two separate unimodal task-specific (emotion and personality) models for temporal feature modeling.

To prevent early cross-task interference induced by corpus-specific modality dominance, we first train ER and PTA models independently. This allows each task to form stable representations under its own supervision regime, without enforcing a shared modality preference from the outset. Cross-task interaction is introduced only in a subsequent stage, enabling controlled information exchange rather than early parameter competition. Each model processes input features in a task-specific latent space, ensuring that biases are learned independently and without contamination from other tasks. The functional mapping of task-specific models operates as follows. Each input is first projected into a shared latent space ${\mathbf{H}}^{EM}$ and ${\mathbf{H}}^{PT}$ with dimensions $D^{EM}$ and $D^{PT}$, respectively:

$${\mathbf{H}}^{EM}=\mathrm{Dropout}\left(\mathrm{LayerNorm}\left({\mathbf{X}}^{EM}{\mathbf{W}}^{EM}\right)\right),{\mathbf{W}}^{EM}\in {\mathbb{R}}^{D^E\times D^{EM}}$$

(1)

$${\mathbf{H}}^{PT}=\mathrm{Dropout}\left(\mathrm{LayerNorm}\left({\mathbf{X}}^{PT}{\mathbf{W}}^{PT}\right)\right),{\mathbf{W}}^{PT}\in {\mathbb{R}}^{D^E\times D^{PT}}$$

(2)

where ${\mathbf{W}}^{EM}$ and ${\mathbf{W}}^{PT}$ denote weight matrices for emotion and personality feature projection, respectively. $\mathrm{LayerNorm}(·)$ stabilizes feature scales, while $\mathrm{Dropout}(·)$ regularizes via stochastic neuron masking. Each sequence is processed through a Temporal Encoder block, composed of N stacked temporal layers ${\mathrm{TEL}}_n(·)$, to capture temporal dependencies:

$${\mathbf{H}}_{n+1}^{EM}={\mathrm{TEL}}_n^{EM}\left({\mathbf{H}}_n^{EM}\right),{\mathbf{H}}_{n+1}^{PT}={\mathrm{TEL}}_n^{PT}\left({\mathbf{H}}_n^{PT}\right)$$

(3)

We use the Mamba and transformer models as Temporal Encoders. The final hidden states are pooled across the time dimension via temporal averaging:

$${\overline{\mathbf{H}}}^{EM}={\displaystyle \frac{1}{T}}\sum _{t=1}^T{\mathbf{H}}_N^{EM}[:,t,:],{\overline{\mathbf{H}}}^{PT}={\displaystyle \frac{1}{T}}\sum _{t=1}^T{\mathbf{H}}_N^{PT}[:,t,:]$$

(4)

This is followed by domain-specific predictors consisting of Fully Connected Layers (FCLs) with Gaussian Error Linear Unit (GELU) activation and normalization:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{y}}}^{EM}& ={\mathbf{W}}_2^{EM}·\mathrm{Dropout}\left(\varphi \left(\mathrm{LayerNorm}\left({\mathbf{W}}_1^{EM}{\overline{\mathbf{H}}}^{EM}\right)\right)\right)\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\widehat{\mathbf{y}}}^{PT}& =\sigma \left({\mathbf{W}}_2^{PT}·\mathrm{Dropout}\left(\varphi \left(\mathrm{LayerNorm}\left({\mathbf{W}}_1^{PT}{\overline{\mathbf{H}}}^{PT}\right)\right)\right)\right)\hfill \end{array}$$

(6)

where $\varphi (·)$ denotes the GELU activation and $\sigma (·)$ is the sigmoid function used for PTs bounded in $[0,1]$. The sizes of weight matrices are ${\mathbf{W}}_{\mathbf{1}}^{EM}\in {\mathbb{R}}^{D^E\times D^{EM}}$, ${\mathbf{W}}_{\mathbf{1}}^{PT}\in {\mathbb{R}}^{D^E\times D^{PT}}$, ${\mathbf{W}}_{\mathbf{2}}^{EM}\in {\mathbb{R}}^{D^{EM}\times C^{EM}}$, and ${\mathbf{W}}_{\mathbf{2}}^{PT}\in {\mathbb{R}}^{D^{PT}\times C^{PT}}$, where $C^{EM}$ and $C^{PT}$ denote the number of emotion and PT classes.

Stage 2: Unimodal Multi-Task Cross-Domain Fusion Model. While joint training can be beneficial for transferring knowledge between tasks, naive parameter sharing may lead to conflicts in cross-domain transferring. To enable controlled interaction, we freeze the single-task models from Stage 1 and introduce cross-attention only at the representation level, allowing mutual refinement without destabilizing task-specific expertise. The non-averaged single-domain hidden states from Stage 1 are passed through domain-specific projectors. Each projector operates similarly to Equations (1) and (2), transforming features of different dimensionalities (${\mathbf{H}}_N^{EM}\in {\mathbb{R}}^{Bs\times T\times D^{EM}}$ and ${\mathbf{H}}_N^{PT}\in {\mathbb{R}}^{Bs\times T\times D^{PT}}$) into a shared dimensional space (${\tilde{\mathbf{H}}}^{EM},{\tilde{\mathbf{H}}}^{PT}\in {\mathbb{R}}^{Bs\times T\times D^{CD}}$, where $D^{CD}$ denotes the hidden state dimension) to enable feature aggregation. The hidden states ${\tilde{\mathbf{H}}}^{EM}$ and ${\tilde{\mathbf{H}}}^{PT}$ are fed into two cross-attention blocks, each comprising N stacked transformer layers $T{L}_n(·)$. These layers operate as follows:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{H}}}_{n+1}^{EM}& ={\mathrm{TL}}_n^{EM⟶PT}\left({\tilde{\mathbf{H}}}_n^{EM},{\tilde{\mathbf{H}}}_n^{PT},{\tilde{\mathbf{H}}}_n^{PT}\right)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{H}}}_{n+1}^{PT}& ={\mathrm{TL}}_n^{PT⟶EM}\left({\tilde{\mathbf{H}}}_n^{PT},{\tilde{\mathbf{H}}}_n^{EM},{\tilde{\mathbf{H}}}_n^{EM}\right)\hfill \end{array}$$

(8)

where the three arguments of a transformer layer are query (Q), key (K), and value (V). This enables each modality to selectively attend to features of the other domain.

After N layers, the emotion and personality representations are concatenated

$${\mathbf{H}}_{\mathrm{fused}}=\mathrm{Concat}({\tilde{\mathbf{H}}}_N^{EM},{\tilde{\mathbf{H}}}_N^{PT})$$

(9)

and pooled using temporal averaging, as in Equation (4). The fused features, ${\overline{\mathbf{H}}}_{\mathrm{fused}}$, are passed through task-specific heads that operate similarly to Equations (5) and (6). To improve robustness and retain unimodal performance, we average Stage 1 and Stage 2 predictions:

$${\mathbf{y}}^{EM}={\displaystyle \frac{1}{2}}\left({\widehat{\mathbf{y}}}^{EM}+{\tilde{\mathbf{y}}}^{EM}\right),{\mathbf{y}}^{PT}={\displaystyle \frac{1}{2}}\left({\widehat{\mathbf{y}}}^{PT}+{\tilde{\mathbf{y}}}^{PT}\right)$$

(10)

where ${\widehat{\mathbf{y}}}^{EM}$ and ${\tilde{\mathbf{y}}}^{EM}$ are emotion predictions from Stage 1 and Stage 2, while ${\widehat{\mathbf{y}}}^{PT}$ and ${\tilde{\mathbf{y}}}^{PT}$ are PT predictions of the same stages.

3.5. Multimodal Multi-Task Cross-Domain Model

Stage 3 integrates unimodal emotion and personality representations and predictions into a unified multimodal prediction, addressing the core challenge of our framework: how to fuse heterogeneous, cross-domain signals while respecting task-specific modality biases. A naive fusion can inherit the corpus-dominant modality implied by the annotation protocol and propagate this bias into a shared space, suppressing complementary but psychologically relevant cues and ultimately degrading performance. To address this, we propose four purpose-built components that enable structured, task-aware knowledge transfer: (i) a Graph Attention Fusion layer that models intermodal relationships without assuming a fixed hierarchy; (ii) a Task-Specific Query-based Multi-Head Cross-Attention Fusion layer that dynamically selects relevant modalities for each task; (iii) predict projectors that inject unimodal predictions into the fused representation, enabling semantic alignment across tasks; and (iv) guide banks that store and refine task-specific prototypes to help stabilize learning. Together, these components enable bidirectional knowledge transfer between ER and PTA while preserving task-specific inductive biases, capturing the interdependence between emotions and PTs.

The pipeline of the proposed multimodal multi-task cross-domain model for MEPR is illustrated in Figure 3. This pipeline presents the complete model architecture, including all proposed components in a symmetric form for both tasks. In Section 4.6, the contribution of each component and the recommended configuration are empirically examined through ablation studies. This model aims to leverage heterogeneous unimodal features and intermediate predictions extracted from unimodal multi-task cross-domain models for five input modalities (face (F), body (B), scene (S), audio (A), and text (T)) to produce multi-task predictions for both ER ($EM$) and PTA ($PT$). Since each unimodal model produces both emotional and personality temporal features, their outputs are concatenated and averaged before being fed into the multimodal model.

Figure 3. Pipeline of the multimodal multi-task cross-domain model for MEPR.

The functional mapping of the multimodal multi-task cross-domain model operates as follows. Let ${\mathbf{X}}^m\in {\mathbb{R}}^{Bs\times D^m}$ denote the feature vector of modality $m\in \{F,B,S,A,L\}$. Each modality is first projected into a shared latent space of dimension D using a modality-specific projector:

$${\mathbf{H}}^m=\mathrm{AdapterFusion}\left(\mathrm{ModalityProjector}\left({\mathbf{X}}^m\right)\right)\in {\mathbb{R}}^{Bs\times D}$$

(11)

The projection block consists of an FCL, Rectified Linear Unit (ReLU) activation, dropout, and layer normalization. Residual adaptation is defined as follows:

$$\mathrm{AdapterFusion}\left(\mathbf{z}\right)=\mathrm{LayerNorm}(\mathbf{z}+{\mathbf{W}}_2\psi \left(\mathrm{Dropout}\left({\mathbf{W}}_1\mathbf{z}\right)\right))$$

(12)

where $\psi (·)$ denotes the ReLU activation and ${\mathbf{W}}_1\in {\mathbb{R}}^{D\times D/2}$ and ${\mathbf{W}}_2\in {\mathbb{R}}^{D/2\times D}$. All projected modality vectors $\left\{{\mathbf{H}}^m\right\}$ are stacked into a tensor $\mathbf{H}\in {\mathbb{R}}^{Bs\times M\times D}$, where M is the number of available modalities. To model intermodality relationships, we apply a Graph Attention layer:

$${\mathbf{H}}^{\prime }=\mathrm{GraphAttention}(\mathbf{H},\mathbf{A}),\mathbf{A}\in {\{0,1\}}^{M\times M}$$

(13)

where $\mathbf{A}$ is a fully connected adjacency matrix and the attention weights are computed using learnable edge-wise coefficients based on concatenated node embeddings, followed by softmax normalization. As a graph layer, we use the vanilla GNN [98]. The output of Graph Attention is then attended to via query vectors, ${\mathbf{Q}}^{EM},{\mathbf{Q}}^{PT}\in {\mathbb{R}}^{1\times D}$, for each task. Thus, emotion ${\mathbf{h}}^{EM}$ and personality ${\mathbf{h}}^{PT}$ features are extracted through Task-Specific Query-based Multi-Head Cross-Attention Fusion:

$$\begin{array}{cc}\hfill {\mathbf{h}}^{EM}& =\mathrm{MHA}({\mathbf{Q}}^{EM},{\mathbf{H}}^{\prime },{\mathbf{H}}^{\prime })\in {\mathbb{R}}^{Bs\times D}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathbf{h}}^{PT}& =\mathrm{MHA}({\mathbf{Q}}^{PT},{\mathbf{H}}^{\prime },{\mathbf{H}}^{\prime })\in {\mathbb{R}}^{Bs\times D}\hfill \end{array}$$

(15)

As an attention mechanism, we use Multi-Head Attention (MHA) [45].

In addition to learned representations, we incorporate auxiliary unimodal emotion and personality predictions ${\mathbf{y}}^{{EM}^m}$ and ${\mathbf{y}}^{{PT}^m}$ predicted from individual modalities:

$${\mathbf{y}}^{{EM}^{\mathrm{avg}}}={\displaystyle \frac{1}{M}}\sum _m{\mathbf{y}}^{{EM}^m},{\mathbf{y}}^{{PT}^{\mathrm{avg}}}={\displaystyle \frac{1}{M}}\sum _m{\mathbf{y}}^{{PT}^m}$$

(16)

The averaged predictions are first projected to the hidden space and added to the task-specific features ${\mathbf{h}}^{EM}$ and ${\mathbf{h}}^{PT}$:

$${\mathbf{h}}^{EM}={\mathbf{h}}^{EM}+{\mathbf{W}}_p^{EM}{\mathbf{y}}^{{EM}^{\mathrm{avg}}},{\mathbf{h}}^{PT}={\mathbf{h}}^{PT}+{\mathbf{W}}_p^{PT}{\mathbf{y}}^{{PT}^{\mathrm{avg}}}$$

(17)

where ${\mathbf{W}}_p^{EM}\in {\mathbb{R}}^{C^{EM}\times D}$ and ${\mathbf{W}}_p^{PT}\in {\mathbb{R}}^{C^{PT}\times D}$ denote the weight matrices employed in the task-specific predict projectors.

To enhance semantic alignment with emotion and personality classes, we introduce guide embeddings in guide banks:

$${\mathbf{G}}^{EM}\in {\mathbb{R}}^{C^{EM}\times D},{\mathbf{G}}^{PT}\in {\mathbb{R}}^{C^{PT}\times D}$$

(18)

The similarity between intermediate multimodal task-specific predictions, ${\mathbf{y}}_{\mathrm{inter}}^{EM}={\mathbf{W}}_{\mathrm{cls}}^{EM}{\mathbf{h}}^{EM}$ and ${\mathbf{y}}_{\mathrm{inter}}^{PT}=\sigma \left({\mathbf{W}}_{\mathrm{reg}}^{PT}{\mathbf{h}}^{PT}\right)$, and guide embeddings is computed using cosine similarity:

$$\begin{array}{cc}\hfill {\mathbf{s}}^{EM}& =\cos ({\mathbf{y}}_{\mathrm{inter}}^{EM},{\mathbf{G}}^{EM})\in {\mathbb{R}}^{Bs\times C^{EM}}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathbf{s}}^{PT}& =\cos ({\mathbf{y}}_{\mathrm{inter}}^{PT},{\mathbf{G}}^{PT})\in {\mathbb{R}}^{Bs\times C^{PT}}\hfill \end{array}$$

(20)

where ${\mathbf{W}}_{\mathrm{cls}}^{EM}\in {\mathbb{R}}^{D\times C^{EM}}$ and ${\mathbf{W}}_{\mathrm{reg}}^{PT}\in {\mathbb{R}}^{D\times C^{PT}}$ denote the weight matrices employed in the classifier and regressor, respectively. Guide banks implement a prototype-based guidance mechanism, whose behavior and performance impact may depend on the task formulation (classification vs. regression). Final predictions are obtained by averaging the similarity-based guidance with outputs from the classifier (cls) and regressor (reg):

$${\mathbf{y}}_{\mathrm{final}}^{EM}={\displaystyle \frac{1}{2}}({\mathbf{y}}_{\mathrm{inter}}^{EM}+{\mathbf{s}}^{EM}),{\mathbf{y}}_{\mathrm{final}}^{PT}={\displaystyle \frac{1}{2}}({\mathbf{y}}_{\mathrm{inter}}^{PT}+\sigma \left({\mathbf{s}}^{PT}\right))$$

(21)

Therefore, the proposed framework offers a principled and novel method for jointly modeling two psychological phenomena—human emotions and PTs—within a unified multimodal, multi-task, cross-domain setting.

3.6. Semi-Supervised Cross-Domain Learning

We train on two task-specific corpora, each devoted to a single prediction target: ER or PTA. Each corpus contains annotations only for its respective task, while labels for the other task are stored as NaN. At each training step, a mini-batch is assembled by randomly drawing samples from both corpora; if $n_1$ and $n_2$ denote the numbers of chosen examples, then $B=n_1+n_2$.

Our loss integrates a supervised component and a semi-supervised component, with task weights adjusted dynamically using an extended GradNorm method [47]. The supervised part ${\mathbf{L}}_s$ is evaluated only on labeled data; entries without ground truth are excluded:

$${\mathbf{L}}_s={\mathbf{w}}_s^{EM}{\mathbf{L}}_s^{EM}+{\mathbf{w}}_s^{PT}{\mathbf{L}}_s^{PT}$$

(22)

where ${\mathbf{L}}_s^{EM}$ is the cross-entropy loss for ER and ${\mathbf{L}}_s^{PT}$ is the Mean Absolute Error (MAE) loss for PTA. The coefficients ${\left\{{\mathbf{w}}_s^t\right\}}_{t\in \mathbb{T}}$ are not fixed but are updated throughout training to maintain balanced gradient magnitudes between tasks.

Unlabeled samples are exploited through pseudo-labeling with task-dependent confidence thresholds. For ER, a pseudo-label is accepted whenever the maximum softmax probability exceeds ${\tau }_{EM}$. For PTA, outputs are binarized at $0.5$ (interpreted as the polarity of the PTs) and kept only if they lie outside an uncertainty margin, either above ${\tau }^{PT}$ or below $1-{\tau }^{PT}$. The semi-supervised part of the loss ${\mathbf{L}}_{ss}$ is defined as

$${\mathbf{L}}_{ss}={\mathbf{w}}_{ss}^{EM}{\mathbf{L}}_{ss}^{EM}+{\mathbf{w}}_{ss}^{PT}{\mathbf{L}}_{ss}^{PT}$$

(23)

where ${\mathbf{L}}_{ss}^{EM}$ corresponds to the cross-entropy loss for ER and ${\mathbf{L}}_{ss}^{PT}$ to the binary cross-entropy loss for PTA. The full training loss is $\mathbf{L}={\mathbf{L}}_s+{\mathbf{L}}_{ss}$.

Both supervised and semi-supervised weights, ${\mathbf{w}}_s^t$ and ${\mathbf{w}}_{ss}^t$, are adapted during training by two separate GradNorm branches. Each branch minimizes its own balancing loss:

$${\mathbf{L}}_s^{GradNorm}=\sum _{t\in \mathbb{T}}\left|{\mathbf{G}}_s^t-{\overline{\mathbf{G}}}_s{\left({\mathbf{r}}_s^t\right)}^{{\alpha }_s}\right|$$

(24)

$${\mathbf{L}}_{ss}^{GradNorm}=\sum _{t\in \mathbb{T}}\left|{\mathbf{G}}_{ss}^t-{\overline{\mathbf{G}}}_{ss}{\left({\mathbf{r}}_{ss}^t\right)}^{{\alpha }_{\mathrm{ss}}}\right|$$

(25)

where ${\mathbf{G}}^t={∥{\nabla }_{{\theta }_{\mathrm{shared}}}\left({\mathbf{w}}^t{\mathbf{L}}^t\right)∥}_2$ is the gradient norm with respect to shared model parameters ${\theta }_{\mathrm{shared}}$; $\overline{\mathbf{G}}={\displaystyle \frac{1}{\left|T\right|}}{\sum }_{j\in T}{\mathbf{G}}^j$ is the average gradient norm across tasks in the branch; and ${\mathbf{r}}^t$ quantifies the relative inverse training speed:

$${\mathbf{r}}^t={\displaystyle \frac{{\mathbf{L}}^t/{\mathbf{L}}_{\left(0\right)}^t}{\frac{1}{\left|\mathbb{T}\right|}{\sum }_{j\in \mathbb{T}}{\mathbf{L}}^j/{\mathbf{L}}_{\left(0\right)}^j}}$$

(26)

where ${\mathbf{L}}_{\left(0\right)}^t$ is the initial loss value used as a baseline for each task. Parameters ${\alpha }_s$ and ${\alpha }_{ss}$ regulate how strongly faster-learning tasks are down-weighted.

Weights are updated by gradient descent on ${\mathbf{L}}^{\mathrm{GradNorm}}$ with branch-specific learning rates:

$${\mathbf{w}}_s^t\leftarrow max\left(w_{floor},{\mathbf{w}}_s^t-{\eta }_s^w·{\nabla }_{{\mathbf{w}}_s^t}{\mathbf{L}}_s^{\mathrm{GradNorm}}\right)$$

(27)

$${\mathbf{w}}_{ss}^t\leftarrow max\left(w_{floor},{\mathbf{w}}_{ss}^t-{\eta }_{ss}^w·{\nabla }_{{\mathbf{w}}_{ss}^t}{\mathbf{L}}_{ss}^{\mathrm{GradNorm}}\right)$$

(28)

where separate learning rates (${\eta }_w^{\mathrm{s}}$ and ${\eta }_w^{\mathrm{ss}}$) with $w_{\mathrm{floor}}$ preventing any task from being deactivated. After each update, weights are renormalized to fixed budgets to ensure their interpretation:

$${\mathbf{w}}_s^t\leftarrow {\mathbf{S}}_s{\displaystyle \frac{{\mathbf{w}}_s^t}{{\sum }_{j\in \mathbb{T}}{\mathbf{w}}_s^j}},{\mathbf{w}}_{ss}^t\leftarrow {\mathbf{S}}_{ss}{\displaystyle \frac{{\mathbf{w}}_{ss}^t}{{\sum }_{j\in \mathbb{T}}{\mathbf{w}}_{ss}^j}}$$

(29)

where ${\mathbf{S}}_s=2.0$ and ${\mathbf{S}}_{ss}=2.0\times \lambda$, initially emphasizing supervised losses.

Compared to the original GradNorm [47] formulation, our version: (i) treats supervised and SSL losses separately to account for different noise levels; (ii) delays the initialization of weights until valid losses are observed; and (iii) constrains the overall task budgets for interpreting their contributions. This hybrid loss supports SSL in merging single-task corpora by combining explicit supervision with pseudo-label guidance, mitigating label scarcity and improving cross-task generalization.

4. Experimental Research

The model training process follows a multi-stage setting: (i) the unimodal single-task single-domain models are trained first, (ii) followed by the unimodal multi-task cross-domain models, and (iii) the multimodal multi-task cross-domain model is trained.

Model configurations were selected via exhaustive grid search on the Development subset. The best-performing configurations of model architectures are presented in Table 3. Training parameters, such as a batch size of 32, a learning rate of $1\times {10}^{-4}$, the Adam optimizer, and a dropout rate of 0.15, were fixed for all models. The thresholds ${\left\{{\tau }^t\right\}}_{t\in \mathbb{T}}$ and other SSL coefficients (${\alpha }_s,{\alpha }_{ss},{\eta }_s^w,{\eta }_{ss}^w,w_{\mathrm{floor}},\lambda$) are tuned on the Development subset via grid search, but only for multimodal models. For unimodal models, the task weights are fixed to 1, which eliminates the need to search over thresholds and other coefficients. The search and best values used in the grid search are listed in Table 4. In contrast, the task weights ${\mathbf{w}}_s^t$ and ${\mathbf{w}}_{ss}^t$ are not tuned manually; they adapt automatically during training.

Table 3. Optimal configurations of model architectures. # means the number of hidden states, feature states, layers and transformer heads.

Model	Hyperparameter
	Temporal Layer	# Hidden States	# Feature States	# Layers	# Transformer Heads
Search values	{Mamba, Transformer}	{128, 256, 512, 1024}	{128, 256, 512, 1024}	{1, 2, 3, 4}	{2, 4, 8, 16}
Body-based PTA	Mamba	1024	256	1	–
Body-based ER	Mamba	256	128	2	–
Body-based MEPR	Transformer	1024	128	1	16
Face-based PTA	Mamba	256	512	1	–
Face-based ER	Mamba	1024	256	1	–
Face-based EPR	Transformer	512	256	3	2
Scene-based PTA	Mamba	512	512	7	–
Scene-based ER	Transformer	512	128	5	4
Scene-based EPR	Transformer	768	256	5	4
Audio-based PTA	Mamba	256	256	3	–
Audio-based ER	Mamba	256	256	3	–
Audio-based EPR	Transformer	256	256	1	4
Text-based PTA	Transformer	512	128	1	8
Text-based ER	Mamba	256	256	4	–
Text-based EPR	Transformer	256	256	2	4
MEPR	Transformer	256	256	1	8

Table 4. Grid search result of SSL hyperparameters.

Hyperparameter	Baseline Value	Search Values	Best Value
Pseudo-label threshold ${\tau }^{EM}$	0.5	{0.5, 0.6, 0.7, 0.8}	0.6
Pseudo-label threshold ${\tau }^{PT}$	0.45	{0.45, 0.5, 0.55, 0.6}	0.6
Gradient balancing controller ${\alpha }_s$	1.0	{1.0, 1.25, 1.50, 1.75}	1.0
Gradient balancing controller ${\alpha }_{ss}$	1.0	{1.0, 1.25, 1.50, 1.75}	1.5
Learning rate ${\eta }_s^w$	0.005	{0.005, 0.01, 0.025}	0.005
Learning rate ${\eta }_{ss}^w$	0.005	{0.005, 0.01, 0.025}	0.01
Budget coefficient $\lambda$	0.3	{0.3, 0.4, 0.5}	0.4
Min value of the task contribution coefficients $w_{\mathrm{floor}}$	${10}^{-3}$	{${10}^{-2}$, ${10}^{-3}$, ${10}^{-4}$}	${10}^{-3}$

4.1. Research Corpora

To date, several corpora are available for ER and PTA. However, large-scale annotated corpora that support both tasks remain unavailable, making it difficult to develop multi-task learning methods. To address this gap, we employ two task-specific corpora: MOSEI [22] (for ER) and FIv2 [12] (for PTA).

MOSEI is the largest multimodal corpus for sentiment analysis and ER. It contains more than 23,500 sentence–utterance videos from over 1000 YouTube speakers. Each video is annotated for six emotions (Anger, Disgust, Fear, Happiness, Sadness, and Surprise) and a sentiment (Positive, Neutral, and Negative). Unlike other emotion corpora, MOSEI employs multi-label annotations. This allows each video to be annotated with multiple emotions at varying intensities. We convert the six-dimensional multi-label vector into a seven-dimensional soft vector (including the Neutral state) that sums to one. The corpus covers a wide range of topics, including product reviews and personal monologues, and includes text, audio, and video data. Despite the multimodal nature of the corpus, annotations were made based on the text modality. The corpus has been split into three subsets: Train (16,274 examples), Development (1861 examples), and Test (4653 examples). Video lengths vary, with an average duration of 4.5 s per example. The corpus exhibits a significant class imbalance, with emotions such as Happiness overrepresented and others such as Fear and Surprise underrepresented (see Figure 4). To address this issue, class weights were calculated using the following formula:

$$w_c={\displaystyle \frac{K-k_c}{k_c}}$$

(30)

where $w_c$ is the weight for class c, K is the total number of examples, and $k_c$ is the number of samples in class c.

Figure 4. Emotion distributions in examples across subsets of MOSEI.

The FIv2 corpus comprises 10,000 short video clips (15 s each) from over 3000 YouTube vlogs, annotated for the Big Five PTs (Openness, Conscientiousness, Extraversion, Agreeableness, and Non-Neuroticism) and hiring suitability. Traits were scored on a scale of 0 to 1 via pairwise comparisons by Amazon Mechanical Turk workers. The corpus is split into Train (6000 examples), Development (2000 examples), and Test subsets (2000 examples). Demographic metadata (gender, ethnicity, and age) is also available. Figure 5 shows the frequency score distributions for PTs in the Train, Development, and Test subsets. The distributions of all traits are approximately normal, with most scores falling between 0.4 and 0.6.

Figure 5. Distributions of PT scores in examples across subsets of FIv2.

Both corpora are diverse and large-scale, containing monologues and in-the-wild video recordings. This enables effective multimodal multi-task cross-domain learning for ER and PTA.

4.2. Performance Measures

To evaluate the performance of the MEPR model, distinct measures are used for each sub-task. For the multi-label ER, we employ two measures: mean Macro F1-score (mMF1) and mean Weighted Accuracy (mWACC). These are calculated as follows:

$${\mathrm{F}1}_j={\displaystyle \frac{2·{\mathrm{TP}}_j}{2·{\mathrm{TP}}_j+{\mathrm{FP}}_j+{\mathrm{FN}}_j}}$$

(31)

$${\mathrm{M}\mathrm{F}1}_i={\displaystyle \frac{1}{C}}\sum _{j=1}^C{\mathrm{F}1}_j$$

(32)

$${\mathrm{WACC}}_i={\displaystyle \frac{{\mathrm{TP}}_i·({\mathrm{TN}}_i+{\mathrm{FN}}_i)/({\mathrm{TP}}_i+{\mathrm{FP}}_i)+{\mathrm{TN}}_i}{2·({\mathrm{TN}}_i+{\mathrm{FN}}_i)}}$$

(33)

where j denotes a binary emotion class i, indicating presence or absence; TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative predictions, respectively, for class j; and C is the number of binary classes.

Prior to computing mMF1 and mWACC, both predicted logits and ground truth labels are binarized. Predicted logits are first normalized using the softmax function. If the probability of the neutral class exceeds $1-1/7$, the entire prediction vector is set to a zero vector. Otherwise, all non-neutral class probabilities greater than $1/7$ are set to 1. Ground truth labels are similarly binarized: all non-zero values are mapped to 1, while the presence of a neutral label results in a zero vector. Finally, both 7-dimensional vectors (including the neutral class) are reduced to 6 dimensions by removing the neutral entry, ensuring that zero vectors unambiguously represent the neutral state. This pre-processing ensures measure compatibility and allows performance comparison with prior SOTA methods.

For the PTA, we employ the Concordance Correlation Coefficient (CCC), defined as follows:

$$\mathrm{CCC}={\displaystyle \frac{2{\sigma }_y{\sigma }_{\widehat{y}}}{{\sigma }_y^2+{\sigma }_{\widehat{y}}^2+{({\mu }_y-{\mu }_{\widehat{y}})}^2}},$$

(34)

where $\mu$ and $\sigma$ denote the mean and standard deviation of the ground truth ($y_j$) and predicted ($\widehat{y_j}$) values for all traits, respectively. As an additional measure, we use mean Accuracy (mACC), defined as 1 minus the MAE for each trait. Since there are five PTs, a final mACC value is computed as the mean over all five traits.

4.3. Video-Based Multi-Task Cross-Domain Model

The video-based experimental results are summarized in Table 5. The experimental results demonstrate the comparative performance of different feature extractors (EmoNext, ResNet-50, and CLIP) when paired with temporal models (Mamba and transformer) in both supervised single-domain and semi-supervised cross-domain settings. Since EmoNext was specifically fine-tuned for facial ER, it was only applied to facial features. In contrast, ResNet-50, pre-trained on general object recognition (including clothing and environmental elements), was used to extract features from body and scene regions. CLIP, a vision–language model trained on a broad multimodal corpus, was applied to all regions of interest (face, body, and scene).

Table 5. Video-based experimental results. LT and LD mean learning type and learning domain, respectively. SL and SSL mean supervised and semi-supervised learning, respectively. SD and CD mean single- and cross-domain learning, respectively. The best results are highlighted in bold.

Exp ID	Configuration				Face					Body					Scene
	Extractor	Model	LT	LD	MOSEI		FIv2			MOSEI		FIv2			MOSEI		FIv2
					mWACC	mMF1	mACC	CCC	Rank	mWACC	mMF1	mACC	CCC	Rank	mWACC	mMF1	mACC	CCC	Rank
V1	EmoNext	Mamba	SL	SD	57.73	53.02	90.18	53.40	6.25	–	–	–	–	–	–	–	–	–	–
V2	EmoNext	Mamba	SSL	CD	57.44	53.57	90.05	53.52	6.25	–	–	–	–	–	–	–	–	–	–
V3	EmoNext	Transformer	SL	SD	56.47	55.30	89.96	55.35	6.25	–	–	–	–	–	–	–	–	–	–
V4	EmoNext	Transformer	SSL	CD	56.91	54.84	89.91	52.69	7.25	–	–	–	–	–	–	–	–	–	–
V5	ResNet-50	Mamba	SL	SD	–	–	–	–	–	58.90	54.18	90.76	61.54	6.25	58.88	53.68	90.01	61.07	6.50
V6	ResNet-50	Mamba	SSL	CD	–	–	–	–	–	58.27	54.03	90.65	61.73	7.25	58.36	53.43	90.06	61.13	6.50
V7	ResNet-50	Transformer	SL	SD	–	–	–	–	–	59.28	55.03	90.67	62.11	5.25	58.85	54.98	89.95	61.10	6.00
V8	ResNet-50	Transformer	SSL	CD	–	–	–	–	–	58.67	54.54	90.60	60.77	7.25	58.89	54.49	88.23	60.08	6.75
V9	CLIP	Mamba	SL	SD	61.78	57.15	91.40	69.21	2.75	62.19	55.52	91.59	70.61	2.00	61.45	54.86	91.53	69.28	3.00
V10	CLIP	Mamba	SSL	CD	61.47	57.44	91.44	69.64	2.00	62.13	56.57	91.70	70.17	2.00	62.02	56.23	91.14	66.57	3.50
V11	CLIP	Transformer	SL	SD	61.47	56.14	91.60	69.41	2.75	62.06	56.75	91.47	69.27	3.25	62.12	56.27	91.41	69.51	1.75
V12	CLIP	Transformer	SSL	CD	61.90	57.00	91.61	69.01	2.25	62.08	56.94	91.51	68.52	2.75	62.47	57.25	91.38	68.39	2.00

The results reveal that CLIP-based models outperform both EmoNext and ResNet-50 across all evaluation measures, regardless of the region of interest. This suggests that the quality of pre-trained extractors has a more significant impact on performance than the choice of temporal architecture. Unlike EmoNext and ResNet-50, CLIP benefits from its joint image–text embedding space, allowing it to extract more discriminative and context-aware features.

The differences between temporal models (Mamba vs. transformer) are marginal, indicating that the choice of sequence modeling architecture plays a less critical role compared to feature extraction. Additionally, semi-supervised cross-domain learning generally leads to slight improvements over supervised single-domain learning, suggesting that leveraging multiple tasks enhances model robustness. The performance across modalities (face, body, and scene) is relatively balanced. However, the body modality is more informative for both tasks. This suggests that whole-body movements provide more stable cues for behavior than facial expressions, which are correlated with the movement of muscles that accompany articulation.

Based on these findings, we have selected the multi-task models obtained by Exp V10 with a rank of 2.00 (CLIP + Mamba + SSL + CD) for the face and body analysis and by Exp V12 with a rank of 2.00 (CLIP + transformer + SSL + CD) for the scene analysis. These models ensure optimal performance while maintaining cross-domain adaptability and supporting the development of a unified MEPR method.

4.4. Audio-Based Multi-Task Cross-Domain Model

The audio-based experimental results are summarized in Table 6. The experimental results demonstrate the comparative performance of different feature extractors (Wav2Vec2.0 and ExHuBERT) when paired with temporal models (Mamba and transformer) in both supervised single-domain and semi-supervised cross-domain settings. The results show clear differences in performance between the two feature extractors. Wav2Vec2.0-based models outperform ExHuBERT in all performance measures and experimental settings.

Table 6. Audio-based experimental results. LT and LD mean learning type and learning domain, respectively. SL and SSL mean supervised and semi-supervised learning, respectively. SD and CD mean single- and cross-domain learning, respectively. The best results are highlighted in bold.

Exp ID	Extractor	Model	LT	LD	MOSEI		FIv2		Rank
					mWACC	mMF1	mACC	CCC
A1	Wav2Vec2.0	Mamba	SL	SD	61.51	60.59	90.25	52.02	1.25
A2	Wav2Vec2.0	Mamba	SSL	CD	61.72	59.42	90.11	51.06	2.00
A3	Wav2Vec2.0	Transformer	SL	SD	60.23	58.70	90.01	49.61	3.25
A4	Wav2Vec2.0	Transformer	SSL	CD	59.61	58.27	90.23	48.15	3.50
A5	ExHubert	Mamba	SL	SD	58.33	55.77	89.15	46.27	6.75
A6	ExHubert	Mamba	SSL	CD	58.42	56.03	89.60	44.83	5.75
A7	ExHubert	Transformer	SL	SD	58.89	55.82	88.44	45.02	6.50
A8	ExHubert	Transformer	SSL	CD	57.49	55.85	89.28	43.52	7.00

The superior performance of Wav2Vec2.0 embeddings, despite covering fewer domains (1 vs. 37), suggests that a larger training corpus (250 h vs. 150 h) may be more crucial for building generalizable audio representations than broader domain coverage. This indicates that data scale outweighs domain diversity for paralinguistic tasks when the target domain aligns with the pre-training data. As for temporal models, the results indicate that Mamba-based models slightly outperform transformers for audio processing, although the differences between temporal models are less significant than those between feature extractors. Based on these findings, we have selected the multi-task model obtained by Exp A2 with a rank of 2.00 (Wav2Vec2.0 + Mamba + SSL + CD) for the audio analysis and the development of a unified MEPR method.

A comparison between the audio model (Exp A2) and the video models (Exp V10 and Exp V12) shows that in ER, the audio model outperforms the video models in the mMF1 measures. The performance difference is more than 2% in favor of audio (59.42 vs. 57.44/56.57/57.25), whereas in PTA, the audio model is inferior to the video models by more than 1% in mACC measures (90.25 vs. 91.44/91.70/91.38) and more than 17% in CCC measures (51.06 vs. 69.64/70.17/68.39). This stark gap reflects the annotation bias of FIv2, where personality labels are derived from visual behavior, making acoustic features alone insufficient for stable PTA.

4.5. Text-Based Multi-Task Cross-Domain Model

The text-based experimental results are summarized in Table 7. The experimental results demonstrate the comparative performance of different feature extractors (Jina-v3 and BGE-en) paired with temporal models (Mamba and transformer) in both supervised single-domain and semi-supervised cross-domain settings. The experimental results demonstrate that the overall performance of BGE-en outperforms that of Jina-v3. This advantage is due to the fine-tuning of BGE-en for English, which is particularly effective for English-language research corpora. However, this language-specific optimization reduces the generalizability of the model to other languages.

Table 7. Text-based experimental results. LT and LD mean learning type and learning domain, respectively. SL and SSL mean supervised and semi-supervised learning, respectively. SD and CD mean single- and cross-domain learning, respectively. The best results are highlighted in bold.

Exp ID	Extractor	Model	LT	LD	MOSEI		FIv2		Rank
					mWACC	mMF1	mACC	CCC
L1	Jina-v3	Mamba	SL	SD	64.78	59.87	88.86	27.94	3.50
L2	Jina-v3	Mamba	SSL	CD	64.18	58.45	88.68	29.19	6.25
L3	Jina-v3	Transformer	SL	SD	63.92	59.00	88.83	30.54	4.75
L4	Jina-v3	Transformer	SSL	CD	62.95	58.44	88.79	29.94	6.00
L5	BGE-en	Mamba	SL	SD	65.02	59.78	88.86	27.09	3.25
L6	BGE-en	Mamba	SSL	CD	64.81	59.29	88.74	28.04	4.50
L7	BGE-en	Transformer	SL	SD	64.69	59.47	88.90	26.83	4.25
L8	BGE-en	Transformer	SSL	CD	64.80	59.22	88.87	29.94	3.00

Regarding temporal model comparisons, Mamba demonstrates significantly superior performance over transformer for ER. In contrast, both models show comparable effectiveness for PTA. This performance discrepancy stems from fundamental differences in corpus structure: FIv2 contains fixed-duration 15-second utterances, while MOSEI comprises utterances of varying length. Mamba’s input-dependent state compression adapts more efficiently to variable-length sequences, whereas transformer’s fixed attention window offers no advantage on uniformly short clips. Based on these findings, we have selected the multi-task model obtained by Exp L8 with a rank of 3.00 (BGE-en + Transformer + SSL + CD) for the text analysis and the development of a unified MEPR method.

Among all unimodal models, the text model (Exp L8) outperforms the audio (Exp A2) and video (Exp V10 and Exp V12) models according to mWACC measures. The performance difference is more than 3% (64.80 vs. 61.72 vs. 61.47/62.13/62.47). In contrast in PTA, the video models outperform other models in the mACC measures by more than 1% (91.44/91.70/91.38 vs. 90.25 vs. 88.87) and in the CCC measures by more than 17% (69.64/70.17/68.39 vs. 51.06 vs. 29.94). These results show that modality performance depends on corpus annotation protocols. For MOSEI, text modality is the most informative because annotations were derived from text. In contrast, for PTA, FIv2 was annotated based on the video modality, so visual features are most discriminative for that task. This modality-specific annotation bias inherently limits the performance improvement in a cross-domain learning setting, as models need to reconcile fundamentally different signal-to-domain relationships across modalities.

4.6. Multimodal Mult-Task Cross-Domain Model

Table 8 summarizes the results of the multimodal experiments. The full model configuration (Exp M1) improves all performance measures, except for mWACC, which drops slightly by 0.90% (63.90 vs. 64.80) compared to the high-performing unimodal text model (Exp L8). mMF1 increases by 4.14% (63.36 vs. 59.22) compared to the high-performing audio-only model (Exp A2). Despite the small mWACC drop, the significant improvement in mMF1 suggests better inter-class discrimination with reduced misclassification bias. This trade-off reflects the importance of text in MOSEI: while using only text maximizes mWACC (Exp L8), using multimodal fusion reduces the overreliance on linguistic cues and leads to more balanced performance across classes (higher mMF1), which is a sign of increased robustness. Further, mACC increases by 0.25% (91.95 vs. 91.70) and CCC by 1.69% (71.86 vs. 70.17) compared to the face-based video model (Exp V10). These improvements underline the value of cross-modal interactions in boosting model performance.

Table 8. Multimodal-based experimental results with ablation study. LT and LD mean learning type and learning domain, respectively. SL and SSL mean supervised and semi-supervised learning, respectively. SD and CD mean single- and cross-domain learning, respectively. The bright gray-highlighted and dull gray-highlighted performance measures represent minor and major informative modalities, respectively. The best results are highlighted in bold.

Exp ID	Model	LT	LD	MOSEI		FIv2		Rank
				mWACC	mMF1	mACC	CCC
V10	Face-based CLIP + Mamba	SSL	CD	61.47	57.44	91.44	69.64	35.50
V10	Body-based CLIP + Mamba	SSL	CD	62.13	56.57	91.70	70.17	32.25
V12	Scene-based CLIP + Transformer	SSL	CD	62.47	57.25	91.38	68.39	34.75
A2	Audio-based Wav2Vec2.0 + Mamba	SSL	CD	61.72	59.42	90.11	51.06	36.50
L8	Text-based BGE-en + Transformer	SSL	CD	64.80	59.22	88.87	29.94	30.00
M1	Full model configuration	SSL	CD	63.90	63.36	91.95	71.86	9.00
M2	w/o Graph Attention Fusion	SSL	CD	63.68	63.23	91.83	72.18	11.75
M3	w/o Cross-Attention Fusion	SSL	CD	63.86	63.33	91.86	71.69	11.75
M4	w/o Emotion Predict Projector	SSL	CD	64.13	63.07	91.83	71.38	13.00
M5	w/o Personality Predict Projector	SSL	CD	63.58	63.25	91.83	71.59	13.75
M6	w/o Emotion Guide Bank	SSL	CD	63.92	62.95	91.94	71.61	11.75
M7	w/o Personality Guide Bank	SSL	CD	64.26	63.54	92.00	73.39	4.75
M8	Exp M7 w/o A	SSL	CD	62.96	62.01	91.90	72.20	15.25
M9	Exp M7 w/o L	SSL	CD	62.34	61.67	92.04	72.95	15.75
M10	Exp M7 w/o S	SSL	CD	63.79	62.78	91.91	72.12	11.75
M11	Exp M7 w/o F	SSL	CD	64.04	63.10	91.90	71.93	10.00
M12	Exp M7 w/o B	SSL	CD	64.48	63.51	91.95	71.90	7.00
M13	Exp M7 w/o A, L	SSL	CD	59.74	58.68	91.79	71.03	28.25
M14	Exp M7 w/o A, S	SSL	CD	62.93	61.87	91.79	70.78	21.75
M15	Exp M7 w/o A, F	SSL	CD	62.93	62.27	91.75	70.71	23.00
M16	Exp M7 w/o A, B	SSL	CD	63.06	62.31	91.79	70.75	20.25
M17	Exp M7 w/o L, S	SSL	CD	61.67	61.12	91.97	72.23	19.00
M18	Exp M7 w/o L, F	SSL	CD	62.85	61.54	91.85	71.44	20.75
M19	Exp M7 w/o L, B	SSL	CD	62.70	61.72	91.94	71.82	18.00
M20	Exp M7 w/o S, F	SSL	CD	63.26	62.43	91.74	70.21	22.00
M21	Exp M7 w/o S, B	SSL	CD	63.56	62.60	91.57	68.09	25.00
M22	Exp M7 w/o F, B	SSL	CD	63.97	63.00	91.64	68.95	21.25
M23	Exp M7 w/o A, L, S	SSL	CD	59.04	57.98	91.79	70.90	29.75
M24	Exp M7 w/o A, L, F	SSL	CD	59.28	58.10	91.70	70.56	32.75
M25	Exp M7 w/o A, L, B	SSL	CD	59.66	58.73	91.82	70.46	29.75
M26	Exp M7 w/o A, S, F	SSL	CD	63.17	61.90	91.65	68.59	26.25
M27	Exp M7 w/o A, S, B	SSL	CD	62.08	61.22	91.44	66.65	33.50
M28	Exp M7 w/o A, F, B	SSL	CD	63.38	62.29	91.51	68.09	26.25
M29	Exp M7 w/o L, S, F	SSL	CD	62.19	61.35	91.74	69.98	28.25
M30	Exp M7 w/o L, S, B	SSL	CD	61.98	60.80	91.61	67.90	33.75
M31	Exp M7 w/o L, F, B	SSL	CD	61.51	60.87	91.77	69.52	30.50
M32	Exp M7 w/o S, F, B	SSL	CD	63.19	61.87	89.97	52.51	30.00
M33	Exp M7 w/o Stage 2	SSL	CD	62.46	61.09	91.77	70.88	25.75
Multimodal Single-Task Models
M34	Exp M7	SSL	CD	70.26	70.55	–	–	1.00
M35	Exp M7 w/o SSL	SL	SD	68.82	68.71	–	–	2.00
M36	Exp M7 w/o SSL and personality features	SL	SD	67.77	67.88	–	–	3.00
M37	Exp M7	SSL	CD	–	–	92.88	78.05	1.00
M38	Exp M7 w/o SSL	SL	SD	–	–	92.59	76.95	2.00
M39	Exp M7 w/o SSL and emotional features	SL	SD	–	–	92.41	76.13	3.00

The ablation study (M1–M7) shows that removing the emotion or personality predict projectors, which are responsible for generating task-specific queries in the cross-attention fusion layers, significantly degrades performance (rank 13.00 and 13.75 compared to 9.00). This confirms their critical role in aligning unimodal predictions with the multimodal fusion. In contrast, removing auxiliary components, such as the Graph Attention Fusion, the Cross-Attention Fusion, or the emotion guide bank, causes a smaller decrease in rank (11.75 compared to 9.00), indicating a secondary contribution. Notably, removing the personality guide bank improves the rank from 9.00 to 4.75, suggesting that prototype-based guidance is sensitive to the task formulation (classification vs. regression): PTs are optimized as continuous regression targets, which yields less separable and less stable prototype structure and increases the risk of overfitting. In contrast, discrete emotion classes provide clearer separation and more stable prototypes. Accordingly, in the current MEPR setting we treat the personality guide bank as optional under continuous PTA and exclude it from subsequent ablations, while retaining prototype guidance for ER.

The ablation study (M8–M9) excluding specific modalities shows that their informativeness varies depending on the task. Excluding the body modality (Exp M12) slightly reduces ER performance, and removing the text modality (Exp M9) similarly affects PTA. However, removing just one modality causes a smaller performance drop than excluding two (M13–M22) or more (M23–M32) modalities. For ER, excluding one to three modalities, starting with the body and expanding to audio, body, and face (M12, M22, and M28), reduces mWACC by 0.88% (63.38 vs. 63.97 vs. 64.48 vs. 64.26) and mMF1 by 1.25% (62.29 vs. 63.00 vs. 63.51 vs. 63.54). For PTA, excluding one to three modalities, starting with text and extending to audio, text, and scene (M9, M21, and M23), reduces mACC by 0.21% (91.79 vs. 91.97 vs. 92.04 vs. 92.00) and CCC by 2.47% (70.90 vs. 72.23 vs. 72.95 vs. 73.39). Thus, the less informative modalities for ER on MOSEI are audio, face, and scene, whereas for PTA on FIv2, the less informative modalities are audio, text, and scene.

Among the most informative modalities, linguistic features remain essential for ER, while facial features contribute most significantly to PTA. When excluding modalities (M8–M23), from one (text) to three (audio, text, and scene) (M9, M13, and M23), in ER, performance measures decline for mWACC by 5.22% (59.04 vs. 59.74 vs. 62.34 vs. 64.26) and mMF1 by 5.56% (57.98 vs. 58.68 vs. 61.67 vs. 63.54). For PTA, exclusion, from one (face) to three (face, body, and scene) (M11, M21, and M32), results in a decrease in mACC by 2.03% (89.97 vs. 91.57 vs. 91.90 vs. 92.00) and CCC by 20.88% (52.51 vs. 68.09 vs. 71.93 vs. 73.39). Notably, modalities that are uninformative for PTA prove highly informative for ER. This asymmetry confirms that the annotation protocol, rather than the modality itself, determines informativeness. For example, text is useful for ER only because MOSEI labels are derived from transcripts, and not because language inherently carries more emotion. This highlights the complexity of joint SSL, where the model must manage conflicting modality relevance while ensuring domain generalization.

In addition, we evaluate the impact of excluding Stage 2 (Exp M33) from the training pipeline on the performance of the proposed framework. The results show that without this intermediate training stage of unimodal multi-task cross-domain models, the performance measures for both corpora decrease. This result highlights the importance of a three-stage learning strategy to model intra- and inter-domain interactions across modalities, from unimodal to multimodal data.

We conduct an additional ablation study to investigate the impact of the order in which modalities are fused (e.g., video–audio–text vs. text–audio–video, etc.) on the performance of the M7 model. The results, summarized in Table 9, show that although the default fusion order (video–audio–text) achieves the best average ranking across measures, all six tested orders yielded highly comparable performance. This indicates that the order of fusion has a minimal impact on final performance, likely due to our architecture, which incorporates novel layers such as Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks. These components effectively normalize interactions between modalities and enable consistent convergence, regardless of input sequence order. Therefore, the proposed improvements are robust to order changes, as they dynamically learn task-relevant modality relationships instead of relying on a static hierarchical aggregation method.

Table 9. Modality order experimental results obtained by the M7 model. The best results are highlighted in bold.

Exp ID	Modality Order	MOSEI		FIv2		Rank
		mWACC	mF1	mACC	CCC
1	Video-Audio-Text	64.26	63.54	92.00	73.39	2.25
2	Video–Text–Audio	64.18	63.40	92.02	73.37	3.00
3	Audio–Video–Text	63.98	63.25	92.00	73.03	5.25
4	Audio–Text–Video	64.18	63.35	92.02	73.42	2.75
5	Text–Video–Audio	63.69	63.13	92.03	73.27	4.25
6	Text–Audio–Video	64.19	63.45	92.03	73.15	2.50

To highlight the contribution of our novel components in proposed model, we also conduct an extensive ablation study by comparing four GNNs. Vanilla GNN [98] computes node representations via masked self-attention over 1-hop neighbours, enabling adaptive weighting of neighbour contributions without requiring pre-defined graph structure priors. Non-Convolutional GNN (NCGNN) [99] replaces conventional message passing with random walks guided by a unified memory and uses Recurrent Neural Networks (RNNs) to aggregate topological and semantic signals along node-anchored walks, mitigating over-smoothing without sparse convolutional kernels. Unitary Convolution GNN (UCGNN) [100] enforces unit-modulus transformations to stabilize deep graph models, avoiding over-smoothing and improving training stability with increasing depth. Hyperbolic GNN (HGNN) [101] recasts message passing as a system of hyperbolic partial differential equations, evolving node states in a solution space spanned by Laplacian eigenvectors to achieve spectral–spatio-temporal interpretability. We also compare several cross-attention fusion layers. MHA [45] linearly projects queries, keys, and values into multiple subspaces, computes scaled dot-product attention in parallel across heads, and concatenates the outputs to capture diverse contextual dependencies. Bidirectional Cross-Attention (BiCA) [102] allows input tokens and latent variables to attend to each other simultaneously, leveraging emergent attention symmetry for balanced bidirectional information flow. Cross-attention Message-Passing Transformer (CrossMPT) [103] uses two cross-attention blocks to iteratively update query and key–value representations, improving multimodal alignment through iterative refinement. Forgetting Attention (FA) [104] introduces a forget gate within the softmax attention mechanism to down-weight unnormalized attention scores in a data-dependent manner, mimicking cognitive filtering of irrelevant signals. As reported in Table 10, the vanilla GNN and MHA outperform newer alternatives in our setting, confirming that the strength of our method lies in the integration strategy, not in the novelty of individual layers. Thus, while individual components of our model include trivial layers, their purposeful combination, guided by psychological principles and tailored to the unique challenges of cross-task, cross-domain, semi-supervised affective and personality computing, constitutes a meaningful methodological advance.

Table 10. Ablation study results of different GNNs and attention mechanism obtained by the M7 model. The best results are highlighted in bold.

Exp ID	Model	MOSEI		FIv2		Rank
		mWACC	mF1	mACC	CCC
1	Vanilla GNN + MHA	64.26	63.54	92.00	73.39	3.75
2	Vanilla GNN + BiCA	63.63	63.16	91.93	73.43	5.75
3	Vanilla GNN + CrossMPT	63.86	63.55	91.88	73.96	4.00
4	Vanilla GNN + FA	63.44	62.97	91.73	73.12	10.00
5	NCGNN + MHA	63.19	62.70	91.87	72.57	11.00
6	NCGNN + BiCA	63.56	62.84	91.98	73.06	8.00
7	NCGNN + CrossMPT	63.83	63.43	91.81	72.55	8.25
8	NCGNN + FA	63.74	63.14	91.78	71.54	10.75
9	UCGNN + MHA	64.50	63.68	91.81	72.09	7.25
10	UCGNN + BiCA	64.69	63.96	91.61	72.76	6.50
11	UCGNN + CrossMPT	63.95	63.73	91.72	73.51	6.00
12	UCGNN + FA	64.48	63.75	91.60	73.10	6.75
13	HGNN + MHA	63.39	62.58	91.77	72.44	13.25
14	HGNN + BiCA	63.19	62.86	91.87	72.30	11.50
15	HGNN + CrossMPT	62.83	62.61	91.88	73.03	10.75
16	HGNN + FA	63.41	63.01	91.75	72.46	11.50

We also assess the performance of the multimodal single-task models (see Table 8). Six different model configurations are tested: (i) the ER model is trained in a semi-supervised cross-domain setting using unimodal emotional and personality features (Exp M34); (ii) the ER model is trained in a supervised single-domain using unimodal emotional and personality features (Exp M35); (iii) the ER model is trained in a supervised single-domain using unimodal emotional features only (Exp M36); (iv) the PTA model is trained in a semi-supervised cross-domain setting using unimodal emotional and personality features (Exp M37); (v) the PTA model is trained in a supervised single-domain setting using unimodal emotional and personality features (Exp M38); and (vi) the PTA model is trained in a supervised single-domain setting using unimodal personality features only (Exp M39). The results show that the models trained in a semi-supervised cross-domain setting (M34 and M37) outperform the models trained in a supervised single-domain setting (M35, M36, M38 and M39) on both tasks. These gains highlight the benefit of using unlabeled data from a different domain through SSL, which enhances the generalization and performance of the model. Additionally, incorporating task-specific auxiliary features, such as PTs for ER or emotional cues for PTA, leads to further performance improvements (Exp M36 vs. Exp M35 on MOSEI; Exp M39 vs. Exp M38 on FIv2). Thus, semi-supervised cross-domain learning improves performance by exposing the model to a diverse range of speakers and recording conditions that are absent from the target domain. Task-specific auxiliary features further enhance representations by modeling additional psychological signals.

Comparison of the performance between single-task and multi-task models highlights that the latter leads to a significant degradation in performance measures; the rank for both tasks decreased from 1.00 to 4.75. However, both learning settings have their problems. Single-task learning introduces overfitting to a single domain, where the model relies on domain-specific modalities that fail to generalize. Multi-task learning, while aiming to balance modalities across corpora, introduces noise and distributional conflicts, impairing domain-invariant feature learning. Thus, the performance gap is not a flaw of multi-task learning, but evidence of irreconcilable annotation biases. The model cannot simultaneously optimize for text-grounded ER and vision-grounded PTA without explicit decoupling, which our architecture partially achieves via task-specific queries, predict projectors, guide banks and SSL with modified the GradNorm method.

5. Discussion

This study presents a semi-supervised multi-task cross-domain learning framework for MEPR, aimed at extracting knowledge about the relationship between transient emotions and stable personality traits, while addressing the scarcity of corpora providing full annotations for both tasks. The model integrates six modalities (text, audio, face, body, and scene) to extract shared representations across different domains using partially labeled data. Furthermore, its multi-task design supports the development of more personalized and context-aware human–computer interaction (HCI) systems grounded in cross-task psychological principles.

Results indicate that modality effectiveness is largely shaped by corpus-specific annotation protocols, which can diverge from how psychological signals are naturally expressed. In MOSEI, linguistic features dominate ER, consistent with text-derived annotations, whereas in FIv2 visual cues are more predictive for PTA, consistent with video-derived annotations. These annotation-induced modality biases create a fundamental tension for MEPR: when ER and PTA are optimized jointly under shared representations, each objective implicitly enforces a different modality preference, yielding partially incompatible learning signals and, consequently, multi-task performance degradation. From this perspective, the observed underperformance of joint training reflects a structural consequence of heterogeneous supervision regimes rather than an inherent limitation of multimodal integration across tasks. At the same time, our results show that semi-supervised cross-domain learning improves both tasks in the single-task setting, suggesting that additional unlabeled observations enhance generalization even under partial labeling. These findings also highlight that effective cross-task knowledge transfer in MEPR requires not only a semi-supervised cross-domain learning strategy, but also architectural mechanisms that regulate how heterogeneous and task-specific features interact. The revised architecture of the MEPR model integrates four novel components: (i) Graph Attention Fusion to model intermodal relationships, (ii) task-specific query-based cross-attention to select relevant modalities, (iii) predict projectors to inject unimodal predictions into the fused representation, and (iv) guide banks to refine task-specific prototypes. Together, these components enable structured, task-aware knowledge transfer. However, their effectiveness varies across task formulations. In particular, guide banks are naturally better suited for the classification task in ER, whereas their impact is more sensitive to the underlying prototype structure in the regression task of PTA. For PTA, alternative trait score representations, such as ordinal bins or categorical levels, may be more appropriate for prototype-based guidance; however, they are not considered in this study, because they are beyond its scope.

Nonetheless, the method has several limitations. Performance is dependent on the annotation strategy of each corpus, which limits full task transferability. Imbalanced corpora and language-specific pre-training also constrain generalization. Furthermore, the current architecture requires a significant amount of computational resources due to the use of graph-based fusion and cross-attention mechanisms. This can become even more challenging when scaling the model to longer sequences or adding additional modalities. While our model achieves impressive results on two large-scale English corpora, its scalability to other languages, domains, or low-resource scenarios has not been tested. Extending this framework to new tasks or modalities will further increase the memory and training time requirements, posing practical challenges for resource-constrained settings.

We report the computational cost of our trained model. Unimodal single-task single-domain and unimodal multi-task cross-domain models of Stages 1 and 2 contain 65 M parameters and require 583 Mb storage memory. Their computational cost is high due to: (i) the use of Mamba and transformer architectures, which support self-attention for a single task, and (ii) the transformer architecture, which facilitates cross-attention across different domains. The multimodal multi-task cross-domain model of Stage 3 contains 2 M parameters and requires 8Mb storage memory. Despite processing multimodal data, its computational cost is reduced through graph-level modality fusion and cross-attention integration. The proposed framework supports real-time inference. The real-time factor for processing 1-second multimodal data is 0.28 s on a Graphics Processing Unit (GPU) (NVIDIA GeForce RTX 4090) and 0.64 s on a Central Processing Unit (CPU) (Intel Core i9). The full source code of the proposed framework is publicly available (https://github.com/LEYA-HSE/SSL-MEPR, accessed on 3 December 2025). All models were trained on an NVIDIA A100 Tensor Core GPU with 80 GB of memory [105].

5.1. Adaptive Loss Coefficient Visualization

Figure 6 shows the adaptation of the task contribution coefficients across training epochs for the best-performing SSL hyperparameters found for the Exp M7 model. These coefficients are dynamically adjusted at each epoch using the double-branch GradNorm method. Their trajectories are plotted in gray; the blue line shows the mMF1-score for ER; and the orange line shows the CCC score for PTA. Both measures increase monotonically throughout training, indicating stable and consistent learning. The coefficients for PTA remain moderate to high in both supervised learning and SSL, indicating that pseudo-labels play a significant role in optimizing this task. In contrast, the coefficients for ERs are lower than those for PTAs. This suggests that the model learns the ER task more quickly and adjusts its relative importance in order to balance the two tasks. Dynamic weight adaptation shows that a balance between performance measures across both tasks is achieved when the task contribution coefficients are set to ${\mathbf{w}}_s^{EM}=0.14$, ${\mathbf{w}}_{ss}^{EM}=0.22$, ${\mathbf{w}}_s^{PT}=1.86$, and ${\mathbf{w}}_{ss}^{PT}=0.58$. Thus, the modified GradNorm method enables dynamic loss contribution weighting not only across tasks but also between learning setting, specifically, supervised learning and SSL.

Figure 6. Visualization of task contribution coefficients during training of the M7 model with the best SSL hyperparameters.

5.2. Model’s Attention Visualization

To visualize the attention of the Exp M7 model when making decisions based on heterogeneous and task-specific features, we use GradCAM [106]. The number of dimensions in the input unimodal features varies, ranging from 256 to 1024 (see Table 3), which is too large for visualization. Therefore, we reduce the number of dimensions of all features to 16, consisting of 8 compressed emotional and 8 personality features. The visualization result is shown in Figure 7. For ER (first column), the model primarily focuses on emotional features in audio and text. This suggests a cross-domain bias, as emotional annotations in MOSEI use textual data. The model relies heavily on linguistic and acoustic cues to detect emotions. For PTA (second–sixth columns), attention shifts toward visual modalities. This is due to the nature of FIv2, where PTs is inferred from visual behavior. The visual nature of the data leads the model to focus on both emotional and personality features.

Figure 7. Visualization of model’s attention across heterogeneous and task-specific features. Less significant features are indicated in blue, and more informative ones are indicated in red.

Across the presented samples, the informativeness of emotional and personality features varies considerably. This suggests that the expression of psychological states varies between individuals: some speakers may express these states through voice or language, while others convey them visually through gestures and facial movements. Moreover, cross-informativeness between features from different tasks can be observed. For example, when predicting the Agreeableness trait (top and bottom subfigures), the model assigns higher importance to emotional features from the face, scene, and audio modalities than to their personality counterparts. In contrast, when predicting the Anger emotion (middle subfigure), personality facial features are more dominant than emotional acoustic and linguistic features. These findings suggest that cross-task knowledge, such as using emotional cues to inform personality prediction and vice versa, can improve model robustness. They also highlight the need for training on more complex psychological signal representations in order to capture more sophisticated, task-independent patterns.

These findings suggest ways to mitigate annotation-induced modality bias. Our model already addresses this implicitly: task-aware fusion (via task-specific query-based cross-attention and guide banks) dynamically reweights modalities by task rather than corpora bias, while predict projectors align emotional and personality features in a shared space, reducing reliance on modality-specific artifacts. For future work, we will consider contrastive learning to disentangle task-agnostic psychological signals from modality bias, alongside cross-corpus pseudo-labeling and modality dropout as regularization strategies.

Overall, the figure shows that while a modality-specific bias remains, the inclusion of complementary feature types (using personality features for ER and emotional features for PTA) and the heterogeneity of modalities provide the model with additional contextual information, which enhances the robustness and informativeness of the final predictions despite residual cross-domain influence.

Although Figure 7 provides qualitative insight into task–modality alignment, the quantitative impact of it is evident in Table 8: the jointly trained multi-task model (Exp M7, row M7) underperforms its single-task counterparts (M34 for ER, M37 for PTA) by 6.0 mWACC/7.0 mMF1 and 0.9 mACC/4.7 mCCC, respectively. This degradation arises because modalities exhibit divergent informativeness across tasks and corpora; e.g., the text modality is dominant for ER on MOSEI but weakly correlated with PTA on FIv2, while facial features are essential for PTA but less critical for ER. The ablation study quantifies this effect: removing the text modality in Exp M9 causes a significant drop of 1.9 mMF1 in ER and a minor 0.4 CCC decline in PTA, whereas removing the face modality in Exp M11 leads to only a 0.4 mMF1 drop in ER but a substantial 1.5 CCC drop in PTA. Thus, the observed degradation is not a flaw of multi-task learning in itself, but rather a consequence of domain-induced modality bias. Our architecture mitigates this bias through the use of task-specific queries, guide banks, and semi-supervised cross-domain alignment.

5.3. Correlation Between Emotions and Personality Traits

Since the proposed framework solves two tasks, we evaluate the correlation between two psychophysical phenomena (Figure 8). Visualization is provided for Test subsets from both corpora. Each PT score is divided into two levels: low (scores range from 0 to 0.55) and high (exceed 0.56). The color gradient shows the strength and direction of the correlation, with red indicating positive correlations and blue indicating negative ones.

Figure 8. Correlations between emotions and PTs. O, C, E, A, and N mean Openness, Conscientiousness, Extraversion, Agreeableness, and Non-Neuroticism traits. Low and high scores range from 0 to 0.55 and exceed 0.56, respectively.

Notably, Happiness exhibits strong negative correlations with low levels of Openness ($-0.31$), Conscientiousness ($-0.20$), Extraversion ($-0.36$), Agreeableness ($-0.21$), and Non-Neuroticism ($-0.28$) and strong positive correlations with high levels of these traits. Conversely, Sadness shows positive correlations with low trait levels (e.g., low Non-Neuroticism: $0.29$) and negative correlations with higher levels (e.g., high Extraversion: $-0.30$). Furthermore, low trait levels are positively correlated with all negative emotions (Anger, Disgust, Fear, and Sadness) and Neutral, while high trait levels are negatively correlated with these same emotions. The Surprise emotion does not show a significant correlation with either low or high trait levels, which can be attributed to its inherent ambiguity, as it can have both positive and negative valences.

Therefore, the figure reveals structured, interpretable knowledge: it demonstrates emotional profiles linked to varying degrees of PTs, providing data-driven evidence of the interplay between transient affective states and stable PTs.

5.4. Comparison with Sate-of-the-Art Methods

Current EPR methods are limited to unimodal frameworks. For example, Li et al. [79] and M. Talaat et al. [7] studied text-based input, and Gao et al. [13] and Li et al. [83] analyzed audio-based input. This work introduces a multimodal solution that is evaluated on two large-scale corpora. Since these corpora have not been explored in a multi-task setting before, we compare our single-domain results with existing single-domain SOTA results in Table 11. The comparison results show that the proposed framework outperforms the SOTA on both corpora, demonstrating its ability for multimodal feature fusion across both domains.

Table 11. Comparison with SOTA methods. Abbreviations: SSL-MER (Semi-Supervised Learning for Multimodal Emotion Recognition) and SSL-MPR (Semi-Supervised Learning for Multimodal Personality Recognition).

Method	Modality	mWACC/mUACC	mMF1
MOSEI
Zhang et al. [53]	Audio, Text, Face	51.2/–	–
Peng et al. [35]	Audio, Text, Face	66.4/–	–
Ryumina et al. [38]	Audio, Text, Face	69.3/–	–
Deng et al. [25]	Audio, Text	–/80.7	–
Ours (SSL-MER)	Audio, Text, Face, Scene, Body	70.3/83.5	70.6
Method	Modality	mACC	CCC
FIv2
Kong et al. [26]	Audio, Face	91.6	–
Zhao et al. [74]	Audio, Face, Scene	91.7	–
Aslan et al. [71]	Audio, Text, Face, Scene	91.8	–
Wang et al. [32]	Audio, Text, Scene	92.1	–
Agrawal et al. [76]	Audio, Text, Face, Scene, Body	92.6	–
Gan et al. [77]	Text, Scene	92.6	–
Ryumina et al. [4]	Audio, Text, Face	92.8	73.4
Ours (SSL-MPR)	Audio, Text, Face, Scene, Body	92.9	78.1

The performance gain is due to three key factors. Firstly, the framework employs semi-supervised learning by using unlabeled data from a different domain, which increases its robustness to variations in recording conditions and speaker diversity. Secondly, it makes use of auxiliary features from a complementary task, such as personality features for ER and emotional features for PTA. Finally, the framework integrates five different modalities (audio, text, face, body, and scene), and explicitly accounts for their varying informativeness across different corpora (for example, text dominance in the MOSEI dataset and visual dominance in FIv2) through the proposed fusion mechanisms.

5.5. Framework Proposed in Intelligent Systems

To demonstrate the practical applicability of the proposed SSL-MEPR system, an interactive multimodal prototype was implemented using the Gradio library (https://www.gradio.app, accessed on 3 December 2025). Figure 9 illustrates the user interface of the developed system.

Figure 9. User interface of the interactive prototype developed for the SSL-MEPR system.

The system accepts a video input via three flexible modes: (i) uploading a local video file, (ii) selecting from a set of preloaded sample videos, or (iii) recording a new video in real time using the user’s webcam and microphone. In addition to input selection, users can configure advanced settings, including the choice of inference model, computational device (CPU/GPU), segment length for analysis, and number of target features to visualize. After adjusting parameters, the user initiates analysis by clicking the ”Run” button.

Upon processing completion, the system provides a detailed analysis of the input data. This includes key frames extracted for the body, face, and scene, an audio waveform visualization, and a transcript generated by Whisper-turbo. The prediction results are presented as bar plots for both ER and PTA. In addition, the system visualizes the model’s attention distribution across different modalities and feature types that contributed most to each prediction. A summary report is then generated, providing a consolidated overview of the individual’s emotional state and personality profile based on knowledge extracted through multimodal, cross-task fusion of heterogeneous behavioral signals.

Such a system clearly demonstrates how multimodal Artificial Intelligence (AI) can extract psychologically grounded knowledge from everyday human behavior, moving beyond black-box predictions. This ability allows for more human-centered AI applications in fields such as personalized education, mental health monitoring, smart customer service, and professional recruitment.

SSL-MEPR combines semi-supervised multi-task cross-domain learning with real-time multimodal processing and intuitive visualization. This combination helps bridge the gap between complex ML models and practical human–computer interaction. As a result, AI systems become not only more accurate, but also more adaptable to diverse behavioral signals and interaction contexts.

5.6. Responses to Research Questions

RQ1: Can semi-supervised multi-task cross-domain learning effectively bridge two large-scale corpora annotated for different psychological tasks (emotion vs. personality) in the absence of joint labels?

As shown in Table 8, semi-supervised single-task cross-domain models (Exp M34 and M37) achieve state-of-the-art performance (70.26 mWACC on MOSEI; 92.88 mACC on FIv2), significantly outperforming their supervised single-domain counterparts (Exp M35 and M36, M38 and M39, respectively). This confirms that SSL can successfully transfer knowledge across corpora despite differences in annotation protocols. However, when both tasks are trained jointly in a multi-task setting (Exp M7), performance decreases (64.26 mWACC; 92.00 mACC), indicating that joint optimization under task-specific modality information bias introduces interference. Therefore, while SSL enables cross-corpus integration, multi-task learning requires careful architecture and optimization design to avoid mutual degradation.

RQ2: To what extent does cross-domain transfer improve or hinder performance, given divergent modality informativeness across annotation protocols?

Cross-domain transfer can be beneficial, but only if modality–task alignment is taken into account. As shown in Section 4.3, Section 4.4 and Section 4.5, the text-derived emotion labels from MOSEI make linguistic features more important for ER (Exp L8: 64.80 mWACC), whereas the video-based personality annotations from FIv2 make the visual modalities (face and body) more informative for PTA (Exp V10 and V12: more than 91.38 mACC and more than 68.39 CCC, respectively). Table 8 (M8-M32) shows that removing the task-relevant modality (e.g., text for ER or face for PTA) leads to a significant performance drop, while removing the task-irrelevant modality has a minor impact. The modality-task bias introduced by the annotation protocols creates a challenge for multi-task learning. The model struggles to balance conflicting priorities between modalities, as shown in Figure 7. This is because features that are optimal for one task can be noisy or even misleading for another task.

RQ3: Does the model leverage cross-task features during joint inference, and how does this reflect the underlying emotion-personality relationship?

The cross-task feature usage is both architecturally feasible and psychologically relevant. The attention maps in Figure 7 show that during ER, the model pays attention not only to emotional acoustic/linguistic features, but also to personality-related facial and scene features (for example, facial expressiveness for Agreeableness). In contrast, during PTA, it uses emotional acoustic and linguistic features. This cross-adaptation is enabled by predict projectors and guide banks, which project unimodal predictions into a shared latent space (Table 8, M4–M5: performance decreases when removed). Quantitatively, this cross-task interaction is confirmed in single-task settings (see Table 8, Exp M34–M39). Leveraging emotional features improves the PTA performance from 76.13 CCC to 78.05 CCC, and incorporating personality features increases ER from 67.77 mWACC to 70.26 mWACC. Additionally, the observed negative correlation between Sadness and high PTs scores, as well as the positive link between Happiness and the same PTs (Figure 8), provide empirical evidence for the psychological interdependence between emotions and PTs. Despite annotation bias, the model learns genuine cross-task dependencies. This suggests that joint modeling, when properly regularized, can uncover deeper affective–behavioral patterns.

6. Conclusions

We present a semi-supervised multi-task cross-domain learning framework for MEPR, called SSL-MEPR, explicitly designed for knowledge extraction across affective and personality modeling tasks. To ensure robustness, we introduce a three-stage learning method. The framework is evaluated on two large-scale corpora: MOSEI, annotated for ER, and FIv2, annotated for five PTs. As no existing corpora combine both tasks, our framework employs SSL to bridge heterogeneous annotations and extract shared representations that reflect psychological interdependencies.

Stage 1 trains unimodal single-domain models. Results demonstrate that selecting optimal feature extractors (CLIP for video, Wav2Vec2.0 for audio, and BGE-en for text) is more critical than choosing temporal architectures (transformer vs. Mamba).

Stage 2 develops unimodal cross-domain models using transformers with cross-attention mechanisms to aggregate information across domains. While this method achieves minor improvements in performance over single-domain models, it provides a cross-domain representation of features.

Stage 3 introduces a novel model architecture based on four new components that jointly facilitate cross-modal and cross-task knowledge extraction. These include a multimodal cross-domain model that incorporates Graph Attention Fusion and Task-Specific Query-based Multi-Head Cross-Attention Fusion. Additionally, there are task-specific predict projectors and guide banks, which enable the system to distill task-invariant and task-specific knowledge from heterogeneous, semi-labeled data. Our findings reveal two key limitations: (i) single-domain learning introduces overfitting to a single corpus, while (ii) cross-domain models struggle due to domain-specific modality informativeness—where text, audio, and scene modalities dominate in ER, while scene, face, and body modalities are more informative for PTA. Notably, despite its three-stage learning, the framework supports real-time inference on both CPU and GPU.

Our experimental results directly address the three research questions:

• RQ1: Semi-supervised learning successfully bridges emotion and personality corpora in single-task settings (70.26 mWACC on MOSEI; 92.88 mACC on FIv2), but multi-task joint training degrades performance (64.26 mWACC; 92.00 mACC), reflecting the inherent tension between ER and PTA annotation protocols.
• RQ2: Cross-domain transfer is beneficial only when modality–task alignment is respected. Linguistic features dominate ER due to the annotation protocol in MOSEI, while visual modalities drive PTA on FIv2. Ignoring this divergence leads to interference, as confirmed by ablation studies and model’s attention visualization.
• RQ3: During joint inference, the model uses both emotional and personality features for each task. For instance, personality-related visual features aid ER, while emotional linguistic cues assist PTA. This shows that the model captures the inherent psychological interdependence between emotions and PTs.

In addition, extensive ablation studies confirm that the proposed model architecture, which is based on four innovative components, is resilient to both the order of modalities and the substitution of additional graph layers or attention mechanisms. This demonstrates the coherence and robustness of its design.

Despite its strengths, our framework has several limitations. First, performance remains sensitive to corpus-specific annotation protocols, which hinders full cross-task transfer. Second, the model is trained and evaluated only on large-scale English corpora, limiting its applicability to other languages or low-resource domains. Third, the use of graph-based fusion and cross-attention mechanisms incurs high computational cost, especially when scaling to longer sequences or additional modalities.

These limitations point to clear future directions: (i) developing annotation-agnostic representation learning to reduce corpus bias; (ii) extending the framework to multilingual and low-resource settings through self-supervised pre-training; and (iii) designing more efficient fusion architectures to improve scalability while preserving cross-task knowledge transfer.