Multi-Source Domain Adaptive EEG Emotion Recognition Based on Dendrite Net

Abstract

Accurate emotion recognition is crucial for enhancing human–computer interaction, and brain–computer interface (BCI) technology offers an efficient means for emotion detection using the EEG signal. However, existing methods face significant challenges due to the inherent inter-individual differences and temporal variability of EEG data. To address these limitations, this paper introduces a multi-source domain adaptive algorithm based on dendrite net (DD-MSDA). The proposed model employs the dendrite network as a shared feature extractor to align feature distributions across multiple source domains, thereby capturing common features among diverse datasets. Experimental validation on cross-subject and cross-session tasks using the SEED and SEED-IV datasets demonstrates that DD-MSDA achieves highly competitive performance, outperforming all compared single-modal EEG-based domain adaptation methods. Moreover, the algorithm demonstrates statistically significant advantages over existing domain adaptation baselines in cross-dataset settings. These results highlight the consistent competitiveness of DD-MSDA across various cross-domain scenarios, and its unsupervised nature underscores its potential for practical online EEG emotion recognition applications.

1. Introduction

Emotion is a multidimensional psychological state that is typically triggered by external stimuli. Maintaining a positive emotional state is crucial for human physical and mental well-being, while long-term negative emotions can have a serious impact on psychological and physiological health [1]. Therefore, it is of great value to develop algorithms that can accurately recognize human emotions through behavioral measurements and physiological signals, which will help computers better respond to human needs, enabling more natural and intelligent interaction. Affective computing focuses on the development of data-driven computational models to identify human emotional states [2]. A variety of human-generated signals can be used to detect and quantify emotions, including facial expressions [3], speech [4], and biological signals such as electrocardiogram (ECG) and electroencephalogram (EEG) [5]. Among them, EEG technology is particularly reliable for objective emotion recognition due to the spontaneous nature of emotions accompanied by the central nervous system and peripheral physiological responses. EEG signals are derived from neural activity and cannot be concealed, thus providing a more authentic reflection of human emotional states [6].

EEG-based emotion recognition has emerged as a popular research topic in brain–computer interface (BCI) studies [7]. Currently, traditional machine learning methods such as Bayesian classifiers, logistic regression, and support vector machines, as well as deep learning methods including convolutional neural networks (CNNs) [8,9], capsule network (CapsNet) [10], long short-term memory (LSTM) networks, and graph neural networks (GNNs) [11,12], are predominantly employed in EEG-based emotion recognition. However, the substantial inter-individual and intra-individual variations in EEG signals across different individuals and time periods pose significant limitations to the practicality of traditional EEG-based emotion recognition models, hindering their real-world applications [13]. As early as 1997, Riding et al. discovered that individual differences remain prominent even during the same physiological activities [14]. Domain adaptation (DA) [15] and domain generalization (DG) [16] techniques, as one of the cutting-edge research directions in machine learning, aim to map data features from different domains (e.g., two different datasets) into a common feature space, so that the data from the source domains could be utilized to improve training for the target domain. Though they are highly demanded in research fields like computer vision, their application in cross-domain emotion recognition using EEG signals remains limited. Considering the variability nature of EEG signals, it is imperative to improve DA and DG methods to address individual differences and increase the generalization capability of emotion recognition models [17].

This paper focuses on solving the cross-domain emotion recognition problem by improving the existing DA methods. Currently, most studies combine EEG data from different subjects and sessions into a single source domain for transfer learning. However, such approaches do not satisfy the assumption of DA, which requires the source domain to have a certain marginal distribution. Employing multi-source DA approaches can solve the above problem, but existing multi-source DA methods often use a three-layer MLP neural network to extract shared feature representations, which may lead to overfitting issues [18]. Furthermore, existing algorithms only align the data distribution features between individual source and target domains, rather than effectively identifying common features among all source domains.

Therefore, we propose a multi-source DA algorithm based on the dendrite (DD) net for cross-domain EEG-based emotion recognition. The DD net is a white-box machine learning algorithm for classification, regression, and system identification proposed by our team [19], and it is utilized as a common feature extractor to prevent overfitting and improve the generalization performance of the model in the current study. Furthermore, we align the multiple feature representation distributions extracted from multiple source domains to extract common features among them. The main contributions of this study are as follows:

1. DD net is proposed to be utilized in the common feature extractor to effectively prevent the overfitting problem.

2. It is proposed to identify the common features among multiple source domains by calculating multiple maximum mean discrepancy (MMD) distances between the feature distributions of multiple branches in the source domain.

3. By integrating individual differences and collective commonalities from multiple source domains, we propose an efficient EEG-based cross-domain emotion recognition network. Comparative experiments demonstrate it achieves excellent performance in cross-subject and cross-session scenarios on the SEED and SEED IV datasets.

4. In the cross-dataset emotion recognition experiments on SEED and DEAP, our proposed algorithm also shows the best performance among the comparison algorithms.

The rest of the paper is organized as follows: Section 2 provides a brief overview of related works. Section 3 introduces the data setups of this study. Section 4 introduces the methodology of the proposed multi-source domain adaptation algorithm based on the dendrite net. Section 5 presents the experimental setup and results. Section 6 discusses the findings. Finally, Section 6 is a full summary of the paper.

2. Related Works

Currently, the DA algorithm is one of the primary methods to address the aforementioned cross-domain emotion recognition problem. DA involves transferring knowledge from one or more source domains, consisting of subjects/sessions, to a new target domain. Several typical DA algorithms aim to learn shared features that can be applied to new subjects/sessions by reducing the distribution differences between the source and target domains. In this way, models trained on labeled source data and a small amount of unlabeled target domain data can make predictions on the remaining target domain data. In terms of cross-domain EEG-based emotion recognition using traditional DA algorithms, Zheng and Lu [20] firstly compared transfer component analysis (TCA) [21], kernel principal component analysis (KPCA) [22], and transductive parameter transfer (TPT); the results showed that TPT achieved the highest accuracy of 76.3% on the SEED dataset, which is a significant improvement of 19.6% compared to a general classifier without DA. Chai et al. [23] proposed an adaptive subspace feature matching (ASFM) strategy, which incorporates subspace alignment (SA) algorithm as its fundamental component. The SA algorithm aligns the PCA subspaces of the training domain data with the subspaces of the testing domain data through linear transformations.

In addition to the classical DA methods, there are also some deep learning-based DA methods. For instance, Wang et al. [24] proposed the deep multiple-source adaptive transfer network (DMATN) algorithm, which adopts a multi-source domain adaptation approach and adds a domain discriminator to identify the most similar individuals to the participant for training. As a result, it achieved an accuracy of 84.22%. Chai et al. [25] proposed an autoencoder architecture to reduce the differences between training and testing subjects in the learned latent space. Domain adversarial neural networks (DANNs) introduced a domain adversarial training strategy composed of a domain classifier and a gradient reversal layer, which can learn domain-invariant representations across source and target domains. Li et al. [26] proposed the dynamic domain adaptation (DDA) algorithm where the global and local divergences are disposed by minimizing the global domain discrepancy and local subdomain discrepancy, respectively. Zhu et al. [27] employed Wasserstein distance to measure the marginal and conditional distributions between the source and target domains. Miao et al. [28] proposed an individual identification method, multi-loss domain adaptor (MLDA), to deal with differences between marginal distribution and conditional distribution caused by different classes. Li et al. [29] utilized a multi-scale approach to represent the connectivity features of EEG signals by combining a multi-scale residual network with meta-transfer learning strategies. In recent years, DG algorithms have also been applied to cross-subject emotion recognition. In comparison to DA, DG does not require access to the target domain data and aims to obtain a universal model, making it more favorable in practical applications. Ma et al. [30] constructed a domain residual network to learn domain-shared weights and domain-specific weights through domain adversarial training and then used the domain-shared weights for the classification of unknown data. In 2021, Zhao et al. [17] proposed a method for handling subject variability using a DG approach, which uses shared encoders and private encoders to capture the subject-invariant emotion representations and the unique components of the source subjects separately in the training phase, and a comparative experiment showed that this method achieved similar performance to DA methods on the SEED dataset. In addition, there are also approaches such as contrastive learning [31] and pairwise learning [32] that are used to narrow the disparities between subjects.

At the same time, in the cross-domain research of EEG emotion recognition, there is a growing trend of exploring the transferability of decoding models across different datasets, in addition to the cross-session and cross-subject studies within the same dataset. In 2019, Lan et al. [33] employed multiple DA techniques to reduce the distribution differences between the DEAP and SEED datasets, using one dataset as the source domain and the other as the target domain, and their experiments showed that the emotion decoding accuracy could be improved by 7.25–13.40% with the use of DA techniques when compared with the baseline method without DA. Dar et al. [34] combined a one-dimensional convolutional recurrent neural network with an extreme learning machine to align the feature distributions of the two datasets. Chen et al. [35] developed several weight extraction methods to quantify patterns in emotional feature distribution, thereby enhancing the stability and effectiveness of the general model in cross-dataset tasks. Jiang et al. [36] proposed a domain symmetry module to address the issue of inconsistent label space in cross-dataset EEG emotion recognition. The module gradually aligns the label space of the target domain with that of the source domain by expanding the target domain through the random selection of samples from the source domain. As human emotional responses can be induced by various stimuli such as images, music and movies, it is likely there are shared brain activation patterns behind such responses. Therefore, it has strong practical application values for the transfer of emotion recognition models across datasets.

3. Materials and Methods

Due to the individual differences and temporal variability of EEG data, the domain adaptive aspects of emotion recognition can be divided into the following three cases: (1) Cross-subject transfer. In one session, EEG data of new subjects are used as target domain data, and the existing EEG data of other subjects are used as source domain data. (2) Cross-session transfer. For a subject, a new session is selected as target domain data, and other existing sessions are selected as the source domain data. In our experiment, the SEED and SEED-IV datasets had three sessions and 15 subjects (see Section 3.1 for details). In the cross-subject transfer, we select 14 subjects in a session as the source domain data. In the cross-session transfer, we use two sessions of a subject as the source domain data, the remaining one as the target domain data, and finally perform cross-validation. (3) Cross-dataset transfer. This refers to the transfer of emotion recognition models between different datasets; we take SEED or DEAP dataset as the target domain and the remaining dataset as the source domain.

3.1. Datasets

Three public datasets were selected in the current study:

SEED [37]: A total of 15 Chinese film clips (positive, neutral, and negative emotions) were chosen as stimuli for the experiment. Fifteen Chinese participants (7 males, 8 females, mean age: 23.27, STD: 2.37) took part in the experiment. Each subject completed the experiment three times, with all 15 film clips being watched in each session, resulting in a total of 15 videos per session. EEG data were collected using 62 channels of the international 10–20 system. The data are downsampled to 200 Hz.

SEED-IV [38]: This is an evolution of the SEED dataset, consisting of three sessions, with 15 participants in each session. In each session, the participants viewed 24 movie clips (Happy, Sad, Neutral, and Fear). The database also includes the eye movement features of the participants during the experiment. In current study, only the EEG data were used, and they are downsampled to 200 Hz.

DEAP [39]: The dataset consisted of 32 subjects. Each subject was instructed to watch 40 1 min music videos as emotional stimuli. EEG data were recorded for 63 s in each trial, with the first 3 s as baseline data. Each subject recorded 40 EEG trials corresponding to a music video that induced an emotion. After watching each video, subjects needed to understand the emotions they felt to assess from four dimensions: (1) valence; (2) arousal; (3) dominance; and (4) liking. Thirty-two channels of the EEG signal were collected for each subject, and the EEG data are downsampled to 128 Hz.

In this paper, SEED and SEED-IV were used for cross-session and cross-subject emotion recognition studies; DEAP and SEED were used for cross-dataset emotion recognition studies.

3.2. Preprocessing and Feature Extraction

In the cross-session and cross-subject experiments, we employed a “linear dynamic system” to eliminate noise and artifacts unrelated to the brain’s electrical activity [40]. We then extracted the differential entropy (DE) [41] features from five frequency bands: Delta (1–4 Hz), Theta (4–8 Hz), Alpha (8–14 Hz), Beta (14–31 Hz), and Gamma (31–50 Hz). Assuming that the EEG signal $\mathrm{X}\left(\mathrm{t}\right)$ follows the Gaussian distribution $\mathrm{x}~\mathrm{N}(\mathrm{\mu },{\mathrm{\sigma }}^2)$, the DE feature can be simplified as

$$\mathrm{D}\mathrm{E}=-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\displaystyle \frac{1}{\sqrt{2\mathrm{\pi }\mathrm{\sigma }}}}\mathrm{e}\mathrm{x}\mathrm{p}{\displaystyle \frac{{\left(\mathrm{x}-\mathrm{\mu }\right)}^2}{2{\mathrm{\sigma }}^2}}\ln \left({\displaystyle \frac{1}{\sqrt{2\mathrm{\pi }\mathrm{\sigma }}}}\mathrm{e}\mathrm{x}\mathrm{p}{\displaystyle \frac{{\left(\mathrm{x}-\mathrm{\mu }\right)}^2}{2{\mathrm{\sigma }}^2}}\right)\mathrm{d}\mathrm{x}$$

$$={\displaystyle \frac{1}{2}}\mathrm{l}\mathrm{n}{2\mathrm{\pi }\mathrm{e}\mathrm{\sigma }}^2$$

(1)

where $\mathrm{\pi }$ is the circumference-to-diameter ratio, $\mathrm{\mu }$ and ${\mathrm{\sigma }}^2$ are the mean and variance of the EEG signal, and $\mathrm{e}$ is the Euler number. It has been proven that for a fixed-length EEG signal, the DE feature is equivalent to the logarithmic energy spectrum in a certain frequency band of the given EEG signals.

The DE features of five frequency bands from all the EEG channels were extracted and then concatenated to form the feature vector for each sample. For the SEED dataset, each sample window was 1 s long, resulting in a total of 3394 samples per participant in a single session. For the SEED-IV dataset, each sample window was 4 s long, resulting in 851/832/822 samples per participant in a single session. In the cross-dataset experiment, the baseline data of the DEAP dataset was removed, and the samples were split with a time window of 1 s. The final obtained feature and label dimensions of the three datasets are shown in

Table 1. Dataset introduction.

Dataset	Feature Data	Label Data
SEED	3 × 15 × 3394 × 310 (session × subject × trial × feature)	3 × 15 × 3394 (session × subject × trial)
SEED-IV	3 × 15 × 851/832/822 × 310 (session × subject × trial × feature)	3 × 15 × 851/832/822 (session × subject × trial)
DEAP	32 × 40 × 40 × 160 (subject × music × trial × feature)	32 × 40 × 40 (subject × music × trial)

.

3.3. The Framework of DD-MSDA

Given an existing set of EEG data and EEG data from a new subject, the goal is to learn a decoding model $\mathrm{\varphi }$ that employs DA methods to train on these known source domain data for predicting the new subject more effectively. As illustrated in Figure 1, DD-MSDA integrates a shared dendrite net for feature alignment, domain-specific branches for individual variability, and a multi-loss optimization framework. The structure of the proposed multi-source domain multi-representation self-adaptive algorithm based on dendrite net (DD-MSDA) is illustrated in Figure 1. It takes multiple source domain data and one target domain data as inputs; these data are then fed into a common feature extractor built based on the DD net to obtain the domain-invariant features. Multiple sets of data extracted by the common extractor are fed back into individual branches to obtain domain-specific features for each subject. The MMDs between source and target features in each branch are calculated as a measure of the discrepancy between the source and target domains. Furthermore, the MMDs between the target-specific features obtained from multiple branches are calculated to obtain a shared feature for all source domain data. Next, an activation layer is added after the domain-specific feature extractor to obtain the predicted labels for the source and target domains in multiple branches. Finally, these target domain predictions are fused to obtain the final predicted label for the target domain. The optimization of the parameters is achieved by minimizing the source-target domain MMD distance, target-target domain MMD distance, source domain classification loss, and variance loss of predictions of target domain. Detailed information on these modules is provided below.

3.3.1. DD Common Extractor

Chen et al. [42] employed a three-layer MLP neural network as the common extractor. However, MLP feedforward neural networks are prone to overfitting, easily falling into local optima, and exhibit a strong dependency on data features.

In this study, we adopted the DD net [19] as the common feature extraction layer. Its white-box properties and controllable precision allow for better generalization capabilities. Our team demonstrated that the DD net exhibits superior generalization capabilities and good overfitting prevention abilities when compared to the MLP architecture that simulates neuron cells (cell body Net) through nine practical applications. Additionally, the DD net performs better in scenarios with few-shot data.

The DD net consists of DD modules and linear modules. In this work, two linear modules were chosen, and a DD module was inserted between them. The representation of the DD module is as follows:

$${\mathrm{A}}^{\mathrm{l}}={\mathrm{W}}^{\mathrm{l},\mathrm{l}-1}{\mathrm{A}}^{\mathrm{l}-1}\circ \mathrm{X}$$

(2)

In this context, ${\mathrm{A}}^{\mathrm{l}-1}$ and ${\mathrm{A}}^{\mathrm{l}}$ represent the input and output of the module, respectively. $\mathrm{X}$ refers to the original input of the DD module. ${\mathrm{W}}^{\mathrm{l},\mathrm{l}-1}$ denotes the weight matrix for the linear transformation from the $\mathrm{l}-1$ module to the $\mathrm{l}$ module. The element-wise Hadamard product operation intrinsically constrains the hypothesis space complexity of the model. This structural constraint prevents the network from overfitting the random fluctuations inherent in EEG signals—which arise from inter-individual differences and recording environments—and instead forces the model to learn stable, underlying features that are common across the source and target domains. This module aids in extracting certain low-level domain-invariant features.

3.3.2. Domain-Specific Extractor

D, the common feature extractor, is followed by a domain private feature extractor; we design $\mathrm{N}$ as separate fully connected layers to correspond to $\mathrm{N}$ source domains, and the source domain-target domain MMD loss is calculated as

$${\mathrm{l}}_{\mathrm{m}\mathrm{m}\mathrm{d}}=\parallel {\displaystyle \frac{1}{{\mathrm{N}}^{\mathrm{S}}}}{\displaystyle \sum _{\mathrm{i}=1}^{{\mathrm{N}}^{\mathrm{S}}}}\mathrm{\varphi }\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{S}}\right)-{\displaystyle \frac{1}{{\mathrm{N}}^{\mathrm{T}}}}{\displaystyle \sum _{\mathrm{i}=1}^{{\mathrm{N}}^{\mathrm{T}}}}\mathrm{\varphi }\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\right){\parallel }_{\mathrm{H}}^2$$

(3)

where ${\mathrm{x}}_{\mathrm{i}}^{\mathrm{S}}$ denotes the source domain data, ${\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}$ denotes the target domain data, and ${\mathrm{N}}^{\mathrm{S}}$ and ${\mathrm{N}}^{\mathrm{T}}$ are the number of the source and target samples. Meanwhile, it is also necessary to identify common features that are well-suited for the target domain across these $\mathrm{N}$ source-target branches; then, the target-target domain MMD distance ${\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}$ is calculated to measure the similarity of the target domain feature distribution obtained from various branches, where ${\mathrm{N}}^{\mathrm{T}-1}$, ${\mathrm{N}}^{\mathrm{T}}$ represent the feature data of different target domains in multiple branches. The target-target domain MMD loss is calculated as

$${\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}=\parallel {\displaystyle \frac{1}{{\mathrm{N}}^{\mathrm{T}-1}}}{\displaystyle \sum _{\mathrm{i}=1}^{{\mathrm{N}}^{\mathrm{T}-1}}}\mathrm{\varphi }\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}-1}\right)-{\displaystyle \frac{1}{{\mathrm{N}}^{\mathrm{T}}}}{\displaystyle \sum _{\mathrm{i}=1}^{{\mathrm{N}}^{\mathrm{T}}}}\mathrm{\varphi }\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\right){\parallel }_{\mathrm{H}}^2$$

(4)

Throughout the above training process, the divergence in feature distribution between the source domain and the target domain within each branch, as well as the distinction in feature distribution among targets in multiple branches, is diminished by minimizing the two MMD losses.

3.3.3. Domain-Specific Classifier

The role of the domain classifier is to predict the features extracted in the domain-specific extractor. The number of source branches $\mathrm{N}$ is equal to the number of source domains used in a given experiment. There are $\mathrm{N}$ softmax classifiers corresponding to each source domain in the domain classifier, and the classification loss is estimated by calculating the cross-entropy during the training process of each classifier. Assuming the true label distribution of the source domain is ${\mathrm{y}}^{\mathrm{S}}$, the predicted label distribution is $\widehat{{\mathrm{y}}^{\mathrm{S}}}$, and the total number of categories is $\mathrm{n}$, the cross-entropy loss function (classification loss) is calculated as follows:

$${\mathrm{l}}_{\mathrm{c}\mathrm{l}\mathrm{s}}=-{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}}{{(\mathrm{y}}^{\mathrm{S}})}_{\mathrm{i}}\mathrm{l}\mathrm{o}\mathrm{g}{\left(\widehat{{\mathrm{y}}^{\mathrm{S}}}\right)}_{\mathrm{i}}$$

(5)

Moreover, if the predicted labels of the target domain obtained based on the $\mathrm{N}$ classifiers are directly averaged, the variance will be high, especially when the target domain samples are at the decision boundary, leading to a serious negative effect. For this reason, the model introduces a measure of the variance loss of the predictions to align the predictions of the $\mathrm{N}$ classifiers. Assuming that the label distributions of the target domain obtained from the predictions of the classifiers based on the jth and kth source domains are $\widehat{{\mathrm{y}}_{\mathrm{j}}^{\mathrm{T}}}$ and $\widehat{{\mathrm{y}}_{\mathrm{k}}^{\mathrm{T}}}$, the variance loss ${\mathrm{l}}_{\mathrm{v}\mathrm{a}\mathrm{r}\mathrm{p}}$ is calculated by (6).

$${\mathrm{l}}_{\mathrm{v}\mathrm{a}\mathrm{r}\mathrm{p}}=-{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}}{\left(\widehat{{\mathrm{y}}_{\mathrm{j}}^{\mathrm{T}}}\right)}_{\mathrm{i}}\mathrm{l}\mathrm{o}\mathrm{g}{\left(\widehat{{\mathrm{y}}_{\mathrm{k}}^{\mathrm{T}}}\right)}_{\mathrm{i}}\left(\mathrm{j},\mathrm{k}\in \mathrm{N},\mathrm{j}\ne \mathrm{k}\right)$$

(6)

In summary, DD-MSDA consists of a DD common extractor, a domain-specific extractor, and a domain-specific classifier. The DD common extractor employs DD net to extract features that are common to all domains, leveraging superior generalization and overfitting prevention abilities. The domain-specific extractor utilizes multiple fully connected separate branches to extract domain-private features and measures the similarity between different domains by calculating the MMD between the source and target domains, as well as the multiple target domains. The domain-specific classifier uses $\mathrm{N}$ softmax classifiers corresponding to $\mathrm{N}$ source domains to assess the classification loss and the variance loss of the prediction results for multiple target domains through cross-entropy calculations. The overall training loss is expressed as

$$\mathrm{l}={\mathrm{l}}_{\mathrm{c}\mathrm{l}\mathrm{s}}+{\mathrm{\alpha }\mathrm{l}}_{\mathrm{m}\mathrm{m}\mathrm{d}}+\mathrm{\beta }{\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}+\mathrm{\gamma }{\mathrm{l}}_{\mathrm{v}\mathrm{a}\mathrm{r}\mathrm{p}}$$

(7)

where $\mathrm{\alpha },\mathrm{\beta },\mathrm{\gamma }$ are hyperparameters. The final predicted label for the target domain is obtained by first averaging the softmax probability vectors from all $\mathrm{N}$ domain-specific classifiers and then taking the class with the maximum averaged probability.

4. Results

In this section, the experimental setup, emotion recognition results, and ablation experiments on the two datasets, SEED and SEED IV, are first detailed. Then, we conducted hyperparameter experiments and visualized the loss function. Finally, the cross-dataset emotion recognition performance is evaluated using the DEAP and SEED datasets.

4.1. Experiment Setups

In the cross-session and cross-subject experiments, the two linear modules in DD net are 310D-128D and 128D-64D, respectively. In the cross-dataset experiment, the linear modules are 160D-128D and 128D-64D. The domain-specific extractor includes a linear layer with dimensions of 64D-32D, and a LeakyReLU activation function [43] is then applied. Due to the sensitivity of EEG data, utilizing ReLU activation may result in information loss, as negative values would be discarded. Therefore, we chose to use LeakyReLU as a compromise solution [44]. The Adam optimizer is used to train the network with an initial learning rate of 0.01. The hyperparameters are set as follows: $\mathrm{\alpha }=2/(1+{\mathrm{e}}^{-10\ast (\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}/\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{s})})-1$, and $\mathrm{\beta }=\mathrm{\gamma }=\mathrm{\alpha }/100$. An example of the changes of $\mathrm{\alpha }$ along with the iterations is depicted in Figure 2. We employ DE as the feature vector and reshape each sample into a 310-D (160-D) vector. Additionally, electrode-wise normalization is used to standardize the data, as shown in Figure 3, which has been shown to improve the performance by Chen et al. [42] and Wang et al. [45]. During the training, the initial learning rate is set to 0.01, and the batch size is 32. The epoch setting varies with the number of samples per subject to ensure that the number of iterations is guaranteed to be slightly greater than 3000, so the epoch number is set to 30 and 120 in SEED and SEED-IV respectively. The Adam optimizer is utilized for gradient descent.

For each cross-session or cross-subject experiment, the data of the target session/subject are randomly split into an unlabeled adaptation set (10% of samples) and a test set (90%). During the training of DD-MSDA, only the unlabeled adaptation set is used together with the source domains to compute the source-target MMD loss, target-target MMD loss, and variance loss; no target labels are involved. After training, the model is evaluated on the held-out test set, and the reported accuracies are based solely on this test set.

4.2. Comparison Methods

To validate the effectiveness of the proposed DD-MSDA, we conduct a comparative analysis of our method against several algorithms. These methods include both classic and recent advances in the field. These algorithms are described as follows:

DDC [46]: Based on a classical deep network architecture, it introduces domain confusion and soft label loss for task transfer.

DNN [47]: Utilizes deep adaptation networks for EEG-based emotion recognition.

DANN [48]: Integrates a gradient inversion layer into the deep network, as well as a domain discriminator, to better distinguish between source and target domains.

DAAN [49]: Represents dynamic learning domain invariability while evaluating the relative importance of features.

MS-MDA [42]: Uses an MLP as a common feature extractor to obtain the classification label of the target domain subjects by measuring multiple losses of the source domain and the target domain.

PPDA [17]: Proposes a plug-and-play domain-adaptive approach to reduce calibration time.

CLISA [50]: Utilizes a contrastive learning approach for domain adaptation.

MMDA-VAE [45]: Uses an autoencoder structure to learn the difference in distribution through adversarial learning (multimodal features).

4.3. Cross-Session and Cross-Subject EEG-Based Emotion Recognition Results

(1) Comparison Results:

Table 2. Cross-subject and cross-session classification accuracies (Mean ± Std (%)) of different methods on SEED and SEED-IV.

Dataset	Method	Cross-Session	Cross-Subject
SEED	DDC	$84.45\pm 8.46$*	$70.12\pm 6.76$*
	DAN	$83.97\pm 8.36$*	$70.31\pm 7.38$*
	DANN	$-$	$79.19\pm 13.14$
	DAAN	$-$	$83.54\pm 9.81$
	MS-MDA	$88.19\pm 9.36$*	$81.86\pm 9.09$*
	PPDA	$-$	$86.70\pm 7.10$
	CLISA	$-$	$86.30\pm 6.40$
	MMDA-VAE	$89.64\pm 11.78$	$85.07\pm 11.81$
	DD-MSDA (ours)	$91.15\pm 7.81$	$87.57\pm 7.43$
SEED-IV	DDC	$67.62\pm 13.43$*	$60.54\pm 9.24$*
	DANN	$-$	$54.63\pm 8.03$
	DAN	$69.96\pm 13.09$*	$58.99\pm 8.95$*
	MS-MDA	$68.66\pm 11.12$*	$64.64\pm 10.83$*
	MMDA-VAE	$73.82\pm 11.74$	$75.52\pm 10.21$
	DD-MSDA (ours)	$75.57\pm 11.71$	$74.79\pm 10.83$

* denotes the accuracies are statistically lower than DD-MSDA (Wilcoxon signed-rank test, p < 0.05). $-$ denotes the accuracies are not given in the original study.

presents the experimental results comparing various algorithms, including our proposed DD-MSDA, on the SEED and SEED IV datasets. The results for DDC, DAN, and MS-MDA are obtained using our experimental methodology and the same hyperparameters, while the results for the other algorithms are quoted from their original papers. In particular, MMDA-VAE leverages multimodal features (EEG and eye movement) on SEED-IV, which provides it with additional discriminative information not available to the other single-modal EEG-based methods in the comparison. Statistical significance was assessed using the paired Wilcoxon signed-rank test (p < 0.05). The experimental results indicate that DD-MSDA significantly outperforms other algorithms with an average classification performance of 91.15 ± 7.81% and 87.57 ± 7.43% in cross-session and cross-subject scenarios on the SEED dataset. For the SEED-IV dataset, DD-MSDA achieves an average classification performance of 75.57 ± 11.71% and 74.79 ± 10.83% in cross-session and cross-subject scenarios, respectively, which is higher than most of the algorithms except MMDA-VAE. The reason may be that MMDA-VAE utilizes multimodal features combining eye movement with EEG features, while our study only uses the EEG features on the SEED-IV dataset. Furthermore, it is worth noting that our method exhibits superior emotion recognition improvements on SEED-IV than SEED. We also draw the confusion matrices obtained by our method in the two datasets, as shown in Figure 4. In each matrix, the column denotes the true label, while the row denotes the predicted label. It is evident that high recognition accuracies are achieved for all emotion categories, as the true positive values are much higher than the true negative and false negative values.

(2) Ablation Study: The proposed DD-MSDA aims to find the common features between source and target domains to achieve the efficient transfer of the decoding model. This is achieved mainly through the DD common extractor and the optimization of the loss ${\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}$. We conduct the following ablation study to validate their effectiveness. The results are presented in

Table 3. Ablation study for cross-subject and cross-session classification accuracies (Mean ± Std (%)) on SEED and SEED-IV.

Dataset	Method	Cross-Session	Cross-Subject
SEED	DD-MSDA	$91.15\pm 7.81$	$87.57\pm 7.43$
	DD-MSDA w/o. $l_{ttmmd}$	$91.09\pm 6.68$	$86.33\pm 8.58$
	DD-MSDA w/o. $l_{mmd}$	$85.62\pm 8.15$	$76.54\pm 10.28$
	DD-MSDA w/o. $l_{varp}$	$90.05\pm 7.32$	$84.75\pm 8.84$
	MLP-MSDA	$90.61\pm 7.37$	$85.32\pm 7.32$
	MLP-MSDA w/o. $l_{ttmmd}$	$88.19\pm 9.36$	$81.86\pm 9.09$
	MLP-MSDA w/o. $l_{mmd}$	$80.96\pm 10.18$	$72.45\pm 11.74$
	MLP-MSDA w/o. $l_{varp}$	$87.62\pm 9.96$	$80.24\pm 10.52$
SEED-IV	DD-MSDA	$75.57\pm 11.71$	$74.79\pm 10.83$
	DD-MSDA w/o. $l_{ttmmd}$	$72.62\pm 13.47$	$70.06\pm 11.55$
	DD-MSDA w/o. $l_{mmd}$	$68.26\pm 14.23$	$62.94\pm 11.48$
	DD-MSDA w/o. $l_{varp}$	$72.59\pm 14.16$	$71.24\pm 12.81$
	MLP-MSDA	$73.12\pm 13.72$	$72.50\pm 11.05$
	MLP-MSDA w/o. $l_{ttmmd}$	$68.66\pm 11.12$	$64.64\pm 10.83$
	MLP-MSDA w/o. $l_{mmd}$	$64.66\pm 12.37$	$60.18\pm 11.82$
	MLP-MSDA w/o. $l_{varp}$	$67.52\pm 12.84$	$63.23\pm 12.51$

, where MLP-MSDA represents using MLP instead of DD net as the common feature extraction layer and DD-MSDA/MLP-MSDA w/o. ${\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}$ refers to training without the loss ${\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}$ ($\mathrm{\beta }$ = 0). It can be seen that the DD net is more effective at extracting common features between source and target domains compared to the MLP. We further compared the role of the loss${\mathrm{l}}_{\mathrm{t}\mathrm{t}\mathrm{m}\mathrm{m}\mathrm{d}}$ during the training; the results indicate that reducing the difference between the targets in the source-target branches could further improve the recognition accuracies for both cross-session and cross-subject scenarios. This suggests that finding common features between the source and target domains is crucial.

(3) Hyperparameters: To further evaluate the generalization of the proposed DD-MSDA, we calculate the classification accuracies using different hyperparameters. It can be seen in Figure 3 that reducing the batch size leads to a slight improvement in accuracy, but the performance does not change significantly, suggesting that DD-MSDA has stronger generalization and overfitting prevention capabilities. Moreover, reducing the learning rate decelerates the model convergence rate and leads to a decrease in performance. Figure 4 illustrates the curves of the five losses including total loss, source-target loss, target-target loss, classification loss and variance loss. It demonstrates that all five losses tend to converge as the number of iterations increases, indicating that the feature distribution of the target and source domains become more similar with the iteration; thus, it is guaranteed that the decoding model can be efficiently migrated from the source domain to the target domain.

4.4. Cross-Dataset EEG-Based Emotion Recognition Results

In the cross-dataset experimental study, we conduct experiments on two datasets, SEED and DEAP, which differ significantly in terms of data collection. Considering that emotions in the DEAP dataset are rated on four number scales, we discretize and categorize the dimensional emotion space in DEAP into positive, neutral and negative categories according to SEED, as conducted in [15]. Fourteen subjects in DEAP who commonly reported the three categories of emotions are selected for the following cross-dataset study. In SEED, all trials are truncated to 185 s. It is worth noting that DEAP only contains 32 channels of EEG data, whereas SEED has 64 channels of EEG data. In the cross-dataset experiment, we use exclusively the same 32-channel EEG data on SEED as the DEAP dataset. Six experiments were conducted to evaluate the cross-dataset emotion recognition performance: SEED1-DEAP, SEED2-DEAP, SEED3-DEAP, DEAP-SEED1, DEAP-SEED2, and DEAP-SEED3. Symbol A-B indicates that dataset A is the source domain and dataset B is the target domain. SEED1-3 represents the data of sessions 1–3 in SEED, respectively. In the first three experiments, the source training data contributed by SEED contains 2775 × 15 = 41,625 samples, and in the last three experiments, the source training data contributed by DEAP contains 180 × 14 = 2520 samples.

Table 4. Cross-dataset classification accuracies (Mean ± Std (%)).

Method	SEED1-DEAP	SEED2-DEAP	SEED3-DEAP	DEAP-SEED1	DEAP-SEED2	DEAP-SEED3
TCA	39.02 (5.37) *	36.95 (6.00) *	39.10 (4.72) *	37.33 (3.24) *	36.23 (2.13) *	38.14 (2.90) *
DDC	50.56 (6.58) *	50.48 (5.70) *	51.31 (4.87) *	49.34 (5.68) *	46.64 (6.79) *	48.19 (8.64) *
DAN	50.99 (5.80) *	50.04 (5.18) *	49.17 (6.33) *	48.57 (4.96) *	46.32 (6.80) *	47.63 (6.80) *
DCORAL	50.00 (5.40) *	51.23 (4.13) *	52.57 (5.46) *	46.58 (4.88) *	45.40 (3.40) *	46.17 (5.10) *
MS-MDA	49.47 (10.35) *	48.78 (9.92) *	55.68 (12.99) *	53.79 (6.87) *	52.36 (7.50) *	53.33 (10.64)
DD-MSDA	58.31 (10.52)	57.39 (9.81)	61.16 (11.62)	56.67 (6.59)	55.43 (7.35)	56.98 (11.91)

* denotes the accuracies are statistically lower than DD-MSDA (Wilcoxon signed-rank test, p < 0.05).

shows the comparison results of the six cross-dataset experiments; the accuracy is averaged from all the subjects in the target domain. Statistical significance was assessed using the paired Wilcoxon signed-rank test (p < 0.05). It can be seen that our proposed DD-MSDA achieves the highest accuracies in all six experiments, indicating that it can not only effectively improve the cross-dataset emotion recognition performance but also have high generalization ability. It should be noted that the average cross-dataset accuracy of 57.66% is significantly above the random chance level of 33.3%. This demonstrates that DD-MSDA captures meaningful emotional patterns beyond random guessing.

5. Discussion

In this study, we use the DD net as a common feature extractor for the source and target domains, calculate multiple losses to narrow the feature representations between the source and target domains, and find the common features suitable for the target domains, so as to improve the classification performance of cross-domain emotion EEG recognition. The classification performance of the proposed DD-MSDA algorithm is validated in three cross-domain experiments using three public emotion EEG datasets. The results indicate that DD-MSDA achieves average accuracies of 91.15 ± 7.81% and 87.57 ± 7.43% in cross-session and cross-subject experiments on the SEED dataset, while the obtained accuracies on the SEED-IV dataset are 75.57 ± 11.71% and 74.79 ± 10.83%, respectively. Furthermore, DD-MSDA achieves an average accuracy of 57.66% in the cross-dataset recognition task based on the DEAP and SEED datasets. We also select several state-of-the-art algorithms for comparison, and the results show that DD-MSDA achieves the highest accuracy in all three cross-domain recognition experiments. These consistent experimental results demonstrate that DD-MSDA is effective in addressing the EEG temporal drift, the inter-subject feature distribution differences, and feature differences caused by EEG acquisition devices, emotional stimulus materials, and experimental protocols.

The ablation experiments show that the DD net employed in the proposed algorithm is beneficial for cross-domain emotion recognition. The improvement observed when replacing MLP with DD Net in the common feature extractor can be attributed to multiple factors. First, DD Net’s controllable logical expression capacity helps mitigate overfitting [51]. Second, DD Net’s white-box nature and multiplicative interaction mechanism may enable more effective extraction of domain-invariant features [52]. Third, as demonstrated in prior work, DD Net exhibits superior generalization capability across nine real-world applications [19]. While the ablation study confirms the overall effectiveness of replacing MLP with DD Net, we acknowledge that the performance gain is not exclusively attributable to overfitting reduction alone. Importantly, when the training dataset is sufficiently large, the advantage of DD Net’s constant logical expression over MLP may diminish, as MLP’s stronger fitting capacity can be fully utilized without severe overfitting. However, in the small-sample regimes typical of EEG-based cross-domain emotion recognition, where each subject contributes only hundreds to thousands of samples, DD Net’s inherent regularization effect becomes particularly advantageous. This also suggests that researchers may adjust the order of the DD Net according to specific experimental requirements to balance commonality and individuality in feature extraction.

During the experimental process, training the model with branches from all source domains can result in high time complexity. In addition, blindly increasing the number of source domains does not necessarily lead to an improvement in classification performance, and it may even produce a negative transfer effect when there is a substantial feature distribution difference between the source and target domains [53]. In future work, it would be beneficial to design a method to identify a “golden source domain subject” group that is particularly suitable for target domain subjects, similar to [54] which employed a Top-h ranked domain classifier selection (TOP) module to select source domain. Such approaches can reduce the number of source domains and significantly reduce the time required for training. When appropriate domain weights are assigned to these selected source domains, it is possible to further improve the performance of cross-domain emotion recognition.

Compared with the DG-based transfer learning methods, one drawback of the proposed DD-MSDA is that the model has to be trained when the EEG data of a new subject is known. Although the proposed DD-MSDA still requires a training phase when EEG data of a new subject arrives, this process holds strong practical value for online applications for two reasons. First, it does not require any labeled target data; only a short unlabeled calibration segment is needed to align the feature distributions, which is more realistic than collecting labeled emotional data online. Second, the model is lightweight and converges quickly, allowing rapid adaptation before the formal online session. In future work, we will further explore pre-training and fine-tuning strategies to reduce this latency and achieve near-instant adaptation, making it even more suitable for real-time emotion recognition.

A limitation of this study is that not all comparison methods could be fully re-evaluated under an identical preprocessing and evaluation protocol due to the diversity of implementations and experimental settings in the literature. This may partially affect the strictness of the quantitative comparison. The development of standardized benchmarks and open-source evaluation pipelines for cross-domain EEG emotion recognition would be highly valuable for fairer and more transparent comparisons in the future.

6. Conclusions

We propose a multi-source domain multi-representation adaptive algorithm based on dendrite net to solve the decoding model transfer problem in electroencephalogram (EEG) emotion recognition. It mainly consists of three modules including the dendrite common feature extractor, domain-specific feature extractor, and domain-specific classifier. The dendrite net helps to extract common features between the source and the target domains, and the feature distribution difference between the source and target domains can be further narrowed by minimizing the four losses in the domain-specific feature extractor and domain-specific classifier. Subsequently, we conducted comparative experiments with state-of-the-art algorithms, and the results show that the proposed algorithm average cross-session and cross-subject accuracies are 91.15% and 87.57% on SEED, and 75.57% and 74.79% on SEED-IV, respectively, along with an average cross-dataset accuracy of 57.66%. They are all the best performances among EEG-only domain adaptation methods across all three cross-domain scenarios.