EEG Emotion Recognition Based on Federated Learning Framework

Abstract

Emotion recognition based on the multi-channel electroencephalograph (EEG) is becoming increasingly attractive. However, the lack of large datasets and privacy concerns lead to models that often do not have enough data for training, limiting the research and application of Deep Learn (DL) methods in this direction. At present, the popular federated learning (FL) approach, which can collaborate with different clients to perform distributed machine learning without sending data to a central server, provides a solution to the above problem. In this paper, we extended the FL method to the field of emotion recognition based on EEG signals and evaluated its accuracy in the DEAP and SEED datasets, where the model accuracy reached 90.74% in our framework. We also divided the DEAP dataset into different clients. The accuracy of emotion recognition decreased by 29.31% compared to the FL method when the clients were trained using local data, which validates the necessity of the FL approach for emotion recognition tasks. In addition, we verified the impact of N-IID data on the accuracy of FL training. The experiment demonstrated that N-IID leads to a 14.89% decrease in accuracy compared to IID.

1. Introduction

Emotion recognition is a hot issue in human–machine interaction systems [1,2], and its accuracy directly affects the user’s interaction experience. At the same time, its application is beneficial for diagnosing diseases such as depression [3], tracking patients’ recovery effects and assisting psychology in studying emotional behavior. Emotion recognition can be based on behavioral signals such as actions and expressions [4,5] and physiological signals such as EEG and ECG [6,7]. Among them, EEG signals provide direct measurements of signals generated by the human nervous system, which is the most direct, reliable and accurate way to reflect human emotional behaviors, and therefore is widely used in emotion recognition.

In recent years, with the advancement of EEG acquisition technology, EEG signals’ temporal resolution and post-processing techniques have been significantly improved, which has laid a solid foundation for applying deep learning techniques to process EEG signals for emotion recognition [8,9]. Due to the temporal asymmetry and instability of EEG signals, poor signal-to-noise ratio and the existence of variability among different brain regions and individuals [10], it poses many challenges for EEG-based emotion recognition. Traditional machine learning methods categorize this task as a classification problem, using algorithms such as decision tree [11], multilayer perceptron [12] and support vector machine [13]. Due to the complexity of EEG signal recognition and categorization, traditional algorithms often have the problem of inaccurate or even unclassifiable classification. Deep learning networks have made significant progress in classification problems in medical image processing, computer vision and other fields due to their strong generalization ability and automatic learning of abstract features [14,15]. They have also achieved advantages far beyond traditional algorithms in the problem of emotion recognition based on EEG signals [16,17]. Deep learning algorithms represented by convolutional neural networks still have problems. Compared with image classification problems that can be processed locally, the classification of EEG signals has discrete characteristics in time and space, making it difficult for traditional convolutional neural networks to achieve higher accuracy rates. Acharya [18] achieved a recognition rate of 87.72 by increasing the number of layers of convolutional neural networks. Rudakov [19] proposed extracting features using differential entropy and power spectra density methods to fit EEG signals’ characteristics further and achieve 96.28% recognition accuracy, which has more value for EEG emotion recognition.

The increase in accuracy of the above emerging EEG emotion recognition algorithms comes at the cost of higher system parameter complexity and the number of training sets, i.e., a sufficient amount of training data is required for the models to be trained to achieve clinical-level accuracy. Since physiological signals such as EEG can reflect the data collector’s personality traits, emotional tendencies and other brain activity characteristics, the development of deep learning has heightened the privacy leakage risks that may arise from data misuse in terms of privacy protection [20]. Governments have started to pay extensive attention to data security and privacy protection issues, with the EU and China adopting the General Data Protection Regulation [21] in May 2018 and the Data Security Law of the People’s Republic of China in June 2021, respectively, requiring the use of personal data to be subject to the consent of the data owner.

Compared with traditional picture data in the field of deep learning, EEG data are subject to technical limitations leading to a more difficult acquisition of EEG data [22], and the high time domain signal-to-noise ratio of EEG data and other characteristics make clean EEG data that can be used for deep learning model training more challenging to obtain. As a result, the EEG data available to various medical institutions are insufficient to train emotion recognition models with high accuracy and robustness under increasingly stringent data security and privacy protection regulations. This makes the research and clinical application of deep learning methods in EEG more challenging and becomes one of the urgent problems to be solved.

In 2017, to address the issues of privacy preservation and data silos in devices, Google proposed federated learning (FL) [23], which provides us with a solution to the above problems. Federation learning is essentially a distributed machine learning method, which abandons the traditional way of training models with centralized data and allows data between devices to participate in training without going out of local by uploading training models for aggregation through local training, and this method protects data privacy.

FL has gained the attention and focus of many researchers in the healthcare field, where data privacy is highly valued because it has the potential to address trust and privacy issues arising from the sensitivity of patient data. Brisimi [24], in 18, proposed an FL model capable of predicting hospitalizations of cardiac patients using inter-institutional EHR data. In order to enable institutions with small amounts of Melanoma Detection data to collaborate in training the model and to address the problem of poor data availability, Agbley [25] extended the FL approach to Melanoma Detection for disease detection. Although the FL algorithm achieves to protect data privacy, it is still possible to attack the uploaded models to obtain data information. Therefore, Malekzadeh [26] proposed an FL system based on differential privacy stochastic gradient descent (DPSGD) and secure aggregation to improve the security of FL further. Moreover, with the explosion of COVID-19, the FL-based approach for COVID-19 detection also attracted the interest of many researchers, among which Feki [27] combined blockchain and FL approaches to achieve cross-institutional co-training of COVID-19 detection models using CTs of COVID-19 patients. At the same time, Zhang [28] designed a dynamic fusion of FL systems to improve the communication efficiency of the FL algorithm for COVID-19 detection accuracy. The application environment of the FL algorithm presents a key data challenge: the distribution of medical data held between different medical institutions is usually non-independently and identically distributed (N-IID). Such N-IID among data has been shown to cause substantial accuracy degradation in traditional image classification domains such as CIFAR-10 or MNIST. However, the N-IID problem of the FL algorithm in EEG data has not been taken seriously [29].

In this study, we expected to use the FL approach to address the problems of insufficient data volume and complexity faced by research and clinical applications in this field, enabling institutions with small datasets to train machine learning models collaboratively. Our contributions mainly include the following:

1.

This paper extends the FL method to EEG signal-based emotion recognition field and evaluates its accuracy in the DEAP and SEED datasets. Our validation shows that the FL method can lead to higher model accuracy;

2.

We constructed different DEAP datasets for evaluating the effect of the diversity of training data on emotion recognition models. It was verified that the accuracy of the emotion recognition model using EEG signals is highly dependent on subjects and that increasing the diversity of subjects can substantially improve the model’s generalization performance, demonstrating the need for the FL method to be applied in this domain;

3.

The impact of the FL method on the accuracy and convergence speed of the emotion recognition model when trained on EEG data with N-IID distribution was evaluated by simulating the N-IID distribution of the inter-client DEAP dataset. Compared with the IID distribution, there is a substantial decrease in the accuracy of the FL-trained emotion recognition model under the N-IID distribution.

2. Materials and Methods

The federated learning framework for training emotion recognition models is shown in Figure 1. The server distributes the emotion recognition models to the clients (individuals or medical institutions) with EEG data participating in the FL task. The client participating in the task processes the data locally into the data format required for training the FL task to train the sentiment recognition model, ends the training after the specified number of training rounds, and uploads the locally trained model to the server. The server aggregates the models uploaded by the client and determines whether to continue the FL training. The methods used in the framework are described in this section.

2.1. Electroencephalography—Emotion Recognition Dataset

The DEAP dataset is a multimodal dataset contributed by Koelstra [30]. This dataset recorded the EEG and peripheral physiological signals of 32 participants. In the experiment, 32 subjects watched music videos of 1 min in length as required, for a total of 40 videos per subject. The experimenters recorded 32 channels of EEG and 8 channels of peripheral physiological information simultaneously using a BioSemi EEG cap according to the international 10–20 standard. Subjects scored each video on four dimensions: validity, arousal, dominance, likeness and familiarity, as required. During pre-processing, the raw 512 hz EEG signal was downsampled and filtered to 128 hz to remove artifacts such as EOG and muscle movement.

The scoring and EEG data of 32 subjects were saved in two formats, MATLAB.mat and Python.dat. The data file corresponding to each subject contains two arrays: data and labels. Data has a data dimension of 40*40*8064, each video saves 40 channels of data, and each channel has 8064. The dimension of data is 40*40*8064, each video has 40 channels of data, and each channel has 8064 saved electrical signals; the dimension of labels is 40*4, and the scores of four aspects were recorded: valence, arousal, dominance, likeness and familiarity. In addition, frontal face clips of 22 subjects were recorded and saved in the face_video.zip file.

The SEED dataset is an emotion classification dataset provided by the BCMI lab of Shanghai Jiao Tong University [31]. The SEED dataset records EEG signals of 15 subjects through 62 channels of electrode caps with a sampling frequency of 1000 Hz according to the 10–20 standard. There were 5 movie clips for each emotion, for a total of 15 movie clips. The SEED dataset provides EEG data down sampled to 200 Hz with a 0.5–75 bandpass filter, which is stored in a .mat file in the Preprocessed_EEG folder, and the corresponding labels are stored in a label.mat file (−1 means negative, 0 means neutral, 1 means positive).

2.2. Signal Pre-Processing

We use fast Fourier transform (FFT) for feature extraction to transform the data relative to time variation into a spectrogram relative to frequency variation, making the model converge faster while having a higher accuracy rate. The sample dimension of the processed EEG signal data becomes (1, 70). These extracted features include five frequency bands: Theta-θ (4–8 Hz), Alpha-α (8–14 Hz), Beta-β (14–30 Hz) and Gamma-γ (31–50 Hz) [32].

FFT is a more efficient and faster computational method for the discrete Fourier transform (DFT) [33], which can be used to transform the signal domain from time to frequency. The use of this algorithm enables the computer to compute the discrete Fourier transform with a much-reduced number of multiplications, thus reducing the computational time as well as the computational complexity.

Since the fluctuations of emotional states are mainly concentrated in the 14 channels [34] in

Table 1. Channel Selection.

Channel	DEAP Index	Seed Index	Channel	DEAP Index	Seed Index
AF3	1	3	AF4	17	4
F3	2	7	F4	19	11
F7	3	5	F8	20	13
FC5	4	15	FC6	21	21
T4	7	23	T8	25	31
P7	11	41	P8	29	49
O0	13	58	O2	31	60

, we selected these channels for training, which can reduce the computational cost of this study and does not affect the accuracy of the model. The time window was set to 2 s, and the update step was updated every 0.125 s.

2.3. Emotion Recognition Model

Convolutional neural network (CNN) is a deep learning algorithm that can automatically learn spatial features in data samples without human extraction. Its advantages of high accuracy and high generalization ability in deep learning compared to traditional machine learning algorithms have made it widely used in the direction of image classification. Therefore, in this study, a standard CNN model was chosen as a machine learning model for evaluating the completion of the emotion classification task in FL methods. The network structure of this model is shown in Figure 2, which mainly consists of three one-dimensional convolutional neural networks, two pooling layers and three fully connected layers, and finally, the lined SoftMax layer was used for emotion classification.

In the experiments of this study, we input the data of shape (70*1) into this network with a convolutional kernel size of 5 and a step size of 1. After the convolutional network operation, the output was subjected to maximum pooling. The pooling layer has a pooling window of 2 and a step size of 1. Going through the pooling layer reduces the dimensionality of the information extracted from the convolutional layer and reduces the computational effort. After the third convolutional layer, the output was pulled into 1 dimension through the Flatten layer and sent to the fully connected layer for further training. All layers in this network use the ReLU function as the activation function, and the output data were normalized to a standard normal distribution with mean 0 and variance 1 by batch normalization to accelerate the model training.

2.4. Federated Learning Algorithm

This section introduces the basic composition structure, model training process and parameter updating methods of federated learning algorithms. Federated learning is a distributed machine learning algorithm that protects data privacy. The standard federated learning framework works together to train high-quality global models through the client, the server and the aggregation framework. In this process, the model obtained by the server through aggregation is usually called the central model, while the model distributed by the server to the client and trained locally is called the local model.

Usually, the FL training process consists of the following three steps:

Step 1: Initialize the FL task. The server determines the FL training task, i.e., the training target of FL task and the data requirements needed for training. Then the FL task is released to request clients who meet the conditions to join the FL task, and the global model and training parameters of the FL task are determined according to the computing power, communication and data volume constraints of the clients joining the training task. The server then distributes the initialized global model and training requirements to the clients participating in the task [35];

Step 2: Execute the FL task to achieve model training. The client uses the local data to train the issued global model W and update the model parameters according to the training requirements, which are usually minimized loss functions. After completing the training, the client uploads the updated local model to the server. The server waits for all clients to upload the local model and then obtains the new global model by weighting the parameters of the local model by the aggregation algorithm to obtain a weighted average;

Step 3: End of FL task. The updated global model is tested, and the FL task is ended when the global model performance meets the task goal. If the model performance does not meet the task target, the global model is resent to the client for parameter update waiting for a new round of model aggregation and testing by repeating step 2.

In the above FL training process, the client usually holds the training data and provides the computational resources needed for the model to be trained locally. The server is usually the commander of a node with reliable computational power, responsible for sending the global model to the client, receiving the local model returned by the client, and implementing the aggregation in the server, in addition to managing the communication between the client and itself. The aggregation algorithm is the core of the FL framework. The local model is updated into a new global model by aggregation algorithm, and the global model obtained by the aggregation algorithm should have good accuracy and generalization performance.

The aggregation algorithm used in this study is federated averaging (FedAvg), which performs a weighted average of the local model parameters for each client. The goal of Fedavg is usually to minimize the loss function of all samples, which can be expressed as Equation (1):

$${\omega }^{\prime }=argminL\left(\omega \right)$$

(1)

where ${\omega }^{\prime }$ denotes the main model parameters and the $L\left(\omega \right)$ function denotes the global loss function. ${\omega }^{\prime }$ parameters are updated by the gradient uploaded by each client for optimization, which usually requires some training rounds. The server distributes the global model to client i. The client i calculates the local gradient $\nabla L_i\left({\omega }_i\right)$ locally using the SGD algorithm and updates the local model with the learning rate λ. The expression of the model update is expressed as follows:

$${\omega }_i\left(t+1\right)={\omega }_i\left(t\right)-\lambda \nabla L_i\left[{\omega }_i\left(t\right)\right].$$

(2)

The client can be trained locally several times before participating in aggregation. After the server gives the aggregation command and completes local training, the clients upload their model parameters to the server, and then the aggregator in the server aggregates the model. The aggregator uses the FedAvg method for aggregation as follows:

$$\omega \left(t+1\right)={\displaystyle \sum }_{i=1}^N\frac{n_i}{n}{\omega }_i\left(t+1\right).$$

(3)

Here, N is the number of clients participating in federated learning, $n$ denotes the total amount of data from clients participating in aggregation, and $n_i$ denotes the amount of data owned by client i. The server aggregates the local models uploaded by each client based on the number of samples from each party with a weighted average of the model parameters to obtain the global model $\omega \left(t+1\right)$ for the next round. The above method can be used to achieve model aggregation at the server after local training of models by the clients in the process of step 2. The training stops when the global loss function, accuracy, or the number of training rounds reaches a threshold, and the global loss function and accuracy are two critical metrics for the same number of training rounds. Finally, this classical federation learning is applied in this study and experimentally proved to have better results.

3. Results

3.1. Experimental Setup

All our experiments were conducted on a Windows 10 computer with an AMD R7–4800H CPU and a GTX1650Ti GPU, and the experimental code was implemented in Python 3.7. We divided each subject’s data in the dataset into 80% training set and 20% test set, and the training data were equally distributed to all clients involved in the federation learning. In contrast, the test set was kept in the server measurement, and the accuracy of the aggregated model was tested and recorded after the completion of the aggregation. In our experiments, we set the optimizer for model training to Adam, the learning rate to 0.001 and the batch size to 2048, and each round of aggregation required the clients to train 1 epoch locally.

3.2. Experimental Results

In this study, we validated the accuracy of the FL framework on the DEAP and SEED datasets. In the experiments, the SEED dataset label is a tri-categorization task (negative, neutral, positive), and the DEAP data label is the subjects’ self-assessment of four feelings of arousal, valence, liking and dominance with scores ranging from 1 to 9. In order to make comparing experimental results from different datasets more intuitive, we divided the DEAP labels into four for the triple classification task by using the scores of 4 and 7 in the DEAP dataset labels as the threshold.

Table 2. Accuracy performance of different training methods on the four emotions of DEAP dataset and SEED dataset.

Method	DEAP-Valence	DEAP-Arousal	DEAP-Dominance	DEAP-Liking	SEED
1 client	0.8569	0.8638	0.8793	0.8582	0.8225
FL-5 clients	0.8843	0.8974	0.9074	0.8719	0.8572
FL-10 clients	0.8794	0.8870	0.9039	0.8687	0.8463

records the tested accuracies of 1 client (without the FL method), the FL method with 5 clients, and the FL method with 10 clients when training 200 Epochs. The experiments show that the FL algorithm of five clients obtains higher accuracy when training 200 Epochs compared to the FL method without FL, where the classification accuracy of DEAP-Dominance reaches 90.74%

Figure 3 shows the variation curves of the loss function of the FL algorithm with the different number of clients when trained on the DEAP dataset with different labels and the SEED dataset. The total amount of training data is the same, but it is evident that the rate of loss decline of the FL algorithm receives the influence of the number of clients when training on the DEAP dataset. The convergence of the model leveled off at 60 epochs; the convergence rate gradually decreases with the increase in the number of clients involved in federal learning, and the FL model converges to level off only at 200 epochs of aggregation when 10 clients are used.

We simulated different training datasets for comparison experiments with the following experimental setup to verify the effect of training source diversity on model generalization performance.

Since real healthcare organizations usually have all the data of one or some subjects, the DEAP dataset is a sentiment classification dataset with 32 subjects. Therefore, we took the overall data of one subject (i.e., 1/32 = 3.125% of the data) as the base, reconstructed the DEAP dataset into the following four different training sets, and assumed that they are owned by four different medical institutions (A, B, C, D):

A. Data owned by two subjects from s01 to s02 (6.25% of DEAP);

B. Data owned by subjects from s01 to s16 (50% of DEAP);

C. Data owned by subjects from s017 to s32 (50% of DEAP);

D. Data owned by subjects from s01 to s32 (100% of DEAP).

In order to exclude the effect of sample size on the accuracy, we replicated the datasets of medical institution A 16 times and the datasets of medical institutions B and C twice, thus ensuring that each medical institution uses the same number of training samples. Finally, each medical institution uses the training data to train the emotion recognition model.

The models were all tested in a test set composed of the complete DEAP subject data, and the experimental data are presented in Figure 4. In the experiments with the label Arousal, the accuracy was 36.25% when training with s01 to s02 subjects and 58.63% when training with s01 to s16 subjects, which was much lower than the accuracy of 87.94% when using all the data. This trend is also shown in the other three labels.

Figure 5 shows the experiments show that the model is tested with an accuracy of up to 91.28% on the data sourced from the training set, but the same model is only 58.82% on the test set composed of all the subjects’ data. This highlights the specificity of EEG data, where each person has some variation in the EEG signal produced in response to external stimuli, and emotion recognition models trained using deep learning methods need to use a large amount of different data to ensure that the model meets the model robustness required for clinical care.

Since the sample collection for emotion recognition is based on the subjective thoughts of the subject or patient, the data distribution of the EEG dataset for emotion recognition available among medical institutions is mostly N-IID. Therefore, we validated the performance variation in the FL algorithm under the N-IID data distribution. In order to ensure intuitive experimental results, we classified the DEAP data according to 1~9 and sampled the DEAP dataset, retaining 10,000 data samples for each label, for a total of 90,000 data samples. Moreover, these data are assigned to different clients using the Dirichlet distribution to achieve the simulated N-IID distribution [36], and the client data distribution is shown in the following

Table 3. The data distribution of the clients after distributing the data using the Dirichlet distribution is shown with the columns indicating the data categories and the horizontal columns indicating the clients. The values in the table represent the data for each category in the different clients.

	Class	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7	Class 8	Class 9	Distribution
Client
Client 1		1	1028	921	34	58	384	755	20	8249
Client 2		34	748	9	85	2708	358	15	576	50
Client 3		154	3036	4	1358	515	211	150	4665	0
Client 4		1008	54	243	1072	1120	278	142	784	83
Client 5		0	329	142	3979	3	639	654	816	1587
Client 6		2101	466	882	1415	12	185	3687	277	0
Client 7		2428	3069	6329	0	0	0	0	0	0
Client 8		397	471	633	121	3	4531	1934	607	31
Client 9		3386	785	355	496	726	3153	57	2255	0
Client 10		491	14	482	1440	4855	261	2606	0	0
All Data		10,000	10,000	10,000	10,000	10,000	10,000	10,000	10,000	10,000

. As a comparison experiment, we equally distributed the resampled DEAP dataset to all clients to simulate the IID distribution.

Figure 6 shows the rising accuracy curves of the FL algorithm for N-IID and IID distributions, where all four labels produce a severe accuracy drop in the N-IID distribution, and an average accuracy decrease of 14.89% can be obtained from

Table 4. The accuracy of the models obtained by training on DEAP datasets with IID and N-IID distributions, respectively.

Method.	Valence	Arousal	Dominance	Liking
IID	0.9200	0.9102	0.9222	0.9286
N-IID	0.7633	0.7590	0.7999	0.7633

.

The convergence rate of the FL algorithm at the data distribution of N-IID, the model’s training, converges only after 450 rounds of aggregation, and the convergence curve at N-IID is more tortuous than the IID distribution when the model has converged after 200 rounds of aggregation.

The FL approach allows the model to achieve higher accuracy when using EEG for emotion classification.

Table 5. Comparison with state-of-the-art techniques using the same DEAP EEG dataset for the emotion recognition task.

Method	Valence	Arousal	Dominance	Liking
Luo et al. [37]	0.7400	0.7800	0.8000	0.8627
Acharya et al. [18]	0.8507	0.8383	0.8143	0.8574
Nawaz et al. [38]	0.7896	0.7762	0.7760	/
Topic et al. [39]	0.7772	0.7661	/	/
Galvao et al. [40]	0.8984	0.8983	/	/
FL-10 Client	0.9200	0.9102	0.9222	0.9286

summarizes the comparison of 600 Epoch trained with 10 clients with other state-of-the-art automatic emotion classification techniques, with our approach generally achieving higher accuracy.

4. Discussion

In recent years, the use of machine learning methods to extract and identify emotional features in EEG [41]. Since the training and validation of models in machine learning methods are highly data-dependent, it makes the research and clinical applications of machine learning methods in EEG fields such as emotion recognition, where there are few publicly available data sets, complex data collection and strict data privacy requirements, more difficult due to the lack of data quantity. For this reason, we investigated federated learning methods in EEG-based emotion recognition to address this problem and evaluate the effectiveness of federated learning methods and the new problems they pose. Our results support our expectation that by using a federated learning approach, owners of EEG sentiment data can use the data to jointly train a global model without sharing the data they have and that the global sentiment recognition models trained by this approach have high accuracy.

Unlike traditional machine learning models that are trained directly in the data to gain experience, federal learning aggregates multiple local models in such a way that the global model gains the experience learned by the local models during training, which makes the process of convergence of the global model affected by multiple local models at the same time. In order to evaluate the variability in the training emergence of emotion recognition models caused by the federal learning approach compared to the traditional machine learning approach, we first designed experiments under the data distribution of IID. We evaluated them on the DEAP and SEED datasets. From the experimental results presented in Section 3, it is clear that using the federation learning method with the same amount of data leads to higher accuracy but may affect the convergence rate of the model. We believe this is because the global model of federal learning learns the local model experience by averaging the gradients of the local model’s parameters, so the global model’s convergence rate is not affected by the number of local models involved in the aggregation. In contrast, when the number of global data is fixed for federation learning, as the number of clients increases, each client has fewer data, and the local model learns less experience after one round of epoch, so the convergence speed becomes slower as the number of clients increases, even though the number of data involved in federation learning is the same.

In order to verify the necessity of federation learning, we reconstructed four DEAP training and testing subsets with the same amount of data according to the source of data collection using the DEAP dataset and show the emotion recognition ability possessed by the model after training under different subsets in Section 3. The results show that the more complex the source of training data is when training a machine learning model, the stronger the classification ability and robustness of the model. This demonstrates that many healthcare organizations with EEG data can obtain more accurate and robust emotion classification models if they can share their data to form more complex datasets for training. Thus, federal learning methods that can accomplish sub-goals become very relevant in the face of data privacy constraints.

Although this study demonstrates that FL methods can solve the problem of insufficient data samples when studying machine learning algorithms in the EEG domain, our accuracy evaluation of FL tasks with N-IID data sample distribution among clients also demonstrates the shortcomings of the current FL algorithms. Experiments show that the accuracy of the global model when the FL algorithms are trained on N-IID data samples is substantially higher when compared to data. The accuracy of the global model decreases significantly when the FL algorithm is trained on a sample of N-IID data compared to the data distribution of IID. However, the distribution of EEG data among institutions with small EEG data must be N-IID, and there are still few studies to address the accuracy degradation of FL algorithms in medical data when facing N-IID data distribution. We believe this issue is one of the problems that must be solved for clinical applications of FL algorithms in EEG, so we plan to investigate this in our future work.

5. Conclusions

In response to the current deep learning research and clinical applications based on EEG signal data receive insufficient data size in public datasets, and EEG data cannot be used mutually between different institutions. This study introduces federated learning into the field of emotion recognition. The method allows for the aggregation of training models by having the client upload the training models so that data between devices can participate in the training without going out of the local area, avoiding the problem of data privacy leakage. The FL method decouples between emotion recognition models, and any emotion recognition model or algorithm can be trained together using the FL method for multiple devices. The effectiveness and superiority of the method for EEG signal emotion recognition are validated on two different tasks. Experiments show that the FL method exhibits higher accuracy, with a 2.75% improvement in accuracy compared to traditional centralized training on a single device. The need for diverse data sources and the necessity of federal learning for emotion recognition models based on EEG data is also demonstrated through experiments using subjects with different DEAP datasets. Finally, N-IID experiments with simulated data distributions show that training an emotion recognition model using the FL method produces a 14.89% decrease in accuracy when faced with N-IID data, a problem that needs to be addressed.