EEG-Based Emotion Recognition Using Deep Learning and M3GP

Round 1

Reviewer 1 Report

Authors conducted this research in the title of "EEG-based emotion recognition using deep learning and M3GP". The paper’s subject could be interesting for readers of journal. Therefore, I recommend this paper for publication in this journal but before that, I have a few comments on the text that should be addressed before publication:

Comments:

1) Abstract: In line 4 of this section authors used this word “We”. Words like “We”, “Me”, “Our” or “Us” are not common in article writing. Other words could be used by the authors. For example, this sentence "our work presented a new model in this work" could be replaced by this "A new model is presented in this work".

2) In Abstract section authors did not mention the main goal of this research. In other words, there is no obvious words about the main question that this research is managed to answer it in this section. If it is added, it would be really helpful for readers to enter and understand main purpose of this research. Furthermore, this would be useful for readers of this article to compare this work with similar works conducted in recent years.

3) Figure 2: The title of this figure is too long. Authors could explain more after or before this figure. The title of figures or tables should be as short as possible and also they should be clean and obvious.

4) Which software has been used in this work to modelling data and export the results?. Also, which software has been used in this work to export the diagrams in this work?. For example, software like MATLAB could be used for modelling and software like SigmaPlot could be utilized to export charts and diagrams.

5) Conclusion: In the conclusion section authors should mention more words about their suggestions to future works. It really can be helpful for future studies and works related with title of this article. For example, authors can mention some words about data base, software, number of indicators and etc.

6) Figure 1 : The space between titles of chart axis and the related axis is too short and they are too close. Authors should increase the space between them. This correction makes this chart looks better and prevents misleading.

7) Abstract : In the line 5 of this section authors mentioned this word "BED" without any previous definition and explanation. Every abbreviation should be defined before it is used in the article. It really would be helpful for readers and users of this work.

8) Since recently it has been proved that artificial intelligence (AI) and machine learning has a numerous applications in all of engineering fields, I highly recommend the authors to add some references in this manuscript in this regard. It would be useful for the readers of journal to get familiar with the application of AI in other engineering fields. I recommend the others to add all the following references, which are the newest references in this field

[1] Sattari, M.A., Roshani, G.H., Hanus, R. 2021. Applicability of time-domain feature extraction methods and artificial intelligence in two-phase flow meters based on gamma-ray absorption technique. Measurement, 168, p.108474.

[2] Reddy, A.P. and Vijayarajan, V., 2020. Audio compression with multi-algorithm fusion and its impact in speech emotion recognition. International Journal of Speech Technology, 23(2), pp.277-285.

[3] Roshani, M., Sattari, M.A., Ali, P.J.M., 2020. Application of GMDH neural network technique to improve measuring precision of a simplified photon attenuation based two-phase flowmeter. Flow Measurement and Instrumentation, 75, p.101804.

Author Response

1) Abstract: In line 4 of this section authors used this word “We”. Words like “We”, “Me”, “Our” or “Us” are not common in article writing. Other words could be used by the authors. For example, this sentence "our work presented a new model in this work" could be replaced by this "A new model is presented in this work". 2) In Abstract section authors did not mention the main goal of this research. In other words, there is no obvious words about the main question that this research is managed to answer it in this section. If it is added, it would be really helpful for readers to enter and understand main purpose of this research. Furthermore, this would be useful for readers of this article to compare this work with similar works conducted in recent years. 7) Abstract : In the line 5 of this section authors mentioned this word "BED" without any previous definition and explanation. Every abbreviation should be defined before it is used in the article. It really would be helpful for readers and users of this work.
Response for 1, 2 and 7.
We replace the abstract to include your suggestions:
This paper addresses a proposal to recognize emotional states through EEG analysis. The novelty of this work resides in the feature selection strategy based on Multiclass Genetic Programming with Multidimensional Populations (M3GP) that improves the feature selection process by implementing an evolutionary technique that selects the most suitable features for a classification process. In this way, we will have more control of the search space by reducing the number of features and using only those features that best define each class. Before and after implementing the M3GP, the results showed an increment of 14.76\% in the recognition rate. The tests were performed on a Biometric EEG Dataset (BED), designed to evoke emotions and record the cerebral cortex's electrical response; this dataset implements a low-cost device to collect the EEG signals, allowing greater viability for the application of the results. The model presented archives up to 92.1\% classification rate. The results are competitive with state-of-the-art features and can simplify the feature selection process by increasing the separability of the spectral features 3) Figure 2: The title of this figure is too long. Authors could explain more after or before this figure. The title of figures or tables should be as short as possible and also, they should be clean and obvious. The description was made simpler and hope that clean too.
The research subjects' responses were distributed according to an arousal and valence scale. Higher diameter circles indicate a higher sample frequency of a class. 4) Which software has been used in this work to modelling data and export the results?. Also, which software has been used in this work to export the diagrams in this work?. For example, software like MATLAB could be used for modelling and software like SigmaPlot could be utilized to export charts and diagrams.
We added the computational and software details to the materials and methods section as:
The computational aspects and software are: Python Scikit Learn libraries, Python 3.7.6, numpy 1.18.5, pillow 8.3.1, imutils 0.5.4, tensorflow gpu 1.15.0, keras 2.2.4, matplotlib 3.1.1, scipy 1.2.0. Intel i7 4gen, 3.6 Ghz, 16 RAM, windows 10 pro, Nvidia Gforce 1080.
5) Conclusion: In the conclusion section authors should mention more words about their suggestions to future works. It really can be helpful for future studies and works related with title of this article. For example, authors can mention some words about data base, software, number of indicators and etc.
We extend the conclusion section by adding an overview of the dataset and including future works.
The results in this research show that the proposal significantly impacts the classification rate. An essential aspect of this work is that we decided to use a novel database built from its conception focused on studying users' emotional reactions, unlike many of the databases that have been conditioned or given off of projects outside the field of study. In addition, this BED explores the use of low-cost devices in a formal study, promoting and establishing some basis for applicability in less controlled environments.
Our future work proposal is based on integrating the feature selection process to the network architecture configuration, that is, implementing the neat GP concept in the generalization and optimization of network operation. Up to our best knowledge, we know that DL has shown outstanding performance when working with EEG signals; however, the community continues to propose architectures that can efficiently solve this type of problem. We hypothesize that we can establish the criteria for the networks' features and topology when implementing an evolutionary strategy. Another proposal for future work linked to this model lies in implementing GP as an AutoML methodology, with the premise that through GP, it autonomously selects the type of features (currently, this process is performed through digital signal processing by extracting spectral coefficients) and DL topology. 6) Figure 1 : The space between titles of chart axis and the related axis is too short and they are too close. Authors should increase the space between them. This correction makes this chart looks better and prevents misleading. The image was replaced in the paper.
8) Since recently it has been proved that artificial intelligence (AI) and machine learning has a numerous applications in all of engineering fields, I highly recommend the authors to add some references in this manuscript in this regard. It would be useful for the readers of journal to get familiar with the application of AI in other engineering fields. I recommend the others to add all the following references, which are the newest references in this field.
The papers has been reviewed, thanks for the recommendations.

Author Response File: Author Response.pdf

Reviewer 2 Report

Decision: Major Revision

1. The abstract of the paper should briefly explain the summary of this work. In the said paper authors should add some motivation and working of their proposed algorithm to avoid confusion for readers. Similarly, authors should add numerical improvements in the abstract against the SOTA.
2. What are the authors' scientific contributions? The hybrid of several DL models is not innovative enough. Many studies have proposed such hybrid DL models. Would you mind clarifying them?
3. How many look-ahead signals, i.e., the emotion horizon, do the authors consider in this research? In addition, the authors must report and analyze the emotional performance in different look-ahead signals.
4. In addition, the authors must provide a sufficient critical review of the literature to indicate the drawbacks of existing approaches and then define the main focus of the research direction. How did those previous studies perform? Specifically, what methodology did they use? Which problem still requires to be solved? Why is the proposed approach suitable for solving the critical problem? Readers need more positive reviews of the literature to indicate the state-of-the-art development.
5. The manuscript, however, does not link well with recent literature on recognition appeared in relevant top-tier journals, e.g., the IEEE Intelligent Systems department on " att-net: Enhanced emotion recognition system using lightweight self-attention module". Also, new trends of AI for recognition “mlt-dnet: recognition using 1D dilated CNN based on multi-learning trick approach” are missing it should be comprised.
6. In Section 3, the authors must introduce their proposed research framework more effectively. For example, the authors could consider some essential brief explanation compared to the text with a total research flowchart or framework diagram for each proposed algorithm to indicate how these employed models are working to receive the experimental results. It is not easy to understand how the proposed approaches work.
7. The authors must perform exploratory data analysis on the datasets.
8. How were the hyperparameter values of the DL architecture chosen? Were they through a grid search? If they were through a grid search, the authors could improve the table by providing the ranges of different hyperparameters they have considered for a grid search.
9. Explain the reasons that the suggested approach provides better performance compared to other previous models? I wonder if the proposed method can be applied to other regions with different mechanical systems and occupant profiles.
10. The readability and presentation of the study should be further improved. The paper suffers from language problems.

Author Response

1. The abstract of the paper should briefly explain the summary of this work. In the said paper authors should add some motivation and working of their proposed algorithm to avoid confusion for readers. Similarly, authors should add numerical improvements in the abstract against the SOTA. Thank you for your comments, we improve the redaction of the abstract: This paper addresses a proposal to recognize emotional states through EEG analysis. The novelty of this work resides in the feature improvement strategy based on Multiclass Genetic Programming with Multidimensional Populations (M3GP) that builds feature by implementing an evolutionary technique that selects, combines, deletes and constructs the most suitable features to ease the classification process of the learning method. In this way, the problem data can be mapped in to a more favorable search space that best define each class. After implementing the M3GP, the results showed an increment of 14.76\% in the recognition rate without changing any setting in the learning method. The tests were performed on a Biometric EEG dataset (BED), designed to evoke emotions and record the cerebral cortex's electrical response; this dataset implements a low-cost device to collect the EEG signals, allowing greater viability for the application of the results. The model presented archives up to 92.1\% classification rate. The results are competitive with state-of-the-art features and can simplify the feature management process by increasing the separability of the spectral features. 2. What are the authors' scientific contributions? The hybrid of several DL models is not innovative enough. Many studies have proposed such hybrid DL models. Would you mind clarifying them? 3) In addition, the authors must provide a sufficient critical review of the literature to indicate the drawbacks of existing approaches and then define the main focus of the research direction. How did those previous studies perform? Specifically, what methodology did they use? Which problem still requires to be solved? Why is the proposed approach suitable for solving the critical problem? Readers need more positive reviews of the literature to indicate the state-of-the-art development. Hope these responses the 2 and 3 observations: The novelty of our proposal lies in applying the ML M3GP tool to convert the search space through selecting, combining, deleting, and constructing the most suitable features to ease the classification process of the learning method. This process allows us to improve the classification stage without changing the architectures, which are often obtained through extensive processes. The literature shows that the community has proposed diverse EEG data processing architectures; some use large and complex neural networks since their features require it. Others argue that low complexity architectures could handle the problem but use distinct spectral feature extractions. The proposed methodology allows the improvement of the recognition rate for this type of analysis without changing the topology. ** citations are added to the document. We also added a more explanatory diagram in Figure 14. 4 How many look-ahead signals, i.e., the emotion horizon, do the authors consider in this research? In addition, the authors must report and analyze the emotional performance in different look-ahead signals. In this research, we try to generalize the analysis of the signals, although we have worked focusing on searching for these identifiers. At the moment, for this work, they were not
considered since we started with the premise that we would obtain an improvement with generalizable situations. 4 The manuscript, however, does not link well with recent literature on recognition appeared in relevant top-tier journals, e.g., the IEEE Intelligent Systems department on " att-net: Enhanced emotion recognition system using lightweight self-attention module". Also, new trends of AI for recognition “mlt-dnet: recognition using 1D dilated CNN based on multi-learning trick approach” are missing it should be comprised. Thank you for the reference, it it’s a very illustrative work. We use it as a SOA reference. 5 In Section 3, the authors must introduce their proposed research framework more effectively. For example, the authors could consider some essential brief explanation compared to the text with a total research flowchart or framework diagram for each proposed algorithm to indicate how these employed models are working to receive the experimental results. It is not easy to understand how the proposed approaches work. Thanks for your contribution, flow chat was added to section 3 and 4. Hope they help to clarify the methodology. Figure 9 and Figure 15. 6 The authors must perform exploratory data analysis on the datasets. We enhance the description and motivation of the use on this dataset in conclusion section as: The results in this research show that the proposal significantly impacts the classification rate. An essential aspect of this work is that we decided to use a novel database built from its conception focused on studying users' emotional reactions, unlike many of the datasets that have been conditioned or given off of projects outside the field of study. In addition, this BED explores the use of low-cost devices in a formal study, promoting and establishing some basis for applicability in less controlled environments. 7 How were the hyperparameter values of the DL architecture chosen? Were they through a grid search? If they were through a grid search, the authors could improve the table by providing the ranges of different hyperparameters they have considered for a grid search. The network configuration was chosen based on previous publications, mainly at https://doi.org/10.1177/1460458216661862 , which shows the configurations and the architecture chosen. Some features such as optimization stages have also been updated, which are described at https://doi.org/10.3390/app10051736. We made changes in the section to make it more straightforward where the decisions for the network configuration can be consulted. 8 Explain the reasons that the suggested approach provides better performance compared to other previous models? I wonder if the proposed method can be applied to other regions with different mechanical systems and occupant profiles. We have investigated in different databases; however, we can say that we chose this dataset since databases such as SEED or DEAP are built-in intensely controlled
environments and with highly controlled equipment. (We have tried to purchase them, but it is not easy to get it if you’re outside of certain regions), so using data obtained with commercial equipment is an excellent opportunity to generalize this type of experiment. We added a reference to these comments in the conclusions and discussions sections. 9 The readability and presentation of the study should be further improved. The paper suffers from language problems We perform a complete review of the document's grammar. Thank you for your comments, and we hope we have met your expectations.

Author Response File: Author Response.pdf

Reviewer 3 Report

This study aimed to recognize emotional states through EEG.  I have the following major suggestions.

1. The abstract should be rewritten and improved by combining the objectives, short methodology, main findings, numerical form of results, and prospective application.
2. What is the novelty of this proposed method?

3. Introduction section needs to be improved. ML/DL based classification studies should be studied in mental workload and cognitive impairment and emotion recognition. Authors should explore state-of-art classification studies in mental workload, disease prediction, mentioning the references, doi:10.1109/ACCESS.2020.3040437, doi:10.3390/brainsci11070900. Please write down the contribution of the study at the end part of the Introduction section in bulleted form.
4. Authors must present the summarized dataset used in this study in a table.

1. Authors should explain the feature sets used in this study. In addition, the procedure of feature extraction method should be demonstrated in detail. What is the size of the epoch in feature extraction?

6. How did the authors deal with dataset class imbalance challenges?
7. How did authors confirm that their models were not overfitted? Authors should report cross-validation procedures and results.
8. Why authors used deep-learning approach (M3GP) for classification? Machine-learning model are quite suitable for this kind of classification problem. Reference can be improved by doi:10.1109/ACCESS.2021.3109806, doi:10.3390/s21216985. Both training and testing confusion matrices need to be shown.

1. Authors should report the specification of hardware (PC) performed this study.
2. The authors need to mention the model parameters or hyperparameters. The performance of the deep-learning model is dependent on the selection of the architecture and/or parameters.

11. Both training and testing performance measures of classifiers are to be shown for DL models.
12. Authors should present the training and validation accuracy graphs of the proposed model with changes in the number of epochs.
13. From the writing point of view, the manuscript needs to be checked for typos and the grammatical issues should be improved.

Author Response

1.- The abstract should be rewritten and improved by combining the objectives, short
methodology, main findings, a numerical form of results, and prospective application.
We rewrite the abstract to include the recommendations.
This paper addresses a proposal to recognize emotional states through EEG analysis. The
novelty of this work resides in the feature improvement strategy based on Multiclass Genetic
Programming with Multidimensional Populations (M3GP) that builds features by implementing
an evolutionary technique that selects, combines, deletes, and constructs the most suitable
features to ease the classification process of the learning method. In this way, the problem
data can be mapped into a more favorable search space that best defines each class. After
implementing the M3GP, the results showed an increment of 14.76\% in the recognition rate
without changing any setting in the learning method. The tests were performed on a Biometric
EEG dataset (BED), designed to evoke emotions, and record the cerebral cortex's electrical
response; this dataset implements a low-cost device to collect the EEG signals, allowing
greater viability for the application of the results. The model presented archives up to 92.1\%
classification rate. The results are competitive with state-of-the-art features and can simplify
the feature management process by increasing the separability of the spectral features.
2.- What is the novelty of this proposed method?
We include both in the abstract and the introduction sections a reference to our contribution;
specifically, in the introduction section, we attach a paragraph in which we define our proposal
and the novelty it refers to for this type of work:
The novelty of our proposal lies in applying the ML tool M3GP to convert the search space
through selecting, combining, deleting, and constructing the most suitable features to ease
the classification process of the learning method. This process allows us to improve the
classification stage without changing the architectures, which are often obtained through
extensive processes.
3.- Introduction section needs to be improved. ML/DL based classification studies
should be studied in mental workload and cognitive impairment and emotion
recognition. Authors should explore state-of-art classification studies in mental
workload, disease prediction, mentioning the references,
doi:10.1109/ACCESS.2020.3040437, doi:10.3390/brainsci11070900. Please write down
the contribution of the study at the end part of the Introduction section in bulleted
form.
Thank you very much for your recommendations; without a doubt, the monitoring proposal is
something that I had not been able to explore, and it helps me expand my references. This
also helps me justify low-cost proposals.
4.- (LMD) Authors must present the summarized dataset used in this study in a table.
The raw dataset is explained in 2.2 Dataset of the BED.
In the results section we have added the transformation tree that was generated by M3GP to
improve the original dataset.
5.- (LMD)Authors should explain the feature sets used in this study. In addition, the
procedure of feature extraction method should be demonstrated in detail. What is the
size of the epoch in feature extraction?
The feature set and epoch(generations) and all evolutionary settings are reported in table 6
M3GP setup. The M3GP procedure is described in 2.4 Features transformation by M3GP
[22]. The feature set can be reduced to less and simpler functions, with the side effect of
limiting the data treatment to build new features.
6.- (lmd)How did the authors deal with dataset class imbalance challenges?
In the standalone M3GP section, we recommend to use all available data without any
previous treatment, to let the evolutionary process of M3GP guide the search to build the
transformation tree.
7.- (LMD) How did authors confirm that their models were not overfitted? Authors
should report cross-validation procedures and results.
Figure 13 is added to clarify the DL overfitting criteria, regarding M3GP in the standalone
M3GP section and table 7 shows training and testing with a high level of overfitting but the
main goal of M3GP is not to classify but to improve the dataset.
8.- (LMD) Why authors used deep-learning approach (M3GP) for classification?
Machine-learning model are quite suitable for this kind of classification problem.
Reference can be improved by doi:10.1109/ACCESS.2021.3109806,
doi:10.3390/s21216985. Both training and testing confusion matrices need to be
shown.
We agree with the reviewer, that's why we use the M3GP, which is a ML method, but the
classification results were not significant, on the other hand the generated features have a lot
of potential shown in the result after M3GP. Training and testing confusion matrices were
added Figure 10 to 14.
9.- Authors should report the specification of hardware (PC) performed this study.
The specifications were added to section 2 as The computational aspects and software are:
Python Scikit Learn libraries, Python 3.7.6, NumPy 1.18.5, pillow 8.3.1, imutils 0.5.4,
TensorFlow gpu 1.15.0, keras 2.2.4, matplotlib 3.1.1, scipy 1.2.0. Intel i7 4gen, 3.6 Ghz, 16
RAM, Nvidia Gforce 1080.
10.- (Adrian DL settings)The authors need to mention the model parameters or
hyperparameters. The performance of the deep-learning model is dependent on the
selection of the architecture and/or parameters.
11.- Both training and testing performance measures of classifiers are to be shown for
DL models.
The confusion matrix of the training score is presented in Figure 14.
12.- Authors should present the training and validation accuracy graphs of the
proposed model with changes in the number of epochs.
The cross-validation test and score accuracy were added in figure 13.
13.- (LMD) From the writing point of view, the manuscript needs to be checked for
typos and the grammatical issues should be improved.
We thank the reviewer, spelling, and typos were checked to the author’s capabilities.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

All the comments have been addressed correctly.

Reviewer 2 Report

Fully addressed 

Reviewer 3 Report

Thanks for addressing the comments.