Review and Classiﬁcation of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review

: Recent developments and studies in brain-computer interface (BCI) technologies have facilitated emotion detection and classiﬁcation. Many BCI studies have sought to investigate, detect, and recognize participants’ emotional affective states. The applied domains for these studies are varied, and include such ﬁelds as communication, education, entertainment, and medicine. To understand trends in electroencephalography (EEG)-based emotion recognition system research and to provide practitioners and researchers with insights into and future directions for emotion recognition systems, this study set out to review published articles on emotion detection, recognition, and classiﬁcation. The study also reviews current and future trends and discusses how these trends may impact researchers and practitioners alike. We reviewed 285 articles, of which 160 were refereed journal articles that were published since the inception of affective computing research. The articles were classiﬁed based on a scheme consisting of two categories: research orientation and domains/applications. Our results show considerable growth of EEG-based emotion detection journal publications. This growth reﬂects an increased research interest in EEG-based emotion detection as a salient and legitimate research area. Such factors as the proliferation of wireless EEG devices, advances in computational intelligence techniques, and machine learning spurred this growth.


Introduction
A Brain Computer Interface (BCI) is a system that takes a biosignal, measured from a person, and predicts (in real-time) certain aspects of the person's cognitive state [1,2].At the outset, BCIs started as assistive technological solutions for individuals with significant speech anomalies.However, the research was rooted in a subject's desire to communicate through either speech or writing or to control his or her immediate environment.BCI systems have also used computer-based recreational activities to stimulate a subject's innate ability to overcome physical disabilities.
Today, BCI-based research has been expanded to include people with and without physical disabilities.The entire system underscores how adaptive systems can enhance analytic methods and application areas.This assistive ability has created a widespread awareness among potential users and researchers alike.In the past 15 years, the increasing numbers of BCI research groups, peer-reviewed During our literature review, we observed that articles on emotion recognition cut across various disciplines, including clinical neurology, rehabilitation, neurobiology, engineering, psychology, computer science, medical physics, and biomedical engineering.Hence, conducting a comparative analysis of articles is difficult since different journals and different scientific domains have different research focuses and methodologies.
Accordingly, the main objective of this review is to classify and summarize research that is relevant to emotion recognition systems and to provide conceptual frameworks for integrating and classifying emotion recognition articles.This system of classification will be useful for literature reviews on emotion recognition research.
The following sections illustrate our proposed classification framework for emotion recognition literature reviews based on research articles.In Section 2, we outline our research methodology.Section 3 describes our proposed classification framework for emotion-recognition-based literature reviews.Section 4 presents our discussion.In Section 5, we provide insights for future research and discuss the challenges and trends in EEG-based emotion recognition.Finally, in Section 6, we present the study's conclusions.

Research Methodology
Articles on emotion recognition systems are scattered across journals of various disciplines and were found in both medical and non-medical journal publications, including clinical neurology, rehabilitation engineering, neurobiology, engineering, psychology, computer science, medical physics, and biomedical engineering.
We searched Web of Science (WoS), https://webofknowledge.com, to obtain a comprehensive bibliography of the academic literature on emotion-recognition-based BCI.The Web of Science Core Collection database provides us with quick, powerful access to the world's leading citation databases, such as Science Direct (Elsevier), IEEE/IEE Library, ACM Digital Library, Springer Link Online Libraries, and Taylor & Francis.
The following subsections describe the procedure we followed in extracting articles, along with our article selection criteria and filtering processes.During our literature review, we observed that articles on emotion recognition cut across various disciplines, including clinical neurology, rehabilitation, neurobiology, engineering, psychology, computer science, medical physics, and biomedical engineering.Hence, conducting a comparative analysis of articles is difficult since different journals and different scientific domains have different research focuses and methodologies.
Accordingly, the main objective of this review is to classify and summarize research that is relevant to emotion recognition systems and to provide conceptual frameworks for integrating and classifying emotion recognition articles.This system of classification will be useful for literature reviews on emotion recognition research.
The following sections illustrate our proposed classification framework for emotion recognition literature reviews based on research articles.In Section 2, we outline our research methodology.Section 3 describes our proposed classification framework for emotion-recognition-based literature reviews.Section 4 presents our discussion.In Section 5, we provide insights for future research and discuss the challenges and trends in EEG-based emotion recognition.Finally, in Section 6, we present the study's conclusions.

Research Methodology
Articles on emotion recognition systems are scattered across journals of various disciplines and were found in both medical and non-medical journal publications, including clinical neurology, rehabilitation engineering, neurobiology, engineering, psychology, computer science, medical physics, and biomedical engineering.
We searched Web of Science (WoS), https://webofknowledge.com, to obtain a comprehensive bibliography of the academic literature on emotion-recognition-based BCI.The Web of Science Core Collection database provides us with quick, powerful access to the world's leading citation databases, such as Science Direct (Elsevier), IEEE/IEE Library, ACM Digital Library, Springer Link Online Libraries, and Taylor & Francis.
The following subsections describe the procedure we followed in extracting articles, along with our article selection criteria and filtering processes.

Data Sources and Procedures for the Extraction of Articles
We searched our selected database (Web of Science Core Collection database) for articles over a span of eleven years, 2005-2016.We used Basic Search to look for topics that fall into our research scope.Using basic search settings, we input search terms and phrases, such as: affective or emotion; emotion detection or recognition; EEG or Electroencephalography; and, Brain-computer interface, Passive BCI or BCI.According to Web of Science search result templates, auto-generated search terms are a result of searches covering articles, meeting abstracts, book chapter(s), and proceedings papers.
The initial search resulted in 766 articles.Table A1 in Appendix A shows a broad categorization and distribution of our search results.We further refined our search based on some predetermined criteria.The following section illustrates our selection criteria.

Selection Criteria
Three criteria were used to select and accept emotion recognition articles for further review.Articles were excluded if they did not meet the following selection criteria:

•
Articles must address one of the Gartner Hype Cycle 2016 trending research topics.To meet this criterion, they must be relatively current.In this regard, we chose articles that were published between 2005 and 2016.This 11-year period could be considered to correspond to the main research period of emotion recognition systems.

•
We excluded meeting abstracts, book chapters, conference proceedings, workshop descriptions, masters and doctoral dissertations, and non-English articles.Notably, the number of conference papers in this domain was 322, the number of book chapters was 3, and the number of meeting abstracts was 6.

•
We also ensured that only peer-reviewed journal articles were included.The logic behind this is that practitioners and academics frequently use journals to both obtain and spread research findings.Thus, journal articles contain the highest level of research.

Filtering/Reviewing Process
In this step, each article was manually reviewed in three rounds.This was done to eliminate non-emotion-recognition-based works and non-BCI-based works.After that step, we classified the filtered articles according to our classification scheme.
During our first round of review, we excluded articles based on our predetermined selection criteria.We excluded all articles that did not address EEG-based emotion recognition systems.After applying our selection criteria, we were left with 435 articles.We directly imported all of the articles into an online EndNote database that facilitated, managed, assessed, and reviewed our result articles.The remaining articles were then scanned and filtered, as described in the next subsection.
Our second round of review involved the manual scanning of titles, abstracts, authors, keywords, and conclusions.This round of review excluded articles in which the central theme of the discussion was centered on subjects other than emotion recognition systems, with emotion recognition systems only being mentioned in passing.By the end of this round, we were left with 340 articles.
The final round involved reading the full texts and analyzing each article according to our classification scheme.This scheme is described in the next section.The infographic in Figure 2 shows the procedure used to filter and extract articles that meet our predetermined criteria.

Classification Method
To systematically reveal and examine research insights on EEG-based emotion recognition, a literature classification scheme was developed.This classification scheme was based on categorizing the research focuses of the 285 selected articles that remained after the filtering processes.A graphical representation of these categories and subcategories and their relationships is presented in Figure 3.

Classification Method
To systematically reveal and examine research insights on EEG-based emotion recognition, a literature classification scheme was developed.This classification scheme was based on categorizing the research focuses of the 285 selected articles that remained after the filtering processes.A graphical representation of these categories and subcategories and their relationships is presented in Figure 3.

Classification Method
To systematically reveal and examine research insights on EEG-based emotion recognition, a literature classification scheme was developed.This classification scheme was based on categorizing the research focuses of the 285 selected articles that remained after the filtering processes.A graphical representation of these categories and subcategories and their relationships is presented in Figure 3.

Application Domain and Field
EEG-based emotion detection applications can be categorized into two broad domains: Medical and Non-medical.They are further described in the following subsections.

Medical Context
EEG emotion recognition is medically contextual if the system is designed to provide assistance, enhancement, monitoring, assessment, and diagnosis of human psychiatric and neurological diseases.

Non-Medical Context
Includes emotion recognition systems based on EEG signals that are designed to entertain, educate, monitor, or play games.

Results and Discussion
We extracted 285 articles on emotion recognition systems from 29 online databases with 160 different journals.Each article was reviewed and classified according to our classification scheme.Although the extensity of our search was limited, it offers a comprehensive insight into EEG-based emotion recognition system research.Furthermore, we provided a descriptive overview of temporal trends of these types of publications, along with a description of publication domains (e.g., publication research area, online database and journal) in Appendix B.

Classification of Articles by Paper Orientation
When we classified journal articles according to paper orientation, we found 20 review articles and 265 experimental design innovation articles.

Review Paper
We found twenty review papers.Seventeen of the articles were narrative reviews, and three were systematic reviews.We classify these papers according to their focus: general/background, signal processing, classification, application, and others.We found seven general reviews, including neuroimaging techniques, emotion modeling, and applications.We also found three signal processing reviews, one classification method, seven application reviews, and two reviews that covered the training protocols, validity in the EEG-neurofeedback optimal performance field, and evidence of neurofeedback learning.Table 1 shows the review article classifications.The objectives of the earliest reviews (2007 and 2008) were to understand the mirror neurons; cover the EEG correlates of emotions, brain rhythms, and functions; and, present a framework and suggestions for a research program [3,12].We also found two published review articles that aim mainly to highlight an emerging field, such as the neuroscience of culture in [9] and social neuroscience in [10].
Other review articles were published in medical fields.These articles focused on EEG applications and how to use EEGs to diagnose and assess medical conditions.These articles also explored the relationships between symptoms and affective states, such as schizophrenia [17,18], depression [20], disorders of consciousness [19], and autism [21].

Design Innovation (Experimental) Paper
A number of design innovation papers (265) were published in the period of 2005-2016.Technical aspects of design and design implications are important, and there are still many technological challenges and obstacles to the growth of EEG-based emotion detection systems.Therefore, information was extracted from each study on the following aspects: affective states investigated, emotion elicitation method, number of participants, acquisition technique, and emotion recognition techniques, including feature extraction, type of classifier, performance of the classifier, and online (real-time) vs. offline classification.
Emotion model: Emotions are traditionally classified on the basis of two models: the discrete and dimensional models of emotion.Dimensional models of emotion propose that emotional states can be accurately represented as combinations of several psychological dimensions.Most dimensional models incorporate valence and arousal.Valence refers to the degree of 'pleasantness' that is associated with an emotion, whereas arousal refers to the strength of the experienced emotion.Discrete theories of emotion propose the existence of small numbers of separate emotions, as characterized by coordinated response patterns in physiology, neural anatomy, and morphological expressions.Six basic emotions that are frequently specified in research papers are happiness, sadness, anger, disgust, fear, and surprise.
Another emotion model is the appraisal model.The appraisal model of emotion is based on the evaluation of currently remembered or imagined circumstances.Basically, the appraisal model proposes that thought precedes emotion and that emotion precedes behavior.The majority of the papers (172 articles, 64.91%) clearly specify that they used a dimensional model, whereas (34 articles, 12.83%) used a discrete model; and, 21.89% of articles used a different model or did not specify which emotion model they used.
Emotion elicitation technique: Different emotion elicitation techniques have been developed and reported.Critical examples of emotion elicitation techniques include standardized emotional stimuli (e.g., pictures, films, and audio), imagination techniques (e.g., guided imagery and autobiographic recall), present social interactions (e.g., games), and directed facial action tasks.We found that researchers have used different methods to elicit target emotions.In addition, other researchers have found that the use of multiple stimuli is more effective in eliciting emotions than the use of single stimuli.Table 2 shows the numbers of articles that used a combination of different emotion elicitation techniques.When using elicitation techniques to evoke emotions, the emotional stimuli (image, audio, and video) are usually taken from reputable sources, such as the International Affective Picture System (IAPS) database and the International Affective Digitized Sounds (IADS).In addition, other databases and video clips can be collected from various resources on the internet (e.g., YouTube and Facebook).Other modalities, such as the recall paradigm, where the subject is asked to repeatedly recall emotional instances from their life, and dyadic interaction, where a facilitator helps to induce various emotions, are also used by researchers.
Single/multimodality: Recently, many studies have shown that combinations of modalities can significantly enhance the emotion recognition accuracy in comparison with single modalities.For example, combining eye movements, which are measured using an eye tracking method, and EEG can considerably improve the performance of emotion recognition systems [38,39].Moreover, in [40], the researchers proposed a multi-modal emotion recognition system using four sensing methods: EEG, heart inter-beat interval, galvanic skin response, and stressor level lever.The purpose of this study was to measure the human stress response during using a powered wheelchair.It also provided a comprehensive background for multi-modal emotional state and proposed a frame work and discussed the feasibility and utility of using the multi-modal approach.In [8], the researchers used multimodal physiological signals EEG, Galvanic skin response, blood volume pressure, respiration pattern, skin temperature, electromyogram, and electrooculogram to classify and predict depression.They proved the potential of the multimodal approach and achieved 85.46% accuracy using support vector machine (SVM).The work in [41] proposes an approach for multi-modal video-induced emotion recognition, based on facial expression and EEG technologies.In [42], the researchers proposed an emotion recognition system that used a large number of modalities: galvanic skin response, heart rate, blood volume pulse, electrocardiogram, respiration, facial electromyograph, and EEG, while the subjects were watching the affective eliciting materials to distinguish six different kinds of emotions (joy, surprise, disgust, grief, anger, and fear).They stated that the integration of these methods facilitates a more detailed assessment of human affective state, which also improved accuracy and robustness.In [43], the researchers examined the role of emotional arousal on subsequent memory in school-age children.They used EEG, heart rate, and respiration.Their findings endorsed the value of combining multiple methods to assess emotion and memory in development.
Most previous research into emotion recognition used either a single modality or multiple modalities of different physiological signals.The former method allows for the limited enhancement of accuracy, and the latter has the disadvantage that its performance can be affected by head or body movements.Furthermore, the latter causes inconvenience to the user due to the sensors that are attached to the body.In our review, we found that the majority of the papers (218 articles, 82.3%) used Appl.Sci.2017, 7, 1239 9 of 34 a single modality, namely, EEG signals, as an objective method in their studies, whereas (45 articles, 17%) used multiple modalities.
Recognition model: The analysis and classification of the EEG signal can be performed either online or offline.The performance time begins when features are extracted and ends when classification has been completed.The number of electrodes that are used during experimentation in emotion detection situations imposes time constraints on the algorithms.For example, in [44], the authors built a system that detects current user affective states and obtained a classification accuracy of 65%.In [45,46], they tested their method online and offline.We observed that the majority of the 187 articles mentioned that they were using offline analysis, whereas only 15 articles used online analysis.
EEG device: A number of EEG devices have been used in EEG-based BCI research in medical and academic settings.We found up to 48 different devices from different companies all over the world.In Table 3, we specify the most commercially available and widespread devices that have been used in more than five articles.

EEG Device Number of Articles
Electrodes: When considering the number of electrodes, time interval required to set up an EEG device, comfort level of subjects, system usability, and number of features to be processed, it is advised, from this standpoint, that fewer electrodes be utilized; for example, five channels were used in [176,190].Nonetheless, most current EEG devices still require a relatively large number of electrodes; for example, 64 channels were used in [93,95], and 32 channels were used in [43,71,191].We found that the maximum number of electrodes that were used when recording EEG signals was 257 in [141], whereas the minimum number was one in [192].In this study, the electrode was placed at the Fpz according to the international 10-20 system, and two reference electrodes were located on the left and right ear lobes.Notably, other physiological signals were recorded in this study, including facial electromyogram, electroencephalography, skin conductivity, and respiration data.
Benchmark EEG emotional databases: Only a small number of benchmark emotional EEG databases with categorized emotions are publicly available for use and to test a new system.These datasets are available to the academic community via a web-based system and have been used in a number of research studies to test and explore proposed systems.Table 4 lists the benchmark datasets, along with descriptions and references to the articles that have used them.However, most of the reviewed articles (239 articles, 88.7%) recorded their own EEG signals and used them during their experiments.
Moreover, in [193], the authors evaluated their approach on three benchmark databases: DEAP, MAHNOB-HCI, and USTC-ERVS.USTC-ERVS is an EEG dataset that contains 197 EEG responses to 92 video stimuli from 28 users, along with users' emotional self-assessments.However, the USTC-ERVS dataset is no longer available online.Either its page was moved or the URL was completely expunged from the initial database location.Participants: The number of participants can vary based on the experiment type and field.In our review, we found that the minimum number of participants was one.In [137], an EEG investigation was carried out on a patient with complete cortical blindness who presented with affective blindness.The maximum number of participants was 300 in [42].In this study, EEG-based physiological signals of Chinese college students were recorded to establish a large effective physiological signal database.
Moreover, different age groups ranging from infants [132,143,169], children [6,43,75,84,89,103,125,126,[128][129][130]140,183,184,[210][211][212][213], adolescents [59,65,70,135,139], and elderly [105,214] in different experiments were considered as study samples.Some studies were conducted on a single gender, such as those investigating women's emotions during motherhood and parental status [133,134,164] and a study of men [215].Some studies targeted a specific type of subject, such as healthy people or patients, to investigate and observe the differences in emotions between two different groups, such as control and healthy groups in [76,107,135], groups of women and men in [46,97,177], young adults versus older adults in [87,216], or children versus adults in [126,143].

EEG correlates of emotion (signals):
Numerous research studies have examined neural correlates of emotion in humans.Frontal EEG asymmetry, event-related desynchronization/synchronization, event-related potentials, and steady-state visually evoked potentials have been found to be associated with emotional states.We found that the majority of the 130 articles used event-related potentials, whereas 48 articles used Frontal EEG asymmetry in their analysis, six articles used event-related desynchronization/synchronization, and four articles used steady-state visually evoked potentials.
Emotion types and numbers: Human feelings and emotions can be quantified for various emotional states.However, only certain types of emotions can be recognized using EEG.Moreover, finding key emotional states to be recognized is mandatory; for example, six emotions were detected in [42,45,106,120,122,192], whereas in [108], a real-time EEG signal to classify happy and unhappy emotions was proposed, and in [113], a fear evaluation system was proposed.In our review, we found that most of the articles aim to detect unpleasant, pleasant, and neutral emotions, such as in [105,217], or positive, negative, and neutral emotions that are based on the valence-arousal dimensional emotion model, as in [159,206].
Computational methods to estimate affective state: Various techniques and approaches have been proposed in the literature in the processing steps to estimate the emotional state from the acquisition signals.Reviewing recent publications and comparing computation methods and results was conducted on a sample of the papers collected.The selection was based on the year 2015-2016 as a timeframe for publication to reflect recent trends and methodologies for emotion detection.The computational methods to extract and classify emotional features from EEG are summarized in Table 5.It is noteworthy to mention that a single feature extraction technique is not optimal across all of the applications.Besides, existing signals are not enough for high accuracy feature extraction.Several approaches introduce more features in different analysis domains to capture extra information about the state of the brain [107,117,200,203,213,216,224].Consequently, feature extraction is one of the major challenges in designing BCI systems; it is determined based on the features and on the appropriate transformation.Although the answer to what are the most emotion-relevant EEG features is still under investigation, power features from different frequency bands are still the most popular in the context of emotion recognition.Studies [26,197,218] have shown that power spectral density (PSD) extracted from EEG signals performs well on distinguishing affective states.
Several machine learning algorithms have been used as emotion classifiers, such as support vector machine (SVM), K-nearest neighbors (K-NN), linear discriminant analysis (LDA), random forest, Naïve Bayes (NB) and Artificial Neural Network (ANN).In general, therefore, the choice of which classification algorithm can be used when designing a BCI depends largely on both the type of brain signal being recoded and the type of application that is being controlled.However, SVM based on frequency domain features as power spectral density (PSD) is shown as the most commonly used method.
Recently, Deep learning methods have been applied to the EEG-based emotion recognition filed.In [49], they proposed a deep belief network (DBN) classifier to recognize three categories of emotions (positive, neutral, and negative).Their experimental results show that the DBN models obtain higher accuracy than SVM and K-NN methods.Also, recurrent neural networks have been used in [74,208,229].In [206], Chai et al. proposed auto-encoder-based deep learning method to investigate emotion recognition on the SEED dataset.

Classification of Articles by Application Domain and Field
We further classified the 285 articles that we extracted into areas of real-life application.Notably, we classified them into medical and non-medical fields of application.The following criteria were used to classify the articles under the medical field: • the paper discussed a medical condition (disorder/disease), such as a psychiatric or neurological case; • the participants of the experiment were patients or it involved two groups: one consisting of healthy people, the other of patients; • the experiment was conducted in a clinical setting; and/or, • the paper was directed toward the medical community and suggested a new method for assistance, enhancement, monitoring, or diagnosis using emotion-based recognition.
After evaluating all 285 articles according to the medical classification criteria listed above, we found that 67 articles, or 23.51%, were medical articles.The remaining 218 articles, or 76.5%, were non-medical.Distributions of medical and non-medical articles according to year of publication are presented in Figure 4. We further classified the articles into areas of application within each domain.Figure 5 shows a treemap of the 285 articles, classified according to field and application domain.

Classification of Articles by Application Domain and Field
We further classified the 285 articles that we extracted into areas of real-life application.Notably, we classified them into medical and non-medical fields of application.The following criteria were used to classify the articles under the medical field: • the paper discussed a medical condition (disorder/disease), such as a psychiatric or neurological case; • the participants of the experiment were patients or it involved two groups: one consisting of healthy people, the other of patients; • the experiment was conducted in a clinical setting; and/or, • the paper was directed toward the medical community and suggested a new method for assistance, enhancement, monitoring, or diagnosis using emotion-based recognition.
After evaluating all 285 articles according to the medical classification criteria listed above, we found that 67 articles, or 23.51%, were medical articles.The remaining 218 articles, or 76.5%, were non-medical.Distributions of medical and non-medical articles according to year of publication are presented in Figure 4. We further classified the articles into areas of application within each domain.Figure 5 shows a treemap of the 285 articles, classified according to field and application domain.

Medical Applications
In the medical field, EEG-based emotion detection systems are used for assisting, monitoring, enhancing, or diagnosing patients' affective states.These medically adapted EEG systems are also used to analyze different types of neurodevelopmental disorders.Some of these neurodevelopmental disorders affect the memory, emotion, learning ability, behavior, and communication of persons

Classification of Articles by Application Domain and Field
We further classified the 285 articles that we extracted into areas of real-life application.Notably, we classified them into medical and non-medical fields of application.The following criteria were used to classify the articles under the medical field: After evaluating all 285 articles according to the medical classification criteria listed above, we found that 67 articles, or 23.51%, were medical articles.The remaining 218 articles, or 76.5%, were non-medical.Distributions of medical and non-medical articles according to year of publication are presented in Figure 4. We further classified the articles into areas of application within each domain.Figure 5 shows a treemap of the 285 articles, classified according to field and application domain.

Medical Applications
In the medical field, EEG-based emotion detection systems are used for assisting, monitoring, enhancing, or diagnosing patients' affective states.These medically adapted EEG systems are also used to analyze different types of neurodevelopmental disorders.Some of these neurodevelopmental disorders affect the memory, emotion, learning ability, behavior, and communication of persons

Classification of Articles by Application Domain and Field
We further classified the 285 articles that we extracted into areas of real-life application.Notably, we classified them into medical and non-medical fields of application.The following criteria were used to classify the articles under the medical field: • the paper discussed a medical condition (disorder/disease), such as a psychiatric or neurological case; • the participants of the experiment were patients or it involved two groups: one consisting of healthy people, the other of patients; • the experiment was conducted in a clinical setting; and/or, • the paper was directed toward the medical community and suggested a new method for assistance, enhancement, monitoring, or diagnosis using emotion-based recognition.
After evaluating all 285 articles according to the medical classification criteria listed above, we found that 67 articles, or 23.51%, were medical articles.The remaining 218 articles, or 76.5%, were non-medical.Distributions of medical and non-medical articles according to year of publication are presented in Figure 4. We further classified the articles into areas of application within each domain.Figure 5 shows a treemap of the 285 articles, classified according to field and application domain.

Medical Applications
In the medical field, EEG-based emotion detection systems are used for assisting, monitoring, enhancing, or diagnosing patients' affective states.These medically adapted EEG systems are also used to analyze different types of neurodevelopmental disorders.Some of these neurodevelopmental disorders affect the memory, emotion, learning ability, behavior, and communication of persons

Classification of Articles by Application Domain and Field
We further classified the 285 articles that we extracted into areas of real-life application.Notably, we classified them into medical and non-medical fields of application.The following criteria were used to classify the articles under the medical field: • the paper discussed a medical condition (disorder/disease), such as a psychiatric or neurological case; • the participants of the experiment were patients or it involved two groups: one consisting of healthy people, the other of patients; • the experiment was conducted in a clinical setting; and/or, • the paper was directed toward the medical community and suggested a new method for assistance, enhancement, monitoring, or diagnosis using emotion-based recognition.
After evaluating all 285 articles according to the medical classification criteria listed above, we found that 67 articles, or 23.51%, were medical articles.The remaining 218 articles, or 76.5%, were non-medical.Distributions of medical and non-medical articles according to year of publication are presented in Figure 4. We further classified the articles into areas of application within each domain.Figure 5 shows a treemap of the 285 articles, classified according to field and application domain.

Medical Applications
In the medical field, EEG-based emotion detection systems are used for assisting, monitoring, enhancing, or diagnosing patients' affective states.These medically adapted EEG systems are also used to analyze different types of neurodevelopmental disorders.Some of these neurodevelopmental disorders affect the memory, emotion, learning ability, behavior, and communication of persons medical).

Medical Applications
In the medical field, EEG-based emotion detection systems are used for assisting, monitoring, enhancing, or diagnosing patients' affective states.These medically adapted EEG systems are also used to analyze different types of neurodevelopmental disorders.Some of these neurodevelopmental disorders affect the memory, emotion, learning ability, behavior, and communication of persons suffering from these conditions.The most common cases are Schizophrenia, Autism, Depression, Huntington's disease (HD), and a myriad of psychiatric and neurological diseases.

Monitoring
These articles outline the performances of patients who are monitored during emotion processing to understand the deficits in emotion and cognition.The data are used to understand neural activity and correlates of emotions in patients with different conditions/disorders using different stimulus types.[5,52,54,73,75,81,98,99,107,114,121,122,124,126,156,181,211,[238][239][240] Other These are review studies that report previous research efforts.They serve as avenues for increasing the awareness of EEG emotional response studies.They also increase the awareness of how EEG may be used in clinical practice to uncover potential neurophysiologic abnormalities.
[ [17][18][19]21] Generally, neurological studies seek to understand how defects in neurobiological processes result in problems associated with neural functioning.These studies are conducted with the goal of understanding how individual differences in brain structure and function influence affective, cognitive, and behavioral outcomes.
Different approaches have been proposed, and several research groups have developed EEG-based BCI systems that aim to detect these affective states.Examples of medical applications are hereunder identified.
Depression is a mental disorder that is related to a state of sadness and dejection.It affects the emotional and physical state of a person.Using EEG-based emotion recognition as a diagnostic test for depression produces conclusive results.A number of studies are based on the automated classification of normal and depression-related EEG signals.This proposed automatic classification system could serve as a useful diagnostic and monitoring tool for the detection of depression [20,79,133,135,226,228,231].
Persons with schizophrenia exhibit drab facial expressions and rarely show positive emotions.These abnormal personality traits may impact social functioning and communication.Whether these deficits reflect an aberrant sensory anomaly, an inability to retain information in their memory, or a dysfunctional integration of these two functions remains unclear.However, studies have shown that sensory processing and memory functioning may be affected in schizophrenic patients [5,17,18,54,62,66,72,73,90,94,98,180,233,236].These studies' experimental protocols were intended to clarify patient deficits in processing emotional faces.
Persons with Parkinson's disease have shown deficits in emotional recognition abilities.Research findings on the reason for this are inconclusive.Nine articles discussed the idea of using EEG-based emotion detection to provide assistance, monitoring, assessment, and diagnosis of Parkinson's disease in patients [47,106,107,114,[118][119][120][121][122].
EEG-based emotion detection systems for cochlear implant patients have been proposed in [156,211,238].EEG signals are used to estimate the ability of these patients to recognize emotional sounds by using EEG signals and different stimuli.Three studies compared the emotional approaches of two groups of study participants.In one group, which was comprised of children with normal hearing, the participants displayed a withdrawal/approach model, whereas the cochlear implant users did not.

Entertainment
These articles observed relationships between multimedia data (music/video) and human emotions.For example, they explored the effects of different types of music on subjects of different ages or genders.In gaming research, some articles sought to detect gamers' affective states to adapt to specific game features, such as the levels of difficulty, punishment, and encouragement.All of these were investigated using EEG-based emotion recognition systems.[34,37,44,53,58,71,88,100,101,110-112, 115,116,151,154,190,193,216,220,222, 276-279] Marketing These articles sought to understand consumer responses toward marketing stimuli by using imaging techniques and recognition of neurophysiological parameters.[102,123,218,280] Education These articles tracked students' engagement and learning.[38,281] Assistance These articles explored how assistive technologies or learning resources were provided to individuals.Thereafter, they were used to identify skills, user experience, and limitations of potential users.They were used to improve behavior, cognition, and emotion regulation.[1,40,95,105,139,194,214,227,282,283] Table 7. Cont.
Recently, a new research area appeared in the marketing field: neuro-marketing.The goal of this new area is to understand consumer responses toward marketing stimuli by using imaging techniques and the recognition of physiological parameters.Because customer feelings in sales areas are strongly influenced by the perception of the surroundings, recognition of emotional responses can reveal true consumer preferences and improve and assist in the (buying) process.Four articles have presented this idea in different applications [102,123,218,280].
Similarly, an EEG-based experimental study [144] is also used to identify the temporal point at which smokers' responses to health warnings begin to differ.Basically, they aim to determine the effects of graphic pictorial cigarette package health warnings by assessing the selective attentional orientation and measuring emotional processing; they reported that smokers are less sensitive to the emotional content of cigarette health warnings.Therefore, future health warning development should focus on increasing the emotional salience of the pictorial health warning content among smokers.
Three research groups have explored EEG emotion detection systems for emotion (stress) monitoring during self-driving of a powered wheelchair [40,227,285].These studies aim to investigate the ability to assist and enhance BCI-based wheelchairs by integrating emotion detection while controlling a wheelchair.
The relationship between music genres and human emotions has been investigated in several recent BCI studies.In these studies, brain signals were recorded using an EEG headset while the subject listens to music [44,53,58,100,110,112,115,116,151,154,190,205,216,220,222,235,276,279].Moreover, the subjects' emotions were recognized as displayed by EEG signals.These signals were then used to tag multimedia data [71,101,193,277].These studies also investigated methods for implicit tagging, wherein users' responses to interactions with the multimedia content are analyzed to generate descriptive tags [71].
Recent BCI research from different disciplines approaches EEG emotion detection and recognition via diverse methods.Some of these methods include feature extraction and selection, machine learning, and pattern recognition methods.These different methods are used to build EEG-based emotion recognition systems.Several research groups (63 articles, 21%) have proposed novel/effective methods for building improved EEG-based emotion recognition systems.Three articles have proposed and described EEG emotion database benchmarks [42,92,101].
Another non-medical application of EEG-based emotion recognition is word processing.Because individual emotional differences impact word processing, differences in interpreting a string of words may elicit different emotional responses.These varying emotional responses are caused by involuntary (implicit) semantic processing, lexical decision tasks (LDTs), and interpretations of perceived positive or negative emotional words [29,31,32,150,160].
Several research groups have explored EEG emotion detection systems as means for monitoring levels of attention and measuring workload [176,188].One important use is to monitor the level of alertness in security-critical tasks such as driving and surveillance.For example, the Air Force Multi-Attribute Task Battery was used as a realistic, ecologically valid multitask environment, in which a participant's workload could be varied [188].
Recently, EEG-based emotion recognition was proposed as a technique that can be used to support a classification task, such as the EEG-based emotional state clustering task [50], image classification [219], and Odor Pleasantness Classification Using Brain and Peripheral Signals [26].
Other non-medical applications of EEG-based emotion recognition include, lie detection applications, security-critical tasks, and driving.For instance, EEG emotion applications are used in untangling criminal cases in legal proceedings and to tell whether an individual is telling the truth [97,139,274,284].
Two articles discussed how EEG-based emotion detection systems could be used to monitor the alertness of humans when performing security-critical tasks.The researchers proposed a real-time low-level attention detection application that can measure a driver's degree of attention [23].Likewise, a driver's emotional and stress levels can be monitored under different conditions [23,283].

Challenges and Future Directions
One challenge with regard to the detection and modeling of emotions in the context of human-computer interaction (HCI), is that it remains complex and requires further exploration.In this context, future research on EEG-based emotion recognition will be explored using the BCI design.From our perspective, Figure 6 provides insights into future research, challenges, and trends in EEG-based emotion recognition.
In general, there are many challenges that are associated with the BCI system.These challenges can be classified as technology related and/or user related.There are differences between the two; however, it is noted that technology-related challenges include, impedance with sensors, system usability, and real-time constraints.When dealing with the device, there are other things to consider, such as the perceived obtrusiveness, information transfer rate, and high error rates.When dealing with user-related challenges, one needs to consider the unfamiliarity of the participating subjects or patients with BCI technologies, discrepancies between ratings, and the duration of setup and preparation.Testing requires assistance from the facilitator in applying the electrodes, which makes the training phase time-consuming.
The BCI design requires multidisciplinary skills from such fields as neuroscience, engineering, computer science, psychology and clinical rehabilitation to achieve the goal of developing an alternative communication and control medium.The number of EEG-based emotion recognition research studies in recent years has been increasing, yet EEG-based emotion recognition is still a new area of research.The effectiveness and efficiency of these algorithms are somewhat limited.Computational methods are being used to estimate emotional state; nevertheless, they can be further improved with technological advancements in order to increase the effectiveness of these algorithms.Some examples of limitations in current algorithms and approaches involve time constraints, accuracy, the number of electrodes, the number of recognized emotions, and benchmark EEG affective databases.
Another limitation is the accuracy and reliability of sensory interfacing and translation algorithms in BCI systems.These factors generally limit the usage of these technologies in clinical settings.Other fields have limitations and challenges as well.For example, engineering challenges focus on the low signal-to-noise ratio in noninvasive electroencephalography (EEG) signals.Moreover, computational challenges include the optimal placement of a reduced number of electrodes and the robustness of BCI algorithms to a smaller set of recording sites.
As we mentioned previously, the challenges currently facing EEG-based BCI systems are two-pronged: technological and user related.We anticipate that some of these challenges will be resolved in the future.For the convenience of description, we have categorized these challenges according to the time that we predict it will take to resolve them: near-term (2-5 years), mid-term (6-9 years), and long-term (10+ years).Although we have categorized these challenges, possible trends and future solutions into three time phases, it is worth noting that some of these anticipated future trends might come earlier, overshoot their time frame, be delayed, or never be achieved.
We anticipate that some of the challenges we have identified may be addressed in 2 to 5 years' time.Some challenges that may be addressed include familiarity of BCI electronic gadgets to study participants or patients; the proliferation of mobile devices' mood and emotion apps; increased multidisciplinary cooperation in BCI systems; and, formulating more accurate and precise definition of the emotion processing in human and neurophysiological studies in EEG correlates of emotion as noted by Kim et al. in [16].
As we mentioned previously, the challenges currently facing EEG-based BCI systems are twopronged: technological and user related.We anticipate that some of these challenges will be resolved in the future.For the convenience of description, we have categorized these challenges according to the time that we predict it will take to resolve them: near-term (2-5 years), mid-term (6-9 years), and long-term (10+ years).Although we have categorized these challenges, possible trends and future solutions into three time phases, it is worth noting that some of these anticipated future trends might come earlier, overshoot their time frame, be delayed, or never be achieved.
We anticipate that some of the challenges we have identified may be addressed in 2 to 5 years' time.Some challenges that may be addressed include familiarity of BCI electronic gadgets to study participants or patients; the proliferation of mobile devices' mood and emotion apps; increased multidisciplinary cooperation in BCI systems; and, formulating more accurate and precise definition of the emotion processing in human and neurophysiological studies in EEG correlates of emotion as noted by Kim et al. in [16].In addition to the above-stated trends, we anticipate extensive advancement in body sensors, cameras, and head-mounted devices.Moreover, improving technology sensors is advancing at a rapid rate, as noted by Alotaiby et al.,in [13].We estimate that these will contribute to the development of rudimentary affective computing systems in approximately six to nine years' time.In ten+ years' time, we anticipate better machine learning and pattern recognition algorithms, in addition to improved accuracy, speed, and elimination of latency in BCI systems.Advances in computational methods have been noted by researchers to reach a reasonable rate of diffusion and serve as key elements for practical online emotion detection systems, as noted by Kim et al. in [16] and by Jenke et al. in [14].In addition to the above-stated trends, we anticipate extensive advancement in body sensors, cameras, and head-mounted devices.Moreover, improving technology sensors is advancing at a rapid rate, as noted by Alotaiby et al.,in [13].We estimate that these will contribute to the development of rudimentary affective computing systems in approximately six to nine years' time.In ten+ years' time, we anticipate better machine learning and pattern recognition algorithms, in addition to improved accuracy, speed, and elimination of latency in BCI systems.Advances in computational methods have been noted by researchers to reach a reasonable rate of diffusion and serve as key elements for practical online emotion detection systems, as noted by Kim et al. in [16] and by Jenke et al. in [14].
Moreover, we believe that, there is potential for using hybrid approaches in classification and feature extraction methods.Another aspect is sharing datasets and making them accessible to researchers for further testing.Third, optimizing the number of electrodes is increasingly recognized as important for computation.Also, combining different modalities with EEG.Together, these contribute towards facilitating the work of future researchers in this domain.

Conclusions
Recent developments and studies in BCI technologies have facilitated emotion detection and classification.These BCI studies set out to investigate, detect, and recognize a participant's emotional affective state.These studies also sought the application of research findings in varied contexts, including communication, education, entertainment, and medical settings.However, increasing numbers of BCI researchers are conducting novel experiments, such as considering different responses in various frequency bands, different ways of eliciting emotions, and various models of affective states.These various approaches have been employed in gauging the emotional states of participants/patients from BCI acquisition signals.
This study set out to review published articles on emotion detection, recognition, classification, and current and future trends.Thereafter, it sought to provide insight into how current and future trends will impact researchers and practitioners alike.To achieve this, we mined 29 online databases, selected 160 journals, and extracted 285 articles on emotion recognition systems.Each article was reviewed, analyzed, and categorized according to publication year, research methods, primary contribution, and publication outlet.
Our classification and descriptive review provides quality reference sources for academics, researchers, and practitioners that are working in the field of emotion detection and recognition.This study also contributes to our understanding of various applied concepts of emotion recognition using BCI in different contextual fields.Our results showed an explosive growth of the number of EEG-based emotion detection journal publications.This reflects increased research interest in EEG-based emotion detection as a salient and legitimate research area.Factors such as the proliferation of wireless EEG devices, advanced computational intelligence techniques, and machine learning, spurred this growth.In general, we expect exponential growth in the amount of EEG-based emotion detection research in the near future.As would be anticipated of any new research field, the field of EEG-based emotion detection and recognition is fraught with challenges.Our results show that the challenges raised herein may be resolved in the near future, thereby causing further growth and increased research interest in BCI systems.
As noted in Section 5, EEG-based emotion detection involves multiple dimensions that need to be considered in research methodologies.Researchers would need to take into account the usability of devices, accessibility of datasets, optimization of computational methods, and combining different modalities in the design of their EEG based studies.
Regarding computational methods that can be used in feature extraction and classification phases, it appears from our review that there is no particular feature extraction nor classification technique that emerges as the single best choice for all of the applications.The choice depends on the specific system paradigm and task.
It has been recommended to consider as many algorithms as possible to determine the validity of the proposed process, including preprocessing and synchronization.In most cases, one should compare performance with a range of features and techniques before settling on a choice that yields adequate performance for the given application.

Appendix B.2. Classification of Articles by Research Area
The distribution of the results, according to the Web of Science categorization, is shown in Table A1.In this table, we can see that most of the articles were published in journals related to neuroscience, biology, physiology, computer science, and electronic engineering.
To classify these articles by their research areas, we had to determine into which broad areas they fit.Such areas as Neuroscience, Engineering, Computer Science, and Healthcare were analyzed.In Figure A2 (shown below), we can see the inter-connectedness of EEG-based emotion detection systems across several disciplines.To create this plot, we performed the following steps: (1) determine a list of sub-subjects that could be related to EEG-based emotion detection; (2) research how each subject is related to the others on an individual basis; and finally (3) visually display, using a Venn diagram, not only how the subjects are related to one another but also how strongly they are related to the main subject, which is the study of EEG-based emotion detection systems.
The strength of that relationship is directly related to the size of the bubble.The relationships between subjects were obtained using Google Scholar and Web of Science.Different published papers that are related to the sub-subjects show stronger correlations between various fields of study.

Appendix B.2. Classification of Articles by Research Area
The distribution of the results, according to the Web of Science categorization, is shown in Table A1.In this table, we can see that most of the articles were published in journals related to neuroscience, biology, physiology, computer science, and electronic engineering.
To classify these articles by their research areas, we had to determine into which broad areas they fit.Such areas as Neuroscience, Engineering, Computer Science, and Healthcare were analyzed.In Figure A2 (shown below), we can see the inter-connectedness of EEG-based emotion detection systems across several disciplines.To create this plot, we performed the following steps: (1) determine a list of sub-subjects that could be related to EEG-based emotion detection; (2) research how each subject is related to the others on an individual basis; and finally (3) visually display, using a Venn diagram, not only how the subjects are related to one another but also how strongly they are related to the main subject, which is the study of EEG-based emotion detection systems.
The strength of that relationship is directly related to the size of the bubble.The relationships between subjects were obtained using Google Scholar and Web of Science.Different published papers that are related to the sub-subjects show stronger correlations between various fields of study.

Appendix B.3. Classification of Articles by Online Database and Journal
There are a total of 285 articles, published by 34 online databases.Classifications of articles by online database and journal are shown in Tables A2 and A3, respectively.
In Table A2, we specify our top ten online article databases.Each of these hosted a minimum of five article publications that matched our search criteria.Of these, Science Direct had the highest

Appendix B.3. Classification of Articles by Online Database and Journal
There are a total of 285 articles, published by 34 online databases.Classifications of articles by online database and journal are shown in Tables A2 and A3, respectively.
In Table A2, we specify our top ten online article databases.Each of these hosted a minimum of five article publications that matched our search criteria.Of these, Science Direct had the highest percentage of articles (103 articles, 36.1%).The likely reason for this is that Science Direct publishes a coterie of different journals, such as Neurocomputing; International Journal of Psychophysiology; Biomedical Signal Processing and Control; Computers in Biology, Medicine, Brain, and Cognition; Brain Research; and Computers in Human Behavior.All of these journals publish EEG-based emotion recognition system articles.Similarly, Springer Link also hosts a cross-section of journal articles from various fields.Consequently, their database recorded the second-highest number of articles: 27 articles, or 9.5% of our total research articles.Other online databases are IEEE Xplore (23 articles, or 8.1%), Frontiers (22 articles, or 7.7%), Taylor & Francis (16 articles, or 5.6%), and Wiley Online Library (12 articles, or 4.2%).Table A3 shows the number of articles in each journal publication.We only documented journals with more than two published articles.Most of these journals were related to neuroscience, biology, physiology, computer science, electronic engineering, and healthcare.Of these, PLOS ONE published 10 articles (or 3.5%); International Journal of Psychophysiology and IEEE Transactions on Affective Computing each published 9 articles (or 3.2%).

Figure 2 .
Figure 2. Procedure that we used to extract and filter articles.

Figure 2 .
Figure 2. Procedure that we used to extract and filter articles.

Figure 2 .
Figure 2. Procedure that we used to extract and filter articles.

Figure 4 .
Figure 4. Classification of articles by field and publication year.

Figure 5 .
Figure 5. Treemap classification of articles by application domain and field ( non-medical  medical).

Figure 4 .
Figure 4. Classification of articles by field and publication year.

•
the paper discussed a medical condition (disorder/disease), such as a psychiatric or neurological case;• the participants of the experiment were patients or it involved two groups: one consisting of healthy people, the other of patients; • the experiment was conducted in a clinical setting; and/or, • the paper was directed toward the medical community and suggested a new method for assistance, enhancement, monitoring, or diagnosis using emotion-based recognition.

Figure 4 .
Figure 4. Classification of articles by field and publication year.

Figure 5 .
Figure 5. Treemap classification of articles by application domain and field ( non-medical  medical).

Figure 5 .
Figure 5. Treemap classification of articles by application domain and field (

Figure 4 .
Figure 4. Classification of articles by field and publication year.

Figure 5 .
Figure 5. Treemap classification of articles by application domain and field ( non-medical  medical).

Figure 4 .
Figure 4. Classification of articles by field and publication year.

Figure 5 .
Figure 5. Treemap classification of articles by application domain and field ( non-medical  medical).

Figure
Figure A1 is a line graph showing articles published in the time frame from 2005 to 2016.Between 2005 and 2009, the graph shows a steady rise in EEG-based emotion detection publications.Thereafter, article publications rose rapidly from 2010 to 2016.The rapid rise in EEG-based emotion detection publications is expected due to increased knowledge of neurobiological processes, computers with faster computational processing, greater availability of devices for recording brain signals, and more powerful signal processing and machine learning algorithms.All of these factors positively contributed to the development of improved affective BCI technology.Aligned with the Gartner Hype Cycle, there was a marked increase in research articles related to Affective Computing.

Figure A1 .
Figure A1.Classification of articles by publication year.The graph also shows that, between 2005 and 2008, only 10 articles related to the subject were published.After that, the volume of publications rose sharply, culminating in a spike in 2009, with 15 articles published.Post-2009, the volume of publications declined, with an average of 17 articles published between 2010 and 2013.Soon after, the number of articles published increased from 46 articles in 2014 to 48 articles in 2015.By 2016, the number of published articles significantly increased to 62.

A1.
Classification of articles by publication year.The graph also shows that, between 2005 and 2008, only 10 articles related to the subject were published.After that, the volume of publications rose sharply, culminating in a spike in 2009, with 15 articles published.Post-2009, the volume of publications declined, with an average of 17 articles published between 2010 and 2013.Soon after, the number of articles published increased from 46 articles in 2014 to 48 articles in 2015.By 2016, the number of published articles significantly increased to 62.

Table 4 .
[92]hmark EEG emotional databases.multimodaldatasetfor the analysis of human affective states.The electroencephalogram (EEG) and peripheral physiological signals of 32 participants were recorded as each watched 40 one-minute-long excerpts of music videos.Participants rated each video in terms of the levels of arousal, valence, like/dislike, dominance, and familiarity.For 22 of the 32 participants, frontal face video was also recorded.The database is described in[92]and is available online at http://www.eecs.qmul.ac.uk/mmv/datasets/deap/. of 15 subjects were recorded while they were watching emotional film clips.For the feedback, participants were told to report their emotional reactions to each film clip by completing a questionnaire immediately after watching each clip.To investigate neural signatures and stable patterns across sessions and individuals, each subject was required to perform the experiments for three sessions.The time interval between two sessions was one week or longer.Facial videos and EEG data were recorded simultaneously.The database is available online at http://bcmi.sjtu.edu.cn/~seed/download.html.
[209]07,208]a multimodal database for emotion recognition and implicit tagging.It includes the physiological signals from 27 participants in response to 20 stimulus videos.Subjects' emotional self-assessments are nine-scale evaluations, from 1 to 9, for both valence and arousal.The database is described in[101]and is available online at http://www.ibug.doc.ic.ac.uk/resources/mahnob-hci-tagging-database/.[41,207,208]eNTERFACE06_EMOBRAINIt is a multimodal database for emotion recognition.It contains physiological signals from both the peripheral (galvanic skin response, respiration, and blood volume pressure) and central (EEG and frontal fNIRS) nervous systems from five subjects in response to picture stimuli.The database is available online at http://www.enterface.net/enterface06/docs/results/databases/eNTERFACE06_EMOBRAIN.[209]

Table 5 .
Computational methods to extract and classify emotional features from EEG.
[20,62,66,76,137,183,226,228] symptoms experienced by the patients.Most of these symptoms are emotional; hence, they are difficult to diagnose using subjective means.The goal of these systems is to objectively detect medical anomalies in a patient's emotional affective state.[20,62,66,76,137,183,226,228]

Table A2 .
Classification of articles based on the online database.

Table A3 .
Classification of articles based on the journal.