Emotions are understood as a complex set of neural and hormonal interactions that can give rise to affective experiences (bodily sensations); generate cognitive processes (feelings, the conscious emotions); imply physiological adjustments to adapt to them; and lead to adaptive behaviors and/or decision making [1
]. Emotions are an important evolutionary factor that allow survival and breeding through adaptation to the environment. However, the mechanisms of emotional processes and the modeling of human emotions are still fairly unknown. Many efforts have been made to unmask the psychobiology of emotions, since a century and a half ago Darwin proposed the first theory that tried to explain its origin [2
]. Still, this is a complicated task by the fact that, even today, there is no consensus regarding the functioning, structure and classification of emotions. One of the most spread theories, Dimensional model of emotions [3
], sustains that emotions can be explained mainly by two dimensions, valence (pleasure/displeasure) and arousal (calm/excited). Depending on the level of activation and polarity of this biphasic dimensions, motivational systems of approach (survival and pleasure) and withdrawal (fight or flight responses) are activated with the intention of adapting behavior to an emotional stimulus [5
]. This theory has increased its popularity due to the affective computing research and applications [6
]. While affective neuro-science’s main objective has focused on the study of the neurobiology of emotions, the affective computing branch has been much more pragmatic, leaving the biology behind and concentrating on its recognition and classification. Nevertheless, neglecting the psychological theories of emotions has filled the affective computing studies with assumptions that undermine its own credibility and effectiveness. Therefore, for the development of functional affective interfaces, it is necessary to contextualize engineering goals based on psychological principles [7
In order to achieve a successful adaptation to the environment, emotions are integrated in the central nervous system (CNS)—what we will call the brain response; and in the autonomic nervous system (ANS)—the bodily response; leading to goal-directed behaviors. The fact that emotions have specific brain and body responses, has motivated the study of the different signals that intervene both in their processing and in their response in order to try to find patterns that allow them to be identified.
At CNS level, the functional Magnetic Resonance Imaging (fMRI) technique, due to its spatial resolution, is the most used—for the study of the neural substrates that underlie emotions; however, the low temporal resolution, high cost and the impossibility of using it in normal life environments and situations, separate it from the picture of affective computing applications [8
]. On the other hand, the temporal resolution, usability, low-cost and wireless nature of the EEG, make it the suitable technique for emotion recognition applications [9
]. In order to characterize the neural structures implied in the emotional process and describe the way in which they work and interact, specific features of the EEG frequency spectrum, electrode location and temporal window have been evaluated. However, the lack of theoretical consensus on the emotional process complicates this task due to the high variability between studies and, therefore, the difficulty of comparing results. Nevertheless, different EEG asymmetry patterns across hemispheres have been observed, left hemisphere activation over frontal and prefrontal regions is linked with positive affect experience and therefore to the approach motivational system; and contrary the right hemisphere presents higher activity when processing negative affective or withdrawal stimuli [10
Regarding the emotional body response, the main structure that regulates and controls the vegetative auto-regulatory processes in order to meet behavioral demands is the ANS [11
], which is closely linked with the CNS emotional part [12
]; and it is believed to be involved in the generation of the physiological arousal of the emotional episode. It is thought to be related with the arousal dimension, but not with the valence scale [13
]. The ANS has two branches, the sympathetic nervous system (SNS), which becomes dominant, increasing the physiological arousal, when either psychological or physical stress is taking place and, the parasympathetic nervous system (PNS), which dominates during periods of rest or safety, maintaining a low degree of physiological arousal. These systems are related with the approach and withdrawal motivational systems since they are responsible for the body’s response [15
]; however, the degree and functionality of the process is still diffuse. The traditional view of the ANS emotional response stands for specific patterns of activation regarding emotional component of the stimulus [16
]. Nevertheless, more evidences supports the undifferentiated arousal theory [17
], suggesting that all emotions present the same or at least similar ANS activation pattern when high arousal stimuli take place. However, this theory does not explain the different responses showed with stimuli presenting the same value in the arousal scale but opposite valence scores. Several techniques have been used for emotion recognition based on bodily responses, such as heart rate, galvanic skin response, respiration, skin temperature and behavioral measures [7
]. Interest in behavioral measures such as facial expressions, voice, and body language emerged because of similarities between cultures found in emotional expression [18
], however these methods have a high cost and require long training times and modelling of the subject [19
]; therefore, physiological measures are preferred. The SNS mediated responses for negative emotions and PNS for positives have been described by several authors and measures as heart rate [20
], skin temperature [22
] and galvanic skin response [24
]. However, the same problem regarding the comparison of the EEG studies apply for the ANS emotional patterns.
Each modality or psychophysiological signal used for emotion recognition has its own pros and cons and an extensive literature behind. Calvo et al. [7
] have pro-posed a set of factors to evaluate the effectivity of a modality to serve as a way for affective computing interfaces. First, the validity of the signal to represent the emotional process. Brain signals are preferred to bodily signals since the latter can be consciously modified and are more unspecific. Second, the reliability of the signal in real-life applications. In general, brain signals have obtained better classification results at the valence scale, whereas bodily signal did for the arousal [25
]; suggesting that both types of signals measure different, but complementary aspects of the emotional state and therefore bringing up the idea of combining the modalities for better performances. Finally, the time resolution, cost and user invasiveness also have to be taken into account. Although EEG technology is now advancing in the development of more accessible and user-friendly devices, its complexity and discomfort is greater than that needed to measure bodily signals, which can be acquired at the time through a simple bracelet [27
In previous studies [28
], we have evaluated some of the technical parameters necessary for emotion recognition based on the EEG signal, but without delving deeper into its biological implications. In the present work, we wanted to study the response of the ANS and its contribution to emotion recognition on the valence scale. Moreover, we would like to test the idea of the synergy between brain and body responses for emotion recognition applications. For this end, we have recorded the EEG, ECG and skin temperature signals of 24 subjects during stimulation using videos with positive and negative emotional content. Each biological modality has been studied individually, in a subject dependent (SD) and independent (SI) way, to finally perform a multimodal classification and correlation analysis in order to see the relationships between body and brain signals, and the emotional subjective ratings provided by the participants.
Understanding the psychophysiology of emotional processes, i.e., the relationship between body and mind, is key to the design of effective and reliable affective brain computer interface applications. Knowing which are and how the activation patterns of the neuronal substrates are involved in the processing of emotions would allow to design more precise computational models and reduce the preparation and training times of the subjects. At the same time, being able to distinguish the emotional peripheral physiological responses and understand the performance of the ANS mechanisms responsible for them would allow a more complete emotional approach and the possibility of developing simpler and more accessible systems.
The belief that emotions are encoded in subcortical and limbic structures, whereas, cognition is encoded in the cortical level has been dismissed, as new evidence, coming from affective neuroscience studies, has supported the statement that emotion and cognition display or overlap along the same cortical nets [46
]. In the meta-analysis conducted by Kober et al. [47
], they tried to identify patterns of co-activation of brain regions and its functional organization in emotional neuroimaging studies without labeling the underlying emotions, i.e., without semantically defining the emotional category, thus overcoming the problem of lack of consensus on emotional theory. They defined six functional groups; lateral occipital or visual association group, medial posterior group, cognitive/motor group, lateral para-limbic group, medial pre-frontal cortex group and the core limbic group; of which the prefrontal, occipital and central-motor cortical lobes should be highlighted. Most of these functional groups were found as relevant regions in our emotional model, thus relating the emotional process with a whole brain network, more than specific isolated areas. Moreover, another important fact revealed by the meta-analysis of Kober et al. was that all cortical structures involved in emotional processing showed co-activation with subcortical structures as the limbic system and the brainstem. The EEG only allows us to assess cortical brain activity, and therefore, it is important to note that we are trying to classify emotions by missing an important part of the puzzle. It is thus interesting and necessary to know the body-mind relationships to have a more complete vision of the process and therefore, define emotions more accurately.
Based on our results, in the case of the frequency domain aspect of emotions, at positive ones, Alpha and both Beta frequencies seemed relevant at the left prefrontal hemisphere; and both Beta1 and Beta2 bandwidths increase its activation at the occipital right hemisphere. On the other hand, negative emotions presented a lateralization pattern towards the right prefrontal cortex and highlight the presence of Beta1 and Beta2 frequencies towards the left hemisphere over occipital regions. These results seem to point to a reverted lateralization pattern of frontal and posterior cortex when processing emotions. Our results agree with the frontal EEG asymmetry theory, described by Davidson et al. [5
], reflecting the activation of motivational systems of approach and withdrawal, also verified by other authors [48
]. In essence, we can conclude that there are interhemispheric differences in the processing of emotions; that this lateralization is also different depending on the emotional category; and that the processing of emotions not only falls on the prefrontal cortex, but rather there seems to be a neural network that expands along the entire cortex [52
], and at all spectrum, excluding the low frequencies of the EEG [29
]. The similarity on the scalp distribution of the different frequencies involved in the processing of positive and negative emotions suggests that, at least at the cortical level, there are no separate neural pathways for processing positive and negative emotions, but there is a network of cortical structures in charge of processing the valence, whose activity varies depending on the polarity of the emotion. Yet, a more in depth study of the relationships between regions and emotional conditions is necessary in order to draw meaningful conclusions about the brain emotional net.
Peripheral psychophysiological reactions constitute an important source of emotional information, therefore, researchers have focused on the different ANS measures. However, if the PNS and SNS are linked with the positive/approach and negative/withdrawal responses [54
], respectively, is more diffuse. As for HRV measures, both our results and those of other authors, showed differences in their response to either emotional dimensions [20
] and discrete emotions [21
]. These differences are present in all levels of the HRV analysis, the time domain, the frequency domain and the Poincare plot. Nevertheless, and as it is customary in the study of emotions, there is no consensus as to which are the HRV variables most representative of the emotional state. In our case, no significant differences were found in any variable with the SI approach, and although there were differences at SD level, they did not show clear patterns across individuals. Moreover, classification results achieved using all HRV measures to differentiate positive and negative emotions were not different than chance; and although after feature selection, precisions of around 60% were reached, these values are far from the percentages achieved by Guo et al. [55
] or Goshvarpour et al. [20
], of 71.4% and 100%, respectively. Differences between our study and the mentioned studies, could be explained by the different methodologies used. Guo et al. applied principal component analysis in order to select the features used for the classification; therefore, not using the “real” HRV measures. On the other hand, Goshvarpour et al. used a neural network method, that although obtained better performance, it doesn’t allow inferences to be made at the biological level as the interaction of the variables with each other and their weight in the classification is not known. Nevertheless, when separating the sample by gender, significant differences were found between positive and negative emotions in the Poincare variable SD2 in women and in the time variable NN50 in men, corroborating the existence of gender differences in young subjects [58
]. The results suggest that in men, the differences are more evident in the short term, responding with greater intensity to positive stimuli; and on the contrary, women respond with greater intensity to negative stimuli, although the difference in this case is observable in the longer term. This may indicate a greater readiness of men for immediate response and greater adaptability to emotional stimulation in women. The variables found as informative, SD2 and NN50, are both regulated by both components of the ANS, so conjectures about the involvement or role that each division plays in the emotional response are not possible.
On the other hand, it has been proven that skin temperature measure could be used for positive and negative emotion discrimination [22
]. The accepted explanation of the role of the skin temperature in the emotional process point out to vasoconstriction responses in order to mobilize blood into the muscular system to allow reaction to an aversive stimulus. Therefore, it seems that the dichotomy between the activation of the approach and withdrawal systems apply at the skin temperature level, however, both vasoconstriction and cooling and vasodilation and warming responses, are mediated by the SNS [23
], pointing to the arousal scale. Therefore, although there are different patterns of response when it comes to processing positive and negative emotions, the functional organization of the activity of the ANS components remains unclear [14
]. Nevertheless, our results showed an opposite pattern of the commonly accepted response [60
], obtaining higher temperatures for negative than for positive emotions. Regarding gender, although there are differences in the thermoregulation of women and men [61
], our results indicate that there are no gender differences at the skin blood volume regulation response when processing positive emotions, but, differences exist while negative stimuli occur; suggesting that women react more intensely to negative emotions than men.
Although the responses of the CNS and ANS systems to emotional stimuli and the relationships that exist between them are not known exactly; our results suggest that in both systems, independently, it is able to differentiate between positive and negative emotions. Koelstra et al. [26
] used the detection of facial expressions and the EEG signal to classify emotions on the valence and arousal scales, demonstrating that classification performance improved when the two signals were combined, reaching percentages of 67.1% and 71.5%, respectively. Torres et al. [25
] evaluated the combination of several biosignals for the detection of emotions on the valence and arousal scales. In the arousal scale, the best classification, 75%, was obtained after the combination of the EEG with physiological signals (heart rate, galnavic skin response (GSR), respiration and skin temperature); however, on the valence scale, the results of the combination of modalities did not improve the percentage achieved by the EEG alone, 58.75%. Using other theories of emotions, Verma and Tiwary [63
] showed the best accuracy obtained in the literature when applying the multimodal approach (CNS: EEG; PNS: GSR, respiration, blood volume pressure and skin temperature) for the classification of discrete emotions, with an average value of 81.45% with a support vector machine. On the other hand, using the same database as Verma and Tiwary, but analyzing the emotional valence dimension, Chen et al. [64
] obtained an accuracy of 83.98%. Our results improve classification performances obtained by other authors and conclude that the CNS per se (0.988 f1-score classification result for the KNN classifier) is most informative than the ANS data (HRV + skin temperature, 0.943 f1 score) in order to classify emotions regarding the valence dimension; and that the combination of the modalities (0.989 f1 score) does not significantly improve the results reached by the EEG alone. Authors like Jatupaiboon et al. [65
] and Torres-Valencia et al. [66
], also concluded, through other methodologies and emotional induction methods but addressing the effectiveness of multimodal emotional approach, that the EEG is the signal with the best capacity of emotional discrimination following the dimensional model of emotions. Therefore, based in our results we can discard the undifferentiated arousal theory that supports that the emotional response in the valence dimension is only reflected at the brain level and not at the peripheral/body level. Moreover, regarding the traditional idea that links the motivational systems of approach and withdrawal with the PNS and SNS systems, respectively, we cannot conclude for or against. Nevertheless, our results make even more evident the incompatibility of having a population-trained model that can be used for particular individuals, due to the high variability observed among subjects. Therefore, in order to make an accurate classification of positive and negative emotions it is necessary to train the computational model for each subject. However, it is possible to specify the most relevant features for the classification of emotions in the dimension of valence at the population level; thus decreasing the dimensionality of the model, the complexity of the system and the temporal cost of classification.
Regarding the gender factor in the classification of emotions, we found differences in the peripheral nervous system response, but not at the CNS. This suggests that emotions are processed in the same way for men and women at the brain level, but the body response is different [67
]. However, one of the main drawbacks of our study is that women and men samples were not balanced, being considerably less number of women than men, and moreover, although our sample of 24 subjects is more than acceptable for this kind of studies, when splitting it into gender, the sample size is not representative of the population for SI analysis regarding the minimum size of 15 subjects stablished for proper classifications [68
]. Therefore, although our results are encouraging, it would be necessary to enlarge the sample size in order to obtain more reliable results in terms of gender differences.
At the peripheral level, it seems that there were differences between the responses to opposing emotional valence stimuli. However, they did not seem to follow the expected pattern of ‘fight or flight’ or ‘calm or safety’ associated with the motivational systems of approach and withdrawal, that are believed to act at the level of the prefrontal EEG asymmetries. The components of the ANS are not activated in an ‘all-or-none’ fashion, rather each tissue is innervated differently by the sympathetic and parasympathetic pathways, which act independently of each other [14
]. It is therefore difficult to attribute approaching or rejecting responses to specific components of the ANS. At this point, it is worth asking if motivational systems represent the same as the dimension of affective valence or if, on the contrary, they are different processes that do not always go hand in hand [69
]. Conversely, it is probable that the arousal is influencing the physiological response, since although there were no significant differences in the arousal rating in the population, polarity existed in specific individuals.