Brain and Body Emotional Responses: Multimodal Approximation for Valence Classification

In order to develop more precise and functional affective applications, it is necessary to achieve a balance between the psychology and the engineering applied to emotions. Signals from the central and peripheral nervous systems have been used for emotion recognition purposes, however, their operation and the relationship between them remains unknown. In this context, in the present work, we have tried to approach the study of the psychobiology of both systems in order to generate a computational model for the recognition of emotions in the dimension of valence. To this end, the electroencephalography (EEG) signal, electrocardiography (ECG) signal and skin temperature of 24 subjects have been studied. Each methodology has been evaluated individually, finding characteristic patterns of positive and negative emotions in each of them. After feature selection of each methodology, the results of the classification showed that, although the classification of emotions is possible at both central and peripheral levels, the multimodal approach did not improve the results obtained through the EEG alone. In addition, differences have been observed between cerebral and peripheral responses in the processing of emotions by separating the sample by sex; though, the differences between men and women were only notable at the peripheral nervous system level.


Introduction
Emotions are understood as a complex set of neural and hormonal interactions that can give rise to affective experiences (bodily sensations); generate cognitive processes (feelings, the conscious emotions); imply physiological adjustments to adapt to them; and lead to adaptive behaviors and/or decision making [1]. Emotions are an important evolutionary factor that allow survival and breeding through adaptation to the environment. However, the mechanisms of emotional processes and the modeling of human emotions are still fairly unknown. Many efforts have been made to unmask the psychobiology of emotions, since a century and a half ago Darwin proposed the first theory that tried to explain its origin [2]. Still, this is a complicated task by the fact that, even today, there is no consensus regarding the functioning, structure and classification of emotions. One of the most spread theories, Dimensional model of emotions [3,4], sustains that emotions can be explained mainly by two dimensions, valence (pleasure/displeasure) and arousal (calm/excited). Depending on the level of activation and polarity of this biphasic dimensions, motivational systems of approach (survival and pleasure) and withdrawal (fight or flight responses) are activated with the intention of adapting behavior to an emotional stimulus [5]. This theory has increased its popularity due to the affective computing research and applications [6]. While affective neuro-science's main objective has focused on the study of the neurobiology of emotions, the affective computing branch has been much effectivity of a modality to serve as a way for affective computing interfaces. First, the validity of the signal to represent the emotional process. Brain signals are preferred to bodily signals since the latter can be consciously modified and are more unspecific. Second, the reliability of the signal in real-life applications. In general, brain signals have obtained better classification results at the valence scale, whereas bodily signal did for the arousal [25,26]; suggesting that both types of signals measure different, but complementary aspects of the emotional state and therefore bringing up the idea of combining the modalities for better performances. Finally, the time resolution, cost and user invasiveness also have to be taken into account. Although EEG technology is now advancing in the development of more accessible and user-friendly devices, its complexity and discomfort is greater than that needed to measure bodily signals, which can be acquired at the time through a simple bracelet [27].
In previous studies [28,29], we have evaluated some of the technical parameters necessary for emotion recognition based on the EEG signal, but without delving deeper into its biological implications. In the present work, we wanted to study the response of the ANS and its contribution to emotion recognition on the valence scale. Moreover, we would like to test the idea of the synergy between brain and body responses for emotion recognition applications. For this end, we have recorded the EEG, ECG and skin temperature signals of 24 subjects during stimulation using videos with positive and negative emotional content. Each biological modality has been studied individually, in a subject dependent (SD) and independent (SI) way, to finally perform a multimodal classification and correlation analysis in order to see the relationships between body and brain signals, and the emotional subjective ratings provided by the participants.

Experimental Procedure and Data Analysis
A total of 24 subjects (mean age: 23.12; range: 19-37; sixteen men and eight women) were emotionally stimulated while EEG brain activity; cardiac activity and skin temperature were recorded. The stimuli consisted of 14 videos, 7 of them labeled as positive and the other 7 as negatives depending on the emotional content. The audiovisual stimuli were rated by the subjects in both valence and arousal scales, obtaining average valence scores of 7.51 (s.d. 1.6) and 2.91 (s.d. 0.98) for the positive and negative categories, respectively. In the arousal scale, positive stimuli were rated with a mean arousal of 3.76 (s.d. 1.62) and 5.47 (s.d. 1.35) for the negatives. The clips, with durations between 43 and 78 s, were selected from the internet, edited with the software Camtasia Studio 8 and presented randomly and counterbalanced to the subjects, in alternation with a 30 s black screen that also served as the initial baseline. The subjects were right-handed, had no personal history of psychiatric or neurological disorders, normal or corrected vision and audition, and were not under medication during the development of the study. They also provided their written consent, supervised by the Ethics Committee of the University Miguel Hernandez.

EEG
64 Ag-AgCl electrode cap was used for the electrical brain recording according to the International 10/10 System [30]. Data were amplified and registered through a NeuroScan SynAmps EEG amplifier (Compumedics, Charlotte, NC, USA), keeping the impedance for every electrode under 25 kΩ [31], and with a sampling rate of 1000 Hz. Data were filtered through a high-pass and low-pass filters, 0.5 Hz and 45 Hz respectively, and electrodes were re-referenced to a Common Average Reference (CAR), by Curry 7 software (Compumedics, Charlotte, NC, USA). For artifact rejection corresponding to electrical non-brain origin signals as eye-blinking, heart rate and muscle activity, the Matlab (The MathWorks Inc., Natick, MA, USA) toolbox EEGLAB [32] was used; artifacts were selected by means of visual inspection on the basis of Independent Component Analysis (ICA) [33]. Detailed data collection and pre-processing steps could be found in previous work [28].
In our previous work [29], a set of 20 features were selected as the most informative in terms of positive and negative emotion classification; however, few could be said about the theoretical interpretation of this result. Therefore, in the present work, the study of cerebral asymmetries was performed. From the 20 frequency-location variables corresponding to Alpha, Beta1, Beta2 and Gamma frequencies at both PreFrontal (PF) left and right locations; Beta 1 and Gamma at Frontal midline (Fm); Alpha and Beta1 at Central midline (Cm); Beta1 and Beta2 at Parieto-Occipital midline (POm); Gamma at both Central (C) left and right hemispheres; Gamma at Parietal (P) right; and Beta1, Beta2 and Gamma at Occipital (O) right; only those which localize in one or both of the hemispheres were used to study cerebral asymmetries through the classical method [34]. The asymmetry index (AI) is calculated for the spectral power of a specific bandwidth on homologous hemispheric regions, according to the Formula (1).
Pairs corresponding to midline regions were not included in the analysis; therefore, 9 combinations of frequencies and locations were evaluated. The AI was calculated for every frequency-location feature in each subject. However, in order to look for significant differences between hemispheres and emotional categories, data from all subjects were assessed together. Mann-Whitney test was used for the statistical analysis [35] between the AI of the positive and negative emotions. The same test was applied at the power spectral values themselves for the comparison between homologous hemispheric regions with either positive and negative categories. The entire EEG asymmetry analysis was performed in the Matlab environment.

ECG
Two electrodes were placed, one on the right side of the sternum and the other on the intercostal space between fifth and sixth rib, using lead II configuration in order to record ECG activity. The recording sampling rate was 1000 Hz and, as for the EEG signal, was performed with Curry 7 technology (Compumedics, Charlotte, NC, USA). After recording, ECG data were resampled to 256 Hz for further analysis. Data from subjects 1, 9, 17, 19 and 22 were not properly recorder, therefore these subjects were excluded from the analysis. Freely available stand-alone Artiifact 2.09 software [36] was used to accomplish the heart rate variability (HRV) analysis. HRV analysis was performed for each of the 24 subjects individually. A high-pass filter with the cut-off frequency at 10 Hz was applied to ECG data in order to extract the inter-beat interval data from every trial. Artifact detection and elimination was carried out through cubic spline interpolation. Finally, we performed time (Mean RR, Median RR, SDNN, RMSSD, NN50, pNN50) and frequency (VLF, LF, HF, LF/HF; frequency bands 0-0.04, 0.04-0.15, 0.15-0.4, respectively. Values for the different band-widths were obtained as a percentage, absolute values and normalized units only for the LF and HF measures) domain HRV measures. On the other hand, Matlab environment was used to assess non-lineal methods as the Poincare plot (SD1, SD2, SD1/SD2) [37][38][39][40]. See Table 1 for a detailed explanation of each HRV measure. Geometrical methods have not been included in the analysis because a minimum time period of 20 minutes' recording is necessary to ensure feasible results [41]. As expected on healthy young people, no abnormalities on the ECG signal as tachycardia, arrhythmias or bundle branch block were found.  [41,42] or instead it is also mainly determined by the PNS [43].
HF: high-frequency component occurs at the frequency of adult respiration (0.15-0.4 Hz), primarily reflects cardiac parasympathetic influence due to respiratory sinus arrhythmia.
LF/HF ratio: This rate is interpreted as an index of sympathovagal balance [44].
Poincare Plot SD1: standard deviation of the instantaneous (short-term) beat-to-beat RR interval variability. As vagal regulation over the sinus node are known to be faster than the sympathetically mediated effects, SD1 is considered a parasympathetic index [45].
SD2: standard deviation of the continuous long-term RR interval variability. There is evidence of both parasympathetic and sympathetic tones influenced on this index [38].
SD1/SD2 ratio: ratio between the short and long interval variation.
Each of the 19 variables evaluated was z-scored and tested for normal distribution with the One-sample Kolmogorov-Smirnov test, resulting in non-normal data distribution. Mann-Whitney test was carried out to test if positive and negative data came from the same distribution at both SD and SI approximations. We have also analyzed the data separating it by gender. Finally, simulated annealing optimization method was used to select the most informative features with K-nearest neighbors (KNN) with 5 neighbors and quadratic discriminant analysis (QDA) classifiers, at the SI approach.

Skin Temperature
Skin temperature signal was recorded at the right wrist of the volunteers through the ActTrust bracelet (Condor Instruments Ltda., São Paulo, Brazil). Due to problems during the experimentation, the data of subjects 1 and 15 could not be included in the analysis, therefore 22 subjects from the total of 24 enter into the analysis. Data were segmented in trials of 28 s lengths, corresponding to every video clip. Values more than three scaled median absolute deviations from the median were consider as outliers and replaced by the mean value. One-sample Kolmogorov-Smirnov test was used for evaluating data normal distribution, resulting as non-normal distributed data. In order to look for differences between the two categories of emotions and the neutral state, the statistical Mann-Whitney test was performed [35], in the SD and SI approximations. Gender differences were also assessed with the same statistical test.

Multimodal Approximation
In previous work [29], we proposed an EEG-based model for the classification of positive and negative emotions. In the present work we have evaluated the relevance that peripheral physiological Sensors 2020, 20, 313 6 of 18 signals could have in classification performance associated to our EEG-based model. In order to obtain same length segments of each type of signal, temperature data was down-sampled and ECG data was up-sampled to match EEG data. Moreover, a total of 6 subjects (numbers 1, 9, 15, 17, 19 and 22) were eliminated from the analysis due to they were missing at least one of the data inputs. Final classification stage was performed with the 20 variables coming from the EEG analysis, 1 variable corresponding to skin temperature, and the significant variable resulting from the ECG analysis. KNN with 5 neighbors, and QDA classifiers were selected based on previous work, and applied in a SD and SI approximations. As there is evidence of gender differences in bodily signals, we also performed gender-based emotional classification (11 men, 6 women). Experimental procedure could be overviewed on Figure 1.  Finally, in order to assess the relationship between the emotional subjective ratings of the stimuli and the 22 different features that conform the model (20 EEG, 1 T, 1 ECG), we performed both a SD and a SI correlation analysis.

ECG
Looking at the statistical differences between positive and negative emotions at the studied HRV variables, no significant differences were found at the SI level. Contrary, significant differences appeared at the SD level for some of the variables and subjects ( Figure 2); however, no clear general patterns were found that would allow conclusions to be drawn about the population. As HRV gender differences are demonstrated in young people, we have assessed differences at the SI level for women and men. Significant differences were found between positive and negative emotions at the SD2 variable (p-value = 0.0498) for women; and at the NN50 variable (p-value = 0.0382) for men. SD2 index showed higher values during the processing of negative emotional stimuli, suggesting greater long term variability during the experience of negative emotions for women. On the contrary, higher values of NN50 variable were presented during the processing of positive emotions in comparison with negative emotions, i.e., there is greater variability between contiguous beats in the positive condition for men.
Statistical analyses have revealed that making common inferences for the population based on differences in HRV measures for positive and negative emotions is not an easy task. When classifying Figure 1. Experimental setup and multimodal approach for emotion classification. We performed the final multimodal classification stage with a total of 22 features (20 EEG frequency-location pairs, 1 HRV (heart rate variability) measure and skin temperature). Tº, skin temperature; IBIs, inter-beat intervals.
Finally, in order to assess the relationship between the emotional subjective ratings of the stimuli and the 22 different features that conform the model (20 EEG, 1 T, 1 ECG), we performed both a SD and a SI correlation analysis.

ECG
Looking at the statistical differences between positive and negative emotions at the studied HRV variables, no significant differences were found at the SI level. Contrary, significant differences appeared at the SD level for some of the variables and subjects ( Figure 2); however, no clear general patterns were found that would allow conclusions to be drawn about the population. As HRV gender differences are demonstrated in young people, we have assessed differences at the SI level for women and men. Significant differences were found between positive and negative emotions at the SD2 variable (p-value = 0.0498) for women; and at the NN50 variable (p-value = 0.0382) for men. SD2 index showed higher values during the processing of negative emotional stimuli, suggesting greater long term variability during the experience of negative emotions for women. On the contrary, higher values of NN50 variable were presented during the processing of positive emotions in comparison with negative emotions, i.e., there is greater variability between contiguous beats in the positive condition for men. classification performance improved, the selected features were not shared by the classifiers, with the exception of the NN50 index. In this context, selecting those variables that best allow discrimination between positive and negative emotions seems complicated, since the results obtained from classification, although favorable, left too much error margin. However, since our objective was to perform a multimodal classification to evaluate the contribution of the central and peripheral nervous systems in the recognition of emotions in the valence dimension, we have selected the NN50 variable to form part of the final classification stage, since it was the only common variable in both optimization groups, although it has only shown significant differences in the men group.

Skin Temperature
Results comparing positive and negative recordings resulted in significant differences (p-value under 0.05) for all subjects in the SD approximation. Moreover, skin temperature data corresponding either to positive and negative emotional conditions, was also significantly different than the baseline period; except for subject 14 at both comparisons positive vs. baseline and negative vs. baseline. At the SI approximation, there were also significant differences between the means of positive vs. negative (p-value = 1.56 × 10 −31 ), positive vs. baseline (p-value = 3.79 × 10 −6 ) and negative vs. baseline (p-value = 3.79 × 10 −6 ) emotional groups. Mean values corresponding to positive, negative and baseline skin temperature were 28.767 °C (s.d. 1.515), 28.847 °C (s.d. 1.486), and 27.087 °C (s.d. 1.552), respectively. Temperature values for negative emotions were slightly higher than for the positive, reflecting an opposite pattern of the expected vasoconstrictor sympathetic regulation in front of adverse stimuli [23].
Looking at gender differences, skin temperature values for positive emotions showed no significant differences between women and men samples (p-value = 0.1502), but negative emotions did (p-value = 0.0017) (Figure 3). If we disaggregate the population sample by gender, significant differences were presented in the women positive vs negative emotions (p-value = 1.7237 × 10 −42 ). Contrary, no differences were found for men (p-value = 0.7738). Statistical analyses have revealed that making common inferences for the population based on differences in HRV measures for positive and negative emotions is not an easy task. When classifying positive and negative emotions based on all the HRV measures, performances below the value of chance were obtained for both QDA and KNN classifiers, with f1 scores equal to 0.355 ± 0.161; and 0.497 ± 0.153, respectively. After simulated annealing optimization, performance improves up to 0.57 ± 0.17 for the QDA classifier, and 0.616 ± 0.125 for KNN; through using 5 (Mean RR, RMSSD, NN50, pNN50 and VLF) and 6 (Median RR, SDNN, NN50, LF [absolute values], HF [absolute values] and SD2) HRV variables, respectively, as inputs for the sorter. Although after feature selection, classification performance improved, the selected features were not shared by the classifiers, with the exception of the NN50 index. In this context, selecting those variables that best allow discrimination between positive and negative emotions seems complicated, since the results obtained from classification, although favorable, left too much error margin. However, since our objective was to perform a multimodal classification to evaluate the contribution of the central and peripheral nervous systems in the recognition of emotions in the valence dimension, we have selected the NN50 variable to form part of the final classification stage, since it was the only common variable in both optimization groups, although it has only shown significant differences in the men group.

Skin Temperature
Results comparing positive and negative recordings resulted in significant differences (p-value under 0.05) for all subjects in the SD approximation. Moreover, skin temperature data corresponding either to positive and negative emotional conditions, was also significantly different than the baseline period; except for subject 14 at both comparisons positive vs. baseline and negative vs. baseline. At the SI approximation, there were also significant differences between the means of positive vs. negative Temperature values for negative emotions were slightly higher than for the positive, reflecting an opposite pattern of the expected vasoconstrictor sympathetic regulation in front of adverse stimuli [23].
Looking at gender differences, skin temperature values for positive emotions showed no significant differences between women and men samples (p-value = 0.1502), but negative emotions did (p-value = 0.0017) (Figure 3). If we disaggregate the population sample by gender, significant differences were presented in the women positive vs negative emotions (p-value = 1.7237 × 10 −42 ). Contrary, no differences were found for men (p-value = 0.7738).

EEG Asymmetries
No differences were found between genders when classifying emotions based on EEG data (Table 2), therefore asymmetry studies were performed with the whole population. In order to have a better overview of the positive and negative emotional brain processes underlying the proposed model, we have studied the interhemispheric asymmetries of the 20 frequency-location pairs. This analysis was done with either SD and SI approaches. With respect to the SD approximation, none of the AI measures resulted as having significant differences between the positive and negative conditions in all subjects, this fact demonstrates once more the high inter-subject brain variability present in the emotional processing. Therefore, in this case, the evaluation of the EEG asymmetries between interhemispheric frequency-location pairs would be more fruitful with a SI approach, trying to focus only on the commonalities between subjects instead of the differences.  [29].
When comparing the AI of positive and negative emotional conditions, we have found differences in 5 of the 9 interhemispheric studied variables: PF-alpha (lateralized towards the left hemisphere at positive emotions and toward right on negative), PF-beta2 (presented the same lateralization pattern as PF-alpha), P-gamma (at both conditions, lateralization occurs towards the left hemisphere, however for positive emotions, power spectral density is higher than for negatives), O-beta1 (positive emotions lateralized towards the right hemisphere and negatives towards the left) and O-gamma (same lateralization pattern as the presented on the PF region). Conversely, if we look at the interhemispheric differences of each emotional category, we found significant differences in positive emotions at the PF-beta1 pair, lateralizing towards the left hemisphere; the PF-beta2, lateralizing towards the left too; and O-beta2 pair, lateralizing this time towards de right hemisphere. For the negative emotions, only O-beta1 and O-beta2 pairs presented significant differences, both lateralizing towards the left hemisphere. Figure 4 represents the AI between the positive and negative emotional categories at the frequency-location features of study.

EEG Asymmetries
No differences were found between genders when classifying emotions based on EEG data (Table 2), therefore asymmetry studies were performed with the whole population. In order to have a better overview of the positive and negative emotional brain processes underlying the proposed model, we have studied the interhemispheric asymmetries of the 20 frequency-location pairs. This analysis was done with either SD and SI approaches. With respect to the SD approximation, none of the AI measures resulted as having significant differences between the positive and negative conditions in all subjects, this fact demonstrates once more the high inter-subject brain variability present in the emotional processing. Therefore, in this case, the evaluation of the EEG asymmetries between interhemispheric frequency-location pairs would be more fruitful with a SI approach, trying to focus only on the commonalities between subjects instead of the differences.  [29].
When comparing the AI of positive and negative emotional conditions, we have found differences in 5 of the 9 interhemispheric studied variables: PF-alpha (lateralized towards the left hemisphere at positive emotions and toward right on negative), PF-beta2 (presented the same lateralization pattern as PF-alpha), P-gamma (at both conditions, lateralization occurs towards the left hemisphere, however for positive emotions, power spectral density is higher than for negatives), O-beta1 (positive emotions lateralized towards the right hemisphere and negatives towards the left) and O-gamma (same lateralization pattern as the presented on the PF region). Conversely, if we look at the interhemispheric Sensors 2020, 20, 313 9 of 18 differences of each emotional category, we found significant differences in positive emotions at the PF-beta1 pair, lateralizing towards the left hemisphere; the PF-beta2, lateralizing towards the left too; and O-beta2 pair, lateralizing this time towards de right hemisphere. For the negative emotions, only O-beta1 and O-beta2 pairs presented significant differences, both lateralizing towards the left hemisphere. Figure 4 represents the AI between the positive and negative emotional categories at the frequency-location features of study.
Sensors 2020, 20, x FOR PEER REVIEW 9 of 17 Figure 4. AI for the nine studied frequency-location pairs at positive and negative emotions for all subjects. The bracket with the asterisk above, which joins two bars of the histogram, indicates those pairs that have shown significant differences (p-value < 0.05) between the positive-negative conditions. Asterisks on the bars indicate significant differences (p-value < 0.05) between interhemispheric, intra-condition (positive-positive or negative-negative).

Multimodal Approximation
After the assessment of the differences between positive and negative emotional categories at the ECG and skin temperature signals, we wanted to evaluate their contribution to the classification of emotions, assessing if it is better to take into account the central nervous system or autonomic nervous system by themselves or if a synergy between brain and body exists. To this end, we have performed a SD and SI classification with the KNN and QDA classifiers using different inputs. Figure  5 shows the f1 scores for the SD classification, and Figure 6 the SI results.
Regarding SD classification, T + ECG condition presented the worst performance at both classifiers (0.602 s.d. 0.123 QDA; 0.943 s.d. 0.03 KNN). This result suggested that, although autonomic nervous system data per se could be used to differentiate positive and negative emotions, it is less effective than using central nervous system data alone (0. All other classifications showed similar results on both classifiers. As the performance obtained with the EEG data alone, was almost perfect, it is difficult to say if the addition of the autonomic nervous system data improves the recognition of emotions, but at least we could conclude that it does not worsen performance.
In the case of the SI classification, we continued having the same inter-subject variability problem observed in our previous work [29]. So that, possible inferences from the results are not conclusive.
Although EEG data did not show differences between genders, peripheral physiological signals did, therefore SI classification regarding gender was performed with all variables. The results obtained for the QDA classifier did not improve the ones obtained for the whole population (0.  . AI for the nine studied frequency-location pairs at positive and negative emotions for all subjects. The bracket with the asterisk above, which joins two bars of the histogram, indicates those pairs that have shown significant differences (p-value < 0.05) between the positive-negative conditions. Asterisks on the bars indicate significant differences (p-value < 0.05) between interhemispheric, intra-condition (positive-positive or negative-negative).

Multimodal Approximation
After the assessment of the differences between positive and negative emotional categories at the ECG and skin temperature signals, we wanted to evaluate their contribution to the classification of emotions, assessing if it is better to take into account the central nervous system or autonomic nervous system by themselves or if a synergy between brain and body exists. To this end, we have performed a SD and SI classification with the KNN and QDA classifiers using different inputs. Figure 5 shows the f1 scores for the SD classification, and Figure 6 the SI results.
Regarding SD classification, T + ECG condition presented the worst performance at both classifiers All other classifications showed similar results on both classifiers. As the performance obtained with the EEG data alone, was almost perfect, it is difficult to say if the addition of the autonomic nervous system data improves the recognition of emotions, but at least we could conclude that it does not worsen performance.
Sensors 2020, 20, x FOR PEER REVIEW 10 of 17 (s.d. 0.109) for men. Although mean classification performance improved, the differences between groups were not significant.  Once we have evaluated the performance of the model taking into account brain and body features together and independently, we wanted to study the correlation between those features and the emotional subjective rating of the stimuli. Figure 7 represents the correlation coefficients of the 22 variables at every subject in an SD analysis and for the whole sample in the SI analysis. In the SD analysis, no clear pattern among subjects was found suggesting positively or negatively correlated specific features with the emotional valence scale. Contrary, each subject presents its own brain and body response in front of subjectively rated positive and negative stimuli. We found the three types of possible responses occurring in all features, (a) increased activity with positive stimuli, (b) increased activity with negative stimuli and (c) same activation with both positive and negative emotional stimuli; suggesting that although all the studied features have relevance in the processing of emotions in the valence scale, their response is not constrained by the polarity of the scale. Moreover, we could also infer in three types of subjects, those who, in general, respond more actively    Once we have evaluated the performance of the model taking into account brain and body features together and independently, we wanted to study the correlation between those features and the emotional subjective rating of the stimuli. Figure 7 represents the correlation coefficients of the 22 variables at every subject in an SD analysis and for the whole sample in the SI analysis. In the SD analysis, no clear pattern among subjects was found suggesting positively or negatively correlated specific features with the emotional valence scale. Contrary, each subject presents its own brain and body response in front of subjectively rated positive and negative stimuli. We found the three types of possible responses occurring in all features, (a) increased activity with positive stimuli, (b) increased activity with negative stimuli and (c) same activation with both positive and negative emotional stimuli; suggesting that although all the studied features have relevance in the processing of emotions in the valence scale, their response is not constrained by the polarity of the scale. Moreover, we could also infer in three types of subjects, those who, in general, respond more actively In the case of the SI classification, we continued having the same inter-subject variability problem observed in our previous work [29]. So that, possible inferences from the results are not conclusive.
Although EEG data did not show differences between genders, peripheral physiological signals did, therefore SI classification regarding gender was performed with all variables. The results obtained for the QDA classifier did not improve the ones obtained for the whole population Although mean classification performance improved, the differences between groups were not significant.
Once we have evaluated the performance of the model taking into account brain and body features together and independently, we wanted to study the correlation between those features and the emotional subjective rating of the stimuli. Figure 7 represents the correlation coefficients of the 22 variables at every subject in an SD analysis and for the whole sample in the SI analysis. In the SD analysis, no clear pattern among subjects was found suggesting positively or negatively correlated specific features with the emotional valence scale. Contrary, each subject presents its own brain and body response in front of subjectively rated positive and negative stimuli. We found the three types of possible responses occurring in all features, (a) increased activity with positive stimuli, (b) increased activity with negative stimuli and (c) same activation with both positive and negative emotional stimuli; suggesting that although all the studied features have relevance in the processing of emotions in the valence scale, their response is not constrained by the polarity of the scale. Moreover, we could also infer in three types of subjects, those who, in general, respond more actively to positive stimuli (e.g., subjects 5,11,14,15,16), to negative stimuli (e.g., subjects 2, 3, 4, 7, 18), and subjects whose features respond in the same way in front of both positive and negative stimuli (e.g., subjects 1, 13, 17). On the other hand, when performing the correlation analysis with the SI approach, the variability observed in the SD results was not present. Although some variables presented slight polarizations, the result reflects that all variables are activated in the same way in both conditions of emotional valence, however, this result could be explained by the high variability presented between subjects. This result could be also reflecting that the process of consciously evaluating the emotional stimuli is not the same as the processing of the emotion.
Sensors 2020, 20, x FOR PEER REVIEW 11 of 17 to positive stimuli (e.g., subjects 5,11,14,15,16), to negative stimuli (e.g., subjects 2, 3, 4, 7, 18), and subjects whose features respond in the same way in front of both positive and negative stimuli (e.g., subjects 1, 13, 17). On the other hand, when performing the correlation analysis with the SI approach, the variability observed in the SD results was not present. Although some variables presented slight polarizations, the result reflects that all variables are activated in the same way in both conditions of emotional valence, however, this result could be explained by the high variability presented between subjects. This result could be also reflecting that the process of consciously evaluating the emotional stimuli is not the same as the processing of the emotion.

Discussion
Understanding the psychophysiology of emotional processes, i.e., the relationship between body and mind, is key to the design of effective and reliable affective brain computer interface applications. Knowing which are and how the activation patterns of the neuronal substrates are involved in the processing of emotions would allow to design more precise computational models and reduce the preparation and training times of the subjects. At the same time, being able to distinguish the emotional peripheral physiological responses and understand the performance of the ANS mechanisms responsible for them would allow a more complete emotional approach and the possibility of developing simpler and more accessible systems.
The belief that emotions are encoded in subcortical and limbic structures, whereas, cognition is encoded in the cortical level has been dismissed, as new evidence, coming from affective neuroscience studies, has supported the statement that emotion and cognition display or overlap along the same cortical nets [46]. In the meta-analysis conducted by Kober et al. [47], they tried to identify patterns of co-activation of brain regions and its functional organization in emotional neuroimaging studies

Discussion
Understanding the psychophysiology of emotional processes, i.e., the relationship between body and mind, is key to the design of effective and reliable affective brain computer interface applications. Knowing which are and how the activation patterns of the neuronal substrates are involved in the processing of emotions would allow to design more precise computational models and reduce the preparation and training times of the subjects. At the same time, being able to distinguish the emotional peripheral physiological responses and understand the performance of the ANS mechanisms responsible for them would allow a more complete emotional approach and the possibility of developing simpler and more accessible systems.
The belief that emotions are encoded in subcortical and limbic structures, whereas, cognition is encoded in the cortical level has been dismissed, as new evidence, coming from affective neuroscience studies, has supported the statement that emotion and cognition display or overlap along the same cortical nets [46]. In the meta-analysis conducted by Kober et al. [47], they tried to identify patterns of co-activation of brain regions and its functional organization in emotional neuroimaging studies without labeling the underlying emotions, i.e., without semantically defining the emotional category, thus overcoming the problem of lack of consensus on emotional theory. They defined six functional groups; lateral occipital or visual association group, medial posterior group, cognitive/motor group, lateral para-limbic group, medial pre-frontal cortex group and the core limbic group; of which the prefrontal, occipital and central-motor cortical lobes should be highlighted. Most of these functional groups were found as relevant regions in our emotional model, thus relating the emotional process with a whole brain network, more than specific isolated areas. Moreover, another important fact revealed by the meta-analysis of Kober et al. was that all cortical structures involved in emotional processing showed co-activation with subcortical structures as the limbic system and the brainstem. The EEG only allows us to assess cortical brain activity, and therefore, it is important to note that we are trying to classify emotions by missing an important part of the puzzle. It is thus interesting and necessary to know the body-mind relationships to have a more complete vision of the process and therefore, define emotions more accurately.
Based on our results, in the case of the frequency domain aspect of emotions, at positive ones, Alpha and both Beta frequencies seemed relevant at the left prefrontal hemisphere; and both Beta1 and Beta2 bandwidths increase its activation at the occipital right hemisphere. On the other hand, negative emotions presented a lateralization pattern towards the right prefrontal cortex and highlight the presence of Beta1 and Beta2 frequencies towards the left hemisphere over occipital regions. These results seem to point to a reverted lateralization pattern of frontal and posterior cortex when processing emotions. Our results agree with the frontal EEG asymmetry theory, described by Davidson et al. [5,10], reflecting the activation of motivational systems of approach and withdrawal, also verified by other authors [48][49][50][51]. In essence, we can conclude that there are interhemispheric differences in the processing of emotions; that this lateralization is also different depending on the emotional category; and that the processing of emotions not only falls on the prefrontal cortex, but rather there seems to be a neural network that expands along the entire cortex [52], and at all spectrum, excluding the low frequencies of the EEG [29,52,53]. The similarity on the scalp distribution of the different frequencies involved in the processing of positive and negative emotions suggests that, at least at the cortical level, there are no separate neural pathways for processing positive and negative emotions, but there is a network of cortical structures in charge of processing the valence, whose activity varies depending on the polarity of the emotion. Yet, a more in depth study of the relationships between regions and emotional conditions is necessary in order to draw meaningful conclusions about the brain emotional net.
Peripheral psychophysiological reactions constitute an important source of emotional information, therefore, researchers have focused on the different ANS measures. However, if the PNS and SNS are linked with the positive/approach and negative/withdrawal responses [54], respectively, is more diffuse. As for HRV measures, both our results and those of other authors, showed differences in their response to either emotional dimensions [20,55] and discrete emotions [21,56,57]. These differences are present in all levels of the HRV analysis, the time domain, the frequency domain and the Poincare plot. Nevertheless, and as it is customary in the study of emotions, there is no consensus as to which are the HRV variables most representative of the emotional state. In our case, no significant differences were found in any variable with the SI approach, and although there were differences at SD level, they did not show clear patterns across individuals. Moreover, classification results achieved using all HRV measures to differentiate positive and negative emotions were not different than chance; and although after feature selection, precisions of around 60% were reached, these values are far from the percentages achieved by Guo et al. [55] or Goshvarpour et al. [20], of 71.4% and 100%, respectively. Differences between our study and the mentioned studies, could be explained by the different methodologies used. Guo et al. applied principal component analysis in order to select the features used for the classification; therefore, not using the "real" HRV measures. On the other hand, Goshvarpour et al. used a neural network method, that although obtained better performance, it doesn't allow inferences to be made at the biological level as the interaction of the variables with each other and their weight in the classification is not known. Nevertheless, when separating the sample by gender, significant differences were found between positive and negative emotions in the Poincare variable SD2 in women and in the time variable NN50 in men, corroborating the existence of gender differences in young subjects [58,59]. The results suggest that in men, the differences are more evident in the short term, responding with greater intensity to positive stimuli; and on the contrary, women respond with greater intensity to negative stimuli, although the difference in this case is observable in the longer term. This may indicate a greater readiness of men for immediate response and greater adaptability to emotional stimulation in women. The variables found as informative, SD2 and NN50, are both regulated by both components of the ANS, so conjectures about the involvement or role that each division plays in the emotional response are not possible.
On the other hand, it has been proven that skin temperature measure could be used for positive and negative emotion discrimination [22,23]. The accepted explanation of the role of the skin temperature in the emotional process point out to vasoconstriction responses in order to mobilize blood into the muscular system to allow reaction to an aversive stimulus. Therefore, it seems that the dichotomy between the activation of the approach and withdrawal systems apply at the skin temperature level, however, both vasoconstriction and cooling and vasodilation and warming responses, are mediated by the SNS [23], pointing to the arousal scale. Therefore, although there are different patterns of response when it comes to processing positive and negative emotions, the functional organization of the activity of the ANS components remains unclear [14]. Nevertheless, our results showed an opposite pattern of the commonly accepted response [60], obtaining higher temperatures for negative than for positive emotions. Regarding gender, although there are differences in the thermoregulation of women and men [61,62], our results indicate that there are no gender differences at the skin blood volume regulation response when processing positive emotions, but, differences exist while negative stimuli occur; suggesting that women react more intensely to negative emotions than men.
Although the responses of the CNS and ANS systems to emotional stimuli and the relationships that exist between them are not known exactly; our results suggest that in both systems, independently, it is able to differentiate between positive and negative emotions. Koelstra et al. [26] used the detection of facial expressions and the EEG signal to classify emotions on the valence and arousal scales, demonstrating that classification performance improved when the two signals were combined, reaching percentages of 67.1% and 71.5%, respectively. Torres et al. [25] evaluated the combination of several biosignals for the detection of emotions on the valence and arousal scales. In the arousal scale, the best classification, 75%, was obtained after the combination of the EEG with physiological signals (heart rate, galnavic skin response (GSR), respiration and skin temperature); however, on the valence scale, the results of the combination of modalities did not improve the percentage achieved by the EEG alone, 58.75%. Using other theories of emotions, Verma and Tiwary [63] showed the best accuracy obtained in the literature when applying the multimodal approach (CNS: EEG; PNS: GSR, respiration, blood volume pressure and skin temperature) for the classification of discrete emotions, with an average value of 81.45% with a support vector machine. On the other hand, using the same database as Verma and Tiwary, but analyzing the emotional valence dimension, Chen et al. [64] obtained an accuracy of 83.98%. Our results improve classification performances obtained by other authors and conclude that the CNS per se (0.988 f1-score classification result for the KNN classifier) is most informative than the ANS data (HRV + skin temperature, 0.943 f1 score) in order to classify emotions regarding the valence dimension; and that the combination of the modalities (0.989 f1 score) does not significantly improve the results reached by the EEG alone. Authors like Jatupaiboon et al. [65] and Torres-Valencia et al. [66], also concluded, through other methodologies and emotional induction methods but addressing the effectiveness of multimodal emotional approach, that the EEG is the signal with the best capacity of emotional discrimination following the dimensional model of emotions. Therefore, based in our results we can discard the undifferentiated arousal theory that supports that the emotional response in the valence dimension is only reflected at the brain level and not at the peripheral/body level. Moreover, regarding the traditional idea that links the motivational systems of approach and withdrawal with the PNS and SNS systems, respectively, we cannot conclude for or against. Nevertheless, our results make even more evident the incompatibility of having a population-trained model that can be used for particular individuals, due to the high variability observed among subjects. Therefore, in order to make an accurate classification of positive and negative emotions it is necessary to train the computational model for each subject. However, it is possible to specify the most relevant features for the classification of emotions in the dimension of valence at the population level; thus decreasing the dimensionality of the model, the complexity of the system and the temporal cost of classification.
Regarding the gender factor in the classification of emotions, we found differences in the peripheral nervous system response, but not at the CNS. This suggests that emotions are processed in the same way for men and women at the brain level, but the body response is different [67]. However, one of the main drawbacks of our study is that women and men samples were not balanced, being considerably less number of women than men, and moreover, although our sample of 24 subjects is more than acceptable for this kind of studies, when splitting it into gender, the sample size is not representative of the population for SI analysis regarding the minimum size of 15 subjects stablished for proper classifications [68]. Therefore, although our results are encouraging, it would be necessary to enlarge the sample size in order to obtain more reliable results in terms of gender differences.
At the peripheral level, it seems that there were differences between the responses to opposing emotional valence stimuli. However, they did not seem to follow the expected pattern of 'fight or flight' or 'calm or safety' associated with the motivational systems of approach and withdrawal, that are believed to act at the level of the prefrontal EEG asymmetries. The components of the ANS are not activated in an 'all-or-none' fashion, rather each tissue is innervated differently by the sympathetic and parasympathetic pathways, which act independently of each other [14]. It is therefore difficult to attribute approaching or rejecting responses to specific components of the ANS. At this point, it is worth asking if motivational systems represent the same as the dimension of affective valence or if, on the contrary, they are different processes that do not always go hand in hand [69]. Conversely, it is probable that the arousal is influencing the physiological response, since although there were no significant differences in the arousal rating in the population, polarity existed in specific individuals.

Conclusions
We have assessed the activation states of the CNS, through EEG data, and ANS, through HRV and skin temperature measures, at positive and negative categories of the dimensional valence of emotion. Population differences were found at the frequency domain of electrical cortical signals showing a lateralization pattern towards the left hemisphere for the positive emotions and towards the right hemisphere for negative at anterior regions, and the inverse pattern at posterior regions. Peripheral bodily differences were also found at the skin temperature response, suggesting that valence dichotomy is also present at the ANS level. Moreover, gender differences presented at both ANS measures but not at the CNS suggest distinct mechanisms at the central and peripheral nervous systems and different gender predisposition. However, the multimodal classification approach did not seem to benefit emotion recognition in comparison with the existing EEG computational models. Our results bring more clarity to the debate between the theory-psychology and practice-engineering of emotions; but, more efforts are needed to finish solving the riddle of the psychobiology of emotional processes.
Author Contributions: J.S. developed the theoretical formalism, performed the experimentation and the analytic calculations, and wrote the manuscript. J.M.F. and E.F. encourage the investigation and supervised the project. This manuscript has been released as a Pre-Print at bioRxiv: https://doi.org/10.1101/638239. All authors have read and agreed to the published version of the manuscript.