Skin Admittance Measurement for Emotion Recognition: A Study over Frequency Sweep

Abstract

The electrodermal activity (EDA) is a reliable physiological signal for monitoring the sympathetic nervous system. Several studies have demonstrated that EDA can be a source of effective markers for the assessment of emotional states in humans. There are two main methods for measuring EDA: endosomatic (internal electrical source) and exosomatic (external electrical source). Even though the exosomatic approach is the most widely used, differences between alternating current (AC) and direct current (DC) methods and their implication in the emotional assessment field have not yet been deeply investigated. This paper aims at investigating how the admittance contribution of EDA, studied at different frequency sources, affects the EDA statistical power in inferring on the subject’s arousing level (neutral or aroused). To this extent, 40 healthy subjects underwent visual affective elicitations, including neutral and arousing levels, while EDA was gathered through DC and AC sources from 0 to 1 kHz. Results concern the accuracy of an automatic, EDA feature-based arousal recognition system for each frequency source. We show how the frequency of the external electrical source affects the accuracy of arousal recognition. This suggests a role of skin susceptance in the study of affective stimuli through electrodermal response.

1. Introduction

Emotions play a fundamental role in the daily life and a possible continuous monitoring of them could be very beneficial in understanding and managing personal well-being promoting mental healthy state [1,2]. In the last decades, several studies have been proposed to increase knowledge on emotion recognition, and, consequently, develop an automatic emotional detection systems. Even though a scientific definition of emotion is still controversial, many models of emotions have been developed [3,4,5,6,7]. Among them, the circumplex model of affect [3] describes emotions as a combination of an arousal level (i.e., the intensity of the emotion perception) and a valence level (i.e., the degree of pleasantness). According to these emotional models, many human–machine systems have been developed in order to automatically recognize humans’ affective and mood states [8,9,10,11,12,13,14,15,16,17,18,19] by interpreting physiological changes as a response to an external triggering event. To this extent, several psychophysiological features extracted from different peripheral biosignals have been widely used in the literature [20,21,22,23]. Some of the most commonly used physiological signs in affective recognition are: electrocardiogram and related heart rate variability series, electrodermal activity (EDA), respiration, muscle activity, peripheral temperature, eye gaze, as well as brain activity [24,25,26,27,28,29].

Among these, EDA has been widely used to assess the arousal level in humans because of its ability to quantify changes in the sympathetic nervous system (SNS) [30,31]. EDA signal is comprised of a low-frequency tonic component, and a higher frequency phasic component [30,32]. Anatomically, EDA changes are due to the sudomotor nerve activity (SMNA), which is part of the SNS, that directly control the eccrine sweat glands [30]. Therefore, EDA can be easily monitored through voltage/current measures between two fingers, where there is a higher concentration of the eccrine sweat glands with respect to other body sites [30].

In this context and aiming at continuously monitoring the EDA signal in a ecological scenario, wearable monitoring systems are the most interesting and advantageous devices. In fact, wearable sensors are greatly valued due to their comfort, portability, non-invasiveness, and their wireless communication capabilities with either a computer, a mobile embedded system or other wearable sensors [33,34,35,36,37].

Referring to EDA measure, three standard methodologies are usually employed. The first one is called endosomatic measurement. This is rarely used and consists in measuring, directly on the skin, the potential difference between two skin sites, in a passive way. It does not need special amplifiers and coupling electrical circuits. Although it is a quite unknown bioprocess, it is accepted that changes in skin potential during sympathetic activity may be provoked by the sodium reabsorption across the duct walls and the consequential change of the ionic potential in the sweat ducts [30].

The other two methods are based on an exosomatic approach, i.e., a small external current is directly injected into the skin. Consequently, two different methods are used: the direct current method (DC) and the alternating current method (AC) at different frequency levels. Generally, they perform a measure of resistance (impedance) or conductance (admittance: “conductance + j susceptance”), where j is the imaginary unit. When DC source is used, resistance and conductance are inversely related. Instead, when AC source is used, the inverse paradigm between resistance and conductance is not valid, but can be applied to the complex impedance and complex admittance. Moreover, in the AC regime, the impedance is represented by a circuit comprised of a resistor-capacitor in series, instead the admittance is represented by a parallel circuit (of a resistor and a capacitor). The Parallel equivalent should be preferred since ionic conduction and polarization are in parallel in biological tissues. More in detail, the skin can be considered as a dielectric whose permittivity and conductivity are complex due to free charges and the AC losses of bound charges [38].

Although AC methods allow measuring capacity changes in the electrodermal responses, DC procedures are the most implemented [39]. Within DC methodology, even if much effort has been spent to standardize, an agreement concerning the use of either constant-voltage or constant-current scheme has not yet been achieved. Therefore, different designs can be found. In constant-voltage sources, the conductance of the skin can be directly measured as an output of the circuit without the need of any further transformation. However, constant-current sources provide more stability and exhibit less tolerance, but, in this method, much attention must be payed to possible damage to the sweat ducts due to the injected current through a small area of the skin [34].

Unfortunately, a very low number of studies have been published on the difference among AC and DC stimulation in EDA measurement. One of the most interesting works [39] showed that, when using an AC source at 88 Hz, the major contribution to EDA is given by the conductance term and not by the susceptance. More specifically, the authors found that, at the instant of the conductance response, there were no susceptance responses, and this could indicate the absence of a significant capacitance in the sweat ducts. Moreover, to the best of our knowledge, in the current literature, no specific wearable systems, which are able to perform both DC and AC electrical stimulations, have already been reported. As a matter of fact, in a previous study, we have already proposed a fabric-based sensorized glove [40]. However, this system was also able to continuously record the EDA using only a DC source.

Starting from the above considerations and from the recommendations provided in [41,42,43], we propose a novel textile wearable system, which is able to perform an exosomatic EDA measurement using both AC and DC methods. Using this novel device, our study aims at investigating whether the admittance contribution at different frequency sources (in the range from DC to 1 kHz of AC) could affect the ability of EDA of inferring the central state during emotional stimulation. To this end, we designed an experimental paradigm using visual affective stimuli, and we developed an automatic arousal recognition system, in order to test potential differences in inferring the arousal state of the two approaches.

2. Materials and Methods

Exosomatic EDA measures changes in the electrical conductance (DC method) or admittance (AC method) due to both the psycho-physiological state of a person and the response to an external event. The EDA signal has a frequency bandwidth of 0–2 Hz [44] and, for a correct analysis, should be decomposed in two sub-components containing different and complementary information: a low-frequency tonic component, which reflects the subject’s general psycho-physiological state and its autonomic regulation [30], and a phasic component, which is the superposed higher-frequency change directly related to an external stimulus [45]. A frequent issue in the decomposition process consists in the overlapping of consecutive phasic responses, which occur in the case of inter-stimulus intervals less than around 10–20 s [46,47].

Measurement of EDA changes provides evidence of eccrine sweat gland activity. The eccrine sweat glands are innervated by sympathetic fibers, and, in normal ambient temperatures, palmar, finger (or plantar) glands reflect responses to psychological rather than thermoregulatory stimuli [48]. Therefore, EDA is considered as an ideal way to monitor the autonomic nervous system and, more specifically, its sympathetic branch [30].

In this section, we first report on the EDA acquisition system prototype and then briefly describe details on the cvxEDA models, which is presented in [49]. Note that this method is able to discern overlapping consecutive electrodermal responses (EDRs), likely to be present in the case of an inter-stimulus interval shorter than the EDR recovery time.

2.1. Multi-Frequency Sensorized Glove

EDA is acquired by a glove where integrated textile electrodes were placed at the distal phalanges of the index and middle fingers (Figure 1). Textile electrodes, provided by Smartex s.r.l. [40] (Pisa, Italy), are made up of 80% polyester yarn knitted with 20% steel wire, with a dimension of 1 × 2 cm. In one of our previous studies [40], we performed a comparison of textile sensors with Ag/AgCl electrodes demonstrating comparable performance. More specifically, reported results on electrode characterization, performed by means of the voltage-current characteristics, and its electric impedance showed that textile electrode achieves a good electrical and thermal coupling with biological site. Moreover, the use of a wearable textile system exhibits several advantages in terms of portability and usability for long-term monitoring, and gives minimal constraints. This latter characteristic is very significant when the system is used in an ecological environment.

The analog front-end of the designed electronics, which is responsible for measuring the DC and AC exosomatic EDA, is based on a variable-gain current-to-voltage operational amplifier. The electric current injected into the skin is variable and programmable (from 0 to 1 kHz), and for this purpose we used the AD9833 provided by Analog Device [50] (Norwood, MA, USA). This chip is a low power, programmable waveform generator, which is used to switch up the frequency of the skin electrical stimulation among 0 (i.e., DC), 10, 100 and 1000 Hz [39]. Moreover, a low-pass filter (cutoff frequency of 3 Hz) and a further amplification stage were applied to the raw EDA data before the successive digitalization step.

The preprocessed EDA signal was digitally converted with a sampling frequency of 15 kHz, thanks to the 12-bit analog-to-digital converter built in the Texas Instrumen (Dallas, TX, USA) MSP430 microcontroller (Figure 2). The MSP430fxx family of microcontrollers are designed to be low cost and, specifically, low power consumption embedded applications. It is a very popular choice especially in wireless networking systems and it is built around a 16-bit RISC (Reduced instruction set computing) CPU. In our prototype, we used the MSP430x6xx Series, which are able to run up to 25 MHz, have up to 512 KB flash memory and up to 66 KB RAM. Moreover, this series includes an innovative power management module for optimal power consumption [51].

Moreover, wireless communication was implemented by an Xbee module (Minnetonka, MN, USA) connected to the USART (Universal Synchronous Receiver-Transmitter) of the MSP430. Specifically, it was used to exchange data between the transceiver and a dedicated multi-platform software application. Finally, a lithium-polymer battery with a voltage of 3.7 V and a capacity of around 750 mAh was chosen as power supply [34]. The large capacitance of the battery allows a long-term-continuous monitoring, and it is an essential characteristic to stream the data wirelessly. An external circuit was developed to support the rechargeable battery through a USB port. Finally, a voltage regulator is responsible for supplying 3.0 volts from the battery to all the components of the device.

2.2. EDA Processing Using cvxEDA Algorithm

CvxEDA proposed a representation of the phasic responses as the output of a linear time-invariant system to a sparse non-negative driver signal. The model assumes that the observed EDA (y) is the sum of the phasic activity (r), a slow tonic component (t), and an additive independent and identically distributed zero-average Gaussian noise term ϵ:

$$y=r+t+ϵ.$$

(1)

Physiologically-plausible characteristics (temporal scale and smoothness) of the tonic input signal can be achieved by means of a cubic spline with equally-spaced knots every 10 s, an offset and a linear trend term:

$$t=B\ell +Cd,$$

(2)

where B is a tall matrix whose columns are cubic B-spline basis functions, ℓ is the vector of spline coefficients, C is an $N\times 2$ matrix (where N is the length of the EDA time series) with $C_{i,1}=1$ and $C_{i,2}=i/N$, d is a $2\times 1$ vector with the offset and slope coefficients for the linear trend.

The phasic component is the result of a convolution between the SMNA, p, and an impulse response $h(t)$ shaped like a biexponential Bateman function [52,53,54]:

$$h(t)=(e^{-\frac{t}{{\tau }_1}}-e^{-\frac{t}{{\tau }_2}})u(t)$$

(3)

where ${\tau }_1$ and ${\tau }_2$ are, respectively, the slow and the fast time constants of the phasic curve shape, and $u(t)$ is the unitary step function.

Taking the Laplace transform of Equation (3) and then its discrete-time approximation with sampling time δ (using a bilinear transformation), we obtain an autoregressive moving average (ARMA) model (see details in [49]) that can be represented in matrix form as

$$H=M^{-1}A,$$

(4)

where M and A are tridiagonal matrices with the MA and AR coefficients along the diagonals. Using an auxiliary variable q such that

$$q=A^{-1}p,r=Mq,$$

(5)

we write the final observation model as

$$y=Mq+B\ell +Cd+ϵ.$$

(6)

Given the EDA model Equation (6), the goal is to identify the maximum a posteriori (MAP) neural driver SMNA (p) and tonic component (t) parametrized by $[q,\ell ,d]$, for the measured EDA signal (y). CvxEDA rewrites the MAP problem as a constrained minimization Quadratic Programming (QP) convex problem (see details in [49,55]):

$$\begin{array}{cc}\hfill \text{minimize}& \frac{1}{2}{∥Mq+B\ell +Cd-y∥}_2^2+\alpha {∥Aq∥}_1+\frac{\gamma }{2}{∥\ell ∥}_2^2\hfill \\ \hfill \text{subj}.\text{to}& Aq\ge 0.\hfill \end{array}$$

(7)

This optimization problem can be re-written in the standard QP form and solved efficiently using one of the many sparse-QP solvers available. After finding the optimal $[q,\ell ,d]$, the tonic component t can be derived from Equation (2) while the sudomotor nerve activity driving the phasic component can be easily found as $p=Aq$.

The objective function Equation (7) to be minimized is a quadratic measure of misfit or prediction error between the observed data and the values predicted by the model. Moreover, the prior knowledge about the spiking sparse nature and nonnegativity of the SMNA (p) and the smoothness of the tonic component are accounted for by the regularizing terms and the constraint.

The strength of the penalty is regulated by α and γ terms. A sparser estimate is yielded by large values of α. Concerning γ, higher values mean a stronger penalization of ℓ, i.e., a smoother tonic curve. Of note, fixed values of ${\tau }_1=0.7s$, ${\tau }_2=3.0s$, $\alpha =0.008$ and $\gamma =0.01$, which were chosen during previous exploratory tests on separate data, were employed throughout this analysis.

CvxEDA algorithm is implemented in Matlab language and the software is available online [56].

2.3. Experimental Protocol

Forty healthy subjects were enrolled in the experiment, aged 26 ± 4 (18 females). All subjects gave written informed consent before taking part in the study, which was approved by the local Ethics Committee. The experiment was designed as following:

• initial resting phase of 1 min;
• maximal expiration task phase of about 1 min;
• affective visual stimulation phase of 2 min;
• final resting phase of 1 min;

(The two elicitation phases will be described in detail in the next sub-sections.) Subjects were comfortably seated in an acoustically insulated room in front of a computer screen while their EDA was recorded using the presented acquisition system.

Textile electrodes were placed for 10 min before the acquisition for achieving a stable skin/dry-electrode coupling and limiting the temporal and thermic effect [57].

Note that the group of 40 healthy subjects was split into four subgroups, each of which comprised of 10 subjects. Each subgroup were acquired with a different exosomathic method such as DC (group 1), AC with a frequency of 10 Hz (group 2), AC with a frequency of 100 Hz (group 3) and AC with a frequency of 1 kHz (group 4).

2.3.1. Maximal Expiration Task

In this session of the experiment, all of the 40 subjects performed a forced maximal expiration task [58], in which they were asked to breathe out with the maximum possible intensity in order to trigger the SNS-mediated expiration reflex.

After the initial resting state session, the subjects breathe normally and rest in front of the computer monitor for about 20 s. Then, they had to perform a deep expiration twice with an inter-stimulus interval of about 20 s, after a neutral visual input on the screen.

The use of the forced expiration task is justified by the need of having a stimulus whose EDA response was as reliable and objective as possible. In fact, previous studies have demonstrated that this stimulation is a reliable way to evoke phasic responses unaffected by emotional change with better reproducibility, less habituation, and more stable waveform patterns than other experimental paradigms (including electrical) [58]. In this way, the presence of at least one phasic response after each stimulus was ascertained. Therefore, we could investigate whether the cvxEDA algorithm was able to identify each phasic response for each acquisition method.

2.3.2. Affective Visual Stimulation

In the second elicitation session, each group of 10 participants was stimulated by projecting on a screen images selected from the official IAPS (International Affective Picture System) database [59]. The IAPS dataset is a collection of images ranked in terms of arousal (i.e., intensity of perception) and valence (pleasantness of perception). This protocol session is designed to assess the pattern recognition system ability on each data group (i.e., of each method) to correctly classify stimulations with different arousal content and provide meaningful information about SNS activation.

The slideshow timeline consists of three neutral images, six aroused images and three other neutral images. Each image was shown for 10 s.

2.4. EDA Analysis and Classification Procedure

For each dataset, the convex-optimization-based EDA model (cvxEDA) described in Section 2.2 was applied to each time series after a Z-score normalization (this is not a mandatory step before applying the cvxEDA algorithm, but an increase in the speed of the QP-solver). Concerning the Impulse Response Function (IRF) parameters considered for this study, values of ${\tau }_2=0.7\text{s}$, ${\tau }_1=0.7\text{s}$, $\alpha =0.4$ and $\gamma =0.01$ were employed throughout this analysis, according to previous exploratory tests on separate data.

In the respiratory stimulation dataset, the presence of an estimated burst of SMNA activity was verified in each 5 s time window following a stimulus onset, in order to prove the model’s ability to correctly detect partially overlapped phasic responses.

As summarized in

Table 1. List of features extracted from Electrodermal Activity (EDA) phasic and tonic components.

Feature	Description
Npeak	number of significant SMNA peaks wrw
AUC	Area under curve of reconstructed phasic signal wrw ($\mathsf{\mu }Ss$)
peak	maximum amplitude of significant peaks of SMNA signal wrw 1 ($\mathsf{\mu }S$)
MeanTonic	Mean value of the tonic component within each image time window ($\mathsf{\mu }S$)

wrw= within response window (i.e., 5 s after stimulus onset).

, we segmented each signal in correspondence to each IAPS image time window, and we extracted several features from both the tonic and phasic component.

Classification Procedure

The feature set, extracted from each single IAPS image, was used as the input of a pattern recognition algorithm in order to classify the two arousal levels, according to the IAPS rates. The supervised classification of the feature set was implemented following a Leave-One-Subject-Out procedure (LOSO) applied to a K-nearest neighbors (K-NN)-based classifier. For each of the N iterations (where N is the total number of participants), the whole dataset was split into a training set including $(N-1)$ subjects and a test set including the cvxEDA feature values of the the remaining subject $N_{th}$. Moreover, for each iteration of the LOSO scheme, a feature selection procedure was performed in order to identify the combination of parameters that resulted in the highest recognition accuracy within the training set examples. Each selected feature constituted a single dimension of the feature space. The LOSO pattern recognition procedure is illustrated in Figure 3.

3. Results

According to the cvxEDA model, all EDA data (Figure 4a) were decomposed into two signals, a sparse component p and a smooth component t. Of note, we interpret p as the activity of the sudomotor nerve (Figure 4b), and t as the tonic level (Figure 4c)).

3.1. Maximal Expiration Task Results

We performed both a visual and a statistical inspection of time series to verify whether the effectiveness of the protocol in eliciting phasic responses was confirmed for all different kinds of acquisition method (DC and AC).

After the application of the cvxEDA model, we considered a time windows of 5 s after the onset of each expiration task, and we looked for peaks of the SMNA signal (in fact, the phasic response is defined as the part of the signal arises within a predefined response window of 1–5 s [30,45]). Of note, due to the stimulus intervals of about 20 s, no overlap between consecutive responses occurred.

Results of an intersubject analysis showed that cvxEDA was able to correctly detect the corresponding phasic peak response over $97.5\%$ of the respiratory stimuli. Moreover, a visual inspection of the small percentage of cases that were not correctly identified revealed a very low signal-to-noise ratio of the signal.

3.2. Automatic Arousal Recognition Results

Results of the arousal-level-classification-procedure on the four datasets, namely, DC, AC 10 Hz, AC 100 Hz, AC 1 kHz, are shown in

Table 2. Confusion matrix of Neutral vs. Arousal images using an cvxEDA feature set extracted with a Direct Current (DC) source.

K-NN (DC)	Neutral	Arousal
Neutral	63.33%	35.00%
Arousal	36.67%	65.00%

K-NN means K-nearest neighbors.

,

Table 3. Confusion matrix of Neutral vs. Arousal images using cvxEDA feature set extracted with Alternating Current (AC) source at 10 Hz.

K-NN (AC 10 Hz)	Neutral	Arousal
Neutral	65.00%	38.33%
Arousal	35.00%	61.67%

,

Table 4. Confusion matrix of Neutral vs. Arousal images using cvxEDA feature set extracted with AC source at 100 Hz.

K-NN (AC 100 Hz)	Neutral	Arousal
Neutral	68.33%	25.00%
Arousal	31.67%	75.00%

and

Table 5. Confusion matrix of Neutral vs. Arousal images using cvxEDA feature set extracted with AC source at 1 kHz.

K-NN (AC 1 kz)	Neutral	Arousal
Neutral	63.33%	38.33%
Arousal	36.67%	61.67%

. The recognition accuracy is reported in the form of a confusion matrix. An element $r_{ij}$ of the confusion matrix indicates a percentage of mismatches, i.e., how many times a pattern belonging to class i was erroneously classified as belonging to class j. Terms $r_{ij}$ on the main diagonal of the confusion matrix correspond to correct classifications.

Both DC and AC measures did not show very high average recognition accuracy. However, it is worth noting that, using 100 Hz of the frequency current source, we obtain an average accuracy significantly higher than in the other cases. More specifically, using DC, 10 Hz and 1 kHz, the average accuracy was in the range of 62.5% to 63.34%, whereas at 100 Hz, the pattern recognition system showed an accuracy of 71.67%.

4. Discussions and Conclusions

In this study, we proposed a novel wearable EDA acquisition system prototype. It consisted of a sensorized glove provided with textile electrodes at the fingertips able to acquire the exosomatic EDA using both DC and AC methods. In order to test the usability of the novel sensorized glove and to investigate about possible differences between DC and AC stimulation (i.e., 10 Hz, 100 Hz, 1 kHz), we designed an experimental paradigm where 40 healthy subjects were stimulated by means of a mechanical expiration task and visual affective stimuli selected from the IAPS database. From this collection of pictures ranked in terms of arousal and valence level, two groups of images were selected: a group of neutral images and a group of negative aroused images.

The EDA signals were analyzed by means of the cvxEDA model [49]. The cvxEDA algorithm is based on the three concepts of sparsity, Bayesian statistics and convex optimization. It provides a decomposition of the EDA in its two components, i.e., phasic and tonic, and estimates the sudomotor nerve activity that control the eccrine sweat process, giving a window into the sympathetic nerve activity.

Results from the application of the cvxEDA algorithm showed no differences in the identification of the phasic peak response after the deep respiration stimulus among the DC and three AC methods. In fact, over 97% of the peaks were identified in the SMNA signal in the time response window of 5 s after the stimulus onset (i.e., directly evoked by the stimulus [30,45]). We could conclude that all of the investigated methods for the exosomatic measurement of the EDA reliably measure the phasic responses to eliciting stimuli.

Considering the four groups of data separately (i.e., DC, AC 10 Hz, AC 100 Hz, AC 1 kHz), in the second part of the experiment, we investigated possible differences in inferring the arousal state. Specifically, we performed a classification procedure of the arousal levels in the four groups of signals. Results showed that an alternating current method at 100 Hz could improve the arousal recognition accuracy up to 71% (while other acquisition modalities did not overcome an average accuracy of 63.5%). These results suggested that not only the skin conductance plays an important role in the electrodermal affective response, but also the susceptance (i.e., imaginary part of the skin impedance) may contain relevant information about the SNS. Moreover, this relationship between AC frequency and recognition accuracy is strongly nonlinear due to the nonlinear relationship between skin impedance, and amplitude and frequency of the external electrical source [60]. Specifically, it is well-known that the current density under a surface plate electrode could be non-uniform, and electrode surfaces present fractal properties creating local areas of different current densities. The onset of non-linearity may therefore be gradual, and start very early at very limited areas on the electrode surface (e.g., it has been shown that very weak non-linearity is measurable at voltages than 100 mV). Hence, it may be difficult to differentiate between the non-linearity of the electrode processes and the tissue processes [61].

We are aware that works stated that the role of the susceptance is less important with respect to the conductance at low frequency [39], but our results seem to indicate that a significant difference in EDA results are frequency dependent even more when they are not mechanical but emotionally evoked.

In other words, we assume that it could be feasible that emotional stimuli may involve a capacitive component in the medium under investigation that has a bigger contribution at 100 Hz.

Moreover, we should take into account that Ohm’s law, given by $J=\sigma E$, in such a medium could be not valid, and it may be useful to treat σ as a complex quantity in order to incorporate dielectric losses and frequency dependence, therefore defining σ as: $\sigma ={\sigma }^{\prime }+j{\sigma }^{″}$.

Future works will investigate the real and imaginary components of the admittance in the analysis of the EDA dynamics by involving time varying methods that could highlight the nonlinear nature of the electrodermal response.