Emotional States versus Mental Heart Rate Component Monitored via Wearables

Featured Application

This technology is an effort towards real-life and real-time emotion monitoring, using mental heart rate component as a derived physiological signal. In this way, an ecological momentary intervention system is developed that allows mental health experts to advise users based upon objective data. This application can allow early detection and prevention of mental disease.

Abstract

Psychiatric illnesses are estimated to account for over 15% of the burden of disease, which is more than all kinds of cancer together. Since mental disease is often preceded by issues in emotion processing, a method to objectively measure emotions in daily life would be needed. The goal of this research is to investigate the possibilities of mental heart rate component, assessed with a real-time individualized algorithm that decomposes total heart rate in a physical, basal, and mental component, to classify discrete emotions. For this aim, twenty participants committed to wearing a wristband 24/7 for three months and to label the occurrence of fourteen emotions on their smartphones. In total, 1255 labels were added. The dynamics of the mental heart rate component responses to emotions were identified via data-based mechanistic transfer function models. For the classification, the numerator and denominator model orders and parameters, the four features that define transfer function models, were used as features in a support vector machine classifier. This resulted in an average classification accuracy of the mental heart rate responses of 51.1% over all participants, compared to a random classifier with an average accuracy of 28.5%. We concluded that the dynamics of emotions are not only highly variable between individuals, but that they are also time varying on an individual basis. To increase accuracy, more and higher quality labels are indispensable.

1. Introduction

Every year, 20% of the world’s population suffers from a diagnosable form of mental problems [1]. Additionally, the number of individuals with a mental disorder shows an increasing trend. Between 1978 and 2015, the prevalence of mental disorders increased by 1.179% [2]. The consequences concerning the burden of disease and costs to the economy are significant, accounting for more than 15% of the total disease burden worldwide. This is more than all kinds of cancer together [3,4]. On the other hand, countries were in 2017 spending less than 2% of their national health budgets on mental health, which is a severe underfunding [5]. The good news is that most mental illnesses are preventable and treatable. By getting a better understanding of mental illness, one could work towards an early detection and prevention of problems [3]. The earlier the treatment can start, the less the mental disorder will develop and the smaller the risk of a comorbid disorder [6].

Mental disease is associated with disturbances in emotion processing or emotional behaviour, hence, a method to objectively measure emotions in daily life would be of added value [7]. When one or more parts of the emotion processing are impaired, the adaptive function of emotions gets disturbed [8]. These disturbances form the core of several psychopathologies, for example, depression or anxiety disorders [8,9]. Although most psychopathologies reflect impairments in emotion processing or emotional responding, they widely vary in nature. Two examples will illustrate this. Firstly, schizophrenia is characterized by a reduced emotional expression of a ‘normal’ experienced emotion. Anxiety disorders, on the other hand, are defined as an excess of fear and/or anxiety experiences, even without a threatening stimulus [8].

For many decades, researchers tried to find a method to objectively measure individuals’ emotional states both in laboratory circumstances and in real-life conditions. Most of the research performed so far in the field of stress and emotions is performed in laboratory settings, while real-life validation often lacks. Since laboratory settings can differ strongly from real-life situations, the classification accuracies achieved in lab conditions cannot be generalized to real-life situations. How well a technology serves in developing mobile applications, depends on how well the experimental setup matches the real-life situation [10]. This is called the issue of ecological validity [11]. Despite the importance of real-life measurements, there are issues that may directly complicate their interpretation. Sometimes subjects simply forget to label their emotional experience. One also needs to consider the number of emotions that must be labelled. As a researcher, you can ask to label a certain number of emotions, but in practice, the participant experiences way more emotions that are not labelled or reported. Moreover, contextual factors, such as smoking, drinking, drug use, etc., are often not labelled but can influence the physiological signal interpretation drastically. Nevertheless, asking for contextual information complicates the study and causes an extra burden for the users during the development or use of the application [10].

Emotion recognition methods can be divided into three groups, namely the ones based on self-report, behavioural parameters, and physiological signals [12]. Self-report is the simplest method and includes asking how someone feels. Nevertheless, self-report of emotions often lacks accuracy, mainly because the possible answers to a questionnaire can influence the given answer, individuals are not always aware of what they are feeling or do not want to give the right answers [13]. Methods based on behavioural parameters, such as facial expression, posture, and speech analysis, are intuitive and effectual. However, in some situations, people can mask them. Therefore, behavioural methods are not always accurate and reliable [12]. Lastly, physiological signals have a higher potential to lead to objective results, because they are an immediate outcome of the human autonomic nervous system (ANS) and the endocrine system, which directly enable emotional responses [12,14]. Different from behavioural parameters, the assignment to a certain emotion is not influenced by the subjective perception of the investigator. As the ANS cannot be controlled actively or consciously, it is impossible to mask or hide an emotional response. This makes physiological ANS responses more robust reflections of emotional states [15]. However, many of these measurement techniques, among others electromyography (EMG) [15], electroencephalography (EEG) [16], electrocardiography (ECG) [17] and functional magnetic resonance imaging (fMRI) [18], are not comfortable and/or possible for real-life monitoring of emotions, since the equipment can restrict free movement or requires too often battery recharging and is therefore not convenient for ambulatory assessment.

As in many other research fields, machine learning (ML) and deep learning (DL) techniques are becoming more popular, instead of the traditional statistical analysis methods. In ML methods, the features for the classifier are hand-crafted. However, this kind of method requires many labelled data, which is the biggest challenge in the field of emotion recognition. Furthermore, feature extraction and selection are very time-consuming and laborious. DL algorithms, on the other hand, can extract and select features automatically [19]. An important disadvantage of most ML and DL models is that they have so many parameters, that it is impossible to estimate them in real-time on each individual subject. Another important disadvantage of DL methods is that they are purely black-box, meaning that it is extremely hard to understand the selected features [20]. Thus, also on the analysis level, there are still many challenges in the field of emotion recognition.

The Mindstretch mobile application was developed by a spin-off company of KU Leuven (Belgium) to monitor individuals’ mental energy use and recovery [21]. These data are used to gain more insights into individuals’ mental energy balance and to early detect or even prevent serious mental health problems. Mental energy can be described as the metabolic energy used to perform cognitive tasks [22,23]. It is calculated based on heart rate (HR) and acceleration data collected by any Fitbit wristband on the market. Mindstretch takes into account that the responses of living organisms are complex, individually different, time-varying and dynamic [24]. The human body is in constant need of energy to provide the basal metabolism, the immune system and the thermal regulation of energy while performing mental and physical activities and to maintain homeostasis. For most people during most of their lifetime, this energy is provided by aerobic metabolism. This means that oxygen is used to produce metabolic energy. This oxygen is pumped to the cells throughout the body via the cardiovascular system. HR measurements can thus be used as a measure for oxygen consumption and consequently also for metabolic energy production [25]. Metabolic energy consumption can be designated into four categories. The first category is basal metabolism, which can be considered the minimal energy consumption of the body, keeping all organs functioning when no need for thermal control of the body in rest [26,27]. The magnitude of the basal component depends on the age, weight, metabolism, etc., of the individual. Secondly, energy is used for thermoregulation. The body temperature needs to be kept constant to make sure that cell processes take place under optimal conditions. Thirdly, physical activity also requires metabolic energy. This component can be estimated based on the dynamic responses of heart rate to the activity monitored by the acceleration data [24,28]. Lastly, there is metabolic energy used for the category ‘mental activity’. Especially when stressed or emotional, energy consumption can rise significantly [21]. The amount of mental metabolic energy consumption can be calculated by subtracting the basal, thermoregulation and physical energy consumption components from the total energy consumption [21]. In this research, mental HR ([bpm]) is linked to mental energy, which is the component of the total HR which is used for mental performance. Piette proved that metabolic energy can be decomposed into its different components and that the mental energy component and mental HR can be used interchangeably in experiments that want to assess cognitive effort [24]. This study decomposes the average HR during the measurement interval, and thus strongly differs from other studies that use heart rate variability (HRV) to monitor emotional states.

The objective of this paper is to use the metabolic mental heart rate component linked to emotions in a so-called ecological momentary intervention (EMI) application. In this case, it means the capturing of emotions in real time during daily life settings. The ‘momentary’ aspect means that the assessment happens continuously over successive points in time [29,30]. In traditional EMIs, prompts are sent by a smartphone as a reminder for the users to label their emotions, additionally to advice that could increase the users’ well-being in the real-life environment [30]. Empirical results in this field of research suggest that EMI can be preventive in the development of depression and other mental illnesses [31]. The emotion recognition based on mental heart rate should be included in the EMI so that labels should not be added manually anymore by the user. Getting information about emotions from questionnaires is subjective, time and cost intensive. The current technology would allow to automatically retrieve emotion data, based on mental heart rate measured with a simple Fitbit bracelet.

2. Materials and Methods

2.1. Data Collection

2.1.1. Participants

Twenty Belgian participants, of which thirteen were women and seven men, took part in the experiment. All participants had Dutch as their native language. The average age of the participants was 24 ± 7 years. All participants were in good physical and mental health. Only one participant had exercise asthma. The other participants did not have any respiratory malfunctions. None of the participants had any form of metabolic or cardiovascular malfunction. All participants confirmed to participate in the study via an Informed Consent Form (ICF). The Social and Societal Ethics Committee of the KU Leuven (G-2021-4184-R3(MAR)) approved the experiment.

2.1.2. Materials

The participants all wore a Fitbit wristband and owned a smartphone on which they were asked to install the Fitbit and Mindstretch applications. Eighteen participants were using Android software, two were using IOS. Fifteen participants were wearing a Fitbit Inspire 2, two were using a Fitbit Versa 2, two a Fitbit Versa 3 and one participant a Fitbit Charge 2. All Fitbit wristbands were equipped with a 3-axis accelerometer to measure steps per minute and an optical heart rate monitor to measure the heart rate ([bpm]). The optical heart rate monitor works according to the polyplethysmography (PPG) principle, using green visible light. HR was sampled at 5 s intervals, except during exercise, then the sampling time was increased to 1 s intervals [32,33,34]. The Mindstretch algorithm used minute averages of these data as raw data to calculate the mental energy use and recovery per minute.

2.1.3. Experimental Methods

Three days and three nights before the official start of the experiment, the participants started to wear their Fitbit trackers. This was needed for the Mindstretch algorithm to adjust to the individual. The data collected during these three days were not used for further analysis. For three months, the participants were asked to continuously wear the Fitbit 24/7 and charge it at regular times to avoid long periods of empty battery.

When they experienced one out of thirteen predefined emotions, they needed to label it, by ticking an icon on the labelling screen included in Mindstretch (Figure 1). The set of emotions was based on the selection in the experiment of Saarimäki et al. (2018) [18] and includes (1) surprise, (2) happiness, (3) disgust, (4) fear, (5) sadness, (6) anger, (7) shame, (8) pride, (9) longing, (10) guilt, (11) love, (12) gratitude, (13) despair. However, if the participant had the feeling that an important emotion in the set of emotions was missing, he/she could create an icon to label this emotion. To make sure that all participants got the same understanding of each emotion label, story lines were used as reference. Each emotion in the Informed Consent Form was described by four short stories, that are known to evoke that particular emotion. The story lines of Saarimäki et al. (2018) were used for that aim [18]. The researcher advised labelling the emotion right after the experience to collect as accurately as possible the start and stop times of the emotion and to minimize the retrospective recall bias [35], but labels could also be added later.

2.2. Modelling

For every participant, an individual dataset resulted with the added emotion labels, their start and stop times and mental HR ([bpm]) collected during the whole course of the experiment. Before the labelled data were modelled, the mental HR data were prepared for further analysis according to the following procedure. The labelled events that overlapped in time were removed. Besides, when an emotion occurred in the period just right before and after another emotion, both emotions were also removed from the dataset. The times right before and after the emotion are defined as having the same duration as the emotion itself. For further pre-processing, the ‘prepz’ function of the CAPTAIN toolbox was used, for smoothing and scaling the mental HR over time, and to get rid of the general trends in the signal [36]. The data were visually presented and inter- and intra-individual differences in mental heart rate responses for the different emotions were observed.

2.2.1. Mental Heart Rate Calculation

The total HR was measured by the wristbands. The basal HR component was estimated before the start of the experiment, by a continuous measurement of the individuals, and was also updated during the experiment. Lastly, the physical component was estimated by a linear autoregressive (AR) SISO model, in which step count data, assessed with an accelerometer, were used as an input, and the physical HR as the model output.

$$H{R}_{Mental}=H{R}_{Total}-H{R}_{Basal}-H{R}_{Physical}-H{R}_{Thermal}$$

(1)

The wearable measures heart rate and activity (steps) which are used as inputs to the data-based-mechanistic (DBM) model as used by Piette [24]. Most of the metabolic energy for most people during their life is produced in an aerobic way. This means that HR, pumping the oxygen to the cell level via de lungs, indicates how much aerobic metabolic energy can be produced. The total HR can be decomposed into a basal component, a physical component and a mental component. The basal one is the amount of HR required to keep the body functioning. The physical one is the part of the total HR to perform a specific physical activity such as running, walking, writing, etc. The mental component is considered as the total HR minus the basal one and the physical one. These different components can be estimated in real time (Figure 2). When the wearable measures activity and HR, the algorithm can detect the minimal HR or basal component for the individual subject by monitoring 24/7. As soon as the subject leaves the bed, the wearable measures the physical activity in time and the response of the HR to this physical performance. From the dynamics of those two signals, the physical component of the total HR can be calculated. The same happens when the outside temperature is varying, and the HR is responding to this by controlling body temperature. From the dynamics, the thermal component is calculated. It has been shown that the mental component of HR correlates with the blood hormone noradrenaline [27,28,37].

Next, this data-based mechanistic (DBM) model was identified for every labelled emotion event of every participant, describing the dynamic mental HR response to an emotion. In this approach, abstract model parameters were estimated based on the mental HR data, which were given a physical or biological meaning afterwards [38,39]. More concretely, single-input single-output (SISO) discrete-time transfer function (TF) models were applied to characterise the relationship between the (step) input, which is a binary input, differentiating between the period right before and after the onset of the emotion, and the output, the mental HR response, associated to it (Figure 3). The model parameters were considered to be time-invariant because the emotion processing of an individual in a single instance was not supposed to change over the course of the experience of that emotion, which is typically very short in duration. Afterwards, the estimated model orders and parameters of the SISO TF were used as features in a support vector machine (SVM) ML classifier, to classify the different emotions. There are several advantages of using DBM models for the current emotion recognition. First, the estimated model parameters can be given a biological meaning and can be interpreted in a meaningful way [39,40]. Second, DBM models can be used for small datasets, while purely ML-based classifiers often need a lot of features and large datasets [39]. Therefore, DBM is a faster method than purely ML-based or deep learning methods for real-time applications in daily life.

2.2.2. SISO TF Modelling

Firstly, SISO TF models were used to identify the relationship between the input u(k − δ), which is a step input around the emotion, consisting in the period right before and after the emotion label, and the output $y\left(k\right)$, the mental HR ([bpm]) response associated with it.

$$y\left(k\right)=\frac{B\left(z^{-1}\right)}{A\left(z^{-1}\right)} u\left(k-\delta \right)+\xi \left(k\right)$$

(2)

u(k − δ) is the assumed deterministic binary step input of the system, with time delay δ (min), which is given the value 1 when the considered data point was labelled as an emotion and a value of 0 otherwise. The period right before and after the step input have the same duration as the emotion itself. This is visually clarified in Figure 4.

$$u\left(k-\delta \right)=\{\begin{array}{c}0\\ 1\end{array} \begin{array}{c} if no emotion labelled\\ if emotion labelled\end{array}$$

(3)

y(k) is the output of the TF system; in this case, the mental HR component. Except for the input to the system, the output is also influenced by external contributions that were not manipulated by the participant, namely disturbing factor ξ(k). This disturbing factor is assumed to be a random variable with zero mean and accounts for the effects of unmodelled inputs, measured noise and modelling errors [38].

$A\left(z^{-1}\right)$ and $B\left(z^{-1}\right)$ are the model polynomials.

$$A\left(z^{-1}\right)=1+a_1{z}^{-1}+a_2{z}^{-2}+\dots +a_{n_a}{z}^{-n_a}$$

(4)

$$B\left(z^{-1}\right)=b_0+b_1{z}^{-1}+b_2{z}^{-2}+\dots +a_{n_b}{z}^{-n_b}$$

(5)

a_j and b_j are the denominator and numerator model parameters, respectively, that have to be estimated, and n_a and n_b are the denominator and numerator polynomial orders. z⁻¹ is called the backward shift operator, meaning that $z^{-1}y\left(k\right)=y\left(k-1\right)$ [39].

The goal of DBM modelling was to determine the optimal TF model structure, defined by the triad [n_a n_b δ], and to estimate the optimal model parameters. To determine the model structure, the ‘rivid’ function of the CAPTAIN toolbox was used, which searched for all possible triads with nominator and denominator polynomials between zero and five, and a time delay between zero and 500 min, with the refined instrumental variable (RIV) approach [36]. The top twenty best models are ranked by increasing Young Information Criterion (YIC). The YIC judges models based on their accuracy, complexity and reliability of the parameter estimations [39].

$$\mathrm{YIC}=lo{g}_e\left(\frac{{\sigma }_e^2}{{\sigma }_y^2}\right)+lo{g}_e\left(\frac{1}{h}{\displaystyle \sum }_{i=1}^h\frac{{\sigma }_e^2{\widehat{p}}_{ii}}{{\widehat{a}}_i^2}\right)$$

(6)

In the equation, ${\sigma }_e^2$ and ${\sigma }_y^2$ are the variances of the model errors and data around the mean, respectively. h is the number of estimated parameters, ${\widehat{p}}_{ii}$ are the diagonal elements of the covariance matrix from the parameter estimations and ${\widehat{a}}_i^2$ the square value of the i-th parameter [38,39]. YIC values have to be as low as possible.

First-order models rise without an overshoot in the mental HR response, while higher-order models do have an overshoot in the response. The results were compared to literature about emotion duration, defined as the interval between the start and end of an emotional episode.

2.2.3. Emotion Classification

To examine the significance of the TF model orders and parameters for emotion recognition, classification accuracies needed to be determined. For this aim, the model orders and parameters of each modelled mental HR response were used as features in subject-specific ML classifiers. Therefore, the data of each participant were randomly divided into a training (70%) dataset and a validation (30%) dataset of the labelled events. Every emotion that was labelled less than three times, was removed from the dataset. Some participants did not have more than one emotion that was labelled at least three times. Therefore, not every participant’s data were used for this part of the analysis.

The performances of three often used ML algorithms in emotion recognition were compared to each other: the k-nearest neighbours (KNN), support vector machine (SVM) and decision trees algorithms. The KNN classifier is based on the assumption that similar samples are located in close proximity to each other. Every sample of the training data set is presented in an n-dimensional space, with n the number of calculated features. Every new sample of the testing data set is classified in the category that is closest by, mostly in terms of Euclidian distance [41]. The SVM algorithm tries to find the optimal distinguishability between two or more classes. Distinguishability is defined as the margin between the different classes. In the n-dimensional feature space, different hyperplanes exist that could separate samples from different classes. The SVM algorithm guarantees that the chosen hyperplane is the one that maximizes the distinguishability between the classes [42]. In the decision tree algorithm, the goal is to classify new data from the testing set based on simple decision rules derived from the calculated features. Each leave of the tree represents a decision rule, after which the data is split into the nodes [41,42].

For every participant, a table was made, in which each row represents an event, and the columns contain the model orders and parameter values. In total, there were four features for each labelled event: number of a-parameters, number of b-parameters, value of the a-parameter(s) and value of the b-parameter(s). The time delay δ was not used as a feature in this classification, because it accounts more for possible misalignments of the data labelling and pre-processing than for a real delay. The first column contained the emotion label for each event. Next, the tables were imported into the Classification learner app in MATLAB and randomly split into a training (70%) and validation (30%) dataset [43]. In this learner, the KNN, SVM and decision trees algorithms with automated hyperparameter optimisation were chosen. For the KNN algorithm, the number of nearest neighbours, the distance metric, distance weight and standardization of the data (yes or no) were optimized. In the SVM algorithm, the kernel function, box constraint level, kernel scale, multiclass method (one VS one or one VS all) and the standardization of the data (yes or no) were optimized. Lastly, for the decision tree algorithm, the maximal number of splits and the split criterion were optimized [44].

The ML algorithms were applied to the data and the classification accuracies of each participant were considered. The classification accuracies were compared to the classification accuracies of a random classifier with the same number of emotions. This percentage was calculated as the inverse of the number of emotions labelled by the participant. Besides that, also the confusion matrices and receiver operating characteristics (ROC) curves were considered to evaluate the classification performance of the best-performing ML algorithm. Classification accuracy is not an optimal performance criterion if there is a big class imbalance, thus, if the samples are far from equally distributed over the different possible classes. The confusion matrix gives additional information on the interpretation of the classification accuracy [45]. The ROC curve for each emotion represents the trade-off between the model sensitivity (true positive rate) and specificity (true negative rate) by plotting the sensitivity against 1-specificity. Ideally, both the sensitivity and the specificity of a model are equal to 1. The area under the ROC curve (AUROC) is, thus, ideally equal to one. The smaller the deviation from 1, the better the predictive value of the model [46]. In the current analysis, the rules of thumb of Hosmer et al. (2013) were followed [47]. An area under the curve (AUC) equal to or smaller than 0.5 implies no discrimination, a classifier that is not better than a random classification. When 0.5 ≤ AUC ≤ 0.7, the classification is hardly better than a random classifier and one can speak about a poor discrimination. From an AUC of 0.7, the discrimination is acceptable and from 0.8 excellent. AUCs equal to or bigger than 0.9 are outstanding [47].

3. Results and Discussion

3.1. Experimental Results

Table 1. Summary of the collected data. Twenty participants (P1–P26) took part in the experiment. For each participant, the gender and age are given in columns two and three, respectively. The number of labels added by each participant is given in the fourth column. The last column shows how many different emotions were labelled by each participant.

Participant	Gender	Age	Number of Labelled Events	Number of Emotions
P1	F	25	17	7
P2	F	22	86	13
P3	F	22	0	0
P4	M	23	17	6
P5	M	51	99	10
P6	F	22	76	9
P7	F	22	21	4
P8	F	22	100	11
P10	F	22	42	8
P11	F	18	583	11
P13	F	26	60	14
P15	F	26	9	5
P16	F	23	13	5
P17	F	22	9	5
P18	M	24	7	6
P22	M	23	23	7
P23	M	21	2	1
P24	F	22	19	4
P25	M	24	75	9
P26	M	22	61	10

summarizes the data that were collected during the experiment. Twenty participants (P1-P26) took part in the experiment, of which thirteen were women and seven were men (Table 1). Since some participants had trouble logging in to their Mindstretch account, a second Mindstretch account was made for those participants. Therefore, the participants’ names in Table 1 go up to P26. The average age of the participants was 24 ± 7 years. From one participant (P3), no data were collected, so only nineteen participants were considered in the analysis. Another participant (P23) only labelled one emotion, so the data of this participant were not suitable for classification. So finally, the results of eighteen participants were analysed.

Table 2. Summary of the number of times that each emotion was labelled by every participant (P1–P26). The emotions in italics are the emotions that were not asked to be labelled but were added on initiative of the participants themselves.

	P1	P2	P4	P5	P6	P7	P8	P10	P11	P13	P15	P16	P17	P18	P22	P23	P24	P25	P26	Total
Anger	4	5		45	3		7	1	47	9		2	1							120
Fear	4	6	1	2	4			2	167	5	1	4			4	2	2	1		201
Guilt	1	6			1		1	2		2										12
Happiness	3	20	5	18	36	9	46	3	228	12	5	2	2	1				26	3	416
Love	1	15			5		3		32	5					1					61
Sadness	2	12			5		8	11	23	3		1	2				4			69
Surprise	1	5	2	12	19		3	18	24	6					8		1			98
Despair		1	5				3		2	3			1							15
Disgust		1							4						2					7
Gratitude		1					1			1							1			4
Longing		3	2				7		20	1										33
Pride		9	1	4	1				21	3	1				1					42
Shame		2			1				14	3		1								21
Calm				2														12	6	20
Frustration				2		2		4			1			1				6	3	29
Stress				7			9							1					3	20
Under time pressure				4		9												5	17	35
Energetic						1					1							10	19	31
Nervous								1												1
Relaxed										6										6
Worried													1						4	5
Excited														2				2	1	5
Exhausted																		7	3	10
Unhappy																		4	1	5
Total	16	86	16	96	75	21	88	42	582	56	9	10	7	5	16	2	8	73	60

summarizes the number of times that each emotion was labelled by every participant.

Figure 5 illustrates the labelled data of participant P13. Every individual plot represents the labelled events of one specific emotion over time and each colour represents a different labelled event of that specific emotion. The dotted lines in the graphs, right before and after the markers, with each time the same duration as the emotion itself, represent the mental HR response just before and after the emotion occurred and give information about the rise and fade out of the emotion. The total length of each graph corresponds with the step input of the models. The y-axis stands for the pre-processed mental HR component in beats per minute (bpm) and the x-axis gives the time in seconds (s).

When the mental HR responses of the same emotion for different events are compared to each other, such as for example the ‘anger’ events of participant P13, it is clear that the mental HR dynamics are highly variable in relation to the emotions at an individual level. This might indicate that contextual information about the circumstances in which an individual is experiencing the emotion is a key aspect, and that the mental HR response to a certain emotion of a specific individual can change over time. Thus, trying to assess the emotional aspect of the mental HR response in real-life conditions seems more challenging than in laboratory-controlled conditions. The context in which emotions take place, for example, whether the person is alone or with friends, or whether the person was expecting it or not, might play a larger role than expected from previous controlled lab conditions. Moreover, some habits of a person, such as drinking, smoking and drug intake might be important factors to take into account when monitoring emotions.

When the mental HR responses of participant P2 are considered (Figure 6), there are some remarkable differences with participant P13. The responses of ‘sadness’ for example, were decreasing curves for participant P13, while they were increasing for participant P2. This is expected as an individual might react differently to the same emotion over time due to the time-varying character of the biological response, but also reacts differently than others to the same emotion due to the individuality of the response [28].

3.2. SISO TF Modelling

After visual look-over of the data as discussed above, the mental HR responses were modelled in function of the step input during the emotion. Over all participants, a first-order response ([1 1 0]) was induced when an emotion arouses. According to the literature about emotion duration, ‘fear’, ‘surprise’, ‘shame’ and ‘disgust’ are the shortest emotions [48]. It was thus expected that these emotions can best be modelled with a first-order model [49]. ‘Sadness’ is considered to be the slowest emotion [49]. It was thus expected that ‘sadness’ could possibly better be described with a higher-order model. However, if the body is considered a complex system and efficient system in using metabolic energy that avoids deviation from homeostasis, it is expected that the mental HR response to an emotion follows a steady increase over time, without overshoot, leading to a first-order model.

3.3. Classification of Emotions

When the optimal model orders and parameters for each labelled event were determined, they were used as features in a different ML classifier. The accuracies achieved per participant and ML algorithm are given in

Table 3. Classification accuracies achieved by the DBM-based classification with ML validation for every participant (P1–P26). The second to fourth columns are the average accuracies for the DBM classification with KNN, SVM and decision tree validation, respectively. The last row includes the accuracies that would have been achieved by a random classifier.

Participant	KNN	SVM	Decision Tree	Random Classifier
P1	60.0%	60.0%	40.0%	50.0%
P2	30.2%	30.2%	30.2%	12.5%
P5	54.3%	57.1%	65.7%	20.0%
P6	50.0%	50.0%	42.9%	16.7%
P7	57.1%	57.1%	42.9%	50.0%
P8	56.7%	56.7%	56.7%	20.0%
P10	66.7%	66.7%	33.3%	50.0%
P11	44.0%	42.8%	37.4%	11.1%
P13	33.3%	38.1%	19.0%	16.7%
P25	54.5%	63.6%	27.3%	33.3%
P26	40.0%	40.0%	26.7%	33.3%
Average	49.7%	51.1%	38.4%	28.5%

. The last column is the classification accuracy that would be achieved by a random classifier. The SVM algorithm was over all participants the best-performing algorithm and performed for all participants better than the random classifier (Table 3).

More information about the classification performance of the SVM classifier running on the model orders and parameters as features was aimed for via confusion matrices. For most participants, the rather high accuracies were caused by a class imbalance of the labels added by the participants. For participant P13 (Figure 7), for example, classifying 76.5% of the events as ‘happiness’, led to an accuracy that was more than double the accuracy of the theoretical random classifier (38.1% versus 16.7%). Few other participants, such as participant P25 (Figure 8), had a confusion matrix in which the elements were more spread over the different emotions.

We concluded that, despite the fact that the accuracies of the SVM algorithm are (sometimes much) higher than those of a random classifier (51.1% on average), the classification performance is often not high. Due to class imbalances of the labels, the classifier often predicts all events of one participant to be the same one or just two particular emotions. Another problem is the inherent subjectivity of the emotion labels. There is no way in which a researcher can assess whether an added label is correct or not. Even if subjects try their very best to label their experiences as accurately as possible, sometimes they just do not know how they feel, or whether this feeling is strong enough to be labelled. Moreover, there is not a large number of labels available. Emotion labelling can be confrontational and time-intensive, so it is hard to gather more labels per individual. Lastly, emotions are for one individual highly variable at different moments in time and in different circumstances. This makes emotion recognition in real-life circumstances a challenging area of research.

Table 4. AUROC evaluation of the emotion classification. For each participant (P1–P26), every labelled emotion is considered as positive class. AUROCs larger than 0.50, marked in bold, indicate that the emotion can better be differentiated from the other emotions than a random classifier would do. The emotions in italics are the emotions that were not asked to be labelled but were added on initiative of the participants themselves.

	P1	P2	P5	P6	P7	P8	P10	P11	P13	P25	P26
Happiness	0.50	0.52	0.61	0.56	0.42	0.37		0.57	0.60	0.67
Anger		0.13	0.62	0.27		0.41		0.51	0.62
Surprise		0.23	0.44	0.32			0.12	0.43
Fear	0.50	0.38		0.38				0.60
Guilt		0.21
Sadness		0.57		0.41		0.48	0.12	0.33	0.32
Love		0.58		0.25				0.50	0.32
Longing						0.38		0.53
Pride		0.20	0.62					0.52	0.32
Shame								0.61	0.19
Stress			0.72			0.69
Under time pressure					0.42						0.44
Calm										0.83
Relaxed									0.32
Energetic										0.50	0.55
Worried											0.31

shows the AUROC values for each participant, for each of the emotions as a positive class. AUROCs larger than 0.50, marked in bold, indicate that the emotion can better be differentiated from the other emotions than a random classifier would do. Over all participants, the following emotions are well discriminable with the current approach: ‘happiness’ (discriminable for 77.8% of the considered participants), ‘stress’ (100.0%) and ‘calm’ (100.0%). The emotions ‘surprise’ (0.0%), ‘guilt’ (0.0%), ‘sadness’ (16.7%), ‘under time pressure’ (0.0%), ‘relaxed’ (0.0%) and ‘worried’ (0.0%) were overall seemingly hard to discriminate.

This research was the first to use mental heart rate as a derived physiological signal to monitor emotions in real-life circumstances. Other physiological signals already have been used for emotion recognition in lab circumstances. In the study of Saarimäki, for example, six basic emotions were classified with an accuracy of 26% and eight non-basic emotions with an accuracy of 15%, based on fMRI measurements [18]. Based on HR measured with PPG in lab conditions, five emotions were recognized with an accuracy of 87.4%. However, these results can poorly be compared with the current result, because ambulatory assessment is more challenging than lab conditions and the number of measured emotions differs. In ambulatory assessment, positive and negative emotions could already be discriminated from each other with an accuracy of 82%, based on voice recordings in a call centre. However, this result can hardly be compared with the 49.7% achieved in the current study, because in the latter a much larger number of emotions were being classified.

4. Current Limitations

Some factors limited the performance of the current algorithms and could be avoided in future research. Firstly, there were too few participants to allow making general statements about all individuals. A priori, a total of twenty participants would have been enough to get statistically significant results. However, there were too few emotions labelled per participant. Often participants forgot to label their emotional experience. Therefore, it is recommended for future research to use an ecological momentary assessment (EMA) system, which works according to the same principles as EMI, but does not have an interventional aspect. These EMAs could remind participants to label their feelings. In this way, more labels can be collected within the same time, which does not increase the burden for the participants. It was also hard for the participants to remind thirteen different emotions, so we recommend decreasing the number of emotions that needs to be labelled. Moreover, shorter questionnaires lead to more reliable answers in EMA or EMI, so the quality of the labels will be enhanced [50].

Besides that, it is important to search for methods that could get rid of the class imbalance in the data set. Methods to reduce the class imbalance already exist, but they mostly rely on under-sampling, which would result in a too low number of training points, or on the generation of simulated data. As emotion recognition based on the mental heart rate component is still in its infancy, not much data are available. The best solution for future research is to collect more balanced data of a lower number of emotions.

Thirdly, the used wearables did not feature thermometers. Therefore, the thermal HR component was estimated by continuous monitoring of the participants. Most big market players are implementing thermometers in their new wearables. The algorithm will improve when working with those wristbands.

Lastly, the sampling frequency of the mental HR needs to be increased. Due to restrictions in the data transfer between Fitbit Inc. and Mindstretch, mental HR could only be calculated every minute. When a higher sampling frequency can be used, the dynamics will be better modelled. Besides, it would probably also be possible to detect certain events in which the emotion was elicited suddenly and subtly, which in the current study might be lost.

5. Conclusions

Mental disorders are a worldwide problem; they are responsible for 15% of the burden of disease worldwide [4]. As the number of incidences increases over time, research in the field of early detection and prevention of mental problems is crucial [2]. Because the core of affective disease often situates in maladaptive emotion processing, it is helpful to develop a method to measure individuals’ emotions in an objective manner, based on the continuous measurement of physiological signals. Emotion scientists already tried to assess individuals’ emotions, via several different behavioural and physiological signals, both so far in ambulatory and lab circumstances. The unique distinctive of this research was the use of the mental heart rate component as a physiological signal. The segregation of the mental component from the total heart rate signal was expected to open a new approach in the field of emotion recognition. The comprehensive goal of this research was to investigate whether it is possible to classify discrete emotions in real life based on the mental heart rate component, estimated by the heart rate and activity variables measured, accurately enough, via a bracelet wearable. Therefore, twenty participants committed to wearing a bracelet 24/7 for three months and to labelling their emotional states.

Each labelled emotion was modelled in a data-based mechanistic matter. The mental heart rate responses of emotions can, in general, be modelled with a first-order model structure. Then, the possibilities of the data-based mechanistic approach for emotion classification were investigated by using the model orders and parameters of the data-based mechanistic models as features in a support vector machine classifier. This machine learning-based validation resulted in an average classification accuracy of 51.1% over all participants and was for all participants higher than the random classifier with an average accuracy of 28.5%. However, when the confusion matrices were considered, it was clear that the classifier often predicted all events of one participant as the same one or two emotions, due to a large class imbalance of the events labelled by the participants. A main reason why the accuracy is not higher might be the inherent subjectivity of emotion labels and the difficulty to collect more labels per individual. Moreover, there are not many labels available. Emotion labelling can be confrontational and time-intensive, so it is hard to gather more labels per individual. Lastly, emotions are for one individual highly variable at different moments in time and in different circumstances. This makes emotion recognition in real-life circumstances a challenging area of research. Over all participants, the following emotions were well discriminable with this approach: ‘happiness’ (discriminable for 77.8% of the considered participants), ‘stress’ (100.0%) and ‘calm’ (100.0%). The emotions ‘surprise’ (0.0%), ‘guilt’ (0.0%), ‘sadness’ (16.7%), ‘under time pressure’ (0.0%), ‘relaxed’ (0.0%) and ‘worried’ (0.0%) were overall hard, or even not possible, to discriminate.

We conclude that it was possible to classify discrete emotions based on the mental heart rate dynamics data, but with yet too limited accuracy. The study shows that the mental heart rate response to emotions is highly variable, not only between different individuals as expected, but also for one individual at different moments in time or in different situations. We suggest that the context information about the circumstances in which an individual is experiencing the emotion, must be taken into account. However, the first steps towards a wearable system for monitoring emotional states, using a classification algorithm based on the link between physiological metrics and emotion in real-life conditions, have been set. Moreover, the insights and identified limitations from this work mark the path for upcoming research in this field.