Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones
2. Hardware System Overview
- Textrodes: The textrodes are electrodes constructed with a textile structure that have an approximate surface of 60 × 40 mm and are used to measure both the ECG and TEB signal, using only four electrodes. These measurements are strongly related to the mental, emotional and physical state of the subject [10,11]. Figure 1 shows one of the textrodes that is included in the vest.
- Vest: The vest design is illustrated in Figure 2. Both comfort and usability were important characteristics in the design of the vest. To ensure a good grip, all contours of the vest are fitted with silicone bands. Furthermore, different sizes were created, and each vest includes adjustable options, as shown in Figure 3. Using these details, we provided a garment that is comfortable for both men and women.
- Recorder: To record the ECG and TEB measurements acquired by the textrodes, a device known as ECGZ2 was specifically designed to record and send data to the smartphone. The ECGZ2 allows sampling of each measurement with a different sampling frequency. For the ECG measurement, the sampling frequency is 250 Hz, and for the TEB, the sampling frequency is 100 Hz. This device is connected to the textrodes through wires included inside the vest. The device is placed in a pocket of the vest, and the design of this device is shown in Figure 4.In the previous version, the information was stored in Micro SD. However, in the new version, we send the information via Bluetooth to the smartphone, which is responsible for receiving and processing data and obtaining results in real time. Additional technical specifications can be found in .
- Smarthphone: The final component of the designed system is a smartphone, which is responsible for processing the information transmitted in real time by the ECGZ2 unit and estimating those parameters related to the monitoring of the SNS activity. In our design, the selected device was the “Samsung Galaxy Pocket” with an 832-MHz CPU, 512 MB RAM and a 1200-mAh battery. Section 3 describes the software included in the smartphone.
3. Software System Design
3.1. Feature Extraction
3.1.1. Features for ECG Measurement
- Step 1: Filtering. The filtering step, as its name indicates, is a step in which the raw ECG measurement is filtered into different frequency bands. First, the signal is low-pass filtered with a cutoff frequency of 0.5 Hz (Low Frequency (LF) block) and then low-pass filtered with a cutoff frequency of 30 Hz (Medium Frequency (MF) block). In both cases, the filter order is . The first filter is an anti-aliasing filter, which allows the use of interpolated finite impulse response (IFIR) filters . Thus, the output of this anti-aliasing filter is applied to two different IFIR filters with a stretch factor of , both with order .The first IFIR (Continuous Frequency (CF) block) is a low-pass filter with a cutoff frequency of 0.1 Hz, and from this block, we obtain the first filtered signal Ṫhe second block (Respiration Frequency (RF) block) is a band-pass filter with cutoff frequencies of 0.1 Hz and 0.5 Hz, from which the filtered signal is obtained. Finally, the last obtained filtered signal is .Each filter in this step has a different computational cost due to the specific structure and the filter order. This computational cost is measured in the number of simple operations per second and affects the global number of operations in the features of the final system . In this case, we use a filter with an order of for the LF and MF blocks. Thus, if the sampling frequency for the ECG measurement is 250 Hz, then the number of simple operations per second is operations per second. For the IFIR filters, the number of operations per second is denoted as , and it is different due to the filter structure, because this filter is an IFIR filter, as shown in Equation (1):
- Step 2: Processing. In this step, we used different algorithms for each signal obtained in Step 1. The signal is decimated to an intermediate frequency Hz, using operations per second. We selected this value because it is sufficient for extracting all information from the signal and allows standardization of the blocks in the next step.The signal known as is processed to obtain information from the breath. For this purpose, we use the block denominated beats per minute (BPM). This block considers the signal to be sine shaped, and it periodically determines the minimum and the maximum values of the signal, thus allowing estimation of the number of breaths per minute and the amplitude of these breaths. These values are used by the next block, denominated as “inter”, that interpolates the signals by generating two Intermediate Frequency (IF)signals using piecewise constant interpolation to the last known value. The first signal known as represents the breaths per minute interpolated to the , and the second signal known as represents the amplitude for each breath interpolated to the . This process takes operations per second.The last signal provided by Step 2 is known as and is obtained through evaluation of the pulsations per minute (PPM). The PPM block uses a simple algorithm based on thresholding of the five-sample differentiation of the signal , thus estimating the position of the pulses using operations per second.For the computational complexity of the blocks of this step, the number of simple operations required was evaluated for each block of the step.
- Step 3: Parameters. This step is necessary for obtaining the parameters that contain relevant information for monitoring the SNS. For this purpose, we calculated the 13 different parameters, listed in Table 1 with the corresponding required number of instructions per second . It is important to note that in the case of the trimmed mean, the median and percentiles data must be sorted, which consumes nearly all of the operations required for the cases (). Therefore, in the case in which several parameters of this type are evaluated for the same signal, the data must be sorted only once, thus reducing the number of operations.Furthermore, with the aim of characterizing the baseline of each signal, the last parameter denominated as baseline was calculated, totaling 14 parameters per signal. This baseline represents the long-term mean value of the signal and is calculated using a first order low-pass IIR (Infinite Impulse Response) filter. Because we use four signals (, , and ) and 14 possible parameters, the number of available features for ECG measurement is 56.
|Parameter||(Operations Per Second)|
|Trimmed mean of 25%||27,805|
|Mean absolute deviation||1200|
3.1.2. Features for TEB Measurement
- The filters in the first step are different due to the use of a different sampling rates of Hz. All signals from this step are determined using IFIR filters because the estimation of the PPM from the TEM is performed in a different manner.
- Because the shape of the pulse-related component in the TEB is completely different from the typical shape of the ECG signal, the algorithm for determining the PPM uses the BMP block, with parameters adapted to the properties of the signal.
3.3. Feature Selection
- In each iteration of the k-fold cross-validation process, the recordings from the i-th subject are selected as a test set, and the recordings from the remaining subjects are used as a design set.
- The design test is used to determine the best subset of features by applying a genetic algorithm. In this process, the following steps are described:
- A “population” of 200 combinations of features is randomly generated.
- If there are two combinations with exactly the same set of features, one of them is modified by randomly replacing one of the features.
- For each combination in the population, if the associated number of instructions per second is greater than the maximum , then a feature is randomly removed. This step is repeated as many times as necessary. If all of the features of a given combination are removed, then one feature is added at random until the condition is satisfied.
- Each combination is ranked using the mean squared error of a least squares linear classifier measured using a -fold cross-validation process over the corresponding design set.
- The best 20 combinations of the population are selected as “survivals” and are used to generate a new population of 200 new solutions using the random crossover of two parents.
- Mutations are added to the population by changing a feature with a probability of 10%. It is important to highlight that the best individual of each population remains unaltered. The process iterates in Step 2 until 100 generations are evaluated.
- To avoid a local minimum in the optimization method, the feature selection process using the genetic algorithm is repeated five times with random initial populations. The final combination is selected as the best combination of all final populations. This solution renders exceptionally good results with a required number of operations per second that is less than the constraining value .
- Once the best combination is finally selected, the obtained reduced set of features is used to train an MLP with 10 neurons. The design set is randomly subdivided into a training set containing the recordings of 90% of the individuals in the database, and the remaining data are used as a validation set to control the training process using an early stop constraint. To avoid local minima, the full training process is repeated five times, and the best network in terms of error over the design set is selected.
4. Description of the Experiments
- Stage 1: In this stage, the subject must watch a fragment of the nature BBC documentary film “Earth” two times. The fragment lasts from the timestamp 00:00:45 to 00:07:06. The first time, the subject simply watches the movie, and the second time, he/she must continuously note the felt emotion from a set of emotions, with the aim of assessing the elicited emotion. We labeled this stage as the neutral state.
- Stage 2: A game based on addition was specifically developed for the second stage. A sum of two digits is shown on the screen; the subject must do the sum mentally and click the correct solution from a set of possible solutions. The number of digits in the to sum increases from two to five every 25 sums. The difficulty of the task is automatically adapted to the ability of the user, such that if the subject responds correctly, less time is available for the next sum, and if he/she does not complete the sum in the time estimated, the time available for the next sum is increased. Each time that the user fails, a disgusting and stressful sound is played. The duration of this stage varies from five and seven minutes depending on the skill of the subject.
- Stage 3: In this stage, the subject has the option to choose among three different films, validated to elicit sadness:
- “I Am Legend” (2007) by Warner Bross. This movie is a post-apocalyptic science fiction/horror film . The selected movie fragments were the most impactful such that they could induce a loaded emotional charge.
- Stage 4: In this stage, the subject must play the well-known game “Tetris”. This game was used to elicit a workload in several papers, such as . To intensify the mental load, we added stressful sounds and increased the rhythm of the game. The difficulty of the game varies from easiest to hardest and attempts to apply the highest work load to the subject. Thus, when the participant removes rows, the speed is slightly increased, and when he/she loses, the game restarts again with a speed slightly slower than the speed at the end of the previous game. The duration of this stage varies from 5 to 7 min depending on the skill of the subject.
- Stage 5: The last film fragment was selected from “Cannibal Holocaust” (1980) by F.D Cinematografica  and was intended to generate disgust, which represents a strong emotional load . The duration of the fragments is close to three minutes, and again, each participant watches the selected movie twice.
- Stage 6: The aim of this stage is to check the functionality of the system (vest + algorithms) with the moving subject and to elicit physical workload. The experiment was performed at university facilities in a location with three floors. For this test, the subject must climb up and down the stairs quickly for 5 min.
- Analysis 1: In the first analysis, classification of the type of activity was used as the target of the analysis. For this purpose, four different situations were used as classes ():
- Neutral activity, which was registered during the last 140 s of the first movie, i.e., the documentary; thus, considering that each individual watched each movie twice, we obtained a total of 280 s for each individual in the database.
- Emotional activity, which was characterized during viewing of the last 70 s of the second and third movies; therefore, we again obtained a total of 280 s per individual.
- Mental activity, which was registered during the last 140 s of both games, produced 280 s in total.
- Physical activity was registered during the last 280 s of the physical activity stage.
- Analysis 2: In the second analysis, the objective was to distinguish among the emotional states during the viewing of the movies, according to three different emotions (): neutral emotion, sadness and disgust. For this purpose, recordings from the last 200 s of each movie were used, generating 400 s for the neutral emotion (the documentary), 400 s for sadness (the second movie) and 400 s for disgust (the third movie).
- Analysis 3: In the third analysis, the objective was to distinguish between low mental activity at the beginning of the games and high mental load at the end of the games (). The first 110 s of each game were labeled as low mental activity, and the last 110 s of each game were labeled as high mental activity (mainly due to the different difficulty of the games from the beginning to the end).
|IF Signal||Analysis 1||Analysis 2||Analysis 3|
|Type of Activity||Emotional State||Mental Activity|
5.1. Analysis 1: Activity Identification
|True Class||Predicted Class||% Error Probability|
5.2. Analysis 2: Emotional State
|True Class||Predicted Class||% Error Probability|
5.3. Analysis 3: Mental Activity
|True Class||Predicted Class||% Error Probability|
|Low Mental Load||High Mental Load|
|Low mental load||572||308||35.00%|
|High mental load||261||619||29.66%|
Conflicts of Interest
- Mehrnia, A.; Willson, A.N., Jr. On optimal IFIR filter design. In Proceedings of the 2004 IEEE International Symposium on Circuits and Systems, Vancouver, BC, Canada, 23–26 May 2004; pp. 111–133.
- Le Fevre, M.; Matheny, J.; Kolt, G.S. Eustress, distress, and interpretation in occupational stress. J. Manag. Psychol. 2003, 18, 726–744. [Google Scholar] [CrossRef]
- The American Institute of Strees. What is Stress? Available online: http://www.stress.org/what-is-stress/ (accessed on 29 September 2015).
- Sloan, R.P.; Shapiro, P.A.; Bagiella, E.; Boni, S.M.; Paik, M.; Bigger, J.T.; Gorman, J.M. Effect of mental stress throughout the day on cardiac autonomic control. Biol. Psychol. 1994, 37, 89–99. [Google Scholar] [CrossRef]
- Jansen, A.; van Nguyen, X.; Karpitskiy, V.; Mettenleiter, T.C.; Loewy, A.D. Central command neurons of the sympathetic nervous system: Basis of the fight-or-flight response. Science 1995, 270, 644–646. [Google Scholar] [CrossRef] [PubMed]
- Taylor, S.E.; Klein, L.C.; Lewis, B.P.; Gruenewald, T.L.; Gurung, R.; Updegraff, J.A. Biobehavioral responses to stress in females: Tend-and-befriend, not fight-or-flight. Psychol. Rev. 2000, 107, 411–429. [Google Scholar] [CrossRef] [PubMed]
- Cannon, W. The Wisdom of the Body; W.W. Norton: New York, NY, USA, 1932. [Google Scholar]
- Silverthorn, D.U.; Ober, W.C.; Garrison, C.W.; Silverthorn, A.C.; Johnson, B.R. Human Physiology: An Integrated Approach; Pearson/Benjamin Cummings: San Francisco, CA, USA, 2009. [Google Scholar]
- Seoane, F.; Mohino, I.; Ferreira, J.; Alvarez, L.; Buendia, R.; Ayllón, D.; Llerena, C.; Gil-Pita, R. Wearable biomedical measurement systems for assessment of mental stress of combatants in real time. Sensors 2014, 14, 7120–7141. [Google Scholar] [CrossRef] [PubMed]
- Gullette, E.C.; Blumenthal, J.A.; Babyak, M.; Jiang, W.; Waugh, R.A.; Frid, D.J.; Krantz, D.S. Effects of mental stress on myocardial ischemia during daily life. Jama 1997, 277, 1521–1526. [Google Scholar] [CrossRef] [PubMed]
- Agrafioti, F.; Hatzinakos, D.; Anderson, A.K. ECG pattern analysis for emotion detection. IEEE Trans. Affect. Comput. 2012, 3, 102–115. [Google Scholar] [CrossRef]
- Healey, J.; Picard, R. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Trans. Syst. 2005, 6, 156–166. [Google Scholar] [CrossRef]
- Healey, J. Wearable and Automotive Systems for Affect recognition From Physiology. Ph.D. Thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA, 2000. [Google Scholar]
- McCraty, R.; Atkinson, M.; Tiller, W.A.; Rein, G.; and Watkins, A.D. The effects of emotions on short-term power spectrum analysis of heart rate variability. Am. J. Cardiol. 1995, 76, 1089–1093. [Google Scholar] [CrossRef]
- Lee, C.; Yoo, S.K.; Park, Y.; Kim, N.; Jeong, K.; Lee, B. Using neural network to recognize human emotions from heart rate variability and skin resistance. In Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society, Shanghai, China, 17–18 January 2006; pp. 5523–5525.
- Billman, G.E. The LF/HF ratio does not accurately measure cardiac sympatho-vagal balance. Front. Physiol. 2013. [Google Scholar] [CrossRef] [PubMed]
- Piccirillo, G.; Vetta, F.; Fimognari, F.L.; Ronzoni, S.; Lama, J.; Cacciafesta, M.; Marigliano, V. Power spectral analysis of heart rate variability in obese subjects: Evidence of decreased cardiac sympathetic responsiveness. Int. J. Obes. Relat. Metab. Disord. 1996, 9, 825–829. [Google Scholar]
- Mladenić, D.; Brank, J.; Grobelnik, M.; Milic-Frayling, N. Feature selection using linear classifier weights: Interaction with classification models. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, 25–29 July 2004; pp. 234–241.
- Sharma, N.; Pal, U.; Kimura, F.; Pal, S. Recognition of off-line handwritten devnagari characters using quadratic classifier. In Computer Vision, Graphics and Image Processing; Springer Heidelberg: Berlin, Germany, 2006; pp. 805–816. [Google Scholar]
- Hagan, M.T.; Demuth, H.B.; Beale, M.H.; Orlando, D.J. Neural Network Design; Pws Pub Co.: Boston, MA, USA, 1996. [Google Scholar]
- Haupt, R.L.; Haupt, E. Practical Genetic Algorithms; John Wiley and Sons Inc.: Hoboken, NJ, USA, 2004. [Google Scholar]
- Chambers, L.D. Practical Handbook of Genetic Algorithms: Complex Coding Systems; CRC Press: Boca Raton, FL, USA, 1998. [Google Scholar]
- Davis, L. Handbook of Genetic Algorithms; Van Nostrand Reinhold: New York, NY, USA, 1991. [Google Scholar]
- Mohino, I.; Goni, M.; Alvarez, L.; Llerena, C.; Gil-Pita, R. Detection of emotions and stress through speech analysis. In Proceedings of the Signal Processing, Pattern Recognition and Application-2013, Innsbruck, Austria, 12–14 February 2013.
- Walters, D.C.; Sheble, G.B. Genetic algorithm solution of economic dispatch with valve point loading. IEEE Trans. Power Syst. 1993, 8, 1325–1332. [Google Scholar] [CrossRef]
- Gross, J.J.; Levenson, R.W. Emotion elicitation using films. Cogn. Emot. 1995, 9, 87–108. [Google Scholar] [CrossRef]
- Pellouchoud, E.; Smith, M.E.; McEvoy, L.; Gevins, A. Mental effort-related EEG modulation during video-game play: Comparison between Juvenile subjects with Epilepsy and normal control subjects. Epilepsia 1999, 40, 38–43. [Google Scholar] [CrossRef] [PubMed]
- Turner, J.R.; Carroll, D. Heart rate and oxygen consumption during mental arithmetic, a video game, and graded exercise: Further evidence of metabolically-exaggerated cardiac adjustments? Psychophysiology 1985, 22, 261–267. [Google Scholar] [CrossRef] [PubMed]
- Carroll, D.; Turner, J.R.; Rogers, S. Heart rate and oxygen consumption during mental arithmetic, a video game, and graded static exercise. Psychophysiology 1987, 24, 112–118. [Google Scholar] [CrossRef] [PubMed]
- Smith, P. American history X. In America First: Naming the Nation in US Film; Routledge: Abingdon, UK, 2007; pp. 244–258. [Google Scholar]
- Giesbrecht, T.; Geraerts, E.; Merckelbach, H. Dissociative experiences are related to commissions in emotional memory. Psychiatry Res. 2003, 150, 277–285. [Google Scholar] [CrossRef]
- Clasen, M. Vampire Apocalypse: A Biocultural Critique of Richard Matheson’s I Am Legend. Philos. Lit. 2010, 34, 313–328. [Google Scholar] [CrossRef]
- Von Jagow, B. Representing the Holocaust, Kertész’s Fatelessness and Benigni’s La vita è bella; Purdue University Press: West Lafayette, IN, USA, 2002. [Google Scholar]
- Fernández, C.; Pascual, J.C.; Soler, J.; García, E. Validación española de una batería de películas para inducir emociones. Psicothema 2011, 23, 778–785. (In Spanish) [Google Scholar]
- Denot-Ledunois, S.; Vardon, G.; Perruchet, P.; Gallego, J. The effect of attentional load on the breathing pattern in children. Int. J. Psychophysiol. 1998, 29, 13–21. [Google Scholar] [CrossRef]
- Fenton, H.; Grainger, J.; Castoldi, G.L. Cannibal Holocaust: And the Savage Cinema of Ruggero Deodato; Fab Press: Surrey, UK, 1999. [Google Scholar]
- Jackson, N. Cannibal Holocaust, Realist Horror, and Reflexivity; Post Script: Jacksonville, FL, USA, 2002; pp. 32–45. [Google Scholar]
- Palatini, P. Recommendations on how to measure resting heart rate. Medicographia 2009, 31, 414–419. [Google Scholar]
- Choi, J.; Gutierrez-Osuna, R. Using heart rate monitors to detect mental stress. In Proceedings of the 6th IEEE International Workshop on Wearable and Implantable Body Sensor Networks, Berkeley, CA, USA, 3–5 June 2009; pp. 219–223.
- Katsis, C.D.; Ganiatsas, G.; Fotiadis, D.I. An integrated telemedicine platform for the assessment of affective physiological states. Diagn. Pathol. 2006, 1. [Google Scholar] [CrossRef] [PubMed]
- Laurent, F.; Valderrama, M.; Besserve, M.; Guillard, M.; Lachaux, J. P.; Martinerie, J.; Florence, G. Multimodal information improves the rapid detection of mental fatigue. Biomed. Signal Process. Control 2013, 8, 400–408. [Google Scholar] [CrossRef]
© 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Mohino-Herranz, I.; Gil-Pita, R.; Ferreira, J.; Rosa-Zurera, M.; Seoane, F. Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones. Sensors 2015, 15, 25607-25627. https://doi.org/10.3390/s151025607
Mohino-Herranz I, Gil-Pita R, Ferreira J, Rosa-Zurera M, Seoane F. Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones. Sensors. 2015; 15(10):25607-25627. https://doi.org/10.3390/s151025607Chicago/Turabian Style
Mohino-Herranz, Inma, Roberto Gil-Pita, Javier Ferreira, Manuel Rosa-Zurera, and Fernando Seoane. 2015. "Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones" Sensors 15, no. 10: 25607-25627. https://doi.org/10.3390/s151025607