Application of Speech on Stress Recognition with Neural Network in Nuclear Power Plant

Abstract

Human failures occur in nuclear power plants when operators are under acute stress. Therefore, an automatic stressed recognition system should be developed for nuclear power work. Previous studies on the prediction of stress are limited because of their reliance on subjective ratings and contact physiological measurement. To solve this problem, we developed a non-intrusive way by using voice features to detect stress. We aim to build a system that can estimate the level of stress from speech which may be applied to nuclear power plants where operators engage in regular verbal communication as part of their duties. In this study, we collected voice recordings from 34 participants during a simulated nuclear plant power task in a time-limited situation that requires high cognitive resources. Mel frequency cepstrum coefficients (MFCCs) were extracted from stressed voice samples and the neural network model was used to assess stress levels continuously. The experimental results showed that voice features can provide satisfactory predictions of the stress state. Mean relative errors of prediction are possible within approximately 5%. We discuss the implications of the use of voice as a minimally intrusive means for monitoring the effects of stress on cognitive performance in practical applications.

1. Introduction

Human error has been the main cause of severe nuclear accidents, such as Three Mile Island in 1979 and Chernobyl in 1986, rather than device malfunction and poor operating environment given the rapid development of technology. An investigation conducted by major nuclear power producing countries illustrated that human failures account for 60%–80% of accidents [1]. Acute stress is an important contributory factor to human error. In the field of human reliability analysis (HRV), acute stress was firstly defined as a physical or mental tension state elicited by various stressors [2]. Salas et al. [3] refined the definition and considered stress to be an appraisal process evoked by certain environmental demands, such as noise, time pressure, and task load, which exceed the mental and physical resources of operators and leads to undesirable physiological, psychological, behavioral, and social outcomes.

Tasks in the master control room may become stressful, overwhelm operators, and make them feel extremely confused and anxious when unexpected circumstances, such as rapid signal changes, a sudden increase in workload, complicated decision-making conditions, and limited operating time occur in the nuclear power plant. In Heponiemi’s study [4], he examined the relationship between vital exhaustion, and temperament during experimentally induced stress. He found that participants felt anxious, tense, fearful, sad, depressed, angry, irritated, and disappointed when experiencing stress. Acute stress elicits not only negative emotions in individuals, such as fear, irritation, and anxiety, but also impairs their cognitive functioning. For instance, Luo found a greater amplitude of SPCN (the sustained posterior contralateral negativity) in the stress group than in the control group, which indicates that it is more difficult for individuals to disengage their attention from threat stimuli. These results illustrated that attention disengagement toward threat stimuli is impaired under acute stress situations [5]. Other psychological studies have shown that individuals with high math anxiety stress demonstrated reduced working memory capacity, which led to increased reaction times and errors in their tasks [6]. In a simulated study, the experiment was carried out in a training simulator that matches the actual control room of a nuclear power plant [7]. The result illustrated that information-seeking performance decreases and reaction time decelerates under stressful situations. These stress responses can lead to fatal and serious accidents. Therefore, immediately recognizing the stress state of nuclear power plant operators and then inhibiting the occurrence of human errors is crucial.

Acute stress can be measured and quantified in a variety of ways. Subjective scales mainly used in psychology measure the process of cognitive appraisal and affective responses under stress, including stress appraisal measures (SAM) [8], the perceived stress scale (PSS) [9], and the positive and negative affect scale (PANAS) [10]. However, most scales assess the chronic stress state caused by life events or catastrophes, such as earthquakes and wars. Otherwise, self-reported measures rely on inaccurate recall or memory and fail to assess stress levels continuously without interrupting the task process. Therefore, these approaches are impracticable for monitoring or predicting the stress state of nuclear power plant operators. Acute stress physiologically activates the hypothalamic pituitary adrenal (HPA) axis and the autonomic nervous system (ANS). The chronic activation of HPA results in the increase of adrenocorticotropin hormone (ACTH) and cortisol levels [11]. Meanwhile, the rapid activation of ANS can lead to changes in heart rate, blood pressure, skin temperature, and skin conductivity. Ekman found that activity in the autonomic nervous system, such as heart rate and skin temperature, distinguished among negative emotions [12]. In addition, Villarejo used ZigBee collected galvanic skin response (GSR) data to detect the stress state successfully [13]. Physiological parameters are regulated by the ANS and endocrine system and remain unaffected by subjective factors. In Nandita’s review, physiological parameters combined with machine learning techniques are used as objective indicators to assess the psychological stress state [14]. However, intrusive measuring devices that are used to collect physiological data may restrain operator movements and cause physical discomfort. The procedure for wearing such devices or equipment is complicated and requires significant ability. The extra mental and physical burden on operators caused by these devices can directly reduce the accuracy of stress detection. Therefore, the use of physiological signals in detecting the stress state is an inconvenient method in the nuclear power mission.

Hence, a real-time and nonintrusive stress monitoring system is necessary in predicting stress from voice samples. This type of system can be useful in situations that require operators to communicate regularly by speaking. Studies have identified a number of vocal correlations with stress. According to Hansen and Patil [15], the increase of the respiration rate of an individual will increase the fundamental frequency during the voiced section and lead to a shortened duration of speech between breaths in the event of stressful situations. Dryness of mouth during stress situations can also affect the muscle activity of the larynx and the condition of the vocal cords. Therefore, speculating that speech features can detect and predict the stress state of operators is logical. We extracted the mel frequency cepstrum coefficient (MFCC) feature for model construction in the present study. For MFCC, the main advantage is that it uses mel-frequency scaling, which is very similar to the human auditory system [16]. According to Besbes, MFCC has been shown to be promising for describing stress [17].

Inducing acute stress in nuclear power plant operators during simulated scenarios is difficult. Many studies in field of psychology have induced stress using the Trier Social Stress Task (TSST), where participants give a speech and perform complex arithmetic in front of a stern panel of evaluators [18]. A social-evaluative threat is the key ingredient for situations that elicit stressful feelings in TSST. However, this type of threat is not the main cause of anxiety and exhaustion in nuclear power operators. According to Lejuez et al. [19], mental arithmetic is a cognitive stressor that can effectively elicit cardiovascular responses and negative emotions. For nuclear power plant operators, the stress may be induced by a sudden fault or excessive workloads. In simulated scenarios, the mental stress may be induced more successfully and easily by increasing the workload compared with the creation of simulated faults. Therefore, we integrated mental arithmetic with short-time memory tasks into simulated nuclear power plant scenarios to elicit psychological stress and verified whether the induction was successful through objective physiological parameters and subjective self-reporting.

A new method that uses speech as the stress identification index was proposed in this study in order to overcome the inapplicability of previous stress detection methods in simulated nuclear power plant scenarios. We built a predictive model of stress from speech recordings using a neural network that treats stress as a continuous measure based on the subjective rating of PANAS to provide the specific stress state information of operators. Our goal was to test whether speech can predict objective stress and estimate the possible performance of machine learning approaches and then provide nuclear power plants with a voice recognition method that can measure the stress state of operators in real time without interfering with their work.

This paper is divided into the following sections: firstly, in the method section, the paper describes the process of the psychological experiment. A complete methodology with all the technologies involved in this system is described, and then the BP neural network design is explained. Secondly, the results are obtained during the tasks carried out as part of the experiment. This paper ends with the conclusions and the discussions arising from the topic.

2. Methods

2.1. Participants

Thirty-four student participants were recruited for an experiment on physiological measurement during a nuclear power plant mission through the university’s online psychology subject pool. All participants were right handed, presented a normal mental and physical state, provided written informed consent prior to the experiment, and were compensated for their participation.

2.2. Measures

2.2.1. Positive and Negative Affect Scale

The state version of the PANAS10 consists of 10 items for state negative affect (scared, afraid, upset, distressed, jittery, nervous, ashamed, guilty, irritable, and hostile), and 10 items for state positive affect (enthusiastic, interested, determined, excited, inspired, alert, active, strong, proud, and attentive). Ratings are performed according to a five-point Likert scale ranging from 1 (very slightly or not at all) to 5 (extremely). PANAS was used before and after the stress task to assess the feelings of participants at any given moment. Cronbach’s alphas (a reliability index of the psychological scale) of the state PA and the state NA of the Chinese version of PANAS were 0.85 and 0.83, respectively.

2.2.2. Physiological Measures

Physiological data were recorded with a Biopac Mp150 and were analyzed using Acqknowledge 5.0 software. Heart rate data were collected using an ECG module and then recorded using electrodes attached to ankles and the right wrist of participants. The skin conductance level (SCL) was collected using a galvanic skin response (GSR) module and then recorded using TSD203 transducers placed on the index and middle fingers of the nondominant hand.

2.3. Acute Psychological Stress Task in the Stimulated Nuclear Power Plant Mission

We revised the interface of PCTRAN, which is a desktop simulator of nuclear reactors invented by Micro-Simulation Technology (MST), to simulate the real nuclear power mission. The nuclear power plant mission was composed of short-time memory and mental arithmetic tasks to induce the stress state of participants.

The interface of the memory task consisted of 60 widgets (Figure 1), and each widget was named according to its location on the interface (e.g., Top 1 and Bottom 6). The short-time memory task required participants to remember digits and typewrite them to the correct widgets. The names of widgets and their matching digits were presented on a grid layout for participants to remember (Figure 2). The number of items to remember was randomly selected from 2 to 4. The grid layout disappeared after a specified amount of time and participants were tasked to key in the correct digits for their corresponding widgets (Figure 1). The time for memorizing and typing was limited and the remaining time was displayed on the progress bar to exert time pressure on the participants.

The short-term memory task was repeated three or four times. The mental arithmetic task immediately started when the short-term memory task ended. Participants performed serial tests involving the subtraction of a number between 10 and 20 from a four-digit numeral. The time for answering was limited and the majority of participants failed to complete the task during the time interval. The remaining time was shown on the progress bar (Figure 3). The mental arithmetic task was repeated three times. Both short-term memory and mental arithmetic tasks entailed visual and auditory feedback to warn participants whenever an erroneous answer was provided.

2.4. Procedure

Participants were asked to sit in a quiet room with all the electrodes and transducers attached to their bodies in the beginning of the experiment, and were informed that the entire experiment was audio recorded.

Participants were initially asked to rest for 10 min and complete the Chinese version of PANAS within the resting baseline period to assess their initial emotional state. Participants then read the instructions for the nuclear power plant mission and completed a brief practice to ensure that they understood the task. The acute psychological stress task (6 min) was integrated in the nuclear power plant mission and completed during the stress-induced session. Participants were instructed to respond as rapidly and as accurately as possible and were told that they would be paid according to their performance. Participants were then asked to calculate some mental arithmetic items that did not require accuracy and efficiency and then speak the items and their answers aloud. These audio recordings were used as the database of the speech stress recognition model. Finally, participants completed the PANAS again to assess their emotional state during the mission.

ECG and SCL data were measured continuously during experimental sessions, and physiological data were ensemble averaged for each session of every participant.

2.5. Data Analysis

Subjective affect rating, heart rate, and SCL before and after the stress induction were analyzed using a dependent t-test for within-group comparisons to verify the induction of mental stress. p values ≤ 0.05 (two-tailed) were considered statistically significant.

2.6. Back Propagation Neural Network for Stress State Recognition

The back propagation (BP) neural network model was used to recognize the stress state with the speech signal. The BP neural network is based on the gradient descent method, which minimizes the sum of squared errors between actual and predicted output values [20]. The neural network consists of input, hidden, and output layers. The structure of the BP neural network is presented in Figure 4. When parameters of the network structure are adjusted manually, network connection weight coefficients and bias are automatically adjusted via the back propagation process.

A total of 558 speech recordings were collected (each subject produced approximately 16 recordings). We used Librosa, the Python package for audio and music signal processing, to extract the MFCC from audio recordings as input data. The first 20 MFCCs were extracted from the spectrogram after speech data were sampled at the default rate (sr = 22,050 Hz). Therefore, the input layer contained 20 neurons that received input data. The graph for data visualization is presented in Figure 5. While considering individual differences in stress sensitivity, we used the degree of variability of PANAS scores before and after the stress induction as output values.

We randomly split the data at a ratio of 8:2 (i.e., 446 training and 112 test samples). In other words, training samples and test samples came from different sentences. The BP neural network was established by minimizing the loss, which was calculated from training data. The loss was evaluated on the basis of mean squared error (MSE), which was calculated using Eq. (1). The network structure can be adjusted to achieve the desired loss error. We used Python 3.7 and TensorFlow 2.0 to perform simulations. The basic steps involved in BP network modelling and parameter optimization are illustrated in Figure 6.

$$\mathrm{MSE}=\frac{1}{\mathrm{N}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{N}}{\left({\mathrm{ŷ}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right)}^2$$

(1)

3. Results

3.1. Verification of the Stress State Induction

The results of the paired sample t-test for the subjective affect rating indicated a significant increase of negative emotion ratings after the stimulated nuclear power plant mission (t = −4.832, p ≤ 0.001). Mean stress ratings reached 19.4 points after the stress induction (S.D. = 5.6, range 10.0–33.0) from a baseline mean of 16.2 points (S.D. = 4.64, range 10.0–30.0). Most subjects (24 individuals) reported intensified negative mood after the stress induction. For some of them, the task didn’t elicit their negative emotion, and two of the subjects reported a slightly diminished negative emotion after the task.

Heart rate significantly increased in response to the stress task (t = −7.198, p ≤ 0.001). The mean heart rate was 85.1 (S.D. = 11.9, range 58.6–88.0) bpm during the stress induction from a mean of 76.2 (S.D. = 8.0, range 60.1–104.9) bpm in the baseline phase.

SCL significantly increased in response to the stress induction (t = −2.432, p ≤ 0.05). The mean SCL was 4.9 (S.D. = 2.9, range 2.0–11.4) μs during the stress induction from a mean of 4.0 (S.D. = 2.9, range 1.14–12.1) μs in the baseline phase.

3.2. Speech Stress Recognition

Parameters of the BP network structure must be appropriately adjusted to obtain satisfactory prediction accuracy. The values of parameters used in the BP network structure are listed in

Table 1. The Parameters of The Model.

Number of layers	4
Neurons	Input: 20
	Hidden: 15
	Hidden: 8
	Output: 1
Activation function	tanh Function
Learning rate	0.005
Optimizer	Adam

. The model used in the present study consisted of an input layer, two hidden layers, and an output layer. The tanh function was set as the activation function (Figure 7) that connected input and hidden layers and was also used as the connection between two hidden layers. We did not set up the activation function between hidden and output layers to obtain the linear output.

If the learning rate is excessively high, then the results become unstable. However, if the learning rate is excessively low, then the network convergence becomes slow. The learning rate was set to 0.005 in this study, and the loss converged to approximately 0.03 after calculating 5000 epochs. An Adam optimizer was developed from the stochastic gradient descent (SGD) algorithm and applied to the model. The Adam optimizer uses momentum methods to combine historical descent information to the new descent direction and adaptive learning rate methods to help the learning rate automatically decay during the training process [21]. Therefore, the Adam optimizer can improve the convergence rate. The results of the loss value generated using the BP neural network are shown in Figure 8.

We should input audio features of test samples after achieving the desired loss error of training data to verify whether the model presents overfitting. The relative error for test data was calculated using Equation (2) to assess the ability of the model. Figure 9 depicts the re-error curves of the model. The optimal performance of test samples was stable at around 0.044 in the end, thereby indicating that the accuracy of the model reached approximately 95%. We randomly selected 20 test samples for comparative analysis. Figure 10 presents the comparative results of the true and predicted values. As shown in the figure, the difference between the predicted and true values of the 20 samples is small.

$$\mathrm{Relative} \mathrm{error}=\frac{1}{\mathrm{N}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{N}}\frac{{\mathrm{ŷ}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}}{{\mathrm{y}}_{\mathrm{i}}}$$

(2)

The results of speech stress recognition indicated that speech features (i.e., MFCC) can be used as effective indicators to distinguish various stress states during the stimulated nuclear power plant mission.

4. Discussion

The present study aimed to develop a monitoring system that can conveniently and effectively assess the stress level of nuclear power plant operators in real time to avoid potential human error. Therefore, the speech-based stress recognition approach was adopted in this study instead of traditional subjective scales and physiological parameters. We first investigated whether the economical and easily applicable simulated nuclear power plant task is an effective tool for inducing high subjective stress levels without focusing on social-evaluative components. Heart activity is correlated with stress [22], and skin conductance increases when the individual is under stress [23] due to the increase of moisture on the skin surface that increases the flow of electricity. The significant increase of both physiological indicators of heart rate and skin conductance in the present study indicated that participants experience distinct stress during the mission. The results of these physiological indicators were consistent with those of the subjective questionnaire, in which participants also experienced relatively high negative feelings. These findings indicated that mental stress is successfully induced by the acute psychological stress task during the stimulated nuclear power plant mission. Stress responses were consistent with significant changes in electrodermal activity and heart rate response to the stressful task obtained in previous studies [24].

Negative effects of digital human–machine interface characteristics on human error, such as increased cognitive load, mode confusion, and loss of situation awareness, were identified in field investigations in nuclear power plants and interviews with operators [25]. Moreover, negative characteristics of nuclear power plants, including complex control systems and false alarms, place an excessive amount of burden on operators and increase their stress levels. Detecting the stress level of operators is necessary; however, the use of physiological indicators as main measures for stress in previous studies will restrain operators’ movement and cause other problematic situations. Determining stress levels through speech is non-invasive, less obtrusive, and less expensive compared with other methods for measuring stress. Such stress monitoring systems can be developed for nuclear power plant operators. The Air Force Research Laboratory in the USA claimed that voice-based stress detection systems can efficiently measure the stress of officers to reduce their workload, improve their effectiveness and thus save lives [26]. Therefore, applying intelligent speech interaction systems to nuclear power plants can detect the stress state of operators without disturbing their work.

An analysis of the speech recordings showed that measurable and systematic voice changes occur over the duration of the experiment. The change of voice features and MFCCs according to different stress levels can be used in predicting the stress state. The BP neural network model was developed using recording data. Compared with previous studies that classify emotions under different stressful conditions, we aimed to predict the precise stress level of operators. As a result, the prediction accuracy of the model can reach approximately 95%. Rodellar-Biarge et al. [27] used the SVM classification algorithm, including PCA processing, and achieved a maximum accuracy of 90% in distinguishing stress with speech recognition. Hence, our developed neural network model can evaluate the stress state more accurately and reliably compared with previous machine learning-based classification models.

The proposed model presents the following limitations that constrain its generalization. (1) The task we applied in the experiment was more of a psychological paradigm that focuses on human fundamental cognitive processes compared with real nuclear power missions. Further investigations will require professional participants and a natural operating environment to apply the model in actual nuclear power plants. (2) The corpus size of this study is small given that we only collected audio data from 34 participants and each participant provided 16 paragraphs of speech material. In addition, some literature would supply a validation process as a separate test design, which we can learn in any future research. The characteristics of speakers, such as gender, personality, accent, and language will limit the generalization of the proposed model. More data will be required to check the feasibility of this.

However, this study mainly aimed to explore the performance of the speech recognition stress model. The main findings of this work verified that objective, continuous, and immediate measures of the stress state can be obtained from voice analysis. The prediction performance obtained in this study proved the validity of the model building approach and makes the tractability of the problem explicit.

According to changes in physiological indicators and the performance of the speech recognition model in the present study, combinations of voice features and physiological data can be used to improve the measurement of stress in developing a real-time stress monitoring system for nuclear power plant operators. In addition, operators can wear portable wristbands instead of being attached to electrodes to collect physiological data, thereby minimizing interference from external devices.

5. Conclusions

The main difficulty of stress detection for operators in nuclear power plants is obtaining the stress level in real time without disturbing their work. We induced stress by asking participants to complete a task in a time-limited situation that requires high cognitive resources using a nuclear power plant simulator. Our results showed that physiological indicators and voice features, such as MFCCs, change with the stress level. The contribution of our work can be summarized in two aspects. Firstly, we introduced a non-intrusive way of using voice features to detect stress. At the same time, voice features can be collected in real time. Secondly, we demonstrated the feasibility of speech recognition stress by introducing a machine learning method. The established neural network model can effectively evaluate the stress state and be used as a prototype of a speech stress recognition system in nuclear power plants. Future investigations can develop a model with increased accuracy and high generalization ability using multi-modal data.