Exploring Seafarers’ Workload Recognition Model with EEG, ECG and Task Scenarios’ Complexity: A Bridge Simulation Study

Abstract

Seafarers are prone to reduce behavioral reliability under high workloads, resulting in human errors and accidents. To explore the changes in seafarers’ workload and physiological activities under complex task conditions, a bridge simulator experiment was conducted to collect the EEG and ECG data of 23 seafarers. The power in different EEG sub-bands was extracted from a one-channel EEG acquisition headset employed by Welch’s method and ratio processing. The features such as root mean square of RR interval difference (RMSSD) were extracted from ECG. Then, an improved seafarer workload recognition method based on EEG combined with ECG and complex task scenarios was proposed, and the performance of the machine learning algorithm was evaluated by cross-validation. Compared with the recognition model that only uses the task scenarios as the workload calibration, the EEG recognition model based on the workload level calibrated by the ECG and the task scenarios is more effective, with an accuracy rate of 92.5%, an increase of 25.9%. The results show that the improved workload recognition model can effectively identify seafarers’ workload, and the model trained with the bagging algorithm has the best performance. Furthermore, time domain features of EEG and ECG fluctuate regularly with the task scenarios’ complexity. The research results can develop online intelligent monitoring, and human–computer interaction active early warning technology and equipment.

1. Introduction

Studies have shown that most maritime accidents are caused by human and organizational factors (HOF) [1,2]. In addition, [3] found through statistical analysis of the ship loss database that most of the accident causes were related to human factors, and technical factors only accounted for 26%. The International Maritime Organization (IMO) has created a regulatory framework for the safety of seafarers and ships by requiring a range of rules and activities to reduce human errors onboard [4]. Human factors on ships are the key issue affecting navigation safety. Therefore, an in-depth discussion of the impact of human factors in the process of seafarers performing complex tasks will help ensure the safety of water transportation and personnel.

The mechanism of ship operation is complicated. It is not operated by one person alone, but by the entire crew working collaboratively to operate multiple modules of control conditions, usually including bridge control, communication, steering gear control, and other posts. Therefore, the captain’s judgment and decision-making will directly affect the performance of the entire ship driving team. For each seafarer, they often need to have information about the environment and the state of the ship in a short time to respond to complex task scenarios in a timely manner.

In addition to the driving operations, the ship’s working mechanism is also very particular. Ships must keep running day and night for navigation and production. Seafarers are often on duty during periods of time when rest is required, which can easily lead to heavy workloads. The high workload will affect the physiological and psychological conditions of the seafarer, causing fluctuations in their cognition, judgment, and responsiveness, which can lead to human error. If the high workload is not detected and dealt with in time, it will become a safety hazard to navigation. Therefore, studying the seafarers’ workload related to complex task scenarios will contribute to the basic research and development of the intelligent monitoring and early warning system for the seafarers’ work status.

Human intentions and implicit psychological traits can be expressed through signals generated spontaneously by the human body, such as electroencephalography (EEG), electrocardiograms (ECG), electromyography (EMG), and blood volume pressure (BVP), and cannot be deliberately concealed by these means. Currently, one of the more interesting areas of emerging research is affective computing [5,6]. Intelligent emotion classifiers detect emotional cues from a person’s responses and integrate emotional responses, ultimately predicting instantaneous emotional states [7]. There have been studies on the use of machine learning to identify emotional states with EEG signals, mostly through algorithmic improvements to develop the accuracy of emotion recognition [8,9,10,11]. Since EEG comes directly from the brain, it is more objective and reliable in capturing the true emotional state of a human being, as well as stress and mental load. Similarly, the detection and integration of workload cues from the EEG and ECG signals of seafarers to discriminate workload states are one of the focuses of this paper.

Considering that the existing studies on seafarers’ workload are mainly based on subjective evaluation and lack of objective evaluation, the main research purposes of this paper are as follows: (1) explore seafarers’ workload variation under different complex tasks using EEG and ECG; (2) propose an EEG-based method for seafarer workload recognition, combining with ECG and task scenarios, to differentiate seafarers’ workload. The workload classification algorithm proposed in this study will solve the problem that current research mostly relies on subjective evaluation and lacks objective data to improve the accuracy of workload identification [12,13,14]. Meanwhile, the portable dry electrode EEG headset used in the experiment has the advantages of high sensitivity and high resolution, and the wireless equipment does not affect the activity, which is suitable for the working environment of seafarers. It helps to achieve real-time monitoring of seafarers’ workload. It is a powerful tool for the intelligent monitoring and early warning system of seafarers’ work status.

The rest of the paper is organized as follows: Section 2 introduces the literature review of the relevant studies. Section 3 introduces the experimental setup and experimental procedure; Section 4 presents the principles and methods of the workload recognition system for seafarers based on ECG and EEG signals; Section 5 gives the workload classification results; Section 6 presents the discussion.

2. Literature Review

2.1. Definition and Measurement of Seafarers’ Workload

The workload is the stress or the processing capacity of the brain that a person is under working [15]. When the workload becomes excessive, performance can be affected and errors begin to occur when time stress is excessive, or when human short-term memory capacity exceeds the limits [16]. The work of seafarers mainly ensures safe navigation by monitoring the ship’s system and making decisions, which requires high mental effort. Therefore, the workload of seafarers studied in this paper mainly focuses on their mental workload.

In the existing research, seafarers’ workload detection methods are mainly divided into three types: subjective assessment, task performance measurement, and physiological data detection. The subjective assessment reflects the workload of the subjects through the quantitative evaluation of their workload in the process of performing the task. Current mainstream subjective workload assessment methods include the Cooper–Harper scale subjective assessment, the NASA-TLX (National Aeronautics and Space Administration-Task Load Index) scale, and the SWAT (Subjective Workload Assessment Technique) subjective workload assessment technique. The task performance measurement refers to the method of using the subjects’ operational performance as the workload evaluation criterion [17], which mainly includes primary task performance measurement and secondary task performance measurement. In ref. [18], the authors used bridge simulation experiments to assess the extent to which fairway navigation benefits from information aids by measuring seafarers’ workload, performance, and affective experience of the work situation. The workload measurement in former research uses NASA TLX, and its derivatives are also widely used in road traffic [19,20].

Seafarers work in difficult environments and complex task scenarios, and their workload calibration is inconvenient due to the narrow working space and the team working mechanism. Physiological data are more quantifiable and intuitive than subjective and performance measures for seafarers’ workload. Commonly used physiological indicators for workload recognition include ECG, EEG, eye movement, etc. Physiological measures of the above pathways have proven research validity in road driving [21]. However, they are not widely used in seafarers’ workload research due to the complex working environment and teamwork mechanism of seafarers. EEG has the advantage of high sensitivity, and portable EEG equipment is suitable for the working environment of seafarers. It helps to achieve real-time monitoring of seafarers’ workload. In addition, complex task scenarios and ECG are used as objective calibration to identify seafarers’ workload and can thus compensate for the inadequacy of subjective methods.

2.2. Application of ECG in Seafarers’ Workload Recognition

The electrocardiogram (ECG) primarily reflects the activity of the heart. Changes in the physical or mental load of the body can lead to changes in heart rate data, making heart rate (HR) a sensitive data indicator that effectively assesses a driver’s workload. HRV is the interval variation between consecutive normal RR (or NN) intervals derived from ECG readings [22], and is measured by calculating the time interval between two consecutive peaks in the heartbeat [23]. HRV can be divided into time-domain and frequency domain metrics. The variability of heart rate data is significantly correlated with workload [24]. Some of the HRV-based features include time domain indicators such as standard deviation of RR intervals (SDNN) and root mean square of RR interval difference (RMSSD), and frequency domain indicators such as low frequency (LF) and high frequency (HF), which can reflect workload variability to some extent.

There have been some exploratory studies on ECG in seafarers’ workload analysis and identification. RR intervals in HRV can assess the mental load of seafarers [25]. RR interval data and LF/HF values were used by [26] to assess the mental workload of operators, and LF/HF discriminated significantly for the workload. The authors of [27] examined seafarer workload levels under different navigation systems using LF/HF and RMSSD based on a ship driving simulator. They found that the above two metrics were effective in differentiating workload levels. The authors of [25] found that based on a ship driving simulator the LF/HF was an effective way to discriminate workload, and the larger the LF/HF value, the higher the workload. Hence, the relevant indicators of HRV can effectively evaluate the change in seafarers’ workload.

2.3. Application of EEG in Seafarers’ Workload Recognition

Electroencephalogram (EEG) signals visually reflect the perception activity of the human brain due to their sensitive nature. Four frequency components that detect the driver’s current state can be obtained from particular EEG frequency intervals as δ (0.5–3 Hz), θ (4–7 Hz), α (8–13 Hz), and β (14–30 Hz), where α activity reflects a relaxed state of wakefulness and decreases with concentration, stimulation or gaze [28]. Increased β activity is associated with levels of alertness and decreases during sleepiness [29]. EEG is also commonly used to study driving fatigue, in road traffic [30].

The research results of EEG on seafarers’ workload are embodied in the following situations. The averaged α band power of EEG epochs was significantly correlated with seafarer workload when seafarers performed Marine Engine Plant Simulator (MEPS) tasks of varying difficulty [31]. In ref. [32] the authors conducted workload experiments in a MEPS and used Pearson correlation coefficients to test the validity of the relationship between seafarer workload and EEG characteristic quantities. They found that EEG sub-band α was sensitive to workload and that EEG sub-band α, β, and θ were sensitive to the amount of subjective workload. In ref. [9] the authors collected seafarers’ EEG data and used SVM to identify the emotions of captains and could achieve an average recognition accuracy of 77.55%. In ref. [33] the authors used a deep learning algorithm to process EEG to recognize the workload levels of seafarers in different positions and found that captains had greater workload than seafarers in other positions. In ref. [34] the authors conducted a one-way ANOVA on EEG sub-band β1 and β2 power spectra and subjective workload and found significant differences in β1 and β2 power spectra for different difficulty-level tasks. In ref. [35] the authors used a physiological assessment system with EEG data as the main parameter to determine workload levels for a reasonable assessment of seafarer training. Therefore, it is feasible for this study to use EEG to objectively assess crew workload levels.

3. Experimental Setup

3.1. Subjects

The subjects in the experiment were seafarers recruited from different companies to take the captain qualification training, with the average age of 37.63 years old (S.D. = 5.43). There were 23 subjects, all of them were male senior seafarers, with normal eyesight and good health. They had over 9 years of extensive sailing experience at sea on average. They were in good mental condition and were all able to complete ship maneuvering instructions. In this experiment, each subject served as the captain and controlled the bridge by giving decision-making instructions. The captain is responsible for navigation status and formulating disposal strategies. The tasks of bridge control, communications, steering gear control, etc., are up to the captain. Since the navigation is operated by the entire crew, the captain’s physiological data reflect the workload of the crew to a certain extent. Therefore, this experiment focuses on the physiological data of the seafarers who are responsible for the captain’s tasks in the bridge simulation. The subjects signed the informed consent form before the start of the experiment and understood the content of the test syllabus. They knew the skill level and ability target they needed to achieve in the experiment and had a comprehension of the workload of different driving tasks.

3.2. Experimental Equipment

The experiment was carried out in the Navi-Trainer Professional 5000, a bridge simulator at Wuhan University of Technology (Figure 1). The advantage of the bridge simulation is that the performance scenes are rich, and it can test the emergency performance of the seafarers in various situations. The simulator is an integrated bridge system on the bridge consisting of radar, electronic charts, navigational aids, and communication systems, which simulates the real state of a ship in various situations. In the experimental scene, there are different driving tasks, including ship berthing, unberthing, safe navigation in narrow waterways, navigation in poor visibility conditions, ship encounter operation, anchor lifting, and anchoring operation, grounding emergency, and 7 kinds of emergency disposal tasks (including main engine failure, steering gear failure, ship fire, personnel overboard, seafarers’ sudden illness, ship collision, and ship oil spill). The simulator provides this experiment with the task and scenario conditions needed to study different workloads.

The heart rate apparatus was the polar V800 watch from Polar Electro, Finland, and an H10 chest strap with a data transmitter, as shown in Figure 2a. The heart rate signal is transmitted at a frequency of 5000 Hz, the chest strap with the sensor measures the RR heart rate spacing, and the transmitter sends the RR heart rate spacing data to the watch. That can automatically change the RR heart rate spacing data to heartbeat and display the changes in heart rate in real time during the experiment.

In actual operation, the seafarer requires high mobility, so the EEG signal acquisition device chosen is NeuroSky Mindwave as shown in Figure 2b, a wireless single-channel EEG acquisition headset with a dry medium and a sampling frequency of 512 Hz.

3.3. Experiment Process

The 23 subjects were informed in advance of the navigation area. They operated in the bridge simulator room, while staff provided scenarios to subjects in a separate control room. The test subjects and staff in the experiment are shown in Figure 3. All scenarios selected were from the simulator database. Before the start of the experiment, the subjects signed the informed consent and basic health information form (see Appendix A) to ensure that they are healthy for experiment. Meanwhile, the chest straps were tightened on the subjects and adjusted to a comfortable area, and the EEG acquisition headphones were worn at the same time. The subjects performed adaptive driving for about 10 min. After the start of the experiment, the EEG and ECG data of each subject were extracted simultaneously, the first five minutes after the start of the experiment served as baseline data. The bridge simulator randomly generates regular and unexpected tasks, which the subjects must complete while keeping the ship safely and smoothly underway, with variable intervals between tasks.

The structure of the experimental scene is similar, and the process is roughly divided into: start sailing, normal sailing, poor visibility, seven random emergencies, and termination of sailing (including operations such as departure, arrival, and anchoring) (see Figure 4). Each part of the scenario process imposes different ability requirements on the subjects. For example, when the ship is sailing normally, the captain needs to continuously observe the weather conditions, hydrological conditions, and traffic conditions, does not need to operate the ship, but issues driving instructions to other personnel. In the ship collision handling task, the captain needs to observe the ship’s navigation trajectory before the collision, judge the possibility of collision, and issue operational instructions accordingly. After the collision, the captain needs to assess the damage and conduct follow-up commands as appropriate.

The duration of each experiment is slightly different according to the actual operation process of the subjects, and the average duration is about 45 min. The EEG and ECG acquisition time during the experiment was the same as the test for each subject.

4. Method

4.1. Data Collection

4.1.1. ECG Collection and Preprocessing

The ECG data of each subject was divided by 60 s as an epoch. Data were processed using Kubios HRV (Heart Rate Variability) Standard (version 3.4) software developed by Kubios Oy. It has pre-processing capabilities for threshold-based heartbeat beat correction and for optimal trend removal from RR data for short-term HRV analysis. Furthermore, there were significant differences in mean heart rate (mHR) and mean RR (mRR) between normal and stressful states during cerebral activity [36]. HF power and RMSSD were significantly lower in the stressed state compared to the relaxed state [37]. During cerebral activity, the mean RR interval was significantly lower than that of the resting state, while there was a tendency for LF/HF to increase under stress conditions [38].

In this study, the time domain indicators of HRV, MHR, MRR, and RMSSD, and the frequency domain indicators, HF and LF/HF were selected as HRV characteristics for further analysis, and the meanings of the indicators are shown in

Table 1. ECG indicators selected for the study and their description.

Indicator Type	Acronym	Description
Time Domain	MHR	Mean of heart rates
	MRR	Mean of RR interval
	RMSSD	Root mean square of successive NN interval differences
Frequency Domain	HF	Power spectrum in the frequency range of 0.15–0.4 Hz
	LF/HF	Ratio of low to high frequency

. The partial RR interval of subject 8 is shown in Figure 5, indicating the change within 450 s.

4.1.2. EEG Collection and Preprocessing

NeuroSky Mindwave acquired EEG signals from 23 subjects at a sampling frequency of 512 Hz in left forehead single channel FP1 over the course of a simulated experiment of about 45 min. The recording electrode was organized with presentation software NeuroView (NeuroSky, San Francisco, CA, USA). In the process of collecting EEG, the activities of the subjects and the power supply of the device will generate interference signals, and the signal of EEG is weak. So, the data were processed by a Matlab-based toolbox with a graphical user interface, EEGLAB (version 2021), invented by [39]. To eliminate linear trends, the raw EEG signals were processed in EEGLAB using band-pass filters with cut-off frequencies of 1 Hz and 30 Hz. Considering the network noise, the center frequency of the notch filter is set to 60 Hz [40].

System function of digital filter H(z):

$$H\left(z\right)=\frac{Y\left(z\right)}{X\left(z\right)}=\frac{{{\displaystyle \sum }}_{i=0}^M{a}_i{z}^{-i}}{1-{{\displaystyle \sum }}_{i=1}^N{b}_i{z}^{-i}}$$

(1)

Direct measures of the brain’s cognitive system can more accurately measure workload levels [15]. As δ reflects individual sleep and will not show high activity during ship driving activities, this study focuses on particular EEG frequency intervals named as θ, α, and β. Ardian et al. found greater accuracy using β-band measurement in a study of workload level estimation based on EEG channel connectivity selection. In complex and cognitively demanding tasks, the EEG sub-band α decreases with increasing workload and the EEG sub-band β connectivity values continue to decrease [15]. The specific frequency allocation and corresponding functional activities are shown in

Table 2. EEG frequency allocation and functional activities.

Category	Frequency Range	Corresponds to the State of Brain Activity
δ	0.5–3 Hz	Deep sleep
θ	4–7 Hz	Light sleep or extreme relaxation
α	8–13 Hz	Conscious and relaxed
β	14–30 Hz	Fully awake and highly focused

.

The higher the difficulty and complexity of the task in the driving simulation experiment, the more focused the subject needs to be and the higher their workload. Jap et al. chose four algorithms for testing EEG-based power ratio activity: α/β, θ/β, (θ + α)/β, and (θ + α)/(θ + β) and found a significant decrease in EEG sub-band β activity as the level of alertness decreased, which reflected the driver’s attention changes in EEG sub-band β under inattentive conditions [41]. Therefore, the above four ratios will be used as observations to examine the attention of seafarers in this experiment of seafarers performing complex navigational tasks to further reflect the workload.

Cognitive and emotional variations indicated by power spectral density estimations of EEG sub-bands are characterized by significant variability. In this study, the band power spectral density (PSD) is applied as the feature to directly observe the frequency information. Commonly used feature extraction methods include the Periodogram method, Bartlett method, and Welch method [42,43], all of which are based on Fourier transform to perform spectral estimation on sample data. A comparison of the three commonly used methods is shown in

Table 3. Comparison of power spectrum estimation methods.

Method	Advantage	Disadvantage
Periodogram	Simple and fast	Sacrificing variance performance
Bartlett	Segmentation to reduce variance	Sacrificing bias and resolution
Welch	Segmented data improves variance	The number of segments and the window function should be selected reasonably

.

In the feature extraction stage of this study, Welch’s method is employed to estimate the band power spectral density and a 50% overlapping Hamming window is used to improve spectral leakage. The power spectrum estimation result ${\tilde{P}}_H\left(\omega \right)$ of Welch’s method can be expressed as:

$${\tilde{P}}_H\left(\omega \right)=\frac{1}{L}{\displaystyle \sum }_{i=1}^L{\widehat{P}}_i\left(\mathsf{\omega }\right)=\frac{1}{NZL}{\displaystyle \sum }_{i=1}^L{\left|{\displaystyle \sum }_{n=1}^N{x}_i\left(n\right)d\left(n\right)e^{-jhn}\right|}^2$$

(2)

In Equation (2), ${\widehat{P}}_i\left(\mathsf{\omega }\right)$ is the power spectrum estimation of section $i$, $N$ is the number of data points in each segment,$L$ is the number of segments; $d\left(n\right)$ is the window function, $Z=\frac{1}{N}{{\displaystyle \sum }}_{n-1}^N{\left|d\left(n\right)\right|}^2$ is the normalization factor.

The log-transformed power (LTP) was then calculated for the three particular EEG frequency interval of interest, θ (4–7 Hz), α (8–13 Hz) and β (14–30 Hz). Where EEG sub-band θ is considered to be slow wave activity, while α and β are fast wave activities [44]. In addition, the four EEG based LTP ratios α/β, θ/β, (θ + α)/β, (θ + α)/(θ + β) were also considered as EEG features. A total of seven feature variables were finally obtained for LTP of particular EEG frequency interval θ, α, β and the EEG based LTP ratios of α/β, θ/β, (θ + α)/β, (θ + α)/(θ + β). As the driving simulator was randomly generated for different experimental durations, the feature analysis unit for each subject was N. The experimentally collected data yielded a feature matrix of 7 × 1 × N for each subject. Part of the EEG of subject is shown in Figure 6, and related EEG feature information is illustrated in Figure 7.

4.2. Workload Analysis Based on EEG and ECG in Task Scenario

Complex tasks are the main source of workload, and tasks of different difficulty bring different workloads. Therefore, the workload analysis based on EEG and ECG in the task scenario can effectively identify the changes in the workload of the subjects.

First of all, the whole experiment process was divided into the early stage, the middle stage and the end of the navigation, and the task scenarios and the corresponding occurrence times were marked. The analysis of ECG features contains three key time stages: pre-task stage, task phase, post-task stage. Second, HRV overall change data were selected and described mood (nervous or calm), they were compared with EEG changes.

Next, this part of the workload was calibrated by heart rate combined with task scenarios. The task scenarios were labeled according to the time demand, effort, and mental demand of the ship pilot to complete the task as a reference. The mean values of MHR and MRR for the first 5 min of every test were used as a baseline for the subject’s ECG in a calm state. Following the law description in Section 4.1, the MHR and MRR changed significantly when a person was under a higher workload, with the MRR generally on a decreasing trend. In addition, an increase in workload also causes a decrease in RMSSD and HF values and an increase in LF/HF values. Therefore, the workload is scaled into two levels, namely high and low. The workload calibration was illustrated by the HRV data of subjects 8 and 9, as detailed in

Table 4. Task scenarios combined with heart rate calibrated workload levels (e.g., subject 8 and 9).

Task Type	Task Scenarios	MHR	MRR	RMSSD	HF	LF/HF	Workload
Regular tasks	Normal navigation	SC	SC	↓	↓	↑	L-L-L
	Overtaking	SC	SC	↓	↑	↓	L-L-H
	Ships’ encounter	SC	SC	↓	↑	↓	L-L-H
	Sparing anchor	SC	SC	↓	↓	↑	L-L-L
	Dropping anchor	SC	SC	↓	↓	↑	L-L-L
	Re-anchoring	SC	SC	↓	↓	↑	H-L-L
	Exchanging information	SC	SC	↓	↑	↑	H-L-L
Sudden events	Adverse weather	SC	SC	↓	↑	↓	L-H-H
	Machine malfunction	SC	SC	↓	↓	↓	H-H-L
	Met boat broke down	SC	SC	↓	↑	↓	L-H-L
	Ship collision	SC	SC	↓	↑	↓	H-H-L
	Water leakage	SC	SC	↑	↑	↓	H-L-H
	Fuel spillage	SC	SC	↓	↓	↑	H-H-L
	Personnel overboard	SC	SC	↑	↑	↓	H-H-L
	Personnel injuries	SC	SC	↓	↑	↓	L-L-H
	Hydraulic pipe bursts	SC	SC	↓	↑	↓	L-H-H
	Fire alarm	SC	SC	↓	↑	↓	L-H-H

(Note: SC = significant change, L = low, H = high, ↑ = rise, ↓ = decline).

. The rightmost column of Table 4 is the change in workload level in the three stages of each task. Due to the randomness of tasks in the process of navigation, there were continuous task operations or multi-task parallelism, so the specific workload classification should not ignore the personal data characteristics of each subject.

4.3. Workload Recognition Based on EEG Data and Task Scenarios

In the process of establishing the workload identification method, we first tried an identification model that only uses EEG to calibrate the task scenarios. The learning method was adopted to classify the EEG data of 23 subjects into high and low workload. The workload was determined by the task’s complexity in different time periods, and the EEG data for these time periods are output through machine learning as high and low workload levels. Figure 8 presents the procedure.

4.4. Improved Workload Recognition Based on ECG, EEG Data, and Task Scenarios

To further improve the recognition accuracy of seafarers’ workload, an improved learning workload classification method based on EEG combined with ECG and complex task scenarios was proposed, as shown in Figure 9. It first classified the workload according to task scenarios combined with the ECG features. Secondly, it dealt with workload recognition based on EEG data. In the second step, the results of the former step combined with EEG data were applied as inputs to train the classifiers. Finally, the binary classification of workload level was carried out by training five machine learning algorithms of Decision Tree, SVM, KNN, NB, and Bagging. Additionally, five-fold cross-validation was conducted.

4.4.1. Step 1Workload Classification Based on ECG Data and Task Scenarios

The method of the first step was to classify the workload by combining the navigation task scenarios with the ECG features. The detailed classification method had been described in Section 4.2. Workloads were divided into high and low levels in step 1.

4.4.2. Step 2 Workload Classification Based on EEG Data

The method of the second step was to use the workload labels from the first step of classification results and the subjects’ EEG feature dataset as input data for the workload recognition system. The EEG feature matrix of 169 × 7 and the label matrix of 169 × 1 were obtained based on the workload labels from the first step. Due to the imbalance of various types of data in binary classification, a fewer number of labeled samples were oversampled to generate new data points to achieve data balance, a 202 × 7 feature matrix and a 202 × 1 label matrix were obtained for classification. The final dataset was classified into the high and low levels of workload through five machine learning algorithms: Decision Tree, SVM, KNN, NB, and Bagging. A five-fold cross-validation was conducted to evaluate the performance of each model.

5. Results

5.1. Workload Classification Based on EEG Data and Task Scenarios

This model compared five machine learning algorithms: Decision Tree, KNN, SVM, Naive Bayesian, and Bagging. The training and test datasets were divided into a 7/3 ratio, and a five-fold cross-validation test was conducted. The performance of each classifier was also evaluated in terms of the area under the subjects’ feature operator curve, and classification accuracy, precision, recall, and F1 score derived from true positive (TP), false positive (FP), true negative (TN), false negative (FN) of the confusion matrix. The performance evaluation metrics for binary classification were calculated and expressed as follows:

$$Accuracy=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FT}+\mathrm{FN}}$$

(3)

$$Precision=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(4)

$$Recall=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(5)

$$F1=\frac{2}{\frac{1}{Precision}+\frac{1}{Recall}}$$

(6)

The specific values are shown in

Table 5. Model performance comparison of algorithms.

Algorithm	AUC	Accuracy	Precision	Recall	F1-Score
SVM	78.0%	67.5%	89.0%	62.2%	73.2%
NB	74.0%	72.5%	78.0%	70.3%	74.0%
Bagging	74.0%	68.5%	84.0%	64.1%	72.7%
DT	69.0%	73.5%	79.0%	71.2%	74.9%
KNN	64.0%	64.0%	77.0%	61.1%	68.1%

(Note: SVM = Support Vector Machine, NB = Naïve Bayesian, DT = Decision Tree, KNN = K-Nearest Neighbor).

.

AUC is the area enclosed by the axis under the ROC curve, and the classifier with the larger value has the higher correct rate (see Figure 10). The accuracy rate indicates the proportion of all samples with positive and negative class predictions that are correct as a percentage of the total sample. Accuracy is specific to the predicted outcome and indicates how many of the samples predicted to be positive are actually positive samples. Recall is specific to the original sample and indicates how many of the positive cases in the sample were correctly predicted. F1 score combines the results of the precision and recall outputs.

The workload classification method achieved the best accuracy rate of only 73.5% by calculation. While the workloads were differentiated to some extent, the classification results were not satisfactory for binary classification. So, the algorithm model needed to be improved.

5.2. Improved Workload Classification Based on ECG, EEG Data, and Task Scenarios

In order to solve the problems existing in the recognition model, the algorithm was improved. This improved model also compared five machine learning algorithms: Decision Tree, KNN, SVM, Naive Bayesian, and Bagging. The specific values are shown in

Table 6. Improved model performance comparison of algorithms.

Algorithm	AUC	Accuracy	Precision	Recall	F1-Score
Bagging	96.0%	92.5%	88.0%	96.7%	92.1%
DT	93.0%	89.5%	85.0%	93.4%	89.0%
KNN	88.0%	88.5%	84.0%	92.3%	88.0%
NB	91.0%	87.5%	87.0%	87.9%	87.5%
SVM	91.0%	85.5%	78.0%	91.8%	84.3%

(Note: DT = Decision Tree; NB = Naive Bayesian).

.

Related AUC statistics are shown as Figure 11.

In terms of classification accuracy and AUC values, the Bagging classifier showed good performance, achieving an accuracy of 92.5% and an AUC of 96.0%. As most ship accidents were caused by human error [45], it is important to ensure both classification accuracy and that the workload status of each driver is not missed when discriminating seafarers’ workload. In summary, the classifier needs to have a high recall rate as well as a high AUC. The classification performance of the Bagging algorithms is superior in all metrics, with a recall of 96.7% and an AUC of 96.0%.

6. Discussion

6.1. Seafarers’ Workload Variation in Task Scenarios

Both EEG and ECG are physiological indicators with strong individual differences. From the analysis of the individual physiological characteristics of the seafarers, when the tasks changed, the changes in the captain’s EEG correlate with EGC, showing significant changes at both key peaks of the complete experimental data. In the identification of multi-person workloads, the personalization of the ECG can offset the large variability of the EEG to a certain extent. Therefore, when seafarers’ EEG signals are recognized, the ECG features can be considered in the classification system to obtain good results. In addition, subjects’ physiological signal fluctuations were affected by complex navigation tasks. ECG features showed significant changes with increasing task difficulty, manifested as decreased RMSSD, HF, and increased LF/HF. The prominence of the above features in workload measurements all validate earlier studies [25,26,27].

In the EEG analysis, the overall trend in the variation of EEG-based power ratio (θ + α)/β features for high and low workloads in the final dataset was generally consistent. However, specific values were still clearly differentiated. During the experiment when complex tasks with temporal stress dimensions such as fire alarms occurred, the subject requires high concentration and high alertness, with a greater increase in α and β activity. Additionally, θ activity was more pronounced when in a calmer state. They were the same as the findings of [29,46]. The extracted EEG features showed significant changes during the experiment. The results demonstrate that EEG signals can effectively cope with seafarers’ workload and stress and support the use of EEG signals to monitor seafarers’ brain status. The findings are consistent with Liu et al. suggesting [33].

6.2. Comparison of the Two Recognition Models

Classified EEG data from 23 subjects with 75% accuracy. This is not a satisfactory result for algorithmic results of binary classification. The main reason is that sensitive EEG devices bring about strong individual differences while extracting subtle physiological fluctuations in subjects. Using priori task scenarios alone as workload calibration in algorithm structure is not restrictive enough for EEG classification. In general, there is a lack of an indicator that restricts individual differences in EEG. Therefore, this approach was abandoned, and an improved workload classification algorithm was subsequently developed to bridge this gap.

Compared to the first workload classification, the improved workload classification algorithm using task scenarios combining ECG and EEG showed better performance. The best classifier was Bagging, achieving an optimal AUC of 96% and recall of 96.7%. The workload classification improved accuracy by 26.0% of the first classification method. A comparison of the evaluation indicators between the two methods can be seen in

Table 7. Comparison of the evaluation indicators between the two methods.

Algorithm		SVM			NB			Bagging			DT			KNN
Model		M1	M2	Diff	M1	M2	Diff	M1	M2	Diff	M1	M2	Diff	M1	M2	Diff
Performance metrics	AUC	78.0	91.0	+3.0	74.0	91.0	+17.0	74.0	96.0	+22.0	69.0	93.0	+24.0	64.0	88.0	+24.0
	Accuracy	67.5	85.5	+18.0	72.5	87.5	+15.0	68.5	92.5	+24.0	73.5	89.5	+16.0	64.0	88.5	+22.5
	Precision	89.0	78.0	−11.0	78.0	87.0	+9.0	84.0	88.0	+4.0	79.0	85.0	+6.0	77.0	84.0	+7.0
	Recall	62.2	91.8	+29.6	70.3	87.9	+17.6	64.1	96.7	+32.6	71.2	93.4	+22.2	61.1	92.3	+31.2
	F1-score	73.2	84.3	+11.1	74.0	87.5	+13.5	72.7	92.1	19.4	74.9	89.0	+14.1	68.1	88.0	+19.9

(Note: M1 = model 1, M2 = model 2, i.e., improved model, Diff = difference, unit: %).

. This result validates the effectiveness of using ECG and EEG metrics to determine seafarer workload. The introduction of the individually discriminative ECG when classifying EEG according to the task scenarios offsets individual differences to a certain extent. That makes the classifier fulfill the need for multi-person workload classification and facilitates rapid recognition of the fitness for the duty of seafarers.

The classification results of the machine learning model proposed in this study validate previous research findings that comprehensive measures based on driver physiological data can better represent workload in complex maneuvers involving varying levels of effort [16]. The study of [35] can measure the mental load and stress experienced by seafarers during training based on EEG, and output indicative recommendations such as “pass”, “retrain”, or “fail”.

6.3. Implications and Limitations

The main contribution of this study is to propose the method for workload recognition of ship pilots based on task scenarios, ECG, and EEG. The method evaluated the performance of multiple machine learning algorithms and found that the Bagging algorithms were the classifiers with the best classification accuracy. It showed good applicability with a classification accuracy of up to 92.5%, which improved accuracy by 25.9% to model 1.

There are some implications of this study. The improved model of seafarer workload classification trained in this study (Figure 9) can develop a multi-dimensional and multi-modal perception of on-the-job status, online intelligent monitoring, dynamic risk identification, and human–computer interaction active early warning technology and equipment. Specifically, it can be used to develop an intelligent monitoring and warning system for seafarers’ on-duty workload. This research will provide a theoretical basis for addressing the challenges of workload recognition and risk behavior modification for seafarers, and also risk early warning trigger mechanism design.

Limitations were also faced in this study. First, the experiment is carried out in a bridge simulator, which lacks the realism brought by the ship. Second, the experiment was conducted while the seafarers were in training, not during exams. There is no real-time evaluation process for the performance of each complex task.

Future studies are recommended to be carried out in the real ship navigation environment. This will further calibrate the environmental perception bias that the bridge simulator brings to the seafarer. Moreover, the experiment should form a complete experimental evaluation system combined with the performance of the pilot tasks.