Develop a Comprehensive Method to Evaluate the Mental Workload of Ship Operators

Abstract

Mental workload has become an important factor affecting the human error of ship operators. Controlling the mental workload of operators within a reasonable range can reduce human errors. The purpose of this study is to develop an evaluation method to evaluate the ship operator’s mental workload. First, the evaluation indices system was constructed according to three types of mental workload measurements. Second, the criteria importance though intercrieria correlation (CRITIC) method and analytic hierarchy process (AHP) method were used to determine the relative weight of each index. Finally, the fuzzy theory was used to calculate ship operator’s mental workload. The experiment results indicated that subjective workload assessment technique, physiological measurement (eye response), and error rate can be integrated into the comprehensive evaluation method to assess the mental workload of ship operators. Thus, this method can be used to comprehensively evaluate the mental workload level and improve the reliability of the assessment results.

1. Introduction

The marine transportation industry plays an increasingly important role in global transportation. Although a lot of work has been done to enhance the safety of maritime transportation, human error remains one of the leading causes of marine accidents. Research reports on maritime accidents show that about 80–85% of accidents involve human error [1]. Ship operation is a heavy mental workload task, particularly in the complex operation procedures. The mental workload has become an important factor affecting human errors. Low or high mental workload will have a negative impact on human performance [2]. Keeping mental workload within a reasonable range can decrease human error and improve system security [3]. As the improvement of automation level of ship control room, the operation tasks are primarily cognitive tasks, which will cause the change of the operator’s mental workload level. Consequently, it is necessary to assess the operators’ mental workload to improve the operators’ performance.

Mental workload is defined as the amount of mental work required by person to complete a specific task [4]. When the operator is engaged in cognitive tasks, information processing needs to call on the operator’s cognitive resources. As the task load increases, more and more resources are needed. When the mental workload exceeds a reasonable range, performance suffers and errors begin to occur. At present, the evaluation methods of mental workload are generally divided into three categories: subjective measurement, physiological measurement, and performance measurement. Subjective measure, such as Positive Affect and Negative Affect Schedule [5], Subjective Workload Assessment Technique (SWAT) [6], Activation-Deactivation Adiective Check List [7], National Aeronautics and Space Administration’s Task Load Index (NASA-TLX) [8], and Modified Cooper-Harper Ratings [9], have been widely used to assess the operator’s mental workload. Most subjective measurement results generally refer to the measurement of mental workload without considering the impact of physical labor [10]. However, subjective measurement is easily influenced by personal factors, such as preferences, prejudices, and protect attitudes. Thus, it is not recommended to use subjective measurement when the evaluation results are easily affected by personal factors [11]. Physiological measurement is to measure mental workload by collecting physiological parameters. This method requires the instrument to be connected with the person during the measurement process, which may restrict the movement of the person. However, this method requires only a few samples to provide accurate results.

Eye response is a physiological measurement method which can measure the operators’ mental workload in real time [12]. Eye response measurement is a technique to analyze the eye response to visual stimuli. Generally, the indexes of pupil diameter, fixation rate, saccadic rate, and blink rate are combined to evaluate mental workload [13,14]. Pupil dilation has a correlation with the amount of cognitive control, attention, and cognitive requirement; it can reflect the mental workload level [15,16]. Nevertheless, the pupil diameter is easily affected by changes of light environment, so the experimental results cannot accurately reflect the level of mental workload. Therefore, pupil dilation is usually combined with blink indices, saccadic rate, and fixation to assess cognitive requirement in different tasks. Blink rate increased with increased load resulting from memory tasks, but it declined with increased workload resulting from processing visual stimuli [17]. Eye fixation refers to the stay of the human eyes in a specific position. This may be because the operator is interested in specific information, or it may be because the information is difficult to understand. It is found that this index has a correlation with task difficulty and can be used to measure mental workload [18]. In addition, the saccadic rate has a negative correlation with mental workload [19].

Through the above analysis, subjective measurement is easily affected by personal factors. Physiological measurement can be affected by external environmental factors and emotions. In addition, performance measurement requires many experiments to obtain reliable results. A single measurement method has certain limitations. The comprehensive measurement method of multiple indices can make up for the shortcomings of various methods. Therefore, the purpose of this research is to develop a method to measure the mental workload of ship operators based on subjective measurement, physiological measurement, and performance measurement. According to the measurement method, an evaluation index system is established, and the CRITIC method and AHP method are used to calculate the relative weight of each index. Finally, the mental workload of the ship operator is evaluated according to the fuzzy theory. The developed method has an important impact on improving human performance and reducing the number of accidents.

2. Comprehensive Evaluation Method

2.1. Comprehensive Evaluation Index System

The first step of the comprehensive evaluation method is to construct an evaluation index system. This study selects workload evaluation indices from three aspects: physiological measurement, subjective measurement, and performance measurement. Physiological measurement methods are easily affected by emotional and physical factors because the device is in contact with the user’s body, which limits the real state of the user. Nevertheless, physiological measurement requires only a few samples to obtain accurate evaluation results. Therefore, this paper selects the eye response measurement method to measure the operator’s mental workload level in real time.

During the experiment, the iView X head mounted eye-tracking device (SensoMotoric Instruments, Teltow, Germany) with pupil/corneal reflection <0.1 and a rate of 50 Hz was used to record the eye response. The experimental data were analyzed and processed using the BeGaze software (version 3.0). Raw eye tracking data were recorded in the form of XY coordinates according to sampling rate.

Subjective workload assessment is becoming more and more widely used. This method is easy to implement and low cost. However, subjective assessment results are easily influenced by personal characteristics, such as bias, errors, and preferences [20] Therefore, only relying on the subjective evaluation results cannot accurately reflect the operator’s workload level. This research used the SWAT method to assess the subjective workload. SWAT evaluates the workload from three dimensions: mental load, time load, and stress load. In this study, an improved version that is more sensitive to the medium-level workload was used to evaluate the workload [21]. Participants marked a visual analog scale with a horizontal length line of 16.5 cm, and made a mark at the beginning, middle, and end of the horizontal line to represent the workload from high to low. The level of each dimension of SWAT method is measured by the marked horizontal line (lowest score 0 cm, highest score 16.5 cm), and the total workload is the sum score of the three dimensions (the highest score is 49.5 cm).

Performance measurement methods can be classified into many categories, such as task time, accuracy, domain-specific measures, worst-case performance, etc. [22]. The error rate is the key index to measure human performance. This index has been applied in many fields to measure workload [23,24]. The operator’s mental workload is not within the reasonable range, which is the cause of the errors, and the errors affect the system operation safety. Thus, this study selected the error rate as the performance measurement index.

2.2. Normalization of Evaluation Matrix

With m evaluating indices and n participants count form an original index value matrix:

$$X=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1m}\\ x_{21}& x_{22}& \cdots & x_{2m}\\ \cdots & \cdots & \cdots & \cdots \\ x_{n1}& x_{n2}& \cdots & x_{nm}\end{array}\right]$$

(1)

where, x_ij is the evaluation data of the ith participant on the jth index.

This matrix is normalized to get Equation (2):

$$H={\left[h_{ij}\right]}_{n\times m}$$

(2)

where, h_ij is the normalized value, and $h_{ij}\in \left[0,1\right]$.

2.3. Determining the Indices Weight

As an important part of fuzzy assessment method, calculating the index weight has an important impact on the evaluation results. The CRITIC method is an objective weighting method based on data volatility. The objective weighting method is based on the influence of quantitative indices of data information on the evaluation results. Because the objective weighting method needs to depend on sample data and actual problem domain, its universality and participation are poor, and it cannot reflect the importance of the judges on different index. Sometimes the determined weight will greatly differ from the actual importance of attributes. AHP method is a weighting method based on subjective evaluation results. In the subjective weighting method, the index weight is easily affected by subjective preference of the evaluator, and it is difficult to measure the importance of the index scientifically. This study combined the CRITIC method and the AHP method to calculate the index weight. The calculation procedure of CRITIC method is as follows [25]:

Calculate the mean square deviation of the ith influencing factor:

$$\overline{h_j}=\frac{1}{n}{\displaystyle \sum _{i=1}^n}{h}_{ij}\hspace{1em}j=1,2,\cdots ,m$$

(3)

$$s_j=\sqrt{\frac{1}{n-1}{\displaystyle \sum _{i=1}^n}{\left(h_{ij}-\overline{h_j}\right)}^2}\hspace{1em}j=1,2,\cdots ,m$$

(4)

where, ${\overline{h}}_j$ stands for the average value of the jth influencing factor, and $s_j$ is the mean square deviation of the jth influencing factor.

Calculate the correlation coefficient of jth index and kth index:

$$r_{jk}=\frac{{\displaystyle \sum _{i=1}^n}\left(x_{ij}-\overline{x_j}\right)\left(x_{ik}-\overline{x_k}\right)}{\sqrt{{\displaystyle \sum _{i=1}^n}{\left(x_{ij}-\overline{x_j}\right)}^2}\sqrt{{\displaystyle \sum _{i=1}^n}{\left(x_{ik}-\overline{x_k}\right)}^2}}\hspace{1em}i=1,2,\cdots ,n;j=1,2,\cdots ,m$$

(5)

Calculate the conflict of the jth index:

$$R_j={\displaystyle \sum _{i=1}^n}\left(1-r_{ij}\right)$$

(6)

Calculate the information content of the jth index

$$T_j=s_j{R}_j=s_j{\displaystyle \sum _{i=1}^n}\left(1-r_{ij}\right)\hspace{1em}i=1,2,\cdots ,n; j=1,2,\cdots ,m$$

(7)

The greater $T_j$, the greater the amount of information contained in the jth evaluation index, and the greater the relative importance of the index.

The calculation result of index weight by CRITIC method is:

$${\delta }_j=\frac{T_j}{{\displaystyle \sum _{j=1}^m{T}_j}}\hspace{1em}j=1,2,\cdots ,m$$

(8)

The procedure of calculating the index weight by AHP method is described as follows [26]:

The first step of AHP is to construct a judgment matrix. n participants judged the relative importance of the pairwise comparison of m indices through collective discussion. The judgment matrix is constructed according to the results of the comparison of m indices:

$$A=\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1m}\\ ⋮& \ddots & ⋮\\ a_{m1}& \cdots & a_{mm}\end{array}\right]$$

(9)

where a_ij represents the importance of ith index to jth index. The importance of the pairwise comparison of two indices is expressed on an integer scale from 1 to 9, the meaning of numbers is shown in

Table 1. Importance scale values.

Importance Scale Value	Description
1	The elements i and j are equally important.
3	The element i is moderately more important than the element j.
5	The element i is strongly more important than the element j.
7	The element i is very strong important than the element j.
9	The element i is extremely more important than the element j.
2,4,6,8	Intermediate value between two adjacent judgments.
Reciprocal	If the importance ratio of indexes i to factor j was aij, then the importance ratio of indexes j to indexes i was aji = 1/aij.

. The element a_ij satisfies a_ij > 0, a_ij = 1/a_ji, and a_ii = 1.

Column standardization:

$$\overline{a_{ij}}=a_{ij}/{\displaystyle \sum _{k=1}^m{a}_{kj}},(i,j=1,2,\cdots ,m)$$

(10)

Summation of the standardized column:

$$\overline{{\omega }_i}={\displaystyle \sum _{j=1}^m\overline{a_{ij}}}$$

(11)

Calculation of the eigenvector:

$${\omega }_i=\overline{{\omega }_i}/{\displaystyle \sum _{i=1}^m\overline{{\omega }_i}}$$

(12)

Find the largest eigenvalue:

$${\lambda }_{max}={{\displaystyle \sum }}_{i=1}^m{\left(A\omega \right)}_i/(m{\omega }_i)$$

(13)

λ_max is the largest eigenvalue of A. ω = (ω₁, ω₂, …, ω₈), the value of ω_i is the weight of the ith factor.

The maximum eigenvalue and eigenvector of the judgment matrix was calculated, and then the consistency index, random consistency ratio, and mean random consistency index were used to test the consistency. The calculation process of consistency index is as follows:

$$CI=({\lambda }_{\max }-m)/(m-1)$$

(14)

The parameter of CR is used to measure the consistency satisfaction for the judgement matrix. The calculation process is as follows:

$$CR=CI/RI$$

(15)

If CR < 0.1, the judgment matrix consistency is acceptable. Otherwise, the judgment matrix needs to be adjusted. The value of the average random consistency index RI is shown in

Table 2. The value of mean random consistency index RI.

m	1	2	3	4	5	6	7	8	9	10
RI	0	0	0.52	0.89	1.12	1.26	1.36	1.41	1.46	1.49

.

The result of the combination of the CRITIC method and the AHP as shown in Equation (16):

$${\eta }_j=\lambda {\delta }_j+\left(1-\lambda \right){\omega }_j$$

(16)

where, $\lambda$ is the preference index, and $\lambda \in \left(0,1\right)$, $\eta =\left[{\eta }_j\right]$ = $\left[{\eta }_1, {\eta }_2\cdots {\eta }_m\right]$.

2.4. Determine Evaluation Criteria

The m indices $U=\left\{\begin{array}{cccc}{u}_1& u_2& \cdots & u_m\end{array}\right\}$ were used to evaluate the operator’s mental workload. For each index, evaluation criteria and ranks are carefully determined. V = {v₁ v₂ v₃ v₄ v₅} = {Very low Low Normal High Very high} is used to describe the five evaluation ranks of each index, the score of each evaluation criteria is H = {1 2 3 4 5}.

2.5. Determining Fuzzy Relation Matrix

According to the evaluation index system and membership function, the fuzzy relation matrix is determined as:

$$F={\left[f_{ij}\right]}_{m\times r}=\left[\begin{array}{ccc}{f}_{11}& \cdots & f_{1r}\\ ⋮& \ddots & ⋮\\ f_{m1}& \cdots & f_{mr}\end{array}\right]$$

(17)

where, $f_{ij}$ represents the fuzzy membership of the ith index belonging to the jth rank.

When the indices, such as pupil dilation, have positive correlation with mental workload level, then the fuzzy function is as follows:

$$f_{ik}\left(x\right)=\{\begin{array}{ccc}0& x\ge {\lambda }_{i(k+1)}\hfill & \\ \frac{{\lambda }_{i(k+1)}-x}{{\lambda }_{i(k+1)}-{\lambda }_{ik}}& {\lambda }_{ik}<x<{\lambda }_{i(k+1)}\hfill & k=1\\ 1& x\le {\lambda }_{ik}\hfill & \end{array}$$

(18)

$$f_{ik}\left(x\right)=\{\begin{array}{ccc}0& x\le {\lambda }_{i(k-1)}, x\ge {\lambda }_{i(k+1)}\hfill & \\ \frac{x-{\lambda }_{i(k-1)}}{{\lambda }_{ik}-{\lambda }_{i(k-1)}}& {\lambda }_{i(k-1)}<x<{\lambda }_{ik}\hfill & k=2,3,4\\ \frac{{\lambda }_{i(k+1)}-x}{{\lambda }_{i(k+1)}-{\lambda }_{ik}}& {\lambda }_{ik}\le x<{\lambda }_{i(k+1)}\hfill & \end{array}$$

(19)

$$f_{ik}\left(x\right)=\{\begin{array}{ccc}0& x\le {\lambda }_{i(k-1)}\hfill & \\ \frac{x-{\lambda }_{i(k-1)}}{{\lambda }_{ik}-{\lambda }_{i(k-1)}}& {\lambda }_{i(k-1)}<x<{\lambda }_{ik}\hfill & k=5\\ 1& x\ge {\lambda }_{ik}\hfill & \end{array}$$

(20)

where, x is the assess value of the ith index; ${\lambda }_{ik}$ is the kth rank threshold of the ith index; and $f_{ik}$ is the fuzzy membership of the ith index belonging to the kth rank.

When the indices, such as saccadic rate, have negative correlation with mental workload level, then the membership $f_{ik}$ can be given as:

$$f_{ik}\left(x\right)=\{\begin{array}{ccc}1& x\ge {\lambda }_{ik}\hfill & \\ \frac{x-{\lambda }_{i(k+1)}}{{\lambda }_{ik}-{\lambda }_{i(k+1)}}& {\lambda }_{i(k+1)}<x<{\lambda }_{ik}\hfill & k=1\\ 0& x\le {\lambda }_{i(k+1)}\hfill & \end{array}$$

(21)

$$f_{ik}\left(x\right)=\{\begin{array}{ccc}0& x\le {\lambda }_{i(k+1)}, x\ge {\lambda }_{i(k-1)}\hfill & \\ \frac{{\lambda }_{i(k-1)}-x}{{\lambda }_{i(k-1)}-{\lambda }_{ik}}& {\lambda }_{ik}<x<{\lambda }_{i(k-1)}\hfill & k=2,3,4\\ \frac{x-{\lambda }_{i(k+1)}}{{\lambda }_{ik}-{\lambda }_{i(k+1)}}& {\lambda }_{i(k+1)}\le x\le {\lambda }_{ik}\hfill & \end{array}$$

(22)

$$f_{ik}\left(x\right)=\{\begin{array}{ccc}1& x\le {\lambda }_{ik}\hfill & \\ \frac{{\lambda }_{i(k-1)}-x}{{\lambda }_{i(k-1)}-{\lambda }_{ik}}& {\lambda }_{ik}<x<{\lambda }_{i(k-1)}\hfill & k=5\\ 0& {\lambda }_{i(k-1)}\le x\hfill & \end{array}$$

(23)

2.6. Calculating Evaluation Result

The membership degree of the evaluation results at different ranks can be calculated through the fuzzy relationship matrix and the index weight vector:

$$M=\eta \times F=\left[{\eta }_1, {\eta }_2\cdots {\eta }_m\right]\times \left[\begin{array}{ccc}{f}_{11}& \cdots & f_{1r}\\ ⋮& \ddots & ⋮\\ f_{m1}& \cdots & f_{mr}\end{array}\right]$$

(24)

Fuzzy comprehensive evaluation scores are:

$$C=H\times M^T$$

(25)

3. Case Study

3.1. Participants

In this study, 22 students with the mean age and standard deviation of 25 ± 1.9 years were invited to participate in the experiment. All participants have excellent engineering education background and are familiar with the computer operations. Moreover, all of them have good vision ability, are right-handed, and had no physical discomfort on the process of the experiment. All participants were recruited in an open manner. Before the experiment, all participants were told the specific content of the experiment in detail. During the experiment, operators’ eye responses were recorded by the eye-tracking device. After the experiment, the participants were invited to fill in the questionnaire. All participants volunteered to participate in the experimental study, and the collected data were also allowed by them. Participants could stop the experiment and withdraw the experiment data without any reason. All experimental methods were implemented in accordance with the relevant regulations, and the experimental scheme was approved by the institutional review committee of Harbin Engineering University.

3.2. Operation Task

The function of the stern tube system is to support the stern shaft or propeller shaft, and make it reliably pass out of the ship, so as not to make a large amount of outboard water leak into the ship, and to prevent the lubricating oil from leaking out. Therefore, the stern tube system has an important impact on the safe operation of ships. In this study, the stern tube system operation procedure was carried out in the simulator MC90-V developed by Kongsberg maritime. The specific operation procedures are shown in

Table 3. Stern tube system operation procedure.

Steps	Operation Procedure
1	Ensure cooling water to stern tube cooler.
2	Refill lubricating oil sump tank if necessary.
3	Select required gravity tank using 3-way valve in filling line.
4	Select correct gravity feed to stern tube.
5	Ensure stern seal isolating valve is open.
6	Start the lubricating oil pump in manual.
7	When one pump is started, set the other pump in Auto.
8	If the running pump is unable to maintain the level in the gravity tank, the stand-by pump starts automatically.
9	Check level of oil in sealing tank, fill from make-up valve. Drain water if required.
10	Stop of pumps to be carried out manually

. Figure 1 is the operation interface of simulator MC90-V. The software was developed based on the real diesel engine data [27].

3.3. Acquisition of Experimental Data

The calculation process of SWAT score includes two steps. The first step is to score each dimension to determine the impact of each dimension on mental workload. In this step, the improved version of visual analysis scales is used to score each dimension. The second step is to sum the scores of the three dimensions as SWAT scores.

Error rate refers to the ratio of the number of incorrect operations to the number of actual operations. The eye response data of all participants were processed, and the average value of the whole task was taken as the final eye response parameter of participants. The area of interest is defined as the human–computer interface. During the experiment, only the data in the area of interest were collected, and all the data outside the area of interest were excluded. The results of the participants’ experimental data are summarized in

Table 4. The evaluation results of all indices.

	Pupil Dilation	Blink Rate	Fixation Rate	Saccadic Rate	SWAT	Error Rate
Mean	50.3	0.43	1.06	1.28	23.8	0.02
Standard deviation	6.2	0.09	0.43	0.25	3.19	0.04

.

3.4. Evaluation Index Weight

Using the data shown in Table 4 and based on Equations (1)–(4), the mean square deviation of the influencing factor is:

$$S_j=\left[ 0.3262 0.3050 0.3470 0.2799 0.2869 0.4289\right]$$

The conflict of index is:

$$R_j=\left[ 3.503 3.671 3.395 5.058 3.282 3.729\right]$$

The information content of the index is:

$$T_j=\left[ 1.143 1.120 1.178 1.416 0.942 1.599\right]$$

The calculation result of index weight by CRITIC method is:

$${\delta }_j=\left[ 0.1545 0.1514 0.1592 0.1914 0.1273 0.2162\right]$$

The judgment matrix of AHP is:

$$A=\left[\begin{array}{cccccc}1& \frac{1}{2}& 1& 1& \frac{1}{5}& \frac{1}{4}\\ 2& 1& 2& 2& \frac{1}{4}& \frac{1}{3}\\ 1& \frac{1}{2}& 1& 1& \frac{1}{5}& \frac{1}{4}\\ 1& \frac{1}{2}& 2& 2& \frac{1}{5}& \frac{1}{4}\\ 5& 4& 5& 5& 1& 2\\ 4& 3& 4& 4& \frac{1}{2}& 1\end{array}\right]$$

According to Equations (10)–(12), the calculation result of index weight by AHP method is as follows:

$$\omega =\left[ 0.0682 0.1189 0.0682 0.0682 0.4041 0.2724\right]$$

CI = 0.011, CR = 0.0088, the consistency of the judgment matrix is considered acceptable.

According to Equation (16), the calculation result of index weight using CRITIC method and AHP method is as follows:

$$\eta =\left[ 0.11 0.14 0.11 0.13 0.27 0.24\right]$$

According to the calculated result of CRITIC and AHP methods, the relative weights of pupil dilation, blink rate, fixation rate, saccadic rate, SWAT, and error rate were 0.11, 0.14, 0.11, 0.13, 0.27, and 0.24, respectively.

3.5. Comprehensive Evaluation Result

Before constructing the fuzzy membership degree matrix, the threshold of each index under different evaluation ranks needs to be determined. In this study, the indices were divided into five evaluation ranks: very low, low, normal, high, and very high. The thresholds of each index under different evaluation ranks are shown in

Table 5. Parameter values in membership function.

		Pupil Dilation	Blink Rate	Fixation Rate	Saccadic Rate	SWAT	Error Rate
K = 1	${\lambda }_{ik}$	40	0.2	0.5	1.65	15	0.01
	${\lambda }_{i\left(k+1\right)}$	45	0.4	0.9	2	20	0.025
K = 2	${\lambda }_{i\left(k-1\right)}$	40	0.2	0.5	1.3	15	0.01
	${\lambda }_{ik}$	45	0.4	0.9	1.65	20	0.025
	${\lambda }_{i\left(k+1\right)}$	50	0.6	1.3	2	25	0.04
K = 3	${\lambda }_{i\left(k-1\right)}$	45	0.4	0.9	0.95	20	0.025
	${\lambda }_{ik}$	50	0.6	1.3	1.3	25	0.04
	${\lambda }_{i\left(k+1\right)}$	55	0.8	1.7	1.65	30	0.055
K = 4	${\lambda }_{i\left(k-1\right)}$	50	0.6	1.3	0.6	25	0.04
	${\lambda }_{ik}$	55	0.8	1.7	0.95	30	0.055
	${\lambda }_{i\left(k+1\right)}$	60	1	2.1	1.3	35	0.07
K = 5	${\lambda }_{i\left(k-1\right)}$	55	0.8	1.7	0.6	30	0.055
	${\lambda }_{ik}$	60	1	2.1	0.95	35	0.07

.

According to Equations (18)–(23), the fuzzy membership matrix F was obtained.

$$F=\left[\begin{array}{ccccc}0& 0& 0.94& 0.06& 0\\ 1& 0& 0& 0& 0\\ 0& 0.6& 0.4& 0& 0\\ 0& 0& 0.94& 0.06& 0\\ 0& 0.24& 0.76& 0& 0\\ 0.33& 0.67& 0& 0& 0\end{array}\right]$$

Using the Equation (24), the membership degree of the comprehensive evaluation results at different ranks is:

$$M=\eta \times F=\left[0.2192 0.2916 0.4748 0.0144 0\right]$$

According to the maximum membership principle, the operator’s mental workload is “Normal” level. Using Equation (25), the fuzzy comprehensive evaluation score is 2.844.

3.6. Validation of Evaluation Results

The developed assessment method was verified by the correlation analysis between the comprehensive evaluation scores and the other indices. The fuzzy comprehensive evaluation score of all participants were shown in Figure 2. According to the maximum membership principle, the number of participants with workload levels in “very low”, “low”, “normal”, and “high” were 3, 9, 8, and 2, respectively.

Table 6. Correlation between the evaluation score and other indices data.

		Pupil Dilation	Blink Rate	Fixation Rate	Saccadic Rate	SWAT	Error Rate
Pupil dilation	Correlation	1
	Sig.(2-tailed)
Blink rate	Correlation	0.21	1
	Sig.(2-tailed)	0.34
Fixation rate	Correlation	0.59 **	0.39	1
	Sig.(2-tailed)	0.00	0.07
Saccadic rate	Correlation	−0.05	0.29	−0.18	1
	Sig.(2-tailed)	0.82	0.18	0.43
SWAT	Correlation	0.45 *	0.33	0.58 **	−0.21	1
	Sig.(2-tailed)	0.04	0.13	0.00	0.35
Error rate	Correlation	0.29	0.10	0.22	0.09	0.57 **	1
	Sig.(2-tailed)	0.18	0.67	0.32	0.70	0.00
Evaluation score	Correlation	0.75 **	0.42	0.84 **	−0.34	0.85 **	0.40
	Sig.(2-tailed)	0.00	0.06	0.00	0.12	0.00	0.06

**: Correlation is significant at 0.01 level. *: Correlation is significant at 0.05 level.

shows the correlation analysis results between the comprehensive evaluation scores and other indices. The result showed that the comprehensive evaluation result has a positive correlation with pupil dilation (r = 0.75, p < 0.01), fixation rate (r = 0.84, p < 0.01), and SWAT (r = 0.85, p < 0.01).

4. Discussion

The purpose of this study is to develop a method to assess the mental workload of ship operators. Eye response, subjective score, and error rate were used to assess the mental workload. Due to the improvement of automation technology and the application of computer technology, the operators realize the monitoring of the system through the digital human–computer interface. The main tasks of the operator are cognitive tasks. Eye response index can reflect the change of mental workload level. SWAT is a subjective assessment method. To obtain the accuracy of the evaluation results, it needs to be combined with the objective evaluation method. Increasing operators’ mental workload, or overload, may cause delayed processing of information since the amount of information exceeds their ability to process information. In contrast, when their mental workload decreases from a suitable level, they feel bored and tend to make errors. Therefore, the human performance is affected by too high or too low mental workload. Therefore, correctly evaluating the workload of operators can optimize mental workload allocation, reduce human errors, improve system safety, and improve operator satisfaction. In the long-term, accurately evaluating the mental workload to realize reasonable mental workload allocation can maintain the mental workload of operators within a reasonable range and reduce the impact of long-term unreasonable mental workload on the health of operators.

The weight of the overall eye response indices is up to 0.49, which indicated that the measurement method is particularly important for evaluating operators’ mental workload. One reason is that the four eye response indices are directly related to cognitive workload, which can usually reflect the level of cognitive workload. In the digital human–computer interface operation, the operator’s task is mainly cognitive task, which determines that the workload is mainly cognitive workload. Therefore, the weight value of the overall eye response indices is the highest among all indices. The weight of blink rate is the highest in all eye response indices, because the blink rate increases with the increase of workload caused by memory task. Pupil dilation is related to cognitive control and attention. It reflects the difficulty of getting information from digital human–computer interface. The pupil diameter in highly complex tasks is larger than that in low complexity tasks. The fixation rate can also be used to reflect the mental workload status, which is positively correlated with task difficulty. Task complexity is an important factor affecting workload. When the operator’s fixation is higher on a specific area, it may indicate greater interest in the target, or the target is more complex. However, eye response methods may be affected by environmental and emotional factors. Therefore, this method is usually combined with subjective workload method to measure the mental workload. The SWAT weight value is 0.27, indicating that this method has a significant impact on the comprehensive evaluation results.

The comprehensive evaluation result indicated that the mental workload is in “normal” level, and the fuzzy evaluation score is 2.844. The effectiveness of the developed method was tested by analyzing the correlation between the comprehensive scores of all participants and other indices. The analysis result indicated that the comprehensive evaluation score is correlated with pupil dilation, fixation rate, and SWAT, demonstrating that the developed method can be used to evaluate the mental workload level. However, there are still some indices that do not show correlation with the comprehensive evaluation results, which may be due to the fact that these indices are not sensitive to mental workload.

However, the developed method has some limitations. First, the experience of participants is different from that of operators with many years of operation experience. Second, this experience used a small number of samples, which reduces the statistical ability. Third, the experiment was implemented on the simulation platform. Therefore, the experiment result needs to be validated in a field study.

5. Conclusions

This research constructed a method to evaluate mental workload. The evaluation index system is established by integrating various workload evaluation methods to improve the rationality of the indices selection. The CRITIC and AHP methods were used to determine the index weight to avoid deviation caused by subjective factors. Taking the average measurement results as the indices level, the comprehensive evaluation results indicated that the mental workload of the operator is at the “normal” level. In addition, the correlation analysis indicated that the scores of the comprehensive evaluation results are related to some evaluation indices. This means that the comprehensive evaluation method can effectively evaluate the mental workload. Therefore, this method can be used as a tool to evaluate the mental workload. Further, future study will investigate other samples which are above 30 years old (not student) to investigate the evaluation results of different ages.