A Multi-Feature Fusion and Situation Awareness-Based Method for Fatigue Driving Level Determination

Abstract

The detection and evaluation of fatigue levels in drivers play a crucial role in reducing traffic accidents and improving the overall quality of life. However, existing studies in this domain often focus on fatigue detection alone, with limited research on fatigue level evaluation. These limitations include the use of single evaluation methods and relatively low accuracy rates. To address these issues, this paper introduces an innovative approach for determining fatigue driving levels. We employ the Dlib library and fatigue state detection algorithms to develop a novel method specifically designed to assess fatigue levels. Unlike conventional approaches, our method adopts a multi-feature fusion strategy, integrating fatigue features from the eyes, mouth, and head pose. By combining these features, we achieve a more precise evaluation of the driver’s fatigue state level. Additionally, we propose a comprehensive evaluation method based on the analytic hierarchy process (AHP) and fuzzy comprehensive evaluation, combined with situational prediction. This approach effectively evaluates the fatigue state level of drivers at specific moments or stages and provides accurate predictions. Furthermore, we optimize the gated recurrent unit (GRU) network using an enhanced marine predator algorithm (MAP), which results in significant improvements in predicting fatigue levels during situational prediction. Experimental results demonstrate a classification accuracy of 92% across various scenarios while maintaining real-time performance. In summary, this paper introduces a novel approach for determining fatigue driving levels through multi-feature fusion. We also incorporate AHP-fuzzy comprehensive evaluation and situational prediction techniques, enhancing the accuracy and reliability of fatigue level evaluation. This research holds both theoretical and practical significance in the field of fatigue driving.

1. Introduction

Since the beginning of the 21st century, with the continuous improvement in people’s living standards, urbanization has been accelerating. To meet the increasing demand for transportation, the demand for cars has also surged. However, along with this, numerous problems have emerged. The most prominent issue is the high frequency of traffic accidents, resulting in significant losses for the country and its people. According to survey statistics, the number of fatalities in traffic accidents in China exceeded 90,000, accounting for 2.3% of the total number of deaths [1]. Furthermore, the average fatality rate in traffic accidents in China is 31.3%, which is considerably higher than in other countries, highlighting the severity of the problem. Analysis of the causes of traffic accidents reveals that over 30% of accidents are attributed to driver-related factors. Real-time monitoring of drivers allows for the detection of early signs of fatigue and enables timely fatigue warnings to be issued, helping drivers adjust their state and effectively reducing the risk of fatigue-related accidents, thereby ensuring the safety of individuals and their property.

Currently, studies on driver fatigue detection can be categorized into three main approaches [2,3,4], as shown in

Table 1. Commonly used fatigue detection methods and evaluation.

Detection Method	Description	Accuracy Rate
Detection based on driver physical data	Detection of changes in the driver’s body’s psychological signals	Good
Detection based on operating vehicle operation status	Detection of driver’s operation behavior and changes in vehicle behavior	General
Facial detection based on computer vision	Detection of the driver’s facial state change	Good

: physical-data-based detection [5,6], vehicle-operation-based detection [7], and facial-feature-analysis-based detection [4,8]. Comparing the three detection methods, it is found that the analysis of facial features based on the driver’s face has the lowest detection cost and achieves the best detection performance. Additionally, the data collection method is relatively simple. However, the pure object detection method for detecting the driver’s fatigue state, although greatly improving the detection accuracy, can easily overlook small features when the driver is in a fatigued state due to the high similarity of facial and head poses. Therefore, it is crucial to study fine-grained features at arbitrary small scales, as it plays an important role in determining the level of driver fatigue. Furthermore, when the driver is engaged in fatigue driving, the level of fatigue varies over time. While computer-vision-based methods are effective in detecting the driver’s facial features, further research is needed to deeply understand the driver’s fatigue level.

Therefore, we propose a comprehensive approach that combines driver facial recognition using the Dlib library with the MAR, EAR, and HPE fatigue detection algorithms. By integrating multiple features from these different sources, our method enables a more comprehensive and accurate evaluation of the driver’s fatigue level. In terms of facial recognition, our method utilizes advanced algorithms and the Dlib library to detect and analyze facial expressions, with a particular focus on fatigue-related cues. This allows us to capture subtle changes in facial features and quantify them, providing indications of fatigue and enhancing the overall effectiveness of our subsequent fatigue level determination. For eye fatigue detection, we employ the Dlib library to track and analyze eye movements and features associated with fatigue, such as drooping eyelids and prolonged blink durations. This enables accurate identification and quantification of the level of eye fatigue experienced by the driver. Similarly, our method includes mouth and head fatigue testing, using the Dlib library to monitor mouth movements and head posture to assess signs of fatigue. By analyzing these specific features, we gain a deeper understanding of the driver’s fatigue state. Overall, the main contribution of this paper is the utilization of driver facial recognition using the Dlib library, combined with the MAR, EAR, and HPE fatigue detection algorithms to extract fatigue features and classify fatigue driving levels. Ultimately, we provide a comprehensive and reliable method for evaluating the driver’s fatigue driving level. By considering multiple fatigue indicators, our method offers a more comprehensive understanding of the driver’s condition, thereby enhancing safety and reducing accident risks.

This paper is structured as follows: Section 1 provides an introduction to the background and significance of fatigue driving. We discuss the motivation behind this paper and highlight our research methods and work in this field. Section 2 presents a detailed literature review, examining the existing studies and theories related to fatigue detection. We discuss the key findings and propose a multi-feature fusion analysis method adopted in this paper. In Section 3, we describe the feasibility of the methods employed in our paper and provide a description and improvement of the relevant methodology. Section 4 explains the data collection process, the experimental setup, and the statistical analysis techniques used to analyze the gathered data and presents the results and findings of our study. We discuss the empirical evidence and present the quantitative and qualitative analysis of the data. We also include visual representations such as figures and tables to support our findings. Finally, in Section 5, we draw conclusions based on our paper’s findings. We summarize the main innovations and contributions of this research and discuss the implications for the fatigue driving field. We also consider the use of the Dlib library for fatigue detection and its applicability in safety-related fields. In summary, this paper aims to provide a comprehensive understanding of driver fatigue and fatigue driving by examining the literature, conducting empirical research, and presenting our findings. The following sections will delve into each aspect in detail.

2. Literature Review

2.1. Face Recognition and 68 Feature Point Positioning

The Dlib library is an open-source utility toolkit that employs powerful algorithms and techniques to achieve accurate and efficient tasks, including face detection, feature localization, pose estimation, and expression analysis. These functionalities make Dlib a popular choice for face-related tasks in both research and practical applications. Developed using C++, the library encompasses mature machine learning algorithms and models, enabling widespread applications and excellent performance in image processing and face feature extraction. The library offers various functionalities for detecting and analyzing facial features [9].

In summary, the Dlib library plays a crucial role in face-related tasks and has made significant contributions to the advancement of facial recognition. Its comprehensive features, reliability, and broad applicability have established its importance in the scientific community.

This paper utilizes the Shape_Predictor_68_face_landmarks.dat [10] model library from the Dlib library to perform accurate detection of the 68 key points on the human face. These key points, as depicted in Figure 1, can be precisely marked on the face and organized in a consistent order. By extracting the coordinates of these facial feature points, along with important information such as face boundaries and facial angles, valuable insights into an individual’s facial expression, state, and overall well-being can be obtained. This information serves as a foundation for assessing the person’s physiological and mental state.

2.2. Eye Fatigue Detection Based on EAR Algorithm

Eye features are an important parameter for judging the fatigue state, which can best reflect the driving state of the driver. Based on the eye aspect ratio (EAR) judgment concept of eye feature points proposed by Soukupova T [11] in 2016, after extracting 68 face key points through relevant face models or environment libraries, there are six key points for each eye, such as the position of the key points of the eye in Figure 2, where (x₃₆,y₃₆), (x₃₇,y₃₇), …, (x₄₁,y₄₁) corresponds to the key points of the left eye [12]. The EAR formula is shown in Formula (1).

$$EAR=\frac{\parallel y_{37}-y_{41}\parallel +\parallel y_{38}-y_{40}\parallel }{2\parallel x_{39}-x_{36}\parallel }$$

(1)

The driver’s eye state is judged by the formula of the EAR; that is, when the human eye is open, the EAR fluctuates up and down at a certain value, and when the human eye is closed, the EAR drops rapidly, and theoretically it will be close to zero. At that time, the face detection model is not yet precise. So, we think that when the EAR is below a certain threshold, the eyes are closed. Therefore, the initial threshold is set to 0.2. When the value of the EAR fluctuates around 0.2, the driver’s eyes are open, indicating that the driver is in a normal state at this time; when the value of the EAR decreases and is 0, the driver’s eyes are closed, indicating that the driver is in an abnormal state at this time.

2.3. Mouth Fatigue Test Based on the MAR Algorithm

There are two states of the mouth: closed and open. When the mouth is in the open state, according to its size, it can be divided into talking, eating, and yawning. Yawning is a reaction when the human body breathes deeply. When the brain feels bored, tired, or distracted, people will subconsciously yawn. Therefore, mouth features are also an important parameter for fatigue determination. The principle is similar to that of the eye parameters. The mouth aspect ratio (MAR) is used to judge the opening and closing of the driver’s mouth [13]. The main reference points of the mouth are shown in Figure 3. The MAR value is determined according to the upper lip and the coordinate information of the key points of the lower lip, and its calculation is shown in Formula (2).

$$\mathrm{MAR}=\frac{\parallel y_{61}-y_{67}\parallel +\parallel y_{62}-y_{66}\parallel +\parallel y_{63}-y_{65}\parallel }{3\parallel x_{64}-x_{60}\parallel }$$

(2)

When the mouth is open, the MAR tends to have a certain value, and when the mouth is closed, the MAR molecule is 0. When a person yawns, the mouth opening is significantly larger than when talking and eating, so the MAR value becomes significantly larger. Because the features of yawning are normal for people. There are obvious differences in the relaxation state, and this feature is set as a secondary fatigue feature detection point in this paper, so individual differences can be almost completely ignored. Based on the comprehensive consideration of multiple test data, this paper initially sets 0.6 as the threshold of the MAR value. Since yawning is a continuous state of the opening and closing of the mouth, Gallup Andrew C et al. [14] found through research that the whole process of opening and closing the mouth takes about 6.5 s when a person yawns once. Accordingly, in combination with the requirements of this paper, the setting conditions are improved, and it is set that the detection is greater than the threshold value of 0.6, and more than 10 times within 60 s are judged as fatigue, and the false detection rate is low at this time.

2.4. Head Fatigue Test Based on HPE Algorithm

The steps of the head pose estimation (HPE) algorithm are as follows: firstly, detect the 2D facial key points, and then utilize the 3D morphable model (3DMM) to fit the corresponding 3D facial models for different individuals. By aligning the 3D facial model, the conversion relationship between the 3D points and their corresponding 2D points is determined. Subsequently, the solvePnP() function in OpenCV is employed to solve the PnP (Perspective-n-Point) problem, allowing the computation of the Euler angles based on the rotation matrix. The Euler angles related to the head pose include the pitch angle (rotation around the x-axis), yaw angle (rotation around the y-axis), and roll angle (rotation around the z-axis), as illustrated in Figure 4 [15]. These angles are utilized to assess the state of the head.

These angles are used to map the two-dimensional coordinates in the image to three-dimensional space through four coordinate systems [16,17]: World Coordinate System (U, V, W), Camera Coordinate System (X, Y, Z), Image Center Coordinate System (u, v), and Pixel Coordinate System (x, y). Figure 5 below shows the relationship between the three coordinate systems.

Previous researchers have employed rotation matrices to solve the Euler angles used in head pose estimation. The rotations around the x-axis, y-axis, and z-axis, denoted by angles α, β, and γ, respectively, correspond to rotation matrices as shown in Equations (3)–(5).

$$R_X(\alpha )=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos (a)& -\sin (a)\\ 0& \sin (a)& \cos (a)\end{array}\right]$$

(3)

$$R_Y(\beta )=\left[\begin{array}{ccc}\cos (\beta )& 0& \sin (\beta )\\ 0& 1& 0\\ -\sin (\beta )& 0& \cos (\beta )\end{array}\right]$$

(4)

$$R_Z(\gamma )=\left[\begin{array}{ccc}\cos (\gamma )& -\sin (\gamma )& 0\\ \sin (\gamma )& \cos (\gamma )& 0\\ 0& 0& 1\end{array}\right]$$

(5)

The sine and cosine values of the Euler angles α, β, and γ around the x, y, and z axes, respectively, denoted as S_x, C_x, S_y, C_y, S_z, and C_z, are obtained as shown in Equation (6). Euler angles can be represented using a rotation matrix R.

$$R=R_X(\alpha )R_Y(\beta )R_Z(\gamma )=\left[\begin{array}{lll}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]=\left[\begin{array}{ccc}{c}_y{c}_z& c_z{c}_x{c}_y-c_x{c}_z& s_x{s}_z+c_x{c}_z{c}_y\\ c_y{s}_z& c_x{c}_z+s_x{s}_y{s}_z& c_x{s}_y{s}_z-c_z{s}_z\\ -s_y& c_y{s}_x& c_z{c}_y\end{array}\right]$$

(6)

The solution to the equation yields the result as shown in Equation (7).

$$\left\{\begin{array}{l}a=\arctan (r_{32}/r_{33})\\ \beta =\arctan (-r_{31}/\sqrt{r_{32}^2+r_{33}^2})\\ \gamma =\arctan (r_{21}/r_{11})\end{array}\right.$$

(7)

Normally, the average range of head movement for adults in yaw is [−79.8°, 75.3°], pitch [−60.4°, 69.6°], and roll [−40.9°, 36.3°]. Based on the observation that the head posture of a drowsy individual exhibits minimal movement in the yaw, this paper considers changes in the pitch and roll angles as the basis for fatigue state evaluation. Following the PERCLOS criteria, this paper determines incorrect head posture in any direction when the pitch angle (Pitch) exceeds [−20°, 20°] or the roll angle (Roll) exceeds [−15.4°, 15.4°].

Combined with the P80 standard, which has the largest correlation coefficient with the objective fatigue degree in the PERCLOS standard [18], this paper sets the Euler angle |Pitch| ≥ 20° for the head down or the Euler angle |Roll| ≥ 15.4° for the head tilt as the fatigue features of the head.

2.5. Multi-Feature Fusion Analysis

Although it is relatively easy to implement the above methods for identifying eye, mouth, and head states, there are some drawbacks. In actual driving environments, fatigue driving is a complex state that involves psychological and physiological factors, and the detection results can be easily influenced by various environmental interferences. If only eye or mouth features are extracted for fatigue detection, the accuracy of the detection will be significantly reduced. Therefore, the fatigue detection method based on individual features needs to be applied under ideal conditions. Some scholars [19] have proposed solutions to alleviate the problem of inaccurate detection of facial key points to some extent. In addition, the size of eyes and mouth can vary among individuals [20,21], and using a fixed threshold may lead to different results for different eye and mouth sizes. Reference [22] suggests training a classification library for each individual driver in advance to address individual differences. However, this method has limited generalization capability and places high demands on computer storage capacity.

Furthermore, since threshold values are set based on eye, mouth, and head poses to determine the driver’s fatigue state, there can be various interfering factors in the data collection process. For example, when the driver smiles, the eyes tend to close; when the driver sings, the mouth tends to yawn; and when the driver looks outside to observe traffic, the head pose can deviate significantly. To reduce the interference of these situations on fatigue state determination, this paper utilizes a fusion of multiple features for comprehensive fatigue evaluation. Based on the aforementioned MAR, EAR, and HPE algorithms, fatigue detection in drivers is performed, and the driver’s fatigue state is determined by setting appropriate thresholds to identify the presence of fatigue features. Specifically, a driver is classified as non-fatigued when the MAR, EAR, |Pitch|, and |Roll| fall within the respective ranges of [0, 0.6), (0.2, 1], [0, 20), and [0, 15.4), as shown in

Table 2. Principles of comprehensive judgment of fatigue status.

Whether It Is Fatigue	MAR	EAR	|Pitch|	|Roll|
Not tired	[0, 0.6)	(0.2, 1]	[0, 20)	[0, 15.4)
Tired	[0.6, 1]	[0, 0.2]	[20, 70]	[15.4, 36]

.

3. Research Methods

3.1. Situation Awareness Structure

Endsley [23] classifies situation perception into three levels: element extraction, evaluation, and prediction. The prediction stage aims to anticipate the “potential” future state. Initially, relevant environmental factors are perceived to understand the driver’s current state. This paper specifically focuses on analyzing head features as part of facial features [24,25] that influence the driving state. The evaluation step involves evaluating the current state, including the driver’s fatigue level and their present situation. Throughout the driver’s journey, various elements are considered to determine the degree of goal attainment and assess the current outcome using different data types and information. Future state projection predicts potential future states based on the current state [26], estimating the driver’s future driving risk. In summary, this paper studies the current and present situation levels to identify potential future trends and issue warnings for high-risk fatigue states. Situational perception technology is commonly applied in network environment warnings, determining the network threat level based on network environment information [27]. The process of fatigue state situational awareness is depicted in Figure 6.

The fatigue state of a driver is determined by calculating parameters related to eye fatigue, mouth fatigue, and head posture. Firstly, the fatigue level of the driver’s fatigue state and evaluation indicators for the fatigue parameters are defined, and mathematical modeling and quantification of each indicator are performed. An AHP-fuzzy comprehensive evaluation algorithm is proposed to assess the driver’s fatigue state. Subsequently, future situation levels are predicted based on past and present states, with warnings issued for higher-level states. To accomplish this, a situation prediction method based on the improved marine predator algorithm (WMPA)-optimized GRU (gate recurrent unit) is proposed.

3.2. Situational Element Extraction

3.2.1. Division of Fatigue Status Level

The Dlib library can be used to obtain the results of facial feature extraction from human faces. The first step in trend perception involves considering these factors as elements of the situation. Subsequently, situation evaluation and situation prediction are conducted, and evaluation rules need to be formulated beforehand. During the situation evaluation process, a comprehensive and reasonable evaluation of various factors is performed. The corresponding evaluation system is set based on the specific circumstances from different perspectives. It is important to note that the results obtained can vary significantly depending on the chosen approach. The focus of this research paper is the fatigue driving status of the driver. Due to the research on fatigue detection based on target detection in fatigue state studies, most studies have focused on detecting fatigue by analyzing the mouth and eyes of the face. However, with the introduction of head pose, the majority of research studies have started incorporating the detection of nodding frequency within a unit of time into the analysis of fatigue using facial features. In this paper, we adopt a method that primarily focuses on facial features with supplementary consideration of head pose. We utilize the Dlib library along with the MAR, EAR, and HPE fatigue detection algorithms to establish the fatigue state thresholds for drivers. By combining these thresholds, we can determine the driver’s fatigue level. Specifically, it investigates the signs of fatigue in the driver’s face and head and classifies the level of fatigue according to

Table 3. Fatigue status level judgment.

Fatigue State Level	Driver’s Face and Head Performance
Level I	The face does not show a special sign of fatigue, or only one fatigue feature is present on the head
Level II	The face only presents one fatigue feature, or the head presents two fatigue features
Level III	One fatigue feature on face and head each, or two fatigue features on the head
Level IV	There are two fatigue features on the face, only one fatigue feature on the head, or the head presents two fatigue features, and the face only presents one fatigue feature
Level V	The face presents two fatigue features, and two fatigue features appear on the head
Level Ⅵ	Arbitrary behavior without seat belt or airbags

below. A higher level indicates a more severe degree of fatigue, requiring increased monitoring and attention.

3.2.2. Construction of the Evaluation System

• Extract evaluation index

The driver’s fatigue state encompasses multiple contributing factors, and the complex interrelationships among these factors make it challenging to directly obtain the desired results. Therefore, it is crucial to extract evaluation indicators relevant to the evaluation objectives. The selection of indicators follows two principles: firstly, clearly defining the subject of the situation evaluation and, secondly, considering the attributes of the research subjects. Considering that the purpose of the situation evaluation is to determine the risk level of fatigue driving, which reflects the degree of driver fatigue, this relationship primarily manifests through the driver’s facial features, including mouth state, eye state, head pitch angle, head roll angle, and the presence of improper driving behaviors. Additionally, the driver’s physiological features and the environmental conditions inside the vehicle (such as temperature and driving duration) serve as indirect factors. Furthermore, weather conditions also influence the driver’s driving state. Based on the research conditions and objectives of this paper, the fatigue state level of fatigue driving will be determined by analyzing the facial features related to the direct factors. Therefore, the identified indicators for fatigue state include mouth state, eye state, head pitch angle, head roll angle, and the presence of improper driving behavior. Indicators 1 to 4 can be used to evaluate levels I to V, while indicator 5 is used to determine level VI. In subsequent algorithms, the scope of the situation evaluation will focus on levels I to V. Figure 7 below shows the fatigue state situation evaluation index system.

2.

Evaluation index quantification

Among the evaluation indicators, the relationship between fatigue and mouth, eyes, and head posture is highly significant. The opening and movement of the eyes, as well as the orientation of the head, directly impact the evaluation results. Head posture can be characterized by three Euler angles: pitch angle (Pitch), yaw angle (Yaw), and roll angle (Roll). In this paper, the changes in mouth state, eye state, head pitch angle, and roll angle are selected as the basis for judging fatigue state. The four evaluation indicators are quantified based on their respective numerical values. Based on the introduction of eye, mouth, and head fatigue detection algorithms in Section 2, the evaluation metrics for the eye and mouth can be quantified using Equation (1) and Equation (2), respectively. Similarly, the evaluation metrics for head pitch angle and roll angle can be quantified using Equation (7).

3.3. Situation Evaluation

The method of combining hierarchical analysis and fuzzy evaluation is referred to as the fuzzy hierarchical analysis method. The model establishment is depicted in Figure 8. The hierarchical analysis method is utilized to calculate the weight of the evaluation indicators that influence the level of driver’s fatigue status. Subsequently, the fuzzy comprehensive evaluation method is employed to construct a fuzzy evaluation matrix and conduct a comprehensive evaluation. Finally, the final result of the driver’s fatigue status level is determined based on the principles of fuzzy theory.

3.3.1. Calculation Weight of Hierarchical Analysis Method

• Build a judgment matrix

The judgment matrix serves as a criterion in determining the element values of the next layer. The values assigned to the matrix elements reflect the subjective understanding and evaluation of the relative importance of different factors based on objective reality. Typically, the values used are 1, 3, 5, 7, and 9, with their reciprocals as the benchmark values and 2, 4, 6, and 8 as intermediate values. The specific labeling method for the median value of adjacent judgments follows the 1–9 standard method proposed by Satty [28], as shown in

Table 4. Label meaning.

Scaling	Meaning
1	Represents equal importance between two elements being compared
3	Represents one element being slightly more important than the other element being compared
5	Represents one element being noticeably more important than the other element being compared
7	Represents one element being strongly more important than the other element being compared
9	Represents one element being extremely more important than the other element being compared
2, 4, 6, 8	Represent the relative importance between two elements being compared at levels between the above descriptions

.

2.

Calculate the maximum feature value

The maximum eigenvalue and eigenvector of each judgment matrix are calculated using the multiplication and summation method. Subsequently, the eigenvectors are normalized to obtain the weight ranking. Based on the weight rankings of each level, the overall weight ranking is determined. The specific calculation steps are as follows:

Step 1: Normalize each column of the judgment matrix P to obtain the normalized matrix $\overline{p}$.

$$\overline{P_{ij}}=\frac{P_{ij}}{{\displaystyle \sum _{k=1}^n{P}_{kj}}}(i,j=1,2,\dots n)$$

(8)

In the formula, $\overline{P_{ij}}$—line i, the value of the element in column j is formally treated; P_ij—line i, the element of column j; ${\displaystyle \sum _{k=1}^n{P}_{kj}}$—the harmony of column j element.

Step 2: Each column of regular judgment matrix $\overline{p}$ is added according to the row:

$${\overline{W}}_i={\displaystyle \sum _{j=1}^n{\overline{P}}_{ij}}\left(i,j=1,2,\dots ,n\right)$$

(9)

In the formula, $\overline{W_i}$—in the normalized judgment matrix, the sum of the ith row.

Step 3: Regularized vector $W={[\overline{W_1},\overline{W_2},\dots ,\overline{W_n}]}^T$:

$$W=\frac{{\overline{W}}_i}{{\displaystyle \sum _{i=1}^n{\overline{W}}_i}}\left(i=1,2,\dots ,n\right)$$

(10)

The obtained W = [W₁, W₂, …, W_n]^T is the feature vector.

Step 4: Calculate the maximum feature value of the matrix λ_max:

$${\lambda }_{\max }=\frac{{\displaystyle \sum _{i=1}^{n}P{W}_i}}{n{W}_i}$$

(11)

In the formula, PW_i is a matrix P and W product.

3.

Consistency check

Consistency check of weight vectors: Due to the complexity of objective phenomena or the potential bias in understanding them, it is necessary to assess the consistency and randomness of the judgment matrix by determining the feature values of the feature vector derived from it. The standardization steps are as follows:

Step 1: Calculate the consistency index (CI): To calculate the consistency index (CI), a survey was conducted among experts in 30 fields, resulting in the following judgment matrix: CI = (λ_max − n)/(n − 1).

Step 2: Calculate the average random consistency index (RI): The RI is obtained by averaging the feature values from multiple randomly generated judgment matrices.

Table 5. The value of RI.

n	1	2	3	4	5	6	7	8	9	10
RI	0	0	0.52	0.89	1.12	1.26	1.36	1.41	1.46	1.49

presents the values of RI for dimensions ranging from 1 to 10.

Step 3: Calculate CR: CR = CI/RI; when CR < 0.1, it is considered to have better consistency to judge the matrix.

4.

Calculation results

This paper formed an expert group consisting of 20 industry experts with senior titles or above in the field of object detection, as well as associate professors from universities. The selection process involved careful consideration of their expertise and industry experience. To ensure a comprehensive evaluation, multiple rounds of questionnaires were distributed to collect the experts’ opinions on the evaluation indices of driver fatigue status. The questionnaire was designed to assess the rationality of the preliminary selection of indicators based on the experts’ professional knowledge and industry experience. Moreover, by comparing the indicators pairwise, the relative importance between two indicators was determined using the 1–9 scale method. This approach helped capture the difference in significance between various indicators. Subsequently, the collected questionnaires underwent rigorous screening and elimination to exclude any that did not meet the requirements. The remaining data were analyzed and processed to obtain the judgment matrix, represented by matrix (12). This matrix reflects the comprehensive opinions and evaluations of the expert group.

$$P=\left[\begin{array}{cccc}1/1& 1/2& 3/1& 4/1\\ 2/1& 1/1& 7/1& 5/1\\ 1/3& 1/7& 1/1& 2/1\\ 1/4& 1/5& 1/2& 1/1\end{array}\right]$$

(12)

The calculation steps are as follows:

Step 1: Normalize each column vector of the pairwise comparison judgment matrix P to obtain $\overline{P}$, as represented by matrix (13).

$$\overline{P}=\left[\begin{array}{cccc}0.279& 0.2713& 0.2608& 0.3333\\ 0.56& 0.54& 0.61& 0.42\\ 0.09& 0.08& 0.09& 0.17\\ 0.07& 0.11& 0.04& 0.08\end{array}\right]$$

(13)

Step 2: Normalize and sum the matrix to obtain $\overline{W_i}$:$\overline{W}$ = [1.4450 2.1261 0.4241 0.3051]^T. The purpose of calculating w is to compute the normalized eigenvector.

Step 3: Calculate the eigenvector (weights), as shown in Equation (14).

$$\left\{\begin{array}{l}{\mathrm{W}}_{\mathrm{MAR}}=1.1445/(1.1445+2.1261+0.4241+0.3051)=0.2861\\ {\mathrm{W}}_{\mathrm{EAR}}=2.1261/(1.1445+2.1261+0.4241+0.3051)=0.5315\\ {\mathrm{W}}_{\mathrm{Pitch}}=0.4241/(1.1445+2.1261+0.4241+0.3051)=0.1060\\ {\mathrm{W}}_{\mathrm{Roll}}=0.3051/(1.1445+2.1261+0.4241+0.3051)=0.0763\end{array}\right.$$

(14)

Step 4: Translate the maximum eigenvalue of the judgment matrix, as shown in Equation (15).

$${\mathsf{\lambda }}_{\max }=\frac{{\displaystyle \sum _{i=1}^{n}P{W}_i}}{n{W}_i}=4.094$$

(15)

Step 5: To perform consistency check on the judgment matrix, we need to calculate the consistency index (CI): $CI=\frac{{\mathsf{\lambda }}_{\max }-n}{n-1}=\frac{4.094-4}{4-1}=0.031$.

For a 5th-order matrix, the average random consistency index (RI) is 0.89. The random consistency ratio (CR) is calculated as follows: $CR=\frac{CI}{RI}=\frac{0.031}{0.89}=0.035<0.10$.

Given that the consistency is reasonable, the weights can be summarized as shown in

Table 6. Weight summary table.

Index	MAR	EAR	Pitch	Roll
Weight	0.2861	0.5315	0.1060	0.0763

below.

3.3.2. Fuzzy Comprehensive Evaluation

• Establish a set of evaluation factors

Establish a set of evaluation factors, U = {u₁, u₂, u₃ …u₄}.

2.

Clear evaluation grade

The evaluation grade is represented by V = {V₁, V₂, V₃ …Vm}. This paper uses five grades to evaluate the driver’s fatigue status level evaluation effect. The five grades correspond to low risk (level Ⅰ), lower risk (level Ⅱ), general risk (level Ⅲ), higher risk (level Ⅳ), and high risk (level Ⅴ).

3.

Constructing a fuzzy comprehensive evaluation matrix

By clarifying the attachment of the lower-level evaluation indicators in the evaluation indicators of each ladder level, the ambiguity matrix R is represented by matrix (16):

$$R=\left[\begin{array}{c}{R}_1\\ R_2\\ \cdots \\ R_m\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1n}\\ r_{21}& r_{22}& \cdots & r_{2n}\\ \cdots & \cdots & \cdots & \cdots \\ r_{m1}& r_{m2}& \cdots & r_{mn}\end{array}\right]$$

(16)

4.

Calculate the evaluation factor right vector

Order W = {a₁, a₂, a₃, …, a_m}. Among them, a_i (indicator weight) represents the affiliation of u_i (evaluation factor set) for the fuzzy subset of the evaluated object. The evaluation factor vector is the weight value of the staircase level in the previous section. In accordance with the different values of the affiliation, this paper obtains the summary table of the affiliation matrix.

5.

Calculate the comprehensive evaluation results

After combining the synthetic comprehensive evaluation result vector with the corresponding level of evaluation factor right vector W and the fuzzy comprehensive evaluation matrix R, calculate the fuzzy comprehensive evaluation result vector B, as shown in Equation (17):

$$B=W\times R=\left[a_1,a_2,a_3,\cdots ,a_m\right]\times \left[\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1n}\\ r_{21}& r_{22}& \cdots & r_{2n}\\ \cdots & \cdots & \cdots & \cdots \\ r_{m1}& r_{m2}& \cdots & r_{mn}\end{array}\right]=\left[b_1,b_2,b_3,\cdots ,b_m\right]$$

(17)

Based on the principle of the most affiliated, the overall effect can be evaluated and analyzed.

3.4. Situation Prediction

After obtaining the situation evaluation result, which represents the “state” in the situation, it is crucial to focus on predicting the future trend of navigation, referred to as the “trend” in the situation. In the process of situation awareness, time plays a significant role as the situation evolves over time. Predicting future situations relies on capturing the long-term dependence of the situation on time. To address this issue, this section proposes an optimized GRU-based situation prediction method using the improved marine predator algorithm [29,30] to predict drivers’ fatigue levels. GRU (gated recurrent unit) [31,32] is a variant of the LSTM (long short-term memory) network [33,34,35], which has a simpler structure and better performance.

It is currently a popular choice in neural networks. In related fields, neural networks are widely used for situation prediction due to their ability to incorporate past and present data and handle temporal continuity. Recurrent neural networks, such as GRU and LSTM, are well-suited for such problems as they can store historical information. However, it should be noted that the activation function of the RNN network cannot be a sigmoid function. The sigmoid function leads to the vanishing gradient problem, where gradients diminish over time, making it difficult for RNNs to handle long-term dependencies. In response to this challenge, a neural network model capable of retaining information over long periods has emerged, known as long short-term memory (LSTM).

The LSTM (Long Short-Term Memory) module unit for information retention is more complex, consisting of cell states, forget gates, input gates, and output gates. This architecture enables LSTM to effectively handle long-term dependencies. Its unit structure is shown in Figure 9. The specific calculation steps for the proposed GRU-based situation prediction method optimized by the improved marine predator algorithm are as follows:

Step 1: The information that needs to be discarded is forgotten and output through the sigmoid function, Output f_t.

$$f_t=\mathsf{\sigma }(w_f{h}_{t-1}+w_f{x}_t+b_f)$$

(18)

w_f is weight, h_t−1 is the information of the previous time, and b_f is offset.

Step 2: First, the previous moment h_t−1 and the current state x_t are updated to the information after the sigmoid function process i_t. Then, use the tanh activation function to obtain the update information.

$$\begin{array}{l}{i}_t=\mathsf{\sigma }(w_i{h}_{t-1}+w_i{x}_t+b_i)\\ {\tilde{c}}_t=\tan h(w_i{h}_{t-1}+w_i{x}_t+b_c)\end{array}$$

(19)

w_i is the weight of the input door, b_i is the input door offset, w_c is the weight of entering the new cell, and b_c is offset.

Step 3: Update the old status information by updating the door, c_t−1, and f_t’s Hadamard accumulation, select whether some old cell information is discarded by the forgotten door, and obtain the new cell status c_t.

$$c_t=f_t\otimes c_{t-1}+i_t\otimes {\tilde{c}}_t$$

(20)

Step 4: Combine the hidden layer status h_t−1 and the current state x_t in the previous moment, and the new output value o_t is processed by the sigmoid function.

$$\begin{array}{l}{o}_t=\mathrm{\sigma }(w_o{h}_{t-1}+w_o{x}_t+b_o)\\ h_t=o_t\otimes \tan h(c_t)\end{array}$$

(21)

Due to the complexity of the LSTM module, its computational process is cumbersome, resulting in longer calculation times. To address this issue, Cho et al. proposed GRU (gated recurrent unit) with gated control cycle units.

Figure 10 illustrates the GRU (Gated Recurrent Unit) architecture, which follows the same computational principles as LSTM but reduces the number of LSTM gates, effectively reducing the calculation time. GRU achieves comparable performance to LSTM using fewer parameters. It combines the forget and input gates of LSTM into a single update gate. The update gate output is obtained by applying the sigmoid function to the previous moment, as shown in the following equation.

$$z_t=\mathsf{\sigma }(w_z{x}_t+u_z{h}_{t-1})$$

(22)

“w_z” corresponds to the weight parameter of the update gate, while “u_z” represents the current weight parameter of the update gate.

Before the reset door decides how much information is available, the linear transformation also processes the output activation value through the sigmoid function.

$$\begin{array}{l}{r}_t=\mathsf{\sigma }(w_r{x}_t+u_z{h}_{t-1})\\ {\tilde{h}}_t=\tan h(w_h{x}_t+r_t\otimes u_z{h}_{t-1})\end{array}$$

(23)

“w_r” stands for the recurrent weight matrix of the reset gate, while “w_h” represents the weight matrix for the candidate hidden state. ${\tilde{h}}_t$ is the local information extracted from the maintained door. After the updated door, the updated state h_t is obtained.

$$h_t=z_t\otimes h_{t-1}+(1-z_t)\otimes {\tilde{h}}_t$$

(24)

The utilization of GRU networks is more suitable for addressing problems that require long-term data prediction. However, adjusting the parameter values of the GRU network can be tedious and complex, and manual tuning may not lead to the optimal state. Therefore, an algorithm that optimizes the weight values to improve learning efficiency is necessary. Figure 11 illustrates the process of optimizing the GRU neural network prediction algorithm using the improved marine predator algorithm.

Improved Marine Predator Algorithm (WMPA)

In the ocean, marine animals exhibit similarities with many animals in nature. They adopt a random walking strategy, wherein the next position or status of predators and prey is dependent on their current location or status. It has been observed that the speed of predators and prey in marine creatures determines their respective predation strategies. The transition between the Lévy flight strategy and Brownian motion strategy enables the identification of optimal strategies for optimization purposes.

In the process of updating the GRU network, the dual adaptive switching probability and the adaptive inertia weight are introduced into the MPA algorithm to speed up the convergence speed and solution accuracy of the algorithm, prevent the ocean capture algorithm from falling into local optimum, and improve the performance of global optimization. The specific improvement methods are as follows:

• Improvement methods

• Improvement Method 1: Adaptive switching probability

In this paper, adaptive learning factors are used to solve this problem, and the adaptive switching probability is used to replace the fixed probability P, as shown in Formula (25). The diversity of eclipses and walks improves the local and global search capabilities of the algorithm.

$$P=0.5-\frac{0.1\times (Max\_iter-Iter)}{Max\_iter}$$

(25)

“Max_iter” represents the maximum number of iterations, and “Iter” represents the current iteration. The adaptive probability P undergoes changes in its value during the drawing process, as illustrated in Figure 12.

From Figure 12, it can be observed that there is a correlation between the number of iterations and the dynamics of the adaptive probability. In the early iterations, the algorithm focuses more on global search, which enhances its performance. However, as the number of iterations increases, the algorithm shifts toward conducting more local exploration when it approaches the global optimal solution. This adjustment aims to improve the accuracy of the solution and the overall search efficiency of the algorithm.

• Improvement Method 2: Dual self-adaptive inertia weight

Inspired by various adaptive weight particle swarm algorithms [36,37], this paper enhances the local and global search capabilities of marine predators through the utilization of dual adaptive inertial weights. Specifically, weight W₁ is designed to enhance the overall search capabilities, while weight W₂ is intended to improve local search capabilities. These weights can be mathematically represented by Formula (26):

$$\begin{array}{l}{W}_1={(1-\frac{Iter}{M\mathrm{a}x\_iter})}^{1-\tan (p\otimes (rand-0.5)\times \frac{rand}{M\mathrm{a}x\_iter})};\\ W_2={(2-\frac{2 \times Iter}{M\mathrm{a}x\_iter})}^{1-\tan (p\otimes (rand-0.5)\times \frac{rand}{M\mathrm{a}x\_iter})};\end{array}$$

(26)

The variable “rand” represents a random number within the range [0, 1]. The dual inertial weights W₁ and W₂ have specific ranges: W₁ ranges from 0 to 0.5, and W₂ ranges from 0.5 to 1.

2.

Structure of WMPA algorithm

First, initialize the type:

$$X_0=X_{\min }+rand(X_{\max }-X_{\min })$$

(27)

X_max and X_min in the formula are the maximum and minimum values of variables, and $rand$ is a uniform random vector defined between [0, 1]. Create the “Elite” matrix (28) and “prey” matrix (29):

$$Elie={\left[\begin{array}{cccc}{X}_{1,1}^I& X_{1,2}^I& \dots & X_{1,\mathrm{d}}^I\\ X_{2,1}^I& X_{2,2}^I& \dots & X_{2,d}^I\\ \dots & \dots & \dots & \dots \\ X_{n,1}^I& X_{n,2}^I& \dots & X_{n,d}^I\end{array}\right]}_{\mathrm{n}\times \mathrm{d}}$$

(28)

$$Prey={\left[\begin{array}{cccc}{X}_{1,1}^{}& X_{1,2}^{}& \dots & X_{1,\mathrm{d}}^{}\\ X_{2,1}^{}& X_{2,2}^{}& \dots & X_{2,d}^{}\\ \dots & \dots & \dots & \dots \\ X_{n,1}^{}& X_{n,2}^{}& \dots & X_{n,d}^{}\end{array}\right]}_{\mathrm{n}\times \mathrm{d}}$$

(29)

In the equation above, X represents the vector of the top predator, and after n iterations, the Elie matrix is replicated. “n” denotes the number of populations, “d” represents the dimension of the search space, and “X_i,j” indicates the jth dimension of the ith prey.

The first stage of the improved marine predator algorithm (WMPA) is characterized by the predator being faster than the prey, and it employs an exploration strategy. This stage involves a global search within the solution space. It is particularly suitable for the initial 1/3 iterations of the algorithm, represented by Equation (30).

$$\begin{array}{l}\overrightarrow{{stepsize}_i}=\overrightarrow{R_B}\otimes (\overrightarrow{{Elite}_i}-\overrightarrow{R_B}\otimes \overrightarrow{{\mathrm{Pr}ey}_i})i=1,\dots n\\ \overrightarrow{{\mathrm{Pr}ey}_i}=w_1\otimes \overrightarrow{{\mathrm{Pr}ey}_i}+P.\overrightarrow{R}\otimes \overrightarrow{{stepsize}_i}\end{array}$$

(30)

In this vector, R consists of random numbers generated from a normal distribution based on Brownian motion. P = 0.5 is a constant, and R is a random number ranging from 0 to 1.

The second stage of improved marine predator algorithm (WMPA) occurs between 1/3 and 2/3 of the total number of iterations. This stage is divided into two parts, where half of the population is responsible for exploitation, and the other half is responsible for exploration. When Iter/Max_iter < 0.5, they are represented by Formulas (31) and (32), respectively. When Iter/Max_iter > 0.5, they are represented by Formulas (33) and (34), respectively.

When Iter/Max_iter < 0.5,

$$\begin{array}{l}\overrightarrow{{stepsize}_i}=\overrightarrow{R_L}\otimes (\overrightarrow{{Elite}_i}-\overrightarrow{R_L}\otimes \overrightarrow{{\mathrm{Pr}ey}_i})i=1,\dots n/2\\ \overrightarrow{{\mathrm{Pr}ey}_i}=w_1\otimes \overrightarrow{{\mathrm{Pr}ey}_i}+P.\overrightarrow{R}\otimes \overrightarrow{{stepsize}_i}\end{array}$$

(31)

$$\begin{array}{l}\overrightarrow{{stepsize}_i}=\overrightarrow{R_B}\otimes (R_B\otimes \overrightarrow{{Elite}_i}-\overrightarrow{{\mathrm{Pr}ey}_i})i=n/2,\dots ,n\\ \overrightarrow{{\mathrm{Pr}ey}_i}=w_1\otimes \overrightarrow{{E\mathrm{lite}}_i}+P.CF\otimes \overrightarrow{{stepsize}_i}\end{array}$$

(32)

When Iter/Max_iter > 0.5,

$$\begin{array}{l}\overrightarrow{{stepsize}_i}=\overrightarrow{R_L}\otimes (\overrightarrow{{Elite}_i}-\overrightarrow{R_L}\otimes \overrightarrow{{\mathrm{Pr}ey}_i})i=1,\dots n/2\\ \overrightarrow{{\mathrm{Pr}ey}_i}=\overrightarrow{{\mathrm{Pr}ey}_i}+w_2\otimes P.\overrightarrow{R}\otimes \overrightarrow{{stepsize}_i}\end{array}$$

(33)

$$\begin{array}{l}\overrightarrow{{stepsize}_i}=\overrightarrow{R_B}\otimes (R_B\otimes \overrightarrow{{Elite}_i}-\overrightarrow{{\mathrm{Pr}ey}_i})i=n/2,\dots ,n\\ \overrightarrow{{\mathrm{Pr}ey}_i}=\overrightarrow{{E\mathrm{lite}}_i}+w_2\otimes P.CF\otimes \overrightarrow{{stepsize}_i}\end{array}$$

(34)

In this stage, R_L represents the random number vector distributed by Lévy motion, and $CF={(1-\frac{Iter}{Max\_iter})}^{(2\frac{Iter}{Max\_iter})}$ is the adaptive parameter that controls the movement of predators.

The third stage of the improved marine predator algorithm (WMPA) focuses on exploitation. It aims to find the optimal solution within the solution space by utilizing the Lévy flight strategy of the predators. The specific process is described by Formula (35):

$$\begin{array}{l}\overrightarrow{{stepsize}_i}=\overrightarrow{R_L}\otimes (\overrightarrow{R_L}\otimes \overrightarrow{{Elite}_i}-\overrightarrow{{\mathrm{Pr}ey}_i})i=1,\dots ,n\\ \overrightarrow{{\mathrm{Pr}ey}_i}=\overrightarrow{{E\mathrm{lite}}_i}+w_2\otimes P.CF\otimes \overrightarrow{{stepsize}_i}\end{array}$$

(35)

The vortex formation and fish aggregating device (FADS) effect play a crucial role in helping the algorithm escape from local optima. It is represented by Formula (36):

$$\overrightarrow{{\mathrm{Pr}ey}_i}=\left\{\begin{array}{ll}\overrightarrow{{\mathrm{Pr}ey}_i}+CF\left[{\overrightarrow{X}}_{\min }+\overrightarrow{R}\otimes ({\overrightarrow{X}}_{\max }-{\overrightarrow{X}}_{\min })\right]\otimes \overrightarrow{U}& ifr\le {FAD}_s\hfill \\ \overrightarrow{{\mathrm{Pr}ey}_i}+\left[{FAD}_s(1-r)+r\right](\overrightarrow{{\mathrm{Pr}ey}_{r1}}-\overrightarrow{{\mathrm{Pr}ey}_{r2}})& ifr>{FAD}_s\hfill \end{array}\right.$$

(36)

In the formula, $\overrightarrow{U}$ is a binary vector with elements of 0 and 1. r is a uniformly distributed random number in the range of [0, 1], and the reduction in r₁ and r₂ represents random indices in the prey matrix.

3.

The implementation step of the WMPA algorithm

The steps of the WMPA algorithm are presented in Algorithm 1 as a pseudo code.

Algorithm 1: The pseudo code of the WMPA algorithm

1.Initialize search agent (prey) population group i = 1, …, n

2.While In the case of not meeting the termination conditions, calculate the adaptation,

construct the elite matrix, and realize memory saving

3. If Iter < Max_Iter/3

4.  Update the position of the current search agent through equivalent (30)

5. else if Iter > Max_iter/3 && Iter < 2*Max_iter/3

6.    For the first half of the group (i = 1, …, n/2)

7.   if (Iter/Max_iter > 0.5)

8.      Update the position of the current search agent through equivalent (33)

9.   else Update the position of the current search agent through equivalent (31)

10.    end

11.   For the other half of the group (i = n/2, …, n)

12.   if (Iter/Max_iter > 0.5)

13.      Update the position of the current search agent through equivalent (34)

14.   else Update the position of the current search agent through equivalent (32)

15.   end

16.    else

17.    Update the position of the current search agent through equivalent (35)

18. end(if)

19. Use the FADS effect to complete memory saving and elite updates, and update

according to the Formula (36)

20. end(while)

4. Result Analysis

4.1. Datasets

The experimental dataset in this section comprises two parts, totaling 500 samples. The first part of the dataset is derived from the Dlib library, which is used to extract facial features. Fatigue driving state videos from the YawDD dataset are fed into Dlib to extract the driver’s eye, mouth, head pitch angle, and roll angle. These features are then converted into vectors, and Python code is employed to calculate and compile the data statistics. Consequently, a new dataset consisting of 400 samples is obtained. The second part of the dataset is a custom-made collection obtained by recording videos of simulated drivers exhibiting both fatigue and non-fatigue states using a camera. These video streams are subsequently processed using the Dlib library, resulting in 100 samples.

4.2. Situation Evaluation Experiment

A fuzzy evaluation model is constructed, and the corresponding quantification is conducted using a 1 to 5 rating scale. The specific quantification criteria are presented in

Table 7. Risk Decision Standard.

Grade	Attributes	Judgement Standard
Level I	Low risk	High degree of safety
Level II	Lower risk	High security
Level III	General risk	The degree of security is average
Level IV	Higher risk	The degree of safety is less secure
Level V	Extremely high risk	The degree of safety is not safe

below.

Fifteen sets of data obtained by Dlib face recognition for 5 s are randomly selected for initial data for situation evaluation, and two decimal places are retained, as shown in

Table 8. Original data table.

Groups	MAR	EAR	|Pitch|	|Roll|
1	0.53	0.31	3.26	5.22
2	0.42	0.10	10.01	3.32
3	0.35	0.32	15.23	5.54
4	0.21	0.14	21.36	6.65
5	0.33	0.34	25.14	8.41
6	0.42	0.40	30.41	6.21
7	0.54	0.33	33.50	9.32
8	0.79	0.62	35.21	10.10
9	0.86	0.54	10.00	6.24
10	0.74	0.52	5.35	5.36
11	0.62	0.41	2.32	3.85
12	0.41	0.70	1.14	5.37
13	0.33	0.66	3.36	7.21
14	0.29	0.35	5.45	3.46
15	0.41	0.14	4.33	5.17

.

In this paper, Table 2 presents the principles of the comprehensive judgment of fatigue status, which defines the thresholds for determining fatigue for each indicator. Table 3, on the other hand, represents the fatigue status level judgment. It adopts a method that primarily focuses on facial features with supplementary consideration of head pose, utilizing the Dlib library along with the MAR, EAR, and HPE fatigue detection algorithms to establish the fatigue state thresholds for drivers. By combining these thresholds, the driver’s fatigue level is determined. In light of this, by integrating the explanations provided in Table 2 and Table 3, the paper derives the Membership Table Standard, which serves as the fuzzy rules for the AHP-fuzzy comprehensive evaluation. Therefore, the resulting Affiliation Table Standard is presented in

Table 9. Standard table of affiliation.

Grade	MAR	EAR	|Pitch|	|Roll|
Level I	[0, 0.6)	(0.2, 1]	[0, 20)	[0, 15.4)
	[0, 0.6)	(0.2, 1]	[20, 70]	[0, 15.4)
	[0, 0.6)	(0.2, 1]	[0, 20)	[15.4, 36]
Level II	[0.6, 1]	(0.2, 1]	[0, 20)	[0, 15.4)
	[0, 0.6)	[0, 0.2]	[0, 20)	[0, 15.4)
	[0, 0.6)	(0.2, 1]	[20, 70]	[15.4, 36]
Level III	[0.6, 1]	(0.2, 1]	[20, 70]	[0, 15.4)
	[0.6, 1]	(0.2, 1]	[0, 20)	[15.4, 36]
	[0, 0.6)	[0, 0.2]	[20, 70]	[0, 15.4)
	[0, 0.6)	[0, 0.2]	[0, 20)	[15.4, 36]
	[0.6, 1]	[0, 0.2]	[0, 20)	[0, 15.4)
Level IV	[0.6, 1]	[0, 0.2]	[0, 20)	[15.4, 36]
	[0.6, 1]	[0, 0.2]	[20, 70]	[0, 15.4)
	[0.6, 1]	(0.2, 1]	[20, 70]	[15.4, 36]
	[0, 0.6)	[0, 0.2]	[20, 70]	[15.4, 36]
Level V	[0.6, 1]	[0, 0.2]	[20, 70]	[15.4, 36]

, as follows.

According to the different value standards of the membership degree, this paper utilizes a frequency-based statistical approach to determine the membership degree. The summarized membership degree matrix,

Table 10. Statistics of frequency of affiliation.

Grade	MAR	EAR	|Pitch|	|Roll|
Level I	33	36	25	30
Level II	26	27	25	30
Level III	40	41	36	31
Level IV	23	21	25	15
Level V	4	3	5	0

, is obtained. Specifically, based on the fuzzy rules established in Table 9, the original evaluation data from Table 8 are analyzed by counting their frequency of occurrence within different membership degree value ranges. The resulting summarized membership degree matrix, Table 10, reflects the higher membership degree for the original evaluation data that appears more frequently within the defined value ranges.

By performing non-dimensional standardization on the data in the table, we can obtain the non-dimensional standardized matrix of membership degrees:

$$\left[\begin{array}{llll}0.8056\hfill & 0.8684\hfill & 0.6452\hfill & 0.9677\hfill \\ 0.6111\hfill & 0.6316\hfill & 0.6452\hfill & 0.9677\hfill \\ 1\hfill & 1\hfill & 1\hfill & 1\hfill \\ 0.5278\hfill & 0.4737\hfill & 0.6452\hfill & 0.4839\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill \end{array}\right]$$

After performing non-dimensional standardization on the data, the next step is to normalize the processed data. The resulting normalized matrix, after transposition, can be denoted as R.

$$\mathrm{R}=\left[\begin{array}{ccccc}\hfill 0.2736& \hfill 0.2075& \hfill 0.3396& \hfill 0.1792& \hfill 0\\ \hfill 0.2920& \hfill 0.2124& \hfill 0.3363& \hfill 0.1593& \hfill 0\\ \hfill 0.2198& \hfill 0.2198& \hfill 0.3407& \hfill 0.2198& \hfill 0\\ \hfill 0.2830& \hfill 0.2830& \hfill 0.2925& \hfill 0.1415& \hfill 0\end{array}\right]$$

Based on the principles of fuzzy evaluation, we can calculate the comprehensive evaluation value.

According to the principles of the analytic hierarchy process (AHP), the weights calculated are denoted as W = [0.2861 0.5315 0.1060 0.0763]^T. Based on the method’s principles, we can further calculate the following:

$$B=\left[\begin{array}{ccccc}0.2736& 0.2075& 0.3396& 0.1792& 0\\ 0.2920& 0.2124& 0.3363& 0.1593& 0\\ 0.2198& 0.2198& 0.3407& 0.2198& 0\\ 0.2830& 0.2830& 0.2925& 0.1415& 0\end{array}\right]\times \left[\begin{array}{c}0.2861\\ 0.5315\\ 0.1060\\ 0.0763\end{array}\right]=\left[\begin{array}{ccccc}0.2784& 0.2172& 0.3343& 0.1700& 0\end{array}\right]$$

After the normalization process, the membership degree evaluation values for the “fatigue state level” of the driver are as follows: B = [0.2784 0.2172 0.3343 0.1700 0]. Based on this, we can obtain the final membership degree evaluation table, denoted as

Table 11. Comprehensive evaluation form of affiliation.

Driver Fatigue State Level	Level I	Level II	Level III	Level IV	Level V
Overall evaluation situation	0.2784	0.2172	0.3343	0.1700	0

.

According to the principle of maximum affiliation, the result of this fuzzy comprehensive evaluation is Level III, indicating a moderate level of risk.

4.3. Situation Prediction Experiment

Based on the evaluation of the driver’s condition, the goal is to predict the driving situation. The fundamental principle of situation prediction is to utilize a neural network approach, which combines the driver’s historical fatigue levels during fatigue driving and the current driver’s state, to forecast future situation levels. This process primarily involves two parts: generating sample data and configuring the parameters of the GRU network.

4.3.1. Sample Settings

Based on the acquisition of human facial feature data using Dlib, special signs are recorded every 5 s, with approximately 15–20 sets of data per cycle. A total of 15 groups of samples are obtained. The situation evaluation method is used to update the intention and trend levels of each sample’s target sequence, denoted as $s_i=[s_1^i,s_2^i\cdots s_t^i]$. s₁ represents the intention level of the target at every 5 s interval, and the intention and trend levels are shown in the table. Since fatigue increases with time during fatigue driving, the evaluation of data is obtained every 10 s to record the driver’s fatigue state over time when the driver is fatigued. This allows for the prediction of fatigue features over time when the driver exhibits fatigue features. Thus, it effectively verifies the long-term dependence of “attitude” on “potential”.

4.3.2. GRU Parameter Settings

A total of 400 targets are selected as the training set, while the remaining 100 targets serve as the test set. The input data for the training set are denoted as $s_{input}=[s_1^i,s_2^i\cdots s_t^i]$, and the output data are denoted as $s_{output}^i=[s_5^i]$. The number of neurons in the input layer is set to 4, and the number of neurons in the output layer is set to 1. The maximum number of training epochs is set to 500.

After optimizing the parameters using the improved marine predator algorithm, the initial population consists of 30 groups. The maximum number of iterations is set to 1000, and the dimensionality (Dim) is set to 3. The lower boundary (LB) parameters are set as [1 × 10⁻³, 10, 1 × 10⁻⁴], while the upper boundary (UB) parameters are set as [1 × 10⁻², 30, 1 × 10⁻¹]. Finally, the target prediction levels are obtained.

4.3.3. Simulation Results

• The improved GRU network training result graph is shown in Figure 13 below.

2.

Comparison chart of the prediction results of each algorithm

Figure 14 and Figure 15 depict the confusion matrices and prediction result comparisons for the improved GRU algorithm: (a), GRU (b), LSTM (c), and BP [38,39] algorithm (d).

From the comparison of confusion matrices, it can be observed that our proposed algorithm (improved GRU algorithm) demonstrates superior performance in the classification task compared to other algorithms, particularly for the collected samples of fatigue levels in the test set over a period of time. The algorithm exhibits higher classification accuracy across all categories. This is evident from the confusion matrices of the four algorithms, where the diagonal elements of our proposed algorithm are relatively larger, indicating a higher correct classification rate.

Similarly, in terms of the comparison of classification prediction results, over a certain period of time, for the collected test set samples of fatigue levels, it can be clearly observed that the algorithm used in this paper (improved GRU algorithm) exhibits superior performance compared to other algorithms in the classification task. The classification prediction result comparison plot demonstrates the improved algorithm’s clearer decision boundaries and more accurate classification boundaries.

Through the analysis of Figure 14 and Figure 15, specifically, the comparison plots reveal the following advantages of the proposed algorithm in terms of classification prediction: Firstly, the algorithm can better differentiate between different categories, leading to tighter clustering of samples within the same category while effectively separating samples from different categories. Secondly, compared to the GRU and LSTM, the proposed algorithm reduces the occurrence of misclassifications during the classification prediction process and more accurately assigns samples to the correct categories. Additionally, in comparison to the traditional BP algorithm, the proposed algorithm demonstrates higher accuracy and robustness in classification prediction.

In addition, the experimental data utilized in this paper exhibit an uneven distribution. The sample data comprise five levels, with levels 1 and 5 having fewer instances and levels 2, 3, and 4 having more instances. The BP, LSTM, and GRU algorithms were compared without any optimization. Based on cost and accuracy evaluation, the algorithm proposed in this paper is suboptimal in terms of calculation time due to the lengthy computation time required by the improved marine predator algorithm. Among the four neural networks, the algorithm and GRU algorithm proposed in this paper employ a combination of three-gate structures for target prediction, while the LSTM algorithm utilizes a single-gate structure, and the BP algorithm adopts a four-layer network structure to achieve the objective. Predictions were randomly selected from these four algorithms and applied to 100 test samples. Figure 14 and Figure 15 demonstrate the enhanced algorithm and the processing effects of the GRU, LSTM, and BP algorithms from left to right. Through a comparison of the confusion matrix and prediction results, it is evident that the GRU algorithm outperforms the BP and LSTM algorithms in terms of classification accuracy. Furthermore, it is observed that the proposed improved GRU algorithm achieves significantly enhanced accuracy (92%) compared to the non-optimized GRU algorithm. The statistical data confirm its ability to attain more precise prediction accuracy, thereby fulfilling the requirements of fatigue driving detection.

5. Conclusions

In conclusion, although this paper proposes a research method combining multi-feature fusion and situational awareness for determining fatigue driving levels, there is still a certain imbalance in the data indicators affecting fatigue state obtained through the Dlib library, which can have an impact on the experimental results. Therefore, this paper performs a fusion analysis of fatigue state indicators and further categorizes the driver’s fatigue level. By employing fusion analysis techniques, the impact of imbalanced data on model accuracy is reduced.

The proposed fatigue driving detection method, which incorporates multi-feature fusion and trend perception, enables the evaluation of the driver’s fatigue state and provides early warning for predicting future situation levels based on past and present states. The AHP-fuzzy evaluation model constructed in this paper effectively integrates the evaluation indicators that influence driver fatigue state and enables comprehensive evaluation over a specific time period or stage. The fatigue level division standard is utilized in the situation evaluation experiment to conduct frequency statistics and generate a summary table of the affiliation matrix. The AHP-fuzzy comprehensive evaluation model is then implemented to obtain evaluation results. Experimental results demonstrate the feasibility of the model and evaluation method.

Furthermore, a GRU-based situation prediction method, optimized by an improved marine predator algorithm, is employed to predict the driver’s fatigue status level. Through simulation experiments, it is observed that although the improved marine predator algorithm achieves satisfactory results in optimizing the parameters of the GRU network and enhancing prediction accuracy in the situation prediction session, reaching 92%, there is still room for further improvement. This is primarily due to the computational time required for calculating the global optimal solution in the improved marine predator algorithm.

In this paper, the Dlib library was utilized for driver fatigue state analysis and fatigue driving level determination. However, the accuracy and performance of the algorithms provided by the Dlib library are crucial in safety-critical applications. Therefore, comprehensive evaluation and validation are necessary to ensure the accuracy and reliability of using the Dlib library for feature extraction in analyzing the driver’s fatigue state and fatigue driving level, meeting the required safety standards and regulations. Certification and compliance requirements also need to be considered, including ISO standards, ASPICE certification in the automotive industry, and road traffic safety regulations. It is essential to ensure that the Dlib library complies with these requirements and can pass the corresponding validation and testing processes. Furthermore, robustness and reliability in different scenarios, such as varying lighting conditions, driver poses, and facial expressions, should be considered for fatigue detection using the Dlib library. Thorough testing and validation are necessary to ensure the stability and robustness of the algorithms.

To summarize, this paper employed the Dlib library for feature extraction and proposed a method for fatigue detection and the determination of fatigue driving level using multi-feature fusion and situational awareness. However, it is important to acknowledge the challenges in using the Dlib library in safety-critical applications. Thorough evaluation, validation, and testing are essential to ensure compliance with safety standards and requirements. Collaboration with domain experts, conducting field tests and validations, and potential further research and development are crucial for ensuring the safety and reliability of the chosen solution.

In conclusion, this paper focuses on driver fatigue level classification and proposes an innovative method using the Dlib library and fatigue detection algorithms. The method incorporates multi-feature fusion and analytical hierarchy process–fuzzy comprehensive evaluation for fatigue level assessment. By optimizing the GRU network with an improved marine predator algorithm, the method achieves significant improvements in fatigue level prediction. Experimental results demonstrate high accuracy and real-time performance across various scenarios.

Accurately assessing driver fatigue levels is vital for enhancing road safety and reducing accidents. This paper provides an innovative method that comprehensively evaluates driver fatigue levels, offering a scientific basis for implementing appropriate measures. The research has implications for driver health and safety, as well as relevance to traffic management and policy making.

It is important to address limitations in the research, such as evaluating the credibility and security of driver facial recognition using the Dlib library and ensuring network security during data transmission. Further comparative analyses with other studies and leveraging deep learning techniques for facial and head feature analysis can enhance the assessment of driver fatigue levels. Additionally, incorporating sensor data, such as heart rate and electroencephalogram, can provide more precise determinations of driver fatigue levels.