A Multi-Feature Fusion Algorithm for Fatigue Driving Detection Considering Individual Driver Differences

Abstract

Fatigue driving is one of the crucial factors causing traffic accidents. Most existing fatigue driving detection algorithms overlook individual driver characteristics, potentially leading to misjudgments. This article presents a novel detection algorithm that utilizes facial multi-feature fusion, thoroughly considering the driver’s individual characteristics. To improve the judging accuracy of the driver’s facial expressions, a personalized threshold is proposed based on the normalization of the driver’s eyes and mouth opening and closing instead of the traditional average threshold, as individual drivers have different eye and mouth sizes. Given the dynamic changes in fatigue level, a sliding window model is designed for further calculating blinking duration ratio (BF), yawning frequency (YF), and nodding frequency (NF), and these evaluation indexes are used in the feature fusion model. The reliability of the algorithm is verified by the actual test results, which show that the detection accuracy reaches 95.6% and shows good application potential in fatigue detection applications. In this way, facial multi-feature fusion and fully considering the driver’s individual characteristics makes fatigue driving detection more accurate.

1. Introduction

Traffic safety remains a paramount concern in our society [1]. Factors such as drunk driving [2], speeding, and fatigue driving [3] not only endanger the lives of drivers but also pose a threat to public transportation safety. Unlike drunk driving and speeding, which are regulated by laws and policies, issues related to fatigue driving can be mitigated through instantaneous detection and warning technologies [4].

Fatigue driving impairs driving stability and decision-making ability [5], representing a significant factor contributing to traffic accidents. According to previous studies, fatigue driving is linked to approximately 16.5% of fatal crashes and 12.5% of injury-related collisions in the United States. Globally, it may account for up to 20% of all traffic accidents [6]. This article aims to mitigate traffic accidents attributable to fatigue driving by proposing a fatigue driving detection algorithm that takes into account individual driver characteristics.

Currently, the detection of fatigue driving is mainly divided into subjective and objective methods [7]. The subjective method relies on questionnaires such as the Stanford ‘Sleepiness Scale (SSS) [8], Visual Analog Scale (VAS) [9], and Karolinska Sleepiness Scale (KSS) [10], which depend on drivers’ subjective perceptions. Due to its subjective nature, this method cannot serve as a standardized approach for detecting fatigue driving. Objective methods involve utilizing auxiliary tools to detect the driver’s physiological characteristics [11,12], vehicle information [13,14], or facial features [15,16] to ascertain fatigue driving. Physiological feature detection using wearable sensors is not only employed in medical contexts [17], but also extensively utilized in fatigue driving detection. While wearable sensors are undeniably becoming smaller, lighter, and more accurate [18,19,20], their costliness and potential discomfort during prolonged use may hinder the widespread adoption of fatigue detection based on physiological characteristics [21,22]. For instance, electroencephalography (EEG) headsets are commonly priced between USD 1000 and 25,000, presenting a substantial economic barrier for widespread deployment. Moreover, user experience studies have reported that extended use of ECG chest straps can cause skin irritation and physical discomfort, which may reduce driver compliance and affect the overall effectiveness of fatigue detection systems [23]. Techniques that indirectly evaluate driver fatigue levels by monitoring alterations in vehicle speed and steering entail diverse data collection and analysis procedures, thereby markedly escalating the complexity of the detection system.

Fatigue detection based on facial features typically involves non-contact methods that minimize interference with the driver and directly reflect the driver’s fatigue state, becoming the mainstream direction of related research [24]. This method usually considers feature variations from the eyes, mouth, and head pose. Li [15] proposed a method utilizing deep learning techniques to analyze facial features, achieving high accuracy in detecting driver fatigue. However, challenges arise when facial features are obscured, such as when drivers wear masks or cover their mouths while yawning or coughing. Qu, S. et al. [25] developed a multi-attention fusion model that improves fatigue detection performance by enhancing feature extraction across multiple facial regions. However, this approach still faces limitations in detecting fatigue when significant facial occlusions occur, such as in poor lighting conditions or when the driver’s head is tilted at extreme angles.

Detection based on human eye features provides the most intuitive characterization of fatigue. Ramzan, M. et al. [1] reported a method for detecting fatigued driving by eye tracking and dynamic template matching using the Hue-Saturation-Intensity (HSI) color model and the Sobel edge operator with an accuracy of up to 88.9%. Percentage of eyelid closure versus pupil change over time (PERCLOS) is one of the most popular algorithms [26], which is accurate but takes a relatively long time to compute fatigue results [27]. A method to calculate blinking frequency (BF) by analyzing specific image frames was proposed in [28]. Although the method gives appropriate criteria for determining the level of fatigue, the number of blinks in a given period of time needs to be counted prior to the computation, which prolongs the response time. As reported in [29,30,31], eye aspect ratio (EAR) calculation is simple to perform and has a fast response; however, a single detection parameter is difficult to reflect an accurate and reliable fatigue level.

In practical applications, fatigue detection must be accurate, reliable, and responsive. However, existing methods often rely on single or limited behavioral features, or apply uniform thresholds across all subjects, which limits their adaptability and leads to potential misjudgments. To address these limitations, this article proposes a fatigue monitoring algorithm based on facial multi-feature fusion, which combines three evaluation indexes: blinking frequency (BF), yawning frequency (YF), and nodding frequency (NF). Additionally, considering individual driver characteristics, a personalized threshold is introduced to assess the driver’s eye, mouth, and head status by normalizing the degree of eye and mouth openness, rather than using a traditional average threshold. Compared to previous approaches, the proposed method provides improved adaptability to individual differences and more comprehensive behavioral representation, thereby enhancing overall detection robustness. It demonstrates significant applicability in a wide range of fatigue detection scenarios.

The main innovations of this article include the following:

• The introduction of a personalized threshold model based on the normalization of eye and mouth openness, which accounts for individual differences in facial features.
• The use of a sliding window model to dynamically track changes in fatigue indicators (BF, YF, and NF) in real time, enhancing detection accuracy.
• The development of a multi-feature fusion approach that integrates multiple fatigue-related behaviors (blinking, yawning, and nodding) to improve overall system reliability and response time.

2. Algorithm and Methodology

2.1. Model Description

The proposed fatigue detection model utilizes a video stream to capture details of the driver’s facial contours. The algorithm operates within the framework depicted in Figure 1. Initially, the video is captured at a frame rate of 30 frames per second (fps) and segmented into a series of image frames with a resolution of 640 × 480 pixels. To reduce computational complexity, each frame is uniformly resized to 320 × 240 pixels. The frames are then converted to grayscale using standard color space transformation techniques, followed by histogram equalization to enhance image contrast. A Gaussian blur with a kernel size of 5 × 5 is subsequently applied to suppress high-frequency noise. Finally, all pixel values are normalized to the range [0, 1] to ensure consistency in illumination across frames. These preprocessing operations improve the robustness of the system by mitigating the effects of lighting variability and image noise, thereby enabling more reliable feature extraction and fatigue detection [32].The illustrations in Figure 1 depict the difference between the images before and after processing. Subsequently, the key facial points are extracted from the preprocessed image. The Dlib toolkit [33] is employed to extract the feature points of the driver’s eyes and mouth. First, the head Euler angle α (HEAα) [34] values are calculated from the head features. Then, to distinguish the facial features of different drivers, the eye and mouth contour curves are fitted using the eye and mouth feature points, and the normalized evaluation indexes (${\eta }_{NAE}$,${\eta }_{NAM}$) for the eyes and mouth are calculated from the open and closed normalization model [30]. Based on the obtained values of ${\eta }_{NAE}$ and${\eta }_{NAM}$, personalized thresholds can be assigned to judge the states of the eyes and mouth for each driver. Subsequently, these values are input into the sliding window model t to further calculate the three evaluation indexes: blinking frequency (BF), yawning frequency (YF), and nodding frequency (NF) [1]. Finally, the obtained three evaluation indexes are input into the feature fusion model to derive the integrated features and assess the degree of fatigue.

2.2. Evaluation Indexes

In this study, Dlib was used to extract 68 facial key points related to the eyes, mouth, and facial contours. These landmarks are illustrated in Figure 2a, and their actual detection results on video frames are shown in Figure 2b. The extracted feature points serve as the basis for calculating fatigue-related indicators and subsequent facial behavior analysis.

Vertical head movement, particularly downward nodding, is widely recognized as a key behavioral indicator of driver fatigue. As illustrated in Figure 3, head pose is described by three rotational angles: roll, pitch, and yaw. Among these, the pitch angle is most directly associated with vertical head motion and was therefore adopted as the primary evaluation metric for fatigue detection in this study.

Head pose estimation is performed based on facial landmark detection using the Dlib toolkit, which extracts 2D image feature points corresponding to predefined 3D facial model coordinates. Combined with the intrinsic parameters of the camera, the Perspective-n-Point (PnP) [36] algorithm is employed to solve for the rotation vector. This vector is then converted into a rotation matrix via the Rodrigues transformation [37]. The pitch angle, denoted as HEAα (head Euler angle α), is calculated from the elements of the rotation matrix using the following formula:

$$\mathrm{HEA}\alpha =\arctan 2(-R_{31},\sqrt{R_{32}^2+R_{33}^2})$$

(1)

where $R_{31}$,$R_{32}$,$R_{33}$ are the elements in the third row of the rotation matrix [37]. This angle effectively captures vertical head movements, which are highly indicative of drowsiness-related behavior. The function arctan2(v,u) used here denotes the two-argument inverse tangent function, which calculates the angle between the point (v,u) and the horizontal axis. It returns values in the range (−π,π]. Unlike the traditional arctan(u/v), the arctan2 function takes into account the sign of both arguments to determine the correct quadrant of the angle, thereby avoiding division-by-zero errors.

According to the biomechanical literature [38], the natural pitch angle range of the human head spans approximately from −50° to 45°. In this study, abnormal nodding is defined as any instance where HEAα falls within the lowest 30% of this range, specifically between −50° and −35°. When HEAα enters this range and meets certain temporal continuity or frequency conditions, it is identified as an abnormal nodding event and used as an indicator of driver fatigue.

To model the opening and closing states of the eyes and mouth, parabolic fitting is applied to their contour curves. This method captures the essential vertical shape variation using limited facial landmarks and ensures a good trade-off between fitting accuracy and computational efficiency. It is well-suited for real-time fatigue detection tasks.

Specifically, the right eye contour curve is segmented into upper and lower parts by nodes 43 and 46, while the left eye contour curve is segmented into upper and lower parts by nodes 37 and 40. Similarly, the mouth contour curve is divided into upper and lower parts by nodes 49 and 55. The upper part of the contour curve is represented by $F_1\left(x\right)$ in Equation (2), while the lower part is represented by $F_2\left(x\right)$ in Equation (2).

$$\left\{\begin{array}{c}{F}_1(x)=a_1{x}^2+b_{1}x+c_1,\\ \hspace{0.33em}{F}_2(x)=a_2{x}^2+b_{2}x+c_2,\\ a_1<0,a_2>0\end{array}.\right.$$

(2)

The data of the upper and lower parts of the nodes of the left eye, the right eye, and the mouth are inputted into $F_1\left(x\right)$ and $F_2\left(x\right)$, respectively. Then, the sum of the error squares is calculated, and the function minimizing this sum is determined as the contour fitting curve. Curve 1 in Figure 4b illustrates the contour fitting curve for the eyes in the open state, while Curve 1 in Figure 4d depicts the contour fitting curve for the mouth in the closed state.

To normalize the openness of the eyes and mouth, the area enclosed by the fitted contours and the area of their circumscribed circle are calculated. The process for constructing the circumscribed circle and calculating the enclosed area is as follows:

(1)

Determining the center of the circumscribed circle:

The x-coordinates of the vertices of the fitted parabolas, $x_1$ = −$b_1$/2$a_1$ and $x_2$ = −$b_2$/2$a_2$, are used to compute the center C = ($x_c$,$y_c$) of the circle:

$$x_c={\displaystyle \frac{x_1+x_2}{2}},y_c={\displaystyle \frac{y_{\mathrm{upper}}^{\max }+y_{\mathrm{lower}}^{\min }}{2}}$$

(3)

(2)

Radius of the circumscribed circle:

The radius R is determined as the maximum Euclidean distance from the center to the sampled points along both parabolas:

$$R=\underset{i}{\max }\sqrt{{(x_i-x_c)}^2+{(y_i-y_c)}^2}$$

(4)

(3)

Area of the circumscribed circle:

The area of the circumscribed circle is given by:

$$A_2=\pi R^2$$

(5)

(4)

Fitted area between the contours:

The area between the upper and lower parabolic contours is computed using the definite integral over the fitting interval:

$$A_1={\displaystyle {\int }_{x_{min}}^{x_{max}}[y_1(x)-y_2(x)]dx}$$

(6)

(5)

Normalized aperture metric:

The final normalized aperture metric is defined as:

$${\eta }_{\mathrm{NA}}={\displaystyle \frac{A_1}{A_2}}$$

(7)

The degree of eye and mouth opening and closing is calculated by Equation (7) and is used as a normalized index for the calibration of opening size. $A_1$ represents the area enclosed by the fitted contour curve of the eyes and mouth, whereas $A_2$ represents the area of their circumscribed circle. In Figure 4b, Curve 2 represents the circumscribed circle of the eye’s feature points in the open state. In Figure 4d, Curve 2 represents the circumscribed circle of the mouth’s feature points in the closed state.

The normalized aperture indicator ${\eta }_{NAE}$ for the eyes is shown in Section 3.1. The normalized aperture indicator ${\eta }_{NAM}$ for the mouth is shown in Section 3.1.

2.3. Personalized Thresholds

Considering the individual differences in eye and mouth sizes among drivers, using uniform average thresholds may reduce the accuracy of facial state detection. To enhance the adaptability and robustness of the system, a personalized threshold database was constructed based on ${\eta }_{NAE}$ (Normalized Eye Area) and ${\eta }_{NAM}$ (Normalized Mouth Area). This database adjusts threshold values according to the driver’s facial features under normal conditions (i.e., eyes open and mouth closed), accommodating variations across different individuals. To improve the reliability of the threshold calibration, initial video frames are selected from the beginning of the driving task, when participants are typically in a rested and alert state. Additionally, subjective questionnaires and manual screening are employed to ensure that the selected frames reflect the driver’s baseline (non-fatigued) condition.

The ${\eta }_{NAE}$ and ${\eta }_{NAM}$ ranges used for threshold classification were derived from the publicly available WIDER FACE dataset. More than 600 representative facial images were selected from this dataset, covering a wide range of diversity in gender, ethnicity, facial structure, and the presence of accessories such as eyewear, facial hair, and masks. As WIDER FACE is a widely used benchmark in facial analysis with extensive demographic variability, the derived thresholds offer better generalizability and robustness across heterogeneous driver populations.

Method for Determining Personalized Thresholds:

1. Initial Data Collection:

• During the first interaction with the system, the initial data are collected by selecting the video frames where the eyes are open and the mouth is closed.
• The${\eta }_{NAE}$and${\eta }_{NAM}$values are calculated from these frames, and the sizes of the driver’s eyes and mouth are assessed to determine whether they fall within the normal range.

As shown in

Table 1. Different sizes of eyes and mouths corresponding to different ${\eta }_{NAE}$ and ${\eta }_{NAM}$ values.

$\mathbf{Range}{\mathit{\eta }}_{\mathit{N}\mathit{A}\mathit{E}}$	Eyes Size	$\mathbf{Range}{\mathit{\eta }}_{\mathit{N}\mathit{A}\mathit{M}}$	Mouth Size
0.15–0.24	undersize	0.10–0.14	undersize
0.25–0.34	normal	0.15–0.24	normal
0.35–0.40	oversize	0.25–0.40	oversize

, the driver’s eye and mouth sizes are evaluated using standardized indexes and categorized based on different facial feature ranges.

2. Adaptive Threshold Adjustment:

The personalized thresholds for each driver are set based on their individual eye and mouth sizes, determined from the initial measurements of ${\eta }_{NAE}$and${\eta }_{NAM}$. For example, if the driver’s eye and mouth sizes fall into the “Normal eyes, Undersize mouth” category, the system sets the corresponding ${\eta }_{NAE}$ and ${\eta }_{NAM}$ thresholds. Other categories are similarly adjusted according to the driver’s individual characteristics. The threshold divisions for eye sizes and mouth sizes are listed in

Table 2. Information base of personalized${\eta }_{NAE}$thresholds.

No.	Eyes Size	${\mathit{\eta }}_{\mathit{N}\mathit{A}\mathit{E}}$ Threshold
1	undersize	0.20
2	normal	0.25
3	oversize	0.20

and

Table 3. Information base of personalized${\eta }_{NAM}$thresholds.

No.	Mouth Size	${\mathit{\eta }}_{\mathit{N}\mathit{A}\mathit{M}}$ Threshold
1	undersize	0.30
2	normal	0.40
3	oversize	0.50

, respectively.

2.4. Sliding Window Model

Driver fatigue is a dynamic process that requires assessing the driver’s state over time. To capture this, a sliding window model is employed, as illustrated in Figure 5. In this model, it is as follows:

• Eye state is labeled as 0 (closed) when ${\eta }_{NAE}$ is below the personalized threshold, and 1 (open) otherwise.
• Mouth state is labeled as 0 (closed) when ${\eta }_{NAM}$ is below the threshold, and 1 (open) otherwise.
• Nodding state is labeled as 0 (normal) when HEAα is below its threshold, and 1 (nodding) otherwise.

The window size is set to 1800 frames, and the step size is 30 frames, ensuring that fatigue state is determined based on continuous feature data from 1800 frames.

After obtaining head Euler angles (HEAs), ${\eta }_{NAE}$ and ${\eta }_{NAM}$, the blinking frequency (BF), yawning frequency (YF), and nodding frequency (NF) can be further calculated from these evaluation indexes, which are calculated as follows:

$$\mathrm{BF},\mathrm{YF},\mathrm{NF}={\displaystyle \frac{n}{T}}\times 100\%$$

(8)

where n represents the number of frames of blinking, yawning, or nodding in a sliding window, and T is set to 1800 frames (equivalent to 1 min of video at 30 frames per second). The blinking frequency (BF) is calculated based on the average of both eyes. Since blinking is a synchronized bilateral behavior, using the average of both eyes provides a reliable and accurate measure.

As shown in Figure 6, the flowchart illustrates the process of calculating BF, YF, and NF using the sliding window model. The specific algorithm for calculating these evaluation indexes is detailed in Algorithm 1.

Algorithm 1 Evaluation Indexes
T = 1800;	# Sliding window size, set to 1800 frames
S = 30;	# The step size of the sliding window, set to 30 frames
$B{F}_i$ = [ ];	# Blink frequency in the i-th sliding window
$Y{F}_i$ = [ ];	# Yawning frequency in the i-th sliding window
$N{F}_i$ = [ ];	# Nodding frequency in the i-th sliding window
$n=⌊{\displaystyle \frac{N-T}{S}}⌋+1;$	# n is the number of sliding windows and N is the total number of frames acquired continuously
i = 1, 2, 3 … n;
$B{F}_i={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}Blin{k}_{(i-1)S+t};$  $Y{F}_i={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{\mathit{Yawn}}_{(i-1)S+t};$  $N{F}_i={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{\mathit{Nod}}_{(i-1)S+t}$

$Blin{k}_t$ represents the blinking state in the t-th frame image, ${Yawn}_t$ represents the yawning state in the t-th frame image, and ${Nod}_t$ represents the nodding state in the t-th frame image. The expression ⌊(N − T)/S⌋ denotes rounding down to ensure that the total number of frames N is not exceeded.

2.5. Feature Fusion Model

Following the normalization, the three evaluation indexes BF, YF, NF are combined into a single fatigue evaluation index $F_{\mathrm{B}\mathrm{Y}\mathrm{N}}$. Its expression is as follows:

$$F_{\mathrm{BYN}}=\sum _{i=1}^3{W}_i\times F_i\hspace{1em}\hspace{1em}\hspace{1em}\sum _{n=1}^3{W}_i=1$$

(9)

The expression indicates that $F_i$ represents the i-th evaluation index, and$W_i$represents the distribution weight of the i-th index. According to the importance and practicability of each physiological behavior in reflecting the fatigue degree, and incorporating the continuously updated$W_i$from the experimental validation in Section 3.6, the values of the W were distributed as: The value of $W_1$ is maximized (valued as 0.4) because research and experimental observations indicate that BF is typically considered one of the most important indicators in fatigue driving detection. A significant increase in BF may indicate that the driver is experiencing fatigue. Additionally, by matching the personalized ${\eta }_{NAE}$ threshold for different drivers, the impact of eye size is effectively eliminated, thereby justifying a higher weight for BF. YF and NF are regarded as secondary indicators. Although they can also reflect fatigue, their changes may not be as significant or frequent as those of BF Therefore, they are assigned lower weights ($W_2$ = 0.3, $W_3$ = 0.3) to reflect their relative importance in detecting fatigue while driving. Assigning weights of 0.4, 0.3, and 0.3 maintains the dominant role of blink frequency while appropriately considering the contributions of yawning and nodding, ensuring that the fatigue driving detection system remains sensitive and accurate under various conditions.

2.6. Fatigue Classification Based on Fuzzy Logic

Fatigue is a gradual, continuous, and subjective process influenced by physiological and psychological fluctuations. Traditional classification methods often segment fatigue into discrete levels—such as awake, mild, moderate, and severe—by applying fixed thresholds to a composite fatigue index. However, such rigid boundaries can lead to classification instability, especially near the threshold regions where small input variations cause abrupt state transitions.

To address this limitation, a fuzzy logic-based classification approach is employed in this study. The normalized fatigue score $F_{\mathrm{B}\mathrm{Y}\mathrm{N}}$ ∈ [0, 1] is mapped to overlapping fuzzy sets, each corresponding to a fatigue level. These sets are defined by triangular or trapezoidal membership functions, allowing a single input to simultaneously belong to multiple states with varying degrees of membership. The final fatigue level is determined by selecting the label with the highest membership value, as described in Algorithm 2.

Algorithm 2 Fatigue Classification Based on Fuzzy Logic

Input:$\mathrm{Normalized}\mathrm{fatigue}\mathrm{evaluation}\mathrm{score}{F}_{\mathrm{B}\mathrm{Y}\mathrm{N}}\in [0,1]$.

Output:$\mathrm{driver}’\mathrm{s}\mathrm{fatigue}\mathrm{level}Y\in \{\mathrm{Awake},\mathrm{Mild},\mathrm{Moderate},\mathrm{Severe}\}$. 1: Define fuzzy membership functions: 2:    ${\mu }_{\mathrm{A}\mathrm{w}\mathrm{a}\mathrm{k}\mathrm{e}}\left(F_{\mathrm{B}\mathrm{Y}\mathrm{N}}\right),{\mu }_{\mathrm{M}\mathrm{i}\mathrm{l}\mathrm{d}}\left(F_{\mathrm{B}\mathrm{Y}\mathrm{N}}\right){,\mu }_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}}\left(F_{\mathrm{B}\mathrm{Y}\mathrm{N}}\right){,\mu }_{\mathrm{S}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{e}}\left(F_{\mathrm{B}\mathrm{Y}\mathrm{N}}\right)$ 3: Initialize fatigue level Y ← null

4:$\mathbf{for}\mathbf{each}\mathrm{fatigue}\mathrm{score}{F}_{\mathrm{B}\mathrm{Y}\mathrm{N}}$ do

5:    $\mathbf{Compute}{\mu }_{\mathrm{A}\mathrm{w}\mathrm{a}\mathrm{k}\mathrm{e}}$     $\mathbf{\text{←}}\mathit{membership}\mathit{degree}\mathit{of}{F}_{\mathrm{B}\mathrm{Y}\mathrm{N}}$ to Awake

6:    $\mathbf{Compute}{\mu }_{\mathrm{M}\mathrm{i}\mathrm{l}\mathrm{d}}$             $\leftarrow \mathit{membership}\mathit{degree}\mathit{of}{F}_{\mathrm{B}\mathrm{Y}\mathrm{N}}$ to Mild Fatigue

7:    $\mathbf{Compute}{\mu }_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}}$     $\mathbf{\text{←}}\mathit{membership}\mathit{degree}\mathit{of}{F}_{\mathrm{B}\mathrm{Y}\mathrm{N}}$ to Moderate Fatigue

8:    $\mathbf{Compute}{\mu }_{\mathrm{S}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{e}}$           $\leftarrow \mathit{membership}\mathit{degree}\mathit{of}{F}_{\mathrm{B}\mathrm{Y}\mathrm{N}}$ to Severe Fatigue

9:     $\mathbf{Construct}\mathbf{dictionary}{\mathit{\mu }}_{\mathbf{d}\mathbf{i}\mathbf{c}\mathbf{t}}$← { 10:      $“\mathbf{Awake}”\mathbf{:}{\mu }_{\mathrm{A}\mathrm{w}\mathrm{a}\mathrm{k}\mathrm{e}}$, 11:      $\mathbf{“}\mathbf{Mild}\mathbf{”}\mathbf{:}{\mu }_{\mathrm{M}\mathrm{i}\mathrm{l}\mathrm{d}}$, 12:      $\mathbf{“}\mathbf{Moderate}\mathbf{”}\mathbf{:}{\mu }_{\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}}$, 13:      $\mathbf{“}\mathbf{Severe}\mathbf{”}\mathbf{:}{\mu }_{\mathrm{S}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{e}}$ 14:    }

15:  Determine output Y ← argmax(${\mu }_{\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}}$)

16: end for

17: return Y

Let $F_{\mathrm{B}\mathrm{Y}\mathrm{N}}\in \left[\mathrm{0,1}\right]$ denote the normalized fatigue score. The membership functions for each class are defined as follows:

Awake (left-shoulder trapezoidal function):

$${\mu }_{\mathrm{Awake}}(F_{\mathrm{BYN}})=\left\{\begin{array}{cc}1,\hfill & \hfill F\le 0.15\\ {\displaystyle \frac{0.30-F}{0.15}},\hfill & \hfill 0.15<F\le 0.30\\ 0,\hfill & \hfill F>0.30\end{array}\right.$$

(10)

Mild Fatigue (triangular function):

$${\mu }_{\mathrm{Mild}}(F_{\mathrm{BYN}})=\left\{\begin{array}{ll}\hspace{0.33em}\hspace{0.33em}0,\hfill & \hfill F\le 0.20\hspace{0.33em}\hspace{0.33em}\mathrm{or}F\ge 0.45\\ {\displaystyle \frac{F-0.20}{0.10}},\hfill & \hfill 0.20<F\le 0.30\\ {\displaystyle \frac{0.45-F}{0.15}},\hfill & \hfill 0.30<F<0.45\end{array}\right.$$

(11)

Moderate Fatigue (triangular function):

$${\mu }_{\mathrm{Moderate}}(F_{\mathrm{BYN}})=\left\{\begin{array}{ll}0,& F\le 0.35\hspace{0.33em}or\hspace{0.33em}F\ge 0.65\hfill \\ {\displaystyle \frac{F-0.35}{0.15}},& 0.35<F\le 0.50\hfill \\ {\displaystyle \frac{0.65-F}{0.15}},& 0.50<F<0.65\hfill \end{array}\right.$$

(12)

Severe Fatigue (right-shoulder trapezoidal function):

$${\mu }_{\mathrm{Severe}}(F_{\mathrm{BYN}})=\left\{\begin{array}{ll}1,& F\le 0.55\hfill \\ {\displaystyle \frac{F-0.55}{0.15}},& 0.55<F\le 0.65\hfill \\ 0,& F>0.65\hfill \end{array}\right.$$

(13)

The final fatigue level is determined using the maximum membership principle:

$$\begin{array}{c}Y=\arg \max {\mu }_l(F)\\ l\in \left\{\mathrm{Awake},\mathrm{Mild},\mathrm{Moderate},\mathrm{Severe}\right\}\end{array}$$

(14)

This fuzzy classification mechanism significantly enhances the continuity and interpretability of fatigue detection. It reduces instability caused by hard thresholds and provides a more flexible and physiologically consistent approach to identifying driver fatigue states. The corresponding membership curves are illustrated in Figure 7.

3. Experimental Results and Analysis

This article evaluates the performance of our proposed facial multi-feature fusion for fatigue detection. Initially, the algorithm’s feasibility and robustness were verified through online experiments with 12 participants, divided into three groups (A, B, and C), each consisting of 4 subjects. All participants completed a 30 min simulated driving task in a laboratory designed to replicate real-world conditions, including ambient noise and driving visuals.

To simulate different fatigue levels, experiments were conducted during both afternoon and late-night hours. Within each group, two participants performed the task in the afternoon, and two others during late-night sessions. This schedule ensured variations in mental alertness and induced mild to moderate fatigue. Participants followed a controlled daily schedule with limited rest, and caffeine intake was prohibited 3 h before the test. The experimental protocol was reviewed and approved by the institutional ethics committee.

Additionally, 45 randomly selected video segments from the YawDD dataset were used to further assess the algorithm’s accuracy in an offline setting, confirming the effectiveness of the personalized thresholds and the algorithm’s overall performance.

The data extraction and analysis platform used the programming language python 3.9 operated in a PyCharm environment based on the Windows 11 system. In addition, the Dlib toolkit was used to detect the face profile information and estimate the head posture for the 3D coordinate establishment. The camera provided an image resolution of 640 × 480 pixels, meeting the image acquisition requirement.

3.1. Changes in ${\eta }_{NAE}$,${\eta }_{NAM}$, and HEAα Values for Fatigue Detection

Figure 8 presents the blinking, yawning, and head movement behavior recorded from a single subject (participant “e”) among the 12 participants. This sample is used to illustrate the dynamic changes in ${\eta }_{NAE}$,${\eta }_{NAM}$, and HEAα values across different fatigue states.

Figure 8a,d,h record the changes in ${\eta }_{NAE}$, ${\eta }_{NAM}$, and HEAα values of an alert individual over a sliding window of 1800 test frames (approximately one min). Specifically, the ${\eta }_{NAE}$values remain mostly on a stable baseline, as shown in Figure 8a. When normal blinking occurs, a significant drop in the ${\eta }_{NAE}$values is observed. However, in a fatigue state [see Figure 8b], the troughs in the ${\eta }_{NAE}$values become more frequent, reflecting the true physiological state of fatigue. As illustrated in Figure 8c, an enlarged view of one${\eta }_{NAE}$wave indicates that, under normal open-eye conditions, the ${\eta }_{NAE}$values fluctuate randomly within a narrow range of 0.20 to 0.26. However, when the eyes are closed or blinking, the ${\eta }_{NAE}$values sharply drop to around 0.13 but return to the original level once the eyes open. In summary, blink detection can be achieved by calculating the ${\eta }_{NAE}$values.

Similarly, the ${\eta }_{NAM}$ values indicate the mouth’s open or closed state, as shown in Figure 8d. Minor fluctuations around the 0.15 mean line represent a closed mouth state, while significant jumps in the ${\eta }_{NAM}$ values occur during talking and yawning. The duration and amplitude of the value increase reflect the differences between normal open-mouth states (such as speaking) and physiological signs of fatigue. When yawning, the peaks in the ${\eta }_{NAM}$ values last longer (over 80 frames) and have greater amplitude. Figure 8e,g detail the changes in ${\eta }_{NAM}$ values during talking and yawning, respectively. This demonstrates that the ${\eta }_{NAM}$ values differentiate between normal and fatigue states.

Generally, when a person is awake, the head posture remains within a limited range of motion angles, whereas involuntary nodding occurs when feeling fatigued. This is well demonstrated in Figure 8h,i, where the HEAα curves in the fatigued state show more frequent instances of HEAα exceeding the threshold. Figure 8j captures a segment of the HEA curve during nodding, revealing that the HEAα values fluctuate within a larger range, with more frequent fluctuations in the fatigued state.

3.2. Fatigue State Changes in Driving Experiments

Figure 9 illustrates the changes in fatigue states among 12 participants from Groups A, B, and C over the course of a 30 min simulated driving experiment. The sessions were conducted during both afternoon and late-night periods. Most participants assigned to the afternoon sessions (Participants “a, d, e, f, i, l”) were in an alert state at the beginning of the experiment and gradually progressed to mild or moderate fatigue. In contrast, participants tested at late-night (Participants “b, c, g, h, j, k”) generally exhibited mild or even moderate fatigue from the outset due to reduced physiological alertness.

Overall, fatigue levels increased progressively as the experiment proceeded. Several individuals showed distinct fatigue progression patterns. For instance, Participant “e” exhibited a rapid transition from an alert state to mild fatigue, eventually progressing to moderate fatigue, indicating a continuous accumulation of fatigue. In contrast, Participant “j” showed a temporary decrease in fatigue level during the middle stage of the experiment, which may suggest a brief recovery or self-regulation phase. These observations highlight the influence of the time of day on fatigue development and emphasize the significant inter-individual variability in fatigue progression.

To evaluate the system’s ability to distinguish between multiple fatigue severity levels, a multi-class classification experiment was conducted based on data from 12 participants across six time points during a 30 min simulated driving task (72 total samples). Ground truth labels were determined through a combination of participant self-reported fatigue ratings and expert annotations of behavioral cues. The predicted fatigue levels were produced by the proposed model using fused behavioral indicators.

As shown in Figure 10, the classification results yielded an overall accuracy of 91.7%, with particularly strong performance in recognizing Awake and Severe states (both achieving 100% classification accuracy). Misclassifications primarily occurred between adjacent classes, such as Mild and Moderate, which are often difficult to distinguish due to their gradual transitions in real-world conditions. The confusion matrix confirms that the proposed system can reliably distinguish between four fatigue levels, supporting its applicability in graded fatigue warning systems.

3.3. Effectiveness of Personalized Thresholds in Improving Detection Accuracy

Figure 11a,b present line plots of yawns and blinks observed during fatigue driving tests on 12 participants. The plots compare the results of two detection methods: the personalized threshold method (blue dots) and the average threshold method (red squares), with the gray dashed line representing the real values.

It is evident from the figures that the personalized threshold detection consistently produces results that are closer to the real values, demonstrating superior detection accuracy. In contrast, the average threshold method shows greater deviation from the real values, reflecting its lower performance.

To better assess the performance of the personalized threshold and average threshold methods in fatigue driving detection, we calculated the Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE), and the results are presented in

Table 4. Comparison of MAE and RMSE for yawn and blink detection.

Method	Yawns MAE	Yawns RMSE	Blinks MAE	Blinks RMSE
Personalized Threshold	0.58	0.76	2.50	2.71
Average Threshold	1.42	1.55	9.25	10.61

. The MAE is calculated as:

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n|y_i-{\widehat{y}}_i|$$

(15)

where $y_i$ represents the actual values, ${\widehat{y}}_i$ represents the predicted values, and n is the number of data points. The RMSE is calculated as:

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}$$

(16)

As shown in Table 4, the MAE and RMSE values of the personalized threshold method are significantly lower than those of the average threshold method, indicating its ability to more accurately approximate the real values in practical applications.

These results clearly highlight the effectiveness of the personalized threshold method, which provides a closer approximation to the real values and outperforms the traditional average threshold method in both yawns and blinks detection. This confirms that the personalized threshold method offers a significant improvement in detection accuracy and reliability, making it a superior choice for fatigue driving detection.

3.4. Fatigue Detection Accuracy

To calculate the average accuracy of the proposed fatigue driving detection algorithm, 45 video segments randomly selected from the YawDD dataset were used to detect fatigue status under various conditions. The multi-feature fatigue detection method records the number of blinks, yawns, and nods, as well as the fatigue status for each video segment, and the detection results are compared with the actual values. The experimental results are shown in

Table 5. YawDD dataset fatigue test.

No.	YawDD (Video ID)	Number of Blinks (Detection/Actual)	Number of Yawns (Detection/Actual)	Number of Nods (Detection/Actual)	Fatigue Status (Detection/Actual)
1	1-FemaleNoGlasses-Yawning.avi	6/7	1/1	0/0	Y/Y
2	6-FemaleNoGlasses-Yawning.avi	6/6	2/2	1/1	Y/Y
3	10-MaleNoGlasses-Normal.avi	5/7	2/2	0/0	Y/Y
4	44-MaleNoGlasses-Yawning.avi	8/10	2/2	1/1	Y/Y
…	…	…	…	…	…
26	17-FemaleSunGlasses-Yawning.avi	0/7	2/2	0/0	N/Y
27	5-MaleSunGlasses-Yawning.avi	0/15	1/2	0/0	N/Y
…	…	…	…	…	…
42	3-FemaleGlasses-Normal.avi	2/3	0/0	0/0	N/N
43	3-MaleGlasses-Normal.avi	3/4	0/0	0/0	N/N
44	2-FemaleNoGlasses-Normal.avi	7/7	0/0	0/0	N/N
45	10-MaleNoGlasses-Talking.avi	12/13	0/0	0/0	N/N

(only a portion of the results is presented).

The results indicate that, out of 45 detections, 43 correctly identified the fatigue status, with only 2 instances where the fatigue state was mistakenly identified as normal. The average accuracy reached 95.6%. Upon review of the dataset, certain conditions were found to be unfavorable for detection, such as talking, laughing, yawning with the mouth covered, facial obstructions, and extreme lighting conditions (either too strong or too weak), all of which reduce detection accuracy. Therefore, future research may focus on improving fatigue detection under these challenging conditions. However, based on the experimental results, the overall detection accuracy remains high, which does not affect the practicality and reliability of the proposed method.

3.5. Comparison with Benchmark Methods

To objectively evaluate the effectiveness and advantages of the proposed multi-feature fusion fatigue detection method, comparative experiments were conducted against existing benchmark methods from the relevant scientific literature. Experimental results from these methods and the proposed algorithm are summarized in

Table 6. Accuracy comparison of different fatigue detection methods.

Method Reference	Indicators Used	Personalized Threshold	Accuracy (%)
Chen et al. [39]	EAR, MAR	No	87.37
Zhao et al. [40]	PERCLOS, POM	No	93.62
Ghourabi et al. [41]	Eye closure, Yawning	No	94.31
Yi et al. [42]	Eye features (EAR, PERCLOS, BF, POR)	No	95
Li et al. [22]	BF, YF	No	95.10
Proposed Method	BF, YF, NF	Yes	95.6

.

As shown in Table 6, the proposed method achieves the highest accuracy (95.6%) among all compared approaches. Traditional methods without personalized thresholds, although integrating one or more fatigue indicators, generally result in slightly lower performance. For example, the method by Chen et al. [39] reached only 87.37%, while Li et al.’s [22] method achieved 95.10% but did not consider nodding behavior. By incorporating blinking, yawning, and nodding frequencies along with a personalized threshold strategy, the proposed approach demonstrates superior detection accuracy.

3.6. Impact of Weight Combinations on Fatigue Detection Accuracy

This section investigates the impact of different weight combinations for blinking frequency (BF), yawning frequency (YF), and nodding frequency (NF) on the accuracy of fatigue detection. As shown in

Table 7. Iterative optimization of weight combinations and corresponding fatigue detection accuracy.

Step	Weight Combinations (BF, YF, NF)	Adjustment Strategy	Detection Accuracy (%)
1	$W_1 =$0.30, $W_2 =$0.30, $W_3 =$0.40	Initial setup (biased towards NF)	88.9
2	$W_1 =$0.30, $W_2 =$0.35, $W_3 =$0.35	Increase YF weight	90.2
3	$W_1 =$0.35, $W_2 =$0.35, $W_3 =$0.30	Balanced BF and YF weights	91.7
4	$W_1 =$0.40, $W_2 =$0.30, $W_3 =$0.30	Increased BF (optimal)	95.6
5	$W_1 =$0.45, $W_2 =$0.25, $W_3 =$0.30	Larger BF weight	93.8
6	$W_1 =$0.40, $W_2 =$0.35, $W_3 =$0.25	Slight increase in YF	94.2
7	$W_1 =$0.40, $W_2 =$0.25, $W_3 =$0.35	Slight increase in NF	93.1

, the weight distribution across multiple experimental steps was optimized to understand how changes in feature weights affect detection performance.

The experiment began with an initial setup (Step 1) that was biased towards NF, resulting in a detection accuracy of 88.9%. The weight of YF was then increased (Step 2), making YF more influential in the detection process, which led to an increase in accuracy to 90.2%. In Step 3, the weights of BF and YF were balanced, and the accuracy further increased to 91.7%.

However, when the weight of BF was increased to 0.4 in Step 4, the highest accuracy of 95.6% was achieved. Increasing the BF weight further in Step 5 to 0.45 caused the accuracy to drop to 93.8%. This suggests that while BF plays a significant role in fatigue detection, its weight cannot be increased indefinitely to improve performance. Beyond a certain point, increasing BF’s weight reduces the influence of other variables, such as YF and NF, leading to a decrease in overall accuracy.

When the weights of YF and NF were slightly adjusted in Steps 6 and 7, the detection accuracies were 94.2% and 93.1%, respectively. This indicates that while YF and NF contribute to fatigue detection, their impact on improving detection accuracy is more limited compared to BF.

These experiments validate the importance of BF in fatigue detection and reveal its non-linear effect on detection performance. The optimal weight combination ($W_1$ = 0.4, $W_2$ = 0.3, $W_3$ = 0.3) was achieved by fine-tuning the weights of BF, YF, and NF, representing the best configuration for this task.

4. Conclusions

In this article, a multi-feature fusion fatigue detection algorithm that accounts for individual driver characteristics is proposed. By analyzing facial data, the algorithm employs personalized thresholds to determine the states of the eyes and mouth. The algorithm evaluates BF, YF, and NF as evaluation indexes, employing a sliding window model to track dynamic changes in fatigue levels. Unlike fatigue detection algorithms based solely on a single feature, our approach comprehensively considers various facial features associated with fatigue behavior, including the eyes, mouth, and head. This comprehensive approach enhances judgment accuracy and response speed. Experimental results validate the accuracy of the algorithm, with an average accuracy of 95.6%. Moreover, the algorithmic model successfully distinguishes between different levels of fatigue, such as Mild, Moderate, or Severe.

Although the proposed method demonstrates high detection accuracy in the experiments, it still has the following limitations:

• The algorithm is sensitive to variations in lighting intensity, and detection accuracy decreases under extreme lighting conditions (e.g., overly strong or weak light).
• The initial calibration of personalized thresholds depends on baseline data from the participants, which may limit large-scale application.
• The feature weight settings are based on manual experience; future work could incorporate machine learning techniques to adaptively optimize the weights.

Future research will aim to overcome these limitations by introducing more robust image preprocessing algorithms (such as illumination normalization techniques) and advanced feature fusion models (such as deep learning). In addition, expanding the sample size and increasing participant diversity—covering a broader range of ages, genders, and facial structures—as well as validating the algorithm across multiple publicly available fatigue detection datasets, are essential steps to further improve the model’s generalizability and practical applicability.