How to Prevent Drivers before Their Sleepiness Using Deep Learning-Based Approach

Abstract

Drowsy driving causes many accidents. Driver alertness and automobile control are challenged. Thus, a driver drowsiness detection system is becoming a necessity. In fact, invasive approaches that analyze electroencephalography signals with head electrodes are inconvenient for drivers. Other non-invasive fatigue detection studies focus on yawning or eye blinks. The analysis of several facial components has yielded promising results, but it is not yet enough to predict hypovigilance. In this paper, we propose a “non-invasive” approach based on a deep learning model to classify vigilance into five states. The first step is using MediaPipe Face Mesh to identify the target areas. This step calculates the driver’s gaze and eye state descriptors and the 3D head position. The detection of the iris area of interest allows us to compute a normalized image to identify the state of the eyes relative to the eyelids. A transfer learning step by the MobileNetV3 model is performed on the normalized images to extract more descriptors from the driver’s eyes. Our LSTM network entries are vectors of the previously calculated features. Indeed, this type of learning allows us to determine the state of hypovigilance before it arrives by considering the previous learning steps, classifying the levels of vigilance into five categories, and alerting the driver before the state of hypovigilance’s arrival. Our experimental study shows a 98.4% satisfaction rate compared to the literature. In fact, our experimentation begins with the hyperparameter preselection to improve our results.

1. Introduction

Hypovigilance is a condition in between being awake and sleepy, and it carries with it the danger of falling asleep with one’s eyes closed for a while. This condition is not chosen and causes the individual to become less alert. Hypovigilance manifests itself in a variety of ways [1], including yawning, slow reflexes, droopy eyelids, tingling eyes, a desire to shut the eyes for a second, the requirement to relax, the need to constantly change positions, episodes of “micro-sleeps” (about 2–5 s), and trouble keeping the head up.

Accidents caused by sleepiness were more common on the highways and major roads. They take place both at night and throughout the day. Driver fatigue increases the risk of a vehicle’s skidding, losing control, or not being stopped in time. Due to these merits, we believe that the drowsiness state should be taken seriously. A significant portion of annual traffic accidents may be attributable to drowsiness, according to experts [2]. Drowsy drivers are more likely to make mistakes and cause accidents because they respond more slowly than sober drivers to potential threats, crises, and other events. These drivers’ severely diminished alertness results in poor task performance because they are unable to appraise any given situation.

Indeed, the areas of medicine and psychology have benefited greatly from recent discoveries in computing. In fact, researchers are trying to pin down the specific environmental and interpersonal factors that contribute to people’s emotional states [3,4,5]. The human–computer interaction (HCI) area known as “affective computing” examines how computers may help humans process and understand our emotions. In 1997, Picard [6] established the core ideas of identifying, interpreting, and expressing affect to allow computers to recognize, comprehend, express, and replicate human emotions. Human driving behavior, as a domain of affective computing, may be studied and understood in more detail via the study of driver tiredness detection. The state of hypovigilance is primarily manifested by a reduction in vigilance with the appearance of various behavioral signals, such as decreased reflexes, the presence of yawning, the heaviness of the eyelids, and the inability to keep one’s head in a frontal position. The suggested solutions have the common ground of relying on the monitoring of the driver’s physiological and behavioral cues. Some methods that are considered intrusive are used. In most circumstances, an electroencephalographic signal is used to determine the driver’s psychological and behavioral states. However, so-called non-invasive procedures that depend merely on visual signals related to a drop in attentiveness make use of behavioral indications. A camera installed in the car may record the driver’s facial expressions and other indicators, such as a yawn, blink, or gaze. So-called non-invasive approaches are more amenable to use in real-world settings because they impose less technological limitations on the deployment. The purpose of our work is to employ a non-intrusive approach based only on footage from a camera mounted in front of the vehicle to assess the driver’s level of tiredness. There has to be a way to identify signs of drowsiness in the motorist before they really fall asleep behind the wheel.

The innovative work in this work includes seven main aspects:

• First, the detection of facial landmarks and the 3D head position MediaPipe Face Mesh which uses approaches linked to machine learning and has shown its robustness in terms of its accuracy and speed compared to other approaches in the literature.
• Second, estimating the driver’s gaze based on the relationship between the iris and the eyes makes it possible to get a better idea of how tired or distracted the driver is.
• Third, the calculation of a normalized image of the iris is seen as a way to add more useful features to the MobileNetV3 model.
• Fourth, the concatenation of several features linked to the eyes and head position gives us more information, which makes it easier to find out if the driver is tired.
• Fifth, the choice of the deep neural networks by the LSTM in our system in order to take not only the current state of the driver but also the previous states.
• Sixth, we conduct our first experiment, which lets us pick the best hyperparameters to make our training model better.
• Seventh, a detection of five levels of driver states is realized in this work in order to alert the driver to their state of hypovigilance to avoid accidents.

This paper is distributed as follows:

In Section 2, we describe the approaches linked to the literature, detailing the work carried out in those that analyze the driver’s eye blinks, those that are based on the location of the mouth and the detection of yawns, and those that use several components of the face. A description and discussion of each approach described in the state-of-the-art section is carried out in order to determine the advantages and disadvantages of each work. This study of the literature allows us to justify the choice of our approach because we find it non-invasive.

In Section 3, a detailed description of the proposed methodology is elaborated. In fact, our approach consists first and foremost of justified pre-processing in order to properly determine the descriptors to be extracted, either by a spatiotemporal analysis or by a transfer learning method. These features constitute the input of our deep learning model, which makes it possible to classify five levels of driver fatigue.

In Section 4, a set of experiments on three databases is carried out. These experiments allow us to validate the proposed approach and compare our results with the existing works.

Section 5 summarizes this paper’s contributions and suggests further study.

2. Related Works

Face recognition [7], facial expression [8], and head tracking [9] are all areas of computer vision research that have made big strides in recent years. These steps have helped image-based technologies grow quickly. We can put these methods into three main groups: ones that use blinking, ones that use yawning, and ones that use both. In this section, we discuss several works that are related to our own work, which is centered on the detection of driver fatigue.

2.1. Eye Blinking Approaches

Approaches based on eye blinking to detect driver fatigue are common. Cyganek et al. [10] suggested using two cameras to identify sleepiness levels in drivers. In this method, the eye areas are found by using a pair of recognition models in a cascade. The first model is used to find the position of the eye, and the second model confirms the position of the eye. To identify a driver’s eyes and an estimated eye state, Gou et al. [11] suggested a cascading regression technique. At each iteration, the image pixel value of the eye’s center and features from its corners and eyelids are used to find the eye’s regions and the probability that it is open. After identifying the driver’s face using Haar feature classifiers, Ibrahim et al. [12] utilized a correlation matching technique to find and follow the driver’s eyes based on their size, intensity, and shape. An eye-based sleepiness detection device was suggested by Mandal et al. [13] to keep tabs on bus drivers. This method uses two different ways to find the eyeballs and then uses spectral regression to estimate on a continuous scale how open both eyes are. Next, Mandal et al. [13] utilized adaptive integration to combine the degrees of openness in both eyes in order to calculate the proportion of time that the bus driver’s eyelids were closed. For the purpose of determining whether or not a driver’s eyes are closed, multiscale histograms of principal-oriented gradients were developed by Song et al. [14]. You at al. [15] captured a motorist’s nighttime picture using a near-infrared camera and then utilized a spline function to match the eyelid curve, all in the name of determining whether or not the driver was sleepy. To develop a deep integrated neural network (DINN) for identifying a driver’s eye state, Zhao et al. [16] combined a convolutional neural network (CNN) with a deep neural network (DNN). To boost the DINN’s efficiency, Zhao et al. [16] pretrained the integrated model using transfer learning. In experiments conducted in a driving context, the DINN identification rates reached 97%. Anitha et al. [17] suggested a continuous surveillance system that monitors the eye gaze to address the issue of driver concentration. Once the driver’s face has been found in the video, the driver’s eye movements will be tracked in real time. Images are learned and classified by using the Viola and Jones face recognition algorithm so that a warning signal sounds if the eyes remain closed for an extended length of time.

2.2. Approaches Based on Mouth and Yawning Detection

When pilots begin to feel sleepy, they generally start yawning. A lot of recent research has focused on detecting driver fatigue using mouth status extraction. To determine if a person yawned or not, Akrout et al. [18] utilized an active contours algorithm to locate the driver’s lips and then estimated the lip region to indicate the mouth’s status using the optical flow technique. Omidyeganeh et al. [19] employed the reverse projection concept for detecting the variations in the driver’s mouth status to identify yawns, as well as a customized version of the Viola–Jones detector for mouth and face localization. In order to identify yawning activity in videos of drivers, Zhang et al. [20] employed convolutional neural networks (CNNs) to generate spatial images of mouth states and a long short-term memory (LSTM) network to categorize these images. After employing deep learning and Kalman filter-based methods to identify and follow the driver’s nose and face, Zhang et al. [21] employed a neural network to identify yawning by analyzing the confidence value of the nose monitoring, properties of the face’s movement, and gradients around the mouth’s borders. Wang et al. [22] used a Kalman filter for mouth tracking based on the extraction of geometric features (height, width, and the height between the upper and lower lips). These features are put together into a vector and fed into the BP neural network to figure out how the mouth is moving (yawning, mouth closed, talking, etc.) to find out if the driver is tired.

2.3. Approaches Based on Several Components of the Face

Both of the above methods are often employed to detect human sleepiness due to the significant changes in the eye or mouth behavior that occur when drivers are tired. If you only look at the eyes or lips, though, you might not be able to tell who someone is because you are missing important clues from the other organs. The results of the study showed that combining the features of the eyes, lips, and other parts of the face can make it much easier to tell if someone is tired or not. Torres et al. [23] proposed a non-invasive technique for discreetly differentiating between the vehicle’s driver and passengers during a message read-out. They put information gathered from smartphone sensors into a machine learning model. They tried out different simulations and looked at seven of the most advanced machine learning methods in different situations. The authors’ model uses a convolutional neural network with gradient boosting. To measure anxiety, Benoit et al. [24] created a simulation tool that takes in both visual (facial expression) and physiological information. In addition to the electrocardiogram (ECG), the model used the driver’s blink rate, yaw rate, and head tilt angle to figure out how stressed the driver was. DrowsyDet was introduced by Chaoui et al. [25] and has now become a popular smartphone app used to monitor sleepiness. These are the criteria for this proposal. To begin, they utilize the facial area and landmarks model to retrieve data from the face recognition model. Chaoui et al. developed three (convolutional neural networks) models to detect face tiredness, eye states, and mouth states. The condition of the driver is then figured out by adding up the data from each model (normal, yawn, or sleep). DrowsyDet is less invasive than other methods because it only needs a cell phone and not access to the Internet. In this work, the authors use face data and the human eye state to examine weariness while driving, although the approach is neither real time nor accurate because of time limits. To identify sleepy drivers, Shih et al. [26] presented a spatial–temporal system. First, a spatial convolutional neural network model was used to take features from each video’s face images that were linked to sleepiness. Then, the temporal variation in sleepiness was built using an LSTM network. To round out the process, they used a temporal smoothing approach to average out the projected driving state scores. On the NTHU database, the suggested model had an accuracy rate of 82.61 percent. Using facial and hand localization, Hossain et al. [27] developed a CNN-based approach to detect fatigued drivers and determine what distracted them. For this purpose, they used four different architectures: CNN, VGG-16, ResNet50, and MobileNetV2. They tested how well their proposed model works by first putting it through a long training process using thousands of photos from a public dataset that shows 10 different distracted driving postures or situations. According to the outcomes, the pretrained MobileNetV2 model is the most effective classifier. Long et al. [28] proposed a driver drowsiness detection approach composed of three steps. Indeed, a multitask convolutional neural network (MTCNN) is used to identify human faces, and then an open-source software library (DLIB) is employed to pinpoint facial key points for the purpose of extracting the fatigue feature vector of each frame, which is fed to the long short-term memory network (LSTM) to yield a final fatigue decision. It was shown by Long et al. that the tiredness state identification algorithm has an accuracy of 90%. Akrout et al. [29] developed a work that automates driver sleepiness control. These non-invasive remedies are based only on visual indicators. These indices are computed using video stream spatiotemporal data. This study analyzes driver faces to identify inappropriate conduct. Three databases assess the proposed fusion system and find several successes but some glaring detecting weaknesses that diminish the accuracy caused by the yawning detection part. Weighted ensembles of CNNs were proposed by Eraqi et al. [30], who advocated for the use of four different CNN designs. Following this, CNNs are learned using the five different image sources that make up the AUC distracted driver set of the data. The findings showed that when trained on raw pictures, individual CNNs performed the best. The results then show that the fusion is more accurate than both solo CNNs and the majority voting fusion. This is performed by combining the predictions from the individual CNNs using a weighted genetic algorithm. An ignition lock was proposed by Arunasalam et al. [31] as part of a real-time sleepiness monitoring system for cars. Drowsiness can be detected with the use of a heart rate monitor and a sensor installed into worn eyewear. This study can identify three distinct stages of sleepiness. When an excessive amount of sleepiness is detected, the ignition will be locked. When a level of sleepiness that could kill the driver is found, the car will slow down and stop.

2.4. Discussion

According to our study of the literature, we have noted several weaknesses. Indeed, the work carried out, which is called “invasive,” leaves the driver a little embarrassed by the installation of electronic components on his body. A lot of work is also conducted on finding only yawning or analyzing blinks to make a binary classification of alertness. This does not make it possible to alert the driver in advance of his state of hypovigilance in order to avoid accidents before they happen. This is why we propose calculating spatiotemporal features that analyze the 3D head pose, the blinking of the eyes, the position of the iris to estimate the driver’s gaze, and the normalization of the iris zone in order for a pretrained learning model to extract these descriptors. We also note that several studies using methods based on machine learning or convolutional neural networks have succeeded in achieving satisfactory results. This is also limited by the choice of the model, which leads us to propose a model that takes into consideration the previous states of the driver and not only their current state. In what follows, we describe the proposed model.

3. Methodology

In the following section, we present all the stages of our approach for multilevel driver fatigue detection. First, using the MediaPipe Face Mesh [32], we can estimate the landmarks of the face, detect and track the eye area, and estimate the head pose that will be used for the next stage. A new approach for iris detection and normalization is used for the second step. The resulting image is considered an input for a MobileNetV3 [33] architecture to pretrain the model and extract features that are enhanced by the calculation of the distances $Ya$, $Pit$, $Ro$, $E_{Po2}$, $E_{Po2}$, and $E_{br}$. The distances $E_{Po1}$ and $E_{Po2}$ represent the distance between the center of the iris and the points representing the intersection of the eyelid, $E_{br}$ is the calculated blinking rate, while $Ya$, $Pit$, and $Ro$ represent the head angles calculated by MediaPipe Face Mesh. These traits will be put into a deep LSTM network that will be able to tell when a driver is getting tired. Figure 1 summarizes the whole process.

3.1. Head Pose Estimation and Facial Features Extraction

The MediaPipe Face Mesh system can calculate 468 unique 3D points on a person’s face in near real time. With machine learning, the 3D face area can be estimated from a single camera feed, so a separate depth sensor is not needed. The method achieves the necessary performance level for real-life experiences by using lighter design structures and GPU accelerators at every step of the pipeline. Indeed, face alignment, also known as face registration, is the process of comparing a face to a preexisting template to forecast its shape. In most cases, you will be asked to find a certain number of landmarks, called key points. Points such as these either contribute to meaningful facial features or have their own independent meanings [34]. MediaPipe Face Mesh instead makes use of a 3D morphable model (3DMM) [35] whose posture, size, and properties are estimated. Principal component analysis (PCA) is often used to generate a 3D mesh model. However, the variety of the collection of faces collected for the model determines the linear manifold covered by the PCA basis, and hence the range of feasible predictions is restricted. The resultant mesh often has many more points. For instance, unlike MediaPipe Face Mesh, many works are unable to accurately depict a face with just one eye closed. The authors suggest a residual neural network architecture for figuring out where the 3D mesh vertices are. Each vertex would act as its own landmark. The 468 points that make up the mesh network are all organized into squares (Figure 2). No depth sensor data are needed for this job; all the model needs is a frame (or a stream of frames) from a single RGB camera as a feed.

In our case, we use MediaPipe Face Mesh to find and track all the parts of the driver’s face that we need and to figure out the distances we set up before. In this step, the yaw, pitch, and roll are calculated and saved to the variables $Ya$, $Pit$, and $Ro$, respectively. Figure 3 shows some examples of the MediaPipe Face Mesh system, which makes it possible to detect the head pose and locate the eyes and other facial components.

3.2. Eye Blinking Rate Detection

The main objective in this section is to calculate a normalized rate for eye blinking from eye landmark detection with MediaPipe Face Mesh. Indeed, as shown in Figure 4, the Iris Landmark Model [36] from MediaPipe Face Mesh detects 14 points for the upper and lower eyelids, 2 points for the eye corners, and 5 points for the iris to define the iris edges and center. In this step, we measure 7 distances from the eye to figure out how often we think the person blinks. In fact, the normalized rate that is fed to the pretrained MobileNetV3 model makes the training more rapid.

Obviously, when people’s eyes are open, the distance between the upper and lower eye point features will be relatively large. When the eyes are closed, the distance becomes smaller. The eye rate is calculated using the distance between the eyelid feature points. Figure 4 shows the eye and iris landmarks for an open eye. The following equation is used to calculate the eye blinking rate, $E_{b}r$:

$$E_{br}=\sum _{n=1}^7\frac{\mid P_n-P_{(m-n)}\mid }{max(\mid P_n-P_{(m-n)}\mid )}\mathrm{where}m=15$$

(1)

where

$$\mid P_n-P_{(m-n)}\mid =\sqrt{{(X_n-X_{(m-n)})}^2+{(Y_n-Y_{(m-n)})}^2}$$

(2)

Knowing that $P_n$ and $P_{(m-n)}$ are the eyelid feature points, X and Y are the coordinates of each point {$Pn,P_{(m-n)}$}, as shown in Figure 4.

3.3. Iris Position According to Eye Corners

We can tell how tired a driver is by looking at where the iris is in relation to the corners of the eyes. In the event that the driver’s head is in the road’s field of vision while he is also looking to the left and right by moving his iris, this information may in fact aid the system in making a more accurate choice. Let us start with $I_c$, the iris center. The eye corners are represented by $C_1$ and $C_2$. The eye position is calculated using the following Equations (3) and (4):

$$E_{Po1}=\frac{D_1}{D_c}$$

(3)

$$E_{Po2}=\frac{D_2}{D_c}$$

(4)

The Euclidean distance between two points can be represented as $Dist(Point1,Point2)$ (Equation (2)). The distances $D_1$, $D_2$, and $D_c$ are calculated as follows:

$$D_1=Dist(I_c,C_1)$$

(5)

$$D_2=Dist(I_c,C_2)$$

(6)

$$D_c=Dist(C_1,C_2)$$

(7)

3.4. Iris and Its Surroundings Features Extraction

The detection of driver fatigue is a complex event that necessitates multiple analyses. In fact, fatigue should not be found in its final stages so that as little damage as possible is done. It is for this reason that we base our general model (Figure 1) on the division of the driver’s drowsiness state into several levels, from 1 to 5. In order to achieve this level of vigilance, our system is fed with several pieces of data that analyze the eyes of the driver. Each of its pieces of data is independent of one another and allows different states of eye blinking to be identified.

In this section, we are interested in analyzing the iris and its surroundings. As a pre-processing step of the inputs to our MobileNetV3 model, we need to segment and normalize the iris. The latter makes it possible to extract these characteristics. In fact, if we normalize the image well before putting it into the deep learning model, we can obtain better results and speed up the learning and testing process afterward.

We notice that the shape of the iris is not a perfect circle. We have decided in our work to treat it as an elliptical shape for better segmentation. In our case, we can calculate the ellipse of the iris by using 2 points and its center, which are detected and explained previously by MediaPipe Face Mesh. Suppose that the center of the iris is presented by the point $I_c$, and the two points $I_2$ and $I_1$ are the two points of the ellipse that surround the iris of the eye. Let the equation of the ellipse be the following:

$$\left\{\begin{array}{c}{x}_e=x_{e0}+a·cos\left(\alpha \right)\hfill \\ y_e=y_{e0}+b·sin\left(\alpha \right)\hfill \end{array}\right.$$

(8)

with $x_e$ and $y_e$ being the coordinates of a point belonging to an ellipse, a and b being the foci, and $x_{e0}$ and $y_{e0}$ being the center coordinates of the ellipse. Using Equation (8) for the two points $I_2$ and $I_1$, we obtain the following equations:

$$\left\{\begin{array}{c}{x}_{I2}=x_{e0}+a\times cos\left(\alpha 1\right)\hfill \\ y_{I2}=y_{e0}+b\times sin\left(\alpha 1\right)\hfill \\ x_{I1}=x_{e0}+a\times cos\left(\alpha 2\right)\hfill \\ y_{I1}=y_{e0}+b\times sin\left(\alpha 2\right)\hfill \end{array}\right.$$

(9)

Now, we have a system of equations with 4 variables, a, b, $\alpha 1$, and $\alpha 2$, which can be calculated by replacing the coordinates of $I_c$, $I_2$, and $I_1$ into Equation (9). The two distances of the local foci of the ellipse are thus calculated, and one can then determine the external contour of the iris. Figure 5 shows a series of driver iris detection examples.

For the MobilenetV3 model to work properly, it is better to standardize the sizes and shapes of the iris localized. In other words, a uniform model would need to be set up to represent any localized iris that also includes part of the eyelids when the person blinks. This kind of model should account for all the different kinds of geometric and colorimetric errors that an iris can show. These deformations are often due to the different shooting conditions, the attitude of the subject, and the behavior of the iris according to its environment. Most of the time, the iris is out of shape, but this can be fixed by using geometric normalization. The goal of this standardization is, among other things, to set up an alignment system that will make it easier to process data in the future. To solve this problem, we propose in our system to use the Daugman technique [37] which consists of linearly unrolling the crown to present the iris in the image, which corresponds to the transition from the Cartesian coordinate system to the polar coordinate system (Figure 6), applied to the 3 matrices, red, green, and blue.

The iris image normalized is scaled automatically to 224 × 224 × 3 according to the equation and the equation for 3 color matrices.

$$x(r,\theta )=r·x_i\left(\theta \right)$$

(10)

$$y(r,\theta )=r·y_i\left(\theta \right)$$

(11)

The point ($x_i\left(\theta \right)$, $y_i\left(\theta \right)$) is located on the iris-sclerotic contour in the same direction (Figure 6), r and $\theta$ are the polar coordinates. After being segmented and normalized, the iris can be seen to be divided into two distinct parts: one containing the iris’s unique properties, and the other including the eyelash and lid hairs, as shown in Figure 7. Indeed, a driver’s blinking varies according to how tired they are.

3.5. MobileNetV3 Transfer Learning for Iris Features Extraction

Transfer learning is an example of a way that deep learning can be used. Multitasking and conceptual drift are also examples of how deep learning can be used. Transfer learning has become more popular in the deep learning community. This is because training deep learning models takes a lot of time and effort, and the datasets on which they are trained are large and difficult. Transfer learning in deep learning works best when the model features learned in the first assignment can be used in other situations. During transfer learning, one network is trained on one dataset and one task, and then the features it learns are used to train a second network on a different dataset and a different task.

The skills acquired via transfer learning are not limited to the original employment but may be used in the new one as well. To solve predictive modeling issues, we apply transfer learning. “Pre-trained model” and “develop model” are the two most common ways of transfer learning. For classifying image data, a huge number of high-performing models have been built and used on ImageNet, which is where the images used in the competitions are located. Because of this, the field of building and training convolutional neural networks has made a lot of progress. Not only that, but a lot of the models were made available under an open-source license. When these models are used for computer vision tasks, they provide a solid base for transfer learning. Because of their useful learning characteristics, the models may be trained to recognize recurring patterns in images. The models still do well at the original tasks they were made for, and they are also the best models today. Figure 8 shows our transfer learning model.

Pretrained model weights are very easy to use because they are open source and can be downloaded for free. Multiple deep learning libraries, such as Keras, are used to obtain the model weights, which are then used in the same model architecture. We employed transfer learning and the deep learning architecture MobileNetV3 to extract the most informative features possible from the training dataset. After being trained on a large set of pictures, this architecture has a strong representation of low-level features such as edges, rotation, lighting, and patterns that can be used to pull deep features from new, noisy pictures. For this reason, these models might be useful for using transfer learning ideas to pull out the qualities you want.

When it comes to apps and embedded devices designed for smartphones, MobileNet [38] is a widely used CNN-based approach. For us, this will be particularly helpful for monitoring the state of our drivers in the event that weariness sets in. Depthwise separable convolution is the main idea that is used to make the computation less complicated. The high computational needs of the classic CNN design are caused by the point-to-point convolution between the image and the filter. Pointwise convolution in MobileNet uses 1 × 1 convolution to make a linear combination of the depthwise-layer output, while depthwise convolution uses a single filter for each input channel. By factorizing the convolutions, both computational time and model size may be drastically reduced, as shown in Figure 9. Consequently, the model is well-suited for use in real-time systems, portable gadgets, and embedded systems.

Two variants of MobileNetV3, MobileNetV3-large and MobileNetV3-small, were presented in [33]. As a successor to the MobileNetV2 framework [39], MobileNetV3 aims to improve upon its predecessor in terms of both latency and precision. MobileNetV3-large, for instance, reduced latency by 20% while improving accuracy by 3.2% compared to MobileNetV2. MobileNetV3-small achieves 6.6% higher accuracy at the same latency. For these reasons, we opted to use MobileNetV3-large in our transfer learning and fine-tuning model. In fact, a network architecture search (NAS) approach to find the best possible network architecture and kernel size for the depthwise convolution was developed in MobileNetV3.

The following components make up MobileNetV3: a 3 × 3 depthwise convolutional kernel, then batch normalization and activation functions, a 1 × 1 convolution (pointwise convolution) for linear combination calculations and feature map extraction within the depthwise separable convolutional layer, and so forth. A decreased feature map is created by a global average pooling layer with a residual block based on the residual skip connection mechanism, which uses bottleneck block networks. Finally, an h-swish activation function with the Rectified Linear Unit (ReLU) activation function.

In reality, the inverted residual block is broken down into the following smaller blocks: The first block makes it possible to learn more sophisticated representations while decreasing the model’s computational load with 1 × 1 expansion and projection convolutional layers and a depthwise convolutional kernel of size 1 × 1. The second block uses a depth-separable convolutional layer and a technique called “residual skip connection” (squeeze-and-excite block) to let an important feature be chosen channel by channel.

3.6. Deep LSTM Network for Multilevel Fatigue Classification

When building the tiredness detection model, much of the prior research neglected to account for the time factor and the effect of the driver’s fatigue condition in earlier time periods. As this was the case, the rate of accurate detection was low. Long periods of driving may cause weariness, which only becomes worse if the driver keeps going. Therefore, LSTM [40] is used in our work as a means of identifying driver fatigue.

The primary benefit of employing LSTM is in the processing of sequence data. In a typical neural network, both the input and output are unrelated to one another. As a result, the results of one instant have no bearing on the results of the next. In LSTM, the current output of a sequence is linked to the previous output because the input of the hidden layer includes both the output of the input layer and the output of the hidden layer from the previous time. Second, unlike the recurrent neural network (RNN) model, LSTM model does not suffer from the issue of gradient dispersion. In reality, LSTM has a quicker learning speed and improved accuracy because the long-term sequence is addressed by inserting a logic gate unit.

In fact, at each time step, the LSTM cell is fed the input $x_t$ and the hidden status $h_{t-1}$ from the previous time step. Because LSTM remembers a unit status $c_t$, it will carry over the unit status $c_{t-1}$ from the previous time step at each time step. At each time step, the above input is weighted to obtain the total input z described by Equation (12).

$$z=W_x{x}_t+W_h{h}_{t-1}+b$$

(12)

$z_i$, $z_f$, $z_o$, and $z_g$ are the input gate (Equation (13)), forget gate (Equation (14)), output gate (Equation (15)), and quasi-cell state input (Equation (16)), respectively, where $\sigma$ represents the sigmoid activation function and tanh represents the hyperbolic tangent function. These four components of the transformation result z are separated and used in the following order.

$$i=\sigma \left(z_i\right)$$

(13)

$$f=\sigma \left(z_f\right)$$

(14)

$$o=\sigma \left(z_o\right)$$

(15)

$$g=tanh\left(z_g\right)$$

(16)

The cell state and the hidden state are represented by the following equations:

$$c_t=f\otimes c_{t-1}\oplus i\otimes g$$

(17)

$$h_t=o\otimes tanh\left(c_t\right)$$

(18)

The following Figure 10 displays the LSTM’s structure.

Indeed, in our case of fatigue detection, we proposed to use a deep LSTM network with two hidden layers (it will be justified in the experimentation section). The output descriptors of our MobilenetV3-large model are associated with the previously calculated 3D head pose estimation parameters yaw, pitch, and roll; the distances $E_{Po1}$ and $E_{Po2}$, which allow us to determine the position of the iris in relation to the eye corners; and finally, the percentage of eyes blinking by the calculated value $E_{br}$ for each iteration. Figure 1 summarizes all the steps described above in order to achieve a fatigue detection system at five levels: awake, a little tired, tired, very tired, and hypovigilance.

4. Experimentation and Analysis

The suggested method was tested on a 2.9 GHz Intel Core i7 10700 vPro with 32 GB of RAM and a 1 TB solid-state drive. The experimental corpus is first described in detail. Next, we conduct a series of experiments and calculate the metrics used to test the validity of our model and provide the outcomes in detail. The entire model was developed using the Python programming language.

4.1. Corpus and Experimental Data

To identify the various levels of driver fatigue in our experiment, we used the databases MiraclHB [41], YawDD [42], and DEAP [43].

The database MiraclHB [41] was collected in our lab under different lighting conditions. Each video consists of a person sitting between 30 and 50 cm in front of a Sony DSC-W510 digital camera. The recordings were made using a Jetion-style steering wheel, pedals, and a computer with driving simulator software installed. In order to study drowsiness, the participants were requested to remain up until midnight. Indeed, we used the same guidelines put forward by Picot et al. [44] in our simulation. All videos were shot in AVI and are 30 fps at a resolution of 640 × 480. Picot et al. [44] carried out an EEG and video data acquisition campaign to validate their hypovigilance detection algorithms. The video was recorded with a 200 fps camera. Hypovigilance was assessed using a 10 min psychomotor alertness task. This task measures an individual’s reaction time to a specific visual stimulus. The campaign was carried out on six subjects. Each subject was recorded twice: once at the end of the afternoon (around 6 p.m.) and a second time around 1 a.m. The first recording was to serve as a reference for awakening. The subjects were asked to stay awake until 1 a.m. in order to observe the states of hypovigilance.

Abtahi et al. [42] set up two cameras in separate locations to guarantee more data collection for many positions. To collect data for the YawDD database, a camera was placed beneath the front mirror in one case and on the dashboard in the other. Both videos were shot using a Canon A720IS digital camera, with the settings for 640 × 480 pixels and 30 fps. For the purpose of creating a more lifelike environment, the participants were instructed to buckle their seat belts in a car. The dataset includes videos of 57 male and 50 female volunteers who range in age, are of diverse nationalities, and have varying facial traits. The 107 people looked very different from one another. People took part in the activity with and without the use of eyeglasses, men with and without facial hair, men with and without mustaches, women with and without scarves, with varied hairstyles, and wearing different clothes.

The multimodal dataset for the human affective state analysis is presented by Koelstra et al. [43]. The physiological and electroencephalographic (EEG) data of 32 participants were monitored as they watched 40 separate 1 min music videos. On a scale that measured arousal, valence, positive and negative emotions, dominance, and level of familiarity, the participants rated each movie. For 22 of the 32 participants, frontal-facing video was also recorded. The camera can record video in 720p at 50 fps. The recordings are being analyzed by researchers in order to determine what factors contribute to the detection of weariness.

In our experiment, the databases described above were divided into three parts. A total of 10% for the test and 90% for the training and validation. Indeed, during the training process, a percentage of 10% is prepared for validation, split from the training dataset.

4.2. Evaluation Metrics

We used the Recall (Equation (19)), the Precision (Equation (20)), the F-measure (Equation (21)), and the Accuracy (Equation (22)) to evaluate our experimentation, which included finding faces and eyes, as well as our multilevel fatigue detection model.

$$Recall=\frac{TP}{TP+FN}$$

(19)

$$Precision=\frac{TP}{TP+FP}$$

(20)

$$F-measure=2\times \frac{Recall\times Precision}{Recall+Precision}$$

(21)

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(22)

where TP, TN, FP, and FN represent the True Positives, True Negatives, False Positives, and False Negatives samples, respectively. The effectiveness of the suggested approach is measured by computing the evaluation metrics for the feature extraction, training, and testing. In fact, these measurements help us fully understand the usefulness of the pre-processing that needs to be conducted and how it affects the proposed model as a whole, so that we can get good results and warn the driver before he falls asleep.

4.3. Experiments for Face Detection

Face detection is an essential step for our system. This step changes the results of the feature extraction, which are the inputs to the deep LSTM network. We compared our MediaPipe Face Mesh detection method to those in the literature of Viola and Jone [45] and Dlib Hog [46]. The results of our experiments with the three approaches we just talked about are shown in the table below, using the three selected databases in our experimentation.

Table 1. F-measure for face detection experiments using MiraclHB, YawDD, and DEAP databases.

	Viola and Jone	Dlib Hog	MediaPipe Face Mesh
DEP	0.86	0.91	0.98
YawDD	0.81	0.9	0.97
MiraclHB	0.91	0.93	0.99
Average	0.86	0.91	0.98

reveals that MediaPipe Face Mesh is very robust in face localization despite the presence of illumination and head rotation effects in all three datasets. There are drawbacks to using Viola and Jone or Dlib Hog. Indeed, neither method is capable of identifying faces that have been rotated or tilted. Additionally, they are affected by the presence of occlusions and changes in the illumination. Indeed, this experiment argues the choice of the MediaPipe Face Mesh method for the detection of faces, which is a primordial step in the model proposed.

4.4. Hyperparameter Tuning, Results, and Discussion

This second experimentation is responsible for building a fine-tuned HyMobLSTM model (Hybrid model MobileNetV3-large and deep LSTM network) for multilevel driver fatigue detection. The deep LSTM network’s input layer is made up of features extracted from MobileNetV3-large, which are enhanced by iris positions and blink rate features. When fine-tuning a model’s settings, many works often resort to the random-search technique. The context necessitates the definition of a search space for various hyperparameters. Consequently, [1–6], [0.01–0.0001], and [128–256–512–1024] are used as the starting points for the search ranges of the hidden-layer number, the learning rate, and the number of neurons in each hidden LSTM layer, respectively. Through the employment of an Adam optimization algorithm, the effectiveness of the training procedure is increased. In this experimentation, we apply a batch size of 50 and 100 epochs. The optimal settings for our proposed model hyperparameters were found using a random search strategy and are shown in

Table 2. HyMobLSTM model best hyperparameters values for random-search technique.

Hyperparameters	Values
Hidden-layer number	2
Learning rate	0.0005
First hidden-layer neurons	1024
Second hidden-layer neurons	512

.

In fact, the YawDD database was used for this second experiment for the fine-tuned HyMobLSTM model. YawwDD has videos that were recorded in an uncontrolled environment. This allows us to put our model to the test in terms of the image light sensitivity.

Table 3. Evaluation of the HyMobLSTM model for fine-tuned experiment.

Fatigue Level	Recall	Precision	F-Measure
Awake	0.932	0.924	0.928
A little tired	0.921	0.912	0.916
Tired	0.925	0.923	0.924
Very tired	0.942	0.952	0.947
Hypovigilance	0.946	0.935	0.940
Average	0.933	0.929	0.931

shows the results obtained during our second experiment. These evaluation results are satisfactory but still need to be improved because we have not yet learned our system on the two DEAP and MiraclHB databases.

In the following third and fourth experiments, we focus on the learning of our proposed model, taking into account the hyperparameters computed empirically by the second experiment described below, using the two databases MiraclHB and DEAP. The Precision, Recall, and F-measure all improved steadily over the course of the third and fourth experimentations (see

Table 4. Evaluation of the HyMobLSTM model using DEAP dataset.

Fatigue Level	Recall	Precision	F-Measure
Awake	0.987	0.988	0.987
A little tired	0.921	0.926	0.923
Tired	0.951	0.923	0.937
Very tired	0.962	0.951	0.956
Hypovigilance	0.989	0.994	0.991
Average	0.962	0.956	0.959

and

Table 5. Evaluation of the HyMobLSTM model using MiraclHB dataset.

Fatigue Level	Recall	Precision	F-Measure
Awake	0.987	0.988	0.987
A little tired	0.981	0.984	0.982
Tired	0.979	0.982	0.980
Very tired	0.984	0.981	0.982
Hypovigilance	0.989	0.983	0.986
Average	0.984	0.984	0.984

). This indicates that our model may be better trained with the classes distributed more similarly in the training and validation. Furthermore, it validates the efficacy of our optimization approach in enhancing both the training procedure and the trained model. Indeed, we find that the iris normalization has played an important role in the improvement in the feature extraction by MobileNetV3-large. The use of the deep LSTM network also made it possible to detect the hypovigilance states with more precision, because the classification findings are based on the study of the driver’s condition, taking into consideration both the present and prior states. Another advantage provided by this work is the multilevel fatigue detection. Indeed, our goal is to alert the driver before arriving at the last final fatigue level.

As can be seen in

Table 6. Comparison of our approach to others in the literature.

Authors	Accuracy	Data Type	Fatigue Levels
Our Work	0.984	Video	5
Akrout et al. [29]	0.979	Video	4
Hossain et al. [27]	0.981	Video	9
Long et al. [28]	0.9	Video	2
Arunasalam et al. [31]	N/A	Video+ECG	3
Xiao et al. [47]	0.991	Video	2
Shang et al. [48]	0.733	Video	4

, despite the relevance of the vigilance levels in the early identification of the hypovigilance situations, they are often overlooked in the literature. We also find that the author uses an invasive approach that requires sensors to be installed to determine the state of the driver fatigue [31]. The driver does not like to use invasive methods.

Indeed, when comparing our suggested HyMobLSTM model to other techniques in the literature, its higher accuracy (98.4%) stands out as a strong point in favor of our system. It is generally accepted that a variety of factors influence this value. Firstly, the MediaPipe Face Mesh was selected as the approach for the face and eye localization, in addition to the calculation of the 3D head pose angles $Ya$, $Pit$, and $Ro$, because it had very high success rates. Secondly, we normalized the iris using Daugman’s method to identify the optimal interest area to feed into the pretrained MobileNetV3-large. Third, the output properties of the MobileNetV3-large pretrained model were enhanced by using the calculated values $E_{Po1}$, $E_{Po2}$, $E_{br}$, $Ya$, $Pit$, and $Ro$. Finally, the experiments were also performed to select the optimal values for the hyperparameters through the Adam optimization algorithm. We also note that our system is characterized by the detection of fatigue at five levels. Indeed, this specification makes it possible to alert the driver of his hypovigilance state before he falls asleep. Hossain et al. [27] proposed nine classes. Indeed, the levels to be detected are levels of small distractions and not fatigue, which is not a major danger for drivers. Our work shows also robustness compared to Xiao et al. [47]. In fact, the work of Xiao et al. [47] reaches an accuracy of 0.991 for only two levels of vigilance. This remains limited in its use in the case where we want to detect hypovigilance states before they are triggered in order to prevent the driver from falling asleep and avoiding road accidents.

5. Conclusions and Future Works

In this work, an approach based on transfer learning and the deep LSTM network was proposed. We are indeed interested in detecting five levels of driver fatigue. Our work is distinguished by a pre-processing step that involves determining the regions of interest using the Google MediaPipe Face Mesh method as well as normalizing the desired features. This is manifested by the calculation of $E_{Po1}$, $E_{Po2}$, $E_{br}$, $Ya$, $Pit$, and $Ro$, as well as the concatenated features of the MobileNetV3-large model transfer learning result. The vectors obtained describe the position of the iris in relation to the eye corners; the blinking behavior; the features of the normalized iris area; and the yaw, pitch, and roll values of the head rotation angles. These vectors are then fed to the deep LSTM network of two hidden layers to classify the five desired fatigue levels: awake, a little tired, tired, very tired, and hypovigilance. We used the Adam optimization algorithm technique to select the most efficient hyperparameters from the three databases used in our experimentation (YawwDD, DEAP, and MiraclHB). The results obtained show the robustness of our proposed HyMobLSTM model compared to the existing work in the literature.

In future work, we hope to take changing light conditions (day and night, for example) into account so that they do not limit our system. The use of an infrared camera can be a possible solution for an evaluation of our system in difficult lighting conditions. The detection of severe states of distraction and the driver’s yawning are considered interesting objectives to improve our system in the future. The metrics that are based on vehicles seem to have a long-term perspective. In fact, a number of metrics, such as departures from the lane position, the steering wheel movement, and the accelerator pedal pressure, can be blended into behavioral measures in order to improve the driver fatigue detection.