Search Results (29)

In distance learning environments, learner engagement directly impacts attention, motivation, and academic performance. Signs of fatigue, negative affect, or critical remarks can warn of growing disengagement and potential dropout. However, most existing approaches rely on a single modality, visual or text-based, without providing a general view of learners’ cognitive and affective states. We propose a multimodal system that integrates three complementary analyzes: (1) a CNN-LSTM model augmented with warning signs such as PERCLOS and yawning frequency for fatigue detection, (2) facial emotion recognition by EmoNet and an LSTM to handle temporal dynamics, and (3) sentiment analysis of feedback by a fine-tuned BERT model. It was evaluated on three public benchmarks: DAiSEE for fatigue, AffectNet for emotion, and MOOC Review (Coursera) for sentiment analysis. The results show a precision of 88.5% for fatigue detection, 70% for emotion detection, and 91.5% for sentiment analysis. Aggregating these cues enables an accurate identification of disengagement periods and triggers individualized pedagogical interventions. These results, although based on independently sourced datasets, demonstrate the feasibility of an integrated approach to detecting disengagement and open the door to emotionally intelligent learning systems with potential for future work in real-time content personalization and adaptive learning assistance. Full article

Drowsy driving contributes to over 6000 fatal incidents annually in the US, underscoring the need for effective, non-intrusive drowsiness detection. This study seeks to address detection challenges, particularly in non-standard head positions. Our innovative approach leverages computer vision by combining facial feature detection using Dlib, head pose estimation with the HOPEnet model, and analyses of the percentage of eyelid closure over time (PERCLOS) and the percentage of mouth opening over time (POM). These are integrated with traditional machine learning models, such as Support Vector Machines, Random Forests, and XGBoost. These models were chosen for their ability to process detailed information from facial landmarks, head poses, PERCLOS, and POM. They achieved a high overall accuracy of 86.848% in detecting drowsiness, with a small overall model size of 5.05 MB and increased computational efficiency. The models were trained on the National Tsing Hua University Driver Drowsiness Detection Dataset, making them highly suitable for devices with a limited computational capacity. Compared to the baseline model from the literature, which achieved an accuracy of 84.82% and a larger overall model size of 37.82 MB, the method proposed in this research shows a notable improvement in the efficiency of the model with relatively similar accuracy. These findings provide a framework for future studies, potentially improving sleepiness detection systems and ultimately saving lives by enhancing road safety. Full article

Advanced Driver Assistance Systems, such as Forward Collision Warning and Lane Departure Warning, play a crucial role in accident prevention by alerting drivers to potential hazards. With the advent of fully autonomous driving technology that requires no driver input, there is now a greater emphasis on monitoring the state of vehicle occupants. This is particularly important because, in emergency situations where control must suddenly be transferred to an unprepared occupant, the risk of accidents increases significantly. To mitigate this risk, new monitoring technologies are being developed to analyze driver behavior and detect states of inattention or drowsiness. In response to the emerging demands of driver monitoring technology, we have developed the Customized Driver Inattention Detection Model (CDIDM). This model employs video analysis and statistical techniques to accurately and rapidly classify information on drivers’ gazes. The CDIDM framework defines the components of inattentive or drowsy driving based on the Driver Monitoring System (DMS) safety standards set by the European New Car Assessment Programme (EuroNCAP). By defining six driving behavior-related scenarios, we have improved the accuracy of driver inattention assessment. The CDIDM estimates the driver’s gaze while simultaneously analyzing data in real-time. To minimize computational resource usage, this model incorporates a series of preprocessing steps that facilitate efficient time series data analysis, utilizing techniques such as DTW Barycenter Averaging (DBA) and K-means clustering. This results in a robust driver attention monitoring model based on time series classification. Full article

Individuals with obstructive sleep apnea (OSA) face increased accident risks due to excessive daytime sleepiness. PERCLOS, a recognized drowsiness detection method, encounters challenges from image quality, eyewear interference, and lighting variations, impacting its performance, and requiring validation through physiological signals. We propose visual-based scoring using adaptive thresholding for eye aspect ratio with OpenCV for face detection and Dlib for eye detection from video recordings. This technique identified 453 drowsiness (PERCLOS ≥ 0.3 || CLOSDUR ≥ 2 s) and 474 wakefulness episodes (PERCLOS < 0.3 and CLOSDUR < 2 s) among fifty OSA drivers in a 50 min driving simulation while wearing six-channel EEG electrodes. Applying discrete wavelet transform, we derived ten EEG features, correlated them with visual-based episodes using various criteria, and assessed the sensitivity of brain regions and individual EEG channels. Among these features, theta–alpha-ratio exhibited robust mapping (94.7%) with visual-based scoring, followed by delta–alpha-ratio (87.2%) and delta–theta-ratio (86.7%). Frontal area (86.4%) and channel F4 (75.4%) aligned most episodes with theta–alpha-ratio, while frontal, and occipital regions, particularly channels F4 and O2, displayed superior alignment across multiple features. Adding frontal or occipital channels could correlate all episodes with EEG patterns, reducing hardware needs. Our work could potentially enhance real-time drowsiness detection reliability and assess fitness to drive in OSA drivers. Full article

Existing vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can be affected by factors like varying lighting conditions and motion. To address these challenges, we propose a non-invasive method for multi-modal fusion fatigue detection called RPPMT-CNN-BiLSTM. This method incorporates a feature extraction enhancement module based on the improved Pan–Tompkins algorithm and 1D-MTCNN. This enhances the accuracy of heart rate signal extraction and eyelid features. Furthermore, we use one-dimensional neural networks to construct two models based on heart rate and PERCLOS values, forming a fatigue detection model. To enhance the robustness and accuracy of fatigue detection, the trained model data results are input into the BiLSTM network. This generates a time-fitting relationship between the data extracted from the CNN, allowing for effective dynamic modeling and achieving multi-modal fusion fatigue detection. Numerous experiments validate the effectiveness of the proposed method, achieving an accuracy of 98.2% on the self-made MDAD (Multi-Modal Driver Alertness Dataset). This underscores the feasibility of the algorithm. In comparison with traditional methods, our approach demonstrates higher accuracy and positively contributes to maintaining traffic safety, thereby advancing the field of smart transportation. Full article

In order to determine the fatigue state of air traffic controllers from air talk, an algorithm is proposed for discriminating the fatigue state of controllers based on applying multi-speech feature fusion to voice data using a Fuzzy Support Vector Machine (FSVM). To supplement the basis for discrimination, we also extracted eye-fatigue-state discrimination features based on Percentage of Eyelid Closure Duration (PERCLOS) eye data. To merge the two classes of discrimination results, a new controller fatigue-state evaluation index based on the entropy weight method is proposed, based on a decision-level fusion of fatigue discrimination results for speech and the eyes. The experimental results show that the fatigue-state recognition accuracy rate was 86.0% for the fatigue state evaluation index, which was 3.5% and 2.2%higher than those for speech and eye assessments, respectively. The comprehensive fatigue evaluation index provides important reference values for controller scheduling and mental-state evaluations. Full article

In response to the problem of high computational and parameter requirements of fatigued-driving detection models, as well as weak facial-feature keypoint extraction capability, this paper proposes a lightweight and real-time fatigued-driving detection model based on an improved YOLOv5s and Attention Mesh 3D keypoint extraction method. The main strategies are as follows: (1) Using Shufflenetv2_BD to reconstruct the Backbone network to reduce parameter complexity and computational load. (2) Introducing and improving the fusion method of the Cross-scale Aggregation Module (CAM) between the Backbone and Neck networks to reduce information loss in shallow features of closed-eyes and closed-mouth categories. (3) Building a lightweight Context Information Fusion Module by combining the Efficient Multi-Scale Module (EAM) and Depthwise Over-Parameterized Convolution (DoConv) to enhance the Neck network’s ability to extract facial features. (4) Redefining the loss function using Wise-IoU (WIoU) to accelerate model convergence. Finally, the fatigued-driving detection model is constructed by combining the classification detection results with the thresholds of continuous closed-eye frames, continuous yawning frames, and PERCLOS (Percentage of Eyelid Closure over the Pupil over Time) of eyes and mouth. Under the premise that the number of parameters and the size of the baseline model are reduced by 58% and 56.3%, respectively, and the floating point computation is only 5.9 GFLOPs, the average accuracy of the baseline model is increased by 1%, and the Fatigued-recognition rate is 96.3%, which proves that the proposed algorithm can achieve accurate and stable real-time detection while lightweight. It provides strong support for the lightweight deployment of vehicle terminals. Full article

Despite sleepiness being considered one of the main factors contributing to road crashes, and even though extensive efforts have been made in the identification of techniques able to detect it, the assessment of fitness-to-drive regarding driving fatigue and sleepiness is still an open issue. In the literature on driver sleepiness, both vehicle-based measures and behavioral measures are used. Concerning the former, the one considered more reliable is the Standard Deviation of Lateral Position (SDLP) while the PERcent of eye CLOSure over a defined period of time (PERCLOS) seems to be the most informative behavioral measure. In the present study, using a within-subject design, we assessed the effect of a single night of partial sleep deprivation (PSD, less than 5 h sleeping time) compared to a control condition (full night of sleep, 8 h sleeping time) on SDLP and PERCLOS, in young adults driving in a dynamic car simulator. Results show that time-on-task and PSD affect both subjective and objective sleepiness measures. Moreover, our data confirm that both objective and subjective sleepiness increase through a monotonous driving scenario. Considering that SDLP and PERCLOS were often used separately in studies on driver sleepiness and fatigue detection, the present results have potential implications for fitness-to-drive assessment in that they provide useful information allowing to combine the advantages of the two measures for drowsiness detection while driving. Full article

Fatigue affects operators’ safe operation in a nuclear power plant’s (NPP) main control room (MCR). An accurate and rapid detection of operators’ fatigue status is significant to safe operation. The purpose of the study is to explore a way to detect operator fatigue using trends in eyes’ blink rate, number of frames closed in a specified time (PERCLOS) and mouse velocity changes of operators. In experimental tasks of simulating operations, the clustering method of Toeplitz Inverse Covariance-Based Clustering (TICC) is used for the relevant data captured by non-invasive techniques to determine fatigue levels. Based on the determined results, the data samples are given labeled fatigue levels. Then, the data of fatigue samples with different levels are identified using supervised learning techniques. Supervised learning is used to classify different fatigue levels of operators. According to the supervised learning algorithm in different time windows (20 s–60 s), different time steps (10 s–50 s) and different feature sets (eye, mouse, eye-plus-mouse) classification performance show that K-Nearest Neighbor (KNN) perform the best in the combination of the above multiple indexes. It has an accuracy rate of 91.83%. The proposed technique can detect operators’ fatigue level in real time within 10 s. Full article

Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving. Full article

Many factors influence the fatigue state of human beings, and fatigue has a significant adverse effect on the health and safety of the haulage operators in the mine. Among various fatigue monitoring systems in mine operations, currently, the Percentage of Eye Closure (PERCLOS) is common. However, work and other environmental factors influence the fatigue state of haul truck drivers; PERCLOS systems do not consider these factors in their modeling of fatigue. Therefore, modeling work and environmental factors’ impact on individual operations fatigue state could yield interesting insights into managing fatigue. This study provides an approach of using operational data sets to find the leading indicators of the operators’ fatigue. A machine learning algorithm is used to model the fatigue of the individual. eXtreme Gradient Boosting (XGBoost) algorithm is chosen for this model because of its efficiency, accuracy, and feasibility, which integrates multiple tree models and has stronger interpretability. A significant number of negative and positive samples are created from the available data to increase the number of datasets. Then, the results are compared with other existing models. A selected algorithm, along with a big data set was able to create a comprehensive model. The model was able to find the importance of the individual factors along with work and environmental factors among operational data sets. Full article

Accidents caused by fatigue occur frequently, and numerous scholars have devoted tremendous efforts to investigate methods to reduce accidents caused by fatigued driving. Accordingly, the assessment of the spirit status of the driver through the eyes blinking frequency and the measurement of physiological signals have emerged as effective methods. In this study, a drowsiness detection system is proposed to combine the detection of LF/HF ratio from heart rate variability (HRV) of photoplethysmographic imaging (PPGI) and percentage of eyelid closure over the pupil over time (PERCLOS), and to utilize the advantages of both methods to improve the accuracy and robustness of drowsiness detection. The proposed algorithm performs three functions, including LF/HF ratio from HRV status judgment, eye state detection, and drowsiness judgment. In addition, this study utilized a near-infrared webcam to obtain a facial image to achieve non-contact measurement, alleviate the inconvenience of using a contact wearable device, and for use in a dark environment. Furthermore, we selected the appropriate RGB channel under different light sources to obtain LF/HF ratio from HRV of PPGI. The main drowsiness judgment basis of the proposed drowsiness detection system is the use of algorithm to obtain sympathetic/parasympathetic nervous balance index and percentage of eyelid closure. In the experiment, there are 10 awake samples and 30 sleepy samples. The sensitivity is 88.9%, the specificity is 93.5%, the positive predictive value is 80%, and the system accuracy is 92.5%. In addition, an electroencephalography signal was used as a contrast to validate the reliability of the proposed method. Full article

Fatigue driving has always received a lot of attention, but few studies have focused on the fact that human fatigue is a cumulative process over time, and there are no models available to reflect this phenomenon. Furthermore, the problem of incorrect detection due to facial expression is still not well addressed. In this article, a model based on BP neural network and time cumulative effect was proposed to solve these problems. Experimental data were used to carry out this work and validate the proposed method. Firstly, the Adaboost algorithm was applied to detect faces, and the Kalman filter algorithm was used to trace the face movement. Then, a cascade regression tree-based method was used to detect the 68 facial landmarks and an improved method combining key points and image processing was adopted to calculate the eye aspect ratio (EAR). After that, a BP neural network model was developed and trained by selecting three characteristics: the longest period of continuous eye closure, number of yawns, and percentage of eye closure time (PERCLOS), and then the detection results without and with facial expressions were discussed and analyzed. Finally, by introducing the Sigmoid function, a fatigue detection model considering the time accumulation effect was established, and the drivers’ fatigue state was identified segment by segment through the recorded video. Compared with the traditional BP neural network model, the detection accuracies of the proposed model without and with facial expressions increased by 3.3% and 8.4%, respectively. The number of incorrect detections in the awake state also decreased obviously. The experimental results show that the proposed model can effectively filter out incorrect detections caused by facial expressions and truly reflect that driver fatigue is a time accumulating process. Full article

This paper describes a real-time fatigue sensing and enhanced feedback system designed for video terminal operating groups. This paper analyzes the advantages and disadvantages of various current acquisition devices and various algorithms for fatigue perception. After comparison, this study uses an eye movement instrument to collect user PERCLOS, and then calculates and determines the user’s fatigue state. A detailed fatigue discrimination calculation method is provided in this paper. The fatigue level is divided into three levels: mild fatigue, moderate fatigue and severe fatigue. Finally, this study uses the fatigue method demonstrated above to achieve real-time discrimination of the fatigue level of the user in front of the video operation terminal. This paper elaborates a method for waking up users and enhancing feedback based on their fatigue level and the importance of information. This study provides a solution for avoiding the operational risks caused by fatigue and lays the foundation for the machine to sense the user and provide different service solutions based on the user’s status. Full article

This study explores how drivers are affected by automation when driving in rested and fatigued conditions. Eighty-nine drivers (45 females, 44 males) aged between 20 and 85 years attended driving experiments on separate days, once in a rested and once in a fatigued condition, in a counterbalanced order. The results show an overall effect of automation to significantly reduce drivers’ workload and effort. The automation had different effects, depending on the drivers’ conditions. Differences between the manual and automated mode were larger for the perceived time pressure and effort in the fatigued condition as compared to the rested condition. Frustration was higher during manual driving when fatigued, but also higher during automated driving when rested. Subjective fatigue and the percentage of eye closure (PERCLOS) were higher in the automated mode compared to manual driving mode. PERCLOS differences between the automated and manual mode were higher in the fatigued condition than in the rested condition. There was a significant interaction effect of age and automation on drivers’ PERCLOS. These results are important for the development of driver-centered automation because they show different benefits for drivers of different ages, depending on their condition (fatigued or rested). Full article