PMMCT: A Parallel Multimodal CNN-Transformer Model to Detect Slow Eye Movement for Recognizing Driver Sleepiness

Sleepiness at the wheel is an important contributor to road traffic accidents. Slow eye movement (SEM) serves as a reliable physiological indicator for the sleep onset period (SOP). To detect SEM for recognizing drivers’ SOP, a Parallel Multimodal CNN-Transformer (PMMCT) model is proposed. The model employs two parallel feature extraction modules to process bimodal signals, each comprising convolutional layers and Transformer encoder layers. The extracted features are fused and then classified using fully connected layers. The model is evaluated on two bimodal signal combinations HEOG + O2 and HEOG + HSUM, where HSUM is the sum of two single-channel horizontal electrooculogram (HEOG) signals and captures electroencephalograph (EEG) features similar to those in the conventional O2 channel. Experimental results indicate that using the PMMCT model, the HEOG + HSUM combination performs comparably to the HEOG + O2 combination and outperforms unimodal HEOG by 2.73% in F1-score, with average classification accuracy and F1-score of 99.89% and 99.35%, outperforming CNN, CNN-LSTM, and CNN-LSTM-Attention models. The model exhibits minimal false positives and false negatives, with average values of 5.2 and 0.8. By combining CNNs’ local feature extraction with Transformers’ global temporal modeling, and using only two HEOG electrodes, the system offers superior performance while enhancing wearable device comfort for real-world applications.