RMDD: Raspberry Pi-Based Multimodal Dangerous Driving Behavior Detection

Abstract

With the continuous growth in motor vehicle ownership, traffic safety risks caused by dangerous driving behaviors remain an important concern. This paper presents RMDD, a Raspberry Pi-based multimodal dangerous driving monitoring feasibility prototype using YOLO26 visual models, personalized facial fatigue estimation, emotion2vec-based speech recognition, and hierarchical reliability-gated BFV fusion (HRG-BFV). HRG-BFV treats object detection as positive-only support for behavior evidence and scales its contribution by development-fold reliability, thereby avoiding hard rejection when the object detector fails. On locked module tests, behavior classification achieved 96.77% Top-1 accuracy, auxiliary detection reached 0.916 mAP@0.5, personalized facial calibration reduced the false-positive rate from 0.877 to 0.211, and speaker-disjoint speech recognition achieved 0.890 accuracy at 0 dB vehicle noise. The historical fixed BFV baseline achieved 0.601 balanced accuracy and 0.506 macro-F1 on P03–P08 (96 clips). In a stricter three-fold leave-one-external-participant-out evaluation on P04/P05/P08 (48 clips), HRG-BFV achieved 0.726 balanced accuracy, 0.562 macro-F1, 0.833 specificity, and 0.677 ROC-AUC, versus 0.655, 0.469, 0.833, and 0.595 for the historical frozen hard gate on the same cohort. Target support reliability was low (0.097–0.194), so the adaptive term appropriately reverted toward behavior evidence rather than producing an unsupported object detection gain. A sealed P08 behavior test exposed substantial cross-subject degradation (0.320 frame accuracy). A cooled 30 min Raspberry Pi 5 run completed without thermal throttling at 0.927 processing windows/s. The evidence supports controlled prototype feasibility but not population-level or on-road generalization.

1. Introduction

With the continued growth in China’s motor vehicle ownership, the risk of traffic accidents has increased significantly. Data from the China Statistical Yearbook show that traffic accidents and their associated casualties and property losses remain an important public-safety concern [1]. Therefore, researching efficient and accurate methods for detecting dangerous driving behaviors holds significant practical importance. Against the backdrop of intelligent connected vehicles and automated-driving technology, precise and timely monitoring of driver status is a critical factor in ensuring safety and comfort during human-machine co-driving.

Methods for detecting dangerous driving behaviors are primarily categorized into those based on visual features and those based on non-visual features [2]. Non-visual technologies include driver physiological signal analysis and vehicle parameter analysis. Physiological approaches identify driving states from indicators such as electroencephalogram and electrooculogram signals [3,4], but their sensing devices can be intrusive. Vehicle parameter analysis uses dynamic information such as steering wheel status [5], lane-keeping behavior [6], and driving speed or pedal-related variables [7]; its performance may vary with driving habits and road conditions. In contrast, vision-based methods monitor facial expressions, including eye closure and yawning, and body movements such as head posture or hand gestures. They are non-contact and practical, but remain sensitive to lighting, occlusion, and viewpoint changes [8].

As a supplement to visual analysis, speech emotion recognition (SER) evaluates driver emotion from acoustic characteristics such as pitch, energy, and Mel-frequency cepstral coefficients. Facial and speech cues provide different views of driver state, and negative emotions such as anger can impair driving cognition and control [9]. A multimodal design can therefore reduce dependence on any single sensing channel, provided that unavailable modalities are handled explicitly rather than treated as negative evidence.

The rapid development of deep learning has driven substantial progress in non-contact detection technology. Object detection algorithms are commonly divided into two-stage and single-stage families [10]. Single-stage methods such as YOLO offer advantages in processing speed and are widely used in driving-monitoring scenarios [11,12]. The present system updates the original YOLOv11 implementation to YOLO26 nano models: a ten-class behavior classifier supplies activity context, while a three-class phone/smoking/drinking detector provides auxiliary object evidence [13].

The Raspberry Pi is an established platform for embedded-system development because of its cost, interfaces, and software ecosystem [14]. This work updates the original Raspberry Pi 4B platform to Raspberry Pi 5 and evaluates the complete prototype on the actual target device [15]. The deployment uses separate visual, face, and voice workers so that incompatible software dependencies and bursty speech computation do not block the core monitoring loop.

However, current research and practical systems still face several issues:

(1)

In terms of hardware, edge-computing resources constrain the deployment of complex models, and sustained performance in dynamic driving scenarios remains difficult. Systems tied to specific vehicle sensors can also be difficult to integrate as general-purpose modules across different in-vehicle electronic architectures.

(2)

Functionally, many systems remain confined to visual analysis and lack integration with modalities such as speech, making it difficult to assess behavior, fatigue-related physiology, and emotion together. False alarms and missing-sensor conditions may further reduce user acceptance.

(3)

Regarding data application and feedback, many systems focus on real-time detection and early warning but provide limited integration of detection records. Without systematic reports or driver profiles, it is difficult for drivers to understand recurring risk patterns and receive targeted feedback.

(4)

Regarding long-duration operation, embedded systems must balance model complexity, latency, thermal stability, memory use, and power demand. The present work measures throughput, latency, CPU, memory, and temperature on Raspberry Pi 5; power consumption was not measured with an external meter and is not claimed as an experimental result.

To address these issues, this study makes the following contributions:

(1)

A Raspberry Pi 5 edge-computing prototype is developed for local monitoring of dangerous driving behavior. Its feasibility is assessed using locked module tests and a cooled 30-min complete-system stability run rather than inferred from platform specifications alone.

(2)

A three-dimensional perception framework integrating behavior, physiology, and emotion is retained from the original RMDD design. It combines YOLO26 behavior classification and auxiliary object detection, personalized facial fatigue estimation, and speech emotion recognition, with explicit reliability and availability handling.

(3)

Drivers are linked with calibration information, detection events, and personalized safety profiles. The implementation supports local visual feedback and structured event storage, providing a basis for retrospective analysis without claiming that the current feasibility study proves long-term behavior correction.

(4)

A hierarchical reliability-gated BFV rule is introduced. Within the visual branch, object evidence can strengthen an existing behavior signal but cannot veto it or independently create a dangerous behavior alert. A chance-corrected reliability term is estimated only from development participants, and the visual, face, and voice branches are then combined by missing-modality-aware noisy-OR [16] with causal confirmation. The design is evaluated by external-participant leave-one-out testing and explicit stress/same-subject supplementary analyses.

The remainder of this paper is organized as follows. Section 2 reviews related work on dangerous driving detection and speech emotion; Section 3 introduces the principal technologies used in the system; Section 4 presents the RMDD method and evaluation design; Section 5 reports the experimental results; Section 6 discusses the findings and limitations; and Section 7 concludes the paper.

2. Related Work

2.1. Overview of Related Research

Detection of dangerous driving behaviors in vehicles is an interdisciplinary research field whose effectiveness depends on the interaction among perception algorithms, computational hardware, and the in-vehicle environment. A successful system requires not only accurate algorithms but also attention to deployment feasibility, resource constraints, and timely response. Accordingly, the original RMDD study reviewed the problem at three levels: the in-vehicle hardware platform that determines implementation feasibility; visual methods that provide non-contact and timely monitoring; and emotion detection from voice features as a supplementary modality for assessing driver state.

2.2. Visual Feature-Based Methods for Detecting Dangerous Driving

Visual feature-based detection methods have gained wide application because of their real-time capability, low cost, and non-invasive nature [17]. Wang et al. proposed a cooperative method for distracted- and fatigued-driving detection using sequential image analysis and SVM classification, with an embedded implementation [18]. Liu et al. combined a convolutional neural network with long short-term memory to model facial appearance and temporal fatigue evolution [19]. Lu et al. fused ResNet image features with pointwise convolutions and evaluated driver action recognition on the State Farm dataset [20]. FatigueView subsequently provided a multi-camera benchmark covering subtle eye behavior and larger fatigue-related movements [8].

Research has continued to improve action recognition while considering the cost of deployment. Huang et al. proposed a deep deformable inverted residual network with attention for real-time driver behavior detection [21]. Li et al. combined multi-scale graph convolutions and spatiotemporal modeling to recognize distracted-driving actions from incomplete skeletons [22]. Kang et al. developed a ResNet-based driver-behavior recognition approach evaluated on public driver datasets [23]. YOLO-based studies have also emphasized efficient detection of small or safety-relevant targets [24,25]. These studies establish the visual foundation of RMDD, while also illustrating the continuing trade-off among accuracy, temporal context, robustness, and embedded computation.

2.3. Emotion Detection Based on Voice Features

Speech recognition and speech emotion analysis provide another route for driving-safety monitoring. Traditional SER methods use acoustic descriptors such as Mel-frequency cepstral coefficients, pitch, and energy [26]. Singh et al. combined modulation-spectral features with deep neural networks [27]. Zhu and Li proposed global-aware fusion over multi-scale feature representations [28]. Li et al. integrated global speech context with locally salient emotional segments for intelligent cockpit emotion detection [29]. Liu et al. used multi-scale, multi-channel feature extraction with a one-dimensional convolutional neural network [30]. Physiological emotion sensing provides a complementary direction: Long et al. used photonic seismocardiography and subject-invariant network features for cross-subject emotion recognition [31]. Such contact sensing may reduce dependence on visible behavior or speech, but it introduces additional hardware, motion artifact, and integration requirements not evaluated in RMDD. Speech can therefore supplement visual monitoring, while noise, speaker variation, intermittent speech, and model cost complicate continuous embedded use.

Existing research has therefore made substantial progress, but the compared approaches emphasize different trade-offs. Visual-only driver action systems are relatively direct to deploy but remain sensitive to viewpoint and subject shift; learned multimodal fusion can model interactions but requires sufficiently representative synchronized training data; physiological systems add less behavior-dependent signals at the cost of contact sensing or specialized hardware. RMDD does not claim general superiority over learned or uncertainty-aware fusion. Its methodological contribution is narrower: HRG-BFV separates correlated behavior/object evidence from cross-modal B/F/V evidence, treats missing target support as unknown rather than negative, and constrains face and voice roles through an auditable participant-held-out protocol suitable for the available edge prototype data.

3. Related Technologies

3.1. Raspberry Pi 5

The Raspberry Pi is a widely used edge-computing platform for localized inference in resource-constrained environments [14]. Local processing can reduce reliance on cloud connectivity and keep sensitive cabin video and audio on the device. The present system uses Raspberry Pi 5 with 8 GB memory, replacing the Raspberry Pi 4B platform described in the original manuscript [15].

At the hardware level, Raspberry Pi 5 provides a quad-core Arm Cortex-A76 processor and interfaces for camera, USB microphone, storage, networking, and display [15]. At the software level, the prototype uses a 64-bit Debian system with isolated Python environments: ONNX Runtime executes the two visual models, while separate Python 3.12 workers run MediaPipe face processing and the emotion2vec speech pipeline [32,33,34]. This separation preserves dependency compatibility and allows face and voice inference to operate asynchronously.

Raspberry Pi 5 is used here because it supports the required sensors and local software stack in a compact platform. Its suitability is assessed through measured latency, throughput, CPU, memory, temperature, and throttling behavior. No external power meter was used, so the paper does not convert the platform’s reputation for efficiency into an experimental power claim.

3.2. YOLO26 Models

YOLO is a single-stage model family designed for efficient end-to-end visual recognition [12]. The current RMDD implementation uses YOLO26 nano variants [13]. A YOLO26n classifier recognizes ten driver activities, including safe driving, phone calling, texting, radio adjustment, drinking, reaching behind, talking to a passenger, and smoking. A separate YOLO26n detector localizes phone, smoking, and drinking objects. The two outputs are not treated as interchangeable: classification provides scene-level activity context, whereas detection supplies corroborating object evidence.

3.3. Facial Physiological State Detection Technology

Facial feature-based physiological state detection captures driver images, analyzes the eyes, mouth, and head pose, and estimates fatigue-related conditions without attaching physiological sensors to the driver [2,17]. The original manuscript used a dlib 68-point model and fixed EAR/MAR thresholds. The updated system uses MediaPipe Face Landmarker [32] and retains interpretable geometric features, including EAR, MAR, pitch, yaw, and roll. Personal calibration, temporal aggregation, quality gating, and guarded online updates are added to reduce sensitivity to stable individual morphology, lighting, pose, and missing landmarks.

3.4. Voice Emotion Recognition Technology

Speech emotion recognition is an interdisciplinary technology that combines signal processing and machine learning to infer emotion from acoustic characteristics such as pitch, speech rate, energy, and spectral structure [26,27,28,29,30]. Driver emotion is relevant to safety because anger and inattention-related errors have been associated with crash involvement [35]. Speech nevertheless remains intermittent and should not be interpreted when no valid speech is present.

RMDD uses emotion2vec, a self-supervised speech representation model, to obtain utterance embeddings [33]. A lightweight classifier maps these embeddings to neutral, happy, angry, sad, and surprise. Unlike the original manuscript’s direct use of model labels, the updated pipeline is trained and evaluated on a speaker-disjoint Mandarin subset of ESD and is tested under controlled DEMAND vehicle-noise mixing. An energy-based voice activity detector marks silent windows as unavailable before fusion.

4. Materials and Methods

4.1. System Architecture

Figure 1 summarizes the processing path. Camera frames enter the behavior classification, auxiliary object, and face branches; microphone segments enter the voice branch only after energy-based voice activity detection (VAD). Each branch produces both a score and an availability flag. The fusion layer omits unavailable modalities rather than converting them to zeros. A two-of-three causal confirmation rule suppresses single-window spikes, and confirmed alerts are written to the local event/profile store and displayed by the PyQt interface.

4.2. Datasets, Splits, and Leakage Controls

Table 1. Datasets and evaluation controls.

Module	Data	Train/Validation/Test	Leakage Control
Behavior classification	Self-collected; 10 classes	3836/1070/711 legacy images; P08 external: 1192 frames, 30 videos	No exact hash overlap; sealed P08 subject test
Auxiliary detection	Self-collected; phone, smoking, drinking	1662/434/291 images; difficult negatives 451/117/76	No exact hash overlap; recording IDs unavailable
Facial fatigue	UTA-RLDD Fold1–Fold2 [36]	24 participants; 72 videos	Participant-group reporting
Speech emotion	ESD Mandarin [37]	6/2/2 speakers; 10,500/3500/3500 clips	Speaker-disjoint split
Fusion	Controlled B/F/V condition clips	Legacy: P01/P02 development, P03–P08 fixed test; HRG: P04/P05/P08 three-fold LOPO	P03 stress only; P06/P07 same-subject supplementary

separates the evidence used by each module. All reported test sets were locked before the corresponding final evaluation. Behavior classification used manually curated ten-class images extracted from the authors’ self-collected recordings; the revision additionally used P06/P07 for model development and sealed P08 as a one-time external participant test. Auxiliary detection used the original self-collected recordings and 644 difficult negatives. Facial fatigue used the official UTA-RLDD Fold1–Fold2 participant groups [36] (24 participants, 72 state videos). Voice evaluation used the ESD Mandarin subset [37] with a speaker-disjoint split and TCAR channel 1 from DEMAND [38]. For fusion, the historical P01/P02-frozen configuration was retained on P03–P08. The new HRG-BFV analysis used only the genuinely behavior-external P04/P05/P08 as a three-fold leave-one-participant-out cohort; P03 was reserved for stress ablation and P06/P07 for same-subject supplementary analysis.

A SHA-256 split audit accounted for all 2387 auxiliary-detection images and all 5617 behavior classification images. The 644 difficult negatives comprise 451 training, 117 validation, and 76 test images. No byte-identical image occurred across train, validation, or test, and no training/validation difficult-negative hash matched a test image. However, the legacy manifests do not retain video_id, session_id, or subject_id, so the audit cannot exclude adjacent or visually near-duplicate frames from the same recording. Accordingly, the revised manuscript does not claim recording- or driver-disjointness for the legacy image tests; the sealed P08 behavior evaluation is reported separately as subject-disjoint evidence.

4.3. Visual Behavior Classification

The classifier recognizes safe driving, left/right phone calling, left/right texting, radio adjustment, drinking, reaching behind, talking to a passenger, and smoking. YOLO26n-cls was trained for 80 epochs at 224 × 224 pixels with batch size 32 and random seed 0. Model selection used validation Top-1 accuracy; the selected checkpoint occurred at epoch 47 (validation Top-1 = 0.8832). The locked test set was evaluated once for the reported result. The checkpoint SHA-256 begins D38A and ends E907, providing a stable link between the manuscript result and archived weights.

4.4. Auxiliary Object Detection

The YOLO26n detector was trained for 100 epochs at 640 × 640 pixels with batch size 4, two workers, and random seed 0. Phone, smoking, and drinking are localized independently. The auxiliary output is not treated as a replacement for the behavior classifier. In fusion, the phone-related behavior score B′ is confirmed only when the classifier indicates a phone class and the auxiliary detector reports a phone at the current window or within a fold-tuned causal hold interval.

4.5. Personalized Facial Fatigue Estimation

MediaPipe Face Landmarker [32] supplies landmarks and a quality signal. Per-frame features include EAR, MAR, pitch, yaw, roll, and visibility/quality indicators. The initial alert-state baseline is the participant median. Initial thresholds were EAR_base × 0.72 (clipped to 0.05–0.60), MAR_base × 1.80 (clipped to 0.003–0.30), and pitch_base + 12°. Eligible alert frames update the baseline with an exponential moving average (α = 0.02), subject to rate limits. A 60 s temporal window aggregates PERCLOS-like eye closure, yawn frequency, and head-down ratio.

Five predefined variants isolate the effect of each design choice: M0 uses fixed thresholds and consecutive frames; M1 adds fixed 60 s temporal statistics; M2 adds personal alert-state calibration but keeps thresholds fixed; M3 adds slow online baseline updating; and M4 adds face-quality gating and the complete temporal fusion/failure-handling logic. M4 and M3 have identical clean aggregate metrics because their decision rules coincide when retained measurements are valid. A targeted deterministic stress test therefore injects a 15 s block of low-confidence but numerically plausible facial measurements into each of the 24 alert recordings and compares clean-control and corrupted runs.

4.6. Speech Emotion and Noise Robustness

The Mandarin subset of ESD [37] contains speakers 0001–0010 and five balanced emotions: neutral, happy, angry, sad, and surprise. Speakers 0001, 0002, 0003, 0006, 0007, and 0008 form the training set; 0004 and 0009 form validation; and 0005 and 0010 form the locked test. Each utterance is mapped to a 1024-dimensional emotion2vec_plus_large embedding [33]. A StandardScaler and multinomial logistic regression classifier were tuned over C ∈ {0.01, 0.1, 1, 10}; C = 0.01 was selected on validation and refitted on training plus validation data.

Noise robustness used TCAR channel 1 from DEMAND [38]. Each of the 3500 clean test clips was mixed at 20, 10, and 0 dB with fixed seed 20260727 and evaluated without retraining. At runtime, the voice branch is available only when energy VAD detects speech. The 0.4 cap was frozen as a conservative design bound using the legacy P01/P02 development data; it was not selected on P03–P08. Because the main alarm threshold is 0.65, voice alone cannot trigger a discrete alert at this cap. A post hoc sensitivity grid (0, 0.2, 0.4, 0.6, and 1.0) is reported without changing the submitted frozen configuration.

4.7. Hierarchical Reliability-Gated BFV Fusion

The historical fixed BFV rule is retained as an auditable comparator. For each available one-second window, B′ denotes phone-related behavior after the hard auxiliary object confirmation, F′ denotes the face risk appropriate to the experimental condition, and V′ denotes angry-speech probability after VAD. The historical v2 rule is:

$$R_{\mathrm{l}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{c}\mathrm{y}}=1-(1-B’)(1-F’)(1-0.4V’).$$

Unavailable terms are omitted from the product; if every modality is unavailable, the fused output is unavailable, and the confirmation history is cleared. An alert requires at least two threshold crossings among the last three available windows. For the controlled F = 1 condition, F′ was defined as clamp(eye_closed_fraction/0.35, 0, 1) so that the closed-eye ground truth was tested directly rather than through the broader fatigue score.

HRG-BFV first resolves the correlation between behavior classification and auxiliary object detection inside the visual branch. Let e_b(t) be the behavior risk evidence and e_o(t) the temporally smoothed phone support confidence. For each outer fold, object-support balanced accuracy BA_o is computed only on the development participants with participant-equal weighting, and chance-corrected reliability is defined as:

$${\rho }_o=\mathrm{m}\mathrm{a}\mathrm{x}(0,2{BA}_o-1).$$

The reliability quantity is target-support discrimination against the behavior condition, not bounding-box mAP. Object evidence is used only as positive corroboration after behavior exceeds a weak-evidence threshold τ_b:

$$E_B\left(t\right)=e_b\left(t\right)+I\left[e_b\right(t)\ge {\tau }_b]{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}{\rho }_o{e}_o\left(t\right)[1-e_b(t\left)\right].$$

Thus, missing target support never reduces e_b, and object evidence cannot create dangerous behavior risk when e_b is zero. The resulting visual evidence is combined with face and voice evidence as:

$$R_{\mathrm{H}\mathrm{R}\mathrm{G}}\left(t\right)=1-[1-E_B(t\left)\right][1-w_F{q}_F(t\left)E_F\right(t\left)\right][1-w_V{q}_V(t\left)E_V\right(t\left)\right].$$

Here q_F and q_V are availability gates. Missing modalities are omitted without renormalizing the others. The visual branch has unit weight; candidate parameters obey w_B ≥ w_F ≥ w_V, w_V < τ_alert ≤ w_F, 0 ≤ λ_max ≤ 0.5, and the same two-of-three causal confirmation. Consequently, voice alone cannot trigger the main alarm, whereas sufficiently strong face or visual evidence can. Phone threshold, object-memory length, τ_b, λ_max, w_F, w_V, and τ_alert are selected only from the two development participants in each outer fold by clip balanced accuracy, macro-F1, and specificity, with semantic-anchor proximity used only to break exact ties.

The original 32 condition videos from P01/P02 were retained as the source for freezing the historical deployment rule. The revision first processed P03–P08 without retuning: each participant contributed two A1 clips for each of the eight B/F/V states, giving 96 condition clips plus four clean calibration clips per participant. The historical configuration used a phone support threshold of 0.05, an 8 s causal hold, a 0.65 alarm threshold, and two-of-three temporal confirmation. For the new HRG-BFV analysis, P04/P05/P08 formed the primary external cohort because they were not used in behavior model development. Three outer folds held out one participant and used only the other two to estimate target reliability and select constrained fusion parameters. P03 was excluded from both tuning and primary testing and used only for stress ablation; P06/P07 were reported separately because their identities overlapped with behavior model development.

4.8. Raspberry Pi 5 Deployment and Evaluation

The final prototype used a Raspberry Pi 5 Model B Rev 1.1 (8 GB; Raspberry Pi Ltd., Cambridge, UK) running 64-bit Debian 13.5 and Python 3.13.5, a 5 MP OV5647 CSI camera module (Hunan Chuang Le Bo Intelligent Technology Co., Ltd., Changsha, Hunan, China; 65° field of view, without night-vision functionality), and a SADA MK21 omnidirectional USB microphone (Saida New Technology (Huizhou) Co., Ltd., Huizhou, Guangdong, China) sampled at 16 kHz. The physical components and sensing connections used in the deployment checks are documented photographically with the deployment results in Section 5.4. Separate Python environments ran ONNX Runtime 1.28.0 under Python 3.13.5 for the visual core, MediaPipe 0.10.15 under Python 3.12 for the face worker, and PyTorch 2.5.1, FunASR 1.3.22, and scikit-learn 1.9.0 under Python 3.12 for the speech worker. PyQt5 5.15.11 provided the local interface. The 30 min measurement used active cooling; enclosure, permanent vehicle wiring, and crash-qualified mounting were not evaluated, and the system is described as a bench/controlled-vehicle prototype rather than a completed in-vehicle product.

Two performance checks were retained. A 50-run visual-only benchmark characterized the two sequential ONNX models. The primary deployment measurement was a cooled 30 min complete-system run with 360 system samples and 1669 application windows. Sidecar profiling timestamped camera acquisition, JPEG decoding, dual-ONNX inference, asynchronous face- and voice-worker returns, audio capture, fusion, state publication, and the complete loop. Because the face and voice workers operate asynchronously, their round-trip times overlap the core loop and are not additive components of its latency. Power consumption was not measured with an external meter.

5. Results

5.1. Visual Modules

The behavior classifier achieved legacy locked-test Top-1 accuracy of 0.9677 and Top-5 accuracy of 1.000. The auxiliary detector achieved precision 0.884, recall 0.907, mAP@0.5 0.916, and mAP@0.5:0.95 0.659 on its legacy locked image test. Because the legacy manifests lack recording identifiers, these values are retained as image-level results rather than recording-disjoint estimates. The classifier and detector are not directly comparable because their labels, outputs, and metrics differ. The normalized confusion matrices and isolated-module results are presented in Figure 2 and

Table 2. Locked-test results for the isolated modules.

Module	Primary Metrics	Additional Detail
Behavior classifier	Top-1 0.9677; Top-5 1.000	10 classes; n = 711
Auxiliary detector	P 0.884; R 0.907; mAP50 0.916	mAP50–95 0.659; n = 291
Facial fatigue (M4)	Balanced acc. 0.7458; macro-F1 0.7058	FPR 0.2113; 24 participants
Speech emotion (clean)	Accuracy 0.9931; macro-F1 0.9931	UAR 0.9931; 3500 clips

, respectively.

On the sealed P08 external behavior set (1192 manually retained frames from 30 videos across all ten classes), the revision model obtained 0.320 frame accuracy, 0.236 frame macro-F1, 0.333 video accuracy, and 0.273 video macro-F1. This large decline is a negative but important result: the legacy image accuracy does not transfer to a new participant/capture domain, so broad behavior generalization claims were removed.

5.2. Facial Personalization and Speech Noise Robustness

Across all 24 UTA-RLDD participants, M0 produced balanced accuracy 0.5303, macro-F1 0.3942, false-positive rate 0.8769, and drowsy recall 0.9375. Personal calibration (M2) increased balanced accuracy to 0.7356 and reduced the false-positive rate to 0.2438. M3 and M4 both reached balanced accuracy 0.7458, macro-F1 0.7058, and false-positive rate 0.2113 on clean retained data. In the targeted low-confidence stress test, clean-control M3/M4 had the same post-block alert rate (0.129). Under injected invalid measurements, M3 performed 3328 invalid baseline updates, and its mean post-block alert rate rose to 0.851, whereas M4 made zero invalid updates and remained at 0.131. This synthetic test isolates the quality-gating mechanism; it is not a naturalistic corruption benchmark.

The clean speaker-disjoint speech test achieved accuracy 0.9931, macro-F1 0.9931, and UAR 0.9931. At 20 and 10 dB, accuracy remained 0.9809 and 0.9734. At 0 dB, it fell to 0.8900 (macro-F1 0.8872); happy recall was 0.6571, whereas angry recall remained 0.9971. Figure 3 places the face ablation and SNR response side by side.

5.3. Multimodal Fusion Feasibility

Table 3. Fixed-config external fusion ablation on P03–P08 (96 controlled condition clips).

Config.	Coverage	Recall	Spec.	Bal. Acc.	Macro-F1	ROC-AUC	Evaluation
B	1.000	0.083	1.000	0.542	0.196	0.548	96 clips
F	1.000	0.571	0.583	0.577	0.478	0.557	96 clips
V	0.573	0.000	1.000	0.500	0.035	0.764	55 clips
BF	1.000	0.619	0.583	0.601	0.506	0.561	96 clips
BV	1.000	0.083	1.000	0.542	0.196	0.774	96 clips
FV	1.000	0.571	0.583	0.577	0.478	0.568	96 clips
BFV	1.000	0.619	0.583	0.601	0.506	0.572	96 clips

reports the fixed-config external condition-level results for P03–P08 (96 clips). BF and BFV both achieved 0.6146 accuracy, 0.6012 balanced accuracy, and 0.5060 macro-F1. F/FV reached 0.5774 balanced accuracy and 0.4776 macro-F1, so the discrete improvement to BF/BFV came from B. VAD produced usable speech in 55 of 96 clips (57.3%); 567 of 2936 two-second voice windows passed VAD and 555 aligned to one-second fusion windows. Among the 48 V = 1 clips, 40 contained at least one VAD-passed window; 196 of 484 passed windows were classified as angry, and 36 clips contained at least one angry-labeled window. Angry evidence was therefore present but intermittent. Because 0.4V cannot cross the 0.65 alarm threshold by itself, the cap/threshold combination was the binding reason that V did not alter discrete alerts: BV equaled B, FV equaled F, and BFV equaled BF. Voice changed continuous ranking slightly (BF ROC-AUC 0.5610 versus BFV 0.5719) but did not establish an incremental alarm benefit.

Participant-level BFV performance remained heterogeneous (

Table 4. Participant-level BFV results on the external controlled evaluation.

Participant	Accuracy	Bal. Acc.	Macro-F1	Diagnostic Note
P03	0.875	0.500	0.467	All clips alerted
P04	0.313	0.607	0.310	Low recall
P05	0.688	0.607	0.543	Manual V1 timing complete
P06	0.625	0.357	0.385	Normal clips false-alarmed
P07	0.625	0.786	0.564	Highest balanced accuracy
P08	0.563	0.750	0.515	Face-driven; B did not trigger

and Figure 4b). P03 alerted on every clip, giving high accuracy but zero specificity; P06 had low balanced accuracy because normal clips were false alarms; P07 and P08 had the highest balanced accuracies, although P08’s result was entirely face-driven because B never triggered. The spread reinforces that this controlled external evaluation is not a population estimate.

Table 5. BFV sensitivity to the voice contribution cap on P03–P08.

Voice Cap	Accuracy	Recall	Specificity	Bal. Acc.	Macro-F1
0.0	0.615	0.619	0.583	0.601	0.506
0.2	0.615	0.619	0.583	0.601	0.506
0.4 (frozen)	0.615	0.619	0.583	0.601	0.506
0.6	0.625	0.631	0.583	0.607	0.513
1.0	0.646	0.655	0.583	0.619	0.528

gives the post hoc voice-cap sensitivity analysis. The frozen 0.4 result is unchanged from weights 0 and 0.2. Increasing the cap to 0.6 or 1.0 produces one or three additional positive clip detections, respectively, without improving specificity. Because these weights were examined after external labels were available, they are reported only as sensitivity values and were not used to replace the frozen configuration. In the P05 manually timed window analysis, all five weights produced identical event recall (0.625), mean first-alert delay (1.6 s), specificity (0.964), and false alarms (2.14 per normal minute).

Table 6. External-participant leave-one-out ablation of HRG-BFV on P04/P05/P08 (48 clips).

Method	Acc.	Recall	Spec.	Bal. Acc.	Macro-F1	ROC-AUC	AP
Historical frozen hard gate	0.521	0.476	0.833	0.655	0.469	0.595	0.895
Outer-fold hard gate	0.604	0.571	0.833	0.702	0.531	0.595	0.895
No object veto	0.646	0.619	0.833	0.726	0.562	0.603	0.896
Fixed object boost	0.646	0.619	0.833	0.726	0.562	0.603	0.896
Reliability object boost	0.646	0.619	0.833	0.726	0.562	0.603	0.896
Complete HRG-BFV	0.646	0.619	0.833	0.726	0.562	0.677	0.930

reports the participant-held-out HRG-BFV ablation on P04/P05/P08 (48 clips). The historical P01/P02-frozen hard gate is shown on the same cohort as a deployment reference; the remaining rows use three outer folds, with one external participant held out and all parameters estimated from the other two. Relative to the historical reference, complete HRG-BFV increased balanced accuracy from 0.655 to 0.726, macro-F1 from 0.469 to 0.562, recall from 0.476 to 0.619, and ROC-AUC from 0.595 to 0.677 while retaining 0.833 specificity. Against the hard gate under the same outer-fold protocol, the balanced-accuracy gain was 0.024, and the macro-F1 gain was 0.031.

The development-fold object-support BA_o ranged from 0.549 to 0.597, giving ρ_o = 0.097–0.194. Only 2 of 1421 held-out windows simultaneously satisfied behavior and object-enhancement conditions. Fixed and reliability-weighted object boosts therefore did not change the current binary decisions or score ranking. Complete HRG-BFV instead improved ROC-AUC/AP through the constrained cross-modal hierarchy while safely reverting toward behavior evidence when object support was unreliable. This result supports failure-aware degradation, not an independent accuracy gain from object weighting. P03 was used only as a stress case; P06/P07 results were retained only as same-subject supplementary evidence.

5.4. Edge Performance and Stability

The visual-only dual-ONNX benchmark required 215.2 ms on average (median 178.2 ms; P95 412.2 ms), corresponding to an estimated 4.65 visual inferences/s with 162.8 MiB peak RSS. During the complete cooled run, the integrated dual-ONNX stage required 175.7 ms on average (median 169.1 ms; P95 196.3 ms), camera/file acquisition required 859.5 ms, JPEG decoding required 20.7 ms, and fusion required 0.091 ms. State publication had a 0.321 ms median and 0.411 ms P95; its 3.633 ms mean reflects one isolated 5.17 s file-write outlier. The complete application loop required 1076.4 ms on average (median 1068.8 ms; P95 1171.7 ms), corresponding to 0.927 windows/s. Face-worker return, 2 s audio capture, and voice-worker return averaged 1077.6, 2200.8, and 918.2 ms, respectively, but operated asynchronously and therefore must not be summed into the core-loop latency. These values characterize application windows rather than camera-frame throughput.

During the 1800.4 s run, mean temperature was 46.01 °C and the maximum was 52.4 °C; every throttling flag was 0x0. System CPU averaged 22.16% (P95 36.69%; maximum 59.46%). Median available memory was 5.99 GiB. The cold-start minimum was 2.39 GiB while the speech model loaded; after the first 60 s, available memory remained between 5.89 and 6.04 GiB. Core + face + voice RSS had a median of 1.48 GiB, a cold-start peak of 4.88 GiB, and a post-warm-up maximum of 1.50 GiB. Post-warm-up available memory changed by +17.9 MiB from first to last sample, and no out-of-memory event occurred. Figure 5 shows the logged traces. The physical prototype is shown in Figure 6, and the complete deployment measurements are summarized in

Table 7. Raspberry Pi 5 complete-system deployment measurements.

Category	Metric	Result
Run	Duration/samples/windows	1800.4 s/360/1669
Application	Throughput	0.927 windows/s
Sensing	Camera capture mean/median/P95	859.5/864.6/960.4 ms
Preprocess	JPEG decode mean/median/P95	20.7/19.0/28.6 ms
Visual	Dual ONNX mean/median/P95	175.7/169.1/196.3 ms
Face (async)	Worker return mean/median/P95	1077.6/1072.0/1172.6 ms
Audio (async)	2 s capture mean/median/P95	2200.8/2135.9/3005.2 ms
Voice (async)	Worker return mean/median/P95	918.2/1054.0/1282.6 ms
Fusion	Mean/median/P95	0.091/0.082/0.109 ms
State publication	Mean/median/P95	3.633/0.321/0.411 ms
Full loop	Mean/median/P95	1076.4/1068.8/1171.7 ms
Thermal	Mean/maximum/throttling	46.01/52.4 °C/none
CPU	System mean/P95/maximum	22.16/36.69/59.46%
Memory	Available median/cold-start min/steady min	5.99/2.39/5.89 GiB
Workers	RSS median/cold-start peak/steady max	1.48/4.88/1.50 GiB

.

6. Discussion

6.1. Main Findings

The strongest result is not a single global accuracy number, but the clearer evidence boundary produced by the revision. The legacy image tests remain useful module checks, yet the sealed P08 behavior result shows severe cross-subject/capture degradation. The face ablation and targeted invalid-measurement test show that calibration improves clean performance and that M4 specifically prevents low-confidence baseline corruption. The speech model remains strong on curated speaker-disjoint clips, but speech is intermittent in controlled driver videos.

Fusion results show why isolated module accuracy cannot be substituted for system evidence. In the historical fixed evaluation on P03–P08, F supplied most discrete detections; phone behavior evidence was strongly participant-dependent and was further suppressed by the old hard object gate. Voice was available in only 57.3% of clips and could not cross the alarm threshold at the frozen 0.4 cap. In the stricter P04/P05/P08 external-participant leave-one-out analysis, removing the object veto improved balanced accuracy without reducing specificity, and complete HRG-BFV further increased ROC-AUC from 0.603 to 0.677 relative to the no-veto rule. The target-support reliability was only 0.097–0.194 and joint behavior/object enhancement occurred in 2/1421 held-out windows; therefore, the defensible innovation is a reliability-controlled fallback that prevents weak object evidence from damaging behavior evidence, not a claim that dynamic object weighting independently improved accuracy.

The edge results also separate feasibility from real-time marketing language. A complete application window was processed approximately every 1.15 s, with stable temperature and memory. That rate is usable for slowly evolving driver states and confirmed events, but it is not equivalent to high-frame-rate perception. The asynchronous design prevented the heavier speech worker from blocking the core loop, at the cost of heterogeneous environments and additional integration complexity.

6.2. Limitations and Validity Boundaries

The fusion evidence covers eight participants in total, but their roles must remain distinct: P01/P02 support historical development, P04/P05/P08 form the three-person external HRG-BFV leave-one-out cohort, P03 is stress-only, and P06/P07 are same-subject supplementary cases. The primary HRG-BFV estimate therefore contains only 48 clips from three external participants. This controlled stationary vehicle sample does not cover road motion, illumination extremes, camera displacement, spontaneous speech, broad traffic acoustics, or safety-critical response under real driving. It supports participant-held-out feasibility evidence, not population-level precision or statistical superiority.

The module datasets also differ from the deployment domain. The SHA-256 audit found no exact cross-split image duplicates, but legacy image manifests cannot establish recording/session independence. The sealed P08 behavior result directly demonstrates domain shift. UTA-RLDD evaluation uses Fold1–Fold2 rather than a full five-fold protocol; the M3/M4 stress test is synthetic rather than naturalistic. ESD contains acted speech, and one DEMAND vehicle-noise channel does not reproduce reverberation, competing speakers, or microphone nonlinearity. Hardware records identify the OV5647 camera, SADA MK21 microphone, and audio format, but not the enclosure, permanent wiring, or qualified vehicle mounting.

Finally, the ONNX agreement verifies deployment format but does not replace the locked PyTorch detector result, because its recall differed by approximately 0.022 under the export evaluation. The 30 min stability run does not establish long-term durability, and no external power meter was used. Any claim of low power would therefore be based on platform reputation rather than measurement and is deliberately omitted.

7. Conclusions

This paper addresses limitations of dangerous driving monitoring, including single-modal sensing, susceptibility to environmental interference, limited feedback, and the difficulty of deploying multiple models on an embedded platform. It retains the original RMDD behavior–physiology–emotion framework while replacing outdated models, datasets, and hardware claims with a reproducible Raspberry Pi 5 implementation and evidence from locked evaluations.

The visual branch now combines a ten-class YOLO26n behavior classifier with a three-class auxiliary object detector. On their locked tests, the classifier achieved 0.9677 Top-1 accuracy, and the detector achieved 0.916 mAP@0.5. The facial branch retained interpretable EAR, MAR, and head-pose cues while adding personal calibration: across 24 UTA-RLDD participants, the false-positive rate decreased from 0.8769 for the fixed-threshold baseline to 0.2113 for the online personalized variants.

The speech branch preserved the original use of emotion information but implemented it through speaker-disjoint emotion2vec embeddings and a lightweight classifier. Clean ESD Mandarin accuracy reached 0.9931 and remained 0.8900 under 0 dB TCAR noise. The historical fixed BFV evaluation on P03–P08 achieved 0.601 balanced accuracy and 0.506 macro-F1. In the external-participant P04/P05/P08 analysis, HRG-BFV achieved 0.726 balanced accuracy, 0.562 macro-F1, 0.833 specificity, and 0.677 ROC-AUC. Its positive-only target support and chance-corrected reliability prevented object misses from vetoing behavior evidence, while the constrained hierarchy preserved face as a secondary main-alarm modality and voice as supplementary evidence.

The prototype also preserves the original goals of local feedback and personalized driver records. Its database and interface can associate calibration data, events, and decisions with a driver profile, but the current experiments do not establish long-term behavior correction or reduced accident incidence.

Raspberry Pi 5 completed the cooled 30-min run without thermal throttling, at 0.927 processing windows/s and 1.076 s mean complete-loop latency. This demonstrates technical feasibility for slowly evolving driver states and confirmed events, but it is not equivalent to high-frame-rate perception. Power consumption was not measured and is not presented as an experimental benefit.

Limitations include legacy image-level split metadata, cross-subject behavior degradation, only three external participants in the primary HRG-BFV evaluation, stationary conditions, acted speech, one noise recording, low target-support reliability, incomplete hardware records, and no external power measurement. The equality of binary results for no-veto, fixed-boost, reliability-boost, and complete HRG-BFV means that the present data demonstrate safe fallback and better score ranking, not an independent object-weighting accuracy gain. Future work should use source-grouped data, preregistered multi-driver testing, externally annotated target boxes, naturalistic speech and missing-modality faults, model-isolated worker profiling, qualified mounting, and external energy instrumentation.