Towards Sustainable Safe Driving: A Multimodal Fusion Method for Risk Level Recognition in Distracted Driving Status

Abstract

Precise driving status recognition is a prerequisite for human–vehicle collaborative driving systems towards sustainable road safety. In this study, a simulated driving platform was built to capture multimodal information simultaneously, including vision-modal data representing driver behaviour and sensor-modal data representing vehicle motion. Multisource data are used to quantify the risk of distracted driving status from four levels, safe driving, slight risk, moderate risk, and severe risk, rather than detecting action categories. A multimodal fusion method called vision-sensor fusion transformer (V-SFT) was proposed to incorporate the vision-modal of driver behaviour and sensor-modal data of vehicle motion. Feature concatenation was employed to aggregate representations of different modalities. Then, successive internal interactions were performed to consider the spatiotemporal dependency. Finally, the representations were clipped and mapped into four risk level label spaces. The proposed approach was evaluated under different modality inputs on the collected datasets and compared with some baseline methods. The results showed that V-SFT achieved the best performance with an recognition accuracy of 92.0%. It also indicates that fusing multimodal information effectively improves driving status understanding, and V-SFT extensibility is conducive to integrating more modal data.

1. Introduction

In the human-machine codriving system, the driver and the intelligent algorithm cooperate and restrict each other to jointly control the vehicle. Accurately identifying the driver’s current driving status or risk is an important basis for the intelligent vehicle system to allocate or switch control rights [1]. Many factors can affect a driver’s driving status, such as alcohol, fatigue, drugs, negative emotions, and lack of concentration, which may significantly reduce the driver’s driving ability and increase potential driving risks. Risky driving not only brings traffic safety hazards [2], but its fuel efficiency may also have a negative impact on environmental protection and sustainability.

Distracted driving and road safety. Distraction is an important cause of risky driving status. The latest data from the National Highway Transportation and Safety Administration (NHTSA) show that approximately 3142 people were killed by distracted driving in 2020, 3119 in 2019, 2628 deaths in 2018, and 3003 deaths in 2017 [3]. Rahman et al. [4] and Sayed et al. [5] focused on studying driving behaviours to precisely anticipate, prevent, and manage road safety programs. Questionnaire measuring [6] and a self-reported survey [7] revealed that risky driving behaviors were associated with traffic safety both directly and indirectly. Compared with risky driving statuses, such as fatigue or drunk driving, distracted driving has a shorter duration, a more unstable time-varying characteristic, and is more susceptible to the influence of objective conditions [8]. Existing studies have explored distracted driving statuses under the influence of different factors. Cognitive driving distraction occurs when the driver’s attention is subjected to some mental burden or when the driver’s attention is on something else, such as carrying on a conversation with a passenger and getting caught up in memories or thoughts [9]. Visual driving distractions can be triggered by anything that takes the driver’s focus away from the driving direction; it can be a navigation screen, a mobile phone, or a street sign [10]. Operational driving distractions refer to the driver performing operations unrelated to driving while controlling the vehicle. Typical actions include adjusting onboard equipment, eating, and editing messages [11]. In a real driving process, distracted driving status is rarely of just one form, but a combination of several types [12]. Existing distracted driving status recognition (DDSR) research is mainly carried out based on data reflecting driver behaviour or vehicle motion states.

Methods based on vision-modal data of driver behaviour. With the development of deep learning algorithms, computer vision technology has become popular because of its high accuracy [13]. Additionally, since visual modality data are collected through a camera, there is little interference with the driver while driving; thus, most of the work is carried out based on this scenario. Craye et al. [14] used machine learning methods such as AdaBoost and the hidden Markov model to judge eye, hand, and head movements while driving based on RGB-D map data captured by an active sensor Kinect. In [15], a novel multi-stream long short-term memory (M-LSTM) network was presented by Behera et al. for recognizing driver activities and transforming action information into semantic information. In other works, Eraqi et al. [16] used a genetically weighted ensemble of convolutional neural networks to obtain distraction detection confidence while combining facial and hand motion segmentation. Xing et al. [17] extracted the driver’s body from the input background using the Gaussian mixture model (GMM) to recognize actions such as checking the rearview mirror, adjusting a device, and using a mobile phone. Although distracted driving behaviour in some studies was divided into multiple action categories, it essentially includes only two categories: distracted and non-distracted. Such work based on only single-modal visual data is largely limited by its data quality. For example, an occluded driver’s body or altered light may reduce accuracy and robustness. In addition, statuses such as cognitive distraction that are not accompanied by significant changes in behavioural actions may be difficult to identify if relying on only single-modal visual data. These behaviours may even be incorrectly identified as a safe driving state, which may create potential safety hazards.

Methods based on sensor-modal data of vehicle motion. Driver behaviour directly affects how the vehicle performs, which can be analyzed by using vehicle motion sensor single-modal data such as lane departure [18], steering wheel angle [19], and longitudinal and lateral acceleration [20]. Such modality data are easy to obtain and have low computational complexity. Lansdown et al. [21] found that multitasking affects the driver’s ability to control the vehicle, manifested as an increase in the number of emergency braking events and changes in the steering wheel angle. Additionally, the vehicle state, such as the vehicle’s driving trajectory, running speed, and acceleration, will also change with the difficulty that the driver experiences in performing the task [22]. However, methods that rely on only single-modal data from vehicle motion sensing may lead to misjudgments due to noisy data. Moreover, driving style specificity may also affect its recognition accuracy, so adjusting the discrimination threshold appropriately may be necessary for drivers with different driving styles, which will greatly influence the generalization performance.

Methods based on multimodal data. Along with the sustainable development of multi-sensor collection techniques, many studies have been conducted with the fusion of multi-sensor data, such as longitudinal vehicle speed estimation and vehicle localization based on global positioning systems (GPS), inertial measurement units (IMU) and wheel speed sensor (WSS) [23,24], improved autonomous vehicle perception by fusing sensor data from camera and lidar [25], and autonomous emergency braking systems (AEBS) by using lidar, radar, and vision sensors [26]. There have been solutions proposed to integrate multimodal data for identifying distracted driving status, but such works are relatively limited [27]. Du et al. [28] confirmed that combining facial expression, speech, and car signals provided a better predictive performance for distraction detection. Rashwan et al. [29] proposed a two-stage model. First, three independent modules were used to process signals to extract features from the audio, image, video, and other signals. Then, the driver’s dangerous state estimation based on the hidden Markov model was output. Finally, the output and context information of each module were fused through the Bayesian network. Streiffer et al. [30] developed a unified data collection and analysis framework, DarNet, which analyzed driving image data through convolutional neural networks and IMU sensor data through recurrent neural networks. Finally, the two outputs were combined through Bayesian networks. The above multimodal learning approaches utilize multimodal information, in which different modalities can complement each other to improve the recognition system performance, but several challenges remain. It is logical to use appropriate subnetworks for feature extraction for different modality data to extract the spatial information of vision-modal data and the temporal information of motion sensor-modal data. However, it will significantly increase the complexity of later integration. Some early fusion strategies involve simply concatenating multimodal features at the input level, while late fusion strategies perform decision voting. They cannot consider the spatiotemporal dependency for multimodal features; in other words, these fusion approaches are difficult to sustainably learn both intramodal and intermodal correlations.

To solve these issues, a novel multimodal fusion method called vision-sensor fusion transformer (V-SFT) was introduced for recognizing distracted driving statuses with different risk levels, which simultaneously processes the vision-modal of driver behaviour and sensor-modal data of vehicle motion. On the one hand, it may help to reduce the number of accidents and road congestion, thereby reducing traffic pollution and carbon emissions; on the other hand, it may reduce the economic losses caused by traffic accidents, thus saving funds for society and businesses, and has a positive impact on driver assistance technology, as well as economic and environmental sustainability.

The distracted driving risk levels were quantified in this study, including safe driving (no risk), slight risk, moderate risk, and severe risk. This method reflects the driver’s safe driving sustainability through the driving risk levels and then provides a basis for allocating or switching the control rights of the human-machine codriving system. The proposed method is composed of three main modules: vision-modal and sensor-modal data early fusion, modality information interaction in the encoder block, and a classifier head for risk level inference. Specifically, feature prefusion was employed to aggregate representations from multiple modality tokens, and then the position-encoded multimodal feature set continuously interact based on the attention mechanism. Finally, the token at a specific position was separated for status classification. During the data acquisition phase, multimodal data during the simulated driving process were recorded synchronously. During the training phase, the model was trained to explicitly describe different risk statuses in distracted driving. To summarize, the main contributions of this study are threefold: (1) The developed end-to-end structure can not only adapt to vision and sensor data simultaneously but can expand and fuse more modal data, (2) feature-level prefusion reduces the complexity of postprocessing operations and provides a prerequisite for human–vehicle information interaction, and (3) taking the risk level as the recognition result to evaluate the status of distracted driving lays the foundation for further research on drivers’ driving abilities and driving right allocation or switching.

2. Materials and Methods

2.1. Data Collection

Existing datasets for distracted driving generally come from real vehicles or simulators. For example, the American University in Cairo (AUC) Distracted Driver [31] and State Farm Distracted Driver datasets [32] contain image data of drivers and were collected through offline motion simulation when the vehicle was stationary. The University of Alcalá (UAH)-DriveSet [33] dataset contains sensing data of a vehicle, which were recorded by the inertial measurement unit (IMU) of an onboard smartphone while the driver imitated a specific driving status in real vehicle experiments. The data recorded by simulator experiments in some other datasets also include eye movement and electrocardiogram (ECG) data, etc., but the intrusiveness of wearable instruments may cause some interference with driving behavior, and their cost is much higher than that of ordinary sensors. Therefore, a new multimodal distracted driving dataset was collected for our research through a rationally designed experiment, which provides both the vision-modal of driver behaviour and sensor-modal data of vehicle motion.

2.1.1. Driving Simulation Platform

Considering the experimental safety, a simulated driving platform was built in the laboratory environment, as shown in Figure 1, which integrates a visual system, a sound system, and a custom I/O board. The driving simulation software UC/win-road was installed on a PC using Microsoft Windows OS. The simulation can simulate driving on various types of roads under various outdoor conditions (weather, light, traffic, etc.) through modelling. The scene was displayed on three LCD screens, and the standard 3D graphics and 120-degree field of view (FOV) provided participants with a realistic driving experience. The interaction between the simulated driving platform and the participants was achieved through the device interface. A Logitech G927 racing steering wheel and pedals were used as input devices. The steering wheel was connected to an active force feedback system, and a passive force feedback mechanism was integrated into the pedals, recreating the feel of the clutch and brake of a real car. In addition, adjustable seats were installed to improve the driving comfort of participants. The data used to infer the driving status were collected by multiple modules, all of which were connected to the host and recorded synchronously. The visual sensor facing the driver’s body recorded the RGB video stream of the driver’s actions. The “other signals” module equipped with the UC/win-road software output various time-sequential signals such as the positions of the steering wheel, accelerator, brake, and clutch pedals in the form of a LOG file during the experiment. The rest of the hardware included an audio system for voice prompts and an iPad for simulating the central control screen to complete driving tasks. When the simulated driving started, the scene reference objects on the screen and the behaviour of the ego vehicle changed sustainably [34] as the driver manipulated the steering wheel and pedals, and the sound system provided real-time engine roar or brake sounds.

2.1.2. Participants and Driving Scenarios

Thirty-six drivers participated in this simulated driving experiment. The average participant age was 25 years old, with a standard deviation of 1.41. Most of the participants were recruited through a social media advertisement, and each participant was paid RMB 50 yuan for their involvement. Both urban driving scenarios and suburban driving scenarios were designed. The two-way three-lane road modelled by asphalt material contained four intersections, the lane width was 3.5 m, and the total distance was 10 km, as shown in Figure 2a,b. Figure 2c shows the driver’s perspective at an urban road intersection. Participants were asked to complete simulated driving on each route at an speed of approximately 60 km/h, and each route was driven only once.

2.1.3. Secondary Task

To simulate the contingency and suddenness of the distracted driving status as realistically as possible, when the participant drove the simulated vehicle to a trigger point on the road, a specific location designed in advance but unknown to the participant, a sound prompt was triggered, and the participant needed to complete the secondary task according to the prompt while driving. Extensive work has demonstrated that the great interference is predicted when a secondary task shares the same sensory modality as the primary driving task, since both tasks require access to the same limited pool of resources [35]. Therefore, the limited resources of the primary task are occupied to varying degrees by designing different combinations of secondary tasks to induce a distracted driving status with different risk levels. Four secondary tasks of different complexity were designed, and the corresponding driver behaviours are shown in Figure 3a–d. These behaviours were triggered by Task_0, Task_1, Task_2, and Task_3, respectively. The specific task combination is described in

Table 1. Secondary task descriptions of varying levels of complexity.

Task Category	Task Type	Description
Task_0	No secondary tasks	No voice command.
Task_1	Cognitive distractions	Triggering the voice command, following the command to calculate the two-digit addition operation that reported in the command, and speaking the result.
Task_2	Cognitive distractions & Visual distractions	Triggering the voice command, following the command to observe the two-digit addition operation that displayed on the screen, and speaking the result.
Task_3	Cognitive distractions & Visual distractions & Operating distractions	Triggering the voice command, following the command to observe the two-digit addition operation that displayed on the screen, and inputting the result by handwriting on the screen.

.

2.1.4. Procedure

When participants arrived at the driving simulator laboratory, they were informed of the purpose of the experiment and related arrangements in detail before signing up to participate. Before the start of the formal experiment, participants performed some practices to adapt to the manipulation of the driving simulation platform, completed the physical condition questionnaire, and confirmed the right to terminate the experiment at any time. In the formal experiment, participants needed only to follow the system prompts and drive to the end along a set route, without the interference of external factors such as experimenters. It took approximately 15 min to complete a route. The physical condition was fed back again while the participant took a break between the two drives. Each participant needed to complete the four simulated driving routes on two road types in total. Even with the same road type, the driving environments of the two routes were different. To avoid predictability and minimize practice effects, the secondary tasks were distributed on the four designed roads in a counterbalanced order [36] by a Latin square design (in

Table 2. Secondary task arrangement on different routes.

	Trigger Point 1	Trigger Point 2	Trigger Point 3	Trigger Point 4
Road 1	Task_2	Task_1	Task_3	Task_0
(Urban)	(27 + 54)	(14 + 17)	(49 + 25)	(/)
Road 2	Task_1	Task_0	Task_2	Task_3
(Urban)	(33 + 19)	(/)	(36 + 48)	(25 + 16)
Road 3	Task_3	Task_2	Task_0	Task_1
(Suburban)	(28 + 34)	(37 + 19)	(/)	(18 + 36)
Road 4	Task_0	Task_3	Task_1	Task_2
(Suburban)	(/)	(43 + 18)	(27 + 15)	(17 + 26)

). As well, the driving order of the four roads was also distributed evenly for each participant. Figure 4 illustrates the secondary task sequence on road one and the recorded data of two modalities, the vision-modal data of the driver’s behaviour and the sensor-modal data of the vehicle motion.

2.2. Methods

The raw data collected in the previous section were processed by data cleaning, extraction, merging, and standard normalization, to obtain a distracted driving status dataset with risk levels. This dataset contained two data modalities. The visual-modal data were acquired by image extraction from the video stream of the driver’s behaviour recorded by the camera. The sensor-modal data expressing the vehicle’s motion state were synchronously recorded through the simulated driving software. There were failure scenarios for status recognition based on only visual data, such as thinking, listening to the radio, and other cognitively distracted driving statuses, because these driving statuses usually do not have apparent changes in visual features. At the same time, relying on only the vehicle motion sensor modality is too sensitive to signal fluctuations, lacks information interaction with the driver, and has insufficient robustness and accuracy. Therefore, an attempt was made to fuse the two data modalities to obtain a joint representation and use the complementary human–vehicle information to enhance the discriminative feature learning. In our previous research [37], superior results were achieved in the distracted driving action classification based on a vision transformer (ViT) [38]. Inspired by this, a vision-sensor fusion transformer (V-SFT) fusion strategy for early fusion and information interaction of multimodal data is proposed. Figure 5 shows the overall architecture of the model. The model input is the preprocessed human–vehicle multimodal data pair, and the output is the risk level of the distracted driving status. The model can be described as three modules.

2.2.1. Module 1: Early Fusion of Vision-Modal Data and Sensor-Modal Data

First, the vision-modal data of driver behaviour were tokenized, as shown in Figure 6. The specific steps are as follows. (1) Resizing the raw image to [224 × 224], converting it into a tensor, and then normalizing its mean and variance to 0.5. (2) Dividing the [3 × 224 × 224] RGB image into 196 [3 × 16 × 16] patches using a convolution kernel with a size of 16, a stride of 16, and 768 convolution kernels. (3) All the patches are flattened and then transposed into a one-dimensional sequence, that is, 196 one-dimensional image tokens with a length of 768. The following Formula (1) describes the process:

$$\left[X_1^i, X_2^i, X_3^i, \dots ,X_{196}^i\right]=T\{Flatten[Conv2D(Normalize(X_{image}), k=\left(16,16\right), s=16)]\}$$

(1)

where $X_{image}$ represents the input visual modality data with a shape of [3 × 224 × 224]; $k$ and $s$ represent kernel size and stride, respectively; $Conv2D$ denotes the two-dimensional convolution; $T$ indicates transpose; and $X^i$ represents a one-dimensional token with a shape of [1 × 768], for a total of 196.

The second step is the tokenization of vehicle motion sensor-modal data, as shown in Figure 7. The sensor-modal data in the input data pair were standardized during preprocessing. Although its initial shape is a one-dimensional series of [1 × 40], its feature size is far from the vision-modal token with a total size of [196 × 768]. If the two are directly embedded, the vision-modal token with a larger number of features may weaken the information expression of the sensor-modal token or even completely cover it. Therefore, a trainable self-learning vector w-token is introduced with a size of [40 × 768], and Xavier initialization [39] is performed to upsample the [1 × 40] sensor series and project it to a [1 × 768] vector space. The obtained vector with shape [1 × 768] can be regarded as the class-token in the vision transformer (ViT) model. However, the improvement is that, compared with the class-token, which does not contain any initial information in the original ViT model, the [1 × 768] obtained vector here contains the prior information of the vehicle’s motion characteristics, which could be the key to improving model performance. In addition, to preserve the advantages of pretraining on large datasets, the pretrained class-token of the original ViT model is also embedded to obtain the final [1 × 768] sensor token. The token contains both the vehicle motion information and the pretrained parameters, which can improve the model performance and speed up model fitting accordingly. The following Formula (2) describes the process:

$$X_{cla}^s=\left(X_{sensor}\otimes W_{token}\right)\oplus Cl{a}_{token}^{pre}$$

(2)

where $X_{sensor}$ represents the [1 × 40] input sensor-modality series, $W_{token}$ represents the [40 × 768] trainable self-learning vector, $Cl{a}_{token}^{pre}$ represents the [1 × 768] pretrained class-token, $X_{cla}^s$ represents the [1 × 768] sensor token, $\otimes$ represents matrix multiplication, and $\oplus$ represents the matrix addition.

The third step is to obtain the joint representation of the two modalities. The acquired [196 × 768] image token and [1 × 768] sensor token are concatenated to output a [197 × 768] sequence group, that is, 197 sequences of shape [1 × 768]. However, at this time, these sequences are not position encoded. The position information is embedded by adding the trainable parameter position embedding to the [197 × 768] sequence group element by element. Finally, the joint token output from two modal data fusion is obtained. The above process can be summarized as Formulas (3) and (4):

$$\left\{ X_{cla}^s,\left[X_1^i, X_2^i, X_3^i, \dots ,X_{196}^i\right]\right\}=Cat\left( X_{token}^{sesor},X_{token}^{image}\right)$$

(3)

$$X_{token}^{joint}=\left\{X_{cla}^s, \left[X_1^i, X_2^i, X_3^i, \dots ,X_{196}^i\right]\right\}\oplus PE$$

(4)

where $Cat$ denotes vector concatenation, $\left\{ X_{cla}^s,\left[X_1^i, X_2^i, X_3^i, \dots ,X_{196}^i\right]\right\}$ represents a [197 × 768] sequence group, $PE$ denotes trainable vector position embedding with the same size, $X_{token}^{joint}$ represents the joint token of the two modal tokens, and $\oplus$ denotes the matrix addition.

2.2.2. Module 2: Modality Information Interaction in the Encoder Block

As shown in Figure 8, the joint representation $X_{token}^{joint}$ enters the repeatedly stacked encoder blocks after passing through a dropout [40] layer. Each encoder block contains a residual multihead self-attention (MSA) block and a residual multilayer perceptron (MLP) block. Specifically, layer norm (LN) is applied before every MSA and MLP, and then a dropout layer and a residual connection [41] are applied after them in sequence. In this module, the sensor-modal token $X_{cla}^s$ representing vehicle motion information and the vision-modal token $\left[X_1^i, X_2^i, X_3^i, \dots ,X_{196}^i\right]$ representing driver behaviour information, and these tokens interact continuously to learn effective abstract representations.

The interaction of different modal information in the MSA block is shown in Figure 9a. The following Formula describes the process:

$$q^i=DP\left(X_{token}^{joint}\right)W_q=\frac{Linear{\left[DP\left(X_{token}^{joint}\right)\right]}_{clip\left[0\right]}}{number of heads}$$

(5)

$$k^i=DP\left(X_{token}^{joint}\right)W_k=\frac{Linear{\left[DP\left(X_{token}^{joint}\right)\right]}_{clip\left[1\right]}}{number of heads}$$

(6)

$$v^i=DP\left(X_{token}^{joint}\right)W_v=\frac{Linear{\left[DP\left(X_{token}^{joint}\right)\right]}_{clip\left[2\right]}}{number of heads}$$

(7)

In Formulas (5)–(7), $q^i$, $k^i$, and $v^i$ are the query, key, and value, respectively, corresponding to the i-th head (branch) of the MSA block, all of which are [197 × 96] vectors. They can be obtained by mapping [197 × 768] $X_{token}^{joint}$ through [768 × 96] vectors $W_q$, $W_k$, and $W_v$. The specific steps are as follows. First, the input [197 × 768] $X_{token}^{joint}$ is linearly transformed into [197 × 768 ×3] through the fully connected layer. Then, through reshaping, permutation, and clipping in turn, [197 × 768 × 3] is divided into three [197 × 768] vectors according to the indexes [0], [1], and [2]. Finally, each [197 × 768] vector is equally divided according to the number of heads; the number of heads here is eight.

$$hea{d}_i=Attention\left(q^i,k^i,v^i\right)=Softmax\left(\frac{q^i{\left(k^i\right)}^T}{\sqrt{h_d}}\right)v^i$$

(8)

In Formula (8), $q^i$, $k^i$, and $v^i$ interact with each other based on the attention mechanism [42] to obtain the output of each [197 × 96] branch $hea{d}_i$, where ${\left(k^i\right)}^T$ is the transpose of $k^i$ and $h_d$ is the dimension of each head, which is 96. $Softmax$ denotes the softmax functions.

$$MSA=\left[Concat\left(hea{d}_1,hea{d}_2,\dots ,hea{d}_i\right)\right]W_O$$

(9)

$$Re{s}_{MSA}=DP\left(X_{token}^{joint}\right)\oplus DP\left(MSA\right)$$

(10)

In Formula (9), after concatenating the output of eight heads, the sequence group [$hea{d}_1,hea{d}_2,\dots ,hea{d}_8$] is [197 × 768]. Then, a [768 × 768] vector $W_O$ is used for linear projection to obtain MSA, which is still [197 × 768]. Last, after MSA passes through a dropout layer, it is added to the $DP\left(X_{token}^{joint}\right)$ to obtain the $Re{s}_{MSA}$ with a shape of [197 × 768] in Formula (10), which is the output of the entire residual MSA block. The MLP block contains two linear layers with Gaussian Error Linear Unit (GELU), as shown in Figure 9b. First, Linear1 transforms the input [197 × 768] sequence $Re{s}_{MSA}$ into [197 × 3072]. Then, after the GELU activation function and dropout, the [197 × 3072] sequence is retransformed back to [197 × 768] by Linear2 and then passes through a dropout layer. Similarly, the output of the residual MLP block is still [197 × 768].

2.2.3. Module 3: Classifier Head for Risk Level Inference

The structure of module 3 is shown in Figure 10. A norm layer is also applied before the classifier head. From the previous section, after 12 consecutive encoder blocks, the output shape is still [197 × 768]. However, at this time, the spatiotemporal dependency of both intramodality and intermodality has been learned. Then, extracting the category sequence class-token in this sequence group is necessary [38]. Since position information encoding was performed in advance in module 1, the [1 × 768] sequence whose index is [0] is removed. Finally, the [1 × 768] features are mapped into four label spaces of risk level by a linear layer, and the number of neurons in the output layer is set to four. The class probability distribution is normalized using the softmax function, as shown in Formula (11).

$$softmax\left(z_j\right)=\frac{e^{z_j}}{{{\displaystyle \sum }}_{c=1}^C{e}^{z_c}}$$

(11)

where $softmax\left(z_j\right)$ represents the value of $j$ after the $C$-dimension vector z is mapped by the softmax functions. $C$ represents the number of classes.

2.3. Evaluation Metrics

To quantitatively measure the testing results, precision, recall, and F1 score were applied to measure the model performance [43]. Accuracy is the overall prediction accuracy of the model. Specifically, four classification results are defined and statistically calculated for each behaviour, including true positive ($T{P}_i$, is samples of behaviour i that are correctly identified), true negative ($T{N}_i$, denotes that samples that do not belong to behaviour $i$ are classified into other behaviours), false positive ($F{P}_i$, is cases in other behaviours that are incorrectly classified into behaviour $i$), and false negative ($F{N}_i$, is cases in behaviour $i$ that are incorrectly predicted as other behaviours). Specifically, the precision of behaviour $i$ ($P_{ri}, i=1, 2, 3\dots$) is calculated as:

$$P_{ri}=\frac{T{P}_i}{T{P}_i+F{P}_i}\times 100\%$$

(12)

The recall of behaviour $i$ ($R_{ei}, i=1, 2, 3\dots$) is calculated as:

$$R_{ei}=\frac{T{P}_i}{T{P}_i+F{N}_i}\times 100\%$$

(13)

The F1-score ($F1$) depends on both $P_{ri}$ and $R_{ei}$ and is the harmonic mean of these two values.

$$F1=2\times \frac{P_{ri}\times R_{ei}}{P_{ri}+R_{ei}}\times 100\%$$

(14)

Finally, the accuracy ($Acc$) is calculated as:

$$Acc=\frac{T{P}_i+T{N}_i}{T{P}_i+T{N}_i+F{P}_i+F{N}_i}\times 100\%$$

(15)

3. Results

In this section, the proposed V-SFT model for distracted driving status risk level recognition was trained and tested. First, in the case of relying on only vision single-modal data, the evaluation results of the proposed V-SFT and some convolutional neural network (CNN) models on the collected dataset were compared. Next, in the case of relying on only sensor single-modal data, the evaluation results of the V-SFT and some benchmark models on the collected dataset were compared. Last, in the case of using both modalities simultaneously, the evaluation results of the proposed V-SFT fusion model on the collected dataset were compared with some other fusion structures. After preprocessing the raw data of the two vision-sensor modalities in the dataset, the training, validation, and test sets were divided at a ratio of 7:2:1, and the corresponding ratio of the sample numbers was 16,800:4800:2400. In order to avoid the influence of other factors, the implementation details, such as hardware platform, training strategy, and hyperparameters, were all set to be consistent during the training process. Specifically, the training and testing of all models were supported by GPU NVIDIA GeForce RTX 3080 Ti. In terms of training strategies and hyperparameter settings, pretraining was performed on the mixed distracted driving datasets of AUC [31] and State Farm [32], and then training was continued for 100 epochs on the collected dataset based on transfer learning. In addition, the batch size was set to 32, and the learning rate was set to 0.001. During the training process, a large amount of labeled data were fed into the model, and the four risk level labels of safe driving, slight risk, moderate risk, and severely risk were used as fitting targets, which were encoded as [1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], and [0, 0, 0, 1].

3.1. Experimental Evaluation of Vision-Only Modality Input

In the case of inputting only vision single-modal data, the validation curves during the training process of the proposed V-SFT (v-only) and several CNN benchmark models, such as MobileNetV3 [44], InceptionV3 [45], and ResNet34 [41] are shown in Figure 11. When the trend of the curve is relatively stable, the validation accuracy of the V-SFT with only vision-modal input (V-SFT v-only) is slightly higher than that of ResNet34 and much higher than that of MobileNetV3 and InceptionV3. This may be largely attributed to the ability of the V-SFT backbone to capture long-range dependencies rather than local features [46]. As shown in

Table 3. The test results of different models with only vision-modal input.

Models	Safe Driving			Slight Risk			Moderate Risk			Severe Risk			$\mathit{A}\mathit{c}\mathit{c}$ (%)
	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)
MobileNet V3	77.1	78.7	77.9	76.3	76.7	76.5	87.4	85.8	86.6	90.3	89.7	90.0	82.7
Inception V3	66.6	74.5	70.3	69.6	59.5	64.2	89.7	84.3	86.9	83.7	91.0	87.2	77.3
ResNet 34	75.0	78.8	76.9	76.7	73.2	74.9	96.2	93.0	94.6	95.0	97.7	96.3	85.7
V-SFT v-only	76.2	80.0	78.1	77.8	74.3	76.0	96.7	93.3	95.0	95.3	98.2	96.7	86.5

, the results on the test set, specifically precision (Pr), recall (Re), F1-score (F1) and accuracy (Acc), were calculated based on the evaluation metrics. They are slightly lower compared to the validation process but still within the expected range.

The confusion matrixs of the test results are shown in Figure 12. The recognition performance of the four models with single-modal visual data input can be intuitively observed through the heatmap. It was shown that all the models were able to identify moderate and severe risk statuses well relying on only vision-modal data. The moderate risk status triggered by task_2 includes not only the cognitive distraction caused by thinking about calculations but also the visual distraction caused by observing the screen through gaze or head movement. In addition to cognitive and visual distractions, the severe risk status triggered by task_3, also includes operational distractions added by manual screen manipulation. It can be seen that all models capture the differences in actions well and make distinctions. The best performance is achieved by the proposed V-SFT (v-only), which has a test accuracy rate of 86.5% for the four statuses. Compared with MobileNetV3, InceptionV3, and ResNet34, it is 3.8%, 9.2%, and 0.8% higher, respectively.

It can be seen that the above four models that rely on only vision-modal data may confuse some safe driving statuses with slight risk statuses, and even the best-performing V-SFT (v-only) has not overcome this problem. Specifically, when there was no secondary driving task (task_0), the driver continued to hold the steering wheel with both hands and look straight ahead. In the slight risk status triggered by task_1, the driver only opened his mouth slightly when answering the mathematical calculation given by the voice prompt. Therefore, it is speculated that when relying on only the vision modality, the model relies heavily on illumination or pixel definition in learning basic features such as lines, edges, and colors. Particularly under real driving conditions, it would be unimaginable if a large number of slight risk statuses were misjudged as safe driving, which is also an issue that this study will solve.

3.2. Experimental Evaluation of Sensor-Only Modality Input

In the case of inputting only sensor single-modal data, the validation curves during the training process of the proposed V-SFT (s-only) and several benchmark models, such as support vector machine (SVM) [47], random forest [48], recurrent neural network (RNN) [49], gated recurrent unit network (GRU) [50], and long short term memory network (LSTM) [51], are shown in Figure 13. It can also be seen that after the curve fluctuation is relatively stable, the validation accuracy of the V-SFT with only sensor-modal input (V-SFT s-only) is higher than the others. As shown in

Table 4. The test results of different models with only sensor-modal input.

Models	Safe Driving			Slight Risk			Moderate Risk			Severe Risk			$\mathit{A}\mathit{c}\mathit{c}$ (%)
	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)
SVM	69.3	72.8	71.0	72.2	71.5	71.8	69.8	62.0	65.7	67.9	72.7	70.2	69.8
Radom Forest	64.5	67.8	66.1	72.4	71.7	72.0	73.0	64.8	68.7	67.1	71.8	69.4	69.0
RNN	77.4	69.8	73.4	73.9	84.7	78.9	66.6	63.5	65.0	68.0	68.0	68.0	71.5
GRU	59.5	81.0	68.6	72.0	51.5	60.0	83.6	73.0	77.9	80.3	84.3	82.3	72.5
LSTM	60.3	79.8	68.7	72.3	52.7	61.0	81.2	70.7	75.6	79.0	85.2	82.0	72.1
V-SFT s-only	78.1	69.0	73.3	73.3	82.2	77.5	76.8	74.7	75.7	72.5	74.2	73.3	75.0

, the test results of the above six models on the collected dataset, specifically precision (Pr), recall (Re), F1-score (F1) and accuracy (Acc), were calculated based on the evaluation metrics in the previous section. Among them, the V-SFT model with only sensor-modal input (V-SFT s-only) achieves the highest test accuracy of 75.0%. Compared with SVM, random forest, RNN, GRU, and LSTM, the test accuracy of V-SFT (s-only) is 5.2%, 6.0%, 3.5%, 2.5%, and 2.9% higher, respectively. This also benefits from the transformer backbone of the V-SFT, which takes advantage of the temporal representation of sensor-modal data [52].

The confusion matrixs of the six models with sensor single-modal data input are shown in Figure 14. Compared to the model that relies on only the vision modality in the previous section, the test accuracy that relies on only the sensor modality is much lower. The reason may be that the feature dimension of sensor data is relatively smaller than that of vision data. In addition, vehicle motion sensor data are easily affected by factors such as road conditions and driving styles, so the vehicle information that can be learned may not be sufficient. Its direct manifestation is that some moderate risk statuses (triggered by task_2) are confused with severe risk statuses (triggered by task_3) when relying on only sensor-modal data. It is speculated that the feature difference of the vehicle motion sensor modality is not as significant as that of the visual modality in these two levels. However, for the distinction between safe driving (triggered by task_0) and slight risk (triggered by task_1), the V-SFT model with only sensor-modal input (V-SFT s-only) showed some improvement. The model is also superior to the other five models in overall recognition accuracy. Therefore, utilizing vehicle information in the sensor modality to supplement driver information in the vision modality also serves as motivation for this study.

3.3. Experimental Evaluation for Vision-Sensor Multimodal Data Input

In this section, under the premise of simultaneous input of vision-sensor multimodal data, CNN-RNN, CNN-GRU, and CNN-LSTM fusion models were built based on the early fusion strategy and compared with the proposed V-SFT on the collected dataset using simultaneous vision-sensor multimodal data input. Figure 15 shows the validation accuracy curves of the above models during the training process; it can be seen that the curve of the V-SFT model is higher than that of the other models after stabilization. The test results, specifically precision (Pr), recall (Re), F1-score (F1) and accuracy (Acc), are also calculated based on the evaluation matrics, as shown in

Table 5. The test results of different models with vision-sensor multimodal input.

Models	Safe Driving			Slight Risk			Moderate Risk			Severe Risk			$\mathit{A}\mathit{c}\mathit{c}$ (%)
	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)	$\mathit{P}\mathit{r}$ (%)	$\mathit{R}\mathit{e}$ (%)	$\mathit{F}1$ (%)
CNN-RNN	76.6	82.5	79.4	80.0	72.8	76.2	85.8	89.7	87.7	92.9	90.0	91.4	83.8
CNN-GRU	86.4	76.0	80.9	77.7	87.5	82.3	93.5	90.8	92.1	92.8	94.8	93.8	87.3
CNN-LSTM	85.1	75.3	79.9	76.8	86.0	81.1	93.7	91.2	92.4	93.1	95.2	94.1	86.9
V-SFT	90.5	83.8	87.0	83.9	90.2	86.9	97.8	94.5	96.1	96.3	99.3	97.8	92.0

. In addition, the parameters of the four fusion models CNN-RNN, CNN-GRU, CNN-LSTM and V-SFT are 81.40 M, 77.92 M, 81.26 M, and 86.33 M respectively.

First, V-SFT achieves the best overall recognition accuracy of 92.0% among all fusion models. Compared with CNN-RNN, CNN-GRU, and CNN-LSTM, the recognition accuracy of V-SFT is 8.2%, 4.7% and 5.1% higher, respectively. Considering that the input of all the above models is vision-sensor multimodal data and they are all built based on the early fusion strategy, it can be inferred that the structural difference is the main reason affecting model performance. The proposed V-SFT uses one unified transformer backbone to simultaneously process two modalities and to preserve the intermodality dependencies. Second, referring to the previous Table 3 and Table 4, it can be found that the V-SFT using vision-sensor multimodal input achieves significantly improved results compared to single-modal data input. Specifically, the recognition accuracy of the V-SFT with multimodal fusion input improves by 5.5% compared with that relying on only vision single-modal input; compared with that relying on only sensor single-modal input, it improves by approximately 17%. This indicates that fusing multimodal information is an effective method for improving driving status understanding.

The confusion matrixs of the four fusion models using vision-sensor multimodal data input are shown in Figure 16. Compared to the previous Figure 12, it can be observed that after increasing the sensor-modality input, the recognition effect of the safe driving status and the slight risk status greatly improves. Similarly, referring to the Figure 14, it can be observed that after increasing the vision-modality input, the recognition effect for the moderate risk status and the severe risk status also improves.

4. Discussion

The purpose of this study is to quantify and identify the distracted driving risk levels, including safe driving (no risk), slight risk, moderate risk, and severe risk. By accurately identifying the risks, sustainable driving safety can be achieved, which can effectively improve public health and traffic flow efficiency. In this experiment, driving statuses with different risk levels were triggered by rationally designing secondary tasks. Thirty-six volunteers were recruited to simulate driving on two road types and four routes with different orders of secondary tasks, and data from different modalities were recorded simultaneously. In addition, a V-SFT model was constructed, and the two modal data types were adapted through one unified backbone network. The vision-modal data representing the driver’s behaviour and the simultaneously recorded sensor-modal data representing the vehicle’s motion interacted and fused.

The performance of the method based on multimodal fusion significantly improved compared with that based on a single modality. Specifically, although some works that use only visual single-modal data for distracted driving status recognition (DDSR) can achieve high accuracy on their datasets [16], there are still some failure scenarios, such as cognitive distraction in real driving conditions [9]. As a frequently occurring risky driving status, cognitive distraction usually presents as a temporary decline in the perception and reasoning ability [22]. In the results of Table 3, it can be seen that for the test results of slight risk status represented by cognitive distraction, the highest recognition recall rate of the methods relying on visual single-modality data, which is 76.7%. The reason may be that the slight risk status represented by cognitive distraction is not accompanied by body movements, which means that the visual modality data features are not evident. Therefore, relying solely on the visual modality is not sufficient to enhance the generalization performance of the model. In Table 5, the average recognition recall rate for slight risk status using the multimodal data fusion methods increases to 84.1%, and that of the proposed V-SFT is the highest at 90.2%, showing the performance enhancement achieved by modality fusion. Similarly, in some other works that rely on only the sensor modality, although typical risky vehicle states such as sharp acceleration or deceleration [20], lane departure [18], and close following distance [19] can be captured, methods based on only this single-modal data may often cause misjudgments. The resulting traffic accidents or low fuel efficiency will be detrimental to social health and environmental sustainability. In the confusion matrix heatmaps in Figure 14, it can be observed that the methods that use only sensor-modal data largely confuse the two statuses of moderate risk and severe risk. The specificity of the driver’s driving patterns [21] and the frequent vehicle starts and stops on certain road sections may have an impact on the recognition accuracy. However, it can be clearly seen in Figure 16 that the multimodal data fusion method can better solve this problem, and the proportion of misjudgments is greatly reduced, which once again reveals that the complementary modal information strengthens the sustainability understanding of driving status.

There are few existing studies on driving status recognition by fusing multimodal data. In fact, different fusion strategies lead to differences in the extensibility and operability of the models, which determines whether it can adapt to the sustainability of the future development of autonomous driving technology and intelligent transportation systems. In terms of sustainable expansion capability, previous studies used traditional convolutional backbone networks such as VGG [53] and AlexNet [54] or improved convolutional backbone networks such as multiscale attention CNN [55] to extract the spatial information of driver actions and used traditional recurrent backbone networks such as LSTM [51] or improved recurrent backbone networks such as attention-based GRU [50] to learn the temporal vehicle motion information. Although these methods take advantage of different types of networks to process corresponding data modalities, the mechanical combination of different network branches ignores the spatiotemporal dependence of multimodal features and limits the sustainable expansion capability of the model. The reason is that non-shareable network parameters need to be trained for different backbones. Additionally, with the addition of more modal data, such as acoustic audio [28], it may be necessary to adjust the entire model structure for compatibility. The transformer backbone network used in this study has proven to be applicable in many fields, such as computer vision, time series classification, and natural language processing [42]. This means that only one unified backbone network can adapt to multiple modal data, which can solve the problem of the insufficient extensibility of existing models. In terms of operationality, some studies required training the models in stages [27,30]. The approach was to obtain a set of driving status confidence probabilities based on the vision modality and another set of confidence probabilities based on the sensor modality. Then, the two sets were probabilistically fused through the Bayesian network [56]. The V-SFT in this study performed feature-level prefusion on different modalities, and the encoder block maximized the information interaction of the vision-sensor (human-vehicle) modality. In addition, this end-to-end structure is more integrated and concise in the process from data preparation and model training to deployment.

However, there are some limitations in this study. Due to the current limited experimental conditions, the recruited participants were generally school students, all of whom were young males with less driving experience. Future work will consider expanding the research sample to further explore the impact of variables such as gender, occupation, and driving experience. In addition, although the human–vehicle information fusion has been achieved, there are still failure or misjudgment scenarios for complex and changeable road traffic conditions. Therefore, an important direction for future work will be to integrate perceived or stored roadside scene information into the model on this basis. Additionally, data collection should also use real vehicles to conduct experiments on real roads as much as possible to simultaneously obtain human–vehicle–road collaborative information [57]. Last, despite the proposed V-SFT having excellent accuracy and operability, its real-time performance needs to improve. To use the V-SFT for real-time recognition, it is critical to further optimize the model parameters before the hardware deployment, and to ensure the data transmission rate and sufficient computing resources of the hardware.

5. Conclusions

Sustainable development requires balancing the interests of multiple aspects, such as the economy, environment, and society. In this study, an end-to-end vision-sensor fusion transformer model, termed V-SFT, was constructed for recognizing distracted driving status risk levels. The model may help to improve transportation and energy efficiency, and its extensibility has great potential for future sustainable developments. It consisted of three main modules: early fusion of vision-modal data and sensor-modal data, modality information interaction in the encoder block, and classifier head for risk level inference. First, the tokenization process was built separately to extract tokens of different modalities, and then feature prefusion was employed to aggregate the respective modality representations. Second, after position encoding, the multimodal joint tokens continuously interacted based on a multi-head attention mechanism. Finally, the token at a specific position was extracted for risk level recognition in the classifier head. To verify the effectiveness of V-SFT, a data collection platform was developed to synchronously record driver behaviour visual signals and vehicle motion sensing signals during simulated driving. The V-SFT was evaluated under different modality inputs on the collected datasets. It was shown that V-SFT can outperform the other compared models regardless of whether only vision-modality or only sensor-modality input was used. When relying on the fusion input of two modalities, V-SFT achieved the best performance with an recognition accuracy of 92%. In future work, to obtain a more robust model, the scope of data collection will be further expanded. Benefiting from the sustainability of the model’s expansion capability, more modal data will be collected from other types of sensors, and different regularization techniques will be applied to optimize the model and its inputs. The model could be integrated into advanced driver assistance systems (ADAS) as a separate module to provide a basis for the allocation or switching of control rights for intelligent vehicles.