PRISM-MTL: Inter-Modal Selective Multi-Task Learning for Assistive Driving Perception

Abstract

Advanced driver assistance systems (ADAS) require a comprehensive understanding of multiple tasks related to the physical and mental states of drivers and traffic situations. Existing ADAS studies perform driver emotion recognition (DER), driver behavior recognition (DBR), traffic context recognition (TCR), and vehicle behavior recognition (VBR) using models designed based on single-task learning, thereby failing to reflect the interactions among tasks in real driving environments. This paper proposes perception and recognition with inter-modal selective multi-task learning (PRISM-MTL), an integrated multimodal and multi-task learning framework that jointly recognizes DER, DBR, TCR, and VBR. The proposed PRISM-MTL consists of a hierarchical stage-wise attention network (HSA-Net)-based multimodal encoder that extracts spatial features from heterogeneous multimodal inputs and task-specific modality fusion (TSMF), which selectively learns effective modality information for each task. This design addresses negative transfer, a key challenge in multi-task learning. In the multimodal encoder, HSA-Net extracts visual modality tokens that emphasize global structural patterns and key spatial regions from multi-view images, while Token-SE generates joint modality tokens that reflect the spatial configuration of joint data. TSMF generates task-specific fusion features that selectively emphasize the modality cues for each task. The generated task-specific fusion features are summarized through temporal mean pooling, and final predictions of driver states and traffic situations are produced by each task head. Experimental results show that the proposed PRISM-MTL achieves state-of-the-art performance on the public AIDE database, with an mAcc of 86.25% ± 0.35 for multi-task recognition of driver states and traffic situations.

1. Introduction

Recently, advanced driver assistance systems (ADAS) have been studied to improve driving safety by monitoring driver states and traffic situations through functions such as automatic emergency braking, lane keeping, and parking assistance. According to the WHO and the U.S. CDC, road traffic accidents cause approximately 1.19 million deaths and about USD 3.6 trillion in annual economic cost worldwide [1,2], motivating research on driver state and traffic situation recognition for accident prevention.

Existing studies on driver state recognition and traffic situation recognition are divided into single-task learning (STL)-based studies, such as driver behavior recognition (DBR) [3], driver emotion recognition (DER) [4], traffic context recognition (TCR) [5], and vehicle behavior recognition (VBR) [6], as shown in Figure 1. Since STL-based driver state and traffic situation recognition studies investigate each task independently, they fail to reflect inter-task correlations in real driving environments; therefore, multi-task learning (MTL)-based driver state and traffic situation recognition studies are being conducted to address this limitation [7].

MTL improves generalization performance by learning shared representations across multiple tasks and is divided into hard parameter sharing and soft parameter sharing methods [8,9]. Hard parameter sharing shares a single encoder across tasks to improve efficiency but may cause negative transfer when task objectives differ. Soft parameter sharing mitigates this by maintaining task-specific networks, at the cost of higher complexity and training cost. Driver states and traffic situations are mutually related, but modality relevance differs across tasks; therefore, existing MTL methods that mainly focus on task-level shared representation learning and task-specific branch design fail to represent task-specific modality relevance, resulting in modality-induced negative transfer.

In this paper, we propose perception and recognition with inter-modal selective multi-task learning (PRISM-MTL), an integrated multimodal and multi-task learning framework that jointly recognizes DER, DBR, TCR, and VBR. To address modality-induced negative transfer in MTL, the proposed PRISM-MTL consists of preprocessing for in-vehicle and out-vehicle video data, a multimodal encoder that extracts visual modality tokens and joint modality tokens, and task-specific modality fusion (TSMF), which selects effective modality information for each task and learns specialized features using learnable task tokens and multimodal tokens.

The multimodal encoder extracts visual modality tokens that emphasize global structural patterns and key spatial regions from multi-view images through HSA-Net and generates joint modality tokens that reflect the spatial configuration of joint data through Token-SE. HSA-Net extracts high-level semantic information by attending to directional structural patterns and global spatial information from input images through a hierarchical attention structure composed of four stages. TSMF selectively emphasizes task-relevant modality cues through task-conditioned cross-attention, and the attended representations are refined into task-specific fusion features. The task-specific fusion features are compressed through temporal mean pooling, and driver states and traffic situations are recognized by each task head. Experimental results show that the proposed PRISM-MTL achieves state-of-the-art performance on the public assistive driving perception (AIDE) database, with a DER accuracy of 84.55% ± 0.91, a DBR accuracy of 80.59% ± 1.67, a TCR accuracy of 94.07% ± 0.80, a VBR accuracy of 85.79% ± 0.83, and a mean accuracy (mAcc) of 86.25% ± 0.35. In addition, generalization performance was verified on the public NTHU-DDD, StateFarm, and HDD databases, and ablation studies verified the reliability of stage-wise attention in HSA-Net, the effectiveness of the HSA-Net and TSMF methods, and the performance improvement achieved by multimodal and multi-task learning.

2. Related Works

Existing studies on driver state and traffic situation recognition are categorized into DER, DBR, VBR, and TCR and have been investigated using STL and MTL methods. Table 1 summarizes the datasets, modalities, methods, and task information used in studies on driver state and traffic situation recognition.

2.1. Single-Task Learning for Driver State Recognition Categories

Studies on driver state recognition have been actively conducted to prevent road traffic accidents. Driver state recognition systems can be divided into DBR and DER. Existing DBR and DER studies are divided into CNN-based recognition techniques and Transformer-based recognition techniques to design adaptive models for unstructured factors in driving environments, such as illumination changes, occlusion, and posture variations.

CNN-based DBR and DER techniques have been studied to improve recognition performance through transfer learning based on large-scale pretrained backbones, custom CNN design, and the integration of attention modules [10,11,12]. ResNet is an architecture that optimizes deep neural networks through residual connections and has been applied in studies aimed at lightweight design and accuracy improvement in DBR and DER [13,14]. Methods that integrate attention modules into CNNs improve DBR and DER performance by emphasizing facial regions and suppressing unnecessary features [15,16]. Custom CNN design methods aim to develop lightweight models considering their applicability to real driving environments [17]. CNN-based DBR and DER studies extract local features and perform efficient inference, but they have limited capability for modeling global relationships, and features can be lost due to occlusion and posture variations.

Transformer-based DBR and DER techniques have been studied to improve recognition performance through strategies such as Vision Transformer (ViT)-based representation learning, semi-supervised representation learning, and temporal data-based learning model design [18]. ViT-based representation learning tokenizes input images into patches and models global relationships through self-attention, making it robust to occlusion, head pose variations, and illumination changes that frequently occur in driving environments [19,20]. Semi-supervised representation learning combines ViT with methods such as semi-supervised learning, pseudo-labeling, and self-learning to secure labels and data, thereby addressing the limitation of obtaining large-scale labels for driving environment data [21]. Temporal data-based learning model design utilizes the temporal continuity of driving situations for DBR and DER [22]. Transformer-based DBR and DER studies achieve high recognition accuracy through global relationship modeling, but their application to real driving environments is difficult because the computational complexity of self-attention increases the computational cost.

Recently, studies on DBR and DER have focused on generalized systems applicable to real driving environments, multimodal data-based recognition, and vision–language model-based recognition by combining the local representational capability of CNNs with the global relationship modeling capability of Transformers. Saadi et al. [23] proposed a hybrid-based DER system that combines CNN and ViT components for lightweight and real-time recognition. The proposed hybrid-based DER system combines ShuffleNet V2 and EfficientViT-M2 to reduce computational cost and improve emotion recognition performance. The hybrid-based DER system was validated on the KMU-FED and KDEF databases, achieving accuracies of 97.3% and 92.44%, respectively, and its processing speed was reported as 3.3 ms. Cui et al. [24] proposed an enhanced Swin Transformer to detect driver behavior in human–machine cooperative driving environments. The proposed enhanced Swin Transformer improves DBR performance by inserting an enhanced channel attention module into the Swin Transformer. The enhanced Swin Transformer-based DBR system was validated on the StateFarm database, achieving an accuracy of 99.42%. Ying et al. [25] proposed a multimodal data-based DER system that combines audio and video. The proposed multimodal data-based system improves emotion recognition performance by compensating for the uncertainty of a single modality and enhancing emotion cues through audio–video feature-level fusion. The multimodal data-based DER system was validated on the RAVDESS and SAVEE databases, achieving accuracies of 93% and 99%, respectively. Manavand et al. [26] proposed ILAB-CNN, which combines an interlaced local attention block (ILAB) and a modified squeeze-and-excitation (MSE) module within a CNN for complex driver facial expression recognition. The proposed ILAB-CNN system emphasizes spatial features important for emotion recognition in facial regions and suppresses unnecessary features based on ILAB, MSE, and self-attention. The ILAB-CNN-based DER system was validated on the FER-2013, RAF-DB, and KMU-FED databases, achieving accuracies of 75.3%, 85.06%, and 98.8%, respectively. Hasan et al. [27] proposed a contrastive language-image pre-training (CLIP)-based DBR system using vision–language pretrained representations. The proposed CLIP-based system classifies driver behavior in a zero-shot manner using the similarity between visual and textual embeddings extracted from video frames and text prompts and improves DBR performance through fine-tuning a linear classifier with a small number of labels. The CLIP-based DBR system was validated on the SynDD, StateFarm, and AUC databases, achieving accuracies of 85.67%, 98.226%, and 88.61%, respectively.

2.2. Single-Task Learning for Traffic Context Perception Categories

Studies on traffic situation recognition have been actively conducted to understand driving environments and perform safe decision-making. Traffic situation recognition systems can be divided into the field of low-level external perception, which consists of recognizing traffic regulation elements, road structures, and dynamic objects, and the field of high-level TCR, which infers the semantic state of driving scenes by integrating perception results.

The field of low-level external perception aims to recognize visual elements that constitute driving environments, such as traffic signs, traffic lights, pedestrians, vehicles, lanes, and road boundaries. Existing studies are mainly divided into object detection-based techniques and road structure estimation-based techniques [28]. Object detection-based techniques have been studied with one-stage detector and two-stage detector architectures, aiming for lightweight design and accurate recognition. One-stage detectors secure real-time object perception performance and robustness in embedded environments by combining lightweight backbone design, attention-based feature enhancement, and proposal quality enhancement [29,30]. Two-stage detectors generate candidate region proposals and perform precise classification and regression by combining prior enhancement in the proposal stage, feature pyramid networks for multi-scale feature fusion, and deformable convolutional networks for deformable feature extraction to ensure robustness in real environments, thereby improving localization accuracy for small objects or complex backgrounds [31,32,33]. Road structure estimation-based techniques strengthen the continuity and spatial consistency of road geometry using segmentation-based estimation, structural representation learning model design strategies, and global relationship learning model design strategies to restore thin and continuous structures, such as lanes, road boundaries, and drivable areas [34]. Segmentation-based estimation predicts spatial structures through pixel-level segmentation and minimizes boundary discontinuities through multi-scale feature fusion and attention-based refinement [35]. Structural representation learning model design strategies represent segmentation results as vector geometry, such as polylines and splines, and model curvature, continuity, and topology to improve structural consistency and post-processing performance [36]. Global relationship learning model design strategies integrate lane connection and branching relationships and scene context based on self-attention and graph convolutional networks, thereby maintaining global consistency under occlusion and viewpoint changes [37]. Low-level external perception methods have high real-time performance and ease of system integration; however, perception performance degradation occurs in real environments due to distant small objects, severe occlusion, and illumination and viewpoint changes. In addition, it is difficult to infer the semantic state of driving scenes because these methods are based on single-frame perception.

High-level traffic context recognition is a technique that recognizes the traffic context of driving scenes by integrating driving videos and data, such as objects extracted in the low-level external perception stage. Existing studies on recognizing traffic context include object relationship modeling-based techniques and global spatiotemporal representation learning-based techniques [38]. Object relationship modeling-based techniques construct graph structures using relative distances, relative velocities, lane-based adjacency, and interaction relationships among objects to recognize traffic context [39]. These techniques model interactions among objects in complex situations that are difficult to distinguish using simple density statistics through graph neural networks based on scene graphs and interaction graphs [40,41]. Object relationship modeling-based approaches have increasing computational costs depending on graph construction rules, object tracking quality, and the scale of the scene. Global spatiotemporal representation learning-based techniques learn global context and temporal patterns of video-level features and bird’s-eye view (BEV)-based spatial representations in an end-to-end manner [42]. These techniques utilize CNN- and Transformer-based video representations and learn scene-level semantics by integrating information on multiple objects and road structures [43,44]. Global spatiotemporal representation learning-based techniques require large-scale driving data and long-sequence learning, and they have high computational and memory costs due to the characteristics of Transformer-based global modeling. High-level TCR methods can infer the semantic states of complex driving scenes by integrating multimodal data and interactions among objects; however, their computational cost for graph construction and interaction modeling increases with the scale of the scene, making them difficult to apply in real environments.

Recent studies on traffic situation recognition have actively focused on recognition strategies based on integrated perception using BEV and 3D spatial representations, end-to-end driving strategies that connect perception, prediction, and planning, and contextual scene understanding strategies based on vision–language models (VLMs). Li et al. [45] proposed a BEVFormer-based 3D external object recognition system that learns BEV representations from multi-camera inputs. The proposed BEVFormer system enhances velocity estimation and object estimation performance under occlusion by using temporal self-attention, which exploits BEV features accumulated along the temporal axis, and spatial cross-attention, which integrates camera-view features into BEV queries. The BEVFormer-based 3D object recognition system was validated on the nuScenes database, achieving an NDS of 56.9%. Hu et al. [46] proposed a UniAD-based integrated autonomous driving perception, prediction, and planning system to optimize the autonomous driving stack in an end-to-end manner in BEV space. The proposed UniAD system does not separate perception, prediction, and planning results into individual modules but instead organizes them as end-to-end planning representations. The UniAD-based integrated autonomous driving perception, prediction, and planning system was validated on the nuScenes database with an minADE of 0.71 and on the planning benchmark database with an avg. L2 of 1.03 and an avg. Col of 0.31. Sima et al. [47] proposed DriveLM, a traffic situation recognition and driving reasoning system based on graph visual question answering, which connects objects, interactions, and planning and performs multi-step reasoning. The proposed DriveLM system organizes object, interaction, and planning QA into a graph structure and converts visual semantic information into language-based structured representations through graph prompting. The DriveLM-Agent-based object, interaction, and planning system was validated in the DriveLM-nuScenes open-loop evaluation, achieving a behavior accuracy of 83.70, a planning ADE of 1.39, and a Col of 1.67. Shao et al. [48] proposed an LMDrive-based closed-loop end-to-end driving traffic situation recognition system that connects driving command understanding, reasoning, and control using a large language model. The proposed LMDrive system integrates driving goals and constraints specified in natural language into the planning and control loop, demonstrating structural scalability for responding to unstructured commands. The LMDrive-based traffic situation recognition system was validated on the LangAuto database, achieving a driving score of 36.2, a route completion of 46.5, and an infraction score of 0.81.

Existing driver state recognition and traffic situation recognition systems have continuously advanced in terms of improving individual recognition performance and lightweight design in neural network-based learning models. However, because driver states and traffic situations are mutually related in real driving environments, MTL-based studies that can jointly learn and integratively infer them are required.

2.3. Multi-Task Learning for ADAS

MTL is a learning method that improves generalization performance by learning shared representations across multiple tasks. MTL methods are divided into hard parameter sharing and soft parameter sharing methods [49]. Hard parameter sharing methods share the parameters of early layers across all tasks and separate task-specific heads in the output layer, thereby improving parameter efficiency and alleviating overfitting. Soft parameter sharing methods maintain independent networks for each task and share parameters through regularization, distance constraints, and other mechanisms.

Xu et al. [50] proposed MTFormer, an MTL network that combines a Transformer architecture with cross-task reasoning. The proposed MTFormer system jointly learns multi-task representations through a shared Transformer encoder–decoder architecture and shares meaningful information across tasks by integrating lightweight task-specific branches, cross-task attention, and cross-task contrastive learning. The MTFormer system was validated on the NYUD-v2 database, achieving a semantic segmentation mIoU of 50.56, a depth estimation RMSE of 0.483, and an overall MTL performance metric δ_m of +4.12%. In addition, it was validated on the PASCAL database, achieving a semantic segmentation mIoU of 74.15, a human part segmentation mIoU of 64.89, a saliency detection mIoU of 67.71, and δ_m of +2.41%.

Recently, studies on MTL-based driver state and traffic situation recognition systems have been actively conducted to learn shared representations across individual tasks. Liu et al. [51] proposed TEM³-Learning, an efficient learning model that considers the interrelationships among drivers, traffic situations, and vehicles. The proposed TEM³-Learning system efficiently extracts spatiotemporal features and learns task-specific features using a Mamba-based multi-view temporal–spatial feature extraction subnetwork and an MTL-based gated multimodal feature integrator. The TEM³-Learning-based driver state and traffic situation recognition system was validated on the AIDE database, achieving a DER accuracy of 75%, a DBR accuracy of 69.31%, a TCR accuracy of 96.29%, a VBR accuracy of 81.68%, and an mAcc of 81.68%. Liu et al. [52] proposed UMD-Net, a multimodal and multi-task learning model that considers the interrelationships among drivers, traffic situations, and vehicles. The proposed UMD-Net system learns spatial feature extraction from multi-view images and complementary feature representations across multimodal inputs using a position-sensitive multi-directional attention feature extraction subnetwork and a recursive dynamic feature fusion method. The UMD-Net-based driver state and traffic situation recognition system was validated on the AIDE database, achieving a DER accuracy of 76.56%, a DBR accuracy of 71.88%, a TCR accuracy of 95.31%, a VBR accuracy of 82.81%, and an mAcc of 81.64%. Liu et al. [53] proposed MMTL-UniAD, a multimodal and multi-task learning framework for multi-task recognition of driver states and traffic contexts. The proposed MMTL-UniAD system learns task-shared and task-specific features through a multi-axis region attention network and a dual-branch multimodal embedding method, thereby alleviating negative transfer across tasks. The MMTL-UniAD-based driver state and traffic situation recognition system was validated on the AIDE database, achieving a DER accuracy of 76.67%, a DBR accuracy of 73.61%, a TCR accuracy of 93.91%, a VBR accuracy of 82.3%, and an mAcc of 82.30%.

MTL-based driver state and traffic situation recognition studies learn shared representations and task-specific representations that reflect the interrelationships among tasks. However, because they mainly focus on task-oriented design, modality-induced negative transfer occurs due to their limited ability to focus on task-relevant modality cues.

Table 1. Recent studies on driver state recognition and traffic context recognition.

Authors	Year	Dataset	Modalities	Method	DER	DBR	TCR	VBR
Saadi et al. [20]	2023	KMU-FED + FER2013	Driver face images	DFER-GCViT	✓	-	-	-
Saadi et al. [23]	2024	KMU-FED + KDEF	Driver face images	ShuffViT-DFER	✓	-	-	-
Duan et al. [3]	2023	SFD + AUCDD-V1	Face + hands video	FRNet	-	✓	-	-
Kim et al. [15]	2025	NTHU-DDD	Face + body video	STFTransNet	-	✓	-	-
Doshi [19]	2025	StateFarm	Driver images	Anchor-ViT	-	✓	-	-
Qu et al. [36]	2021	TuSimple + CULane	Front camera images	FOLOLane	-	-	✓	-
Ko et al. [35]	2022	TuSimple + CULane	Front camera images	PINet	-	-	✓	-
Yan et al. [37]	2025	TuSimple + CULane	Front camera images	MHFS-FORMER	-	-	✓	-
Wasi et al. [6]	2024	DAAD	Multi-view video + gaze	M2MVT	-	-	-	✓
Liu et al. [51]	2025	AIDE	Multi-view video + joints	TEM3-Learning	✓	✓	✓	✓
Liu et al. [52]	2025	AIDE	Multi-view video + joints	UMD-Net	✓	✓	✓	✓
Liu et al. [53]	2025	AIDE	Multi-view video + joints	MMTL-UniAD	✓	✓	✓	✓

Table 1. Recent studies on driver state recognition and traffic context recognition.

Authors	Year	Dataset	Modalities	Method	DER	DBR	TCR	VBR
Saadi et al. [20]	2023	KMU-FED + FER2013	Driver face images	DFER-GCViT	✓	-	-	-
Saadi et al. [23]	2024	KMU-FED + KDEF	Driver face images	ShuffViT-DFER	✓	-	-	-
Duan et al. [3]	2023	SFD + AUCDD-V1	Face + hands video	FRNet	-	✓	-	-
Kim et al. [15]	2025	NTHU-DDD	Face + body video	STFTransNet	-	✓	-	-
Doshi [19]	2025	StateFarm	Driver images	Anchor-ViT	-	✓	-	-
Qu et al. [36]	2021	TuSimple + CULane	Front camera images	FOLOLane	-	-	✓	-
Ko et al. [35]	2022	TuSimple + CULane	Front camera images	PINet	-	-	✓	-
Yan et al. [37]	2025	TuSimple + CULane	Front camera images	MHFS-FORMER	-	-	✓	-
Wasi et al. [6]	2024	DAAD	Multi-view video + gaze	M2MVT	-	-	-	✓
Liu et al. [51]	2025	AIDE	Multi-view video + joints	TEM3-Learning	✓	✓	✓	✓
Liu et al. [52]	2025	AIDE	Multi-view video + joints	UMD-Net	✓	✓	✓	✓
Liu et al. [53]	2025	AIDE	Multi-view video + joints	MMTL-UniAD	✓	✓	✓	✓

3. Perception and Recognition with Inter-Modal Selective Multi-Task Learning

This paper proposes PRISM-MTL, which dynamically selects important modality subspaces for each task rather than simply combining multimodal features, to address the modality-induced negative transfer problem in existing MTL-based driver state and traffic situation recognition systems, which fail to focus on task-relevant modality cues. As shown in Figure 2, PRISM-MTL consists of a multimodal encoder that extracts modality-specific tokens from heterogeneous multimodal inputs using HSA-Net and Token-SE and TSMF, which selectively emphasizes effective modality cues for each task using the extracted modality-specific tokens and learnable task tokens.

Table 2. Notation of the major symbols and variables used in PRISM-MTL.

Symbol	Description
$\mathcal{T}$	Task set (DER, DBR, TCR, VBR)
$\mathcal{M}$	Modality set (face, body, scene, posture, gesture)
E	Modality embedding dimension (E = 128)
D	Shared attention dimension of TSMF (D = 256)
H	Number of attention heads in TSMF (H = 2)
k	Kernel size of the spatial attention (k = 7)
tk	Learnable task token used as the query for the k-th task
Q, K, V	Query, key, and value in the cross-attention
αt,m	Task-conditioned modality relevance of task t for modality m
FFN	Feed-forward network refining task-specific features

summarizes the major symbols and variables used to describe the proposed PRISM-MTL, including modality/task counts, task tokens, and key hyperparameters.

3.1. Problem Formulation

This paper addresses a multimodal multi-task learning problem that simultaneously performs $K=4$ recognition tasks. The task set is defined as $\mathcal{T}=\{DER,DBR,TCR,VBR\}$, and the modality set is defined as $\mathcal{M}=\{f,b,s,p,g\}$. Here, $f$, $b$, $s$, $p$, and $g$ denote the driver face, driver body, scene, posture, and gesture, respectively.

Each training sample consists of heterogeneous multimodal inputs and a multi-task label set, as defined in Equation (1).

$$\mathcal{D}=\left\{\right({\mathcal{X}}_i,{\mathcal{Y}}_i){\}}_{i=1}^N,{\mathcal{X}}_i=\{X_i^m{\}}_{m\in \mathcal{M}},{\mathcal{Y}}_i=\{y_i^k{\}}_{k\in \mathcal{T}}.$$

(1)

$X_i^m$ denotes the input of modality $m$ for the $i$-th sample, and $y_i^k$ denotes the ground-truth label for task $k$. The objective of PRISM-MTL is to learn a mapping function $F_{\Theta }$ that simultaneously generates predictions for all tasks through a single multimodal learning framework, as defined in Equation (2).

$$F_{\Theta }\left({\mathcal{X}}_i\right)=\{{\widehat{y}}_i^k{\}}_{k=\mathcal{T}}^K.$$

(2)

Naive shared-fusion-based multimodal MTL integrates frame-level modality features into a single shared representation through concatenation or weighted summation.

$$z_t^{\mathrm{s}\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{d}}=\varphi \left([h_t^f;h_t^b;\cdots \hspace{0.17em};h_t^g]\right),$$

(3)

Naive shared-fusion is defined in Equation (3), where $h_t^m$ denotes the frame-level feature of modality $m$ at time step $t$. The naive shared-fusion formulation cannot finely reflect the different modality relevance of each task. Since DER, DBR, TCR, and VBR extract key information from different modality cues, sharing the same fused representation across all tasks causes modality-induced negative transfer, in which task-relevant modality cues are not sufficiently emphasized. To alleviate this problem, this paper designs TSMF, which uses learnable task tokens as queries and heterogeneous modality tokens as keys and values.

3.2. Multimodal Data Preprocessing

PRISM-MTL uses in-vehicle and out-vehicle multimodal inputs, and each sample is sampled as a clip composed of $T=16$ frames. The visual modalities consist of the driver face, driver body, and scene, while the joint modalities consist of posture and gesture.

Driver face and driver body are cropped from inside-view videos based on bounding box annotations. The driver face input $X^f$ and driver body input $X^b$ are constructed in the form of $B\times 3\times T\times H\times W$. The scene modality $X^s$ includes four camera views, namely front, left, right, and inside views, and is therefore defined in the form of $B\times 4\times 3\times T\times H\times W$. Here, $B$ denotes the batch size, $T$ denotes the clip length, and $H$ and $W$ denote the height and width of the input image, respectively. In the actual implementation, the scene input is constructed in the form of $B\times 3\times 4T\times H\times W$, and then the time and view axes are separated to average-aggregate the four-view scene features at each time step.

The joint input is used by separating the posture and gesture subsets from the entire keypoint set. Each keypoint is represented by an x-coordinate, y-coordinate, and confidence score. The posture keypoint tensor $K^p$ and gesture keypoint tensor $K^g$ are constructed in the forms of $B\times 3\times T\times K_p\times 1$ and $B\times 3\times T\times K_g\times 1$, respectively. Here, $K_p$ and $K_g$ denote the numbers of keypoints used for each modality. Posture reflects the global body configuration of the driver, while gesture reflects hand movement and fine-grained interaction cues.

3.3. Hierarchical Stage-Wise Attention Network

The visual modality encoder is constructed based on HSA-Net, as shown in Figure 3. HSA-Net attends to vertical, horizontal, and global spatial information inside and outside the vehicle by progressively applying different attention operators across four convolutional stages. For an input visual frame $x_t^m$, the stage-wise transformation of HSA-Net is defined in Equation (4).

$$F_s^m={\mathcal{A}}_s\left({\mathcal{C}}_s\left(F_{s-1}^m\right)\right),s\in \{\mathrm{1,2},\mathrm{3,4}\}.$$

(4)

$${\mathcal{A}}_1\left(F\right)=F.$$

(5)

This design prevents the loss of local details caused by excessive attention gating in the early stage. In Stage 2, directional spatial attention with a kernel size of $k$ = 7 is applied to emphasize structural patterns in the vertical and horizontal directions, which provides a sufficient receptive field to capture local structural cues while keeping the computational cost low. First, a two-channel spatial descriptor is generated through channel-wise average pooling and max pooling, as computed in Equation (6).

$$M\left(F\right)=\left[{\mathrm{A}\mathrm{v}\mathrm{g}}_c\right(F);{\mathrm{M}\mathrm{a}\mathrm{x}}_c(F\left)\right].$$

(6)

The directional attention gate is computed by combining vertical and horizontal convolutions, as defined in Equation (7).

$$G_{\mathrm{d}\mathrm{i}\mathrm{r}}=\sigma \left({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{k\times 1}\left(M\right(F\left)\right)+{\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{1\times k}\left(M\right(F\left)\right)\right),{\mathcal{A}}_2\left(F\right)=F\odot G_{\mathrm{d}\mathrm{i}\mathrm{r}}.$$

(7)

Here, $\sigma (\cdot )$ denotes the sigmoid function, and $\odot$ denotes element-wise multiplication. Directional spatial attention emphasizes structural features with strong horizontal and vertical directionality, such as lane boundaries, vehicle contours, and driver body silhouettes. Therefore, Stage 2 does not simply perform spatial attention; rather, it selectively emphasizes directional structural cues that repeatedly appear in driving environments and driver postures.

In Stage 3, spatial attention is applied to emphasize object-centric semantic saliency based on the structural features emphasized in Stage 2, as defined in Equation (8).

$$G_{\mathrm{s}\mathrm{p}\mathrm{a}}=\sigma \left({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}_{k\times k}\left(M\right(F\left)\right)\right),{\mathcal{A}}_3\left(F\right)=F\odot G_{\mathrm{s}\mathrm{p}\mathrm{a}}.$$

(8)

While the directional attention in Stage 2 emphasizes morphological structural information, the spatial attention in Stage 3 selectively emphasizes semantic regions that are important for task recognition. Examples include facial expression regions in the driver face, regions where body movement occurs, and salient regions related to the traffic state in the scene.

In Stage 4, channel-wise importance is reweighted for high-level semantic representations, as defined in Equation (9).

$$G_{\mathrm{c}\mathrm{h}}=\sigma \left(\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{G}\mathrm{A}\mathrm{P}\right(F\left)\right)+\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{G}\mathrm{M}\mathrm{P}\right(F\left)\right)\right),{\mathcal{A}}_4\left(F\right)=F\odot G_{\mathrm{c}\mathrm{h}}.$$

(9)

Here, $GAP(\cdot )$ and $GMP(\cdot )$ denote global average pooling and global max pooling, respectively. Channel attention enhances the discriminative power of the final visual representation by emphasizing semantic channels that are effective for task recognition in high-level feature maps.

Finally, the frame-level embedding for visual modality $m\in \{f,b,s\}$ is computed as shown in Equation (10).

$$h_t^m=P_m\left(\mathrm{G}\mathrm{A}\mathrm{P}\left({\Phi }_{\mathrm{H}\mathrm{S}\mathrm{A}}\left(x_t^m\right)\right)\right)\in {\mathbb{R}}^E.$$

(10)

${\Phi }_{HSA}(\cdot )$ denotes the HSA-Net-based visual encoding function, and $P_m$ is a learnable projection layer that transforms the backbone feature dimension into the modality embedding dimension $E$. The scene modality consists of four-view inputs. After extracting the HSA-Net features for each view, view-wise average aggregation is performed, as defined in Equation (11).

$$h_t^s={\displaystyle \frac{1}{4}}{\displaystyle {\sum }_{v=1}^4}{P}_s\left(\mathrm{G}\mathrm{A}\mathrm{P}\left({\Phi }_{\mathrm{H}\mathrm{S}\mathrm{A}}\left(x_{t,v}^s\right)\right)\right).$$

(11)

Equation (11) integrates the scene features extracted from the front, left, right, and inside views into a single frame-level scene token. Through this process, the face, body, and scene visual inputs are transformed into modality tokens with the same embedding dimension and used as inputs to TSMF.

3.4. Joint Modality Encoder with Token-SE

Posture and gesture are represented as keypoint coordinate sequences, unlike visual modalities. At each frame $t$, the keypoint tensor of joint modality $q\in \{p,g\}$ is flattened and then fed into an MLP encoder, as defined in Equation (12).

$$u_t^q={\mathrm{M}\mathrm{L}\mathrm{P}}_q\left(\mathrm{v}\mathrm{e}\mathrm{c}\left(K_t^q\right)\right)\in {\mathbb{R}}^E.$$

(12)

Here, $K_t^q$ denotes the joint tensor at time step $\mathrm{t}$, and $vec(\cdot )$ denotes an operation that transforms keypoint coordinates into vector form. $ML{P}_q(\cdot )$ generates the frame-level embedding for joint modality $q$.

The entire temporal joint embedding sequence is constructed as $U^q=[u_1^q,\dots ,u_T^q]\in {\mathbb{R}}^{B\times T\times E}$. In this paper, Token-SE is applied to the joint embedding sequence to emphasize important embedding channels across the entire temporal sequence. First, sequence-level channel descriptors are computed using temporal average pooling and temporal max pooling, and a channel-wise gate is generated through the shared excitation MLP $\psi (\cdot )$, as defined in Equation (13).

$$e^q=\sigma \left(\psi \left({\displaystyle \frac{1}{T}}{\displaystyle {\sum }_{t=1}^T}{u}_t^q\right)+\psi \left(\underset{t}{\mathrm{m}\mathrm{a}\mathrm{x}} u_t^q\right)\right).$$

(13)

The final joint modality token is computed as shown in Equation (14). Token-SE emphasizes joint embedding channels that are important for task recognition in posture and gesture sequences. The posture token reflects the global pose configuration of the driver, while the gesture token reflects hand movement and fine-grained interaction cues. Through this process, driver posture and hand movement information, which are difficult to capture using only visual modalities, is provided to TSMF as modality tokens.

$$h_t^q=u_t^q\odot e^q,q\in \{p,g\}.$$

(14)

3.5. Task-Specific Modality Fusion

Figure 4 presents the architecture of TSMF. Simple concatenation or weighted summation of heterogeneous modality features fails to reflect task-specific modality sensitivity and often introduces bias toward particular modalities. In contrast, TSMF selectively emphasizes multimodal features extracted by the multimodal encoder on a per-task basis and generates task-specific fusion sequences.

The architecture of TSMF is shown in Figure 4. TSMF constructs a modality token matrix by stacking five modality tokens along the modality axis at each time step $t$ through the multimodal encoder, as defined in Equation (15).

$$Z_t=\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{c}\mathrm{k}(h_t^f,h_t^b,h_t^s,h_t^p,h_t^g)\in {\mathbb{R}}^{B\times M\times E}.$$

(15)

Here, $h_t^f$, $h_t^b$, $h_t^s$, $h_t^p$, and $h_t^g$ denote the face, body, scene, posture, and gesture tokens, respectively. $M=5$ denotes the number of modalities, and $\mathrm{E}$ denotes the modality embedding dimension.

TSMF projects the modality tokens into a shared dimension $D$ for attention computation, as defined in Equation (16). In this study, the modality embedding dimension is set to $E = 128$ and the shared attention dimension to $D = 256$.

$${\widehat{Z}}_t=Z_t{W}_z\in {\mathbb{R}}^{B\times M\times D}.$$

(16)

Here, $W_z\in {\mathbb{R}}^{E\times D}$ is a learnable projection matrix. This projection aligns modality tokens generated from different sources into a common latent space, enabling task-to-modality attention.

$$Q=[q_1;q_2;\cdots \hspace{0.17em};q_K]\in {\mathbb{R}}^{K\times D}.$$

(17)

TSMF uses learnable task tokens corresponding to each task, as defined in Equation (17). Here, $q_k$ is the query for task $\mathrm{k}$, and the modality tokens ${\widehat{Z}}_t$ are used as keys and values. TSMF is a task-to-modality cross-attention operator that selectively emphasizes task-relevant modality cues among heterogeneous modality tokens based on task tokens.

Through task tokens, TSMF generates different task-specific representations for DER, DBR, TCR, and VBR from the same modality token set, thereby directly learning task-specific modality relevance in addition to task-level representation learning.

3.6. Multi-Head Task-to-Modality Cross-Attention

TSMF learns diverse modality interaction subspaces using multi-head cross-attention. For head $r\in \{1,\dots ,H\}$, the query, key, and value are defined in Equation (18). In this study, $H = 2$ attention heads are used, where the number of heads $H$ was selected based on the ablation study.

$$Q^{\left(r\right)}=\mathrm{L}\mathrm{N}\left(Q\right)W_Q^{\left(r\right)},K_t^{\left(r\right)}=\mathrm{L}\mathrm{N}\left({\widehat{Z}}_t\right)W_K^{\left(r\right)},V_t^{\left(r\right)}=\mathrm{L}\mathrm{N}\left({\widehat{Z}}_t\right)W_V^{\left(r\right)}.$$

(18)

Here, $W_Q^{\left(r\right)}$, $W_K^{\left(r\right)}$, and $W_V^{\left(r\right)}$ are the projection matrices of the $\mathrm{r}$-th attention head, and $d_h=D/H$ is the head dimension. Layer normalization stabilizes the scale of the query and modality tokens.

The attention coefficient assigned by task $k$ to modality $m$ and the head-specific fused representation are computed as shown in Equation (19).

$${\alpha }_{t,k,m}^{\left(r\right)}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}(Q_k^{\left(r\right)}{\left(K_{t,m}^{\left(r\right)}\right)}^{\mathrm{\top }}/\sqrt{d_h})}{{\displaystyle {\sum }_{m^{\prime }=1}^M}\mathrm{e}\mathrm{x}\mathrm{p}(Q_k^{\left(r\right)}{\left(K_{t,m^{\prime }}^{\left(r\right)}\right)}^{\mathrm{\top }}/\sqrt{d_h})}},O_{t,k}^{\left(r\right)}={\displaystyle {\sum }_{m=1}^M}{\alpha }_{t,k,m}^{\left(r\right)}{V}_{t,m}^{\left(r\right)}.$$

(19)

Here, ${\alpha }_{t,k,m}^{\left(r\right)}$ is the task-conditioned modality relevance coefficient for time $t$, task $k$, modality $m$, and attention head $r$. TSMF learns for each task to focus on the modality cues required for that task from the same multimodal token set.

After concatenating the outputs of all attention heads, an output projection is applied, and the task-specific representation is refined through a residual connection and a feed-forward network (FFN), as computed in Equation (20).

$${\tilde{X}}_t=X_t+\mathrm{F}\mathrm{F}\mathrm{N}\left(\mathrm{L}\mathrm{N}\left(X_t\right)\right),X_t=Q+\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(O_t^{\left(1\right)},\dots ,O_t^{\left(H\right)})W_O.$$

(20)

${\tilde{X}}_t\in {\mathbb{R}}^{B\times K\times D}$ is the task-specific fusion representation generated at time step $t$. Through Equations (15)–(20), TSMF operates not as simple feature concatenation but as a task-conditioned fusion mechanism that selectively emphasizes the modality cues required for each task. In particular, ${\alpha }_{t,k,m}^{\left(r\right)}$ can be interpreted as task-specific modality relevance learned without explicit modality supervision.

The key novelty of TSMF is that it does not simply combine modality information but selectively attends to heterogeneous modality tokens by using learnable task tokens as queries. The contribution of TSMF lies not in cross-attention itself but in its task-token-conditioned formulation for explicit modality selection. Conventional multimodal cross-attention typically uses a modality-derived feature as the query to generate a task-agnostic fused representation, whereas existing task-specific fusion approaches commonly rely on separate branches or gating modules for individual tasks. In contrast, TSMF uses learnable task tokens as queries and modality tokens as keys and values. All tasks share the same cross-attention operator, while their respective task tokens generate distinct modality relevance distributions and task-specific representations. Unlike the recursive fusion in UMD-Net, the task-shared and task-specific branches in MMTL-UniAD, and the task-specific gates in TEM³-Learning, TSMF performs task-to-modality routing within a single shared fusion module.

3.7. Clip-Level Prediction and Multi-Task Objective

The output of TSMF is a task-specific temporal fusion sequence $\tilde{X}\in {\mathbb{R}}^{B\times K\times T\times D}$. The clip-level representation for each task is computed through temporal mean pooling and layer normalization, as defined in Equation (21).

$$z_k=\mathrm{L}\mathrm{N}\left({\displaystyle \frac{1}{T}}{\displaystyle {\sum }_{t=1}^T}{\tilde{x}}_{t,k}\right).$$

(21)

Here, ${\tilde{x}}_{t,k}\in {\mathbb{R}}^D$ denotes the fusion representation generated for task $k$ at time step $t$. Temporal mean pooling summarizes the frame-level task-specific representations across the entire clip and generates a compact representation used for video-level prediction.

Each task-specific classifier takes $z_k$ as input and generates the prediction for each task. PRISM-MTL jointly optimizes the cross-entropy losses for DER, DBR, TCR, and VBR, as defined in Equation (22).

$${\widehat{y}}_k=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(W_k{z}_k+b_k\right), {\mathcal{L}}_{\mathrm{M}\mathrm{T}\mathrm{L}}={\displaystyle {\sum }_{k=1}^K}{\lambda }_k{\mathcal{L}}_{\mathrm{C}\mathrm{E}}({\widehat{y}}_k,y_k).$$

(22)

In this paper, the loss weights of all tasks are set equally. Specifically, ${\lambda }_k=1$, so DER, DBR, TCR, and VBR are learned with equal importance. Finally, PRISM-MTL optimizes all trainable parameters of HSA-Net, Token-SE, TSMF, and the task-specific prediction heads in an end-to-end manner.

4. Experimental Results

This section introduces the public assistive driving perception (AIDE), StateFarm [54], National Tsing Hua University driver drowsiness detection (NTHU-DDD) [55], and Honda driving dataset (HDD) [56] databases, along with the data preprocessing procedures, evaluation metrics, and implementation details. It then presents the performance comparison results between the proposed PRISM-MTL and existing models. Finally, the ablation study results of PRISM-MTL are reported, followed by visual analyses of the attention maps generated by HSA-Net and TSMF.

4.1. Dataset

The public AIDE database is composed of multi-view, multimodal, and multi-task data and consists of 2898 samples. The AIDE database was collected in real driving situations at a resolution of $1920\times 1080$ and 15 fps, and the samples were constructed in 3 s units, reflecting diversity in illumination conditions, weather environments, and traffic situations. Each sample includes front-view, left-view, right-view, and inside-view videos corresponding to multi-view data, face and body videos corresponding to multimodal data, and joint data for gesture and posture. The multi-task setting consists of DER, DBR, TCR, and VBR, and

Table 3. Class composition of each task.

Task	Count	Class
driver emotion	5	Anxiety, Peace, Weariness, Happiness, Anger
driver behavior	7	Smoking, Making Phone, Looking Around, Dozing off, Normal Driving, Talking, Body Movement
traffic context	3	Traffic Jam, Waiting, Smooth Traffic
vehicle behavior	5	Parking, Turning, Backward Moving, Changing Lane, Forward Moving

provides the class composition information for each task.

For the evaluation of PRISM-MTL, the public AIDE database was preprocessed. The driver face and body videos were cropped from the inside-view videos based on bounding box coordinate annotations and resized to $224\times 224$. In this study, each sample, composed of 45 frames, was uniformly sampled at intervals of three frames to ensure temporal alignment across multimodal inputs and maintain computational efficiency, and normalize into a 16-frame sequence per sample. The driver posture and gesture joints were mapped into the $\left(x,y,c\right)$ coordinate format based on joint annotations. Here, $x$ and $y$ denote the two-dimensional position coordinates of each joint, and $c$ denotes the confidence score of joint detection.

In addition, the public StateFarm, NTHU-DDD, and HDD databases were used to verify the generalization performance of the proposed PRISM-MTL. The StateFarm database consists of 22,424 driver images captured by an in-vehicle dashboard camera and includes 10 distracted driver behavior classes, such as safe driving, texting, talking on the phone, operating the radio, and drinking. The NTHU-DDD database is a driver drowsiness detection dataset collected from 36 drivers under day and night illumination conditions and glasses and sunglasses conditions, covering drowsiness-related situations such as normal driving, yawning, slow blinking, falling asleep, and laughing. The HDD database is a 104 h human driving dataset collected in real driving environments in the San Francisco Bay Area. It includes front-facing camera, GPS, and CAN-bus sensor information from an instrumented vehicle and is used to evaluate driver behavior and driving scene understanding.

4.2. Evaluation Metrics

The multi-task recognition performance of the proposed PRISM-MTL was evaluated using accuracy, which represents the correct classification rate; the F1-score, which is the harmonic mean of precision and recall; and mean average precision (mAP), which is the mean of class-wise average precision, as defined in Equations (23)–(26). In addition, the performance was evaluated using mAcc, which calculates the average accuracy across all tasks.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+FN+TN}}$$

(23)

$$F1-score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(24)

$$\mathrm{m}\mathrm{A}\mathrm{P}={\displaystyle \frac{1}{C}}{\displaystyle {\sum }_{c=1}^C}{\mathrm{A}\mathrm{P}}_c$$

(25)

$$mAcc={\displaystyle \frac{1}{m}}{\displaystyle {\sum }_{k=1}^m}{Acc}_k$$

(26)

Here, $m$ denotes the number of tasks in the multi-task setting, and $Ac{c}_k$ denotes the accuracy of each task.

4.3. Implementation Details

The hardware used to evaluate the performance of the proposed PRISM-MTL consisted of an Intel(R) Core i5-13600K CPU (Intel Corporation, Santa Clara, CA, USA), 32 GB RAM, and an NVIDIA RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA), and the software environment consisted of Ubuntu 22.04 and Visual Studio Code (version 1.126.0).

For fairness and reproducibility, all evaluations of PRISM-MTL were repeated five times under a multi-seed setting using the public AIDE, NTHU-DDD, StateFarm, and HDD databases. For each public database, PRISM-MTL was independently retrained rather than directly transferring or fine-tuning the weights learned on the AIDE database. StateFarm and NTHU-DDD were configured as single-task settings and HDD as a two-task setting, so that only the task heads were adjusted for each database, while the data splits, preprocessing, and training hyperparameters were kept identical across all databases. For the ablation and robustness studies, the seed was fixed at 42 under identical input and preprocessing settings. PRISM-MTL was trained for 50 epochs with a batch size of 8, using the AdamW optimizer with a learning rate of 1 × 10⁻⁴ and the GeLU activation function. The dataset was divided into training, validation, and test sets at a ratio of 64:16:20.

4.4. Comparison with the State-of-the-Art

To verify the generalization performance of the proposed PRISM-MTL, additional comparative experiments were conducted using the public AIDE database for multimodal and multi-task learning, the public StateFarm and NTHU-DDD databases for single-task learning, and the public HDD database for the two tasks of VBR and TCR. The StateFarm database consists of a DBR task for recognizing distracted driver behaviors, while the NTHU-DDD database consists of a driver state recognition task for recognizing driver drowsiness states. The HDD database consists of TCR and VBR tasks for recognizing vehicle driving states and traffic environment elements. In this experiment, we verified whether PRISM-MTL can learn effective representations not only in a multimodal multi-task setting such as AIDE but also in single-task driver state recognition environments and heterogeneous multimodal multi-task settings.

Figure 5 compares the driver state and traffic situation recognition performance of PRISM-MTL with existing studies on the public Assistive Driving Perception (AIDE) database. PRISM-MTL achieved state-of-the-art performance with DER, DBR, TCR, and VBR accuracies of 84.55% ± 0.91, 80.59% ± 1.67, 94.07% ± 0.80, and 85.79% ± 0.83, respectively, and an mAcc of 86.25% ± 0.35. Compared with MMTL-UniAD [53], PRISM-MTL improved DER, DBR, TCR, VBR, and mAcc by 7.88, 6.98, 0.16, 0.79, and 3.95 percentage points, respectively.

Table 4. Comparative analysis of existing studies and PRISM-MTL using the StateFarm database.

Author	Modality	Method	Acc. (%)	F1-Score
Khan et al. [16] (2023)	Driver area	EFFNet-CA	99.58	1.000
Li et al. [57] (2024)	Driver area	CoViT	97.89	-
Doshi [19] (2025)	Driver area	Anchor-ViT	92.30 ± 0.30	0.924
Kim et al. [15] (2025)	Driver area	STFTransNet	99.65 ± 0.13	0.996 ± 0.001
Ours	Face/Body	PRISM-MTL	99.55 ± 0.09	0.996 ± 0.001

presents the comparative analysis results of the proposed PRISM-MTL and existing methods for driver inattention recognition using the public StateFarm database. Based on face and body inputs, PRISM-MTL achieved an accuracy of 99.55% ± 0.09 and an F1-score of 0.996 ± 0.001. Compared with Anchor-ViT, PRISM-MTL improved the recognition accuracy by 7.32 percentage points. Although its accuracy was 0.10 percentage points lower than that of STFTransNet, PRISM-MTL showed comparable performance. These results verify that PRISM-MTL, although designed as an AIDE-based multimodal multi-task learning architecture, also achieves good generalization performance on a driver inattention recognition database.

Table 5. Comparative analysis of existing studies and PRISM-MTL using the NTHU-DDD database.

Author	Method	Model	Acc. (%)	F1-Score
Bai et al. [58] (2022)	Face landmark	2s-STGCN	92.70	0.881
Yang et al. [59] (2024)	Face area	VBFLLFA	91.30	-
Huang et al. [60] (2024)	Face area	SMGA-Net	81.00	0.811
Kim et al. [15] (2025)	Driver area	STFTransNet	95.86 ± 0.17	0.957 ± 0.002
Ours	Face/Body	PRISM-MTL	96.47 ± 0.21	0.964 ± 0.002

presents the comparative analysis results of PRISM-MTL and existing methods for driver drowsiness recognition using the public NTHU-DDD database. Based on face and body inputs, PRISM-MTL achieved an accuracy of 96.47% ± 0.21 and an F1-score of 0.964 ± 0.002. Compared with SMGA-Net and STFTransNet, PRISM-MTL improved recognition accuracy by 15.47 and 0.61 percentage points, respectively. These results verify that PRISM-MTL achieves good generalization performance in driver drowsiness recognition. Bold values indicate the best performance.

Table 6. Comparative analysis of existing studies and PRISM-MTL using the HDD database.

Author	Method	Model	mAP (%)
Chen et al. [61] (2022)	Sensors	GateHUB	32.1
Wang et al. [62] (2023)	Sensors	MAT	32.7
Cao et al. [63] (2023)	Scene	E2E-LOAD	48.1
Guo et al. [64] (2024)	Sensors	BEDL	33.0
Ours	Scene/Sensors	PRISM-MTL	64.45 ± 0.01

presents the comparative analysis results of PRISM-MTL and existing studies for driving behavior recognition using the public HDD database. PRISM-MTL achieved a VBR accuracy of 89.06% ± 0.01, an F1-score of 0.629 ± 0.001, and an mAP of 64.45% ± 0.01 on the HDD database. In terms of VBR mAP, PRISM-MTL outperformed GateHUB, MAT, E2E-LOAD, and BEDL by 32.35, 31.75, 16.35, and 31.45 percentage points, respectively. PRISM-MTL also performed TCR, achieving a TCR accuracy of 81.53% ± 0.01 and an F1-score of 0.570 ± 0.001, with an overall mAcc of 85.29% ± 0.01. These results show that PRISM-MTL effectively learns traffic context and vehicle behavior cues on the HDD database. Since existing HDD-based studies mainly evaluate goal-oriented driving behavior mAP, which corresponds to VBR, this paper compares PRISM-MTL with existing studies based on VBR mAP and presents TCR and mAcc as additional multi-task recognition results.

To verify that the observed gains were not attributable to random variation, a one-sample t-test (one-sided, $n$ = 5) against the best competing method confirmed significant improvements (p < 0.05) in AIDE mAcc, StateFarm and NTHU-DDD accuracy, and HDD VBR mAP. For an additional controlled significance analysis on the public AIDE database, we constructed a baseline by replacing the cross-attention mechanism of TSMF with concatenation-based fusion and removing HSA-Net. The controlled baseline and PRISM-MTL were evaluated using the same five seeds and data splits, and their seed-matched results were compared using a one-sided paired t-test and an exact one-sided Wilcoxon signed-rank test. PRISM-MTL significantly improved mAcc by 2.33 percentage points over the controlled baseline (paired t-test: p < 0.001; Wilcoxon signed-rank test: p < 0.05). Significant improvements were also observed in mF1 (+3.39 percentage points) and DER, DBR, and TCR accuracy (p < 0.05), whereas the 0.93-percentage-point improvement in VBR accuracy was not statistically significant.

4.5. Ablation Studies

In this section, ablation studies were conducted to analyze the contribution of each component of the proposed PRISM-MTL to driver state and traffic situation recognition. The experiments were designed to evaluate the stage-wise attention configuration of HSA-Net, the contribution of multimodal input groups, the effectiveness of multi-task learning, the influence of task combinations, the individual contributions of HSA-Net and TSMF, the effectiveness of task-specific modality fusion, and the effect of the TSMF multi-head configuration. In addition, qualitative visualizations were provided to examine the stage-wise attention responses of HSA-Net and the task-specific modality attention patterns learned by TSMF.

Table 7. Ablation study results on HSA-Net stages.

HSA-Net Stage			DER	DBR	TCR	VBR	mAcc. (%)
Stage 2	Stage 3	Stage 4	Acc.	Acc.	Acc.	Acc.
√			83.68	79.14	93.96	85.55	85.58
	√		84.69	78.79	93.45	86.03	85.74
		√	85.17	78.79	93.45	84.83	85.56
√	√		82.59	76.72	94.48	85.34	84.78
	√	√	84.31	79.31	94.66	85.69	85.99
√		√	84.48	79.14	94.83	85.00	85.86
√	√	√	83.79	79.48	95.34	86.38	86.25

presents the stage attention ablation results for analyzing changes in multi-view data processing performance according to the stage-wise attention configuration of HSA-Net. The results show that when the stage-wise attention components of HSA-Net were ablated, mAcc decreased by up to 1.47 percentage points and by at least 0.26 percentage points, verifying the effectiveness of the hierarchical stage-wise attention structure.

Table 8. Ablation study results on multimodal inputs.

Multimodal Data			DER	DBR	TCR	VBR	mAcc. (%)
Face + Body	Scene	Post + Gest	Acc.	Acc.	Acc.	Acc.
√			83.28	79.31	90.52	82.24	83.84
	√		83.28	78.28	94.48	85.86	85.86
		√	70.00	71.90	84.48	74.31	75.17
√	√		85.34	77.07	93.10	85.86	85.34
	√	√	84.31	79.48	93.62	85.86	85.82
√		√	82.07	76.38	90.17	82.24	82.72
√	√	√	83.79	79.48	95.34	86.38	86.25

presents the ablation study results for the face, body, scene, posture, and gesture to examine the contribution of each modality and the effect of multimodal input configurations. In addition, when the face + body group or the posture + gesture group was removed, PRISM-MTL maintained an mAcc of 85.82% and 85.34%, respectively, corresponding to a degradation within 1 percentage point relative to the full-modality model, which demonstrates robustness to partial modality absence in real driving situations. When the scene group was removed, the mAcc decreased to 82.72%, a drop of 3.53 percentage points relative to the full-modality model, confirming that scene cues serve as important information for driver-state and traffic situation recognition.

Table 9. STL performance for each task.

Multi-Task				DER	DBR	TCR	VBR
DER	DBR	TCR	VBR	Acc.	Acc.	Acc.	Acc.
√				81.14	-	-	-
	√			-	79.14	-	-
		√		-	-	94.48	-
			√	-	-	-	86.21
√	√	√	√	83.79	79.48	95.34	86.38

presents the comparison results between STL-based prediction for each task and MTL performance to analyze performance changes caused by inter-task interactions. Compared with MTL, STL-based prediction decreased task accuracy by up to 2.65 percentage points and by at least 0.17 percentage points, demonstrating the interrelationships among tasks and verifying the necessity of MTL.

Table 10. Ablation study results on multi-task configuration.

Multi-Task				DER	DBR	TCR	VBR
DER	DBR	TCR	VBR	Acc.	Acc.	Acc.	Acc.
√	√			82.76	78.97	-	-
		√	√	-	-	92.93	86.03
√	√	√		84.83	79.14	92.93	-
√	√		√	84.31	78.62	-	85.17
√		√	√	80.86	-	92.93	86.27
	√	√	√	-	77.07	93.79	85.69
√	√	√	√	83.79	79.48	95.34	86.38

presents the task ablation study results for DER, DBR, TCR, and VBR to verify the effectiveness of four-task MTL. The task ablation studies were divided into configurations consisting of highly related driver state tasks, namely DER and DBR; traffic situation tasks, namely TCR and VBR; and configurations in which one of the four tasks was excluded. In the ablation studies on driver state tasks and traffic situation tasks, DER, DBR, TCR, and VBR accuracies decreased by 2.07, 0.17, 2.41, and 0.35 percentage points, respectively. In the experiments where one of the four tasks was ablated, DBR accuracy decreased by up to 2.41 percentage points, TCR accuracy decreased by up to 2.41 percentage points, and VBR accuracy decreased by up to 1.21 percentage points, verifying the superior performance of four-task MTL.

Table 11. Method ablation study results and model complexity.

Method		DER	DBR	TCR	VBR	mAcc. (%)	Params (M)	FLOPs (G)
HSA-Net	TSMF	Acc.	Acc.	Acc.	Acc.
		82.41	81.21	93.28	81.38	82.41	34.51	175.07
√		82.59	75.34	91.72	82.93	83.15	34.61	175.09
	√	82.93	79.45	93.62	85.17	85.29	34.68	175.12
√	√	83.79	79.48	95.34	86.38	86.25	34.77	175.13

presents the method ablation study results and model complexity comparison results for evaluating the individual contributions of HSA-Net and TSMF, which are the core components of PRISM-MTL. The method ablation study results show that removing each component decreased mAcc by up to 3.84 percentage points and by at least 0.96 percentage points, verifying the contributions of HSA-Net and TSMF to PRISM-MTL. PRISM-MTL is a lightweight architecture with 34.77M parameters and 175.13G FLOPs. On an NVIDIA RTX 4090 with a batch size of 8, it achieves 82.10 ms per batch (97.44 samples/s), supporting its potential for real-time processing in practical driving environments. Its peak allocated GPU memory usage is 2.63 GB under the same setting. TEM³-Learning reports approximately 5.99 M parameters and an inference speed of 142.32 FPS. Although PRISM-MTL requires more parameters, it achieves a substantially higher AIDE mAcc of 86.25%, compared with 81.68% for TEM³-Learning. Because these measurements were obtained using different hardware and batch-size settings, the reported inference speeds should not be interpreted as a direct comparison.

Table 12. Ablation study results on task-specific modality fusion.

Method	DER	DBR	TCR	VBR	mAcc. (%)
	Acc.	Acc.	Acc.	Acc.
task-agnostic attention	83.62	76.55	93.79	85.52	84.87
TSMF	83.79	79.48	95.34	86.38	86.25

presents the comparative experimental results between TSMF and the task-agnostic attention method, which performs cross-attention among modalities without using task tokens, to verify the effectiveness of TSMF. Task-agnostic attention is applied by sequentially using each modality as a query. The experimental results show that TSMF outperformed the task-agnostic attention method by 0.17, 2.93, 1.55, 0.86, and 1.38 percentage points in DER accuracy, DBR accuracy, TCR accuracy, VBR accuracy, and mAcc, respectively, demonstrating the superiority of the task token learning strategy in TSMF.

Table 13. Ablation study results on TSMF attention head configuration.

TSMF Head Rate	DER	DBR	TCR	VBR	mAcc. (%)
	Acc.	Acc.	Acc.	Acc.
1 Head	84.31	78.45	94.66	86.03	85.86
2 Head	86.38	78.62	94.66	85.34	86.25
4 Head	84.66	78.10	94.48	86.72	85.99
8 Head	84.31	79.66	93.28	86.90	86.03

presents the ablation results according to the number of attention heads in TSMF. The 2-head setting achieved the highest overall performance, with an mAcc of 86.25%, which was 0.39 percentage points higher than that of the 1-head setting. The 4-head and 8-head settings achieved mAcc values of 85.99% and 86.03%, respectively, showing lower performance than the 2-head setting. These results indicate that increasing the number of heads does not directly lead to performance improvement. The TSMF attention head ablation study results show that although TSMF can learn diverse modality interaction subspaces through multi-head cross-attention, an appropriate number of heads should be selected by considering the embedding dimension and the number of tasks.

Table 14. Ablation study results on embedding dimension.

Embedding Dimension (E, D)	DER	DBR	TCR	VBR	mAcc. (%)
	Acc.	Acc.	Acc.	Acc.
(64, 128)	84.31	79.48	93.28	85.52	85.65
(128, 256) default	83.79	79.48	95.34	86.38	86.25
(256, 512)	84.83	79.48	93.97	84.48	85.69

presents the embedding dimension ablation results, analyzing the performance changes according to the sizes of the modality embedding dimension E and the shared attention dimension D. The default setting (E = 128, D = 256) achieved the highest mAcc of 86.25% among the three settings, and it recorded the highest accuracies for TCR at 95.34% and VBR at 86.38%. The (E = 64, D = 128) setting achieved an mAcc of 85.65%, 0.60 percentage points lower than the default setting. The (E = 256, D = 512) setting achieved the highest DER at 84.83%, but its mAcc was 85.69%, 0.56 percentage points lower than the default setting. The DBR accuracy was identical at 79.48% across the three settings. These results show that increasing the embedding dimension does not directly lead to performance improvement and that the dimension should be set by considering the number of tasks and the number of modalities. We used the (E = 128, D = 256) setting as the default.

Figure 6 shows the visualization results of the stage-wise attention in HSA-Net. Stage 1 broadly responds to local visual patterns across the input image, while Stage 2 emphasizes structural cues, such as vehicle contours, road boundaries, and driver body silhouettes. Stage 3 selectively highlights spatially salient regions that are important for task recognition, and Stage 4 shows that attention is concentrated on key regions containing high-level semantic cues. These results demonstrate that HSA-Net progressively refines visual information from local features to semantic representations as the stages become deeper, qualitatively verifying the effectiveness of the proposed HSA-Net.

Figure 7 shows the visualization of the modality cues on which the proposed TSMF focuses on for each task. Figure 7a shows the task-modality attention heatmap averaged over the temporal axis. The DER task showed relatively high attention to the body, face, and scene modalities, while the DBR task showed the highest attention to the body modality. In addition, the TCR and VBR tasks showed the highest attention to the scene modality, confirming the importance of scene cues in traffic-related tasks. These task-wise attention patterns show that TSMF selectively refers to the modalities required for the recognition of each task, thereby alleviating the negative transfer caused by task-irrelevant modalities. Figure 7b shows the frame-wise changes in task-modality attention within the input sequence. In the DER and DBR tasks, attention to the face and body modalities remained generally high, whereas in the TCR and VBR tasks, the scene modality showed high attention in most frames. These results further confirm that TSMF effectively learns task-specific modality relevance.

Figure 8 shows the task-token attention distribution for four representative test samples. In all four samples, the driver-centric tasks DER and DBR assign high attention to the face and body modalities, and the traffic-centric tasks TCR and VBR assign high attention to the scene modality. The per-sample distributions differ, but the task-to-modality selection trend is consistent across samples. These results show that each task does not share a single fused representation but instead selectively uses different modalities.

5. Conclusions

Although current ADAS technologies have advanced, road traffic accidents continue to occur. Existing driver state and traffic situation recognition systems are based on STL and investigate each task independently, which limits their ability to reflect inter-task correlations. The proposed PRISM-MTL is an integrated multimodal and multi-task learning framework that simultaneously recognizes DER, DBR, TCR, and VBR. PRISM-MTL improves the negative transfer problem through multimodal data preprocessing, a multimodal encoder for extracting spatial features from multimodal data, and TSMF for learning task-specific modality features. Experimental results show that the proposed PRISM-MTL achieves state-of-the-art performance on the public AIDE database, with a DER accuracy of 84.55% ± 0.91 and an F1-score of 0.791 ± 0.012, a DBR accuracy of 80.59% ± 1.67 and an F1-score of 0.757 ± 0.035, a TCR accuracy of 94.07% ± 0.80 and an F1-score of 0.904 ± 0.010, a VBR accuracy of 85.79% ± 0.83 and an F1-score of 0.754 ± 0.015, an mAcc of 86.25% ± 0.35, and an mF1 of 0.802 ± 0.010. The proposed PRISM-MTL achieves a lightweight architecture and real-time inference speed and remains robust even under partial modality absence. However, the full four-task multi-task setting was validated only on the single AIDE database, and because PRISM-MTL relies on synchronized multi-view cameras and a separate pose-estimation pipeline, its generalization and deployment in real vehicle environments require further validation. In future work, we plan to study a lightweight MTL system based on quantization-based compression for the application of MTL in real driving environments.