Cross-Modal Behavioral Intelligence in Regard to a Ship Bridge: A Rough Set-Driven Framework with Enhanced Spatiotemporal Perception and Object Semantics

Abstract

Aberrant or non-standard operations by ship drivers are a leading cause of water traffic accidents, making the development of real-time and reliable behavior detection systems critically important. However, the environment within a ship’s bridge is significantly more complex than typical scenarios, such as vehicle driving or general security monitoring, which results in poor performance when applying generic algorithms. In such settings, both the accuracy and efficiency of existing methods are notably limited. To address these challenges, this paper proposes a cross-modal behavioral intelligence framework designed specifically for a ship’s bridge, integrating multi-target tracking, behavior recognition, and feature object association. The framework employs ByteTrack, a high-performance multi-object tracker that maintains stable tracking even when subject to occlusions or motion blur through its novel association mechanism, using both high and low confidence detection boxes, for multi-driver tracking. Combined with an improved Temporal Shift Module (TSM) algorithm for behavior recognition, which effectively resolves issues concerning target association and action ambiguity in complex environments, the proposed framework achieves a Top-1 accuracy of 82.1%, based on the SCA dataset. Furthermore, the method incorporates a multi-modal decision optimization strategy, based on spatiotemporal correlation rules, leveraging YOLOv7-e6 for simultaneous personnel and small object detection, and introduces the Accuracy of Focused Anomaly Recognition (AFAR) metric to enhance the anomaly detection performance. This approach improves the anomaly detection rate, up to 81.37%, with an overall accuracy of 80.66%, significantly outperforming single-modality solutions.

1. Introduction

A ship’s bridge, serving as the command center, is a critical workspace, equipped with various navigation and communication tools, such as radars, a Global Positioning System (GPS), electronic chart systems, an Automatic Identification System (AIS), a Very High Frequency (VHF) radio communication system, and more. Multiple ship drivers are typically required to operate the ship and process the large amount of information coming from diverse equipment. Ship crew members who violate maritime transportation regulations face severe consequences, including as a result of causing serious injuries, substantial property damage, significant water pollution, and other adverse effects. For instance, on 6 August 2017, a recreational motor cruiser, James 2, and a commercial fishing vessel, Vertrouwen, collided in Sussex Bay, southeast of Shoreham harbor, resulting in James 2 sinking and three of its four crew members drowning, due to ineffective lookouts, a lack of lifejackets, improper navigation lights, and the watchkeeper being distracted by social media and vessel administration [1]. This incident underscores the necessity of the MCA’s MGN 638 [2], which provides essential guidance on maintaining proper lookouts and avoiding distractions to enhance maritime safety. Therefore, it is crucial to enhance ship safety by developing an autonomous recognition system for monitoring crew behavior.

While driver behavior identification in automobiles has made significant progress [3], similar technologies have found limited application within the domain of ship navigation. Compared to automobiles, the environment within a ship’s bridge is more complex. Firstly, the ship’s bridge is notably spacious. In contrast to a car, which typically has only one driver, a ship requires multiple crew members to operate the vessel together. Furthermore, automobile drivers are generally restricted to a seated position when driving. Ship crew may exhibit various postures, such as walking, sitting, or standing, with a significantly broader range of operational areas. Additionally, due to issues related to lighting or occlusion, the accuracy in detecting small objects, such as cigarettes or mobile phones, tends to be lower on a ship. Many algorithms perform excellently in scenarios with well-defined boundaries [4]; however, their efficacy diminishes significantly within the complex environment of a ship’s bridge. The challenges faced in this environment increase the requirements for developing and optimizing behavior recognition algorithms.

Based on the input, behavior recognition algorithms can be classified into two categories: single-frame recognition and multi-frame recognition. Single-frame recognition algorithms attempt to identify actions from a single image. Object detection algorithms can help determine whether a crew member is smoking or playing on a phone by detecting cigarettes, mobile phones, or other items in the crew member’s hand. However, these objects are often small and have ambiguous features, leading to higher chances of false detection, especially in complex environments, like a ship’s bridge. Consequently, object detection algorithms may produce unsatisfactory results, due to their limitations in regard to handling such ambiguity. In contrast, multi-frame behavior recognition algorithms process multiple consecutive frames simultaneously. By analyzing the sequences of frames, these algorithms can effectively extract motion features, such as the act of raising a hand while smoking or turning the steering wheel while driving, along with other temporal dynamics. This capability significantly enhances the feature extraction performance, providing more robust and accurate behavior identification over time. Moreover, the collection and analysis of high-quality data are crucial for maximizing model performance. The use of high-quality data not only enhance the training process, but also ensure that the model can generalize well to new, unseen scenarios, thereby substantially increasing its recognition accuracy.

To address these gaps, this paper introduces a novel cross-modal behavioral intelligence framework, driven by rough set theory, which uniquely integrates spatiotemporal perception and object semantics, for ship bridge environments. Our approach distinguishes itself from prior work in regard to the following three key aspects:

(1) Original dataset construction. We present the first publicly available ship crew behavior dataset, the SCA. This dataset comprises 4788 annotated action sequence samples, captured from video surveillance systems within the ship bridge of passenger vessels operated by Zhoushan Haihua Passenger Transport Co., Ltd. During navigation, these vessels maintain a bridge team consisting of one helmsman, and two lookouts. The SCA dataset covers seven behavior categories: sitting, standing, walking, driving, lying, playing on a phone, and smoking. Each behavior sample contains a minimum of eight consecutive frames to ensure adequate temporal context for action recognition. Additionally, we provide a specialized subset of 3400 images, with detailed object-level annotations (humans, phones, cigarettes), to support feature object detection tasks. By combining comprehensive behavior sequences with precise object annotations in authentic maritime operational contexts, the SCA dataset addresses a significant research gap and enables reproducible studies in regard to maritime computer vision applications.

(2) Hierarchical recognition architecture with enhanced spatiotemporal modeling. We propose a lightweight framework that innovatively combines ByteTrack multi-object tracking with an improved Temporal Segment Network (TSM). Our novel integration of EfficientNet-B3, as the backbone network for computational efficiency, and Coordinate Attention (CA) mechanisms, for spatiotemporal feature enhancement, achieves a Top-1 accuracy of 82.1%, based on the SCA dataset, surpassing existing methods, in complex ship environments.

(3) Rough set-driven decision optimization. Departing from conventional unimodal approaches, we introduce a feature object-assisted method that establishes spatiotemporal association rules between personnel and objects via rough set theory. The proposed joint reasoning engine and multi-source fusion strategy increase the Accuracy of Focused Anomaly Recognition (AFAR) by 7.07% and the Comprehensive Classification Accuracy (CCA) by 2.2% above the baseline methods.

2. Literature Review

2.1. Multi-Object Tracking

Multi-object tracking involves identifying moving targets in each frame of a video series and assigning the same identification label to the same object across successive frames, resulting in separate motion trajectories for individual objects [5].

Wojke et al. [6] proposed the DeepSORT algorithm, which utilized a proprietary residual network trained on pedestrian re-identification datasets to extract deep appearance features mapped onto a hypersphere. These features were then used to determine the minimal cosine distance between identified and tracked targets. During the association phase, the Mahalanobis distance and cosine distance, linearly weighted according to the tracking prediction box generated from Kalman filtering and the detection box in the current frame, were used to compute a cost matrix. Cascade matching was then applied to pair the anticipated boxes with the detected boxes. This compact tracking strategy incorporated motion and appearance information, achieving high multi-object tracking accuracy (MOTA) in real time, while efficiently minimizing occlusion concerns. Zhang et al. [7] presented ByteTrack, a simple, efficient, and general method for data association. Instead of focusing only on high-scoring detection boxes, it associates each detection box with a track by leveraging its resemblance to pre-existing trajectories. This approach utilizes the similarities between low-scoring detection boxes and existing tracks to distinguish genuine targets from background detections. Both high- and low-scoring detection boxes were fully utilized in regard to this method, which can be directly integrated into standard tracking techniques, such as DeepSORT, resulting in varying degrees of improvement. With the growing popularity of Transformers, new techniques like TrackFormer [8] and TransTrack [9] have emerged, which combine Transformers with multi-object tracking. These techniques enhance tracking performance by drawing inspiration from the principles of Transformer encoders and decoders. However, the computational burden they introduce is significant and cannot be overlooked.

Multi-target tracking algorithms currently have a wide range of applications. Hong et al. [10] combined DeepSORT with YOLOv5 in order to follow Bactrocera minax and recognize its grooming behavior. Although this method provided a viable approach for multi-object behavior identification, the tracker still requires significant improvements. Wang et al. [11] used multi-target tracking for pedestrian detection to gather data for autonomous vehicle driving. They enhanced the YOLOv7 backbone network by incorporating spatial depth convolutional layers and applied the concept of convolutional structural reparameterization to create a comprehensive feature extraction block. This optimized block was integrated into the DeepSORT tracker, resulting in an improved system for pedestrian detection. Sukkar et al. [12] combined YOLOv8 with StrongSORT, systematically tuning the parameters across various dimensions to achieve optimal adaptability. From a vehicular perspective, these two techniques show promising results in addressing pedestrian tracking issues. Zhao et al. [13] introduced a new Intersection over Union (IoU) calculation method and applied it to SORT, DeepSORT, and other methods, achieving excellent results in regard to the multi-object tracking of ships in waterways. However, due to the limited experimental circumstances employed to evaluate this strategy, its generalizability cannot be guaranteed.

2.2. Behavior Recognition

With the development of hardware resources in recent years, behavior recognition has become a hot topic. Many methods have added increasingly deeper network layers to improve recognition efficacy, which has led to an explosion in the number of parameters involved. However, using large models to enhance recognition invariably results in a higher threshold for model usage. Achieving a balance between accuracy and model size is crucial in many application scenarios.

Convolutional neural network (CNN)-based behavior recognition algorithms are typically divided into two types, namely 2D CNN and 3D CNN, depending on whether they employ convolution in the time dimension. While 3D CNN-based methods can achieve promising performance, they are computationally intensive. To address this issue, several studies have aimed to reduce the computational complexity and simplify the structure to enhance the real-time performance of 3D convolutions, with notable examples including C3D [14], I3D [15], P3D [16], and R (2 + 1) D [17]. Although these methods simplify the structure and enhance the performance over typical 3D convolutions, they still struggle with real-time processing.

While 2D convolution is less effective than 3D convolution at capturing temporal features, it requires significantly fewer parameters and computational resources. This makes 2D convolution more suitable for real-time applications [18]. Many pieces of research have proposed methods to extract features similar to those achieved with 3D convolution using 2D convolution. For instance, Wang et al. [19] introduced the Temporal Segment Network (TSN), a video classification algorithm based on a 2D CNN. TSN employs a sparse temporal sample strategy and video-level supervision to effectively learn from entire action videos. It consists of spatial stream ConvNets and temporal stream ConvNets that process short, sparsely sampled snippets from the video sequence. Each stream generates an initial action class prediction, and the final video-level prediction is formed by aggregating predictions from these snippets. However, this method does not adequately capture the temporal relationships between snippets, which limits its ability to accurately recognize events defined by precise timing cues.

Lin et al. [20] proposed TSM, a generic and effective module, with high efficiency and performance. TSM shifts part of the channels along the temporal dimension on the feature map, thereby increasing information exchange among neighboring frames. It can be integrated into a 2D CNN to achieve temporal modelling, without the need for additional computation or parameters. TSM performs well in scenarios with clear boundaries. In the context of sharing internal video footage from self-driving cars, Rodrigues et al. [21] proposed a fusion surveillance solution for identifying violent and other unruly behaviors of car occupants, using algorithms such as SlowFast, TSN, and TSM to achieve excellent recognition and real-time performance. Xue et al. [22] suggested a behavior recognition model based on SlowFast and the EPSA attention mechanism to address the issues arising from complex lighting conditions. However, the effectiveness of these methods is significantly hindered in environments with dim lighting and unclear boundaries.

2.3. Feature Extraction Network

Since feature extraction networks form the foundation of image processing algorithms, they are crucial for recognition algorithms. With the advent of ResNet [23], these networks have seen significant improvements in recent years. The primary innovation of ResNet is the introduction of residual connections, which facilitate the training of extremely deep layers, while mitigating the vanishing gradient problem. ResNet has demonstrated impressive performance in tasks such as image classification, object detection, and semantic segmentation. Due to the increased use of embedded systems and mobile devices, there is a growing demand for lightweight 2D feature extraction networks. Traditional deep networks often perform inadequately under resource-constrained conditions, necessitating more efficient solutions. The introduction of the MobileNet [24] series addressed this need. These networks utilize depthwise separable convolutions, which reduce the computational requirements by employing separate depthwise and pointwise convolutions. Consequently, they are well-suited to mobile devices and embedded systems. EfficientNet [25] further advanced this field by proposing automated network structure adjustments. This approach optimizes model performance by balancing the network depth, width, and resolution. Not only does this increase model efficiency, but it also enables automatic adjustments based on the specific requirements of different tasks.

Research on using the EfficientNet network for behavior recognition is currently underway. Luo et al. [26] applied an improved version of the EfficientNet backbone network to classify behavior data collected by LiDAR, albeit based on a limited dataset. Zhou et al. [27] integrated EfficientNet as the feature extraction component in a dual-stream network and enhanced its performance with a multi-head attention mechanism, effectively addressing the challenge of distinguishing similar actions. For detecting car driver behavior, Khan et al. [28] proposed incorporating the EfficientNet-B0 network into a channel attention mechanism. Through the use of comparative analysis with established methodologies, they achieved the highest recognition accuracy with their proposed model. However, it is important to note that channel attention only establishes relationships at the channel level. Incorporating multidimensional attention mechanisms into the backbone network could further improve model performance.

2.4. Rough Set

Rough set (RS) theory, introduced by Zdzislaw Pawlak [29], is a mathematical tool to deal with vagueness and uncertainty. The core concept of RS theory involves utilizing existing knowledge bases to approximate and characterize imprecise or uncertain information through established knowledge repositories. This method provides an objective way to describe and process uncertainties without requiring any prior information beyond the dataset being processed. It relies solely on intrinsic data relationships derived from equivalence classes and indiscernibility relations.

Integrating RS with machine learning algorithms can enhance the accuracy of behavior classification. By learning from historical behavior data, systems can predict behaviors that may occur in the future. Senthilnathan [30] developed a three-level model based on RS to understand complex Indian consumer retail shopping behavior in the grocery segment, treating behavioral traits as rules. Henry [31] incorporated RS with reinforcement learning, which could be applied to a monocular vision system for tracking moving targets and studying swarm behavior. Chu et al. [32] introduced a multi-modal and multi-criteria conflict analysis approach, combining RS with deep learning. This approach used a deep residual network as a feature extractor and classified objects according to decision attributes derived from RS. Sanchez-Riera et al. [33] made use of deep learning to train a set of gestures, which served as a rough estimate of hand pose and orientation. Bell et al. [34] proposed a video mining learning pattern of behavior, granulating detailed action sequences into molecular behavioral units, then applying RS for attribute reduction and extracting decision rules from the resulting feature table, enabling interpretable pattern mining, validated through preliminary experiments. The generated rule sets are highly explainable and can be directly translated into decision-making criteria for behavior recognition.

The previous discussion highlights that traditional behavior recognition methods struggle to capture the temporal evolution characteristics of actions, typically achieving recognition only in narrow-domain single-person scenarios. Monomodal visual approaches exhibit weak discrimination capabilities for similar behaviors and fail to resolve ambiguities within actions. Consequently, this paper integrates ByteTrack with TSM to address real-time multi-target tracking and behavior recognition challenges in multi-driver scenarios. Additionally, it introduces a decision optimization method based on RS, effectively distinguishing ambiguous actions, such as smoking and playing on a phone.

3. The Proposed Approach

The bridge of a ship is a unique environment, with several drivers operating the vessel simultaneously. As illustrated in Figure 1, this paper proposes a cross-modal behavioral intelligence recognition approach.

The procedure for this approach is as follows: Firstly, the YOLOv7 model is utilized for the detection of individuals and feature objects. Subsequently, the ByteTrack algorithm is introduced for the tracking of multiple drivers, with an improved TSM method applied for the preliminary recognition of target behaviors. To address the challenge in regard to differentiating between two types of abnormal driver behaviors, namely smoking and playing on a phone, an optimization approach aided by feature objects is proposed based on RS. This involves the development of personnel object spatiotemporal correlation rules and the implementation of a joint reasoning engine for actions and feature objects, leading to enhanced identification accuracy in terms of these similar behaviors.

The Multi-person behavior recognition algorithm is given as Algorithm 1:

Algorithm 1 Multi-person behavior recognition algorithm

Input: Real-time video stream from ship bridge cameras Output: Anomaly alerts with behavior classification 1: Initialize YOLOv7 model, ByteTrack tracker, Improved TSM algorithm, Decision Optimization Engine 2:      FOR each frame in video_stream DO 3:      Perform Personnel and Feature Object Detection (frame) 4:      Perform Personnel and Feature Object Localization (frame, YOLOv7) 5:      IF Personnel detected THEN 6:        Perform Personnel Tracking Support (frame, ByteTrack) 7:        Assign Feature Object Scores (frame, YOLOv7) 8:      ENDIF 9:     Perform Spatiotemporal Alignment Module (frame) 10:     Associate Personnel–Feature Object (frame, tracking results, feature scores) 11:     Perform Behavior Recognition (frame, Improved TSM) 12:     Calculate Action Score (behavior recognition results) 13:     Generate Anomaly Alert (action score, Decision Optimization Engine) 14:      END FOR 15: RETURN Anomaly Alerts

3.1. Personnel and Feature Object Detection Based on YOLOv7

The YOLO series of detection algorithms are well-known for their lightweight and efficient performance, making them highly suitable for edge scene requirements. In each frame, YOLOv7 is used to identify all the drivers and abnormal objects, such as mobile phones and cigarettes. For the detected individuals, the ByteTrack algorithm is employed for multi-target tracking.

ByteTrack operates based on tracking according to the detection paradigm. It achieves person tracking by associating all the person detection boxes and assigns unique identifiers (IDs) to each individual. Additionally, ByteTrack stores continuous action frames of individuals, providing essential data support for behavior recognition. This combination of YOLOv7 and ByteTrack ensures both accurate detection and robust tracking, laying a solid foundation for subsequent behavior analysis.

3.2. Improved TSM for Behavior Recognition

Considering the particularities of a ship’s bridge, this research improves the TSM model, as depicted in Figure 2. The process involves several stages: random sampling, feature extraction, feature augmentation, and classification. Firstly, videos are divided into multiple segments at the random sampling layer, where one frame is randomly selected from each segment to serve as the pre-input for the model. These sample frames then undergo pre-processing using various image enhancement techniques, such as padding, random cropping, and flipping, etc. This pre-processing step helps maintain relevant feature information, while reducing noise and improving the model’s convergence rate. In the feature extraction layer, the EfficientNet-B3 network is employed as the core feature extraction module. Due to its reduced parameter count, it significantly reduces the computational costs, while enhancing recognition performance. After obtaining the final feature map, we introduce a feature enhancement layer that models the feature map in two dimensions, using the CA module. This allows the network to simultaneously extract internal potential connections in regard to both channel and spatial dimensions. Finally, the feature map is classified using a fully connected layer acting as a classifier. The fused final category score is determined by uniformly averaging the outputs of a set of frames through consensus. The consensus mechanism plays a pivotal role in this framework. Given that the model processes multiple frames sampled from video segments, their respective outputs may exhibit slight variations. To achieve robust result aggregation, the proposed method employs consensus-based averaging of classification scores across all the frame outputs. This critical processing stage effectively utilizes the inherent redundancy in multi-frame information to mitigate individual frame prediction errors, thereby generating more stable and accurate fused category scores.

Table 1. The backbone network.

Stage	Operator	Resolution	Channels	Layers
$\mathbf{i}$	${\mathbf{F}}_{\mathbf{i}}$	${\mathbf{H}}_{\mathbf{i}}\ast {\mathbf{W}}_{\mathbf{i}}$	${\mathbf{C}}_{\mathbf{i}}$	${\mathbf{L}}_{\mathbf{i}}$
1	Conv3 × 3	448 × 448	32	1
2	MBConv1, k3 × 3	224 × 224	16	2
3	MBConv6, k3 × 3	224 × 224	24	3
4	MBConv6, k5 × 5	112 × 112	40	3
5	MBConv6, k3 × 3	56 × 56	80	4
6	MBConv6, k5 × 5	28 × 28	112	4
7	MBConv6, k5 × 5	28 × 28	192	5
8	MBConv6, k3 × 3	14 × 14	320	2
9	Conv1 × 1	14 × 14	1280	1
10	CA_Block	14 × 14	1280	1
11	Pooling&FC	14 × 14	7	1

outlines the structural parameters of the behavior recognition network, with each row describing a stage $\mathrm{i}$ containing ${\mathrm{L}}_{\mathrm{i}}$ layers, input resolution ${\mathrm{H}}_{\mathrm{i}}\ast {\mathrm{W}}_{\mathrm{i}}$, and output channels ${\mathrm{C}}_{\mathrm{i}}$.

In regard to the classical TSM, random cropping is employed to augment the training samples. Nonetheless, the bridge images typically have a rectangular shape and are essentially consistent with the characters. A direct random crop of the source image will remove the majority of the character information. The robustness of the model will suffer if random cropping is not applied. In order to guarantee data randomization and prevent the loss of significant image features, we fill the image grayscale according to a 1:1 ratio, before random cropping.

Similarly, we apply an expansion coefficient and first enlarge the target box, before intercepting it when capturing the target image from the original video frame, after the target tracking stage. The reason for this is that there may be some character movements in the original video that overlap with the background. For instance, the driving action and the wheel usually overlap, so you will miss vital details if you simply clip and fit the person’s image. As a result, we intercept the extended target image, fill it with grayscale, crop it randomly, and then use the model to make inferences.

The core of the model is the feature extraction network, which has a significant influence on recognition performance. Because of its distinct residual structure, the ResNet series feature extraction network typically improves the model’s performance as the number of stacking layers rises. Nevertheless, the model’s size directly affects how difficult it is to deploy. Before heedlessly expanding the model size to improve the effect, it is imperative to keep the balance between the scale and the effect. The model performance can be impacted by the width, depth, and resolution of the feature extraction network. EfficientNet involves a new scaling method that uniformly scales all the dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. Google provided eight combination structures with progressively larger scales, ranging from EfficientNet-B0 to EfficientNet-B7. The structures Efficientnet-B0 through to Efficientnet-B7 have much fewer parameters and calculations than the ResNet series of feature extraction networks. Furthermore, EfficientNet models outperformed ResNet networks based on public datasets of comparable size.

This research compares the performance of different feature extraction networks based on the self-constructed SCA dataset, and the results are shown in Figure 3. The horizontal axis reflects the computational complexity of the backbone network, while the Top-1 accuracy of the behavior identification is represented by the vertical axis. EfficientNet-B3 is easily found to have the best effect, as a result of the comparison. It achieves great recognition accuracy, with a significant reduction in computational complexity and the number of parameters involved, as compared to the ResNet series of networks. As a result, the feature extraction network used in this study is EfficientNet-B3.

The size of the feature map keeps getting smaller, the number of channels keeps growing, and the features of the model extracts get more abstract, as the feature extraction network gets deeper. As a result, a lot of the features are redundant and are not useful for classification. Thus, we insert a feature improvement layer before the completely connected layer. It primarily employs a CA module to improve the network’s representation of useful information, while weakening irrelevant information for classification. The coordinate information embeds positional information. Specifically, it exploits two 1D global pooling operations to aggregate the input features, along the vertical and horizontal directions, respectively, into two separate direction-aware feature maps. These two feature maps with embedded direction-specific information are then separately encoded into two attention maps, each of which captures the long-range dependencies of the input feature map along one spatial direction. The positional information can thus be preserved in the generated attention maps. Both attention maps are then applied to the input feature map via multiplication to emphasize the representations of interest.

3.3. Multi-Source Feature Fusion Based on RS

Following the localization and recognition of individuals and characteristic objects, it is imperative to establish their associations across both temporal and spatial dimensions. From a temporal perspective, consideration must extend beyond the current frame of the drivers being tracked. It is necessary to comprehensively analyze all the feature objects within the temporal span covered by the tracking drivers. This approach ensures a holistic understanding of the drivers’ dynamics over time. Spatially, assessing the intersection between the bounding boxes of the tracking drivers and those of the feature objects is crucial for determining their spatial relationships accurately.

Specifically, for a tracking target $P_i$ in the frame $t$, its associated feature object set $O_i^t$ is

$$O_i^t=\left\{o_j|IoU\left(B_p^t,B_o^t\right)>\delta \right\}$$

(1)

where $B_p^t$ denotes the bounding box of the tracked driver in the frame $t$, while $B_o^t$ represents the bounding box of a feature object appearing in the frame $t$. $IoU\left(B_p^t,B_o^t\right)$ indicates the IoU between the driver and the abnormal object. The parameter $\delta$ denotes the threshold, which is set to 0.3.

Within a behavior sampling period, the associated characteristic object set $O_i$ for $P_{\mathrm{i}}$ is defined as the union of all the frames’ associated characteristic object sets in the sampled frame sequence $\{F_{t-n},F_{t-n+1},\dots ,F_t\}$_.

$$O_i={\bigcup }_{k=t-n}^t O_i^k$$

(2)

where $O_i^k$ represents the set of associated characteristic objects for $P_i$ in the frame $k$, and $n$ denotes the sampling length for behavior recognition. By employing the spatiotemporal alignment module to detect and associate abnormal characteristic objects in each frame, the model can effectively mitigate the issue of insufficient sensitivity to instantaneous behaviors.

Table 2. The decision table.

Condition Attribute	Confidence of Associated Feature Objects	$C_o\left(\mathit{cigarette}\right) \in [0,1]$
		$C_o\left(\mathit{phone}\right) \in [0,1]$
	Behavior Confidence	$C_b\left(driving \right) \in [0,1]$
		$C_b\left(smoking\right) \in [0,1]$
		$C_b\left(\mathit{playing} \mathit{on} a \mathit{phone}\right) \in [0,1]$
		$C_b\left(sitting\right) \in [0,1]$
		$C_b\left(\mathrm{standing}\right) \in [0,1]$
		$C_b\left(walking\right) \in [0,1]$
		$C_b\left(\mathit{lying}\right) \in [0,1]$
	Coverage Rate of Anomalous Features	${\mathrm{S}}_{\mathrm{o}}\left(\mathit{cigarette}\right) \in [0,1]$
		${\mathrm{S}}_{\mathrm{o}}\left( \mathit{phone}\right) \in [0,1]$
Decision Attribute	Anomalous Behavior	$D_a\in${non-driving (smoking), non-driving (playing on a phone), distracted driving (smoking), distracted driving (playing on a phone)}
	Basic Mutually Exclusive Behavior	$D_b\in \left\{sitting, standing, walking, lying \right\}$

is a decision table formulated based on feature data extracted through the use of spatiotemporal association rules and behavior classification outcomes derived from the improved TSM. The condition attributes ensemble comprises:

(1) The Confidence of Associated Objects. Denoted as $C_o$, this metric represents the average confidence level of associated features of the same category within the sampling time frame, pertinent to the driver under observation.

(2) The Coverage Rate of Anomalous Features. Specifically defined for objects such as cigarettes, $S_o$ denotes the coverage ratio of associated anomalous objects during the behavioral sampling period of the driver under scrutiny. Mathematically, it is expressed as:

$$S_o\left(cigarette\right)={\displaystyle \frac{n_1}{n}}$$

(3)

where $n_1$ denotes the number of frames in which the associated feature object (e.g., cigarette) appears, and $n$ represents the total number of sampled frames. An elevated feature coverage rate signifies a higher frequency of interactions between the observed drivers and the feature objects, which is positively correlated with the likelihood of the corresponding behavior’s occurrence.

(3) The Behavior Confidence. $C_b$ refers to the output generated by the improved TSM model concerning the confidence level of the detected behaviors.

The Anomalous Behavior Set $D_a$ includes non-driving behaviors (smoking), non-driving behaviors (playing on a phone), distracted driving (smoking), and distracted driving (playing on a phone). The basic mutually exclusive behavior set $D_b$ encompasses four fundamental states: sitting, standing, walking, and lying. These mutually exclusive behaviors are foundational for classifying various activities, providing a clear distinction between different types of behaviors observed within the study context.

Due to the significant similarities in motion characteristics between smoking and playing on a phone, such as the interaction of specific objects with the person’s head and the pronounced arm movements involved in both answering a phone call and bringing a cigarette to the person’s mouth. The TSM model, despite its efficacy in extracting temporal features from adjacent frames, encounters challenges in accurately distinguishing the subtle local differences between these two behaviors. This limitation often results in confusion, suggesting a potential action association between smoking and playing on a phone during the behavior classification process. To address this issue, the confidence levels of the two behaviors are proportionally integrated to generate an action score. This action score is subsequently combined with the object feature score to compute the final abnormal behavior score for classification purposes. The decision rule set is as follows:

$$D_a=\left\{\begin{array}{c}non-driving \left(playing on a phone\right) IF C_b\left(driving \right)<\tau \wedge M\left(playing on a phone\right)>\phi \\ non-driving \left(smoking\right) IF C_b\left(driving \right)<\tau \wedge M\left(smoking\right)>\phi \\ distracted driving \left(smoking\right) IF C_b\left(driving \right)<\tau \wedge M\left(smoking\right)>\phi \\ distracted driving \left(playing on a phone\right) IF C_b\left(driving \right)<\tau \wedge M\left(playing on a phone\right)>\phi \end{array}\right.$$

(4)

where $\phi$ denotes the threshold score for identifying abnormal behaviors, while $\tau$ represents the threshold score for assessing the driving state. Specifically, $M\left(smoking\right)$ and $M(playing on a phone)$ refer to the scores for abnormal behaviors corresponding to smoking and playing on a phone, respectively, which are defined as follows:

$$M\left(smoking\right)=\alpha ·S_o\left(\mathit{cigarette}\right)·C_o\left(\mathit{cigarette}\right)+\beta ·C_b\left(smoking\right)+\gamma ·C_b\left(\mathit{playing} \mathit{on} a \mathit{phone}\right)$$

(5)

$$M(playing on a phone)=\alpha ·S_o\left(\mathit{phone}\right)·C_o\left(\mathit{phone}\right)+\beta ·C_b\left(\mathit{playing} \mathit{on} a \mathit{phone}\right)+\gamma ·C_b\left(\mathit{smoking}\right)$$

(6)

where $\alpha$ denotes the weight coefficient associated with the feature score, whereas $\beta$ and $\gamma$ represent the weight coefficients corresponding to the direct and indirect action scores, respectively. Utilizing the grid search method within the SCA dataset, with the Comprehensive Classification Accuracy (CCA) as the evaluation metric, and setting the search range to [0, 1], with a step size of 0.05, it is experimentally determined that the highest CCA of 89.4% is attained with the parameter configuration $(\alpha ,\beta ,\gamma ,\phi ,\tau )=\left(\mathrm{0.55,0.25,0.2,0.7,0.7}\right)$.

As discussed above, the basic mutually exclusive behavior set $D_b$ comprises four behaviors: sitting, standing, walking, and lying. Given the mutual exclusivity of these behaviors, which dictates that only one behavior can occur at any given time, the final classification of an observed behavior is determined by identifying the behavior associated with the highest confidence score $C_b$.

$$D_b=\mathit{arg}max\left(C_b\left(sitting\right),C_b\left(standing\right),C_b\left(walking\right),C_b\left(lying\right)\right)$$

(7)

Therefore, if $C_b\left(sitting\right)$ is the highest, $D_b$ is classified as sitting; if $C_b\left(standing\right)$ is the highest, $D_b$ is classified as standing; and, similarly, for each behavior within the set.

The Top-1 accuracy is effective for evaluating the recognition effect of mutually exclusive behaviors. However, it inadequately assesses the detection effectiveness of anomalous behaviors in scenarios where basic behaviors and abnormal behaviors overlap. To more comprehensively evaluate the anomaly alerting capability of the methods, an Anomaly-Focused Accuracy Rate (AFAR) metric has been designed based on the following principles:

(1) Decoupling the influence of the driving states. This involves disregarding the accuracy of distinguishing between driving states (non-driving vs. distracted driving) in regard to the classification of anomalous behaviors, focusing instead solely on the identification of specific anomalous behavior categories (e.g., smoking, playing on a phone).

(2) The differential penalty mechanism. Considering the practical application context, this mechanism imposes differential penalties for cross-category misjudgments (e.g., misclassifying smoking as playing on a phone) and missed detections (the failure to detect anomalous behavior), based on the severity of the misjudgment consequences. This ensures that the evaluation not only considers correct identifications, but also appropriately penalizes errors, according to their impact.

The AFAR is defined as follows:

$$AFAR={\displaystyle \frac{N_{correct}}{N_{correct}+N_{confusion}+{2N}_{miss}}}$$

(8)

where $N_{correct}$ represents the number of samples correctly classified into the anomaly behavior categories, $N_{confusion}$ denotes the number of misclassifications, and $N_{miss}$ indicates the number of anomaly samples not detected. Given that missed detections generally result in more severe consequences than misclassifications in real-world scenarios, the formula imposes a heavier penalty for misses to reflect their impact. This AFAR metric only evaluates the detection effectiveness based on anomaly samples.

In order to prevent an overemphasis on anomaly classification at the expense of normal sample detection, an evaluation metric CCA that comprehensively considers both anomaly and normal behavior detection accuracies is introduced, which is given by:

$$CCA={\displaystyle \frac{N_{correct\_normal}+N_{correct\_anomaly}}{N_{total}}}$$

(9)

where $N_{correct\_normal}$ represents the number of normal samples correctly identified as not exhibiting any anomaly behavior, $N_{correct\_anomaly}$ denotes the number of anomaly samples that are correctly classified, and ${\mathrm{N}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ is the total number of samples.

This approach ensures that the evaluation framework not only emphasizes the critical aspect of detecting anomalous behaviors accurately, but also maintains high standards for the correct identification of normal behaviors, thereby providing a comprehensive assessment of the model’s performance.

4. Experiments

4.1. Datasets

To validate the proposed approach, a dataset is utilized of footage from CCTV surveillance systems installed on ship bridges. This dataset, designed to enhance model robustness, includes a diverse array of bridge environments. The video frames are meticulously cropped and categorized with behavior labels to create a specialized dataset for driver behavior detection.

We have created a ship bridge behavior SCA dataset. The dataset encompasses seven distinct behaviors: sitting, standing, walking, lying, driving, smoking, and playing on a phone. It comprises approximately 4788 samples in total, with each sample containing at least eight consecutive action frames. The samples are evenly distributed across the seven behavioral classes to ensure balanced representation.

Table 3. Sample distribution of the SCA dataset.

Behavior	Number of Samples
Sitting	764
Standing	728
Walking	652
Lying	660
Driving	452
Smoking	920
Playing on a phone	612

shows the sample distribution of this dataset, while Figure 4 shows some of the samples.

Additionally, from the SCA dataset, frames containing feature objects are extracted and annotated to construct a subset for personnel and object detection. This subset consists of 3400 images, covering various cockpit scenarios and includes three object categories: humans, mobile phones, and cigarettes.

4.2. Implementation Details

An experimental platform is established based on the Windows 10 operating system, leveraging a Nvidia GeForce RTX 3060 GPU, with 12 GB of memory, and an Intel Core i512400F CPU, for computational tasks. The framework utilizes PyTorch version 1.13+cu116. The SCA dataset is employed for testing purposes, with the relevant training parameters detailed in

Table 4. Parameter settings.

Category	Parameters	Value
Multi-object Tracking	Conf_thres	0.1–0.7
	IOU_thres	0.1–0.7
Behavior Recognition	Input_size	448 px × 448 px
	Num_segment	8
	Learning_rate	2 × 10−5
	Batch_size	1
	Dropout	0.8
	Epoch	50

. Considering the limited size of the SCA dataset, transfer learning is adopted in each experiment to mitigate the risk of model overfitting. Specifically, the feature extraction networks ResNet and EfficientNet are utilized, with the weights pretrained based on the ImageNet [35] dataset, followed by fine tuning based on the SC Action dataset. This methodology ensures the enhanced generalization capabilities of the model, despite the constrained dataset size.

4.3. Results

4.3.1. Results of Personnel and Feature Object Detection

The subset is partitioned into training and validation sets, with a ratio of 8:2. The detection performance of various YOLOv7 models are evaluated based on this dataset, as shown in

Table 5. Comparative results of personnel and feature object detection using YOLOv7 series models.

Model	Image Size	Precision	Recall	mAP@0.5	mAP@0.5:0.95	Latency (ms)
YOLOv7	640 px × 640 px	0.7454	0.6806	0.7029	0.4086	12.5
YOLOv7-w6	1280 px × 1280 px	0.812	0.726	0.78	0.503	30.9
YOLOv7-e6	1280 px × 1280 px	0.822	0.745	0.781	0.514	49.3
YOLOv7-d6	1280 px × 1280 px	0.827	0.7434	0.7964	0.5101	60.6
YOLOv7-e6e	1280 px × 1280 px	0.838	0.703	0.776	0.513	71.6

. The experimental results presented in Table 5 reveal an important trade-off in regard to model performance. The baseline YOLOv7 model achieves the fastest processing speed, with a latency of just 12.5 milliseconds, owing to its smaller 640 px × 640 px input resolution. However, this advantage comes at the cost of a reduced detection capability, particularly for small objects like cigarettes and mobile phones that appear at low resolution in the input frames. The YOLOv7-e6 variant demonstrates superior performance, with a recall rate of 0.745, and achieves the highest mAP@0.5:0.95 score of 0.514. This model exhibits particularly impressive efficiency, delivering comparable mAP@0.5 accuracy, while achieving an 18.6% reduction in inference latency relative to the computationally heavier YOLOv7-d6 architecture. The YOLOv7-e6e model, while attaining peak precision of 0.838 among all the tested models, suffers from a substantially lower recall rate of 0.703. This performance imbalance raises significant concerns regarding potential missed detections of critical feature objects, which could severely compromise the system’s ability to accurately identify and classify anomalous behaviors in practical deployment scenarios.

After a comprehensive evaluation of the detection accuracy, small-object sensitivity, and deployment feasibility, we selected YOLOv7-e6 as the core detection model. Its superior recall rate and robust feature extraction capabilities provide a stable and reliable foundation for target localization, which is critical for subsequent behavior recognition tasks. This architecture demonstrates an optimal balance between detection performance in regard to challenging small targets and practical implementation requirements.

Therefore, the YOLOv7-e6 model is selected for object detection throughout this research, providing a robust and reliable foundation for object localization. This, in turn, supports subsequent behavior recognition tasks.

4.3.2. Results of Driver Tracking

The tracking efficacy is affected by the confidence threshold and the IoU threshold. Therefore, experiments are conducted to evaluate the tracking performance under various configurations of these parameters. The assessment is based on three metrics: MOTA, precision, and recall. As shown in Figure 5, the optimal performance is achieved when both the confidence threshold and the IoU threshold are set to 0.4, resulting in an MOTA score of 0.957, a precision of 0.97, and a recall of 0.991.

4.3.3. Results of Behavior Recognition

In regard to the behavior recognition module, a comparative analysis is conducted among the classical TSM, the improved TSM, and the popular SlowFast algorithm, using the SCA dataset. The comparison is based on three dimensions: GMAC, the number of parameters, and Top-1 accuracy. As can be observed from

Table 6. Comparison of different models. GMAC values represent billions of multiply–accumulate operations.

Model	Backbone	GMAC	Parameters/M	Top-1 Accuracy/%
TSM	ResNet50	132.17	23.52	77.79
	ResNet101	251.62	42.51	81.4
	EfficientNet-B0	13.07	4.02	76.34
	EfficientNet-B1	19.26	6.52	80.65
	EfficientNet-B2	22.16	7.71	78.50
	EfficientNet-B3	32.23	10.71	80.11
	EfficientNet-B4	50	17.56	77.42
SlowFast	ResNet50	101.16	33.66	75.1
	ResNet101	163.88	52.87	76.66
Improved TSM	EfficientNet-B3	32.28	10.93	82.1

, the improved TSM algorithm demonstrates superior performance compared to both the original TSM model and the SlowFast model.

To further validate the effectiveness of the proposed improved TSM method, ablation experiments are conducted, and the results are presented in

Table 7. Ablation study results. A checkmark symbol (√) denotes the inclusion of a component in the model configuration, while empty cells indicate its exclusion.

Component	M1	M2	M3
TSM–EfficientNet-B3	√	√	√
+Image Augmentation		√	√
+CA Attention			√
Computational Cost (GMAC)	32.23	32.23	32.26
Top-1 Accuracy (%)	79.03	80.32	82.1

.

The baseline model M1 is based on the TSM architecture, with the EfficientNet-B3 backbone. M2 improves upon M1 by incorporating image augmentation, utilizing a super-resolution reconstructed dataset, and refining the pre-processing strategy. As a result, the recognition accuracy increases by 1.29 percentage points. Building on M2, M3 introduces the CA attention module into the feature extraction layers. With only a marginal increase in the computational cost, this modification effectively enhances the model’s representational capability, achieving a Top-1 accuracy of 82.1%.

To comprehensively evaluate the recognition performance of the improved TSM model, comparative experiments are conducted on two widely used benchmark datasets, UCF101 and HMDB51, which contain more diverse action categories than typical evaluation datasets. The Top-1 accuracy serves as the primary evaluation metric, with the detailed comparative results systematically presented in

Table 8. Comparative results based on public datasets.

Method	Backbone Network	Parameters/M	UCF101 /%	HMDB51 /%
TSN [19]	InceptionV2	10.7	84.4	53.3
I3D [15]	InceptionV2	25	95.6	74.8
TSM [20]	ResNet50	23.52	96.0	73.2
VidTr [36]	VidTr-L	86	96.7	74.4
AMD [37]	ViT-B	87	97.1	79.6
Improved TSM	Improved EfficientNet-B3	10.93	96.7	75.1

.

The improved TSM is benchmarked against state-of-the-art approaches to action recognition, including the 2D baseline TSN, I3D, and Vision Transformer-based methods [38]. The experimental results demonstrate that the improved TSM model achieves a competitive Top-1 accuracy of 96.7% and 75.1% based on the UCF101 and HMDB51 datasets, respectively. While slightly lower than the ViT-based AMD method, it exhibits significant advantages in regard to its lightweight design, with substantially fewer parameters, namely 10.93 M, compared to Transformer-based architectures.

The feature extraction capability is a critical determinant of a model’s performance. By comparing the shallow feature maps of identical input samples across different model architectures, deeper insights into the initial stages of feature recognition can be gained, as illustrated in Figure 6.

In regard to the TSM–ResNet series of models, the shallow feature maps exhibit a notable homogenization tendency. As shown in Figure 6b,c, the feature responses at different locations show a high level of similarity. This phenomenon indicates that the model fails to effectively capture the multidimensional characteristics of the input data in regard to the initial layers, thereby limiting the richness and expressiveness of subsequent deep features. Figure 6 demonstrates that the improved model has solved this problem, while simultaneously increasing the feature map diversity.

To evaluate the real inference effect, comparisons are made between the TSM–ResNet50 model and the improved TSM model. For an intuitive comparison, the behaviors are sorted in descending order of likelihood and displayed within the corresponding detection boxes, as shown in Figure 7. The probability of the driving behavior is represented by the action probability, which uses a sigmoid function to map behavior classification scores in the range (0,1). When the action probability is less than 0.5, the model predicts that the target does not exhibit the behavior, indicating a negative confidence score. As the action’s probability increases, the model interprets this as a higher likelihood of the target exhibiting the behavior.

From Figure 7, it is evident that the TSM–ResNet50 model inaccurately recognizes the left driver’s sitting behavior (indicated by the red box), suggesting instead that the driver is more likely standing. In contrast, the improved TSM model demonstrates a greater likelihood of correctly identifying the sitting behavior compared to standing. Although the standing confidence level remains relatively high, this suggests that the improved model is moving in the right direction, towards more accurate behavior recognition.

The experiments also involved a class-wise accuracy comparison across different behavior categories, as shown in

Table 9. Comparison of class-wise accuracy for different models.

Behavior	SlowFast–ResNet50 (%)	TSM–ResNet50 (%)	Improved TSM (%)
Sitting	83.24	84.53	89.27
Standing	82.45	85.08	88.63
Walking	81.03	83.47	87.15
Lying	84.12	85.62	88.29
Driving	80.58	82.14	91.35
Playing on a phone	58.31	62.08	65.42
Smoking	60.17	64.25	68.09
Top-1 accuracy	75.10	77.79	82.10

.

The improved TSM model achieves performance gains in all the categories, with the most notable improvement in the driving category, where the accuracy reaches 91.35%, representing an increase of 9.21 percentage points. This improvement is primarily attributed to the collaborative modeling of the bounding box expansion and the CA attention mechanism, which effectively captures contextual information from the dashboard background. In contrast, the model performs relatively less well in regard to abnormal behaviors, such as playing on a phone and smoking, suggesting that further optimization is needed to better address the unique characteristics of these behaviors.

4.3.4. Multi-Source Feature Fusion Inference Results

To validate the effectiveness of the proposed multi-source feature fusion inference method, a comparison is conducted between the pre-optimization and post-optimization decision-making processes. Prior to decision optimization, the output of the method consists of normalized confidence scores for behaviors, which are not directly suitable for anomaly detection. Consequently, a decision function must be applied to transform these confidence scores into prediction results.

$$D_{pre}=\left\{\begin{array}{c}{smoking }_{\wedge C_b\left(smoking\right)>C_b\left(playing on a phone\right)}^{IF C_b\left(smoking\right)>{\phi }_{pre}}\\ {playing on a phone}_{ \wedge C_b\left(playing on a phone\right)>C_b\left(smoking\right)}^{ IF C_b\left(playing on a phone\right)>{\phi }_{pre}}\end{array}\right.$$

(10)

where $D_{pre}$ represents the anomaly behavior classification without employing multi-source feature fusion. The optimal parameter ${\phi }_{pre}=0.65$ is determined through the use of a grid search method. In contrast, the method utilizing multi-source feature fusion initially decouples the driving state and, subsequently, evaluates it using the AFAR. The test data are derived from the SCA dataset, comprising a total of 954 samples, including 306 anomaly samples. Among these anomaly samples, 184 instances involve a person smoking and 122 instances involve a person playing on a phone. The relevant evaluation metrics and detailed test data are summarized in

Table 10. Comparison of anomalous behavior detection results using different methods.

Method	AFAR/%			CCA/%
	All	Smoking	Playing on a Phone
TSM–ResNet50	74.46	76.55	71.3	80.36
Improved TSM	76.3	77.57	74.38	81.84
Multi-Source Feature Fusion	83.37	84.47	81.69	83.66

.

To evaluate the reasoning effectiveness of the optimized decision-making method, two surveillance video segments featuring a person smoking and a person playing on a phone are employed for testing. As illustrated in Figure 8a, a driver is observed holding a cigarette. However, due to the absence of critical smoking action features, the action score remains low. Consequently, after feature fusion, the behavior is correctly classified as normal. In Figure 8b, a diver exhibits key action characteristics indicative of smoking, which increases the action score, thereby elevating the overall behavioral score beyond the predefined threshold, resulting in successful anomaly detection. Moreover, fundamental mutually exclusive behaviors, such as sitting and non-driving, are accurately identified. Prior to its application, the lack of auxiliary judgment based on object features often leads to inaccurate behavior categorization by the model. Figure 8c depicts a typical ship’s bridge scene filled with electronic instruments, where a driver is partially obscured by another passing individual, while positioned at the control panel. The proposed feature fusion approach accurately identifies both the driver and the mobile phone by considering the driver’s actions and their association with object features comprehensively. Consequently, the driver’s behavior is successfully categorized as standing, distracted driving (playing on a phone).

Furthermore, a surveillance video containing multiple drivers is selected for analysis to evaluate the effectiveness of the proposed method in crowded scenarios. As illustrated in Figure 9, the method demonstrates strong capabilities in regard to person localization and behavior recognition, even under conditions involving severe occlusion. In Figure 9a, despite the mobile phone being nearly indistinguishable from the background, it is successfully detected. By combining this detection with the distinct postural cue of the driver bowing their head, the system accurately identifies two drivers playing on a phone. Similarly, in Figure 9b, the method detects a cigarette near the crew member’s mouth. By incorporating contextual information from the control panel, the model correctly classifies the behavior as standing–distracted driving (smoking).

These results demonstrate the robustness and practical applicability of the proposed approach in regard to complex operational environments. The multi-level feature fusion strategy effectively addresses key challenges such as target occlusion overlapping and weakened feature representation in dense scenes.

5. Conclusions

This paper presents a novel rough set-driven cross-modal framework for intelligent recognition method of multi-driver behaviors in ship bridge scenarios. Our work makes three foundational contributions to maritime behavior analysis: (1) original algorithmic integration. We propose an architecture combining ByteTrack multi-driver tracking with an improved TSM, enhanced by a spatiotemporal-aware CA mechanism. This design significantly outperforms unimodal approaches, achieving an 81.37% AFAR and 80.66% CCA, significantly outperforming methods relying solely on visual modality. (2) Pioneering feature object association. We introduce a rough set-based joint inference model that establishes spatiotemporal correlation rules between personnel and objects. This innovation enables interpretable reasoning for abnormal behaviors like smoking or playing on a phone, addressing a critical gap in maritime safety systems. (3) A benchmark dataset and metrics. The SCA dataset, the first publicly available ship bridge behavior dataset, with 4788 annotated sequences and 3400 feature object images, provides a foundation for reproducible research. The proposed AFAR metric offers a novel evaluation protocol for anomaly focused maritime applications. By unifying rough set theory, spatiotemporal perception, and object semantics, this work establishes a new paradigm for intelligent ship bridge monitoring, with implications for autonomous navigation and human–AI collaboration in maritime environments.

Future research will focus on the following aspects: (1) Given the critical importance of datasets for model training, the current sample size of the SCA dataset is insufficient. In the future, it is necessary to collect more data under extreme weather conditions and nighttime scenarios. Considering the introduction of Generative Adversarial Networks (GANs) for data augmentation could enhance the generalization capabilities. Additionally, this research utilizes a default frame sampling rate; the experimental findings indicate that increasing the sampling frame rate can effectively improve the temporal modeling performance. Therefore, future dataset expansions should prioritize samples with higher frame rates over those with lower frame rates. (2) The existing system primarily relies on visual modalities and could benefit from the integration of multi-modal data, such as millimeter-wave radar and cabin sensors. Exploring the application of voiceprint recognition technology to detect abnormal sounds and incorporating eye-tracking techniques to assess driver distraction levels are promising avenues for enhancing system robustness and reliability. (3) Considering the characteristics of embedded devices onboard ships, future research should focus on Neural Architecture Search (NAS) technologies to design lightweight models specifically tailored for maritime applications. Investigating model quantization and knowledge distillation strategies could further compress models, while maintaining high accuracy, making them suitable for deployment in resource-constrained environments.