MAAT: A Marine-Aware Adaptive Tracker for Robust and Real-Time Multi-Object Tracking in Maritime Environments

Abstract

Multi-object tracking (MOT) is a key technology for enabling autonomous navigation of unmanned surface vehicle (USV) as it provides continuous perception of surrounding maritime targets and supports navigation decision-making. However, videos acquired on maritime platforms typically suffer from challenges such as platform-induced jitter and nonlinear object motion, which significantly degrade tracking performance. To address these challenges, this paper builds upon ByteTrack by incorporating an adaptive Kalman filtering scheme and proposing a density-aware association strategy, resulting in a novel tracker termed the Marine-Aware Adaptive Tracker (MAAT). Specifically, an adaptive Kalman filter is introduced to increase the contribution of high-confidence detections during the state update process, thereby enhancing the stability and robustness of state estimation. Furthermore, to better mitigate the frequent identity switches caused by severe platform jitter from the USV observation platform, a density-aware association strategy is proposed. This strategy dynamically adjusts the composition of the cost matrix according to the density of high-confidence targets, enabling more reliable data association under varying scene conditions. Finally, the proposed tracking algorithm is evaluated against several state-of-the-art methods on the Singapore Maritime Dataset. It achieves competitive performance, attaining 44.37 MOTA and 43.857 IDF1. Moreover, MAAT operates in real time, running at 41.4 FPS. The experimental results demonstrate that MAAT is capable of performing accurate and real-time multi-object tracking in dynamic maritime environments with surface fluctuations, thereby providing effective technical support for intelligent maritime surveillance applications.

1. Introduction

MOT is an essential perception module for USVs, enabling the continuous monitoring and tracking of surrounding maritime targets from onboard vision systems. MOT aims to identify and track multiple targets through video analysis. Existing MOT algorithms can generally be categorized into online and offline approaches [1,2]. Online methods perform target tracking at each frame using only the current frame and a limited number of past frames, thereby emphasizing real-time performance and low latency [3,4], such as SORT [5], DeepSORT [6], and FairMOT [7]. In contrast, offline methods exploit information from a larger temporal context, typically leveraging historical frames or analyzing the entire video sequence to correct errors in the current frame. Such approaches are generally adopted when more accurate trajectory estimation and long-term tracking are required, for example, Siamese-based symmetric networks and their variants [8]. However, in certain application scenarios, offline MOT methods struggle to satisfy real-time requirements.

Vision-based multi-object tracking from USVs requires real-time processing of maritime video data captured by visible-light sensors to ensure timely detection and tracking. Therefore, online methods, due to their real-time capability and causal nature, have become the primary research focus in maritime MOT. To date, Tracking-by-Detection (TBD) has emerged as the most effective paradigm for online MOT tasks [3,4,5,7]. Many recent TBD approaches are built upon SORT, DeepSORT, and ByteTrack [9]. These SORT-like MOT algorithms typically first employ an object detector to localize targets in each frame, then associate current-frame detections with existing trajectories from the previous frame using ReID features or IOU-based matching, and finally update the trajectories via a Kalman filter. Most of these methods rely on two key assumptions: (1) A target can only be assigned the same identity if the IoU between detections is greater than zero. (2) The motion of the target relative to the USV observation platform follows a linear model.

However, unlike the observation perspectives commonly encountered in MOT scenarios—such as fixed surveillance cameras [10,11,12] or vehicle-mounted cameras [13,14]—maritime videos present unique challenges. (1) USV platform jitter: Under most weather conditions, ocean waves induce varying degrees of instability in the USV observation platform, resulting in frequent identity switches. (2) Nonlinear target motion: The superposition of platform-induced motion and intrinsic target motion violates the linear motion assumption of the Kalman filter, leading to significant estimation bias and degraded tracking accuracy.

These challenges render the aforementioned two assumptions inadequate for complex and dynamic maritime environments. In such scenarios, sea–surface fluctuations induce significant jitter in the observation platform, which severely degrades tracking performance. As illustrated in Figure 1, two target trajectories are initialized in frame (a). In frame (b), where the motion remains relatively stable, detections can be successfully associated with the predicted tracklets based on IoU distance. However, between frame (b) and frame (c), strong platform-induced jitter leads to a significant deviation between the Kalman filter prediction and the detection bounding box, resulting in an IoU of zero. Consequently, although the detections correspond to the same target, the association fails, leading to identity switches and trajectory fragmentation. In such cases, the assumption of linear motion between the target and the USV observation platform no longer holds, resulting in substantial estimation bias in the Kalman filter [15,16]. The Singapore Maritime Dataset (SMD) is adopted in this study to evaluate the proposed method, as it provides a representative benchmark for real-world maritime surveillance scenarios. Compared with other datasets, SMD contains challenging conditions such as severe platform-induced jitter, which are highly consistent with the problem settings addressed in this work. Therefore, using SMD enables a more realistic and comprehensive evaluation of tracking performance under complex maritime environments. Experimental results indicate that most existing methods are unable to fully address these issues.

Motion cues serve as a powerful signal for target association in maritime video scenarios. To address the large estimation errors of the Kalman filter caused by nonlinear motion, we first introduce an adaptive Kalman filtering scheme that increases the weight of high-confidence detections during the state update stage, thereby enhancing the stability and reliability of state estimation. Furthermore, to better mitigate the frequent identity switches induced by severe USV platform jitter, we propose a density-aware association strategy, which dynamically adjusts the composition of the cost matrix according to the target density. The resulting tracker is termed MAAT, which is equipped with YOLOv8—the detector demonstrating superior performance in maritime environments—to generate bounding box detections, and integrates them with the proposed data association framework.

The main contributions of this paper are summarized as follows:

(1)

To address the degradation of state estimation caused by nonlinear motion under platform jitter, an adaptive Kalman filtering mechanism is introduced to dynamically adjust the contribution of observations during the update stage, thereby improving the robustness and reliability of motion prediction in complex maritime environments.

(2)

To address the failure of conventional IoU-based association caused by severe platform-induced jitter, a density-aware association strategy is proposed. By adaptively adjusting the composition of the cost matrix, the proposed method enhances association robustness and effectively reduces identity switches under unstable observations.

The remainder of this paper is organized as follows. Section 2 reviews the related work, and Section 3 describes the architecture and components of the proposed MAAT in detail.

2. Related Work

With the increasing maturity of USV technology, USVs have been playing an increasingly important role in various applications, including maritime patrol, search and rescue, and marine scientific research [17,18,19,20]. In these scenarios, accurate and real-time monitoring of maritime targets is of critical importance. Among the enabling technologies, vision-based MOT from USVs serves as a key component for achieving intelligent maritime surveillance [21,22,23].

2.1. Tracking by Detection

In recent years, however, the rapid advancement of deep learning, particularly the significant improvements in object detector performance [24,25,26,27,28,29], has led to a substantial transformation in the field of MOT. TBD has gradually emerged as the dominant paradigm in multi-object tracking. By leveraging advanced object detectors, such as YOLO [24,25], Faster R-CNN [26], RetinaNet [27], SSD [28], CenterNet [29], and the lightweight EfficientDet, MOT systems are able to obtain more accurate and robust detections at higher speeds. This emerging trend has enabled MOT algorithms to better handle complex environments and diverse challenges.

At present, most TBD-based approaches decompose the MOT task into two stages: object detection and data association. Object detectors can generally be categorized into one-stage detectors [24,25,27] and two-stage detectors [26]. Among them, the YOLO series has been widely adopted in many MOT frameworks [3,4,7] due to its high real-time performance. Based on experimental evaluations of the YOLO family, it is selected as the detector in this work owing to its superior balance between accuracy and speed. Data association aims to link detection results across different frames to form consistent object trajectories. This process typically relies on various association techniques, such as Kalman filtering, the Hungarian algorithm [30], and Maximum A Posteriori (MAP) estimation [31]. These methods facilitate the matching of detections in individual frames with existing tracks and enable the update of target state information. The TBD paradigm enhances the scalability and generalization capability of multi-object tracking systems, as it allows flexible integration of different detectors and association algorithms. However, in maritime environments, sea–surface fluctuations and platform-induced jitter introduce significant disturbances to the tracking process. These factors lead to deviations in both motion prediction and data association within the TBD framework, thereby increasing the risk of tracking failure. The limitations of motion modeling and association strategies will be further analyzed in the following.

2.2. Tracking by Transformer

Another paradigm in MOT extends query-based object detectors to the tracking task [32,33,34]. Unlike the aforementioned Tracking-by-Detection framework, these methods typically do not require additional post-processing for data association. Instead, they employ tracking queries to progressively decode and update the locations of tracked targets across frames.

Inspired by DETR [35], most of these approaches employ learnable object queries for newborn target detection, while tracking queries are used to localize previously tracked targets. TransTrack [32] leverages two sources of queries to obtain adaptive cues. The first query type is similar to that in transformer-based detectors and provides general object detection results. The second type utilizes features of previously detected targets to form tracking queries. New trajectories are then generated by matching detection boxes and tracking boxes based on IoU. TrackFormer [33] and MOTR [34] are extended from Deformable DETR. These methods predict object bounding boxes and update tracking queries to detect the same targets in subsequent frames. TrackFormer employs the same Transformer decoder as TransTrack for both detection and tracking, followed by Non-Maximum Suppression (NMS) with a high IoU threshold to eliminate duplicate bounding boxes. Unlike previous approaches, MOTR constructs a fully end-to-end Transformer framework for multi-object tracking, eliminating complex post-processing steps. In this framework, object queries are extended into tracking queries, where each tracking query is responsible for continuously tracking a specific target throughout the video. Once successfully matched, a tracking query remains dedicated to that target without requiring additional association operations.

These transformer-based approaches learn deep representations from visual cues and object trajectories. Although they have achieved impressive performance, their application in maritime environments still faces several challenges, such as blurred sea–surface scenes and nonlinear target motion. These limitations result in a considerable gap in both accuracy and computational efficiency compared with state-of-the-art TBD-based methods.

2.3. Motion Models

Most recent state-of-the-art TBD algorithms are built upon motion models. Typically, these models adopt Bayesian estimation [36] to predict the next state by maximizing the posterior probability. The Kalman Filter, one of the most classical motion models, assumes linear motion and operates as a recursive Bayesian filter following the standard prediction–update cycle. Given that the linear motion assumption limits the applicability of the KF, variants such as the Extended Kalman Filter (EKF) [37] and the Unscented Kalman Filter (UKF) [38] have been proposed, employing first-order and third-order Taylor approximations, respectively, to handle nonlinear motion. However, these approaches still rely on Gaussian prior assumptions inherent to the KF framework and require predefined motion models. On the other hand, particle filtering addresses nonlinear motion through sampling-based posterior estimation, but its computational complexity increases exponentially with the state dimension. As a result, these KF variants and particle filtering methods are rarely adopted in vision-based multi-object tracking, where most motion models remain KF-based. Nevertheless, in many complex scenarios, particularly those involving platform jitter, the resulting nonlinear motion may lead to significant prediction errors in the KF. To mitigate this issue, many researchers employ camera motion compensation (CMC) techniques [39,40] to correct image instability caused by camera movement. While CMC can improve tracking accuracy, it typically introduces substantial computational overhead, thereby significantly reducing real-time performance. These limitations indicate that existing motion modeling approaches are insufficient to handle the nonlinear disturbances caused by platform jitter in maritime environments, making it necessary to develop a more adaptive motion model that can dynamically adjust to observation uncertainty.

2.4. Data Association

Data association lies at the core of MOT. It first computes the similarity between detection bounding boxes and existing trajectories using motion and/or appearance cues, and then performs matching based on these similarity measures. Position, motion, and appearance cues are all crucial for reliable association. Position and motion cues are combined in a relatively simple manner in SORT. Specifically, SORT employs a Kalman filter to predict the locations of trajectories in the new frame and measures the similarity between predicted boxes and detection boxes using the IoU metric. The Hungarian algorithm is then applied to perform optimal matching between detections and tracklets. Building upon the SORT framework, DeepSORT introduces a cascade matching strategy, in which detections are first associated with recently updated tracks and then with previously lost tracks. Additionally, an independent Re-Identification (Re-ID) model is employed to extract appearance features from detection boxes. By incorporating deep visual features into the association process, DeepSORT significantly improves tracking performance.

MOTDT [40] breaks from the conventional pipeline by first matching detection boxes using appearance-based cosine distance, followed by IoU-based matching for the remaining unmatched trajectories. ByteTrack proposes a two-stage association strategy in which all detection boxes in each frame are divided into high-confidence and low-confidence sets according to a threshold, and matched in two successive steps. This strategy preserves low-confidence detections and re-associates tracklets that would otherwise remain unmatched due to the removal of low-score boxes. As a result, it effectively addresses missed associations caused by occlusion or motion blur and improves trajectory continuity. BoT-SORT [39] introduces camera motion compensation and an improved Kalman filter state representation that directly predicts bounding box width and height instead of aspect ratio, enabling more accurate localization of tracking boxes. It also fuses IoU distance with Re-ID cosine similarity to enhance the robustness of detection-to-track association. Compared with other SORT-like methods, OC-SORT [41] shifts from an estimation-centric paradigm to an observation-centric one. It proposes Observation-Centric Online Smoothing (OOS), an Observation-Centric Momentum (OCM) term, and Observation-Centric Recovery (OCR). These strategies improve robustness in scenarios involving nonlinear motion and mitigate the impact of occlusion or target disappearance by relying more directly on detection observations.Despite these advances, most existing association strategies still rely heavily on IoU or appearance similarity. In maritime scenes with severe camera motion, these cues become unreliable, often leading to identity switches and fragmented trajectories. This highlights the need for a more adaptive and robust association mechanism that can handle dynamic target distributions and unstable observations.

3. Method

This section first presents the overall pipeline of MAAT, followed by a detailed explanation of the main innovations and core modules of the proposed framework, including the adaptive Kalman filtering algorithm and the density-aware association strategy. Finally, the high-performance detector YOLOv8 adopted in MAAT is described.

3.1. ByteTrack

The proposed method is built upon the recent Kalman filter-based tracking algorithm ByteTrack, which can be regarded as an improved tracker derived from the SORT baseline. SORT relies on the linear motion assumption for object tracking and employs the Hungarian algorithm to associate detection bounding boxes from the object detector with Kalman filter-predicted boxes based on IoU distance. When the observation platform is stable, the linear motion assumption is generally valid for modeling target displacement between adjacent video frames. However, in blurred scenarios, the detector tends to assign lower confidence scores to blurred targets. As a result, SORT may lose the corresponding trajectories and fail to correctly associate targets across frames.

ByteTrack incorporates low-confidence detections into the tracking process and designs a two-stage association strategy. Specifically, high-confidence detections are first matched with existing tracklets, followed by a second-stage association in which the remaining unmatched low-confidence detections are considered. The key difference between the two stages lies in the treatment of low-confidence boxes, which are assumed to contain a certain degree of noise and unreliable appearance features. Compared with other methods, ByteTrack improves the utilization of low-confidence detections, effectively alleviating track fragmentation caused by occlusion. However, in complex maritime environments, it still faces challenges such as scene blur and observation platform jitter. The proposed MAAT is developed based on ByteTrack, inheriting its overall framework, including the Hungarian algorithm for association based on the constructed cost matrix.

3.2. Marine-Aware Adaptive Track

Marine-Aware Adaptive Tracker (MAAT) is a motion-based tracking algorithm built upon ByteTrack. Figure 2 illustrates the overall tracking framework of MAAT. Given the input image and the trajectories from the previous frame $T_{t-1}$, YOLOv8 is employed as the detector to obtain detection results. According to their confidence scores, the detections are divided into high-confidence detections $D_{high}$ and low-confidence detections $D_{low}$ using a predefined confidence threshold. In this study, the threshold is set to 0.65, where detections with confidence scores higher than 0.65 are assigned to $D_{high}$, while the remaining detections are categorized as $D_{low}$. Meanwhile, an adaptive Kalman filter prior is introduced to generate trajectory predictions under the linear motion assumption. Unlike ByteTrack, MAAT adopts a density-aware association strategy for high-confidence targets, dynamically adjusting the composition of the cost matrix to improve the robustness of data association.

Specifically, Algorithm 1 presents the pseudo-code of MAAT. Given a video sequence V, at the current time step the object detector outputs detections $Det$, which, together with the output of the adaptive Kalman filter from the previous time step, serve as the inputs to MAAT. The output of MAAT is the set of trajectories T over the video sequence. The key parameters include ${\tau }_{high}$, ${\tau }_{low}$, and $\upsilon$. Among them, ${\tau }_{high}$ and ${\tau }_{low}$ are used to distinguish high-confidence and low-confidence detections, respectively. The parameter r is used to determine whether the current high-confidence detections are in a dense or sparse state, while $\upsilon$ controls whether new trajectories can be initialized. The trajectory set $\mathbf{T}$ is defined as a collection in which each element contains the bounding boxes and identity labels of targets across frames.

Algorithm 1 Pseudo-code of the MAAT tracking algorithm

Require:  Video sequence V, detector $Det$, NSA-KF filter, confidence thresholds ${\tau }_{high}$ and ${\tau }_{low}$, trajectory confidence threshold $\upsilon$, density threshold $r_0$ Ensure:  Tracked trajectory set T   1: Initialize $T\leftarrow \varnothing$   2: for each frame $f_k$ in V do (1) Detection confidence splitting   3:       $D_k\leftarrow Det\left(f_k\right)$,    $D_{high}\leftarrow \varnothing$,    $D_{low}\leftarrow \varnothing$   4:       for each detection $d\in D_k$ do   5:             if $d.score\ge {\tau }_{high}$ then   6:                   $D_{high}\leftarrow D_{high}\cup \left\{d\right\}$   7:             else if $d.score\ge {\tau }_{low}$ then   8:                   $D_{low}\leftarrow D_{low}\cup \left\{d\right\}$   9:             end if 10:       end for (2) Trajectory state prediction 11:       $T_{pred}\leftarrow \varnothing$ 12:       for each trajectory $t\in T$ do 13:             $t\leftarrow NSA\text{-}KF\_predict\left(t\right)$ 14:             $T_{pred}\leftarrow T_{pred}\cup \left\{t\right\}$ 15:       end for (3) First-stage association (high-confidence detections) 16:       Associate $T_{pred}$ with $D_{high}$ using Density-Aware Association Strategy 17:       Obtain unmatched trajectories $T_{rem}$ and unmatched detections $D_{rem}$ (4) Second-stage association (low-confidence detections) 18:       $D_{rem}\leftarrow D_{rem}\cup D_{low}$ 19:       Associate $T_{rem}$ with $D_{low}$ using IOU distance 20:       Obtain remaining unmatched trajectories $T_{r\mathit{e-}rem}$ and detections $D_{r\mathit{e-}rem}$ (5) Trajectory management 21:       Remove unmatched trajectories: $T\leftarrow T\setminus T_{r\mathit{e-}rem}$ 22: end for 23: return T

Algorithm 1 performs two association stages sequentially. In the first stage, a density-aware association strategy is adopted to associate $D_{high}$ with the predicted trajectories $T_{prediction}$. The remaining unmatched detections $D_{remain}$ and trajectories $T_{remain}$ are then recorded.

In the second stage, the IoU distance metric is used to associate $T_{remain}$ with $D_{low}$. After association, unmatched trajectories are removed, and NSA-KF is applied to estimate the current states of the remaining trajectories. Finally, the remaining detections $D_{r\mathit{e-}remain}$ with confidence scores greater than $\upsilon$ are used to initialize new trajectories.

3.2.1. Noise-Scale Adaptive Kalman Filter

In the target tracking algorithm, the state vector is defined as $x=[x_c,y_c,w,h,{\dot{x}}_c,{\dot{y}}_c,\dot{w},\dot{h}]$, while the observation vector is defined as $z={[u,v,w,h,c]}^{\top }$. In the original ByteTrack algorithm, continuous frame-by-frame prediction is performed using the standard Kalman filter, as shown in Equation (1).

$$\left\{\begin{array}{c}{\widehat{x}}_{t|t-1}=F{x}_{t-1}\hfill \\ P_{t|t-1}=F{P}_{t-1}{F}^T+Q\hfill \end{array}\right.$$

(1)

where F denotes the state transition matrix, $P_{t|t-1}$ represents the state covariance matrix, and Q is the process noise covariance matrix, which characterizes the uncertainty of the motion model. During the measurement update stage, the observation $Z_t$ is used to correct the predicted state. The state estimate is dynamically adjusted through the Kalman gain $K_t$, as expressed in the following, Equation (2).

$$\left\{\begin{array}{c}{K}_t=P_{t|t-1}{H}^T{\left(H{P}_{t|t-1}{H}^T+R\right)}^{-1}\hfill \\ {\widehat{x}}_t={\widehat{x}}_{t|t-1}+K_t\left(Z_t-H{\widehat{x}}_{t|t-1}\right)\hfill \end{array}\right.$$

(2)

However, in maritime scenarios, when the camera undergoes severe oscillation, target vessels often appear blurred or even distorted in the image, leading to a significant decrease in the detector’s confidence scores. This degradation further causes prediction errors to accumulate over time and amplifies matching errors in the subsequent association stage. To enhance the robustness of the filtering process under such uncertainty, the Noise-Scale Adaptive Kalman Filter (NSA-KF) introduces a confidence-aware adaptive observation noise modeling mechanism, allowing the observation noise covariance to be dynamically adjusted according to detection quality. Specifically, when the detection confidence is low, the observation noise is automatically increased, making the update step rely more heavily on the predicted state to prevent erroneous detections from corrupting the trajectory. Conversely, when the confidence is high, the observation noise is reduced, enabling the filter to place greater trust in the measurement and thereby improving the timeliness and accuracy of state correction. This strategy effectively suppresses noise induced by detection jitter, image blur, or external disturbances. Without introducing additional computational overhead, it significantly reduces motion prediction errors and enhances overall tracking robustness in complex maritime environments. The gain computation formula of the NSA Kalman Filter is given in Equation (3).

$$\left\{\begin{array}{c}{R}_{\mathrm{NSA}}=R_0\left(1+\alpha (1-c)\right)\hfill \\ K_t=P_{t|t-1}{H}^T{\left(H{P}_{t|t-1}{H}^T+R_{\mathrm{NSA}}\right)}^{-1}\hfill \\ {\widehat{x}}_t={\widehat{x}}_{t|t-1}+K_t\left(z_t-H{\widehat{x}}_{t|t-1}\right)\hfill \end{array}\right.$$

(3)

In the equation, $R_0$ denotes the initial observation noise covariance matrix, $H_t$ represents the observation matrix at time step t, $R_{\mathrm{NSA}}$ is the predefined observation noise covariance matrix, and c denotes the detection confidence score.

3.2.2. Density-Aware Association Strategy

In multi-object tracking tasks, data association serves as the core step to ensure trajectory continuity, aiming to determine which historical track each observation in the current frame should correspond to. The original ByteTrack improves matching stability through a confidence-based stratification mechanism: high-confidence detections are prioritized for association, while low-confidence detections are incorporated in a second stage to supplement unmatched tracks. However, both stages rely on the same IoU-based distance metric to construct the cost matrix. When target motion is stable and detection quality is reliable, this strategy performs effectively. Nevertheless, in maritime scenarios, such a fixed association mechanism is prone to failure.

Specifically, the relationship between the number of high-confidence detections and the number of active tracks reflects the local scene density and detection reliability. When the number of active tracks exceeds that of high-confidence detections, it often indicates severe camera jitter, intensified wave disturbances, or short-term occlusions, which can lead to blurred detections and increased fluctuations in object scale. Under such conditions, the IoU between bounding boxes may be significantly underestimated due to shape variations. If the cost matrix is still constructed solely based on IoU, large-scale mismatches are likely to occur, resulting in frequent trajectory fragmentation and increased ID switches. Conversely, when the number of high-confidence detections exceeds the number of active tracks, it typically suggests that a large number of new targets are entering the scene from the image boundaries. In this case, incorporating ship category information on top of the IoU metric helps strengthen the association constraints and improves matching accuracy. These observations indicate that data association in maritime scenarios should be capable of perceiving changes in target density and adaptively adjusting the composition of the cost matrix, rather than relying on a fixed IoU-based metric. Motivated by this, a density-aware data association strategy is proposed, which enables adaptive switching of matching metrics based on the relationship between the number of tracks and high-confidence detections, thereby significantly enhancing tracking stability under platform jitter and in densely populated scenes.

The above analysis indicates that data association in maritime scenarios should be capable of perceiving variations in target density and detection confidence, and accordingly adjust the construction of the cost matrix, rather than relying on a fixed IoU-based metric. Motivated by this observation, we propose a density-aware association strategy, which adaptively switches the matching metric according to the relationship between the number of active trajectories and the number of high-confidence detections. This adaptive mechanism significantly enhances tracking stability under severe camera jitter, occlusion, and densely populated target scenarios. The pseudo-code of the proposed strategy is presented in Algorithm 2.

The core mechanism of the density-aware association strategy lies in the introduction of a density threshold r. By comparing the distribution density of current high-confidence targets, denoted as $r_{cur}$, with the threshold r, the cost matrix is adaptively constructed in a differentiated manner according to their relative magnitude. The computation process of $r_{cur}$ is shown in Equation (4).

$$r_{cur}={\displaystyle \frac{T}{D_{high}}}$$

(4)

Here, T denotes the number of active trajectories in the current frame, and $D_{\mathrm{high}}$ represents the number of high-confidence detection boxes in the current frame.

Algorithm 2 Density-Aware Association Strategy (DAAS)

Require:  Predicted trajectory set $T_{pred}$ at time t, high-confidence detections $D_{high}$, density threshold r, direction-consistency weight $\lambda$ Ensure:  Matched trajectory–detection pairs M, unmatched trajectories $T_{remain}$, unmatched detections $D_{remain}$   1: $N_T\leftarrow \left|T_{pred}\right|$,    $N_D\leftarrow \left|D_{high}\right|$   2: if $N_T=0$ or $N_D=0$ then   3:       $M\leftarrow \varnothing$   4:       $T_{remain}\leftarrow T_{pred}$   5:       $D_{remain}\leftarrow D_{high}$   6:       return $M,T_{remain},D_{remain}$   7: end if (1) Density estimation   8: $r_{cur}\leftarrow N_T/N_D$ (2) IoU cost computation   9: $C_{IoU}\leftarrow IO{U}_{cost}(T_{pred},D_{high})$ (3) Density-adaptive cost construction 10: if $r_{cur}>r$ then 11:       $V_T\leftarrow DirectionFeature\left(T_{pred}\right)$ 12:       $V_D\leftarrow DirectionFeature\left(D_{high}\right)$ 13:       $C_v\leftarrow VelocityDirectionCost(V_T,V_D)$ 14:       $C\leftarrow C_{IoU}+\lambda C_v$ 15: else 16:       $C\leftarrow Fuse\_Distance(T_{pred},D_{high})$ 17: end if (4) Hungarian matching 18: $(M,U_T,U_D)\leftarrow Hungarian\left(C\right)$ 19: $T_{remain}\leftarrow \{T_{pred}\left[i\right]\mid i\in U_T\}$ 20: $D_{remain}\leftarrow \{D_{high}\left[j\right]\mid j\in U_D\}$ 21: return $M,T_{remain},D_{remain}$

When $r_{cur}<r$, the current state is determined as the det-heavy mode. In this case, the scene is typically relatively simple and often corresponds to situations where new targets enter from the image boundaries or where the detector produces scattered false positives. Under this mode, the number of detections significantly exceeds the number of existing trajectories, and the predictive constraint provided by trajectory motion is relatively limited. Relying solely on the IoU metric for association is therefore susceptible to detection box jitter and spatial misalignment, leading to unstable matching results. To enhance the discriminative capability between targets and trajectories, we further incorporate category consistency and bounding-box scale information on top of the traditional IoU distance, and propose a fused matching metric termed Fuse-distance. This metric jointly considers geometric overlap, category consistency, and target similarity, providing richer and more stable association cues in scenarios with many detections but fewer trajectories. Consequently, it effectively reduces mismatching probability and enhances overall tracking robustness. The computation of Fuse-distance is given in Equation (5).

$${\mathrm{Fuse\_distance}}_i=1-{\mathrm{IoU}}_i·{\mathrm{Mat}}_i(\mathit{cls},\mu )$$

(5)

$$\mathrm{IoU}={\displaystyle \frac{\mathrm{Area}\mathrm{of}\mathrm{Intersection}}{\mathrm{Area}\mathrm{of}\mathrm{Union}}}$$

(6)

$$\mathrm{Mat}{(\mathit{cls},\mu )}_{(a,b)}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{track\_cls}\left(a\right)=\mathrm{det\_cls}\left(b\right)\cap {\mu }_{\mathrm{track}}={\mu }_{\det }\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(7)

Here, ${\mathrm{IoU}}_i$ denotes the IoU similarity matrix in the i-th frame, computed between all detection boxes and predicted track boxes. The variable $\mathit{cls}$ represents the target category, and $\mu$ denotes the aspect ratio (width-to-height ratio) of the bounding box. ${\mathrm{Mat}}_i$ represents the similarity matrix in the i-th frame that fuses target category and aspect ratio information, which is then multiplied element-wise with the IoU similarity matrix. Specifically, $\mathrm{track\_cls}\left(a\right)$ and $\mathrm{det\_cls}\left(b\right)$ denote the category labels of the a-th predicted box and the b-th detection box, respectively, while ${\mu }_{\mathrm{track}}$ and ${\mu }_{\det }$ represent the aspect ratios of the predicted box and the detection box.

When $r_{cur}>r$, the current state is determined as the track-heavy mode, indicating that the scene is subject to significant external disturbances, such as severe camera jitter or ocean wave motion, which cause drastic variations in detection box scales and reduce the reliability of the IoU metric. Relying solely on IoU to construct the cost matrix under such conditions may lead to severe matching failures when target positions fluctuate substantially, thereby triggering identity switches. Therefore, in the track-heavy mode, inspired by the idea of OC-SORT, historical observations (i.e., variations in the centers of detection boxes) are utilized to estimate a more stable motion direction. A direction-consistency constraint is then incorporated into the cost matrix construction. This constraint complements the IoU metric, enabling the cost matrix to simultaneously encode spatial overlap information and temporal continuity cues, thereby significantly improving matching reliability under highly dynamic motion scenarios. Given N existing trajectories and M detections at the current time step, the computation procedure of the association matrix is described in Equation (8).

$$C(\widehat{\mathbf{X}},\mathbf{Z})=C_{\mathrm{IoU}}(\widehat{\mathbf{X}},\mathbf{Z})+\lambda C_v(\mathbf{Z},\mathbf{Z})$$

(8)

Here, $\widehat{\mathbf{X}}\in {\mathbb{R}}^{N\times 7}$ denotes the set of estimated object states, and $\mathbf{Z}\in {\mathbb{R}}^{M\times 5}$ represents the set of observations at the current time step. The parameter $\lambda$ is a weighting coefficient. The set $\mathbf{Z}$ includes the observations corresponding to all existing trajectories. $C_{\mathrm{IoU}}(·,·)$ computes the IoU distance between targets, while $C_v(·,·)$ calculates the directional angular difference between two historical observations within an existing trajectory and the angular difference between the new observation to be associated and the historical observations of the trajectory. Specifically, $C_v$ contains all angular differences $\Delta \theta =\left|{\theta }^{track}-{\theta }^{intention}\right|$. In implementation, the motion direction angle is computed in radians as Equation (9)

$$\theta =arctan\left({\displaystyle \frac{v_1-v_2}{u_1-u_2}}\right)$$

(9)

where $(u_1,v_1)$ and $(u_2,v_2)$ denote observations at two different time steps. The computation procedure of the velocity direction consistency term $C_v$ is illustrated in Figure 3.

3.3. Detector

In theory, MAAT can accept the output of any object detector as its input. However, in practical maritime surveillance scenarios, the algorithm must not only maintain robustness under complex backgrounds and severe camera jitter, but also satisfy real-time processing requirements. Within the TBD paradigm, detection quality directly affects subsequent multi-object association and overall tracking stability. Therefore, selecting a reliable and high-performance detector is of critical importance. As a representative one-stage detection framework, the YOLO series achieves a favorable balance between speed and accuracy, and has evolved into multiple advanced versions, including YOLOv5, YOLOv8, YOLOv9 [42], YOLOv10 [43], YOLOv11, and YOLOv12 [44]. On the maritime vessel dataset used in this study, we conducted systematic comparative experiments among several representative YOLO models. Key performance indicators—including Precision, Recall, mAP@0.5, mAP@0.5:0.95, and inference speed—were comprehensively evaluated to determine the most suitable detector for integration into MAAT. The corresponding performance comparison is presented in

Table 1. Performance comparison of different YOLO-based object detectors.

Method	P	R	mAP@0.5	mAP@0.5:0.95	Parameters (M)	FPS
YOLOv5	0.974	0.948	0.986	0.780	2.51	92.59
YOLOv8	0.972	0.968	0.988	0.806	3.10	125.00
YOLOv9	0.974	0.948	0.988	0.803	2.00	50.00
YOLOv10	0.955	0.943	0.981	0.790	2.71	103.09
YOLOv11	0.961	0.973	0.986	0.786	2.59	121.95
YOLOv12	0.966	0.963	0.988	0.779	2.57	88.50

.

After conducting comparative experiments among multiple YOLO-series detection models, clear performance differences can be observed from Table 1. YOLOv8 achieves the highest mAP@0.5:0.95 among all models, while maintaining a competitive mAP@0.5, indicating its superior localization accuracy and overall detection performance. In addition, YOLOv8 attains the highest inference speed, which is significantly higher than that of YOLOv5 and YOLOv9, demonstrating its strong real-time capability. Although YOLOv11 achieves the highest recall, its mAP@0.5:0.95 is lower than that of YOLOv8, suggesting a trade-off between detection sensitivity and localization precision. YOLOv9 exhibits the smallest number of parameters, indicating a lightweight design; however, its relatively low inference speed limits its practical deployment in real-time maritime applications. Compared with these models, YOLOv10 and YOLOv12 show relatively balanced performance across different metrics, but neither surpasses YOLOv8 in terms of overall accuracy and efficiency. Therefore, considering both detection accuracy and real-time performance, YOLOv8 provides the most favorable trade-off and is selected as the object detection module in the proposed MAAT framework.

4. Experiment

4.1. Datasets and Metrics

To validate the effectiveness and performance of the proposed method, comprehensive evaluations were conducted on the Singapore Maritime Dataset [45]. The Singapore Maritime Dataset is a specialized benchmark designed for maritime scene analysis, containing multiple shore-based and vehicle-mounted video sequences captured in Singapore waters. The dataset covers complex maritime environments under diverse weather and illumination conditions. In this study, representative video sequences were selected and manually annotated for performance evaluation. All videos have a resolution of 1080 × 1920. The selected sequences include seven common maritime object categories: Ferry, Buoy, Vessel, Speed boat, Boat, Kayak, and Sailboat. These scenarios are highly challenging, typically involving platform jitter, nonlinear target motion, scale variation, and background interference, thereby providing a rigorous testbed for assessing the robustness and generalization capability of tracking algorithms in real-world maritime environments. The Singapore Maritime Dataset used in this study is publicly available and can be accessed via its official GitHub repository¹. It includes annotated image frames for maritime object detection and tracking tasks. Based on this dataset, the proposed method is compared with multiple mainstream state-of-the-art approaches to comprehensively evaluate its performance and advantages.

In terms of evaluation metrics, HOTA [46] and IDF1 [47] are adopted to assess maritime multi-object tracking performance. HOTA introduces a higher-dimensional tracking accuracy measure, enabling a more comprehensive and balanced evaluation of multi-object tracker performance. IDF1 evaluates the identity preservation capability and stability of the tracker. A higher IDF1 score indicates that the algorithm can more accurately maintain target identities over longer temporal durations.

The computation of HOTA is given in Equation (10), where DetA denotes the detection accuracy score and AssA represents the association accuracy score. Here, TP denotes the number of true positives; FN represents the number of positive samples incorrectly predicted as negative by the model; and FP denotes the number of negative samples incorrectly predicted as positive. Let c be a point belonging to TP. Based on this point, a unique ground-truth (GT) trajectory can be determined. $A\left(c\right)$ represents the association accuracy, as defined in Equation (11), where $\mathrm{TPA}\left(c\right)$ denotes the accuracy of correctly associated predictions, $\mathrm{FPA}\left(c\right)$ denotes the accuracy of predictions that are either unassociated or incorrectly associated, and $\mathrm{FNA}\left(c\right)$ denotes the accuracy penalty for predictions that are missing associations or have erroneous associations.

$$\mathrm{HOTA}=\sqrt{\mathrm{DetA}·\mathrm{AssA}}=\sqrt{{\displaystyle \frac{{\sum }_{c\in \mathrm{TP}}A\left(c\right)}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}}}}$$

(10)

$$A\left(c\right)={\displaystyle \frac{\mathrm{TPA}\left(c\right)}{\mathrm{TPA}\left(c\right)+\mathrm{FPA}\left(c\right)+\mathrm{FNA}\left(c\right)}}$$

(11)

The computation of IDF1 is given in Equation (12). Here, IDTP (ID True Positives) denotes the number of correctly matched identities, IDFP (ID False Positives) represents the number of incorrectly matched identities, and IDFN (ID False Negatives) denotes the number of missed identity matches.

$$\mathrm{IDF}1={\displaystyle \frac{2\mathrm{IDTP}}{2\mathrm{IDTP}+\mathrm{IDFP}+\mathrm{IDFN}}}$$

(12)

In addition, two other metrics are employed to further evaluate the model’s performance: the number of identity switches (IDS) and the average frames per second (FPS). Higher values of HOTA, IDF1, and FPS, together with lower IDS values, indicate better overall model performance.

4.2. Implementation Details and Parameter Settings

To ensure fairness, both comparative experiments and ablation studies were conducted using YOLOv8 as the detector under the PyTorch 2.9.1 framework. For detector training, the Singapore Maritime Dataset was used, and the model was trained on a single NVIDIA 4060Ti GPU. The input image resolution was set to $1080\times 1920$. The optimizer was SGD with a batch size of 16, weight decay of $5\times {10}^{-4}$, and momentum of 0.98. Data augmentation techniques included Mosaic augmentation, horizontal flipping, translation, scaling, and HSV-space augmentation. The initial learning rate was set to $1\times {10}^{-2}$ with a warm-up period of 5 epochs. A cosine annealing schedule was adopted to gradually reduce the learning rate to $2\times {10}^{-3}$. The model was trained for 150 epochs.

In the data association stage, detection boxes were divided into high-confidence and low-confidence sets according to predefined score thresholds ${\tau }_{high}$ and ${\tau }_{low}$, where ${\tau }_{high}=0.65$ and ${\tau }_{low}=0.1$. The matching threshold match_thresh = 0.98 was applied during the three-stage trajectory-to-detection association process, and matches with an area ratio greater than 4 or less than $\frac{1}{4}$ between detections and trajectories were discarded to avoid obvious mismatches. Meanwhile, the density threshold r was set to 0.9. In the track-heavy mode, to better adapt to severe vessel jitter in the scene, the direction-consistency coefficient was set to 0.4. The tracking threshold track_thresh = 0.7 indicates that detections with confidence scores higher than 0.7 that cannot be matched to existing trajectories are used to initialize new tracks.

4.3. Quantitative Experiment Analysis

4.3.1. Comparative Experiments

To comprehensively evaluate the performance and robustness of the proposed MAAT framework in complex maritime environments, extensive comparative experiments were conducted against a series of state-of-the-art multi-object tracking algorithms. The comparison benchmarks include mainstream TBD approaches, such as Deep-OC-SORT [48], ByteTrack [9], BoT-SORT [39], OC-SORT [41], and Strong-SORT [49]. All methods adopt YOLOv8 as the detector to ensure fair input conditions. In addition, since Deep-OC-SORT, BoT-SORT, and StrongSORT incorporate appearance cues during data association, dedicated Re-ID models were separately trained for these methods to guarantee the completeness and fairness of the comparison. The quantitative comparison results are presented in

Table 2. Performance comparison of different MOT methods.

MOT Method	Re-ID	HOTA↑	IDF1↑	IDS↓	FPS↑
OC-SORT		42.446	42.664	24	41.89
Deep-OC-SORT	✓	42.540	42.131	24	17.48
Strong-SORT	✓	42.510	42.464	23	16.43
ByteTrack		42.909	42.516	24	41.94
BoT-SORT	✓	43.289	42.915	20	16.18
MAAT		44.370	43.857	20	41.40

✓ indicates that the Re-ID module is enabled. ↑ indicates that higher values are better, while ↓ indicates that lower values are better.

.

As shown in Table 2, the evaluated trackers exhibit noticeable differences in key metrics, including overall accuracy (HOTA), identity preservation (IDF1), and the number of identity switches (IDS). MAAT achieves the best or jointly best performance across these metrics, demonstrating strong robustness under complex maritime disturbances.

For the HOTA metric, which reflects the overall trajectory quality, MAAT achieves a score of 44.37, outperforming ByteTrack and BoT-SORT by 1.461 and 1.081 points, respectively. Since HOTA jointly evaluates detection and association quality, this improvement indicates that the proposed framework effectively alleviates the instability of IoU-based association under camera jitter and target scale variations through adaptive filtering and velocity direction consistency constraints.

For identity preservation, MAAT achieves the highest IDF1 score of 43.857, exceeding ByteTrack and BoT-SORT by 1.341 and 0.942 points, respectively. Because IDF1 reflects identity assignment accuracy and long-term identity consistency, these results suggest that MAAT can more reliably recover correct identities after occlusion, detection jitter, or short-term target loss. Notably, although BoT-SORT integrates Re-ID features to enhance identity consistency, MAAT achieves superior IDF1 performance without using appearance information, indicating that the improvement mainly originates from enhanced motion modeling and a more robust association strategy.

In terms of efficiency, MAAT runs at 41.4 FPS, which is close to the lightweight methods OC-SORT (41.89 FPS) and ByteTrack (41.94 FPS). At the same time, it operates more than 2.5 times faster than Re-ID based methods such as BoT-SORT (16.18 FPS) and StrongSORT (16.43 FPS). This demonstrates that MAAT achieves strong tracking robustness while maintaining high computational efficiency, making it suitable for real-time maritime surveillance.

A comparison among the baseline trackers further reveals several observations. OC-SORT and Deep-OC-SORT show very similar performance, indicating that introducing Re-ID does not necessarily provide consistent improvements in this dataset. Under maritime conditions characterized by blur, scale variation, and low resolution, appearance descriptors may be less reliable. Although StrongSORT slightly reduces identity switches, its HOTA and IDF1 scores remain lower than those of ByteTrack. These results suggest that the main challenge in this scenario lies in motion disturbance and association uncertainty rather than appearance discrimination. By introducing adaptive filtering and a density-aware association strategy, MAAT provides constraints that better match the characteristics of maritime scenes and consequently achieves the best overall performance without relying on Re-ID features.

To provide an intuitive comparison of the tracking performance of different algorithms under severe maritime scene jitter, qualitative visualization analysis is conducted in this section. The comparison results are illustrated in Figure 4.

As shown in Figure 4, particularly in Frame 134 and Frame 195, clear differences in tracking performance among different methods can be observed. For classical methods such as ByteTrack and OC-SORT that do not rely on Re-ID, strong camera oscillation significantly degrades tracking stability. In these frames, ByteTrack exhibits both identity switches (ID Switch) and missed detections (Miss), while OC-SORT also suffers from frequent identity switches. Although both methods can maintain relatively stable trajectories for large vessels, small-scale targets exhibit noticeable fluctuations in bounding box position and scale under heavy image jitter, which weakens IoU-based association.

Compared with these methods, BoT-SORT and Deep-OC-SORT partially mitigate this problem by incorporating Re-ID features and camera motion compensation (CMC), resulting in more stable tracking of small vessels. However, occasional mismatches can still be observed in Frame 134, indicating that these methods remain sensitive to severe platform jitter. In addition, the additional appearance extraction and motion estimation introduce substantial computational overhead, reducing the runtime to around 17 FPS.

StrongSORT further improves appearance discrimination and maintains relatively stable tracking in most frames. However, identity switches still occur when large vessels experience severe jitter, as shown in Frame 134, while the heavy reliance on Re-ID also leads to relatively low inference speed.

In contrast, the proposed MAAT achieves consistently stable tracking results across all frames, without observable identity switches or missed detections. Compared with all the above methods, MAAT maintains correct identity assignment even under severe platform-induced jitter, as demonstrated in Frame 134 and Frame 195. This improvement is mainly attributed to the adaptive noise-aware motion modeling and density-aware association strategy, which effectively enhance robustness under unstable observations.

4.3.2. Ablation Experiments

The quantitative and qualitative comparisons presented above have demonstrated the superiority of the proposed MAAT framework. However, to scientifically and quantitatively isolate the specific contribution of each innovative module, comprehensive ablation studies are conducted in this section. In these experiments, ByteTrack is adopted as the baseline model, and the core components of MAAT are incrementally integrated, including the NSA-KF and the density-aware association strategy. In addition, whether to incorporate the Fuse-distance metric into the density-aware association mechanism is also treated as a separate ablation factor. The corresponding results are reported in

Table 3. Ablation study of different components in the MAAT framework.

ByteTrack	NSA-KF	DAAS	Fuse-Distance	HOTA ↑	IDF1 ↑	FPS ↑
✓	×	×	×	42.909	42.516	41.94
✓	✓	×	×	42.978	42.525	41.87
✓	×	✓	×	44.129	43.559	41.68
✓	✓	✓	×	44.342	43.587	41.53
✓	✓	✓	✓	44.37	43.857	41.4

✓ indicates that the corresponding component is applied, while × indicates that the component is not applied. ↑ indicates that higher values are better.

.

The comparison results in Table 3 illustrate the tracking performance variations when NSA-KF, DAAS, and Fuse-distance are progressively integrated into the ByteTrack baseline. Using the original ByteTrack alone, HOTA and IDF1 reach 42.909 and 42.516, respectively, indicating that the single IoU-based matching mechanism provides limited robustness in complex maritime environments.

After introducing NSA-KF, HOTA slightly increases to 42.978 while IDF1 remains nearly unchanged, suggesting that direction-aware adaptive Kalman filtering can moderately improve state prediction but contributes less to identity association. When the density-aware DAAS is introduced independently, HOTA and IDF1 significantly increase to 44.129 and 43.559, respectively, demonstrating that the density-aware association mechanism effectively captures the imbalance between detections and trajectories in maritime scenes and substantially improves association accuracy and identity consistency.

Combining NSA-KF with DAAS further improves both metrics, indicating that the two modules complement each other by enhancing motion modeling and association robustness simultaneously. In the complete model, after incorporating Fuse-distance, HOTA reaches 44.370 and IDF1 increases to 43.857, achieving the best overall performance. This improvement suggests that integrating category consistency and bounding box scale information into the IoU metric provides more discriminative constraints for trajectory–detection association. Meanwhile, FPS decreases only slightly from 41.94 to 41.4, indicating negligible computational overhead. Overall, the proposed multi-module design significantly improves tracking accuracy while maintaining strong real-time performance, making it well suited for maritime target tracking applications.

5. Conclusions

In this paper, a Marine-Aware Adaptive Tracker (MAAT) was proposed for vision-based multi-object tracking in complex maritime environments. Built upon the ByteTrack framework, MAAT introduces a Noise-Scale Adaptive Kalman Filter to improve state estimation under unstable maritime disturbances and a density-aware association strategy to enhance matching robustness under varying scene densities. In addition, a Fuse-distance metric was designed to provide more discriminative constraints in det-heavy scenarios.

Experimental results on the Singapore Maritime Dataset demonstrated that the proposed method achieves superior overall performance compared with several state-of-the-art trackers. MAAT not only improves HOTA and IDF1, but also maintains real-time processing capability at 41.4 FPS. The ablation studies further confirmed the effectiveness of each proposed module, especially the substantial contribution of the density-aware association strategy to association accuracy and identity consistency. These results indicate that MAAT is capable of achieving robust and efficient maritime multi-object tracking without relying on Re-ID features or explicit camera motion compensation.

The proposed MAAT framework has important practical implications for maritime surveillance and intelligent navigation systems. By improving tracking robustness under severe platform-induced jitter disturbances, the method enhances the reliability of continuous vessel monitoring, which is critical for maritime traffic management, collision avoidance, and safety supervision.

Despite the promising performance of the proposed MAAT framework, several limitations remain. Although the proposed approach improves robustness under platform-induced jitter, its performance may still be affected under extreme environmental conditions such as heavy fog or low visibility. In addition, the current framework relies on predefined parameters (e.g., confidence thresholds), which may require further tuning for different datasets or application scenarios. Future work will focus on addressing these limitations by improving tracking performance under severe occlusion, low-visibility conditions, and more densely populated maritime scenes.