AttriMOT: Semantic-Aware Multimodal 3D Multi-Object Tracking with Attribute-Level Alignment

Abstract

3D multi-object tracking (MOT) in complex and dynamic environments remains challenging due to the time-varying reliability of sensor modalities, severe occlusions, and the difficulty of distinguishing instances with similar appearances. Existing methods mainly rely on coarse category-level semantics or heuristic multimodal fusion strategies, which limits fine-grained instance discrimination and leads to unstable trajectory association under complex scenarios. Moreover, current 3D MOT frameworks generally lack the ability to leverage attribute-level semantic information for robust tracking and semantic-aware target retrieval. To address these limitations, we propose AttriMOT, a semantic-aware multimodal 3D MOT framework. Specifically, a category semantic anchoring and competition suppression mechanism is introduced to preserve discriminative fine-grained attribute information among visually similar instances. An attribute-level multimodal alignment module establishes structured correspondences across 3D geometry, 2D appearance, and textual semantics, enabling robust cross-modal representation learning. Furthermore, a parameter-free adaptive confidence fusion strategy dynamically balances LiDAR- and camera-derived trajectory confidence to improve tracking stability under varying environmental conditions. In addition, a semantic-aware trajectory selector is designed to support text-specified target retrieval and trajectory locking, enabling controllable semantic-guided 3D tracking. Extensive experiments on challenging 3D MOT benchmarks demonstrate that AttriMOT consistently outperforms state-of-the-art methods in tracking accuracy and robustness. In particular, AttriMOT achieves 1.33% improvement in HOTA and 0.54% improvement in MOTA compared with the best existing method, while also providing enhanced semantic controllability and text-guided tracking capability.

1. Introdution

Multi-object tracking (MOT) in autonomous driving and robotic perception requires robust and consistent identity association across time under complex and dynamic environments [1]. In autonomous driving systems, 3D multi-object tracking is commonly achieved by fusing data from cameras and LiDAR sensors to overcome the inherent limitations of single-modal approaches [2,3,4]. Specifically, camera-acquired images contain rich semantic information and appearance details, which facilitate stable object tracking in the two-dimensional image plane. Meanwhile, LiDAR-generated point clouds provide high-precision three-dimensional geometric information, thereby enhancing the spatial perception of objects [5]. By fully exploiting the complementary strengths of image and point cloud data, the accuracy and robustness of multi-object tracking can be significantly improved.

Building upon this multimodal sensing paradigm [6], a number of camera–LiDAR fusion based 3D MOT methods have been proposed to better exploit visual and geometric complementarity. MSA-MOT [7] introduces a reliability-aware multi-stage association framework with adaptive track management to improve matching stability, while DeepFusion-MOT [8] employs a deep association mechanism to enable dynamic 2D–3D trajectory fusion for a better balance between accuracy and efficiency. In addition, PolarMOT [9] adopts a geometry-centric paradigm by modeling 3D detections as graph nodes and encoding spatial–temporal relations via polar coordinates, enabling robust data association through graph-based reasoning across diverse scenes. Although existing 3D tracking methods have achieved promising performance, most of them treat semantic category prediction merely as the final classification output or directly fuse category-level semantic embeddings with tracking features. Such designs suffer from two key limitations. First, category semantics tend to dominate the shared embedding space, thereby suppressing the fine-grained attribute information that is crucial for distinguishing instances within the same category. Second, due to the lack of structured semantic modeling, appearance features are often incorporated into the tracking pipeline in a heuristic manner, leading to poor robustness under viewpoint changes or illumination variations.

Moreover, how to effectively integrate 2D and 3D trajectory confidence remains an open and critical problem. In practical multimodal 3D MOT systems, the reliability of different sensing modalities varies significantly across temporal and environmental conditions. LiDAR-based representations often suffer from sparse or incomplete point clouds under long-range observations, occlusions, and limited viewpoints, while image-based trajectory estimation is more sensitive to illumination variations, motion blur, and visually similar distractors. In addition, multimodal fusion frameworks commonly face challenges related to cross-modal registration inconsistency and the increased computational complexity introduced by LiDAR processing and feature alignment [10]. Recent advances in monocular 3D reasoning and 3D visual grounding, such as Mono3DVG [11], have demonstrated that spatial query modeling and localized feature fusion can significantly enhance semantic-aware 3D perception and object localization. However, these ideas have not been sufficiently explored in multimodal 3D MOT, particularly for robust trajectory association and semantic-guided tracking under dynamic environments. Consequently, existing fusion strategies, which usually rely on fixed or heuristic fusion mechanisms, often fail to dynamically adapt to varying modality reliability, resulting in suboptimal multimodal association and trajectory estimation.

In recent years, the rapid development of vision–language models has provided new perspectives for multimodal semantic modeling, enabling effective alignment between visual features and textual semantics. However, directly introducing textual embeddings into object tracking frameworks remains nontrivial. Figure 1 illustrates the limitations of existing VLM-based multimodal tracking frameworks. Current methods mainly rely on coarse category-level semantic representations and lack effective fine-grained attribute modeling, making it difficult to distinguish visually similar instances in complex environments. Existing large-model-based tracking approaches do not yet achieve true semantic intelligence; instead, they mainly exploit latent category- or attribute-level semantics of objects, without the ability to explicitly specify or consistently track a particular target. Moreover, naïve multimodal fusion often intensifies the competition between category-level semantics and attribute-level semantics, which can even degrade tracking performance. In addition, existing methods usually fail to explicitly distinguish attributes dominated by geometric cues from those primarily determined by appearance cues, leading to ambiguous semantic supervision and suboptimal utilization of multimodal information.

To address the above challenges, we propose a multi-stage semantic-aware multimodal tracking framework that achieves fine-grained vision–language alignment for robust instance-level tracking. By anchoring category semantics while suppressing their dominance, and aligning geometric, appearance, and motion attributes across 3D, 2D, and textual modalities, our method enables structured and discriminative object representation. An adaptive Softmax-based fusion further balances LiDAR and camera confidence, and a semantic-aware trajectory selector supports precise text-guided target tracking beyond coarse category-level association. Specifically, our contributions are summarized as follows:

• To address the limitation of existing 3D MOT methods that rely on coarse category-level semantics and struggle with fine-grained instance discrimination and semantic-aware target retrieval, we propose AttriMOT, a novel semantic-aware multimodal 3D MOT framework. The proposed framework explicitly aligns object-level attributes to enable controllable instance discrimination and text-guided tracking.
• To address the issue that fine-grained attribute information is easily dominated by category-level semantics in existing multimodal representation learning methods, we design a category semantic anchoring and competition suppression mechanism, which preserves discriminative attribute information by treating category embeddings as stable semantic anchors while suppressing their dominant components in the shared embedding space.
• To address the lack of structured attribute-level correspondence modeling in existing cross-modal association methods, which leads to unstable association and weak interpretability, we introduce an attribute-based multimodal alignment paradigm that establishes fine-grained structured correspondences between visual features and textual embeddings for robust and interpretable cross-modal association.
• To address the inability of existing multimodal fusion methods to adapt to dynamically varying modality reliability, we develop a parameter-free adaptive confidence fusion strategy that dynamically balances LiDAR- and camera-derived information through a Softmax mechanism, thereby improving the stability of trajectory confidence estimation.

2. Relation Work

2.1. 3D Multiobject Tracking

Existing 3D multi-object tracking (3D MOT) methods generally take 3D detection results as input and aim to improve tracking performance by enhancing detection reliability, association robustness, and trajectory consistency. To mitigate the impact of noisy detections, several studies focus on detection-level refinement [12]. Poly-MOT [13] preprocesses multi-class detection results using non-maximum suppression and confidence filtering, effectively reducing inconsistencies between detection and tracking stages. Building upon this framework, Fast-Poly [14] further introduces voxel masking and a coarse-to-fine association strategy, achieving significant improvements in tracking efficiency.

Beyond detection refinement, a substantial body of work concentrates on modeling the association between detections and existing trajectories [15]. UG3DMOT [16] characterizes the uncertainty between trajectories and detections using Jensen–Shannon divergence to measure similarities between multidimensional distributions. Graph-based approaches, such as 3DMOTFormer [17] and LEGO [18], formulate data association as inference over trajectory–detection graphs, leveraging graph transformers or joint learning and optimization to improve matching accuracy. InterTrack [19] aggregates global contextual information via attention mechanisms and evaluates affinities through learned regression, while MC-MOT [20] designs a unified global association step based on fractional optimal transport. For multimodal fusion, JMODT [21] employs attention mechanisms to jointly learn from point cloud and image data; however, its cross-modal alignment during data processing limits localization accuracy. In addition, multistage or multi-similarity association strategies adopted by EagerMOT [22], ByteTrackv2 [23], and MCCA-MOT [24] improve matching performance but may suffer from identity switches or increased computational overhead, particularly when 3D IoU values are low.

Trajectory-level optimization has also been extensively explored to maintain temporal consistency [25]. PF-Track [26] incorporates both historical and future state inference to refine trajectory associations, while YONTD-MOT [27] and TG3MOT [28] unifies detection and tracking within a single model and represents trajectory states using regression confidence. PnPNet [29] introduces a novel trajectory representation based on point cloud inputs and exploits trajectory-level features for motion prediction. Rethink MOT [30] and RobMOT [31] further enhance tracking robustness through trajectory repair and multi-stage observation gating mechanisms, respectively.

Overall, although existing 3D MOT approaches have achieved notable progress by emphasizing detection refinement, association modeling, and trajectory maintenance, their primary focus remains on geometric, motion, and temporal consistency. The intrinsic attributes of the objects themselves are largely overlooked, and current methods rarely distinguish targets based on their inherent semantic or appearance-level properties, which limits their effectiveness in scenarios involving visually or structurally similar objects.

2.2. Pre-Trained Vision-Language Models

In recent years, vision–language (VL) supervision has emerged as a prominent research direction in computer vision. The core idea is to construct a unified cross-modal semantic embedding space through large-scale image–text joint learning, thereby enhancing model generalization capability and open-set understanding in complex visual environments. A representative work in this line of research is CLIP [32], which adopts a contrastive learning paradigm to align images with their corresponding textual descriptions in a shared embedding space, enabling visual understanding driven by language prompts. Owing to its strong transferability and open-vocabulary capability, CLIP has attracted significant attention and has been widely extended to various downstream tasks [33].

Building upon CLIP, a series of vision–language pre-trained models have been proposed to accommodate different visual tasks. For instance, ALIGN [34] further strengthens cross-modal alignment by leveraging larger-scale image–text datasets, while UniCL [35] improves multi-task transferability through a unified contrastive learning framework. These approaches primarily focus on image-level semantic understanding and provide strong semantic priors for downstream vision tasks.

For object-level perception, ViLD [36] and RegionCLIP [37] incorporate VL representations into object detection frameworks, enabling open-vocabulary object recognition. GLIP [38] further unifies object detection and phrase grounding into a single framework, achieving object localization and recognition under language guidance. Subsequently, Grounding DINO [39] improves both detection accuracy and cross-modal alignment quality. In addition, X-DETR [40] and OWL-ViT [41] explore open-vocabulary object detection from the perspectives of Transformer-based architectures and visual encoder design, respectively.

At a finer-grained perception level, LSeg [42], DenseCLIP [43], OpenSeg [44], and MaskCLIP [45] extend vision–language representations to pixel-level semantic modeling, enabling open-vocabulary image segmentation. These methods demonstrate that VL models are capable of capturing not only global semantic information but also localized regions and fine-grained attribute cues.

Although the aforementioned vision-language approaches are primarily developed for static vision tasks such as classification, detection, and segmentation, their strengths in semantic understanding, attribute modeling, and cross-category generalization offer new perspectives for object tracking. Traditional tracking methods typically rely on geometric cues, motion consistency, or appearance similarity for data association, which can be fragile in scenarios involving similar-looking objects, category ambiguity, or long-term occlusions. In contrast, vision–language models can explicitly encode semantic attributes and discriminative characteristics through language supervision, providing higher-level semantic cues for cross-frame identity association in tracking.

3. Proposed Method

AttriMOT is a semantic-aware multimodal 3D multi-object tracking framework that integrates LiDAR, camera, and textual information to achieve fine-grained, text-guided tracking. As shown in the Figure 2, it operates in four stages: first, 2D visual features are aligned with category-level textual embeddings while removing category-dominant components to preserve instance-level attributes; second, object semantics are decomposed into geometric, appearance, and motion attributes and aligned across 3D, 2D, and textual modalities for robust cross-modal representation; third, trajectory confidence from LiDAR and camera is dynamically fused via a parameter-free Softmax mechanism according to modality reliability; and fourth, candidate trajectories are evaluated against a textual query using attribute-level semantic alignment to select the target. By combining geometry, appearance, and textual semantics at the attribute level, AttriMOT achieves robust, interpretable, and text-specified 3D multi-object tracking. Our overall tracking framework is built upon the TG3MOT baseline.

3.1. Stage I: Semantic Anchoring with Category Competition Suppression

The first stage aims to establish category-level semantic anchors while explicitly suppressing semantic competition between category and attribute representations. Instead of treating category prediction as a final classification outcome, we reinterpret category semantics as anchor points that define a reference frame in the vision-language embedding space. Based on this, we introduce a Category Semantic Anchoring and Competition Suppression (SACS) module, as shown in Figure 3. In this stage, region-level visual features are extracted using RegionCLIP to provide high-quality vision–language aligned representations.

Given a set of region-level visual features $\left\{{\mathbf{v}}_i^t\right\}$ extracted from the detector at time step t, we construct a category anchor set:

$${\mathcal{T}}^{\mathrm{anchor}}=\{{\mathbf{t}}_1,{\mathbf{t}}_2,\dots ,{\mathbf{t}}_C\},$$

(1)

where each textual embedding encodes only category-level semantics without attribute descriptions.

For each region feature ${\mathbf{v}}_i^t$, we compute its similarity to all category anchors and obtain a category response distribution:

$$s_{i,c}=\cos ({\mathbf{v}}_i^t,{\mathbf{t}}_c),{\mathbf{p}}_i^{\mathrm{cat}}=\mathrm{Softmax}\left({\displaystyle \frac{s_{i,c}}{\tau }}\right),$$

(2)

where $\tau$ denotes a temperature parameter controlling the sharpness of the distribution. Unlike hard category assignment, the soft distribution ${\mathbf{p}}_i^{\mathrm{cat}}$ preserves semantic uncertainty and enables robust modeling under ambiguous observations.

To mitigate the dominance of category semantics in the shared embedding space, we further perform semantic debiasing by removing the category-induced components from the visual representation:

$${\tilde{\mathbf{v}}}_i^t={\mathbf{v}}_i^t-{\displaystyle \sum _{c=1}^C}{\mathbf{p}}_i^{\mathrm{cat}}·{\mathbf{t}}_c.$$

(3)

The resulting debiased feature ${\tilde{\mathbf{v}}}_i^t$ retains attribute-sensitive information while being anchored to a stable category reference. This operation effectively suppresses category-level semantic competition and prepares the representation for fine-grained attribute alignment in the subsequent stage.

3.2. Stage II: Attribute-Level Fine-Grained Multimodal Alignment

To effectively exploit complementary information across multiple modalities in 3D object tracking, we introduce an attribute-level fine-grained multimodal alignment module (AFMA), as shown in Figure 4. By explicitly modeling object semantics as a set of interpretable attributes, the proposed module establishes structured correspondences among 3D geometry, 2D appearance, and textual semantics, thereby enhancing identity consistency over time.

At time step t, the i-th object is represented by a 3D bounding box predicted by the 3D detector,

$${\mathbf{b}}_i^t=(x_i^t,y_i^t,z_i^t,l_i^t,w_i^t,h_i^t,{\theta }_i^t),$$

(4)

where $(x_i^t,y_i^t,z_i^t)$ denote the 3D center location, $(l_i^t,w_i^t,h_i^t)$ describe the physical dimensions, and ${\theta }_i^t$ is the object orientation. Based on these geometric parameters, geometry-driven semantic attributes are derived through deterministic mapping functions,

$$a_i^{\mathrm{size}}=f_{\mathrm{size}}(l_i^t,w_i^t,h_i^t),a_i^{\mathrm{dist}}=f_{\mathrm{dist}}\left(z_i^t\right),$$

(5)

where $f_{\mathrm{size}}(·)$ and $f_{\mathrm{dist}}(·)$ discretize continuous geometric quantities into semantically meaningful categories.

To associate each 3D object with its 2D appearance representation, we adopt a simple yet robust mapping strategy. Specifically, for each 3D object, the 2D detection or tracking box associated with it in the same frame is directly used as the corresponding image-plane region,

$${\mathcal{R}}_i^t=(u_{i,1}^t,v_{i,1}^t,u_{i,2}^t,v_{i,2}^t),$$

(6)

where $(u_{i,1}^t,v_{i,1}^t)$ and $(u_{i,2}^t,v_{i,2}^t)$ denote the top-left and bottom-right corners of the 2D bounding box, respectively. This mapping does not rely on explicit camera geometry, but instead leverages spatial consistency between 3D and 2D detections.

Given the associated 2D region, an appearance feature vector is extracted using a pretrained 2D visual backbone,

$${\mathbf{v}}_i^t\in {\mathbb{R}}^d,$$

(7)

which serves as a unified representation of the visual modality for subsequent attribute-level alignment.

For appearance-related attributes such as color, we construct a set of textual attribute embeddings,

$${\mathcal{T}}^{\mathrm{color}}=\{{\mathbf{t}}_1^{\mathrm{color}},{\mathbf{t}}_2^{\mathrm{color}},\dots ,{\mathbf{t}}_K^{\mathrm{color}}\},$$

(8)

where each embedding corresponds to a fine-grained semantic description of a color attribute. The color attribute of object i is determined by maximizing the similarity between its appearance feature and the textual attribute embeddings,

$$a_i^{\mathrm{color}}=\arg \underset{k}{\max }\mathrm{sim}({\mathbf{v}}_i^t,{\mathbf{t}}_k^{\mathrm{color}}),$$

(9)

where $\mathrm{sim}(·,·)$ denotes cosine similarity. This process explicitly aligns local visual appearance with semantic attribute descriptions in the textual embedding space.

Motion attributes are estimated from temporal changes in 3D object positions. Given two consecutive frames, the instantaneous velocity of object i is computed as

$${\mathbf{u}}_i^t={\displaystyle \frac{(x_i^t,y_i^t,z_i^t)-(x_i^{t-1},y_i^{t-1},z_i^{t-1})}{\Delta t}},$$

(10)

and mapped to a discrete motion attribute via

$$a_i^{\mathrm{motion}}=f_{\mathrm{motion}}(\parallel {\mathbf{u}}_i^t\parallel ).$$

(11)

By integrating geometric, appearance, and motion attributes, the attribute-level representation of object i at time t is defined as

$${\mathbf{a}}_i^t=\{a_i^{\mathrm{color}},a_i^{\mathrm{size}},a_i^{\mathrm{dist}},a_i^{\mathrm{motion}}\}.$$

(12)

This compact attribute set serves as an intermediate representation that bridges 3D geometry, 2D appearance, and textual semantics, enabling structured multimodal alignment.

The hand-crafted priors used in this work, including size, distance, and motion information, are mainly introduced as lightweight auxiliary constraints rather than strict deterministic rules. Moreover, the parameter ranges of these priors are carefully designed according to realistic physical-world driving scenarios and object motion characteristics. Specifically, the constraints on object size, spatial distance, and motion continuity are defined based on common geometric and temporal properties observed in autonomous driving environments.

During data association, an attribute alignment score is introduced to measure the fine-grained semantic consistency between an existing trajectory i and a candidate detection j,

$$S_{\mathrm{attr}}(i,j)=\sum _k{w}_k·s\left(a_i^k,a_j^k\right),$$

(13)

where k indexes attribute types, $w_k$ denotes the importance weight of each attribute, and $s(·,·)$ is an attribute-level similarity function. For discrete attributes, $s(·,·)$ can be defined as an indicator function, while for soft or continuous attributes, normalized similarity measures can be applied. The attribute alignment score is combined with geometric and motion consistency terms to guide trajectory association and update.

3.3. Stage III: Adaptive Softmax-Based Confidence Fusion

In practical 3D object tracking systems, trajectory confidence is often estimated from multiple heterogeneous modalities, such as LiDAR-based 3D regression scores and image-based 2D appearance scores. However, the reliability of these modalities varies dynamically over time due to occlusion, sparsity of point clouds, illumination changes, and viewpoint variations. Directly applying fixed-weight fusion or simple averaging fails to account for such temporal and modality-dependent uncertainty, often leading to unstable confidence estimation and identity switches.

To address this issue, as shown in Figure 4, we introduce an adaptive Softmax-based confidence fusion (ASCF) mechanism that dynamically balances the contributions of 3D and 2D modalities according to their relative confidence levels, without introducing additional learnable parameters.

Given the historical confidence scores of an object trajectory from LiDAR and camera modalities, denoted as $s^{\mathrm{lidar}}t$ and $s^{\mathrm{camera}}t$, we first apply a lightweight temporal smoothing operation to suppress short-term fluctuations while preserving the overall confidence trend. Specifically, a weighted average is computed over recent scores, with larger weights assigned to more recent observations:

$${\overline{s}}_t^m={\displaystyle \frac{{\sum }_{k=1}^T{w}_k{s}_{t-k}^m}{{\sum }_{k=1}^T{w}_k}},m\in \{\mathrm{lidar},\mathrm{camera}\}$$

(14)

where $w_k$ increases linearly with time, emphasizing recent measurements.

Instead of manually assigning fusion weights, we compute modality-specific weights using a temperature-scaled Softmax function:

$${\alpha }^m={\displaystyle \frac{\exp ({\overline{s}}^m/\tau )}{{\sum }_{n\in \mathrm{lidar},\mathrm{camera}}\exp ({\overline{s}}^n/\tau )}},$$

(15)

where $\tau$ is a temperature parameter controlling the sharpness of the weighting distribution. A smaller $\tau$ encourages the fusion to focus on the modality with higher confidence, while a larger $\tau$ leads to a more balanced contribution.

This design enables an exponential amplification of relative confidence differences, allowing the fusion process to automatically favor the more reliable modality under challenging conditions, without explicitly suppressing the other modality.

The final trajectory confidence score is obtained by a weighted sum of the smoothed modality scores:

$$s^{\mathrm{fusion}}={\alpha }^{\mathrm{lidar}}·{\overline{s}}^{\mathrm{lidar}}+{\alpha }^{\mathrm{camera}}·{\overline{s}}^{\mathrm{camera}}.$$

(16)

Compared with variance-based or heuristic fusion strategies, the proposed method preserves confidence continuity when both modalities improve simultaneously and avoids penalizing consistent score growth. Moreover, since the fusion mechanism is parameter-free and differentiable, it can be seamlessly integrated into existing tracking pipelines without additional training or computational overhead.

3.4. Stage IV: Semantic-Aware Trajectory Selector

Based on the fine-grained vision–language alignment established in the previous stages, we further introduce a semantic-aware trajectory selector to identify the target trajectory that best matches a given textual description, as shown in the Figure 5. Unlike conventional tracking frameworks that rely solely on geometric proximity or motion consistency, our selector leverages semantic attribute representations to perform text-conditioned trajectory retrieval.

Let $\mathcal{T}=\{{\tau }_1,{\tau }_2,\dots ,{\tau }_K\}$ denote the set of candidate trajectories at the current time step. For each trajectory ${\tau }_k$, we aggregate the attribute-sensitive visual representations obtained in Stage I and Stage II into a trajectory-level semantic descriptor:

$${\mathbf{a}}_k={\displaystyle \frac{1}{|{\tau }_k|}}\sum _{t\in {\tau }_k}{\tilde{\mathbf{v}}}_k^t,$$

(17)

where ${\tilde{\mathbf{v}}}_k^t$ denotes the category-debiased visual feature at time step t.

Given a textual query embedding ${\mathbf{t}}^{\mathrm{query}}$, which encodes attribute-oriented descriptions of the target object, we compute the semantic consistency score between the query and each candidate trajectory as:

$$S_{\mathrm{sem}}({\tau }_k,{\mathbf{t}}^{\mathrm{query}})=\cos ({\mathbf{a}}_k,{\mathbf{t}}^{\mathrm{query}}).$$

(18)

The final trajectory is selected by ranking all candidates according to their semantic consistency scores, optionally combined with motion and confidence cues:

$${\tau }^{*}=\arg \underset{{\tau }_k\in \mathcal{T}}{\max }S\left({\tau }_k\right).$$

(19)

By explicitly incorporating fine-grained semantic attributes into trajectory selection, the proposed selector enables robust identification of text-specified targets under complex multi-object scenarios.

4. Experiment

4.1. Datasets

We evaluate AttriMOT on the KITTI multi-object tracking benchmark, which provides synchronized RGB images, 3D LiDAR scans, and trajectory annotations for multiple object categories, including cars, pedestrians, and cyclists. The dataset consists of 21 training sequences and 29 test sequences.

4.2. Evaluation Metrics

In this paper, we evaluate the proposed method using two widely adopted tracking performance metrics, namely HOTA and a set of classical tracking measures. The classical metrics include multi-object tracking accuracy (MOTA), multi-object tracking precision (MOTP), and the number of identity switches (IDSW), which together reflect localization quality, overall tracking accuracy, and identity consistency in multi-object tracking. Specifically, MOTA comprehensively evaluates tracking performance by considering false positives, missed detections, and identity association errors, while MOTP measures the localization precision between predicted trajectories and ground truth targets. IDSW quantifies the frequency of identity changes during trajectory association and is commonly used to assess identity preservation capability.

HOTA is the primary evaluation metric and provides a balanced assessment by jointly considering detection accuracy (DetA) and association accuracy (AssA), enabling a comprehensive evaluation of tracking performance under complex scenarios. Compared with traditional metrics, HOTA better reflects the balance between detection quality and association consistency, making it more suitable for evaluating modern multi-object tracking systems.

4.3. Experimental Setup

All experiments were carried out on a workstation equipped with an NVIDIA RTX 4090 GPU with 24 GB memory. Our framework is implemented on top of the TG3MOT tracker, which serves as the baseline tracking system. VoxelRCNN is used to generate 3D detections and initialize object tracks. For the visual–language representation, we adopt RegionCLIP to extract region-level image features from 2D detection boxes and project them into a shared vision–language embedding space. Category-level textual prompts are manually defined following a unified template format, such as “a photo of a red car”, “a photo of a blue truck”, and “a photo of a white van”, and are encoded by the RegionCLIP text encoder to provide semantic anchors for our proposed modules.

All experiments in this work are conducted on the KITTI benchmark dataset. Therefore, the sensor configuration, calibration parameters, and sensor placement follow the official KITTI data acquisition setup. Specifically, the dataset employs a synchronized multimodal sensing system consisting of RGB cameras and a Velodyne LiDAR sensor, where the calibration and coordinate alignment information between different modalities are provided by the official benchmark. In our framework, the multimodal feature extraction and trajectory association processes are performed based on these predefined calibration settings to ensure consistent spatial alignment between image and point cloud representations.

4.4. Experimental Analysis

4.4.1. Comparison with State-of-the-Art Methods

Table 1. Performance comparison on the Kitti test sets.

Method	Published	HOTA (%) ↑	DetA (%) ↑	AssA (%) ↑	LocA (%) ↑	MOTA (%) ↑	IDSW ↓
MSA-MOT [7]	Sensors 2022	78.52	75.19	82.56	87.00	88.01	91
DeepFusion-MOT [8]	RA-L 2022	75.46	71.54	80.05	86.70	84.63	84
PolarMOT [9]	ECCV 2022	75.16	73.94	76.95	87.12	85.08	462
EagerMOT [22]	ICRA 2021	74.39	75.27	74.16	87.17	87.82	239
YONTD-MOT [27]	RA-L 2024	78.08	74.16	82.86	88.23	85.09	42
TG3MOT [28]	Applied Science 2025	78.72	74.59	83.69	87.64	86.15	35
Mono-3D-KF	FUSION 2021	75.47	74.10	77.63	85.48	88.48	162
PNAS-MOT [46]	RA-L 2024	67.32	77.69	58.99	86.94	89.59	751
APPTracker+ [47]	IJCV 2024	75.19	75.55	75.36	86.59	89.09	176
S3MOT [48]	Arxiv 2025	76.86	76.95	77.41	87.87	86.93	543
C-TWIX [49]	Pattern Recognition 2025	77.58	76.97	78.84	86.95	89.68	381
SG-LKF [50]	Arxiv 2025	79.59	77.27	82.53	87.09	90.55	160
CollabMOT [51]	Access 2024	75.26	75.46	75.74	86.44	89.08	227
mmMOT [52]	ICCV 2019	62.05	72.29	54.02	86.58	83.23	733
PC3T [53]	TITS 2021	77.80	74.57	81.59	86.07	88.81	225
3DMLA [54]	Remote Sensing 2023	75.65	71.92	80.02	86.62	85.03	39
Ours	-	80.05	77.85	83.00	87.53	90.32	174

reports the quantitative comparison between the proposed method and a range of state-of-the-art multi-object tracking approaches on the KITTI tracking benchmark. Early methods such as mmMOT exhibit relatively low HOTA and association accuracy, indicating limited robustness in complex traffic scenarios. With the introduction of stronger motion modeling and sensor fusion strategies, approaches like EagerMOT, PC3T, and Mono-3D-KF achieve noticeable improvements in both detection and association performance. In particular, PC3T and MSA-MOT demonstrate competitive AssA scores, reflecting the effectiveness of enhanced data association mechanisms.

More recent methods further explore multi-modal collaboration and semantic cues. For example, DeepFusion-MOT and CollabMOT improve detection-related metrics but still suffer from suboptimal association consistency. C-TWIX achieves strong MOTA performance by leveraging temporal interaction modeling, while YONTD-MOT obtains competitive association accuracy and low identity switches, benefiting from its stable tracking strategy.

Among all compared approaches, the proposed method achieves the best overall HOTA score of 80.05%, outperforming previous state-of-the-art methods by a clear margin. Notably, our method also attains the highest DetA (77.85%) and MOTA (90.32%), demonstrating superior detection quality and overall tracking accuracy. These gains indicate that the introduced semantic-aware modeling effectively enhances both object localization reliability and long-term trajectory maintenance.

Compared with recent state-of-the-art methods, the proposed AttriMOT achieves the best overall HOTA score of 80.05, outperforming both S3MOT and SG-LKF, which demonstrates a better balance between detection accuracy and association quality. Although SG-LKF achieves slightly higher AssA performance due to its motion-driven learnable Kalman filtering strategy, our method obtains superior DetA performance and more robust overall tracking quality through semantic-aware multimodal association and adaptive confidence fusion.

Our framework is built upon TG3MOT, which already provides strong association performance, as evidenced by its high AssA and low IDSW. By incorporating explicit semantic features, our method further strengthens global semantic consistency, leading to improved HOTA and MOTA. However, it is observed that the number of identity switches increases compared with TG3MOT. This phenomenon can be attributed to the sensitivity of explicit appearance and motion features to scene variations. Changes in illumination, object color distribution, and target velocity across different sequences may introduce feature inconsistency between adjacent frames, which can occasionally disrupt identity association.

Despite this trade-off, the proposed method achieves a more favorable balance between detection accuracy and association quality, as reflected by the consistent improvements in holistic metrics such as HOTA. The results suggest that explicit semantic representation plays a critical role in enhancing overall tracking performance, while further improving feature robustness under dynamic scene changes remains an important direction for future work.

4.4.2. Ablation Experiment

Table 2. Ablation study of different modules on the Kitti training set.

AFMA	SACS	ASCF	HOTA (%) ↑	DetA (%) ↑	AssA (%) ↑	MOTA (%) ↑	MOTP (%) ↑	IDSW ↓	IDF1 (%) ↑	$\Delta$FPS ↓
-	-	-	77.785	73.846	82.083	82.354	88.514	8	90.914	0.0
✓	-	-	78.626	74.293	83.340	82.057	89.652	5	90.347	0.4
-	✓	-	78.214	74.018	82.947	82.481	89.103	6	90.562	0.3
-	-	✓	78.037	73.925	82.811	82.436	88.947	7	90.481	0.2
✓	✓	-	79.068	74.748	83.772	82.700	89.485	4	90.827	0.8
-	✓	✓	78.843	74.516	83.514	82.618	89.337	5	90.744	0.7
✓	-	✓	78.957	74.603	83.635	82.674	89.421	5	90.793	0.8
✓	✓	✓	79.257	75.067	83.817	83.138	89.486	5	90.966	1.1

reports the ablation results of the proposed modules built upon TG3MOT as the baseline tracker. By progressively introducing AFMA, SACS, and ASCF into the baseline framework, we analyze both the individual effectiveness and the complementary contributions of each component, while also evaluating the additional runtime overhead in terms of $\Delta$FPS.

The baseline TG3MOT already achieves strong performance in HOTA, AssA, and IDF1, demonstrating robust association capability. However, there remains room for improvement in detection quality, semantic discrimination, and overall tracking consistency.

After introducing the AFMA module alone, HOTA increases from 77.785% to 78.626%, while DetA and AssA are improved to 74.293% and 83.340%, respectively. In addition, IDSW decreases significantly from 8 to 5, indicating that attribute-aware multimodal modeling effectively improves instance discriminability and stabilizes target association, especially under appearance ambiguity and occlusion scenarios. Meanwhile, the additional runtime overhead is only 0.4 FPS, demonstrating the lightweight nature of the proposed module.

Similarly, incorporating SACS alone also leads to consistent gains over the baseline, with HOTA increasing to 78.214% and MOTA improving to 82.481%. These results demonstrate that the proposed semantic anchoring and competition suppression strategy can effectively alleviate semantic dominance among same-category objects, thereby preserving fine-grained attribute information beneficial for instance-level discrimination. The corresponding overhead is limited to only 0.3 FPS. When ASCF is individually introduced, the tracker also achieves noticeable improvements compared with the baseline while introducing only 0.2 FPS additional overhead, verifying the effectiveness and efficiency of adaptive cross-modal confidence fusion in handling dynamic modality reliability variations.

Further combining multiple modules yields additional performance improvements. In particular, integrating AFMA and SACS achieves the lowest IDSW value of 4, demonstrating strong identity preservation capability and enhanced temporal association robustness. Despite the improved performance, the additional computational overhead remains below 1 FPS.

Finally, incorporating AFMA, SACS, and ASCF simultaneously achieves the best overall performance. HOTA further improves to 79.257%, while DetA, AssA, and MOTA reach 75.067%, 83.817%, and 83.138%, respectively. In addition, the proposed framework achieves the highest IDF1 score of 90.966%. Although IDSW slightly increases compared with the AFMA + SACS configuration, it remains significantly lower than the baseline, indicating that adaptive confidence fusion effectively balances multimodal reliability and improves global tracking stability.

Overall, the complete framework introduces only 1.1 FPS additional runtime overhead compared with the baseline while consistently improving tracking accuracy, association robustness, and trajectory continuity. These results demonstrate that the proposed modules achieve an effective balance between tracking performance and computational complexity.

To further evaluate the effectiveness of each proposed module on more challenging autonomous driving scenarios, additional ablation experiments were conducted on the nuScenes training set. The corresponding results are summarized in

Table 3. Ablation study of different modules on the nuScenes training set.

Base	AFMA	SACS	ASCF	AMOTA ↑	MOTA ↑	MOTP ↑	IDSW ↓	RECALL ↑
✓	-	-	-	0.742	0.610	0.274	386	0.767
✓	✓	-	-	0.746	0.607	0.282	350	0.769
✓	✓	✓	-	0.751	0.616	0.287	351	0.772
✓	✓	✓	✓	0.757	0.624	0.291	330	0.778

.

Table 3 presents the ablation study results of different modules on the nuScenes training set. The experimental results show that each proposed module contributes positively to the overall tracking performance.

After introducing the AFMA module, AMOTA improves from 0.742 to 0.746, while IDSW decreases significantly from 386 to 350, demonstrating that AFMA can effectively enhance feature representation capability and improve tracking stability. Although MOTA shows a slight decrease, the improvements in MOTP and IDSW indicate better localization quality and association robustness. With the addition of the SACS module, the model further improves AMOTA, MOTA, and RECALL to 0.751, 0.616, and 0.772, respectively, demonstrating that SACS can effectively strengthen temporal association capability and improve target recovery performance in complex scenes.

After integrating the ASCF module, the complete framework achieves the best performance across almost all metrics, reaching 0.757 AMOTA, 0.624 MOTA, 0.291 MOTP, and 0.778 RECALL, while reducing IDSW to 330. Compared with the baseline, the proposed framework achieves consistent improvements in tracking accuracy, association robustness, and trajectory continuity, demonstrating the effectiveness and complementarity of the proposed modules.

Overall, the ablation results clearly show that each proposed module contributes positively, and their combination yields the best performance, validating the effectiveness of semantic anchoring, attribute-level multimodal alignment, and adaptive confidence fusion in AttriMOT.

4.4.3. Qualitative Results and Visualization Analysis

Figure 6 presents qualitative comparisons between the baseline method and our proposed AttriMOT under challenging occlusion scenarios. Both methods successfully track the vehicle with ID 15 before frame 94. However, when two vehicles undergo a cross-occlusion at frame 101, the baseline tracker fails to maintain the identity and loses ID 15, whereas AttriMOT consistently preserves the correct identity and continues tracking the target. This demonstrates the robustness of our attribute-aware representation in handling severe inter-object interactions.

Figure 7 illustrates another challenging case involving partial and static occlusion. At frame 78, the vehicle with ID 12 becomes temporarily occluded by a roadside pole. Due to the abrupt disappearance and appearance changes, the baseline method fails to re-associate the object and permanently loses the track at frame 84. In contrast, AttriMOT successfully recovers and maintains the trajectory of ID 12 after the occlusion. This can be attributed to our semantic-aware attribute alignment and adaptive confidence fusion, which jointly enhance identity continuity under short-term occlusions.

Overall, these qualitative results confirm that AttriMOT is more effective at preserving object identities in complex scenes with cross-occlusion and partial obstruction, highlighting its superior robustness and instance-level discrimination capability.

Figure 8 presents the visualization comparisons between the settings without debiasing and with debiasing under the same scene and target conditions. It should be noted that the proposed enhancement strategy is not a conventional backbone-level feature fusion operation, but a hierarchical semantic enhancement mechanism involving semantic debiasing, attribute-level multimodal alignment, and adaptive confidence fusion.

As shown in the heatmap visualizations, without semantic debiasing, several non-target regions still produce strong activations even though no red vehicle exists in the scene. This phenomenon indicates that the model is heavily influenced by category-level common semantics and tends to incorrectly activate semantically similar background structures or irrelevant object regions. Consequently, the learned representations are dominated by coarse category information rather than fine-grained attribute cues.

After introducing the proposed enhancement strategy, especially the semantic debiasing mechanism, these non-target high-response regions are significantly suppressed. The model gradually concentrates its responses on regions that are more consistent with the target attributes, while irrelevant activations are effectively reduced. In addition, the target boundaries and local discriminative structures become more concentrated and continuous, demonstrating that the proposed framework can better preserve fine-grained attribute information and improve semantic discrimination capability.

5. Discussion

5.1. Analysis of Tracking Performance with the Same 3D Detector

Table 4. Tracking performance on KITTI training sequences using the same 3D detector.

Method	3D Detector	2D Detector	HOTA (%) ↑	AssA (%) ↑	MOTA (%) ↑	IDSW ↓	IDF1 (%) ↑
YONTD-MOT	VoxelRCNN	FasterRCNN	77.52	82.14	82.11	10	90.75
	PVRCNN	MaskRCNN	76.49	81.12	80.35	8	88.85
	VoxelRCNN	FasterRCNN	75.38	82.33	75.47	14	87.91
	PVRCNN	MaskRCNN	75.05	81.78	75.14	13	87.68
TG3MOT	VoxelRCNN	RegionClip	77.78	82.08	82.35	8	90.91
	PVRCNN	RegionClip	75.93	81.27	78.61	4	89.01
Ours	VoxelRCNN	RegionClip	79.26	83.82	83.14	5	90.97
	PVRCNN	RegionClip	76.71	81.28	79.83	7	88.87

presents a fair comparison of tracking performance on the KITTI training set using identical 3D and 2D detector configurations. The results compare YONTD-MOT, TG3MOT, and our method under VoxelRCNN and PVRCNN detectors.

YONTD-MOT shows noticeable performance variations across different detector combinations, with fluctuating HOTA and MOTA scores and increased ID switches, indicating limited robustness to detector quality and appearance variations.

TG3MOT benefits from RegionCLIP-based semantic cues and consistently outperforms YONTD-MOT under the same 3D detectors. In particular, it achieves improved HOTA and association accuracy, demonstrating the effectiveness of language-aware representations. However, its reliance on coarse category-level semantics limits further gains in instance-level discrimination.

Building on TG3MOT, our method achieves the best overall performance under both detector settings. With VoxelRCNN, it improves HOTA from 77.78% to 79.26%, AssA from 82.08% to 83.82%, and MOTA from 82.35% to 83.14%, while maintaining comparable IDF1 and ID switch levels. Similar improvements are observed with PVRCNN. These results indicate that explicit attribute-level alignment and adaptive multimodal fusion enhance tracking robustness and association quality beyond category-level semantics.

Overall, the consistent gains across detector settings confirm that the improvements stem from the proposed semantic-aware tracking framework rather than detector-specific advantages.

5.2. Discussion on the Semantic-Aware Trajectory Selector

The semantic-aware trajectory selector is designed to retrieve and track a target based on a textual query, leveraging the fine-grained attribute-level alignment established in previous stages. A key component of this mechanism is the category debiasing performed in Stage I, which removes the dominant category-level components from visual features to preserve discriminative attribute information.

Figure 9 illustrates the effect of this debiasing. When the prompt specifies a “red car,” without debiasing, the tracker incorrectly associates a nearby blue car with the query, resulting in a tracking error. After applying category debiasing, the tracker successfully focuses on the red car, ignoring other visually similar but category-matched objects. This demonstrates that debiasing is critical for precise, attribute-sensitive, text-guided tracking. By combining this capability with adaptive confidence fusion and attribute-level multimodal alignment, the trajectory selector ensures robust retrieval and consistent tracking of the specified target even under occlusions and challenging scene dynamics.

5.3. Discussion on the Different Object Categories

To further evaluate the generalization capability of the proposed method across different object categories, additional experiments were conducted on the Pedestrian category of the KITTI dataset, and the corresponding results are presented in

Table 5. Comparison between the baseline and the proposed method on pedestrian categories.

Category	Method	HOTA ↑	DetA ↑	MOTA ↑
Pedestrian	Baseline	52.12	53.09	68.33
	Ours	53.77	54.23	69.85

.

Compared with vehicle targets, pedestrian targets usually exhibit smaller object scales, more irregular motion patterns, and frequent occlusions, making pedestrian tracking substantially more challenging in multi-object tracking tasks. As shown in Table 5, the proposed method improves HOTA from 52.12 to 53.77, DetA from 53.09 to 54.23, and MOTA from 68.33 to 69.85, respectively. These improvements demonstrate that the proposed framework can effectively enhance both detection accuracy and temporal association performance in complex pedestrian tracking scenarios.

5.4. Sensitivity Analysis of Hyperparameters

We analyze the sensitivity of the ASCF to two key hyperparameters in Stage III: the temperature parameter $\tau$ and the temporal smoothing weight start $w_{start}$ (Figure 10).

As shown in Figure 10a, HOTA first increases and then decreases as $\tau$ grows, with the best result at $\tau =0.3$ (HOTA = 79.257). A too-small $\tau$ makes modality weights overly sharp and more sensitive to short-term noise, while a too-large $\tau$ makes fusion close to uniform averaging and weakens adaptive modality selection.

As shown in Figure 10b, varying $w_{start}$ causes only limited HOTA fluctuation, indicating good robustness to temporal smoothing strength. Smaller $w_{start}$ emphasizes recent observations more strongly (higher responsiveness but more jitter), whereas larger $w_{start}$ yields smoother but less responsive confidence evolution. The best trade-off is achieved at $w_{start}=0.5$.

Overall, these results show that ASCF is stable within a practical hyperparameter range, and that the default setting $(\tau =0.3,w_{start}=0.5)$ provides a favorable balance between adaptivity and stability.

6. Conclusions

In this work, we proposed AttriMOT, a semantic-aware multimodal 3D multi-object tracking framework that explicitly aligns object-level attributes across 3D geometry, 2D appearance, and textual semantics. By introducing category semantic anchoring, attribute-level cross-modal alignment, adaptive confidence fusion, and a semantic-aware trajectory selector, AttriMOT achieves fine-grained, interpretable, and text-guided tracking, overcoming the limitations of conventional category-based or unstable multimodal methods. Extensive experiments on challenging 3D MOT benchmarks demonstrate that AttriMOT outperforms existing approaches in tracking accuracy, robustness, and semantic controllability, highlighting the effectiveness of attribute-level multimodal integration for precise and flexible 3D multi-object tracking.

In future work, we plan to further explore more robust semantic reasoning and adaptive multimodal association under complex dynamic environments, especially in scenarios involving severe occlusion, sparse point clouds, and long-range targets. In addition, we aim to investigate lightweight and real-time semantic-aware tracking frameworks for large-scale deployment, as well as richer language-guided spatial understanding and interaction mechanisms to further enhance controllable and intelligent 3D tracking capabilities.