Text-Guided Spatio-Temporal 2D and 3D Data Fusion for Multi-Object Tracking with RegionCLIP

Abstract

3D Multi-Object Tracking (3D MOT) is a critical task in autonomous systems, where accurate and robust tracking of multiple objects in dynamic environments is essential. Traditional approaches primarily rely on visual or geometric features, often neglecting the rich semantic information available in textual modalities. In this paper, we propose Text-Guided 3D Multi-Object Tracking (TG3MOT), a novel framework that incorporates Vision-Language Models (VLMs) into the YONTD architecture to improve 3D MOT performance. Our framework leverages RegionCLIP, a multimodal open-vocabulary detector, to achieve fine-grained alignment between image regions and textual concepts, enabling the incorporation of semantic information into the tracking process. To address challenges such as occlusion, blurring, and ambiguous object appearances, we introduce the Target Semantic Matching Module (TSM), which quantifies the uncertainty of semantic alignment and filters out unreliable regions. Additionally, we propose the 3D Feature Exponential Moving Average Module (3D F-EMA) to incorporate temporal information, improving robustness in noisy or occluded scenarios. Furthermore, the Gaussian Confidence Fusion Module (GCF) is introduced to weight historical trajectory confidences based on temporal proximity, enhancing the accuracy of trajectory management. We evaluate our framework on the KITTI dataset and compare it with the YONTD baseline. Extensive experiments demonstrate that although the overall HOTA gain of TG3MOT is modest (+0.64%), our method achieves substantial improvements in association accuracy (+0.83%) and significantly reduces ID switches (−16.7%). These improvements are particularly valuable in real-world autonomous driving scenarios, where maintaining consistent trajectories under occlusion and ambiguous appearances is crucial for downstream tasks such as trajectory prediction and motion planning. The code will be made publicly available.

1. Introduction

Multi-Object Tracking (MOT) is a core task in computer vision, widely used in autonomous driving, robotics, and surveillance. Its goal is to assign consistent identities to multiple objects across frames. Traditional MOT methods are commonly grouped into Tracking-by-Detection (TBD) [1,2,3,4,5,6,7], Joint Detection and Embedding (JDE) [8,9], and Joint Detection and Tracking (JDT) [10,11,12]. TBD separates detection and association, often suffering from latency and association complexity. JDE integrates detection and tracking for greater efficiency and joint optimization. JDT further improves by explicitly modeling temporal relationships, enhancing performance in challenging scenarios with occlusions or fast motion.

In recent years, deep learning-based trackers have significantly advanced MOT performance by introducing stronger detectors, re-identification modules, and temporal modeling techniques [6,13]. Despite this progress, MOT in unconstrained environments still faces difficulties such as long-term occlusions, crowded scenes, and drastic illumination changes [8]. These limitations are especially critical in safety-related applications like autonomous driving.

Most existing 3D MOT methods depend on single-modal data, such as RGB images or LiDAR, which may lack robustness. Multimodal fusion combines complementary cues from different sensors (e.g., LiDAR, camera, radar), showing superior results. However, current multimodal methods largely follow the TBD pipeline, limiting their potential due to complex association steps.

3D MOT introduces added challenges, such as sparse data, occlusions, and scale variations [14]. Recent methods [15,16] propose end-to-end frameworks to integrate detection and tracking, reducing reliance on hand-crafted associations and achieving state-of-the-art results without additional training. Still, many approaches over-rely on visual cues, making them fragile in ambiguous appearance conditions.

To address these limitations, several studies have explored multimodal and cross-modal tracking. For instance, Camera-LiDAR fusion has been used to improve spatial localization and robustness in autonomous driving [17,18]. Moreover, radar sensing has shown strong robustness under adverse environmental conditions such as rain, fog, and snow. The K-Radar dataset demonstrates that 4D radar (with elevation information) outperforms LiDAR-based methods for 3D object detection under challenging weather [19]. RadarNet further integrates radar and LiDAR using early and late fusion strategies to improve detection and velocity estimation, especially for moving objects in degraded visual conditions [20]. These advances highlight radar’s value as a complementary sensor in adverse weather environments. However, most existing fusion frameworks focus primarily on low-level geometric and appearance cues rather than high-level semantic understanding, which limits their ability to resolve ambiguous cases where appearance alone is insufficient.

Meanwhile, Vision-Language Models (VLMs) such as CLIP [21] and ALIGN [22] have shown strong cross-modal reasoning in tasks such as image-text retrieval [23], VQA [24], and zero-shot learning [25]. Recent works such as BLIP [26] and Flamingo [27] further demonstrate that large-scale vision-language pretraining enables richer semantic understanding and generalization across domains. These advances suggest that language supervision could serve as an additional modality to disambiguate visually similar objects and guide association in 3D MOT. However, their application in MOT remains underexplored, as illustrated in Figure 1.

Building on the success of [28], we propose TG3MOT, a novel framework that integrates Vision-Language Models (VLMs) into the 3D MOT pipeline. Specifically, we leverage CLIP to extract text-modal features, enabling the model to incorporate semantic cues from natural language descriptions. This enhances its ability to handle challenging scenarios where traditional visual or geometric cues fall short—such as severe occlusions, appearance variations, or identity ambiguity. To further improve robustness, we introduce three key modules: Target Semantic Matching, Gaussian Confidence Fusion, and 3D Feature EMA, which synergize multimodal fusion with text-guided understanding. Together, these innovations push the boundaries of multi-object tracking performance in complex real-world environments.

Our contributions include the following:

• Extending the YONTD [29] framework with a semantic-aware RegionCLIP detector, achieving strong zero-shot detection without extra training;
• Designing a Target Semantic Matching Module (TSM) to filter false positives based on semantic consistency;
• Proposing a 3D Feature EMA module to improve regression accuracy and reduce false detections;
• Introducing Gaussian Confidence Fusion (GCF) to stabilize trajectory confidence using historical weights;
• Achieving superior results on KITTI [30], with our code publicly released to foster future research.

2. Related Work

2.1. Traditional MOT Methods

Multi-Object Tracking (MOT) methods mainly fall into Tracking-by-Detection (TBD) and Joint Detection and Embedding (JDE) paradigms. TBD separates detection and tracking: objects are detected per frame and then associated using algorithms such as the Hungarian method [31] or graph-based optimization [32]. Although effective, TBD is heavily dependent on the quality of the detection and the complex association, which can be error-prone.

JDE integrates detection and feature embedding into a single model, streamlining the pipeline and improving speed. However, it often struggles with balancing detection accuracy and discriminative feature learning, especially under occlusion. Recent Joint Detection and Tracking (JDT) methods further unify both tasks in an end-to-end framework, eliminating explicit association steps. While promising, they usually require extensive labeled data and complex training.

2.2. 3D MOT Methods

3D MOT extends tracking into spatial depth, critical for autonomous driving and robotics [33]. It introduces challenges such as depth ambiguity and sparse data. LiDAR-based methods like AB3DMOT [15] and PointRCNN [16] exploit point clouds for detection and tracking, while multimodal fusion approaches like MMF [34] and CenterPoint [12] enhance robustness by integrating LiDAR and camera inputs. YONTD [29] offers an end-to-end solution, achieving state-of-the-art performance without extra training.

Camera-based 3D MOT methods, such as Mono3D [35] and Pseudo-LiDAR [36], reconstruct 3D cues from monocular images. Graph-based models like GNN3DMOT [37] capture spatial relationships via graph neural networks, improving performance in dense scenes. Overall, integrating sensor modalities and unifying detection and tracking tasks is key to advancing 3D MOT.

2.3. Vision-Language Models (VLMs) in MOT

Vision-Language Models (VLMs), such as CLIP [21], have shown promise in aligning visual and textual modalities. RegionCLIP [28] extends this to region-level tasks, highlighting potential for MOT applications like semantic association and occlusion handling.

Recent efforts like LaMOT [38], CiteTracker [39], and Adaptive Text Feature Updating [40] leverage textual cues for improved object association and tracking consistency. However, VLM integration in MOT remains limited, with most works relying on purely visual or geometric features.

Our method addresses this gap by incorporating RegionCLIP into the MOT framework, enabling zero-shot detection and semantic reasoning. Built upon YONTD, our end-to-end system introduces the Target Semantic Matching (TSM), 3D Feature-EMA (3D F-EMA), and Gaussian Confidence Fusion (GCF) modules to enhance tracking in complex scenes. Full details are presented in the following section.

3. Proposed Method

In this section, we present the proposed Text-Guided 3D Multi-Object Tracking framework, a novel approach that integrates Vision-Language Models (VLMs) into the YONTD architecture to address key challenges in 3D MOT. Our framework, illustrated in Figure 2, consists of several innovative components designed to enhance detection, tracking, and trajectory management. The overall architecture maintains an end-to-end structure, enabling seamless integration of semantic information without requiring additional training. The framework begins with LiDAR point cloud data [41] as input, which is processed by a 3D detector (embed the 3D Feature EMA module) to generate the current frame’s 3D trajectory states [42,43]. These 3D trajectories are then projected onto the 2D image plane, where the 2D trajectories and the current image frame are fed into a multimodal detector to obtain trajectory scores and semantic features [44]. The Target Semantic Matching Module is employed to filter out false positives by analyzing the semantic information of candidate trajectories. Subsequently, the Gaussian Confidence Fusion Module fuses the 2D and 3D historical trajectories, assigning weights based on a Gaussian distribution to compute the final trajectory score, which guides the Non-Maximum Suppression (NMS) process to determine the final trajectories and associations for the current frame. In the Trajectory Management Module, we adopt the strategy from DeepFusionMOT [45]. Each of these components is detailed in the following subsections.

3.1. VLM and RegionCLIP

Most existing Multi-Object Tracking (MOT) methodologies predominantly rely on visual or geometric features, often overlooking the rich semantic information provided by textual modalities. A critical challenge addressed in our framework is the effective integration of semantic information into the classical 3D MOT framework. Our initial approach involved leveraging the CLIP [21] model on 2D images in conjunction with traditional detectors such as Faster R-CNN [46]. However, directly applying CLIP within the MOT framework resulted in suboptimal performance due to a significant domain shift: CLIP was trained to match an entire image to a textual description, lacking the capability to establish fine-grained alignment between specific image regions and corresponding text spans. Additionally, traditional detectors, typically trained on COCO datasets, exhibit a performance gap when applied to tracking scenarios, further exacerbating the issue. To address these limitations, we adopt RegionCLIP [28], a multimodal open-vocabulary detector that enables fine-grained alignment between image regions and textual concepts. Building upon the YONTD [29] framework, we integrate RegionCLIP as the 2D detector. For each object to be tracked, a set of textual descriptions is provided as $Text=[tex{t}_1,tex{t}_2,\dots ,tex{t}_N]$. These descriptions are encoded using RegionCLIP’s text encoder to obtain corresponding text features $F_{text}={[f{t}_1,f{t}_2,\dots ,f{t}_N]}^T$. For the current frame t, let $D_t^{3D}$ denote the 3D detection results, $T_t^{3D}$ represent the 3D trajectories, and $S_t^{3D}$ indicate the regression confidences of the 3D trajectories. $T_t^{3D}$ are projected onto the 2D image plane to generate region proposals, which are then fed into RegionCLIP to extract region image features $F_t^{region}={[f{r}_1,f{r}_2,\dots ,f{r}_M]}^T$. The matching score is computed using the following formula:

$$S_{\mathrm{match}}\in {\mathbb{R}}^{M\times N}={\displaystyle \frac{F_t^{\mathrm{region}}·F_{text}^T}{\parallel F_t^{\mathrm{region}}\parallel ·\parallel F_{text}^T\parallel }}$$

(1)

This matching score is used to derive the classification output of the 2D detector. Furthermore, the maximum value of $S_{match}$ corresponding to the predicted class is selected, and the 2D trajectory regression confidence $S_t^{2D}$ is computed as

$$S_t^{2D}=\underset{\dim =1}{max}{S}_{\mathrm{match}}·S_t^{3D}$$

(2)

This approach not only bridges the domain gap between detection and tracking scenarios but also enhances the framework’s ability to leverage semantic information for more accurate and robust object tracking. By integrating RegionCLIP, our framework achieves fine-grained alignment between image regions and textual concepts, significantly improving performance in complex tracking environments.

3.2. Target Semantic Matching Module

Although RegionCLIP offers robust detection capabilities in open-vocabulary scenarios, it faces significant challenges in complex tracking environments, particularly due to issues such as blurring, occlusion, and ambiguous object appearances. Even with the integration of semantic information, these challenges remain substantial.

To address these limitations, we focus on enhancing the alignment between image region features and textual features. Specifically, the matching score matrix $S_{match}$ is computed to identify the most semantically relevant textual description for each image region. Through extensive experimentation, we observe that when the objects within a region are clear and easily identifiable, the matching score distribution across textual descriptions exhibits significant discriminability. Conversely, in cases of ambiguity or occlusion, the distribution becomes notably smoother, as demonstrated in Figure 3. Building on these insights, we propose a Target Semantic Matching Module (TSM), which quantifies the uncertainty of each region’s semantic alignment by calculating the variance of its matching score distribution, denoted as $V=\{v_1,v_2,\dots ,v_M\}$. To further incorporate the influence of 3D detection confidence, we define the TSM score for each region as follows:

$$S_{\mathrm{TSM}}={\left[v_i^{1-S_i^{3D}}\right]}_{i=1}^M$$

(3)

where $S_i^{3D}$ represents the 3D detection confidence for the i-th region. Regions with $S_{TSM}$ below a predefined threshold are considered noisy and are subsequently discarded. This approach effectively filters out unreliable regions, ensuring that only semantically consistent and high-confidence regions contribute to the tracking process.

3.3. 3D Feature Exponential Moving Average

Multi-Object Tracking (MOT) inherently operates on temporally sequential data, where frames are intrinsically interconnected over time. However, conventional 3D object detectors often process each frame independently, neglecting the valuable temporal correlations that exist across consecutive frames. This limitation becomes particularly pronounced in scenarios affected by sensor noise, occlusions, or dynamic environmental changes, where trajectories detected in frame $t-1$ may be lost in frame, leading to false negatives, as illustrated in Figure 4. To tackle this challenge, we propose a novel 3D Feature Exponential Moving Average (EMA) Module, which ingeniously incorporates temporal information into the 3D detection framework through historical feature fusion, thereby enhancing the robustness and continuity of object tracking.

The core idea of the 3D Feature EMA Module is to leverage the temporal consistency of object features across frames. Specifically, for each detected object, we maintain a dynamically updated feature representation that aggregates historical information from previous frames. Let $f_t^i$ denote the feature vector of the i-th object detected in frame t. The EMA-updated feature ${\tilde{f}}_t^i$ is computed as follows:

$${\tilde{f}}_t^i=\alpha ·f_t^i+(1-\alpha )·{\tilde{f}}_{t-1}^i$$

(4)

where $\alpha \in (0,1)$ is a smoothing factor that controls the contribution of the current frame’s feature relative to the historical features. This recursive formulation ensures that the feature representation of each object evolves smoothly over time, effectively mitigating the impact of transient detection failures caused by noise or occlusions.

3.4. Gaussian Confidence Fusion Module

In the original YONTD [29] framework, a trajectory confidence fusion mechanism is proposed to address the challenges posed by “strong objects” and “weak objects” in the tracking process. This module traditionally computes the regression confidence for the current frame t by averaging the trajectory scores from the previous k frames. However, this approach fails to account for the temporal weighting of different frames, which is crucial for accurate tracking.

We hypothesize that the state of a tracked object at frame t is more similar to its state at frame $t-1$ than at frame $t-2$, implying that temporal proximity correlates with state similarity. To incorporate this temporal dependency, a weighted regression confidence fusion method is proposed, leveraging a Gaussian distribution to assign weights to the historical trajectory confidences. Specifically, the confidence for the current frame t is computed as a weighted sum of the confidences from the previous k frames, where the weights are derived from a Gaussian function centered at the current frame. This approach ensures that frames closer in time to the current frame have a greater influence on the computed confidence. The mathematical formulation of this weighted fusion is as follows:

$${\widehat{S}}_t^n={\displaystyle \frac{{\sum }_{i=1}^k{\omega }_i·S_{t-i+1}^n}{{\sum }_{i=1}^k{\omega }_i}}$$

(5)

where ${\widehat{S}}_t^n$ is the fusion regression confidence of the n-th trajectory at frame t, $S_{t-i+1}^n$ represents the confidence of the n-th trajectory at frame $t-i+1$, and ${\omega }_i$ is the weight assigned to the i-th historical frame, calculated using a Gaussian distribution:

$${\omega }_i=exp\left(-{\displaystyle \frac{i^2}{2{\sigma }^2}}\right)$$

(6)

Here, $\sigma$ is the standard deviation of the Gaussian distribution, controlling the rate at which the weights decay with increasing temporal distance from the current frame. This method effectively captures the temporal dynamics of object states, leading to more accurate and robust tracking performance. The implementation of Gaussian Confidence Fusion ($k=9$) is illustrated in Figure 5.

4. Experiments

In this section, we present the experimental setup, results, ablation studies, and discussions to evaluate the effectiveness of the proposed framework. All experiments are conducted on the KITTI dataset [30], and the performance is compared against the baseline YONTD framework, as well as other state-of-the-art methods.

4.1. Experimental Setup

Hardware: All experiments were conducted on a workstation with an NVIDIA RTX 4090 GPU (24GB VRAM) (NVIDIA, Santa Clara, CA, USA), Intel Xeon Platinum 8480+ CPU, and 2TB RAM (Intel, Santa Clara, CA, USA). Although not optimized for real-time performance, this setup enables efficient batch processing of large-scale LiDAR and image data.

Dataset: The proposed framework was evaluated on the KITTI tracking benchmark, which includes synchronized 3D LiDAR scans, RGB images, and annotations for multiple object categories (e.g., cars, pedestrians, cyclists). It consists of 21 training and 29 testing sequences.

Implementation: For 3D detection, we employed PVRCNN [47] and VoxelRCNN [48]. RegionCLIP [28] was adopted to extract region-level image features and align them with textual prompts. The textual prompts were manually designed following the template style used in RegionCLIP, such as “a photo of a car”, “a photo of a pedestrian”, and “a photo of a truck”. These standardized prompts ensure consistency across different categories and were encoded using RegionCLIP’s text encoder to guide semantic alignment. The TSM threshold was set to 0.0085 and the 3D F-EMA decay factor to 0.9.

Tracking performance was primarily assessed using Higher-Order Tracking Accuracy (HOTA) [49], which integrates detection, association, and localization quality. Additional metrics were also reported to provide a comprehensive evaluation:

MOTA (Multiple-Object Tracking Accuracy): Measures overall tracking accuracy by combining false positives, false negatives, and identity switches.

MOTP (Multiple-Object Tracking Precision): Measures localization precision of correctly matched targets.

AssA (Association Accuracy): Evaluates the correctness of identity association between frames.

LocA (Localization Accuracy): Measures the accuracy of object localization, independent of identity assignment.

DetA (Detection Accuracy): Reflects the quality of object detection, ignoring association errors.

IDSW (ID Switches): Counts the number of times a tracked identity changes, indicating identity fragmentation.

4.2. Experimental Results

Table 1. Performance comparison with SOTA 3D MOT methods on the KITTI test set. The optimal results are highlighted in bold.

Method	Published	HOTA (%) ↑	DetA(%) ↑	AssA (%) ↑	LocA(%) ↑	MOTA(%) ↑	MOTP (%) ↑	IDSW ↓
mmMOT [50]	ICCV 2019	62.05	72.29	54.02	86.58	83.23	85.03	733
EagerMOT [51]	ICRA 2021	74.39	75.27	74.16	87.17	87.82	85.69	239
PC3T [52]	TITS 2021	77.80	74.57	81.59	86.07	88.81	84.26	225
Mono-3D-KF [53]	FUSION 2021	75.47	74.10	77.63	85.48	88.48	83.70	162
MSA-MOT [54]	Sensors 2022	78.52	75.19	82.56	87.00	88.01	85.45	91
DeepFusion-MOT [45]	RA-L 2022	75.46	71.54	80.05	86.70	84.63	85.02	84
PolarMOT [55]	ECCV 2022	75.16	73.94	76.95	87.12	85.08	85.63	462
3DMLA [56]	Remote Sensing 2023	75.65	71.92	80.02	86.62	85.03	84.93	39
CollabMOT [57]	Access 2024	75.26	75.46	75.74	86.44	89.08	84.97	227
PNAS-MOT [58]	RA-L 2024	67.32	77.69	58.99	86.94	89.59	85.44	751
APPTracker+[59]	IJCV 2024	75.19	75.55	75.36	86.59	89.09	85.03	176
C-TWIX [60]	Pattern Recognition 2025	77.58	76.97	78.84	86.95	89.68	85.50	381
YONTD-MOT [29]	—	78.08	74.16	82.86	88.23	85.09	86.98	42
TG3MOT (Ours)	—	78.72	74.59	83.69	87.64	86.15	86.26	35

reports our results on the KITTI test set, compared with several state-of-the-art (SOTA) methods, including mmMOT, DeepFusion-MOT, PolarMOT, C-TWIX, YONTD-MOT, and others. Our method, TG3MOT, achieved the highest HOTA (78.72%), outperforming MSA-MOT (78.52%) and C-TWIX (77.58%). TG3MOT also achieved superior association accuracy(AssA: 83.69%), exceeding PC3T (81.59%) and MSA-MOT (82.56%). Although PNAS-MOT reported higher DetA (77.69%), it suffered from 751 ID switches, in contrast to only 35 for TG3MOT, demonstrating our framework’s consistency. While C-TWIX and PNAS-MOT achieved slightly higher MOTA, their elevated IDSW indicates inferior identity preservation.

A head-to-head comparison with YONTD-MOT (

Table 2. Tracking performance on KITTI training sequence using the same 3D detector for TG3MOT and YONTD.

	3D Detector	2D Detector	HOTA (%)↑	AssA(%)↑	MOTA(%)↑	IDSW↓	IDF1↑
YONTD-MOT	VoxelRCNN [48]	FasterRCNN [46]	77.52	82.14	82.11	10	90.75
		MaskRCNN [61]	76.49	81.12	80.35	8	88.85
YONTD-MOT	PVRCNN [47]	FasterRCNN	75.38	82.33	75.47	14	87.91
		MaskRCNN	75.05	81.78	75.14	13	87.68
TG3MOT (Ours)	VoxelRCNN	RegionClip	77.78	82.08	82.35	8	90.91
	PVRCNN	RegionClip	75.93	81.27	78.61	4	89.01

), using the same 3D detector, shows TG3MOT consistently outperforms across all key metrics. Figure 6 visualizes bird’s-eye-view trajectories, highlighting TG3MOT’s superior temporal consistency and reduced tracking errors. These improvements are attributed to the effective fusion of 2D–3D features and the incorporation of semantic information.

4.3. Ablation Study

We performed ablation studies to evaluate each component’s contribution (

Table 3. Impact of different modules on tracking performance metrics. (Results are reported with three-decimal precision to highlight marginal contributions.) The symbols ↑/↓ indicate that higher/lower values are better.

Module	Components			Performance Metrics
	TSM	3D F-EMA	GCF	HOTA (%) ↑	DetA(%) ↑	AssA(%) ↑	MOTA(%) ↑	MOTP(%) ↑	IDSW ↓	IDF1(%) ↑
Base	×	×	×	76.432	71.852	81.453	79.382	88.402	5	88.423
TSM	✓	×	×	77.741	73.412	82.481	81.732	88.521	2	90.821
3D F-EMA	×	✓	×	76.441	71.863	81.462	79.403	88.413	5	88.434
GCF	×	×	✓	76.592	72.271	80.041	80.842	88.423	5	88.743
TSM+GCF	✓	×	✓	77.782	73.842	82.351	82.351	88.512	8	90.902
Full	✓	✓	✓	77.785	73.846	82.083	82.354	88.514	8	90.914

):

TSM: Removing TSM reduced HOTA and AssA and increased IDSW, indicating its role in filtering noisy regions and improving association. Specifically, the semantic filtering of TSM suppresses visually similar but semantically irrelevant proposals, which reduces mismatches across frames. This directly improves association accuracy (AssA) and identity consistency (lower IDSW), while also leading to higher HOTA.

3D F-EMA: Excluding this module decreased DetA and MOTA, suggesting temporal feature fusion improves stability and continuity. Without 3D F-EMA, the detector relies only on per-frame features, making it more vulnerable to occlusions and motion blur. By integrating historical embeddings through an exponential moving average, temporal consistency is reinforced, thus enhancing detection accuracy (DetA) and overall multi-object tracking accuracy (MOTA). To further clarify its contribution, we report results with three-decimal precision. Compared to the configuration with only TSM+GCF, adding 3D F-EMA yields consistent improvements across all metrics: HOTA (+0.003), DetA (+0.004), MOTA (+0.003), and IDF1 (+0.012). Although these gains appear small, they highlight the stabilizing role of temporal information and contribute to more reliable identity preservation in long-term tracking.

GCF: Disabling GCF led to lower HOTA and DetA, confirming the benefit of adaptive confidence weighting. Gaussian Confidence Fusion models the distribution of detection confidence with a Gaussian kernel and incorporates the temporal weighting of different frames. By assigning larger weights to frames closer to the current time step, GCF adaptively fuses historical trajectory confidences, which enhances robustness under ambiguous detections and reduces the impact of noisy predictions. This leads to improved detection accuracy (DetA) and overall tracking performance (HOTA).

All Modules Combined: Full integration yielded the best results across all metrics, demonstrating the complementary effects of semantic filtering, temporal fusion, and confidence refinement. In particular, TSM ensures the semantic correctness of proposals, 3D F-EMA stabilizes temporal information, and GCF adaptively calibrates confidence. Their combination provides balanced improvements across detection (DetA), association (AssA), and identity-related metrics (IDF1, IDSW).

4.4. Discussion

Text-Guided Features: The integration of semantic cues enhances tracking robustness, especially in dense or occluded scenarios, by improving inter-object discrimination and reducing ID switches.

Practical Implications: While the overall HOTA gain of TG3MOT compared to the YONTD baseline is modest (+0.64%), the framework delivers clear advantages in terms of association accuracy (+0.83%) and ID switch reduction (-16.7%). These improvements translate into more consistent and reliable object trajectories, which are critical for downstream tasks such as trajectory prediction, motion planning, and behavior analysis in autonomous driving. In real-world deployments, reducing ID switches mitigates trajectory fragmentation, thereby providing more stable inputs to decision-making modules. Moreover, the integration of multimodal semantics introduces a new paradigm for 3D MOT, demonstrating that even incremental performance gains in benchmarks can yield disproportionately high benefits in safety-critical applications.

Module-Level Insights: Although the numerical gains from the 3D F-EMA module appear modest when combined with TSM and GCF, results with higher-precision reporting confirm consistent improvements (HOTA +0.003, DetA +0.004, MOTA +0.003, IDF1 +0.012). These seemingly small increments reflect enhanced temporal stability and reduced identity fragmentation, which are crucial in long-term tracking scenarios. Importantly, stable identity preservation ensures smoother trajectories that directly benefit downstream tasks such as prediction and planning, underscoring the practical value of even marginal module-level contributions.

Limitations and Future Work: The framework’s computational demands limit real-time deployment. Future research may focus on efficiency optimization via model compression, lightweight language models, or hardware acceleration. Additionally, robustness in dynamic environments could be improved by incorporating temporal priors or multimodal inputs (e.g., radar, thermal).

Dataset Scope: Our evaluation is currently restricted to the KITTI dataset, which, while widely adopted, does not cover the full diversity of real-world driving scenarios. As a result, the generalizability of TG3MOT to other domains remains to be fully validated. Future work will extend experiments to broader benchmarks such as nuScenes, Waymo, and BDD100K to strengthen the robustness and applicability of our findings.

Despite the benefits of integrating Vision-Language Models (VLMs), several limitations remain regarding their use for guiding visual perception. A key concern is the robustness to unseen or noisy textual prompts, as models trained on specific datasets may not generalize well to novel or corrupted inputs, leading to performance degradation. This challenge is further intensified by exposure bias, where discrepancies between the training distribution and the inference scenario cause errors to compound during deployment. Recent work by Pozzi et al. [62] addresses this problem in the context of large language model distillation using an imitation learning approach, showing improved generalization under distributional shifts. Exploring similar strategies in multimodal tracking could enhance the reliability of prompt-based guidance, especially in dynamic or ambiguous scenarios.

5. Conclusions

In this paper, we propose TG3MOT, a novel Text-Guided 3D Multi-Object Tracking framework that integrates Vision-Language Models to enhance tracking performance. Our key innovation lies in the seamless fusion of semantic information with traditional 3D MOT pipelines, enabling more robust and accurate object tracking in complex environments. By leveraging RegionCLIP for fine-grained region-text alignment and introducing target semantic matching, temporal feature fusion, and confidence-based trajectory refinement, our method effectively mitigates challenges such as occlusions, appearance ambiguity, and false positives. Extensive experiments on the KITTI benchmark demonstrate that TG3MOT outperforms existing approaches, establishing a new state-of-the-art in Text-Guided 3D MOT.

While our current evaluation primarily compares TG3MOT with representative baselines such as YONTD and recent works, we acknowledge that a broader range of multimodal approaches could provide additional context. Future work will aim to extend comparisons to other multimodal methods, including sensor fusion and text-guided tracking frameworks, as well as to evaluate performance on diverse benchmarks beyond KITTI, in order to further validate the generalization and robustness of TG3MOT.

Although the overall HOTA improvement appears modest, TG3MOT achieves clear practical benefits by improving association accuracy and substantially reducing ID switches. These gains lead to more consistent and reliable trajectories, which are critical for downstream tasks such as trajectory prediction and motion planning in autonomous driving. Thus, TG3MOT not only advances performance metrics but also strengthens the real-world applicability of 3D MOT through multimodal semantic integration.

We hope this work inspires further research into multimodal learning for 3D object tracking, bridging the gap between vision and language in autonomous perception systems.