A Transformer-Based Multi-Task Learning Model for Vehicle Traffic Surveillance

Abstract

Vehicle traffic surveillance (VTS) systems are based on the automatic analysis of video sequences to detect, classify, and track vehicles in urban environments. The design of new VTS systems requires computationally efficient architectures with high performance in accuracy. Conventional approaches based on multi-stage pipelines have been successfully used during the last decade. However, these systems need to be improved to face the challenges of complex, high-mobility traffic environments. This article proposes an efficient system based on transformer architectures for VTS channels. The proposed analysis system is evaluated in scenarios with high vehicle density and occlusions. The results demonstrate that the proposed scheme reduces the computational complexity required for multi-object detection and tracking and exhibits a Multiple Object Tracking Accuracy (MOTA) of 0.757 and an identity F1 score (IDF1) of 0.832 when compared to conventional multi-stage systems under the same conditions and parameters, along with achieving a high detection precision of 0.934. The results show the viability of implementing the proposed system in practical applications for high-density vehicle VTS channels.

1. Introduction

For smart city applications, intelligent transportation systems (ITSs) have emerged as an economically, socially, and environmentally sustainable solution to the challenges of modern transportation [1,2]. ITSs interconnect vehicles, pedestrians, and infrastructure through sensing, communication, and computation technologies to improve transportation efficiency and safety [3,4].

As a key component of ITS, VTS systems automatically analyze traffic scenes to extract high-level semantic information, such as vehicle trajectories, speeds, traffic density, and lane occupancy. These data form the basis for higher-level ITS applications, including collision detection [5], route optimization [6], and intelligent traffic management [7,8].

Due to the complex nature of traffic scenes, a VTS system is typically broken into small, relatively independent tasks, such as vehicle detection, tracking, and classification. These tasks are often addressed under a single-task learning (STL) paradigm, in which each task is learned separately and later combined to solve the overall problem [9,10]. However, STL methods fail to leverage the shared and complementary knowledge among tasks, which may limit overall performance.

To overcome these limitations, multi-task learning (MTL) has emerged as a promising machine learning paradigm for simultaneously learning several tasks [11]. In contrast to STL, MTL leverages shared representations across tasks, which introduces an inductive bias that constrains the hypothesis space to promote solutions that generalize better and reduce both the risk of overfitting and computational complexity, particularly when the tasks are closely related [11,12]. Prior studies have consistently shown that MTL outperforms STL. For instance, ref. [13] reported that MTL improved document classification performance while also reducing computation time by 27%. Another study reported simultaneous improvements in depth estimation and semantic segmentation [14], highlighting the effectiveness of shared representations.

Recent advances in neural network architectures have further enhanced shared representation learning by introducing mechanisms that regularize the interactions between shared and task-specific knowledge. In particular, Transformer-based architectures have shown remarkable success for extracting more expressive and transferable features across related tasks in computer vision tasks via self-attention mechanisms [15,16,17]. Unlike a convolutonal neural network (CNN), which uses convolutional filters to extract local spatial features, vision transformers treat an image as a sequence of patches and leverage self-attention mechanisms to capture long-range relationships and global context-aware across patches. This makes transformers particularly useful for MTL, in which related tasks can benefit from shared, context-aware representations.

Inspired by these ideas, this study presents a transformer-based MTL model for traffic monitoring. This work extends our previous research on multi-view and MTL VTS systems [18] by incorporating modern neural architectures to enhance feature abstraction and data representation. Specifically, we extend the MOTR framework [19], originally developed for multi-object tracking in human motion analysis, as an end-to-end multi-task model for vehicle detection, classification, and tracking in traffic surveillance.

The main contributions of this work are summarized as follows:

• We present a high-performance transformer-based MTL model for VTS systems that jointly addresses detection, tracking, and classification.
• We perform extensive experiments on standard VTS benchmarks demonstrate that the proposed approach achieves competitive state-of-the-art performance across tasks.
• To the best of our knowledge, this is the first end-to-end MTL model for vehicle detection, tracking, and classification, providing a scalable and generalizable solution for real-world traffic analysis.

The remainder of this paper is organized as follows: Section 2 reviews related work on vehicle detection, tracking, and classification. Section 3 formally defines the problem, followed by a comprehensive description of the MOTR architecture in Section 4. The experimental results and discussion are presented in Section 5 and Section 6, respectively. Finally, Section 7 concludes the paper and outlines future directions.

2. Related Work

The typical pipeline of a VTS system involves a series of related tasks for an accurate semantic understanding of traffic scenes. These tasks commonly include vehicle detection, tracking, and classification. Each of them contributes to the overall system: detection localizes vehicles in each frame, tracking maintains their temporal identities, and classification assigns semantic categories such as car, bus, or truck to each vehicle. In the following subsections, we review representative approaches to each of these tasks, emphasizing the evolution from traditional handcrafted methods to modern deep learning (DL).

2.1. Moving Vehicle Detection Task

Moving vehicle detection is an essential and challenging part of VTS systems, as it involves accurately identifying vehicles in complex environments affected by adverse weather, shadows, visual occlusions, and varying lighting conditions, and where multiple objects coexist, including cyclists, pedestrians, and waving vegetation [20].

In general, vehicle detection methods can be grouped into three main approaches: appearance-based, motion-based, and DL-based.

• Appearance-based approaches model vehicle appearance using handcrafted descriptors such as color and edges [21], the Histogram of Oriented Gradients (HOG) [22], Scale-Invariant Feature Transform (SIFT) [23], Haar-like features [24], Local Binary Patterns (LBPs) [25,26], and wavelet-based representations [27]. Other studies have combined multiple descriptors using tensor decompositions, including Tucker decomposition to fuse HOG, LBP, and Four-Direction Features (FDFs) for vehicle detection at night [28].
• Motion-based approaches exploit temporal changes of pixel values through frame differencing [29,30], background subtraction [31,32,33,34], optical flow [35,36], and subspace methods [37,38].
• DL-based approaches have recently improved vehicle detection by automatically learning robust appearance features, overcoming the limitations of handcrafted descriptors. Notable models include YOLO [39], Faster R-CNN [40], and Mask R-CNN [41]. More recently, transformer-based architectures like Detection Transformer (DETR) [42] introduced an end-to-end detection paradigm that formulates object detection as a direct set prediction problem, leveraging attention mechanisms to capture long-range spatial dependencies and global contextual relationships across the entire scene.

Occlusion Handling

A major challenge of vehicle detection is visual occlusion, in which vehicles or parts of them are hidden behind other objects in the traffic scene. Occluded vehicles often lead to missed detections, inaccurate localization, or the misclassification of partially visible vehicles, potentially degrading overall vehicle detection performance.

Occlusion handling can fall into two main approaches: heuristic-based and DL-based.

• Heuristic-based approaches typically exploit prior knowledge of vehicle geometry and motion—such as shape convexity, symmetry, aspect ratio, and bounding box overlapping—to detect occluded vehicles [9,43,44,45,46]. However, such assumptions often fail in complex scenarios involving heavy occlusion, irregular vehicle shapes, and unpredictable motion patterns.
• DL-based approaches have significantly improved occlusion handling, as these models can not only detect and resolve partially hidden vehicles but also infer or reconstruct occluded parts by leveraging learned features and contextual information [47,48,49,50,51].

2.2. Vehicle Tracking Task

Tracking methods model the temporal evolution of vehicles, including motion trajectories and appearance features, across frames. In VTS systems, accurately tracking vehicles is challenging due to the presence of multiple vehicles and other moving objects. For multi-object tracking (MOT), a common strategy is tracking-by-detection, in which vehicles are first detected in each frame and then associated with those detected in previous frames using appearance or motion cues.

Typically, vehicle tracking approaches fall into three categories: appearance-based, motion-based, and DL-based approaches.

• Appearance-based methods rely on visual similarity measures, such as color histograms, texture descriptors, and local features extracted from vehicle images [52,53,54]. However, these methods are sensitive to illumination, perspective, and occlusions, which can degrade tracking performance.
• Motion-based methods model vehicle dynamics using IoU heuristics or probabilistic filters such as Kalman and particle filters, followed by assignment solvers like the Hungarian algorithm [9,55,56,57]. Classical motion-based trackers include SORT, which uses Kalman filtering and the Hungarian algorithm to estimate bounding box states and associate tracks in real-time [58], and ByteTrack, which extends SORT by also associating low-confidence detections to maintain consistent tracking in crowded or occluded scenes [59]. Nevertheless, they often fail under abrupt maneuvers, irregular trajectories, or heavy occlusions.
• DL methods have improved tracking robustness by leveraging automatically learned feature embeddings for better discrimination and data association [60]. DeepSORT extends SORT by extracting appearance features via a CNN for re-identification after occlusions [61]. Other neural network approaches include Tracktor [62] and CenterTrack [63]. Transformer-based architectures—including TrackFormer [17] and TransTrack [64]—leverage attention mechanisms to capture long-term temporal dependencies, enabling more robust tracking compared to traditional approaches such as Kalman filters, which rely on simplified observation models. Recently, MOTR [19] introduced an end-to-end detection and tracking pipeline with track queries for identity propagation, leveraging transformers to learn nonlinear temporal dependencies directly from the data. MOTRv2 [65] have further extended this framework by incorporating a pretrained YOLOv5 backbone to enhance detection performance.

2.3. Vehicle Classification Task

The vehicle classification task assigns detected vehicles to categories such as cars, buses, trucks, and motorcycles. These methods can fall into two main groups: appearance-based and DL-based approaches.

• Appearance-based approaches model vehicle appearance using handcrafted features—including Image Moments, HOG, SIFT, and color histograms—that capture discriminative visual cues such as shape, texture, and color for different vehicle categories. Subsequently, these features are fed to traditional machine learning classifiers (e.g., support vector machines [9]), or clustering techniques (e.g., K-means [66]). However, the performance of such methods strongly depends on the quality and expressiveness of the handcrafted features.
• DL approaches leverage CNNs to automatically learn discriminative vehicle features directly from images. Modern architectures, including YOLO [39], VGGNet [67], ResNet [68], DenseNet [69], and EfficientNet [70], have consistently outperformed traditional methods, enabling robust intra- and inter-class classification in challenging traffic scenarios.

Table 1. Summary of representative approaches in VTS systems.

Task	Approach	Method	Limitations
Detection	Appearance	HOG [22]	Sensitive to lighting, occlusions, and viewpoint variations
		SIFT [23]	Same as above
		Haar-like features [24]	Same as above
		LBP [25,26]	Same as above
	Motion	Background subtraction [31,32,33,34]	Fails under camera motion, dynamic background, or abrupt vehicle movement
		Optical flow [35,36]	Same as above
		Subspace models [37,38]	Same as above
	DL	YOLO [39]	Requires large annotated datasets; higher computational cost
		Faster R-CNN [40]	Same as above
		Mask R-CNN [41]	Same as above
		DETR [42]	Same as above
Occlusion	Heuristic	Geometric priors [9,43,44,45,46]	Limited robustness for heavy occlusion or irregular shapes
	DL	Feature reconstruction [47,48]	Computationally expensive; requires large datasets with occlusions
		Occlusion context-aware [49,50,51]	Same as above
Tracking	Appearance	Color histograms [52,53]	Sensitive to illumination, perspective changes, and occlusions
		Texture descriptors [54]	Same as above
	Motion	Kalman filter [9]	Fail in presence of nonlinear dynamics
		SORT [58]	Fails under abrupt maneuvers, nonlinear motion, or crowded scenes
		ByteTrack [59]	Same as above
	DL	DeepSORT [61]	High computational cost; requires large annotated sequences
		TrackFormer [17]	Same as above
		MOTR [19]	Limited detection performance
Classification	Appearance	SIFT and Color histograms [9,66]	Performance limited by feature design; sensitive to occlusions
		Image Moments [9]	Same as above
	DL	YOLO, VGGNet, ResNet [39,67,68,69,70]	Requires large labeled datasets; high computational resources

summarizes representative approaches in VTS systems. Nonetheless, despite substantial progress in vehicle detection, occlusion handling, tracking, and classification, most existing methods address these tasks independently, limiting their ability to learn shared representations and capture inter-task dependencies.

3. Problem Statement and Mathematical Formulation

3.1. Problem Statement

Given a traffic surveillance video recorded from a static camera at a specific frame rate, let ${\mathcal{I}}^{\left(1\right)},\dots ,{\mathcal{I}}^{\left(N\right)}$ denote a temporal sequence of frames, where ${\mathcal{I}}^{\left(n\right)}\in {\mathbb{R}}^{H\times W\times 3}$ is the n-th frame and N the total number of frames. Then, the goal of a VTS system is to analyze the sequence of frames to infer high-level semantic traffic information by jointly solving multiple tasks, such as vehicle detection, classification, and tracking.

3.2. Mathematical Formulation

Consider a collection of M supervised tasks in a VTS system. For the m-th task, we seek to learn a mapping ${\mathcal{T}}_m:{\mathbb{R}}^{H\times W\times 3}\to {\mathcal{Y}}_m$ that predicts, with high probability, the corresponding ground truth $y_m\in {\mathcal{Y}}_m$ for a given frame $\mathcal{I}$.

In our case study, we consider the following tasks:

• Vehicle detection (${\mathcal{T}}_1$): predicts the bounding box $y_1^{(n,k)}={\mathit{b}}^{(n,k)}$ corresponding to the k-th vehicle on the road in frame n. Each bounding box is parameterized by the pixel coordinates of its top-left corner and its spatial size (width and height).
• Vehicle classification (${\mathcal{T}}_2$): assigns a category label $y_2^{(n,k)}=c^{(n,k)}$ to the k-th vehicle detected in the frame n.
• Vehicle tracking (${\mathcal{T}}_3$): maintains temporal identity $y_3^{(n,k)}={\mathrm{id}}^{(n,k)}$ for the k-th vehicle detected in the frame n.

Following the standard formulation of MTL [11], we further consider the parametric hypothesis class ${\mathcal{T}}_m=h_m\circ g$ for the m-th task, where $g:{\mathbb{R}}^{W\times H\times 3}\to S$ denotes a mapping from the image space to some shared latent space S parameterized by ${\theta }_{sh}$, and $h_m:S\to {\mathcal{Y}}_m$ represents the task-specific mapping parameterized by ${\theta }_m$. The shared parameters ${\theta }_{sh}$ encode features jointly learned from all tasks, while each task-specific $h_m$ specializes these features to solve the m-th task. And we also consider that each task is associated with a loss function, ${\mathcal{L}}_m:{\mathcal{Y}}_m\times {\mathcal{Y}}_m\to {\mathbb{R}}_{+}$, that evaluates the discrepancy between the prediction and its corresponding ground truth.

MTL then seeks to simultaneously learn the set of task functions, ${\mathcal{T}}_1,\dots ,{\mathcal{T}}_M$, by minimizing a weighted combination of their individual task losses, referred to as the multi-task empirical risk [71], as expressed in Equation (1):

$$\underset{{\theta }_1,\dots ,{\theta }_M,{\theta }_{sh}}{min}\sum _{m=1}^M{\displaystyle \frac{{\lambda }_m}{N}}\sum _{n=1}^N{\mathcal{L}}_m\left(h_m\left(g\left({\mathcal{I}}^{\left(n\right)}\right)\right),Y_m^{\left(n\right)}\right),$$

(1)

where ${\mathcal{I}}^{\left(1\right)},\dots ,{\mathcal{I}}^{\left(N\right)}$ is a sequence of training frames and, for the m-th task, ${\lambda }_m$ denotes its task-importance weight, and $Y_m^{\left(n\right)}:=\{y_m^{(n,1)},\dots ,y_m^{(n,K_n)}\}\subseteq {\mathcal{Y}}_m$ denotes the set of $K_n$ ground truth instances associated with the n-th frame ${\mathcal{I}}^{\left(n\right)}$.

Figure 1 shows a block diagram that visually summarizes the proposed MTL setup and complements the mathematical formulation. For a more detailed visual description of the architecture and the interactions between modules, the reader is referred to Figure 2.

4. Transformer-Based Multi-Task Learning Model for Vehicle Traffic Surveillance

Traditional VTS systems rely on multi-stage pipelines that sequentially address detection, classification, and tracking tasks for the effective monitoring of traffic environments. In such pipelines, each stage is addressed independently, which may lead to suboptimal performance due to their limited capacity to leverage shared representations across tasks.

The MTL formulation in Equation (1) addresses these limitations by enabling a unified end-to-end framework that jointly performs object detection, classification, and tracking, while promoting a shared representation across tasks. Nevertheless, two main challenges arise from this formulation. First, feature sharing must ensure that learned representations capture complementary spatial patterns across tasks. Second, temporal coherence must maintain vehicle identities and representations under occlusions, illumination variations, and dense traffic interactions.

Recent advances in DL have significantly enhanced the performance of VTS systems. For instance, models based on the YOLO architecture have demonstrated remarkable accuracy for vehicle detection and classification [72,73]. However, their inherent multi-stage design treats detection and temporal association independently, which limits their ability to capture spatio-temporal dependencies. To overcome these limitations, the proposed system adapts the MOTR framework for VTS applications. In the next subsection, we will describe the MOTR architecture.

4.1. Proposed Model

The overall architecture of the proposed VTS system is illustrated in Figure 2. Video streams from static traffic surveillance cameras are processed frame-by-frame and fed into the MOTR network, which forms the core of the VTS model. MOTR consists of a transformer-based encoder-decoder architecture designed for end-to-end multi-object tracking. The backbone encoder, based on ResNet-50 with deformable transformer layers [16], extracts rich spatial features, while the transformer decoder utilizes two types of learnable queries: detection queries, responsible for identifying newly appearing vehicles, and tracking queries, which propagate the identities of vehicles detected in previous frames.

This encoder–decoder mechanism allows MOTR to detect and re-identify vehicles in continuous video streams, producing reliable bounding boxes, high-confidence classification outputs, and maintaining consistent vehicle identities across frames.

Let ${\mathcal{Q}}_d$ denote the learnable detection queries and ${\widehat{\mathcal{Q}}}_{tr}^{\left(n\right)}$ the predicted tracking queries at frame n. For the first frame ($n=1$), the model only uses detection queries, i.e., ${\widehat{\mathcal{Q}}}_{tr}^{\left(0\right)}=⌀$. For subsequent frames ($n>1$), the query set is recursively concatenated, as Equation (2) shows:

$${\widehat{\mathcal{Q}}}_{tr}^{\left(n\right)}={\mathcal{Q}}_d\cup {\widehat{\mathcal{Q}}}_{tr}^{(n-1)}.$$

(2)

To correctly associate predicted queries with ground-truth vehicle instances, a tracklet-aware label assignment (TALA) mechanism is employed. Following the principles in [19], this mechanism separates label assignment for detection and tracking queries:

• Detection queries are matched exclusively to newborn vehicles using bipartite matching between predictions of detect queries, ${\widehat{\mathcal{Q}}}_d^{\left(n\right)}$, and the ground truth of newborn objects, ${\mathcal{O}}_{new}^{\left(n\right)}$, at frame n.
• Tracking queries are matched according to their temporal identity consistency, inheriting their assignments from the previous frame to preserve temporal identity.

Formally, let ${\omega }_d^{\left(n\right)}$ and ${\omega }_{tr}^{\left(n\right)}$ denote the label assignments for detection and tracking queries, respectively. Then, the total assignment for the n-th frame is given via Equation (3),

$$\begin{array}{c}\hfill {\omega }_{tr}^{\left(n\right)}={\omega }_{tr}^{(n-1)}\cup {\omega }_d^{(n-1)}\\ \hfill {\omega }_d^{\left(n\right)}=\underset{{\omega }_d^{\left(n\right)}\in {\Omega }^{\left(n\right)}}{arg\; min}\mathcal{L}\left({\widehat{\mathcal{Q}}}_d^{\left(n\right)}|{\omega }_d^{\left(n\right)},{\mathcal{O}}_{\mathrm{new}}^{\left(n\right)}\right),\end{array}$$

(3)

where $\mathcal{L}$ denotes the pairwise matching cost, as defined in DETR [42], and ${\Omega }^{\left(n\right)}$ represents the space of all possible bipartite associations between detection queries and newborn objects [19].

This formulation ensures that newborn vehicles (e.g., cars entering the monitored area) are detected via ${\widehat{\mathcal{Q}}}_d^{\left(n\right)}$, while ongoing vehicle tracks are maintained through ${\widehat{\mathcal{Q}}}_{tr}^{\left(n\right)}$. This mechanism is particularly relevant to VTS applications, where vehicles frequently enter and exit the camera’s field of view due to occlusions, lane changes, or traffic flow dynamics.

During inference, the model processes frames sequentially and updates the query sets in an online manner. The hidden states corresponding to the tracked vehicles are passed to the next frame through a Query Interaction Module (QIM), maintaining inter-frame consistency [19].

4.2. Multi-Task Learning Framework

The MOTR learning problem can be modeled as a joint optimization problem that simultaneously learns to detect, classify, and track multiple objects within a unified transformer framework. This formulation enables the model to exploit shared representations between the two related tasks: object detection and temporal identity association. Compared to traditional multi-stage VTS pipelines, transformer-based architectures offer a key advantage: they learn long-range spatial and temporal relationships in complex traffic scenes—including vehicle appearance and occlusion variations, and complex maneuvers—without relying on naive assumptions or hand-crafted features.

Formally, given a sequence of N frames, ${\mathcal{I}}^{\left(1\right)},\dots ,{\mathcal{I}}^{\left(N\right)}$ with ground truths $Y=$ ${\left\{(Y_1^{\left(n\right)},Y_2^{\left(n\right)},Y_3^{\left(n\right)})\right\}}_{n=1}^N$, their corresponding predictions, $\widehat{Y}={\left\{({\widehat{Y}}_1^{\left(n\right)},{\widehat{Y}}_2^{\left(n\right)},{\widehat{Y}}_3^{\left(n\right)})\right\}}_{n=1}^N$, and the set of matching assignments between predictions and ground truth instances, $\omega ={\left\{{\omega }^{\left(n\right)}\right\}}_{n=1}^N$, we seek to minimize the overall loss, ${\mathcal{L}}_o$, of the entire video sequence, called the collective average loss (CAL), as defined in Equation (4):

$${\mathcal{L}}_o\left(\widehat{Y}\right|\omega ,Y)={\displaystyle \frac{{\displaystyle \sum _{n=1}^N\left(\mathcal{L}\left({\widehat{\mathcal{Q}}}_{tr}^{\left(n\right)}|{\omega }_{tr}^{\left(n\right)},{\mathcal{Q}}_{tr}^{\left(n\right)}\right)+\mathcal{L}\left({\widehat{\mathcal{Q}}}_d^{\left(n\right)}|{\omega }_d^{\left(n\right)},{\mathcal{Q}}_d^{\left(n\right)}\right)\right)}}{{\displaystyle \sum _{n=1}^N{V}^{\left(n\right)}}}}.$$

(4)

Here, $V^{\left(n\right)}=V_{tr}^{\left(n\right)}+V_d^{\left(n\right)}$ is the total number of ground-truth objects in the frame n, where $V_{tr}^{\left(n\right)}$ and $V_d^{\left(n\right)}$ are the numbers of tracked objects and newborns, respectively. The function $\mathcal{L}$ represents the loss for a single frame, which is defined as in Equation (5):

$$\mathcal{L}({\widehat{Y}}^{\left(n\right)}\mid {\omega }^{\left(n\right)},Y^{\left(n\right)})={\lambda }_{cls}{\mathcal{L}}_{cls}+{\lambda }_1{\mathcal{L}}_1+{\lambda }_{giou}{\mathcal{L}}_{giou},$$

(5)

where ${\mathcal{L}}_{cls}$ is the focal loss [74], ${\mathcal{L}}_1$ is the ${\ell }_1$ loss, and ${\mathcal{L}}_{giou}$ is the generalized intersection-over-union (gIoU) loss [75], while ${\lambda }_{cls}$, ${\lambda }_1$, and ${\lambda }_{giou}$ are their corresponding weights.

5. Experimental Results

In this section, the performance of the model is evaluated using the UA-DETRAC benchmark dataset [76]. Our experiments assess the robustness of the model under various environmental conditions and traffic densities, as well as its ability to maintain stable multi-object detection and tracking accuracy. Quantitative results are reported in terms of standard MOT metrics (see Section 5.2). All experiments were conducted on a workstation equipped with an Intel Core i7-13650HX CPU, 16 GB of RAM, and an NVIDIA RTX 4050 GPU, running Windows 11 with CUDA 12.2 and PyTorch 2.1.0.

5.1. Dataset Description

We employ the UA-DETRAC benchmark [76] for evaluation. UA-DETRAC is a fully annotated, large-scale corpus for vehicle detection and multi-object tracking in real-world traffic surveillance. It contains more than $140,000$ frames from 100 video sequences at a native resolution of $960\times 540$ and 25 FPS, all acquired with fixed cameras. The sequences span urban scenes with diverse traffic densities, illumination conditions, and weather, yielding frequent partial/heavy occlusions and pronounced scale variation—conditions that stress both detection and association.

Each frame provides tight bounding boxes for every vehicle, together with a persistent object identifier (obj_id), to maintain temporal consistency across frames. Additional per-instance attributes include the vehicle category (car, bus, van, others), the occlusion level (none, partial, heavy), and the truncation ratio, defined as the fraction of the object area outside the image bounds. These annotations support a controlled analysis of tracking performance under visibility degradation and class-specific traffic composition.

5.2. Quantitative Results

The performance of the MOTR model was evaluated using ten representative videos from the UA-DETRAC dataset [76], covering diverse scenarios, including heavy traffic, partial occlusions, and varying motion patterns. The evaluation follows standard multi-object tracking protocols, reporting MOTA, IDF1, identity switches (IDS), identity precision (IDP), identity recall (IDR), precision, and recall metrics [76,77]. These are typical metrics used to evaluate detection reliability and identity continuity, where higher values of MOTA, IDP, IDR, IDF1, precision, and recall suggest superior overall performance, while lower IDS values mean enhanced temporal identity preservation across the sequence.

Table 2. MOTR performance on ten representative UA-DETRAC videos, where the best value for each metric is highlighted in bold.

Video	MOTA	IDS	IDP	IDR	IDF1	Precision	Recall
MVI_40962	0.863	4	0.981	0.868	0.921	0.988	0.875
MVI_40981	0.796	3	0.909	0.877	0.893	0.913	0.881
MVI_40963	0.710	32	0.888	0.740	0.807	0.927	0.773
MVI_41073	0.822	17	0.922	0.875	0.898	0.934	0.886
MVI_20063	0.776	8	0.894	0.823	0.857	0.922	0.849
MVI_20033	0.754	2	0.972	0.769	0.859	0.977	0.773
MVI_40212	0.649	12	0.902	0.695	0.785	0.922	0.711
MVI_40204	0.697	24	0.873	0.743	0.803	0.910	0.775
MVI_20064	0.707	19	0.923	0.706	0.800	0.963	0.736
MVI_20065	0.700	20	0.853	0.769	0.809	0.889	0.801

presents the quantitative results. The results show that sequences with moderate traffic density achieve superior performance, with MVI_40962 and MVI_40981 obtaining MOTA values of 0.863 and 0.796, respectively. These sequences also maintain low identity switch counts of 4 and 3, confirming effective identity preservation under favorable conditions. In contrast, sequences that exhibit heavy occlusion and dense traffic exhibit reduced performance, with MVI_40963 and MVI_40204 showing MOTA values of 0.710 and 0.697, along with elevated identity switch counts of 32 and 24, respectively.

The overall system performance in

Table 3. Average MOTR performance over the ten representative UA-DETRAC videos.

Metric	MOTA	IDF1	IDP	IDR	Precision	Recall
Mean	0.757	0.832	0.906	0.767	0.934	0.796
Std. Dev.	0.065	0.048	0.034	0.063	0.024	0.058

shows a mean MOTA of 0.757 and IDF1 of 0.832 across the evaluated sequences. The high precision value of 0.934 indicates minimal false positives, while the recall of 0.796 reflects the system’s capability to maintain detection coverage in challenging environments. The standard deviation values demonstrate consistent performance across different traffic scenarios.

Figure 3 shows sample frames of four distinct UA-DETRAC videos in different traffic scenarios to supplement the numerical evaluation found in Table 2 and Table 3.

6. Discussion

The implementation of the transformer-based architecture in the proposed VTS system demonstrates robust multi-object tracking capabilities across diverse urban scenarios. The experimental results show that the system achieves a MOTA of 0.757 and an IDF1 of 0.832, while simultaneously maintaining a high detection precision of 0.934 across different traffic conditions.

Per-sequence analysis reveals performance variations aligned with environmental challenges. Sequences such as MVI_40962 and MVI_41073 achieve superior performance with MOTA values of 0.863 and 0.822, respectively, demonstrating a high performance under moderate traffic density and good visibility conditions. In contrast, sequences such as MVI_40212 and MVI_40204 present reduced performance metrics, particularly in recall values of 0.711 and 0.775, highlighting limitations in handling severe occlusions and significant scale variations.

Identity preservation is further characterized by identity metrics. The high IDP value of 0.906 indicates that, once a track is established, identity assignments are rarely incorrect. Conversely, a lower IDR of 0.767 reveals that the system may fail to recover or maintain identities when targets reappear after occlusions or when appearance cues are degraded.

The number of IDS ranges from 2 to 32 per sequence, with elevated counts being aligned with challenging sequences: MVI_40963 (32), MVI_40204 (24), and MVI_20065 (20). These sequences also exhibit comparatively lower recall, supporting the hypothesis that identity fragmentation is primarily driven by missed detections during occlusion events. In contrast, MVI_20033 achieves the lowest IDS (2) while maintaining high precision (0.977), suggesting that the system can effectively stabilize identity assignment when scene dynamics are moderate.

The missed detections observed in Figure 3 are primarily attributed to the limited spatial resolution of the corresponding vehicles, which prevents the model from extracting sufficiently distinctive visual features. This is further supported by the fact that vehicles exhibiting similar visibility and occlusion conditions—but with a higher spatial resolution—are effectively detected. It is important to highlight that the UA-DETRAC benchmark does not consider these vehicles for tracking evaluation, so they do not impact the reported performance metrics.

6.1. Failure Modes and Operating Conditions

From the results in Table 2, the following observations can be made:

(a)

Density and occlusion dominate errors. Sequences characterized by dense traffic and frequent partial occlusions (e.g., MVI_40212 and MVI_40204) show a lower recall and a higher IDS. This is consistent with identity drops caused by lower spatial visibility, rather than tracker confusion.

(b)

Precision is uniformly strong. All sequences maintain precision $\ge 0.889$, with several above $0.96$ (MVI_20064: $0.963$; MVI_20033: $0.977$; and MVI_40962: $0.988$). This suggests that false positives are well handled, and further improvements should focus on increasing recall while preserving precision.

(c)

Association is reliable when observations persist. High IDP across sequences indicates that the association cost is well calibrated for temporally consistent tracks. Drops in IDR occur mostly when targets leave the field of view or are heavily truncated.

6.2. Implications and Future Improvements

The above findings imply that the current pipeline is well suited for deployment in fixed-camera traffic monitoring, where false alarms must be minimized. In highly congested scenes, performance is bounded by missed detections during transient occlusions. Accordingly, two avenues are expected to yield measurable improvements:

• Occlusion-aware re-identification. Incorporating long-range appearance embeddings with explicit occlusion modeling and temporal memory (e.g., tracklet-level re-ID with motion gating) should raise IDR and reduce IDS in clips with frequent hide-and-reveal events.
• Recall-oriented detection tuning. Threshold calibration and hard-example mining targeted at small, truncated, or partially visible vehicles can improve recall while preserving the current precision regime.

6.3. Summary

In summary, the model delivers competitive tracking with high precision and stable identity assignment across diverse operating conditions (mean MOTA $0.757$, IDF1 $0.832$). Performance degradations concentrate in dense-traffic, occlusion-heavy scenes, where recall and IDR are most affected, leading to higher IDS. Given that precision is already high and uniform, future work should prioritize recall enhancements and robust re-identification to close the remaining gap without compromising the low false-positive rate.

7. Conclusions

This paper has presented a comprehensive evaluation of a transformer-based model for multi-vehicle tracking in urban environments. The proposed system demonstrates consistent performance across various challenging scenarios, achieving a mean MOTA of 0.757 and IDF1 of 0.832 on the UA-DETRAC benchmark. The system maintains a high precision of 0.934 while effectively preserving vehicle identities with an average of 15 identity switches per sequence.

The unified architecture successfully handles vehicle detection, classification, and tracking within a single framework, eliminating the complex multi-stage pipelines typically required in conventional VTS systems. The model exhibits high robustness under diverse conditions, including high traffic density, moderate partial occlusions, and varying illumination. The system maintains tracking consistency for traffic conditions, as reflected in the very low identity inconsistency observed in sequences such as MVI_20033, which contains only two identity switches. In contrast to multi-stage pipelines, which run an STL model for each task, significantly increasing inference time and memory resources, the proposed model learns a shared representation that substantially reduces computational complexity, a crucial aspect in VTS systems, especially for resource-constrained environments.

Future work will focus on enhancing re-identification capabilities during prolonged occlusion events and optimizing the system for real-time deployment in ITS applications. The results demonstrate the viability of the proposed approach to practical VTS systems operating in complex urban environments, providing a solid foundation for next-generation urban traffic monitoring solutions.