Multi-Modal Artificial Intelligence for Smart Cities: Experimental Integration of Textual and Sensor Data

Abstract

Smart city decision-making increasingly relies on heterogeneous urban data sources. Dense traffic sensor streams provide continuous quantitative measurements, while citizen-generated textual reports offer event-driven contextual information. However, integrating these modalities remains challenging due to temporal misalignment, textual sparsity, and semantic noise. This paper investigates multi-modal learning for traffic congestion severity prediction through an experimental integration of open traffic sensor data (METR-LA: Los Angeles, USA) and citizen-generated textual reports (NYC 311: New York City, USA). Congestion severity is formulated as a four-class classification task derived from traffic speed measurements. We propose an end-to-end framework that combines: (i) sensor time-series encoding using a GRU-based temporal encoder, (ii) textual representation learning using a BERT-based encoder, (iii) a symmetric time-window alignment strategy (±Δ) to associate irregular reports with sensor time steps, and (iv) multiple fusion architectures, including early fusion, late fusion, and a cross-attention module for cross-modal interaction modeling. Experiments on publicly available datasets show that multi-modal early fusion achieves the best overall performance (Accuracy = 0.8283, Macro-F1 = 0.8231) compared to uni-modal baselines. In the studied cross-city setting with sparse and weakly aligned textual signals, the proposed cross-attention fusion does not outperform the strong sensor-only baseline, suggesting that the sensor modality dominates when cross-modal signal strength is limited. These results highlight both the potential and the practical constraints of multi-modal fusion in heterogeneous smart-city environments, emphasizing the importance of alignment design, modality relevance, and transparent experimental validation.

1. Introduction

Smart cities increasingly depend on data-driven technologies to enhance urban efficiency, sustainability, and quality of life. The rapid deployment of Internet of Things (IoT) infrastructures and city-scale sensing platforms has enabled continuous monitoring of critical domains such as traffic, energy, public safety, and environmental conditions [1,2,3,4]. These sensing systems generate high-frequency numerical streams (e.g., traffic speed, flow, and occupancy) that provide objective measurements of urban dynamics.

In parallel, cities also generate abundant unstructured textual information through citizen-facing platforms and digital communication channels. Examples include 311 service requests, incident reports, social media posts, and administrative logs. Such text-based records frequently contain rich contextual cues, human observations, and descriptions of causes (e.g., accidents, road closures, events) that may not be immediately captured through sensors alone [2,5,6]. Therefore, integrating sensor data with textual reports is increasingly viewed as a key direction for robust urban intelligence, enabling models to combine the precision of physical measurements with the interpretability and context of human-generated narratives [7,8].

Urban traffic congestion remains one of the most persistent problems affecting smart city mobility, economic productivity, and overall livability. Timely prediction of congestion severity can support adaptive traffic control, route optimization, incident management, and planning of public mobility services [9,10,11]. Many existing traffic analytics approaches rely primarily on sensor-based time-series forecasting, which provides reliable quantitative indicators but often fails to incorporate contextual triggers that explain sudden changes in congestion patterns [7,12]. Conversely, textual reports can provide early warnings and explanations for traffic disruptions but are noisy and sparsely available, and their timestamps may not align with sensor sampling intervals [7,8].

Accordingly, this paper investigates traffic congestion severity prediction as a concrete case study to evaluate multi-modal integration in smart city environments. Formally, given (i) a window of traffic sensor observations over a past horizon and (ii) temporally associated textual reports within a configurable alignment window, our objective is to predict the next-step congestion severity label y (t + 1). Congestion severity is modeled as a four-class ordinal classification problem (Low, Moderate, High, Severe), derived from traffic speed using quantile-based discretization computed on the training split (Section 3.3).

This study aims to:

• Design a practical multi-modal pipeline that integrates traffic sensor streams with textual urban reports for congestion severity prediction.
• Develop a temporal alignment strategy that links irregular, sparse textual signals to regularly sampled sensor time steps.
• Evaluate fusion architectures (early fusion, late fusion, and cross-attention fusion) for modeling cross-modal interactions.
• Provide reproducible experimental comparisons and ablation analysis, and discuss robustness considerations under sparse/noisy text conditions.

• Study Challenges:

The experimental integration of textual and sensor data in smart city traffic analytics introduces several technical challenges:

• Temporal misalignment: Sensor measurements are typically sampled at fixed intervals, while textual reports occur irregularly. Reports may arrive before or after congestion becomes observable in sensor streams, making alignment non-trivial [7,9].
• Text sparsity and noise: Textual reports may be missing for many time steps, may contain informal language, and may include irrelevant or ambiguous information. This can reduce the reliability of text-only prediction models [7,13].
• Heterogeneous feature spaces: Sensor time-series are numerical and structured, whereas text is unstructured and semantic. Combining these representations requires effective embedding and fusion mechanisms that preserve complementary information [7,8].
• Real-time constraints: Smart city systems often require timely predictions. Multi-modal models must be computationally feasible for practical deployment and robust to incomplete or delayed inputs [3,14,15].
• Cross-city pairing and weak cross-modal relevance: When sensor streams and textual reports originate from different cities, the aligned text may be temporally overlapping but weakly related to the underlying traffic dynamics observed by the sensors. This limits interpretability, increases noise in multimodal fusion, and may reduce the expected benefits of interaction-based fusion mechanisms.

Therefore, this study emphasizes robustness-oriented multimodal integration, focusing on alignment and fusion behavior under sparse and weak auxiliary textual signals.

Table 1. Comparison of related multi-modal fusion studies in smart city and urban computing.

Study	Modalities	Task	Fusion/Model	Key Limitation vs. Our Work
[1]	Conceptual	Smart cities + AI overview	—	No experimental model or multimodal fusion pipeline
[7]	Various	Systematic review	—	Review only; does not propose or evaluate a specific fusion architecture
[11]	Traffic (multi-agent)	Flow prediction + anomaly	Multi-agent AI	Not multimodal text + sensor; focuses on agents rather than cross-modal fusion
[15]	Sensors + social/text	Traffic event detection	Multi-modal GAN	Focuses on event detection rather than severity classification; alignment analysis limited
[16]	Cross-domain	Taxonomy + advances	—	Broad outlook; no end-to-end validated pipeline for text + traffic sensors
[17]	Multimodal	Survey	—	General multimodal fusion; not tailored to smart city traffic severity prediction
[18]	Multimodal	Smart city multimodal framework	Positional-attention DL	Not focused on sensor time-series + textual reports alignment for traffic severity
[19]	Sensors (traffic + env)	Smart city optimization	ML fusion	Fusion of sensors only; no textual modality
[20]	Sensors	Deep supervised learning support	Sensor fusion	No textual modality; not designed for multimodal urban reports

summarizes the studies relevant to the topic of the study and highlights the research gap that the study addresses.

2. Related Work

The rapid evolution of smart city infrastructures has led to the availability of heterogeneous urban data sources, including large-scale sensor streams and unstructured textual records. Consequently, modern urban intelligence increasingly relies on learning frameworks capable of integrating multiple modalities to support predictive and decision-making tasks. In this section, we review (i) sensor-based traffic prediction and congestion modeling, and (ii) multimodal learning and fusion methods in smart city and urban computing. We then position the research gap addressed by this work, emphasizing the need for robust temporal alignment and advanced cross-modal fusion for congestion severity prediction.

2.1. Sensor-Based Traffic Prediction and Congestion Modeling

Traffic prediction and congestion modeling have been widely studied within smart city mobility research, driven by the demand for improved traffic management, routing optimization, and intelligent transportation systems. Traditional data-driven approaches rely primarily on structured sensor measurements such as speed, flow, and occupancy, which provide reliable indicators of traffic conditions at high temporal resolution [3,4]. With the growth of IoT-based sensing infrastructures, traffic analytics has increasingly benefited from machine learning methods that model temporal dependencies and non-linear traffic dynamics [2,3,21].

Deep learning models have become central in traffic forecasting due to their ability to capture complex temporal patterns. Recurrent architectures such as LSTM and GRU are widely used for sequential modeling, enabling prediction of traffic states based on historical sensor measurements [11]. More recently, attention-based architectures and Transformer-like models have been adopted to better model long-range dependencies and dynamic temporal correlations in traffic time-series data [8,22]. These approaches generally improve predictive performance and stability compared to traditional linear models, especially when traffic patterns exhibit non-stationary behavior.

Beyond continuous-value forecasting, several smart city applications require categorical modeling of traffic conditions (e.g., congestion severity levels), which is often formulated as a classification problem. Severity-based models are practical for decision support, because discrete levels can trigger policy actions such as signal timing adjustments, incident alerts, or rerouting strategies [9,10]. However, a major limitation of purely sensor-based congestion modeling is that sensor streams may not fully capture contextual causes of sudden congestion, such as accidents, roadworks, or special events. This limitation motivates the integration of additional modalities—particularly textual urban reports—into congestion prediction pipelines [7].

Related research also extends traffic modeling toward anomaly detection and event-driven analytics, where the goal is to detect disruptions or abnormal traffic behaviors. Multi-agent and AI-driven strategies have been explored to enhance predictive capabilities and decision support under dynamic urban scenarios [11,23]. Nevertheless, most sensor-based approaches remain limited in capturing semantic context, which is often essential for interpreting and anticipating congestion-causing events.

2.2. Multi-Modal Learning and Fusion in Smart Cities

Multi-modal learning aims to integrate heterogeneous data representations to improve prediction accuracy, interpretability, and robustness. In smart city contexts, multi-modal fusion has been increasingly recognized as a crucial capability for integrating sensory measurements with other information sources such as text, images, audio, and remote sensing data [7,8,24]. Recent surveys highlight that smart city analytics is inherently multimodal, and that fusion methods play a central role in enabling data-centric governance and scalable urban intelligence [5,7,14,25].

A broad body of research has proposed fusion taxonomies, distinguishing between early fusion (feature-level concatenation), late fusion (decision-level aggregation), and intermediate fusion methods that integrate modalities within neural architectures [7,8,9]. While early fusion is computationally simple, it often fails to capture cross-modal dependencies beyond linear combination. Late fusion improves modularity and robustness but may not fully exploit complementary information between modalities. Intermediate fusion—especially attention-based mechanisms—has gained significant interest because it enables explicit modeling of interactions between modalities [8,9,26].

Multi-modal fusion in urban analytics has attracted growing attention due to the increasing availability of heterogeneous urban data. Systematic reviews highlight the importance of information fusion methods for smart cities and provide taxonomies of fusion strategies across modalities and domains [7,14,27]. Some studies explore multimodal frameworks and attention-based mechanisms for smart city data processing [18,22,28], while others investigate multimodal learning for specific event detection tasks in traffic contexts [12]. Moreover, recent surveys on deep multimodal fusion emphasize that advanced fusion mechanisms—such as attention-based cross-modal interaction—are generally more effective than simple feature concatenation in capturing complementary information [8,29,30].

In smart city applications, multi-modal approaches have been explored across multiple domains. Urban computing studies propose general frameworks for processing multi-modal city data using deep learning and attention-based representations [18,22,31]. In traffic-related contexts, multi-modal models have been applied to event detection, demonstrating that integrating sensor observations with additional modalities can improve detection of traffic incidents and unusual patterns [12]. Furthermore, systematic reviews on information fusion emphasize that temporal misalignment and modality imbalance (e.g., dense sensors vs. sparse text) are among the key challenges in urban multi-modal systems [7,32,33,34].

Recent work also indicates growing interest in generative AI and large language models (LLMs) for smart city management and cognitive digital twins, where multi-modal data can support decision-making and knowledge representation [35,36,37,38,39]. While these directions highlight the importance of advanced semantic understanding for smart city intelligence, the integration of textual and numerical sensor streams for time-dependent prediction tasks still requires carefully designed fusion strategies and robust experimental validation [40,41].

Overall, the literature shows strong motivation for multi-modal fusion in smart cities; however, effective integration of time-series sensors with irregular textual streams remains technically challenging due to differences in structure, frequency, and reliability. This motivates approaches that include explicit temporal alignment and cross-modal interaction modeling.

2.3. Research Gap

Despite these advances, there remains a limited number of experimentally validated frameworks that (i) explicitly handle temporal alignment between irregular textual signals and regularly sampled traffic sensors, and (ii) evaluate advanced fusion architectures (e.g., cross-attention) for congestion severity prediction with thorough ablation experiments and robustness discussion. This motivates the present work, which focuses on an end-to-end multi-modal pipeline with explicit temporal alignment and cross-modal fusion for smart city traffic intelligence.

To provide a conceptual overview of existing methodological directions and to clearly delineate the contribution of this study, Figure 1 synthesizes prior multimodal fusion strategies in smart city traffic analytics into a unified taxonomy.

3. Materials and Methods

3.1. Study Overview and Pipeline

This study proposes an experimental multi-modal learning framework that integrates traffic sensor streams with textual urban reports to predict traffic congestion severity in smart city settings. The framework is designed to handle two heterogeneous sources: (i) regularly sampled numerical time-series data captured by traffic sensors, and (ii) irregular, event-driven textual reports that may provide early semantic signals about disruptions, incidents, or contextual triggers.

-

Inputs and Output:

Let t denote a discrete time index aligned with the sensor sampling interval. The proposed system takes the following inputs:

• Traffic sensor window: A historical sensor window of length L ending at time t:

  $${\mathrm{X}}_{\mathrm{s}}\left(\mathrm{t}\right)=\left\{{\mathrm{x}}_{\mathrm{s}}\left(\mathrm{t}-\mathrm{L}+1\right),\dots ,{\mathrm{x}}_{\mathrm{s}}\left(\mathrm{t}\right)\right\}$$

  (1)

  where ${\mathrm{x}}_{\mathrm{s}}\left(·\right)$ denotes the traffic speed measurement vector across all sensor stations at each time step.
• Aligned textual context: A set of textual reports temporally associated with time t using an alignment window Δ:

  $$\mathrm{T}\left(\mathrm{t}\right)=\left\{r_i\right| \left|\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\right(r_i)-\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}(\mathrm{t}\left)\right|\le \mathsf{\Delta }\}$$

  (2)

  where $r_i$ denotes an individual text report.

The prediction target is a discrete congestion severity label at the next time step:

At each time step t, the multimodal input to the model can be formally represented as a paired structure ${(\mathrm{X}}_{\mathrm{s}}\left(\mathrm{t}\right),H_t)$, where ${\mathrm{X}}_{\mathrm{s}}\left(\mathrm{t}\right)\in {\mathrm{R}}^{L \times N}$ denotes the sensor window of length L across N sensors, and $H_t=\left[{\mathrm{h}}_{t,i},\dots ,{\mathrm{h}}_{\mathrm{t},{\mathrm{n}}_{\mathrm{t}}}\right]$ is the set of aligned report embeddings within the temporal window, with ${\mathrm{h}}_{t,i}\in {\mathrm{R}}^{d_t}$. This formulation highlights the structured nature of the multimodal input and clarifies the dependency of the fusion mechanism on the availability of aligned reports.

where $h_{t,i}$ denotes the embedding of the i-th textual report aligned to time step t, and nt is the number of aligned reports within the window Δ. Thus, $H_t$ is a list (set) of report embeddings, not paired elements.

where N is the number of sensors and $d_t$ is the text embedding dimension.

$$\mathrm{y}\left(\mathrm{t}+1\right)\in \left\{1,2,3,4\right\}$$

(3)

corresponding to Low, Moderate, High, and Severe congestion levels, derived from the traffic speed at time t + 1. Predicting $\mathrm{y}\left(\mathrm{t}+1\right)$ from history up to t avoids using the same measurement to define the label.

-

Processing Pipeline:

The overall processing pipeline consists of the following stages:

• Data acquisition from open datasets (traffic sensors + textual reports).
• Preprocessing of sensor time series (cleaning, normalization, windowing) and text (cleaning, tokenization).
• Temporal alignment to associate irregular textual reports with regularly sampled sensor time steps using a configurable window Δ.
• Modality-specific encoding, where sensor sequences and aligned text are converted into compact embeddings using dedicated encoders.
• Multi-modal fusion using early fusion, late fusion, or cross-attention-based fusion to model interactions between the sensor and textual embeddings.
• Classification, where a prediction head outputs the congestion severity level.
• Evaluation, including comparisons with baseline models and ablation analysis and robustness discussion.

To provide a clear and structured visualization of the above-described stages, Figure 2 illustrates the overall multi-modal framework for traffic congestion severity prediction.

The framework aligns irregular text reports to sensor time steps and fuses modality embeddings via cross-attention.

3.2. Open Datasets

To ensure reproducibility and practical relevance, the proposed framework is evaluated using publicly available open datasets representing traffic sensor streams and textual urban reports. This selection reflects a typical smart city setting where dense sensing infrastructure is complemented by human-generated textual signals.

3.2.1. Traffic Sensor Dataset (METR-LA)

The traffic sensing modality is represented using the METR-LA dataset, which has been widely adopted in urban traffic modeling research. It contains traffic speed measurements collected from a network of roadway sensors in the Los Angeles metropolitan area. The measurements are aggregated at a fixed sampling interval (5 min), enabling the construction of regular time-series sequences suitable for deep learning-based temporal modeling.

• Key characteristics of METR-LA include:

• Spatial scale: multiple sensors distributed across the city road network.
• Temporal scale: continuous time-series spanning multiple months.
• Measurement type: traffic speed (primary variable), which can be used to derive congestion levels.

Although METR-LA provides a structured and dense sensor stream, it does not directly encode semantic context about external triggers (e.g., accidents, events). This motivates the integration with textual urban reports as an additional modality [7].

3.2.2. Textual Dataset (NYC 311)

The textual modality is represented using the NYC 311 service requests dataset, which contains large-scale public service reports submitted by citizens. These reports include unstructured text descriptions, timestamps, and location-related attributes. Such text streams provide qualitative, human-centered descriptions of urban issues (e.g., road conditions, disruptions), and can serve as a proxy for real-world city reporting channels relevant to smart city monitoring.

• Key characteristics of NYC 311 include:

• Text content: short-to-medium descriptions of incidents and service issues.
• Temporal coverage: long-term, continuously updated records.
• Spatial attributes: borough/zip code/geo-location fields.

Textual data are inherently irregular and noisy. Reports may be sparse relative to sensor sampling, requiring explicit alignment and robust encoding strategies [7,13].

3.2.3. Data Integration Rationale

In real smart city deployments, traffic monitoring often combines dense sensor streams with irregular human-generated reports from the same city. In this work, we construct an experimental multimodal benchmark by integrating two widely used open datasets that originate from different cities (METR-LA, NYC 311). Because the two streams do not describe the same roadway network or incident events, temporal overlap does not guarantee semantic correspondence. Consequently, the aligned text modality should be interpreted as a weak and potentially irrelevant auxiliary stream. We therefore position this study as an experimental robustness investigation of temporal alignment and fusion strategies under sparse/noisy and weakly related text signals, rather than as a fully unified real-city multimodal deployment. All timestamps are normalized to a common reference (UTC) before applying the alignment window.

Table 2. Dataset statistics and modality characteristics.

Modality	Dataset	Data Type	Temporal Resolution	Spatial Scale	Time Span Used in This Study	Size Available in Span	Size Used in Experiments	Key Fields	Notes
Sensor	METR-LA	Traffic speed time-series	5 min	207 loop detectors	1 March 2012–27 June 2012	207 sensors × 34,272-time steps	Same (full)	timestamp, sensor ID, speed	Regular sampling; missing values handled via interpolation; z-score normalization
Text	NYC 311	Citizen textual reports	Irregular (event-driven)	Citywide (NYC)	1 March 2012–27 June 2012	570,998 reports	120,000 reports (sampled)	created date, complaint type, descriptor, location	Noisy and sparse relative to sensors; cleaned and temporally aligned using ±Δ window

summarizes the open datasets used in this study. METR-LA provides dense traffic speed measurements sampled every 5 min from 207 sensors, spanning 34,272-time steps between 1 March 2012 and 27 June 2012. NYC 311 provides irregular citizen-generated textual reports; after restricting to the same temporal window, 570,998 reports are available. For computational efficiency, we use a fixed subset of 120,000 reports after preprocessing (removing empty or extremely short texts) and randomly sampled from the full set of reports within the study period using a fixed random seed for reproducibility. We note that any subsampling may alter report density patterns and is therefore reported explicitly as an experimental constraint.

A descriptive inspection of the METR-LA speed data reveals clear temporal variability consistent with daily traffic patterns, where speed values tend to decrease during peak commuting periods and increase during off-peak hours. Similarly, the NYC 311 reports exhibit irregular temporal density, with bursts of activity during certain periods and long intervals with sparse reporting. These characteristics motivate the need for robust temporal alignment and fusion mechanisms that can handle uneven cross-modal availability.

3.3. Label Construction (Congestion Severity)

In this study, traffic congestion severity is formulated as a four-class classification task derived from continuous traffic speed measurements. Speed is a widely used proxy for congestion, where lower speeds typically indicate heavier congestion levels. To construct discrete labels in a reproducible manner, we transform the speed values into four ordinal severity classes: Low, Moderate, High, and Severe.

3.3.1. Severity Levels

Let v(t) denote the citywide average traffic speed (i.e., averaged across the 207 METR-LA sensors) at time t. We define the congestion severity label y(t) as a four-class ordinal variable derived from v(t). In our experiments, the model uses sensor history up to time t to predict the next-step severity label y(t + 1). This formulation defines a citywide congestion severity index rather than per-sensor labels, which simplifies evaluation and supports robustness analysis.

$$\mathrm{y}\left(\mathrm{t}\right)\in \left\{\mathrm{L}\mathrm{o}\mathrm{w},\mathrm{M}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e},\mathrm{H}\mathrm{i}\mathrm{g}\mathrm{h},\mathrm{S}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{e}\right\}$$

(4)

3.3.2. Thresholding Strategy

To avoid arbitrary manual thresholds and to ensure adaptability across different sensors, we adopt a quantile-based thresholding strategy. Specifically, the thresholds are computed using the empirical distribution of speed values in the training set only, ensuring that label construction does not leak information from validation or test splits.

Let Q0.25, Q0.50, and Q0.75 denote the 25th, 50th, and 75th percentiles of the training speed distribution, respectively. The congestion severity label is then defined as:

$$y\left(t\right)=\{\begin{array}{c}Severe , v\left(t\right)\le Q^0{.}^{25}\\ High , Q^0{.}^{25}< v\left(t\right)\le Q^0{.}^{50}\\ Moderate , Q^0{.}^{50}< v\left(t\right)\le Q^0{.}^{75}\\ Low , v\left(t\right)> Q^0{.}^{75}\end{array}$$

(5)

Quantile-based thresholds provide a data-driven and reproducible labeling strategy that avoids subjective or city-specific speed cutoffs. This approach also yields relatively balanced class distributions, which is important for stable training and fair comparison across models. While traffic speed varies naturally by time-of-day, the empirical speed distribution implicitly reflects these temporal variations, allowing the quantile-based discretization to capture congestion severity relative to typical operating conditions in the dataset.

This approach provides a balanced partitioning of the speed data into four classes and supports robust evaluation under varying traffic conditions. Similar discretization strategies have been used in smart city and traffic analytics when categorical decision outputs are required for operational response and monitoring [7].

The four congestion severity classes and their quantile-based speed thresholds (computed on the training split) are defined in

Table 3. Congestion severity label definition and thresholds.

Congestion Severity	Speed Range (Quantile-Based)	Interpretation
Low	v(t) > Q0.75	Free-flow traffic; low congestion
Moderate	Q0.50 < v(t) ≤ Q0.75	Mild congestion; minor slowdown
High	Q0.25 < v(t) ≤ Q0.50	Heavy congestion; sustained slowdown
Severe	v(t) ≤ Q0.25	Severe congestion; critical slowdown

.

Figure 3 shows the distribution of traffic speeds and the derived intensity thresholds.

Speed distribution in the training set with quantile thresholds used to define four congestion severity classes.

3.4. Preprocessing

This section describes preprocessing procedures for both sensor time-series data and textual reports. Given the heterogeneous nature of the two modalities, preprocessing is performed independently to ensure that each input is represented in a consistent and model-ready format before alignment and fusion.

3.4.1. Sensor Preprocessing

Traffic sensor data in METR-LA are organized as regularly sampled time-series streams. The following preprocessing steps are applied:

• Data cleaning: Sensor readings are checked for missing entries or invalid values (e.g., negative speeds). Invalid values are treated as missing.
• Imputation: Missing values are handled using linear interpolation across time for each sensor station. This method preserves temporal continuity and is commonly adopted in traffic time-series modeling.

Handling gaps and sensor outages: METR-LA contains occasional missing values. In this experimental setup, we apply linear interpolation within each sensor stream to impute missing readings and keep all 207 sensors. We do not apply additional sensor exclusion rules; therefore, if a sensor exhibits long missing segments, its interpolated values may be less reliable. This is a standard practical trade-off and is reported explicitly for reproducibility.

Linear interpolation is commonly adopted in traffic time-series modeling when missing intervals are relatively short, as it preserves temporal continuity without introducing abrupt artifacts. Given the moderate frequency of missing values in METR-LA and the objective of maintaining all 207 sensors for robustness analysis, interpolation provides a practical trade-off between data completeness and modeling simplicity.

3.

Normalization: Each sensor’s speed values are normalized using z-score normalization computed from the training split:

$$v^{\prime }\left(t\right)={\displaystyle \frac{\mathrm{v}\left(\mathrm{t}\right)-\mathsf{\mu }}{\sigma }}$$

(6)

where μ and σ are the mean and standard deviation of the training data for the corresponding sensor.

4.

Sliding window construction: For each target time step t, we construct a historical window of length L time steps:

$${\mathrm{X}}_{\mathrm{s}}\left(\mathrm{t}\right)=\left\{{\mathrm{x}}_s\left(\mathrm{t}-\mathrm{L}+1\right),\dots ,{\mathrm{x}}_{\mathrm{s}}\left(\mathrm{t}\right)\right\}$$

(7)

In this study, we set L = 12, corresponding to one hour of sensor history at a 5 min sampling interval. This window length provides sufficient short-term temporal context while maintaining computational efficiency.

These steps produce standardized sensor sequences suitable for temporal encoding with recurrent or attention-based models [3,11].

3.4.2. Text Preprocessing

Textual urban reports are naturally noisy and contain informal language, abbreviations, and inconsistent formatting. We apply the following preprocessing steps:

• Cleaning rules:

-

Convert text to lowercase.

-

Remove URLs, email addresses, and excessive whitespace.

-

Remove non-informative characters (e.g., repeated punctuation).

-

Keep alphanumeric tokens and common separators.

Text field selection: Each NYC 311 record contains multiple structured text fields (e.g., Complaint Type and Descriptor). In this study, we construct a single input string per report by concatenating the Complaint Type and Descriptor fields (separated by a delimiter). We do not use location identifiers as text input. Beyond restricting the time span and removing empty/very short texts, we do not apply aggressive topic filtering; therefore, the text stream may include non-traffic-related reports. We treat this as an experimental constraint and discuss it as a key limitation and direction for future work.

2.

Tokenization: Text reports are tokenized using the tokenizer of the pretrained language model bert-base-uncased (BERT-base).

3.

Maximum sequence length: Each report is truncated or padded to a maximum length of 128 tokens, which balances coverage and computational cost.

4.

Embedding extraction (BERT): The textual representation is obtained using a pretrained BERT encoder. Given a tokenized report ${\mathrm{r}}_i$, we extract its embedding $h_{t,i}$ as:

$${\mathrm{h}}_{t,i}={\mathrm{f}}_{\mathrm{t}}\left(r_i\right)$$

(8)

where ${\mathrm{f}}_{\mathrm{t}}\left(·\right)$ denotes the BERT encoder (bert-base-uncased). We use the pooled [CLS] representation as a sentence-level embedding. In our experiments, BERT is used as a frozen feature extractor (no fine-tuning) to reduce computational cost.

Using a frozen BERT encoder ensures stable and reproducible text representations while significantly reducing training complexity and computational overhead. Given the experimental focus on alignment and fusion mechanisms rather than large-scale language model fine-tuning, this choice provides a controlled and computationally efficient setup.

For time steps with multiple aligned reports, embeddings can be aggregated by mean pooling to form a single fixed-length textual context vector (used in the text-only and early/late fusion baselines). For the cross-attention fusion model, we retain the set of per-report embeddings within the alignment window and apply attention over the report set (Section 3.6).

$$\overline{h_t}={\displaystyle \frac{1}{\left| T^{\left(t\right)}\right|}} \sum _{r_i \in T^{\left(t\right)}}{f}_t\left(r_i\right)$$

(9)

The preprocessing steps applied to sensor time-series and textual reports are summarized in

Table 4. Preprocessing configuration for sensors and text.

Component	Step	Configuration	Rationale
Sensor	Missing values	Linear interpolation	Preserves temporal continuity
Sensor	Normalization	z-score (train split only)	Prevents leakage; stabilizes training
Sensor	Windowing	L = 12 steps (1 h)	Captures short-term dynamics efficiently
Text	Cleaning	lowercase, remove URLs, normalize spaces	Reduces noise and non-semantic tokens
Text	Tokenization	BERT tokenizer	Consistent with pretrained encoder
Text	Max length	128 tokens	Balanced coverage vs cost
Text	Embedding	BERT pooled output ([CLS])	Captures sentence-level semantics
Text	Multi-report aggregation	Mean pooling	Stable fixed-size context per time step

, highlighting the key configuration choices used throughout the experiments.

3.5. Temporal Alignment Strategy

A central challenge in integrating sensor streams with textual reports is the mismatch in temporal structure: traffic sensors provide regularly sampled time-series, whereas textual reports arrive irregularly and may not exist at every prediction time step. Therefore, a temporal alignment mechanism is required to associate text records with sensor time steps in a consistent and reproducible manner [7].

In this work, alignment is performed using a configurable time-window association strategy. For each sensor time step t, we collect all textual reports whose timestamps fall within a symmetric window around t. This design is simple, practical, and suitable for real smart city systems where reports may occur slightly before or after a measurable effect appears in sensors.

3.5.1. Alignment Window Definition

Let time (t) denote the real timestamp corresponding to sensor time index t. Let each textual report $r_i$ have an associated timestamp $\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\left({\mathrm{r}}_i\right).$ Given an alignment window Δ, we define the set of aligned reports for time step t as:

$$\mathrm{T}\left(\mathrm{t}\right)=\left\{{\mathrm{r}}_i\right| \left|\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\left({\mathrm{r}}_i\right)-\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\left(\mathrm{t}\right)\right|\le \mathsf{\Delta }\}$$

(10)

In this study, we set Δ = 30 min as the default alignment window. This choice reflects a practical delay tolerance in city reporting platforms and provides a balanced trade-off between capturing relevant contextual reports and introducing unrelated text. Because the paired datasets originate from different cities, the aligned text may be weakly related to the sensor conditions; therefore, Δ should be interpreted as an experimental hyperparameter rather than a calibrated real-world reporting delay.

3.5.2. Construction of a Fixed-Length Text Context per Time Step

Because the number of aligned reports varies over time, we represent the aligned text either as (i) a mean-pooled fixed-length vector $\overline{h_t}$ for baselines, or (ii) a set of per-report embeddings $H_t$ for the cross-attention model. We apply the following strategy:

• For each report ${\mathrm{r}}_i\in \mathrm{T}\left(\mathrm{t}\right)$, compute its embedding $h_{t,i}={\mathrm{f}}_{\mathrm{t}}\left({\mathrm{r}}_i\right)$ using the BERT encoder (Section 3.4.2).
• Aggregate the embeddings into a single vector using mean pooling:

  $$\overline{h_t}={\displaystyle \frac{1}{\left| T^{\left(t\right)}\right|}} \sum _{r_i \in T^{\left(t\right)}}{h}_{t,i}$$

  (11)

This produces a stable and noise-tolerant representation of the textual context around time step t. Mean pooling is computationally efficient and empirically robust for aggregating multiple short reports [8].

3.5.3. Handling Missing Text Reports (No-Text Cases)

Text reports may be absent for many sensor time steps, especially when the reporting stream is sparse. To ensure that the model can operate under such conditions, we define a no-text handling mechanism:

If $\mathrm{T}\left(\mathrm{t}\right)=\mathrm{\varnothing }$, then:

$$\overline{h_t}={\mathrm{h}}_{none}$$

where ${\mathrm{h}}_{none}$ is a learned embedding vector representing “no textual information available”. This approach is preferred over using an all-zero vector because it allows the model to learn a meaningful default representation and reduces bias introduced by systematically missing text [7,8].

Implementation note: if report embeddings are precomputed once per report, the alignment step can be implemented with an efficient two-pointer scan over the sorted timestamps in O (N + M) time.

Figure 4 illustrates an example of time synchronization between sensor readings and text reports.

Temporal alignment of irregular text reports to a sensor time step using a symmetric window of ±Δ minutes. Reports within the window are encoded and aggregated to form a fixed-length context vector $\overline{h_t}$; if no reports exist, a learned no-text embedding is used.

3.6. Proposed Multi-Modal Fusion Model

This section presents the proposed multi-modal model designed to integrate traffic sensor time-series and aligned textual context for congestion severity prediction. The model consists of: (i) a sensor encoder to capture temporal traffic dynamics, (ii) a text encoder to extract semantic representations from urban reports, (iii) a cross-attention fusion module to model cross-modal interactions, and (iv) a prediction head to output congestion severity classes. The overall architecture is illustrated in Figure 5.

3.6.1. Sensor Encoder

Traffic sensors generate dense time-series data sampled at regular intervals. To encode the temporal dynamics, we adopt a recurrent neural network (GRU) as the default sensor encoder due to its efficiency and strong performance in sequential modeling. Given a historical sensor window ${\mathrm{X}}_{\mathrm{s}}\left(\mathrm{t}\right)$ (Section 3.1), the sensor encoder produces a compact representation:

$${\mathrm{h}}_{\mathrm{s}}^{\left(t\right)}={\mathrm{f}}_s\left({\mathrm{X}}_s\left(\mathrm{t}\right)\right)$$

(12)

where ${\mathrm{f}}_{\mathrm{s}}\left(·\right)$ denotes the GRU-based temporal encoder and ${\mathrm{h}}_{\mathrm{s}}^{\left(t\right)}\in {\mathrm{R}}^{d_s}$ is the sensor embedding at time step t. In practice, ${\mathrm{h}}_{\mathrm{s}}^{\left(t\right)}$ is obtained from the final hidden state of the GRU.

We select GRU for its computational efficiency under real-time constraints; however, the framework supports alternative temporal encoders such as temporal Transformers.

3.6.2. Text Encoder

Text reports associated with each sensor time step are encoded using a pretrained language model. For each aligned repor $r_i \in T\left(t\right)$, we compute a per-report embedding using BERT:

$${\mathrm{h}}_{t,i}={\mathrm{f}}_{\mathrm{t}}\left(r_i\right), \mathrm{f}\mathrm{o}\mathrm{r} r_i\in \mathrm{T}\left(\mathrm{t}\right)$$

(13)

where ${\mathrm{f}}_{\mathrm{t}}\left(·\right)$ denotes the BERT-based text encoder (bert-base-uncased). Each report embedding ${\mathrm{h}}_{t,i}\in {\mathrm{R}}^{d_t}$. For baselines that require a single fixed-length text vector per time step, we apply mean pooling to obtain $\overline{h_t}\in R^{d_t}$ (Section 3.5). For the cross-attention fusion model, we retain the report set ${\mathrm{H}}_{\mathrm{t}}=\left[{\mathrm{h}}_{t,1},\dots ,{\mathrm{h}}_{t,n_t}\right]$ and attend over the report embeddings (Section 3.6).

We treat the aligned text at time t as a set of ${\mathrm{n}}_t$ report embeddings $H_t$, where each element corresponds to one report.

3.6.3. Cross-Attention Fusion Module (Proposed)

Simple concatenation-based fusion may fail to capture interaction patterns between traffic dynamics and textual semantics. To address this limitation, we adopt a cross-attention fusion design in which the sensor embedding acts as a query and attends over the set of aligned report embeddings ${\mathrm{H}}_{\mathrm{t}}$. This enables the model to weight multiple aligned reports differently (when available) and to down-weight noisy or irrelevant reports conditioned on the current traffic state.

We project the sensor and text embeddings into a shared attention space of dimension d and compute single-head dot-product attention over the report set:

$$\mathrm{Q}={\mathrm{W}}_{\mathrm{q}}{\mathrm{h}}_{\mathrm{s}}\left(\mathrm{t}\right), K_i={\mathrm{W}}_{\mathrm{k}}{\mathrm{h}}_{t,i}, V_i={\mathrm{W}}_{\mathrm{v}}{\mathrm{h}}_{t,i} \mathrm{f}\mathrm{o}\mathrm{r} \mathrm{i},1,\dots ,{\mathrm{n}}_{\mathrm{t}}$$

(14)

where ${\mathrm{W}}_{\mathrm{q}}\in {\mathrm{R}}^{d \times d_s}$, ${\mathrm{W}}_{\mathrm{k}}\in {\mathrm{R}}^{d \times d_t},$ and ${\mathrm{W}}_{\mathrm{v}}\in {\mathrm{R}}^{d\times d_t}$ are trainable projection matrices, and ${\mathrm{n}}_{\mathrm{t}}$ is the number of reports aligned to time step t (Algorithm 1).

Algorithm 1: Temporal alignment of irregular text reports with sensor time steps. Temporal alignment procedure (time-window association).

- Input: sensor time steps $\left\{{\mathrm{t}}_1,\dots ,{\mathrm{t}}_N\right\}$ with timestamps time(t); text reports $\left\{{\mathrm{r}}_1,\dots ,{\mathrm{r}}_{\mathrm{M}}\right\}$ with timestamps time $\left(r_i\right)$ alignment window Δ; text encoder ${\mathrm{f}}_{\mathrm{t}}\left(·\right)$ learned “no-text” embedding ${\mathrm{h}}_{\mathrm{n}\mathrm{o}\mathrm{n}\mathrm{e}}$. - Output: for each sensor time step t, a list of aligned report embeddings ${\mathrm{H}}_{\mathrm{t}}=[{\mathrm{h}}_{t,1},\dots ,{\mathrm{h}}_{t,n_t} (\mathrm{or} [{\mathrm{h}}_{\mathrm{n}\mathrm{o}\mathrm{n}\mathrm{e}}]$ if no reports). A mean-pooled text vector $\overline{h_t}$ can be computed from Ht when required (e.g., for early fusion baselines). - Steps: ○ Sort text reports by timestamp. ○ For each sensor time step t (chronological order), collect all reports $r_i$ such that $\left|\mathrm{t}ime\left(r_i\right)-\mathrm{t}ime(t\right)|\le \mathsf{\Delta }$. ○ If no reports exist for t, set ${\mathrm{H}}_{\mathrm{t}}\leftarrow \left[{\mathrm{h}}_{\mathrm{n}\mathrm{o}\mathrm{n}\mathrm{e}}\right]$. Otherwise, compute embeddings for each report in the window: ${\mathrm{h}}_{t,i}\leftarrow {\mathrm{f}}_{\mathrm{t}}\left(r_i\right)$. ○ (Optional for baselines) Compute $\overline{h_t}\leftarrow \mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}\left({\mathrm{H}}_{\mathrm{t}}\right)$.                                   Return {${\mathrm{H}}_{\mathrm{t}}$} (and $\overline{h_t}$ when used).

The attention weights and attended text representation are computed as:

$${\mathsf{\alpha }}_{\mathrm{i}}\left(\mathrm{t}\right)={\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{i}}\left({\displaystyle \frac{{\mathrm{Q}}^{\mathrm{T}}{\mathrm{K}}_{\mathrm{i}}}{\sqrt{\mathrm{d}}}}\right),\mathrm{i}=1,\dots ,{\mathrm{n}}_{\mathrm{t}}$$

(15)

$${\mathrm{a}}^{\left(t\right)}={\mathsf{\Sigma }}_{i=1}^{n_t} {\alpha }_i\left(\mathrm{t}\right) V_i$$

where the softmax is computed over $i = 1,\dots ,n_t$.

(When no reports exist, we set ${\mathrm{H}}_{\mathrm{t}}=\left[{\mathrm{h}}_{\mathrm{n}\mathrm{o}\mathrm{n}\mathrm{e}}\right]$, hence ${\mathrm{n}}_{\mathrm{t}}=1$.)

where a (t) ∈ R^d represents the attended textual signal given the sensor query. This mechanism provides a structured way to aggregate multiple aligned reports; however, its benefit depends on the relevance and quality of the aligned text.

Finally, we construct the fused representation by concatenating the sensor embedding with the attended textual embedding:

$${\mathrm{h}}_{\mathrm{f}}^{\left(t\right)}=\left[{\mathrm{h}}_{\mathrm{s}}^{\left(t\right)};{\mathrm{a}}^{\left(t\right)}\right]$$

(16)

This fusion mechanism conditions textual aggregation on the current sensor state to model cross-modal interactions. Its effectiveness depends on the relevance of the aligned textual reports and the alignment quality, particularly in sparse/noisy settings [8].

3.6.4. Prediction Head and Training Objective

The fused representation ${\mathrm{h}}_{\mathrm{f}}\left(\mathrm{t}\right)$ is fed into a multilayer perceptron (MLP) followed by a softmax layer to predict the next-step congestion severity class:

$$\mathrm{y}\left(\mathrm{t}+1\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_{\mathrm{o}}·\mathsf{\varphi }\left({\mathrm{h}}_{\mathrm{f}}\left(\mathrm{t}\right)\right)+{\mathrm{b}}_{\mathrm{o}}\right)$$

(17)

where ϕ(·) denotes the MLP transformation with a non-linear activation, and ${\mathrm{W}}_{\mathrm{o}},{\mathrm{b}}_{\mathrm{o}}$ are trainable parameters.

The model is trained using the standard cross-entropy loss:

$$L= - \sum _{c=1}^4{y}_c^{\left(t\right)}log{(\widehat{y_c}}^{\left(t\right)})$$

(18)

where y(t + 1) is the one-hot encoded ground truth label.

3.7. Baseline Models

To validate the effectiveness of the proposed cross-attention fusion architecture, we compare it against multiple baseline models that represent common uni-modal and multi-modal integration strategies. These baselines are designed to isolate the contributions of (i) each modality individually and (ii) the fusion mechanism. Such comparisons follow established evaluation practices in multimodal learning and urban data fusion research [7,8].

3.7.1. Sensor-Only Baseline

This baseline uses only the traffic sensor time-series input ${\mathrm{X}}_{\mathrm{s}}\left(\mathrm{t}\right)$. The sensor window is encoded using the same GRU encoder employed in the proposed model, followed by an MLP classifier. This provides a strong reference that reflects traditional sensor-based congestion modeling.

3.7.2. Text-Only Baseline

This baseline uses only the aligned textual context T(t). Each report is encoded using BERT, and report embeddings within the alignment window are aggregated by mean pooling to form a fixed-length text vector $\overline{h_t}$. An MLP classifier then predicts congestion severity. This baseline evaluates the predictive signal available from text alone.

3.7.3. Early Fusion Baseline (Concatenation)

Early fusion is a common baseline in multimodal learning, where modality embeddings are concatenated and directly passed to a classifier. Specifically, the sensor embedding $h_s\left(t\right)$ and the mean-pooled text embedding $\overline{h_t}$ are concatenated:

$${\mathrm{h}}_{\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{l}\mathrm{y}}\left(\mathrm{t}\right)=\left[{\mathrm{h}}_{\mathrm{s}}\left(\mathrm{t}\right);\overline{h_t}\right]$$

(19)

And an MLP classifier predicts the severity label. This baseline assesses whether simple fusion is sufficient without explicit cross-modal interaction modeling.

3.7.4. Late Fusion Baseline (Ensemble)

Late fusion combines decisions from independent models trained separately on each modality. We train the sensor-only model and text-only model independently, then fuse their outputs at the decision level by averaging class probabilities:

$${\mathrm{ŷ}}_{\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e}}(\mathrm{t}+1)=1/2\left({\mathrm{ŷ}}_{\mathrm{s}}\right(\mathrm{t}+1)+{\mathrm{ŷ}}_{\mathrm{t}}(\mathrm{t}+1).$$

(20)

For fair comparison, we evaluate the proposed approach against standard uni-modal and multi-modal baselines, which are summarized in

Table 5. Baseline models and fusion strategies.

Model ID	Baseline	Modalities Used	Core Encoder(s)	Fusion Strategy	Description
B1	Sensor-only	Sensor	GRU	None	Sensor window → GRU → MLP → Softmax
B2	Text-only	Text	BERT	None	Aligned text → BERT → pooling → MLP → Softmax
B3	Early Fusion	Sensor + Text	GRU + BERT	Concatenation	$\left[{\mathrm{h}}_{\mathrm{s}}\left(\mathrm{t}\right);\overline{h_t}\right]$ → MLP → Softmax
B4	Late Fusion	Sensor + Text	GRU + BERT	Decision-level averaging	Avg (ŷs, ŷt)
P (Ours)	Cross-Attention Fusion	Sensor + Text	GRU + BERT	Cross-attention + concat	Sensor query attends over report set ${\mathrm{H}}_{\mathrm{t}}\to \mathrm{a}\left(\mathrm{t}\right);$$\mathrm{then} \left[{\mathrm{h}}_{\mathrm{s}}\left(\mathrm{t}\right);\mathrm{a}\left(\mathrm{t}\right)\right]$ → MLP

.

4. Experimental Setup

This section describes the experimental protocol used to train and evaluate the proposed model and baseline methods. The setup is designed to ensure reproducibility and fair comparison across models.

4.1. Data Splitting

The traffic sensor time-series is split chronologically to avoid temporal leakage, following standard practice in time-series learning. We partition the data into:

• Training set: 70%
• Validation set: 10%
• Test set: 20%

All preprocessing statistics (normalization parameters and quantile thresholds for label construction) are computed using the training set only (Section 3.3 and Section 3.4). The NYC 311 text records are restricted to the same temporal window as METR-LA and are aligned to sensor time steps using the procedure described in Section 3.5.

4.1.1. Input Configuration

• Sensor window length: L = 12 (1 h history with 5 min sampling)
• Alignment window: Δ = 30 min (default)
• Text maximum length: 128 tokens
• Number of classes: 4 (Low, Moderate, High, Severe)

4.1.2. Training Details

All models are trained using the Adam optimizer with early stopping based on validation Macro-F1 (patience = 5 epochs; the best checkpoint is selected by validation Macro-F1). Each experiment is repeated three times with different random seeds, and mean results are reported to reduce variance.

We use fixed random seeds for reproducibility and keep all hyperparameters identical across baselines to ensure fair comparison. The default settings (Δ = 30 min, L = 12) were selected based on validation performance and retained across all experiments.

The experimental protocol and model hyperparameters used for training and evaluation are reported in

Table 6. Experimental settings and hyperparameters.

Category	Parameter	Value
Data	Sensor window length L	12-time steps (1 h)
Data	Sampling interval	5 min
Data	Alignment window Δ	30 min
Data	Number of Severity classes	4
Text	Max token length	128
Model	Sensor encoder	GRU (hidden size = 64)
Model	Text encoder	BERT-base (bert-base-uncased), pooled [CLS], frozen
Model	Fusion	Cross-attention (single-head, report-level) + concat
Model	MLP head	2 layers (64 → 32 → 4)
Training	Optimizer	Adam
Training	Learning rate	1 × 10−4
Training	Batch size	64
Training	Epochs	30 max (early stopping, patience = 5)
Training	Dropout	0.2
Evaluation	Runs	3 seeds, report mean
Model	Attention dimension d	64
Reproducibility	Report sampling	Fixed subset of 120,000 reports (uniform over time, fixed seed)

.

Hyperparameters were selected based on validation performance and kept fixed across all compared models.

4.2. Evaluation Metrics

To evaluate congestion severity prediction performance, we adopt standard classification metrics that capture both overall correctness and class-balanced performance.

• Accuracy: the fraction of correctly predicted instances:

  Accuracy = correct predictions/total predictions
• Macro-F1 (primary): the unweighted mean of class-wise F1 scores, which is robust to potential class imbalance:

  $$\left\{Macro-F1\right\}={\displaystyle \frac{1}{C}}\sum _{c=1}\mathrm{F}1\mathrm{c}$$

  (21)
• Precision and Recall: reported as macro-averaged scores across classes.

Because congestion severity may exhibit imbalance across labels, Macro-F1 is used as the primary metric for model comparison. Additionally, inference efficiency is discussed qualitatively in Section 5 as a practical consideration for smart city deployment [7,8].

5. Results

This section reports the empirical evaluation of the multimodal framework for traffic congestion severity prediction. We compare fusion strategies against unimodal baselines and conduct an ablation analysis to examine the contribution of key components. We also discuss robustness considerations related to the alignment window Δ and sensor history length Lqualitatively (without claiming a full sensitivity sweep in this experimental setting).

5.1. Main Quantitative Results

Table 7. Main performance comparison on the test set.

Model	Modalities	Accuracy	Macro-F1	Macro-Precision	Macro-Recall
B1: Sensor-only	Sensor	0.8113	0.8072	0.8135	0.8023
B2: Text-only	Text	0.2257	0.0921	0.0564	0.2500
B3: Early Fusion (Concat)	Sensor + Text	0.8283	0.8231	0.8296	0.8187
B4: Late Fusion (Ensemble)	Sensor + Text	0.8063	0.8038	0.8167	0.7965
Proposed: Cross-Attention Fusion	Sensor + Text	0.8113	0.8072	0.8161	0.8023

reports the main test performance comparison among all models. Overall, sensor-based models provide strong predictive capability due to the dense temporal structure of traffic observations. Text-only performance is notably lower, reflecting the sparsity and noise of citizen reports and the weak cross-modal correspondence in the cross-city pairing. Multi-modal early fusion improves performance over the sensor-only baseline, while the cross-attention fusion model does not outperform the sensor-only baseline in our experiments.

All baselines are implemented using the same data splits, preprocessing, and evaluation protocol, and the same GRU/BERT encoders where applicable to ensure comparability.

Interpretation. The results indicate that the sensor modality remains the primary source of predictive power, while textual reports alone are insufficient for reliable congestion severity classification. Multi-modal early fusion provides measurable benefits, particularly in Macro-F1. The cross-attention fusion model performs similarly to the sensor-only baseline, suggesting that attention-based interaction modeling may require stronger modality relevance (e.g., same-city spatiotemporal alignment or traffic-specific text filtering) to provide gains [7,8].

5.2. Ablation Study

To assess the impact of each component, we conduct ablation experiments by removing modalities and comparing fusion strategies (

Table 8. Ablation study results.

Variant	Text	Sensor	Alignment	Fusion	Macro-F1	Δ vs. Best Fusion
Best Fusion (Concat)	✓	✓	✓	Concatenation	0.8231	—
− Text (Sensor-only)	✗	✓	—	None	0.8072	−0.0159
− Sensor (Text-only)	✓	✗	✓	None	0.0921	−0.7310
Late Fusion (Ensemble)	✓	✓	✓	Avg probs	0.8038	−0.0193
Proposed (Cross-Attention)	✓	✓	✓	Cross-attention	0.8072	−0.0159

Note: ✓/✗ denote presence/absence of a component; — indicates not applicable; Δ denotes performance difference relative to the best fusion model.

). Removing the sensor modality causes a sharp decrease in performance, confirming that traffic sensors capture the dominant congestion dynamics. Removing text reduces performance relative to the best multimodal setting (early fusion/concatenation), indicating that the text stream can provide complementary information when combined at the representation level. In contrast, the cross-attention fusion variant performs similarly to the sensor-only baseline in this experimental setting, suggesting that interaction-based fusion may not provide gains when the aligned text signal is weak or noisy.

6. Discussion

This section discusses the empirical findings of the proposed multi-modal framework for traffic congestion severity prediction. The results confirm that dense traffic sensor time-series remain the dominant source of predictive power, while textual urban reports provide auxiliary contextual signals that can enhance performance when properly aligned and fused. The observed trends provide practical insights into how multimodal learning can support smart city operations under real-world constraints such as irregular reporting, missing text, and heterogeneous data quality.

The main quantitative results indicate that sensor-only prediction achieves strong performance, which is consistent with the continuous and high-frequency nature of traffic measurements. In contrast, text-only performance remains substantially lower, reflecting the sparsity, noise, and indirect relationship between citizen-generated reports and real-time traffic dynamics. Importantly, early fusion improves overall performance, particularly in Macro-F1, suggesting more balanced classification across congestion severity levels. These gains suggest that textual signals can provide a complementary context in which they can function as a complementary “urban context” that helps to explain or predict disturbances that may not be fully captured by sensor streams alone [7,8].

Among the evaluated fusion strategies, early fusion (concatenation) yields the strongest improvement in the reported experiments. This outcome suggests that, under sparse textual conditions, a stable representation-level integration of sensor and text embeddings is sufficient to deliver reliable gains. Although the cross-attention fusion design is conceptually motivated and provides a structured way to aggregate multiple aligned reports (Section 3.6), it does not outperform the sensor-only baseline in this experimental setting. A likely explanation is the weak cross-modal correspondence introduced by the cross-city pairing and the broad topical coverage of 311 reports, which can cause the model to down-weight text or to learn minimal additional signal beyond sensors. This observation aligns with multimodal fusion literature, where the advantage of interaction-based mechanisms typically increases when the auxiliary modality contains higher-quality and more directly relevant information [8].

From a smart city deployment perspective, the findings support multimodal pipelines that treat textual reports as auxiliary signals for decision support rather than as a primary predictor. In practice, traffic management systems often require semantic explanations for observed congestion (e.g., construction, road disruption, or service issues), and same-city textual streams can provide such cues. In this study, however, interpretability claims are necessarily limited by the cross-city pairing. Regarding real-time feasibility, GRU-based sensor encoding supports efficient streaming inference, while the BERT encoder is used in feature-extraction mode and can be precomputed offline per report to reduce online latency. We do not claim measured end-to-end latency in this experimental study; instead, we discuss computational considerations qualitatively based on model structure and dominant costs (BERT encoding).

This research has limitations. First, multimodal integration is performed experimentally using open datasets from different cities, which enables controlled evaluation of alignment and fusion under realistic sparsity and noise conditions but does not provide direct geo-spatial or event-level correspondence between textual reports and sensor locations. Second, the temporal alignment strategy relies on a fixed ±Δ window, which introduces uncertainty because not all reports within the window are necessarily relevant to the sensor time step. Third, NYC 311 reports cover a wide range of urban topics and, beyond basic preprocessing, the current experimental setup does not perform traffic-specific topic filtering; this can dilute cross-modal signal. Finally, for computational reasons we subsample the text stream (120,000 reports), which may alter report density patterns. These factors limit interpretability and help explain why interaction-based fusion does not provide gains over sensor-only performance in the reported experiments [7].

We therefore interpret the multimodal setting as a robustness test under weak auxiliary text rather than evidence of strong cross-modal causality.

Future research can address these limitations by integrating spatial modeling and stronger cross-modal alignment. Incorporating road-network graphs or spatial adjacency structures may better capture how congestion propagates across connected road segments. Additionally, unified multimodal datasets collected within a single city—where sensors and reports are geo-temporally aligned—would enable stronger validation of cross-attention interaction modeling and more realistic incident-level case studies. Finally, recent advances in large language models suggest opportunities for improving semantic understanding and filtering of urban reports; lightweight LLM-assisted categorization or summarization could reduce noise and improve relevance before fusion, potentially increasing the effectiveness of attention-based multimodal learning in real-world smart city deployments [35,36,37].

Future work may extend this framework to highway scenarios with reduced sensor coverage or sparse sensing infrastructure, where textual reports and human-generated signals may play a more influential role. Such settings could better highlight the benefits of multimodal fusion when sensor density is limited.

7. Conclusions

This paper demonstrates that integrating textual urban reports with traffic sensor streams is feasible within an end-to-end multimodal pipeline. The experimental results show that early fusion provides the most consistent performance gains, while cross-attention fusion remains sensitive to the strength and relevance of aligned textual signals. A key strength of the proposed framework is its reproducible design, including explicit temporal alignment and systematic baseline comparisons. However, the study is limited by the cross-city pairing of datasets and the potential inclusion of non-traffic-related reports, which may weaken cross-modal signal quality. From a practical perspective, the findings suggest that multimodal fusion is most beneficial when textual streams are semantically relevant and reliably aligned with sensor measurements, emphasizing the importance of dataset design and modality relevance in real-world smart city deployments.