MMHFormer: Multi-Source and Multi-View Hierarchical Transformer for Traffic Flow Prediction

Abstract

Traffic flow prediction is a vital component of Intelligent Transportation Systems (ITSs), playing a key role in proactive traffic management and the optimization of urban mobility. However, the complex spatial–temporal dependencies, dynamic variations, and external factors in traffic networks present significant challenges for accurate predictions. In this paper, we propose MMHFormer, a novel multi-source, multi-view hierarchical Transformer model specifically designed for traffic flow prediction. MMHFormer incorporates three key innovations: (1) a multi-source gated embedding layer that integrates diverse multidimensional inputs, including spatial Laplacian embeddings, temporal periodic embeddings, and traffic occupancy embeddings, to better capture the complex dynamics of traffic conditions; (2) a hierarchical multi-view spatial attention module that models global, local, and dynamic similarity-based spatial dependencies, effectively addressing the spatial heterogeneity of traffic flows; (3) a hierarchical two-stage temporal attention mechanism that captures global temporal dependencies while adapting to node-specific temporal variations. Extensive experiments conducted on four benchmark traffic datasets demonstrate that MMHFormer outperforms state-of-the-art methods, achieving significant improvements in prediction accuracy.

1. Introduction

Traffic flow prediction is one of the core technologies of Intelligent Transportation Systems (ITSs). By analyzing historical data and utilizing prediction models, it helps traffic managers forecast future traffic volume and road conditions [1]. With accurate traffic predictions, traffic management departments can develop strategies for road condition optimization, plan and navigate in advance, thereby improving traffic operational efficiency, ensuring travel safety, and optimizing urban transportation networks [2]. However, due to the complex temporal and spatial dependencies of traffic flow, achieving accurate predictions remains challenging.

Traffic flow data, typically collected from road sensors as time-series data, has traditionally been analyzed using linear methods such as Autoregressive Integrated Moving Average (ARIMA) [3], Vector Autoregression (VAR) [4], and Support Vector Regression (SVR) [5]. However, these methods struggle to effectively capture the nonlinear characteristics of traffic patterns and the spatial attributes of the transportation network, limiting their application in modern traffic flow prediction. With advancements in deep learning techniques and improvements in hardware capabilities, an increasing number of neural network-based models have been employed to more effectively capture the dynamic temporal and spatial dependencies in traffic flow prediction. In the temporal dimension, Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) [6] and Gated Recurrent Units (GRU) [7], are employed to capture the dynamic temporal features of traffic flow. In the spatial dimension, some researchers utilize Convolutional Neural Networks (CNNs) to partition the road network into distinct grids to extract spatial correlations. Researchers have combined these models to extract spatial–temporal features [8]. However, this approach is only applicable to grid-like Euclidean spatial graphs, while traffic road networks are typically non-Euclidean. Subsequently, Graph Neural Networks (GNNs) have been shown to be more suitable for modeling non-Euclidean traffic road networks [9]. For instance, Zhao et al. proposed the T-GCN model [10], which combines Graph Convolutional Networks (GCNs) and GRU to simultaneously capture both spatial and temporal dependencies in traffic data, achieving significant results.

Despite the significant achievements in exploring the spatial–temporal features of traffic networks, several challenges still remain to be addressed. Firstly, the spatial–temporal features of traffic flow are highly complex and intertwined. In the input embedding stage, previous studies typically input spatial and temporal data into separate spatial and temporal modules for feature extraction, and then fuse the extracted results in subsequent stages. However, this approach fails to effectively integrate spatial–temporal features, leading to insufficient capture and utilization of deeper information correlations. Additionally, existing multidimensional embedding methods have notable limitations, particularly the lack of effective fusion strategies. These methods typically treat data from different dimensions equally, without considering the relative importance and interdependencies of each dimension, thereby limiting the model’s performance to some extent. Moreover, most early models primarily rely on a single data source (such as historical traffic flow data) for predictions, neglecting other factors that significantly influence traffic flow. For example, an increase in traffic occupancy during peak hours may lead to congestion, significantly affecting the distribution of traffic flow within the region. Ignoring these critical influencing factors may prevent the model from fully addressing the complexity of real-world scenarios, thereby reducing its reliability and practical applicability.

Secondly, traffic networks exhibit significant spatial heterogeneity. Although some nodes are spatially adjacent (e.g., different locations on the same road segment), their traffic flow and variation trends may be similar. However, as shown in Figure 1b, even when two nodes are far apart and not on the same road segment, they may still exhibit similar traffic patterns, often driven by shared traffic characteristics or demand patterns. This indicates that spatial correlations in traffic networks are not solely dependent on physical distance. However, numerous studies have demonstrated that GNNs tend to suffer from over-smoothing during node feature aggregation [11], which undermines the model’s ability to capture long-range dependencies and limits its performance in modeling spatial heterogeneity.

Finally, traffic flow exhibits significant temporal heterogeneity. For example, Figure 1c shows that the traffic flow correlation between Location A and Location B is relatively low during the morning peak period, reflecting a disparity in their traffic patterns [12]. However, during the evening peak period, their traffic patterns tend to align. Conversely, as shown in Figure 1d, the traffic flow correlation between Node B and Node C is stronger during the morning peak, but weaker at other times. Furthermore, Figure 1a shows that even at the same location, traffic patterns on weekdays may differ significantly from those on non-working days. Weekday traffic flows exhibit clear periodicity, whereas non-working days lack such regularity, reflecting differences in travel behaviors on different types of days. Notably, given the significant temporal heterogeneity in traffic flow across different periods, a single temporal processing method may be insufficient to comprehensively capture its complex dynamic variations.

To address the aforementioned complex traffic flow scenarios, this paper proposes a model based on the Transformer encoder architecture—MMHFormer. To provide enriched input representations for subsequent spatial–temporal encoding, the input embedding layer utilizes gated convolution to integrate traffic features, temporal characteristics, spatial attributes, and external environmental factors. To capture spatial dependencies among traffic nodes from multiple perspectives, we introduce a hierarchical multi-view spatial attention module that explicitly models both global and local spatial dependencies. Additionally, to address cross-node temporal pattern variations, we propose a hierarchical two-stage temporal attention module. In the first stage, a temporal attention mechanism captures global temporal dependencies. In the second stage, convolution is performed at the node level as a secondary query, followed by an additional round of temporal attention to emphasize traffic pattern differences across nodes. The main contributions of MMHFormer are summarized as follows:

• This paper proposes a novel MMHFormer model, which effectively addresses challenges in multi-source information fusion, multi-view interaction, and dynamic temporal dependency modeling.
• MMHFormer is designed as a unified framework that integrates multi-source feature fusion, multi-view spatial reasoning, and hierarchical temporal modeling. A multi-source gated embedding layer fuses spatial Laplace, time-period, and traffic occupancy embeddings to represent complex traffic conditions. On this basis, a hierarchical multi-view spatial attention mechanism captures global, geospatial, and dynamic-similarity dependencies, while a two-stage temporal attention module learns both global temporal correlations and node-specific dynamics, enabling adaptive modeling of evolving traffic patterns across time and space.
• Experimental results on four real-world traffic datasets show that MMHFormer outperforms current state-of-the-art methods, validating the model’s effectiveness and generalization capability. Furthermore, we conducted a single-step evaluation of the hourly forecast, demonstrating the model’s effectiveness in long-term prediction.

2. Related Works

2.1. Traffic Flow Prediction

Traffic flow prediction is a critical component of intelligent transportation systems, enabling the forecasting of future traffic conditions based on historical data, which significantly aids traffic management and control. Researchers typically use statistical methods, machine learning approaches, and deep learning techniques to predict traffic flow. Traditional statistical models, such as exponential smoothing, ARIMA, VAR, and historical average (HA) [13] approach traffic forecasting by assuming linear dependencies. However, they are less effective at capturing the complex nonlinear variations inherent in traffic flow.

Recognizing the limitations of linear models, researchers shifted their focus to treating traffic prediction as a nonlinear problem. Machine learning techniques, such as support vector machines (SVM) [14], Bayesian networks, autoencoders, and k-nearest neighbors (KNN) [15], were introduced to extract the correlation between data.However, these methods relied on manually crafted features, making feature engineering extremely time-consuming for large datasets, which further limited their effectiveness for large-scale traffic flow prediction.

In recent years, deep learning has gradually become the mainstream method for traffic flow prediction due to its powerful modeling capabilities. For example, Zhang et al. proposed a hybrid deep learning framework combining CNN and LSTM for short-term traffic flow forecasting [15]. The 1D CNN captures spatial features, while LSTM learns short-term variations and periodic features. Liu et al. introduced a module combining convolution and LSTM, referred to as the Conv-LSTM module, to extract spatial–temporal features from traffic flow data [16]. Furthermore, the model includes a Bidirectional LSTM (Bi-LSTM) module to capture the periodic characteristics of traffic flow. Lv et al. proposed the LC-RNN model, which employs a CNN to extract dynamic features from surrounding areas, while an RNN learns long-term temporal patterns [17]. Additionally, LC-RNN integrates periodic information and contextual factors (such as weather and holidays) and adaptively fuses this information to improve prediction accuracy. These models, primarily based on CNNs and RNNs designed for Euclidean data such as grids or sequences, struggle to model traffic networks effectively, as these networks are inherently non-Euclidean and exhibit complex, irregular topologies. Consequently, these methods fail to fully capture the spatial dependencies of road networks, leading to suboptimal performance.

2.2. Graph Convolution Network

Graph Convolutional Networks are designed to analyze non-Euclidean data. Unlike CNNs, which are limited to regular grid-based data, GCNs excel at aggregating feature information from neighboring nodes to compute node representations, effectively leveraging graph topology. Consequently, researchers have applied GCNs to traffic flow prediction to extract spatial features from non-Euclidean traffic networks. For example, Yu et al. proposed the STGCN, which leverages graph convolutional layers to capture spatial topology and temporal convolutional layers to model temporal dynamics, thereby overcoming the limitations of recurrent networks [18]. Guo et al. proposed the ASTGCN [19], which consists of three independent components that model recent, daily, and weekly dependencies in traffic flow, respectively. Each component includes a spatial–temporal attention mechanism to effectively capture dynamic spatial–temporal correlations and a spatial–temporal convolution module that combines graph convolution and standard convolution to capture spatial patterns and temporal features. Although previous studies acknowledge the importance of capturing both spatial and temporal features for traffic forecasting, most methods still rely on predefined adjacency matrices to model spatial correlations, neglecting dynamic features that evolve over time. Therefore, in recent years, some researchers have focused more on the dynamic construction of adjacency matrices to better capture spatial–temporal dependencies between nodes. Wu et al. proposed Graph WaveNet [20], which uses an adaptive dependency matrix to capture hidden spatial dependencies in the data, thereby avoiding reliance on predefined adjacency matrices. In addition, Graph WaveNet employs stacked dilated causal convolutions to capture temporal dependencies, enabling it to handle long time series. For example, Bai et al. proposed the NAPL module and the DAGG module to enhance the capabilities of GCNs [21]. The NAPL module is used to learn node-specific patterns, while the DAGG module automatically generates dependencies between nodes based on the data.

With the advancement of research, the limitations of a single adjacency matrix in capturing complex spatial dependencies have become increasingly apparent. Researchers have begun to explore the use of multiple graph adjacency matrices to more comprehensively capture spatial dependencies. Guo et al. proposed the HGCN [22], which combines micro-level and macro-level graph layers. The micro-level layer represents nodes in a road network and their connections, while the macro-level layer is constructed by clustering the micro-level layer to form larger traffic regions. HGCN uses pooling methods to achieve a hierarchical structure and introduces a dynamic transfer module that facilitates interaction between micro and macro features, thereby enhancing the model’s accuracy and generalization ability in traffic forecasting. Ye et al. proposed the DMGNN [23], which constructs multiple types of spatial–temporal graphs to represent different relationships between road nodes, providing richer prior knowledge and better capturing complex relationships within the traffic system. To dynamically reflect changes between nodes, DMGNN includes a dynamic graph adjustment module that updates the adjacency matrix during each training iteration, allowing the model to adapt to temporal variations in dependency relationships. Yin et al. introduced the M-SDCGCN [24], which incorporates multiple types of graphs (including adaptive, dynamic, and static graphs) to model complex dependencies between road nodes, thus capturing interactions at different levels. The model leverages meta-learning to enhance connections between spatial–temporal features and jointly models both static and dynamic factors. Huang et al. presented the MVDGCN [25], which integrates multi-view encoder-decoder modules, dynamic relationship matrix generation, and coupled graph convolutions to capture spatial–temporal dependencies at different time scales. The model extracts traffic flow patterns from hourly, daily, and weekly perspectives, and uses a dynamic fusion module to integrate these features for more accurate traffic flow forecasting.

Beyond the aforementioned graph structures, several studies have increasingly focused on spatial–temporal modeling of heterogeneous traffic flow with heterogeneous spatial–temporal graphs. Zhong et al. [26] proposed a heterogeneous spatial–temporal graph convolution framework that constructs multiple graphs to jointly encode geographical and dynamic correlations, while simultaneously handling missing traffic data via spatial–temporal completion. Xu et al. [27] built a global heterogeneous traffic spatial–temporal graph and introduced HTSTGC to capture interactions among different traffic elements within a unified framework. More recently, Wu et al. [28] presented MHGNet, a multi-heterogeneous graph neural network that jointly represents multiple heterogeneous spatial–temporal graphs and performs prediction on clustered heterogeneous subgraphs. Collectively, these works underscore the importance of explicitly modeling heterogeneous nodes, edges, and relations in traffic networks. However, they mainly focus on structural heterogeneity at the graph level and rarely integrate rich multi-source traffic attributes or explicitly address temporal heterogeneity.

2.3. Attention Mechanism

Transformers [29], known for their powerful sequence modeling capabilities and multi-head attention mechanisms, have achieved significant success in natural language processing and have recently been introduced into the domain of traffic flow forecasting. Compared to traditional models such as GCNs and RNNs, Transformers have a greater advantage in capturing long-term spatial–temporal dependencies, especially when handling dynamic changes and complex patterns in traffic flow. For instance, Jiang et al. proposed PDFormer [30], which employs a spatial self-attention module to capture dynamic spatial dependencies and combines short-range and long-range graph masks to capture local and remote dependencies. Furthermore, PDFormer introduces a delay-aware feature transformation module to explicitly model the delay characteristics of spatial information propagation. Cai et al. introduced LCDFormer [31], which incorporates a temporal aggregation approach that aggregates and compresses historical data over long time windows, effectively retaining long-term historical information while reducing the impact of redundant data. Additionally, LCDFormer proposes a novel spatial–temporal attention module that combines topology-based local attention and node similarity-based global attention, enabling the model to capture local short-term spatial features as well as to discover long-distance dynamic spatial correlations. Geng et al. presented STGAFormer [32], which adopts a Transformer encoder architecture combined with a gated temporal self-attention module and a distance-based spatial self-attention module to extract complex spatial–temporal features. The gated temporal self-attention module enhances the extraction of both local and global temporal features, while the distance-based spatial self-attention module employs thresholding to selectively extract crucial spatial features. Li et al. proposed DDGFormer [33], which introduces direction and distance-aware self-attention modules to capture relative position and directionality in traffic flow sequences. In addition, the model includes a dynamically enhanced adaptive graph convolutional network module, which improves the capture of dynamic spatial correlations in traffic systems.

Compared with the above existing studies, our proposed MMHFormer offers the following advantages and distinctions: (1) A multi-source gated embedding layer adaptively fuses spatial Laplacian, temporal periodic, and occupancy features, effectively mitigating the limitations of single-source or uniformly weighted embeddings. (2) A hierarchical multi-view spatial attention module explicitly captures global, local, and dynamic-similarity dependencies, enabling comprehensive modeling of spatial heterogeneity. (3) A two-stage temporal attention mechanism jointly models global dependencies and node-specific variations, substantially improving robustness to diverse temporal dynamics.

3. Problem Formalization

Traffic flow is defined as $\mathcal{X}=\{X_1,X_2,\dots ,X_T\}\in R^{T\times N\times C}$, where $X_t\in R^{N\times C}$ represents the observed values at time step t for N nodes with C feature dimensions. Here, $C=1$ corresponds to the traffic flow feature dimension. Our goal is to train a function $\mathcal{F}$ that predicts the sequence for the next Q time steps. This prediction is based on the past P time steps of traffic flow data $\mathcal{X}$. Therefore, the traffic prediction problem can be formulated as:

$$Y_{(t+1)},\dots ,Y_{(t+Q)}=\mathcal{F}({\mathcal{X}}_{(t-P+1)},\dots ,{\mathcal{X}}_t)$$

(1)

4. Methodology

Figure 2 illustrates the framework of MMHFormer, which primarily consists of three modules: the multi-source gated embedding layer, the spatial–temporal encoder layer, and the output layer.

To effectively capture the spatial–temporal features of traffic flow while considering road conditions. The multi-source gated embedding layer integrates multidimensional data inputs, including raw traffic flow, spatial Laplacian embeddings, temporal periodic embeddings, and traffic occupancy embeddings, using gated convolutions.

The spatial–temporal encoder layer employs a hierarchical multi-view spatial attention module. A global spatial attention mechanism captures global spatial dependencies between nodes. To enhance sensitivity to critical local information, a geospatial attention mechanism focuses on the local dynamic features of neighboring nodes while discarding connections to distant nodes. Additionally, dynamic similarity spatial attention is incorporated, allowing the model to ignore distance and extract long-range spatial features by capturing the dynamic similarity of traffic patterns between nodes.

To adapts to the varying traffic patterns across different nodes. In the hierarchical two-stage temporal attention module, the first stage captures global temporal dependencies, while the second stage refines the query focus and re-applies temporal attention to emphasize traffic pattern differences between nodes.

The output layer employs skip connections and convolutional layers to transform the outputs into the final dimensions required for prediction. In this section, we provide a detailed description of the MMHFormer architecture.

4.1. Multi-Source Gated Embedding Layer

At each time interval, traffic flow in different urban areas is influenced by various factors, including traffic flow in neighboring areas and external environmental conditions. For instance, congestion in one area can lead to a sudden decrease in traffic flow across the city. Similarly, during holidays, urban traffic flow often increases significantly compared to weekdays. Based on this observation, we propose a multi-source gated embedding layer that integrates multidimensional data to capture the spatial–temporal dependencies and the impact of road conditions on traffic flow.

Specifically, the raw traffic flow input X is transformed into $X_f\in R^{T\times N\times d_f}$ via a Linear layer and combined with three types of embeddings: spatial Laplacian embeddings to encode the spatial structural features of the road network, temporal periodic embeddings to capture the periodic variations in traffic flow, and traffic occupancy embeddings to reflect external environmental impacts. Finally, a gated convolutional network achieves efficient fusion of multi-source information, effectively integrating the spatial, temporal, and external influences on traffic flow.

Spatial–temporal embedding: The Laplacian matrix is used to learn the correlations between nodes in the road network, embedding the graph into Euclidean space to obtain the spatial embedding representation $X_{spe}\in R^{N\times d_{st}}$. Considering the periodicity of urban traffic flow, we introduce weekly and daily periodic embeddings [30], represented as $t_w\left(T\right)$ and $t_d\left(T\right)\in R^d$, where $w\left(T\right)$ and $d\left(T\right)$ convert time t into week and minute indices. Temporal embeddings $X_w$ and $X_d\in R^{T\times d_{st}}$ are added to the spatial embeddings $X_{spe}$ to obtain the final spatial–temporal representation $X_{st}\in R^{T\times N\times d_{st}}$.

Traffic occupancy embedding: Variations in traffic flow and occupancy across different nodes may reflect the characteristics of the roads in those regions (e.g., major or minor roads), providing the model with the ability to recognize differences between nodes. To achieve this, we introduce the traffic occupancy embedding mechanism. Initially, raw traffic occupancy data is processed to extract key features, and a linear layer is applied to generate the traffic occupancy embedding representation $X_o\in R^{T\times N\times d_o}$.

Information fusion: The traffic flow input representation $X_f$, spatial–temporal embedding $X_{st}$, and traffic occupancy embedding $X_o$ are concatenated to produce the fused representation $X_e\in R^{T\times N\times d}$, where $d=d_f+d_{st}+d_o$ is the hidden dimension:

$$X_{fus}=concat(X_f,X_{st},X_o)$$

(2)

$$X_e=Conv(Conv\left(X_{fus}\right)\odot silu\left(Conv\left(X_{fus}\right)\right))$$

(3)

$$X_e=X_e+PE\left(X_e\right)$$

(4)

The gated convolution mechanism dynamically adjusts feature weights based on different input data, enabling more comprehensive fusion. Here, $Conv$ denotes $1\times 1$ convolution, $PE$ represents temporal positional encoding, $silu$ denotes the Sigmoid Linear Unit, and ⊙ represents the Hadamard product.

4.2. Hierarchical Multi-View Spatial Attention Module

As illustrated in Figure 3, global spatial attention is first introduced to capture the dependencies between traffic nodes and all other nodes. For the input feature representation $X_e\in R^{T\times N\times d}$, query, key, and value matrices are generated at each time step t through convolutional operations as follows:

$$\begin{array}{c}{Q}_t^{\left(S1\right)}=X_{e(t::)}{W}_Q,K_t^{\left(S1\right)}=X_{e(t::)}{W}_K,\hfill \\ V_t^{\left(S1\right)}=X_{e(t::)}{W}_V\hfill \end{array}$$

(5)

Here, $W_Q,W_K,W_V\in R^{d\times d^{\prime }}$ are learnable parameter matrices, where $d^{\prime }$ is the dimensionality of the queries, keys, and values. The attention score matrix $A^{\left(S1\right)}$ is computed as:

$$A^{\left(S1\right)}={\displaystyle \frac{Q_t^{\left(S1\right)}{K}_t^{\left(S1\right)T}}{\sqrt{d^{\prime }}}}$$

(6)

The softmax function is applied to $A^{\left(S1\right)}$, generating attention weights for each node relative to all other nodes. The global spatial representation $X_s^{\left(Glo\right)}$ is then obtained:

$$X_s^{\left(Glo\right)}=\mathrm{softmax}\left(A^{\left(S1\right)}\right)V_t^{\left(S1\right)}$$

(7)

In practical traffic scenarios, the dynamic features of a target node are often significantly influenced by its neighboring nodes, particularly during abrupt changes. To better capture the local dynamic characteristics of nodes, geospatial attention is introduced. This mechanism focuses on information from neighboring nodes while discarding connections with distant nodes. Based on the global spatial attention output $X_s^{\left(Glo\right)}$, a new query $Q_t^{\left(S2\right)}$ is generated at time step t using a $1\times 1$ convolution and treated as $V_t^{\left(S2\right)}$. The secondary attention weight $A^{\left(S2\right)}$ is computed as:

$$A^{\left(S2\right)}={\displaystyle \frac{Q_t^{\left(S2\right)}{\left(Q_t^{\left(S1\right)}\right)}^T}{\sqrt{d^{\prime }}}}$$

(8)

To ensure that the attention mechanism focuses only on nearby nodes, a geospatial mask matrix $M_{\mathrm{geo}}$ is defined. The matrix is undirected, and only nodes with distances less than $\lambda$ are considered for extracting important features. The geospatial representation $X_s^{\left(Geo\right)}$ is obtained as:

$$X_s^{\left(Geo\right)}=\mathrm{softmax}(A^{\left(S2\right)}\odot M_{geo})V_t^{\left(S2\right)}$$

(9)

Although some traffic nodes are not geographically adjacent, they may exhibit similar traffic patterns driven by shared traffic characteristics or demand patterns. Dynamic similarity spatial attention is introduced to dynamically adjust attention weights based on the similarity of traffic patterns between nodes, identifying potential correlations among non-adjacent nodes. We use a sliding 12-step time window to analyze traffic data and construct a dynamic similarity mask matrix $M_{dyn}$ using the fast Fourier transform (FFT). Specifically, we apply FFT to the past 12 steps of each node to extract short-term periodicity and local oscillation patterns, and then compute the Euclidean distances between the resulting frequency-domain features. This matrix encodes the relationships between each node and its K-most similar nodes, with a weight of 1 assigned to the most similar nodes and 0 to others.

The choice of a 12 step window is driven by the forecasting task, which involves predicting the next 12 steps based on the previous 12 steps. A window of 12 steps corresponds to one hour of traffic data, offering a sufficient segment for capturing stable frequency components, while also being short enough to account for transient variations and peak-hour fluctuations.

Based on $X_s^{\left(Glo\right)}$, a new query $Q_t^{\left(S3\right)}$ is generated at time step t using a $1\times 1$ convolution and treated as $V_t^{\left(S3\right)}$. Under the dynamic similarity mask, the dynamic similarity attention weight $A^{\left(S3\right)}$ is calculated as:

$$A^{\left(S3\right)}={\displaystyle \frac{Q_t^{\left(S3\right)}{\left(Q_t^{\left(S1\right)}\right)}^T}{\sqrt{d^{\prime }}}}$$

(10)

The dynamic similarity representation $X_s^{\left(Dyn\right)}$ is given by:

$$X_s^{\left(Dyn\right)}=\mathrm{softmax}(A^{\left(S3\right)}\odot M_{dyn})V_t^{\left(S3\right)}$$

(11)

Finally, the model integrates global spatial, geospatial, and dynamic similarity attention mechanisms to model multi-perspective spatial dependencies. The aggregated spatial representation is expressed as:

$$\begin{array}{cc}\hfill X_s& =\mathrm{Concat}\left(X_s^{\left(\mathrm{Dyn}\right)},X_s^{\left(\mathrm{Geo}\right)},X_s^{\left(\mathrm{Glo}\right)}\right)W_F\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill X_s& =\mathrm{FFN}\left(\mathrm{LN}\left(X_s\right)\right)+X_s\hfill \end{array}$$

(13)

Here, $W_F\in R^{3d^{\prime }\times d}$ is a learnable parameter. The feed-forward network (FFN) performs nonlinear transformations with two linear layers and a GELU activation function, while LN denotes the layer normalization operation.

4.3. Hierarchical Two-Stage Temporal Attention

4.3.1. Global Temporal Attention

In the first stage, to capture global dynamic temporal patterns, we first project the input tensor $X_s\in {\mathbb{R}}^{T\times N\times d}$ into temporal query, key, and value matrix using learnable $1\times 1$ convolutions:

$$\begin{array}{ccc}{Q}_t^{\left(T\right)}=X_{s(t,:,:)}{W}_Q^{\left(T\right)},& K_t^{\left(T\right)}=X_{s(t,:,:)}{W}_K^{\left(T\right)},& V_t^{\left(T\right)}=X_{s(t,:,:)}{W}_V^{\left(T\right)}\end{array}$$

(14)

Here, $W_Q^{\left(T\right)},W_K^{\left(T\right)},W_V^{\left(T\right)}\in R^{d\times d}$ are learnable parameters.

Next, we compute the scaled dot-product between the queries and keys at each node to obtain the temporal attention scores. The output of the first-stage temporal attention module is then calculated by applying the attention scores to the values:

$$X_t^{\left(T1\right)}=\mathrm{softmax}({\displaystyle \frac{Q_t^{\left(T\right)}{\left(K_t^{\left(T\right)}\right)}^{\top }}{\sqrt{d}}})V_t^{\left(T\right)}.$$

(15)

4.3.2. Spatially-Informed Temporal Attention

Time series data not only have long-term dependence, but also show complex cross-node changes due to the heterogeneity and dynamics of time series characteristics of each node. It is difficult to describe this heterogeneous evolution with a single time attention. To this end, as shown in Figure 4, we propose the spatially-informed temporal attention Module. Attention is combined with adaptive graph convolution to model the dynamic changes between nodes. Specifically, adaptive graph convolution dynamically adjusts the adjacency matrix by learning, fuses spatial topology information with the dynamic interaction between nodes, and can flexibly model the interdependence between different nodes. In the case of large cross-time changes between nodes, it can effectively distinguish and strengthen the timing pattern at critical moments. This mechanism enables the model not only to capture the dynamic evolution in time, but also to identify the temporal pattern differences between nodes, which further enhances the sensitivity to cross-node changes.

In the second stage, based on the output of the first stage, we capture temporal patterns that span multiple nodes by incorporating adaptive graph learning to infer latent inter-node dependencies. These learned dependencies are then used to compute the query matrix through a graph convolution step.

$$G=\mathrm{softmax}\left(\mathrm{ReLU}\left(E{E}^{\top }\right)\right)$$

(16)

$$Q_t^{\left(T2\right)}=G{X}_t^{\left(T1\right)}{W}_Q^{\left(T2\right)}$$

(17)

Here, $W_Q^{\left(T2\right)}\in R^{d\times d}$ is a learnable projection matrix, and $E\in R^{N\times a}$ denotes the learnable node embeddings.

In this stage, the first stage key $K_t^{\left(T\right)}$ and output $X_t^{\left(T1\right)}$ serve as the key K and value V, respectively, to compute the secondary temporal attention weights. The weight matrix is normalized and applied to $X_t^{\left(T1\right)}$, which passes through a linear layer and an FFN to generate the final temporal context $X_t^{\left(T2\right)}$, completing the two stage aggregation of temporal features:

$$X_t^{\left(T2\right)}=\mathrm{softmax}({\displaystyle \frac{Q_t^{\left(T2\right)}{\left(K_t^{\left(T\right)}\right)}^{\top }}{\sqrt{d}}})X_t^{\left(T1\right)}.$$

(18)

$$X_t=\mathrm{FFN}\left(\mathrm{LN}\left(X_t^{\left(T2\right)}\right)\right)+X_t^{\left(T2\right)}$$

(19)

4.4. Output Layer

To resize the spatial–temporal encoder output, we apply 1 × 1 convolution skip connections, summing each layer’s output to obtain $X_h\in R^{T\times N\times d_{sk}}$.

$$\begin{array}{c}\hfill \widehat{X}=Conv2\left(Conv1\left(X_h\right)\right)\end{array}$$

(20)

For multi-step prediction, $Conv1$ and $Conv2$ adjusts $X_h$ to the prediction window dimension, reducing error and enhancing model effectiveness.

5. Experiments

5.1. Datasets

The datasets used for the experiments in this paper include PeMS03, PeMS04, PeMS07, and PeMS08, which are all derived from the California State Traffic Management System (CSTMS) and collected by the California Department of Transportation (Caltrans) through its traffic monitoring network. The data cover highway traffic conditions across various regions, collected every 5 min by real-time sensors deployed on the roadways, providing multidimensional features such as traffic volume, speed, and roadway occupancy for time-efficient monitoring of traffic flow. These multidimensional features serve as inputs to the traffic flow prediction model, which effectively captures the traffic patterns and dynamic changes in each region, revealing regional traffic differences. Due to its large size and high frequency, the PeMS dataset has become a cornerstone of ITS research, providing a valuable testbed for data-driven traffic decision-making and system optimization. Specific information about each dataset is detailed in

Table 1. The detailed information of datasets.

Dataset	Sensors	Edges	Time Range	Time Steps
PeMS03	358	547	September 2018–November 2018	26,208
PeMS04	307	340	January 2018–February 2018	16,992
PeMS07	883	866	May 2017–August 2017	28,224
PeMS08	170	295	July 2016–August 2016	17,856

.

5.2. Baselines

We compare MMHFormer with the following baseline models: traditional time series forecasting models, graph neural network-based models, and transformer-based models.

• VAR [4]: VAR is based on the assumption that traffic flow follows an autoregressive pattern, meaning that future values can be predicted using past data in the series.
• SVR [5]: SVR leverages the principles of Support Vector Machines (SVMs) to effectively model traffic flow in a nonlinear way, unlike traditional linear regression methods.
• DCRNN [34]: DCRNN models traffic flow as a diffusion process on a directed graph, capturing spatial dependencies through bidirectional random walks. It also models temporal dependencies using a sequence-to-sequence structure combined with scheduled sampling.
• GraphWaveNet [20]: GraphWaveNet automatically generates the graph adjacency matrix by adaptively learning node embeddings, which enables more accurate capturing of hidden spatial dependencies. Additionally, it utilizes stacked dilated causal convolutions to effectively handle long-term temporal dependencies.
• AGCRN [21]: AGCRN introduces the NAPL and DAGG modules, which are designed to capture node-specific patterns and automatically infer interdependencies across different traffic series. This enables AGCRN to effectively capture fine-grained spatial and temporal correlations within traffic series data, enhancing its ability to model complex traffic dynamics.
• STGCN [18]: STGCN is a model that combines graph convolutions to capture spatial dependencies between traffic nodes and 1D convolutions to model temporal dynamics, effectively handling the complex spatio-temporal correlations in traffic flow prediction.
• MTGNN [35]: MTGNN exploits the underlying spatio-temporal dependencies by automatically learning the relationships between variables. The framework includes a graph learning layer, a graph convolution module, and a temporal convolution module, which adaptively learns the graph structure to capture spatial dependencies between variables and integrates multi-frequency temporal patterns to enhance prediction performance.
• GMAN [36]: GMAN adopts an encoder-decoder structure, with both the encoder and decoder comprising multiple spatial–temporal attention blocks to model the complex spatio-temporal correlations of traffic systems. By incorporating a transformer attention mechanism, GMAN aims to reduce error propagation in long-term forecasting.
• ASTGCN [19]: ASTGCN consists of three independent components that model the recent, daily, and weekly dependencies of traffic flow. Each component includes spatial–temporal attention mechanisms and spatial–temporal convolution modules to capture the dynamic spatio-temporal correlations and features of traffic data.
• STFGNN [37]: STFGNN is a novel spatiotemporal fusion graph neural network for traffic flow prediction. Using data-driven “temporal graphs” to complement traditional spatial graphs, it effectively captures hidden spatiotemporal dependencies. STFGNN’s fusion operation processes multiple spatial and temporal graphs in parallel, integrating with a gated convolution module to handle long sequences and capture richer dependencies.
• STID [38]: By adding spatial and temporal identity information, this model addresses the issue of sample indistinguishability in the spatio-temporal dimensions, thereby enhancing the model’s predictive capability.
• GDGCN [39]: GDGCN systematically explores spatial, temporal, and feature dimensions of data by combining parameter-sharing and independent modules. It designs a novel temporal graph convolution block to process the dynamic relationships of historical time slices in graph form. Additionally, a dynamic graph constructor is introduced to model time-specific spatial dependencies and the dynamic interaction relationships between different time slices.
• PDFormer [30]: PDFormer addresses the limitations of current GNN-based models in static modeling, short-range spatial dependencies, and the neglect of propagation delays. PDFormer introduces a dynamic spatial self-attention module to capture dynamic spatial dependencies and employs both geographic and semantic graph mask matrices to simultaneously capture short-range and long-range dependencies.
• DDGformer [33]: DDGformer captures the directional and relative positional relationships in traffic data using a direction- and distance-aware self-attention module. It also uses a dynamically enhanced adaptive graph convolution network to capture dynamic patterns in traffic systems.
• STGAFormer [32]: STGAFormer effectively integrates both local and global dynamic spatio-temporal features and employs a distance-based self-attention module to capture critical features between different regions. The model incorporates multidimensional inputs, including traffic flow attributes, periodicity, proximity adjacency matrices, and adaptive adjacency matrices, to better capture the spatio-temporal characteristics of traffic flow.

5.3. Experimental Settings

In this experiment, data from the past 12 time steps (one hour) are used to predict the traffic flow for the next 12 time steps, based on current mainstream traffic flow prediction methods. The dataset is divided into training, validation, and test sets in the ratio of 6:2:2 to fully evaluate the generalization ability of the model. The experiments were conducted on an NVIDIA RTX 4090 GPU (NVIDIA, Santa Clara, CA, USA). The batch size was set to 16 for PeMS03, PeMS04, and PeMS08, and to 6 for PeMS07, with 200 epochs of training to ensure sufficient model convergence and to avoid overfitting. The model’s spatial–temporal encoder comprises 6 layers (L), with the hidden dimension (d) set to 64. The optimizer uses AdamW with an initial learning rate of 0.001 and combines the weight decay mechanism to effectively prevent overfitting. In addition, overfitting is further prevented during training by employing an early stopping criterion, which halts training once the validation performance stops improving.

5.4. Evaluation Metrics

To comprehensively assess the model performance, we introduced three commonly used evaluation metrics: mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE). To ensure the accuracy of the results, we filtered the missing data.

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n|y_i-{\widehat{y}}_i|$$

(21)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}$$

(22)

$$MAPE={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|{\displaystyle \frac{y_i-{\widehat{y}}_i}{y_i}}\right|\times 100\%$$

(23)

Here, $y_i$ and ${\widehat{y}}_i$ denote the actual and predicted traffic flow values at node i for a given time step, respectively. The variable n represents the total number of nodes.

5.5. Comparison Results

Table 2. Comparison Table of Model Performance. Bold: best; underline: second best.

Model	PeMS03			PeMS04			PeMS07			PeMS08
	MAE	RMSE	MAPE (%)	MAE	RMSE	MAPE (%)	MAE	RMSE	MAPE (%)	MAE	RMSE	MAPE (%)
SVR [5]	27.40	26.46	44.51	28.66	44.59	19.15	32.97	50.15	15.43	23.25	36.15	14.71
VAR [4]	23.65	38.26	24.51	24.54	38.61	17.24	50.22	75.63	32.22	19.19	29.81	13.10
DCRNN [34]	17.99	30.31	18.34	21.22	33.44	14.17	25.22	38.61	11.82	16.82	26.36	10.92
GraphWaveNet [20]	19.12	32.77	18.89	39.66	31.72	17.29	26.39	41.50	11.97	18.28	30.05	12.15
AGCRN [21]	15.98	28.25	15.23	19.83	32.26	12.97	22.37	36.55	9.12	15.95	25.22	10.09
STGCN [18]	17.55	30.42	17.34	21.16	34.89	13.83	25.33	39.34	11.21	17.50	27.09	11.29
MTGNN [35]	15.85	26.23	15.55	19.08	31.56	12.96	20.82	34.09	9.03	15.40	24.93	10.17
ASTGCN [19]	17.34	29.56	17.21	22.93	35.22	16.56	24.01	37.87	10.73	18.25	28.06	11.64
STFGNN [37]	16.77	28.34	16.30	19.83	31.88	13.02	22.07	35.80	9.21	16.64	26.22	10.60
GDGCN [39]	14.66	24.30	13.94	18.44	29.79	12.52	20.15	33.21	8.50	14.82	23.87	9.35
GMAN [36]	16.87	27.92	18.23	19.14	31.60	13.19	20.96	34.10	9.05	15.31	24.92	10.13
STID [38]	15.33	27.40	16.40	18.29	29.82	12.49	19.54	32.82	8.25	14.20	23.49	9.28
PDFormer [30]	14.94	25.39	15.82	18.32	29.97	12.10	19.83	32.87	8.53	13.58	23.51	9.05
DDGFormer [33]	15.01	24.95	15.89	18.04	30.06	11.76	18.99	32.25	7.93	13.37	23.15	8.83
STGAFormer [32]	14.56	24.94	14.69	18.18	29.78	11.98	19.65	32.62	8.45	13.06	22.43	8.87
MMHFormer (ours)	14.33	24.80	14.68	17.99	29.53	11.64	18.74	32.10	7.79	12.94	22.30	8.77

presents the performance of our proposed MMHFormer model compared to other baseline models on next-hour predictions across four real-world datasets. The best-performing results are highlighted in bold, while the second-best results are underlined. To validate the superiority of the model, most baseline results were obtained from official documentation and related studies [28]. Our model achieved the best performance across all metrics on the four datasets: PeMS03, PeMS04, PeMS07, and PeMS08.

To evaluate the long-term prediction capability of the model, we visualized its MAE, RMSE, and MAPE across the next 12 steps, as shown in Figure 5. Compared to other models, our model demonstrates smaller variations in all metrics throughout the prediction horizon, highlighting its superior stability and robustness. This consistent performance further confirms its suitability for long-term prediction tasks.

DDGformer, PDFormer, and STGAFormer are three representative attention-based models. However, they fuse features across different dimensions through simple addition during data input embedding. In contrast, MMHFormer employs a concatenation approach across dimensions, which not only preserves the complete representation of spatial–temporal features but also incorporates traffic occupancy as an additional data source, enabling a more accurate representation of real-world traffic conditions. Additionally, the introduction of a gating mechanism allows the model to dynamically adjust its focus on each dimension under different scenarios, thereby effectively retaining and utilizing information from each dimension.

Although PDFormer and STGAFormer are designed to address short-range and long-range spatial correlations, their focus is limited to local spatial relationships, failing to adequately capture global spatial features. In contrast, MMHFormer introduces two masking matrices to capture global, local, and long-distance dynamic similarity patterns of nodes simultaneously, enabling multi-view modeling of spatial features. Furthermore, in the temporal dimension, other models use single temporal attention to capture temporal features, which may be insufficient to adapt to significant temporal heterogeneity. In contrast, MMHFormer employs a hierarchical two-stage temporal attention mechanism to flexibly adjust to the temporal features of different nodes, dynamically adapting to varying traffic patterns.

To further demonstrate the effectiveness and practicality of MMHFormer, the computational complexity of the model is also analyzed in this section. The computational complexity of each module in STGAFormer’s spatial–temporal encoder is analyzed as follows: the distance-based spatial self-attention module requires $O(4N^2{d}_{\mathrm{model}}+2N^2)$ operations, the gated temporal self-attention module consumes $O(2T^2{d}_{\mathrm{model}}+T^2)$, and the position-wise feed-forward network contributes $O\left(NT{d}_{\mathrm{model}}^2\right)$. Consequently, the overall time complexity of STGAFormer’s spatial–temporal encoder amounts to $O\left(l(4N^2{d}_{\mathrm{model}}+2T^2{d}_{\mathrm{model}}+T^2+2N^2+NT{d}_{\mathrm{model}}^2)\right)$, where l denotes the number of encoding layers, T represents the number of time steps, N indicates the number of sensors in the traffic network, and $d_{\mathrm{model}}$ corresponds to the hidden dimension size. In the MMHFormer framework, the hierarchical multi-view spatial attention module has a complexity of $O(6N^2{d}_{\mathrm{model}}+3N^2)$. The hierarchical two-stage temporal attention mechanism requires $O(2T^2{d}_{\mathrm{model}}+2T^2)$ operations. The feed-forward network maintains $O\left(NT{d}_{\mathrm{model}}^2\right)$ complexity, resulting in a total time complexity of $O\left(l(6N^2{d}_{\mathrm{model}}+2T^2{d}_{\mathrm{model}}+2T^2+3N^2+NT{d}_{\mathrm{model}}^2)\right)$.

While MMHFormer demonstrates a moderately higher theoretical complexity compared to the best-performing baseline, STGAFormer, primarily due to its multi-view spatial attention design, this increased computational overhead is strategically justified. The hierarchical architecture enables more comprehensive spatial–temporal representation learning by simultaneously capturing global, local, and dynamic similarity patterns. The model remains practically feasible for real-world applications, as the polynomial complexity is manageable for typical urban traffic networks where N and T are constrained by physical infrastructure. The additional computational investment results in significant improvements in prediction accuracy and model interpretability, as shown by the experimental results across multiple benchmark datasets.

5.6. Long-Range Forecasting

To further evaluate the long-horizon forecasting capability of different models, we conduct experiments on the PEMS04 and PEMS08 datasets under an extended prediction setting. Specifically, we increase the prediction horizons to 36, 48, and 60 steps and report the performance at each horizon as well as their average.

When the horizon is extended to 60 steps, models that perform competitively at short horizons—such as DCRNN, Graph WaveNet, and MTGNN—exhibit pronounced degradation in MAE, RMSE, and MAPE. In contrast, attention-based architectures such as GMAN and the Transformer-based PDFormer show greater robustness and consistently outperform the recurrent and convolutional baselines, highlighting the advantage of explicitly modeling global temporal dependencies for long-range traffic forecasting. The detailed numerical results are summarized in

Table 3. Performance on long-range traffic flow forecasting. Bold: best; underline: second best.

Dataset	Method	@Horizon 36			@Horizon 48			@Horizon 60			Average
		MAE	RMSE	MAPE	MAE	RMSE	MAPE	MAE	RMSE	MAPE	MAE	RMSE	MAPE
PEMS04	DCRNN [34]	23.67	37.62	16.03%	24.47	38.86	16.81%	25.39	39.92	18.10%	22.79	36.19	15.70%
	STGCN [18]	26.85	41.78	18.32%	27.27	42.40	18.74%	28.90	44.54	19.65%	26.66	41.58	18.30%
	GraphWaveNet [20]	24.25	39.13	17.94%	24.32	39.29	18.01%	24.98	39.97	18.48%	24.26	39.20	17.78%
	MTGNN [35]	23.18	37.12	15.63%	23.78	37.80	16.17%	25.11	39.16	17.65%	22.37	35.88	15.25%
	AGCRN [21]	23.03	36.76	15.72%	23.47	37.65	16.38%	24.93	39.38	17.87%	22.29	35.58	15.72%
	GMAN [36]	22.45	36.91	16.07%	22.72	39.13	16.33%	23.20	40.48	17.00%	22.26	35.66	15.93%
	PDFormer [30]	22.18	35.57	15.26%	22.45	36.40	15.33%	27.64	37.73	15.91%	21.34	34.55	14.60%
	MMHFormer (ours)	21.01	35.38	14.73%	21.49	35.19	14.95%	22.28	36.11	15.53%	20.43	33.34	14.43%
PEMS08	DCRNN [34]	20.48	31.73	13.73%	21.38	33.03	14.56%	23.12	35.22	16.35%	20.02	30.93	13.58%
	STGCN [18]	27.25	41.28	17.02%	27.63	41.97	17.27%	28.80	43.55	18.24%	26.83	40.86	16.77%
	GraphWaveNet [20]	21.27	35.06	13.75%	21.54	35.76	13.96%	22.01	36.38	14.95%	21.00	34.63	13.63%
	MTGNN [35]	19.57	31.55	13.14%	20.45	32.74	14.19%	22.10	34.57	15.95%	18.86	30.36	12.52%
	AGCRN [21]	20.06	31.92	14.39%	20.49	32.71	14.56%	21.58	34.04	14.80%	19.10	30.56	12.99%
	GMAN [36]	17.69	30.69	13.75%	18.13	31.20	14.06%	18.95	32.73	14.52%	17.48	30.14	13.56%
	PDFormer [30]	17.61	28.98	12.22%	18.24	29.87	12.80%	19.52	31.40	14.12%	16.93	28.05	11.83%
	MMHFormer (ours)	16.89	28.72	11.54%	17.29	29.73	11.96%	17.98	31.18	12.78%	16.26	27.49	11.11%

.

Compared with these baselines, MMHFormer achieves the lowest errors across all long horizons on both PEMS04 and PEMS08. On PEMS04, it reduces the average long-horizon MAE by about 4% compared with PDFormer, while on PEMS08 it yields relative improvements of around 8–9% at the most challenging 60-step horizon. These results demonstrate that the multi-source gated embedding, hierarchical multi-view spatial attention, and two-stage temporal attention enable MMHFormer to effectively capture long-range spatial–temporal dependencies, resulting in consistently superior long-horizon prediction performance.

5.7. Ablation Study

The MMHFormer model comprises three key modules: the multi-source gated embedding layer, the hierarchical multi-view spatial attention module, and the hierarchical two-stage temporal attention module. To validate the effectiveness of each component, we conducted ablation experiments on the PeMS08 dataset.We compared MMHFormer against the following variants:

• MMHFormer w/o multi-source embedding: This variant removes all auxiliary spatial, temporal, and traffic occupancy embeddings, using only raw traffic flow data as input.
• MMHFormer w/o gated embedding: This variant retains all the embedding sources but removes the gating mechanism, instead fusing the features through simple addition.
• MMHFormer w/o hierarchical multi-view spatial attention: The multi-view spatial attention mechanism is replaced by a global spatial attention mechanism for direct spatial feature extraction.
• MMHFormer w/o hierarchical two-stage temporal attention: The second stage temporal attention structure is removed, and only the first stage temporal attention mechanism is retained.

Figure 6 illustrates the comparison results of these variants. We can observe that removing the multi-source embedding leads to a significant increase in MAE, RMSE, and MAPE, highlighting the importance of integrating diverse embeddings, such as spatial Laplacian, temporal periodic, and traffic occupancy, in effectively capturing spatial–temporal dependencies. Replacing the gated embedding mechanism with simple feature addition results in a slight performance decline, suggesting that the gating mechanism is crucial for dynamically adjusting feature contributions, thereby improving model accuracy.

The inclusion of the hierarchical multi-view spatial attention module enables MMHFormer to account for global spatial dependencies while also capturing geographic and dynamic similarity-based spatial dependencies. When this module is removed, the RMSE, MAE, and MAPE metrics increase, underscoring the importance of multi-view spatial information for modeling the spatial dependencies of traffic flow.

The hierarchical two-stage temporal attention module also proves highly impactful. Removing the second stage temporal attention structure results in increased RMSE, MAE, and MAPE values, demonstrating its effectiveness in capturing temporal dependencies within traffic flows. This module is particularly advantageous in addressing sudden traffic flow fluctuations and varying temporal patterns across nodes. The second-stage temporal attention mechanism, in particular, enhances the model’s adaptability to diverse traffic patterns at different nodes.

5.8. Parameter Sensitivity Analysis

To further investigate the influence of different parameter settings on our proposed model for the traffic forecasting task, we conduct a parameter sensitivity analysis for MMHFormer. Specifically, we explored various values for each hyperparameter within predefined search spaces: [4,5,6,7] for the geographic distance threshold $\lambda$, and [6,7,8,9] for the number of nearest neighbors K based on dynamic similarity mask matrix. These hyperparameters are pre-trained using only the training set data, ensuring no information leakage during the training or inference processes. This analysis allowed us to evaluate the impact of different configurations on the performance of our MMHFormer model.

The results, as shown in Figure 7, reveal the following observations: (1) The geographic distance threshold $\lambda =5$ best preserves spatial dependencies, enabling the model to capture relevant relationships without unnecessary complexity. Smaller values (e.g., $\lambda =4$) place too much emphasis on local dependencies, resulting in a loss of broader spatial context, while larger values (e.g., $\lambda =7$) lead to overfitting and reduce efficiency by including irrelevant or weak connections. (2) Increasing the number of nearest neighbors K based on FFT-calculated similarity improves model performance up to $K=7$. Beyond this point, further increases provide minimal gains. Specifically, when K is smaller than 7 (e.g., $K=6$), the model may fail to capture long-range dependencies adequately, leading to underfitting and a less accurate prediction of traffic flow. On the other hand, when K exceeds 7 (e.g., $K=8$ or $K=9$), the model becomes overly sensitive to noise, which can lead to overfitting and reduced generalization ability. A value of $K=7$ strikes a balance, capturing both local and global patterns effectively while avoiding unnecessary complexity.

5.9. Traffic Occupancy Embedding

To highlight the importance of traffic occupancy data as an input embedding, this study conducts an in-depth analysis of the daily variations in both traffic flow and traffic occupancy. Twenty specific sensors were selected from the PeMS08 dataset, and heatmaps of both traffic flow and traffic occupancy were generated. Figure 8 visually demonstrates that, during peak periods (such as 7–9 a.m. and 5–7 p.m.), both traffic flow and traffic occupancy reach peak values at multiple nodes simultaneously. These significant peak regions indicate strong synchronization between flow and road occupancy during peak hours. Moreover, data from Node 16 shows that, although traffic flow decreases during certain periods, traffic occupancy remains relatively high. This may be due to factors such as reduced speed and increased vehicle density, leading to localized congestion. In such cases, relying solely on traffic flow may not accurately reflect congestion conditions. Traffic occupancy, as a supplementary indicator, can assist the model in identifying abnormal features under varying congestion states. Additionally, variations in traffic flow and occupancy across different nodes may reflect the features of the roads in the region, providing the model with the ability to identify differences between nodes. Therefore, embedding traffic occupancy into the model helps capture the complex spatial–temporal dependencies of traffic flow, thereby improving the model’s prediction accuracy and adaptability.

5.10. Case Study

To gain insights into the model’s spatial reasoning capabilities, We visualized the normalized attention weights of the geographic spatial attention during the morning peak (9:00–10:00) and the dynamic similarity spatial attention during the off-peak period (15:00–16:00). Figure 9 demonstrates that two attention mechanism captures distinct spatial patterns: During the morning peak (9:00–10:00), traffic congestion is typically higher on major arterial roads and key intersections, as people commute to work or school. The Geographic Spatial Attention mechanism captures this phenomenon by emphasizing the nodes (roads or intersections) with immediate neighbors that are highly trafficked, thus modeling the flow patterns in densely populated areas. As seen in Figure 9a, the attention heatmap predominantly highlights these localized clusters of road nodes, reflecting the high traffic density and the strong temporal dependencies at this time. The corresponding geospatial mask (Figure 9b) further refines this attention by suppressing connections that are distant or irrelevant to the immediate traffic flow. Dynamic similarity spatial attention (Figure 9c) reveals interesting long-range dependencies between nodes sharing similar traffic patterns, such as residential areas that experience synchronized morning outbound traffic flows. Similarly, by comparing the dynamic similarity attention heatmap (Figure 9c) with its mask matrix (Figure 9d), we observe that the mask strategically preserves connections between functionally similar nodes regardless of geographical distance, while filtering out irrelevant correlations. This enables the model to identify and leverage synchronized traffic behaviors across the network, capturing complex urban mobility patterns that transcend physical connectivity.

5.11. Forecasting Results and Visualization

After selecting two specific nodes from the PeMS08 dataset, we visualized the prediction results for the test set (horizon = 12), as shown in Figure 10a. The true traffic flow curve exhibits relatively smooth variations with a clear trend. In this scenario, the model performs exceptionally well, almost perfectly capturing the overall trend of traffic flow, especially at peak and trough points. Furthermore, as shown in Figure 10b, despite the larger and more frequent fluctuations in the actual traffic flow, the model successfully identifies various abrupt changes and responds quickly, closely approximating the true values even in high-frequency fluctuation intervals. This stable prediction of extreme fluctuations further highlights the model’s superiority in handling complex and unstable traffic patterns.

6. Conclusions

In this paper, we proposed MMHFormer, a novel multi-source and multi-view hierarchical Transformer model designed for traffic flow prediction, effectively tackling the challenges of complex spatial–temporal dependencies, dynamic variations, and external influences in traffic networks. MMHFormer incorporates a multi-source gated embedding layer to dynamically fuse multidimensional features, including spatial, temporal, and external conditions, enhancing the representation of traffic scenarios. Additionally, it employs a hierarchical multi-view spatial attention module to capture global, local, and similarity-based spatial dependencies, effectively addressing spatial heterogeneity. To further improve adaptability, the model leverages a hierarchical two-stage temporal attention mechanism, which models global temporal patterns while adapting to node-specific variations. Extensive experiments on four benchmark datasets (PeMS03, PeMS04, PeMS07, and PeMS08) demonstrate that MMHFormer consistently outperforms state-of-the-art methods across various evaluation metrics, including MAE, RMSE, and MAPE. The model also exhibits remarkable long-term prediction stability and effectiveness, making it a reliable tool for intelligent transportation systems. Future work will aim to enhance MMHFormer by incorporating a broader range of external contextual features, such as various weather conditions and traffic incidents, and extending its capabilities to support multimodal data inputs.