Short-Term Traffic Speed Prediction and Pattern Discovery Using Sensor Data ^†

Abstract

The sustained growth of urban populations has worsened chronic traffic congestion, with detrimental impacts on multiple dimensions of urban livability, including increased commute times, reduced road safety, and the degradation of local air quality. Thus, traffic speed prediction (TSP) is an important component of intelligent transportation systems (ITSs) because urban traffic behavior is dynamic, nonlinear, and time-dependent. This study addresses the challenge of short-term TSP using time-series regression techniques applied to the METR-LA dataset utilizing a multi-sensor subset containing one week of data. After reconstruction into a tabular format, the data were preprocessed using interpolation via forward/backward filling, and transformed using temporal and lag-based features. On the modeling side, a comparative analysis with Linear Regression (LR), K-Nearest Neighbors (KNN), Decision Tree (DT), and Random Forest (RF) models was conducted, while clustering provided additional insights into traffic-state patterns. Results indicated that KNN achieved the best performance after tuning (MAE = 2.22, RMSE = 4.38, R² = 0.899). The findings illustrated that simple interpretable models can support TSP and traffic-state understanding.

1. Introduction

The number of vehicles has increased rapidly in recent years, which has led to growth in the complexity of urban mobility structures. This has made the use of intelligent transportation systems (ITSs) a necessity, and traffic prediction is an important part of this field. Urbanization has made ITSs a requirement for smart cities. Without such systems, increased travel durations, high fuel consumption and adverse environmental effects continue to increase. Traffic control and management systems are a key part of minimizing these effects and aid towards smarter city development. Traffic forecasting is a field extensively studied by urban planners and researchers to ensure optimized traffic flow [1,2]. This can help governments not only develop an efficient transportation system, but also wider smart city and sustainable transport objectives.

Traffic prediction is the process of predicting future traffic congestion or condition based on past data. It depends on variables, such as speed, traffic movement and travel time. It can be divided into multiple horizons, such as short-, medium- or long-term forecasting [3]. Short-term traffic prediction can help control traffic movement, and mitigate congestion resulting in efficient transportation flow. Medium- and long-term predictions are often utilized at government policy level in order to incorporate mechanisms or develop infrastructures in a way that would ensure maintaining an efficient flow of transportation for urban areas [4,5].

Several factors influence traffic behavior, which in turn affects traffic systems. Time of the day is one such factor, which plays a major role in traffic systems planning. Morning and evening rush hours are mostly monitored for most cities. The road network in a location is also important. Considering such factors supports making policies regarding traffic systems [6,7]. External factors like weather conditions or seasonality can also play a role. These factors can cause uncertainty in a smooth-running system and must be considered [3]. Factors like accidents, construction works, emergency or protocol movements can also cause traffic congestion. Weekends, public holidays, festivals or cultural events also affect traffic congestion [2].

Advancements in sensing and communication technologies have led to the generation and collection of extensive amounts of data related to traffic. Systems such as GPS, mobile devices, Internet of Things (IoT) sensors and surveillance cameras gather huge data that could help to build efficient models that predict traffic behavior [4,8]. The existence of such large-scale datasets has redirected attention towards more traditional model-based methods to Machine Learning (ML) methods that are capable of directly extracting patterns out of data.

This study focuses on short-term traffic speed prediction (TSP) and traffic pattern discovery, using simple and interpretable ML techniques. The work uses a reduced METR-LA subset to keep the analysis clear and computationally manageable while preserving important time-based traffic patterns. Starting with data engineering, preprocessing to convert the data into a well-fitted tabular format, an exploratory analysis and feature engineering using K-Means clustering was conducted to help identify repeated seasonal traffic states. On the modeling side, the aim was to evaluate how Linear Regression (LR), K-Nearest Neighbors (KNN), Decision Tree (DT), and Random Forest (RF) perform in predicting traffic speed using temporal and lag-based features.

In the remainder of this paper, Section 2 reviews the related literature, Section 3 explains the proposed methodology, while Section 4 discusses and evaluates results, and finally Section 5 concludes with directions for future work.

2. Related Work

Data-driven techniques have been widely used for TSP because traffic data contain hidden patterns and relationships gathered through multiple sources. Initial studies on this topic utilized classic statistical and time-series models, which were useful for understanding basic traffic behavior, but were limited in capturing the nonlinearity and dynamic nature of traffic datasets. With increasing amounts of data and their complex relationships, recent studies have started utilizing ML methods to better understand data relationships and support improved traffic prediction systems [9,10].

Traffic datasets face challenges like noise, missing sensor values and inconsistent data quality throughout multiple data collecting sources. These challenges could lead to poor model performance if not handled properly before modeling. Traffic data also have strong temporal dependency, where traffic patterns follow daily and weekly cycles, but may also change suddenly due to congestion, accidents, events, road works, or other external factors [8,11]. Therefore, preprocessing and feature engineering are important before applying prediction models, especially in short-term TSP where the recent past strongly affects the near future.

Classic statistical models such as ARIMA, SARIMA, Kalman filtering, historical averaging, and exponential smoothing have been used in traffic forecasting because of their interpretability and lower computational requirements [12,13,14,15,16,17]. These methods are useful as baseline forecasting approaches, but they usually depend on assumptions, such as linearity, stationarity, or predefined temporal patterns. Real-world traffic systems are more complex because traffic behavior is nonlinear, time-dependent, and affected by several uncertain factors. This has encouraged the use of ML models that can learn relationships directly from data.

ML approaches have become widely used in traffic prediction due to the difficulties of classic statistical models in handling nonlinear, high-dimensional and dynamic traffic data. In this study, the focus is kept on interpretable ML models rather than complex deep learning (DL) models. LR is used as a baseline because it is simple and easy to understand. KNN is used because it predicts future values based on similarities between current observations and historical traffic states. DT and RF are used because they can capture nonlinear relationships in the data, while RF reduces the overfitting risk of a single DT through ensemble learning [18,19,20]. On the unsupervised side, K-Means clustering is used to identify similar traffic states from traffic speed and time-based information [21].

DL models, such as Recurrent Neural Networks (RNNs), with gating mechanisms, like Long Short-Term Memory (LSTMs) or Gated Recurrent Units (GRUs), Convolutional Neural Networks (CNNs) and graph-based models have shown strong performance in traffic prediction problems [1,7,22]. These methods are useful because they can learn temporal or spatial relationships from large traffic datasets. However, these models are more complex, require larger computational resources, and are often less interpretable. Since the aim of this study is to demonstrate a clear and reproducible ML workflow, the analysis is limited to classic classification and clustering methods. This enhances understandability while demonstrating strong prediction and pattern discovery using traffic sensor data.

Recent works have also highlighted the importance of traffic congestion prediction, missing-data handling, and sensor-based traffic analysis in ITSs. Mystakidis et al. [3] reviewed emerging methods for traffic congestion prediction and showed the continued importance of reliable prediction methods for smart transportation systems. Tsalikidis et al. [23] proposed a multi-step approach for urban traffic congestion prediction using sensor and weather data. Other recent works have also explored interpretable sequence-to-sequence approaches for multi-step time-series regression for traffic prediction [24], with seasonal and weather features [25]. These studies support the relevance of traffic prediction, preprocessing, and interpretable modeling within smart city applications.

Although DL and graph-based models often achieve strong performance in TSP, this study focuses more on interpretable ML models. This approach allows analysis to emphasize on transparency, computational efficiency, and reproducibility in a reduced-data setting. Therefore, the comparison with DL methods is considered mainly from a methodological perspective, while a direct experimental comparison with state-of-the-art DL models can be further explored in future work.

3. Methodology

3.1. Dataset and Preprocessing

Our application domain is urban traffic pattern discovery and TSP. Traffic prediction is an important component of an ITS because it facilitates better traffic management, congestion mitigation, and efficient route planning. In addition to predictive modeling, understanding basic traffic patterns is also important for extracting useful insights from transportation data. Therefore, this study focuses on both regression-based prediction and clustering-based pattern discovery within a unified ML workflow.

For this task, we use the METR-LA dataset, popular in traffic forecasting research. This dataset contains traffic speed measurements gathered from 207 loop detectors across the highway network in Los Angeles County. The data were recorded at 5 min intervals, suitable for short-term TSP. METR-LA has been used in similar traffic forecasting studies, especially in spatiotemporal and graph-based prediction research. For example, Li et al. [6] used METR-LA in the DCRNN study to model both spatial and temporal dependency in traffic forecasting. The dataset has also been used together with PEMS-BAY in benchmark traffic forecasting studies. Compared with these larger-scale DL studies, the present work uses a reduced METR-LA subset and focuses on interpretable K-Means clustering and predictive methods.

The original dataset is stored in Hierarchical Data Format (HDF5), where the data are organized in structured arrays. The dataset was reconstructed into tabular format using the stored index, column identifiers, and value arrays. The resulting data structure consists of time-indexed traffic speed observations for multiple sensors, where each column represents a specific sensor and each row corresponds to a timestamp. Data from the first seven days and five sensors were used for the analysis. This subset kept the experiment computationally manageable while still preserving important temporal variation and sensor-level differences. One sensor was selected as the target variable for prediction, while the selected sensor group was also used for exploratory traffic pattern analysis.

The dataset contained a high amount of zero values, which do not show real traffic conditions but were considered to be affected by missing values or misread sensor data. To overcome this, zeros were treated as missing values and were replaced, using time-based interpolation to estimate values from nearby data points. The remaining missing values were filled using forward and backward filling. By applying this method, the data were transformed into a clean and complete dataset for further analysis.

3.2. Feature Engineering

As mentioned, the original dataset was converted into csv format using Pandas library [26]. The index was converted to a datatime format, which is a standard indexing methodology for time series forecasting. Feature engineering was used to create three time-based features: hour of day, day of week and a binary weekend indicator. This process was derived from the exploratory procedure in Section 3.3 that explained why traffic behavior and speed depend on these time features. Additionally, lag features were also created to help the model learn the past speed pattern from sensors. Traffic speed from one sensor was selected as an output feature, while the input features included temporal features along with the created lag features. The train–test split of 80:20 was used for the experiment [27,28]. This chronological split was adopted to reduce information leakage and to evaluate model performance on later observations.

3.3. Exploratory Clustering Setup

During data preprocessing, an exploratory data analysis was conducted to understand traffic patterns across the selected sensors. This setup was motivated by the fact that traffic observations often present temporal and seasonal behavior, with previous traffic prediction studies indicating the usage of temporal characteristics and seasonality-related information to improve forecasting performance [25].

The experiment was implemented via K-Means clustering in order to identify possible seasonal traffic behavior, accompanied with clear time-based patterns. Clustering outcomes were not used as an additional input feature in the regression models. Instead, they were used as an exploratory step to examine whether the selected sensor subset contained distinguishable traffic-state regimes. The insights from this step guided us towards the usage of temporal features, such as hour of day, day of week, and weekend indicators, by showing that traffic conditions vary across time-dependent states.

3.4. Traffic Speed Prediction Modeling

For short-term TSP, four regression models were utilized: LR, RF, DT, and KNN. LR was used as the baseline model, while KNN, DT, and RF were used to compare nonlinear and similarity-based learning behavior. Standardization was performed before training the KNN model, as it is a distance-based model.

The evaluation metrics used for this experiment are Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and the coefficient of determination (R²) [3]. Parameter tuning was performed on the best performing model (KNN) after an initial model comparison. The number of neighbors were changed for determining the most suitable value. This was done to examine how parameter tuning affects model performance. Figure 1 shows the methodology framework developed for this experiment.

Figure 1. Methodological framework.

4. Results and Discussion

4.1. Exploratory Analysis of Traffic Data

After cleaning the dataset, the traffic speed was analyzed for the selected sensors to understand how the traffic speed changed over time. Figure 2 shows how the traffic speed changes over time for these sensors. It clearly demonstrates traffic patterns over time. Similar daily patterns are shown in Figure 2, which means that the traffic conditions change depending on the time of day. In different sensors, sudden drops in speed can be observed, showing traffic congestion. By observing such changes, it is shown that traffic behavior is complex and changes with time, which is why ML techniques need to observe and study these patterns. Also, the differences between sensors show that traffic conditions vary in different areas of the network.

Figure 2. Traffic speed over time over five sensors.

4.2. Exploratory Traffic Pattern Discovery Using Clustering

To further explore the seasonality of the METR-LA subset, K-Means clustering was used to find hidden traffic patterns in the dataset. For applying clustering, the traffic speed and time-of-day data were used, which helped identify different traffic conditions. This analysis was used as an exploratory traffic-pattern analysis step before interpreting the prediction results.

Clustering results in Figure 3 show three different groups that represent different traffic conditions. One group shows high-speed traffic, which usually occurs during off-peak hours, when the roads are free and traffic flows freely. The second group shows moderate-speed or average-speed traffic conditions, while the third group shows low-speed traffic, because of traffic congestion, which could possibly happen during peak hours. These findings show that traffic behavior is not random but follows clear time-based patterns. The clustering results give useful insights into traffic conditions and support the use of temporal features in the regression models.

Figure 3. Traffic state clustering using K-Means.

4.3. Model Performance Comparison

To evaluate the prediction performance of different ML predictive techniques, four regression models were used: LR, KNN, DT, and RF. The performance of each model was evaluated using MAE, RMSE, and R². As Table 1 shows, the four models illustrate different prediction performance. Among them, KNN provided the best initial results, with the lowest prediction error and the highest R² value.

Table 1. Results of the four evaluated models.

Model	MAE	RMSE	R2
LR	2.464631	4.570303	0.890481
KNN	2.231291	4.404525	0.898282
DT	2.748815	6.161186	0.800966
RF	2.358864	4.827594	0.877803

Figure 4 shows a visual representation of model performance across all metrics. It can be observed that KNN outperforms the other models, in terms of both MAE and RMSE, while also achieving the highest R² value. LR is simple and easy to understand, but it may not capture the complex nonlinear patterns in the data. DT showed weaker performance, which may be linked to its sensitivity to tree depth and data variation. RF uses ensemble learning, which is why it performs better than DT, but in this study, it still did not perform better than the KNN model.

Figure 4. Model comparison for TSP.

4.4. Effect of Parameter Tuning

In order to further analyze the impact of model parameters, different values of k were tested for the KNN model. The results are shown in Figure 5, showing that the lower values for k make the model more sensitive to noise and cause higher variation in predictions, while higher values of k make the prediction too smooth. The best performance was achieved with $k=7$, which provided a suitable balance between bias and variance for this dataset, showing the importance of parameter tuning in the models’ performances.

Figure 5. KNN performance for different numbers of neighbors k.

4.5. Performance of the Tuned KNN Model

After reaching the conclusion that KNN is the best-performing model, parameter tuning was performed to find the best number of neighbors. The results showed that $k=7$ gave the best performance. The final tuned KNN model achieved an MAE = 2.22, RMSE = 4.38 and R² = 0.899.

These results show the capability of the model for accurate prediction of traffic speed, with almost 90% of the changes in the target variable explained by the model. Figure 6 shows that the predicted values closely follow the actual traffic speed, especially in showing the overall trends and short-term changes. Although there are small differences during sudden changes, generally the model is still able to record the traffic patterns successfully.

Figure 6. KNN actual vs predicted traffic speed with $k=7$.

The combined results from regression and clustering analysis show that ML techniques are effective for both prediction and knowledge discovery. The KNN model proved to be the most suitable for this dataset due to its ability to capture local patterns and nonlinear relationships. At the same time, clustering analysis provided a clear understanding of different traffic conditions by showing repeated congestion patterns and time-based changes. The results confirm that using simple ML methods together with proper data preprocessing and feature engineering can give strong performance in TSP and provide useful insights into traffic behavior.

5. Conclusions

This work presented a compact and interpretable application of ML techniques in the field of short-term TSP. The METR-LA dataset was reduced to five sensors and one week of data for the analysis. The novelty of this study is the demonstration that simple and computationally efficient ML models can provide useful predictive performance and interpretable TSP insights within limited-data settings.

Starting with data engineering, preprocessing to convert the data into a well-fitted tabular format, an exploratory analysis and feature engineering using K-Means clustering was conducted to help identify repeated seasonal traffic states. This analysis indicated the usage of features like hour of the day and traffic speed to capture similar states of traffic throughout the day. This analysis provided useful insights into the temporal behavior of traffic data, showing the effectiveness of ML for both predictive and exploratory analysis. On the supervised modeling side, the aim was to compare LR, KNN, DT, and RF performance on a TSP task, using temporal and lag-based features.

For the regression part, the results show the ability of models to capture the dynamic behavior of traffic. KNN was able to capture the data behavior and nonlinearity most effectively, achieving an RMSE of 4.38, MAE 2.22, and an R² value of 0.899. The performance of KNN shows the importance of capturing similar states in data and its effect on model performance. In addition, hyperparameter tuning was performed for KNN, which helped to identify 7 as the optimal number of neighbors.

The analysis faced several challenges, such as the HDF5 data format, missing and invalid values, and the appropriate treatment of time-series data. These difficulties were addressed through preprocessing, chronological data splitting, and feature engineering, which produced a stable and dependable modeling pipeline. The results also show that classic ML techniques, when applied in a systematic manner, can perform well in traffic forecasting tasks and for the discovery of meaningful patterns in the data.

The main contribution of this work is the demonstration that simple and interpretable ML methods can provide useful predictive performance and traffic-state understanding, even when applied to a limited sensor subset. Moreover, regression modeling combined with clustering analysis offers a useful approach to understanding traffic dynamics. Therefore, both techniques can be considered useful for ITSs.

Ongoing work expands this study towards broader experimental settings, already producing promising results. Extensions include increasing dataset temporal coverage, a larger number of sensors and more data, as well as evaluating multi-step forecasting horizons. Finally, external variables, such as weather or incident information, horizon-aware and advanced spatiotemporal modeling, can also be incorporated in the future to improve predictive performance.