A Hybrid LSTM-CPS Approach for Long-Term Prediction of Train Delays in Multivariate Time Series

Abstract

In many big cities, train delays are among the most complained-about events by the public. Although various models have been proposed for train delay prediction, prior studies on both primary and secondary train delay prediction are limited in number. Recent advances in deep learning approaches and increasing availability of various data sources has created new opportunities for more efficient and accurate train delay prediction. In this study, we propose a hybrid deep learning solution by integrating long short-term memory (LSTM) and Critical Point Search (CPS). LSTM deals with long-term prediction tasks of trains’ running time and dwell time, while CPS uses predicted values with a nominal timetable to identify primary and secondary delays based on the delay causes, run-time delay, and dwell time delay. To validate the model and analyse its performance, we compare the standard LSTM with the proposed hybrid model. The results demonstrate that new variants outperform the standard LSTM, based on predicting time steps of dwell time feature. The experiment results also showed many irregularities of historical trends, which draws attention for further research.

1. Introduction

Recent trends have showed a great need for the adoption of intelligent transport systems (ITS), especially in metropolises. This would generate various impacts on both passenger transport and freight logistics [1,2,3], which helps to reduce traffic emissions and energy consumption [4,5,6,7]. Moreover, train transport plays an essential role in a multimodal transport system for both inter-city and intra-city travelers. The train timetabling problem (TTP) aims to find a periodic timetable that provides a preoperational schedule for a set of trains and follows operational constraints [8,9]. Nevertheless, train delays may cause a scheduled timetable to become infeasible and lead to inefficient operations, poor services, and longer travel time to complete passenger journeys. Therefore, accurate prediction of train delays is a vital task for control of railway traffic. Traditional model-driven methods have been widely studied in train delay prediction, such as micro and macro simulation methods [10,11,12]. In simulation-based approaches, each simulation model requires numerous parameters. Due to time-varying operating conditions and interactions between different subsystems subject to infrastructure and operational rules, parameter calibration is a significant challenge in building a successful simulation model [13].

In railway transport systems, diverse data are collected from various sources containing sensors, smart cards, Global Positioning System (GPS), and video detectors. With the fast development of big data techniques, many studies have focused on machine learning algorithms to learn the general rules that map a set of inputs to outputs in spatio-temporal prediction. State-of-the-art delay prediction models can be divided into two categories: deterministic and stochastic prediction models. Deterministic prediction models, such as neural networks and ensemble learning, usually use past observations to produce a fixed best-estimate value, while stochastic prediction models such as Bayesian networks consider a probability distribution. A diversity of factors may cause train delays. For example, primary congestion predictive factors have the most significant impacts on congestion delay, including meets, passes, and overtakes [14]. Hence, although the train delay can be reduced, it cannot be avoided. The types of train delay prediction can be classified as real-time/short-term, and medium-/long-term prediction [15]. Train delays can be classified as primary and secondary delays [16]. Due to the effects of the delay propagation, a primary delay (an initial delay) occurring at a station often causes secondary delays (or knock-on delays) at subsequent stations.

Ref. [17] proposed a fuzzy Petri net (FPN) model to simulate train primary delays and knock-on delays in a railway system. Ref. [18] developed an artificial neural network (ANN) to predict delays of passenger trains. The application of support vector regression (SVR) achieved better performance than ANN [19]. Ref. [20] used a microscopic graph model for online prediction of event times based on a directed acyclic graph (DAG) with dynamic arc weights. Ref. [21] presented and compared the accuracy and computation time of global and local data-driven approaches for calibrating the microscopic prediction model in real-time. Ref. [22] constructed a decision tree to estimate the root causes and effects of knock-on delays. Ref. [23] established a dynamic train delay prediction system (DTDPS) based on shallow extreme learning machine (SELM) and deep extreme learning machine (DELM), which used a repeated learning process with historical train movement data and exogenous weather data for long-term prediction of train delays in a large-scale railway network.

Furthermore, Ref. [24] demonstrated a dynamic stochastic model that predicts conditional probability distributions of train delays in real-time. The model applied prior probabilities for parent nodes to update calibration results of Bayesian networks with the probability distribution and regression coefficients for every two related events. Similarly, Ref. [25] presented an algorithmic performance comparison of three Bayesian network training methods, including hill climbing, primitive linear, and hybrid structures. The hybrid structure combined data-driven approaches with some domain knowledge which outperformed the other models. Ref. [26] proposed a data-driven ensemble forecasting model for medium-term prediction of train delays. The model combined a context-aware Random Forest (RF) and a regression with a mesoscopic simulation-based approach. Ref. [27] indicated a bi-level RF approach to estimate near-term train delays. The approach incorporated an RF classifier for predicting the increase, decrease, or lack of change for a current delay at the primary level, and three RF regression models for measuring the change in delay at the secondary level. Ref. [28] proposed a generic delay prediction framework to establish a connection between expert knowledge with long short-term memory (LSTM). The Convolutional Neural Network (CNN) [29], Gated Recurrent Unit (GRU) [30], and LSTM can be combined to form a more complex structure. In some circumstances, hybrid methods (e.g., Bidirectional LSTM and GRU-LSTM) can reach a better performance than a simple structure for unlocking hidden patterns in data [31,32]. Ref. [33] showed a hybrid method to merge the knowledge of the network and the experience of the operators, and data-driven models based on operational data and exogenous information to predict running time, dwell time, train delay, penalty cost, and train overtaking. Ref. [34] presented a train spatio-temporal graph convolutional network (TSTGCN), which included weather variables as input variables to predict the number of delayed trains in a certain period. Ref. [15] proposed a rule-driven automation approach for multi-scenario real-time prediction, which consisted of a delay status labeling algorithm, resilience of section and resilience of station factors.

Although various models have been proposed for real-time or long-term train delay prediction, to the best of our knowledge, no prior work has applied a deep learning approaches to both primary and secondary delay prediction. In this paper, we propose a comprehensive architecture of deep learning methodology for long-term train delay prediction. The contributions of this paper are summarized as: (1) The causes of train delay, run-time delay and dwell delay are accurately investigated and distinguished. We aim to predict the running time and dwell time instead of performing train delay prediction directly. (2) Several machine learning models are trained and evaluated using General Transit Feed Specification (GTFS) time series dataset, and the experimental results show the effectiveness and efficiency of the proposed model. (3) Detailed guidelines on generating multivariate regression features and building a robust train delay forecasting system are introduced.

The remainder of the paper is organized as follows. Section 2 describes the train delay and interprets the causes of train delays; this section also introduces a preliminary critical point search (CPS) classification algorithm, supervised learning setting, and multivariate time series models. Section 3 of experiments discusses static and real-time train operation data fusion, evaluation methods, and prediction results. Section 4 presents a summary of this study.

2. Methodology

2.1. Train Delay Problem

A railway network can be expressed as a graph. The nodes on the graph present a series of stations. Passenger trains are typically run along railway tracks based on regular schedules. Each trip has an original station, a destination station, and some intermediate stations. In this paper, we consider a train T serving a series of stations $S_m$, where $S_m \in \left\{S_1,S_2,\dots ,S_l\right\}$. The train operational variables and spatio-temporal variables are defined as actual arrival time $a{t}_a \left(T,S_m\right)$, actual departure time $a{t}_d \left(T,S_m\right)$, scheduled arrival time $s{t}_a \left(T,S_m\right)$, scheduled departure time $s{t}_d \left(T,S_m\right)$, actual dwell time $d_a = a{t}_d \left(T,S_m\right) - a{t}_a\left(T,S_m\right),$ and actual running time $r_a = a{t}_a \left(T,S_m\right) - a{t}_d \left(T,S_{m - 1}\right)$, respectively. If the difference of arrival time $a{t}_a \left(T,S_m\right) - s{t}_a \left(T,S_m\right)$ and departure time $a{t}_d \left(T,S_m\right) - s{t}_d \left(T,S_m\right)$ are more than 30 s and 60 s, an arrival delay and a departure delay occur at the corresponding station, respectively. Furthermore, each trip has a real-time record from which it is to retrieve the category of the trip. Each station has a unique sequence value from which it is to specify the order of the stations.

2.2. Critical Point Search Classification

Classifying the predicted data by train delay categories helps us to understand the model output. A classification algorithm is required to incorporate domain knowledge to extract the information from the output of a predictive model. CPS is a rule-based classification method to search primary, secondary and on-time points [28]. It can be extended to identify those train stations that cause major disruption to services. If it is able to predict the primary delay and secondary delay, we can reduce or avoid delays in advance. The CPS algorithm (Algorithm 1) has If–Then rules to identify the primary delay and secondary delay at the corresponding arrival stations from arrival delays of a trip list $A{D}_{S_m}$. By comparing the difference of arrival delays of adjacent train stations, the two types of delay points or on-time running points can be added to the lists, $L_1, L_{2 }$ and $L_3$, where $L_1$ denotes a list of on-time running points, $L_2$ represents a list of secondary delay points, and $L_3$ is a list of primary delay points. The algorithm also can be applied to departure delay calculation.

Algorithm 1. CPS Algorithm

Input: Given a trip R with a set of arrival delay $A{D}_{S_m} = \left(A{D}_{S_1},A{D}_{S_2}\dots A{D}_{S_l}\right)$,

and pre-defined thresholds $V_1, V_2$

Output: $L_1,L_2,L_3$

m = 0

For each arrival delay $\left(A{D}_{S_1},A{D}_{S_2}\dots A{D}_{S_l}\right)$

if $A{D}_{S_{m+1}}<V_1$:

$L_1.add\left(A{D}_{S_{m+1}}\right)$

else:

if $A{D}_{S_{m+1}} – A{D}_{S_m} \ge V_2$:

$L_3.add\left(A{D}_{S_{m+1}}\right)$

else:

$L_2.add\left(A{D}_{S_{m+1}}\right)$

End for

2.3. Supervised Learning Setting

Figure 1 demonstrates the architecture of data visualization for the feature generation process. For a sequence-to-sequence model [35], the time series data as inputs are shown by $X^d$, where $d$ presents the number of samples (or stations) for each sequential matrix. The illustration of $X^d$ is shown in Figure 1 as a set of sequential matrices. The train delays, $X^d$, which can be indicated in vector form as $X^d = \left[X_t^d,X_{t+1}^d,X_{t+2}^d\dots ,X_{t + n - 1}^d\right]$. $X^d$ with a time-step is from $t$ to $t+n - 1$. For example, in $X_t^d$, the time steps of the lag $w$ are from $t - w + 1$ to $t$, which can be expressed explicitly in matrix form as

$$X_t^d = \left[\begin{array}{cccc}{x}_t^1& x_t^2& \cdots & x_t^d\\ x_{t - 1}^1& x_{t - 1}^2& \cdots & x_{t - 1}^d\\ ⋮& ⋮& \ddots & ⋮\\ x_{t - w + 1}^1& x_{t - w + 1}^2& \cdots & x_{t - w + 1}^d\end{array}\right]$$

(1)

2.4. Multivariate Time Series Forecasting Model

This section demonstrates the design and development of a comprehensive LSTM-CPS network architecture for multivariate delay predictions. As demonstrated in Figure 2, a multivariate forecasting model is composed of Long Short-Term Memory (LSTM), two rectified linear units (ReLU), two dense layers, a dropout layer, and a CPS component. The last dense layer outputs a range of future values. Additionally, the outputs are passed to the CPS as inputs in a list. In addition, the CPS classifies the predicted values, which include not only the primary delays, secondary delays, and on-time running of corresponding stations, but also the status of the entire trip in the future at a given history. Furthermore, different machine learning methods and model hyperparameters are introduced in the following content.

Ref. [36] introduced an LSTM network, which consisted of an input gate $i$, a forget gate $f$, a cell $c$, and an output gate $o$. As shown in Figure 3, a standard LSTM contains two types of states: the long-term state and the short-term state. Specifically, the short-term state $h_{t - 1}$ and the current input $X^d$ are fed to three Sigmoid functions and a $tanh$ activation function. Moreover, the long-term state $c_{t - 1}$ traverses the network to $c_t$. Firstly, it drops some memories in the forget gate, secondly, some new memories are added to the input gate by using the addition operation. Additionally, then the long-term state is passed through the $tanh$ activation function. Finally, the output gate filters the results, generating the output $y_t$ and the short-term state $h_t$ ($y_t$ is equal to $h_t$ at the time step). $W_{xi}$, $W_{xf}$, $W_{xc}$, and $W_{xo}$ denote the weight matrices that connect the input to the corresponding gates. $W_{hi}$, $W_{hf}, W_{hc}$, and $W_{ho}$ present the weight matrices that connect the hidden vector to the corresponding gates.$b_i, b_f$, $b_c$, and $b_o$ refer to bias vectors. In this paper, we follow the version of fully connected LSTM (FC-LSTM) from [37]. The key equations of LSTM are indicated in Equation (2) below, where $\sigma$ is the logistic sigmoid function and the operator ‘$\circ$’ is the Hadamard product:

$$i_t = \sigma \left(W_{xi} X^d+W_{hi}{h}_{t - 1}+W_{ci}\circ c_{t - 1} + b_i\right)$$

$$f_t = \sigma \left(W_{xf} X^d+W_{hf}{h}_{t - 1} + W_{cf}\circ c_{t - 1} + b_f\right)$$

$$c_t = f_t\circ c_{t - 1} + i_t\circ tanh(W_{xc}{X}^d+W_{hc}{h}_{t - 1}+b_c)$$

(2)

$$o_t = \sigma \left(W_{xo} X^d+W_{ho}{h}_{t - 1} + W_{co}\circ c_t+b_o\right)$$

$$h_t = o_t\circ tanh\left(c_t\right)$$

Multiple LSTMs can be combined to build a more complex network structure. Such a model is capable of solving real-world sequence learning problems. In some cases, it generates good outcomes where the model presents performance improvement, rather than a simple structure for discovering trends, seasonality, and cyclicity patterns [31,32]. Since the absolute error is more robust to outliers, we use an L1 loss to measure the performance of our model. Hence, the loss function of the hybrid LSTM-CPS learning architecture for multivariate prediction tasks can be expressed as

$${\widehat{h}}_n = \arg min\frac{1}{n}{\displaystyle \sum }_{i = 1}^{n}l\left|y_i - h{\left(X^d\right)}_i\right|$$

(3)

where ${\widehat{h}}_n$ are training loss and validation loss; $y_i$ is the ground truth; and $h{\left(X^d\right)}_i$ expresses the prediction of the model.

We sum up the main characteristics of our hybrid LSTM-CPS learning architecture in the following.

• The CPS algorithm can obtain primary delay and secondary delays on each trip, which presents essential time–space relationships in the status of the railway network.
• A multivariate LSTM model is developed to estimate regression coefficients for multiple stations.
• The hybrid LSTM-CPS is a batch-based prediction architecture to process structured time series. It is not only able to have multi-inputs and multi-outputs (MIMO) features but is also able to be extended to tackle spatio-temporal sequence learning tasks. Notably, this architecture allows deep learning models to reach higher efficiency and accuracy for large-scale railway networks.

3. Experiments

3.1. Data Description

To evaluate the performance of the proposed LSTM-CPS network architecture, comprehensive experiments have been conducted on the fused train dataset from the Transport for NSW’s open data hub. The dataset contains 161-day historical observations of the trip ‘600D’ (8:30 a.m.–9:00 a.m.) in the period 11 April to 21 November in 2019. The line is from Sydney’s Bondi Junction Station, Platform 1 to Hurstville Station, Platform 4 in the Monday to Friday morning peak hours. Meanwhile, the corresponding schedule data and geographical information are obtained in the same time period. This dataset includes scheduled arrival time, scheduled departure time, station name, longitude, and latitude.

GTFS offers detailed schedules and associated geographic information in an open-source data format [38]. A data extraction and pre-processing tool is implemented to obtain real-time information from a GTFS-realtime application programming interface (API) provided by Transport for NSW. This dataset contains arrival delay, departure delay, and stop ID.

The proposed learning architecture is implemented in Python using the TensorFlow Framework [39] and trained using the Adam algorithm [40].

3.2. Evaluation Metrics

In this study, the proposed model is evaluated based on five standard metrics: mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), coefficient of determination (${\mathrm{R}}^2$) and Adjusted R-squared (${\mathrm{Adjusted} \mathrm{R}}^2$) values, as shown in Equations (4)–(8), where $y_t$ is the actual value at time step $t$, ${\widehat{y}}_t$ is the predicted value, $\overline{y}$ is the mean of the observed values of the dependent variable, n is the total sample size, and k is the number of predictors.

$$\mathrm{MAE} = \frac{1}{n}{\displaystyle \sum }_{i = 1}^n\left|y_t - {\widehat{y}}_t\right|$$

(4)

$$\mathrm{RMSE} = \sqrt{\frac{1}{n}{\displaystyle \sum }_{i = 1}^n{\left(y_t - {\widehat{y}}_t\right)}^2}$$

(5)

$$\mathrm{MAPE} = \frac{1}{n}{\displaystyle \sum }_{i = 1}^n\left(\frac{y_t - {\widehat{y}}_t}{y_{t }}\right)$$

(6)

$${\mathrm{R}}^2 = 1 - \frac{{{\displaystyle \sum }}_{i = 1}^n{\left(y_t - {\widehat{y}}_t\right)}^2}{{{\displaystyle \sum }}_{i = 1}^n{\left(y_t - \overline{y}\right)}^2}$$

(7)

$${\mathrm{Adjusted} \mathrm{R}}^2 = 1 - \frac{n - 1}{n - \left(k + 1\right)}\left(1 - {\mathrm{R}}^2\right)$$

(8)

For training a deep learning model, the first 100 days of the 161-day dataset is compiled as the training dataset and the remaining data is used as the validation dataset, roughly 70/30 split. According to the network structure of Figure 2, we select a batch size of 32 and 150 epochs as the parameters of the training model. Figure 4 illustrates the loss during the training and validation procedures. Both the training loss and validation loss converge after approximately 60 epochs. In this experiment, the batch size and the number of epochs are set as 32, and 60 for running time prediction and 20 for dwell time prediction, respectively. MAE is exploited as a loss function in the training process. Since the performance difference between training and validation is acceptable, the model does not overfit the training data.

3.3. Model Comparison

Figure 5 reports the performance of our proposed models with different LSTM variants or combinations of CNN, LSTM, and GRU for predicting one day ahead running time of seven intermediate stations from the trip ‘600D’. All x-axes express the corresponding metrics. The y-axis of RMSE and MAE shows the error in seconds; the y-axis of MAPE denotes the proportion of the error; the y-axis on the right of the figure represents the training time (seconds) of the corresponding models. In addition to Random Forest, other models do not show significant differences in model performance. The standard LSTM reaches the lowest MAPE value. Random Forest reaches minimum training time. In summary, the standard LSTM has a sufficiently good prediction performance in terms of all validation metrics.

The expressions for the x-axis and y-axis in Figure 6 are the same as in Figure 5, but the performance of the proposed models for dwell time prediction, and estimation accuracies in terms of all validation metrics with one day ahead are reported instead. Random Forest provides the highest accuracy in RMSE. For MAE, and MAPE, GRU-LSTM achieves the highest prediction accuracy; however, the time cost of the model is the highest. To sum up, compared with the LSTM variants, the Random Forest, standard LSTM, and CNN have more substantial errors and less training time. LSTM variants do not have significant performance differences.

Table 1. Performance on running time prediction.

Model	RMSE (sec)	MAE (sec)	MAPE (%)	Training Time (sec)
Random Forest	148.431	107.15	62.825	1.2
CNN	20.0	15.411	14.583	8.3
LSTM	20.419	14.973	11.779	56.6
Bidirectional-LSTM	25.256	19.136	14.907	63.7
CNN-LSTM	24.218	17.076	15.066	47.1
GRU-LSTM	21.527	18.086	15.960	85.2

and

Table 2. Performance on dwell time prediction.

Model	RMSE (sec)	MAE (sec)	MAPE (%)	Training Time (sec)
Random Forest	13.418	9.980	23.688	1.2
CNN	18.128	14.491	24.014	3.8
LSTM	16.670	12.125	18.897	17.7
Bidirectional-LSTM	15.586	11.147	17.107	24.3
CNN-LSTM	14.837	10.229	15.498	19.8
GRU-LSTM	14.717	9.844	14.480	29.9

demonstrate the results of the proposed models. In Table 1, it is observed that Random Forest has the lowest training time. For 7-steps ahead prediction, LSTM achieves lower MAE and MAPE, compared to others. However, in Table 2, GRU-LSTM performs well for 7-steps ahead prediction. Additionally, the differences between all LSTM variants in performance are slight.

As shown in Figure 1, the trend volatility line of running time (red line) is more regular than the dwell time (green line). It shows that dwell time demonstrates higher randomness, due to delays caused by the non-fixed daily number of passengers or uncertain events that occurred at the corresponding train station. For run-time prediction, a simpler neural network model can quickly achieve good performance. For dwell time prediction, a more complex network structure can better capture hidden patterns and trends in the dataset.

To establish a real-world deep learning application, we also need to balance or even trade-off accuracy and training time and prevent overfitting of the model. Time series data have a certain degree of randomness. In some cases, certain model approaches a high accuracy, but overfitting occurs. It dramatically reduces the portability of the model. In addition to accurately assessing the maximum predictability of the dataset, in further studies, we need to explore automated model selection techniques to identify an appropriate predictor for the corresponding inputs.

Finally, we compared R-squared and Adjusted R-squared values resulting from the proposed models. Figure 7 reports a standard LSTM for running time prediction and best performance LSTM variants for dwell time prediction from one day ahead. Moreover, a baseline model predicts $\overline{y}$, which has default values ${\mathrm{R}}^2 = 0$ and ${\mathrm{R}}_{\mathrm{adj}}^2 = 0$. The paper compares the proposed standard LSTM and LSTM variants against the baseline model. The result shows that the running time has a small variation with high accuracy, which is to a great extent explained by predictors in the standard LSTM. The proposed architecture presents reliable predictive power. Additionally, the running time shows a weak dependence on train delays. Data visualization in Figure 1 indicates that most trains run at their best performance regardless of departure delays. The impact of running time on the peak hours of workdays is also weak. Therefore, dwell time is more closely related to arrival delays.

Figure 7 shows the R-squared and Adjusted R-squared values for one day ahead. There is an amount of unexplainable uncertainty, which is represented by R-squared and Adjusted R-squared. It can be observed that most of the randomness comes from the dwell time. Our model chooses the appropriate numbers of epochs to achieve the smallest value ${\widehat{h}}_n$ through the method of Figure 4. If we increase the number of epochs to reach the higher values of R-squared and Adjusted R-squared, it will cause the model to be overfitting or underfitting.

4. Conclusions

This paper dealt with establishing a hybrid deep learning architecture for long-term train delay prediction using real-world data collected from different sources in an Internet of Things (IoT)-empowered public transport agency. The LSTM and LSTM variants could achieve high prediction accuracy by exploiting the long-term temporal dependency patterns. Several machine learning models, including Random Forest, CNN, LSTM, and LSTM variants, were applied to predict the running time and dwell time. Moreover, CPS utilizes experimental results to implement the predicted primary delays and the predicted secondary delays. The solution can be directly used for long-term prediction in urban railway systems. In future work, a stochastic probability model, such as a conditional Bayesian model, can be used to adjust the delay information of the entire network in real-time. Meanwhile, we will provide more comparisons with stronger baselines such as kernel methods, as meaningful future work.