Collaborative Optimization Method for Multi-Train Energy-Saving Control with Urban Rail Transit Based on DRLDA Algorithm

Abstract

With the traffic congestion problem deteriorating, people increasingly choose urban rail transit (URT) to travel. Although URT alleviates traffic congestion, the long-term operation of a large number of trains leads to huge energy consumption. In order to adapt the major social development concept of “Low carbon”, a multi-train energy-saving control collaborative optimization method is proposed in this paper. First, the composition of single train operating conditions is determined by the conversion of operating conditions between stations and the force changes under the premise of ensuring safe and on-time train operation. A single-train energy consumption calculation combinatorial optimization model with the dual control objectives of reducing passengers’ average waiting time as well as train traction energy consumption is established. The energy saving control strategy of a single train is investigated by ARMA-Radial Basis Function Neural Network (ARMA-RBFNN) and Genetic Algorithm (GA). Next, the queuing theory is introduced to analyze the variation in passenger waiting time for multiple trains at different arrival intervals. A Deep Reinforcement Learning (DRL) algorithm is designed to obtain the correlation among passenger waiting time, arrival interval and train stopping time. The optimization objective is to minimize the multi-train traction energy consumption and the average passenger waiting time while considering conditions such as train operating safety interval, speed limit, multiple operating state and single train energy-saving models, etc. Then, a multi-train cooperative energy-saving control model is proposed based on the Dragonfly Algorithm (DA). Finally, a case study of Beijing Metro Line 4 is conducted to illustrate the effectiveness of the proposed method. The results demonstrate that the total traction energy consumption and passenger waiting time are reduced by 3.1% and 5 s, respectively, compared with the method of independently optimizing the single-train control strategy. The findings can aid in the development of energy-saving strategies and also provide a basis for energy-saving operation control of multiple trains.

1. Introduction

In recent years, Urban Rail Transit (URT) systems have established independent operation lines. Compared with urban buses, taxis and other forms of public transport, URT systems have developed significantly due to their advantages in safety, high punctuality and reliability. The construction of URT systems has become an important way to alleviate traffic congestion. In China, the operating mileage of the URT systems increased from 5032.7 km in 2017 to 9192.62 km in 2021 Q3 [1,2]. The average annual growth rate is about 19.16% [2]. With the increase in the operating mileage, the energy consumption of the URT systems has drawn more attention. In 2020, the annual energy consumption of the URT systems was about 172.4 billion kWh [3]. Among them, traction energy consumption accounted for 8.4 billion kwh or 48.7% [3]. From the perspective of environmental protection and operational cost reduction, it is necessary to study how to reduce the energy consumption in URT systems. Therefore, it is of great importance to reduce energy consumption by optimizing the multi-train operating conditions and statuses. As a result, the reduction in traction energy and the improvement of multi-train operating control strategies are two effective measures to save energy in URT systems.

Regarding single-train energy-saving operating control strategies, most studies focus on the line condition, train operating speed and condition conversion point. Howlett, Pudney and Vu [4] calculated the critical switching points and key speeds so as to obtain an optimal and feasible solution for train energy saving. Bocharnikov, Tobias and Roberts [5] presented a single-train trajectory optimization to obtain minimum energy consumption with maximum regenerated energy. Albrecht et al. [6,7] proposed a method to obtain an energy-efficient driving strategy for a train’s journey on an undulating track with steep grades. Miao et al. [8] proposed an energy-saving operation optimization model of a single train based on time discretization. Li et al. [9] presented a Pareto multi-objective genetic algorithm to optimize the energy consumption of the train. Li et al. [10] established an energy-saving optimal control model of a train under a timing constraint considering ramp, speed limit of the line and feedback of regenerative braking energy. Various intelligent algorithms were applied to obtain the best combination sequence and accurate driving mode switching point based on different operation conditions and train types [11]. Yin, Chen and Li [12] presented a regression tree to deal with design and analysis of energy-efficient train speed profiles. Fernández-Rodríguez, Fernández-Cardador and Cucala [13] used fuzzy optimization to calculate an optimal driving strategy that balanced energy-efficiency and computational efficiency. Bao et al. [14] proposed a design and analysis of energy-efficient train speed profiles to predict the actual load value of microgrid energy. Dou et al. [15] established a speed-flow relationship model to analyze the highway situation based on the data passing through sensor network nodes. To sum up, the train has many possible control strategies for each segment when the running status is determined. Additionally, the traction energy consumption for each control strategy is normally different. However, most researchers have tried to seek for an energy-saving control strategy for a given ideal environment with the train operation statuses of adjacent stations, so as to balance the sequence of formulating strategies and the switching point of operating conditions.

As demonstrated in the literature review, when the running state is determined, the train has many possible control strategies for each operating segment. Meanwhile, the traction energy consumption for each control strategy is normally different. However, most researchers have tried to seek for an energy-saving control strategy for a given ideal environment with the train operation statuses of adjacent stations, so as to balance the sequence of strategy formulation and the switching points of operating conditions. This can reduce the practicality of the control strategy and its control model in a real-life case.

A train control model (TCM), consisting of multi-train and carriages, takes train running status as a mass model. Although coupling between adjacent carriages is simplified, it is still a common and effective method to develop a train control strategy. Thus, some studies have carried out research on train traction conditions. Yang et al. [16] proposed a cooperative scheduling method to optimize the timetable so that the energy generated by the braking train could be directly used to accelerate the train. Ye and Liu [17] established a multi-phase optimal control method for multi-train control and scheduling on railway lines. Huang and Shuai [18] applied a customer-oriented dynamic cyclical adjustment approach to optimize a high-speed railway train scheduling plan. Liu, Guo and Yu [19] formulated a cooperative control model, considering the utilization of regenerative energy. Long and Yin [20] developed hyper-speed train operation control and energy-saving control URT strategies. Liu et al. [21] proposed a cooperative scheduling approach to optimize the timetable based on variable traction force and braking action. Feng et al. [22] presented an energy-saving operation optimization method for URT trains based on the recycling of braking energy. Cao [23] constructed a multiple energy consumption model to optimize the manipulation strategy. Pineda-Jaramillo, J et al. [24] used four basic features to predict the traction power based on different machine learning models. Deng et al. [25] created an energy consumption optimization method for train operation with these multiple parameters. Liu, Yang and Yang [26] proposed multiple high-speed trains following a characteristic model and a cooperative optimal control strategy. Oh et al. [27] investigated and classified representative applications for railway safety, mainly focusing on deep learning approaches. Zhang et al. [28] established a multi-train collaborative energy consumption optimization model to minimize the total energy consumption. Jieli and Tao [29] constructed a collaborative optimization model of multi-train operation to maximize the overlap time for regenerative braking. The energy performance method for the energy-efficient train model is still a popular topic of research. Wang et al. [30] proposed an integrated energy-efficient train operation method to minimize the systematic energy. Gu, Tang and Ma [31] proposed a multiple-optimization-model-based energy-efficient operation method to reduce energy consumption and arrival delay for following trains. Su et al. [32] proposed a train regulation method to use the dwell time supplements sufficiently, which can reduce energy consumption. He et al. [33] presented a multi-particle train model to realize train trajectory optimization within a given timetable. Huang et al. [34] proposed a joint scheduling model to improve service quality and reduce energy consumption in urban rail transit networks.

In summary, many existing studies have proposed different train control models and their corresponding optimization methods. However, the above studies have affected passenger travel comfort due to the multiple adjustments of station dwell time, inter-station operating time and other operation parameters, as well as the large adjustment range of the current train operation schedule. At the same time, the combination of multiple operating conditions, such as multi-traction or multi-cruises, as well as the actual operating conditions, which consist of line speed limit and passengers’ travel waiting time, have not been considered, which reduces the practicability of the model to a certain extent. Although several studies focused on trains at a single station or adjacent stations in the same power supply interval, few studies considered the utilization of regenerative energy for multiple trains at multiple stations. In addition, most of the previous works discuss single-objective optimization problems, but train performance is often influenced by many factors.

The control strategy optimization and the train timetable optimization attempt to achieve global minimization of energy consumption. Several studies focus on train timetable optimization based on dynamic passenger flow. Sun et al. [35] formulated three optimization models to design demand-sensitive timetables by demonstrating train operation using equivalent time. Li et al. [36] adopted the Modular Multilevel Converter Type Railway Power Conditioner (MMC-RPC) with distributed super-capacitor (SC) energy storage (ES) scheme. Gao et al. [37] proposed a comprehensive quantitative optimization model of the route scheme for the desert railway based on Interval Number and TOPSIS. Niu, Zhou and Gao [38] calculated and adjusted train timetables for a rail corridor with given time-varying origin-to-destination passenger demand matrices. Considering dwell time uncertainty and traction force, a timetable optimization method was proposed to minimize the energy consumption and train travel time [39,40]. Su et al. [41] presented an integrated train operation approach by jointly optimizing the train timetable and driving strategy, which can minimize the systematic net energy consumption, including the traction energy consumption and the reused regenerative energy. Gao and Yang [42] proposed an integrated optimization method for trains’ timetables and speed profiles. The train speed profile can be optimized to achieve on-time and precise stopping through a sequence of operating modes and associated tracking/braking accelerations at each position. Li et al. [43] proposed a multi-objective programming model consisting of three objective functions to minimize the energy consumption, passenger waiting time at stations and waiting time of transfer passengers at transfer stations. In recent years, data-driven energy saving control methods have been applied in URT systems. A series of machine learning method are employed in data-driven multi-train energy control strategy optimization. Bai et al. [44] presented a cooperative control method to optimize each switching point of the driving regimes. Yin et al. [45] used a deep neural network (DNN) to improve the performance of train energy control. Zhang et al. [46] proposed a policy-based reinforcement learning train control approach to reduce the traction energy consumption of the train. Liu et al. [47] applied a model predictive control method to the minimum safety tracking interval. Due to the multiple parameters of the trains’ dynamics, Cao et al. [48] and Cao, Wen and Ma [49] applied proportional integral differential (PID) and fuzzy controllers to optimize train control and the speed–time curve. Liu and Liu [50] built a dynamic planning model using rail running time to find the optimal control strategy. Zhu et al. [51] proposed multi-train timetable optimization based on the improved Seeker Optimization Algorithm (SOA). Zhang et al. [52] proposed a novel distance measurement on hesitant fuzzy sets (HFSs) based on equal-probability transformation and their application to decision-making regarding traffic control. In a word, the above research mainly focuses on the passenger waiting time at the station, and does not consider actual constraints such as train multi-stage traction energy consumption, actual train capacity, safety speed limit, etc. The established mathematical model has not yet been perfected and adapted to the actual operation conditions.

Nevertheless, there are significant limitations to the previous studies. First, the above studies mainly focus on the passenger waiting time at the station, and do not consider actual constraints such as train multi-stage traction energy consumption, actual train capacity, safety speed limit, etc. The established mathematical model has not yet been perfected and adapted to the actual operation conditions. Second, reducing passenger waiting time and energy consumption through train scheduling has a great impact on improving service quality and energy efficiency from an operational perspective [43]. However, there is a lack of studies on coordinated optimization of train traction energy consumption of multiple trains that also consider time-varying passenger demand and train operating state variation. Third, the above studies reduce the total energy consumption by optimizing the timetable or speed profile, respectively. However, the train timetable and speed profile are not independent elements of train operation. Both have a direct impact on traction energy consumption and the utilization of regenerative braking energy, and should be considered together.

In summary, at present, both research on single train energy-saving operation and multi-train timetable optimization have achieved rich findings, but there are still some urgent problems regarding energy-saving train control that deserve in-depth study.

On one hand, in terms of train control, the energy-saving control method and mathematical model, based on the combination of time-varying passenger flow and multi-train operating conditions, have not been fully established. When multiple trains run at the same time, due to the short departure interval, any slight change in the train operation process will affect the operation of other trains in the network and indirectly influence the use of train traction energy consumption. In terms of ensuring the safety of multi-train workshop operation, few studies have been conducted on the coordinated optimization control method of train traction energy consumption based on the intersective relationship between time-varying passenger demand and train operating state variation.

On the other hand, although these data-driven models have significantly improved the accuracy and flexibility of TCM, there are still shortcomings: the existing data-driven TCMs are based on black-box models that do not explain how they use the full range of inputs in physical operation. However, TCM is a multi-objective optimization problem concerning safety, punctuality, passenger comfort and energy saving, which is a complex learning problem. The focus of this study is combined with deep learning and neural networks, which have demonstrated their ability to simulate a significant amount of high-dimensional data to predict the movements of helicopters, robots and patients. Most of these models are based on multi-layered neural networks without physical properties. As mentioned above, TCMs are sensitive to variations in operating conditions and train parameters, whereas the data-driven models are black boxes and, thus, difficult to use in practice. Therefore, how to combine deep learning with a dynamic model of physical operating parameters to predict and control multi-train energy consumption is still an outstanding problem.

In view of multi-train energy-saving control collaborative optimization, a set of mathematic models are presented in which the objective function is to minimize the multi-train traction energy consumption and average passenger waiting time. Then, a DRLDA algorithm is designed to obtain the optimal objective value and cooperative control optimization strategy for all trains. The rest of this paper is organized as follows. Section 2 provides an energy-saving control model of a single train that is multiple objective-driven. In Section 3, the mathematical models for multi-train cooperative energy-saving control are established based on the DRLDA algorithm. In Section 4, the effectiveness of the proposed approach is demonstrated according to the case studies. Section 5 summarizes the main conclusions.

2. Multiple Objective-Driven Energy-Saving Control Model of a Single Train

2.1. Passenger Flow Prediction of URT

When studying the energy-saving train control operation strategy, the influence of passenger flow variations should be considered. Using the metro’s historical passenger flow data to effectively predict the movement of passengers in or out of the stations is of great significance for studying train traction energy consumption and reducing the cost of train operation. In the process of metro passenger flow prediction, when there are large, random fluctuations in passenger flow, the predicting accuracy of traditional models is often greatly affected. Thus, this paper uses the change-point algorithm to handle the time series consisting of passenger flow data. Then, the wavelet transform method is applied to denoise the change point set. Finally, the URT passenger flow is predicted by using the ARMA-RBFNN algorithm.

2.1.1. The Change-Point Algorithm

The passenger flow data are random in nature; these data present continuous variation in a sawtooth shape. Some change points are ignored when predicting directly with the original data, which affects the prediction accuracy. Therefore, the set of change points is defined as b(w). The change-point algorithm is used to obtain the b(w) in the time series of passenger flow. The change-point algorithm can improve the fitting accuracy of the model by dividing the time series of passenger flow into time periods with different characteristics based on qualitative changes in the observed function values at a certain time point. The specific steps of the algorithm are as follows:

Step 1: The cycle number is set as u, the number of change point as w and the auxiliary a variable as e. It is supposed that u = 1, w = 1, e = 1, ${\beta }_j=l_j$ (j = 1, 2, …, k), where l_j is the valley point on the peak curve (or the peak point of the valley curve).

Step 2: On the curve, ${\beta }_u$ is the starting point. Two convex waves are taken along the direction of the time axis, which are denoted as $\left({\beta }_u-{\beta }_{u+1}-{\beta }_{u+2}\right)$. The wave variance of b(w) is calculated within this range. Each point in the wave is taken as the middle point, and then evaluated. The next step is to calculate the maximum and minimum values of $t^{*}$.

$$Y\left({\beta }_u,{\beta }_{u+1},{\beta }_{u+2}\right)={{\displaystyle \sum _{t={\beta }_u}^{{\beta }_{u+1}}\left(x_t-\overline{x}\left|{}_{{\beta }_u}^{{\beta }_{u+1}}\right.\right)}}^2+{{\displaystyle \sum _{t={\beta }_{u+1}+1}^{{\beta }_{u+2}}\left(x_t-\overline{x}\left|{}_{{\beta }_{u+1}+1}^{{\beta }_{u+2}}\right.\right)}}^2$$

(1)

$$\max \left({\beta }_u,t_1{}^{*},{\beta }_{u+2}\right)=\max \left[{{\displaystyle \sum _{t={\beta }_u}^t\left(x_t-\overline{x}\left|{}_{{\beta }_u}^t\right.\right)}}^2+{{\displaystyle \sum _{t=t+1}^{{\beta }_{u+2}}\left(x_t-\overline{x}\left|{}_{t+1}^{{\beta }_{u+2}}\right.\right)}}^2\right]$$

(2)

$$\min \left({\beta }_u,t_2{}^{*},{\beta }_{u+2}\right)=\min \left[{{\displaystyle \sum _{t={\beta }_u}^t\left(x_t-\overline{x}\left|{}_{{\beta }_u}^t\right.\right)}}^2+{{\displaystyle \sum _{t=t+1}^{{\beta }_{u+2}}\left(x_t-\overline{x}\left|{}_{t+1}^{{\beta }_{u+2}}\right.\right)}}^2\right]$$

(3)

where $x_t$ is the value of the passenger flow time series at time t. $t_1{}^{*}$ represents the maximum values. $t_2{}^{*}$ represents the minimum values.

Step 3: Change point estimation:

$$R_u=\frac{2Y\left({\beta }_u,{\beta }_{u+1},{\beta }_{u+2}\right)}{\max \left({\beta }_u,t_1{}^{*},{\beta }_{u+2}\right)}+\min \left({\beta }_u,t_2{}^{*},{\beta }_{u+2}\right)$$

(4)

$$c_{ru}=\frac{\max \left({\beta }_u,t_1{}^{*},{\beta }_{u+2}\right)}{\min \left({\beta }_u,t_2{}^{*},{\beta }_{u+2}\right)}$$

(5)

Step 3.1: If $\max \left({\beta }_u,t_1{}^{*},{\beta }_{u+2}\right)\approx \min \left({\beta }_u,t_2{}^{*},{\beta }_{u+2}\right)$, when $1\le c_{ru}<1.3$, the time series of passenger flow is slightly changed. There is no need to delineate the change point in the range of (l_u,l_u+2).

Step 3.2: If $R_u\le \frac{1}{4}+\frac{3}{2}\left(2c_{ru}+1\right)$, whether it satisfies $t_1{}^{*}={\beta }_{u+1}$ or not, then $b^{(w)}={\beta }_{u+1}$, where $b^{(w)}$ represents a set of change points on the curve.

Step 3.3: If $\frac{1}{4}+\frac{3}{2}\left(2c_{ru}+1\right)\le R_u\le 1$, search for ${\beta }_{u+1}$.

Step 3.4: If $\exists t_c\in \left({\beta }_{u+1}^L,{\beta }_{u+1}^R\right)$, the following condition satisfies $Dev\left({\beta }_u,t_c,{\beta }_{u+2}\right)=\min \left[Dev\left({\beta }_u,{\beta }_{u+1}^L,{\beta }_{u+2}\right),Dev\left({\beta }_u,{\beta }_{u+1}^R,{\beta }_{u+2}\right)\right]$, where Dev(·) indicates variance. $b^{(w)}=t_c$ can be obtained.

Step 3.5: If $R_u>1$, so get $b^{(w)}=t_2^{*}$.

The validity of $b^{(w)}$ is determined; if $T(w-1)>\epsilon +1$, the change point is retained. Otherwise, it is deleted.

Step 3.6: If $b^{(w)}$ exists, so u = u + 1, $b^{(w)}={\beta }_u$, ${\beta }_{u+1}={\beta }_{u+1}$, ${\beta }_{u+2}={\beta }_{u+2}$, w = w + 1, e = 0. Otherwise, e = 1, ${\beta }_u={\beta }_u$, ${\beta }_{u+1+e}={\beta }_{u+1+e}$, ${\beta }_{u+2+e}={\beta }_{u+2+e}$. When ${\beta }_{u+2}={\beta }_k$, the search is complete and the change-point algorithm ends. Otherwise, Step 2 is repeated.

2.1.2. Wavelet Transform Method

The original data are processed again on the basis of wavelet transform to ensure the validity. Wavelet transform has the characteristic of multi-resolution analysis, which can focus on any detail of the passenger flow data for multi-resolution time-frequency domain analysis and can also effectively remove the noise from the passenger flow data. The equation for wavelet transform is as follows:

$$T_x(a,b)=\frac{1}{\sqrt{a}}{\displaystyle \underset{-\infty }{\overset{+\infty }{\int }}x(t)\omega (\frac{\overline{t-b}}{a}})dt=〈x(t),{\omega }_{a,b}(t)〉$$

(6)

where a is the scale of passenger flow data and b is the translation amount of the passenger flow data. x(t) is the time-varying passenger flow data, w(t) is the basic wavelet and ${\omega }_{a,b}(t)$ is a function set of basic wavelets. t is the sampling time interval.

Step 1: The sensor data is input.

Step 2: Wavelet transform multi-scale decomposition is used, including Haar, Db, Sym and Coif.

Step 3: Wavelet coefficients of each scale are denoised and processed with different layers.

Step 4: The signal-to-noise ratio is calculated. Compared with the results, the wavelet basis functions are selected. The final passenger flow data are obtained.

2.1.3. ARMA-RBFNN Algorithm

The data processed by the change-point algorithm and wavelet transform method are non-stationary time series. The non-stationary time series are transformed into stationary time series by difference computing. However, the ARMA model could not predict the nonlinearity of these data. The ARMA-RBFNN algorithm is constructed to reduce the prediction error. The specific steps are described as follows:

Step 1: The data are input after secondary processing.

Step 2: The stationarity, as well as the value of both the autocorrelation function and the partial autocorrelation function, are calculated. This assumes that the covariances of two random variables are defined as x_t and x_t−k, separately, in b(w) with k period intervals. The autocovariance function (defined as cov_k), with the lag of k period, is calculated by Equation (7).

$${\mathrm{cov}}_k=\mathrm{cov}\left[b(w),b(w-k)\right]=E\left[\left(b(w)-\overline{u}\right)\left(b(w-k)-\overline{u}\right)\right]$$

(7)

where $\overline{u}$ is the mean value of the passenger flow time series.

The autocorrelation function of the passenger flow time series is shown as ${\rho }_k=\frac{{\mathrm{cov}}_k}{{\vartheta }_{x_{t-k}}{\vartheta }_{x_t}}$, where ${\vartheta }_{x_t}^2=E\left[x_t-E(x_t)\right]$. When the passenger flow time series is stationary, the autocorrelation function is defined as ${\rho }_k=\frac{{\mathrm{cov}}_k}{{\vartheta }_0}$.

The data sample is presented as ${\widehat{\rho }}_k=\frac{{\displaystyle \sum _{t=1}^{n-k}\left[x_t-\overline{x}\right]}\left[x_{t-k}-\overline{x}\right]}{{\displaystyle \sum _{t=1}^n{\left[x_t-\overline{x}\right]}^2}}$, where $\overline{x}=\frac{{\displaystyle \sum _{t=1}^n\overline{x}}}{n}$. The partial autocorrelation degree of the data sample is defined as ${\widehat{\varphi }}_{kk}=\left\{\begin{array}{ll}{\widehat{\rho }}_k,\hfill & k=1\hfill \\ \frac{{\widehat{\rho }}_k-{\displaystyle \sum _j^{k-1}{\widehat{\varphi }}_{k-1,j}{\widehat{\rho }}_{k-j}}}{1-{\displaystyle \sum _j^{k-1}{\widehat{\varphi }}_{k-1,j}{\widehat{\rho }}_{k-j}}},\hfill & k=2,3,\cdots \hfill \end{array}\right.$.

If the graph has tailing characteristics, it belongs to a stationary time series, and Step 4 should be completed. Otherwise, it belongs to a non-stationary time series, and Step 3 should be completed.

Step 3: Difference computing transforms non-stationary time series into stationary time series. Due to the periodicity of the passenger flow entering the station, periodic differences are established for non-stationary time series.

$$y_t=x_{t+s}-x_t$$

(8)

Step 4: It is assumed that the passenger flow time series consists of linear and nonlinear components, i.e., L(t) is the linear part and N(t) is the nonlinear part.

$$y_t=L(t)+N(t)$$

(9)

Step 5: p and q are determined. The rank of ARMA algorithm is determined by the AIC criterion.

Step 6: Aiming at the linear part of the passenger flow time series, the ARMA model is used for passenger flow prediction. The prediction results are defined as p(t), and the nonlinear prediction error (defined as e(t)) is calculated as follows:

$$e(t)=y_t-p(t)$$

(10)

Step 7: e(w) is randomly divided into two parts: 75% error for the training set and 25% for the test set.

Step 8: The K-MEANS clustering algorithm is applied to determine the radial basis function center. The average method is used for the radial basis function widths so as to determine the core parameters of the RBFNN algorithm.

Step 9: The training set is used to train the RBFNN algorithm. The RBFNN algorithm is used to predict the results of the test set.

Step 10: If the prediction value of RBF is defined as ${\widehat{N}}_t$, the equation represents the prediction result.

$${\widehat{y}}_t={\widehat{L}}_t+{\widehat{N}}_t$$

(11)

2.2. Energy-Saving Control Model of a Single Train

In Section 2.1, the variation pattern of passenger flow was obtained by the ARMA-RBFNN algorithm. When passengers board the train, how to control the train operation and reduce traction energy consumption has become an important part of the train energy-saving model.

2.2.1. Parameters and Notations

As shown in

Table 1. Notation explanation.

Notation	Description
i	The station and station spacing number
j	The train number
$Q_{up}^i$	The number of passengers getting on at the ith station
$Q_{off}^i$	The number of passengers getting off at the ith station
$Q_i$	The number of passengers on the train after departure from the ith station
$T_j^i$	The scheduled operation time of the jth train to the next station
$\Delta t_i$	The optimization time at the ith station
m	The total weight of the train
$m_p$	The weight of the train
$m_{Q_i}$	The weight of the passengers on the train
$S_i$	The distance between the ith and i + 1th stations
$s_i^n$	The position of the nth operating condition conversion point at the ith station, $n\in N$
$t_i^n$	The conversion time of the nth operating condition at the ith station spacing, $n\in N$
$v_{s_i^n}$	The speed at which the position of the nth operating condition conversion point at the ith station
$a_{s_i^n}$	The acceleration at which the position of the nth operating condition conversion point at the ith station
$t_B$	The time between the train braking with maximum braking force and stopping
$F_{s_i^n}$	The force of the nth operating condition at the ith station spacing
$E^{s_i^n}$	The energy consumption of the nth operating condition at the ith station spacing
$d_j^i$	The stopping time of the jth train at the ith station
$u_j^i$	The arrival time of the jth train at the ith station
$r_j^i$	The departure time of the jth train at the ith station
$\Delta T_j^i$	The maximum passenger waiting time at the ith platform
$t_{q_k}^i$	The time of arrival of the kth passenger at the ith platform, $k=1,2,\cdots ,Q_{up}^i$
$\Delta t_{q_k}^i$	The waiting time of the kth passenger at the ith platform
Ftraction	The traction
W	The resistance
B	The braking force

, the relevant notations and parameters throughout this paper are listed below to describe the problem more conveniently.

2.2.2. Operating Condition Analysis

In the force analysis of the train, it can be clearly seen that under the action of different forces, the train travels in different operating conditions. The operating condition of a train during operation can be divided into four types: traction, cruise, coasting and braking. In the traditional sense, there are only three train operating conditions: traction, coasting and braking. Traction conditions include primary traction, secondary traction and multiple traction. Cruising conditions mean that the running resistance and traction force of the train are balanced without resultant force, and the train runs at a fixed speed. Thus, the resultant force is analyzed under each operating condition.

$$F=\left\{\begin{array}{ll}{F}_{traction}-W\hfill & tractionoperatingcontion\hfill \\ 0\hfill & cruiseoperatingcontion\hfill \\ -W\hfill & coastingoperatingcontion\hfill \\ -(B+W)\hfill & brakingoperatingcontion\hfill \end{array}\right.$$

(12)

where F_traction is the traction; W is resistance; B is braking force.

2.2.3. Energy Consumption Calculation

Most studies only calculate the traction energy consumption of the train under the four operating conditions while ignoring the secondary traction, which reduces the accuracy of the results. Thus, due to comprehensive consideration of secondary traction, it is assumed that the train operation process consists of six stages in Figure 1: traction, coasting, secondary traction, cruise, coasting and braking. The energy consumption is calculated, respectively, as follows.

Step 1: Traction stage 0–s₁: the train starts with a certain acceleration traction to overcome the resistance, resulting in traction energy consumption.

$$F_{s_i^1}-m\cdot (1+\gamma )\cdot g\cdot W=m\cdot (1+\gamma )\cdot a_{s_i^1}$$

(13)

$$v=a_{s_i^1}\cdot t_i^1$$

(14)

$$\begin{array}{ll}{E}^{s_i^1}={\displaystyle {\int }_0^{t_i^1}{F}_{s_i^1}}\cdot vdt& ={\displaystyle {\int }_0^{t_i^1}[m(1+\gamma )(g\cdot W+a_{s_i^1})]a_{s_i^1}\cdot tdt}\\ & ={\displaystyle {\int }_0^{t_i^1}M(\Delta t})\cdot (1+\gamma )(g\cdot W+a_{s_i^1})\cdot a_{s_i^1}\cdot tdt\end{array}$$

(15)

where $\gamma$ is the rotary mass coefficient. $\gamma =0.08$.

Step 2: Coasting stage s₁–s₂: the train is only affected by resistance during this process. The speed gradually decreases, and there is no traction energy consumption.

Step 3: Secondary traction stage s₂–s₃: the train tracts for the second time with a certain acceleration, causing traction energy consumption.

$$v_{s_i^2}=v_{s_i^1}-a_{s_i^2}{t}_i^2$$

(16)

$$v_{s_i^3}=v_{s_i^2}+a_{s_i^3}\cdot t_i^3=v_{s_i^1}-a_{s_i^2}{t}_i^2+a_{s_i^3}\cdot t_i^3$$

(17)

$$F_{s_i^3}-m\cdot (1+\gamma )\cdot g\cdot W=m\cdot (1+\gamma )\cdot a_{s_i^3}$$

(18)

$$\begin{array}{ll}{E}^{s_i^3}& ={\displaystyle {\int }_{t_i^2}^{t_i^3}{F}_{s_i^3}\cdot vdt=}{\displaystyle {\int }_{t_i^2}^{t_i^3}m\cdot (1+\gamma )\cdot (g\cdot W+a_{s_i^3})\cdot (v_{s_i^1}-a_{s_i^2}\cdot t_i^2+a_{s_i^3}\cdot t_i^3)dt}\\ & ={\displaystyle {\int }_{t_i^2}^{t_i^3}M(\Delta t)\cdot }(1+\gamma )\cdot (g\cdot W+a_{s_i^3})\cdot (v_{s_i^1}-a_{s_i^2}\cdot t_i^2+a_{s_i^3}\cdot t_i^3)dt\end{array}$$

(19)

Step 4: Cruise stage s₃–s₄: the resultant force on the train is zero. At the same time, the train runs at a constant speed of $v_{s_i^3}$. The traction energy consumption is calculated according to the resistance of the train.

$$F_{s_i^4}=m\cdot (1+\gamma )\cdot g\cdot W=M(\Delta t)\cdot (1+\gamma )\cdot g\cdot W$$

(20)

$$E^{s_i^4}={\displaystyle {\int }_{t_i^3}^{t_i^4}{F}_{s_i^4}}\cdot vdt={\displaystyle {\int }_{t_i^3}^{t_i^4}M\cdot (1+\gamma )}\cdot g\cdot W\cdot v_{s_i^3}dt$$

(21)

Step 5: Coasting stage s₄–s₅: the train is only affected by resistance in this process and performs a deceleration motion; the traction energy consumption is zero.

Step 6: Breaking stage s₅–s₆: the train is only affected by two forces, resistance and braking force, and the traction force does not do any work, so the traction energy consumption is zero.

In summary, the energy consumption of train operation in this zone, defined as E³, is the sum of the energy consumption of operation, consisting of primary traction, secondary traction and the cruising stage, which is calculated by Equation (22).

$$\begin{array}{ll}{E}^3=E^{s_i^1}+E^{s_i^3}+E^{s_i^4}& ={\displaystyle {\int }_0^{t_i^1}M(\Delta t})\cdot (1+\gamma )(g\cdot W+a_{s_i^1})\cdot a_{s_i^1}\cdot tdt\\ & +{\displaystyle {\int }_{t_i^2}^{t_i^3}M(\Delta t)\cdot }(1+\gamma )\cdot (g\cdot W+a_{s_i^3})\cdot (v_{s_i^1}-a_{s_i^2}\cdot t_i^2+a_{s_i^3}\cdot t_i^3)dt\\ & +{\displaystyle {\int }_{t_i^3}^{t_i^4}M(\Delta t)\cdot (1+\gamma )}\cdot g\cdot W\cdot (a_{s_i^1}\cdot t_i^1-a_{s_i^2}\cdot t_i^2+a_{s_i^3}\cdot t_i^3)dt\\ & =H(\Delta t_i,t_i^1,t_i^2,t_i^3,t_i^4)\end{array}$$

(22)

2.2.4. Single-Train Control Model

The process of urban rail transit operation can be divided into two main bodies: passengers and operators. From the perspective of passengers, there is a “benefit paradox” relationship between the high demand of passengers for a certain service level and train operation energy consumption. Passengers always hope to board the train as soon as it arrives as well as for the waiting time at the platform to be shortened. However, such hopes lead to an increase in train running speed and traction energy consumption, which, in turn, increases operation cost. From the perspective of the operator, the operator wants to appropriately reduce the train’s running speed and traction energy consumption in order to lower operation costs while transporting as many passengers as possible and ensuring the safety of train operation. The contradiction between supply and demand between the two main bodies is becoming increasingly prominent and influential. Therefore, this paper comprehensively considers strategies to meet the high service level requirements for passengers and co-optimize the traction energy consumption of trains.

Step 1: Control optimization objectives

• Average passenger waiting time

In an actual situation, when passengers arrive at the platform to wait for the train, they automatically disperse according to the current queuing situation. Finally, the number of people queuing in front of each door tends to be balanced. Assuming that the travel time of passengers between the gate and the platform is ignored, the passenger waiting time is calculated as follows.

The departure time of the train at the ith platform is defined as $r_j^i$, which is calculated by Equation (23).

$$r_j^i=T_j^i+\Delta t_i+d_j^i$$

(23)

Similarly, the departure time of the previous train at the ith platform is defined as $r_{j-1}^i$, which is shown as Equation (24).

$$r_{j-1}^i=T_{j-1}^i+\Delta t_i+d_j^i$$

(24)

The maximum passenger waiting time at the ith platform is calculated by Equation (25).

$$\Delta T_j^i=T_j^i+\Delta t_i-r_{j-1}^i=T_j^i+t_i-(T_{j-1}^i+\Delta t_i+d_j^i)=T_j^i-T_{j-1}^i-d_j^i$$

(25)

The number of passengers waiting to board the train at the ith platform is the number of passengers arriving during the time difference between the departure of the previous train from the platform and the arrival of this train at the platform.

$$Q_{up}^i=f_i(\Delta t_i)={\displaystyle {\int }_1^{T_j^i+\Delta t_i}{h}_i}(t)dt$$

(26)

The time when the kth passenger arrives at the ith platform is defined as $\Delta t_{q_k}^i$; the passenger waiting time at the ith platform is calculated by Equation (27).

$$\Delta t_{q_k}^i=T_j^i+\Delta t_i-t_{q_k}^i$$

(27)

Therefore, the average passenger waiting time is calculated by Equation (28).

$$\overline{\Delta t_{q_k}^i}=\frac{{\displaystyle \sum _{k=1}^{Q_{up}^i}\Delta t_{q_k}^i}}{Q_{up}^i}=\frac{{\displaystyle \sum _{k=1}^{{\displaystyle {\int }_1^{T_j^i+\Delta t_i}{h}_i(t)dt}}(T_j^i+\Delta t_i-t_{q_k}^i)}}{{\displaystyle {\int }_1^{T_j^i+\Delta t_i}{h}_i(t)dt}}$$

(28)

• Traction energy consumption

The total traction energy consumption of the train on the whole line is the comprehensive energy consumption between every two adjacent stations. It consists of traction energy consumption and cruise energy consumption. Assuming that the train line has I-1 station spacing, it can be seen from Equation (22) that the total traction energy consumption of the jth train on this line is calculated by Equation (29).

$$E_j={\displaystyle \sum _1^{I-1}{E}^3}={\displaystyle \sum _1^{I-1}H(\Delta t_i},t_i^1,t_i^2,t_i^3,t_i^4)$$

(29)

Step 2: Constraint condition

To ensure safe train operation, the train operating status needs to be subject to the following constraints:

• Speed constraint

According to multiple factors, such as the physical conditions of the train line, train operating safety and passenger comfort, the train operation will be limited by the maximum speed, v_max.

In order to ensure the punctual arrival of the train and maximize the transportation efficiency, the train will be restricted by a minimum speed, v_min, during traction operation. When the train runs in coasting conditions, the train speed is close to the minimum speed, v_min. At this time, the train should be accelerated immediately to ensure that the running speed is greater than v_min, to guarantee that the train can reach the next station on time.

• Train dwell time constraint

When a train enters a station and stops, it needs to be given some time for passengers to fully disembark/embark before moving to the next station. Thus, the train dwell time (defined as $d_j^i$) should meet the following conditions.

$$d_{\min }\le d_j^i\le d_{\max }$$

(30)

• Passenger waiting time constraint

Considering the complex psychological characteristics of passengers, the train should meet their traveling and psychological needs. Therefore, the maximum waiting time that passengers can accept at the platform is defined as ${(\Delta t_q)}_{\max }$, i.e., passenger waiting time (defined as $\Delta t_{q_k}^i$) is calculated by Equation (31).

$$0<\Delta t_{q_k}^i\le {(\Delta t_q)}_{\max }$$

(31)

Step 3: Modeling

Considering the cost benefit of the two main bodies, namely the operator and the passenger, a single train combination optimization model is established that aims to balance the traction energy consumption and the average passenger waiting time. The model control objectives are summarized as follows.

$$\begin{array}{l}\min J=\alpha \cdot E_j+\beta \cdot \overline{\Delta t_{q_k}^i}=\alpha \cdot H(\Delta t_i,t_i^1,t_i^2,t_i^3,t_i^4)+\beta \cdot K(\Delta t_i)\\ s.t.\left\{\begin{array}{l}\Delta t_i\in (0,60)\hfill \\ t_i^1+t_i^2+t_i^3+t_i^4+t_B\le T_j^i+\Delta t_i\hfill \\ v_{\min }\le v_{s_i^n}\le v_{\max }\hfill \\ d_{\min }\le d_j^i\le d_{\max }\hfill \\ 0<\Delta t_{q_k}^i\le {(\Delta t_q)}_{\max }\hfill \end{array}\right.\end{array}$$

(32)

• Step 4: The model is solved by GA.

Step 4.1: The process is initialized and basic data are input, including line name, train data, train model parameters, single side slope, curve radius, speed limit of running line, etc.

Step 4.2: The control model and its objective function are determined based on Equation (32).

Step 4.3: The encoding mode is determined. The real number encoding mode represents the conversion point $s_i^n$ in train operating conditions. Since $s_i^n$ is a model variable, the conversion point is represented according to genes. The chromosome represents the sequence of conversion points. A chromosome contains multiple genes: $S_I=\left\{s_i^1,s_i^2,\cdots ,s_i^n\right\}$.

Step 4.4: The reciprocal of the objective function is selected as the fitness function to dynamically search the internal information, and then the individual evaluation is conducted.

$$fit=\frac{1}{\alpha \cdot E_j+\beta \cdot \overline{\Delta t_{q_k}^i}}$$

(33)

Step 4.5: GA is designed according to roulette selection and adaptive crossover/mutation operation.

Step 4.6: Operation parameters, including population size, iteration times, selection operator, crossover operator, mutation operator, etc., are determined.

Step 4.7: It is judged whether genetic algebra has been reached. If reached, the result is output and the algorithm ends. Otherwise, Step 4.4 should be repeated and the calculation should be continued. The flowchart of GA is shown in Figure 2.

3. Multi-Train Cooperative Energy-Saving Control Model Based on Multi-Objective Combinatorial Optimization

3.1. Scenario Assumptions

When studying a multi-train cooperative control method, it is necessary to optimize the arrival time, departure interval and maximum interval speed at each station while considering the benefits of both passengers and operators. Combined with the actual operation of the train, the following assumptions are made.

For passengers: on one hand, it is assumed that passengers show no transfer behavior. On the other hand, when the train stops regularly at a station, it is assumed that passengers board the train in an orderly manner, and the mechanism of “first come, first served” is adopted.

For train operation: on one hand, it is assumed that the trains depart in order at the starting station, only one train is allowed to stop at each station at the same time and trains cannot overtake the previous train on the route. On the other hand, it is assumed that the train follows the “stop at station by station” principle during operation. The trains are not allowed to stop during inter-station operation, i.e., trains can only stop at stations.

3.2. Constraint Condition

When multiple trains are running on the line, in addition to meeting the constraints of the single train control model, they also satisfy the train safety interval constraints. Only one train is allowed to stop at the station and cannot stop on the running line based on scenario assumption. Thus, a safety interval is required between two trains. The minimum safety interval of the train includes the maximum braking distance and safety protection distance required for emergency braking. To simplify the calculation, the minimum safe distance of the train is defined as T_aq.

$$T_j^i-T_{j-1}^i\ge T_{aq}$$

(34)

3.3. Methodology

3.3.1. Modeling

Step 1: The “queuing theory” is introduced to study the relationships among time-varying passenger flow prediction, arrival interval and dwell time. Service time is defined as train dwell time. Train arrival interval ($\Delta T_j^i$) is defined as vacation time. Therefore, trains that enter and leave the stations and passengers who embark/disembark belong to a service queuing model, i.e., a single vacation system in which every service enters a vacation. The average queue length of waiting passengers is defined as L. Arrival interval is defined as $\Delta T_j^i$. Dwell time is defined as $d_j^i$. According to Ning, Niu and Zhang [53]:

$$L=\tilde{\rho }+\frac{{\lambda }^2[b^2+\frac{2}{d_j^i}E(\Delta T_j^i)+E({(\Delta T_j^i)}^2)]}{2[1-\rho -\tilde{\lambda }E(\Delta T_j^i)]}+\frac{\tilde{\lambda }E({(\Delta T_j^i)}^2)}{2E(\Delta T_j^i)}$$

(35)

where $\overline{\lambda }$ represents the input conversion parameter of passenger arrival rate $\lambda$. $\rho$ and $\tilde{\rho }$ both represent the service intensity, i.e., $\rho =\frac{\tilde{\lambda }}{\mu }$ and $\tilde{\rho }=\rho +\tilde{\lambda }E(\Delta T_j^i)$. b² represents the second moment of service time distribution.

Considering Equation (26):

$$Q_{up}^i=f_i(\Delta t_i)={\displaystyle {\int }_{T_{j-1}^i+\Delta t_i+d_j^i}^{T_j^i+\Delta t_i}{h}_i(t)dt}$$

(36)

The average passenger arrival rate during the arrival interval is calculated by Equation (37).

$$\overline{{\lambda }_i}=\frac{Q_{up}^i}{\Delta T_j^i}=\frac{{\displaystyle {\int }_{T_{j-1}^i+\Delta t_i+d_j^i}^{T_j^i+\Delta t_i}{h}_i(t)dt}}{T_j^i-T_{j-1}^i-d_j^i}$$

(37)

The average passenger waiting time is calculated by Equation (38).

$$\Delta T=\frac{L_i}{\overline{{\lambda }_i}}$$

(38)

The service time for one passenger is defined as Y_k; the total service time for all arriving passengers is calculated by Equation (39).

$$X(t)={\displaystyle \sum _1^{Q_{up}^i}{Y}_k}$$

(39)

As the dwell time of the train is certain, the service time of each passenger is the same, and is calculated by Equation (40).

$$E\left\{X(t)\right\}=\lambda tE(Y_k)$$

(40)

The average service time of passengers per unit time is calculated by Equation (41).

$$H=\frac{E\left\{X(t)\right\}}{t}$$

(41)

The minimum service intensity of the train is Ds/person within the dwell time of a train, which is calculated by Equation (42).

$$\tilde{\lambda }=\frac{H}{D}=\frac{\lambda E(Y_k)}{D}$$

(42)

If the arrival time of the train obeys the negative exponential distribution, the dwell time obeys the deterministic distribution, and the second moment is $b^2=D^2$. Service time is $D=u_j^i$.

The arrival interval is expressed based on the actual arrival and departure times:

$$E(\Delta T_j^i)=u_j^i-u_{j-1}^i=T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i$$

(43)

$$E[{(\Delta T_j^i)}^2]=D(\Delta T_j^i)+{[E(\Delta T_j^i)]}^2=2{(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)}^2$$

(44)

Equations (36)–(44) are substituted into Equation (45), giving Equation (45).

$$\begin{array}{ll}\Delta T& =\frac{1}{\overline{\lambda }}\left\{\begin{array}{l}\frac{\tilde{\lambda }}{d_j^i}+\tilde{\lambda }(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)+\frac{2\tilde{\lambda }{(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)}^2}{2(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)}\\ +\frac{{(\tilde{\lambda })}^2[{(d_j^i)}^2+2{(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)}^2+2{(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)}^2]}{2[1-\frac{\tilde{\lambda }}{d_j^i}-\tilde{\lambda }(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)]}\end{array}\right\}\\ & =P(\tilde{\lambda },\Delta t_j^i)\end{array}$$

(45)

where the service time is the minimum dwell time and the number of passengers that the train can serve is Q. The remaining number of passengers for a given service time is defined as $\left|\overline{\lambda }\times \Delta T-Q\right|$.

During vacation time, when $\overline{\lambda }\times \Delta T>Q$, the train arrival interval should be shortened to reduce the number of waiting passengers. The dwell time correspondingly extends and the number of passengers on board should increase. When $\overline{\lambda }\times \Delta T<Q$, the arrival interval is extended to reduce the traction energy consumption of the train. The optimal passenger waiting time is taken as the control objective. The function is established by Equation (46).

$$\begin{array}{l}{Y}_1={\displaystyle \sum _{i=1}^I{\displaystyle \sum _{j=1}^J\Delta T}}\\ s.t.\left\{\begin{array}{l}\tilde{\lambda }{d}_j^i+\tilde{\lambda }(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)<1\\ T_j^i-T_j^i\ge T_{aq}\end{array}\right.\end{array}$$

(46)

Step 2: Traction energy consumption of multiple trains

The multi-train cooperative model is not a single objective optimization problem, but a multi-objective optimization problem. The main optimization objectives are described as follows:

• Passenger waiting time

Passenger waiting time is naturally formed when passengers arrive at the platform through the gate and wait for the train to arrive. As known in Section 3.3, $\Delta T_j^i$ is defined as the vacation time of the queuing model. The passenger waiting time can be expressed as follows.

$$Y_1={\displaystyle \sum _{i=1}^I{\displaystyle \sum _{j=1}^J\Delta T}}$$

(47)

• Traction energy consumption

According to a single train energy-saving control model, a multi-train traction energy consumption control function is calculated by Equation (48).

$$Y_2={\displaystyle \sum _1^J{E}_j}={\displaystyle \sum _1^J{\displaystyle \sum _1^{I-1}{E}^3}}$$

(48)

Combined with the weight summation method, the energy-saving cooperative control method for multiple trains is constructed by the train operation safety, equipment working status and other factors or constraint conditions, which can be expressed as follows.

$$\begin{array}{l}\min Y=\theta Y_1+\omega Y_2\\ s.t.\left\{\begin{array}{l}\Delta t_j^i\in (0,60)\\ t_i^1+t_i^2+t_i^3+t_i^4+t_B\le T_j^i+\Delta t_i\\ v_{\min }\le v_{s_i^n}\le v_{\max }\\ T_j^i-T_{j-1}^i\ge T_{aq}\\ d_{\min }\le d_j^i\le d_{\max }\\ T_{aq}\le \Delta T\le {(\Delta t_q)}_{\max }\\ \tilde{\lambda }{d}_j^i+\tilde{\lambda }(T_j^i+\Delta t_j^i-T_{j-1}^i-\Delta t_{j-1}^i)<1\end{array}\right.\end{array}$$

(49)

3.3.2. DRLDA Algorithm for Multi-Train Energy-Saving Control Strategy

The external force, speed, dwell time, passenger waiting time, traction energy consumption and other factors are multiple basic variables that are interrelated and mutually restricted in the train operation control process. The speed, passenger flow, position and distance are the cumulative results of external forces on a train over time. In turn, the choice is also affected by the state of other basic variables. Moreover, when a train arrives at the station, the speed should reach a specified value, such as zero, and the actual operation time must be less than or equal to the planned operation time. At the same time, the relationships among time-varying passenger flow prediction, train arrival time, dwell time and traction energy consumption should also be considered before the train enters the station. During the train’s operation, the speed must not exceed the speed limit of the specific location. To deal with these complex constraints, the implementation of multi-train energy-saving control using DRL is a challenging cooperative optimization problem, because the learning process will involve a long-term trial and error process. Thus, the DRL is suitable to be combined with DA to handle the constraints. The learning system retains the global exploration and learning ability of reinforcement learning (RL), avoiding the locality and incompleteness of pure RL-guided learning. Under DA hybrid learning, it compares the current state of DRL with its own expectations. If they are inconsistent, it means that the agent has made an incorrect decision. Then, DA intervenes with the agent in time through weight distribution. The agent receives specific feedback actions and instructs the controller to make a reliable decision about the state. Finally, the input data for a given train/passenger flow and railway line are used to learn from the train energy consumption and passenger travel data to achieve the goals of safety, energy saving, comfort, punctuality, etc.

RL can make the optimal decision by learning the optimal control strategy incrementally, so as to maximize the cumulative reward value and improve the system performance. DRL combines the perception ability of Deep Learning (DL) with the decision-making ability of RL. Before applying the DRLDA algorithm to the multi-train energy saving control, the basic elements are defined as state space, action space, reward mechanism setting, strategy and value function, respectively.

Step 1: State space

To accurately present the current train information, multiple parameters are selected as environment states, which are related to different attributes of the environment: speed (defined as $v_{s_i^n}$), passenger waiting time (defined as Y₁), traction energy consumption (defined as Y₂), current position (defined as x_i) and the remaining operating time of the train (defined as $t_{res{t}_i}$). The state space is defined as Equation (50):

$$\begin{array}{l}{s}_i=[v_{s_i^n},Y_1,Y_2,x_i,t_{res{t}_i}]\\ s.t.\left\{\begin{array}{l}{v}_{s_i^n}\in \left[v_{\min },v_{\max }\right]\hfill \\ Y_1\in [T_{aq},\Delta T]\hfill \\ Y_2\in [0,E_j]\hfill \\ x_i\in \left[0,x_{end}\right]\hfill \\ t_{res{t}_i}\in [0,t_{plan}]\hfill \end{array}\right.\end{array}$$

(50)

where x_end is the end position of the railway line. $t_{res{t}_i}=t_{plan}-t_{operting}$ represents the rest operating time of a train. t_plan is the train travel planning time. t_operating indicates the time that the train has been running.

The process of train operation is discrete; at each time point t, there are two kinds of states for trains, i.e., the stop at station state and the running state. For the stop at station state, the value ranges for the variables are $v_{s_i^n}=0$, $\Delta T\ge Y_1\ge T_{aq}$, $Y_2=0$, $x_i\in x_{stop}$, $0\le t_{res{t}_i}\le t_{dwell}=d_{\max }$ to satisfy the upper limits of dwell time and train speed. For the running state, when train runs at the railway line, the value ranges for the variables are $v_{\min }\le v_{s_i^n}\le v_{\max }$, $\Delta T\ge Y_1\ge T_{aq}$, $Y_2\in E_j$, $x_i\in x_{end}-x_{stop}$, $t_{dwell}-\Delta t\le t_{res{t}_i}\le t_{res{t}_i}^{\max }$ (where $\Delta t$ is time intervals), and should satisfy the speed limit and the upper limit of the remaining operating time. During state transition, the state space s_i of multiple trains can be obtained by generating all candidate states within the value range.

Step 2: Action space

Trains generate corresponding traction or braking forces to control multiple operation based on the continuous output of the main control handle or URT system. Therefore, the action space A is composed of the variation tendency of the control force level (defined as CFL_i) according to the current operating states. The action set is defined as follows:

$$\begin{array}{l}A=\left\{a_{n_i}\right\}=\left|CF{L}_i\right|,CF{L}_i\in \left[-100\%,100\%\right]\\ s.t.\left\{\begin{array}{l}{U}_1=CF{L}_i\ast F_{\mathrm{traction}\_\max }CF{L}_i>0\hfill \\ U_2=CF{L}_i\ast F_{\mathrm{coasting}}CF{L}_i<0\hfill \\ U_3=CF{L}_i\ast F_{\sec \mathrm{ondary}\text{\_}\mathrm{traction}}CF{L}_i>0\hfill \\ U_4=0\ast CF{L}_i=0\hfill \\ U_5=CF{L}_i\ast B_{\max }CF{L}_i<0\hfill \end{array}\right.\end{array}$$

(51)

where U_i represents the different effects under different operating conditions and running states. F_{traction_max} and B_max are the maximum traction force and the maximum braking force of the train, respectively. F_{secondary_traction} is the secondary traction force. F_coasting is the train coasting force. The train with variation tendency of the control force level range is [−100%, 100%]. The maximum traction force is 100%, and the maximum braking force is −100%. A value of 0% means that train runs in a cruise state.

Step 3: Reward mechanism setting

An agent is able to randomly generate control strategies based on the definition of state and action. Reward function is of great importance in the realistic optimization objective field of RL applications. Different reward values can guide the algorithm to learn better control strategies in order to maximize the benefit. A reasonable reward and punishment mechanism can improve the convergence speed. Generally, under the cooperative setting, the common reward (defined as $R_{\mathrm{total}}\left(s_i,a_{n_i}\right)$) is obtained by the independent individual reward and the cooperative individual reward based on the relationship between multiple trains. In order to reduce the energy consumption of all trains, $R_{\mathrm{total}}\left(s_i,a_{n_i}\right)$ is calculated by Equation (52).

$$R_{\mathrm{total}}\left(s_i,a_{n_i}\right)=\min Y=\theta Y_1+\omega Y_2$$

(52)

Step 4: Strategy and value function

In RL, the train selects an action mapping a probability distribution to execute in the current state, which is defined as strategy matching $\pi$. The agent selects a dynamic state-action function Q with a maximum value. The current state-action function Q is defined as:

$$Q\left[s_i,\pi \left(s_i\right)\right]=E\left\{\left(1-\kappa \right)R_{\mathrm{total}}\left[s_i,\pi \left(\left.a_{n_i}\right|s_i\right)\right]+\kappa Q\left[s_{i+1},\pi \left(\left.a_{n_i}\right|s_{i+1}\right)\right]\right\}$$

(53)

where $\kappa$ is a discount factor; the larger $\kappa$ is, the greater the focus on future cumulative returns. The cumulative return value of the action strategy is defined as a function $J\left(\pi \right)$, calculated by $J\left(\pi \right)=E_{~\tau }\left\{Q\left[s,\pi \left(\left.a\right|s\right)\right]\right\}$. Thus, the optimal control strategy can be calculated by Equation (54).

$$optima{l}_{\pi }=\arg {\max }_{\pi }J\left(\pi \right)$$

(54)

Step 5: DRLDA strategy design

This algorithm consists two kinds of networks, i.e., actor network and critic network, experience replay pool, multi-train energy-saving control model and reward function. The DRLDA algorithm learning framework is illustrated in Figure 3. The two main networks, including the actor network and the critic network, output the train state and control strategy to the train cooperative model to simulate the train operation process. The reward function evaluates the action strategy. Then, the reward values and train states are stored in an experience replay pool through a replay memory mechanism. When the pool is full of data, small batches of random samples are used to train the two main networks, which can effectively reduce the correlations among samples and improve the learning speed. The training process of the proposed algorithm is divided into a state-action function update and an action strategy update. Two main networks combine the advantages of value function update and strategy function update. The critic network adopts the single-step update method based on the value function to evaluate the current multi-train control strategy. The network parameter is defined as ${\xi }^Q$. The actor network adopts the strategy gradient update method, which can better solve the problem of continuous action space, and is used to output the multi-train energy-saving control strategy. The network parameter is defined as ${\xi }^{\pi }$.

• Critic Network

The critical network is similar to the Deep Q-Network (DQN) algorithm, which uses a fully connected neural network to fit each state action value, that is, $Q\left(s,A\left|{\xi }^Q\right.\right)=Q\left(s,A\right)$. The state observations and the actions of sample data are taken as the input. The state action value Q is selected as the output. A larger Q value indicates a higher evaluation of that action strategy. The new weight of the Q neural network parameters is calculated by Equation (55).

$${\xi }_{t+1}^Q={\xi }_t^Q+{\zeta }_Q\Delta y\nabla Q\left(s,a\left|{\xi }_t^Q\right.\right)$$

(55)

$\Delta y={y^{\prime }}_t-y_t$ is established, where ${y^{\prime }}_t=R_{tota{l}_t}+\kappa Q_{t+1}\left[s_{t+1},\pi {\left(s_i\right)}_t\left(s_{t+1}\left|{\xi }^{\pi {\left(s_i\right)}_t}\right.\right)\left|{\xi }^Q\right.\right]$, $y_t=Q_t\left[s_t,\pi {\left(s_i\right)}_t\left(s_t\left|{\xi }^{\pi {\left(s_i\right)}_t}\right.\right)\left|{\xi }^Q\right.\right]$. When calculating, the above equations are substituted into Equation (55). Where ${y^{\prime }}_t$ is target value, ${\zeta }_Q$ is the learning rate. $\nabla Q\left(s,a\left|{\xi }_t^Q\right.\right)$ is the update gradient calculated by the critical neural network based on the Q value, which is output to the actor network as the basis for updating the action strategy. If the gradient is positive, it means that the optimal strategy is close to this control strategy. The larger the gradient, the faster the approach speed.

• Actor Network

The division of action space has a great impact on RL. If the space is too small, it is impossible to accurately describe the train movement and operation process. If the division space is too large, it leads to too large a dimensionality, so the train cannot learn all the actions. The actor network is applied to Q neural network parameters to estimate the train control strategy. The state observations of the sample data are taken as the input, and the action strategy is taken as the output. The different types of strategies can be divided into stochastic and deterministic strategies. In the process of train operation, there is a certain optimal control strategy in each state. Thus, this paper selects the deterministic strategy. The deterministic strategy outputs the corresponding optimal action in the current state, that is, $a=\pi {\left(s_i\right)}_{\xi }$. According to gradient ascent (Lillicrap et al. [54]), the derivative of $J\left(\pi \right)$ is calculated for obtaining the strategy gradient, which is shown in Equation (56).

$${\nabla }_{{\xi }^{\pi \left(s_i\right)}}J\left(\pi \right)\approx E\left[{\nabla }_{a}Q\left(s,a\left|{\xi }^Q\right.\right)\left|{}_{a=\pi \left(s_i\right)}\right.\cdot {\nabla }_{{\xi }^{\pi \left(s_i\right)}}\pi \left(s\left|{\xi }^{\pi \left(s_i\right)}\right.\right)\right]$$

(56)

When performing a gradient ascent to update strategies in a continuous control problem, the maximization of J maximizes the corresponding Q value. Therefore, an attempt is made to suggest that the agent proactively learns the random sampling data, and the hybrid control increments ($\Delta {\pi }_{{\xi }_i}^{hr}$, i = 1, 2, …, N) are brought into Equation (56) as the unbiased estimation of the expectation. Equation (57) is rewritten to obtain the calculation of the strategy gradient.

$$\begin{array}{ll}{\nabla }_{{\xi }^{\pi \left(s_i\right)}}J\left(\pi \right)& \approx \frac{\upsilon }{N}{\displaystyle \sum _i\left[{\nabla }_{a}Q\left(s,a\left|{\xi }^Q\right.\right){\nabla }_{{\xi }^{\pi \left(s_i\right)}}\pi \left(s\left|{\xi }^{\pi \left(s_i\right)}\right.\right)\right]}\\ & +\frac{1-\upsilon }{N}{\displaystyle \sum _i\left[\pi \left(s\left|{\xi }^{\pi \left(s_i\right)}\right.\right)-\Delta {\pi }_{{\xi }_i}^{hr}\right]}{\nabla }_{{\xi }^{\pi \left(s_i\right)}}\pi \left(s\left|{\xi }^{\pi \left(s_i\right)}\right.\right)\end{array}$$

(57)

where $s=s_i$ and $a=\pi \left(s_i\right)$ are new weight calculation equations of the Q neural network parameters, respectively; that is, ${\xi }_{t+1}^Q={\xi }_t^Q+a_{\pi }{\nabla }_{{\xi }^{\pi \left(s_i\right)}}J(\pi )$.

• Parameters Update and Their Mechanism

To increase the stability of algorithm training and reduce large changes to the neural network, as shown in Equations (58) and (59), the actor network and critic network should track the learning by “soft update” instead of directly changing the weights.

$${\xi }_{t+1}^Q=\sigma {\xi }_t^Q+(1-\sigma ){\xi }_{t-1}^Q$$

(58)

$${\xi }_{t+1}^{\pi \left(s_i\right)}=\psi {\xi }_t^{\pi \left(s_i\right)}+(1-\psi ){\xi }_{t-1}^{\pi \left(s_i\right)}$$

(59)

where ${\xi }_t^{\pi \left(s_i\right)}$ and ${\xi }_t^Q$ are the parameters of the actor network and critic network before updating, respectively. ${\xi }_{t+1}^{\pi \left(s_i\right)}$ and ${\xi }_{t+1}^Q$ are calculated as the parameters of the actor network and critic network after updating, respectively.

The RL in continuous action space generally performs exploration by adding noise in order to learn new action strategies; that is, $a_t=\pi \left(s\left|{\xi }^{\pi \left(s_i\right)}\right.\right)+\Re$, where $\Re$ refers to the noise process. It is of great importance to improve the exploration efficiency. According to the actual train operation, the maximum traction level of the control strategy after adding noise is limited to no more than 100%, and the maximum braking level is no less than −100%.

• The Pseudo Code of DRLDA Algorithm (Algorithm 1)

Algorithm 1: DRLDA algorithm for multi-train energy-saving control model

1: Initialize replay memory and the common reward, $R_{\mathrm{total}}\left(s_i,a_{n_i}\right)$

2: Initialize actor network and critic network.

3: Initialize target actor network and critic network.

4: For i = 1:

5:        Reset state space si.

6:        Initialize a random process $\mathbb{Z}\mathrm{for}Action\left|State\right.$

7:        For step i = 1 to t:

8:               Select action $\Delta {\pi }_i=\pi \left(s_i\left|{\xi }^{\pi \left(s_i\right)}\right.\right)+{\mathbb{Z}}_i$ based on current strategy and noise.

9:           Set actor network, critic network and parameters update and its mechanism with ${\xi }^Q\mathrm{and}{\xi }^{\pi }$

10:       Obtain hybrid controller output, $\Delta {\pi }_{{\xi }_i}^{hr}$

11:       Execute simulation using ${\pi }_{{\xi }_i}\leftarrow {\pi }_{{\xi }_i}^{hr}$ and observe reward $R_{{\mathrm{total}}_i}$

12:       Obtain next state, si + 1.

13:       Calculate ultimate action increment, $\Delta {\pi }_{{\xi }_i}={\pi }_{{\xi }_i}-{\pi }_{{\xi }_{i-1}}$

14:       Store $\left(s_i,\left|CF{L}_i\right|,R_{total},Q_i\right)$ in replay memory.

15:       Sample random mini-batch of $\left(s_i,\left|CF{L}_i\right|,R_{total},Q_i\right)$ with size N.

16:       Set $J\left(\pi \right)$

17:         Initialize DA parameters, including the speed and position of population, the initial value of the weight, the number of iterations, the radius, etc.

18:           Calculate the objective function value of each Dragonfly individual according to Equation (49).

19:       Release the feasibility of all objective functions to external files.

20:          Compare the number of feasible solutions put into the external file with the maximum capacity of the external file. If it is greater than the maximum capacity, delete the redundant feasible solutions according to the dynamic maintenance strategy. If it is less than the maximum total, go to the next step.

21:             Update the weight factor of Dragonfly behavior and radius parameters.

22:             Update the population location by using the mixed mutation method.

23:             Update the critic network using Equation (55).

24:             Update the actor network using the sampled strategy gradient, Equation (56).

25:             Update the target networks in a soft manner.

26:             Condition judgment: if the maximum number of iterations is reached, execute the next step; if the maximum number of iterations is not reached, go to Step 18.

27:             Output the optimal solution.

28:      End

29: End

4. Case Study

In order to evaluate the effectiveness of the DRLDA algorithm and the ARMA-RBFNN-GA method, we used actual sensor data from Beijing Metro Line 4 (such as Xinjiekou, Pinganli, Xisi and Lingjinghutong) for simulation experiments, as shown in Figure 4.

In practice, these data include train parameters, passenger flow data and railway line data. The train parameters were a type B train with 6-train marshaling, SFM05 model, a total staffing capacity of 1440, etc. The train adopted DC 750 V third rail power supply. The designed maximum speed was 80 km/h. The passenger flow data were derived from historical operation data, including dynamic passenger flow demand data and train operation schedule data. For example, the metro passenger flow data were collected from the Automatic Fare Collection (AFC) data, survey data were provided by the Beijing Metro Company and the railway passenger flow data were obtained from statistics provided by the Passenger Transport Center, Beijing Railway Administration. Since the railway timetable is affected by many factors, it cannot be easily changed. The planning time range for the metro should be determined based on the Beijing Metro Line 4 (such as Xinjiekou, Pinganli, Xisi and Lingjinghutong) and the actual travel time range of passengers. In addition, in order to ensure that the case study was representative, we investigated the original passenger flow data of the Xinjiekou, Pin’anli, Xisi and Lingjinghutong stations of Beijing Line 4 from 7:00 to 9:00 on weekdays for one month, with 30 s interval statistics. The passenger flow data from one week is visualized in Figure 5. Railway line data include stations, line length, planned train operating time, speed limit, etc. Actual energy consumption data were obtained by the Beijing Metro Company.

As shown in Figure 5, passengers embarking on/disembarking from the Beijing metro conform to the actual passenger flow distribution.

In AnyLogic simulation software, as shown in Figure 6, the pedestrian library was used to construct a simulation of passengers entering and leaving the station and queuing for trains. The PedSource module was used to generate passengers. The passengers were shown to enter the platform through the gate, disperse in front of each door and wait for the train to arrive and stop before boarding. Then, the PedSink module was applied to demonstrate the disappearance of the passengers. Some passengers disembarked from the train, made their way from the platform to the gates and left through those gates. These passengers disappeared, thus completing the simulation of passengers entering and leaving the station and embarking on/disembarking from the train.

When building the train operation simulation, the Train Source module was used to generate the train. In the simulation, the train runs through track operation into the platform, which was completed by the Train Move To module. The TrainUpload module was used to represent the process of passengers embarking on/disembarking from the train. Then, the train goes to the next station. According to the actual operation of the train, several simulation parameters of the train (such as train and line length, running speed, etc.) were set with the property inspector of Train Source and Train Move To, as shown in Figure 7.

4.1. Comparison Results of Passenger Flow Prediction

To test and verify the performance of the ARMA-RBFNN algorithm, the original passenger flow data from the Xinjiekou, Pin’anli, Xisi and Lingjinghutong stations of Beijing Line 4 were collected from 7:00 to 9:00 on weekdays for one month. This paper takes 3–4 s as the interval period to process the passenger flow data by using the change-point algorithm. As shown in Figure 8, the change-point algorithm can search for the passenger flow data with change points; here, we use the Xinjiekou station as an example.

After the change-point algorithm processing, wavelet functions are used to denoise the passenger flow data, including Haar wavelet, db wavelet, sym wavelet and coif wavelet. A variety of wavelet decomposition levels are used for denoising. The signal-to-noise ratio (SNR) is calculated by four kinds of wavelet functions and their changes under different decomposition levels, which are shown in

Table 2. The signal-to-noise ratio of passengers embarking on the train.

Wavelet Function	Haar	db2	db3	db4	db5	db6	Sym2	Sym3	Sym4	Sym5	Sym6	Coif1	Coif2	Coif3	Coif4	Coif5
SNR	25.55	34.93	34.95	35.07	34.19	34.25	28.99	27.99	28.62	27.88	29.02	29.76	30.09	29.64	30.04	29.55

and

Table 3. The signal-to-noise ratio of passenger disembarking from the train.

Wavelet Function	Haar	db2	db3	db4	db5	db6	Sym2	Sym3	Sym4	Sym5	Sym6	Coif1	Coif2	Coif3	Coif4	Coif5
SNR	27.63	35.82	33.57	34.59	36.28	35.35	30.05	28.26	26.49	29.18	28.13	28.39	30.21	28.27	31.35	29.68

.

Table 2 shows that the larger the signal-to-noise ratio, the better the denoising effect. When the decomposition level of db wavelet function is 4, the signal-to-noise ratio is at its largest. Therefore, the wavelet function of db4 can be used to denoise the passenger flow data for embarking on the train. Similarly, as shown in Table 3, it can be observed that the wavelet function of db5 is used to denoise the passenger flow data for disembarking from the train.

The condition of using the ARMA algorithm is a stable time series; the stability of passenger flow when entering/leaving the station should be tested before using ARMA. As shown in Figure 9 and Figure 10, the value and characteristics of the autocorrelation function and partial autocorrelation function are calculated and analyzed training both embarking and disembarking states.

Figure 9 and Figure 10 demonstrate that the autocorrelation function decreases from 1.0 to 0.0 and fluctuates nearby. The fluctuation amplitude gradually decreases. The partial autocorrelation function decays from 1.0 to 0.0 and oscillates at 0.0 and nearby, infinitely close to 0.0. Both of them are tailing and belong to stable time series, which satisfies the applicability conditions of the ARMA algorithm.

In this section, the rank of the ARMA model is determined based on AIC criterion. According to experimental calculation, when p = 2, q = 3, AIC obtains the minimum value. Thus, the rank of the ARMA model is determined as ARMA (2, 3). Then, the selection and performance of the RBFNN algorithm’s parameters are shown in Figure 11. Finally, the performance of ARMA-RBFNN is compared with other deep learning models in the datasets.

Specifically, some commonly used time series prediction models (i.e., LSTM variation (Yin, Ning and Tang [55]) and MQ-RNN (Wen et al. [56]) are selected as benchmark models. The benchmark models are detailed as follows. First, the LAG-LSTM parameters are set as learning rate = 0.002, batch size = 80, pervious time steps = 14 and time delay = 5. The dimension of the hidden state is set as 12. The dropout is 0.2 (Yin, Ning and Tang [55]). Next, MQ-RNN is constructed as a simple combination of state and control variables as input data for the standard RNN network. The parameters are set as $h_t\in {ℝ}^{12}$ and N = 14 (Wen et al. [56]).

In order to evaluate the performance of different algorithms, the Mean Absolute Percentage Error (MAPE) is selected as the evaluation index. As the ARMA-RBFNN algorithm predicts the passenger flow, it is necessary to evaluate the performance of this method as follows (see Figure 12 and

Table 4. Comparison prediction error of MAPE.

Model	ARMA-RBFNN	LAG-LSTM	MQ-RNN
<10%	93.78%	73.06%	69.07%
10–15%	6.22%	20.16%	18.65%
15–20%	--	7.25%	5.18%
>20%	--	0.52%	3.11%

).

$$MAPE=\frac{100\%}{n}{\displaystyle \sum _{i=1}^n\left|\frac{y_i-{\widehat{y}}_i}{y_i}\right|}$$

(60)

where $y_i$ is the actual passenger flow data and ${\widehat{y}}_i$ represents the predicting data.

As shown in Figure 12, it can be seen that the MAPE of ARMA-RBFNN, LAG-LSTM and MQ-RNN are 2.23%, 6.38% and 7.88%. Table 4 shows that the prediction error of ARMA-RBFNN was the smallest, superior to other algorithms. Therefore, the ARMA-RBFNN algorithm can be used to improve the prediction accuracy of passenger flow in URT systems.

4.2. Evaluation of Single-Train Control Effect

In this set of experiments, the single-train control effect is tested by GA, which solves the global optimal value of the model based on the secondary processed datasets. The train parameters adopt the type B train of Beijing line 4. Other parameters, including single side slope, curve radius (2000 m) and speed limit (80 km/h), are also selected. The station spacing is shown in

Table 5. Distance between stations.

Origination/Destination Station	Distance [m]
Xinjiekou/Pinganli	1100
Pinganli/Xisi	1100
Xisi/Lingjinghutong	869

.

• Selection Operator

The selection operator is changed without changing other operators. The relationship between the fitness function value and the number of iterations is shown in Figure 13.

Figure 13 shows that the number of iterations required when the fitness function value tends to be stable decreases as the selection probability increases. When the selection probability value is 0.75, the fitness function value tends to be stable in the 26th generation. When the selection probability value is 0.85, the fitness function value tends to be stable in the 22th generation. When the selection probability value is 0.95, the fitness function value tends to be stable in the 8th generation. Thus, when the selection probability is between [0.80, 0.95], the convergence speed of GA is improved.

• Crossover Operator

The crossover operator is changed without changing other operators. The relationship between the fitness function value and the number of iterations is shown in Figure 14.

Figure 14 shows that the number of iterations of the algorithm has little effect on the stability of the fitness function value based on changing the crossover operator. However, the function value has the effect of tending to the optimal value. The values of the crossover operator are between 0.7 and 0.8, and the fitness function value is close to the optimal value. When the values of the crossover operator are between 0.8 and 0.9, the fitness function value is far from optimal, with a large error. Therefore, a crossover operator value between [0.75, 0.85] can improve the evolution effect of GA.

• Mutation Operator

The mutation operator is changed without changing other operators. The relationship between the fitness function value and the number of iterations is shown in Figure 15.

Figure 15 demonstrates that the variance probability is taken as 0.003; the value of the fitness function tends to be smooth in the 6th generation. When the variance probability is taken as 0.004, the value of the fitness function tends to be smooth in the 11th generation. When the variance probability is taken as 0.005, the value of the fitness function tends to be smooth in the 16th generation, and at the value of 0.004, the value of the fitness function is closer to the optimal value. Therefore, the algorithm evolves better when the variation probability is taken between [0.003, 0.005].

In summary, the selection operator, crossover operator and mutation operator are determined. The GA is run, and the optimal fitness value is related to the number of iterations, as shown in Figure 16.

Figure 16 shows that when the number of iterations is about 53, the fitness curve tends to be stable. This means that the optimal value of the fitness function has been reached, and the curve is convergence.

The speed and operating condition conversion points among the four stations are shown in Figure 17.

Figure 17 indicates that the operating conditions of the train between Xinjiekou-Lingjinghutong stations consist of four types: traction, cruise, coasting and braking, which verifies the effectiveness of the method proposed in this paper. Meanwhile, considering the operation efficiency and control effect, the optimal recommended speed of trains between stations obtained by the proposed algorithm can effectively satisfy the constraints of the speed limit and the safety of train operation, which further verifies the applicability of the proposed algorithm.

The solution results of GA were entered into the Anylogic simulation software for verification and analysis. The operation process is shown in Figure 18 and Figure 19. Figure 18 shows the interface where passengers were generated and entered the platform. Figure 19 shows the interface where passengers started to embark on/disembark from the train after tit stopped at the station.

As shown in

Table 6. Comparison of optimization results.

Index	Xinjiekou/Pinganli Station	Pinganli/Xisi Station	Xisi/Lingjinghutong Station
Actual average passenger waiting time [s]	105.00	102.00	93.00
Calculated average passenger waiting time [s]	103.00	101.80	96.00
Actual energy consumption [kW·h]	14.67	14.28	13.64
Calculated energy consumption [kW·h]	13.85	13.94	12.88
Energy efficiency [%]	5.58	2.38	5.57

, traction energy consumption and average passenger waiting time during train operation are calculated.

Table 6 shows that when the train ran at Xinjiekou/Pinganli station, the energy-saving efficiency was 5.58%. The average passenger waiting time at Pinganli station was reduced by 2 s. The energy-saving efficiency was 2.38%. The average passenger waiting time at Xisi station was reduced by 0.2 s during the operation between Pinganli and Xisi station. The energy-saving efficiency was 5.57%. The average passenger waiting time at Lingjinghutong station was extended by 3 s. For a single train, the overall energy-saving efficiency was 4.51%. The average passenger waiting time was extended by 0.8 s, thus realizing energy saving and achieving the expected effect.

4.3. Evaluation of Multi-Train Control Effect

To test and verify the reliability and performance of the DRLDA algorithm, different cases for simulations were made. The training process of DDPG-RS and DQN-RS (Shang, Zhou and Fujita [57]) was compared with the proposed method in an ordinary scenario. According to the relationship complexities among states, actions and evaluation indexes, Neural Network structures of different depths can be employed. Since the states and actions are multi-dimensional for the problem of train energy-saving control, the Q networks, actor networks and critic networks are composed of several fully connected hidden layers, with some neurons for the fully connected hidden layers, respectively. The output layer of the actor networks and critic networks have only one neuron in continuous action space (Kou et al. [58]; Wang et al. [59]; Zhao et al. [60]). The related parameters for training are shown in

Table 7. Training parameters.

Parameter	DRLDA	DDPG-RS	DQN-RS
State transition step	100	100	100
The batch size N of transfer samples from replay memory	32	32	32
Single training step	20	20	20
Actor network learning radio	0.0001	0.001	--
Critic network learning radio	0.0001	0.001	0.001
Soft update parameter	0.0001	0.001	0.001
Sample data	125	125	125
Replay memory capacity	12,000	10,000	10,000
Discount factor	0.99	0.8	0.9
Line length	3069	3069	3069
Other parameters	--	See Ref. [57]	See Ref. [57]

. If the termination condition described in the DRLDA strategy were triggered during the training, the current single training would be stopped. The train would be reset to the initial state to start a new training session. The levels were discretized into eleven different control levels [100%, −80%, −60%, −40%, −20%, 0%, 20%, 40%, 60%, 80%, 100%]. The higher the number of discretions, the more difficult the training. After adding noise in the exploration process, three different algorithms were selected as the closest control levels.

As shown in Figure 20, the average reward value of the variety process curve is shown for the training of three different algorithms.

Figure 20 demonstrates that the average reward values of three different algorithms tend to be stable after rising. The DRLDA algorithm more easily learns the action space, so the training process is more stable. The average episode reward value is stable at about −135, with a small amplitude of vibration. The average reward value per 200 episodes for DRLDA is generally higher than the reward values for DDPG-RS and DQN-RS in Figure 20. This implies that DA-DRL learns better than these two algorithms. With the increase in training times, the average episode reward values of the DDPG-RS and DQN-RS algorithms have been rising in vibration. The values of these two algorithms began to rise slowly after reaching −160 and −200, respectively.

The results, compared with the computer resources demanded by the above three and other alternative approaches, are shown in

Table 8. Comparing the training session results of the proposed method with other algorithms.

	Number of Training	Number of State	Average Rewards Value	Single Training Time [s]
DRL-DA	10,000	751,442	−104	0.0008
DDPG-RS	10,000	735,117	−141	0.02
DQN-RS	10,000	680,985	−181	0.06
DDPG	10,000	640,088	−282	0.83
DQN	10,000	601,128	−369	2.04
DRL-DA	20,000	837,022	−119	0.001
DDPG-RS	20,000	802,441	−150	0.21
DQN-RS	20,000	750,088	−193	0.35
DDPG	20,000	721,672	−294	1.19
DQN	20,000	682,309	−382	2.20
DRL-DA	30,000	915,147	−132	0.032
DDPG-RS	30,000	870,036	−158	0.33
DQN-RS	30,000	834,011	−205	0.68
DDPG	30,000	802,113	−303	1.54
DQN	30,000	779,554	−394	2.31
DRL-DA	40,000	1,012,271	−141	0.074
DDPG-RS	40,000	953,046	−167	0.56
DQN-RS	40,000	903,117	−218	1.04
DDPG	40,000	880,121	−312	1.81
DQN	40,000	840,455	−408	2.57

.

As shown in Table 8, the DRL-DA algorithm has more state transfers than other algorithms, and the average reward value obtained is higher, indicating that the DRL-DA triggers less termination of training and better learning. Due to the algorithm structure of Q learning, Q learning chooses discrete vectors for the action space, which has some limitations in simulation generalization ability and is slightly less effective. However, due to the relative simplicity of the algorithm, the training time is relatively fast. The DQN algorithm is relatively slow in computing efficiency due to the complexity of the algorithm compared to the reinforcement learning algorithm. At the same time, because the artificial neural network training has a certain degree of randomness, the training results are prone to fluctuations, which can be improved by adjusting the reward function weights and improving the neural network. Compared with other algorithms, the proposed algorithm requires less data. Through the direct interaction between the environment and the agent, it continuously learns and corrects each step of the action without the need for complex operation conditions settings. Therefore, compared with other algorithms, the proposed algorithm is more convenient and shows better performance.

Figure 21 shows that in the process of solving with the DRLDA algorithm, there is a convergence curve between the optimal value and the number of iterations. The operating time of multiple trains at Xinjiekou, Pinganli, Xisi and Lingjinghutong stations can be obtained by inputting different train arrival intervals in Figure 22.

Figure 21 demonstrates that the proposed algorithm converges to a relatively satisfactory solution after more than 100 generations of computation. The time to obtain the optimal solution is about 371 s. As seen in Figure 22, the multi-train operating time optimization diagram, drawn according to the optimization results obtained by the algorithm, can better reflect the variation pattern of passenger flow with time, and the train operating interval changes dynamically with the passenger flow demand. During the morning peak hours of 7:00–9:00 a.m., the operating lines are more closely distributed, which can maximize passenger waiting time, rationalizing train departures and reducing operation costs.

For verification, the solution results of the DRLDA algorithm were entered into Anylogic. In the multi-train train control model, the relationship between train traction energy consumption and passenger waiting time is mainly studied by changing the arrival interval of the train. The TrainSource module in Anylogic sets the train arrival interval, which can be adjusted in minutes or seconds. According to the solution results, the train arrival interval is adjusted by running the simulation.

Figure 23 shows that passengers arrive at the platform and disperse to each door of the train to wait for it. Figure 24 indicates that passengers have formed a queue in front of the doors and as the train is about to enter the station, and passengers are waiting to board.

As shown in

Table 9. Comparison of operating times.

	The Planned Operating Time				Model Optimization Time
Train Number	Xinjiekou	Pinganli	Xisi	Lingjinghutong	Xinjiekou	Pinganli	Xisi	Lingjinghutong
1	7:02	7:04	7:07	7:09	7:02:02	7:04:40	7:07:03	7:09:39
2	7:05	7:07	7:10	7:12	7:05:42	7:07:42	7:10:32	7:12:39
3	7:08	7:10	7:13	7:15	7:08:12	7:10:12	7:13:41	7:15:39
4	7:12	7:14	7:17	7:19	7:12:42	7:14:42	7:17:11	7:19:09
5	7:16	7:18	7:21	7:23	7:16:23	7:18:40	7:21:03	7:23:39
6	7:20	7:22	7:25	7:27	7:20:02	7:22:15	7:25:03	7:27:39

, the planned operating times of the trains are compared with the optimization results of the proposed model.

Considering the punctuality of the train in operation, the train operating time is optimized within the range of (0.60 s). It can be seen from Table 9 that the train’s arriving time at each station also changes with different train arrival intervals and is affected by passenger waiting time. On this basis, the multi-train energy consumption at the four stations and the average passenger waiting time were calculated and compared, and are shown in

Table 10. Comparison results.

Train Number	Actual Energy Consumption [kW·h]	Actual Average Waiting Time [s]	Energy Consumption by DRLDA [kW·h]	Average Waiting Time by DRLDA [s]	Time Reduction [s]	Energy Efficiency [%]
1	81.50	235.00	76.40	232.00	3.00	6.30
2	83.60	231.00	78.20	230.00	1.00	6.50
3	82.70	238.00	84.50	243.00	−5.00	−2.20
4	89.60	307.00	85.30	301.00	6.00	4.80
5	88.50	311.00	90.10	315.00	−4.00	−1.80
6	92.50	304.00	87.70	300.00	4.00	5.20
Total	518.40	1626.00	502.20	1621.00	5.00	3.10

.

As a result, the number of metro passengers waiting at the station is determined by the arrival rate of metro passengers at the station and the following headway. The train arrival time at each station also changes with different train arrival intervals and is affected by passenger waiting time. Table 10 shows that with the arrival of the morning rush, the trains gradually entered the normal operation stage during the day, the passenger flow increased and the scheduled departure interval gradually increased. When the train arrival interval was changed, the average passenger waiting time increased. More passengers entered the train, resulting in an increase in the frequency of train operation and a gradual increase in the number of train traction runs, which increased the train traction energy consumption. However, from the perspective of multi-train collaborative operation, the total traction energy-saving efficiency of these six trains was 3.1%. The passenger waiting time was reduced by 5 s, which achieves the energy-saving effect and the expected goal. Furthermore, with multi-train collaborative operation, there is a positive effect, i.e., the trains numbered 1, 2, 4 and 6. Meanwhile, there is also a negative effect, i.e., the trains numbered 3 and 5. The possible reason is that the passenger flow is assumed to be dynamic, which means that there is a spatiotemporal aspect to the passenger flow. Therefore, when the following headway, running time and dwell time change slightly, the impact on the train traction energy consumption value is rather large.

5. Conclusions

The rapid development of urban rail transit has brought about huge energy consumption problems due to its large capacity and long transportation hours. With the development concept of “low carbon” and “sustainable development”, how to reduce train traction energy consumption has become an important element in the sustainable development of rail transit. The focus of this paper is to reduce the train traction energy consumption while reducing the waiting time of passengers on the platform to satisfy their travel demands. Thus, a DRLDA algorithm was proposed for multi-train energy-saving collaborative optimized control to deal with complicated multidimensional constraints. Firstly, the change-point algorithm was used to find the changing points in the time series of passenger flow curve, which constitute the variation point sets. The wavelet variation method was used for denoising to ensure the authenticity of the data. Then, the train operation between stations was studied to determine the combination of train operating conditions, and a single train’s energy consumption was calculated. Considering the multidimensional conditions, the single-train energy-saving control model was established, the goal of which was to minimize the multi-train traction energy consumption and the average passenger waiting time. An ARMA-RBFNN algorithm was proposed and applied to implement the single-train energy-saving control strategy. Next, the queuing theory was introduced to study the relationship between passenger waiting time and train arrival intervals and stopping times. The multi-train cooperative energy-saving control model was established based on many factors and conditions. The global optimal value of the multi-train cooperative energy-saving control model was solved by the Dragonfly Algorithm. The DRLDA algorithm was designed to handle multi-train cooperative energy-saving control, and a proactive constraint handling system was added to DRL to adjust learning directions and avoid learning oscillations. The simulation results demonstrated that the MAPE of ARMA-RBFNN was 2.23%; this value was the smallest, which is superior to other algorithms. Moreover, for a single train, the overall energy-saving efficiency was 4.51% and the average passenger waiting time was extended by 0.8 s. For multiple trains, the DRLDA showed advantages over DDPG-RS and DQN-RS in accelerating the learning convergence process and obtaining smooth results, satisfying multidimensional constraints. At the same time, the total traction energy consumption and passenger waiting time were reduced by 3.1% and 5 s, respectively, achieving the energy-saving effect.

In the future, our research will mainly focus on the following major aspects: (1) When calculating the passenger waiting time according to the actual situation, this time changes, and can be set to an interval range to restore the authenticity of the data in the future. (2) The multi-train collaborative control model, in the case of interchange, should be studied in depth. (3) We hope to obtain more concrete operational data to study transfer behavior in a larger-scale rail transit network.