A Novel Approach for Train Tracking in Virtual Coupling Based on Soft Actor-Critic

Abstract

The development of virtual coupling technology provides solutions to the challenges faced by urban rail transit systems. Train tracking control is a crucial component in the operation of virtual coupling, which plays a pivotal role in ensuring the safe and efficient movement of trains within the train and along the rail network. In order to ensure the high efficiency and safety of train tracking control in virtual coupling, this paper proposes an optimization algorithm based on Soft Actor-Critic for train tracking control in virtual coupling. Firstly, we construct the train tracking model under the reinforcement learning architecture using the operation states of the train, Proportional Integral Derivative (PID) controller output, and train tracking spacing and speed difference as elements of reinforcement learning. The train tracking control reward function is designed. Then, the Soft Actor-Critic (SAC) algorithm is used to train the virtual coupling train tracking reinforcement learning model. Finally, we took the Deep Deterministic Policy Gradient as the comparison algorithm to verify the superiority of the algorithm proposed in this paper.

1. Introduction

Urban transit systems are a cornerstone of modern urban planning and development. They make cities more livable, equitable, and economically vibrant places by improving the environment and overall urban sustainability. As the size of cities increases, subway lines become more complex, and train operation control tasks are arduous. This will lead to a waste of resources. Virtual coupling in urban subway systems provides solutions to a wide range of challenges, including optimizing train formation, reducing energy consumption, enhancing passenger experience, adapting to variable demand, and ensuring safety. It offers a more responsive and adaptable approach to subway operations, ultimately benefiting both passengers and subway operators. Train tracking control is a crucial component in the operation of virtual coupling trains. It plays a pivotal role in ensuring the safe and efficient movement of trains within the train and along the rail network, and ensures that trains are accurately positioned, aligned, and coordinated during coupling and throughout their journey.

Train tracking control is a crucial component in the operation of virtual coupling trains. It plays a pivotal role in ensuring the safe and efficient movement of trains within the train and along the rail network and ensures that trains are accurately positioned, aligned, and coordinated during coupling and throughout their journey. In virtual coupling, multiple trains form a train formation in which back trains track front trains and there should be a method used to control back trains to front ones. At present, many traditional methods, such as Model Predictive Control (MPC) and Proportional Integral Derivative (PID) control, are used in the train tracking problems in virtual coupling. However, the control performance of these methods relies on the manual setting of feedback parameters and has the disadvantage of too frequent acceleration and deceleration switching, which cannot guarantee the comfort of passengers. If we can find a method so that the parameters of the control method can be adjusted according to the control effect, the existing train tracking control method can have a better control performance.

In this study, we proposed a novel approach for train tracking in virtual coupling based on Soft Actor-Critic (SAC), which belongs to the reinforcement learning (RL) area. The SAC tends to be sample-efficient, meaning it can learn high-quality policies with relatively fewer samples. And, it is based on the maximum entropy reinforcement learning framework. It employs a maximum entropy policy to balance exploration and exploitation, enhancing the algorithm’s robustness and performance. The SAC is used to optimize the parameters of PID by taking the output of PID and the train motion status as the states and the parameters of PID as actions. Then, PID controls the operation of the back train. It is the first time that the SAC has been used in the Virtual Coupling of urban railway transportation. We set up two trains in the Virtual Coupling in this study.

This study is organized as follows: Section 2 presents a literature review about train tracking control problems for railway train operation, especially in Virtual Coupling. Section 3 introduces the train tracking control algorithm in virtual coupling based on the SAC. Section 4 carries out the simulation of the train tracking methods based on SAC proposed in this paper and discusses the results based on real data. Section 5 summarizes the work in this paper and introduces the research work to be performed in the future.

2. Literature Review

In 1999, virtual coupling was first proposed by Bock et al. [1] as well as the new train operation mode, and in 2000, Bock et al. [2] introduced the methods to design and develop the virtual coupling system, which uses train-to-train communication to replace the physical coupler links between train carriages, so that adjacent trains can operate cooperatively and shorten the train tracking interval. Other technical details such as system framework, communication, positioning, etc., are explained in detail in references [3], [4], and [5], respectively.

With the continuous development of train-to-train communication, train positioning, and other technologies, more and more scholars carry out research on the virtual coupling operation mode and train tracking control. Cao et al. [6] proposed a train operation control method based on local leader–follower to establish the virtual coupling implementation scheme, and then a controller used for train tracking was proposed on the basis of parameter identification of the nonlinear virtual coupled train operation process [7]. Lin et al. [8] studied the velocity and constrained tracking control problem to propose a distributed cooperative tracking control algorithm for virtual coupling trains. For high-speed railway virtual coupling, Liu et al. [9] designed a method based on Pontryagin’s maximum principle to solve the optimal control problem considering constraints of safe spacing, operation limits, and train dynamic performance. Luo et al. [10] presented a distributed adaptive model predictive control system to coordinate the driving of each unit train in Virtual coupling. Wang et al. [11] illustrated the virtual coupling by a cooperative game model and solved it based on the particle swarm optimization algorithm. Chen et al. [12] proposed an iterative learning control scheme aiming to track the desired reference displacement and velocity in which more general nonlinear uncertainties are imposed on the dynamic model of the train, and that would be closer to reality. Felez et al. [13] proposed a robust MPC method by defining a robust controller into the MPC to control train tracking. A similar work is proposed by Su [14].

The reinforcement learning methods provide a new way for optimized control in many other areas. Wu et al. [15] first investigated the problems of optimal false data injection attack scheduling and countermeasure design for car-like robots in the framework of deep reinforcement learning. In the area of data-based secure control, the Soft Actor-Critic is used to solve the secure control problem [16]. As for the urban railway, He et al. [17] combined the convolutional neural network with long short-term memory to construct the CNN-LSTM hybrid model from the perspective of trajectory prediction, which can obtain the features both in space and time from measurement data at the same time. They also combined LSTM with the Kalman filter to obtain the long-term dependencies [18]. Huang et al. [19] proposed a cooperative tracking control based on a consensus algorithm and artificial potential field theory to realize the train tracking within a distance range. For the displacement-speed trajectory tracking of the automatic train control system with unknown parametric/nonparametric uncertainties and speed constraints, Li et al. [20] proposed a constrained spatial adaptive iterative learning controller. Zhou et al. [21] proposed an improved disturbance observer-based control method to ensure that the automatic train control (ATC) system can operate safely and control accurately even when the train is affected by uncertainties. Wang et al. [22] introduced deep learning into train tracking to calculate the reference speed trajectory according to the real-time driving condition.

3. Methodology

In this section, we introduce the basic principle of the SAC method used in this article. Then, the train tracking dynamic model is built, which is used as the environment for the reinforcement learning architecture. At last, the train tracking control algorithm based on SAC is illustrated.

3.1. Soft Actor-Critic

SAC [23] is a RL algorithm developed based on the idea of maximum entropy. It uses a randomly distributed policy function (Stochastic Policy) and is an off-policy, Actor-Critic algorithm. It is most similar to other RL algorithms. The difference is that while SAC optimizes the strategy to obtain higher cumulative returns, it also maximizes the entropy of the strategy. SAC has excellent performance in various commonly used benchmarks and real robot control tasks, and its performance is stable and has strong anti-interference ability.

The purpose of standard reinforcement learning is to find the policy that maximizes the expected sum of rewards:

$${\pi }^{*}=\underset{\pi }{arg\; max}\sum _t{\mathbb{E}}_{(s_t,a_t)\sim {\rho }_{\pi }}\left[r(s_t,a_t)\right]$$

(1)

where $\pi$ is the policy in the SAC, $s_t$ is the state at time t, $a_t$ is the action at time t, $r(s_t,a_t)$ is the reward gained by action $a_t$ under state $s_t$, and ${\rho }_{pi}$ is the distribution of agent’s state $s_t$ and action $a_t$ under policy $\pi$.

After introducing the maximum entropy, the target policy of the reinforcement learning can be written as Equation (2):

$${\pi }^{*}=\underset{\pi }{arg\; max}\sum _t{\mathbb{E}}_{(s_t,a_t)\sim {\rho }_{\pi }}[r(s_t,a_t)+\alpha H\left(\pi (·|s_t)\right)]$$

(2)

where $\alpha$ is a hyperparameter called the temperature parameter, which determines the relative importance of the entropy term against the reward, and thus controls the stochasticity of the optimal policy; $H\left(\pi (·|s_t)\right)$ is the entropy of $\pi (·|s_t)$ calculated by Equation (3).

$$H\left(\pi (·|s_t)\right)=E[-log\pi (·|s_t)]$$

(3)

The action value function of SAC is defined by Equation (4).

$$Q(s_t,a_t)=E_{s_{t+1}\sim \mathcal{D}}[r(s_t,a_t)+\gamma V^{\pi }\left(s_{t+1}\right)]$$

(4)

where $\mathcal{D}$ is the distribution of previously sampled states and actions or a replay buffer.

The state value function is defined by Equation (5).

$$V\left(s_t\right)=E_{a_t\sim \pi }[Q(s_t,a_t)-\alpha log\pi (·|s_t)]=E_{a_t\sim \pi }[Q(s_t,a_t)+H\left(\pi (·|s_t)\right)]$$

(5)

The loss function of the two q-critic networks can be calculated as Equation (6).

$$\begin{array}{cc}\hfill & J_Q\left(\theta \right)={\mathbb{E}}_{(s_t,a_t,s_{t+1})\sim D,a_{t+1}\sim {\pi }_{\varphi }}\hfill \\ \hfill & \left[\frac{1}{2}{(Q_{\theta }(s_t,a_t)-(r(s_t,a_t)+\gamma (Q_{\theta }(s_{t+1},a_{t+1})-\alpha log\left({\pi }_{\varphi }\left(a_{t+1}\right|s_{t+1})\right))))}^2\right]\hfill \end{array}$$

(6)

The loss function of the actor-network can be calculated as Equation (7).

$$J_{\pi }\left(\varphi \right)={\mathbb{E}}_{s_t\sim \mathcal{D},\epsilon \sim \mathcal{N}}\left[\alpha log{\pi }_{\varphi }\left(f_{\varphi }\left({\epsilon }_t;s_t\right)\mid s_t\right)-Q_{\theta }\left(s_t,f_{\varphi }\left({\epsilon }_t;s_t\right)\right)\right]$$

(7)

where $\mathcal{N}$ is the standard normal distribution and f is the probability density function of policy $\pi$.

And lastly, the loss function of the temperature parameter $\alpha$ is shown as Equation (8).

$$J\left(\alpha \right)={\mathbb{E}}_{a_t\sim {\pi }_t}\left[-\alpha log{\pi }_t\left(a_t\mid {\pi }_t\right)-\alpha {\mathcal{H}}_0\right]$$

(8)

The process of the SAC algorithm is shown as Algorithm A1 in Appendix A.

3.2. Train Tracking Model Based on Reinforcement Learning

For train tracking in the virtual coupling, it is necessary to ensure that the distance between the two trains is not too large, otherwise the transportation efficiency of the virtual coupling will become low. It is also necessary to ensure that there is a sufficient safe distance between the two trains to ensure the safe operation of the trains in the virtual coupling.

Figure 1 shows the scenario of train tracking in the virtual coupling. In the virtual coupling, the back train (Train 2 shown in Figure 1) changes its position $p_{t,2}$ and speed $v_{t,2}$ at time t by controlling the output of its traction system based on the position $p_{t,1}$ and speed $v_{t,1}$ of the front train (Train 1 shown in Figure 1), which makes the distance $\Delta p_t$ between Train 2 and Train 1 within a reasonable range. The positions and speeds of Train 1 and Train 2 have the following relationship.

$$\Delta p=p_1-p_2-L$$

(9)

$$\Delta v=v_2-v_1$$

(10)

where L is the length of Train 1.

Since the position and speed of Train 1 is the reference target of Train 2, we take Train 2 as the control object and here give its dynamic model as described in our previous paper [24].

PID control is widely used in urban rail transit train operation control due to its simple, real-time, robust, and adjustable characteristics. At the beginning of the operation, there is an initial distance $\Delta p_0$ between train 2 and train 1. If the speed of train 2 is controlled to be as close as possible to the speed of train 1, $\Delta p$ will remain close to $\Delta p_0$. Therefore, the speed of train 2 can be the control target of PID control. Set $\Delta v_t$ as the difference between $v_1$ and $v_2$ at time t ($v_i=v_{t,i}$, for $i\in \{1,2\}$).

$$\Delta v_t=v_{t,2}-v_{t,1}$$

(11)

The PID control variable $\Delta u$ at time t can be calculated by Equation (12) [25].

$$\Delta u=K_P(\Delta v_t-\Delta v_{t-1})+K_I\Delta v_t+K_D(\Delta v_t-2\Delta v_{t-1}+\Delta v_{t-2})$$

(12)

where $K_P$, $K_I$, and $K_D$ denote the coefficients for the proportional, integral, and derivative terms, respectively.

Although the PID algorithm is widely used, its parameters need to be finely adjusted. Reinforcement learning learns how to make optimal decisions in different situations through the continuous interaction between the agent and the environment. This article uses the reinforcement learning method to learn the three coefficients of the PID algorithm.

The elements of the RL architecture are described as follows.

Agent: we take the PID controller as the agent of the RL structure, because it can change the operation states of the train by changing the traction or breaking forces of the train.

Action space: Since the RL method is used to learn the three coefficients of the PID algorithm, the action $a_t$ of the RL should be a three-tuple representing the three coefficients.

$$a=\{a_P,a_I,a_D\}$$

(13)

It is better to limit the value of action to $\left[-1,1\right]$ as shown in Equation (14), and we transfer $a_P$, $a_I$, $a_D$ to $K_P$, $K_I$, $K_D$ by Equation (15).

$$a_j\in \left[-1,1\right],forj\in \left\{P,I,D\right\}$$

(14)

$$K_j=K_{j,min}+\frac{(a_j+1)(K_{j,max}-K_{j,min})}{2},forj\in \left\{P,I,D\right\}$$

(15)

where $K_{j,max}$ is the maximum of $K_j$ and $K_{j,min}$ is the minimum of $K_j$ for $j\in \left\{P,I,D\right\}$.

State space: We take the output u of the PID controller as one of the state elements. The speed $v_t$, acceleration $ac{c}_t$, and position $p_t$ of the back train at time t are also elements of the state, since they represent the operation characteristics of the train. The differences of the speed $\Delta v_t$ and the difference of the position $\Delta p_t$ between the trains in virtual coupling at time t are also in the state of the RL structure. With the limitations on actual train operation conditions shown in Equation (17), the state at time t can be described as Equation (16).

$$s_t=\left(u_t,ac{c}_t,v_t,p_t,\Delta v_t,\Delta p_t\right)$$

(16)

$$\left\{\begin{array}{c}{u}_t\in [0,1]\\ ac{c}_t\in [ac{c}_{min},ac{c}_{max}]\\ v_t\in [0,v_{max}\left(t\right)]\\ p_t\in [0,P_L]\\ \Delta v_t\in [\Delta v_{min},\Delta v_{max}]\\ \Delta p\in (0,P_L]\end{array}\right.$$

(17)

where the range of the acceleration of the train is $[ac{c}_{min},ac{c}_{max}]$, and $P_L$ is the length of the whole railway.

Environment: The agent outputs an action and then the environment returns a new state as well as a reward. In this work, the environment should construct the relevance of the output of the PID controller, the state of train’s operation and the distance between the two trains in virtual coupling. So, we take the train dynamics model and the train tracking model as the environment of the RL architecture.

Reward: The purpose of train tracking is to keep the distance and the difference of speed between the two trains in virtual coupling in a small range, which means that the distance and the difference of speed should be smaller. Then, we can define the reward $r\left(t\right)$ at time t as Equation (18). As $\Delta v_t$ and $\Delta p_t$ decrease, $r\left(t\right)$ will increase. Therefore, reinforcement learning will learn in the direction where both $\Delta v_t$ and $\Delta p_t$ decrease.

$$r\left(t\right)=\frac{1}{1+{\eta }_1\left|\Delta v_t\right|}+\frac{1}{1+{\eta }_2\left|\Delta p_t\right|}$$

(18)

where ${\eta }_1$ and ${\eta }_2$ are the adjusting coefficients that prevent the reward value from being too large.

Note that $\Delta p_t$ should always be a positive number since the trains can not collide. The penalty item of the reward is shown as Equation (19).

$$r\left(t\right)=-Cif\Delta p_t<0$$

(19)

where C is a large positive number.

3.3. The Train Tracking Control Method Based on SAC

Figure 2 illustrates the architecture of the train tracking control method based on SAC. The SAC generates action $a_t$ and transfers it to $(K_P,K_I,K_D)$. Then, the PID controller generates u by receiving $(K_P,K_I,K_D)$ and derives the train movement under its dynamics. After every step is executed, the operation data are stored in the Replay Memory for the training of SAC. Also, the Replay Memory will be gradually updated by the operation data.

4. Case Study

To verify the superiority of this work, we took the DDPG algorithm as the comparison algorithm applied to the virtual coupling train tracking control along with the SAC, which is based on the real subway in Beijing. The parameters of the train and the information of the track are similar to our previous paper [24]. Other parameters used in this paper are shown in

Table 1. Parameters of the PID control and virtual coupling.

Parameter Names	Parameters
L /m	92
$\Delta p_0$/m	5.92
$\left\{K_{P,min},K_{P,max}\right\}$	$\left\{0,5\right\}$
$\left\{K_{I,min},K_{I,max}\right\}$	$\left\{0,1.5\right\}$
$\left\{K_{D,min},K_{D,max}\right\}$	$\left\{0,1.5\right\}$
${\eta }_1$	1000
${\eta }_2$	1000
C	100

.

For the two reinforcement learning methods, we conducted 10,000 episodes of training, respectively. The changing trend of the rewards, the loss, and the train tracking states of virtual coupling are used to illustrate the simulation.

Figure 3 and Figure 4 illustrate the changing trend of rewards in two algorithms. Figure 3 shows that the reward of SAC changes a lot before the 4000th episode, and after that, the fluctuation of the original curve decreases and the smoothed curve becomes flat. However, the reward of DDPG shows a downward trend as the training episodes increase, and there is no sign of convergence as shown in Figure 4. The SAC has better performance in this paper than DDPG from the aspect of changing trend of the rewards.

Figure 5 shows the changing trend of training losses of SAC and Figure 6 shows that of DDPG. The loss of SAC increases before the 2000th episode, and then decreases until it is stable after the 5000th episode, as shown in Figure 5. It shows that the SAC algorithm keeps learning before the 5000th episode, and then it reaches a stable result for the whole RL progress. As shown in Figure 6, the loss of DDPG does not have a gradual trend. Instead, it suddenly changes dramatically after 8000 episodes of training. This shows that the learning efficiency of DDPG is very low and its training is unstable.

We can conclude that the DDPG algorithm does not meet the demand of the train tracking control, as its reward does not converge and it has lower learning efficiency than SAC, the SAC performs better than DDPG, and the reward can reach a stable level through the above analysis. For the virtual coupling, the distance and the difference of speed between two trains in virtual coupling are crucial indicators, which are analyzed as follows.

Figure 7 shows the operation profiles of the front train (Train 1) and the back train (Train 2). The trajectories of these two trains are almost similar to each other. The difference in positions and speeds are shown in Figure 8 and Figure 9, respectively. The initial distance between two trains is $5.92$ m and from Figure 8 we can see that during the whole operation process, the distance between two trains is between $5.88$ m and $5.98$ m, and the maximum of $\Delta p$ is $0.6$ m. The two trains in the Virtual Copuling did not collide, which ensures the operation safety in train tracking. The distance between the two trains is not much different from the initial distance. As for the difference of speed, known as $\Delta v$, the maximum of $|\Delta v|$ is within $0.15$ m/s, as shown in Figure 9, which means that the train tracking efficiency is improved by this algorithm.

From the above analysis, it can be seen that under the control method combining SAC and PID, the operation of the two trains in the virtual coupling always maintains a highly consistent state, and the trajectories of these two trains are almost similar to each other. The difference in positions and speeds stays within a small range.

5. Conclusions

The development of virtual coupling technology provides solutions to the challenges faced by urban rail transit systems. Train tracking control is a crucial component in the operation of virtual coupling which plays a pivotal role in ensuring the safe and efficient movement of trains within the train and along the rail network. In order to ensure the high efficiency and safety of train tracking control in virtual coupling, this paper proposes an optimization algorithm based on Soft Actor-Critic for train tracking control in virtual coupling. First, the train tracking model in virtual coupling is constructed including the relationship between the position and speed among trains in virtual coupling, as well as the PID controller. Then, the reinforcement learning model based on SAC for train tracking control is constructed. The train tracking control reward function is designed through the distance and speed difference of the trains. The SAC algorithm is used to train the train tracking reinforcement learning model. The experimental results show that the proposed train tracking control algorithm based on SAC can improve the train tracking efficiency and ensure the safety of virtual coupling.

The proposed method in this article can provide a new way for train tracking control technology in virtual coupling. Combining with traditional control methods can help adjust parameters in traditional methods, making train tracking control more efficient and safer. However, this article lacks research on the robustness of the train tracking process, which can be studied in the author’s future research.