Deep Reinforcement Learning for Ecological and Distributed Urban Traffic Signal Control with Multi-Agent Equilibrium Decision Making

Abstract

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent’s decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network.

1. Introduction

With the rapid development of urbanization, traffic congestion, traffic accidents, traffic pollution and energy shortages have become important factors restricting sustainable development. Intersection is not only the place where traffic congestion and traffic accidents occur, but also the main place where traffic pollution emissions increase and fuel consumption intensifies [1,2]. Therefore, reasonable traffic signal control methods at intersections can improve the efficiency of urban traffic movement and reduce environmental pollution. However, traditional traffic signal control methods rely heavily on a given traffic flow model, which cannot adapt well to the dynamic traffic flow, and the control efficiency decreases with the increasing traffic flow [3].

The development of information technology and communication technology promotes the traffic signal control method to the intelligent direction. As the core part of modern intelligent transportation, traffic signal control has important practical significance. From the perspective of vehicle road collaboration [4], through optimizing the scheduling of signals, traffic signal control is able to promote the cooperation between vehicles and roads to achieve a more efficient and smooth traffic flow distribution, thus improving the efficiency of vehicular traffic and reducing traffic congestion, as well as reducing energy consumption and pollution emissions. From the perspective of an intelligent connected vehicle [5], an intelligent traffic signal system adjusts signal timing in real time according to the information exchanged with vehicles and real-time road conditions, optimizing traffic flow to the greatest extent possible, reducing the probability of traffic accidents and improving traffic safety. In addition, traffic signal control [6] can also be combined with intelligent driving technology to achieve safe cooperative driving between vehicles, providing technical support for the development of intelligent transportation.

Urban traffic signal control is essentially a sequential decision-making problem, and Reinforcement Learning (RL) methods are well suited to deal with such problems. RL methods do not need to build a traffic model but rather learn to improve the traffic control scheme by interacting directly with the traffic environment on a continuous basis. A common approach is Single Agent Reinforcement Learning (SARL) [7], where a single agent controls all intersections. At each time step, the agent observes the traffic conditions at all target intersections and selects an action for all intersections; after taking the action, the environment feeds back a reward to the agent. However, this approach is only applicable to traffic signal control in isolated small road networks. Therefore, more researchers have focused on the study of Multi-Agent Reinforcement Learning (MARL) traffic signal control methods [8], where an independent RL agent controls an intersection signal in a large urban road network, and each agent responds to the neighboring traffic state in real time and makes autonomous decisions, which is more in line with the real-world traffic signal control. However, there are new challenges in MARL traffic signal control methods. (1) The state of each agent is affected by the decisions of its neighborhood agents, and there is a non-smooth problem [9]. To solve this problem, interactive communication between agents is expected. However, full interactive communication between agents reduces the value density of collaborative information, while interacting only with neighboring agents limits the scope of collaboration. This is because traffic signal control is a “multilevel hopping” problem, where agents controlling an intersection have to interact on the basis of sensing traffic information representations including multilevel neighbors. However, most of the current MARL traffic signal control methods still cannot achieve effective agent interaction. (2) There is a collaboration problem among agents [10,11]. The relationship among agents in MARL is difficult to handle. Introducing game theory into RL can handle agent interrelationships well, and game theory provides effective solution concepts to analyze the behaviors of multi-agents and describe the learning results of Multi-Agent Systems (MAS) [12]. In addition, through the solution concept of game theory to guide each iteration of the agent, the result can be well converged to the equilibrium point, the convergence speed is faster, and the benefit of each agent is relatively larger. The equilibrium solution of game theory replaces the optimal solution of MARL, and the strategy obtained is also reasonable and effective. However, existing MARL studies only use game theory to empirically update or balance MAS objectives.

At the same time, vehicle actions such as acceleration, deceleration and stopping increase exhaust emissions, and these actions occur with great frequency at intersections, so signal control at intersections directly affects how much exhaust emissions are emitted by vehicles. However, most MARL traffic signal control methods aim to reduce congestion costs by designing more robust algorithms, ignoring the protection of the ecological environment.

To address the above deficiencies, this paper simplifies the multi-intersection traffic environment into an undirected graph model from the perspective of eco-transportation, in which a node represents an intersection agent. In order to accurately capture the spatio-temporal characteristics of each intersection observation in the traffic network, Gate Recurrent Unit (GRU), a convolutional network, is introduced; and multiple GRUs are stacked to achieve the expansion of the collaborative range of the target intersection agent while reducing the communication load of the agents. Secondly, in order to clarify the relationship between the intersection agents and optimize the decision-making process of the agents, the concept of a mixed-strategy Nash equilibrium solution in game theory is introduced into MARL, and a decision-making process based on the mixed-strategy Nash equilibrium is designed for the agent controlling intersections to optimize and enhance the control capability of the MARL algorithm in the traffic signal control system. Specifically, the contributions of this study are as follows:

• A Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model framework is constructed, which clarifies the relationship between the agents through the game theory to make the collaboration between the agents more reasonable and expands the sensing domain of the intersection agents through the multi-layer stacked GRU network to make the collaboration between the agents more effective, which greatly improves the real-time response speed of the traffic signals and the traffic efficiency of traffic road network.
• The Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is designed, which introduces the concept of a mixed-strategy Nash equilibrium solution in the game theory to optimize the agent’s decision-making, so that the DCNPG-TSC model can choose an optimal signal phase for each intersection to respond to the real-time changing traffic flow. The DCNPG-TSC model can choose an optimal signal phase for each intersection in respond to real-time changes in the traffic flow and then reduce traffic congestion.
• Taking eco-transportation as the starting point and incorporating the concept of green transport, the reward value is set to multiple optimization objectives including reducing pollutant gas emission and fuel consumption, thus reducing the pollutant gas emission and energy consumption at intersections.

The rest of the paper is organized as follows. Section 2 presents the existing research work on RL-based traffic signal control. Section 3 establishes the DCNPG-TSC model for the problem and describes the design of Partially Observable Stochastic Game and RL setup in detail. Section 4 describes the specific implementation details of the DCNPG-TSC model, including the design of the agent network structure, the agent decision process, the agent learning process and the MADNPG algorithm. Experimental results and performance analysis are given in Section 5. Section 6 illustrates the conclusions of this paper as well as future perspectives.

2. Related Works

Traditional traffic signal control methods such as fixed-time control, inductive control [13] and adaptive control [14,15,16] have been widely studied in the past decades. However, with the growth of traffic volume, the complexity of traffic signal control in urban areas is climbing. The traditional traffic signal control methods cannot adapt to the complex and changing traffic flow, and the control effect is unsatisfactory. Thus, scholars have proposed the use of RL methods for traffic signal control.

Early applications of RL to single intersection traffic signal control showed strong advantages. When the road network is extended to multiple intersections, SARL trains a centralized agent to decide the actions at all intersections, which cannot be learned well due to the large dimensionality of the joint action space. Arel et al. [17] proposed a traffic signal control method based on the Q-Learning algorithm and feed-forward neural network, where the outbound agent provides local traffic statistic information to the central agent to collaborate; the central agent learns a value function driven by local and neighboring traffic conditions. However, this method faces scalability problems in deployment and is difficult to be applied in large-scale road networks. To address the above issues, researchers have applied MARL to the traffic signal control problem by training an agent for each intersection and have proposed more complex deep MARL algorithms under the paradigm of centralized training distributed execution. Li et al. [18] proposed the Multi-Agent Deep Deterministic Policy Gradient method to coordinate the traffic signal control, which reduces the waiting time of vehicles by adjusting the phase and duration of traffic lights. Wu et al. [19] proposed a Multi-Agent Recurrent Deep Deterministic Policy Gradient algorithm based on Deep Deterministic Policy Gradient for how to coordinate traffic controllers at multiple intersections. The algorithm adopts the framework of centralized learning decentralized execution; in centralized learning, agents can estimate other agent strategies during the decision-making process, thus realizing mutual coordination among agents and alleviating the problem of a poor learning effect due to unstable environments; in decentralized execution, each agent can make decisions independently, and the participant network with shared parameters also speeds up the training process and reduces the memory occupation.

The MARL algorithm overcomes the scalability problem in large-scale road networks [20], but it also faces the challenges of how to cooperate, coordinate and share information among agents. The impact of one agent’s behavior on the environment is also influenced by the activities of other agents. In addition, individual agents are self-interested, and they want to maximize their returns, and that game relationship leads to locally optimal strategies. Therefore, in traffic signal control applications, optimal decisions need to be made between agents through information sharing and collaboration. However, most of the current research on MARL in traffic signal control focuses on Q-Learning. Abdoos [21] utilizes game theory to coordinate MAS interactive decision making and proposes a two-mode agent structure, in which, in the independent mode, a single agent controls an intersection independently by using Q-Learning; in the cooperative mode, game theory is used to coordinate the cooperation between agents to dynamically control the signal of multiple intersections. Qu [22] designed a distributed control approach to prevent congestion on urban road networks caused by traffic flow demand perturbations by integrating MARL and mixed-strategy Nash equilibrium.

Meanwhile, there are few existing studies on the decision-making process of MARL. MARL makes decisions based on accumulated historical experience, in other words, the ability to make effective decisions is obtained through trial and error. For example, Rasheed et al. [23] applied Deep Q Network (DQN) to traffic signal control to solve the dimensionality problem in traffic congestion and proposed the Mutil-Agent DQN method. Although the concept of game theory is combined in MARL, most of the studies [24,25] use game theory for empirical updating or balancing MAS objectives. Existing studies usually use $\epsilon$-greedy and softmax as strategies for action selection. Decision making in MARL based on mixed-strategy Nash equilibrium, which exhibits a preference for potentially optimal actions in the action selection process and can accelerate MARL convergence. Wu et al. [26] designed two game theory-assisted RL algorithms, Nash Advantage Actor–Critic and Nash Asynchronous Advantage Actor–Critic, by using Nash equilibrium and RL; Wei et al. [27] proposed an algorithm for MARL based on Nash equilibrium and Deep reinforcement learning. The simulation results of the above two methods illustrate that the Nash equilibrium solution of game theory can optimize the agent’s decision to achieve adaptive traffic signal control in complex urban road networks, but the above two methods can only be applied to traffic signal control problems with explicit equilibrium solutions, and the scope of application is limited. Meanwhile, there is a gap in the research of RL-based traffic signal control methods on the green emission problem. Li et al. [28] applied a decision tree method in their study, which successfully reduced CO₂ emission and vehicle delay; however, this method is too simple to be applied to complex road networks.

In summary, existing research on traffic signal control based on MARL and game theory focuses on small-scale communication and collaboration with Q-Learning, ignoring the design of the agent decision-making process and failing to consider the green emission problem. For this reason, this paper constructs a DCNPG-TSC model for distributed traffic signal control, which not only considers the communication problem among agents on a large-scale road network, but also takes the collaboration among agents and the agent decision-making process into account, realizes MARL effective communication and collaboration, and substantially improves the control efficiency of traffic signals and alleviates traffic congestion. For the first time, this paper combines mixed-strategy Nash equilibrium and Multi-Agent Deep Deterministic Policy Gradient and proposes the MADNPG algorithm, which uses mixed-strategy Nash equilibrium to coordinate the relationship between agents and optimize the agent’s decision-making process and greatly improves the traffic efficiency of the traffic network; at the same time, this paper incorporates the exhaust emission and fuel consumption into the MARL signal control system, which largely reduces the vehicle exhaust emissions and fuel consumptions and protects the ecological environment of the city.

3. DCNPG-TSC Model for Multi-Intersection Traffic Signal Control

Existing studies on MARL traffic signal control focus on solving the communication between agents, while ignoring the interaction and collaboration between agents. In addition, there are few studies on the combination of MARL and game theory. Therefore, this paper combines MARL and game theory to propose the DCNPG-TSC model. In this section, the general concept of RL control is first introduced from a simple single-agent RL traffic signal control model. Then, the DCNPG-TSC model is proposed to address the shortcomings of the existing research. Finally, the overall structure of the DCNPG-TSC model and the RL settings in the model are described in detail.

3.1. DCNPG-TSC Model Framework

Urban traffic signal control is a typical MAS, where each intersection is abstracted as a single independent agent. Figure 1a illustrates a real road network, which can be simplified to a graph model as in Figure 1b. In the figure, nodes circled in red represent agents; nodes circled in blue denote the set of neighboring nodes that are directly adjacent to the agent; and the purple circle describes a portion of the agent’s perceptual domain.

Due to the large scale of urban road network and the complex and changing environment, it is costly and unsatisfactory to consider all intersections at the same time. Therefore, this paper defines the multi-intersection traffic signal control problem as a RL task of a multi-agent game. The whole process is modeled as a Partially Observable Stochastic Game, which consists of a multi-agent game, and described as an eight-tuple, $G=\left\{N,S,\left\{{\mathcal{A}}_i\right\},\mathcal{P},\Omega ,\left\{{\mathcal{O}}_i\right\},\left\{{\mathcal{R}}_i\right\},\gamma \right\}$, where N denotes the set of finite intersection agents in the traffic signal control region, S represents the set of observed states of all intersection agents in the region, ${\mathcal{A}}_i$ represents the set of actions that can be taken by intersection agent i, $\mathcal{P}$ represents the state transfer function, $\Omega$ represents the observation function, ${\mathcal{O}}_i$ represents the set of observations of intersection agent i, ${\mathcal{R}}_i$ represents the reward function of intersection agent i, and $\gamma$ denotes the reward discount factor. Figure 1c shows the process of agent i controlling a traffic signal, in any moment t, agent i observes the real-time traffic flow and obtains a local observation $o_i^t$; then, based on the existing observed state $o_i^t$, it takes a corresponding action $a_i^t$ to obtain an instantaneous reward $r_i^t$, and the environment transfers to the next moment’s state $o_i^{t+1}$; agent i observes the impact of this action on the intersection traffic, evaluates the effect of the action just chosen and optimizes the strategy until it converges to the optimal “state and strategy”.

In the framework of a Partially Observable Stochastic Game, the agent is in a multi-agent environment, and the agent’s decisions are not only affected by the environment but also by the decisions of other agents. Meanwhile, since multi-intersection traffic signal control involves interactions among multiple agents, problems involving interactions among multiple agents usually need to deal with the relationships among multiple agents. In addition, game theory is a powerful tool for portraying the interrelationships among multiple agents, which can deal with the interrelationships among agents well. In order to optimize the strategies acquired by agents and improve the learning efficiency, this paper introduces game theory to clarify the influence of other agents on the current agent decision. In addition, taking all agents into account, the MARL traffic signal control problem requires a high cost, and the RL algorithm that considers all agents at the same time does not substantially improve the control efficiency of the traffic signal control system. The handling of the relationship between agents at intersections is the key to achieving collaborative MARL traffic signal control. In order to achieve collaborative control of traffic light signals at multiple intersections, this paper adopts game theory to clarify the relationships among neighborhood agents, and at the same time, in order to improve the access efficiency of the urban road network and to reduce the emission of pollutants at the intersections, this paper proposes a distributed MARL framework based on the game equilibrium–the DCNPG-TSC model framework. Figure 2 shows the overall structure of the DCNPG-TSC model.

In this framework, each signal control intersection is abstracted as an agent, and each agent completes the signal control scheme formulation for one intersection with multiple distributed agents learning in parallel. In each time step, all traffic signal controllers (agents) collect traffic state data from each intersection. Then, agents make action decisions based on the observed traffic state, information exchanged with neighboring agents and the game mechanism and select an optimal action for each agent and deliver it to the corresponding intersection for implementation. Subsequently, the environment feeds back an immediate reward and moves to the next state. Finally, agents evaluate the actions they have just taken and optimize their strategies. Agents thus learn by interacting with the environment to obtain optimal equilibrium strategies in response to real-time changes in the traffic flow on the road network.

In order to achieve global coordinated control, the network design of the DCNPG-TSC model is designed by stacking multiple convolutional layers to expand the sensory domain of the agent, and the training of the agent is based on the observation information and actions of the neighborhood agents to find the optimal equilibrium strategy. The agent network structure is similar to the brain of the DCNPG-TSC model, while the agent learning process makes the DCNPG-TSC model show “intelligence”, and the combination of the two forms an “Agent” that can respond to the traffic environment in real time. These two parts will be elaborated in Section 4. The part framed by the blue line in Figure 2 is the network structure of the agent, and the red line is the agent learning process. MLP is a multi-layer perception machine, GRU stands for Gate Recurrent Unit, and the combination of the two realizes the encoding of traffic observation states; the stacked network of multiple GRUs realizes the extraction of potential traffic features in the gradually increasing perception domain; and Q is the action value function of RL.

3.2. RL Traffic Settings

As the core part of the DCNPG-TSC model is MARL, the definition of the state space, action space and reward function as the three basic elements of MARL is a key part of the DCNPG-TSC model. The definition of the traffic state, action space and reward function are described in detail below.

(1) Intersection observation information $o_i^t$ indicates that at the moment t, the real-time traffic environment information recorded by the intersection traffic cameras and sensors constitutes the local part of the observable state vector of the intersection agent ($i\in \mathcal{N}:=\left\{1,2,\dots ,N\right\}$), which contains $q_i^t$, $d_i^t$. $q_i^t$ indicates the length of vehicles in the queue on all entrance lanes of the intersection; and $d_i^t$ indicates the average delay to vehicles on the entrance lanes of the intersection.

(2) Action $a_i^t$ represents the set of phases taken at an intersection (agent i) at the moment t, which is the set of all traffic signal phases combined by the traffic signal controller. For example, in a synthetic road network consisting of standard “cross-shaped” intersections with 14 lanes, each direction of traffic at the intersection has a separate left-turn lane, and the red and green signals of the traffic signal controllers at each intersection are combined into four phases to form a set of actions corresponding to the intersection, as shown in Figure 3. Among them, the S-N straight and right-turn phases represent the first phase; the S-N left-turn phase represents the second phase; the E-W straight and right-turn phases represent the third phase; and the E-W left-turn phase represents the fourth phase. Since each intersection in a four-phase synthetic road network is identical, each intersection agent in a four-phase synthetic road network has the same set of actions $A=\left\{a_1,a_2,a_3,a_4\right\}$. The number and order of phases varies from intersection to intersection in a realistic road network, and the set of actions of the intersection agent is determined by the actual situation.

Table 1. The correspondence relationships between agent actions, phases, action codes and traffic signal sequences in the synthetic road network.

Phases	I	II	III	IV
Action	$a_1$	$a_2$	$a_3$	$a_4$
Encoder in agent	0	1	2	3
Traffic signal sequence	GGrrrrrGGrrrrr	rrGrrrrrrGrrrr	rrrGGGrrrrGGGr	rrrrrrGrrrrrrG

shows the correspondence between intersection agent actions, phases, action codes and traffic signal sequences in synthetic road networks. For example, phase I corresponds to action $a_1$ that is encoded as “0”, and the traffic signal sequence is “GGrrrrrGGrrrrr”. Wherein “G” represents a green signal and “r” represents a red signal. In order to facilitate understanding, “GGrrrrrGGrrrrr” can be regarded as a traffic signal sequence composed of [3, 4, 3, 4] characters. The first three characters “GGr” indicate the three lanes of the north entrance, which indicates that the right turn and straight lanes are green and the left turn lane is red; “rrrr” indicates the four lanes of the east entrance, where the right turn lane, the two straight ahead lanes and the left turn lane are all red. The next three characters and four characters represent the traffic light status of the lanes at the south and west entrances, respectively.

(3) Reward $r_i^t$ indicates the immediate reward that the environment feeds back to the intersection agent i at the moment t. The urban road network traffic signal control problem has a variety of different control objectives, such as reducing emissions, reducing queue lengths, etc. Different reward functions need to be set for different control objectives. In this paper, the optimization objective is positioned to improve the traffic efficiency of the traffic road network while reducing the emissions as much as possible, so the reward is set to be the sum of the weights of the traffic metrics such as queue length, vehicle waiting time, fuel consumption and tailpipe emissions. In order to investigate the performance of this modeling framework in fuel consumption and tailpipe emissions, this paper adopts the intersection evaluation metrics generator tool that comes with SUMO (Simulation of Urban Mobility, SUMO) to estimate the fuel consumption and the emissions of pollutant gases CO and NOx. Therefore, the reward function is defined as

$$r_i^t=-\left({\displaystyle c_1\times \sum _{j\in I_{-i}}\left({queue}_{l_{ji}}^{t+\Delta t}+{wait}_{l_{ji}}^{t+\Delta t}\right)+c_2\times \left(CO\left[i\right]+NOx\left[i\right]\right)}\right)$$

(1)

where $c_1$, $c_2$ is the weight coefficient, $I_{-i}$ is the set consisting of agent i’s neighboring agents, ${queue}_{l_{ji}}^{t+\Delta t}$ is the queue length measured along the inlet lane $l_{ji}$ at the time t, and ${wait}_{l_{ji}}^{t+\Delta t}$ is the sum of the vehicle waiting time in the inlet lane $l_{ji}$ at the time t. It is worth noting that the reward is lagged, so both ${queue}_{l_{ji}}^{t+\Delta t}$ and ${wait}_{l_{ji}}^{t+\Delta t}$ are measured at time $t+\Delta t$.

4. MADNPG Algorithm for Multi-Intersection Traffic Signal Control

The core parts of the DCNPG-TSC model in Figure 2 are the design of the agent network structure and the design of the agent learning process. In the agent network structure, the traffic state is firstly passed into the Observation Encoder composed of the Fully Connected network and the GRU network for encoding. Furthermore, it is then passed into the GRU network to extract the implicit features, and multiple GRUs are stacked to increase the agent perception domain, so that the collaboration range is no longer limited to the directly adjacent intersections. In the agent’s learning process, in order to find the mixed-strategy Nash equilibrium solution, this paper uses the Boltzmann exploration to predict the strategies of neighbor agents; agent i makes the action decision based on the local observation and the local mixed-strategy Nash equilibrium and proposes the MADNPG algorithm. The MADNPG algorithm is used to train the DCNPG-TSC model to find the optimal equilibrium policy of the multi-intersection traffic signal control system, thus realizing the coordinated and efficient control of multi-intersection traffic signals. Specifically, the MADNPG algorithm trains the agents in the DCNPG-TSC model to find the optimal equilibrium policy, which determines the actions that the agents should take at the moment t. Section 4.1 elaborates the overall design of the network structure and the specific design of the agent network structure. Section 4.2 introduces the decision-making process based on the mixed-strategy Nash equilibrium, which is the solution process of the Nash policy. Section 4.3 demonstrates the agent learning process, and Section 4.4 gives the specific MADNPG algorithm implementation process.

4.1. Neural Network Structure Design

Considering that traffic flow is complex spatio-temporal data, in order to prevent the intersection agent from only knowing the current state and causing the Markov Decision Process to become non-stationary, this paper uses GRUs that can save hidden states and memorize short-term histories to design the network structure of the DCNPG-TSC model. GRU is a Recurrent Neural Network proposed to solve the problems of long-term memory and gradient in backpropagation, which is a variant of Long Short-Term Memory with a simpler structure than Long Short-Term Memory. Compared with Long Short-Term Memory, it is easier to train using GRU, which can improve the training efficiency to a great extent and can achieve considerable results. Figure 4a shows the overall design diagram of the network structure in this paper, in which the queue state, wait state, CO and NOx, as well as the information of neighboring agents, are firstly inputted into the Fully Connected layer for processing. Then, all hidden neurons are combined and inputted into the GRU layer for processing, and at the same time, multiple convolutional layers are stacked to make the extracted regional environmental embedded in the regional environmental features more concentrated. Finally, the output layer performs a softmax operation on the actor and linear operation on the critic.

Figure 4b shows the design details of the agent network structure, where first, the local observation vector $o_i^t$ acquired by the Agent is embedded into the MLP to obtain the intermediate value $c_i^t$, which is then fed into the GRU to obtain the hidden state $h_i^t$. Where $c_i^t=\sigma \left(o_i^t·W_e+b_e\right)$, $h_i^t=GRU\left(c_i^t\right)$. $W_e$ is the weight matrices, $b_e$ is the bias vector, and the $\sigma$ function is a Single Layer Perceptron (SLP) with the Relu’s nonlinear activation function. Next, the second layer GRU extracts the traffic environment features of the merging intersection i and the area where its domain intersection is located to generate the implied feature vector $h_i^{\prime t}$. Continuing to stack GRUs, the perceptual domain of the implied feature vector becomes larger, and the traffic information embedded in the extracted regional environmental features becomes more concentrated. That is, when stacking one GRU, the traffic state observation information acquired by the agent contains the feature vectors of the neighbor agents. When stacking two GRUs, the traffic state observation information acquired by agent i includes the feature vectors of the output of the first GRU and the state information acquired by the second GRU. However, no matter how many GRUs are superimposed, agent i only communicates with its neighbor agents, a feature that makes the method proposed in this paper better adapted to traffic signal control on urban road networks because each agent only plays a mixed-strategy Nash equilibrium game for a limited area rather than a strategy game for the entire urban road network.

4.2. Decision-Making Process Based on Mixed-Strategy Nash Equilibrium

Under distributed conditions, agents should consider not only the environment state but also the strategies of other agents when making decisions. The agent’s strategy at game equilibrium is the optimal strategy to cope with other agents’ strategies. When the MAS is in equilibrium, the distribution of agent’s own experience is the mixed-strategy Nash equilibrium solution to cope with the neighboring agents. In order to obtain the mixed-strategy Nash equilibrium solution of the agent, which is the distribution of the agent’s optimal decisions, this paper adopts Boltzmann’s exploration to predict the strategy of neighboring agents and designs a decision-making process based on the mixed-strategy Nash equilibrium for the agent controlling the intersection. This decision-making mechanism not only accelerates the convergence of MARL but also improves the effect of traffic signal control in urban road networks and reduces the time delay and queue length of vehicles at intersections, as well as reducing the emission of pollutant gases and fuel consumption.

In order to make the MAS achieve the global optimal state when it reaches the final steady state, this paper constructs a general sum game based on the definition of agent’s utility function of the game, the utility function of agent i is

$$\begin{array}{cc}\hfill u_i\left(a_i,a_{-i},s_i\right)=& {\displaystyle \sum _{j\in I_{-i}}{\varpi }_{l_{ji}}^1\left({queue}_{l_{ji}}^{pre}\left(a_i,a_{-i},s_i\right)-{queue}_{l_{ji}}\right)}\hfill \\ \hfill & +{\varpi }_{l_{ji}}^2\left({wait}_{l_{ji}}^{pre}\left(a_i,a_{-i},s_i\right)-{wait}_{l_{ji}}\right)\hfill \end{array}$$

(2)

where $u_i\left(a_i,a_{-i},s_i\right)$ denotes the game payoff function when agent i performs action $a_i$ in the state $s_i$, and its neighbor agents perform the action $a_{-i}$, where agent j is the direct neighbor agent of the agent i, $j\in I_{-i}$; $l_{ji}$ is the road section between the intersection controlled by the agent j and the intersection controlled by the agent i; ${queue}_{l_{ji}}^{pre}\left(a_i,a_{-i},s_i\right)$ is the expected vehicle queue length on the road section $l_{ji}$ when agent i is in the state $s_i$ and its own action $a_i$ is responding to the action $a_{-i}$ adopted by its neighbor agent j; ${queue}_{l_{ji}}$ is the existing vehicle queue length on the road section $l_{ji}$; ${wait}_{l_{ji}}^{pre}\left(a_i,a_{-i},s_i\right)$ and ${wait}_{l_{ji}}$ are similar to ${queue}_{l_{ji}}^{pre}\left(a_i,a_{-i},s_i\right)$ and ${queue}_{l_{ji}}$ in the previous section; ${\varpi }_{l_{ji}}^1$ and ${\varpi }_{l_{ji}}^2$ are both weighting factors indicating the extent to which the queue length and queue waiting time of section $l_{ji}$ are in urgent need of improvement, respectively.

To obtain mixed-strategy Nash equilibrium solution, it is necessary to predict the strategy choice of neighborhood agents. When equilibrium is reached, the global of the MAS also realizes smooth convergence. At this time, the experience distribution of agent is the mixed-strategy Nash equilibrium solution for coping with neighboring agents. Therefore, in this paper, the MARL experience value approximation is used to replace the neighbor agents strategy choice expectation, which is to use Boltzmann exploration to predict the strategies of neighboring agents.

Assuming that agent i is in state $s_i$ and its neighbor agent j is in state $s_j$, $a_j\in A_j\left(s_j\right)$ denotes the actions that can be taken by agent j in state $s_j$, and the basic components of the mixed strategy are denoted by the $Q_j\left(s_j,a_j\right)$ distribution:

$${Pr}_j=\frac{exp\left[Q_j\left(s_j,a_j\right)\right]}{{\displaystyle \sum _{a_j\in A_j\left(s_j\right)}exp\left[Q_j\left(s_j,a_j\right)\right]}}$$

(3)

Let ${\sigma }_j\leftarrow {Pr}_j$ to obtain the mixed strategy ${\sigma }_j$ (${\sigma }_j\in {\sigma }_{-i}$) of neighbor agent j. The mixed-strategy Nash equilibrium solution of the agent i can be obtained using the known expected mixed strategies of neighborhood agents:

$${\sigma }_i^{*}=argma{x}_{{\sigma }_i}{u}_i\left({\sigma }_i,{\sigma }_{-i},s_i\right)$$

(4)

Then, the mixed-strategy Nash equilibrium solution ${\sigma }_i^{*}$ of agent i can be obtained. Subsequently, the execution action $a_i^{MSNE}$ of agent i is obtained by randomly sampling on the probability distribution $\widehat{{Pr}_i}\leftarrow {\sigma }_i^{*}$.

4.3. Agent Learning Process Design

In order to realize the scalability of MARL traffic signal control and to address the non-smoothness of learning among agents, this paper adopts a learning framework with centralized training distributed execution to design the agent learning process, as shown in Figure 5. The agent learning specific process is divided into two parts. In the centralized training phase, the critic of each agent is trained based on the local observations and the observations and policies of neighboring agents. Then, the actor is trained to update with the help of Q-value and mixed-strategy Nash equilibrium solution given by the critic. In the testing phase, decisions are made using the trained actor decision function.

In order to break the correlation between the data and improve the experience utilization, the MADNPG algorithm adopts an experience replay mechanism. During training, the trained data tuples $\left(\mathcal{O},\mathcal{A},\mathcal{R},{\mathcal{O}}^{\prime }\right)$ are first stored in the experience replay buffer D, where $\mathcal{O}=\left\{o_1,o_2,\dots ,o_N\right\}$ is the set consisting of all agents’ local observations, $\mathcal{A}=\left\{a_1,a_2,\dots ,a_N\right\}$ is the set consisting of all agents’ actions, $\mathcal{R}=\left\{r_1,r_2,\dots ,r_N\right\}$ is the set consisting of all agents’ rewards, and ${\mathcal{O}}^{\prime }=\left\{{o_1}^{\prime },{o_2}^{\prime },\dots ,{o_N}^{\prime }\right\}$ is the set consisting of all agents’ local observations at the next moment. Then, sample data of size B are randomly selected from D for end-to-end training from perception to action. In this case, the critic network is trained by minimizing the squared value loss between the target Q value and the estimated Q value, and the value loss function of the agent i is given in Equation (5).

$${\displaystyle L\left({\theta }_i\right)=\frac{1}{B}\sum _j^B\left[{\left(Q_i^{\pi }\left(o_i^j,o_{nei1}^j,\dots ,o_{neiM}^j;a_i^j,a_{nei1}^j,\dots ,a_{neiM}^j\right)-y^j\right)}^2\right]}$$

(5)

where $y^j=r_i^j+\gamma Q_i^{\pi \prime }\left(o_i^{\prime j},o_{nei1}^{\prime j},\dots ,o_{neiM}^{\prime j};a_i^{\prime },a_{nei1}^{\prime },\dots ,a_{neiM}^{\prime }\right){\mid }_{a_k^{\prime }={\pi }_k^{\prime }\left(o_k^j\right)}$ is the target Q value, $Q_i^{\pi \prime }\left(o_i^{\prime j},o_{nei1}^{\prime j},\dots ,o_{neiM}^{\prime j};a_i^{\prime },a_{nei1}^{\prime },\dots ,a_{neiM}^{\prime }\right)$ denotes the Q value of agent i at the next moment, and $\gamma$ is the reward discount factor.

Whereas, the actor network is trained by maximizing the cumulative expected return through gradient descent. The policy gradient of agent i can be written as

$${\displaystyle {\nabla }_{{\theta }_i}J\left({\pi }_i\right)=\frac{1}{B}\sum j\left[{\nabla }_{{\theta }_i}{\pi }_i\left(o_i^j\right)·Q_i^{\pi }\left(o_i^j,o_{nei1}^j,\dots ,o_{neiM}^j;a_i^j,a_{nei1}^j,\dots ,a_{neiM}^j\right){\mid }_{a_i^j={\pi }_i\left(o_i^j\right)}\right]}$$

(6)

where ${\pi }_i\left(o_i^j\right)$ denotes the action of agent i under observation $o_i^j$; The evaluation function of agent i is $Q_i^{\pi }\left(o_i^j,o_{nei1}^j,\dots ,o_{neiM}^j;a_i^j,a_{nei1}^j,\dots ,a_{neiM}^j\right)$, its inputs are local observation information and neighbor’s observation information, as well as local action and neighbor agents’ action, and its output is the Q value of agent i. ${\theta }_i$ is the parameter of agent i’s strategy network.

4.4. MADNPG Algorithm

The goal of the DCNPG-TSC model is to train agents to learn an optimal joint equilibrium strategy in response to the real-time changing traffic flow in the road network. In order for the DCNPG-TSC model to produce the best equalization phase strategy for the next moment, this paper proposes the MADNPG algorithm, which finds the optimal network parameters for each agent by constantly updating the agent network parameters. In order to stabilize the learning process and improve the learning efficiency, the MADNPG algorithm also employs two techniques, the target network and the experience replay. Among them, the target network technique contains an evaluation network and a target network, both of which are obtained by the value function approximation network. The evaluation network first continuously interacts with the traffic environment and stores the interaction experience into the experience replay pool. Then, a batch of experiences is randomly drawn from the experience replay pool to train the evaluation network, and the parameters of each step are updated to the target network using soft update method. The target network updates the parameters of the evaluation network using gradient descent method according to the real label of the evaluation network.

The selection of the execution phase of each intersection is very important and needs to balance the utilization and exploration of RL. In the MADNPG algorithm, at the beginning of the training, it needs to spend more effort on exploration, and the exploration rate is set to $\epsilon =0.1$, and as the number of iterations increases, the exploration rate $\epsilon$ decreases to 0.01. The agent of the MADNPG algorithm chooses its local mixed-strategy Nash equilibrium action with probability $1-\epsilon$, which is

$$a_1,a_2,\dots ,a_N=a_1^{MSNE},a_2^{MSNE},\dots ,a_N^{MSNE}$$

(7)

Algorithm 1 gives the pseudo-code of the MADNPG algorithm. First, each local agent collects experiences according to the policy for reaching the mixed-strategy Nash equilibrium until enough experiences are collected to allow for small batch updates. Next, a small batch of experiences is randomly sampled to update each actor network and critic network. Then, backpropagation is performed using the first-order gradient optimizer Adam. Finally, the training process is terminated if the maximum step size is reached or a certain stopping condition is triggered.

Algorithm 1 Multi-Agent Distributional Nash Policy Gradient

Input:  Agent set N, State Space S and Action Space A, Reward discount rate $\gamma$, Total simulation time T, batch data B, Actor network learning rate ${\eta }_w$, critic network learning rate ${\eta }_{\theta }$ Output:  trained parameters of target networks   1: Simulation time $t\leftarrow 0$, $B=\varphi$   2: Initialize parameters $\omega$, $\theta$ with random values   3: for Episode = 1 to M do   4:     while $t<T_{max}$ do   5:       for Each Agent $i\in 1,2,\dots ,N$ do   6:       for Each Agent $i\in 1,2,\dots ,N$ do   7:          Initialize sequence $o_i^t$   8:          Receive $Q_i^t$   9:          Find mixed-strategy Nash equilibrium solution ${\sigma }_i^{*}$ based on the neighbor agent’s $Q_{nei}^t$ 10:          Make decisions based on ${\sigma }_i^{*}$ ($a_i^t\sim {\sigma }_i^{*}$) and random seed 11:         end for 12:         execute actions $a=\left(a_1^t,\dots ,a_N^t\right)$ and observe new state $o^{\prime t}$ and receive the reward $r^t$ 13:         if $\left|\mathcal{D}\right|>{\mathcal{D}}_{max}$ then 14:          Remove the oldest experiences in the memory 15:         end if 16:         Store transition $\left(\mathcal{O},\mathcal{A},{\mathcal{O}}^{\prime },\mathcal{R}\right)$ in $\mathcal{D}$ 17:         if $\left|\mathcal{D}\right|>B$ then 18:          Select B samples $\left(o^j,a^j,r^j,o^{\prime j}\right)$ from $\mathcal{D}$ 19:          $y^j=r_i^j+\gamma Q_i^{\pi \prime }\left(o_i^{\prime j},o_{nei1}^{\prime j},\dots ,o_{neiM}^{\prime j};a_i^{\prime },a_{nei1}^{\prime },\dots ,a_{neiM}^{\prime }\right){\mid }_{a_k^{\prime }={\pi }_k^{\prime }\left(o_k^j\right)}$ 20:          Update critic by minimizing the loss function using Equation (5) 21:          Update actor using the sample policy gradient using Equation (6) 22:         end if 23:        end for 24:        Update target network parameters for each Agent i: ${\theta }_i^{\prime }=\tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$ 25:      end while 26: end for 27: return : ${\theta }_i$, ${\theta }_i^{\prime }$ $i\in 1,2,\dots ,N$

5. Simulation Experiments and Analysis of Results

SUMO [29], as an open source traffic simulation software, is not only powerful but also highly flexible. With SUMO, a variety of complex traffic simulation scenarios can be flexibly set up to meet specific simulation needs. Therefore, this paper adopts SUMO software (version: 1.3.1) to build a traffic simulation environment, and uses Tensorflow (version: 1.14.0) [30], a deep learning framework, to construct an agent network. To realistically and fairly compare the control effects of multiple traffic signal control methods with the DCNPG-TSC model proposed in this paper, two challenging experimental scenarios are designed which are (a) a synthetic traffic road network and (b) a real road network of 43 intersections in Jianye District, Nanjing City, Jiangsu Province, China, extracted from the real world using OpenStreetMap.

5.1. Baseline Methods

In order to verify the effectiveness and robustness of the DCNPG-TSC model in this paper, it is experimentally compared with the four most advanced RL-based traffic signal control methods. These methods include IQL-LR (Independent Q-Learning-Linear Regression), IQL-DNN (Independent Q-Learning-Deep Neural Network), IA2C (Independent Advantage Actor Critic) and MA2C (Multi-Agent Advantage Actor Critic). IQL is an independent RL method, where each agent learns its own strategy independently without considering neighboring agents information and only makes decisions based on part of the traffic environment information it perceives. IQL-LR is a (linear regression, LR) LR-based IQL, and IQL-DNN is a DNN-based IQL, and the difference lies in the different methods of fitting the Q-function. IA2C extends the IQL idea on A2C (Advantage Actor–Critic) and employs a policy-based A2C to train agents, each of which independently interacts with the intersection environment. The MA2C method, on the other hand, builds on A2C by adjusting the neighbor fingerprints to improve the observability of the domain information, and by introducing a spatial discount factor to reduce the dimensionality of the traffic states and reward signals of the neighboring intersections, so that the agent can focus more on improving the traffic conditions in the region.

5.2. Experimental Parameter Setting

To ensure a fair comparison, all traffic signal control methods were performed in the same experimental environment and used the same action space, state space and interaction frequency. The experimental hyper-parameters include (1) SUMO hyper-parameters, such as, yellow light time $t_y$, simulation time per episode $T_{max}$; (2) RL hyper-parameters, including the discount factor $\gamma$, experience replay buffer size D and batch random sampling size B; and (3) optimization algorithm hyper-parameters, for instance, parameters $\alpha$ and $\beta$ of the RMSProp optimization algorithm, which were set as experiential values. The specific settings of the parameters are shown in

Table 2. Parameter settings in the DCNPG-TSC model.

Types	Hyper-Parameter	Description	Value
Simulator	$t_y$	Yellow time interval	2
parameters	$T_{max}$	Episode time length	3600
RL parameters	$c_1$	Weighting factor in Equation (1)	0.5
	$c_2$	Weighting factor in Equation (1)	0.5
	$\gamma$	Reward discount factor	0.99
	B	Batch size of training episode	20
	$\mathcal{D}$	Size of replay memory	$1\times {10}^3$
	${\varpi }_{l_{ji}}^1$	Weighting factor in Equation (1)	0.6
	${\varpi }_{l_{ji}}^2$	Weighting factor in Equation (1)	0.4
DL parameters	${\alpha }_{RMSProp}$	RMSProp optimization parameter $\alpha$	0.99
	${\beta }_{RMSProp}$	RMSProp optimization parameter $\beta$	$1\times {10}^{-5}$
	${num}_{MLP}$	Number of MLP units in the encoding layer	64
	${num}_{Hidden}$	Three hidden layer in the actor and critic Network	64 + 64 + 64
	${num}_{Output}$	Output layer in the actor and critic Network	Number of Agents

.

5.3. $4\times 4$ Synthesis Grid Net

The synthetic road network consists of 16 controlled intersections, as shown in Figure 6a, each of which is a six-entrance lane intersection with a two-lane east–west speed limit of 70 km/h and a single lane north–south speed limit of 40 km/h. The action space of each intersection contains four optional phases, which are S-N straight and right-turn phase, S-N left-turn phase, E-W straight and right-turn phase and E-W left-turn phase. In order to make the Markov Decision Process problem more challenging, the experiment simulates four time-varying traffic flow groups in the form of origin–destination, and at the beginning moment of the simulation, Router 1 is generated by the four traffic streams $x1\text{-}x9$, $x2\text{-}x10$, $x3\text{-}x11$ and $x4\text{-}x12$, and Router 2 is generated by the four traffic streams $x5\text{-}x13$, $x6\text{-}x14$, $x7\text{-}x15$ and $x8\text{-}x16$. In the figure, the blue solid line and the red dashed line represent the Router 1 and the Router 2, respectively, and the Router 3 and the Router 4 are their counterflow. Fifteen minutes after the beginning of the simulation, the traffic flow of Router 1 and Router 2 starts to decrease and the traffic flow of Router 3 and Router 4 start to be generated. Figure 6b shows the variation of the four traffic flows over time. The above defines the variation in the traffic flow with the simulation time, while the route of each vehicle is randomly generated at runtime without experimental setup.

In order to reflect the overall congestion of the traffic road network, the experiment uses average queue length to evaluate the model. Figure 7 gives a graph of the average queue length in a synthetic road network with simulation time for five RL-based traffic signal control methods including the DCNPG-TSC model in this paper, where the solid line indicates the average value and the shading indicates the standard deviation of the evaluation set. As can be seen from Figure 7, the average queue lengths of IQL-DNN, IQL-LR, MA2C and IA2C show a gradual increasing trend, and the curve increases steeply after 900 s, which is due to the generation of convective traffic groups Router 3 and Router 4 leading to more and more traffic into the synthetic road network after 15 min. From the curves in Figure 7, it can be seen that the average queue length of the DCNPG-TSC methods is significantly smaller than the other four methods because the DCNPG-TSC method tends to choose a more stable mixed-strategy Nash equilibrium that exploits the information game at the neighborhood intersections to find a lower congestion level strategy.

Figure 8 plots the variation in average intersection delay with simulation time in the synthetic road network; the average delay curve is similar to the average queue length curve in that both show a gradual increase. From the figure, it can be seen that the delay curves of IQL-DNN and IQL-RL show a monotonically increasing trend and keep on increasing sharply with the simulation time, indicating that they are unable to recover from congestion. The delays in IA2C, MA2C and MADNPG increase slowly, and the MADNPG algorithm also outperforms the other MARL traffic signal control methods by consistently maintaining the lowest level of delay.

In order to further validate the effectiveness of each traffic signal control method, vehicle-based measurements are used in this paper to evaluate the effectiveness of various traffic signal control methods in terms of vehicle energy consumption and pollutant gas emissions.

Table 3. Vehicle-based measurements in a $4\times 4$ synthetic road network.

Metric	IQL-DNN	IQL-LR	IA2C	MA2C	DCNPG-TSC
Avg. Speed (m/s)	1.306	1.586	2.016	2.135	2.102
Avg. Queue (m)	13.138	12.874	9.765	9.447	8.580
Avg. Wait (s)	277.435	216.797	47.569	44.775	32.970
Fuel consumption (mL/s)	1.118	1.109	0.896	0.903	0.889
CO NOx emission (mg/s)	65.367	59.483	45.422	44.988	44.745

summarizes the vehicle-related information measured for various RL-based traffic signal control methods in the synthetic road network, including the average vehicle speed (avg. speed, unit: m/s), average queue length (avg. queue, unit: m), average stopping time (avg. waits, unit: s), fuel consumption (unit: mL/s), and average CO and NOx emission (unit: mg/s). It can be seen that the IQL-DNN performs poorly in all aspects due to the fact that IQL-DNN learns the traffic conditions independently while also suffering from high variance. The DCNPG-TSC model of the synthetic road network performs the best in all aspects, having the lowest average queue length, average stopping time, fuel consumption, and the least CO and NOx gas emissions, and the average speed of the vehicles is second only to the MA2C method. This is due to the fact that the DCNPG-TSC model learns a joint optimal policy to optimize signal control at intersections in urban road networks through a neighborhood game.

5.4. Real Urban Road Network

In order to verify the performance of the DCNPG-TSC model in a real road network, a real traffic road network in Jianye District, Nanjing City, Jiangsu Province, China, is selected for simulation experiments in this section, as shown in Figure 9a. This real road network has various types of roads and intersections, with a total of 43 controlled intersections, 42 two-phase intersections and 1 three-phase intersection. In order to test the performance of the algorithm in the challenging Adaptive Traffic Signal Control (ATSC) scenario, dense, stochastic and time-varying traffic flows are designed to simulate rush hour traffic. Specifically, four traffic flow groups are generated from multiples of unit flow and randomly sampled origin–destination pairs in a given area, as shown in Figure 9b. Among them, F1 and F2 transform the simulated traffic flows every 5 min interval for the first 40 min, and the eight simulated flows transformed are [1, 2, 4, 4, 4, 4, 4, 4, 2, 1] multiples of the unit flow, whereas the traffic flows F3 and F4 are generated starting from 15 min and ending the generation at 55 min.

In order to verify the performance of the model in a complex real road network, simulation experiments on a real road network are conducted in this paper. Figure 10 plots the variation in average vehicle queue length with simulation time for various RL control methods in the real road network. From the figure, it can be seen that the IQL-LR and IQL-DNN methods perform poorly because their queue lengths show an incremental change with the simulation time, which cannot effectively inhibit the growth of queue lengths. The IA2C, MA2C, and DCNPG-TSC models all gradually reduce the vehicle queue lengths after reaching the inflection point, which indicates that these methods can effectively inhibit the queue lengths from increasing. Among them, the DCNPG-TSC model performs the best, presenting the lowest queue length.

In order to more accurately assess the performance of the DCNPG-TSC model, Figure 11 plots the average delay time of vehicles at intersections with the simulation time for various RL control methods in a real road network. From Figure 11, it can be seen that both IQL-DNN and IQL-LR methods perform poorly. MA2C, IA2C and DCNPG-TSC are all able to reduce delay time after peaks; however, both MA2C and IA2C fail to maintain consistently low intersection delays. This is because the upstream intersections of MA2C and IA2C greedily maximize their local flows leading to congestion, whereas DCNPG-TSC is more able to distribute traffic flows more evenly by gaming its neighboring intersections, which in turn leads to lower and more sustainable intersection delays.

In order to further validate the performance of the DCNPG-TSC model in this paper in terms of fuel consumption and pollutant gas emissions, experimental validation of each of the five methods is carried out, and the results are shown in

Table 4. Vehicle-based measurements in the real road network in Jianye District, Nanjing, Jiangsu Province, China.

Metric	IQL-DNN	IQL-LR	IA2C	MA2C	DCNPG-TSC
Avg. Speed (m/s)	4.877	2.414	3.827	3.904	3.946
Avg. Queue (m)	2.756	4.173	2.537	2.550	2.404
Avg. Wait (s)	84.413	114.932	64.075	58.634	52.646
Fuel consumption (mL/s)	1.446	1.299	1.266	1.274	1.242
CO NOx emission (mg/s)	59.070	67.462	53.477	52.944	53.179

. Table 4 summarizes the various traffic performance metrics of each RL method in the real road network. The fuel consumption of IA2C and MA2C are, respectively, reduced by $12.4\%$ and $11.9\%$ compared to the IQL-DNN method. The exhaust emissions of IA2C and MA2C are, respectively, reduced by $9.5\%$ and $10.3\%$ compared to the IQL-DNN, whereas DCNPG-TSC reduces the fuel consumption by $14.1\%$ and exhaust emissions by $10\%$ compared to IQL-DNN. It can be seen that the DCNPG-TSC model outperforms the other baseline methods in almost all the metrics, achieving the lowest average queue lengths and the lowest intersection waiting times, as well as the lowest fuel consumption and less CO and NOx gas emissions. This is attributed to the fact that the agent network design allows the intersection agents to have access to a larger perceptual domain for better equilibrium gaming and collaboration, and the agent gaming mechanism results in a reasonable distribution of traffic flow, which optimizes the overall operation of the traffic system and improves traffic efficiency.

6. Conclusions

To deal with the increasingly severe traffic congestion problem, this paper constructs a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which is an adaptive multi-intersection traffic signal control model based on mixed-strategy Nash equilibrium and MARL. Specifically, firstly, through the design of the agent network structure, the DCNPG-TSC model can obtain more real-time traffic spatio-temporal state data and realize effective communication and collaboration of agents. Secondly, the Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm used in the DCNPG-TSC model finds an optimal joint equalization strategy for each agent in the traffic road network to respond to the real-time changing traffic flow. Finally, the DCNPG-TSC model combines the green transportation concept with MARL traffic signal control and sets up a multi-optimization objective reward mechanism including reducing pollutant gas emissions and fuel consumption to reduce energy consumption and pollutant gas emissions. The experimental results of the synthetic road network and the real road network show that the DCNPG-TSC model proposed in this paper outperforms the other most advanced RL traffic signal control methods, which proves the effectiveness and scalability of the DCNPG-TSC model.

In spite of the progress in this paper, the research has the following limitations and constraints. Firstly, the gaming process takes more time, which affects the efficiency of model training; secondly, only the vehicle traffic at intersections is considered, and pedestrian traffic is not taken into account, which has some limitations in real traffic; thirdly, only the pollutant gases emitted by vehicles are focused on, and the particulate matter emitted by vehicles is ignored, which is an omission in comprehensively evaluating the pollutant emissions from vehicles.

Despite some limitations, this research still has important practical significance, especially for stakeholders such as traffic control centers and environmental agencies. For traffic control centers, the research can assist in designing more effective signal control strategies to reduce congestion and improve the efficiency of vehicular traffic. For environmental agencies, the research can help to accurately monitor the environmental impacts of vehicular pollutant gas emissions and develop effective environmental protection strategies to improve environmental quality and protect public health.

There is still some work required to be further studied. Firstly, the gaming process could be accelerated to reduce the gaming time consumed during the training of the DCNPG-TSC model and improve the training efficiency of the model. Secondly, more external data could be considered, such as pedestrians, weather, and so on, to further improve the performance of the DCNPG-TSC model. Thirdly, the measurement of vehicle pollutants, including particulate matter, can be improved to achieve a more accurate assessment of vehicle pollutant emissions.