A Q-Learning-Based Approximate Solving Algorithm for Vehicular Route Game

Abstract

Route game is recognized as an effective method to alleviate Braess’ paradox, which generates a new traffic congestion since numerous vehicles obey the same guidance from the selfish route guidance (such as Google Maps). The conventional route games have symmetry since vehicles’ payoffs depend only on the selected route distribution but not who chose, which leads to the precise Nash equilibrium being able to be solved by constructing a special potential function. However, with the arrival of smart cities, the real-time of route schemes is more of a concerned of engineers than the absolute optimality in real traffic. It is not an easy task to re-construct the new potential functions of the route games due to the dynamic traffic conditions. In this paper, compared with the hard-solvable potential function-based precise method, a matched Q-learning algorithm is designed to generate the approximate Nash equilibrium of the classic route game for real-time traffic. An experimental study shows that the Nash equilibrium coefficients generated by the Q-learning-based approximate solving algorithm all converge to 1.00, and still have the required convergence in the different traffic parameters.

1. Introduction

Traffic congestion has become a major issue for city managers, since it causes serious direct and indirect economic losses in modern cities [1,2,3]. Vehicle route guidance (VRG) can adjust the trajectories of travelers in the space dimension to mitigate the traffic congestion [4]. According to the quantity of agents, route guidance can be divided into the single-agent-based selfish VRG and the multi-agent-based coordinated VRG.

A selfish VRG system provides a single vehicle’s optimal path without considering others’ route schemes [5]. The drivers divert to the recommended free-flowing roads after receiving the congestion information. For example, based on the requirements of vehicles, the studies in [6,7,8] use the traffic data obtained from the selfish route guidance systems to choose paths by predicting traffic trends. In the past, due to the limitations of the old on-board computers, the vehicles struggle to interact with other agents in fast-moving [9]. Selfish route guidance systems (such as the Google Maps) are adopted as the mainstream method for vehicle navigation [10].

With the proliferation of selfish VRG systems, however, researchers find that the road efficiency is reduced due to Braess’ paradox (the traffic congestion switches to an alternative road since the same optimal route is recommended to numerous vehicles) [11]. Braess’ paradox may reduce the running efficiency of the discrete systems such as VRG, that has been studied in many journals such as Operations Research [12,13,14,15].

Recently, with the emergence of the Internet of vehicles (IoVs), the multi-agent-based coordinated VRG (modeled by route game) becomes possible for alleviating the Braess’ paradox [16]. The route game model can consider users’ routes systematically [17]. For example, the studies in [18,19] design two route games based on the assumptions of mixed and pure strategy, respectively. The research team in [20] investigates the information perturbations of IoVs on the proposed route game in [18]. Moreover, based on the route game in [19], the work in [21] builds an information perturbation-based route game model under the mixed-strategy assumption. In the above route games, the vehicles and their requirements are set as player and payoffs, respectively. The Nash equilibrium represents the final route schemes of the coordinated VRG.

For solving route games that have multi-players, some milestone theorems have been proved. The study in [22] proposes the famous “potential function” to solve the congestion games which are usually used to simulate the behaviors of independent players competing for the limited resources (such as the route game in VRG problem). Subsequently, the authors in [23,24,25] further study the potential functions in various structure-based congestion games (such as the weighted congestion games). The potential function-based solving method is a precise algorithm to generate the pure strategy Nash equilibrium of the existing route games.

In today’s smart cities, the real-time of traffic management systems is more concerned by engineers [26]. However, the work of building the potential function is a manual task. Moreover, it is not an easy task to re-construct the new potential functions for the route games due to the dynamic traffic conditions [27]. Hence, compared with the hard-solved absolute optimality, the approximate optimal routes that can be solved effectively are more significant in real coordinated VRG.

For the existing literature of approximate Nash equilibrium, intelligent algorithms (such as heuristic, machine learning and reinforcement learning) are often adopted to calculate the approximate solutions of game theories [28]. For example, the study in [29] presents a concurrent learning to obtain an approximate Nash equilibrium for an N-player nonzero-sum game. The work in [30] uses a stackelberg game to model the interactions between the content providers and the edge caching devices, and designs a reinforcement learning to solve the approximate Nash equilibrium for the proposed game model. In [31], the approximate Nash equilibrium is used to solve a multi-objective optimization problem combining by the particle swarm optimization and the self-organizing mapping neural network. Among the above intelligent algorithm, the reinforcement learning is a method that can approximate the optimal solutions by self-learning in training [32]. It can describe the multistage discrete decisions such as in the pure strategy-based game theory.

To adapt to the real-time of smart traffic, compared with the existing hard-solvable potential function-based precise algorithms, a matched Q-learning-based solving method (a type of reinforcement learning) is designed in this paper to generate the approximate Nash equilibrium for the classic route game. Specifically, based on the features of the route game, the virtual strategy set of all vehicles is set as the current state. Only one vehicle updates its virtual strategy based on the optimization rules in each iteration. For the reward function, the concept of Nash equilibrium coefficient (quantifying the degree of Nash equilibrium) is introduced to evaluate the action in each iteration. Using the established Q-learning algorithm, the approximate Nash equilibrium of the route game can converge to a unique solution in the basic traffic scenarios.

Finally, a microscopic traffic scenario and two typical experimental methods (the potential function-based precise algorithm and the designed Q-learning approximate method) are designed to test the validity and robustness of the contributions. With the example comparisons, the results show that the Nash equilibrium coefficients generated by the Q-learning approximate solving algorithm all converge to 1.00, which means that the obtained approximate routes are the same as the precise Nash equilibrium generated by the potential function. Moreover, the robustness analysis shows that the Q-learning-based solving algorithm still has required convergence in the different traffic parameters. The main contributions of this paper are summarized as follows:

• To adapt to the real-time of smart traffic, the Nash equilibrium coefficient of route games is proposed in this paper, which means the proportion of vehicles whose current route strategies are optimal;
• A Q-learning-based approximate solving algorithm is designed to generate the coordinated route schemes for the classic route game. This approximate Nash equilibrium-based method is more suitable for the dynamic traffic compared with the hard-solved precise solving algorithm;
• A decentralized route coordination framework (that can be applied to large road networks based on the assumption of IoVs) is built to alleviating Braess’ paradox in VRG.

The rest of this paper is organized as follows. Section 2 illustrates the assumptions and formalizations of the art. Section 3 describes the contributions of the model and solving algorithm. Section 4 presents the running experiment and comparative results. Conclusions and potential future works are given in Section 5.

2. Basic Assumptions and Formalizations

In this section, the preliminary notions and the classic route game are formalized based on the available assumptions.

2.1. Concepts of the Art

There are some concepts that have not been explained at length in the introductory section:

• Braess’ paradox: with the proliferation of selfish VRG systems, the road efficiency is reduced since numerous vehicles obey the same guidance from the selfish VRG systems [15];
• Rout games: In a route game, the vehicles and their requirements are set as players and payoffs, respectively. The Nash equilibrium represents the final coordination routes. The route game can generate the route strategies systematically for a group of vehicles based on agents’ interaction relationship [17];
• Pure/mixed strategy: Pure strategy denotes a certain action chosen by each player in a game theory. Mixed strategy denotes a random action distribution with certain probability [33];
• Symmetrical games: All agents are non-personalized, which leads to the symmetry of the payoff matrix in the game. The symmetry makes the symmetrical games have some special mathematical properties [20].

2.2. Formalization of the Route Game

2.2.1. Description of the Traffic Scenario

Route game is recognized as an effective way to alleviate Braess’ paradox, which generates a new traffic congestion on one alternative road since numerous vehicles obey the same guidance from the selfish VRG systems (such as Google Maps). For the sake of a concise model, referring to the study in [34] (which researches the price of anarchy for selfish routing), a basic traffic scenario with two contrastive paths is adopted to describe the classic route game. The details of the micro road network are shown in Figure 1.

In Figure 1, let $I=\left\{i\left|i=1,2,\dots ,n\right.\right\}$ be the set of vehicles and $t_i$ be the travel time of vehicle i. To reduce the calculations without weakening the contribution description, the pure strategy is adopted in this paper’s route game, and it is assumed that the vehicles have the same origin-destination from point O to point D. All vehicles are equipped with IoVs devices and no overtaking occurs. Based on the IoVs technique, the driving data can be exchanged between vehicles in fast-moving [35].

Each vehicle has to choose a path to its destination D. Let $\mathrm{E}=\left\{\epsilon \left|\epsilon =1,2,\dots ,m\right.\right\}$ be the set of the alternative paths. In Figure 1, there are two optional paths for each vehicle i ($\epsilon =1$: $\mathrm{O}\to 3\to 1\to 2\to \mathrm{D}$; $\epsilon =2$: $\mathrm{O}\to 3\to 4\to 2\to \mathrm{D}$). Assume that the traffic lights are all green to eliminate the influence from non-variable factors. All traffic parameters use standardized units.

In driving, all vehicles originally chose the shortest paths as their route strategies. This could generate traffic congestion since the road load exceeds the limit of flow. When traffic congestion occurs, the widely used selfish VRG systems (such as Google Maps) usually recommend the optimal alternative path to all vehicles. Due to the same route diversion, the selfish VRG may lead to Braess’ paradox, where traffic congestion switches from one road to an alternative one. Route games can alleviate the negative impact of Braess’ paradox by considering users’ routes systematically.

2.2.2. Establishment of the Classic Route Game

A complete route game consists of four elements: players, strategies, payoff functions, and game rules [36]. Similar to the existing route games [18,19,20,21], let vehicle $i\left(i\in I\right)$ be the player, the alternative path set $\mathrm{E}$ be the pure strategy set, and the static as the rules. Let ${\theta }_i$ (${\theta }_i={\epsilon }_i\in \mathrm{E}$ in pure strategy) and ${\pi }_i$ be the strategy and payoff of vehicle i, respectively. The value of ${\pi }_i$ depends on the selection of ${\theta }_i$ and ${\theta }_{-i}$:

$${\theta }_{-i}=({\theta }_1,{\theta }_2,\cdots ,{\theta }_{i-1},{\theta }_{i+1},\cdots ,{\theta }_n).$$

Travel time $t_i$ is usually adopted as payoff ${\pi }_i$ for vehicle i. For the travel time of vehicles, the engineers usually adopt the BPR formulation:

$${\pi }_i({\theta }_i,{\theta }_{-i})=t_i=t_{{\epsilon }_i}(1+\alpha {(\frac{N_{{\epsilon }_i}}{C_{{\epsilon }_i}})}^{\beta })$$

(1)

to estimate, where $t_{{\epsilon }_i}$ denotes the basic travel time of path ${\epsilon }_i$, $C_{{\epsilon }_i}$ denotes the capacity of path ${\epsilon }_i$, $N_{{\epsilon }_i}$ denotes the current vehicle volume of edge ${\epsilon }_i$, and $\alpha$ and $\beta$ are constants that are fit by huge traffic data. $N_{{\epsilon }_i}$ depends only on the vehicle route strategies:

$$N_{{\epsilon }_i}={\displaystyle \sum _{i^{\prime }\in -i}^{}}\mathrm{T}\{{\theta }_{i^{\prime }}={\theta }_i\}.$$

(2)

In the built pure strategy-based route game, each vehicle tries to improve its payoff $\pi$ by selecting an optimal strategy $\theta$. Let $\Theta =({\theta }_1,{\theta }_2,\dots ,{\theta }_n)$ be the selected strategies of all vehicles. The Nash equilibrium means the final coordination routes maintain a steady-state for all vehicles.

Definition 1.

A Nash equilibrium in the pure strategy-based game theory is a special strategy set ${\Theta }^{*}=({\theta }_1^{*},{\theta }_2^{*},{\theta }_3^{*},\dots ,{\theta }_n^{*})$ satisfying

$${\pi }_i\left({\Theta }^{*}\right)\le {\pi }_i({\theta }_1^{*}\dots ,{\theta }_{i-1}^{*},{\theta }_i,{\theta }_{i+1}^{*}\dots ,{\theta }_n^{*}),\forall i\in I,{\theta }_i\in \mathrm{E}.$$

It means that any player who changes strategy alone will not get a better payoff [37]. The route game steps are as follows:

• Step 1. The game information (such as the strategies and payoff functions) is obtained by vehicles based on the IoVs technology;
• Step 2. Based on the built route game $G=\{\Theta ;{\pi }_1,{\pi }_1,\cdots {\pi }_n\}$, the on-board computer of each vehicle predicts the route strategies of others;
• Step 3. Based on the prediction result of other vehicles’ route strategies, all vehicles simultaneously calculate their own optimal route strategies θ and receive the payoff π.

The payoff function Equation (1) shows that the route game is symmetric. It means the payoff of each vehicle depends only on the strategy distribution but not who chooses (all players are non-personalized). A large number of basic road units form a complex traffic network. The built classic route game in this paper can be extended to the large road network due to its decentralized feature.

3. Contributions to Solving Algorithms

In this section, based on the symmetry of the classic route game, the precise solving algorithm is constructed by the potential function as the control group. Subsequently, considering the limitation of the precise algorithm on smart roads, a matched Q-learning-based approximate solving algorithm is designed to calculate the coordination routes of the classic route game.

3.1. Potential Function-Based Precise Algorithm

The symmetry of payoff matrix is the significant feature in the classic route game. In [38], based on this mathematical nature, the existence of the pure strategy Nash equilibrium is proved in symmetrical games. Subsequently, the study in [22] finds that the solutions of the potential functions are the Nash equilibrium in potential games. Considering the features of the route game built in Section 2.2, the proof processes of the precise Nash equilibrium are as follows:

When the path ${\epsilon }_i$ is chosen by vehicle i, there is

$$x_{{\epsilon }_i}=\left\{\begin{array}{c}1\mathrm{path}{\epsilon }_i\mathrm{is}\mathrm{selected}\mathrm{by}\mathrm{vehicle}i\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.{\epsilon }_i=1,2,\cdots ,m$$

(3)

where ${\displaystyle \sum _{{\epsilon }_i=1}^m}{x}_{{\epsilon }_i}=1$. That is, each vehicle chooses one of the alternative path $\epsilon$ as its route strategy. Let the selection times of path $\epsilon$ ($\epsilon \in \mathrm{E}$) be $N_{\epsilon }$:

$$N_{\epsilon }-\sum _{i=1}^n\mathrm{T}\{\epsilon ={\epsilon }_i\}=0.$$

(4)

Let $c_{\epsilon }\left(k_{\epsilon }\right)$ be the expected payoff of the vehicles who select the path $\epsilon$ chosen by $k_{\epsilon }$ vehicles. The value of $c_{\epsilon }\left(k_{\epsilon }\right)$ is calculated by Equation (1). The potential function of the established route game can be constructed by

$$min\sum _{\epsilon \in \mathrm{E}}\sum _{k=0}^{N_{\epsilon }}{c}_{\epsilon }\left(k_{\epsilon }\right).$$

(5)

Subject to:

$$\left\{\begin{array}{c}{x}_{{\epsilon }_i}=\left\{\begin{array}{c}1\mathrm{path}{\epsilon }_i\mathrm{is}\mathrm{selected}\mathrm{by}\mathrm{vehicle}i\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.\hfill \\ {\displaystyle \sum _{{\epsilon }_i=1}^m}{x}_{{\epsilon }_i}=1\hfill \\ N_{\epsilon }={\displaystyle \sum _{i=1}^n}\mathrm{T}\{\epsilon ={\epsilon }_i\}\hfill \\ c_{\epsilon }\left(k_{\epsilon }\right)=t_{free-flow}^{\epsilon }(1+\alpha {(\frac{k_{\epsilon }}{C_{\epsilon }})}^{\beta }).\hfill \end{array}\right.$$

(6)

Let ${\mathrm{E}}^{*}=({\epsilon }_1^{*},{\epsilon }_2^{*},\dots ,{\epsilon }_n^{*})$ be a coordination route set for all vehicles, which satisfies the solution of the potential function Equation (5). If route set ${\mathrm{E}}^{*}$ is not a Nash equilibrium, there must exist a path $\epsilon {}_i{}^{\prime }$ to make vehicle i arrive a shorter travel time. That is

$$c_{\epsilon {}_i{}^{\prime }}(k_{\epsilon {}_i{}^{\prime }}+1)<c_{\epsilon {}_i{}^{*}}\left(k_{\epsilon {}_i{}^{*}}\right).$$

(7)

When the selected path ${\epsilon }_i^{*}$ of vehicle i changes to $\epsilon {}_i{}^{\prime }$, let $N_{\epsilon }{}^{\prime }$ denote the vehicle quantity choosing the path $\epsilon$. Inspired by the conclusions in [22], there exists

$$\begin{array}{c}{\displaystyle \sum _{\epsilon \in \mathrm{E}}}{\displaystyle \sum _{k=0}^{N_{\epsilon }{}^{\prime }}}{c}_{\epsilon }\left(k_{\epsilon }\right)\hfill \\ ={\displaystyle \sum _{\genfrac{}{}{0pt}{}{\epsilon \in \mathrm{E}\hfill }{\genfrac{}{}{0pt}{}{\epsilon \notin \epsilon {}_i{}^{\prime }\hfill }{\epsilon \notin {\epsilon }_i^{*}\hfill }}}}{\displaystyle \sum _{k=0}^{N_{\epsilon }}}{c}_{\epsilon }\left(k_{\epsilon }\right)+{\displaystyle \sum _{k=0}^{N_{\epsilon {}_i{}^{\prime }}}}{c}_{\epsilon {}_i{}^{\prime }}\left(k_{\epsilon {}_i{}^{\prime }}\right)+{\displaystyle \sum _{k=0}^{N_{{\epsilon }_i^{*}}}}{c}_{{\epsilon }_i^{*}}\left(k_{{\epsilon }_i^{*}}\right)\hfill \\ ={\displaystyle \sum _{\genfrac{}{}{0pt}{}{\epsilon \in \mathrm{E}\hfill }{\epsilon \notin {\epsilon }_i^{*}\hfill }}}{\displaystyle \sum _{k=0}^{N_{\epsilon }}}{c}_{\epsilon }\left(k_{\epsilon }\right)+c_{\epsilon {}_i{}^{\prime }}(k_{\epsilon {}_i{}^{\prime }}+1)+{\displaystyle \sum _{k=0}^{N_{{\epsilon }_i^{*}}}}{c}_{{\epsilon }_i^{*}}\left(k_{{\epsilon }_i^{*}}\right)\hfill \\ ={\displaystyle \sum _{\epsilon \in \mathrm{E}}}{\displaystyle \sum _{k=0}^{N_{\epsilon }}}{c}_{\epsilon }\left(k_{\epsilon }\right)+c_{\epsilon {}_i{}^{\prime }}(k_{\epsilon {}_i{}^{\prime }}+1)-c_{{\epsilon }_i^{*}}\left(k_{{\epsilon }_i^{*}}\right)\hfill \\ <{\displaystyle \sum _{\epsilon \in \mathrm{E}}}{\displaystyle \sum _{k=0}^{N_{\epsilon }}}{c}_{\epsilon }\left(k\right).\hfill \end{array}$$

The existence of the precise Nash equilibrium is deduced mathematically in the built route game.

If Equation (7) is true, there must exist $\epsilon {}_i{}^{\prime }$ to make the value of Equation (5) smaller. This conclusion is inconsistent with ${\mathrm{E}}^{*}$ being the solution of Equation (5). It is proved that Equation (7) does not exist. That is, the vehicle route set satisfying the potential function Equation (5) is the pure strategy Nash equilibrium of the route game.

3.2. Q-Learning-Based Approximate Solving Algorithm

3.2.1. Definition of Approximate Nash Equilibrium

In today’s smart cities, the real-time of traffic management systems is more concerned by engineers. However, the work of building the potential function is a manual task. And it is not an easy work to re-construct the new potential functions due to the dynamic traffic conditions. Compared with the hard-solved potential function-based precise algorithm, an approximate route scheme that can be solved effectively is more significant in real traffic [39].

In this section, to fill the above gap, the concept of the Nash equilibrium coefficient $\eta$:

$$\eta =\frac{{\displaystyle \sum _{i=1}^n}\mathrm{T}\{{\pi }_i({\theta }_i,{\theta }_{-i})\le {\pi }_i(\theta {}_i{}^{\prime },{\theta }_{-i})\}}{n}$$

is introduced into the route game, where $\mathrm{T}\{·\}$ denotes the indicator function and $\theta {}_i{}^{\prime }$ denotes any selectable path for vehicle i. The value of Nash equilibrium coefficient $\eta$ is calculated by this definition.

According to the definition of Nash equilibrium coefficient $\eta$, if other vehicles’ route strategies ${\theta }_{-i}$ remain the same, the value of $\mathrm{T}\{{\pi }_i({\theta }_i,{\theta }_{-i})\le {\pi }_i(\theta {}_i{}^{\prime },{\theta }_{-i})\}$ is 1.00 when the current route strategy ${\theta }_i$ is optimal for vehicle i. Nash equilibrium coefficient $\eta$ means the proportion of the vehicles whose current route strategy is optimal.

In the route game proposed in this paper, the approximate Nash equilibrium $\eta$ is created to replace the function of the precise Nash equilibrium. When the Nash equilibrium coefficient reaches 1.00 ($\eta =1.00$), the corresponding route strategy set denotes the pure strategy Nash equilibrium of the route game.

3.2.2. Q-Learning Matched with the Route Game

For the existing literature regarding approximate Nash equilibrium, intelligent algorithms (such as heuristic, machine learning and reinforcement learning) are often adopted to calculate the approximate solutions of game theories [28]. Among them, Q-learning, which approximates the optimal solutions by self-learning in training, can describe the multistage discrete decisions in the pure strategy-based game theory [21].

In this subsection, combined with the features of the route game, a matched Q-learning-based solving method (a type of reinforcement learning) is designed to generate the approximate Nash equilibrium for the classic route game. A complete Q-learning algorithm consists of four elements—reward function, optimization rule, state set and action set [18,19,20].

For the $k_{th}$ iteration in Q-learning, the virtual strategy set of all vehicles is set as the state $s_k$:

$$s_k=({\theta }_1^k,{\theta }_2^k,\dots ,{\theta }_i^k,\dots ,{\theta }_n^k),{\theta }_i^k\in \Theta$$

where $\theta {{}_i}^K$ denotes the virtual strategy of vehicle i in $k_{th}$ iteration. That is, the state of each iteration is actually a feasible route scheme for the route game.

For the selection action $a_k$ in the state $s_k$, only one vehicle updates its virtual strategy based on the optimization rules. There is

$$a_k=(i,{\theta }_i^{k+1}),i\in I,{\theta }_i^{k+1}\in \Theta$$

where ${\theta }_i^{k+1}$ denotes the virtual strategy of vehicle i in state $s_{k+1}$. The value of ${\theta }_i^{k+1}$ is generated by famous $\epsilon -\mathrm{Greedy}$ algorithm (note that here $\epsilon$ is not the notation of the selectable path). The $\epsilon -\mathrm{Greedy}$ algorithm is a conventional algorithm, whose specific steps are not described in this paper due to the space limitation.

About the state updating, there is

$$s_{k+1}=\Phi (s_k,a_k)$$

(8)

where $\Phi$ denotes the certain mapping function from $s_k$ to $s_{k+1}$. It means that the next state $s_{k+1}$ depends only on the current state $s_k$ and action $a_k$.

For the reward function R, the Nash equilibrium coefficient $\eta$ is used to evaluate the action $a_k$ in each iteration:

$$R_k(s_k,a_k)=\frac{{\displaystyle \sum _{i=1}^n}\mathrm{T}\{{\pi }_i\left(s_{k+1}\right)\le {\pi }_i({{\theta }_i}^{{}^{\prime }},{\theta }_{-i}^{t+1})\}}{n}.$$

(9)

R means the proportion of the vehicles whose virtual route strategies are optimal. The values of $s_{k+1}$ and ${\pi }_i$ are generated by Equations (8) and (1), respectively.

The value-action matrix Q is

$$Q(s_k,a_k)=(1-\sigma )Q(s_k,a_k)+\sigma ·(R_k+\gamma ·\underset{a\in A}{max}Q(s_k,a)).$$

(10)

where $\sigma$ denotes the learning step and $\gamma$ denotes the discounted factor. The running details of the designed Q-learning are shown in Figure 2.

In Figure 2, there may exist more than one Nash equilibrium in the route game. The final route solution ${\Theta }^e$ depends on initial ${\Theta }^0$. The dimension of the Q-value matrix is $\left|S\right|\times \left|A\right|$, where S denotes the set of states and A denotes the set of actions. The training task is to converge the value-action matrix Q. The time complexity of the designed Q-learning algorithm is $O(n^3)$ (note that here n is not the notation of vehicle quantity).

Due to the limited space, the calculation details of the Q-learning-based approximate Nash equilibrium are not described in this section. Its convergence is analyzed in Section 4 with numerical experiments.

3.3. Discussion

In this section, firstly, the potential function-based precise solving algorithm is constructed as the control group. Subsequently, considering the limitation of the precise algorithm on real-time roads, a matched Q-learning-based algorithm is designed to generate the approximate Nash equilibrium for the route game. In the Q-learning-based approximate solving method, the concept of approximate Nash equilibrium is introduced into the route game. This approximate Nash equilibrium-based method is more suitable for the dynamic traffic compared with the hard-solved precise solving algorithm

4. Numerical Experiments

In this section, a representative traffic scenario is built to verify the proposed contributions in the micro level. Firstly, as a control group, the potential function-based precise solving algorithm is worked on four traffic scenarios with differentiated traffic parameters. It is used to prove the feasibility of the route game established in this paper. Then, the convergence of the Q-learning-based approximate solving algorithm is studied in the same traffic scenarios, and the converged values are compared with the precise solutions. Finally, the robustness of the Q-learning-based approximate solving algorithm is tested.

4.1. Preparation

For the numerical experiments, the running speed depends on the computer performance and programming language. In this paper, MATLAB R2016a tool is used. The default parameters are accepted, and the processor of the computer is AMD Ryzen 74800U with Radeon Graphics 1.80 GHz.

The basic traffic scenario in Figure 1 is adopted to verify the contributions in this paper. To reduce the error caused by randomness, four different parameters (shown in

Table 1. Traffic parameters of numerical experiments.

Traffic Scenario	n	m	${\mathit{C}}_{\mathit{\epsilon }=1}$	${\mathit{C}}_{\mathit{\epsilon }=2}$	${\mathit{t}}_{\mathit{free}-\mathit{flow}}^{\mathit{\epsilon }=2}$	${\mathit{t}}_{\mathit{free}-\mathit{flow}}^{\mathit{\epsilon }=1}$
Scenario 1	10	2	10	10	30	26
Scenario 2	10	2	15	10	25	20
Scenario 3	10	2	10	15	30	24
Scenario 4	10	2	30	30	30	30

Note: All traffic parameters adopt the standardized units.

) are used in the micro road network.

As shown in Table 1, in four traffic scenarios, the vehicle number $n=10$, 10 and the quantity of alternative paths $m=2$. Traffic scenarios 1, 2 and 3 are general, whose alternative roads are different. For traffic scenario 4, it is a special road network that the traffic conditions of all alternative roads are undifferentiated.

In the payoff function Equation (1), let constants $\alpha$ and $\beta$ be 1.00. For the parameters of Q-learning, similar to the general parameter settings, we set learning step $\sigma =0.2$, discounted factor $\gamma =0.9$, greedy coefficient $\epsilon =0.8$, and maximum iteration times $\mathrm{M}=100$. The variation of these parameters will affect the speed of convergence but not the accuracy [40].

4.2. Effectiveness of the Precise Algorithm (Control Group)

A potential function-based precise solving algorithm for the route game is described in Section 3.1. It is used to prove the effectiveness of the route game that can alleviate Braess’ paradox. The precise solving algorithm is set as a control group to test the accuracy of the following Q-learning-based approximate solving algorithm.

Let Method 1 be the random route selection from the alternative path set $\mathrm{E}$, Method 2 be the selfish VRG without considering the interaction between vehicles’ decisions, and Method 3 be the route game solved by the potential function-based precise algorithm.

Let the number of samples be 20. Running the built potential function-based precise algorithm under the traffic parameters in Section 4.1, the results are shown in

Table 2. The results of the precise solving algorithm.

	Method 1			Method 2			Method 3
	$\mathbf{\eta }$	$\mathbf{\tau }$	$\mathit{t}$	$\mathbf{\eta }$	$\mathbf{\tau }$	$\mathit{t}$	$\mathbf{\eta }$	$\mathbf{\tau }$	$\mathit{t}$
Scenario 1	–	0.99:1.00	43.47	–	0:1	52.00	1.00	2:3	41.76
Scenario 2	–	0.98:1.00	32.56	–	0:1	40.00	1.00	2:3	31.86
Scenario 3	–	1.01:1.00	36.97	–	0:1	40.00	1.00	3:7	35.44
Scenario 4	–	1.00:1.00	35.00	–	1:0	40.00	1.00	1:1	35.00

Note: $\eta$ denotes the averaged Nash equilibrium coefficient of 20 samples. $\tau$ denotes the averaged flow ratio of $\epsilon$ = 1 to $\epsilon$ = 2. t denotes the averaged travel time of 20 samples.

. Since solving Equation (5) in the control group is not the main task of this paper, the specific steps of this potential function-based precise method are not described in detail.

As shown in Table 2, the selfish VRG (Method 2) obtains a longer travel time compared with the random path selection (Method 1) in all traffic scenarios. This means that the road efficiency is reduced due to the occurrence of Braess’ paradox (a congestion switches to an alternative road since the same optimal route is recommended to numerous vehicles). For the route game-based coordinated VRG (Method 3), the precise algorithm generates the shortest travel time compared with Method 1 and 2 by considering users’ routes systematically. Braess’ paradox is mitigated. In other words, the feasibility of the route game is verified. It should be noted that the travel time of vehicles depends only on the flow ratio due to the symmetry of the payoff matrix.

4.3. Availability of the Q-Learning-Based Approximate Solving Algorithm

In today’s smart cities, the real-time nature of traffic management systems is a primary concern for engineers. However, the work of building the potential function for precise algorithm is a manual task, which cannot adapt to dynamic traffic conditions. Hence, compared with the hard-solved absolute optimality, a Nash equilibrium coefficient-based approximate solving method that can be solved effectively is designed in Section 3.2.

Let the number of samples be 20 (the initial route schemes of 20 samples are different), the and training times be $1\times {10}^4$, $1\times {10}^5$ and $5\times {10}^5$, respectively. Running the built Q-learning-based approximate solving algorithm (designed in Section 3.2) on traffic scenario 1, the results are shown in Figure 3 with 20 different color curves.

As shown in Figure 3a, when the training times are set to $1\times {10}^4$, the Nash equilibrium coefficients of all samples distribute from 0.30 to 1.00 due to the insufficient training, and the corresponding travel times do not converge to a unique value. It means that the generated coordination routes are not the Nash equilibrium. When the training times are increased to $1\times {10}^5$, the results (shown in Figure 3b) are similar to Figure 3a that the calculated coordination routes still do not receive the Nash equilibrium.

As shown in Figure 3c, with the training times increased to $5\times {10}^5$, the Nash equilibrium coefficients of samples all converge to 1.00, and the corresponding travel times converge to a unique value 41.76. The Nash equilibrium routes are received. Compared with the results in Table 2, it is found that the Q-learning-based approximate solving algorithm receives the same coordination routes as the potential function-based precise solving algorithm. The validity of the designed approximate solving algorithm is demonstrated in this section. It is worth noting that the operation time of Figure 3c is about 335 s in the MATLAB environment described in Section 4.1.

4.4. Robustness of the Q-Learning-Based Approximate Solving Algorithm

In the verification of the Q-learning-based approximate solving algorithm, only traffic scenario 1 is considered in Figure 3. To verify the universality of the Q-learning-based solving algorithm in various traffic scenarios, the convergence of the designed Q-learning needs to be further tested under different traffic parameters. The robustness of the designed approximate solving algorithm is shown in Figure 4.

As shown in Figure 4a, the Q-learning-based solving algorithm still exists required convergence in traffic scenarios 2, 3 and 4 (the Nash equilibrium coefficients coverage to 1.00). As shown in Figure 4b, the corresponding travel times converge to the unique values 31.86, 35.44, and 35.00, respectively. Compared with the results in Table 2, it is found that the Q-learning-based approximate solving algorithm receives the same route schemes as the potential function-based precise solution algorithm in traffic scenarios 2, 3, and 4. The robustness of the Q-learning-based approximate solving algorithm is verified.

4.5. Discussion

In this section, firstly, the potential function-based precise solving algorithm is used as the control group. The results show that Braess’ paradox is mitigated by the built route game. Then, the convergence of the Q-learning-based approximate solving algorithm is studied in the same traffic scenarios. The converged approximate route schemes show that the Q-learning-based approximate solving algorithm receives the same Nash equilibrium as the potential function-based precise solving algorithm. Finally, the robustness analysis shows that the Q-learning-based approximate solving algorithm still has the required convergence in different traffic parameters.

5. Conclusions and Future Works

The route game is recognized as an effective way to alleviate Braess’ paradox. In this paper, compared with the existing hard-solvable precise solving algorithms, a matched Q-learning-based approximate solving algorithm is designed to generate the coordination routes for the route game. In the Q-learning-based approximate solving algorithm, the concept of the approximate Nash equilibrium is introduced into the route game, which is more suitable for the real-time nature of smart traffic. An experimental study shows that the Nash equilibrium coefficients generated by the Q-learning-based approximate solving algorithm all converge to 1.00, and the obtained route schemes receive the same Nash equilibrium as the potential function-based precise solving algorithm. The robustness analysis shows that the Q-learning-based approximate solving algorithm still has the required convergence in different traffic parameters.

The built approximate Nash equilibrium-based route game is a decentralized approach that can be applied to large road networks based on IoVs. Meanwhile, there still exist some limitations to be further studied. For example, (1) the conventional route games have symmetry, since vehicles’ payoffs depend only on the selected route distribution. However, for a VRG model with personalized vehicles, the payoff matrix is not necessarily symmetric. The convergence of the proposed Q-learning-based approximate solving algorithm deserves further study. (2) The main purpose of this paper is to propose the basic Q-learning framework to prove the feasibility of approximate Nash equilibrium in route games. It is necessary to design a faster multi-agent-based Q-learning for bigger scale cities, that can be compared with the previous approximate algorithms such as heuristic.