Bifurcation Mechanism Design—From Optimal Flat Taxes to Better Cancer Treatments

Abstract

Small changes to the parameters of a system can lead to abrupt qualitative changes of its behavior, a phenomenon known as bifurcation. Such instabilities are typically considered problematic, however, we show that their power can be leveraged to design novel types of mechanisms. Hysteresis mechanisms use transient changes of system parameters to induce a permanent improvement to its performance via optimal equilibrium selection. Optimal control mechanisms induce convergence to states whose performance is better than even the best equilibrium. We apply these mechanisms in two different settings that illustrate the versatility of bifurcation mechanism design. In the first one we explore how introducing flat taxation could improve social welfare, despite decreasing agent “rationality,” by destabilizing inefficient equilibria. From there we move on to consider a well known game of tumor metabolism and use our approach to derive potential new cancer treatment strategies.

1. Introduction

The term bifurcation, which means splitting in two, is used to describe abrupt qualitative changes in system behavior due to smooth variation of its parameters. Bifurcations are ubiquitous and permeate all natural phenomena. Effectively, they produce discrete events (e.g., rain breaking out) out of smoothly varying, continuous systems (e.g., small changes to humidity or temperature). Typically, they are studied through bifurcation diagrams, multi-valued maps that prescribe how each parameter configuration translates to possible system behaviors (e.g., Figure 1).

Bifurcations arise in a natural way in game theory. Games are typically studied through their Nash correspondences, a multi-valued map connecting the parameters of the game (i.e., payoff matrices) to system behavior, in this case Nash equilibria. As we slowly vary the parameters of the game, typically the Nash equilibria will also vary smoothly, except at bifurcation points where, for example, the number of equilibria abruptly changes as some equilibria appear/disappear altogether. Such singularities may substantially impact both system behavior and system performance. For example, if the system state was at an equilibrium that disappeared during the bifurcation, then a turbulent transitionary period ensues where the system tries to reorganize itself at one of the remaining equilibria. Moreover, the quality of all remaining equilibria may be significantly worse than the original. Even more disturbingly, it is not a priori clear that the system will equilibrate at all. Successive bifurcations that lead to increasingly more complicated recurrent behavior is a standard route to chaos [1], which may have devastating effects on system performance.

Game theorists are particularly aware of the need to produce “robust" predictions, i.e., predictions that allow for deviations from an idealized exact specification of the parameters of the setting [2]. For example, $ϵ$-approximate Nash equilibria allow for the possibility of computational bounded agents, whereas $ϵ$-regret outcomes allow for persistently non-equilibrating behavior [3]. These approaches, however, do not really address the problem at its core as any solution concept defines a map from parameter space to behavioral space and no such map is immune to bifurcations. If pushed hard enough any system will destabilize. The question is what happens next?

Well, a lot of things may happen. It is intuitively clear that if we are allowed to play around arbitrarily with the payoffs of the agents then we can reproduce any game and no meaningful analysis is possible. Using payoff entries as controlling parameters is problematic for another reason. It is not clear that there exists a compelling parametrization of the payoff space that captures how real life decision-makers deviate from the Platonic ideal of the payoff matrix. Instead, we focus on another popular aspect of economic theory: agent “rationality”.

We adopt a standard model of boundedly rational learning agents. Boltzmann Q-learning dynamics [4,5,6] is a well studied behavioral model in which agents are parameterized by a temperature/rationality term T. Each agent keeps track of the collective past performance of his/her actions (i.e., learns from experience) and chooses an action according to a Boltzmann/Gibbs distribution with parameter T. When applied to a multi-agent game, the behavioral fixed points of Q-learning are known as quantal response equilibria (QREs) [7]. Naturally, QREs depend on the temperature T. As $T\to 0$ players become perfectly rational, and play approaches a Nash equilibrium,1 whereas as $T\to \infty$ all agents use uniformly random strategies. As we vary the temperature the QRE(T) correspondence moves between these two extremes producing bifurcations along the way at critical points where the number of QREs changes (Figure 1).

Our goal in this paper is to quantify the effects of these rationality-driven bifurcations to the social welfare of two-player two-strategy games. At this point, a moment of pause is warranted. Why is this a worthy goal? Games of small size ($2\times 2$ games in particular) are rarely seem like a subject worthy of serious scientific investigation. This, however, could not be further from the truth.

First, the correct way to interpret this setting is from the point of population games where each agent is better understood as a large homogeneous population (e.g., men and women, attackers and defenders, cells of phenotype A, and cells of phenotype B). Each of a handful of different types of users has only a few meaningful actions available to them. In fact, from the perspective of applied game theory, only such games with a small number of parameters are practically meaningful. The reason should be clear by now. Any game theoretic modeling of a real life scenario is invariably noisy and inaccurate. In order for game-theoretic predictions to be practically binding, they have to be robust to these uncertainties. If the system intrinsically has a large number of independent parameters, e.g., 20, then this parameter space will almost certainly encode a vast number of bifurcations, which invalidate any theoretical prediction. Practically useful models need to be small.

Secondly, game theoretic models applied for scientific purposes are often small. Specifically, the exact setting studied here with Boltzmann Q-learning dynamics applied in $2\times 2$ games has been used to model the effects of taxation to agent rationality [9] (see Section 6.2 for a more extensive discussion) as well as to model the effects of treatments that trigger phase transitions to cancer dynamics [10] (see Section 6.1). Our approach yields insights to explicit open questions in both of these applications areas. In fact, direct application of our analysis can address similar inquiries for any other phenomenon modeled by Q-learning dynamics applied in $2\times 2$ games.

Finally, the analysis itself is far from straightforward as it requires combining sets of tools and techniques that have so far been developed in isolation from each other. On one hand, we need to understand the behavior of these dynamical systems using tools from topology of dynamical systems, whose implications are largely qualitative (e.g., prove the lack of cyclic trajectories). On the other hand, we need to leverage these tools to quantify at which exact parameter values bifurcations occur and produce price-of-anarchy guarantees, which by definition are quantitative. As far as we know, this is the first instance of a fruitful combination of these tools. In fact, not only do we show how to analyze the effects of bifurcations to system efficiency, we also show how to leverage this understanding (e.g., knowledge of the geometry of the bifurcation diagrams) to design novel types of mechanisms with good performance guarantees.

Our Contribution

We introduce two different types of mechanisms: hysteresis and optimal control mechanisms.

Hysteresis mechanisms use transient changes to the system parameters to induce permanent improvements to its performance via optimal (Nash) equilibrium selection. The term hysteresis is derived from an ancient Greek word that means “to lag behind.” It reflects a time-based dependence between the system’s present output and its past inputs. For example, let’s assume that we start from a game theoretic system of Q-learning agents with temperature $T=0$ and assume that the system has converged to an equilibrium. By increasing the temperature beyond some critical threshold and then bringing it back to zero, we can force the system to provably converge to another equilibrium, e.g., the best (Nash) equilibrium (Figure 1, Theorem 4). Thus, we can ensure performance equivalent to that of the price of stability instead of the price of anarchy. One attractive feature of this mechanism is that from the perspective of the central designer it is relatively “cheap" to implement. Whereas typical mechanisms require the designer to continuously intervene (e.g., by paying the agents) to offset their greedy tendencies, this mechanism is transient with a finite amount of total effort from the perspective of the designer. Further, the idea that game theoretic systems have effectively systemic memory is rather interesting and could find other applications within algorithmic game theory.

Optimal control mechanisms induce convergence to states whose performance is better than even the best Nash equilibrium. Thus, we can at times even beat the price of stability (Theorem 5). Specifically, we show that by controlling the exploration/exploitation tradeoff, we can achieve strictly better states than those achievable by perfectly rational agents. In order to implement such a mechanism, it does not suffice to identify the right set of agents’ parameters/temperatures so that the system has some QRE whose social welfare is better than the best Nash. We need to design a trajectory through the parameter space so that this optimal QRE becomes the final resting point.

2. Preliminaries

2.1. Game Theory Basics: $2\times 2$ Games

In this paper, we focus on $2\times 2$ games. We define it as a game with two players, and each player has two actions. We write the payoff matrices of the game for each player as

$$\mathit{A}=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\mathit{B}=\left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right)$$

(1)

respectively. The entry $a_{ij}$ denotes the payoff for Player 1 when s/he chooses action i and his/her opponent chooses action j; similarly, $b_{ij}$ denotes the payoff for Player 2 when s/he chooses action i and his/her opponent chooses action j. We define x as the probability that the Player 1 chooses his/her first action, and y as the probability that Player 2 chooses his/her first action. We also define two row vectors $\mathit{x}={(x,1-x)}^T$ and $\mathit{y}={(y,1-y)}^T$ as the strategy for each player. For simplicity, we denote the i-th entry of vector $\mathit{x}$ by $x_i$. We call the tuple $(x,y)$ as the system state or the strategy profile.

An important solution concept in game theory is the Nash equilibrium, where each user cannot make profit by unilaterally changing his/her strategy, that is

Definition 1 (Nash equilibrium).

A strategy profile $(x_{NE},y_{NE})$ is a Nash equilibrium (NE) if

$$\begin{array}{cc}{x}_{NE}\in arg\underset{x\in [0,1]}{max}{\mathit{x}}^T\mathit{A}{\mathit{y}}_{NE}\hfill & \hfill y_{NE}\in arg\underset{y\in [0,1]}{max}{\mathit{y}}^T\mathit{B}{\mathit{x}}_{NE}.\end{array}$$

We call $(x_{NE},y_{NE})$ a pure Nash equilibrium (PNE) if both $x_{NE}\in \{0,1\}$ and $y_{NE}\in \{0,1\}$. A Nash equilibrium assumes each user is fully rational. An alternative solution concept is the quantal response equilibrium [7], where it assumes that each user has bounded rationality:

Definition 2 (Quantal response equilibrium).

A strategy profile $(x_{QRE},y_{QRE})$ is a QRE with respect to temperature $T_x$ and $T_y$ if

$$\begin{array}{ccc}\hfill x_{QRE}& =\frac{e^{\frac{1}{T_x}{\left(\mathit{A}{\mathit{y}}_{QRE}\right)}_1}}{{\sum }_{j\in \{1,2\}}{e}^{\frac{1}{T_x}{\left(\mathit{A}{\mathit{y}}_{QRE}\right)}_j}}\hfill & \hfill 1-x_{QRE}=\frac{e^{\frac{1}{T_x}{\left(\mathit{A}{\mathit{y}}_{QRE}\right)}_2}}{{\sum }_{j\in \{1,2\}}{e}^{\frac{1}{T_x}{\left(\mathit{A}{\mathit{y}}_{QRE}\right)}_j}}\\ \hfill y_{QRE}& =\frac{e^{\frac{1}{T_y}{\left(\mathit{B}{\mathit{x}}_{QRE}\right)}_1}}{{\sum }_{j\in \{1,2\}}{e}^{\frac{1}{T_y}{\left(\mathit{B}{\mathit{x}}_{QRE}\right)}_j}}\hfill & \hfill 1-y_{QRE}=\frac{e^{\frac{1}{T_y}{\left(\mathit{B}{\mathit{x}}_{QRE}\right)}_2}}{{\sum }_{j\in \{1,2\}}{e}^{\frac{1}{T_y}{\left(\mathit{B}{\mathit{x}}_{QRE}\right)}_j}}.\end{array}$$

Analogous to the definition of Nash equilibria, we can consider the QREs as the case where each player is not only maximizing the expected utility but also maximizing the entropy. We can see that the QREs are the solutions to maximizing the linear combination of the following program:

$$\begin{array}{cc}\hfill {\mathit{x}}_{QRE}& \in arg\underset{\mathit{x}}{max}\left\{{\mathit{x}}^T\mathit{A}{\mathit{y}}_{QRE}-T_x\sum _j{x}_{j}ln{x}_j\right\}\hfill \\ \hfill {\mathit{y}}_{QRE}& \in arg\underset{\mathit{y}}{max}\left\{{\mathit{y}}^T\mathit{B}{\mathit{x}}_{QRE}-T_y\sum _j{y}_{j}ln{y}_j\right\}.\hfill \end{array}$$

This formulation has been widely seen in Q-learning dynamics literature (e.g., [9,11,12]). With this formulation, we can find that the two parameters $T_x$ and $T_y$ control the weighting between the utility and the entropy. We call $T_x$ and $T_y$ the temperatures, and their values define the level of irrationality. If $T_x$ and $T_y$ are zero, then both players are fully rational, and the system state is a Nash equilibrium. However, if both $T_x$ and $T_y$ are infinity, then each player is choosing his/her action according to a uniform distribution, which corresponds to the fully irrational players.

2.2. Efficiency of an Equilibrium

The performance of a system state can be measured via the social welfare. Given a system state $(x,y)$, we define the social welfare as the sum of the expected payoff of all users in the system:

Definition 3.

Given a $2\times 2$ game with payoff matrices $\mathit{A}$ and $\mathit{B}$, and a system state $(x,y)$, the social welfare is defined as

$$SW(x,y)=xy(a_{11}+b_{11})+x(1-y)(a_{12}+b_{21})+y(1-x)(a_{21}+b_{12})+(1-x)(1-y)(a_{22}+b_{22}).$$

In the context of algorithmic game theory, we can measure the efficiency of a game by comparing the best social welfare with the social welfare of equilibrium system states. We call the strategy profile that achieves the maximal social welfare as the socially optimal (SO) strategy profile. The efficiency of a game is often described as the notion of the price of anarchy (PoA) and the price of stability (PoS). Given a set of equilibrium states S, we define the PoA/PoS as the ratio of the social welfare of the socially optimal state to the social welfare of the worst/best equilibrium state in S, respectively. Formally,

Definition 4.

Given a $2\times 2$ game with payoff matrices A and B, and a set of equilibrium system states $S\subseteq {[0,1]}^2$, the price of anarchy (PoA) and the price of stability (PoS) are defined as

$$\begin{array}{cc}PoA\left(S\right)=\frac{{max}_{(x,y)\in {[0,1]}^2}SW(x,y)}{{min}_{(x,y)\in S}SW(x,y)}\hfill & \hfill PoS\left(S\right)=\frac{{max}_{(x,y)\in {[0,1]}^2}SW(x,y)}{{max}_{(x,y)\in S}SW(x,y)}.\end{array}$$

3. Our Model

3.1. Q-Learning Dynamics

In this paper, we are particularly interested in the scenario when both players’ strategies are evolving under Q-learning dynamics:

$$\begin{array}{cc}{\dot{x}}_i=x_i\left[{\left(\mathit{A}\mathit{y}\right)}_i-{\mathit{x}}^T\mathit{A}\mathit{y}+T_x\sum _j{x}_{j}ln(x_j/x_i)\right]\hfill & \hfill {\dot{y}}_i=y_i\left[{\left(\mathit{B}\mathit{x}\right)}_i-{\mathit{y}}^T\mathit{B}\mathit{x}+T_y\sum _j{y}_{j}ln(y_j/y_i)\right].\end{array}$$

(2)

Q-learning dynamics has been studied because of its connection with multi-agent learning problems. For example, it has been shown in [13,14] that Q-learning dynamics captures the system evolution of a repeated game, where each player learns his/her strategy through Q-learning and Boltzmann selection rules. More details are provided in Appendix A.

An important observation on the dynamics of Equation (2) is that it demonstrates the exploration/exploitation tradeoff [14]. We can find that the right hand side of Equation (2) is composed of two parts. The first part $x_i[{\left(\mathit{A}\mathit{y}\right)}_i-{\mathit{x}}^T\mathit{A}\mathit{y}]$ is exactly the vector field of replicator dynamic [15]. Basically, the replicator dynamics drives the system to the state of higher utility for both players. As a result, we can consider this as a selection process in terms of population evolutionary, or an exploitation process from the perspective of a learning agent. Then, for the second part, $x_i[T_x{\sum }_j{x}_{j}ln(x_j/x_i)],$ we show in the appendix that if the time derivative of $\mathit{x}$ contains this part alone, this results in an increase of the system entropy.

The system entropy is a function that captures the randomness of the system. From the population evolutionary perspective, the system entropy corresponds to the variety of the population. As a result, this term can be considered as the mutation process. The level of the mutation is controlled by the temperature parameters $T_x$ and $T_y$. Besides, in terms of the reinforcement learning, this term can be considered as an exploration process, as it provides the opportunity for the agent to gain information about the action that does not look the best so far.

3.2. Convergence of the Q-Learning Dynamics

By observing the Q-learning dynamics of Equation (2), we can find that the interior rest points for the dynamics are exactly the QREs of the $2\times 2$ game. It is claimed in [16] (albeit without proof) that the Q-learning dynamics for a $2\times 2$ game converges to interior rest points of probability simplexes for any positive temperature $T_x>0$ and $T_y>0$. We provide a formal proof in Appendix B. The idea is that, for positive temperatures, the system is dissipative and, by leveraging the planar nature of the system, it can be argued that it converges to fixed points.

3.3. Rescaling the Payoff Matrix

At the end of this section, we discuss the transformation of the payoff matrices that preserves the dynamics in Equation (2). This idea is proposed in [17,18], where the rescaling of a matrix is defined as follows

Definition 5

([18]). ${\mathit{A}}^{\prime }$ and ${\mathit{B}}^{\prime }$ is said to be a rescaling of $\mathit{A}$ and $\mathit{B}$ if there exist constants $c_j,d_i$, and $\alpha >0$, $\beta >0$ such that $a_{ij}^{\prime }=\alpha a_{ij}+c_j$ and $b_{ji}^{\prime }=\beta b_{ji}+d_i$.

It is clear that rescaling the game payoff matrices is equivalent to updating the temperature parameters of the two agents in Equation (2). Therefore, it suffices to study the dynamics under the assumption that the $2\times 2$ payoff matrices $\mathit{A}$ and $\mathit{B}$ are in the following diagonal form.

Definition 6.

Given $2\times 2$ matrices $\mathit{A}$ and $\mathit{B}$, their diagonal form is defined as

$${\mathit{A}}_D=\left(\begin{array}{cc}{a}_{11}-a_{21}& 0\\ 0& a_{22}-a_{12}\end{array}\right){\mathit{B}}_D=\left(\begin{array}{cc}{b}_{11}-b_{21}& 0\\ 0& b_{22}-b_{12}\end{array}\right)$$

Note that, although rescaling the payoff matrices to their diagonal form preserves the equilibria, it does not preserve the social optimality, i.e., the socially optimal strategy profile in the transformed game is not necessarily the socially optimal strategy profile in the original game.

4. Hysteresis Effect and Bifurcation Analysis

4.1. Hysteresis effect in Q-Learning Dynamics: An Example

We begin our discussion with an example:

Example 1 (Hysteresis effect).

Consider a $2\times 2$ game with reward matrices

$$A=\left(\begin{array}{cc}10& 0\\ 0& 5\end{array}\right)B=\left(\begin{array}{cc}2& 0\\ 0& 4\end{array}\right)$$

(3)

There are two PNEs in this game: $(x,y)=(0,0)$ and $(1,1)$. By fixing different $T_y$, we can plot different QREs with respect to $T_x$ as in Figure 2 and Figure 3, which we call the bifurcation diagrams. For simplicity, we only show the value of x in the figure, since, according to Equation (4), given x and $T_y$, the value of y can be uniquely determined. Assuming the system follows the Q-learning dynamics, as we slowly vary $T_x$, x tends to stay on the line segment that is the closest to where it was originally corresponding to a stable but inefficient fixed point. We consider the following process:

1. 

Where the initial state is $(0.05,0.14)$, where $T_x\approx 1$ and $T_y\approx 2$, plot x versus $T_x$ by fixing $T_y=2$ in Figure 3.

2. 

Fix $T_y=2$ and increase $T_x$ to where there is only one QRE correspondence.

3. 

Fix $T_y=2$ and decrease $T_x$ back to 1. Now $x\approx 0.997$.

In the above example, we can find that, although at the end the temperature parameters are set back to their initial value, the system state ends up being an entirely different equilibrium. This behavior is known as the hysteresis effect. In this section, we would like to answer the question of when this is going to happen. Further, in the next section, we will show how can we take advantage of this phenomenon.

4.2. Characterizing QREs

We consider the bifurcation diagrams for QREs in $2\times 2$ games. Without loss of generality, we consider a properly rescaled $2\times 2$ game with payoff matrices in the diagonal form:

$${\mathit{A}}_D=\left(\begin{array}{cc}{a}_X& 0\\ 0& b_X\end{array}\right),{\mathit{B}}_D=\left(\begin{array}{cc}{a}_Y& 0\\ 0& b_Y\end{array}\right)$$

We can also assume that the action indices are ordered properly and rescaled properly so that $a_X>0$ and $|a_X|\ge |b_X|$. For simplicity, we assume $a_X=b_X$ and $b_X=b_Y$ do not hold at the same time. At QRE, we have

$$\begin{array}{cc}x={\displaystyle \frac{e^{{\displaystyle \frac{1}{T_x}}y{a}_X}}{e^{{\displaystyle \frac{1}{T_x}}y{a}_X}+e^{{\displaystyle \frac{1}{T_x}}(1-y)b_X}}}\hfill & \hfill y={\displaystyle \frac{e^{{\displaystyle \frac{1}{T_y}}x{a}_Y}}{e^{{\displaystyle \frac{1}{T_y}}x{a}_Y}+e^{{\displaystyle \frac{1}{T_y}}(1-x)b_Y}}}.\end{array}$$

(4)

Given $T_x$ and $T_y$, there could be multiple solutions to Equation (4). However, we find that, if we know the equilibrium states, then we can recover the temperature parameters. We solve for $T_x$ and $T_y$ in Equation (4) and get

$$\begin{array}{cc}{T}_X^I(x,y)=\frac{-(a_X+b_X)y+b_X}{ln(\frac{1}{x}-1)}\hfill & \hfill T_Y^I(x,y)=\frac{-(a_Y+b_Y)x+b_Y}{ln(\frac{1}{y}-1)}.\end{array}$$

(5)

We call this the first form of representation, where $T_x$ and $T_y$ are written as functions of x and y. Here the capital subscripts for $T_X$ and $T_Y$ indicate that they are considered as functions. A direct observation of Equation (5) is that both of them are continuous function over $(0,1)\times (0,1)$ except for $x=1/2$ and $y=1/2$.

An alternative way to describe the QRE is to write $T_x$ and y as a function of x and parameterize with respect to $T_y$ in the following second form of representation. This will be the form that we use to prove many useful characteristics of QREs.

$$\begin{array}{cc}\hfill T_X^{II}(x,T_y)& =\frac{-(a_X+b_X)y^{II}(x,T_y)+b_X}{ln(\frac{1}{x}-1)}\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill y^{II}(x,T_y)& ={\left(1+e^{\frac{1}{T_y}(-(a_Y+b_Y)x+b_Y)}\right)}^{-1}.\hfill \end{array}$$

(7)

In this way, if we are given $T_y$, we are able to analyze how $T_x$ changes with x. This helps us understand how to answer the question of what the QREs are given $T_x$ and $T_x$ in the system.

We also want to analyze the stability of the QREs. From dynamical system theory (e.g., [19]), a fixed point of a dynamical system is said to be asymptotically stable if all of the eigenvalues of its Jacobian matrix have a negative real part; if it has at least one eigenvalue with a positive real part, then it is unstable. It turns out that, under the second form representation, we are able to determine whether a segment in the diagram is stable or not.

Lemma 1.

Given $T_y$, the system state $\left(x,y^{II}(x,T_y)\right)$ is a stable equilibrium if and only if

1. 

$\frac{\partial T_X^{II}}{\partial x}(x,T_Y)>0$ if $x\in (0,1/2)$;

2. 

$\frac{\partial T_X^{II}}{\partial x}(x,T_Y)<0$ if $x\in (1/2,1)$.

Proof. 

The given condition is equivalent to the case where both eigenvalues of the Jacobian matrix of the dynamics (2) are negative. ☐

Finally, we define the principal branch. In Example 1, we call the branch on $x\in (0.5,1)$ the principal branch given $T_y=2$, since, for any $T_x>0$, there is some $x\in (0.5,1)$ such that $T_X^{II}(x,T_y)=T_x$. Analogously, we can define it formally as in the following definition with the help of the second form representation.

Definition 7.

Given $T_y$, the region $(a,b)\subset (0,1)$ contains the principal branch of QRE correspondence if it satisfies the following conditions:

1. 

$T_X^{II}(x,T_y)$ is continuous and differentiable for $x\in (a,b)$.

2. 

$T_X^{II}(x,T_y)>0$ for $x\in (a,b)$.

3. 

For any $T_x>0$, there exists $x\in (a,b)$ such that $T_X^{II}(x,T_y)=T_x$.

Further, for a region $(a,b)$ that contains the principal branch, $x\in (a,b)$ is on the principal branch if it satisfies the following conditions:

1. 

The equilibrium state $(x,y^{II}(x,T_y))$ is stable.

2. 

There is no $x^{\prime }\in (a,b),x^{\prime }<x$ such that $T_X^{II}(x^{\prime },T_y)=T_X^{II}(x,T_y)$.

4.3. Coordination Games

We begin our analysis with the class of coordination games, where we have all $a_X$, $b_X$, $a_Y$, and $b_Y$ positive. Additionally, without loss of generality, we assume $a_X\ge b_X$. In this case, there is no dominant strategy for either player, and there are two PNEs.

Let us revisit Example 1, we can make the following observations from Figure 2 and Figure 3:

• Given $T_y$, there are three branches. One is the principal branch, while the other two appear in pairs and occur only when $T_x$ is less than some value.
• For small $T_y$, the principal branch goes toward $x=0$; for a large $T_y$, the principal branch goes toward $x=1$.

Now, we are going to show that these observations are generally true in coordination games. The proofs in this section are deferred to Appendix D, where we will provide a detailed discussion on the proving techniques.

The first idea we are going to introduce is the inverting temperature, which is the threshold of $T_y$ in Observation (2). We define it as

$$T_I=max\left\{0,\frac{b_Y-a_Y}{2ln(a_X/b_X)}\right\}.$$

We note that $T_I$ is positive only if $b_Y>a_Y$, which is the case where two players have different preferences. When $T_y<T_I$, as the first player increases his/her rationality from fully irrational, i.e., $T_x$ decreases from infinity, s/he is likely to be influenced by the second player’s preference. If $T_y$ is greater than $T_I$, then the first player prefers to follow his/her own preference, making the principal branch move toward $x=1$. We formalize this idea in the following theorem:

Theorem 1 (Direction of the principal branch).

Given a $2\times 2$ coordination game, and given $T_y$, the following statements are true:

1. 

If $T_y>T_I$, then $(0.5,1)$ contains the principal branch.

2. 

If $T_y<T_I$, then $(0,0.5)$ contains the principal branch.

The second idea is the critical temperature, denoted as $T_C\left(T_y\right)$, which is a function of $T_y$. The critical temperature is defined as the infimum of $T_x$ such that, for any $T_x>T_C\left(T_y\right)$, there is a unique QRE correspondence under $(T_x,T_y)$. Generally, there is no close form for the critical temperature. However, we can still compute it efficiently, as we show in Theorem 2. Another interesting value of $T_y$ we should point out is $T_B=\frac{b_Y}{ln(a_X/b_X)}$, which is the maximum value of $T_y$ that QREs not on the principal branch are presenting. Intuitively, as $T_y$ goes beyond $T_B$, the first player ignores the decision of the second player and turns his/her face to what s/he thinks is better. We formalize the idea of $T_C$ and $T_B$ in the following theorem:

Theorem 2 (Properties about the second QRE).

Given a $2\times 2$ coordination game, and given $T_y$, the following statements are true:

1. 

For almost every $T_x>0$, all QREs not lying on the principal branch appear in pairs.

2. 

If $T_y>T_B$, then there is no QRE correspondence in $x\in (0,0.5)$.

3. 

If $T_y>T_I$, then there is no QRE correspondence for $T_x>T_C\left(T_y\right)$ in $x\in (0,0.5)$.

4. 

If $T_y<T_I$, then there is no QRE correspondence for $T_x>T_C\left(T_y\right)$ in $x\in (0.5,1)$.

5. 

$T_C\left(T_y\right)$ is given as $T_X^{II}(x_L,T_y)$, where $x_L$ is the solution to the equality

$$y^{II}(x,T_y)+x(1-x)ln\left(\frac{1}{x}-1\right)\frac{\partial y^{II}}{\partial x}(x,T_y)=\frac{b_X}{a_X+b_X}.$$

6. 

$x_L$ can be found using binary search.

The next aspect of the QRE correspondence is their stability. According to Lemma 1, the stability of the QREs can also be inspected with the advantage of the second form representation by analyzing $\frac{\partial T_X^{II}}{\partial x}$. We state the results in the following theorem:

Theorem 3 (Stability).

Given a $2\times 2$ coordination game, and given $T_y$, the following statements are true:

1. 

If $a_Y\ge b_Y$, then the principal branch is continuous.

2. 

If $T_y<T_I$, then the principal branch is continuous.

3. 

If $T_y>T_I$ and $a_Y<b_Y$, then the principal branch may not be continuous.

4. 

If $T_x$ is fixed, for the pairs of QREs not lying on the principal branch, the one with the lowest distance to $x=0.5$ is unstable, while the other one is stable.

Note that Part 3 in Theorem 3 infers that there is potentially an unstable segment between segments of the principal branch. This phenomenon is illustrated in Figure 4 and Figure 5. Though this case is weaker than other cases, this does not hinder us from designing a controlling mechanism as we are going to do in Section 5.3.

4.4. Non-Coordination Games

Due to space constraints, the analysis for non-coordination games is deferred to Appendix C.

5. Mechanism Design

In this section, we aim to design a systematic way to improve the social welfare in a $2\times 2$ game by changing the temperature parameters. We focus our discussion on the class of coordination games. Recall that any $2\times 2$ game has more than one PNE if and only if its diagonal form is a coordination game. This means that, in a coordination game, given any temperature parameters, there could be more than one equilibrium correspondences. In this case, we are not guaranteed to achieve the socially optimal equilibrium state even if we set the system to the correct temperatures due to the hysteresis effects that we have discussed in the previous section. Therefore, the main task for us in this section is to determine when and how we can get to the socially optimal equilibrium state. In Section 5.3, we consider the case when the socially optimal state is one of the PNEs. Since rescaling the payoff matrices to their diagonal form does not preserve the social optimality, in Section 5.1, we generalize our discussion to the case when the social optimal state does not coincide with any PNE.

5.1. Hysteresis Mechanism: Select the Best Nash Equilibrium via QRE Dynamics

First, we consider the case when the socially optimal state is one of the PNEs. The main task for us in this case is to determine when and how we can get to the socially optimal PNE. In Example 1, by sequentially changing $T_x$, we move the equilibrium state from around $(0,0)$ to around $(1,1)$, which is the social optimum state. We formalize this idea as the hysteresis mechanism and present it in Theorem 4. The hysteresis mechanism mainly takes advantage of the hysteresis effect we have discussed in Section 4—that we use transient changes of system parameters to induce permanent improvements to system performance via optimal equilibrium selection.

Theorem 4 (Hysteresis Mechanism).

Consider a $2\times 2$ game that satisfies the following properties:

1. 

Its diagonal form satisfies $a_X,b_X,a_Y,b_Y>0$.

2. 

Exactly one of its pure Nash equilibrium is the socially optimal state.

Without loss of generality, we can assume $a_X\ge b_X$. Then there is a mechanism to control the system to the social optimum by sequentially changing $T_x$ and $T_y$ if (1) $a_Y\ge b_Y$ and (2) the socially optimal state is $(0,0)$ do not hold at the same time.

Proof. 

First, note that, if $a_Y\ge b_Y$, by Theorem 1, the principal branch is always in the region $x>0.5$. As a result, once $T_y$ is increased beyond the critical temperature, the system state will no longer return to $x<0.5$ at any positive temperature. Therefore, $(0,0)$ cannot be approached from any state in $x>0.5$ through the QRE dynamics.

On the other hand, if $a_Y\ge b_Y$ and the socially optimal state is the PNE $(1,1)$, then we can approach that state by first getting onto the principal branch. The mechanism can be described as

(C1)	(a)	Raise $T_x$ to some value above the critical temperature $T_C\left(T_y\right)$.
	(b)	Reduce $T_x$ and $T_y$ to 0.

Though in this case the initial choice of $T_y$ does not affect the result, if the social designer is taking the costs from assigning large $T_x$ and $T_y$ values into account, s/he is going to trade off between $T_C$ and $T_y$ since a typically smaller $T_y$ induces a larger $T_C$.

Next, consider $a_Y<b_Y$. If we are aiming for state $(0,0)$, then we can undergo the following procedure:

(D1)	(a)	Keep $T_y$ at some value below $T_I=\frac{b_Y-a_Y}{2ln(a_X/b_X)}$. Now the principal branch is at $(0,0.5)$.
	(b)	Raise $T_x$ to some value above the critical temperature $T_C\left(T_y\right)$.
	(c)	Reduce $T_x$ to 0.
	(d)	Reduce $T_y$ to 0.

On the other hand, if we are aiming for state $(1,1)$, then the following procedure suffices:

(D2)	(a)	Keep $T_y$ at some value above $T_I=\frac{b_Y-a_Y}{2ln(a_X/b_X)}$. Now the principal branch is at $(0.5,1)$.
	(b)	Raise $T_x$ to some value above the critical temperature $T_C\left(T_y\right)$.
	(c)	Reduce $T_x$ to 0.
	(d)	Reduce $T_y$ to 0.

Note that, in the last two steps, only by reducing $T_y$ after $T_x$ keeps the state around $x=1$. We recommend that the interested reader refers to Figure 11 for Case (D1) and Figure 12 for Case (D2) for more insights. ☐

5.2. Efficiency of QREs: An Example

A question that arises with the solution concept of QRE is whether QRE improves social welfare? Here we show that the answer is yes. We begin with an example to illustrate:

Example 2.

Consider a standard coordination game with the payoff matrices of the form

$$A=\left(\begin{array}{cc}ϵ& 1\\ 0& 1+{ϵ}^{\prime }\end{array}\right)B=\left(\begin{array}{cc}1+ϵ& 0\\ 1& {ϵ}^{\prime }\end{array}\right)$$

(8)

where $ϵ>{ϵ}^{\prime }>0$ are some small numbers. Note that, in this game, there are two PNEs, $(x,y)=(1,1)$ and $(x,y)=(0,0)$, with social welfare values $1+2ϵ$ and $1+2{ϵ}^{\prime }$, respectively. We can see that for small ϵ and ${ϵ}^{\prime }$ values, the socially optimal state is $(x,y)=(1,0)$, with social welfare value 2. In this case, the state $(x,y)=(1,1)$ is the PNE with the best social welfare. However, we are able to achieve a state with better social welfare than any NE through QRE dynamics. We illustrate the social welfare of the QREs with different temperatures of this example in Figure 6. In this figure, we can see that, at PNE, which is the point $T_x=T_y=0$, the social welfare is $1+2ϵ$. However, we are able to increase the social welfare by increasing $T_y$. We will show in Section 5.3 a general algorithm for finding particular temperature as well as a mechanism, which we refer to as the optimal control mechanism, that drives the system to the desired state.

5.3. Optimal Control Mechanism: Better Equilibrium with Irrationality

Here, we show a general approach to improve the PoS bound for coordination games from Nash equilibria by QREs and Q-learning dynamics. We denote $QRE(T_x,T_y)$ as the set of QREs with respect to $T_x$ and $T_y$. Further, denote $QRE$ as the set of the union of $QRE(T_x,T_y)$ over all positive $T_x$ and $T_y$. Additionally, denote the set of pure Nash equilibria system states as $NE$. Since the set $NE$ is the limit of $QRE(T_x,T_y)$ as $T_x$ and $T_y$ approach zero, we have the bounds:

$$PoA\left(QRE\right)\ge PoA\left(NE\right),PoS\left(QRE\right)\le PoS\left(NE\right).$$

Then, we define QRE-achievable states:

Definition 8.

A state $(x,y)\in {[0,1]}^2$ is a QRE-achievable state if for every $ϵ>0$, there is a positive finite $T_x$ and $T_y$ and $(x^{\prime },y^{\prime })$ such that $|(x^{\prime },y^{\prime })-(x,y)|<ϵ$ and $(x^{\prime },y^{\prime })\in QRE(T_x,T_y)$.

Note that, with this definition, pure Nash equilibria are QRE-achievable states. However, the socially optimal states are not necessarily QRE-achievable. For example, we illustrate in Figure 7 the set of QRE-achievable states for Example 2. We can find that the socially optimal state, $(x,y)=(1,0)$, is not QRE-achievable. Nevertheless, it is easy to see from Figure 7 and Figure 8 that we can achieve a higher social welfare at $(x,y)=(1,0.5)$, which is a QRE-achievable state. Formally, we can describe the set of QRE-achievable states as the positive support of $T_X^I$ and $T_Y^I$:

$$\begin{array}{cc}\hfill S=& \left\{\left\{x\in \left[\frac{1}{2},1\right],y\in \left[\frac{b_X}{a_X+b_X},1\right]\right\}\cup \left\{x\in \left[0,\frac{1}{2}\right],y\in \left[0,\frac{b_X}{a_X+b_X}\right]\right\}\right\}\hfill \\ & \cap \left\{\left\{x\in \left[\frac{b_Y}{a_Y+b_Y},1\right],y\in \left[\frac{1}{2},1\right]\right\}\cup \left\{x\in \left[0,\frac{b_Y}{a_Y+b_Y}\right],y\in \left[0,\frac{1}{2}\right]\right\}\right\}.\hfill \end{array}$$

An example for the region of a game with $a_Y\ge b_Y$ is illustrated in Figure 7. For the case $a_Y<b_Y$, we demonstrate it in Figure 9.

In the following theorem, we propose the optimal control mechanism for a general process to achieve an equilibrium that is better than the PoS bound from Nash equilibria.

Theorem 5 (Optimal Control Mechanism).

Given a $2\times 2$ game, if it satisfies the following property:

1. 

Its diagonal form satisfies $a_X,b_X,a_Y,b_Y>0$.

2. 

None of its pure Nash equilibrium is the socially optimal state.

Without loss of generality, we can assume $a_X\ge b_X$. Then

1. 

there is a stable QRE-achievable state whose social welfare is better than any Nash equilibrium;

2. 

there is a mechanism to control the system to this state from the best Nash equilibrium by sequentially changing $T_x$ and $T_y$.

Proof. 

Note that, given those properties, there are two PNEs $(0,0)$ and $(1,1)$. Since we know neither of them is socially optimal, the socially optimal state must be either $(0,1)$ or $(1,0)$.

First, consider $a_Y\ge b_Y$. In this case, we know from Theorem 3 that all $x\in (0.5,1)$ states belong to a principal branch for some $T_y>0$ and are stable, while for $x<0.5$ not all of them are stable. We illustrate the region of stable QRE-achievable states in Figure 10. By Theorems 2 and 3, we can infer that the states near the border $x=0$ are stable. As a result, we can claim that the following states are what we are aiming for:

(A1)

If $(1,1)$ is the best NE and $(0,1)$ is the SO state, then we select $(0.5,1)$.

(A2)

If $(1,1)$ is the best NE and $(1,0)$ is the SO state, then we select $(1,0.5)$.

(A3)

If $(0,0)$ is the best NE and $(0,1)$ is the SO state, then we select $\left(0,\frac{b_X}{a_X+b_X}\right)$.

(A4)

If $(0,0)$ is the best NE and $(1,0)$ is the SO state, then we select $\left(\frac{b_Y}{a_Y+b_Y},0\right)$.

It is clear that these choices of states improve the social welfare. It is known that for the class of games we are considering, the price of stability is no greater than 2. In fact, in Cases A1 and A2, we reduce this factor to $4/3$. Additionally, in Cases A3 and A4, we reduce this factor to ${\left(\frac{1}{2}+\frac{b_X/2}{a_X+b_X}\right)}^{-1}$.

The next step is to show the mechanism to drive the system to the desired state. Due to symmetry, we only discuss Cases A1 and A3, where Cases A2 and A4 can be done analogously. For Case A1, the state corresponds to the temperatures $T_x\to \infty$ and $T_y\to 0$. For any small $\delta >0$, we can always find the state $(0.5+\delta ,1-\delta )$ on the principal branch of some $T_y$. This means that we can achieve this state from any initial state, not only from the NEs. With the help of the first form representation of the QREs in Equation (5), given any QRE-achievable system state $(x,y)$, we are able to recover them to corresponding temperatures through $T_X^I$ and $T_Y^I$. The mechanism can be described as follows:

(A1)	(a)	From any initial state, raise $T_x$ to $T_X^I(0.5+\delta ,1-\delta )$.
	(b)	Decrease $T_y$ to $T_Y^I(0.5+\delta ,1-\delta )$.

For Case A3, the state we selected is not on the principal branch. This means that we cannot increase the temperatures too much; otherwise, the system state will move to the principal branch and will never return. We assume initially the system state is at $(\delta ,\delta )$ for some small $\delta >0$, which is some state close to the best NE. Additionally, we can assume the initial temperatures are $T_x=T_X^I(\delta ,\delta )$ and $T_y=T_Y^I(\delta ,\delta )$. Our goal is to arrive at the state $\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$ for some small ${\delta }_1>0$ and ${\delta }_2>0$ such that $\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$ is stable. We present the mechanism in the following:

(A3)	(a)	From the initial state $(\delta ,\delta )$, move $T_x$ to $T_X^I\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$.
	(b)	Increase $T_y$ to $T_Y^I\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$.

Here, note that Step (b) should not proceed before Step (a) because, if we increase $T_y$ first, then we risk leaving the principal branch.

Next, consider the case where $a_Y<b_Y$. Similarly to the previous case, we know from Theorems 2 and 3 that states near the borders $x=0,0.5,1$ and $y=0,0.5,1$ are basically stable states. Hence, we can claim the following results:

(B1)

If $(1,1)$ is the best NE and $(0,1)$ is the SO state, then we select $\left(\frac{b_Y}{a_Y+b_Y},1\right)$.

(B2)

If $(1,1)$ is the best NE and $(1,0)$ is the SO state, then we select $(1,0.5)$.

(B3)

If $(0,0)$ is the best NE and $(0,1)$ is the SO state, then we select $\left(0,\frac{b_X}{a_X+b_X}\right)$.

(B4)

If $(0,0)$ is the best NE and $(1,0)$ is the SO state, then we select $\left(0.5,0\right)$.

It is clear that these choices of states create improvement on the social welfare. An interesting result for this case is that basically these desired states can be reached from any initial state. Due to symmetry, we demonstrate the mechanisms for Cases (B3) and (B4), and the remaining ones can be done analogously.

For Case (B3), we are aiming for the state $\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$ for some small ${\delta }_1>0$ and ${\delta }_2>0$. We propose the following mechanism:

(B3)	Phase 1: Getting to the principal branch. (a) From any initial state, fix $T_y$ at some value less than $T_I=\frac{b_Y-a_Y}{2ln(a_X/b_X)}$. (b) Increase $T_x$ above the critical temperature $T_C\left(T_y\right)$. (c) Decrease $T_x$ to $T_x^I\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$.
	Phase 2: Staying at the current branch. (a) Increase $T_y$ to $T_Y^I\left({\delta }_1,\frac{b_X}{a_X+b_X}-{\delta }_2\right)$.

This process is illustrated in Figure 11 and Figure 12. In Phase 1, as we are keeping low $T_y$, meaning the second player is of more rationality. As the first player getting more rational, s/he is more likely to be influenced by the second player’s preference, and eventually getting to a Nash equilibrium. In phase 2, we make the second player more irrational to increase the social welfare. The level of irrationality we add in phase 2 should be capped to prevent the first player to deviate his/her decision.

For Case (B4), since our desired state is on the principal branch, the mechanism will be similar to Case (A1).

(B4)	(a)	From any initial state, raise $T_x$ to $T_X^I(0.5+\delta ,\delta )$.
	(b)	Decrease $T_y$ to $T_Y^I(0.5+\delta ,\delta )$.

 ☐

As a remark, in Cases (A3) and (A4), if we do not start from $(\delta ,\delta )$ but from some other states on the principal branch, we can instead aim for state $(0.5,1)$. This state is not better than the best Nash equilibrium, but still makes improvements over the initial state. The process can be modified as

(A3’)	(a)	From any initial state, raise $T_x$ to $T_X^I(0.5+\delta ,1-\delta )$ (above $T_C\left(T_y\right)$).
	(b)	Reduce $T_y$ to $T_Y^I(0.5+\delta ,1-\delta )$.

6. Applications

6.1. Evolution of Metabolic Phenotypes in Cancer

Evolutionary game theory (EGT) has been instrumental in studying evolutionary aspects of the somatic evolution that characterizes cancer progression. As opposed to conventional game theory, in evolutionary game theory, the strategies are fixed for the player and constitute its phenotype. Tumors are very heterogeneous, and frequency-dependent selection is a driving force in somatic evolution. While evolutionary outcomes can change depending on initial conditions or on the exact features and microenvironment of the relevant tumor phenotypes, evolutionary game theory can explain why certain clonal populations, usually the more aggressive and faster proliferating ones, emerge and overtake the previous ones. Tomlinson and Bodmer were the first to explore the role of cell–cell interactions in cancer using EGT [20]. This pioneering work was followed by others that built on those initial ideas to study the role of key aspects of cancer evolution, such as the role of space [21] treatment [22,23] or metabolism [10,24].

Work by Kianercy and colleagues [10] shows how microenvironmental heterogeneity impacts somatic evolution. Kianercy and colleagues show how the tumor’s genetic instability adapts to the heterogeneous microenvironment (with regard to oxygen concentration) to better tune metabolism to the dynamic microenvironment. While evolutionary dynamics can help a tumor population evolve to acquire all relevant mutations to become an aggressive cancer [25], they also help them become treatment-resistant, which leads to treatment failure as well as increased toxicity for the patient, which can result in patient death. Researchers such as Axelrod and colleagues [26] have speculated that tumor cells do not need to acquire all the hallmarks of cancer to become an aggressive cancer but that the cooperation between different cells with different abilities might allow the tumor as a whole to acquire all the hallmarks. A few years ago, Hanahan and Weinberg updated their original research to include disregulated metabolisms as one of the hallmarks of cancer [27]. Here we suggest that cooperation between cells with different metabolic needs and abilities could allow the tumor to grow faster but also present a new therapeutical target that could be clinically exploited. Namely, this cooperation, as described by Kinaercy and colleagues, allows for hypoxic cells to benefit from the presence of oxygenated non-glycolytic cells with modest glucose requirements, whereas cells with aerobic metabolism can benefit from the lactic acids that are the byproduct of anaerobic metabolism (see Figure 13).

By targeting this cooperation, a tumor’s growth and progression could be disrupted using novel microenvironmental pH normalizers. What our work suggests is that small perturbations could return the system back to a state different from the one it started so that the microenvironmental impact does not need to be too substantial for the therapy to have an impact. The work we have described here supports the hypothesis that hysteresis would allow us to apply treatments for a short duration of time with the aim of changing the nature of the game instead of killing tumor cells. This would have the combined advantages of reducing toxicity and side effects and decreasing selection for resistant tumor phenotypes and thus reducing the emergence of resistance to the treatment. For instance, treatments that aim to reduce the acidity of the environment [28] would impact not only acid producing cells but also the acid-resistant normoxic ones.

Our techniques (the hysteresis mechanism and the optimal control mechanism) can be applied to the cancer game [10] with two types of tumor phenotypic strategies: hypoxic cells and oxygenated cells (

Table 1. Payoff matrix for the cancer game in [10], where $L>G_o/2$. This $2\times 2$ game represents the tumor metabolic symbiosis rewards (ATP generation). The row agent represents hypoxic cells, and the column one represents oxygenated cell energy generation values based on their collective actions. Specifically, oxygenated cells can use both glucose and lactate for energy generation, whereas the hypoxic cells can use only glucose. Empirical data as discussed in [10] suggests that $L>Go/2$.

Hypoxic/Oxygenated	Glucose	Lactate
Glucose	$G_h/2,G_o/2$	$G_h,L$
Lactate	$0,G_o$	$0,0$

). These cells inhabit regions where oxygen could be either abundant or lacking. In the former, oxygenated cells with regular metabolism thrive but in the latter, hypoxic cells whose metabolism is less reliant on the presence of oxygen (but more on the presence of glucose) have higher fitness.

6.2. Taxation

A direct application for the solution concept of QRE is to analyze the effect of taxation, which has been discussed in [9]. Unlike Nash equilibria, for QREs, if we multiply the payoff matrix by some factor $\alpha$, the equilibrium does change. This is because, by multiplying $\alpha$, effectively we are dividing the temperature parameters by $\alpha$. This means that, if we charge taxes to the players with some flat tax rate $\alpha -1$, the QREs will differ. Formally, we define the base temperature $T_0$ as the temperature when no tax is applied for both players. Then, we can define the tax rate for each player as ${\alpha }_x=1-T_0/T_x,{\alpha }_y=1-T_0/T_y$, respectively.

We demonstrate how the hysteresis mechanism can be applied in a $2\times 2$ game via taxation with Example 1. Recall that in Example 1, we have two types of agents. We can consider these two types of agents as corresponding to two different sectors of the economy (e.g., aircraft manufacturing versus car manufacturing), which need to coordinate on their choice between two different competing technologies that are related to both sectors (e.g., 3D-printing). We can consider the row player as being the aircraft manufacturer and the column player as being the car manufacturer, with payoff matrices specified in

Table 2. Payoff matrix for a coordination game between two agents where neither of the two pure Nash Pareto dominates the other. States where both agents play the first strategy (Technology 1) are nearly socially optimal and they can be selected via a bifurcation argument.

Sector A/Sector B	Technology 1	Technology 2
Technology 1	$10,2$	$0,0$
Technology 2	$0,0$	$5,4$

. By assuming both players are of bounded rationality with temperature 1, we assume the base temperatures for both players are $T_0=1$. In this game, the equilibrium where both players choose Technology 1 has greater social welfare than the equilibrium where both players choose Technology 2. Consider the situation where, initially, the system is in an equilibrium state where both players choose Technology 2 with high probability. Then, with taxation, we have shown in the previous sections that we are able to increase the social welfare via the hysteresis mechanism or the optimal control mechanism. Here, we demonstrate how the simplified process that we have described in Example 1 can improve the social welfare in this game (see Figure 2 for the bifurcation diagram of this game):

• The initial state is $(0.05,0.14)$, where the row agent chooses Technology 1 with probability $0.05$ and the column agent chooses Technology 1 with probability $0.14$. This is an equilibrium state when we impose the tax rate ${\alpha }_x\approx 0$ to the row agent and the tax rate ${\alpha }_y\approx 0.5$ to the column agent (where $T_x\approx 1$ and $T_y\approx 2$).
• Fix the tax rate for the column agent at ${\alpha }_y=0.5$ (where $T_y=2$) and increase the tax rate for the row agent to ${\alpha }_x=0.8$ (where $T_x=5$). Under this assignment of tax rates, there is only one QRE correspondence.
• Fix the tax rate for the column agent at ${\alpha }_y=0.5$ (where $T_y=2$) and decrease the tax rate ${\alpha }_x$ for the row agent back to 0 (where $T_x=1$). Now $x\approx 0.997$, where both agents choose Action 1 with high probability.

In [9], they considered three approaches—“anarchy,” “socialism,” and “market”—of how the taxes can be dynamically adjusted by the society, depending on whether the taxes are determined in a decentralized manner, by an external regulator, or through bargaining, respectively. The concept of our mechanisms is a variant of the “socialism” scheme since in our model the mechanism, who can be thought as an external regulator, determines the tax rates. Our mechanisms are systematic approaches that optimize an objective where, in [9], the trajectories toward maximizing expected utilities are considered.

7. Connection to Previous Works

Recently, there has been a growing interplay between game theory, dynamical systems, and computer science. Examples include the integration of replicator dynamics and topological tools [29,30,31] in algorithmic game theory, and Q-learning dynamics [5] in multi-agent systems [6]. Q-learning dynamics has been studied extensively in game settings, e.g., by Sato et al. in [13] and Tuyls et al. in [14]. In [12], Q-learning dynamics is considered as an extension of replicator dynamics driven by a combination of payoffs and entropy. Recent advances in our understanding of evolutionary dynamics in multi-agent learning can be found in the survey in [32].

We are particularly interested in the connection between Q-learning dynamics and the concept of QRE [7] in game theory. In [11], Cominetti et al. study this connection in traffic congestion games. The hysteresis effect of Q-learning dynamics was first identified in 2012 by Wolpert et al. [9]. Kianercy et al. in [16] observed the same phenomenon and provided discussions on bifurcation diagrams in $2\times 2$ games. The hysteresis effect has also been highlighted in recent follow-up work by [10] as a design principle for future cancer treatments. It was also studied in [33] in the context of minimum-effort coordination games. However, our current understanding is still mostly qualitative and in this work we have pushed towards a more practically applicable, quantitative, and algorithmic analysis.

Analyzing the characteristics of various dynamical systems has also been attracting the attention of computer science community in recent years. For example, besides the Q-learning dynamics, the (simpler) replicator dynamics has been studied extensively due to its connections [30,34,35] to the multiplicative weight update (MWU) algorithm in [36].

Much attention has also been devoted to biological systems and their connections to game theory and computation. In recent work by Mehta et al. [37], the connection with genetic diversity was discussed in terms of the complexity of predicting whether genetic diversity persists in the long run under evolutionary pressures. This paper builds upon a rapid sequence of related results [38,39,40,41,42,43]. The key result is [39,40], where it was made clear that there is a strong connection between studying replicator dynamics in games and standard models of evolution. Follow-up works show how dynamics that incorporate errors (i.e., mutations) can be analyzed [44] and how such mutations can have a critical effect on ensuring survival in the presence of dynamically changing environments. Our paper makes progress along these lines by examining how noisy dynamics can introduce, for example, bifurcations.

We were inspired by recent work by Kianercy et al. establishing a connection between cancer dynamics and cancer treatment and studying Q-learning dynamics in games. This is analogous to the connections [39,40,45] between MWU and evolution detailed above. It is our hope that by starting off a quantitative analysis of these systems we can kickstart similarly rapid developments in our understanding of the related questions.

8. Conclusions

In this paper, we perform a quantitative analysis of bifurcation phenomena connected to Q-learning dynamics in $2\times 2$ games. Based on this analysis, we introduce two novel mechanisms, the hysteresis mechanism and the optimal control mechanism. Hysteresis mechanisms use transient changes to the system parameters to induce permanent improvements to its performance via optimal (Nash) equilibrium selection. Optimal control mechanisms induce convergence to states whose performance is better than the best Nash equilibrium, showing that by controlling the exploration/exploitation tradeoff, we can achieve strictly better states than those achievable by perfectly rational agents.

We believe that these new classes of mechanisms could lead to interesting new questions within game theory. Importantly they could also lead to a more thorough understanding of cancer biology and how treatments could be designed not to kill tumor cells but to induce transient changes in the game with long-lasting consequences, impacting the equilibrium in ways that would be therapeutically useful.