Toward Convergence in Multi-Agent Reinforcement Learning: Best-Response Space Shrinking as a Sufficient Condition

Abstract

We study convergence in multi-agent reinforcement learning (MARL) through the lens of sufficient conditions, using a single-point-of-failure analysis applied to multi-agent policy iteration integrated with linear programming (MAPI-LP), where results are proven for pure coordination games, and extension to broader settings is conjectured. We identify two sufficient conditions for convergence to Markov Perfect Equilibrium (MPE). The first is stability in best-response space that emerges from value monotonicity. The second, monotonic best-response space shrinking (MBRSS), is a novel condition requiring that each agent’s best-response space contracts monotonically across iterations until it collapses to a stable space. Furthermore, we show that MBRSS does not necessarily imply monotonic improvement in this setting. However, value monotonicity and stability in best-response space imply each other when a complementary condition is applied. Building on this hierarchical relationship, we propose conjectures on sufficient condition relationships in both serial and parallel MARL. In addition, we propose conjectures on generalized MBRSS to arbitrary finite repeated games and validation of stability in best-response space. We further discuss connections between MBRSS and existing related frameworks, and outline directions toward a taxonomy of sufficient conditions for MARL convergence.

1. Introduction

MARL problems consist of three fundamental components: a game model defining the strategic environment, solution concepts specifying desired equilibria, and algorithms providing computational methods to reach these solutions [1]. From an algorithm development perspective, researchers typically pursue three primary directions: generalizing single-agent reinforcement learning algorithms to multi-agent settings [2,3,4,5,6], improving existing MARL algorithms for better reliability and stability [7,8,9], or adapting algorithms to new classes of game models or new solution concepts [10,11,12]. However, changing any component or a combination of them introduces non-stationarity or compatibility issues that prevent convergence. To address these problems, new sufficient conditions are needed to explain and predict when convergence will succeed or fail and to guide algorithm design and improvement.

To pursue this line of understanding, we begin from the generalization of policy iteration, the simplest extension of reinforcement learning to the multi-agent setting. We propose and analyze a multi-agent policy iteration with linear programming (MAPI-LP) in the context of repeated games, aimed at convergence to Markov Perfect Equilibria (MPE) under a sufficient condition and convergence analysis. The existence of MPE is guaranteed in any finite stochastic game, and thus also in repeated games [13]. However, MAPI-LP, a model of best-response dynamics that intuitively satisfies the definition of MPE, does not guarantee convergence (Section 3). Because of this, our approach adopts a single-point-of-failure perspective: by asking where and how MAPI-LP could fail, we identify sufficient conditions that ensure convergence.

In Section 4, first we explain why value monotonicity (Definition 12) is not generally sufficient for MPE convergence. Then we provide two main theoretical contributions. The first is stability in best-response space (Definition 13). The second is monotonic best-response space shrinking (MBRSS, Theorem 1), a condition requiring that each agent’s best-response set decreases monotonically across iterations until it collapses to a stable space. We further show that MBRSS is not merely abstract, by showing that sequential MAPI-LP provably satisfies MBRSS in pure coordination games (Theorem 3), converging to a stable best-response space, where all of its members are MPE, in at most n iterations. Building on these conditions, we propose several conjectures related to MBRSS generalization and relationships between sufficient conditions, respectively (Section 5).

Finally, we discuss broader implications and future directions, including connections between MBRSS and existing frameworks, as well as directions for extending MBRSS to broader game classes and another application, and the development of a taxonomy of sufficient conditions for MARL convergence (Section 6). Together, these results provide both theoretical guarantees in simplified settings and a roadmap for investigating convergence in more complex multi-agent environments.

2. Preliminary

2.1. Sufficient Conditions and Where to Find Them

Sufficient conditions for convergence guarantee that a sequence, series, or iterative process will converge. In the context of MARL, such conditions typically involve assumptions, constraints, or specific properties of the components within the scope of learning towards a given solution concept. Examining these components through theoretical analysis or empirical observation can help identify sufficient conditions that may lead to significant improvements in algorithms.

For example, ref. [8] found that the full factorization of the action-value function used in VDN [14] is not necessary, and then proposed QMIX, which ensures a condition known as the Individual-Global-Max (IGM). Later, ref. [9] observed that both QMIX and VDN rely on two sufficient conditions for IGM: additivity and monotonicity. However, there exist tasks in which these conditions are not satisfied, motivating the development of QTRAN. Hence, depending on the interpretation of IGM, we can trace the evolution of algorithms within the value decomposition category. For example, ref. [15] stated that the sufficient condition for IGM sacrifices the completeness of the action-value function class, resulting in a parameter space structure that fails to guarantee optimality. Ref. [16] proposed a sufficient condition of identifiability on expected KL-divergence in an attempt to achieve equilibrium in a fully decentralized manner, and found that one of their algorithm’s components shares similarities with IGM.

In contrast, sufficient conditions can also emerge from the properties of certain components. For example, ref. [17] developed an advantage estimation method based on a sufficient condition for policy invariance, a desirable property in which the optimal policy remains unchanged under reward shaping. Ref. [18] investigated convergence probabilities with respect to the absorbing best-response graph, under the implication of a sufficient condition in weakly acyclic stochastic games (as studied by [19]). This graph corresponds to the best-response graph modeled as an absorbing Markov chain.

These works collectively highlight that sufficient conditions for convergence in MARL are often tied to specific structural assumptions, which may not hold across all tasks. When these assumptions fail, the algorithm in question can struggle to converge optimally, encouraging developments of alternative methods or improvements of existing methods. Examining single points of failure, where an algorithm is vulnerable to convergence failure, can offer valuable insight into identifying new sufficient conditions. In other words, by pinpointing the exact component or assumption that breaks down, we can better understand the necessities behind convergence and design more generalizable and robust algorithms. For example, along these lines, ref. [20] employed a strictly positive real property to establish a sufficient condition ensuring practical finite-time convergence for the stability of interconnected subsystems in a nonlinear control framework.

2.2. Multi-Agent Policy Iteration

The policy iteration is a classic reinforcement learning for Markov decision processes that alternates between two steps: policy evaluation ($\stackrel{E}{\to }$) and policy improvement ($\stackrel{I}{\to }$) [21]. Its greedy improvement step, which guarantees convergence to the optimal solution, naturally generalizes to best-response dynamics in multi-agent stochastic games, with Markov Perfect Equilibrium (MPE) [22] as the solution concept. For this generalization, we define $\stackrel{E}{\to }$, $\stackrel{I}{\to }$, and MPE as follows:

Definition 1.

Given an input joint policy π, the process $\stackrel{E}{\to }$ computes the value function $v^{\pi }$ using the Bellman operator. In other words, it finds $v^{\pi }$ such that

$$\forall i,s,v_i^{\pi }\left(s\right)=\sum _a\pi (a\mid s)\sum _{s^{\prime }\in S}\mathcal{P}(s^{\prime }\mid s,a)\left[{\mathcal{R}}_i(s,a,s^{\prime })+\gamma v_i^{\pi }\left(s^{\prime }\right)\right],$$

(1)

where $\mathcal{P}$ is the state transition probability and $R_i$ is the reward function for agent i.

Definition 2.

Given an input joint policy $\pi ={\left({\pi }_i\right)}_i$, the process $\stackrel{I}{\to }$ outputs an updated joint policy ${\pi }^{\prime }$, where each agent’s policy ${\pi }_i^{\prime }$ is a state-wise best response to the others, i.e.,

$${\pi }^{\prime }\in \left\{{\left({\pi }_i^{\prime }\right)}_i|\forall i,s,\forall {\pi }_i^{″},v_i^{({\pi }_i^{\prime },{\pi }_{-i})}\left(s\right)\ge v_i^{({\pi }_i^{″},{\pi }_{-i})}\left(s\right)\right\}.$$

(2)

Definition 3.

A joint policy ${\pi }^{*}=({\pi }_i^{*},{\pi }_{-i}^{*})$ is said to be (stationary) MPE if, for every agent i and state s,

$$\forall {\pi }_i,v_i^{({\pi }_i^{*},{\pi }_{-i}^{*})}\left(s\right)\ge v_i^{({\pi }_i,{\pi }_{-i}^{*})}\left(s\right),$$

(3)

where $v_i^{\pi }\left(s\right)$ denotes the expected return for agent i starting from state s under the joint policy π.

Following the proof structure of [13], the existence of an MPE in discounted finite stochastic games can be established using fixed-point theorems applied to the policy evaluation and improvement processes individually:

• Banach’s fixed-point theorem ensures that $\stackrel{E}{\to }$ yields a unique value function $v^{\pi }$ for each joint policy $\pi$.
• Kakutani’s fixed-point theorem guarantees that $\stackrel{I}{\to }$ admits a fixed point, which corresponds to MPE under certain conditions on input joint policy $\pi$.

Therefore, starting from an arbitrary initial joint policy, alternating between $\stackrel{E}{\to }$ and $\stackrel{I}{\to }$ (as in [21]) can determine an MPE only in some cases, but does not guarantee convergence to an MPE.

2.3. Repeated Games

A repeated game (Definition 4) is a special class of stochastic games with n agents and a single state, where the state transitions back to itself at every time step, having only one value function to be calculated for each agent (Lemma 1).

Definition 4.

A discounted finite repeated game with n agents and bounded rewards, where the state transitions back to itself every time step, is defined by $G=(I,{\left\{A_i\right\}}_i,{\left\{{\mathcal{R}}_i\right\}}_i,\gamma )$, which consists of:

• $I=\{1,\dots ,n\}$, a finite set of agents,
• $A_i$, a finite set of actions available to agent i; the set of joint actions is defined as $A={\prod }_i{A}_i$.
• ${\mathcal{R}}_i:A\to [R_{\mathit{min}},R_{\mathit{max}}]$, a deterministic reward function of agent i, where constants $R_{\mathit{min}}\le R_{\mathit{max}}$.
• $\gamma \in [0,1)$, a discount factor.

Since the only state always repeats to itself with a given joint policy, $\stackrel{E}{\to }$ (Definition 1) then evaluates the value function as follows.

Lemma 1.

In a repeated game, from a given joint policy, if we follow process $\stackrel{E}{\to }$, the value function for agent i is

$$v_i^{\pi }={\displaystyle \frac{1}{1-\gamma }}\sum _a\pi \left(a\right){\mathcal{R}}_i\left(a\right).$$

(4)

Repeated games have been widely studied as a testbed for understanding the emergence of cooperative behavior in multi-agent systems. For example, ref. [23] investigates two-agent repeated games under Q-learning, demonstrating that the interplay between memory strength and future discounting determines whether agents converge to cooperative or defective policies. Their work highlights that even in the simplest repeated game setting, convergence behavior is highly sensitive to algorithm parameters and the structure of agent interactions. Hence, its simplicity makes it convenient and effective for analyzing the convergence of algorithms and provides a natural stepping stone toward generalizing to stochastic games with more steps. For these reasons, we choose the repeated game as our model of choice.

To support our analysis, Definitions 1–3 are generalized to the context of repeated games. In addition, we formally define the concepts of policy space and joint policy space as follows:

Definition 5.

The policy space for agent i, denoted ${\Delta }_{A_i}$, is the set of all possible probability distributions over actions in $A_i$ that agent i can choose. Formally,

$${\Delta }_i\doteq \left\{{\left({\pi }_i\left(a_i\right)\right)}_{a_i\in A_i}|\sum _{a_i\in A_i}{\pi }_i\left(a_i\right)=1\mathit{and}\forall a_i\in A_i,{\pi }_i\left(a_i\right)\ge 0\right\}.$$

(5)

Thus, ${\Delta }_i$ is a standard simplex over the action set $A_i$.

Definition 6.

The joint policy space, denoted Δ, is the set of all possible joint policies. Formally,

$$\Delta \doteq \prod _i{\Delta }_i.$$

(6)

3. Linear Programming Formulation

Solving $\stackrel{I}{\to }$ in Definition 2 is equivalent to solving a linear programming (LP) problem specified in Definition 7 for each agent i at iteration k. It seeks the best-response policy ${\pi }_i^{k+1}$ to the opponents’ policies ${\pi }_{-i}^k$, while ensuring a better response in expected value, and the output policy ${\pi }_i^{k+1}$ lies within the policy space ${\Delta }_i$.

Definition 7.

For agent i, assuming that $v_i:\Delta \to \mathbb{R}$ is a continuous linear function, at iteration k, the LP problem, seeking best-response policy for agent i to ${\pi }_{-i}^k$, is defined as follows:

• Maximize:

  $$v_i^{({\pi }_i^{k+1},{\pi }_{-i}^k)}$$

  (7)
• Subject to:

  $$v_i^{({\pi }_i^{k+1},{\pi }_{-i}^k)}\ge v_i^{{\pi }^k}$$

  (8)

  $$\sum _{a_i\in A_i}{\pi }_i^{k+1}\left(a_i\right)=1$$

  (9)

  $$\forall a_i\in A_i,{\pi }_i^{k+1}\left(a_i\right)\ge 0$$

  (10)

Remark 1.

The LP defined in Definition 7 has the following properties:

(a) 

The feasible region is a non-empty convex polytope defined by its linear constraints.

(b) 

There exists an optimal solution at a vertex of the feasible region.

(c) 

If multiple vertices yield the optimal solution, then all points within the convex hull of those vertices are also optimal.

(d) 

The set of all optimal solutions is a non-empty compact convex set.

As a note for Remark 1, the feasible region is a closed subset of the probability simplex ${\Delta }_i$, which is compact; hence the feasible region is compact. Since the objective function is continuous and linear, the set of all optimal policies is a closed convex subset of the feasible region; therefore it is compact and convex.

Remark 1(c) suggests that the convergence behavior depends not only on the trajectory of the update chain of $\stackrel{E}{\to }$ and $\stackrel{I}{\to }$, but also on the choice of best responses. We then separate policy improvement into two parts for each agent: identifying the set of best responses $(\stackrel{I_1}{\to })$ and selecting a specific best response $(\stackrel{I_2}{\to })$.

Definition 8.

The best-response function solving the LP problem (Definition 7) for agent i is denoted by a set-valued function

$${BR}_i:{\Delta }_{-i}\rightrightarrows {\Delta }_i,{BR}_i\left({\pi }_{-i}\right)=\underset{{\pi }_i}{argmax}{v}_i^{({\pi }_i,{\pi }_{-i})},$$

(11)

Definition 9.

At iteration k, consider agent i and a given joint policy ${\pi }^k$. The process $\stackrel{I_1}{\to }$ finds all vertices of optimal solution for LP defined in Definition 7 by vertex testing to obtain the set of best responses ${BR}_i\left({\pi }_{-i}^k\right)$, which is a convex hull of these optimal vertices. In other words, we can call ${BR}_i\left({\pi }_{-i}^k\right)$ the best-response space of agent i, at iteration k. By this process, it is guaranteed that ${BR}_i\left({\pi }_{-i}^k\right)$ is convex and compact (Remark 1(d)).

Definition 10.

At iteration k, considering agent i, the process $\stackrel{I_2}{\to }$ arbitrarily selects a policy

$${\pi }_i^{k+1}\in {BR}_i\left({\pi }_{-i}^k\right).$$

(12)

By combining $\stackrel{E}{\to }$, $\stackrel{I_1}{\to }$, and $\stackrel{I_2}{\to }$, we obtain a version of multi-agent policy iteration integrated with LP, called MAPI-LP, that simultaneously solves LP for each agent.

Definition 11.

MAPI-LP process at iteration k proceeds as follows:

$${\pi }^k\stackrel{E}{\to }{v}^{{\pi }^k}\to \left\{\begin{array}{c}({\pi }_{-1}^k,v_1^{{\pi }^k})\stackrel{I_1}{\to }{BR}_1\left({\pi }_{-1}^k\right)\stackrel{I_2}{\to }{\pi }_1^{k+1}\\ ⋮\\ ({\pi }_{-i}^k,v_i^{{\pi }^k})\stackrel{I_1}{\to }{BR}_i\left({\pi }_{-i}^k\right)\stackrel{I_2}{\to }{\pi }_i^{k+1}\\ ⋮\\ ({\pi }_{-n}^k,v_n^{{\pi }^k})\stackrel{I_1}{\to }{BR}_n\left({\pi }_{-n}^k\right)\stackrel{I_2}{\to }{\pi }_n^{k+1}\end{array}\right\}\to {\pi }^{k+1}.$$

(13)

MAPI-LP generates a sequence of joint policies ${\left\{{\pi }^k\right\}}_{k\ge 0}$.

Since $\stackrel{I}{\to }$ is equivalent to the combination of $\stackrel{I_1}{\to }$ and $\stackrel{I_2}{\to }$, we know that it is possible to find an MPE with MAPI-LP, although convergence is not guaranteed, as stated in Section 2.2.

4. Sufficient Conditions for MPE Convergence

To address sufficient conditions, one can examine the outcome of each step in the process. If any step tends to violate the solution concept, in this case the MPE, then some form of sufficient condition must exist to prevent such violations and ensure convergence. In other words, it must be possible to find sufficient conditions on single points of convergence failure. For completeness, we assume the following:

Assumption 1.

The joint policy sequence ${\left\{{\pi }^k\right\}}_{k\ge 0}$ is an infinite sequence and is generated by MAPI-LP, where the notion of $\left\{{\pi }^k\right\}$ refers to this.

Assumption 2.

The function $v_i:\Delta \to \mathbb{R}$ is linear continuous for all i.

4.1. Monotonic Improvement

In our representation of best-response dynamics, each agent proposes a new policy that is a best response to the others, in terms of value function. Hence, one of the desired properties is value-monotonic (Definition 12).

Definition 12

(Value Monotonicity). The sequence ${\left\{{\pi }^k\right\}}_k$ is improving its value monotonically, if

$$\forall i,k\ge 0,v_i^{{\pi }^k}\le v_i^{({\pi }_i^{k+1},{\pi }_{-i}^k)}\le v_i^{{\pi }^{k+1}}.$$

(14)

Corollary 1.

If $\left\{{\pi }^k\right\}$ is value-monotonic, ${\left\{v_i^{{\pi }^k}\right\}}_{k\ge 0}$ is a non-decreasing monotone sequence. By monotone convergence theorem, for all i, the sequence of value functions converges to a constant $c_i$, that is

$$\underset{k\to \infty }{lim}{v}_i^{{\pi }^k}=\underset{k}{sup}{v}_i^{{\pi }^k}=c_i.$$

(15)

Corollary 2.

If $\left\{{\pi }^k\right\}$ is value-monotonic, it may hold that two successive joint policies ${\pi }^k$ and ${\pi }^{k+1}$ are distinct, i.e., ${\pi }^k\ne {\pi }^{k+1},$ while their value functions coincide, $v^{{\pi }^k}=v^{{\pi }^{k+1}}.$

By Corollary 1, value monotonicity guarantees that each agent value sequence ${\left\{v_i^{{\pi }^k}\right\}}_{k\ge 0}$ is monotone non-decreasing and converges to a constant limit $c_i$. In particular, Corollary 2 suggests that the sequence of joint policies may continue to move within a set of value-equivalent policies even after the values have stabilized, and potentially cause a cycle. Therefore, value monotonicity establishes convergence in value space, but not sufficient convergence of the policies as an MPE.

4.2. Stability in Best-Response Space

Considering $\stackrel{I_1}{\to }$, we can collectively assemble its results as a sequence of best-response spaces. The idea of MPE can derive stability in best-response space (Definition 13) as another sufficient condition for MAPI-LP.

Definition 13.

The sequence $\left\{{\pi }^k\right\}$ is said to be stable in best-response space ${\Delta }^{*}$ at iteration K, if there exists ${\Delta }^{*}={\prod }_i{\Delta }_i^{*}\subseteq \Delta$, such that

$$\forall i,{\Delta }_i^{*}={BR}_i\left({\pi }_{-i}^K\right)={BR}_i\left({\pi }_{-i}^{K+1}\right)=\dots ,$$

(16)

Since ${\Delta }_i^{*}$ does not change from $k\ge K$, then we can alternatively define it without iteration index as

$${\Delta }_i^{*}=\left\{{\pi }_i\right|{\pi }_i\in {BR}_i\left({\pi }_{-i}\right),{\pi }_{-i}\in {\Delta }_{-i}^{*}\},$$

(17)

where ${\Delta }_{-i}^{*}={\prod }_{j\in I\setminus \left\{i\right\}}{\Delta }_j^{*}$.

To see why Equation (17) is well-defined, note that once stability is reached at iteration K, the best-response space of each agent no longer changes. Therefore, any arbitrary selection of ${\pi }_i^k$ where $k\ge K$ produces the same stable space ${\Delta }_i^{*}$, and so ${\Delta }^{*}={\prod }_i{\Delta }_i^{*}$. Since $\left\{{\pi }^k\right\}$ is infinite by Assumption 1, there exists an infinite subsequence of best responses beyond iteration K for each i, all mapping within ${\Delta }_i^{*}$, which exhaustively generates ${\Delta }_i^{*}$ itself. This gives Equation (17) its self-referential character: the stable space is exactly the set of all best responses that agents can play against one another within ${\Delta }^{*}$, without reference to any specific iteration index.

These lemmas establish the connection between stability in the best-response space and MPE. Lemma 2 shows that once the learning dynamics enter a stable best-response space ${\Delta }^{*}$, all subsequent iterations remain within that space. Building on this property, Lemma 3 proves that every joint policy in ${\Delta }^{*}$ is MPE.

Lemma 2.

If $\left\{{\pi }^k\right\}$ is stable in ${\Delta }^{*}$ at iteration K, $\forall i,k\ge K,{\pi }_i^k\in {\Delta }_i^{*}$.

Proof. 

Immediate from the definition of stability in best-response space, assuming $\left\{{\pi }^k\right\}$ is generated by MAPI-LP. □

Lemma 3.

All π in a stable best-response space ${\Delta }^{*}$ are MPEs.

Proof. 

By Definition 8, we know that

$$\forall i,\forall {\pi }_{-i}\in {\Delta }_{-i},\forall {\pi }^{\prime },{\pi }^{″}\in {\mathrm{BR}}_i\left({\pi }_{-i}\right),v_i^{({\pi }_i^{\prime },{\pi }_{-i})}=v_i^{({\pi }_i^{″},{\pi }_{-i})}.$$

(18)

If ${\pi }_{-i}\in {\Delta }_{-i}^{*}$, then ${\mathrm{BR}}_i\left({\pi }_{-i}\right)\in {\Delta }_i^{*}$ by Definition 13. Hence, in this case,

$$\forall i,\forall {\pi }_{-i}\in {\Delta }_{-i}^{*},\forall {\pi }^{\prime },{\pi }^{″}\in {\Delta }_i^{*},v_i^{({\pi }_i^{\prime },{\pi }_{-i})}=v_i^{({\pi }_i^{″},{\pi }_{-i})}.$$

(19)

Since quantifiers of (19) can exhaustively compare any combination of joint policies in ${\Delta }^{*}$, then it is equivalent to

$$\forall i,\forall \pi ,{\pi }^{\prime }\in {\Delta }^{*},v_i^{\pi }=v_i^{{\pi }^{\prime }}.$$

(20)

Since $\pi \in {\Delta }^{*}$ is no better than other outside policies, then all $\pi \in {\Delta }^{*}$ are MPEs. □

The key observation is that stability implies mutual best-response consistency among all policies in ${\Delta }^{*}$, while the best-response function guarantees value indifference between all admissible best responses. Consequently, no agent can improve its value by deviating from any policy in ${\Delta }^{*}$, establishing the entire stable space as an equilibrium set (Theorem 1).

Theorem 1.

If $\left\{{\pi }^k\right\}$ is stable in best-response space ${\Delta }^{*}$ at K, then, for all $k\ge K$, ${\pi }^k$ is an MPE.

4.3. Monotonic Best-Response Space Shrinking

Another issue is that the choice of the best response is significant in $\stackrel{I_2}{\to }$, according to the possibility of equally effective policies (Remark 1(c)). Since the process in an ideal trajectory ultimately converges to a single joint policy corresponding to MPE, iteratively reducing the size of the set of possible best responses can theoretically lead to the desired outcome, possibly as a contraction mapping on policy space itself that converges to a stable best-response space. Therefore, with this idea, we define the following sufficient condition:

Theorem 2

(Monotonic Best-Response Space Shrinking, MBRSS). If ${\left\{{\pi }^k\right\}}_{k\ge 0}$ satisfies

$$\forall i,\forall k\ge 0,{BR}_i\left({\pi }_{-i}^{k+1}\right)\subseteq {BR}_i\left({\pi }_{-i}^k\right),$$

(21)

then

$$\underset{k\to \infty }{lim}{BR}_i\left({\pi }_{-i}^k\right)={\Delta }_i^{*}$$

(22)

and all $\pi \in {\prod }_i{\Delta }_i^{*}={\Delta }^{*}$ are MPEs.

Proof. 

Assuming that (21) holds, BR is a sequence of compact best-response spaces, and each of them is a subset of its prior best-response spaces. Therefore,

$$\underset{k\to \infty }{lim}{\mathrm{BR}}_i\left({\pi }_{-i}^k\right)=\bigcap _{k\ge 1}{\mathrm{BR}}_i\left({\pi }_{-i}^k\right)={\Delta }_i^{*}.$$

(23)

Since ${\mathrm{BR}}_i$ will always contain at least one member, ${\mathrm{BR}}_i\left({\pi }_i^0\right)$ will never be empty, and neither will ${\Delta }_i^{*}$.

When the best-response space of each agent converges to ${\Delta }_i^{*}$, $\left\{{\pi }^k\right\}$ becomes stable in ${\Delta }^{*}={\prod }_i{\Delta }_i^{*}$. Thus, by Lemma 3, for all $\pi \in {\Delta }^{*}$. □

In addition, this implies that the joint policy space, formed as the Cartesian product of the best-response sets of each agent i, also shrinks monotonically over iterations.

4.4. Satisfiability of MBRSS in Pure Coordination Games

MBRSS is defined over the policy space and becomes particularly useful in games where multiple joint actions yield the same reward, such as in pure coordination games [24]. This condition can help anchor the update direction, reduce variability, and narrow the trajectory toward a best-response path. In fact, we can view this sufficient condition as a set-valued contraction mapping. Accordingly, we demonstrate that sequential MAPI-LP satisfies MBRSS in the context of pure coordination games (Definition 14), as shown in Theorem 3.

Definition 14.

A pure coordination game is a repeated game in which all agents share the following reward structure:

$${\mathcal{R}}_i\left(a\right)=\left\{\begin{array}{cc}r,\hfill & a\in A^{\ell },\hfill \\ 0,\hfill & a\notin A^{\ell },\hfill \end{array}\right.$$

(24)

where the alignment set $A^{\ell }\subseteq A$ is not empty, and r is a positive constant. For each agent i, we define the individual alignment set as $A_i^{\ell }=\{a_i\mid \exists a_{-i},(a_i,a_{-i})\in A^{\ell }\}=\{a_i^1,\dots ,a_i^{|A_i^{\ell }|}\}$, representing the set of alignment actions available to agent i.

Definition 15.

We define a count function $\mathcal{C}$ to return the total number of unique alignments in the alignment set $A^{\ell }$ that contain its input. For example, if $A^{\ell }=\{(a_1^1,a_2^1),(a_1^1,a_2^2)\}$, then $\mathcal{C}\left(a_1^1\right)=2,\mathcal{C}\left(a_2^1\right)=1$ and $\mathcal{C}(a_1^1,a_2^1)=1$.

Lemma 4.

If $\mathcal{C}\left({\left(a_i\right)}_{i\in I^{\prime }}\right)\ge 1$ where $I^{\prime }\subset I$, then there exists $a_j\in A_j^{\ell }$ where $j\notin I^{\prime }$ that satisfies $\mathcal{C}({\left(a_i\right)}_{i\in I^{\prime }},a_j)\ge 1$.

Proof. 

This is a direct implication when $|A^{\ell }|\ge 1$. □

Definition 16.

Sequential MAPI-LP is a modified MAPI-LP, where at iteration k, only agent $i=(kmodn)+1$ selects a policy which is a vertex of its best-response space ${BR}_i\left({\pi }_{-i}^k\right)$, while for other agents the policies remain unchanged.

Theorem 3.

Assuming that the initial policies ${\pi }_i^0$ are uniformly distributed for all agents i and all agents follow sequential MAPI-LP (Definition 16), in any pure coordination game, MBRSS is satisfied and is stable in some best-response space at iteration $k=n$. Therefore, for all $k\ge n$, ${\pi }^k$ is an MPE.

Proof. 

At $k=0$, for each vertex of policy space ${\pi }_1^1\left(a_1\right)=1$ where $a_1\in A_1$,

$$v_i^{{\pi }_1^1,{\pi }_{-1}^0}=\left\{\begin{array}{cc}{\displaystyle \frac{r}{1-\gamma }}·{\displaystyle \frac{\mathcal{C}\left(a_1\right)}{{\prod }_{j\in I\setminus \left\{1\right\}}\left|A_j^{\ell }\right|}},\hfill & a_1\in A_1^{\ell },\hfill \\ 0,\hfill & a_1\notin A_1^{\ell }.\hfill \end{array}\right.$$

(25)

By process $\stackrel{I_1}{\to }$ the best-response space is

$${\mathrm{BR}}_1\left({\pi }_{-1}^0\right)=\mathrm{Hull}\left\{{\pi }_1^1\left(a_1\right)=1|a_1\in \underset{a_1\in A_1^{\ell }}{argmax}{\displaystyle \frac{\mathcal{C}\left(a_1\right)}{{\prod }_{j\in I\setminus \left\{1\right\}}\left|A_j^{\ell }\right|}}\right\},$$

(26)

where $\mathrm{Hull}\{·\}$ is the convex hull of the given set. Since $|A^{\ell }|$ is not empty, ${\mathrm{BR}}_i\left({\pi }_i^0\right)$ is not empty, so we can arbitrarily select a vertex of it as ${\pi }_1^1$ where ${\pi }_1^1\left(a_1\right)=1$, for the next iteration.

At the iteration $k<n$, agents $i\in 1,\dots ,k-1$ already select ${\pi }_i^i={\pi }_i^{i+1}=\dots ={\pi }_i^k$ where some ${\pi }_i^k\left(a_i\right)=1$ by sequential constraints on $\stackrel{I_2}{\to }$. This implies that $\mathcal{C}(a_1,a_2,\dots ,a_{k-1})\ge 1$. By Lemma 4, there exists $a_k\in A_k^{\ell }$ such that $\mathcal{C}(a_1,a_2,\dots ,a_{k-1},a_k)\ge 1$, which means

$${\mathrm{BR}}_k\left({\pi }_{-k}^k\right)=\mathrm{Hull}\left\{{\pi }_k^k\left(a_k\right)=1|a_k\in \underset{a_k\in A_k^{\ell }}{argmax}{\displaystyle \frac{\mathcal{C}(a_1,a_2,\dots ,a_k)}{{\prod }_{j\in I\setminus \{1,\dots ,k\}}\left|A_j^{\ell }\right|}}\right\}$$

(27)

is subset or equal to ${\mathrm{BR}}_k\left({\pi }_{-k}^{k-1}\right)$ due to the restrictions of positive $\mathcal{C}(·)$ imposed on each agent’s alignment set $A_i^{\ell }$.

Hence, at the smallest iteration $k\ge n$, all agents already choose some alignment $a^{\ell }={\left(a_i\right)}_i\in A^{\ell }$ where $\mathcal{C}\left(a^{\ell }\right)\ge 1$, which is an optimal choice. So agent $i=(kmodn)+1$ can only arbitrarily choose

$${\pi }_i^{k+1}\in {\mathrm{BR}}_i\left({\pi }_{-i}^k\right)=\mathrm{Hull}\left\{{\pi }_i^{k+1}\left(a_i\right)=1|\exists a_i\in A_i^{\ell },\mathcal{C}({\left(a_j\right)}_{j\in I\setminus \left\{i\right\}},a_i)=1\right\},$$

(28)

which does not improve value function and does not change the best-response space. So, the sequence of best-response spaces satisfies MBRSS (Theorem 2) and is stable in

$${\Delta }^{*}=\prod _i\mathrm{Hull}\left\{{\pi }_i^{k+1}\left(a_i\right)=1|\exists a_i\in A_i^{\ell },\mathcal{C}({\left(a_j^{\ell }\right)}_{j\in I\setminus \left\{i\right\}},a_i)=1\right\},$$

(29)

for some alignment $a^{\ell }={\left(a_i^{\ell }\right)}_i$ that all agents previously resemble at iteration k. Since we can identify ${\Delta }^{*}$ at iteration $k=n$, by Lemma 2, for all $k\ge n$, ${\pi }^k$ is an MPE. □

However, in this setting, MBRSS does not imply value monotonicity.

Corollary 3.

Under the same assumption of Theorem 3, the generated sequence of joint policies does not necessarily satisfy value monotonicity (Definition 12).

Proof. 

Let $\left\{{\pi }^k\right\}$ be the generated sequence, then for some agent i at iteration $k\le n$, $v_i^{({\pi }_i^k,{\pi }_{-i}^{k-1})}$ is not necessarily less than or equal to $v_i^{({\pi }_i^{k+1},{\pi }_{-i}^k)}$, because the ratio of count and fixed chosen policy is the same as in (27). So, $\left\{{\pi }^k\right\}$ also does not satisfy value monotonicity. □

Please note that Theorem 3 has very narrow settings, but it is a sufficient illustrative case that there exists a class of repeated games that naturally satisfies MBRSS.

5. Conjectures on Sufficient Conditions

5.1. Existence of Best-Response Dynamics for Generalized MBRSS

Sequential MAPI-LP (Theorem 3) induces a dimensional contraction mapping on the best-response space of the joint policy, where, for all i, the nested sequence ${\mathrm{BR}}_i^1\left({\pi }_{-i}^0\right)\supseteq {\mathrm{BR}}_i^1\left({\pi }_{-i}^1\right)\supseteq \dots \supseteq {\Delta }_i^{*}$ contracts monotonically in dimension, with the specific trajectory determined by agents’ best-response selection.

For example, consider a pure coordination game with two agents with three available actions each that consist of: $I=\{1,2\}$, $\forall i,A_i=\{a_i^1,\dots ,a_i^3\}$ and alignment set $A^{\ell }=\{(a_1^1,a_2^1),(a_1^1,a_2^2),(a_1^2,a_2^3)\}$. By using sequential MAPI-LP (Theorem 3), the sequence of generated best-response sets is illustrated in Figure 1, showing a sufficient case that all possible trajectories satisfy MBRSS under sequential policy selection.

Figure 1. Example of trajectories that satisfy MBRSS according to Theorem 3. The yellow area shows the best-response space corresponding to each iteration, and the black diamond shows the current policy for each agent. In this scenario, after agent 2 selects a policy, agent 1’s best response remains constant, and agent 1’s value function also remains unchanged (Lemma 2). Similarly, agent 2’s best response and value function do not change afterward.

However, MAPI-LP is heavily restricted by both the policy selection mechanism and the class of games in which MBRSS is proven to hold, which is the class of pure coordination games. Hence, a natural question is to ask whether there exist best-response dynamics beyond MAPI-LP that satisfy a generalized notion of MBRSS. This motivates the following conjecture, which relaxes MBRSS while keeping the property of stable best-response space:

Conjecture 1.

For any finite repeated game $G=(I,{\left\{A_i\right\}}_i,{\left\{R_i\right\}}_i,\gamma )$, if stable best-response space ${\Delta }^{*}$ exists for this game, there exists a set-valued best-response map $\mathcal{M}:{\Delta }_A\rightrightarrows {\Delta }_A$ that generates ${\left\{{\pi }^k\right\}}_{k\ge 0}$ that satisfies the following:

(a) 

$\forall k\ge 0,\exists k^{\prime }>k,\mathcal{M}\left({\pi }^{k^{\prime }}\right)\subseteq \mathcal{M}\left({\pi }^k\right)$

(b) 

${lim}_{k\to \infty }\mathcal{M}\left({\pi }^k\right)={\Delta }^{*}$

In addition, from the characteristic of stability in best-response space, by infinite iteration and arbitrary best-response selection, ${\Delta }_i^{*}$ can be expressed by (17), which is equivalent to checking if all joint policies in ${\Delta }^{*}={\prod }_i{\Delta }_i^{*}$ can map back to themselves, that is

$$\forall \pi \in {\Delta }^{*},{\Delta }^{*}=\prod _{i}B{R}_i\left({\pi }_{-i}\right)=\prod _i{\Delta }_i^{*}.$$

(30)

In case that ${\left\{{\pi }^k\right\}}_{k\ge 0}$ is stable in ${\Delta }^{*}$ at iteration k, we can test only all combinations of vertices in ${\Delta }^{*}$ to confirm this property, so Conjecture 2 is true. However, it will fail if at least one joint policy is within the convex hull and does not map back to itself in terms of best-response space.

Conjecture 2.

Consider the sequence $\left\{{\pi }^k\right\}$. Let ${BR}_i^V\left({\pi }_{-i}^k\right)$ be a set of vertices of ${BR}_i\left({\pi }_{-i}^k\right)$ and ${BR}^V\left({\pi }^k\right)={\prod }_i{BR}_i^V\left({\pi }_{-i}^k\right)$. At some iteration k, if

$$\forall \pi \in {BR}^V\left({\pi }^k\right),{BR}^V\left(\pi \right)={BR}^V\left({\pi }^k\right),$$

(31)

then

$${\Delta }^{*}=Hull\left({BR}^V\left(\pi \right)\right)=Hull\left({BR}^V\left({\pi }^k\right)\right)$$

(32)

and $\left\{{\pi }^k\right\}$ is stable in ${\Delta }^{*}$ at iteration $k=n$, which indicates convergence to an MPE-consistent space.

5.2. Relationship Between Sufficient Conditions Within the Same Algorithm

5.2.1. In Serial MARL

Both value monotonicity (Definition 12) and stability in best-response space (Definition 13) are sufficient conditions for MARL. However, the value monotonicity is defined in terms of value function improvement and does not involve details about how individual policies are chosen. In contrast, stability in best-response space is defined based on the selection of best responses, which depends on only the value function determined by $\stackrel{I_1}{\to }$, that is the best better-responses. This implies that a sufficient condition defined on the sequence ${\left\{v_i^{{\pi }^k}\right\}}_k$ has broader scope than one defined on the sequence ${\left\{{\mathrm{BR}}_i\left({\pi }_{i-1}^k\right)\right\}}_k$.

Corollary 4.

Suppose that $\left\{{\pi }^k\right\}$ is value-monotonic and ${lim}_{k\to \infty }{v}_i^{{\pi }^k}=c_i$ for all i. If there exists K such that, for all i, $k\ge K$, $v_i^{{\pi }^k}=c_i$ and ${BR}_i\left({\pi }_{-i}^k\right)={BR}_i\left({\pi }_{-i}^{k+1}\right)$, then $\left\{{\pi }^k\right\}$ is stable in best-response space at iteration K.

Corollary 5.

Suppose that $\left\{{\pi }^k\right\}$ is stable in best-response space ${\Delta }^{*}$ at iteration K. If, for all i, $0\le k\le K,v_i^{{\pi }^k}\le v_i^{({\pi }_i^{k+1},{\pi }_{-i}^k)}\le v_i^{{\pi }^{k+1}}$, then $\left\{{\pi }^k\right\}$ is also value-monotonic.

As in Corollaries 4 and 5, we see the potential relationship between sufficient condition, which leads to the following conjecture:

Conjecture 3.

Suppose that ${\mathbf{C}}_{\mathbf{1}}$ and ${\mathbf{C}}_{\mathbf{2}}$ are sufficient conditions for the convergence of a given MARL algorithm, that serially depend on each other, sharing some scopes of computation. Then there exists a complementary condition $\mathbf{A}$ such that

$${\mathbf{C}}_{\mathbf{1}}\wedge \mathbf{A}\Rightarrow {\mathbf{C}}_{\mathbf{2}}.$$

(33)

5.2.2. In Parallel MARL

Conjecture 3 is formulated from the perspective of serial processes, where algorithm components operate sequentially. This raises a question: does this hierarchical relationship between sufficient conditions also apply to parallel processes commonly found in modern MARL architectures?

Consider the actor-critic model [7,25], where the actor network focuses on policy updates, while the critic network learns to estimate value functions or similar functions. Both components take the state as input, but the actor requires additional feedback from the critic’s value estimation. This creates a parallel yet interdependent computational structure that differs fundamentally from our serial policy iteration framework.

In such parallel architectures, we hypothesize that sufficient conditions might exhibit more complex relationships than simple hierarchical implications. The complementary condition $\mathbf{A}$ in Conjecture 3 may need to account for synchronization between parallel processes. The relationship between ${\mathbf{C}}_{\mathbf{1}}$ and ${\mathbf{C}}_{\mathbf{2}}$ may involve dependencies rather than a nested scope. Hence, we conjecture that the hierarchical relationship can be extended to parallel processes as follows:

Conjecture 4.

Let ${\mathbf{C}}_{\mathbf{1}}$ and ${\mathbf{C}}_{\mathbf{2}}$ be sufficient conditions for convergence in parallel components of a MARL algorithm, where ${\mathbf{C}}_{\mathbf{2}}$ is parallelly dependent on ${\mathbf{C}}_{\mathbf{1}}$. Then, there exists a synchronization condition $\mathbf{S}$, ensuring proper coordination between the parallel components, and a complementary condition $\mathbf{A}$, such that

$${\mathbf{C}}_{\mathbf{1}}\wedge \mathbf{A}\wedge \mathbf{S}\Rightarrow {\mathbf{C}}_{\mathbf{2}}.$$

(34)

Recent advances in decentralized multi-agent actor-critic models demonstrate the importance of agent synchronization within the communication network as a fundamental assumption for convergence guarantees [26], highlighting the critical role of coordination mechanisms in parallel multi-agent systems. Hence, this extended conjecture opens several promising directions for formal verification of sufficient synchronization conditions. Examples include identifying the minimal amount of information required for communication, understanding the structure of information exchange, and characterizing process dependencies that affect convergence.

6. Discussion

6.1. Connections of MBRSS to Existing Frameworks

We clarify that MBRSS is not a reformulation of existing set-based convergence notions, but operates on a different axis. The following two comparisons make this precise.

6.1.1. Iterated Elimination of Dominated Strategies

Iterated elimination of dominated strategies (IEDS) is a classical approach to finding equilibria by iteratively removing dominated strategies, which are strategies that always result in a worse outcome for an agent compared to another available strategy, regardless of what the other agents do. This is based on the idea that rational agents are expected to avoid dominated strategies, as there is always a better alternative, making such strategies unlikely to be played. In this sense, each iteration of elimination shrinks the feasible joint policy space.

More precisely, IEDS removes strategies that are dominated with respect to all possible opponent strategies across the entire joint action space, independent of the learning trajectory. MBRSS, by contrast, tracks the shrinkage of each agent’s best-response set relative to the current opponent policy profile at each iteration. A strategy that survives all rounds of IEDS may still be pruned from the best-response space under MBRSS at a later iteration, and conversely, a strategy eliminated by IEDS could remain in an early MBRSS best-response set if it happens to be a best response to the current profile. The two processes therefore operate on orthogonal axes: IEDS is a static equilibrium-refinement procedure, while MBRSS is a dynamic convergence condition on the algorithm’s trajectory.

6.1.2. Absorbing Best-Response Graphs

A related treatment of convergence within the best-response space appears in [18] (called the best-reply set in [18]). Their work proposes a decentralized MARL algorithm for stochastic games with general state spaces, where each agent learns an estimated best-response space by running quantized Q-learning during exploration phases. The global policy-updating dynamics is then modeled as a Markov chain over the joint policy space, and convergence to equilibrium is characterized in closed form via the theory of absorbing Markov chains. Their framework shares a common object with MBRSS: both operate on a best-response space that collects all policies a rational agent might adopt given the current behavior of others. However, the two frameworks use this object in structurally different ways.

In [18], the best-response space enters as a static graph structure that encodes which joint policies are reachable from which other joint policies under one round of best-reply updates. Convergence is therefore a probabilistic property of how agents sample from their best-response spaces. In particular, when multiple best responses exist, an agent may sample among them at random, and the probability of reaching any given equilibrium is derived from the canonical form of the absorbing Markov chain.

Instead, MBRSS requires that the best-response space itself contracts monotonically across iterations. Convergence under MBRSS is therefore deterministic. There is no role for a sampling distribution, and no need to characterize absorption probabilities, because shrinkage of the space itself guarantees the outcome. Consequently, the two frameworks are complementary: ref. [18] asks with what probability does a best-response-based dynamics reach equilibrium within a induced best-response graph, while MBRSS asks under what set-contraction condition does the algorithm necessarily converge.

6.2. Future Directions

Our analysis of MBRSS in repeated games opens several promising avenues for further research. We outline three directions that extend both the theoretical scope and the practical implications of this sufficient condition.

6.2.1. Finding Game Classes and Algorithms That Satisfy MBRSS

Introduced by [27], the ratio-bounded game is a class of two-player non-cooperative games in which the strategy sets are Hilbert spaces and the defining parameters are derived from the game itself. Within this class, three of the four subclasses (distinguished by parameter configurations) admit a unique Nash equilibrium and can be solved via an affine-relaxed best-response algorithm. A key property enabling this result is that the best-response correspondence is single-valued.

In contrast, MBRSS relaxes this requirement and assumes only non-empty correspondence, due to $\stackrel{I_1}{\to }$. As a consequence, the class of games and algorithms that satisfy MBRSS is fundamentally different, raising natural questions about their characterization and properties. Identifying such games therefore becomes an important future direction of study. At minimum, there exists at least one class of games that satisfies MBRSS, providing a starting point for investigation.

A natural first step is to examine the implications of this class by considering simple canonical examples, beginning with pure coordination games (as discussed in Section 4.3). From there, one can seek to identify broader classes of games that meet the MBRSS criteria. By doing so, we can deepen our understanding of the dynamics at play and potentially uncover new strategies that could be applied in various game-theoretic contexts.

6.2.2. A Possible Application in Search Problem

The structural properties of MBRSS suggest a natural connection to search problems in which the feasible solution space can be reduced iteratively. Specifically, the monotone contraction of each agent’s best-response set in Theorem 2 resembles a set-valued pruning process, that stabilizes at ${\Delta }^{*}$ (Definition 13). This dynamic is analogous to the progressive narrowing of a search space in constraint-satisfaction, branch-and-bound, and version-space learning frameworks, where feasibility conditions are applied iteratively to eliminate candidate solutions.

More precisely, the stability condition in Definition 13 ensures that the search has effectively terminated when no further pruning is possible or necessary. This mirrors the role of a feasibility check in combinatorial search, where a node is pruned precisely when no extension of the current partial solution can yield an improvement.

This analogy motivates a conjecture that MBRSS-type conditions could serve as a principled basis for search space reduction in multi-agent settings. Consider, for example, a cooperative task-assignment problem in which agents must jointly select actions from a large combinatorial space. If the agents’ best-response spaces satisfy MBRSS, then the effective search space contracts monotonically across iterations, and the algorithm is guaranteed to concentrate its computation within a progressively smaller region of policy space. This property would be practically valuable in settings where exhaustive search is intractable and heuristic pruning lacks formal guarantees.

A formal development of this connection would require translating MBRSS from the continuous policy simplex into discrete or mixed search domains, characterizing the rate of contraction as a function of game structure, and identifying the classes of search problems for which the stability condition in stability in best-response space (Definition 13) can be verified efficiently (when Conjecture 2 is true, or more effective approximation approach). We leave these questions as directions for future work, noting that the single-point-of-failure methodology introduced here may offer a useful lens for analyzing when and why search procedures succeed or fail to concentrate on high-quality solutions.

6.2.3. Sufficient Condition Taxonomy

Conjectures 3 and 4 point toward a broader investigation into the landscape of sufficient conditions for MARL convergence, suggesting future work on classifying and unifying such conditions. As noted in Section 2.1, existing sufficient conditions suggest several different directions for improvement. A systematic taxonomy of these conditions organized by properties, such as static versus dynamic, serial versus parallel, or local versus global, could clarify their interrelationships and relative strengths. Such a taxonomy would help us understand the connections between various MARL categories, identify missing links, and inspire new conditions that may lead to novel MARL models. Since sufficient conditions indirectly indicate how corresponding MARL algorithms work, it would be desirable to identify appropriate algorithms based on the characteristics of the game and the desired solution concept.

7. Conclusions

We have studied convergence in multi-agent reinforcement learning through the lens of sufficient conditions, using MAPI-LP in repeated games as a concrete framework for analysis. By adopting a single-point-of-failure perspective, we identified two sufficient conditions for monotonic convergence to MPE. The first is stability in best-response space (Definition 13). The second, monotonic best-response space shrinking (Theorem 1), is a novel set-valued condition requiring that each agent’s best-response set contracts monotonically across iterations until collapsing to a stable best-response space. In contrast, value monotonicity establishes convergence in value space but does not guarantee convergence to an MPE. And, we further proved that sequential MAPI-LP satisfies MBRSS in pure coordination games, converging to a stable best-response space, where all of its members are MPE, in at most n iterations (Theorem 3).

Building on these results, we proposed four conjectures: the first two are on MBRSS and stability in best-response space, while the others point to relationships between sufficient conditions within the same system (Section 5). Conjecture 1 asks whether generalized best-response dynamics satisfying MBRSS exist for arbitrary finite repeated games, motivated by the restriction of Theorem 3 to pure coordination games. Conjecture 2 proposes a more effective way to check stability in best-response space, instead of exhaustively checking every iteration infinitely. Conjecture 3 extends the sequential relationship, inspired by Corollaries 4 and 5. Conjecture 4 provisions the parallel relationship between components.

We also discussed connections between MBRSS and classical frameworks, including iterated elimination of dominated strategies and absorbing best-response graphs, highlighting both the similarities and the structural distinctions that make MBRSS a complementary perspective on convergence.

Together, these results offer a theoretical foundation and a structured roadmap for understanding convergence in multi-agent settings. The most immediate open problems are the proof or disproof of Conjectures 1 and 2 for the extension and the application of MBRSS in other game classes, from other repeated games to stochastic games where the policy space becomes high-dimensional and coupled through transition dynamics. We would like to develop a taxonomy of sufficient conditions that systematically organizes existing and future conditions by their scope, structural assumptions, and relationships to one another. We hope that the single-point-of-failure methodology introduced here, and the sufficient conditions and conjectures yielded in this analysis, provide a useful starting point for theoretical investigations in the future.