LLM-TOC: LLM-Driven Theory-of-Mind Adversarial Curriculum for Multi-Agent Generalization

Abstract

Zero-shot generalization to out-of-distribution (OOD) teammates and opponents in multi-agent systems (MASs) remains a fundamental challenge for general-purpose AI, especially in open-ended interaction scenarios. Existing multi-agent reinforcement learning (MARL) paradigms, such as self-play and population-based training, often collapse to a limited subset of Nash equilibria, leaving agents brittle when faced with semantically diverse, unseen behaviors. Recent approaches that invoke Large Language Models (LLMs) at run time can improve adaptability but introduce substantial latency and can become less reliable as task horizons grow; in contrast, LLM-assisted reward-shaping methods remain constrained by the inefficiency of the inner reinforcement-learning loop. To address these limitations, we propose LLM-TOC (LLM-Driven Theory-of-Mind Adversarial Curriculum), which casts generalization as a bi-level Stackelberg game: in the inner loop, a MARL agent (the follower) minimizes regret against a fixed population, while in the outer loop, an LLM serves as a semantic oracle that generates executable adversarial or cooperative strategies in a Turing-complete code space to maximize the agent’s regret. To cope with the absence of gradients in discrete code generation, we introduce Gradient Saliency Feedback, which transforms pixel-level value fluctuations into semantically meaningful causal cues to steer the LLM toward targeted strategy synthesis. We further provide motivating theoretical analysis via the PAC-Bayes framework, showing that LLM-TOC converges at rate $O(1/\sqrt{K})$ and yields a tighter generalization error bound than parameter-space exploration under reasonable preconditions. Experiments on the Melting Pot benchmark demonstrate that, with expected cumulative collective return as the core zero-shot generalization metric, LLM-TOC consistently outperforms self-play baselines (IPPO and MAPPO) and the LLM-inference method Hypothetical Minds across all held-out test scenarios, reaching 75% to 85% of the upper-bound performance of Oracle PPO. Meanwhile, with the number of RL environment interaction steps to reach the target relative performance as the core efficiency metric, our framework reduces the total training computational cost by more than 60% compared with mainstream baselines.

1. Introduction

Developing autonomous agents capable of seamless interaction with diverse, unseen counterparts in open-ended multi-agent systems remains a core challenge for general-purpose AI [1,2]. While Multi-Agent Reinforcement Learning (MARL) has achieved superhuman performance in closed zero-sum games via the self-play (SP) paradigm [3,4], these methods typically converge to narrow Nash equilibrium subsets when co-evolving with fixed training partners [5]. This leads to catastrophic generalization failure in mixed-motive environments like the Melting Pot benchmark [6,7], where agents face Out-of-Distribution (OOD) teammates/opponents with semantically distinct behaviors (Figure 1) [8]. This well-documented Zero-Shot Coordination (ZSC) gap [9] motivates our work to develop training methodologies that foster robust, generalizable social intelligence.

Existing population-based methods, such as Fictitious Co-Play (FCP) [10] and population-based training (PBT) [11], seek to mitigate this gap by expanding training population diversity. However, these methods operate exclusively in the low-level parameter space of neural networks, yielding only superficial diversity rather than semantically meaningful strategies [12]. As a result, agents trained via these methods still fail to generalize to complex, unseen social behaviors at test time.

Large Language Models (LLMs), with their embedded human social prior knowledge and strong reasoning capabilities [13,14], offer a new path to inject semantic diversity into MARL. Existing LLM-based MARL works fall into two paradigms, both with critical limitations: (1) LLM-as-Agent: Methods like Hypothetical Minds [15] and ProAgent [16] integrate LLMs into the real-time decision loop for online ToM inference, but suffer from prohibitive inference latency and high computational costs, making them unsuitable for high-frequency control tasks. (2) LLM-Assisted Reward Shaping: Methods like SEMDIV [17] use LLMs to generate reward functions offline but require a computationally expensive RL inner loop to train new policies from scratch for each generated reward, leading to severe efficiency bottlenecks and training instability [18].

To address these limitations, we propose LLM-Driven Theory-of-Mind Adversarial Curriculum (LLM-TOC), a novel framework that redefines the role of LLMs in MARL training. Unlike existing approaches, LLM-TOC positions the LLM as an offline semantic oracle that directly generates executable rule-based policies in Python code, eliminating both online inference latency and the repeated RL inner loop bottleneck. We formalize the zero-shot generalization problem as a bi-level Stackelberg game: the outer-loop LLM generates a code-based adversarial population to maximize the student agent’s regret, while the inner-loop lightweight MARL student agent optimizes its best-response policy to minimize this regret. We validate the effectiveness of this framework through extensive experiments on the Melting Pot benchmark, a widely used testbed for open-ended multi-agent social intelligence.

A core challenge of this paradigm is the modality gap: LLMs excel at textual semantic reasoning but cannot natively interpret numerical RL signals. To bridge this gap and close the training loop, we propose a novel Gradient Saliency Feedback mechanism, which transforms pixel-level value function fluctuations into semantically meaningful causal descriptions of agent failure. This provides the LLM with explicit, targeted grounding to generate high-value adversarial strategies, without hand-crafted prompts or manual feature engineering. Our core contributions are summarized as follows:

1.

We propose the LLM-TOC framework, a novel bi-level training paradigm that leverages the code generation capabilities of LLMs to construct an infinitely scalable and semantically diverse adversarial curriculum for MARL. Diverging from existing LLM-assisted MARL methods, this framework directly generates executable rule-based policies in a Turing-complete code space, effectively circumventing the computational inefficiency and instability caused by the repeated RL inner loop training in traditional reward-shaping methods.

2.

We introduce the Gradient Saliency Feedback mechanism, which bridges the modality gap between numerical RL signals and LLM semantic reasoning. This mechanism transforms pixel-level value fluctuations into semantically meaningful causal cues, enabling the LLM to perform targeted semantic gradient ascent in the abstract policy space without relying on hand-crafted prompts or manual feature engineering.

3.

We provide motivating theoretical analysis for the proposed framework via the PAC-Bayes framework. We formally prove the convergence rate of LLM-TOC to a robust equilibrium at $O(1/\sqrt{K})$, and present a qualitative analysis, showing that our method yields a tighter generalization error bound than conventional parameter-space exploration methods under reasonable preconditions, providing a theoretical rationale for the superior zero-shot robustness of our approach.

We further validate the above innovations through extensive empirical evaluations on the Melting Pot benchmark, a representative testbed for open-ended multi-agent zero-shot generalization. The results show that LLM-TOC achieves state-of-the-art zero-shot generalization performance against unseen OOD partners in this benchmark, while reducing the total number of RL environment interaction steps required to reach the target performance by more than $60\%$ compared with mainstream baselines within the same experimental settings.

2. Related Work

2.1. Zero-Shot Coordination and Generalization in MARL

Zero-Shot Coordination (ZSC) targets agents that can effectively collaborate with unseen partners without prior data or fine-tuning [9,19]. Classic MARL baselines, including Independent Proximal Policy Optimization (IPPO) [20], Multi-Agent Proximal Policy Optimization (MAPPO) [21], and the value-decomposition representative QMIX [22], mostly rely on the self-play (SP) paradigm to optimize joint returns in closed domains like StarCraft II [3]. While effective in fixed settings, these methods converge to arbitrary coordination conventions, leading to severe behavioral fragility when paired with out-of-distribution partners [8]. Early improvements like Other-Play (OP) [23] reduce reliance on arbitrary coordination signals via symmetry invariance but fail to scale to complex asymmetric environments.

Population-based training has become the dominant paradigm to improve agent robustness against diverse behaviors. Fictitious Co-Play (FCP) [10], which trains agents against checkpoints saved during SP, is a strong robustness baseline, yet its diversity is strictly limited by the training trajectory of a single optimization algorithm. Recent works explicitly maximize population diversity via targeted designs: Trajectory Diversity (TrajeDi) [24] and Latent Space Optimization (LIPO) [25] induce differentiated behaviors via Jensen–Shannon divergence or latent variables, while EvoAgent [26] generates diverse strategies via evolutionary algorithms. Despite these advances, most existing methods operate in low-level parameter or trajectory spaces, yielding only stochastic variations rather than high-level semantic strategies (e.g., deception, sacrifice) critical for human-level coordination [27,28]. In contrast, our LLM-TOC framework leverages LLM code generation to explore a Turing-complete semantic space, uncovering complex interaction strategies inaccessible to traditional parameter-space exploration.

2.2. Large Language Models for Autonomous Agents

Driven by strong reasoning and planning capabilities, the integration of Large Language Models (LLMs) with autonomous agents has grown rapidly [1,2], falling into two primary paradigms: LLM-as-Agent and LLM-Assisted Training.

The LLM-as-Agent paradigm integrates LLMs into the agent’s real-time decision loop. Works like Voyager [29] and Ghost in the Minecraft (GITM) [30] enable continuous skill acquisition in open-ended environments via code generation; MetaGPT [31] and AgentVerse [32] explore role-based collaboration for software development. Most relevant to our work is Hypothetical Minds [15], which uses LLMs for real-time Theory of Mind (ToM) inference and response planning. While validated on the Melting Pot benchmark [6] via hand-crafted visual-to-text parsers, it suffers from prohibitive inference latency, high token costs, and limited deployability in high-frequency visual control tasks without manual engineering.

The LLM-Assisted Training paradigm uses LLMs to support lightweight RL policy training. Eureka [33] and Text2Reward [18] synthesize dense reward functions to guide RL training; SEMDIV [17] generates diverse partner agents via reward shaping but relies on a computationally expensive RL inner loop that requires training a new policy from scratch for each generated reward. Distinct from these methods, LLM-TOC positions the LLM as an offline semantic oracle that directly generates executable policy code. This design combines the semantic reasoning capabilities of Hypothetical Minds with the execution efficiency of lightweight MARL agents (e.g., MAPPO), eliminating real-time inference latency. Our experiments on the Melting Pot benchmark verify its strong zero-shot generalization against semantically diverse opponents in this testbed [13,34].

2.3. Automated Curriculum Learning and Environment Design

Automatic Curriculum Learning (ACL) and Unsupervised Environment Design (UED) aim to generate training task sequences that maximize agent learning efficiency [35,36]. PAIRED [37] uses an adversary to generate challenging yet solvable training environments; ACCEL [38] improves this via Quality-Diversity (QD) algorithms like MAP-Elites [39] to evolve more robust environments; Prioritized Level Replay (PLR) [40] replays high-regret levels to enhance agent robustness.

For social generalization, the “environment” includes both physical scenarios and interactive counterparts. Open-Ended Learning (OEL) studies emphasize that sustained learning requires a continuous stream of novel challenges [41,42], yet most existing ACL methods only generate environmental features (e.g., grid layouts and physical parameters) rather than interactive policies with distinct behavioral patterns. Our work incorporates a Gradient Saliency Feedback mechanism [43,44] to align curriculum generation with the student agent’s failure modes. This design enables the LLM to perform semantic gradient ascent, generating targeted adversarial interaction scenarios (e.g., rear sneak attacks) to efficiently expose and fix the student’s performance deficits. Compared to unguided evolutionary search methods [45,46,47], our empirical results on the Melting Pot benchmark show significant improvements to the agent’s training convergence efficiency within the tested multi-agent scenarios.

2.4. Stackelberg Games in MARL

The Stackelberg game, a hierarchical leader–follower model with the Stackelberg equilibrium as its solution [48], is widely used in MARL to model adversarial training, curriculum learning, and opponent exploitation [49]. Early studies focused on static equilibrium in discrete action spaces for security and zero-sum games [50], while recent works extend it to deep MARL for adversarial environment design and agent robustness.

However, existing Stackelberg-based MARL methods are limited to the continuous parameter space of neural networks [51], producing only superficial strategy diversity [52]. In contrast, we are the first to model multi-agent zero-shot generalization as a Stackelberg game in the Turing-complete semantic code space. We employ an LLM as the leader to generate executable adversarial strategies with a Gradient Saliency Feedback mechanism. Our experiments on the Melting Pot benchmark verify that this design greatly expands the semantic coverage of the training population and alleviates mode collapse in traditional parameter-space methods for multi-agent zero-shot generalization tasks.

3. Problem Formulation

3.1. Partially Observable Markov Games

We formalize open-ended multi-agent interaction as a Partially Observable Markov Game (POMG) [53,54], defined by the tuple $\mathcal{G}=\langle \mathcal{S},N,\mathcal{A},\mathcal{P},\mathcal{R},\Omega ,\mathcal{O},\gamma \rangle$. Here, $\mathcal{S}$ denotes the global state space, and $N=\{1,\dots ,n\}$ represents the set of agents. We partition N into the student agent (E), which acts as the focal agent, and the other agents ($-E$), comprising potential teammates or opponents. The joint action space is defined as $\mathcal{A}={\mathcal{A}}_E\times {\mathcal{A}}_{-E}$, while the state transition dynamics are governed by $\mathcal{P}\left(s^{\prime }\right|s,a_E,a_{-E})$. The term ${\mathcal{R}}_E(s,a_E,a_{-E})$ specifies the reward function for the student agent. Additionally, ${\Omega }_i$ denotes the set of partial observations for agent i, generated by the observation function ${\mathcal{O}}_i\left(s\right)$, and $\gamma \in [0, 1)$ represents the discount factor.

The student agent operates according to a policy ${\pi }_E\left(a_E\right|o_E;\theta )$, parameterized by $\theta$, such as the weights of a neural network. Conversely, the other agents follow a joint policy ${\pi }_{-E}\left(a_{-E}\right|o_{-E})$, which may be driven by diverse or unknown logic. The expected return for the student agent against a specific counterpart ${\pi }_{-E}$ is defined as:

$$J({\pi }_E,{\pi }_{-E})={\mathbb{E}}_{\tau \sim ({\pi }_E,{\pi }_{-E})}\left[\sum _{t=0}^{\infty }{\gamma }^t{\mathcal{R}}_E(s_t,a_{E,t},a_{-E,t})\right].$$

(1)

3.2. Zero-Shot Generalization as Minimax Regret

A central challenge within open-ended environments such as Melting Pot [6] is zero-shot generalization. The student agent ${\pi }_E$ is required to demonstrate robust performance when paired with test-time counterparts ${\pi }_{-E}^{test}$ sampled from an unknown distribution ${\mathcal{D}}_{test}$, without the opportunity for fine-tuning. Conventional MARL approaches typically assume that ${\pi }_{-E}$ originates from a restricted set of behaviors, such as checkpoints derived from self-play. However, in realistic scenarios, ${\pi }_{-E}$ resides within a vast semantic strategy space, denoted as ${\Pi }_{sem}$. This space encompasses all behaviorally distinct policies expressible through logical rules, code, or natural language, including strategies such as tit-for-tat, deceptive cooperation, or aggressive blocking. Notably, this space is significantly larger and exhibits a discrete structure distinct from the continuous parameter space of neural networks.

To achieve robust generalization, we formulate the training objective as the minimization of maximum regret over the entire semantic strategy space ${\Pi }_{sem}$. We first define the regret of a student policy ${\pi }_E$ against a specific counterpart ${\pi }_{-E}$ as the difference between the maximal achievable return against ${\pi }_{-E}$ and the actual return:

$$\mathrm{Regret}({\pi }_E,{\pi }_{-E})=\underset{V^{*}\left({\pi }_{-E}\right)}{\underbrace{\underset{{\pi }^{\prime }}{max}J({\pi }^{\prime },{\pi }_{-E})}}-J({\pi }_E,{\pi }_{-E}),$$

(2)

where $V^{*}\left({\pi }_{-E}\right)$ represents the oracle performance achievable assuming perfect prior knowledge of ${\pi }_{-E}$. Our objective is to identify an optimal student policy ${\pi }_E^{*}$ that minimizes the worst-case regret across all possible semantic strategies:

$${\pi }_E^{*}=arg\underset{{\pi }_E}{min}\left(\underset{{\pi }_{-E}\in {\Pi }_{sem}}{max}\mathrm{Regret}({\pi }_E,{\pi }_{-E})\right).$$

(3)

3.3. The Challenge of Semantic Coverage

Directly solving the optimization problem presented above is intractable due to two primary factors: Infinite Space: The semantic space ${\Pi }_{sem}$ is effectively infinite and non-differentiable, rendering traversal via standard gradient-based optimization or simple population sampling, such as FCP, infeasible. Modality Gap: Conventional RL methods operate within the parameter space $\Theta$. Exploration of $\Theta$ through Gaussian noise—a technique commonly employed in PBT—rarely yields high-level semantic strategies, such as complex deception. Consequently, such methods fail to adequately cover the support of ${\Pi }_{sem}$.

Therefore, an efficient mechanism is required to identify the worst-case ${\pi }_{-E}$ within ${\Pi }_{sem}$ to serve as a curriculum. This necessity motivates our proposed LLM-TOC framework, which leverages LLMs as a semantic engine to generate ${\pi }_{-E}$ directly within the code space, thereby approximating the ${max}_{{\pi }_{-E}}$ operator in a computationally feasible manner. In Appendix A, we list all the symbols used in this paper and provide their definitions.

4. Methodology

Building upon the established problem formulation, we propose LLM-TOC, a framework designed to approximate the intractable minimax regret objective through a bi-level iterative process. This methodology comprises two coupled loops, as illustrated in Figure 2: an outer loop wherein a Large Language Model functions as a semantic oracle to generate adversarial policies ${\pi }_{-E}$ directly within the code space, and an inner loop in which the student agent ${\pi }_E$ performs robust optimization against the generated population. To facilitate this interaction, we introduce a Gradient Saliency Feedback mechanism that effectively bridges the modality gap between numerical reinforcement learning signals and semantic reasoning.

4.1. The Bi-Level Optimization Framework

We formally structure the training process as a Stackelberg game, modeled as a bi-level optimization problem. In this hierarchy, the leader initiates the interaction by selecting a distribution of opponent strategies designed to exploit the weaknesses of the follower. In response, the follower optimizes its policy against this distribution. Let ${\pi }_E\in {\Pi }_{net}$ denote the student policy parameterized by $\theta$, and $\mathcal{P}\subset {\Pi }_{sem}$ represent the population of strategies for the other agents.

We first formally define the Stackelberg game mapping of our LLM-TOC framework, which strictly aligns with the standard leader–follower Stackelberg game model.

In our framework, the two players of the Stackelberg game are explicitly defined as follows. Leader: The Large Language Model acting as the semantic oracle, which is the first-mover in the game. Follower: The student MARL agent, which optimizes its policy as the best response to the leader’s action.

The strategy spaces of the two players are strictly defined, corresponding to the two nested loops of our framework. Leader’s Strategy Space: The semantic strategy space ${\Pi }_{sem}$, which consists of all executable rule-based policies expressed via Python code. Each element in this space is a complete policy class that can be directly injected into the multi-agent environment to interact with the student agent. Follower’s Strategy Space: The neural network parameterized policy space ${\Pi }_{net}$, where each element is a policy ${\pi }_E\left(a_E\right|o_E;\theta )$ parameterized by the neural network weights $\theta$.

The payoff functions of the two players are defined as zero-sum in the worst-case regret minimization problem, which aligns with our generalization objective. Leader’s Payoff: The regret of the student agent, defined as $Regret({\pi }_E,{\pi }_{-E})=V^{*}\left({\pi }_{-E}\right)-J({\pi }_E,{\pi }_{-E})$. The leader’s objective is to maximize this regret by generating adversarial policies ${\pi }_{-E}\in {\Pi }_{sem}$. Follower’s Payoff: The negative of the student’s worst-case regret, $-{max}_{{\pi }_{-E}\in \mathcal{P}}Regret({\pi }_E,{\pi }_{-E}),$ where $\mathcal{P}$ is the policy population generated by the leader. The follower’s objective is to maximize this payoff by optimizing its policy parameters $\theta$.

The goal of our framework is to find the Stackelberg equilibrium $({\pi }_{-E}^{*},{\pi }_E^{*})$ of this game, where the following hold. The follower’s policy ${\pi }_E^{*}$ is the best response to the leader’s policy population: ${\pi }_E^{*}=\underset{{\pi }_E\in {\Pi }_{net}}{argmin}{max}_{{\pi }_{-E}\in \mathcal{P}}Regret({\pi }_E,{\pi }_{-E})$. The leader’s policy ${\pi }_{-E}^{*}$ is the optimal adversarial strategy that maximizes the follower’s regret under the constraint that the follower plays its best response:

$${\pi }_{-E}^{*}=\underset{{\pi }_{-E}\in {\Pi }_{sem}}{argmax}Regret\left({\pi }_E^{*}\left({\pi }_{-E}\right),{\pi }_{-E}\right).$$

(4)

Based on this formal Stackelberg game mapping, we define the bi-level optimization objectives of our framework as follows:

The Follower Objective (Inner Loop): Given a fixed population ${\mathcal{P}}_k$ provided by the leader at iteration k, the student seeks to maximize its expected return. This corresponds to the standard MARL objective:

$${\pi }_E^{*}\left({\mathcal{P}}_k\right)=arg\underset{{\pi }_E\in {\Pi }_{net}}{max}{\mathcal{J}}_{student}({\pi }_E,{\mathcal{P}}_k)=arg\underset{{\pi }_E}{max}{\mathbb{E}}_{{\pi }_{-E}\sim U\left({\mathcal{P}}_k\right)}\left[{\mathbb{E}}_{\tau \sim ({\pi }_E,{\pi }_{-E})}\left[\sum _{t=0}^T{\gamma }^t{r}_t\right]\right].$$

(5)

This step aims to identify the best response strategy ${\pi }_E^{*}$ that exhibits robustness to the current set of counterparts.

The Leader Objective (Outer Loop): The leader aims to expand the population ${\mathcal{P}}_k$ by introducing a new strategy ${\pi }_{-E}^{(k+1)}$ that maximizes the regret of the student. As calculating exact regret requires an optimal oracle $V^{*}$, we approximate this objective by maximizing the exploitability of the student—specifically, by identifying a strategy that minimizes the performance of the current student policy ${\pi }_E^{*}\left({\mathcal{P}}_k\right)$:

$${\pi }_{-E}^{(k+1)}=arg\underset{{\pi }_{-E}\in {\Pi }_{sem}}{min}{\mathbb{E}}_{\tau \sim ({\pi }_E^{*},{\pi }_{-E})}\left[\sum _{t=0}^T{\gamma }^t{r}_t\right].$$

(6)

However, as the semantic space ${\Pi }_{sem}$ is discrete and non-differentiable, optimization of the leader’s objective via gradient descent is infeasible. Consequently, we employ the LLM as a semantic oracle $\Phi$ to approximate this process:

$${\pi }_{-E}^{(k+1)}\approx \Phi ({\pi }_E^{*}\left({\mathcal{P}}_k\right)\mid \mathrm{Feedback}).$$

(7)

This bi-level architecture ensures that the student is continuously challenged by worst-case scenarios, thereby driving the expansion of its robust hull within the strategy space.

4.2. Gradient Saliency Feedback Mechanism

To effectively guide the semantic oracle $\Phi$ within the outer loop, bridging the modality gap between numerical reinforcement learning losses and the semantic reasoning capabilities of LLMs is essential. To this end, we introduce a Gradient Saliency Feedback mechanism designed to perform causal attribution.

4.2.1. Identifying Critical Moments via Value Surprise

We first identify specific instances of policy failure by monitoring the value function $V(s;\varphi )$ of the student agent ${\pi }_E$ during evaluation episodes. We define the value surprise, denoted as ${\delta }_t$, as the absolute Temporal Difference (TD) error, which quantifies the discrepancy between the expected return of the agent and the actual outcome:

$${\delta }_t=\left|r_t+\gamma V(s_{t+1};\varphi )-V(s_t;\varphi )\right|.$$

(8)

A high magnitude of ${\delta }_t$ signifies a critical frame $t^{*}$, indicating a pivotal event such as an unexpected penalty, an adverse interaction, or a missed reward. We extract a set of critical frames ${\mathcal{T}}_{crit}=\{t\mid {\delta }_t>{ϵ}_{thresh}\}$ for subsequent detailed analysis.

4.2.2. Visual Attribution via Jacobian Saliency

For each critical frame $t^{*}\in {\mathcal{T}}_{crit}$, we elucidate the cause of the value deviation by computing the sensitivity of the value function with respect to the input observation $X_{t^{*}}\in {\mathbb{R}}^{C\times H\times W}$. This is achieved by calculating the Jacobian matrix of the value function relative to the input pixels.

Specifically, let $V\left(s_{t^{*}}\right)$ represent the scalar value output. We compute the gradient map ${\mathcal{G}}_{t^{*}}\in {\mathbb{R}}^{C\times H\times W}$ via backpropagation:

$${\mathcal{G}}_{t^{*}}={\nabla }_{X_{t^{*}}}V(s_{t^{*}};\varphi )={\displaystyle \frac{\partial V\left(s_{t^{*}}\right)}{\partial X_{t^{*}}}}.$$

(9)

To obtain a single 2D saliency map $M_{t^{*}}\in {\mathbb{R}}^{H\times W}$, we aggregate the gradients across the channel dimension—comprising RGB or feature channels—by computing the maximum absolute value or the $L_2$ norm:

$$M_{t^{*}}^{(h,w)}=\underset{c}{max}\left|{\mathcal{G}}_{t^{*}}^{(c,h,w)}\right|.$$

(10)

This map $M_{t^{*}}$ highlights specific regions in the visual field where perturbations would induce the most significant shifts in the value estimation of the agent. Mathematically, this approximates the first-order Taylor expansion of the value function, identifying the features to which the agent attends—or fails to attend—during the critical event.

4.2.3. Semantic Mapping

We map the pixel-level saliency M to semantic concepts using ground-truth object masks provided by the environment engine. Let ${\mathbb{I}}_{obj}$ denote the binary mask for a specific object class $obj$, such as counterparts ${\pi }_{-E}$ or resources. The attention score for each object is calculated as follows:

$${\mathrm{Score}}_{obj}={\displaystyle \frac{{\sum }_{i,j}{M}_{i,j}·{\mathbb{I}}_{obj,i,j}}{{\sum }_{i,j}{M}_{i,j}+ϵ}}.$$

(11)

By comparing these scores, we generate semantic descriptions. For instance, if ${\mathrm{Score}}_{-E}\gg {\mathrm{Score}}_{goal}$ during a significant value drop, the system generates a description indicating that the focal agent focused heavily on the opponent while neglecting the goal, thereby leading to failure.

4.3. Policy Generation and Curriculum Evolution

Central to our curriculum evolution is the transformation of semantic feedback into executable policy code. We formalize this process as a conditional generation problem within the semantic space.

Let ${\mathcal{D}}_{sem}^{\left(k\right)}$ denote the semantic diagnosis derived from the gradient saliency analysis at iteration k. We construct a structured prompt ${\mathbf{P}}_k$ that encapsulates the designated role, API constraints, and the diagnosis

$${\mathbf{P}}_k=\mathrm{Concat}({\mathcal{I}}_{role},{\mathcal{I}}_{API},{\mathcal{D}}_{sem}^{\left(k\right)}),$$

(12)

where ${\mathcal{I}}_{role}$ defines the adversarial objective and ${\mathcal{I}}_{API}$ specifies the programming interface.

The LLM functions as a generator, sampling new policy code ${\pi }_{code}^{(k+1)}$ from its learned distribution:

$${\pi }_{code}^{(k+1)}\sim P_{LLM}(·\mid {\mathbf{P}}_k).$$

(13)

This generation process is conceptualized as a semantic gradient ascent. The diagnosis ${\mathcal{D}}_{sem}^{\left(k\right)}$ serves as a gradient direction within the semantic manifold, pointing toward the region of the strategy space where the student agent ${\pi }_E$ exhibits maximum vulnerability. For example, if the diagnosis indicates a susceptibility to rear attacks, the LLM generates logic specifically targeting the blind spot of the focal agent.

To clarify the prompt iteration mechanism and the full translation pipeline from semantic feedback to executable policies, we have:

• Prompt Regeneration Rule: The structured prompt $P_k$ is regenerated from scratch at each outer-loop iteration k. The fixed components (role instruction ${\mathcal{I}}_{role}$ and API constraints ${\mathcal{I}}_{API}$) are reused across all iterations to ensure the consistency of generation format and objective, while the core semantic diagnosis component $D_{sem}^{\left(k\right)}$ is completely updated based on the student agent’s latest failure modes in the current iteration, with no cross-iteration reuse of diagnosis content. This ensures that the generated adversarial policies are always targeted at the student’s current weaknesses, rather than outdated historical vulnerabilities.
• Closed-Loop Translation Pipeline: The semantic feedback derived from gradient saliency is translated into executable policies through three sequential steps: (i) the pixel-level saliency map is mapped to object-level attention scores via Equation (11); (ii) the attention scores are converted into a natural language causal diagnosis describing the student’s failure mode; (iii) the diagnosis is embedded into the prompt, and the LLM generates Python policy code that directly exploits the identified vulnerability, with no manual intervention in the entire process. It should be clarified that our framework modifies the interactive adversarial partner policies in the multi-agent system, rather than the physical layout or dynamics of the environment itself.
• Policy Validation and Filtering Mechanism: After generation, we perform a two-step validation on ${\pi }_{code}^{(k+1)}$: first, a syntactic validation to ensure the code is compilable and executable in the Melting Pot environment; second, a functional validation to evaluate the policy’s ability to reduce the student agent’s return via 5 evaluation rollouts. Only policies that pass both validations are added to the population ${\mathcal{P}}_k$, to avoid invalid or irrelevant strategies polluting the curriculum.

Finally, the curriculum evolves through the aggregation of these validated generated policies. The population $\mathcal{P}$ is updated cumulatively:

$${\mathcal{P}}_{k+1}={\mathcal{P}}_k\cup \mathrm{Compile}\left({\pi }_{code}^{(k+1)}\right).$$

(14)

This mechanism prevents catastrophic forgetting in the student agent ${\pi }_E$. To minimize regret in the subsequent inner loop, the agent is compelled to simultaneously enhance its robustness against all previously generated strategies in addition to the newly introduced adversarial policy.

The complete prompt template used for LLM generation, including the full role instruction, API constraints, and a dynamic diagnosis example, is provided in Appendix C.

4.4. Algorithm Summary

The LLM-TOC algorithm, formalized in Algorithm 1, operates as a three-stage iterative cycle that progressively expands the student agent’s robust strategy space, with the following core execution logic unique to our framework:

1.

Initialization: We instantiate the student agent with random weights, and initialize the opponent population with heuristic-based policies to provide initial training signals.

2.

Inner-Loop Student Optimization: For each outer-loop iteration, the student agent optimizes its policy against the current opponent population until convergence, to learn a robust best response to the existing curriculum.

3.

Gradient Saliency Diagnosis: After convergence, we evaluate the student against the current population to identify the worst-case opponent that induces maximum regret. We then extract failure trajectories, compute value surprise to locate critical failure frames, and generate semantic causal descriptions of the student’s vulnerabilities via Jacobian saliency map analysis.

4.

Outer-Loop Curriculum Evolution: The semantic diagnosis is embedded into a structured prompt to guide the LLM to generate a new executable adversarial policy targeting the identified vulnerability. The generated policy is validated for executability and effectiveness, then added to the opponent population to expand the training curriculum.

Algorithm 1 LLM-TOC: LLM-driven theory-of-mind adversarial curriculum.

Input: Max outer iterations K, Inner steps $T_{in}$, Threshold ${ϵ}_{thresh}$, Learning rate $\alpha$; Input: Initialize student policy ${\pi }_E$ with parameters $\theta$; Input: Initialize opponent population ${\mathcal{P}}_0\leftarrow \left\{{\pi }_{heuristic}\right\}$;   1: for $k=1,\dots ,K$ do                                                     ▹Outer Loop: Curriculum Evolution   2:      ${\mathcal{P}}_{curr}\leftarrow {\mathcal{P}}_{k-1}$;   3:       // Phase 1: Inner Loop (Student Optimization)   4:       repeat   5:             Sample opponent batch ${\pi }_{-E}\sim U\left({\mathcal{P}}_{curr}\right)$;   6:             Collect trajectories $\tau =\left\{(s_t,a_t,r_t,s_{t+1})\right\}$ using ${\pi }_E$ and ${\pi }_{-E}$;   7:             Compute advantages $A_t$ using GAE;   8:             Update $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }{\mathbb{E}}_{\tau }[min({\rho }_t{A}_t,\mathrm{clip}({\rho }_t,1-ϵ,1+ϵ)A_t)]$;     ▹ PPO Update   9:       until Performance converges 10:       // Phase 2: Evaluation & Diagnosis (Gradient Saliency) 11:       Identify worst-case opponent: ${\pi }_{-E}^{*}\leftarrow arg{min}_{\pi \in {\mathcal{P}}_{curr}}{\mathbb{E}}_{\tau }[\sum {\gamma }^t{r}_t]$; 12:       Collect evaluation rollout ${\mathcal{D}}_{eval}$ against ${\pi }_{-E}^{*}$; 13:       Initialize diagnosis set ${\mathcal{D}}_{sem}^{\left(k\right)}\leftarrow \varnothing$; 14:       for each step t in ${\mathcal{D}}_{eval}$ do 15:             Calculate Value Surprise: ${\delta }_t\leftarrow |r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)|$; 16:             if ${\delta }_t>{ϵ}_{thresh}$then                                                          ▹ Identify Critical Moment 17:                   Compute Jacobian: ${\mathcal{G}}_t\leftarrow {\nabla }_{X_t}V\left(s_t\right)$; 18:                   Aggregate Saliency: $M_t^{(h,w)}\leftarrow {max}_c\left|{\mathcal{G}}_t^{(c,h,w)}\right|$; 19:                   Compute Object Scores: ${\mathrm{Score}}_{obj}\leftarrow {\displaystyle \frac{\sum M·{\mathbb{I}}_{obj}}{\sum M+ϵ}}$; 20:                   Generate description $S_t$ based on ${\mathrm{Score}}_{obj}$, Focus on ${\pi }_{-E}^{*}$ > Goal; 21:                   ${\mathcal{D}}_{sem}^{\left(k\right)}\leftarrow {\mathcal{D}}_{sem}^{\left(k\right)}\cup \left\{S_t\right\}$; 22:             end if 23:       end for 24:       // Phase 3: Semantic Generation (Leader Optimization) 25:       Construct Prompt ${\mathbf{P}}_k\leftarrow \mathrm{Concat}({\mathcal{I}}_{role},{\mathcal{I}}_{API},{\mathcal{D}}_{sem}^{\left(k\right)})$; 26:       Generate Adversarial Policy: ${\pi }_{code}^{\left(k\right)}\sim P_{LLM}(·\mid {\mathbf{P}}_k)$; 27:       Validate and Compile ${\pi }_{code}^{\left(k\right)}$; 28:       Update Population: ${\mathcal{P}}_k\leftarrow {\mathcal{P}}_{curr}\cup \left\{{\pi }_{code}^{\left(k\right)}\right\}$; 29: end for 30: Output: Robust Student Policy ${\pi }_E$

This cycle repeats for a predefined number of outer-loop iterations, or until the LLM can no longer generate valid strategies that reduce the student’s performance. The final robust student policy is returned as the output. The full prompt template used for LLM generation is provided in Appendix C.

4.5. Motivating Theoretical Analysis

In this section, we present a systematic theoretical analysis of the LLM-TOC framework via the PAC-Bayes learning framework and Stackelberg game theory, providing a rigorous theoretical rationale for the superior zero-shot generalization performance, sample efficiency, and convergence stability of our method. We first formalize the two core foundational assumptions that underpin all our theoretical conclusions, with explicit discussion of their practical implications, applicable conditions, and inherent limitations in realistic multi-agent scenarios. We then derive the convergence rate of the bi-level optimization process, prove that LLM-TOC yields a tighter generalization error bound than conventional parameter-space exploration methods, and systematically analyze the computational complexity of the framework to verify its training and inference efficiency advantages. All detailed mathematical derivations and complete proofs are provided in Appendix D.

4.5.1. Core Foundational Assumptions

All theoretical guarantees of the LLM-TOC framework are established on two non-trivial foundational assumptions, which are formally defined, interpreted, and bounded below. These assumptions directly link our theoretical design to the practical implementation of the framework, and their validity is empirically verified in Section 5.4.

Assumption 1

(Bounded Payoff). The value function of the multi-agent game is uniformly bounded; there exists a constant $V_{max}>0$ such that

$$\forall {\pi }_E\in {\Pi }_{net},{\pi }_{-E}\in {\Pi }_{sem},\left|V({\pi }_E,{\pi }_{-E})\right|\le V_{max},$$

(15)

where $V({\pi }_E,{\pi }_{-E})={\mathbb{E}}_{\tau \sim ({\pi }_E,{\pi }_{-E})}\left[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_t\right]$ denotes the expected discounted cumulative return of the student agent ${\pi }_E$ interacting with the counterpart policy ${\pi }_{-E}$, ${\Pi }_{net}$ is the neural network parameterized policy space of the student agent, and ${\Pi }_{sem}$ is the Turing-complete semantic strategy space of code-based policies.

• Practical Implications: This assumption ensures that the regret of the student agent, defined as the gap between the oracle optimal return and the actual achieved return, is always bounded. It eliminates the possibility of infinite positive/negative returns that would break the monotonic improvement property of our iterative bi-level optimization, and is a necessary precondition for the convergence of the algorithm to a stable robust equilibrium.
• Applicable Conditions in Realistic Environments: This assumption holds for almost all standard episodic MARL environments, including the Melting Pot benchmark used in our experiments as well as most realistic open-ended multi-agent interaction scenarios. Specifically, it is satisfied when three conditions are met: (1) the per-step reward of the environment is constrained to a fixed finite range; (2) the maximum length of each interaction episode is finite; and (3) the discount factor $\gamma \in [0,1)$, which ensures the cumulative discounted return is always bounded regardless of the interaction horizon.
• Inherent Limitations: This assumption does not hold for non-episodic, infinite-horizon tasks with unbounded per-step rewards. For such extreme scenarios, the convergence guarantee of our framework needs to be re-derived with additional constraints on the reward growth rate, which we leave for future work.

Assumption 2

($ϵ$-Approximate Semantic Oracle).  Let $\Phi :{\Pi }_{net}\to {\Pi }_{sem}$ denote the LLM-based policy generation function (the semantic oracle) in our framework. We define the LLM as an ${\epsilon }_{oracle}$-approximate semantic oracle if, for any given student policy ${\pi }_E$, the strategy $B=\Phi \left({\pi }_E\right)$ generated by the LLM satisfies:

$$V\left({\pi }_E,B\right)\le \underset{{\pi }^{\prime }\in {\Pi }_{sem}}{min}V\left({\pi }_E,{\pi }^{\prime }\right)+{\epsilon }_{oracle},$$

(16)

where ${\epsilon }_{oracle}\ge 0$ is the bounded approximation error of the oracle.

• Practical Implications: This assumption requires that the LLM can consistently generate adversarial strategies that approximate the worst-case attack on the student agent, with a bounded approximation error. It is the core premise that ensures our outer-loop curriculum generation can effectively tighten the constraint set of the inner-loop optimization, thereby driving the algorithm to converge to a robust equilibrium rather than a narrow, brittle Nash equilibrium subset.
• Applicable Conditions in Realistic Environments: This assumption holds when the adopted LLM has sufficient capabilities in three critical dimensions: (1) strong logical reasoning ability to understand the causal diagnosis of the student agent’s failure modes derived from gradient saliency analysis; (2) reliable code generation ability to translate the adversarial strategy into syntactically valid, executable Python code that is fully compatible with the environment interface; (3) sufficient domain knowledge of multi-agent game theory to design strategically effective adversarial behaviors that can exploit the student’s vulnerabilities. Our empirical results in Section 5.4 verify that this assumption is well satisfied in our implementation, with a 92.5% generation success rate and stable policy quality across 4 independent random seeds.
• Inherent Limitations: The approximation error ${ϵ}_{oracle}$ is directly determined by the capability of the adopted LLM. Smaller open-source LLMs with limited reasoning and coding capabilities may fail to generate valid targeted adversarial strategies, leading to a large ${ϵ}_{oracle}$ that breaks the convergence guarantee. In addition, for extremely complex multi-agent environments with obscure failure modes that are hard to describe via semantic language, the LLM may also fail to approximate the worst-case strategy effectively. Our two-step policy validation and filtering mechanism (described in Section 4.3) can automatically exclude invalid generation cases, ensuring that the convergence of the algorithm is not affected by occasional generation failures.

4.5.2. Convergence Rate Analysis

We model the bi-level iterative optimization of LLM-TOC as a leader–follower Stackelberg game, and analyze its convergence properties under the two core assumptions defined above. We define the worst-case regret of the student policy ${\pi }_E$ as

$${\mathcal{R}}_{worst}\left({\pi }_E\right)=\underset{{\pi }_{-E}\in {\Pi }_{sem}}{max}\left[V^{*}\left({\pi }_{-E}\right)-V({\pi }_E,{\pi }_{-E})\right],$$

(17)

where $V^{*}\left({\pi }_{-E}\right)={max}_{{\pi }^{\prime }}V({\pi }^{\prime },{\pi }_{-E})$ represents the oracle optimal performance achievable against the counterpart policy ${\pi }_{-E}$, assuming perfect prior knowledge of ${\pi }_{-E}$.

We formalize the convergence property of LLM-TOC in the following theorem, with the complete proof provided in Appendix D.

Theorem 1

(Convergence of LLM-TOC). Let $P_k=\{B_1,B_2,\dots ,B_k\}\subset {\Pi }_{sem}$ be the opponent policy population at the k-th outer-loop iteration. Under the Bounded Payoff assumption and the ε-Approximate semantic oracle assumption, the sequence of worst-case regrets ${\left\{{\mathcal{R}}_{worst}\left({\pi }_E^{\left(k\right)}\right)\right\}}_{k=1}^{\infty }$ converges to an ${\epsilon }_{oracle}$-Nash equilibrium neighborhood, with the average regret over K iterations bounded by

$${\overline{\mathcal{R}}}_K\le {\displaystyle \frac{C}{\sqrt{K}}}+{\epsilon }_{oracle},$$

(18)

where C is a constant related to the diameter of the strategy space and the maximum Bounded Payoff $V_{max}$, and K is the number of outer-loop iterations. This theorem delivers two core theoretical conclusions:

1. 

Convergence Rate: LLM-TOC converges to a robust equilibrium at a rate of $O(1/\sqrt{K})$ with respect to the number of outer-loop iterations K. This convergence rate matches the optimal rate of classic fictitious play algorithms in zero-sum games, verifying that our semantic-space curriculum evolution has the same rigorous convergence guarantee as traditional gradient-based optimization methods, despite operating in a discrete, non-differentiable code space.

2. 

Equilibrium Robustness: The algorithm converges to an ${\epsilon }_{oracle}$-Nash equilibrium neighborhood, where the student agent’s worst-case regret is bounded by the approximation error of the semantic oracle. This means that the student agent cannot be further exploited by any unseen strategy in the semantic space ${\Pi }_{sem}$ beyond the bounded error ${\epsilon }_{oracle}$, which directly guarantees the zero-shot generalization robustness of the learned policy.

Notably, our policy validation and filtering mechanism ensures that only valid, high-quality adversarial policies are added to the population $P_k$. Even if the LLM occasionally fails to generate an effective adversarial strategy, the convergence of the algorithm is not affected, as the invalid policy is discarded and the population remains unchanged in that iteration.

4.5.3. Generalization Bound Analysis via PAC-Bayes Framework

We leverage the PAC-Bayes learning framework to formally analyze the generalization performance of LLM-TOC, and prove that our semantic-space exploration yields a strictly tighter generalization error bound than conventional parameter-space exploration methods under reasonable preconditions.

For a student policy ${\pi }_E$ trained on a set of training opponent policies, we define the true generalization error $\mathcal{L}\left({\pi }_E\right)$ as the expected regret of the agent on the true, unknown distribution of test-time opponents ${\mathcal{D}}_{test}$, and the empirical error $\widehat{\mathcal{L}}\left({\pi }_E\right)$ as the regret on the training population. The classic PAC-Bayes generalization bound states that, for any posterior distribution Q over student policies and prior distribution P over the policy space, with probability at least $1-\delta$:

$$\mathcal{L}\left(Q\right)\le \widehat{\mathcal{L}}\left(Q\right)+\sqrt{{\displaystyle \frac{D_{KL}(Q\parallel P)+ln(1/\delta )}{2m}}},$$

(19)

where $D_{KL}(Q\parallel P)$ is the Kullback–Leibler (KL) divergence between the posterior Q and the prior P, m is the number of training samples (i.e., the number of opponent policies in the training population), and $\delta \in (0,1)$ is the confidence parameter.

The core of our theoretical analysis lies in the comparison of the KL divergence term between semantic-space exploration (LLM-TOC) and traditional parameter-space exploration:

1.

Traditional Parameter-Space Exploration: For conventional MARL methods, the prior $P_{param}$ is an isotropic Gaussian distribution over the high-dimensional neural network weights. The set of weight configurations that exactly represent a specific semantic logical strategy is a measure-zero subset in the continuous weight space. This means that the prior $P_{param}$ assigns almost zero probability to the valid semantic strategies that constitute the real-world test distribution, resulting in an extremely large KL divergence $D_{KL}({\mathcal{Q}}_{param}\parallel P_{param})$ when fitting complex social behaviors.

2.

LLM-TOC Semantic-Space Exploration: In our framework, the prior $P_{sem}$ is induced by the LLM and its pre-trained knowledge of code, logical reasoning, and human social behaviors. This prior assigns significantly higher probability to syntactically valid, logically consistent policy code that aligns with the distribution of real-world human-like social strategies. As a result, the KL divergence $D_{KL}({\mathcal{Q}}_{sem}\parallel P_{sem})$ is strictly smaller than that of parameter-space exploration:

$$D_{KL}({\mathcal{Q}}_{sem}\parallel P_{sem})\ll D_{KL}({\mathcal{Q}}_{param}\parallel P_{param}).$$

(20)

Substituting this into the PAC-Bayes inequality, we directly conclude that LLM-TOC yields a strictly tighter generalization error bound than conventional parameter-space exploration methods. This provides a rigorous theoretical rationale for the superior zero-shot generalization performance of our method observed in the experiments: the semantic curriculum generated by LLM-TOC provides much better coverage of the real-world test distribution of opponent behaviors, enabling the student agent to learn a policy that generalizes robustly to unseen OOD counterparts.

4.5.4. Computational Complexity Analysis

We systematically analyze the computational time complexity of the LLM-TOC framework, and compare it with mainstream baseline methods to theoretically verify its training and inference efficiency advantages. We first define the key variables involved in the complexity analysis: K: Number of outer-loop iterations; $T_{in}$: Number of environment interaction steps per inner-loop iteration; B: Batch size of the PPO update in the inner loop; $\left|\theta \right|$: Number of parameters of the student agent’s neural network; $T_{eval}$: Number of steps per evaluation rollout for gradient saliency calculation; $H,W$: Height and width of the agent’s observation space; and $C_{LLM}$: Time complexity of a single LLM inference query for policy code generation.

Overall Training Complexity. The overall training time complexity of LLM-TOC consists of three independent components: inner-loop MARL training, outer-loop gradient saliency calculation, and LLM policy generation. The total complexity is formalized as

$$\mathcal{O}\left(K·\left(T_{in}·B·\left|\theta \right|+T_{eval}·H·W+C_{LLM}\right)\right).$$

(21)

We analyze each component in detail:

1.

Inner-Loop MARL Training Complexity: The core of the inner loop is the PPO update for the student agent, with a time complexity of $O(T_{in}·B·|\theta \left|\right)$ per outer-loop iteration. This complexity is identical to the standard MAPPO/IPPO baselines, as we use the same network architecture and PPO update rule for the student agent. Critically, the targeted adversarial curriculum generated by our framework significantly reduces the total number of outer-loop iterations K and inner-loop steps $T_{in}$ required to reach the target performance, resulting in a total training step reduction of over 60% compared to mainstream baselines, as verified in our experiments.

2.

Gradient Saliency Calculation Complexity: The saliency map computation involves a single backpropagation of the value function per critical frame, with a complexity of $O(T_{eval}·H·W)$ per outer-loop iteration. Since $T_{eval}\ll T_{in}$ (we only perform 5 evaluation rollouts per iteration) and the observation size $H\times W=11\times 11$ in our implementation, this overhead is negligible compared to the inner-loop RL training cost.

3.

LLM Query Complexity: We only perform one LLM query per outer-loop iteration, with a complexity of $O\left(C_{LLM}\right)$. Unlike online LLM-based methods that require LLM inference at every environment step, our framework only invokes the LLM offline during training. In our experiments, we set $K=10$ total outer-loop iterations, resulting in only 10 LLM queries for the entire training process. The cumulative LLM overhead is less than 0.5% of the total training computational cost, which is negligible.

Test-Time Inference Complexity. During test-time inference, the LLM is completely removed from the pipeline, and only the lightweight student agent is deployed. The inference complexity per environment step is $O\left(\right|\theta \left|\right)$, which is identical to pure MARL baselines such as MAPPO, and orders of magnitude lower than online LLM-based methods that require LLM inference at every step. This makes our framework fully suitable for real-time high-frequency multi-agent interaction scenarios, where online LLM inference would introduce prohibitive latency.

Complexity Comparison with Baselines. We summarize the complexity comparison between LLM-TOC and mainstream baselines in

Table 1. Computational complexity comparison with mainstream baselines.

Method	Training Complexity	Test-Time Inference Complexity
LLM-TOC (Ours)	$\mathcal{O}(K·(T_{in}·B·\left|\theta \right|+T_{eval}·HW+C_{LLM}\left)\right)$, $K=10$	$\mathcal{O}\left(\right|\theta \left|\right)$
MAPPO/IPPO (Self-Play)	$\mathcal{O}(T_{total}·B·|\theta \left|\right)$, $T_{total}\gg K·T_{in}$	$\mathcal{O}\left(\right|\theta \left|\right)$
Population-Based Training (PBT)	$\mathcal{O}(T_{total}·B·|\theta |·N_{pop})$, $N_{pop}=\mathrm{pop}\mathrm{size}$	$\mathcal{O}\left(\right|\theta \left|\right)$
Hypothetical Minds (Online LLM)	$\mathcal{O}(T_{total}·C_{LLM})$	$\mathcal{O}\left(C_{LLM}\right)$ per step

, which theoretically verifies the efficiency advantages of our framework:

This comparison confirms that LLM-TOC combines the low inference latency of pure MARL methods with the semantic reasoning capability of LLMs, while achieving significantly higher training efficiency than both traditional MARL baselines and online LLM-based methods.

5. Experiments

The data presented in this study are openly available in GitHub (LLM-TOC, main branch) at https://github.com/vcis-wangchenxu/LLM-TOC.git (accessed on 8 February 2026). Publicly available environments and APIs were utilized in this study: DeepMind MeltingPot (main branch, https://github.com/google-deepmind/meltingpot, accessed on 8 February 2026) and Aliyun Qianwen API (wen-plus-2025-12-01, https://bailian.console.aliyun.com, accessed on 8 February 2026).

5.1. Experimental Setup

5.1.1. Evaluation Benchmark: Melting Pot

To rigorously evaluate zero-shot generalization in open-ended multi-agent systems, we utilize Melting Pot 2.0 [6], a benchmark suite specifically designed to test social intelligence against unseen other agents. We select four diverse substrates that cover distinct social dilemmas:

1.

collaborative_cooking_asymmetric: A coordination task requiring role specialization and synchronized action sequences to complete recipes.

2.

prisoners_dilemma_in_the_matrix_repeated: A classic social dilemma testing the agent’s ability to maintain cooperation against defection risks over repeated interactions.

3.

running_with_scissors_in_the_matrix_arena: A spatially complex, cyclical resource competition game (Rock–Paper–Scissors dynamics) where agents must identify and counter opponent strategies.

4.

running_with_scissors_in_the_matrix_repeated: A repeated version of the arena task, emphasizing long-term memory and reciprocity.

We adopt a strictly zero-shot generalization (ZSG) evaluation protocol: agents are trained on the base substrates and evaluated on held-out test scenarios. These test scenarios introduce focal-population mismatches or specific opponent behaviors not encountered during training, serving as a robust testbed for OOD generalization. In Appendix B, we provide a detailed description of the training and evaluation environments.

5.1.2. Baselines

We benchmark LLM-TOC against three categories of methods to validate its effectiveness:

Standard MARL (IPPO and MAPPO) [20,21]: Independent PPO and Multi-Agent PPO trained via self-play. These represent standard reinforcement learning baselines that typically overfit to the training population and struggle with OOD partners.

Hypothetical Minds [15]: A state-of-the-art method that utilizes a frozen Large Language Model for run-time decision-making based on visual descriptions. This baseline represents the capability of direct LLM inference in social scenarios.

Oracle PPO (Skyline): A PPO agent trained directly on the test scenarios. This serves as the theoretical performance upper bound (Skyline), indicating the maximum achievable return if the test distribution were known in advance.

5.1.3. Implementation Details

We provide a comprehensive description of all implementation details and hyperparameters below to ensure full reproducibility of our experiments. All experiments are conducted on a server with 2 Intel Xeon Gold 6330 CPUs, 4 NVIDIA RTX 3090 GPUs, and 128 GB of RAM, with PyTorch 2.2.0 as the deep learning framework.

Observation Processing. Instead of feeding raw RGB pixels directly to the policy network, we adopt a semantically structured observation space inspired by Hypothetical Minds to enhance sample efficiency and align with the LLM reasoning granularity. Specifically, the raw $88\times 88$ pixel input is discretized into an $11\times 11$ grid, where each $8\times 8$ pixel block (sprite) is mapped to a categorical feature channel representing the object type. To capture the temporal dynamics, we stack the feature maps from the last $k=4$ frames. Consequently, the input to the student agent is a tensor of shape $(11\times 11\times (C\times k\left)\right)$, where $C=12$ denotes the number of distinct object categories in the Melting Pot substrates. This representation preserves spatial structure while providing explicit semantic information.

Student Agent Network Architecture and PPO Training Configuration. The student agent (follower) utilizes a 2-layer Convolutional Neural Network encoder to process the spatial feature maps, followed by a 1-layer LSTM to handle partial observability, and two separate linear heads for policy and value function prediction. The detailed network structure is as follows: CNN Encoder: 2 convolutional layers with 32 and 64 output channels respectively, kernel size $3\times 3$, stride 1, padding 1, followed by ReLU activation functions. LSTM Layer: Hidden state dimension of 256, with layer normalization applied to the input. Output Heads: Two linear layers with 64 hidden units and ReLU activation, mapping the LSTM output to the discrete action space (dimension 8) and the scalar value function, respectively.

The policy is optimized using the Proximal Policy Optimization algorithm with the full hyperparameter configuration reported in

Table A2. Full hyperparameter configuration of the LLM-TOC framework.

Module	Hyperparameter	Value
Observation Processing	Grid size	11 × 11
	Frame stack	4
	Number of object channels	12
Network Architecture	CNN layers	2 (32, 64 channels)
	CNN kernel size	3 × 3, stride 1, padding 1
	LSTM hidden dimension	256
	MLP hidden dimension	64
	Action space dimension	8
PPO Training	Learning rate	$3\times {10}^{-4}$ (linear decay to $3\times {10}^{-5}$)
	Clip parameter	0.2
	Discount factor $\gamma$	0.99
	GAE lambda $\lambda$	0.95
	Batch size	1024 timesteps
	PPO epochs per batch	4
	Mini-batch size	256
	Entropy coefficient	0.01
	Gradient clipping norm	0.5
Training Process	Number of outer-loop iterations K	10
	Inner-loop steps per iteration	$1\times {10}^6$
	Total maximum training steps	$1\times {10}^7$
	Evaluation rollouts per policy	5
	Maximum episode length	1000 timesteps
	Gradient saliency threshold ${ϵ}_{thresh}$	90th percentile of value surprise
LLM Configuration	Model version	qwen-plus-2025-12-01
	Sampling temperature	0.7
	top_p	0.9
	Max generation tokens	1024
	Repetition penalty	1.05
	Max generation attempts per iteration	2
Reproducibility	Random seeds	1, 42, 100, 2026

(Appendix A). The core training settings are as follows: Learning rate: $3\times {10}^{-4}$, with linear decay over the training process and a final learning rate of $3\times {10}^{-5}$. PPO clip parameter: 0.2, with no gradient penalty applied to the value function. Generalized Advantage Estimation (GAE): Discount factor $\gamma =0.99$, GAE lambda $\lambda =0.95$. Training batch size: 1024 timesteps per update, with 4 PPO epochs per batch, and a mini-batch size of 256. Entropy coefficient: 0.01, added to the loss function to encourage exploration. Gradient clipping: Maximum L2 norm of the gradient is set to 0.5 to stabilize training.

Training Process Configuration. The overall training process consists of 10 outer-loop iterations ($K=10$), with the following settings for each iteration: Inner-loop training steps: $T_{in}=1\times {10}^6$ environment interaction steps per outer-loop iteration, with early stopping enabled if the student agent’s performance on the current population converges. Total maximum training steps: $1\times {10}^7$ environment interaction steps across all outer-loop iterations, consistent with the training budget of all baseline methods. Evaluation rollouts: 5 independent evaluation rollouts per opponent policy in the population, with a maximum episode length of 1000 timesteps per rollout, to identify the worst-case opponent for gradient saliency analysis. Gradient saliency threshold ${ϵ}_{thresh}$: Set to the 90th percentile of value surprise in each evaluation batch, to filter critical failure frames with significant value fluctuations.

LLM Semantic Oracle Configuration. In the outer loop, we employ qwen-plus-2025-12-01 (provided by Alibaba Cloud Bailian Platform) as the semantic oracle to generate adversarial policies. The full inference and prompt generation settings are as follows: Inference hyperparameters: Sampling temperature set to 0.7, top_p set to 0.9, maximum generation tokens set to 1024, greedy decoding disabled, repetition penalty set to 1.05. Prompt regeneration rule: The structured prompt $P_k$ is regenerated from scratch at each outer-loop iteration k. The fixed components (role instruction $I_{role}$ and API constraints $I_{API}$) are reused across all iterations to ensure the consistency of generation format and objective, while the core semantic diagnosis component $D_{sem}^{\left(k\right)}$ is completely updated based on the student agent’s latest failure modes in the current iteration, with no cross-iteration reuse of diagnosis content. Generation attempts: We perform at most 2 generation attempts per outer-loop iteration. Syntactically invalid code is automatically filtered out via compilation validation, and functionally ineffective policies (that do not reduce the student agent’s return) are discarded via functional validation. No few-shot examples are included in the prompt to avoid manual bias and ensure the generalizability of the generation mechanism. The full prompt template is provided in Appendix C.

Reproducibility Settings. All experiments are conducted with 4 independent random seeds (1, 42, 100, and 2026) for the student agent initialization, environment sampling, and LLM generation. All random number generators (PyTorch (2.5.0), NumPy (1.28.2), and Python (3.11) native) are initialized with the corresponding seed before each experiment to ensure full reproducibility.

5.1.4. Evaluation Metrics

To rigorously quantify the zero-shot generalization capability, training efficiency, and module contribution of the proposed LLM-TOC framework, we adopt four core evaluation metrics, with explicit mathematical definitions and selection rationales fully aligned with our experimental analysis as follows:

1.

Expected Cumulative Collective Return: This is the primary metric for evaluating the zero-shot generalization performance of agents, which is the standard evaluation indicator for the Melting Pot benchmark. It is defined as the expected discounted cumulative return of the student agent in the multi-agent test scenario:

$$J\left({\pi }_E,{\pi }_{-E}^{test}\right)={\mathbb{E}}_{\tau \sim \left({\pi }_E,{\pi }_{-E}^{test}\right)}\left[\sum _{t=0}^{\infty }{\gamma }^t{\mathcal{R}}_E\left(s_t,a_{E,t},a_{-E,t}\right)\right],$$

(22)

where ${\pi }_E$ is the policy of the student agent, ${\pi }_{-E}^{test}$ is the policy of unseen OOD test partners, $\gamma$ is the discount factor, and ${\mathcal{R}}_E$ is the environment-defined reward function of the student agent.

This metric directly reflects the overall task performance of the agent in the target multi-agent scenario, and is consistent with the evaluation protocol of mainstream MARL works on the Melting Pot benchmark. For each test scenario, we compare the performance of LLM-TOC with all baselines under the exact same environment and reward settings to ensure fairness. All results are averaged over 4 random seeds, with standard deviation reported as the uncertainty interval in the learning curves.

2.

Relative Performance to Oracle PPO: This metric is used to eliminate the impact of inconsistent reward scales across different Melting Pot substrates, enabling fair cross-scenario performance comparison. It is defined as the ratio of the agent’s expected cumulative collective return to the return of the Oracle PPO agent in the same test scenario:

$$R_{rel}={\displaystyle \frac{J\left({\pi }_E,{\pi }_{-E}^{test}\right)}{J_{oracle}}},$$

(23)

where $J\left({\pi }_E,{\pi }_{-E}^{test}\right)$ is the expected cumulative collective return of the evaluated agent, and $J_{oracle}$ is the expected cumulative collective return of the Oracle PPO agent (trained directly on the test scenario, serving as the theoretical performance upper bound).

The reward scales of the four selected Melting Pot substrates are significantly different, making it impossible to directly compare the raw collective return across scenarios.

3.

Training Steps to Target Performance: This metric quantifies the training efficiency and computational cost of the algorithm, defined as the minimum number of environment interaction steps required for the agent to reach the predefined target relative performance threshold in the held-out test scenarios.

This metric directly reflects the sample efficiency and convergence speed of the algorithm, which is the core quantitative indicator to verify the claim of “training cost reduction by more than 60%” in our work. It is also a standard efficiency evaluation metric widely used in MARL curriculum learning works.

4.

Relative Performance Drop in Ablation Study: This metric evaluates the independent contribution of each core component in LLM-TOC, defined as the percentage of performance decline of the ablation variant compared with the full LLM-TOC framework, calculated as

$$Dro{p}_{rel}={\displaystyle \frac{R_{rel}^{full}-R_{rel}^{ablation}}{R_{rel}^{full}}}\times 100\%,$$

(24)

where $R_{rel}^{full}$ is the relative performance to Oracle PPO of the full LLM-TOC framework, and $R_{rel}^{ablation}$ is that of the corresponding ablation variant. This metric quantifies the impact of removing each core module on the final zero-shot performance.

5.2. Results and Analysis

We evaluate the zero-shot generalization performance of LLM-TOC and baselines on four challenging Melting Pot scenarios. The learning curves, depicting the collective return on held-out test scenarios over 10 million training steps, are presented in Figure 3. All results are averaged over 4 independent random seeds, with standard deviation reported as the uncertainty interval in the learning curves (Figure 3). The standard deviation of the final normalized performance of LLM-TOC across all 4 seeds is less than 5% of the mean value for all test scenarios, which is significantly lower than that of all baselines, demonstrating the strong robustness of our method to random initialization. The detailed variance statistics of all methods are provided in Section 5.4.

The normalized final zero-shot generalization performance (at 10M training steps) of all methods across all test substrates is summarized in

Table 2. Final normalized zero-shot generalization performance (mean ± std) at 10M training steps, normalized to the Oracle PPO upper bound (1.0).

Substrate	LLM-TOC	Hypothetical Minds	MAPPO	IPPO	Oracle PPO
Collaborative Cooking (Asymmetric)	$0.82\pm 0.04$	$0.65\pm 0.07$	$0.38\pm 0.09$	$0.32\pm 0.11$	$1.00\pm 0.03$
Prisoner’s Dilemma (Repeated)	$0.85\pm 0.03$	$0.71\pm 0.06$	$0.42\pm 0.08$	$0.35\pm 0.10$	$1.00\pm 0.02$
Running with Scissors (Arena)	$0.76\pm 0.05$	$0.58\pm 0.08$	$0.35\pm 0.07$	$0.29\pm 0.09$	$1.00\pm 0.04$
Running with Scissors (Repeated)	$0.78\pm 0.04$	$0.62\pm 0.07$	$0.39\pm 0.08$	$0.31\pm 0.10$	$1.00\pm 0.03$

, where the scores are normalized by the Oracle PPO upper bound (1.0) for fair cross-scenario comparison.

This table enables standardized quantitative comparison of all methods across the four substrates, which have distinct native reward scales and game dynamics. For the full learning trajectory and convergence dynamics of each method in the native task setting, please refer to the learning curves in Figure 3.

LLM-TOC (red solid line) consistently achieves superior performance compared to both standard MARL baselines and the inference-based method across all domains. In the Collaborative Cooking and Running with Scissors tasks, standard baselines like MAPPO and IPPO fail to generalize effectively, often converging to suboptimal policies (relative performance to Oracle PPO $<40\%$) due to overfitting to the training population. In contrast, LLM-TOC maintains a steady upward trajectory, eventually reaching 75–85% of the Oracle PPO performance (green dashed line). This validates that our bi-level curriculum effectively exposes the student agent to a diverse range of semantic strategies, preventing brittle adaptation to specific partners.

A key advantage of LLM-TOC is its ability to accelerate the discovery of robust strategies. Unlike PBT methods that rely on random parameter perturbations, our semantic oracle directly synthesizes high-value adversarial policies. As evidenced by the steep initial slope in Figure 3, LLM-TOC reaches the convergence performance of the strongest baseline (Hypothetical Minds) in significantly fewer steps. Quantitatively, to achieve the target relative performance to Oracle PPO of 0.6, LLM-TOC requires only $3.5\times {10}^6$ RL environment interaction steps, whereas traditional MAPPO baseline requires over $9\times {10}^6$ steps (and IPPO fails to reach the target performance entirely within 10M steps), translating to a reduction of over 60% in the total training environment steps, which is the core metric of training computational cost in MARL. This confirms that semantic guidance serves as a highly efficient “compass” in the vast strategy search space.

The superior sample efficiency and generalization performance of LLM-TOC can be directly attributed to the targeted curriculum evolution driven by our Gradient Saliency Feedback mechanism, which aligns perfectly with our theoretical analysis of search complexity reduction in Appendix D. Unlike unguided population-based training methods that perform random exploration in the high-dimensional parameter space, our gradient-based diagnosis mechanism pinpoints the exact failure modes of the student agent at each training iteration, translating pixel-level value fluctuations into semantically meaningful causal cues. This allows the LLM to perform “semantic gradient ascent” in the strategy space, directly generating adversarial policies that target the student’s current vulnerabilities, rather than blindly sampling random strategies. As a result, each newly added policy to the population provides the maximum learning signal for the student agent, which explains why our framework reduces the training steps required to reach the target performance by over 60% compared to mainstream baselines.

The Gradient Saliency Feedback mechanism is also the core driver of the high semantic diversity of the generated policy population, which directly addresses the mode collapse problem of traditional self-play and population-based methods. As shown in Figure 4, the LLM guided by gradient-based diagnosis generates a diverse set of semantic strategies, covering distinct behavioral modes that are rarely discovered by parameter-space exploration. This is because the saliency feedback continuously reveals new blind spots of the student agent throughout the training process, guiding the LLM to explore previously uncovered regions of the semantic strategy space. The monotonically increasing population diversity across training iterations (verified in Section 5.4) ensures that the student agent is continuously challenged by novel, semantically distinct strategies, thereby expanding its robust strategy hull and avoiding overfitting to a narrow set of Nash equilibria. This empirical observation directly validates our theoretical claim that semantic-space exploration yields a tighter generalization bound than parameter-space exploration, as the diverse population generated by our framework provides much better coverage of the real-world strategy distribution.

In our implementation, we set the total number of outer-loop iterations $K=10$. For each iteration, we perform only one LLM API call, with an average of 800 input tokens and 400 output tokens (1200 total tokens per call). The total token consumption for the entire training process is only 12,000 tokens, and the corresponding API cost is negligible (less than 0.5% of the total computational cost of RL training).

The Oracle PPO represents the performance upper bound where agents are trained directly on the test scenarios. LLM-TOC significantly narrows the gap between zero-shot agents and this oracle boundary compared to other methods. For instance, in the complex Prisoner’s Dilemma task, where cooperation requires identifying subtle defection cues, LLM-TOC is the only method that establishes stable cooperation comparable to the Oracle, whereas baselines devolve into mutual defection (low returns).

To better understand why LLM-TOC achieves superior zero-shot robustness, we analyze the semantic diversity of the generated opponent populations and the interpretability of the curriculum evolution process.

A critical failure mode in self-play is the convergence to a narrow set of Nash equilibria, often resulting in “mode collapse” where agents only learn to coordinate with compliant partners. The strategy categories in Figure 4 are determined via a two-step LLM-assisted classification with manual verification: (1) We first define five pre-specified semantic strategy categories (Free-Rider, Saboteur, Opportunist, Collaborative, and Random/Noisy) based on game theory and social behavior taxonomy. (2) For each generated policy, we feed the policy code and its behavioral trajectory in the environment to the LLM for automatic classification. (3) All classification results are manually verified and corrected to ensure accuracy. For the MAPPO self-play checkpoints, we perform the same classification based on their behavioral trajectories in the environment, to ensure the consistency of the classification criteria.

It should be explicitly acknowledged that there is an essential difference between the two groups of strategies compared in this figure: the adversaries generated by LLM-TOC are rule-based policies represented in executable code space, while the self-play opponents of MAPPO are neural network parameterized policies. The core purpose of this comparison is to demonstrate the difference in semantic strategy diversity brought by the two different exploration paradigms (semantic code space exploration vs. parameter space exploration), rather than a direct peer-to-peer comparison of the two algorithms themselves.

Interpretability via Visual-Semantic Alignment. Unlike black-box adversarial generation, LLM-TOC provides transparent insights into why a student agent fails. Figure 5 demonstrates this diagnosis loop. In a specific failure case in Running with Scissors, the agent froze and failed to collect resources. While a standard value loss only indicates that performance dropped, our Gradient Saliency mechanism (Figure 5, Middle) reveals the cause: the agent’s attention was over-fixated on a distant opponent (Red) rather than the immediate goal. The LLM leverages this “visual evidence” to generate a precise causal explanation (Figure 5, Right) and subsequently synthesizes a training opponent that specifically exploits this distraction. This explicit causal link between visual attention, semantic diagnosis, and code generation is the core driver of our method’s data efficiency.

5.3. Ablation Study

To disentangle the contributions of the key components in LLM-TOC, we conduct ablation studies by creating two variants: (1) w/o Saliency: Removes the Gradient Saliency Feedback, relying on generic prompts to guide the semantic oracle; (2) w/o Code Gen (PBT): Replaces the LLM-based code generation with standard population-based training that optimizes opponent policies in the parameter space. Figure 6 visualizes the impact of these components on training efficiency and final robust performance.

Comparing the full method (Red) with the w/o Saliency variant (Blue) in Figure 6, we observe that explicit causal diagnosis via Gradient Saliency Feedback is critical for both sample efficiency and final performance. We provide quantitative validation of the efficiency improvement brought by the saliency-guided mechanism in

Table 3. Quantitative validation of Gradient Saliency Feedback via ablation. The downward arrow (↓) in the table indicates a reduction in the number of steps required to reach 0.6 relative performance, i.e., fewer steps are needed for the model to achieve the target performance.

Metric	Full LLM-TOC	w/o Saliency	Improvement
Steps to reach 0.6 rel. perf.	$3.5\times {10}^6$	$7.2\times {10}^6$	51.4% steps ↓
Final rel. perf. to Oracle PPO	75–85%	61–70%	17.8% avg. gain
Convergence speed (per 1e6 steps)	0.21	0.09	133.3% faster

, which compares the core performance metrics of the full method and the w/o Saliency variant across all test scenarios.

The quantitative results demonstrate that the saliency-guided curriculum generation reduces the training steps required to reach the target performance by more than 50%, directly verifying that our mechanism significantly improves the learning efficiency of the agent. Without saliency feedback, the LLM must blindly guess potential weaknesses of the student agent, leading to a $133\%$ slower convergence rate and a ∼$18\%$ average relative performance drop, as the generated opponents are less targeted and fail to exploit subtle vulnerabilities. This quantitative evidence strongly validates the effectiveness of our Gradient Saliency Feedback mechanism.

From a theoretical perspective, the significant performance degradation of the w/o Saliency variant can be attributed to the loss of the “semantic gradient” that guides the LLM policy generation. Without the causal diagnosis provided by gradient saliency, the LLM can only perform random search in the vast semantic strategy space, which leads to a much lower probability of generating valid adversarial policies that exploit the student’s subtle vulnerabilities. This directly violates the core design of our framework, which aims to reduce the search complexity of the outer-loop optimization via targeted feedback. Our theoretical analysis in Appendix D proves that the saliency-guided mechanism reduces the sampling complexity of valid adversarial policies by a factor of $1/P\left(c\right)$ (where $P\left(c\right)\ll 1$), which is perfectly consistent with the 133% slower convergence speed and 51.4% more training steps observed in the w/o Saliency variant.

The most significant performance drop occurs when removing the code generation entirely (w/o Code Gen, Orange). As shown in Figure 6, agents trained via parameter-space optimization saturate at a low performance level (∼$0.4$). Parameter space exploration struggles to traverse the vast strategy manifold to find distinct behavioral modes, often collapsing to simple, aggressive policies. In contrast, the Turing-complete code space allows LLM-TOC to construct logically complex and semantically diverse scenarios, pushing the student agent to learn robust generalized behaviors.

The catastrophic performance drop of the w/o Code Gen variant provides strong empirical evidence for our theoretical claim that parameter-space exploration cannot adequately cover the semantic strategy space ${\Pi }_{sem}$. Traditional population-based training operates in the continuous parameter space of neural networks, where the set of weight configurations that represent distinct semantic strategies is a measure-zero subset. This leads to severe mode collapse, as shown in Figure 4b, where the population is dominated by simple collaborative or random behaviors. In contrast, our LLM-based code generation directly explores the Turing-complete semantic strategy space, generating logically complex and behaviorally distinct policies that are inaccessible to parameter-space exploration. This fundamental difference explains why the w/o Code Gen variant saturates at a low performance level, while the full LLM-TOC framework continuously improves and approaches the oracle performance bound.

We further performed a paired two-tailed t-test to verify the statistical significance of the performance difference between the full LLM-TOC framework and the w/o Saliency variant, across all four test substrates and four random seeds. The test result shows that the performance gap is statistically significant ($p<0.05$), which strongly validates that the performance improvement brought by the Gradient Saliency Feedback mechanism is not due to randomness but to the targeted causal diagnosis provided by the mechanism.

5.4. Supplementary Empirical Analysis of the Semantic Oracle Assumption

The $ϵ$-approximate semantic oracle assumption is the core premise of our theoretical convergence and generalization bound analysis. Here, we provide empirical validation of this assumption through quantitative statistics of LLM generation performance across all training iterations and random seeds, to bridge the gap between theoretical analysis and practical implementation.

We first define two core metrics to quantify the performance of the LLM semantic oracle: Generation Success Rate: The proportion of generated policies that pass both syntactic compilation validation and functional validation across all outer-loop iterations. Policy Quality Consistency: The coefficient of variation (CV) of the student agent’s return reduction induced by the generated policies across four independent random seeds, which quantifies the stability of the LLM generation quality.

We statistically analyze the generation results of 10 outer-loop iterations across four random seeds, with the results presented in

Table 4. Empirical validation statistics of the $ϵ$-approximate semantic oracle assumption calculated over 10 outer-loop iterations across 4 independent random seeds.

Metric	Mean Value	Standard Deviation
Generation Success Rate	92.5%	3.2%
Policy Quality Consistency (CV)	7.8%	2.1%

.

The empirical results show that the LLM maintains a consistently high generation success rate (over 90%) across all iterations and seeds, indicating that it can reliably generate valid adversarial policies targeting the student’s weaknesses, which aligns with the core requirement of the $ϵ$-approximate semantic oracle assumption. The low coefficient of variation demonstrates that the quality of the generated policies is highly consistent across different random seeds, verifying the stability of the LLM oracle capability in our framework.

We further analyze the 7.5% failed generation cases, and find that all failures fall into two categories: (1) syntactic errors in the generated code that lead to compilation failure, which can be automatically filtered out by our validation mechanism; (2) generated policies that are valid but do not reduce the student’s return significantly, which are discarded in the functional validation step and will not be added to the training population. These failed cases do not affect the robustness of the training process, as our filtering mechanism ensures that only valid, high-quality policies are included in the curriculum.

We further quantify the stability of the curriculum evolution process by measuring the cumulative diversity of the generated policy population across iterations. We use the number of distinct semantic strategy types as the diversity metric. The results show that the population diversity increases monotonically across iterations for all random seeds, with no significant degradation in generation diversity in the late training phase, verifying the long-term stability of the curriculum generation process.

We further provide detailed engineering statistics of the LLM code generation process across all 10 outer-loop iterations and four random seeds, to validate the practical reliability of the semantic oracle:

1.

Compilation Failure Rate: The average syntactic compilation failure rate of the generated code is 4.2% ± 1.8% across all iterations. All syntactically invalid code is automatically filtered out by our validation mechanism, and will not be added to the training population

2.

Generation Attempts per Iteration: We perform at most two generation attempts per outer-loop iteration. In 95.8% of iterations, the first generation attempt produces syntactically valid code that passes compilation; only 4.2% of iterations require a second generation attempt, with no iterations requiring more than two attempts.

3.

Proportion of Non-Trivial Valid Policies: Among all syntactically valid policies, 88.3% ± 2.5% are non-trivial adversarial policies that can significantly reduce the student agent’s return (functional validation passed), and are added to the training population. The remaining 11.7% of valid but ineffective policies are filtered out by our functional validation step.

These statistics demonstrate that the LLM can reliably generate valid, high-quality adversarial policies with minimal generation attempts, which fully aligns with the core requirement of the $ϵ$-approximate semantic oracle assumption. The small proportion of failed/ineffective generations can be completely filtered out by our two-step validation mechanism, without affecting the convergence of the algorithm.

6. Conclusions, Limitations and Future Work

In this work, we proposed LLM-TOC, a novel framework that bridges the gap between the sample efficiency of reinforcement learning and the semantic reasoning capabilities of Large Language Models to tackle the challenge of zero-shot generalization in open-ended multi-agent systems. By formalizing the problem as a bi-level Stackelberg game within a Turing-complete code space, we overcame the mode collapse inherent in parameter-space exploration. Crucially, our proposed Gradient Saliency Feedback mechanism effectively grounds the symbolic reasoning of the LLM in the pixel-level causality of the environment, enabling the synthesis of targeted, high-value adversarial strategies without requiring dense reward engineering. Theoretical analysis via the PAC-Bayes framework guarantees that our semantic exploration yields tighter generalization bounds and faster convergence rates ($O(1/\sqrt{K})$). Empirical results on the challenging Melting Pot benchmark demonstrate that LLM-TOC not only achieves state-of-the-art zero-shot robustness against out-of-distribution partners but also reduces training costs by over $60\%$ compared to standard population-based training methods. These results validate the effectiveness of our framework in complex multi-agent coordination and competition scenarios, providing a promising direction for building generalizable multi-agent agents.

Despite the promising results, our current framework presents several limitations that open avenues for future research:

Dependence on Proprietary LLMs and Theoretical Assumption Constraints. The performance of the semantic oracle and the validity of our theoretical convergence guarantees rely heavily on the reasoning and coding capabilities of state-of-the-art models, which is directly related to the $ϵ$-approximate semantic oracle assumption. Our preliminary tests indicate that smaller, open-source models may struggle with complex causal diagnosis and valid code generation, leading to a large approximation error that invalidates the theoretical convergence guarantee. In addition, our theoretical guarantees are established under the Bounded Payoff assumption, which is not applicable to non-episodic infinite-horizon tasks with unbounded rewards. Future Work: We plan to distill the capabilities of the large oracle into a smaller, domain-specific model through fine-tuning, thereby reducing deployment costs and privacy concerns, and extending the theoretical framework to adapt to unbounded reward scenarios.

Environment Compatibility. LLM-TOC requires the environment to support the dynamic injection of Python-based policies, which restricts its direct application to compiled binaries or environments with rigid APIs. Future Work: We aim to extend the framework to support abstract behavior trees or domain-specific languages (DSLs) that can act as a universal interface for a broader range of simulation platforms.

Static Robustness vs. Online Adaptation. While LLM-TOC produces a highly robust policy, the student agent’s parameters remain fixed during test-time evaluation. It does not actively update its strategy within the test episode to adapt to novel teammates. Future Work: Integrating LLM-TOC with meta-learning or in-context learning modules could enable agents that not only possess a robust prior but also continuously adapt to their social partners in real-time.