Enhancing Commentary Strategies for Guandan: A Study of LLMs in Game Commentary Generation

Abstract

Recent advancements in large language models (LLMs) have unlocked the potential for generating high-quality game commentary. However, producing insightful and engaging commentary for complex games with incomplete information remains a significant challenge. In this paper, we introduce a novel commentary method that combines reinforcement learning (RL) and LLMs, tailored specifically for the Chinese card game Guandan. Our system leverages RL to generate intricate card-playing scenarios and employs LLMs to generate corresponding commentary text, effectively emulating the strategic analysis and narrative prowess of professional commentators. The framework comprises a state commentary guide, a Theory of Mind (ToM)-based strategy analyzer, and a style retrieval module, which seamlessly collaborate to deliver detailed and context-relevant game commentary in the Chinese language environment. We empower LLMs with ToM capabilities and refine both retrieval and information filtering mechanisms. This facilitates the generation of personalized commentary content. Our experimental results demonstrate a significant improvement in the system’s effectiveness in generating accurate, coherent, and engaging commentary when applied to open-source LLMs, surpassing GPT-4 across multiple evaluation metrics.

1. Introduction

Large language models (LLMs) have made significant advances in the field of natural language processing in recent years, particularly demonstrating exceptional capabilities in complex text generation and context understanding. This opens up new possibilities in the field of game commentary generation, enabling the production of insight-rich and contextually detailed commentary. The concept of symmetry plays a crucial role in this domain, as it pertains to the balanced and harmonious generation of commentary, mirroring the game’s strategic elements while maintaining a consistent, unbiased narrative flow. This alignment with symmetry ensures that the generated commentary remains coherent and well structured, reflecting the underlying order and dynamics of the game itself. However, generating high-quality game commentary still faces multiple challenges, including a deep understanding of game rules and strategies, the smooth generation of real-time commentary, and engaging narrative skills.

Previous work makes some progress in game commentary generation. For example, refs. [1,2] report preliminary explorations into chess commentary, but these methods are mainly based on simple rules and limited by the scale and quality of datasets. Ref. [3] applies advanced techniques in Shogi games, generating commentary by extracting key terms from game states and integrating them with language models, marking a step towards more complex generation methods. However, game commentary generation still faces a series of challenges, especially in handling incomplete information games [4], simulating human commentators’ advanced strategy analysis, and with applications in non-English environments. Moreover, existing methods still fall short in data-driven deep learning applications, limiting the naturalness and richness of generated commentary [5,6].

To assess the performance of LLMs in cooperative and incomplete information environments, we select the popular Chinese card game Guandan https://en.wikipedia.org/wiki/Guandan (accessed on 1 July 2025). The game’s objective is for two players on the same team to play out all their cards as quickly as possible to score higher team points. Consistent with recent work, we evaluate commercial and Chinese open-source models in a zero-training setup on the state-of-the-art RL agent Danzero+’s [7] playing behavior to assess their ability to understand and interpret agent actions in a cooperative, incomplete information environment.

In this study, we develop a novel commentary method that combines RL and LLM, specifically designed for the complex, incomplete information card game Guandan. By utilizing RL to generate complex card-playing scenarios, and LLM to generate corresponding commentary text, our system effectively simulates the strategy analysis and narrative skills of professional commentators. The system consists of several core components, including a state commentary guide, a ToM-based strategy analyzer [8], and a style retrieval module, as illustrated in Figure 1. These modules work together to provide detailed and context-relevant game commentary, while optimizing commentary effects in the Chinese environment. Experimental results show that our commentary framework significantly enhances the performance of open-source LLMs, significantly outperforming GPT-4 on multiple evaluation metrics.

We introduce a novel commentary method that combines Reinforcement Learning (RL) and Large Language Models (LLMs) to enhance commentary generation for the complex card game Guandan.

• We equip LLMs with Theory of Mind (ToM) capabilities, optimizing retrieval and information filtering mechanisms. This allows LLMs to effectively analyze the game-playing processes of RL agents and produce more personalized, context-aware commentary.
• We design a specialized commentary agent framework tailored for Guandan, a card game characterized by incomplete information. This framework enables the agent to handle complex game situations and generate insightful commentary without requiring specific training in the game environment.
• Our experimental results demonstrate that the proposed framework significantly enhances the performance of open-source LLMs, outperforming GPT-4 across multiple evaluation metrics.

In this paper, we first introduce the problem of generating game commentary for the Chinese card game Guandan, focusing on the challenges posed by incomplete information and the need for strategic analysis. In Section 2, we review related work in the field of game commentary generation, including both traditional rule-based methods and more recent approaches using large language models (LLMs). Section 3 details the methodology, where we present our proposed framework, including the key components of our system, such as reinforcement learning (RL) integration and theory of mind (ToM) for strategic analysis. The experimental setup, evaluation metrics, and results are discussed in Section 4, followed by a summary of conclusions and potential future work in Section 5.

2. Related Work

2.1. Game Commentary Generation

In the field of game commentary generation, there has been significant progress in research in recent years. Initially, due to the lack of large-scale training data, researchers primarily used rule-based methods, such as generating commentary through rules in sports and chess games to enhance the viewer experience [1,2]. With the development of deep learning technologies, neural networks began to be applied to commentary generation, such as detailed commentary for soccer and baseball games [9,10], and using encoder-decoder models [6] and hierarchical models to generate commentary for eSports [11] games like League of Legends. Additionally, the rise of LLMs brought a new dimension to game commentary, handling multiple tasks through zero-shot learning [12] and providing a superior user experience.

Researchers have also explored methods combining visual and structured data, such as in racing games where [13] generates automatic commentary by combining multiple data sources. Meanwhile, ref. [14] introduces a large-scale commentary dataset in the field of chess, improving the accuracy and fluency of the generated commentary. More recently, ref. [15] applies LLMs to the generation of commentary for fighting games, demonstrating the model’s potential in generating diverse and preferential commentary. Based on these research findings, this paper aims to further explore the application of methods combining reinforcement learning and an LLM in Guandan, a complex card game with incomplete information, to further advance the field of game commentary generation.

2.2. Retrieval-Augmented Generation

In the field of Retrieval-Augmented Generation (RAG), significant progress has been made recently. One study [16] introduces the RAGAs evaluation framework for rapid assessment of Retrieval-Augmented Generation system performance, which is particularly suitable for the rapid deployment of LLMs.

Another study investigates the domain adaptability of RAG models and proposes the RAG-end2end approach, enabling the models to adapt to specific domain knowledge bases [17]. Additionally, researchers have proposed MuRAG, a multi-modal Retrieval-Augmented Generator. It leverages external non-parametric multi-modal memory for enhanced language generation, demonstrating outstanding cross-dataset performance in tasks involving image and text-based questions and answers [18]. Finally, another study develops the ARM-RAG system, which enhances problem-solving performance by storing and retrieving inference chains. This effectively boosts the intelligence of large language models while reducing training costs [19]. These advancements demonstrate the potential of RAG technology in enhancing the accuracy of knowledge access, the quality of generation, and the narrative ability. Our research explores the integration of RAG with domain-specific strategies to enhance the strategic depth of game commentary.

3. Method

The study introduces a modular approach (Figure 1) that enables a commentary agent based on LLMs to deliver detailed commentaries on cooperation and strategic interactions with opponent agents in Guandan, a card game of incomplete information, without the need for specialized training in Chinese text environments. This task is broken down into several core components, including a State Commentary Guider, ToM Strategy Analyzer, and Style Retriever. The inputs received by the LLM include game rules, observation rules (i.e., prompts that guide the LLM to convert low-level game state information into readable text), and historical game context. We provide a detailed description process demonstrating how to guide the LLM to utilize its knowledge, reasoning capabilities, and ToM to perform these module functions and effectively navigate the complexities inherent in incomplete information games.

The commentary generation process follows a structured pipeline: Game State + History + Rules → State Commentary Guider → Obsr, Hisr → ToM Analyzer (First/Second-Order Prompted Inference) → Style Retrieval and CoT Filtering → Commentary Coordinator → Final Output. This design enables the system to progressively transform raw game inputs into strategic, stylistically aligned, and context-aware commentary through a sequence of reasoning and retrieval-enhanced modules.

3.1. State Commentary Guider

In commentary for the Guandan game, the process starts with the State Commentary Guider, which converts the low-level game state information into a preliminary commentary text. This initial text provides basic context about the game, such as which cards were played and the current situation. Next, the Theory of Mind (ToM)-Based Strategy Analyzer analyzes this commentary to identify the psychological state of the players and predict their next moves. This step is crucial in adding depth to the commentary by providing strategic insights about players’ likely actions. After this, the Style Retrieval Module applies a tree-based retrieval method to filter the commentary generated so far, selecting the text that aligns with the desired commentary style (e.g., formal or casual, detailed or high-level). Finally, the Commentary Coordinator integrates all the components to produce a coherent, contextually rich commentary. This step ensures that the final output is fluent, informative, and logically consistent with the game’s progression.

A State Commentary Guide module is proposed, aimed at assisting the language model in transforming the game state into readable commentary text. This module integrates game rules, observation transformation rules, and historical transformation rules, guiding the model to convert the current low-level state and historical information into descriptive text to support the commentary process. Please refer to Appendix A for detailed rules. We design structured prompts to help the LLM understand the game rules and current state information, including card types, scoring rules, and win/loss conditions.

An example of the game rule template is as follows: The game rule template is shown as follows:

• General Rules: A brief game introduction, team position rules, all single cards, and cards ranking.
• Card Type that cannot beat other types: {Description of Card Type 1, Example of Card Type 1}, {Description of Card Type 2, Example of Card Type 2}, ...;
• Card Type that can beat other types: {Description of Card Type 1, Example of Card Type 1}, {Description of Card Type 2, Example of Card Type 2}, ...;
• Single Win/Loss Rule: The scoring rules for four players in a single game, with different combinations of cards being played out in different order.
• Whole Win/Loss Rule: The overall win/loss conditions.

We also design templates for observation rules and historical transformation rules, which include input interpretations and transformation prompts. Through these templates, the model can convert low-level game states and history into vivid commentary text, making it easier for the audience to understand the game’s progression and players’ strategies. The template is shown as follows:

• Input Explanation: The input types, like dictionaries and lists, are clearly specified. A description of every component in the input is also provided.
• Conversion Tips: Additional instructions guide converting the low-level game state representations into natural language descriptions.

We can effectively convert low-level game states and history records into human-readable text, denoted as ${\mathit{Obs}}_r$ and ${\mathit{His}}_r$, respectively, by employing the game rule, observation conversion rule, and history conversion rule. In detail, Obs_r refers to a sequence of observations, indexed from 1 to N, where N is the length of the generated commentary. This process enhances the clarity and comprehensibility of the game commentary, providing more vivid and understandable content for the audience. The conditional distribution for each element ${\mathit{Obs}}_r\left[j\right]$ within the generated text can be modeled using the prompts ${\mathit{Prompt}}_{obs}$ as:

$$\mathbb{P}\left(Ob{s}_r\right)\sim \prod _{j=1}^{N}L{M}_{\theta }\left(Ob{s}_r\left[j\right]\mid Promp{t}_{obs},Rule,Rul{e}_{obs},Ob{s}_r[1,\dots ,j-1]\right)$$

(1)

${\mathit{LM}}_{\theta }$ represents the language model parameterized by $\theta$, and $\mathit{N}$ is the length of the generated text ${\mathit{Obs}}_r$. This State Commentary Guide module provides crucial support for the model to commentate in games of incomplete information. A theoretical guarantee for the mechanism design is provided in the following content.

The theory of mechanism design is crucial for the architecture of large language models (LLMs) framework design, as it provides a principled framework to align model outputs with desired objectives through incentive-compatible structures. Below is a detailed proof based on mathematical mechanism design, demonstrating that Compliant Commentary Generation is feasible.

Each state–rule pair $(s_t,R)$ is encoded as a vector $v_t$. It is then proven that the covered space $V\subset {\mathbb{R}}^d$ is compact (since both S and R are finite). By the universal approximation theorem [20], there exists a neural network ${\mathrm{LM}}_{\theta }$ such that:

$$\underset{v\in V}{sup}∥{\mathrm{LM}}_{\theta }\left(v\right)-\mathbb{P}(c\mid v)∥<\epsilon$$

(2)

where c denotes a Compliant Commentary Generation. The generation process forms a Markov chain whose stationary distribution $\pi \left(c\right)$ satisfies:

$$\pi \left(c\right)=\sum _v\mathbb{P}(c\mid v)\mathbb{P}\left(v\right)$$

(3)

After that, the convergence theorem is guaranteed by the Perron–Frobenius theorem [21],

$$\mathbb{P}\left({\mathbf{LM}}_{{\theta }^{*}}\left(s,{\mathit{His}}_r\right)\mathbf{is}\mathbf{compliant}\right)\ge 1-\epsilon .$$

(4)

Theorem 1.

Let S be a finite state space and R a finite rule set. Then, each state–rule pair $(s_t,R)$ can be encoded as a vector $v_t\in {\mathbb{R}}^d$, such that the space of all encoded vectors $V\subset {\mathbb{R}}^d$ is compact.

It is to be shown that the set

$$V=\{v_t\in {\mathbb{R}}^d\mid v_t\mathrm{encodes}\mathrm{the}\mathrm{state}-\mathrm{rule}\mathrm{pair}(s_t,R),s_t\in S,R\in \mathcal{R}\}$$

(5)

is compact, since both S and $\mathcal{R}$ are finite. The Cartesian product $S\times \mathcal{R}$ is finite. The encoding function

$$f:S\times \mathcal{R}\to {\mathbb{R}}^d,(s_t,R)\mapsto v_t$$

(6)

assigns a vector $v_t$ in ${\mathbb{R}}^d$ to each state–rule pair. Therefore, the set $V=f(S\times \mathcal{R})$ must be finite because it is the image of a finite set under the function f.

Any finite subset of ${\mathbb{R}}^d$ is bounded. In particular, there exists some $M>0$ such that for every $v_t\in V$, the norm satisfies $\parallel v_t\parallel \le M.$ A finite set has no limit points other than its own elements, and every sequence in a finite set eventually becomes constant (or repeats elements). Thus, all limit points of V are contained in V itself, implying that V is closed.

Then, in ${\mathbb{R}}^d$, the Heine–Borel theorem [22] tells us that a subset is compact if, and only if, it is closed and bounded. Since we have shown that V is both closed and bounded, it follows that V is compact.

Theorem 2.

Let $V\subset {\mathbb{R}}^d$ be a compact set, and let $\mathbb{P}(c\mid v)$ denote the conditional distribution over compliant commentary c given input vector $v\in V$. Then, by the universal approximation theorem [20], there exists a neural network ${\mathrm{LM}}_{\theta }$ such that:

$$\underset{v\in V}{sup}∥{\mathrm{LM}}_{\theta }\left(v\right)-\mathbb{P}(c\mid v)∥<\epsilon ,$$

(7)

for any $\epsilon >0$, where ${\mathrm{LM}}_{\theta }\left(v\right)$ approximates the target distribution $\mathbb{P}(c\mid v)$.

Assume that the target function $f\left(v\right)=\mathbb{P}(c\mid v),$ which returns the probability (or an appropriate representation) of a Compliant Commentary Generation c based on the encoding v, is continuous on the compact set V.

The universal approximation theorem (see [20]) asserts that for any continuous function defined on a compact subset of ${\mathbb{R}}^d$, and for any $\epsilon >0$, there exists a feedforward neural network ${\mathrm{LM}}_{\theta }$ with appropriate architecture and parameters $\theta$ such that

$$\underset{v\in V}{sup}∥{\mathrm{LM}}_{\theta }\left(v\right)-f\left(v\right)∥<\epsilon .$$

(8)

Substituting $f\left(v\right)=\mathbb{P}(c\mid v)$ yields

$$\underset{v\in V}{sup}∥{\mathrm{LM}}_{\theta }\left(v\right)-\mathbb{P}(c\mid v)∥<\epsilon .$$

(9)

Theorem 3.

Let c denote a compliant commentary generation. Suppose the generation process induced by the language model ${\mathrm{LM}}_{\theta }$ forms a Markov chain over the commentary space. Then, the stationary distribution $\pi \left(c\right)$ of this process is given by:

$$\pi \left(c\right)=\sum _v\mathbb{P}(c\mid v)\mathbb{P}\left(v\right),$$

(10)

where v denotes the encoded state–rule representations. Furthermore, by the Perron–Frobenius theorem [21], the Markov chain converges to $\pi \left(c\right)$, and the learned model ${\mathrm{LM}}_{{\theta }^{*}}$ satisfies:

$$\mathbb{P}\left({\mathrm{LM}}_{{\theta }^{*}}\left(s,{\mathit{His}}_r\right)\mathit{is}\mathit{compliant}\right)\ge 1-\epsilon .$$

(11)

Assume that the commentary generation process evolves over discrete time steps t such that the generated commentary $c_t$ only depends on the previous commentary $c_{t-1}$. Thus, the process forms a Markov chain with the state space being the set of all compliant commentaries. Let the transition probabilities be given by

$$\mathbb{P}(c^{\prime }\mid c)=\mathbb{P}(c_{t+1}=c^{\prime }\mid c_t=c),$$

(12)

there exists a unique stationary distribution $\pi \left(c\right)$ that satisfies

$$\pi \left(c^{\prime }\right)=\sum _c\pi \left(c\right)\mathbb{P}(c^{\prime }\mid c).$$

(13)

This stationary distribution represents the long-term behavior of the chain. Then, the Perron–Frobenius theorem applied to the transition probability matrix P (a non-negative matrix) guarantees that: (1) The spectral radius of P is 1. (2) There is a unique positive eigenvector (up to scaling) corresponding to the eigenvalue 1.

Assume that the training of the neural network ${\mathrm{LM}}_{{\theta }^{*}}$ is designed so that the target (or true) process overwhelmingly produces compliant commentaries. In other words, the intended stationary distribution $\pi \left(c\right)$ for a compliant commentary c satisfies

$$\pi \left(c\right)\ge 1-\epsilon .$$

(14)

This is a consequence of the training objective and approximation guarantees (e.g., from the universal approximation theorem), ensuring that the probability of compliance asymptotically is at least $1-\epsilon$.

Since the generation process is implemented via the neural network ${\mathrm{LM}}_{{\theta }^{*}}$, we have by the convergence property of the Markov chain:

$$\underset{t\to \infty }{lim}\mathbb{P}\left({\mathrm{LM}}_{{\theta }^{*}}(s,{\mathit{His}}_r)\mathrm{produces}c\right)=\pi \left(c\right).$$

(15)

Thus, when the system has run for a sufficiently long time, the probability that ${\mathrm{LM}}_{{\theta }^{*}}(s,{\mathit{His}}_r)$ produces a compliant commentary is at least $1-\epsilon$:

$$\mathbb{P}\left({\mathrm{LM}}_{{\theta }^{*}}(s,{\mathit{His}}_r)\mathrm{is}\mathrm{compliant}\right)\ge 1-\epsilon .$$

(16)

3.2. TOM-Based Strategy Analyzer

In the commentary of the card game, Guandan, the psychological tactics and strategic interactions within the game are difficult to accurately assess from a single player’s perspective. Therefore, we have adopted the ToM [4,8,23] approach to enhance the depth and strategy of the commentary. The commentary model is based on processing incomplete information of the game and inferring the psychological states of players, thereby providing accurate analysis and predictions in a complex multi-player interaction environment. For further information, refer to Algorithm 1.

• First-order ToM: We utilize first-order ToM for basic strategy analysis. By analyzing players’ past actions and the current situation, we infer the possible hand types each player may hold and analyze their potential strategies. This information is utilized to construct commentary content, explaining players’ actions and potential counter-strategies by opponents. For example, if a player consistently chooses to play certain cards, we can infer that they might hold strong cards and possibly intend to suppress their opponents.
  Second-order ToM: We introduce second-order ToM for a deeper level of strategy analysis. At this stage, we not only consider players’ strategies and actions but also predict opponents’ cognition and reactions to these strategies. Through this approach, we can interpret the game progression and players’ strategic tendencies more comprehensively. For instance, if a player adopts a relatively adventurous strategy in a certain situation, we may speculate that they believe their opponents would not anticipate this move, deliberately selecting this strategy.

Algorithm 1 Theory of Mind (ToM) Inference Step

Input:    obs (current game observations)    history (game history up to the current point)    player_id (ID of the player whose strategy is being inferred)   Output:    commentary (generated commentary based on ToM inference)   procedure tom_inference_step (obs, history, player_id)     first_order ← LLM.generate (“What is player player_id’s likely strategy?”, context = obs + history)     second_order ← LLM.generate (“What does player player_id think opponents       expect them to do?”, context = obs + history)     commentary ← integrate_to_commentary (first_order, second_order)     return commentary end procedure

To ensure the logical coherence and fluency of the commentary content, after stepwise reasoning on strategies, we introduce a Commentary Coordinator to review the output of the game commentary. The Commentary Coordinator is responsible for integrating various commentary parts, ensuring the content is both accurate and easy to understand, and seamlessly blending with the game progression. Through this method, we can provide audiences with deeper game commentary, allowing them to better understand the strategic competition and psychological tactics among players.

3.3. Style Retrieval and Extraction

We introduce a Style Retrieval and Extraction module specifically designed for the Guandan game to enhance the accuracy and relevance of information in the game commentary system. The module is divided into two main stages: data retrieval and information filtering.

• Data Retrieval: In the data retrieval phase of Guandan game commentary, we adopt a tree structure retrieval method to efficiently extract information most relevant to user queries from a corpus specifically designed for the Guandan game. In this process, each document is broken down into individual document nodes, each containing the content of the original document and a unique identifier, to facilitate subsequent vector indexing. The content of these document nodes is then converted into vector form and indexed using a vector space model.

During the query execution phase, the system first converts the user’s query request into vector form to ensure that the query can be effectively compared with the vectorized document nodes. Then, the system searches the constructed vector index, returning only document nodes whose similarity exceeds a preset threshold. Our threshold is set to 2, meaning that the system filters out nodes with a similarity greater than this value, ensuring that only the most relevant information is extracted.

• Information Filtering: In the information filtering stage, the model meticulously filters the retrieved data. The system examines the relevance of each data item, retaining only those that meet high standards for subsequent processing. The content outputted by the Style Retrieval and Extraction module will provide curated, relevant data to the Commentary Coordinator to support the analysis and explanation of game situations and player strategies during the commentary process.

4. Experiments and Result Analysis

4.1. Implementation Details

To study the performance of an LLM-based agent in Guandan commentary under conditions of incomplete information, lack of communication, and dynamic collaboration, we choose Guandan as the experimental environment. In our tests, we use Danzero+ [7] as the game agent. Danzero+ employs a distributed framework to train the reinforcement learning model using the deep Monte Carlo method. They demonstrate their capability to perform at a human level. All experiments are conducted in a Chinese environment. We conducted tests on both open-source and close-source language models, including OpenAI’s commercial model GPT-4 and GPT-3.5 [24], and Chinese-language open-source models like Qwen-32B [25], Yi-34B [26], ChatGLM-4 [27].

In the data processing stage, we first remove non-text elements and stopwords from the texts to reduce noise. Then, we conducted text normalization, including unifying the case of the text to eliminate differences caused by case inconsistency. We also applied part-of-speech tagging and tokenization techniques. Finally, we use the Porter Stemmer algorithm to simplify word inflections [28].

4.2. Dataset

Our dataset includes professionally validated commentary texts covering 250 game sessions (5832 segments) across eight Guandan variants.

• Professional commentary: Transcribed from match videos with two-stage validation:

• Technical accuracy (≥90% compliance) verified by game experts (2+ years experience)
• Strategic relevance scored by professionals (avg. 4.5/5)

These texts demonstrate high-level strategic insights and culturally authentic expression.

• Generated commentary: Produced by the commentary model based on actual match data, simulating the style and content of professional commentary.

4.3. Metrics

To comprehensively evaluate the quality and applicability of the commentary texts, we use the following metrics:

• Cosine Similarity: We employ the TF-IDF vectorization method [29] to convert processed texts into vector form and calculate the cosine similarity between professional and test texts. This measures the semantic closeness of professional and generated texts, reflecting the model’s capability to capture semantics.
  Sentiment Analysis: Through sentiment analysis [30] tools, we assign sentiment polarity scores to the texts to compare the emotional expression differences between the two types of texts.
  Lexical Diversity: We use the Type-Token Ratio (TTR) [31] to assess the lexical diversity of the texts. This metric measures the proportion of different words in the text relative to the total number of words, reflecting the richness and diversity of the language.
  SNOWNLP: The SNOWNLP [32] score is a sentiment score ranging from 0 (most negative) to 1 (most positive), computed using a Chinese text sentiment analysis library based on Naive Bayes.
  Human Evaluation: We also conduct human evaluations, including match consistency and fluency. These evaluations are completed by human reviewers to verify whether the commentary texts accurately reflect the actual card game situations and assess the readability and naturalness of the texts. Reviewers need to have a certain knowledge and experience of Guandan to accurately judge the accuracy of the text descriptions.

• Match Consistency
  • Key Event Identification (KEI): Assess whether the commentary texts capture key events in the match, such as major turnarounds or critical decision points [33].
  • Detail Accuracy: Check if the text descriptions of card types, scores, and player strategies are precise and correct.
• Fluency
  • Naturalness: Assess if the text language is smooth and free of grammatical errors or unnatural expressions.
  • Information Organization: Evaluate if the text’s information is properly organized and can be understood in a logical order throughout the development of the match.
  • Logical Coherence: Check if the narrative of events in the text is coherent and free from logical jumps or contradictions.

4.4. Results

In this experiment, we conduct a comparative analysis of the data presented in

Table 1. Evaluation results of commentary for open source and closed source models.

	Sentiment Analysis				Cosine Similarity	Lexical Diversity	SNOWNLP
	Neg	Neu	Pos	Compound
GPT-3.5	0.0	1.0	0.0	0.0	0.0032	0.09	0.0
GPT-4	0.0	1.0	0.0	0.0	0.0380	1.0	0.0
Yi-34B	0.0	1.0	0.0	0.0	0.0250	0.55	0.99
GLM-4	0.0	1.0	0.0	0.0	0.0050	0.86	0.98
Our	0.0	1.0	0.0	0.0	0.7955	0.95	0.0
Original	0.0	1.0	0.0	0.0	-	1.0	0.0

, highlighting the substantial benefits gained from utilizing the Retrieval-Augmented Generation (RAG) framework for commentary. The integration of techniques such as card memory and the retrieval mechanism in RAG results in a more authentic and professionally styled commentary, offering key improvements over standard models. Even with its larger parameter count, GPT-4 without retrieval underperforms compared to RAG-enabled models. The retrieval component enables commentators to provide more accurate analyses and predictions, resulting in deeper insights into match progress and a stronger emotional connection to the content.

For performance metrics: Accuracy refers to the correctness of the generated commentary in describing game elements (e.g., card types, scores, and player strategies), measured by our Detail Accuracy metric; Match Consistency evaluates logical progression across game states and alignment with previous moves/decisions, quantified through Key Event Identification (KEI) and Detail Accuracy metrics; Fluency assesses linguistic presentation quality, including naturalness, information organization, and narrative coherence. These metrics are deliberately separated because they measure fundamentally distinct aspects: Match Consistency focuses on factual/logical correctness in the game context, while Fluency evaluates linguistic presentation quality. This distinction follows established evaluation frameworks for game commentary systems [5,33].

When applied to card game commentary, such as Guandan, RAG utilizes Chain-of-Thought (CoT) retrieval, which further enhances the model’s ability to emulate the linguistic norms of expert commentators. Models equipped with retrieval features outperform large models without retrieval or those fine-tuned solely on commentary data.

All evaluated models exhibit neutrality in sentiment analysis. GPT-3.5, for instance, shows relatively low lexical diversity (0.09), whereas Qwen equipped with RAG and GPT-4 demonstrate higher diversity in the generated text. The near-perfect SnowNLP score (0.99) for the Yi-34B model does not indicate a clear advantage or reliable ability to mimic the commentary. Additionally, all models struggle with cosine similarity when compared to the original text, likely due to the improvisational nature of live commentary. Among them, Qwen with RAG exhibits a slight advantage in aligning with the source content.

The theoretical guarantees are empirically validated using the following methods: The Compliance Verification method involves human evaluation of detail accuracy (measuring accuracy), which scores 0.97 (as shown in

Table 2. Human evaluation of different benchmarks and our model.

	Match Consistency			Fluency
	KEI	Detail Accuracy		Naturalness	Information Organization	Logical Coherence
GPT-3.5	0.37	0.95		0.86	0.69	3.75
GPT-4	0.46	0.91		0.90	0.80	4.46
Yi-34B	0.23	0.87		0.82	0.62	3.52
GLM-4	0.32	0.83		0.84	0.58	2.72
Our	0.81	0.97		0.95	0.89	4.34

), confirming the $\epsilon$-bound compliance. The Vector Convergence method shows that a cosine similarity of 0.7955 (compared to 0.038 for GPT-4) demonstrates the preservation of compact encoding $v_t$.

• Human Evaluation Analysis. We recruit 20 human annotators with Guandan experience for scoring. As shown in Table 2, in terms of match consistency, our model significantly outperforms other models, particularly excelling in Key Event Identification (KEI) [34]. This demonstrates that our model accurately captures crucial moments in the game, effectively reflecting the game’s turning points and climactic sections. In Detail Accuracy, our model also performs exceptionally well, accurately describing game elements such as card types, scores, and strategic actions. Regarding fluency, our framework scores high in naturalness (0.95), information organization (0.89), and logical coherence (4.34), highlighting the model’s ability to generate commentary text that is grammatically correct, logically structured, and contextually coherent.

In contrast, other models like GPT-4, despite performing well in general language generation tasks, often fail to show the same acuity in specialized game commentary scenarios. These models are often not sensitive enough to key events, sometimes missing significant turning points in the game or failing to fully utilize game-specific terminology and expressions. Additionally, in terms of fluency, although they can generate structurally sound sentences, they sometimes lack the ability to present information in a logical and captivating manner.

• Case Study. As shown in Figure 2, our method not only focuses on the types of cards played but also delves into detailed analysis of the playing patterns and potential strategies of the players and their opponents. It provides recommendations based on the current and predicted game state, advising when to play high or low cards. This includes discussions on when to play single cards or pairs to maximize gameplay advantage, helping players not only understand the game rules but also master advanced strategies to enhance their gameplay level. In contrast, the Yi-34B model’s commentary seems to focus more on the sequence of cards played without deeply exploring the reasoning or strategic implications behind these choices. It lists actions such as leading with certain cards or choosing to pass, which, while informative, do not delve into the strategic significance of these decisions. See Appendix B for more output examples.

4.5. Ablation Studies

• Ablation Studies on RAG and ToM.

  Table 3. Ablation study on the effect of the Tree-Based Retrieval Method (RAG) in commentary generation. The table compares different configurations of the system, including models with and without RAG and varying levels of Theory of Mind (ToM). The evaluation includes sentiment analysis (negative, neutral, positive, and compound scores), cosine similarity (measuring alignment with the original commentary), lexical diversity (indicating vocabulary richness), and SNOWNLP score (sentiment analysis score). Rows represent the following configurations: (1) Our(w/o RAG)(Vanilla): base model without RAG; (2) Our(w/o RAG)(1st-ToM): base model with first-order ToM; (3) Our(w/o RAG)(2nd-ToM): base model with second-order ToM; (4) Our(w RAG)(1st-ToM): model with RAG and first-order ToM; (5) Our(w RAG)(2nd-ToM): model with RAG and second-order ToM; and (6) Original: the reference commentary. Higher cosine similarity and lexical diversity indicate better performance in mimicking human-like commentary.

  	Sentiment Analysis				Cosine	Lexical	SNOWNLP
  	Neg	Neu	Pos	Compound	Similarity	Diversity
  Our(w/o RAG)(Vanilla)	0.0	0.0	1.0	0.0	0.0	0.87	1.0
  Our(w/o RAG)(1st-ToM)	0.0	0.0	1.0	0.0	0.0126	1.0	1.0
  Our(w/o RAG)(2nd-ToM)	0.0	1.0	0.0	0.0	0.0380	1.0	1.0
  Our(w RAG)(1st-ToM)	0.0	1.0	0.0	0.0	0.7519	0.92	0.0
  Our(w RAG)(2nd-ToM)	0.0	1.0	0.0	0.0	0.7955	0.95	0.0
  Original	0.0	1.0	0.0	0.0	-	1.0	0.0

  shows that removing RAG (e.g., “Our(w/o RAG)(Vanilla)”) results in a cosine similarity of 0.0 with the original text, despite producing the maximum SNOWNLP score of 1.0 and a reasonably high lexical diversity (0.87 or 1.0). This outcome implies that, without retrieval, the generated text deviates significantly from the source content, even though it may appear stylistically diverse. In contrast, when RAG is introduced (“Our(w RAG)(1st-ToM)” or “Our(w RAG)(2nd-ToM)”), the cosine similarity rises markedly (up to 0.7955), indicating stronger alignment with the original text. However, this improvement is accompanied by a slight decrease in both lexical diversity and SNOWNLP scores, suggesting that retrieval imposes some constraints on free-form generation. Overall, these results underscore the importance of RAG for achieving higher semantic fidelity in commentary.
  Ablation Result Analysis. The RAG model with the retrieval component displays significant disparities from the model lacking it across various dimensions. In terms of lexical diversity, the model incorporating retrieval exhibits somewhat constrained diversity, suggesting a potential limitation imposed by the retrieval component. Sentiment analysis using the SnowNLP tool reveals that the model without retrieval yields more pronounced sentiment results, diverging notably from the original text. This deviation may arise from the model’s greater freedom in generating emotional expressions, albeit resulting in a less faithful imitation of the source text. Conversely, regarding text semantic similarity, the model integrating retrieval showcases a distinct advantage, effectively highlighting the crucial role of the retrieval component in maintaining semantic coherence and bolstering text relevance.

In addition to retrieval, second-order ToM demonstrates a clear advantage over first-order ToM in semantic alignment. Without RAG, second-order ToM improves cosine similarity from 0.0126 to 0.0380, and, with RAG, from 0.7519 to 0.7955. These gains indicate that higher-order ToM facilitates a deeper understanding of strategic interactions and player intentions, enhancing the commentary’s accuracy. In Guandan, for instance, second-order ToM captures opponents’ potential counteractions and anticipations, resulting in more precise narration of pivotal turning points. As such, second-order ToM is instrumental in producing in-depth and contextually grounded game commentary.

5. Conclusions

In this study, we introduce a novel commentary framework that combines reinforcement learning and large language models to generate detailed, context-relevant commentary for the complex, incomplete information card game Guandan. Our modular approach, consisting of a State Commentary Guider, ToM Strategy Analyzer, and Style Retrieval module, enables the LLM to deliver insightful game commentary without the need for specialized training. Experimental results demonstrate that our commentary framework significantly enhances the performance of open-source LLMs, outperforming GPT-4 on multiple evaluation metrics. In the future, we aim to extend our commentary framework to other complex games and explore the integration of additional modalities, such as audio and video data, to further enrich the commentary generation process.

6. Code Availability

Our code is available at https://github.com/heimy2000/guandan, accessed on 1 July 2025.