Enhancing Multi-Agent Reinforcement Learning via Knowledge-Embedded Modular Framework for Online Basketball Games

Abstract

High sample complexity presents a major challenge in applying multi-agent reinforcement learning (MARL) to dynamic, high-dimensional sports such as basketball. To address this problem, we proposed the knowledge-embedded modular framework (KEMF), which partitions the environment into offense, defense, and loose-ball modules. Each module employs specialized policies and a knowledge-based observation layer enriched with basketball-specific metrics such as shooting success and defensive accuracy. These metrics are also incorporated into a dynamic and dense reward scheme that offers more direct and situation-specific feedback than sparse win/loss signals. We integrated these components into a multi-agent proximal policy optimization (MAPPO) algorithm to enhance training speed and improve sample efficiency. Evaluations using the commercial basketball game Freestyle indicate that KEMF outperformed previous methods in terms of the average points, winning rate, and overall training efficiency. An ablation study confirmed the synergistic effects of modularity, knowledge-embedded observations, and dense rewards. Moreover, a real-world deployment in 1457 live matches demonstrated the robustness of the framework, with trained agents achieving a 52.43% win rate against experienced human players. These results underscore the promise of the KEMF to enable efficient, adaptive, and strategically coherent MARL solutions in complex sporting environments.

1. Introduction

Recent advances in artificial intelligence (AI) have led to the widespread application of reinforcement learning (RL) in various domains, including gaming, finance, robotics, and autonomous driving [1,2,3,4]. RL is a learning paradigm whereby an agent interacts with its environment and acquires an optimal policy by repeatedly taking actions and receiving rewards, ultimately aiming to maximize the cumulative rewards [5]. Game environments that allow large-scale experimentation in virtual settings and relatively safe agent training have consequently garnered considerable attention as representative applications of RL [6]. In this context, notable achievements have been reported in diverse genres—including strategy simulations, board games, and shooting games—leading to growing interest in RL-based technologies in both the gaming industry and academia [7,8,9].

Deep reinforcement learning (DRL), which combines deep neural networks with RL algorithms, has achieved superhuman performance in various challenging environments, including board games such as Go [10,11] and Chess [11], video games such as Atari [12] and StarCraft II [1], and complex strategy games such as Dota 2 [7]. In multi-agent settings, multi-agent reinforcement learning (MARL) can extend these capabilities by enabling multiple autonomous agents to learn cooperative or competitive behaviors through environmental interactions [13]. Recent advances in MARL have demonstrated considerable potential in team-based scenarios where coordination, communication, and strategic planning are essential for success [14,15].

Among various game genres, sports games enjoy global popularity [16]. In particular, basketball—alongside soccer—has been studied extensively as a representative environment for RL applications in sports gaming. Notably, as game environments become increasingly realistic owing to technological advances, incorporating knowledge-based data from actual game competitions can offer considerable advantages when training RL agents in these virtual settings. Conversely, game AI developed through RL not only provides new experiences across the broader gaming industry but can also demonstrate the potential for “real-world match” applications, similar to the examples of Chess and Go [10,11].

Despite these benefits, the application of RL in basketball game environments remains relatively underexplored. Compared with more commonly studied RL domains, sports game environments, such as soccer and basketball, typically exhibit higher-dimensional state-action spaces, rapidly changing dynamic states, stochastic elements, and sparse events, all of which can contribute to high sample complexity in learning and, thereby, lower overall sample efficiency [17]. Moreover, research on multi-agent, full-game environments suitable for real-world online commercial applications is still in its infancy. Nonetheless, recent advances in transformer-based multi-agent reinforcement learning (MARL) [13,18] have shown promise in handling complex sequential dependencies, although they can introduce computational overhead and latency challenges that may be incompatible with real-time gaming requirements [19].

In this study, we identified high sample complexity as the principal challenge in team sports games and proposed a knowledge-embedded modular RL framework to address this problem. Specifically, our proposed method mitigates the high-dimensional state-action space by modularizing the model for each continually changing game context. Learning outcomes are further improved by introducing observation-data optimization (via knowledge-based data) and a dense reward structure for each module. The utilization of knowledge-based data refers to extracting global task-level semantic contexts—such as strategically or tactically significant metrics (e.g., shot success probability and defensive accuracy)—from raw game data collected by a simulator, and then integrating this context into the agent’s observation space. For example, a “shot success probability,” not explicitly contained in the raw data, can be indirectly inferred through multistage feature engineering as well as statistical or regression techniques, drawing on fundamental coordinate information such as the rim location, agent location, and defender location. This process enables the agent to acquire more refined tactical knowledge—such as the likelihood of a shot succeeding at a specific distance or angle. To design a dense reward structure, we extended beyond reward schemes that relied solely on scoring or winning by subdividing and quantifying the quality of intermediate decisions, thereby providing more granular feedback. This approach ensured that agents received concrete guidance, even during the early stages of random exploration, ultimately enhancing both the convergence speed and stability of the learned policy.

More specifically, the proposed knowledge-embedded modular framework (KEMF) was implemented through the following pipeline: (1) The environment can be partitioned into three distinct modules—that is, the offense, defense, and loose-ball modules—each governed by specialized policies trained independently before integration, based on modular RL principles [13,20,21]. (2) A knowledge-based observation layer can extract domain-specific features including the shooting success probability, defensive accuracy metrics, and spatial relationships, inspired by sports analytics methods [22,23,24]. (3) Each module can employ tailored reward functions with metrics such as the shooting success rate (for offense), defensive interception accuracy (for defense), and ball recovery rate (for loose-ball), building on reward shaping techniques [25,26]. (4) A situation determinator can check the predefined conditions in the environment and switch to the appropriate contextual module at each timestep [27]. (5) The dynamic dense reward mechanism can provide more informative feedback than sparse win/loss signals by correlating intermediate decisions with semantically enriched feedback [28,29].

The key contributions can be summarized as follows:

(1)

Situation-Specific Modular Reinforcement Learning: By partitioning various game states in team sports into individual sub-models for training, we can reduce the complexity of high-dimensional state-action space and facilitate the learning of optimal policies for each situation. This approach mitigates the impact of high sample complexity and enables specialized decision making in different contexts.

(2)

Knowledge-Embedded Observation Layers: Through multistage feature engineering of the raw game data, we can derive a high-dimensional semantic context—such as shot success probability—and incorporate it into the agent’s modularized observation space. This domain-specific state representation can help to counter low sample efficiency by allowing the agent to learn more effectively with limited data.

(3)

Dynamic and Dense Reward Scheme: To complement the sparse rewards offered by simple metrics (such as scoring or winning), we can assign more finely grained rewards based on the tactical indicators extracted in (2). This enables the agent to receive immediate feedback from the outset of training, thereby improving the sample efficiency and aligning more closely with knowledge-based data for stable and rapid policy formation.

In this study, we applied the proposed method to a multi-agent proximal policy optimization (MAPPO) algorithm [14]. To validate our framework, an ablation study was conducted in a commercial online basketball game environment—namely, JoyCity’s “Freestyle”—where we evaluated and analyzed trained models in terms of the winning rate, points earned, and action-level performance. These experimental results confirmed the effectiveness of the proposed method.

2. Background

2.1. Sample Complexity in Reinforcement Learning and Sports Game Environment

In RL, sample complexity typically refers to the number of environment interactions required for an agent to reach optimal (or ϵ-approximately optimal) performance [30]. A key factor driving sample complexity is the intrinsic difficulty of the underlying environment, which may be termed environment complexity. In a tabular setting, this environment complexity can be approximated by the following simplified measure [31]:

$$C(E)={\displaystyle \frac{\left|S\right| \cdot \left|A\right|}{{(1 - \gamma )}^p}},$$

(1)

where ∣S∣ and ∣A∣ represent the sizes of the state and action spaces, respectively, γ denotes the discount factor, and p denotes a positive constant reflecting how the horizon scales with (1 − γ). Intuitively, this expression indicates that larger state-action spaces and longer planning horizons (i.e., γ ≈ 1) make the environment harder to learn, resulting in higher sample complexity.

Although the basic environment complexity C(E) can be approximated by Equation (1) in a tabular setting, sports game environments (such as basketball), present substantially higher complexity owing to several factors. For example, the theoretical foundations of this study’s complexity analysis were grounded upon recent advances in multi-agent RL theory. Zhang et al. [32] established the first near-optimal sample complexity bounds for multi-agent RL in zero-sum Markov games with formal mathematical proofs, whereas recent breakthrough work by Zhang et al. [33] achieved minimax-optimal regret bounds in online RL. These theoretical frameworks provide a mathematical foundation for understanding the complexity factors in competitive multi-agent environments (such as basketball). It is important to note that the complexity formulations presented below serve as conceptual frameworks for understanding the factors contributing to sample complexity rather than formal theoretical bounds. These provide intuitive illustrations of how various problems can compound in sports environments.

One major source of increased complexity is the multi-agent nature of team sports, whereby each agent (player) operates with its own state and action space. In scenarios with n agents, these spaces combine to form a joint representation of size ∣$S_1$∣ ×, …, × ∣$S_i$∣ and ∣$A_1$∣ ×, …, × ∣$A_i$∣. Consequently, the complexity of a multi-agent environment can be roughly modeled as follows:

$$C_{multi}(E)\approx {\displaystyle \frac{\prod _{i=1}^n\left|S_i\right| \cdot \prod _{i=1}^n\left|A_i\right|}{{(1-\gamma )}^p}}.$$

(2)

In Equation (2), we model the complexity of a multi-agent environment as the product of individual agent complexities. Let $\left|S_i\right|$ and $\left|A_i\right|$ denote the state and action space sizes for agent i, respectively. The multiplicative formulation (2) captures the combinatorial explosion arising from joint state-action spaces when n agents operate simultaneously [13,32]. This exponential growth in complexity—as opposed to additive scaling—reflects the fundamental problem that each agent must reason about the behaviors of all other agents, leading to a joint space that grows multiplicatively [33]. The discount factor γ and horizon parameter p are inherited from the single-agent formulation [30,31], indicating that longer planning horizons can further compound this multi-agent complexity.

Moreover, basketball games involve rapidly shifting tasks—the agent may transition from offense to defense or even to a loose-ball situation—unlike many RL tasks with relatively static objectives. To account for these transitions, it is useful to view the game as a collection of subtasks {$E_1$, $E_2$, … $E_k$}. If $a_i$ denotes the fraction of time spent in subtask $E_i$ and C($E_i$) denotes its complexity, an aggregated metric can be written as follows:

$$C_{context}(E)\approx {\sum }_{i=1}^k{a}_i\cdot C\left(E_i\right).$$

(3)

Another challenge arises from the sparse reward signals, where scoring or winning may be the primary source of the reward. In such cases, agents—particularly off-ball players—receive little immediate feedback, complicating the credit assignment process. To incorporate the notion that a reduced reward density ($d$ → 0) elevates the sample complexity, a reward sparsity factor β($d$) can be introduced that increases as $d$ decreases, yielding:

$$C_{sparse}\left(E\right)\approx \beta \left(d\right)\cdot {\displaystyle \frac{\left|S\right| \cdot \left|A\right|}{{(1-\gamma )}^p}}.$$

(4)

Finally, stochasticity introduced for realism or entertainment can further complicate learning. In several sports simulations, even identical states and actions can result in different outcomes—for example, a shot may succeed with probability n%. This randomness amplifies variance in transitions and rewards, effectively broadening the exploration horizon of an agent. A variance factor σ can be used to capture this effect, leading to:

$$C_{var}\left(E\right)\approx \sigma \cdot {\displaystyle \frac{\left|S\right| \cdot \left|A\right|}{{(1-\gamma )}^p}}.$$

(5)

To represent the combined impact of these factors—namely multi-agent joint spaces, shifting subtasks, sparse reward signals, and stochastic outcomes—it is helpful to define a composite complexity measure. For a multi-agent, multi-context case in a basketball environment, the following conceptual form can be illustrated:

$$\tilde{C}\left(E\right)\approx \left[\beta \left(d\right)\cdot \sigma \right]\cdot {\sum }_{i=1}^k{a}_i\cdot {\displaystyle \frac{{\prod }_{j=1}^n\left|S_i^j\right| \cdot \prod _{j=1}^n\left|A_i^j\right|}{{(1-\gamma )}^{p_i}}},$$

(6)

where $a_i$ denotes the fraction of time in subtask $E_i$, $\beta \left(d\right)$ denotes the reward sparsity, σ denotes the degree of randomness, and ${(1-\gamma )}^{p_i}$ encodes the effective horizon in each subtask. The multiplicative relationship between β(d) and σ can be theoretically justified by recent advances in RL complexity analysis. Shi & Chi [34] demonstrated that complexity factors interact multiplicatively in fundamental bounds for distributionally robust RL, whereby the state space complexity, concentrability, discount factor, and accuracy all appear as multiplicative terms. This multiplicative structure can be further supported by concentration inequalities and minimax lower bounds that consistently exhibit a multiplicative form—that is, Ω(d·σ²/ε²) rather than additive Ω(d + σ²/ε²) [35]. The multiplicative relationship reflects the compounding effect of these challenges, whereby both sparse rewards and stochastic outcomes can simultaneously increase the learning difficulty. Although these formulations are largely conceptual, they illustrate how team sports environments can exhibit substantially higher sample complexity than various standard RL tasks.

In addition to the aforementioned mathematical formulations, numerous studies on the application of RL to sports game environments have indicated that the inherent complexity of these environments can make convergence to an optimal policy challenging. For example, in Google Research Football [36], which is widely recognized as a representative sports game environment for RL research, Zhao et al. [37] identified multi-agent interactions, sparse rewards, and stochastic dynamics as primary obstacles. Similarly, Scott et al. [38] emphasized multi-agent complexity, low scoring frequency, and constant situational changes as factors that can further increase the difficulty of learning optimal policies.

Although fewer studies have been conducted on basketball game environments than on soccer game environments, basketball games are considered to have high complexity owing to the multi-agent dynamics, shifting objectives (contexts), sparse rewards, and stochastic factors [39,40]. To train agents in basketball tasks, Choi et al. [41] employed an A2C algorithm enhanced by behavioral cloning from expert policies, whereas Lee et al. [42] adopted a QMix algorithm based on human gameplay data. Both methods produced [39] meaningful results to a certain extent; however, they were limited by their reliance on external resources, such as expert policies and user data. Moreover, neither study addressed the training efficiency, particularly the amount of time required for the algorithm to converge. A similar problem was observed in Jia et al. [43], where even the proposed curriculum learning method required the creation of new environments, ultimately resulting in additional time and effort.

2.2. Multi-Agent Proximal Policy Optimization (MAPPO)

Proximal policy optimization (PPO) is a policy-based algorithm introduced by OpenAI in 2017 [44] and has been widely employed in various RL studies [45,46,47]. The core idea of PPO is to update the policy parameters in a manner that avoids excessively large policy changes, thereby striking a balance between sufficient exploration and stable performance. Let θ denote the current policy parameters and ${\theta }_{old}$ the old policy parameters; PPO maximizes the following clipped surrogate objective:

$$L^{clip}\left(\theta \right)={\widehat{\mathbf{E}}}_t\left[\mathrm{m}\mathrm{i}\mathrm{n}\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\left(r_t\left(\theta \right), 1-ϵ, 1+ϵ\right){\widehat{A}}_t\right)\right],$$

(7)

where

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t \right| s_t)}{{\pi }_{{\theta }_{old}}\left(a_t \right| s_t)}}.$$

(8)

Here, ${\widehat{A}}_t$ denotes an estimator of the advantage function at time t, and ϵ denotes a hyperparameter (often 0.1 or 0.2) that defines the clipping range. This design prevents the policy from deviating too far from the previous one, promoting stable and reliable learning. The algorithm alternates between collecting trajectories through environment interaction and performing multiple epochs of stochastic gradient ascent on $L^{clip}\left(\theta \right).$

Although the PPO algorithm is designed for single-agent settings, multi-agent PPO (MAPPO) [14] can extend PPO to multi-agent tasks using a centralized critic while maintaining decentralized actors (Figure 1). Consider an environment with n agents, each observing $o_t^i$ at time $t$ and taking an action $a_t^i$. Let $\theta =\{{\theta }_1,{\dots , \theta }_n\}$ denote the set of policy parameters for all agents and $\mathrm{\varnothing }$ represent the parameters of a centralized value function $V_{\varnothing }\left(o_t\right)$, conditional on the joint observation $o_t=\left(o_t^1,\dots ,o_t^n\right)$ or the global state $s_t$. In MAPPO, the critic can be trained to minimize the value loss in the form:

$$L_{MAPPO}^{CLIP}\left(\theta \right)={\widehat{\mathbf{E}}}_t[\sum \mathrm{m}\mathrm{i}\mathrm{n}\left(r_t^i\right(\theta ){\widehat{A}}_t^i,\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}(r_t^i\left(\theta \right),1-ϵ, 1+ϵ){\widehat{A}}_t^i)],$$

(9)

where

$$r_t^i\left(\theta \right)={\displaystyle \frac{{\pi }_{{\theta }_i}\left(a_t^i \right| o_t^i)}{{\pi }_{{\theta }_i^{old}}\left(a_t^i \right| o_t^i)}}, {\widehat{A}}_t^i=Q_{\varnothing }\left({\mathit{o}}_t, {\mathit{a}}_t\right)-V_{\varnothing }\left({\mathit{o}}_t\right).$$

(10)

Here, ${\widehat{A}}_t^i$ denotes an advantage estimate for agent i, typically computed through a centralized action-value function $Q_{\mathrm{\varnothing }}$ or other multi-agent advantage estimator. The clip function, parameterized by ϵ, prevents excessively large updates—thus preserving the stability and monotonic improvement properties of PPO.

Because of its ability to handle multi-agent coordination while retaining PPO’s stable policy updates, MAPPO has demonstrated leading performance on several cooperative benchmarks. In particular, when evaluated on the Google Research Football task—widely used for multi-agent RL—MAPPO has demonstrated superior sample efficiency and higher final performance compared to a variety of MARL algorithms [14]. Moreover, Song et al. [15] reported that for complex tasks, such as sports games, policy-based algorithms tend to outperform value-based methods at a fundamental level.

Building on these insights, we adopted MAPPO as the foundational algorithm for our basketball environment and integrated it with a modular architecture, knowledge-based observation design, and dynamic dense reward structure (as detailed in the following section). This approach leverages MAPPO’s proven stability and coordination benefits while directly addressing the high dimensionality, frequent context switches, and sparse feedback challenges inherent to sports game tasks.

Figure 1. Multi-agent proximal policy optimization (MAPPO) structure.

Figure 1. Multi-agent proximal policy optimization (MAPPO) structure.

Although recent advances in large language models (LLMs) and federated learning have shown promise in various domains, these methods can present considerable challenges for real-time gaming applications. LLMs typically require substantial computational resources and introduce latency that is incompatible with the millisecond-level response times demanded by competitive gaming environments [48]. Similarly, federated learning methods, although valuable for privacy-preserving distributed training, can introduce communication overhead and synchronization delays that are unsuitable for real-time decision making [49].

2.3. Modular Reinforcement Learning

Modular RL can decompose a complex environment into multiple more tractable subtasks, each handled by a specialized sub-policy [50]. By isolating functional or contextual components—such as the offense and defense phases in sports games—this method effectively localizes the state-action space, mitigating the problem of dimensionality and improving the sample efficiency [51]. Previous studies have demonstrated that monolithic policies trained in vast heterogeneous environments can often struggle with convergence owing to the difficulty in managing diverse dynamics and reward structures. By contrast, modular methods enable each sub-policy to focus on a narrower range of scenarios, facilitating more stable gradient updates and reducing unproductive exploration [52].

Although hierarchical RL (HRL) and option-based methods can also pursue task decomposition, they typically employ a vertical structure whereby high-level policies invoke temporally extended lower-level behaviors. By contrast, modular RL adopts a horizontal partitioning strategy, assigning different subtasks or contexts to distinct modules without requiring a top-down control hierarchy. Although modular RL can incorporate HRL-like components if necessary, its emphasis is on context-based module selection and parallel specialization rather than purely temporal abstraction.

From a theoretical perspective, the complexity of a modularized environment E can be approximated by summing the complexities of each individual module. Suppose we partition E into a set of modules $\{M_1,$ …, $M_n\}$, each associated with a distinct subtask or context. Let $a_i$ denote the fraction of time or probability the agent spends in module $M_i$ and $C\left(M_i\right)$ denote the effective complexity of module $M_i$. The aggregate complexity can then be expressed as:

$$C_{module}(E)\approx {\sum }_{i=1}^k(a_i \cdot C\left(M_i\right)).$$

(11)

This indicates that each module must handle only a subset of the overall problem space. Because the dimensionality of each subtask’s state-action distribution is considerably smaller than that of the full environment, modular RL can achieve reduced sample complexity compared to monolithic methods. Moreover, context detection—whether implemented through rule-based heuristics or learned gating mechanisms—enables dynamic module selection, ensuring that sub-policies are trained only on relevant data [27]. Jeon et al. [20] proposed a modular RL approach to the Game of Tron, decomposing the gameplay into two distinct phases—that is, normal and abnormal phases—and significantly improving the sample efficiency. Yu et al. [53] proposed a modular RL framework to enhance the low-level control performance of a quadrotor, particularly by directly controlling the yaw motion, thereby improving the sample efficiency. In StarCraft II AI, Lee et al. [21] proposed a modular architecture that decomposed a game into several subtasks handled by independent modules trained using RL.

Similarly, in our basketball game environment, we divided the gameplay into distinct contexts (e.g., offense, defense, and loose-ball situations) and assigned each context to a specialized module. By focusing on these narrower subtasks, the state-action space for each module remained more tractable, ultimately reducing the sample complexity and leading to faster convergence. Although additional engineering is required to define module boundaries and coordinate transitions, the resulting flexibility and scalability render modular RL a compelling framework for addressing highly complex multi-agent environments, such as basketball games.

2.4. Knowledge-Based Feature Engineering

Recent advances in modular and distributed learning have further demonstrated the effectiveness of decomposition strategies. For example, research on efficient federated learning with adaptive model pruning and momentum-based batch adjustment [54] has shown how modular methods can improve the training efficiency in distributed settings. Additionally, studies on reinforcing neural network stability with attractor dynamics [55] provide a theoretical foundation for understanding stable policy learning in modular architectures. Moreover, recent work on the surprising effectiveness of PPO in cooperative multi-agent games [14] and empirical studies on multi-agent scenarios in sports environments [15] have validated the potential of policy-based methods in complex team settings.

In most academic studies on RL, it has been customary to provide the agent with rich state information alongside sparse global rewards to attain broad generalization capabilities on its own. Although this approach can facilitate sophisticated performance and extensive generalization, it can also lead to a substantial increase in the time required for convergence, primarily because of the limitations in sample efficiency [56]. This problem can become more pronounced as an agent’s action space expands [57]. Although techniques, such as using auto-encoders, can be employed to automatically extract pertinent features within a neural network, they ultimately introduce additional layers and learning processes.

From the perspective of companies deploying AI in real-time commercial services, the elevated computational overheads and prolonged convergence times have become critical bottlenecks. These factors can result in increased resource consumption and operational costs, imposing considerable constraints on real-time updates and reliable service provision.

Consequently, achieving rapid and stable convergence often requires more than offering abundant state information with sparse rewards, making it crucial to design both the observation space and reward structure for enhanced specificity to a given task. A representative strategy involves complementing the raw data originally used to provide abundant state observations with a knowledge-based methodology that introduces domain-informed features directly relevant to the task. Liu et al. [58] highlighted the underutilization of soccer-specific domain knowledge in complex full-length soccer games. In the Google Research Football environment, Song et al. [59] demonstrated a considerable performance boost by incorporating inter-entity information such as distances and angles into baseline raw observations, compared to relying exclusively on default observations.

This domain knowledge can also be instrumental in reward design. For example, in the multi-objective beach problem domain, Mannion et al. [25] demonstrated that providing different rewards that reflected each agent’s contribution, or employing potential-based rewards that encoded state preferences, could effectively guide exploration and enhance learning outcomes. Similarly, Abe et al. [28] observed that sparse rewards alone were insufficient for efficient cooperative behavior acquisition in a soccer environment. By employing reward shaping, they substantially improved the success rate at which low-level motions and cooperative behaviors were learned. In a full-game StarCraft scenario, Liu et al. [29] analyzed expert replays and integrated heuristic signals favoring objective attainment into a reward structure, which subsequently yielded a high win rate against the built-in AI.

In our proposed method, we embedded domain-specific knowledge—such as estimated shooting success probabilities, defensive accuracy metrics, and context-specific positional relationships—directly into both the observation space and reward function. For example, an agent on offense could receive features indicating the best shooter on its team or the distance and angle to the rim, thus enabling more informed pass or shot decisions. In defense, the agent could observe its target’s shooting success probability and defensive accuracy requirements, guiding better positioning and timely stealing or blocking attempts. We further shaped the reward function to provide dense feedback based on these metrics. For example, when moving closer to an optimal defensive stance or passing to a high-probability shooter, the agent could receive incremental rewards proportional to the measured improvement. This knowledge-driven approach helped agents learn more efficiently by reducing unproductive exploration and aligning intermediate decisions with domain expertise.

3. Methods

In this section, inspired by previous research aimed at improving the sample efficiency under complex, high-dimensional conditions, we propose the KEMF, specifically tailored for multi-agent basketball game environments (Figure 2). The principal objective of the KEMF is to embed domain knowledge directly into the training pipeline while partitioning the environment into context-specific submodules, thereby mitigating high sample complexity inherent in dynamic multi-context tasks.

First, building on modular RL methods that demonstrate how breaking down large problems into discrete submodules can significantly enhance the sample efficiency [20,21,53], we devised a modular architecture that divided the game into major contexts, such as offense, defense, and loose-ball situations. Second, drawing on prior work emphasizing the effectiveness of knowledge-based feature engineering in sports simulations [48,49], we embedded domain-informed features—such as the shooting success probabilities, defensive accuracy, and spatial relationships—into each submodule’s observation space, thereby providing more targeted and semantically rich state information. Finally, reflecting on the merits of context-specific reward shaping in multi-agent and sports-based RL [15,28,29], we introduced a dynamic dense reward structure tailored to each situation, designed to guide the agent from action-level decisions to the final objective of winning. By integrating these techniques into MAPPO, a modern multi-agent policy-gradient algorithm, we could streamline unnecessary exploration steps and accelerate overall convergence.

3.1. Modular Architecture for Context-Specific Policies

Team sports environments, such as basketball, can exhibit rapidly shifting objectives and interactive dynamics among multiple agents, making a single monolithic policy insufficient for handling all scenarios effectively. To address this problem, we can divide the environment into discrete situations or modules. Each module can be governed by a specialized sub-policy trained independently under a shared overall framework. At each time step, the situation determinator can check predefined conditions in the environment and switch to the appropriate contextual module.

Two critical principles must be rigorously maintained when adopting a modular RL approach. Let S represent the entire state space and let each module i be responsible for a subset $s_i\bigcap s_j$. First, each state within the environment must activate a module at any given time step. Formally, for all $i\ne j$,

$$s_i\bigcap s_j=\varnothing ,$$

(12)

ensuring that no state belongs to multiple modules simultaneously. If multiple modules are triggered concurrently, conflicting policies may arise, causing ambiguity in action selection and destabilizing the learning process. Therefore, the criteria for defining the operational scope of each module should be formulated as mutually exclusive and deterministically identifiable.

Second, every possible state encountered during gameplay must correspond to at least one module. This implies that

$${\bigcup }_{i=1}^k{s}_i=S$$

(13)

thereby guaranteeing complete coverage of the state space environment. Failure to ensure comprehensive state coverage can prevent the agent from selecting a valid action, resulting in undefined or unintended behavior. The verification of these constraints can be achieved through deterministic rule-based conditions in the basketball environment. For example, the module assignment relies on clear game state indicators—that is, the ball possession status, game phase transitions, and contextual triggers. For disjoint mapping (Equation (12)), we ensured mutual exclusivity through hierarchical decision rules where ball possession took precedence, followed by defensive assignment, then loose-ball detection. For complete coverage (Equation (13)), we implemented comprehensive state space coverage verification during development, testing all possible game scenarios to ensure every state mapped to exactly one module.

Similarly to other ball games, in basketball the ball serves as the central element that determines the flow and dynamics of gameplay. Figure 3 illustrates how the ball transitions among three overarching situations—that is, offense, defense, and loose balls. Although the diagram does not depict the full complexity of a multi-agent Markov decision process (MDP), it serves as a simplified blueprint for designing a complete MDP. This high-level structure serves as a conceptual diagram that succinctly illustrates the transitions in basketball games and can be used as a reference point for future environmental design or algorithm implementation.

Specific definitions of the states, transition probabilities, and action spaces can be further refined or expanded to fit actual training scenarios. Under this modular design, each situation corresponds directly to a specialized subtask, allowing one to assign one-to-one mapping between these context-dedicated MAPPO sub-policies. Each context maintains its own independent rollout buffer that operates in isolation from other modules. This separation ensures that the training samples are logged and updated strictly within the relevant module, preventing cross-contamination between different game situations. During training, the samples from each buffer are independently sampled and used exclusively for updating the corresponding module’s policy and value networks. This architectural choice improves the sample efficiency by ensuring that each specialized policy learns only from contextually relevant experiences, while promoting the development of highly specialized policies tailored to specific game scenarios.

The sample efficiency was measured through comparative performance analysis in our ablation study, where we evaluated the score differentials achieved by different model configurations against the same baseline opponents. Higher sample efficiency was demonstrated when a model achieved better performance (higher scores) with equivalent training data, as shown in the ablation results where our complete KEMF appreciably outperformed the partial implementations and baseline methods.

A pivotal consideration is that the rapidly alternating contexts that characterize team-sport environments can render recurrent architectures suboptimal for our modular framework. Although recurrent neural networks (RNNs) can leverage sequential information to track temporal dependencies, abrupt shifts between the offense, defense, and loose-ball contexts can effectively break the continuity of this sequence of data. When a context switch occurs, the hidden state generated in the preceding module becomes irrelevant to the subsequent module, rendering it impossible to coherently resume any prior sequence once the original context becomes active. For example, consider the sequence offense → defense → offense. The hidden state (or sequence information) built up during an offense is no longer semantically meaningful when transitioning to defense. If the environment later reverts to offense, previously accumulated sequence data cannot be reintroduced in a principled manner. Such discontinuities imply that the temporal relationships that RNNs rely on can provide only limited benefit, as they are constantly being reset.

To circumvent this limitation, we treated the game (at the macro level) as a multiple context-specific task and assigned a dedicated, feed-forward MAPPO policy to each context (offense, defense, and loose-ball). Because feed-forward networks do not rely on persistent hidden memories, each module could operate independently, remain robust to frequent context switches, and store data in an isolated rollout buffer, thereby improving the sample efficiency and fostering highly specialized policies (see Figure 4).

3.2. Knowledge-Embedded Observation Layers

In this study, drawing on prominent spatial tactic analysis techniques and player performance prediction models that are widely used in team sports, we proposed a four-layer structure for systematically processing raw game data. First, at the 0th stage (raw level), we collect fundamental information such as the basic coordinates directly extracted from the game engine, ball possession status, and match scores. This serves as the starting point for applying commonly used spatial data processing methods in sports analytics and provides essential data for understanding the relative positions of players, the ball, and the rim [22]. In the 1st stage, we derive the spatial relationship indicators (e.g., distance, angle, and relative positions) from the 0th-stage data. For example, in basketball tactics analysis, the Euclidean distance and relative angle between the ball holder and the rim (or defender) can be used as key predictors of the “shot attempt likelihood” and “pass route effectiveness” [23].

Subsequently, in the 2nd stage, we leverage the spatial indicators computed in the 1st stage to generate single-context tactical or strategic metrics, such as the shooting success rate and level of defensive accuracy. The shooting success rate is typically estimated via statistical or regression methods that integrate factors, including the distance to the rim and defender positioning, whereas defensive accuracy is quantified by reflecting “how closely a player is positioned both near the opponent they must guard and, simultaneously (multiplicatively), between the opponent and rim [24]”. Specifically, the shooting success rate incorporates: (1) the Euclidean distance to the rim, (2) angular deviation from the optimal shooting angle relative to the rim, (3) degree to which defenders are positioned along the direct line between the shooter and rim, and (4) proximity distance to the closest defending player. The defensive accuracy metric similarly integrates: (1) the distance to the assigned target player, (2) positioning relative to the line connecting the target player and the rim, and (3) angular coverage that maximizes the interference with potential shots. As improving the accuracy of these metrics can require additional statistical modeling or regression analysis, we collected sufficient log data by repeatedly running custom measurement scenarios in a training simulator and then used these data to numerically approximate the metrics through correlation analysis and regression techniques.

Finally, in the 3rd stage, we recombine the 2nd-stage metrics to produce multi-context indicators, such as “the player on the current team with the highest shooting success rate” or “the player closest to the ball in a loose-ball situation.” This is intended to capture the complexity of cooperative play and the rapid situational changes observed in real basketball, thus enabling the modeling of team-level decision making (e.g., pass, ball tracking) at a higher level.

By applying specialized feature engineering at each stage, the agent can more effectively learn complex tactical situations that may be overlooked when only the raw coordinates or state information are provided (see Figure 5).

Additionally, when coupled with the modular structure described in Section 3.1, this method enables the agent to receive the observational data filtered for the relevant situation (offense, defense, or loose-ball), thus reducing both the exploration cost and difficulty of parameter updates during training (

Table 1. Knowledge-embedded observation-data information.

Situation	Knowledge-Embedded Observation Data
Offense	Start flag
	Ball possession state
	Distance, angle to the closest opponent agent
	Distance, angle to the rim
	Action, state of the closest opponent agent
	Personal, team average shoot success rate
	Distance, angle to the closest team agent
	Highest, lowest shoot success rate agent in team
Defense	Start flag
	Ball possession state of the mark target
	Distance, angle to the mark target
	Action, state of the mark target
	Shoot success rate of the mark target
	Personal, team average defensive accuracy
	Steal action available
	Block action available
Loose-ball	Start flag
	Distance to the ball
	Angle to the ball
	Ball state
	Ball’s height
	Closest agent to the ball in team
	Rebound action available

). Unlike a traditional setting whereby the agent “learns everything from scratch,” this method strategically incorporates the accumulated basketball expertise, statistical models, and in-house analytical results to facilitate faster and more stable learning. Indeed, multiple studies on sports AI have reported enhanced learning efficiency by integrating existing match analysis methods and statistical models [38,39,42]. Accordingly, this study actively adopted a knowledge-based observational data architecture to effectively address frequent situational changes and complex team play in basketball.

Feature Estimation for Knowledge-Embedded Observation

The knowledge-based observation layer employs several estimators to extract meaningful features from the raw game states, following established sports analytics methodologies [22,23]. The shooting success estimator uses a logistic regression model trained on historical match data, taking as input the shooter’s position, angle to rim, defender positions, and distance factors—metrics commonly used in basketball shot predictions [24]. The defensive pressure estimator employs weighted distance calculations to quantify the spatial control around the ball carrier, similar to defensive skill characterization methods in professional basketball analysis [24].

These estimator models were pre-trained on a separate dataset collected from 10,000 simulated matches and remain fixed during policy training to ensure stability, following the practice of separating feature extraction from policy learning [56,57]. To mitigate overfitting to simulator-specific statistics, we applied domain randomization by varying the physical parameters during data collection. Additionally, all estimator outputs were normalized to the range [0, 1] using min-max scaling based on the empirically observed bounds from the training dataset, consistent with standard feature normalization practices in deep RL [56].

3.3. Dynamic Dense Reward Design

RL aims to maximize the cumulative reward through interactions with the environment, a widely recognized definition in this field. From a more technical standpoint, the agent learns a function approximating the reward r resulting from the transition from state s to s’ via action a. If the reward is overly sparse or exhibits weak correlations with these state transitions, the agent may struggle to converge to an optimal policy. Conversely, stronger alignment between state transitions and the reward structure enables more efficient learning with fewer samples [26]. Building on these insights, we designed a dynamic dense reward structure by actively incorporating the ‘situation-specific contextual observations’ introduced in Section 3.2.

In contrast to conventional methods that rely on sparse or static dense rewards, our method provides fine-grained action-level feedback. Crucially, the domain knowledge-based metrics (e.g., the shooting success rates, defensive accuracy, inter-player distances, and angles) described in Section 3.2 serve as key references for distributing rewards more precisely. In most game environments, movement constitutes a considerable portion of the agent actions. Hence, the ability to move to an appropriate location in a rapidly changing context is a prerequisite for correct decision making and action execution. To address this problem, our reward mechanism included (1) movement rewards for each situation, (2) action-level rewards, and (3) the final overall reward (

Table 2. Dynamic dense reward information.

Situation	Action	Condition	Reward
Offense	Movement	Only for movement actions	(Previous team average shoot success rate–Current team average shoot success rate)/Max step
	Pass	Only when possessing the ball and lowest shoot success rate	Average team shoot success rate × Scale factor
	Shoot	Only when possessing the ball and highest shoot success rate	Shoot success rate × Scale factor
Defense	Movement	Only for movement actions	(Previous defensive accuracy rate–Current defensive accuracy rate)/Max step
	Face Up	Only when distance to mark target is lower than defense effective range	Defensive accuracy rate × Scale factor
	Steal	Only when distance to mark target is lower than defense effective range	Defensive accuracy rate × Scale factor
	Block	Only when distance to mark target is lower than defense effective range	Defensive accuracy rate × Scale factor
Loose-ball	Movement	Only for movement actions and when closest to the ball on team	(Previous distance to the ball–Current distance to the ball)/Max step
		Only for movement actions and when not closest to the ball on team
	Rebound	Only when ball height is higher than minimum height to rebound	Distance to the ball × Scale factor

).

For example, successfully performing defensive actions such as stealing, blocking, or face-ups requires the agent not only to mark the target player from close range but also to position itself between the player and rim.

Here, the “defensive accuracy” metric (Section 3.2) quantifies how effectively the defender covers both the offensive player and the rim, thus informing the design of the movement rewards. Subsequently, the dynamic rewards are assigned based on how closely the agent’s attempted defensive actions align with the “ideal response actions” identified through repeated simulations (e.g., the timing, distance, and angular proximity). For example, the closer the agent’s action is to the statistically optimal pattern—calculated from metrics such as the shooting success rates or distances and angles between players—the higher the reward. Combining position-based rewards with action-level incentives encourages the agent to learn more sophisticated defensive policies.

However, segmenting the rewards in this manner and using a variety of contextual metrics, as discussed in Section 3.2, requires balancing the relative size of each reward element, potentially compromising the learning stability. In this study, we attempted to mitigate this limitation by adopting a modular environment, which allowed learning to operate independently on a module-by-module basis. This approach makes it easier to elaborate on the contextual design of rewards and perform the requisite individual tuning.

Nonetheless, to achieve higher learning stability, we compensated for reward bias by applying a scaling factor to the action-level reward formula. The scale variable measured the proportion of situations where each behavior could be attempted during each neural network update cycle. Accordingly, we applied min-max normalization to calculate a value that was inversely proportional to this proportion (where this proportion denotes the fraction of time steps in the current training batch during which the corresponding action was executed).

Consequently, we designed the reward scale to be automatically larger for behaviors that were relatively infrequently encountered and, therefore, had less learning experience, effectively balancing rewards during the module-by-module learning phase and ensuring the consistency and stability of the overall policy. Through extensive sensitivity analysis, we tested various scaling methods—that is, uniform scaling (1.0 for all actions), low inverse frequency scaling (0.001–1.0 range), medium scaling (0.01–2.0 range), and high scaling (0.1–10.0 range) methods.

The results demonstrated that uniform scaling led to approximately 13.1% slower convergence owing to the inadequate learning of rare actions such as blocking and stealing. Medium inverse frequency scaling factors achieved optimal performance with 15–20% faster convergence compared to that of uniform weighting, whereas high scaling factors caused training instability. The optimal scaling range was empirically determined to be 0.001–2.0 for most action types, with rare defensive actions requiring higher multipliers. The specific scaling factors used remain proprietary owing to commercial considerations.

4. Experimental Results

4.1. Experimental Environment

In this study, we employed the “Freestyle” simulator—an online basketball game developed and published by JoyCity Co., Ltd.—as the training and testing environment for validating our proposed method (Figure 6). Unlike standard basketball, “Freestyle” features a three-on-three multiplayer setting with a single rim on a half-court, without enforcing rules such as fouls or double dribbling.

A notable characteristic of this game is that the action outcomes are governed by probabilistic events. For example, even if a shot is taken from the same position and angle, it may succeed or fail with a certain probability, and defensive maneuvers such as stealing or blocking are likewise subject to probabilistic elements. This uncertainty partially reflects unpredictability, providing players with an engaging challenge to adapt to unforeseen situations.

However, from an RL perspective, such probabilistic outcomes further exacerbate the reward sparsity associated with success or failure. In scenarios where the reward is directly tied to the action success, the same action may yield different rewards solely based on chance, making it difficult for the agent to pinpoint which specific action actually produces the reward. Consequently, even when the agent selects the correct state-action combination, there is always a possibility that it will not receive any positive feedback, thereby complicating the exploration process and increasing the overall training time.

Moreover, if actions that fail consistently yield zero (or very small) rewards, the agent rarely experiences meaningful feedback, leading to substantially reduced sample efficiency. In environments in which success or failure constitutes the primary source of rewards, frequent random failures can intensify the reward sparsity problem, ultimately decreasing the signal-to-noise ratio. In Freestyle, the agent’s behaviors are drawn from the same list of actions, as shown in

Table 3. Available actions per situation.

Situation	Actions
Common	Movement (8 directions), Stop
Offense	Shoot, Pass, Breakthrough
Defense	Steal, Block, Face Up
Loose-ball	Rebound, Diving Catch

. Behavior changes depend on the situation, and there are no restrictions on the number of instances or stamina.

4.2. Setting

In this study, we employed a self-play training paradigm for our multi-agent RL setup rather than relying on a conventional finite-state machine (FSM) approach. Self-play is well-known for fostering robust policy learning in multi-agent environments, as each agent continually interacts with both the environment and its adversarial or cooperative counterparts that simultaneously undergo learning [60]. By iteratively competing against and adapting to increasingly capable opponents, agents can discover sophisticated strategies and naturally adjust to evolving gameplay scenarios.

By contrast, FSM-based methods typically rely on carefully handcrafted state definitions and transitions that encode domain-specific knowledge. Although FSMs can be straightforwardly implemented for small or highly structured tasks, they often result in inflexible behaviors because each possible situation must be enumerated and codified in advance. Consequently, an FSM design may struggle to manage unforeseen game states or complex dynamic scenarios. Moreover, adding new features or modifying an existing environment frequently necessitates extensive revisions to the FSM logic, thus complicating long-term maintenance.

By adopting self-play, agents can autonomously explore the state space and learn from repeated trial-and-error interactions—both with the environment and with each other [61]. This process can endow agents with greater adaptability to stochastic outcomes (e.g., probabilistic shot successes or defensive maneuvers) and emergent strategies. Therefore, self-play naturally increases the diversity of experiences encountered during training, thereby leading to more generalized policies that can consistently perform well against a wide range of opponent skill levels.

However, a practical caveat is that when multiple models learn simultaneously, the environment can become non-stationary, which can hinder stable convergence. A common solution is to periodically alternate training between models, ensuring that only one model is updated at any given time. In our setup, we implemented a structured alternating schedule where one model trained for 100 episodes while the other remained fixed, after which we switched roles. This 100-episode cycle was empirically determined through preliminary experiments that balanced the learning stability with computational efficiency. The alternating schedule prevents volatile learning dynamics and helps maintain a more stable training reference, ultimately resulting in more robust policy evolution.

Here, 100 episodes refers to this periodic alternation cycle rather than the total training duration, which extends over multiple such cycles until convergence is achieved. Consequently, our learning framework provides a stronger foundation for scalable, maintainable, and robust agent behavior in the Freestyle simulator, ultimately facilitating more realistic and strategically rich gameplay.

Algorithm 1. Pseudocode for the knowledge-embedded modular MAPPO training procedure. The algorithm implements alternating self-play training between two teams (lines 4–8), where one team’s parameters are updated while the other remains frozen for 100 episodes before switching. For each timestep, the situation determinator selects the appropriate module (Offense/Defense/Loose-ball) based on the game state (line 15), and actions are sampled from the corresponding module policy (line 18). Transitions are stored in team-specific buffers (line 19) to prevent cross-contamination. The PPO update (lines 39–49) uses generalized advantage estimation (GAE) with parameters γ and λ for computing advantages (lines 27–34), followed by clipped surrogate objective optimization with entropy regularization.

Algorithm 1. Knowledge-Embedded Modular MAPPO Training

Require: (1) Env. interface $ℰ$ (states, actions, rewards);     (2) MAPPO models for each module (Offense, Defense, Loose-ball);     (3) γ, λ, ε;     (4) Teams: {0, 1} each with N agents.

1: Initialize: Set seeds; Create buffers $𝒟$0, $𝒟$1    Load Knowledge-embedded modular MAPPO params;    if available; Reset trackers;  2: for episode e = 1, … do  3:  Reset env; Clear $𝒟$0, $𝒟$1  4:  if (e mod 200) < 100 then  5:   Update team 0, freeze team 1  6:  else  7:   Update team 1, freeze team 0  8:  end if  9:  while not done do 10:   Get state s from $ℰ$ 11:   for team ∈ {0, 1} do 12:      if team is frozen then 13:       // Only forward pass, no param update 14:      end if 15:      Determine module M_team (Offense/Defense/Loose-ball) 16:      Build local obs oi for each agent; masks mi 17:      Global state g_team 18:      (ai, log π(ai|oi)) ← MAPPO_{M_team}(oi) 19:      Exec ai in env; Store transition (oi, ai, ri, …) in $𝒟$_team 20:   end for 21:   Env step; Compute rewards rt; done ← (time = 0) 22:  end while 23:  for team ∈ {0, 1} do 24:   if team not frozen then 25:      Group transitions in $𝒟$_team by trajectory 26:      for each traj. τ = {(st, ot, rt)}T−1_{t=0} do 27:       for t = T − 1 down to 0 do 28:         δt = rt + γV_φ(st+1) − V_φ(st) 29:         if t = T − 1 then 30:           At = δt 31:         else 32:           At = δt + γλ At+1 33:         end if 34:       end for 35:       for t = 0 to T − 1 do 36:         Rt = At + V_φ(st) 37:     end for 38:      end for 39:      PPO Update: 40:      for minibatch in $𝒟$_team do 41:       for sample t in minibatch do 42:         r(θ) ← π_θ(at|st) / π_{θ_old}(at|st) 43:         L^{CLIP}_t = min(r(θ) At, clip(r(θ), 1−ε, 1+ε) At) 44:         L^{VF}_t = (Rt − V_φ(st))2 45:       end for 46:       L^{CLIP} = mean(L^{CLIP}_t); L^{VF} = mean(L^{VF}_t) 47:       L(θ, φ) = −L^{CLIP} + c1 L^{VF} + c2 Entropy(π_θ) 48:       θ, φ ← θ, φ − η∇L(θ, φ) 49:      end for 50:   end if 51:  end for 52:  Save model if training 53: end for

To apply the proposed KEMF method, we adopted the training procedure and hyperparameter configurations specified in

Table 4. Hyperparameters for proposed method.

Hyperparameter	Value
Learning rate	0.00001
Gamma	0.99933
Mini batch size	64
GAE lambda	0.95
Clip ratio	0.2
Gradient clipping max norm	0.5
Hidden layer size	64
Layer	2 fully connected layers
Activation	Hyperbolic tangent

. These settings were determined through iterative experimentation and tuning, involving pilot runs and systematic parameter sweeps to assess the impact of various learning rates, batch sizes, discount factors, and other critical factors. Throughout this process, we aimed to balance the training stability with final performance. The neural networks employed standard feed-forward architectures with hyperbolic tangent activations. The actor networks had two fully connected 64-unit hidden layers, whereas the critics used similar architectures. The choice of relatively simple architectures was intentional to focus the computational resources on the modular framework and knowledge-embedded features rather than the network.

4.3. Comparison Experiment with Competing Methods

To rigorously evaluate the effectiveness of our proposed KEMF-based method, we conducted direct performance comparisons with two baseline methods—namely, (1) a prior version model [42] and (2) an FSM, which has been extensively deployed in the commercial Freestyle game environment. The FSM baseline represents a sophisticated rule-based probabilistic system that has been refined over years of commercial operation in the Freestyle environment. Owing to intellectual property constraints and its ongoing commercial deployment, we cannot provide detailed internal logic. However, we can characterize it as a comprehensive state machine that encodes expert basketball knowledge through carefully handcrafted rules and has demonstrated robust performance in live service environments over an extended period, making it a challenging benchmark.

Specifically, we organized multiple 3-on-3 matches under standardized conditions—such as identical team compositions, controlled initial ball possession, and randomized seeds—to ensure that each policy faced comparable in-game scenarios. This design allowed us to isolate and compare the core decision-making capabilities of the three approaches. Notably, our proposed method required only 28 d of training time, whereas the prior model required more than 180 d of training yet still exhibited lower performance, demonstrating the efficiency gains achieved through the KEMF.

Figure 7 presents the average scores on these tests. When pitted against the former model, the KEMF-based policy achieved an average of 18.8 points, surpassing the baseline’s 14.9. Against the FSM opponent, the old model trailed badly—10.1 points versus the FSM’s 19.9, a gap of 9.8 points. By contrast, the KEMF policy posted 12.5 points versus the FSM’s 15.8, trimming the deficit to 3.3 points. Although the KEMF has not yet overtaken the handcrafted FSM, this sizable narrowing of the gap—along with its clear win over the earlier learning-based model—highlights the proposed method’s improved competitiveness. Coupled with its markedly shorter training time, these results underscore the enhanced efficiency and robustness of the KEMF method in the stochastic, adversarial setting of Freestyle basketball.

4.4. Ablation Study

To isolate the contributions of each component in the KEMF, we conducted an ablation study using four distinct experimental settings. By selectively enabling or disabling specific elements of our proposed method—namely, knowledge-embedded observation, dense reward, and modular policy architecture—we aimed to understand the factors that most enhanced the learning performance. The four experimental variants labeled (a)–(d) were:

(a)

Baseline (No Proposed Methods): This setting mirrored the training regime of our previous model and did not incorporate any newly proposed techniques. The agent received raw game state observation data and relied solely on sparse rewards using a single-model approach.

(b)

Knowledge-Embedded Observation and Dynamic Dense Reward (No Modular): For this variant, we embedded the domain knowledge into the agent’s observation data, as described in Section 3.2, and applied our dense reward strategy. However, the modular policy component was omitted; therefore, the agent employed a single unified policy across all contexts.

(c)

Modular Policy with Raw Game State Observation and Sparse Reward: This setting adopted a modular RL design while retaining the original raw game state observation and sparse reward used in the baseline approach. This allowed us to isolate the impact of modular decomposition on the learning performance without the additional influence of knowledge-embedded features or denser reward signals.

(d)

Full KEMF (All Proposed Methods): Finally, we evaluated the complete KEMF where the knowledge-embedded observation, dynamic dense reward, and modular architectures were employed. This served as a reference point for comparison with the other three variations.

Although the empirical results are presented in a subsequent section, this ablation study setup offered a systematic approach for identifying the relative importance and synergistic effects of each proposed component. By examining the effects of each incremental modification on policy learning, we could gain deeper insights into the mechanisms underlying the performance gains afforded by the KEMF.

Figure 8 presents a bar chart visualizing the outcomes of direct matches between the FSM and models trained according to the ablation procedure described above. On the x-axis, from left to right, are the game’s built-in FSM-based AI and four ablated versions of our model: (1) a baseline without the proposed knowledge-embedded or modular strategies, (2) a modular-only variant (K(x),M(o)), (3) a knowledge-embedded-only variant (K(o),M(x)), and finally (4) a fully integrated model (K(o),M(o)) that employs both proposed techniques. The y-axis indicates the average score for each model obtained in head-to-head competitions.

Statistical analysis of the score differentials revealed highly significant differences between configurations. One-way ANOVA on the score gaps revealed F(3,36) = 69.63, p < 0.001, with a large effect size (η² = 0.853). The score gaps decreased systematically from 30.9 ± 5.57 points [K(x),M(x)] to 25.2 ± 4.58 points [K(x),M(o)] to 11.4 ± 4.62 points [K(o),M(x)] to 3.3 ± 4.32 points [K(o),M(o)]. Moreover, post hoc Tukey Honestly Significant Difference tests confirmed significant differences between all pairwise comparisons (all p < 0.01). The complete KEMF demonstrated a large effect size compared to the baseline (Cohen’s d = 5.54), indicating considerable practical significance. The results demonstrated that models adopting the proposed strategies progressively reduced the performance gap with the FSM. This pattern suggests that models incorporating the proposed strategies could defend against the FSM’s scoring opportunities more effectively while simultaneously enhancing their own offensive success. Notably, the performance improved significantly when both knowledge-embedded and modular components were applied in tandem, underscoring the efficacy of combining these two core elements of the proposed approach.

To investigate the impact of the reward scaling magnitude on model performance, we conducted a sensitivity analysis by varying the scaling factor across a wide range (0.001–10.0). As shown in Figure 9, the performance exhibited a clear U-shaped pattern with respect to the scaling magnitude. Scaling factors in the range of 0.5 –2.0 achieved optimal performance with a score gap of approximately 3.3 points against the FSM baseline. Below this range (<0.1), insufficient reward differentiation led to slower convergence and suboptimal policies. Conversely, scaling factors above 5.0 caused training instability owing to the exaggerated reward signals, resulting in erratic policy updates. These findings support our choice of medium-range scaling factors (0.5–2.0) for the final KEMF implementation and demonstrate the importance of careful reward scaling in dynamic dense reward designs.

4.5. Experiment on Real-World Live Services

To further validate the proposed KEMF-based method under practical real-world conditions, we deployed trained agents on Freestyle’s live servers. The agents played for approximately 5900 h a week and participated in 1457 matches, ultimately achieving a win rate of approximately 52.43%. This outcome was particularly noteworthy given that Freestyle—a team-based basketball game in continuous operation for over 20 years—features deeply refined tactics and highly strategic gameplay, honed by a large community of skilled human players. It is important to acknowledge that the proposed method had access to raw game data (such as precise player coordinates and the ball state), which provided certain advantages over human players who had to rely on visual information processing. This information asymmetry introduced several considerations for interpreting the reported 52.43% win rate against human teams, similar to considerations raised in human-AI game competitions [10,11].

First, the AI’s access to complete state information provides inherent advantages in spatial awareness and anticipation that human players cannot replicate through visual processing alone [22]. Second, human players face cognitive load limitations when processing multiple simultaneous events, whereas the AI system processes all information instantaneously. Third, human reaction times (typically 200–300 ms) impose fundamental constraints that do not affect the AI system.

To contextualize these results, we note that the relatively modest win rate margin (52.43% vs. expected ~50% for balanced competition) suggests that despite data access advantages, the strategic complexity and nuanced decision making required in basketball present major challenges even when provided with complete state information [39,40]. The human players in our evaluation represented experienced competitors with years of gameplay expertise, making this a rigorous benchmark.

The achieved win rate of 52.43% indicates that although the AI method performed slightly better than the human players on average, the margin was relatively modest considering its data access advantage. We acknowledge that achieving true fairness in human-AI competition remains an open challenge [10], and that reported win rates should be interpreted as demonstrations of tactical competence rather than claims of superiority under equivalent perceptual conditions.

This suggests that the strategic complexity and nuanced decision making required in basketball present major challenges even when provided with complete state information. Nonetheless, the ability to maintain competitive in such a mature and strategically rich environment demonstrates that the proposed KEMF-based method can effectively navigate real-time user decisions and adapt to the dynamic complexities inherent in team-based gaming scenarios. This real-world deployment serves as substantial validation that goes beyond controlled laboratory experiments, providing evidence of the method’s practical effectiveness in commercial gaming environments in which performance and player-compatible metrics are critical success metrics.

5. Conclusions

This study proposed a KEMF to address the elevated sample complexity associated with MARL in a commercial online basketball environment. By partitioning the game into context-specific modules—that is, the offense, defense, and loose ball modules—we effectively reduced the dimensionality of the state-action space and mitigated the adverse impacts of frequent and rapid situation switches. Moreover, the incorporation of domain-specific knowledge, such as estimated shooting success rates and defensive accuracy, substantially enhanced the observation space, thereby accelerating the learning process and improving policy stability. To counter the well-known problems of sparse reward signals, we introduced a dynamic dense reward mechanism that correlated intermediate decisions, including spatial movement and action selection, with semantically enriched feedback. Empirical evaluations in the Freestyle simulator indicated that our framework outperformed both previously developed baseline models, achieving higher average scores in competitive match settings while requiring considerably less training time (180 d to 28 d). Although the proposed method has not yet surpassed the FSM baseline, the 66.3% reduction in performance gap (from 9.8 to 3.3 points) demonstrates considerable progress. Moreover, the KEMF-based method was deployed on live servers for approximately 5900 h, engaging in 1457 matches and attaining a win rate of approximately 52.43% against skilled human opponents. These real-world results underscore the practical viability of the proposed method in highly dynamic and stochastic team sports. Moreover, using specific optimizations—including refined reward structures, fine-grained situational decomposition, and module-specific hyperparameter tuning—could result in further performance improvements.

Although the proposed method exhibited promise in the basketball gaming domain, we acknowledge certain limitations. The method incurs engineering overheads for knowledge inference and contextual demarcation, and broader validation across different sporting environments could strengthen its generalizability claims. Additionally, statistical limitations owing to computational constraints during development could be addressed in future work through more extensive multi-seed experimentation. Nonetheless, this research demonstrated that a judicious blend of modularization, knowledge-based feature engineering, and refined reward structuring could substantially improve performance in complex multi-agent sporting simulations.

We also acknowledge that recent advances in MARL have introduced novel methods for multi-agent cooperation. For example, Tang et al. [62] proposed an intention-strategy optimization model (ISO-MAGCG) that integrated intention recognition with strategy optimization, demonstrating considerable improvements in cooperative scenarios. Future extensions could include direct experimental comparisons with recent MARL algorithms, automated detection of multi-context scenarios, advanced probabilistic modeling for enhanced robustness, and exploration of the framework’s applicability to other team sport domains. Future extensions could also include the automated detection of multi-context scenarios, advanced probabilistic modeling for enhanced robustness, and exploration of the framework’s applicability to other team sport domains.