Algorithms for Game AI

Games have long been benchmarks for AI algorithms and, with the boost of computational power and the application of new algorithms, AI systems have achieved superhuman performance in games for which it was once thought that they could only be mastered by humans due to their high complexity. Since Deep Blue beat professional human players in chess [1], many milestones have been reported in various games, from board games like Go [2,3,4] and card games like Texas Hold’em [5,6,7] to video games like StarCraft [8], Dota 2 [9], and HoK [10].

There are two reasons why games are excellent benchmarks for AI algorithms. First, games are born to test and challenge human intelligence—just as children learn about the world by playing with games and toys during their first years of life, the diversity of games can provide a rich context to test different cognitive and decision-making capabilities. For example, board games, with their formalized state representation and perfect information, require searching and planning from the current game state. Card games, with their non-deterministic transition and imperfect information, reveal more sophisticated strategies, such as bluffing and deception, skills that are normally reserved for humans. Video games, with their high-dimensional states and long episodes, require feature extraction, memorization, long-term planning, and multi-agent cooperation and competition. These characteristics make games strong testbeds for the gradual skill progression of AI.

Second, solving complex real-world problems usually requires repeated trial and error, which can be very costly. By simulating or emulating real-world scenarios, games provide a low-cost or even zero-cost platform to validate various algorithms and solutions. In a broad sense, games are not limited to those that people play for entertainment, but any decision-making problems can be modeled and simulated as games, from simple robot control to complex scenarios of resource allocation, scheduling, and routing across various industrial fields. Algorithms proposed and developed to play games are eventually applied to various real-world problems for social benefit in all aspects of life.

This Special Issue entitled “Algorithms for Game AI” covers new and innovative approaches for solving game AI problems. These approaches range from traditional algorithms like planning and searching, to modern algorithms such as deep reinforcement learning. The papers in this Special Issue address both the theoretical and practical challenges of the application of these algorithms.

AI techniques have been implemented in gaming industry with a wide range of applications [11], such as procedure content generation (PCG), where algorithms autonomously create game worlds, levels, and assets to enhance replayability, and AI-driven non-player character (NPC) behavior, simulating human-like decision-making and emotional responses for better engagement. Nevertheless, we mainly focused on the application of building game-playing agents, because of the inherent challenges and the potential for solving real-world decision-making problems. In this field, game AI algorithms can be divided into several categories, such as the following:

• Search algorithms are used for real-time planning. These algorithms are designed to expand a search tree from the game state currently encountered to evaluate future states under different sequences of actions. Some algorithms like A* [12] leverage heuristic functions to guide the selection of unexplored nodes, which evaluate game states based on the domain knowledge of human players. In general, searching more deeply usually yields better policies because it eliminates some errors of value estimation by looking more steps ahead. However, searching more deeply costs more computation, and the search depth is usually, in practice, fixed or decided by iterative deepening to limit the time cost. In multi-agent competitive games, adversarial search algorithms such as Minimax [13] are often used, where the agents must act against opponents and maximize their own benefits. Monte Carlo Tree Search (MCTS) [14] is one of the most popular search algorithms due to its efficiency and robustness with no domain knowledge needed. Many variants of MCTS have been proposed to further improve its efficiency by incorporating upper confidence bounds [15], domain knowledge [16], and parallelization [17].
• Optimization algorithms refine solutions to achieve specific goals, which are used to train agents with parameterized models. Among these, linear programming algorithms optimize objective functions under a set of constraints; evolutionary methods, such as Genetic Algorithms (GA) and Evolutionary Strategies (ES) [18], create a population of individuals where the fitter ones have a higher probability of reproducing and inheriting part of their structures, inspired by the process of natural selection; gradient descent is the foundation of deep learning, which achieves the efficient parameter tuning of modern artificial neural networks. These algorithms train policy models or value models containing prior knowledge before actual gameplay, which can be combined with real-time planning algorithms during the inference phase.
• Supervised learning is a data-driven method to learn patterns and relationships from labeled data. In the context of game AI, the data usually refer to the game states or observations, and the task is to learn a policy model or value model that predicts the action or the estimated value under the current state. These algorithms require a great deal of labeled data in the form of state–action or state–value pairs, usually collected from data on human gameplay or data generated by other game-playing algorithms. In general, there are two types of applications based on the data source. Applying supervised learning on human data can learn implicit human knowledge and store it in policy models or value models. Another type of application, known as policy distillation [19], relies on data generated by other models, which is used to distill a lightweight model from a larger one to improve computational efficiency or to consolidate multiple task-specific policies into a single unified policy.
• Reinforcement learning (RL) studies how agents should take actions in an environment to maximize cumulative rewards over time. Typically, RL models the environment as a Markov Decision Process (MDP), where the transition probability and rewards satisfy the Markov property that they are only related to the current state and action. When the transition model is known, generalized policy iteration, such as value iteration, uses dynamic programming to solve the optimal policy and its value function based on Bellman Equation. However, in most cases, the environment model is unknown, and model-free algorithms are preferable as they learn from experiences by interacting with the environment. There are two kinds of model-free algorithms:

  ○

  Value-based algorithms optimize the policy by approximating the values of states or state–action pairs and selecting better actions based on these values. There are different ways to update the value function [20]. A Monte Carlo (MC) algorithm updates the value function based on the cumulative rewards towards the end of the episode, while a temporal difference (TD) algorithm updates the value function based on the current reward and the value of the next state in a bootstrapping manner. Algorithms such as DQN [21] use deep neural networks to approximate the state–action value function, which can be applied to games with large state space.

  ○

  o Policy-based algorithms directly learn parameterized policies based on gradients of some performance measure using the gradient descent method. For example, REINFORCE [22] samples full episode trajectories with Monte Carlo methods to estimate returns as the loss function. However, such pure policy-based algorithms suffer from high variance, and actor–critic algorithms [23] have been proposed, which use actors to learn parameterized policies and critics to learn value functions, allowing the policy updates to consider the value estimates to reduce the variance. Actor–critic algorithms include DDPG [24], A3C [25], IMPALA [26], TRPO [27], and PPO [28], to name but a few.

• Regret-based algorithms seek to find the Nash equilibrium in games with imperfect information. The basic idea is to choose actions to minimize the regret of not having chosen other actions in previous games. Counterfactual regret minimization (CFR) [29] traverses the game tree for multiple iterations to calculate the cumulative regrets of each state–action pair. Many variants of CFR, such as Discounted CFR [30] and MCCFR [31], have been proposed to improve efficiency by incorporating sampling, discounting, and reweighting techniques. For games with large state space, algorithms such as Deep CFR [32] and DREAM [33] adopt neural networks to approximate the regrets and policies.
• Self-play algorithms are often used in competitive multi-agent environments. In fictitious play (FP) [34], each agent calculates its best response to the opponent’s average policies, which has been proven to converge to the Nash equilibrium in theory. When applied to games with large state space, neural fictitious self-play (NFSP) [35] adopts neural networks as the policy model, using reinforcement learning to calculate best responses and supervised learning to learn average policies. Double oracle (DO) [36] starts from a small policy subset, where each agent iteratively calculates the Nash equilibrium under the current strategy set and adds it to the set. Policy-space response oracles (PSRO) [37] provide a unified view of them using a policy pool to train new policies to be added, which has become the common practice of multi-agent RL training.
• Multi-agent RL (MARL) algorithms extend single-agent RL algorithms to multi-agent settings, usually following a training paradigm called centralized training decentralized execution (CTDE). CTDE jointly trains multiple agents in a centralized manner but keeps their independence in execution, which can provide a mechanism of communication to eliminate the problems of unstable dynamics in independent training. For example, value-based algorithms like Value Decomposition Network (VDN) [38] and QMIX [39] are variants of DQN in cooperative multi-agent settings that adopt centralized state-action value functions, using summation and mixing networks to combine individual Q-networks. Multi-agent DDPG (MADDPG) [40] is a policy-based algorithm that generalizes DDPG to multi-agent settings, and many variants of it have been proposed to improve its performance and efficiency.

This Special Issue presents ten papers covering a wide range of game AI topics, including the quantification of non-transitivity in chess, the expressiveness of level generators in Super Mario Bros, Mahjong as a new game AI benchmark, new MARL algorithms to reduce Q-value bias, surveys of various AI algorithms in cyber defense, energy areas and games, the application of MCTS in Amazons, the application of deep reinforcement learning in autonomous vehicle driving, and the application of transformers in both offline RL and imitation learning. The list of contributions to this Special Issue is as follows:

Contribution 1 [41] examines the issue of cyber defense from the perspective of decision-making and proposes a framework to study the problem characteristics of different cyber defense contexts. The framework identifies four stages in the life cycle of threats—pentest, design, response, and recovery—integrating both the attacker’s and defender’s perspectives. By analyzing the decision-making scenarios in each of these four stages, this paper presents a comprehensive survey of existing research on the application of reinforcement learning in cyber defense, providing a review of both the problem boundary and current methods.

Contribution 2 [42] measures the degree of non-transitivity in chess and investigates the implications of non-transitivity for population-based self-play algorithms. By measuring both the Nash clusters and the length of the longest transition cycles in the policies from billions of human games, this paper reveals that the strategy space of chess indicates a spinning top geometry where middle-level strategies have longer non-transitive cycles than top-level or worst-performing ones. Further experiments with a fixed-memory fictitious play algorithm, which simulates the behavior of common population-based self-play methods, indicate not only that larger populations are required for training to converge, but that the minimum size of the population is also related to the degree of non-transitivity of the strategy space.

Contribution 3 [43] reviews machine learning algorithms to solve combinatorial optimization problems (COPs) in energy areas. COPs are a class of NP-hard problems used to optimize an objective function within a discrete domain, which have been the focus of traditional game AI research and can be seen across various industrial fields, from resource allocation and scheduling to routing scenarios. This paper focuses on the context of energy areas, including petroleum supply chains, steel making, electric power systems, and wind power. It presents a systematic review of ML algorithms used to solve COPs in these scenarios, including supervised learning (SL), deep learning (DL), reinforcement learning (RL), and recently proposed game theory-based methods. Application of these algorithms are discussed in detail, including recent advances and the challenges and future directions of this field.

Contribution 4 [44] reviews algorithms and paradigms to build high-performance game-playing agents. Many game AI algorithms, from searching and planning to SL, RL, and game theory-based methods are presented as basic components of game AI systems. By summarizing the implementation of recent state-of-the-art game AI systems in various games, which have been achieving superhuman performance, this paper provides a comprehensive comparison of these milestones via decomposing each into its components and relating them to the characteristics of the games to which they are applied. Three types of paradigms are concluded from these milestones, with their scope and limitations discussed in detail. Stemming from these comparisons and analyses, the paper claims that deep reinforcement learning is likely to become a general methodology for game AI as future trends unfold.

Contribution 5 [45] introduces Official International Mahjong as a new benchmark for game AI research. Mahjong presents challenges for AI research due to its multi-agent nature, rich hidden information, and complex scoring rules. This paper presents in detail Mahjong Competition Rules (MCR), a Mahjong variant widely used in competitions due to its complexity, as well as a series of Mahjong AI competitions held in IJCAI, which adopts the duplicate format to reduce variance. By comparing the algorithms and the performance of AI agents in these competitions, the paper shows that supervised learning and reinforcement learning are currently the state-of-the-art methods in this game and perform much better than heuristic methods. While the top AI agents still cannot beat professional human players, this paper claims that this game can be a new benchmark for AI research due to its complexity and popularity.

Contribution 6 [46] explores the application of language models in offline RL, and proposes Action-Translator Transformer (ATT) as the policy model. ATT is built upon the Sequence-to-Sequence (Seq2Seq) model structure used in text translation tasks. Different from Decision Transformer (DT), which predicts next states as well as rewards and actions, ATT only predicts actions to maximize cumulative rewards, given partial trajectories containing states, actions, and rewards. Positional encoding typically used in NLP tasks is also devised to adapt to new input elements of RL. Experiments performed in several Mujoco environments show that ATT performs better than other offline RL algorithms such as DT.

Contribution 7 [47] discusses the issue of Q-value overestimation in MADDPG, and proposes two variants to reduce estimation bias, called Multi-Agent Mutual Twin Delayed Deep Deterministic Policy Gradient (M2ATD3) and Multi-Agent Softmax Twin Delayed Deep Deterministic Policy Gradient (MASTD3). Both methods introduce a second Q-network for each agent and incorporate either the minimum or maximum value of two Q-values to reduce bias in the opposite direction. MASTD3 further extends the Softmax Bellman operation to multi-agent settings by fixing others’ actions and only changing its own actions. Experiments on two multi-agent environments, Particles and Tank, show some performance improvement and Q-value bias reduction compared to baseline methods such as MATD3.

Contribution 8 [48] investigates the quality of PCG and specifically evaluates the expressive range of three level generators for Super Mario Bros (SMB) under different algorithms, including Genetic Algorithms (GA), Generative Adversarial Networks (GAN), and Markov Chains (MC). By defining nine metrics for each SMB level covering different types of characteristics, this paper visualizes the expressive ranges based on the metric distribution of generated levels and concludes that GA and MC have a much wider expressive range than GAN. The practice of expressive range analysis (ERA) presented in this paper can help game developers to recognize potential problems early in the PCG process to guarantee high-quality game content.

Contribution 9 [49] explores the application of the MCTS algorithm in Amazons, and proposes MG-PEO, a combination of several optimization strategies called “Move Groups” and “Parallel Evaluation”, to improve performance and efficiency. Move groups is a method used to divide a move into two parts, which can largely reduce the branching factor of the game tree and focus the visited nodes of MCTS more on promising subtrees. Parallel evaluation uses multiple threads to speed up the evaluation of leaf nodes. A technique called “Shallow Rollout” is also used to expand the small, fixed depth of nodes and apply the evaluation function to them. Ablation experiments show that MG-PEO agents achieve higher win rate than vanilla MCTS agents, indicating the effectiveness of the proposed optimization strategies.

Contribution 10 [50] explores the application of vision transformer (ViT) to imitate the strategy of human Go players. While AlphaGo has achieved superhuman performance in Go, it often comes up with moves beyond human players’ comprehension. This paper adopts supervised learning to train a policy network with vision transformer as its backbone architecture, which achieves professional-level play while mimicking the decision-making of human players. Experiments show that this ViT model achieves higher accuracy than models based on convolutional neural networks (CNNs).

Contribution 11 [51] develops a vehicle driving game and explores the application of different reinforcement learning, including PPO and SAC [52], to train autonomous driving agents. By designing a series of tasks with rising complexity as a curriculum, the agent can gradually learn to navigate through multiple targets within lane boundaries without running into obstacles. Rewards are carefully designed in each task for effective and robust driving. Experiments show that though SAC agents learn faster in early phase, PPO agents achieve better performance and adaptability than SAC, shedding light on the applications of autonomous systems in real-world scenarios.

Finally, as the Guest Editor, it was my pleasure to work with the editorial staff of Algorithms to prepare this Special Issue.